[SalesForce] How to open page in a new popup window

You can try to use window.open method:

window.open('some_URL');

It will open an URL in a new browser window. Use onclick behavior for that.

Example:

<apex:outputLink onclick="window.open('/apex/SomePage')">Click me!</apex:outputLink>

Or just using pure html:

<script>
function urlOpener(url){
    window.open(url);
}
</script>

<a href="#" onclick="urlOpener('/apex/SomePage');">Click me!</a>

You can use either Window.postMessage to safely transfer data between the popup and the original window; this is the preferred method. Alternatively, you can also set an Apex Cookie that can be shared between the two windows. Using expando properties of window.opener should not be used, because you can't even verify that the receiving page is even the original page which opened the popup; this might cause undesirable behavior.