[SalesForce] Custom button to visualforce page in new window popup size

It's in technically possible (although unsupported) to use a custom button to inject arbitrary content into the page. See Changing the color of a custom button

You can likley modify this technique to REQUIRESCRIPT jquery+jqueryUI, and then REQUIRESCRIPT your own custom script file that creates the modal window.

If this works it would be subject to possibly breaking in future releases.

The commandButton by default will execute both the onClick event as well as the apex action. You will need to replace the commandButton with a normal HTML input button, then from within your JavaScript popup you need to call an apexFunction that will call the actual apex method. So your code should look something like this:

<apex:page>
    <apex:form>
        <script>
            function showDialog()
            {
                // show the dialog with with another button to proceed
                // which will have onclick event to call the function below
            }

            function dialogButtonClicked()
            {
                // do something more JS if needed and then call the apex login method
                controllerLogin();
            }
        </script>
        <apex:actionFunction name="controllerLogin" action="{!login}" />

        <!-- your login form here -->
        <input type="button" value="Login" onclick="showDialog();" />
    </apex:form>
</apex:page>