Well, you are doing it correctly. You will be required to add and event handler on the <a> tag you have defined. I tried it using jQuery dialog.
Apex Page:
<apex:page controller="ErrorMSG" >
<apex:pageMessages escape="false" rendered="true" ></apex:pageMessages>
<link rel="stylesheet" href="//code.jquery.com/ui/1.11.4/themes/smoothness/jquery-ui.css"/>
<script src="//code.jquery.com/jquery-1.10.2.js"></script>
<script src="//code.jquery.com/ui/1.11.4/jquery-ui.js"></script>
<link rel="stylesheet" href="/resources/demos/style.css"/>
<div id="dialog" title="Basic dialog">
<p>This is the default dialog which is useful for displaying information. The dialog window can be moved, resized and closed with the 'x' icon.</p>
</div>
<script>
$j = jQuery.noConflict();
$j( "#popup" ).click(function(){
$j( "#dialog" ).dialog();
})
</script>
</apex:page>
Controller
public class ErrorMSG {
public ErrorMSG(){
ApexPages.addMessage(new ApexPages.Message(ApexPages.Severity.WARNING , 'You have received an error <b><u> <span id="popup">Click for Details</span><u></b>'));
}
}
Result
You will be required to refine the code and surround the dialog by outputpanel and re-render it if there is a failure and hide dialog at start.
Best Answer
You can display error messages using the addError method on sobject or its fields.
Salesforce docs on the sobject instance addError method: https://developer.salesforce.com/docs/atlas.en-us.198.0.apexcode.meta/apexcode/apex_methods_system_sobject.htm#apex_System_SObject_addError
More info on exception handling: http://wiki.developerforce.com/page/An_Introduction_to_Exception_Handling