[SalesForce] How to add a button from custom object to a page layout like Account

Try using

window.open('{!URLFOR($Action.Contact.NewContact, Contact.Id )}');

Update : try this : click on inspect element on the lookup button and you can find Id of the lookup as con4_lkwgt (I would assume this to be same since its a standard field) and click on the account name field and get the ID as con4

/003/e?con4={!Account.Name}&con4_lkwgt={!Account.Id}&&retURL=/{!Account.Id}

Refer this blog for more insight on how to override the buttons http://raydehler.com/cloud/clod/salesforce-url-hacking-to-prepopulate-fields-on-a-standard-page-layout.html

You only get this option if there are Visualforce Pages available that implement the correct controller. If you see a page like this in your org, you should have the option enabled.

<apex:page standardController="MyObject__c">

This StandardController implementation also enables you to use the Visualforce Page in Detail Buttons.

There may be other attributes. The only one that is relevant is recordSetVar. If this attribute is set (to any value), then it implements StandardSetController instead of StandardController and can be used in List Buttons instead.