Try creating a new Javascript button, type should be "list button" (last option, without checkboxes). Call it "New Payment".
Implement your check there, something like
if({!Opportunity.Product_Rollup__c} == 0){
alert('Uh oh');
} else {
location.href = '(url to make new Payments here)';
}
Finally go to the Opportunity page layout (all of them if you have several?), edit related lists, untick the checkbox next to standard "New" button and add your custom one.
As this is managed and in future the package devs might decide they need new parameters passed to this VF page or something crazy like that I'd advise you to try not to make the url /apex/SomePage... and even not /a07/e?someParams (which format is OK for normal "new" pages) but rather try using URLFOR function.
P.S. Make no mistake - I'd recommend preventing saving by adding a Validation rule or some other "hard check" (trigger?) to the object anyway. It's a bit of security by obscurity, Salesforce power user might be able to hand-craft the link, access the Payments tab and hit "New" there or he'll use a mobile device with an app ;) This is valid visual client-side helper/reminder but such logic should be replicated server-side.
EDIT to include comparison of old & new URLs:
Original URL:
https://(instance).salesforce.com/setup/ui/recordtypeselect.jsp?
ent=01Id0000000jpap&
retURL=%2F006K00000069AnT&
save_new_url=%2Fa0B%2Fe%3FCF00Nd0000004bboJ%3DTest%2BPerson-Conference%2BRegistration-%252410-2013-01-18%26CF00Nd0000004bboJ_lkid%3D006K00000069AnT%26scontrolCaching%3D1%26retURL%3D%252F006K00000069AnT
Custom "new":
https://(instance).salesforce.com/setup/ui/recordtypeselect.jsp?
ent=01Id0000000jpap&
retURL=%2F006K00000069AnT&
save_new_url=%2Fa0B%2Fe%3FretURL%3D%252F006K00000069AnT
So the only difference is in the content of form prepopulation to be performed when user cicks "Save & New". Let's urldecode it:
/a0B/e?
CF00Nd0000004bboJ=Test+Person-Conference+Registration-%2410-2013-01-18&
CF00Nd0000004bboJ_lkid=006K00000069AnT&
scontrolCaching=1&
retURL=%2F006K00000069AnT
So - the package developers just made sure that Opportunity Id & Name will be prefilled in "Save & New" too. But this works for me already (I have custom object under Account, not Opportunity and the Account Name is prefilled on subsequent "save & new" without problems).
Long story short - experiment but I doubt you need it ;)
I think it's the way that you're attempting to nest inside that URLFOR. Try restructuring it so that the Id param is outside the CASE (since it's gonna be the same no matter what), and also don't use quotes around those $Action references. maybe like this (whitespaced for readability)
<apex:page standardController="Abv_Core_KAM_Engagement_Plan__c" action="{!URLFOR(
CASE(Abv_Core_KAM_Engagement_Plan__c.RecordType.Name,
'Abv_Core_KAM_Engagement_Plan', $Action.Abv_Core_KAM_Engagement_Plan__c.Edit,
'Abv_Core_KAM_Strategic_Initiative', $Action.Abv_Core_KAM_Engagement_Plan__c.Edit,
$Action.Abv_Core_KAM_Engagement_Plan__c.Edit
), Abv_Core_KAM_Engagement_Plan__c.Id) }" >
Although, I'm a bit confused because it looks like you're sending it to the same place every time...
Best Answer
You need to make sure the page tag is set up correctly, something like this:
You also need to make sure that the page is visible to the user, in the security section of the page, and that the user has create permission on the custom object.