Formula for button help

Try creating a new Javascript button, type should be "list button" (last option, without checkboxes). Call it "New Payment".

Implement your check there, something like

if({!Opportunity.Product_Rollup__c} == 0){
    alert('Uh oh');
} else {
    location.href = '(url to make new Payments here)';
}

Finally go to the Opportunity page layout (all of them if you have several?), edit related lists, untick the checkbox next to standard "New" button and add your custom one.

As this is managed and in future the package devs might decide they need new parameters passed to this VF page or something crazy like that I'd advise you to try not to make the url /apex/SomePage... and even not /a07/e?someParams (which format is OK for normal "new" pages) but rather try using URLFOR function.

P.S. Make no mistake - I'd recommend preventing saving by adding a Validation rule or some other "hard check" (trigger?) to the object anyway. It's a bit of security by obscurity, Salesforce power user might be able to hand-craft the link, access the Payments tab and hit "New" there or he'll use a mobile device with an app ;) This is valid visual client-side helper/reminder but such logic should be replicated server-side.

EDIT to include comparison of old & new URLs:

Original URL:

https://(instance).salesforce.com/setup/ui/recordtypeselect.jsp?
ent=01Id0000000jpap&
retURL=%2F006K00000069AnT&
save_new_url=%2Fa0B%2Fe%3FCF00Nd0000004bboJ%3DTest%2BPerson-Conference%2BRegistration-%252410-2013-01-18%26CF00Nd0000004bboJ_lkid%3D006K00000069AnT%26scontrolCaching%3D1%26retURL%3D%252F006K00000069AnT

Custom "new":

https://(instance).salesforce.com/setup/ui/recordtypeselect.jsp?
ent=01Id0000000jpap&
retURL=%2F006K00000069AnT&
save_new_url=%2Fa0B%2Fe%3FretURL%3D%252F006K00000069AnT

So the only difference is in the content of form prepopulation to be performed when user cicks "Save & New". Let's urldecode it:

/a0B/e?
CF00Nd0000004bboJ=Test+Person-Conference+Registration-%2410-2013-01-18&
CF00Nd0000004bboJ_lkid=006K00000069AnT&
scontrolCaching=1&
retURL=%2F006K00000069AnT

So - the package developers just made sure that Opportunity Id & Name will be prefilled in "Save & New" too. But this works for me already (I have custom object under Account, not Opportunity and the Account Name is prefilled on subsequent "save & new" without problems).

Long story short - experiment but I doubt you need it ;)

You can achieve this with the help of a Visual Force page and a Controller. In the Visual Force page put the action name in the apex:page tag itself so that the action method invoked when this page is requested by the server. Pass the Case Id and current (list view) page URL to the Controller on the click of HYPERLINK Formula field. The current Page URL will be used to return to the same page where the HYPERLINK formula field is clicked. In the Controller update the Case and create a PageReference based on the retURL parameter. At the end forward to this PageReference and you will get the updated list view.

HYPERLINK Formula field

HYPERLINK("javascript:function encURI(){return encodeURIComponent(window.location);};javascript:window.location='/apex/casepage?caseid="+ Id +"&retURL='+encURI()" ,'Accept Case', '_self')

Visualforce Page

<apex:page controller="MyCaseController" action="{!updateCase}" >
</apex:page>

Controller

public class MyCaseController {
    public String retUrl {get;set;}
    public String caseId {get;set;}
    public MyCaseController(){
        retUrl = ApexPages.currentPage().getParameters().get('retURL');
        caseId = ApexPages.currentPage().getParameters().get('caseid');
    }
    public PageReference updateCase(){
        PageReference retPage = new PageReference(retUrl);
        //Put your own logic here
        Case cs = new Case(Id=caseId);
        cs.Subject = 'New Case Subject';
        update cs;
        return retPage;
    }
}