How can I use a formula to calculate list price based on custom fields for certain products? I though I could I set the list price at 0 since it's required then create a process builder to update the field based on the fields that are filled out on the product line, but that doesn't appear to be possible.
List Price is always associated to a Custom Pricebook. You are only able to add a List Price within that Pricebook. Refer to the below from Considerations for Setting Prices documentation:
So, in order for you to update the List Price whenever you are adding an Opportunity Product, you have to basically update the List Price on the Pricebook Entry associated with that Product.
With a quick test, the following Process Builder works fine. You will need to create a Process Builder on Opportunity Product and choose the associated Pricebook Entry's List Price to be updated. You can adjust the formula based on your custom fields here.
While this works, I will not recommend to use this approach though. List Price are associated with a Price book. So wherever that Price book will be used, changes with this approach will have an impact on all such Opportunity Products associated to different Opportunities wherever the Price book is referenced. Instead you should create another field on the Opportunity Product and use that for your use case.