Is it possible, from a related list, to have a custom button that opens a lightning modal dialog, as is the behavior of the standard buttons of these related lists?
My research tells me that it isn't possible given that the onclick JavaScript buttons are no longer allowed in the Lightning experience but would like to confirm this with you.
Here are some pictures to show better what I want.
Same as answered in your earlier question. You need to add the button to the Salesforce1 and Lightning Actions section for the button to show in Lightning Experience. Currently it will just show in Classic version only.
You could use Visualforce with standard list controller. Using a standard list controller is very similar to using a standard controller. First you set the standardController attribute on the component, then you set the recordSetVar attribute on the same component.
For example, to associate a page with the standard list controller for accounts, use the following markup:
<apex:page standardController="Account" recordSetVar="accounts">
The recordSetVar attribute not only indicates that the page uses a list controller, it sets the variable name of the record collection. This variable can be used to access data in the record collection.
Below is stack exchange link for one use case where im selecting a list of records on the list view and iterating on them to export into excel in the below example.
Note: even though the below link is for a different issue I just wanted to show you the use case. Issues with List API in Read Only mode VF page
Im my above example value="{!selected}" is the equivalent for GETRECORDIDS.
Also to get the selected record id's in apex I'm pointing here for another SFSE link Creating a custom list-view button that handles multi-record selection
I tested my vf code in lightning and it had a weird bug before summer 18 where the the selected records were inconsistent but they fixed it with summer 18 and everything looks good now. I will update here with the link when I get the chance.
Option 2: I know flow's are not as flexible as apex but you could give it a shot too based on what you are trying to achieve. Below is a link from Andrew Facet blog about leveraging flow Visual Flow with List View and Related List Buttons
Best Answer
With the new Winter 17 release you can Launch a Lightning Component from an Action
So if your modal would be this lightning component, you can launch it as an action