[SalesForce] How to place a Custom Button on list view (Lightning)

You can create a Visualforce page using the standard list controller, and define it as a list button.

Refer this link: Adding Custom List Buttons using Standard List Controllers

These display as expected in Lightning:

You can create new actions for your objects, in the form of buttons and links, by defining them on an object. JavaScript buttons and links aren’t supported in Lightning Experience, but Visualforce (and URL) items are. The process of defining Visualforce buttons and links is identical to that in Salesforce Classic, so we won’t bother to show it here.

Don't forget the Custom Object must be Searchable in order to add your new button to the List View Search Layout.

You can then use Lightning for Visualforce (Lightning Out) to render a lightning component in your VF page if you're doing a primarily lightning component project.

Trailhead Module: Use Visualforce in Lightning Experience

JavaScript buttons don't work the way you expect, and I'm pretty sure that the code you're trying to use won't work. That said, you have an alternative you can use (tested/verified).

First, create a new Apex Class:

public class CaseAcceptOwnership {
    PageReference cancel;
    Case[] records;
    public CaseAcceptOwnership(ApexPages.StandardSetController c) {
        records = (Case[])c.getSelected();
        cancel = c.cancel();
    }
    public PageReference updateCases() {
        for(Case record: records) {
            record.OwnerId = UserInfo.getUserId();
        }
        update records;
        return cancel.setRedirect(true);
    }
}

Next, create a Visualforce page:

<apex:page standardController="Case" recordSetVar="cases" extensions="CaseAcceptOwnership" action="{!updateCases}">
</apex:page>

After that, go to Setup > Customize > Cases > Buttons, Links, and Actions, and create a new List Button (with select checkboxes), choose Visualforce for the Content Source, and choose the new page you created.

Finally, go ahead and add this button to your Search Layout, and you're done. The user can select records, and when they click the button, they will own the selected cases. You may want to add some error handling or something, but this should get you started.

Edit: Example Unit Test

Again, does not cover failure conditions, so make sure you add additional error handling if there's a possibility of failure.

@isTest class CaseAcceptOwnershipTest {
    @isTest public static void test() {
        // Get All Fields
        String[] fields = new List<String>(sObjectType.User.fields.getMap().keySet());
        // Get Test User Data
        Id userId = UserInfo.getUserId();
        User tempUser = Database.query('select '+String.join(fields,',')+' from user where Id = :userId');
        // Clone and clear out problematic fields
        tempUser = tempUser.clone(false, false, false);
        tempUser.FederationIdentifier = null;
        tempUser.Alias = 'testUser';
        tempUser.CommunityNickname = 'testUser';
        tempUser.UserName = 'testUser'+Math.random()+'@not-a-real-domain.or-tld';
        
        // This automatically does a DML, saves us one. Also avoids MIXED_DML errors.
        System.runAs(tempUser) {
            Account accountRecord = new Account(Name='Test');
            Contact contactRecord = new Contact(LastName='Test');
            Case caseRecord = new Case(Subject='Test',Status='Open', OwnerId=tempUser.Id);
            // Multi-Object insert for performance
            insert new sObject[] { accountRecord, contactRecord, caseRecord };
    
            contactRecord.AccountId = accountRecord.Id;
            caseRecord.ContactId = contactRecord.Id;
            caseRecord.AccountId = accountRecord.Id;
            // Assocate records together
            update new sObject[] { contactRecord, caseRecord };
        }
        Test.startTest();
        // Instantiate controller as it would be from the Visualforce page
        ApexPages.StandardSetController standardSetController = new ApexPages.StandardSetController([
            select id from case
        ]);
        standardSetController.setSelected(standardSetController.getRecords());
        // And the extension controller
        CaseAcceptOwnership controller = new CaseAcceptOwnership(standardSetController);
        // Avoid MIXED_DML error, run as yourself
        System.runAs(new User(Id=UserInfo.getUserId())) {
            ApexPages.PageReference result = controller.updateCases();
        }
        Test.stopTest();
        // Verify ownership change
        System.assertEquals(1, [SELECT COUNT() FROM Case WHERE OwnerId = :UserInfo.getUserId()]);
    }
}