Access to the file is based on access to the parent entity it is shared with. In your case, anyone who can see the custom object will have access to the file. If the org wide default sharing for the custom object is private, limited people would have access to the record, such as the record creator and administrators.
If the file is shared with a public group, then everyone would have access to the file because everyone has access to the group. To limit file access to only group members, you would need to share the file with private groups.
You can share files with groups through the Connect API by creating a FeedItem on the group. If the file is already uploaded into Salesforce, then while creating the FeedItem, you can specify the ContentVersion Id and it will share the file.
To share a file with a private group without creating a FeedItem (such as if you want to avoid the "noise" of doing many shares), you can use the SObject API and the ContentDocumentLink entity. This can create a share between the file and a parent entity (such as a group).
If your use requires the file to also be shared with the custom object, then I would suggest uploading it to the custom object as you are and having the custom object share settings be private. Then, either create a FeedItem or ContentDocumentLink share to PRIVATE groups.
If it is not required to be shared with the custom object, then you can do the initial upload (as a FeedItem) to a private group, and then share it to additional private groups through FeedItems or ContentDocumentLinks.
Best Answer
I think you can do this using a dataloader update on the user object, I'm pretty sure the field you want is "DefaultGroupNotificationFrequency" with the options: