PUBG Battlegrounds – How to Recover 50% of Lost Crates after Update #1.0

Your problem is most likely due to the controller not being assigned to your person.

The other controller, even though it has a headset plugged in is treated as belonging to a guest (or other profile if it is signed in) and, hence, will not have access to the party.

You can fix the issue by pairing the controller with your person manually. However, reading more on the topic, it seems that if you have set up Kinect Sign-in, the console will automatically recognise that you picked up a different controller and will assign it to you as you would expect.

Source: How to assign a profile to an Xbox One Wireless Controller

The problem is that overkill haven't sorted out the connection problems with the game. Its not that people aren't trying to join you its the fact that every time someone try's it just comes up with connection errors, failed to join. Overkill are supposed to be worked on a patch but that was over a week ago.