Xbox One included chat headset not working

Your problem is most likely due to the controller not being assigned to your person.

The other controller, even though it has a headset plugged in is treated as belonging to a guest (or other profile if it is signed in) and, hence, will not have access to the party.

You can fix the issue by pairing the controller with your person manually. However, reading more on the topic, it seems that if you have set up Kinect Sign-in, the console will automatically recognise that you picked up a different controller and will assign it to you as you would expect.

Source: How to assign a profile to an Xbox One Wireless Controller

The kind of sharing you are referring to, requires your brothers HOME xbox to be the one you are playing from and vice versa.

SO if you have 2 Xboxes and 2 Accounts:

Your brothers home console needs to be your xbox, where you then log in with your account to play on.

Your home console needs to be on your brothers xbox, where he is then logged in with his profile, still able to play your games.

This way is how i have been using it for the past 3 years without any issues. The important step is to make sure you set each others Home Console on the other persons Xbox, to ensure he has access to the content, even tho he is not logged in.

Hope this clears things up.