Impossible to manage castle

The best way of getting a castle or town is to capture one yourself. Keep an eye out along the borders where towns and castles frequently change hands. Those places usually have a rather weak garrison so you should be able to take it without too much trouble.

When you capture a town or castle, you can make the request to your king that you want to keep it. Unless you've just recently been rewarded a fief, the King will almost always grant you the town/castle you've just captured. Don't count on the king giving you one otherwise unless your reputation with him and the other lords are all super high and you manage to convince them to support your cause.

If the castle near your village is already owned by a lord (almost 100% of the time it is), then you are out of luck, you can't get that one unless an enemy faction captures it and you retake it.

Circumventing this limit is rather tedious and requires you to manually add more spots for the lords to stand in in the castle/city halls' configuration files. The lords stand at pre-determined spots, and when there aren't enough spots left, the rest of the lords are omitted from showing up. Exiting and re-entering the town shuffles which lords show up on visually on the feast, though it would take quite some exiting/reentering to greet every single lord in attendance.

Some castles and cities do have larger halls, allowing more lords to visibly attend. Which ones are the larger ones I am unsure, but the Nord have the largest I can recall at least.