Fallout – When I enter the Institute for the first time I can’t progress

The Battle of Bunker Hill is a quest that is given to you by the Institute, which is why they weren't interested in being warned - it was their mission.

The reason you're allowed to inform other sides is because you'll always get this mission regardless of faction allegiance however if you're working for the Railroad they'll want informing because the synths you've been sent to kill by the SRB are in the process of being "saved"/smuggled out of the Commonwealth by the Railroad.

If you're aligned with the Brotherhood of Steel and warn them, then they'll request that you kill the synths.

At this point in the game it is possible to still be on friendly terms with every faction, if this is the case you can turn up to this mission and walk through multiple opposing factions trying to kill each other, with all of them acknowledging you as on their side.

The Molecular Level quest both is and isn't a point of no return, fundamentally it requires you to make a choice - what you do need to consider is that speaking to a faction will provide you the infiltration quest for the Institute but also make the equivalent quest for the other factions unavailable. It is possible to reconnect with the factions you didn't choose after completing Institutionalized though.

The game is seemingly inconsistent in warning you when you're about to make permanent changes to your faction reputations. The actual point of no return varies by faction alignment.

In the case of the Institute quest line, speaking to Allie to start the Mass Fusion quest will pop up a nice warning stating you're about to become permanently hostile to the Brotherhood of Steel. There isn't a subsequent warning when you do the same to the Railroad, you'll just get lots of failed quest spam if you've got quests outstanding but in fairness, the mission is specifically to kill the Railroad so...

For the Railroad, I believe the switch occurs silently at the start of the quest Precipice of War. I definitely didn't receive a message and the conversation to start Precipice of War seemed to flow continually from the conversation that ended Operation Ticonderoga - the Brotherhood of Steel just stormed in and then I got lots of quest failure spam.

For the Brotherhood of Steel, the mission that makes you hostile with the Railroad is Tactical Thinking.