Is there a point of no return in Fallout 4?

The Molecular Level quest both is and isn't a point of no return, fundamentally it requires you to make a choice - what you do need to consider is that speaking to a faction will provide you the infiltration quest for the Institute but also make the equivalent quest for the other factions unavailable. It is possible to reconnect with the factions you didn't choose after completing Institutionalized though. The actual point of no return varies by faction alignment.

The game is seemingly inconsistent in warning you when you're about to make permanent changes to your faction reputations.

In the case of the Institute quest line, speaking to Allie to start the Mass Fusion quest will pop up a nice warning stating you're about to become permanently hostile to the Brotherhood of Steel. There isn't a subsequent warning when you do the same to the Railroad, you'll just get lots of failed quest spam if you've got quests outstanding but in fairness, the mission is specifically to kill the Railroad so...

For the Railroad, I believe the switch occurs silently at the start of the quest Precipice of War. I definitely didn't receive a message and the conversation to start Precipice of War seemed to flow continually from the conversation that ended Operation Ticonderoga - the Brotherhood of Steel just stormed in and then I got lots of quest failure spam.

For the Brotherhood of Steel, the mission that makes you hostile with the Railroad is Tactical Thinking.

Yes, once you've finished the "story" you can continue playing, but there will be less faction missions available because of who you destroyed in order to finish the story.