Local phase invariance of our lagrangian

This will only answer part of your question, namely "What is the physical motivation for wanting our lagrangian to be invariant wrt to local phase changes? What are some cases where we want local U(1) invariance(?)". I will try to explain why gauge invariance is forced upon us in a particular example. On a general note, you should usually think of local/gauge invariance as a redundancy of our description of nature and not a fundamental symmetry of nature (these are global symmetries).

The Lagrangian for a theory is constrained in various ways. One very important way is by the assertion that all particles must be describable by unitary (irreducible) representations of the Poincare group, since that is the observed symmetry group in our universe. The analysis of the possible representations of the Poincare group is an exercise in Lie group theory. Turns out, to describe a spin 1 massless particle (photon), we need two degrees of freedom (for each momentum). But, if we want to write a local theory, a straightforward way to achieve that is to write a Lagrangian in terms of scalar fields, (4-)vector fields, higher rank tensor fields, etc. To embed two degrees of freedom, the smallest of these objects is a 4-vector field. So, the question arises, where do the two other degrees of freedom come from? And the answer is that those aren't physical degrees of freedom. So if you write down some form of the Lagrangian, you better ensure that those redundancies are taken care of. In simple terms, that is the origin of gauge invariance in the free theory of the photon. Now if you go on to couple this field to a charged scalar particle, for example, you'll need to make sure that this gauge/local symmetry transformation doesn't screw up the full coupled Lagrangian, and this leads to the phase ambiguity you have mentioned.

For detailed mathematical justifications of the claims I make, you can check out Ch. 8 of Matt Schwartz's book on QFT and Standard Model.

If you use Noether's theorem, there is a locally conserved current associated with local $U(1)$ symmetry. We can then identify this current as electric current. Without the symmetry, we wouldn't have a locally conserved current to use for electric current, so it's pretty important. As an additional note, we can find Lagrangians that have $SU(2)$ or $SU(3)$ local gauge invariance. When we do this, we find that the Lagrangian for $SU(3)$ describes the behavior of the strong interaction between quarks.