Why are the Lagrangian points $L_1$, $L_2$ & $L_3$ unstable?

I) OP asks (v5):

Why are the Lagrangian points $L_1$, $L_2$ & $L_3$ unstable?

Actually, that's a great & non-trivial question. One should not make conclusions based on merely the effective potential $V$ (consisting of gravity & the centrifugal potential) alone, because the Coriolis force may be important.

For instance, the global maximum points $L_4$ & $L_5$ of $V$ (which naively would be the last places to suspect a stable equilibrium point), are sometimes stable, cf. e.g. this Phys.SE post!

II) Let us here analyze the stability of an arbitrary point with small initial velocity $v_0$ in the orbital plane$^1$ $mathbb{R}^2cong mathbb{C}$ in the presence of the Coriolis force. Let us pick coordinates $zinmathbb{C}$ so that the initial position is in the origin $z=0$.

Let us start by studying critical/stationary points, i.e. Lagrange points. We use the following theorem mentioned in Ref. 1:

Theorem. Given a Hessian ${bf H}$ for an effective potential $V:mathbb{R}^2to mathbb{R}$ at a Lagrange point. Let $Omega$ be the angular velocity. The following 3 conditions are necessary & sufficient conditions for the Lagrange point to be stable:

1. $C~:=~det({bf H})~geq~0. $

2. $B~:=~{rm tr}({bf H})+4Omega^2~geq~0. $

3. $D~:=~B^2-4C~geq~0. $

NB: The theorem ignores higher-order terms in $V$, which could become important if $det({bf H})=0$.

A proof of the theorem is given in my Phys.SE answer here, which also discusses the case of a local maximum.

1. Case of a local minimum$^2$ $det({bf H})geq 0 wedge {rm tr}({bf H})geq 0$: It is stable because all 3 conditions are satisfied. To see the third condition, observe that $$begin{align} D~=~&B^2-4C~=~({rm tr}({bf H})+4Omega^2)^2-4 det({bf H}) cr ~=~&{rm tr}({bf H})-4 det({bf H}) + 8{rm tr}({bf H})Omega^2 +16Omega^4 cr ~=~&(H_{11}-H_{22})^2 +4H_{12}^2 + 8underbrace{{rm tr}({bf H})}_{geq 0}Omega^2 +16Omega^4 ~>~ 0.end{align}tag{1} $$ $Box$

2. Case of a strong saddle point $det({bf H})<0$: This is the case of the Lagrange points $L_1$, $L_2$ & $L_3$, which OP asks about. It is unstable, because it violates the first condition. $Box$

III) To gain some intuition for above conclusions, let us also analyze what happens away from the Lagrange points.

3. Case of a non-critical point: The specific effective force $f=left. -nabla Vright|_{z=0}neq 0$ is non-zero. Then in the region where we can treat the specific effective force as constant, the EOM with specific Coriolis force $2vec{v}timesvec{Omega}$ reads$^3$ $$ begin{align}ddot{z} ~approx~&underbrace{-2iOmega dot{z}}_{text{Coriolis}} +fcr Updownarrow ~& crfrac{d}{dt}left( expleft{2iOmega t right}dot{z}right)~approx~&expleft{2iOmega t right}f. end{align}tag{2}$$ Hence the velocity is $$ dot{z}~approx~frac{f}{2iOmega}+ left(v_0-frac{f}{2iOmega}right) expleft{-2iOmega t right}, tag{3} $$ and the position is $$ z~approx~ underbrace{frac{ft}{2iOmega}}_{text{drift}}+ underbrace{frac{1}{2iOmega}left(frac{f}{2iOmega}-v_0right)}_{text{complex amplitude}} left(underbrace{expleft{-2iOmega t right}}_{text{circle}}-1right) tag{4} $$ $$~approx~ v_0t + left(f-2iOmega v_0right)frac{t^2}{2}left(1- frac{2iOmega t}{3}right) + {cal O}(t^4) . tag{5} $$ The test particle performs a circular motion with a drift along equipotential lines in the orbital plane. (To the $color{red}{rm right}$ if you are looking $color{red}{rm downhill}$.) In any case, the test particle is unstable at a non-critical point. $Box$

   

   $uparrow $ Fig. 1: A possible horseshoe orbit along equipotential lines.

The above description provides some inside into why local minimums and (sometimes) local maximums are stable, while saddle points are always unstable. If the test particle tends to drift along equipotential lines, then near local minimums and maximums, it ends up in small concentric loops, while equipotential lines always point away from saddle points.

$uparrow $ Fig. 2: A test particle tends to drift along equipotential lines due to the Coriolis force.

References:

1. J. Binney & S. Tremaine, Galactic Dynamics, 2nd edition (2008); p. 181-182.

--

$^1$ There could also be an oscillatory motion perpendicular to the orbital plane, which we ignore.

$^2$ The above analysis is valid for any effective potential $V$. In the restricted 3-body problem, which OP asks about, the effective potential $V$ has no local minimum.

$^3$ Here we choose the orientation of the angular velocity $vec{Omega}$ to be prograde/counter-clockwise/positive direction of rotation.

I find the most intuitive way to think about Lagrange points is to consider the gravitational potential energy and a surface representing the potential.

The Lagrange points exist in the space around two massive objects, say the Sun and the Earth. You can think of both of these as producing a gravitational potential well shaped a bit like a cone or funnel pointing down but with sides that make a curved profile. The gravitational potential as a function of distance $r$ is given by $U = -Gm/r$. He's a nice picture grabbed from Wikipedea 2. ^{Image source: By AllenMcC. - Own work, CC BY-SA 3.0, Graviational Potential Well}

Now, since there are two massive bodies there are two of these wells near each though the one centred on the more massive body will be bigger. You can imagine that there would be a saddle point between the two wells were the surface would be flat. This would represent a point where the forces of gravity pulling towards each body are in balance. (I couldn't find a good image of two wells.)

The thing is that the smaller body (say the Earth) is orbiting the larger body (Sun) or more accurately they are both orbiting their common barycentre. The Lagrange points are stationary as seen from the Earth so we need to consider things from a frame of reference that is rotating with the Earth. In such a rotating frame of reference we have to introduce a fictitious centrifugal force away from the centre of rotation. This the addition of this force changes potential surface with the gravitational wells. Since the force is acting away from the centre an object near the centre would have a higher potential than one farther away so the potential for this force would look kind of like an inverted bowl. When you put together the effects of gravity from both bodies and the centrifugal force you get somewhat complicated effective potential surface. Here's an image, thanks once again to Wikipedia.

^{Image source: By User:cmglee - Own work, CC BY-SA 3.0,, Lagranian Points Equipotential}

So, the five Lagrange points are places where the slope of this potential surface is zero. You could imagine yourself sitting there and not sliding away from them if you were careful. The $L_1$, $L_2$, $L_3$, are all saddle shaped whereas the $L_4$ and $L_5$ points occur at local maxima. They are like hilltops but quite flat hilltops.

Nonetheless, the the $L_4$ and $L_5$ points are not local minima so how are they stable? Well, in the rotating reference frame the centrifugal force is not the only fictitious force that has to be introduced. Once something starts moving it also experiences a Coriolis force perpendicular to its motion. If a satellite moves away from the $L_4$ or $L_5$ points it's this Coriolis force that will nudge it back if not exactly back to the Lagrange point then to an orbit around it.

Your analysis is missing two key elements-- the force of gravity, and how that force (and the centrifugal force) vary with r. You need those. Go into the rotating frame around the center of mass of the system, and put in the gravity of both objects, as well as the centrifugal force (it's there because we are in a rotating frame of reference). You should find that all these forces add to zero at the L1 and L3 points. Then notice that the force points away from the L1 and L3 points if you vary r a little bit. If you don't have the calculus mathematical skills to be able to see this, you can just say it-- an object right at L1 or L3 can go around with the same orbital period as the orbiting gravity sources, but any deviation away from that spoils that. This is what you said, but you never specified the forces involved, so it is not enough to show that it is unstable because the instability does depend on those forces. How much you want to show that, or just say it, depends on the depth of your presentation.