Invariance of the relativistic interval

You are asking a question right at the foundations of relativity.

There are two ways to develop special relativity. In the first approach one talks about inertial frames, the principle of relativity, and the invariance of the maximum speed (called speed of light). In the second approach one merely announces that spacetime is a kind of space (mathematically an example of a 'differentiable manifold') and that invariant distances in the space are given by the interval (as in the equations you are asking about).

In the first approach one can derive the Lorentz transformation from the two main principles (relativity and speed of light), and then one can prove that the interval is invariant. I wouldn't call it obvious, but it does come out reasonably quickly. In the second approach one defines the Lorentz transformation as that kind of coordinate transformation which preserves the interval. One then derives other results, such as that something traveling at the maximum speed relative to one coordinate frame will also have that same speed relative to others. It must do, because it moves between events separated by a null interval. Such an interval is null no matter who is measuring it, since it is invariant after all!

You get if from the postulate of the constancy of the speed of light. If we take the speed of light to be the same in all inertial frames, then it follows that

$$c^2= left(frac{dmathbf{r}}{dt}right)^2 = left(frac{dmathbf{r'}}{dt'}right)^2$$

That is,

$$dr^2 = c^2dt^2 implies dr^2 - c^2dt^2 =0$$

and

$$dr'^2 = c^2dt'^2 implies dr'^2 - c^2dt'^2 =0$$

Setting the two equal we get

$$ dr^2 - c^2dt^2 = dr'^2 - c^2dt'^2.$$

From this derivation, you can easily see that the choice of the Minkowski metric signature being 2 or -2 is arbitrary.

A proof can be made with the following thought experiment. (In relativity thought experiments are ubiquitous). I'm sorry if I can't make drawings, but if you search "light clock experiment" you will see what I'm talking about with drawings and possibly a more detailed explanation.

Take a box with two reflective walls at distance $d/2$ and imagine that a beam of light is bouncing back and forth, and imagine that a detector at one of the two walls counts the light bounces and uses it to track time.

If you look at the box in its rest frame you'll see the light beam bouncing every $Delta t = d/c$ seconds. Let's say that there is a built in mechanism that makes the box, i don't know, ring an alarm, after $n$ bounces. That is, the box alarm will go off after a time $n Delta t$.

Now let's go into a reference frame moving at constant velocity $v$ with respect to the original frame, along the direction orthogonal to the light beam. The time interval for a full bounce is $Delta t'$ and the distance travelled is $$ mathrm{dist} = 2sqrt{(Delta x')^2 + (Delta y')^2},,qquadDelta x = frac{vDelta t'}{2},, $$ where $Delta y'$ is the distance travelled in the vertical direction in the new frame. It must equal $d/2$ because intervals orthogonal to $v$ don't get transformed under a boost (see later for an explanation of that). From the fact that $c$ is the same in both frames we conclude $$ c^2 (Delta t')^2 = v^2 (Delta t')^2 + d^2;;Rightarrow;;(Delta t')^2 = frac{c^2}{c^2-v^2}Delta t^2,. $$ We need to show that $Delta y' = Delta y = d/2$. There's another funny little thought experiment to prove that 1. Say I have two halves of a pipe painted differently and kept separated. They are running towards each other along the main axis of the pipe. If the transverse directions would shrink after a Lorentz transformation then in one frame I would see one half pipe enclosing the other, and in one other frame the opposite. Clearly the color seen by an observer after the pipes meet must be a frame invariant thing, so the solution is that the pipes don't shrink/enlarge along the orthogonal directions.

Ok, now let's come to the punch line. The events $E_1$ and $E_2$ are, respectively, the light clock starts and the light clock alarm rings. Let us compute the distance $$ S^2=c^2(t_1 - t_1)^2 - (vec{x}_1 - vec{x}_2)^2,, $$ in the two cases. For the original frame $S^2 = (ncDelta t )^2$. For the other frame $$ S^2 = (n c Delta t')^2 - (n v Delta t')^2 = (c^2 - v^2) frac{(ncDelta t )^2}{c^2 - v^2},. $$ So the two factors simplify and it works out!

But we are not done yet. This proves that the distance is invariant under arbitrary boosts for positive intervals (usually called time-like). Another answer given previously proved it for null intervals (called light-like) in full generality. We are left to prove it for negative intervals (space-like) and also for pure rotations. Now the pure rotation case is just trivial because if we have $Delta t = Delta t'$ that's just the statement that the Euclidean norm is invariant under rotation. Whereas the proof for space-like intervals can be done with another thought experiment. Take two detectors along the $x$ direction and shine a laser from the middle point. In the rest frame $Delta t = 0$ while in a frame boosted along $x$ the events won't be simultaneous. The math will work out similarly so you can do it as an exercise.

Let me review an argument given in the most textbooks. First let's start by formulating postulates of special relativity.

Postulate 1 "The laws of nature and the results of all experiments performed in a given frame of reference are independent of the translational motion of the system as a whole. More precisely, there exists a triply infinite set of equivalent Euclidean reference frames moving with constant velocities in rectilinear paths relative to one another in which all physical phenomena occur in the identical manner" (taken from 11.1 p. 517 Classical Electrodynamics 3rd ed)

We will use the following consequence of this postulate:

If two distinct object are observed to be moving with the same velocity $vec v$ in an inertial reference frame $K$, then those two objects will also be observed to be moving with the same velocity $vec v'$ in any other reference frame $K'$.

As a direct consequence of this observation we will conclude that any coordinate transformation between $K$ and $K'$ has to be an affine map. Indeed an object moving at constant speed follows a line and any other object with the same speed follows a parallel line. But a coordinate transformation that carries straight lines into straight lines and preserves parallelism, has to be affine, by this non obvious fact. Let $X=(x,y,z,t)$ ($X'=(x',y',z',t')$) be the coordinates of an event $E$ in $K$ (resp. in $K'$). The transformation between $K$ and $K'$ will be denoted by $L$, and $X'=LX$.

Now let's formulate the second postulate:

Postulate 2 "The speed of light in the vacuum is the same and is isotropic with respect any inertial frame, regardless the motion of the source."(taken from 1.3 "From Special Relativity to Feynman Diagrams")

We will prove now the following fact:

Fact Let $eta=diag(1,1,1,-1)$ and let $S$ be a symmetric real matrix. If $X^Teta X=0$ implies $X^T S X=0$, then there is a constant $lambda$ such that $X^T S X= lambda X^Teta X$, for all $X$.

Proof : Take $X=(1,0,0, pm 1)$, then $S_{11}+S_{00} pm 2S_{01}=0$. It follows $S_{11}=-S_{00}$ and $S_{01}=0$. Similarly we get $S_{22}=S_{33}=-S_{00}$ and $S_{02}=S_{03}=0$. Now take $X=(1,1,0, sqrt 2)$, then $0=S_{11}+S_{22}+2S_{00}+2S_{12}$, it follows $S_{12}=0$. Similarly $S_{23}=S_{13}=0$. Let $lambda=-S_{00}$, the conclusion follows.

The argument presented here follows 1.3 "From Special Relativity to Feynman Diagrams". Without loss of generality we may assume that $K'$ moves away from $K$ with speed $V$ in the $x$-direction and we have the two origins $O$ and $O'$ coincide at $t=t'=0$. In this case $L$ is linear.

Observe that the planes $xz$ and $x'z'$ coincide at all times. Then $y=0$ implies $y'=0$. But $L$ is linear so $y'= ax +by+cz +dt$, it follows $y'=by$. We can regard $b$ as a function of $V$, $b=b(V)$. Now if we change the orientation of the $x'$-axis and of the $x$-axis, then the equation for $y'$ and $y$ reads, $y'=b(-V)y$. Then $b(-V)=b(V)$. But $K$ moves with speed $-V$ with respect to $K'$, therefore $y=b(-V)y'$. It follows that $y=b(-V)y'=b(V)y'=b(V)^2y$, then $b(V)=pm 1$. But the axes $y$ and $y'$ are oriented in the same direction so $y'=y$. Similarly we get $z'=z$.

For the observer in $K$ the equation of motion the plane $y'z'$ is $x=Vt$, but for the observer in $K'$ it is $x'=0$. However $L$ is linear so we may write $x'= ax +by+cz +dt$, for some $a$, $b$, $c$, and $d$. Then it follows that $aV+d=0$, so $x'=a(x-Vt)$. Therefore $xt$ transforms linearly into $x't'$.

We are ready now to apply Postulate 2 in order to deduce the invariance of the interval. Assume that at $t=t'=0$ a light source emits a signal from $O=O'$. The signal propagates isotropically in all the directions with the speed of light $c$. In $K$ an observer sees a spherical wave-front with radius $r=ct$, similarly in $K'$ a spherical wave-front of radius $r'=ct'$ is observed. Now to simplify the relations we choose units in which $c=1$. For the observer in $K$, the equation of wave-front reads: $X^T eta X = x^2+y^2+z^2-t^2=0$, similarly in $K'$: $0=X'^T eta X'=X^T L^Teta L X$. Let $S=L^Teta L$. By the Fact proven above, there is a constant $lambda$ such that $X'^T eta X'=X^T S X =lambda X^Teta X$. By comparing the coefficient in $y^2$ on both sides of the equation $X'^T eta X'=lambda X^Teta X$, we deduce that $lambda =1$. This proves the invariance of the interval.

You quote the correct equations for invariance of the spacetime interval but yet say you feel you are missing something important. First you need to realize that the absolute constancy of the speed of light is a very bizarre idea! There is a real dilemma from our intuition of space and time and first you need to understand this paradox exists. And the solution really is counter intuitive but seemingly inevitable. Easiest to imagine this in empty space. If you run away or towards light we would expect said light to hit us either faster or slower. Unless the light is like sound and is vibrations of some ethereal medium and then it simply propagates at light speed relative to this. Choice 1: light radiates away from source at speed c. So light from a receding star should arrive later than from its binary companion rushing toward us. This is simply not observed in astronomy .. the light doesn’t slow down nor speed up. Choice 2 with an all pervading ether to “support” light is horrible idea and would say that if we move through the ether then light must slow down or speed up which experiments show false. So we run out of logical choices. This was a massive dilemma for the brilliant physicists like JJ Thomson who discovered electron and was no dummy. Einstein realized our logic was based on intuitive assumptions about space and time .. and he found a mathematically simple change to solve all the paradoxes but our intuition about relative motion , simultaneity and so forth takes a big knock! That spacetime invariance formula is very counterintuitive and you need appreciate the dilemmas before you can appreciate that it is a new perception of space and time which has nothing more fundamental to explain it. Like why is space 3 dimensional.. there’s no simple reason based on something more basic, it just turns out to be this way. You need to see the dilemmas first.