How to demonstrate mathematically the conditions of reflection in transmission line?

Surprised there aren't any proper answers yet. The question is asking about how to demonstrate this property mathematically, not intuitively. This is a great question, so let's derive some stuff.

SHORT ANSWER

Solve the Telegrapher's equations, to get the general form of the voltage and current on the line. For the lossless case, this is: $$ v(z) = V_1 e^{-jbeta z} + V_2 e^{j beta z} $$ For unknown complex "amplitude constants" V1 and V2, and constant beta, which is $$ beta = omega sqrt{LC} $$ And the current is $$ i(z) = frac{V1}{Z_0} e^{-jbeta z} - frac{V2}{Z_0} e^{jbeta z}$$ Where we introduce the constant Z0, $$ Z_0 = sqrt{frac{L}{C}} $$ Which comes out of the math.

Constants V1 and V2 are determined by the boundary conditions. The condition at the load gives: $$ V_2 = Gamma V_1 $$ That is, the amplitude of the reflection is Gamma * V1. Gamma is called the reflection coefficient: $$ Gamma = frac{R_L - Z_0}{R_L + Z_0} $$ When RL = Z0, Gamma = 0, and there is no reflected wave. When RL=0, Gamma is -1, implying full reflection with phase inversion (reflected wave out of phase with incident). When RL=infinity, Gamma is +1, implying full reflection with no phase inversion (reflected wave in phase with incident). The other results follow from plugging in values for Gamma.

MORE RIGOROUS ANSWER

Fundamentally, a distributed element or transmission line is a circuit element with a spatial dimension (call it z) such that, for real constants R, L, G, and C, and time- and space-dependent voltage and current V(t,z) and I(t,z), the following partial differential equations hold: $$ frac{partial V}{partial z} = -L frac{partial I}{partial t} - RI$$ $$ frac{partial I}{partial z} = -C frac{partial V}{partial t} - GV$$ These are called the telegrapher's equations, well-known PDEs describing transmission lines. Note that they are only PDEs because we have both time and space dependence.

If we limit ourselves to phasors, then V(t,z) and I(t,z) become v(z) and i(z), complex valued functions of only space. The Telegrapher's equations then become: $$ frac{d v}{d z} = -i(R+jomega L) $$ $$ frac{d i}{d z} = -v(G+jomega C) $$ For angular velocity omega, $$ omega = 2 pi f $$ For assumed frequency f.

Great, ordinary differential equations. For simplicity, assume R = G = 0. The general solution to these equations is well-known; we find for v(z), $$ v(z) = V_1 e^{-jbeta z} + V_2 e^{j beta z} $$ For unknown complex "amplitude constants" V1 and V2, and complex constant beta, which is $$ beta = omega sqrt{LC} $$ (When R and G are not zero, beta becomes complex, with a real part the same as beta here, and an imaginary part representing attentuation with increasing z).

Using this expression, we can now solve for i, $$ i(z) = frac{V1}{Z_0} e^{-jbeta z} - frac{V2}{Z_0} e^{jbeta z}$$ Where we introduce the constant Z0, $$ Z_0 = sqrt{frac{L}{C}} $$

It now remains to determine the constants V1 and V2. These amplitude constants are determined by the particular problem at hand; specifically, we can provide two boundary conditions (e.g. V(0) = 1 and V(5) = 0), and then plug them into our expression for v(z) to find V1 and V2. Simple enough. To find boundary conditions, we use KVL and KCL, and Ohm's Law. KVL and KCL imply that at the boundary of the distributed element (where the transmission line connects to a lumped circuit) the current must be continuous and the voltage must be continuous. This is hugely helpful.

Let's solve the basic transmission line problem. This image shows the situation.

Note that the spatial coordinates only apply to the distributed element, and more particularly, to this distributed element. If there was another transmission line, it would have its own coordinate system.

For conventional reasons, make z=0 the end of the transmission line, right before the load RL, with +z in the direction shown, so that the beginning of the transmission line is at z = -length. We have then two boundary conditions. First, $$ frac{v(0)}{i(0)} = R_L $$ Due to continuity at the boundary and Ohm's law. Second, using KVL, $$ frac{V_s - v(-l)}{i(-l)} = R_s $$ Where l is the length of the line. We have our boundary conditions, and we can solve for the amplitude constants V1 and V2 (note that we haven't clarified what V1 and V2 represent intuitively, only that they are mathematical constants inherent in the problem as we constructed it).

From the first condition, we can substitute in v(z) and i(z) setting z=0 (recall that we have these, from above) to get: $$ frac{v(0)}{i(0)} = frac{V_1 e^0 + V_2 e^0}{V_1/Z_0 e^0 - V_2/Z_0 e^0} = R_L $$ So: $$ Z_0 frac{V_1 + V_2}{V_1 - V_2} = R_L $$ Which gives, by some algebra: $$ V_2 = frac{R_L - Z_0}{R_L + Z_0} V_1 $$ We will denote that constant fraction by Gamma, as shown: $$ Gamma = frac{R_L - Z_0}{R_L + Z_0} $$ So that $$ V_2 = Gamma V_1 $$ And our expression for v(z) becomes: $$ v(z) = V_1(e^{-jbeta z} + Gamma e^{jbeta z}) $$ Now we're all done.

Let's interpret our result now (which, mind you, we have refrained from doing to avoid projecting our own intuition on something mathematical). v(z) is a phasor, meaning its "amplitude" is actually the amplitude of a sinusoidal signal, in time, varying with frequency w. The phase is the phase angle of the sinusoid. Essentially, v(z) represents the envelope of the actual voltage on the line, which has a frequency w.

The complex exponential in the expression for v(z) can be expanded as $$ e^{-jbeta z} = cos beta z - j sin beta z $$ Which makes the 'oscillating' or 'wave' nature of the complex exponential terms clear. What we're seeing is, as we look at different positions z, the amplitude and phase of the phasor change, and they change sinusoidally, giving an envelope which looks like a sine wave, and as time goes in, the whole envelope moves to the +z direction. This is the "forward traveling wave."

By similar reasoning, when we have $$ e^{jbeta z} $$
(with +j instead of -j), the same thing occurs, but now the envelope waveform slides backwards, a "backward traveling wave." And so, we can rewrite v(z) as $$ v(z) = V^+(z) + V^-(z) $$ Where $$ V^+(z) = V_1 e^{-jbeta z} $$ Is the forward-traveling wave with amplitude V1, and $$ V^-(z) = Gamma V_1 e^{j beta z} $$ Is the backward-traveling wave, often called the reflection.

We see that the amplitude of the reflection is Gamma * V1. And, in turn, Gamma is: $$ Gamma = frac{R_L - Z_0}{R_L + Z_0} $$ And Z0 is: $$ Z_0 = sqrt{frac{L}{C}} $$ Which is characteristic of the line (the characteristic impedance, called an impedance due to its similarity to the fact that it relates v(z) and i(z), but note that it does not have units of ohms). When RL = Z0, Gamma = 0, and there is no reflected wave. When RL=0, Gamma is -1, implying full reflection with phase inversion (reflected wave out of phase with incident). When RL=infinity, Gamma is +1, implying full reflection with no phase inversion (reflected wave in phase with incident). The other results follow from plugging in values for Gamma.