Two spring-coupled mass system

Thanks for defining $x$, with that, I can explain what the problem is. I assume that by the way you drew $x_1$ and $x_2$, this also defines the positive values of $x_1$ and $x_2$.

The thing is that you give $x_1$ and $x_2$ with reference to some equilibrium positions $x_{1e}$ and $x_{2e}$. You can define the harmonic motion in this coordinate system, but it's a bit tricky to justify exactly why the terms aris the way they do.

The no-brain way of doing it is to start with $x_1$ and $x_2$ with reference to the same origin. Say $x_1$ were the distance of mass 1 from the point where the left spring is attached to the wall, and $x_2$ were the distance of mass 2 from the same point. Then you can go ahead and model all three springs. The length of the first spring would be $x_1$, the length of the second spring would be $x_2 - x_1$ and the lenght of the third spring is $L - x_2$. Clearly, $x_1$ feels the forces due to the first and second spring, and each force is the spring constant times the length, so $$m ddot{x}_1 = - k_0 x_1 + k (x_2 - x_1).$$ The signs are there because the left spring pulls $x_1$ to the left (against the positive direction of the coordinate system we just chose) while the middle spring gives rise to a pull to the right (in the positive direction of the coordinate system). Similarly, you find $$m ddot{x}_2 = k_0 (L - x_2) - k (x_2 - x_1).$$ This time, the right spring pulls to the right while the middle spring pulls to the left, so you get the opposite signs. Therefore, you have a matrix equation $$frac{textrm{d}^2}{textrm{d}t^2}begin{pmatrix} x_1 x_2end{pmatrix} = begin{pmatrix} - (k + k_0) & k k & -(k + k_0)end{pmatrix}begin{pmatrix} x_1 x_2end{pmatrix} + k_0 L begin{pmatrix} 0 1end{pmatrix},$$ and you can proceed to calculate the frequencies from the frequency matrix.

Link between descriptions

I had kind of a brainblockage yesterday, so I got a bit confused about the use of different equilibrium positions. So to get this straight: You can transform the above equation to arbitrary coordinate origins for $x_1$ and $x_2$. Let's write new coordinates with the transform $x_1 mapsto x_1' = x_1 - x_{1e}$ and $x_2 mapsto x_2' = x_2 - x_{2e}$. Evidently, $textrm{d}^2/textrm{d}t^2 x_i' = textrm{d}^2/textrm{d}t^2 x_i$ for $i in {1,2}$, and by insertion we have $$frac{textrm{d}^2}{textrm{d}t^2}begin{pmatrix} x_1' x_2'end{pmatrix} = begin{pmatrix} - (k + k_0) & k k & -(k + k_0)end{pmatrix}begin{pmatrix} x_1' + x_{1e} x_2' + x_{2e}end{pmatrix} + k_0 L begin{pmatrix} 0 1end{pmatrix}.$$ Now if you deliberately choose the positions $x_{1e}$ and $x_{2e}$ to be the equilibrium posistion, i.e. $$begin{pmatrix} - (k + k_0) & k k & -(k + k_0)end{pmatrix}begin{pmatrix} x_{1e} x_{2e}end{pmatrix} + k_0 L begin{pmatrix} 0 1end{pmatrix} = 0,$$ which is just the requirement that the masses feel no acceleration in this particular configuration, you arrive at $$frac{textrm{d}^2}{textrm{d}t^2}begin{pmatrix} x_1' x_2'end{pmatrix} = begin{pmatrix} - (k + k_0) & k k & -(k + k_0)end{pmatrix}begin{pmatrix} x_1' x_2'end{pmatrix}.$$

If you just have one spring with mass you have two possibilities to choose the force sign $~+~$ or $~-~$

$$mddot{x}=pm k,x$$

if you choose $~-~$ sign you obtain the solution with the initial conditions $x(0)=x_0$ and $dot{x}(0)=0$

$$x(t)=x_0,cos(omega,t)$$

where $omega=sqrt{frac km}$

if you choose $~+~$ sign you obtain the solution

$$x(t)=frac 12 x_0 e^{omega t}$$

but this solution is wrong because you expect sine or cosine motion.

thus the minus sign is correct.

follow this rule you obtain this equations of motion

$$m,ddot x_1=-k_0,x_1-k,(x_1-x_2)$$ $$m,ddot x_2=-k_0,x_2+k,(x_1-x_2)$$

I suggest stick to convention and do the free body diagram with positive displacements towards the right.

In fact I drew the above with $x_2 > x_1$ to help with the determination of the middle spring force direction.

From the above I have

$$ begin{aligned} F_1 & = k_0 (x_1) F_2 & = k (x_2-x_1) F_3 & = k_0 (x_2) end{aligned} $$

and the equations of motion

$$ begin{aligned} F_2 - F_1 & = m_1 ddot{x}_1 -F_2 - F_3 & = m_2 ddot{x}_2 end{aligned} $$

Combined the above produces

$$begin{bmatrix} m_1 & 0 0 & m_1 end{bmatrix} pmatrix{ddot{x}_1 ddot{x}_2 } = -begin{bmatrix} k+k_0 & -k -k & k+k_0 end{bmatrix} pmatrix{x_1 x_2} $$

Now you can go and flip the sign of $x_1$ and of $ddot{x}_1$ if you want.