Change of coordinates of Lagrangian

You should make a change of variables to the center of mass reference frame. The center of mass as you should know has coordinates $$R=frac{m_1r_1+m_2r_2+m_3r_3}{m_1+m_2+m_3} $$. Then you should define a new position for every particle from the center of mass. This way you should get a nice lagrangian with no crossed terms.

Studying a system from the center of mass perspective is useful when dealing with many bodies, it is not an arbitrary choice.

A general quadratic form in $x_1,ldots,x_n$ can be written as

$$L(x_1,ldots,x_n) = sum_{ile jle n}a_{ij}x_ix_j = sum_{i,j=1}^n A_{ij}x_ix_j$$

where $A_{ij} = A_{ji}$, so that $A_{ij} = a_{ij} / 2$ if $i < j$. Then $A_{ij}$ forms a symmetric matrix, so it can be diagonalized by an orthogonal transformation. This change of basis is a linear change of variables in which cross-terms are eliminated.

In your example the terms involving $dot x_i$ are already free of cross-terms, so we can disregard them. The remaining part, written out, is

$$-frac12kleft(x_1^2 - 2x_1x_2 + 2x_2^2 - 2x_2x_3 + x_3^2right) = -frac12kbegin{pmatrix}x_1 & x_2 & x_3end{pmatrix}Abegin{pmatrix}x_1 x_2 x_3end{pmatrix}$$

where

$$A = begin{pmatrix}phantom{-}1 & -1 & phantom{-}0 -1 & phantom{-}2 & -1 phantom{-}0 & -1 & phantom{-}1end{pmatrix}$$

Normalized orthogonal eigenvectors are

$$frac1{sqrt3}begin{pmatrix}1 1 1end{pmatrix},frac1{sqrt2}begin{pmatrix}1 0 -1end{pmatrix},frac1{sqrt6}begin{pmatrix}1 -2 1end{pmatrix}$$

with eigenvalues $0$, $1$, $3$, so for

$$y_1 = frac{x_1 + x_2 + x_3}{sqrt3}, y_2 = frac{x_1 - x_3}{sqrt2}, y_3 = frac{x_1 - 2x_2 + x_3}{sqrt6},$$

we get

$$x_1^2 - 2x_1x_2 + 2x_2^2 - 2x_2x_3 + x_3^2 = y_2^2 + 3y_3^2 = frac12(x_1 - x_3)^2 + frac12(x_1 - 2x_2 + x_3)^2.$$