Values of a pair of determinants

First, by transposing and interchanging two rows $S({bf x},{bf y}) = -left|begin{matrix} x_0 & y_0 & 2y_2 \ x_1 & y_1 & y_0 \ x_2 & y_2 & y_1 end{matrix} right|$. Notice also that if $T = left[begin{matrix}0&0&2 \ 1&0 &0 \ 0& 1& 0end{matrix} right]$ then $T{bf y} = left[begin{matrix} 2y_2 \ y_0 \ y_1end{matrix}right]$. Hence, $S({bf x},{bf y}) = -left| {bf x} {bf y} T{bf y} right|$

Suppose $Ain mathcal G$ then for all ${bf x},{bf y} in mathbb F^3$ $$ S({bf x},{bf y}) = S(A{bf x}, A{bf y}) = -left|{bf x} {bf y} |A|A^{-1}TA{bf y}right|. $$ Now when ${bf x} = e_3$ then this gives $$ y_0^2 - 2y_1y_2 = y_0langle|A|A^{-1}TA{bf y}, e_2rangle - y_1langle|A|A^{-1}TA{bf y}, e_1rangle. $$ Subsequently, when ${bf y} = e_1$ this gives $1 = langle|A|A^{-1}TAe_1, e_2rangle$, and when ${bf y} = e_2$ this gives $0 = langle|A|A^{-1}Tae_2, e_1rangle$. Moreover, when ${bf y} = e_1+e_3$ then after some computation $$langle|A|A^{-1}TAe_1, e_1rangle = langle|A|A^{-1}TAe_3, e_3rangle$$

When ${bf x} = e_2$ this gives $$ 2y_2^2 - y_0y_1 = y_2langle|A|A^{-1}TA{bf y}, e_1rangle - y_0langle|A|A^{-1}TA{bf y}, e_3rangle, $$ ${bf y} = e_1$ gives $0 = langle|A|A^{-1}TAe_1, e_3rangle$, and ${bf y} = e_3$ gives $2 = langle|A|A^{-1}Tae_3, e_1rangle$. Moreover, when ${bf y} = e_1+e_2$ then $$langle|A|A^{-1}TAe_1, e_1rangle = langle|A|A^{-1}TAe_2, e_2rangle$$

When ${bf x} = e_1$ this gives $$ y_1^2 - y_0y_2 = y_1langle|A|A^{-1}TA{bf y}, e_3rangle - y_2langle|A|A^{-1}TA{bf y}, e_2rangle $$ ${bf y}=e_2$ gives $1 = langle|A|A^{-1}TAe_2, e_3rangle$ and ${bf y} = e_3$ gives $0 = langle|A|A^{-1}TAe_3, e_2rangle$

If we define $lambda := langle|A|A^{-1}TAe_1, e_1rangle$ then the above argument concludes $$ |A|A^{-1}TA = left[begin{matrix}lambda&0&2 \ 1&lambda &0 \ 0& 1& lambdaend{matrix} right] = lambda I + T $$ and so $$ TA = frac{1}{|A|}lambda A + frac{1}{|A|}AT. $$ One won't get anymore out of these equations. Furthermore, any $A$ satisfying this equation gives $$ S(A{bf x}, A{bf y}) = -|{bf x} {bf y} lambda{bf y} + T{bf y}| = S({bf x}, {bf y}) $$ by column replacement. Your $M$ corresponds to the case where $lambda = 0$, that is commutes with $T$, and $|A| = 1$. However, there may be more matrices in $mathcal G$.

Therefore, $$ mathcal G = left{ Ain {rm GL}_3(mathbb F) : TA = frac{1}{|A|}lambda A + frac{1}{|A|}AT, lambdain mathbb F right} $$