Range of quadratic function using discriminant

Others have pointed out where you went wrong. I just want to provide an alternative proof.

$$x^2-2xy-3y^2=4=(x+y)(x-3y)$$

Denote $z=x+y, w=x-3y$, then $zw=4, x=(3z+w)/4, y=(z-w)/4$.

We want to find the range of $$2x^2-2xy+y^2=frac{1}{16} (5w^2+14wz+13z^2)=frac 72+frac{1}{16} (5w^2+13z^2) tag 1$$ under the constraint $zw=4$.

Clearly $(1)$ can be as large as you want. And if you apply AM-GM you get the minimum $$frac 72+frac{1}{16} (5w^2+13z^2) ge frac 72 + frac{1}{16} 2 sqrt{5} sqrt{13}|wz|=frac{7+sqrt{65}}{2}$$

Consider the curves $x^2-2xy-3y^2=4 (C)$ and $2x^2-2xy+y^2=k^2+4 (E)$ At the points where they intersect, we can substitute the value of $-2xy$ from the first into the second to get $$x^2+4y^2=k^2 (E)$$ This represents an ellipse centered at the origin. Now, either with a bit of graph sketching, or by considering $(C)$ as a quadratic in $x^2$, deduce that $y$ can range anywhere in the real numbers, and that $(C)$ is a hyperbola. Increasing $k$ in $(E)$ just enlarges the ellipse, and it’s not hard to see that there is no upper bound on $k$ for $(E)$ and $(C)$ to intersect. For obtaining a lower bound, we need to find $k$ such that the two curves touch. Note that if $(x,y)$ lies on the two curves, then so does $(-x,-y)$. So we’re looking for exactly two intersections. Then substituting $x=pmsqrt{k^2-4y^2}$ in $(C)$ gives a quadratic in $y^2$: $$65y^4-y^2(18k^2-56)+(k^2-4)^2 = 0 $$ Setting the discriminant equal to zero will give $k^2=frac{sqrt{65}-1}{2}$ and hence we have the desired range: $$frac{sqrt{65}+7}{2} le 2x^2-2xy+y^2 =k^2+4lt infty $$

What was wrong in your approach? What you stated was a necessary condition on $a$, but that doesn’t mean that $a$ could really equal anything in that range.

Method$#1:$

For real $t,$ $$a^2-7a-4ge0$$

If $x=ty$

$$4=x^2-2xy-3y^2=y^2(t^2-2t-3)$$

$$t^2-2t-3=dfrac4{y^2}>0$$

$$implies(t-3)(t+1)>0$$

Either $t<-1$ or $t>3$

So, the values of $a$ must satisfy this condition as well

Method$#2:$

$$4=(x-y)^2-(2y)^2$$

WLOG $y=tan t, x-y=2sec timplies x=2sec t+tan t$

$$a=2(2sec t+tan t)^2-2(2sec t+tan t)tan t+tan^2t$$

Multiplying both sides by $cos^2t=1-sin^2t,$

$$(1+a)sin^2t+4sin t+8-a=0$$

What if $a+1=0?$

Else
$$sin t=dfrac{-2pmsqrt{a^2-7a-4}}{a+1}$$

As $sin t$ is real, the discriminant must be $ge0$

But that is not sufficient, we need $$-1ledfrac{-2pmsqrt{a^2-7a-4}}{a+1}le1$$ as well for real $t$

Also, $dfrac xy=2csc t+1$ $impliesdfrac xyge2+1$ or $dfrac xyle-2+1$