number of integer points inside a triangle and its area

The bound on $|E|$ can certainly be improved: $|E|le|E|$; so, we get the "perfect" (but completely useless) upper bound, $|E|$, on $|E|$.

To make the problem meaningful, we need to specify the terms in which the upper bound is to be expressed. Counting the unit squares intersecting the boundary of the triangle, it is easy to see that $$|E|ll s:=a+b+1,$$ where $a$ and $b$ are the lengths of the horizontal and vertical sides of the right triangle (in terms of your $alpha,beta,gamma$, we have $a=|gamma-beta/alpha|$ and $b=|beta-alphagamma|$; your bound is missing "$+1$"). Note that $s>1$.

Let us show that $s$ is the best (up to a constant factor) upper bound on $|E|$ in terms of $s$. Indeed, take any real $s>1$ and consider the right triangle $T$ with vertices $(0,0),(0,a),(a,a)$, where $a:=(s-1)/2>0$, so that $a+a+1=s$. Then the area of $T$ is $A=a^2/2$ and the number $N$ of integral points that are either inside $T$ or on the boundary of $T$ is the number of pairs $(i,j)$ of integers such that $0le ile jle a$. The latter triple inequality can be rewritten as $0le ile jle n$, where $n:=lfloor arfloorge0$, so that $nle a<n+1$. So, $N=(n+1)(n+2)/2$ and $A<(n+1)^2/2$. Hence, $$|E|=|N-A|=N-A \ >frac{(n+1)(n+2)}2-frac{(n+1)^2}2 \ =frac{n+1}2gefrac{2(n+1)+1}6>frac{2a+1}6=frac s6.$$

Thus, the best upper bound on $|E|$ in terms of $s$ is $,asymp s$, as claimed.

---

Working a bit harder, one can show that $|E|le s$. So, the best upper bound on $|E|$ in terms of $s$ is between $s/6$ and $s$.