Density of particles in hexagonal lattice

Ok, finally solved it in a very simple geometrical way.

IF we take a square slanted lattice in the hexagonal lattice, like in the image, which is $N$ particles along each side. Then the number of particles inside is $N^2$. The volume of that area is just $V=(Na)^2cos(30)$, and so: $$rho a^2=frac{2}{sqrt 3}$$.

I'm sorry for asking, I was frustrating myself with this.

I think your result is right but your procedure of calculation maybe wrong.

Following your image：

You may find that the area of this parallelogram is $V=(N-1)^2a^2cos30$ because the length of each edge should be $(N-1)a$.

Then we should consider that some points on the edges are shared by multiple parallelograms. For example, the red points are shared by 2 parallelograms and can only be considered as half points. Only the gray points completely inside the parallelogram can be viewed as full points.

Finally, we will have: point number = $(N-2)^2$ (full points) + $4*1/2*(N-2)$ (half points) + $1$ (all corner points), which equals to $N^2-4N+4+2N-4+1=(N-1)^2$.

Since $V=(N-1)^2a^2cos30$, the $rho$ will still equals to $2/(sqrt3*a^2)$.