How can I determine the radius of 4 identical circles inside an equilateral triangle $ABC$?

You asked to find $QR$. Connect the centers of three external circles, whose circumradius is $2r$. Then its side is from the sine theorem: $$frac{QR}{sin 60^circ}=2(2r) Rightarrow QR=2sqrt{3}r.$$ Now you can finish.

Another solution. Using the Tangent-secant theorem: $$AScdot AK=AM^2 Rightarrow (R-3r)(R-r)=left(frac{AB-MN}{2}right)^2Rightarrow\ left(frac{a}{sqrt3}-3rright)left(frac{a}{sqrt3}-rright)=left(frac{a-2sqrt3r}{2}right)^2Rightarrow a=frac{a}{4sqrt3}.$$

Following your approach: $text{In right} Delta AMQ, AM=QMcot30^circ=rsqrt{3}$

$$implies MT=AT-AM=dfrac{a}{2}-rsqrt3$$ Now, use Pythagorean theorem in the right $Delta $ with legs $dfrac{a}{2}-rsqrt3, dfrac{a}{2sqrt3}-r$ and hypotenuse $PQ=2r$ as follows $$(2r)^2=left(dfrac{a}{2}-rsqrt3right)^2+left(dfrac{a}{2sqrt3}-rright)^2$$

After expanding, $r^2$ terms cancel out and we get

$$bbox[15px, #ffd,border:2px solid green]{r=frac{a}{4sqrt{3}}}$$

The picture shown below makes it immediately obvious that $$6r = frac{sqrt{3}}{2} a,$$ or $$r = frac{a}{4sqrt{3}}.$$

Draw some extra lines to enclose each circle in a smaller equilateral triangle. Since the radii of all the circles are equal, the sidelengths of the small triangles are equal also, and as the diagram makes clear, the radius of one circle is the inradius of an equilateral triangle with half the sidelength:

From there, it's easy to show that if $s = $ side of original triangle, $r =$ radius of one small circle, we get $r = s/4sqrt{3}$, as claimed.

Join the center $P$ of central circle to the vertex $A$. The length of $AP$ will be equal to the circum-radius of equilateral $Delta ABC$ i.e. $AP=dfrac{a}{sqrt{3}}$.

$text{In right} Delta AMQ, AQ=dfrac{QM}{sin30^circ}=dfrac{r}{1/2}=2r$

From the above figure, we have $$AQ+QP=AP$$ $$2r+2r=frac{a}{sqrt3}$$ $$bbox[15px, #ffd,border:1px solid green]{r=frac{a}{4sqrt{3}}}$$