How is Grover's operator represented as a rotation matrix?

This is essentially the same calculation I outlined in this other answer (though it might not be immediately obvious).$newcommand{ket}[1]{lvert#1rangle}newcommand{ketbra}[2]{lvert#1rangle!langle#2rvert}$

Let us denote with $Pi_Y$ and $Pi_N=I-Pi_Y$ the projectors onto the "yes space" and the "no space". Given an initial state $ketpsi$, the goal is getting as close to a state in $Pi_Y$ as possible, as fast as possible.

Because ${Pi_Y,Pi_N}$ define a separation of the full space, any state can be decomposed using these operators. In particular, we can write $$ketpsi=costhetaketalpha+sinthetaketbeta,$$ where $theta$ is defined via $costhetaequiv|Pi_Yketpsi|$, while $ketalphaequivPi_Yketpsi/costheta$ and $ketbetaequivPi_Nketpsi/sintheta$ (I'm using a slightly different notation with the $theta$ than the one in the OP, sorry about that).

The Grover iterator is defined as $G=-S_psi S_Y$, where $S_Y$ and $S_psi$ are reflections in state space, that is, operators which leave untouched some subspace and change the sign on everything else. More specifically $S_Y$ flips the "yes space", while $S_psi$ flips the direction corresponding to the initial state $ketpsi$ (that is, it leaves the direction of the initial state untouched and flips everything else). Mathematically, these reflections can be written as $$S_Yequiv I - 2Pi_Y, qquad S_psiequiv I - 2ketpsi!langlepsirvert.$$ It follows that the Grover operator reads $$G=(I-2ketpsi!langlepsirvert)(2Pi_Y-I).$$ Expanding this product we get $$G=2Pi_Y-I-4 lvertpsirangle!langlepsirvertPi_Y + 2ketpsi!langlepsirvert.$$ Expanding $ketpsi$ in terms of $ketalpha$ and $ketbeta$, and remembering the property of $ketalpha$ that $Pi_Yketalpha=ketalpha$, you can readily verify that this expression becomes, after a bit of algebra, the following (let me use here the shorthand notation $cequivcostheta$ and $sequivsintheta$):

$$G=2Pi_Y-I+2s^2ketbrabetabeta-2c^2 ketbraalphaalpha +2cs (ketbraalphabeta-ketbrabetaalpha).$$

Now this represents a rotation in state space of some angle with respect to some axis, but because we are only interested in the action of $G$ on states spanned by the $ketalpha$ and $ketbeta$ states, we need only analyse how $G$ acts on these two states. We then readily get: begin{align} Gketalpha&=-cos(2theta)ketalpha-sin(2theta)ketbeta, Gketbeta&=phantom{-}sin(2theta)ketalpha-cos(2theta)ketbeta. end{align} Collecting the corresponding amplitudes in a matrix, we conclude that the action of $G$ in the space spanned by $ketalpha$ and $ketbeta$ can be represented as $$Gdoteqbegin{pmatrix}-cos(2theta) &sin(2theta)-sin(2theta) & -cos(2theta)end{pmatrix}.$$

Note that this shows that the result holds in a more general scenario that the one often used when first explaining Grover's algorithm. You can however easily reduce to the standard situation (which you also use in your post) by having $ketpsi$ be a balanced superposition of all basis states, and $Pi_Y$ a trace-1 projector (that is, a projector over a one-dimensional subspace).

---

Another derivation with different notation

Let us use the shorthand notation $psiequivketbrapsipsi$, $YequivPi_Y$ and $NequivPi_N$. We then have begin{align}G = (I-2psi)(2Y-I) = &Y(1-2psi)Y - 2Npsi Y & +2 Ypsi N -N(1-2psi)N. end{align} Then, specialising to the case $ketpsi=cos(theta)ket y+sin(theta)ket n$, we get $$G = Y(I - 2cos^2(theta) ketbra y y)Y + sin(2theta)(ketbra y n - ketbra n y) - N(I- 2 sin^2(theta) ketbra n n)N,$$ and in particular its action on $ket y$ and $ket n$ is begin{align} Gket y &= -cos(2theta)ket y - sin(2theta)ket n, Gket n &= phantom{-}sin(2theta) ket y - cos(2theta) ket n. end{align}