A discussion about $text{SO}(1,3)$ transformation generators on states and four-momentum little group

Let me make some statements which may help organize things a bit. Explicit forms for the Lorentz transformations on the states are not necessary to see what's going on with the little group, and may obfuscate things a bit.

Let $P^mu$ be our 4-momentum operator and suppose that our states are written $|p,sigmarangle$ where $sigma$ is any index needed for these vectors to span the Hilbert space. I will assume that $sigma$ is discrete (I think this requirement can be relaxed, but would make things technically more complex).

By definition, $P^mu|p,sigmarangle=p^mu|p,sigmarangle$. If we suppose that $U(Lambda)$ is the unitary operator implementing Lorentz transformations on our Hilbert space, then it forms a representation, $U(Lambda)U(Lambda^prime)=U(LambdaLambda^prime)$, and must map a $p$-momentum state to a $Lambda p$-momentum state.

This can be explicitly shown from the commutator algebra with the momentum operator, but in the end we have $$ P^mu(U(Lambda)|p,sigmarangle)=(Lambda^mu_nu p^nu)(U(Lambda)|p,sigmarangle). $$ From this it follows that $U(Lambda)|p,sigmarangle$ must be some linear combination of eigenvectors of $P^mu$, all with eigenvalue $Lambda^mu_nu p^nu$. Necessarily then there exist coefficients $C_{sigma,sigma^prime}(Lambda,p)$ such that $$ U(Lambda)|p,sigmarangle=sum_{sigma^prime}C_{sigma,sigma^prime}(Lambda,p)|Lambda p,sigma^primerangle. $$ I have made no statements about what these coefficients are yet, so this is the most general possible result for the action of a Lorentz transformation on a state.

Now that some notation has been established, my goal will be to show two things: these coefficients must form a representation of the little group, and that the little group for a massive particle must be SO(3), and hence diagonalizing the little group representations is equivalent to decomposing the system into spin representations.

Denoting the standard momenta $k^mu=(mc,boldsymbol 0)$ (the following also works for standard momenta $(1,-1,0,0)$), I will generally write $W$ to indicate an element of the little group, satisfying $W^mu_nu k^nu=k^mu$. We can also define the special Lorentz transformation $L(p)$ which maps from the standard momenta to any given momenta $p$.

Now, since the $sigma$'s are just some indices, we have some freedom in which state we define $|p,sigmarangle$ to represent. In particular, we may define the symbol $|p,sigmarangle$ by $|p,sigmarangle=U(L(p))|k,sigmarangle$ (this can probably be though analogous to the statement that given $boldsymbol{hat x}$ in $mathbb{R}^3$, there is some freedom in what we call $boldsymbol{hat y}$ and $boldsymbol{hat z}$ if our only demands are they be orthogonal to each other and form a right-handed coordinate system). There are some fine points about choosing state normalizations to make sure the normalizations themselves remain Poincare invariant, but they will not matter for what I would like to say here.

With this, we are able to write $$ U(Lambda)|p,sigmarangle=U(Lambda L(p))|k,sigmarangle=U(L(Lambda p))U(L^{-1}(Lambda p)Lambda L(p))|k,sigmarangletag{1} $$ by inserting the identity. If we stare at $W(Lambda,p)equiv L^{-1}(Lambda p)Lambda L(p)$ for long enough, we may also note that this is a little group element since it maps $krightarrow prightarrow Lambda prightarrow k$.

For general little group elements $W$ we have, for some coefficients $D_{sigmasigma^prime}(W)$, $U(W)|k,sigmarangle=sum_{sigma^prime}D_{sigmasigma^prime}(W)|k,sigma^primerangle$ as a special case of the general statement about the action of Lorentz transformations on our states. In particular, $U(W^prime)U(W)=U(W^prime W)$ implies that these coefficients satisfy $$ D_{sigmasigma^prime}(W^prime W)=sum_{sigma^{primeprime}}D_{sigmasigma^{primeprime}}(W^prime)D_{sigma^{primeprime}sigma^prime}(W)tag{2} $$ and hence form a representation of the little group.

Combining (1) and (2), we are now able to write $$ U(Lambda)|p,sigmarangle=U(L(Lambda p))sum_{sigma^prime}D_{sigmasigma^prime}(W(Lambda, p))|k,sigma^primerangle=sum_{sigma^prime}D_{sigmasigma^prime}(W(Lambda, p))|Lambda p,sigma^primerangle.tag{3} $$

Hence, we have found $C_{sigmasigma^prime}(Lambda,p)=D_{sigmasigma^prime}(W(Lambda,p))$, the coefficients $D$ already shown to form a representation of the little group. With this, the spectrum of states may be further decomposed into irreducible representations of the little group.

In the case of a massive particle, $k^mu=(mc,boldsymbol{0})$, so the little group is just SO(3). Hence the irreducible representations of the little group that our states should be organized by will in general be spin. So it's not that you can ensure the states will be eigenstates of your $hat Sigma$ operator, but rather that the states can always be reorganized such that this is true.

The arguments I have given above are very similar to those in chapter 2 of Weinberg's The Quantum Theory of Fields Volume I (where I learned this from). More details can be found there.

Based on a discussion in the comments, I feel there are a few more things I should add to this answer. First, I would like to point out that the main result of this argument was to show that the coefficients $C_{sigmasigma^prime}(Lambda, p)$, whose existence is guaranteed on general grounds, can be expressed in terms of the coefficients $D_{sigmasigma^prime}(W)$ which were noted to form a representation of the little group. In particular, (3) implies (up to some normalizations which don't matter here but are worked out in Weinberg) $$ C_{sigmasigma^prime}(Lambda,p)=D_{sigmasigma^prime}(W(Lambda,p)), $$ and so the $C$'s also form a representation of the little group. A fairly accessible description of representation theory can be found in sections 1.1-1.5 of Howard Georgi's Lie Algebras in Particle Physics.

For a massive particle, the little group was noted to be $SO(3)$, so let's discuss the relationship between $SO(3)$ and spin, which requires we first talk about $SU(2)$. Everything I am about to describe can be found in chapters 2 and 3 of Georgi's book, mentioned above. There are three generators which span the Lie algebra of $SU(2)$. At least some of this should be familiar from standard accounts of spin $1/2$, though the generalization to higher spin is typically left out. In any case, all the finite dimensional representations of $SU(2)$ can be classified by the spin, which is the largest eigenvalue of the diagonalized generator (typically $S_z$ by convention).

So, whenever someone refers to some state or some such as being "spin-$j$", the meaning is that this object transforms under the representation of $SU(2)$ with spin $j$. The dimensionality of the spin-$j$ representation is always $2j+1$, so for $j=1/2$ the dimension is $2$, which we know had to be the case because the Pauli matrices are used when discussing spin-$1/2$ particles/states.

This index $sigma$ runs over the dimensionality of the representation. So in some sense it's misleading to call $sigma$ "spin" when spin is understood to have the meaning above.

Relating this back to the case where our little group is $SO(3)$, note that the Lie algebras of $SO(3)$ and $SU(2)$ are isomorphic, so all the statements about representations of $SU(2)$ carry over to the case of $SO(3)$. There are some subtleties relating to global properties of the two groups, but I think worrying about these details would distract from the main idea here.

We can go even further to explore how these $C$'s work, what these $sigma$ indices are supposed to be, and how this relates to the $C$'s forming a representation.

A large majority of what I said above about $SU(2)$ really only applies to so-called irreducible representations of the group. It is a theorem that all representations of any finite group can be decomposed into the direct sum of irreducible representations. In fact, this is precisely what we are doing whenever we compute a Clebsch-Gordon decomposition.

With this in mind, the $C$'s don't need to form an irreducible representation (which are the things actually classified by the spin $j$), but we can always rotate our basis vectors such that the $C$'s have been block diagonalized, and hence we have decomposed our states into sectors transforming under irreducible representations of $SU(2)$. These sectors are typically what we would identify as "spin-$j$ particles."

Within a single sector, say for simplicity a spin-$1/2$ block, our states are spanned by the states $|p;1/2;sigma_{1/2}rangle$ where now because we are within a single block (subspace) of the Hilbert space, the $sigma$ runs over the indices of the spin-$1/2$ representation. This is a two-dimensional representation, so $sigma_{1/2}=1,2$, or more colloquially, is either spin up or spin down.