Superposition of eigenstates in statistical mechanics

The question that we are concerned with is the following, given a point in the $k$ space $(k_{1},k_{2},k_{3})$ and an infinitesimal volume centred at this point, how many states can be found within this volume? The superposed state $$frac{1}{sqrt{2}}left(|1,0,0 rangle + |0,1,0rangleright)$$ is found in two different volumes at a finite separation from each other in the $k$ space and is therefore, of no relevance.

The point is not that there's a priori something special about separable states compared to non-separable states, the point is that you'd like to sum over a complete set of states -- that is, a basis for the Hilbert space. The separable states you describe happen to be particularly convenient states, because they are energy eigenstates. This is especially important when you want to derive something like the density of states. But if you wanted to calculate the partition function, for instance, you're technically free to choose any basis you'd like (although if you know the energy eigenstates, then those will usually be the most convenient).

One frequently confusing point when comparing quantum mechanics to classical mechanics is that every quantum system, no matter how small, has an infinite number of possible states. An easy example is a qubit: whereas a classical bit only has the states 0 and 1, a qubit has any linear superposition $alpha | 0 rangle + beta | 1 rangle$, of which there is a continuous infinity. But when you're calculating (say) a partition function of a Hamiltonian for a single qubit, you only need to sum over a complete basis, which will contain just two states.

It's essentially because you're counting dimensions or degrees of freedom, and not every possible state.

Just to give a different style of answer, when we get into the canonical ensemble, say, we will find that the system is described by a state matrix $$ rho propto exp(-beta hat H) $$ which will contain, as you say, a component that looks like $$ |{1,0}ranglelangle{1,0}| + |{0,1}ranglelangle{0,1}|.tag{1} $$ (In 2D for simplicity.) They both have the same energy, so they both have the same prefactor, whatever that is.

As you say there is an alternative expression, in the basis $sqrt{frac12}big(|{0,1}rangle pm|{1,0}ranglebig).$ These are new vectors that span the same plane, by the logic of “we count two dimensions” it is the same number of “states” we come to if we reckon this way. But, crucially, there is a consistency issue. Each of these states also has the same energy as the two we see above, but the contribution to the state matrix now looks like something different,$$ frac12big(|{0,1}rangle+|{1,0}ranglebig)big(langle{0,1}|+langle{1,0}|big) + frac12big(|{0,1}rangle-|{1,0}ranglebig)big(langle{0,1}|-langle{1,0}|big).tag{2} $$ If you expand these out, you may be relieved to find that the first term expands out into four terms, and the second term also expands out into four terms, but they are the same terms. Two cancel with minus signs, the remaining two sum together with $frac12 +frac12=1,$ and we recover equation (1) from equation (2). So it is indeed consistent!

In fact, I have been asserting that the prefactors in (1) and (2) are the same but this is not terribly physicist-y of me, for one crucial reason which is that as physicists we kind of get a pass to ignore degeneracy. Like, mathematicians cannot commute partial derivatives because there are degenerate cases where partial derivatives do not commute; they cannot invert matrices because there are degenerate cases where matrices have eigenvalues of 0 and Jordan blocks; we basically sweep this all under the rug and say that unless there is a conservation principle forcing some sort of equation that implies degeneracy, we don't have it, our world is noisy. (You can tell the quantum textbooks written by the mathematically inclined, because the very physical minded people have just one index of summation, but the mathematicians have separate indexes for things like spin or left and right traveling particles, because their model has a degeneracy there. The mathematician has to sum over all of the spin states because the Hamiltonian does not put them at different energies. And then a lazy physicist is instead like, I am going to perform this experiment somewhere on Earth, Earth has a magnetic field which is going to split the two different spin states, so this energy index that I am summing over secretly also has the spin index crammed into it because those different spin states don't have precisely the same energy level.)

And in some sense that is what we should do here. The question only really arises because you have chosen for the 2D box to be a perfect square. But if you let me say that there is noise in the box’s manufacture, then the one side is an angstrom longer than the other and there is a fine-structure splitting of the energy levels and there is no degeneracy.

Then I can just tell you that your provided state $sqrt{frac12}big(|{0,1}rangle+|{1,0}ranglebig)$ is not actually a state of definite energy due to the fine structure, and we only “count” the states of definite energy. This is perhaps a clearer physicist-y answer.