Does a complete set of commuting observables always exist?

Sometimes textbooks on introductory quantum theory give the impression that every self-adjoint operator represents an observable, and many simple models do have that property. Such simple models always have a CSCO. However, the general principles of quantum theory don't require that every self-adjoint operator be an observable. In fact, some models don't even have any CSCOs.

So the short answer to the question is: counterexamples do exist. I'll start by reviewing some natural counterexamples that actually arise in physics, but the math is deep. To compensate for that, I'll finish the answer with a contrived counterexample in which the math is elementary.

Natural (but difficult) counterexamples

Models exist in which the algebra of observables is a type III factor, which cannot have a CSCO in any representation on a separable Hilbert space. I'll explain what this jargon means in the next section, but first I'll give the examples.

The counterexamples come from relativistic quantum field theory (RQFT). At least in one axiomatic system for RQFT, typical free-field RQFTs (in continuous spacetime) can be shown to have the property that the algebra of observables associated with a bounded region of spacetime is a type-III von Neumann algebra. Below, I'll prove that such an algebra cannot have a CSCO. By choosing an arbitrarily-large but bounded region of spacetime and calling that the whole model, we obtain a physically-natural example of a model with no CSCOs. More information about this is given in ref 1. Here's an excerpt from the abstract:

In relativistic quantum field theory (RQFT), ... factors of type III occur naturally. The same holds true in quantum statistical mechanics of infinite systems.

And an excerpt from pages 3-4:

Investigations on the foundations of relativistic quantum field theory (RQFT) revealed already more than 40 years ago that the algebras generated by observables that are localized in bounded regions of space-time cannot be type I and that they are, in fact, generically of type III. This is far from being obvious but turns out to be a consequence of general physical requirements of RQFT. Another case where type I is excluded is the statistical physics of infinite systems with nonzero density [11].

Free-field RQFTs are not the only counterexamples. Asymptotically-free models like quantum chromodynamics (QCD) presumably also have this property, but that has not yet been proven: we do not yet have a proof that QCD satisfies the axioms of RQFT in continuous spacetime from which the type-III property is deduced. Counterexamples that are physically natural and mathematically tractable (but still not easy) can be found in non-relativistic quantum physics. A few are listed on page 273 in ref 2.

Technical background

Here's some math background to explain what all that jargon means:

• A von Neumann algebra can be defined as a set $M$ of operators on a Hilbert space that is self-contained with respect to adjoints and that has the property $(M')'=M$, where $M'$ denotes the set of all operators that commute with $M$. This isn't the usual definition of a von Neumann algebra, but it's equivalent to the usual definition, thanks to the double commutant theorem.

• A factor is a von Neumann algebra $M$ in which operators proportional to the identity operator are the only ones that commute with everything else in $M$.

• A factor is called type I if it is equivalent to the algebra of all operators on some Hilbert space. By the way, throughout this answer, "operator" means "linear operator that is defined on the whole Hilbert space". Although it is often convenient to represent observables by operators that are only defined on part of the Hilbert space, this is never necessary.

• A factor $M$ is called type III if it has this property: for every non-zero projection operator $Pin M$, there is another operator $Win M$ such that $W^*W = 1$ and $WW^* = P$. The asterisk here denotes the operator adjoint, which is more commonly denoted $W^dagger$ in the physics literature.

Proof that a type III algebra cannot have a CSCO

The counterexamples mentioned above are counterexamples because, in quantum physics, a type III factor cannot have a CSCO. To prove this, remember that quantum physics requires that the Hilbert space be separable — it must have a countable orthonormal basis. On such a Hilbert space, the algebra generated by a CSCO (via the double commutant, so that it is topologically complete) must include projection operators that project onto one-dimensional subspaces. Let $P$ be such a projection operator. The claim that the algebra is type III means that it also includes an operator $W$ for which $W^*W = 1$ and $WW^* = P$. Multiplying the first equation on the left by $W$ gives $PW=W$, and multiplying this on the left by $W^*$ gives $W^*PW=1$, which is impossible if $P$ projects onto a one-dimensional subspace. Therefore, a CSCO cannot exist in a (faithful representation of a) type III factor on a separable Hilbert space.

An easier (but contrived) counterexample

The examples mentioned above all involve an infinite-dimensional Hilbert space, and they are mathematically difficult. Can we contrive easier examples that use a finite-dimensional Hilbert space? The answer is yes.

I'll use the Heisenberg picture, where all time-dependence is carried by the observables. (This was also implicit in the natural counterexamples mentioned above.) At any given time $t$, the model has only five observables: $A_1(t),A_2(t),A_3(t),A_4(t)$ and the time-indepenent Hamiltonian $H$. Choose the four self-adjoint operators $A_k(0)$ to have these properties: $$ A_j(0)A_k(0)=-A_k(0)A_j(0)neq 0 tag{1} $$ and $$ big(A_k(0)big)^2 = 1. tag{2} $$ Below, I'll show that such operators do exist. The time-dependence of the observables is defined by $$ A_k(t) equiv e^{iHt}A_k(0) e^{-iHt}. tag{3} $$ If we take the Hamiltonian to be $H=0$, then $A_k(t)=A_k(0)$, so the model has only five observables altogether, if we count the (trivial) Hamiltonian.

To prove that such a model cannot have a CSCO, first note that the smallest Hilbert space on which such a system of operators can be faithfully represented is a four-dimensional Hilbert space. Equation (2) says that each observable has no more than two eigenvalues, namely $+1$ and $-1$, and equation (1) says that none of the observables commute with each other (except for the trivial observable $H$), so every set of commuting observables has the form $(A_k(0),H)$ for some $k$. Such a pair cannot be a CSCO, because it has only two eigenvalue-pairs, namely $(1,0)$ and $(-1,0)$. This is not enough to discern all four of the state-vectors that any complete basis in this Hilbert space must have, so a CSCO cannot exist.

That completes the proof, except that we still need to show that operators with the properties (1)-(2) actually exist. To show this, first define the $2times 2$ self-adjoint matrices begin{gather} X = left(begin{matrix} 0&1 1&0 end{matrix}right) hspace{1cm} Y = left(begin{matrix} 0&i-i&0 end{matrix}right) Z = left(begin{matrix} 1&0 0&-1 end{matrix}right) hspace{1cm} I = left(begin{matrix} 1&0 0&1 end{matrix}right), tag{4} end{gather} and notice that $X^2=Y^2=Z^2=I$ and that $X,Y,Z$ all anticommute with each other. Then define these self-adoint $4times 4$ matrices: begin{gather} A_1(0) = left(begin{matrix} 0&X X&0 end{matrix}right) hspace{1cm} A_2(0) = left(begin{matrix} 0&Y Y&0 end{matrix}right) A_3(0) = left(begin{matrix} 0&Z Z&0 end{matrix}right) hspace{1cm} A_4(0) = left(begin{matrix} 0&iI -iI&0 end{matrix}right). tag{5} end{gather} These satisfy the conditions (1) and (2), which shows that operators having those properties do exist.

A challenge

In the contrived example shown above, the observables don't all commute with each other. This is typically what people mean when they say that a model is "quantum." However, we could also adopt a more narrow definition of "quantum," such as requiring incompatibility with any non-contextual hidden-variables model. I haven't tried to determine whether or not the contrived model above has that property. If it doesn't, then finding a no-CSCO model on a finite-dimensional Hilbert space that also has this extra property might be an interesting challenge.

---

References:

1. Yngvason (2004), "The Role of Type III Factors in Quantum Field Theory" (http://arxiv.org/abs/math-ph/0411058)

2. Ruetsche and Earman, "Interpreting probabilities in quantum field theory and quantum statistical mechanics" (http://pitt.edu/~jearman/EarmanRuetsche2011a.pdf)