Symmetry transformations of states and operators

If you are only making a basis change, you have to transform both, otherwise you will end up with inconsistent results. The physics of the system must not depend on the choice of basis. Using $|psi'rangle = hat{U}^dagger |psirangle$ and $hat{A}' = hat{U}^dagger hat{A} hat{U}$, the transformation exactly cancels when calculating $$langlepsi'|hat{A}'|psi'rangle = langlepsi|hat{A}|psirangle$$ as it needs to be.

The laws of physics should be the same in all reference frames. This means that if your measurement apparatus is measuring a value in one Reference Frame (RF), then it should measure the same thing in another RF. The only way to satisfy this is if you apply the transformation $hat{U}$ to both the kets and the operators, so that begin{align} langle hat{A} rangle_{F'} &= langle psi vert hat{A} vert psi rangle_{F'} &= (langle psi verthat{U}^{dagger}) (hat{U} hat{A} hat{U}^{dagger})(hat{U}vert psi rangle)_F &= langle psi verthat{A}vert psi rangle_F &= langle hat{A} rangle_F end{align}

One way to look at this is thinking the transformation on the kets are rotating the particle and the transformation on the operator is rotating the measurement apparatus alongside, so that the measurement result is the same.

As a side note, although I talk about measurement apparatuses, the same conclusion can be reached from more abstract terms. We can say that we expect the expectation value of a given operator to be observer independent. In particular, a given eigenstate of an operator $hat{A}$ should still be an eigenstate, regardless of the observer, and should have the same eigenvalue for all observers. This is accomplished by the above transformation on both $hat{A}$ and the eigenstate.

If you want, on the other hand, not describe the system in another reference frame, but make a physical transformation on the system (i.e. rotate a particle by rotating the beamer that is producing it, or evolve the system in time), then you should apply $hat{U}$ only to one of the objects: either the kets or the operators, not both. The fact that you can choose which one you are applying the transformation to is, as you mentioned, analogous to the situation of the Heisenberg/Schroedinger pictures.

TL;DR: Changes of reference frame transform both objects; Physical transformations on the system transform either, but only one.

The "changing frames" language might be confusing the issue. Coordinate systems and observers are two independent concepts. All of the observers in any given scenario can be described using a single coordinate system. The choice of coordinate system doesn't matter at all, aside from mathematical convenience. Some coordinate systems might be more convenient for some observers, but that's the only connection between them. It's a matter of convenience, not necessity.

So, the answer depends on what you mean by "changing frames":

• If it means changing coordinates, so that the given unitary transformation is being used to represent a mere change of coordinates, then we need to apply it to everything (operators and states) so that it doesn't change any predictions. Changing the coordinate system can't have any physical consequences, because coordinates are just labels.

• If it means changing the observer, then the answer depends on what kind of model we're using. I'll call them in-practice models and in-principle models.

In practice, solving the Schrödinger equation for a one multi-electron atom is already difficult enough, nevermind solving the Schrödinger equation for a person made of jillions of molecules. So, instead of treating the observer as a physical entity, we usually omit the observer, and instead we artificially specify which observable is being measured. That's what I mean by an in-practice model. If $O$ is an observable and $U$ is a unitary operator that implements a rotation, then $U^dagger OU$ is a different observable. In any given state, we could choose to measure either $O$ or $U^dagger OU$, and these should typically give different results. So in this case, we should apply $U$ only to the observable, not to the state. Equivalently, we could apply $U$ only to the state, not the observable. These two options are equivalent, just like the Schrödinger and Heisenberg pictures are equivalent. (This equivalence comes from the fact that applying the same $U$ to both the observable and the state doesn't have any physical consequences.)

In principle, quantum theory allows using a model that includes complex macroscopic observers (like people) as part of the physical quantum system. That's what I mean by an in-principle model. In this type of model, changing the observer means changing the state, because the state describes the whole physical system including the observer. But this isn't a symmetry operation. If a person moves to a different location in the laboratory, then the state — which describes both the laboratory and the person — has changed in some very complicated way. Such a complicated change cannot be described by anything as simple as an overall translation or rotation. We can still describe it as a unitary transformation if we want to, because any change from one given state to another given state can be described by a unitary transformation, but it's not a symmetry.

By the way, even with an in-principle model, we still need to tell the theory which outcome we actually experience when an observable is measured. Using an in-principle model doesn't avoid the need for the usual projective-measurement rules, but it does allow us to defer application of those rules until after the physical process of observation — which the model itself describes — is complete.