In what way observables are a representation of the symmetry group?

Physical symmetries are in particular a certain transformation of the system. Translations displace the system from a point to another point. Rotations rotate the system. In general, in order to make sense of this statement one needs to give a prescription for the way that this transformation acts on your description of the system. In particular, you need to prescribe the way that the symmetry acts on your observables. In other words, the symmetry group should act on your space of observables.

Now, a priori, I don't see why this action should be linear. The only example I can think of right now is a gauge transformation of the EM field $A_mumapsto A_mu+partial_muLambda$, which is affine. This is not a great example though because it is not an observable. In any case, in QM there is an additional piece of information which restricts the symmetries to be linear. This is that symmetries are described by unitary/anti-unitary operators. This is known as Wigner's theorem. As a consequence, they have to form a representation on the Hilbert space of your theory.