Optomechanical interaction

I do not have a tidy, rigorous answer ready for you, but since you seem to be stuck at the level of intuition, consider a couple of things:

1. Total momentum is conserved, as you would expect, but that does not mean a subset of the system will behave in the intuitive way, as if that subset were an isolated system of its own. You need to consider the entire system: (a) the light coming in from free space and reflected back out, (b) the net force on the input coupling mirror, and (c) the force on the mobile back mirror.

You can calculate the (first-order) reflection coefficient of the resonator for a given frequency, and then calculate the net radiation pressure on the entire system via momentum conservation. This net pressure will of course be in the intuitive “recoil” direction, no matter the position of the mobile mirror or color of the light.

The difference between this calculation and yours is that yours is concerned with the pressure on only the mobile mirror, a single constituent among two making up the material system. (As an aside, which direction do you suppose is the net force on the input mirror? Into the cavity or away?)

2. We might calculate the optical force density using an expression generalized from the microscopic Lorentz Force Law, such as: $$F = (Pcdotnabla)E + Jtimes B,$$ where $E$ = electric field, $B$ = magnetic field, $P$ = electric polarization density $propto E$, and $J$ = current density $propto E$. This is the most commonly used force density expression (but there are others).

Notice that force is second order in field (and take my word for it that this globally conserves momentum). What’s crucial in this equation is the phase relationship between the material sources ($P$ and $J$) and the fields. Indeed, the relative phases determine the sign of the time-averaged force. Now consider what happens at the surface of a mirror in a resonator when the resonator is tuned off resonance. When you think about it this way, you realize that it is complicated. The sign of the force on either of the mirrors is not a simple question of local, simplistic momentum conservation. In a complex situation, the fields can conspire to give a complex result.