How did River Tam end up in Alliance hands between the series and movie

At the end of the 3-part comics series, which are the events after the final episodes, the operative is sent after River and Simon. So the breakout hologram scene must refer to events before the series. However the operative is in the current timeframe, he is viewing a past event. Also it is known that Simon spent “years" getting the clearances to gain access to River, something that could not have happened after the events of the series, since he was a known “fugitive" by that time.

The Firefly in universe sequence goes like this:

• River is taken to the government facility
• Simon rescues her
• They board the Serenity
• Lots of adventures
• Events of the movie

However it's shown to us in this order:

• They board the Serenity
• Lots of adventures
• River is taken to the government facility
• Simon rescues her
• Events of the movie

I think she was re-captured, when Simon originally broke her out, he wasn't aboard the Serenity yet, and it's the Serenity ship they board, so she must have been captured somewhere between the series ending and the movie beginning.

I always assumed that she was captured again because she's not in cryo during the holographic recording shown at the beginning of the film, and originally, Simon didn't break her out. He paid other people to sneak her out in cryo and only met the on Persephone afterward. I guess Whedon thought he had to punch up the back story for the movie.

It is a flashback on how Simon had a key role in rescuing River from the facility.

The question's premise is wrong.

River does not end up in Alliance hands between the series and movie.

The scene you are referring to is a holographic recording.

You can tell it's a hologram because of the lighting difference between the room and the image being projected.

It eventually details how Simon rescues his sister (watched by 'The Operative'), which leads you up to the beginning of the Firefly series, and cuts out to the beginning of the movie.

The script also backs this up,

THE OPERATIVE (O.S.)

Stop.

The action freezes.

(CONTINUING)
Lovely. Lovely. Backtrack.

The action REVERSES, taking us back to the moment of Simon and River on the gurney just before it rises.

THE OPERATIVE (O.S.)

(CONTINUING)
Stop.

There is a motionless beat, River frozen in that crouch, and he steps through what we now see is a hologram of the event.

And then later on:

THE OPERATIVE

River was your greatest success. A prodigy -- A phenomenon. Until her brother walked in eight months ago and took her from you.

(emphasis added)

So not only are the events a hologram, but the script shows they occurred 8 months prior to the film, which could easily place them before the series began.

I always assumed that the sequence at the start of the movie was a flashback to before the start of the TV series.

I guess that this isn't particularly clear - but the actual sequence is revealed to be a recording.