What does it mean to "bring back a rider" in bike racing?

I know drafting might come into play too but to simplify things, let's forget about drafting

You can't forget about drafting because it is a huge effect in road racing; quoting from How much benefit do we really get from drafting? in Cycling Tips:

Studies have shown drag reductions of between 27% and 50% for riders that are drafting

When a team in the peloton decides to bring that rider back, every rider on the team, maybe excluding the team lead/sprinter¹, will take their "turn" at the front of the train, before dropping back to let the next rider have a turn. At worst, this means they get a 27% advantage 6/7ths of the time (Tour de France teams have 8 riders), or still more than a 20% advantage. That advantage only gets bigger in more optimal drafting conditions or if any other teams contribute to the chase.

The rider out on their own gets none of that advantage, so your assumption that they would be more fresh isn't correct - in fact, it's not uncommon to see riders that have been out on their own for a long time spit out the back of the peloton soon after being caught as they have expended so much energy riding on their own.

^{1. In general, a team is trying to bring a rider back for a reason; for example, on a flat stage, they want to set the finish up for their sprinter. There's no point bringing the race back together just to have their sprinter tired from the chase and unable to compete effectively in the sprint, so the sprinter may take shorter, fewer or even no turns at the front of the train.}