How do I fill in these holes from curtain rail?

I would start by checking the brick that fixing (rawlplug) goes into.

If the brick is damaged behind the plaster, then anything else you put in there will come out sooner or later.

I would consider, if the brick is sound, gluing (epoxy or similar strong glue) a wooden dowel into the hole - whittled to fit well. Once dry I would pre-drill the dowel (so the screw does not split it) and use a suitable screw.

If the brick is not sound, the epoxy may be sufficient or you will be looking at either re-locating the fixings or doing something with the brick, not easy though.