Insulation around the pipe outside of a kitchen exhaust

Context

I have a kitchen exhaust which works well enough, this question is about insulation. Last winter was quite a cold one for where I live, and I felt a nice fall of cold air coming from there. I want to fix this before next winter. I have no problem about ventilation in the house, so this pipe can be completely closed when not in use.

The pipe itself will receive an anti wind valve (not sure how it’s called in English – but it is something between the pipe and the grid outside, it has 2 wings with a light spring. When the exhaust is on, the wings are open, when it’s off the springs will close the wings).

Actual question

My question is what to do around the pipe, on the outside. There is quite a hole in the bricks:

• between the pipe and the bricks, I can easily pass 2 or 3 fingers, for about 6cm (2 inches+) deep
• further down there’s some isolation, but I can still pass one finger between the pipe and the isolation.

I could of course just fill the hole with insulation foam, but I’m not sure that it’s smart (then the pipe will be stuck there and cannot be moved any more).

I could put a snug layer of glass wool (or equivalent, suggestion welcome). It feels better, at least it can be removed, but I’m worried about possible rain going there. Would it be an issue?

What would be the smartest way forward?

Update: There is a grid on top, held in place by screws and the 4 holes that can be seen. I removed it for the picture. This grid does next to nothing for insulation.