How to ventilate a chimney taken down to attic (roof line) level?

I had a leak in my chimney flashing, so the roofer recommended taking the chimney down to attic level. He repaired the roof where the chimney was (board, felt and roof tiles).

All was good, but a few days later I noticed there was condensation forming and dripping (back down the chimney) on the non-breathable felt just above the open chimney in the attic.

As a temp. workaround I stuffed the open chimney with some insulation roll and the condensation problem stopped. However, I’ve read that’s not a good idea as eventually the roll will become saturated.

So options:

1. Close up the chimney in the attic with a (patio) slab?
2. Add tile vents to allow the condensation to escape?
3. Any other ideas?