How to find gaps in a tile and geometry of tables in a faster way

I have a table that contains polygons within a tile and it is huge. I want to find the gaps in a tile, i.e. where polygons are not present within a tile. I came up with this query but it takes a lot of time.

SELECT (st_dump(ST_DIFFERENCE(m.geom::geography::geometry, 
ST_Union( ST_MakeValid(e.geom::geography::geometry))))).geom
    FROM tile m
    JOIN geom_table e on e.tile_id = m.feat_id
    WHERE  st_isvalidreason(e.geom) = 'Valid Geometry'
    GROUP BY e.tile_id, m.geom;

mainly because I think it is doing union of all geometries in a table. Could there be a better approach to it using PostGIS functions?