TransWikia.com

Intersecting large shapefiles onto a grid in QGIS

Geographic Information Systems Asked by Shaun Doolin on April 2, 2021

I have 21 separate shapefiles containing data for Ireland as various
polygons on QGIS 3.10. I’m trying to intersect each shapefile onto a
250m² grid of Ireland so that I’ll have 21 separate shapefiles in a grid
format. I’ll then spatially join each dataset using the corresponding grid
"ID"s to create one shapefile containing all the data in a 250m² grid
format for Ireland.

My problem is that the intersections onto the grid are taking days to
complete. One intersection is currently on 56% after 31 hours, and the
intersected shapefile only contains 26 features and 1 column of data,
although there will be 1,000,000+ features in the 250m² grid output.
Other datasets that will also need to be intersected onto the grid are
much larger (4,000,000+ features) as they were converted from rasters.

I have enabled "Render layers in parallel using many CPU cores" and
have created spatial indexes for the two layers and don’t know how
else to speed up the processing. I’m running QGIS 3.10 on a PC with
8GB RAM and an i5 processor. Is the lack of processing power on my PC
the only reason the intersection is taking so long, or is there
another way to increase the speed of the intersection?

I’m performing the intersections by 'Vector > Geoprocessing Tools > Intersection'. With the Input layer as my 250m grid, and the Overlay layer as the shapefile to be intersected onto the grid.

One Answer

When processing big data sets the biggest cost is usually related to moving your data om pace for processing. If your processing takes a long time while your CPU performance is still low you, should start looking at how your data is accessed. Working on data placed on a network drive is typically something which will lead to long processing time.

I would suggest two alternative approaches to be able to do this more efficiently:

  1. Load the files into PostGIS and process it there using spatial SQL query. It is relatively easy to set up a job using spatial queries. You can install PostGIS locally on your computer. Remember do index your data.
  2. Avoid using Shapefiles. Convert your data to Geopackage. Then try your processing again.

Answered by ragnvald on April 2, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP