TransWikia.com

Does GDAL support multi-process ReadAsArray

Geographic Information Systems Asked on October 5, 2021

I have a very large TIFF file that I need to read and process block by block through ReadAsArray. If I have multiple processes calling this function at the same time, can I return the correct result?

One Answer

Yes, it's no problem**. Just make sure you pass each process a filepath not gdal.Dataset object. That's not so much of a problem with multiple processes, as passing an open gdal.Dataset will just fail, but can cause issues if you try to use multiple threads:

One important point is that the same ​GDALDataset object should not be accessed by several threads at the same time. But of course, it is fine to use 2 different handles pointing to the same file in 2 threads.

You shouldn't try and write to the same output file from different processes though. Use your main process to farm out the tasks to the child processes, collect the results and write the results.

** Note that if your process is IO limited rather than CPU limited (e.g if the processing of each block is quick), you may find it faster to process sequentially in a single process rather than multiple processes.

Correct answer by user2856 on October 5, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP