TransWikia.com

Speeding up execution of Python code in QGIS's Python console

Geographic Information Systems Asked by timpan on January 13, 2021

I have learned so much from this community. About two weeks ago I had separate code for each file that I was working with, now I am executing the following code:

from pathlib import Path
import shutil
import glob
import os

mapp_csv = "/Users/name/Desktop/Regnmappen/steg1/"
InFlPth = "file://"+mapp_csv
end = "?type=csv&quote=&escape=&skipEmptyFields=Yes&maxFields=10000&detectTypes=yes&xyDms=yes&xField=lon&yField=lat&crs=EPSG:4326&spatialIndex=yes&subsetIndex=no&watchFile=no"

mapp_lager = "/Users/name/Desktop/Regnmappen/NUTS_uppdelade/"
add = "|layername="

mapp_attribut = "/Users/name/Desktop/Regnmappen/attributefterposition/"

exists = 0
missing = 0

for region_filepath in glob.glob(f"{mapp_lager}*.gpkg"):
    region_file = Path(region_filepath).name
    filename2 = os.fsdecode(region_file)
    if region_file.endswith(".gpkg"):

        for year_file in os.listdir(mapp_csv):

            if year_file.endswith(".csv"):

            # year_content = os.fsdecode(year_file)
                if Path(os.path.join(mapp_attribut, f"{region_file.replace('.gpkg', '')}{year_file.replace('.csv', '')}.gpkg")).exists():
                    exists += 1
                    continue
                else:

                    missing += 1
                    uri=InFlPth+year_file+end
                    vlayer = QgsVectorLayer(uri, "temp", "delimitedtext")
                    path_to_gpkg=mapp_lager+filename2+add+filename2[:-5]
                    wlayer = QgsVectorLayer(path_to_gpkg, "", "ogr")
                    output = mapp_attribut+filename2[:-5]+year_file[:-4]
                    processing.run("qgis:joinattributesbylocation", {'INPUT':vlayer,'JOIN':path_to_gpkg,'PREDICATE':[5],'JOIN_FIELDS':[],'METHOD':0,'DISCARD_NONMATCHING':True,'PREFIX':'','OUTPUT':output})
                    continue

print(f"Found {exists} files, but missing {missing}")

for file in os.listdir(mapp_attribut):

    filename = fs.decode(file)
    if filename.endswith(".gpkg"):

        uri = mapp_attribut+filename
        vlayer = QgsVectorLayer(uri, "", "ogr")
        feats_count = vlayer.featureCount()
        if(feats_count>=3):
            
            singelton="/Users/name/Desktop/Regnmappen/singelton/"+filename
            processing.run("native:multiparttosingleparts", {'INPUT':vlayer,'OUTPUT':singelton})
            
            vlayer = QgsVectorLayer(singelton, "", "ogr")
            voronoi_out="/Users/name/Desktop/Regnmappen/voronoi_out/"+filename
            processing.run("qgis:voronoipolygons", {'INPUT':vlayer,'BUFFER':10000,'OUTPUT':voronoi_out})
            
            overlay="/Users/name/Desktop/Regnmappen/NUTS_uppdelade/"+filename[:-11]+".gpkg"+add+filename[:-11]
            layername=QgsVectorLayer(voronoi_our,"","ogr")
            output = "/Users/name/Desktop/Regnmappen/tre/"file
            processing.run("native:clip", {'INPUT':layername,'OVERLAY':overlay,'OUTPUT':output})
            
            continue
        if(feats_count==2):
            dirrdirr2="/Users/name/Desktop/Regnmappen/två/"
            move=dirrdirr2+filename4
            shutil.move(layername, move)
        if(feats_count==1):
            dirrdirr1= "/Users/name/Desktop/Regnmappen/ett/"
            move=dirrdirr1+filename4
            shutil.move(layername, move)
        if(feats_count==0):
            dirrdirr="/Users/name/Desktop/Regnmappen/noll/"
            move=dirrdirr+filename4
            shutil.move(layername, move)

The code it running fine, however slow. I was hoping to have generated 160 000 files in the first part of the code during my 6 h sleep but woke up to less than 70 000 generated files. Is there anyway to speed up my code? I have a 2,7 GHz Quad-Core Intel Core i7, maybe that is too weak? I am using the QGIS built in Python console as I have been unable to set it up externally.

I need my code to do the following things:

  1. Open CSV-files in one folder and perform a join attributes by location with files in another folder.
  2. Asses how many observation the new file contains.
  3. If it contains three or more observations, make multiparts into singelparts, create a Voronoi and then a native clip.
  4. If it contains less than three observations, move to the file to a different folder (one folder for all files containing 2 observations, one for all files containing 1 observation and so on.

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP