Multiprocessing in a QGIS Plugin

I have developed a QGIS plugin that processes an image using a bunch of other information, but it runs slowly. I would like to parallelize, as I can do with Multiprocessing.Process outside a QGIS plugin. I have seen several questions about using Multiprocessing in a QGIS plugin (for example this one), but no real answers or examples. As far as I can tell, the problem is related to management of processes, but I don’t understand exactly how.

I’m using the following structure to create multiple (e.g. 5) processes:

def doprocessing():

### Test multiprocessing ###
import multiprocessing as multi
import worker

njobs = 5
jobs = []
for i in range(njobs):
    p = multi.Process(target=worker.worker)
    jobs.append(p)
    p.start()

for p in jobs:
    p.join()
    if str(p.exitcode) != '0':
        print ("Error running process_chunk as process")
        sys.exit(1)

The worker in my test just does something silly:

def worker():
"""worker function"""
#print 'Worker'

import random
import time

value = int(random.random() * 1000000)
time_of_process = time.time()

filename = "D:/Anders/SDB/multiprocessingtest_" + str(value) + ".txt"
output_file = open(filename, "w")
output_file.write(str(time_of_process))
output_file.close()

return 0

I call this from the plugin code like this (just showing you the last bit here):

    def run(self):
    """Run method that performs all the real work"""
    # show the dialog
    self.dlg.show()
    # Run the dialog event loop
    result = self.dlg.exec_()
    # See if OK was pressed
    if result:
        import multiprocessing_test_process as process
        process.doprocessing()

When I run this through my plugin, all kinds of crazy breaks loose, with multiple new instances of QGIS starting up, and the original QGIS freezing.

If I change the first code snippet to the following and run it outside QGIS (in PyCharm), it works as it should:

def doprocessing():

### Test multiprocessing ###
import multiprocessing as multi
import worker

if __name__ == '__main__':
    njobs = 5
    jobs = []
    for i in range(njobs):
        p = multi.Process(target=worker.worker)
        jobs.append(p)
        p.start()

    for p in jobs:
        p.join()
        if str(p.exitcode) != '0':
            print ("Error running process_chunk as process")
            sys.exit(1)

doprocessing()

So it’s related to the use of main or not. Any help, pointers, and preferably examples that are known to work in QGIS plugins would be great! Alternatively, if you simply know of a plugin QGIS that successfully uses Multiprocessing, please just point me to it and I will take it from there.

import subprocess
basepath = os.path.dirname(os.path.abspath(__file__))  # Get plugin path
fileToRun = basepath + "/my_code_that_uses_parallel_processing.py"
subprocess.Popen(["python", fileToRun, argsFile])