TransWikia.com

Memory leak while using ParallelTable

Mathematica Asked on August 3, 2021

I am populating a table using ParallelTable. The entries of this table are given by a function written as a module, which depends on several modules itself. Here’s a rough skeleton of the code:

func1[vars1_]:=Module[{locals1}, stuff happens; Return[answer1]

func2[vars2_]:=Module[{locals2}, other stuff happens including: func1[vars2]; Return[answer2]

datarun = ParallelTable[func2[vars2],{i,1,n},{j,1,m}]

For small values of n and m, everything runs fine. However, for larger values, the memory usage on my machine slowly creeps up, until by the end it’s using about 98% of the memory. This also has the effect of vastly slowing down the computation, which is the reason I’m so annoyed–if I evaluate the function func2[vars2] for one set of vars2, it takes a couple seconds, whereas in the loop, once the memory has been eaten up, it takes a few minutes.

Things I’ve tried: I put ClearSystemCache[] and $HistoryLength=0 into each of my modules (after reading about similar problems here). I also included the statement ClearAll["Global'*"] in my main function. These things helped a fair bit (the problem used to be much worse, I won’t go into it), but it still seems that I have a memory leak somewhere. That all being said, my questions are:

  1. Is there something about the way I’ve nested modules that makes such a thing ripe for a memory leak?

  2. Being new to all this, how would I go about identifying where a memory leak was occuring?

  3. Is there anything I can just add to my code that ensures it won’t keep around anything unnecessary that might cause a leak? (I thought that’s what ClearAll["Global'*"] would do)

Also, in case it matters, my ParallelTable is using 12 cores, and the amount of memory that I have is 16 GB.

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP