Cause of large delay occurring before rendering of each frame?

I’m trying to do background CYCLES rendering on a multi-gpu server using NUMA. However, adding any gpu’s beyond 2 offers no speeds up as a unstated delay occurs before each keyframe. The more gpus I add the larger this delay gets. The renderer takes this delay into account but doesn’t display where it is coming from, see the terminal output below.

Fra:0 Mem:32.61M (0.00M, Peak 44.36M) | Time:00:02.25 | Remaining:00:00.00 | Mem:1.82M, Peak:1.82M | Scene, View Layer | Rendered 2/4 Tiles
Fra:0 Mem:28.61M (0.00M, Peak 44.36M) | Time:00:02.27 | Remaining:00:00.00 | Mem:1.82M, Peak:1.82M | Scene, View Layer | Rendered 3/4 Tiles
Fra:0 Mem:24.61M (0.00M, Peak 44.36M) | Time:00:02.28 | Mem:1.82M, Peak:1.82M | Scene, View Layer | Rendered 4/4 Tiles
Fra:0 Mem:24.61M (0.00M, Peak 44.36M) | Time:00:02.30 | Mem:1.82M, Peak:1.82M | Scene, View Layer | Finished
Fra:0 Mem:22.55M (0.00M, Peak 44.36M) | Time:00:02.63 | Compositing
Fra:0 Mem:22.55M (0.00M, Peak 44.36M) | Time:00:02.63 | Compositing | Determining resolution
Fra:0 Mem:22.55M (0.00M, Peak 44.36M) | Time:00:02.63 | Compositing | Initializing execution
Fra:0 Mem:46.61M (0.00M, Peak 46.61M) | Time:00:02.71 | Compositing | Tile 1-4
Fra:0 Mem:46.61M (0.00M, Peak 46.61M) | Time:00:02.72 | Compositing | Tile 2-4
Fra:0 Mem:46.61M (0.00M, Peak 46.61M) | Time:00:02.72 | Compositing | Tile 3-4
Fra:0 Mem:46.61M (0.00M, Peak 46.61M) | Time:00:02.72 | Compositing | Tile 4-4
Fra:0 Mem:46.55M (0.00M, Peak 46.61M) | Time:00:02.72 | Compositing | De-initializing execution
Fra:0 Mem:27.55M (0.00M, Peak 46.61M) | Time:00:02.72 | Sce: Scene Ve:0 Fa:0 La:0
Saved: '/dev/shm/blender_proc_33f1cc3d05404208b75dd9758392d710/rgb_0000.png'
 Time: 00:02.95 (Saving: 00:00.23)

Fra:1 Mem:23.23M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Synchronizing object | Suzanne
Fra:1 Mem:23.36M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Synchronizing object | Icosphere.002
Fra:1 Mem:23.37M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Synchronizing object | Icosphere.001
Fra:1 Mem:23.38M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Synchronizing object | Icosphere
Fra:1 Mem:23.40M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Synchronizing object | Cylinder.002
Fra:1 Mem:23.41M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Synchronizing object | Cylinder.001
Fra:1 Mem:23.42M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Synchronizing object | Cylinder
Fra:1 Mem:23.44M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Synchronizing object | Cube.002
Fra:1 Mem:23.44M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Synchronizing object | Cube.001
Fra:1 Mem:23.44M (0.00M, Peak 46.61M) | Time:00:01.55 | Mem:0.00M, Peak:0.00M | Scene, View Layer | Initializing

The above shows one of the delays happening between frames 0 and 1. As you can see over 50% of rendering is spent in this unmarked state (assumed both frames take similar time to render). What could be a possible cause for this? Is this something wrong with blender or a potential configuration fix and is there any way for me to log what blender is doing in the time period?

This render is being done with the BlenderProc which is based on blender. I’ve also made a small script to render an object using blender alone and the problem still occurs.