Image Stitching with large Parallax

I need to stitch images of 5 cameras together. The cameras are mounted on the sides of an underwater remote operated vehicle. It’s basicly a cupoid with sizes of 2.1m x 1.3m x 1.85m. The cameras are mounted at the sides of this cupoid. So one camera at the front, one at the back, one at the right side, one at the left side and one at the bottom pointing downwards. All of the cameras have fisheye lenses with a viewing angle of 180° or more and have a 4k resolution. The goal is to stitch the images in realtime and to view the result with a VR headset.

I’ve tried a pure geometric approach where I only used the positions and parameters of the cameras. This worked ok, but the seam between the images is only valid for a very certain viewing distance. Otherwise the seam is very visible and I only managed to stitch the horizontal cameras and not the bottom one.

I’ve also tried vrWorks 360 from NVidia but this only works for horizontal aligned cameras so the bottom camera is ignored. Also the stitching result for the horizontal cameras was really bad.

I also stumbled across Parallax-tolerant Image Stitching with this implementation. But it also only works for horizontal aligned images and doesn’t allow fisheye lenses.

My last approach was to use the calibration of PTGui and use this calibration in stitchEm which used to be a commercial stitching software but is now open source. This delivered the best results so far. But there are still some alignment errors and it only works okay if there are little to no near objects.

I also tried to use openCV directly but this was too overwhelming for me, since there are so many options to use the api and so many modules to configure.

Has anyone an idea how I could stitch these images efficiently together? I’d prefer an free open source solution but a payed closed source application is also acceptable if it gets the job done.