How to speed up passing of images to a GPU

Question

I am using Ubuntu 16.04 LTS installed on a 4TB HDD. I am working with a large dataset (more than 30 GB and 150,000 images). I have a single 11GB GTX 1080 Ti GPU card on my system. I am training a mask RCNN written in Keras. I noticed that passing the entire dataset of 150,000 images to a single GPU from the HDD will take few weeks at least to process a single epoch. Hence I want to speed up passing an image to a GPU.

So I bought a 256 GB SSD, mounted it and copied only the images directory to the SSD. Then I began to train the Keras model. All python files including the Anaconda install are on the HDD. I noticed that merely moving the images to SSD did not speed up the training. Does anyone know how to speed up the training process by sending images from the SSD to the GPU? Do I need to move the python files and anaconda install to the SSD as well? I do not wish to reinstall the entire OS on the SSD.

Media · Answer

Try to use batch version for training. You have to implement a function to manage your data which exist in the memory. For each batch, you can load the data into your memory as much as possible, high RAM size can help. Then you can pass a batch to your network, the size of graphical memory matters here, using the current data which is in memory now. By doing that you reduce the number of disk calls. When it happens to load data from disk, you fill the memory as much as possible, the disk speed matters here; consequently, high speed SSDs are valuable for facilitating the reading operation.

Answered by Media on December 4, 2020

How to speed up passing of images to a GPU

One Answer

Add your own answers!

Ask a Question