CUDA_ERROR_OUT_OF_MEMORY: out of memory. How to increase batch size?

It could be the case that your GPU cannot manage the full model (Mask RCNN) with batch sizes like 8 or 16.

I would suggest trying with batch size 1 to see if the model can run, then slowly increase to find the point where it breaks.

You can also use the configuration in Tensorflow, but it will essentially do the same thing - it will just not immediately block all memory when you run a Tensorflow session. It will only take what it needs, which (given a fixed model) will be defined by batch size.

---

You should alter your code example to be:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)

Environment:
1.CUDA 10.0
2.cuNDD 10.0 3.tensorflow 1.14.0
4.pip install opencv-contrib-python
5.git clone https://github.com/thtrieu/darkflow
6.Allowing GPU memory growth

Reference

If you are using Jupyter Notebook you should run the following code to clear your GPU memory so that you train perfectly

import gc
gc.collect()

If the problem still persists ,use smaller batch size like 4.