Bib number recognition using Keras

We can break down the task into two parts.

In the first part, our aim will be to localize regions where the bibs could be found. This is an object detection task, so you can use YOLO, Faster RCNN etc. YOLO could be a better choice as it is fast and efficient. We don't need semantic segmentation here. A number of GitHub repositories are available which enable to train a custom dataset for detecting bibs in images.

In the second part, as mentioned in the question, a CNN trained on MNIST will not be enough. We will need an OCR ( Optical Character Recognition ) which would get the text ( number ) from the image.

The localizer will produce bounding box coordinates. The image will be then cropped by those coordinates. This cropped image could be passed into an OCR engine and finally, you will get the number present on the bib.

For training YOLO on a custom dataset, see here.

For an OCR engine, see here.

Here are a few other ways than what Shubham suggested, with probably a lower development time.

Option 1: EAST & OCR

Instead of using Object recognition, you can use the EAST algorithm for scene text localization, and then OCR it to recognize the bib number. EAST is quite robust, so this will probably not be very noisy. You will need a bit of preprocessing before carrying out OCR. For OCR, you can use tesseract-ocr.

Option 2: Number plate recognition

Your problem is very close to recognizing number plates, which is a well-studied problem at this point.

Option 3: Off the shelf Scene text detection

DeepSceneTextReader