Training CNN for object detection

I have an idea for build object detection model and I would like to share it with community in order to see if it makes sense.

Dataset:

• I’ve gathered images with different shapes and annotated them with labeled bounding boxes (I am trying to predict 1 class only).
• Images are really large (e.g. 5000×3000), which is why I’ve trained a CNN Classification model on their PATCHES (299 x 299) for binary classification.

My question is this, is it possible for me to:

1. Build new neural network but this time train it for BOUNDING BOX REGRESSION on full images (remember that I have annotations (i.e. bbox coordinates))
2. Use previously trained PATCH CLASSIFIER in order to classify ROI’s that were detected by newly trained network for bounding box regression (remember that bounding box coordinates are localizing an object that previous patch classifier was trained to classify).

Is this a "solid" idea for object detection? Or do you guys suggest to fine-tune Faster RCNN or something? Problem is that my data is quite specific (medical data) and I would need to use my own classifier for feature extraction (not a pre-trained classifier on image net).

Thanks!