Object detection is not improving although loss is decreasing

I am working on a project for detecting buildings from satellite images (using OpenStreetMap data as labels) using the Tensorflow Object Detection API. I recently upgraded to Tensorflow 2 and chose EfficientDet D4 from the Model ZOO.

I train (fine-tune) this pre-trained model on my data. Throughout the training, I keep making predictions using always the current state of the model. The point is to see how the performance of the model gets better with training.

Problem: The ability to predict buildings in validation dataset is not improving at all. In fact, it is getting worse! However, the loss during training is constantly decreasing.

Here is an example – in blue are predictions of the model after less than 1 epoch while in red are predictions by a model trained much more:

To me, it doesn’t make any sense. I’ve spent loads of time trying different variations of this, using another dataset for evaluation, etc. But still I get worse results (in terms of precision, recall, f1) when I train the model more. And this is not due to overfitting, the model’s performance starts to decrease almost immediately.

Earlier, I was using Tensorflow 1 and SSD Inception V2 as a model. That worked as expected, i.e. the model’s ability to detect buildings was improving as the training progressed.

Question: What could be the cause of this and what can I try to solve this problem?I have run out of ideas.

I am new to machine learning so I am likely missing some important information here. Please let me know if more details are needed to answer this.