Classification vs Regression Training

I’m working on a problem where I have images(600) and corresponding scores. I have used VGG19 pretrained architecture where I connected the last CNN layer of VGG19 with 3 new fully dense layer. As a general process I’m doing the whole model training Vgg19(incremental) + 3 FC layers.

As a part of training exercise, initially I divided the scores in to few classes and trained the same architecture as a classficaion model. Accuracy is close to 99% on testing data as well.

Now if I’m using the same architecture for regression problem (after adjusting the necessary changes), I observed altogether different behavior.

From the first epoch itself, even for the training data it always predict the same outcome. It get changed with every epoch, but predict the the same logit for all the enteries in the dataset.

I’m not sure what could be the reason.

Please provide your answer.