CNN Multi-Class Model Only Predicts 1 class for all test images

I am trying to build a CNN model to predict 42 classes. I used pre-trained models for this. I used Xception.

This is how I have imported my dataset:

train_datagen = ImageDataGenerator(rescale =1.0/255.0, 
                                   zoom_range = 0.2, 
                                   shear_range = 0.2, 
                                   horizontal_flip = True,
                                  validation_split = 0.2)
training_data = train_datagen.flow_from_directory(train_path,
                                                  target_size = (299,299),
                                                  batch_size = 32,
                                                  class_mode = 'categorical',
                                                 subset = 'training')
validation_data = train_datagen.flow_from_directory(train_path, 
                                                   target_size = (299,299),
                                                   batch_size = 32,
                                                   class_mode = 'categorical',
                                                   subset = 'validation')

I then built my model:

import keras
prior = keras.applications.Xception(include_top = False, weights = 'imagenet', input_shape = (299,299,3))
model = Sequential()
model.add(prior)
model.add(Flatten())
model.add(Dense(256, activation = 'relu'))
model.add(Dropout(0.1))
model.add(Dense(42, activation = 'softmax', name = 'Output'))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

My model performs quite well but I do not know why when I predict it on the test set, it predicts all the same class.

This is the epochs:

Epoch 1/3
2636/2636 [==============================] - ETA: 0s - loss: 1.9625 - accuracy: 0.4816
Epoch 00001: val_accuracy improved from -inf to 0.56412, saving model to xception.hdf5
2636/2636 [==============================] - 5041s 2s/step - loss: 1.9625 - accuracy: 0.4816 - val_loss: 1.6792 - val_accuracy: 0.5641
Epoch 2/3
2636/2636 [==============================] - ETA: 0s - loss: 1.4015 - accuracy: 0.6224
Epoch 00002: val_accuracy improved from 0.56412 to 0.64584, saving model to xception.hdf5
2636/2636 [==============================] - 5101s 2s/step - loss: 1.4015 - accuracy: 0.6224 - val_loss: 1.3240 - val_accuracy: 0.6458
Epoch 3/3
2636/2636 [==============================] - ETA: 0s - loss: 1.2084 - accuracy: 0.6747
Epoch 00003: val_accuracy did not improve from 0.64584
2636/2636 [==============================] - 4968s 2s/step - loss: 1.2084 - accuracy: 0.6747 - val_loss: 1.4000 - val_accuracy: 0.6373

However, when I predict its all one category:

['39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',

Any help is appreciated. I would also like to know how I can use pre-trained models for these.

base_model = keras.applications.xception.Xception(weights="imagenet", include_top=False)
model = keras.layers.GlobalAveragePooling2D()(base_model.output)
output = keras.layers.Dense(n_classes, activation="softmax")(model)
model = keras.Model(inputs=base_model.input, outputs=output)

for layer in base_model.layers:
    layer.trainable = False

optimizer = keras.optimizers.SGD(lr=0.2, momentum=0.9, decay=0.01)
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(train_set, epochs=15, validation_data=valid_set)

for layer in base_model.layers:
    layer.trainable = True

optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, decay=0.001)
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(train_set, epochs=5, validation_data=valid_set)