Training accuracy is ~97% but validation accuracy is stuck at ~40%

I am trying to classify images into 27 classes using a Conv2D network. The training accuracy rises through epochs as expected but the val_accuracy and val_loss values fluctuate severely and are not good enough.

I am using separate datasets for training and validation. The images are 256 x 256 in size and are binary threshold images.

There are 22127 images in the training set (~800 for each class) and
11346 images in the validation set (~400 for each class) so I believe there is no class imbalance.

Here is my architecture:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 126, 126, 32)      320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 63, 63, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 61, 61, 32)        9248      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 30, 30, 32)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 28800)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 128)               3686528   
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 96)                12384     
_________________________________________________________________
dropout_2 (Dropout)          (None, 96)                0         
_________________________________________________________________
dense_3 (Dense)              (None, 64)                6208      
_________________________________________________________________
dense_4 (Dense)              (None, 27)                1755      
=================================================================



Total params: 3,716,443
Trainable params: 3,716,443
Non-trainable params: 0

Found 22127 images belonging to 27 classes.
Found 11346 images belonging to 27 classes.

here is the code I used to make this model:

classifier = Sequential()

classifier.add(Convolution2D(32, (3, 3), input_shape=(sz, sz, 1), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))
classifier.add(Convolution2D(32, (3, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))

classifier.add(Flatten())

classifier.add(Dense(units=128,
                     activation='relu'))
classifier.add(Dropout(0.40))


classifier.add(Dense(units=96,
                     activation='relu'))
classifier.add(Dropout(0.40))


classifier.add(Dense(units=64,
                     activation='relu'))

classifier.add(Dense(units=27, activation='softmax'))

optimizer used is adam and loss function I used is categorical_crossentropy.

from keras import optimizers

opt = optimizers.Adam(learning_rate = 0.001)

classifier.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])

classifier.summary()

This is the code for flowing the data from the directories to the model

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

training_set = train_datagen.flow_from_directory('Dataset2/Training Data',
                                                 target_size=(sz, sz),
                                                 batch_size=10,
                                                 color_mode='grayscale',
                                                 class_mode='categorical',
                                                 shuffle = True)

test_set = test_datagen.flow_from_directory('Dataset2/Test Data',
                                            target_size=(sz , sz),
                                            batch_size=10,
                                            color_mode='grayscale',
                                            class_mode='categorical',
                                            #shuffle = True
                                            )

classifier_obj = classifier.fit_generator(
        training_set,
        #steps_per_epoch=22127,
        epochs=10,
        validation_data=test_set,
        #validation_steps=11346
    verbose = 1)

the output this code produced was:

Epoch 1/10
2213/2213 [==============================] - 302s 136ms/step - loss: 0.0130 - accuracy: 0.7108 - val_loss: 0.0604 - val_accuracy: 0.3293
Epoch 2/10
2213/2213 [==============================] - 312s 141ms/step - loss: 0.0035 - accuracy: 0.9371 - val_loss: 0.0279 - val_accuracy: 0.4069
Epoch 3/10
2213/2213 [==============================] - 305s 138ms/step - loss: 0.0024 - accuracy: 0.9558 - val_loss: 0.0260 - val_accuracy: 0.4266
Epoch 4/10
2213/2213 [==============================] - 336s 152ms/step - loss: 0.0018 - accuracy: 0.9680 - val_loss: 0.0452 - val_accuracy: 0.4323
Epoch 5/10
2213/2213 [==============================] - 310s 140ms/step - loss: 0.0015 - accuracy: 0.9731 - val_loss: 0.0339 - val_accuracy: 0.3659
Epoch 6/10
2213/2213 [==============================] - 317s 143ms/step - loss: 0.0014 - accuracy: 0.9746 - val_loss: 0.0415 - val_accuracy: 0.4496
Epoch 7/10
2213/2213 [==============================] - 297s 134ms/step - loss: 0.0012 - accuracy: 0.9777 - val_loss: 0.0379 - val_accuracy: 0.4512
Epoch 8/10
2213/2213 [==============================] - 285s 129ms/step - loss: 0.0012 - accuracy: 0.9782 - val_loss: 0.0157 - val_accuracy: 0.4603
Epoch 9/10
2213/2213 [==============================] - 274s 124ms/step - loss: 0.0012 - accuracy: 0.9795 - val_loss: 0.0289 - val_accuracy: 0.4430
Epoch 10/10
2213/2213 [==============================] - 274s 124ms/step - loss: 9.1970e-04 - accuracy: 0.9837 - val_loss: 0.0459 - val_accuracy: 0.4800
Model Saved
Weights saved

Seeing promising results on this, I increased the number of epochs in hopes of getting a higher val_Accuracy but the result was that the val_accuracy was lower than the previous one.

I changed some parameters and ran the code with 100 epochs overnight. This was the output:

Some of the things that I found in other answers and have tried but failed:

1. used SGD optimizer
2. used regularizers
3. played around with the number of units in Dense layers.

Based on other answers, I think that either my model is overfitting or the data is bad. Please help me as I am completely clueless.

import tensorflow.keras as keras

model = keras.models.Sequential()

model.add(keras.applications.ResNet50(include_top=False, weights='imagenet', input_shape=(128, 128, 3)))

model.add(keras.layers.Flatten())

model.add(keras.layers.Dense(27, activation='softmax')) 

model.layers[0].trainable=False

model.compile(keras.optimizers.Adam(0.001), loss='categorical_crossentropy', metrics=['accuracy'])

model.summary()

model_obj = model.fit_generator(training_set, epochs=1, validation_data=test_set, verbose = 1)