LSTM-Model - Validation Accuracy is not changing

I am working on classification problem, My input data is labels and output expected data is labels
I have made X, Y pairs by shifting the X and Y is changed to the categorical value

Labels  Count
1      94481
0      65181
2      60448

    X   Y
    2   1.0
    1   2.0
    1   1.0
    2   1.0
    2   2.0

encoder = LabelEncoder()
test_labels = to_categorical(encoder.fit_transform(values[:,1]),num_classes=3)
train_X,test_X,train_y,test_y= train_test_split(values[:,0], test_labels,test_size = 0.30,random_state = 42)

print(train_X.shape)
print(train_y.shape)
print(test_X.shape)
print(test_y.shape)

(154076,)
(154076, 3)
(66033,)
(66033, 3)
Converting this to LSTM format

train_X = train_X.reshape(train_X.shape[0],1,1)
test_X = test_X.reshape(test_X.shape[0],1,1)

# configure network
n_batch = 1
n_epoch = 10
n_neurons = 100

ModelArchitecture

tf.keras.backend.clear_session()

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(n_neurons, batch_input_shape=(n_batch, train_X.shape[1],train_X.shape[2]), stateful=True),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(100, activation = 'relu',kernel_regularizer=regularizers.l2(0.0001)),
    tf.keras.layers.Dense(3, activation='softmax')
])

model.summary()
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['acc'])
history = model.fit(train_X,train_y,validation_data=(test_X, test_y),epochs=n_epoch, batch_size=n_batch, verbose=1,shuffle= False)

Validation Accuracy is not Changing

Epoch 1/5
154076/154076 [==============================] - 356s 2ms/step - loss: 1.0844 - acc: 0.4269 - val_loss: 1.0814 - val_acc: 0.4310
Epoch 2/5
154076/154076 [==============================] - 354s 2ms/step - loss: 1.0853 - acc: 0.4256 - val_loss: 1.0813 - val_acc: 0.4310
Epoch 3/5
154076/154076 [==============================] - 355s 2ms/step - loss: 1.0861 - acc: 0.4246 - val_loss: 1.0814 - val_acc: 0.4310
Epoch 4/5
154076/154076 [==============================] - 356s 2ms/step - loss: 1.0874 - acc: 0.4228 - val_loss: 1.0825 - val_acc: 0.4310
Epoch 5/5
154076/154076 [==============================] - 353s 2ms/step - loss: 1.0887 - acc: 0.4208 - val_loss: 1.0828 - val_acc: 0.4310

What can be the changes to improve the model.

Update2

After Increasing the learning rate of Rmsprop to 0.5 , Below is the training loss and validation loss

Epoch 1/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.4314 - acc: 0.3594 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 2/10
176087/176087 [==============================] - 369s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 3/10
176087/176087 [==============================] - 371s 2ms/step - loss: 1.3209 - acc: 0.3592 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 4/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.3209 - acc: 0.3592 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 5/10
176087/176087 [==============================] - 368s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 6/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 7/10
176087/176087 [==============================] - 368s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 8/10
176087/176087 [==============================] - 372s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 9/10
176087/176087 [==============================] - 371s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9379 - val_acc: 0.2739
Epoch 10/10
176087/176087 [==============================] - 369s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854

train_X = train_X.reshape(train_X.shape[0],10,1)
test_X = test_X.reshape(test_X.shape[0],10,1)