1D convolutional neural network validation improvement

I created 1D CNN in Keras, but I’m having issues with validation loss and accuracy.
I have 24k records, 22 features. Is my model overfitting or what is going on so validation loss and accuracy is inconsistent? They rise and drop over epochs.
Here is my model

#model
n_obs, feature, depth = X_train.shape
batch_size = 20
kernel_size = 2
pool_size = 2
filters = 16


inp = Input(shape=(feature, depth))
C = Conv1D(filters=filters, kernel_size=kernel_size, strides=1)(inp)

C11 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(C)
A11 = Activation("relu")(C11)
C12 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(A11)
S11 = Add()([C12, C])
A12 = Activation("relu")(S11)
M11 = MaxPooling1D(pool_size=pool_size, strides=2)(A12)


C21 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(M11)
A21 = Activation("relu")(C21)
C22 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(A21)
S21 = Add()([C22, M11])
A22 = Activation("relu")(S11)
M21 = MaxPooling1D(pool_size=pool_size, strides=2)(A22)


C31 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(M21)
A31 = Activation("relu")(C31)
C32 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(A31)
S31 = Add()([C32, M21])
A32 = Activation("relu")(S31)
M31 = MaxPooling1D(pool_size=pool_size, strides=2)(A32)


C41 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(M31)
A41 = Activation("relu")(C41)
C42 = Conv1D(filters=filters, kernel_size=kernel_size, strides=1, padding='same')(A41)
S41 = Add()([C42, M31])
A42 = Activation("relu")(S41)
M41 = MaxPooling1D(pool_size=pool_size, strides=2)(A42)

F1 = Flatten()(M41)

D1 = Dense(21)(F1)
A6 = Activation("relu")(D1)
D2 = Dense(16)(A6)
D22 = Dense(8)(D2)
D3 = Dense(2)(D22)
A7 = Softmax()(D3)

model = Model(inputs=inp, outputs=A7)

lrate = LearningRateScheduler(0.001)
adam = Adam(lr = 0.001, beta_1 = 0.9, beta_2 = 0.999)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
history = model.fit(X_train, y_train, 
                    epochs=200, 
                    batch_size=batch_size, 
                    verbose=2, 
                    validation_data=(X_valid, y_valid),
                    callbacks=[lrate])

Training set is about 15k records, valid 1.5k and test 6k records. Although test set achieve 96% correct classification I’m wondering if this validation inconsistent behaviour is something I should fix.
What can I do about it?

EDIT: example data