Keras stateful LSTM returns NaN for validation loss

I’m having some trouble interpreting what’s going on in the training and validation loss, sensitivity, and specificity for my model. My validation sensitivity and specificity and loss are NaN, and I’m trying to diagnose why.

My training set has 50 examples of time series with 24 time steps each, and 500 binary labels (shape: (50, 24, 500)). My validation set has shape (12, 24, 500). Of course, I expect a neural network to overfit massively.

Because I was interested in a stateful LSTM, I followed philipperemy’s advice and used model.train_on_batch with batch_size = 1. I had multiple inputs: one called seq_model_in that is a time series, and one called feat_in that is not a time series (and so is concatenated in to the model after the LSTM but before the classification step).

As my classes are highly imbalanced, I also used Keras’s class_weights function. To make this function work in a multi-label setting, I concatenated two columns to the front of my responses (one of all 0s and one of all 1s), such that the final shape of the response is (50, 502).

 feat_in = Input(shape=(1,), batch_shape=(1, 500), name='feat_in')
 feat_dense = Dense(out_dim, name='feat_dense')(feat_in)
 seq_model_in = Input(shape=(1,), batch_shape=(1, 1, 500), name='seq_model_in')
 lstm_layer = LSTM(10, batch_input_shape=(1, 1, 500), stateful=stateful)(seq_model_in)
 merged_after_lstm = Concatenate(axis=-1)([lstm_layer, feat_dense])
 dense_merged = Dense(502, activation="sigmoid")(merged_after_lstm)

I have coded this model in Keras for time series prediction (multi-label prediction at the next time step):

The training and validation metrics and loss do not change per epoch, which is worrisome (and, I think, a symptom of overfitting), but I’m also concerned about understanding the graphs themselves.

Here are the TensorBoard graphs:

The training loss should (roughly) be decreasing per epoch, as should the validation loss. The training sensitivity and specificity are 92% and 97.5%, respectively (another possible sign of overfitting).

My questions are:

• Am I right in thinking the sensitivity and specificity graphs should all have the same general shape as the train_spec graph?
• Why does the training sensitivity graph look like this?
• Why does the validation loss return NaN?

def clipped_squared_error_loss(y_true, y_pred):
    y_true = K.clip(y_true, K.epsilon(), 1)
    y_pred = K.clip(y_pred, K.epsilon(), 1)
    return K.sum(K.square(y_pred - y_true))