How to model properly sequential data when the output has to be used as part of the next input? Model off completely when it makes single mistake

I have time series data and am fitting a (LSTM) neural network. The time series data include let’s say a brain wave (var1) as well as the previous state (prev_state) and I want to predict a state (y_pred) based on the data and values over several steps before. The problem I run into is I can fit a neural network, or any type of model, on the training data but the training data already has all the correct state values; but when predicting one record at a time over the same training data everything gets thrown off and the model can get stuck in one state because a state prediction is used as input to the next record.

For example:

Training Data:
Var1    prev_state    y_true
22.2    1             0
20.1    0             1
25.1    1             1
28.5    1             1
30.0    1             1

Predicting not one a time but with already states known:
Var1    prev_state    y_pred
22.2    1             0
20.1    0             1
25.1    1             0 (HERE THE MODEL MADE THE WRONG PREDICTION BUT IT'S OK BECAUSE THE TRAINING DATA ALREADY KNOWS THE TRUE STATE IN THE NEXT RECORD)
28.5    1             1
30.0    1             1

Predicting one at a time and then inserting the prediction in the input of the next record:
Var1    prev_state    y_pred
22.2    1             0
20.1    0             1
25.1    1             0 (HERE THE MODEL MADE THE WRONG PREDICTION AND IN THE NEXT RECORDS ENTIRE PREDICTION SEQUENCE GETS THROWN OFF)
28.5    0             0
30.0    0             0

Is this a problem with over or under fitting? Not sure how to approach this. I’m wondering if the setup is just plain wrong and I’m wasting my time.