ANN regression accuracy and loss stuck

Question

I have a data set on predicting solar power generation,the dataset has 20 independent var and 1 dependent. The accuracy of my model is stuck at 60%. I have tried several models but this accuracy is best I could get other sucks even more.
Here is my code:
data_path = r'drive/My Drive/Proj/S.P.F./solarpowergeneration.csv'
dts = pd.read_csv('solarpowergeneration.csv')
dts.head()
X = dts.iloc[:, :-1].values
y = dts.iloc[:, -1].values
print(X.shape, y.shape)
y = np.reshape(y, (-1,1))
y.shape
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
from sklearn.preprocessing import StandardScaler
sc= StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
y_train = sc.fit_transform(y_train)
y_test = sc.transform(y_test)

import keras.backend as K  
def calc_accu(y_true, y_pred):
  return K.mean(K.equal(K.round(y_true), K.round(y_pred)))

def get_spfnet():
  ann = tf.keras.models.Sequential()

ann.add(Dense(X_train.shape[1], activation='relu'))
# ann.add(BatchNormalization())
  ann.add(Dropout(0.3))
  ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
 # ann.add(BatchNormalization())
  ann.add(Dropout(0.3))
  ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
 # ann.add(BatchNormalization())
  ann.add(Dropout(0.3))

ann.add(Dense(1))
  ann.compile(loss='mse',
              optimizer='adam',
              metrics=[tf.keras.metrics.RootMeanSquaredError(), calc_accu])
  return ann

spfnet = get_spfnet()
#spfnet.summary()
hist = spfnet.fit(X_train, y_train, batch_size=32, epochs=250, verbose=2)

the accuracy and loss graphs are
plt.plot(hist.history['calc_accu'])
plt.title('Model Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.show()
plt.plot(hist.history['root_mean_squared_error'])
plt.title('Model error')
plt.xlabel('Epochs')
plt.ylabel('error')
plt.show()

after 50 epochs nothing seems to improve, neither curve seems to overfit on the data
I tried other models like reducing layers and removing kernel regularizes, using
kernel_initlizers='normal' and 'he-normal'

but they perform poorly stuck at 20%.
dataset:

desertnaut · Accepted Answer

There are two issues here.
First, you are in a regression setting, where accuracy is meaningless - see What function defines accuracy in Keras when the loss is mean squared error (MSE)?
Although the linked thread is not directly applicable here (since you use your own custom calc_accu function), the essence of the argument remains: assessing K.equal(K.round(y_true), K.round(y_pred)) is not the correct way to measure the performance of a regression model. In such settings often the loss (MAE, MSE, RMSE) and the metric are the same; what you do here (accuracy aside), i.e having an MSE loss and an RMSE metric is also valid.
Second, there is no way to tell if an RMSE of ~ 0.5 is "good" or "bad" in itself, even more since here you scale your output y before feeding it to the model, hence the calculated RMSE is on these scaled values, and not in the true, unscaled ones. What you have to do is calculate the "true" RMSE in the original unscaled values, and then decide if it is good enough for your purpose or not. Essentially what you have to do is:

Get predictions for your (scaled) test data
Transform back your predictions to the initial scale of the original test data
Get the RMSE between these invert-transformed predictions and the original data

To do so (and a good practice in general), it is necessary to define different scalers for your x and y, i.e.:
from sklearn.preprocessing import StandardScaler

# feature scaling
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.transform(X_test)

# outcome scaling:
sc_y = StandardScaler()
y_train = sc_y.fit_transform(y_train)    
y_test = sc_y.transform(y_test)

Now, assuming the data have been scaled as I show above, here is how you should calculate your RMSE on the original (unscaled) test data:
from sklearn.metrics import mean_squared_error

y_pred = model.predict(X_test) # get model predictions (scaled inputs here)
y_pred_orig = sc_y.inverse_transform(y_pred) # unscale the predictions
y_test_orig = sc_y.inverse_transform(y_test) # unscale the true test outcomes

RMSE_orig = mean_squared_error(y_pred_orig, y_test_orig, squared=False)

It is this RMSE_orig that you should decide if it is satisfactory for your problem; plotting the original unscaled predictions y_pred_orig against the original (unscaled) outcomes y_test_pred might also be useful and informative.
See own answer in SO thread How to interpret MSE in Keras Regressor for a similar case.

Other than that, it is speculative to say anything more since we do not have your data; nevertheless, a general rule is that we don't use dropout by default - only when we suspect that our model might overfit the data. So, I suggest experimenting with all dropout layers removed, and possibly putting them back in only gradually, and only in the case you observe overfitting.

ANN regression accuracy and loss stuck

One Answer

Add your own answers!

Ask a Question