TransWikia.com

Keras: Very high loss for Autoencoder

Data Science Asked by Animeartist on June 13, 2021

I am trying to implement an autoencoder for prediction of multiple labels using Keras. This is a snippet:

input = Input(shape=(768,))
hidden1 = Dense(512, activation='relu')(input)
compressed = Dense(256, activation='relu', activity_regularizer=l1(10e-6))(hidden1) 
hidden2 = Dense(512, activation='relu')(compressed)
output = Dense(768, activation='sigmoid')(hidden2) # sigmoid is used because output of autoencoder is a set of probabilities

model = Model(input, output)
model.compile(optimizer='adam', loss='categorical_crossentropy') # categorical_crossentropy is used because it's prediction of multiple labels
history = model.fit(x_train, x_train, epochs=100, batch_size=50, validation_split=0.2)

I ran this in Jupyter Notebook (CPU) and I am getting loss and validation loss as:
loss: 193.8085 - val_loss: 439.7132
but when I ran it in Google Colab (GPU), I am getting very high loss and validation loss:
loss: 28383285849773932.0000 - val_loss: 26927464965996544.0000.

What could be the reason for this behavior?

One Answer

You should not use the categorical cross-entropy loss, but the binary cross-entropy.

The categorical cross-entropy is meant for categorical probability distribution. In your scenario, this means that only one label would be active at the same time. If multiple labels can be active at the same time, then they are independent, and you need to compute the binary cross-entropy.

Answered by noe on June 13, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP