What loss function should I use for auto encoders for text

Question

I'm trying to implement an autoencoder for text. But I don't know which loss function I should use ? I tried using the mse but I get a huge loss 1063442. I'm trying to build a very simple autoencoder using only LSTM layers in Keras. On the keras-blog they use binary-crossentropy but I think the reason for this is because they use black and white images.

Thanks

mevoki · Answer

The cause of your huge loss is probably a lack of normalization. As an example, in the Keras-blog post you linked, they conduct such normalization by dividing the input by 255, which is the maximum value an element in the input can be of.

As for which loss function to use, if you have a look at this keras-blog post, they construct a LSTM-based auto encoder for text which uses categorical crossentropy as a loss function.

Ali Oskooei · Answer

From my experience, the cosine similarity loss (tf.keras.losses.CosineSimilarity) works best for text autoencoders with Keras. You can find the documentation at:
https://www.tensorflow.org/api_docs/python/tf/keras/losses/CosineSimilarity

What loss function should I use for auto encoders for text

2 Answers

Add your own answers!

Ask a Question