TransWikia.com

SparseCategoricalcrossEntropy(from_logits=True) internally apply softmax?

Data Science Asked on July 18, 2021

Regarding Tensorflow/Keras SparseCategoricalcrossEntropy.

SparseCategoricalcrossEntropy(from_logits=True) expects the logits that has not been normalized by softmax. Then does SparseCategoricalcrossEntropy applies softmax internally?

I suppose applying softmax would not be required internally as taking argmax would give the index expected.

Please help clarify.

One Answer

Yes, Softmax function is called when logit=True

Infact, if we check the keras code [Link], the softmax output is ignored in every condition and tf.nn.sparse_softmax_cross_entropy_with_logits is called. This function calculate softmax prior to cross_entropy as explained [Here]

I suppose applying softmax would not be required internally as taking argmax would give the index expected.

Predicting the class is not the only task here. It has to calculate the appropriate loss value to provide feedback to the network
If the target is in probability values[OHE] i.e. sum=1, not an open-ended logit i.e. [-inf, inf]. So the output should be similarly normalized prior to the loss calculation.

Correct answer by 10xAI on July 18, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP