Loss function range normalization

Question

This is from a referee report in a conference to which I submitted my paper - I don't quite get it and I'm not sure what I need to do about it.

I use Euclidean loss and Softmax cross-entropy (pixelwise) loss in my model. The referee pointed out that the losses are not in the same range and they are not weighted.

I've never seen that done in other models (e.g. Faster R-CNN, Mask R-CNN). Any suggestions how to address this issue?

Adrien D · Answer

I agree with the referee. When your loss is composed from different terms, you should add a mechanism that enable you to adjust the contribution off those terms.
Just create a new hyperparameter (usually called lambda) and use it like that:
$L = L_1 + lambda L_2$
You have to find the right lambda with cross-validation.
This is used in most of the papers.
For example : https://arxiv.org/pdf/1806.08462v1.pdf page 3

Loss function range normalization

One Answer

Add your own answers!

Ask a Question