Cross-entropy loss explanation

The answer from Neil is correct. However I think its important to point out that while the loss does not depend on the distribution between the incorrect classes (only the distribution between the correct class and the rest), the gradient of this loss function does effect the incorrect classes differently depending on how wrong they are. So when you use cross-ent in machine learning you will change weights differently for [0.1 0.5 0.1 0.1 0.2] and [0.1 0.6 0.1 0.1 0.1]. This is because the score of the correct class is normalized by the scores of all the other classes to turn it into a probability.

Let's see how the gradient of the loss behaves... We have the cross-entropy as a loss function, which is given by

$$ H(p,q) = -sum_{i=1}^n p(x_i) log(q(x_i)) = -(p(x_1)log(q(x_1)) + ldots + p(x_n)log(q(x_n)) $$

Going from here.. we would like to know the derivative with respect to some $x_i$: $$ frac{partial}{partial x_i} H(p,q) = -frac{partial}{partial x_i} p(x_i)log(q(x_i)). $$ Since all the other terms are cancelled due to the differentiation. We can take this equation one step further to $$ frac{partial}{partial x_i} H(p,q) = -p(x_i)frac{1}{q(x_i)}frac{partial q(x_i)}{partial x_i}. $$

From this we can see that we are still only penalizing the true classes (for which there is value for $p(x_i)$). Otherwise we just have a gradient of zero.

I do wonder how to software packages deal with a predicted value of 0, while the true value was larger than zero... Since we are dividing by zero in that case.

The problem is that the probabilities are coming from a 'complicated' function that incorporates the other outputs into the given value. The outcomes are inter-connected, so this way we are not deriving regarding to the actual outcome, but by all the inputs of the last activation function (softmax), for each and every outcome.

I have found a very nice description here where the author shows that the actual derivative is $p_i - y_i$.

Other neat description can be found here.

I think that using a simple sigmoid as a last activation layer would lead to the approved answer, but using softmax indicates different answer.

Let's start with understanding entropy in information theory: Suppose you want to communicate a string of alphabets "aaaaaaaa". You could easily do that as 8*"a". Now take another string "jteikfqa". Is there a compressed way of communicating this string? There isn't is there. We can say that the entropy of the 2nd string is more as, to communicate it, we need more "bits" of information.

This analogy applies to probabilities as well. If you have a set of items, fruits for example, the binary encoding of those fruits would be $log_2(n)$ where n is the number of fruits. For 8 fruits you need 3 bits, and so on. Another way of looking at this is that given the probability of someone selecting a fruit at random is 1/8, the uncertainty reduction if a fruit is selected is $-log_{2}(1/8)$ which is 3. More specifically,

$$-sum_{i=1}^{8}frac{1}{8}log_{2}(frac{1}{8}) = 3$$ This entropy tells us about the uncertainty involved with certain probability distributions; the more uncertainty/variation in a probability distribution, the larger is the entropy (e.g. for 1024 fruits, it would be 10).

In "cross"-entropy, as the name suggests, we focus on the number of bits required to explain the difference in two different probability distributions. The best case scenario is that both distributions are identical, in which case the least amount of bits are required i.e. simple entropy. In mathematical terms,

$$H(bf{y},bf{hat{y}}) = -sum_{i}bf{y}_ilog_{e}(bf{hat{y}}_i)$$

Where $bf{hat{y}}$ is the predicted probability vector (Softmax output), and $bf{y}$ is the ground-truth vector( e.g. one-hot). The reason we use natural log is because it is easy to differentiate (ref. calculating gradients) and the reason we do not take log of ground truth vector is because it contains a lot of 0's which simplify the summation.

Bottom line: In layman terms, one could think of cross-entropy as the distance between two probability distributions in terms of the amount of information (bits) needed to explain that distance. It is a neat way of defining a loss which goes down as the probability vectors get closer to one another.