When using Absolute Error in Gradient Descent, how to calculate the derivative?

Question

What is the derivative of the Loss Function (Absolute Error) with respect to the feature weights that is used to update the weights?

Couldn't find anything specific about it anywhere.

Philipp · Accepted Answer

MAE is not continuous in $y_{pred} = y_{true}$ and therefore there is no defined (bounded, direction independent) derivative at that point.

Elsewhere you have +1, where  $y_{pred} > y_{true}$ and -1 where  $y_{pred} < y_{true}$

Usually frameworks like TensorFlow, Keras, etc... use an approximate derivative for that point.

Media · Answer

You can simply approximate $f(x)=|x|$ by $f(x)=sqrt{x^2+c}$ where $c>0$.
You can also utilize subderivative method.

2 Answers