Derivative of activation function used in gradient descent algorithms

As the name suggests, Gradient Descent ( GD ) optimization works on the principle of gradients which basically is a vector of all partial derivatives of a particular function. According to Wikipedia,

In vector calculus, the gradient is a multi-variable generalization of the derivative.

At its core, GD computes derivatives ( in terms of Neural Networks ) of a composite function ( a neural network is itself a composite function ) because of the gradient descent update rule, which is,

$Large theta = theta - alpha frac{partial J}{partial theta}$

Where $theta$ is the parameter which needs to be optimized. In a neural network, this parameter could be a weight or a bias. $J$ is the objective function ( loss function in NN ) which needs to be minimized. So for $frac{partial J}{partial theta}$, we need to repeatedly apply the chain rule till we have a derivative of the loss function with respect to that parameter.

Intuition:

Sorry for the weird image. When GD is far away from the function minima ( where it tends to reach ) the value of $frac{partial J}{partial theta}$ is greater and therefore the updated value of $theta$ is smaller than the previous one. This updated value is scaled by the learning rate ( $alpha$ ). The negative sign indicates that we are moving in the opposite direction of the gradient.