Gradient computation

Question

I am beginner in data-science. I am trying to understand this PyTorch code for gradient computation using custom autograd function:

class MyReLU(torch.autograd.Function):

@staticmethod
def forward(ctx, x):

ctx.save_for_backward(x)
    return x.clamp(min=0)

def backward(ctx, grad_output):

x, = ctx.saved_tensors
    grad_x = grad_output.clone()
    grad_x[x < 0] = 0
    return grad_x

However, I don't understand this line : grad_x[x < 0] = 0. Can anyone explain this part?

user12075 · Accepted Answer

The example you find is calculating the gradient for the ReLU function, whose gradient is
$$text{ReLU}'(x)=left{
begin{array}{c l}	
     1 & text{if } x>0
     0 & text{if } x<0
end{array}right.$$

Therefore when x<0, you make the gradient 0 by grad_x[x < 0] = 0.

Gradient computation

One Answer

Add your own answers!

Ask a Question