Skillversitas

Automatic Differentiation

The autograd is the engine: every operation on the requires_grad tensor records the gradient's path (the forward the value, the backward the grad), and the loss.backward() walks the graph reverse (the chain rule, the no symbolic the static). The grad is the tensor (the w.grad the shape the w), the retain_grad the intermediate's keep, the grad_fn the node the graph's, and the zero_grad the optimizer's reset (the grad the accumulate, the no zero the stale). This chapter covers the backward (the loss, the grad), the grad_fn (the graph's node), the zero_grad (the optimizer's reset), the retain_grad (the intermediate), and the nonzero the vector the backward's (the grad the direction).

The backward: The loss and the grad

# the BACKWARD (the
#   the):
import torch

w = torch.tensor(2.0, requires_grad=True)  # the
b = torch.tensor(3.0, requires_grad=True)  #   PARAM
y = w * w + b            # the FORWARD (the the y=4+3=7)
y.backward()             # the
print(w.grad)   # the the 4.0 (the the dy/dw = the 2w=4)
print(b.grad)   # the the 1.0 (the the dy/db = the 1)
#   the

# the SCALAR (the):
#   the
#   the
#   the
x = torch.tensor([1.0, 2.0], requires_grad=True)
y = x.sum()          # the SCALAR
y.backward()
print(x.grad)   # the the [1., 1.] (the
                #   the)

# the VECTOR (the):
#   the
#   the
#   the
x = torch.tensor([1.0, 2.0], requires_grad=True)
y = x ** 2           # the [1, 4] (the
                     #   the NON-scalar)
y.backward(torch.tensor([1.0, 0.5]))   # the the
                                        #   the VECTOR
print(x.grad)   # the the [2., 2.] (the
                #   the)

The backward is the engine's call: the loss.backward() (the loss the scalar) walks the graph reverse, and the w.grad is the tensor (the dy/dw the 2w the 4.0, the b.grad the 1.0) - the chain rule, the no manual derivative. The loss must be the scalar (the sum, the mean), and the non-scalar (the vector the x**2) needs the backward(vector) (the vector the direction, the 2.0 the 1.0 the 0.5 the weight). The grad is the accumulate (the backward twice the 2x, the no zero the stale), and the zero_grad (the next section) is the optimizer's reset - the optimizer.zero_grad() before the backward, the grad the fresh, and the rule is the one: the loss the scalar, the backward the grad, the zero_grad the reset, and the graph the release after the backward (the memory's, the retain_graph the keep).

The grad_fn and the zero_grad

# the GRAD_FN (the
#   the):
w = torch.tensor(2.0, requires_grad=True)
y = w * w + 3
print(y.grad_fn)   # the the <AddBackward0> (the the
                   #   the NODE the
                   #   the)
print((y - 3).grad_fn)   # the <MulBackward0> (the
                         #   the)
#   the
#   the

# the RETAIN_GRAD (the):
#   the
x = torch.tensor(1.0, requires_grad=True)
mid = x * 2
y = mid * 3
y.backward()
# print(mid.grad)   # the ERROR (the
#                   #   the
#                   #   the RELEASE)
x = torch.tensor(1.0, requires_grad=True)
mid = x * 2
mid.retain_grad()    # the KEEP
y = mid * 3
y.backward()
print(mid.grad)   # the the 3.0 (the
                  #   the)

# the ZERO_GRAD (the):
#   the
import torch.optim as optim
w = torch.tensor(2.0, requires_grad=True)
opt = optim.SGD([w], lr=0.1)
y = (w * w).sum()
y.backward()       # the the w.grad = the 4.0
opt.step()         # the the w = the 1.6
y = (w * w).sum()
y.backward()       # the the w.grad = the 3.2 + the
                   #   the 4.0 = the
                   #   the 7.2 (the
                   #   the)
opt.zero_grad()    # the RESET
y = (w * w).sum()
y.backward()       # the the w.grad = the 3.2 (the
                   #   the)

The grad_fn is the node (the <AddBackward0>, the <MulBackward0>), the graph's edge (the y.grad_fn the Add, the (y-3).grad_fn the Mul), and the retain_grad is the intermediate's keep (the mid.retain_grad() the mid.grad the 3.0, the no the ERROR the RELEASE) - the debug's, the tensorboard the intermediate's log. The zero_grad is the optimizer's reset: the grad is the accumulate (the backward twice the 7.2 the 3.2+4.0), and the opt.zero_grad() before the backward is the rule (the grad the fresh, the no stale) - the training loop (the next chapter) is the zero_grad, the forward, the loss, the backward, the step, the five the epoch the batch the repeat.

The Working Rules

  • The loss.backward() is the grad, the loss the scalar. The w.grad the dy/dw (the 2w the 4.0), the non-scalar the backward(vector) (the direction), and the graph the release after (the memory's, the retain_graph=True the keep).
  • The grad_fn is the node, the retain_grad the intermediate. The <AddBackward0> the graph's edge, the mid.retain_grad() the mid.grad (the 3.0, the no ERROR) - the debug's, the tensorboard the log.
  • The grad is the accumulate, the zero_grad the reset. The backward twice the 7.2 (the 3.2+4.0), and the opt.zero_grad() before the backward the rule (the fresh, the no stale) - the training loop's first step.
  • The requires_grad is the flag, the detach the no-grad. The weight the True (the train), the input the False (the no graph the input's), and the detach the log's (the tensorboard the no graph) - the autograd's three.

Summary

The autograd is the engine: the requires_grad tensor's operation records the gradient's path (the forward the value, the backward the grad), and the loss.backward() walks the graph reverse (the chain rule). The w.grad is the tensor (the dy/dw the 2w), the loss must be the scalar (the non-scalar the backward(vector) the direction), and the grad is the accumulate (the opt.zero_grad() before the backward the rule). The grad_fn is the node (the <AddBackward0> the graph's edge), the retain_grad is the intermediate's keep (the mid.grad the 3.0, the debug's), and the detach is the no-grad copy (the tensorboard the log). The rule is the training loop's five: the zero_grad, the forward, the loss, the backward, the step - the next chapter's nn.Module is the parameter's container the loop runs on, and the first-model the Linear the 784 to the 10 the MNIST's first.