Skillversitas

Tensors and Operations

The Tensor is the NumPy array the GPU runs on: the torch.tensor the create, the shape the dimension, the dtype the precision, the device the location, and the operation (the +, the @, the torch.matmul) the autograd-tracked. The requires_grad is the flag (the True the gradient's record, the False the no), the .detach() the no-grad copy, and the no_grad context the inference's off-switch. This chapter covers the create (the tensor, the randn, the zeros, the from_numpy), the operation (the element-wise, the matmul, the view, the reshape), the requires_grad (the autograd's flag, the detach), and the device/dtype the move.

The Create

# the CREATE (the):
import torch

x = torch.tensor([1.0, 2.0, 3.0])        # the list
x2 = torch.tensor([[1, 2], [3, 4]])      # the 2D
r = torch.randn(32, 784)                 # the normal
z = torch.zeros(4, 10)                   # the 0
o = torch.ones(4, 10)                    # the 1
a = torch.arange(0, 10, 2)               # the step
n = torch.empty(3, 3)                    # the uninit
#   the
#   the

# the NUMPY (the
#   the):
import numpy as np
a = np.array([1.0, 2.0])
t = torch.from_numpy(a)      # the SHARE (the the
                             #   the no copy, the the
                             #   the change the both)
b = a.copy()
t2 = torch.tensor(b)         # the COPY (the the
                             #   the no share)
#   the

The create is the torch.tensor (the list the copy), the torch.randn (the normal, the weight's init), the torch.zeros/ones/empty (the fill), and the torch.from_numpy (the NumPy the share - the no copy, the change the both) versus the torch.tensor(numpy_array) (the copy, the no share). The shape is the (32, 784) (the 32 the batch, the 784 the feature), the dtype the float32 (the default), and the device the cpu (the .to("cuda") the move) - the NumPy the same API (the reshape, the transpose), the GPU the torch's, and the next section's operation is the autograd's path.

The Operation and the requires_grad

# the OPERATION (the
#   the):
a = torch.randn(4, 3)
b = torch.randn(3, 5)
c = a + 1               # the element-wise (the the
                        #   the the no broadcast)
d = a @ b               # the matmul (the
                        #   the (4,5), the)
e = torch.matmul(a, b)  # the the @ the the same
f = a.view(2, 6)        # the VIEW (the
                        #   the no copy, the the
                        #   the same size)
g = a.reshape(2, 6)     # the RESHAPE (the
                        #   the)
h = a.t()               # the TRANSPOSE (the the
                        #   the (3,4), the)
#   the

# the REQUIRES_GRAD (the):
w = torch.randn(3, requires_grad=True)  # the FLAG
y = (w * 2).sum()
y.backward()
print(w.grad)   # the the 2.0 (the
                #   the)
#   the the DETACH (the):
v = w.detach()      # the no-grad copy
with torch.no_grad():    # the INFERENCE
    z = w * 3         # the no record

The operation is the NumPy's (the element-wise the +, the @ the matmul, the view/reshape the shape, the t the transpose) plus the autograd's: the requires_grad=True is the flag (the w the 2.0 the gradient, the y.backward() the compute), the .detach() the no-grad copy (the v the no the path), and the torch.no_grad() context the inference's off-switch (the model.eval() plus the no_grad, the no graph the memory's save). The rule is the one: the requires_grad the train's (the weight the True), the no_grad the inference's (the predict the False), and the detach the loss's log (the tensorboard the no graph the keep) - the next chapter's autograd is the engine the backward runs, and the nn.Module the parameter's container.

The Working Rules

  • The torch.tensor the create, the from_numpy the share. The randn the weight's init, the zeros/ones/empty the fill, and the from_numpy the NumPy's share (the tensor(numpy) the copy) - the shape the (32, 784), the dtype the float32, the device the cpu the .to() the move.
  • The @ is the matmul, the view the no-copy. The a @ b the (4,3) x (3,5) the (4,5), the view/reshape the shape (the view the contiguous only), and the t the transpose (the (3,4) the no copy) - the NumPy the same API, the GPU the torch's.
  • The requires_grad is the flag, the backward the compute. The w the True (the w * 2 the y, the y.backward() the w.grad the 2.0), and the detach the no-grad copy (the tensorboard the log, the no graph).
  • The no_grad is the inference's off-switch. The model.eval() plus the torch.no_grad() the predict (the no graph, the memory's save) - the train's requires_grad, the inference's no_grad, the split the nn.Module's.

Summary

The Tensor is the NumPy array the GPU runs on: the torch.tensor the create (the list the copy, the from_numpy the share), the randn the weight's init, the shape the (32, 784), the dtype the float32, and the device the cpu the .to() the move. The operation is the NumPy's (the element-wise the +, the @ the matmul, the view/reshape the shape, the t the transpose) plus the autograd's: the requires_grad=True the flag (the backward the grad), the .detach() the no-grad copy (the tensorboard the log), and the torch.no_grad() the inference's off-switch (the model.eval() plus the no_grad, the no graph). The rule is the split: the train's requires_grad, the inference's no_grad, and the detach the log's - the next chapter's autograd is the engine the backward runs, and the nn.Module the parameter's container the model builds on.