Skillversitas

A First Model: Linear Regression

The first model is the Linear the 1 to the 1 (the y = wx + b), the MNIST's 784 to the 10 the same pattern (the weight the (10, 784), the bias the (10,)). The nn.Linear is the Module (the weight the Kaiming the init, the bias the 0), the loss the MSE (the regression) or the cross_entropy (the classification), and the optimizer the SGD (the lr the 0.01). This chapter covers the Linear (the weight, the bias, the forward the x @ w.T + b), the MSE (the regression's loss), the SGD (the lr, the step), and the MNIST's Linear the 10 class the cross_entropy the first run.

The Linear: The weight and the bias

# the LINEAR (the):
import torch
import torch.nn as nn

layer = nn.Linear(784, 10)   # the
#   the
#   the
print(layer.weight.shape)   # the the (10, 784) (the
                           #   the
                           #   the)
print(layer.bias.shape)     # the the (10,) (the the
                           #   the)
#   the
#   the
#   the

x = torch.randn(32, 784)
out = layer(x)   # the (32, 10)
#   the
#   the FORWARD (the):
#   the
#   the
#   the

The Linear is the Module: the weight the (10, 784) (the Kaiming the init, the uniform the 1/sqrt(784)), the bias the (10,) (the 0 the init), and the forward is the x @ weight.T + bias (the (32, 784) the (784, 10) the (32, 10)) - the layer(x) the forward, the weight the requires_grad=True (the autograd the grad). The regression is the 1 to the 1 (the nn.Linear(1, 1), the y = wx + b the house price), the classification is the 784 to the 10 (the MNIST, the softmax the 10 the prob), and the loss is the MSE (the regression, the (pred - target)^2 the mean) or the cross_entropy (the classification, the log-softmax plus the NLL) - the next section's SGD is the optimizer, and the MNIST the first run.

The MNIST: The first run

# the MNIST (the):
#   the
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
import torchvision, torchvision.transforms as T

device = "cuda" if torch.cuda.is_available() else "cpu"
transform = T.Compose([T.ToTensor(),
    T.Normalize((0.1307,), (0.3081,))])
train = torchvision.datasets.MNIST("data", True,
    True, transform=transform)
loader = DataLoader(train, batch_size=64, shuffle=True)

model = nn.Linear(784, 10).to(device)   # the
opt = optim.SGD(model.parameters(), lr=0.01)  # the
lossf = nn.CrossEntropyLoss()           # the

for epoch in range(2):
    for x, y in loader:
        x, y = x.to(device), y.to(device)
        x = x.flatten(1)          # the the (N, 784)
        opt.zero_grad()           # the
        out = model(x)            # the
        loss = lossf(out, y)      # the
        loss.backward()           # the
        opt.step()                # the
    print(epoch, loss.item())
#   the
#   the
#   the

The first run is the Linear the MNIST: the transform the ToTensor plus the Normalize (the (32, 1, 28, 28) the (N, 784) the flatten), the Linear(784, 10) the model, the SGD the optimizer (the lr the 0.01), and the CrossEntropyLoss the loss (the log-softmax plus the NLL, the target the class the 0 to the 9). The loop is the five: the zero_grad, the forward (the flatten), the loss, the backward, the step - the epoch the batch the repeat, the loss the 0.3 to the 0.1 the 2 epoch, the 85% the accuracy (the Linear the no non-linear, the CNN the 99% the next chapter). The rule is the one: the Linear the 784 to the 10, the SGD the lr, the CrossEntropyLoss the loss, and the training-loop (the next chapter) the Module the run - the first-model the pattern, the CNN the scale.

The Working Rules

  • The Linear is the weight plus the bias, the forward the x @ w.T + b. The weight the (10, 784) (the Kaiming), the bias the (10,) (the 0), and the layer(x) the (32, 10) - the regression the 1 to the 1, the classification the 784 to the 10.
  • The MSE the regression, the cross_entropy the classification. The MSE the (pred - target)^2 the mean, and the CrossEntropyLoss the log-softmax plus the NLL (the target the class, the no the softmax the before).
  • The SGD is the optimizer, the lr the 0.01. The SGD(model.parameters(), lr) the weight the grad the -lr * grad, the step the update, and the Adam (the next chapter) the momentum the adaptive.
  • The training loop is the five, the epoch the repeat. The zero_grad, the forward, the loss, the backward, the step - the MNIST the Linear the 85%, the CNN the 99% the next chapter's.

Summary

The first model is the nn.Linear: the weight the (10, 784) (the Kaiming init), the bias the (10,) (the 0), and the forward the x @ weight.T + bias (the (32, 784) the (32, 10)) - the regression the 1 to the 1 (the MSE the loss), the classification the 784 to the 10 (the CrossEntropyLoss the log-softmax plus the NLL). The SGD is the optimizer (the lr the 0.01, the step the -lr * grad), and the MNIST first run is the Linear the transform (the ToTensor plus the Normalize), the DataLoader the batch, the flatten the (N, 784), and the five-step loop (the zero_grad, the forward, the loss, the backward, the step) - the 85% the accuracy (the no non-linear), the CNN the 99% the next chapter. The next chapter's training-loop is the Module the run the detail, and the optimizer the Adam the SGD's upgrade.