Skillversitas

The Training Loop

The training loop is the five-step cycle: the opt.zero_grad() (the grad the reset), the out = model(x) (the forward, the pred), the loss = lossf(out, y) (the scalar), the loss.backward() (the grad the compute), and the opt.step() (the weight the update) - the batch the inner, the epoch the outer, the DataLoader the batch the shuffle. The eval is the no_grad plus the model.eval() (the accuracy the test), the AMP the autocast (the float16 the speedup), and the grad clip the stability (the clip_grad_norm_). This chapter covers the five-step loop (the train, the eval), the AMP (the autocast, the GradScaler), the grad clip (the norm), and the best the save (the epoch's argmax).

The Five-Step Loop

# the TRAIN (the):
def train(epoch):
    model.train()          # the
    total, correct = 0, 0
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        opt.zero_grad()      # the 1
        out = model(x)       # the 2
        loss = lossf(out, y) # the 3
        loss.backward()      # the 4
        opt.step()           # the 5
        total += y.size(0)
        correct += (out.argmax(1) == y).sum().item()
    return loss.item(), correct / total

# the EVAL (the):
def evaluate():
    model.eval()             # the
    correct = 0
    with torch.no_grad():    # the
        for x, y in test_loader:
            x, y = x.to(device), y.to(device)
            out = model(x)
            correct += (out.argmax(1) == y).sum().item()
    return correct / len(test_loader.dataset)

# the RUN (the):
best = 0
for epoch in range(5):
    loss, acc = train(epoch)
    test_acc = evaluate()
    print(epoch, f"{loss:.3f}", f"{acc:.3f}",
            f"{test_acc:.3f}")
    if test_acc > best:      # the
        best = test_acc
        torch.save(model.state_dict(), "best.pt")

The train is the five-step loop: the model.train() (the BatchNorm the running, the Dropout the on), the zero_grad (the 1, the grad the reset), the forward (the 2, the pred), the loss (the 3, the scalar), the backward (the 4, the grad), and the step (the 5, the update) - the accuracy the argmax the count. The eval is the no_grad plus the model.eval() (the no graph, the BatchNorm the running, the Dropout the off) - the test's accuracy, the train's the loss. The best is the save: the test_acc > best the state_dict the file (the best.pt the epoch's argmax, the overfit the later the no the keep) - the AMP (the next section) is the speedup, and the grad clip the stability.

The AMP and the grad clip

# the AMP (the):
#   the
scaler = torch.amp.GradScaler("cuda")
for x, y in train_loader:
    x, y = x.to(device), y.to(device)
    opt.zero_grad()
    with torch.amp.autocast("cuda"):  # the
        out = model(x)                #   the the
        loss = lossf(out, y)          #   the the
    scaler.scale(loss).backward()     # the
    scaler.step(opt)                  # the
    scaler.update()                   # the
#   the
#   the

# the GRAD CLIP (the):
#   the
loss.backward()
torch.nn.utils.clip_grad_norm_(
    model.parameters(), 1.0)   # the
opt.step()
#   the
#   the

The AMP is the speedup: the torch.amp.autocast("cuda") the float16 (the compute the 2x, the memory the half), the GradScaler the underflow's guard (the scale the loss, the backward the scaled, the step the unscale plus the skip the inf, the update the scale the adapt) - the CUDA the only (the MPS the no the AMP). The grad clip is the stability: the clip_grad_norm_(model.parameters(), 1.0) the grad the norm the 1.0 (the explode the RNN's, the transformer's the standard) - the backward plus the clip plus the step, the RNN chapter the required. The rule is the one: the AMP the CUDA's speedup (the autocast plus the scaler), the grad clip the RNN's stability (the norm the 1.0), and the best the save (the test_acc the argmax) - the next chapter's Datasets and the DataLoader is the data's half the loop runs on.

The Working Rules

  • The five-step is the loop, the epoch the outer. The zero_grad (the 1), the forward (the 2), the loss (the 3), the backward (the 4), the step (the 5) - the batch the inner, the DataLoader the shuffle, and the model.train() the mode's start.
  • The eval is the no_grad plus the model.eval(). The no graph (the memory's), the BatchNorm the running, the Dropout the off - the test's accuracy, and the argmax the pred the count.
  • The AMP is the CUDA's, the scaler the underflow. The autocast the float16 (the 2x), the GradScaler the scale/step/update (the inf the skip) - the MPS the no AMP, the CPU the no AMP.
  • The grad clip is the RNN's, the best the save. The clip_grad_norm_(parameters, 1.0) the explode's guard (the RNN, the transformer), and the test_acc > best the state_dict (the best.pt the overfit the no).

Summary

The training loop is the five-step cycle: the opt.zero_grad() (the grad the reset), the out = model(x) (the forward), the loss = lossf(out, y) (the scalar), the loss.backward() (the grad the compute), and the opt.step() (the update) - the batch the inner, the epoch the outer, the model.train() the mode. The eval is the no_grad plus the model.eval() (the no graph, the accuracy the test), the AMP is the CUDA's speedup (the autocast the float16, the GradScaler the underflow the scale/step/update), and the grad clip is the RNN's stability (the clip_grad_norm_ the norm the 1.0). The best is the save (the test_acc > best the state_dict, the best.pt the overfit the no). The next chapter's Datasets and the DataLoader is the data's half: the Dataset the __getitem__, the DataLoader the batch plus the shuffle plus the worker, and the transform the pipeline.