Skillversitas

Best Practices

The practice is the train the rule: the seed (the torch.manual_seed, the numpy, the CUDA, the reproducibility the 1 line), the determinism (the CUBLAS the env, the CUDA the 100% the no), the eval the no_grad (the predict the no the graph, the memory the save), the checkpoint the every epoch (the best the keep), the LR the find (the range the sweep, the loss the plot), and the overfit the 1 batch (the 1 epoch the 0% the loss the check the debug). This chapter covers the seed (the 4 the set, the determinism the env), the eval the no_grad (the inference the rule), the checkpoint (the every, the best), the LR the find (the sweep), and the overfit the 1 batch (the debug).

The seed and the determinism

# the SEED (the 4 set, the reproducibility):
import torch, numpy as np, random

seed = 42
random.seed(seed)          #   the Python's
np.random.seed(seed)       #   the NumPy's
torch.manual_seed(seed)    #   the CPU's
torch.cuda.manual_seed_all(seed)   # the
                              # CUDA's all

# the DETERMINISM (the env, the CUDA's
#   the 100% the no):
#   the 1: the CUBLAS (the 100% the yes,
#     the slow the 2x):
#   CUBLAS_WORKSPACE_CONFIG=:4096:8
#   the 2: the deterministic the algo:
torch.use_deterministic_algorithms(True)
#   the warn: the 1 op the no (the
#     the index_add's, the fallback the
#     the threads the 1)
torch.set_num_threads(1)
#   the: the CPU the 100%,
#     the CUDA the 90% (the non-determ
#     the op the log, the accept)

The seed is the 4 the set: the random (the Python's shuffle), the numpy (the data the aug), the torch.manual_seed (the CPU the init the op), and the cuda.manual_seed_all (the GPU the all) - the 1 line the reproducibility the init the order (the weight the same, the shuffle the same). The determinism is the env: the CUBLAS_WORKSPACE_CONFIG=:4096:8 (the CUDA the 100%, the slow the 2x), the use_deterministic_algorithms(True) (the op the non-determ the error, the index_add the fallback), and the set_num_threads(1) (the CPU the thread the no the race) - the CPU the 100%, the CUDA the 90% (the 1 op the no, the accept the log) - the reproducibility is the seed plus the version (the torch, the CUDA, the driver) plus the code the git (the 3 the record, the run the replay). The eval is the next section: the no_grad the inference the rule.

The eval, the no_grad, and the checkpoint

# the INFERENCE (the eval, the no_grad,
#   the memory the save):
model.eval()
with torch.no_grad():
    out = model(x_test)
    acc = (out.argmax(1) == y_test).float() \
        .mean()
#   the 1: the graph the no (the memory
#     the 0, the speed the 2x)
#   the 2: the BatchNorm the running
#     (the batch's stat the no)
#   the 3: the Dropout the off
#     (the predict the stable)

# the CHECKPOINT (the every epoch, the
#   best the keep, the resume):
for epoch in range(10):
    train()
    v = evaluate_val()
    torch.save({"epoch": epoch,
        "model": model.state_dict(),
        "opt": opt.state_dict(),
        "val": v}, "last.pt")   # the last
    if v > best:
        best = v
        torch.save({"epoch": epoch,
            "model": model.state_dict(),
            "opt": opt.state_dict(),
            "val": v}, "best.pt")   # the
                                       # best
#   the 2 file: the last (the crash the
#     resume) + the best (the final)

The eval plus the no_grad is the inference the rule: the model.eval() (the BatchNorm the running, the Dropout the off) and the torch.no_grad() (the graph the no, the memory the 0, the speed the 2x) - the predict the stable (the Dropout the no the mask), the argmax the acc, and the train mode the eval the no (the BatchNorm the batch the stat the update the eval the no). The checkpoint is the 2 the file: the last.pt (the every epoch, the crash the resume the 1 epoch the no the loss) and the best.pt (the val the improve the save, the final the deploy) - the dict (the epoch, the model, the opt, the val) the save (the optimizer the momentum the Adam's), and the LR is the next section: the find the sweep.

The LR find and the overfit check

# the LR FIND (the sweep, the loss the plot,
#   the 1 hour the right lr):
opt = optim.SGD(model.parameters(), lr=1e-7)
losses, lrs = [], []
gamma = 10.0 / 100   # the 1e-7 to the 1e1
for batch in train_loader:
    x, y = batch
    opt.zero_grad()
    loss = lossf(model(x), y)
    loss.backward()
    opt.step()
    lrs.append(opt.param_groups[0]["lr"])
    losses.append(loss.item())
    for g in opt.param_groups:
        g["lr"] *= gamma   #   the 10x per
                           #   the 100 batch
#   the plot: the log(lr) the x, the loss
#     the y (the 1 dip the lr's the right)
#   the the loss the 1e-7 the 1e-4 the 1e-2:
#     the 1e-4 the dip (the pick)

# the OVERFIT 1 BATCH (the debug, the 0 loss
#   the 1 epoch the check):
xb, yb = next(iter(train_loader))
for i in range(200):
    opt.zero_grad()
    loss = lossf(model(xb), yb)
    loss.backward()
    opt.step()
print(loss.item())   # the < 0.01 (the model
                     #   the 1 batch the
                     #   memorize, the debug
                     #   the pass)

The LR the find is the sweep: the 1e-7 start, the 10x per 100 batch (the gamma the 1e-4 the 1 epoch), the loss the log(lr) the plot (the 1 dip the lr the right, the diverge the high the cut) - the 1 hour the right the lr (the 1e-4 the dip the pick), and the CNN the 0.1 the SGD (the high), the transformer the 1e-4 the AdamW (the low). The overfit the 1 batch is the debug: the 200 step the 1 batch (the xb, the yb the fixed), the loss the 0.01 the below (the model the 1 batch the memorize, the loop the correct, the no the converge the data the check) - the debug the pass the no the overfit (the batch the no the shuffle, the 1 batch the same). The rule is the one: the seed the 4 the set, the eval the no_grad the inference, the checkpoint the 2 the file, the LR the find the 1 hour, and the overfit the 1 batch the debug - the practice the train the reliable, the reproducible, and the debug the fast.

The Working Rules

  • The seed is the 4, the determinism the env. The random plus the numpy plus the torch.manual_seed plus the cuda.manual_seed_all (the 1 line the reproducibility), and the CUBLAS_WORKSPACE_CONFIG plus the use_deterministic_algorithms plus the set_num_threads(1) (the CPU the 100%, the CUDA the 90%) - the version plus the git the record.
  • The eval plus the no_grad is the inference. The BatchNorm the running, the Dropout the off, and the graph the no (the memory the 0, the speed the 2x) - the predict the stable, the train mode the no.
  • The checkpoint is the 2, the last plus the best. The every epoch the last.pt (the crash the resume), and the val the improve the best.pt (the final the deploy) - the dict the epoch plus the model plus the opt (the momentum) plus the val.
  • The LR the find is the 1 hour, the overfit the 1 batch. The 1e-7 to the 1e1 the 10x per 100 (the loss the log(lr) the dip the pick), and the 200 step the 1 batch (the 0.01 the below the debug the pass) - the CNN the 0.1, the transformer the 1e-4.

Summary

The practice is the train the rule: the seed (the random plus the numpy plus the torch.manual_seed plus the cuda.manual_seed_all the 4 the set, the determinism the env the CUBLAS the CUDA the 90%), the eval plus the no_grad (the inference the rule, the BatchNorm the running, the graph the no), the checkpoint (the 2 the file, the last the crash the resume, the best the final, the dict the opt the momentum), the LR the find (the 1e-7 to the 1e1 the 10x per 100, the dip the pick, the 1 hour), and the overfit the 1 batch (the 200 step the 0.01 the debug the pass) - the train the reliable the reproducible, the debug the fast, and the PyTorch tutorial's end: the Tensor the autograd the nn.Module the loop the train, the CNN the RNN the transformer the model, the DDP the compile the export the deploy.