Skillversitas

Regularization and Generalization

The overfit is the train the good the val the bad (the gap the signal): the Dropout the neuron the mask (the 0.25 the CNN, the 0.1 the transformer), the weight decay the L2 (the shrink, the 1e-4 the 1e-2), the early stop the val the stuck (the patience the 5), and the data the augment (the best, the 10x the data the proxy). The generalize is the val the test the close (the gap the small), and the batch size the regularize (the small the noise the good). This chapter covers the Dropout (the neuron, the rate), the weight decay (the L2, the AdamW), the early stop (the patience), and the data (the augment, the batch).

The Dropout and the weight decay

# the DROPOUT (the):
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 512),
    nn.ReLU(),
    nn.Dropout(0.5),      # the
    nn.Linear(512, 10),
)
#   the
#   the
#   the

# the WEIGHT DECAY (the):
#   the
import torch.optim as optim
opt = optim.AdamW(model.parameters(),
    lr=1e-3, weight_decay=1e-2)
#   the
#   the
#   the

The Dropout is the neuron the mask: the 0.5 the MLP (the 50% the off the train), the 0.25 the CNN (the less, the feature the shared), the 0.1 the transformer (the minimal, the pretrain the big), and the eval the no the mask (the scale the train the compensate) - the ensemble the 2^N the model the proxy (the neuron the co-adapt the no). The weight decay is the L2: the 1e-4 the CNN (the mild), the 1e-2 the transformer (the strong, the AdamW the decouple), and the BatchNorm the bias the 0 the decay (the 1D the no the shrink) - the L1 the sparse (the 0 the force, the feature the select) the less the use, and the next section's early stop is the val the stuck the stop.

The early stop and the data

# the EARLY STOP (the):
#   the
best, patience, wait = val_acc, 5, 0
for epoch in range(100):
    train()
    v = evaluate_val()
    if v > best:
        best, wait = v, 0
        torch.save(model.state_dict(), "best.pt")
    else:
        wait += 1
        if wait >= patience:   # the
            break              #   the
#   the
#   the

# the DATA (the):
#   the
#   the
#   the
#   the
#   the

The early stop is the val the stuck the stop: the patience the 5 (the epoch the no the improve), the best the save (the state_dict the keep), and the break the train the end (the overfit the the the guard, the compute the save) - the val the test the no the peek (the 1 the final, the tune the val the only). The data is the best: the augment the 10x the proxy (the flip the crop the color, the RandAugment the auto), the batch the small the noise (the 32 the 64 the regularize, the large the fast the less), and the 10 the more the data the best the regularize (the 100x the data the 1% the error the cut) - the model the small the data the few (the capacity the match), the model the big the data the many (the scale the law), and the rule is the one: the Dropout the neuron, the weight decay the L2, the early stop the val, the data the best - the next chapter's distributed is the GPU the multi the train the fast the DDP's path.

The Working Rules

  • The Dropout is the neuron, the rate the task. The 0.5 the MLP, the 0.25 the CNN, the 0.1 the transformer (the minimal), and the eval the no the mask (the scale the compensate) - the ensemble the 2^N the proxy, the co-adapt the no.
  • The weight decay is the L2, the AdamW the decouple. The 1e-4 the CNN (the mild), the 1e-2 the transformer (the strong), and the BatchNorm the bias the 0 the decay (the 1D the no the shrink) - the L1 the sparse the less the use, the feature the select.
  • The early stop is the val, the patience the 5. The no the improve the wait, the best the save (the state_dict), and the break the end (the overfit the guard, the compute the save) - the val the test the no the peek (the 1 the final, the tune the val the only).
  • The data is the best, the batch the small. The augment the 10x the proxy, the batch the 32 the noise (the regularize), and the 10 the more the data the best (the 100x the 1% the cut) - the model the small the data the few, the model the big the data the many (the scale the law).

Summary

The overfit is the train the good the val the bad (the gap the signal), and the regularize is the gap the close: the Dropout the neuron the mask (the 0.5 the MLP, the 0.25 the CNN, the 0.1 the transformer, the eval the no the mask), the weight decay the L2 (the 1e-4 the CNN, the 1e-2 the transformer, the AdamW the decouple, the BatchNorm the 0), and the early stop the val the stuck (the patience the 5, the best the save). The data is the best: the augment the 10x the proxy, the batch the small the noise (the 32 the regularize), and the 10 the more the data the best (the 100x the 1% the cut) - the model the small the data the few, the big the many (the scale the law). The next chapter's distributed is the GPU the multi: the DDP the process the per-GPU, the data parallel the batch the split, and the FSDP the shard the big the model the 4 the GPU the run.