Skillversitas

Optimizers and Learning Rate Schedulers

The optimizer is the weight the update: the SGD (the momentum, the 0.9 the standard), the Adam (the momentum plus the adaptive, the 1e-3 the default), and the AdamW (the decouple the weight decay, the transformer the standard). The lr is the hyperparam the critical (the high the diverge, the low the stuck), the scheduler the lr the time (the warmup the linear, the cosine the decay), and the weight decay the regularize (the L2 the shrink). This chapter covers the SGD (the momentum, the nesterov), the Adam/AdamW (the adaptive, the decouple), the scheduler (the warmup, the cosine), and the lr the group (the backbone the low, the head the high).

The SGD and the AdamW

# the SGD (the):
import torch.optim as optim

opt = optim.SGD(model.parameters(), lr=0.1,
    momentum=0.9, weight_decay=1e-4)
#   the
#   the
#   the

# the ADAM (the):
#   the
opt = optim.Adam(model.parameters(), lr=1e-3)
#   the
#   the
#   the

# the ADAMW (the):
#   the
opt = optim.AdamW(model.parameters(), lr=1e-3,
    weight_decay=0.01)
#   the
#   the
#   the

The SGD is the baseline: the lr the 0.1 (the CNN the high), the momentum the 0.9 (the velocity the smooth, the local the escape), and the weight_decay the 1e-4 (the L2 the shrink, the overfit the guard) - the CNN the SGD the best (the generalize the Adam the less). The Adam is the adaptive: the momentum (the m the 1st) plus the RMSProp (the v the 2nd the per-param the lr), the 1e-3 the default (the SGD's 0.1 the 100x the low), and the transformer the Adam the from - the AdamW is the decouple: the Adam the L2 the adaptive the conflict (the decay the scale the param), and the AdamW the decay the no the gradient (the transformer the standard, the 1e-3 plus the 0.01 the decay) - the next section's scheduler is the lr the time, and the group the transfer's tool.

The scheduler and the group

# the SCHEDULER (the):
#   the
sched = optim.lr_scheduler.CosineAnnealingLR(
    opt, T_max=10)   # the
#   the
#   the

# the WARMUP (the):
#   the
class WarmupCosine:
    def __init__(self, opt, warmup=1000, max=100000):
        self.opt = opt
        self.warmup = warmup
        self.max = max
    def step(self, i):
        if i < self.warmup:
            lr = 1e-3 * i / self.warmup
        else:
            import math
            p = (i - self.warmup) / (self.max - self.warmup)
            lr = 1e-3 * 0.5 * (1 + math.cos(math.pi * p))
        for g in self.opt.param_groups:
            g["lr"] = lr
#   the

# the GROUP (the):
#   the
opt = optim.AdamW([
    {"params": model.backbone.parameters(),
     "lr": 1e-5},        # the
    {"params": model.head.parameters(),
     "lr": 1e-3},        # the
], weight_decay=0.01)
#   the

The scheduler is the lr the time: the StepLR (the gamma the step), the ReduceLROnPlateau (the val the stuck the halve), and the CosineAnnealingLR (the T_max the epoch, the 1e-3 to the 1e-6 the smooth) - the warmup is the transformer's required (the 1000 step the 1e-6 to the 1e-3 the linear, the early the stability the Adam's 2nd the estimate the bad), and the cosine the decay (the 1e-3 the 1e-6 the half the period). The group is the transfer's tool: the backbone the 1e-5 (the pretrain the keep), the head the 1e-3 (the new the learn), and the weight_decay the 0 the BatchNorm the bias (the decay the no the 1D) - the rule is the one: the CNN the SGD the 0.1, the transformer the AdamW the 1e-3 plus the warmup, the group the transfer, and the next chapter's data-pipelines is the augment the regularize the data's half.

The Working Rules

  • The SGD is the CNN's, the momentum the 0.9. The lr the 0.1 (the high), the momentum the velocity (the smooth, the escape), and the weight_decay the 1e-4 (the L2 the shrink, the overfit the guard) - the generalize the Adam the less, the CNN the SGD the best.
  • The Adam is the adaptive, the 1e-3 the default. The momentum plus the RMSProp (the per-param the lr), the 1e-3 the 100x the SGD's low, and the transformer the Adam the from - the 2nd the estimate the early the bad, the warmup the guard.
  • The AdamW is the decouple, the transformer's. The L2 the adaptive the conflict (the Adam), the decay the no the gradient (the AdamW), and the 1e-3 plus the 0.01 the decay (the standard, the LLM the same) - the weight decay the 0 the BatchNorm the bias.
  • The scheduler is the time, the group the transfer. The CosineAnnealingLR the T_max (the smooth), the warmup the 1000 step (the linear, the stability), and the group the backbone 1e-5 plus the head 1e-3 (the pretrain the keep, the new the learn) - the ReduceLROnPlateau the val the stuck.

Summary

The optimizer is the weight the update: the SGD (the 0.1 the CNN, the momentum the 0.9, the weight_decay the 1e-4), the Adam (the adaptive, the 1e-3 the default), and the AdamW (the decouple the decay, the transformer the standard) - the CNN the SGD the generalize, the transformer the AdamW the 1e-3 plus the 0.01. The scheduler is the lr the time: the CosineAnnealingLR the T_max (the smooth), the warmup the 1000 step (the linear, the stability, the 2nd the estimate the early the bad), and the ReduceLROnPlateau the val the stuck the halve. The group is the transfer's tool: the backbone the 1e-5, the head the 1e-3, the BatchNorm the bias the 0 the decay. The rule is the one: the CNN the SGD the 0.1, the transformer the AdamW the warmup the cosine, the group the transfer - the next chapter's data-pipelines is the augment the regularize the data's half, and the save-load the checkpoint the resume the train's tool.