Skillversitas

Transfer Learning

The transfer is the pretrain weight the your data: the torchvision.models (the ResNet18, the EfficientNet, the ConvNeXt) the ImageNet's 1000 class the weight the download, the feature (the edge, the texture) the domain the agnostic (the dog the car the same the edge), and the fine-tune the head the your class (the 1000 to the 10). The freeze is the feature the no the update (the requires_grad=False), the unfreeze the all the low the lr (the 1e-5 the 1e-3 the head), and the adapter the LoRA the parameter-efficient (the LLM the modern). This chapter covers the pretrain (the torchvision.models, the weight), the freeze (the feature the head only), the unfreeze (the low the lr), and the LoRA (the adapter the few the param).

The pretrain and the freeze

# the PRETRAIN (the):
import torchvision.models as M

#   the
model = M.resnet18(weights=M.ResNet18_Weights.IMAGENET1K_V1)
#   the
#   the

# the FREEZE (the):
#   the
for p in model.parameters():
    p.requires_grad = False   # the
#   the

# the HEAD (the):
#   the
in_features = model.fc.in_features   # the 512
model.fc = nn.Linear(in_features, 10)   # the the
for p in model.fc.parameters():
    p.requires_grad = True   # the
#   the
#   the

The pretrain is the torchvision.models: the M.resnet18(weights=M.ResNet18_Weights.IMAGENET1K_V1) the ImageNet's 1000 class the weight (the download the auto, the feature the edge the texture the domain the agnostic) - the dog the car the same the edge, the 100 image the class the from the scratch the 10000 the need the no. The freeze is the feature the no the update: the for p in model.parameters(): p.requires_grad = False the backbone the frozen, the model.fc the head the 10 class (the 512 to the 10), and the for p in model.fc.parameters(): p.requires_grad = True the head the train - the SGD the head only (the param the 8K, the GPU the no the need), the 1 epoch the converge (the feature the good, the head the map) - the next section's unfreeze is the accuracy the push, and the LoRA the modern.

The unfreeze and the LoRA

# the UNFREEZE (the):
#   the
for p in model.parameters():
    p.requires_grad = True   # the
#   the
opt = optim.SGD([
    {"params": model.features.parameters(),
     "lr": 1e-5},        # the
    {"params": model.fc.parameters(),
     "lr": 1e-3},        # the
], momentum=0.9)
#   the
#   the

# the LoRA (the):
#   the
#   the
#   the
#   the
#   the

The unfreeze is the accuracy the push: the all the requires_grad=True, and the optimizer the group (the backbone the 1e-5 the low, the head the 1e-3 the high) - the feature the nudge (the your domain the adapt), the head the learn (the class the new) - the 10 epoch the 95% the 1% the gain. The LoRA is the parameter-efficient: the Linear the freeze (the W the no the update), the adapter the low-rank (the A the (d, r), the B the (r, d), the r=8 the param the 1%), and the output = Wx + BAx (the W the frozen, the BA the train) - the LLM the 7B the 1 GPU the fine-tune, the Peft library the apply. The rule is the one: the pretrain the weight, the freeze the head only (the few the data), the unfreeze the low the lr (the many the data), the LoRA the adapter (the LLM the big) - the next chapter's RNN is the sequence the model the CNN's sibling.

The Working Rules

  • The pretrain is the ImageNet's weight, the feature the agnostic. The M.resnet18(weights=IMAGENET1K_V1) the 1000 class (the edge, the texture), and the 100 image the class the scratch the 10000 the no - the domain the edge the same, the dog the car the shared.
  • The freeze is the head only, the few the data. The requires_grad=False the backbone, the model.fc the 10 class (the 512 to the 10), the True the head - the 8K the param, the 1 epoch the converge, the CPU the run.
  • The unfreeze is the low lr, the group the optimizer. The all the True, the backbone the 1e-5, the head the 1e-3 - the feature the nudge, the head the learn, the 10 epoch the 95% the push.
  • The LoRA is the adapter, the LLM's. The W the freeze, the BA the low-rank (the r=8 the 1% the param), the output = Wx + BAx - the 7B the 1 GPU, the Peft the apply, the LLM the modern's path.

Summary

The transfer is the pretrain weight the your data: the torchvision.models (the resnet18, the IMAGENET1K_V1 the 1000 class) the feature the edge the texture the domain the agnostic (the 100 image the class the scratch the 10000 the no). The freeze is the head only: the requires_grad=False the backbone, the model.fc the 10 class (the 512 to the 10), the True the head - the 8K the param, the 1 epoch the converge, the CPU the run. The unfreeze is the accuracy the push: the all the True, the group the lr (the backbone the 1e-5, the head the 1e-3), the 10 epoch the 95%. The LoRA is the adapter (the W the freeze, the BA the low-rank the r=8, the 1% the param, the 7B the 1 GPU) - the LLM the modern the path. The next chapter's RNN is the sequence the model: the GRU the LSTM the gate, the time the step the recur, the text the next the predict.