Skillversitas

Distributed Training

The distributed is the multi-GPU the train: the DataParallel (the 1 process, the batch the split, the GIL the slow, the legacy) and the DDP (the DistributedDataParallel, the 1 process the GPU the peer, the gradient the all-reduce the sync, the fast the standard). The DDP is the torchrun the spawn (the rank the id, the world_size the count), the Sampler the split (the DistributedSampler the shard), and the FSDP the shard the param (the big the model the 1 the GPU the no). This chapter covers the DDP (the torchrun, the rank, the Sampler), the all-reduce (the gradient the sync), the effective batch (the N the GPU the multi), and the FSDP (the shard, the sharding_strategy).

The DDP: The torchrun and the rank

# the DDP (the 1 process the GPU, the peer):
import torch.distributed as dist
import torch.nn as nn

dist.init_process_group("nccl")   # the NCCL (the
                                   #   NVIDIA's, the)
model = CNN().to("cuda")
model = nn.parallel.DistributedDataParallel(
    model, device_ids=[dist.get_rank()])

rank = dist.get_rank()         # the 0..N-1 (the
                               #   process's id)
world = dist.get_world_size()  # the N (the GPU's)

# the SAMPLER (the shard per rank):
from torch.utils.data import DataLoader, \
    DistributedSampler
sampler = DistributedSampler(train,
    num_replicas=world, rank=rank,
    shuffle=True)
loader = DataLoader(train, batch_size=64,
    sampler=sampler)

The DDP is the one process per GPU: the torchrun --nproc_per_node=4 train.py spawns the 4 process (the rank the 0 to the 3, the world_size the 4), each holds the full model copy on its GPU, and the DistributedDataParallel wraps it (the device_ids the local GPU). The DistributedSampler is the split: the 60000 image the 4 shard (the 15000 each, the rank the its), so each process sees a disjoint slice - the effective batch is the 64 * 4 = 256 (the lr the scale: the 0.1 the 0.4 the linear rule). The init_process_group("nccl") is the backend (the NVIDIA's NCCL the fast, the gloo the CPU), and the torchrun sets the MASTER_ADDR/PORT the peer connect. The DataParallel (the legacy) is the one process the multi-GPU (the batch the split the main thread, the GIL the bottleneck) - the DDP the replace, the all-reduce the next section.

The all-reduce and the FSDP

# the ALL-REDUCE (the grad the sync, the
#   every backward the collect):
loss.backward()   # the hook: the all-reduce
                  #   the grad (the mean, the)
                  #   the 4 process's
opt.step()        # the 4 update the same

# the FSDP (the shard the param, the 4 GPU
#   the 1 model's run):
from torch.distributed.fsdp import (
    FullyShardedDataParallel as FSDP)
model = FSDP(
    Transformer(),
    auto_wrap_policy=size_based_auto_wrap_policy)
#   the param the shard (the 1/4 per GPU)
#   the forward the gather (the 1 layer)
#   the backward the reduce-scatter (the grad)

The all-reduce is the sync: the backward() triggers the hook, the gradient from the 4 process collects and averages (the NCCL the GPU-to-GPU the fast), and the opt.step() applies the same update on all - the 4 copy stays identical, the effective batch the 256 (the 1 step the 256 sample the gradient). The speedup is the 4x the ideal (the all-reduce the overhead, the NVLink the fast, the 4 GPU the 3.5x the realistic). The FSDP is the shard: the param splits across the 4 GPU (the 1/4 the memory each), the forward gathers the layer on demand (the 1 layer the 4 GPU the compute), and the backward the reduce-scatter (the grad the shard) - the 7B model the 4 GPU the run (the DDP the 1 GPU the full the no). The rule is the one: the DDP the torchrun plus the DistributedSampler plus the lr the scale (the N the GPU the multi), the FSDP the shard (the big the model), and the next chapter's torch.compile is the graph the speedup the single process.

The Working Rules

  • The DDP is the 1 process the GPU, the torchrun the spawn. The rank the 0..N-1, the world_size the N, the nccl the backend (the gloo the CPU), and the DistributedDataParallel the wrap (the device_ids the local) - the DataParallel the legacy (the GIL the slow).
  • The DistributedSampler is the shard, the effective batch the N the multi. The 60000 the 4 slice (the 15000 each), the 64 the 4 the 256, and the lr the linear scale (the 0.1 the 0.4) - the shuffle the sampler the set, the epoch the set_epoch the seed.
  • The all-reduce is the sync, the 3.5x the realistic. The backward the collect (the mean the 4 process), the step the same (the 4 copy the identical), and the NVLink the overhead the cut (the PCIe the slow) - the 1 step the 256 sample.
  • The FSDP is the shard, the big the model. The param the 1/4 per GPU (the memory the cut), the forward the gather (the 1 layer), the backward the reduce-scatter - the 7B the 4 GPU (the DDP the 1 GPU the no), the sharding_strategy the OFFLOAD the CPU.

Summary

The distributed is the multi-GPU: the DDP (the DistributedDataParallel) is the standard - the torchrun --nproc_per_node=4 the 4 process (the rank the 0..3, the world_size the 4), each the full copy on its GPU, the DistributedSampler the shard (the 15000 each), and the all-reduce the backward the gradient the mean the sync (the 4 copy the identical, the effective batch the 256, the lr the linear scale). The FSDP is the shard (the param the 1/4 per GPU, the forward the gather, the backward the reduce-scatter - the 7B the 4 GPU the run). The DataParallel is the legacy (the GIL the bottleneck). The next chapter's torch.compile is the single-process speedup: the graph the capture, the kernel the fuse, the 1.2x to the 2x the eager's replace.