Skillversitas

Compiling with torch.compile

The torch.compile is the 2.0+'s speedup: the model = torch.compile(model) the one line, the first call the Dynamo the trace (the graph the capture, the FX the intermediate), the backend the optimize (the inductor the default, the kernel the CUDA the generate), and the graph break the stop (the Python the op the no trace, the eager's fall back). The mode is the speed/reduce-overhead/max-autotune (the tradeoff the time), and the fullgraph=True the break's error (the fix's force). This chapter covers the compile (the Dynamo, the inductor), the mode (the 3 the tradeoff), the graph break (the no the op, the fullgraph), and the 1.2x to the 2x the realistic.

The Dynamo and the inductor

# the COMPILE (the 1 line, the Dynamo's):
model = CNN().to(device)
model = torch.compile(model)
#   the 1 call: the Dynamo the trace
#     (the FX graph the capture, the slow)
#   the 2+: the inductor the fast
#     (the kernel the CUDA the generate)

# the MODE (the 3, the tradeoff the time):
m = torch.compile(model, mode="default")
#   the speed the time the balance
m = torch.compile(model, mode="reduce-overhead")
#   the CUDA graph the capture (the overhead
#     the cut, the 1.2x the 1.5x)
m = torch.compile(model, mode="max-autotune")
#   the kernel the 50 the tune (the compile
#     the long, the 1.5x the 2x)

# the FULLGRAPH (the break's error, the fix):
m = torch.compile(model, fullgraph=True)
#   the 1 break the RuntimeError (the
#     the no-silent, the trace's complete)

The torch.compile is the two: the Dynamo (the 1 call, the bytecode the trace, the FX graph the intermediate the Python the op the no trace the graph break) and the inductor (the FX the optimize, the CUDA kernel the generate the fuse the 4 op the 1 kernel, the memory the bandwidth the save). The 1 call is the slow (the 10s to the min the compile), the 2+ is the fast (the 1.2x the CNN, the 2x the transformer the realistic). The mode is the 3: the default (the balance), the reduce-overhead (the CUDA graph the kernel launch the cut, the 1.5x), and the max-autotune (the 50 the kernel the tune, the compile the long the nightly the yes, the 2x). The backend is the inductor (the default), the aot_autograd (the inference's), the eager (the debug's, the no-optimize) - the next section's graph break is the no-optimize's source.

The graph break and the realistic

# the BREAK (the no-trace, the eager's fall):
torch._dynamo.config.verbose = True
model = torch.compile(model)
out = model(x)
#   the log: the "graph break" the 3
#     - the if the data-dependent (the
#       the shape the no-compile)
#     - the print the mid-forward (the
#       the Python's op)
#     - the .item() the sync (the CPU's)

# the FIX (the no-break, the fullgraph's):
def forward(self, x):
    x = self.features(x)
    x = x.flatten(1)
    return self.head(x)   # the no .item(),
                          #   the no print,
                          #   the no data-dep if
m = torch.compile(model, fullgraph=True)

The graph break is the no-optimize: the Dynamo hits the op the no-trace (the data-dependent if the shape the no-compile, the print the Python's, the .item() the CPU's sync), falls back to the eager for that segment, and the 2 graph the 1 slow (the break the overhead the host the GPU the transfer). The torch._dynamo.config.verbose = True is the log (the 3 break's location), and the fullgraph=True is the error (the RuntimeError the 1 break, the fix's force - the if the static the branch, the .item() the no the loop, the print the tensorboard). The realistic is the 1.2x to the 2x: the CNN the 1.2x (the kernel the already the fast), the transformer the 2x (the small op the fuse the bandwidth the save), the big batch the 1.1x (the compute the bound) - the compile the once the slow, the run the fast, and the rule is the one: the default the start, the reduce-overhead the kernel the launch the bound, the max-autotune the nightly, the fullgraph the break the fix - the next chapter's ONNX is the export the deploy the runtime the no-PyTorch.

The Working Rules

  • The torch.compile is the 1 line, the Dynamo the trace. The 1 call the FX graph (the 10s to the min), the 2+ the inductor the CUDA kernel (the 4 op the 1 the fuse), and the eager backend the debug's (the no-optimize, the step-through).
  • The mode is the 3, the tradeoff the time. The default the balance (the 1.2x), the reduce-overhead the CUDA graph (the launch the cut, the 1.5x), and the max-autotune the 50 the tune (the compile the long, the 2x, the nightly the yes).
  • The graph break is the no-optimize, the fullgraph the fix. The verbose=True the log (the location), the data-dependent if the print the .item() the source, and the fullgraph=True the RuntimeError (the 1 break the error, the silent the no) - the 2 graph the 1 slow.
  • The 1.2x to the 2x is the realistic. The CNN the 1.2x (the kernel the already), the transformer the 2x (the fuse the bandwidth), the big batch the 1.1x (the compute the bound) - the once the slow, the run the fast.

Summary

The torch.compile is the 2.0+'s speedup: the 1 line (the model = torch.compile(model)), the Dynamo the 1 call the FX graph (the bytecode the trace, the graph break the no the op), and the inductor the CUDA kernel (the fuse the 4 op the 1, the bandwidth the save). The mode is the 3 (the default the balance, the reduce-overhead the CUDA graph the launch the cut, the max-autotune the 50 the tune the nightly), the fullgraph=True the break the error (the verbose the log, the fix's force), and the realistic is the 1.2x the CNN to the 2x the transformer. The next chapter's ONNX is the export: the torch.onnx.export the opset the graph, the Runtime the no-PyTorch, and the TorchServe the serve the production the REST the endpoint.