Exporting to ONNX and Deployment

The export is the ONNX (the Open Neural Network Exchange, the format the no-PyTorch): the torch.onnx.export the trace (the 1 forward the graph the record), the opset the version (the 17 the modern, the Runtime the support), and the dynamic_axes the batch (the 1 to the 32 the 1 file). The Runtime is the onnxruntime (the CPU the fast the no the GPU), the TorchServe the serve (the REST the endpoint, the batch the dynamic), and the quantize the int8 (the 4x the small, the mobile the edge). This chapter covers the export (the trace, the opset), the dynamic_axes (the batch), the onnxruntime (the load, the run), and the TorchServe (the serve, the REST).

The export: The trace and the opset

# the EXPORT (the 1 forward the record):
import torch

dummy = torch.randn(1, 1, 28, 28)   # the trace
                                      #   the shape
torch.onnx.export(
    model,               # the eval the
                        #   the mode first
    dummy,               # the input (the
                        #   the shape's the)
    "model.onnx",        #   the file
    opset_version=17,    #   the opset (the
                        #     the 17 the
                        #     the modern)
    input_names=["image"],
    output_names=["logits"],
    dynamic_axes={       # the batch the
        "image": {0: "N"},   #   the dim
        "logits": {0: "N"}  #     the 0
    },
)
#   the 1 forward the run (the trace)
#   the graph the save (the .onnx)

The torch.onnx.export is the trace: the 1 forward with the dummy input (the (1, 1, 28, 28)), the op the record (the ONNX graph the save), and the model.eval() the before (the Dropout the off, the BatchNorm the running - the train mode the export the no). The opset_version=17 is the op's version (the 17 the modern the Runtime the support, the 9 the old the mobile the yes), the input_names/output_names the label (the image, the logits the Runtime the read), and the dynamic_axes is the batch: the {"image": {0: "N"}} the dim 0 the var (the 1 to the 32 the 1 file, the static the 1 the no). The trace is the if the data the static (the shape the dummy the freeze), and the torch.jit (the script) is the alternative (the Python the compile, the TorchScript the runtime) - the next section's onnxruntime is the load the run the no-PyTorch.

The onnxruntime and the TorchServe

# the ONNXRUNTIME (the load, the run, the
#   no-PyTorch, the CPU the fast):
import onnxruntime as rt

sess = rt.InferenceSession("model.onnx")
x = np.random.randn(4, 1, 28, 28).astype(
    np.float32)   # the batch the 4 (the
                  #   dynamic's the N)
out = sess.run(None, {"image": x})
print(out[0].shape)   # the (4, 10) (the
                      #   the logits)
#   the CPU the 2x the eager (the no
#     Python's loop, the kernel the
#     optimized)

# the TORCHSERVE (the REST, the production):
#   the 1: the model the stage
ts_model = Model(model_name="digit",
    model_file="model.py")
ts_model.serialize_model("model.mar")
#   the 2: the serve
torchserve --start --model-store ./store \
    --models digit=model.mar
#   the 3: the call (the batch, the
#     the image the base64)
curl -T image.jpg http://localhost:8080/ \
    predictions/digit

The onnxruntime is the no-PyTorch: the InferenceSession the load (the .onnx the 1 file), the sess.run(None, {"image": x}) the run (the 4 batch the N the dynamic), and the CPU the 2x the eager (the Python loop the no, the kernel the optimize the AVX512) - the no the GPU the deploy, the latency the low, the edge the mobile the run. The TorchServe is the production: the Model the stage (the model.py the handle the pre/post the process, the .mar the archive), the torchserve --start the serve (the port the 8080), and the curl -T image.jpg the REST (the base64 the image, the batch the dynamic the auto, the GPU the multi the worker) - the scale the worker the add, the A/B the 2 version the serve. The quantize is the int8 (the 4x the small, the onnxruntime the quant, the mobile the edge the yes, the accuracy the 0.5% the drop) - the next chapter's TorchHub is the model's load the ecosystem the HuggingFace the timm.

The Working Rules

The torch.onnx.export is the trace, the eval the before. The 1 forward the dummy (the shape the freeze), the opset_version=17 the modern (the 9 the mobile), and the input_names/output_names the label - the train mode the no (the Dropout the on, the export the wrong).
The dynamic_axes is the batch, the 1 file the N the var. The {"image": {0: "N"}} the dim 0 the var (the 1 to the 32), and the static the 1 the no (the 1 file the batch the 1) - the Runtime the run the 4 the no-reexport.
The onnxruntime is the no-PyTorch, the CPU the 2x. The InferenceSession the load (the 1 file), the sess.run the batch the N, and the CPU the fast (the no Python loop, the kernel the AVX512) - the edge the mobile the run, the no GPU the deploy.
The TorchServe is the REST, the worker the scale. The Model the stage (the handle the pre/post, the .mar), the torchserve --start the 8080, and the curl the image (the batch the dynamic, the A/B the 2 version) - the quantize the int8 the 4x the small (the edge, the 0.5% the drop).

Summary

The export is the ONNX: the torch.onnx.export the 1 forward the dummy the trace (the op the record, the model.eval() the before), the opset_version=17 the modern (the 9 the mobile), and the dynamic_axes the batch (the {"image": {0: "N"}} the 1 file the N). The onnxruntime is the no-PyTorch (the InferenceSession the load, the sess.run the 4 batch, the CPU the 2x the edge the mobile), and the TorchServe is the production (the Model the .mar, the torchserve the REST the 8080, the worker the scale, the A/B the 2 version) - the quantize the int8 the 4x the small the edge. The next chapter's TorchHub is the ecosystem: the torch.hub.load the pretrain the 1 line, the HuggingFace the transformer the library, the timm the CNN the 500, and the Lightning the train the loop the framework.