Training frameworks optimize for flexibility and backward passes. Production inference wants maximum tokens or images per second at a quality bar, on a specific GPU, often in FP16 or INT8. NVIDIA TensorRT is a compiler and runtime for that job: ingest ONNX (or framework graphs), apply layer fusion, kernel auto-tuning, and precision calibration, then emit an engine bound to a GPU architecture. The same ResNet or Transformer that trained in PyTorch can run several times faster once it is an engine — if you accept the workflow of building engines per SKU and validating accuracy after quantization.
Fusion, tactics, and why the builder takes minutes
TensorRT's builder searches a space of "tactics" (kernel implementations) for each layer given the target GPU, batch size, and precision. That search is why engine build is slow and why you persist engines as artifacts in CI rather than building on every box at boot. Fusion reduces memory round trips: a conv-bias-relu becomes one kernel. Transformers add attention fusions and later sparse or FP8 paths on new architectures. Dynamic shapes complicate the search; you give optimization profiles (min/opt/max) so the engine can handle variable sequence lengths without a rebuild per request.
INT8 calibration uses a representative dataset to pick scale factors. A bad calibration set (wrong domain) produces a fast, wrong model. Teams must measure the product metric — not only top-1 ImageNet — on the engine, not on the PyTorch fp32 graph. Mixed precision (FP16) is easier but still not bit-exact. Some layers stay FP32 because they are numerically brittle.
Deployment: Triton, plugins, and the ONNX gap
Unsupported operators fall out to plugins or force a fallback. Real graphs always have one custom op. TensorRT's plugin interface is how NVIDIA and vendors extend the compiler; it is also how engines become non-portable C++. NVIDIA Triton Inference Server commonly hosts TensorRT engines with batching and multi-model scheduling. The operational loop is: export ONNX, simplify, build engine in CI for each GPU generation you run, canary accuracy, then serve.
Version skew is the outage. An engine built for SM 8.0 will not run on a different major architecture. A TensorRT major upgrade rebuilds everything. Pin the builder in the same image that serves, or at least in the same release train. Dynamic batching in the server can invalidate assumptions you tuned at batch=8 in the builder; include the real batch distribution in optimization profiles.
The steal if you never touch TensorRT is the split: train in a flexible graph compiler, deploy with a specialized one that knows the chip. ONNX is the handshake. Accuracy tests after lowering are mandatory.
Batching policy belongs next to the engine. A static engine built for batch 32 will underperform or fail on batch 1 chat traffic. Either build multiple engines or use profiles that cover the live histogram. Throughput-oriented vision APIs and latency-oriented token APIs should not share one optimization profile "because they are both TensorRT."
What you can borrow
- Compile inference graphs for the target GPU and batch profile; do not serve training-mode graphs unchanged.
- Persist engines as CI artifacts; tactic search does not belong in the request path.
- Validate product accuracy after FP16/INT8 lowering with a domain-correct calibration set.
- Pin TensorRT and GPU architecture per fleet slice; engines are not portable across generations.
- Budget custom plugins for the one op ONNX will not express, and test it as hard as the rest of the net.