Accelerating edge AI models: ONNX, TensorRT and INT8 quantization

Model acceleration means running a trained model on the target hardware with lower latency and higher throughput while keeping its accuracy within measurable limits. At the edge this usually means the PyTorch → ONNX → TensorRT or ONNX Runtime chain, plus lower numeric precision (FP16, INT8).

I explained why video is processed on site in edge or cloud; the price is fitting as many camera streams as you have onto a limited GPU. The budget calculation in real-time detection with Python and YOLO (“12 cameras × 8 frames = 96 inferences per second”) is the starting point here: how to make a model fit a budget it currently misses.

The chain: from PyTorch to hardware

Stage	Tool	Output
Training	PyTorch / Ultralytics	.pt weights
Export	torch.onnx.export, Ultralytics export	.onnx graph
Quantization (optional)	NVIDIA ModelOpt, ONNX Runtime quantization	.onnx with Q/DQ nodes
Build	trtexec, ONNX Runtime TensorRT EP	Hardware-specific TensorRT engine
Run	TensorRT, ONNX Runtime	Boxes, scores

ONNX is the shared contract: the same file can run on TensorRT, OpenVINO or ONNX Runtime. A built TensorRT engine is the opposite; it belongs only to the environment it was built in.

First, versions: TensorRT 11 and strong typing

At the time of writing NVIDIA’s current release is TensorRT 11.3.0, and the 11.x series brought two changes that directly affect edge work.

First, networks are now always strongly typed. According to NVIDIA’s trtexec migration guide, the --fp16, --int8, --best and --calib flags have been removed and trtexec exits with an error if you pass them. For FP16 the model is first converted to mixed precision with ModelOpt AutoCast; for INT8 it is quantized ahead of time with ModelOpt. Implicit quantization and the IInt8Calibrator interface are gone.

Second, NVIDIA’s Jetson migration page states that TensorRT 11.3.0 does not support JetPack and that Jetson deployments must stay on a TensorRT 10.x release supported by their JetPack version.

	TensorRT 10.x (e.g. Jetson/JetPack)	TensorRT 11.x (x86 server GPUs)
FP16	trtexec --fp16	Model converted to FP16 with ModelOpt AutoCast
INT8	--int8 --calib=<cache> or a Q/DQ model	Q/DQ model only (ModelOpt)
Ultralytics quantize=8	Uses the legacy calibrator	Installs and uses ModelOpt automatically

The practical upshot: if servers and Jetsons share a project, do the quantization in the model. Explicit quantization (ONNX with Q/DQ nodes) is supported in TensorRT 10.x as well, so both sides can build engines from the same ONNX file. Check the build flags against the documentation of the TensorRT version on your device.

Step 1: Export from PyTorch to ONNX

For an Ultralytics model one line is enough. On YOLO26, nms=False selects the NMS-free end-to-end head and the output becomes (N, 300, 6) rows of [x1, y1, x2, y2, score, class], so you do not have to write NMS in post-processing. By default (no nms argument) you get the raw (N, 4+classes, 8400) output and NMS is your job. In Ultralytics’ published COCO results the default head gives slightly higher mAP, so this is a speed–accuracy trade-off to measure on your own data.

For a generic PyTorch model you use torch.onnx.export. Since PyTorch 2.9, dynamo=True is the default; dynamic dimensions are now given with dynamic_shapes rather than dynamic_axes (which is being deprecated).

"""PyTorch -> ONNX: (A) an Ultralytics YOLO model, (B) any torch.nn.Module."""
import onnx
import torch
from ultralytics import YOLO

# (A) Ultralytics: dynamic axes + YOLO26's NMS-free end-to-end head
YOLO("best.pt").export(format="onnx", imgsz=640, dynamic=True, nms=False, simplify=True)
# -> best.onnx; input "images" (batch, 3, H, W), output (batch, 300, 6)


# (B) Generic PyTorch model: dynamo=True has been the default since PyTorch 2.9
class TinyNet(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = torch.nn.Conv2d(3, 16, 3, padding=1)
        self.head = torch.nn.Linear(16, 4)

    def forward(self, x):
        return self.head(self.conv(x).mean(dim=(2, 3)))


model = TinyNet().eval()
torch.onnx.export(
    model,
    (torch.randn(1, 3, 640, 640),),
    "tinynet.onnx",
    input_names=["images"],
    output_names=["logits"],
    dynamic_shapes=({0: "batch"},),  # only the batch is dynamic; H/W stay fixed
    opset_version=18,  # pick an opset your target runtime supports
    external_data=False,  # small model: keep weights in one file, not a separate .data file
)

m = onnx.load("tinynet.onnx")
onnx.checker.check_model(m)
for t in m.graph.input:
    dims = [d.dim_param or d.dim_value for d in t.type.tensor_type.shape.dim]
    print("input", t.name, dims)  # e.g. input images ['batch', 3, 640, 640]

Three decisions:

• Dynamic axes or fixed size? A dynamic batch adds flexibility, but in TensorRT every dynamic dimension needs a min/opt/max range (an optimization profile). If cameras are always fed at 640×640, keeping height and width fixed simplifies both the build and the measurements.
• Opset. Choose opset_version according to what the target runtime supports; the newest is not always the best. ModelOpt INT8 quantization needs opset 19 or higher and upgrades lower versions itself.
• External data. external_data is on by default and writes the weights to a separate .data file. For small edge models, setting it to False and shipping a single file makes deployment easier.

Step 2: Run with ONNX Runtime and pick execution providers

ONNX Runtime runs the graph through hardware-specific “execution providers” (EPs). The list order is the priority: if the first EP does not support a node, it falls back to the next. The common ones:

Execution provider	Hardware	Note
TensorrtExecutionProvider	NVIDIA GPU	Compiles subgraphs into TensorRT engines; the engine cache is a must
CUDAExecutionProvider	NVIDIA GPU	Fallback for nodes TensorRT does not support
OpenVINOExecutionProvider	Intel CPU/GPU/NPU	Intel-based edge boxes
CoreMLExecutionProvider	Apple	Trying things on a development machine
CPUExecutionProvider	Everywhere	The default, always last in the list

The ONNX Runtime documentation recommends registering the CUDA EP alongside the TensorRT EP so that nodes TensorRT does not support run on CUDA. Because the TensorRT EP builds the engine on first start, creating a session can take minutes; trt_engine_cache_enable writes the engine to disk and brings later starts down to seconds. The compatibility table in the documentation pairs the TensorRT EP with TensorRT 10.x; check the version there before installing.

"""YOLO inference with ONNX Runtime: letterbox preprocessing, execution provider selection, postprocessing."""
from __future__ import annotations

import cv2
import numpy as np
import onnxruntime as ort


def letterbox(img: np.ndarray, size: int = 640, color=(114, 114, 114)):
    """Fits the image into a size x size canvas keeping the aspect ratio; returns scale and offset for undoing it."""
    h, w = img.shape[:2]
    r = min(size / h, size / w)
    nh, nw = round(h * r), round(w * r)
    resized = cv2.resize(img, (nw, nh), interpolation=cv2.INTER_LINEAR)
    top, left = (size - nh) // 2, (size - nw) // 2
    canvas = np.full((size, size, 3), color, dtype=np.uint8)
    canvas[top:top + nh, left:left + nw] = resized
    return canvas, r, (left, top)


def preprocess(frames_bgr: list[np.ndarray], size: int = 640):
    """BGR frames -> (N, 3, size, size) float32, 0-1 range, RGB. Must match the training preprocessing."""
    batch, metas = [], []
    for f in frames_bgr:
        img, r, pad = letterbox(f, size)
        batch.append(img[:, :, ::-1].transpose(2, 0, 1))  # BGR->RGB, HWC->CHW
        metas.append((r, pad))
    x = np.ascontiguousarray(np.stack(batch), dtype=np.float32) / 255.0
    return x, metas


def make_session(model_path: str, use_trt: bool = True, cache_dir: str = "./trt_cache"):
    """Priority: TensorRT EP -> CUDA EP -> CPU. EPs that are not installed are left out."""
    available = ort.get_available_providers()
    providers: list = []
    if use_trt and "TensorrtExecutionProvider" in available:
        providers.append(("TensorrtExecutionProvider", {
            "trt_fp16_enable": True,
            "trt_engine_cache_enable": True,  # write the engine to disk: next start takes seconds, not minutes
            "trt_engine_cache_path": cache_dir,
            "trt_timing_cache_enable": True,
        }))
    if "CUDAExecutionProvider" in available:
        providers.append("CUDAExecutionProvider")
    providers.append("CPUExecutionProvider")

    so = ort.SessionOptions()
    so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    sess = ort.InferenceSession(model_path, sess_options=so, providers=providers)
    print("Active EPs:", sess.get_providers())  # always check which one is really in use
    return sess


def postprocess(out: np.ndarray, metas, conf_thres: float = 0.35, iou_thres: float = 0.5):
    """Boxes from end-to-end output (N, 300, 6) [x1,y1,x2,y2,score,class] or raw output (N, 4+nc, A)."""
    results = []
    for i, (r, (left, top)) in enumerate(metas):
        o = out[i]
        if o.ndim == 2 and o.shape[-1] == 6:  # YOLO26 nms=False: no NMS needed
            keep = o[:, 4] >= conf_thres
            boxes, scores, cls = o[keep, :4].copy(), o[keep, 4], o[keep, 5].astype(int)
        else:  # raw output: (4+nc, A) -> xywh + class scores, needs NMS
            o = o.T
            cls_scores = o[:, 4:]
            cls = cls_scores.argmax(1)
            scores = cls_scores[np.arange(len(o)), cls]
            keep = scores >= conf_thres
            xywh, scores, cls = o[keep, :4], scores[keep], cls[keep]
            boxes = np.column_stack([xywh[:, 0] - xywh[:, 2] / 2, xywh[:, 1] - xywh[:, 3] / 2,
                                     xywh[:, 0] + xywh[:, 2] / 2, xywh[:, 1] + xywh[:, 3] / 2])
            idx = cv2.dnn.NMSBoxes(xywh_to_tlwh(xywh).tolist(), scores.tolist(), conf_thres, iou_thres)
            idx = np.array(idx, dtype=int).reshape(-1)
            boxes, scores, cls = boxes[idx], scores[idx], cls[idx]
        boxes[:, [0, 2]] = (boxes[:, [0, 2]] - left) / r  # undo the letterbox
        boxes[:, [1, 3]] = (boxes[:, [1, 3]] - top) / r
        results.append((boxes, scores, cls))
    return results


def xywh_to_tlwh(xywh: np.ndarray) -> np.ndarray:
    tlwh = xywh.copy()
    tlwh[:, 0] -= xywh[:, 2] / 2
    tlwh[:, 1] -= xywh[:, 3] / 2
    return tlwh


if __name__ == "__main__":
    import sys

    sess = make_session(sys.argv[1])
    frame = cv2.imread(sys.argv[2])
    x, metas = preprocess([frame])
    inp = sess.get_inputs()[0].name
    out = sess.run(None, {inp: x})[0]
    for boxes, scores, cls in postprocess(out, metas):
        for b, s, c in zip(boxes, scores, cls):
            print(f"class={c} score={s:.2f} box={np.round(b).astype(int).tolist()}")

Do not skip the sess.get_providers() line: if the CUDA libraries cannot be found, the session silently falls back to CPU.

Step 3: Build the TensorRT engine

The shortest route is Ultralytics:

from ultralytics import YOLO

YOLO("best.pt").export(format="engine", imgsz=640, quantize=16, nms=False)  # FP16

According to the Ultralytics documentation, the half and int8 arguments have been replaced by quantize=16 and quantize=8; the old names are still accepted with a deprecation warning. With TensorRT 11, Ultralytics uses ModelOpt AutoCast for FP16 and ModelOpt explicit quantization for INT8 on its own.

Doing the same with trtexec is more transparent if you want to control the build pipeline:

# TensorRT 11.x: convert to mixed precision first, then build without precision flags
python -m modelopt.onnx.autocast --onnx_path best.onnx --output_path best_fp16.onnx
trtexec --onnx=best_fp16.onnx --saveEngine=best_fp16.engine \
  --minShapes=images:1x3x640x640 --optShapes=images:4x3x640x640 --maxShapes=images:8x3x640x640 \
  --timingCacheFile=trt.timing.cache

# TensorRT 10.x (e.g. JetPack): precision is set with a flag
trtexec --onnx=best.onnx --saveEngine=best_fp16.engine --fp16 \
  --minShapes=images:1x3x640x640 --optShapes=images:1x3x640x640 --maxShapes=images:4x3x640x640

The engine file is not portable: TensorRT profiles and tunes it on the GPU it was built on. Build the engine on the target device, or on a machine with the same GPU model, driver and TensorRT version. The timing cache (--timingCacheFile) should likewise only be reused on the same hardware and software configuration.

Step 4: INT8 and the calibration set

Post-training quantization (PTQ) runs representative images through the model to find a scale for each activation tensor. The key word is “representative”. The calibration set should:

• Contain frames from the cameras on site, from different times of day, including night/IR mode.
• Cover crowded scenes as well as empty ones.
• Go through exactly the inference preprocessing: letterbox, BGR→RGB, 0–1 scaling.
• Not be made of augmented copies of the training set.

The Ultralytics documentation relays NVIDIA’s recommendation of at least 500 calibration images, and ModelOpt’s example repository gives the same number for CNN and ViT models. It is a guideline for a minimum, not a hard rule.

"""Builds an INT8 calibration array from frames selected on site: (N, 3, 640, 640) float32 -> calib.npy."""
from __future__ import annotations

import random
import sys
from pathlib import Path

import cv2
import numpy as np

from yolo_ort import preprocess

src, out, n = Path(sys.argv[1]), sys.argv[2], int(sys.argv[3]) if len(sys.argv) > 3 else 500
files = sorted(p for p in src.rglob("*") if p.suffix.lower() in {".jpg", ".jpeg", ".png"})
random.seed(0)
files = random.sample(files, min(n, len(files)))  # mix cameras, times of day, day/night

arr = np.empty((len(files), 3, 640, 640), dtype=np.float32)  # about 2.5 GB of RAM for 500 frames
for i, f in enumerate(files):
    x, _ = preprocess([cv2.imread(str(f))])  # exactly the inference preprocessing
    arr[i] = x[0]
np.save(out, arr)
print(f"{len(files)} frames -> {out} {arr.shape}")

500 frames × 3 × 640 × 640 × 4 bytes comes to about 2.5 GB, so reserve that memory on the machine that runs calibration. Then quantize with ModelOpt:

pip install --extra-index-url https://pypi.nvidia.com "nvidia-modelopt[all]"
python -m modelopt.onnx.quantization \
  --onnx_path=best.onnx \
  --quantize_mode=int8 \
  --calibration_data_path=calib.npy \
  --calibration_method=entropy \
  --calibration_shapes=images:1x3x640x640 \
  --output_path=best.int8.onnx
trtexec --onnx=best.int8.onnx --saveEngine=best_int8.engine \
  --minShapes=images:1x3x640x640 --optShapes=images:4x3x640x640 --maxShapes=images:8x3x640x640

Three details in this command make a difference on site. If --calibration_data_path is missing, ModelOpt calibrates with random data; the command does not fail, but the model falls apart in the field. If the model is also dynamic in height and width, you need --calibration_shapes; otherwise unknown dimensions are taken as 1. Third, --high_precision_dtype defaults to fp16: layers that are not quantized end up in FP16, not FP32.

If you are on TensorRT 10.x on a Jetson, the least painful route is to export on the device itself: YOLO("best.pt").export(format="engine", quantize=8, data="site.yaml"). Ultralytics uses the legacy calibrator on those versions and explicitly stresses that calibration is device-specific and that the export should run on the deployment device.

When PTQ is not enough, the next step is quantization-aware training (QAT). In Ultralytics, train(..., quantize=8) fine-tunes a pretrained model and the scales travel with the checkpoint, so no calibration data is needed at export.

Step 5: Measure the accuracy loss

INT8 should not go into the field before its accuracy loss is measured. The method is simple but needs discipline: same validation set, same input size, same head.

"""Compares FP32 / FP16 / INT8 model accuracy on the same validation set."""
from ultralytics import YOLO

DATA = "site_val.yaml"  # validation set labelled on site (not used in training)
MODELS = {"fp32 (pt)": "best.pt", "fp16 (engine)": "best_fp16.engine", "int8 (engine)": "best_int8.engine"}

base = None
for name, path in MODELS.items():
    m = YOLO(path, task="detect").val(data=DATA, imgsz=640, batch=1, plots=False, verbose=False)
    map50_95, map50 = m.box.map, m.box.map50
    base = base if base is not None else map50_95
    print(f"{name:14s} mAP50-95={map50_95:.4f}  mAP50={map50:.4f}  delta={map50_95 - base:+.4f}")

My advice is to write down the acceptable loss before measuring. Tie it to the business metric: daily counting error for people counting, the rate of missed violations for safety. mAP is a general indicator; the real decision should be made on clips recorded on site and labelled by hand, using the business metric. Look at the results per class and per scene type (small objects, night, crowds); a subgroup can degrade while the average looks fine. I covered why models degrade over time and how to manage the dataset in keeping detection models alive in the field.

If the loss is too high, try these in order: enlarge and diversify the calibration set, compare the entropy and max calibration methods, keep sensitive layers out of quantization (--nodes_to_exclude, --op_types_to_exclude), move to QAT, or stay on FP16 for that model.

Step 6: Measure latency and throughput correctly

Measurement errors are the most common problem in acceleration projects. The rules:

1. Warm up. The first calls include memory allocation, kernel selection and, with the TensorRT EP, the engine build; they do not belong in the measurement.
2. Many iterations, percentiles. Time hundreds of calls and report p50, p95 and p99. The stalls you see in the field show up in p95/p99, not in the average.
3. Two separate measurements. Model-only time and end-to-end time including decoding, preprocessing and post-processing answer different questions.
4. Synchronisation. GPU calls in PyTorch are asynchronous; without torch.cuda.synchronize() the measured time is wrong.
5. Fix the environment. Power mode, clocks and temperature change the result. NVIDIA’s documentation explains that clocks can be locked with nvidia-smi -lgc for deterministic measurements; on a Jetson, fix the power mode too. On fanless boxes, keep measuring for several minutes to see thermal throttling.
6. Compare like with like. Head (nms), input size, precision and hardware must be the same.

"""Latency and throughput measurement: warm-up, fixed input, percentiles (p50/p95/p99)."""
from __future__ import annotations

import sys
import time

import numpy as np
import onnxruntime as ort

from yolo_ort import make_session


def bench(sess: ort.InferenceSession, batch: int, size: int = 640,
          warmup: int = 50, iters: int = 500) -> dict:
    name = sess.get_inputs()[0].name
    x = np.random.rand(batch, 3, size, size).astype(np.float32)
    for _ in range(warmup):  # first calls: memory allocation, kernel selection, TensorRT engine build
        sess.run(None, {name: x})
    lat = np.empty(iters)
    t_start = time.perf_counter()
    for i in range(iters):
        t0 = time.perf_counter()
        sess.run(None, {name: x})  # run() returns once results are copied to the CPU; no extra sync needed
        lat[i] = (time.perf_counter() - t0) * 1000
    total = time.perf_counter() - t_start
    p50, p95, p99 = np.percentile(lat, [50, 95, 99])
    return {"batch": batch, "p50_ms": p50, "p95_ms": p95, "p99_ms": p99,
            "img_per_s": batch * iters / total}


if __name__ == "__main__":
    session = make_session(sys.argv[1])
    for b in (1, 2, 4, 8):
        r = bench(session, b)
        print(f"batch={r['batch']:>2}  p50={r['p50_ms']:.2f} ms  p95={r['p95_ms']:.2f} ms  "
              f"p99={r['p99_ms']:.2f} ms  {r['img_per_s']:.1f} img/s")

trtexec does the same at engine level and, by default, warms up for at least 200 ms and runs for at least 10 iterations or 3 seconds:

trtexec --loadEngine=best_int8.engine --shapes=images:1x3x640x640 \
  --warmUp=500 --duration=60

In the output, Throughput (inferences per second), the median and percentile(95%) values on the Latency line, and GPU Compute Time are what matter. In TensorRT 11, host–GPU data transfers are excluded from the measurement by default; add --includeDataTransfers if you want the numbers to reflect the real pipeline. If Enqueue Time is longer than GPU Compute Time, the bottleneck is the host side, not the GPU.

The effect of batch size and resolution

Batching fills the GPU more efficiently and raises images per second, but because the batch has to fill up, the latency of a single image rises too. For a safety rule that needs an immediate reaction, batch 1; for a counting system that tolerates a few hundred milliseconds, a larger batch makes sense. The benchmark script above prints both numbers side by side for batch 1, 2, 4 and 8; base the decision on that table.

For resolution the arithmetic is rougher but useful: going from 640×640 to 1280×1280 quadruples the pixel count, and the work of the convolutional layers grows roughly in the same proportion. Instead of raising the resolution, it is often cheaper to crop the region where distant objects appear and process it separately.

General criteria for choosing hardware

I will not make price/performance claims about specific devices; they are meaningless until measured with your own model. The criteria I look at:

• Supported precisions. Does the hardware accelerate FP16 and INT8 natively? If not, quantization may not bring a speed-up; the ONNX Runtime documentation also notes that quantization may not help on older hardware.
• Memory. If you run one process per camera, each process loads the model separately; total memory can be the first thing that limits the camera count.
• Hardware video decoder. As the camera count grows, H.264/H.265 decoding becomes as important as inference; check how many concurrent streams the hardware decoder can handle.
• Software roadmap. Which TensorRT versions the device will receive, and until when, matters; TensorRT 11 not supporting JetPack is the current example. The Ultralytics documentation also notes that TensorRT 10.7 was the last release with DLA support.
• Power and heat. A fanless box inside a closed cabinet may not sustain its bench result for long.
• Ecosystem and supply. Does your runtime have an EP for that hardware, and will the device still be available in a few years?

Borrow one of the candidates, measure your own model with the script above and trtexec, and decide on that table. I covered packaging the built engine into containers and rolling it out to sites in scaling GPU video analytics services with Docker and Kubernetes.

Checklist

• The TensorRT version on the target device is known (10.x or 11.x) and the build pipeline was chosen accordingly.
• Opset, dynamic axes and external data were chosen deliberately in the ONNX export; input/output shapes were checked.
• Active EPs are logged in ONNX Runtime; the TensorRT EP engine cache is on.
• The engine was built on the target device (or an identical one).
• The calibration set comes from the site, covers day/night and crowded/empty scenes, and went through the inference preprocessing.
• FP32, FP16 and INT8 were compared on the same validation set and on the business metric; the acceptance threshold was written down in advance.
• Latency was measured after warm-up as p50/p95/p99, for batch 1 and the target batch.
• Power mode, clocks and temperature were recorded during measurement.