Scaling GPU video analytics services with Docker and Kubernetes

Scaling GPU video analytics services means packaging per-camera inference processes as containers and managing their GPU access, health and updates in a repeatable way. Docker Compose on a single server and Kubernetes for multi-server or multi-site deployments are the two scales of the same job.

In this article I turn the script from real-time object detection with Python and YOLO into a service. Compiling the model into a TensorRT engine is covered in ONNX, TensorRT and INT8; here we deal with rolling that engine out to sites.

The architectural decision: one process per camera

I run a separate container per camera (a separate pod in Kubernetes). Because tracking state is specific to a camera, separating processes keeps the code simple; a stream problem on one camera does not affect the others; and updates can be done camera by camera. The price is GPU memory: each process loads the model separately.

Approach	Pros	Cons
One process per camera	Failure isolation, simple code, per-camera updates	Each process loads the model separately
One process for many cameras	Batched inference, less memory	One failure stops every camera, more complex code

Moving to a multi-camera process makes sense once the camera count starts to strain GPU memory; until then, one process per camera is the easier option to operate.

Step 1: GPUs in Docker: the NVIDIA Container Toolkit

The NVIDIA driver must be installed on the host; the CUDA libraries come from inside the image. The NVIDIA Container Toolkit is the layer that connects the driver to containers. After adding the package repository as described in NVIDIA’s installation guide:

sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi   # is the GPU visible?

If the last command prints the nvidia-smi table, containers can see the GPU.

Step 2: A multi-stage Dockerfile

The first stage installs the dependencies into a virtual environment; the second stage takes only that environment and the application files. The pip cache and build leftovers never reach the final image.

# Pin versions; the TensorRT that builds the engine must be the one that runs it.
ultralytics==8.4.163
tensorrt-cu13==11.3.0.99      # the torch 2.14 wheel on PyPI pulls CUDA 13 libraries
nvidia-modelopt[onnx]==0.47.0 # on TensorRT 11, FP16/INT8 export goes through ModelOpt
paho-mqtt==2.1.0
prometheus-client==0.26.0

# syntax=docker/dockerfile:1

# --- stage 1: install dependencies into a virtual environment -------------------
FROM python:3.12-slim-bookworm AS build
ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1
RUN python -m venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH
COPY requirements.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt

# --- stage 2: runtime only --------------------------------------------------------
FROM python:3.12-slim-bookworm AS runtime
# system libraries needed by OpenCV (opencv-python)
RUN apt-get update \
 && apt-get install -y --no-install-recommends libgl1 libglib2.0-0 \
 && rm -rf /var/lib/apt/lists/*
COPY --from=build /opt/venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH \
    PYTHONUNBUFFERED=1 \
    YOLO_CONFIG_DIR=/tmp/ultralytics \
    NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
RUN useradd --uid 10001 --no-create-home app
WORKDIR /app
COPY --chown=app:app rtsp_yolo_events.py observability.py ./
# The model is not baked in: a TensorRT engine is specific to the GPU and TensorRT version; mount it at /app/models.
USER 10001
EXPOSE 9108
# The container is unhealthy if the main loop has not updated the heartbeat file for 30 s
HEALTHCHECK --interval=15s --timeout=3s --start-period=120s --retries=3 \
  CMD ["python", "-c", "import os,sys,time; sys.exit(0 if time.time()-os.path.getmtime('/tmp/heartbeat')<30 else 1)"]
ENTRYPOINT ["python", "rtsp_yolo_events.py"]

A few decisions:

• The model is not baked into the image. A TensorRT engine is specific to the GPU and TensorRT version it was built with; if the same image runs on sites with different GPUs, the engine is built on each site and mounted under /app/models.
• Versions are pinned. The TensorRT version that builds the engine must be the one that runs it; a latest tag and unpinned packages break that match.
• ModelOpt is in the image. With TensorRT 11, Ultralytics tries to install ModelOpt itself on first use; that install cannot happen in a container running as a non-root user, so I add the package up front.
• video in NVIDIA_DRIVER_CAPABILITIES is needed for the container to reach the hardware video decoder; compute is for CUDA and utility for nvidia-smi.
• A non-root user, and a writable YOLO_CONFIG_DIR for Ultralytics’ settings file.

For health checks and Prometheus metrics I add a small module to the service:

"""Adds Prometheus metrics and a heartbeat file to the analytics service (prometheus-client)."""
from __future__ import annotations

import os
import time
from pathlib import Path

from prometheus_client import Counter, Gauge, Histogram, start_http_server

CAMERA = os.getenv("CAMERA_ID", "cam-07")
HEARTBEAT = Path(os.getenv("HEARTBEAT_FILE", "/tmp/heartbeat"))

FRAMES = Counter("va_frames_processed_total", "Processed frames", ["camera"])
EVENTS = Counter("va_events_total", "Events emitted", ["camera", "type"])
RECONNECTS = Counter("va_rtsp_reconnects_total", "RTSP reconnects", ["camera"])
INFER = Histogram("va_inference_seconds", "Detection+tracking time per frame", ["camera"],
                  buckets=(0.005, 0.01, 0.02, 0.04, 0.08, 0.16, 0.32))
LAST_FRAME = Gauge("va_last_frame_timestamp_seconds", "Time of the last processed frame", ["camera"])

_last_beat = 0.0


def start(port: int = 9108) -> None:
    start_http_server(port)  # /metrics


def heartbeat() -> None:
    """Call on every turn of the main loop; writes the file at most once per second.
    If the loop keeps turning while the camera is down, the process is healthy: a restart will not fix the camera."""
    global _last_beat
    now = time.time()
    if now - _last_beat >= 1.0:
        HEARTBEAT.write_text(str(int(now)))
        _last_beat = now


def frame_done(seconds: float) -> None:
    """Call for every processed frame."""
    FRAMES.labels(CAMERA).inc()
    INFER.labels(CAMERA).observe(seconds)
    LAST_FRAME.labels(CAMERA).set(time.time())

Wiring it into the script takes a few lines: obs.start(9108) at the start of main(), obs.heartbeat() on every turn of the main loop, timing the model.track call and passing it to obs.frame_done(seconds), obs.EVENTS.labels(CAMERA_ID, kind).inc() when an event is emitted, and obs.RECONNECTS.labels(CAMERA_ID).inc() when the reader reconnects.

It matters why the heartbeat is written on every loop turn rather than on every processed frame. When a camera goes down, the process is healthy and the problem is the camera; if the heartbeat depended on frames, the container would be restarted on every camera fault, which would not fix the camera. The Kubernetes documentation also warns that tying liveness checks to external dependencies can lead to cascading restarts. Camera outages are tracked as an alert on a metric instead.

Step 3: One service per camera with docker compose

# One service per camera; shared settings live in one place via a YAML anchor (x-analytics).
x-analytics: &analytics
  image: registry.example.com/cx/video-analytics:1.4.2
  restart: unless-stopped
  env_file: common.env        # SITE, MODEL_PATH, MQTT_HOST, MQTT_CA
  volumes:
    - ./certs/ca.crt:/etc/cx/ca.crt:ro
    - ./models:/app/models:ro   # best.engine built on this machine
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            device_ids: ["0"]
            capabilities: [gpu]
  logging:
    driver: json-file
    options: { max-size: "10m", max-file: "3" }

services:
  cam-01:
    <<: *analytics
    environment:
      CAMERA_ID: cam-01
      RTSP_URL: ${CAM01_RTSP_URL}
      MQTT_USER: cam-01
      MQTT_PASS: ${CAM01_MQTT_PASS}
      PROCESS_FPS: "8"
  cam-02:
    <<: *analytics
    environment:
      CAMERA_ID: cam-02
      RTSP_URL: ${CAM02_RTSP_URL}
      MQTT_USER: cam-02
      MQTT_PASS: ${CAM02_MQTT_PASS}
      PROCESS_FPS: "15"

Shared settings live in one place through a YAML anchor (x-analytics); each camera adds only its own ID, stream URL and MQTT credentials. Passwords go into the .env file that Compose reads for variable substitution, not into common.env, so one camera’s stream URL never ends up in another container’s environment. Keep .env out of version control.

According to the Docker documentation, the capabilities field is mandatory and count and device_ids cannot be used together. Docker does not put a memory limit between containers sharing the same GPU; each container can use all of the GPU memory, so decide the camera count by measuring memory.

Building the engine once on the site machine with the same image, then starting the services:

docker run --rm --gpus all --user "$(id -u):$(id -g)" -v "$PWD/models:/app/models" \
  --entrypoint yolo registry.example.com/cx/video-analytics:1.4.2 \
  export model=/app/models/best.pt format=engine imgsz=640 quantize=16 nms=False
docker compose up -d
docker compose ps        # the STATUS column should show (healthy)

Step 4: GPUs in Kubernetes: device plugin and GPU Operator

Kubernetes does not know about GPUs by itself; you need the NVIDIA device plugin, which advertises the node’s GPUs as the nvidia.com/gpu resource. The NVIDIA GPU Operator brings it along with the driver, container toolkit, GPU Feature Discovery, DCGM Exporter and MIG Manager in a single Helm install:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
helm install --wait --generate-name -n gpu-operator --create-namespace \
  nvidia/gpu-operator --version=v26.7.1
# If the driver and toolkit are already installed on the host: --set driver.enabled=false --set toolkit.enabled=false

The rules from the Kubernetes documentation: GPUs are requested only under limits, requests must equal limits if given, and fractional GPUs cannot be requested. A Deployment per camera:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: va-cam-07
  namespace: video-analytics
  labels: { app: video-analytics, camera: cam-07 }
spec:
  replicas: 1
  strategy:
    type: Recreate            # never let two pods count the same camera
  selector:
    matchLabels: { app: video-analytics, camera: cam-07 }
  template:
    metadata:
      labels: { app: video-analytics, camera: cam-07 }
    spec:
      # runtimeClassName: nvidia   # needed on k3s with a standalone device plugin (see below)
      nodeSelector:
        nvidia.com/gpu.present: "true"
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
      containers:
        - name: analytics
          image: registry.example.com/cx/video-analytics:1.4.2
          env:
            - { name: CAMERA_ID, value: cam-07 }
            - { name: SITE, value: gebze-01 }
            - { name: PROCESS_FPS, value: "8" }
            - { name: MODEL_PATH, value: /app/models/best.engine }
            - { name: MQTT_HOST, value: mosquitto.iot.svc }
            - { name: MQTT_CA, value: /etc/cx/ca.crt }
            - { name: MQTT_USER, value: cam-07 }
            - name: RTSP_URL
              valueFrom: { secretKeyRef: { name: cam-07-rtsp, key: url } }
            - name: MQTT_PASS
              valueFrom: { secretKeyRef: { name: mqtt-cam-07, key: password } }
          ports:
            - { name: metrics, containerPort: 9108 }
          resources:
            requests: { cpu: "1", memory: 1Gi }
            limits:
              memory: 3Gi
              nvidia.com/gpu: 1       # GPUs only in limits; must be an integer
          startupProbe:               # up to 5 minutes for model loading and TensorRT set-up
            exec:
              command: ["python", "-c", "import os,sys,time; sys.exit(0 if time.time()-os.path.getmtime('/tmp/heartbeat')<30 else 1)"]
            periodSeconds: 10
            failureThreshold: 30
          livenessProbe:              # restart if the main loop has not turned for 30 s (stuck)
            exec:
              command: ["python", "-c", "import os,sys,time; sys.exit(0 if time.time()-os.path.getmtime('/tmp/heartbeat')<30 else 1)"]
            periodSeconds: 15
            failureThreshold: 4
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - { name: models, mountPath: /app/models, readOnly: true }
            - { name: ca, mountPath: /etc/cx, readOnly: true }
            - { name: tmp, mountPath: /tmp }
      volumes:
        - name: models              # engine built on this node, for this GPU
          hostPath: { path: /opt/cx/models, type: Directory }
        - name: ca
          secret:
            secretName: mqtt-ca
            items: [{ key: ca.crt, path: ca.crt }]
        - name: tmp
          emptyDir: {}

Secrets are not written into the manifest:

kubectl create namespace video-analytics
kubectl -n video-analytics create secret generic cam-07-rtsp --from-literal=url='rtsp://<user>:<password>@10.0.0.21:554/stream2'
kubectl -n video-analytics create secret generic mqtt-cam-07 --from-literal=password='<password>'
kubectl -n video-analytics create secret generic mqtt-ca --from-file=ca.crt=./certs/ca.crt

The decisions in the manifest:

• strategy: Recreate. The default RollingUpdate keeps the old pod running until the new one is ready: two pods count the same camera and produce duplicate events. On top of that, if no GPU is free the new pod stays Pending and the rollout hangs. Recreate accepts a gap of a few seconds and removes both problems.
• No CPU limit, but a memory limit. A CPU limit can throttle decoding threads and add latency; my preference is a request only for CPU and a limit for memory.
• nodeSelector uses the nvidia.com/gpu.present label set by GPU Feature Discovery.
• startupProbe allows up to 5 minutes for model loading and TensorRT set-up; liveness does not start until it succeeds.
• readOnlyRootFilesystem makes the image read-only; /tmp is the only writable place.
• In current GPU Operator releases CDI (Container Device Interface) is on by default and the Operator no longer makes the nvidia runtime the default handler, so runtimeClassName is commented out in the example. On k3s it is different (see below).

Step 5: Sharing one GPU: time-slicing, MPS, MIG

By default one pod gets a GPU to itself. To avoid needing twelve GPUs for twelve camera pods, you need sharing.

Time-slicing advertises a GPU as a number of replicas:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config-all
  namespace: gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4

kubectl create -n gpu-operator -f time-slicing-config.yaml
kubectl patch clusterpolicies.nvidia.com/cluster-policy -n gpu-operator --type merge \
  -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config-all", "default": "any"}}}}'

With this configuration each physical GPU shows up as four nvidia.com/gpu. NVIDIA’s documentation is explicit about the caveats: there is no memory or fault isolation between replicas; requesting more than one replica does not guarantee proportional compute; and with time-slicing enabled, DCGM Exporter cannot associate metrics with containers. If one pod fills the GPU memory, the others on the same GPU are affected too.

MPS is supported in the device plugin as an experimental feature; it limits each client’s memory to an equal share and also caps its compute capacity, and it does not work on MIG-enabled devices.

MIG (Multi-Instance GPU) partitions supporting GPUs into instances with hardware-level memory and fault isolation. In the GPU Operator you choose mig.strategy (single or mixed) and assign a profile to the node with the nvidia.com/mig.config label; pods request resources such as nvidia.com/mig-1g.10gb. It is available only on MIG-capable GPUs.

Method	Memory isolation	Fault isolation	Where
Time-slicing	None	None	Any NVIDIA GPU
MPS (experimental)	Equal-share limit	Limited	GPUs without MIG enabled
MIG	Yes	Yes	MIG-capable GPUs

A newer option: according to the GPU Operator 26.7.1 release notes, with an R615 or later driver the NVIDIA_GPU_MEMORY_REQUEST and NVIDIA_GPU_MEMORY_LIMIT environment variables (in MiB) can set soft and hard per-container CUDA memory limits. That could close the memory-isolation gap of time-slicing; it is new, so do not rely on it before testing with your own driver and version combination.

Step 6: Lightweight Kubernetes at the edge: k3s

A full Kubernetes cluster is heavy for a single GPU server in a store or a factory. k3s is a fully compliant Kubernetes distribution shipped as a single binary; it lets you use the same manifests and the same rollback method across many sites. For a single-server, single-site installation, Compose is often enough.

According to the k3s documentation, the order is: install the NVIDIA driver and container runtime on the host, then install (or restart) k3s; k3s finds the runtime, adds it to the containerd configuration, and ships a ready-made nvidia RuntimeClass.

curl -sfL https://get.k3s.io | sh -
sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml   # was the runtime found?
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin && helm repo update
helm upgrade -i nvdp nvdp/nvidia-device-plugin -n nvidia-device-plugin --create-namespace \
  --set runtimeClassName=nvidia --set-file config.map.config=dp-time-slicing.yaml

dp-time-slicing.yaml holds the same content as the any key of the ConfigMap above. If the default runtime has not been changed, pods need runtimeClassName: nvidia; uncomment that line in the Deployment. Pulling images over thin site links is another issue: k3s can be configured through /etc/rancher/k3s/registries.yaml to pull from a local mirror registry.

Step 7: Monitoring: Prometheus and DCGM Exporter

Monitoring has two layers. The GPU layer comes from DCGM Exporter; it is included by default with the GPU Operator and, when installed on its own, publishes /metrics on port 9400. The application layer comes from the service’s own metrics. If you use the Prometheus Operator, for the service metrics:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: video-analytics
  namespace: video-analytics
spec:
  selector:
    matchLabels: { app: video-analytics }
  podMetricsEndpoints:
    - port: metrics
      interval: 30s

The queries I use for alerts and dashboards:

# No frames processed from a camera for 60 s (camera, network or stream problem)
time() - va_last_frame_timestamp_seconds > 60
# Processed frame rate per camera
rate(va_frames_processed_total[5m])
# p95 inference time per camera
histogram_quantile(0.95, sum by (le, camera) (rate(va_inference_seconds_bucket[5m])))
# GPU memory usage (approximate)
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)
# Hardware decoder utilisation and the last XID error
DCGM_FI_DEV_DEC_UTIL
DCGM_FI_DEV_XID_ERRORS > 0

DCGM Exporter’s default counter list includes GPU utilisation, memory, temperature, power, encoder/decoder utilisation and XID errors. In video analytics, always watch decoder utilisation: in my experience, as the camera count grows, the bottleneck often turns out to be decoding rather than inference.

Step 8: Update strategy

• Immutable tags. Use version tags such as 1.4.2, never latest, and never move a tag to a different image.
• Image and model are versioned separately. Name engine files by date (best-2026-09-25.engine) and select one with MODEL_PATH. Rolling back is as easy as pointing at the old file.
• Camera-by-camera canary. Update one camera’s Deployment first, watch its processed frame rate, p95 time and event counts for a while, then move on to the others.
• Rebuild the engine on the target. When the driver, TensorRT or GPU changes, rebuilding the engine on that hardware should be part of the release pipeline.
• Have the rollback command ready:

kubectl -n video-analytics set image deployment/va-cam-07 analytics=registry.example.com/cx/video-analytics:1.4.3
kubectl -n video-analytics rollout status deployment/va-cam-07
kubectl -n video-analytics rollout undo deployment/va-cam-07     # if something goes wrong

Do driver and GPU Operator upgrades by draining the node and within a maintenance window; with a single node per site, that means analytics at that site stops for a short time. For moving events securely over MQTT, see secure IoT with MQTT.

Checklist

• The driver and NVIDIA Container Toolkit are installed on the host; docker run --gpus all ... nvidia-smi works.
• The image is multi-stage, runs as a non-root user, has pinned versions; the model lives outside the image.
• The TensorRT engine was built on the target machine with the TensorRT version in the image.
• Passwords and stream URLs are in Secrets or .env, not in version control.
• GPUs are requested only under limits; a memory limit and a CPU request are set.
• Camera pods use the Recreate strategy; no double counting.
• The GPU sharing method (time-slicing, MPS, MIG) matches the isolation requirement; memory was measured.
• Liveness is tied to the main loop; camera outages are tracked by alerts.
• DCGM Exporter and service metrics are in Prometheus; decoder utilisation is on the dashboard.
• Updates are tried on one camera first; the rollback command and model file history are ready.