The first thing we ask a customer is not the camera brand but what the camera puts out. If there is an ONVIF-compliant IP camera producing H.264 or H.265, ninety percent of the job is done. Every system we have built in five years works on that assumption: we do not sell new cameras, we turn the picture from the existing one into something that makes a decision.
The chain has five links. Each has its own bottleneck, and in projects the trouble usually shows up where nobody predicted it.
1. Taking the stream
We pull the stream from the camera over RTSP. Two practical choices here: main stream or sub-stream, TCP or UDP. The main stream is 1080p and above, the sub-stream usually 640×360 or 704×576. People counting and zone intrusion run comfortably on the sub-stream; licence plates and PPE detection want the main stream. UDP has less latency but a lost packet corrupts the frame; on factory networks we always use TCP, the latency difference stays under a few hundred milliseconds and the broken-frame problem disappears.
The thing that eats the most time on site is the NVR itself. Some NVRs block direct access to the camera and force you to take the stream through the NVR. When the NVR's RTSP server serves 8 cameras at once and applies a reduced bit rate per channel, picture quality drops. We no longer quote before measuring this during discovery.
2. Decoding the frame
Leave H.264/H.265 decoding to the CPU and the server chokes at 3–4 cameras. We hand decoding to the GPU's hardware decoder (NVDEC). At 25 fps, 8–16 cameras can be processed on a single GPU server; the number depends on camera resolution and how many models run at the same time. We write that range into quotes as it is. There is no such thing as "unlimited cameras".
Not every frame needs processing. People counting is fine with 8–10 frames per second, fall detection wants 15, forklift proximity should not go below 20. The frame-skip ratio is set per camera; that setting directly determines the GPU budget.
3. Detection and tracking
For detection we use CNN-based single-stage models; the YOLO family has produced the fewest surprises in the field. Detection time per frame is under 40 milliseconds; that figure holds for 1080p input and a model compiled with TensorRT. However good the model, you cannot count without tracking: the same person is counted 30 times in 30 frames. Multi-object tracking (ByteTrack-style) gives each detection an identity, and the count happens when the identity crosses a line.
Tracking has two classic failures. Identity switch: two people overlap and part, and the identities swap. Identity loss: someone walks behind a pillar and comes out 2 seconds later with a new identity, counted twice. The cure for both is not the model, it is camera placement. I will cover that in a separate article; in short, when the camera is on the ceiling looking straight down, most of the problems solve themselves.
4. The rule engine
A detected object on its own means nothing. What turns "there is a person" into an event is a set of spatial and temporal rules. Spatial rule: inside this polygon. Temporal rule: for longer than 10 seconds. Combine them and you get the event "person staying in a restricted zone for more than 10 seconds". Queue length, waiting time, distance between a forklift and a pedestrian, an operator passing without a helmet; all of these are different rule sets on the same engine.
The reason for keeping the rule engine separate from the model is simple: when the customer wants to change a rule, the model should not need retraining. Zone polygon, time threshold, distance limit all change from the interface; the model keeps running untouched.
5. Event and metric output
Output comes in two kinds. Event: instant, singular, usually an alarm or a notification (webhook, e-mail, MQTT). Metric: aggregated, time series, for dashboards and BI tools (JSON, CSV, REST API). The retail customer wants metrics, the safety customer wants events. The manufacturing customer wants both and expects them to talk to the MES.
The event schema has to be fixed on day one. In ours every event carries the camera identifier, timestamp, event type, confidence score, the related frame (optional, depending on the privacy setting) and the rule identifier. As long as this schema does not change, the integration on the customer's side does not break either.
Where things go wrong
- The network, not the model. In half of the projects the first week goes to switches, VLANs and NVR settings. The camera network and the server not being on the same segment is a job in itself.
- Night footage. A camera that switches to IR mode gives a black-and-white picture, and a model trained on daytime data drops at night. No acceptance test is run before night data is added to the training set.
- Clocks. If camera, NVR and server are on different times you cannot match events with POS or MES records. NTP is mandatory.
- Bandwidth. 16 cameras × 4 Mbps = 64 Mbps of continuous traffic. Projects that try to push that to the cloud over a store's 20 Mbps line die in the first month. That is why we process on site; only events and metrics go to the cloud.
The whole chain runs on one GPU server, inside the store or the factory. The video does not leave. What comes out is a line like "at 14:32 the queue at till 3 exceeded 6 people". That is exactly what we mean when we say the camera stops recording and starts deciding.