In our first year we trained the model once and threw it into the field. Three months later the customer called: "the counting has broken". We looked: same model, same camera; what had changed was the store switching to its winter collection. Thick coats, hats, scarves. The training set had summer clothing. Since that day a model is not a product but a system with maintenance.
Where drift comes from
The sources of deviation we see in the field, in order of frequency:
- Season and clothing. The winter–summer difference causes trouble not in person detection but in classification (staff vests, PPE). A yellow vest under a coat becomes "no vest".
- Light. The window lighting was renewed, new LEDs installed in the plant, the days got longer. When exposure changes, the distribution the model sees changes.
- Layout. A shelf moved, a stand placed in front of the door, the production line rearranged. The model does not break, but the rule polygons become meaningless.
- Camera. Replaced, knocked, lens fogged, autofocus failed. The sneakiest one: a camera firmware update changing the image-processing settings (sharpness, noise reduction).
- New objects. A new type of forklift arrived at the warehouse, a shape the model has never seen. Detection rate for that vehicle is near zero.
Noticing the drift
You need to notice before the customer calls. We watch three signals:
Confidence distribution: the daily average detection confidence per camera. A camera going from 0.85 to 0.70 means something has changed. Count consistency: an alert if today's count deviates more than 20% from the same day last week; sometimes real (a campaign), mostly technical. Empty-frame ratio: hours in which the camera produces no detections at all, night excluded. All of this on one monitoring panel, per camera. A sample frame is requested from the camera showing deviation and someone looks.
Dataset discipline
Every customer, where they have agreed, contributes to the dataset. The rules:
- Face blurring before labelling, automatic. The labeller never sees a face.
- Hard examples first. Not random frames but frames where the model detected with low confidence or tracking lost an identity. A hundred hard examples over a thousand easy ones.
- A fixed label schema. "person", "vest", "helmet", "forklift", "pallet-jack"... Adding a new class is a decision; a class is not opened on every customer request.
- Source tags. Every sample carries which site, which camera type, which season. For a new customer the nearest subset can be used for fine-tuning.
- The test set is untouchable. The test set held out from every customer never enters training. Model versions are compared on this set; the sentence "the new model is better" is not said without that table.
The retraining loop
The general model is retrained once a quarter. Customer-specific fine-tuning when needed: a new forklift type, a special uniform, an unusual camera angle. Fine-tuning runs on top of the general model with a few hundred of that customer's samples and finishes in a few hours. The result is checked not only on that customer's test set but on the general test set too; fine-tuning may have broken something else.
Training happens centrally, on a machine with a GPU. No training on site servers; the GPU there is reserved for inference.
Versioning and rollout
The model is compiled with TensorRT and goes to the sites as an update. Every site server has two model slots: active and candidate. The candidate runs in shadow mode on the same stream for a day, its output is compared with the active one, the difference report is reviewed, then the switch. Rollback is a single command: swap the slots. Before we built this, one update broke counting in 12 stores at the same time; shadow mode was born that week.
Every event record carries the model version. Without "the first half of the month was counted with v14, the second half with v15", monthly comparison is impossible.
Solutions outside the model
Half of the deviations are fixed with settings, not the model. Locking the camera exposure, turning off autofocus, updating the rule polygon, raising the frame rate. That list is checked before going to retraining. Retraining the model on every problem is expensive and usually a misdiagnosis.
Not a summary, a lesson
Training a model takes two days. Keeping it accurate in 40 locations for a year is the team's real job. That is why we write the maintenance item separately in quotes, and when the customer says "the model is done, why maintenance?" we send them this article.