There is no such thing as a single people-counting accuracy figure. An accuracy rate given without saying at what hour, at which door, with how large a sample and under which error definition it was measured is a marketing sentence. The 96.2% we quote is a rate measured in several stores, compared against manual counts, calculated over the sum of entries and exits. What follows are the notes that piled up in the notebook on the way to that number.
Measurement method first
A person stands at the door with a clicker and counts every hour; whatever the system counted in the same hour is compared. Three slots of the day are measured separately: the quiet morning, the busy midday, the mixed evening. Saturday separately. One day of measurement is not enough; at least three days, preferably a week.
Error runs both ways: over-counting and under-counting. The two can cancel each other out, and a system that looks 98% in total may actually be counting 6% over and 4% under. That is why we measure absolute error: (|over| + |under|) / actual. We give the customer this definition in writing.
Camera angle above everything
Two thirds of the low accuracy we see in the field comes from camera placement. The ranking:
- Top-down camera (ceiling, 3–4 metres, 90 degrees). Almost no occlusion, a person becomes a circle, the tracking identity does not get lost. Its only drawback: existing security cameras are not mounted this way.
- Angled camera (45–60 degrees). Most existing security cameras are like this. It works, but two people entering one behind the other hide each other. Accuracy stays in the 92–95% band.
- Frontal camera (horizontal, eye level). Not suitable for counting. Groups entering together become one person. We do not set up counting on this camera; we recommend an additional camera, and even though the quote says "0 extra hardware", this is the exception and we say so openly.
Reaching 96% with an existing angled camera is possible, but with line and threshold settings, below.
Where the counting line goes
The line should not sit right over the door. Detection jitters while the door opens and closes, people stop on the line to look at their phone, someone waiting for the automatic door sensor steps back and forth. We pull the line 1–1.5 metres inside the door. Instead of a single line we use two parallel lines: the person is counted when they cross first one, then the other; direction comes from the order as well. Someone who waits at the door and turns back never reaches the second line and is not counted.
In a shopping-mall store, the person who stops at the window and does not come in creates the difference between "passing" and "entering". We count the two separately; window interest rate is a metric in its own right.
Staff
Store staff pass through the door 40–60 times a day. In a store with 300 visitors that is a 15% systematic error. The fix is not face recognition; we do not do that. The fix is the staff vest or uniform: a separate class based on colour and pattern. Where there is no uniform, staff entry and exit times are taken from the shift plan and subtracted from the count. The second is crude but works.
Night and light
When the window lighting comes on in the evening and the store interior is dimmed, the camera changes its exposure; a camera that switches to IR gives a black-and-white picture. We saw the model trained on daytime data drop 10% in the evening hours in the first project. Since that day the training set has evening footage from every store and the acceptance test is not run without covering the evening slot.
Children and shopping trolleys
Should children be counted? Depends on the customer; the toy store counts them, the menswear store does not. The height threshold in the model is adjustable. Shopping trolleys and pushchairs are separate classes and are never counted as people. On the top-down camera a mother entering with a pushchair is one person; on the angled camera sometimes two. This too comes back to the angle question.
Accuracy decaying over time
A system at 96% at installation can be down to 90% six months later. Reasons in order: the store layout changed, a display stand was placed in front of the door; the camera was not cleaned, the lens fogged; the season changed, thick coats and hats in winter. We added a monthly automatic check: the daily count is compared with the same day of the previous week, and when the deviation exceeds 20% an alert goes out and someone looks. A yearly site visit is planned separately.
Acceptance criterion
The sentence we write into the contract: "Absolute error under 5% when compared with manual counts in three slots of the day over three working days." The system is not handed over until this is met. When it cannot be met the reason is almost always the camera angle, and then the thing to talk about is not the model but an additional camera.