Methodology · 7 min read
How to scope and measure a retail video analytics pilot
Most pilots in this category fail for the same two reasons: the KPIs were never defined precisely enough to be measured, and there was no baseline to compare against. Both are avoidable in week one.
1. Pick three or four KPIs. Not twelve.
A pilot with a dozen metrics produces a dozen half-measured numbers and no decision. Three or four, chosen because someone will act differently depending on the answer, is enough. If nobody can say what they would do differently, the metric is interesting rather than useful — leave it out.
2. Write the definition down before anything is measured
“Queue length” is not a definition. The definition is the zone, the count threshold, the duration condition, the hours it applies to and the exclusions. Write it in a sentence both your operations lead and the engineer can sign. Here is what that looks like in practice:
Queue events above a threshold
- Definition
- Count of occasions where the defined checkout zone held more than N people for more than M consecutive minutes, during trading hours.
- Where it goes wrong
- Without the duration condition, a brief cluster of three shoppers registers as a queue and the alert volume becomes unusable.
Visits by entrance
- Definition
- Count of distinct entries through a named doorway per hour, excluding staff entrances and deliveries.
- Where it goes wrong
- Staff crossing the line all day can inflate this by a wide margin. Decide how staff movement is handled before you trust the number.
Dwell in a defined zone
- Definition
- Aggregate time that activity persists within a drawn polygon, reported as a distribution rather than a single average.
- Where it goes wrong
- An average hides the shape. A zone with two shoppers for twenty minutes looks identical to twenty shoppers for two.
Time from event to review
- Definition
- Elapsed time between an event being raised and an authorised person recording an action against it.
- Where it goes wrong
- This measures your process, not the software - which is precisely why it is worth tracking from day one.
3. Insist on a human-reviewed ground truth sample
Take a sample of footage — a few hours spread across different times and lighting conditions — and have a person count what actually happened. That sample is the only way to know whether the automated count is right. Agree the minimum acceptable precision and recall before you see the results, otherwise the threshold will quietly move to whatever the system achieved.
4. Capture the baseline first
You cannot show a change without a before. Capture at least a few weeks of baseline on the agreed definitions before anyone changes a layout, a rota or a promotion. A pilot that starts with an intervention has thrown away its own control.
5. Write down the confounders while they happen
Weather, holidays, a competitor closing, road works, a staffing change, a neighbouring unit being refitted. Keep a dated log during the pilot. At the end, this log is what separates a finding from a coincidence — and nobody can reconstruct it afterwards from memory.
6. Agree what “stop” looks like
Decide in advance what result would make you walk away, and who gets to call it. A pilot with no stop condition is a procurement exercise with extra steps. The three honest outcomes are scale, improve and stop — and a vendor who cannot describe the third is telling you something.
Sightora’s own pilot method follows this structure, step by step, on the pilot methodology page.