Every AI-visibility tool has an instrument problem, and we have not seen one publish its noise floor. So we measured ours. Our classifier is an AI that reads AI answers and assigns a verdict. If it flickers, the volatility we report could be partly our own tool, not the engines. The test settles it: feed the classifier the same answer text, repeatedly. The input never changes, so any disagreement is pure instrument noise. No human labelling needed.
The numbers
196 answers — every answer in a 699-answer client baseline where the classifier makes a judgment call — each classified five times, on two consecutive days.
| Measure | Day 1 (5 Jul) | Day 2 (6 Jul) |
|---|---|---|
| Pairwise self-disagreement | 1.22% | 1.02% |
| Answers fully stable across 5 repeats | 97.4% | 98.0% |
| “Recommended” verdicts stable | 115 / 115 | 115 / 115 |
Day over day, the classifier gave the same verdict on the same answer 24 hours apart in 195 of 196 cases — 99.5%. The single exception was one genuinely ambiguous answer that both praises and warns about a brand, flipping between “negative” and “recommended.” That is the number that answers “does your tool drift?”
What it means, stated plainly
- The volatility we publish is 19% of same-question repeats changing verdict. The instrument’s noise floor on identical text is ~1%. The wobble is overwhelmingly the engines, and that sentence now carries a measurement rather than a hope.
- One honest caveat, raised before anyone raises it for us: between-run comparisons involve different answer texts, so the identical-text floor is a lower bound on the instrument’s total contribution. Bounding the rest needs human-labelled answers; that audit is scheduled and this reading will be updated with it. Even so, the gap between 1% and 19% is not the kind volatility hides in.
- Negative verdicts are the softest class (all the flicker lived there). We are adding majority-of-five voting on negatives. Until it ships, read negative counts ±1–2 — including in our own past reports.
Why we published it
“Measured, not estimated” should include the measuring stick. If a number we report can be challenged as our instrument shaking, the honest move is to measure the shake and show you. The test script re-runs whenever the classifier changes.
Download a sample of the per-answer verdicts (CSV) · method in full: how we measure.
Measured 5–6 July 2026 · classifier gpt-4o-mini, version 3 · one client baseline