Every dataset in our benchmark battery except one is a snapshot: a fixed pool of (query, candidate answer) pairs, scored once, split once. SemCacheLMArena, SemCacheSearchQueries, and Quora Question Pairs all behave this way, and on all three, fine-tuning a verifier on a tenant's own gray-zone feedback held up — no measurable decay over the test window.
The one dataset that isn't a snapshot is real Twitter customer-support traffic from comcastcares, spanning multiple years. And on that dataset, fine-tuning didn't just fail to help. It turned actively harmful.
What actually happened
The gray-zone positive rate — the fraction of similarity-ambiguous cache hits that are genuinely correct — isn't a fixed property of a task. It's a property of this month's traffic. Support topics shift, products get discontinued, new complaint categories emerge. On comcastcares, that rate drifted by roughly 5x over the traffic window we tested against.
A verifier fine-tuned on an earlier slice of that traffic had learned a threshold calibrated to the old rate. When the real rate moved, the threshold didn't move with it — and a model that used to improve on the stock verifier started approving more wrong answers than it caught. The fine-tuned model was, by the numbers, worse than doing nothing.
This is the uncomfortable result underneath the tidier ones. Fine-tuning is the validated remedy for a real off-the-shelf verifier's weak accuracy (PAPER.md 5.7-5.8) — it tolerates real label noise, it needs a few hundred to a thousand examples depending on the task, and it doesn't decay on three of our four benchmarks. But "doesn't decay on three of four" is a different, narrower claim than "doesn't decay," and the one dataset where it decayed is also the one dataset that looks most like a real production workload: long-running, non-stationary, subject to whatever a support org's customers are actually complaining about this quarter.
Why a label-based monitor isn't enough on its own
The first fix we shipped watches the thing that broke directly: the gray-zone positive rate itself, against the rate the active model was trained on. Two detectors run in parallel —
- a chunked two-proportion z-test, checking whether a recent window's positive rate differs significantly from the training-window baseline, and
- a Page-Hinkley test, a sequential change-point detector built for exactly this — catching a gradual drift that a single-window z-test might average away before it crosses a threshold.
Either one flagging sets a tenant's drift status to flagged. For Managed tenants this isn't just a dashboard color change: it triggers an out-of-cycle recalibration automatically, on top of whatever weekly/daily schedule the tier already runs, and it marks any Conformal-Risk-Control-certified threshold as stale_recalibration_pending until a fresh job completes.
That catches the comcastcares failure mode directly — the positive rate moved, the monitor saw it move. But it's a label-based signal, and labels are the thing callers have to go out of their way to provide (POST /v1/feedback). There's a second, quieter failure mode it can miss entirely: the verifier's own score distribution shifting — more candidates clearing a fixed threshold, or fewer — while the underlying label error rate happens to stay flat for a while. Nothing about that shows up in a positive-rate test, because a positive-rate test only ever looks at labels.
The detector we added for the case labels don't catch
That's what a third detector, shipped this month, is for: a two-sample Kolmogorov-Smirnov test comparing the verifier's own score distribution on recent traffic against a baseline sample captured at fine-tune time. It's recorded automatically — every POST /v1/feedback call now scores the pair with the active verifier and stores the score alongside the label, no extra work from the caller — and it flags a covariate shift in the score distribution itself, independent of whether the label-based detectors have caught anything yet.
Any one of the three detectors flagging is enough to set status: "flagged" on GET /v1/monitor/drift-status. None of them individually claims to be sufficient; together they cover the label-drift case we actually found in comcastcares and the score-covariate-drift case that traffic didn't happen to exercise but the underlying theory says is possible.
What this means if you're evaluating a verifier for your own traffic
The honest version of this story isn't "verification always decays" — three of four benchmarks showed no decay at all, including datasets with real semantic diversity. It's narrower and more useful than that: decay is a property of non-stationary traffic, not of verification as a technique, and the only way to know whether your own traffic is non-stationary in a way that matters is to watch it, not to assume the answer from a one-time evaluation.
If your traffic looks like comcastcares — long-running, subject to product/topic churn, the kind of workload where "what counts as a correct answer" quietly shifts under you — a one-time Health Check tells you where you start, not where you'll be in six months. That's the entire reason GET /v1/monitor/drift-status exists as an ongoing check rather than a one-off report.
Full methodology, the complete dataset battery, and the chronological-split protocol these numbers come from are on the Research page.