Conformal risk control overshoot in a semantic cache

POST /v1/finetune/jobs?target_risk=0.01 certifies a threshold using Conformal Risk Control (CRC) — a distribution-free, finite-sample guarantee that accepting only candidates scoring above the certified threshold keeps false-reuse risk at or below your target, provided future traffic resembles what it was calibrated on. It's not a best-effort estimate; it's a real statistical guarantee, with a proof behind it.

It's also a guarantee that comes with an assumption baked into its name: provided future traffic resembles what it was calibrated on. We went looking for exactly how much that assumption costs when it's violated, because "exact under ideal conditions" and "exact in practice" are different claims, and shipping a feature called a guarantee without checking the gap between them felt like the wrong place to take a shortcut.

Two ways to test a guarantee, and why we don't use the easy one

The textbook way to validate a conformal guarantee is a random split: shuffle the pool, hold out a random subset, certify against the rest. Under a random split, CRC's guarantee on our data held essentially exactly — at under 3% utility cost versus an oracle that somehow knew the ground truth in advance. That's a genuinely strong result, and it's the version of the experiment the underlying theorem is actually proven for.

It's also not how a real deployment works. A real tenant's calibration data is history — everything fine-tuned and certified so far — and the traffic the certified threshold then has to hold up against is the future. That's a chronological split, not a random one, and a chronological split is where a guarantee that assumes stationarity actually gets tested against non-stationarity, the same way the drift story from our previous post is about traffic that changes shape over time rather than traffic sampled uniformly from one fixed distribution.

So that's the protocol we actually certify under, and it's a harder bar than the one the theorem strictly covers. We think that's the right choice for a number people are going to rely on. It's also the choice that produced an uncomfortable result.

What the chronological test actually showed

Across our benchmark battery, the chronological-split result was mixed in an informative way:

• On LmArena, the guarantee held — realized false-reuse risk stayed at or below the certified target.
• On SearchQueries, it overshot by roughly 20-25% — a real gap, traced to covariate drift in the verifier's own score distribution over the chronological window (the same kind of shift the KS-test drift detector described in our last post exists to catch going forward).
• On Quora, the apparent overshoot looked considerably larger than either of those — large enough, on its own, to be the headline result of the whole experiment.

That third number is the one we didn't publish as-is. It's the one we went back and checked.

The overshoot that wasn't real drift

Three things about that result didn't hold up when we looked closer:

• Quora's "stream order" isn't time. It's the order in which each question first appears in the QQP training file. A chronological split of that order is a split by file position, not a calibrate-on-the-past, test-on-the-future split.
• The difficulty gradient behind the overshoot comes from the labels. The overshoot is driven by the gray-zone error rate climbing steadily with file position. Under the original QQP labels, the early-vs-late gap is +0.16. We had an independent LLM judge (DeepSeek) relabel the same pairs, and under those labels the gap is +0.01 — the gradient essentially disappears.
• The verifier's own scores never moved. Its mean score is 3.59 at the start of the stream and 3.56 at the end. Real covariate drift, the kind SearchQueries shows, moves the score distribution. Quora's doesn't.

Put together, the overshoot comes from a batch effect in how Quora Question Pairs was crowd-annotated, not from anything changing over time. It's not a property of Conformal Risk Control, and not evidence that the guarantee fails harder on Quora-shaped traffic than the SearchQueries result already showed. The corrected picture: one real, moderate overshoot (SearchQueries, ~20-25%, from a genuine covariate drift in the verifier's score distribution, mean 5.57 → 6.04), and one that was a labeling problem in the benchmark, not a modeling one.

Correction, 2026-10-06: an earlier version of this section blamed the Quora result on duplicate pairs concentrating unevenly across the split. That's not what the analysis in Paper D §4.2 found; the explanation above is.

What we actually shipped because of this

A few things changed as a direct result of running this check rather than trusting the random-split number:

• certified_pool_candidate_concentration now ships on every certified fine-tune job — the fraction of the certification pool sharing its single most-repeated candidate answer. Above roughly 20%, certification_note adds an explicit caveat. It's a separate check from the Quora finding: CRC assumes the certification pool is representative, and one answer dominating it usually means the feedback was self-selected — a condition worth flagging on any tenant's own data.
• certification_status distinguishes a certification that's still trustworthy from one that isn't: stale_recalibration_pending once any drift detector flags (see the previous post), superseded once a newer model is active. A number computed correctly six weeks ago isn't automatically still correct today, and the field says so instead of implying otherwise by silence.
• The online adaptive threshold work in Paper D (ACI, restoring tracking under drift instead of just detecting it) exists specifically because "the guarantee overshoots under real drift" is a finding you either build a response to or you don't — detection alone doesn't fix the SearchQueries gap, it just tells you it's there.

Why we're publishing the number that made us look worse

A 20-25% overshoot on one of three benchmarks isn't the result we'd have picked if we were choosing which number to feature. We're publishing it — and the fact that it took a second pass to find that the larger apparent overshoot on a different dataset wasn't real — because a risk guarantee that's only checked once, under the easiest protocol, isn't one we'd trust from someone else. certified_threshold on /v1/finetune/jobs is informational, never auto-applied to /v1/verify, precisely so a number like this can carry an honest caveat instead of a silent one.

Full protocol, both split methodologies, and the complete numbers are on the Research page.