Our own risk guarantee overshot its target — here's what we found when we checked

POST /v1/finetune/jobs?target_risk=0.01 certifies a threshold using Conformal Risk Control (CRC) — a distribution-free, finite-sample guarantee that accepting only candidates scoring above the certified threshold keeps false-reuse risk at or below your target, provided future traffic resembles what it was calibrated on. It's not a best-effort estimate; it's a real statistical guarantee, with a proof behind it.

It's also a guarantee that comes with an assumption baked into its name: provided future traffic resembles what it was calibrated on. We went looking for exactly how much that assumption costs when it's violated, because "exact under ideal conditions" and "exact in practice" are different claims, and shipping a feature called a guarantee without checking the gap between them felt like the wrong place to take a shortcut.

Two ways to test a guarantee, and why we don't use the easy one

The textbook way to validate a conformal guarantee is a random split: shuffle the pool, hold out a random subset, certify against the rest. Under a random split, CRC's guarantee on our data held essentially exactly — at under 3% utility cost versus an oracle that somehow knew the ground truth in advance. That's a genuinely strong result, and it's the version of the experiment the underlying theorem is actually proven for.

It's also not how a real deployment works. A real tenant's calibration data is history — everything fine-tuned and certified so far — and the traffic the certified threshold then has to hold up against is the future. That's a chronological split, not a random one, and a chronological split is where a guarantee that assumes stationarity actually gets tested against non-stationarity, the same way the drift story from our previous post is about traffic that changes shape over time rather than traffic sampled uniformly from one fixed distribution.

So that's the protocol we actually certify under, and it's a harder bar than the one the theorem strictly covers. We think that's the right choice for a number people are going to rely on. It's also the choice that produced an uncomfortable result.

What the chronological test actually showed

Across our benchmark battery, the chronological-split result was mixed in an informative way:

• On LmArena, the guarantee held — realized false-reuse risk stayed at or below the certified target.
• On SearchQueries, it overshot by roughly 20-25% — a real gap, traced to covariate drift in the verifier's own score distribution over the chronological window (the same kind of shift the KS-test drift detector described in our last post exists to catch going forward).
• On Quora, the apparent overshoot looked considerably larger than either of those — large enough, on its own, to be the headline result of the whole experiment.

That third number is the one we didn't publish as-is. It's the one we went back and checked.

The overshoot that wasn't real drift

The Quora Question Pairs dataset, as commonly distributed, has a structural property that most similarity-cache benchmarks don't have to think about: a meaningful share of its (query, candidate) pairs are duplicates or near-duplicates of pairs used elsewhere in the pool, an artifact of how the original dataset was assembled for a duplicate-question-detection task, not a semantic-cache-reuse task. Under a random split that's mostly invisible — duplicates scatter evenly across the split. Under a chronological split, it isn't: the assembly artifact concentrates unevenly across the calibration/test boundary in a way that looks, if you don't check for it, exactly like covariate drift.

When we traced the large Quora overshoot back to its source, that's what it was — a dataset-assembly artifact in how Quora Question Pairs was built, not a property of Conformal Risk Control, and not evidence that the guarantee fails harder on Quora-shaped traffic than the SearchQueries result already showed. The corrected picture: one real, moderate overshoot (SearchQueries, ~20-25%, attributable to genuine covariate drift) and one that was a measurement problem, not a modeling one.

What we actually shipped because of this

A few things changed as a direct result of running this check rather than trusting the random-split number:

• certified_pool_candidate_concentration now ships on every certified fine-tune job — the fraction of the certification pool sharing its single most-repeated candidate answer. Above roughly 20%, certification_note adds an explicit caveat. This doesn't fix Quora's specific artifact; it flags the general condition that made it possible to miss, on any tenant's own data.
• certification_status distinguishes a certification that's still trustworthy from one that isn't: stale_recalibration_pending once any drift detector flags (see the previous post), superseded once a newer model is active. A number computed correctly six weeks ago isn't automatically still correct today, and the field says so instead of implying otherwise by silence.
• The online adaptive threshold work in Paper D (ACI, restoring tracking under drift instead of just detecting it) exists specifically because "the guarantee overshoots under real drift" is a finding you either build a response to or you don't — detection alone doesn't fix the SearchQueries gap, it just tells you it's there.

Why we're publishing the number that made us look worse

A 20-25% overshoot on one of three benchmarks isn't the result we'd have picked if we were choosing which number to feature. We're publishing it — and the fact that it took a second pass to find that the larger apparent overshoot on a different dataset wasn't real — because a risk guarantee that's only checked once, under the easiest protocol, isn't one we'd trust from someone else. certified_threshold on /v1/finetune/jobs is informational, never auto-applied to /v1/verify, precisely so a number like this can carry an honest caveat instead of a silent one.

Full protocol, both split methodologies, and the complete numbers are on the Research page.