Does semantic cache verification actually work?

Production feedback (thumbs up/down, ticket reopens) is noisier than a research dataset's ground truth. We took a fixed 4,000-example slice per dataset and randomly flipped labels at six target noise rates, always evaluating against the untouched, clean held-out test set.

All three curves decay smoothly and now cross zero in nearly the same range — ~32% noise (32.2% LmArena, 32.1% SearchQueries, 31.4% Quora; Core paper §5.7). The shape holds across datasets, so we treat this as a conservative red line: below it, self-serve fine-tuning is safe by default; above it, the service now holds the new model back automatically instead of just warning — a job whose train_label_disagreement_rate clears a 30% cutoff comes back model_status: "held_for_review" rather than auto-activating (see the docs). A Health Check Report — a dry run against your own data, nothing deployed — is still the cheapest way to check where you land before running a real job.

A new tenant starts with zero labels. We fine-tuned independent models on increasing prefixes of each dataset's training stream — from 50 examples up to the full set — to find the point where it reliably helps rather than hurts.

All three curves keep climbing with more data — no plateau, even at the full training set. But below roughly 1,000 examples, SearchQueries — the dataset with the weakest untuned baseline — shows fine-tuning can actually be negative, not just "smaller gains"; Quora dips close to zero around 500 examples but stays positive, and LmArena is positive from the smallest sample size we tested. The risk is concentrated in domains where the untuned baseline is already weak. 1,000 examples is the smallest point where every dataset we tested turned reliably positive, so that's our default minimum before we recommend switching on fine-tuning.

We split each dataset into 8 chronological chunks, fine-tuned a fresh model on each chunk as an "anchor," then evaluated every anchor against every chunk — including ones far in its future — and averaged the gain over baseline by time distance.

After the initial same-window drop (distance 0→1, a memorization effect, not real generalization), LmArena and Quora both stay flat out to a distance of 7 chunks — no sign of decay. SearchQueries doesn't: its own baseline accuracy declines gradually over the same window, and the fine-tuning gain decays alongside it — down roughly a third from distance 1 to distance 7. That's real but mild drift, nowhere near the catastrophic decay in the Comcast counter-example below. Two of three benchmarks argue against needing high-frequency retraining; the third is a reminder to actually monitor for drift rather than assume it away.

Fine-tune jobs can certify a threshold via Conformal Risk Control (Angelopoulos et al., arXiv 2208.02814): a distribution-free, finite-sample guarantee that accepting only candidates above the certified threshold holds false-reuse risk at or below a target you set. Paper B tests how much that guarantee is worth, in two regimes. (Today, certification is informational — it reports the threshold that would satisfy your target, separately from whatever threshold /v1/verify actually enforces: Youden’s J by default, or a cost-sensitive pick — see Question 6 below.)

Under a random calibration/test split, the guarantee is exact and nearly free. Across all three benchmarks and four risk targets (5% down to 0.5%), realized risk lands almost exactly on target, at under 3% utility cost versus an oracle that knows the full score distribution — shrinking further with more data. A side finding: a verifier’s AUC ranking does not predict its usefulness under a fixed risk budget — SearchQueries has the lowest AUC of the three (0.60) yet certifies a higher reuse rate than Quora (AUC 0.63) at every target.

Under a chronological split — calibrate on past traffic, deploy on future traffic, which is what a real tenant does — it depends on how the traffic drifts:

• • LmArena: the guarantee held, conservatively, at every target.
• • SearchQueries: a real but mild overshoot (~20–25% above target), from a slow covariate drift in the verifier’s own score distribution as the cache fills — the detectable kind.
• • Quora: every target overshot, badly (up to 80% at the tightest). But Paper D re-checked this with an independent judge relabel and found the “drift” was an artifact of how the QQP dataset was assembled — its record order isn’t time, and the apparent difficulty gradient vanishes under fresh labels. Not a real production-drift failure.

The fix for the real case is an online adaptive threshold. Paper D adapts the certified threshold online in stream order (Adaptive Conformal Inference; Gibbs & Candès, 2021): realized risk returns to target at all four levels, at a modest hit-rate cost (16.4% → 13.4%). On genuinely drifting production traffic (the Comcast stream below), it showed no clear advantage over a static threshold — an honest null. Practically: a certified threshold is not good forever once computed, which is exactly why a certification’s certification_status tracks stale_recalibration_pending off the same drift monitor described below.

Every experiment above inherits one convention from vCache's benchmark harness: every request gets written into the cache, hit or miss. A real system doesn't — a hit reuses an existing entry and writes nothing. Paper B (Formal Risk Control) §4.2 tests what that changes.

When the cache only grows on misses, its own past decisions shape which candidates are available later: an answer that keeps getting successfully reused never gets its own entry written, so future near-duplicates fall back on some other, less-exact entry that still happens to clear the verifier. This is a self-selection feedback loop, and its harm scales precisely with how often the cache is hit directly:

• • Quora (3.7% direct-hit rate): no measurable effect.
• • SearchQueries (28%): a moderate, statistically significant increase in realized risk.
• • LmArena (73%): realized risk more than triples.

Online recalibration compensates — up to a point. On Quora and SearchQueries it fully offsets the harm (and then some). On LmArena it helps but doesn't fully close the gap. And the obvious cheaper mitigation — occasionally rewrite popular entries instead of always — is a worse bargain than it looks: rewriting on 3 of every 4 hits still only removes about a third of the residual harm. The higher your cache's direct-hit rate — exactly the profile of a high-value, high-redundancy workload — the more a certified risk bound can drift looser than measured, and the more a recalibration cadence matters.

Every threshold above this point was picked via Youden's J — the operating point maximizing true-positive rate minus false-positive rate, which implicitly treats a wrong approval and a missed cache as equally costly. For most real businesses they aren't: a wrong answer served with a cache hit's confidence — the wrong refund policy, the wrong cancellation step — usually costs more than the one extra LLM call a miss costs. Paper B (Formal Risk Control) §4.3 swept a cost ratio r (how many times more expensive a wrong approval is than a miss) up to 1,000× across three datasets and found the cost-minimizing operating point moves meaningfully away from Youden's J's once r departs from 1 — so a large gap between a cost-sensitive pick and the default isn't a bug. A 2026-09-10 follow-up measured, on real calibration data, how sensitive the actual picked threshold is to r itself, and the answer is lopsided: wide, near-tied-threshold plateaus really do exist, but almost entirely below r≈1 or above r≈5–20 — the ranges where the win/loss question above is already settled either way. In the range most businesses' true r plausibly falls into — roughly single digits to the teens — the picked threshold changes densely (25–55 distinct thresholds within about one decade of r, across the datasets tested, a ~1–7% step), so a rough order-of-magnitude guess isn't enough to land on the right threshold in that range.

Fine-tune jobs now report threshold_by_cost_ratio — what threshold would be picked at r = 0.5/1/2/5/10 on your own calibration data — and accept an optional cost_ratio to pick the LIVE threshold that way instead of Youden's J. This is Phase 1 and 2 of a 3-phase rollout: the numbers are surfaced (Phase 1) and opt-in cost-sensitive selection works end to end (Phase 2). The Fine-tune Jobs dashboard also turns that same fixed grid into a plain-language picker ("a wrong answer is roughly twice as costly as a missed cache") with a one-click way to carry a preset row into your next job — a lighter-weight step toward Phase 3's original goal, not that goal itself: a true guided flow (ask what a wrong answer and an escalation roughly cost you, derive a custom r from the answer instead of picking from five fixed presets) isn't built yet. The threshold-sensitivity finding above makes that gap more consequential, not less — if picked thresholds really did sit on wide plateaus throughout, a rough five-preset guess would be good enough; since they don't in the range that matters, a derived r is worth more than the preset picker can offer. Paper B §4.3 itself is explicit that no real business's actual r has been measured — the research validates the mechanism, not any specific ratio; see the docs for how to use it on your own data.

The cheapest defense against a wrong cache hit isn't a verifier at all — it's turning up your cache's similarity threshold so fewer borderline candidates get reused. No latency, no dependency, no data leaving your box; you just lose some hit rate. Any honest case for adding a verifier has to clear that bar: at a matched error rate, does routing the gray zone through a verifier buy back more hit rate than simply tightening the threshold would have? The Core paper §5.11 swept the upper similarity cutoff τ_high (the point above which similarity alone is trusted) across a full grid and compared each result to the pure raise-the-threshold frontier on the same traffic. (For a standalone walkthrough of how those cutoffs work and how to set one, see Semantic cache similarity thresholds.)

The verifier's edge over just tightening the threshold is real but small, and entirely dependent on how good the verifier is on your data. On LmArena, where the off-the-shelf verifier already has genuine discriminative power (AUC 0.72), it's worth about +1.4pp of hit rate at the standard operating point (τ_high = 0.97, the table's own column — the paper's earlier fair-comparison estimate was +1.9pp, the gap is resampling noise) — and up to ~5pp only if you also push τ_high to 0.99, an aggressive setting that hands the verifier the whole high-similarity band and, per the row below, backfires on a weaker-verifier domain. On Quora it's a rounding error at every setting — the QQP task's own score-separability ceiling, not threshold choice, is the binding constraint. And on SearchQueries, where the untuned verifier is near-random (AUC 0.60), widening the gray zone too far (τ_high = 0.99) makes things actively worse than the threshold alone would have — the verifier introduces more bad calls than it prevents. This is the same message as Question 2 and the counter-example below, in a different metric: the verifier only earns its place when it's measurably more trustworthy than raw similarity on your traffic, which is exactly what a Health Check measures and what fine-tuning (Question 2) is for. The corollary for pricing and adoption: if a Health Check shows your gray zone is Quora-shaped, raising the threshold is the right call and you don't need this.

And on total cost, not just the hit-rate / error-rate trade-off

The table above holds the error rate fixed and compares hit rate — which still treats a wrong answer and a missed cache as equally bad. They aren't. Paper B §4.3 redoes the comparison on a single total-cost axis: let r = (cost of one wrong reused answer) ÷ (cost of one extra LLM call), sweep it from 0.1 to 500, and at each r take the cheapest operating point for "just raise the threshold" and the cheapest for "keep the threshold, verify the gray zone" — then ask which policy costs less. Same held-out traffic, same datasets.

The break-even r is where the verifier stops costing money and starts saving it — ≈ 0.7 on LmArena, ≈ 1.4–4 (a window) then ≈ 9+ on SearchQueries, ≈ 5 on Quora, all for the untuned verifier. Read plainly: once a wrong answer costs more than about one extra model call — which is almost any support, billing, or policy answer — verification is the cheaper policy on every dataset here. Where a miss is as expensive as a wrong answer, or more (r ≲ 1), you should just raise the threshold and aim for maximum hit rate. This is the curve a buyer actually needs: not "the model is more accurate" (it is, by AUC) but "at your cost ratio, does the accuracy pay for the dependency."

Stated limitations from Paper B §4.3: r is a free parameter — this has never been measured for a real business, so every number above is "if your r lands here," not a claim about where it does. The cost model is linear (cost(r) = r · error_rate + (1 − hit_rate)); it assumes a run of wrong answers costs a simple multiple of one, which the reputational version of the cost probably isn't.

Synchronous verification runs on the request path, so it costs latency — but only on gray-zone candidates, not every request. Measured server-side on the production no-GPU box: the verifier's model forward pass is ~17ms; the full server stack (auth, tenant lookup, serialization, reverse proxy, no external network) is p50 ~33ms / p95 ~44ms / p99 ~47ms. Deployed in the same region as your cache, expected added latency is ~9–19ms per request.

That figure excludes the client-to-origin network hop, which depends on deployment topology. One real data point: 150 calls from a US-East client (Virginia) to the production API measured a full round trip of p50 ~33ms / p95 ~39ms (2026-09-09) — nearly identical to the server-only figure, i.e. network overhead was close to zero for that specific US-domestic path. That's a single geography and a single run, not a claim that network overhead is negligible everywhere — a client on another continent, or not co-located with our CDN egress path, will see a different number.

An LLM-judge fallback — used by asynchronous designs like Krites — is far heavier: a real API-class judge measured a mean ~1.7s, though that's an upper bound over an international network path with cold connections. This is the cost a scoring cross-encoder avoids by staying a scoring model, not a generative one. (Core §5.5; the FAQ has the current production numbers.)

Every number above is measured on natural benchmark traffic — hard cases the datasets happen to contain, not cases built to beat the verifier. Paper C builds them on purpose: an LLM generates query pairs that look nearly identical but need opposite answers, across five axes — negation ("can I" vs. "can I not"), quantity swap, direction reversal ("convert A to B" vs. "B to A"), named-entity swap, and action-verb swap ("cancel" vs. "pause").

On 306 adversarial pairs that clear the similarity gate, the off-the-shelf verifier false-accepts 84.0% (95% CI [79.7%, 87.9%]) — it confidently approves an answer that is the literal opposite of correct. Negation is the worst axis (96%), consistent with the base model never being trained for polarity.

In-domain fine-tuning — the remedy that repairs natural-data accuracy — gives no protection here (87.6%, confidence interval almost fully overlapping). Learning a dataset's natural noise distribution and learning robustness to constructed attacks are different capabilities.

Targeted adversarial training helps, partially. Mixing a small batch of adversarial examples (3.8% of the training set) covering the five known axes drops the false-accept rate to 53.6%, with no loss of natural-data AUC. Paper D pushes further — a word-level diff of what changed between the two questions, plus swapping the base model to one pretrained on natural-language inference rather than passage ranking — reaching 5.9% on LmArena, 3.6% on Quora, 15.7% on SearchQueries. Named-entity swap is the one axis no amount of training data moved; only the pretraining-objective change did.

The honest summary: this is mitigated, not solved. A deployment with real adversarial exposure — a public support bot where users may phrase things to get the answer they want — should not rely on natural-data accuracy numbers alone.

A verifier is only worth adding if cheaper alternatives don't do the job. Several don't:

• • Bucket queries by their action verb — a popular suggestion: forbid "cancel" and "pause" from sharing a cache bucket. Tested on three datasets, it never raised discriminative power, and on LmArena it caused a statistically significant regression (−0.098 AUC). It screens out the easy negatives and leaves the verifier facing only the hard ones — same verb, opposite object direction. (Core §5.10)
• • Swap in a bigger or more broadly trained off-the-shelf reranker. A 12-layer model and a different-lineage reranker both produced near-identical, small gains (about +0.02 AUC each) on the dataset that separates the two variables — far short of what fine-tuning on your own data reaches. (Core §5.12)
• • Rewrite a rejected candidate to fit the new query instead of calling it a miss (the TweakLLM idea). The rewritten answers were judged wrong 22–61% of the time, and whether a rewrite succeeded was nearly unrelated to whether the rejection was justified in the first place. (Core §5.13)
• • Use the verifier's own confidence to decide when to escalate to an LLM judge. Once you compare against a threshold already tuned to your cost ratio, the score's distance to that threshold carries no exploitable structure. (Paper D §4.1)
• • Add a per-request "is this still in-distribution" gate on the risk certificate. The distribution shift on real streams is too diffuse to localize to individual requests — and the gate rejects correct reuses slightly more often than errors. (Paper D §4.2)

The one free improvement that did work: feeding the similarity score — which the gray-zone gate normally discards once the verifier runs — back into the decision as a second input. It's a measurable Pareto gain, but only where the verifier's own signal is weak (+1.7pp hit rate on SearchQueries; −1.1pp on LmArena, where the verifier is already strong and the extra parameter mostly adds estimation noise). A noise-injection experiment confirms the direction is causal. Same lesson as every question above: the verifier earns its place only when it's measurably more trustworthy than raw similarity on your traffic. (Paper D §4.5)