How semantic cache similarity thresholds work (and why tuning them isn't enough)

A semantic cache retrieves the nearest previously-answered query by embedding similarity and then has to decide what to do with it. In practice that's two thresholds, not one:

Most single-threshold setups just collapse τ_low and τ_high into the same number, which means every borderline candidate is forced into either "serve" or "skip" with nothing in between. Widening the gap creates room to handle the uncertain ones differently.

1. 1

   Fix the embedding model first

   A threshold is only meaningful relative to one embedding space. Pick the model your cache will use to compare queries and treat swapping it later as invalidating every number below.

2. 2

   Collect labelled pairs from your own traffic

   A few hundred real (query, cached-answer) pairs, each marked as a correct or incorrect reuse. Synthetic pairs don't predict where your real gray zone sits.

3. 3

   Sweep the cutoff and record both rates

   Step the similarity cutoff across its usable range and, at each point, measure hit rate (how many queries get served from cache) and error rate (how many of those were wrong). You're mapping a trade-off curve, not looking for one magic value.

4. 4

   Set τ_high where the error rate is acceptable

   τ_high is the point above which similarity alone is trusted to serve. Pick it against the error rate your use case tolerates — a billing answer and a documentation pointer don't get the same number.

5. 5

   Set τ_low where candidates are clearly unrelated

   Below τ_low, skip the candidate and call the model. The band between τ_low and τ_high is the gray zone: similar enough to tempt, not similar enough to trust.

6. 6

   Route the gray zone somewhere that reads the pair

   Send only the gray-zone band to a verification step or a fresh model call. This is where a similarity number has run out of information and something has to actually judge the answer.

7. 7

   Re-run the sweep on a schedule

   The right threshold drifts as your traffic changes. A cutoff calibrated six months ago is a guess today — drift monitoring watches for exactly this.

Raising τ_high always works in the narrow sense — fewer borderline candidates get served, so fewer wrong ones do. The real question is what it costs you in hit rate, and whether routing that same gray-zone band through a verifier instead buys the correctness back more cheaply. PAPER.md 5.11 swept τ_high across a full grid and compared verifier-gated reuse to the pure raise-the-threshold frontier, at a matched error rate:

The verifier's edge over just tightening τ_high is real but small, and entirely dependent on how good the verifier is on your data. Where an off-the-shelf verifier already has genuine discriminative power (LmArena, AUC 0.72), routing more of the high-similarity band through it is worth up to ~5pp of hit rate at the same error rate. Where it's near-random (SearchQueries, AUC 0.60), widening the gray zone too far makes things actively worse than the threshold alone. Threshold tuning has a ceiling; so does an untuned verifier.

The table above holds the error rate fixed, which quietly treats a wrong answer and a missed cache as equally bad. They usually aren't. PAPER.md 5.17 redoes the comparison on a single total-cost axis: let r = (cost of one wrong reused answer) ÷ (cost of one extra model call), sweep it, and at each r compare the cheapest "just raise the threshold" operating point to the cheapest "keep the threshold, verify the gray zone" one.

Read plainly: once a wrong answer costs more than about one extra model call — which is almost any support, billing, or policy answer — verification is the cheaper policy on every dataset here. Where a miss is as expensive as a wrong answer or more (r ≲ 1), raise the threshold and aim for maximum hit rate instead. The break-even r for the untuned verifier is ≈ 0.7 on LmArena, ≈ 5 on Quora, with a window around 1.4–4 then ≈ 9+ on SearchQueries.

If a Health Check shows your gray zone is Quora-shaped — the verifier is a rounding error at every τ_high because the task's own score-separability ceiling, not threshold choice, is the binding constraint — then raising the threshold is the right call and a verification layer isn't worth the dependency. This page is not an argument that everyone needs a verifier; it's an argument for measuring which case you're in before deciding.