Why Similarity Fails

Every semantic cache picks candidates by similarity — cosine similarity over embeddings, usually. Above some threshold, a candidate is confident enough to serve without a second look; below some lower threshold, it's clearly unrelated and gets skipped. In between is the gray zone: candidates similar enough to be tempting, not similar enough to be certain.

The gray zone isn't an edge case. It's where a cache backend's own similarity score stops being able to answer the question that actually matters — not "how alike are these two pieces of text," but "does this candidate correctly answer this query, right now." Two candidates can score identically on similarity and split completely on correctness — see the live example on the homepage. A plain threshold, no matter how carefully tuned, can't separate them: it's answering a different question than the one that decides whether serving the cached answer was the right call.

This is the specific gap a verification step closes — not by picking a better threshold, but by asking a model to look at the actual (query, candidate) pair and judge correctness directly, on exactly the candidates where similarity alone couldn't decide. See POST /v1/verify for the mechanics.

Not every gray-zone mismatch looks the same. An LLM red-teaming pass (PAPER.md 5.18) generated adversarial candidates designed to score high on similarity while being wrong, across a handful of named categories — negated meaning ("can I cancel" vs. "can I not cancel"), entity substitution (the right structure, the wrong product/plan name), and quantity substitution (the right sentence, a different number or amount). Pooled across categories, the verifier let a striking 84% of these adversarial candidates through — a real finding, worth taking seriously.

But "worth taking seriously" isn't the same as "safe to operationalize without checking." We went back and tested the two easiest-to-automate categories — negation and number mismatch — against real historical feedback from two actual tenants (1,190 rows each, AmazonHelp and LmArena support traffic), asking two separate questions: does this pattern show up often in real traffic, and when it does, does it actually predict a higher error rate the way the red-team data implied?

The patterns themselves are common in real traffic — far from rare edge cases (number mismatch alone was 77.9% of LmArena's gray-zone rows). But negation mismatch didn't predict a higher error rate on either real dataset — the direction was backwards on both. The likely explanation: an LLM generating adversarial examples uses negation deliberately, to flip meaning on purpose ("can" vs. "can't"). Real-world negation mismatches are mostly incidental wording differences that don't change what's actually true — a different phenomenon wearing the same label. Number mismatch held up better — a real signal on LmArena, inconclusive on AmazonHelp (whose baseline error rate was already near-ceiling for unrelated reasons, leaving no room for the pattern to show a gap either way).

A wrong cache hit and a wrong LLM answer look similar from the outside — both are one bad reply. But if the cache also functions as an agent's working memory (a common pattern in multi-turn agent pipelines, where a "confirmed" fact from an earlier turn gets referenced in later ones), a false hit is worse than a one-off mistake. The agent doesn't just answer wrong once — it treats the wrong answer as an established fact and keeps building on it every subsequent turn, the same way it would treat anything else it retrieved successfully. There's no signal in the pipeline distinguishing "this came from a verified source" from "this came from an unverified near-miss that happened to score high on similarity."

That compounding is what separates a false cache hit from an ordinary wrong answer: the caller has to actively notice and correct it before it propagates, instead of it just being one bad response the user moves past. Cache hit rate — the metric most dashboards actually show — measures how often an LLM call was skipped. It says nothing about how many of those skipped calls were skipped correctly.

The same question can be asked many surface-different ways without changing what a correct answer looks like — "how do I cancel" and "what's the process to end my subscription" should both match the same cached answer. A similarity-based cache has to generalize across that surface variation to be useful at all — too literal a match, and semantically identical queries phrased differently miss the cache entirely, defeating the point of semantic caching in the first place.

The open question is whether the SAME generalization that correctly collapses paraphrases together also collapses queries that are close in surface form but meaningfully different in what they're actually asking — and whether that failure mode is distinguishable, in practice, from the entity/number substitution patterns in Section 2, or is really the same underlying phenomenon described from a different angle. We don't have a real-data answer to that yet.

A wrong answer to "how do I reset my password" costs a support ticket. A wrong answer to "what's covered under my policy" or "what dose should I take" costs a lot more. Treating every gray-zone candidate with the same risk tolerance — one threshold, one cost assumption, applied uniformly across every kind of question a tenant's traffic contains — is a simplification, not a design choice made because it's correct.

CacheVerifier's answer to this today is global, not per-bucket: cost_ratio lets a tenant express how many times more expensive a wrong approval is than a missed cache, and picks one threshold for the whole account by minimizing expected cost at that ratio instead of treating a false approval and a miss as equally bad by default. That's a real, shipped improvement over a single one-size-fits-all threshold — but it's still one number for the whole tenant. Splitting that same idea by domain — a stricter effective threshold for "billing dispute," a looser one for "where's your documentation" — isn't built. It would need a caller-supplied domain/intent tag on every verify and feedback call (a new API contract, not just a new dashboard number) and enough data volume in each bucket to calibrate it separately without the small-sample problems Section 2's per-category numbers already ran into.