Cache hit rate answers one question: how often was an LLM call skipped. It says nothing about how many of those skipped calls actually served the right answer — the same gap we've written about before, in the context of drift and a certified risk guarantee that overshot its target. This post is about two more specific, mechanically different ways one wrong hit turns into more than one bad answer. Neither shows up as a change in how similar the query looked, which is exactly why tuning a similarity threshold — however carefully — can't catch either of them.
Failure mode one: the cache re-serves its own mistake
A wrong cache hit and a wrong LLM answer look identical from the outside — both are one bad reply. But if the cache also doubles as working memory across turns (a common pattern in multi-turn agent pipelines, where a "confirmed" fact from an earlier turn gets referenced in later ones), a false hit is worse than a one-off mistake: the agent doesn't just answer wrong once, it treats the wrong answer as an established fact and keeps building on it every subsequent turn, the same way it would treat anything else it retrieved successfully. Nothing in the pipeline distinguishes "this came from a verified source" from "this came from an unverified near-miss that happened to score high on similarity." We cover the mechanics of this in more depth in the hallucination-memory section of Why Similarity Fails.
The self-reinforcing version of this is worse still: if the wrong answer itself gets written back into the cache — because the pipeline treats "we didn't get a complaint" as "the answer was fine" — a single bad reply doesn't just compound within one session, it becomes the stored answer for every future query that scores similar to it. The error stops being a one-time mistake and becomes the cache's own ground truth.
Failure mode two: the cache doesn't know whose question it's answering
The second failure mode is older than semantic caching and has nothing to do with embeddings: it's a cache-key design bug. If a semantic cache's similarity search runs over a single shared namespace instead of one scoped per tenant, per user, or per data-access boundary, two different callers asking similar-enough questions can retrieve the same stored answer — even when the correct answer depends on who's asking. A billing question that should resolve differently for two different accounts, a support answer that references one customer's own order details, a RAG lookup over documents two users shouldn't both see — any of these can cross a boundary that a similarity score was never designed to check, because similarity search only ever asks "does this embedding sit close to that one," never "is the asker allowed to see that stored answer at all."
This is a well-understood category of cache bug in general — cache-key scoping mistakes predate LLMs by decades — but a similarity-based cache makes it easier to trigger by accident: an exact-match cache only collides on an identical key, while a semantic cache collides on anything that merely reads as similar, which is a much larger surface if the namespace boundary isn't enforced explicitly in how the lookup is scoped.
Neither is a tuning problem
It's tempting to read both of these as arguments for a stricter similarity threshold. They aren't. Tightening the threshold reduces how often any hit fires, wrong or right — it doesn't teach the cache which hits are wrong, and it does nothing at all about a cache-key scoping bug, which fires regardless of how confident the similarity score is. A threshold answers "how alike do these two pieces of text look"; both failure modes above are actually about a different question — "was this specific stored answer ever validated for reuse, by anyone, for this asker" — that similarity alone was never built to answer.
That's also the shape of what some 2026 write-ups on LLM caching have started calling a verified semantic cache, as distinct from a plain one: restricting reuse to answers that have actually been checked, instead of trusting any hit that clears a similarity bar. It's the same idea CacheVerifier is built around — POST /v1/verify runs a correctness check on the specific (query, candidate) pair before a gray-zone hit gets served, which is a check on the pair itself, not a looser or tighter number to clear. It doesn't fix a cache-key scoping bug on its own — that's an architecture decision your cache backend has to get right — but it does mean a hit that gets through is one that was actually evaluated, not just one that happened to score high enough.
See Why Similarity Fails for the full breakdown of the gray zone this sits inside, and the API Reference for how a verify call works mechanically.