Prompt caching vs semantic caching: which one you actually need

Prompt caching makes a repeated request cheaper by not re-processing a prompt prefix the provider has already seen. Semantic caching skips the model call entirely when a new, differently-worded request is judged close enough to one you've already answered. The first is exact-match and can't be wrong; the second matches on meaning and sometimes is.

Prompt caching (Anthropic prompt caching, OpenAI's automatic prompt caching, Gemini context caching) is a provider-side optimization. When consecutive requests share a long identical prefix — a big system prompt, a fixed tool schema, a document you're asking several questions about — the provider keeps the computed attention state for that prefix and charges a reduced rate to reuse it instead of recomputing it.

It fires only on a byte-for-byte (or provider-defined near-exact) prefix match. The response is still generated fresh by the model every time. There is no similarity judgement, no candidate answer being reused, and therefore nothing that can be "a wrong hit" — the worst case is a cache miss and the normal price.

Semantic caching (GPTCache, Redis LangCache, a self-built Redis or vector-store lookup) sits in your application. It embeds each incoming query, searches for a previously-answered query whose embedding is close, and — if one is close enough — returns that stored answer without calling the model at all. "How do I cancel" and "what's the process to end my plan" can hit the same cached answer even though the text differs.

That's the point of it, and also the risk: embedding similarity measures how alike two questions look, not whether an old answer is still a correct, current answer to the new one. Two candidates at the same similarity can split on correctness — one right, one a stale detail or an adjacent-but-different intent. The band where that happens is the gray zone, and it's what POST /v1/verify exists to check.

You want prompt caching whenever you send a large, stable prefix on many calls — a long system prompt, few-shot examples, a document under repeated questioning. It's close to free to turn on, there's no downside, and for most providers it's on by default or a single flag. There is nothing to evaluate.

You want semantic caching when the same question reaches you in many different wordings and a full model call per phrasing is the cost you're trying to cut — FAQ deflection, support assistants, high-volume RAG. This one you do have to evaluate: how often your gray zone is actually reached, and how often a hit there is wrong on your traffic. That's what a Health Check measures.

They're not alternatives. A support assistant can serve a differently-worded repeat question straight from a semantic cache (no model call), and on the questions that do reach the model, prompt caching discounts the shared system-prompt prefix. Verification only applies to the semantic side — a prompt-cache hit is exact by construction and has nothing to check.

A KV cache is neither of these. It lives inside a single model's forward pass, holding the attention key/value tensors for tokens already processed so the next token in the same context doesn't recompute them. It's an inference-time detail of one call, not a cross-request response store, so — like prompt caching — there's no reuse ambiguity for it to get wrong. If a doc says "KV cache," it's talking about model internals, not a caching layer you'd verify.