GPTCache + CacheVerifier: a correctness check for cache hits

GPTCache is the open-source semantic cache for LLM apps. Its SimilarityEvaluation hook is the point where a wrong hit slips through — or gets caught. CacheVerifier drops straight into that hook.

CacheVerifier is a verification layer for semantic caches, not a cache itself. This page is a straight comparison, kept as fair as we can make it — see Why Similarity Fails for the underlying argument.

What GPTCache is

GPTCache (from Zilliz) is the widely-used open-source semantic cache for LLM queries, with built-in LangChain and LlamaIndex integrations. It’s built to be pluggable: you choose the embedding function, the vector store, and the SimilarityEvaluation — the component that decides whether a matched entry is close enough to serve.

The false-positive problem

GPTCache’s own documentation recommends carefully assessing it for false positives on cache hits, and independent comparisons have found it produces significant false hits on non-duplicate queries. The stock SimilarityEvaluation strategies are distance- and embedding-based, so they inherit the gray-zone limitation: a candidate can clear the similarity bar and still be the wrong answer.

The drop-in

CacheVerifierEvaluation implements GPTCache’s SimilarityEvaluation interface. You pass it as similarity_evaluation= in cache.init(...) — no fork, no PR. Gray-zone candidates get a cross-encoder correctness check; obvious hits and misses pass through cheaply.

pip install "cacheverifier[gptcache]"

from gptcache import cache
from cacheverifier.integrations.gptcache import CacheVerifierEvaluation

cache.init(
    embedding_func=...,
    data_manager=...,
    similarity_evaluation=CacheVerifierEvaluation(api_key="cv_..."),
)

Full example: examples/gptcache_example.py in the cacheverifier-python repo.

Side by side

	Stock GPTCache SimilarityEvaluation	CacheVerifier
Decides on	Embedding distance / string similarity	Cross-encoder reading the (query, answer) pair
Gray-zone hits	Served or dropped by threshold	Sent for a correctness judgement
Adapts to your data	Manual threshold tuning	Fine-tune on your feedback (optional)
Setup	Built in	pip install, pass one argument
Cost per lookup	Local compute only	One API call on gray-zone candidates (obvious hits/misses skip it)

When you don’t need it

If your GPTCache traffic is near-duplicate queries with a comfortable similarity margin — FAQ deflection on tightly-scoped phrasing, for instance — the stock evaluation is fine and a verify call is overhead. The adapter earns its place when queries are varied enough that the gray zone is where a real fraction of your hits land.

Where to go next

• Why Similarity Fails — the gray zone in detail: why a similarity score and a correctness judgement are different questions.
• Research — measured results on when fine-tuning a verifier helps, how much data it needs, and where it broke.
• API Reference — how /v1/verify, fine-tuning, and drift monitoring work mechanically.
• Start free