Your semantic cache is probably wrong more often than your dashboards show
CacheVerifier is a semantic cache verification platform for LLM apps. A similarity threshold can't tell a real cache hit from a confident-looking near-miss. Serve one of those, and if an agent treats it as fact, it's a hallucination you never chose.
Gray-zone precision was under 2% in our tests against real support traffic. CacheVerifier adds a synchronous verification call plus fine-tuning on your own feedback — no switching cache backends.
The problem
When semantic caching serves the wrong answer
Same similarity score, opposite verdict — one of these cached answers is right and one is wrong, and a similarity threshold sees them as identical.
Query: "How do I cancel my subscription?"
"Go to Settings > Billing > Cancel subscription."
"Go to Settings > Billing > Pause subscription for a month."
Pausing keeps the subscription active — a different action than canceling.
Both candidates score high on similarity — a plain threshold can't separate them. Illustrative example, not a live API call; try /v1/verify on your own pairs for free.
Fit
Is this your traffic?
Verification helps a specific shape of workload, not every semantic cache. Four things that need to be true — and the honest read if they aren't.
The same question reaches you worded many different ways.
FAQ deflection, support assistants, high-volume RAG — enough of your hits are genuine paraphrases, not near-duplicates, that a similarity score has real work to do.
A wrong reused answer costs more than one extra model call.
The wrong refund policy, cancellation step, or dosage is expensive; a miss just costs a regeneration. If the two are close, raising your threshold for maximum hit rate is the better move.
The gray zone is a real fraction of your hits.
Not near-zero — then there's nothing to verify. Not near-everything — then your threshold is just set wrong. Somewhere in between, a verify call actually changes the outcome.
The verifier can tell your gray-zone hits apart — on your data.
Not something you can eyeball. The free Health Check measures whether the stock model, or a fine-tune on your feedback, is actually more trustworthy than raw similarity in that band.
If a Health Check shows your gray zone is Quora-shaped — where the task's own score ceiling, not your threshold, is the limit — raising the threshold is the right call and you don't need this. We'd rather tell you that before you integrate.
Drop-in
One line in a pipeline you already have
It implements GPTCache's own SimilarityEvaluation extension point — no fork, no PR. Already on LangChain or a self-built Redis lookup? /v1/verify is a plain REST call.
from cacheverifier.integrations.gptcache import CacheVerifierEvaluation
cache.init(
similarity_evaluation=CacheVerifierEvaluation(api_key="cv_..."),
...
)Integrations
How it pairs with your cache
CacheVerifier sits behind the cache you already run — it doesn't replace it. How that looks next to the common options:
Capabilities
What CacheVerifier gives you
Synchronous verification
POST /v1/verify takes a (query, candidate_answer) pair and returns approve/reject with a score and latency — call it inline on any gray-zone hit, whatever cache backend produced it.
Self-serve fine-tuning
Fine-tuning is cheap and fast (seconds to minutes) so it's free to run yourself: POST /v1/feedback accumulates your own correctness signal, POST /v1/finetune/jobs trains a tenant-specific verifier on it.
Drift & stability monitoring
GET /v1/monitor/drift-status is a narrow slice of AI observability for one failure mode: model drift in your verifier's gray-zone positive rate, the kind that quietly turns a fine-tuned model harmful — no extra inference required.
Certified risk budgets
Certify a threshold via Conformal Risk Control: a distribution-free guarantee that risk stays below your target — if later traffic is exchangeable with its calibration data. We found a case where it wasn't. Details below.
ROI, not just a hit rate
Your dashboard turns this period's real verify-call volume and your active verifier's own held-out numbers into calls avoided, wrong cache hits avoided, and — if you tell it what one LLM call costs you — an estimated $ saved. No invented cost assumption on our end.
Gray-zone tuning
GET /v1/monitor/gray-zone-threshold replays your feedback against your current verifier to recommend where your cache's own confidence cutoff should sit — so a verify call only fires on the hits that are genuinely uncertain, not the obvious ones.
Don't want to own the retraining loop?
Verify and fine-tune yourself for free, or let Managed run recalibration on a schedule, watch for drift, and alert you before it hurts precision — no cron job, no on-call for your verifier.
Proof
We tried to break this before shipping it
Three questions the underlying research didn't answer — how noisy feedback can get, how little data is enough, whether gains decay over time — checked on three public benchmarks and real production support traffic, not just one dataset.
Fine-tuning gain (AUC delta) vs. label noise — all three datasets cross zero in the same range
4
datasets, including real support traffic
~30%
label-noise threshold, consistent across datasets
1,000
feedback rows before fine-tuning reliably helps everywhere
Both limits are enforced for you automatically — Health Check fails closed below the AUC bar, and fine-tune jobs hold back instead of activating once label noise clears the danger zone.
Architecture
Where CacheVerifier sits in your system
You keep your cache backend and decide what counts as gray-zone. CacheVerifier only runs on the candidates you route to it, and the learning loop below runs asynchronously — neither ever blocks a request.
Request flow · synchronous
A user asks a question.
GPTCache, LangChain, a Redis lookup — you set the gray zone.
Gray-zone hits only — a confident hit skips this and serves straight from cache.
approved → serve the cached answer
rejected → call your LLM, then re-cache
Async learning loop · never on the request path
Was the served answer actually correct?
Fine-tune job retrains your verifier · drift monitor watches its stability.
Flows back into /v1/verify — the next gray-zone hit uses the updated model.
Setup
How it works
Start free
Register gets you a tenant, a dashboard, and an API key — no card required.
Point your gray zone at /v1/verify
Wherever your cache backend flags an uncertain hit — GPTCache, LangChain, a self-built Redis lookup.
Run a health check
See baseline vs. fine-tuned precision on a sample of your real traffic before committing to anything.
Let us run the loop
Move to managed calibration + drift monitoring once you know it's worth automating.
FAQ
Common questions
Data handling, self-hosting, SLA, fine-tuning requirements and the rest are on the full FAQ.
+What is semantic cache verification?
+Do I have to replace my existing cache?
+Is it really free?
+When is verification worth adding — and when isn’t it?
+Is this the same as prompt caching?
+How much latency does a verify call add?
Self-serve is free forever.
Verify and fine-tuning stay free, always. The Health Check Report is free too. Managed — hosted continuous recalibration — is live starting at $149/mo.