How CacheVerifier decides, and where it can be wrong

CacheVerifier tells your application whether a cached LLM answer fits a new question. Its verdicts come from a statistical model, so some will be wrong. This page sets out what it does, how it decides, how often it is wrong on the data we've measured, what it can't catch, and what you can do about it.

What it does — and what it doesn't

A semantic cache finds a stored answer whose question looks similar to a new one. CacheVerifier checks the borderline matches: given the new question and the cached answer, it scores whether that answer is a correct reply to the new question, and your application serves the hit only when the verdict approves it.

It doesn't check whether the cached answer was true. If the original model call produced a wrong answer, the verifier can approve reusing it. See hallucination memory.
It doesn't know when facts change. A cached price, date or policy can go out of date while still fitting the question; mark such feedback as stale and expire those entries in your cache.
It isn't a content filter. It doesn't judge whether an answer is safe, polite or allowed, only whether it fits the question.
It shouldn't be the only check where a wrong answer can seriously harm someone — medical, legal, financial or safety-critical answers, or decisions about a person. The Terms require human review or a fresh model response there, and the Docs describe how.

How a verdict is made

Model. A small cross-encoder reads the new question and the cached answer together (plus the cached question, if you send it) and outputs a score; the hit is approved when the score clears your workspace's threshold. Clear matches above your own cache's similarity cutoff never reach it.
Starting point. Every workspace starts from the same off-the-shelf model, trained on public data. It is not trained on any customer's data.
Fine-tuning. When you send feedback on verdicts, you can fine-tune a model for your workspace. It is trained and evaluated only on your own feedback, never on another customer's, and its accuracy is reported on held-out feedback it didn't train on.
Updates. Your workspace's fine-tuned model changes only when you fine-tune it or activate a version yourself.

How often it is wrong, on data we've measured

These figures come from our published research and live tests on public datasets. Your error rate depends on your traffic; measure it with randomized audit rather than assuming ours.

Fine-tuning on a workspace's own feedback is what makes the verifier useful; the off-the-shelf model alone adds little on most datasets. Details and per-dataset numbers are on the Research page.
On deliberately misleading phrasing (negations, swapped numbers or names), an off-the-shelf verifier approved 84% of wrong pairs, and fine-tuning on ordinary feedback didn't fix that. Adversarial hardening with the matching settings brought it to roughly 5–16%. More
A certified risk target held on traffic from the same period, but ran 1.4–1.8 times over target on later traffic that had drifted. More
On one long-running customer-support dataset, fine-tuning made the verifier worse once the share of correct matches shifted over time; the drift monitor is built to catch that. More

Known limitations

Deliberately misleading phrasing still gets through sometimes, even with hardening.
Accuracy and any certified bound degrade when your traffic changes; neither updates itself.
Fine-tuning on fewer than roughly 1,000 rows can be a net negative on hard data; more and fresher feedback helps.
Feedback you choose to send is not a random sample, so accuracy measured on it can look better than it is; randomized audit corrects for that.
The public benchmarks behind our research have noisy labels; we have corrected published numbers where that mattered, with dates, on the Research page.

Your data

Your queries, answers and labels are used only to serve and fine-tune your own workspace.
Adversarial hardening is the only feature that sends any of it to an outside AI provider (a sample of queries to OpenAI), and only when you turn it on for a fine-tune job.
You can export everything and delete your account at any time. Retention periods and every sub-processor are listed in the Privacy Policy; the DPA covers processing on your behalf.

Safeguards and defaults

Safeguard	What it does	Default
Python client fails closed	If a verify call times out, can't connect, or gets a server error, the client returns approved = false, so you fall through to your LLM instead of serving an unchecked hit.	On (fail_open=False)
Cold start	Before your workspace has a fine-tuned model, verify only approves hits once a Health Check on your own data shows the stock model is accurate enough. Otherwise it rejects, says why, and shows what the stock model would have decided without acting on it.	On
Held for review	A newly fine-tuned model isn't switched on automatically when one of its checks flags a problem, for example scoring worse than the model you have now. You decide whether to activate it.	On
Randomized audit	Rejects a small random share of approved hits and asks for a label, so your error rate is measured on a fair sample.	On for new workspaces, adjustable
Drift monitor	Flags when your traffic stops looking like the data your model was trained and certified on.	On once you have a fine-tuned model; alerts on paid plans
Certified risk target	Certifies a threshold that bounds expected false-reuse risk while traffic stays like the calibration data. Informational: never applied automatically.	Opt-in
Adversarial hardening	Mixes synthetic hard pairs (negation, number, entity swaps) into training. Sends a sample of your queries to OpenAI to generate them.	Opt-in

Reporting a problem

A wrong verdict: send it as feedback with POST /v1/feedback; it improves your next fine-tune and shows up in your error-rate numbers. For a pattern you think is a product bug, email support@cacheverifier.com.
A security issue: see security.txt.
Something on this page is wrong: tell us at the same address; we'll correct it and note the date.

Last reviewed October 24, 2026.