# CacheVerifier

> Hosted semantic cache verification: a REST API that checks gray-zone cache
> hits (the similarity range where a plain threshold match might be wrong)
> and lets you fine-tune a verifier on your own feedback data. Backend-agnostic
> by design -- drop-in with GPTCache, LangChain, or a self-built Redis cache.

CacheVerifier does not replace a semantic cache or do similarity search
itself. Callers run their own cache lookup first, then call `/v1/verify` on
the candidates where similarity alone isn't a confident enough signal (the
"gray zone"). Self-serve verify and fine-tuning are free forever; Managed
plans add hosted, scheduled recalibration on top, plus drift monitoring --
on both the gray-zone positive rate and the verifier's own score
distribution -- that now triggers an out-of-cycle recalibration
automatically when flagged, not just an alert.

Who it helps: verification pays off for a specific shape of workload, not
every semantic cache. Roughly: (1) the same question arrives worded many
different ways, so a real fraction of hits are paraphrases rather than
near-duplicates; (2) a wrong reused answer costs meaningfully more than one
extra model call (r >= ~1-9, dataset-dependent, in Paper B section 4.3's cost model) -- below that,
raising the similarity threshold for maximum hit rate is the better move;
(3) the gray zone is a real fraction of hits (not near-zero, not
near-everything); (4) the verifier is actually more discriminative than raw
similarity on the caller's own domain (measured by the free Health Check --
not something to assume). Where a Health Check shows the gray zone is
"Quora-shaped" (the task's own score-separability ceiling, not threshold
choice, is the binding constraint), the honest answer is to raise the
threshold and not adopt this. Don't pitch it as universally useful.

Not the same thing as provider-side prompt caching (Anthropic's, OpenAI's),
which reuses an exact-or-near-exact prompt prefix to skip redundant token
processing and has no ambiguity to verify. Semantic caching matches on
meaning, not exact text, which is what creates the gray zone in the first
place -- a system can use both layers at once.

Also not the same thing as a KV cache (the attention key/value tensors an
LLM caches within its own forward pass to avoid recomputing already-seen
tokens): that's an inference-time optimization inside one model call, not a
cross-request response cache, so there is no reuse ambiguity for it to get
wrong. Semantic caching reuses a previous response across separate requests
based on query similarity -- a different layer, a different failure mode.

Not a RAG evaluation framework either (Ragas, Langfuse, Braintrust, etc.):
those score retrieval/generation quality offline against an eval set.
CacheVerifier runs synchronously on live traffic, deciding per-request
whether a specific gray-zone cache hit is safe to serve -- complementary to
offline RAG evaluation, not a replacement for it.

Similarity and correctness are different signals: a candidate can score high
on similarity and still be the wrong answer (e.g. a stale policy detail, or
an adjacent-but-different user intent like "cancel" vs. "pause"). A plain
similarity threshold can't tell these apart -- that's the gap CacheVerifier
checks, and why a false hit that gets treated as ground truth by an
LLM-backed agent (rather than corrected) functions as a self-inflicted
hallucination, not just a wrong answer.

## Project

- Developer: Chengyou Xin / LoopDot AI Research. Wikidata entities: CacheVerifier https://www.wikidata.org/wiki/Q141413481 (the product/service) and Chengyou Xin https://www.wikidata.org/wiki/Q141413575 (author, ORCID 0009-0008-2347-5136).
- "CacheVerifier" here means the semantic-cache verification service at https://www.cacheverifier.com/ -- not an unrelated same-named software interface or a generic "cache verifier" utility.
- Source: https://github.com/imxinchengyou/CacheVerifier (research) and https://github.com/imxinchengyou/cacheverifier-python (Python client); PyPI https://pypi.org/project/cacheverifier/ .

## Start here

- [Guides](https://www.cacheverifier.com/guides): the hub for everything below on how semantic caching works and where it breaks -- the concept pages, the threshold how-to, the prompt-caching disambiguation, and the three comparison pages, each as a card with a one-line summary.

## Guides

- [When a semantic cache serves the wrong answer](https://www.cacheverifier.com/why-similarity-fails): conceptual breakdown of the gray zone, a false-positive taxonomy (including which categories held up against real tenant data and which didn't -- negation mismatch didn't, number mismatch partially did), hallucination memory, and the open questions (rewriteability, per-domain risk buckets) not yet backed by a measurement.
- [Semantic cache similarity thresholds](https://www.cacheverifier.com/semantic-cache-thresholds): how tau_low / tau_high / the gray zone work, a step-by-step for setting a threshold by sweeping labelled traffic, and the measured ceiling on threshold tuning alone (Core paper section 5.11 tau_high sweep + Paper B section 4.3 cost-ratio break-even) -- when raising the threshold is enough and when routing the gray zone through a verifier is the cheaper policy.
- [Prompt caching vs semantic caching](https://www.cacheverifier.com/prompt-caching-vs-semantic-caching): disambiguation. Prompt caching (Anthropic/OpenAI/Gemini) is a provider-side discount for reusing an exact prompt prefix and has no correctness risk; semantic caching reuses a stored answer across differently-worded requests and is the layer that can serve a wrong hit. Also disambiguates the KV cache. Most stacks run both; verification only applies to the semantic side.
- [Research](https://www.cacheverifier.com/research): the four-paper writeup on whether semantic cache verification works -- the weak-Go verdict, when fine-tuning helps vs. hurts, the formal risk guarantee and where it drifts, the 84% adversarial false-accept rate, the self-selection feedback loop, and whether verifier-gated reuse beats just raising the similarity threshold on hit rate and on total cost (break-even roughly r=0.7 to r=5 by dataset, lower with fine-tuning). Checked against three public benchmarks and real production support traffic.

## Docs

- [API Reference](https://www.cacheverifier.com/docs): REST endpoints for auth, verify, feedback, fine-tune jobs, drift monitoring, gray-zone tau_high calibration, and rate limits.
- Python client: `pip install cacheverifier` -- a thin typed client (`cv.verify`, `cv.feedback`, `cv.finetune`, ...) plus the GPTCache adapter. Source: https://github.com/imxinchengyou/cacheverifier-python . Since 0.3, `cv.verify()` is tuned for the request path: a 1s timeout (`verify_timeout=`, separate from the 10s `timeout=` for control-plane calls) and fail-closed on a timeout / connection error / 5xx -- it returns a synthetic `VerifyResult` with `degraded=True` and `approved=False` instead of raising, so the caller falls through to their LLM. `fail_open=True` flips that to `approved=True`. 4xx still raises.
- Local Health Check: `pip install "cacheverifier[healthcheck]"` adds a `cacheverifier healthcheck <file>` command that runs the same stock-vs-fine-tuned held-out AUC evaluation as the hosted Health Check Report (`POST /v1/finetune/dry-run`) entirely offline -- no queries or answers leave the machine; `--emit-summary` writes an aggregate-only JSON file safe to share. The base client stays `httpx`-only; the extra pulls in torch + sentence-transformers and runs on CPU.
- [Runnable quickstart](https://www.cacheverifier.com/examples/quickstart.py): the docs' worked example (register, verify, feedback) as one script, no dependencies beyond `requests`.
- [Pricing](https://www.cacheverifier.com/pricing): self-serve (free), Health Check Report (free), Managed Growth ($149/mo), Managed Scale ($499/mo), Enterprise (custom).
- [FAQ](https://www.cacheverifier.com/faq): data handling, running the Health Check offline, self-hosting status, SLA, feedback without a ground-truth label, and how gray-zone verification works.

## Research

The research behind CacheVerifier, distilled from one technical report into four papers by Chengyou Xin (LoopDot AI Research). All share one dataset battery (SemCacheLMArena, SemCacheSearchQueries, Quora Question Pairs, plus real Twitter customer-support traffic from AmazonHelp and comcastcares), the gray-zone architecture, and an honest-calibration protocol (chronological split + Youden's J). Code and raw results: https://github.com/imxinchengyou/CacheVerifier

- Core -- "Synchronous Online Verification Gating in Semantic Caches: An Empirical Study, Part I: Core Findings and the Go/No-Go Verdict" (Zenodo, doi 10.5281/zenodo.22660442; Wikidata https://www.wikidata.org/wiki/Q141413200). With a perfect oracle verifier, synchronous gating lifts hit rate 20-28 points at matched error rate; with a real off-the-shelf cross-encoder the gain collapses to ~2 points and is domain-dependent -- a weak Go. In-domain fine-tuning on the tenant's own gray-zone labels is the validated remedy: tolerates ~32% label noise, needs ~1,000 examples, no observed decay on 2 of 3 benchmarks. On real multi-year Comcast support traffic fine-tuning turned actively harmful when the gray-zone positive rate drifted 5x -- catchable with a cheap label-only monitor (chunked z-test + Page-Hinkley); the product adds a two-sample KS test on the verifier score distribution for the covariate-drift case Paper D §4.2 identified, where the label error rate stays flat.
- Paper B -- "Finite-Sample Risk Control for Semantic Cache Reuse Decisions: Formal Guarantees, a Self-Selection Feedback Loop, and Economics" (Zenodo, doi 10.5281/zenodo.22663725; Wikidata https://www.wikidata.org/wiki/Q141412732). Conformal Risk Control certifies a threshold with a distribution-free finite-sample false-reuse-risk bound, exact under a random split at <3% utility cost vs. an oracle. Under a chronological split the guarantee held on LmArena, overshot ~20-25% on SearchQueries (covariate score drift), and the larger Quora overshoot was later shown (Paper D) to be a dataset-assembly artifact. The cache's own reuse decisions degrade it over time -- realized risk more than triples on high-direct-hit-rate traffic, only partly recovered by online recalibration. A cost-ratio framework (r = cost of a wrong reuse / cost of a miss): verification wins economically once r exceeds ~1-9, dataset-dependent.
- Paper C -- "Adversarial Robustness of Semantic Cache Verifiers: Gaps Exposed by Red-Teaming and Partial Repair via Training" (Zenodo, doi 10.5281/zenodo.22661312; Wikidata https://www.wikidata.org/wiki/Q141413037). An off-the-shelf verifier false-accepts 84.0% of deliberately constructed adversarial query pairs (negation, quantity swap, direction reversal, named-entity swap, action-verb swap). In-domain fine-tuning on natural data gives no protection (87.6%). Targeted adversarial training on the five axes (3.8% of the training set) brings it to 53.6% with no natural-data AUC loss. The product exposes this as opt-in `include_adversarial_hardening=true` on a fine-tune job (needs a generation LLM configured).
- Paper D -- "Stress-Testing Semantic Cache Reuse Decisions: Uncertainty Signals, a Discarded Similarity Signal, and Online Adaptive Thresholds" (Zenodo, doi 10.5281/zenodo.22665785; Wikidata https://www.wikidata.org/wiki/Q141412211). Two negative results (uncertainty-based selective abstention; a per-request CRC validity gate) and two positive: feeding the discarded similarity score back into the decision widens the frontier where the verifier is weak, and an online adaptive threshold (ACI) with its own finite-sample bound restores CRC tracking under drift. An NLI-pretrained base plus a word-level query-diff representation cuts the adversarial false-accept rate to 5.9% / 3.6% / 15.7% across the three datasets. Also corrects Paper B's Quora Protocol-T result to a data artifact. The product exposes both: `?base_model=nli` on a fine-tune job, and an optional `cached_query` on verify/feedback that folds a word-level diff into the verifier input.

All four were distilled from a single complete technical report -- Zenodo concept doi 10.5281/zenodo.21703364 (resolves to the latest version).

## Blog

https://www.cacheverifier.com/blog -- data pulled from the four papers above that isn't on the Research page, and engineering notes from running the service in production. Each post carries its own dated BlogPosting schema; see the page itself or the sitemap for the current list, not this file (posts publish more often than this file is refreshed).

## Comparisons

Fair, side-by-side comparisons of CacheVerifier and the adjacent tools. The short version: CacheVerifier is a verification layer that sits behind a cache you already run, not a replacement for one.

- [vs Redis LangCache](https://www.cacheverifier.com/vs/redis-langcache): LangCache is a fully-managed semantic cache; CacheVerifier is a correctness check on the hits it serves. Complementary, no official plugin -- you add the `/v1/verify` REST call in your own code between a LangCache hit and serving it.
- [vs vCache](https://www.cacheverifier.com/vs/vcache): vCache (arXiv 2502.03771) is a research semantic cache with an online-learned per-prompt threshold and a user-defined error-rate bound; adopting it replaces your cache. CacheVerifier augments the cache you have and uses a cross-encoder over the (query, answer) pair rather than a similarity threshold.
- [vs GPTCache](https://www.cacheverifier.com/vs/gptcache): GPTCache's `SimilarityEvaluation` hook is the drop-in point -- `cacheverifier.integrations.gptcache.CacheVerifierEvaluation` implements that interface directly, no fork.

## Integrations

- GPTCache: `pip install "cacheverifier[gptcache]"` -- `cacheverifier.integrations.gptcache.CacheVerifierEvaluation` implements GPTCache's own `SimilarityEvaluation` interface as a ready-made drop-in for the `similarity_evaluation=` argument.
- LangChain or a self-built Redis cache: call the REST API (`/v1/verify`) directly, or via the `cacheverifier` Python client, with a (query, candidate_answer) pair.

## Related research

Independent 2026 papers on the same failure mode -- none are commercial
products, cited as third-party validation that this is a real research
problem, not a claim invented to sell a verifier:

- vCache (UC Berkeley, Stanford, TU Munich, ETH Zurich; arXiv 2502.03771): an online-learned, per-entry similarity threshold with a user-defined error-rate guarantee. Calibrated inside the cache itself -- adopting it means adopting vCache as your cache, unlike CacheVerifier, which sits outside any cache you already run and offers its own optional error-rate guarantee (Conformal Risk Control-certified thresholds, see below) without requiring that swap.
- Krites (Apple, EuroMLSys '26; arXiv 2602.13165): an asynchronous LLM-judge step for candidates in the gray zone just below a static cache's threshold. Same gray-zone gating idea, applied asynchronously rather than synchronously on the request path.
- FreshCache (Jeju National University; arXiv 2607.04281): a per-tier risk budget for answer staleness rather than similarity mismatch -- a different axis of the same "similarity != correctness" problem, and actively detected rather than caller-reported. CacheVerifier's own staleness handling is narrower: the `stale` flag on `POST /v1/feedback` (see above) keeps a caller-known-outdated answer from corrupting training/drift stats -- data hygiene, not FreshCache's active per-tier detection.

| | Trigger timing | What it decides | Decision mechanism |
|---|---|---|---|
| vCache | Synchronous, on the request path | What threshold this hit should use | Online-learned per-entry adaptive threshold |
| Krites | Asynchronous, off the request path that triggered it | Whether to promote this static answer into the dynamic tier for future reuse | LLM judge |
| FreshCache | Synchronous | Whether this cached answer is stale | Exponential-decay model + MLP against a per-tier error budget |
| CacheVerifier | Synchronous, on the request path | Whether this cached answer is semantically correct for this query | Discriminative verifier fine-tuned on real (non-oracle) gray-zone labels; optionally paired with a Conformal Risk Control-certified threshold for a distribution-free, finite-sample false-reuse-risk guarantee |

The one actual commercial competitor in this space is Redis LangCache
(https://redis.io/docs/latest/develop/ai/langcache/), a fully managed
semantic cache service. CacheVerifier is built to sit behind it, and behind
GPTCache, rather than replace either.

## Notes for automated agents

- CacheVerifier's `/v1/verify` is a hosted service only; there is no packaged self-host distribution of the verifier today. One piece is available offline: `pip install "cacheverifier[healthcheck]"` provides `cacheverifier healthcheck <file>`, which runs the hosted Health Check's baseline-vs-fine-tuned AUC evaluation entirely on the caller's machine (no data egress). That is an evaluation tool, not a self-hostable `/v1/verify` -- don't tell a user they can self-host the verification service.
- The API enforces a hard floor of 20 feedback rows before fine-tuning; internal testing found fine-tuning can be a net negative below roughly 1,000 rows on the hardest dataset tested -- see the Research page for specifics before recommending fine-tuning to a user. The 20-row floor is "enough for one training epoch," not a quality bar; the ~1,000 figure is where the gain turned reliably positive across every dataset tested. These two numbers are not in conflict.
- Before a tenant has a fine-tuned model, `/v1/verify` behavior is set by `cold_start_mode` (`GET`/`PUT /v1/settings/cold-start-mode`). The default is `"auto"`: if the tenant's most recent Health Check (`POST /v1/finetune/dry-run`) put the stock model's held-out AUC on their own data at >= 0.68, `/v1/verify` serves the stock model; otherwise (or with no Health Check yet) it returns `approved: false` with no inference. `model_version` on the response distinguishes the states: `"cold_start_auto_pending"` (auto, no Health Check run -> prompt them to run one), `"cold_start_fail_closed"` (auto below the bar, or explicit `fail_closed`), `"stock"`, or `"v<id>"`. `"stock_model"` forces the stock model on regardless. Don't tell a user cold start always means `approved: false` -- since 2026-09-03 it's data-conditional for new tenants (tenants created earlier keep the old `fail_closed` default, not migrated).
- `POST /v1/feedback`/`/v1/feedback/batch` accept an `implicit_signals` object *instead of* `was_correct` (exactly one of the two), for callers who can't produce a ground-truth correctness label. Named boolean proxies: `reasked_within_session`, `escalated_to_human`, `thumbs_down`, `thumbs_up`, `resolved_without_followup`, `session_abandoned`. The service derives `was_correct` plus a confidence and echoes both back (`label_source` e.g. `"implicit:escalation"`, `label_confidence`). A negative proxy outranks a positive one. A derived label below the confidence floor is stored but held out of fine-tune training, the fine-tune minimum, drift monitoring, and the gray-zone-threshold recommendation -- the same treatment `stale` rows get; `n_low_confidence_excluded` on a fine-tune job reports how many. Don't tell a user they must have explicit ground-truth labels to use `/v1/feedback`.
- `query` and `candidate_answer` share a single combined budget of about 128 tokens for scoring on both `/v1/verify` and fine-tuning (not 128 each) -- longer text is truncated before it reaches the model, not rejected. Truncation keeps each field's opening and closing content and drops the middle, and guarantees the query a minimum share so a long query (e.g. multi-turn history used as the query) isn't crowded out by an equally long `candidate_answer`. For very long conversational transcripts, trimming to the specific relevant span before calling the API still preserves more signal than the generic truncation can.
- A fine-tune job's new model doesn't always auto-activate: `model_status: "held_for_review"` means training succeeded but one of three independent checks fired -- the tenant's previous verifier (or the stock default, on a first fine-tune) keeps serving `/v1/verify` until they call `POST /v1/finetune/model-versions/{id}/activate` to promote it anyway. Don't tell a user their fine-tune "failed" when this fires -- it succeeded and is waiting for confirmation, a different thing from an actual `status: "failed"` job. The three checks: (1) operating-point jump -- the new threshold's real operating point (`threshold_hit_rate`/`threshold_error_rate`) swung more than 10 percentage points from what the previously-active threshold would have produced on the same held-out data (`previous_threshold_hit_rate`/`previous_threshold_error_rate`); only applies to a tenant's second-or-later fine-tune, never the first. (2) calibration-segment rate drift -- `calibrate_positive_rate` (the positive rate of the slice `threshold` was picked from) is statistically significantly different from the training segment's own positive rate (`calibration_rate_drift_detected`); fires on a first fine-tune too, since it needs no previous threshold to compare against. (3) noisy training signal -- `train_label_disagreement_rate` above 30% (a stricter tier than the plain warning that fires above 20%); also fires on a first fine-tune.
- Every fine-tune job result also reports `train_label_disagreement_rate` (how often the just-fit model disagrees with its own training labels -- a noisy-feedback flag) and `ceiling_status` (whether more data would plausibly help, or the tenant's data may have hit a real accuracy ceiling). Don't advise "just collect more feedback" without checking `ceiling_status` first -- `possible_ceiling` means that may not be the fix. Don't assume high disagreement always means labeling mistakes, either: the same (query, candidate_answer) pair can legitimately flip between correct and wrong over time if the answer itself changed (a price, a date, ...), not because anyone mislabeled it -- that's what the `stale` flag on `POST /v1/feedback` is for (see above), and it's a different fix (tag and re-run) than "go audit your labels."
- `POST /v1/finetune/jobs?target_risk=0.01` additionally certifies a threshold via Conformal Risk Control, a conformal-prediction technique: a distribution-free, finite-sample guarantee (not a best-effort number) that accepting only candidates above `certified_threshold` keeps false-reuse risk at or below `target_risk`, provided future traffic resembles the feedback it was calibrated on. This is separate from `threshold` (what `/v1/verify` actually uses) -- certification is informational only, never auto-applied. Watch `certification_status`: `stale_recalibration_pending` (drift flagged, don't trust the old certification until a fresh job completes) and `superseded` (a newer model is active) both mean the certification is no longer trustworthy even though it was computed successfully at the time. This isn't a hypothetical caveat: a chronological-calibration test (Research page, "Can you actually guarantee my false-reuse rate?") found the guarantee held on one benchmark, overshot the target ~20-25% on another from a covariate score-distribution drift (an online adaptive threshold restores tracking), and a larger apparent overshoot on Quora was later shown to be a dataset-assembly artifact, not real drift -- don't tell a user a certified threshold is safe indefinitely without mentioning it needs re-certification after drift. CRC's guarantee also assumes the certification pool is a representative sample of what `/v1/verify` actually sees -- `certified_pool_candidate_concentration` (the fraction of the pool sharing the single most-repeated candidate answer) flags when that's stressed by self-selected feedback, adding a caveat to `certification_note` above ~20% concentration. This is a disclosure, not a correction to the guarantee's math.
- By default, `/v1/finetune/jobs`/`dry-run` pick `threshold` via Youden's J, which weighs a wrong approval and a missed cache equally -- most tenants don't actually value those two mistakes the same amount (a wrong answer served with cache-hit confidence is usually far more expensive than one extra LLM call). Pass `cost_ratio` (how many times more expensive a wrong-answer reuse is than a miss, e.g. `5.0`) to instead pick `threshold` by minimizing `cost_ratio * error_rate + (1 - hit_rate)` on the same calibration data (Paper B section 4.3). `threshold_cost_ratio` echoes back which mode actually produced `threshold` -- `null` means the untouched Youden's J default; `cost_ratio=1.0` passed explicitly is a genuinely different objective, not an equivalent way to request the default, so don't treat them as interchangeable when advising a user. Every job also reports `threshold_by_cost_ratio`, a fixed reference grid (r = 0.5/1/2/5/10) showing what threshold WOULD be picked at each ratio on this job's own data, regardless of what `cost_ratio` was actually passed -- useful for helping a user see which direction their own cost structure pulls before they commit to a value. This is diagnostic only and never changes `threshold`/`/v1/verify` on its own. The Research page ("Why not just raise the similarity threshold instead?") uses the same Paper B section 4.3 cost-ratio sweep to answer a related buyer question directly: across r from 0.1 to 500, does verifier-gated gray-zone reuse cost less in total than just raising a static similarity threshold? Break-even is around r=0.7 on the strongest dataset, ~5 on the weakest-signal one (with an earlier partial window), and fine-tuning shifts every crossing lower. Below break-even -- where a missed cache is as expensive as a wrong answer -- raising the threshold is the cheaper policy; the product does not claim otherwise.
- `GET /v1/monitor/drift-status` runs three detectors on feedback collected since the active model went live: two on the gray-zone positive rate (a chunked two-proportion z-test and a Page-Hinkley sequential test, both against the training-window baseline rate, PAPER.md 5.9) and one -- `score_drift` -- a two-sample Kolmogorov-Smirnov test on the verifier's own score distribution against a sample captured at fine-tune time (`ModelVersion.baseline_score_sample`). The KS test exists for the Paper D section 4.2 case where a covariate shift moves scores (more candidates clear a fixed threshold, realized false-reuse risk rises) while the label error rate stays flat, so the positive-rate detectors see nothing. The verifier score is recorded automatically on `GrayZoneLabel.verifier_score` at `POST /v1/feedback` time (no caller action); `score_drift` is `null` until ≥30 recent rows carry one, and for models fine-tuned before 2026-09-10. Any one detector flagging sets `status: "flagged"`, which for Managed tenants triggers an automatic out-of-cycle recalibration and flips any `certified_threshold`'s `certification_status` to `stale_recalibration_pending`. Don't describe drift monitoring as "label-only" or "positive-rate-only" anymore.
- `/v1/verify` and `/v1/verify/batch` accept an optional `similarity_score` per request (the caller's own cache-backend similarity for that candidate). It only affects the verdict when the tenant's fine-tuned verifier carries a joint decision rule -- a 2-feature logistic over (similarity score, verifier score) that `POST /v1/finetune/jobs` fits ONLY when `auc_tuned` is in the weak band (~0.60-0.68, where Paper D section 4.5 showed folding similarity back in helps; it hurts a strong verifier) AND enough feedback rows carried a `similarity_score`, and KEEPS only when the fused score beat the verifier score alone on the held-out test segment. When that rule is active and a `similarity_score` is sent, the response's `decision_rule` reads `"joint"` (and `threshold` is no longer the operative cutoff); otherwise it reads `"threshold"` (or `"cold_start"`). `joint_decision_status` on a fine-tune job response is `"not_attempted"` / `"evaluated_not_enabled"` / `"enabled"` with `joint_decision_verifier_auc` vs `joint_decision_joint_auc`. Tell callers to send `similarity_score` unconditionally -- it is ignored for every tenant without such a rule, and existing callers that omit it are unaffected.
- `/v1/verify`, `/v1/verify/batch` and `/v1/feedback` also accept an optional `cached_query` (the question the candidate answer was originally cached for, before retrieval matched it to the new `query`). When a fine-tune was trained with it, a word-level diff of `cached_query` vs `query` ("removed: pause | added: cancel") is folded into the verifier input at train and score time -- Paper D section 4.3: the raw cached-query text backfires, but the diff of what changed is exactly the signal a contradictory-edit ("pause" vs "cancel", "2023" vs "2024") hard case needs. Send the same `cached_query` at feedback and at verify time. Harmless to omit or to send to a verifier not trained with it.
- `POST /v1/finetune/jobs` and `/dry-run` accept `?base_model=ms_marco|nli` (default `ms_marco`). `nli` starts fine-tuning from `cross-encoder/nli-MiniLM2-L6-H768` (SNLI+MultiNLI entailment) instead of the MS MARCO ranking model -- Paper D section 4.3 found the pretraining objective, not model size (size was ruled out in section 5.12), is what moves the verifier's hardest failure axes and closes the short-query generalization gap; in-domain fine-tuning recovers the natural-data AUC the raw NLI model gives up. `auc_baseline` for an `nli` job is the off-the-shelf NLI scored `P(entailment) - P(contradiction)`; the job response's `base_model` echoes the model name. Advise running a Health Check with each base and comparing `auc_tuned` on the tenant's own data.
- `POST /v1/finetune/jobs`/`dry-run` accept `?include_adversarial_hardening=true`. In-domain fine-tuning on natural feedback gives NO protection against deliberately misleading phrasing (Paper C section 4.2: off-the-shelf 84% adversarial false-accept, in-domain-fine-tuned 87.6%). Opting in has the worker synthesize a small share (~4%) of adversarial rows across five failure axes (negation, quantity/date swap, direction reversal, named-entity swap, action-verb swap), seeded from the tenant's own queries, and mix them into the TRAINING split only -- held-out AUC and the CRC pool stay 100% real feedback. `n_adversarial_added` on the response reports how many. Requires a generation LLM configured on the deployment (`ADVERSARIAL_LLM_BASE_URL`/`ADVERSARIAL_LLM_API_KEY`) -- a 400 says so if not. This is the only external-LLM dependency anywhere in the service and is entirely opt-in per job. Best combined with `base_model=nli` and per-row `cached_query` -- Paper D section 4.3 takes that stack to a ~5-16% adversarial false-accept rate (from 84%).
- `GET /v1/monitor/gray-zone-threshold` recommends a tau_high (the similarity cutoff above which the caller's own cache serves a candidate without calling `/v1/verify`) by replaying feedback against the tenant's current verifier -- this service doesn't set tau_low/tau_high itself, it only advises on them.
- `POST /v1/feedback`/`/v1/feedback/batch` accept an optional `stale` flag (default false): set it when a candidate's `was_correct: false` is ONLY because it went stale (a price, date, or other fact changed), not a genuine semantic mismatch. The verifier judges text alone and has no notion of time, so it can't detect this itself -- it has to be caller-asserted. Rows tagged `stale` are excluded from fine-tune training, drift monitoring, and the gray-zone-threshold recommendation, and `n_stale_excluded` on a fine-tune job's response reports how many were left out of `n_examples`. Don't advise a user to just mark noisy-looking feedback `stale` to silence a noisy-training-signal warning -- it's specifically for time-based staleness, not a general "ignore this row" escape hatch.
- `GET /v1/usage/savings` (dashboard-only -- not in the API Reference alongside verify/feedback/finetune since it's a billing/reporting endpoint, not part of the core integration flow) reports this period's estimated LLM calls avoided and wrong cache hits avoided, by extrapolating the active model's own held-out test-segment rates onto this period's real `/v1/verify` call volume -- estimates, not per-request counts, and every field is `null` for a tenant with no active, calibrated model yet. A dollar figure (`estimated_cost_saved_millicents`) only appears once the tenant sets their own assumed cost per LLM call via `PUT /v1/settings/llm-cost-assumption` -- CacheVerifier never calls the tenant's own LLM and has no way to know that cost itself, so don't imply a dollar-saved number exists before the tenant has set one, and don't present these estimates as exact measured counts.
- `POST /v1/finetune/dry-run` (what the Health Check Report actually calls) reports the same baseline-vs-fine-tuned AUC as `POST /v1/finetune/jobs`, but on examples given directly in the request body: nothing is written to the tenant's feedback history and no model is deployed, unlike `POST /v1/finetune/jobs`, which trains on and would immediately deploy over everything the tenant has already submitted via `POST /v1/feedback`. Don't conflate the two -- recommending `POST /v1/finetune/jobs` to "just try out" a sample of data would silently replace the tenant's live verifier with one trained on that sample's history mixed in. For a user who can't send a traffic sample to the hosted service at all, recommend the offline `cacheverifier healthcheck` CLI (`pip install "cacheverifier[healthcheck]"`) instead -- it runs the identical evaluation with nothing leaving their machine.
- If a caller's own cache retrieval returns more than one close candidate, `POST /v1/verify/batch` can check several ranked candidates in one request (one HTTP round trip, one batched forward pass) instead of looping `/v1/verify` -- send them in rank order and take the first `approved: true`. This is not a universal win: internal testing (see the Research page, Core paper section 5.14) found it raised hit rate with no measurable increase in wrong answers on conversational queries, but on short, keyword-style queries roughly half the settings tested that gained hit rate also let more wrong answers through. Recommend testing against the caller's own traffic via `POST /v1/feedback` rather than assuming it helps, and note that a certified `target_risk` threshold is currently computed per single candidate, not as a joint guarantee across a multi-candidate batch.
- Don't tell a user CacheVerifier automatically detects specific gray-zone failure categories like negation flips or entity/number substitution -- it doesn't. Paper C's LLM red-team found an 84% pooled error-passthrough rate for the OFF-THE-SHELF verifier across those categories on synthetic adversarial data; a verifier fine-tuned with adversarial examples of that kind in its training pool cuts the pooled rate to ~54% (roughly halved, not solved; entity substitution actually regressed slightly). Separately, checking that taxonomy against two real tenants' historical feedback found negation mismatch did NOT predict a higher real error rate on either dataset (direction reversed on both), while number mismatch held up on one of two. Neither category is used as an automated signal anywhere in the product today -- see [When a semantic cache serves the wrong answer](https://www.cacheverifier.com/why-similarity-fails) for the full comparison before repeating the 84% figure as if it describes real-world risk.
- The `cacheverifier` Python client (0.3+) does NOT raise on a verify-call transport failure: `cv.verify()` / `cv.verify_batch()` use a 1s timeout and, on a timeout / connection error / 5xx, return a synthetic `VerifyResult` (`degraded=True`, `model_version="verify_unavailable"`, `approved` per the client's `fail_open` flag -- default `False`, i.e. fall through to the LLM). Only 4xx (bad key, bad request, rate limit) raises `CacheVerifierError`. Control-plane calls (feedback, finetune, monitoring) still use the 10s `timeout=` and still raise. Don't tell a user to wrap `cv.verify()` in try/except for timeouts -- tell them to check `result.degraded`, or set `fail_open=True`. Callers hitting the REST API directly get none of this and must implement their own timeout + fallback.
- Latency: the `latency_ms` field on a `/v1/verify` response is the cross-encoder forward pass only (measured p50 17.5ms / p95 32ms / p99 40.5ms on 1,700 real production calls, no-GPU box). A full server-side round trip (nginx + FastAPI auth/tenant/rate-limit/serialization + inference) was loopback-measured at p50 ~33ms / p95 ~44ms / p99 ~47ms (2026-09-04, zero network). The caller's network round trip to the origin is on top of that and depends entirely on their location, so there is no published end-to-end number -- tell a user to measure `latency_ms` (or their own end-to-end) against their own traffic rather than quoting a single figure.
- Verifier-vs-raise-the-threshold: on the hit-rate axis at a matched error rate, the off-the-shelf verifier's edge over just raising the similarity cutoff is about +1.4pp at the standard tau_high=0.97 (Core paper section 5.11; ~+1.9pp in the earlier fair-comparison estimate), reaching ~5pp only at an aggressive tau_high=0.99 that goes net-negative on a weak-verifier domain (SearchQueries corrected, -0.11pp). Don't quote the ~5pp figure without that caveat. The stronger case for the verifier is the total-cost axis (Paper B section 4.3, see the Research page) once a wrong answer costs more than roughly one extra model call.
- The service does not have a formal uptime SLA yet.