API Reference

Quickstart

From zero to a verified call in two requests.

curl -X POST https://www.cacheverifier.com/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "email": "you@example.com",
    "password": "at-least-8-chars",
    "tenant_name": "acme",
    "accept_terms": true
  }'

# accept_terms: true says you accept the Terms of Service and Privacy Policy; signup fails without it.
# => { "token": "...", "api_key": "cv_...", "user": { ... } }
# Save api_key now -- it's only ever returned this once.

curl -X POST https://www.cacheverifier.com/v1/verify \
  -H "Content-Type: application/json" \
  -H "X-API-Key: cv_..." \
  -d '{
    "query": "when will my order arrive",
    "candidate_answer": "In-stock items ship within 48 hours and usually arrive 2–3 days after shipping."
  }'

# A new account's first calls are all rejected -- nothing has been checked against your data yet:
# => { "approved": false, "model_version": "cold_start_auto_pending", "decision_rule": "cold_start",
#      "notice": "No verifier has been checked against your data yet, so every call is rejected for now ...",
#      "shadow": { "approved": true, "score": 1.202, "threshold": 0.0, "latency_ms": 17.5, "model_version": "stock" } }
#
# After a Health Check clears it (or cold_start_mode = "stock_model"), a model decides:
# => { "approved": true, "score": 1.202, "latency_ms": 17.5, "model_version": "stock", "threshold": 0.0, "decision_rule": "threshold" }

From here: send was_correct ground truth to /v1/feedback as you get it, and once you have a couple hundred rows, kick off a fine-tune job — or just run the Health Check Report in the dashboard, which does the feedback-upload-and-fine-tune loop for you on a sample.

Integration pattern

CacheVerifier doesn't replace your cache — you already have one (Redis, GPTCache, a vector database like Pinecone or Weaviate, a RAG pipeline's retrieval cache, your own). The problem it solves is the gray zone: a candidate that clears your cosine similarity threshold — typically somewhere around 0.8–0.9 — without actually being right. Raising that cosine similarity threshold doesn't fix it either: push it high enough to kill the false positives and you start rejecting real duplicate questions, because near-miss and true-match pairs often sit within a few hundredths of each other in cosine similarity. Serving a false positive is worse than a cache miss — from the user's side, a confidently wrong cached answer looks a lot like a hallucination, except your system had the right answer on file the whole time. A worked example — a support bot backed by a similarity-search cache:

user_query = "can I get a refund if my flight is delayed"
candidate = my_cache.search(user_query)
# candidate.answer = "Delays over 2 hours qualify for one free rebooking."
# candidate.similarity = 0.86 -- close, but "refund" vs. "rebooking" is a real difference

resp = requests.post(
    "https://www.cacheverifier.com/v1/verify",
    headers={"X-API-Key": API_KEY},
    json={"query": user_query, "candidate_answer": candidate.answer},
).json()
# => { "approved": false, "score": 0.41, "threshold": 0.5, ... }

if resp["approved"]:
    answer = candidate.answer          # cache hit confirmed -- skip the LLM call
else:
    answer = call_real_llm(user_query) # not trustworthy -- fall through, then re-cache
    my_cache.store(user_query, answer)

requests.post(
    "https://www.cacheverifier.com/v1/feedback",
    headers={"X-API-Key": API_KEY},
    json={"query": user_query, "candidate_answer": answer, "was_correct": True},
)

When the verify call itself fails. /v1/verify is on your request path, so decide up front what a timeout or a 5xx means for you. The safe default is fail closed: treat an errored verify like approved: false and fall through to your LLM — you pay for one regeneration, never serve an unverified near-miss. Only fail open (serve the cached answer anyway) if you've decided a stale-or-near-miss answer is acceptable for that traffic. Warm calls are tens of milliseconds server-side (see the FAQ for measured percentiles), but the first call after a deploy is slower while the model loads, so a ceiling around 1s with fail-closed handling covers both. The cacheverifier Python client (0.3+) does this by default — verify() uses a 1 s timeout and returns approved: false with degraded=true on a timeout or 5xx; pass fail_open=True to serve the cached answer instead, or verify_timeout= to change the ceiling. Calling the REST API directly, you set both yourself. There is no formal uptime SLA yet (see the Terms), which is another reason to make the fallback path explicit.

From there, either call POST /v1/finetune/jobs yourself once feedback builds up (self-serve, free forever), or subscribe to a Managed plan and let it happen on a schedule automatically, with drift alerts if the gray zone starts behaving differently than it did at training time.

Python client. pip install cacheverifier gives you a typed client for every endpoint on this page — cv.verify(...), cv.feedback(...), cv.finetune(). Source, examples, and the GPTCache adapter are in cacheverifier-python.

Already on GPTCache? pip install "cacheverifier[gptcache]", then pass CacheVerifierEvaluation(api_key=...) as GPTCache's similarity_evaluation= argument — the if/else above happens inside GPTCache's own pipeline, no extra code required.

On LangChain? pip install "cacheverifier[langchain]", then wrap the cache you already use: set_llm_cache(VerifiedCache(RedisSemanticCache(...), api_key=...)), with VerifiedCache from cacheverifier.integrations.langchain. Each hit is verified against the user's latest message before it is served; a rejected hit is a miss, so your model is called as usual.

Prefer no dependencies? examples/quickstart.py is this same worked example as one runnable script using only requests, and registers a free tenant for you if you don't already have an API key.

High-stakes domains

The verifier is a statistical model, so some verdicts will be wrong. When a reused answer goes to someone making a health, legal, money or safety decision — a medication question, a filing deadline, an account balance, a flight change — one wrong reuse can cost far more than the LLM call it saved. In those flows:

• Don't cache what must be current or personal. Route questions about a specific person's account, order, booking or case straight to a fresh model call (or your system of record), whatever the similarity score.
• Start strict and measure. Keep cold-start fail-closed until a fine-tune has been checked on your own feedback, request a certified risk target, and turn on randomized auditing so the error rate you see is measured, not assumed.
• Keep a person in the loop where the answer drives a decision about someone — credit, insurance, employment, housing, care.
• Tell your users. A short note such as "this answer was matched to an earlier, similar question" lets them ask again if it doesn't fit.
• Watch for drift. A certified threshold holds only while traffic looks like the data it was calibrated on; the Drift Monitor flags when it stops.

Authentication

Tenant-scoped endpoints (verify, feedback, fine-tune, drift) accept either credential — whichever is more convenient for the caller:

• X-API-Key: cv_... — a raw tenant API key, for server-to-server integrations. Issued once at registration or by an admin; only the hash is stored, so it can't be recovered if lost.
• Authorization: Bearer <token> — a dashboard session token from /v1/auth/login or /v1/auth/register, valid for 24 hours.

X-Admin-Token is separate and only guards POST /v1/admin/tenants (internal tenant provisioning).

Rate limits

Enforced at the edge, keyed by X-API-Key when present (so multiple tenants behind the same NAT/corporate IP don't share a bucket), falling back to client IP for unauthenticated calls and dashboard session-token requests. Exceeding a limit returns a plain 429 with a Retry-After header — back off and retry. A 429 from /v1/verify without Retry-After is different: your monthly verify quota is used up, and retrying won't help until you top up, subscribe or the month resets. The Python client (0.4.1+) treats a rate-limited verify like an outage — a degraded result, so you fall through to your LLM — and still raises on the quota case.

The general limit's burst allowance is sized for bulk flows like the Health Check Report, which submits many /v1/feedback rows back-to-back — a single run of a hundred or so rows won't get throttled. The auth limit is intentionally tight: it's there to slow down credential stuffing and mass signup, not to accommodate normal traffic.

Verify calls have their own, higher limit because they sit on your request path — about 20 a second per API key, with room for short spikes. Past that, use /v1/verify/batch (many pairs per request) or local scoring, which has no request limit at all.

Auth

{
  "email": "you@example.com",
  "password": "at-least-8-chars",
  "tenant_name": "acme"
}

{
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "api_key": "cv_hSDoVrYkFJo2CvoaL1_N9SSQDOWjW6NMSV4DPj9FQ5g",
  "user": { "id": 1, "email": "you@example.com", "tenant_id": 1 }
}

{ "email": "you@example.com", "password": "at-least-8-chars" }

{
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "user": { "id": 1, "email": "you@example.com", "tenant_id": 1 }
}

{ "id": 1, "email": "you@example.com", "tenant_id": 1 }

{ "api_key": "cv_hSDoVrYkFJo2CvoaL1_N9SSQDOWjW6NMSV4DPj9FQ5g" }

rotate-key requires a dashboard login (Authorization: Bearer), not the API key itself — otherwise a leaked key could keep itself alive. Update any server-to-server integration with the new key immediately; the old one stops working the instant this call succeeds.

Verify

{
  "query": "when will my order arrive",
  "candidate_answer": "In-stock items ship within 48 hours and usually arrive 2–3 days after shipping.",
  "similarity_score": 0.91,             // optional
  "runner_up_similarity_score": 0.88,   // optional
  "cached_query": "how long does delivery take"  // optional
}

{
  "approved": true,
  "score": 1.202,
  "latency_ms": 17.5,
  "model_version": "stock",
  "threshold": 0.0,
  "decision_rule": "threshold"
}

similarity_score is optional — your own cache backend's similarity for this candidate. It only affects the verdict when your fine-tuned verifier carries a joint decision rule: a fine-tune fits a rule that combines similarity with the verifier score and keeps it only if it beat the verifier score alone on held-out data (CacheVerifier Paper D §4.5). When that rule is active and you send asimilarity_score, decision_rule in the response reads "joint"instead of "threshold", and threshold is no longer the operative cutoff. A joint rule still rejects an answer the verifier itself scores as clearly wrong, however high the similarity, so a stale or rewritten cache entry can't pass on similarity alone. Send it unconditionally — it's ignored (decision_rule: "threshold") for every tenant that doesn't have such a rule. "cold_start" is the third value, before you have a model of your own. It isn't range-checked, because backends report cosine, inner product or distance on different scales: send whatever yours reports, the same way every time. Only non-finite values are rejected.

runner_up_similarity_score is optional — your cache's similarity for its second-best match for this query (a top-2 search returns it for free). The gap between your best and runner-up match tells a fine-tuned rule how crowded your cache is around this query, which the verifier itself can't see. A fine-tune fits a 3-feature rule on it and keeps it only if it beat both the verifier score and the 2-feature rule on held-out data; with it active, calls that send both similarity scores get decision_rule: "joint_gap". Once auditing is on and enough audits exist, these rules are fit on your rejected + audited labels, inverse-probability weighted, instead of volunteered feedback. Send it together with similarity_score on /v1/feedback too.

cached_query is optional — the question candidate_answer was originally the answer to, before your cache retrieved it for this query. When your verifier was fine-tuned with it (see Feedback), what changed between it and query is folded into the score — CacheVerifier Paper D §4.3 found the raw text backfires but "what changed" is exactly the signal a contradictory-edit hard case needs. The stock model and any verifier fine-tuned without it ignore it, so it is safe to always send.

latency_ms is the cross-encoder forward pass only (~17.5 ms p50). A full call on our side, end to end, measured p50 33 ms / p95 44 ms / p99 47 ms on a loopback probe of the origin (300 warm calls, 2026-09-04). Your network round trip to us is on top of that and depends on where you call from, so measure end-to-end against your own traffic. One reference point: 150 calls from a US-East client measured a full round trip of p50 ~33 ms / p95 ~39 ms (2026-09-09) — essentially the same as the server-only figure above, i.e. network overhead was close to zero on that specific path. That's a single geography and run, not a guarantee for yours. The first call after a deploy is slower while the model loads.

threshold is the score >= threshold cutoff this call was decided against —0.0 on the shared stock model, tenant-specific once a fine-tune job has calibrated one (see Fine-tune Jobs below).

Three more fields only matter once you turn on randomized auditing: verifier_approved (the verifier's own verdict, equal to approved except on an audited call), audit (true when this approved call was picked for an audit, in which case approved is deliberately false), and verification_id (the id to send back on POST /v1/feedback; null while auditing is off).

model_version tells you which verifier decided the call: "v<id>" for your own fine-tuned model, "stock" for the shared un-tuned cross-encoder, or — before you have a fine-tuned model — one of two cold-start values that ran no inference and always returned approved: false: "cold_start_auto_pending" (the default: run a Health Check so we can tell whether the stock model is accurate enough on your data to serve — a Health Check that clears the accuracy bar on a test split large enough to trust flips this on automatically; a smaller one leaves it pending, see health_check_sufficient on GET /v1/settings/cold-start-mode) or "cold_start_fail_closed" (a Health Check came back below that bar, or you set cold-start mode to fail_closed explicitly). PUT /v1/settings/cold-start-mode overrides the default in either direction. Cold-start responses also carry a plain-language notice saying why the call was rejected and what to do next, and a shadow object with what the shared stock model would have decided for that pair — a preview only, since the call is still rejected. Both are null whenever a model made the decision. The Python client adds one more value client-side — "verify_unavailable" on the synthetic result it returns when a verify call times out or 5xxes (see the fail-closed note above); the API itself never sends it.

query and candidate_answer together share a combined budget of about 128 tokens for scoring, not 128 each — longer text is truncated before it reaches the model, not rejected. Truncation isn't a blind cutoff either: when a pair doesn't fit, each field keeps its own opening and closing content and loses the middle, and query is guaranteed a minimum share so an unusually long query (e.g. multi-turn history used as the query) isn't crowded out entirely by an equally long candidate_answer. This is a real limit, not a formality: a long conversational transcript that gets cut off mid-thought can genuinely score differently than the full text would. If your candidates are long (support-chat transcripts, multi-turn context), this truncation already keeps the most likely-relevant parts automatically — but for very long inputs, trimming to the specific relevant span yourself before calling this endpoint will still preserve more signal than any generic heuristic can.

{
  "items": [
    { "query": "when will my order arrive", "candidate_answer": "Usually 2–3 days after shipping." },
    { "query": "can I get a refund if my flight is delayed", "candidate_answer": "Delays over 2 hours qualify for a free refund." }
  ]
}

{
  "results": [
    { "approved": true, "score": 1.202, "latency_ms": 23.5, "model_version": "stock", "threshold": 0.0, "decision_rule": "threshold" },
    { "approved": true, "score": 0.98, "latency_ms": 23.5, "model_version": "stock", "threshold": 0.0, "decision_rule": "threshold" }
  ]
}

The whole batch is quota-checked and billed as a single unit: admitted only if capacity covers every item, rejected entirely otherwise — never partially processed. latency_ms on a batch response is the whole batch's time, not a true per-item measurement (there's no way to isolate one pair's share of a single forward pass). Each item may carry its own optional similarity_score, same meaning as on /v1/verify.

candidates = my_cache.search(user_query, top_k=2)  # ranked, most similar first

resp = requests.post(
    "https://www.cacheverifier.com/v1/verify/batch",
    headers={"X-API-Key": API_KEY},
    json={"items": [
        {"query": user_query, "candidate_answer": c.answer} for c in candidates
    ]},
).json()

hit = next((r for r, c in zip(resp["results"], candidates) if r["approved"]), None)
answer = hit and candidates[resp["results"].index(hit)].answer or call_real_llm(user_query)

Feedback

{
  "query": "when will my order arrive",
  "candidate_answer": "In-stock items ship within 48 hours and usually arrive 2–3 days after shipping.",
  "was_correct": true,
  "similarity_score": 0.86
}

{
  "id": 42,
  "was_correct": true,
  "label_confidence": 1.0,
  "label_source": "explicit",
  "label_check": null
}

label_check flags a row labelled wrong whose cached_query is the same question as query once case, spacing and punctuation are ignored ("same_question_marked_wrong"), or the same words reordered ("same_words_reordered_marked_wrong"). Either the label is a mistake or the cached answer itself is bad (refresh or evict that entry). On public semantic-caching benchmarks most such labels turned out to be mistakes. The row is stored as sent; the flag is only a prompt to review it, and it is always null without a cached_query. The batch endpoint returns the same flags as label_checks, in request order.

If you send cached_query on only some of your feedback, label those rows the way your traffic really is. When they are correct much more or less often than the rest, a fine-tuned model can learn that a cached_query by itself means a wrong (or right) answer, and shift the score of every request that sends one. POST /v1/finetune/jobs warns when it sees that gap.

Pass an Idempotency-Key header to make retries safe — retrying the same key returns the original row's id instead of creating a duplicate label. Reusing a key for different feedback (another pair or another label) is a 422, so use a new key per submission. Omit it and the same (query, candidate_answer) pair is free to get independent feedback rows, which is the common case (different users hitting the same cached answer).

If you don't have was_correct — most teams running a semantic cache don't have a clean "was this answer right" signal — send an implicit_signals object instead (exactly one of the two; both, or neither, is a 422). Set any of the proxies you can observe and the service derives was_correct plus a confidence, echoed back on the response:

{
  "query": "when will my order arrive",
  "candidate_answer": "In-stock items ship within 48 hours.",
  "implicit_signals": {
    "reasked_within_session": true,   // user asked the same thing again
    "escalated_to_human": false,      // handed off to a human agent
    "thumbs_down": false,             // explicit negative rating
    "thumbs_up": false,               // explicit positive rating
    "resolved_without_followup": false, // session ended clean, no re-ask
    "session_abandoned": false        // user left mid-flow
  }
}
// -> { "id": 43, "was_correct": false,
//      "label_confidence": 0.6, "label_source": "implicit:reask" }

A negative proxy outranks a positive one (a thumbs-up that's then re-asked is, on balance, a wrong reuse). Strong proxies — an explicit rating, an escalation, a re-ask — produce a confident label used exactly like was_correct. Weak ones on their own — a lone abandoned session, a lone clean resolution — land below the confidence floor: the row is still stored (and still counts toward your raw history) but is held out of fine-tune training, the fine-tune minimum, drift monitoring, and the gray-zone threshold recommendation, the same treatment a stale row gets. See n_low_confidence_excluded below.

similarity_score is optional — your own cache backend's raw similarity score (cosine similarity, in most implementations) for this candidate, if you still have it in scope when you learn was_correct. It isn't used by fine-tuning or drift monitoring; it's what GET /v1/monitor/gray-zone-threshold replays against to recommend a tau_high for your own cache's gray-zone boundary. Omit it and that row is simply excluded from that calculation — everything else about feedback works the same.

cached_query is optional — the question candidate_answer was originally cached for. When you send it, fine-tuning folds a word-level diff of it against query into the verifier's input (Paper D §4.3), and you then send the same cached_query on /v1/verify at serving time. If your cache tracks which stored question each hit came from, plumbing this through is the single highest-leverage input for hard "pause vs cancel" / "2023 vs 2024" cases.

Once you're on a fine-tuned model, the service also records its score for this pair at submission time — nothing for you to send. That stream is what the drift monitor's KS test compares against calibration, so a shift in the score distribution itself gets caught, not just a shift in how often the answer was right.

stale is optional (default false) — set it instead of leaving was_correct: false unqualified when the ONLY reason this candidate is wrong now is that it went stale (a price, date, or other fact changed), not a genuine semantic mismatch. This is a real, and different, correctness axis: the verifier judges semantic match from text alone and has no notion of time, so it can't infer staleness — it has to be caller-asserted, the same way was_correct itself is. Rows tagged stale are excluded from fine-tune training, drift monitoring, and the gray-zone threshold recommendation — a stale-caused "wrong" would otherwise teach the model (or a drift/threshold calculation) something that has nothing to do with semantic accuracy. See n_stale_excluded below. A stale report also lists the entry on Entries to refresh.

cache_entry_id is optional — your own key for the cache entry candidate_answer came from (up to 256 characters). It groups feedback per entry for Entries to refresh, which hands it back so you can refresh or drop that entry directly. Without it, entries are grouped by answer text. Fine-tuning and monitoring don't use it.

verification_id is optional — the id from the /v1/verify response this feedback is about, returned while randomized auditing is on. It is how you answer an audit, and it is worth sending for a rejected call too (you called your LLM anyway, so that label is free). Linked rows come back as label_source: "audit" or "reject". Each id can be linked once (a second, different report is a 409; retrying with the same Idempotency-Key is not), an id that isn't yours is a 404, the query and candidate_answer must be the ones the id was issued for (otherwise a 409), and it requires an explicit was_correct rather than implicit_signals.

{
  "items": [
    { "query": "when will my order arrive", "candidate_answer": "Usually 2–3 days after shipping.", "was_correct": true },
    { "query": "when will my order arrive", "candidate_answer": "Ships within 48 hours.", "was_correct": false }
  ]
}

{ "ids": [42, 43] }

Local scoring (Team and Scale)

On a Managed plan you can run the verifier we fine-tuned for you inside your own stack, instead of calling /v1/verify: no network round trip on your request path, your query and answer texts stay with you, and local calls aren't metered. Recalibration, drift monitoring, audits and certification keep running on our side, on the feedback you send.

pip install "cacheverifier[local]"

from cacheverifier import LocalVerifier

cv = LocalVerifier(api_key="cv_...")   # downloads your live model in the background
r = cv.verify(query, cached_answer, similarity_score=0.87, cached_query=cached_question)
if r.approved:
    ...                                # serve from cache
else:
    answer = call_llm(query)
    if r.audit:                        # a randomized audit: report whether the cached answer was right
        cv.feedback(query, cached_answer, was_correct=same_answer(answer, cached_answer), result=r)

• It decides exactly as /v1/verify would with the same model. Every model download comes with reference verdicts from the service; a model that doesn't reproduce them on your machine is not used.
• When a newer model goes live (after a fine-tune or a scheduled recalibration), the client downloads it, checks it the same way, and switches over without a restart.
• While auditing is on, each local decision is uploaded as scores, verdicts and a keyed hash of the pair — never the texts — so audits and GET /v1/monitor/risk work as they do for hosted calls.
• The audit draw happens in the client. The risk bound assumes it is independent of whether the answer is right, which the published client guarantees unless it is modified.
• Until a model is ready (first download, or no fine-tune yet), calls go to the hosted API by default, or are rejected with fallback="reject". The online ACI threshold isn't available locally yet.

The endpoints it uses, if you're writing your own client: GET /v1/models/active (manifest: files with hashes, decision rules, reference verdicts), GET /v1/models/{id}/files/{name} (resumable downloads), and POST /v1/verify/decisions (upload decisions, get back the verification_ids to send with feedback).

Local Health Check (offline)

The hosted Health Check Report runs the stock-vs-fine-tuned AUC comparison for you, but it needs a sample of your real cache hits uploaded to run it. If that's a blocker — a compliance review, or just not wanting production traffic leaving your network on day one — run the identical evaluation on your own machine instead. Nothing is sent: your queries and answers never leave the host, and the only thing that can leave is an aggregate summary (a handful of floats and counts, no text) that you explicitly choose to write out.

It's the healthcheck subcommand of the cacheverifier Python package. It's an opt-in extra — the base client is httpx-only; the extra adds torch and sentence-transformers and runs the same evaluation as the hosted dry-run, on CPU.

pip install "cacheverifier[healthcheck]"

# rows: {"query": "...", "candidate_answer": "...", "was_correct": true}
#   (optional per row: "stale": true)
# order matters -- the train/calibrate/test split is chronological
cacheverifier healthcheck traffic.jsonl

# or, to also write an aggregate-only summary you can share for a human read:
cacheverifier healthcheck traffic.jsonl --emit-summary summary.json

CacheVerifier local Health Check  --  traffic.jsonl
==================================================================
rows read:            4200
usable (non-stale):   4187

fine-tuning locally (nothing sent) -> /tmp/cacheverifier_healthcheck_xxxx
this takes a few minutes on CPU; the base model downloads once on first run...

results
------------------------------------------------------------------
  train / calibrate / test:         3349 / 419 / 419
  stock verifier   held-out AUC:    0.6120
  fine-tuned       held-out AUC:    0.7080   (delta +0.0960)
  label-noise proxy (disagreement): 11.4%
  train / test positive rate:       48.0% / 47.0%
  ceiling status:                   still_improvable

verdict
------------------------------------------------------------------
  IMPROVED   -- fine-tuning on your own data helps this traffic

The first run downloads the ~90 MB base model from Hugging Face; after that it's fully offline. No GPU needed. For what the numbers mean, see the Research page (label-noise tolerance, cold start, and the "vs. raising the threshold" comparison are all there). The reference implementation also lives as scripts/local_health_check.py in the service repo, if you'd rather run it from a source checkout.

Fine-tune Jobs

{
  "id": 7,
  "status": "queued",
  "n_examples": 214,
  "n_stale_excluded": 0,
  "n_low_confidence_excluded": 0,
  "warning": null,
  "result_model_version": null,
  "auc_baseline": null,
  "auc_tuned": null,
  "train_label_disagreement_rate": null,
  "ceiling_status": null,
  "ceiling_precision_at_top_decile": null,
  "calibrate_positive_rate": null,
  "calibration_rate_drift_detected": null,
  "threshold_hit_rate": null,
  "threshold_error_rate": null,
  "previous_threshold_hit_rate": null,
  "previous_threshold_error_rate": null,
  "threshold_by_cost_ratio": null,
  "threshold_cost_ratio": null,
  "model_status": null,
  "certified_risk_target": null,
  "certified_threshold": null,
  "certified_pool_n": null,
  "certified_pool_candidate_concentration": null,
  "certification_note": null,
  "certification_status": null,
  "test_n": null,
  "threshold": null,
  "threshold_calibrated": null,
  "error": null,
  "dry_run": false
}

{
  "examples": [
    { "query": "when will my order arrive", "candidate_answer": "In-stock items ship within 48 hours and usually arrive 2–3 days after shipping.", "was_correct": true },
    { "query": "when will my order arrive", "candidate_answer": "In-stock items ship within 48 hours.", "was_correct": false }
  ]
}

{
  "id": 8,
  "status": "queued",
  "n_examples": 20,
  "n_stale_excluded": 0,
  "n_low_confidence_excluded": 0,
  "warning": null,
  "result_model_version": null,
  "auc_baseline": null,
  "auc_tuned": null,
  "train_label_disagreement_rate": null,
  "ceiling_status": null,
  "ceiling_precision_at_top_decile": null,
  "calibrate_positive_rate": null,
  "calibration_rate_drift_detected": null,
  "threshold_hit_rate": null,
  "threshold_error_rate": null,
  "previous_threshold_hit_rate": null,
  "previous_threshold_error_rate": null,
  "threshold_by_cost_ratio": null,
  "threshold_cost_ratio": null,
  "model_status": null,
  "certified_risk_target": null,
  "certified_threshold": null,
  "certified_pool_n": null,
  "certified_pool_candidate_concentration": null,
  "certification_note": null,
  "certification_status": null,
  "test_n": null,
  "threshold": null,
  "threshold_calibrated": null,
  "error": null,
  "dry_run": true
}

[
  { "id": 7, "status": "done", "n_examples": 214, "n_stale_excluded": 6, "n_low_confidence_excluded": 3, "result_model_version": 3, "auc_baseline": 0.71, "auc_tuned": 0.86, "train_label_disagreement_rate": 0.06, "ceiling_status": "still_improvable", "ceiling_precision_at_top_decile": 0.94, "calibrate_positive_rate": 0.31, "calibration_rate_drift_detected": false, "threshold_hit_rate": 0.31, "threshold_error_rate": 0.08, "previous_threshold_hit_rate": 0.29, "previous_threshold_error_rate": 0.09, "threshold_by_cost_ratio": [{ "cost_ratio": 0.5, "threshold": 1.42, "hit_rate": 0.38, "error_rate": 0.05 }, { "cost_ratio": 1, "threshold": 0.98, "hit_rate": 0.31, "error_rate": 0.03 }, { "cost_ratio": 2, "threshold": 0.61, "hit_rate": 0.24, "error_rate": 0.02 }, { "cost_ratio": 5, "threshold": 0.05, "hit_rate": 0.11, "error_rate": 0.0 }, { "cost_ratio": 10, "threshold": null, "hit_rate": 0.0, "error_rate": null }], "threshold_cost_ratio": null, "model_status": "active", "certified_risk_target": 0.01, "certified_threshold": 2.14, "certified_pool_n": 128, "certified_pool_candidate_concentration": 0.07, "certification_note": "Certified: based on 128 of your own past feedback examples, this threshold is expected to keep the false-reuse rate at or below 1.0% — as long as your future traffic looks statistically similar to the feedback you've submitted so far.", "certification_status": "certified", "joint_decision_status": "not_attempted", "joint_decision_note": "…", "base_model": "cross-encoder/ms-marco-MiniLM-L6-v2", "n_adversarial_added": 0, "test_n": 43, "threshold": 0.184, "threshold_calibrated": true, "warning": null, "error": null, "dry_run": false }
]

{
  "id": 7,
  "status": "done",
  "n_examples": 214,
  "n_stale_excluded": 6,
  "n_low_confidence_excluded": 0,
  "warning": null,
  "result_model_version": 3,
  "auc_baseline": 0.71,
  "auc_tuned": 0.86,
  "train_label_disagreement_rate": 0.06,
  "ceiling_status": "still_improvable",
  "ceiling_precision_at_top_decile": 0.94,
  "calibrate_positive_rate": 0.31,
  "calibration_rate_drift_detected": false,
  "threshold_hit_rate": 0.31,
  "threshold_error_rate": 0.08,
  "previous_threshold_hit_rate": 0.29,
  "previous_threshold_error_rate": 0.09,
  "threshold_by_cost_ratio": [
    { "cost_ratio": 0.5, "threshold": 1.42, "hit_rate": 0.38, "error_rate": 0.05 },
    { "cost_ratio": 1, "threshold": 0.98, "hit_rate": 0.31, "error_rate": 0.03 },
    { "cost_ratio": 2, "threshold": 0.61, "hit_rate": 0.24, "error_rate": 0.02 },
    { "cost_ratio": 5, "threshold": 0.05, "hit_rate": 0.11, "error_rate": 0.0 },
    { "cost_ratio": 10, "threshold": null, "hit_rate": 0.0, "error_rate": null }
  ],
  "threshold_cost_ratio": null,
  "model_status": "active",
  "certified_risk_target": 0.01,
  "certified_threshold": 2.14,
  "certified_pool_n": 128,
  "certified_pool_candidate_concentration": 0.07,
  "certification_note": "Certified: based on 128 of your own past feedback examples, this threshold is expected to keep the false-reuse rate at or below 1.0% — as long as your future traffic looks statistically similar to the feedback you've submitted so far.",
  "certification_status": "certified",
  "test_n": 43,
  "threshold": 0.184,
  "threshold_calibrated": true,
  "error": null,
  "requested_at": "2026-10-25T05:29:26Z",
  "started_at": "2026-10-25T05:29:27Z",
  "completed_at": "2026-10-25T05:34:22Z",
  "steps_done": 480,
  "steps_total": 480,
  "cancel_requested": false,
  "dry_run": false
}

{
  "id": 8,
  "status": "running",
  "cancel_requested": true,
  "steps_done": 120,
  "steps_total": 480,
  ...
}

[
  { "id": 3, "job_id": 7, "status": "active", "base_model": "cross-encoder/nli-MiniLM2-L6-H768", "created_at": "2026-10-20T08:00:00", "deactivated_at": null, "train_n": 171, "test_n": 43, "auc_baseline": 0.71, "auc_tuned": 0.86, "threshold": 0.184, "threshold_calibrated": true, "threshold_hit_rate": 0.31, "threshold_error_rate": 0.08, "test_positive_rate": 0.33, "last_activated_at": "2026-10-20T08:00:05", "last_activated_by": "auto, fine-tune job #7", "hold_reasons": [], "notes": [] },
  { "id": 2, "job_id": 5, "status": "superseded", "base_model": "cross-encoder/nli-MiniLM2-L6-H768", "created_at": "2026-10-13T08:00:00", "deactivated_at": "2026-10-20T08:00:00", "train_n": 120, "test_n": 30, "auc_baseline": 0.70, "auc_tuned": 0.83, "threshold": 0.201, "threshold_calibrated": true, "threshold_hit_rate": 0.29, "threshold_error_rate": 0.11, "test_positive_rate": 0.32, "last_activated_at": "2026-10-13T08:00:04", "last_activated_by": "auto, fine-tune job #5", "hold_reasons": [], "notes": [] }
]

{
  "id": 3,
  "is_active": true,
  "threshold": 0.184
}

Drift Monitor

A fine-tuned verifier is calibrated against the traffic it was trained on — a new user segment, a product launch, or a rewritten support macro can quietly shift that traffic until the same threshold stops matching reality. That's model drift, and it's the one failure mode this endpoint is purpose-built to catch: a narrow slice of AI observability for the verifier's own gray zone, not a general-purpose LLM monitoring dashboard. Three independent statistical tests run against the training-window baseline — a chunked z-test and a Page-Hinkley test on the gray-zone positive rate, plus a two-sample KS test on the verifier's own score distribution — so a single noisy chunk of feedback doesn't trigger a false alarm on its own, and a covariate shift that moves scores without moving the label error rate (Paper D §4.2) still gets caught. Any one test flagging marks the tenant flagged.

{
  "status": "stable",
  "n_labels": 96,
  "baseline_positive_rate": 0.31,
  "chunked_z_test": {
    "baseline_rate": 0.31,
    "n_baseline": 214,
    "chunks": [ { "chunk": 0, "n": 12, "positive_rate": 0.33, "z": 0.2, "p_value": 0.84, "flagged": false } ],
    "first_flagged_chunk": null
  },
  "page_hinkley": { "flagged": false, "flag_index": null, "final_statistic": 0.04 },
  "score_drift": {
    "statistic": 0.07, "p_value": 0.42, "flagged": false,
    "n_baseline": 500, "n_recent": 96,
    "baseline_mean": 1.83, "recent_mean": 1.79
  }
}

score_drift is null until you have at least 30 recent feedback rows carrying a verifier score (recorded automatically on every POST /v1/feedback once you're on a fine-tuned model) — and for models fine-tuned before this check existed. The score-distribution shift it catches is exactly the one that can quietly break a certified_threshold's risk guarantee, so a flag here also flips a certification's certification_status to stale_recalibration_pending.

Entries to refresh

The verifier catches a bad hit when it's served; this list tells you which cache entries to fix so they stop coming up. It reads your feedback, per cache entry, and lists an entry when someone reported its answer stale, or when it has been reported wrong repeatedly and more often than right — the answer itself is wrong, or the entry keeps matching questions it doesn't answer. Available on every plan, and in the dashboard under Entries to refresh.

curl https://www.cacheverifier.com/v1/monitor/entries-to-refresh   -H "X-API-Key: $CACHEVERIFIER_API_KEY"

{"entries": [{
  "key": "3f9c0d2a6b1e4f8a9c7d5e3b1a2f4c6d",
  "cache_entry_id": "faq:shipping",
  "reason": "stale",
  "candidate_answer": "Standard parcels arrive in 3-5 business days.",
  "cached_query": "how long does shipping take",
  "stale_reports": 1, "wrong_reports": 0, "correct_reports": 4,
  "first_reported_at": "2026-10-30T09:12:00Z", "last_reported_at": "2026-10-30T09:12:00Z",
  "sample_queries": ["when will my parcel arrive"]
}]}

curl -X POST https://www.cacheverifier.com/v1/monitor/entries-to-refresh/3f9c0d2a6b1e4f8a9c7d5e3b1a2f4c6d/resolve   -H "X-API-Key: $CACHEVERIFIER_API_KEY"

• Only an entry's current answer counts: once feedback for a cache_entry_id arrives with a new answer text, reports about the old one stop counting, so a refreshed entry drops off by itself.
• resolve marks it refreshed (or dropped): earlier feedback stops counting, and it's listed again only if new reports arrive.
• On Scale, newly listed entries are also pushed to your alert webhook, once each: {"event": "entries_to_refresh", "entries": [{"key", "cache_entry_id", "reason"}]} plus a Slack-compatible text. Keys and reasons only — answer and query texts stay behind the API.

Randomized audit

Feedback you volunteer is not a random sample: a wrong answer is likelier to get reported than a right one, so no error rate computed from it can be promised. And the calls that matter most, the ones you served from cache, are exactly the ones you never learn the truth about. Randomized auditing closes that gap. With it on, the service picks a random fraction of approved verdicts and returns them as approved: false, audit: true. You serve a fresh LLM answer as you would on any miss, so the user never gets an unverified answer, and then report whether the cached answer would have been right. Because the pick is random and doesn't depend on the score or the text, those audits are a simple random sample of approved calls, and GET /v1/monitor/risk can put an exact upper bound on how often an approved call is wrong.

{ "audit_rate": 0.05 }

{ "audit_rate": 0.05, "min_nonzero_audit_rate": 0.01 }

resp = requests.post(f"{BASE}/v1/verify", headers=H,
                     json={"query": user_query, "candidate_answer": candidate.answer}).json()
if resp["approved"]:
    answer = candidate.answer
else:
    answer = call_real_llm(user_query)
    if resp["audit"] or not resp["verifier_approved"]:
        # You called the LLM anyway: say whether the cached answer would have been right.
        # Any equivalence check works; an LLM judge comparing the two answers is the usual choice.
        requests.post(f"{BASE}/v1/feedback", headers=H, json={
            "query": user_query, "candidate_answer": candidate.answer,
            "was_correct": same_meaning(candidate.answer, answer),
            "verification_id": resp["verification_id"],
        })

A client that ignores audit stays safe: it just sees an extra miss. Report every audit you get. With /v1/verify/batch ranked candidates, treat the whole lookup as a miss if any result has audit: true; moving on to a lower-ranked approved candidate would leave the audit unanswered. Each audit costs one extra LLM call. To pin an approved-call error rate r down to ±h you need about 1.96² × r(1−r) / h² reported audits, for example about 460 for r = 5% and h = 2 points, so choose the rate that gets there within the window you care about at your approved-call volume.

{
  "status": "ok",
  "window_days": 7,
  "confidence": 0.95,
  "audit_rate": 0.05,
  "n_decisions": 25000,
  "n_approved": 20000,
  "n_audits_issued": 1000,
  "n_audits_returned": 980,
  "n_audit_wrong": 31,
  "n_audit_stale": 2,
  "audit_completion_rate": 0.98,
  "approved_error_rate": 0.0316,
  "approved_error_rate_upper": 0.0425,
  "approved_error_rate_upper_conservative": 0.0605,
  "approval_rate": 0.8,
  "gray_zone_error_rate_upper": 0.034,
  "note": "Bounds assume whether an audit gets reported back is unrelated to whether the cached answer was right."
}

approved_error_rate_upper is an exact one-sided (Clopper-Pearson) bound. It assumes that whether you report an audit has nothing to do with how it turned out. approved_error_rate_upper_conservative drops that assumption by counting every unreported audit as wrong; rely on it when audit_completion_rate is low. gray_zone_error_rate_upper multiplies the bound by approval_rate, giving the share of all gray-zone requests you send to /v1/verify that end up served a wrong cached answer. Stale answers count as wrong here, unlike in fine-tuning and drift monitoring, because the user still got an outdated answer. status is "no_audit" while auditing is off (there is no valid estimate) and "insufficient" below 100 reported audits in the window (the bound is valid but wide). Only decisions made by your current model are counted. "Wrong" means whatever your equivalence check says, so an LLM judge's own mistakes flow into the estimate.

Online adaptive threshold (ACI)

A certified threshold is fixed when the fine-tune runs, and real traffic drifts away from it (on time-ordered benchmark data its realized risk ran 1.4–1.8× over target on later traffic). Opt in to ACI and every verdict is decided against a threshold that keeps moving instead: a little lower after each verdict, higher by step / audit_rate after each audited approval, and back down by the same amount when that audit's feedback says the cached answer was correct. Over time this holds the share of gray-zone requests served a wrong answer near target_risk without assuming your traffic stays put. While on, it replaces your fine-tuned threshold and any joint rule (decision_rule: "aci"; threshold in the response is the live ACI value on the score scale). It needs auditing on, and an audit you never report counts as wrong, so unreported audits make it stricter, not looser. A new active model, or turning it back on, restarts it from that model's own threshold.

{ "enabled": true, "target_risk": 0.05, "step": 0.01 }

{ "enabled": true, "target_risk": 0.05, "step": 0.01, "current_threshold": null }

Gray-zone threshold

tau_low/tau_high — the boundaries of your own cache's gray zone, above which a candidate is trusted without ever calling /v1/verify — are entirely your own cache backend's decision; this service never sees or sets them (see the Integration pattern above). What it can tell you is what to do with them: a fine-tuned verifier's auc_tuned (Fine-tune Jobs) says how much you can trust it, but not how that translates into a tau_high value. This endpoint closes that gap by replaying your own feedback against your current verifier: for a grid of candidate tau_high values, it scores what decision accuracy you'd get if everything above that cutoff were served blindly and everything below it were sent through /v1/verify, then recommends whichever value scores highest.

{
  "status": "ok",
  "n_labels": 214,
  "baseline_accuracy": 0.81,
  "recommended_tau_high": 0.87,
  "recommended_accuracy": 0.93,
  "grid": [
    { "tau_high": 0.75, "n_above": 180, "n_below": 34, "accuracy": 0.88 },
    { "tau_high": 0.87, "n_above": 96, "n_below": 118, "accuracy": 0.93 },
    { "tau_high": 0.95, "n_above": 12, "n_below": 202, "accuracy": 0.90 }
  ],
  "label_lint_rows_checked": 150,
  "label_lint_same_question_marked_wrong": 0,
  "label_lint_same_words_reordered_marked_wrong": 0,
  "label_lint_note": null,
  "similarity_p_value": 0.005,
  "similarity_note": null
}

status: "insufficient_data" (with a null recommendation) means fewer than 20 of your feedback rows had similarity_score set — nothing to reject, just not enough signal yet. baseline_accuracy is what you'd get with tau_high set above every candidate (verification never runs); grid is every value tried, not just the winner, so you can see how flat or peaked the curve is around the recommendation rather than trusting one number blindly — same reasoning auc_baseline/auc_tuned get surfaced instead of a bare verdict.

The label_lint_* fields run the same check as label_check on feedback over the rows used here that carried a cached_query. Wrong labels on what is really the same question push the recommended tau_high up: fewer direct hits, more verify calls, for no real gain. When any are found, label_lint_note says so; review a few of those rows before adopting the recommendation.

The best of ~19 grid points beats baseline_accuracy a little even when similarity carries no information, so similarity_p_value reruns the search with your similarity scores shuffled across rows: it is the share of shuffles that do as well. At 0.05 or above, similarity_note says your scores don't separate right from wrong answers in this feedback, and the recommendation then reflects the verifier alone.

Admin

{ "name": "acme" }

{
  "tenant_id": 1,
  "api_key": "cv_hSDoVrYkFJo2CvoaL1_N9SSQDOWjW6NMSV4DPj9FQ5g"
}