API Reference

Quickstart

From zero to a verified call in two requests. Swap in your own deployment's base URL.

curl -X POST https://www.cacheverifier.com/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "email": "[email protected]",
    "password": "at-least-8-chars",
    "tenant_name": "acme"
  }'

# => { "token": "...", "api_key": "cv_...", "user": { ... } }
# Save api_key now -- it's only ever returned this once.

curl -X POST https://www.cacheverifier.com/v1/verify \
  -H "Content-Type: application/json" \
  -H "X-API-Key: cv_..." \
  -d '{
    "query": "how do I cancel my subscription",
    "candidate_answer": "Go to Settings > Billing > Cancel."
  }'

# => { "approved": true, "score": 1.202, "latency_ms": 17.5, "model_version": "stock", "threshold": 0.0, "decision_rule": "threshold" }

From here: send was_correct ground truth to /v1/feedback as you get it, and once you have a couple hundred rows, kick off a fine-tune job — or just run the Health Check Report in the dashboard, which does the feedback-upload-and-fine-tune loop for you on a sample.

Integration pattern

CacheVerifier doesn't replace your cache — you already have one (Redis, GPTCache, a vector database like Pinecone or Weaviate, a RAG pipeline's retrieval cache, your own). The problem it solves is the gray zone: a candidate that clears your cosine similarity threshold — typically somewhere around 0.8–0.9 — without actually being right. Raising that cosine similarity threshold doesn't fix it either: push it high enough to kill the false positives and you start rejecting real duplicate questions, because near-miss and true-match pairs often sit within a few hundredths of each other in cosine similarity. Serving a false positive is worse than a cache miss — from the user's side, a confidently wrong cached answer looks a lot like a hallucination, except your system had the right answer on file the whole time. A worked example — a support bot backed by a similarity-search cache:

user_query = "how do I cancel my subscription"
candidate = my_cache.search(user_query)
# candidate.answer = "Go to Settings > Billing > Pause subscription for a month."
# candidate.similarity = 0.86 -- close, but "cancel" vs. "pause" is a real difference

resp = requests.post(
    "https://www.cacheverifier.com/v1/verify",
    headers={"X-API-Key": API_KEY},
    json={"query": user_query, "candidate_answer": candidate.answer},
).json()
# => { "approved": false, "score": 0.41, "threshold": 0.5, ... }

if resp["approved"]:
    answer = candidate.answer          # cache hit confirmed -- skip the LLM call
else:
    answer = call_real_llm(user_query) # not trustworthy -- fall through, then re-cache
    my_cache.store(user_query, answer)

requests.post(
    "https://www.cacheverifier.com/v1/feedback",
    headers={"X-API-Key": API_KEY},
    json={"query": user_query, "candidate_answer": answer, "was_correct": True},
)

When the verify call itself fails. /v1/verify is on your request path, so decide up front what a timeout or a 5xx means for you. The safe default is fail closed: treat an errored verify like approved: false and fall through to your LLM — you pay for one regeneration, never serve an unverified near-miss. Only fail open (serve the cached answer anyway) if you've decided a stale-or-near-miss answer is acceptable for that traffic. Warm calls are tens of milliseconds server-side (see the FAQ for measured percentiles), but the first call after a deploy is slower while the model loads, so a ceiling around 1s with fail-closed handling covers both. The cacheverifier Python client (0.3+) does this by default — verify() uses a 1 s timeout and returns approved: false with degraded=true on a timeout or 5xx; pass fail_open=True to serve the cached answer instead, or verify_timeout= to change the ceiling. Calling the REST API directly, you set both yourself. There is no formal uptime SLA yet (see the Terms), which is another reason to make the fallback path explicit.

From there, either call POST /v1/finetune/jobs yourself once feedback builds up (self-serve, free forever), or subscribe to a Managed plan and let it happen on a schedule automatically, with drift alerts if the gray zone starts behaving differently than it did at training time.

Python client. pip install cacheverifier gives you a typed client for every endpoint on this page — cv.verify(...), cv.feedback(...), cv.finetune(). Source, examples, and the GPTCache adapter are in cacheverifier-python.

Already on GPTCache? pip install "cacheverifier[gptcache]", then pass CacheVerifierEvaluation(api_key=...) as GPTCache's similarity_evaluation= argument — the if/else above happens inside GPTCache's own pipeline, no extra code required.

Prefer no dependencies? examples/quickstart.py is this same worked example as one runnable script using only requests, and registers a free tenant for you if you don't already have an API key.

Authentication

Tenant-scoped endpoints (verify, feedback, fine-tune, drift) accept either credential — whichever is more convenient for the caller:

• X-API-Key: cv_... — a raw tenant API key, for server-to-server integrations. Issued once at registration or by an admin; only the hash is stored, so it can't be recovered if lost.
• Authorization: Bearer <token> — a dashboard session token from /v1/auth/login or /v1/auth/register, valid for 24 hours.

X-Admin-Token is separate and only guards POST /v1/admin/tenants (internal tenant provisioning).

Rate limits

Enforced at the edge, keyed by X-API-Key when present (so multiple tenants behind the same NAT/corporate IP don't share a bucket), falling back to client IP for unauthenticated calls and dashboard session-token requests. Exceeding a limit returns a plain 429 with no body — back off and retry.

The general limit's burst allowance is sized for bulk flows like the Health Check Report, which submits many /v1/feedback rows back-to-back — a single run of a hundred or so rows won't get throttled. The auth limit is intentionally tight: it's there to slow down credential stuffing and mass signup, not to accommodate normal traffic.

Auth

{
  "email": "[email protected]",
  "password": "at-least-8-chars",
  "tenant_name": "acme"
}

{
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "api_key": "cv_hSDoVrYkFJo2CvoaL1_N9SSQDOWjW6NMSV4DPj9FQ5g",
  "user": { "id": 1, "email": "[email protected]", "tenant_id": 1 }
}

{ "email": "[email protected]", "password": "at-least-8-chars" }

{
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "user": { "id": 1, "email": "[email protected]", "tenant_id": 1 }
}

{ "id": 1, "email": "[email protected]", "tenant_id": 1 }

{ "api_key": "cv_hSDoVrYkFJo2CvoaL1_N9SSQDOWjW6NMSV4DPj9FQ5g" }

rotate-key requires a dashboard login (Authorization: Bearer), not the API key itself — otherwise a leaked key could keep itself alive. Update any server-to-server integration with the new key immediately; the old one stops working the instant this call succeeds.

Verify

{
  "query": "how do I cancel my subscription",
  "candidate_answer": "Go to Settings > Billing > Cancel.",
  "similarity_score": 0.91,             // optional
  "cached_query": "how do I pause my subscription"  // optional
}

{
  "approved": true,
  "score": 1.202,
  "latency_ms": 17.5,
  "model_version": "stock",
  "threshold": 0.0,
  "decision_rule": "threshold"
}

similarity_score is optional — your own cache backend's similarity for this candidate. It only affects the verdict when your fine-tuned verifier carries a joint decision rule: on domains where the verifier alone is weak (held-out AUC in the ~0.60–0.68 band), a fine-tune fits a 2-feature logistic over (similarity, verifier score) and keeps it only if it beat the verifier score alone on held-out data (CacheVerifier Paper D §4.5). When that rule is active and you send asimilarity_score, decision_rule in the response reads "joint"instead of "threshold", and threshold is no longer the operative cutoff. Send it unconditionally — it's ignored (decision_rule: "threshold") for every tenant that doesn't have such a rule. "cold_start" is the third value, before you have a model of your own.

cached_query is optional — the question candidate_answer was originally the answer to, before your cache retrieved it for this query. When your verifier was fine-tuned with it (see Feedback), a word-level diff of it against query ("removed: pause  |  added: cancel") is folded into the score — CacheVerifier Paper D §4.3 found the raw text backfires but "what changed" is exactly the signal a contradictory-edit hard case needs. Harmless to send to a verifier that wasn't trained with it.

latency_ms is the cross-encoder forward pass only (~17.5 ms p50). A full call on our side — nginx, auth, tenant lookup, rate-limit, serialization, inference — measured p50 33 ms / p95 44 ms / p99 47 ms on a loopback probe of the origin (300 warm calls, 2026-09-04). Your network round trip to us is on top of that and depends on where you call from, so measure end-to-end against your own traffic. One reference point: 150 calls from a US-East client measured a full round trip of p50 ~33 ms / p95 ~39 ms (2026-09-09) — essentially the same as the server-only figure above, i.e. network overhead was close to zero on that specific path. That's a single geography and run, not a guarantee for yours. The first call after a deploy is slower while the model loads.

threshold is the score >= threshold cutoff this call was decided against —0.0 on the shared stock model, tenant-specific once a fine-tune job has calibrated one (see Fine-tune Jobs below).

model_version tells you which verifier decided the call: "v<id>" for your own fine-tuned model, "stock" for the shared un-tuned cross-encoder, or — before you have a fine-tuned model — one of two cold-start values that ran no inference and always returned approved: false: "cold_start_auto_pending" (the default: run a Health Check so we can tell whether the stock model is accurate enough on your data to serve — AUC ≥ 0.68 flips this on automatically) or "cold_start_fail_closed" (a Health Check came back below that bar, or you set cold-start mode to fail_closed explicitly). PUT /v1/settings/cold-start-mode overrides the default in either direction. The Python client adds one more value client-side — "verify_unavailable" on the synthetic result it returns when a verify call times out or 5xxes (see the fail-closed note above); the API itself never sends it.

query and candidate_answer together share a combined budget of about 128 tokens for scoring, not 128 each — longer text is truncated before it reaches the model, not rejected. Truncation isn't a blind cutoff either: when a pair doesn't fit, each field keeps its own opening and closing content and loses the middle, and query is guaranteed a minimum share so an unusually long query (e.g. multi-turn history used as the query) isn't crowded out entirely by an equally long candidate_answer. This is a real limit, not a formality: a long conversational transcript that gets cut off mid-thought can genuinely score differently than the full text would. If your candidates are long (support-chat transcripts, multi-turn context), this truncation already keeps the most likely-relevant parts automatically — but for very long inputs, trimming to the specific relevant span yourself before calling this endpoint will still preserve more signal than any generic heuristic can.

{
  "items": [
    { "query": "how do I cancel", "candidate_answer": "Settings > Billing > Cancel." },
    { "query": "how do I pay", "candidate_answer": "Settings > Billing > Payment methods." }
  ]
}

{
  "results": [
    { "approved": true, "score": 1.202, "latency_ms": 23.5, "model_version": "stock", "threshold": 0.0, "decision_rule": "threshold" },
    { "approved": true, "score": 0.98, "latency_ms": 23.5, "model_version": "stock", "threshold": 0.0, "decision_rule": "threshold" }
  ]
}

The whole batch is quota-checked and billed as a single unit: admitted only if capacity covers every item, rejected entirely otherwise — never partially processed. latency_ms on a batch response is the whole batch's time, not a true per-item measurement (there's no way to isolate one pair's share of a single forward pass). Each item may carry its own optional similarity_score, same meaning as on /v1/verify.

candidates = my_cache.search(user_query, top_k=2)  # ranked, most similar first

resp = requests.post(
    "https://www.cacheverifier.com/v1/verify/batch",
    headers={"X-API-Key": API_KEY},
    json={"items": [
        {"query": user_query, "candidate_answer": c.answer} for c in candidates
    ]},
).json()

hit = next((r for r, c in zip(resp["results"], candidates) if r["approved"]), None)
answer = hit and candidates[resp["results"].index(hit)].answer or call_real_llm(user_query)

Feedback

{
  "query": "how do I cancel my subscription",
  "candidate_answer": "Go to Settings > Billing > Cancel.",
  "was_correct": true,
  "similarity_score": 0.86
}

{
  "id": 42,
  "was_correct": true,
  "label_confidence": 1.0,
  "label_source": "explicit"
}

Pass an Idempotency-Key header to make retries safe — retrying the same key returns the original row's id instead of creating a duplicate label. Omit it and the same (query, candidate_answer) pair is free to get independent feedback rows, which is the common case (different users hitting the same cached answer).

If you don't have was_correct — most teams running a semantic cache don't have a clean "was this answer right" signal — send an implicit_signals object instead (exactly one of the two; both, or neither, is a 422). Set any of the proxies you can observe and the service derives was_correct plus a confidence, echoed back on the response:

{
  "query": "how do I cancel my subscription",
  "candidate_answer": "Go to Settings > Billing > Pause.",
  "implicit_signals": {
    "reasked_within_session": true,   // user asked the same thing again
    "escalated_to_human": false,      // handed off to a human agent
    "thumbs_down": false,             // explicit negative rating
    "thumbs_up": false,               // explicit positive rating
    "resolved_without_followup": false, // session ended clean, no re-ask
    "session_abandoned": false        // user left mid-flow
  }
}
// -> { "id": 43, "was_correct": false,
//      "label_confidence": 0.6, "label_source": "implicit:reask" }

A negative proxy outranks a positive one (a thumbs-up that's then re-asked is, on balance, a wrong reuse). Strong proxies — an explicit rating, an escalation, a re-ask — produce a confident label used exactly like was_correct. Weak ones on their own — a lone abandoned session, a lone clean resolution — land below the confidence floor: the row is still stored (and still counts toward your raw history) but is held out of fine-tune training, the fine-tune minimum, drift monitoring, and the gray-zone threshold recommendation, the same treatment a stale row gets. See n_low_confidence_excluded below.

similarity_score is optional — your own cache backend's raw similarity score (cosine similarity, in most implementations) for this candidate, if you still have it in scope when you learn was_correct. It isn't used by fine-tuning or drift monitoring; it's what GET /v1/monitor/gray-zone-threshold replays against to recommend a tau_high for your own cache's gray-zone boundary. Omit it and that row is simply excluded from that calculation — everything else about feedback works the same.

cached_query is optional — the question candidate_answer was originally cached for. When you send it, fine-tuning folds a word-level diff of it against query into the verifier's input (Paper D §4.3), and you then send the same cached_query on /v1/verify at serving time. If your cache tracks which stored question each hit came from, plumbing this through is the single highest-leverage input for hard "pause vs cancel" / "2023 vs 2024" cases.

Once you're on a fine-tuned model, the service also records its score for this pair at submission time — nothing for you to send. That stream is what the drift monitor's KS test compares against calibration, so a shift in the score distribution itself gets caught, not just a shift in how often the answer was right.

stale is optional (default false) — set it instead of leaving was_correct: false unqualified when the ONLY reason this candidate is wrong now is that it went stale (a price, date, or other fact changed), not a genuine semantic mismatch. This is a real, and different, correctness axis: the verifier judges semantic match from text alone and has no notion of time, so it can't infer staleness — it has to be caller-asserted, the same way was_correct itself is. Rows tagged stale are excluded from fine-tune training, drift monitoring, and the gray-zone threshold recommendation — a stale-caused "wrong" would otherwise teach the model (or a drift/threshold calculation) something that has nothing to do with semantic accuracy. See n_stale_excluded below.

{
  "items": [
    { "query": "how do I cancel", "candidate_answer": "Settings > Billing > Cancel.", "was_correct": true },
    { "query": "how do I cancel", "candidate_answer": "Settings > Billing > Pause.", "was_correct": false }
  ]
}

{ "ids": [42, 43] }

Local Health Check (offline)

The hosted Health Check Report runs the stock-vs-fine-tuned AUC comparison for you, but it needs a sample of your real cache hits uploaded to run it. If that's a blocker — a compliance review, or just not wanting production traffic leaving your network on day one — run the identical evaluation on your own machine instead. Nothing is sent: your queries and answers never leave the host, and the only thing that can leave is an aggregate summary (a handful of floats and counts, no text) that you explicitly choose to write out.

It's the healthcheck subcommand of the cacheverifier Python package. It's an opt-in extra — the base client is httpx-only; the extra adds torch and sentence-transformers and runs the same evaluation as the hosted dry-run, on CPU.

pip install "cacheverifier[healthcheck]"

# rows: {"query": "...", "candidate_answer": "...", "was_correct": true}
#   (optional per row: "stale": true)
# order matters -- the train/calibrate/test split is chronological
cacheverifier healthcheck traffic.jsonl

# or, to also write an aggregate-only summary you can share for a human read:
cacheverifier healthcheck traffic.jsonl --emit-summary summary.json

CacheVerifier local Health Check  --  traffic.jsonl
==================================================================
rows read:            4200
usable (non-stale):   4187

fine-tuning locally (nothing sent) -> /tmp/cacheverifier_healthcheck_xxxx
this takes a few minutes on CPU; the base model downloads once on first run...

results
------------------------------------------------------------------
  train / calibrate / test:         3349 / 419 / 419
  stock verifier   held-out AUC:    0.6120
  fine-tuned       held-out AUC:    0.7080   (delta +0.0960)
  label-noise proxy (disagreement): 11.4%
  train / test positive rate:       48.0% / 47.0%
  ceiling status:                   still_improvable

verdict
------------------------------------------------------------------
  IMPROVED   -- fine-tuning on your own data helps this traffic

The first run downloads the ~90 MB base model from Hugging Face; after that it's fully offline. No GPU needed — it's a 6-layer MiniLM cross-encoder. For what the numbers mean, see the Research page (label-noise tolerance, cold start, and the "vs. raising the threshold" comparison are all there). The reference implementation also lives as scripts/local_health_check.py in the service repo, if you'd rather run it from a source checkout.

Fine-tune Jobs

{
  "id": 7,
  "status": "queued",
  "n_examples": 214,
  "n_stale_excluded": 0,
  "n_low_confidence_excluded": 0,
  "warning": null,
  "result_model_version": null,
  "auc_baseline": null,
  "auc_tuned": null,
  "train_label_disagreement_rate": null,
  "ceiling_status": null,
  "ceiling_precision_at_top_decile": null,
  "calibrate_positive_rate": null,
  "calibration_rate_drift_detected": null,
  "threshold_hit_rate": null,
  "threshold_error_rate": null,
  "previous_threshold_hit_rate": null,
  "previous_threshold_error_rate": null,
  "threshold_by_cost_ratio": null,
  "threshold_cost_ratio": null,
  "model_status": null,
  "certified_risk_target": null,
  "certified_threshold": null,
  "certified_pool_n": null,
  "certified_pool_candidate_concentration": null,
  "certification_note": null,
  "certification_status": null,
  "test_n": null,
  "threshold": null,
  "threshold_calibrated": null,
  "error": null,
  "dry_run": false
}

{
  "examples": [
    { "query": "how do I cancel my subscription", "candidate_answer": "Go to Settings > Billing > Cancel subscription.", "was_correct": true },
    { "query": "how do I cancel my subscription", "candidate_answer": "Go to Settings > Billing > Pause subscription for a month.", "was_correct": false }
  ]
}

{
  "id": 8,
  "status": "queued",
  "n_examples": 20,
  "n_stale_excluded": 0,
  "n_low_confidence_excluded": 0,
  "warning": null,
  "result_model_version": null,
  "auc_baseline": null,
  "auc_tuned": null,
  "train_label_disagreement_rate": null,
  "ceiling_status": null,
  "ceiling_precision_at_top_decile": null,
  "calibrate_positive_rate": null,
  "calibration_rate_drift_detected": null,
  "threshold_hit_rate": null,
  "threshold_error_rate": null,
  "previous_threshold_hit_rate": null,
  "previous_threshold_error_rate": null,
  "threshold_by_cost_ratio": null,
  "threshold_cost_ratio": null,
  "model_status": null,
  "certified_risk_target": null,
  "certified_threshold": null,
  "certified_pool_n": null,
  "certified_pool_candidate_concentration": null,
  "certification_note": null,
  "certification_status": null,
  "test_n": null,
  "threshold": null,
  "threshold_calibrated": null,
  "error": null,
  "dry_run": true
}

[
  { "id": 7, "status": "done", "n_examples": 214, "n_stale_excluded": 6, "n_low_confidence_excluded": 3, "result_model_version": 3, "auc_baseline": 0.71, "auc_tuned": 0.86, "train_label_disagreement_rate": 0.06, "ceiling_status": "still_improvable", "ceiling_precision_at_top_decile": 0.94, "calibrate_positive_rate": 0.31, "calibration_rate_drift_detected": false, "threshold_hit_rate": 0.31, "threshold_error_rate": 0.08, "previous_threshold_hit_rate": 0.29, "previous_threshold_error_rate": 0.09, "threshold_by_cost_ratio": [{ "cost_ratio": 0.5, "threshold": 1.42, "hit_rate": 0.38, "error_rate": 0.05 }, { "cost_ratio": 1, "threshold": 0.98, "hit_rate": 0.31, "error_rate": 0.03 }, { "cost_ratio": 2, "threshold": 0.61, "hit_rate": 0.24, "error_rate": 0.02 }, { "cost_ratio": 5, "threshold": 0.05, "hit_rate": 0.11, "error_rate": 0.0 }, { "cost_ratio": 10, "threshold": null, "hit_rate": 0.0, "error_rate": null }], "threshold_cost_ratio": null, "model_status": "active", "certified_risk_target": 0.01, "certified_threshold": 2.14, "certified_pool_n": 128, "certified_pool_candidate_concentration": 0.07, "certification_note": "Certified: based on 128 of your own past feedback examples, this threshold is expected to keep the false-reuse rate at or below 1.0% — as long as your future traffic looks statistically similar to the feedback you've submitted so far.", "certification_status": "certified", "joint_decision_status": "not_attempted", "joint_decision_note": "…", "base_model": "cross-encoder/ms-marco-MiniLM-L6-v2", "n_adversarial_added": 0, "test_n": 43, "threshold": 0.184, "threshold_calibrated": true, "warning": null, "error": null, "dry_run": false }
]

{
  "id": 7,
  "status": "done",
  "n_examples": 214,
  "n_stale_excluded": 6,
  "n_low_confidence_excluded": 0,
  "warning": null,
  "result_model_version": 3,
  "auc_baseline": 0.71,
  "auc_tuned": 0.86,
  "train_label_disagreement_rate": 0.06,
  "ceiling_status": "still_improvable",
  "ceiling_precision_at_top_decile": 0.94,
  "calibrate_positive_rate": 0.31,
  "calibration_rate_drift_detected": false,
  "threshold_hit_rate": 0.31,
  "threshold_error_rate": 0.08,
  "previous_threshold_hit_rate": 0.29,
  "previous_threshold_error_rate": 0.09,
  "threshold_by_cost_ratio": [
    { "cost_ratio": 0.5, "threshold": 1.42, "hit_rate": 0.38, "error_rate": 0.05 },
    { "cost_ratio": 1, "threshold": 0.98, "hit_rate": 0.31, "error_rate": 0.03 },
    { "cost_ratio": 2, "threshold": 0.61, "hit_rate": 0.24, "error_rate": 0.02 },
    { "cost_ratio": 5, "threshold": 0.05, "hit_rate": 0.11, "error_rate": 0.0 },
    { "cost_ratio": 10, "threshold": null, "hit_rate": 0.0, "error_rate": null }
  ],
  "threshold_cost_ratio": null,
  "model_status": "active",
  "certified_risk_target": 0.01,
  "certified_threshold": 2.14,
  "certified_pool_n": 128,
  "certified_pool_candidate_concentration": 0.07,
  "certification_note": "Certified: based on 128 of your own past feedback examples, this threshold is expected to keep the false-reuse rate at or below 1.0% — as long as your future traffic looks statistically similar to the feedback you've submitted so far.",
  "certification_status": "certified",
  "test_n": 43,
  "threshold": 0.184,
  "threshold_calibrated": true,
  "error": null,
  "dry_run": false
}

{
  "id": 3,
  "is_active": true,
  "threshold": 0.184
}

Drift Monitor

A fine-tuned verifier is calibrated against the traffic it was trained on — a new user segment, a product launch, or a rewritten support macro can quietly shift that traffic until the same threshold stops matching reality. That's model drift, and it's the one failure mode this endpoint is purpose-built to catch: a narrow slice of AI observability for the verifier's own gray zone, not a general-purpose LLM monitoring dashboard. Three independent statistical tests run against the training-window baseline — a chunked z-test and a Page-Hinkley test on the gray-zone positive rate, plus a two-sample KS test on the verifier's own score distribution — so a single noisy chunk of feedback doesn't trigger a false alarm on its own, and a covariate shift that moves scores without moving the label error rate (Paper D §4.2) still gets caught. Any one test flagging marks the tenant flagged.

{
  "status": "stable",
  "n_labels": 96,
  "baseline_positive_rate": 0.31,
  "chunked_z_test": {
    "baseline_rate": 0.31,
    "n_baseline": 214,
    "chunks": [ { "chunk": 0, "n": 12, "positive_rate": 0.33, "z": 0.2, "p_value": 0.84, "flagged": false } ],
    "first_flagged_chunk": null
  },
  "page_hinkley": { "flagged": false, "flag_index": null, "final_statistic": 0.04 },
  "score_drift": {
    "statistic": 0.07, "p_value": 0.42, "flagged": false,
    "n_baseline": 500, "n_recent": 96,
    "baseline_mean": 1.83, "recent_mean": 1.79
  }
}

score_drift is null until you have at least 30 recent feedback rows carrying a verifier score (recorded automatically on every POST /v1/feedback once you're on a fine-tuned model) — and for models fine-tuned before this check existed. The score-distribution shift it catches is exactly the one that can quietly break a certified_threshold's risk guarantee, so a flag here also flips a certification's certification_status to stale_recalibration_pending.

Gray-zone threshold

tau_low/tau_high — the boundaries of your own cache's gray zone, above which a candidate is trusted without ever calling /v1/verify — are entirely your own cache backend's decision; this service never sees or sets them (see the Integration pattern above). What it can tell you is what to do with them: a fine-tuned verifier's auc_tuned (Fine-tune Jobs) says how much you can trust it, but not how that translates into a tau_high value. This endpoint closes that gap by replaying your own feedback against your current verifier: for a grid of candidate tau_high values, it scores what decision accuracy you'd get if everything above that cutoff were served blindly and everything below it were sent through /v1/verify, then recommends whichever value scores highest.

{
  "status": "ok",
  "n_labels": 214,
  "baseline_accuracy": 0.81,
  "recommended_tau_high": 0.87,
  "recommended_accuracy": 0.93,
  "grid": [
    { "tau_high": 0.75, "n_above": 180, "n_below": 34, "accuracy": 0.88 },
    { "tau_high": 0.87, "n_above": 96, "n_below": 118, "accuracy": 0.93 },
    { "tau_high": 0.95, "n_above": 12, "n_below": 202, "accuracy": 0.90 }
  ]
}

status: "insufficient_data" (with a null recommendation) means fewer than 20 of your feedback rows had similarity_score set — nothing to reject, just not enough signal yet. baseline_accuracy is what you'd get with tau_high set above every candidate (verification never runs); grid is every value tried, not just the winner, so you can see how flat or peaked the curve is around the recommendation rather than trusting one number blindly — same reasoning auc_baseline/auc_tuned get surfaced instead of a bare verdict.

Admin

{ "name": "acme" }

{
  "tenant_id": 1,
  "api_key": "cv_hSDoVrYkFJo2CvoaL1_N9SSQDOWjW6NMSV4DPj9FQ5g"
}