Blog
Research data and production notes from CacheVerifier: what our semantic cache experiments showed, what broke, and what we changed.
Latest· 5 min readSemantic cache verification tested on 3,400 live pairsOn two public benchmarks, a fine-tuned verifier lifted recall from 74% to 94% at the same false-accept rate. Where it pays off, and where a threshold is enough.production testfine-tuningbenchmarksRead · 4 min readSemantic cache hallucinations: two failure modesA wrong semantic cache hit can be repeated back as fact or served to a different user, and a similarity threshold sees neither. Why, and what catches them.productcache correctnessRead · 4 min readFine-tuning won't stop adversarial semantic cache hitsAn off-the-shelf verifier false-accepts 84% of adversarial query pairs; fine-tuning on natural feedback pushes it to 87.6%. What actually worked instead.researchadversarial robustnessRead · 5 min readConformal risk control overshoot in a semantic cacheA Conformal Risk Control threshold overshot its risk bound under a chronological test on one dataset; a second apparent overshoot was a dataset artifact.researchrisk certificationRead · 4 min readSemantic cache drift: when fine-tuning turned harmfulOn multi-year Comcast support traffic, the gray-zone positive rate drifted 5x and a fine-tuned verifier went from helping to hurting. How we now catch it.researchdrift monitoringRead