Does this actually work? We tried to break it.
Three questions the underlying research didn't answer, checked against three public benchmarks and real production support traffic, using the same held-out AUC evaluation throughout (cross-encoder/ms-marco-MiniLM-L6-v2 base model).
4
independent datasets
3 public benchmarks + real customer-support traffic
~180K
gray-zone examples evaluated
across all four datasets combined
~30%
label-noise breakeven point
consistent across LmArena and Quora
500–1000
examples to reliably help
the floor where cold-start turns positive everywhere
How noisy can feedback labels be before fine-tuning backfires?
Production feedback (thumbs up/down, ticket reopens) is noisier than a research dataset's ground truth. We took a fixed 4,000-example slice per dataset and randomly flipped labels at six target noise rates, always evaluating against the untouched, clean held-out test set.
All three curves decay smoothly and cross zero in the same 30–40% range — LmArena and Quora both flip negative around ~31–32% noise; SearchQueries stayed positive through the full tested range. The shape holds across datasets, so we treat ~30% as a conservative red line: below it, self-serve fine-tuning is safe by default; above it, we recommend running a Health Check Report before enabling continuous calibration.
How much feedback do you need before fine-tuning is worth it?
A new tenant starts with zero labels. We fine-tuned independent models on increasing prefixes of each dataset's training stream — from 50 examples up to the full set — to find the point where it reliably helps rather than hurts.
All three curves keep climbing with more data — no plateau, even at the full training set. But below ~500 examples, two of the three datasets (SearchQueries, Quora) show fine-tuning can actually be negative, not just "smaller gains" — the risk is concentrated in domains where the unfine-tuned baseline is already weak. 500–1000 examples is the smallest point where every dataset we tested turned reliably positive, so that's our default minimum before we recommend switching on fine-tuning.
Does a fine-tuned verifier decay over time?
We split each dataset into 8 chronological chunks, fine-tuned a fresh model on each chunk as an "anchor," then evaluated every anchor against every chunk — including ones far in its future — and averaged the gain over baseline by time distance.
After the initial same-window drop (distance 0→1, a memorization effect, not real generalization), gains stay flat out to a distance of 7 chunks — no sign of decay. This held even on SearchQueries, whose baseline accuracy itself barely moves over time, ruling out "the data got easier" as a confound. Across three benchmarks, this argued against needing high-frequency retraining schedules.
The one place it broke — and why that's useful
All three questions above were checked against public research benchmarks. We then reran the same battery on real production data: 3.5 years of Comcast and Amazon customer-support conversations (Kaggle's Customer Support on Twitter dataset). Amazon confirmed everything above. Comcast didn't — fine-tuning was negative, more data made it worse (AUC briefly fell below random chance), and gains decayed with time distance instead of holding flat.
The root cause wasn't noisy labels or too little data — it was that the gray-zone hit rate itself drifted 5× between the training window and later traffic (1.0% → 5.2%). A model calibrated on one base rate silently stopped applying to a different one. That's a distinct failure mode from "the verifier got stale," and it's why drift monitoring watches positive-rate stability, not just accuracy — the same chunked z-test and Page-Hinkley detector caught this shift several chunks before it would have shown up as a Group-level accuracy drop.
Limitations, stated plainly
- • Noise was injected by uniform random label flips, not a modeled real-world feedback error distribution — real thumbs-down noise may be systematic rather than random.
- • Three of four datasets are public research benchmarks, not private customer data — query style, label quality, and traffic patterns may differ systematically from any one business.
- • The real-traffic dataset (2014–2017 Twitter support) predates today's AI-assisted support workflows and used embedding-similarity clustering as a proxy for "equivalent question," not human labels.
- • "~30% noise" and "500–1000 examples" are reasonable starting points validated across the datasets we tested — not universal constants for every domain.
Want the same evaluation run on your own traffic?
Get a health check report