Your benchmark is not your production distribution

Every model I have deployed performed worse in production than in evaluation. Not occasionally. Every one. After enough repetitions this stops looking like bad luck and starts looking like a property of how evaluation is normally done.

The gap is structural. Understanding why makes it manageable, and it never fully closes.

Where the gap comes from

Your test set came from the same place as your training set. The standard split takes one dataset and divides it. That measures generalisation to unseen samples from a distribution you have already seen. Production is a different distribution, gathered at a different time, by different people, under different conditions. The split tells you about variance within your data, not about the shift between your data and the world.

Real inputs are worse than collected ones. Data assembled for training tends to be data that was easy to assemble. In speech that means audible recordings with cooperative speakers. Production gives you crosstalk, background noise, a phone in a pocket, accents underrepresented in the corpus, and someone talking over the person you care about. Nobody curated it, because it arrived.

Aggregate metrics hide the failures that matter. A single number averages over inputs that carry very different consequences. Our quantisation benchmark showed conversational accuracy holding up well under aggressive compression, which looked like permission to ship the smaller model. Segmented by vocabulary type, drug names and dosages had degraded badly. The aggregate said yes and the segment said absolutely not, and the segment was the one that mattered in a clinical setting.

Distributions move. Even a perfectly representative test set describes one moment. Users change behaviour, upstream systems change their output format, an adversary adapts. Fraud detection makes this obvious because the adversary is deliberate, but the drift exists everywhere, just more slowly.

What to do instead

Build the test set from production data, not from a split. This is the single highest value change. Sample real inputs, label them properly, and keep that set separate from anything used in training. It is slower and less convenient than a split, and it is the only way to measure the thing you care about. On client engagements this is where the work starts, because an accuracy claim against a public benchmark tells the client nothing about their own recordings.

Segment every metric. Never report a single number without reporting the slices underneath it. Segment by input condition, by user cohort, by the categories where errors carry the most cost. Any aggregate that has not been broken apart is hiding something, and the only question is whether you find out before or after deployment.

Evaluate the failure mode, not just the score. Accuracy tells you how often the system is wrong. It says nothing about what happens when it is. A model that is wrong loudly, by flagging low confidence and abstaining, is safer than one with a better score that is wrong silently. Measure the behaviour of the errors, not only their frequency.

Assemble an adversarial set on purpose. Collect the inputs you expect to break the system, and keep them permanently. Accents, edge cases, malformed inputs, ambiguous language, the transcript that caused a problem last quarter. Every production failure becomes a permanent test. Over time this becomes the most valuable evaluation asset you own, because unlike a benchmark it was built by reality.

Instrument for the metrics you cannot compute offline. Some quality signals only exist in production. In a documentation product, the edits a clinician makes to a draft are a direct measurement of where the draft was wrong. Capturing them properly was worth more than any offline benchmark, because it is continuous, it is real, and it does not go stale.

The expectation to set

The useful mental model is not that evaluation predicts production performance. It is that evaluation establishes an upper bound, and the job is to narrow the gap between that bound and what actually happens.

That reframing changes the questions. Instead of asking whether the score is good enough to ship, you ask how far production is likely to fall below it, which inputs will cause that, and whether the system fails in a way you can detect. Those are answerable questions, and answering them is what makes a deployment predictable.

Being wrong is not the problem. Being wrong in a way nobody anticipated, on inputs nobody tested, without any signal that it is happening, is the problem. Evaluation is how you buy that signal in advance.