Fluency is not accuracy

A language model asked to summarise a medical consultation will always produce something that reads like a clinical note. That is what it was trained to do. Whether the note describes the consultation that actually happened is a separate question, and almost nothing in the standard evaluation toolkit answers it.

This is the problem that shaped how I build. Not accuracy in the abstract, but the specific failure where a system produces confident, well structured, entirely plausible output that contains something nobody said.

Why the usual metrics miss it

Summarisation is typically evaluated with reference based metrics. You collect human written summaries, generate machine written ones, and measure similarity. ROUGE counts overlapping sequences of words. BERTScore compares contextual embeddings, so it credits a paraphrase where ROUGE would not.

Both are useful. Neither was designed to catch fabrication.

Consider a consultation where a patient mentions they have been taking ibuprofen. The model produces a note stating the patient takes ibuprofen and paracetamol. Against a reference note, that output is close to perfect. Nearly every word overlaps, the embeddings are almost identical, the structure is right, the tone is right. The metric will reward it. And a medication the patient never mentioned is now in their record.

The failure is small in the space the metric measures and serious in the space the patient occupies. That mismatch is not a tuning problem. The metric is answering a question about similarity, and safety is a question about provenance.

Grounding as a separate measurement

The approach that works is to stop treating the note as one object to be scored, and start treating it as a set of claims, each of which has to earn its place.

The pipeline decomposes a generated note into individual clinical assertions. Patient reports chest pain on exertion. Patient takes ibuprofen. Blood pressure recorded at 140 over 90. Each assertion is then matched back against the transcript, looking for a span of speech that supports it. Word level timestamps from the speech recognition step make this tractable, because the search is over an aligned sequence rather than a wall of text.

An assertion that finds no supporting span is not paraphrased into something more defensible. It is dropped. This is worth being blunt about, because the instinct when a system produces something ungrounded is to repair it, and a confident repair is indistinguishable from an invention. The only safe behaviour is to remove the claim and let the gap be visible.

What this gives you is a grounding rate, reported separately from readability. Two numbers instead of one, measuring two different things, neither of which can hide inside the other.

The evaluation has to be able to fail

An evaluation that never fails is not an evaluation. It is reassurance.

Alongside the automated checks we ran structured review sessions with clinicians whose explicit task was to find fabricated content. Not to rate the notes, not to say whether they were helpful, but to hunt for anything asserted that was not said. Twenty sessions, adversarial by design.

One of them found a hallucinated clinical detail that had passed every automated check we had at the time. That single finding was worth more than any aggregate score, because it told us where the automated evaluation was blind rather than confirming what it already believed.

This is the part of the work that has no obvious return until it does. A review protocol built to look for failure will find some, and every one of those becomes a test case, an error category, and usually a change to the pipeline.

What it looks like in the product

The grounding work is not invisible to the user. When a clinician reviews a draft, they see which part of the conversation produced each line. Selecting a statement highlights the speech it came from.

This matters more than the accuracy number. A clinician who can check a claim in one second will trust the system appropriately, which means trusting it less on the lines where the provenance looks thin. Calibrated trust is the goal, not maximal trust. A system that is right ninety percent of the time and shows you which ten percent to check is safer than one that is right ninety five percent of the time and offers no way to tell.

The general form

None of this is specific to healthcare. The pattern generalises to any system where generated text will be acted on.

Decide what an atomic claim is in your domain. Require every claim to trace to evidence. Report grounding separately from quality, because a single blended score lets one hide behind the other. Drop what you cannot support rather than smoothing it over. Build a review process whose job is to find failures, and treat every failure it finds as a permanent test.

The model is rarely the hard part. Deciding what the system is allowed to say, and then enforcing it, is most of the work.