Speech intelligence, clinical AI
Nora Health
On device clinical documentation that turns a consultation into a structured note without the audio leaving the room.
- Role
- Founder, Machine Learning Engineer
- Organisation
- Nora Health
- Period
- 2024 to present
- Status
- Active
- Domain
- Speech intelligence, clinical AI
01
Overview
Nora Health listens to a clinical consultation and produces the note a clinician would have written afterwards. Transcription, structuring, and drafting all run on the clinician device, so patient audio is never uploaded, never stored in a third party account, and never used to train a model elsewhere.
The product exists because the documentation burden in healthcare is measured in hours per clinician per day, and because the obvious solution, streaming consultations to a hosted model, is the one most clinical governance teams will not approve.
02
Problem
Clinicians spend a significant part of the working day writing notes rather than seeing patients. The available automation almost all depends on sending recorded speech to a cloud service, which puts identifiable patient data into someone else data centre and turns a clinical tool into a procurement and information governance problem.
The second problem is quieter and more serious. A language model asked to summarise a consultation will produce a fluent note whether or not it understood the conversation. A note that reads well but records a medication the patient never mentioned is more dangerous than no note at all, because a fluent error invites trust.
03
Solution
The system runs entirely on the device. Speech recognition is a quantised model small enough for consumer hardware, and note generation is constrained so every clause in the output has to trace back to something that was actually said.
Rather than asking a model for a note and hoping, the pipeline extracts candidate clinical facts first, grounds each one against a span of the transcript, and only then composes prose from the facts that survived. Anything the model asserts without a supporting span is dropped before a clinician ever sees it, and the interface shows which part of the conversation produced each line.
04
Technical architecture
- 01
Capture
Audio is buffered in memory with voice activity detection to segment speech, and is discarded once a segment has been transcribed. Nothing is written to disk.
- 02
Transcription
A quantised speech recognition model runs locally with word level timestamps, which become the anchors the grounding step later depends on.
- 03
Extraction
A constrained decoding pass pulls structured clinical entities from the transcript into a typed schema covering presentation, history, examination, and plan.
- 04
Grounding
Every extracted fact is matched back to a transcript span. Facts without an anchor are discarded rather than rewritten, because a confident repair is indistinguishable from an invention.
- 05
Composition
The surviving facts are rendered into a clinical note through templates that constrain structure while a local model handles phrasing.
- 06
Review
The clinician sees the draft alongside its provenance, edits in place, and signs off. Their edits are the evaluation signal the system improves against.
05
Technologies
Speech
- Whisper
- CTranslate2
- ONNX Runtime
- Voice activity detection
Language models
- Quantised local inference
- Constrained decoding
- Structured output
Application
- Python
- FastAPI
- TypeScript
- React
Evaluation
- BERTScore
- Span level grounding checks
- Clinician review harness
06
Challenges
Running useful models on hardware you do not control
A clinic laptop is not a training cluster. Quantisation, batching, and a streaming architecture were all necessary before the pipeline could keep pace with normal speech, and every accuracy decision had to be weighed against latency on the slowest machine the product supports.
Proving a note is grounded rather than assuming it
Fluency is not accuracy, and no summarisation metric on its own tells you whether a clinical fact was invented. The grounding step exists because the evaluation had to be able to fail. Structured review sessions were run specifically to find fabricated content, and one of them caught a hallucinated detail that would otherwise have reached a patient record.
Clinical language is not general language
Drug names, abbreviations, and dosages are exactly where a general purpose recogniser is weakest and where an error carries the most weight. Domain vocabulary handling and a bias toward flagging uncertainty over guessing were both needed.
07
Outcomes
- 87%Grounding quality on the production pipeline, by BERTScore
- 20Structured test sessions, one of which caught a hallucination before it reached a record
- 0Patient audio recordings leaving the device
- Documentation time reduced to review and sign off rather than authoring
08
Key learnings
- The model was never the hard part. Deciding what the system is allowed to say, and enforcing it, was most of the work.
- Discarding an ungrounded fact is better product behaviour than paraphrasing it into something defensible. Users trust a system that admits a gap.
- On device is a design constraint that pays for itself. Removing the network removed an entire class of governance, privacy, and latency problems at once.
- Clinician edits are the most valuable evaluation dataset in the product, and capturing them properly mattered more than any offline benchmark.