The case for running models on the edge

The default architecture for an AI product is to send the input to a hosted model and return the response. It is faster to build, it gives you the strongest available model, and it is what every tutorial demonstrates.

For the products I work on it is also unavailable, and I have come to see that as an advantage rather than a limitation.

The constraint that starts the conversation

Healthcare, finance, and public sector organisations share a property that shapes everything: the data cannot leave. Sometimes that is regulation, sometimes it is an information governance policy, and often it is an institutional memory of a vendor incident.

The consequence is a specific and repeated pattern. A clinical team has a documentation burden measured in hours per day. A solution exists. It requires streaming patient consultations to a third party, so it never clears governance. The team continues writing notes by hand, not because the technology does not work, but because the architecture is unacceptable.

These are the organisations with the most to gain from automation and the least ability to adopt the standard approach. That gap is a market, and the way into it is to remove the network.

What you actually give up

Being honest about the cost matters, because the trade is real.

You give up the frontier model. A model that runs on a clinic laptop is smaller than one running in a data centre, and there is no argument that makes that untrue. You give up elastic compute, so the slowest machine a customer owns becomes your performance budget. You give up instant model upgrades, because a new version has to be distributed rather than switched on. And you take on distribution complexity, packaging inference across hardware you do not control.

For general purpose reasoning these costs would be prohibitive. For a narrow, well specified task they are much smaller than they first appear, because a task specific pipeline running a smaller model with strong constraints often outperforms a large model given a vague instruction.

What you get back

The governance problem disappears. Not reduced, removed. There is no data processing agreement to negotiate over patient audio, because no patient audio is transmitted. Procurement conversations that would have taken months stop happening. This is the single largest commercial effect, and it does not appear anywhere in a technical comparison.

Latency becomes predictable. No network round trip, no rate limits, no third party incident taking your product down mid consultation. A local pipeline running at a consistent hundred milliseconds is more useful in a live clinical setting than a hosted one averaging four hundred with an occasional two second outlier.

Costs stop scaling with usage. Inference runs on hardware the customer already owns. A per consultation API cost that grows with adoption is replaced by a fixed engineering cost, which changes both the unit economics and how you are able to price.

It works offline. Community clinics, field settings, and buildings with unreliable connectivity are all reachable. A hosted product simply does not function there.

The constraint forces better engineering. This one surprised me. When you cannot solve a quality problem by reaching for a bigger model, you solve it by understanding the problem. Every accuracy improvement in our pipeline came from better segmentation, better domain vocabulary handling, and a grounding step that removes unsupported claims, rather than from more parameters. The resulting system is more predictable than one built on a model whose behaviour can change underneath you.

Where it genuinely does not fit

Edge deployment is wrong for open ended reasoning over large context, for anything needing current external information, and for early stage products still discovering what they are. In that last case, ship with a hosted model, find out whether anyone wants it, and move inference locally once the requirements have stopped moving.

The decision rests on one question: is the task narrow and stable enough that a small model with good scaffolding can do it well? If yes, and the data is sensitive, the edge is the better architecture and not the compromise position.

The part worth remembering

The strongest engineering constraints on these projects turned out to be the reason for winning the work. Building for on device from the start meant a product that could be deployed in environments where competitors were structurally unable to operate.

Constraints narrow the design space. A narrower space is easier to search, and the answer you find in it tends to be one you understand completely.