Keep the state outside the prompt

The straightforward way to build a conversational system is to keep the message history and send it with every request. It works immediately, which is why almost everything starts this way.

It stops working at length, and the failure is gradual enough to be mistaken for a model problem.

How it degrades

A tutoring session runs for forty minutes. By the end the history holds dozens of turns, and three things have gone wrong at once.

Cost grows quadratically with the conversation, because every turn resends everything before it. Quality drops, because material in the middle of a long context gets less attention than material at either end, so a misconception identified early becomes progressively less influential exactly as it matters most. And behaviour becomes unpredictable, because what the model attends to in twelve thousand tokens of transcript is not something you can reason about or test.

The last one is the real problem. You can pay for tokens. You cannot debug a system whose memory is an undifferentiated pile of text.

The alternative

Keep an explicit state object. Let the prompt be a rendering of it.

In the tutoring engine that state was a typed record, held per student and separate from the conversation entirely. Which concepts had been demonstrated, which were shaky, which specific misconception best explained the errors seen so far, and a running summary of ground covered. After every turn the state is updated. Before every turn the prompt is assembled from the current state, the relevant curriculum context, and the selected next move.

The transcript is not the memory. The state is the memory, and the transcript is raw material the state was derived from.

What this changes

Prompt size stops growing. State is bounded because its schema is fixed. A forty minute session and a five minute session produce prompts of roughly the same size, which flattens the cost curve and keeps the model operating in the range where it behaves well.

The system becomes inspectable. You can look at what the system currently believes about a student. You can see that it has decided the misconception is about denominators rather than multiplication. When behaviour is wrong you can tell whether the diagnosis was wrong or the response to a correct diagnosis was wrong. Those are different bugs with different fixes, and with history as memory they are indistinguishable.

It becomes testable. State is data, so you can construct it directly. Given a student with these three concepts shaky and this misconception open, what does the system do next? That is a unit test. There is no equivalent test for a system whose behaviour depends on the shape of an entire transcript.

Policy gets somewhere to live. The tutoring engine had to avoid giving away answers, which is directly against a model default disposition to be maximally helpful. Instructions in a prompt are a request, followed inconsistently. With explicit state and a decision layer above generation, the choice of whether to explain, question, or work an example is made in code, and generation only handles phrasing. The behaviour became reliable when it stopped being a matter of persuasion.

The cost of doing it

This is more work than appending to a list, and it is worth naming what it costs.

You have to design the schema, which means deciding in advance what the system needs to remember. Getting that wrong means the state cannot represent something important, and you will find out later than you would like. You need an update step, which is itself a model call and can itself be wrong, so a bad summary poisons subsequent turns in a way a raw transcript would not. And information is genuinely lost, because compression discards detail, some of which will occasionally have mattered.

The mitigation for that last point is to keep the transcript stored, just not in the prompt. State drives behaviour, and the full record remains available for the rare case that needs it, and for offline analysis.

When to bother

Do not do this for a three turn interaction. History as memory is correct for short conversations, and the extra structure would be pure overhead.

Do it when sessions run long, when behaviour has to stay consistent across a session, when the assistant needs to remember something specific rather than generally, and when you need to explain after the fact why the system did what it did.

The underlying principle is not specific to language models. Deriving behaviour from an explicit, typed, inspectable representation of what a system knows is how you keep it debuggable. The prompt is a view over that representation, and treating it as the representation itself is what makes long running assistants so hard to reason about.