The Harness — From Tokens to Agents

Here's a fact that surprises people: the chatbot you talk to is not "the model." It's the model plus a substantial amount of code wrapped around it. That wrapper is the harness (also called scaffolding), and it's where a raw next-token predictor becomes a usable product.

If the model is a brain in a jar, the harness is the body, the senses, the reflexes, and the rules of conduct. This post is about what's in that wrapper — because most of the difference between a flaky demo and a reliable product lives here, not in the model.

The harness gives the brain a body

What the raw model actually gives you

Strip away everything and a base model offers a brutally simple interface:

Code

   text in  ──►  [model]  ──►  text out

That's the entire contract. No memory of previous calls, no ability to do anything but emit tokens, no notion of "you" versus "it," no rules. Everything else — the conversation that persists, the personality, the safety behavior, the ability to use tools — is constructed around this core by the harness.

What a harness does

Let's build one up, piece by piece, starting from that bare text in → text out.

1. It manages the conversation. The model is stateless (post #2). So before every call, the harness assembles the full context: the system prompt, the conversation history so far, and your new message — concatenated into one big input.

Code

   each turn, the harness sends the model the WHOLE conversation again:

   [system prompt][user: hi][assistant: hello!][user: what's 2+2?]
                                                        └─ your new message

The "memory" of a chat is an illusion maintained by re-sending the transcript every turn. Which means when a conversation outgrows the context window (post #6), the harness has to do something: drop old messages, summarize them, or otherwise prune. That pruning logic is pure harness, and it's why long chats sometimes seem to forget earlier details.

2. It sets the system prompt. Before you ever type anything, the harness has already placed a hidden system prompt at the front of the context. This is the model's standing instructions: who it is, how to behave, what it can and can't do, how to format responses, what today's date is. It's often long and carefully engineered (post #9), and it's the single biggest lever the product builder has over behavior. A huge fraction of what feels like a model's "personality" is actually its system prompt.

3. It parses the output. The model emits a stream of tokens. The harness turns that stream into something structured: rendering markdown, detecting when the model wants to call a tool (post #11), extracting JSON, catching when it's done. The model produces raw text; the harness gives it meaning.

4. It enforces rules and safety. The harness is where guardrails live:

Code

   user input ─► [input checks] ─► model ─► [output checks] ─► you
                  block disallowed         filter unsafe content,
                  requests, detect          redact secrets, enforce
                  injection attempts        format, verify claims

These checks — content filters, rate limits, refusal handling, secret redaction — are policy decisions made in code, not emergent properties of the model. When a product feels "safe" or "locked down," that's largely the harness.

5. It handles the messy reality. Real systems need retries when a call fails, timeouts, fallback to a smaller model under load, logging, cost tracking, caching (post #7), and streaming the response to your screen (post #6). None of this is "AI" — it's ordinary, essential software engineering that makes the AI dependable.

The loop: where it gets interesting

So far the harness is a sophisticated wrapper around a single call. The leap that unlocks everything in the rest of this series is making the harness loop.

Instead of text in → text out → done, the harness can call the model, look at what it produced, do something with it, and call the model again with the result:

The loop — act, observe, repeat, until done

This loop is the dividing line between a chatbot and an agent. A chatbot harness calls the model once per turn and hands you the text. An agent harness keeps the loop running — letting the model act, see results, and decide what to do next — until the task is done (posts #11–12). Same model; a more ambitious harness.

Why the harness explains so much

Reframing AI products as "model + harness" clarifies a lot of confusion:

Why the same model feels different across products. Two apps using the identical underlying model can behave very differently because their system prompts, tools, memory handling, and guardrails differ. The harness is the differentiator.
Why behavior changes without a new model. A product can get noticeably better or worse overnight with zero change to the model — someone edited the system prompt or adjusted the loop.
Where many "bugs" actually are. Recall the deleted-file example from post #1. "It ran the command without asking" is a harness decision (no confirmation step), not a model failure. A surprising share of AI mishaps are harness design choices, not model limitations.
Why "just use the API" isn't the whole story. Calling the raw model is easy; building the harness that makes it reliable, safe, and useful is most of the actual work in shipping an AI product.

The takeaway

The harness is the code around the model that turns a stateless text predictor into a product: it manages conversation memory, sets the system prompt, parses output, enforces safety, handles failures, and — most importantly — can wrap the model in a loop. That loop is what we're about to exploit. First, though, the model needs a way to reach beyond its own text and actually do things. That's tool use.