Inside an AI agent: the agent loop

One turn, then another

What is an agent? ends on three words: model, tools, loop. This article is the loop. Every agent you have used - a coding assistant, a research tool, a ticket triager - runs the same cycle underneath: assemble a context from the current state, call the model once, read its decision, run the tool it asked for, record what happened, update the state, and go again. One pass through that cycle is a turn. An agent run is a sequence of turns that ends when something says stop.

The stages are worth naming individually because each one is a place where production agents differ from demos. The diagram below is the spine of the article; sections 03 to 09 take the stages one at a time, and section 10 argues why the loop, not the model, is the thing you are actually engineering.

One turn runs left to right · the return arrow is the loop · Decision is where it branches

Two things to notice before going stage by stage. The model appears exactly once per turn, and it is stateless: every call is a fresh function of whatever context you hand it. Everything that persists between turns lives in the state, on your side of the line. And the only exit is at Decision - a run ends when the model stops asking for tools, or when something in the harness decides that it should.

Agent loop, control loop

Strip the AI vocabulary away and the agent loop is a control loop in the classic sense: sense, decide, act, repeat. A thermostat reads the temperature, compares it to a target, switches the heater, and reads again. A game loop reads input, updates the world, renders, and reads again. An agent reads the observation, asks the model what to do, runs the tool, and reads again. What is new is the decide step - a probabilistic model instead of a rule - and that one substitution is where every hard problem in this article comes from.

The code that runs the loop is the harness. It owns the state, calls the model, dispatches tools, enforces limits, and decides when to stop. The model never owns any of that; it sees a context and returns a reply. Three shapes of harness are common, and the choice between them is mostly about how much structure you want to make explicit.

A plain loop. A while with the stages inline, like the code in section 07. Simplest to write, easiest to step through in a debugger, and where most production agents start. Its weakness is that branching, parallel tool calls, and pausing for a human all end up as ad hoc conditions inside one function.
A graph or state machine. Each stage is a node; the loop is a cycle in the graph; an approval gate is a node that suspends. Frameworks in this family make interrupts, retries, and parallel branches first-class and give you a picture of the run for free. The cost is ceremony - a small agent becomes a lot of wiring - and a second mental model on top of the loop itself. AI Agent Frameworks compares the main options.
A hosted loop. The provider runs the harness: you supply tools and instructions, it runs turns and calls back when a tool is needed or the run ends. The Claude Platform and the OpenAI Platform both offer one. Least code, least control over the stages below, and the right answer when the stages below are not your differentiator.

Whichever shape you pick, the stages are the same. The rest of the article is about what each one has to do, and the trade-offs you meet doing it.

State

State is everything the run knows that the model does not remember - which, since the model remembers nothing between calls, is everything. Context is not state; context is the rendering of state into a prompt, rebuilt on every turn. Keeping the two apart is the single most useful habit in agent design, because it lets you change what the model sees without losing what the run knows.

Transcript

What was said and done

Messages, tool calls, tool results, in order. Grows every turn, and a single file read can add more than the whole conversation. This is what fills the window first.

Working memory

The plan and the notes

A scratchpad the model writes and rewrites: the plan, what it has found, what is still open. Small, current, and the thing that survives when the transcript is trimmed.

External memory

Files, databases, indexes

Outside the window entirely, pulled in on demand by a tool. Notes written to disk, a vector index, the project itself. Survives the run, and often the day.

Run metadata

What the harness knows

Turn count, spend, elapsed time, permissions already granted, the last checkpoint. The model rarely sees it in full; the termination logic reads it every turn.

The trade-off runs along one axis: how much of the state to put in the window. Everything in the window is simplest and works until the window fills, which on real tasks is sooner than you expect - What's in an agent's context? has the arithmetic. Compacting the transcript (summarising old turns, dropping stale tool results) keeps runs going but loses detail, and the model does not know what it lost. Externalising - the agent writes findings to a file and reads them back when needed - is the most robust for long runs, at the price of extra turns and a model that has to remember to look. Mature coding agents do all three: a full transcript until it gets heavy, a compaction step, and a notes file the model maintains itself. The four words this section keeps apart - working memory, state, long-term memory, retrieval - get an article of their own in Inside agent memory.

Planning

Planning is how the model decides not just the next step but the shape of the whole task. There are two ways to get it, and the choice shows up in the transcript as either nothing at all or a visible plan artifact.

Implicit plan

Decide one step at a time

The model reasons briefly, picks one action, observes, and reasons again - the ReAct pattern.
Adapts instantly to whatever the last observation turned up.
Loses the thread on long tasks: redoes work, drifts from the goal, forgets a sub-task it noticed ten turns ago.
Best for short, exploratory work: debugging, research, a question that needs three lookups.

Explicit plan

Write it down, then work it

The first turn produces a plan - a task list, a checklist file - and later turns execute against it and tick items off.
Stays on task across dozens of turns and survives transcript compaction, because the plan is state, not chat.
Costs a turn up front, and plans go stale: the harness has to make re-planning cheap when an observation contradicts the plan.
Best for long, multi-part work: a migration, a multi-file change, a research report.

In practice the line is drawn by task length. Coding agents that started as pure step-by-step loops have all grown an explicit task list, because a forty-turn refactor without one wanders. The failure to watch for on the other side is a plan the model will not abandon: an observation says the approach is wrong and the agent keeps ticking boxes. Re-planning has to be an expected move, not an exception.

Tool selection

The model chooses a tool by reading its name, description, and input schema - nothing else. That makes tool descriptions a design surface, not documentation: a vague description gets a tool called at the wrong time, an ambiguous pair of names gets them confused, and a schema with optional fields the model does not understand gets arguments invented. The best single improvement to most agents is rewriting the tool descriptions from the model's point of view: when to call this, when not to, what a good argument looks like.

The problem that arrives with scale is the size of the catalogue. Every tool definition is in the window on every turn, so forty tools cost tokens before the task starts, and selection accuracy drops as the list grows - the model has more ways to be nearly right. Three responses, in rough order of effort:

A small, fixed set per agent. Give each agent the eight tools its job needs, not the sixty the organisation has. Cheapest and most reliable; it stops working when one agent genuinely needs breadth.
Tool routing. Load definitions on demand. A search_tools meta-tool, or a namespaced catalogue where only the relevant group is expanded, keeps the resident set small. Model Context Protocol (MCP) servers make this pressing - each server brings its whole catalogue - and the ecosystem has grown the same on-demand pattern in response.
Sub-agents. Split the breadth across agents, each holding a subset, with an orchestrator that delegates. This is where multi-agent orchestration begins, and it costs coordination.

Whichever you choose, validate arguments against the schema in the harness before running anything. A malformed call is not an exception to crash on; it is an observation to send back so the model can correct it.

Observation

The observation is the only channel through which reality reaches the model. It never sees your filesystem, your database, or the web; it sees the string your tool returned. Whatever is in that string is the world, and whatever is not in it did not happen. That makes the shape of tool results the second design surface after tool descriptions, and the one teams neglect longest.

Three rules cover most of it. Keep it small. A tool that returns a 600-line file has just spent a page of the window on lines the model will not use; return the relevant region, paginate, or summarise, and say what was cut. Keep it structured. Return the fields the next decision needs - a path, a status, a count - rather than raw HTML or a stack of log lines the model has to parse. Keep it honest. An error, a timeout, an empty result, a permission denial: each is an observation and belongs in the transcript in plain words. A tool that swallows its failure and returns nothing has just told the model that nothing is wrong.

The tension is between honesty and noise. Every observation is paid for on every later turn, so a verbose error stack shown once is re-read twenty times. The usual resolution is to shape observations in the tool wrapper: pass through what changes the model's next decision, log the rest, and never lie by omission.

Retry and error recovery

Things fail at every stage: the model API rate-limits, a tool times out, the model calls a function with a path that does not exist, the run loops on the same failing call. Recovery works in layers, and the design question at each layer is the same: does the model need to know?

Transient failures - hidden. A 429, a network blip, a 503 from a downstream service. Retry with backoff inside the harness and never mention it. The model gains nothing from knowing, and the tokens it would spend reasoning about it are wasted.
Lasting failures - shown. File not found, a query that errored, a test that failed. These go back as observations, in the words a colleague would use, because the model's next decision depends on them. Hide these and the agent proceeds as if the step succeeded, which is the most expensive failure an agent has.
Bad arguments - corrected. Schema validation fails before the tool runs; the validation message is the observation. The model fixes the call on the next turn without anything having executed.
Stuck loops - interrupted. The same call with the same arguments three times, or the same error twice, is a signal the harness can detect cheaply. Inject a nudge ("this approach has failed twice; try a different one") or stop the run. Left alone, a stuck agent runs to the budget cap.
Crashes and pauses - resumed. Persist the state after every turn. A checkpointed run survives a process restart, a network outage, or a human who wants to sleep on an approval. It also makes a post-mortem possible: the transcript is the log.
Wrong actions - undone. Where a tool has side effects, give the run something to undo with: a git branch, a sandbox, a dry-run flag, a staging environment. Recovery from a bad write is a property of the environment, not the model.

Here is a harness with the stages so far in place. callModel wraps your model API; tools is a map of name to definition, each with a run function and a needsApproval flag; the helpers do what their names say.

            JavaScript
            harness.js · the loop with its controls
          

async function runAgent(goal, tools, limits) {
  const state = { goal, transcript: [], notes: "", turns: 0, spentUsd: 0 }; // state, not context
 
  while (true) {
    if (state.turns++ >= limits.maxTurns || state.spentUsd > limits.maxUsd) {
      return { status: "stopped", reason: "budget", state };                 // termination: a cap tripped
    }
 
    const context = buildContext(state, tools);   // state -> prompt, rebuilt every turn
    const reply = await callModel(context);       // the model: one call, one reply
    state.spentUsd += reply.costUsd;
    state.transcript.push(reply.message);
 
    if (!reply.toolCall) {
      return { status: "done", answer: reply.text, state };                  // termination: natural stop
    }
 
    const call = reply.toolCall;                  // the decision
    const tool = tools[call.name];
    if (tool.needsApproval && !(await askHuman(call))) {                    // human approval gate
      state.transcript.push(toolResult(call, "Denied by the user. Try another approach or stop."));
      continue;
    }
 
    let observation;
    try {
      observation = await withRetry(() => tool.run(call.args), { attempts: 3 }); // transient: retried, unseen
    } catch (err) {
      observation = "Error: " + err.message;      // lasting: shown to the model
    }
    state.transcript.push(toolResult(call, clip(observation, 4000)));         // the observation, trimmed
    state.notes = await compactIfNeeded(state);   // update state before the next turn
  }
}
          

Everything interesting is in the comments' right-hand column. The model is one line. The other thirty are the loop deciding what to remember, what to allow, what to retry, what to show, and when to stop - and none of them exist in a single tool-calling request.

Termination conditions

A loop that cannot stop is a bug, and an agent has more ways to fail to stop than a normal program: a model that keeps finding one more thing to check, a stuck retry, a plan with no last item. Production harnesses run several stop conditions at once, and treat the reason a run stopped as data - it is the first thing you look at when a run did not do what you expected.

Stop condition	Who triggers it	What the run should return
Natural stop	The model: a reply with no tool call	The final answer, with the transcript kept for review
Explicit finish	The model: a `finish` tool that takes a structured result	Typed output your code validates before accepting it
Budget cap	The harness: turns, tokens, dollars, or wall-clock time exceeded	A partial result marked partial - never a fabricated success
Verification gate	The harness: tests, a linter, a schema check on the output	Pass: done. Fail: the failure goes back in as an observation
Escalation	Either side: the model asks, or a rule says a person must decide	A paused run with its state saved, resumable after the decision
Kill switch	A person or a monitor	Immediate stop; state persisted for the post-mortem

The natural stop is the one everybody implements and the one you can trust least on its own: the model decides it is done, and models are optimistic about their own work. Pairing it with a verification gate - run the tests, check the output against a schema, diff the result against the request - turns "the model says it finished" into "the model finished and here is the evidence". An explicit finish tool is the cheap version of the same idea: it forces the final answer through a shape you can check.

Budget caps are the trade-off condition. Too tight and legitimate long runs get cut off mid-task; too loose and a stuck agent spends the monthly budget by lunch. Set them per task type rather than globally, and log how close finished runs came to the cap - that distribution tells you where the cap should be. What an agent actually costs has the numbers behind why this matters.

Human approval

The approval gate sits between Decision and Tool: the model has chosen an action, and before it runs, a person gets to say no. In the code above it is one if; in a real harness it is the mechanism that decides how much autonomy the agent has, and it is a setting rather than a property of the agent.

Three modes cover the range, and mature tools expose all three. Ask on everything - the safe default for a new agent or an unfamiliar codebase; slow, and it teaches you what the agent tends to do. Allowlist - reads, searches, and anything inside a sandbox run freely; writes, sends, and deletes ask. This is where most day-to-day use lands. Autonomous inside a boundary - no questions, because the environment itself is the guardrail: a container, a branch, a staging account, a spending limit. The right mode depends on the blast radius of the worst tool in the set, not on how much you trust the model.

What needs approval is any action that is irreversible or leaves the boundary: sending a message, moving money, deleting data, deploying, calling a third party's API with your credentials. Reads almost never do. Writes inside a sandbox rarely do. The mistake in both directions is common: gating file reads until the user approves reflexively and stops reading the prompts, or letting an agent send email because "it only sends to internal addresses".

Mechanically, an approval is a pause: the harness persists the state, surfaces the proposed call with its arguments, and waits. The answer becomes part of the run. An approval runs the tool; a denial goes into the transcript as an observation - "denied, with this reason" - so the model can choose another route instead of retrying the same call. Two details separate a usable gate from an annoying one: batch related approvals into one decision when the model proposes several similar calls, and keep an audit log of every gated call with who approved it, because that log is what you read when something went wrong anyway. Claude Code's permission modes are a working example of all of this in one tool.

Why an agent isn't an LLM plus tools

Tool calling has been a model feature for years: send a request with tool definitions, get back a request to call one. That is one turn. It is a stateless function - the same input gives the same shape of output, nothing accumulates, nothing has to be decided about stopping. Most of the confusion about agents comes from assuming that adding a loop around this function is a small step. It is not, and the previous seven sections are the reason.

The loop introduces time. Once turns accumulate, the system has state that has to be kept and rendered, decisions that depend on earlier decisions, errors that compound instead of surfacing once, cost that grows with every turn, and a stop that somebody has to decide. None of those exist in a single tool-calling request. All of them exist in a forty-turn run. An agent is not a model with a for-loop around it any more than a web service is a function with a socket around it: the loop is where the engineering lives, and the model is a component the loop calls.

That is why the vocabulary of production agents has settled on the loop's stages rather than on the model. Context engineering is the Context stage. Tool routing is Tool selection. Memory is State. Planning is Planning. Recovery is the branch out of Observation. Orchestration is what happens when one loop delegates to others. Operational controls - budgets, permissions, logging, evaluation - are the harness's half of Decision and Stop. Read an architecture guide for agents from any vendor this year and you will find those seven headings, in roughly that order, because they are the stages of the diagram at the top of this page with production names.

The consequence for anyone building one: spend your design time on the harness. The model will improve on its own schedule and you can swap it. The state design, the tool descriptions, the observation shaping, the stop conditions, and the approval boundary are yours, and they are what decides whether the agent finishes the job. Why agents fail is the catalogue of what happens when one of those stages is skipped; How to evaluate an agent is how you find out which one.