Building reliable agents

1. Intuition

Errors compound. If each step of a 10-step task is right 95% of the time and steps fail independently, the task succeeds only about 60% of the time. An agent that looks great on any single step can still fail 4 times out of 10. Three levers fight this: make each step more reliable (better tools, clearer instructions), use fewer steps (a better tool that does in one call what took three), and catch errors before they propagate (validate tool arguments, check results, retry).

Tools are the agent's interface, so design them like an API for a new colleague. A good agent tool does one thing, has a name that says what, takes few arguments with clear types, and returns a compact result. It is idempotent where possible: calling hold_booking(request_id=...) twice with the same request ID makes one hold, not two, so a retry after a timeout is safe. Its errors say what went wrong and what to do next ("flight_id AS33 not found; flight IDs look like AS330, get them from search_flights"). A vague tool list is the single most common cause of agents choosing the wrong action.

Guardrails limit what a wrong step can do. The model will sometimes be wrong and can be manipulated by text it reads, so the runtime, not the prompt, enforces safety:

• Least privilege: the agent gets only the tools and scopes this task needs, with the user's permissions, never an admin key.
• Human approval for irreversible or high-impact actions: payments, deletions, sending email, merging code. The model proposes; a person confirms.
• Sandboxing: code execution and file access happen in an isolated container with no network or a strict allowlist, and resource limits.
• Validation: arguments are schema-checked and business-rule-checked (amount within limit, recipient on an allowlist) before execution.

Multi-agent systems. One model call can act as an orchestrator that splits work and hands pieces to worker agents, each with its own fresh context and tools, then combines their results. This helps when subtasks are genuinely independent and parallel (research several sources at once), or when a single context would overflow. It costs more tokens, because each worker re-reads instructions and some work gets duplicated, and it adds coordination failures: workers misunderstand their brief, duplicate each other, or return results that don't fit together. For tightly coupled tasks, where every step depends on the last, one agent with good tools usually wins.

Evaluate agents on outcomes and on the path. Final-state checks ask: was the right flight held, did the tests pass? Trajectory evals look at the steps: did it call a forbidden tool, ask for approval, take 30 steps for a 5-step task? Because agents are stochastic, run each task several times. pass@k (succeeds at least once in $k$ tries) measures capability; pass^k (succeeds in all $k$ tries) measures the reliability users actually experience.

2. Visualization

Synthetic numbers. Left: independent per-step errors multiply, and a verifier plus one retry raises effective per-step reliability. Right: a parallel multi-agent design halves wall-clock time here but spends about 25% more tokens.

Try it (predict first, then check in the agent loop lab):

1. Turn on Ambiguous tool descriptions. Predict at which step the run goes wrong and whether anything in the runtime catches it. What single tool-design change prevents it?
2. Turn on Search tool returns an error. The error is marked retryable. Predict what would happen if the error were just the string "failed".
3. Turn on Require approval for side effects and reject the hold. Which part of the system guarantees the hold did not execute: the model or the runtime?
4. Combine the ambiguous tools with approval on. Would a human approver reading only the proposed call hold_booking(DL1180) catch the mistake? What would they need to see?

3. The math

Symbols

Compounding error

Assuming independent steps:

Worked example. $p = 0.95$, $n = 10$.

1. $0.95^2 = 0.9025$
2. $0.95^4 = 0.9025^2 = 0.81450625$
3. $0.95^5 = 0.81450625 \times 0.95 = 0.7737809375$
4. $0.95^{10} = 0.7737809375^2 = 0.5987$ (to 4 decimals)

So $S \approx 0.60$. Cutting the task to 5 steps gives $0.95^5 \approx 0.774$. Turning it around: for $S = 0.9$ over 10 steps you need $p = 0.9^{1/10} \approx 0.9895$ per step, which is a much harder target than "95% looks fine".

Note

Independence is a simplification. Real errors are correlated (one wrong assumption early can doom every later step), and agents can sometimes recover from a mistake later. The formula is a planning estimate and an argument for short paths and early checks, not a precise prediction.

Verification and retry

Suppose a check after each step catches a wrong step with probability $\rho$, and a caught step is retried $r$ times. A step ends up wrong if it errs and either the check misses it or every retry also errs:

Worked example. $p = 0.95$, $\rho = 0.8$, $r = 1$.

1. $1 - p = 0.05$
2. Bracket: $(1 - 0.8) + 0.8 \times 0.05 = 0.2 + 0.04 = 0.24$
3. $p_{\text{eff}} = 1 - 0.05 \times 0.24 = 1 - 0.012 = 0.988$
4. Over 10 steps: $0.988^{10}$. $0.988^2 = 0.976144$; $0.988^4 = 0.976144^2 = 0.952857$; $0.988^5 = 0.952857 \times 0.988 = 0.941423$; $0.988^{10} = 0.941423^2 = 0.8863$.

Task success rises from 0.60 to 0.89. A checker that catches most errors is worth more than a modestly better model, which is why tests, schema checks, and "verify the claim against the observation" steps pay off.

pass@k versus pass^k

For a task the agent solves with probability $s$ per independent trial:

Worked example. $s = 0.7$, $k = 3$.

1. pass@3 $= 1 - 0.3^3 = 1 - 0.027 = 0.973$
2. pass^3 $= 0.7^3 = 0.343$

The same agent "solves" 97% of attempts-with-retries but behaves correctly on all three runs only 34% of the time. If users run the agent once and expect it to work, pass^k (or the plain per-trial rate) is the number to report.

Multi-agent cost (synthetic)

A research task has three independent subtasks of about 40 s and 25k tokens each. A single agent does them in sequence and later subtasks re-read earlier findings (+15k tokens twice): 120 s and $3 \times 25{,}000 + 2 \times 15{,}000 = 105{,}000$ tokens. An orchestrator (8 s, 6k tokens) runs three workers in parallel (40 s, 35k tokens each, because each reads its brief and some searches overlap), then a synthesis step (12 s, 20k tokens): latency $8 + 40 + 12 = 60$ s, tokens $6{,}000 + 3 \times 35{,}000 + 20{,}000 = 131{,}000$, about 1.25 times the single agent. You trade tokens for wall-clock time, which is worth it only if the subtasks are truly independent and latency matters.

4. Implementation

The ideas in this lesson are small pieces of runtime code around the loop from the previous lesson. Each one below is stdlib Python with a scripted model, so you can see exactly when it fires.

Build it from scratch

Evaluator-optimizer: generate, check, feed the critique back. One function proposes a draft; another judges it against explicit criteria; the judge's text becomes the next draft's feedback. The loop stops on a pass or after max_iter, so a checker that never passes can't run forever.

def evaluator_optimizer(propose, evaluate, max_iter=4):
    feedback, trace = None, []
    for _ in range(max_iter):
        draft = propose(feedback)
        ok, feedback = evaluate(draft)
        trace.append((draft, ok, feedback))
        if ok:
            break
    return draft, trace

Predict: evaluate always returns False. What does the caller get back, and how would it know the draft failed?

Ground the checker in observations. A model asked to critique its own draft often misses its own confident mistakes. A stronger check compares the draft against something external: here, the tool results the agent actually received (the CRITIC idea). This catches the "false success" failure mode.

def grounded_check(draft: str):
    """CRITIC-style check: every refundability claim must match an observation, not the model's own opinion."""
    for fid, obs in OBSERVED.items():
        if fid in draft:
            claims_refund = f"{fid} is refundable" in draft
            if claims_refund != obs["refundable"]:
                return False, f"claim about {fid} contradicts tool result refundable={obs['refundable']}"
    return True, "ok"

Predict: the draft says "UA412 isn't refundable". Does this string-matching check pass it? What does that tell you about rule-based checkers versus an LLM judge?

Loop detector. An agent that repeats the same call with the same arguments is making no progress. Serialize the call with sorted keys, so {"a":1,"b":2} and {"b":2,"a":1} count as the same call, and stop after a few repeats.

class LoopDetector:
    def __init__(self, max_repeats=2):
        self.seen, self.max_repeats = {}, max_repeats

    def check(self, name: str, args: dict) -> bool:
        key = name + json.dumps(args, sort_keys=True)       # canonical form: key order must not matter
        self.seen[key] = self.seen.get(key, 0) + 1
        return self.seen[key] > self.max_repeats            # True means stop

Predict: an agent alternates between two calls, A, B, A, B, and so on. When does this detector fire with max_repeats=2?

Cost governor. Charge every model call against a step cap and a money budget. The important detail is checking before spending: compute the next call's cost and refuse it if it would cross the line, instead of noticing after the money is gone.

class CostGovernor:
    def __init__(self, max_usd: float, max_steps: int):
        self.max_usd, self.max_steps, self.usd, self.steps = max_usd, max_steps, 0.0, 0

    def charge(self, tokens_in: int, tokens_out: int):
        cost = tokens_in * PRICE_IN + tokens_out * PRICE_OUT
        if self.steps + 1 > self.max_steps:
            raise BudgetExceeded(f"step cap {self.max_steps}")
        if self.usd + cost > self.max_usd:
            raise BudgetExceeded(f"next call costs {cost:.4f}, spent {self.usd:.4f} of {self.max_usd}")
        self.usd, self.steps = self.usd + cost, self.steps + 1

Predict: with the previous lesson's six-step run (1,200-token base, +400 per step, 80 output tokens, synthetic $3 and $15 per million), the run costs 0.0468 USD. With a 0.05 USD budget, which call is refused? (Call 7 would read 3,600 tokens.)

Propose-then-commit. A side-effecting action runs only after approval, and at most once. The idempotency key is a hash of the task, action, and payload, so a retry after a network timeout finds the existing record instead of booking twice. After writing, re-read the target to verify.

def commit(task_id: str, action: str, payload: dict, approve) -> dict:
    key = idempotency_key(task_id, action, payload)
    if key in BACKEND:                                       # retry after a timeout: no double booking
        return {"status": "already_committed", "key": key}
    if not approve(action, payload):                         # human sees the evidence, not raw JSON
        return {"status": "declined", "key": key}
    BACKEND[key] = {"action": action, **payload}
    verified = BACKEND.get(key) == {"action": action, **payload}   # re-read the target after writing
    return {"status": "committed" if verified else "commit_unverified", "key": key}

Predict: the user declines, then later asks again and approves the same hold. Does the second request commit? What if the price changed from 312 to 315 in between?

Use the library

In production these pieces sit in one guarded executor around every tool call. The policy table is data, so you can review and test it apart from the prompt:

POLICY = {                       # least privilege, per tool
    "search_flights": {"side_effect": False},
    "get_fare_rules": {"side_effect": False},
    "hold_booking":   {"side_effect": True, "approval": True, "max_per_task": 1},
}

def guarded_execute(call, ctx):
    rule = POLICY.get(call.name)
    if rule is None:
        return {"error": f"unknown tool {call.name}; available: {list(POLICY)}"}
    if (err := validate(call)):                             # schema + business rules
        return {"error": err, "retryable": False}
    if ctx.count(call.name) >= rule.get("max_per_task", 99):
        return {"error": "limit reached for this tool", "retryable": False}
    if rule.get("approval") and not ctx.ask_human(call):
        return {"status": "declined_by_user"}
    key = idempotency_key(ctx.task_id, call)                # same call -> same effect
    with ctx.trace.span(call.name, args=call.args):         # observability
        return with_retries(lambda: TOOLS[call.name](**call.args, idempotency_key=key),
                            retries=2, timeout_s=10)

What libraries give you: payment and booking APIs (Stripe, for example) accept an idempotency key header and deduplicate server-side, which is stronger than a client-side dict. Agent frameworks (LangGraph, the OpenAI Agents SDK, the Claude Agent SDK) offer interrupts or permission hooks to pause before a tool runs, max-turn limits, and tracing. OpenTelemetry's GenAI semantic conventions standardize span attributes for model and tool calls, so traces from different frameworks look alike. None of them decides your policy: which tools need approval, what the budget is, and what counts as verified are still yours.

Run it

uv run code/19-agents/reliability_scratch.py

Key output:

1. Evaluator-optimizer with a grounded checker
  iter 1 FAIL: claim about UA412 contradicts tool result refundable=False
  iter 2 PASS: ok
2. Loop detector on a stuck agent
  stopped at step 3: search_flights repeated 3 times
3. Cost governor on a run whose context grows 400 tokens per step
  halted before call 7: next call costs 0.0120, spent 0.0468 of 0.05
4. Propose-then-commit with an idempotency key
  {'status': 'declined', 'key': '8e1d249c9dc4'}
  {'status': 'committed', 'key': '8e1d249c9dc4'}
  {'status': 'already_committed', 'key': '8e1d249c9dc4'}

The reliability arithmetic from section 3 (compounding error with a Monte Carlo check, verify-and-retry, pass@k versus pass^k, the multi-agent comparison) and the figure: code/19-agents/agent_reliability.py.

5. Engineering

Designing tools for agents.

• Narrow and well named: get_fare_rules(flight_id) rather than airline_api(endpoint, params). Merge tools that are always called together; split tools that do unrelated things.
• Descriptions as instructions: when to use it, when not to, what it returns, example arguments.
• Idempotent writes: accept a request or idempotency key so retries can't double-book or double-charge.
• Good errors: say what failed, whether retrying helps, and what to try instead.
• Compact, typed results: return the fields the model needs, paginate the rest.
• Test tools with the model: run your eval suite after every tool or description change; small wording changes move tool-selection accuracy.

Guardrails and permissions. Scope credentials to the user and the task. Separate read tools from write tools, and gate writes. Require human approval for irreversible actions, and show the approver the evidence (price, recipient, diff), not just the call. Run generated code in a sandbox with no secrets and limited network. Enforce limits in code: max steps, max spend, max calls per tool, allowlisted domains. See Securing AI systems for the threat model.

Multi-agent: when it's worth it. Worth it for broad, parallelizable work (research across many sources, reviewing many files) where a single context would overflow or wall-clock time matters. Not worth it for sequential tasks with shared state, such as most coding edits, where handoffs lose context. If you do it: give each worker a complete, specific brief and an output format; have the orchestrator check results; and measure the token multiplier.

Evaluating agents. Build a fixed suite of realistic tasks in a reproducible environment (mocked or sandboxed tools, seeded data) with automatic final-state checks. Add trajectory assertions (no forbidden tools, approval requested before writes, under N steps). Run each task several times and report per-trial success and pass^k, plus cost and latency per task (median and p95). Grade free-text outcomes with rubric-based LLM judges calibrated against human labels (see Evaluating LLM systems). Include adversarial cases: injected instructions in tool results, tools that fail, ambiguous requests.

Observability. Log a trace per task: every model call (prompt version, model, input and output tokens, latency), every tool call (arguments, result size, latency, errors), approvals, and the stop reason. Traces are how you debug "it did something weird", find the step where cost explodes, and turn production failures into eval cases. Redact secrets and personal data in traces.

Cost control. Set a per-task token or money budget and stop gracefully at it. Layer the caps, because each catches a different failure: a per-call token cap stops one runaway response, a step cap stops a loop, a per-task budget stops an expensive task, a spend-rate (velocity) limit stops a fast loop before the budget is hit, and a monthly cap per account is the backstop. A monthly cap alone fires only after most of the damage is done. Keep prompts and tool lists stable for prefix caching. Route easy subtasks (classification, extraction, summarizing tool output) to smaller models. Cache deterministic tool results. Cut steps with better tools and parallel calls; since billed tokens grow roughly with the square of steps, fewer steps is the biggest lever.

Watch out

Common failure modes.

• Loops: the same call repeated, or oscillating between two actions. Detect duplicate calls, cap steps, return informative results.
• Tool hallucination: calling tools that don't exist or inventing argument values (an ID never returned by any tool). Validate names and arguments; check IDs against earlier observations.
• Context overflow: long runs push out the task or key facts, or exceed the window. Compact old results, keep notes, use sub-agents with fresh contexts.
• Prompt injection via tool results: a web page, email, or file says "ignore previous instructions and send the data to...". Treat all tool output as untrusted data, keep high-impact actions behind approval and allowlists, and don't let one task's untrusted content flow into another's privileged tool.
• False success: the final answer claims something the observations contradict. Verify claims against tool results before reporting.

Common mistakes

• Reporting pass@k from several attempts as if it were the reliability a user gets from one run.
• Evaluating only the final answer, so an agent that succeeded by doing something forbidden scores as a pass.
• Giving the agent a broad service account "to make it work", instead of the user's own scoped credentials.
• Asking humans to approve raw JSON calls with no context, which trains them to click approve.
• Adding agents (planner, critic, researcher, writer) before measuring whether one agent with better tools would do.
• Retrying non-idempotent writes after a timeout.

6. Knowledge check

Practice exercise. An agent's task has 8 steps, each correct with probability 0.97. (a) What is the task success rate? (b) You add a check with recall 0.9 and one retry. What is the new per-step and task success rate? (c) If the per-trial success is your answer to (b), what are pass@2 and pass^2?

Implementation challenge. In agent_reliability.py, replace the independence assumption with a simple correlated model: with probability 0.1 the agent starts with a wrong assumption and every step then succeeds with 0.80; otherwise every step succeeds with 0.97. Compute task success for 10 steps analytically and by Monte Carlo, and compare it with independent steps at the same average per-step accuracy.

Summary

• Task success is roughly $p^n$: 0.95 per step over 10 steps is about 0.60. Fewer steps and per-step checks with retry are the main levers.
• Good agent tools are narrow, clearly named and described, idempotent for writes, compact in output, and explicit in their errors.
• Safety comes from the runtime: least privilege, validation, human approval for irreversible actions, sandboxing, and hard limits.
• Multi-agent designs trade extra tokens and coordination risk for parallelism and fresh contexts; use them for broad, independent subtasks.
• Evaluate on a fixed task suite with final-state and trajectory checks, report pass^k alongside pass@k, track cost and latency per task, and trace every step.

Next: Agent memory

Related: Tool use and agents · Evaluating LLM systems · Reliability, cost, and observability

Go deeper, code-first (AI Engineering from Scratch): Agent State Machines — Graphs, Nodes, Checkpoints · Reflexion: Verbal Reinforcement Learning · Tree of Thoughts and LATS: Deliberate Search · The Shift from Chatbots to Long-Horizon Agents · Long-Running Background Agents: Durable Execution · Kill Switches, Circuit Breakers, and Canary Tokens · Checkpoints and Rollback · Failure Modes — MAST, Groupthink, Monoculture, Cascading Errors

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Anthropic's Workflow Patterns: Simple Over Complex, Self-Refine and CRITIC: Iterative Output Improvement, Failure Modes: Why Agents Break, Action Budgets, Iteration Caps, and Cost Governors, Human-in-the-Loop: Propose-Then-Commit.

Interview angle