Production architecture

1. Intuition

Follow one request through a support assistant:

1. The client sends a question.
2. The API gateway authenticates the user, enforces rate limits and quotas, and rejects oversized inputs.
3. Orchestration picks a prompt template and a model (routing), checks the cache, calls retrieval with the user's permission filters, assembles the prompt, calls the model, and validates the output against a schema. If the model proposes a tool call ("look up order 123"), orchestration checks it against an allowlist and the user's permissions, executes it with a timeout, and feeds the result back.
4. Every step emits traces and logs (with secrets and personal data redacted) to monitoring.
5. Samples of traffic flow into evaluation (offline test sets, online feedback), and governance controls retention, access, audits, and policy.

Batch versus online. Batch inference processes large volumes on a schedule (nightly classification of all tickets): cheap, high throughput, latency doesn't matter. Online inference answers live requests: latency and availability matter, and capacity must handle peaks.

Hosted API versus self-hosted. Hosted APIs give top models with no infrastructure, at per-token prices and with data leaving your network. Self-hosting open models gives control over data, latency, and cost at scale, but you own GPUs, serving software, and upgrades.

2. Visualization

Try it (predict first, then check):

1. Trace the Normal scenario and predict which hop dominates the latency before you reach it.
2. Switch to Outage. Which two patterns keep the request from failing, and what would happen without a timeout?
3. In Tool call, find the step where the application, not the model, decides whether the refund runs.

3. The math

Symbols

Capacity: Little's law

At 20 requests per second with an average of 3 seconds per request, about 60 requests are in flight at once, so the model backend and connection pools must support at least 60 concurrent requests, with headroom for peaks.

Cost per request

Worked example: routing

Two models: a small one costs 0.1 cents per request and is good enough on 80% of traffic (simple FAQ questions); a large one costs 1.5 cents. Sending everything to the large model costs 1.5 cents per request. A router that sends the easy 80% to the small model and the rest to the large model costs $0.8 \times 0.1 + 0.2 \times 1.5 = 0.08 + 0.30 = 0.38$ cents, a 75% saving, if the router's misroutes don't hurt quality. Measure quality per route on the evaluation set before trusting the saving.

Worked example: capacity

Peak traffic is 50 requests per second; p95 end-to-end latency is 4 seconds. Little's law with $W = 4$ as a conservative bound gives $L = 200$ concurrent requests at peak. If one GPU server handles 25 concurrent generations at acceptable latency, you need 8 servers, plus redundancy for failures and deployments.

4. Implementation

Build it from scratch

The full script is code/20-production-ai/canary_scratch.py. It builds the three things that keep a model change from hurting users: a request path that degrades instead of failing, a shadow comparison, and a canary that rolls itself back. Models are fake functions with synthetic latency, cost, and error rates.

A fake model. Each "model" returns text, a latency, and a cost, or raises to simulate a 503. Latency is jittered so percentiles mean something.

def make_model(name, latency_ms, cost, error_rate, length):
    """A fake model: returns (text, latency_ms, cost_usd) or raises on a simulated failure."""
    def call(prompt, rng):
        if rng.random() < error_rate:
            raise RuntimeError(f"{name}: 503")
        lat = latency_ms * rng.uniform(0.7, 1.6)
        return f"{name} answer to {prompt!r}"[: length], lat, cost
    call.name = name
    return call

Predict: with error_rate=0.40, roughly how many of 1,000 calls raise?

The request path. Try the primary, fall back once, and treat "too slow" the same as "failed". The user always gets something: an answer or an honest apology. Nothing hangs.

def handle(prompt, primary, fallback, rng, timeout_ms=2000):
    """One request: try the primary under a deadline, fall back once, never hang."""
    for model in (primary, fallback):
        try:
            text, lat, cost = model(prompt, rng)
        except RuntimeError:
            continue                      # dependency failed: try the next model
        if lat > timeout_ms:
            continue                      # too slow: treat as a timeout
        return {"model": model.name, "text": text, "latency_ms": lat, "cost": cost}
    return {"model": "none", "text": "Sorry, try again shortly.", "latency_ms": timeout_ms, "cost": 0.0}

Predict: if both models are down, what latency does the request report, and why is that the right number to log?

Shadow mode. Send the same prompts to production and to the candidate. Only production's answer reaches the user; the candidate's numbers are logged. This catches cost blow-ups, latency regressions, and hard failures with zero user impact. It cannot tell you whether users prefer the new answers.

def shadow(prompts, prod, cand, seed=0):
    """Send the same prompts to both; users only ever see prod. Log the deltas."""
    rng = random.Random(seed)
    rows = []
    for p in prompts:
        a = handle(p, prod, prod, rng)
        b = handle(p, cand, cand, rng)        # candidate output is logged, never returned
        rows.append((a["latency_ms"], b["latency_ms"], a["cost"], b["cost"], b["model"] == "none"))
    return {
        "latency_ratio": statistics.median(r[1] for r in rows) / statistics.median(r[0] for r in rows),
        "cost_ratio": sum(r[3] for r in rows) / sum(r[2] for r in rows),
        "cand_fail_rate": sum(r[4] for r in rows) / len(rows),
    }

Predict: why compare medians of latency but sums of cost?

Gates. A canary passes a stage only if each guard metric stays within a multiple of the baseline. The thresholds are set above the run-to-run noise, because LLM outputs and latencies vary even for identical inputs.

STAGES = [0.01, 0.10, 0.25, 0.50, 1.00]
GATES = {"p95_ms": 1.5, "cost": 1.2, "error_rate": 2.0}   # canary may be at most this multiple of baseline

def p95(xs):
    xs = sorted(xs)
    return xs[int(0.95 * (len(xs) - 1))]

Predict: a candidate's p95 is 1.4 times baseline. Does it pass?

Measuring a stage. Count errors separately from successful latencies. The 1e-3 floor stops a zero-error baseline from making any single canary error look infinitely worse.

def measure(model, n, rng):
    lats, costs, errs = [], [], 0
    for i in range(n):
        try:
            _, lat, cost = model(f"q{i}", rng)
            lats.append(lat)
            costs.append(cost)
        except RuntimeError:
            errs += 1
    return {"p95_ms": p95(lats), "cost": statistics.mean(costs), "error_rate": max(errs / n, 1e-3)}

The rollout. Walk 1% → 10% → 25% → 50% → 100%. At each stage compare canary to baseline measured at the same time, and stop at the first breach. Rolling back is a config flip, so it takes seconds.

def canary(baseline, cand, requests_per_stage=4000, seed=1):
    """Shift traffic stage by stage; compare canary to baseline at each stage; roll back on any breach."""
    rng = random.Random(seed)
    for share in STAGES:
        n_cand = max(200, int(share * requests_per_stage))      # floor so early stages have signal
        base, can = measure(baseline, requests_per_stage, rng), measure(cand, n_cand, rng)
        breaches = [k for k, mult in GATES.items() if can[k] > base[k] * mult]
        print(f"  {share:>4.0%}  p95 {can['p95_ms']:6.0f} vs {base['p95_ms']:6.0f} ms  "
              f"err {can['error_rate']:.3f} vs {base['error_rate']:.3f}  "
              f"{'ROLLBACK: ' + ','.join(breaches) if breaches else 'pass'}")
        if breaches:
            return "rolled back"
    return "promoted"

Predict: the 1% stage only sees 40 canary requests at 4,000 per stage. Why does the code use at least 200?

Use the library

Structured output with validation, so free text never flows into code unchecked:

from pydantic import BaseModel, Field, ValidationError

class TicketTriage(BaseModel):
    category: str = Field(pattern="^(billing|auth|shipping|other)$")
    urgency: int = Field(ge=1, le=5)
    summary: str = Field(max_length=280)

def triage(ticket: str) -> TicketTriage:
    for attempt in range(2):                                   # one bounded retry
        raw = llm(PROMPT.format(ticket=ticket), response_format="json")
        try:
            return TicketTriage.model_validate_json(raw)
        except ValidationError as e:
            log.warning("invalid model output", extra={"attempt": attempt, "error": str(e)})
    return TicketTriage(category="other", urgency=3, summary="needs human review")   # safe fallback

A tool call is a request from the model; the application decides whether to run it:

ALLOWED = {"get_order_status": get_order_status}           # explicit allowlist

def run_tool(call, user):
    fn = ALLOWED.get(call.name)
    if fn is None:
        raise PermissionError(f"tool not allowed: {call.name}")
    args = OrderLookup.model_validate(call.arguments)      # validate arguments with a schema
    if not user.can_view(args.order_id):                   # authorization uses the USER's rights
        raise PermissionError("forbidden")
    return with_timeout(fn, args, seconds=5)

For progressive delivery itself you rarely write the loop above. Feature-flag systems (for example via the vendor-neutral OpenFeature API) hold the traffic split, a model registry pins the exact model version, and tools such as Argo Rollouts or Flagger automate stage-by-stage promotion against metric queries. The scratch version shows what those tools check for you.

Run it

From guide/:

uv run code/20-production-ai/canary_scratch.py

Key output:

Request path (primary flaky, fallback small):
  served by: {'flaky-v1': 601, 'none': 6, 'small-v1': 393}
Shadow (candidate = cheaper, faster model):
  latency ratio 0.36  cost ratio 0.20  candidate failures 0.0%
Canary cand-good:
  ...
  100%  p95   1272 vs   1249 ms  err 0.011 vs 0.011  pass
  -> promoted
Canary cand-slow:
    1%  p95   2198 vs   1245 ms  err 0.015 vs 0.011  ROLLBACK: p95_ms
Canary cand-pricey:
    1%  p95   1256 vs   1245 ms  err 0.015 vs 0.011  ROLLBACK: cost
all checks passed

The 40% flaky primary still leaves only 6 of 1,000 requests unanswered (both models failed or timed out). The slow and expensive candidates are stopped at 1% of traffic, before most users ever see them.

5. Engineering

Prompt templates are code. Version them, review changes, and evaluate before deploying.

Model versioning. Pin model versions; upgrades are deployments that need evaluation and gradual rollout. Record prompt, model, and index versions in every trace.

Agents. An agent is a system that plans or loops across model and tool calls. Bound it: maximum steps, maximum cost, timeouts, and human approval for consequential actions.

Watch out

Failure modes. Trusting raw model text as code or commands; tools running with service-wide credentials instead of the user's permissions; retries that duplicate side effects (charging a card twice); unbounded agent loops; logs containing secrets or personal data; no fallback when the model provider is down.

Shipping a model change: shadow, then canary

A new model, prompt, or index version is a deploy. Treat it like one.

1. Offline eval on the golden set (see LLM evaluation).
2. Shadow: mirror real traffic to the candidate, log its outputs, tokens, latency, and errors, and return nothing from it. This is a smoke test for cost and breakage, not a quality test.
3. Canary: shift traffic in stages (for example 1%, 10%, 25%, 50%, 100%) and gate each stage on latency percentiles, cost per request, error and refusal rate, output length, and user feedback, all compared to the baseline over the same window.
4. Rollback must be a flag flip plus a pinned previous model version. If rolling back needs a redeploy, fix that first.

Set gate thresholds above the natural noise. The same prompt can produce different outputs and latencies from run to run (sampling, batch composition, floating-point reduction order), so "stable" means "within the expected spread", not "identical to baseline".

Common mistakes

• Parsing free-form prose with regular expressions instead of using structured outputs and schemas.
• Building the whole system around one model provider with no abstraction or fallback.
• Shipping prompt changes without evaluation.

6. Knowledge check

Practice exercise. Traffic averages 8 requests per second and each request takes 2.5 seconds. How many requests are in flight on average? If a peak is 3× average traffic, how many concurrent slots do you need?

Implementation challenge. Build a minimal FastAPI service for the ticket-triage example: an endpoint that validates input size, calls a (mock or real) model, validates the output with Pydantic, retries once, falls back safely, and emits a structured log line with latency, token counts, model version, and prompt version.

Summary

• The model is one component; gateway, orchestration, retrieval, tools, cache, observability, evaluation, and governance make it a product.
• Use structured outputs and schemas; treat tool calls as requests your code authorizes and validates.
• Route between models by quality, latency, cost, and privacy, and measure the trade-off.
• Little's law ($L = \lambda W$) sizes concurrency; version prompts, models, and indexes like code.

Next: Reliability, cost, and observability

Related: Securing AI systems · The RAG pipeline

Go deeper, code-first (AI Engineering from Scratch): LLM Routing Layer — LiteLLM, OpenRouter, Portkey · Production Runtimes: Queue, Event, Cron · AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Building a Production LLM Application, Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs.

Interview angle