🏗️ Building High-Quality AI Agents 🤖 — A Comprehensive, Actionable Field Guide 📚

A synthesis of hard-won lessons from Claude Code, OpenHands, SWE-agent, GoClaw, Nanobot, PicoClaw, and the emerging discipline of harness engineering. This is the guide we wish existed when we started building agents.

The goal: an AI agent that is fast, scalable, capable, reliable, efficient, and secure — not by accident, but by design.

Companion reads: 🏗️ Building Agents Like Claude Code — A Source-Derived Blueprint 📘, 🙌 OpenHands — Deep Dive & Build-Your-Own Guide 📚, 🤖 SWE-agent — Deep Dive & Build-Your-Own Guide 📘, 🦊 GoClaw Deep Dive 🤖 — A Builder's Guide to a Multi-Tenant AI Agent Platform 📘, 🤖 nanobot: A Comprehensive Build-Your-Own Guide 📚, and 🦀 PicoClaw Deep Dive 🤖 — A Field Guide to Building an Ultra-Light AI Agent in Go 🐹.

📖 How to read this guide

• Read top-to-bottom for the full mental model. Each section builds on the previous one.
• Skim the boxes if you only want the takeaways — every section ends with an Actionable rules box.
• Jump to Part 14 — The Build-Your-Own Roadmap if you already know the theory and want a sequenced plan.
• Bookmark Part 15 — Anti-Patterns for design reviews.

📋 Table of Contents

• ⚡ Part 0 — The Core Equation
• 🧠 Part 1 — Mental Model: What an AI Agent Actually Is
• 🔄 Part 2 — The Agent Loop (the Kernel)
• 🛠️ Part 3 — Tools: The Agent's Hands
• 💭 Part 4 — Context Engineering
• 💾 Part 5 — Memory (Long-Term Knowledge)
• ⚡ Part 6 — Concurrency & Multi-Agent Patterns
• 🔧 Part 7 — Reliability: Error Recovery, Stuck Detection, Autosubmit
• 🔒 Part 8 — Security: Defense-in-Depth
• 🏢 Part 9 — Multi-Tenancy from Day One
• 🚀 Part 10 — Performance & Efficiency
• 🔌 Part 11 — Provider Abstraction & Resilience
• 📡 Part 12 — Channels & Integration Surface
• 📊 Part 13 — Observability & Evaluation
• 🗺️ Part 14 — The Build-Your-Own Roadmap
• ⚠️ Part 15 — Anti-Patterns to Avoid
• 🎯 Part 16 — Closing: The Harness Mindset

⚡ Part 0 — The Core Equation: Agent = Model + Harness

The single most important insight in agent engineering, framed by Vivek Trivedy of LangChain: "If you're not the model, you're the harness."

   Reliability  ≈  Model capability  ×  Harness quality
                          (fixed)        (your job)

The model is fixed for the duration of your project. The harness — system prompts, tools, sandboxes, memory, hooks, orchestration, observability — is where 80% of agent quality comes from. After hundreds of production agent sessions across many teams, the pattern is consistent:

It's almost never a model problem. It's a configuration problem.

Teams who blame weak results on the model are usually wrong. Teams who treat harness engineering as a discipline — equal in rigor to model selection — ship reliably.

This guide is about the harness.

🧰 What the harness contains

Every Part below covers one of these layers in depth.

🧠 Part 1 — Mental Model: What an AI Agent Actually Is

Before any code: hold the right picture in your head.

📖 The one-paragraph definition

An AI agent is a streaming, cancellable, recursive state machine that emits typed actions against an environment, observes the consequences, and runs in a loop until a typed terminal state. The model is a function from history to next action. Everything else — memory, sub-agents, security, hooks — is a small module hanging off that single loop.

🔁 The canonical loop

loop while not done:
   state    = compress(state)               # context fits the budget
   response = await model(state)            # streaming
   yield response                           # to UI / caller
   if no tool_calls: return completed
   batches  = partition(response.tool_calls)# concurrent vs serial
   for batch in batches:
     results = run(batch)                   # parallel where safe
     yield results
     state += results

That's it. Five lines. Every agent in this guide — Claude Code, OpenHands, SWE-agent, GoClaw, Nanobot, PicoClaw — implements that exact shape, with variations in how each piece is realized.

🎛️ The five universal abstractions

Every production agent reduces to these. Implement them as first-class modules, not helpers attached to a god object.

📐 The four design principles to lift wholesale (from OpenHands V1)

1. Optional isolation, not mandatory sandboxing. The agent runs in-process by default; swap LocalWorkspace → DockerWorkspace for isolation without changing agent code. Don't make sandboxing a build-time decision.
2. Stateless components, single source of truth. Agent, Tool, LLM, and Condenser are immutable models. The only mutable thing is ConversationState. State changes happen by appending events — never by mutating objects.
3. Strict separation of concerns. The SDK never imports applications. The CLI, GUI, and your custom integration all consume the SDK as a library.
4. Two-layer composability. Compose at package level (swap workspaces, swap servers) and at component level (swap tools, prompts, condensers, LLMs).

📏 Five rules carry 80% of the design (from Claude Code's source)

1. 🔄 The loop is an async generator. Backpressure, cancellation, and typed terminal states fall out for free.
2. 📝 Every tool is self-describing (schema, permissions, concurrency safety). The loop never special-cases tools.
3. 🛡️ Safety is per-invocation, not per-tool. Bash("ls") ≠ Bash("rm -rf").
4. 💾 Prompt cache is architecture, not optimization. Every design either preserves cache hits or busts them.
5. 📁 Memory is files. A small LLM picks which to load. No vector DB, no embeddings (yet). Trust through transparency.

✅ Actionable rule: Build those five things well and you have ~80% of a production agent. Everything else is layering and polish.

🔄 Part 2 — The Agent Loop (the Kernel)

The loop is the only place control flow lives. Everything else — tools, hooks, sub-agents — yields through it.

2.1 🔀 Why an async generator (not a callback web)

Three concrete reasons:

1. Backpressure for free. A generator yields only when the consumer calls .next(). The UI naturally pauses if it can't render fast enough.
2. Typed terminal states. The generator's return is a discriminated union of why execution stopped: completed, max_turns, error, aborted_streaming, aborted_tools, prompt_too_long, image_error, model_error, stop_hook_prevented, hook_stopped. The compiler enforces exhaustive handling.
3. Composability. Inner generators delegate via yield*. No callback nesting.

In Python, use async def + async for. In Go, use channels or a method that returns a (StepOutput, error) tuple per call.

2.2 🗂️ Phases worth memorizing (the 5-phase shape)

Borrowed from OpenHands' Agent.step() and SWE-agent's forward_with_handling:

def step(self):
    # 1. Drain confirmed actions waiting to execute (confirmation flow).
    if pending := drain_pending(state):
        return execute(pending)

    # 2. Honor user-input or stop hooks that could block this turn.
    if blocked := check_hooks(state):
        return finished(reason=blocked)

    # 3. Build the LLM prompt — may return a Condensation event instead.
    msgs = prepare_llm_messages(state.events, condenser, llm)
    if isinstance(msgs, Condensation):
        return msgs

    # 4. Call the LLM, with explicit context-window handling.
    try:
        response = call_llm(msgs, tools, on_token=stream)
    except ContextWindowExceeded:
        return CondensationRequest()  # next step compresses

    # 5. Classify and dispatch.
    match classify(response):
        case TOOL_CALLS:    return execute_tools(response)
        case CONTENT:       return emit_message(response)
        case EMPTY:         return retry_with_nudge()

The hardest bug in agent loops is "the LLM responded but my code didn't know what to do with it." Explicit response classification kills that bug. Branch on a tagged union, not on a string match.

2.3 ♻️ Continue-states (don't return, just continue)

Some failures should re-enter the loop, not terminate it:

Name each transition (collapse_drain_retry, reactive_compact_retry, max_output_tokens_escalate). Every test asserts which transition fired.

Hard rule: Guards prevent infinite loops. hasAttemptedReactiveCompact one-shot flags. Hard caps on recovery attempts. Never run stop-hooks on an error response — that creates "error → hook blocks → retry → error" spirals.

2.4 🔗 The tool_use / tool_result invariant

This is the single most common bug source.

Every tool_use your agent emits MUST have a paired tool_result in message history before the next API call.

Anthropic and OpenAI APIs both reject assistant messages that contain a tool_use block without a matching tool_result in the next user message.

On any cancellation path — Esc, timeout, error — drain queued tools by emitting synthetic tool_result blocks:

"Cancelled: parallel tool call Bash(mkdir build) errored"

signal.reason distinguishes hard aborts from "submit interrupts" (a new user message), so you skip redundant interruption stubs in the latter case.

2.5 🧹 Loop hygiene every iteration

Before each LLM call, OpenHands' Runner.run enforces these invariants:

• Drop orphan tool_results — if the LLM forgot to emit tool_use for a tool_result, strip it.
• Backfill missing tool_results — if the LLM emitted tool_use with no matching result, synthesize an error placeholder so the trace is well-formed.
• Microcompact — fast pre-call truncation of large blobs.

2.6 💰 Concrete budgets

These defaults are battle-tested across the open-source agent ecosystem:

Cost is the primary stop signal, not steps. Step counts vary wildly across models — Claude uses few long steps, GPT-5 uses many short ones. A $3-per-task cap normalizes this.

✅ Actionable rules

• The loop is one async generator with five named phases.
• Branch on a typed union of LLM response classifications.
• Pair every tool_use with a tool_result before the next API call.
• Cap iterations and cost; cost is the primary signal.
• Errors that should retry use continue states; errors that shouldn't return terminal states.
• Never run stop-hooks on error responses.

🛠️ Part 3 — Tools: The Agent's Hands

If the loop is the kernel, tools are the kernel's syscalls.

3.1 🔌 The tool interface

A tool is parameterized by three types: Input, Output, Progress. Five members are critical:

interface Tool<I, O, P> {
    name: string
    description: string
    inputSchema: Schema           // Zod / Pydantic / JSON Schema
    isConcurrencySafe(input): boolean   // PER INVOCATION, not per type
    checkPermissions(input, ctx): allow | deny | ask | passthrough
    validateInput(input, ctx): ok | error
    call(input, ctx): Result<O>
}

3.2 🏭 The buildTool() factory pattern (fail-closed)

Never construct a tool literal directly. Wrap it in a factory that fills in safe defaults:

const SAFE_DEFAULTS = {
    isEnabled:        () => true,
    isParallelSafe:   () => false,    // serial unless proven otherwise
    isReadOnly:       () => false,    // assume writes
    isDestructive:    () => false,
    checkPermissions: (input) => ({ behavior: 'allow', updatedInput: input }),
}

If a tool author forgets isConcurrencySafe, they get serial execution — slow, but never corrupting. The opposite default would silently produce race conditions.

3.3 🛡️ Safety is per-invocation, not per-type

Bash("ls -la") is concurrency-safe. Bash("rm -rf build/") is not. Same tool, different inputs, different verdicts.

Pass the parsed input to the safety check. If parsing fails or the check throws, default to serial execution. Always fail closed.

3.4 🪜 The 14-step tool execution pipeline

Every tool call in Claude Code goes through this single pipeline:

Implement this as one function (checkPermissionsAndCallTool). Skipping any step will hurt later.

3.5 ⚡ Concurrency partitioning

def partition(calls):
    batches = []
    current = ConcurrentBatch()
    for call in calls:
        tool = lookup(call.name)
        parsed = tool.schema.safe_parse(call.input)
        safe = parsed.success and tool.is_concurrency_safe(parsed.data)
        if safe and current.kind == 'concurrent':
            current.push(call)
        elif safe:
            batches.push(current); current = ConcurrentBatch([call])
        else:
            if current: batches.push(current)
            batches.push(SerialBatch([call]))
            current = ConcurrentBatch()
    if current: batches.push(current)
    return batches

Example: [Read, Read, Grep, Edit, Read] → [concurrent[Read, Read, Grep], serial[Edit], concurrent[Read]].

Yield results in submission order, not completion order — even if c.ts finishes before a.ts, the conversation history must remain a, b, c.

3.6 🚄 Speculative streaming execution

Watch the model stream. The moment a tool_use block is fully parsed (often seconds before the response finishes), start that tool — provided admission rules allow.

Admission rule: a tool can start executing iff no tool is currently running, or both the new tool and all currently-running tools are concurrency-safe.

Sequential timeline: stream 2.5s + 3 serial tools = 3.1s.
Speculative: stream 2.5s overlapped with tools 1–2 = 2.6s.

3.7 🚦 Three concurrency flags every tool needs

The runner partitions a turn's tool calls honoring these. That's how you get fast parallel reads without races on writes — without a real scheduler.

3.8 📏 Result budgeting: per-tool size caps

Above the cap, write the full content to a <persisted-output> file and return a preview pointing to it. An aggregate ContentReplacementState tracks per-conversation budgets.

3.9 🖥️ The Agent-Computer Interface (ACI) — the SWE-agent thesis

The single most-imitated idea in coding agents:

"Language models are a new kind of end user, and they need software interfaces designed for them, not for humans."

The empirical claim: same model, same problem, roughly 2× SWE-Bench score with ACI tools versus raw bash. The agent isn't smarter; it's better-housed.

🎛️ Four ACI design rules

1. Concise, bounded output. No cat-the-whole-file. No unbounded grep -rn. Every command's output has a maximum size and a structured shape.
2. Persistent state across turns. The runtime owns the cursor (CURRENT_FILE, FIRST_LINE). The agent never has to reconstruct "where am I" from history.
3. Guard rails on destructive actions. Edits run through a linter; failures auto-revert. Bash is -n-checked before execution.
4. Predictable, tiny argument grammar. Each command has 1–2 positional args max. Multi-line bodies bracketed by sentinels (end_of_edit).

🚀 The four flagship ACI tools

• Windowed file viewer (100 lines, 2-line overlap on scroll, goto N puts N about 1/6 down the window so the agent gets ~17 lines of preceding context).
• Bounded search (filenames + counts only, hard cap at 100 files, explicit start/end markers).
• Line-targeted edit with flake8 + auto-rollback. This is the most carefully engineered piece. On lint failure, show the bad state and the kept state side-by-side, with line numbers, both bracketed; revert automatically; tell the agent "DO NOT re-run the same failed edit command."
• Submit sentinel. A unique string (<<SWE_AGENT_SUBMISSION>>) means any tool can declare completion without a separate channel.

💡 CodeAct: code as the universal action (OpenHands)

A different direction reaches the same conclusion: instead of giving the LLM 20 bespoke tools each with its own JSON schema, give it bash + Python + a file editor + a browser and let it write code. Empirically this generalizes far better and dramatically reduces parsing errors.

Trade-off: relies on the model being a strong code generator. With Claude / GPT-5, "give it a shell" is the strongest baseline. With weaker models you may need narrower, more guided tools.

Pick your stance: narrow ACI tools (SWE-agent) or broad code-as-action (OpenHands). Both work. Don't do both at once — the prompt cost compounds.

3.10 🗂️ Tool registry assembly: cache-aware ordering

final = sort(builtins, alpha) ++ sort(mcpTools, alpha)

Sort within each partition, then concatenate. A flat sort across all tools would interleave MCP tools into built-in positions, busting the prompt cache whenever MCP servers change.

3.11 ⏳ Deferred tool loading

Tools marked shouldDefer: true send only { name, description, defer_loading: true } to the API. The model has to call ToolSearch to load full schemas. Three benefits:

• Smaller initial prompt
• Adding/removing a deferred tool changes the prompt by a few tokens, not hundreds — prompt cache stays warm
• Less tool-soup confusion for the model

✅ Actionable rules

• Wrap every tool in a buildTool() factory with safe-by-default flags.
• Concurrency safety is per-invocation, with read_only / concurrency_safe / exclusive metadata.
• Funnel every tool call through one 14-step pipeline.
• Pick narrow ACI tools or broad code-as-action — don't blend both.
• Cap tool output sizes; persist overflow to disk; return previews.
• Sort tools within partitions before concatenating; defer tools that aren't always needed.

💭 Part 4 — Context Engineering (the Brain's Working Memory)

The model's context window is your most expensive, most fragile resource. Treat it that way.

4.1 🧱 The dumb zone

LLMs degrade as context fills up. The relationship is non-linear: the last 20% of the window is dramatically lower-quality than the first 60%. Every token you waste in the first 60% pushes signal into the dumb zone.

Three forces fight for context:

1. System prompt (identity, tools, conventions) — static-ish
2. Memory (CLAUDE.md, AGENTS.md, project context) — semi-static
3. Working history (turns, tool results, observations) — pure churn

Keep #1 and #2 lean and stable. Compress #3 aggressively.

4.2 🗜️ The 4-layer compression pipeline

Run before every API call, in this strict order:

Why ordering matters: if collapse alone gets tokens below the auto-compact threshold, auto-compact never runs — so you keep fine-grained recent history. Cheap layers first.

4.3 📉 Budget thresholds

• Auto-compact triggers at effectiveContextWindow − 13,000 tokens.
• Hard blocking limit at effectiveContextWindow − 3,000.
• Mid-loop compaction triggers at 75% of context window during iteration. Summarize the first ~70% of in-memory messages, keep the last ~30%.
• Post-run compaction at 50 messages OR 75% context window.

Token counting blends authoritative API usage numbers with rough estimates for messages added since the last response — biased conservative so compaction fires slightly early.

Instrument both estimated and authoritative token counts; log the delta. When the delta drifts, your estimator is broken and your safety margins are wrong.

4.4 ✂️ Last-N-observations: the brutal default

A history processor that drops all but the most recent N observations from the messages array, keeping actions and thoughts in place but blanking out stdout of older steps. For a 50-turn trajectory, this is the difference between context overflow and a clean run.

SWE-agent's default is n=5. Aggressive but works; the agent's own thoughts plus recent state-command output give it enough to keep going.

4.5 🪄 The condenser pattern (OpenHands)

def maybe_condense(events, summarizer, max_size=80, keep_first=4):
    if len(events) <= max_size:
        return events
    head = events[:keep_first]                  # system prompt + task
    tail = events[-(max_size // 2):]            # recent work
    middle = events[keep_first:-(max_size // 2)]
    summary = summarizer.complete(
        "Summarize concisely, preserving decisions, findings, current state:\n"
        + dump(middle))
    return head + [SummaryEvent(text=summary)] + tail

Two trigger paths:

1. Proactive — check size on each step.
2. Reactive — when the LLM raises LLMContextWindowExceedError, emit a CondensationRequest event and try again next step.

Don't summarize on every step — only when over a threshold. Cache aggressively. The cheapest thing you can do is just truncate with a small head + recent tail; LLM summarization is the upgrade.

4.6 📂 Progressive disclosure: skills and tools

A separate compression strategy: don't load it until you need it.

Two-phase skill loading (the killer pattern):

• Phase 1 (startup): parse YAML frontmatter only — name, description, when_to_use. Inject as a directory.
• Phase 2 (invocation): load full markdown body, substitute $ARGUMENTS, execute inline shell commands, prepend as a user message.

You pay the token cost only when the skill actually runs. 50+ skills cost 50 lines, not 50 docs.

4.7 🔥 Sub-agents as context firewalls

The single biggest qualitative win in context engineering. Instead of letting research/grep output pollute the main conversation, delegate to a sub-agent with isolated context. The sub-agent returns a condensed answer with citations (e.g. auth.py:142-158).

Use sub-agents for:

• Codebase exploration
• Research and grep operations
• Long Q&A with simple final answers
• Anything that produces 200+ lines of intermediate output you don't need to keep

The parent never sees the 200 lines of grep output — only the sub-agent's 3-line summary.

4.8 🧩 The system prompt is 19+ composable sections

Not one giant blob — a list of small sections assembled at request time. Order matters. Build it like this (Claude Code / GoClaw):

1. Identity (channel-aware)
2. First-run bootstrap notice
3. Persona (SOUL.md, IDENTITY.md) — early "primacy zone"
4. Tooling (filtered + sandbox-aware)
5. Credentialed CLI context
6. Safety preamble + identity anchoring
7. Self-evolution rules (if applicable)
8. Skills inline (≤ 15) OR via skill_search
9. MCP tools inline OR via mcp_tool_search
10. Workspace info
11. Team workspace (if team agent)
12. Sandbox container info
13. User identity
14. Time (UTC)
15. Channel formatting hints
16. Extra context (<extra_context> tags)
17. Project/bootstrap context files
18. Sub-agent spawning rules
19. Runtime info (agent ID, model, pricing)
20. Persona reminder — late "recency zone" — fights "lost in the middle"
21. Memory reminders ("run memory_search first")

Two reinforcement zones (primacy + recency) are the cheapest reliability win in agent prompting. Put critical rules at both ends.

4.9 ❌ The "one big AGENTS.md" anti-pattern

A monolithic AGENTS.md crowds out task and code context. Too much guidance becomes non-guidance. It rots without maintenance.

Better: Treat AGENTS.md / CLAUDE.md as a table of contents (~60–100 lines max) pointing to deeper docs/ sources. Include build commands, test commands, key conventions, structure pointers. Exclude directory listings, conditional rules, auto-generated content.

✅ Actionable rules

• Run a 4-layer compression pipeline before every LLM call.
• Cap last-N observations (default N=5) — keep actions/thoughts, blank stdout.
• Sub-agents are context firewalls, not just parallelism.
• Skills load in two phases — frontmatter at startup, body on invocation.
• The system prompt is a list of small sections assembled at request time.
• AGENTS.md is a table of contents under 100 lines, not an encyclopedia.
• Instrument estimated vs authoritative token counts and alert on drift.

💾 Part 5 — Memory (Long-Term Knowledge)

Memory is where most agents either over-engineer (vector DB on day one) or under-engineer (everything fits in context, surely?).

5.1 📁 Why files, not a database (Claude Code / Nanobot)

• Transparency — users open .md files and see exactly what the agent remembers. Trust through observability, not capability.
• Modification time is a built-in epistemological signal: "when was this observation recorded?"
• Git is free — diffable, recoverable, human-editable, auditable.
• Zero infrastructure — no schema migrations, no indexes, no backups.

Add embeddings + a vector store later, when you have measured the failure modes that demand them. Most agents don't need them.

5.2 🗂️ The file layout

~/.claude/projects/<sanitized-git-root>/memory/
  MEMORY.md                        # always loaded; index only; ≤200 lines
  user_role.md                     # one memory per file
  feedback_testing.md
  project_migration_q2.md
  team/                            # shared via symlink
  logs/YYYY/MM/YYYY-MM-DD.md       # append-only daily logs

5.3 🏷️ The four-type taxonomy

Derivability test: if git log / git blame / the code itself can answer it, don't memorize it. No code patterns, no architecture, no debug fix recipes.

5.4 📝 Frontmatter contract

---
name: <title>
description: <one-line summary used by recall LLM>
type: user | feedback | project | reference
---

<body — for feedback/project, structure as: rule → **Why:** → **How to apply:**>

The description field carries the most weight — it's the LLM-recall index.

5.5 🔍 Two-tier retrieval

• Tier 1 (always loaded): MEMORY.md index (~3,000 tokens for ~150 entries). Lines after 200 are truncated.
• Tier 2 (on-demand): an async side-query LLM gets the manifest (type, name, date, description), the user's current query, and recent tool history. Returns up to 5 filenames. Validated against the file list to catch hallucination.

This trades a few hundred ms of latency for semantic precision keyword-matching cannot achieve — especially for negation (do NOT use mocks).

5.6 🏛️ The 3-tier memory model (GoClaw, when you grow up)

Once a file system isn't enough, GoClaw's tiering is the right next step:

L0 — Working Memory                L1 — Episodic                L2 — Semantic
┌────────────────────┐             ┌──────────────────┐         ┌──────────────────┐
│ Current session    │             │ Session summaries│         │ Knowledge Graph  │
│ messages           │   ───────►  │ + L0 abstracts   │ ──────► │ entities +       │
│ Threshold-based    │             │ (~50 tokens ea.) │         │ relations        │
│ compaction         │             │ + embeddings     │         │ + temporal       │
│                    │             │ 90-day retention │         │   validity       │
└────────────────────┘             └──────────────────┘         └──────────────────┘
        ▲                                  ▲                            ▲
   auto-inject                        memory_search                memory_expand
   (free, no tool call)               (top-K)                      (full doc)

Hybrid search formula:

combined_score = vector_score * 0.7 + fts_score * 0.3

With BM25 / tsvector for FTS and pgvector for embeddings. Per-user 1.2× boost. Dedup: per-user wins over global.

5.7 🔄 Event-driven consolidation

run.completed
       │
       ▼
EpisodicWorker → extract summary + abstract via LLM
       │
       ▼ episodic.created
SemanticWorker → extract entities/relations into KG
       │
       ▼ entity.upserted
DedupWorker → embedding-similarity merge

(separately, debounced 10m)
DreamingWorker → batch unpromoted summaries scored by:
                 0.30 * frequency + 0.35 * relevance +
                 0.20 * recency  + 0.15 * freshness
              → LLM synthesis → write to long-term memory

The dream phase (Nanobot / GoClaw) is the agent literally writing notes to itself, then git commit-ing them. Auditable, diffable, recoverable.

5.8 🕰️ Staleness policy: annotate, don't expire

Don't expire memories. Annotate with age. Today/yesterday → no caveat. Older → human-readable warning ("This memory is 47 days old — code claims may be outdated"). Models reason better about "47 days ago" than ISO timestamps.

Per-line age suffixes (← 30d) computed from git blame let the LLM naturally deprioritize stale entries.

✅ Actionable rules

• Start with files + git. Add a vector store only when measured failures demand it.
• Always-loaded MEMORY.md index → on-demand LLM recall side-query for tier 2.
• Four memory types: user, feedback, project, reference. Derivability test rules out the rest.
• Hybrid search: 0.7×vector + 0.3×FTS. Per-user boost. Dedup.
• Consolidation is event-driven (workers subscribed to run.completed).
• Annotate stale entries; don't expire them.

⚡ Part 6 — Concurrency & Multi-Agent Patterns

6.1 🔒 Per-session serial, cross-session concurrent (Nanobot)

The simplest correct concurrency model for multi-tenant chat agents:

• Each session gets a lock.
• Within a session: strictly serial (no race on history).
• Across sessions: concurrent (one user's slow tool call doesn't block another's).

No threads, no actors, no Redis.

6.2 📬 The pending queue: mid-turn message injection

Users send follow-ups while the agent is still working. Don't ignore them; queue them.

# Top-level dispatcher
async def run(self):
    while running:
        msg = await bus.consume_inbound(timeout=1.0)
        key = effective_session_key(msg)
        if key in pending_queues:
            # session is mid-turn → enqueue, don't start a 2nd task
            pending_queues[key].put_nowait(msg)
            continue
        task = asyncio.create_task(self._dispatch(msg))

When the current turn ends, drain the queue back into inbound for the next turn. Per-session lock + pending queue is the entire multi-turn concurrency model. You will be tempted to use a state machine. Don't.

6.3 🕹️ Steering: mid-loop user correction (PicoClaw)

"The user can correct the agent at any moment. Make that a first-class concern."

Per-session FIFO queue polled at four checkpoints:

1. Loop initialization
2. After each tool completes
3. After each non-tool LLM response
4. Before turn finalization

If a queued message exists at any of those points:

• Remaining tool calls in the current LLM response are skipped, each receiving the synthetic result "Skipped due to queued user message." so the model still understands what did/didn't run.
• The queued message is appended as a new user turn.
• The loop re-enters the LLM stage.

Why this matters:

• Side-effect safety: a user yelling "don't send that email" actually stops the email.
• Compute savings: a planned batch of three 4s tool calls = 12s avoided.
• Model awareness: skipping is announced via tool-result so the model adapts.

6.4 🤖 Sub-agents: the recursive primitive

Single-agent capability has a hard ceiling. Spawn child agents that are the same loop with isolated state.

Hard limits to enforce:

Recursive fork prevention (defense in depth):

1. Primary: tag child's context with a querySource flag. AgentTool checks this before allowing fork.
2. Fallback: scan message history for the boilerplate XML tag if the flag was lost in transit.

6.5 🍴 Fork agents: cache-driven subprocess design

The point of a fork is byte-identical request prefix to the parent, so children pay ~10% input-token cost (90%+ savings on prompt cache).

Three mechanisms make this work:

1. System prompt threading — pass parent's already-rendered bytes via override.systemPrompt. Don't regenerate; feature flags or session date may have changed.
2. Exact tool passthrough — useExactTools: true. No filtering, no reordering, no re-serialization.
3. Placeholder tool results — clone the parent's last assistant message. For each tool_use, insert a constant placeholder string ("Fork started -- processing in background"). Same string for every child → same bytes.

Result: [...shared_history, assistant(all_tool_uses), user(placeholders..., directive)]. Only the final directive differs. With a 48,500-token shared prefix and 5 children, savings exceed 90% on input tokens for children 2–5.

6.6 🗺️ Three coordination patterns

Pick the right shape for the work:

📋 Coordinator mode rules

• Three tools only: Agent (spawn), SendMessage (talk), TaskStop (kill). By design.
• "The coordinator's job is to think, plan, decompose, and synthesize. Workers do the work."
• Critical principle: never delegate understanding. Coordinators must give workers exact file paths, exact line numbers, exact change descriptions — not "based on the research, fix the bug."

Workflow phases: research → synthesis (coordinator!) → implementation → verification.

🐝 Swarm rules (when peers collaborate)

• File-based mailboxes for inter-peer messages.
• Messages delivered between tool rounds, never mid-execution. No race conditions.
• Three interruption levels: abort current work / shutdown request / kill.
• Hard cap on teammate state. A real production incident reached 36.8 GB across 292 agents. Cap message history; budget unbounded fan-out from day one.

6.7 📋 Teams: SQL-claimed task boards (GoClaw)

When agents collaborate on a structured workflow, give them a board:

-- Atomic, race-safe claim — no distributed lock needed
UPDATE team_tasks
SET status = 'in_progress', owner_agent_id = $1
WHERE id = $2 AND status = 'pending' AND owner_agent_id IS NULL
-- 1 row updated = claimed; 0 rows = someone else got it

Task states: pending / in_progress / in_review / completed / failed / cancelled / blocked / stale. Dependencies via blocked_by UUID[] array; completing a task auto-unblocks dependents.

6.8 🏗️ SubTurn: hierarchical sub-agents (PicoClaw)

Why context derives from context.Background(), not the parent's ctx: an independent timeout on a child shouldn't surprise it when the parent finishes early. Cascading cancellation is opt-in.

✅ Actionable rules

• Per-session lock + pending queue is the entire multi-turn concurrency model.
• Steering: poll a per-session FIFO at 4 checkpoints; skip remaining tools with explicit "Skipped" results.
• Sub-agent limits: depth 1, ≤5 children, ≤8 concurrent. Hard-enforce.
• Forks share bytes with the parent for 90%+ prompt cache savings.
• Coordinator: 3 tools only. Workers get exact paths, never "based on research."
• Cap teammate state. Bound every fan-out queue.
• Atomic SQL claim beats distributed locks.

🔧 Part 7 — Reliability: Error Recovery, Stuck Detection, Autosubmit

7.1 🪜 The error recovery ladder (not a fallback)

Order matters. From least to most aggressive:

Each layer catches a different failure mode. Together they make the agent feel "self-correcting" even though the LM is just being asked to try again with more context.

7.2 🚀 The autosubmit pattern (SWE-agent's killer move)

Every error path ends in autosubmit, not crash. A 30-step trajectory that hits a cost limit at step 31 has probably made some real progress. A git diff of the work-in-progress is a partial patch that may pass some tests. Throwing it away for an exception is a bug; autosubmitting is the feature.

def handle_error_with_autosubmission(self, exit_status, message):
    try:
        # Run submit one more time, capture whatever git diff exists
        observation = self._env.communicate("submit", timeout=10)
        submission = self.extract_submission(observation)
    except Exception:
        submission = ""    # even submit failed, ship empty
    return StepOutput(
        done=True,
        exit_status=exit_status,    # "exit_cost", "exit_context", etc.
        submission=submission,
        observation=message,
    )

Failure modes turn into degraded successes. Cost overrun, context overflow, total timeout, consecutive errors — all autosubmit, none crash.

7.3 🛡️ Three guardrails that compound (SWE-agent)

Each layer catches a different failure mode. Together they make the agent feel self-correcting:

• bash -n syntax check before execution. Malformed commands never run; the agent sees the error and retries.
• flake8 + auto-revert on edits. Broken code is never persisted. Side-by-side before/after diff in the error message.
• Format-error requery. Up to 3 re-prompts when parse / blocklist / bash-syntax fails.

7.4 🚨 Stuck detection (OpenHands)

Without this, agents burn money in loops. Run a StuckDetector on the event log every step. It flags five patterns:

Comparison is semantic, not object identity: actions matched by tool name + content (timestamps and metrics ignored). When stuck, the agent transitions to ERROR or emits a LoopRecoveryAction.

This 100-LOC detector saves more money than any other optimization.

7.5 ⚖️ The SWE-agent caveat: don't add semantic guardrails unless 100% precision

SWE-agent has no semantic stuck detection. The team tried and abandoned them: false positives were too high.

Their stance: don't add a guardrail unless its false-positive rate is low. SWE-agent's existing guardrails (cost, syntax check, lint+revert) are all 100%-precision: they only fire when something is definitely wrong.

Reconciling with OpenHands: stuck detection works when the false-positive cost is low (you abort/notify, not crash). Tune the thresholds for your tolerance.

7.6 ✅ Verification is the difference between hallucinated "done" and real done

The single most important prompting lesson from OpenHands' CodeActAgent:

Make your agent re-run the test suite as the last action before finish.

The four-phase methodology baked into the system prompt:

1. Exploration — read the repo, find relevant files, understand the surface area before doing anything.
2. Analysis — form a hypothesis about what to change and why. The ThinkTool exists for this slot.
3. Implementation — make the smallest change that addresses the analysis.
4. Verification — re-run the tests, lints, build. Only call finish when verification passes.

Without verification, the agent declares victory on broken code. With it, "ran for 30 minutes and didn't break anything" becomes a real story.

7.7 🔐 Confirmation policy + risk analyzer (OpenHands)

Two layers:

1. Risk analyzer. Every Action gets a SecurityRisk ∈ {LOW, MEDIUM, HIGH, UNKNOWN} score. The default LLMSecurityAnalyzer adds a security_risk field to every tool's JSON schema, so the LLM scores its own action inline — no extra call.
2. Confirmation policy. AlwaysConfirm, NeverConfirm, or ConfirmRisky(threshold=HIGH). With ConfirmRisky, low-risk actions auto-execute; risky ones pause until approved.

Headless mode hard-disables confirmation (it's NeverConfirm always). That means headless mode's blast radius is whatever the workspace allows — which is exactly why headless mode wants Docker.

7.8 💰 Cost ceilings: the day-one minimum

Don't ship a headless agent without all of these.

7.9 💾 Session checkpoint on every iteration

Cheap to write, makes /stop and crashes feel free instead of catastrophic. Before each turn the loop saves a runtime checkpoint of intermediate tool messages. On /stop or crash the next turn restores them so a half-finished tool sequence isn't lost.

✅ Actionable rules

• The error recovery is a ladder, not a fallback. Each layer is testable.
• Autosubmit, never crash. Every error path produces a degraded success.
• Three compounding guardrails: bash -n, lint+revert, format requery.
• Add stuck detection — but tune for low-false-positive cost.
• Verification is the last action before "done." No exceptions.
• Day-one ceilings: max iterations, max retries, max cost, max wallclock.
• Checkpoint every iteration. Restore on resume.

🔒 Part 8 — Security: Defense-in-Depth

Each layer is independent — even if one is bypassed, the others still protect.

Layer 1 — 🌐 Transport

• CORS allow-list validation
• WebSocket message size limit: 512 KB
• HTTP body limit: MaxBytesReader 1 MB
• Timing-safe token comparison (crypto/subtle.ConstantTimeCompare)
• Rate limiting (token bucket per user / per IP)
• Ping/pong every 30s; read deadline 60s; write deadline 10s

Layer 2 — 🔍 Input Validation (InputGuard)

Six regex patterns scan every user message. Catches:

Four action modes: off / log / warn (default) / block.

Layer 3 — ⚙️ Tool Execution

• Shell deny groups — 15 classes denied by default: destructive_ops, data_exfiltration, reverse_shell, code_injection, privilege_escalation, dangerous_paths, env_injection, container_escape, crypto_mining, filter_bypass, network_recon, package_install, persistence, process_control, env_dump. Live-reloadable via pub/sub.
• Path traversal prevention — resolvePath() runs filepath.Clean() and verifies the result starts with the workspace prefix on every filesystem op.
• SSRF guards — block 127.0.0.1 / localhost / RFC1918 ranges for provider base URLs and web_fetch.
• Credentialed CLI gate — when calling registered binaries (gh, gcloud, aws, kubectl, terraform), inject encrypted env vars directly into the child process (no shell), unwrap sh -c wrappers up to depth 3 to prevent bypass, and fail-closed on DB error.
• Domain allow/block — web_fetch honors per-tenant allow/block lists.

For Bash, parse the command via a real bash AST parser, split on && || ; |, classify each subcommand. If the parser fails, return fail-safe behavior — assume any command it can't parse is unsafe.

Layer 4 — 🧹 Output Sanitization

• Credential scrubber — static regex patterns (OpenAI sk-, Anthropic sk-ant-, GitHub ghp_, AWS AKIA, generic 64-char hex) + dynamic registry of runtime values. Replaces with [REDACTED]. Always-on.
• Output sanitizer — 7 steps applied to LLM output before delivery:
  1. Strip garbled tool XML (<tool_call>, etc. from broken models)
  2. Strip downgraded text-format tool calls ([Tool Call: ...])
  3. Strip thinking tags (<think>, <thinking>, <antThinking>)
  4. Strip final wrapper tags (preserve inner content)
  5. Strip echoed [System Message] blocks
  6. Collapse consecutive duplicate paragraphs (model stuttering)
  7. Strip leading blank lines

Layer 5 — 🔒 Isolation

• Per-user workspace — base + "/" + sanitize(userID), injected via WithToolWorkspace(ctx).
• Docker / bwrap sandbox — read-only root, dropped capabilities, scoped per-session.
• Subagent depth limit — max depth 1, max children 5/parent, max concurrent 8 system-wide.
• MCP servers spawned in isolated processes. Treat MCP servers as untrusted child processes. One buggy server must not crash the agent.

🚧 Hard security boundaries to set early

🔑 The secret registry (OpenHands)

A separate vault that:

• Stores secrets per-session, late-bound (resolved only at exec time).
• Masks them in stdout/stderr (<secret-hidden>).
• Encrypts at rest, supports rotation, supports callable resolvers.
• The shell tool scans commands for known secret keys, exports them as env vars, and replaces matches in output.

✅ Actionable rules

• Five layers, all independent. Don't pick one and stop.
• 15 shell deny groups by default. Live-reloadable.
• Every filesystem op runs through resolvePath(). Path traversal dies.
• Output sanitizer is a 7-step pipeline. Always-on.
• Credential scrubber: static + dynamic. Mask in real-time.
• MCP servers run in isolated processes. MCP skills never execute shell.
• .security.yml separate file, separate perms. AES-256-GCM at rest. SHA-256 + constant-time for auth.

🏢 Part 9 — Multi-Tenancy from Day One

This is the single most consequential design decision — and the one most projects skip until it's painful.

📜 Three rules, never broken

1. Every isolatable table has tenant_id NOT NULL. 40+ tables in GoClaw enforce this.
2. Every query includes WHERE tenant_id = $N. No exceptions. Fail-closed.
3. Tenant flows through context.Context. Resolved at the gateway, propagated everywhere, never taken from client headers (which can be spoofed).

🚪 Tenant resolution at the gateway

⚙️ Per-tenant overrides

Each tenant gets its own:

• LLM provider configs and API keys
• Tool settings (web_search providers, TTS voice, etc.)
• Skills enabled/disabled
• MCP servers + per-user credentials
• Channel instances

🔐 Storage hardening

• API keys: SHA-256 at rest, constant-time compare for validation.
• Provider/MCP/custom-tool secrets: AES-256-GCM with aes-gcm: prefix + 12-byte nonce + ciphertext + tag, base64'd.
• Master scope guard: writes to global tables (builtin_tools, config.*) require IsMasterScope(ctx) — otherwise tenant admin only.

🪪 Identity propagation

You don't authenticate end-users. The upstream service (your SaaS backend, your auth proxy) provides user_id, opaque, max 255 chars. Convention for multi-tenant: tenant.{tenantId}.user.{userId}.

✅ Actionable rule: Retrofitting multi-tenancy is one of the most painful migrations in software. Make tenant_id a column on day one, even if you only have one tenant. The migration cost from "single-tenant + tenant_id column" to "multi-tenant" is hours. The migration cost from "single-tenant" to "multi-tenant" is months.

🚀 Part 10 — Performance & Efficiency

10.1 🏛️ Prompt caching is architecture, not optimization

Every design decision either preserves cache hits or busts them. Anthropic's prompt cache gives ~90% input-token discount on cached prefixes. With long conversations, this dominates economics.

Cache scopes:

The dynamic boundary — a literal marker in your system prompt:

• Above (cacheScope: global): identity, system rules, task guidance, tool usage, tone.
• Below (per-session): session guidance, project memory, env info, language, MCP instructions, output style.

Rule: Every runtime if above the boundary doubles the cache key space. 3 conditionals = 8 prefixes. 5 = 32. Compile-time feature flags are fine; runtime checks must live below the boundary.

Global scope is disabled when MCP tools are present — user-specific tool definitions would fragment global cache into millions of unique prefixes.

10.2 🔒 Sticky latches

Five session-scoped boolean flags (null | true) that, once set, can't unset for the rest of the session. They control beta/feature headers. Reason: mid-session toggles change the server-side cache key — flipping a flag would bust 50–70K tokens of cached context.

type Latch = boolean | null   // null = "not yet evaluated"

function shouldSendBetaHeader(active: boolean): boolean {
  const latched = getAfkLatch()
  if (latched === true) return true
  if (active) { setAfkLatch(true); return true }
  return false
}

Use Once(value) semantics for any cache-influencing config.

10.3 📦 Output token slot reservation

Production p99 output ≈ 4,911 tokens. Default SDK reservation = 32K–64K. Over-reservation = 8–16×.

Strategy: cap default max_tokens at 8K. On the rare truncation (<1% of requests), retry with 64K. Recovers 12–28% of context window for free.

10.4 🌊 Streaming: skip the SDK helper

The SDK's BetaMessageStream calls partialParse() on every input_json_delta — repeatedly re-parsing growing JSON from scratch (O(n²)). Use raw stream events and accumulate tool-input strings yourself.

10.5 🐕 Watchdog and fallback

• Idle watchdog: setTimeout(90s) reset on every chunk. At 45s, warn. At 90s, abort and retry non-streaming.
• Non-streaming fallback activates when streaming dies mid-response (network, stall, truncation, proxies returning 200 with non-SSE bodies).
• Disable fallback when streaming tool execution is active — duplicate tool runs would corrupt state.

10.6 🔌 API preconnection at boot

Fire a HEAD request to your LLM API during init. TCP+TLS handshake (100–200 ms) overlaps with setup. Connection is warm by the time the user submits.

10.7 💸 Cheap-first model routing (PicoClaw)

A rule-based classifier scores each turn 0..1 on five language-agnostic features:

With threshold 0.35, trivial chat stays on a cheap "light" model (Gemini Flash, Haiku); code/attachments/tool-active goes to the heavy model. This alone cuts API spend dramatically for chatty workloads.

Each agent has both Candidates and LightCandidates — primary and cheap fallback chains. Routing only picks the chain; fallback logic inside the chain is generic.

10.8 🪶 Lean runtime tricks (PicoClaw, Go)

• Static linking: no shared-library footprint.
• -ldflags="-s -w" strips symbol table and DWARF info (~30% size reduction).
• -trimpath removes file system paths from binaries.
• Bounded queues everywhere — turns "memory bug" into "rejected request" you can monitor and tune.
• Lazy initialization — channels, hooks, skill registries init only when enabled.
• membench regression gate in CI — peak RSS measured per PR.

10.9 📈 Concrete cost economics

Two anchor points:

• CodeActAgent v1.8 on Claude 3.5 Sonnet: 26% on SWE-Bench Lite at $1.10 per instance.
• OpenHands V1 default condenser: cuts API spend by ~2× on long sessions.

Order-of-magnitude expectations:

• Trivial task (few file edits, no tests): $0.05–$0.30 per run on a frontier model.
• SWE-Bench-style real fix: $0.50–$3 per task.
• Multi-hour autonomous run: $5–$30, easily more without a condenser.

10.10 🎯 Bootstrap targets

For interactive agents:

• Boot time < 300 ms.
• First token streamed < 1 second.
• Fast-path dispatch: --version / --help → dynamic-import only that handler, exit. Don't load React, telemetry, MCP.
• 50+ profiling checkpoints sampled at 100% of internal users / 0.5% of external. Without instrumentation you can't tell what to optimize.

✅ Actionable rules

• Treat the prompt cache as architecture. Static-then-dynamic boundary, sticky latches, byte-identical fork prefixes.
• Cap max_tokens at 8K, escalate on truncation. Recovers 12–28% of context.
• Cheap-first routing: 5-feature classifier, threshold 0.35, light/heavy model chains.
• Bounded queues, lazy init, regression gates.
• Boot < 300ms, first token < 1s. Profile from day one.

🔌 Part 11 — Provider Abstraction & Resilience

11.1 🔌 The Provider interface

Tiny. Everything that's hard about LLMs lives inside this seam.

type Provider interface {
    Name() string
    DefaultModel() string
    Chat(ctx, req) (Response, error)
    ChatStream(ctx, req, onChunk) (Response, error)
}

Every backend — Anthropic native HTTP+SSE, OpenAI-compatible (Groq, DeepSeek, Gemini, Mistral, OpenRouter, vLLM, LM Studio, Ollama all speak the same wire), Claude CLI subprocess, Bedrock, Azure, Vertex AI, ACP JSON-RPC, DashScope — implements this interface. The agent loop never knows which one it's talking to.

Use LiteLLM (Python) or equivalent — 100+ providers for free.

11.2 🧱 The reliability stack (the part most projects miss)

When a provider call fails, the wrapper consults:

1. Error classifier — Auth? Rate-limit? Network blip? 5xx? 9 canonical reasons.
2. Cooldown — if Auth/Quota, mark this provider unavailable for N minutes.
3. Rate limiter — token bucket to keep us under contractual TPM/RPM.
4. Fallback — try next candidate in chain (heavy → light, or primary → secondary key).
5. Retry — exponential backoff with jitter; honors Retry-After; retries 5xx + network only (not 4xx).
6. Cache middleware — caches identical prompts within a TTL.
7. Service tier — picks priority / flex / auto per request.

The agent never sees this — it sees one logical "send" that either returns a response or gives up after the chain is exhausted.

11.3 🔧 Wire-format quirks live in adapters, not the loop

Examples:

• Anthropic uses x-api-key; OpenAI-compat uses Bearer; Codex uses OAuth + token refresh.
• Claude CLI is a subprocess speaking stdio; ACP is JSON-RPC 2.0 over stdio.
• DashScope wraps Qwen with custom thinking-budget mapping.
• Some providers return "200 with error body" (Slack-style). Normalize at the adapter.

Force every quirk through one interface and you keep the agent loop boringly simple.

11.4 🧠 Reasoning/thinking blocks are first-class

Anthropic extended thinking → ThinkingBlocks; OpenAI o-series reasoning → ReasoningItemModel. Persist these on ActionEvent / MessageEvent so they're replayable and can be fed back to the model on the next turn — required to maintain reasoning continuity for o-series and Sonnet thinking-mode.

11.5 🔄 Non-native function calling

For models without native tool support, serialize tools into a structured prompt and parse responses with regex. Lets even small open-source models drive the agent loop. Pattern: detect, then either call native function-calling or fall back to prompt-and-parse — same agent code path.

11.6 🗺️ Routers all the way down

RouterLLM — abstract base; subclass with select_llm(messages) → str. Real example: route image-containing messages to a vision model and text-only to a cheap model. Composes recursively (a router can route to a router), so you can build cost-optimization trees.

11.7 📋 Rich LLMResponse error fields (Nanobot's hard-won detail)

class LLMResponse:
    content: str | None
    tool_calls: list[ToolCallRequest]
    finish_reason: str
    usage: dict[str, int]
    reasoning_content: str | None
    thinking_blocks: list[dict] | None
    error_status_code: int | None
    error_should_retry: bool | None
    retry_after: float | None

Capture rich error metadata in the response object itself. The retry layer becomes a one-page, provider-agnostic policy instead of a forest of except clauses.

✅ Actionable rules

• Tiny Provider interface. Every quirk lives behind it.
• 7-layer reliability stack: classify → cooldown → rate-limit → fallback → retry → cache → tier.
• Persist reasoning/thinking blocks for continuity.
• Pattern-detect native vs non-native function calling — same agent code path.
• Rich error fields on the response object. One retry policy, not N.

📡 Part 12 — Channels & Integration Surface

12.1 🔌 Channels as pluggable adapters

Each external messaging platform is an adapter that:

• Translates platform events to a unified InboundMessage.
• Translates unified OutboundMessage to platform replies.

Two functions: Listen() and Send(). Keep the agent loop ignorant of platform specifics.

12.2 🏷️ First-class fields beat metadata bags

Don't bury chatId, senderId, messageId inside generic metadata maps. Hoist them to typed fields:

type InboundMessage struct {
    Peer       Peer        // platform + chat + topic
    MessageID  string
    Sender     SenderInfo  // canonical identity ("telegram:42")
    Body       string
    Media      []MediaRef
    ReceivedAt time.Time
}

This is the contract that session allocation, routing, and hooks rely on. Put it in your design from day one — retrofitting is painful.

12.3 🎭 Capability-based polymorphism (PicoClaw)

Every platform sub-package embeds BaseChannel and implements the minimum interface. Optional capabilities are separate interfaces:

type MediaSender         interface { SendMedia(...) error }
type TypingCapable       interface { ShowTyping(...) error }
type ReactionCapable     interface { React(...) error }
type PlaceholderCapable  interface { SendPlaceholder(...) (id string, err error) }
type MessageEditor       interface { Edit(...) error }
type WebhookHandler      interface { HandleWebhook(...) }
type HealthChecker       interface { Check(ctx) error }

The manager probes channels with if c, ok := ch.(MediaSender); ok { ... }. Adding VoiceCapable to one platform doesn't change anyone else.

12.4 🔄 The manager owns retries, not the channel

Centralize in the manager:

• Worker queue with rate limit per channel.
• Outbound message splitting — long replies broken at sentence/word boundaries below the platform's per-message limit.
• Retries with backoff on transient errors classified by sentinel error types.
• Typing/reaction indicators as transparent decorations of long turns.

Platforms only know how to send a single chunk. Everything fancy happens above them. 30 channels × N retry strategies = 0 duplication.

12.5 📌 Self-registration via blank imports

// channels/telegram/telegram.go
func init() {
    channels.Register("telegram", New)
}

// main.go
import _ "yourapp/channels/telegram"  // side-effect registration

The main binary just imports for side effects. The channel becomes available in the registry. No registry plumbing.

12.6 🚌 The bus is the universal IPC (Nanobot)

class MessageBus:
    inbound:  asyncio.Queue[InboundMessage]
    outbound: asyncio.Queue[OutboundMessage]

That's the entire seam. Every channel just needs to (a) translate platform events into InboundMessage and publish_inbound, and (b) listen to consume_outbound for messages addressed to its channel name and translate back.

Side effect: cron jobs, sub-agent results, heartbeat triggers, and inter-agent messages all use the same bus — they're just synthetic InboundMessage events with channel="system". Uniformity = small code.

12.7 🔑 Session keys: structured identity

agent:{agentId}:{channel}:direct:{peerId}     ← DM
agent:{agentId}:{channel}:group:{groupId}     ← Group
agent:{agentId}:subagent:{label}              ← Subagent
agent:{agentId}:cron:{jobId}:run:{runId}      ← Cron run

Or content-addressed: sk_v1_<sha256> (PicoClaw) — stable, opaque, source of truth, with legacy aliases resolved transparently.

12.8 🤝 Pairing flow (DM access policies)

Three policies:

allowFrom empty = deny all. The right default for personal agents.

✅ Actionable rules

• Channels: two methods (Listen, Send). Everything else lives in the manager.
• First-class typed fields on InboundMessage. Never metadata bags.
• Capability interfaces beat optional methods on a god-interface.
• Manager owns retries, splitting, rate-limit. Channels send one chunk.
• The bus carries inbound, outbound, system events. Uniformity = small code.
• DM access: pairing / allowlist / open. Empty = deny all.

📊 Part 13 — Observability & Evaluation

13.1 🔎 Trace everything

Three span types: agent, llm_call, tool_call. Wrap every LLM call in a span. Wrap every tool call in a span. Trace tree mirrors the run shape.

13.2 💰 Cost tracking from step 1

Every API response runs through a cost accumulator:

• Per-model usage in bootstrap state.
• Reports to OpenTelemetry.
• Recursively processes nested model calls (sub-agents, recall queries).
• Persists to project config on process exit.
• Restores on next session if persisted session ID matches.

Histograms use reservoir sampling (Algorithm R) with 1,024 entries to compute p50/p95/p99. Averages hide tail latency, and tail latency is what users feel.

Even in v0, instrument cost and latency. You cannot decide what to optimize from feel.

13.3 ⏮️ Replayable trajectories

Every step() writes a .traj JSON file containing history, model output, observations, costs. SWE-agent's run-replay re-executes any old run. The append-only event log is the source of truth.

Worth it just for debugging. When the agent does something weird at minute 47, you can rewind to any event and try a different model or prompt.

13.4 🧪 Eval taxonomies

Three layers of evaluation:

13.5 📝 Trace grading

Grade agent traces directly — especially helpful for multi-step tasks where final output alone doesn't reveal process quality. Use a separate LLM as a judge with a clear rubric.

13.6 🎯 Skill-level evals

Measure whether a specific skill actually helps using:

• Bounded tasks — reproducible inputs.
• Deterministic verifiers — automated pass/fail.
• No-skill baseline — does the skill move the needle?
• Trace review — human-spotcheck the failures.

13.7 📡 Infrastructure noise

Runtime configuration can move coding benchmark scores by more than many leaderboard gaps.

Infrastructure choices may matter more than model intelligence. The same model with a better harness, better tools, better verification, lands a higher score.

13.8 📒 Activity log for every admin action

Every admin write to global tables (settings, permissions, tool config) appends to an audit log: { tenant_id, actor_id, action, target, timestamp, ip }. Cheap to write, invaluable when "who changed X?" comes up.

✅ Actionable rules

• Spans on every LLM and tool call. Trace tree mirrors the run.
• Cost + reservoir-sampled latency from day one.
• Append-only event log = replayable trajectories.
• Eval at three layers: single-step, full-run, multi-turn. Trace-grade.
• Skill-level evals with no-skill baselines. If it doesn't move the needle, drop it.
• Audit log for every admin action.

🗺️ Part 14 — The Build-Your-Own Roadmap

A pragmatic order to implement everything above. Each step compiles and runs on its own.

🌱 Milestone 0 — Foundation (1–2 days)

• Pick the language. Go for small/portable; Python for ML/research/speed.
• Pick the DB: PostgreSQL + pgvector if you ever want vector search.
• Skeleton: cmd/, internal/, pkg/, migrations/, docs/, Makefile, docker-compose.yml.
• Define the Provider interface (4 methods).
• Implement one provider — start with OpenAI-compatible (covers Groq, DeepSeek, Together for free).
• cmd/serve loads config, makes one HTTP request, prints the response.

🔁 Milestone 1 — Minimum Viable Agent Loop (1 week)

• Define Tool interface: name, description, schema, execute(ctx, args).
• Implement 3 tools: read_file, write_file, list_files — workspace-scoped, with resolvePath() traversal guard.
• Build the loop: for i := 0; i < 20; i++ { think; if no tools break; act; observe }.
• Persist sessions: SessionStore interface + in-memory implementation.
• Emit events via callback. Three only: run.started, tool.call, run.completed.
• HTTP endpoint /v1/chat/completions (OpenAI-compatible). One agent. No streaming yet.

You now have an LLM that can read/write files in a workspace.

🧩 Milestone 2 — System Prompt Architecture (3–4 days)

• Bootstrap files: agent_context_files (agent-level) + user_context_files (per-user). 6 known files: SOUL, IDENTITY, AGENTS, TOOLS, BOOTSTRAP, USER.
• ContextFileInterceptor — when a tool reads/writes a known name, route to DB instead of disk.
• System prompt builder — assemble from sections. Persona early, persona reminder late.
• Two modes: PromptFull and PromptMinimal.
• Per-user file seeding on first chat.

🏢 Milestone 3 — Multi-Tenancy from the Start (3–4 days)

• tenants and api_keys tables. UUID v7 PKs.
• tenant_id NOT NULL on every table that holds tenant data.
• WithTenantID(ctx) / TenantIDFromContext(ctx) helpers.
• Resolve API key → SHA-256 lookup → set tenant on ctx at the gateway.
• Update every store query to add WHERE tenant_id = $N. Audit the diff.
• Master tenant for legacy/single-user data; master scope guard for global writes.

🔧 Milestone 4 — Pipeline Refactor (1 week)

Once your loop has > 3 conditional branches, split it:

• Define Stage interface, StageResult enum, RunState struct.
• Implement ContextStage, ThinkStage, ToolStage, ObserveStage, CheckpointStage, FinalizeStage. Add PruneStage later.
• Pipeline.Run orchestrates: setup → iteration loop → finalize.
• Feature flag (pipeline_enabled) so V2 (monolithic) and V3 (pipeline) coexist during migration.

💾 Milestone 5 — Memory & Search (1–2 weeks)

• memory_documents + memory_chunks tables. tsvector (FTS) + vector(1536) columns.
• MemoryInterceptor — auto-chunks + embeds on .md writes inside memory/*.
• Hybrid search: 0.7 * vector + 0.3 * fts. Per-user 1.2× boost. Dedup.
• memory_search and memory_get tools.
• Later: episodic_summaries + EpisodicWorker subscribed to run.completed.
• Later: kg_entities + kg_relations with temporal validity for L2.

🛡️ Milestone 6 — Tool Registry Hardening (1 week)

• Funnel every tool call through Registry.ExecuteWithContext.
• Token-bucket rate limiting per session key (defaults: 60/min, burst 5).
• Credential scrubber — start with 5–10 high-value patterns.
• Policy engine: profiles (full / coding / messaging / minimal), groups, allow/deny lists.
• Shell deny groups (start with: destructive_ops, reverse_shell, dangerous_paths, package_install).
• Capability metadata on every tool.

📡 Milestone 7 — Channels (per channel, ~2 days each)

• Define Channel interface: Listen(ctx, onMessage), Send(ctx, OutboundMessage) error.
• Telegram first (simplest, long-polling).
• channel_instances table with tenant_id baked in.
• Outbound dispatcher routes by channel_instance_id.
• Pairing flow: 8-char code, 60-min TTL.
• Then: Discord, Slack, WhatsApp, Feishu, Zalo.

📊 Milestone 8 — Observability (3–4 days)

• traces and spans tables. Three span types.
• Wrap every LLM call and tool call in a span.
• BatchCreateSpans in batches of 100; on failure, retry individually.
• Verbose mode (TRACE_VERBOSE=1) for full input/output, truncated at 50 KB.
• Optional: OpenTelemetry exporter.

🔄 Milestone 9 — Resilience (3–4 days)

• Wrap providers with retry middleware.
• Per-model cooldown.
• Failover chain.
• Mid-loop compaction at 75%; post-run at 50 messages or 75%.
• Per-session TryLock for compaction goroutine.
• Stuck detector (5 patterns, semantic comparison).
• Autosubmit on every fatal error path.

🤖 Milestone 10 — Multi-Agent (1–2 weeks)

• subagent table. Limits: depth 1, max 5 children, max 8 concurrent.
• spawn tool (async return), delegate tool (sync with timeout).
• agent_links table for delegation eligibility.
• When ready: teams, agent_team_members, team_tasks, team_messages.
• Atomic task claim: UPDATE … WHERE status = 'pending' AND owner_agent_id IS NULL.

🔒 Milestone 11 — Production Hardening (ongoing)

• Add the remaining 4 security layers (input guard, output sanitizer, isolation).
• AES-256-GCM encryption for all at-rest secrets. aes-gcm: prefix convention.
• API keys: 16 random bytes, SHA-256 hash, constant-time compare.
• Activity log for every admin action.
• Hourly snapshot aggregations.
• Per-tenant config UI.

🧩 Milestone 12 — Optional Surface Area

• Knowledge Vault with wikilinks ([[target]]).
• MCP bridge (stdio + SSE + streamable-http transports, per-agent + per-user grants).
• Custom shell tools (DB-stored, hot-reloaded).
• Cron jobs.
• Browser automation (headless Chrome).

🕰️ Save for last (don't build until milestone 12)

• Fork agents (cache-driven sub-agents)
• Swarm teams
• Remote tasks across machines
• KAIROS continuous-mode logs
• Auto-mode permission classifier
• Renderer optimization (cell-diffing, BSU/ESU)
• Bitmap search index for huge filesystems

⚠️ Part 15 — Anti-Patterns to Avoid

Each row is a trap that's burned multiple production teams.

🔄 Loop / control flow

🛠️ Tools

🔐 Permissions

⚡ Caching / API

💾 Memory

🤖 Multi-agent

🏢 Multi-tenancy

🪝 Bootstrap / hooks

🔌 Provider / API

🎯 Part 16 — Closing: The Harness Mindset

Three closing observations distilled from every source.

1. 🎯 Push complexity to the boundaries

Permission resolution, protocol translation, state reconciliation, tool I/O — these are the messy edges. Concentrate the mess there. Keep the loop, the tool composition, the memory recall, and the streaming logic clean and exhaustively typed.

2. 🔁 The agent is a function from event history to next event, run in a loop

Everything else is a hook into that one loop:

• "Function" → stateless Agent.
• "Event history" → append-only EventLog.
• "Next event" → Action, executed by Workspace, producing Observation.
• "Run in a loop" → Conversation, until Finish or stuck.

There is no big design. There is one tight kernel and a lot of small components hanging off it.

3. 🔧 Iterate on failures

The single most important cultural practice from the harness-engineering discipline:

Anytime an agent makes a mistake, engineer a solution so it never makes that mistake again.

Ship first. Add configuration reactively. Throw away what doesn't help. Distribute battle-tested configurations. Treat technical debt as a high-interest loan.

After many production incidents the pattern is the same:

• "GPT-6 will fix it" → almost always wrong.
• "It's a configuration problem" → almost always right.

The fix is in your harness — context management, tool selection, verification loops, handoff artifacts, prompt reinforcement zones, hook ordering, error ladders.

🍳 The shortest possible recipe

If you only build six things well, you have a great agent:

1. An async-generator loop with typed terminal states and a continue-state ladder for recovery.
2. A self-describing tool registry with per-invocation safety, the 14-step pipeline, and bounded output.
3. A 4-layer context compression pipeline preserving the prompt cache architecture.
4. File-based memory with always-loaded index + LLM recall side-query.
5. Defense-in-depth security with five independent layers.
6. Multi-tenancy on day one — tenant_id NOT NULL everywhere.

Build those, and you've shipped a real agent. The rest of this guide is layering and polish.

📚 Appendix — Source Map

"It's not a model problem. It's a configuration problem." — every team, after enough incidents.

If you found this helpful, let me know by leaving a 👍 or a comment!, or if you think this post could help someone, feel free to share it! Thank you very much! 😃