Build a Python agent harness you can extend and test without calling a model

Wed, 21 Oct 2026 12:50:26 +0000

A team ships a small Python agent with two tools. A month later someone renames an argument on one tool function and forgets the hand-written JSON schema sitting a few lines above it. The model keeps sending the old argument name, the function raises on every call, and nobody finds out until a real task hits that tool, because the only way anyone ever tested the loop was by running it against a live model.

The layout caused this one. When tool contracts are written by hand and the loop only runs end to end, every change to a tool risks the whole agent, and every test costs an API call. A harness you can live with keeps three things apart: the loop, the tool contracts, and the tests that prove the loop behaves.

The loop has three parts, and only one of them should change when you add a tool

A minimal agent harness in Python has a loop over the model's stop_reason, a dispatcher that maps a tool name to a Python function, and a messages list that grows every turn. The loop keeps running while the model asks for tools (stop_reason == "tool_use") and returns the text answer when it sends end_turn.

Of those three, only the set of tools should grow as the agent gains abilities. The loop itself should not change when you add a tool, and the message list should follow the same rules on every turn. A production harness wraps more around this core, such as sandboxing, permission checks, context compaction and session state. Those layers are easier to add later when the core stays this small.

Generate the schema from the function's input type

Every tool needs a name, a description and an input_schema that tells the model which arguments it accepts. Writing that JSON Schema by hand is how the opening bug happens. The schema and the function live in two places, and only one of them gets edited.

Describe each tool's inputs with a Pydantic model and generate the schema from it:

One model per tool. A ReadFile model with a single path: str field describes the input of read_file.
Let the library write the schema. ReadFile.model_json_schema() returns a plain dict with path as a required string, which is the input_schema shape the Anthropic API validates against.
Edit the type and the schema follows. Add a field or mark one optional and the generated schema updates itself. This method ships with Pydantic v2.

The Pydantic model becomes the single source of truth for a tool's inputs, and the function's arguments have to match its fields. The dispatch fixture in the testing section below fails the moment the two drift apart, long before a real task finds it.

Dispatch by name so the loop never names a tool

Keep a dict called HANDLERS that maps each tool name to its function, and dispatch with one line: HANDLERS[block.name](**block.input). The loop reads the tool name from the model's tool_use block and never refers to a specific tool.

That one decision is what makes the harness extendable. Adding a tool touches three places, and none of them is the loop:

Define the input model. A WriteFile model with path and content fields.
Add one entry to the tool list. Name, description, and WriteFile.model_json_schema() as the schema.
Register the function. HANDLERS["write_file"] = write_file.

The agent can now write files as well as read them, and run_agent did not change by a single character. If adding a tool ever forces an edit inside the loop, a tool name has leaked into it.

Treat what comes back as part of the contract

The input side is only half of a tool's contract. The output has rules too. The content of a tool_result must be a string or a list of content blocks. A read_file that returns a string passes straight through, while a list_files that returns a Python list has to be serialized first, for example with json.dumps.

Put that rule in the loop once, so no single tool has to remember it. The same goes for pairing: copy the id from each tool_use block into the result's tool_use_id, and append the assistant message before the user message that carries the results. These are loop rules, and they belong next to the dispatch line, where every tool gets them for free.

Test the loop without calling a model

Testing a harness by running it against a live model and reading the output costs money, takes seconds per run, and only covers the path the model happened to take that time.

The loop is ordinary Python, so test it like ordinary Python. Mock the messages.create call to return fixtures, and assert on what the loop does with them:

A tool-use fixture drives one turn. Return a response with stop_reason set to tool_use and a fake tool_use block, then check that the matching handler ran with the arguments in the block.
The append order holds. After that turn, the assistant message should sit before the user message that carries the result, and the result's tool_use_id should match the block's id.
Parallel calls arrive together. Return a fixture with two tool_use blocks and assert that both results go back in a single user message before the next call.
Non-string results get serialized. Point a fake tool at a list return and check that the content the loop sends back is a string.
The loop exits on end_turn. Follow the tool turn with an end_turn fixture and check that run_agent returns its text.
The ceiling fires. Feed tool_use fixtures forever and check that the loop raises once it reaches max_turns, instead of spinning.

One detail matters when you read the mock's recorded calls: run_agent keeps appending to the same messages list, so copy the list inside the mock if you want to assert on what each call saw. These tests run in milliseconds and never reach the API, and they cover the rules that break a harness without anyone noticing. Run them on every change, and keep the live-model run for checking behavior that only a model can show.

Know what a framework would take over

The OpenAI Agents SDK runs the same loop for you through Runner.run. You describe tools as Python functions with type annotations, and the SDK raises MaxTurnsExceeded when a run passes its max_turns, which is the same safety valve as the manual counter.

Reach for the SDK when you need multi-agent handoffs, built-in guardrails or tracing. Keep the manual loop when you want zero framework dependency or need to see exactly what happens on every turn. OpenAI Swarm is an experimental, stateless project, and the Agents SDK has superseded it for anything you plan to run in production.

Either way, the design rule carries over. Tool definitions stay separate from loop logic, and switching to a framework later holds no surprises in the control flow.

Pitfalls that keep showing up

Hand-written schema dicts. They drift from the function the first time someone edits a signature.
Tool names inside the loop. Every new tool turns into a change to the control flow.
Raw Python objects in tool_result. The API expects a string or content blocks, so a returned list fails validation.
A missing max_tokens. It is required on every Anthropic request and has no default, so the SDK raises before the model ever runs.
Live-model runs as the only test. They are slow, they cost money, and they never force the edge cases.

The full program, with the tool definitions, the loop and the stop conditions in one file you can run, is in the build guide with the full code.

A loop that never names a tool, schemas generated from types and tests that need no model call together keep a harness small. Every new tool after that is a few lines next to the others.

Coder Legion - Recent posts tagged agent-harness