Coder Legion - Recent posts tagged agentical

Your Agent Died at Step 7 of 12: Durable Execution for Long-Running Agents

Mon, 21 Sep 2026 09:25:59 +0000

The agent was twenty minutes into a migration task. It had read the schema, planned eleven steps, executed six of them — two of which wrote to production — and was waiting on the model for step seven when the pod was rescheduled.

Everything the agent knew lived in a Python list called messages, in the memory of a process that no longer exists.

You now have three bad options. Redo the whole run, and steps 1–6 happen a second time, including the two that wrote to production. Abandon it, and leave a migration half-applied. Or have a human reconstruct what happened from logs, which is what most teams actually do, at 11 p.m.

None of these are retry problems. Retries handle a 429 or a 529 inside a process that is still alive — a separate and well-understood layer. This is what happens when the process itself goes away, and the fix is architectural: the run has to live in the database, not in the heap.

Why long agent runs die, and why it is not exotic

A ten-second chat completion effectively never gets interrupted. A twenty-minute agent run gets interrupted constantly, because twenty minutes is long enough to collide with ordinary operations:

A deploy. You ship twice a day. Any run longer than your drain timeout is killed by your own CI.
A platform timeout. Lambda caps at 15 minutes. Cloud Run request timeouts, ALB idle timeouts, and Cloudflare's limits all sit below "agent doing real work".
OOM. Agent runs accumulate tool results — a large file read, a big query result — and the process footprint grows with the transcript.
A long backoff. You hit a rate limit with a retry-after of 90 seconds, and something upstream decides the worker is hung.
A crash in your own tool code. An unhandled exception in step 7 takes the loop down with it, and the loop was holding the only copy of steps 1–6.

The frequency is what surprises people. If a run takes 20 minutes and you deploy twice a day with a 30-second drain, a meaningful fraction of in-flight runs die every single day. At one run an hour that is invisible; at a thousand it is a daily incident.

The model: a run is a state machine, not a for-loop

The mental shift is small and everything follows from it. Today your loop probably looks like this:

def run_agent(task: str) -> str:
    messages = [{"role": "user", "content": task}]
    while True:
        resp = client.messages.create(model=MODEL, messages=messages, tools=TOOLS, max_tokens=8000)
        messages.append({"role": "assistant", "content": resp.content})
        if resp.stop_reason != "tool_use":
            return text_of(resp)
        results = [execute(b) for b in resp.content if b.type == "tool_use"]
        messages.append({"role": "user", "content": results})

messages is the state, the heap is the storage, and the process is the transaction boundary. Kill the process and the state is gone.

The durable version separates three things that the loop above conflates:

The run — a row with a status and a task. It outlives every worker.
The steps — an append-only log of what happened, in order. This is the conversation; messages becomes a projection of it.
The worker — a disposable process that leases a run, advances it one step, commits, and repeats. Any worker can pick up any run at any time.

Step 7 crashing now means: the run is still running, steps 1–6 are committed, and the next worker to pick up the lease rebuilds the message list from the log and continues at step 7. Nothing is redone.

The schema

CREATE TABLE agent_run (
    id            UUID PRIMARY KEY,
    task          TEXT NOT NULL,
    status        TEXT NOT NULL,          -- pending|running|succeeded|failed|cancelled
    model         TEXT NOT NULL,          -- pin it: see the append-only section
    lease_owner   TEXT,                   -- worker id holding this run
    lease_until   TIMESTAMPTZ,            -- heartbeat expiry
    attempt       INT  NOT NULL DEFAULT 0,
    result        TEXT,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE agent_step (
    run_id        UUID NOT NULL REFERENCES agent_run(id),
    seq           INT  NOT NULL,          -- 0,1,2... strictly increasing per run
    kind          TEXT NOT NULL,          -- model_turn | tool_result
    payload       JSONB NOT NULL,         -- the raw content blocks, verbatim
    request_id    TEXT,                   -- provider request id, for support tickets
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (run_id, seq)
);

Two constraints carry most of the weight.

PRIMARY KEY (run_id, seq) makes double-writes impossible. If two workers somehow both try to append step 7, one of them gets a unique violation and loses. That is not an edge case to handle later; it is the safety property that lets you run more than one worker.

Steps are append-only. Never UPDATE agent_step. A retried tool call is a new step with a new seq and its own error payload, not an overwrite of the failed one. You lose nothing, you can replay any run exactly, and post-mortems stop being fiction.

payload is verbatim. Store the provider's content blocks as they arrived — do not normalize, do not strip fields you think are noise. The next section is about why that is not merely good hygiene.

Rebuilding the conversation: append-only history is now enforced

Resume means reconstructing messages from the step log:

def rebuild_messages(db, run_id: str) -> list[dict]:
    steps = db.query(
        "SELECT kind, payload FROM agent_step WHERE run_id = %s ORDER BY seq", run_id
    )
    messages = [{"role": "user", "content": steps[0]["payload"]["task"]}]
    for step in steps[1:]:
        role = "assistant" if step["kind"] == "model_turn" else "user"
        messages.append({"role": role, "content": step["payload"]})
    return messages

Simple — and there is one trap in it that turns a working resume into a hard 400 on the current generation of models.

Anthropic's Fable 5.1 made conversation history append-only. Each thinking block is cryptographically bound to the system prompt, the tool definitions, and the history that preceded it. Modify any of that on the client and you get Invalid "signature" in "thinking" block — a 400, not a degraded response. (Enforcement is implicit for accounts created from 31 August 2026 onward; older accounts can opt in with the thinking-binding-controls-2026-08-01 beta to find out now rather than later.)

Resume is precisely the operation that tempts you to modify history. Three rules keep you on the right side of it:

Store payloads verbatim. A "cleanup" that drops a field you considered redundant will break the signature. Serialize what the SDK gave you, whole.
Pin the model on the run row. thinking blocks are bound to the model that produced them; 5.1 can read blocks from older models, but not the reverse. A worker that picks up an old run and reads MODEL from environment config — which you just bumped — resumes with the wrong model and fails at a distance from the cause.
Pin the system prompt and tool definitions too, or version them. Deploying a reworded system prompt mid-run invalidates the binding for every in-flight run. Store a prompt_version on the run and resolve the text from a versioned registry.

That last rule catches teams weeks later: resume worked perfectly in testing, then the day someone edits the system prompt, every run that survives the deploy fails on its next turn. Note that server-side compaction and context editing do not count as client-side modification — this constraint is about what you rewrite.

Idempotency: the step that spends money

Persisting the transcript fixes the "redo the whole run" problem. It does not fix the narrower and more expensive one: a tool call executed, and the process died before the result was committed.

You cannot tell those two apart from the outside. All you know at resume time is that step 7 has no result. The tool may have done nothing, or it may have charged a card.

At-least-once delivery is the only guarantee a crash-safe system can offer. Exactly-once execution is achieved the same way it is everywhere else in distributed systems: at-least-once delivery plus idempotent effects.

import hashlib, json

def idempotency_key(run_id: str, seq: int, tool: str, args: dict) -> str:
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(f"{run_id}:{seq}:{tool}:{canonical}".encode()).hexdigest()

The key is derived from position in the run, not from time or a UUID. Two attempts at step 7 produce the same key; step 8 produces a different one. Pass it to anything with a side effect — Stripe, your own API, a queue — and the second attempt returns the first result instead of performing the action again.

For tools you own and that have no natural idempotency support, record the effect in the same transaction as the step:

def execute_step(db, run, seq, tool_use):
    key = idempotency_key(run["id"], seq, tool_use.name, tool_use.input)

    cached = db.fetchone("SELECT result FROM tool_effect WHERE key = %s", key)
    if cached:
        return cached["result"]                  # already happened; do not repeat

    result = TOOLS[tool_use.name](**tool_use.input)

    with db.transaction():
        db.insert("tool_effect", key=key, result=result)
        db.insert("agent_step", run_id=run["id"], seq=seq,
                  kind="tool_result", payload=serialize(result))
    return result

There is still a window — between TOOLS[...]() returning and the transaction committing — where a crash loses the record of an effect that happened. It cannot be closed without a distributed transaction, and you do not want one. Shrink it instead: for genuinely dangerous tools, write an intent row before calling, so resume finds "we were about to charge this card" and can ask a human rather than guess. High-risk actions are also exactly where a human approval gate belongs, and an approval record is itself a durable checkpoint — the two designs reinforce each other.

Tool classification, done once, at design time:

Class	Example	Resume policy
Pure read	`get_schema`, `search_docs`	Just re-run it
Idempotent write	`set_status(id, "done")`	Re-run; same end state
Non-idempotent	`charge_card`, `send_email`, `post_message`	Idempotency key, mandatory
Irreversible	`DROP TABLE`, `terminate_instance`	Intent row + human confirmation

The row that gets skipped is the third one, and it is the one that shows up in a customer complaint.

Leases: stopping two workers from running the same run

Once runs are in a table and workers are disposable, the obvious failure is two workers picking up the same run — one that appeared dead but was only in a long backoff. Both advance it, both append steps, and the primary key saves you from corruption but not from duplicate tool calls.

A lease with a heartbeat fixes it:

def claim_run(db, worker_id: str, lease_seconds: int = 60):
    return db.fetchone("""
        UPDATE agent_run
           SET lease_owner = %s,
               lease_until = now() + make_interval(secs => %s),
               status      = 'running',
               attempt     = attempt + 1
         WHERE id = (
             SELECT id FROM agent_run
              WHERE status IN ('pending', 'running')
                AND (lease_until IS NULL OR lease_until < now())
              ORDER BY created_at
              FOR UPDATE SKIP LOCKED
              LIMIT 1
         )
     RETURNING *;
    """, worker_id, lease_seconds)

FOR UPDATE SKIP LOCKED is what makes this safe under concurrency without a separate queue: two workers polling simultaneously get two different runs, never the same one. The worker renews lease_until on every committed step. If it dies, the lease expires and the run becomes claimable — with no supervisor process needed to notice the death.

Keep the lease short (60 seconds) and renew often, rather than long and hopeful. A long lease means a dead worker's run sits idle for the length of the lease. Renew on each step and, if a single step can exceed the lease, from a background heartbeat thread.

Guard against poison runs with attempt: a run that has been claimed eight times is failing deterministically and must go to a dead-letter status. Without that check, one malformed run will be retried by your fleet forever.

Cancellation, for free

A benefit that pays for the whole design on its own: with the run in a table, cancellation is an UPDATE.

# in the worker, checked before each step
if db.value("SELECT status FROM agent_run WHERE id = %s", run_id) == "cancelled":
    return

Try cancelling an in-flight while True loop inside an anonymous process. You cannot, which is why so many agent products have no stop button — and why so many have a stop button that only hides the UI while the tokens keep burning.

The same table makes the operational surface trivial: runs by status, oldest pending run, p95 steps per run, attempts distribution. That is a dashboard and an alert, built from data you already have. Pipe run_id into every log line and a support question ("what did it do for this customer?") becomes one query instead of a log-grep expedition. This is the ordinary SRE treatment applied to agents, and it works because you made the agent ordinary software.

When to use a workflow engine instead

Everything above is roughly 200 lines on top of Postgres. Temporal, Restate, and similar durable-execution engines give you the same guarantees with the replay semantics handled for you.

Build it yourself when: you already run Postgres, your steps are minutes not days, and you want the step log in the same database as your application data so you can join against it. The debugging story — SELECT * FROM agent_step WHERE run_id = ... — is genuinely better than an opaque engine's event history.

Use an engine when: runs span days and involve human waits, you need timers and signals, you are coordinating multiple agents that must rendezvous, or you already run one for other workloads. Do not adopt one only for agents; the operational cost is real, and an agent run is a simpler shape than the workflows these engines were built for.

The part that is not optional either way is idempotency. Temporal will happily re-execute your non-idempotent activity during a replay. Durable execution guarantees that the workflow resumes correctly, never that your side effects are safe.

FAQ

Does this apply to a 10-second chat completion?
No. If a run cannot outlive a deploy, an in-memory loop is the right amount of machinery. The threshold is roughly "longer than your drain timeout" — past that, interruption stops being an edge case and becomes a daily event.

Can I just store the messages list as a JSON blob after every turn?
It survives a crash, which is most of the benefit, and it is a reasonable first version. What it does not give you is a per-step primary key (so concurrent workers can corrupt it), a record of failed attempts, or a queryable history. The step table is maybe forty more lines and you stop rewriting the same row under concurrency.

Do I need a queue like SQS or Celery in front of this?
Not for correctness — FOR UPDATE SKIP LOCKED is a perfectly good work queue at moderate volume, and keeping the queue in the same database means claiming a run and committing a step share one transaction. Add a broker when you need fan-out to many worker types or cross-service routing.

What happens to a run whose model was retired mid-flight?
It fails, and it should. Pinning the model on the run row is what makes that a clean, early, obvious failure instead of a confusing one. Drain long runs before removing a model from your rotation, and treat any run still pinned to a dead model as a dead-letter case for a human.

Is streaming compatible with this?
Yes. Stream to the user for responsiveness, and commit the assembled message to the step log before dispatching any tool. The checkpoint boundary is the completed turn, not the token.

Checklist

[ ] Every model turn and tool result is committed to an append-only step log before the next call
[ ] PRIMARY KEY (run_id, seq) — duplicate steps are impossible, not merely unlikely
[ ] Payloads stored verbatim; model, system prompt, and tool definitions pinned per run
[ ] Every tool classified: pure / idempotent / non-idempotent / irreversible
[ ] Non-idempotent tools take a key derived from run_id + seq + args
[ ] Irreversible tools write an intent row before executing
[ ] Leases with FOR UPDATE SKIP LOCKED and a heartbeat; attempt capped with a dead-letter status
[ ] Cancellation is a status change the worker checks between steps
[ ] A run_id in every log line
[ ] A test that kills the worker mid-run and asserts the resumed run does not repeat a side effect

That last one is the only way to know any of this works. Run the agent, SIGKILL the worker between steps 3 and 4, start a new worker, and assert the effect count. Do it in CI. Crash-safety that has never been crashed is a hypothesis.

The short version

Agent runs are long enough to be interrupted by routine operations, and an in-memory messages list is not a place to keep work you cannot afford to redo. Move the run into a table, make the step log append-only, rebuild the conversation on resume, and make every side-effecting tool idempotent by position. None of this is AI-specific — it is the same durability discipline as any long-running job. The only genuinely new constraint is that modern models bind their reasoning blocks to the exact history that produced them, so "rebuild the conversation" now means byte-identical, not equivalent.

I build and teach production AI systems at Cursuri-AI.ro, Eastern Europe's AI education platform. If you are taking an agent from a working demo to something that survives a deploy, the production AI SaaS track covers this end to end.

Why Your RAG Pipeline Returns Wrong Answers (7 Failure Modes and Fixes)

Thu, 18 Jun 2026 13:21:58 +0000

You built a RAG system. You embedded your docs, wired up a vector store, stuffed the top chunks into the prompt, and shipped. Then the questions came back wrong — confidently, fluently wrong — and your first instinct was to blame the model or reach for a bigger one.

Here's the uncomfortable truth after debugging a lot of these: the model is almost never the problem. In a Retrieval-Augmented Generation pipeline, the model can only answer from what you put in its context. If retrieval hands it the wrong chunks, even a frontier model will produce a beautifully written wrong answer. RAG failures are retrieval failures wearing a generation costume.

This is a field guide to the seven failure modes that cause most bad RAG answers, how to recognize each one, and the concrete fix. No new framework to adopt — just the debugging order that actually finds the bug.

First, the one diagnostic that saves you hours

Before touching anything, answer one question: was the correct information in the retrieved chunks at all?

Log the actual chunks your retriever returned for a failing query and read them with your own eyes. This splits every RAG bug into two worlds:

The answer was not in the retrieved chunks → you have a retrieval problem (failure modes 1–5 below). The generator never stood a chance.
The answer was in the chunks but the model ignored or mangled it → you have a generation problem (failure modes 6–7).

Skipping this step is why people waste days tuning prompts when the real bug is that the right paragraph was never retrieved. Do this first, every time.

One more split worth making early: is the question single-hop ("what is our refund window?") or multi-hop ("how does our refund window compare to the EU default?"). Multi-hop questions need facts from several documents at once, and a top-k retriever tuned for single-hop will reliably fetch one of the two needed pieces and miss the other. If your failures cluster on comparative or multi-part questions, the fix isn't a better embedding — it's query decomposition (break the question into sub-queries, retrieve for each, then synthesize), which is a different lever than everything below.

Failure mode 1: Your chunking destroys meaning

The single most common cause of bad retrieval is naive chunking — splitting documents every N characters with no regard for structure. You end up with chunks that cut a sentence in half, separate a heading from its content, or split a table from the row that gives it meaning. An embedding of half a thought retrieves poorly because it is half a thought.

Symptoms: retrieved chunks are topically close but never contain the complete fact; answers are vague or stitched from fragments.

The fix: chunk along the document's natural structure, not a fixed byte count.

Split on semantic boundaries — paragraphs, sections, Markdown headings — not blind character windows.
Add overlap (e.g. 10–20%) so a fact straddling a boundary survives in at least one chunk.
Keep chunks coherent: one chunk should answer one thing. For structured docs, keep a table or list with its caption.
Attach metadata to each chunk (source, section title, date) — you'll use it for filtering and citations later.

There's no universal chunk size; it depends on your documents and your embedding model's sweet spot. Measure it (see failure mode 7), don't guess. Chunking is deceptively deep — it's where a structured course on RAG and retrieval-augmented generation spends real time, because getting it right fixes more bugs than any prompt tweak.

Failure mode 2: The wrong embedding model (or a mismatch)

Your embedding model decides what "similar" means. Two failures hide here.

First, a weak or generic embedding model can't tell apart concepts that matter in your domain — legal clauses, medical terms, internal product names all collapse into the same neighborhood. Second, and more insidious: a mismatch between query and document embeddings. If you embed documents with one model and queries with another, or you embed a terse 6-word question and expect it to land near a dense 400-token passage, the vectors live in different worlds.

Symptoms: retrieval returns plausibly-related-but-not-relevant chunks; performance is mediocre across the board rather than failing on specific queries.

The fix:

Use a strong, current embedding model suited to your domain and language — for non-English content, verify it was actually trained for it.
Embed queries and documents with the same model and the same preprocessing.
Consider query transformation: rewrite the short user question into a richer form (or generate a hypothetical answer and embed that — the HyDE technique) so it sits closer to real passages.

Failure mode 3: Pure vector search misses exact terms

Dense vector search is great at semantics and surprisingly bad at exact matches — error codes, SKUs, function names, dates, proper nouns. A user asking about ERR_CONN_4021 doesn't want "semantically similar" connection errors; they want that string. Embeddings smear it into a neighborhood and the exact doc ranks fifth.

Symptoms: great on conceptual questions, weak on specific identifiers, names, or numbers.

The fix: hybrid search. Combine dense vector search with classic keyword search (BM25/full-text) and fuse the results (Reciprocal Rank Fusion is a simple, strong default). You get semantics and exact-match precision. Most production-grade vector stores support hybrid retrieval now — turn it on.

Failure mode 4: No re-ranking, so the best chunk isn't on top

First-stage retrieval optimizes for recall — cast a wide net and return, say, the top 50 candidates. But you only feed a handful to the model, and the ordering of that first stage is coarse. The genuinely best chunk is often in your top 50 but not your top 5, so it never reaches the prompt.

Symptoms: when you log the top 20–50 candidates, the right chunk is in there — just ranked too low to make the cut.

The fix: add a re-ranking stage. Retrieve a generous candidate set, then run a cross-encoder reranker that scores each candidate against the query directly (far more accurate than the initial vector distance) and keep the top few after re-ranking. This two-stage "retrieve wide, re-rank narrow" pattern is one of the highest-ROI upgrades in RAG and often fixes the "it was almost right" class of bugs outright. Designing this multi-stage retrieval well is core to a deeper course on advanced LLM integration.

Failure mode 5: Stale, duplicated, or contradictory sources

Retrieval is only as good as the corpus. If your index contains last quarter's pricing alongside this quarter's, three near-duplicate copies of the same policy, or contradictory drafts, retrieval will faithfully surface the wrong-but-similar one. The pipeline isn't broken — your data is.

Symptoms: answers are outdated, contradict each other across runs, or cite a deprecated document.

The fix:

Deduplicate before indexing, and keep the index fresh — re-index or invalidate on source changes, don't let it drift.
Store a date/version in chunk metadata and filter or boost by recency at query time.
Have a single source of truth per fact; archive superseded docs out of the active index.
Garbage in, garbage retrieved. Data hygiene is unglamorous and it outperforms most clever tricks.

Failure mode 6: The context is right but the prompt undermines it

Now the generation side. Suppose the correct chunk is in context but the answer is still wrong. Often the prompt is the culprit: it doesn't tell the model to ground its answer in the provided context, doesn't give it permission to say "I don't know," or buries the relevant chunk among ten irrelevant ones so the signal drowns (the "lost in the middle" effect, where models attend least to the middle of a long context).

Symptoms: the right info is in the prompt, yet the model invents, ignores it, or pads with prior knowledge.

The fix:

Instruct explicitly: "Answer only using the context below. If the answer isn't there, say you don't know." This single line kills a surprising amount of hallucination.
Pass fewer, better chunks — re-ranking (mode 4) makes this possible. More context is not more better; irrelevant chunks actively hurt.
Put the most relevant chunks at the start or end, not buried in the middle.
Ask for citations — make the model name which chunk each claim came from. It both reduces fabrication and gives you a debugging trail.

Deciding what deserves a place in the context window — and what to leave out — is its own discipline, increasingly called context engineering, and it's the focus of a dedicated course on context engineering for AI agents.

This is the meta-failure that makes all the others permanent. If "is it working?" is answered by eyeballing a few queries, you can't tell whether a change helped, hurt, or just moved the failures around. Every fix above becomes a coin flip.

Symptoms: you "improve" the pipeline and have no idea if it actually got better; regressions ship silently.

The fix: build an evaluation set and measure, separately, the two halves of the pipeline.

Retrieval metrics — on a set of questions with known relevant documents, measure recall@k and precision@k. This tells you whether the right chunk is even being retrieved (the diagnostic from the intro, made systematic).
Generation metrics — given the retrieved context, measure faithfulness (does the answer stay grounded in the context?) and answer relevance. LLM-as-judge scoring works well here when calibrated.

Even 30–50 representative question/answer pairs turn RAG tuning from vibes into engineering: you change one thing, you re-run, you see the number move. Treating evals as first-class is exactly the muscle a course on AI evals in production builds, and it's the difference between "I think the reranker helped" and "recall@5 went from 0.61 to 0.84."

The debugging order, in one list

When a RAG answer is wrong, resist the urge to swap models. Work the pipeline in this order:

Log the retrieved chunks. Was the answer even in there? This alone routes you to retrieval vs. generation.
If not retrieved → check chunking (1), embeddings (2), hybrid search (3), re-ranking (4), and data freshness (5), roughly in that order of likelihood.
If retrieved but ignored → fix the prompt and context layout (6).
Throughout, measure with evals (7) so every change is verified, not hoped.

Nine times out of ten the bug is in steps 1–2, not the model. Working the steps in order also stops you from stacking fixes blindly — change one variable, re-measure, and you'll usually find a single failure mode is responsible for most of your bad answers.

Conclusion

RAG looks simple in the tutorial — embed, search, prompt — and that simplicity is exactly why broken RAG is so common. The hard parts are invisible until you debug them: how you split documents, how you embed and search, how you re-rank, how clean your corpus is, and whether you can measure any of it. None of those are model problems, and none are solved by paying for a bigger model.

The teams whose RAG actually works aren't using a secret model. They're disciplined about retrieval and they measure everything. Start with the diagnostic — log your chunks — fix the failure mode it points to, and put an eval set behind every change. That's how a RAG demo becomes a RAG system people can trust.

The courses linked throughout are part of Cursuri-AI.ro, an AI-learning platform with hands-on, current tracks on RAG, LLM integration, context engineering, and evaluating AI systems in production.

Sources & further reading:

Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (original RAG paper)
Liu et al. — Lost in the Middle: How Language Models Use Long Contexts
Gao et al. — Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE)

This article is educational content. Techniques and tooling evolve quickly; validate approaches against your own data and current library documentation.

Coder Legion - Recent posts tagged agentical

Your Agent Died at Step 7 of 12: Durable Execution for Long-Running Agents

Why long agent runs die, and why it is not exotic

The model: a run is a state machine, not a for-loop

The schema

Rebuilding the conversation: append-only history is now enforced

Idempotency: the step that spends money

Leases: stopping two workers from running the same run

Cancellation, for free

When to use a workflow engine instead

FAQ

Checklist

The short version

Why Your RAG Pipeline Returns Wrong Answers (7 Failure Modes and Fixes)

First, the one diagnostic that saves you hours

Failure mode 1: Your chunking destroys meaning

Failure mode 2: The wrong embedding model (or a mismatch)

Failure mode 3: Pure vector search misses exact terms

Failure mode 4: No re-ranking, so the best chunk isn't on top

Failure mode 5: Stale, duplicated, or contradictory sources

Failure mode 6: The context is right but the prompt undermines it

Failure mode 7: You have no evals, so you're debugging blind

The debugging order, in one list

Conclusion