Coder Legion - Recent posts tagged langgraph

Stop calling functions: my first LangGraph graph, where state does the calling

Wed, 28 Oct 2026 09:39:31 +0000

Three parts in, JobCopilot had layers that worked but couldn't talk to each other. Part 1 gave me a structured-output layer — ParseOk/ParseError as data. Part 2 gave me retrieval — build_context() pulling the top-8 resume chunks for a job description. The pipeline was one line on paper; in code it was nested calls, and the awkward questions had no home: who retries when the model drifts into prose? Who stops the retries? Where does the score live between "parsed" and "shown to the user"?

I did what a React dev does in a new language: I nested functions. The retry loop closed over a closure that closed over a callback, and I couldn't tell you who owned what. Rewriting the flow as a graph — my first with LangGraph — made the nesting disappear. Not because a framework absorbed it, but because the mental model changed: functions stop calling each other; state flows through a graph, and nodes return patches of that state.

It survived everything I threw at it — after I fixed one wire that made every job come back "error", no matter how perfectly the model answered.

1. The state is a pydantic model; nodes return patches

If the graph is the wiring, state is everything a run knows. I made it one pydantic model — deliberately the same discipline as part 1:

# state.py — the whole run in one model
from __future__ import annotations

from typing import Literal

from pydantic import BaseModel, Field

THRESHOLD = 60   # score at or above this = "worth applying"
MAX_CALLS = 3    # 1 first try + 2 repairs — the budget from part 1


class AgentState(BaseModel):
    """One run of the copilot, end to end."""

    jd_text: str = ""
    context: str = ""              # resume chunks, from part 2
    last_raw: str = ""             # what the model last said, verbatim
    error: str | None = None       # latest parse failure, if any
    attempt: int = 0               # how many LLM calls so far

    match_score: int | None = None
    missing_skills: list[str] = Field(default_factory=list)
    verdict: Literal["match", "no_match", "error"] | None = None

The rule that makes a graph a graph: a node never calls another node. It receives state, does one job, and returns a dict with only what changed — a patch. LangGraph merges the patch and hands state to the next node. retrieve fills context; judge fills the score fields; nobody knows about anybody else.

Fields without a custom reducer are last-writer-wins — I proved it to myself before trusting it:

def note_a(state: AgentState) -> dict:
    return {"missing_skills": ["Python"]}

def note_b(state: AgentState) -> dict:
    return {"missing_skills": ["Terraform"]}

# a -> b in that order: missing_skills == ["Terraform"] — b overwrote a

For my state that's correct by design: every field is overwrite-on-purpose, so zero reducers. If a list should accumulate across nodes, opt in with Annotated:

from typing import Annotated


def merge_lists(left: list[str] | None, right: list[str] | None) -> list[str]:
    if left is None:
        return right or []
    if right is None:
        return left
    return list(left) + [x for x in right if x not in left]


class AccumState(BaseModel):
    missing_skills: Annotated[list[str], merge_lists] = Field(default_factory=list)

# two nodes appending "Python" then "Terraform" -> ["Python", "Terraform"]

And why pydantic for state instead of the TypedDict every tutorial starts with? A patch is data from somewhere else, and at merge time nobody type-checks a plain dict:

# TypedDict state, merged by dict spread — type-checkers scream, Python shrugs:
>>> {**{"attempt": 0, "verdict": None}, "verdict": 21, "attempt": "two"}
{"attempt": "two", "verdict": 21}          # garbage, happily stored

# pydantic model:
>>> PS(attempt="two", verdict=21)
pydantic.ValidationError: attempt — Input should be a valid integer...

Same lesson as part 1: shape is a contract, and contracts live in pydantic, not in comments.

2. The graph is a handful of declared edges

Here is the entire graph.py — wiring, not logic:

# graph.py
from langgraph.graph import END, START, StateGraph

from nodes import Chat, fail_node, judge_node, retrieve_node, route_after_parse
from state import AgentState


def build_graph(chat: Chat):
    g = StateGraph(AgentState)

    g.add_node("retrieve", retrieve_node)
    g.add_node("judge", lambda s: judge_node(s, chat))  # closure: node gets the LLM client
    g.add_node("fail", fail_node)

    g.add_edge(START, "retrieve")
    g.add_edge("retrieve", "judge")

    g.add_conditional_edges(
        "judge",
        route_after_parse,          # returns "ok" | "retry" | "give_up"
        {"ok": END, "retry": "judge", "give_up": "fail"},
    )
    g.add_edge("fail", END)

    return g.compile()

Coming from React, this shape clicked instantly: it's a component tree. Components don't import each other to call each other — data flows down a declared structure, and the structure decides what happens next.

The lambda is deliberate: the graph is built once with a concrete LLM client, and no node ever imports a provider. That's what makes the graph testable — hand it a fake chat and the whole flow runs with zero network. Which is exactly how I caught the bug.

3. All routing logic in one boring function — and the wire that said "error" to everything

add_conditional_edges takes a router: a pure function from state to a string, plus a mapping from strings to destinations. Mine is twelve lines and holds the entire loop policy of the agent:

# nodes.py
from state import MAX_CALLS


def route_after_parse(state: AgentState) -> str:
    """All loop logic in one boring function."""
    if state.error is None:
        return "ok"
    if state.attempt < MAX_CALLS:
        return "retry"
    return "give_up"

Compare with my first wiring attempt, written because the router concept felt foreign:

g.add_edge("retrieve", "judge")
g.add_edge("judge", "fail")     # "temporarily — I'll wire the real branches next"

Then I ran it — with a fake chat returning a perfect JSON verdict. Result: verdict = "error". Every single job. The parse was flawless, the score valid, and the graph marched it into fail_node anyway: I had declared exactly one outgoing edge from judge, and a graph never "falls through". Nothing raised, nothing warned — the state machine politely followed the only road I built:

>>> # plain edge judge -> fail, no conditional edges, perfect JSON from the model
>>> app.invoke(AgentState(jd_text="junior"))
verdict = "error"     # a valid parse, executed through the wrong edge

The fix was the conditional edges above, plus two habits that came with it:

Route values are data, not flow control. If the router returns a key that's not in the mapping, LangGraph doesn't guess — it throws KeyError at runtime. I know because my typo did exactly that. An unmapped route crashes loudly; a wrongly-wired edge fails silently. I'll take the crash.
Loops are declared, not nested. "retry": "judge" makes a cycle in the graph. The loop exists as an edge, its budget lives in MAX_CALLS, and its stop condition is one pure function I can unit-test without LLM, network, or shame.

The shift, internalized: in nested functions, control flow was who calls whom — invisible at a glance. In the graph, it's data in the edges — visible in one screenful.

4. Closing the repair loop with state, not recursion

Last piece: judge_node, the only node that talks to the model. It reads state to decide which prompt to send — first pass or repair — and returns a patch either way:

# nodes.py
from schemas import ParseError, ParseOk, parse_match
from state import AgentState, THRESHOLD

MATCH_PROMPT = """You are a skeptical technical recruiter.
Compare the job description to the candidate's resume excerpt.
Reply with ONLY a JSON object:
{{"match_score": <int 0-100>, "missing_skills": [<string, ...>]}}

JOB DESCRIPTION:
{jd}

RESUME EXCERPT:
{context}
"""

FIX_PROMPT = """Your previous reply could not be parsed.

PREVIOUS REPLY:
{previous}

PARSER ERROR:
{error}

Reply with ONLY a valid JSON object:
{{"match_score": <int 0-100>, "missing_skills": [<string, ...>]}}
"""


def judge_node(state: AgentState, chat: Chat) -> dict:
    """One LLM call, one parse, a patch of state. Loops are the graph's job now."""
    if state.error is None:
        prompt = MATCH_PROMPT.format(jd=state.jd_text, context=state.context)
    else:
        prompt = FIX_PROMPT.format(previous=state.last_raw, error=state.error)

    raw = chat(prompt)
    result = parse_match(raw)          # the part-1 contract, unchanged

    patch: dict = {"last_raw": raw, "attempt": state.attempt + 1}
    if isinstance(result, ParseOk):
        score = result.data.match_score
        patch.update(
            match_score=score,
            missing_skills=result.data.missing_skills,
            verdict="match" if score >= THRESHOLD else "no_match",
            error=None,
        )
    else:
        patch.update(error=result.message)
    return patch


def fail_node(state: AgentState) -> dict:
    """The model never produced valid JSON. Say so — as state, not a stack trace."""
    return {"verdict": "error"}

Because build_graph takes chat as a parameter, the end-to-end test is a fake answer card — call one drifts into prose, call two complies:

def flaky_llm():
    calls = {"n": 0}

    def chat(prompt: str) -> str:
        calls["n"] += 1
        if calls["n"] == 1:
            return "Sure! The match is quite good, I'd say around 70 out of 100."
        return '{"match_score": 70, "missing_skills": ["Terraform"]}'

    return chat


app = build_graph(flaky_llm())
final = app.invoke(AgentState(jd_text="Senior Frontend Engineer (React, TS)"))

verdict: match | score: 70 | attempts: 2 | error: None

The prose landed in last_raw, parse_match filled error, the router said "retry", and the second call got FIX_PROMPT with the model's own prose quoted back at it — the part-1 trick, now driven by edges instead of a while-loop. When the model never complies:

verdict: error | error: no JSON object found in model output | attempts: 3

No exception slicing through three stack frames. fail_node writes verdict: "error" and the graph ends — for an agent run overnight over forty postings, a run that ends with a recorded failure beats ten runs that crashed and took the batch with them. Part 1's "errors as data", promoted from one function to the shape of the whole system.

Wrapping up

The first graph, in one sentence: state is a pydantic contract, nodes return patches, a router returns strings, loops live in edges.

Three things this React-dev-learning-Python took from it:

Patches, not calls. The merge is last-writer-wins by default; accumulators are opt-in via Annotated. Every field in my state is overwrite-on-purpose — zero reducers, but verified behavior before trusting it.
Routing is a pure function. Twelve lines, no LLM, unit-testable — and an unmapped return value crashes with a KeyError. That's the framework refusing to guess, and I've come to love it.
Giving up is a node, not a throw. fail writes verdict: "error" and the graph ends cleanly. The batch survives; the log tells the truth.

And the bug that taught me most: a graph executes exactly the edges you declared — my single judge → fail wire turned perfect parses into errors with zero pushback from the framework. A state machine has no common sense. That's the feature.

Next Tuesday, part 4: evals. The judge says 72 — but is 72 right? Fixtures of job descriptions with expected score ranges, a negative case that must score LOW (a Python-backend posting should not match a frontend resume), and the first time JobCopilot tells me I'm wrong.

Follow if you want to watch the rest of the copilot get built — including the parts that break. Build log + daily code: t.me/novamind_hub. Part 1, in case you missed it: Stop prompting for valid JSON: an LLM output layer that does not break.

Do you build agent graphs with a framework or hand-roll the loop? And what's the first thing your graph did that you didn't expect?

LangGraph4j 1.9.0: What's new and the roadmap

Sat, 19 Sep 2026 13:05:32 +0000

LangGraph4j 1.9.0 is out!

LangGraph4j 1.9.0 is out! With the 1.8.x release stream now in LTS, development moves forward on 1.9 with improvements to streaming, persistence, and the infrastructure for building agents in Java.

This release is an important step toward workflows that are easier to follow while they run and easier to inspect after they finish. A node can now report progress output before returning its result, checkpoint savers have a richer execution lifecycle, and Studio gets an UI refresh.

Let's look at what changes for an application built on 1.8.x, with particular attention to custom output from nodes and checkpoint management. The official migration guide provides details API reference for the changes discussed here.

Emit custom output while a node is running

Imagine a node that retrieves documents, processes them, and prepares a response. The caller may want to show progress throughout that work. Waiting for the final node output gives the user minimal information about what is happening in the meantime.

In 1.9, a node action can emit its own typed NodeOutput values through the active graph execution stream (graph.stream(...)). They arrive alongside the usual START, node, and END outputs, before the node has necessarily finished.

The mechanism is straightforward:

Define a subclass of NodeOutput carrying the information you want to expose.
Obtain a dispatcher from the node's RunnableConfig.
Dispatch custom outputs during execution.
Handle those types when consuming the graph stream.

Here is a small example.

public final class ProgressOutput extends NodeOutput<AgentState> {
    private final String message;

    public ProgressOutput(String node, AgentState state, String message) {
        super(node, state);
        this.message = message;
    }

    public String message() {
        return message;
    }
}

Inside the node, the dispatcher publishes a message while the returned map still provides the normal state update:

AsyncNodeActionWithConfig<AgentState> process = (state, config) -> {
    var dispatcher = config.<AgentState, ProgressOutput>customDispatcher();

    dispatcher.dispatchAsync(
        new ProgressOutput(config.nodeId(), state, "Processing documents"));

    // Perform the node's work here.

    return completedFuture(Map.of("result", "Documents processed"));
};

Once this action is registered in a compiled graph, the caller can recognize its progress messages:

graph.stream(GraphInput.noArgs(), RunnableConfig.empty()).forEachAsync( output -> {
        if (output instanceof ProgressOutput progress) {
            System.out.println(progress.message());
        } else {
            System.out.println("Graph step: " + output.node());
        }
    });

This gives us a useful channel for progress notifications, intermediate results, and application-specific events. Dispatching a custom output does not update graph state. If a value must influence routing, be available to later nodes, or become part of a persisted result, include it in the node's returned state update.

There are two dispatch options. dispatchSync(...) waits until the stream accepts the value and can throw InterruptedException; dispatchAsync(...) submits without waiting. The dispatcher is available during node execution in an active graph stream, including the supported hook context. Keep its use within that lifecycle: accessing it outside has an unpredicable result.

Custom events are intended for graph.stream(...) consumers; they do not change the result returned by graph.invoke(...). See the custom-output tutorial for the complete contract and a hook-based example.

The streaming engine behind it

This capability comes with a refactoring of the internal streaming engine around AsyncGeneratorFlow from async-generator 5.0 project.

Ordinary iteration over stream() remains source-compatible. Applications extending directly streaming generators, supplying a BlockingQueue, or depending on AsyncGenerator.WithResult need migration work.

Note: 👀

This refactoring is a groundwork for Reactor Flow support in a future LangGraph4j 2.0; that support is a future direction.

Checkpoint Savers: A richer execution lifecycle

Checkpoint persistence is central to long-running agents and Human-in-the-Loop workflows. In 1.9, the changes cover what happens when execution finishes, how released runs can be retrieved, how subgraph savers participate, and how interruptions and failures are recorded. These are the checkpoint changes I would review first when upgrading an existing application.

Completed runs release their thread by default

With a checkpoint saver configured, a graph that completes normally now releases its active thread automatically. Depending on the saver, release archives or tags its checkpoints and removes the active checkpoint set.

This matters if your application completes a graph, updates the same active thread's state, and executes it again. To retain the previous lifecycle, explicitly disable automatic release when compiling:

var compileConfig = CompileConfig.builder()
    .checkpointSaver(saver)
    .releaseThread(false) // backward compatible option
    .build();

var graph = workflow.compile(compileConfig);

An interruption or an exception does not automatically release the thread. Interrupted runs remain available for Human-in-the-Loop continuation, and failed runs remain available for the application's recovery strategy. graph.updateState(...) continues to work in those situations as before.

The practical upgrade check is to examine what your code does after successful completion. Any inspection, replay, resume, or manual-release logic that assumes an active checkpoint set needs attention.

Experimental versioning for released runs

Checkpoint tags gain optional version information. BaseCheckpointSaver.Tag exposes threadId(), version() as an Optional<Integer>, checkpoints(), and lastCheckpoint().

Two lookup methods make released runs accessible:

Optional<Tag> tag(RunnableConfig config, Integer version) throws Exception;

Optional<Tag> lastTag(RunnableConfig config) throws Exception;

The version argument to tag(...) is nullable. lastTag(...) retrieves the latest tag regardless of whether it is versioned.

GraphResult.asLastCheckpointStateData() now obtains its data through Tag.lastCheckpoint(), preserving its observable result. This is useful continuity for callers retrieving the final checkpoint state while the underlying release model evolves.

Parent and Subgraph savers work together

Subgraphs increasingly carry real agent behavior, so their persistence lifecycle must follow the parent execution.

BaseCheckpointSaver now supports putSubGraphSaver(...) to register a subgraph saver and listSubGraphSaver(...) to retrieve registrations associated with a parent configuration. SubCompiledGraphNodeAction registers its saver with the parent; AbstractCheckpointSaver.release(...) then cascades release to the registered subgraph threads.

If you implement a custom saver, AbstractCheckpointSaver provides a reference implementation of the registration mechanism. Review that lifecycle when adapting your implementation, especially if your agents contain nested compiled graphs.

Interruption and Error Metadata

The saver contract also exposes explicit lifecycle hooks:

void registerInterruption(
    RunnableConfig config,
    InterruptionMetadata interruptionMetadata) throws Exception;

Tag releaseCheckpointsOnError(
    RunnableConfig config,
    Throwable error) throws Exception;

These allow a saver to retain interruption information and distinguish an error release from normal completion. InterruptionMetadata now also exposes the interruption reason. That gives later analysis more context about why an execution stopped.

The error-release hook does not change the default described above: an exception leaves the thread available, and the caller chooses its recovery or release strategy.

Note: 👀

To avoid leaving lots of unfinished records in the checkpoint persistence layer, we recommend calling saver.releaseOnError(RunnableConfig, Throwable) within exception handling logic. We are currently considering automating this in a future release by having the graph execution engine invoke the method whenever an error occurs.

The in-memory, file-system, Redis, Postgres, Oracle, MySQL, CockroachDB, DynamoDB, and Hazelcast savers have been aligned with this contract through default implementations. The amount of information retained remains a concern of the specific saver implementation.

SQLite and PostgreSQL V2 savers

Both relational integrations add implementations backed by versioned SQL resources:

Integration	New implementation	Existing implementation
SQLite	`SQLiteSaverV2`	`SQLiteSaver`, for the V1 schema
PostgreSQL	`PostgresSaverV2`	`PostgresSaver`, for the V1 schema

The V2 implementations align with the updated release, error, and interruption contract. The original classes remain available for their V1 schemas. Treat adoption of a V2 saver as a persistence change to review against your existing database.

Other changes worth knowing before upgrading

Explicit Graph Inputs

GraphInput becomes the preferred API for expressing whether an execution is starting or resuming:

graph.stream(GraphInput.args(Map.of("input", "Hello")), config);
graph.stream(GraphInput.resume(), config);
graph.stream(GraphInput.resume(Map.of("approval", "APPROVED")), config);

Use GraphInput.noArgs() for a new execution without arguments. The older map-based stream and invoke overloads remain available in 1.9, but are deprecated for removal. Moving away from a null map makes the intended operation explicit.

Diagnostics and State Serialization

RunnableConfig.nodePath() supplies the current node path for regular nodes and subgraphs, replacing the older partial graphPath() usage. A node can also intentionally interrupt execution by raising GraphInterruptException.

State cloning remains enabled by default. RunnableConfig.builder().disableCloneState() opts out, which can help during early development with objects that are not yet serializable. Exposed state may then reflect later mutations, so consider how consumers retain or modify it.

StateSerializer.declareTransientAttributes(...) excludes named attributes from serialized data and restores them from the same serializer's in-memory transient storage. Those values do not survive a restart, a different process, or a new serializer instance. Keep anything needed for durable resume in persisted state. GsonStateSerializer is deprecated for removal; move toward Jackson-based serialization.

Agent infrastructure

The experimental core skills API introduces SkillSource, SkillPath, and SkillParser.

Spring AI builds on this with SubAgent, CustomSubAgent, SkilledReactSubAgent, and SkillResource, making compiled agents reusable as tools. I covered the pattern in Skill-Based Sub-Agents with LangGraph4j and Spring AI.

LangGraph4j Studio gets a visual refresh

Studio 1.8

Studio 1.9

In the new Studio the active node receives a blue highlight and a loading indicator, making execution easier to follow at a glance.

LangGraph4j DSL

The release also adds langgraph4j-dsl, whose JsonDslGenerator exports graphs, including parallel nodes and nested subgraphs, as JSON:

String json = compiledGraph.reduce(new JsonDslGenerator<>());

This representation has been used for the Studio refactoring toward React Flow library.

What's Next for LangGraph4j

The next phase of development will focus on three areas. These are roadmap priorities, with scope and delivery to evolve as implementation progresses.

Persist more of the execution context

I want to improve graph execution-context persistence so that it supports monitoring and post-processing analysis more effectively. The checkpoint lifecycle work in 1.9 is a step in that direction: understanding completed, interrupted, and failed runs is essential when agentic workflows move into production.

The aim is to make retained execution information more useful for understanding behavior and for subsequent analysis and processing.

Redesign parallel node execution

Another priority is to redesign the parallel-node implementation, removing current limitations and improving efficiency. As workflows grow, parallel branches need to become easier to compose and manage.

This is upcoming work. Applications using 1.9 should continue to account for the currently documented parallel-execution limitations.

Improve built-in Agents for Spring AI and LangChain4j

Finally, I want to enhance the built-in agent infrastructure for both Spring AI and LangChain4j. The skills and sub-agent work demonstrates how much can be built on top of graph execution; the goal is to make those building blocks more useful and easier to compose across both integrations.

Try the Release

Update the LangGraph4j BOM to 1.9.0 to keep module versions aligned:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.bsc.langgraph4j</groupId>
      <artifactId>langgraph4j-bom</artifactId>
      <version>1.9.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Conclusions

For an existing 1.8.x application, start with the migration guide, review thread release and stored checkpoint compatibility, then exercise interruption, resume, and completion paths with your chosen saver.

I'm interested in feedback from real workflows, especially around custom progress output and checkpoint management. Check out LangGraph4j, try the release, and let me know what would help your next agentic application. If you find the project useful, leave a star ⭐️ and... happy AI coding! 👋

Managed Agent Infrastructure: What Changes When the Runtime Is Handled For You

Fri, 14 Aug 2026 22:11:13 +0000

Managed agent frameworks address a specific infrastructure challenge: streams that don't timeout, memory that persists across sessions, and auth that prevents data leakage. Building one that handles all three in production is where most teams stall.

The Shift From DIY Agent Plumbing to Managed Runtimes

When you build an AI agent yourself, roughly half the engineering effort has nothing to do with the agent's actual behavior - it goes into scaffolding: wiring up a streaming layer, managing sandboxed code execution (an isolated environment so agent-generated code can't touch your production systems), persisting memory between conversations, and handling authentication so users only see what they're supposed to see.

According to LangChain's framework documentation, managed agent frameworks provide infrastructure as defaults rather than something you assemble. The agent runtime is the execution environment that actually runs your agent loop, handles retries, streams tokens back to the user, and checkpoints state. This comes pre-built. You define the tools, the model, and the logic. The platform handles how that runs reliably at scale.

For teams where the AI engineer and the product builder are the same person, this shift is particularly valuable. If you're a forward deployed engineer shipping an internal tool, or a data scientist turning a prototype into something real users touch, you don't have time to maintain bespoke streaming infrastructure.

Real Example

Here's roughly what the shift looks like in practice. A DIY setup for a simple memory-enabled agent in LangGraph might require you to manually wire a checkpointer (the component that saves conversation state between turns):

from langgraph.checkpoint.memory import MemorySaver
from langgraph.prebuilt import create_react_agent

checkpointer = MemorySaver()
agent = create_react_agent(
 model=llm,
 tools=tools,
 checkpointer=checkpointer
)
config = {"configurable": {"thread_id": "user-session-42"}}
result = agent.invoke({"messages": [...]}, config=config)

In a managed setup, the checkpointer, the streaming layer, and the sandbox are provisioned for you - you hand off tools and model, declare the agent, and deploy. The operational surface you maintain shrinks considerably. You still own the agent's behavior; you just don't own the plumbing it runs through.

The tradeoff is real: managed runtimes introduce platform dependency and can constrain how you debug or customize the execution loop. For teams that need full control over retry logic or want to inspect every state transition, self-hosted is still the right call.

Key Takeaways

Agent scaffolding (streaming, memory, sandboxing, auth) consumes a disproportionate share of build time - managed runtimes specifically target that overhead.
The agent's behavior stays yours - tools, prompts, model choice, and logic remain in your control; the runtime layer is what's abstracted.
Managed isn't always better - teams needing deep observability or custom execution control may find self-hosted LangGraph or similar frameworks give more flexibility.

If you've shipped an agent to production, which layer gave you the most unexpected trouble - memory, streaming, auth, or something else entirely?

Sources referenced: LangChain Blog - "Why managed agents are the next big thing in agent building"

From RAG to Agentic RAG. How I Added LangGraph to My Local

Thu, 30 Jul 2026 19:49:17 +0000

In my previous article, I built a fully local RAG assistant Ollama, ChromaDB, LangChain, all running in Docker. It answered technical support questions by searching through documentation and citing sources. It worked.

But after using it for a while, I noticed something uncomfortable: it treated every question the same way.

Ask it "how to close monthly payroll?" it searches the docs. Fine.
Ask it "the server crashes at startup" it also searches the docs. Less fine.
Ask it something completely outside the documentation it searches the docs. Useless.

A real support technician doesn't do that. They first assess the situation, then decide what to do: look it up, run a diagnosis, or escalate to a human. My RAG had no such judgment.

That's what this article is about how I evolved the system into an Agentic AI architecture using LangGraph, where the assistant first decides which strategy to use, then acts accordingly.

The Core Limitation of Classic RAG

Classic RAG is a linear pipeline. Every query follows the exact same path:

Question → Embed → Retrieve → Prompt → LLM → Answer

No branching. No decision-making. No memory between steps.

This works perfectly for procedural questions where the answer lives in the docs. But technical support involves at least three distinct scenarios:

Scenario	Example	Best strategy
Procedural question	"How do I create an account?"	Search documentation
Known error code	"ERR-COMP-001 appears"	Lookup error database
Unknown incident	"Server crashes, no idea why"	Diagnose + escalate if needed

A single RAG pipeline handles the first case well and the other two poorly. The solution is to add a layer of reasoning before retrieval.

What Agentic AI Adds

The shift from RAG to Agentic AI comes down to one thing: the system plans before it acts.

Instead of one fixed pipeline, you have:

Question
    ↓
Classifier (what kind of question is this?)
    ↓
    ├── Procedural → RAG Agent (search docs)
    ├── Error code → Diagnostic Agent (lookup + LLM analysis)
    └── Complex    → Diagnostic Agent → Escalation Agent (generate ticket)

Each branch is an agent; an autonomous function that receives a shared state, does its job, updates the state, and returns it. LangGraph connects these agents into a directed graph with conditional routing between them.

The Shared State Foundation of LangGraph

Before building agents, you define the state they all share. Think of it as a case file that every agent reads and updates:

# src/agents/state.py

from typing import TypedDict, List, Annotated, Sequence
from langchain_core.messages import BaseMessage
import operator

class AgentState(TypedDict):
    messages: Annotated[Sequence[BaseMessage], operator.add]
    question: str          # Original question never changes
    plan: List[str]
    current_step: int
    past_steps: List[str]  # Audit trail of what each agent did
    response: str
    next_agent: str
    niveau_support: int    # 1=resolved, 2=partial, 3=needs human

The niveau_support field is key it's how agents communicate urgency to each other. If an agent sets it to 3, the orchestrator escalates automatically.

Annotated[Sequence[BaseMessage], operator.add] means messages accumulate — each agent appends rather than overwrites. You get a full conversation trace at the end.

Agent 1 The RAG Agent

The pipeline from my previous article, repackaged as an autonomous agent:

# src/agents/rag_agent.py

def rag_agent(state: AgentState) -> AgentState:
    """Searches the documentation and generates a grounded answer."""

    docs = vectorstore.similarity_search(state["question"], k=3)

    if not docs:
        return {
            **state,
            "messages": list(state["messages"]) + [
                AIMessage(content="[RAG] No relevant document found.")
            ],
            "past_steps": state["past_steps"] + ["RAG: no results"],
            "niveau_support": 3   # Nothing found — escalate
        }

    context = "\n\n".join([
        f"[Source: {d.metadata.get('module', '?')}]\n{d.page_content}"
        for d in docs
    ])

    answer = llm.invoke(
        f"Answer from context only.\nContext: {context}\nQuestion: {state['question']}"
    )

    return {
        **state,
        "messages": list(state["messages"]) + [AIMessage(content=f"[RAG] {answer}")],
        "past_steps": state["past_steps"] + ["RAG: documentation search completed"],
        "niveau_support": 1
    }

The critical difference from the classic RAG: the agent returns an updated state, not just an answer. Every agent follows this same pattern — receive state, do work, return updated state.

Agent 2 The Diagnostic Agent

Handles error codes and technical incidents with two modes:

# src/agents/diagnostic_agent.py

KNOWN_ERRORS = {
    "ERR-COMP-001": {
        "description": "Debit/credit imbalance",
        "solution": "Check Total Debit = Total Credit. Go to Accounting > Journal Entry."
    },
    "ERR-PAIE-042": {
        "description": "Payroll line missing rate",
        "solution": "Go to Settings > Payroll Lines, fill in the missing rate."
    },
}

def diagnostic_agent(state: AgentState) -> AgentState:
    question = state["question"]
    found_codes = [code for code in KNOWN_ERRORS if code.lower() in question.lower()]

    if found_codes:
        # Mode 1 : known error: instant answer, no LLM call needed
        responses = [
            f"Code {code}: {KNOWN_ERRORS[code]['description']}\n"
            f"Solution: {KNOWN_ERRORS[code]['solution']}"
            for code in found_codes
        ]
        return {
            **state,
            "messages": list(state["messages"]) + [
                AIMessage(content=f"[Diagnostic] {chr(10).join(responses)}")
            ],
            "past_steps": state["past_steps"] + ["Diagnostic: known error, resolved"],
            "niveau_support": 1
        }

    # Mode 2 : unknown error: LLM analysis
    answer = llm.invoke(
        f"Analyze this technical issue. If unsure, say so.\nIssue: {question}"
    )
    return {
        **state,
        "messages": list(state["messages"]) + [AIMessage(content=f"[Diagnostic] {answer}")],
        "past_steps": state["past_steps"] + ["Diagnostic: LLM analysis, uncertain"],
        "niveau_support": 2
    }

The known error lookup runs in milliseconds without touching the LLM; a meaningful performance gain for common incidents.

Agent 3 : The Escalation Agent

Called only when niveau_support >= 3. It doesn't try to answer; it generates a structured ticket with the full diagnostic history already attached:

# src/agents/escalade_agent.py

def escalade_agent(state: AgentState) -> AgentState:
    ticket = f"""
=== SUPPORT TICKET — LEVEL 3 ===
Date     : {datetime.now().strftime("%Y-%m-%d %H:%M")}
Priority : HIGH

ISSUE:
{state["question"]}

DIAGNOSTIC STEPS ALREADY TAKEN:
{chr(10).join(state["past_steps"])}

CONCLUSION: Human intervention required.
================================"""

    return {
        **state,
        "messages": list(state["messages"]) + [AIMessage(content=ticket)],
        "past_steps": state["past_steps"] + ["Escalation: level 3 ticket generated"],
        "niveau_support": 3
    }

This is what separates a real support system from a chatbot; it knows when to stop and hand off, with context already attached so the user doesn't have to repeat themselves.

The Orchestrator : Wiring It All Together

# src/agents/orchestrator.py

def classifier(state: AgentState) -> str:
    question = state["question"].lower()

    if re.search(r'err-\w+-\d+', question):
        return "diagnostic"

    problem_words = ["error", "crash", "bug", "not working", "fails", "problem"]
    if any(word in question for word in problem_words):
        return "diagnostic"

    return "rag"  # Default for procedural questions


def should_escalate(state: AgentState) -> str:
    return "escalade" if state.get("niveau_support", 1) >= 3 else END


def create_orchestrator():
    workflow = StateGraph(AgentState)

    workflow.add_node("rag_agent", rag_agent)
    workflow.add_node("diagnostic_agent", diagnostic_agent)
    workflow.add_node("escalade_agent", escalade_agent)

    workflow.set_conditional_entry_point(
        classifier,
        {"rag": "rag_agent", "diagnostic": "diagnostic_agent"}
    )

    workflow.add_edge("rag_agent", END)
    workflow.add_conditional_edges(
        "diagnostic_agent",
        should_escalate,
        {"escalade": "escalade_agent", END: END}
    )
    workflow.add_edge("escalade_agent", END)

    return workflow.compile()

Seeing It In Action

Three questions, three different paths:

Procedural question

Input  : "How to close monthly payroll?"
Path   : classifier → rag_agent → END
Output : agents_used: ["RAG: documentation search completed"]
         niveau: 1

Known error code

Input  : "ERR-COMP-001 appears on journal validation"
Path   : classifier → diagnostic_agent → END
Output : agents_used: ["Diagnostic: known error, resolved"]
         niveau: 1  (no LLM call, instant response)

Unknown incident

Input  : "Server crashes at startup, no idea why"
Path   : classifier → diagnostic_agent → escalade_agent → END
Output : agents_used: ["Diagnostic: LLM analysis", "Escalation: ticket generated"]
         niveau: 3

What I Would Do Differently

The classifier is fragile. Keyword matching breaks on edge cases "I don't have a problem, just need help configuring payroll" contains "problem" and routes to diagnostic instead of RAG. A small LLM classification call, or a fine-tuned classifier, would be more robust.

Agents reload the embedding model on every call. Instantiating HuggingFaceEmbeddings on each agent invocation adds unnecessary overhead. Initialize once at startup and pass the instance through the state.

The escalation threshold is hardcoded. A feedback loop where user ratings influence the confidence threshold over time would make this genuinely adaptive that's what reinforcement learning adds in the next phase of this project.

RAG vs Agentic AI : When to Use Which

Criteria	Classic RAG	Agentic AI
Best for	Uniform question types	Mixed question types
Complexity	Low	Medium-High
Latency	One LLM call	One or more LLM calls
Flexibility	Fixed pipeline	Extensible — add agents without rewriting
When to choose	Start here	When RAG fails on edge cases

My honest take: start with RAG. Get it working, understand its failure modes, then add agents where you actually need branching logic. Adding LangGraph too early is over-engineering.

Key Takeaways

Design your AgentState first. It's the contract between all components. Changing it mid-project means updating every agent get it right upfront.

Agents should do one thing. The moment an agent tries to retrieve and diagnose, it becomes untestable and unroutable. Keep them single-responsibility.

past_steps is your audit trail. When a ticket escalates to a human, they see exactly what the system tried. Small design choice, professional result.

The classifier is your most important and most brittle piece of code. Test it with as many edge cases as you can before trusting it.

These two articles cover Phases 1 and 2 of my internship project. The logical next step is a reinforcement learning feedback loop; user ratings adjusting the classifier's routing decisions over time; which I may write about once it's implemented and tested.

I'm a Master's student in AI & Big Data, building these systems with open-source tools and no GPU. If you're working on multi-agent systems or ran into similar design decisions; especially around the classifier or state design; I'd love to hear your approach in the comments.

The cost of a serverless agent isn't the Lambda. It's the loop.

Fri, 17 Jul 2026 05:46:41 +0000

When I moved Wingman onto Lambda, I assumed the thing I'd have to watch was compute duration. That's the serverless mental model everyone starts with: you pay per millisecond, so keep the function fast and cheap.

That model was wrong for an agent. The Lambda bill turned out to be the boring, predictable part. The two things that actually move cost and latency are the cold start and the retry loop, and neither of them shows up where you'd look first.

Here's what the code taught me.

The setup: what Wingman actually is

Wingman is a LangGraph worker-evaluator agent. A worker node (running gpt-4o-mini) does the task. An evaluator node (also gpt-4o-mini) grades the answer against a success criteria. If the criteria isn't met, control routes back to the worker for another attempt, up to five times.

It runs as a container image on Lambda, fronted by API Gateway and CloudFront, with session history in DynamoDB. The compute is stateless: every request rebuilds the conversation from DynamoDB, so there's no persistent checkpointer to worry about across cold and warm starts. I wrote about that state design in an earlier post. This one is about what it costs to run.

Cold start #1: the image

A Lambda container cold start begins by unpacking the image. Wingman's image carries langchain, langgraph, langchain-community, langchain-experimental, boto3, plus reportlab and sendgrid for the agent's tools. That's not a small pile of Python.

The one decision I'm glad I made is in the Dockerfile. The project has a local dependency group with playwright, gradio, and uvicorn, and a dev group on top of that. None of those belong in Lambda. So the build exports only the core deps and drops the rest:

RUN uv export \
      --no-dev \
      --no-group dev \
      --no-group local \
      --format requirements-txt \
      -o /tmp/requirements.txt \
    && pip install -r /tmp/requirements.txt --no-cache-dir \
      --target "${LAMBDA_TASK_ROOT}"

Playwright alone would have pulled a browser runtime into the image. Keeping it in the local group means it exists for running the agent on my laptop and never ships to Lambda. Smaller image, faster unpack, faster cold start. This is the kind of thing you only notice you did right when you compare it to the version where you didn't.

Cold start #2: the init you pay for on import

The bigger cold-start cost is in the handler, and it's easy to miss because it happens at import time:

# backend/main.py — runs once per new execution environment
wingman = Wingman()
wingman.setup()

_dynamodb = boto3.resource("dynamodb", region_name=os.getenv("AWS_REGION", "us-east-1"))
_table = _dynamodb.Table(_dynamodb_table_name)

setup() isn't cheap. It builds the tool list, constructs two ChatOpenAI clients, binds tools to the worker, wraps the evaluator in structured output, and compiles the LangGraph:

def setup(self):
    self.tools = get_tools()
    worker_llm = ChatOpenAI(model="gpt-4o-mini")
    self.worker_llm_with_tools = worker_llm.bind_tools(self.tools)
    evaluator_llm = ChatOpenAI(model="gpt-4o-mini")
    self.evaluator_llm_with_output = evaluator_llm.with_structured_output(EvaluatorOutput)
    self._build_graph()

Putting this at module level is the right call. It runs once when a new execution environment spins up, then every warm invocation reuses the same compiled graph and the same clients. If I'd built the graph inside the request handler instead, I'd be paying graph compilation on every single /api/chat call, warm or not. Module-level init trades a heavier cold start for near-zero per-request setup, which is the trade you want when warm invocations vastly outnumber cold ones.

The lever for cold-start speed is memory:

resource "aws_lambda_function" "backend" {
  timeout     = 300   # 5 minutes
  memory_size = 1024
}

On Lambda, CPU scales with memory. At 1024MB you get roughly a full vCPU, so bumping memory doesn't just buy RAM headroom, it makes that import-time init finish faster. Memory is a latency knob here, not only a capacity one.

The real cost: the retry loop, not the duration

Now the part that surprised me.

Look at the timeout: 300 seconds. Lambda's max is 900. That's a generous ceiling for a function whose real job takes a few seconds. So why so high? Because the retry loop can run the worker up to five times, and each turn is a full round of model calls plus whatever tools the worker decides to invoke.

Here's the accounting that matters. turn_count only increments in the evaluator:

def evaluator(self, state: State) -> Dict[str, Any]:
    ...
    return {
        ...
        "turn_count": state.get("turn_count", 0) + 1,
    }

def route_based_on_evaluation(self, state: State) -> str:
    if state["success_criteria_met"] or state["user_input_needed"]:
        return "END"
    if state.get("turn_count", 0) >= self.MAX_TURNS:
        return "END"
    return "worker"

So a single /api/chat request is not one model call. In the worst case it's five worker calls, five evaluator calls, and every tool call the worker makes in between. The Lambda duration for that is trivial to reason about and cheap. The token cost is the thing that swings, and it swings by a factor of five depending on how many times the evaluator sends the work back.

That reframed the whole cost model for me. Moving to stateless compute on Lambda made the infrastructure cost flat and predictable. What it did was push all the cost variance out of the infrastructure and into the model loop. The bill I have to watch isn't Lambda-seconds. It's tokens times retries.

The 29-second cliff

One more thing the config quietly documents. API Gateway has a hard 29-second integration timeout. A five-turn loop with tool calls can blow past that even though the Lambda itself is happily configured for 300 seconds. So the request dies at the gateway while the function is still working.

The fix is in the Terraform: a Lambda Function URL that bypasses API Gateway entirely for long tasks.

resource "aws_lambda_function_url" "backend" {
  function_name      = aws_lambda_function.backend.function_name
  authorization_type = "NONE"
}

Short chats go through CloudFront and API Gateway. Long agent runs go straight to the Function URL and get the full timeout budget. It works, but authorization_type = "NONE" and allow_origins = ["*"] are both flagged in the code as things to tighten after the first deploy, and they're still open. An unauthenticated public URL that invokes a model loop is a bill waiting to happen if someone finds it. That one's on my list.

What I'd do differently

The retry loop has no token budget. It caps at five turns, but a "turn" can be an expensive worker call with several tool invocations, and nothing stops a run from burning far more tokens than the answer is worth. A turn cap is a crude proxy for a cost cap. If I rebuilt this, the evaluator would carry a token budget alongside turn_count, and the loop would stop on whichever it hit first.

Takeaway

For a serverless agent, Lambda duration is the cheap, predictable line on the bill. The cost that actually moves lives in the cold-start init you pay on import and the retry loop you pay per request, so budget those, not the milliseconds.

Building a worker-evaluator retry loop in LangGraph (and where it bites you on Lambda)

Fri, 17 Jul 2026 05:44:06 +0000

Most agent demos stop at "the model called a tool and gave an answer." That answer is often wrong, and nothing in the loop notices. I wanted an agent that checks its own work against a success criteria before returning, and retries when it falls short. That is the whole idea behind Wingman: a worker that does the task, and a separate evaluator that decides whether the task is actually done.

Here is how the loop is built, the one decision that shaped everything, and the part that broke once it ran on Lambda behind API Gateway.

The problem

A single-pass agent has no idea when it failed. It calls a tool, produces text, and returns. If the output misses the point, the user finds out, not the system.

I wanted a gate. Something that reads the assistant's last response, compares it against a stated success criteria, and either accepts it or sends the worker back to try again with feedback. Two roles, not one. The worker produces. The evaluator judges.

The catch is that a retry loop with no ceiling is a bill waiting to happen. So the loop needs a hard stop, and it needs to know the difference between "the assistant needs to try again" and "the assistant is stuck and should ask the user."

The approach and the key decision

The graph has three nodes: worker, tools, and evaluator. The state carries the conversation, the success criteria, the latest feedback, two boolean flags, and a turn counter.

class State(TypedDict):
    messages: Annotated[List[Any], add_messages]
    success_criteria: str
    feedback_on_work: Optional[str]
    success_criteria_met: bool
    user_input_needed: bool
    turn_count: int

The worker runs first. If its last message contains tool calls, the router sends it to the tool node and back. If not, it goes to the evaluator.

def worker_router(self, state: State) -> str:
    last = state["messages"][-1]
    return "tools" if (hasattr(last, "tool_calls") and last.tool_calls) else "evaluator"

The key decision is making the evaluator a structured-output call, not a free-text one. It returns a typed object, so the routing logic reads clean booleans instead of parsing prose.

class EvaluatorOutput(BaseModel):
    feedback: str = Field(description="Feedback on the assistant's response")
    success_criteria_met: bool = Field(description="Whether the success criteria have been met")
    user_input_needed: bool = Field(
        description="True if more input is needed from the user, or the assistant is stuck"
    )

Both the worker and the evaluator run on gpt-4o-mini. The worker is bound to tools. The evaluator is bound to the schema above with .with_structured_output.

The routing after evaluation is where the loop's whole personality lives:

def route_based_on_evaluation(self, state: State) -> str:
    if state["success_criteria_met"] or state["user_input_needed"]:
        return "END"
    if state.get("turn_count", 0) >= self.MAX_TURNS:
        return "END"
    return "worker"

Three ways out. The criteria is met. The assistant is stuck and needs the user. Or the loop has spent its five turns. Otherwise it loops back to the worker, and this time the worker gets the feedback baked into its system prompt:

if state.get("feedback_on_work"):
    system_message += f"""
Previously you thought you completed the assignment, but your reply was rejected because the success criteria was not met.
Feedback:
{state["feedback_on_work"]}
Please continue the assignment..."""

That feedback injection is what makes the retry worth anything. The worker does not just try again blind. It tries again knowing why the last attempt was rejected.

One design detail I care about: the evaluator is told not to confuse "not done yet" with "stuck." A failed criteria means retry, not stop.

For user_input_needed: set True ONLY if the assistant explicitly asked the user a question or stated it cannot proceed.
Do NOT set user_input_needed=True just because the criteria was not met — the assistant should simply retry.

Without that line, the evaluator bails to the user the first time the worker misses, and the retry loop never actually retries.

What broke, and what I changed

The loop is stateless by design. Wingman reconstructs the full conversation from DynamoDB on every request and runs a fresh graph invocation with a throwaway thread ID, so there is no in-memory state to lose between Lambda cold starts. That part held up.

The timeout did not.

A worker-evaluator loop is slow. Five turns, each turn a worker call plus tool calls plus an evaluator call, adds up to real wall-clock time. I set the Lambda timeout to 300 seconds to give the loop room:

# LangGraph agent loops can take a while.
timeout     = 300  # 5 minutes (Lambda max is 15 min)
memory_size = 1024

That number is a lie if the request comes through API Gateway. The HTTP API has a hard 29-second integration cap that Lambda's 300 seconds cannot override. So a long agent run would keep working inside Lambda while the gateway had already given up on the caller. The function billed for the full run. The user got nothing.

The fix was to stop routing long tasks through the gateway at all. I added a Lambda Function URL, which is a direct HTTPS endpoint with no 29-second ceiling:

# Lambda function URL — direct HTTPS invoke, bypasses API Gateway 29s timeout.
resource "aws_lambda_function_url" "backend" {
  function_name      = aws_lambda_function.backend.function_name
  authorization_type = "NONE"
}

Now the gateway stays for the fast endpoints, and the agent loop can run against the Function URL without a proxy in the middle deciding it took too long.

There is a second sharp edge I have not fully closed. turn_count only increments inside the evaluator node:

"turn_count": state.get("turn_count", 0) + 1,

But the worker can loop worker -> tools -> worker -> tools several times before it ever hands off to the evaluator, and none of those hops count against MAX_TURNS. So the five-turn cap is really "five evaluator turns," not "five model calls." A task that keeps calling tools without producing a final answer can burn far more time than the cap implies. The honest fix is to count worker iterations too, or add a wall-clock budget inside the run, rather than trusting the evaluator counter to bound cost. That one is still on my list.

Takeaway

An evaluator that gates the worker's output is the cheap half of a self-correcting agent. The expensive half is making the retry ceiling account for every model call, not just the ones you happened to count.

Memory in LLM Agents, Explained: From Stateless Calls to Long-Term Memory

Sun, 21 Jun 2026 08:23:59 +0000

Introduction

Most explanations of agent memory start at the wrong altitude. They jump straight to "use a vector database" without explaining what problem that solves, or why a vector database is sometimes the wrong tool entirely.

This post walks through memory the way I built it: as a progression of seven small, runnable scripts, starting from a completely stateless LLM call and ending at a two-layer system with separate short-term and long-term memory. Each step introduces exactly one new concept. By the end, the line between short-term and long-term memory stops being fuzzy and becomes a concrete architectural decision: what is scoped to a thread, and what is scoped to a user.

All code referenced here is in the open-source repo at the end of this post. Every script is self-contained and runnable on its own.

1. Stateless: the baseline problem

The simplest possible setup is a single call to an LLM with no history attached. Tell it your name, ask it your name in the next message, and it has no idea. Every call starts from zero.

This is not a bug. It is the literal absence of memory, and it is the baseline every other pattern in this guide is solving for.

2. Short-term memory in RAM

The most direct fix: keep the full conversation in a Python list, and send that list with every call to the model.

history.append({"role": "user", "content": user_input})
response = llm.invoke(history)
history.append({"role": "assistant", "content": response.content})

Now the model can recall earlier turns, because it is literally being shown them every time. The catch: this history lives in process memory. Restart the script and it is gone. This pattern is memory in the loosest sense, useful for a single uninterrupted session, useless across sessions.

3. Persistence and threads

The next step is durability. Write the conversation history to SQLite after every turn, and reload it on startup.

This alone solves the restart problem. The second piece is the thread_id: rather than one global history, each thread_id is its own isolated conversation. Multiple conversations can live in the same store without bleeding into each other, and you can switch between them on demand.

At this point memory has gone from "exists only while the process is running" to "exists independently of the process, scoped to a conversation."

4. Sliding window

A real conversation grows without bound, and sending the full history on every call gets expensive fast, both in tokens and in latency. The sliding window pattern fixes the cost problem directly: a @before_model middleware hook trims the stored history down to the last N messages right before every call to the model.

@before_model
def trim_to_window(messages):
    return messages[-WINDOW:]

Token cost is now bounded, regardless of how long the conversation runs. The tradeoff is blunt: anything said more than N turns ago is permanently inaccessible to the model. If the user mentioned an important constraint 10 turns ago and the window is 4, that constraint is gone.

This is the right tool when recent context is what matters and the conversation is expected to be short or self-contained.

5. Summarization

Summarization is the more graceful version of the same idea. Instead of discarding old messages outright, a @before_model middleware hook compresses them into a short summary once the stored message count crosses a threshold, while keeping the most recent few messages verbatim alongside that summary.

@before_model
def summarize_if_needed(messages):
    if len(messages) > SUMMARIZE_AFTER:
        old, recent = messages[:-KEEP_RECENT], messages[-KEEP_RECENT:]
        summary = llm.invoke([SUMMARY_PROMPT, *old])
        return [summary_message(summary), *recent]
    return messages

This retains the gist of everything that happened earlier in the conversation, at the cost of losing fine-grained detail. A sliding window would have kept the literal text of the last N turns and nothing before that; summarization keeps a compressed signal of everything before that, plus the literal text of the most recent few turns.

Neither pattern is strictly better. Sliding window is cheaper and simpler. Summarization preserves more of the conversation's shape, at the cost of an extra LLM call to generate the summary.

6. Long-term memory: the user profile

Every pattern so far is still short-term memory. It lives within one conversation thread and, even with persistence, none of it is shared if the user starts a different thread.

Long-term memory is a different scope entirely: durable facts about a user that should be available no matter which conversation they are currently having. The pattern here uses a second LLM call after every turn to extract durable facts (name, job, stated preferences) and write them to a key-value store (LangGraph's SqliteStore):

facts = extract_facts_llm.invoke([extraction_prompt, *recent_turns])
store.put(namespace=("user", user_id), key="profile", value=facts)

On the next session, even a completely separate thread, those facts are loaded from the store and injected into the system prompt. This is the first pattern in the series where memory survives across sessions that have no shared conversation history at all.

7. Combining short-term and long-term memory

The full architecture uses both layers together, each with a different scope:

Short-term memory, via SqliteSaver: conversation history, scoped to a thread_id. Restored automatically on reconnect to the same thread.
Long-term memory, via SqliteStore: durable behavioral rules, scoped to a user_id. The agent saves rules autonomously via a save_rule tool, and those rules apply to every future session for that user, regardless of thread.

The clarifying detail: open a new thread for the same user, and the conversation history resets, but the rules that user taught the agent in a previous thread are already active. Switch to a different user entirely, and that user inherits none of the first user's rules. Each user's long-term memory is fully isolated.

This is the insight that took the fuzziness out of "short-term vs long-term memory" for me. They are not the same kind of memory at different durations. They are scoped along different dimensions: STM resets per thread, LTM persists per user across every thread that user ever opens.

Choosing a pattern

Situation	Pattern
Single-session prototype, no persistence needed	STM in RAM
Conversation needs to survive restarts	STM plus SQLite with threads
Long conversations, recent context matters most, cost-sensitive	Sliding window
Long conversations, full context shape matters	Summarization
Facts about the user should persist across sessions	Long-term memory, user profile
Production agent with both ongoing conversations and persistent user behavior	STM and LTM combined

Conclusion

Memory in LLM applications is not one feature, it is a set of independent tradeoffs: durability versus simplicity, recall versus token cost, thread-scoped versus user-scoped. Building each pattern in isolation, in order, makes those tradeoffs visible rather than hidden behind a single "add memory" abstraction.

Full source, all 7 scripts, self-contained and runnable: https://github.com/f2015537/llm-memory-patterns

Skill-Based Sub-Agents with LangGraph4j and Spring AI

Thu, 30 Apr 2026 21:42:33 +0000

Combine Tools & Skill to create Agent

Modern agentic systems usually expose two strong but separate abstractions:

Tools, which give an LLM executable capabilities
Skills, which package reusable instructions for a narrow task

This article shows how to combine both ideas in [LangGraph4j] so that a skill is not just prompt text, but a fully operational sub-agent exposed as a tool. The result is a practical pattern for building modular multi-agent systems in Java with [Spring AI] and [LangGraph4j], while keeping each agent focused, reusable, and low in context cost.

Starting from a standard ReACT agent, the approach implemented is quite simple.

Load a SKILL.md file
Read its front matter to define the tool contract ( name and argument description)
Use the skill body as the sub-agent system prompt
Restrict the sub-agent to only the tools declared by the skill
Register the sub-agent in a parent ReAct agent as if it were a normal tool

In other words, a skill becomes an executable tool-backed agent.

Why This Pattern Matters

The standard ReAct pattern already gives an LLM a strong execution model: reason, choose a tool, inspect the result, and continue until the task is complete. That model is useful because the LLM does not need to commit to a fixed plan up front. It can adapt after every tool call.

The next step is to let a tool itself be implemented by another agent.

Once that happens, the parent agent is no longer forced to know every operational detail. It only needs to know:

which specialized capability exists
when to invoke it
what input to pass

The specialized logic is delegated to a narrower agent with its own instructions and its own restricted toolset. This gives us a clean sub-agent architecture without inventing a new orchestration primitive. We reuse the same tool-calling semantics that ReAct agents already understand well.

Why Reusing Tool Semantics Is Powerful

One of the most valuable properties of LLM-based agents is their ability to build and revise a tool execution plan while reasoning over intermediate results. ReAct made this practical by combining thought, action, and observation in a loop.

When a tool call can transparently invoke another agent, we preserve that planning model instead of replacing it.

The parent agent can:

decide that a specialized capability is needed
provide only the task-relevant context
wait for a result exactly as it would for a standard tool

The child agent can:

run its own reasoning loop
invoke its own allowed tools
return a compact outcome to the parent

This gives hierarchical coordination without exposing the full inner workflow to the parent prompt.

Minimizing Context and Token Usage

This pattern also improves context discipline.

Without sub-agents, a single generalist agent often needs:

every tool definition
all operating instructions for every domain
every procedural rule in one large system prompt

That quickly becomes expensive and unreliable.

With skill-based sub-agents:

the parent only sees high-level capabilities
each child sees only the instructions relevant to its task
each child receives only the tools it is allowed to use

This narrower context has practical benefits:

fewer tokens sent to the model
lower prompt interference between unrelated domains
clearer tool choice
better modularity as the system grows

The allowed-tools skill property plays an important role here anabling tool filtering. It turns tool access into an explicit design decision instead of an accidental side effect of what happened to be registered globally.

Reference implementation

I've started implementation in the new [LangGraph4j] release stream 1.9 currently in active development. The implementation is centered on [`SkilledReactSubAgent`], which turns a markdown skill into both:

a compiled LangGraph4j sub-graph
a Spring AI tool (i.e. ToolCallback)

That means the same component can participate in the graph as an agent and present itself to the parent LLM as a callable tool.

In particular the flow is:

Parse the SKILL.md
Extract name and description from front matter
Optionally filter tools using allowed-tools
Use the markdown body as the sub-agent instructions
Compile the agent
Wrap it into a function tool whose input is a single context field

The important design choice is that the tool definition is derived from the skill metadata, while the agent behavior is derived from the skill content.

Skill Format

The supported skill format is

name:
description: |

    <multi line description>

allowed-tools:

### From Skill to Tool-Backed Agent The conversion happens in `SkilledReactSubAgent.Builder#build(...)`. This is the crucial bridge where: * the skill metadata defines the external API * the skill body defines the internal operating behavior * the allowed tools define the sub-agent execution boundary The resulting agent is then wrapped as a Spring AI function tool. > From the parent agent point of view, this is just another tool. Internally, it is a complete ReAct-capable agent workflow. ## Examples ### Purchase Assistant Agent This is an agent that allow to select a produdct from a reference marketplace and purchase it. Below the simplified component diagram. ![](https://bsorrentino.github.io/bsorrentino/assets/langgraph-java/sub-agents/PurchaseAssistantAgent1.png) ### Marketplace Skill

---
name: agent-marketplace
description: |
    marketplace agent, ask for information about products. This is agent that provides the information on the product marketplace.
    required data are:
    * product name
allowed-tools: searchByProduct
---

## Retrieve product information
We need all information about about the product of interest in particular is mandatory to have
* product name

with such information we have to invoke tool `searchByProduct` and return the result

### Payment Skill

---
name: agent-payment
description: |
    payment agent, this is the agent that provides payment service.
    required data are:
    * Product name
    * Product price
    * Product currency
    * the International Bank Account Number (IBAN) - optional

allowed-tools:
    - submit-payment
    - retrieve-iban
---

## submit a payment for purchasing a specific product
We need all information about purchasing to allow the payment

* Product name
* Product price
* Product price currency
* International Bank Account Number (IBAN)

With such information you must invoke tool `submit-payment` and return the result of the payment transaction plus the IBAN.
If IBAN is not provided invoke tool `retrieve-iban` to retrieve the required IBAN.

Observing the skills above they define three different concerns: * `name`: the tool name exposed to the parent agent * `description`: the tool description shown to the model * `allowed-tools`: the only tools available inside the sub-agent Everything after the front matter becomes the sub-agent system prompt. > Again, this is a strong separation of concerns. The parent agent sees a concise tool contract, while the child agent receives richer operating instructions. ## Finally the code Below a representative code snippet that show how wires two skill-based sub-agents into a parent agent: ```java var subAgentMarketplace = SkilledReactSubAgent.builder() .chatModel(chatModel) .tools(Marketplace.tools()) .build(compileConfig, SkillSource.of(Paths.get("skills/agent-marketplace/"))); var subAgentPayment = SkilledReactSubAgent.builder() .chatModel(chatModel) .tools(Payment.tools()) .build(compileConfig, SkillSource.of(Paths.get("skills/agent-payment/"))); var purchaseAssistantAgent = AgentExecutorEx.builder() .chatModel(chatModel) .tool(subAgentMarketplace) .tool(subAgentPayment) .build(compileConfig); var input = """ search for product 'X'. If found proceed to payment with IBAN US82WEST1234567890123456 to purchase it """; var result = purchaseAssistantAgent.invoke( GraphInput.args( Map.of("messages", new UserMessage(input) )), runnableConfig); ``` This composition is interesting because the purchase agent (the parent) does not need detailed knowledge of: * how product lookup works * how payment submission works * when IBAN retrieval is required Those rules are delegated to the sub agents themselves. That is a simple but good example of behavioral encapsulation. The parent agent only decides to involve marketplace agent to retrieve product and after that to involve payment agent to purchase. This is a very basic but representative example, it is useful because it shows the intended layering: * the user gives a single business request * the parent agent decomposes the work * specialized sub-agents execute their own logic * the final answer comes back through the normal tool-response path > Interesting note that each sub agent, potentially, can relies on different LLM chat model so we can further optimize overall agent execution process Below a more detailed componet diagram generated by [LangGraph4j] itself. ![](https://bsorrentino.github.io/bsorrentino/assets/langgraph-java/sub-agents/PurchaseAssistantAgent.png) From this diagram it is quite clear that the sub agents are nothing but sub-graphs ## Design Advantages This approach has several practical advantages. 1. **Reuse Existing Agent Patterns** There is no need to invent a separate orchestration protocol for sub-agents. A sub-agent is exposed as a tool, so the parent agent can use standard ReAct behavior without extra prompting complexity. 2. **Better Modularity** Each skill owns: * its tool-facing identity * its internal instructions * its allowed tool boundary That makes the system easier to evolve than a single monolithic agent prompt. 3. **Smaller and Cleaner Context** The child agent receives only what it needs. This reduces token usage and lowers the risk that unrelated instructions distort tool choice or task execution. 4. **Safer Capability Scoping** By filtering tools from `allowed-tools`, the framework prevents a child agent from accidentally seeing tools outside its intended domain. 5. **Better Reusability** A skill can be packaged, versioned, and reused across different parent agents. The same marketplace or payment skill can participate in many workflows as long as the tool contract remains stable. ## Tradeoffs and Implementation Considerations This pattern is strong, but it is not free of tradeoffs. 1. **The Parent Must Summarize Well** Because the child receives a single `context` field, the parent agent needs to pass the relevant facts clearly. If critical information is omitted, the child may need extra recovery steps or may fail to act. 2. **Tool Descriptions Matter More** Since the parent sees the child as a tool, the `name` and `description` fields in the skill front matter become part of the routing surface for the LLM. Weak metadata will reduce invocation quality. 3. **Prompt Boundaries Need Discipline** The skill body is effectively the child system prompt. If it is vague, contradictory, or overloaded, the benefits of modularity disappear. Skills need to stay focused. 4. **Observability Is Important** Nested agents can make execution harder to inspect if tracing is weak. LangGraph4j's graph structure and step streaming help here, but production systems should still invest in logging and traceability. ## Why LangGraph4j Fits This Well [LangGraph4j] is a good fit for this pattern because it already models agent workflows as explicit graphs and integrates naturally with Spring AI tool abstractions. That matters for two reasons: * the sub-agent is not just prompt composition; it is a compiled graph with state and execution hooks * the parent integration remains ergonomic because Spring AI already treats capabilities as tools * [LangGraph4j] support interruptions in subgraph enabling **Human In The Loop** also when involving sub agents So the implementation is not forcing two incompatible ideas together. It is aligning: * LangGraph4j graph execution * Spring AI tool calling * markdown-defined skills into a single reusable agent-building pattern. ## Conclusion Turning a skill into a sub-agent is a small change with a large impact. It lets us: * preserve the strengths of the ReAct loop * delegate specialized behavior cleanly * reduce context size * constrain tool access per domain * build multi-agent systems out of reusable skill modules The idea is implemented in a compact and pragmatic way: a `SKILL.md` file becomes the source of both the sub-agent prompt and the tool contract exposed to the parent agent. For Java teams building agentic applications with Spring AI and LangGraph4j, this is a practical pattern for moving from a single large agent toward a modular multi-agent design without abandoning the tool semantics that current LLMs already handle well. The implementation is in progress on release `1.9-SNAPSHOT` in develop branch of [LangGraph4j] repository but the first result are very promising. Hope this could help and encourage usage of [LangGraph4j] for your next Agentic Workflow. Checkout project, try it and let me know your feedback and... happy AI coding! [LangGraph4j]: https://github.com/langgraph4j/langgraph4j [Spring AI]: https://spring.io/projects/spring-ai

LangGraph4j

Sat, 07 Feb 2026 15:07:31 +0000

Why LangGraph4j

A while ago, when I started working in AI, I chose LangChain to build LLM-powered apps because it made introduce LLM into applications practical and effective. Then, in January 2024, LangChain announced LangGraph: a module built on top of LangChain that enables cyclical graphs for real-world agent runtimes. That was the missing piece for building agents in a controlled and safe way.

I work with teams whose IT stack is mainly Java, and they want to experiment with LLMs in production-grade systems. That’s why I built LangGraph4j: to bring the LangGraph approach to the Java ecosystem, leverage my experience with LangChain and LangGraph, and deliver on a simple promise: "learn once, apply everywhere".

LangGraph4j is inspired by [LangGraph] and designed to help Java developers build sophisticated AI-powered agents and applications. It integrates cleanly with established Java LLM frameworks like LangChain4j and Spring AI.

What you get

Controlled execution with explicit graph paths and safe loops
Parity with LangGraph concepts, without leaving the Java ecosystem
Drop-in integration with Java LLM frameworks and existing codebases

How It Works

LangGraph4j allows you to build stateful, multi-agent applications by defining cyclical graphs where different components (agents, tools, or custom logic) interact with a shared state.

⭐ Key Features

Core Graph Model

Stateful Execution - Manage shared state across graph nodes for memory and context awareness
Cyclical Graphs - Support for loops needed in real agent-based architectures
Explicit Control Flow - Clear, definable paths and transitions

Reliability & Long-Running Workflows

Checkpoints - Save and replay graph state at any point, enabling long-running workflows and Human-in-the-Loop (HITL)

Integration & Modularity

Modularity - Build complex systems from reusable graphs (for example, a SubGraph)
Flexibility - Integrate with various LLM providers and custom Java logic

Observability

Observability & Debugging - Use hooks as interception points around node and edge execution to add logging, metrics, or traces without contaminating business logic

How to Get Involved

Start here: try the Quickstart in the docs and run your first graph in minutes.

Documentation: Visit the official documentation
DeepWiki: Check out the community docs on DeepWiki
Latest Releases: Check the GitHub releases page
Requirements: Minimum Java 17 supported
Community: Join the Discord community
Github Repository: langgraph4j/langgraph4j
Maven Central: Available on Maven Central
License: MIT - Open for contributions

Conclusion

f you're intrigued by this project and want to support me in putting effort into this development journey, checkout the project, try it out, and let me know your feedback. Don't forget to star it ⭐️!
In the meantime, happy AI coding

LangChain vs LangGraph vs LangSmith: Understanding the Ecosystem

Sat, 17 Jan 2026 13:36:52 +0000

Building LLM apps isn’t just about prompts anymore.
It’s about composition, orchestration, and observability.

TL;DR

LangChain provides the foundational building blocks for creating LLM applications through modular components and a unified interface for working with different AI providers.
LangGraph extends this foundation with stateful, graph-based orchestration for complex multi-agent workflows requiring loops, branching, and persistent state.
LangSmith completes the picture by offering observability, tracing, and evaluation tools for debugging and monitoring LLM applications in production.

Use:

LangChain for straightforward chains and RAG systems
LangGraph when you need sophisticated state management and agent coordination
LangSmith throughout development and production for visibility into behavior

Hands-on GitHub Repositories

LangChain RAG Project → https://github.com/rajkundalia/langchain-rag-project
LangGraph Analyzer → https://github.com/rajkundalia/langgraph-analyzer
LangSmith Learning → https://github.com/rajkundalia/langsmith-learning

Introduction

The landscape of LLM application development has evolved rapidly since 2022.

What began as simple prompt–response interactions has grown into multi-step workflows involving retrieval systems, tool usage, autonomous agents, and long-running processes. This evolution introduced new problems at each stage of the development lifecycle.

The composition problem → How do you connect prompts, models, tools, and data?
The orchestration problem → How do you manage branching, retries, loops, and shared state?
The observability problem → How do you debug, evaluate, and monitor these systems?

The LangChain ecosystem emerged to address each layer:

Problem	Tool	Year
Composition	LangChain	2022
Orchestration	LangGraph	2024
Observability	LangSmith	2023–2024

Each tool targets a specific layer in the LLM application stack.

LangChain: The Foundation

LangChain is the core framework for building LLM-powered applications.

Its primary goal is abstraction: different LLM providers expose different APIs, capabilities, and quirks. LangChain hides these differences behind a unified interface.

Core Building Blocks

LangChain is composed of modular, swappable components:

Prompts – Templates and structured inputs for models
Models – OpenAI, Anthropic, Google, or local LLMs
Memory – Conversation history and contextual state
Tools – Function calls to external systems
Retrievers – Vector databases and RAG pipelines

LCEL: LangChain Expression Language

What ties everything together is LCEL.

LCEL introduces a declarative, pipe-based syntax for composing chains:

prompt | model | output_parser

Instead of writing imperative glue code, you describe data flow.

Why LCEL Matters

LCEL enables:

Automatic async, streaming, and batch execution
Built-in LangSmith tracing
Parallel execution of independent steps
A unified Runnable interface (invoke, batch, stream)

This makes chains faster, cleaner, and easier to reason about.

Multi-Provider Support

LangChain supports dozens of LLM providers and integrations.

You can switch providers by changing one line of configuration, enabling:

Vendor independence
A/B testing across models
Cost and latency optimization

When LangChain Is Enough

Use LangChain when your workflow is primarily:

Input → Process → Output

Typical use cases include:

Chatbots with memory
RAG-based Q&A systems
Natural language → SQL generation
Linear tool pipelines

If your application doesn’t need complex branching or shared long-lived state, LangChain is the right tool.

LangGraph: Stateful Agent Orchestration

LangGraph solves the orchestration problem.

As soon as your application needs to:

make decisions,
loop,
retry,
or coordinate multiple agents, linear chains start to break down.

Graph-Based Architecture

LangGraph models your application as a directed graph:

Nodes → processing steps or agents
Edges → execution flow between nodes

This enables patterns that are hard or impossible with chains:

Loops and retries
Conditional branching
Parallel execution
Shared, persistent state

State as a First-Class Concept

Every LangGraph workflow operates on a shared state object.

Nodes receive the current state
They compute updates
Updates are merged back into state

This allows multiple agents to collaborate naturally.

Example:

Research agent gathers sources
Fact-checking agent validates claims
Synthesis agent produces the final answer

All without complex message passing.

Conditional Routing

LangGraph supports conditional edges.

A function decides which node runs next based on runtime state:

Route customer queries to specialist agents
Loop back when required information is missing
Retry until success conditions are met

Persistence & Checkpointing

LangGraph includes built-in checkpointing:

Persist state across restarts
Resume long-running workflows
Support human-in-the-loop pauses
Enable time-travel debugging

This is critical for production-grade agent systems.

Visualization Support

LangGraph workflows are inspectable and exportable:

Mermaid diagrams for documentation
PNG images for presentations
ASCII graphs for terminal debugging

This makes complex agent systems understandable and communicable.

When You Need LangGraph

Choose LangGraph when you need:

Explicit shared state
Runtime decision-making
Retry and failure recovery
Multi-agent coordination
Long-running workflows

A classic example is an autonomous research agent that iteratively searches, reads, verifies, and synthesizes information.

LangSmith: The Observability Layer

LangSmith answers the question:

“What is my LLM application actually doing?”

It doesn’t build workflows — it illuminates them.

Tracing Everything

LangSmith captures full execution traces:

Prompts and responses
Token usage and latency
Component call stacks
Errors and retries

You can drill down from:

a full workflow run
→ to a single LLM call.

This makes debugging dramatically easier.

Evaluation & Regression Testing

LangSmith allows you to:

Create evaluation datasets
Run structured tests
Track quality metrics
Compare prompts and models

This enables regression testing for LLM apps — a must-have for production systems.

Production Monitoring

In production, LangSmith tracks:

Response times
Error rates
Token and cost trends
Usage by workflow or user

Alerts help you catch issues early and optimize costs.

Framework-Agnostic

While LangSmith integrates seamlessly with LangChain and LangGraph, it’s not limited to them.

You can instrument any LLM application with LangSmith.

Quick Comparison

Tool	Solves	Use When
LangChain	Composition	Linear workflows, RAG, simple agents
LangGraph	Orchestration	Branching, loops, shared state, multi-agent
LangSmith	Observability	Debugging, evaluation, production monitoring

The Broader Ecosystem

LangFlow

LangFlow provides a visual, drag-and-drop interface for building LangChain workflows.

Great for prototyping
Helpful for non-technical collaboration
Often exported to code for production

Model Context Protocol (MCP)

MCP (by Anthropic) standardizes tool and resource access for LLMs.

Works at the tool/retriever layer
Complements LangChain and LangGraph
Reduces custom integration effort
Framework-agnostic

MCP does not replace orchestration tools — it enhances connectivity.

Conclusion

The LangChain ecosystem is layered, not competitive.

LangChain builds the core logic
LangGraph manages complex workflows
LangSmith makes everything observable

Most serious LLM applications will use more than one of these tools.

Start simple, add complexity only when needed, and never ship without observability.

Creating a chatbot for SQLite Databases

Tue, 14 Jan 2025 19:49:46 +0000

We all use chatbots like ChatGPT, Perplexity, and Claude. But wouldn’t it be cool if a chatbot could directly talk to your database? And keep your data secure at the same time? Let’s build one!

Here’s how we can build it, step by step:

Technologies We’ll Use:

Frontend
Next.js (React framework)
Copilotkit (for interacting with sql agent)
Backend
Fastapi (Python backend framework)
langgraph and langchain (for building sql agent)

We’ll create a SQL agent using LangGraph. This agent will connect to your database and allow you to chat with it in a structured way.

Once the agent is ready, we’ll integrate it into a Next.js app using Co Agents.

Instead of setting up everything from scratch, you can use my starter kit.

Building the sql agent

You need to edit the agent.py in ./agent/my_agent directory.

Add all the imports

```python
from langchain_community.utilities import SQLDatabase
from langchain_core.messages import AIMessage, BaseMessage
from langchain_core.runnables import RunnableLambda
from langchain_core.output_parsers import StrOutputParser
from langchain.prompts import PromptTemplate
from langchain_groq import ChatGroq
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode
from langgraph.checkpoint.memory import MemorySaver
from pydantic import BaseModel, Field
from typing import Literal, Annotated, List
from typing_extensions import TypedDict
from dotenv import load_dotenv
```

You can load environment variables using:

 ```py
    load_dotenv()
    ```

We need a SQLite database for the SQL agent. For this example, I’m using the Chinook db.

We will use SQLDatabase to create an engine for querying the database and retrieving its context.

```py
db = SQLDatabase.from_uri(r"sqlite:///data/data.db")

def db_query_tool(query: str) -> str:
    result = db.run_no_throw(query)
    if not result:
        return "Error: Query failed. Please rewrite your query and try again."
    return result

def get_context() -> str:
    return db.get_table_info()
```

We need an LLM to interact with users. Here, we’re using the Llama 3.3 model via Groq.

```py
llm = ChatGroq(model="llama-3.3-70b-versatile")
```

Now comes the fun part - creating AgentState, edges and nodes for our LangGraph agent.

```py
class AgentState(TypedDict):
    """
    Attributes:
        messages (List[BaseMessage]): List of messages exchanged during the interaction.
        user_query (str): The user-provided query.
        sql_query (str): The generated SQL query.
        query_results (str): Results from executing the SQL query.
    """

    messages: Annotated[List[BaseMessage], add_messages]
    user_query: str
    sql_query: str
    query_results: str
```

We need to verify if the user's query is relevant to the database.

We can use structured outputs to obtain relevance from LLM using the Pydantic model by using withured_outputs() method.

```py
def check_relevance(state: AgentState) -> Literal["YES", "NO"]:
    class QueryRelevance(BaseModel):
        query_relevant: str = Field(
            description="Relevance of user query to database context. The response must be 'YES' or 'NO'."
        )

    relevance_prompt = PromptTemplate.from_template(
        """
        You are a relevance evaluation assistant. Your task is to determine if a user's query is relevant to a given database context. 
        Respond with 'YES' if the query is relevant, and 'NO' if it is not.

        Database Context: {context}.

        Example Queries (may not be related to context):
        1. "What were the total sales in 2010?" -> YES
        2. "Who is the CEO of the company?" -> NO
        3. "Which sales agent had the highest sales in 2009?" -> YES
        4. "What is the weather today?" -> NO

        Now evaluate the following query and respond with 'YES' or 'NO':
        {query}
        """
    )

    model = llm.with_structured_output(QueryRelevance)
    relevance_generator = relevance_prompt | model

    inputs = {"context": get_context(), "query": state["messages"][-1].content}
    relevance = relevance_generator.invoke(inputs)

    return relevance.query_relevant
```

Basically we attach prompt to the model and invoke the chain with our inputs (query - user query, context - database schema info).

Now, let’s define the nodes for query execution, answer generation, and handling irrelevant queries.

```py

def sql_query_execution(state: AgentState) -> AgentState:
    """
    Executes the SQL query and retrieves the results.

    Args:
        state (AgentState): The current state of the agent.

    Returns:
        AgentState: Updated state with the SQL query results.
    """
    state["query_results"] = db_query_tool(state["sql_query"])
    return state


def answer_generation(state: AgentState) -> AgentState:
    """
    Generates a natural language response based on the user query and SQL query results.

    Args:
        state (AgentState): The current state of the agent.

    Returns:
        AgentState: Updated state with the generated response.
    """
    answer_generation_prompt = PromptTemplate.from_template(
        """
    You are a highly skilled assistant responsible for generating user-friendly answers based on a user query and the SQL query's output.
    You will use the provided SQL output to craft a clear, concise, and helpful response that directly answers the user's query.

     User Query: {query}

        SQL Query Output:
        {sql_output}

        Instructions:
        - Analyze the SQL query output.
        - Generate a response that answers the user query in natural language.
        - Be specific and provide numerical values or details where applicable.

        Output:
        Provide only the answer to the user query in plain text, without any additional formatting or explanation.
        """
    )

    inputs = {
        "query": state["user_query"],
        "sql_output": state["query_results"],
    }

    answer_generator = answer_generation_prompt | llm
    return {**state, "messages": state["messages"] + [answer_generator.invoke(inputs)]}


def query_not_relevant(state: AgentState) -> AgentState:
    """
    Handles cases where the user's query is not relevant to the database context.

    Args:
        state (AgentState): The current state of the agent.

    Returns:
        AgentState: Updated state with a message indicating irrelevance.
    """
    return {
        **state,
        "messages": state["messages"]
        + [AIMessage(content="Given user query is not relevant to database context.")],
    }
```

Now, let's create a workflow that compiles into the agent.

```py
workflow = StateGraph(AgentState)

workflow.add_node("sql_query_generation", sql_query_generation)
workflow.add_node("sql_query_execution", sql_query_execution)
workflow.add_node("answer_generation", answer_generation)
workflow.add_node("query_not_relevant", query_not_relevant)

workflow.add_conditional_edges(
    START, check_relevance, {"YES": "sql_query_generation", "NO": "query_not_relevant"}
)
workflow.add_edge("query_not_relevant", END)
workflow.add_edge("sql_query_generation", "sql_query_execution")
workflow.add_edge("sql_query_execution", "answer_generation")
workflow.add_edge("answer_generation", END)
```

Finally, lets compile the workflow with memory.

```py
graph = workflow.compile(checkpointer=MemorySaver())
```

Let's run this agent.

```bash
# in terminal
poetry install
poetry run demo
```

UI

In the starter kit, the /copilotkit route is already set up to connect the agent to our Next.js app.

On the homepage(/), I’ve just rendered a heading and the chatbot provided by CopilotKit with some custom colors.

Let's run this app

```bash
pnpm i
pnpm dev
```

You can also add your own custom chatbot UI, as CopilotKit provides the useCopilotChat() hook.

SQL Agent Chatbot is created!

More Features

You can add more features to this app, such as:

Changing the database via the ui
Adding animations to the ui
Displaying sql query execution results to the user

Feel free to add anything you can imagine—there are no limits!

Here’s the source code for this project: kom-senapati/DataChat

Wrapping it up

CopilotKit is an innovative tool for adding AI copilots into your project. You can connect these copilots to deployed LangGraph ai agents.

LangGraph. is a python library that allows you to create stateful ai agents as graphs and supports multi-agent workflows.

We should use these technologies to keep up with the rapidly changing landscape of the software industry.

Dont forget to start my project DataChat, CopilotKit, LangGraph ⭐

Coder Legion - Recent posts tagged langgraph

Stop calling functions: my first LangGraph graph, where state does the calling

1. The state is a pydantic model; nodes return patches

2. The graph is a handful of declared edges

3. All routing logic in one boring function — and the wire that said "error" to everything

4. Closing the repair loop with state, not recursion

Wrapping up

LangGraph4j 1.9.0: What's new and the roadmap

LangGraph4j 1.9.0 is out!

Emit custom output while a node is running

The streaming engine behind it

Checkpoint Savers: A richer execution lifecycle

Completed runs release their thread by default

Experimental versioning for released runs

Parent and Subgraph savers work together

Interruption and Error Metadata

SQLite and PostgreSQL V2 savers

Other changes worth knowing before upgrading

Explicit Graph Inputs

Diagnostics and State Serialization

Agent infrastructure

LangGraph4j Studio gets a visual refresh

Studio 1.8

Studio 1.9

LangGraph4j DSL

What's Next for LangGraph4j

Persist more of the execution context

Redesign parallel node execution

Improve built-in Agents for Spring AI and LangChain4j

Try the Release

Conclusions

Managed Agent Infrastructure: What Changes When the Runtime Is Handled For You

From RAG to Agentic RAG. How I Added LangGraph to My Local

The Core Limitation of Classic RAG

What Agentic AI Adds

The Shared State Foundation of LangGraph

Agent 1 The RAG Agent

Agent 2 The Diagnostic Agent

Agent 3 : The Escalation Agent

The Orchestrator : Wiring It All Together

Seeing It In Action

What I Would Do Differently

RAG vs Agentic AI : When to Use Which

Key Takeaways

The cost of a serverless agent isn't the Lambda. It's the loop.

The setup: what Wingman actually is

Cold start #1: the image

Cold start #2: the init you pay for on import

The real cost: the retry loop, not the duration

The 29-second cliff

What I'd do differently

Takeaway

Building a worker-evaluator retry loop in LangGraph (and where it bites you on Lambda)

The problem

The approach and the key decision

What broke, and what I changed

Takeaway

Memory in LLM Agents, Explained: From Stateless Calls to Long-Term Memory

Introduction

1. Stateless: the baseline problem

2. Short-term memory in RAM

3. Persistence and threads

4. Sliding window

5. Summarization

6. Long-term memory: the user profile

7. Combining short-term and long-term memory

Choosing a pattern

Further reading

Conclusion

Skill-Based Sub-Agents with LangGraph4j and Spring AI

Combine Tools & Skill to create Agent

Why This Pattern Matters

Why Reusing Tool Semantics Is Powerful

Minimizing Context and Token Usage

Reference implementation

Skill Format

allowed-tools:

LangGraph4j

Why LangGraph4j

What you get