Coder Legion - Recent posts tagged spring-ai

Skill-Based Sub-Agents with LangGraph4j and Spring AI

Thu, 30 Apr 2026 21:42:33 +0000

Combine Tools & Skill to create Agent

Modern agentic systems usually expose two strong but separate abstractions:

Tools, which give an LLM executable capabilities
Skills, which package reusable instructions for a narrow task

This article shows how to combine both ideas in [LangGraph4j] so that a skill is not just prompt text, but a fully operational sub-agent exposed as a tool. The result is a practical pattern for building modular multi-agent systems in Java with [Spring AI] and [LangGraph4j], while keeping each agent focused, reusable, and low in context cost.

Starting from a standard ReACT agent, the approach implemented is quite simple.

Load a SKILL.md file
Read its front matter to define the tool contract ( name and argument description)
Use the skill body as the sub-agent system prompt
Restrict the sub-agent to only the tools declared by the skill
Register the sub-agent in a parent ReAct agent as if it were a normal tool

In other words, a skill becomes an executable tool-backed agent.

Why This Pattern Matters

The standard ReAct pattern already gives an LLM a strong execution model: reason, choose a tool, inspect the result, and continue until the task is complete. That model is useful because the LLM does not need to commit to a fixed plan up front. It can adapt after every tool call.

The next step is to let a tool itself be implemented by another agent.

Once that happens, the parent agent is no longer forced to know every operational detail. It only needs to know:

which specialized capability exists
when to invoke it
what input to pass

The specialized logic is delegated to a narrower agent with its own instructions and its own restricted toolset. This gives us a clean sub-agent architecture without inventing a new orchestration primitive. We reuse the same tool-calling semantics that ReAct agents already understand well.

Why Reusing Tool Semantics Is Powerful

One of the most valuable properties of LLM-based agents is their ability to build and revise a tool execution plan while reasoning over intermediate results. ReAct made this practical by combining thought, action, and observation in a loop.

When a tool call can transparently invoke another agent, we preserve that planning model instead of replacing it.

The parent agent can:

decide that a specialized capability is needed
provide only the task-relevant context
wait for a result exactly as it would for a standard tool

The child agent can:

run its own reasoning loop
invoke its own allowed tools
return a compact outcome to the parent

This gives hierarchical coordination without exposing the full inner workflow to the parent prompt.

Minimizing Context and Token Usage

This pattern also improves context discipline.

Without sub-agents, a single generalist agent often needs:

every tool definition
all operating instructions for every domain
every procedural rule in one large system prompt

That quickly becomes expensive and unreliable.

With skill-based sub-agents:

the parent only sees high-level capabilities
each child sees only the instructions relevant to its task
each child receives only the tools it is allowed to use

This narrower context has practical benefits:

fewer tokens sent to the model
lower prompt interference between unrelated domains
clearer tool choice
better modularity as the system grows

The allowed-tools skill property plays an important role here anabling tool filtering. It turns tool access into an explicit design decision instead of an accidental side effect of what happened to be registered globally.

Reference implementation

I've started implementation in the new [LangGraph4j] release stream 1.9 currently in active development. The implementation is centered on [`SkilledReactSubAgent`], which turns a markdown skill into both:

a compiled LangGraph4j sub-graph
a Spring AI tool (i.e. ToolCallback)

That means the same component can participate in the graph as an agent and present itself to the parent LLM as a callable tool.

In particular the flow is:

Parse the SKILL.md
Extract name and description from front matter
Optionally filter tools using allowed-tools
Use the markdown body as the sub-agent instructions
Compile the agent
Wrap it into a function tool whose input is a single context field

The important design choice is that the tool definition is derived from the skill metadata, while the agent behavior is derived from the skill content.

Skill Format

The supported skill format is

name:
description: |

    <multi line description>

allowed-tools:

### From Skill to Tool-Backed Agent The conversion happens in `SkilledReactSubAgent.Builder#build(...)`. This is the crucial bridge where: * the skill metadata defines the external API * the skill body defines the internal operating behavior * the allowed tools define the sub-agent execution boundary The resulting agent is then wrapped as a Spring AI function tool. > From the parent agent point of view, this is just another tool. Internally, it is a complete ReAct-capable agent workflow. ## Examples ### Purchase Assistant Agent This is an agent that allow to select a produdct from a reference marketplace and purchase it. Below the simplified component diagram. ![](https://bsorrentino.github.io/bsorrentino/assets/langgraph-java/sub-agents/PurchaseAssistantAgent1.png) ### Marketplace Skill

---
name: agent-marketplace
description: |
    marketplace agent, ask for information about products. This is agent that provides the information on the product marketplace.
    required data are:
    * product name
allowed-tools: searchByProduct
---

## Retrieve product information
We need all information about about the product of interest in particular is mandatory to have
* product name

with such information we have to invoke tool `searchByProduct` and return the result

### Payment Skill

---
name: agent-payment
description: |
    payment agent, this is the agent that provides payment service.
    required data are:
    * Product name
    * Product price
    * Product currency
    * the International Bank Account Number (IBAN) - optional

allowed-tools:
    - submit-payment
    - retrieve-iban
---

## submit a payment for purchasing a specific product
We need all information about purchasing to allow the payment

* Product name
* Product price
* Product price currency
* International Bank Account Number (IBAN)

With such information you must invoke tool `submit-payment` and return the result of the payment transaction plus the IBAN.
If IBAN is not provided invoke tool `retrieve-iban` to retrieve the required IBAN.

Observing the skills above they define three different concerns: * `name`: the tool name exposed to the parent agent * `description`: the tool description shown to the model * `allowed-tools`: the only tools available inside the sub-agent Everything after the front matter becomes the sub-agent system prompt. > Again, this is a strong separation of concerns. The parent agent sees a concise tool contract, while the child agent receives richer operating instructions. ## Finally the code Below a representative code snippet that show how wires two skill-based sub-agents into a parent agent: ```java var subAgentMarketplace = SkilledReactSubAgent.builder() .chatModel(chatModel) .tools(Marketplace.tools()) .build(compileConfig, SkillSource.of(Paths.get("skills/agent-marketplace/"))); var subAgentPayment = SkilledReactSubAgent.builder() .chatModel(chatModel) .tools(Payment.tools()) .build(compileConfig, SkillSource.of(Paths.get("skills/agent-payment/"))); var purchaseAssistantAgent = AgentExecutorEx.builder() .chatModel(chatModel) .tool(subAgentMarketplace) .tool(subAgentPayment) .build(compileConfig); var input = """ search for product 'X'. If found proceed to payment with IBAN US82WEST1234567890123456 to purchase it """; var result = purchaseAssistantAgent.invoke( GraphInput.args( Map.of("messages", new UserMessage(input) )), runnableConfig); ``` This composition is interesting because the purchase agent (the parent) does not need detailed knowledge of: * how product lookup works * how payment submission works * when IBAN retrieval is required Those rules are delegated to the sub agents themselves. That is a simple but good example of behavioral encapsulation. The parent agent only decides to involve marketplace agent to retrieve product and after that to involve payment agent to purchase. This is a very basic but representative example, it is useful because it shows the intended layering: * the user gives a single business request * the parent agent decomposes the work * specialized sub-agents execute their own logic * the final answer comes back through the normal tool-response path > Interesting note that each sub agent, potentially, can relies on different LLM chat model so we can further optimize overall agent execution process Below a more detailed componet diagram generated by [LangGraph4j] itself. ![](https://bsorrentino.github.io/bsorrentino/assets/langgraph-java/sub-agents/PurchaseAssistantAgent.png) From this diagram it is quite clear that the sub agents are nothing but sub-graphs ## Design Advantages This approach has several practical advantages. 1. **Reuse Existing Agent Patterns** There is no need to invent a separate orchestration protocol for sub-agents. A sub-agent is exposed as a tool, so the parent agent can use standard ReAct behavior without extra prompting complexity. 2. **Better Modularity** Each skill owns: * its tool-facing identity * its internal instructions * its allowed tool boundary That makes the system easier to evolve than a single monolithic agent prompt. 3. **Smaller and Cleaner Context** The child agent receives only what it needs. This reduces token usage and lowers the risk that unrelated instructions distort tool choice or task execution. 4. **Safer Capability Scoping** By filtering tools from `allowed-tools`, the framework prevents a child agent from accidentally seeing tools outside its intended domain. 5. **Better Reusability** A skill can be packaged, versioned, and reused across different parent agents. The same marketplace or payment skill can participate in many workflows as long as the tool contract remains stable. ## Tradeoffs and Implementation Considerations This pattern is strong, but it is not free of tradeoffs. 1. **The Parent Must Summarize Well** Because the child receives a single `context` field, the parent agent needs to pass the relevant facts clearly. If critical information is omitted, the child may need extra recovery steps or may fail to act. 2. **Tool Descriptions Matter More** Since the parent sees the child as a tool, the `name` and `description` fields in the skill front matter become part of the routing surface for the LLM. Weak metadata will reduce invocation quality. 3. **Prompt Boundaries Need Discipline** The skill body is effectively the child system prompt. If it is vague, contradictory, or overloaded, the benefits of modularity disappear. Skills need to stay focused. 4. **Observability Is Important** Nested agents can make execution harder to inspect if tracing is weak. LangGraph4j's graph structure and step streaming help here, but production systems should still invest in logging and traceability. ## Why LangGraph4j Fits This Well [LangGraph4j] is a good fit for this pattern because it already models agent workflows as explicit graphs and integrates naturally with Spring AI tool abstractions. That matters for two reasons: * the sub-agent is not just prompt composition; it is a compiled graph with state and execution hooks * the parent integration remains ergonomic because Spring AI already treats capabilities as tools * [LangGraph4j] support interruptions in subgraph enabling **Human In The Loop** also when involving sub agents So the implementation is not forcing two incompatible ideas together. It is aligning: * LangGraph4j graph execution * Spring AI tool calling * markdown-defined skills into a single reusable agent-building pattern. ## Conclusion Turning a skill into a sub-agent is a small change with a large impact. It lets us: * preserve the strengths of the ReAct loop * delegate specialized behavior cleanly * reduce context size * constrain tool access per domain * build multi-agent systems out of reusable skill modules The idea is implemented in a compact and pragmatic way: a `SKILL.md` file becomes the source of both the sub-agent prompt and the tool contract exposed to the parent agent. For Java teams building agentic applications with Spring AI and LangGraph4j, this is a practical pattern for moving from a single large agent toward a modular multi-agent design without abandoning the tool semantics that current LLMs already handle well. The implementation is in progress on release `1.9-SNAPSHOT` in develop branch of [LangGraph4j] repository but the first result are very promising. Hope this could help and encourage usage of [LangGraph4j] for your next Agentic Workflow. Checkout project, try it and let me know your feedback and... happy AI coding! [LangGraph4j]: https://github.com/langgraph4j/langgraph4j [Spring AI]: https://spring.io/projects/spring-ai

Designing APIs for the AI Era with Spring AI and MCP

Mon, 22 Sep 2025 13:49:16 +0000

As backend developers, we've been building robust REST APIs for years. We design them to be consumed by our UIs, mobile apps, or other microservices. It's a paradigm we've mastered. But what if I told you that with minimal effort, that same API could have a second interface—one that allows AI agents to interact with your business logic using natural language?

Today, thinking about artificial intelligence isn't an afterthought; it's a design strategy. It's not just about creating a service that responds to GET or POST requests, but about asking ourselves: how can an AI model consume my service to provide extra value?

For those of us developing with Java, thanks to Spring AI, this idea has shifted from a complex task to a simple extension of what we already do. In this post, we'll explore how incredibly easy it is to add an MCP (Model Context Protocol) Server to an existing Spring Boot application, using a real-world API for managing a product catalog as our example.

The Dual API Paradigm: REST for Machines, MCP for AI ????

A REST API is great for structured machine-to-machine communication. But an AI agent doesn't think in terms of JSON or endpoints.

This is where the MCP Server comes in. It acts as a translator, exposing your application's logic in a way that an LLM (Large Language Model) can understand and use. This allows you to have the best of both worlds:

Your traditional REST API, stable and reliable for your applications.
An MCP Server, offering a native interface for AI integration.

This approach forces you to design from the outset with the functions (Tools), data (Resources), and guides (Prompts) that an intelligent agent would need to operate autonomously within your domain.

What is the Model Context Protocol?

The Model Context Protocol (MCP) is the standard that defines how an AI model can discover and use your application's capabilities in a secure and structured way. It provides the AI with the necessary context about what actions it can perform and what information it can query.

This protocol is built on three pillars: Tools, Resources, and Prompts. If you want to dive deeper into how it works and what each of these concepts means, you can read my more detailed post: Model Context Protocol Exposed: The Power of Resources and Prompts Beyond Tools.

For this article, the key takeaway is seeing how easy it is to implement these three pillars with Spring AI.

From @RestController to MCP Server

To demonstrate this, I'll use my Product Catalog project.

Step 0: Build Your Traditional Spring Boot Service

I won't spend much time on this point because creating the foundation for a Spring Boot project is almost trivial using https://start.spring.io/. I'll also leave a link to their documentation if you want more details: Getting Started | Building an Application with Spring Boot.

Step 1: Configuration is a Breeze

Once you have your Spring Boot service, you just need to add the Spring AI dependency and enable the server. To manage versions centrally, we add the Spring AI "Bill of Materials" (BOM) to the <dependencyManagement> section of your pom.xml:

<dependencyManagement>  
    <dependencies>  
        <dependency>  
            <groupId>org.springframework.ai</groupId>  
            <artifactId>spring-ai-bom</artifactId>  
            <version>1.0.0</version>  
            <type>pom</type>  
            <scope>import</scope>  
        </dependency>  
    </dependencies>  
</dependencyManagement>

Next, in your <dependencies> section, you only need to add the MCP server starter, which in this case is spring-ai-starter-mcp-server-webmvc:

<dependencies>  
    <dependency>  
        <groupId>org.springframework.ai</groupId>  
        <artifactId>spring-ai-starter-mcp-server-webmvc</artifactId>  
    </dependency>  
</dependencies>

Finally, in your application.properties, you enable and configure the server. It's that simple:

# application.properties

# Enable the MCP server  
spring.ai.mcp.server.enabled=true  
spring.ai.mcp.server.name=product-catalog-mcp-server  
spring.ai.mcp.server.version=1.0.0

# Define general instructions for the AI  
spring.ai.mcp.server.instructions=A server for managing a product catalog with AI-powered tools, resources, and guides.

# Enable the capabilities you want to expose  
spring.ai.mcp.server.capabilities.tool=true  
spring.ai.mcp.server.capabilities.resource=true  
spring.ai.mcp.server.capabilities.prompt=true

With just these few lines, your application now has a functional MCP server.

Step 2: Implementing the MCP Features

Now that we have the Spring AI dependencies and the MCP starter, it's time to dive in.

2.1: Expose Your Services as Tools

You already have the business logic in your @Service classes. To expose a method as a tool for the AI, you just need one annotation: @Tool.

@Tool(
    description =
        "Creates a new product in the catalog with comprehensive validation and audit logging. Required fields: name, price, category, stockQuantity, minStockLevel. Optional fields: description, SKU (auto-generated if not provided), weight, dimensions, imageUrl. SKU must be globally unique if provided. Automatically creates audit log entry. Throws DuplicateSkuException for duplicate SKUs. Use this for adding new products to the inventory system.")
@Transactional
public Product createProduct(
    @ToolParam(
        description =
            "Product object with required fields: name, price, category, stockQuantity, minStockLevel. Optional: description, sku, weight, dimensions, imageUrl")
    Product product) {
...
}

And that's it! Your createProduct method is now a tool that any AI agent can invoke.

2.2: Expose Live Data with Resources

A Resource acts as a live data feed for the AI. Instead of having static data in a prompt, you can expose information that changes within your application, like a list of product categories.

To do this, we define a URI (e.g., categories://overview/all) and associate it with a function that returns the data.

public List<McpServerFeatures.SyncResourceSpecification> getResourceSpecifications() {
    return List.of(
            // All categories overview resource
            new McpServerFeatures.SyncResourceSpecification(
                    new McpSchema.Resource(
                            "categories://overview/all",
                            "All Categories Overview",
                            "Complete overview of all product categories with descriptions and guidelines",
                            "application/json",
                            null),
                    this::getAllCategoriesResource),
            ...
            );
}

private McpSchema.ReadResourceResult getAllCategoriesResource(
        McpSyncServerExchange exchange, McpSchema.ReadResourceRequest request) {
    try {
        List<Map<String, Object>> categoriesInfo =
                Arrays.stream(Category.values())
                        .map(this::createCategoryOverview)
                        .collect(Collectors.toList());

        String jsonContent =
                objectMapper.writeValueAsString(
                        Map.of(
                                "totalCategories",
                                Category.values().length,
                                "categories",
                                categoriesInfo,
                                "description",
                                "Complete catalog of product categories with their descriptions and key characteristics",
                                "lastUpdated",
                                Instant.now().toString()));

        return new McpSchema.ReadResourceResult(
                List.of(
                        new McpSchema.TextResourceContents(
                                request.uri(), "application/json", jsonContent)));
    } catch (Exception e) {
        log.error("Failed to get all categories resource", e);
        throw new RuntimeException("Failed to retrieve categories overview", e);
    }
}

2.3: Expose Your Workflows with Prompts

Prompts are structured guides that teach the AI how to perform complex, multi-step tasks. They are like pre-designed prompt templates that you can invoke, combining instructions, data from Resources, and Tools.

Imagine you want a guide for analyzing inventory. You could create a Prompt named inventory-analysis that instructs the AI to first fetch low-stock products (using a Tool) and then generate a report.

public List<McpServerFeatures.SyncPromptSpecification> getPromptSpecifications() {
    return List.of(
            ...

            // Inventory analysis prompt
            new McpServerFeatures.SyncPromptSpecification(
                    new McpSchema.Prompt(
                            "inventory-analysis",
                            "Inventory Analysis and Recommendations",
                            List.of(
                                    new McpSchema.PromptArgument(
                                            "analysisType",
                                            "Type of analysis (stock-health/reorder/turnover)",
                                            true),
                                    new McpSchema.PromptArgument(
                                            "category", "Focus on specific category", false))),
                    this::createInventoryAnalysisPrompt),
            
            ...
            );
}

private McpSchema.GetPromptResult createInventoryAnalysisPrompt(
        McpSyncServerExchange exchange, McpSchema.GetPromptRequest request) {
    try {
        String analysisType = (String) request.arguments().get("analysisType");
        String category = (String) request.arguments().getOrDefault("category", "ALL");

        String promptText = createInventoryAnalysisPromptText(analysisType, category);

        McpSchema.PromptMessage promptMessage =
                new McpSchema.PromptMessage(
                        McpSchema.Role.ASSISTANT, new McpSchema.TextContent(promptText));

        return new McpSchema.GetPromptResult(
                "Inventory analysis guidance for " + analysisType + " analysis",
                List.of(promptMessage));
    } catch (Exception e) {
        log.error("Failed to create inventory analysis prompt", e);
        throw new RuntimeException("Failed to generate inventory analysis prompt", e);
    }
}

private String createInventoryAnalysisPromptText(String analysisType, String category) {
    return switch (analysisType) {
        case "stock-health" -> """
            You are an inventory management specialist conducting a stock health analysis%s.

            ## Stock Health Analysis Framework

            ### Key Metrics to Evaluate:
            1. **Out of Stock Items** - Immediate attention required
            2. **Low Stock Warnings** - Items below minimum levels
            3. **Overstock Situations** - Items with excessive inventory
            4. **Stock Turnover Rates** - How quickly inventory moves
            5. **Dead Stock** - Items with no recent sales

            ### Analysis Process:
            1. Retrieve current inventory levels using available tools
            2. Compare against minimum stock thresholds
            3. Identify critical alerts requiring immediate action
            4. Calculate stock health scores by category
            5. Generate actionable recommendations

            ### Recommended Actions:
            - **Critical**: Immediate reorders for out-of-stock items
            - **Warning**: Schedule reorders for low-stock items
            - **Optimization**: Review slow-moving inventory for discounts
            - **Planning**: Adjust minimum stock levels based on trends

            Please conduct a comprehensive stock health analysis and provide prioritized recommendations.
            """
                .formatted(category.equals("ALL") ? "" : " for " + category + " category");

        case "reorder" -> """
            You are a procurement specialist developing reorder recommendations%s.

            ## Reorder Analysis Framework

            ### Reorder Triggers:
            1. Current stock ≤ minimum stock level
            2. Projected stockouts based on demand trends
            3. Lead time considerations
            4. Economic order quantities
            5. Seasonal demand adjustments

            ### Analysis Steps:
            1. Identify products below reorder points
            2. Calculate optimal order quantities
            3. Consider supplier minimums and volume discounts
            4. Factor in lead times and safety stock
            5. Prioritize based on sales velocity and margin

            ### Reorder Recommendations:
            - High priority: Fast-moving items near stockout
            - Medium priority: Steady sellers below minimum
            - Low priority: Slow movers with adequate runway

            Please generate a prioritized reorder plan with specific quantities and timing.
            """
                .formatted(category.equals("ALL") ? "" : " for " + category + " category");

        case "turnover" -> """
            You are a financial analyst evaluating inventory turnover performance%s.

            ## Inventory Turnover Analysis

            ### Key Calculations:
            1. **Turnover Ratio** = Cost of Goods Sold / Average Inventory Value
            2. **Days Sales Inventory** = 365 / Turnover Ratio
            3. **Stock Velocity** = Units Sold / Average Units in Stock
            4. **Carrying Cost Impact** = Average Inventory × Carrying Cost Rate

            ### Performance Benchmarks:
            - Electronics: 8-12 turns/year (Fast-moving technology)
            - Books: 3-5 turns/year (Diverse demand patterns)
            - Clothing: 4-6 turns/year (Seasonal collections)
            - General: 6-8 turns/year (Standard retail)

            ### Analysis Focus:
            1. Compare actual vs. target turnover rates
            2. Identify slow-moving inventory
            3. Calculate carrying cost implications
            4. Recommend optimization strategies

            Please analyze turnover performance and suggest improvements for capital efficiency.
            """
                .formatted(category.equals("ALL") ? "" : " for " + category + " category");

        default -> "Please specify analysis type: stock-health, reorder, or turnover.";
    };
}

Step 3: Configure the Beans to Expose Everything

Once you have all the pieces, exposing Tools, Resources, and Prompts is as simple as creating Spring Beans for them.

@SpringBootApplication
public class ProductCatalogSpringApplication {

    public static void main(String[] args) {
        SpringApplication.run(ProductCatalogSpringApplication.class, args);
    }

    @Bean
    public ToolCallbackProvider productCatalogTools(ProductService productService) {
        return MethodToolCallbackProvider.builder().toolObjects(productService).build();
    }

    @Bean
    public List<McpServerFeatures.SyncResourceSpecification> productCatalogResources() {
        List<McpServerFeatures.SyncResourceSpecification> productCatalogResources =
                new ArrayList<>();
        ...
        return productCatalogResources;
    }

    @Bean
    public List<McpServerFeatures.SyncPromptSpecification> productCatalogPrompts(
            ProductManagementPromptProvider productManagementPromptProvider) {
        return productManagementPromptProvider.getPromptSpecifications();
    }
}

With these simple steps, you've built a direct and secure bridge between your business logic and the world of AI.

Conclusion: The Next Natural Step for Your APIs

As we've seen, adding an MCP Server to your Spring Boot application isn't a Herculean task. It's the next logical step in the evolution of our services. Spring AI abstracts away all the complexity, allowing us to reuse the code we already have to open our applications to a universe of new possibilities with artificial intelligence.

Designing with AI in mind from the start is no longer an option—it's a competitive advantage.

What's Next? Your Turn to Act ????

I hope this post has inspired you to look at your APIs from a new perspective. Here are a few ideas to get you started:

Explore the source code: All the code from this example is available in my Product Catalog API GitHub repository. Clone it, run it with Docker, and experiment for yourself. The best way to learn is by doing!
Start small: You don't need a complex architecture. Pick a key method from one of your current Spring Boot applications and try exposing it as a @Tool. You'll see how simple it is.
Join the conversation: What do you think of this approach to AI-native APIs? What other use cases can you think of for an MCP Server in your projects?
Dive into the documentation: If you want to learn all the details, the official Spring AI documentation is the best place to keep learning.

Power Up Java with AI: Integrate LLaMA 3 Using Spring Boot

Wed, 30 Jul 2025 07:25:28 +0000

In today’s world of artificial intelligence, creating a personal AI assistant has become surprisingly accessible thanks to various LLM models. With Spring Boot and open-source AI models, we can build a conversational AI within Java applications. Let’s explore how this works.

In this example, we’ll install and use Ollama locally, and create an endpoint that accepts a user’s prompt and returns a response using AI.

Step 1: Install Ollama
To get started, install Ollama using this link.
Once the installation is complete, download one of the available models. For this example, we’ll use the LLaMA 3 model:

ollama run llama3 // or any other model

After the model is downloaded, you can also interact with it directly through the terminal:

>>> hello
Hello! It's nice to meet you. Is there something I can help you with, or would you like to chat?
>>>

Step 2: Connect to Ollama from a Spring Boot Application
Now it’s time to connect Ollama with a Spring Boot application.
Let’s quickly generate a Spring Boot project using start.spring.io.
Here’s an example of the build.gradle file with the necessary dependencies:

plugins {
 id 'java'
 id 'org.springframework.boot' version '3.5.0'
 id 'io.spring.dependency-management' version '1.1.7'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'

java {
 toolchain {
  languageVersion = JavaLanguageVersion.of(21)
 }
}

repositories {
 mavenCentral()
}

ext {
 set('springAiVersion', "1.0.0")
}

dependencies {
 implementation 'org.springframework.ai:spring-ai-starter-model-ollama'
}

dependencyManagement {
 imports {
  mavenBom "org.springframework.ai:spring-ai-bom:${springAiVersion}"
 }
}

tasks.named('test') {
 useJUnitPlatform()
}

Step 3: Create a REST Endpoint
After setting up the dependencies, all we need is a simple REST endpoint that receives a user prompt, sends it to the LLaMA 3 model, and returns the response.
For this, we can use the ChatModel class.

import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.ollama.api.OllamaModel;
import org.springframework.ai.ollama.api.OllamaOptions;

@RestController
@RequiredArgsConstructor
public class ChatController {
    private final ChatModel chatModel;

    @GetMapping("/message")
    public String handleMessage(@RequestParam String message) {
        var response = chatModel.call(
                new Prompt(
                        message,
                        OllamaOptions.builder()
                                .model(OllamaModel.LLAMA3)
                                .temperature(0.4)
                                .build()
                ));
        return response.getResult().getOutput().getText();
    }
}

Key Components:
ChatModel — Interface that calls the AI model to generate responses.
Prompt — Wraps the user’s message and AI configuration into a single request.
OllamaOptions — Configures model settings such as temperature and model selection.
OllamaModel.LLAMA3 — Specifies the LLaMA 3 model for processing the input.
temperature — Controls the creativity of the response (0 = strict, 1 = more random).

Its time to test:

Conclusion
Using Ollama locally with Spring AI makes it easy to build AI-powered APIs without relying on external cloud services. Without spending anything for the LLM model, an AI-powered chat endpoint is ready. We can do a little bit of prompt engineering to make it more specialized.

Learn how to write GenAI applications with Java using the Spring AI framework and utilize RAG for improving answers.

Sun, 22 Sep 2024 09:21:49 +0000

Generative AI (GenAI) is currently a hot topic in the tech world. Retrieval-Augmented Generation (RAG) is a technique that enhances the accuracy and reliability of generative AI models by grounding them in external knowledge sources.

In this blog post, we'll look at how to write GenAI applications with Java using the Spring AI framework and utilize RAG for improving answers.

What is Spring AI?

Spring AI is a framework for building generative AI applications in Java. It provides a set of tools and utilities for working with generative AI models and architectures, such as large language models (LLMs) and retrieval augmented generation (RAG). Spring AI is built on top of the Spring Framework, which is a popular Java framework for building enterprise applications, allowing those already familiar with or involved in the Spring ecosystem the ability to incorporate GenAI strategies to their already existing applications and workflow.

There are also other options for GenAI in Java, such as Langchain4j, but we'll focus on Spring AI for this post.

Creating a project

To get started with Spring AI, you'll need to either create a new project or add the appropriate dependencies to an existing project. You can create a new project using the Spring Initializr at https://start.spring.io/, which is a web-based tool for generating Spring Boot projects.

When creating a new project, you'll need to add the following dependencies:

Spring Web
OpenAI (or other LLM model, such as Mistral, Ollama, etc.)
Neo4j Vector Database (other vector database options also available)
Spring Data Neo4j

If you're adding these dependencies manually to an existing project, you can see the dependency details in today's related Github repository.

The Spring Web dependency allows us to create a REST API for our GenAI application. We need the OpenAI dependency to access the OpenAI model, which is a popular LLM. The Neo4j Vector Database dependency allows us to store and query vectors, which are used for similarity searches. Finally, adding the Spring Data Neo4j dependency provides support for working with Neo4j databases in Spring applications, allowing us to run Cypher queries in Neo4j and map entities to Java objects.

Go ahead and generate the project, and then open it in your favorite IDE. Looking at the pom.xml file, you should see that the milestone repository is included. Since Spring AI is not a general-availability release yet, we need to include the milestone repository to get the pre-release version of the dependencies.

A bit of boilerplate

First thing that we need is a Neo4j database. I like to use the Neo4j Aura free tier because the instance is managed for me, but there are also Docker images and other methods.

Depending on the LLM model you chose, you will also need an API key. For OpenAI, you can get one by signing up at OpenAI.

Once you have a Neo4j database and an API key, you can set up the config in the application.properties file. Here's an example of what that might look like:

spring.ai.openai.api-key=<YOUR API KEY HERE>
spring.neo4j.uri=<NEO4J URI HERE>
spring.neo4j.authentication.username=<NEO4J USERNAME HERE>
spring.neo4j.authentication.password=<NEO4J PASSWORD HERE>
spring.data.neo4j.database=<NEO4J DATABASE NAME HERE>

Note: It's a good idea to keep sensitive information like API keys and passwords in environment variables or other location external to the application. To create environment variables, you can use the export command in the terminal or set them in your IDE.

We can set up Spring Beans for the OpenAI client and the Neo4j vector store that will allow us to access necessary components wherever we need them in our application.

We can put these in our SpringAiApplication class by adding the related code to the class.

The EmbeddingClient bean creates a client for the OpenAI API and passes in our API key environment variable. Lastly, the Neo4jVectorStore bean configures Neo4j as the store for embeddings (vectors). We customize the configuration by specifying the label for the nodes that will store the embeddings, as Spring's default looks for Document entities. We also specify our index name for the embeddings (default is spring-ai-document-index).

Data set

For this example, we'll use a dataset of books and reviews from Goodreads. You can pull a curated version of the dataset from here. The dataset contains information about books, as well as related reviews.

I have already generated embeddings using OpenAI's API, so if you want to generate your own, you will need to comment out the final Cypher statement in the script and instead run the generate-embeddings.py script (or your custom version) to generate and load the review embeddings to Neo4j.

Application model

Next, we need to create a domain model in our application to map to our database model. In this example, we'll create a Book entity that represents a book node. We'll also create a Review entity that represents a review of a book. The Review entity will have an embedding (vector) associated with it, which we'll use for similarity searches.

These entities are standard Spring Data Neo4j code, so I won't show the code here. However, full code for each class is available in the Github repository - Book class, Review class.

We also need a repository interface defined so that we can interact with the database. While we will need to define a custom query, we'll come back and add that in a bit later.

public interface BookRepository extends Neo4jRepository<Book, String> {
}

Next, the core of this application where all the magic happens is the controller class. This class will contain the logic for taking a search phrase provided by the user and calling the Neo4jVectorStore to calculate and return the most similar ones. We can then pass those similar reviews into a Neo4j query to retrieve connected entities, providing additional context in the prompt for the LLM. It will use all the information provided to respond with some similar book recommendations for the original searched phrase.

Controller

Our controller class contains a couple of common annotations, to start. We'll also inject the Neo4jVectorStore and BookRepository beans that we defined earlier, as well as the OpenAiChatClient for our embedding client.

The next thing is to define a string for our prompt. This is the text that we will pass to the LLM to generate the response. We'll use the search phrase provided by the user and the similar reviews we find in the database to populate our prompt parameters in a few minutes. Next, we define the constructor for the controller class, which will inject the necessary beans.

@RestController
@RequestMapping("/")
public class BookController {
    private final ChatClient client;
    private final Neo4jVectorStore vectorStore;
    private final BookRepository repo;

    String prompt = """
            You are a book expert providing recommendations from high-quality book information in the CONTEXT section.
            Please summarize the books provided in the context section.
                        
            CONTEXT:
            {context}
                        
            PHRASE:
            {searchPhrase}
            """;

    public BookController(ChatClient.Builder builder, Neo4jVectorStore vectorStore, BookRepository repo) {
        this.client = builder.build();
        this.vectorStore = vectorStore;
        this.repo = repo;
    }

    //Retrieval Augmented Generation with Neo4j - vector search + retrieval query for related context
    @GetMapping("/rag")
    public String generateResponseWithContext(@RequestParam String searchPhrase) {
        List<Document> results = vectorStore.similaritySearch(SearchRequest.query(searchPhrase).withTopK(10));

        //more code shortly!
    }
}

Finally, we define a method that will be called when a user makes a GET request to the /rag endpoint. This method will first take a search phrase as a query parameter and pass that to the vector store's similaritySearch() method to find similar reviews. I have also added a couple of customization filters to the query.

Then, we map the similar Review nodes back to Document entities because Spring AI expects a general document type. The Neo4jVectorStore class contains methods to convert Document to a custom record, as well as the reverse for record to Document conversion.

Back in our controller method for book recommendations, we now have similar reviews for the user's searched phrase. But reviews (and their accompanying text) aren't really helpful in giving us book recommendations. So now we need to run a query in Neo4j to retrieve the related books for those reviews. This is the retrieval augmented generation (RAG) piece of the application.

Let's write the query in the BookRepository interface to find the books associated with those reviews.

public interface BookRepository extends Neo4jRepository<Book, String> {
    @Query("MATCH (b:Book)<-[rel:WRITTEN_FOR]-(r:Review) " +
            "WHERE r.id IN $reviewIds " +
            "AND r.text <> 'RTC' " +
            "RETURN b, collect(rel), collect(r);")
    List<Book> findBooks(List<String> reviewIds);
}

In the query, we pass in the ids of the reviews from the similarity search ($reviewIds) and pull the Review -> Book pattern for those reviews. We also filter out any reviews that have the text 'RTC' (which is a placeholder for reviews that don't have text). We then return the Book nodes, the relationships, and the Review nodes.

Now we need to call that method in our controller and pass the results to a prompt template. We will pass that to the LLM to generate a response with a book recommendation list based on the user's search phrase (we hope!). :)

//Retrieval Augmented Generation with Neo4j - vector search + retrieval query for related context
@GetMapping("/rag")
public String generateResponseWithContext(@RequestParam String searchPhrase) {
    List<Document> results = vectorStore.similaritySearch(SearchRequest.query(searchPhrase).withTopK(10));

    List<Book> bookList = repo.findBooks(results.stream().map(Document::getId).collect(Collectors.toList()));
    System.out.println("--- Book list ---");
    System.out.println(bookList);

    var template = new PromptTemplate(prompt, Map.of("context", bookList.stream().map(b -> b.toString()).collect(Collectors.joining("\n")), "searchPhrase", searchPhrase));
    System.out.println("----- PROMPT -----");
    System.out.println(template.render());

    return client.prompt(template.create()).call().content();

}

Starting right after the similarity search, we call our new findBooks() method and pass in the list of review ids from the similarity search. The retrieval query returns to a list of books called bookList. Next, we create a prompt template with the prompt string, the context data from the graph, and the user's search phrase, mapping the context and searchPhrase prompt parameters to the graph data (list with each item on new line) and the user's search phrase, respectively. I have also added a System.out.println() to print the prompt to the console so that we can see what is getting passed to the LLM.

Finally, we call the template's create() method to generate the response from the LLM. The returning JSON object has a contents key that contains the response string with the list of book recommendations based on the user's search phrase.

Let's test it out!

Running the application

To run our Goodreads AI application, you can use the ./mvnw spring-boot:run command in the terminal. Once the application is running, you can make a GET request to the /rag endpoint with a search phrase as a query parameter. Some examples are included next.

http ":8080/rag?searchPhrase=happy%20ending"
http ":8080/rag?searchPhrase=encouragement"

Sample call and output + full prompt

Call and returned book recommendations:

jenniferreif@elf-lord springai-goodreads % http ":8080/rag?searchPhrase=encouragement"
    
The books recommended in the context section are all inspirational and encouraging reads. They include "An Autobiography," "The Greatest Gift," "The Cross and the Switchblade," "David and Goliath," "The Art of Recklessness," "90 Minutes in Heaven," "Just Kids," and "The Five People You Meet in Heaven." These books offer great insight, hope, and encouragement to readers looking for uplifting stories and messages.

Application log output:

----- PROMPT -----
You are a book expert providing recommendations from high-quality book information in the CONTEXT section.
Please summarize the books provided in the context section.

CONTEXT:
Book[book_id=125441, title=An Autobiography, isbn=0717806677, isbn13=9780717806676, reviewList=[Review[id=d664f3888ecdfd78ec50606fe5f9b58c, text=Inspiring, rating=5]]]
Book[book_id=1488663, title=The Greatest Gift: The Original Story That Inspired the Christmas Classic It's a Wonderful Life, isbn=0670862045, isbn13=9780670862047, reviewList=[Review[id=b74851666f2ec1841ca5876d977da872, text=Inspiring, rating=4]]]
Book[book_id=772852, title=The Cross and the Switchblade, isbn=0515090255, isbn13=9780515090253, reviewList=[Review[id=f70c68721a0654462bcc6cd68e3259bd, text=encouraging, rating=4]]]
Book[book_id=15751404, title=David and Goliath: Underdogs, Misfits, and the Art of Battling Giants, isbn=0316204366, isbn13=9780316204361, reviewList=[Review[id=8c8bf791101b405c2cd134da193d2701, text=Inspiring, rating=4]]]
Book[book_id=7517330, title=The Art of Recklessness: Poetry as Assertive Force and Contradiction, isbn=1555975623, isbn13=9781555975623, reviewList=[Review[id=2df3600d488e182a3ef06bff7fc82eb8, text=Great insight, great encouragement, and great company., rating=4]]]
Book[book_id=89375, title=90 Minutes in Heaven: A True Story of Death and Life, isbn=0800759494, isbn13=9780800759490, reviewList=[Review[id=85ef80e09c64ebd013aeebdb7292eda9, text=inspiring & hope filled, rating=5]]]
Book[book_id=341879, title=Just Kids, isbn=006621131X, isbn13=9780066211312, reviewList=[Review[id=3eab2a67ac0bde179dd017014cb212bf, text=Inspiring, rating=5], Review[id=d731490fcfefe6547e459bdff204df4a, text=inspiring., rating=5]]]
Book[book_id=3431, title=The Five People You Meet in Heaven, isbn=1401308589, isbn13=9781401308582, reviewList=[Review[id=3ad8018734f177fd1fe47e6ada1c5df8, text=inspirational, rating=4], Review[id=2b94facb74fe09afd1fd54f0db2251c8, text=Inspiring, rating=5]]]

PHRASE:
encouragement

We can see that the LLM generated a response with a list of book recommendations based on the books found in the database (CONTEXT section of prompt). The results of the similarity search + graph retrieval query for the user's search phrase are in the prompt, and the LLM's answer uses that data for a response.

Wrapping Up!

I hope this post helps to get you started with Spring AI and beyond. Happy coding!

Resources

Code (Github repository): Spring AI Goodreads
Webpage: Spring AI project
API: Spring AI - Neo4jVectorStore