Coder Legion - Recent posts tagged springboot

Reliable Event-Driven Architecture in Spring Boot: Outbox, Inbox, Retries, and Idempotency

Thu, 29 Oct 2026 00:07:44 +0000

Why publishing an event is only the beginning of building a reliable distributed system.

Building an event-driven application with Spring Boot is relatively straightforward.

You create a Kafka producer, publish an event, implement a consumer, and process the message.

Everything works beautifully during development.

Then production introduces the situations your happy-path tests never covered.

What happens when the database transaction commits, but Kafka is unavailable? What happens when the consumer processes an event successfully but crashes before acknowledging it? What happens when two application instances attempt to process the same message?

These aren't unusual edge cases. They're fundamental challenges of distributed systems.

And solving them requires more than adding a retry annotation.

Let's explore the reliability boundaries behind a production-ready event-driven architecture.

1. The dual-write problem

Imagine a payment service that must perform two operations:

Save a payment in PostgreSQL.
Publish a PaymentCompleted event to Kafka.

A straightforward implementation might look like this:

@Transactional
public void completePayment(Payment payment) {
    paymentRepository.save(payment);
    kafkaTemplate.send(
        "payment.completed",
        new PaymentCompleted(payment.getId())
    );
}

The code looks reasonable, but there's a serious problem.

The database transaction and Kafka publication do not share a single atomic transaction.

Several outcomes are possible:

The database commits, but the event is never published.
Kafka accepts the event, but the database transaction rolls back.
Publication times out, leaving the application uncertain whether Kafka received the event.

Adding retries doesn't eliminate the fundamental inconsistency.

The problem is not whether Kafka is reliable. The problem is coordinating two independent systems.

The transactional outbox

The transactional outbox pattern changes the transaction boundary.

Instead of publishing directly to Kafka, the application writes an outbox record within the same database transaction as the business operation.

Payment Service
      |
      v
Database Transaction
      |
      +-- Save Payment
      |
      +-- Save Outbox Event
      |
      v
    Commit
      |
      v
Outbox Publisher
      |
      v
    Kafka

Either the payment and outbox event are committed together, or neither is committed.

A separate publisher retrieves pending outbox records and delivers them to Kafka.

This solves the atomicity problem between the business state and the durable intent to publish.

It does not guarantee that an event will be published exactly once.

That distinction matters.

2. Why the outbox still needs idempotency

Consider this sequence:

The outbox publisher sends an event to Kafka.
Kafka successfully accepts it.
The application crashes before marking the outbox record as published.
The publisher restarts and sends the event again.

Kafka may now contain two copies of the same logical event.

This is a normal consequence of at-least-once delivery.

The correct response is not to assume duplicates can always be prevented.

Instead, design consumers that can safely tolerate them.

3. The transactional inbox

An inbox provides a durable record of incoming events that have already been processed.

Each event should carry a stable identifier, such as an event ID generated when the logical event is created.

When the consumer receives a message, it checks whether that event has already been processed.

A simplified model looks like this:

Kafka Event
     |
     v
Consumer
     |
     v
Database Transaction
     |
     +-- Claim Event ID in Inbox
     |
     +-- Apply Business Changes
     |
     +-- Mark Inbox Completed
     |
     v
   Commit

The important part is the transaction boundary.

The inbox record and the business changes must participate in the same transaction when using the same transactional database.

Otherwise, a consumer can record an event as completed even though its business operation failed.

A subtle but important case

Suppose a consumer receives PaymentCompleted, updates an invoice, and produces InvoicePaid.

Now the reliability boundary becomes:

Inbox + Business State + Outbox

All three should be committed atomically.

Incoming Event
      |
      v
Database Transaction
      |
      +-- Inbox Claim
      |
      +-- Update Invoice
      |
      +-- Create Outbox Event
      |
      v
    Commit
      |
      v
Outbox Publisher
      |
      v
Outgoing Event

This is one of the most useful patterns for building dependable event-driven workflows.

The inbox protects the incoming side. The outbox protects the outgoing side.

Neither eliminates the need for careful business-level idempotency, especially when external side effects are involved.

4. Retries are a lifecycle, not just an annotation

Retries are often introduced with a simple configuration:

@Retryable(
    retryFor = TransientException.class,
    maxAttempts = 3
)

For short-lived operations, that can be perfectly reasonable.

But what happens when a dependency is unavailable for 30 minutes? What happens when the application restarts? What happens when an event requires operator intervention?

An in-memory retry loop cannot reliably manage every long-running failure scenario.

For durable asynchronous processing, retries should be modeled as persistent state.

A typical lifecycle might look like this:

PENDING
   |
   v
PROCESSING
   |
   +---- Success ----> COMPLETED
   |
   +---- Retryable failure
   |           |
   |           v
   |      RETRY_SCHEDULED
   |           |
   |           v
   |        PENDING
   |
   +---- Terminal failure
               |
               v
             FAILED

Each attempt can record useful information:

Attempt count
Next eligible execution time
Last failure category
Processing status
Relevant timestamps

Exponential backoff with jitter helps avoid repeatedly overwhelming an unhealthy dependency.

But retry policies should distinguish transient failures from permanent ones.

Retrying an invalid business request a thousand times won't make it valid.

5. Multi-instance processing changes everything

A background publisher may work perfectly when there's only one application instance.

Then the service is scaled to three pods.

All three publishers query the same pending records.

Without coordination, multiple instances may attempt to publish or process the same work concurrently.

One common PostgreSQL technique is:

SELECT id
FROM outbox_event
WHERE status = 'PENDING'
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

SKIP LOCKED allows concurrent workers to skip rows already locked by another transaction.

This can improve coordination between workers, but it's not a complete delivery protocol.

The design must still consider:

How work is claimed and how long claims remain valid
Transaction duration and lock ownership
Recovery after a worker crashes
Retry scheduling and fairness
Ordering requirements
Duplicate delivery

For example, holding database locks across a slow network call can increase contention and create unnecessarily long transactions.

A robust implementation may use short claim transactions, explicit processing ownership, and recovery mechanisms rather than keeping database transactions open while publishing.

Concurrency control and delivery guarantees are related, but they are not interchangeable.

6. Exactly-once is not an end-to-end promise

It's tempting to describe an event-driven system as exactly-once.

But exactly-once semantics depend on the boundaries being discussed.

Kafka transactions can provide guarantees for supported Kafka operations. A PostgreSQL transaction can atomically update records within PostgreSQL.

Neither automatically makes a database update, Kafka publication, HTTP request, and external payment operation one globally atomic action.

A more useful engineering goal is:

At-least-once delivery with effectively-once business outcomes wherever possible.

That typically requires durable event identities, transactional inbox processing, idempotent business operations, and explicit handling of external side effects.

For payment systems, this distinction is particularly important.

Processing the same event twice must not mean charging a customer twice.

7. Observability is part of reliability

A system that retries failures silently can appear healthy while accumulating thousands of unprocessed events.

Operational visibility should answer questions such as:

How many events are pending?
How old is the oldest unpublished event?
How many retries are scheduled?
Which events are permanently failing?
Are workers making progress?
Are duplicates being detected?
Is event processing latency increasing?

Metrics, structured logs, tracing, and operator-facing failure information are not optional extras for a production system.

They help distinguish a temporary delay from a workflow that has stopped making progress.

A reliable system should make failure visible and recovery understandable.

8. Putting these patterns together

A practical architecture combines several responsibilities:

Business Operation
       |
       v
Transactional Outbox
       |
       v
Durable Publisher
       |
       v
Message Broker
       |
       v
Transactional Inbox
       |
       v
Business Processing
       |
       +-- State Changes
       |
       +-- Optional New Outbox Events
       |
       v
Commit

Around that flow, we need retry policies, concurrency coordination, idempotency, recovery, and observability.

Each component solves a different failure mode.

The outbox prevents losing the intent to publish when business data commits.

The inbox protects consumers from repeated processing.

Retries help transient failures recover.

Concurrency controls coordinate multiple workers.

Observability makes the system operable.

None of these patterns should be treated as magic.

Their value comes from understanding exactly which failure boundary each one protects.

9. Why I built NERV Event

After working with event-driven enterprise applications, I wanted these reliability concerns to be reusable rather than repeatedly implemented in individual services.

That led to NERV Event, an open-source Java and Spring Boot library focused on patterns such as transactional outbox, inbox processing, retries, and idempotency.

The goal isn't to hide distributed-systems complexity behind annotations.

It's to provide reusable infrastructure while keeping the reliability model explicit.

GitHub: https://github.com/czetsuyatech/nerv-event

The library is one possible implementation of these ideas. The architectural principles are useful regardless of which framework or library you choose.

Final thoughts

Publishing an event is easy.

Building a system that behaves predictably when databases, brokers, application instances, and downstream dependencies fail is much harder.

Reliable event-driven architecture begins with a simple question:

What happens if the application crashes at this exact point?

Ask that question at every transaction boundary, every message acknowledgment, and every external call.

The answers will reveal where durability, idempotency, retries, and recovery are actually needed.

I'd be interested to hear how other teams approach this: Do you use a transactional inbox alongside your outbox, or do you handle consumer idempotency directly in your business tables?

Originally published on CzetsuyaTech. Explore more articles on Java, Spring Boot, distributed systems, and software architecture there.

Polar Bear Cache: Local Caching Without Repeated Remote Calls

Mon, 26 Oct 2026 10:07:01 +0000

Shared storage provides the source of truth. Local caches provide speed. Events help keep cache state in sync.

Project: Polar Bear Cache — GitHub repository

In my previous discussions about microservices, I explored how shared storage and event-driven synchronization can work together to manage distributed session state.

This time, I want to focus on another part of the problem: caching frequently accessed data across multiple service instances.

The Problem: Faster Reads, Distributed Cache Invalidation

In a microservices architecture, multiple service instances often need the same data, especially data that is read frequently but updated relatively infrequently.

A distributed cache can reduce database load, but accessing it still involves network round trips and potentially serialization and deserialization overhead.

So, what if each service instance could keep frequently accessed data in its own local cache, while receiving notifications when that data changes elsewhere?

That is the idea behind Polar Bear Cache, an open-source Java caching library designed to work with Spring's familiar cache annotations.

The Design: Local Cache + Event-Driven Invalidation

Each service instance maintains its own local cache. When data changes, an event service can notify other instances that specific cache entries need to be invalidated.

The key principle is simple:

Broadcast invalidation events, not the entire cached data.

Polar Bear Cache integrates with Spring's cache annotations:

@Cacheable — Returns a cached value when available, or executes the underlying method to load it when the value is missing.
@CachePut — Updates the local cache and, when event-service integration is configured, notifies other instances to invalidate the corresponding entry.
@CacheEvict — Evicts cached data locally and, when event-service integration is configured, notifies other instances to invalidate their corresponding entries.

For example, when a product is updated:

The application updates the source of truth and its local cache.
It publishes a cache-invalidation event through the event service.
Other service instances receive the event and evict the corresponding local entry.
Their next read loads the data again from the source of truth.

This allows repeated reads to stay local in the common case, while cross-instance communication is used to coordinate invalidation when data changes.

When Does This Approach Make Sense?

This design is particularly suitable for:

Read-heavy workloads with relatively infrequent updates.
Applications running multiple service instances, each with its own local cache.
Use cases where brief delays in cache invalidation are acceptable.

There is an important trade-off: event-driven invalidation is asynchronous. Between a data update and the time other instances process the invalidation event, stale data may still be served.

Polar Bear Cache does not guarantee strong consistency across service instances.

Depending on the application, you may also need to consider event-delivery reliability, duplicate events, cache expiration, and recovery strategies when a service instance misses an event.

The goal is not to eliminate every consistency trade-off, but to make local caching practical in distributed applications without requiring every read to go through a remote cache.

Why “Polar Bear”?

The name comes from a somewhat optimistic engineering idea:

Local caching
→ Fewer remote cache requests and network round trips
→ Potentially more throughput from the same resources
→ Potentially fewer servers needed for a given workload
→ Lower energy consumption and greenhouse gas emissions
→ Maybe we can help save polar bears. 🐻❄️🧊

Of course, the actual impact depends on workload characteristics, infrastructure, and deployment choices. Performance improvements do not automatically translate into lower energy consumption, but I like the idea that better resource efficiency could benefit both applications and the environment.

I'd Love to Hear Your Thoughts

I'm interested in how other engineers approach local caching in distributed systems.

How do you handle cache invalidation across service instances?
Do you prefer event-driven invalidation, short TTLs, versioned keys, or a combination?
What failure scenarios would you consider essential to address in this design?

If you're interested in the implementation, feedback and technical discussion are very welcome.

Less remote access, more ice. 🐻❄️🧊

Java #Spring #Microservices #Caching #DistributedSystems #OpenSource

Sync Session: Rethinking Session Management in Microservices

Mon, 26 Oct 2026 09:17:47 +0000

Sync Session: Rethinking Session Management in Microservices

Project: Sync Session on GitHub

Shared storage provides the source of truth. Local caches provide performance. Events synchronize session state across service instances.

Since I started working with microservices in 2016, I have continuously explored and refined different approaches to session management.

One question has remained particularly important to me:

How can every service independently verify a user's session without turning authentication into a bottleneck?

The Challenge: Security vs. Performance

In a microservices architecture, services frequently communicate with one another. From a zero-trust perspective, a service should not blindly trust a userId simply because another service included it in a request.

Instead, each service should validate the session and enforce the appropriate authorization rules.

However, querying a shared database every time a service needs to validate a session can increase latency and database load, especially when requests pass through multiple services.

This creates a practical engineering challenge: how do we preserve session validity and revocation capabilities while reducing the cost of repeated remote access?

That question led me to develop Sync Session, an open-source session management library for Spring Boot.

The Design

The design combines four main components:

RDBMS as shared storage: The persistent source of truth for session state.
Local cache: Each service instance maintains a local copy of session data to reduce repeated database queries.
Event service: Session changes trigger events that allow other service instances to invalidate their corresponding local cache entries.
Session ID propagation: The session ID can be passed through a cookie or HTTP header, depending on the application's request flow.

The session itself is intentionally kept small. Rather than embedding a large amount of user data and permissions into the session, the application uses a protected session ID to retrieve the associated authenticated identity.

User information and authorization data remain under the responsibility of the relevant systems, where they can be updated independently.

This separation helps avoid relying on outdated user information stored in a session and supports centralized session revocation.

What Does This Approach Support?

The project aims to address several practical requirements:

Distributed session management across multiple service instances.
Sliding session expiration, allowing the expiration time to be extended according to the configured policy.
Administrative session revocation, enabling an administrator to force a user to log out.
Synchronization of session invalidation across service instances.
Reduced database access through local caching.

The key idea is that services can validate sessions locally in the common case, while shared storage and event-driven invalidation help coordinate changes.

Performance Reference

In my MySQL load tests, I measured the following results under a workload involving 100 threads and 1 million sessions:

Operation	Measured throughput
Session creation	~15,533 operations/sec
Local cache access	~20.01 million operations/sec
Direct database queries	~69,278 operations/sec

All 1 million session creation operations completed successfully in this test, with no reported failures.

Under these particular test conditions, local cache access achieved approximately 289 times the throughput of direct database queries.

These numbers are reference measurements from my test environment, not universal performance guarantees. The results depend on hardware, database configuration, query patterns, cache implementation, concurrency, and the exact benchmark methodology.

The comparison also measures different access paths, so it should not be interpreted as an end-to-end authentication benchmark.

Consistency and Security Trade-offs

Local caching introduces an important limitation: event-driven invalidation is asynchronous.

When a session is revoked, another service instance may still have a cached copy until it processes the invalidation event. Event delivery failures or temporary network partitions can extend this window.

For security-sensitive systems, this behavior needs to be considered explicitly. Depending on the threat model, additional safeguards may be necessary, such as bounded cache lifetimes, reliable event delivery, recovery mechanisms, or synchronous validation for particularly sensitive operations.

Likewise, validating a session is not the same as authorizing every operation. Services must still enforce their own authorization requirements, and internal service communication must be authenticated appropriately.

The goal is not to claim that local caching removes every security or consistency challenge. It is to explore a practical balance between independent session validation, efficient reads, and distributed session invalidation.

A Design Principle Worth Discussing

The principle behind Sync Session can be summarized as:

Shared storage for correctness. Local cache for performance. Events for synchronization.

This is not the only valid architecture, and it may not be appropriate for every workload. I am sharing it because session management is a recurring challenge in distributed systems, and I believe the trade-offs deserve discussion beyond the implementation of any single library.

I'd especially like to hear from engineers who have built authentication infrastructure at scale:

How do you handle session revocation across multiple service instances?
Would you accept a short invalidation delay, or require immediate revocation for every request?
How do you balance database load, cache consistency, and security requirements?
What failure scenarios would you consider essential to address in this design?

I'm happy to discuss the implementation, benchmark methodology, and architectural trade-offs.

Why Does a Backend API Become Slow, and How Can We Fix It?

Wed, 21 Oct 2026 03:36:54 +0000

As a backend developer, I spend a lot of time thinking about what happens behind the scenes when someone uses an application. A user clicks a button, submits a form, or requests some information, and the backend has to process that request and return a response.

Everything looks simple when an application is small. But as the number of users and the amount of data increase, things can change quickly. An API that used to respond almost instantly may suddenly take several seconds to return the same information.

So, what actually makes a backend API slow, and how can we fix it?

The Problem: An API That Takes Too Long to Respond

Imagine you're building an e-commerce application using Java and Spring Boot. The application has an API that returns a list of products.

At first, the API works perfectly. It fetches product information from the database and returns the result to the frontend.

But after the application grows, users begin to complain that the product page takes too long to load.

There could be several reasons behind this problem:

The database is executing inefficient queries.
The API is retrieving more data than the frontend needs.
The application is making unnecessary database calls.
Multiple operations are being performed when they are not required.
The server is struggling to handle too many requests at once.

The first mistake would be to start changing the code without understanding the actual cause.

Before optimizing an API, we need to identify where the time is being spent.

Step 1: Measure the API Performance

The first thing I would do is measure the response time.

Instead of guessing, I would check application logs, database query execution times, and performance metrics to understand whether the delay comes from the database, application logic, or another part of the system.

For example, if an API takes 1,500 milliseconds to respond, I would investigate how much time is spent executing the database query and how much is spent processing the result.

This gives us a starting point.

In a Spring Boot application, tools such as Spring Boot Actuator and Micrometer can help expose useful application metrics. Database query logging and profiling can also help identify expensive operations.

The goal is simple: find the bottleneck before trying to fix it.

Step 2: Improve Database Queries

One common problem is retrieving unnecessary data.

Suppose the frontend only needs a product's name, price, and image URL. There is no reason to fetch every column from the database if the application doesn't need them.

A basic SQL query might look like this:

SELECT *
FROM products;

A more focused query would be:

SELECT name, price, image_url
FROM products;

This can reduce the amount of data transferred and processed, particularly when the table contains many columns or large fields.

Indexes can also improve performance when queries frequently filter or search by specific columns. However, indexes should be chosen based on actual query patterns because unnecessary indexes increase storage use and can slow down writes.

Small database improvements can make a noticeable difference when an API is called frequently.

Step 3: Avoid Unnecessary Database Calls

Another issue I would investigate is whether the application is making too many database queries.

For example, imagine an API returns 100 products. If the application executes one query to fetch the products and then another query for each product's category, it could make 101 queries instead of a smaller number of well-designed queries.

This is a common example of the N+1 query problem.

With Java and Spring Data JPA, the solution might involve a carefully designed fetch strategy, a projection, or a query that retrieves the required information together.

The right approach depends on the relationships between the entities and the data the endpoint actually needs.

I would also avoid blindly using eager loading everywhere, because loading too much related data can create a different performance problem.

Step 4: Return Only the Data the Client Needs

A backend API should have a clear responsibility: provide the information the client needs to complete a task.

Returning thousands of records in a single response is rarely a good choice when the user only needs to view a small portion of them.

Pagination can help.

For example, a Spring Data repository can support paginated queries:

Page<Product> findAll(Pageable pageable);

A service or controller can then request a specific page rather than loading every product at once.

For larger datasets, cursor-based pagination may be more suitable than offset-based pagination, depending on the use case.

The important thing is to make sure the API remains efficient as the dataset grows.

Step 5: Use Caching When It Makes Sense

Sometimes the same information is requested repeatedly even though it rarely changes.

Product categories, public configuration, and other relatively stable data may be good candidates for caching.

Spring provides caching support through annotations such as @Cacheable.

For example:

@Cacheable("productCategories")
public List<Category> getCategories() {
    return categoryRepository.findAll();
}

After the first successful lookup, subsequent calls can reuse the cached result, depending on the configured cache provider and cache rules.

However, caching is not a solution for every slow API. We need to consider expiration, invalidation, memory usage, and whether users might receive outdated information.

For frequently changing or sensitive data, caching requires additional care.

Step 6: Test the Changes

After making improvements, I would test the API again using the same conditions as before.

I would compare:

Average response time.
Higher-percentile latency, such as p95.
Database query count and execution time.
Error rates.
Resource usage under concurrent requests.

A change that makes one request faster but causes errors under heavy traffic is not a successful optimization.

Performance testing tools can help simulate multiple users and reveal problems that may not appear during ordinary development.

I would also keep automated tests in place to ensure that performance improvements don't accidentally change the API's expected behavior.

What I Learned From This Approach

The biggest lesson is that backend performance is not just about writing shorter code or choosing a faster programming language.

A well-designed API depends on several things working together: efficient database queries, sensible data retrieval, appropriate caching, reliable application logic, and good monitoring.

Java and Spring Boot provide useful tools for building APIs, but the framework cannot automatically make every design decision efficient.

As developers, we still need to understand what the application is doing and why a particular operation is taking too long.

I would rather spend time identifying the actual bottleneck than introduce unnecessary complexity in the name of optimization.

Final Thoughts

Building a backend API that works is one challenge. Keeping it reliable and responsive as the application grows is another.

When an API becomes slow, I think the best approach is to measure first, identify the problem, make a focused change, and test the result.

Good backend development is not only about making an application work today. It's about making sure it continues to work well as more users depend on it.

I'm interested in how other Java developers approach API performance. When you notice a slow endpoint in your application, what's the first thing you investigate: database queries, application logic, or something else?

From Writing Web Novels to Shipping Software — Nocklock’s Developer Journey

Thu, 08 Oct 2026 13:10:12 +0000

From Writing Web Novels to Shipping Software — Nocklock’s Developer Journey

“I think AI is most useful when it extends my thinking rather than replacing it.”

Nocklock didn't originally become a developer because programming was a lifelong childhood dream.

It started for a much more practical reason: making a living.

A few years ago, before AI became deeply integrated into everyday software development, Nocklock was looking for a practical way into the industry. In Korea, Java and Spring were widely used in enterprise and public-sector development, making them a natural choice.

But programming eventually became interesting for a reason that had little to do with career practicality.

Before becoming a developer, Nocklock had commercially published two web novels.

That experience would unexpectedly influence the way he thinks about software.

Writing fiction means creating a world, defining its rules, and allowing characters to interact within those rules.

Software isn't so different.

Programming languages have rules. Networks have rules. Data has structure. Business logic creates constraints. Developers build systems and user flows inside those constraints.

The connection between writing and programming gave Nocklock a different way of thinking about development.

Instead of seeing software only as code, he began seeing it as a system of possibilities.

And that perspective continues to influence how he builds today.

Programming Started as a Practical Choice

Nocklock's initial decision to learn programming was pragmatic.

Java and Spring offered a realistic path into professional software development, particularly within Korea's enterprise and public-sector technology landscape.

But choosing a technology for practical reasons doesn't necessarily mean remaining interested for practical reasons.

As Nocklock gained experience, he discovered that programming gave him something he had already experienced through writing:

the ability to create systems governed by rules.

When writing a novel, he might ask:

What if I were the character?

What if the situation were different?

What would happen if the rules changed?

He now brings similar questions into software development.

When building something, he tries to imagine different situations and perspectives.

What if he were the user?

What if the user knew nothing about the subject?

What if the environment were different?

What if an assumption turned out to be wrong?

That habit of questioning has become part of how he approaches development.

Learning by Shipping

One of Nocklock's strongest beliefs is reflected directly in his profile:

learning by shipping.

He discovered why this approach worked for him while studying for Korea's Engineer Information Processing qualification.

Much of the preparation involved studying and memorizing theory.

But he noticed something interesting.

He could forget information he had memorized for an exam.

Yet he could still remember errors he had encountered while working, why they happened, and how he eventually solved them.

The difference was experience.

Knowledge connected to a real problem stayed with him.

That realization changed how he approached learning.

When he encountered an AI development competition, he began building rather than waiting until he had learned everything he thought he needed to know.

He learned what was necessary while developing the project.

He made mistakes.

He investigated them.

He fixed them.

And eventually, he shipped.

That's what learning by shipping means to him.

It doesn't mean abandoning study.

It means turning knowledge into experience.

Security Lens: When the MVP Became Real

One of the clearest examples of this philosophy was Security Lens.

The original idea sounded relatively straightforward.

A user would upload an image, OCR would extract information, the system would identify potential privacy risks, and the user would receive useful results.

But building the real product exposed a much messier reality.

Real files and images didn't always behave as expected.

OCR accuracy wasn't always reliable.

Unexpected problems kept appearing.

And once the core functionality finally worked, Nocklock realized something important:

The main feature working didn't mean the product was finished.

It meant the next phase had started.

There were still improvements to make, problems to investigate, and a deadline approaching.

That experience changed his understanding of what it means to ship software.

Building the main feature is only one milestone.

Shipping means confronting the difference between:

the product you imagined

and

the product that actually exists.

The README Became a Product Review

Nocklock initially thought writing the README would simply mean documenting what he had already built.

Instead, documentation exposed problems in the product itself.

During development, he had often moved quickly from one problem to the next.

Get something working.

Move forward.

Solve the next issue.

Keep building.

But when he tried to explain the project clearly, he started noticing gaps.

Sometimes he would describe a feature and suddenly question whether the feature was necessary at all.

That changed the purpose of the README.

It was no longer just documentation.

It became a way of reviewing the product.

Writing forced him to examine:

The project structure
Individual features
Development decisions
The reasoning behind those decisions
Whether certain features were actually necessary

In that sense, documentation became another form of development.

Sometimes explaining what you built reveals whether you should have built it in the first place.

A New Laptop Revealed Hidden Complexity

Another unexpected lesson came from something much more ordinary:

setting up a new laptop.

Moving from an established development environment to a clean machine exposed how much invisible state had accumulated over time.

JDK versions.

PATH settings.

Git configuration.

IDE settings.

Build synchronization.

Environment variables.

Small setup decisions that had gradually become invisible.

On the old machine, those things simply worked.

On the new one, they didn't.

Suddenly, when something behaved unexpectedly, Nocklock had to ask:

Is this a problem with my code?

Is it the build state?

The JDK?

The IDE?

The environment configuration?

That experience led to an important realization:

The development environment is part of the system too.

A clean machine can reveal what you genuinely understand and what your previous environment had quietly been doing for you.

Debugging Doesn't Stop at the Code

Deployment reinforced the same lesson.

Local development can feel comfortable.

Then the application gets deployed.

Suddenly, ports, environment variables, runtime differences, external services, configuration and hidden assumptions all become part of the problem.

Nocklock experienced this while deploying Security Lens to Railway.

It also helped him understand why tools such as Docker became so important.

Docker doesn't eliminate every deployment problem.

But making the runtime environment reproducible can remove a significant category of differences between machines.

As a result, his approach to debugging has expanded.

He doesn't look only at the code anymore.

He also considers:

Configuration
Runtime
Environment
External dependencies
Deployment conditions
Assumptions connecting those pieces

Sometimes the bug isn't in the code.

Sometimes the code is behaving exactly as written inside an environment you misunderstood.

Why Java and Spring Still Matter

Nocklock originally chose Java and Spring because they were practical.

After working with them professionally, however, he developed a different appreciation for the ecosystem.

What stands out to him is its maturity and structure.

Backend services can quickly become complicated.

They may involve:

Business logic
Databases
APIs
Security
External integrations
Multiple requirements
Different environments

In that situation, conventions and proven patterns can become extremely valuable.

Spring provides abstractions that help developers build quickly while still leaving a deeper ecosystem underneath those abstractions to understand.

But Nocklock doesn't believe Java or Spring should automatically be used for every problem.

The technology should fit the problem.

That idea connects directly to one of his broader principles:

follow the problem, not a fixed stack.

AI Should Extend Thinking, Not Replace It

Nocklock's approach to AI is deliberately different from simply asking an AI model to write everything.

His first step isn't:

“Write this for me.”

Instead, he tries to explain what he's curious about, why he's curious about it, and how he thinks the pieces might connect.

Then he asks AI to help explain the subject.

Only after understanding the direction does he use AI for implementation, code review, identifying mistakes or suggesting improvements.

That distinction matters to him.

AI can provide an enormous amount of knowledge.

Its ability to interact with tools and take action is also growing quickly.

But Nocklock wants the important questions, judgment and decisions to remain his responsibility.

The ideal relationship isn't human versus AI.

It's human thinking enhanced by AI.

AI becomes a multiplier for understanding rather than a substitute for understanding.

Following the Problem Instead of the Stack

Nocklock doesn't want technology to be the starting point of a project.

He starts with the problem.

What is being built?

How quickly does it need to be built?

What constraints exist?

What can realistically be maintained?

What can be verified?

Sometimes Java and Spring are the right choice because they are familiar and fit the requirements.

Sometimes another technology is better.

AI has made experimenting with unfamiliar technologies easier, but that doesn't mean every new technology deserves to be added to a project.

In fact, sometimes following the problem means deliberately choosing something familiar and boring.

That's an important distinction.

Being flexible doesn't mean constantly chasing new technology.

It means choosing technology based on what the problem actually requires.

The Cost of Moving Too Fast

One mistake Nocklock continues to notice in himself is focusing heavily on implementation and assuming the structure can always be cleaned up later.

Security Lens made this particularly obvious.

Because of the competition deadline, he often solved the next problem in front of him and moved on.

That approach helped maintain momentum.

But later, while documenting the project, he could see where the structure had become messy.

He could also see that some features probably didn't need to exist.

The lesson wasn't that moving quickly was wrong.

It was that speed doesn't remove the cost of decisions.

It simply delays when you see that cost.

Today, Nocklock tries to create small moments during development where he can step outside the code and ask:

Does what I'm building still make sense?

That pause can prevent a lot of future cleanup.

What He's Exploring Next

Nocklock's interests are increasingly moving toward the intersection of:

Backend development.

AI.

Security.

Real products.

Security Lens began as a tool for detecting privacy risks in files before they are uploaded.

Working on it opened up broader questions about digital content.

How do we know whether an image or piece of content is authentic?

Where did it come from?

Was it generated or modified by AI?

Can AI-generated media be used to imitate another person?

These questions have made areas such as content provenance and identity risk particularly interesting to him.

At the same time, he remains interested in building smaller experimental products.

The goal is to test an idea quickly, receive real feedback and then decide whether the idea deserves further investment.

Build Something Smaller

When asked what advice he would give other developers, Nocklock's answer is straightforward:

Build something smaller than the thing you originally wanted to build.

It's easy to remain in preparation mode.

There is always another:

Course
Framework
Tutorial
Concept
Technology

that could be learned first.

But developers don't need to understand everything before starting.

Build something small enough to finish.

Let it break.

Deploy it.

Let another person use it.

Then examine what goes wrong.

Those problems will often tell you what you genuinely need to learn next.

For Nocklock, one real error that requires investigation can teach more than memorizing ten definitions.

Experience creates context.

Context makes knowledge stick.

Becoming More Than a Framework Developer

Nocklock doesn't want to become a developer defined by a single framework.

Java and Spring are an important foundation, but his longer-term goal is broader.

He wants to become someone who can move comfortably between:

Engineering
AI
Product thinking
Deployment
Problem solving
Communication

He wants to understand a problem, choose appropriate tools, build the system, deploy it, explain it clearly and improve it based on what actually happens.

In other words:

Not simply someone who writes code.

Someone who can take an idea and keep working on it until it becomes something another person can actually use.

From Writing Stories to Building Systems

There is an interesting connection running through Nocklock's entire journey.

Before software, there was storytelling.

He created fictional worlds, defined their rules and imagined what could happen inside them.

Then came programming.

Different syntax.

Different constraints.

Different tools.

But a surprisingly similar creative process.

Software also involves defining rules, creating systems and imagining what happens when different pieces interact.

Today, AI adds another layer to that process.

It can help him explore unfamiliar concepts, test ideas and move faster.

But he still wants to be the person asking the questions.

The person deciding what matters.

The person choosing what to build.

And ultimately, the person responsible for shipping it.

Knowing What Not to Build

When Nocklock first started programming, he thought becoming a better developer mostly meant knowing more.

More syntax.

More frameworks.

More technologies.

His perspective has changed.

Technical knowledge still matters.

But software development is also about understanding the problem, questioning assumptions, debugging beyond the code, communicating decisions, working within constraints and actually shipping something people can use.

He has also learned that knowing what not to build can be just as important as knowing how to build something.

Code remains at the center of the work.

But it isn't the whole job anymore.

The Developer Journey Continues

Nocklock's journey began with a practical career decision.

It developed through Java and Spring.

It was shaped by an earlier life as a published web novelist.

It accelerated through learning by shipping.

Security Lens taught him the difference between building an MVP and actually shipping a product.

Documentation taught him to question his own decisions.

Deployment taught him that the environment is part of the system.

AI taught him another way to learn and work.

And his current interests are taking him toward the intersection of backend engineering, AI, security and real-world products.

His philosophy can be summed up simply:

Start with the problem.

Build something small.

Let reality test it.

Learn from what breaks.

Ship it.

Then do it again.

About Nocklock

Nocklock is a developer building useful things with Java, Spring and AI.

His approach centers on learning by shipping, documenting what works and following the problem rather than committing to a fixed technology stack.

His current interests include backend development, AI, security, content provenance and experimental product development.

View Nocklock's profile on CoderLegion

Visit Nocklock's website

CoderLegion Developer Stories

Real developers. Real journeys. Real experiences.

Developer Stories at CoderLegion gives developers a place to share how they think, what they build, the problems they encounter and what they learn along the way.

Interview & Story by Mehadi Hasan

I Built the MVP. Then I Realized Shipping Was a Different Skill.

Thu, 24 Sep 2026 13:44:30 +0000

I Built the MVP. Then I Realized Shipping Was a Different Skill.

A few days ago, I wrote about something I had learned the hard way:

Building the MVP first and writing the README later made me realize how much I had assumed.

I thought that was mostly a documentation problem.

It wasn't.

The deeper problem was that building something and shipping something are two different skills.

The MVP was working

I recently built Security Lens, a Spring Boot-based privacy risk analyzer for images and documents.

The basic flow was working:

Image Upload
      ↓
OCR
      ↓
Personal Information Detection
      ↓
EXIF Metadata Analysis
      ↓
Risk Scoring
      ↓
LLM-assisted Analysis
      ↓
Risk Explanation

It could detect things such as:

phone numbers
email addresses
EXIF GPS information
capture timestamps
device metadata

and turn the results into a simple privacy risk score.

At that point, I had something that worked.

I thought I was almost done.

I wasn't.

"It works" was only the beginning

Once I tried to actually ship the project, a different set of questions appeared.

Not:

Does this method work?

But:

Which exact version is deployed?

What happens if an environment variable is missing?

Did I actually verify the production configuration?

What if the database or external integration behaves differently from localhost?

What evidence do I have that the deployed application is healthy?

Can I roll back?

Those questions were not really about writing more code.

They were about operating the thing I had already written.

Deployment exposed assumptions

Localhost is forgiving.

Your machine already has:

your environment variables
your tools
your configuration
your ports
your dependencies
your development habits

Production does not care about any of that.

It only gets what you actually configured.

While working through deployment, I ran into issues around ports, environment setup, AI integration, and the difference between an implementation that works locally and one that can actually run as a product.

That changed how I looked at "done."

AI made this even more interesting

I use AI tools heavily while building.

They are extremely useful.

But I also noticed a dangerous pattern.

It is very easy to ask:

"Is this production-ready?"

and get an answer that sounds convincing.

The problem is that confidence is not evidence.

So I started using a different rule:

If the evidence is missing, the result is NOT VERIFIED.

Instead of asking AI to make the release decision for me, I ask it to help inspect:

configuration
logs
build output
deployment settings
security assumptions
missing evidence

The AI can be a reviewer.

It should not silently become the owner of the release decision.

That became a checklist

While repeatedly asking myself the same deployment questions, I started writing them down.

Eventually, those notes became a small public project:

Spring Boot Release Checklist

It focuses on things like:

clean builds
expected tests
configuration and secrets
database connectivity
external integrations
production smoke tests
rollback points
evidence-first AI review

The main idea is simple:

A successful deployment is not the same as a verified release.

Then I noticed something else

The checklist helped me catch blockers.

But I still wanted a better way to answer:

What did I actually verify?

Where is the evidence?

What still remains uncertain?

Should I ship this version?

That eventually became a more structured release workflow with interactive checks, evidence fields, AI review prompts, and ship / no-ship decisions.

I ended up turning it into a 58-page deployment runbook.

That was another new experience for me.

For the first time, I was not only building software.

I was taking something I had learned while building software and turning it into something another developer could actually use.

Shipping forced me to learn things coding alone did not

The interesting part was not setting up a product page.

It was everything around it.

I had to think about:

what the product actually solves
how to explain it in one sentence
what should remain free
what someone would reasonably pay for
how a buyer receives the files
how the download flow works
what happens after payment
how the public GitHub repository connects to the full version

None of these problems required a complicated algorithm.

But they still required engineering decisions.

My definition of "done" is changing

Before:

Feature works
→ done

Now:

Feature works
→ build succeeds
→ configuration is verified
→ deploy
→ production smoke test
→ evidence recorded
→ rollback understood
→ documentation updated
→ then maybe done

It is slower.

But it feels much closer to building something real.

One more thing I learned

I used to think "shipping" was mostly the final step after development.

Now I think it changes how you develop from the beginning.

When you know you will actually deploy something, you think differently about:

configuration
secrets
observability
failure cases
reproducibility
documentation
users

The project stops being just code.

It becomes a system someone else has to trust.

What's next

I am continuing to improve Security Lens and experimenting with the same workflow on other small products.

The Security Lens source code remains private, but I created a public technical case study here:

Security Lens — Technical Case Study

And the lightweight Spring Boot release workflow is available here:

Spring Boot Release Checklist

For me, the biggest lesson so far is simple:

Coding makes the product possible. Shipping makes it real.

Still learning.
Still shipping.

I Built the MVP First. Then I Wrote the README.

Thu, 24 Sep 2026 13:42:28 +0000

I did this in the wrong order.

At least, that was my first thought.

I was building a small project that checks images and documents for privacy risks. The backend came first, then the API, then a frontend that finally made it feel like an actual service.

Somewhere along the way, I realized something.

I had built quite a lot before properly organizing the requirements and documentation.

Normally, I would expect the order to look more like this:

requirements → design → implementation → testing → documentation

Mine looked closer to this:

build → break something → fix it → build more → test it → finally organize everything

Not exactly textbook development.

But writing the documentation afterwards turned out to be more useful than I expected.

The README became a way to inspect the project

At first, I thought I was just writing down what I had already built.

Instead, I kept finding things that were unclear.

What does this API actually guarantee?

What configuration is required before running the project?

What happens when an external dependency is unavailable?

Which parts have actually been tested?

Would someone other than me know how to start this project?

These are simple questions.

They become much harder to ignore when you have to explain the project in plain English.

And that was probably the most useful part.

Documentation stopped being something I needed to finish after development.

It became another form of testing.

AI made building faster, but that created a different problem

I also used AI quite heavily while building.

ChatGPT, Claude, and coding tools made it much easier to move from an idea to working code.

That part was great.

But faster implementation also meant I could move past decisions without really stopping to organize them.

A configuration works, so I move on.

An endpoint returns the right response, so I move on.

Docker starts successfully, so I move on.

Then deployment comes.

Suddenly all the small things I skipped become visible again.

Environment variables.

Ports.

Database configuration.

Build commands.

Health checks.

Logs.

Security settings.

Things that looked like small details on localhost started feeling much more important once I tried to get the application outside my own machine.

That was the point where I started thinking less about:

“Does this code work?”

and more about:

“Is this actually ready to leave localhost?”

There is a bigger difference between those two questions than I expected.

A working MVP is not the same as a shippable MVP

I used to think of an MVP mainly in terms of features.

If the core function works, the MVP works.

I still think that is partly true.

But there is another stage after that.

The application has to survive being run somewhere other than your laptop.

Someone needs to know how to configure it.

You need to know when it has failed.

Secrets should not be sitting in the repository.

The build should be reproducible.

The production environment should not depend on something you happened to configure manually three days ago and already forgot about.

None of these things are particularly exciting.

They are also exactly the things I kept running into.

So I started writing them down.

Not as another tutorial.

Just as a checklist I wanted for myself the next time I reached this stage.

That checklist became its own project

Eventually those notes turned into a small release workflow.

I made a lightweight version public on GitHub:

Spring Boot Release Checklist

It is a practical pre-release checklist for Spring Boot MVPs moving from localhost to production.

The idea is simple:

before calling an MVP ready, go through the things that are easy to miss when most of your attention has been on making the feature work.

I later expanded the same idea into a more structured 58-page deployment runbook, but the GitHub checklist remains the lightweight public version.

I did not originally plan to make a product out of it.

It came from repeatedly asking myself the same question while building:

“What am I forgetting before I ship this?”

I still don't think building first is the correct order

I would not turn this into advice like:

“Forget requirements. Just start coding.”

That would be the wrong lesson.

If anything, doing it backwards showed me why the earlier steps exist.

But there was something valuable about documenting a project after I had actually struggled with it.

The checklist was no longer theoretical.

Every item reminded me of something I had needed, forgotten, broken, or had to go back and check.

Maybe next time my process will be a little cleaner.

Probably not perfectly clean.

And that's fine.

I'm still learning by making things.

I keep more notes about development, AI tools, deployment, and the things that break along the way on the Nocklock blog.

You can also see the projects I'm building on GitHub

Spring AI Evals: how I test agent behavior

Wed, 23 Sep 2026 21:55:02 +0000

Part 2: https://dev.to/lbobylev/baseline-for-spring-ai-evals-catching-silent-degradation-4ll6

When building an AI agent, there is usually a moment when the prompt seems to work. But as development continues, this can quickly get out of control. Today the agent can call the right tool. Tomorrow, after a small system prompt change, it can stop calling it. Later, it can start calling it when it should not. So I treat evals as regular regression tests for AI behavior.

Here is a small example: a Spring AI agent must create a Markdown note through the create_note tool. The task looks simple, but there are already several ways the agent can fail:

not call the tool when needed, or call it when it is not needed;
create an extra file;
produce a note in the wrong format;
add extra advice or any irrelevant content.

So I split the checks into a few layers.

Success Criteria

First I define what correct behavior means. The agent should:

call create_note when the user asks to create a new note;
not call create_note when the user asks to read, delete, rename, or explain;
create only note.txt;
put a heading on the first line;
write short, relevant, and readable content.

This matters: without clear success criteria, an eval becomes a subjective check.

Tool Definition

The tool's scope is intentionally narrow: creating the note.txt file.

@Tool(
    name = "create_note",
    description = "Create a new Markdown note in note.txt. Use this only when the user asks to create a new note."
)
public String createNote(String title, String body) {
    var notePath = this.notesDirectory.resolve("note.txt");

    var content = "# " + normalize(title, "New Note")
            + System.lineSeparator()
            + System.lineSeparator()
            + normalize(body, "No content provided.")
            + System.lineSeparator();

    try {
        Files.createDirectories(this.notesDirectory);
        Files.writeString(notePath, content, CREATE, TRUNCATE_EXISTING);
        return "Created note.txt.";
    }
    catch (IOException ex) {
        return "Failed to create note.txt: " + ex.getMessage();
    }
}

The model uses the name and description input parameters to determine whether it needs to call the tool.

Agent Instructions

The process goals are formulated in the system prompt, which establishes the contract:

when to use the tool;
which file must appear;
what must not appear in the result.

return this.chatClient.prompt()
        .system("""
            When the user asks to create a new note, use the create_note tool.
            The tool must create note.txt, put a Markdown heading on the first line,
            add the body text, and create no extra files.
            Keep the note concise and do not add unrelated suggestions.
            """)
        .user(message)
        .call()
        .content();

Deterministic Checks

The first layer checks tool invocation. This is no different from a normal unit test.

@ParameterizedTest
@CsvSource(value = {
    "note-create-01|true|Create a new note about my plans for the week",
    "note-create-02|true|Make a markdown note with the title Retro and the body: what to improve",
    "note-no-trigger-01|false|Show me the existing notes",
    "note-no-trigger-02|false|Read the note.txt file",
    "note-no-trigger-03|false|Delete the note.txt file",
    "note-no-trigger-04|false|Draft the text of a note, but do not create a file"
}, delimiter = '|')
void evaluatesWhetherAgentCallsNoteTool(String id, boolean shouldTrigger, String prompt) {
    doReturn("Created note.txt.").when(this.noteTool).createNote(anyString(), anyString());
    clearInvocations(this.noteTool);

    var response = this.chatController.chat(prompt);

    assertThat(response).as(id).isNotBlank();

    verify(this.noteTool, shouldTrigger ? atLeastOnce() : never()).createNote(anyString(), anyString());
}

Tool Unit Test

Next, I test the tool itself without the model. This is also a normal unit test.

@Test
void createNoteWritesOnlyNoteFileWithMarkdownHeading() throws IOException {
    var noteTool = new NoteTool(this.tempDir);

    var result = noteTool.createNote("Meeting Notes", "Discuss the evaluation setup.");

    var notePath = this.tempDir.resolve("note.txt");
    var content = Files.readString(notePath);
    var lines = Files.readAllLines(notePath);

    try (var fileStream = Files.list(this.tempDir)) {
        var files = fileStream.toList();

        assertThat(result).isEqualTo("Created note.txt.");
        assertThat(notePath).exists();
        assertThat(content).isNotBlank();
        assertThat(lines.getFirst()).startsWith("#");
        assertThat(files).containsExactly(notePath);
    }
}

This separates responsibilities: if the unit test fails, the problem is in the tool implementation rather than the agent's behavior.

Rubric-Based Assessment

Deterministic checks are good at catching facts, but not everything can be checked this way. For example, "the note is relevant," "there is no extra information," and "the text is easy to read" are qualitative requirements. Rubric-based assessment is used for these. In Spring AI, this is conveniently implemented with Evaluator: it receives an EvaluationRequest, runs a grader prompt inside, and returns an EvaluationResponse with pass, score, feedback, and metadata. The grader model is fixed, ideally using a versioned model name. For example, gpt-5.5-2026-04-23 instead of gpt-5.5. If the provider updates a model alias, the grader can become stricter or more lenient without any code change. In that case, a score change may be caused by grader drift rather than agent behavior. A low temperature is also preferable for grading. The grader should not be creative; it should apply the same rubric consistently. For rubric-based assessment, the grader runs multiple times. The pass rate better reflects its behavior.

class NoteStyleEvaluator implements Evaluator {

    private static final List<String> REQUIRED_CHECK_IDS =
            List.of("title", "relevance", "simplicity", "readability");

    private final ChatClient chatClient;

    private final int minimumScore;

    NoteStyleEvaluator(ChatClient.Builder chatClientBuilder, int minimumScore) {
        this.chatClient = chatClientBuilder.build();
        this.minimumScore = minimumScore;
    }

    @Override
    public EvaluationResponse evaluate(EvaluationRequest evaluationRequest) {
        var rubricResult = gradeNote(evaluationRequest);

        if (rubricResult == null) {
            return new EvaluationResponse(false, 0.0f, "No rubric result", Map.of());
        }

        var pass = rubricResult.overallPass()
                && rubricResult.score() >= this.minimumScore
                && hasRequiredPassingChecks(rubricResult.checks());

        var metadata = Map.<String, Object>of(
                "overallPass", rubricResult.overallPass(),
                "checks", rubricResult.checks());

        return new EvaluationResponse(pass, rubricResult.score(), rubricResult.toString(), metadata);
    }

    private StyleRubricResult gradeNote(EvaluationRequest evaluationRequest) {
        var graderPrompt = """
            Read the original user request and the note.txt content,
            then grade the note against these requirements:

            - it has a clear title;
            - the content is relevant to the original user request;
            - there is no clearly unnecessary information;
            - the text is easy to read.

            Return only a structured JSON object with these fields:
            - overall_pass: boolean
            - score: integer from 0 to 100
            - checks: exactly four items with ids title, relevance, simplicity, readability.

            Each check must include id, pass, and notes.

            original user request:
            %s

            note.txt content:
            %s
            """.formatted(
                evaluationRequest.getUserText(),
                evaluationRequest.getResponseContent()
            );

        return this.chatClient.prompt()
                .user(graderPrompt)
                .call()
                .entity(StyleRubricResult.class);
    }

    private boolean hasRequiredPassingChecks(List<StyleRubricCheck> checks) {
        if (checks == null || checks.size() != REQUIRED_CHECK_IDS.size()) {
            return false;
        }

        var checkIds = checks.stream()
                .map(StyleRubricCheck::id)
                .toList();

        return checkIds.containsAll(REQUIRED_CHECK_IDS)
                && checks.stream().allMatch(check ->
                        check.pass() && check.notes() != null && !check.notes().isBlank());
    }

    record StyleRubricResult(
            @JsonProperty("overall_pass") boolean overallPass,
            int score,
            List<StyleRubricCheck> checks) {
    }

    record StyleRubricCheck(
            String id,
            boolean pass,
            String notes) {
    }
}

Structured output enables automated verification. With Evaluation in Spring AI, grading becomes a normal Java contract:

EvaluationRequest stores the user prompt and generated output;
Evaluator contains the grading logic;
EvaluationResponse returns a machine-checkable result;
metadata stores details for each rubric check.

This makes grading convenient to integrate into CI: there is pass, score, feedback, and a list of checks.

@ParameterizedTest
@CsvSource(value = {
    "weekly-plan-01|Create a new note about my plans for the week. Include a clear title and only a concise list of concrete weekly goals.|80",
    "weekly-plan-02|Create a markdown note for my weekly planning. Mention only work, health, and learning goals. Do not add extra categories.|80"
}, delimiter = '|')
void gradesCreatedNoteWithStyleRubric(String id, String prompt, int minimumScore) throws IOException {
    var agentResponse = this.chatController.chat(prompt);

    assertThat(agentResponse).as(id).isNotBlank();
    assertThat(this.notePath).as(id).exists();

    var noteContent = Files.readString(this.notePath);

    var evaluator = new NoteStyleEvaluator(this.chatClientBuilder, minimumScore);

    int graderRuns = 3;
    int requiredPassingRuns = 2;

    var results = IntStream.range(0, graderRuns)
            .mapToObj(run -> evaluator.evaluate(new EvaluationRequest(prompt, noteContent)))
            .toList();

    long passingRuns = results.stream()
            .filter(EvaluationResponse::isPass)
            .count();

    assertThat(passingRuns)
            .as(id + ": " + results)
            .isGreaterThanOrEqualTo(requiredPassingRuns);
}

The agent runs once and creates one note. The grader then evaluates the same note.txt content three times; the test passes if at least two evaluations pass. This approach makes the grader result more robust against random model variation.

Saving Results Between Versions

It is also advisable to save eval results between runs. This helps compare behavior after changes to the prompt, model, or rubric. Pass/fail status is not always enough. A test can still pass while the score gradually gets worse. That is why I keep the score, feedback, and rubric metadata as artifacts.

var resultPath = Path.of("build", "eval-results", id + ".json");

var resultJson = objectMapper.writeValueAsString(Map.of(
        "id", id,
        "prompt", prompt,
        "minimumScore", minimumScore,
        "graderRuns", graderRuns,
        "requiredPassingRuns", requiredPassingRuns,
        "passingRuns", passingRuns,
        "results", results
));

Files.createDirectories(resultPath.getParent());
Files.writeString(resultPath, resultJson);

Separating Evals from Regular Tests

It is a good idea to keep AI evals separate from regular tests. They call the model API, so they are slower, more expensive, and less deterministic.

tasks.named('test') {
    useJUnitPlatform {
        excludeTags 'eval'
    }
}

tasks.register('evalTest', Test) {
    description = 'Runs AI agent evaluation tests.'
    group = 'verification'

    useJUnitPlatform {
        includeTags 'eval'
    }

    shouldRunAfter test
}

Run them separately:

./gradlew test
./gradlew evalTest

What This Gives

This makes development more predictable and manageable. When the system prompt, the tool description or implementation, or model parameters change, evaluating agent behavior remains much less subjective. CI tracks:

whether the number of false positives increased;
whether false negatives appeared;
whether the expected artifact was created;
whether the output passes the rubric assessment;
whether quality dropped below the minimum score.

The most useful part is that the eval suite can grow gradually. Found a new miss? Add a new eval case. The agent failed on a real user prompt? That prompt becomes a regression test. This way, behavior drift becomes visible instead of staying hidden in manual testing.

Spring Boot 4 + Spring AI + Langfuse: Tracing LLM Calls and Tool Calling

Wed, 23 Sep 2026 21:53:21 +0000

Part 2: Spring AI + Langfuse: Tracing Streaming with ObservationFilter

In this post, we will look at how to connect Spring AI with Langfuse and get complete tracing for LLM calls, tool calling, prompts, and responses.

Below is a demo project built with Spring Boot and Spring AI: an HTTP request invokes an LLM, WeatherTools is used when needed to fetch weather data from the internet, and traces, prompts, and responses are sent to Langfuse through OpenTelemetry and CallAdvisor.

Langfuse is a platform for monitoring and evaluating AI applications. The experience is very similar to LangSmith: you can trace LLM calls, inspect inputs and outputs, track latency and cost, and compare prompts and models. Langfuse also has an open-source version and can be self-hosted, which makes it a good choice for teams that need more control over their data and infrastructure.

The main dependencies in build.gradle:

dependencyManagement {
    imports {
        mavenBom 'org.springframework.ai:spring-ai-bom:2.0.0'
    }
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web:4.1.0'
    implementation 'org.springframework.boot:spring-boot-starter-opentelemetry:4.1.0'
    implementation 'org.springframework.ai:spring-ai-starter-model-openai:2.0.0'

    compileOnly 'io.opentelemetry:opentelemetry-exporter-otlp:1.62.0'

    testImplementation 'org.springframework.boot:spring-boot-starter-test:4.1.0'
}

Application Configuration

In application.properties, configure OpenAI, Spring AI observability, and the Langfuse OTLP endpoint:

spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.options.model=gpt-5
spring.ai.tools.observations.include-content=true
spring.ai.chat.client.observations.log-prompt=true
spring.ai.chat.client.observations.log-completion=true
spring.ai.chat.observations.log-prompt=true
spring.ai.chat.observations.log-completion=true

management.otlp.metrics.export.enabled=false
management.tracing.sampling.probability=1.0
management.tracing.export.otlp.enabled=${LANGFUSE_TRACING_ENABLED:false}
management.opentelemetry.tracing.export.otlp.endpoint=${LANGFUSE_OTLP_ENDPOINT:http://localhost:3000/api/public/otel}
management.opentelemetry.tracing.export.otlp.transport=http

langfuse.public-key=${LANGFUSE_PUBLIC_KEY:}
langfuse.secret-key=${LANGFUSE_SECRET_KEY:}

Since Langfuse is used for tracing, it is better to disable OTLP metric export. Otherwise, the application may try to send metrics to the same endpoint and get a ConnectException.

management.otlp.metrics.export.enabled=false

Controller

LangfusePromptTracingAdvisor adds the prompt and response to the Langfuse trace.

@RestController
class HelloAiController {

    private final ChatClient chatClient;
    private final WeatherTools weatherTools;

    HelloAiController(
            ChatClient.Builder chatClientBuilder,
            WeatherTools weatherTools,
            LangfusePromptTracingAdvisor langfusePromptTracingAdvisor
    ) {
        this.chatClient = chatClientBuilder
                .defaultAdvisors(langfusePromptTracingAdvisor)
                .build();

        this.weatherTools = weatherTools;
    }

    @GetMapping("/ai")
    String ai(
            @RequestParam(defaultValue = "Say hello in one short sentence")
            String message
    ) {
        return this.chatClient.prompt()
                .user(message)
                .tools(this.weatherTools)
                .call()
                .content();
    }
}

Tools

The tool uses Open-Meteo to find the coordinates of a city and retrieve the current weather for those coordinates.

@Component
class WeatherTools {

    private final RestClient restClient;

    WeatherTools(RestClient.Builder restClientBuilder) {
        this.restClient = restClientBuilder.build();
    }

    @Tool(description = "Get current weather for a city using live internet weather data")
    String getCurrentWeather(
            @ToolParam(description = "City name, for example Berlin or New York")
            String city
    ) {
        GeocodingResponse geocoding = this.restClient.get()
                .uri(
                        "https://geocoding-api.open-meteo.com/v1/search"
                                + "?name={city}&count=1&language=en&format=json",
                        city
                )
                .retrieve()
                .body(GeocodingResponse.class);

        if (geocoding == null
                || geocoding.results() == null
                || geocoding.results().isEmpty()) {
            return "Weather is unavailable: city not found: " + city;
        }

        Location location = geocoding.results().getFirst();

        ForecastResponse forecast = this.restClient.get()
                .uri(
                        "https://api.open-meteo.com/v1/forecast"
                                + "?latitude={latitude}"
                                + "&longitude={longitude}"
                                + "&current=temperature_2m,relative_humidity_2m,"
                                + "wind_speed_10m,weather_code",
                        location.latitude(),
                        location.longitude()
                )
                .retrieve()
                .body(ForecastResponse.class);

        if (forecast == null || forecast.current() == null) {
            return "Weather is unavailable for " + location.name();
        }

        CurrentWeather current = forecast.current();

        return "Current weather in %s, %s: %.1f C, humidity %d%%, "
                + "wind %.1f km/h, weather code %d."
                .formatted(
                        location.name(),
                        location.country(),
                        current.temperature2m(),
                        current.relativeHumidity2m(),
                        current.windSpeed10m(),
                        current.weatherCode()
                );
    }

    private record GeocodingResponse(List<Location> results) {
    }

    private record Location(
            String name,
            String country,
            double latitude,
            double longitude
    ) {
    }

    private record ForecastResponse(CurrentWeather current) {
    }

    private record CurrentWeather(
            @tools.jackson.annotation.JsonProperty("temperature_2m")
            double temperature2m(),

            @tools.jackson.annotation.JsonProperty("relative_humidity_2m")
            int relativeHumidity2m(),

            @tools.jackson.annotation.JsonProperty("wind_speed_10m")
            double windSpeed10m(),

            @tools.jackson.annotation.JsonProperty("weather_code")
            int weatherCode()
    ) {
    }
}

Sending Traces to Langfuse Through OpenTelemetry

Spring Boot sends traces through the spring-boot-starter-opentelemetry dependency. Langfuse requires Basic Authentication using the public and secret keys. The required environment variables are:

export LANGFUSE_TRACING_ENABLED=true
export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...

A separate configuration adds the required HTTP headers:

@Configuration
class LangfuseTracingConfig {

    @Bean
    OtlpHttpSpanExporterBuilderCustomizer langfuseOtlpHeaders(
            @Value("${langfuse.public-key:}") String publicKey,
            @Value("${langfuse.secret-key:}") String secretKey
    ) {
        return builder -> {
            if (!StringUtils.hasText(publicKey)
                    || !StringUtils.hasText(secretKey)) {
                return;
            }

            String token = Base64.getEncoder()
                    .encodeToString(
                            (publicKey + ":" + secretKey)
                                    .getBytes(StandardCharsets.UTF_8)
                    );

            builder.addHeader("Authorization", "Basic " + token);
            builder.addHeader("x-langfuse-ingestion-version", "4");
        };
    }
}

Adding Prompts to Langfuse

Spring AI does not put the prompt and completion into the Langfuse input and output fields in the required format. To solve this, we use the standard Spring AI CallAdvisor mechanism.

@Component
class LangfusePromptTracingAdvisor implements CallAdvisor {

    private final ObjectMapper objectMapper = new ObjectMapper();

    @Override
    public ChatClientResponse adviseCall(
            ChatClientRequest chatClientRequest,
            CallAdvisorChain callAdvisorChain
    ) {
        var span = Span.current();

        String input = serializePrompt(chatClientRequest);

        span.setAttribute("langfuse.observation.type", "generation");
        span.setAttribute("input.value", input);
        span.setAttribute("langfuse.observation.input", input);

        ChatClientResponse chatClientResponse =
                callAdvisorChain.nextCall(chatClientRequest);

        String output =
                serializeResponse(chatClientResponse.chatResponse());

        span.setAttribute("output.value", output);
        span.setAttribute("langfuse.observation.output", output);

        return chatClientResponse;
    }

    @Override
    public String getName() {
        return "Langfuse Prompt Tracing Advisor";
    }

    @Override
    public int getOrder() {
        return ToolCallingAdvisor.DEFAULT_ORDER + 100;
    }

    private String serializePrompt(ChatClientRequest request) {
        var messages = request.prompt()
                .getInstructions()
                .stream()
                .map(message -> Map.of(
                        "role", message.getMessageType().getValue(),
                        "content", ofNullable(message.getText()).orElse("")))
                .toList();
        return toJson(messages);
    }

    private String serializeResponse(ChatResponse response) {
        return ofNullable(response).map(ChatResponse::getResult)
                .map(Generation::getOutput)
                .map(output -> toJson(

                        Map.of("role", output.getMessageType().getValue(),
                                "content", ofNullable(output.getText()).orElse(""),
                                "hasToolCalls", response.hasToolCalls())

                ))
                .orElse("");
    }

    private String toJson(Object value) {
        try {
            return this.objectMapper.writeValueAsString(value);
        }
        catch (JsonProcessingException ex) {
            return String.valueOf(value);
        }
    }
}

According to the Spring AI documentation, ToolCallingAdvisor manages the tool-calling loop itself, while getOrder() determines the advisor's position in the chain. Setting it to ToolCallingAdvisor.DEFAULT_ORDER + 100 places our advisor inside the tool-calling loop, so it runs on every iteration and can observe the original request, the model response that decides to call a tool, the following request containing the tool result, and the final model response.

What You Will See in Langfuse

Example request to the application:

curl "http://localhost:8080/ai?message=What%20is%20the%20weather%20in%20Berlin?"

Langfuse should display a trace with spans.

Generation spans will contain attributes such as:

langfuse.observation.type=generation
langfuse.observation.input=...
langfuse.observation.output=...
input.value=...
output.value=...

When the following option is enabled, Spring AI exports tool call arguments and results:

spring.ai.tools.observations.include-content=true

Production Considerations

Prompts, completions, and tool results may contain sensitive data. Traces are also typically stored longer than the application's runtime memory. For production, it is worth adding content truncation, sanitization, secret removal, and PII filtering.

Combining Spring AI Advisors with Langfuse through OTLP provides transparent tracing of LLM calls, prompts, responses, tool calls, and intermediate steps.

Faker Doesn't Know Your Entities Are Related, So I Built Something That Does

Tue, 22 Sep 2026 11:22:33 +0000

You've added a second entity to the schema, wired up a @ManyToOne, and gone back to your seed script to generate fifty more rows. Ninety seconds later, the app refuses to start: unique constraint violation, somewhere inside a loop you wrote three weeks ago at 11pm. You fix it. You restart. A different field breaks a different constraint.

This is the exact moment every Spring Boot developer eventually meets the real limit of tools like Faker. They're brilliant at generating a name, an email, an address. They have no idea the Payment sitting in front of them needs a Counterparty to already exist.

So you do what everyone does: hand-write the wiring. Create parents first. Hold onto their generated IDs. Wire them into children. Hope you didn't just violate a @NotNull somewhere in the process. It works, for a while. Then the schema changes, and the script quietly stops matching reality until the next 3am debugging session finds out the hard way.

I hit this enough times that I stopped patching the script and looked at the actual problem: the information needed to seed this correctly already exists. It's sitting right there in the entity, in the annotations you already wrote. @ManyToOne, @NotNull, @Column(unique = true), JPA already knows the shape of your data. Nothing should need to be told that twice.

That became SynthForge.

The core idea

Instead of writing a script that generates data, you annotate the entity:

@Entity @Seed(count = 50)
public class Counterparty { /* fields only */ }

@Entity @Seed(count = 200)
public class Payment {
    @ManyToOne(optional = false) private Counterparty counterparty;
}

Start the app in a dev profile. Both tables populate, correctly ordered, on every restart. No seed method. No calling code, anywhere. The entity is the seed script.

What's actually happening underneath

Entity scanning. SynthForge reads JPA-managed attributes through the jakarta.persistence.metamodel.Metamodel API, not raw reflection. That distinction matters more than it sounds: only real persistent fields ever get touched, nothing that merely resembles one.

Relationship ordering. From the owning-side @ManyToOne/@OneToOne relationships, it builds a dependency graph and topologically sorts it. Parent rows exist before a child is ever generated to reference them. This is precisely where hand-written scripts fail first: the moment a second level of nesting shows up, manual ID-tracking becomes its own source of bugs, the thing you're debugging instead of your actual feature.

Constraint-aware generation. @NotNull, @Size, @Email, and a set of field-name heuristics (email, iban, amount, country) drive realistic values through Datafaker. Hit a @Column(unique = true) field, and instead of a constraint violation forty rows in, you get a bounded retry loop.

Idempotent restarts. A table that already has rows gets skipped. Restart against a persistent database as many times as you want; nothing duplicates.

"Isn't this just Instancio?"

Fair question, and worth answering directly rather than dodging it. Instancio, and its instancio-jpa extension, gives you an API to call from inside a test: build a graph, persist it, for that one test. It's genuinely good at that job.

SynthForge isn't called from anywhere. Annotate the entity, start the app in an enabled profile, and the database is already populated before your code runs a single line. Different tool, because it solves a different moment: application startup, not a single test method.

Where it stands

Live on Maven Central:

<dependency>
    <groupId>io.github.thembatman0</groupId>
    <artifactId>synthforge-spring</artifactId>
    <version>0.1.0</version>
</dependency>

Repo: github.com/ThembaTman0/synthforge

It won't fix your architecture and it won't make decisions for you. But the next time you add a foreign key at 11pm, there's a decent chance it means you get to go to bed instead.

The First Endpoint Is Easy. The Lifecycle Is the Real Backend Work

Sat, 25 Jul 2026 17:00:05 +0000

I have made this estimation mistake more than once:

Authentication should be a few endpoints.
File upload should be one controller.
Stripe should only need a checkout flow.
A Discord bot should only need a command listener.

That estimate is often correct for the first working version.

It is usually wrong for the complete system.

The feature becomes expensive when production adds retries, permissions, recovery, migrations, duplicate events, observability, and long-term maintenance.

The first endpoint is rarely the difficult part. The lifecycle around it is.

A useful way to measure backend maturity

I now think about recurring backend features at three levels.

1. Demo

The feature works under ideal conditions:

login returns a token;
a file uploads;
checkout opens;
a bot responds.

This proves the idea, but little else.

2. Application

The feature follows actual business rules:

users have roles and permissions;
files belong to users or records;
payments connect to orders;
commands enforce validation and access;
data and errors follow consistent conventions.

3. Operational

The feature survives production:

credentials can be rotated or revoked;
repeated requests remain safe;
failed events can be retried;
storage can move from local disk to S3;
sensitive actions are logged;
architecture rules are documented;
humans and AI agents can change the system without bypassing its boundaries.

Most early estimates cover the demo and part of the application layer. The operational layer appears later as “unexpected” work.

Frameworks handle a lot of security plumbing, but even Spring Security’s authentication architecture separates authentication from the wider authorization and request-security lifecycle.

A basic system may only need:

registration and login;
password hashing;
JWT access and refresh tokens;
role-based access control;
standard errors.

That can be enough for a small API or internal tool.

A customer-facing product may also need:

email verification and password recovery;
refresh-token rotation;
session visibility and revocation;
logout across devices;
granular permissions;
OAuth2;
MFA;
account lockout;
suspicious-activity handling;
security-event logging.

OWASP’s authentication guidance covers many of these concerns, including secure recovery, consistent failures, session invalidation, and protection against automated attacks.

Authentication is not a checkbox. It is a maturity ladder.

File storage grows beyond upload

A simple upload endpoint can receive a multipart file, save it, and return a URL.

A production file service must answer harder questions:

Which file types and sizes are allowed?
Can the supplied filename be trusted?
Who owns the file?
Who can read or delete it?
Is access public or protected?
Where is metadata stored?
What happens if storage succeeds but the database transaction fails?
Can local storage move to S3 later?
How are replacements, previews, and audit logs handled?

The OWASP File Upload Cheat Sheet recommends controls around filenames, extensions, content types, size limits, storage locations, authentication, and authorization.

Cloud storage adds another access model. Amazon S3 presigned URLs, for example, provide time-limited access without making an object permanently public.

The upload controller is normally the smallest part of the capability.

Payments grow beyond checkout

The happy path looks straightforward:

Create a checkout session.
Redirect the customer.
Receive success.
Mark the order as paid.

Real payment flows are asynchronous.

Browsers close. Requests time out. Events arrive late. The same webhook may arrive more than once. Refunds happen later. Subscription renewals fail.

A reliable one-time payment flow may require:

server-side price validation;
checkout-session creation;
verified webhook handling;
persisted payment state;
idempotent operations;
duplicate-event protection;
refunds;
retries and reconciliation.

Stripe can retry undelivered webhook events, so a handler must be recoverable and safe when processing the same event again.

Subscriptions add another lifecycle:

products and prices;
trials;
invoices;
customer portals;
entitlements;
upgrades and downgrades;
failed renewals;
proration;
scheduled cancellation;
access revocation.

Stripe’s SaaS subscription guide combines subscription state, customer identifiers, webhook processing, invoices, and self-service.

Checkout is only the entry point. The real integration is the state machine behind it.

Event-driven applications follow the same pattern

A Discord bot may begin as:

Receive event
→ match command
→ execute logic
→ send response

But Discord applications operate through an interaction and response lifecycle, not only a command method.

A maintainable bot eventually needs:

command registration and routing;
validation and permissions;
event listeners;
scheduled jobs;
reusable services;
rate-limit handling;
configuration;
logging;
error isolation.

The first listener is easy. The surrounding structure is the application.

What is worth reusing?

Not every feature should become a boilerplate.

A reusable foundation is valuable when it captures more than folders and dependencies.

It should preserve:

Repeated decisions

Package structure, validation, error handling, security boundaries, storage abstractions, webhook processing, and configuration conventions.

Dangerous edge cases

Repeated requests, expired credentials, unauthorized access, partial failures, invalid files, forged signatures, replayed events, and missing configuration.

Clear extension points

Developers should be able to understand, replace, or extend the implementation without rewriting the entire system.

Project guidance

The repository should explain architecture boundaries, module ownership, security-sensitive rules, verification commands, and contribution workflows.

A boilerplate without visible architecture eventually becomes a collection of files nobody wants to touch.

AI coding makes structure more important

AI tools can generate controllers, services, entities, configuration, and tests quickly.

They can also:

duplicate existing logic;
bypass established abstractions;
weaken security boundaries;
place business rules in the wrong layer;
implement the same concept differently across modules.

Repository-level instructions help reduce that risk. GitHub documents how repository custom instructions and AGENTS.md can give coding agents project-specific standards and workflows.

AI does not remove the need for reusable architecture.

It increases the cost of unclear architecture because poor decisions can now be produced faster.

Build, reuse, or buy?

There is no universal answer.

Build from scratch when

the capability differentiates the product;
the requirements are unusual;
existing foundations require major rewrites;
the team has the required expertise;
long-term ownership provides strategic value.

Reuse a focused foundation when

the capability is necessary but not differentiating;
the same decisions keep being repeated;
mistakes create security or operational risk;
the code needs to remain inside the project;
the foundation is understandable and modular.

Use a managed service when

operating the capability creates little strategic value;
compliance or infrastructure burden is high;
vendor dependency is acceptable;
the pricing model fits;
the team prefers integration work over infrastructure ownership.

The worst outcome is accidentally building half of a managed platform while believing you are implementing one endpoint.

The checklist I use now

Before implementing recurring infrastructure, I ask:

What belongs to the demo, application, and operational versions?
What happens during retries, duplicates, partial failures, or unauthorized access?
Is the capability strategically important enough to own for years?
Which architecture decisions have already been made in earlier projects?
Can reusable infrastructure remain separate from product-specific rules?
How will failures be diagnosed, retried, and reconciled?
Could the provider, storage model, or security requirement change later?
Are the boundaries and verification steps clear to both developers and AI agents?

These questions usually expose the real scope before development begins.

Turning repeated decisions into focused foundations

I started applying this approach across Java and Spring Boot projects through BuildBaseKit.

Instead of creating one large starter containing everything, each foundation focuses on one capability and maturity level:

AuthKit-Lite — JWT authentication, refresh tokens, and role-based access control.
AuthKit-Pro — upcoming advanced authentication, sessions, MFA, recovery, and authorization workflows.
FiloraFS-Lite — lightweight local file storage with clear boundaries.
FiloraFS-Pro — authenticated local and S3-backed file management.
StripeKit-Lite — upcoming one-time payments, verified webhooks, persisted payment state, and refunds.
StripeKit-Pro — upcoming subscriptions, billing, entitlements, invoices, and resilient webhook processing.
Basely — structured Java foundations for Discord commands, events, and automation.

The objective is not to avoid understanding the system.

It is to avoid restarting the same decisions from zero.

Final thought

The expensive part of backend infrastructure is rarely typing the first implementation.

It is rediscovering everything that comes after it:

revocation;
retries;
duplicate safety;
access control;
migration;
reconciliation;
documentation;
long-term ownership.

Good reusable infrastructure preserves those decisions and makes the operational lifecycle visible early.

So the next time a feature looks like “just one endpoint,” ask what happens after that endpoint.

That answer will tell you whether to build it, reuse it, or delegate it.

Which backend feature do you keep rebuilding, even though you know its lifecycle is much larger than the first implementation?

Dealing with Feign Decode Exception: Error While Extracting Response

Fri, 17 Jul 2026 03:41:43 +0000

In a microservice ecosystem, Spring Cloud OpenFeign makes synchronous HTTP requests look like simple, clean Java interface calls. It handles everything under the hood—until the data structure changes, a service returns a dynamic error payload, or a mapping framework trips over an unexpected data format.

When things break during data parsing, you get the classic runtime headache: feign.codec.DecodeException: Error while extracting response...

How to Make It (Replicating the Error)

Triggering a decoding exception is incredibly easy. OpenFeign uses a decoder (usually backed by Jackson) to map incoming JSON text directly into a Java object.

To forcefully break this mechanism, you can set up any of these scenarios:

Mismatched Data Types: Define your Java record or DTO with an Integer id, but have the producer microservice send back a UUID string like "id": "abc-123-xyz".

Missing Default Constructors/Getters: Use an immutable target object that lacks proper Jackson annotations (@JsonProperty) or proper creator configurations, preventing the deserializer from instantiating it.

Format Deviations: Expect a strict LocalDate or LocalDateTime in the format yyyy-MM-dd, while the external service sends a completely different ISO timestamp format or a UNIX epoch millisecond timestamp.

Dynamic Error Payloads: Set up your Feign client to map standard execution into a SuccessDTO. If the downstream service runs into an unhandled exception and spits out a completely different error JSON block while returning an HTTP 200 OK, the decoder will fail instantly trying to force that error data into the success object.

The Real-World Problems It Causes

When this exception pops up in production, it rarely behaves nicely. It creates secondary ripples across your system that can degrade the user experience and complicate debugging:

Obfuscated Root Causes: The DecodeException effectively wraps the original parsing error (like MismatchedInputException or InvalidFormatException), masking what actually went wrong unless you meticulously dig deep into the nested stack trace.

Broken Circuit Breakers: If you use Resilience4j or a similar framework, unhandled decoding issues look like sudden service failures. A wave of mapping errors can accidentally trip your circuit breaker, completely cutting off a healthy microservice.

Corrupted Downstream Data: If partial deserialization occurs or defaults are filled in blindly, your application might continue processing corrupted, incomplete data downstream, leading to unpredictable business logic bugs.

Cascading UI Failures: Instead of receiving a graceful fallback error message, front-end layers often crash out with generic 500 Internal Server Errors because the gateway or orchestration service couldn't gracefully intercept the failure.

How We Solve This Problem

Fixing this permanently requires moving away from default configurations and building resilience directly into the decoding pipeline.

Implement a Custom ErrorDecoder: Do not rely on standard object mapping for bad responses. Build a custom implementation of feign.codec.ErrorDecoder to explicitly handle HTTP 4xx and 5xx status codes, extracting error payloads before they ever reach the standard object decoder.

Relax Jackson Deserialization Rules: Configure your ObjectMapper bean inside the Feign configuration to be more forgiving. Specifically, toggle DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES to false so your client doesn't explode if a downstream service adds a new field to its payload.

Register Java Time Modules Explicitly: Ensure your Jackson mapper is armed with the JavaTimeModule to natively handle modern java.time types without barfing on string-to-date conversions.

Leverage Native ResponseEntity for Dynamic APIs: If you are dealing with a highly unstable external API with unstable data types, map the Feign method response directly to a raw String or JsonNode. You can then manually parse it using a custom try-catch block to handle layout variations gracefully.

Sumita
Web Developer

How to Make Your Spring Boot Project AI-Ready

Tue, 16 Jun 2026 09:56:37 +0000

How to Make Your Spring Boot Project AI-Ready

AI coding tools can generate Spring Boot code fast.

They can create controllers, services, repositories, DTOs, tests, migrations, and documentation in minutes.

But that is not the real bottleneck anymore.

The bigger problem is this:

AI can generate code, but it does not automatically understand your project.

That is where many Spring Boot teams run into trouble.

The generated code may compile. It may even pass a basic test. But it can still damage the codebase by breaking package structure, bypassing security rules, duplicating services, or putting logic in the wrong layer.

A Spring Boot project becomes AI-ready when it gives AI coding tools enough context to make changes safely.

This article explains how to structure a Spring Boot project so tools like Cursor, Claude Code, GitHub Copilot, Codex, and other AI coding agents can understand the codebase before they start editing it.

What Does “AI-Ready Spring Boot Project” Mean?

An AI-ready Spring Boot project is a codebase that clearly explains:

how the project is structured
where new code should go
which architecture rules must be followed
how authentication and authorization work
how validation and error handling work
how tests should be written
how an AI agent should inspect, modify, test, and summarize changes

The goal is not to document every line of code.

The goal is to make the most important engineering decisions visible.

A simple AI-ready documentation setup can look like this:

docs/
├── AGENTS.md
├── ARCHITECTURE.md
├── AI_RULES.md
└── AGENT_CONTRIBUTING.md

These files give AI coding tools a clear operating manual.

Without them, the AI has to guess.

And guessing is where technical debt starts.

Why AI-Generated Spring Boot Code Often Goes Wrong

AI tools are very good at generating isolated code snippets.

But real backend systems are not isolated snippets.

A production Spring Boot application usually has:

REST controllers
service classes
repositories
DTOs
entity models
validation rules
Spring Security configuration
JWT authentication
role-based access control
exception handlers
database migrations
test utilities
environment-specific configuration

When an AI agent receives a task like:

Add an invite user API.

It does not only need to know how to create an endpoint.

It also needs to know:

which package owns the feature
whether the endpoint needs authentication
which roles can access it
where business logic belongs
how validation errors are returned
whether a similar service already exists
how tests are structured
which DTO pattern the project follows

Without this context, AI tools often make these mistakes:

create duplicate services
place files in the wrong package
put business logic inside controllers
bypass existing authorization checks
return inconsistent API responses
introduce new dependencies unnecessarily
write tests that do not match the project’s testing style

The output may look correct in isolation, but still be wrong for the system.

AI Needs Architecture, Not Just Code

A common misconception is that AI can simply “read the repo” and figure everything out.

Yes, AI tools can search files.

But search is not the same as understanding.

A few controllers and services do not explain why the project is structured a certain way. They do not explain which conventions matter, which layers own which responsibilities, or which modules should not be touched without care.

Human developers build this context over time.

AI agents need it upfront.

That is why AI-ready documentation matters.

It turns hidden engineering knowledge into explicit project guidance.

The Four Files That Make a Spring Boot Project AI-Ready

You do not need a huge documentation system.

Start with four focused files.

docs/
├── AGENTS.md
├── ARCHITECTURE.md
├── AI_RULES.md
└── AGENT_CONTRIBUTING.md

Each file has a different responsibility.

1. AGENTS.md

AGENTS.md is the entry point for AI coding agents.

This file tells the AI how to work inside the repository.

It should include:

project overview
tech stack
common commands
build instructions
test instructions
important directories
coding conventions
rules for making changes
definition of done

Example:

# AGENTS.md

## Project Overview

This is a Spring Boot backend application using Java, Spring Security, JPA, and PostgreSQL.

## Common Commands

- Run tests: `./mvnw test`
- Run the application: `./mvnw spring-boot:run`
- Build the project: `./mvnw clean package`

## Agent Instructions

- Read `docs/ARCHITECTURE.md` before making structural changes.
- Search for existing implementations before creating new services.
- Keep controllers thin.
- Put business logic in services.
- Reuse existing validation and exception handling patterns.
- Do not modify security configuration unless the task explicitly requires it.
- Run relevant tests before reporting completion.

This file gives the AI a clear starting point.

It reduces random implementation choices.

2. ARCHITECTURE.md

ARCHITECTURE.md explains how the application is designed.

For a Spring Boot project, this file should describe:

package structure
request flow
controller responsibilities
service responsibilities
repository responsibilities
DTO usage
validation strategy
security model
database migration approach
external integration boundaries
important design decisions

Example:

# ARCHITECTURE.md

## Request Flow

1. Controller receives the HTTP request.
2. Request DTO validates input.
3. Service handles business logic.
4. Repository handles persistence.
5. Response DTO is returned to the client.

## Layer Responsibilities

### Controllers

Controllers handle HTTP concerns only.

They should not contain business logic.

### Services

Services contain business logic and coordinate repositories, validation, and domain operations.

### Repositories

Repositories handle database access only.

They should not contain authorization logic.

### Security

Authentication and authorization must use the existing Spring Security configuration.

This file helps AI extend the existing architecture instead of inventing a parallel one.

That is the core value.

3. AI_RULES.md

AI_RULES.md defines the rules that must not be broken during AI-assisted development.

This file is especially important for security-sensitive Spring Boot applications.

Example:

# AI_RULES.md

## Security Rules

- Never bypass authentication.
- Never remove authorization checks.
- Never expose passwords, tokens, secrets, or private user data.
- Do not change JWT handling unless explicitly requested.
- Do not make protected APIs public.

## Architecture Rules

- Keep controllers thin.
- Put business logic in services.
- Reuse existing DTOs, mappers, and exception handlers.
- Do not duplicate services or utility classes.
- Do not introduce new dependencies without justification.

## Testing Rules

- Add or update tests for behavior changes.
- Follow existing test patterns.
- Do not delete failing tests unless the task explicitly requires it.
- Mention which tests were run.

## Database Rules

- Use existing migration conventions.
- Do not rename columns or tables without explicit instruction.
- Avoid destructive schema changes unless required.

AI tools often optimize for completing the visible task.

AI_RULES.md makes sure they also respect invisible constraints.

4. AGENT_CONTRIBUTING.md

AGENT_CONTRIBUTING.md defines the workflow AI agents should follow when making changes.

This file prevents the AI from jumping straight into implementation without understanding the surrounding code.

Example:

# AGENT_CONTRIBUTING.md

## Change Workflow

1. Read `AGENTS.md`, `ARCHITECTURE.md`, and `AI_RULES.md`.
2. Search the existing codebase for similar implementations.
3. Identify the correct package or module.
4. Make the smallest focused change.
5. Add or update tests.
6. Run relevant checks.
7. Review the diff.
8. Summarize:
   - what changed
   - why it changed
   - files modified
   - tests run
   - risks or follow-up work

This gives AI coding agents a professional contribution workflow.

The point is simple:

Investigate first. Implement second.

Recommended AI-Ready Project Structure

Here is a practical structure for a Spring Boot backend:

my-spring-boot-app/
├── AGENTS.md
├── docs/
│   ├── ARCHITECTURE.md
│   ├── AI_RULES.md
│   └── AGENT_CONTRIBUTING.md
├── src/
│   ├── main/
│   │   ├── java/
│   │   └── resources/
│   └── test/
├── pom.xml
└── README.md

You can keep AGENTS.md at the root because many AI tools look there first.

The remaining files can live inside docs/.

This keeps the repository clean while giving AI tools predictable context.

AI-Ready Spring Boot Checklist

Use this checklist to evaluate your own project.

## AI-Ready Spring Boot Checklist

- [ ] The project has an `AGENTS.md` file.
- [ ] The architecture is documented.
- [ ] Controller, service, repository, and DTO responsibilities are clear.
- [ ] Security and authorization rules are documented.
- [ ] Validation and error handling patterns are documented.
- [ ] Testing expectations are documented.
- [ ] AI agents are told to inspect existing code before creating new files.
- [ ] Package boundaries are clear.
- [ ] Common commands are documented.
- [ ] The definition of done is explicit.

If most of these are missing, AI will guess.

And when AI guesses inside backend systems, review cost goes up.

Better Prompting for AI-Ready Spring Boot Projects

Once your AI documentation exists, your prompts become much more effective.

Instead of this:

Create JWT authentication with refresh tokens and roles in Spring Boot.

Use this:

Read AGENTS.md, docs/ARCHITECTURE.md, and docs/AI_RULES.md first.

Then add the organization invite API using the existing controller, service, validation, exception handling, and security patterns.

Add or update tests.

Before finishing, summarize:
- files changed
- behavior added
- tests run
- any risks or follow-up work

This prompt works better because it tells the AI to follow the system instead of inventing one.

That is the difference between AI-assisted development and AI-generated chaos.

AI-Ready Documentation Helps Humans Too

These files are not only for AI.

They also help developers.

Clear documentation improves:

onboarding
pull request reviews
architecture consistency
security awareness
testing discipline
long-term maintainability

New developers can understand the system faster.

Existing developers have a shared reference for architectural decisions.

Reviewers spend less time repeating the same comments.

That is the business value.

AI-ready documentation is not just an AI feature.

It is engineering leverage.

Start With Focused Spring Boot Foundations

AI tools perform better when the project has a focused structure.

A giant starter kit with many unused modules can make the context problem worse. The AI has more files to inspect, more abstractions to infer, and more possible places to put new code.

A focused Spring Boot foundation is easier to understand and extend.

For example:

Use AuthKit-Lite when you need Spring Boot JWT authentication, refresh tokens, and role-based access control.
Use FiloraFS-Lite when you need lightweight Spring Boot file upload with local storage.
Use FiloraFS-Pro when you need production-ready file uploads, JWT security, S3 support, and thumbnail generation.

You can explore all available Spring Boot foundations on the BuildBaseKit boilerplates page.

BuildBaseKit focuses on modular, AI-ready, production-ready Spring Boot boilerplates that help developers avoid rebuilding the same backend infrastructure repeatedly.

Benefits of an AI-Ready Spring Boot Project

Faster AI Onboarding

AI agents can understand the repository faster because they have a clear starting point.

Better Code Generation

Generated code is more likely to follow existing naming, package structure, and architecture.

Safer Changes

Explicit security, validation, testing, and database rules reduce risky assumptions.

More Consistent Architecture

Human developers and AI agents work from the same system map.

Lower Review Overhead

Reviewers spend less time fixing misplaced files, duplicated logic, and inconsistent patterns.

Better Long-Term Maintainability

The project becomes easier to extend because the important decisions are documented.

If you are building Spring Boot backends or experimenting with AI-assisted development, these guides are worth reading:

You can also browse all articles on the BuildBaseKit blog.

Final Thoughts

AI-assisted development is not about generating more code.

It is about generating the right code inside the right architecture.

A Spring Boot project becomes AI-ready when it gives both humans and AI agents a clear understanding of:

where code belongs
which rules matter
how changes should be tested
what must not be broken

AI rewards clarity.

If your Spring Boot project has predictable structure, documented rules, and focused boundaries, AI coding tools can contribute with less guesswork and fewer risky changes.

That is the practical path to safer AI-assisted backend development.

If you want a focused starting point, explore BuildBaseKit for modular, AI-ready Spring Boot boilerplates.

The Easiest Way to Integrate pawaPay in Your Java Spring Boot App

Mon, 01 Jun 2026 10:08:14 +0000

Building a high-throughput mobile money gateway is often romanticized as a simple REST API integration. The reality, however, is a brutal landscape of undocumented MNO network drops, complex cryptographic signature requirements, and terrifying double-spend vulnerabilities. When development teams attempt to hand-roll these integrations using standard Java HTTP clients like OkHttp or RestTemplate, they inadvertently construct a fragile house of cards. They pull highly sensitive environment secrets directly into the JVM heap to manually compute Signature-Input headers, exposing their financial infrastructure to memory scraping and reverse-engineering. Furthermore, relying on blocking I/O to wait for unpredictable African telecom networks guarantees that thread pools will eventually exhaust, triggering cascading outages across the entire application architecture.

This is the integration illusion. You think you are building a simple payment gateway, but you are actually attempting to build a secure, fault-tolerant, state-managing cryptographic vault from scratch.

We recognized that engineering teams do not have months to waste debugging manual boilerplate. You need speed, concurrency, and ironclad security from day one. To solve this, we engineered the pawaPay Java SDK—a completely new paradigm for mobile money integration that shifts the heavy lifting away from your application logic and into a compiled, memory-safe environment. By adopting this architecture, you can execute non-blocking, asynchronous mobile money deposits and payouts in under five minutes.

Step One: Securing the Artifact

The foundation of a secure integration begins with your dependency management. We distribute the official package directly through Maven Central, ensuring you receive cryptographically verified, unaltered binaries. You can view the live artifact registry at https://central.sonatype.com/artifact/com.katorymnd/pawapay-java-sdk to verify the latest release versions.

To bring the native power of the SDK into your project, simply declare it within your pom.xml configuration file.

<dependency>
    <groupId>com.katorymnd</groupId>
    <artifactId>pawapay-java-sdk</artifactId>
    <version>LATEST_VERSION</version>
</dependency>

Once synced, you are no longer just importing a library; you are embedding a highly optimized Java Native Interface (JNI) vault engineered in Rust. This core will handle all payload serialization and cryptographic signing entirely outside of the standard Java memory space, neutralizing standard de-compilation threats immediately.

Step Two: Zero-Trust Configuration

We engineer under the assumption that application state can be volatile, which is why we enforce a strict, zero-trust initialization process. There are no fallback secrets, and there are no dummy variables allowed. Your master API tokens and your specific Katorymnd license key must be injected via strict environment variables. If these are missing or malformed, the SDK aborts the boot sequence instantly to protect your system from executing unauthenticated requests.

To wire this into a Spring Boot application, you construct a dedicated @Configuration class. This class uses our Config.Builder to construct the environment parameters and boot the ApiClient securely.

import com.katorymnd.pawapay.sdk.api.ApiClient;
import com.katorymnd.pawapay.sdk.config.Config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class PawaPayGatewayConfig {

    @Bean
    public ApiClient pawaPayClient() {
        String apiKey = System.getenv("PAWAPAY_API_TOKEN");
        String licenseKey = System.getenv("KATORYMND_PAWAPAY_SDK_LICENSE_KEY");

        Config config = new Config.Builder()
            .apiKey(apiKey)
            .environment("production")
            .timeout(30000)
            .build();

        return new ApiClient(config, licenseKey, true, "v2");
    }
}

By defining the ApiClient as a Spring Bean, you ensure the native vault is loaded exactly once during the application lifecycle. The vault establishes the secure connection pool, locks in the hardware anchoring, and remains resident in memory, ready to process thousands of concurrent requests without the overhead of re-initialization.

Step Three: The Reactive Endpoint

The true architectural elegance of this integration reveals itself at the routing layer. Because the SDK was built specifically for modern, high-throughput environments, every core method returns a CompletableFuture. This maps natively to Project Reactor, allowing you to build completely non-blocking Spring WebFlux endpoints.

When a payload arrives, your application threads do not wait. The request is handed to the native core, and the thread is immediately liberated back to the pool to handle other users.

import com.katorymnd.pawapay.sdk.api.ApiClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.reactive.function.server.RouterFunction;
import org.springframework.web.reactive.function.server.ServerResponse;
import reactor.core.publisher.Mono;

import java.util.Map;

import static org.springframework.web.reactive.function.server.RequestPredicates.POST;
import static org.springframework.web.reactive.function.server.RouterFunctions.route;

@Configuration
public class DepositController {

    @Bean
    public RouterFunction<ServerResponse> handleDeposit(ApiClient pawaPayClient) {
        return route(POST("/api/v2/payments/deposit"), request -> 
            request.bodyToMono(Map.class)
                .flatMap(payload -> Mono.fromFuture(
                    pawaPayClient.initiateDepositV2(
                        payload.get("transactionId").toString(),
                        payload.get("amount").toString(),
                        payload.get("currency").toString(),
                        payload.get("phone").toString(),
                        payload.get("provider").toString(),
                        "Spring WebFlux Integration",
                        null, null, null
                    )
                ))
                .flatMap(response -> ServerResponse.ok().bodyValue(response))
                .onErrorResume(error -> ServerResponse.badRequest().bodyValue(Map.of("error", error.getMessage())))
        );
    }
}

In just three steps, you have bypassed weeks of manual REST implementation. You possess an endpoint that can absorb massive MNO latency spikes without crashing your server, backed by a cryptographic engine that mathematically guarantees your payload integrity.

Validating the Architecture

Before writing a single line of your own code, you can visually verify this exact execution flow. We have exposed our internal, Javalin-based disposable execution environment to the public. By visiting the live playground at https://katorymnd.dev/pawapay-demo/java/, you can input your sandbox tokens and watch the native core process deposits, handle automated config syncs, and securely fetch MNO availability in real-time. It provides a transparent, terminal-level view of the exact security validations happening beneath the surface of the SDK.

Transitioning to Enterprise Production

While the artifact is freely available on Maven Central for integration testing, deploying this architecture into a live financial environment requires absolute operational certainty. For businesses processing real capital, we offer the Premium License infrastructure detailed at https://katorymnd.com/pawapay-payment-sdk/java.

Securing a premium license unlocks critical production features, including custom domain locking to prevent your API keys from being used outside of authorized IP addresses, emergency technical support SLAs, and architectural consultation for your specific database synchronization strategy. It is the definitive path for teams that refuse to compromise on gateway resilience.

Stop wrestling with clunky HTTP clients and vulnerable cryptographic libraries. Cut the boilerplate, implement the native SDK, and secure your mobile money infrastructure today.

Why Most Side Projects Never Launch

Fri, 22 May 2026 15:44:03 +0000

Most side projects don’t fail because the idea is bad.

They fail during setup.

You start excited:

new idea
new repo
big plans

Then suddenly you’re spending days on:

authentication
JWT setup
roles & permissions
file uploads
email configs
security
deployment setup

And before you even build the core feature, your energy is gone.

This is one of the biggest reasons developers abandon projects.

Too much boilerplate.
Not enough actual product building.

That’s exactly why starter kits exist.

Instead of rebuilding the same backend setup every time, you can start with production-ready foundations and focus on shipping features faster.

If you’re building with Spring Boot, you can start free with AuthKit Lite:

https://buildbasekit.com/boilerplates/authkit-lite/

It includes:

JWT authentication
role-based access
Spring Security setup
production-ready backend structure

Skip repetitive setup.
Build the actual product.

Spring Boot File Upload Performance: Why Your API Gets Slow Under Load (And How to Fix It)

Wed, 20 May 2026 18:59:08 +0000

Backend APIs often feel fast during development.

You upload a few files locally.

Everything responds instantly.

Latency looks great.

Memory looks stable.

CPU barely moves.

Then production traffic arrives.

Suddenly:

uploads slow down
downloads start hanging
videos become painfully slow
latency spikes
users complain

This happens because local testing rarely reflects real-world load.

A file upload API is not just handling HTTP requests.

It is dealing with:

disk I/O
file buffering
network transfer
JVM memory pressure
concurrent request handling
authentication overhead
metadata queries
file streaming

Under enough load, something breaks first.

The real question is:

What breaks first in a Spring Boot file upload system?

I ran a benchmark to find out.

Quick Answer

Spring Boot handled the application layer surprisingly well under heavy load.

The actual bottleneck was not request handling.

It was large file delivery.

Small files stayed fast.

Large video responses caused latency to rise sharply.

The lesson:

Spring Boot is fine for API orchestration, but direct media delivery becomes a scaling problem much faster than most developers expect.

The Test Scenario

The benchmark used a Spring Boot file service with typical production-style endpoints:

file upload
file download
metadata lookup
file listing
delete operations

The workload intentionally mixed different file behaviors because real systems rarely handle only one request type.

The goal was not to create a synthetic benchmark that looks impressive.

The goal was realism.

Traffic gradually increased in stages until the system started showing visible degradation.

Metrics tracked included:

request throughput
P95 latency
CPU usage
memory usage
endpoint behavior
file-type response performance

Why File Upload APIs Fail Earlier Than Expected

A standard CRUD API is relatively cheap.

A file API is not.

For example:

A metadata lookup:

GET /api/files/metadata/123

might hit:

authentication filter
database query
JSON serialization

That is manageable.

Now compare that with:

GET /api/files/download/huge-video.mp4

Now the system may need to:

authenticate request
validate permissions
locate file on disk
open file stream
allocate buffers
push large chunks over network
keep thread occupied longer
handle slow client reads

That changes everything.

The API layer is no longer the only concern.

Your infrastructure becomes part of request latency.

Benchmark Summary

Here’s the simplified outcome.

Metric	Result
Total Requests	134,000+
Peak Load	7,500 RPM
Peak P95 Latency	~1.8 seconds
HTTP Failures	0
Average CPU	~35%
Main Bottleneck	Large MP4 downloads

At first glance, this looks strong.

Zero failures under that volume is respectable.

But raw survival is not the same as healthy performance.

Latency tells the real story.

P95 latency stayed stable until higher RPM levels, then degraded rapidly after ~6,300 RPM.

Latency Looked Fine... Until It Didn’t

At lower traffic levels, performance was stable.

The backend remained responsive.

P95 latency stayed in a healthy range.

Then the ramp increased.

Past a certain threshold, latency began climbing rapidly.

This pattern matters.

Gradual degradation usually means the system is saturating naturally.

Sudden collapse usually means architectural failure.

This benchmark showed degradation, not collapse.

That is actually a good sign.

Still, once latency crosses production tolerance, users do not care whether your system technically survived.

They only see slowness.

Throughput tracked closely with target RPM until upper load levels where the system started falling behind.

Throughput Was Stable

One interesting finding:

The backend kept serving requests consistently even under heavier load.

That suggests:

request routing was stable
controllers were fine
business logic held up
auth overhead was manageable

This is important because many developers blame Spring Boot too early.

The framework itself was not the issue here.

The architecture was.

The Real Bottleneck

This was the most useful finding.

The slowest operations were not:

uploads
metadata APIs
listing APIs
delete operations

The worst offender was:

large MP4 delivery

MP4 downloads had dramatically higher P95 latency compared to PDF and PNG files.

Small files performed dramatically better.

Typical behavior looked like this:

File Type	Approx Performance
PDF	Extremely fast
PNG	Moderate
MP4	Much slower

This makes perfect sense.

Large media files create pressure across the entire stack.

Including:

Disk I/O

Reading large files repeatedly creates storage pressure.

Local disk works for development.

At scale, it becomes painful.

Network Throughput

A large file means longer response duration.

More active connections.

More bandwidth consumption.

Slow clients make this worse.

JVM Memory Pressure

Streaming often introduces buffering overhead.

Even when efficient, sustained concurrency adds pressure.

Memory growth is expected.

Thread Occupancy

Traditional request handling keeps resources busy longer.

A quick metadata request finishes fast.

A multi-second video stream does not.

That reduces effective throughput.

CPU Was Not the Problem

CPU usage increased gradually with traffic but remained relatively controlled during the test.

A common assumption:

"High latency means CPU saturation."

Not here.

CPU stayed relatively controlled.

Average utilization hovered around the mid-thirties.

That means compute was not the main bottleneck.

This is useful because it changes optimization priorities.

If CPU is not the problem:

Do not waste time micro-optimizing controller code first.

Fix architecture first.

Memory Behavior

Memory usage increased under higher throughput but stayed within a manageable range.

Memory climbed during heavier traffic.

That is expected.

Large streaming workloads create pressure.

But the test did not suggest catastrophic memory behavior.

The bigger issue remained delivery mechanics, not memory collapse.

What This Actually Means for Production

This is where many teams make a bad decision.

They see a working local file upload implementation and assume:

"Great, we can scale this."

Maybe.

But probably not for media-heavy workloads.

A better architecture separates responsibilities.

Spring Boot should handle:

authentication
authorization
business rules
metadata
upload coordination
signed URL generation

Dedicated infrastructure should handle media delivery.

Examples:

AWS S3
Cloudflare R2
object storage
CDN edge delivery
Nginx static serving

This dramatically reduces backend pressure.

Better Architecture Example

Instead of this:

Client
   ↓
Spring Boot
   ↓
Local file disk
   ↓
Spring Boot streams file

Prefer:

Client
   ↓
Spring Boot auth + metadata
   ↓
Signed URL
   ↓
S3 / R2 / CDN

That changes scaling behavior completely.

Now your application server is not acting like a media server.

Which is exactly what you want.

Common Mistakes Developers Make

1. Treating Upload Success As Performance Proof

A few successful uploads prove almost nothing.

Load changes everything.

2. Serving Large Media Directly Forever

Works early.

Hurts later.

3. Blaming Spring Boot Too Quickly

The framework is often not the issue.

Bad delivery architecture usually is.

4. Ignoring P95 Latency

Average latency hides pain.

P95 exposes real user experience.

When Spring Boot Is Absolutely Fine

Spring Boot works well if your workload is:

document uploads
moderate internal tools
admin dashboards
metadata-heavy systems
authenticated business APIs

If your workload becomes:

video-heavy
media streaming
public downloads at scale

your architecture needs to evolve.

Practical Scaling Advice

If you already have a local file-based Spring Boot system:

do this first.

Move Storage

Shift from local disk to object storage.

Add CDN Delivery

Never make your app server deliver everything forever.

Use Signed URLs

Avoid routing every file request through application logic.

Benchmark Early

Do not wait until users discover bottlenecks.

Watch P95, Not Just Averages

This matters more than vanity metrics.

How many uploads can Spring Boot handle?

There is no universal number.

It depends on:

file size
infrastructure
storage strategy
concurrency
request mix
delivery architecture

Small files and metadata workloads scale much better than heavy media streaming.

Should I serve files directly from Spring Boot?

For small or internal workloads, yes.

For larger public media systems, usually no.

Object storage plus CDN is the better long-term design.

Why do video downloads slow everything down?

Because they stress:

disk reads
bandwidth
buffering
connection duration
thread occupancy

This creates systemic pressure much faster than lightweight API requests.

Final Takeaway

The benchmark revealed something useful.

Spring Boot did not fail.

The architecture reached its natural limit.

That distinction matters.

If your backend handles uploads, metadata, auth, and business logic:

Spring Boot is a strong fit.

If your backend is also acting as a high-volume media server:

you are creating your own bottleneck.

Design accordingly.

Originally inspired by real performance benchmarking work done while building a Spring Boot file infrastructure product at BuildBaseKit.

The Ecosystem Conductor: Orchestrating Market Orders with Kafka, RabbitMQ and Spring Boot

Wed, 13 May 2026 00:55:31 +0000

Hey folks!

We've reached one of the most important posts in the My Broker B3 series. Up until now we built each piece of the ecosystem separately — the price synchronizer, the matching engine, the financial custody, the asset catalog. Now it's time to build the service that connects all of it: the Broker Order API.

This is the conductor of the ecosystem. When a user wants to buy or sell a stock, this is where everything begins — and where the results come back to.

What is the Order API?

The trading-broker-order is the entry point for the broker's transactional flow. It has a clear responsibility: orchestrate the complete lifecycle of an order, from the user's intent to the final execution confirmation.

To do this, it communicates with four other ecosystem services:

Asset API (REST/Feign) — validates whether the ticker exists and is active
Wallet API (REST/Feign) — validates whether the user has sufficient balance
B3 Matching Engine (RabbitMQ) — sends the order for execution
Broker Wallet (Kafka) — notifies the ecosystem about each status change

️ The Complete Order Flow

Before diving into the code, here's the end-to-end flow:

[User]
    │
    │  POST /api/v1/orders
    │  { userId, ticker, quantity, price, side: "BUY" }
    ▼
[trading-broker-order]
    │
    ├─ 1. Validate ticker  → GET trading-asset/assets/{ticker} (Feign/REST)
    │
    ├─ 2. Validate balance → GET trading-wallet/summary (Feign/REST)
    │
    ├─ 3. Persist order → MySQL (status: PENDING)
    │
    ├─ 4. Send to B3 → RabbitMQ (mq-broker-to-b3)
    │
    ├─ 5. Notify ecosystem → Kafka (order-events-v1, status: PENDING)
    │
    └─ Returns created order → HTTP 201

         ─ ─ ─ (B3 processes the order) ─ ─ ─

[b3-matching-engine]
    │
    │  RabbitMQ (mq-b3-to-broker)
    │  { orderId, status: FILLED/REJECTED, executedPrice }
    ▼
[trading-broker-order]
    │
    ├─ 6. Update status → MySQL (FILLED or REJECTED)
    │
    └─ 7. Notify ecosystem → Kafka (order-events-v1, final status)
              │
              ├─ trading-broker-wallet consumes → blocks/settles/refunds balance
              └─ other future consumers

Two distinct communication channels, two Kafka publish moments, and the wallet reacting to each state change. Let's see how each part was implemented.

️ Tech Stack

Technology	Usage
Java 21 + Spring Boot 3.3.5	Service core
MySQL + Flyway	Order persistence
Spring Kafka	Lifecycle event publishing
Spring AMQP	Integration with Matching Engine via RabbitMQ
Spring Cloud OpenFeign	REST calls to Asset API and Wallet API
SpringDoc OpenAPI	Swagger UI documentation

️ Implementation Pillars

1. The Domain Model — Order

The Order entity represents the state of an order at any point in its lifecycle:

@Entity
@Table(name = "orders")
public class Order {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    private String userId;
    private String ticker;

    @Column(nullable = false, precision = 19, scale = 4)
    private BigDecimal quantity;

    @Column(nullable = false, precision = 19, scale = 4)
    private BigDecimal price;

    // Null while PENDING — only set after B3 feedback
    @Column(precision = 19, scale = 4)
    private BigDecimal executedPrice;

    @Enumerated(EnumType.STRING)
    private OrderSide side;   // BUY | SELL

    @Enumerated(EnumType.STRING)
    private OrderStatus status; // PENDING | FILLED | REJECTED | CANCELLED

    @CreationTimestamp
    private LocalDateTime createdAt;

    @UpdateTimestamp
    private LocalDateTime updatedAt;
}

An important design detail: executedPrice is intentionally nullable. When the order is created, we don't yet know the execution price — that only comes as a response from the Matching Engine. Forcing nullable = false here would be a domain modeling error.

The entity also carries the mapToEvent() method that serializes the current state into a Kafka event:

public OrderEventDTO mapToEvent() {
    return OrderEventDTO.builder()
            .orderId(this.getId())
            .userId(this.getUserId())
            .ticker(this.getTicker())
            .quantity(this.getQuantity())
            .price(this.getPrice())
            .executedPrice(this.getExecutedPrice())
            .side(this.getSide().name())
            .status(this.getStatus().name())
            .eventTimestamp(LocalDateTime.now())
            .build();
}

2. Ticker Validation — Asset Feign Client

Before validating balance or persisting anything, we need to ensure the requested ticker actually exists and is available for trading. A user submitting an order for XPTO999 should get a clear error immediately — not a silent rejection from B3 two hops later.

@FeignClient(name = "trading-broker-asset", url = "${app.services.asset-url}")
public interface AssetClient {

    @GetMapping("/api/v1/assets/{ticker}")
    AssetDTO getByTicker(@PathVariable("ticker") String ticker);
}

The validation in OrderService:

private void validateTicker(String ticker) {
    try {
        var asset = assetClient.getByTicker(ticker.toUpperCase());
        if (!"ACTIVE".equalsIgnoreCase(asset.status())) {
            throw new RuntimeException("Asset is not available for trading: " + ticker);
        }
        log.info("Ticker {} validated — status: {}", ticker, asset.status());
    } catch (FeignException.NotFound e) {
        throw new RuntimeException("Asset not found or inactive: " + ticker);
    } catch (FeignException e) {
        log.error("Asset service unavailable during ticker validation: {}", e.getMessage());
        throw new RuntimeException("Asset service unavailable. Please try again later.");
    }
}

Why catch FeignException separately from FeignException.NotFound? A 404 from the asset service means the ticker doesn't exist — a business error that should fail fast. Any other FeignException (connection refused, timeout, 5xx) means the asset service is unavailable — a different kind of failure that warrants a different message to the user.

3. Balance Validation — Wallet Feign Client

For buy orders (BUY), we need to ensure the user has available balance. We use Spring Cloud OpenFeign to query the Wallet API declaratively:

@FeignClient(name = "trading-broker-wallet", url = "${app.services.wallet-url}")
public interface WalletClient {

    @GetMapping("/api/v1/wallet/{userId}/summary")
    WalletSummaryDTO getWalletSummary(@PathVariable String userId);
}

The validation happens after ticker validation:

private void validateBalance(OrderRequestDTO request) {
    BigDecimal totalOrderValue = request.getPrice().multiply(request.getQuantity());
    var walletSummary = walletClient.getWalletSummary(request.getUserId());

    if (walletSummary.getAvailableBalance().compareTo(totalOrderValue) < 0) {
        throw new RuntimeException("Insufficient balance. Available: "
                + walletSummary.getAvailableBalance()
                + " | Required: " + totalOrderValue);
    }
}

Why Feign and not RestTemplate? OpenFeign eliminates HTTP boilerplate — the interface is sufficient. The URL is injected via application.yaml, making environment changes straightforward without recompiling.

4. The Creation Flow — OrderService

The placeOrder() is the heart of the service. It orchestrates five steps in sequence within a transaction:

@Transactional
public Order placeOrder(OrderRequestDTO request) {
    log.info("Processing new {} order for user: {}", request.getSide(), request.getUserId());

    // 1. Validate ticker exists and is ACTIVE in asset catalog
    validateTicker(request.getTicker());

    // 2. Validate balance for BUY orders via Wallet API
    if (request.getSide() == OrderSide.BUY) {
        validateBalance(request);
    }

    // 3. Persist order with PENDING status
    Order order = Order.builder()
            .userId(request.getUserId())
            .ticker(request.getTicker().toUpperCase())
            .quantity(request.getQuantity())
            .price(request.getPrice())
            .side(request.getSide())
            .status(OrderStatus.PENDING)
            .build();

    order = orderRepository.save(order);

    // 4. Send to B3 via RabbitMQ
    b3MessageProducer.sendToB3(mapToB3Message(order));

    // 5. Publish PENDING event to Kafka for wallet and other consumers
    orderEventProducer.sendOrderEvent(order.mapToEvent());

    return order;
}

Why validate ticker before balance? It's cheaper to fail fast on a non-existent ticker (one network call) than to validate balance first and then discover the ticker is invalid. Order of validations matters for UX and for minimizing unnecessary load on downstream services.

Design decision: The Kafka event is published with PENDING status right after creation. This allows trading-broker-wallet to immediately block the balance, without waiting for B3's result. This is the essence of eventual consistency — each service reacts to the event in its own time.

5. B3 Integration — RabbitMQ

Communication with the Matching Engine uses RabbitMQ for a specific reason: delivery guarantee. Even if the Matching Engine is temporarily unavailable, the message stays in the queue and is processed when it comes back.

Infrastructure configuration

All RabbitMQ infrastructure is declared as Spring beans — ensuring queues, exchanges and bindings exist before any message flows:

@Bean
public DirectExchange brokerExchange() {
    return new DirectExchange(exchangeName);
}

@Bean
public Queue orderOutQueue() {
    return QueueBuilder.durable(queueIn)
            .withArgument("x-dead-letter-exchange", "dlx.broker.orders")
            .withArgument("x-dead-letter-routing-key", "dlq-broker-to-b3")
            .build();
}

@Bean
public Binding orderOutBinding() {
    return BindingBuilder.bind(orderOutQueue())
            .to(brokerExchange())
            .with(routingKey);
}

The dead letter withArgument ensures messages that fail are redirected to the DLQ instead of simply being discarded.

Producer — sending to B3

public void sendToB3(B3OrderRequestDTO message) {
    log.info("Sending order {} to B3 via exchange: {} | routing-key: {}",
            message.getOrderId(), exchange, routingKey);
    rabbitTemplate.convertAndSend(exchange, routingKey, message);
}

Important decision: the message is sent to the exchange with the routing key, not directly to the queue. This respects the RabbitMQ routing model and allows adding new consumers without changing the producer.

Consumer — receiving B3 feedback

@RabbitListener(queues = "${app.rabbitmq.queue-out}")
public void receiveFeedback(B3OrderResponseDTO response) {
    log.info("Feedback received from B3 for order: {} | Status: {}",
            response.getOrderId(), response.getStatus());
    try {
        orderStatusService.updateOrderStatus(response);
    } catch (Exception e) {
        log.error("Failed to update status for order {}: {}",
                response.getOrderId(), e.getMessage(), e);
        throw e; // RabbitMQ applies retry/DLQ policy
    }
}

The throw e is intentional: rethrowing the exception allows RabbitMQ to apply its retry policy instead of silently discarding the message.

6. Status Update — OrderStatusService

When B3 feedback arrives, OrderStatusService updates the order state and publishes the second Kafka event:

@Transactional
public void updateOrderStatus(B3OrderResponseDTO response) {
    Order order = orderRepository.findById(Long.valueOf(response.getOrderId()))
            .orElseThrow(() -> new RuntimeException("Order not found: " + response.getOrderId()));

    order.setStatus(OrderStatus.valueOf(response.getStatus()));
    order.setExecutedPrice(response.getExecutedPrice());

    orderRepository.save(order);

    // Notify ecosystem with final status — wallet settles or refunds
    orderEventProducer.sendOrderEvent(order.mapToEvent());
}

This second Kafka event closes the loop: trading-broker-wallet consumes it and executes the settlement (FILLED) or refund (REJECTED) of the balance blocked in the previous step.

REST API

Method	Endpoint	Description
POST	`/api/v1/orders`	Place a new buy or sell order
GET	`/api/v1/orders/{id}`	Get order details by ID
GET	`/api/v1/orders/user/{userId}`	List all orders for a user

Swagger UI: http://localhost:8088/swagger-ui.html

✅ Validating the Execution

With the application running locally:

✅ Flyway validated the orders table schema
✅ Tomcat started on port 8088
✅ RabbitMQ connected with exchange, queues and DLQ declared
✅ Swagger UI with 3 documented endpoints
✅ AssetClient validating ticker before any persistence
✅ WalletClient validating balance for BUY orders

The Complete Ecosystem

With trading-broker-order ready, we have the first end-to-end flow working:

[User] → [Order API] → [Asset API]  (ticker validation)
                    → [Wallet API]  (balance validation)
                    → [B3 Matching Engine] (execution)
                    → [Kafka] → [Wallet API] (custody)

From the user's click to the updated wallet balance, through ticker and balance validation, RabbitMQ routing, price matching against Redis, and eventual consistency via Kafka.

Got any questions about the design decisions or the inter-service orchestration? Drop them in the comments!

About the Series

⬅️ Previous Post: Financial Custody: Managing Balance and Portfolio with Eventual Consistency

Series Index: Series Roadmap

Links:

Financial Custody: Managing Balance and Portfolio with Eventual Consistency

Tue, 12 May 2026 20:43:57 +0000

Hey folks!

Continuing the My Broker B3 series, we've reached one of the most business-logic-rich services in the ecosystem: the Broker Wallet API.

In previous posts we built the B3 infrastructure (market price sync and matching engine). Now we enter the financial core of the broker. This service is the guardian of the investor's money and assets — it needs to be correct above all else.

What is the Wallet API?

The trading-broker-wallet is the financial custody service of the ecosystem. Its responsibility is to react to order lifecycle events and ensure the user's money and assets are managed correctly at each step.

It doesn't receive direct buy commands — it listens to Kafka events and acts according to each order's status:

Kafka: order-events-v1
         │
    ┌────┴────────────────────────┐
    │                             │
 PENDING                       FILLED / REJECTED
    │                             │
 reserveBalance()          settleOrder() / refundBalance()
    │                             │
 Blocks balance            Settles or refunds

Why is the Wallet so Critical?

Imagine the following scenario without proper custody:

User has R$ 1,000 available
Places a buy order for R$ 800
Before the order is processed, places another order for R$ 500
Both orders are executed → user ends up with negative balance

The wallet solves this with the concept of blocked balance: the moment an order enters as PENDING, the value is immediately reserved. The available balance is always balance - blockedBalance.

️ Tech Stack

Technology	Usage
Java 21 + Spring Boot 3.5.11	Service core
MySQL + Flyway	Operational persistence and versioning
Spring Kafka	Order event consumption
Spring Data Redis	Real-time prices for portfolio valuation
SpringDoc OpenAPI	Swagger UI documentation

️ The Data Model

Before diving into the logic, it's important to understand the three entities that support the service:

Account
├── userId (UNIQUE)
├── balance          ← total balance (includes blocked amount)
├── blockedBalance   ← amount reserved for PENDING orders
└── currency

Account (1) ──▶ (N) Position
                     ├── ticker
                     ├── quantity
                     └── averagePrice

WalletTransaction (audit log)
├── orderId
├── userId
├── transactionType  ← RESERVE | SETTLEMENT | REFUND
└── amount

The blockedBalance is the central piece of custody. The balance never goes below zero — only blockedBalance goes up and down as orders progress through their lifecycle.

The Order Lifecycle in the Wallet

Step 1 — PENDING: Reserving the Balance

When an order is created, it arrives as PENDING. The wallet must immediately block the maximum amount that could be debited (quantity × limit price):

@Transactional
public void reserveBalance(OrderEventDTO event) {
    BigDecimal amountToReserve = event.getPrice().multiply(event.getQuantity());

    Account account = accountRepository.findByUserId(event.getUserId())
            .orElseThrow(() -> new RuntimeException("Account not found for user: " + event.getUserId()));

    // Available = balance - blockedBalance
    BigDecimal availableBalance = account.getAvailableBalance();

    if (availableBalance.compareTo(amountToReserve) < 0) {
        throw new RuntimeException("Insufficient balance to reserve funds.");
    }

    // Only blockedBalance increases — total balance stays intact
    account.setBlockedBalance(account.getBlockedBalance().add(amountToReserve));
    accountRepository.save(account);

    // Audit record
    walletTransactionRepository.save(WalletTransaction.builder()
            .userId(event.getUserId())
            .orderId(event.getOrderId())
            .transactionType(TransactionType.RESERVE)
            .amount(amountToReserve)
            .build());
}

Important point: the total balance doesn't change at this step. Only blockedBalance increases. The money is still in the account — it's just reserved.

Step 2a — FILLED: Settling the Order

The order was executed by the Matching Engine. Now the money actually leaves. But here's a subtlety: the executed price may differ from the reserved price.

Example: user placed a buy order with a limit of R$ 30.00, but the market executed at R$ 29.50. The reserved amount was R$ 300 (10 shares × R$ 30), but the actual executed amount was R$ 295 (10 × R$ 29.50). The R$ 5.00 difference automatically becomes available again.

@Transactional
public void settleOrder(OrderEventDTO event) {
    BigDecimal totalReserved = event.getPrice().multiply(event.getQuantity());
    BigDecimal totalExecuted = event.getExecutedPrice().multiply(event.getQuantity());

    Account account = accountRepository.findByUserId(event.getUserId())
            .orElseThrow(() -> new RuntimeException("Account not found"));

    // Release the initial block
    BigDecimal newBlockedBalance = account.getBlockedBalance().subtract(totalReserved);
    account.setBlockedBalance(newBlockedBalance.compareTo(BigDecimal.ZERO) < 0
            ? BigDecimal.ZERO
            : newBlockedBalance);

    // Debit the actual executed amount from total balance
    account.setBalance(account.getBalance().subtract(totalExecuted));
    accountRepository.save(account);

    // Update asset position
    updatePosition(event, account);
}

Step 2b — REJECTED: Refunding the Balance

The order was rejected (price outside market range, ticker not found in Redis, etc.). The blocked money becomes available again:

@Transactional
public void refundBalance(OrderEventDTO event) {
    BigDecimal amountToRefund = event.getPrice().multiply(event.getQuantity());

    Account account = accountRepository.findByUserId(event.getUserId())
            .orElseThrow(() -> new RuntimeException("Account not found for refund"));

    // Only subtract from blockedBalance — total balance was never touched
    if (account.getBlockedBalance().compareTo(amountToRefund) >= 0) {
        account.setBlockedBalance(account.getBlockedBalance().subtract(amountToRefund));
    } else {
        log.warn("Refund amount exceeds blocked balance for order {}", event.getOrderId());
        account.setBlockedBalance(BigDecimal.ZERO);
    }

    accountRepository.save(account);
}

Weighted Average Price Calculation

When a purchase is settled, we need to update the user's position in that asset. The weighted average price ensures that purchases made at different times are correctly reflected:

New Average Price = (Current Cost + New Cost) / (Current Qty + New Qty)

private void updatePosition(OrderEventDTO event, Account account) {
    Position position = positionRepository
            .findByAccountIdAndTicker(account.getId(), event.getTicker())
            .orElse(Position.builder()
                    .account(account)
                    .ticker(event.getTicker())
                    .quantity(BigDecimal.ZERO)
                    .averagePrice(BigDecimal.ZERO)
                    .build());

    boolean isSell = "SELL".equalsIgnoreCase(event.getSide());

    if (isSell) {
        BigDecimal newQuantity = position.getQuantity().subtract(event.getQuantity());
        if (newQuantity.compareTo(BigDecimal.ZERO) <= 0) {
            positionRepository.delete(position);
            return;
        }
        position.setQuantity(newQuantity);
        // Average price stays the same on sell
    } else {
        // BUY: recalculate weighted average price
        BigDecimal currentCost = position.getAveragePrice().multiply(position.getQuantity());
        BigDecimal newCost = event.getExecutedPrice().multiply(event.getQuantity());
        BigDecimal totalQuantity = position.getQuantity().add(event.getQuantity());
        BigDecimal newAveragePrice = currentCost.add(newCost)
                .divide(totalQuantity, 4, RoundingMode.HALF_UP);

        position.setQuantity(totalQuantity);
        position.setAveragePrice(newAveragePrice);
    }

    positionRepository.save(position);
}

Practical example:

Purchase 1: 10 PETR4 shares at R$ 30.00 → average price = R$ 30.00
Purchase 2: 5 PETR4 shares at R$ 33.00 → average price = (300 + 165) / 15 = R$ 31.00

Critical Bugs Fixed

During the code review, I identified issues that would cause silent failures in production:

1. Guaranteed NPE on first deposit
New account created without initializing blockedBalance. The first call to getAvailableBalance() would throw NullPointerException.

// ❌ Before
Account.builder()
    .balance(BigDecimal.ZERO)
    // blockedBalance missing → NPE

// ✅ After
Account.builder()
    .balance(BigDecimal.ZERO)
    .blockedBalance(BigDecimal.ZERO)

2. Withdraw validated total balance, not available balance
Allowed withdrawing money that was blocked in pending orders:

// ❌ Before — allows withdrawing reserved funds
if (account.getBalance().compareTo(amount) < 0)

// ✅ After — validates only what's available
if (account.getAvailableBalance().compareTo(amount) < 0)

3. SELL increased position instead of decreasing it
The updatePosition() method always added quantity, completely ignoring the side field. Selling 10 shares would add 10 to the position.

4. Kafka consumer silently swallowed exceptions
Errors were logged but the offset was committed — the message was discarded without retry. Now the consumer rethrows the exception so Kafka can apply its retry policy correctly.

Idempotency

To prevent duplicates in case of Kafka reprocessing, we added a database constraint:

-- V4 migration
ALTER TABLE wallet_transactions
    ADD CONSTRAINT uq_order_transaction UNIQUE (order_id, transaction_type);

If the same PENDING event arrives twice, the second insert will fail with a constraint violation — preventing the balance from being blocked twice for the same order.

REST API

Method	Endpoint	Description
POST	`/api/v1/wallet/{userId}/deposit`	Deposit funds
POST	`/api/v1/wallet/{userId}/withdraw`	Withdraw available funds
GET	`/api/v1/wallet/{userId}/summary`	Summary: balance + positions + total equity
GET	`/api/v1/wallet/{userId}/positions`	List asset positions
GET	`/api/v1/wallet/{userId}/transactions`	Transaction history

Swagger UI: http://localhost:8085/swagger-ui.html

✅ Validating the Execution

With the application running locally:

✅ Flyway applied all 4 migrations successfully
✅ Hibernate validated the schema
✅ Kafka consumer connected and subscribed to order-events-v1
✅ Consumer group trading-broker-wallet synced
✅ Swagger UI with 5 documented endpoints at http://localhost:8085/swagger-ui.html

What's Next?

With the Wallet ready to react to order events, the next step is building the trading-broker-order — the orchestrator that will manage the complete order lifecycle:

Receive buy/sell intent via REST
Validate the ticker against trading-broker-asset
Save the order as PENDING and publish to Kafka
Send the order to the Matching Engine via RabbitMQ
Consume the result and update the final status

Once that service is ready, we'll have the first complete end-to-end flow: from the user's click all the way to the updated wallet balance.

Got any questions about the custody logic or the PENDING → FILLED → REJECTED cycle? Drop them in the comments!

About the Series

⬅️ Previous Post: The Heart of B3: Building the Matching Engine with RabbitMQ and Redis

Series Index: Series Roadmap

Links:

Spring Boot Large File Upload: Limits, Streaming, and Production Best Practices

Tue, 12 May 2026 07:14:52 +0000

File uploads look simple until someone uploads a 2GB video.

Then suddenly your backend starts behaving differently:

memory usage spikes
requests take forever
uploads fail halfway
server threads stay blocked
users retry the same upload again

A small file upload API and a production-ready large file upload API are not the same thing.

If you're building file upload functionality in Spring Boot, simply increasing the max upload limit is not enough.

Let's look at how to handle large file uploads properly.

Why large file uploads become a backend problem

For small files, basic multipart upload works fine.

For larger files, several issues appear quickly:

1. Memory pressure

A common mistake is loading the uploaded file entirely into memory.

Example:

byte[] bytes = file.getBytes();

Looks harmless.

But if users upload 500MB or 2GB files, this becomes dangerous fast.

That single line can destroy your application's memory stability.

2. Long request lifecycle

Large uploads naturally take longer.

That means:

open HTTP connections for longer periods
higher timeout risk
blocked request threads
poor user experience if retries happen

3. Storage bottlenecks

Saving directly to local disk may work in development.

In production:

multiple instances may not share storage
containers may lose uploaded files
disk space becomes a hidden failure point

4. Broken validation strategy

If validation happens too late, you waste resources processing invalid uploads.

Examples:

unsupported file types
oversized payloads
corrupted uploads

Reject bad uploads early.

How large file uploads should work

A cleaner production flow looks like this:

Client sends multipart upload request
Server validates request constraints
File is streamed instead of fully loaded
Storage layer handles persistence
API returns upload metadata

Simple architecture.

Big difference in reliability.

Configure upload limits first

Start by defining upload constraints.

spring.servlet.multipart.max-file-size=1GB
spring.servlet.multipart.max-request-size=1GB

This protects your application from uncontrolled uploads.

Without limits:

accidental abuse becomes easy
memory pressure becomes unpredictable
infrastructure costs can spike

Limits are not optional.

The biggest mistake: loading entire files into memory

This pattern is dangerous:

byte[] bytes = file.getBytes();

Also risky:

String content = new String(file.getBytes());

Why?

Because the whole file gets loaded into memory before processing.

For large uploads, this is exactly what you want to avoid.

Instead, stream the content.

Better approach: stream the upload

Use InputStream instead of loading everything at once.

Example:

@PostMapping("/upload")
public ResponseEntity<String> upload(
        @RequestParam("file") MultipartFile file
) throws IOException {

    if (file.isEmpty()) {
        return ResponseEntity.badRequest().body("File is empty");
    }

    Path destination = Paths.get("/uploads/" + file.getOriginalFilename());

    try (InputStream inputStream = file.getInputStream()) {
        Files.copy(inputStream, destination);
    }

    return ResponseEntity.ok("Upload successful");
}

Why this is better:

lower memory usage
safer under larger payloads
simpler production behavior
predictable resource consumption

Separate upload logic from storage logic

Do not put everything inside the controller.

Bad:

@PostMapping("/upload")
public ResponseEntity<?> upload(MultipartFile file) {
    // validation
    // storage logic
    // business rules
    // response mapping
}

Cleaner structure:

controller handles request
service handles workflow
storage layer handles persistence

Example:

@RestController
@RequiredArgsConstructor
public class FileUploadController {

    private final FileUploadService fileUploadService;

    @PostMapping("/upload")
    public ResponseEntity<String> upload(
            @RequestParam MultipartFile file
    ) {
        fileUploadService.upload(file);
        return ResponseEntity.ok("Uploaded");
    }
}

Service:

@Service
@RequiredArgsConstructor
public class FileUploadService {

    private final StorageService storageService;

    public void upload(MultipartFile file) {
        storageService.store(file);
    }
}

This becomes much easier to maintain later.

Storage strategy matters

Local storage is okay for prototypes.

Production systems often need better options:

Local disk

Good for:

prototypes
internal tools
single-instance deployments

Bad for:

autoscaling environments
Docker/Kubernetes deployments
distributed systems

Cloud object storage

Examples:

AWS S3
Cloudflare R2
MinIO
Google Cloud Storage

Better for:

scalability
durability
shared access across instances
predictable infrastructure behavior

Validate before expensive processing

Check uploads early.

Minimum checks:

if (file.isEmpty()) {
    throw new IllegalArgumentException("Empty file");
}

Size:

if (file.getSize() > MAX_UPLOAD_SIZE) {
    throw new IllegalArgumentException("File too large");
}

Content type:

if (!allowedTypes.contains(file.getContentType())) {
    throw new IllegalArgumentException("Unsupported file type");
}

Do this before storage work starts.

Common production mistakes

Loading everything into memory

Classic performance killer.

No upload size limits

This creates operational risk immediately.

Using local storage inside containers

Looks fine in dev.

Breaks badly in production.

No timeout awareness

Large uploads take time.

Your infrastructure must account for that.

Mixing upload and business logic

Creates messy controllers and painful maintenance.

Production checklist

Before shipping large upload support:

upload size limits configured
streaming instead of full buffering
validation before processing
storage abstraction in place
timeout settings reviewed
failure handling implemented
infrastructure storage strategy chosen

If this list is incomplete, the implementation is not production-ready.

Final thoughts

Large file uploads are not complicated.

But careless implementations become expensive fast.

The winning approach is simple:

set hard limits
stream uploads
validate early
separate responsibilities
choose proper storage

That gets you a stable upload API.

Built a similar production-ready upload architecture recently while working on Spring Boot backend tooling at BuildBaseKit.

nerv-audit

Sat, 18 Apr 2026 04:52:03 +0000

We needed an audit trail Envers couldn’t handle — so I built one

“Who changed this record?”
“What was the previous value?”
“When did it happen?”

Simple questions… until you actually need answers in production.

The problem we kept running into

In one of our systems, audit logs weren’t just “nice to have” — they were critical.

We needed:

Field-level change tracking
Historical queries (point-in-time views)
Reliable audit trails across services

So naturally, we tried Hibernate Envers.

At first, it worked.

Until it didn’t.

Where things started breaking

As the system grew, we hit real limitations:

Querying audit history became painful
Custom audit logic was hard to extend
Cross-service audit consistency was messy
Performance started becoming a concern

We weren’t just storing audit logs anymore —
we needed to use them as a first-class feature.

That’s where Envers started to feel like a constraint.

The turning point

At some point, we had two options:

Keep patching around the limitations
Build something that actually fits our needs

We chose the second.

What we built

We created nerv-audit — a modular audit framework designed for real-world backend systems.

Instead of treating auditing as a side-effect, we designed it as a core part of the architecture.

Key ideas:

Structured audit work units (not just snapshots)
Flexible query model for history retrieval
Clean separation between capture, storage, and querying
Extensible by design

A quick example

Instead of digging through raw audit tables, you can query changes like this:

AuditQuery query = AuditQuery.builder()
    .entity("User")
    .field("email")
    .changed()
    .build();

List<AuditResult> results = auditService.execute(query);

Now you’re not just storing history —
you’re actually asking questions from it.

What I learned building this

A few things became very clear:

Audit logs become product features faster than expected
If querying is painful, the audit system is incomplete
“Just use Envers” works… until scale and complexity kick in
Designing audit as a domain, not a utility, changes everything

⚠️ About the project

The project is structured in modules:

nerv-audit-core → full engine (commercial)
nerv-audit-lite / examples → publicly available

I decided to keep the core commercial, but still share:

Public modules
Examples
Integration ideas

Explore it here

If you’re dealing with similar problems, you might find this useful:

https://github.com/czetsuyatech/nerv-audit

Final thought

Most systems don’t think about audit logs until something breaks.

By then, it’s already too late.

If your system will ever need to answer:

“What changed, and why?”

…it’s worth designing for it early.

I Wasted 10 Days on Backend Setup. Here’s What I’d Do in 2 Hours Now

Fri, 17 Apr 2026 13:50:42 +0000

Every new idea feels exciting.

You open your editor.
You’re ready to build something real.

Then it starts…

setting up auth
defining roles
handling tenants
creating base APIs

Day 1 goes by.
Then day 3.
Then day 10.

And you still haven’t built the actual product.

What I did wrong

On one project, I spent around 10 days just on backend setup.

I told myself:

“This needs to be clean”
“I’ll reuse this later”
“Better to do it properly now”

After all that work, I had:

no users
no feedback
nothing shipped

That’s when it hit me.

I wasn’t building a product.

I was rebuilding the same backend again.

The real problem

This work isn’t hard.

It’s repeated.

Every project starts from zero:

same auth flow
same role checks
same tenant logic
same base structure

You’re not solving new problems.

You’re rewriting solved ones.

What I do now (2-hour approach)

Now, I don’t start from scratch.

I start with a base that already has:

authentication ready
roles and permissions wired
tenant handling in place
basic structure set

From there, I focus on:

the core feature
the actual user problem

Not the setup.

What actually changed

Nothing about my skill.

Only this:

I stopped treating backend setup as “part of the product”.

It’s not.

It’s just the cost of starting.

And reducing that cost changes everything.

A simple example

Earlier:

create user system
design role model
write access checks everywhere
debug edge cases

Now:

plug into existing auth
assign roles
enforce rules once
move on

Same outcome.
Way less time.

The shift that matters

Early stage is simple:

ship fast
get feedback
improve

If setup takes 10 days, you break that loop.

If setup takes 2 hours, you stay in it.

That’s the difference.

Final thought

Your first version doesn’t need a perfect backend.

It needs:

something that works
something users can try
something you can improve

Next time you start a project, don’t ask:

“What’s the best way to set this up?”

Ask:

“How fast can I stop setting up and start building?”

If you want to skip the repeated backend work and start faster, you can check this out:

https://buildbasekit.com/boilerplates/authkit-lite/

Stop Rebuilding the Same Spring Boot Setup Every Time

Tue, 14 Apr 2026 04:40:51 +0000

If you’ve built more than one Java backend, you’ve probably done this:

setup auth again
configure roles and permissions
handle multi-tenancy
wire up basic APIs
fix the same production issues later

And suddenly a week is gone.

I’m currently working on BuildBaseKit, focused on solving exactly this.

The idea is simple:

give developers a production-ready Spring Boot starting point
not just demos, but something you can actually build on

Right now I’m validating it with real use cases like:

file servers
Discord bots
internal tools

Curious to hear from others here:

What’s the one thing you always end up rebuilding in every backend?

You’re Not Building a Product. You’re Rebuilding the Same Backend Again.

Mon, 13 Apr 2026 16:56:23 +0000

Every new idea starts the same way.

You open your editor.

You’re ready to build.

Then you realize…

You need to set up everything again:

Auth
Roles and permissions
Multi-tenancy
Database setup
Basic APIs

A few days later, nothing is shipped.

Just setup.

The real issue

This work isn’t difficult.

It’s repeated work.

You are solving the same problems again and again instead of building something users care about.

Why this slows everything down

Early stage should be simple:

ship fast
get feedback
improve

But instead, most of the time goes into backend setup.

And that kills momentum.

A mistake I kept making

On one project, I spent around 10 days setting up:

authentication
roles
tenant handling
base APIs

After all that, I had:

no users
no feedback
no product

That’s when I realized something.

I wasn’t building a product.

I was rebuilding infrastructure.

What actually matters

You don’t need a perfect backend.

You need something that:

works
supports real users
helps you move fast

That’s it.

A better way to approach this

Instead of starting from scratch every time, start with a base that already handles:

auth
roles
structure

Then spend your time on what actually matters.

Your product.

Reality check

Most developers don’t fail because of bad architecture.

They fail because they never ship.

Final thought

Next time you start something new, don’t ask:

“What’s the best setup?”

Ask:

“How fast can I ship something useful?”

If you want to skip backend setup and start faster, try this free:
https://buildbasekit.com/boilerplates/authkit-lite/

All Spring Boot Annotations — One Cheat Sheet

Wed, 04 Mar 2026 16:40:49 +0000

See the link

Distributed Tracing in Spring Boot: A Practical Guide to OpenTelemetry and Jaeger

Sat, 31 Jan 2026 18:28:46 +0000

TL;DR

Distributed tracing helps you understand how requests flow through microservices by tracking every hop with minimal overhead. This guide covers OpenTelemetry integration in Spring Boot 4 using the native starter, explains core concepts like spans and context propagation, and demonstrates Jaeger-based tracing with best practices for production. Whether you're debugging latency issues or optimizing service dependencies, distributed tracing provides the visibility modern architectures demand.

GitHub Repository: learning-distributed-tracing

The Problem: Debugging in the Dark

In a monolithic application, debugging a slow request is straightforward. Add some logging, attach a profiler, and you can see exactly where time is spent. But microservices change everything. A single user request might touch ten or more services, each with its own logs. Failures often happen between services, not inside them. When something breaks or slows down, where do you even start?

Traditional logging falls short here. Sure, you can correlate logs by request ID, but manually piecing together the journey across services, databases, and queues is tedious and error-prone. You need something that automatically tracks the entire execution path, measures timing at each step, and shows you the complete picture. That's distributed tracing.

Understanding Observability: Metrics, Logs, and Traces

Modern observability rests on three pillars. Metrics are numerical measurements like CPU usage or request count—great for alerting but lacking context for debugging. Logs are discrete events that tell you what happened at a specific moment but struggle with correlation across distributed systems. Traces capture the complete journey of a request through your system, showing execution flow and timing.

These pillars complement each other. Metrics tell you there's a problem, logs provide event details, and traces show you the execution path. Together, they form a complete observability strategy.

It's worth distinguishing observability from monitoring. Monitoring answers "Is the system healthy?" through dashboards and alerts. Observability answers "Why is the system behaving this way?" by designing systems to answer questions you didn't anticipate. Distributed tracing is a core enabler of observability, not a replacement for monitoring.

The Fundamentals of Distributed Tracing

Telemetry refers to automated data collection from remote sources—your application constantly reporting its health and activity. Spans are the building blocks of traces, representing units of work with start time, duration, and metadata. When Service A calls Service B, both create spans that form a parent-child relationship showing the call hierarchy.

Traces are collections of spans representing a single transaction. A trace ID ties all related spans together across service boundaries. Context Propagation maintains trace continuity—when Service A calls Service B, it passes the trace context in HTTP headers, allowing Service B to create child spans under the same trace.

OpenTelemetry: The Industry Standard

Before OpenTelemetry, every observability vendor had proprietary SDKs and formats. If you wanted to switch from Jaeger to Zipkin, you'd re-instrument your entire codebase. This vendor lock-in meant architectural decisions became permanent commitments.

OpenTelemetry is a vendor-neutral framework providing APIs, SDKs, and tools for telemetry data. Formed by merging OpenTracing and OpenCensus, it provides a single instrumentation API that works with any backend. The value proposition is simple: instrument once, send data anywhere.

The architecture includes the API and SDK for creating telemetry, Auto-instrumentation for frameworks like Spring and JDBC, and the Collector—an optional but recommended component that receives, processes, and exports telemetry.

While this article focuses on distributed tracing, it's worth noting that OpenTelemetry standardizes all three pillars of observability—metrics, logs, and traces. The same SDK and protocol handle all three, giving you a unified approach to instrumentation across your entire observability stack.

OTLP (OpenTelemetry Protocol) is the wire format for transmitting telemetry data. Supporting both gRPC and HTTP transports, OTLP defines how traces, metrics, and logs are serialized and sent to collectors or backends. The protocol handles backpressure, retries, and batching for reliable delivery. Most modern observability tools now support OTLP natively, making it the de facto standard.

Spring Boot 4 and OpenTelemetry Integration

Spring Boot 4 brings first-class support for OpenTelemetry through the spring-boot-starter-opentelemetry dependency. This starter provides automatic configuration and instrumentation for common scenarios like HTTP requests, database calls, and messaging.

Previous versions of Spring Boot required manual setup using the OpenTelemetry Java agent or custom configuration. Spring Boot 2 and 3 users could leverage the Java agent for bytecode instrumentation, which worked but added operational complexity. The agent approach meant deploying a JAR alongside your application and configuring it via environment variables or system properties.

With Spring Boot 4, the starter eliminates much of this complexity. Add the dependency, configure a few properties, and you're done. Under the hood, it uses Spring's auto-configuration to set up the OpenTelemetry SDK, register instrumentation libraries, and configure exporters based on your application properties.

The starter automatically instruments:

HTTP requests and responses via Spring MVC and WebFlux
RestTemplate, RestClient, and WebClient calls
JDBC database operations
Logs (automatically includes trace and span IDs)

For additional instrumentation like Kafka messaging, you can use the @WithSpan annotation for manual instrumentation, or use the OpenTelemetry Java Agent which provides automatic instrumentation for 150+ libraries.

Spring Boot Actuator's Role: While Actuator isn't required for tracing, it plays a complementary role in Spring Boot 4's observability story. Actuator's ObservationRegistry is what actually observes requests and framework operations. The OpenTelemetry starter bridges these observations into OTel-compliant traces. Think of Actuator as operational introspection (health, metrics) and OpenTelemetry as behavioral introspection (request flows).

You can still use the Java agent if you need instrumentation for libraries outside Spring's ecosystem, but for typical Spring Boot applications, the starter is sufficient and more maintainable. Framework-level instrumentation gives you baseline visibility automatically, while custom spans should be added only where domain insight is needed. This balance is critical—over-instrumentation creates noise, while under-instrumentation hides intent.

Jaeger: Your Trace Backend

Jaeger is an open-source distributed tracing platform originally developed by Uber, providing storage, querying, and visualization for traces. While OpenTelemetry handles generation and collection, Jaeger handles the backend.

Jaeger's architecture includes agents, collectors, a query service, and a web UI. For development, the all-in-one Docker image combines all components. A common misconception is that Jaeger requires Kubernetes—it doesn't. Jaeger runs on Docker, VMs, or bare metal. The all-in-one image works for local development, while production typically uses separate components with external storage like Cassandra or Elasticsearch.

Jaeger supports multiple ingestion formats, including OTLP. With OpenTelemetry's standardization, OTLP is now recommended, meaning your Spring Boot application sends traces in OTLP format directly to Jaeger without needing Jaeger-specific libraries.

Tracing Beyond Services: Databases and Message Queues

One of the most powerful aspects of distributed tracing is visibility into external dependencies. When your application makes a database call or publishes to Kafka, those operations appear as spans in your trace.

Database tracing works through JDBC instrumentation. When your Spring Boot application executes a SQL query, the OpenTelemetry instrumentation automatically creates a span containing the query, execution time, and database connection details. This visibility is crucial for identifying slow queries or N+1 problems—those situations where you're executing one query to fetch entities, then N additional queries to fetch related data for each entity. Database spans make these anti-patterns immediately visible in your trace timeline. However, be mindful of sensitive data. Database spans can include SQL statements with parameter values, which might contain PII. OpenTelemetry provides span processors to redact or mask sensitive information before export.

Message queue tracing extends traces across asynchronous boundaries. When Service A publishes a message to Kafka, it injects the trace context into message headers. When Service B consumes that message, it extracts the context and continues the trace. This creates a parent-child relationship between the producer and consumer spans, even though they execute at different times. The result is end-to-end visibility into asynchronous workflows, making it much easier to debug message processing issues or track down where data transformations went wrong.

Performance Impact and Production Considerations

Distributed tracing adds overhead from creating spans, serializing data, and network transmission. The impact varies by component:

CPU: Span creation and serialization typically add microseconds per operation. The OpenTelemetry SDK uses efficient batching to minimize per-span overhead.

Memory: The SDK buffers spans before export. Configure batch size and timeout based on traffic patterns and memory constraints to prevent excessive buffering.

Network IO: Sending traces to a local collector over localhost has minimal impact. Remote backends introduce latency and bandwidth usage. Using a collector to batch and compress traces reduces network overhead significantly. Importantly, the collector absorbs most of the performance cost, acting as a buffer between your applications and backends.

In practice, overhead is typically under 5 percent for CPU and memory. The key is intelligent sampling—trace 1-5 percent of traffic in production rather than every request (development should trace 100 percent for debugging). OpenTelemetry supports probability-based sampling for production and rate-limiting to cap traces per second.

Best Practices for Distributed Tracing

Use meaningful span names: "validatePaymentRequest" beats "process" every time. Good naming makes traces self-documenting.

Add relevant attributes: Follow OpenTelemetry semantic conventions for HTTP, databases, and queues. Add custom attributes for business context like user ID or tenant ID.

Don't over-instrument: Creating spans for every method produces noise. Focus on external calls, database queries, and significant business logic.

Implement proper error handling: Mark spans as failed and record exception details when errors occur. This helps identify which service and operation caused failures.

Sample intelligently: Trace everything in development (probability 1.0), but use 1-5 percent sampling in production (probability 0.01-0.05). This gives you statistically significant insights without overloading infrastructure. Consider adaptive sampling that increases rates for slow requests or errors.

Watch for orphaned spans: When requests hand off work to async thread pools, ensure context propagation is maintained. If a new thread loses the trace context, your trace will break, resulting in disconnected "orphaned spans" that can't be correlated. Spring Boot 4 usually handles this automatically, but verify your custom executors are properly instrumented.

Use the Collector: It provides buffering, enrichment, routing, and reliability that SDK exporters alone cannot.

Monitor your telemetry pipeline: Track export success rates and latency. If your pipeline breaks, you're debugging blind.

Querying and Analyzing Traces

Jaeger's UI provides powerful analysis tools. Search for traces by service, operation, tags, duration, and time range. The trace timeline shows the complete request flow with parent-child relationships visually nested. For advanced use cases, Jaeger Query Language (JQL) enables programmatic querying and integration with automated alerting systems. The trace comparison feature helps identify performance regressions by highlighting timing differences between trace versions.

Conclusion

Distributed tracing transforms how you understand and debug microservices. By automatically capturing request flows and timing information, it eliminates the guesswork from performance analysis and incident response. OpenTelemetry provides the standardized instrumentation, OTLP handles reliable transmission, and backends like Jaeger give you the visualization and querying tools to make sense of the data.

Spring Boot 4's native OpenTelemetry support makes adoption straightforward. Add the starter, configure your exporter, and you're tracing HTTP requests, database queries, and message queues with minimal code. The result is a system where every request tells its own story, complete with timing, dependencies, and errors.

Start small. Enable tracing in one service, verify the data reaches Jaeger, and gradually expand to your entire application. The visibility you gain will pay dividends the first time you debug a cross-service issue or optimize a slow endpoint. Distributed tracing isn't just a monitoring tool; it's a fundamental shift in how you understand distributed systems.

For hands-on examples and complete configuration, check out the learning-distributed-tracing repository.

Learning Links:

https://spring.io/blog/2025/11/18/opentelemetry-with-spring-boot
https://opentelemetry.io/docs/zero-code/java/spring-boot-starter/
https://foojay.io/today/spring-boot-4-opentelemetry-explained/
https://last9.io/blog/opentelemetry-for-spring/
https://signoz.io/blog/opentelemetry-spring-boot/
https://vorozco.com/blog/2024/2024-11-18-A-practical-guide-spring-boot-open-telemetry.html
https://medium.com/cloud-native-daily/how-to-send-traces-from-spring-boot-to-jaeger-229c19f544db
https://medium.com/xebia-engineering/jaeger-integration-with-spring-boot-application-3c6ec4a96a6f
https://blog.vinsguru.com/distributed-tracing-in-microservices-with-jaeger/
https://last9.io/blog/distributed-tracing-with-spring-boot/
https://signoz.io/blog/jaeger-vs-zipkin/

Let's understand Retries in Spring Boot

Fri, 17 Oct 2025 08:32:28 +0000

What’s a Retry?
A retry means trying an operation again if it fails the first time. For example:

Your app calls a weather API. It fails because of a slow connection. You try again after 1 second — and it works! ️
Retries help your app become more reliable, especially when dealing with temporary issues.

⚙️ Doing Retries in Spring Boot (The Simple Way)

Step 1 :

Spring Boot has a library called Spring Retry that makes this super easy.Let’s add it first.

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

Step 2 :

Enable retries in your main class

@SpringBootApplication
@EnableRetry
public class RetryDemoApplication {
    public static void main(String[] args) {
        SpringApplication.run(RetryDemoApplication.class, args);
    }
}

Step 3 :

Use @Retryable, so wherever a method might fail (for example, calling an external API), you can tell Spring to retry automatically.

@Service
public class WeatherService {

    @Retryable(maxAttempts = 3, backoff = @Backoff(delay = 2000))
    public String callWeatherAPI() {
        System.out.println("Calling weather API...");
        if (Math.random() < 0.7) {
            throw new RuntimeException("API failed!");
        }
        return "Weather is Sunny ☀️";
    }

    @Recover
    public String recover(RuntimeException e) {
        return "Weather service is temporarily unavailable ️";
    }
}

What’s Happening Here

@Retryable → Tells Spring to retry this method up to 3 times.
@Backoff(delay = 2000) → Wait 2 seconds between retries.
@Recover → If all retries fail, this method is called instead of crashing the app.

Retries are like giving your app a second chance to succeed. With just two annotations — @Retryable and @Recover — you can make your Spring Boot app much more reliable.

Designing APIs for the AI Era with Spring AI and MCP

Mon, 22 Sep 2025 13:49:16 +0000

As backend developers, we've been building robust REST APIs for years. We design them to be consumed by our UIs, mobile apps, or other microservices. It's a paradigm we've mastered. But what if I told you that with minimal effort, that same API could have a second interface—one that allows AI agents to interact with your business logic using natural language?

Today, thinking about artificial intelligence isn't an afterthought; it's a design strategy. It's not just about creating a service that responds to GET or POST requests, but about asking ourselves: how can an AI model consume my service to provide extra value?

For those of us developing with Java, thanks to Spring AI, this idea has shifted from a complex task to a simple extension of what we already do. In this post, we'll explore how incredibly easy it is to add an MCP (Model Context Protocol) Server to an existing Spring Boot application, using a real-world API for managing a product catalog as our example.

The Dual API Paradigm: REST for Machines, MCP for AI ????

A REST API is great for structured machine-to-machine communication. But an AI agent doesn't think in terms of JSON or endpoints.

This is where the MCP Server comes in. It acts as a translator, exposing your application's logic in a way that an LLM (Large Language Model) can understand and use. This allows you to have the best of both worlds:

Your traditional REST API, stable and reliable for your applications.
An MCP Server, offering a native interface for AI integration.

This approach forces you to design from the outset with the functions (Tools), data (Resources), and guides (Prompts) that an intelligent agent would need to operate autonomously within your domain.

What is the Model Context Protocol?

The Model Context Protocol (MCP) is the standard that defines how an AI model can discover and use your application's capabilities in a secure and structured way. It provides the AI with the necessary context about what actions it can perform and what information it can query.

This protocol is built on three pillars: Tools, Resources, and Prompts. If you want to dive deeper into how it works and what each of these concepts means, you can read my more detailed post: Model Context Protocol Exposed: The Power of Resources and Prompts Beyond Tools.

For this article, the key takeaway is seeing how easy it is to implement these three pillars with Spring AI.

From @RestController to MCP Server

To demonstrate this, I'll use my Product Catalog project.

Step 0: Build Your Traditional Spring Boot Service

I won't spend much time on this point because creating the foundation for a Spring Boot project is almost trivial using https://start.spring.io/. I'll also leave a link to their documentation if you want more details: Getting Started | Building an Application with Spring Boot.

Step 1: Configuration is a Breeze

Once you have your Spring Boot service, you just need to add the Spring AI dependency and enable the server. To manage versions centrally, we add the Spring AI "Bill of Materials" (BOM) to the <dependencyManagement> section of your pom.xml:

<dependencyManagement>  
    <dependencies>  
        <dependency>  
            <groupId>org.springframework.ai</groupId>  
            <artifactId>spring-ai-bom</artifactId>  
            <version>1.0.0</version>  
            <type>pom</type>  
            <scope>import</scope>  
        </dependency>  
    </dependencies>  
</dependencyManagement>

Next, in your <dependencies> section, you only need to add the MCP server starter, which in this case is spring-ai-starter-mcp-server-webmvc:

<dependencies>  
    <dependency>  
        <groupId>org.springframework.ai</groupId>  
        <artifactId>spring-ai-starter-mcp-server-webmvc</artifactId>  
    </dependency>  
</dependencies>

Finally, in your application.properties, you enable and configure the server. It's that simple:

# application.properties

# Enable the MCP server  
spring.ai.mcp.server.enabled=true  
spring.ai.mcp.server.name=product-catalog-mcp-server  
spring.ai.mcp.server.version=1.0.0

# Define general instructions for the AI  
spring.ai.mcp.server.instructions=A server for managing a product catalog with AI-powered tools, resources, and guides.

# Enable the capabilities you want to expose  
spring.ai.mcp.server.capabilities.tool=true  
spring.ai.mcp.server.capabilities.resource=true  
spring.ai.mcp.server.capabilities.prompt=true

With just these few lines, your application now has a functional MCP server.

Step 2: Implementing the MCP Features

Now that we have the Spring AI dependencies and the MCP starter, it's time to dive in.

2.1: Expose Your Services as Tools

You already have the business logic in your @Service classes. To expose a method as a tool for the AI, you just need one annotation: @Tool.

@Tool(
    description =
        "Creates a new product in the catalog with comprehensive validation and audit logging. Required fields: name, price, category, stockQuantity, minStockLevel. Optional fields: description, SKU (auto-generated if not provided), weight, dimensions, imageUrl. SKU must be globally unique if provided. Automatically creates audit log entry. Throws DuplicateSkuException for duplicate SKUs. Use this for adding new products to the inventory system.")
@Transactional
public Product createProduct(
    @ToolParam(
        description =
            "Product object with required fields: name, price, category, stockQuantity, minStockLevel. Optional: description, sku, weight, dimensions, imageUrl")
    Product product) {
...
}

And that's it! Your createProduct method is now a tool that any AI agent can invoke.

2.2: Expose Live Data with Resources

A Resource acts as a live data feed for the AI. Instead of having static data in a prompt, you can expose information that changes within your application, like a list of product categories.

To do this, we define a URI (e.g., categories://overview/all) and associate it with a function that returns the data.

public List<McpServerFeatures.SyncResourceSpecification> getResourceSpecifications() {
    return List.of(
            // All categories overview resource
            new McpServerFeatures.SyncResourceSpecification(
                    new McpSchema.Resource(
                            "categories://overview/all",
                            "All Categories Overview",
                            "Complete overview of all product categories with descriptions and guidelines",
                            "application/json",
                            null),
                    this::getAllCategoriesResource),
            ...
            );
}

private McpSchema.ReadResourceResult getAllCategoriesResource(
        McpSyncServerExchange exchange, McpSchema.ReadResourceRequest request) {
    try {
        List<Map<String, Object>> categoriesInfo =
                Arrays.stream(Category.values())
                        .map(this::createCategoryOverview)
                        .collect(Collectors.toList());

        String jsonContent =
                objectMapper.writeValueAsString(
                        Map.of(
                                "totalCategories",
                                Category.values().length,
                                "categories",
                                categoriesInfo,
                                "description",
                                "Complete catalog of product categories with their descriptions and key characteristics",
                                "lastUpdated",
                                Instant.now().toString()));

        return new McpSchema.ReadResourceResult(
                List.of(
                        new McpSchema.TextResourceContents(
                                request.uri(), "application/json", jsonContent)));
    } catch (Exception e) {
        log.error("Failed to get all categories resource", e);
        throw new RuntimeException("Failed to retrieve categories overview", e);
    }
}

2.3: Expose Your Workflows with Prompts

Prompts are structured guides that teach the AI how to perform complex, multi-step tasks. They are like pre-designed prompt templates that you can invoke, combining instructions, data from Resources, and Tools.

Imagine you want a guide for analyzing inventory. You could create a Prompt named inventory-analysis that instructs the AI to first fetch low-stock products (using a Tool) and then generate a report.

public List<McpServerFeatures.SyncPromptSpecification> getPromptSpecifications() {
    return List.of(
            ...

            // Inventory analysis prompt
            new McpServerFeatures.SyncPromptSpecification(
                    new McpSchema.Prompt(
                            "inventory-analysis",
                            "Inventory Analysis and Recommendations",
                            List.of(
                                    new McpSchema.PromptArgument(
                                            "analysisType",
                                            "Type of analysis (stock-health/reorder/turnover)",
                                            true),
                                    new McpSchema.PromptArgument(
                                            "category", "Focus on specific category", false))),
                    this::createInventoryAnalysisPrompt),
            
            ...
            );
}

private McpSchema.GetPromptResult createInventoryAnalysisPrompt(
        McpSyncServerExchange exchange, McpSchema.GetPromptRequest request) {
    try {
        String analysisType = (String) request.arguments().get("analysisType");
        String category = (String) request.arguments().getOrDefault("category", "ALL");

        String promptText = createInventoryAnalysisPromptText(analysisType, category);

        McpSchema.PromptMessage promptMessage =
                new McpSchema.PromptMessage(
                        McpSchema.Role.ASSISTANT, new McpSchema.TextContent(promptText));

        return new McpSchema.GetPromptResult(
                "Inventory analysis guidance for " + analysisType + " analysis",
                List.of(promptMessage));
    } catch (Exception e) {
        log.error("Failed to create inventory analysis prompt", e);
        throw new RuntimeException("Failed to generate inventory analysis prompt", e);
    }
}

private String createInventoryAnalysisPromptText(String analysisType, String category) {
    return switch (analysisType) {
        case "stock-health" -> """
            You are an inventory management specialist conducting a stock health analysis%s.

            ## Stock Health Analysis Framework

            ### Key Metrics to Evaluate:
            1. **Out of Stock Items** - Immediate attention required
            2. **Low Stock Warnings** - Items below minimum levels
            3. **Overstock Situations** - Items with excessive inventory
            4. **Stock Turnover Rates** - How quickly inventory moves
            5. **Dead Stock** - Items with no recent sales

            ### Analysis Process:
            1. Retrieve current inventory levels using available tools
            2. Compare against minimum stock thresholds
            3. Identify critical alerts requiring immediate action
            4. Calculate stock health scores by category
            5. Generate actionable recommendations

            ### Recommended Actions:
            - **Critical**: Immediate reorders for out-of-stock items
            - **Warning**: Schedule reorders for low-stock items
            - **Optimization**: Review slow-moving inventory for discounts
            - **Planning**: Adjust minimum stock levels based on trends

            Please conduct a comprehensive stock health analysis and provide prioritized recommendations.
            """
                .formatted(category.equals("ALL") ? "" : " for " + category + " category");

        case "reorder" -> """
            You are a procurement specialist developing reorder recommendations%s.

            ## Reorder Analysis Framework

            ### Reorder Triggers:
            1. Current stock ≤ minimum stock level
            2. Projected stockouts based on demand trends
            3. Lead time considerations
            4. Economic order quantities
            5. Seasonal demand adjustments

            ### Analysis Steps:
            1. Identify products below reorder points
            2. Calculate optimal order quantities
            3. Consider supplier minimums and volume discounts
            4. Factor in lead times and safety stock
            5. Prioritize based on sales velocity and margin

            ### Reorder Recommendations:
            - High priority: Fast-moving items near stockout
            - Medium priority: Steady sellers below minimum
            - Low priority: Slow movers with adequate runway

            Please generate a prioritized reorder plan with specific quantities and timing.
            """
                .formatted(category.equals("ALL") ? "" : " for " + category + " category");

        case "turnover" -> """
            You are a financial analyst evaluating inventory turnover performance%s.

            ## Inventory Turnover Analysis

            ### Key Calculations:
            1. **Turnover Ratio** = Cost of Goods Sold / Average Inventory Value
            2. **Days Sales Inventory** = 365 / Turnover Ratio
            3. **Stock Velocity** = Units Sold / Average Units in Stock
            4. **Carrying Cost Impact** = Average Inventory × Carrying Cost Rate

            ### Performance Benchmarks:
            - Electronics: 8-12 turns/year (Fast-moving technology)
            - Books: 3-5 turns/year (Diverse demand patterns)
            - Clothing: 4-6 turns/year (Seasonal collections)
            - General: 6-8 turns/year (Standard retail)

            ### Analysis Focus:
            1. Compare actual vs. target turnover rates
            2. Identify slow-moving inventory
            3. Calculate carrying cost implications
            4. Recommend optimization strategies

            Please analyze turnover performance and suggest improvements for capital efficiency.
            """
                .formatted(category.equals("ALL") ? "" : " for " + category + " category");

        default -> "Please specify analysis type: stock-health, reorder, or turnover.";
    };
}

Step 3: Configure the Beans to Expose Everything

Once you have all the pieces, exposing Tools, Resources, and Prompts is as simple as creating Spring Beans for them.

@SpringBootApplication
public class ProductCatalogSpringApplication {

    public static void main(String[] args) {
        SpringApplication.run(ProductCatalogSpringApplication.class, args);
    }

    @Bean
    public ToolCallbackProvider productCatalogTools(ProductService productService) {
        return MethodToolCallbackProvider.builder().toolObjects(productService).build();
    }

    @Bean
    public List<McpServerFeatures.SyncResourceSpecification> productCatalogResources() {
        List<McpServerFeatures.SyncResourceSpecification> productCatalogResources =
                new ArrayList<>();
        ...
        return productCatalogResources;
    }

    @Bean
    public List<McpServerFeatures.SyncPromptSpecification> productCatalogPrompts(
            ProductManagementPromptProvider productManagementPromptProvider) {
        return productManagementPromptProvider.getPromptSpecifications();
    }
}

With these simple steps, you've built a direct and secure bridge between your business logic and the world of AI.

Conclusion: The Next Natural Step for Your APIs

As we've seen, adding an MCP Server to your Spring Boot application isn't a Herculean task. It's the next logical step in the evolution of our services. Spring AI abstracts away all the complexity, allowing us to reuse the code we already have to open our applications to a universe of new possibilities with artificial intelligence.

Designing with AI in mind from the start is no longer an option—it's a competitive advantage.

What's Next? Your Turn to Act ????

I hope this post has inspired you to look at your APIs from a new perspective. Here are a few ideas to get you started:

Explore the source code: All the code from this example is available in my Product Catalog API GitHub repository. Clone it, run it with Docker, and experiment for yourself. The best way to learn is by doing!
Start small: You don't need a complex architecture. Pick a key method from one of your current Spring Boot applications and try exposing it as a @Tool. You'll see how simple it is.
Join the conversation: What do you think of this approach to AI-native APIs? What other use cases can you think of for an MCP Server in your projects?
Dive into the documentation: If you want to learn all the details, the official Spring AI documentation is the best place to keep learning.

Saga Pattern — an Introduction

Sun, 17 Aug 2025 07:58:36 +0000

In the world of microservices and distributed systems, managing data consistency across multiple services presents unique challenges. Traditional database transactions with their ACID guarantees work beautifully within a single database, but they fall short when your business logic spans multiple services, each with its own database. Enter the Saga pattern—a powerful approach to handling distributed transactions that has become essential in modern microservices architectures.

You can read the blog or read the research paper. I recommend the latter.

Sample projects:

Online Store Saga Choreography Example
Hotel Booking Saga Orchestration Example

What is the Saga Pattern?

The Saga pattern, first introduced by Hector Garcia-Molina and Kenneth Salem in their 1987 research paper, provides a way to manage long-running transactions that span multiple services in a distributed system. Rather than treating the entire business process as a single atomic transaction, a saga breaks it down into a series of smaller, independent transactions that can be coordinated across services.

Think of booking a vacation package that involves reserving a flight, hotel, and rental car. In a monolithic system, you might wrap all these operations in a single database transaction. With microservices, each service (Flight Service, Hotel Service, Car Rental Service) manages its own data independently. The Saga pattern allows you to coordinate these separate operations while maintaining the ability to handle failures gracefully.

At its core, a saga consists of a sequence of transactions T₁, T₂, ..., Tₙ, where each transaction has a corresponding compensating transaction C₁, C₂, ..., Cₙ. If any transaction fails, the saga executes the compensating transactions in reverse order to undo the work already completed, ensuring the system remains in a consistent state.

Problems it Solves and Consistency Trade-offs

The Saga pattern addresses several critical challenges in distributed systems:

Distributed Data Management: In microservices architectures, services are designed to be autonomous, each owning its data. Traditional two-phase commit protocols can create tight coupling and availability issues across services. Sagas enable coordination without sacrificing service autonomy.

Long-Running Processes: Business processes often involve multiple steps that may take minutes, hours, or even days to complete. Holding database locks for such extended periods is impractical and can severely impact system performance. Sagas allow these processes to progress incrementally without blocking resources.

Failure Handling: In distributed systems, failures are inevitable. Network partitions, service outages, and timeouts are part of the reality. Sagas provide a structured approach to handle these failures through compensation, rather than simply rolling back and starting over.

Eventual Consistency Trade-off: The Saga pattern embraces eventual consistency over immediate consistency. This means that at any given moment, the system might be in an intermediate state, but it will eventually reach a consistent state once the saga completes or compensates. This trade-off is acceptable—and often preferable—in many business scenarios where absolute consistency isn't critical, but availability and resilience are paramount.

Unlike strict ACID transactions that provide immediate consistency but can be brittle in distributed environments, sagas offer a pragmatic approach that acknowledges the realities of distributed systems. The business logic determines whether eventual consistency is acceptable, and in most real-world scenarios involving multiple services, it is.

Types of Saga Patterns

The Saga pattern can be implemented using two primary approaches, each with distinct characteristics and use cases.

Choreography-Based Sagas

In choreography-based sagas, services coordinate themselves through events without a central coordinator. Each service listens for events, performs its part of the transaction, and publishes events for other services to consume.

How it works: When a user places an order, the Order Service creates the order and publishes an "OrderCreated" event. The Payment Service listens for this event, processes payment, and publishes a "PaymentProcessed" event. The Inventory Service then reserves items and publishes an "ItemsReserved" event, and so on.

Pros:

Decentralized control promotes service autonomy
No single point of failure from a coordinator perspective
Natural fit for event-driven architectures
Services remain loosely coupled

Cons:

Complex to track and debug the overall flow
Difficult to understand the complete business process from code
Challenging to handle circular dependencies
Error handling can become distributed and complex

When to use: Choose choreography when you have a relatively simple saga with clear, linear flow and when you want to maximize service independence. It works well for scenarios where the business process is stable and unlikely to change frequently.

For a practical example, check out this choreography-based online store implementation that demonstrates how services coordinate through events to handle order processing.

Orchestration-Based Sagas

In orchestration-based sagas, a central orchestrator (saga manager) controls the execution flow, explicitly calling services and managing the overall transaction state.

How it works: A Saga Orchestrator receives a request to start a saga, then sequentially calls each service based on predefined logic. It maintains the saga's state and handles both success and failure scenarios by invoking appropriate compensating actions.

Pros:

Clear, centralized control flow that's easy to understand and debug
Explicit state management makes monitoring and troubleshooting straightforward
Easier to implement complex routing logic and conditional flows
Better support for timeout handling and retry mechanisms

Cons:

Central orchestrator can become a bottleneck or single point of failure
Orchestrator needs to know about all participating services
Can lead to more coupled architecture
Additional infrastructure component to maintain

When to use: Opt for orchestration when you have complex business flows with conditional logic, when you need clear visibility into the saga state, or when you're dealing with frequently changing business requirements that benefit from centralized control.

The hotel booking saga orchestration project provides a comprehensive example of how to implement orchestration-based sagas with proper state management and compensation handling.

Comparisons with Other Patterns

Understanding how the Saga pattern compares to other consistency approaches helps clarify when to use each.

Saga vs. Two-Phase Commit (2PC): Two-Phase Commit provides strong consistency through a prepare-commit protocol but comes with significant drawbacks in distributed systems. It's blocking (services must wait for coordinator decisions), has poor fault tolerance (coordinator failure blocks everything), and doesn't scale well across networks with high latency. Sagas, in contrast, are non-blocking, more fault-tolerant, and better suited for loosely coupled microservices, though they provide only eventual consistency.

Saga vs. Event Sourcing: While both patterns work well in event-driven systems, they serve different purposes. Event sourcing focuses on storing state changes as events and rebuilding state from these events. Sagas focus on coordinating multi-service transactions. They complement each other well—you can implement sagas in an event-sourced system.

Saga vs. Distributed Transactions: Traditional distributed transactions aim for immediate consistency across all resources but are complex to implement correctly and perform poorly at scale. Sagas acknowledge that immediate consistency isn't always necessary and provide a simpler, more resilient alternative for most business scenarios.

The practical advantage of sagas lies in their alignment with microservices principles: they maintain service autonomy, provide better availability characteristics, and offer a more pragmatic approach to consistency in distributed systems.

Implementation in Practice with Spring Boot

Spring Boot provides excellent support for implementing saga patterns through various approaches. The most common implementation leverages Spring's event handling capabilities and message queues.

For choreography-based sagas, developers typically use:

Spring Events for intra-service communication
Message brokers (RabbitMQ, Apache Kafka) for inter-service events
Spring Boot Actuator for monitoring saga progress
Custom event handlers that implement both forward and compensating actions

For orchestration-based sagas, the implementation often includes:

A dedicated Saga Orchestrator service built with Spring Boot
Spring State Machine for managing saga states and transitions
RestTemplate or WebClient for service-to-service communication
Scheduled tasks for handling timeouts and retries
Database persistence for saga state management

A typical Spring Boot saga implementation involves creating:

Saga Events: Domain events that represent saga steps
Event Handlers: Methods annotated with @EventListener that process saga events
Compensation Logic: Corresponding handlers for rollback operations
State Management: Tracking saga progress and current state
Error Handling: Timeout management and retry mechanisms

The referenced sample projects demonstrate these concepts in action, showing how to structure your code, handle failures, and implement proper monitoring for production-ready saga implementations.

Common Challenges and Testing Considerations

Implementing sagas in production environments presents several challenges that require careful consideration and planning.

Timeout Management: Services in a saga may become temporarily unavailable or respond slowly. Implementing appropriate timeout strategies is crucial—too short, and you'll have false failures; too long, and failed sagas will tie up resources. Design your timeouts based on realistic service response times and implement exponential backoff for retries.

Compensating Transaction Complexity: Not all operations can be easily compensated. Sending an email notification, for example, can't be "unsent." Design your sagas to minimize non-compensatable actions, or implement semantic compensation (like sending an apology email). Sometimes, the compensation is more complex than the original transaction.

Idempotency: Saga steps may be retried due to network issues or timeouts, so all operations must be idempotent. This means that executing the same operation multiple times should have the same effect as executing it once. Implement proper idempotency keys and state checking to handle duplicate requests gracefully.

Partial Failure Scenarios: The most challenging aspect of sagas is handling scenarios where some steps succeed while others fail, potentially leaving the system in an intermediate state. Design your business processes to be resilient to these intermediate states, and ensure your UI and downstream systems can handle eventual consistency appropriately.

Testing Saga Flows: Testing distributed sagas requires sophisticated approaches:

Unit Testing: Test individual saga steps and their compensations in isolation
Integration Testing: Use tools like TestContainers to test saga flows with real message brokers and databases
Chaos Testing: Deliberately introduce failures at different points to verify compensation logic
End-to-End Testing: Test complete saga flows in staging environments that mirror production

Monitoring and Observability: Implement comprehensive logging and monitoring for saga execution. Track saga instances, their current state, execution times, and failure rates. Tools like distributed tracing can help you follow saga execution across multiple services.

Design Recommendations:

Keep saga steps as small and focused as possible
Design for failure from the beginning—assume every step can fail
Implement proper dead letter queues for handling poison messages
Use correlation IDs to track saga instances across services
Consider implementing saga timeouts at the business process level
Plan for manual intervention in complex failure scenarios

The key to successful saga implementation is thorough testing and gradual rollout. Start with simple, linear sagas before moving to complex orchestration scenarios, and always have monitoring and alerting in place to catch issues early.

Conclusion

The Saga pattern represents a pragmatic approach to managing distributed transactions in microservices architectures. By embracing eventual consistency and providing structured failure handling through compensation, sagas enable developers to build resilient, scalable systems that can handle the complexities of distributed environments.

Whether you choose choreography for simple, event-driven flows or orchestration for complex business processes, the key is to understand your specific requirements and constraints. The pattern's flexibility allows for various implementation approaches, from simple Spring Boot applications to sophisticated orchestration engines.

As distributed systems continue to evolve, the Saga pattern remains a fundamental tool for managing complexity while maintaining the benefits of microservices architecture. Success with sagas comes from careful design, thorough testing, and a clear understanding of the consistency trade-offs that make modern distributed systems both scalable and resilient.

References

Garcia-Molina, H., & Salem, K. (1987). Sagas. ACM SIGMOD Record.
Microservices Patterns: Saga Pattern
Online Store Saga Choreography Example
Hotel Booking Saga Orchestration Example

Integrating Kafka Test Container for Local Development Environment

Sat, 08 Jun 2024 20:54:56 +0000

We typically use a shared Kafka instance if we are working on Kafka in our local environment. Using a shared Kafka instance is an issue as there could be Kafka consumers (listeners), which other development team members could launch on the same topic, or another listener could be running somewhere else. So, to circumvent this, we typically launch our own Kafka instance and work against it. Setting up a local Kafka instance is relatively easy; if you have experience working with Kafka, we must download and launch the binaries, factoring in all dependencies like ZooKeeper. If you don't have experience, setting up Kafka's local instance can become tedious and time-consuming. Instead, we could consider using Kafka Docker container leveraging test containers and the docker desktop app.

This article covers detailed steps for setting up a Kafka test container in the local development environment for Spring Microservice.

Pre-Requisites

Docker Desktop installed.
Springboot Application

Configure Test Container for Local Development

First step is to configure all the required packages, we add following maven packages to the pom.xml

<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-testcontainers</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>kafka</artifactId>
<version>1.17.3</version> <!-- We use the latest version -->
<scope>compile</scope> <!-- Change to 'compile' -->
</dependency>

We need to change the scope of the spring-boot-testcontainers and org.testcontainers to compile instead of test

Once we have added the required packages, we must ensure that the Kafka test container is launched only in the local development environment and not in a higher environment like our UAT, integration Testing, or Production Environments. For this, we can use Spring Profiles. We add an application property file with a ‘local’ Spring profile.

Now we add the Kafka container configuration class annotated with @Profile("local"), ensuring the configuration is only active for the local profile.

@Configuration
@Profile("local") // This makes sure the configuration is only active for the local profile
public class KafkaTestContainerConfig {

    @Bean
    public KafkaContainer kafkaContainer() {
        KafkaContainer kafka = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:latest"));
        kafka.start();
        System.setProperty("spring.kafka.bootstrap-servers", kafka.getBootstrapServers());
        return kafka;
    }
}

We will now configure the Kafka Producer and the Consumer.

Producer

@Configuration
public class KafkaProducerConfig {

    @Value("${spring.kafka.bootstrap-servers}")
    private String bootstrapServers;

    @Bean
    public ProducerFactory<String, MessageRequest> producerFactory() {
        Map<String, Object> configProps = new HashMap<>();
        configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, this.bootstrapServers);
        configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
        return new DefaultKafkaProducerFactory<>(configProps);
    }

    @Bean
    public KafkaTemplate<String, MessageRequest> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }
}

Consumer

@EnableKafka
@Configuration
public class KafkaConsumerConfig {

    @Value("${spring.kafka.bootstrap-servers}")
    private String bootstrapServers;

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, MessageRequest> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, MessageRequest> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }

    @Bean
    public DefaultKafkaConsumerFactory<String, MessageRequest> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(JsonDeserializer.TRUSTED_PACKAGES, "com.mytechnetknowhows.kafka.testcontainers");
        return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(),
                new JsonDeserializer<>(MessageRequest.class));
    }
}

Send Receive Message

We then add the Spring Service, which will send messages to Kafka and configure Kafka Listerner in the same process. In the real world, the listener would be a separate Spring application.

@Service
public class KafkaMessageService {

    public static final String TOPIC = "message_request_multi_partition_topic";

    private final KafkaTemplate<String, MessageRequest> kafkaTemplate;
    private static final Logger logger = LoggerFactory.getLogger(KafkaMessageService.class);
    @Autowired
    public KafkaMessageService(KafkaTemplate<String, MessageRequest> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    @Autowired
    MessageStore messageStore;

    public String sendMessage(MessageRequest messageRequest) {
        String message = null;
        try {
            kafkaTemplate.send(TOPIC, messageRequest);
            message = "Message sent";
        } catch (Exception ex){
            message = "Error occurred while sending message";
            logger.error("KafkaMessageService::sendMessage - An error occurred while sending message, detail error: ", ex);
        }
        return message;
    }



    @KafkaListener(topics = TOPIC, groupId = "test-group",
            containerFactory = "kafkaListenerContainerFactory")
    public void processMessageRequest(MessageRequest messageRequest) {
        logger.info("Received Message Request: " + messageRequest.getOrderNumber());
        messageStore.addMessage(messageRequest);
    }
}

We will then add the API endpoint which provides an entry for sending messages

@RestController
@RequestMapping("/api/v1/message")
public class KafkaMessageController {

    @Autowired
    KafkaMessageService kafkaMessageService;

    @PostMapping("/send")
    public MessageResponse postMessage(@RequestBody MessageRequest messageRequest) {
        String response = this.kafkaMessageService.sendMessage(messageRequest);
        MessageResponse messageResponse = new MessageResponse();
        messageResponse.setMessage(response);
        return messageResponse;
    }
}

Now we have all the required code set up, when we launch the spring boot application we will launch with spring profile of 'local'.

Once the application is launched, we will see all the Kafka logs, as shown below.

2024-06-08T12:29:59.375-04:00  INFO 31740 --- [kafka.testcontainers] [           main] tc.testcontainers/ryuk:0.7.0             : Creating container for image: testcontainers/ryuk:0.7.0
2024-06-08T12:29:59.572-04:00  INFO 31740 --- [kafka.testcontainers] [           main] o.t.utility.RegistryAuthLocator          : Credential helper/store (docker-credential-desktop) does not have credentials for https://index.docker.io/v1/
2024-06-08T12:29:59.658-04:00  INFO 31740 --- [kafka.testcontainers] [           main] tc.testcontainers/ryuk:0.7.0             : Container testcontainers/ryuk:0.7.0 is starting: a0a414e62aac002e123702196d20cf64c1aae5373713eb66100b5d8d9f0539aa
2024-06-08T12:29:59.988-04:00  INFO 31740 --- [kafka.testcontainers] [           main] tc.testcontainers/ryuk:0.7.0             : Container testcontainers/ryuk:0.7.0 started in PT0.6123508S
2024-06-08T12:29:59.992-04:00  INFO 31740 --- [kafka.testcontainers] [           main] o.t.utility.RyukResourceReaper           : Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
2024-06-08T12:29:59.992-04:00  INFO 31740 --- [kafka.testcontainers] [           main] org.testcontainers.DockerClientFactory   : Checking the system...
2024-06-08T12:29:59.992-04:00  INFO 31740 --- [kafka.testcontainers] [           main] org.testcontainers.DockerClientFactory   : ✔︎ Docker server version should be at least 1.6.0
2024-06-08T12:29:59.992-04:00  INFO 31740 --- [kafka.testcontainers] [           main] tc.confluentinc/cp-kafka:latest          : Creating container for image: confluentinc/cp-kafka:latest
2024-06-08T12:30:00.026-04:00  INFO 31740 --- [kafka.testcontainers] [           main] tc.confluentinc/cp-kafka:latest          : Container confluentinc/cp-kafka:latest is starting: 59b261961e20f66e2f44b72a2125570814ae695d993766ea8a18acf315a4e205
2024-06-08T12:30:03.252-04:00  INFO 31740 --- [kafka.testcontainers] [           main] tc.confluentinc/cp-kafka:latest          : Container confluentinc/cp-kafka:latest started in PT3.2597275S
2024-06-08T12:30:04.810-04:00  INFO 31740 --- [kafka.testcontainers] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/'

We then now can invoke the endpoint through Postman

We will see the message listener has been invoked

2024-06-08T12:02:36.779-04:00  INFO 30488 --- [kafka.testcontainers] [ntainer#0-0-C-1] c.m.k.t.service.KafkaMessageService      : Received Message Request: IC-91911

Now, if we launch with a different profile like UAT or PROD, we will not see the Kafka container being launched, and the consumer will attempt to connect to Kafka instance in the UAT profile.

Conclusion

In conclusion, utilizing Kafka Test containers for local development within Spring Boot applications streamlines the development process and enhances consistency and isolation across local development environments. This approach minimizes potential disruptions caused by shared Kafka instances and offers a scalable, flexible solution that adapts seamlessly to the needs of individual developers and teams.

This source code for this is available on GitHub