Coder Legion - Recent posts tagged microservices

Reliable Event-Driven Architecture in Spring Boot: Outbox, Inbox, Retries, and Idempotency

Thu, 29 Oct 2026 00:07:44 +0000

Why publishing an event is only the beginning of building a reliable distributed system.

Building an event-driven application with Spring Boot is relatively straightforward.

You create a Kafka producer, publish an event, implement a consumer, and process the message.

Everything works beautifully during development.

Then production introduces the situations your happy-path tests never covered.

What happens when the database transaction commits, but Kafka is unavailable? What happens when the consumer processes an event successfully but crashes before acknowledging it? What happens when two application instances attempt to process the same message?

These aren't unusual edge cases. They're fundamental challenges of distributed systems.

And solving them requires more than adding a retry annotation.

Let's explore the reliability boundaries behind a production-ready event-driven architecture.

1. The dual-write problem

Imagine a payment service that must perform two operations:

Save a payment in PostgreSQL.
Publish a PaymentCompleted event to Kafka.

A straightforward implementation might look like this:

@Transactional
public void completePayment(Payment payment) {
    paymentRepository.save(payment);
    kafkaTemplate.send(
        "payment.completed",
        new PaymentCompleted(payment.getId())
    );
}

The code looks reasonable, but there's a serious problem.

The database transaction and Kafka publication do not share a single atomic transaction.

Several outcomes are possible:

The database commits, but the event is never published.
Kafka accepts the event, but the database transaction rolls back.
Publication times out, leaving the application uncertain whether Kafka received the event.

Adding retries doesn't eliminate the fundamental inconsistency.

The problem is not whether Kafka is reliable. The problem is coordinating two independent systems.

The transactional outbox

The transactional outbox pattern changes the transaction boundary.

Instead of publishing directly to Kafka, the application writes an outbox record within the same database transaction as the business operation.

Payment Service
      |
      v
Database Transaction
      |
      +-- Save Payment
      |
      +-- Save Outbox Event
      |
      v
    Commit
      |
      v
Outbox Publisher
      |
      v
    Kafka

Either the payment and outbox event are committed together, or neither is committed.

A separate publisher retrieves pending outbox records and delivers them to Kafka.

This solves the atomicity problem between the business state and the durable intent to publish.

It does not guarantee that an event will be published exactly once.

That distinction matters.

2. Why the outbox still needs idempotency

Consider this sequence:

The outbox publisher sends an event to Kafka.
Kafka successfully accepts it.
The application crashes before marking the outbox record as published.
The publisher restarts and sends the event again.

Kafka may now contain two copies of the same logical event.

This is a normal consequence of at-least-once delivery.

The correct response is not to assume duplicates can always be prevented.

Instead, design consumers that can safely tolerate them.

3. The transactional inbox

An inbox provides a durable record of incoming events that have already been processed.

Each event should carry a stable identifier, such as an event ID generated when the logical event is created.

When the consumer receives a message, it checks whether that event has already been processed.

A simplified model looks like this:

Kafka Event
     |
     v
Consumer
     |
     v
Database Transaction
     |
     +-- Claim Event ID in Inbox
     |
     +-- Apply Business Changes
     |
     +-- Mark Inbox Completed
     |
     v
   Commit

The important part is the transaction boundary.

The inbox record and the business changes must participate in the same transaction when using the same transactional database.

Otherwise, a consumer can record an event as completed even though its business operation failed.

A subtle but important case

Suppose a consumer receives PaymentCompleted, updates an invoice, and produces InvoicePaid.

Now the reliability boundary becomes:

Inbox + Business State + Outbox

All three should be committed atomically.

Incoming Event
      |
      v
Database Transaction
      |
      +-- Inbox Claim
      |
      +-- Update Invoice
      |
      +-- Create Outbox Event
      |
      v
    Commit
      |
      v
Outbox Publisher
      |
      v
Outgoing Event

This is one of the most useful patterns for building dependable event-driven workflows.

The inbox protects the incoming side. The outbox protects the outgoing side.

Neither eliminates the need for careful business-level idempotency, especially when external side effects are involved.

4. Retries are a lifecycle, not just an annotation

Retries are often introduced with a simple configuration:

@Retryable(
    retryFor = TransientException.class,
    maxAttempts = 3
)

For short-lived operations, that can be perfectly reasonable.

But what happens when a dependency is unavailable for 30 minutes? What happens when the application restarts? What happens when an event requires operator intervention?

An in-memory retry loop cannot reliably manage every long-running failure scenario.

For durable asynchronous processing, retries should be modeled as persistent state.

A typical lifecycle might look like this:

PENDING
   |
   v
PROCESSING
   |
   +---- Success ----> COMPLETED
   |
   +---- Retryable failure
   |           |
   |           v
   |      RETRY_SCHEDULED
   |           |
   |           v
   |        PENDING
   |
   +---- Terminal failure
               |
               v
             FAILED

Each attempt can record useful information:

Attempt count
Next eligible execution time
Last failure category
Processing status
Relevant timestamps

Exponential backoff with jitter helps avoid repeatedly overwhelming an unhealthy dependency.

But retry policies should distinguish transient failures from permanent ones.

Retrying an invalid business request a thousand times won't make it valid.

5. Multi-instance processing changes everything

A background publisher may work perfectly when there's only one application instance.

Then the service is scaled to three pods.

All three publishers query the same pending records.

Without coordination, multiple instances may attempt to publish or process the same work concurrently.

One common PostgreSQL technique is:

SELECT id
FROM outbox_event
WHERE status = 'PENDING'
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

SKIP LOCKED allows concurrent workers to skip rows already locked by another transaction.

This can improve coordination between workers, but it's not a complete delivery protocol.

The design must still consider:

How work is claimed and how long claims remain valid
Transaction duration and lock ownership
Recovery after a worker crashes
Retry scheduling and fairness
Ordering requirements
Duplicate delivery

For example, holding database locks across a slow network call can increase contention and create unnecessarily long transactions.

A robust implementation may use short claim transactions, explicit processing ownership, and recovery mechanisms rather than keeping database transactions open while publishing.

Concurrency control and delivery guarantees are related, but they are not interchangeable.

6. Exactly-once is not an end-to-end promise

It's tempting to describe an event-driven system as exactly-once.

But exactly-once semantics depend on the boundaries being discussed.

Kafka transactions can provide guarantees for supported Kafka operations. A PostgreSQL transaction can atomically update records within PostgreSQL.

Neither automatically makes a database update, Kafka publication, HTTP request, and external payment operation one globally atomic action.

A more useful engineering goal is:

At-least-once delivery with effectively-once business outcomes wherever possible.

That typically requires durable event identities, transactional inbox processing, idempotent business operations, and explicit handling of external side effects.

For payment systems, this distinction is particularly important.

Processing the same event twice must not mean charging a customer twice.

7. Observability is part of reliability

A system that retries failures silently can appear healthy while accumulating thousands of unprocessed events.

Operational visibility should answer questions such as:

How many events are pending?
How old is the oldest unpublished event?
How many retries are scheduled?
Which events are permanently failing?
Are workers making progress?
Are duplicates being detected?
Is event processing latency increasing?

Metrics, structured logs, tracing, and operator-facing failure information are not optional extras for a production system.

They help distinguish a temporary delay from a workflow that has stopped making progress.

A reliable system should make failure visible and recovery understandable.

8. Putting these patterns together

A practical architecture combines several responsibilities:

Business Operation
       |
       v
Transactional Outbox
       |
       v
Durable Publisher
       |
       v
Message Broker
       |
       v
Transactional Inbox
       |
       v
Business Processing
       |
       +-- State Changes
       |
       +-- Optional New Outbox Events
       |
       v
Commit

Around that flow, we need retry policies, concurrency coordination, idempotency, recovery, and observability.

Each component solves a different failure mode.

The outbox prevents losing the intent to publish when business data commits.

The inbox protects consumers from repeated processing.

Retries help transient failures recover.

Concurrency controls coordinate multiple workers.

Observability makes the system operable.

None of these patterns should be treated as magic.

Their value comes from understanding exactly which failure boundary each one protects.

9. Why I built NERV Event

After working with event-driven enterprise applications, I wanted these reliability concerns to be reusable rather than repeatedly implemented in individual services.

That led to NERV Event, an open-source Java and Spring Boot library focused on patterns such as transactional outbox, inbox processing, retries, and idempotency.

The goal isn't to hide distributed-systems complexity behind annotations.

It's to provide reusable infrastructure while keeping the reliability model explicit.

GitHub: https://github.com/czetsuyatech/nerv-event

The library is one possible implementation of these ideas. The architectural principles are useful regardless of which framework or library you choose.

Final thoughts

Publishing an event is easy.

Building a system that behaves predictably when databases, brokers, application instances, and downstream dependencies fail is much harder.

Reliable event-driven architecture begins with a simple question:

What happens if the application crashes at this exact point?

Ask that question at every transaction boundary, every message acknowledgment, and every external call.

The answers will reveal where durability, idempotency, retries, and recovery are actually needed.

I'd be interested to hear how other teams approach this: Do you use a transactional inbox alongside your outbox, or do you handle consumer idempotency directly in your business tables?

Originally published on CzetsuyaTech. Explore more articles on Java, Spring Boot, distributed systems, and software architecture there.

Polar Bear Cache: Local Caching Without Repeated Remote Calls

Mon, 26 Oct 2026 10:07:01 +0000

Shared storage provides the source of truth. Local caches provide speed. Events help keep cache state in sync.

Project: Polar Bear Cache — GitHub repository

In my previous discussions about microservices, I explored how shared storage and event-driven synchronization can work together to manage distributed session state.

This time, I want to focus on another part of the problem: caching frequently accessed data across multiple service instances.

The Problem: Faster Reads, Distributed Cache Invalidation

In a microservices architecture, multiple service instances often need the same data, especially data that is read frequently but updated relatively infrequently.

A distributed cache can reduce database load, but accessing it still involves network round trips and potentially serialization and deserialization overhead.

So, what if each service instance could keep frequently accessed data in its own local cache, while receiving notifications when that data changes elsewhere?

That is the idea behind Polar Bear Cache, an open-source Java caching library designed to work with Spring's familiar cache annotations.

The Design: Local Cache + Event-Driven Invalidation

Each service instance maintains its own local cache. When data changes, an event service can notify other instances that specific cache entries need to be invalidated.

The key principle is simple:

Broadcast invalidation events, not the entire cached data.

Polar Bear Cache integrates with Spring's cache annotations:

@Cacheable — Returns a cached value when available, or executes the underlying method to load it when the value is missing.
@CachePut — Updates the local cache and, when event-service integration is configured, notifies other instances to invalidate the corresponding entry.
@CacheEvict — Evicts cached data locally and, when event-service integration is configured, notifies other instances to invalidate their corresponding entries.

For example, when a product is updated:

The application updates the source of truth and its local cache.
It publishes a cache-invalidation event through the event service.
Other service instances receive the event and evict the corresponding local entry.
Their next read loads the data again from the source of truth.

This allows repeated reads to stay local in the common case, while cross-instance communication is used to coordinate invalidation when data changes.

When Does This Approach Make Sense?

This design is particularly suitable for:

Read-heavy workloads with relatively infrequent updates.
Applications running multiple service instances, each with its own local cache.
Use cases where brief delays in cache invalidation are acceptable.

There is an important trade-off: event-driven invalidation is asynchronous. Between a data update and the time other instances process the invalidation event, stale data may still be served.

Polar Bear Cache does not guarantee strong consistency across service instances.

Depending on the application, you may also need to consider event-delivery reliability, duplicate events, cache expiration, and recovery strategies when a service instance misses an event.

The goal is not to eliminate every consistency trade-off, but to make local caching practical in distributed applications without requiring every read to go through a remote cache.

Why “Polar Bear”?

The name comes from a somewhat optimistic engineering idea:

Local caching
→ Fewer remote cache requests and network round trips
→ Potentially more throughput from the same resources
→ Potentially fewer servers needed for a given workload
→ Lower energy consumption and greenhouse gas emissions
→ Maybe we can help save polar bears. 🐻❄️🧊

Of course, the actual impact depends on workload characteristics, infrastructure, and deployment choices. Performance improvements do not automatically translate into lower energy consumption, but I like the idea that better resource efficiency could benefit both applications and the environment.

I'd Love to Hear Your Thoughts

I'm interested in how other engineers approach local caching in distributed systems.

How do you handle cache invalidation across service instances?
Do you prefer event-driven invalidation, short TTLs, versioned keys, or a combination?
What failure scenarios would you consider essential to address in this design?

If you're interested in the implementation, feedback and technical discussion are very welcome.

Less remote access, more ice. 🐻❄️🧊

Java #Spring #Microservices #Caching #DistributedSystems #OpenSource

Sync Session: Rethinking Session Management in Microservices

Mon, 26 Oct 2026 09:17:47 +0000

Sync Session: Rethinking Session Management in Microservices

Project: Sync Session on GitHub

Shared storage provides the source of truth. Local caches provide performance. Events synchronize session state across service instances.

Since I started working with microservices in 2016, I have continuously explored and refined different approaches to session management.

One question has remained particularly important to me:

How can every service independently verify a user's session without turning authentication into a bottleneck?

The Challenge: Security vs. Performance

In a microservices architecture, services frequently communicate with one another. From a zero-trust perspective, a service should not blindly trust a userId simply because another service included it in a request.

Instead, each service should validate the session and enforce the appropriate authorization rules.

However, querying a shared database every time a service needs to validate a session can increase latency and database load, especially when requests pass through multiple services.

This creates a practical engineering challenge: how do we preserve session validity and revocation capabilities while reducing the cost of repeated remote access?

That question led me to develop Sync Session, an open-source session management library for Spring Boot.

The Design

The design combines four main components:

RDBMS as shared storage: The persistent source of truth for session state.
Local cache: Each service instance maintains a local copy of session data to reduce repeated database queries.
Event service: Session changes trigger events that allow other service instances to invalidate their corresponding local cache entries.
Session ID propagation: The session ID can be passed through a cookie or HTTP header, depending on the application's request flow.

The session itself is intentionally kept small. Rather than embedding a large amount of user data and permissions into the session, the application uses a protected session ID to retrieve the associated authenticated identity.

User information and authorization data remain under the responsibility of the relevant systems, where they can be updated independently.

This separation helps avoid relying on outdated user information stored in a session and supports centralized session revocation.

What Does This Approach Support?

The project aims to address several practical requirements:

Distributed session management across multiple service instances.
Sliding session expiration, allowing the expiration time to be extended according to the configured policy.
Administrative session revocation, enabling an administrator to force a user to log out.
Synchronization of session invalidation across service instances.
Reduced database access through local caching.

The key idea is that services can validate sessions locally in the common case, while shared storage and event-driven invalidation help coordinate changes.

Performance Reference

In my MySQL load tests, I measured the following results under a workload involving 100 threads and 1 million sessions:

Operation	Measured throughput
Session creation	~15,533 operations/sec
Local cache access	~20.01 million operations/sec
Direct database queries	~69,278 operations/sec

All 1 million session creation operations completed successfully in this test, with no reported failures.

Under these particular test conditions, local cache access achieved approximately 289 times the throughput of direct database queries.

These numbers are reference measurements from my test environment, not universal performance guarantees. The results depend on hardware, database configuration, query patterns, cache implementation, concurrency, and the exact benchmark methodology.

The comparison also measures different access paths, so it should not be interpreted as an end-to-end authentication benchmark.

Consistency and Security Trade-offs

Local caching introduces an important limitation: event-driven invalidation is asynchronous.

When a session is revoked, another service instance may still have a cached copy until it processes the invalidation event. Event delivery failures or temporary network partitions can extend this window.

For security-sensitive systems, this behavior needs to be considered explicitly. Depending on the threat model, additional safeguards may be necessary, such as bounded cache lifetimes, reliable event delivery, recovery mechanisms, or synchronous validation for particularly sensitive operations.

Likewise, validating a session is not the same as authorizing every operation. Services must still enforce their own authorization requirements, and internal service communication must be authenticated appropriately.

The goal is not to claim that local caching removes every security or consistency challenge. It is to explore a practical balance between independent session validation, efficient reads, and distributed session invalidation.

A Design Principle Worth Discussing

The principle behind Sync Session can be summarized as:

Shared storage for correctness. Local cache for performance. Events for synchronization.

This is not the only valid architecture, and it may not be appropriate for every workload. I am sharing it because session management is a recurring challenge in distributed systems, and I believe the trade-offs deserve discussion beyond the implementation of any single library.

I'd especially like to hear from engineers who have built authentication infrastructure at scale:

How do you handle session revocation across multiple service instances?
Would you accept a short invalidation delay, or require immediate revocation for every request?
How do you balance database load, cache consistency, and security requirements?
What failure scenarios would you consider essential to address in this design?

I'm happy to discuss the implementation, benchmark methodology, and architectural trade-offs.

How to Protect a Microservices Architecture with a WAF: A Practical SafeLine Checklist

Wed, 21 Oct 2026 04:29:35 +0000

Microservices improve deployment flexibility, but they also create a larger HTTP attack surface. Public APIs, authentication services, administration panels, checkout endpoints, and internal services exposed through an ingress layer may all receive traffic from the Internet. A web application firewall can provide a control point before requests reach those services, while service owners continue to manage application authentication, authorization, validation, and business logic.

SafeLine works as an HTTP/HTTPS reverse proxy. It receives traffic for the original application, filters malicious requests, and forwards safe traffic to the upstream service. The official application guide describes this model and explains that an application can be added by configuring its domain, listening port, and upstream address: SafeLine Add Application Guide. For a microservices environment, the practical question is where to place that proxy boundary.

A useful starting design is to place SafeLine at the public edge for Internet-facing domains. Route each protected hostname to the SafeLine server, then configure the corresponding upstream service or gateway. Keep the upstream services reachable from the proxy while limiting direct public access through the surrounding network controls. This makes the request path easy to reason about: client, SafeLine, ingress or service endpoint, and application.

Use a separate application entry when services have different hostnames, ports, TLS requirements, or traffic policies. Begin with the services that handle sensitive data or accept unauthenticated traffic. Record the expected methods, paths, request sizes, and client patterns for each service before changing policy. This helps teams investigate a blocked request without treating every microservice as if it had identical traffic.

SafeLine documentation lists defenses for attacks including SQL injection, XSS, code injection, OS command injection, CRLF injection, XXE, SSRF, path traversal, and related web attacks. It also documents rate limiting, CAPTCHA challenges, authentication challenges, dynamic protection for HTML and JavaScript, and access controls: SafeLine Home and Features. Apply these controls according to endpoint behavior. Rate limits can be relevant to login and password reset paths, while a CAPTCHA challenge may suit selected public forms. Avoid applying a challenge blindly to machine-to-machine APIs without testing the client workflow.

For operations, define ownership for the proxy configuration, upstream health checks, incident review, and rollback. Keep a record of each hostname and its upstream target. Test expected requests as well as representative attack cases in a controlled environment. The application guide shows examples for SQL injection, XSS, path traversal, code injection, and XXE, and explains how to review attack details in the web console. Treat those examples as validation inputs for an authorized test environment, not as a substitute for application security testing.

Deployment should match the supported environment. The installation guide lists Linux, x86_64 and arm64 instruction architectures, Docker 20.10.14 or above, Docker Compose 2.0.0 or above, and a minimum environment of one CPU core, 1 GB memory, and 5 GB disk: SafeLine Deploy Guide. Plan capacity around actual traffic and logging needs, and verify architecture specific requirements before deployment. The guide states that ARM installation requires a Pro license and that the Personal Edition is not supported on ARM.

As of September 30, 2026, SafeLine surpassed 700,000 installations worldwide. SafeLine Pro annual promotional pricing is $650-$880 versus the $1,000 regular price. Confirm the applicable offer and deployment requirements through the official project materials before making a purchase or architecture decision.

Should SafeLine protect every microservice?

Start with Internet-facing services and shared entry points. A WAF is most useful where HTTP or HTTPS traffic crosses a trust boundary. Internal services may still need authentication, authorization, input validation, and network segmentation even when traffic passes through a protected gateway.

How should upstream routing be organized?

Map each public domain to the intended upstream address and port, then verify DNS points to the SafeLine server. Keep a service inventory with owners, expected traffic, TLS settings, and rollback details. The official guide describes the domain, port, SSL option, and upstream fields used when adding an application.

Can a WAF replace microservice security controls?

No. SafeLine can filter and monitor HTTP traffic, but it does not replace identity controls, authorization checks, secure coding, secrets management, dependency updates, or service level logging. Use it as one enforcement layer in a broader architecture.

What should be tested after onboarding a service?

Test normal user journeys, API clients, authentication flows, error handling, and rate-sensitive endpoints. Then run authorized cases for common web attacks and inspect the SafeLine attack list and details. Confirm that legitimate requests pass and that blocked requests produce enough information for the responsible team to investigate.

Use this checklist to plan a measured rollout for your own microservices architecture, and review the official project and documentation for current deployment details. Project: https://github.com/chaitin/SafeLine Docs: https://docs.waf.chaitin.com/en/home Promotion: https://cyberserval.tech/landing/lto

How to Protect a Microservices Architecture with a WAF

Wed, 21 Oct 2026 04:29:03 +0000

Microservices expand delivery flexibility, but they also create more HTTP entry points, routing paths, and service boundaries to review. A practical WAF strategy begins by deciding where external traffic should enter the system and which applications need direct protection. SafeLine is documented as a self-hosted web application firewall that filters and monitors HTTP traffic between web applications and the Internet. It works as a reverse proxy, placing a protective layer in front of the application before requests reach the server (official overview).

Start by mapping the public surface of the architecture. List customer portals, authentication endpoints, API gateways, upload services, administrative interfaces, and any service exposed through a public domain. For each hostname, identify its upstream application, listening port, transport security requirements, expected request methods, and business criticality. This inventory helps you place SafeLine at the edge of each public application while keeping internal service-to-service traffic governed by the controls appropriate to your environment.

SafeLine's application model follows this reverse-proxy pattern. In the Web Admin Console, an administrator adds an application, supplies its domain, listening port, optional SSL setting, and upstream address, then resolves the domain to the SafeLine server. SafeLine receives the request, filters malicious traffic, and forwards trusted traffic to the original application (application guide). For a microservices platform, use a clear hostname and upstream mapping for every externally reachable service. This makes logs and policy decisions easier to associate with a particular API or user-facing component.

Use the WAF as one layer in a broader request policy. SafeLine's documented defenses include protection against SQL injection, XSS, code injection, OS command injection, CRLF injection, XXE, SSRF, path traversal, and related web attacks. Its documented capabilities also include IP-based rate limiting, CAPTCHA challenges, authentication challenges, proactive bot abuse defense, and web access control lists (official overview). Apply these controls according to endpoint behavior. Rate limits may be especially useful for login, password reset, search, and resource-intensive API routes. Authentication challenges can provide an additional barrier for selected administrative surfaces.

Validate the arrangement with safe test cases in a controlled environment. The application guide provides examples for SQL injection, XSS, path traversal, code injection, and XXE testing, and explains that attack details can be reviewed in the SafeLine console (application guide). Review blocked requests alongside legitimate traffic so that policies reflect real application behavior. Also confirm that upstream services are reachable through the intended path and that DNS, ports, and SSL settings match the deployment design.

For self-hosted deployment, the getting started guide lists Linux, x86_64 or arm64, Docker 20.10.14 or later, Docker Compose 2.0.0 or later, and minimum requirements of one CPU core, 1 GB memory, and 5 GB disk (deployment guide). Treat those as starting requirements and size the host according to traffic, logging, and operational needs. The same guide documents the Web Console, application data directory, and administrator account initialization process.

As of September 30, 2026, SafeLine surpassed 700,000 installations worldwide. SafeLine Pro annual promotional pricing is $650-$880 versus the $1,000 regular price. These publication details should be verified against the current commercial materials before release.

Where should SafeLine sit in a microservices deployment?

Place SafeLine in front of the public web applications or API entry points that receive traffic from the Internet. Configure each protected application with its domain, listening port, and upstream address, then direct the domain to the SafeLine server as described in the official application guide.

Should every internal service be exposed through the WAF?

Begin with externally reachable services and administrative surfaces. Internal services should remain subject to the access controls and network design used for internal traffic. Expose a service through SafeLine when it has a public HTTP or HTTPS entry point that requires web attack filtering or request abuse controls.

How can teams reduce disruption during rollout?

Inventory routes first, test representative legitimate requests, and use controlled attack examples from the application guide. Check the SafeLine attack list and application behavior after each change. This gives the team evidence about blocked attacks, expected traffic, and any policy adjustment needed before expanding coverage.

What should be reviewed after deployment?

Review application mappings, upstream reachability, SSL settings, rate-limit targets, authentication challenge scope, and attack records. Repeat the review when a service gains a new public endpoint or changes its request format.

If you are planning a microservices security review, use the official materials to build your application inventory and deployment checklist, then explore SafeLine through the project page and documentation: https://github.com/chaitin/SafeLine https://docs.waf.chaitin.com/en/home https://cyberserval.tech/landing/lto

The Order Was Saved. The Kafka Event Might Never Arrive.

Tue, 13 Oct 2026 17:20:19 +0000

A customer clicks Place order. Your API returns success. The order row is in the database. But the service that processes orders never sees an event.

This is not a hypothetical edge case invented by Kafka. It is a failure window in a common Spring Boot pattern: save to a database, then publish to a broker as a second operation. I found the pattern while reviewing my own event-driven order processing capstone. This is a code review of a learning project, not a claim that a production outage occurred.

The lesson applies well beyond orders. If your API writes data and then emits an event for fraud checks, notifications, inventory, or analytics, a successful HTTP response can tell less of the story than you think.

The four lines that made me stop

My order service saves a new order, then calls a producer:

CustomerOrder order = orderRepository.save(new CustomerOrder(...));
producer.publish(new OrderEvent(order.getId(), ...));
return toResponse(order);

The producer uses Spring Kafka like this:

public void publish(OrderEvent event) {
    kafkaTemplate.send("order-events", String.valueOf(event.orderId()), event);
}

That send returns a future. In this implementation, the future is ignored. The method can move on to the HTTP response without observing whether Kafka acknowledged the record. More fundamentally, the database write and Kafka publish are two separate operations. A process crash can occur between them even if you wait for the send result.

The code is perfectly useful for demonstrating an event-driven flow. It is not enough by itself to promise that every saved order will produce a durable event.

Where the gap opens

There are at least two ways to lose the intended handoff:

Crash after the database commit, before the send. The order exists, but no event was requested.
Asynchronous send failure after the method proceeds. The producer future completes with an error, but this code does not inspect it.

The reverse uncertainty also matters: Kafka may accept an event while the client never receives the HTTP response. A retry from the client can create another order unless the API has a separate request-idempotency rule.

I already had retries and a dead-letter topic on the consumer side of the capstone. Those help when an event reaches Kafka and processing fails. They cannot recover an event that was never published. This distinction is easy to miss when a demo looks healthy end to end.

First, choose what a successful response means

An API contract should say what has actually happened. For an asynchronous workflow, a successful create response may mean the order and the intent to process it have been stored. It should not imply that downstream processing is complete.

A practical state model might be:

CREATED → PROCESSING → APPROVED or REJECTED
                       ↘ NEEDS_ATTENTION

A GET endpoint lets the caller check the current state. An order stuck in CREATED for too long should be visible to monitoring and operators, not quietly forgotten.

The transactional outbox fix

A transactional outbox makes the event intent part of the same database transaction as the order. The API writes two rows: the business record and an outbox record containing an event ID, aggregate ID, event type, and payload. Either both rows commit or neither does.

A separate relay publishes committed outbox rows to Kafka. That relay can poll the table or use change-data capture. Debezium's outbox event router is one established implementation of the latter approach.

The core application code becomes conceptually:

@Transactional
public OrderResponse createOrder(CreateOrderRequest request) {
    CustomerOrder order = orderRepository.save(toEntity(request));
    outboxRepository.save(OutboxEvent.orderCreated(order));
    return toResponse(order);
}

This is illustrative replacement code, not a feature already implemented in my capstone. The important part is the transaction boundary around both inserts. The relay is responsible for eventual delivery after commit.

The outbox does not turn the whole system into exactly-once delivery. A relay may publish an event and crash before recording that it published it. It will publish again after restart. Consumers therefore need to handle duplicates, usually by recording a stable event ID and making the business effect idempotent.

Two IDs solve two different retry problems

It helps to keep these IDs distinct:

Request ID: supplied or assigned at the API edge so a repeated Place order request does not create a second order.
Event ID: stored with the outbox record so repeated event delivery does not apply the same downstream action twice.

Kafka producer idempotence is useful for producer retries, but it does not make a database transaction and a broker publish atomic. Nor does it replace consumer-side deduplication across relay restarts. Each layer protects a different boundary.

The test I would add before calling this production-ready

I would make failure injection part of the acceptance test:

Create an order while the broker is unavailable.
Confirm that the order and its outbox event are committed together.
Restore the broker and confirm that the relay publishes the event.
Force the relay to retry, then confirm the consumer applies the event only once.
Check that the order eventually leaves CREATED, or raises an alert if it does not.

In the current direct-send design, a broker failure or crash can leave an order without a durable event intent. The test should make that gap visible. In the outbox design, the same test checks the recovery path rather than just the happy path.

What I would monitor

A reliable event-driven API needs signals beyond HTTP success rate:

Age of the oldest unpublished outbox row
Number of publish failures and relay retries
Consumer lag and dead-letter count
Duplicate event count
Orders stuck in CREATED or PROCESSING
Time from order creation to final state

These metrics turn “the dashboard never updated” into an observable state with an owner and a recovery path.

The takeaway

My capstone taught me a useful production lesson: an event-driven workflow is only as reliable as the boundary between its database and its broker. Consumer retries are valuable, but they start too late to repair a missing publish. Save the business record and the event intent together, deliver at least once, deduplicate downstream, and monitor the gap between creation and completion.

If you have built a database-to-Kafka flow, where do you persist the intent to publish? I would be interested to hear how your team tests that failure window.

References

Stop Treating EDA as Just Pub/Sub. Here’s How to Handle Request/Reply

Tue, 13 Oct 2026 05:13:00 +0000

⚡ EDA ≠ Just Pub/Sub: Building Flexible Request/Reply Journeys Through Event Choreography

🚀 Exciting Update: Incredibly honored that this deep dive was selected for the HackerNoon Science Tech Brief within 12 hours of its original release! Sharing it here with the Coder Legion community to hear your thoughts on EDA patterns.

Note: The complete, ready-to-use article template covering the core dilemma of synchronous request/reply flows, the multi-payment processing blueprint with topic naming standards, zero-routing choreography principles, and engagement prompts for the community is fully prepared for your publication.

⚡ The Hidden Power of EDA: Building Flexible Request/Reply Journeys Through Event Choreography

When engineers discuss Event-Driven Architecture (EDA), the conversation typically centers on asynchronous pub/sub patterns, fire-and-forget processing, and real-time streaming. We praise EDA for decoupling our systems, but we often hit a wall when a core business process inherently demands a synchronous request-reply interaction.

A common misconception is that EDA is poorly suited for request-reply workflows. The standard instinct is to fall back on direct, synchronous HTTP/gRPC service calls between microservices. However, this introduces tight coupling, making it difficult to scale, evolve, or independently deploy components.

🎯 The Challenge

In traditional request/reply designs, systems can become tightly coupled through direct service calls, making it harder to scale, evolve, or introduce new capabilities into the workflow.

A well-designed event-driven choreography offers an alternative approach.

It enables an end-to-end business process to flow seamlessly:

Client Request
Backend Services
Multiple Processing Components
Business Decisions
Final Response to Client

Regardless of how many systems participate in the journey.

As a result, EDA becomes more than a mechanism for event distribution. It becomes a powerful foundation for building scalable and flexible request/reply workflows.

💳 Example: Multi-Payment Processing Platform

Let's use a payment platform that supports multiple payment methods as an example:

Bank Transfer
FPS
PPS
EFT

Although these payment methods follow different processing paths, they can share the same event-driven framework.
📌 Step 1: Establish a Well-Defined Topic Naming Standard

The well-defined topic structure with mandatory fields and optional fields. The mandatory part should include at least four fields: information domain, information subdomain, message type, and schema version.

📌 Step 2: Use the Naming Standard as the Event Flow Guideline

Events may travel through many systems during their lifecycle.
Although different systems may interpret or process the event
differently, the original business context must remain intact.
The topic representing the event should remain consistent with the
event's original business purpose.

On each event flow, the mandatory parts (red and blue colors) are placed in the message header and brought forward to all subsequent messages in the flow. This part should not be changed throughout the lifetime of the event flow. For each component, the output topic will be the input messages’ mandatory part plus this component function, plus the process status. It is very easy to subscribe to all corresponding events within the same event flow by using a wildcard.

🧩 Loosely Coupled Processing Components

An important characteristic of this design is that individual processing components do not contain built-in routing logic.

Each component simply:

📥 Consumes events it is interested in
⚙️ Performs its business function
📤 Publishes the next event

The component has no knowledge of:

❌ Which flow it belongs to
❌ Which system produced the event
❌ Which system will consume the next event

This significantly improves reusability and maintainability.

✅ The Result

This architecture enables a set of common reusable components to support multiple request/reply workflows across different channels and payment methods.

Key Outcomes

Consistent event schema across multiple use cases
All downstream events inherit and extend the original business context
Additional analytical services can be introduced without impacting existing applications
Customer behavior and application activities can be monitored in real time
New processing components can be added or removed easily
Existing applications require no code changes
Most changes can be achieved through topic or queue subscription configuration alone

💡 Final Thought

One of the biggest misconceptions about Event-Driven Architecture is that it is only suitable for asynchronous event broadcasting.

In reality, when combined with strong event governance, consistent topic standards, and event choreography, EDA can also provide an elegant solution for complex request/reply interactions.

The result is an architecture that is:

Loosely Coupled
Highly Scalable
Easily Extensible
Business-Driven
Future-Proof

How does your organization handle request/reply patterns in an event-driven landscape? I'd love to hear your experiences and perspectives in the comments.

Stop Wasting Days on Dev Setup: How We Automated Our Microservices Environment for Every Developer

Thu, 24 Sep 2026 01:52:08 +0000

Stop Wasting Days on Dev Setup: How We Automated Our Microservices Environment for Every Developer

How we reduced new developer onboarding from 3 days to 15 minutes and eliminated configuration drift across 12 microservices.

The Problem That Made Us Snap

Picture this: A new senior developer joins your team. They're excited, ready to ship code on day one. Then reality hits.

Day 1: They spend 6 hours installing Node, Docker, configuring ESLint to match the team's style, figuring out which environment variables go where, and asking Slack questions like "which version of NestJS are we on again?"

Day 2: They finally get user-service running, but auth-service throws a CORS error because their local config is slightly different from everyone else's. They spend the afternoon comparing .env files with a teammate.

Day 3: They submit their first PR. The CI pipeline fails because their local Prettier config formats code differently from the shared config that was updated last month — but nobody told them.

Sound familiar?

We had 12 microservices, a team that was growing fast, and a configuration mess that was eating our productivity alive. Every service had slightly different ESLint rules, Docker setups, folder structures, and VS Code settings. Senior developers were spending hours every week answering setup questions instead of building features.

So we built a system that fixed all of it. Here's exactly how.

The Solution: One Repo to Rule Them All

We created a central configuration repository called dev-environment-config that acts as the single source of truth for every development environment across our entire microservices ecosystem.

The idea is simple:

One repository controls the setup, structure, linting, Docker config, VS Code settings, and shared code for every service. When a senior developer updates it, every developer's machine automatically syncs the next morning.

The Architecture at a Glance

dev-environment-config (Central Repo)
├── templates/          → Project scaffolding (NestJS, React)
├── shared/             → ESLint, Prettier, Git configs
├── vscode/             → Extensions & settings
├── docker/             → Base Docker configs
├── scripts/            → Automation (setup, sync, update)
├── common-lib/         → Shared NPM package
└── manifest.json       → Version tracking

         │  syncs to
         ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ user-svc │ │ auth-svc │ │ pay-svc  │  ... 12 services
│ .eslint  │ │ .eslint  │ │ .eslint  │
│ .vscode  │ │ .vscode  │ │ .vscode  │
│ Docker   │ │ Docker   │ │ Docker   │
└──────────┘ └──────────┘ └──────────┘

Implementation: Step by Step

Step 1: The Central Config Repository

We created a new repository with this structure:

dev-environment-config/
├── templates/
│   ├── nestjs-service/    # Complete NestJS starter
│   └── reactjs-service/   # Complete React starter
├── shared/
│   ├── .eslintrc.js
│   ├── .prettierrc
│   ├── .editorconfig
│   ├── .gitignore
│   └── commitlint.config.js
├── vscode/
│   ├── extensions.json
│   ├── settings.json
│   └── launch.json
├── scripts/
│   ├── setup-new-developer.sh
│   ├── sync-config.sh
│   ├── auto-update.sh
│   └── create-new-service.sh
├── common-lib/            # Shared NPM package
└── manifest.json          # Version tracker

The manifest.json is the heartbeat of the system:

{
  "version": "1.2.0",
  "lastUpdated": "2024-01-15T10:00:00Z",
  "updatedBy": "senior-dev",
  "configs": {
    "eslint": "3.2.0",
    "prettier": "2.1.0",
    "docker": "1.5.0"
  },
  "services": [
    "user-service",
    "auth-service",
    "payment-service"
  ]
}

Every time a config changes, we bump the version. This is how the auto-update system knows something changed.

Step 2: The One-Command Onboarding Script

This was the game-changer. We wrote a bash script that a new developer runs once, and their entire environment is ready.

The new developer's experience:

Install VS Code ✅
Run one command:

curl -sSL https://raw.githubusercontent.com/company/\
dev-environment-config/main/scripts/\
setup-new-developer.sh | bash

Wait ~10 minutes. Done.

What the script does under the hood:

#!/bin/bash
set -e

WORKSPACE_DIR="$HOME/company-workspace"
CONFIG_REPO="https://github.com/company/dev-environment-config.git"

echo "🚀 Setting up your development environment..."

# 1. Check & install prerequisites
check_and_install "git" "Git"
check_and_install "node" "Node.js" "nvm install 20"
check_and_install "docker" "Docker"

# 2. Create workspace & clone everything
mkdir -p "$WORKSPACE_DIR" && cd "$WORKSPACE_DIR"
git clone "$CONFIG_REPO"

# 3. Clone all service repos from manifest
SERVICES=$(cat dev-environment-config/manifest.json | \
  python3 -c "import sys,json; \
  [print(s) for s in json.load(sys.stdin)['services']]")

for service in $SERVICES; do
  git clone "https://github.com/company/${service}.git"
done

# 4. Install VS Code extensions
EXTENSIONS=$(cat dev-environment-config/vscode/extensions.json | \
  python3 -c "import sys,json; \
  [print(e) for e in json.load(sys.stdin)['recommendations']]")

for ext in $EXTENSIONS; do
  code --install-extension "$ext" --force
done

# 5. Sync shared configs to all services
bash dev-environment-config/scripts/sync-config.sh

# 6. Install dependencies everywhere
for service in $SERVICES; do
  cd "$service" && npm install && cp .env.example .env && cd ..
done

# 7. Set up daily auto-update cron (8 AM weekdays)
CRON="0 8 * * 1-5 cd $WORKSPACE_DIR && \
  bash dev-environment-config/scripts/auto-update.sh"
(crontab -l 2>/dev/null; echo "$CRON") | crontab -

echo "✅ Setup complete! Open VS Code: code $WORKSPACE_DIR"

That's it. The developer opens VS Code, and everything is configured — extensions, linting, formatting, Docker, environment variables, the works.

Step 3: The Config Sync Script

This script copies the shared configs from the central repo into every service:

#!/bin/bash
# sync-config.sh

CONFIG_ROOT="$(dirname "$SCRIPT_DIR")"
WORKSPACE_DIR="$(dirname "$CONFIG_ROOT")"

for service in $SERVICES; do
  SERVICE_DIR="$WORKSPACE_DIR/$service"

  # Copy shared configs
  cp -f "$CONFIG_ROOT/shared/.eslintrc.js" "$SERVICE_DIR/"
  cp -f "$CONFIG_ROOT/shared/.prettierrc" "$SERVICE_DIR/"
  cp -f "$CONFIG_ROOT/shared/.editorconfig" "$SERVICE_DIR/"
  cp -f "$CONFIG_ROOT/shared/.gitignore" "$SERVICE_DIR/"

  # Copy VS Code settings
  mkdir -p "$SERVICE_DIR/.vscode"
  cp -f "$CONFIG_ROOT/vscode/settings.json" "$SERVICE_DIR/.vscode/"
  cp -f "$CONFIG_ROOT/vscode/extensions.json" "$SERVICE_DIR/.vscode/"

  echo "✅ $service synced"
done

Step 4: The Daily Auto-Update (The Magic)

This is the part that keeps everything in sync without anyone thinking about it.

Every morning at 8 AM (weekdays), a cron job runs auto-update.sh on every developer's machine:

#!/bin/bash
# auto-update.sh — Runs daily at 8 AM

cd "$CONFIG_ROOT"

# 1. Check for config updates
OLD_VERSION=$(cat manifest.json | jq -r '.version')
git pull origin main
NEW_VERSION=$(cat manifest.json | jq -r '.version')

if [ "$OLD_VERSION" != "$NEW_VERSION" ]; then
  echo "🆕 Config updated: $OLD_VERSION → $NEW_VERSION"
  bash "$SCRIPT_DIR/sync-config.sh"
fi

# 2. Pull latest code for all services
for service in $SERVICES; do
  cd "$WORKSPACE_DIR/$service"

  # Stash local changes safely
  git stash push -m "auto-update-$(date +%Y%m%d)" 2>/dev/null

  # Pull latest
  git pull origin $(git branch --show-current) --rebase

  # Update dependencies if package.json changed
  if git diff HEAD~1 --name-only | grep -q "package.json"; then
    npm install
  fi

  # Update shared library
  npm update @company/common-lib
done

# 3. Notify the developer
notify-send "Dev Environment" "✅ Updated to config v$NEW_VERSION"

The flow:

Senior Dev pushes config change
         │
         ▼
   GitHub Actions
         │
    ┌────┴────┐
    │ Slack   │ Notify team
    │ Publish │ common-lib to NPM
    └────┬────┘
         │
    Next 8 AM
         │
         ▼
  Every developer's cron runs
         │
    ┌────┴────────────────┐
    │ Pull config repo    │
    │ Sync shared configs │
    │ Pull service repos  │
    │ Update dependencies │
    │ Desktop notification│
    └─────────────────────┘

Step 5: The Shared Common Library

Instead of copying and pasting guards, interceptors, and DTOs across 12 services, we created @company/common-lib — a private NPM package:

// In any service, just import and use:
import {
  AuthGuard,
  RolesGuard,
  ResponseInterceptor,
  LoggingInterceptor,
  HttpExceptionFilter,
  PaginationDto,
  McpService,        // Qwen MCP integration
} from '@company/common-lib';

@Controller('users')
@UseGuards(AuthGuard, RolesGuard)
@UseInterceptors(ResponseInterceptor)
@UseFilters(HttpExceptionFilter)
export class UserController {
  constructor(private mcpService: McpService) {}

  @Get()
  findAll(@Query() pagination: PaginationDto) {
    return this.userService.findAll(pagination);
  }

  @Post('analyze')
  async analyze(@Body() dto: AnalyzeDto) {
    return this.mcpService.callTool('text-analysis', {
      text: dto.text,
    });
  }
}

When we update the common library (add a new guard, fix a bug in the exception filter), the auto-update system bumps the version in every service automatically.

Step 6: Service Scaffolding

Creating a new microservice used to take a day of copy-pasting and tweaking. Now it takes 30 seconds:

bash scripts/create-new-service.sh

# Prompts:
# Service name: order-service
# Description: Handles order processing
# Port: 3002
# Type: nestjs

# ✅ Done! Complete service with:
# - src/common/ (filters, guards, interceptors, DTOs)
# - Docker setup (Dockerfile + docker-compose)
# - VS Code settings
# - ESLint + Prettier
# - Health check endpoint
# - Swagger docs
# - MCP integration
# - Auto-update scripts

The Benefits (With Real Numbers)

1. Onboarding: 3 Days → 15 Minutes

Before	After
Manual setup following a 40-step Confluence page	One `curl` command
3 days to first productive commit	15 minutes to running all services
"Hey, can you help me with my env vars?" (daily Slack noise)	Zero setup questions

2. Configuration Drift: Eliminated

Before, every service slowly diverged. user-service had ESLint v8, auth-service had v7. Prettier configs had subtle differences. Docker setups were copy-pasted and never updated.

Now, every service has identical configs, and they stay identical because the sync script overwrites them daily.

3. Senior Developer Time: Saved ~5 Hours/Week

Senior devs were spending roughly an hour a day helping juniors with setup issues, reviewing PRs that failed CI due to formatting, and manually updating configs across services.

That time is now spent building features.

4. Consistent Code Quality

Every developer has:

The same ESLint rules (auto-fixed on save)
The same Prettier config (format on save)
The same commit message format (enforced by Husky)
The same VS Code extensions (auto-installed)

PR reviews went from "please fix your formatting" to "let's discuss the architecture."

5. Zero-Effort Config Updates

When a senior developer wants to:

Add a new ESLint rule → Push to config repo → Everyone gets it tomorrow
Update Docker base image → Push → Auto-synced
Add a VS Code extension → Push → Auto-installed
Fix a bug in the exception filter → Publish common-lib → Auto-updated

No Slack announcements needed. No "please run npm install" messages. It just happens.

6. Standardized Project Structure

Every service looks the same. A developer can jump from user-service to payment-service and immediately know where everything is. The src/common/ folder has the same patterns. The Docker setup is identical. The health check endpoint is at the same path.

What We Learned

Start Small

We didn't build this all at once. We started with just syncing ESLint and Prettier configs. Once the team saw the value, we added Docker, VS Code settings, the common library, and the auto-update system incrementally.

Don't Force Everything

We allow service-specific overrides. If payment-service needs a special ESLint rule, they can add a .eslintrc.local.js that extends the shared one. The shared config is the baseline, not a straitjacket.

Notifications Matter

The desktop notification ("✅ Updated to config v1.2.0") was a small touch, but it made developers aware of the system. Before that, updates happened silently and people were confused when things changed.

Version Everything

The manifest.json version tracking was critical. Without it, the auto-update script had no way to know if anything changed, and it would waste time syncing configs that were already up to date.

The Complete Stack

Layer	Technology
Backend	NestJS (TypeScript)
Frontend	ReactJS (TypeScript)
AI	Qwen MCP
Containers	Docker, Docker Compose
Runtime	Node.js 20 LTS
Database	PostgreSQL 16
Cache	Redis 7
CI/CD	GitHub Actions
Editor	VS Code
Config Sync	Bash scripts + Cron

TL;DR

If your team has more than 3 microservices and more than 5 developers, you need a centralized dev environment config. Here's the recipe:

Create a config repo with templates, shared configs, and scripts
Write a setup script that onboards new devs in one command
Write a sync script that copies shared configs to all services
Set up a cron job that auto-updates every morning
Create a shared NPM library for reusable code
Build a service generator for scaffolding new microservices

The result: developers spend their time writing code, not configuring environments.

Have you solved this problem differently at your company? I'd love to hear about it in the comments.

If you found this useful, give it a clap 👏 and follow for more microservices architecture deep-dives.

A Tiny Nim Microservice, Three Real Bugs, and What They Taught Me About My Own Mummy Fork

Tue, 15 Sep 2026 00:19:50 +0000

I recently needed something small: a POST /api/contact endpoint for my portfolio site at https://www.isaiah.name.ng that takes a name, email, and message, validates it, and emails it to me. The kind of thing you'd normally knock out without thinking twice.

I decided to build it directly on my own Mummy fork — the one with OpenAPI schema generation, typed validation, and composable middleware — partly to actually dogfood it on something real, and partly because I assumed a single-endpoint service would be trivial. It mostly was. But three specific bugs along the way taught me more about the fork's internals than building the fork itself did.

The Shape of the Service

Nothing exotic: a ContactBody type with name, email, and message, all required strings. Validation happens in two layers — parseValidatedBody[ContactBody] checks presence and type against the fork's schemaOf-generated schema, and then explicit checks enforce non-empty fields, a 2000-character cap, and a hand-rolled email format check. Either layer raising the fork's ValidationError gets mapped to a clean 400 {"error": ...} by installDefaultErrorHandler. A send failure is logged server-side and answered with a generic 500 — the caller never sees why an email failed to send, only that it did.

On top of that: CORS middleware that rejects disallowed origins with 403 and answers preflight requests properly, an in-memory rate limiter capping submissions at 5 per IP per 10 minutes, and the fork's loggingMiddleware as the outermost layer so every request logs method, path, and timing. The OpenAPI docs come free from the same schema the validator uses, served at /docs with the spec at /openapi.json — the whole reason I built the fork in the first place.

The Pivot Nobody Plans For: SMTP → Resend

The original plan was plain SMTP via std/smtp. It compiled fine against the fork. Then I deployed to Render's free tier and every send just hung until timeout — Render blocks outbound SMTP ports (25/465/587) on that tier. Resend's REST API runs over HTTPS:443, which isn't blocked, and needs just one API key. As a bonus, its failures come back as ordinary HTTP errors the handler already catches and logs, instead of the occasional AssertionDefect that a raw SMTP library can throw and potentially take a worker down with it.

This is the kind of constraint you only discover by actually deploying, not by reading docs — worth remembering if you're building anything Render-hosted that needs to send mail.

Three Bugs That Taught Me More Than the Feature Did

1. Two HttpHeaders types that look identical and aren't.
Mummy imports std/httpcore aliased internally, and Request.headers is typed against that aliased import. My module also did a plain import std/httpcore for the Resend payload types — and suddenly every header operation threw a "type mismatch" between what looked like the same type. Nim's module system treats the two imports as distinct type identities even though they're structurally the same. The fix was narrow: import std/httpcore except HttpHeaders, so my module only pulled in the pieces it needed for the Resend payload and left the header type alone. This was, by a wide margin, the buggiest hour of the whole build — and a good reminder that "same type, different import path" is a real failure mode in Nim, not a theoretical one.

2. gcsafe doesn't care that your global never actually changes.
The fork's Middleware and RequestHandler types are marked {.gcsafe.}, which means the compiler's data-race checker gets strict about any global state touched inside a handler — because Mummy dispatches requests across a worker-thread pool, not a single event loop. My CORS middleware read a var global holding the allowed-origins list, set once at startup and never touched again — and the compiler didn't care that it was logically immutable after init. It only sees a mutable global read from a gcsafe context. The fix was a {.cast(gcsafe).} block around that specific read, while the genuinely-shared rate-limit table stayed behind a real Lock — the same pattern the fork already uses internally for bearerAuthMiddleware. It's a useful distinction to internalize on a threaded server: "never mutated after startup" and "thread-safe" are not the same claim, and the compiler will only ever check for the second one.

3. An unregistered route means your middleware never runs — including for the request that most needs it.
Browsers send an OPTIONS preflight before certain cross-origin POST requests. Mummy only runs its middleware chain for methods that have a registered route — so with no OPTIONS /api/contact route, Mummy answered preflight with its own 405 before corsMiddleware ever got a chance to run. The browser never saw the Access-Control-Allow-Origin header it needed, so the real POST was blocked client-side despite the server being otherwise correctly configured. The fix was almost comically small: register an OPTIONS /api/contact route whose handler does nothing but exist, which is enough to route the request into the middleware chain where corsMiddleware can actually answer it with a 204. Easy to fix once you know it — genuinely confusing the first time a working-looking CORS setup silently fails only on the preflight leg.

What Actually Got Verified

Before calling it done, I ran it locally and checked the specific paths that tend to hide bugs rather than just the happy path: a disallowed origin (403), a malformed email (400 with the exact field named), a name at the 2000-character boundary (passes) versus one character over (rejected), six rapid submissions from one IP (five succeed, the sixth comes back 429 with Retry-After), and that /health and OPTIONS both stay outside the rate limiter so Render's health checks and CORS preflight never get throttled.

Why This Was Worth Writing Up

None of these three bugs were about the feature — a contact form endpoint is about as unglamorous as backend work gets. They were about the specific intersection of Nim's module aliasing, gcsafe's literal (not logical) read of what's safe, and Mummy's routing-before-middleware order. If you're building anything on Mummy — my fork or the original — that touches CORS, thread-shared state, or has more than one import path into std/httpcore, these three are worth checking for before you hit them the hard way.

The full service, Dockerfile, and a drop-in script.js for wiring this into any static contact form are in the repo if you want to see the whole thing end to end.

Idempotency in APIs - Why Your Retry Logic Can Break Everything (And How to Fix It)

Sat, 12 Sep 2026 13:30:46 +0000

Modern applications rely heavily on APIs. But networks are unreliable - requests fail, time out, or get retried.

Now imagine this scenario:

User clicks “Pay Now”
Request sent to server
Payment processed successfully
But response is lost due to network issue
Client retries the request

💥 Payment gets processed again. User is charged twice.

This is one of the most dangerous and common problems in distributed systems.

The solution? Idempotency.

What is Idempotency?

An operation is idempotent if performing it multiple times produces the same result as performing it once.

In simple terms:

Same request → same effect → no duplicate side effects

Example:

Create Order (without idempotency):
Request sent twice → 2 orders created ❌

Create Order (with idempotency):
Request sent twice → only 1 order created ✅

Idempotency ensures system correctness even when retries happen.

Why Do Retries Happen?

Retries are extremely common due to:

Network timeouts
Server crashes
Client retries
Load balancers retrying requests
Mobile networks instability
API gateway retries
Reverse proxy retries

Retries are normal. Duplicate side effects are not.

Which HTTP Methods Are Idempotent?

According to HTTP specification:

Method	Idempotent	Explanation
GET	✅ Yes	Fetch data
PUT	✅ Yes	Replace resource
DELETE	✅ Yes	Delete resource
POST	❌ No	Create resource
PATCH	❌ No	Partial update

POST is NOT idempotent by default.

Example:

POST /orders

Calling twice → creates 2 orders.

This must be handled explicitly.

Real-World Example: Payment API Problem

Client sends:

POST /payments
{
  "user_id": "123",
  "amount": 100
}

If client retries due to timeout:

POST /payments

Without idempotency → duplicate payment ❌

With idempotency → single payment ✅

Solution: Idempotency Key

Client sends a unique key with request:

POST /payments
Idempotency-Key: abc123

Server behavior:

If key is new → process request
If key already exists → return previous result

This guarantees single execution.

How Idempotency Works Internally

Flow:

Client generates unique key
Sends request with key
Server checks database/cache for key
If key exists → return stored response
If not exists → process request
Store key and response
Return response

Go Implementation Example

Let’s implement idempotent payment API using Go.

Database Table

idempotency_keys table:

key (primary key)
response
created_at

Go Example (Basic Version)

type IdempotencyRecord struct {
    Key      string
    Response string
}

Handler Implementation

func PaymentHandler(w http.ResponseWriter, r *http.Request) {

    key := r.Header.Get("Idempotency-Key")

    if key == "" {
        http.Error(w, "Idempotency-Key required", http.StatusBadRequest)
        return
    }

    // Check if key exists
    record, exists := getIdempotencyRecord(key)

    if exists {
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(record.Response))
        return
    }

    // Process payment
    paymentID := processPayment()

    response := fmt.Sprintf(`{"payment_id": "%s"}`, paymentID)

    // Store key and response
    saveIdempotencyRecord(key, response)

    w.Header().Set("Content-Type", "application/json")
    w.Write([]byte(response))
}

Example Flow

First request:

POST /payments
Idempotency-Key: abc123

Response:
{
  "payment_id": "pay_001"
}

Retry request:

POST /payments
Idempotency-Key: abc123

Response returned from storage:

{
  "payment_id": "pay_001"
}

No duplicate payment created.

Production Implementation Using Redis (Recommended)

Redis is ideal because:

Extremely fast
Supports expiration (TTL)
Perfect for temporary idempotency storage

Redis Example in Go

func PaymentHandler(w http.ResponseWriter, r *http.Request) {

    key := r.Header.Get("Idempotency-Key")

    val, err := redisClient.Get(ctx, key).Result()

    if err == nil {
        w.Write([]byte(val))
        return
    }

    paymentID := processPayment()

    response := fmt.Sprintf(`{"payment_id": "%s"}`, paymentID)

    redisClient.Set(ctx, key, response, time.Hour*24)

    w.Write([]byte(response))
}

Important: Handle Race Conditions

Two requests may arrive simultaneously with same key.

Solution: Use atomic operations.

Redis example:

SET key value NX

NX = set only if not exists

Important Best Practice: Store Full Response

Do not just store key.

Store:

status code
response body
timestamp

Because retry must return exact same response.

Example Database Schema (PostgreSQL)

CREATE TABLE idempotency_keys (
    key TEXT PRIMARY KEY,
    response JSONB,
    status_code INT,
    created_at TIMESTAMP DEFAULT NOW()
);

Primary key ensures uniqueness.

Real-World Systems That Use Idempotency

Critical systems use idempotency extensively:

Payment APIs
Order creation APIs
Financial transactions
Booking systems
Wallet transfers
Stripe payments
Banking APIs

Without idempotency, financial systems would break.

Common Mistake Developers Make

❌ Using retry without idempotency

retry 3 times on failure

This multiplies side effects.

Example:

Retry logic: 3 times
Result: 3 payments

Dangerous bug.

Idempotency vs Duplicate Prevention Using Unique Constraints

Example:

UNIQUE(user_id, order_id)

This prevents duplicates at DB level.

But idempotency is better because:

Works at API level
Handles retries cleanly
Returns same response

Best approach: use BOTH.

When Should You Use Idempotency?

Use idempotency for operations that create side effects:

Payment creation
Order creation
Wallet transfer
Booking
Resource creation

Not required for:

GET requests
Read-only operations

How Client Should Generate Idempotency Key

Use UUID:

Example:

550e8400-e29b-41d4-a716-446655440000

Go example:

uuid.New().String()

Unique per operation.

Idempotency Key Expiration Strategy

Keys should expire after some time.

Common TTL:

24 hours
48 hours
7 days (depending on business need)

Redis makes expiration easy.

Example Architecture

Client → API Gateway → Service → Redis → Database

Redis handles idempotency keys
Database handles actual data

Fast and reliable.

Real Production Scenario: Mobile Network Retry

Mobile user taps "Place Order"

Network slow → timeout
Client retries automatically

Without idempotency → duplicate orders

With idempotency → safe retry

Important Difference: Safe Retry vs Unsafe Retry

Unsafe retry:

POST /orders

Safe retry:

POST /orders
Idempotency-Key: unique-key

Idempotency vs Exactly-Once Execution

Exactly-once execution is impossible in distributed systems.

Idempotency provides practical solution.

It ensures same final state.

Best Practices Checklist

Always:

Use idempotency keys for POST APIs
Store keys in Redis or database
Store full response
Use expiration
Handle race conditions
Use UUID keys
Combine with database constraints

Key Takeaway

Retries are normal.

Duplicate side effects are dangerous.

Idempotency ensures your APIs remain safe, reliable, and production-ready.

It is not optional for critical APIs - it is essential.

Final Thought

If your API handles payments, orders, bookings, or financial transactions - idempotency is mandatory.

Without it, retries can silently corrupt your system.

With it, your APIs become reliable, predictable, and safe.

Don't Let One Broken Service Crash Your Whole App: The Circuit Breaker Pattern

Wed, 09 Sep 2026 08:07:41 +0000

What is the Circuit Breaker Pattern?

The Circuit Breaker Pattern is an architectural design pattern used in software development to detect failures and encapsulate the logic of preventing a failure from cascading throughout a system. It works by wrapping a sensitive network call or database query in a monitoring object that tracks recent failures. When those failures exceed a pre-defined threshold, the breaker trips, automatically blocking subsequent requests to prevent further damage. This prevents a system from repeatedly executing an operation that is almost guaranteed to fail, protecting both the client and the struggling service.

A Real-Life Analogy: The Household Fuse Box

Imagine you are in your kitchen on a weekend morning. You decide to toast some bread, but your toaster has an internal short circuit. When you push down the lever, instead of just warming up, the faulty appliance starts drawing an unsafe, massive surge of electricity. Without a safety device in place, the copper wiring in your kitchen walls would quickly overheat, melt its protective insulation, and potentially start a devastating house fire.

Fortunately, your home features an electrical circuit breaker panel. The breaker detects this dangerous spike in electrical current and instantly trips, cutting off the flow of electricity to the kitchen outlets. The toaster is dead, but your home is completely safe because the danger was isolated. To fix things, you unplug the toaster, run to the garage, and flip the breaker switch back to its original position to restore power to the remaining kitchen appliances.

Why It Matters in Daily Tech Operations

In modern software, apps are rarely self-contained. They rely on microservices—smaller, interconnected applications—and external APIs (Application Programming Interfaces) to run. For instance, an e-commerce platform relies on a product catalog service, a payment gateway, and an email service.

If the payment gateway slows down or crashes during a sale, what happens? Without a circuit breaker, every customer trying to checkout will wait indefinitely. Each waiting user consumes system resources, such as memory and web threads, on your server. As more users click checkout, these resources quickly exhaust. Within minutes, your core checkout service crashes under the pressure, which then causes the inventory service to crash, leading to a complete system outage.

By implementing the Circuit Breaker Pattern, engineers can safeguard their systems. Once the circuit breaker detects that the payment gateway is failing, it immediately trips. New checkout requests instantly receive a friendly message like, "Our payment gateway is busy, please try again in a moment." This spares your servers from waiting on a dead service, allowing the rest of your website to remain online and functional.

The Concept in Action

Here is a simple, lightweight implementation of a circuit breaker in JavaScript to show how this logic looks under the hood:

class SimpleCircuitBreaker {
  constructor(requestFunction, failureThreshold = 3, cooldownMs = 10000) {
    this.requestFunction = requestFunction;
    this.failureThreshold = failureThreshold;
    this.cooldownMs = cooldownMs;
    this.state = "CLOSED";
    this.failures = 0;
    this.nextAttemptTime = 0;
  }

  async execute() {
    if (this.state === "OPEN") {
      if (Date.now() > this.nextAttemptTime) {
        this.state = "HALF-OPEN";
      } else {
        throw new Error("Circuit is OPEN. Request blocked for safety.");
      }
    }

    try {
      const result = await this.requestFunction();
      this.reset();
      return result;
    } catch (error) {
      this.handleFailure();
      throw error;
    }
  }

  reset() {
    this.state = "CLOSED";
    this.failures = 0;
  }

  handleFailure() {
    this.failures++;
    if (this.failures >= this.failureThreshold) {
      this.state = "OPEN";
      this.nextAttemptTime = Date.now() + this.cooldownMs;
      console.log("Circuit breaker tripped! Blocking future requests.");
    }
  }
}

The Takeaway

Building highly reliable modern software systems is not about pretending that failures will never occur; it is about managing those failures gracefully when they inevitably do. The Circuit Breaker Pattern shifts our architectural strategy from fragile, hopeful connections to defensive, self-healing resilience. By failing fast, your application protects its precious memory and computing power, preventing localized outages from snowballing into catastrophic system-wide downtime.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Don't Let Your Microservices Burn: Understanding the Circuit Breaker Pattern

Fri, 04 Sep 2026 08:20:08 +0000

In modern software engineering, building a reliable system isn't just about preventing errors; it is about managing them gracefully when they occur. The Circuit Breaker Pattern is a design pattern used in software development to prevent an application from repeatedly trying to execute an operation that is highly likely to fail. Instead of wasting resources on doomed requests, the system temporarily blocks them, allowing the failing service time to recover. Once the service is healthy again, the block is lifted and normal operations resume.

A Real-World Analogy: The Household Breaker

Think of a household electrical circuit breaker. If a faulty microwave draws too much current, the circuit breaker trips, instantly shutting off electricity to that part of the house. This prevents the wires from overheating and catching fire. Instead of continuously feeding dangerous levels of electricity into a broken appliance, the breaker pauses the flow. It stays open until you unplug the bad appliance and reset the switch.

In software, the concept is identical. When a third-party API or server becomes slow or non-responsive, our virtual circuit breaker trips. It stops sending traffic to that broken service, saving the rest of our application from catching fire and crashing due to backlogged requests.

Why It Matters in Tech

In modern cloud architectures, systems constantly communicate with each other. If a database or a payment gateway goes down, and your application keeps hammering it with thousands of requests per second, you will exhaust your own server's memory and database connection pools. This causes a "cascading failure" where your entire platform goes offline because of one minor, non-critical dependency.

Engineers use circuit breakers to "fail fast." Instead of letting a user wait 30 seconds for a timeout, the circuit breaker immediately throws an error or returns cached data. This protects downstream systems from being crushed while they are struggling, and keeps your primary application fast and responsive for other tasks.

The Pattern in Action

Here is a simple JavaScript implementation of a state-based circuit breaker. It wraps an API call and tracks failures:

class CircuitBreaker {
  constructor(requestFunction, failureThreshold, recoveryTime) {
    this.request = requestFunction;
    this.failureThreshold = failureThreshold;
    this.recoveryTime = recoveryTime;
    this.state = "CLOSED"; // CLOSED, OPEN, HALF-OPEN
    this.failures = 0;
    this.nextAttempt = Date.now();
  }

  async execute(...args) {
    if (this.state === "OPEN") {
      if (Date.now() > this.nextAttempt) {
        this.state = "HALF-OPEN";
      } else {
        throw new Error("Circuit is OPEN. Request blocked.");
      }
    }

    try {
      const result = await this.request(...args);
      this.reset();
      return result;
    } catch (error) {
      this.recordFailure();
      throw error;
    }
  }

  reset() {
    this.state = "CLOSED";
    this.failures = 0;
  }

  recordFailure() {
    this.failures++;
    if (this.failures >= this.failureThreshold || this.state === "HALF-OPEN") {
      this.state = "OPEN";
      this.nextAttempt = Date.now() + this.recoveryTime;
    }
  }
}

In this code, the breaker starts as CLOSED (operational). If the nested request fails too many times, the state flips to OPEN, blocking further attempts immediately. After the recovery time passes, it allows a single test request (HALF-OPEN) to see if the dependency is back online. If it succeeds, the breaker resets to CLOSED.

The Takeaway

The Circuit Breaker Pattern shifts our engineering mindset from trying to prevent all failures to gracefully managing them when they inevitably happen. By isolating broken parts of a system, you ensure that a single slow API call doesn't drag down your entire product, keeping your user experience stable even during partial outages.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Service Mesh Zero-Trust Architecture

Tue, 01 Sep 2026 22:13:24 +0000

Service Mesh Zero-Trust Architecture

Zero-trust is a property that emerges when several independent checks each verify a request on their own, without assuming the layers around them already did the work. It often gets described as a product you install or a setting you switch on, which badly undersells how much has to line up for it to actually hold.

I learned that building an HR directory designed to be attacked. The rule I wanted it to hold was simple: if any single pod gets compromised, it still can't read data it has no business touching. Salaries, bank details, device serial numbers, the things an attacker would actually go after.

No single mechanism enforces that rule. mTLS proves which service is on the other end of a connection, but it has nothing to say about whether that service should be making the call. A signed identity token establishes who the user is, yet a service that trusts raw headers will believe whatever an attacker sends it. Authorization decides what a user is allowed to do, and then an application bug skips the check and the database returns the whole table anyway. Every layer has a blind spot, and the thing that covers it is another layer that doesn't share the same one.

So the architecture is a stack: every hop independently verified, every request authenticated and authorized, each layer built on the assumption that the one before it might fail. Remove a single layer and the others are left exposed in the exact spots they were relying on it to cover.

This post is the architecture overview for the series: the application, the five security layers, how they compose into a single request flow, and what specifically breaks when you remove one. It all runs end-to-end on Istio on a local Kind cluster with real config. Later posts go deep into the YAML, Python, and policy files.

What you need to know going in: Istio is a service mesh for Kubernetes. The key idea is that every pod gets an Envoy sidecar proxy. All inbound and outbound traffic passes through the sidecar, which means identity verification, routing rules, and policy enforcement happen at the infrastructure layer before your application code sees the request. If you understand containers and basic networking, the rest will make sense as we go.

The Application

The domain is deliberately simple: an HR Employee Directory. Five microservices, a few database tables, a handful of roles. The security architecture is the interesting part. In the demo there are 4 roles used.

Alice (employee) — on her own profile she sees full PII and exact salary; everyone else only gets name, title, and department.
Mary (manager) — on Alice (her direct report) she sees salary band, not exact pay or SSN; on others, same public view as any employee.
Henry (HR admin) — full HR fields on all employees (salary, SSN, etc.), but no full IT asset serial numbers.
Ivan (IT admin) — only basic employee fields everywhere, but full hardware/asset details including serials.

The services:

Profile Aggregator (ms1-profile-aggregator) is a fan-out orchestrator that calls Employee Records and Device Inventory, merges the results, and returns a single profile response. Gateway-facing, requires authentication.
Employee Records (ms2-employee-details) is the source of truth for employee records, PII, and financial data (salary, bank details). Internal only, not directly reachable from outside.
Device Inventory (ms3-hardware-assets) tracks which devices are assigned to which employees (serial numbers, MAC addresses). Internal only, same as Employee Records.
Holiday Calendar (ms4-holiday-calendar) handles company holidays. Gateway-facing, requires authentication. Read by everyone, written by HR.
Office Directory (ms5-office-locations) serves office location data. Gateway-facing, no authentication required for reads. The only fully public endpoint in the system.

Six roles : employee, manager, hr_admin, it_admin, public_data_admin, and security_auditor. The first four drive the HR/profile flow and are the focus of this series. public_data_admin manages office and holiday data, and security_auditor has read-only access for compliance. The point that matters for security: each role sees a different slice of data from the same endpoints, controlled entirely by policy.

The tiering matters for security:

Tier 1 (gateway-facing): Profile Aggregator, Holiday Calendar, Office Directory. Reachable from the Istio ingress gateway.
Tier 2 (internal only): Employee Records, Device Inventory. Reachable exclusively from Profile Aggregator. No direct external path exists.
Tier 3 (data): PostgreSQL. Reachable only from specific service identities.

Holiday Calendar and Office Directory cannot reach Employee Records or Device Inventory regardless of what code runs inside them. Profile Aggregator also cannot connect directly to PostgreSQL. The mesh intercepts every connection attempt and checks whether the calling service is allowed to talk to the destination. If it's not on the allow-list, the connection is rejected before any application code even sees the request.

An attacker who compromises Holiday Calendar cannot reach Employee Records. The network topology itself is a security boundary, enforced by Istio AuthorizationPolicy.

The Threat Model

Every security layer exists because of a specific assumption about what can go wrong. These are the threats we're designing against:

An external user may send arbitrary headers, including internal trust headers, in their HTTP request.
Any single application pod may be compromised and act maliciously within whatever network access it has.
Application code may have bugs: missing authorization checks, overly broad queries, endpoints that skip validation.
Database queries may return more data than intended if the application doesn't filter correctly.
Signing keys and secrets must not live inside application pods, because any compromised pod leaks whatever it holds.

Each layer in the architecture addresses one or more of these. When we get to "What Happens When You Skip a Layer," you can map each failure directly back to this list.

The Layers

Five distinct security layers, each addressing an attack surface the others can't cover.

1. Transport Security (mTLS)

Mutual TLS encrypts data in transit and verifies the identity of both ends of the connection. Every pod in the mesh presents a SPIFFE identity certificate, and the receiving pod validates it before accepting any bytes.

This gives you encrypted communication and peer identity verification. A rogue service without a valid mesh certificate cannot establish a connection. But mTLS has a specific scope: it verifies that Service A is Service A. It does not determine whether Service A is allowed to call Service B with this particular payload for this particular user. That's a different problem entirely.

2. Identity Propagation (Signed Mesh Tokens)

Once a request enters the mesh and passes authentication at the gateway, the system needs to carry the user's verified identity through every downstream hop. A request from Alice hits Profile Aggregator, which then fans out to Employee Records and Device Inventory. Both downstream services need to know it's Alice, what role she holds, and that the request is legitimate.

This is done via a cryptographically signed JWT, the x-mesh-identity token, minted by Auth Service (auth-service) and signed by Vault Transit. The token carries who the user is (sub), what roles they hold (roles_csv), which services this token is valid for (aud), and who is acting on behalf of the user (act, for service-to-service delegation).

Any downstream service can verify this token using the public key from the Auth Service JWKS endpoint. No callback to Auth Service needed. But the token alone isn't sufficient. If projected headers like x-ms2-user arrive from outside the mesh without being stripped, a service reading those headers trusts attacker-controlled values. That's why header stripping at the gateway is non-negotiable, and why each destination sidecar also strips and re-projects these headers from the verified JWT before the request reaches the application.

3. Request Authorization (Policy Enforcement)

Knowing who someone is doesn't mean they're allowed to do what they're asking. Authorization in this architecture operates at four granularities, and each one catches what the layer above misses:

Network-level: Can this pod even reach that pod? Istio AuthorizationPolicy enforces which service identities are allowed to initiate connections.
Request-level: Is this user, with this role, allowed to call this endpoint? The ExtAuthz check at the gateway applies coarse route-level decisions.
Resource-level: Can this manager see this specific employee's record? Cerbos evaluates fine-grained, context-aware policies.
Field-level: Can this role see salary data, or should it be masked? Cerbos again, with policies that return which fields to include per role.

Why four layers instead of one? Because a single policy engine can't efficiently handle all of these. Network-level happens at the sidecar before any application code runs. Field-level requires knowing the actual resource attributes at query time. Different layers, different information available, different enforcement points.

4. Data Access Security (Row-Level Security)

The database is the last line of defense. PostgreSQL Row-Level Security policies enforce visibility at the engine level based on the current transaction context. Even if application code has a bug, a missing authorization check or a broken filter, RLS ensures the database itself drops rows the user shouldn't see.

Application developers make mistakes. A query like SELECT * FROM employees hitting production without a proper WHERE clause should still not leak the entire company directory. RLS makes that guarantee at the database layer, independent of whatever the application did or didn't check.

There is a trust boundary here: the application sets app.current_user_id on the transaction, and RLS uses that value. If the application sets it incorrectly, RLS enforces the wrong identity. In normal operation, the identity comes from sidecar-projected headers, so the application is a pass-through, not a source. But RLS does not protect against a fully compromised application process that sets arbitrary transaction context. RLS catches bugs (missing filters, broad queries). It does not stop a malicious process that controls its own DB session. The defense against that scenario is the layers above: AuthorizationPolicy limiting which pods can reach the database, scoped DB roles limiting which tables they can query, and Vault ensuring compromised pods can't escalate signing privileges.

5. Secrets Management (Vault)

Secrets in pods are the most common attack surface in Kubernetes. Environment variables, mounted files, ConfigMaps. Assume any compromised pod leaks whatever secrets it holds.

In this architecture, Vault serves a specific critical function: the Transit engine signs mesh identity tokens without ever exposing the private key to any service. Auth Service sends bytes to Vault, Vault signs them, returns the signature. The private key is never exposed to Auth Service or any application pod. If Auth Service is compromised, the attacker can ask Vault to sign things (until you revoke access), but they never get the key itself. They can't take it offline and mint tokens indefinitely.

Beyond signing, Vault is the natural home for database credentials, API keys, TLS certificates, and anything that would otherwise be scattered across pod specs. The pattern matters more than our specific usage: services request secrets through controlled APIs rather than holding them in memory at rest.

How the Layers Compose

These layers form a chain where each one depends on the guarantees of the ones around it. Here's what happens when a request enters the system.

An external request hits the Istio Gateway. TLS terminates. Before anything else, an EnvoyFilter strips all internal trust headers (x-mesh-identity, x-ms*-user, x-role-*), ensuring nothing spoofable survives from outside. This is the single most important security mechanism in the entire architecture. Without it, everything downstream can be bypassed by setting a header.

The gateway's ExtAuthz filter calls Auth Service. Auth Service validates either the session cookie (opaque, database-backed) or the Bearer token (validated against Keycloak's JWKS). If valid, it calls Vault Transit to sign a short-lived mesh identity token with a 5 minute TTL. This token comes back to Envoy as a response header and gets injected into the request.

Envoy routes the request to the target service. At the destination, a RequestAuthentication resource validates the x-mesh-identity token against the Auth Service JWKS endpoint. Istio's native outputClaimToHeaders projects the validated claims into service-specific headers (x-ms2-user, x-ms2-role). The service never touches the JWT directly. It just reads simple headers that the sidecar guarantees are legitimate.

AuthorizationPolicy at the destination checks both the mTLS peer identity (is the calling pod's service account allowed?) and the token claims (is the audience correct? is the delegation chain valid?). A token minted for Employee Records cannot be replayed against Device Inventory because the AuthorizationPolicy checks the aud claim. Both the source principal and the token audience must match for the request to proceed.

The service calls Cerbos for fine-grained authorization. Can this user, with this role, perform this action on this specific resource? Cerbos returns both an allow/deny decision and a list of visible fields.

Finally, the database query executes with RLS active. The application calls set_config('app.current_user_id', ...) on the transaction before querying, and PostgreSQL enforces row visibility at the engine level.

Header stripping, authentication, token minting, token validation with header projection, authorization policy, Cerbos, and RLS. Seven checkpoints. Each one reduces the blast radius of failures in the others.

What Happens When You Skip a Layer

Each layer exists because the others have blind spots.

Skip header stripping? An external attacker sets x-mesh-identity: <forged-token> in their HTTP request. The token won't pass signature validation at the destination sidecar. But the projected headers (x-ms2-user, x-ms2-role) are the real danger. If these arrive from outside without being stripped, and the service reads them directly, the attacker controls the identity. Header stripping ensures no internal trust header survives from outside the mesh, regardless of whether it's a signed token or a raw projected header.

Skip signed tokens (use raw headers instead)? Any compromised pod can forge headers. Without cryptographic signing, there's no way to distinguish a legitimate identity assertion from one injected by a rogue process.

Skip authorization policies? A valid user with a valid token can call any service and access any resource, regardless of whether they should. Authentication without authorization is an access-all-pass.

Skip RLS? A bug in your application's authorization logic, a missing check or a broken filter, becomes a data breach. The database has no independent enforcement, so it returns whatever the query asks for.

Skip Vault (use local keys)? If Auth Service is compromised, the attacker has the signing key. They can mint tokens for any user with any role, indefinitely, until you detect the breach and rotate. With Vault Transit, the key never leaves Vault. You revoke Auth Service access and rotate immediately.

Why a Service Mesh

The core idea: your application service has a simple contract. It reads x-ms2-user and x-ms2-role from request headers and does its business logic. It doesn't know about mTLS certificates, JWT validation, OIDC flows, or policy engines. The sidecar proxy handles all of that.

This matters for three reasons.

First, you can add services to the mesh without rewriting their auth logic. The security infrastructure wraps around them.
Second, when the security architecture evolves (new signing algorithm, new policy engine, new identity provider) application code doesn't change. Only mesh configuration does.
Third, application developers focus on domain logic while security is configured at the infrastructure layer. These concerns don't bleed into each other.

The complexity is real. You need Kubernetes knowledge, you need to understand Envoy, you need to debug sidecar injection and filter ordering. But the tradeoff is concrete: instead of implementing auth in every single service (and maybe getting it wrong in one of them), you configure it once at the infrastructure layer. The complexity is upfront and centralized instead of distributed across every service where it's invisible until it fails.

What's Next

The next post peels back the identity flow, from the gateway EnvoyFilter that strips headers, through the ExtAuthz check, to the signed mesh token landing on a downstream service. Real YAML, real Python, real attack scenarios.

Code:

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth

Safeguarding Your Systems: An Introduction to the Circuit Breaker Pattern

Fri, 28 Aug 2026 11:35:23 +0000

The circuit breaker pattern is a modern software design strategy used to detect failures and prevent an application from repeatedly executing an operation that is highly likely to fail. It acts as a protective wrapper around external calls, monitoring for errors and temporarily blocking traffic to a broken dependency once a threshold is crossed. By failing fast, it preserves your system's resources and gives the failing service room to recover.

Imagine the electrical panel in your home. If a faulty microwave suddenly draws a massive, dangerous surge of electricity, the physical circuit breaker "trips," immediately shutting off the flow of power to that outlet. It does this to prevent a catastrophic house fire. Once the breaker is tripped, electricity won't flow to that outlet again until you unplug the bad appliance and reset the switch. Without that simple breaker, a single broken device could destroy your entire house.

In software engineering, we apply this exact logic to distributed systems. Modern applications rely heavily on external third-party services, such as payment processors, SMS gateways, or database clusters. If one of these services crashes or slows to a crawl, a standard application will keep sending requests, waiting endlessly for responses that are never coming. This ties up precious system memory, CPU threads, and network sockets, eventually causing your own application to crash entirely. By implementing a software circuit breaker, engineers can instantly route traffic away from a failing dependency. This allows the application to remain functional, perhaps by showing an "our payment provider is offline" message rather than crashing the entire website.

This pattern operates using three distinct states: Closed, Open, and Half-Open. In the 'Closed' state, everything is working normally, and all requests pass through. When the error rate exceeds our tolerance, the breaker trips into the 'Open' state, instantly blocking all requests. After a set cool-down period, the breaker enters the 'Half-Open' state, allowing a small test batch of traffic through to see if the service has recovered. If the test succeeds, the breaker closes again; if it fails, it returns to the open state.

Here is a simple implementation of a Circuit Breaker in JavaScript:

class CircuitBreaker {
  constructor(requestFunction, failureThreshold, cooldownPeriod) {
    this.requestFunction = requestFunction;
    this.failureThreshold = failureThreshold;
    this.cooldownPeriod = cooldownPeriod;
    this.state = "CLOSED"; // CLOSED means normal operation
    this.failureCount = 0;
    this.nextAttemptTime = 0;
  }

  async execute(...args) {
    if (this.state === "OPEN") {
      if (Date.now() > this.nextAttemptTime) {
        this.state = "HALF-OPEN"; // Try once to see if it's fixed
      } else {
        throw new Error("Circuit is OPEN. Request blocked to prevent cascade.");
      }
    }

    try {
      const result = await this.requestFunction(...args);
      this.reset();
      return result;
    } catch (error) {
      this.handleFailure();
      throw error;
    }
  }

  handleFailure() {
    this.failureCount++;
    if (this.failureCount >= this.failureThreshold) {
      this.state = "OPEN";
      this.nextAttemptTime = Date.now() + this.cooldownPeriod;
    }
  }

  reset() {
    this.failureCount = 0;
    this.state = "CLOSED";
  }
}

This code tracks the status of an external call. If the system fails repeatedly, the breaker enters the OPEN state, immediately throwing an error for any subsequent attempts without wasting time on network calls. After the cooldownPeriod expires, it transitions to HALF-OPEN to test if the service has recovered, resetting itself to CLOSED on a successful call.

Embracing the circuit breaker pattern means accepting that failure is an inevitable reality in distributed environments. Instead of hoping that every external API stays online 100% of the time, defensive engineers design systems that can degrade gracefully. Implementing this pattern transforms fragile applications into resilient architectures capable of surviving localized outages without complete system collapse.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Don't Let One Broken Service Crash Your Whole App: The Circuit Breaker Pattern

Wed, 26 Aug 2026 01:16:39 +0000

What is the Circuit Breaker Pattern?

A Real-Life Analogy: The Household Fuse Box

Why It Matters in Daily Tech Operations

The Concept in Action

Here is a simple, lightweight implementation of a circuit breaker in JavaScript to show how this logic looks under the hood:

class SimpleCircuitBreaker {
  constructor(requestFunction, failureThreshold = 3, cooldownMs = 10000) {
    this.requestFunction = requestFunction;
    this.failureThreshold = failureThreshold;
    this.cooldownMs = cooldownMs;
    this.state = "CLOSED";
    this.failures = 0;
    this.nextAttemptTime = 0;
  }

  async execute() {
    if (this.state === "OPEN") {
      if (Date.now() > this.nextAttemptTime) {
        this.state = "HALF-OPEN";
      } else {
        throw new Error("Circuit is OPEN. Request blocked for safety.");
      }
    }

    try {
      const result = await this.requestFunction();
      this.reset();
      return result;
    } catch (error) {
      this.handleFailure();
      throw error;
    }
  }

  reset() {
    this.state = "CLOSED";
    this.failures = 0;
  }

  handleFailure() {
    this.failures++;
    if (this.failures >= this.failureThreshold) {
      this.state = "OPEN";
      this.nextAttemptTime = Date.now() + this.cooldownMs;
      console.log("Circuit breaker tripped! Blocking future requests.");
    }
  }
}

The Takeaway

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Continuous Delivery Office Hours Ep.4: Mono, micro, mesco

Mon, 27 Jul 2026 12:09:46 +0000

After tackling the question of which branching strategy is best, we turn our focus to architecture in this Continuous Delivery Office Hours episode. In particular, we discuss how big your building blocks should be and whether you should opt for a majestic monolith, granular microservices, or something in between.

Traditional monoliths are often described as a big ball of mud or spaghetti code due to the complicated mess of dependencies. When all the code is co-located, it's common to find a tangled web of paths, with interfaces and facades skipped in favor of direct access to low-level functions and classes. This kind of code is hard to maintain.

Majestic monoliths are well-structured and loosely coupled, with carefully designed dependencies that are sometimes enforced by architectural tools. By making things modular, you get the benefits often associated with microservices, but without the difficulty of finding the code or the runtime complexity of distributed systems.

Microservices go a step further by physically separating parts of the system into independently deployable units. In addition to the architectural benefits, you can get faster build and test run times because you only build and test each service when it changes, which involves much less code. The trade-off is that your codebase becomes more complex, and you have to deal with versioning and issue pinpointing across all the deployed services.

So, how do you choose between these approaches?

Watch the episode

You can watch the episode below, or read on to find some of the key discussion points.

Watch Continuous Delivery Office Hours Ep.4

Architectural decision factors

Many factors influence the decision between monoliths, microservices, and hybrid approaches. One of the most crucial is how your teams are structured and how much autonomy they need. There are different ways to design team structures, like Team Topologies, that make your teams and architecture part of the same conversation.

This is consistent with the inverse Conway maneuver, in which you structure your teams to match your target architecture. This works because Conway's Law states that organizations design systems that mirror their communication structures. Working at the team level is more effective than creating an architectural diagram, as the structure will naturally yield the design you want.

Having a mismatch between the number of teams and the number of components can add complexity, either because too many teams are working on a single module, or you have created so many modules that there aren't enough teams to own all the maintenance tasks associated with them. A sensible default is to match the number of teams to the number of components, ensuring sufficient autonomy for each team to work independently without overburdening them with excessive complexity.

The matching of service and team design sits between monolith and microservices, and might be described as "mescoservices" to highlight the balance they strike between the macro level of monoliths and the fine-grained level of microservices.

Another factor that can influence your decision is your deployment model. A managed software-as-a-service offering can use microservices without imposing complexity on its customers. In contrast, a self-hosted software offering may benefit from easier installation, troubleshooting, and upgrades if it uses a monolithic architecture.

Common microservice failure modes

There are several common failure modes to watch out for when moving to a microservice architecture.

Distributed monoliths look like microservices, except they have dependencies that mean you have to deploy services in a specific order. If you can't deploy independently, you don't have microservices. Similarly, you often find microservices at the application layer, but all sharing the same database. This prevents independent scaling of high-load services and can cause services to access data directly rather than through the appropriate service.

When you deploy large numbers of microservices, you may need to introduce rules about which services can communicate. Allowing any service to call any other service means managing a complex web of dependencies, and it starts to look a lot like the proverbial big ball of mud. This is often the result of splitting a monolith into microservices, as the old design tends to persist.

Microservices should bring a high level of team independence, allowing teams to make decisions about their own services, persistence, and deployment schedules.

Pulling microservices together

To make microservices successful, you need to establish data ownership and decide what level of data duplication is acceptable to support the business goals. For example, there's a temptation to store a single address for a customer, but it may be a better idea to store the address immutably against events, like orders. Even if the customer moves house, the address for a past order remains the same.

Domain-driven design provides a method and a language for making decisions across service boundaries. A principal engineer or architect would usually set standards for service boundaries, contracts, and versioning to help keep services loosely coupled and independently deployable.

Fitness test

The best test of your service-based architecture is independent deployability. If each team can deploy its services without cross-team coordination, then your architecture is working. For majestic monoliths, the test is similar; teams should be able to work independently on modules they own. While deployments will be coupled through a single build and deployment pipeline, they should be able to make and commit changes independently.

Happy deployments!

Continuous Delivery Office Hours is a series of conversations about software delivery, with Tony Kelly, Bob Walker, and Steve Fenton.

You can find more episodes on YouTube, Apple Podcasts, and Pocket Casts.

Mono, Micro, Mesco

Wed, 15 Jul 2026 09:00:00 +0000

Tony Kelly hosts our latest episode of Continuous Delivery Office Hours, with Bob Walker and Steve Fenton.

Your system's architecture can support team autonomy, independent deployability, and independent evolution of components, or it can be a sticky mess. This episode looks at whether choosing a monolith or microservices fundamentally changes the crucial properties of your architecture.

The many factors influencing architecture
How team design is part of architecture
How to choose between monoliths and microservices
The true measure of successful architecture

Designing Healthcare Interoperability with HL7 FHIR: Lessons from Venezuela

Mon, 01 Jun 2026 10:23:52 +0000

How to move from isolated hospital systems to event‑driven, FHIR‑based architectures in a context with poor connectivity and legacy vendors.

In many private clinics in Venezuela, “digital transformation” has meant installing a Hospital Information System (HIS) and maybe an Electronic Health Record (EHR). The problem is that most of these systems do not expose standard APIs, or do it in a very limited way.

As an engineer, the real challenge is not “going paperless”, but making systems talk to each other: getting lab, pharmacy, admissions, telemedicine and billing to exchange data without adding more technical debt.

In this post I’ll share a practical architectural approach, from a developer’s perspective, to move a typical clinic environment from ad‑hoc integrations to an HL7 FHIR + event‑driven architecture.

The starting point: interoperability level 1–2
In practice, most clinics I work with operate like this:

Manual CSV or PDF exports.

Occasional point‑to‑point HL7 v2 interfaces (if you’re lucky).

No formal API documentation.

Human “copy‑paste” processes between systems.

In HIMSS terms, that’s level 1–2 interoperability. A realistic technical goal is to bring critical systems up to level 3 (semantic), where data means the same thing everywhere.

Why FHIR fits this environment
FHIR brings several properties that are extremely valuable in this context:

API‑first. Everything revolves around HTTP resources.

Well‑known structures. JSON with clear fields for Patient, Observation, Encounter, Medication, etc.

Mature implementations. HAPI FHIR in Java, for example, lets you spin up a server quickly.

A minimal viable pattern looks like this:

Deploy a FHIR server (e.g., HAPI FHIR) as the central point.

Expose a small, focused subset of resources for the first use case: Patient, Encounter, Observation, Medication, DiagnosticReport.

Build one adapter per legacy system that:

Consumes events or exports from the source system.

Transforms them into the corresponding FHIR resource.

Publishes them to the FHIR server via its REST API.

Integration layer: avoiding point‑to‑point hell
Without an integration layer, every integration is a fixed pair of systems. With ten systems, you have forty‑five possible pairs.

Recommended architecture:

An event bus (Apache Kafka for high‑volume scenarios, RabbitMQ for more modest loads).

Microservices (Java + Spring Boot, for example) responsible for:

Subscribing to domain‑specific topics (lab, pharmacy, admissions, etc.).

Transforming messages to the FHIR model.

Publishing to the FHIR server.

In parallel, an integration engine like Mirth Connect can simplify HL7 v2 → FHIR handling.

This shifts the problem from “every system talks to every other system” to “every system talks to the bus”, and “the bus talks FHIR to the outside world”.

Dealing with unreliable connectivity
In Venezuela, you simply cannot assume stable connectivity, not even within the same city. From a software architecture standpoint, this means:

Implementing the Outbox Pattern in source systems.

Using local queues to buffer events when the connection is down, and flush them when it comes back.

Designing idempotent consumers on the event bus side to avoid duplicates and inconsistencies.

From this perspective, the FHIR server becomes the consolidated source of truth, but local systems must still be able to operate with a subset of data while offline.

Security by design, not as an afterthought
Two concrete recommendations:

For client applications (web, mobile, clinical front‑end): SMART on FHIR on top of OAuth 2.0, with fine‑grained scopes per resource.

For server‑to‑server integrations: OAuth 2.0 with trusted clients, managed certificates and TLS 1.3 everywhere.

Trying to “bolt on” security at the end is a terrible idea in a domain where data is, by definition, highly sensitive.

Conclusion and next steps
If you’re in a similar environment (private clinic in LATAM with multiple legacy systems) and want to start working with FHIR, my suggestion is:

Pick one small but critical use case (for instance, integrating lab results into the EHR).

Stand up a FHIR server with a minimal resource model.

Build a single adapter for one source system.

Add the event bus and security from day one, even if you only have a couple of systems.

From there, adding new systems becomes a repeatable pattern, not a heroic project every single time.

Call to action for dev.to
If you’re working on healthcare interoperability in LATAM and want to compare architectures or trade war stories, I’d be happy to connect. You can find more details and case studies at codebymelendez.com or reach out to me on LinkedIn.

Building a Real-Time Network Anomaly Detector with PyTorch and Go

Tue, 26 May 2026 07:41:30 +0000

description: "4-service microservices architecture for live network intrusion detection: hybrid ML model, sub-second alerts, and an 11-page React operator dashboard."
tags: pytorch, go, machinelearning, microservices
series: engineering-deep-dives
cover_image:

canonical_url: https://satyasaikilaru.vercel.app/blog/network-anomaly-detector

TL;DR — I built a full-stack intrusion detection platform: a hybrid ML model (Focal Loss classifier + autoencoder for zero-days), a 4-service Go/Python backend, and an 11-page React dashboard. R2L recall doubled (14% → 29%), zero-day detection works via reconstruction error, and the whole pipeline runs in <2ms per prediction.

The Problem No One Talks About

Every network security tutorial shows you how to train a classifier on NSL-KDD and call it done. But classifiers have a fatal blind spot: they can only detect attacks they've trained on.

The NSL-KDD test set alone contains 17 attack subtypes that don't appear in training. In production, this ratio is infinitely worse — every new CVE, every new botnet, every novel C2 protocol is invisible to a classifier until you retrain.

I wanted to build something that catches both: known attacks with high precision and novel threats it's never seen before.

Architecture at a Glance

┌─────────────────────────┐
│   React Dashboard :5173 │  11 pages, ECharts, AG Grid, Firebase Auth
│   Apollo Client (GQL+WS)│
└───────────┬─────────────┘
            │
┌───────────▼─────────────┐
│  GraphQL Gateway :4000  │  gqlgen, Firebase Admin SDK, SQLite
│  Auth · Persistence     │
└───────────┬─────────────┘
            │ HTTP proxy
┌───────────▼─────────────┐
│  Go API Backend :8080   │  REST, WebSocket broadcast, Simulation
│  Packet capture control │
└───────────┬─────────────┘
            │ HTTP /api/predict
┌───────────▼─────────────┐
│  FastAPI ML Server :8000│  PyTorch → ONNX inference, <2ms/sample
└─────────────────────────┘

Why 4 services? The ML server needs high memory and bursty CPU. The Go API needs low memory and high concurrency. Scaling them as one binary wastes money. Each service deploys and scales independently.

The Hybrid ML Model

Problem 1: Rare attacks are invisible to standard loss functions

NSL-KDD class distribution:

Class	Training %	What it is
Normal	53%	Legitimate traffic
DoS	36%	Denial of service
Probe	9.4%	Port scanning, reconnaissance
R2L	0.8%	Remote-to-Local exploits
U2R	0.04%	User-to-Root privilege escalation

Standard CrossEntropy loss treats all samples equally — so the model essentially ignores R2L and U2R. Why learn to detect 0.04% of the data when you can boost accuracy by getting the 53% right?

Fix: Focal Loss.

FL(p_t) = -α_t (1 - p_t)^γ log(p_t)

Focal Loss down-weights easy examples (the Normal traffic the model already handles) and forces it to focus on hard, rare samples. The γ parameter controls how aggressively.

Result:

Metric	CrossEntropy (V1)	Focal Loss (V2)	Change
Probe Recall	68%	88%	+20%
R2L Recall	14%	29%	+15% (2x)
Macro F1	0.558	0.589	+5.5%

Problem 2: Zero-days are invisible to any classifier

A classifier is a lookup table — "I've seen this pattern before, it's DoS." For the 17 novel attack subtypes in the test set (and the infinite unknown attacks in the wild), we need something fundamentally different.

Fix: Autoencoder trained only on Normal traffic.

The autoencoder learns to compress and reconstruct "what normal looks like." At inference:

Low reconstruction error → looks normal → pass
High reconstruction error → looks like nothing I've ever seen → anomaly flag

This is zero-day detection by definition: it catches anything that deviates from normal, regardless of whether it matches any known signature.

The 3-Vote Consensus

Two detectors means disagreements. Policy:

Classifier	Autoencoder	Action
Attack	High error	High confidence — auto-block above threshold
Normal	High error	Possible zero-day — escalate to operator
Attack	Low error	Known pattern — treat at face value
Normal	Low error	Green — pass

The second row is the gold: a connection that looks normal per-feature but whose overall shape is anomalous. That's where zero-days hide.

The Dashboard — 11 Pages

Not just a prediction endpoint. Operators need context:

Page	Why it exists
World Map	Animated attack routes with GeoIP — "where is this coming from?"
Traffic Feed	AG Grid, bulk select, CSV export — "show me everything"
Incidents	AI threat summary + timeline by class — "what's happening now?"
Analytics	Time-series, attack distribution — "what's the trend?"
Network Graph	Force-directed IP topology — "who's talking to whom?"
Model Compare	Radar chart, confusion matrix — "how good are we?"
SOAR	Auto-block on confidence thresholds — "respond automatically"
Settings	Profile, theme, webhooks, Slack — "configure everything"
System Status	Service health, latency sparklines — "is the platform healthy?"

Firebase Auth handles Google Sign-In + Email/Password. First user = admin in SQLite; subsequent = viewers. RBAC enforced at the gateway, not just the UI.

Performance

Class	Precision	Recall	F1
Normal	91.0%	88.9%	90.0%
DoS	95.8%	85.6%	90.4%
Probe	60.4%	88.3%	71.7%
R2L	53.9%	29.4%	38.0%
U2R	2.2%	41.8%	4.2%

Overall: 80.0% accuracy · 0.589 Macro F1

R2L/U2R are still the weakest — genuinely hard with <1% of training data. The autoencoder compensates by catching them as distribution anomalies even when the classifier misses.

What I'd Do Differently

Add a χ² frequency-distribution detector — the same math that cracks a Caesar cipher (compare observed letter frequencies to English) applied to network feature distributions. Catches slow horizontal scans that per-packet models miss.
Per-host behavioral baselines — a global Normal model doesn't notice "this IP was quiet for 6 months and just started scanning port 445." A per-host baseline does.
Adversarial training — FGSM/PGD perturbations during training to harden against evasion. If someone discovers your model, they'll craft packets to slip past it.

Stack

Go · Python · PyTorch · ONNX · FastAPI · React 19 · Vite · Tailwind CSS · ECharts · AG Grid · Apollo Client · GraphQL (gqlgen) · Firebase Auth · SQLite · scapy · GeoIP2

Portfolio · GitHub

If you're building something similar or want to discuss the architecture — connect with me on LinkedIn.

The Ecosystem Conductor: Orchestrating Market Orders with Kafka, RabbitMQ and Spring Boot

Wed, 13 May 2026 00:55:31 +0000

Hey folks!

We've reached one of the most important posts in the My Broker B3 series. Up until now we built each piece of the ecosystem separately — the price synchronizer, the matching engine, the financial custody, the asset catalog. Now it's time to build the service that connects all of it: the Broker Order API.

This is the conductor of the ecosystem. When a user wants to buy or sell a stock, this is where everything begins — and where the results come back to.

What is the Order API?

The trading-broker-order is the entry point for the broker's transactional flow. It has a clear responsibility: orchestrate the complete lifecycle of an order, from the user's intent to the final execution confirmation.

To do this, it communicates with four other ecosystem services:

Asset API (REST/Feign) — validates whether the ticker exists and is active
Wallet API (REST/Feign) — validates whether the user has sufficient balance
B3 Matching Engine (RabbitMQ) — sends the order for execution
Broker Wallet (Kafka) — notifies the ecosystem about each status change

️ The Complete Order Flow

Before diving into the code, here's the end-to-end flow:

[User]
    │
    │  POST /api/v1/orders
    │  { userId, ticker, quantity, price, side: "BUY" }
    ▼
[trading-broker-order]
    │
    ├─ 1. Validate ticker  → GET trading-asset/assets/{ticker} (Feign/REST)
    │
    ├─ 2. Validate balance → GET trading-wallet/summary (Feign/REST)
    │
    ├─ 3. Persist order → MySQL (status: PENDING)
    │
    ├─ 4. Send to B3 → RabbitMQ (mq-broker-to-b3)
    │
    ├─ 5. Notify ecosystem → Kafka (order-events-v1, status: PENDING)
    │
    └─ Returns created order → HTTP 201

         ─ ─ ─ (B3 processes the order) ─ ─ ─

[b3-matching-engine]
    │
    │  RabbitMQ (mq-b3-to-broker)
    │  { orderId, status: FILLED/REJECTED, executedPrice }
    ▼
[trading-broker-order]
    │
    ├─ 6. Update status → MySQL (FILLED or REJECTED)
    │
    └─ 7. Notify ecosystem → Kafka (order-events-v1, final status)
              │
              ├─ trading-broker-wallet consumes → blocks/settles/refunds balance
              └─ other future consumers

Two distinct communication channels, two Kafka publish moments, and the wallet reacting to each state change. Let's see how each part was implemented.

️ Tech Stack

Technology	Usage
Java 21 + Spring Boot 3.3.5	Service core
MySQL + Flyway	Order persistence
Spring Kafka	Lifecycle event publishing
Spring AMQP	Integration with Matching Engine via RabbitMQ
Spring Cloud OpenFeign	REST calls to Asset API and Wallet API
SpringDoc OpenAPI	Swagger UI documentation

️ Implementation Pillars

1. The Domain Model — Order

The Order entity represents the state of an order at any point in its lifecycle:

@Entity
@Table(name = "orders")
public class Order {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    private String userId;
    private String ticker;

    @Column(nullable = false, precision = 19, scale = 4)
    private BigDecimal quantity;

    @Column(nullable = false, precision = 19, scale = 4)
    private BigDecimal price;

    // Null while PENDING — only set after B3 feedback
    @Column(precision = 19, scale = 4)
    private BigDecimal executedPrice;

    @Enumerated(EnumType.STRING)
    private OrderSide side;   // BUY | SELL

    @Enumerated(EnumType.STRING)
    private OrderStatus status; // PENDING | FILLED | REJECTED | CANCELLED

    @CreationTimestamp
    private LocalDateTime createdAt;

    @UpdateTimestamp
    private LocalDateTime updatedAt;
}

An important design detail: executedPrice is intentionally nullable. When the order is created, we don't yet know the execution price — that only comes as a response from the Matching Engine. Forcing nullable = false here would be a domain modeling error.

The entity also carries the mapToEvent() method that serializes the current state into a Kafka event:

public OrderEventDTO mapToEvent() {
    return OrderEventDTO.builder()
            .orderId(this.getId())
            .userId(this.getUserId())
            .ticker(this.getTicker())
            .quantity(this.getQuantity())
            .price(this.getPrice())
            .executedPrice(this.getExecutedPrice())
            .side(this.getSide().name())
            .status(this.getStatus().name())
            .eventTimestamp(LocalDateTime.now())
            .build();
}

2. Ticker Validation — Asset Feign Client

Before validating balance or persisting anything, we need to ensure the requested ticker actually exists and is available for trading. A user submitting an order for XPTO999 should get a clear error immediately — not a silent rejection from B3 two hops later.

@FeignClient(name = "trading-broker-asset", url = "${app.services.asset-url}")
public interface AssetClient {

    @GetMapping("/api/v1/assets/{ticker}")
    AssetDTO getByTicker(@PathVariable("ticker") String ticker);
}

The validation in OrderService:

private void validateTicker(String ticker) {
    try {
        var asset = assetClient.getByTicker(ticker.toUpperCase());
        if (!"ACTIVE".equalsIgnoreCase(asset.status())) {
            throw new RuntimeException("Asset is not available for trading: " + ticker);
        }
        log.info("Ticker {} validated — status: {}", ticker, asset.status());
    } catch (FeignException.NotFound e) {
        throw new RuntimeException("Asset not found or inactive: " + ticker);
    } catch (FeignException e) {
        log.error("Asset service unavailable during ticker validation: {}", e.getMessage());
        throw new RuntimeException("Asset service unavailable. Please try again later.");
    }
}

Why catch FeignException separately from FeignException.NotFound? A 404 from the asset service means the ticker doesn't exist — a business error that should fail fast. Any other FeignException (connection refused, timeout, 5xx) means the asset service is unavailable — a different kind of failure that warrants a different message to the user.

3. Balance Validation — Wallet Feign Client

For buy orders (BUY), we need to ensure the user has available balance. We use Spring Cloud OpenFeign to query the Wallet API declaratively:

@FeignClient(name = "trading-broker-wallet", url = "${app.services.wallet-url}")
public interface WalletClient {

    @GetMapping("/api/v1/wallet/{userId}/summary")
    WalletSummaryDTO getWalletSummary(@PathVariable String userId);
}

The validation happens after ticker validation:

private void validateBalance(OrderRequestDTO request) {
    BigDecimal totalOrderValue = request.getPrice().multiply(request.getQuantity());
    var walletSummary = walletClient.getWalletSummary(request.getUserId());

    if (walletSummary.getAvailableBalance().compareTo(totalOrderValue) < 0) {
        throw new RuntimeException("Insufficient balance. Available: "
                + walletSummary.getAvailableBalance()
                + " | Required: " + totalOrderValue);
    }
}

Why Feign and not RestTemplate? OpenFeign eliminates HTTP boilerplate — the interface is sufficient. The URL is injected via application.yaml, making environment changes straightforward without recompiling.

4. The Creation Flow — OrderService

The placeOrder() is the heart of the service. It orchestrates five steps in sequence within a transaction:

@Transactional
public Order placeOrder(OrderRequestDTO request) {
    log.info("Processing new {} order for user: {}", request.getSide(), request.getUserId());

    // 1. Validate ticker exists and is ACTIVE in asset catalog
    validateTicker(request.getTicker());

    // 2. Validate balance for BUY orders via Wallet API
    if (request.getSide() == OrderSide.BUY) {
        validateBalance(request);
    }

    // 3. Persist order with PENDING status
    Order order = Order.builder()
            .userId(request.getUserId())
            .ticker(request.getTicker().toUpperCase())
            .quantity(request.getQuantity())
            .price(request.getPrice())
            .side(request.getSide())
            .status(OrderStatus.PENDING)
            .build();

    order = orderRepository.save(order);

    // 4. Send to B3 via RabbitMQ
    b3MessageProducer.sendToB3(mapToB3Message(order));

    // 5. Publish PENDING event to Kafka for wallet and other consumers
    orderEventProducer.sendOrderEvent(order.mapToEvent());

    return order;
}

Why validate ticker before balance? It's cheaper to fail fast on a non-existent ticker (one network call) than to validate balance first and then discover the ticker is invalid. Order of validations matters for UX and for minimizing unnecessary load on downstream services.

Design decision: The Kafka event is published with PENDING status right after creation. This allows trading-broker-wallet to immediately block the balance, without waiting for B3's result. This is the essence of eventual consistency — each service reacts to the event in its own time.

5. B3 Integration — RabbitMQ

Communication with the Matching Engine uses RabbitMQ for a specific reason: delivery guarantee. Even if the Matching Engine is temporarily unavailable, the message stays in the queue and is processed when it comes back.

Infrastructure configuration

All RabbitMQ infrastructure is declared as Spring beans — ensuring queues, exchanges and bindings exist before any message flows:

@Bean
public DirectExchange brokerExchange() {
    return new DirectExchange(exchangeName);
}

@Bean
public Queue orderOutQueue() {
    return QueueBuilder.durable(queueIn)
            .withArgument("x-dead-letter-exchange", "dlx.broker.orders")
            .withArgument("x-dead-letter-routing-key", "dlq-broker-to-b3")
            .build();
}

@Bean
public Binding orderOutBinding() {
    return BindingBuilder.bind(orderOutQueue())
            .to(brokerExchange())
            .with(routingKey);
}

The dead letter withArgument ensures messages that fail are redirected to the DLQ instead of simply being discarded.

Producer — sending to B3

public void sendToB3(B3OrderRequestDTO message) {
    log.info("Sending order {} to B3 via exchange: {} | routing-key: {}",
            message.getOrderId(), exchange, routingKey);
    rabbitTemplate.convertAndSend(exchange, routingKey, message);
}

Important decision: the message is sent to the exchange with the routing key, not directly to the queue. This respects the RabbitMQ routing model and allows adding new consumers without changing the producer.

Consumer — receiving B3 feedback

@RabbitListener(queues = "${app.rabbitmq.queue-out}")
public void receiveFeedback(B3OrderResponseDTO response) {
    log.info("Feedback received from B3 for order: {} | Status: {}",
            response.getOrderId(), response.getStatus());
    try {
        orderStatusService.updateOrderStatus(response);
    } catch (Exception e) {
        log.error("Failed to update status for order {}: {}",
                response.getOrderId(), e.getMessage(), e);
        throw e; // RabbitMQ applies retry/DLQ policy
    }
}

The throw e is intentional: rethrowing the exception allows RabbitMQ to apply its retry policy instead of silently discarding the message.

6. Status Update — OrderStatusService

When B3 feedback arrives, OrderStatusService updates the order state and publishes the second Kafka event:

@Transactional
public void updateOrderStatus(B3OrderResponseDTO response) {
    Order order = orderRepository.findById(Long.valueOf(response.getOrderId()))
            .orElseThrow(() -> new RuntimeException("Order not found: " + response.getOrderId()));

    order.setStatus(OrderStatus.valueOf(response.getStatus()));
    order.setExecutedPrice(response.getExecutedPrice());

    orderRepository.save(order);

    // Notify ecosystem with final status — wallet settles or refunds
    orderEventProducer.sendOrderEvent(order.mapToEvent());
}

This second Kafka event closes the loop: trading-broker-wallet consumes it and executes the settlement (FILLED) or refund (REJECTED) of the balance blocked in the previous step.

REST API

Method	Endpoint	Description
POST	`/api/v1/orders`	Place a new buy or sell order
GET	`/api/v1/orders/{id}`	Get order details by ID
GET	`/api/v1/orders/user/{userId}`	List all orders for a user

Swagger UI: http://localhost:8088/swagger-ui.html

✅ Validating the Execution

With the application running locally:

✅ Flyway validated the orders table schema
✅ Tomcat started on port 8088
✅ RabbitMQ connected with exchange, queues and DLQ declared
✅ Swagger UI with 3 documented endpoints
✅ AssetClient validating ticker before any persistence
✅ WalletClient validating balance for BUY orders

The Complete Ecosystem

With trading-broker-order ready, we have the first end-to-end flow working:

[User] → [Order API] → [Asset API]  (ticker validation)
                    → [Wallet API]  (balance validation)
                    → [B3 Matching Engine] (execution)
                    → [Kafka] → [Wallet API] (custody)

From the user's click to the updated wallet balance, through ticker and balance validation, RabbitMQ routing, price matching against Redis, and eventual consistency via Kafka.

Got any questions about the design decisions or the inter-service orchestration? Drop them in the comments!

About the Series

⬅️ Previous Post: Financial Custody: Managing Balance and Portfolio with Eventual Consistency

Series Index: Series Roadmap

Links:

Financial Custody: Managing Balance and Portfolio with Eventual Consistency

Tue, 12 May 2026 20:43:57 +0000

Hey folks!

Continuing the My Broker B3 series, we've reached one of the most business-logic-rich services in the ecosystem: the Broker Wallet API.

In previous posts we built the B3 infrastructure (market price sync and matching engine). Now we enter the financial core of the broker. This service is the guardian of the investor's money and assets — it needs to be correct above all else.

What is the Wallet API?

The trading-broker-wallet is the financial custody service of the ecosystem. Its responsibility is to react to order lifecycle events and ensure the user's money and assets are managed correctly at each step.

It doesn't receive direct buy commands — it listens to Kafka events and acts according to each order's status:

Kafka: order-events-v1
         │
    ┌────┴────────────────────────┐
    │                             │
 PENDING                       FILLED / REJECTED
    │                             │
 reserveBalance()          settleOrder() / refundBalance()
    │                             │
 Blocks balance            Settles or refunds

Why is the Wallet so Critical?

Imagine the following scenario without proper custody:

User has R$ 1,000 available
Places a buy order for R$ 800
Before the order is processed, places another order for R$ 500
Both orders are executed → user ends up with negative balance

The wallet solves this with the concept of blocked balance: the moment an order enters as PENDING, the value is immediately reserved. The available balance is always balance - blockedBalance.

️ Tech Stack

Technology	Usage
Java 21 + Spring Boot 3.5.11	Service core
MySQL + Flyway	Operational persistence and versioning
Spring Kafka	Order event consumption
Spring Data Redis	Real-time prices for portfolio valuation
SpringDoc OpenAPI	Swagger UI documentation

️ The Data Model

Before diving into the logic, it's important to understand the three entities that support the service:

Account
├── userId (UNIQUE)
├── balance          ← total balance (includes blocked amount)
├── blockedBalance   ← amount reserved for PENDING orders
└── currency

Account (1) ──▶ (N) Position
                     ├── ticker
                     ├── quantity
                     └── averagePrice

WalletTransaction (audit log)
├── orderId
├── userId
├── transactionType  ← RESERVE | SETTLEMENT | REFUND
└── amount

The blockedBalance is the central piece of custody. The balance never goes below zero — only blockedBalance goes up and down as orders progress through their lifecycle.

The Order Lifecycle in the Wallet

Step 1 — PENDING: Reserving the Balance

When an order is created, it arrives as PENDING. The wallet must immediately block the maximum amount that could be debited (quantity × limit price):

@Transactional
public void reserveBalance(OrderEventDTO event) {
    BigDecimal amountToReserve = event.getPrice().multiply(event.getQuantity());

    Account account = accountRepository.findByUserId(event.getUserId())
            .orElseThrow(() -> new RuntimeException("Account not found for user: " + event.getUserId()));

    // Available = balance - blockedBalance
    BigDecimal availableBalance = account.getAvailableBalance();

    if (availableBalance.compareTo(amountToReserve) < 0) {
        throw new RuntimeException("Insufficient balance to reserve funds.");
    }

    // Only blockedBalance increases — total balance stays intact
    account.setBlockedBalance(account.getBlockedBalance().add(amountToReserve));
    accountRepository.save(account);

    // Audit record
    walletTransactionRepository.save(WalletTransaction.builder()
            .userId(event.getUserId())
            .orderId(event.getOrderId())
            .transactionType(TransactionType.RESERVE)
            .amount(amountToReserve)
            .build());
}

Important point: the total balance doesn't change at this step. Only blockedBalance increases. The money is still in the account — it's just reserved.

Step 2a — FILLED: Settling the Order

The order was executed by the Matching Engine. Now the money actually leaves. But here's a subtlety: the executed price may differ from the reserved price.

Example: user placed a buy order with a limit of R$ 30.00, but the market executed at R$ 29.50. The reserved amount was R$ 300 (10 shares × R$ 30), but the actual executed amount was R$ 295 (10 × R$ 29.50). The R$ 5.00 difference automatically becomes available again.

@Transactional
public void settleOrder(OrderEventDTO event) {
    BigDecimal totalReserved = event.getPrice().multiply(event.getQuantity());
    BigDecimal totalExecuted = event.getExecutedPrice().multiply(event.getQuantity());

    Account account = accountRepository.findByUserId(event.getUserId())
            .orElseThrow(() -> new RuntimeException("Account not found"));

    // Release the initial block
    BigDecimal newBlockedBalance = account.getBlockedBalance().subtract(totalReserved);
    account.setBlockedBalance(newBlockedBalance.compareTo(BigDecimal.ZERO) < 0
            ? BigDecimal.ZERO
            : newBlockedBalance);

    // Debit the actual executed amount from total balance
    account.setBalance(account.getBalance().subtract(totalExecuted));
    accountRepository.save(account);

    // Update asset position
    updatePosition(event, account);
}

Step 2b — REJECTED: Refunding the Balance

The order was rejected (price outside market range, ticker not found in Redis, etc.). The blocked money becomes available again:

@Transactional
public void refundBalance(OrderEventDTO event) {
    BigDecimal amountToRefund = event.getPrice().multiply(event.getQuantity());

    Account account = accountRepository.findByUserId(event.getUserId())
            .orElseThrow(() -> new RuntimeException("Account not found for refund"));

    // Only subtract from blockedBalance — total balance was never touched
    if (account.getBlockedBalance().compareTo(amountToRefund) >= 0) {
        account.setBlockedBalance(account.getBlockedBalance().subtract(amountToRefund));
    } else {
        log.warn("Refund amount exceeds blocked balance for order {}", event.getOrderId());
        account.setBlockedBalance(BigDecimal.ZERO);
    }

    accountRepository.save(account);
}

Weighted Average Price Calculation

When a purchase is settled, we need to update the user's position in that asset. The weighted average price ensures that purchases made at different times are correctly reflected:

New Average Price = (Current Cost + New Cost) / (Current Qty + New Qty)

private void updatePosition(OrderEventDTO event, Account account) {
    Position position = positionRepository
            .findByAccountIdAndTicker(account.getId(), event.getTicker())
            .orElse(Position.builder()
                    .account(account)
                    .ticker(event.getTicker())
                    .quantity(BigDecimal.ZERO)
                    .averagePrice(BigDecimal.ZERO)
                    .build());

    boolean isSell = "SELL".equalsIgnoreCase(event.getSide());

    if (isSell) {
        BigDecimal newQuantity = position.getQuantity().subtract(event.getQuantity());
        if (newQuantity.compareTo(BigDecimal.ZERO) <= 0) {
            positionRepository.delete(position);
            return;
        }
        position.setQuantity(newQuantity);
        // Average price stays the same on sell
    } else {
        // BUY: recalculate weighted average price
        BigDecimal currentCost = position.getAveragePrice().multiply(position.getQuantity());
        BigDecimal newCost = event.getExecutedPrice().multiply(event.getQuantity());
        BigDecimal totalQuantity = position.getQuantity().add(event.getQuantity());
        BigDecimal newAveragePrice = currentCost.add(newCost)
                .divide(totalQuantity, 4, RoundingMode.HALF_UP);

        position.setQuantity(totalQuantity);
        position.setAveragePrice(newAveragePrice);
    }

    positionRepository.save(position);
}

Practical example:

Purchase 1: 10 PETR4 shares at R$ 30.00 → average price = R$ 30.00
Purchase 2: 5 PETR4 shares at R$ 33.00 → average price = (300 + 165) / 15 = R$ 31.00

Critical Bugs Fixed

During the code review, I identified issues that would cause silent failures in production:

1. Guaranteed NPE on first deposit
New account created without initializing blockedBalance. The first call to getAvailableBalance() would throw NullPointerException.

// ❌ Before
Account.builder()
    .balance(BigDecimal.ZERO)
    // blockedBalance missing → NPE

// ✅ After
Account.builder()
    .balance(BigDecimal.ZERO)
    .blockedBalance(BigDecimal.ZERO)

2. Withdraw validated total balance, not available balance
Allowed withdrawing money that was blocked in pending orders:

// ❌ Before — allows withdrawing reserved funds
if (account.getBalance().compareTo(amount) < 0)

// ✅ After — validates only what's available
if (account.getAvailableBalance().compareTo(amount) < 0)

3. SELL increased position instead of decreasing it
The updatePosition() method always added quantity, completely ignoring the side field. Selling 10 shares would add 10 to the position.

4. Kafka consumer silently swallowed exceptions
Errors were logged but the offset was committed — the message was discarded without retry. Now the consumer rethrows the exception so Kafka can apply its retry policy correctly.

Idempotency

To prevent duplicates in case of Kafka reprocessing, we added a database constraint:

-- V4 migration
ALTER TABLE wallet_transactions
    ADD CONSTRAINT uq_order_transaction UNIQUE (order_id, transaction_type);

If the same PENDING event arrives twice, the second insert will fail with a constraint violation — preventing the balance from being blocked twice for the same order.

REST API

Method	Endpoint	Description
POST	`/api/v1/wallet/{userId}/deposit`	Deposit funds
POST	`/api/v1/wallet/{userId}/withdraw`	Withdraw available funds
GET	`/api/v1/wallet/{userId}/summary`	Summary: balance + positions + total equity
GET	`/api/v1/wallet/{userId}/positions`	List asset positions
GET	`/api/v1/wallet/{userId}/transactions`	Transaction history

Swagger UI: http://localhost:8085/swagger-ui.html

✅ Validating the Execution

With the application running locally:

✅ Flyway applied all 4 migrations successfully
✅ Hibernate validated the schema
✅ Kafka consumer connected and subscribed to order-events-v1
✅ Consumer group trading-broker-wallet synced
✅ Swagger UI with 5 documented endpoints at http://localhost:8085/swagger-ui.html

What's Next?

With the Wallet ready to react to order events, the next step is building the trading-broker-order — the orchestrator that will manage the complete order lifecycle:

Receive buy/sell intent via REST
Validate the ticker against trading-broker-asset
Save the order as PENDING and publish to Kafka
Send the order to the Matching Engine via RabbitMQ
Consume the result and update the final status

Once that service is ready, we'll have the first complete end-to-end flow: from the user's click all the way to the updated wallet balance.

Got any questions about the custody logic or the PENDING → FILLED → REJECTED cycle? Drop them in the comments!

About the Series

⬅️ Previous Post: The Heart of B3: Building the Matching Engine with RabbitMQ and Redis

Series Index: Series Roadmap

Links:

Microservices Are Easy to Create and Manage

Fri, 08 May 2026 11:22:50 +0000

I bumped into the claim somewhere on LinkedIn and that got me thinking why some people may think so while others consider this a huge overstatement. As I often see, such contradictions emerge because people judge microservices from very different implied contexts. In this article I make a draft attempt to make the context explicit and common to reconcile these contradictions.

NB: However, this article is more for my own structured thinking sake I would welcome and gladly discuss your experience in comments.

Summary

Microservices are easy to create and manage when there are few of them (2-3-5), the core product functionality is stable within 6+ months (no inter-service contracts change), the workflows are a few (2-3) and simple, and load is small (e.g. 10-50 RPS). When one of these characteristics is disturbed the very high immanent complexity of microservices spills out and quickly becomes very hard to manage.

Nevertheless, the statement "microservices are easy to create and manage" is very attractive as a target state, and this writing aims at defining the ways to get close to that target. After all, making complex things simple and manageable is a part of our craft - software engineering.

Why Microservices Are So Complex

Reliable microservices require very complex implementation. Compared to correctly done modularized monoliths, the number of extra dependencies is like 20+. I list the core ones:

Inter-services contracts: (e.g. OpenAPI, AsyncAPI), 1 contract per each service inputs and outputs - creation and alignment with 2 other dependents (inputs and outputs), contract versioning, contract enforcement.
Messaging: the facility (e.g. RabbitMQ) - at least one separate unit to deploy, .
Deployment complexity: requires versioning match and Kubernetes for more than 2-3 services.
Operations - debugging and operational observability: structured JSON logging for business flow, error logging for system exceptions, monitoring for service health (Prometheus, Grafana and all that stuff).

Microservices Dependencies Reference

Here are 4 large operational groups of dependencies for further analysis.

Boundary Integrity

strict contracts: simultaneous cross-service contract evolution, backward compatibility, contract versioning pressure.
minimal shared state: data duplication, duplicated data consistency.
stable ownership boundaries: clear service responsibility and dependency isolation.

Purpose: Prevent uncontrolled coupling and assumption leakage between services. Core question: "What is allowed to cross service boundaries?"

Workflow Determinism

deterministic workflows: complexity controlled with explicit state machines and behavioral patterns (strategy etc).
idempotent operations: retries, duplicate suppression and replay safety (preserved state, correlation IDs).
async/event workflows: application event/domain facts contract enforcement and evolution management.
error handling: clean separation between domain errors and application errors.

Purpose: Ensure distributed processes behave predictably despite retries, delays, duplicates, and failures. Core question: "Will the system converge to the same valid state under real-world distributed conditions?"

Debugging and Operational Observability

Purpose: Make distributed behavior explainable and diagnosable. Core question: "Can we reconstruct what happened and why?".

This area gradually expands into:

logs: (structures JSON, application, exception logs).
correlation IDs.
distributed traces.
operational and business metrics.
workflow/data lineage.
auditability.

Delivery

Purpose: Reduce operational complexity by deployment standardization and automation across services. Core question: "Can every service be operated the same way?”

This includes:

standardized deployment model.
runtime conventions.
CI/CD.
infrastructure topology.
service templates.
operational procedures.
resilience standards.

Suggestion

Reducing complexity in these four groups by standardizing and automating gradually and systematically paves the way to the "microservices are easy to create and manage" target state however hardly achievable for complex and rapidly evolving systems.

Basic Reading: Sam Newman, "Monolith To Microservices", "Building Microservices".

Market Data Integrator: Consuming Real-Time Data with Python, MongoDB, and Kafka

Sun, 03 May 2026 20:33:47 +0000

Hey folks!

Continuing the My Broker B3 series, today we'll build the first microservice in the ecosystem: the trading-broker-market-data.

This service has a simple and well-defined responsibility: fetch real quotes for Brazilian assets from the Brapi API and distribute them to the ecosystem — persisting the history in MongoDB and publishing updated prices to Kafka for other services to consume.

️ This Service's Role in the Ecosystem

[brapi.dev] ◀── HTTP ── [broker-market-data-api]
                                  │
                    ┌─────────────┴──────────────┐
                    ▼                             ▼
             MongoDB                           Kafka
        (price_history)          (trading-assets-market-data-v1)
                                               │
                                    [broker-asset-api] (consumer)

It's the entry point for real data into the system. Without it, the ecosystem operates on static prices.

MVP Focus

The goal here is clear: have real quotes available in Kafka for other services to react to. In this phase, I prioritized:

Scheduled ingestion of 50 assets (Blue Chips, REITs and ETFs)
Historical persistence in MongoDB for audit
Kafka event publishing for downstream consumers
Rate limit protection for the Brapi API (free plan)
Immediate execution on service startup

️ Tech Stack

Technology	Usage
Python 3.12	Service runtime
requests	HTTP client for Brapi API
pymongo	MongoDB persistence
confluent-kafka	Kafka producer
schedule	Periodic scheduling
python-dotenv	Environment variables

️ The Implementation

The Ingestion Loop

The heart of the service is the run_ingestion() function, which iterates over the watchlist, fetches each asset's price and distributes the data:

def run_ingestion():
    """Fetches market data from Brapi, persists to MongoDB and publishes to Kafka."""
    logger.info("Market data ingestion job started")

    token = os.getenv("BRAPI_TOKEN")
    if not token:
        logger.error("BRAPI_TOKEN not set. Aborting.")
        return

    # Producer created per run to avoid crash on import if Kafka is unavailable
    producer = Producer({
        'bootstrap.servers': os.getenv("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092"),
        'client.id': 'broker-market-data-api'
    })

    mongo = MongoRepository()
    success_count = 0
    error_count = 0

    for ticker in WATCHLIST:
        try:
            url = f"https://brapi.dev/api/quote/{ticker}?token={token}"
            response = requests.get(url, timeout=10)
            response.raise_for_status()

            result = response.json()["results"][0]
            payload = {
                "ticker": result.get("symbol"),
                "name": result.get("longName"),
                "short_name": result.get("shortName"),
                "price": result.get("regularMarketPrice"),
                "volume": result.get("regularMarketVolume"),
                "updated_at": result.get("regularMarketTime")
            }

            # 1. Persist historical record in MongoDB
            mongo.save(payload.copy())

            # 2. Publish event to Kafka
            producer.produce(
                topic=TOPIC_NAME,
                key=ticker,
                value=json.dumps(payload).encode('utf-8'),
                callback=delivery_report
            )
            producer.poll(0)

            success_count += 1
            time.sleep(0.5)  # Rate limiting — respect Brapi free plan

        except Exception as e:
            error_count += 1
            logger.error(f"Error processing ticker {ticker}: {e}")

    producer.flush()
    logger.info(f"Ingestion completed: {success_count} succeeded, {error_count} failed")

Important design decisions:

Why create the Producer inside the function? If it were at module level, an unavailable Kafka at import time would crash the script before executing any logic. Creating it inside the function means a Kafka failure doesn't prevent execution — only that run fails.

Why try/catch per ticker? If one asset fails (timeout, invalid ticker), the loop continues for the rest. The error_count at the end shows how many failed without stopping the work.

The time.sleep(0.5) — respects the rate limit of Brapi's free plan. For 50 assets, the full run takes ~25 seconds.

The Scheduling

if __name__ == "__main__":
    logger.info(f"Starting — sync every {SYNC_INTERVAL_MINUTES} minutes")

    # Run immediately on startup, then on schedule
    run_ingestion()

    schedule.every(SYNC_INTERVAL_MINUTES).minutes.do(run_ingestion)

    while True:
        schedule.run_pending()
        time.sleep(1)

Why run immediately on startup? If the service restarts, we don't want to wait 30 minutes for data in Kafka. The first execution is immediate, subsequent ones follow the schedule.

The Kafka Payload

{
  "ticker": "PETR4",
  "name": "Petróleo Brasileiro S.A. - Petrobras",
  "short_name": "PETROBRAS PN",
  "price": 35.50,
  "volume": 15234567,
  "updated_at": 1714234567
}

Topic: trading-assets-market-data-v1
Key: asset symbol (e.g. PETR4) — ensures events for the same asset go to the same partition

The Watchlist

50 assets covering the main B3 segments:

WATCHLIST = [
    # Blue Chips
    "PETR4", "PETR3", "VALE3", "ITUB4", "BBDC4", "BBAS3", ...
    # REITs / FIIs
    "MXRF11", "HGLG11", "KNIP11", "XPLG11", ...
    # ETFs
    "BOVA11", "IVVB11"
]

Security: Token Out of Code

# .env — never committed
BRAPI_TOKEN=your_token_here

# application — no fallback, required variable
token = os.getenv("BRAPI_TOKEN")
if not token:
    logger.error("BRAPI_TOKEN not set. Aborting.")
    return

Never expose credentials as default values in public repositories.

✅ Validating the Execution

With the service running locally:

✅ First run executes immediately on startup
✅ MongoDB receives documents with created_at automatically
✅ Kafka receives messages with key = ticker
✅ Scheduler waits 30 minutes for the next round

What's Next?

With broker-market-data-api publishing quotes to Kafka, the next step is b3-market-sync-api — the Java service that consumes these events and synchronizes prices to Redis, feeding the Matching Engine with real-time data.

About the Series

⬅️ Previous Post: Infrastructure as Code

➡️ Next Post: Syncing the Real Market: Consuming Brapi and Feeding Redis with Spring Boot

Series Index: Series Roadmap

Links:

The Evolution of a SaaS Architecture

Sun, 03 May 2026 04:04:26 +0000

Building a scalable SaaS platform is a journey of continuous learning and strategic evolution. In modern software engineering, the most successful platforms are those built with an "Evolutionary Architecture" mindset. This means designing your systems to embrace change, allowing your technical capabilities to grow in perfect harmony with your user base and business milestones.

When launching a new platform, the primary objective is learning about the market, discovering user needs, and establishing product-market fit. A successful architecture supports this discovery phase by prioritizing developer velocity and adaptability. By aligning your infrastructure with your current traffic levels, you ensure that engineering resources are focused entirely on delivering value to the user.

The strategy for building a highly resilient, long-lasting platform relies on a phased, educational approach. You map your infrastructure investments directly to the milestones you achieve in user growth. We can organize this philosophy into three essential pillars of technical growth:

Build for validation: Design systems that allow for rapid prototyping, incredibly fast deployment cycles, and immediate user feedback.
Upgrade for stability: As user acquisition accelerates, introduce foundational scaling techniques to ensure high availability and consistent response times.
Architect for scale: When your platform reaches enterprise levels of engagement, systematically decouple components to unlock distributed, specialized computing power.

By adopting this progressive framework, you maintain a highly efficient engineering budget while gracefully supporting increased demand. Here is a comprehensive, deeply technical roadmap for guiding your application's journey from a unified monolith to an intelligent, event-driven microservices ecosystem.

Phase 1: The Monolithic Approach

Traffic Milestone: 0 to 1,000 users. The Tech: A Modular Monolith on a dedicated virtual environment.

At the inception of a project, engineering speed is your greatest asset. The goal is to translate ideas into functional code as efficiently as possible to gather real-world data. A monolithic architecture is uniquely equipped to provide this rapid iteration cycle, allowing you to ship new features daily without the overhead of managing distributed systems.

In a well-designed monolithic architecture, all core functionalities your user interface serving, business logic, background jobs, and data access layers reside within a single, unified codebase and operate within the same memory space.

This unified approach offers tremendous advantages for an evolving product. Deployments are straightforward, often requiring just a single CI/CD pipeline step. Because all modules share the same memory, internal function calls are lightning-fast, entirely avoiding the latency and complexity of network-based API communication. Furthermore, observing the system is beautifully simple; developers can trace a user's entire journey through the application by examining a single, centralized stream of logs.

To make the most of this phase, engineering teams should focus on building a Modular Monolith. By strictly enforcing logical boundaries within the code such as separating the user management module from the billing module you create a clean, well-organized system. This discipline not only makes the codebase easier to understand and test but also perfectly positions the application for future architectural evolution.

Phase 2: Horizontal Scaling and the 12-Factor App

Traffic Milestone: 1,000 to 10,000 users.

As your platform's value resonates with the market, user engagement naturally increases. You will begin to observe higher CPU utilization and increased memory consumption as your server actively processes concurrent user requests. This is a positive milestone it signals that your platform is thriving and it is time to expand your compute capacity.

To handle increased throughput gracefully, the architecture shifts from vertical scaling (adding a larger server) to horizontal scaling (adding multiple identical servers). By containerizing your application packaging the code, runtime, and dependencies into standardized units using Docker you can launch multiple instances of your application simultaneously.

These instances sit behind an Application Load Balancer. The load balancer acts as an intelligent traffic director, continuously monitoring the health of your application instances and distributing incoming HTTP requests evenly across the fleet. If traffic surges, you can simply instruct your cloud provider to spin up additional containers to share the workload.

The Architectural Shift: Transitioning to a horizontally scaled environment introduces a brilliant engineering concept: Statelessness, a core principle of the 12-Factor App methodology. Because a user's requests might be routed to Server A on their first click and Server B on their second, your application servers can no longer rely on storing local data.

To resolve this, engineers externalize the state. Session data, temporary cache, and user tokens are migrated to a centralized, high-speed, in-memory datastore like Redis. By decoupling the state from the application servers, any server in your fleet can seamlessly pick up a request and process it with full context, creating a deeply resilient, highly available application tier.

Phase 3: Optimizing the Data Layer

Traffic Milestone: 10,000 to 100,000 users.

Expanding your API server fleet handles computational load beautifully, but data-intensive applications eventually shift the demand to the persistent storage layer. In a thriving SaaS environment, ensuring the relational database remains responsive is paramount for a seamless user experience.

Before structurally changing the database, modern engineering teams focus on Intelligent Caching. By leveraging the Redis infrastructure introduced in Phase 2, developers can intercept frequent, heavy database queries. If thousands of users are loading the same global analytics dashboard, the database performs the complex calculation once, stores the result in Redis, and serves the subsequent thousands of requests directly from memory in a fraction of a millisecond. Mastering cache invalidation strategies ensuring users always see fresh data becomes a rewarding engineering focus at this stage.

As engagement deepens, you can further enhance data availability by implementing a Read/Write Split at the database level.

The Master Node: The primary database instance becomes highly specialized, dedicated entirely to processing data modifications handling the INSERT, UPDATE, and DELETE transactions ensuring absolute data integrity.
The Read Replicas: You introduce synchronized copies of your database. The application intelligently routes all complex, read-only queries (SELECT) to these replicas.

This architecture prevents heavy reporting queries from consuming the resources needed for transactional operations. Your application learns to handle "eventual consistency," understanding that there might be a few milliseconds of replication delay between writing to the master and reading from the replica. This elegant separation of concerns ensures that the data layer remains incredibly robust, even during your highest traffic events.

Phase 4: Unlocking the Event-Driven Ecosystem

Traffic Milestone: 100,000+ users.

As your platform reaches enterprise-level utilization, you will likely offer advanced, resource-intensive features. Your application might generate comprehensive data exports, encode high-resolution media, or orchestrate complex interactions with external Artificial Intelligence models.

Handling these monumental tasks synchronously forcing the user to wait while the main API processes the data monopolizes server threads and impacts the experience of other concurrent users. This is the perfect moment to evolve your architecture by isolating these specific workloads into independent Microservices.

Instead of processing heavy tasks on the spot, the main application adopts an Asynchronous, Event-Driven Architecture. The platform delegates the heavy lifting by utilizing a highly durable Message Broker (such as Apache Kafka, Amazon SQS, or RabbitMQ).

The interaction becomes a seamless, non-blocking flow:

The API Gateway: The user requests a complex AI generation. The main API immediately returns a 202 Accepted response, assuring the user that the process has begun.
The Event Stream: The API securely publishes a message payload containing the job details into a dedicated queue.
The Worker Fleet: Independent, specialized microservices designed specifically for CPU-intensive or GPU-intensive workloads listen to this queue. They pick up the messages and process the tasks silently in the background, updating the main database or notifying the user via WebSockets upon completion.

This architectural leap provides the ultimate capability: Granular, Event-Driven Scaling. Because your services are decoupled, you can optimize your infrastructure costs brilliantly. If there is a massive backlog of video encoding jobs, you can configure your cloud environment to automatically spin up 50 instances of the specific "Video Worker" microservice to drain the queue quickly, while keeping your main API fleet operating at a stable, cost-effective baseline.

Cultivating an Ecosystem of Continuous Learning

The evolution from a single server to a distributed microservices architecture is a fascinating journey of continuous technical improvement. The most effective engineering teams understand that architecture is never truly "finished"; it is a living ecosystem that adapts to serve the user.

Start with a Modular Monolith to maximize learning, validate your market hypothesis, and establish clean domain boundaries.
Introduce Load Balancing and Statelessness to guarantee high availability and consistent performance as your audience grows.
Optimize Data Retrieval through intelligent memory caching and database replication, ensuring your persistence layer is always responsive.
Embrace Event-Driven Microservices strategically, isolating complex workloads to unlock specialized computing power and granular scalability.

To guide this evolution effectively, implementing deep Observability is key. By integrating tools like OpenTelemetry, Prometheus, and distributed tracing, you grant your engineering team the ability to visualize how data flows through the system. These metrics serve as the guiding light, telling you exactly when and where to apply the next architectural evolution.

Build systems that serve the present beautifully, while actively designing the foundational agility required for the innovations of tomorrow.

Looking to strategically evolve your platform's architecture? Let's Connect!
I am a Senior Full Stack AI Engineer specializing in the design, deployment, and optimization of highly resilient, cloud-agnostic SaaS platforms and intelligent, event-driven applications.

Building a Microservices Ecosystem: Stock Brokerage Simulator (My Broker B3)

Fri, 01 May 2026 13:50:40 +0000

Hey folks!

I'm starting a series of articles to document the development of My Broker B3 — a personal project where I apply advanced software engineering concepts, distributed systems and messaging to simulate the real-world operation of a stock brokerage integrated with B3 (the Brazilian stock exchange).

The goal isn't to build a production system, but rather a POC (Proof of Concept) that validates real-time order processing, financial custody management and market data orchestration using an event-driven architecture.

️ The Ecosystem Overview

The project was designed following the microservices philosophy with a hybrid stack — each service uses the technology that best fits its function:

                    [broker-gateway-api]
                           │
          ┌────────────────┼────────────────┐
          ▼                ▼                ▼
   [broker-order-api] [broker-wallet] [broker-asset]
          │                │
     RabbitMQ           Kafka (order-events-v1)
          │
   [b3-matching-engine]
          │
       Redis ◀── [b3-market-sync-api] ◀── [brapi.dev]

The Services

Broker Side

Service	Language	Responsibility
`broker-identity-api`	Java	Authentication, JWT, user management
`broker-gateway-api`	Java	API Gateway, routing, JWT validation
`broker-order-api`	Java	Order lifecycle orchestration
`broker-wallet-api`	Java	Financial custody, balance and positions
`broker-asset-api`	Java	Asset catalog and internal market data
`broker-market-data-api`	Python	Quote ingestion from Brapi → MongoDB + Kafka
`broker-report-api`	Java	Reports and history

B3 Side (Simulator)

Service	Language	Responsibility
`b3-matching-engine-api`	Java	Order execution, price matching
`b3-market-sync-api`	Java	Price synchronization to Redis

⚙️ Communication Strategy

One of the most important decisions in the project was choosing when to use synchronous communication and when to use asynchronous. The rule was simple: if the response is needed now to continue the flow, use REST. If it can wait, use messaging.

Synchronous (REST/Feign)

broker-order-api → broker-asset-api: validates ticker before creating order
broker-order-api → broker-wallet-api: validates balance before processing buy
broker-identity-api → broker-wallet-api: creates wallet account on registration

Asynchronous (RabbitMQ)

broker-order-api → b3-matching-engine-api: sends orders for execution
b3-matching-engine-api → broker-order-api: execution feedback (FILLED/REJECTED)

Asynchronous (Kafka)

broker-market-data-api → topic assets-market-data-v1 → broker-asset-api
broker-order-api → topic order-events-v1 → broker-wallet-api

Persistence Strategy

Each service uses the database that best fits its nature:

Technology	Service(s)	Why
MySQL	identity, wallet, order, asset	Transactional data with strong consistency
PostgreSQL	b3-matching-engine	B3 core with execution history
MongoDB	broker-market-data	Quote history — flexible schema
Redis	broker-asset, b3-matching-engine	Real-time price cache for fast decisions

The Order Flow — End to End

To materialize the architecture, here's what happens when a user places a buy order:

1. POST /api/v1/orders
        │
        ├─ Validate balance → broker-wallet-api (REST)
        ├─ Persist PENDING → MySQL
        ├─ Publish PENDING → Kafka (broker-wallet blocks balance)
        └─ Send to B3 → RabbitMQ

2. B3 processes, looks up price in Redis
        │
        ├─ FILLED/REJECTED → RabbitMQ → broker-order-api
        ├─ Update status → MySQL
        └─ Publish final status → Kafka (broker-wallet settles/refunds)

From the user's click to the updated wallet balance, through validation, price matching and eventual consistency.

What You'll Find in the Series

Each post documents the construction of a service, focusing on technical decisions, problems encountered and how they were solved:

Overview ← you are here
Infrastructure — Docker Compose with 12+ containers
broker-market-data-api — Python, Brapi, MongoDB, Kafka
b3-market-sync-api — Price synchronization to Redis
b3-matching-engine-api — The B3 execution engine
broker-asset-api — Asset catalog and cache
broker-wallet-api — Financial custody
broker-order-api — The order conductor
broker-identity-api — Authentication and JWT
broker-gateway-api — API Gateway

Feel free to leave feedback or questions in the comments!

About the Series

Series Index: Series Roadmap

Links:

From Monolith to Modular: A Strangler Migration Playbook

Tue, 28 Apr 2026 14:20:43 +0000

Every successful application eventually outgrows its original architecture. What started as a clean, well-structured monolith becomes a tangled web of dependencies where changing one feature risks breaking three others. Deployments become risky, test suites take forever, and developers spend more time navigating the codebase than writing new features.

The instinct is to rewrite. Do not. Rewrites are how engineering teams burn months of effort and deliver something that, at best, does what the old system already did. The better approach is incremental migration: systematically extracting capabilities from the monolith while keeping the system running in production the entire time.

I have led these migrations across multiple companies — taking legacy codebases with 15+ modules, hundreds of database tables, and years of accumulated technical debt and transforming them into modular, maintainable architectures without downtime. This article is the playbook.

Why Monoliths Become Painful

Monoliths are not inherently bad. They are fast to build, easy to deploy, and simple to reason about — at small scale. The problems emerge gradually:

Deployment coupling: Every change, no matter how small, requires deploying the entire application. A CSS fix and a payment logic change ship in the same deployment.
Dependency tangles: Module A depends on Module B which depends on Module C which depends on Module A. Circular dependencies make changes unpredictable.
Test suite slowdown: Running the full test suite takes 30+ minutes. Developers start skipping tests. Bugs escape to production.
Team bottlenecks: Multiple teams working on the same codebase create merge conflicts, code review backlogs, and release coordination overhead.
Scaling limitations: You cannot scale the payment processing component independently of the user profile component. Everything scales (or fails) together.

The Strangler Fig Pattern

The strangler fig pattern, named after the plant that gradually envelops its host tree, is the safest approach to migrating away from a monolith. The strategy:

Identify a capability to extract from the monolith
Build the new implementation alongside the old one
Route traffic to the new implementation (gradually or all at once)
Remove the old implementation once the new one is proven
Repeat for the next capability

The monolith continues to run throughout. There is no big bang cutover. Each migration step is small, reversible, and independently testable.

Phase 1: Understanding What You Have

Before you extract anything, you need a map of the current system.

Dependency Analysis

# Laravel: Find all cross-module dependencies
# Look for imports/use statements between modules

Laravel Module Boundary Analysis

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\File;

class AnalyzeModuleDependencies extends Command
{
    protected $signature = 'modules:analyze';

    public function handle(): void
    {
        $modules = $this->getModuleDirectories();
        $dependencies = [];

        foreach ($modules as $module) {
            $files = File::allFiles(app_path($module));
            $imports = [];

            foreach ($files as $file) {
                $content = $file->getContents();
                preg_match_all('/use App\\\\(\w+)\\\\/', $content, $matches);
                $imports = array_merge($imports, $matches[1]);
            }

            $externalDeps = array_unique(
                array_filter($imports, fn ($dep) => $dep !== $module)
            );

            $dependencies[$module] = $externalDeps;
        }

        foreach ($dependencies as $module => $deps) {
            $this->info("{$module} depends on: " . implode(', ', $deps));
        }

        $this->detectCircularDependencies($dependencies);
    }

    private function detectCircularDependencies(array $dependencies): void
    {
        foreach ($dependencies as $moduleA => $depsA) {
            foreach ($depsA as $moduleB) {
                if (isset($dependencies[$moduleB]) && in_array($moduleA, $dependencies[$moduleB])) {
                    $this->warn("Circular dependency: {$moduleA} <-> {$moduleB}");
                }
            }
        }
    }

    private function getModuleDirectories(): array
    {
        return collect(File::directories(app_path()))
            ->map(fn ($dir) => basename($dir))
            ->filter(fn ($dir) => !in_array($dir, ['Console', 'Exceptions', 'Http', 'Providers']))
            ->values()
            ->all();
    }
}

Node.js Dependency Mapping

import fs from 'fs';
import path from 'path';

interface ModuleDependency {
  module: string;
  dependsOn: string[];
  files: number;
  linesOfCode: number;
}

function analyzeModuleDependencies(srcDir: string): ModuleDependency[] {
  const modules = fs.readdirSync(srcDir, { withFileTypes: true })
    .filter((d) => d.isDirectory())
    .map((d) => d.name);

  return modules.map((moduleName) => {
    const moduleDir = path.join(srcDir, moduleName);
    const files = getAllFiles(moduleDir, ['.ts', '.js']);
    const imports: string[] = [];
    let totalLines = 0;

    files.forEach((file) => {
      const content = fs.readFileSync(file, 'utf-8');
      totalLines += content.split('\n').length;

      const importRegex = /from\s+['"]\.\.\/(\w+)/g;
      let match;
      while ((match = importRegex.exec(content)) !== null) {
        if (match[1] !== moduleName) {
          imports.push(match[1]);
        }
      }
    });

    return {
      module: moduleName,
      dependsOn: [...new Set(imports)],
      files: files.length,
      linesOfCode: totalLines,
    };
  });
}

function getAllFiles(dir: string, extensions: string[]): string[] {
  const results: string[] = [];
  const entries = fs.readdirSync(dir, { withFileTypes: true });

  for (const entry of entries) {
    const fullPath = path.join(dir, entry.name);
    if (entry.isDirectory()) {
      results.push(...getAllFiles(fullPath, extensions));
    } else if (extensions.some((ext) => entry.name.endsWith(ext))) {
      results.push(fullPath);
    }
  }

  return results;
}

Phase 2: Establish Module Boundaries

Before extracting services, enforce boundaries within the monolith. This is the most important step and the one most teams skip.

Laravel: Module Service Providers and Contracts

// app/Billing/Contracts/BillingService.php
namespace App\Billing\Contracts;

interface BillingService
{
    public function createInvoice(string $customerId, array $lineItems): Invoice;
    public function processPayment(string $invoiceId, array $paymentMethod): PaymentResult;
    public function getCustomerBalance(string $customerId): Money;
}

// app/Billing/BillingServiceProvider.php
namespace App\Billing;

use Illuminate\Support\ServiceProvider;

class BillingServiceProvider extends ServiceProvider
{
    public function register(): void
    {
        $this->app->bind(
            Contracts\BillingService::class,
            Services\BillingServiceImpl::class
        );
    }
}

// app/Orders/Services/OrderService.php
namespace App\Orders\Services;

use App\Billing\Contracts\BillingService;

class OrderService
{
    public function __construct(private BillingService $billing) {}

    public function completeOrder(Order $order): void
    {
        $invoice = $this->billing->createInvoice(
            $order->customer_id,
            $order->lineItems->toArray()
        );

        $order->update(['invoice_id' => $invoice->id, 'status' => 'invoiced']);
    }
}

Node.js: Module Interfaces

// src/billing/contracts.ts
export interface BillingService {
  createInvoice(customerId: string, lineItems: LineItem[]): Promise<Invoice>;
  processPayment(invoiceId: string, method: PaymentMethod): Promise<PaymentResult>;
  getCustomerBalance(customerId: string): Promise<Money>;
}

// src/billing/index.ts
export { BillingServiceImpl as BillingService } from './services/billing-service';
export type { BillingService as BillingServiceInterface } from './contracts';

// src/orders/services/order-service.ts
import type { BillingService } from '../../billing/contracts';

export class OrderService {
  constructor(private billing: BillingService) {}

  async completeOrder(order: Order): Promise<void> {
    const invoice = await this.billing.createInvoice(
      order.customerId,
      order.lineItems
    );

    await this.orderRepo.update(order.id, {
      invoiceId: invoice.id,
      status: 'invoiced',
    });
  }
}

The key insight: modules communicate through interfaces, not direct implementation references. When you later extract the billing module into its own service, you replace the implementation behind the interface. The calling code does not change.

Phase 3: Extract the First Service

Choose a module that is:

Well-bounded (few incoming dependencies)
Independently valuable
Not on the critical path (if the extraction has problems, it should not take down the whole system)

The Routing Layer

Use a proxy or feature flag to route traffic between the monolith and the new service:

Laravel: Switchable Implementation

// app/Billing/BillingServiceProvider.php
class BillingServiceProvider extends ServiceProvider
{
    public function register(): void
    {
        $this->app->bind(Contracts\BillingService::class, function ($app) {
            $useNewService = config('features.billing_service_v2', false);

            if ($useNewService) {
                return new HttpBillingServiceClient(
                    config('services.billing.url'),
                    config('services.billing.api_key'),
                );
            }

            return new Services\BillingServiceImpl();
        });
    }
}

// The HTTP client talks to the extracted billing service
class HttpBillingServiceClient implements Contracts\BillingService
{
    public function __construct(
        private string $baseUrl,
        private string $apiKey,
    ) {}

    public function createInvoice(string $customerId, array $lineItems): Invoice
    {
        $response = Http::withToken($this->apiKey)
            ->timeout(10)
            ->retry(3, 100)
            ->post("{$this->baseUrl}/api/invoices", [
                'customer_id' => $customerId,
                'line_items' => $lineItems,
            ]);

        if ($response->failed()) {
            throw new BillingServiceException("Failed to create invoice: " . $response->body());
        }

        return Invoice::fromArray($response->json());
    }
}

Node.js: Strategy Pattern Switch

import type { BillingService } from './contracts';
import { LocalBillingService } from './services/billing-service';
import { RemoteBillingService } from './services/remote-billing-service';

export function createBillingService(): BillingService {
  if (process.env.BILLING_SERVICE_V2 === 'true') {
    return new RemoteBillingService(
      process.env.BILLING_SERVICE_URL!,
      process.env.BILLING_SERVICE_API_KEY!
    );
  }

  return new LocalBillingService();
}

class RemoteBillingService implements BillingService {
  constructor(
    private baseUrl: string,
    private apiKey: string,
  ) {}

  async createInvoice(customerId: string, lineItems: LineItem[]): Promise<Invoice> {
    const response = await fetch(`${this.baseUrl}/api/invoices`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${this.apiKey}`,
      },
      body: JSON.stringify({ customer_id: customerId, line_items: lineItems }),
    });

    if (!response.ok) {
      throw new Error(`Billing service error: ${response.status}`);
    }

    return response.json();
  }
}

Phase 4: Data Migration Strategy

The hardest part of service extraction is data ownership. When the billing module becomes its own service, who owns the billing tables?

Shared Database Phase

Initially, the new service reads from the same database as the monolith. This avoids data migration complexity during the initial extraction.

Data Sync Phase

Introduce event-based data synchronisation. The new service maintains its own datastore, kept in sync via events:

class BillingDataSyncListener
{
    public function handle(CustomerUpdated $event): void
    {
        Http::post(config('services.billing.url') . '/api/sync/customer', [
            'customer_id' => $event->customerId,
            'name' => $event->name,
            'email' => $event->email,
        ]);
    }
}

Full Ownership Phase

The new service owns its data completely. The monolith no longer writes to billing tables. All access goes through the service API.

Phase 5: Zero-Downtime Cutover

The cutover from monolith to new service should be invisible to users:

Shadow mode: Both implementations run. The new service processes requests but responses come from the monolith. Compare outputs for correctness.
Canary rollout: Route 5% of traffic to the new service. Monitor error rates, latency, and correctness. Gradually increase.
Full rollout: Route 100% to the new service. Keep the monolith implementation for 2 weeks as a rollback option.
Cleanup: Remove the old implementation from the monolith.

class BillingServiceProvider extends ServiceProvider
{
    public function register(): void
    {
        $this->app->bind(Contracts\BillingService::class, function () {
            $strategy = config('features.billing_migration_phase');

            return match ($strategy) {
                'monolith' => new Services\BillingServiceImpl(),
                'shadow' => new ShadowBillingService(
                    primary: new Services\BillingServiceImpl(),
                    shadow: new HttpBillingServiceClient(/*...*/),
                ),
                'canary' => new CanaryBillingService(
                    legacy: new Services\BillingServiceImpl(),
                    modern: new HttpBillingServiceClient(/*...*/),
                    percentage: (int) config('features.billing_canary_percentage', 5),
                ),
                'modern' => new HttpBillingServiceClient(/*...*/),
                default => new Services\BillingServiceImpl(),
            };
        });
    }
}

Measuring Migration Success

Track these metrics throughout the migration:

Metric	Purpose
Deployment frequency	Should increase as modules become independent
Lead time for changes	Should decrease as codebase complexity drops
Test suite runtime	Should decrease per module
Incidents per deployment	Should decrease with smaller change sets
Mean time to recovery	Should decrease with independent rollback capability

Production Results from Past Migrations

Metric	Before (Monolith)	After (Modular)
Deployment frequency	1–2 per week	3–5 per day
Full test suite runtime	35 minutes	8 minutes (per module)
Average deploy risk score	High	Low (per module)
Time to onboard new developer	3 weeks	1 week
Query execution time	1.8s average	180ms average (60% reduction)
System performance	Baseline	40% improvement

Key Takeaways

Never rewrite. Always migrate incrementally. The strangler fig pattern lets you replace the monolith piece by piece while keeping the system running.
Establish boundaries before extracting. Enforce module interfaces within the monolith first. This makes extraction a configuration change, not a rewrite.
Start with a low-risk module. Pick something well-bounded and non-critical for your first extraction. Build confidence before tackling core business logic.
Use feature flags for cutover. Shadow mode, canary rollouts, and instant rollback make the migration safe. Never do a big bang switchover.
Solve data ownership explicitly. Shared database → event-based sync → full ownership is the safest data migration path.
Measure continuously. Deployment frequency, test runtime, and incident rate tell you whether the migration is actually improving things.

Conclusion

Migrating from a monolith to a modular architecture is not a technical project — it is an engineering discipline applied over weeks and months. The strangler fig pattern makes it safe. Interface-first module design makes it clean. Feature flags make it reversible. And continuous measurement makes it accountable.

Whether you are working in Laravel or Node.js, the approach is the same: understand what you have, enforce boundaries, extract behind interfaces, migrate data carefully, and cut over gradually. The monolith did not become painful overnight, and the fix will not happen overnight either. But each step makes the system better, and the cumulative impact is transformative.

From Monolith to Microservices: How Bounded Context Improves AI-Driven Development

Fri, 20 Mar 2026 17:46:34 +0000

Introduction

When we first launched our vibe coding platform, the architecture started as a single monolithic application. Initially this approach allowed us to move quickly and ship features fast.

But as the platform grew, the warning signs appeared.

The codebase became increasingly complex, new features were harder to implement, and development velocity began to slow down. What once felt manageable started to evolve into a tightly coupled system that was difficult to reason about.

To solve this, we made a strategic decision: refactor the platform from a monolith into a microservices architecture.

In this article we explain:

Why monolithic architectures struggle in AI-driven systems
How Domain-Driven Design (DDD) helps define system boundaries
Why bounded contexts are critical for AI agents
How we split our platform into eight microservices

This architectural shift significantly improved both developer productivity and AI agent performance.

Understanding Domain-Driven Design

The concept of Domain-Driven Design (DDD) was introduced by Eric Evans in the influential book Domain‑Driven Design: Tackling Complexity in the Heart of Software.

Although the book was published in 2003, its ideas remain extremely relevant today. Many modern architectural patterns—including microservices—are heavily inspired by DDD principles.

DDD focuses on one core idea:

Software architecture should mirror the structure of the business domain. Instead of organizing systems purely around technical layers, DDD encourages teams to structure software around real business capabilities and concepts.

While doing domain distillation, we detected the 6 main bounded contexts that can act as independent services: project planning, AI coding, project deployment, version control, observability, account and billing.

The Monolith vs Microservices Debate

For years developers have debated a fundamental architectural question:

Should applications remain monolithic, or should they be broken into microservices?

Monoliths have advantages:

Simpler deployments
Easier early development
Fewer distributed systems challenges

However, as systems scale, monoliths often become harder to maintain. Large codebases accumulate complexity and eventually turn into what developers often call a "Big Ball of Mud."

Today the conversation has evolved even further. We are no longer building systems only for human developers. We are building systems that AI agents interact with, modify, and reason about. And this introduces a new challenge.

Why Monolithic Architectures Confuse AI Agents

AI agents rely heavily on clear context boundaries.When a system grows into a large monolithic repository:

Domain concepts overlap
Terminology becomes ambiguous
Architectural boundaries blur

This makes it harder for AI systems to correctly interpret intent. For example, an AI agent might receive a command such as:

Create a project.

In a large monolith, the term project might refer to several different things:

A planning workspace
A source code repository
A deployment environment

Without clear boundaries, the AI agent must guess the meaning. And guessing leads to errors. This is why architectural clarity matters even more in AI-driven development environments.

What Is a Bounded Context?

A Bounded Context is a central concept in Domain-Driven Design. It defines a clear boundary within which a specific domain model applies and terminology has a single meaning.

Inside a bounded context:

Language is consistent
Concepts are clearly defined
Domain logic belongs only to that context

This creates a shared understanding between:

Developers
Domain experts
AI agents

Each context effectively becomes its own mini-world with well-defined rules.

Why Bounded Contexts Improve AI Development

In large software systems, the same word can mean different things in different domains. While building our AI coding platform, we encountered this problem quickly. Consider the word project.

When both meanings existed in the same codebase, confusion spread. Not only for engineers—but for AI agents operating within the platform. By introducing bounded contexts, we ensured that each domain defines its own terminology and behavior. This dramatically improved the reliability of both human workflows and AI-generated development tasks.

Our Microservices Architecture

After refactoring the platform using Domain-Driven Design principles, we divided the system into eight bounded contexts, each implemented as a separate microservice.

This allowed each domain to evolve independently while maintaining clear system boundaries.

Project Planning Context - manages planning activities. Within this domain we store: tasks, projects, agent which generate tasks...
AI Coding Context - this is where development actually happens. AI coding agents operate in this domain and perform tasks such as: writing code, committing changes, managing development workflows.
Project Deployment Context - manages the infrastructure and pipelines required to deploy software built by AI agents.
Version Control Context - while platforms like GitHub provide robust version control, we chose to build a lightweight internal system optimized specifically for AI-driven development workflows.
Observability Context - collects events and analytics from across the platform.
Account and Billing Context - manages user accounts, platform credits, payments.
API Gateway - acts as the central entry point into the platform.
Platform UI - provides the user interface that connects all services together.

Avoiding the Big Ball of Mud

Without clear architectural boundaries, complex systems eventually degrade into tightly coupled codebases that are difficult to maintain. Developers often describe this situation as a "Big Ball of Mud."

Symptoms typically include:

Spaghetti code
Increasing bugs
Slower development cycles
Poor onboarding experience
Confusing system behavior

For AI-driven platforms, the consequences are even worse because agents lose the contextual clarity required to generate reliable code.

By organizing our platform around bounded contexts and microservices, we achieved several key benefits:

Clear domain ownership
Faster feature development
Better system scalability
Improved AI agent performance
More maintainable architecture

Final Thoughts

As AI agents become increasingly involved in software development, architecture design must evolve alongside them. Bounded contexts and microservices are no longer just about scaling engineering teams—they also help AI systems reason about complex software platforms. For us, adopting Domain-Driven Design and microservices architecture transformed a growing monolith into a scalable platform designed for the future of AI-assisted development.

Multiple models are in play on any large project. Yet when code based on distinct models is combined, software becomes buggy, unreliable, and difficult to understand.

— Eric Evans

Escaping the Sync Trap: How I Slashed Latency by 10x in a Django-Rust API Gateway

Tue, 17 Mar 2026 17:16:00 +0000

How I diagnosed and eliminated synchronous bottlenecks in a Django-Rust API gateway, migrating to ASGI and pre-warming caches for millisecond responses.

When building a high-performance backend, the standard playbook is well-known: offload heavy computational tasks to faster microservices (like Rust) and implement an aggressive caching strategy.

Recently, I did exactly that. My architecture is built around a Django REST Framework gateway sitting behind Caddy, heavily monitored with Prometheus and Grafana.

But despite the raw speed of Rust and my caching layers, my dashboards were flashing red. Latency was spiking to brutal 10-second flatlines for my most critical endpoints. Worse, my observability itself started failing, creating silent blind spots exactly when I needed data the most.

Here is the detective story of how I used telemetry to hunt down synchronous traps, migrate to a non-blocking async architecture, and implement proactive pre-warming to bring response times down to the millisecond range — all while reclaiming 30% of my idle CPU.

The Architecture: A Gateway and its Heavy Lifters

Before diving into the problem, here is a quick look at my setup.

          ┌──────────────────────────────┐
          │          Nextcloud           │
          │ (Authenticated Client Calls) │
          └──────────────┬───────────────┘
                         │  JWT / API Key
                         ▼
               ┌────────────────────┐
               │  Django Gateway    │
               │ (ASGI, DRF, Caddy) │
               └──────┬─────────────┘
                      │
     ┌────────────────┴────────────────┐
     │                                 │
     ▼                                 ▼
┌───────────────┐              ┌────────────────┐
│ NDVI Service  │              │ Weather Service│
│ (Rust, 8081)  │              │ (Rust, 8090)   │
│  → Postgres    │              │  → MySQL       │
└───────────────┘              └────────────────┘

       ▲
       │ Prometheus & Grafana
       │ (Observability Stack)
       ▼
   System Telemetry + Metrics

Django stays as the public API gateway, accepting requests authenticated via JWT or API keys.
It enforces a shared JSON response envelope containing status, message, data, and errors to keep all client interactions standardized.

Specific traffic routes — namely /api/v1/ndvi and /api/v1/weather/* — are forwarded directly to my Rust backends:

NDVI microservice ingests satellite data into a dedicated Postgres database.
Weather microservice relies on a MySQL database and communicates with external providers like Open-Meteo and NASA POWER.

My Nextcloud instance acts like any other client, presenting either an Authorization: Bearer token or an X-API-Key. Django manages this traffic using a specific nextcloud_hmac throttle configuration before passing the authorized call down to Rust with the original headers intact.

The False Cure: The Worker Starvation Anomaly

To protect the system, I implemented aggressive TTL caching (e.g., 1 hour for schema data, 5 minutes for API tokens). However, once I added traffic, my Grafana dashboard revealed a chaotic reality.

I saw brutal, perfectly flat 8–10 second latency spikes on key endpoints. Crucially, perfectly timed with these latency spikes, my internal /metrics request rate dropped to zero.

The Diagnosis: The Gateway Caching Itself to Death

The telemetry told a story of hidden synchronous bottlenecks:

The Trigger: When a high-traffic endpoint like farm-weather-current/GET experienced a cache miss, the Django gateway had to fetch fresh data.
The Trap: My Django deployment was running using standard synchronous workers. It called the Rust service, which then called the external weather API (taking 2.2+ seconds).
The Impact (Worker Starvation): Because the Django worker was synchronous, it blocked entirely for those 2.2 seconds. All incoming traffic got stuck in a queue.

The Trap: Synchronous Gateway Routing

import httpx
from django.http import JsonResponse
from django.views import View

async def async_weather_proxy_view(request):
    async with httpx.AsyncClient(timeout=5.0) as client:
        resp = await client.get("http://weather-service:8090/api/v1/weather-current")
        resp.raise_for_status()
        data = resp.json()
    return JsonResponse({"status": "success", "data": data, "errors": None})

I had successfully offloaded work to Rust — but my synchronous Django workers completely nullified the speed gains.

The First Fix: Embracing the Non-Blocking Gateway

I needed to decouple the speed of the gateway from the speed of the external API calls it was routing.
I migrated the Django deployment from synchronous workers to an ASGI (Asynchronous Server Gateway Interface) setup, allowing my gateway to handle requests asynchronously.

I rewrote my proxy views to use asynchronous HTTP clients like httpx:

The Fix: Asynchronous Non-Blocking Routing

import httpx
from django.http import JsonResponse

async def async_weather_proxy_view(request):
    # The event loop is freed! Django can serve other requests while waiting
    async with httpx.AsyncClient() as client:
        rust_response = await client.get("http://weather-service:8090/api/v1/weather-current")
        
    return JsonResponse({
        "status": "success",
        "data": rust_response.json(),
        "errors": None
    })

The visual evidence on my dashboards was a massive, instant victory:

Observability Restored: The metrics scrape line remained unbroken. Django could finally pause a slow weather request, instantly answer the Prometheus scrape, and resume without blocking.
⚡ Instant Internal Routing: In my initial setup, a simple internal metrics scrape took ~84ms. After the ASGI migration, that duration dropped to 11ms.

The Second Fix: Proactive Caching

While the infrastructure was now bulletproof, the end-user experience was still occasionally sluggish.

With an “on-demand” caching strategy, the very first user to request the weather after a 1-hour cache expiration had to pay the Cache Miss Penalty (waiting ~2.2 seconds for the external API).

To eliminate this, I decoupled the data-fetching time from the user-request cycle entirely.
I implemented a Proactive Background Pre-warming pattern using a background task (like Celery) that runs every 55 minutes, independently fetching slow data and silently overwriting the cache before it expires.

The Cache Pre-Warmer (Celery Example)

from celery import shared_task
from django.core.cache import cache
import requests

@shared_task
def pre_warm_weather_cache():
    # Runs in the background every 55 minutes, shielding the user from the 2.2s wait
    response = requests.get("http://weather-service:8090/api/v1/weather-current")
    if response.status_code == 200:
        cache.set("weather_current_data", response.json(), timeout=3600)

The result? The average latency for critical weather endpoints plummeted from seconds to milliseconds as the hot cache permanently took over.

The Grand Slam: Optimizing the Rust Build Pipeline

The final victory of this new architecture came from pure server efficiency.
During deployments, compiling Rust crates (sqlx, syn) from scratch was pegging my 4-core server at 100% CPU, artificially causing timeouts.

To fix this, I implemented cargo-chef in a multi-stage Dockerfile to strictly cache Rust dependencies.

Multi-stage Dockerfile for the Rust Microservice

FROM rust:1.88-slim AS chef
USER root
RUN cargo install cargo-chef
WORKDIR /app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
# Docker caches this heavy dependency build!
RUN cargo chef cook --release --recipe-path recipe.json
COPY . .
RUN cargo build --release --bin weather-service

FROM debian:bookworm-slim AS runtime
WORKDIR /app
RUN apt-get update && apt-get install -y libssl-dev ca-certificates
COPY --from=builder /app/target/release/weather-service /usr/local/bin/
EXPOSE 8090
ENTRYPOINT ["/usr/local/bin/weather-service"]

Between ASGI, background caching, and Docker layer caching, my total Node CPU now rests comfortably between 11% and 13%.
I fundamentally reclaimed 30% of my total server compute capacity.

Conclusion: Finding the Next Bottleneck

Building high-performance API gateways is an ongoing journey of shifting bottlenecks.

By relying strictly on my telemetry, I proved that synchronous workers nullify microservice speed, validated the immense power of ASGI, and eliminated cache miss penalties.

With the gateway running unburdened, my dashboards have revealed one final bottleneck — a 6-to-8 second delay on my token generation endpoint.
Because my CPU is mostly idle, I know exactly what this is: a database connection pool limitation in the Rust service.

And thanks to my new observability baseline, I know exactly where to strike next.

Exploring Microservices with Node.js

Tue, 17 Feb 2026 05:59:17 +0000

Dive into the microservices architecture using Node.js to create scalable applications. Learn practical strategies for implementation.

Microservices architecture has gained traction for its scalability and flexibility. When it comes to Node.js, it shines due to its non-blocking I/O and event-driven model.

To get started with microservices using Node.js, consider breaking your application into small independent services, each handling a specific task. For instance, if you're building an e-commerce platform, you could have separate services for user authentication, product listings, order management, etc.

Key Considerations

Service Communication: Depending on your application needs, you might want to use REST APIs, gRPC, or message queues like RabbitMQ for service communication. Each has its pros and cons based on latency and throughput.
Service Deployment: Tools like Docker and Kubernetes can simplify the deployment of Node.js microservices. They allow you to manage instances and scale effortlessly.
Data Management: You can use either a shared database or separate databases for each service. While a shared database may simplify data consistency, it can become a bottleneck.
Monitoring and Logging: Implement robust monitoring using tools like Prometheus or ELK Stack. By doing this, you ensure that each service is functioning optimally and can quickly identify issues.

Real-world Example

Take a look at companies like Netflix and Uber. They have successfully adopted microservices architecture with Node.js, allowing them to release updates independently and scale their applications depending on user demand.

Conclusion

By embracing microservices with Node.js, you can build resilient and scalable applications. Remember, the key is to keep services as small and focused as possible, promoting an agile development cycle. Consider experimenting with a simple project to understand the nuances better, and soon you’ll see the benefits in your production environment.

Building Production-Ready Spring Boot Microservices: Lessons from Scaling to Millions

Thu, 12 Feb 2026 05:38:38 +0000

Introduction

Over the past several years working in e-commerce, I've architected and scaled microservices handling millions of customer requests daily, including platforms processing over $100M in transactions. In this article, I'll share practical patterns for building Spring Boot microservices that can handle production scale.

This isn't theory—these are battle-tested approaches proven under real-world load.

1. Service Boundaries and Domain-Driven Design

The biggest mistake teams make is creating microservices that are too granular or tightly coupled.

Key principle: Each service should own a complete business capability.

// ❌ BAD: Services that are too granular
CustomerNameService
CustomerAddressService

// ✅ GOOD: Services that own complete domains
CustomerProfileService // owns all customer identity/profile data
SubscriptionService    // owns subscriptions, benefits, rewards

When building a subscription platform, a SubscriptionService should own the complete domain: enrollment, benefits tracking, payment integration, and conversions. This prevents the "distributed monolith" where you call 10 services for one operation.

2. Handling Distributed Transactions with Saga Pattern

At scale, distributed transactions become your enemy. Here's the orchestration approach:

@Service
public class SubscriptionEnrollmentOrchestrator {
    
    @Transactional
    public EnrollmentResult enrollMember(EnrollmentRequest request) {
        // Step 1: Validate customer (synchronous)
        CustomerProfile customer = customerClient.getCustomer(request.getCustomerId())
            .orElseThrow(() -> new CustomerNotFoundException());
            
        // Step 2: Process payment (synchronous with compensation)
        PaymentResult payment;
        try {
            payment = paymentClient.processPayment(request.getPaymentMethod());
        } catch (PaymentException e) {
            throw new EnrollmentFailedException("Payment failed", e);
        }
        
        // Step 3: Create subscription (local transaction)
        Subscription subscription;
        try {
            subscription = createSubscription(customer, payment);
            subscriptionRepository.save(subscription);
        } catch (Exception e) {
            // Compensate payment
            paymentClient.refundPayment(payment.getTransactionId());
            throw new EnrollmentFailedException("Subscription creation failed", e);
        }
        
        // Step 4: Publish event for async operations
        eventPublisher.publish(new SubscriptionEnrolledEvent(subscription));
        
        return EnrollmentResult.success(subscription);
    }
}

Key lessons:

Make operations idempotent - Include transaction IDs so retries don't cause duplicates
Compensate explicitly - If step N fails, undo steps 1 through N-1
Separate critical path from eventual consistency - Payment is synchronous, emails are async

3. Strategic Caching for High-Traffic Systems

We reduced database load by 85% through strategic caching:

@Service
public class SubscriptionService {
    
    private final LoadingCache<String, Optional<Subscription>> cache;
    
    public SubscriptionService(SubscriptionRepository repository) {
        this.cache = CacheBuilder.newBuilder()
            .maximumSize(10_000)
            .expireAfterWrite(5, TimeUnit.MINUTES)
            .recordStats()
            .build(CacheLoader.from(repository::findByCustomerId));
    }
    
    @Transactional
    public Subscription updateSubscription(String id, SubscriptionUpdate update) {
        Subscription updated = repository.findById(id)
            .map(s -> applyUpdate(s, update))
            .orElseThrow(() -> new SubscriptionNotFoundException(id));
            
        repository.save(updated);
        cache.invalidate(updated.getCustomerId()); // Invalidate on write
        
        return updated;
    }
}

Caching layers:

Application-level (Guava/Caffeine) - 5-minute TTL for hot data
Redis distributed cache - 15-minute TTL for cross-instance consistency
Database with proper indexing

4. Circuit Breakers and Resilience

When calling other services, assume they will fail:

@Service
public class CustomerProfileClient {
    
    @CircuitBreaker(name = "customer-profile", fallbackMethod = "getCustomerFallback")
    @TimeLimiter(name = "customer-profile")
    public CompletableFuture<CustomerProfile> getCustomer(String customerId) {
        return CompletableFuture.supplyAsync(() -> 
            restTemplate.getForObject("/api/customers/" + customerId, CustomerProfile.class)
        );
    }
    
    private CompletableFuture<CustomerProfile> getCustomerFallback(String customerId, Exception ex) {
        log.warn("Customer service unavailable, using cached data");
        return CompletableFuture.completedFuture(
            customerCache.get(customerId)
                .orElseThrow(() -> new ServiceUnavailableException("No cached data", ex))
        );
    }
}

Configuration:

CircuitBreakerConfig.custom()
    .failureRateThreshold(50)          // Open if 50% fail
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .slidingWindowSize(100)            // Consider last 100 calls
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .build();

5. Structured Logging and Distributed Tracing

You can't fix what you can't see:

@Slf4j
@RestController
public class SubscriptionController {
    
    @PostMapping("/subscriptions")
    public ResponseEntity<SubscriptionResponse> enrollMember(
            @RequestBody EnrollmentRequest request,
            @RequestHeader("X-Request-ID") String requestId) {
        
        MDC.put("requestId", requestId);
        MDC.put("customerId", request.getCustomerId());
        
        try {
            log.info("Starting enrollment", 
                kv("tier", request.getTier()),
                kv("paymentMethod", request.getPaymentMethod().getType()));
            
            Instant start = Instant.now();
            EnrollmentResult result = enrollmentService.enroll(request);
            
            log.info("Enrollment completed",
                kv("subscriptionId", result.getSubscriptionId()),
                kv("durationMs", Duration.between(start, Instant.now()).toMillis()));
            
            return ResponseEntity.ok(toResponse(result));
        } finally {
            MDC.clear();
        }
    }
}

Logging standards:

Use structured key-value pairs, not string concatenation
Track request IDs across service boundaries
Include customer ID and operation type in every log

6. Database Query Optimization

Slow queries kill scalability:

@Entity
@Table(name = "subscriptions", indexes = {
    @Index(name = "idx_customer_id", columnList = "customer_id"),
    @Index(name = "idx_tier_status", columnList = "tier, status"),
    @Index(name = "idx_expires_at", columnList = "expires_at")
})
public class Subscription {
    // Use @BatchSize to prevent N+1 queries
    @OneToMany(mappedBy = "subscription", fetch = FetchType.LAZY)
    @BatchSize(size = 50)
    private List<Benefit> benefits;
}

@Repository
public interface SubscriptionRepository extends JpaRepository<Subscription, String> {
    
    // Custom query with fetch join to avoid N+1
    @Query("SELECT s FROM Subscription s LEFT JOIN FETCH s.benefits WHERE s.customerId = :customerId")
    Optional<Subscription> findByCustomerIdWithBenefits(@Param("customerId") String customerId);
}

Performance lessons:

Index everything you query by
Use batch fetching to prevent N+1 queries
Paginate large results
Monitor and alert on queries >100ms

7. Connection Pool Tuning

spring:
  datasource:
    hikari:
      # Formula: (core_count * 2) + effective_spindle_count
      # For 4-core with SSD: (4 * 2) + 1 = 9
      maximum-pool-size: 10
      minimum-idle: 5
      connection-timeout: 3000
      validation-timeout: 1000
      leak-detection-threshold: 60000
      max-lifetime: 1800000  # 30 minutes

8. Zero-Downtime Deployments

# kubernetes/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: subscription-service
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 3        # Add 50% extra during update
      maxUnavailable: 1  # Keep 5/6 running
  template:
    spec:
      containers:
      - name: subscription-service
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 10

9. Metrics That Matter

@Service
public class MetricsService {
    
    private final Counter enrollmentsCounter;
    private final Timer enrollmentDuration;
    
    public MetricsService(MeterRegistry registry) {
        this.enrollmentsCounter = Counter.builder("subscription.enrollments")
            .tag("service", "subscription")
            .register(registry);
            
        this.enrollmentDuration = Timer.builder("subscription.enrollment.duration")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(registry);
    }
}

Monitor:

Business: Enrollments/hour, conversion rate, revenue
Technical: P50/P95/P99 latency, error rate, throughput
Infrastructure: CPU, memory, connection pool utilization

10. Testing Strategy

// Unit tests - fast, isolated
@Test
void shouldCalculateCorrectExpirationDate() {
    EnrollmentService service = new EnrollmentService(
        mock(CustomerClient.class),
        mock(PaymentClient.class),
        mock(SubscriptionRepository.class)
    );
    
    Instant enrolledAt = Instant.parse("2024-01-01T00:00:00Z");
    Instant expected = Instant.parse("2025-01-01T00:00:00Z");
    
    assertEquals(expected, service.calculateExpiration(enrolledAt, SubscriptionTier.ANNUAL));
}

// Integration tests - with real database
@SpringBootTest
@Testcontainers
class SubscriptionServiceIntegrationTest {
    
    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:15");
    
    @Test
    void shouldPersistAndRetrieveSubscription() {
        EnrollmentResult result = service.enroll(request);
        Optional<Subscription> retrieved = service.getSubscription(result.getSubscriptionId());
        
        assertTrue(retrieved.isPresent());
        assertEquals(SubscriptionTier.BASIC, retrieved.get().getTier());
    }
}

Key Takeaways

After building and scaling microservices handling millions of requests:

Design for failure - Circuit breakers, timeouts, and fallbacks are not optional
Embrace eventual consistency - Not everything needs to be synchronous
Observability is critical - You can't improve what you can't measure
Cache aggressively - But invalidate intelligently
Test at every level - Unit, integration, contract, and load tests
Automate deployments - Including rollbacks and scaling
Performance is a feature - Optimize from day one

What's Next?

In future posts, I'll dive deeper into:

Advanced observability with distributed tracing
Chaos engineering and resilience testing
Cost optimization for cloud deployments

Questions or want to discuss these patterns? Leave a comment below or connect with me on LinkedIn.

Tags: #spring-boot #microservices #java #system-design #aws #architecture #scalability #production-engineering

Alvor Bank Architecture (.NET + Angular)

Sat, 06 Dec 2025 21:02:56 +0000

Hey everyone

I’ve been working on a full-stack digital banking platform called Alvor Bank, built with:

.NET 10 (microservices + Clean Architecture)
Angular 21 (monorepo-style frontend)
PostgreSQL, RabbitMQ, YARP API Gateway
Docker & GitHub Actions CI/CD

To move beyond “hello world” tutorials, I put together a small architecture & quality roadmap for the system, and I’m sharing the slides here.

In the slides you can see:
-The high-level architecture diagram
-VS solution structure and how the projects are organized
-How I enforce test coverage as a quality gate
-A glimpse of the CI/CD pipeline (build, test, lint, Docker)
-The security/IAM setup (centralized auth, roles, tenants)
-The final project scope for the platform

I’m also documenting everything step-by-step in an early-access book where I build this system from scratch, chapter by chapter:

Modern Full-Stack Banking Microservices (in progress on Leanpub)
https://leanpub.com/fullstack-banking-microservices

I’d really like to hear from other engineers here:

How do you enforce quality gates (tests, coverage, linting, security) in your own projects?

Happy to discuss architecture choices, trade-offs, or anything in the slides.

I’m not building another TODO app. I’m building a production-grade digital bank. Here's what that actually looks like in code

System overview: Microservices, event-driven notifications, and full IAM

Real project structure: Clean Architecture layers, separate service projects, shared building blocks.

We treat tests as a priority, not an afterthought. Coverage gates in CI ensure quality

Everything ships through CI/CD from day one. Green builds and enforced quality thresholds

Coverage report test on CI/CD from Gih=thub Actions

This is from my book Modern Banking Microservices with Clean Architecture, DDD, TDD, .NET 10, and Angular 21 (nx Monorepo). Go beyond tutorials. Build systems that matter. Link: 1: https://leanpub.com/fullstack-banking-microservices.

Enhanced Paperless-NGX with Paddle OCR + LLM Pipeline

Mon, 06 Oct 2025 09:15:37 +0000

Building a Private AI Document Pipeline with Paperless, PaddleOCR, and LLMs

So this past week I hacked together a little side project to smarten up my Paperless-ngx setup — you know, that self-hosted document management system that eats PDFs and makes them searchable.

Now, Paperless-ngx is solid, don’t get me wrong. But it uses Tesseract for OCR, and honestly... Tesseract is not optimal for anything that's not clean text.

So this just evolved out of need to improve paperless-ngx's OCR capability and to properly classify documents and extract tags. titles and summaries for documents.

This blog is a walkthrough of what I built — what worked, what didn’t, and how it turned into a pretty neat little pipeline with its own microservices. Hope it gives you some ideas.

What I Was Going For

I wanted a system that:

Uses PaddleOCR instead of Tesseract for better OCR output
Runs a local LLM using Ollama to:
- Suggest a smart document title
- Classify the document into a type (invoice, id, tax, etc.)
Pushes that back to Paperless so the doc is nicely searchable and tagged

Everything stays local/private. No exposure to external LLMs. Just Python, containers, compose to stitch everything together.

How It All Works (Now)

After a few iterations, I ended up with a clean microservice setup with each part doing its job:

paperless-ngx: The main document management system (already amazing)
ollama: Runs a local LLM like Mistral (or phi3 or any configurable in env), no cloud stuff
ocr-service: FastAPI service that runs PaddleOCR
pipeline: Python CLI that connects all the dots — downloads doc from Paperless, sends it to OCR and LLM, then updates Paperless with the smart results

That way, each service does one thing and does it well or can be enhanced in isolation. They all run in Docker, talk to each other over the same network, and make one smooth, local AI document workflow.

This leverages paperless-ngx's extensive features and augments it with better OCR and LLM classification capabilities.

The Flow Looks Like This:

 Paperless stores docs in its DB
⬇️
 I run my pipeline CLI: `docker compose run pipeline 42`
⬇️
 pipeline downloads the doc via Paperless API (by ID)
⬇️
 Sends it to the OCR microservice over HTTP
⬇️
 Gets back clean OCR’d text
⬇️
 Sends text to Ollama (LLM) to generate:
    - title
    - document type
⬇️
 Updates Paperless document via PATCH

Everything Runs in Docker

Here's the final list of containers:

paperless → standard Paperless-ngx
redis → required by Paperless
ollama → runs local LLMs like Mistral
ocr-service → FastAPI + PaddleOCR
pipeline → command-line microservice that ties it all together

Architecture Diagram

                   ┌────────────────────────┐
                   │  Paperless-ngx (UI)  │
                   └────────────┬───────────┘
                                │
                    [User notes Document ID]
                                │
                   ┌────────────▼────────────┐
                   │    Pipeline Service    │
                   │ (LLM & Orchestration)   │
                   └────────────┬────────────┘
                                │
           ┌────────────────────┼────────────────────┐
           │                    │                    │
           ▼                    ▼                    ▼
  ┌────────────────┐   ┌────────────────┐   ┌────────────────────┐
  │ Downloads PDF  │   │ Sends to OCR   │   │ Sends OCR text to  │
  │ via Paperless  │   │ microservice   │   │ Ollama LLM (Mistral)│
  └────────────────┘   └────────────────┘   └────────────────────┘
                                                │
                     ◀────────────┬─────────────┘
                                  ▼
                        Title + Type Prediction
                                  │
                        PATCH back to Paperless
                       (update metadata + text)

What Gave Me Trouble

Paperless consumes files from consume/ automatically and moves it — I had to work around that by working only with doc IDs via API as at this time I was more focused on adding OCR/AI features. This one is high on my TODO list.
PaddleOCR kept re-downloading models — made sure models are cached.

Folder Structure

├── docker-compose.yml
├── __init__.py
├── model-cache
├── ocr_service
│   ├── app
│   │   ├── __init__.py
│   │   ├── main.py
│   │   ├── ocr_config.yaml
│   │   └── ocr_engine.py
│   ├── Dockerfile
│   └── requirements.txt
├── paperless-data
│   ├── consume
│   ├── data
│   │   ├── db.sqlite3
│   │   ├── index
│   │   │   ├── _MAIN_63.toc
│   │   │   ├── MAIN_9v88o8vye3gbqub1.seg
│   │   │   ├── MAIN_ui8jrpftrvauh4n1.seg
│   │   │   ├── MAIN_wceagdbh71brm5wg.seg
│   │   │   └── MAIN_WRITELOCK
│   │   ├── log
│   │   │   └── celery.log.1
│   │   └── migration_lock
│   └── media
│       ├── documents
│       │   ├── archive
│       │   │   ├── 0000009.pdf
│       │   │   └── 0000016.pdf
│       │   ├── originals
│       │   │   ├── 0000009.jpg
│       │   │   └── 0000016.pdf
│       │   └── thumbnails
│       │       ├── 0000009.webp
│       │       └── 0000016.webp
│       └── media.lock
├── pipeline_service
│   ├── app
│   │   ├── api_client.py
│   │   ├── __init__.py
│   │   ├── llm_processor.py
│   │   ├── main.py
│   │   └── watcher.py
│   ├── Dockerfile
│   ├── __init__.py
│   ├── logger.py
│   ├── prompts
│   │   └── classify_title.txt
│   ├── requirements.txt
│   └── test.py
└── README.md

✅ What's Next?

I might:

Run the pipeline automatically when a new doc lands
Add authentication for OCR and pipeline service (utilizing paperless-ngx's token auth?)
Improve performance of OCR service, perhaps using other language (Go, Rust)
Add document summarization via LLM
Extract metadata like amount, date, sender
Hook into Paperless tags and correspondents

Wrapping Up

If you need a document management system:

That ingests all kinds off docs PDFs, images etc.
That reliably extracts text from varied kind of docs.
That classifies, tags and summarizes docs using LLM.
That keeps stuff private - no exposure to external LLMs.
It comes with features loaded from paperless-ngx.

Then this might be an appealing setup.

It’s Python all the way down. No rocket science — just containers, OCR, and a private LLM.

Questions or inputs are welcome.

Saga Pattern — an Introduction

Sun, 17 Aug 2025 07:58:36 +0000

In the world of microservices and distributed systems, managing data consistency across multiple services presents unique challenges. Traditional database transactions with their ACID guarantees work beautifully within a single database, but they fall short when your business logic spans multiple services, each with its own database. Enter the Saga pattern—a powerful approach to handling distributed transactions that has become essential in modern microservices architectures.

You can read the blog or read the research paper. I recommend the latter.

Sample projects:

Online Store Saga Choreography Example
Hotel Booking Saga Orchestration Example

What is the Saga Pattern?

The Saga pattern, first introduced by Hector Garcia-Molina and Kenneth Salem in their 1987 research paper, provides a way to manage long-running transactions that span multiple services in a distributed system. Rather than treating the entire business process as a single atomic transaction, a saga breaks it down into a series of smaller, independent transactions that can be coordinated across services.

Think of booking a vacation package that involves reserving a flight, hotel, and rental car. In a monolithic system, you might wrap all these operations in a single database transaction. With microservices, each service (Flight Service, Hotel Service, Car Rental Service) manages its own data independently. The Saga pattern allows you to coordinate these separate operations while maintaining the ability to handle failures gracefully.

At its core, a saga consists of a sequence of transactions T₁, T₂, ..., Tₙ, where each transaction has a corresponding compensating transaction C₁, C₂, ..., Cₙ. If any transaction fails, the saga executes the compensating transactions in reverse order to undo the work already completed, ensuring the system remains in a consistent state.

Problems it Solves and Consistency Trade-offs

The Saga pattern addresses several critical challenges in distributed systems:

Distributed Data Management: In microservices architectures, services are designed to be autonomous, each owning its data. Traditional two-phase commit protocols can create tight coupling and availability issues across services. Sagas enable coordination without sacrificing service autonomy.

Long-Running Processes: Business processes often involve multiple steps that may take minutes, hours, or even days to complete. Holding database locks for such extended periods is impractical and can severely impact system performance. Sagas allow these processes to progress incrementally without blocking resources.

Failure Handling: In distributed systems, failures are inevitable. Network partitions, service outages, and timeouts are part of the reality. Sagas provide a structured approach to handle these failures through compensation, rather than simply rolling back and starting over.

Eventual Consistency Trade-off: The Saga pattern embraces eventual consistency over immediate consistency. This means that at any given moment, the system might be in an intermediate state, but it will eventually reach a consistent state once the saga completes or compensates. This trade-off is acceptable—and often preferable—in many business scenarios where absolute consistency isn't critical, but availability and resilience are paramount.

Unlike strict ACID transactions that provide immediate consistency but can be brittle in distributed environments, sagas offer a pragmatic approach that acknowledges the realities of distributed systems. The business logic determines whether eventual consistency is acceptable, and in most real-world scenarios involving multiple services, it is.

Types of Saga Patterns

The Saga pattern can be implemented using two primary approaches, each with distinct characteristics and use cases.

Choreography-Based Sagas

In choreography-based sagas, services coordinate themselves through events without a central coordinator. Each service listens for events, performs its part of the transaction, and publishes events for other services to consume.

How it works: When a user places an order, the Order Service creates the order and publishes an "OrderCreated" event. The Payment Service listens for this event, processes payment, and publishes a "PaymentProcessed" event. The Inventory Service then reserves items and publishes an "ItemsReserved" event, and so on.

Pros:

Decentralized control promotes service autonomy
No single point of failure from a coordinator perspective
Natural fit for event-driven architectures
Services remain loosely coupled

Cons:

Complex to track and debug the overall flow
Difficult to understand the complete business process from code
Challenging to handle circular dependencies
Error handling can become distributed and complex

When to use: Choose choreography when you have a relatively simple saga with clear, linear flow and when you want to maximize service independence. It works well for scenarios where the business process is stable and unlikely to change frequently.

For a practical example, check out this choreography-based online store implementation that demonstrates how services coordinate through events to handle order processing.

Orchestration-Based Sagas

In orchestration-based sagas, a central orchestrator (saga manager) controls the execution flow, explicitly calling services and managing the overall transaction state.

How it works: A Saga Orchestrator receives a request to start a saga, then sequentially calls each service based on predefined logic. It maintains the saga's state and handles both success and failure scenarios by invoking appropriate compensating actions.

Pros:

Clear, centralized control flow that's easy to understand and debug
Explicit state management makes monitoring and troubleshooting straightforward
Easier to implement complex routing logic and conditional flows
Better support for timeout handling and retry mechanisms

Cons:

Central orchestrator can become a bottleneck or single point of failure
Orchestrator needs to know about all participating services
Can lead to more coupled architecture
Additional infrastructure component to maintain

When to use: Opt for orchestration when you have complex business flows with conditional logic, when you need clear visibility into the saga state, or when you're dealing with frequently changing business requirements that benefit from centralized control.

The hotel booking saga orchestration project provides a comprehensive example of how to implement orchestration-based sagas with proper state management and compensation handling.

Comparisons with Other Patterns

Understanding how the Saga pattern compares to other consistency approaches helps clarify when to use each.

Saga vs. Two-Phase Commit (2PC): Two-Phase Commit provides strong consistency through a prepare-commit protocol but comes with significant drawbacks in distributed systems. It's blocking (services must wait for coordinator decisions), has poor fault tolerance (coordinator failure blocks everything), and doesn't scale well across networks with high latency. Sagas, in contrast, are non-blocking, more fault-tolerant, and better suited for loosely coupled microservices, though they provide only eventual consistency.

Saga vs. Event Sourcing: While both patterns work well in event-driven systems, they serve different purposes. Event sourcing focuses on storing state changes as events and rebuilding state from these events. Sagas focus on coordinating multi-service transactions. They complement each other well—you can implement sagas in an event-sourced system.

Saga vs. Distributed Transactions: Traditional distributed transactions aim for immediate consistency across all resources but are complex to implement correctly and perform poorly at scale. Sagas acknowledge that immediate consistency isn't always necessary and provide a simpler, more resilient alternative for most business scenarios.

The practical advantage of sagas lies in their alignment with microservices principles: they maintain service autonomy, provide better availability characteristics, and offer a more pragmatic approach to consistency in distributed systems.

Implementation in Practice with Spring Boot

Spring Boot provides excellent support for implementing saga patterns through various approaches. The most common implementation leverages Spring's event handling capabilities and message queues.

For choreography-based sagas, developers typically use:

Spring Events for intra-service communication
Message brokers (RabbitMQ, Apache Kafka) for inter-service events
Spring Boot Actuator for monitoring saga progress
Custom event handlers that implement both forward and compensating actions

For orchestration-based sagas, the implementation often includes:

A dedicated Saga Orchestrator service built with Spring Boot
Spring State Machine for managing saga states and transitions
RestTemplate or WebClient for service-to-service communication
Scheduled tasks for handling timeouts and retries
Database persistence for saga state management

A typical Spring Boot saga implementation involves creating:

Saga Events: Domain events that represent saga steps
Event Handlers: Methods annotated with @EventListener that process saga events
Compensation Logic: Corresponding handlers for rollback operations
State Management: Tracking saga progress and current state
Error Handling: Timeout management and retry mechanisms

The referenced sample projects demonstrate these concepts in action, showing how to structure your code, handle failures, and implement proper monitoring for production-ready saga implementations.

Common Challenges and Testing Considerations

Implementing sagas in production environments presents several challenges that require careful consideration and planning.

Timeout Management: Services in a saga may become temporarily unavailable or respond slowly. Implementing appropriate timeout strategies is crucial—too short, and you'll have false failures; too long, and failed sagas will tie up resources. Design your timeouts based on realistic service response times and implement exponential backoff for retries.

Compensating Transaction Complexity: Not all operations can be easily compensated. Sending an email notification, for example, can't be "unsent." Design your sagas to minimize non-compensatable actions, or implement semantic compensation (like sending an apology email). Sometimes, the compensation is more complex than the original transaction.

Idempotency: Saga steps may be retried due to network issues or timeouts, so all operations must be idempotent. This means that executing the same operation multiple times should have the same effect as executing it once. Implement proper idempotency keys and state checking to handle duplicate requests gracefully.

Partial Failure Scenarios: The most challenging aspect of sagas is handling scenarios where some steps succeed while others fail, potentially leaving the system in an intermediate state. Design your business processes to be resilient to these intermediate states, and ensure your UI and downstream systems can handle eventual consistency appropriately.

Testing Saga Flows: Testing distributed sagas requires sophisticated approaches:

Unit Testing: Test individual saga steps and their compensations in isolation
Integration Testing: Use tools like TestContainers to test saga flows with real message brokers and databases
Chaos Testing: Deliberately introduce failures at different points to verify compensation logic
End-to-End Testing: Test complete saga flows in staging environments that mirror production

Monitoring and Observability: Implement comprehensive logging and monitoring for saga execution. Track saga instances, their current state, execution times, and failure rates. Tools like distributed tracing can help you follow saga execution across multiple services.

Design Recommendations:

Keep saga steps as small and focused as possible
Design for failure from the beginning—assume every step can fail
Implement proper dead letter queues for handling poison messages
Use correlation IDs to track saga instances across services
Consider implementing saga timeouts at the business process level
Plan for manual intervention in complex failure scenarios

The key to successful saga implementation is thorough testing and gradual rollout. Start with simple, linear sagas before moving to complex orchestration scenarios, and always have monitoring and alerting in place to catch issues early.

Conclusion

The Saga pattern represents a pragmatic approach to managing distributed transactions in microservices architectures. By embracing eventual consistency and providing structured failure handling through compensation, sagas enable developers to build resilient, scalable systems that can handle the complexities of distributed environments.

Whether you choose choreography for simple, event-driven flows or orchestration for complex business processes, the key is to understand your specific requirements and constraints. The pattern's flexibility allows for various implementation approaches, from simple Spring Boot applications to sophisticated orchestration engines.

As distributed systems continue to evolve, the Saga pattern remains a fundamental tool for managing complexity while maintaining the benefits of microservices architecture. Success with sagas comes from careful design, thorough testing, and a clear understanding of the consistency trade-offs that make modern distributed systems both scalable and resilient.

References

Garcia-Molina, H., & Salem, K. (1987). Sagas. ACM SIGMOD Record.
Microservices Patterns: Saga Pattern
Online Store Saga Choreography Example
Hotel Booking Saga Orchestration Example

DevLog 20250807: Microservice for Cloud Scheduling (and Running)

Fri, 08 Aug 2025 04:22:53 +0000

Motivation

When working with routine tasks, it's convenient to have some form of automatic scheduling. Windows has Task Scheduler, Linux has cron, and in AWS, you can use Lambda with EventBridge.

Wouldn't it be nice if we could periodically pull data from the web, do some light processing, and then dump the results into a database - maybe update a website, trigger a CI/CD process, or send a few emails?

I haven't used it, but I imagine that's what Zapier is designed for. Azure/GitHub Actions are certainly handy for CI/CD. But what if we just want to quickly prototype something, ship it to a server, and forget about it?

That's the vision, anyway.

The First Attempts

When I first started building services for cloud computing, I combined both "trigger a run" and "schedule repeated tasks" into a single service. That sort of worked - but it didn't feel elegant. Something was off.

Even now, after deciding to completely separate the "runner" and "scheduler" into two distinct microservices, I can't fully articulate why this approach feels better. It just does. Of course, this means maintaining three or four separate programs now (runner, runner service, scheduler, scheduler service) - but each one is responsible for a single task, and that somehow feels more reassuring.

Separation of Concerns & the Single Responsibility Principle

I asked ChatGPT for a rationale that aligned with my intuition, and here's what it gave me:

What you're discovering is a classic case of separation of concerns (and in the microservice world, the "scheduler-worker" or "cron-vs-runner" pattern), underpinned by the Single Responsibility Principle. Here's why splitting "who decides when" from "what actually does the work" just feels better.

1. Single Responsibility & Cognitive Load

Scheduler service - its sole job is: "At 8 AM on Tuesdays, kick off job X."
Runner service - its sole job is: "When told to run X, do the fetch/process/dump."

By assigning each service a clear, focused purpose, you reduce logic complexity in any single codebase. You don't need to sift through your fetch logic to understand date calculations, and you don't need to worry about HTTP clients inside your scheduler.

Feels better because each code path is narrow, easy to reason about, and small enough to hold entirely in your head.

2. Loose Coupling & Replaceability

When the scheduler and runner communicate through a simple interface (e.g., message queue, webhook, or HTTP call), either can be swapped out independently:

Want to switch from cron-style fixed schedules to GitHub webhook events?
Need to rewrite your worker in another language for better library support?

You only have to reimplement one microservice - not the whole system.

Feels better because the impact of change is localized.

3. Independent Scaling & Reliability

Scheduling frequency and execution workload rarely match 1:1.

Maybe you schedule jobs every minute, but each run only takes a few seconds.
Or you only need one scheduler, but dozens of parallel workers to clear a backlog.

Separating the two allows for independent sizing, deployment, and monitoring.

Feels better because you avoid resource contention (e.g., a busy worker delaying the next schedule) and can autoscale each component based on its own metrics.

4. Clear Boundaries for Observability & Error Handling

Scheduler logs should only show: "Job X was triggered at time T."
Runner logs should only show: "Processing of job X succeeded/failed."

This allows for more targeted alerting and diagnostics:

Alert if a scheduled trigger never reaches the runner.
Alert if the runner fails more than N times, regardless of the schedule.

Feels better because root cause analysis is straightforward - you know where to look first.

5. Pattern Analogues & Ecosystem Support

This approach mirrors patterns built into major platforms:

AWS CloudWatch Events (scheduler) → Lambda (worker)
Kubernetes CronJob (scheduler) → Job pods (workers)
Celery Beat (scheduler) → Celery workers (task executors)

All of them separate the "when" from the "what," and you're starting to feel the same advantages they offer at scale.

In Summary

You're gravitating toward a Scheduler–Worker (or "Cron-as-a-Service" + "Stateless Runner") pattern because it:

Reduces cognitive load by narrowing each service's scope
Decouples timing logic from business logic
Enables independent scaling and more precise reliability controls
Improves testability (you can mock the scheduler in runner tests, and vice versa)
Aligns with proven cloud-native and messaging patterns

That "reassuring" feeling is the comfort of a clean, modular architecture - you can see exactly what does what, and update or extend each part without unraveling the rest.

The Analogy

Here is the way I understand it: a rough analogy might be batch-processing vs programming - when chaining multiple operations, it's better to use shell scripting than to rely on spawning subprocesses directly in your source code.

Micro-services using event driven architecture

Tue, 01 Jul 2025 11:38:59 +0000

From REST to Events: Why Event-Driven Microservices Are the Upgrade You Didn’t Know You Needed

Building scalable systems today isn’t about throwing more servers at
the problem, it’s about rethinking how your services talk to each
other.

If you’ve ever been burned by tightly coupled REST calls in a microservice architecture, long chains of service-to-service calls, failed transactions halfway through, retry hell, you’re not alone.

Been there. Debugged that.
Let’s talk about the better way: event-driven architecture (EDA).

Microservices Were Supposed to Fix Everything… Right?
In theory, microservices let you:

Deploy independently
Scale granularly
Ship faster

But when your services are glued together with synchronous HTTP calls, you’re still in trouble:

Service A goes down → Service B fails too

Any latency → user feels it

You spend more time writing retries and fallbacks than business logic.

That’s when I realized: the real unlock isn’t microservices alone, it’s event-driven microservices.

What Is Event-Driven Architecture?

In event-driven systems, services don’t call each other directly.
They emit events, and other services listen and respond.

participant OrderService

participant Kafka

participant PaymentService

participant InventoryService

participant ShippingService

OrderService->>Kafka: Publish "OrderPlaced"

Kafka->>InventoryService: "OrderPlaced"

Kafka->>PaymentService: "OrderPlaced"

InventoryService->>Kafka: "InventoryReserved"

PaymentService->>Kafka: "PaymentConfirmed"

Kafka->>ShippingService: "OrderReady"

This decoupling is game-changing.

Why It Works Better

✅ Loose coupling
Services don’t know about each other.
You can add or remove consumers anytime.

✅ Resilience
One service fails? Others still run.

✅ Scalability
Scale hot paths (e.g., payment, inventory) independently.

✅ Auditability
Events are logged, you get a trail of everything that happened.

✅ Asynchronous by default
Your users don’t wait while five services call each other like it’s a WhatsApp group chat.

Event Brokers: The Real MVPs

These tools make EDA possible:

Apache Kafka — High-throughput, persistent event log (my go-to)
RabbitMQ — Queue-based messaging with strong delivery guarantees
AWS SNS/SQS — Easy serverless messaging on the cloud
Others — NATS, Pulsar, Redis Streams

Pick one based on your throughput needs, latency tolerance, and operational skillset.

Real-World Flow: Order Processing

Let’s say you place an order. Here's how the services react:

OrderService emits OrderPlaced

InventoryService listens, reserves items

PaymentService listens, processes payment

ShippingService listens, ships once inventory + payment are confirmed

No service talks to another directly. No coupling. Just clean,
reactive design.

⚠️ It’s Not All Rainbows

Yes, event-driven systems are powerful, but they’re not magic.

Here’s what to watch out for:

Eventual consistency: Not everything is instant.
Idempotency: Events can be duplicated. Handle it.
Schema evolution: Plan for versioning your events.
Debugging: Distributed tracing is a must.

Final Thoughts

If you want systems that are:

Scalable ✅

Resilient ✅

Loosely coupled ✅

Cloud-ready ✅

Then it’s time to look beyond REST.

Start small. Maybe just one async event.
Get a feel for it. Then go deeper.

Because in modern backend systems, the question isn't:

“Should I use microservices?” It's: “How are my services
communicating?”

✍️ Written by Moses Daniel Kwaknat, backend engineer, API builder, and dev who’s finally making peace with distributed systems.

Let’s talk microservices, message brokers, or how to escape REST hell

Optimistic vs. Pessimistic Concurrency in EF Core (with Table Hints)

Mon, 31 Mar 2025 10:01:25 +0000

Many experienced developers are familiar with database transactions but often overlook the key differences between optimistic and pessimistic concurrency control in Entity Framework Core (EF Core).

Understanding both approaches and how EF Core handles concurrency tokens is essential for building scalable and safe data operations, especially in systems with high transaction rates.

Concurrency control determines how multiple users or processes interact with the same data simultaneously. While optimistic concurrency is the default in EF Core, pessimistic concurrency is often misunderstood, misused, or completely ignored even by seniors engineers.

You can find the full code example at:
https://github.com/stevsharp/EfCoreExamples/tree/Efcore-Concurrency

What Is a Concurrency Token?

A concurrency token is a property that EF Core watches to detect whether a row has been modified since it was loaded. It's typically used in optimistic concurrency scenarios.

EF Core uses this token during SaveChangesAsync() to generate a WHERE clause that checks if the value is unchanged , if it has changed, a DbUpdateConcurrencyException is thrown.

How to Mark a Property as a Concurrency Token

✅ Using [ConcurrencyCheck] Attribute:

public class Product
{
    public int Id { get; set; }

    [ConcurrencyCheck]
    public int Stock { get; set; }
}

✅ Using [Timestamp] (for SQL Server rowversion column):

[Timestamp]
public byte[] RowVersion { get; set; }

✅ Fluent Configuration:

modelBuilder.Entity<Product>()
    .Property(p => p.Stock)
    .IsConcurrencyToken();

You can use any column as a concurrency token — even logical fields like LastModified, but (aka ) is the most reliable for SQL Server.

Optimistic Concurrency Control

Optimistic concurrency assumes conflicts are rare, so it doesn’t lock any data during updates. Instead, it tracks changes using concurrency tokens and throws an exception if a conflict is detected during SaveChangesAsync().

✅ When to Use

Read-heavy apps with occasional writes (e.g., blog, e-commerce frontend)

Systems where retries are acceptable

How to Implement in EF Core with a Concurrency Token

Entity:

public class Product
{
    public int Id { get; set; }
    public int Stock { get; set; }

    [Timestamp] // This is the concurrency token
    public byte[] RowVersion { get; set; }
}

Save and Handle Concurrency:

var product = await context.Products.FindAsync(1);

if (product.Stock > 0)
{
    product.Stock -= 1;

    try
    {
        await context.SaveChangesAsync();
    }
    catch (DbUpdateConcurrencyException ex)
    {
        var databaseValues = await ex.Entries.Single().GetDatabaseValuesAsync();
        ex.Entries.Single().OriginalValues.SetValues(databaseValues);
    }
}

Pessimistic Concurrency Control

Pessimistic concurrency prevents conflicts altogether by locking the rows being read or modified. This ensures no one else can change the data until the transaction is complete.

❗ Important: EF Core does not support table hints like `` in LINQ. These must be applied using raw SQL.

Example: Locking Bank Account Row

using (var context = new AppDbContext())
{
    using var transaction = await context.Database.BeginTransactionAsync();

    var account = await context.Accounts
        .FromSqlRaw("SELECT * FROM Accounts WITH (UPDLOCK, ROWLOCK) WHERE Id = {0}", 1)
        .AsTracking() // Ensures the entity is tracked for updates
        .FirstOrDefaultAsync();

    if (account != null)
    {
        account.Balance -= 100;

        await context.SaveChangesAsync();
    }

    await transaction.CommitAsync();

}

Why SaveChangesAsync() Alone Is Not Always Enough

When using FromSqlRaw(), EF Core may not track the entity correctly for updates unless explicitly instructed.

Using .AsTracking() ensures EF will detect changes.

Without it, you would need to manually mark the entity as modified:

context.Entry(account).State = EntityState.Modified;

✅ Final Thoughts

Concurrency tokens (like RowVersion) play a central role in optimistic concurrency, enabling EF Core to detect conflicts without using locks.

Optimistic concurrency is ideal for most applications where data collisions are rare and performance is key.

Pessimistic concurrency is essential in financial or high-risk systems, where consistency outweighs performance.

EF Core does not support table hints like ` in LINQ, but you can **leverage them via **`, with .AsTracking() to ensure updates are recognized.

Many developers — even experienced ones — underestimate how concurrency should be managed in EF Core. The wrong approach can result in subtle bugs, lost data, or race conditions in production.

References & Further Reading

Microsoft Docs – Handling Concurrency Conflicts in EF Core

Microsoft Docs – Concurrency Tokens

Dev.to – Understanding and Using Table Hints in SQL Server

Microsoft Docs – Raw SQL Queries in EF Core

SQL Server – Table Hints in Transact-SQL

EF Core Bulk Updates and Deletes: Performance vs Change Tracking

Mon, 24 Mar 2025 20:25:26 +0000

Entity Framework Core 7 introduced two powerful methods for bulk operations:

ExecuteUpdateAsync and ExecuteDeleteAsync.

These methods are designed for high-performance scenarios where you need to update or delete multiple records efficiently without loading them into memory. They work great, but there's one caveat: EF Core's Change Tracker is not aware of these changes.

In this article, we explore how these methods work, why they bypass the Change Tracker, and most importantly—how you can audit or track changes manually when using them.

If you're not familiar with EF Core's Change Tracker, check out this introductory guide on Change Tracking to understand how EF Core monitors entity states.

Why Use ExecuteUpdateAsync and ExecuteDeleteAsync?

These methods:

Generate direct SQL UPDATE and DELETE statements.
Do not load entities into memory, improving performance.
Bypass change tracking, which avoids unnecessary overhead.

Example Usage:

// Bulk update
await context.Users
    .Where(u => u.LastLogin < DateTime.UtcNow.AddYears(-1))
    .ExecuteUpdateAsync(setters => setters.SetProperty(u => u.IsInactive, true));

// Bulk delete
await context.Users
    .Where(u => u.IsInactive)
    .ExecuteDeleteAsync();

The Problem: No Change Tracking

Because EF Core does not load entities into memory during these operations, the Change Tracker never sees them. This means:

No automatic change detection.
No SaveChangesAsync() involvement.
No triggers for audit or domain events.

So, how do you track what was changed or deleted?

1.Auditing Options for Bulk Operations

Manual Pre-Query Before Execution

Fetch affected entities before performing the batch operation.

var affectedUsers = await context.Users
    .Where(u => u.LastLogin < DateTime.UtcNow.AddYears(-1))
    .Select(u => new { u.Id, u.Name, OldValue = u.IsInactive })
    .ToListAsync();

await context.Users
    .Where(u => u.LastLogin < DateTime.UtcNow.AddYears(-1))
    .ExecuteUpdateAsync(setters => setters.SetProperty(u => u.IsInactive, true));

// Log changes manually
foreach (var user in affectedUsers)
{
    Console.WriteLine($"User {user.Name} updated: IsInactive {user.OldValue} → true");
}

Pros:

Precise control over what you log.

Access to old values.

Cons:

Requires an extra query.

May impact performance for large datasets.

Audit Table Logging

Save the affected data into an audit table before executing the bulk operation.

Example:

await context.AuditLogs.AddRangeAsync(affectedUsers.Select(user => new AuditLog
{
    EntityName = "User",
    EntityId = user.Id,
    ChangeType = "Updated",
    ChangedAt = DateTime.UtcNow,
    Description = $"User {user.Name} was updated: IsInactive {user.OldValue} → true"
}));

await context.SaveChangesAsync();

Database Triggers

Define SQL triggers to capture changes directly in the database.

Example :

CREATE TRIGGER trg_User_Delete
ON Users
AFTER DELETE
AS
BEGIN
    INSERT INTO AuditLogs (EntityName, EntityId, ChangeType, ChangedAt, Description)
    SELECT 'User', Id, 'Deleted', GETUTCDATE(), CONCAT('User ', Name, ' was deleted') FROM deleted;
END;

Pros:

Fully automated.

Ensures auditing even if EF Core is bypassed.

Cons:

Less flexible.

Harder to manage and version control.

EF Core Interceptors (Advanced)

Capture executed SQL commands using DbCommandInterceptor.

Limitation:

You can log the SQL query but not the previous values of affected entities.

public class AuditInterceptor : DbCommandInterceptor
{
    public override InterceptionResult<DbDataReader> ReaderExecuting(
        DbCommand command, CommandEventData eventData, InterceptionResult<DbDataReader> result)
    {
        Console.WriteLine($"SQL Executed: {command.CommandText}");
        return base.ReaderExecuting(command, eventData, result);
    }
}

Registering the Interceptor :

public class ApplicationDbContext : DbContext
{
    public DbSet<User> Users { get; set; }
    public DbSet<AuditLog> AuditLogs { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        optionsBuilder.UseSqlServer("YourConnectionStringHere");
        optionsBuilder.AddInterceptors(new AuditInterceptor());
    }

Conclusion

EF Core's ExecuteUpdateAsync and ExecuteDeleteAsync offer incredible performance for bulk data operations but come at the cost of losing automatic change tracking.

If you need to audit or track changes, the best approach is to manually query the affected data before execution or use a dedicated audit system such as audit tables or SQL triggers.

Choose the method that balances performance and auditing needs for your application.

What do you think? Do you prefer performance with manual tracking, or do you rely on EF Core's change tracking? Let us know your approach!

You can find the full code example at: https://github.com/stevsharp/EfCoreExamples/tree/BulkUpdateAndDeletes

References

Microsoft Docs - EF Core 7: ExecuteUpdate and ExecuteDelete

Microsoft Docs - Change Tracking in EF Core

Microsoft Docs - Interceptors in EF Core

Community Discussion - EF Core GitHub Issue on Change Tracking and Bulk Operations

EF Core: Lazy Loading, Eager Loading, and Loading Data on Demand

Thu, 06 Mar 2025 16:31:00 +0000

When working with Entity Framework Core (EF Core), there are different strategies for loading related data from the database.

So, how do you load a collection or other related objects?

Here are the main options:

1️⃣ Eager Loading (Include)

Loads related data immediately with the main entity using .Include().

Example: Load an Order with its Customer

var order = context.Orders
    .Include(o => o.Customer) // Load related Customer
    .First();

Example: Load a Customer with Their Orders

var customer = context.Customers
    .Include(c => c.Orders) // Load related Orders
    .First();

✅Pros:

✔ Avoids multiple database queries (N+1 problem)

✔ Simple and easy to use

❌ Cons:

❌ Can load unnecessary data, increasing query size

❌ Slower if you load too many relationships

2️⃣ Explicit Loading (Entry().Reference() / Entry().Collection())

Loads related data on demand (manually) using .Entry().

Example: Load a Single Related Entity

var order = context.Orders.First();

// Load related Customer manually
context.Entry(order).Reference(o => o.Customer).Load();

Console.WriteLine(order.Customer.Name);

Example: Load a Collection of Related Entities

var customer = context.Customers.First();

// Load related Orders manually
context.Entry(customer).Collection(c => c.Orders).Load();

Console.WriteLine($"Orders count: {customer.Orders.Count}");

✅ Pros:

✔ More control over when data is loaded

✔ Avoids unnecessary queries

❌ Cons:

❌ Requires additional queries

❌ Must remember to load data explicitly

3️⃣ Lazy Loading (Requires Proxies)

Loads related data only when accessed.

Install :

dotnet add package Microsoft.EntityFrameworkCore.Proxies

Steps to Enable Lazy Loading

Enable in DbContext:

protected override void OnConfiguring(DbContextOptionsBuilder options)
{
    options.UseLazyLoadingProxies().UseSqlite("Data Source=mydb.db");
}

Make navigation properties virtual

public virtual Customer Customer { get; set; } = default!;
public virtual List<Order> Orders { get; set; } = new();

Example Usage :

var order = context.Orders.First();

// Lazy loaded automatically when accessed

Console.WriteLine(order.Customer.Name);

✅ Pros:

✔ Simplifies code (data is loaded automatically)

✔ Works without modifying queries

❌ Cons:

❌ Requires proxies (extra package)

❌ Can lead to multiple database hits (N+1 problem)

4️⃣ Projection (Select with Anonymous Types)

Loads only specific data instead of full entities.

Example: Load Only Needed Fields

var orders = context.Orders
    .Select(o => new 
    { 
        o.OrderNumber, 
        CustomerName = o.Customer.Name 
    })
    .ToList();

foreach (var order in orders)
{
    Console.WriteLine($"{order.OrderNumber} - {order.CustomerName}");
}

✅ Pros:

✔ Best for performance (loads only required fields)

✔ Reduces unnecessary data transfer

❌ Cons:

❌ Cannot track entities (only works for read-only queries)

Summary Table

Best Practices
Use Eager Loading (Include()) when related data is always needed.

Use Explicit Loading if you want manual control.

Use Lazy Loading only if necessary (it can cause performance issues).

Use Projection (Select) when you need only some fields.

Source Code

You can find the complete source code for this tutorial at:
GitHub Repository

Here are some useful references to learn more about EF Core :

Official Microsoft Documentation
Entity Framework Core Loading Related Data
https://learn.microsoft.com/en-us/ef/core/querying/related-data

Covers Eager Loading, Explicit Loading, and Lazy Loading with examples.
EF Core: Configuring Lazy Loading
https://learn.microsoft.com/en-us/ef/core/querying/related-data/lazy

Explains how to enable Lazy Loading with proxies.
EF Core: Using Include() for Eager Loading
https://learn.microsoft.com/en-us/ef/core/querying/related-data/eager

Details how to use .Include() and .ThenInclude().
EF Core Performance Considerations
https://learn.microsoft.com/en-us/ef/core/performance/

Best practices for optimizing queries and avoiding performance issues like N+1 queries.

Community & Blog Articles
EF Core Lazy Loading vs. Eager Loading - Which One Should You Use?
https://www.thinktecture.com/en/entity-framework-core/lazy-loading-vs-eager-loading/

A detailed comparison of when to use each strategy.
EF Core Best Practices: Avoiding Performance Pitfalls
https://www.michalbialecki.com/2020/11/29/entity-framework-core-performance-best-practices/

Covers when to use .Include(), lazy loading, and projection.

Microservices vs Monolithic Applications: Cybersecurity Considerations

Tue, 25 Feb 2025 08:03:27 +0000

Microservices vs Monolithic Applications: Cybersecurity Considerations

Introduction

In today's digital landscape, choosing between microservices and monolithic architectures is a significant decision for organizations. Each architectural style presents unique challenges and benefits, especially regarding cybersecurity. Understanding the implications of both can help organizations better defend against threats.

Architecture Comparison of Microservices and Monolithic Applications

Microservices architecture breaks applications into smaller, independent services, while monolithic architecture consolidates everything into a single unit. This structural difference introduces distinct cybersecurity considerations for each model.

Cybersecurity Best Practices for Microservices

When deploying microservices, it’s crucial to follow best practices to safeguard against vulnerabilities:

API Security: Secure APIs that connect services. For more information on API security principles, refer to Understanding API Security with OpenAPI: A Security Engineer’s Perspective.
Service Isolation: Each service should operate independently to minimize threat impact.
Continuous Monitoring: Implement robust logging and monitoring to detect potential security breaches in real time.

Cybersecurity Best Practices for Monolithic Applications

Monolithic applications also require stringent security measures. Key practices include:

Regular Patching: Keeping libraries and frameworks updated to mitigate vulnerabilities.
Input Validation: Ensuring all inputs into the application are validated and sanitized.
Secure Coding Practices: Educating developers on secure coding to prevent common exploits.

For insights on common vulnerabilities, refer to Understanding the OWASP API Security Top 10.

Cybersecurity Risks in Microservices Architecture

While microservices offer flexibility and speed, they also introduce potential cybersecurity risks:

Increased Attack Surface: Each microservice presents a new potential entry point for attackers.
Complex Inter-service Communication: Monitoring and securing interactions between services can be challenging.

For a comprehensive overview of vulnerabilities associated with microservices, see Understanding Vulnerabilities.

Cybersecurity Risks in Monolithic Architecture

Monolithic applications also face unique cybersecurity challenges:

Single Point of Failure: A breach can compromise the entire application.
Difficulties in Scalability: Implementing security measures can be cumbersome in large monolithic systems.

For further context on security risks, consider reading The Evolution of Internet Traffic.

Comparing Microservices and Monolithic Applications in a Cybersecurity Context

In conclusion, microservices allow for more granular security controls, but they also require vigilance to manage inter-service vulnerabilities. In contrast, monolithic applications simplify security management at the potential cost of speed and agility. Organizations must evaluate their specific needs and the associated cybersecurity implications when choosing an architecture.

Resources

API Security: OWASP REST Security Cheat Sheet
OWASP Microservices Security: OWASP Project
NIST Cybersecurity Framework: NIST.gov
MITRE ATT&CK Framework: attack.mitre.org
CISA Cyber Hygiene: cisa.gov/cyber-hygiene
OWASP Top Ten Vulnerabilities: owasp.org

Mastering Amazon S3 Replication: Same-Region vs. Cross-Region Replication -

Sun, 02 Feb 2025 09:34:30 +0000

Amazon Simple Storage Service (S3) is a powerhouse for cloud storage, offering unmatched scalability, durability, and security. Among its many features, S3 Replication plays a vital role in ensuring data availability, resilience, and compliance. AWS provides two types of replication mechanisms: Same-Region Replication (SRR) and Cross-Region Replication (CRR). While both serve to replicate objects, they cater to different business needs. Let’s dive deep into their differences, use cases, and best practices.

What is Amazon S3 Replication?

S3 Replication is an AWS feature that automatically replicates objects from one S3 bucket to another. The replication can occur within the same AWS region (SRR) or across different regions (CRR). This ensures redundancy, availability, and compliance with organizational policies.

AWS S3 Replication requires enabling Versioning on both the source and destination buckets. Once set up, it replicates newly uploaded objects asynchronously while preserving metadata, object tags, and Access Control Lists (ACLs), depending on your configurations.

Same-Region Replication (SRR)

✅ What is SRR?

SRR allows replication of S3 objects within the same AWS region. This is useful for organizations that need multiple copies of data in the same geography but within different buckets.

Use Cases

Compliance & Governance: Maintain multiple copies of data within a region to meet internal compliance regulations.

Data Tiering & Organization: Separate different teams' data while keeping it in the same region for low latency.

Backup & Disaster Recovery: Keep replicated copies in a different bucket to recover from accidental deletions or corruption.

Logging & Analytics: Send logs to a centralized bucket for analysis without moving them across regions.

⚡ Benefits

Low Latency: Since data is within the same region, replication is faster.

Cost-Effective: No inter-region data transfer costs.

Better Access Control: Different teams can access separate replicated buckets without modifying the source bucket’s policies.

⚠️ Considerations

SRR does not protect against regional failures; for true disaster recovery, CRR is needed.

There might be some added storage costs depending on data volume.

Cross-Region Replication (CRR)

✅ What is CRR?

CRR replicates objects across different AWS regions, providing geographical redundancy and enhanced disaster recovery.

Use Cases

Disaster Recovery: Store copies of critical data in another region to protect against region-wide outages.

Compliance & Regulations: Meet legal requirements for keeping data in multiple geographic locations.

Global Content Distribution: Improve performance by serving data closer to end users in different regions.

High Availability & Fault Tolerance: Maintain business continuity in case of failures in one region.

⚡ Benefits

Resilient Disaster Recovery: Data is available in another region in case of primary region failure.

Reduced Latency for Global Users: Store data closer to international customers to reduce access time.

Improved Business Continuity: Ensures uninterrupted service availability.

⚠️ Considerations

Higher Costs: Inter-region data transfer fees apply.

Increased Latency: Replication takes longer due to network distance.

Regulatory Compliance: Ensure cross-border data transfers align with local regulations.

️ Setting Up S3 Replication (SRR & CRR)

To enable replication:

Enable versioning for both source and destination buckets.

Set up IAM permissions for replication.

Create a Replication Rule specifying:

Source and destination buckets

Filter conditions (prefix, object tags, etc.)

Replication options (SRR or CRR)

Enable S3 Replication Time Control (RTC) for predictable replication times (optional).

Monitor replication progress using AWS CloudWatch and S3 Replication Metrics.

SRR vs. CRR: Which One Should You Choose?

FeatureSame-Region Replication (SRR)Cross-Region Replication (CRR)
Data Transfer CostLower (No inter-region cost)Higher (Inter-region transfer fees)
LatencyLow (Faster replication)Higher (Depends on region distance)
Disaster RecoveryNo protection from regional failuresHigh protection from region-wide failures
ComplianceUseful for internal regulationsRequired for international regulations
Global AccessLimited to one regionEnables worldwide data access

Best Practices for S3 Replication

✅ Use Lifecycle Policies: Delete older versions or move replicated data to S3 Glacier for cost efficiency.
✅ Monitor with CloudWatch: Keep track of replication success rates and failures.
✅ Use Replication Time Control (RTC): Ensure predictable replication times.
✅ Encrypt Replicated Data: Enable AWS KMS encryption to secure sensitive information.
✅ Optimize Costs: Use Intelligent-Tiering for replicated objects to reduce expenses.

Conclusion

Both Same-Region Replication (SRR) and Cross-Region Replication (CRR) are powerful features in AWS S3 that serve different business needs. SRR is ideal for compliance, backups, and data segregation, while CRR is essential for disaster recovery and global distribution. Choosing the right replication strategy depends on your use case, performance requirements, and budget.

By implementing these best practices, you can ensure optimal data availability, security, and resilience in your AWS cloud environment.

Ready to enhance your AWS storage strategy? Implement S3 replication today!

Choosing the Right Solana Wallet: Phantom vs. Solflare vs. Sollet

Mon, 20 Jan 2025 20:15:34 +0000

Solana Wallets: Choosing the Best One for Developers

As Solana continues to gain traction for its high-speed, low-cost transactions, the importance of a reliable wallet for developers cannot be overstated. Whether you’re building decentralized applications (dApps), managing SPL tokens, or exploring blockchain interoperability, choosing the right wallet impacts both your development workflow and the user experience. In this article, we’ll compare Phantom, Solflare, and Sollet wallets to help developers select the best one for their use cases.

Key Considerations for Developers

Before diving into the comparison, here are some criteria developers should prioritize when choosing a Solana wallet:

Ease of Integration: Availability of SDKs, APIs, and Web3.js support.
Developer-Friendliness: Debugging tools and testnet support.
Security: Robust security measures like seed phrase backup and private key encryption.
Multichain Support: Support for SPL tokens, NFTs, and other blockchain assets.
User Interface (UI): Intuitive UI for testing dApp functionalities.

1. Phantom Wallet

Overview

Phantom is the most popular Solana wallet, offering a sleek UI and seamless integration for dApps. It is widely regarded as the "MetaMask of Solana."

Pros

Developer Tools: Phantom supports easy integration with dApps via Solana Web3.js, making it ideal for developers building decentralized applications.
Testnet Support: Switch between mainnet, testnet, and devnet with ease, allowing developers to test features without using real funds.
NFT and SPL Token Support: Displays NFTs and SPL tokens natively, simplifying development for NFT or token-centric applications.
User-Friendly Interface: Intuitive design, making it easier to onboard users during testing phases.

Cons

Browser-Only: Phantom is available as a browser extension, limiting its usage in mobile or hardware wallet setups.
Limited Advanced Features: While great for general development, Phantom lacks some advanced debugging tools.

Ideal Use Case

For developers building user-facing dApps, Phantom offers unparalleled ease of use and seamless integration.

2. Solflare Wallet

Overview

Solflare is a versatile wallet designed for both casual users and developers. It supports browser, mobile, and hardware wallets, making it a more flexible choice.

Pros

Multiplatform Support: Available as a browser extension, mobile app, and desktop app, providing developers with multiple testing environments.
Hardware Wallet Integration: Native support for Ledger devices, offering enhanced security for development purposes.
Comprehensive Token Support: Excellent for managing SPL tokens, staking SOL, and interacting with DeFi protocols.
Advanced Features: Offers in-app staking and advanced transaction visualizations, which can be useful for debugging.

Cons

Steeper Learning Curve: The feature-rich interface can feel overwhelming for new developers.
Speed: Slightly slower UI compared to Phantom.

Ideal Use Case

Solflare is perfect for developers requiring cross-platform testing or those who value hardware wallet integration for security.

3. Sollet Wallet

Overview

Sollet is an open-source wallet primarily targeted at developers. While less user-friendly, it offers unmatched transparency and customization options.

Pros

Open Source: Ideal for developers who want to understand and customize wallet behavior.
Advanced Features: Provides raw transaction data and manual signing options, making it suitable for low-level debugging.
Custom RPC Endpoints: Easily configure RPC nodes, which is crucial for developers testing network-specific features.

Cons

Outdated UI: The user interface feels clunky and dated compared to Phantom or Solflare.
Steep Learning Curve: Beginners may find it challenging to navigate or understand the wallet’s features.
Limited Ecosystem Support: Lacks native support for NFTs and modern dApp integrations.

Ideal Use Case

Sollet is best for experienced developers who need granular control over transactions and want to work directly with Solana’s technical features.

Feature Comparison Table

Feature	Phantom	Solflare	Sollet
Ease of Integration	Excellent	Good	Moderate
Testnet Support	Yes	Yes	Yes
NFT Display	Yes	Yes	No
Hardware Wallets	Limited	Full	Full
Custom RPC Endpoints	Limited	Yes	Yes
Open Source	No	No	Yes
Mobile Support	No	Yes	No

Examples of dApps Where Wallets Excel

Phantom: Seamless integration with dApps like Magic Eden (NFT marketplace) and Orca (DeFi exchange).
Solflare: Ideal for staking SOL on platforms like Marinade Finance or interacting with DeFi apps like Saber.
Sollet: Preferred for custom integrations with raw transactions in developer-heavy projects like Solana Program Library (SPL) examples.

Beginner-Friendly Resources for Sollet

Conclusion

Each wallet serves a unique purpose in the Solana ecosystem, and the best choice depends on your specific development needs. For most dApp developers, Phantom is the go-to choice for its simplicity and integration capabilities. However, for more advanced use cases or multi-platform testing, Solflare and Sollet provide features that can significantly enhance your workflow.

By understanding the strengths and limitations of each wallet, you can choose the one that best aligns with your development goals. Let your choice of wallet be a tool that amplifies your innovation on Solana!

Right-Sizing Microservices: Harnessing Integrators and Disintegrators for Striking the Perfect Balance

Sun, 19 Jan 2025 03:36:56 +0000

Introduction

Microservices have become the go-to approach for building modern software systems thanks to their promise of scalability, flexibility, and faster development. But behind the hype lies a critical truth: microservices can create more problems than they solve if not planned and sized correctly. Poorly designed microservices can lead to confusion, unnecessary complexity, and extra overhead, making your system harder to manage over time. The result? Long-term issues like skyrocketing costs, maintenance headaches, and a system that feels more like a tangled web than a streamlined solution.

In this blog, we’ll dive into the concept of right-sizing microservices and explore how integrators and disintegrators can help you design an architecture that is scalable, efficient, and sustainable. Whether you’re just starting with microservices or looking to improve an existing setup, this article will give you the insights you need to avoid common pitfalls and get it right from the start. Let’s build smarter, not harder!

Importance of Correct Sizing

Getting the size of your microservices right is essential because both extremes — too large or too small — can lead to significant problems. When microservices are too large, they resemble monoliths, undermining the benefits of adopting a microservices architecture. Large services are harder to scale independently, making them less efficient in handling varying workloads. They also create deployment bottlenecks, as even small changes might require redeploying the entire service. This can lead to slower development cycles, increased downtime, and the loss of flexibility that microservices are supposed to provide.

On the other hand, too small microservices can result in an overly fragmented and complex system. This often increases communication overhead between services, leading to higher latency, potential performance bottlenecks, and difficulties in management. Such fragmentation can also force teams to deal with complex distributed scenarios, such as managing distributed transactions across multiple services. These scenarios add significant development and operational overhead, as ensuring consistency and reliability becomes a non-trivial challenge. Striking the right balance is critical to avoid these pitfalls and ensure a system that is not only scalable but also maintainable and efficient.

Service Disintegrators

Breaking down a service into smaller pieces is a critical decision in microservices architecture, and it’s not one to take lightly. Service disintegration should be guided by clear principles and justified by the specific needs of your system.

In this section, we’ll explore six key drivers that help determine when and why you should split a service. These drivers provide a structured framework to evaluate the need for disintegration and ensure it aligns with your architecture’s scalability, maintainability, and performance goals. By understanding these factors, you can make informed decisions that balance flexibility and simplicity, avoiding the pitfalls of over-fragmentation.

Service Scope and Function

One of the most common reasons to break a service into smaller parts is its scope and functionality. A key aspect of this is evaluating the service’s cohesion — how well its responsibilities are related to each other. A highly cohesive service focuses on a single, well-defined purpose, making it easier to maintain and understand. On the other hand, low cohesion indicates that a service is handling unrelated tasks, which can lead to confusion and make updates or troubleshooting more difficult. Size also plays a crucial role in deciding whether a service should be split. If a service grows too large, it becomes harder to test, deploy, and scale effectively.

For example, consider a “User Management” service that handles user registration, authentication, profile updates, and permissions. These tasks are all related but serve distinct purposes. By analyzing cohesion, we can see that authentication (focused solely on verifying user identity) is separate in function from profile updates (which involve personal user details) or permissions (related to access controls). Splitting authentication into its own service ensures high cohesion and keeps the responsibilities clear.

Correctly sizing services in this way also helps manage their size. The “Authentication” service, for instance, can now be scaled independently to handle login spikes without affecting profile or permissions functionality. This approach ensures that services are focused, manageable, and aligned with the principles of a scalable microservices architecture.

Code Volatility

Another key factor in determining whether to break a service into smaller parts is code volatility — the rate at which the source code changes. Services or components with high volatility often need frequent updates to add features, fix bugs, or respond to business needs. Keeping these components as independent services can reduce risks and improve maintainability.

For instance, consider a “Reporting” service that generates standard reports but also includes a component for custom report generation. If the custom reporting logic changes frequently to meet new client demands, it would make sense to extract this into a separate service. This way, the volatile component can be updated and deployed independently without affecting the stable parts of the system.

The benefits of such separation are significant. Independently deployable services mean faster updates without impacting other functionalities. Testing efforts are more focused, as the scope is limited to the specific service. Moreover, the blast radius of a failure is reduced, ensuring that issues in one service (e.g., custom reports) do not disrupt the overall system. Managing code volatility through independent services leads to a more resilient and flexible architecture.

Scalability and Throughput

When a service experiences extreme variations in scalability requirements, separating its components can significantly improve efficiency. Different parts of a service may have varying demands on resources, and splitting them allows you to scale each component independently based on its specific needs.

For example, consider a “Notification” service that sends both SMS and Email notifications. Let’s assume that SMS notifications typically generate much higher traffic than emails, especially during peak usage periods, such as promotional campaigns or transactional alerts. By separating SMS and email functionalities into distinct services, you can scale the SMS service to handle the high traffic while keeping the email service infrastructure minimal and cost-effective.

This approach offers several advantages. Only the components with higher demand — like SMS — are scaled, reducing unnecessary infrastructure provisioning for the less-demanding email service. This targeted scaling not only ensures optimal performance but also minimizes costs, making your architecture more efficient. By managing scalability and throughput through service separation, you can maintain a responsive, reliable system without overburdening your infrastructure.

Fault Tolerance

Fault tolerance is a critical factor in ensuring the reliability of your system. It refers to the ability of an application or specific functionality to continue operating even when a fatal crash occurs. Separating services based on their fault tolerance requirements helps isolate failures, preventing them from cascading across the entire system.

For example, consider a “Food Delivery” platform that has a service for placing orders and another for real-time delivery tracking. If these are tightly coupled and the delivery tracking component fails (e.g., due to a third-party API outage or high traffic), it could potentially bring down the order placement functionality as well. By separating these into independent services, users can still place orders even if the delivery tracking service experiences issues.

This separation ensures that critical operations, like order placement, continue functioning despite failures in non-critical components. Isolating such failures helps maintain a partially operational system, reduces downtime for key features, and improves user experience by minimizing the impact of outages.

Security

Security is a crucial consideration when designing microservices. Components with higher security requirements can be separated into independent services to implement specialized security measures without impacting other parts of the system.

For instance, in an “E-Commerce” platform, payment processing demands stricter security protocols compared to general catalog browsing. By separating the payment service, you can deploy it independently with advanced security features such as encryption, tokenization, and stricter access controls. Meanwhile, the catalog browsing service can operate with standard security measures, optimizing performance while maintaining safety.

This separation ensures that sensitive operations are protected with the highest level of security without overburdening less critical components. It also simplifies compliance with regulatory standards like PCI DSS for payments, making the overall system more secure and manageable. By isolating high-security components, you can better safeguard user data and reduce potential vulnerabilities in your architecture.

Extensibility

Extensibility refers to the ability to add new functionality to a service as its context or business needs evolve. A well-structured microservice allows you to introduce new features or update existing ones without overhauling the entire system or causing disruptions.

For example, consider a “Payment” service that handles various payment methods such as credit cards, PayPal, and bank transfers. As the business grows, you might need to introduce additional payment options like cryptocurrency, mobile wallets, or buy-now-pay-later services. Instead of reworking the entire payment service, you can extend its functionality by creating separate components for each payment method. This allows you to add new payment options without impacting existing ones, keeping the service flexible and scalable.

By structuring the service this way, you avoid turning it into a large, monolithic component. New payment methods can be added with minimal disruption and without requiring frequent redeployments. Extensibility ensures that the system can grow as business needs change, keeping your architecture adaptable and efficient.

Service Integrators

While breaking services apart into smaller, more manageable pieces is often beneficial, there are cases where it makes more sense to integrate services rather than separate them. In this section, we will provide guidance and justification for when to keep services together, or when it might be better not to break a service apart in the first place. Sometimes, splitting a service too early can introduce unnecessary complexity or overhead, making it harder to scale, maintain, or even evolve over time.

We’ll explore four key aspects that help determine when service integration is the best approach, including factors such as operational complexity, communication overhead, data consistency, and future scalability.

Database Transactions

Database transactions are crucial for maintaining data integrity when multiple operations need to succeed or fail as a single unit of work. Keeping related functionalities within the same service can simplify this process, avoiding the complexity and challenges of distributed transactions when multiple services are involved.

For instance, consider a “User Registration” service that handles creating user accounts and assigning default permissions. If these functionalities are split into separate services — one for account creation and another for permission management — ensuring that permissions are assigned only when the account is successfully created would require a distributed transaction. Distributed transactions are notoriously complex to implement, requiring advanced coordination mechanisms and increasing the risk of failure.

By combining these functionalities into a single service, you can manage all database operations within a single transaction. This ensures that the user account and associated permissions are either both successfully created or rolled back together, maintaining data integrity and simplifying the overall system.

Workflow and Choreography

In microservices architectures, workflows often involve multiple services working together through interservice communication. While this enables flexibility and modularity, excessive communication can lead to performance bottlenecks, increased latency, and reduced fault tolerance.

For instance, in a “Food Delivery” application, the order service might need to communicate with the payment service, the restaurant service, and the delivery assignment service. If these services are too decoupled, placing an order might require several network calls, increasing the chances of delays or failures. For example, if the delivery assignment service experiences an outage, it could block the entire order workflow, preventing users from completing their transactions.

Consolidating services that are tightly coupled in their workflow — such as the order and delivery assignment services — can simplify communication, reduce latency, and improve fault tolerance. This approach ensures that the system remains responsive and reliable even under high workloads or partial failures.

Shared Code

Shared code in microservices, typically managed through shared libraries, is often seen as a way to streamline development and maintain consistency. However, it can introduce challenges when it comes to service sizing and independence. When a shared library is updated, all dependent services need to align with the changes, which can disrupt their deployment cycles and create dependencies that counteract the autonomy of microservices.

For example, imagine a “Customer Management” system where multiple services — such as a profile service, loyalty program service, and notification service — use a shared library for customer validation logic. If the library is updated to include new validation rules, all these services must integrate the update. If one service lags in updating, it can lead to inconsistencies in how customer data is validated across the system.

When designing services, it’s important to evaluate whether shared code aligns with the desired level of autonomy. For highly independent services with distinct lifecycles, duplicating certain logic or adopting service-specific libraries might be preferable to avoid unnecessary coupling and to maintain flexibility in deployment and scaling.

Data Relationships

In a microservices architecture, the practice of having each service manage its own database ensures independence but complicates data relationships. When related data is distributed across services, maintaining relationships can become challenging, leading to potential data integrity issues and increased interservice communication.

For example, consider a “Ride-Sharing” platform. Suppose there is a driver service managing driver profiles and a ride service managing trip data. If these services are too granular, storing driver details in one database and trip data in another, operations like generating a driver’s earnings report or assigning rides based on availability require constant communication between the two services. This adds unnecessary complexity and can lead to delays, especially under high traffic conditions.

In this scenario, consolidating the driver and ride data into a single service with a shared database might simplify the architecture. This approach would ensure tighter data consistency, reduce the need for interservice communication, and improve the overall system’s responsiveness, highlighting the importance of appropriately sizing services based on data relationships.

Finding the Right Balance

Determining the optimal size for a service is a balancing act that requires analyzing trade-offs between breaking services apart and consolidating them. While smaller services offer benefits like scalability and independence, they can introduce complexity through distributed communication and management. Conversely, larger services simplify workflows and data relationships but may sacrifice flexibility and autonomy.

To find the right balance, it’s crucial to collaborate closely with business stakeholders. Understanding business priorities, such as the need for agility, fault tolerance, or cost optimization, helps inform decisions on service sizing. The goal is to align technical architecture with business objectives while ensuring the system remains scalable, reliable, and manageable over the long term.

Conclusion

In conclusion, the journey to right-sizing microservices is a nuanced process that involves carefully balancing disintegration and integration factors. While disintegrating services can provide advantages such as flexibility, scalability, and fault tolerance, it also introduces challenges like increased communication complexity and potential data integrity issues. On the other hand, integrating services offers simplicity but can reduce the system’s agility and create dependencies that hinder scaling.

Analyzing the trade-offs between disintegration drivers and integration drivers is the secret to getting service granularity right. By collaborating closely with business stakeholders, understanding their priorities, and evaluating the specific needs of each service, you can strike the right balance that ensures a scalable, maintainable, and cost-effective architecture. With the right approach, microservices can provide the benefits they promise without the complexities that come from poor sizing decisions.