Coder Legion - Recent posts tagged eventdriven

Pub/Sub Isn't Event-Driven Architecture: Here's What Most Teams Miss

Fri, 23 Oct 2026 14:38:21 +0000

Pub/Sub is a messaging pattern. Event-Driven Architecture is an architectural discipline. Every EDA implementation may use Pub/Sub, but not every Pub/Sub implementation is truly Event-Driven Architecture.

For many years, I would have agreed with the statement that Pub/Sub equals Event-Driven Architecture (EDA).

Why? Because early discussions around EDA focused primarily on three characteristics:

Real-time communication
Publish/Subscribe messaging
Asynchronous processing

At that time, implementing a Pub/Sub broker was often considered synonymous with implementing EDA.

Today, however, the answer is no.

As organizations have adopted EDA at scale, the discipline has evolved from a messaging concept into a comprehensive architectural practice. Modern EDA encompasses far more than simply publishing and subscribing to messages.

📡EDA Is More Than Messaging

A mature Event-Driven Architecture consists of multiple foundational elements:
📝Event Definition

Events must be clearly defined and categorized.

Examples include:
Business Events: CustomerRegistered, OrderPlaced, PaymentCompleted
System Events: FileUploaded, ServiceStarted, DatabaseUpdated, EOD

Without a consistent event model, organizations quickly end up with thousands of loosely defined messages that are difficult to understand and maintain.

👤Event Ownership

Just like data, events are business assets.

Every event should have a clearly identified owner responsible for:
Event definition
Schema management
Versioning
Lifecycle governance

When nobody owns an event, everyone assumes somebody else does.
📏Event and Queue Standards

Topic (event) and queue naming conventions form the foundation of a scalable EDA platform.
A well-defined naming structure enables:
Event discoverability
Consistent routing
Easier troubleshooting
Better governance
Without standards, the event landscape quickly becomes chaotic.

🔀Event Flow Design

EDA is not merely about sending messages.

It is about how events collaborate to achieve business outcomes.

Modern architectures require a clear understanding of:
Event relationships
Event choreography
Event dependencies
Business process visibility

While services remain loosely coupled, the overall event flow must still be intentionally designed.

🛡️Event Governance

Governance is the difference between a sustainable platform and an uncontrolled messaging ecosystem.

Effective governance covers:
Schema management
Data classification
Security controls
Access management
Event lifecycle management
Compliance and auditing

Governance should be embedded throughout the entire lifecycle, from design through implementation and operations.

📬Pub/Sub Is Only One Pattern Within EDA

Pub/Sub is an important communication pattern, but it is only one building block of a broader architecture.
A Pub/Sub platform without governance, ownership, standards, and event design is simply a collection of disconnected messages.

In such environments, organizations commonly encounter problems such as:
⚠️Consumer Failures

Consumers may crash when:
Unexpected event types are received
Event schemas change without coordination
Required fields are removed or modified
Data Exposure Risks

Sensitive information, including Personally Identifiable Information (PII), may be unintentionally delivered to systems or teams that should not have access to it.
Message Loss or Misrouting

Poorly governed subscriptions can result in:
Events being consumed by unintended systems
Important messages being ignored
Business processes becoming unreliable
Rising Infrastructure Costs

Unnecessary event distribution increases:
Network traffic
Broker workload
Infrastructure consumption
Operational costs
What begins as architectural flexibility can quickly become architectural sprawl.

💡 Final Thoughts

Many organizations begin their event-driven journey by deploying a message broker and enabling Pub/Sub communication. While this is an important first step, it is only the foundation.
The real challenge is not moving events between systems. The challenge is managing events as long-lived business assets.

Without clear ownership, governance, standards, and lifecycle management, organizations often end up with an event ecosystem that is difficult to understand, secure, and evolve. Teams gain asynchronous communication, but lose visibility, consistency, and control.

Successful Event-Driven Architecture is not measured by how many events are produced each day. It is measured by how effectively those events enable business capabilities while remaining discoverable, governed, and trusted across the enterprise.

In other words:
A message broker can make your systems event-enabled.
Governance, ownership, and standards make your organization truly event-driven.

❓ What do you think? Have you seen organizations mistake Pub/Sub implementation for Event-Driven Architecture? What governance challenges emerged as event adoption grew?

This typically drives significantly more comments and discussion from architects, developers, and integration professionals.

🚦 EDA Without Governance: A Fast Track to Enterprise Chaos

Thu, 15 Oct 2026 16:03:25 +0000

In today's digital enterprise, Event-Driven Architecture (EDA) has become a preferred architectural pattern for building scalable, loosely coupled, and real-time systems. By enabling applications to publish and subscribe to events independently, EDA provides the flexibility and agility required to support modern business demands.

However, there is a critical truth that many organizations discover only after their event ecosystem starts to grow:

⚠️ The success of EDA depends less on technology and more on governance.

Without proper governance, an event bus can quickly evolve into a chaotic integration landscape where dependencies are unclear, ownership becomes fragmented, and changes introduce widespread downstream impacts.

🌐 EDA Is More Than Event Distribution

EDA is fundamentally a distribution design that allows applications to publish and consume events across the enterprise.

✅ New publishers can share information with multiple consumers.
✅ New consumers can subscribe to existing event streams.
✅ Applications remain loosely coupled and independently deployable.

But this flexibility comes with responsibility.

An event published today may be consumed by numerous applications, either directly or indirectly. As adoption grows, every design decision becomes part of a shared enterprise contract.

Without governance, what starts as flexibility can quickly become complexity.

⚠️ The Hidden Risk of Uncontrolled Growth

When onboarding new applications into an event-driven ecosystem, governance standards and processes should be established before implementation begins.

Without these controls:

🔴 Topic structures become inconsistent.
🔴 Applications subscribe to unnecessary events.
🔴 Event routing becomes difficult to understand and maintain.
🔴 Sensitive data may be exposed to unauthorized consumers.
🔴 Small changes trigger large-scale application rework.

The impact is not limited to event broker configurations.

A seemingly simple change to an event topic, payload structure, or routing pattern may require modifications across multiple consuming applications, testing cycles, operational procedures, and business processes.

In large-scale enterprises, a single event redesign can affect dozens of interconnected systems.

🔄 Governance Must Cover the Entire Lifecycle

EDA governance is not a one-time architecture review.

It should span the entire delivery lifecycle:

📋 Design
👨💻 Development
🧪 Testing
🚀 Deployment
⚙️ Production Operations

Any governance gap can result in:

❌ Increased technical debt
❌ Integration failures
❌ Security and compliance risks
❌ Operational complexity
❌ Higher maintenance costs

The most successful event-driven organizations balance architectural flexibility with disciplined governance.

🎯 Key Areas of EDA Governance

🏷️ 1. Topic Naming Standards

Topics are the foundation of event discoverability and maintainability.

Every topic should follow a mandatory naming standard that clearly identifies:

✅ Business domain
✅ Business capability
✅ Event type
✅ Environment

A consistent naming convention helps architects, developers, and support teams quickly understand event purpose and ownership.

Good governance starts with naming things correctly.

📬 2. Queue Subscription Governance

Applications should subscribe only to events that are required for their business functions.

Governance should ensure:

✅ Controlled subscription approval
✅ Consumer registration and ownership
✅ Dependency visibility
✅ Periodic subscription reviews

This prevents unnecessary event consumption and reduces future change impacts.

Not every application needs every event.

🔀 3. Event Flow Design

A well-designed event flow enables systems to evolve without disrupting the broader ecosystem.

Governance should enforce:

✅ Standard routing principles
✅ Clear producer and consumer responsibilities
✅ Traceable event paths
✅ Flexible addition and removal of components

A properly governed event flow allows organizations to grow their architecture while maintaining control and predictability.

Flexibility should be intentional, not accidental.

🔐 4. Data Governance and Access Control

Not every consumer should have access to every event.

Organizations must ensure applications can subscribe only to authorized events, especially when they contain:

🔒 Personally Identifiable Information (PII)
🔒 Customer information
🔒 Financial transactions
🔒 Business-sensitive data

Strong governance ensures:

✅ Data privacy
✅ Regulatory compliance
✅ Security controls
✅ Consumer authorization

The principle should be simple:

🔑 Subscribe only to the data you are authorized to consume.

🏆 Governance Determines EDA Success

Many organizations focus heavily on event brokers, event meshes, streaming platforms, and messaging technologies.

While these technologies are important, they are not what ultimately determines the success of an Event-Driven Architecture.

The true differentiator is governance.

Well-defined standards, ownership models, review processes, security controls, and lifecycle management enable organizations to scale their event ecosystem confidently and sustainably.

Without governance, flexibility becomes chaos.

With governance, EDA becomes a strategic enabler of enterprise agility, innovation, and business growth.

💡 Final Thought

🚦 EDA without governance is simply message transportation.

🚀 EDA with governance becomes sustainable enterprise architecture.

The question is not whether your organization should adopt Event-Driven Architecture.

The real question is:

Do you have the governance needed to keep it from becoming enterprise chaos?

Stop Treating EDA as Just Pub/Sub. Here’s How to Handle Request/Reply

Tue, 13 Oct 2026 05:13:00 +0000

⚡ EDA ≠ Just Pub/Sub: Building Flexible Request/Reply Journeys Through Event Choreography

🚀 Exciting Update: Incredibly honored that this deep dive was selected for the HackerNoon Science Tech Brief within 12 hours of its original release! Sharing it here with the Coder Legion community to hear your thoughts on EDA patterns.

Note: The complete, ready-to-use article template covering the core dilemma of synchronous request/reply flows, the multi-payment processing blueprint with topic naming standards, zero-routing choreography principles, and engagement prompts for the community is fully prepared for your publication.

⚡ The Hidden Power of EDA: Building Flexible Request/Reply Journeys Through Event Choreography

When engineers discuss Event-Driven Architecture (EDA), the conversation typically centers on asynchronous pub/sub patterns, fire-and-forget processing, and real-time streaming. We praise EDA for decoupling our systems, but we often hit a wall when a core business process inherently demands a synchronous request-reply interaction.

A common misconception is that EDA is poorly suited for request-reply workflows. The standard instinct is to fall back on direct, synchronous HTTP/gRPC service calls between microservices. However, this introduces tight coupling, making it difficult to scale, evolve, or independently deploy components.

🎯 The Challenge

In traditional request/reply designs, systems can become tightly coupled through direct service calls, making it harder to scale, evolve, or introduce new capabilities into the workflow.

A well-designed event-driven choreography offers an alternative approach.

It enables an end-to-end business process to flow seamlessly:

Client Request
Backend Services
Multiple Processing Components
Business Decisions
Final Response to Client

Regardless of how many systems participate in the journey.

As a result, EDA becomes more than a mechanism for event distribution. It becomes a powerful foundation for building scalable and flexible request/reply workflows.

💳 Example: Multi-Payment Processing Platform

Let's use a payment platform that supports multiple payment methods as an example:

Bank Transfer
FPS
PPS
EFT

Although these payment methods follow different processing paths, they can share the same event-driven framework.
📌 Step 1: Establish a Well-Defined Topic Naming Standard

The well-defined topic structure with mandatory fields and optional fields. The mandatory part should include at least four fields: information domain, information subdomain, message type, and schema version.

📌 Step 2: Use the Naming Standard as the Event Flow Guideline

Events may travel through many systems during their lifecycle.
Although different systems may interpret or process the event
differently, the original business context must remain intact.
The topic representing the event should remain consistent with the
event's original business purpose.

On each event flow, the mandatory parts (red and blue colors) are placed in the message header and brought forward to all subsequent messages in the flow. This part should not be changed throughout the lifetime of the event flow. For each component, the output topic will be the input messages’ mandatory part plus this component function, plus the process status. It is very easy to subscribe to all corresponding events within the same event flow by using a wildcard.

🧩 Loosely Coupled Processing Components

An important characteristic of this design is that individual processing components do not contain built-in routing logic.

Each component simply:

📥 Consumes events it is interested in
⚙️ Performs its business function
📤 Publishes the next event

The component has no knowledge of:

❌ Which flow it belongs to
❌ Which system produced the event
❌ Which system will consume the next event

This significantly improves reusability and maintainability.

✅ The Result

This architecture enables a set of common reusable components to support multiple request/reply workflows across different channels and payment methods.

Key Outcomes

Consistent event schema across multiple use cases
All downstream events inherit and extend the original business context
Additional analytical services can be introduced without impacting existing applications
Customer behavior and application activities can be monitored in real time
New processing components can be added or removed easily
Existing applications require no code changes
Most changes can be achieved through topic or queue subscription configuration alone

💡 Final Thought

One of the biggest misconceptions about Event-Driven Architecture is that it is only suitable for asynchronous event broadcasting.

In reality, when combined with strong event governance, consistent topic standards, and event choreography, EDA can also provide an elegant solution for complex request/reply interactions.

The result is an architecture that is:

Loosely Coupled
Highly Scalable
Easily Extensible
Business-Driven
Future-Proof

How does your organization handle request/reply patterns in an event-driven landscape? I'd love to hear your experiences and perspectives in the comments.

Asistente Zyro

Sun, 11 Oct 2026 00:34:39 +0000

¿Qué es Asistente Zyro?

Asistente Zyro es un agente autónomo de IA diseñado para la automatización de procesos B2B. A diferencia de los chatbots convencionales, Zyro actúa como un operador multicanal asíncrono capaz de crear automatizaciones recurrentes e interactuar con el mundo físico (telefonía) mediante infraestructura Event-Driven.

Características y Capacidades Clave

Creación de Automatizaciones en Tiempo Real: Puedes pedirle a Zyro que clasifique documentos en tu Google Drive. El agente interactúa contigo mediante lenguaje natural y botones dinámicos en el chat para crear reglas recurrentes (por ejemplo, analizar correos entrantes diariamente y guardar adjuntos específicos en carpetas auto-creadas).
Integración Telefónica Real (PBX) y Negociación: Zyro no solo envía correos. Puede realizar llamadas asíncronas a clientes para dar seguimiento a un documento, negociar horarios de reunión por voz y, tras colgar, reportar el nuevo estado y agendar la cita en tu calendario.
Orquestación Multiusuario por Correo: Al poner a *Emails are not allowed* en copia en un hilo de correo, el agente analiza los calendarios de los involucrados, encuentra huecos comunes y propone la fecha ideal, liberando a tu equipo del caos administrativo.
Flujos de Trabajo de Extremo a Extremo: En un solo prompt, Zyro puede: generar un borrador, convertirlo en Google Doc, enviarlo por Gmail, llamar al destinatario y actualizar tu Google Calendar.
Seguridad Zero-Retention: La IA nunca almacena ni posee tus datos. Toda la ejecución de código (lectura de correos, modificación de Drive) se realiza inyectando tokens OAuth2 nativos de Google Workspace mediante middleware, garantizando seguridad de nivel corporativo.

¿Para quién es el software?

Zyro está diseñado para empresas, agencias y consultorías que necesitan escalar su productividad sin aumentar su carga administrativa. Es la solución para directivos que buscan delegar las tareas de seguimiento (llamadas, agendamiento de reuniones, clasificación documental) a un sistema predecible, seguro y sin curva de aprendizaje.

📊 Can Kafka Handle Every Integration Use Case? Understanding the Architectural Trade-offs

Wed, 07 Oct 2026 13:39:23 +0000

Compared with traditional queue-based messaging platforms, Kafka is primarily designed for event streaming. It excels at distributing data from producers to one or many consumers at massive scale. However, unlike many traditional messaging platforms, Kafka intentionally minimizes broker-side routing and content-based filtering capabilities. Message selection is typically achieved through topic design, partitioning strategies, stream processing, or consumer-side filtering rather than broker-managed routing.

This raises an important architectural question:

❓ Can Kafka be used for all integration and messaging use cases?

The answer is yes, but organizations should understand and address several architectural and operational challenges before adopting Kafka as a universal messaging platform.

📈 1. Scaling Considerations

Kafka's distributed architecture allows horizontal scaling by adding additional brokers to support increasing workloads and client connections. However, consumer scalability is fundamentally tied to the number of partitions within a topic.

Key considerations:

✅ One partition can be consumed by only one consumer within the same consumer group.
✅ One consumer can consume multiple partitions.

⚠️ The maximum parallelism of a consumer group is limited by the partition count.

As a result, partitioning strategy becomes a critical design decision. Under-partitioning can limit future scalability, while over-partitioning may introduce unnecessary operational overhead, metadata overhead, replication traffic, and recovery times, making partition planning a long-term architectural decision.

Partition planning should be treated as a long-term architectural decision, with future scalability requirements carefully considered during initial design.

🛡️ 2. Reliability and Data Durability

Kafka provides flexible durability options through replication and acknowledgment settings.

Publishers can choose different acknowledgment modes, balancing performance against durability requirements.

This flexibility offers advantages, but it also introduces governance challenges:

⚠️ Different applications may adopt different reliability settings.
⚠️ Organizations need governance and standards to ensure mission-critical applications meet reliability requirements.

In addition, replication significantly reduces the risk of data loss, but durability guarantees ultimately depend on replication factors, acknowledgment settings, in-sync replica requirements, and storage configuration. Kafka's design prioritizes minimizing the probability of data loss while maintaining high throughput. Kafka achieves durable storage through persistent logs and replication. While Kafka leverages the operating system page cache to maximize throughput, durability ultimately depends on replication, acknowledgment settings, storage configuration, and broker availability. These mechanisms provide strong durability characteristics, although architects should carefully evaluate configurations based on business requirements.

Architects should therefore carefully assess durability requirements and select appropriate replication, acknowledgment, and storage configurations.

🔄 3. Consumer Group Rebalancing

Consumer groups provide scalability and high availability, but they come with an operational cost.

Whenever a consumer joins, leaves, fails, or recovers within a consumer group, Kafka may initiate a rebalance process.

During rebalancing:

Partition ownership is reassigned
Consumer-to-partition mappings are recalculated.
Message consumption may be temporarily paused.

For dynamic cloud-native applications that frequently scale in and out, excessive consumer group rebalancing can disrupt message processing and increase latency. By default, Kafka does not guarantee partition ownership stability during rebalances. To minimize disruption, consider using the Cooperative Sticky Assignor or Static Membership.

⚙️ 4. Group Coordinator Bottlenecks

Kafka's data plane is highly scalable and typically handles large volumes of producers and consumers efficiently.

However, consumer groups rely on a control-plane component known as the Group Coordinator.

The Group Coordinator is responsible for:

📌 Managing group membership
📌 Handling consumer rebalancing
📌 Processing heartbeats
📌 Managing offset commits and retrievals
📌 Enforcing group protocols and coordination logic

In very large deployments with many consumer groups and frequent rebalances, Group Coordinator load can become an important operational consideration. When many consumer groups rebalance simultaneously, Group Coordinator load may increase significantly, potentially leading to longer rebalance times and higher consumer recovery latency.

🔀 5. Message Distribution Models

Traditional message brokers and Kafka use fundamentally different message distribution approaches.

Message Bus Model

✅ Multiple consumers can subscribe to the same non-exclusive queue.
✅ The broker distributes messages among available consumers.
✅ Consumer scaling can be automated using tools such as Kubernetes Event-Driven Autoscaling (KEDA) based on queue depth.

Kafka Model

✅ Messages are distributed across partitions.
✅ Each partition is processed by only one consumer within a consumer group.
✅ Scaling is achieved through adding partitions and consumers.

However:

⚠️ Message distribution occurs at the partition level.
⚠️ New messages continue to be sent to assigned partitions regardless of individual consumer performance.

If one consumer becomes significantly slower than others, backlog accumulation can occur on specific partitions, resulting in uneven processing rates across the consumer group.

Kafka has introduced the Share Group model (KIP-932), which aims to provide queue-like work-distribution semantics similar to traditional non-exclusive queues. However, the feature is still evolving and has not yet reached the same maturity level as established enterprise messaging platforms.

🌐 6. Network Bandwidth Consumption

One of the key differences between Kafka and many traditional message buses is the absence of native broker-side routing and filtering.

In many messaging platforms, consumers receive only messages that match specific routing rules.

With Kafka:

📥 Consumers typically subscribe to topics.
📥 Kafka favors topic-based distribution. As consumer requirements diverge, additional topics, stream processing, or filtering may be needed, increasing operational overhead.
📥 When consumer requirements diverge significantly, organizations may need additional topics, stream processing, or consumer-side filtering, potentially increasing operational complexity and network traffic.

For environments with large event volumes and diverse consumer requirements, this approach may increase:

📡 Network bandwidth usage
🖥️ Consumer processing overhead

Increasing the number of Kafka topics is one way to address bandwidth and scalability challenges. However, this approach can significantly increase operational overhead and application complexity. Architects should carefully evaluate topic design and event segmentation strategies to minimize unnecessary data movement and avoid excessive topic proliferation.

📋7. Message Ordering vs Scalability

One of Kafka's key strengths is its ability to preserve message order, but this comes with an important architectural trade-off.

Kafka guarantees message ordering only within a partition.

To increase processing throughput and consumer parallelism, organizations often increase the number of partitions. However, as partitions increase, maintaining a single global order across all messages becomes more difficult.

📦 8. Topic and Metadata Management

In very large Kafka deployments, metadata size and frequent metadata updates can increase client startup time, subscription management overhead, and cluster coordination activity. While Kafka is designed to scale efficiently, excessive topic and partition growth can create additional operational and management challenges.

🎯 Final Thoughts

Kafka is an exceptional event-streaming platform and has become a popular foundation for modern event-driven architecture. However, "one platform for every use case" is not always the optimal architectural approach.

The key question is not whether Kafka can support a use case, but whether it is the most appropriate tool for that use case.

Successful architecture decisions require balancing:

📈 Scalability
🛡️ Reliability
⚙️ Operational Complexity
🔄 Consumer Distribution Requirements
🌐 Network Efficiency

High-scale Kafka environments require strong operational observability. Consumer lag, partition imbalance, replication health, broker utilization, and rebalancing behavior must be continuously monitored to ensure reliable service operation.

However, Kafka is intentionally designed with relatively simple architecture to deliver high-performance event streaming at scale. As a result, it does not provide the rich routing and message filtering capabilities commonly found in traditional message buses.

Understanding these trade-offs enables architects to select the right messaging paradigm for the right business problem rather than forcing every workload into a single technology pattern.

💡 Kafka can support most integration workloads, but its architectural model introduces trade-offs that architects must understand. The most successful implementations are those that align Kafka's strengths with business requirements, rather than assuming that a streaming platform is the optimal solution for every messaging scenario.

Building Event-Driven Autonomous Agents (Zyro)

Wed, 07 Oct 2026 11:36:23 +0000

Hi everyone at CoderLegion. The leap from simple API wrappers to actual autonomous agents in production presents massive architectural challenges—especially when dealing with long-running tasks. I wanted to share a real-world demo of how we handle this in the backend of Zyro Workspace.

In this video, I put the system to the test with an asynchronous, multi-stage workflow. Here is how our architecture handles the flow under the hood:

Inversion of Control (IoC) and Zero-Retention:
I ask the assistant to take its draft from the whiteboard, turn it into a Google Doc, and send it via email. The AI never "owns" my credentials or executes code directly. Our Node.js middleware intercepts the JSON payload with its intent, validates the security, and executes the action by injecting my native OAuth2 token. The security boundaries are enforced by the infrastructure, not by a fragile System Prompt.
True Asynchrony (Event-Driven):
The real architectural challenge comes when I ask Zyro to call the client on the phone to confirm receipt of the email and schedule a meeting. Keeping an HTTP connection open while a human answers and negotiates would destroy any server's resources. Here we apply strict asynchrony: Zyro dispatches the task to our PBX (telephone switchboard) and frees the main thread (it "goes to sleep").
State Management and Self-Adjustment -:
When the physical phone call ends, the PBX fires an asynchronous Webhook that "wakes up" Zyro by injecting the transcript. As seen in the video, the client rejects the Tuesday 10:00 appointment and negotiates for Wednesday at 11:00. The agent evaluates the new state autonomously, informs me via chat, and properly registers the final event in my Google Calendar.

This entire flow is backed by our Context Footprint Management engine. To survive such a long recursive loop (text generation, API consumption, and call transcripts) without overflowing the LLM's context window, the system purges and compacts its memory milliseconds before each new inference.

The technical challenge of autonomous agents isn't making them sound smart, but building the guardrails (circuit breakers) and asynchronous state management that prevent the model from collapsing.

I'd love to hear your thoughts: How are you managing long-running tasks and tool-calling loops in your AI agents in production?

Building Fault-Tolerant, Event-Driven Kafka Pipelines in Go: Reliable Reprocessing & Dead Letter Queues

Sat, 26 Sep 2026 10:35:10 +0000

A practical guide to building reliable event-driven systems in Go using Apache Kafka. Learn how to implement tiered retry strategies with delayed reprocessing, route permanently failed messages to dead letter queues in Golang with Sarama.

Prerequisites

What do you need to follow along?

Working knowledge of Golang.
Go & Docker installed on your PC.

What is an Event-Driven Architecture?

An Event-Driven Architecture (EDA) is a design approach where services communicate by producing and responding to events. Each service operates independently, producing or reacting to events as they happen.

What are Events?

An event is a record of something that has happened in a system, typically representing a state change or a significant action. An event contains data (payload) describing what happened. An example of an event could be:

A user signing up for a service.
A user placing an order in your system.

Components of an Event-Driven Architecture

To understand how events flow through a system, we need to know three key players:

Event Producers: They are the sources of events. They generate and publish events like signup events, order placed events, etc. Producers generate events and transmit them to the rest of the system. They do not know who is listening for or handling the events.

Event Brokers: They sit between producers and consumers, decoupling them so neither needs a direct connection to the other. Brokers receive event messages, maintain their chronological order, make them available for consumption, and route them to the right consumers. Apache Kafka is an example of an event broker, and it's the one we'll use throughout this guide.

Event Consumers: They handle the processing tasks. They listen on event channels and react when an event they are subscribed to is published, then they process the event, which can include making API calls, updating a database, triggering other events, or logging information.

The Complete Flow

With those three pieces in place, the flow of an EDA looks like this:

When an app or service performs an action that another part of the system needs to know about, it publishes a new event to the broker.
The broker receives the event, maintains the event's order relative to others in the stream, and then routes it to the appropriate consumer.
An event consumer then ingests the message in real time or at a later relevant instance and processes it to trigger another action.

Apache Kafka, Defined

Apache Kafka is an open-source, distributed, event-streaming platform used to publish, store, process, and consume real-time data streams. It is based on a publish-subscribe messaging model and is designed to support fault-tolerant, scalable, high-throughput, and low-latency data pipelines, event-driven applications, and stream-processing systems.

Key Apache Kafka Components

Messages: A unit of data comprised of two parts: a key and a value. The value is the actual content being sent, while the key identifies what the message relates to.
Producers: Applications that publish events or messages to Kafka topics. A producer can publish to one or more topics and can optionally choose the partition that stores the data.
Topics: A topic is a named channel or routing mechanism through which events are published and delivered to consumers. Applications write to and read from topics.
Partitions: Subdivisions of a topic that distribute records across multiple brokers. Partitions let Kafka scale horizontally while preserving record order within each partition.
Brokers: Kafka servers that store topic partitions, handle client requests, and replicate data across the cluster for fault tolerance.
Consumers: Applications that subscribe to topics and read records from partitions.
Consumer Groups: Allow multiple consumers to work together to process records from a topic.
Offsets: Unique sequential identifiers assigned to records within a partition. Consumers use offsets to track their reading position and replay data when needed.

Together, these components allow organizations to reliably stream, process, and distribute high-volume data in real time across complex, distributed environments.

Setting Up Kafka Locally with Docker

Before we write any Go code, we need a running Kafka cluster. We will use Docker Compose to spin up Kafka locally:

services:
  broker:
    image: apache/kafka:latest
    hostname: broker
    container_name: broker
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      CLUSTER_ID: 59Uj60pTdxFKv5bUMDYZFA
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:29093
      KAFKA_LOG_DIRS: /tmp/kraft-combined-logs
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

Since the introduction of KRaft, Kafka no longer requires Apache ZooKeeper® for managing cluster metadata. It uses Kafka itself instead. One advantage of the new KRaft mode is that you can have a single Kafka broker to handle both metadata and client requests in a small, local development environment. The docker-compose.yml file for this tutorial uses this approach, leading to faster startup times and simpler configuration. Note that, in a production setting, you'll have distinct Kafka brokers for handling requests and operating as a cluster controller.

Start the cluster with:

docker compose up -d

Defining Topics and Consumer Groups

Before we build the producer and consumer, let's define the topics and consumer groups that both sides of our pipeline will share. We will keep these in a separate pkg/topics package so both the producer and consumer can import them:

// pkg/topics/topics.go

package topics

type Topics string

const (
	ORDERS_CREATED Topics = "orders_created"
	RETRY_5M       Topics = "retry_5m"
	RETRY_20M      Topics = "retry_20m"
	RETRY_40M      Topics = "retry_40m"
	ORDERS_DLQ     Topics = "orders_dlq"
)

func (t Topics) String() string {
	return string(t)
}

type ConsumerGroup string

const (
	ORDERS_MAIN_GROUP      ConsumerGroup = "orders_main_group"
	ORDERS_5M_RETRY_GROUP  ConsumerGroup = "orders_retry_5m_group"
	ORDERS_20M_RETRY_GROUP ConsumerGroup = "orders_retry_20m_group"
	ORDERS_40M_RETRY_GROUP ConsumerGroup = "orders_retry_40m_group"
)

func (t ConsumerGroup) String() string {
	return string(t)
}

We have a main topic for fresh orders, three retry topics with increasing delays (5 minutes, 20 minutes, 40 minutes), and a dead letter queue (DLQ) for messages that have exhausted all retry attempts. The consumer groups map one-to-one with these topics so each tier has its own independent group.

Creating the Producer

Now that Kafka is running and our shared definitions are in place, let's build the producer side of our pipeline. The producer is responsible for receiving an order request via a REST API and publishing that event to our Kafka broker.

We'll be using Sarama throughout this guide, Sarama is an MIT-licensed Go client library for Apache Kafka.

Connecting to Kafka

First, we need a function that establishes a connection to our Kafka broker and returns a synchronous producer:

func ConnectProducer(brokersUrl []string) (sarama.SyncProducer, error) {
	config := sarama.NewConfig()

	sarama.MaxRequestSize = 100 * 1024 * 1024
	sarama.MaxResponseSize = 100 * 1024 * 1024
	config.Producer.MaxMessageBytes = 100 * 1024 * 1024

	config.Producer.Return.Successes = true
	config.Producer.RequiredAcks = sarama.WaitForAll
	config.Producer.Idempotent = true
	config.Producer.Retry.Max = 3
	config.Net.MaxOpenRequests = 1

	conn, err := sarama.NewSyncProducer(brokersUrl, config)
	if err != nil {
		return nil, err
	}

	return conn, nil
}

A few things worth noting here. RequiredAcks = sarama.WaitForAll means the producer will wait for all in-sync replicas to acknowledge the message before considering it successfully sent. We also set Idempotent = true to prevent duplicate messages in case of retries, and MaxOpenRequests = 1 is required for idempotent producers to guarantee ordering.

Publishing Messages

Next, our publisher function which produces a message to our broker:

func PushOrderToQueue(producer sarama.SyncProducer, topic string, message []byte) error {

	msg := &sarama.ProducerMessage{
		Topic: topic,
		Value: sarama.StringEncoder(message),
	}

	partition, offset, err := producer.SendMessage(msg)
	if err != nil {
		return err
	}

	fmt.Printf("Message is stored in topic(%s)/partition(%d)/offset(%d)", topic, partition, offset)
	return nil
}

The REST API Entry Point

We will use a basic REST API POST request to handle incoming user requests that will then trigger our producer to publish an event to the broker:

func main() {

	brokersUrl := []string{"localhost:9092"}

	producer, err := ConnectProducer(brokersUrl)
	if err != nil {
		log.Fatalf("Failed to initialize Kafka producer: %v", err)
	}
	defer producer.Close()

	router := gin.Default()

	api := router.Group("/api/v1")

	api.POST("/order", createOrderHandler(producer))

	router.Run()
}

The Order Handler

Now our createOrderHandler, which will produce the event on success:

type Order struct {
	Text string `form:"text" json:"text"`
}

func createOrderHandler(producer sarama.SyncProducer) gin.HandlerFunc {
	return func(c *gin.Context) {
		topic := topics.ORDERS_CREATED.String()
		order := new(Order)

		if err := c.ShouldBindJSON(&order); err != nil {
			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
			return
		}

		orderInBytes, err := json.Marshal(order)
		if err != nil {
			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
			return
		}

		err = PushOrderToQueue(producer, topic, orderInBytes)
		if err != nil {
			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
			return
		}

		c.JSON(http.StatusOK, gin.H{
			"success": true,
			"message": "Order pushed successfully",
			"order":   order,
		})
	}
}

Complete Producer Code

Here is the full producer code with everything wired together:

// producer/producer.go

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"net/http"

	"github.com/IBM/sarama"
	"github.com/gin-gonic/gin"
	"github.com/umohsamuel/edd-kafka-golang/pkg/topics"
)

type Order struct {
	Text string `form:"text" json:"text"`
}

func main() {

	brokersUrl := []string{"localhost:9092"}

	producer, err := ConnectProducer(brokersUrl)
	if err != nil {
		log.Fatalf("Failed to initialize Kafka producer: %v", err)
	}
	defer producer.Close()

	router := gin.Default()

	api := router.Group("/api/v1")

	api.POST("/order", createOrderHandler(producer))

	router.Run()
}

func ConnectProducer(brokersUrl []string) (sarama.SyncProducer, error) {
	config := sarama.NewConfig()

	sarama.MaxRequestSize = 100 * 1024 * 1024
	sarama.MaxResponseSize = 100 * 1024 * 1024
	config.Producer.MaxMessageBytes = 100 * 1024 * 1024

	config.Producer.Return.Successes = true
	config.Producer.RequiredAcks = sarama.WaitForAll
	config.Producer.Idempotent = true
	config.Producer.Retry.Max = 3
	config.Net.MaxOpenRequests = 1

	conn, err := sarama.NewSyncProducer(brokersUrl, config)
	if err != nil {
		return nil, err
	}

	return conn, nil

}

func PushOrderToQueue(producer sarama.SyncProducer, topic string, message []byte) error {

	msg := &sarama.ProducerMessage{
		Topic: topic,
		Value: sarama.StringEncoder(message),
	}

	partition, offset, err := producer.SendMessage(msg)
	if err != nil {
		return err
	}

	fmt.Printf("Message is stored in topic(%s)/partition(%d)/offset(%d)", topic, partition, offset)
	return nil
}

func createOrderHandler(producer sarama.SyncProducer) gin.HandlerFunc {
	return func(c *gin.Context) {
		topic := topics.ORDERS_CREATED.String()
		order := new(Order)

		if err := c.ShouldBindJSON(&order); err != nil {
			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
			return
		}

		orderInBytes, err := json.Marshal(order)
		if err != nil {
			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
			return
		}

		err = PushOrderToQueue(producer, topic, orderInBytes)
		if err != nil {
			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
			return
		}

		c.JSON(http.StatusOK, gin.H{
			"success": true,
			"message": "Order pushed successfully",
			"order":   order,
		})

	}
}

Creating the Consumer

Now for the other side of the pipeline. The consumer listens for events on topics, processes them, and handles failures by routing messages through a tiered reprocessing system.

Connecting Consumer Groups

Next, let's create our consumer group and producer connections. The consumer side also needs a producer because when a message fails processing, it needs to publish the message to the next retry topic:

func ConnectConsumerGroup(brokersUrl []string, groupID string) (sarama.ConsumerGroup, error) {
	config := sarama.NewConfig()

	config.Consumer.Return.Errors = true
	config.Consumer.Offsets.AutoCommit.Enable = false
	config.Consumer.Offsets.Initial = sarama.OffsetOldest

	return sarama.NewConsumerGroup(brokersUrl, groupID, config)
}

func ConnectProducer(brokersUrl []string) (sarama.SyncProducer, error) {
	config := sarama.NewConfig()

	config.Producer.RequiredAcks = sarama.WaitForAll
	config.Producer.Return.Successes = true
	config.Producer.Idempotent = true
	config.Net.MaxOpenRequests = 1

	return sarama.NewSyncProducer(brokersUrl, config)
}

Pay attention to config.Consumer.Offsets.AutoCommit.Enable = false. By default, Sarama will automatically commit offsets in the background on a timer, which means it could mark a message as "done" before your code has actually finished processing it. If your app crashes between the auto-commit and the actual processing, that message is lost. By disabling auto-commit, we take full control of when offsets get committed. This means we have to manually call session.MarkMessage and session.Commit in our handler to commit an offset.

The Main Consumer Group Handler

This is where the processing logic lives. When a message is consumed, we will attempt to process it. If processing fails, we route the message to the next retry tier:

type ConsumerGroupHandler struct {
	producer sarama.SyncProducer
}

func (h *ConsumerGroupHandler) Setup(sarama.ConsumerGroupSession) error {
	return nil
}

func (h *ConsumerGroupHandler) Cleanup(sarama.ConsumerGroupSession) error {
	return nil
}

func (h *ConsumerGroupHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {

	for msg := range claim.Messages() {
		fmt.Printf("Received message: key=%s, value=%s, partition=%d, offset=%d\n", string(msg.Key), string(msg.Value), msg.Partition, msg.Offset)

		err := processBusinessLogic(msg.Value)

		if err != nil {

			log.Printf("Processing failed on topic %s. Routing to next stage...", msg.Topic)

			errRoute := h.routeToNextStage(msg)
			if errRoute != nil {
				log.Printf("CRITICAL: Failed to route message: %v", errRoute)
				return errRoute
			}
		}

		session.MarkMessage(msg, "")
		session.Commit()
	}

	return nil
}

Notice how after we are done with a message, whether it was processed successfully or routed to the next retry stage, we call session.MarkMessage(msg, "") followed by session.Commit(). This is the manual offset commit we set up earlier when we disabled auto-commit. MarkMessage tells Sarama "I am done with this message" and Commit actually persists that offset to Kafka. This way, if the consumer crashes mid-processing, Kafka knows to redeliver the message because the offset was never committed.

Routing Failed Messages

The routeToNextStage method is the heart of our retry strategy. It reads the x-retry-count header from the message to determine how many times this message has already been retried, then routes it to the next topic accordingly. After three failed attempts, the message lands in the DLQ:

func (h *ConsumerGroupHandler) routeToNextStage(msg *sarama.ConsumerMessage) error {
	retryCount := 0
	for _, header := range msg.Headers {
		if string(header.Key) == "x-retry-count" {
			retryCount, _ = strconv.Atoi(string(header.Value))
		}
	}

	var nextTopic string
	switch retryCount {
	case 0:
		nextTopic = topics.RETRY_5M.String()
	case 1:
		nextTopic = topics.RETRY_20M.String()
	case 2:
		nextTopic = topics.RETRY_40M.String()
	default:
		nextTopic = topics.ORDERS_DLQ.String()
	}

	now := time.Now()
	headers := []sarama.RecordHeader{
		{Key: []byte("x-retry-count"), Value: []byte(strconv.Itoa(retryCount + 1))},
		{Key: []byte("x-failed-at"), Value: []byte(now.Format(time.RFC3339))},
		{Key: []byte("x-original-topic"), Value: []byte(msg.Topic)},
	}

	retryMsg := &sarama.ProducerMessage{
		Topic:   nextTopic,
		Key:     sarama.ByteEncoder(msg.Key),
		Value:   sarama.ByteEncoder(msg.Value),
		Headers: headers,
	}

	log.Printf("Routing message to topic: %s (Attempt #%d)", nextTopic, retryCount+1)
	_, _, err := h.producer.SendMessage(retryMsg)
	return err
}

We attach three headers to every retried message: x-retry-count so the next handler knows which attempt this is, x-failed-at so the retry handler can calculate the correct delay, and x-original-topic for traceability. The switch statement maps the retry count to the appropriate next topic, escalating through 5 minute, 20 minute, and 40 minute delays before finally landing in the DLQ.

The Retry Consumer Group Handler

The retry handler works similarly to the main handler, but with one key difference: it enforces a delay before reprocessing the message. It reads the x-failed-at header to determine when the message originally failed, then calculates how much time has already passed. If the message arrived before the required delay has elapsed, it sleeps for the remaining duration. If processing fails again, it escalates to the next retry tier:

type RetryConsumerGroupHandler struct {
	producer      sarama.SyncProducer
	delayDuration time.Duration
}

func (h *RetryConsumerGroupHandler) Setup(sarama.ConsumerGroupSession) error   { return nil }
func (h *RetryConsumerGroupHandler) Cleanup(sarama.ConsumerGroupSession) error { return nil }

func (h *RetryConsumerGroupHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
	for msg := range claim.Messages() {

		var failedAt time.Time
		for _, header := range msg.Headers {
			if string(header.Key) == "x-failed-at" {
				failedAt, _ = time.Parse(time.RFC3339, string(header.Value))
			}
		}

		timeElapsed := time.Since(failedAt)
		if timeElapsed < h.delayDuration {
			remainingDelay := h.delayDuration - timeElapsed
			log.Printf("[%s] Message arrived early. Sleeping for %v...", msg.Topic, remainingDelay)
			time.Sleep(remainingDelay)
		}

		err := processBusinessLogic(msg.Value)
		if err != nil {
			log.Printf("[%s] Retry failed. Escalating to next tier.", msg.Topic)

			mainHandler := &ConsumerGroupHandler{producer: h.producer}
			errRoute := mainHandler.routeToNextStage(msg)
			if errRoute != nil {
				return errRoute
			}
		}

		session.MarkMessage(msg, "")
		session.Commit()
	}
	return nil
}

Simulating Business Logic

For testing purposes, here is a dummy function that simulates our business logic. You can toggle between returning nil (success) and returning an error (failure) to test both paths:

func processBusinessLogic(data []byte) error {
	// return nil
	return fmt.Errorf("forced simulation error")
}

Complete Consumer Code

Here is the full consumer code with everything connected together. We create a shared producer (for routing failed messages), spin up consumer groups for the main topic and each retry tier, and run them all concurrently in separate goroutines:

// consumer/consumer.go

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"os/signal"
	"strconv"
	"syscall"
	"time"

	"github.com/IBM/sarama"
	"github.com/umohsamuel/edd-kafka-golang/pkg/topics"
)

func main() {
	brokersUrl := []string{"localhost:9092"}

	sharedProducer, err := ConnectProducer(brokersUrl)
	if err != nil {
		log.Fatalf("Failed to start DLQ Producer: %v", err)
	}
	defer sharedProducer.Close()

	mainGroup, err := ConnectConsumerGroup(brokersUrl, topics.ORDERS_MAIN_GROUP.String())
	if err != nil {
		log.Fatalf("Failed to start Main Consumer Group: %v", err)
	}
	defer mainGroup.Close()

	retryGroup5m, err := ConnectConsumerGroup(brokersUrl, topics.ORDERS_5M_RETRY_GROUP.String())
	if err != nil {
		log.Fatalf("Failed to start 5m Retry Group: %v", err)
	}
	defer retryGroup5m.Close()

	retryGroup20m, err := ConnectConsumerGroup(brokersUrl, topics.ORDERS_20M_RETRY_GROUP.String())
	if err != nil {
		log.Fatalf("Failed to start 20m Retry Group: %v", err)
	}
	defer retryGroup20m.Close()

	retryGroup40m, err := ConnectConsumerGroup(brokersUrl, topics.ORDERS_40M_RETRY_GROUP.String())
	if err != nil {
		log.Fatalf("Failed to start 40m Retry Group: %v", err)
	}
	defer retryGroup40m.Close()

	ctx, cancel := context.WithCancel(context.Background())
	sigchan := make(chan os.Signal, 1)
	signal.Notify(sigchan, syscall.SIGINT, syscall.SIGTERM)

	fmt.Println("Consumers have entered the fray...")

	go func() {
		handler := &ConsumerGroupHandler{
			producer: sharedProducer,
		}

		for {
			if err := mainGroup.Consume(ctx, []string{topics.ORDERS_CREATED.String()}, handler); err != nil {
				log.Printf("Error from main consumer group: %v", err)
			}

			if ctx.Err() != nil {
				return
			}
		}
	}()

	go func() {
		handler := &RetryConsumerGroupHandler{
			producer:      sharedProducer,
			delayDuration: 5 * time.Minute,
		}

		for {
			if err := retryGroup5m.Consume(ctx, []string{topics.RETRY_5M.String()}, handler); err != nil {
				log.Printf("Error from %s group: %v", topics.RETRY_5M.String(), err)
			}

			if ctx.Err() != nil {
				return
			}
		}
	}()

	go func() {
		handler := &RetryConsumerGroupHandler{
			producer:      sharedProducer,
			delayDuration: 20 * time.Minute,
		}

		for {
			if err := retryGroup20m.Consume(ctx, []string{topics.RETRY_20M.String()}, handler); err != nil {
				log.Printf("Error from %s group: %v", topics.RETRY_20M.String(), err)
			}

			if ctx.Err() != nil {
				return
			}
		}
	}()

	go func() {
		handler := &RetryConsumerGroupHandler{
			producer:      sharedProducer,
			delayDuration: 40 * time.Minute,
		}

		for {
			if err := retryGroup40m.Consume(ctx, []string{topics.RETRY_40M.String()}, handler); err != nil {
				log.Printf("Error from %s group: %v", topics.RETRY_40M.String(), err)
			}

			if ctx.Err() != nil {
				return
			}
		}
	}()

	<-sigchan
	fmt.Println("Interruption detected, shutting down all groups gracefully...")
	cancel()

}

func ConnectConsumerGroup(brokersUrl []string, groupID string) (sarama.ConsumerGroup, error) {
	config := sarama.NewConfig()

	config.Consumer.Return.Errors = true
	config.Consumer.Offsets.AutoCommit.Enable = false
	config.Consumer.Offsets.Initial = sarama.OffsetOldest

	return sarama.NewConsumerGroup(brokersUrl, groupID, config)
}

func ConnectProducer(brokersUrl []string) (sarama.SyncProducer, error) {
	config := sarama.NewConfig()

	config.Producer.RequiredAcks = sarama.WaitForAll
	config.Producer.Return.Successes = true
	config.Producer.Idempotent = true
	config.Net.MaxOpenRequests = 1

	return sarama.NewSyncProducer(brokersUrl, config)
}

func processBusinessLogic(data []byte) error {
	// return nil
	return fmt.Errorf("forced simulation error")
}

type ConsumerGroupHandler struct {
	producer sarama.SyncProducer
}

func (h *ConsumerGroupHandler) Setup(sarama.ConsumerGroupSession) error {
	return nil
}

func (h *ConsumerGroupHandler) Cleanup(sarama.ConsumerGroupSession) error {
	return nil
}

func (h *ConsumerGroupHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {

	for msg := range claim.Messages() {
		fmt.Printf("Received message: key=%s, value=%s, partition=%d, offset=%d\n", string(msg.Key), string(msg.Value), msg.Partition, msg.Offset)

		err := processBusinessLogic(msg.Value)

		if err != nil {

			log.Printf("Processing failed on topic %s. Routing to next stage...", msg.Topic)

			errRoute := h.routeToNextStage(msg)
			if errRoute != nil {
				log.Printf("CRITICAL: Failed to route message: %v", errRoute)
				return errRoute
			}
		}

		session.MarkMessage(msg, "")
		session.Commit()
	}

	return nil
}

func (h *ConsumerGroupHandler) routeToNextStage(msg *sarama.ConsumerMessage) error {
	retryCount := 0
	for _, header := range msg.Headers {
		if string(header.Key) == "x-retry-count" {
			retryCount, _ = strconv.Atoi(string(header.Value))
		}
	}

	var nextTopic string
	switch retryCount {
	case 0:
		nextTopic = topics.RETRY_5M.String()
	case 1:
		nextTopic = topics.RETRY_20M.String()
	case 2:
		nextTopic = topics.RETRY_40M.String()
	default:
		nextTopic = topics.ORDERS_DLQ.String()
	}

	now := time.Now()
	headers := []sarama.RecordHeader{
		{Key: []byte("x-retry-count"), Value: []byte(strconv.Itoa(retryCount + 1))},
		{Key: []byte("x-failed-at"), Value: []byte(now.Format(time.RFC3339))},
		{Key: []byte("x-original-topic"), Value: []byte(msg.Topic)},
	}

	retryMsg := &sarama.ProducerMessage{
		Topic:   nextTopic,
		Key:     sarama.ByteEncoder(msg.Key),
		Value:   sarama.ByteEncoder(msg.Value),
		Headers: headers,
	}

	log.Printf("Routing message to topic: %s (Attempt #%d)", nextTopic, retryCount+1)
	_, _, err := h.producer.SendMessage(retryMsg)
	return err
}

type RetryConsumerGroupHandler struct {
	producer      sarama.SyncProducer
	delayDuration time.Duration
}

func (h *RetryConsumerGroupHandler) Setup(sarama.ConsumerGroupSession) error   { return nil }
func (h *RetryConsumerGroupHandler) Cleanup(sarama.ConsumerGroupSession) error { return nil }

func (h *RetryConsumerGroupHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
	for msg := range claim.Messages() {

		var failedAt time.Time
		for _, header := range msg.Headers {
			if string(header.Key) == "x-failed-at" {
				failedAt, _ = time.Parse(time.RFC3339, string(header.Value))
			}
		}

		timeElapsed := time.Since(failedAt)
		if timeElapsed < h.delayDuration {
			remainingDelay := h.delayDuration - timeElapsed
			log.Printf("[%s] Message arrived early. Sleeping for %v...", msg.Topic, remainingDelay)
			time.Sleep(remainingDelay)
		}

		err := processBusinessLogic(msg.Value)
		if err != nil {
			log.Printf("[%s] Retry failed. Escalating to next tier.", msg.Topic)

			mainHandler := &ConsumerGroupHandler{producer: h.producer}
			errRoute := mainHandler.routeToNextStage(msg)
			if errRoute != nil {
				return errRoute
			}
		}

		session.MarkMessage(msg, "")
		session.Commit()
	}
	return nil
}

Running the Application

To run the application, first start the Docker Compose cluster (if you already did this earlier, you can skip it), then run the producer and consumer in separate terminals:

docker compose up -d

cd producer && go run .

cd consumer && go run .

Testing with cURL

You can test the pipeline by sending a POST request to the producer's API:

curl -X POST http://localhost:8080/api/v1/order \
  -H "Content-Type: application/json" \
  -d '{"text": "New order from Lagos"}'

To see the full retry flow in action, keep the processBusinessLogic function returning the dummy error. Send a request and watch the consumer logs as the message fails, gets routed to the 5 minute retry topic, fails again, moves to the 20 minute retry, then the 40 minute retry, and finally lands in the DLQ.

Once you have confirmed that flow, change processBusinessLogic to return nil, restart the consumer service, and send another request to confirm the success path works cleanly, with the message being marked and committed on the first pass.

Lastly, while the consumer service is running, try hitting Ctrl+C as well. You should see each consumer group shut down gracefully in turn, without dropping any in-flight messages.

Demo

Here is a demo of how all these concepts come together:

Demo Link

Conclusion

In this guide, we built a fault-tolerant event-driven pipeline using Apache Kafka and Go. The key takeaways are:

Event-Driven Architecture decouples your services, letting producers and consumers operate independently through a message broker.
Tiered retry topics & reprocessing with increasing delays (5m, 20m, 40m) give transient failures time to resolve before reprocessing, instead of hammering the same operation repeatedly.
Dead letter queues catch messages that have exhausted all retry attempts, so they can be inspected and handled manually without blocking the pipeline.
Consumer groups let you scale consumption horizontally while Kafka handles partition assignment and offset tracking.

These patterns apply to any Go service that needs reliable, asynchronous message processing, whether you are processing orders, sending notifications, or syncing data between services.

(If you want to see the full codebase, checkout the repository. Play around with it and lmk what you think)

References

Kafka in hospitals: real-world event architecture for HIS, LIS and billing in a Venezuelan mid-size

Wed, 29 Jul 2026 06:37:11 +0000

Most Kafka “healthcare interoperability” examples are theoretical or focused on US/EU EHRs. I wanted to share a concrete case from a mid‑size hospital in Venezuela where we used Kafka as the event backbone between HIS, LIS and billing systems.

Context:

4–8 systems needed to know the same things in near real time: admissions, lab results, charges.

Point‑to‑point integrations had become a mess: custom scripts, duplicate records, fragile error handling.

The hospital knew it would keep adding systems (BI, SAP, new clinical modules) over the next few years.

What we ended up doing:

Defined topics by clinical domain (admissions, lab-results, orders, billing-events) instead of “one topic for everything”.

Partitioned by patient or encounter ID to preserve clinical event order per case.

Used replication factor 3 and min.insync.replicas=2 because losing a lab result is not just a business data issue.

Enabled idempotent + transactional producers (exactly-once semantics) for lab results and financial events, where duplicates are harmful.

The article goes into more detail on:

How we mapped HL7/FHIR messages to Kafka topics.

Trade‑offs in infra cost vs. resilience for a hospital context.

When Kafka was clearly the wrong tool (small clinics with only 2–3 systems and low event volume).

Link (in Spanish, but I’m happy to summarize or answer questions in English here):

Kafka in hospitals

Message Queue Example using RabbitMQ in Go

Sun, 03 May 2026 05:44:27 +0000

Project Overview

In this example, we will build a simple advertisement platform. This platform will send a notification if the advertisement is created. The message queue is utilized as a buffer between the advertisement service and the notification service.

This is the architecture diagram of this project.

Develop the Project

Project Setup

The project is implemented using microservices architecture. These are the key components of the project:

Advertisement service: the service to handle advertisement feature including create a new advertisement.
Notification service: the service for handling the advertisement notification.
API Gateway: the main entrypoint for client to accessing the available services.

The project is implemented using microservices architecture. These are the key components of the project:

Advertisement service: the service to handle advertisement feature including create a new advertisement.
Notification service: the service for handling the advertisement notification.
API Gateway: the main entrypoint for client to accessing the available services.

The project structure is implemented using Go workspace. This is the overview of the project structure.

go-mini-ads
 ┣ ads
 ┣ common
 ┃ ┣ models
 ┃ ┣ rabbitmq
 ┃ ┣ utils
 ┣ gateway
 ┣ notification
 ┣ .env
 ┣ .gitignore
 ┣ docker-compose.yml

The very first step is to create a Go workspace. Initialize the Go Workspace using this command.

go work init

After creating the workspace, notice that the go.work file is created. Adjust the go.work file to include the required packages.

go 1.25.5

use (
./ads
./gateway
./notification
./common
)

Create the .env file to store the required configurations.

APP_MODE="production"

ADS_SERVICE_BASE_URL="http://ads-service:1323"

RABBITMQ_USER="admin"
RABBITMQ_PASS="secretpassword123"
RABBITMQ_HOST="rabbitmq-service"
RABBITMQ_PORT="5672"

If not using Docker compose, ensure to change the app mode to dev and change ads-service & RABBITMQ_HOST to localhost.

Create Common Utilities

Create a new directory called common. After that, initialize a go module for this package.

go mod init common

Install the required packages.

go get github.com/spf13/viper
go get github.com/rabbitmq/amqp091-go

Inside the common directory, create a new directory called models to store required entities for the services.

Inside the models directory, create a new file called ad.go to store the data model of the advertisement.

package models

type Ad struct {
ID       string `json:"id"`
Title    string `json:"title"`
Content  string `json:"content"`
Category string `json:"category"`
}

Inside the models directory, create a new file called request.go to store the request entity for creating a new advertisement.

package models

type AdRequest struct {
Title    string `json:"title"`
Content  string `json:"content"`
Category string `json:"category"`
}

Inside the models directory, create a new file called response.go to store the response entity. This entity is using generic which means the type of the data can be adjusted dynamically.

package models

type Response[T any] struct {
Success bool   `json:"success"`
Message string `json:"message"`
Data    T      `json:"data"`
}

After the entities are created, create a new directory called utils to store the utility functions and constants. The utils package will be used across the services.

Create a file called constants.go to store the constant values. The constant values are used for ensuring the consistency of the value.

package utils

const ADS_QUEUE = "ads"

// env value keys
const (
APP_MODE             = "APP_MODE"
ADS_SERVICE_BASE_URL = "ADS_SERVICE_BASE_URL"
RABBITMQ_USER        = "RABBITMQ_USER"
RABBITMQ_PASS        = "RABBITMQ_PASS"
RABBITMQ_HOST        = "RABBITMQ_HOST"
RABBITMQ_PORT        = "RABBITMQ_PORT"
)

const (
APP_MODE_PRODUCTION = "production"
)

This application uses .env file for storing and fetching the configurations. To fetch the values from the .env file, create a new utility function called GetConfig inside the config.go file.

package utils

import (
"log"
"os"

"github.com/spf13/viper"
)

func GetConfig(key string) string {
isDevelop := os.Getenv(APP_MODE) != APP_MODE_PRODUCTION

if isDevelop {
viper.AddConfigPath(".")
viper.SetConfigFile(".env")

if err := viper.ReadInConfig(); err != nil {
log.Fatalf("error when reading configuration file: %s\n", err)
}

return viper.GetString(key)
}

return os.Getenv(key)
}

To interact with RabbitMQ service conveniently, create a dedicated directory called rabbitmq. Inside this directory, create a new file called rabbitmq.go.

Inside the rabbitmq.go, the RabbitMQ instance will be created, this instance can initialize required setups (like: connection, queue and channel) and provides basic functionalities like publish and consume messages.

First, import the required package and create a struct that act as a RabbitMQ instance.

package rabbitmq

import (
"common/utils"
"context"
"errors"
"fmt"

"github.com/rabbitmq/amqp091-go"
amqp "github.com/rabbitmq/amqp091-go"
)

type RabbitMQ struct {
Channel *amqp.Channel
Queue   amqp.Queue
}

Create a function to retrieve the RabbitMQ address.

func GetRabbitMQAddress() string {
return fmt.Sprintf("amqp://%s:%s@%s:%s/",
utils.GetConfig(utils.RABBITMQ_USER),
utils.GetConfig(utils.RABBITMQ_PASS),
utils.GetConfig(utils.RABBITMQ_HOST),
utils.GetConfig(utils.RABBITMQ_PORT),
)
}

Create a function to initialize the RabbitMQ connection. This function returns a connection object that can be used for interacting with the RabbitMQ service.

func InitConnection(address string) (*amqp.Connection, error) {
conn, err := amqp.Dial(address)
if err != nil {
return nil, fmt.Errorf("failed to connect: %s\n", err)
}

return conn, nil
}

Create a function to initialize the connection channel. This function returns a connection channel that can be used for using RabbitMQ features like send and consume messages.

func InitChannel(conn *amqp.Connection) (*amqp.Channel, error) {
ch, err := conn.Channel()
if err != nil {
return nil, fmt.Errorf("failed to open channel: %s\n", err)
}

return ch, nil
}

Create a function to initialize the queue. The queue uses FIFO principle which means the first message comes in, that message will be retrieved first.

func DeclareQueue(ch *amqp.Channel) (amqp.Queue, error) {
q, err := ch.QueueDeclare(
utils.ADS_QUEUE, // name
true,            // durability
false,           // delete when unused
false,           // exclusive
false,           // no-wait
amqp.Table{
amqp.QueueTypeArg: amqp.QueueTypeQuorum,
},
)
if err != nil {
return amqp.Queue{}, fmt.Errorf("failed to create queue: %s\n", err)
}

return q, nil
}

Create a function to initialize the RabbitMQ instance. When initializing this instance, the previous created functions are called and assembled together.

Using this approach makes the code more clean and easy to refactor when there is an additional changes when initializing the instance.

func InitRabbitMQ() (*RabbitMQ, error) {
address := GetRabbitMQAddress()
conn, connErr := InitConnection(address)
ch, chErr := InitChannel(conn)
q, qErr := DeclareQueue(ch)

err := errors.Join(connErr, chErr, qErr)

if err != nil {
return nil, fmt.Errorf("failed to create RabbitMQ instance: %s\n", err)
}

return &RabbitMQ{
Channel: ch,
Queue:   q,
}, nil
}

Create a method for RabbitMQ instance to publish a message to the queue. This method requires context object, queue name and the message content.

func (rq *RabbitMQ) PublishWithCtx(ctx context.Context, queueName string, body []byte) error {
return rq.Channel.PublishWithContext(ctx,
"",        // exchange
queueName, // routing key
false,     // mandatory
false,     // immediate
amqp.Publishing{
ContentType: "application/json",
Body:        body,
})
}

Create a method for RabbitMQ instance to consume the messages from the queue. This method requires queue name.

func (rq *RabbitMQ) Consume(queueName string) (<-chan amqp091.Delivery, error) {
return rq.Channel.Consume(
queueName, // queue
"",        // consumer
true,      // auto-ack
false,     // exclusive
false,     // no-local
false,     // no-wait
nil,       // args
)
}

Create Ads Service

Create a new directory called ads. After that, initialize a go module for this package.

go mod init ads-service

Install the required packages.

go get github.com/google/uuid
go get github.com/labstack/echo/v5
go get github.com/labstack/echo/v5/middleware
go get github.com/rabbitmq/amqp091-go

Create a new file called main.go to store the main logic inside this service.

Import the required packages.

package main

import (
"common/models"
"common/rabbitmq"
"context"
"encoding/json"
"fmt"
"log"
"net/http"
"time"

"github.com/google/uuid"
"github.com/labstack/echo/v5"
"github.com/labstack/echo/v5/middleware"
)

var RMQ *rabbitmq.RabbitMQ

Inside the main function, initialize the Echo application and RabbitMQ instance.

func main() {
var err error

e := echo.New()
e.Use(middleware.RequestLogger())

RMQ, err = rabbitmq.InitRabbitMQ()

if err != nil {
log.Fatalf("failed to init RabbitMQ: %s\n", err)
}

// create controller here

// start the server
if err := e.Start(":1323"); err != nil {
e.Logger.Error("failed to start server", "error", err)
}
}

After the Echo and RabbitMQ instance are initialized, Create a controller for creating the advertisement data.

func main() {
// other codes...
e.POST("/ads/create", func(c *echo.Context) error {
req := new(models.AdRequest)

// bind request body
if err := c.Bind(req); err != nil {
return c.JSON(http.StatusBadRequest, models.Response[any]{
Success: false,
Message: "request invalid",
})
}

// create advertisement record
record := models.Ad{
ID:       uuid.NewString(),
Title:    req.Title,
Content:  req.Content,
Category: req.Category,
}

// send record to the message queue
if err := sendToQueue(record); err != nil {
return c.JSON(http.StatusInternalServerError, models.Response[any]{
Success: false,
Message: err.Error(),
})
}

// return created advertisement record
return c.JSON(http.StatusCreated, models.Response[models.Ad]{
Success: true,
Message: "ad created",
Data:    record,
})
})
}

Notice that in the create advertisement controller, the sendToQueue is called for sending the advertisement record to the message queue.

Create a function to store the advertisement record to the message queue called sendToQueue. Create this function outside the main function.

func sendToQueue(data models.Ad) error {
// create context object
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

// parse advertisement data
body, err := json.Marshal(data)
if err != nil {
return fmt.Errorf("failed to parse data: %s\n", err)
}

// send advertisement data to the message queue
err = RMQ.PublishWithCtx(ctx, RMQ.Queue.Name, body)
if err != nil {
return fmt.Errorf("failed to publish message: %s\n", err)
}

return nil
}

The sendToQueue function sends the advertisement record to the message queue by calling the PublishWithCtx method from the RabbitMQ instance.

Create a Dockerfile for running this service as a Docker container.

FROM golang:1.25 AS build-stage

WORKDIR /app

COPY . .

WORKDIR /app/ads

RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux go build -o /adsapp

FROM gcr.io/distroless/base-debian11 AS build-release-stage

WORKDIR /

COPY --from=build-stage /adsapp /adsapp

EXPOSE 1323

USER nonroot:nonroot

ENTRYPOINT [ "./adsapp" ]

Create API Gateway Service

Create a new directory called gateway. After that, initialize a go module for this package.

go mod init gateway-service

Install the required packages.

go get github.com/labstack/echo/v5
go get github.com/labstack/echo/v5/middleware

Create a main function inside the main.go file. Then, import the required packages.

package main

import (
"bytes"
"common/models"
"common/utils"
"encoding/json"
"fmt"
"net/http"
"time"

"github.com/labstack/echo/v5"
"github.com/labstack/echo/v5/middleware"
)

Inside the main function, initialize the Echo application and start the server as well.

func main() {
e := echo.New()
e.Use(middleware.RequestLogger())

e.POST("/api/ads", createAd)

if err := e.Start(":8080"); err != nil {
e.Logger.Error("failed to start server", "error", err)
}
}

Create a controller called createAd to create a new advertisement record. This controller sends HTTP request to the advertisement service to create a new advertisement record.

Before sending the HTTP request, parse the request body and initialize the HTTP client.

func createAd(c *echo.Context) error {
adReq := new(models.AdRequest)

if err := c.Bind(adReq); err != nil {
return c.JSON(http.StatusBadRequest, models.Response[any]{
Success: false,
Message: "request invalid",
})
}

// create HTTP client with 10-second request timeout
client := &http.Client{
Timeout: 10 * time.Second,
}

// parse request body as a JSON
body, err := json.Marshal(adReq)
if err != nil {
return c.JSON(http.StatusBadRequest, models.Response[any]{
Success: false,
Message: "request invalid",
})
}

adsServiceUrl := utils.GetConfig(utils.ADS_SERVICE_BASE_URL)
adsEndpoint := fmt.Sprintf("%s/ads/create", adsServiceUrl)

// create HTTP POST request to the advertisement service
request, err := http.NewRequest(http.MethodPost, adsEndpoint, bytes.NewBuffer(body))

if err != nil {
return c.JSON(http.StatusInternalServerError, models.Response[any]{
Success: false,
Message: "failed to create a request",
})
}

request.Header.Set("Content-Type", "application/json")
}

After the request body is parsed and HTTP client is initialized, send the HTTP request and store the response body in a JSON format.

func createAd(c *echo.Context) error {
// other codes...
// send HTTP request
response, err := client.Do(request)
if err != nil || response.StatusCode != http.StatusCreated {
return c.JSON(http.StatusInternalServerError, models.Response[any]{
Success: false,
Message: "failed to create ad",
})
}

// close the response body after parse process is completed
defer response.Body.Close()

// variable to store the response body
var data models.Response[models.Ad]

// parse response body and store in the "data" variable
err = json.NewDecoder(response.Body).Decode(&data)
if err != nil {
return c.JSON(http.StatusInternalServerError, models.Response[any]{
Success: false,
Message: "failed to parse ad data",
})
}

return c.JSON(http.StatusCreated, data)
}

Create a Dockerfile for running this service as a Docker container.

FROM golang:1.25 AS build-stage

WORKDIR /app

COPY . .

WORKDIR /app/gateway

RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux go build -o /gatewayapp

FROM gcr.io/distroless/base-debian11 AS build-release-stage

WORKDIR /

COPY --from=build-stage /gatewayapp /gatewayapp

EXPOSE 1323

USER nonroot:nonroot

ENTRYPOINT [ "./gatewayapp" ]

Create Notification Service

Create a new directory called notification. After that, initialize a go module for this package.

go mod init notification-service

Install the required package.

go get github.com/rabbitmq/amqp091-go

Create a main function inside the main.go file. Then, import the required packages.

package main

import (
"common/models"
"common/rabbitmq"
"encoding/json"
"fmt"
"log"
"strings"
)

var RMQ *rabbitmq.RabbitMQ

In the main function, initialize the RabbitMQ instance and consume the incoming messages from the queue.

The incoming messages are retrieved from the message queue via delivery channel. Consume the messages inside the goroutine.

func main() {
var err error

// initialize RabbitMQ instance
RMQ, err = rabbitmq.InitRabbitMQ()

if err != nil {
log.Fatalf("failed to init RabbitMQ: %s\n", err)
}

// consume messages
msgs, err := RMQ.Consume(RMQ.Queue.Name)

if err != nil {
log.Fatalf("failed to consume queue: %s\n", err)
}

var forever chan struct{}

// parse the messages
go func() {
for d := range msgs {
parseMessage(d.Body)
}
}()

log.Println("[*] Waiting for messages. To exit press CTRL+C")
<-forever
}

The parseMessage function parses the incoming messages from the queue. This is the complete implementation of the function.

func parseMessage(body []byte) {
var record models.Ad

if err := json.Unmarshal(body, &record); err != nil {
log.Fatalf("failed to parse message: %s\n", err)
}

fmt.Println("send notification...")
fmt.Println(strings.Repeat("=", 5))
fmt.Println("title: ", record.Title)
fmt.Println("content: ", record.Content)
fmt.Println("category: ", record.Category)
fmt.Println(strings.Repeat("=", 5))
fmt.Println("notification sent")
}

Create a Dockerfile for running this service as a Docker container.

FROM golang:1.25 AS build-stage

WORKDIR /app

COPY . .

WORKDIR /app/notification

RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux go build -o /notifapp

FROM gcr.io/distroless/base-debian11 AS build-release-stage

WORKDIR /

COPY --from=build-stage /notifapp /notifapp

EXPOSE 1323

USER nonroot:nonroot

ENTRYPOINT [ "./notifapp" ]

Configure the Services

The services can be run and managed easily using Docker compose. Check out the complete file here

Run the Project

The services can be run using this command. This command run all the specified services in the Docker compose file.

docker compose up -d

Try to send the HTTP request. This request is sent to the gateway service.

curl -X POST "http://localhost:8080/api/ads" \
  -H "Content-Type: application/json" \
  -d '{"title":"new ad","content":"ad content","category":"anything but new"}'

If successful, the notification service should be able to parse the message from the queue. Check it using docker logs notification-service

send notification...
=====
title:  new ad
content:  ad content
category:  anything but new
=====
notification sent

Sources

⭐️ If you found this post is helpful, don't forget to like and save it. If you have any feedbacks, please let me know in the comment section down below.

Thank you

The Evolution of a SaaS Architecture

Sun, 03 May 2026 04:04:26 +0000

Building a scalable SaaS platform is a journey of continuous learning and strategic evolution. In modern software engineering, the most successful platforms are those built with an "Evolutionary Architecture" mindset. This means designing your systems to embrace change, allowing your technical capabilities to grow in perfect harmony with your user base and business milestones.

When launching a new platform, the primary objective is learning about the market, discovering user needs, and establishing product-market fit. A successful architecture supports this discovery phase by prioritizing developer velocity and adaptability. By aligning your infrastructure with your current traffic levels, you ensure that engineering resources are focused entirely on delivering value to the user.

The strategy for building a highly resilient, long-lasting platform relies on a phased, educational approach. You map your infrastructure investments directly to the milestones you achieve in user growth. We can organize this philosophy into three essential pillars of technical growth:

Build for validation: Design systems that allow for rapid prototyping, incredibly fast deployment cycles, and immediate user feedback.
Upgrade for stability: As user acquisition accelerates, introduce foundational scaling techniques to ensure high availability and consistent response times.
Architect for scale: When your platform reaches enterprise levels of engagement, systematically decouple components to unlock distributed, specialized computing power.

By adopting this progressive framework, you maintain a highly efficient engineering budget while gracefully supporting increased demand. Here is a comprehensive, deeply technical roadmap for guiding your application's journey from a unified monolith to an intelligent, event-driven microservices ecosystem.

Phase 1: The Monolithic Approach

Traffic Milestone: 0 to 1,000 users. The Tech: A Modular Monolith on a dedicated virtual environment.

At the inception of a project, engineering speed is your greatest asset. The goal is to translate ideas into functional code as efficiently as possible to gather real-world data. A monolithic architecture is uniquely equipped to provide this rapid iteration cycle, allowing you to ship new features daily without the overhead of managing distributed systems.

In a well-designed monolithic architecture, all core functionalities your user interface serving, business logic, background jobs, and data access layers reside within a single, unified codebase and operate within the same memory space.

This unified approach offers tremendous advantages for an evolving product. Deployments are straightforward, often requiring just a single CI/CD pipeline step. Because all modules share the same memory, internal function calls are lightning-fast, entirely avoiding the latency and complexity of network-based API communication. Furthermore, observing the system is beautifully simple; developers can trace a user's entire journey through the application by examining a single, centralized stream of logs.

To make the most of this phase, engineering teams should focus on building a Modular Monolith. By strictly enforcing logical boundaries within the code such as separating the user management module from the billing module you create a clean, well-organized system. This discipline not only makes the codebase easier to understand and test but also perfectly positions the application for future architectural evolution.

Phase 2: Horizontal Scaling and the 12-Factor App

Traffic Milestone: 1,000 to 10,000 users.

As your platform's value resonates with the market, user engagement naturally increases. You will begin to observe higher CPU utilization and increased memory consumption as your server actively processes concurrent user requests. This is a positive milestone it signals that your platform is thriving and it is time to expand your compute capacity.

To handle increased throughput gracefully, the architecture shifts from vertical scaling (adding a larger server) to horizontal scaling (adding multiple identical servers). By containerizing your application packaging the code, runtime, and dependencies into standardized units using Docker you can launch multiple instances of your application simultaneously.

These instances sit behind an Application Load Balancer. The load balancer acts as an intelligent traffic director, continuously monitoring the health of your application instances and distributing incoming HTTP requests evenly across the fleet. If traffic surges, you can simply instruct your cloud provider to spin up additional containers to share the workload.

The Architectural Shift: Transitioning to a horizontally scaled environment introduces a brilliant engineering concept: Statelessness, a core principle of the 12-Factor App methodology. Because a user's requests might be routed to Server A on their first click and Server B on their second, your application servers can no longer rely on storing local data.

To resolve this, engineers externalize the state. Session data, temporary cache, and user tokens are migrated to a centralized, high-speed, in-memory datastore like Redis. By decoupling the state from the application servers, any server in your fleet can seamlessly pick up a request and process it with full context, creating a deeply resilient, highly available application tier.

Phase 3: Optimizing the Data Layer

Traffic Milestone: 10,000 to 100,000 users.

Expanding your API server fleet handles computational load beautifully, but data-intensive applications eventually shift the demand to the persistent storage layer. In a thriving SaaS environment, ensuring the relational database remains responsive is paramount for a seamless user experience.

Before structurally changing the database, modern engineering teams focus on Intelligent Caching. By leveraging the Redis infrastructure introduced in Phase 2, developers can intercept frequent, heavy database queries. If thousands of users are loading the same global analytics dashboard, the database performs the complex calculation once, stores the result in Redis, and serves the subsequent thousands of requests directly from memory in a fraction of a millisecond. Mastering cache invalidation strategies ensuring users always see fresh data becomes a rewarding engineering focus at this stage.

As engagement deepens, you can further enhance data availability by implementing a Read/Write Split at the database level.

The Master Node: The primary database instance becomes highly specialized, dedicated entirely to processing data modifications handling the INSERT, UPDATE, and DELETE transactions ensuring absolute data integrity.
The Read Replicas: You introduce synchronized copies of your database. The application intelligently routes all complex, read-only queries (SELECT) to these replicas.

This architecture prevents heavy reporting queries from consuming the resources needed for transactional operations. Your application learns to handle "eventual consistency," understanding that there might be a few milliseconds of replication delay between writing to the master and reading from the replica. This elegant separation of concerns ensures that the data layer remains incredibly robust, even during your highest traffic events.

Phase 4: Unlocking the Event-Driven Ecosystem

Traffic Milestone: 100,000+ users.

As your platform reaches enterprise-level utilization, you will likely offer advanced, resource-intensive features. Your application might generate comprehensive data exports, encode high-resolution media, or orchestrate complex interactions with external Artificial Intelligence models.

Handling these monumental tasks synchronously forcing the user to wait while the main API processes the data monopolizes server threads and impacts the experience of other concurrent users. This is the perfect moment to evolve your architecture by isolating these specific workloads into independent Microservices.

Instead of processing heavy tasks on the spot, the main application adopts an Asynchronous, Event-Driven Architecture. The platform delegates the heavy lifting by utilizing a highly durable Message Broker (such as Apache Kafka, Amazon SQS, or RabbitMQ).

The interaction becomes a seamless, non-blocking flow:

The API Gateway: The user requests a complex AI generation. The main API immediately returns a 202 Accepted response, assuring the user that the process has begun.
The Event Stream: The API securely publishes a message payload containing the job details into a dedicated queue.
The Worker Fleet: Independent, specialized microservices designed specifically for CPU-intensive or GPU-intensive workloads listen to this queue. They pick up the messages and process the tasks silently in the background, updating the main database or notifying the user via WebSockets upon completion.

This architectural leap provides the ultimate capability: Granular, Event-Driven Scaling. Because your services are decoupled, you can optimize your infrastructure costs brilliantly. If there is a massive backlog of video encoding jobs, you can configure your cloud environment to automatically spin up 50 instances of the specific "Video Worker" microservice to drain the queue quickly, while keeping your main API fleet operating at a stable, cost-effective baseline.

Cultivating an Ecosystem of Continuous Learning

The evolution from a single server to a distributed microservices architecture is a fascinating journey of continuous technical improvement. The most effective engineering teams understand that architecture is never truly "finished"; it is a living ecosystem that adapts to serve the user.

Start with a Modular Monolith to maximize learning, validate your market hypothesis, and establish clean domain boundaries.
Introduce Load Balancing and Statelessness to guarantee high availability and consistent performance as your audience grows.
Optimize Data Retrieval through intelligent memory caching and database replication, ensuring your persistence layer is always responsive.
Embrace Event-Driven Microservices strategically, isolating complex workloads to unlock specialized computing power and granular scalability.

To guide this evolution effectively, implementing deep Observability is key. By integrating tools like OpenTelemetry, Prometheus, and distributed tracing, you grant your engineering team the ability to visualize how data flows through the system. These metrics serve as the guiding light, telling you exactly when and where to apply the next architectural evolution.

Build systems that serve the present beautifully, while actively designing the foundational agility required for the innovations of tomorrow.

Looking to strategically evolve your platform's architecture? Let's Connect!
I am a Senior Full Stack AI Engineer specializing in the design, deployment, and optimization of highly resilient, cloud-agnostic SaaS platforms and intelligent, event-driven applications.

Designing Audit-Proof Financial Workflows with Immutable Logs

Fri, 17 Apr 2026 10:09:35 +0000

In financial software, data does not just need to be correct — it needs to be provably correct. Regulators, auditors, and compliance teams do not take your word for it. They need an unbroken chain of evidence showing exactly what happened, when it happened, and who triggered it.

I have built financial platforms where a missing audit trail meant weeks of manual reconciliation. I have also built systems where every state change was captured in an append-only log that could answer any auditor's question in seconds. The difference is not complexity — it is intentional design.

This article covers how to build audit-proof financial workflows using immutable event logs, covering architecture decisions, implementation patterns in both Laravel and Node.js, and the operational practices that make the system trustworthy under scrutiny.

Why Mutable Records Break Under Audit

Most web applications store the current state of a record. When an invoice status changes from "pending" to "paid," the database row is updated in place. The previous state is gone.

This works fine for most applications. For financial systems, it is a liability:

No history: You cannot prove what the invoice status was yesterday at 3pm.
No attribution: You cannot tell who changed the status or why.
No reconciliation path: When numbers do not match, there is no trail to follow.
No tamper evidence: A modified record looks identical to a legitimate one.

Financial regulations — PCI DSS, SOX, FCA guidelines — require that systems maintain a complete, tamper-evident record of all financial transactions and state changes. An UPDATE statement destroys exactly the evidence you are required to preserve.

The Immutable Log Pattern

The solution is to never update or delete financial records. Instead, every change is recorded as a new, append-only event:

[Event 1] Invoice #1042 CREATED — amount: £5,000 — by: user_42 — at: 2025-03-01T09:15:00Z
[Event 2] Invoice #1042 APPROVED — approved_by: user_15 — at: 2025-03-01T14:30:00Z
[Event 3] Invoice #1042 PAYMENT_RECEIVED — amount: £5,000 — ref: PAY_8821 — at: 2025-03-05T11:00:00Z
[Event 4] Invoice #1042 RECONCILED — matched_to: BANK_TXN_4401 — at: 2025-03-06T08:45:00Z

The current state of the invoice is derived by replaying the events in order. The audit trail is the events themselves. Nothing is lost, nothing is overwritten.

Architecture Overview

[Client Request]
       |
       v
[Application Layer]
  - Validate business rules
  - Create event record
       |
       v
[DB Transaction]
  - Write event to audit_events table (append-only)
  - Update materialised view / current-state table
       |
       v
[Audit Events Table]           [Current State Table]
  (immutable, append-only)      (derived, rebuildable)
       |
       v
[Async Consumers]
  - Compliance reporting
  - Reconciliation engine
  - Notification service

The critical design decision: the audit events table is the source of truth. The current-state table is a projection that can be rebuilt from events at any time.

Implementation in Laravel

Audit Events Migration

Schema::create('audit_events', function (Blueprint $table) {
    $table->uuid('id')->primary();
    $table->string('aggregate_type');
    $table->string('aggregate_id');
    $table->unsignedBigInteger('sequence_number');
    $table->string('event_type');
    $table->json('payload');
    $table->json('metadata');
    $table->string('actor_id');
    $table->string('actor_type');
    $table->string('ip_address')->nullable();
    $table->string('checksum');
    $table->timestamp('occurred_at');
    $table->timestamp('created_at');

    $table->unique(['aggregate_type', 'aggregate_id', 'sequence_number']);
    $table->index(['aggregate_type', 'aggregate_id']);
    $table->index('event_type');
    $table->index('occurred_at');
    $table->index('actor_id');
});

The checksum column stores a hash of the event data plus the previous event's checksum, forming a hash chain. Any tampering with a historical record breaks the chain.

Audit Event Service

namespace App\Services;

use App\Models\AuditEvent;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Str;

class AuditEventService
{
    public function record(
        string $aggregateType,
        string $aggregateId,
        string $eventType,
        array $payload,
        string $actorId,
        string $actorType = 'user',
        ?string $ipAddress = null,
    ): AuditEvent {
        return DB::transaction(function () use (
            $aggregateType, $aggregateId, $eventType,
            $payload, $actorId, $actorType, $ipAddress
        ) {
            $lastEvent = AuditEvent::where('aggregate_type', $aggregateType)
                ->where('aggregate_id', $aggregateId)
                ->orderByDesc('sequence_number')
                ->lockForUpdate()
                ->first();

            $sequenceNumber = $lastEvent ? $lastEvent->sequence_number + 1 : 1;
            $previousChecksum = $lastEvent?->checksum ?? 'GENESIS';

            $eventData = [
                'aggregate_type' => $aggregateType,
                'aggregate_id' => $aggregateId,
                'sequence_number' => $sequenceNumber,
                'event_type' => $eventType,
                'payload' => $payload,
                'actor_id' => $actorId,
                'occurred_at' => now(),
            ];

            $checksum = hash('sha256', $previousChecksum . json_encode($eventData));

            return AuditEvent::create([
                'id' => Str::uuid(),
                ...$eventData,
                'metadata' => [
                    'previous_checksum' => $previousChecksum,
                    'schema_version' => '1.0',
                ],
                'actor_type' => $actorType,
                'ip_address' => $ipAddress,
                'checksum' => $checksum,
                'created_at' => now(),
            ]);
        });
    }

    public function verifyChain(string $aggregateType, string $aggregateId): bool
    {
        $events = AuditEvent::where('aggregate_type', $aggregateType)
            ->where('aggregate_id', $aggregateId)
            ->orderBy('sequence_number')
            ->get();

        $previousChecksum = 'GENESIS';

        foreach ($events as $event) {
            $eventData = [
                'aggregate_type' => $event->aggregate_type,
                'aggregate_id' => $event->aggregate_id,
                'sequence_number' => $event->sequence_number,
                'event_type' => $event->event_type,
                'payload' => $event->payload,
                'actor_id' => $event->actor_id,
                'occurred_at' => $event->occurred_at,
            ];

            $expectedChecksum = hash('sha256', $previousChecksum . json_encode($eventData));

            if ($expectedChecksum !== $event->checksum) {
                return false;
            }

            $previousChecksum = $event->checksum;
        }

        return true;
    }

    public function getHistory(
        string $aggregateType,
        string $aggregateId
    ): \Illuminate\Support\Collection {
        return AuditEvent::where('aggregate_type', $aggregateType)
            ->where('aggregate_id', $aggregateId)
            ->orderBy('sequence_number')
            ->get();
    }
}

Using It in a Financial Workflow

class InvoiceService
{
    public function __construct(
        private AuditEventService $auditService,
    ) {}

    public function createInvoice(array $data, string $userId, string $ip): Invoice
    {
        return DB::transaction(function () use ($data, $userId, $ip) {
            $invoice = Invoice::create([
                'number' => $this->generateInvoiceNumber(),
                'client_id' => $data['client_id'],
                'amount' => $data['amount'],
                'currency' => $data['currency'],
                'status' => 'draft',
                'due_date' => $data['due_date'],
            ]);

            $this->auditService->record(
                aggregateType: 'invoice',
                aggregateId: (string) $invoice->id,
                eventType: 'invoice.created',
                payload: [
                    'number' => $invoice->number,
                    'amount' => $invoice->amount,
                    'currency' => $invoice->currency,
                    'client_id' => $invoice->client_id,
                ],
                actorId: $userId,
                ipAddress: $ip,
            );

            return $invoice;
        });
    }

    public function approveInvoice(Invoice $invoice, string $approverId, string $ip): Invoice
    {
        return DB::transaction(function () use ($invoice, $approverId, $ip) {
            $invoice->update(['status' => 'approved']);

            $this->auditService->record(
                aggregateType: 'invoice',
                aggregateId: (string) $invoice->id,
                eventType: 'invoice.approved',
                payload: [
                    'previous_status' => 'draft',
                    'new_status' => 'approved',
                ],
                actorId: $approverId,
                ipAddress: $ip,
            );

            return $invoice->fresh();
        });
    }
}

Implementation in Node.js

Prisma Schema

model AuditEvent {
  id              String   @id @default(uuid())
  aggregateType   String   @map("aggregate_type")
  aggregateId     String   @map("aggregate_id")
  sequenceNumber  Int      @map("sequence_number")
  eventType       String   @map("event_type")
  payload         Json
  metadata        Json
  actorId         String   @map("actor_id")
  actorType       String   @map("actor_type") @default("user")
  ipAddress       String?  @map("ip_address")
  checksum        String
  occurredAt      DateTime @map("occurred_at")
  createdAt       DateTime @default(now()) @map("created_at")

  @@unique([aggregateType, aggregateId, sequenceNumber])
  @@index([aggregateType, aggregateId])
  @@index([eventType])
  @@index([occurredAt])
  @@index([actorId])
  @@map("audit_events")
}

Audit Event Service

import { PrismaClient } from '@prisma/client';
import crypto from 'crypto';

const prisma = new PrismaClient();

interface RecordEventParams {
  aggregateType: string;
  aggregateId: string;
  eventType: string;
  payload: Record<string, unknown>;
  actorId: string;
  actorType?: string;
  ipAddress?: string;
}

export async function recordAuditEvent(params: RecordEventParams) {
  return prisma.$transaction(async (tx) => {
    const lastEvent = await tx.auditEvent.findFirst({
      where: {
        aggregateType: params.aggregateType,
        aggregateId: params.aggregateId,
      },
      orderBy: { sequenceNumber: 'desc' },
    });

    const sequenceNumber = lastEvent ? lastEvent.sequenceNumber + 1 : 1;
    const previousChecksum = lastEvent?.checksum ?? 'GENESIS';

    const eventData = {
      aggregate_type: params.aggregateType,
      aggregate_id: params.aggregateId,
      sequence_number: sequenceNumber,
      event_type: params.eventType,
      payload: params.payload,
      actor_id: params.actorId,
      occurred_at: new Date().toISOString(),
    };

    const checksum = crypto
      .createHash('sha256')
      .update(previousChecksum + JSON.stringify(eventData))
      .digest('hex');

    return tx.auditEvent.create({
      data: {
        aggregateType: params.aggregateType,
        aggregateId: params.aggregateId,
        sequenceNumber,
        eventType: params.eventType,
        payload: params.payload,
        metadata: {
          previous_checksum: previousChecksum,
          schema_version: '1.0',
        },
        actorId: params.actorId,
        actorType: params.actorType ?? 'user',
        ipAddress: params.ipAddress,
        checksum,
        occurredAt: new Date(),
      },
    });
  });
}

export async function verifyAuditChain(
  aggregateType: string,
  aggregateId: string
): Promise<boolean> {
  const events = await prisma.auditEvent.findMany({
    where: { aggregateType, aggregateId },
    orderBy: { sequenceNumber: 'asc' },
  });

  let previousChecksum = 'GENESIS';

  for (const event of events) {
    const eventData = {
      aggregate_type: event.aggregateType,
      aggregate_id: event.aggregateId,
      sequence_number: event.sequenceNumber,
      event_type: event.eventType,
      payload: event.payload,
      actor_id: event.actorId,
      occurred_at: event.occurredAt.toISOString(),
    };

    const expectedChecksum = crypto
      .createHash('sha256')
      .update(previousChecksum + JSON.stringify(eventData))
      .digest('hex');

    if (expectedChecksum !== event.checksum) {
      return false;
    }

    previousChecksum = event.checksum;
  }

  return true;
}

export async function getAggregateHistory(
  aggregateType: string,
  aggregateId: string
) {
  return prisma.auditEvent.findMany({
    where: { aggregateType, aggregateId },
    orderBy: { sequenceNumber: 'asc' },
  });
}

Express Route for Audit Queries

import { Router } from 'express';

const router = Router();

router.get('/audit/:aggregateType/:aggregateId', async (req, res) => {
  const { aggregateType, aggregateId } = req.params;

  const history = await getAggregateHistory(aggregateType, aggregateId);

  res.json({
    aggregate: { type: aggregateType, id: aggregateId },
    eventCount: history.length,
    events: history.map((e) => ({
      sequence: e.sequenceNumber,
      type: e.eventType,
      payload: e.payload,
      actor: e.actorId,
      occurredAt: e.occurredAt,
    })),
  });
});

router.get('/audit/:aggregateType/:aggregateId/verify', async (req, res) => {
  const { aggregateType, aggregateId } = req.params;
  const valid = await verifyAuditChain(aggregateType, aggregateId);

  res.json({
    aggregate: { type: aggregateType, id: aggregateId },
    chainValid: valid,
    verifiedAt: new Date().toISOString(),
  });
});

Retention and Archival Strategy

Audit logs grow fast. A system processing 10,000 transactions per day generates millions of events per year. You need a retention strategy:

Hot storage (0–90 days): Primary database. Full query capability. Used for day-to-day operations and recent audits.
Warm storage (90 days–2 years): Move older events to a read-optimised store (e.g., partitioned table, read replica, or columnar database). Still queryable but not on the critical path.
Cold storage (2+ years): Archive to object storage (S3, GCS) in a structured format (Parquet, JSON Lines). Retained for regulatory compliance periods (typically 5–7 years for financial data).

Laravel Archival Command

class ArchiveAuditEvents extends Command
{
    protected $signature = 'audit:archive {--older-than=90}';

    public function handle(): void
    {
        $cutoff = now()->subDays((int) $this->option('older-than'));

        $events = AuditEvent::where('occurred_at', '<', $cutoff)
            ->orderBy('occurred_at')
            ->cursor();

        $batch = [];
        $batchCount = 0;

        foreach ($events as $event) {
            $batch[] = $event->toArray();

            if (count($batch) >= 10000) {
                $this->uploadBatch($batch, $batchCount);
                $batch = [];
                $batchCount++;
            }
        }

        if (!empty($batch)) {
            $this->uploadBatch($batch, $batchCount);
        }

        AuditEvent::where('occurred_at', '<', $cutoff)->delete();

        $this->info("Archived and purged events older than {$cutoff->toDateString()}.");
    }

    private function uploadBatch(array $batch, int $index): void
    {
        $filename = sprintf('audit-archive/%s/batch-%04d.json', now()->format('Y/m'), $index);

        Storage::disk('s3')->put(
            $filename,
            collect($batch)->map(fn ($e) => json_encode($e))->implode("\n")
        );
    }
}

Reconciliation with Immutable Logs

One of the most valuable outcomes of immutable logs is automated reconciliation. Instead of manually comparing spreadsheets, you can programmatically verify that every financial event has a matching counterpart:

class ReconciliationService
{
    public function reconcilePayments(string $date): array
    {
        $internalEvents = AuditEvent::where('event_type', 'payment.completed')
            ->whereDate('occurred_at', $date)
            ->get()
            ->keyBy(fn ($e) => $e->payload['gateway_reference']);

        $bankStatements = $this->fetchBankStatements($date);

        $matched = [];
        $unmatchedInternal = [];
        $unmatchedBank = [];

        foreach ($bankStatements as $statement) {
            if ($internalEvents->has($statement->reference)) {
                $matched[] = $statement->reference;
                $internalEvents->forget($statement->reference);
            } else {
                $unmatchedBank[] = $statement;
            }
        }

        $unmatchedInternal = $internalEvents->values()->all();

        return [
            'date' => $date,
            'matched_count' => count($matched),
            'unmatched_internal' => count($unmatchedInternal),
            'unmatched_bank' => count($unmatchedBank),
            'discrepancies' => [...$unmatchedInternal, ...$unmatchedBank],
        ];
    }
}

Production Results

After implementing immutable audit logging across financial platforms:

Metric	Before	After
Reconciliation time (weekly)	4+ hours manual	15 minutes automated
Audit preparation time	2–3 weeks	2 days
Unresolved discrepancies	8–12 per month	0–1 per month
Compliance audit findings	Multiple per review	Zero material findings
Data dispute resolution time	3–5 days	< 1 hour

Key Takeaways

Never update financial records in place. Append new events instead. The current state is a projection of the event history.
Use hash chains for tamper evidence. Each event's checksum includes the previous event's checksum. Any modification breaks the chain.
Record who, what, when, and why. Every event needs an actor, a timestamp, and the business context. IP addresses and session identifiers add forensic value.
Plan for retention from day one. Audit logs grow quickly. Tiered storage (hot/warm/cold) keeps costs manageable while satisfying compliance requirements.
Automate reconciliation. Immutable logs enable programmatic reconciliation that replaces hours of manual spreadsheet comparison.
Write events and business data in the same transaction. If they are not atomic, you risk inconsistency between your source of truth and your audit trail.

Conclusion

Audit-proof financial workflows are not about generating more logs — they are about designing systems where every state change is captured, chained, and verifiable. Append-only writes, hash-chain integrity, and tiered retention create a system that answers any auditor's question with evidence, not explanations.

Whether you build in Laravel or Node.js, the patterns are the same. This infrastructure pays dividends every time a regulator asks a question or a discrepancy needs resolution.

From Delayed Data to Real-Time Decisions: Building an Event-Driven Pipeline on AWS

Thu, 19 Mar 2026 21:11:10 +0000

At 2:17 AM, the dashboard wasn’t moving.

Transactions were still coming in, but the numbers on the screen were frozen. The system was running, but the data was already 10 minutes behind.

At that scale, 10 minutes wasn’t just delay — it meant decisions based on outdated information.

That was the moment everything had to change.

The Problem: When Data Arrives Too Late

In many systems, data pipelines are built around batch processing or synchronous APIs. They work — until they don’t.

As traffic grows, these systems start to break in predictable ways:

Increasing latency
Bottlenecks in database reads
Inconsistent data across services
Delayed analytics and reporting

The core issue is simple:
the system reacts too late.

To solve this, the architecture had to shift from request-driven to event-driven.

The Shift: Thinking in Events

Instead of asking for data, services react to events as they happen.

Every transaction becomes an event. Every state change is recorded as a fact.

This changes everything:

Systems become asynchronous
Components are loosely coupled
Scalability comes from partitioning, not vertical scaling

The goal was clear:

Process more than 15,000 events per second with sub-50ms latency — reliably.

Designing the Pipeline

The first version of the system was built as a distributed architecture using open-source tools.

Core components:

- Apache Kafka for event streaming

- Kafka Streams for real-time processing

- Spring Boot for processing services

- PostgreSQL for durable storage

- Redis for low-latency reads

- Prometheus and Grafana for observability

Event Flow

The pipeline follows a simple but powerful structure:

Producer → Kafka → Stream Processing → Storage → Analytics

Producers publish transaction events
Events are serialized and validated
Kafka distributes them across partitions
Stream processors handle transformations and aggregations
Results are stored and made available for real-time queries

This allowed continuous processing instead of waiting for batches.

The Challenges (Where Systems Usually Break)

Designing the pipeline was only part of the work. Making it reliable at scale required solving real issues:

Event Duplication
Ensuring exactly-once processing required transactional guarantees and careful handling of offsets.
Latency Spikes
Under heavy load, consumer lag increased. This was mitigated with parallel consumers and optimized batching.
Throughput Optimization
Producer batching (32KB) and Snappy compression significantly improved performance.
Resource Bottlenecks
Connection pooling and efficient serialization reduced pressure on downstream systems.

These are the kinds of problems that don’t show up in diagrams — only in real systems.

Results

After multiple iterations and optimizations, the system achieved:

Throughput: 15,000+ events per second

Latency (P99): under 50ms

Availability: 99.95%

Data loss: 0% (exactly-once processing)

More importantly, this enabled near real-time anomaly detection.

What used to take minutes could now happen in milliseconds.

Moving to AWS: Same Architecture, Less Infrastructure

Once the system was stable, the next step was moving to the cloud.

The key decision was not to redesign the system — only to replace infrastructure with managed services.

Cloud Architecture:

Event ingestion via Amazon EventBridge or Amazon MSK
Processing with AWS Lambda
Workflow orchestration using AWS Step Functions
Storage with DynamoDB or Amazon RDS
Monitoring through Amazon CloudWatch

Each component maps directly to the original architecture.

The logic stays the same. The operations overhead disappears.

The Key Insight

The most important lesson wasn’t about tools.

It was about understanding the system.

Event-driven architectures are built on a few core principles:

Events are immutable
Systems react asynchronously
Scalability comes from partitioned streams
State is derived from event logs

Once these are clear, moving between technologies becomes straightforward.

Final Thoughts

Real-time systems are not about speed alone.

They are about enabling better decisions.

When data arrives late, the system might still work — but the business doesn’t.

Designing event-driven pipelines is ultimately about reducing the gap between what happens and what the system knows.

Design, therefore I exist.

Schema-First Approach with ThingsDB

Fri, 12 Dec 2025 06:48:09 +0000

In many architectures, the separation between database and application code introduces a dangerous gap where data integrity can fail. The application code is responsible for validation, but what if multiple services access the same data?

This post explores a different approach. We'll show you how to leverage ThingsDB's core features—Typed Collections, Procedures, and Rooms—to make the database itself the enforcing authority.

Follow along as we set up a secure, event-driven API for an Order application, demonstrating how to lock down access and ensure every data change is both valid and consistent.

1. Setting the Stage: The Blank Slate

We begin by creating a fresh collection for our application.

(execute the code snippet in the /thingsdb scope)

new_collection('OrderApp');

2. Understanding Typed Collections (The Core Defense)

The first step toward protection is establishing a schema. By default, a new ThingsDB collection starts with an empty, unrestricted object (a basic "thing") at its root. Any user with write access can add anything to it.

We want to lock this down. We create a type, App, to serve as the structured blueprint for the collection's entire existence.

(execute the code snippet in the //OrderApp scope)

// 1. Create a type for the root of my collection
new_type('App');

// 2. Convert the collection root to the 'App' type
.to_type('App');

Why this matters: The collection is now restricted by the App type. No data can be added or modified unless the modification conforms to changes made to the App type definition. This is your primary schema defense.

3. Defining the Data Model

Next, we define the structure of the data we want to store—the Order object—and where to keep it within the App root.

(execute the code snippet in the //OrderApp scope)

// Create an enumerator for consistent order status values
set_enum('OrderStatus', {
    OPEN: 'Open',
    PENDING: 'Pending',
    CLOSED: 'Closed',
});

// Define the Order structure
set_type('Order', {
    id: '#',               // Return ID as `id` when wrapped
    name: 'str',
    price: 'float',
    status: 'OrderStatus', // Enforcing status consistency
    created: 'datetime',
});

// Update the App type: Add a property 'orders' as an array of Order objects
mod_type('App', 'add', 'orders', '[Order]');

The mod_type(..) call automatically creates the orders property as an empty list on the collection root, ready to hold our data.

4. Establishing an Event Channel

For a modern microservice or component-based application, polling for data changes is inefficient. We set up an event channel using a Room to enable real-time communication.

(execute the code snippet in the //OrderApp scope)

// Add a room property to the App type for event handling
mod_type('App', 'add', 'order_events', 'room');

// For easy identification, we give the room a fixed name
.order_events.set_name('api_order_events');

Benefit: Any component can now join this room and instantly react to events like new-order or change-order-status without needing to constantly query the collection.

5. Encapsulating Logic with Procedures (The API Layer)

To guarantee that only valid changes are made to the collection, we expose all actions only through Procedures. We prefix these procedures with api_ for easy security whitelisting later.

(execute the code snippet in the //OrderApp scope)

// 1. Add a new order (with validation on input types)
new_procedure('api_order_add', |name, price| {
    order = Order{
        name:,
        price:,
    };
    .orders.push(order);

    // Emit the 'new-order' event for all listeners
    .order_events.emit('new-order', order.wrap());
    nil;
});

// 2. Change order status
new_procedure('api_order_set_status', |order_id, status| {
    // Input validation: ensures the new status is a valid enum value
    new_status = OrderStatus{||status.upper()};

    // Access and update the specific Order object
    Order(order_id).status = new_status;

    // Emit the 'change-order-status' event
    .order_events.emit('change-order-status', order_id, new_status.value());
    nil;
});

// 3. Retrieve orders by status
new_procedure('api_orders_by_status', |status| {
    order_status = OrderStatus{||status.upper()};
    .orders.filter(|order| order.status == order_status).map_wrap();
});

6. Strict Security: The Locked-Down API User

The final and most crucial step is to create a user that can only execute these specific procedures. This prevents any external service from bypassing your defined business logic.

(execute the code snippet in the /thingsdb scope)

// Create a new user for microservice access
new_user('api');

// Grant access to the collection:
// CHANGE (to modify data), JOIN (to listen to events), RUN (to execute procedures)
grant('//OrderApi', 'api', CHANGE|JOIN|RUN);

// Whitelist: This is the security lock.
// The user can only execute procedures and join rooms starting with 'api_'
whitelist_add('api', 'procedures', /^api_.*/);
whitelist_add('api', 'rooms', /^api_.*/);

// Generate and return the authentication token
new_token('api');

With this setup, the api user is completely restricted. It cannot arbitrarily delete the collection, change its structure, or modify data without going through the validated input and logic defined in your api_ procedures. Your collection is now truly protected by code!

In the next post, we will switch gears to the client side. We will demonstrate how a microservice uses the generated token to connect to the OrderApp collection, how to perform the authenticated api_ queries, and, crucially, how to "listen" to the real-time event changes emitted through the api_order_events room.