Coder Legion - Recent posts tagged distributedsystems

Pub/Sub Isn't Event-Driven Architecture: Here's What Most Teams Miss

Fri, 23 Oct 2026 14:38:21 +0000

Pub/Sub is a messaging pattern. Event-Driven Architecture is an architectural discipline. Every EDA implementation may use Pub/Sub, but not every Pub/Sub implementation is truly Event-Driven Architecture.

For many years, I would have agreed with the statement that Pub/Sub equals Event-Driven Architecture (EDA).

Why? Because early discussions around EDA focused primarily on three characteristics:

Real-time communication
Publish/Subscribe messaging
Asynchronous processing

At that time, implementing a Pub/Sub broker was often considered synonymous with implementing EDA.

Today, however, the answer is no.

As organizations have adopted EDA at scale, the discipline has evolved from a messaging concept into a comprehensive architectural practice. Modern EDA encompasses far more than simply publishing and subscribing to messages.

📡EDA Is More Than Messaging

A mature Event-Driven Architecture consists of multiple foundational elements:
📝Event Definition

Events must be clearly defined and categorized.

Examples include:
Business Events: CustomerRegistered, OrderPlaced, PaymentCompleted
System Events: FileUploaded, ServiceStarted, DatabaseUpdated, EOD

Without a consistent event model, organizations quickly end up with thousands of loosely defined messages that are difficult to understand and maintain.

👤Event Ownership

Just like data, events are business assets.

Every event should have a clearly identified owner responsible for:
Event definition
Schema management
Versioning
Lifecycle governance

When nobody owns an event, everyone assumes somebody else does.
📏Event and Queue Standards

Topic (event) and queue naming conventions form the foundation of a scalable EDA platform.
A well-defined naming structure enables:
Event discoverability
Consistent routing
Easier troubleshooting
Better governance
Without standards, the event landscape quickly becomes chaotic.

🔀Event Flow Design

EDA is not merely about sending messages.

It is about how events collaborate to achieve business outcomes.

Modern architectures require a clear understanding of:
Event relationships
Event choreography
Event dependencies
Business process visibility

While services remain loosely coupled, the overall event flow must still be intentionally designed.

🛡️Event Governance

Governance is the difference between a sustainable platform and an uncontrolled messaging ecosystem.

Effective governance covers:
Schema management
Data classification
Security controls
Access management
Event lifecycle management
Compliance and auditing

Governance should be embedded throughout the entire lifecycle, from design through implementation and operations.

📬Pub/Sub Is Only One Pattern Within EDA

Pub/Sub is an important communication pattern, but it is only one building block of a broader architecture.
A Pub/Sub platform without governance, ownership, standards, and event design is simply a collection of disconnected messages.

In such environments, organizations commonly encounter problems such as:
⚠️Consumer Failures

Consumers may crash when:
Unexpected event types are received
Event schemas change without coordination
Required fields are removed or modified
Data Exposure Risks

Sensitive information, including Personally Identifiable Information (PII), may be unintentionally delivered to systems or teams that should not have access to it.
Message Loss or Misrouting

Poorly governed subscriptions can result in:
Events being consumed by unintended systems
Important messages being ignored
Business processes becoming unreliable
Rising Infrastructure Costs

Unnecessary event distribution increases:
Network traffic
Broker workload
Infrastructure consumption
Operational costs
What begins as architectural flexibility can quickly become architectural sprawl.

💡 Final Thoughts

Many organizations begin their event-driven journey by deploying a message broker and enabling Pub/Sub communication. While this is an important first step, it is only the foundation.
The real challenge is not moving events between systems. The challenge is managing events as long-lived business assets.

Without clear ownership, governance, standards, and lifecycle management, organizations often end up with an event ecosystem that is difficult to understand, secure, and evolve. Teams gain asynchronous communication, but lose visibility, consistency, and control.

Successful Event-Driven Architecture is not measured by how many events are produced each day. It is measured by how effectively those events enable business capabilities while remaining discoverable, governed, and trusted across the enterprise.

In other words:
A message broker can make your systems event-enabled.
Governance, ownership, and standards make your organization truly event-driven.

❓ What do you think? Have you seen organizations mistake Pub/Sub implementation for Event-Driven Architecture? What governance challenges emerged as event adoption grew?

This typically drives significantly more comments and discussion from architects, developers, and integration professionals.

🚦 EDA Without Governance: A Fast Track to Enterprise Chaos

Thu, 15 Oct 2026 16:03:25 +0000

In today's digital enterprise, Event-Driven Architecture (EDA) has become a preferred architectural pattern for building scalable, loosely coupled, and real-time systems. By enabling applications to publish and subscribe to events independently, EDA provides the flexibility and agility required to support modern business demands.

However, there is a critical truth that many organizations discover only after their event ecosystem starts to grow:

⚠️ The success of EDA depends less on technology and more on governance.

Without proper governance, an event bus can quickly evolve into a chaotic integration landscape where dependencies are unclear, ownership becomes fragmented, and changes introduce widespread downstream impacts.

🌐 EDA Is More Than Event Distribution

EDA is fundamentally a distribution design that allows applications to publish and consume events across the enterprise.

✅ New publishers can share information with multiple consumers.
✅ New consumers can subscribe to existing event streams.
✅ Applications remain loosely coupled and independently deployable.

But this flexibility comes with responsibility.

An event published today may be consumed by numerous applications, either directly or indirectly. As adoption grows, every design decision becomes part of a shared enterprise contract.

Without governance, what starts as flexibility can quickly become complexity.

⚠️ The Hidden Risk of Uncontrolled Growth

When onboarding new applications into an event-driven ecosystem, governance standards and processes should be established before implementation begins.

Without these controls:

🔴 Topic structures become inconsistent.
🔴 Applications subscribe to unnecessary events.
🔴 Event routing becomes difficult to understand and maintain.
🔴 Sensitive data may be exposed to unauthorized consumers.
🔴 Small changes trigger large-scale application rework.

The impact is not limited to event broker configurations.

A seemingly simple change to an event topic, payload structure, or routing pattern may require modifications across multiple consuming applications, testing cycles, operational procedures, and business processes.

In large-scale enterprises, a single event redesign can affect dozens of interconnected systems.

🔄 Governance Must Cover the Entire Lifecycle

EDA governance is not a one-time architecture review.

It should span the entire delivery lifecycle:

📋 Design
👨💻 Development
🧪 Testing
🚀 Deployment
⚙️ Production Operations

Any governance gap can result in:

❌ Increased technical debt
❌ Integration failures
❌ Security and compliance risks
❌ Operational complexity
❌ Higher maintenance costs

The most successful event-driven organizations balance architectural flexibility with disciplined governance.

🎯 Key Areas of EDA Governance

🏷️ 1. Topic Naming Standards

Topics are the foundation of event discoverability and maintainability.

Every topic should follow a mandatory naming standard that clearly identifies:

✅ Business domain
✅ Business capability
✅ Event type
✅ Environment

A consistent naming convention helps architects, developers, and support teams quickly understand event purpose and ownership.

Good governance starts with naming things correctly.

📬 2. Queue Subscription Governance

Applications should subscribe only to events that are required for their business functions.

Governance should ensure:

✅ Controlled subscription approval
✅ Consumer registration and ownership
✅ Dependency visibility
✅ Periodic subscription reviews

This prevents unnecessary event consumption and reduces future change impacts.

Not every application needs every event.

🔀 3. Event Flow Design

A well-designed event flow enables systems to evolve without disrupting the broader ecosystem.

Governance should enforce:

✅ Standard routing principles
✅ Clear producer and consumer responsibilities
✅ Traceable event paths
✅ Flexible addition and removal of components

A properly governed event flow allows organizations to grow their architecture while maintaining control and predictability.

Flexibility should be intentional, not accidental.

🔐 4. Data Governance and Access Control

Not every consumer should have access to every event.

Organizations must ensure applications can subscribe only to authorized events, especially when they contain:

🔒 Personally Identifiable Information (PII)
🔒 Customer information
🔒 Financial transactions
🔒 Business-sensitive data

Strong governance ensures:

✅ Data privacy
✅ Regulatory compliance
✅ Security controls
✅ Consumer authorization

The principle should be simple:

🔑 Subscribe only to the data you are authorized to consume.

🏆 Governance Determines EDA Success

Many organizations focus heavily on event brokers, event meshes, streaming platforms, and messaging technologies.

While these technologies are important, they are not what ultimately determines the success of an Event-Driven Architecture.

The true differentiator is governance.

Well-defined standards, ownership models, review processes, security controls, and lifecycle management enable organizations to scale their event ecosystem confidently and sustainably.

Without governance, flexibility becomes chaos.

With governance, EDA becomes a strategic enabler of enterprise agility, innovation, and business growth.

💡 Final Thought

🚦 EDA without governance is simply message transportation.

🚀 EDA with governance becomes sustainable enterprise architecture.

The question is not whether your organization should adopt Event-Driven Architecture.

The real question is:

Do you have the governance needed to keep it from becoming enterprise chaos?

Keeping a Legacy Cache Consistent During an Incremental Migration

Wed, 14 Oct 2026 04:04:39 +0000

During an incremental system migration, I ran into a subtle consistency problem: our new service could successfully update the database while the legacy service continued reading stale data from its own in-memory cache.

The obvious solutions such as distributed locks, cache invalidation, or adding another piece of infrastructure didn't fit our constraints.

The solution ended up being much simpler: the legacy system already had a lease mechanism that we could reuse.

This is the story of how I found it, how we used it, and how we handled the failure cases.

Architecture

Our system currently has three main components:

Main Service: The client-facing service that sends business requests.
Legacy Service: An older, high-performance engine that maintains an embedded in-memory cache. It loads a subset of database tables into its own memory for fast access.
New Service: A modern service that is gradually replacing the Legacy Service using the Strangler Fig pattern.

The New Service currently acts as a proxy between the Main Service and the Legacy Service.

If an endpoint has already been migrated, the New Service handles the request directly. Otherwise, it forwards the request to the Legacy Service.

The architecture looks roughly like this:

The Problem

The problem appeared when the New Service handled a migrated request.

The New Service writes updates directly to the database. However, the Legacy Service has its own in-memory copy of the data and has no way of knowing that the database changed.

That creates a dangerous sequence:

The Legacy Service caches status = pending.
A request is migrated to the New Service.
The New Service updates the database to status = completed.
A later request reaches the Legacy Service.
The Legacy Service reads status = pending from its local cache.

The database is correct, but the Legacy Service is operating on stale state.

We discovered these issues during QA testing. As more endpoints moved to the New Service, the problem would become increasingly common.

So we needed a way to guarantee that the Legacy Service could not continue operating on stale data while a migrated endpoint was modifying the same resource.

The Constraints

The textbook solutions weren't a good fit for this system.

We had several constraints:

The Legacy and Main Services should not be modified. The Legacy Service is largely a black box, so modifying its behavior is risky.
The Legacy Service could not temporarily operate on stale data. For these operations, we needed to prevent conflicting legacy operations rather than accept eventual consistency.
The Legacy Service had limited cache synchronization capabilities. It automatically reloaded its cache only periodically, or through a manual refresh operation.
We wanted to avoid introducing new infrastructure. Adding something like Redis purely for distributed locking would introduce another dependency to provision, operate, and maintain.

These constraints ruled out several conventional approaches.

The Naive Solution

My first thought was:

What if the New Service acquires a distributed lock, performs the database update, and then triggers a cache refresh on the Legacy Service?

That sounds reasonable, but there were two problems.

1. The Legacy Service wouldn't know about the lock

An external distributed lock only coordinates operations if the participating systems can use it.

The Legacy Service was a black box and wouldn't check our new lock before accessing its local cache.

We could add that behavior, but that violated one of our main constraints.

2. Refreshing the entire cache was too expensive

The Legacy Service's manual refresh operation reloaded a large amount of data.

Doing that after every migrated request would introduce significant latency and unnecessary work.

We needed something more targeted.

The Solution: Reusing an Existing Lease

While investigating the Legacy Service's existing APIs, I found something useful: it already had a built-in lease mechanism.

The Legacy Service exposed APIs to:

Acquire a lease for a specific resource.
Release the lease when the operation was complete.

More importantly, releasing the lease had an existing side effect: the Legacy Service refreshed the cached data associated with that resource.

The lease mechanism was already part of how the Legacy Service protected its own critical sections.

This changed the problem completely.

Instead of introducing a new distributed lock, we could use the same coordination mechanism the Legacy Service already understood.

The New Service could wrap a migrated operation like this:

1. Acquire lease for the resource
2. Update the database
3. Perform business logic
4. Release the lease
5. Legacy Service refreshes that resource's cached data

The important part is that the lease wasn't merely an arbitrary lock exposed by the Legacy Service. The Legacy Service's own operations already align with the same lease mechanism.

That gave us two properties we needed:

Conflicting Legacy Service operations could not proceed while the resource was leased.
Releasing the lease caused the Legacy Service to refresh the affected resource rather than performing a full cache reload.

So instead of adding Redis, Kafka, or another synchronization system, I reused a primitive that already existed in the system.

What If Releasing the Lease Fails?

The happy path was straightforward.

But distributed systems rarely fail only on the happy path.

Consider this sequence:

New Service
    |
    | acquire lease
    v
Legacy Service
    |
    | lease acquired
    |
    v
New Service
    |
    | update database
    |
    X---- network failure ----> release request never arrives

The database update succeeded, but the release request never reached the Legacy Service.

If leases could remain indefinitely, the resource could become permanently locked.

So I looked at how the Legacy Service handled abandoned leases.

It already had two mechanisms:

TTL (Time-To-Live): Every lease automatically expires after a configured duration.
Background reaper: A background process periodically detects expired leases and cleans them up.

Even better, lease cleanup also triggered the same cache refresh behavior.

That meant the failure path became:

Acquire lease
     |
     v
Update database
     |
     X  release request lost
     |
     v
Lease expires
     |
     v
Background reaper
     |
     v
Lease cleared + cache refreshed

We therefore didn't need to build a separate retry system just to recover from a lost release request.

The existing lease lifecycle already provided a recovery mechanism.

An Important Invariant

The key correctness property here is that the resource remains protected by the lease until the Legacy Service releases or expires it.

As long as the database update completes before the lease is released or expires, the Legacy Service cannot resume normal access to the resource using its old cached state during that protected window.

When the lease is eventually cleaned up, the cache refresh brings the Legacy Service back in sync with the database.

This is an important distinction: the TTL isn't what provides consistency by itself. The consistency comes from the lease preventing conflicting access; the TTL and reaper provide failure recovery when the normal release path doesn't happen.

The Tradeoff

The solution wasn't free.

The main cost was latency.

If a resource has many related records that need to be refreshed, releasing the lease and synchronizing the cache can take noticeable time and I verified this through tests.

So I didn't view this as a permanent architecture.

It's a migration bridge.

During the migration, migrated endpoints use the lease mechanism to safely coordinate with the Legacy Service.

As more functionality moves to the New Service, individual resources eventually become fully owned by the New Service.

At that point, the Legacy Service no longer needs to read or modify those resources.

That's our exit condition.

Once a resource is completely owned by the New Service, we can remove the lease and cache-synchronization logic for it.

This is important because migration code has a tendency to become permanent if nobody defines when it can be deleted. So I documented this as well.

What I Learned

1. Start with invariants, not technologies

My first instinct was to think about distributed locks and infrastructure such as Redis.

But the more useful question was:

What must never happen?

In this case, the important invariant was that the Legacy Service must not perform conflicting operations against a resource while the New Service is modifying it.

Once that was clear, the technology choice became much easier.

2. Existing systems often contain useful primitives

At first, I thought of the Legacy Service as something we couldn't modify and therefore mostly had to work around.

But "don't modify the legacy system" doesn't mean "don't understand the legacy system."

By digging into its existing APIs and lease lifecycle, I found a mechanism that already provided most of the coordination and recovery behavior we needed.

Sometimes the best solution isn't introducing another system. It's discovering what the current system can already do.

3. Temporary migration code needs an exit condition

A migration introduces a lot of code that ideally won't exist forever.

The lease wrapper is useful while ownership is split between the Legacy and New Services. But once the New Service completely owns a resource, keeping the wrapper would only add unnecessary latency and complexity.

Defining the exact conditions under which the lease logic can be removed makes it much less likely that temporary migration infrastructure becomes permanent technical debt.

Final Thoughts

This experience changed how I think about solving distributed systems problems.

It's easy to reach for familiar tools like Redis for locking, Kafka for asynchronous communication, an outbox for reliable delivery.

Sometimes those are exactly the right tools.

But before introducing another component, it's worth understanding the guarantees and primitives that already exist in the system.

In this case, the legacy system already had a resource-level lease, automatic expiration, a background reaper, and targeted cache synchronization.

The challenge wasn't building a new distributed system.

The best solution wasn't to build another system around the legacy one. It was to understand the system we already had well enough to make it work for us.

Realtime Is a Hint, Not Your Source of Truth

Thu, 17 Sep 2026 08:20:15 +0000

A notification badge looks like a small feature. In a distributed application, it can expose a large architectural assumption.

I recently investigated a badge that sometimes stayed stale in an ASP.NET Core and Blazor application running across multiple server instances. The notification itself had been created successfully. Durable state was correct. A separate push path could also complete. Yet the visible count did not always change until the user left the screen and returned.

The bug was not really about arithmetic. It was about assigning too much authority to a realtime transport.

The hidden assumption

The UI initially loaded its count from the server and then relied on live events to remain current. That feels reasonable while developing against one server with a stable connection. In production, several ordinary conditions weaken the assumption:

The client can disconnect briefly.
A load balancer can place the sender and receiver on different instances.
A workflow can create the same durable state without passing through the page that happens to broadcast.
A transport or hub can fail after the database operation has already succeeded.
An idempotent retry can revisit work that should not increment the UI again.

None of these conditions makes realtime technology a poor choice. They simply mean that a realtime message should usually be treated as a hint that fresh state is available, not as the state itself.

A three-layer reliability pattern

The repair became much clearer once the responsibilities were separated into three layers.

First, persist the truth

The durable record must be committed independently of whether any connected client is listening. If the user closes the app, changes networks, or connects to another instance, the correct unread state must still be recoverable later.

This ordering matters. Broadcasting first can produce a convincing UI update for data that subsequently fails to commit. Persisting first means a missed broadcast can be healed; an invented broadcast cannot be trusted.

Second, broadcast from the business chokepoint

The live update originally depended on a narrower UI workflow. Other creation paths could produce the same durable result without emitting the event. Moving the broadcast to the shared dispatch path made the behavior consistent across interactive and background work.

The broadcast was also tied to the outcome of the durable write. A genuinely new record emits a hint. An idempotent replay that finds the work already completed does not. That prevents duplicate increments and makes the event reflect a real state transition rather than merely an attempted command.

Third, reconcile periodically

Even a correctly placed broadcast can be missed. The client therefore refreshes the authoritative count on a bounded interval. The loop is cancelled when the component is disposed, and a transient refresh failure does not kill future attempts.

This is not polling as the primary experience. Realtime still gives the fast path. Polling is the repair path.

The trade-off

Hybrid delivery is not free.

Periodic reconciliation adds read traffic. The interval also defines a maximum period of visible staleness when a realtime event is missed. A very short interval increases load and can waste battery or bandwidth on mobile clients. A long interval lowers cost but leaves stale UI visible for longer.

That makes the interval a product and operational decision, not a magic constant. Ask how damaging stale information is, how expensive the read is, how many active clients exist, and whether the endpoint can use a cheap aggregate or cache.

For a badge, bounded staleness may be acceptable. For a payment confirmation or permission change, the interaction may require an immediate authoritative read instead. The pattern stays the same, but the recovery budget changes.

Failure boundaries matter too. Once durable work has succeeded, a hub failure should not turn the whole operation into a failure. Otherwise a temporary transport problem can encourage retries of already-completed business work. Log the broadcast failure, observe it, and let reconciliation repair the presentation.

Test the contract, not the transport

The most useful tests were not attempts to prove that a network never fails. They pinned the business boundary:

New durable state produces one realtime hint.
Previously completed or deduplicated work produces no new hint.
A failing realtime transport does not invalidate successful durable work.

Client-side tests can add the complementary guarantees: an initial authoritative load occurs, the reconciliation loop refreshes state, transient errors do not stop later ticks, and disposal cancels the loop.

These tests document the intended hierarchy. The database-backed read is authoritative. Realtime reduces latency. Reconciliation restores convergence.

A practical review checklist

When reviewing a live-update feature, I now ask five questions:

What is the source of truth?
Where is the single shared point that knows a real state transition occurred?
Can the live event be safely missed or duplicated?
How and how quickly does the client converge again?
What happens if the transport fails after the business operation succeeds?

If those answers are vague, the feature may look realtime in a demo while remaining fragile in production.

The broader lesson is modest: reliability often comes from combining mechanisms with explicit roles. Durable storage provides truth. Realtime provides speed. Periodic reconciliation provides healing. None needs to pretend it can do the others’ job.

That separation made a small badge easier to trust, and it is a pattern I would reuse for dashboards, counters, presence indicators, job progress, and other derived UI state.

Why Local LLM Inference Needs Admission Control Before It Needs More Hardware

Fri, 11 Sep 2026 02:47:45 +0000

Running an LLM locally is easy to demonstrate.

Start Ollama or vLLM, load a model, send a request, and watch it generate tokens.

The interesting problems start when you send more than one kind of request.

Imagine a local inference server handling two workloads:

Interactive requests where a user is waiting for an answer.
Background jobs generating longer responses that can tolerate delay.

With plenty of spare capacity, both workloads coexist peacefully.

Under contention, they don't.

A few large background requests can occupy enough inference capacity that interactive requests wait behind work that was never latency-sensitive in the first place.

At that point, adding another HTTP worker or increasing a concurrency limit doesn't necessarily solve the problem.

You have a scheduling problem.

Concurrency Is Not Capacity Management

A common first approach is to put a semaphore around inference:

maximum concurrent requests = 4

That's useful. It prevents unlimited work from entering the system.

But it doesn't answer a more important question:

Which four requests should be allowed to run?

Suppose all four slots are occupied by long-running batch generations.

An interactive request arrives.

The concurrency limit is working exactly as designed, yet the user still waits.

The system protected itself from overload, but it didn't protect the workload that mattered most.

This is where admission control becomes different from simple concurrency limiting.

Concurrency asks:

How much work can execute simultaneously?

Admission control asks:

Which work should consume scarce capacity right now?

Tokens Make the Problem More Interesting

LLM workloads also aren't uniform.

Consider two requests:

Request A
input: 500 tokens
maximum output: 200 tokens

Request B
input: 2,000 tokens
maximum output: 4,000 tokens

Counting both as "one request" hides a large difference in potential resource consumption.

A better admission decision can account for both request concurrency and estimated token footprint.

Conceptually:

estimated footprint = input tokens + maximum output tokens

Before admitting a request, the controller asks whether enough capacity remains for that footprint.

The estimate doesn't need to predict the exact number of generated tokens perfectly. It needs to provide a useful bound for deciding whether admitting more work is safe.

When the request finishes, unused capacity can be returned.

This gives the system another dimension of protection beyond request count alone.

Static Limits Still Leave a Policy Problem

You can go further and reserve capacity.

For example:

Interactive: 3 slots
Batch:       1 slot

Now interactive traffic is protected.

But you've created another problem.

What happens when there are no interactive requests?

If three reserved slots sit unused while batch work waits, utilization suffers.

The obvious improvement is borrowing.

Let batch jobs temporarily consume unused interactive capacity:

Interactive demand low
        ↓
Batch borrows spare capacity
        ↓
Interactive demand returns
        ↓
Borrowed capacity must be restored

That last step is where things become interesting.

Borrowing idle capacity is easy.

Safely reclaiming it is the hard part.

Borrowing Without Restoration Isn't Really a Reservation

Suppose an interactive class is guaranteed three units of capacity.

During an idle period, batch workloads borrow two of them.

Then interactive traffic suddenly returns.

If those batch requests can't be interrupted, the interactive reservation doesn't immediately exist anymore. It exists only after the borrowers finish.

That means the actual guarantee depends on restoration behavior.

A useful reservation policy therefore needs to define more than:

interactive floor = 3

It also needs to answer:

How quickly can that floor become usable again?

This distinction matters especially with inference because some resources can be reclaimed immediately while others cannot.

A controller may stop granting new batch work immediately.

But an upstream inference request already running on a GPU may continue until completion.

So "capacity restored" can mean at least two different things:

The scheduler has restored the entitlement.
The physical resource is actually available again.

Those are not necessarily the same moment.

Testing This Against a Real Local Model

I've been experimenting with this problem while building MoFlux, an admission-control system for shared workloads.

One of the useful milestones was running the controller against a real local Ollama instance instead of only synthetic workloads.

The first result wasn't interesting because MoFlux somehow made the model faster.

It didn't.

The interesting result was that the admission policy could operate in front of real local inference and observe the same contention problem that appears in simulations.

That changes what can be tested.

Instead of asking only:

Does the controller make the correct decision?

we can ask:

What happens to real inference when interactive and batch workloads
compete for the same model?

Now measurements such as these become meaningful:

Time to first token
End-to-end latency
Interactive success rate
Batch throughput
Admission rejection rate
Capacity restoration time
Protected-capacity violations

This is where benchmarks start becoming more useful than raw throughput numbers.

Throughput Is Not Always the Goal

Imagine two configurations.

Configuration A completes 110 requests per minute.

Configuration B completes 100.

At first glance, A wins.

But suppose under A:

interactive latency becomes unpredictable,
user-facing requests frequently miss their SLO,
background work dominates capacity during bursts.

Configuration B might complete fewer total requests while completing substantially more useful requests within their objectives.

That's the difference between throughput and goodput.

For shared inference infrastructure, the question isn't simply:

How many generations can this GPU complete?

A more useful question is:

How much useful work can this GPU complete while preserving the
guarantees of the workloads sharing it?

That is a capacity-allocation problem.

Why This Matters for Local Inference

Local inference is becoming increasingly practical.

Developers are running models on workstations, edge machines, private servers, and internal GPU clusters.

The first optimization instinct is often hardware-oriented:

Need more throughput?
Buy a faster GPU.

Sometimes that's the right answer.

But if the actual problem is uncontrolled contention between workloads, more hardware can simply postpone the same failure mode.

Before scaling hardware, it's worth understanding:

Who is consuming capacity?

How much are they allowed to consume?

Which workloads need protection?

Can idle capacity be borrowed?

How quickly can it be reclaimed?

What happens when demand suddenly changes?

Those questions exist whether your inference server is a laptop running Ollama or a production cluster serving thousands of requests.

The scale changes.

The scheduling problem doesn't.

The Bigger Lesson

A single local LLM request is mostly a model-serving problem.

A shared inference server is a distributed-systems problem.

Once multiple workloads compete for a finite model, GPU, memory budget, or concurrency pool, you need policy.

Rate limiting controls how often work arrives.

Concurrency limiting controls how much work runs simultaneously.

Admission control decides whether the system should accept the work at all, given its current obligations.

And workload isolation decides whose work gets protected when capacity becomes scarce.

You may eventually need another GPU.

But before buying one, send your local model two very different workloads at the same time.

You might discover that the first thing you need isn't more capacity.

It's a better way to decide who gets to use the capacity you already have.

Foundations of System Design: Learn It the Way I Wish Someone Taught Me

Sun, 06 Sep 2026 13:32:05 +0000

When I first started learning system design, I felt overwhelmed.

Load balancers. Caches. Message queues. Replication. Sharding. Distributed systems.

There are a lot of terms. And most guides throw them at you like a vocabulary test.

But here is the thing: I don't think the best way to learn system design is to memorize diagrams or vocabulary.

I prefer starting with something much simpler:

Start with one request. Then introduce a new concept only when the system actually needs it.

That is what we are going to do in this guide.

We will start with a very small application and gradually make it capable of handling more traffic and more failures. Every new concept we introduce will be because the system actually broke and needed it.

If you have ever felt like system design is "too advanced" or "not for you" yet, I promise you it is. You just need the right entry point.

Let's build it together.

What We Will Cover

What system design actually means (and why it is not just for senior engineers)
How a browser request travels to your server
Where the database fits and why relational databases are useful
Transactions and why they matter
Scaling: vertical and horizontal
Load balancers and how they distribute traffic
Health checks and detecting failed servers
Redundancy and why it matters
The architecture we built and the problems we have not solved yet
The mental model for approaching any system design problem

Let's start.

What Is System Design?

In simple terms, system design is about deciding how the different parts of a software system should work together as the system grows.

When your application has 10 users, almost any reasonable architecture can work.

When it has 100,000 users, things start getting interesting.

Your server has limited CPU and memory.

Your database has limits.

Networks fail.

Servers crash.

Requests take longer than expected.

Two users can try to change the same data at the same time.

So system design is not just:

"How do I make this application work?"

It becomes:

"How do I make this application continue working as traffic, data, and failures increase?"

If that question excites you, you are in the right place.

A Request From the Browser

Imagine you have a frontend application running in the browser.

The browser needs a list of products.

It sends a request:

Browser
   |
   | GET /products
   ↓
api.example.com

But api.example.com is not the server's IP address.

It is a domain name.

The browser needs to find the IP address associated with that domain.

This is where DNS (Domain Name System) comes in.

Very roughly:

Browser
   |
   | "Where is api.example.com?"
   ↓
DNS
   |
   | "It is available at this IP"
   ↓
IP address
   |
   ↓
Server

DNS is basically the system that helps translate human-readable domain names into network addresses.

You do not need to remember the entire DNS process yet.

For now, just remember:

Domain name → DNS resolution → IP address → server

The Server Processes the Request

Now the request reaches our server.

The server receives:

GET /products

It needs to retrieve the products from the database.

So our system currently looks like this:

Client
   |
   | Request
   ↓
Server
   |
   | Query
   ↓
Database

The database returns the data:

Database
   |
   | Products
   ↓
Server
   |
   | JSON response
   ↓
Client

The complete flow is:

Client
   ↓
Server
   ↓
Database
   ↓
Server
   ↓
Client

At this point, everything looks simple.

And that is exactly what we want.

Do not introduce a load balancer, Redis, Kafka, Kubernetes and 15 other things when one server is perfectly capable of handling the application.

Where Does the Database Fit?

Let us say our application has these entities:

Users
Products
Orders
Accounts
Sessions

This data has structure.

A user has an ID, name and email.

A product has an ID, name and price.

An order belongs to a user and contains products.

This is where a relational database can be a very good fit.

Examples include:

PostgreSQL
MySQL
Oracle
SQLite

Relational databases organize data into tables, which contain rows and columns.

For example:

users

id: 1
name: Noor
email: Emails are not allowed
id: 2
name: Ali
email: Emails are not allowed

And:

products

id: 1
name: Keyboard
price: 100
id: 2
name: Mouse
price: 50

The important thing is not memorizing the names of databases.

The important question is:

What kind of data and guarantees does my application need?

Why Relational Databases Are Useful

One major advantage of relational databases is that they let us model relationships between data.

Suppose Noor buys a keyboard.

We might have:

users
  ↓
orders
  ↓
order_items
  ↓
products

The database can represent those relationships explicitly.

We can then query information such as:

Which products did this user purchase?

This is where SQL and operations such as JOIN become useful.

For example, conceptually:

SELECT users.name, products.name
FROM users
JOIN orders ON orders.user_id = users.id
JOIN order_items ON order_items.order_id = orders.id
JOIN products ON products.id = order_items.product_id;

You do not need to memorize this query.

The important idea is that relational databases are very good when your data has clear relationships and you need strong consistency around those relationships.

Transactions

Now we have reached our first really important system-design problem.

Imagine a user places an order.

Two things need to happen:

1. Create the order
2. Deduct the user's balance

What happens if step 1 succeeds but step 2 fails?

We could end up with:

Order created
Money not deducted

That is an invalid state for many applications.

This is where transactions become important.

A transaction allows us to group multiple database operations into one logical operation.

Conceptually:

BEGIN TRANSACTION

Create order
Deduct balance

COMMIT

If something goes wrong:

BEGIN TRANSACTION

Create order
Deduct balance ← FAIL

ROLLBACK

The database can roll back the changes made by the transaction.

The simple mental model is:

Either the required operations succeed together, or the transaction does not commit their changes.

Transactions are built around the ACID properties:

A → Atomicity
C → Consistency
I → Isolation
D → Durability

You do not need to master ACID right now.

Just understand why transactions exist:

They help us keep related database operations in a valid state when something fails or when multiple operations happen concurrently.

Our First Bottleneck

So far we have:

                  ┌──────────┐
Client ────────→ │  Server  │
                  └────┬─────┘
                       │
                       ↓
                  ┌──────────┐
                  │ Database │
                  └──────────┘

Now imagine our application becomes popular.

We start getting:

10 requests/second

Then:

100 requests/second

Then:

1,000 requests/second

Our server has limited resources.

It has:

CPU
RAM
Network
Disk

At some point, one machine may no longer be enough.

Now we have to think about scaling.

This is where system design starts becoming interesting.

Vertical Scaling

The first thing we can do is make the existing server stronger.

Maybe we currently have:

4 CPU cores
8 GB RAM

We could upgrade it to:

16 CPU cores
64 GB RAM

This is called vertical scaling.

In simple terms:

Make the machine bigger.

It is easy to understand and often the simplest solution.

But it has limits.

Problem 1: Hardware Has Limits

You cannot infinitely increase the CPU and RAM of one machine.

Eventually you reach the limits of the available hardware or the cost becomes unreasonable.

Problem 2: Single Point of Failure

We still have one server.

If that server goes down:

Server crashes
   ↓
Application unavailable

This is a single point of failure.

A single point of failure is a component whose failure can bring down an important part of the system.

So we have solved one problem but created another question:

Can we run multiple servers?

Horizontal Scaling

Instead of making one server bigger, we can add more servers.

For example:

         Server 1
       /
Client ── Server 2
       \
         Server 3

This is called horizontal scaling.

Instead of:

1 very powerful server

we have:

3 servers

Now if one server fails, the other servers may still be able to handle requests.

But we have created a new problem.

How does the client know which server to talk to?

We do not want clients manually choosing:

server-1.example.com
server-2.example.com
server-3.example.com

We need something in front of them.

Load Balancer

This is where a load balancer comes in.

A load balancer receives incoming requests and decides which server should handle each request.

Our architecture becomes:

                    ┌──────────┐
                    │ Server 1 │
                  /
Client → Load Balancer → Server 2
                  \
                    └──────────┘
                         Server 3

The client does not need to know about all the application servers.

It talks to the load balancer.

The load balancer distributes traffic between available servers.

Now we have:

Client
   ↓
Load Balancer
   ↓
┌────────┬────────┬────────┐
│Server 1│Server 2│Server 3│
└────────┴────────┴────────┘

This gives us more capacity and can improve availability.

But now we need to decide:

How does the load balancer choose a server?

Load-Balancing Algorithms

There is not one universal algorithm.

Different algorithms work better for different workloads.

Round Robin

The load balancer sends requests in sequence:

Request 1 → Server 1
Request 2 → Server 2
Request 3 → Server 3
Request 4 → Server 1

Simple and useful when servers have similar capacity and requests have roughly similar cost.

Weighted Round Robin

Maybe our servers are not equally powerful.

Server 1 → weight 1
Server 2 → weight 2
Server 3 → weight 3

The stronger server receives more traffic.

Least Connections

The load balancer sends a new request to the server currently handling the fewest active connections.

Server 1 → 30 connections
Server 2 → 12 connections
Server 3 → 21 connections

New request → Server 2

This can be useful when requests have different durations.

IP Hash

The load balancer hashes the client's IP address and uses the result to choose a server.

Conceptually:

Client IP
   ↓
Hash
   ↓
Server selection

This can provide a form of request affinity, although relying on IP alone has limitations.

Random

The load balancer randomly chooses an available server.

Simple, but whether it is appropriate depends on the workload and implementation.

Least Response Time

The load balancer can consider response times and direct traffic toward servers that are responding faster.

This can be useful when server load is not evenly distributed.

Consistent Hashing

Consistent hashing is useful in systems where we want related keys to map predictably to servers while minimizing how much mapping changes when servers are added or removed.

It becomes especially useful in distributed caches and partitioned systems.

Do not worry if that sounds complicated.

We will come back to it when we actually need it.

But What If a Server Is Dead?

Imagine we have:

Server 1 → healthy
Server 2 → healthy
Server 3 → crashed

We do not want the load balancer to continue sending traffic to Server 3.

This is where health checks come in.

The load balancer periodically checks whether a server is healthy.

For example:

GET /health

The server might respond:

{
  "status": "ok"
}

If the server stops responding or reports that it is not ready, the load balancer can stop sending new traffic to it.

So now:

              Load Balancer
               /         \
              ↓           ↓
         Server 1      Server 2
         healthy       healthy

         Server 3
        unhealthy
            X

This is much better than blindly trusting that every server is alive.

If you want a ready-made health check implementation you can drop into an Express, Fastify or Hono app, I built one for Blockend. It handles liveness and readiness checks out of the box:

https://blockend.noorulhassan.com/docs/02-blocks/07-health-check

Health Checks

A load balancer should not blindly send traffic to every server.

Instead, it should periodically check whether each server is healthy.

For example:

Load Balancer
       /         \
      ↓           ↓
 Server 1      Server 2
 healthy       healthy

 Server 3
unhealthy
    X

This is much better than blindly trusting that every server is alive.

If you want a ready-made health check implementation you can drop into an Express, Fastify, or Hono app, I built one for Blockend. It handles liveness and readiness checks out of the box:

https://blockend.noorulhassan.com/docs/02-blocks/07-health-check

Health Checks Are More Than "Is the Process Running?"

There is an important distinction here.

A server process can be running while the application is not actually ready to serve traffic.

For example:

Node process → running
Database     → unavailable

The process itself is alive.

But the application might not be able to perform useful work.

This is why production systems often distinguish between two different concepts:

Liveness
Readiness

Liveness

Liveness asks something like:

Is the application process alive?

Readiness

Readiness asks:

Is the application ready to receive traffic?

This distinction becomes important when deploying applications, restarting services, and handling failures.

Redundancy

We have now introduced another important idea:

Redundancy.

Instead of depending on one component:

Server

we have:

Server 1
Server 2
Server 3

If one fails, the others can continue serving traffic.

The same principle can apply to other parts of the system.

For example:

Multiple application servers
Multiple database replicas
Multiple cache nodes
Multiple availability zones

The goal is to avoid unnecessary single points of failure.

But redundancy creates another problem.

If we have three application servers, where is the user's session stored?

And if all three servers need the same data, how do they share it?

This is where our simple architecture starts becoming a distributed system.

The Architecture We Have Built

Let us stop here and look at how far we have come.

We started with:

Client → Server → Database

Then traffic increased.

We added vertical scaling.

Then we reached the limits of one machine.

We added horizontal scaling.

Then we needed something to distribute traffic.

We added a load balancer.

Then we needed to detect failed servers.

We added health checks.

Our architecture now looks roughly like:

                            ┌───────────┐
                            │ Database  │
                            └─────▲─────┘
                                  │
                                  │
                            ┌─────┴─────┐
                            │   Load    │
Client ─────────────────→ │ Balancer  │
                            └─────┬─────┘
                            ┌─────┼─────┐
                            ↓     ↓     ↓
                        Server Server Server
                           1     2     3

Notice something important.

We did not start with all of these components.

We earned each component by encountering a problem.

That is how I think system design should be learned.

The Problems We Have Not Solved Yet

Our system is better, but we are not finished.

We still have several problems.

Database Bottleneck

All servers are talking to one database.

What happens when database traffic becomes too high?

We might need:

Indexes
Query optimization
Connection pooling
Caching
Read replicas
Partitioning
Sharding

But we should not add these just because they exist.

We add them when the problem requires them.

Repeated Expensive Requests

Suppose thousands of users request the same product:

GET /products/123

Every request goes:

Server → Database

Why query the database 10,000 times if the data does not change frequently?

This is where caching becomes useful.

Background Work

Suppose creating an account also requires:

Send email
Generate report
Process image
Notify another service

Do we really want the HTTP request to wait for all of that?

Probably not.

This leads us toward:

Queues
Background workers
Asynchronous processing

Service Failures

What happens if another service is temporarily unavailable?

Do we retry?

How many times?

How quickly?

What if retries make the problem worse?

This leads us toward:

Timeouts
Retries
Backoff
Circuit breakers
Idempotency

Data Growth

What happens when our database grows from:

1 GB

to:

1 TB

Eventually we may need to think about:

Partitioning
Archiving
Replication
Sharding

And this is where system design starts becoming much deeper.

The Mental Model I Want You to Keep

If you are learning system design, do not try to memorize diagrams.

Instead, keep asking:

What is the bottleneck?

Then:

What failure can happen here?

Then:

What happens when traffic increases?

Then:

What happens when this component goes down?

Then:

What consistency does the application actually need?

Then:

What is the simplest solution that solves the problem?

That last question is important.

A system with:

Client
  ↓
Server
  ↓
PostgreSQL

is not a bad architecture simply because it does not have Kafka, Redis, Kubernetes, and 20 microservices.

If that system handles the workload reliably, it is a good system.

Good system design is not about adding more components.

It is about making the right tradeoffs for the problem you actually have.

Where We Go From Here

We started with one request:

Client
  ↓
Server
  ↓
Database

Then we introduced concepts only when we needed them:

More traffic
     ↓
Scaling
     ↓
Horizontal scaling
     ↓
Load balancer
     ↓
Health checks
     ↓
Redundancy

And this is only the beginning.

From here, we can start introducing:

Caching
   ↓
Database scaling
   ↓
Read replicas
   ↓
Queues
   ↓
Background workers
   ↓
Retries and backoff
   ↓
Idempotency
   ↓
Consistency
   ↓
Replication
   ↓
Partitioning
   ↓
Sharding
   ↓
Distributed systems

We will introduce each one when there is a real problem that requires it.

Because that is the way I wish someone had taught me system design.

Not:

"Here are 50 components. Memorize them."

But:

"Here is a system. Let's see what breaks when we push it."

That is where system design actually starts.

What You Learned

Let us take a step back.

You started with a single browser request and ended up with:

A server that processes requests
A relational database that stores structured data
Transactions that keep related operations consistent
Vertical scaling to handle more load
Horizontal scaling to handle even more
A load balancer that distributes traffic across servers
Health checks that detect failed servers
Redundancy that avoids single points of failure

And you did not just memorize these concepts.

You encountered each problem and solved it with the right tool.

That is the mental model. That is what matters.

The next time someone says "you need a load balancer," you will not just know what it is.

You will know why it exists.

And that makes all the difference.

The Bug That's Been Breaking Software for 40 Years Is Now Breaking Your AI Agents, Too

Tue, 25 Aug 2026 00:00:00 +0000

Picture a Java developer shipping an AI agent into production. The agent gets a prompt and starts working through a task, 50 steps, each one a call to a model like OpenAI or Anthropic, a database write, maybe a few dollars of compute. At step 49 of 50, something fails: a bug, a network blip, an AWS outage, a Kubernetes scale-down mid-run. What happens next, according to Yaron Schneider, CTO and co-founder of Diagrid, is the part nobody budgets for. "The agent is going to need to pick back up all the way from the beginning, and you're going to be paying for all of those LLM calls right from the very beginning," he said. "Which is a great way to go bankrupt."

That failure mode is what Diagrid's new integration, diagrid-spring-ai, is built to close. Announced this week, it brings durable execution to Spring AI, the framework Java developers already use to build agents, without asking them to rethink how they write one.

The same 40-year-old problem, with worse consequences

Schneider doesn't frame this as a new category of failure. It's the same reliability problem distributed systems have always had. "The exact same problems that made software unreliable for the past 40 years are still true for AI agents," he said. "Only with AI agents, the consequences are much bigger." A crashed script used to mean lost time. A crashed agent means lost time and a re-billed stack of model calls, at a cost that scales with however many steps the agent had already completed.

Schneider said the companies Diagrid works with see failures on roughly 30% of production agent runs, on average, not rare edge cases, but a routine cost of running agents at scale.

Zero code changes, by design

The integration's core pitch is unusually low-friction for infrastructure-level tooling. "With Spring AI, literally nothing," Schneider said, when asked what a developer has to change to adopt it. "All you need to do is pull in the package. There is zero code changes. You do not need to recompile your code." Underneath, the integration runs on Dapr Workflow, the open-source project Schneider originally built at Microsoft and later spun into Diagrid, now used by more than 30,000 companies, including over half the Fortune 500 and, notably, NASA.

How it actually avoids re-running side effects

The obvious follow-up question for anyone who's built distributed systems: how do you resume an interrupted agent without re-executing a step that already sent an email or wrote to a database? Diagrid's answer is an immutable ledger. "We keep an immutable record of everything that the agent did. We're essentially journaling," Schneider said. When an agent fails mid-run, the platform reads the journal, recovers the state, exactly what was completed, its inputs, outputs, and memory, and resumes from that point rather than replaying anything already finished.

Schneider was direct about the confidence customers should have in that recovery: "If they're using our technology, they know for 100%, for a fact, that we will always resume where they left off mid-run." He added that customers can and do test this themselves, running failure simulations in their own environments to verify the behavior before trusting it in production.

Why this isn't just a Spring AI feature

A natural question is why durable execution lives as a separate integration rather than something Spring AI ships with directly. Schneider's answer comes down to what a workflow engine actually requires. "Workflow engines are incredibly hard to get right. They require their own separate infrastructure, and Spring is just not that," he said. Bundling a database and deployment model into a lightweight developer framework would hurt adoption, not help it. Building that infrastructure in-house isn't a reasonable alternative either: Schneider pointed to Microsoft's own investment, a team of 15 engineers spending more than $10 million a year maintaining its workflow engine, as a sense of the actual cost most teams would take on trying to solve this themselves.

The claim Schneider thinks is overrated

Asked what's overstated in the broader AI agent conversation right now, Schneider didn't hesitate: model accuracy. "Models are going to continue getting better. They're going to continue being cheaper," he said. "I see a lot of emphasis being put on that instead of, you know, how do you orchestrate multiple agents needing to solve a task and all of them failing at the same time." In his view, the infrastructure problem, keeping agents reliable when things break, is a bigger and less-discussed risk than whatever incremental accuracy gain the next model release brings.

Beyond Spring

diagrid-spring-ai is one piece of a broader set of integrations Diagrid maintains across roughly ten agent frameworks spanning Python, C#, and Java, including LangChain, LangGraph, and CrewAI. Developers can try the platform directly at diagrid.io, where Schneider said quick-start guides let teams test the failure-recovery behavior themselves before committing to it in production.

Retries Can Make Outages Worse

Sun, 23 Aug 2026 12:35:25 +0000

Retries are one of the easiest resilience patterns to add and one of the easiest to get wrong.

The idea is sound. Transient failures happen. Networks drop packets. Load balancers close idle connections. A database primary moves. A single failed call should not always fail the whole operation.

But retries also create traffic.

During a healthy moment, that extra traffic is small. During an outage, it can be enormous. Every timeout becomes multiple calls. Every slow dependency receives new requests from callers that have already given up on older ones. The dependency is not just handling user traffic anymore. It is handling user traffic plus the system's anxiety about user traffic.

That is how retries turn a partial failure into a larger one.

Retry Storms

A retry storm starts when many clients observe the same failure at roughly the same time.

Without jitter, they retry at roughly the same time too.

If the dependency is overloaded, those synchronized retries make recovery harder. The service needs quiet time to drain queues, rebuild caches, elect leaders, or scale out. Instead it receives a second wave of traffic. Then a third.

The clients think they are being persistent. The dependency experiences persistence as pressure.

Timeouts Come First

Retries without timeouts are not retries. They are a way to accumulate stuck work.

Every retry policy needs a per-attempt timeout and an overall deadline. The per-attempt timeout prevents one slow call from consuming the entire budget. The overall deadline prevents retries from continuing after the user or caller no longer cares.

In Go, that usually means retries must be tied to context.Context, not a loop with time.Sleep.

func callWithRetry(ctx context.Context, attempts int) error {
    for i := 0; i < attempts; i++ {
        if err := ctx.Err(); err != nil {
            return err
        }

        attemptCtx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
        err := callDependency(attemptCtx)
        cancel()

        if err == nil {
            return nil
        }
        if !isRetriable(err) {
            return err
        }

        if !sleepWithJitter(ctx, i) {
            return ctx.Err()
        }
    }
    return errors.New("dependency failed after retries")
}

The exact helper functions matter less than the shape: bounded attempts, bounded time, cancellation-aware sleep, and retry only for errors that are actually retriable.

Retry Budgets

A retry budget limits how much extra traffic retries may create.

For example, a client might allow retries to add no more than 10 percent extra request volume over a rolling window. When the budget is exhausted, the client stops retrying and returns errors faster.

That sounds harsh, but it protects the dependency and the caller. If the system is already failing, unlimited retries mostly buy longer queues and slower recovery.

Retries should be treated as a shared resource.

Not Every Error Is Retriable

Retrying a deterministic failure is just load generation.

Good retry policies distinguish between:

connection resets
timeouts
rate limits
temporary 5xx responses
validation failures
authentication failures
not found responses
conflicts caused by stale state

Some of those may deserve a retry. Many do not. A 400 response should usually not be retried. A 401 should not become three 401s. A permanent validation error is not going to become valid because the client waited 100 milliseconds.

The retry policy should be closer to the protocol semantics than to the transport library.

Retries Need Backpressure

Retries and backpressure belong together.

If the dependency returns 429 Too Many Requests or a clear overload signal, callers should respect it. If there is a Retry-After header, callers should use it. If the local queue is already full, the client should stop adding retry work to it.

The purpose of a retry is to survive a transient failure, not to argue with a saturated system.

The Lesson

Retries are useful when they are humble.

They should have a deadline. They should have jitter. They should have a budget. They should understand which errors are worth retrying. They should stop when the caller no longer needs the answer. They should listen when the dependency says "slow down."

The question is not "should we retry?"

The question is:

How much extra pressure are we allowed to create while trying to recover?

The Difference Between a Cache and a Lie

Wed, 19 Aug 2026 15:51:36 +0000

A cache is a promise with an expiration date.

That sounds obvious until a production system starts depending on cached state as if it were truth. Then the cache is no longer an optimization. It is a second source of truth with worse durability, weaker ownership, and fewer people watching it.

That is where cache bugs become painful.

The hard part of caching is not storing a value. The hard part is deciding what the value is allowed to mean.

A Cache Should Have a Narrow Job

The safest caches answer narrow questions:

Have we recently computed this result?
Have we recently seen this object absent?
Have we recently fetched this configuration?
Can we avoid doing the same expensive work again?

The more dangerous caches answer authority-shaped questions:

Does this object exist?
Is this user allowed?
Is this state current?
Has this workflow completed?

Sometimes a cache must sit near one of those questions. That is fine. But the design needs to say exactly how wrong the cache is allowed to be and what happens when it is wrong.

The Three Questions Every Cache Needs

Before adding a cache, I like to ask three questions.

First:

What is the source of truth?

If the team cannot answer this clearly, the cache will eventually become part of the data model by accident.

Second:

What is the failure mode when the cache is unavailable?

Fail-open and fail-closed are both valid choices, but they belong to different classes of systems. A performance cache in front of an idempotent delete path can usually fail open. An authorization decision probably cannot.

Third:

What is the failure mode when the cache is stale?

This is where many designs get uncomfortable. A stale cache entry might mean an extra database read, which is fine. It might mean stale search results for a minute, which may be acceptable. It might mean granting access after permission was revoked, which is not the same kind of risk.

TTL Is Not Invalidation

TTL is a recovery mechanism. It is not a complete correctness story.

A TTL says:

If we miss an invalidation, the system will eventually heal.

It does not say:

The system is correct until the TTL expires.

If stale data can cause a correctness bug, the system still needs an invalidation path, a version check, a source-of-truth read, or a design that makes stale data harmless.

TTL is useful because production systems are imperfect. It should not be the only thing standing between the user and a wrong answer.

Fail-Open vs Fail-Closed

The right behavior depends on what the cache protects.

For a performance cache:

cache hit   -> use cached result
cache miss  -> recompute or read from source
cache error -> bypass cache

This keeps correctness independent of cache availability.

For a safety or authorization cache, bypassing the cache may be wrong. If the cache is the only fast path to know whether an operation is allowed, fail-open can turn an outage into a security issue. In those systems, the right answer may be to fail closed, degrade the feature, or require a source-of-truth check.

The cache design should make that tradeoff explicit.

When a Cache Becomes a Lie

A cache becomes a lie when the rest of the system forgets that it is allowed to be wrong.

Common warning signs:

no documented source of truth
no owner for invalidation
no metric for hit rate, miss rate, or stale reads
business logic that only exists in cache-writing code
cache entries with no version, generation, or TTL
code that treats cache errors as successful reads
tests that mock the cache but never test cache loss

The worst cache bugs are not caused by Redis being down. They are caused by Redis being trusted too much.

The Lesson

A good cache makes a system faster without becoming more important than the data it stores.

It is allowed to be missing. It is allowed to be cold. It is allowed to be flushed. It may be useful when present and boring when absent.

The moment a cache cannot be lost, rebuilt, bypassed, or corrected, it is no longer just a cache.

It is part of the system of record, whether the architecture diagram admits it or not.

Remote Execution, Explained for People Who Just Run make

Tue, 28 Jul 2026 15:51:22 +0000

Most engineers think of the build as something that happens on their laptop. You type a command, the fans spin up, and a few minutes later you have a binary. That model is so ingrained that we rarely question it — until the codebase gets big enough that it quietly collapses.

At a certain scale, the laptop build is too big for one machine, too slow to repeat, and too wasteful to run from scratch on every change. Ten engineers rebuild the same unchanged files ten times before lunch. A clean build takes forty minutes. CI is the bottleneck for the whole team. Remote execution is the answer large codebases reach for, and once you understand it, a lot of otherwise-baffling build-system design suddenly makes sense. Here is the mental model, minus the vendor jargon.

The core idea: the build is a graph of pure functions

Stop thinking of a build as a sequence of commands and start thinking of it as a graph. Each node is an action: a single command plus the exact set of inputs it reads and outputs it produces. Compiling one file is an action. Linking is an action. Generating code from a schema is an action. The edges are dependencies — the linker action consumes the object files the compile actions produced.

Here is the key insight that everything else rests on. If you treat each action as a pure function — same inputs always produce the same outputs — then two things become possible that are not possible with a pile of shell scripts. You can run independent actions anywhere, on any machine, in parallel. And you can cache the result of any action, because identical inputs mean you already know the output.

Remote execution is what you get when you take those two possibilities seriously.

What an action actually is

Concretely, an action is described by its command, the content of every input it depends on, and the environment it runs in. Crucially, inputs are identified by the hash of their content, not their path or timestamp:

Action {
  command:       ["clang", "-c", "parser.c", "-o", "parser.o"]
  input_digests: { "parser.c": sha256(abc1...),
                   "parser.h": sha256(def2...) }
  platform:      { os: linux, arch: x86_64, ... }
}

action_digest = sha256(serialize(Action))

That action_digest is the core lookup key. It fingerprints everything the build system has declared as relevant to the output. Correctness therefore depends on complete input and environment declarations: an undeclared dependency is invisible to the key. The build system computes the digest and asks a shared service one question: have you seen this exact declared action before?

Cache hit: the service returns the output blobs immediately. No compilation happens at all — not locally, not remotely. You just fetched the answer.
Cache miss: the service schedules the action on a remote worker, the worker runs the command in a clean sandbox, uploads the resulting parser.o to a content-addressed store, and records the action-to-output mapping so the next person gets a hit.

Multiply that across a team and a CI fleet and the effect is dramatic. The first engineer to build a given commit pays the cost; everyone else, and every CI run on that commit, gets cache hits. Work is done once and shared.

The two pieces of machinery

Underneath, remote execution rests on two services that are simpler than they sound.

The first is a content-addressed store (the CAS). It is a blob store keyed by the hash of the content. Put bytes in, get a hash back; present a hash, get the bytes. Because the key is the content's fingerprint, identical files can be deduplicated and a retained blob cannot become stale under the same digest. Storage policies may evict blobs, so clients still have to handle missing content. Source files, intermediate objects, and final binaries can all live here.

The second is the action cache: a map from action_digest to the set of output digests it produced. That is the lookup that turns "compile this" into "you already compiled this, here are the bytes."

Bazel and Buck2 both support the common Remote Execution API for these services, which allows compatible deployments to point either client at the same class of remote backend. You do not need to know the protocol to use it, but it helps to know it exists: the build tool on your laptop coordinates the graph while remote workers and shared caches handle much of the expensive execution and storage.

Why this reshapes how build systems are designed

Once you see the build as a graph of cacheable pure functions, the design decisions of modern build systems stop looking arbitrary.

Why does Bazel make you declare every input and output explicitly, instead of letting a command read whatever files it wants? Because the action digest has to capture every input. An undeclared input is an input the cache cannot see, which means the cache will hand you a stale result. Why the obsession with hermetic, sandboxed actions? Because a worker on the other side of the cluster has to be able to run your action with nothing but the declared inputs and get the same answer your laptop would. The strictness that feels like bureaucracy when you are writing a BUILD file is the exact thing that makes the action a pure function — and a pure function is the thing you can distribute and cache.

Where it gets hard

It is not free, and the failure modes are worth knowing before you adopt it.

The network becomes the new bottleneck. You have traded local CPU for cache lookups and blob transfers, so a slow link or a far-away CAS can make a "cached" build feel slower than a local one. Large inputs and outputs strain the CAS and the network; build graphs with giant artifacts need care. And the whole edifice rests on one assumption that deserves its own discussion: that actions are actually deterministic. The moment an action produces different output from identical inputs, the cache starts handing out wrong or useless answers, and the value proposition quietly inverts.

That last point is important enough that it is the subject of its own post. For now, the mental model is the takeaway: a build is a graph of actions, each action is a pure function fingerprinted by its inputs, and remote execution is just the infrastructure for running those functions anywhere and remembering their answers. Once that clicks, the rest of the modern build world — the explicit inputs, the sandboxing, the content hashing — reads as the obvious consequence it is.

Request-Response: The Foundation of Modern APIs

Sun, 26 Jul 2026 06:46:57 +0000

Most developers use Request-Response every day.

Few stop to think about what actually happens after clicking a button.

You open an app.

You tap "Login".

A response appears almost instantly.

Simple, right?

Not exactly.

Behind that seemingly simple interaction lies one of the most important communication patterns in software engineering:

Request-Response.

Whether you're using ChatGPT, ordering food through Uber Eats, checking your bank balance, or watching Netflix, countless Request-Response interactions occur every second.

Understanding this pattern is one of the first steps toward understanding modern backend engineering.

What Is Request-Response?

At its core, Request-Response is a conversation between two systems.

One system asks for something.

Another system answers.

Client                    Server
   |                         |
   |------ Request --------->|
   |                         |
   |<----- Response ---------|
   |                         |

The client initiates communication.

The server processes the request and returns a response.

This interaction is typically short-lived and synchronous.

The client sends a request and waits for an answer.

A Real-World Analogy

Imagine you're sitting in a restaurant.

You call the waiter.

Customer -> "I'd like a burger."

The waiter takes your order to the kitchen.

The kitchen prepares the meal.

The waiter returns with your food.

Kitchen -> "Here's your burger."

That's Request-Response.

Customer = Client
Waiter = Network
Kitchen = Server
Burger = Response

Most web applications operate using exactly this pattern.

The Journey of a Request

Consider a simple login request.

POST /login

It may look simple from the browser's perspective.

But internally:

Browser
   |
Load Balancer
   |
API Gateway
   |
Authentication Service
   |
Database

The response must travel back through the same chain.

Database
   |
Authentication Service
   |
API Gateway
   |
Load Balancer
   |
Browser

Every step introduces latency, failure possibilities, and complexity.

Latency: The Hidden Cost of Distance

Latency is the time required for a request to travel through the system and return a response.

Request Sent --------------------->

<-------------------- Response Received

Users don't care about architecture diagrams.

They care about speed.

When someone clicks a button, they expect a response immediately.

Even small delays are noticeable.

Typical user perception:

Response Time	User Experience
0–100 ms	Instant
100–300 ms	Fast
300–1000 ms	Noticeable
1–3 sec	Slow
3+ sec	Frustrating

At scale, reducing latency becomes a competitive advantage.

This is one reason companies invest heavily in:

CDNs
Caching
Edge Computing
Database Optimization
Load Balancing

Timeouts: Waiting Forever Is Not an Option

Imagine calling a restaurant and nobody answers.

How long should you wait?

Five seconds?

Five minutes?

An hour?

Software faces the same problem.

A timeout defines the maximum amount of time a system is willing to wait for a response.

Client ---- Request ---->
                ...
                ...
                ...
Timeout Reached

Without timeouts, systems can become stuck waiting forever.

This leads to:

Hanging requests
Resource exhaustion
Poor user experience

Every production-grade system uses timeouts.

Retries: What If the Request Fails?

Networks are imperfect.

Packets get lost.

Servers restart.

Databases become temporarily unavailable.

A single failure doesn't necessarily mean the operation should fail.

Instead, many systems retry.

Attempt #1 -> Failed

Attempt #2 -> Failed

Attempt #3 -> Success

Retries improve reliability.

But they also introduce a dangerous problem.

What if the original request actually succeeded?

Idempotency: Preventing Duplicate Operations

Imagine clicking:

Pay Now

The payment succeeds.

But the response never reaches your browser.

Your application thinks the request failed.

It retries.

Charge Card

Again.

Without protection, the customer could be charged twice.

This is why payment systems use idempotency.

An idempotent operation produces the same result even if executed multiple times.

Request ID: abc123

If the same request arrives again:

Request ID: abc123

The server recognizes it and returns the previous result instead of processing the payment again.

This concept is critical in systems like:

Stripe
PayPal
Banking Platforms
Order Processing Systems

Fan-Out: One Request Can Become Many

A request rarely stays simple for long.

Imagine opening a product page.

The backend may need:

Product Service
Review Service
Inventory Service
Pricing Service
Recommendation Service

One incoming request becomes many downstream requests.

                 Product
                /
Client ---- API Gateway
                \
                 Reviews

                 Inventory

                 Pricing

This pattern is called Fan-Out.

Fan-Out increases functionality.

But it also increases risk.

If one dependency becomes slow, the entire response can become slow.

Cascading Failures

Now imagine the Review Service becomes overloaded.

Requests begin timing out.

The Product Service waits.

Threads become blocked.

Connection pools fill up.

More requests pile up.

Soon:

Review Service Failure
        |
        v
Product Service Slow
        |
        v
API Gateway Slow
        |
        v
Entire Application Slow

A small issue spreads throughout the system.

This is called a Cascading Failure.

Many major outages start exactly this way.

A single dependency struggles.

Everything connected to it suffers.

Observability: Seeing What the System Sees

When users complain:

"The app feels slow."

Where exactly is the problem?

The database?

The cache?

The network?

The payment service?

Without visibility, debugging becomes guesswork.

Modern systems rely on observability.

Three pillars are commonly used:

Logs

Individual events.

User logged in
Payment created
Order shipped

Metrics

System health measurements.

CPU Usage
Memory Usage
Request Rate
Error Rate
Latency

Traces

The complete journey of a request.

Client
  -> Gateway
     -> Auth Service
        -> Database

Tracing reveals where time is spent.

At scale, observability becomes essential.

Why Request-Response Eventually Becomes a Bottleneck

Request-Response is simple.

That's its greatest strength.

But it's also its limitation.

The client must wait.

Request --->

<--- Response

No response means no progress.

As systems grow, engineers begin exploring alternatives:

Polling
Long Polling
Server-Sent Events (SSE)
WebSockets
Pub/Sub
Event-Driven Architectures

These patterns exist because traditional Request-Response is not always enough.

But every one of them builds upon the foundation established by Request-Response.

Final Thoughts

Request-Response appears deceptively simple.

A client asks.

A server answers.

Yet hidden beneath that simplicity are some of the most important challenges in software engineering:

Latency
Timeouts
Retries
Idempotency
Fan-Out
Cascading Failures
Observability

Understanding these concepts is what separates someone who can build APIs from someone who can design reliable systems.

Before learning WebSockets, Event Streaming, or Distributed Systems, it's worth mastering the communication pattern that powers nearly every application on the internet:

Request-Response.

I Doubled My Servers and Wrecked My Availability

Sat, 27 Jun 2026 09:58:47 +0000

“Reliability is the most important feature of any system.” — Bill Gates

Why does your bank never show the wrong balance, but your favorite app sometimes takes a few seconds to update? Why do some systems feel fast but crash under load, while others feel slower but never fail?

Performance answers: “How fast is the system right now?”
Scalability answers: “What happens when demand increases?”

These questions point to a fundamental tension in system design: performance vs scalability.

A system can improve performance metrics like throughput and latency, yet fail to scale under load. This article demonstrates exactly that.

In this article, we’ll explore this trade-off through a simple analogy, and then connect it to real-world systems using measurable metrics like latency, throughput, and error rate.

The Burger Shop Analogy

Imagine a small burger shop run by a single worker, let’s call them A.

Let’s assume the following things:

Burgers are prepared automatically by machines, so A only handles taking and serving orders.
Finished burgers are placed in a shared box, from which A picks them up and delivers them to customers.

We intentionally ignore preparation time and assume burgers are instantly ready. This allows us to focus purely on how requests are handled, rather than how they are produced.

The Single Worker System

In this setup, a single worker handles all incoming requests.

When you enter the shop, you join a single queue and wait your turn before placing an order. Every request is processed sequentially (one at a time).

A few key observations:

Single queue, sequential processing

With only one line, every customer must wait for those ahead of them. This results in higher waiting time, which translates to high latency and low throughput.
No contention

Since only one request is handled at a time, there is no competition for shared resources. The system operates without conflicts or synchronization issues.
Efficient resource usage

Resources are used strictly on demand. Each burger is prepared only when needed, with no excess consumption or overhead.

What does this imply?

High latency → Each request experiences delay due to queuing
Low throughput → Only one request is processed at any given time
No contention → No conflicts between concurrent operations

Despite its limitations, this system is remarkably stable.

There are no race conditions, no lost work, and no unexpected failures. Every request is handled in a predictable and controlled manner.

Slow, but predictable.

Scaling the System

So far, the limitation has been clear: a single worker handling all requests.

To improve this, we introduce more machines. Instead of relying on a single machine, worker A can now distribute incoming orders across multiple machines.

Let’s examine what changes:

Parallel processing

With multiple machines available, several orders can be handled simultaneously. This increases the number of requests processed per unit time, leading to higher throughput and reduced latency.
Decision overhead (routing)

Worker A must now decide which machine should handle each request. This introduces a small coordination overhead — similar to how a load balancer (e.g., Nginx) routes requests in real systems.

For simplicity, we will ignore this overhead in our analysis.

What happens to the metrics?

Multiple orders can be processed simultaneously.
Waiting time decreases
Throughput increases

This approach is known as horizontal scaling, increasing system capacity by adding more resources rather than making a single component more powerful.

The Hidden Problem

So far, scaling seems like a clear improvement.

But consider the shared burger box , the place where all machines deposit finished burgers.

What happens if multiple machines try to place burgers into the box at the same time?

This introduces two critical failure scenarios:

Lost Work

One burger makes it into the box, but another falls to the ground.

That burger is wasted, and the customer never receives their order.

In system terms, this is a failed request.
System Blockage

Both burgers get stuck at the entrance of the box, blocking it completely.

Now no further burgers can be placed.

In system terms, this resembles contention or resource locking, where the system slows down or stops entirely.

What Changed ?

By introducing multiple machines, we improved performance:

Higher throughput, lower latency

Multiple burgers can now be prepared simultaneously. This increases throughput and reduces waiting time for customers, improving overall performance.
Increased coordination and state management

With two machines handling requests, worker A must decide where each order goes. Both machines can process requests, so their state must be tracked and logs must be maintained. This introduces coordination overhead.
Inefficiency due to lost work

Some burgers may fall to the ground before reaching the box. These orders are wasted, and the customer never receives their burger. In system terms, this represents failed or lost requests.
Unpredictable latency

Consider a customer waiting for an order that never arrives. If they leave after 30 seconds, that request now has very high latency. Even if other customers are served quickly, overall latency becomes inconsistent and harder to predict.This is the core trade-off.

In system terms, we have introduced:

Contention for shared resources
Increased system complexity
Possibility of failure

The Core Trade-off

This is the fundamental trade-off in system design.

Some customers are served faster, but others may never get their order.

Performance

Performance refers to how effectively a system executes tasks within a given time frame. It is typically evaluated in terms of speed, responsiveness, and how efficiently system resources are utilized under load.

To understand and measure performance, we rely on a few key metrics:

Latency

Latency is the time it takes for a system to respond to a request.

In simpler terms, it measures how long a user has to wait after initiating an action.

Throughput

Throughput represents the number of requests or transactions a system can handle per unit of time.

It reflects the system’s capacity to process workload at scale.

Resource Utilization

Resource utilization measures how much of the system’s resources, such as CPU, memory, and network bandwidth are consumed while handling requests.

Efficient systems aim to maximize output while minimizing unnecessary resource consumption.

Efficiency

Efficiency describes how well a system converts resources into useful work.

A highly efficient system achieves high throughput and low latency while using minimal resources.

Scalability

Scalability refers to a system’s ability to handle increasing load by proportionally utilizing additional resources, while maintaining acceptable performance.

In other words, a scalable system does not degrade significantly as the number of users or requests grows, it adapts by efficiently distributing the workload.

To evaluate scalability, we observe how key performance metrics behave under increasing load, including:

Latency under load
Throughput under load
Resource utilization
Error rates (failures, timeouts)
Number of concurrent users supported

A system is considered scalable if these metrics remain stable or degrade gracefully as demand increases.

Availability

Availability is the measure of a system’s ability to respond to requests within an acceptable time frame.

A system is considered available if it is operational and capable of returning responses, even under load. High availability systems are designed to minimize downtime and ensure that users can reliably access the service.

“A system can be highly performant but not scalable. for example, a single powerful server may respond quickly, but fail when user load increases.”

The Monolith vs. The Load-Balanced Fleet (A vs 2A’s)

To move beyond theory, we need to observe how a system behaves when pushed to its absolute limits. To test the relationship between compute scaling, database bottlenecks, and overall availability, an A/B load test was constructed.

The code for the experiments can be found in attached github repository: {% github https://github.com/c0d3l0v3r-HeHe/performance-vs-scalability-experiment %}

Assumptions

Before running the load test, the architecture was built on three core hypotheses:

Linear Scalability: Doubling the compute resources (adding a second application server) will linearly increase the system’s maximum throughput.
Testing Default Database Limits: We hypothesized that the default connection pool of a single MongoDB instance would be sufficient to handle the concurrent load of doubled compute resources.
The Load Balancer Penalty is Negligible: The network hop introduced by routing traffic through a reverse proxy will be outweighed by the massive gains in request processing speed.

Setup Methodology

The experiment was conducted locally using Docker to containerize and isolate the environments. The testing tool, k6, was configured to inject a stepped load simulating aggressive traffic spikes: starting at 100 concurrent virtual users and ramping up to 2,000 users over an 80-second window.

The workload itself was a mixed Read/Write operation. Every request forced the application to write a new document to the database and immediately read a document back, simulating a standard transactional endpoint.

We tested three distinct architectures:

Architecture A (The Baseline): A single Node.js/Express application connected directly to a standalone MongoDB instance. This represents a traditional, unscaled monolith.
Architecture 2A (The Distributed Compute): Two identical Node.js/Express applications sitting behind an Nginx load balancer. Nginx utilized a round-robin strategy to distribute incoming k6 traffic across both nodes. Crucially, both application nodes were wired to the exact same single MongoDB instance.

const express = require("express");
const mongoose = require("mongoose");
const os = require("os");

const app = express();


mongoose.connect("mongodb://mongo:27017/testdb", {
  maxPoolSize: 50,
});

const TestSchema = new mongoose.Schema({
  value: Number,
});

const Test = mongoose.model("Test", TestSchema);

app.get("/", async (req, res) => {
  const start = Date.now();

  try {
    await Test.create({ value: Math.random() });
    await Test.findOne();

    const latency = Date.now() - start;

    res.json({
      latency,
      server: os.hostname(), 
    });
  } catch (err) {
    res.status(500).json({ error: "DB error" });
  }
});

app.listen(3000, () => console.log(`Server running on ${os.hostname()}`));

Results

The k6 load test yielded drastically different saturation points for the two architectures.

These reports are present as the .html files in the github respository.

Architecture A (Single Node Baseline):

The latency trade-off. With a single Node.js event loop acting as a natural throttle, the p95 latency sat at ~796ms. The system was relatively slow under heavy load, but perfectly stable.

Baseline throughput. The single-node architecture safely processed a total of 176,584 requests, hitting a hard bottleneck at roughly 2,193 requests per second.

100% Availability. Because the compute layer could not overwhelm the database connection pool, every single read/write transaction was successfully completed without a single dropped request.

Total Requests Processed: 176,584
Throughput: 2,193 requests/second
Latency (p95): ~796 ms
Failure Rate: 0.00% (100% Success)

Architecture 2A (Distributed Compute):

The smoking gun. While the p95 latency for successful requests actually dropped to ~492ms, the maximum latency spiked to an unusable 36 seconds. The explicit HTTP 500 errors confirm the single MongoDB instance was pushed into connection exhaustion.

The illusion of scale. Adding a load balancer and a second application node successfully pushed our throughput to over 3,600 requests per second, processing 350,261 total requests.

The breaking point. Scaling the compute layer while leaving the data layer untouched resulted in a massive 26.5% failure rate, actively degrading the system’s availability.

Total Requests Processed: 350,261
Throughput: 3,638 requests/second
Latency (p95): ~492 ms (for successful requests)
Failure Rate: 26.57% (93,069 failed requests, including explicit HTTP 500 database errors and massive timeouts spiking up to 36 seconds).

Discussion

The experimental data explicitly demonstrates the systemic risks of isolated horizontal scaling. While adding compute resources improved specific performance metrics, it fundamentally compromised system reliability, proving that scaling a single tier does not equate to scaling the system as a whole.

1. The Shifting Bottleneck and Resource Exhaustion The baseline load test (Architecture A) established a clear system ceiling: a single Node.js instance capped throughput at 2,193 requests per second with a p95 latency of ~796ms. Crucially, the failure rate was 0.00%. In this configuration, the compute layer acted as a strict upstream bottleneck. Because a single server could not process requests fast enough to overwhelm the database’s connection pool (configured to a maximum of 50 connections), the database was naturally shielded from resource exhaustion. The system was relatively slow under peak load, but highly stable.

By introducing a load balancer and a second application node (Architecture 2A), the compute-layer bottleneck was removed. The system accepted more concurrent traffic, driving throughput up by roughly 65% (to 3,638 req/sec). However, because both nodes funneled this doubled traffic into the single, unscaled MongoDB instance, the bottleneck immediately shifted to the data layer. The sudden influx of concurrent requests exhausted the database connection pool, leading to connection lockups and explicit HTTP 500 errors.

2. The False Positive of “Improved” Performance A superficial reading of Architecture 2A’s metrics might suggest a performance win: the p95 latency for successful requests actually dropped by ~38% (from 796ms down to 492ms), and total processed requests nearly doubled.

However, a rigorous analysis reveals this “scale” is an illusion. The data shows a catastrophic failure rate of 26.57% (93,069 failed requests). The reduction in p95 latency for successful requests is effectively a survivorship bias metric; it only accounts for the requests that managed to secure a database connection. Meanwhile, the maximum latency for blocked requests spiked to an unusable 36 seconds. The experiment proves that forcing high throughput at the compute layer without matching capacity at the data layer destroys system availability.

3. The Inadequacy of Isolated Scaling The results strictly invalidate the hypothesis that doubling compute resources linearly increases maximum throughput safely. Instead, it proves that architectural scaling must be holistic.

When Architecture 2A pushed the database beyond its concurrency limits, the system did not degrade gracefully — it fractured, dropping more than a quarter of all user requests.

Conclusion

The data validates a core principle of distributed systems:

throughput and availability are often inversely correlated when shared resources are unscaled. Architecture A was unscalable but highly available (100% success). Architecture 2A was highly performant for a subset of users but critically unavailable for the rest (26.57% failure).

To successfully scale this system beyond the 2,193 req/sec ceiling of Architecture A without incurring the 26% failure penalty of Architecture 2A, the architecture must evolve to protect the database. Based strictly on these failure modes, future iterations must either synchronously scale the data layer (e.g., database sharding/replicas to handle the higher connection volume) or implement an asynchronous buffer (e.g., a message queue) to decouple the request ingestion rate from the database write rate.

Acknowledgements & References

The experiments and conclusions drawn in this article were directly informed by studying foundational distributed systems literature. Translating these concepts from theory into a live, containerized k6 experiment was made possible by the following resources:

Performance, Scalability & Throughput

MIT 6.004: Performance Measures (YouTube): Provided the classic “laundry analogy” illustrating why pipelining improves overall throughput but not individual request latency.
Performance vs. Scalability (Prof. Beekums): Clarified the boundaries between making a single server faster versus designing an architecture to handle increased load.
A Word on Scalability (Werner Vogels): Foundational principles from Amazon’s CTO on building systems that scale gracefully.
Latency vs Throughput (CS.fyi) & Understanding Latency vs Throughput (Cadence): Core definitions and metrics for measuring system speed and capacity.

The CAP Theorem & Consistency Patterns

CAP Theorem Revisited (Robert Greiner) & A Plain English Introduction to CAP (ksat.me): Explained the core mathematical trade-off: why our partitioned, load-balanced system had to sacrifice availability when tied to a single database.
Consistency Patterns (CS.fyi): Detailed breakdowns of Weak, Eventual, and Strong Consistency, which directly inspired the Redis asynchronous queue experiment.

High Availability, Failover & Database Replication

Availability in Distributed Systems (GKCS — YouTube): A practical war story detailing a real-world database migration, downtime management, and understanding the “9s of availability.”
Scalability, Availability & Stability Patterns (SlideShare): A comprehensive visual overview of architectural patterns.
High Availability System Design Basics (DesignGurus) & System Design Availability Patterns (Dev.to): Practical guides on identifying and removing single points of failure.
Active-Passive vs Active-Active Failover (Serverion): The blueprint used for configuring our Nginx load balancer to distribute compute traffic.
Introduction to Database Replication (EnjoyAlgorithms): A deep dive into separating read/write workloads and scaling the data tier.

"Done" Is Not a State

Wed, 24 Jun 2026 11:00:12 +0000

Completion generates silence. A task that finished and a task that was silently dropped produce the same signal: nothing.

Originally published at vibeagentmaking.com

On December 16, 2024, a developer filed a bug report against Trigger.dev, the open-source background job framework. A routine nightly server restart had caused random tasks to get stuck in a "queued" state. The system's recovery logic, working exactly as designed, detected the stalled tasks and requeued them. Then it detected them again. And again. By the time anyone checked the dashboard, 3,800 duplicate tasks were sitting in the queue, each one a faithful copy of work that had already been completed.

The monitoring system showed no errors. Every task had succeeded. The duplicates were executing successfully too. From the system's perspective, nothing was wrong.

This is the kind of bug that makes senior engineers go quiet. Not because it's complicated — the explanation fits in a sentence — but because the implications are uncomfortable. The system didn't malfunction. It did exactly what it was designed to do: detect abandoned work and retry it. The problem is that "completed successfully" and "abandoned silently" produce the same signal from the outside. Both go quiet.

Two Generals and No Good Options

The theoretical foundation for this is older than most production systems running today. In 1985, Fischer, Lynch, and Paterson published their impossibility result in the Journal of the ACM: given the possibility of even a single faulty process, it is impossible for a system of processes to agree on a decision. The paper is formally about consensus, but its practical implication is about something more mundane. It's about acknowledgments.

You send a message. Did it arrive? You wait for an acknowledgment. Did the acknowledgment arrive? You could send an acknowledgment of the acknowledgment, but that just moves the problem one level up. This is the Two Generals Problem, and it has no solution. Not "no known solution" — no solution, period. It is a mathematical impossibility, as fundamental to distributed computing as the halting problem is to computation itself.

Tyler Treat crystallized the practical consequence in a 2015 essay that has since become something of a canonical reference: "You Cannot Have Exactly-Once Delivery." There are, he argued, exactly two real delivery semantics. At-most-once: acknowledge the message before processing it, accept that crashes will lose data. At-least-once: acknowledge after processing, accept that retries will duplicate work. Everything else is one of these two, wearing a better outfit.

"Exactly-once delivery in practice," Treat wrote, "is by faking it" — through idempotent operations, deduplication layers, or application-level state machines that make repeated processing safe even when the underlying transport cannot make it impossible. Apache ZooKeeper's Zab protocol demonstrates the approach: state changes are idempotent, so applying the same change multiple times produces no inconsistencies. But this is an application-level guarantee, not a network-level one. The network still delivers messages more than once. The application just learned not to care.

The theory says duplicates are inevitable. The question isn't whether your system will duplicate work. It's whether it will notice.

The Industry Said This Out Loud

Here is the part that makes the Trigger.dev incident less surprising and more damning. The largest cloud providers in the world don't just acknowledge duplicate execution. They document it as expected behavior.

Google Cloud Tasks states it plainly: "In situations where a design trade-off must be made between guaranteed execution and duplicate execution, the service errs on the side of guaranteed execution." Their published metric: more than 99.999% of tasks are executed only once. Five nines of uniqueness sounds impeccable until you do the arithmetic. At one million tasks per day — a modest load for any serious deployment — 99.999% means ten duplicates daily. Three thousand six hundred and fifty per year. Whether that number is acceptable depends entirely on whether each task is counting page views or charging credit cards.

AWS is equally explicit. Standard SQS queues guarantee "at-least-once" delivery, and the documentation enumerates three specific scenarios in which Lambda functions will be invoked more than once for the same message: the Lambda service fails to delete the message from SQS before the visibility timeout expires; the Lambda service sends the event but fails to receive acknowledgment; an intermittent issue causes SQS to return the same message on a subsequent poll. The documented mitigation is to store message IDs in DynamoDB and check before processing. But this adds latency, cost, and its own failure modes. What if the DynamoDB write succeeds but the SQS delete fails? You have added a deduplication layer that itself needs deduplication. The turtles go all the way down.

The documentation exists. The warnings are in print. Almost nobody reads them until after the incident.

Airflow's Four-Year War Against "Done"

If the Trigger.dev case is a snapshot, Apache Airflow's relationship with stuck-queued tasks is a time-lapse.

With CeleryExecutor — Airflow's most common production deployment pattern — tasks would routinely get stuck in a "queued" state for hours. Sometimes indefinitely. The GitHub issue tracker accumulated reports across several major versions: #21225, tasks stuck in queued state; #13542, tasks stuck scheduled or queued; #26773, tasks stuck after upgrade; #13808, tasks incorrectly marked as orphaned. The core issue was architectural. When a scheduler process died, its tasks became orphans. A different scheduler was supposed to "adopt" them. But if a task had already been marked as STARTED in the Celery results database while remaining QUEUED in Airflow's internal state, no scheduler would ever transfer it out. The task existed in a kind of superposition: simultaneously complete in one system and waiting in another.

Neither system was wrong. They just disagreed about what "done" meant.

Airflow 2.6.0, released in April 2023, finally addressed the problem — and the fix is more instructive than the bug. The team didn't write a better timeout algorithm. They didn't add smarter retry logic. They consolidated three separate timeout configurations — kubernetes.worker_pods_pending_timeout, celery.stalled_task_timeout, and celery.task_adoption_timeout — into a single parameter: scheduler.task_queued_timeout. The fix was moving the "is this task stuck?" question from the executor to the scheduler, giving one component authoritative ownership of the completion state. Even then, Airflow 2.6.3 had to patch additional edge cases where tasks could still get permanently stuck.

The lesson is worth stating directly. You cannot fix a missing state by building better detection of its absence. If three components each maintain a partial view of "done," no amount of timeout tuning will make them agree. The number of components that can declare a task complete is inversely proportional to the system's ability to notice when nothing has.

When Correct Systems Duplicate Correctly

The Trigger.dev and Airflow cases are at least recognizable as engineering problems — recoverable, diagnosable, fixable. What happened to Coinbase customers in February 2018 is something different.

Between January 22 and February 11, customers found duplicate charges on their credit and debit cards. Not two or three charges — seventeen to fifty repetitions of a single cryptocurrency purchase. The root cause was not a software failure. Visa had changed the Merchant Category Code for digital currency transactions. When major banks and card issuers reclassified purchases under the new code, the processing systems refunded original transactions and recharged them under the updated category. Many customers saw the recharge before the refund cleared, producing what looked like mass duplicate billing. Worldpay, Coinbase, and Visa worked together to reverse the duplicates.

No system malfunctioned. Every component did precisely what it was designed to do. A category reclassification is not a retry — but it triggers the same downstream effect as one. The most dangerous duplicates don't come from bugs. They come from correct systems responding correctly to a state change that nobody modeled as a duplication event.

The Invisible State

There is a pattern running through every one of these incidents, and it isn't strictly about distributed systems theory. It's about visibility.

Consider the states a task can occupy: queued, dispatched, running, retrying, failed. Every one of these generates observable activity. Queued tasks sit in a list. Running tasks consume resources. Failed tasks fire alerts. Even retrying tasks produce log entries. Each state is loud.

Completion generates silence.

From the perspective of any monitoring system, any reclamation process, any orphan-detection algorithm, a task that completed successfully and a task that was silently dropped look identical. Both stopped producing signals. Both stopped consuming resources. Both went quiet. The only difference between them is that one finished its work and the other didn't — and no system that relies on the absence of activity can distinguish between the two.

This is why "done" cannot be treated as the default — the thing that happens when nothing else is happening. "Done" must be an explicit transition, a first-class state with its own signal, its own timestamp, its own acknowledgment path. A task that completes must announce its completion as loudly as a task that fails announces its failure. Otherwise, every recovery system, every health check, every dashboard that monitors for activity will interpret completion as disappearance.

The Idempotency Deflection

The standard engineering response to all of this is: make your operations idempotent. If running a task twice produces the same result as running it once, duplicates are harmless. Problem solved.

This is true and incomplete. Idempotency makes duplicate execution safe. It does not make it visible. A pipeline that silently runs every task three times and produces correct results is not a well-functioning system — it is a system burning three times the compute, making three times the API calls, and generating three times the cost, while its dashboard reports 100% success with a clean conscience. Idempotency is a seatbelt, not a steering wheel. It protects you from the consequences of the crash. It does not prevent the crash, and it does not tell you one happened.

The deeper fix is architectural: one component owns the definition of "done." One system has the authority to mark a task complete, and every other system defers to it. This is what Airflow 2.6.0 did. This is what Trigger.dev's self-hosted deployments still needed as of late 2024. This is what every team eventually learns after their third duplicate-execution incident. The solution isn't making duplicates safe. It's making completion loud.

The Dashboard Said 100%

The most unsettling detail in the Trigger.dev report isn't the 3,800 duplicates. It's that every one of them succeeded. The monitoring dashboard showed a perfect success rate because every task — original and duplicate alike — completed without error. The system was not failing. It was succeeding too many times.

We build monitoring to detect failure. We set up alerts for errors, timeouts, crashes. We watch for the system to go red. But the most expensive failure mode in distributed computing isn't the one that trips the alarm. It is the one that generates a clean bill of health while quietly tripling your workload, your costs, and your confidence in a number that was never what you thought it meant.

Silence, in a distributed system, is not peace. It's ambiguity. And until your system learns to announce "done" as loudly as it announces "broken," you are trusting that ambiguity to mean what you hope it means.

Your dashboard says 100%. It might even be right. But "right" and "once" are not the same thing.

Sources: M. Fischer, N. Lynch, M. Paterson, "Impossibility of Distributed Consensus with One Faulty Process," Journal of the ACM 32(2), April 1985. T. Treat, "You Cannot Have Exactly-Once Delivery," Brave New Geek, 2015. Google Cloud, "Issues and limitations — Cloud Tasks," cloud.google.com. AWS, "Using Lambda with Amazon SQS," docs.aws.amazon.com. Trigger.dev GitHub Issue #1566, December 2024. RNHTTR, "Unsticking Airflow: Stuck Queued Tasks Are No More in 2.6.0," Apache Airflow Blog, 2023. Apache Airflow GitHub Issues #21225, #13542, #26773, #13808. CNBC, "Worldpay and Visa are reversing duplicate transactions for Coinbase users," February 17, 2018. TechCrunch, "Visa confirms Coinbase wasn't at fault for overcharging users," February 16, 2018.

Your system announces "broken" with alerts and dashboards. Does it announce "done" with equal conviction? Chain of Consciousness treats agent completion as an explicit, anchored event — every decision signed, every state transition timestamped, every completion producing a verifiable artifact rather than silence. One component owns the record, and every downstream system defers to it — the architectural fix this essay prescribes, applied to agent work. Verify an agent's decision chain | Follow a claim through its evidence | pip install agent-rating-protocol

Simple Auth Service to a Distributed Authentication Platform with Kafka, Debezium, and Observability

Wed, 17 Jun 2026 18:53:34 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

What I Built

I built a Distributed Authentication System as a long-term learning project to explore real-world backend and distributed systems concepts through a familiar use case: user authentication.

The project started as a simple authentication service with signup and login functionality. Over time, it evolved into a distributed architecture that incorporates event-driven communication, change data capture (CDC), observability, horizontal scaling, and performance testing.

The current system consists of:

PostgreSQL as the primary source of truth for user credentials
Redis for refresh token storage and session management
Kafka as the event streaming platform
Debezium for Change Data Capture (CDC) from PostgreSQL
MongoDB for materialized user profile documents
Nginx for load balancing across multiple service instances
Prometheus and Grafana for monitoring and observability
k6 for load testing and performance analysis

When a user signs up, the authentication service writes user data to PostgreSQL. Debezium captures the database change and publishes it to Kafka. A separate profile service consumes the event and creates a corresponding profile document in MongoDB. This allows services to communicate asynchronously while keeping the architecture loosely coupled.

Beyond implementing features, the main goal of this project was to understand the trade-offs involved in building distributed systems. Throughout development I conducted load-testing experiments, measured replication lag, analyzed bottlenecks, and documented the architectural decisions that shaped the system.

This project has become my personal distributed systems playground where I can experiment with new ideas, evaluate design decisions, and learn how real systems behave under load.

Demo

GitHub Repository

Repository:
https://github.com/c0d3l0v3r-HeHe/distributed-auth-system

The repository contains additional architecture diagrams, performance experiments, load-testing results, observability metrics, and detailed documentation covering the design decisions and trade-offs explored throughout the project.

Rather than reproducing all of those results here, I've kept this submission focused on the project's journey and evolution. If you're interested in the deeper technical details, bottleneck analysis, replication lag measurements, scaling experiments, or observability setup, you'll find them documented in the repository.

The Comeback Story

This project was not originally intended to be a standalone distributed authentication platform.

In May, I started building it as the authentication backend for a larger job portal project. The initial goal was fairly straightforward: implement signup, login, token management, and the supporting infrastructure needed for user authentication.

After building the core authentication service and setting up the initial infrastructure, I shifted my focus to other work and the project was left unfinished. While the foundation existed, many of the ideas I wanted to explore—distributed systems patterns, observability, scalability, and performance analysis—were still missing.

A few weeks later, I came across the GitHub Finish-Up-A-Thon Challenge and decided to revisit the project instead of letting it remain another abandoned repository.

Rather than simply cleaning up old code, I used the opportunity to significantly expand the project and turn it into a distributed systems playground.

During the revival, I:

Added Redis-based refresh token management
Implemented a CDC pipeline using PostgreSQL, Debezium, and Kafka
Added a dedicated profile service backed by MongoDB
Introduced Prometheus metrics and Grafana dashboards
Added k6 load-testing infrastructure
Scaled the authentication service horizontally behind Nginx
Conducted multiple performance experiments and documented the results
Created architecture diagrams and expanded the project documentation

One of the most valuable outcomes of revisiting the project was discovering bottlenecks that only became visible under load. While scaling the authentication service improved CPU utilization, load testing revealed that MongoDB and the CDC pipeline became the primary bottlenecks. Investigating these trade-offs taught me far more than simply building the original authentication service.

By the end of the challenge, the project had evolved from an unfinished backend component into a fully documented distributed authentication system that I can continue using to explore distributed systems concepts, scalability patterns, and performance engineering.

My Experience with GitHub Copilot

GitHub Copilot acted as an implementation and exploration partner throughout the revival of this project.

For many components, I first designed the architecture and wrote the interfaces, function signatures, and high-level implementation plan. I then used Copilot to generate boilerplate code, suggest implementations, and accelerate repetitive development tasks.

Copilot was particularly useful for:

Generating service scaffolding and repetitive CRUD logic
Assisting with Docker Compose configuration
Helping configure Prometheus metrics collection
Suggesting Kafka consumer and producer implementations
Writing integration and infrastructure tests
Explaining configuration options for Debezium and Kafka Connect
Speeding up refactoring and cleanup work

One workflow I found especially effective was defining the architecture and API contracts myself, leaving implementation placeholders, and then using Copilot to generate an initial implementation. This allowed me to focus more on system design decisions and less on repetitive coding.

I also used Copilot while setting up and validating the distributed architecture. It helped me troubleshoot configuration issues, understand service interactions, and quickly iterate on infrastructure changes during development.

The biggest benefit wasn't code generation itself—it was the ability to move from an idea to an experiment much faster. Since this project is intended as a distributed systems learning playground, that rapid feedback loop allowed me to spend more time investigating architectural trade-offs, performance bottlenecks, and scalability challenges.

Feedback Welcome

This project started as a learning exercise and has evolved into my personal distributed systems playground.

If you're an experienced backend or distributed systems engineer and happen to read this submission, I'd genuinely appreciate any feedback on the architecture, design decisions, bottlenecks, or trade-offs discussed throughout the project.

Some areas I'm currently thinking about include:

Improving the CDC pipeline under higher load
Reducing replication lag and read amplification
Better approaches to profile materialization
Scaling strategies beyond the current setup
Observability improvements and production-readiness considerations

Constructive criticism, alternative approaches, and architecture suggestions are all welcome. One of the main goals of this project is to learn from engineers who have solved these problems in real systems.

Reconciling P2P Collaborative States via WebRTC Data Channels

Tue, 09 Jun 2026 07:16:53 +0000

title: "Reconciling P2P Collaborative States via WebRTC Data Channels"
published: true
description: "Mastering peer-to-peer signaling, NAT traversal mechanics (STUN/TURN), and robust client-side state synchronization using WebRTC Data Channels."
tags: webdev, javascript, typescript, distributedsystems
canonical_url: https://akinseinde.netlify.app/articles/webrtc-sync

series: "Web Engineering Series"

>This is an excerpt. The full article includes a live client-to-client WebRTC simulator — walk step-by-step through a simulated P2P handshake (Offer, STUN check, Signaling relay, Answer, Connection) and watch direct message sync bypass the central server entirely. Read the full interactive version →

The Peer-to-Peer Paradigm

Modern web architectures almost universally rely on client-server protocols (HTTP, WebSockets). While this simplifies state coordination, it introduces overhead: latency, server infrastructure costs, and single-point-of-failure bottlenecks.

WebRTC (Web Real-Time Communication) allows browsers to establish direct peer-to-peer (P2P) channels. Once connected, data flows directly between user devices with sub-10ms latencies, completely bypassing intermediate servers.

While WebRTC is widely known for audio/video streaming, its most powerful asset for application engineering is the RTCDataChannel API — allowing arbitrary, high-throughput, low-latency binary or text transfer directly between browsers.

The P2P Paradox: NATs and Signaling

Two browsers cannot simply talk directly to each other out of the box. Security boundaries and network topologies get in the way. Specifically, browsers face two barriers:

1. The NAT / Firewall Barrier

Almost all consumer devices sit behind Network Address Translators (NATs) and firewalls. A device knows its internal private IP (e.g. 192.168.1.45), but has no idea what its public-facing IP is, nor can it accept unsolicited incoming connections from the open web.

To cross this barrier, WebRTC utilizes ICE (Interactive Connectivity Establishment) alongside two types of auxiliary servers:

STUN (Session Traversal Utilities for NAT): A lightweight server that simply tells a requesting client: "Here is your public-facing IP address and port." This is cheap, fast, and succeeds in ~85% of standard residential network setups.
TURN (Traversal Using Relays around NAT): If symmetric NATs or strict corporate firewalls block direct connection, a TURN server acts as a fallback relay. All traffic is routed through it. This guarantees connection but incurs bandwidth costs.

2. The Signaling Bottleneck

To negotiate a direct connection, peers must exchange configuration details — specifically Session Description Protocol (SDP) objects (containing codecs, encryption keys, and connection profiles) and ICE Candidates (discovered IP/port routing paths).

Since they don't have a direct connection yet, they must use an out-of-band Signaling Server (typically running WebSockets or SSE) to exchange these initial handshakes.

Peer A                     Signaling Server                     Peer B
  │                                │                                │
  │ ─── 1. Send SDP Offer ───────> │ ─────────────────────────────> │
  │                                │                                │
  │ <── 2. Send SDP Answer ─────── │ <───────────────────────────── │
  │                                │                                │
  │ ─── 3. Exchange ICE Candidates │ ─────────────────────────────> │
  │                                │                                │
  │ <── [DIRECT P2P DATA CHANNEL ESTABLISHED (BYPASSES SERVER)] ──> │

Implementing WebRTC Data Channels in TypeScript

Here is a clean, dependency-free implementation of a WebRTC initiator client:

class P2PPeer {
  private pc: RTCPeerConnection;
  private dataChannel: RTCDataChannel | null = null;
  private signalingSocket: WebSocket;

  constructor(signalingUrl: string) {
    this.signalingSocket = new WebSocket(signalingUrl);
    
    // Configure standard public STUN servers
    this.pc = new RTCPeerConnection({
      iceServers: [
        { urls: "stun:stun.l.google.com:19302" },
        { urls: "stun:stun1.l.google.com:19302" }
      ]
    });

    this.setupIceGathering();
  }

  // Initiator: Create Offer and send to Peer
  async initiateConnection(targetPeerId: string) {
    // 1. Create the data channel on the local connection
    this.dataChannel = this.pc.createDataChannel("p2p-sync-channel", {
      ordered: true // Guarantees in-order delivery of packets
    });
    this.bindChannelEvents(this.dataChannel);

    // 2. Generate SDP Offer
    const offer = await this.pc.createOffer();
    await this.pc.setLocalDescription(offer);

    // 3. Dispatch to signaling server
    this.signalingSocket.send(JSON.stringify({
      type: "offer",
      target: targetPeerId,
      sdp: offer
    }));
  }

  // Receiver: Accept Offer, create Answer
  async handleOffer(offerSdp: RTCSessionDescriptionInit, senderId: string) {
    // 1. Set remote description
    await this.pc.setRemoteDescription(new RTCSessionDescription(offerSdp));

    // 2. Listen for the channel incoming event
    this.pc.ondatachannel = (event) => {
      this.dataChannel = event.channel;
      this.bindChannelEvents(this.dataChannel);
    };

    // 3. Generate SDP Answer
    const answer = await this.pc.createAnswer();
    await this.pc.setLocalDescription(answer);

    // 4. Send back via signaling
    this.signalingSocket.send(JSON.stringify({
      type: "answer",
      target: senderId,
      sdp: answer
    }));
  }

  private setupIceGathering() {
    // Broadcast discovered local ICE routes to remote peer
    this.pc.onicecandidate = (event) => {
      if (event.candidate) {
        this.signalingSocket.send(JSON.stringify({
          type: "ice-candidate",
          candidate: event.candidate
        }));
      }
    };
  }

  private bindChannelEvents(channel: RTCDataChannel) {
    channel.onopen = () => console.log("Direct P2P Data Channel Opened!");
    channel.onmessage = (event) => {
      const data = JSON.parse(event.data);
      console.log("Received P2P Message:", data);
    };
    channel.onclose = () => console.log("Data Channel Closed.");
  }

  send(message: any) {
    if (this.dataChannel && this.dataChannel.readyState === "open") {
      this.dataChannel.send(JSON.stringify(message));
    }
  }
}

State Reconciliations over P2P

Direct connections introduce concurrency challenges. If both clients edit a shared local state concurrently, how do they sync without a server?

The standard solution is pairing RTCDataChannel with a CRDT (Conflict-free Replicated Data Type) engine (like Y.js or Automerge). The data channel acts as a raw transport layer, while the CRDT handles deterministic state updates.

Because the data channel supports binary buffers (arraybuffer), you can transmit highly compact, compressed state updates directly:

channel.send(Y.encodeStateAsUpdate(ydoc));

Engineering Takeaways

WebRTC enables direct P2P connection, dropping transport latency to the raw physical limits of the network.
Signaling servers are only required for connection setup (SDP/ICE exchange) — once connected, the signaling server can crash and the P2P connection remains active.
STUN servers are free and easy to host, but always configure TURN servers as relays for symmetric or enterprise networks.
RTCDataChannel handles ordered or unordered data transmission, making it highly customizable for games, file sharing, or collaborative documents.

>The full article features a step-by-step WebRTC handshake visualizer — walk through the signaling architecture with a live interactive simulation of Offer generation, STUN query routing, and P2P connection establishment.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Distributed Systems: Implementing the Raft Consensus Protocol from Scratch

Tue, 09 Jun 2026 07:14:44 +0000

title: "Distributed Systems: Implementing the Raft Consensus Protocol from Scratch"
published: true
description: "Deconstructing distributed consensus, state machine replication, leader election algorithms, and split-brain mitigation in distributed clusters."
tags: computerscience, javascript, typescript, distributedsystems
canonical_url: https://akinseinde.netlify.app/articles/raft-consensus

series: "Systems Engineering Series"

>This is an excerpt. The full article includes a live interactive Raft consensus cluster simulator — control a 5-node distributed cluster, trigger leader elections, partition nodes to simulate network splits, commit client state updates, and watch the term-clock and replication logs resolve conflicts in real time. Read the full interactive version →

The Distributed State Problem

How do you get multiple independent computer nodes to agree on a single sequence of events, especially when individual nodes can crash, experience network delays, or drop packets?

This is the core challenge of Distributed Consensus. In a distributed database, all replicas must execute incoming commands in identical order to maintain a consistent state.

For years, the industry standard was Paxos — an algorithm famously powerful but incredibly complex to understand and implement without error.

Raft was designed as an alternative. It breaks down the consensus problem into three modular sub-problems: Leader Election, Log Replication, and Safety Invariants.

The Three Raft Node States

At any given moment, every node in a Raft cluster operates in one of three distinct roles:

        ┌───────────────────────────────────────┐
        │                FOLLOWER               │
        └───────────────────────────────────────┘
          │ (Times out, starts election)   ▲
          ▼                                │ (Discovers current leader)
        ┌───────────────────────────────────────┐
        │               CANDIDATE               │
        └───────────────────────────────────────┘
          │ (Receives majority votes)      
          ▼                                
        ┌───────────────────────────────────────┐
        │                 LEADER                │
        └───────────────────────────────────────┘

Follower: Completely passive. Responds to incoming RPC calls from Candidates and Leaders. If a follower receives no communication for a randomized timeout period, it assumes the leader has crashed and transitions to a Candidate.
Candidate: Increments the cluster "term", votes for itself, and broadcasts requests for votes to all other nodes.
Leader: Manages all client writes. Coordinates log replication and broadcasts periodic "heartbeat" RPCs to assert authority and reset follower timeout clocks.

Sub-Problem 1: Leader Election

To prevent split votes, followers wait for a randomized election timeout (typically between 150ms and 300ms) before initiating an election.

Once a node's timeout expires, it:

Increments its local currentTerm counter.
Transitions to the Candidate state.
Votes for itself.
Sends a RequestVote RPC to all other nodes.

If a candidate receives votes from a majority of nodes in the cluster (e.g., 3 out of 5), it immediately transitions to Leader and starts broadcasting heartbeats.

Sub-Problem 2: Log Replication

Once a Leader is elected, it acts as the single gateway for all client write requests.

Client ─── Write "X = 5" ───> Leader
                                │
          ┌─────────────────────┴─────────────────────┐
          │ (Append entry, broadcast AppendEntries)   │
          ▼                                           ▼
      Follower A                                  Follower B

The Replication Steps:

The client sends a command (e.g., set x=5) to the Leader.
The Leader appends the command to its local log.
The Leader broadcasts an AppendEntries RPC containing the new log entry to all followers.
Followers verify the entry and append it to their logs, sending a success confirmation back.
Once a majority of followers acknowledge the write, the Leader commits the entry to its state machine and returns a success response to the client.
In subsequent heartbeats, the Leader notifies followers of the newly committed entry, prompting them to commit it to their local state machines.

TypeScript Raft Node Implementation

Here is the baseline state container and interface schema for a Raft consensus node in TypeScript:

type NodeRole = "Follower" | "Candidate" | "Leader";

interface LogEntry {
  term: number;
  command: string;
}

interface RequestVoteArgs {
  term: number;
  candidateId: string;
  lastLogIndex: number;
  lastLogTerm: number;
}

interface RequestVoteReply {
  term: number;
  voteGranted: boolean;
}

class RaftNode {
  public id: string;
  public currentTerm = 0;
  public role: NodeRole = "Follower";
  public votedFor: string | null = null;
  public log: LogEntry[] = [];
  
  private commitIndex = 0;
  private electionTimeout: NodeJS.Timeout | null = null;
  private peers: string[] = [];

  constructor(id: string, peers: string[]) {
    this.id = id;
    this.peers = peers;
    this.resetElectionTimeout();
  }

  private resetElectionTimeout() {
    if (this.electionTimeout) clearTimeout(this.electionTimeout);
    
    // Randomized timeout between 150ms and 300ms prevents split votes
    const timeout = 150 + Math.floor(Math.random() * 150);
    this.electionTimeout = setTimeout(() => this.startElection(), timeout);
  }

  // Handle incoming vote request from Candidate node
  public handleRequestVote(args: RequestVoteArgs): RequestVoteReply {
    this.resetElectionTimeout();

    // 1. Term check: reject candidates with stale terms
    if (args.term < this.currentTerm) {
      return { term: this.currentTerm, voteGranted: false };
    }

    if (args.term > this.currentTerm) {
      this.currentTerm = args.term;
      this.role = "Follower";
      this.votedFor = null;
    }

    // 2. Log completeness check: candidate log must be at least as up-to-date as ours
    const lastIndex = this.log.length - 1;
    const lastTerm = lastIndex >= 0 ? this.log[lastIndex].term : 0;
    const logIsUpToDate =
      args.lastLogTerm > lastTerm ||
      (args.lastLogTerm === lastTerm && args.lastLogIndex >= lastIndex);

    // 3. Vote check
    const canVote = this.votedFor === null || this.votedFor === args.candidateId;

    if (canVote && logIsUpToDate) {
      this.votedFor = args.candidateId;
      return { term: this.currentTerm, voteGranted: true };
    }

    return { term: this.currentTerm, voteGranted: false };
  }

  private startElection() {
    this.role = "Candidate";
    this.currentTerm++;
    this.votedFor = this.id;
    this.resetElectionTimeout();

    console.log(`Node ${this.id} starting election for Term ${this.currentTerm}`);
    
    let votesReceived = 1; // Vote for self
    const lastLogIndex = this.log.length - 1;
    const lastLogTerm = lastLogIndex >= 0 ? this.log[lastLogIndex].term : 0;

    for (const peerId of this.peers) {
      // Simulate dispatching network RPC calls to peer cluster
      this.sendRequestVoteRPC(peerId, {
        term: this.currentTerm,
        candidateId: this.id,
        lastLogIndex,
        lastLogTerm
      }).then((reply) => {
        if (this.role !== "Candidate") return;

        if (reply.voteGranted) {
          votesReceived++;
          if (votesReceived > (this.peers.length + 1) / 2) {
            this.becomeLeader();
          }
        }
      });
    }
  }

  private becomeLeader() {
    this.role = "Leader";
    if (this.electionTimeout) clearTimeout(this.electionTimeout);
    console.log(`Node ${this.id} elected Leader for Term ${this.currentTerm}!`);
    this.startHeartbeats();
  }

  private startHeartbeats() {
    // Send AppendEntries heartbeats at 50ms intervals to maintain authority
    setInterval(() => {
      if (this.role !== "Leader") return;
      for (const peerId of this.peers) {
        this.sendAppendEntriesRPC(peerId);
      }
    }, 50);
  }

  // Mock network transporters for simulation environment
  private async sendRequestVoteRPC(peerId: string, args: RequestVoteArgs): Promise<RequestVoteReply> {
    return { term: this.currentTerm, voteGranted: Math.random() > 0.3 };
  }

  private async sendAppendEntriesRPC(peerId: string) {
    // Heartbeat logic
  }
}

Split-Brain and Minority Partitions

What happens when a network partition cuts a 5-node cluster into two segments: a majority partition (3 nodes) and a minority partition (2 nodes)?

  MAJORITY PARTITION (Can commit)         MINORITY PARTITION (Cannot commit)
   [Node A] ─── [Node B (Leader 1)]         [Node D] ─── [Node E (Leader 2)]
      │                                        │
   [Node C]                                    X (Network Partition cut)

The minority partition might elect a new leader because they can no longer hear from the original leader. However, this minority leader cannot commit any new client writes because they can never achieve the required cluster majority (3 nodes).

Once the partition heals:

The minority leader receives a heartbeat containing a higher term clock from the majority leader.
The minority leader immediately steps down back to a Follower.
The minority nodes discard their uncommitted local log differences and synchronize their states with the majority leader's canonical log.

Raft guarantees that the cluster always converges safely back to a single source of truth.

Engineering Takeaways

Raft guarantees Safety and Liveness in distributed environments with randomized timeouts.
Commitment requires absolute majority consensus — protecting clusters from split-brain state divergence.
Term clocks act as global logical time — higher term numbers override all stale cluster leaders.

>The full article features a live 5-node Raft consensus cluster simulator — trigger network cuts to form minority and majority partitions, watch node roles transition, commit client writes, and view live log conflict resolution directly in your browser.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Monolithic Thinking Is the Biggest Bug in Your Microservice Testing

Tue, 19 May 2026 18:19:07 +0000

Monolithic Thinking Is the Biggest Bug in Your Microservice Testing

If you started your QA career testing a monolithic application, you already have a foundation. But that foundation can quietly work against you in a microservice system — because the rules have changed, and most junior QA engineers don't realize it until something breaks in a way they can't explain.

This article will help you understand how microservice systems are built, why they behave differently, and what you need to start thinking about as a QA engineer working in one.

What Even Is a Microservice Architecture?

In a traditional monolithic application, everything lives in one place. The user interface, the business logic, the database — all bundled together. When you test a login feature, you're testing one system.

Microservice architecture breaks that single system into smaller, independently running services. Each service:

Handles one specific business capability (e.g., payments, user accounts, notifications)
Manages its own database
Communicates with other services through APIs or events

Here's what a simplified e-commerce system might look like:

API Gateway
              │ 
  ┌───────────┼───────────┐
  ▼           ▼           ▼
User       Order       Payment
Service    Service     Service
  │           │           │
  ▼           ▼           ▼
User DB    Order DB   Payment DB

Each of those services is developed, deployed, and scaled by a separate team — independently of the others.

Flexible? Yes.
Complex to test? Absolutely.

Why Your Old Testing Approach Breaks Here

In a monolith, a login flow looks like this:

User enters credentials → Business logic validates → Database checks → Response returned

That's one system. One place to look when something breaks.

In a microservice system, the exact same login might look like this:

Request hits API Gateway
       ↓
  Auth Service (validates credentials)
       ↓
  User Service (fetches user profile)
       ↓
  Session Service (creates session token)
       ↓
  Response returned

Four services involved in a single login.

If anything fails — a timeout in the Auth Service, a slow response from the Session Service, a data mismatch in the User Service — the whole flow breaks. And the error message the user sees might give you zero clue about which service actually caused it.

This is why junior QA engineers often feel lost in microservice environments. They're testing the output without understanding the system producing it.

The Mindset Shift: From Feature Tester to System Observer

In a monolith, the right question is:

“Does this feature work correctly?”

In microservices, the right question is:

“Which services are involved, and what happens when one of them fails?”

This is the core shift.

You're no longer just validating a response — you're observing a system.

When you pick up a test scenario, train yourself to ask:

Which services participate in this flow?
What data does each service depend on?
What happens if one service responds slowly or not at all?
Is the data consistent across services after the transaction completes?

You don't need to know how to build microservices to ask these questions. You just need to understand the architecture well enough to think through the flow.

Step 1: Get the Architecture Diagram (And Actually Read It)

This is the most underrated habit in QA.

Before you write a single test case for a feature, ask your team for:

A service interaction diagram (which services talk to which)
An event flow diagram (what triggers what)
A deployment overview (how services are grouped)

These diagrams tell you things that no test case document will:

Where does the data originate?
Which service owns a piece of data?
How do services notify each other when something happens?

If you're testing a payment flow but you don't know that the Payment Service sends an event to the Notification Service after a transaction, you'll miss testing whether the user gets their confirmation email — because that's not the Payment Service's job.

Without the diagram, you're testing in the dark.

Step 2: Understand How Services Talk to Each Other

Microservices don't communicate the same way in all situations. There are two main patterns, and you need to test them differently.

Synchronous Communication (HTTP / gRPC)

One service directly calls another and waits for a response.

Service A ──── REST API call ───→ Service B
               (waits for response)

What to test:

Does the API return the correct data when everything works?
What happens when Service B is slow? Does Service A handle the timeout?
What error does the user see if Service B is down?
Are the API contracts (request/response formats) being respected?

Asynchronous Communication (Events / Message Queues)

One service publishes an event. Another service picks it up later — independently.

Service A → publishes event to Kafka → Service B consumes it → Service C is triggered

This is trickier to test because there's no instant response. The flow happens over time.

What to test:

Is the event published in the correct format?
Does Service B actually consume the event and process it correctly?
What happens if the event is published but Service B is temporarily down — does the message get processed later?
Are there duplicate events being sent? How does the system handle that?

As a junior QA engineer, asynchronous flows are where most real bugs hide. If you're not thinking about them, you're missing a big part of your system.

Step 3: Understand Who Owns the Data

In a monolith, there's usually one database. You can query it directly to verify that data was saved correctly.

In microservices, every service has its own database:

Order Service   → Order Database
Payment Service → Payment Database
User Service    → User Database

This creates a testing challenge called eventual consistency.

When an order is placed, the Order Service records it immediately — but the Payment Service might take a few milliseconds (or longer) to update its records. During that window, the data across the two databases is temporarily inconsistent.

What junior QA engineers often miss:

Checking the Order Database but forgetting to verify the Payment Database
Not accounting for the delay in asynchronous updates
Assuming that if one service confirms success, all services have updated

A good habit: after any cross-service transaction, check the relevant database in each service that was involved. Don't just trust the API response.

Step 4: Learn to Follow a Request Through the System

When a bug happens in a microservice system, “it's broken” tells you almost nothing. You need to know where it broke.

Teams handle this with observability tools — tools that track what a request does as it moves through services:

Centralized logging (e.g., ELK Stack, Datadog): collects logs from all services in one place
Distributed tracing (e.g., Jaeger, Zipkin): shows you the full path of a request across services, with timing
Monitoring dashboards: give you a real-time view of service health

Here's what a trace might show you:

Request ID: abc-123
API Gateway        → 5ms
Auth Service       → 12ms
User Service       → 340ms  ← slow!
Session Service    → 8ms
Total: 365ms

That trace tells you the User Service is the bottleneck — not the Auth Service, not the gateway. Without tracing, you'd have no idea.

As a QA engineer, you don't need to set up these tools — but you do need to learn how to read them. Ask your team to walk you through how they trace requests. It will make you significantly more effective at diagnosing failures.

Step 5: Test What Happens When Things Go Wrong

This is where microservice testing gets interesting — and where most junior QA engineers don't spend enough time.

In a monolith, you mostly test happy paths and a few negative scenarios. In microservices, failure is a normal part of the system.

Services go down.
Networks get slow.
Events get delayed.

The system is supposed to handle these gracefully. Your job is to verify that it does.

Failure scenarios worth testing:

Scenario	What to verify
A dependent service is down	Does the system return a clear error? Does it retry?
Network latency is high	Does the service timeout correctly?
An event is published twice	Does the consumer handle duplicates without creating duplicate records?
A service restarts mid-transaction	Is the transaction completed, rolled back, or left in a broken state?

Concrete example:

Your team builds a checkout flow. The Payment Service is unavailable.

Questions to test:

Does the Order Service still create an order record? Should it?
Does the user see “Payment failed, please try again” — or a generic 500 error?
Is there a retry mechanism? How many times does it retry?
If the payment eventually succeeds on retry, does the order status update correctly?

These are not edge cases.
These are real production scenarios.

Test them.

The Testing Pyramid for Microservices

Effective microservice testing doesn't happen at one level — it happens at four:

1. Unit Testing

Developers test individual functions in isolation.
You won't own this, but you should know it exists.

2. Service Testing

Each microservice is tested independently, with its dependencies mocked.
You verify that the service behaves correctly on its own.

3. Integration Testing

Two or more services are tested together.
You verify that Service A and Service B communicate correctly under real conditions.

4. End-to-End Testing

You test the complete user workflow across all services.
This is where most QA engineers spend their time, and where most real-world failures are caught.

The further up the pyramid you go, the slower and more expensive the tests become — but also the more realistic.

Balance matters.

What This Means for Your Career

The QA role is evolving.

In modern teams, a QA engineer who only knows how to write UI test scripts has a ceiling.

A QA engineer who understands distributed systems, can read traces, and thinks in flows rather than features — that person is genuinely hard to replace.

You don't need to become a software architect. But you do need to understand the system you're testing well enough to ask the right questions and catch the bugs that only appear at the seams between services.

Final Thought

The hardest bugs in microservice systems don't live inside a single service.

They live in the space between services — in the assumptions each one makes about the others.

If you're only testing the API response in front of you, you're seeing one part of a much larger picture.

Start with the architecture diagram.
Follow the flow.
Test the failures.

That's where the real testing work is.

Shazzadur Rahman
Senior QA Engineer | AI Dashcam & Telematics Testing

CAP Theorem Explained: What I Learned Setting Up MySQL Replication

Mon, 11 May 2026 11:37:29 +0000

title: CAP Theorem Explained: What I Learned Setting Up MySQL Replication
published: true

tags: systemdesign, mysql, distributedsystems, beginners

I recently started my journey into system design and data-intensive applications. This experience has completely changed how I look at software — especially around the hard tradeoffs of consistency, availability, and what happens when things go wrong at scale.

Let me start with a scenario that surprised me when I first encountered it:

Imagine you just inserted a record in your app. A millisecond later, a read on a different server returns the old data. How? Welcome to distributed systems.

What is CAP Theorem?

CAP stands for Consistency, Availability, and Partition Tolerance. The theorem states that a distributed system can only guarantee 2 out of these 3 properties at any given time.

Consistency — Every read returns the most recent write, or an error. All nodes in the system see the same data at the same time.

Availability — Every request gets a response (not an error), even if some nodes are down. The system is always up.

Partition Tolerance — The system keeps running even when network communication between nodes is lost or delayed.

The Real Choice: CP vs AP

Here is the most important thing to understand: you cannot actually drop Partition Tolerance.

Network partitions are not a design choice — they are a reality. Hardware fails, packets drop, data centers lose connectivity. Any system that runs across multiple machines will eventually face a partition.

So the real decision every distributed system makes is: when a partition happens, do you stay consistent or stay available?

A Simple Analogy: The Bank Branch

Imagine a bank with its HQ in New York and a branch in Boston. The branch always calls HQ before confirming any transaction.

A customer in Boston deposits $500. Right at that moment, the network between Boston and New York goes down.

The Boston branch now has two options:

Option 1 — CP (Consistency + Partition Tolerance):
The branch refuses to process any transactions until it can reach HQ. The data stays consistent, but the service is unavailable during the outage. If you walk up to the counter, you get turned away.

Option 2 — AP (Availability + Partition Tolerance):
The branch processes the deposit locally and syncs with HQ once the network recovers. The service stays available, but if someone checks the account balance at the New York branch before the sync, they see the old amount — temporarily inconsistent data.

Neither option is wrong. The right choice depends entirely on what your application can tolerate.

Hands-On: MySQL Replication as an AP System

To make this concrete, I built a small master-slave replication setup using Docker to see these tradeoffs in action.

You can find the full setup in this repository.

How it works:

One MySQL container acts as the master — it accepts all writes
A second container acts as the slave — it monitors the master and replays every change
The master records every write to a binary log (binlog)
The slave runs two threads: an IO thread that pulls the binlog from the master, and a SQL thread that replays those events locally

What I observed:

When I inserted rows on the master, they showed up on the slave within milliseconds under normal conditions. But when I inserted 10,000 rows rapidly, I could watch the Seconds_Behind_Master value in SHOW SLAVE STATUS grow and then shrink back to zero as the slave caught up.

That gap — replication lag — is the tradeoff in action. During those seconds, a read on the slave returns stale data. The slave is always available to serve reads, but it cannot guarantee it has the latest write.

This makes the setup an AP system: Available and Partition Tolerant, but not strongly Consistent.

If you wanted CP behaviour, you would switch to synchronous replication — the master waits for the slave to confirm every write before acknowledging it to the client. You get consistency, but every write now pays the latency cost of that round-trip.

When to Choose CP vs AP

Prefer CP when a wrong answer is worse than no answer:

Bank balances and financial transactions
Seat reservation systems (cinema, flights)
Distributed locks and coordination

Prefer AP when brief staleness is acceptable:

Social media feeds
Product catalogs and shopping carts
DNS resolution
Caching layers

Beyond CAP: Consistency is a Spectrum

CAP theorem is a useful mental model, but it is a simplification. In practice, consistency is not binary — there is a whole spectrum:

Linearizability — the strongest guarantee; reads always reflect the latest write
Sequential consistency — operations appear in the same order across all nodes, but may lag
Eventual consistency — all nodes will converge to the same value, given enough time (this is what MySQL async replication gives you)

If you want to go deeper, Designing Data-Intensive Applications by Martin Kleppmann covers replication in Chapter 5 and consistency models in Chapter 9. It is genuinely the best resource I have found on this topic.

Conclusion

CAP is not a theorem you apply once and move on — it is a lens for thinking about tradeoffs. Every distributed system you design makes a CAP choice, whether you make it explicitly or not. Understanding that choice, and being able to justify it, is what separates a system that works in production from one that only works in theory.

Why Your "Fail-Fast" Strategy is Killing Your Distributed System (and How to Fix It)

Tue, 05 May 2026 19:11:27 +0000

It's 2 AM. PagerDuty fires. Redis master is down. Your application, trained to fail fast, dutifully fails — every single request, all at once. By the time Sentinel promotes a new master 12 seconds later, you've already generated 40,000 errors and three escalation calls. The system recovered on its own. Your application didn't let it.

This is the story of how "good engineering" can make a 12-second infrastructure event into a 12-minute outage — and how to design boundaries that prevent it.

tl;dr — During infrastructure failovers (Redis, Kafka, etcd), blind fail-fast amplifies instability. Bounded retry — centralized, time-boxed, invisible to business logic — absorbs the 10–15 second recovery window without leaking infrastructure noise to users. Resilience is not a library. It is a contract between layers.

The Core Question

When your session storage — Redis, Memcached, or any stateful dependency — goes temporarily unavailable, you face a fundamental architectural choice:

Should you fail fast? Or should you retry?

We all learned fail-fast as gospel. And it is — until it isn't. During transient infrastructure events like leader elections, blind fail-fast propagates instability instead of containing it. The response you choose determines whether the incident resolves itself in 12 seconds or snowballs into a 12-minute outage with three bridge calls.

What Actually Happens During Failover

To understand why fail-fast can backfire, look at the mechanics of a Redis Sentinel failover:

Phase	Duration	What Happens
Detection	~10–12s	Sentinel quorum detects master is down
Election	~1–2s	Sentinels agree on a new master
Promotion	~1s	Replica promoted, clients notified
Reconnection	~1–3s	Clients re-establish connections

Note: these phases overlap. Total failover typically completes in 12–15 seconds, not the sum of individual phases. Reconnection time also depends heavily on your client library — a Sentinel-aware client with topology refresh (e.g., Lettuce, go-redis with Sentinel support) reconnects in under a second, while a naive connection pool can take 30s+.

During this window, your application sees TCP dial timeouts and connection resets. Nothing is broken. No data is lost. The system is doing exactly what it was designed to do — electing a new leader. Your application just needs to not panic for 12 seconds.

If your application fails immediately on the first connection timeout during this window, four things happen in rapid succession:

1. Instability Amplification

A 3-second infrastructure blip becomes a user-visible outage. Every request during the failover window returns an error, even though the system would have recovered on its own.

2. Infrastructure Semantics Leak Upward

Your business layer now exposes raw infrastructure details — "Redis connection refused" — to clients that have no idea what Redis is or why it matters.

3. Uncontrolled Client Retries

Clients receiving errors start retrying independently. If you have 1,000 concurrent users and each retries 3 times, you just turned 1,000 QPS into 3,000 QPS — hitting an infrastructure layer that's already struggling to stabilize.

4. Retry Storms

This is the catastrophic outcome. Unbounded retries create cascading load amplification. CPU spikes prevent recovery. The system enters an instability feedback loop where the act of trying to recover keeps the system down. I've seen retry storms take down entire regions.

"Your timeout config was technically correct. Your system was functionally down. That's not a timeout problem — that's a design problem."

Here's the distinction that actually matters in production: the failure TYPE must determine your recovery strategy.

	Infrastructure-Level	Business-Level
Examples	Network jitter, leader election, connection reset, `READONLY` replica response	Validation error, permission denial, domain rule violation
Nature	Transient — will resolve on its own	Permanent — retrying won't help
Strategy	ABSORB — retry within bounds	FAIL FAST — return error immediately

Treating a leader election timeout the same as a schema validation error is an architectural mistake. One will resolve in seconds; the other will never succeed no matter how many times you retry.

The Failure Boundary Model

This is the architectural pattern that makes everything work:

The retry boundary sits in the infrastructure client wrapper — the thin layer between your business code and the dependency client. Not in HTTP middleware, not in individual service handlers, not in a sidecar. In the client wrapper itself.

Why does this matter? Because if retry logic exists at multiple layers, you get retry amplification. I've seen teams with retry in the HTTP handler, the service layer, AND the Redis client — producing 3 × 3 × 3 = 27 attempts per original request. That's not resilience. That's a DDoS against your own infrastructure.

Key principles:

Retry belongs at the infrastructure boundary — one place, one policy.
Business logic must remain fail-fast — semantic errors should never be retried.
By the time an error reaches the client, it has been vetted and classified. We are designing for predictability.

Bounded Retry: Implementation

If we're going to retry, we must do it with discipline. Four pillars:

1. Centralized

Retry logic lives in one place — the infrastructure client wrapper. Not in individual handlers, not in middleware, not in the business layer. One retry boundary per dependency, one policy, one set of metrics.

2. Time-Bounded

We define a retry budget — for example, 15 seconds. Why 15? Because it encapsulates the 10–12 second Sentinel detection window plus a margin for stabilization and reconnection. Time-based budgets are superior to pure attempt counts because they normalize across different failure modes — a retry that takes 5s per attempt behaves very differently from one that takes 100ms.

3. Attempt-Limited with Jitter

Maximum 2–3 retry attempts within the budget window, with exponential backoff and jitter. Without jitter, synchronized retries from multiple application instances create a thundering herd — everyone hits the new master at exactly the same moment.

4. Invisible to Business Logic

If the retry succeeds within the budget, the business layer never knew there was a problem. If it fails, the business layer receives a clean, classified error — not a raw TCP stack trace that means nothing to anyone above the infrastructure layer.

Here's what this looks like in practice:

// Bounded retry wrapper — lives in the infrastructure client layer
func withBoundedRetry(ctx context.Context, budget time.Duration, maxAttempts int, op func() error) error {
    deadline := time.Now().Add(budget)
    var lastErr error

    for attempt := 0; attempt < maxAttempts; attempt++ {
        if time.Now().After(deadline) {
            break
        }

        lastErr = op()
        if lastErr == nil {
            return nil // success — business layer never knew
        }

        if !isRetryable(lastErr) {
            return normalizeError(lastErr) // permanent failure — fail fast
        }

        // Exponential backoff with jitter
        backoff := time.Duration(1<<attempt) * 500 * time.Millisecond
        jitter := time.Duration(rand.Int63n(int64(backoff / 2)))
        select {
        case <-time.After(backoff + jitter):
        case <-ctx.Done():
            return ctx.Err()
        }
    }

    return normalizeError(lastErr) // budget exhausted — fail deterministically
}

┌─────────────────────────────────────────────┐
│          Retry Budget: 15 seconds           │
│                                             │
│  Attempt 1  →  timeout (5s)  →  backoff     │
│  Attempt 2  →  timeout (5s)  →  backoff     │
│  Attempt 3  →  success                      │
│                                             │
│  Total elapsed: ~11s                        │
│  Application impact: ZERO                   │
│                                             │
│  ─── OR ───                                 │
│                                             │
│  Budget exhausted → FAIL DETERMINISTICALLY  │
│  Clean, classified error to business layer  │
└─────────────────────────────────────────────┘

"Retry is not infinite. Retry is time-boxed. Once the budget is exhausted, we fail deterministically."

Error Normalization

This is where most teams get it wrong. They retry everything — or nothing. The retry decision must be driven by error classification:

Raw Error	Normalized To	Retryable?	Why
`TCP dial timeout`	`UNAVAILABLE`	Yes	Connection not established, may recover
`Connection reset`	`UNAVAILABLE`	Yes	Transient network disruption
`READONLY` (replica)	`UNAVAILABLE`	Yes	Sentinel failover in progress — replica not yet promoted
`Leader election in progress`	`UNAVAILABLE`	Yes	Raft/consensus transition
`OOM command not allowed`	`RESOURCE_EXHAUSTED`	No	Backpressure — retrying makes it worse
`WRONGTYPE`	`INVALID_ARGUMENT`	No	Schema error — will never succeed
`NOPERM` / `Permission denied`	`PERMISSION_DENIED`	No	Auth failure — will never succeed
`NOT_FOUND`	`NOT_FOUND`	No	Semantic absence — retry won't create the resource

The READONLY case deserves special attention. During Sentinel failover, a replica that hasn't been promoted yet responds with READONLY to write commands. If your retry layer treats this as a permanent error, your circuit breaker trips, clients get errors, and a 12-second failover becomes a 5-minute outage while someone manually resets the breaker. Classify READONLY as UNAVAILABLE — it will resolve when the new master is promoted.

The rule is simple: you cannot leak internal implementation details up the stack. Your retry layer must inspect and reclassify errors — not just map them 1:1. Error semantics must align across every layer.

The Relationship with Circuit Breakers

Bounded retry is the inner loop — it handles transient failures within a known recovery window. But what if the dependency is truly down, not just transitioning?

That's where circuit breakers serve as the outer loop:

Bounded retry absorbs transient events (leader election, network jitter) — seconds.
Circuit breaker protects against sustained outages (dependency truly dead) — minutes.

Without a circuit breaker, sustained failures chew through retry budgets on every request, wasting resources. Without bounded retry, every transient blip trips the circuit breaker unnecessarily. They are complementary, not redundant.

Observability: Instrument the Boundary

A production retry boundary must emit metrics. Without them, you're flying blind:

retry_attempt_total — how often retries fire (by dependency, by error type)
retry_budget_exhausted_total — how often the full budget is consumed without success
retry_success_on_attempt — which attempt number succeeds (histogram)
error_classification — distribution of retryable vs non-retryable errors

The key alert: if retry budget exhaustion rate exceeds ~5%, either your budget is too tight or your dependency is degraded beyond transient. This is the signal that distinguishes a leader election from a real outage — and it's the signal that should trigger your circuit breaker.

Beyond Redis: A Universal Pattern

If this looks Redis-specific, zoom out. The bounded retry pattern applies to any stateful dependency with leader election:

Redis Sentinel — master failover with quorum detection, 10–15s window
NATS JetStream — stream leader election in the Raft group, typically 2–5s with default election timeout
etcd / Consul — Raft leader election, ~1–2s with default settings, but watch streams may buffer longer
Kafka — partition leader election via controller, typically 5–15s depending on replica.lag.time.max.ms and ISR size
CockroachDB / TiKV — range leader election, similar Raft mechanics

The mechanics are the same everywhere: a detection window, a brief period of unavailability, and then recovery. Design your retry budget to absorb that window. Calibrate the budget to the specific system — 15s for Redis Sentinel, 5s for NATS, 20s for Kafka.

The Cross-Layer Contract

Resilience is not a library you import. It is a contract between layers:

Layer	Responsibility
Infrastructure	Absorbs transient instability via bounded retry
Business	Remains fail-fast for semantic integrity
Client	Retries only when signaled retryable

When failure is bounded and classified, the system becomes predictable. And predictability is the foundation of operational confidence.

Resilience Checklist

[ ] Retry Budget: Is my retry window matched to the dependency's failover time (e.g., 15s for Redis)?
[ ] Jitter: Do my retries have randomized sleep to avoid the "Thundering Herd"?
[ ] Error Classification: Does my code distinguish between READONLY (retryable) and PERMISSION_DENIED (not retryable)?
[ ] Centralization: Is my retry logic in the client wrapper, not leaked across handlers?
[ ] Observability: Do I have an alert if "Retry Budget Exhausted" exceeds 5%?

Key Takeaways

Fail fast — but not during transient infrastructure events. A leader election is not a business error. Don't treat it like one.
Retry must be bounded. Time-boxed, attempt-limited, with jitter. No open-ended retry loops.
Retry must be centralized. One retry boundary per dependency, at the infrastructure layer. Retry in multiple layers = retry amplification.
Failure semantics must be normalized. Retryable vs non-retryable must be explicit. Watch for READONLY — the most common Sentinel failover gotcha.
Resilience requires cross-layer alignment. Bounded retry (inner loop) + circuit breaker (outer loop) + observability = production-grade resilience.

Frequently Asked Questions

Should distributed systems always fail fast?

No. Fail fast for business-level errors (validation, permission, domain rules), but use bounded retry for transient infrastructure failures like leader election and temporary network instability.

What is a reasonable retry budget for Redis Sentinel failover?

In many production setups, 12-15 seconds is a practical starting point because it usually covers Sentinel detection, promotion, and client reconnection. Calibrate with your own failover timings and SLOs.

If the service already retries, should the client also retry?

Only when explicitly signaled retryable. Blind retries at both layers often create retry amplification and can trigger a retry storm.

How is bounded retry different from a circuit breaker?

Bounded retry handles short transient windows (inner loop). Circuit breaker handles sustained dependency failure and stops repeated expensive attempts (outer loop).

Why not use a Service Mesh (Istio) for retries?

While Mesh can retry, the application layer has better "semantic awareness." Only the app knows if a specific error is safe to retry based on idempotency.

When should I NOT use Bounded Retry?

For non-idempotent operations unless you have a robust request-ID tracking system. For business errors (400s), always fail fast.

Final Thought

Distributed systems are not about avoiding failure.
They are about designing boundaries.

If retry is everywhere, the system becomes unpredictable.
If retry is nowhere, transient instability leaks upward.

The goal is not infinite retry.
The goal is bounded retry.

That boundary is what keeps systems stable.

Resilience is not a library. It is a contract between layers.

Based on a talk I gave on failure boundary design in distributed systems.

Originally published at harrisonsec.com. Listen to the deep dive audio for a detailed walkthrough.

Building a Microservices Ecosystem: Stock Brokerage Simulator (My Broker B3)

Fri, 01 May 2026 13:50:40 +0000

Hey folks!

I'm starting a series of articles to document the development of My Broker B3 — a personal project where I apply advanced software engineering concepts, distributed systems and messaging to simulate the real-world operation of a stock brokerage integrated with B3 (the Brazilian stock exchange).

The goal isn't to build a production system, but rather a POC (Proof of Concept) that validates real-time order processing, financial custody management and market data orchestration using an event-driven architecture.

️ The Ecosystem Overview

The project was designed following the microservices philosophy with a hybrid stack — each service uses the technology that best fits its function:

                    [broker-gateway-api]
                           │
          ┌────────────────┼────────────────┐
          ▼                ▼                ▼
   [broker-order-api] [broker-wallet] [broker-asset]
          │                │
     RabbitMQ           Kafka (order-events-v1)
          │
   [b3-matching-engine]
          │
       Redis ◀── [b3-market-sync-api] ◀── [brapi.dev]

The Services

Broker Side

Service	Language	Responsibility
`broker-identity-api`	Java	Authentication, JWT, user management
`broker-gateway-api`	Java	API Gateway, routing, JWT validation
`broker-order-api`	Java	Order lifecycle orchestration
`broker-wallet-api`	Java	Financial custody, balance and positions
`broker-asset-api`	Java	Asset catalog and internal market data
`broker-market-data-api`	Python	Quote ingestion from Brapi → MongoDB + Kafka
`broker-report-api`	Java	Reports and history

B3 Side (Simulator)

Service	Language	Responsibility
`b3-matching-engine-api`	Java	Order execution, price matching
`b3-market-sync-api`	Java	Price synchronization to Redis

⚙️ Communication Strategy

One of the most important decisions in the project was choosing when to use synchronous communication and when to use asynchronous. The rule was simple: if the response is needed now to continue the flow, use REST. If it can wait, use messaging.

Synchronous (REST/Feign)

broker-order-api → broker-asset-api: validates ticker before creating order
broker-order-api → broker-wallet-api: validates balance before processing buy
broker-identity-api → broker-wallet-api: creates wallet account on registration

Asynchronous (RabbitMQ)

broker-order-api → b3-matching-engine-api: sends orders for execution
b3-matching-engine-api → broker-order-api: execution feedback (FILLED/REJECTED)

Asynchronous (Kafka)

broker-market-data-api → topic assets-market-data-v1 → broker-asset-api
broker-order-api → topic order-events-v1 → broker-wallet-api

Persistence Strategy

Each service uses the database that best fits its nature:

Technology	Service(s)	Why
MySQL	identity, wallet, order, asset	Transactional data with strong consistency
PostgreSQL	b3-matching-engine	B3 core with execution history
MongoDB	broker-market-data	Quote history — flexible schema
Redis	broker-asset, b3-matching-engine	Real-time price cache for fast decisions

The Order Flow — End to End

To materialize the architecture, here's what happens when a user places a buy order:

1. POST /api/v1/orders
        │
        ├─ Validate balance → broker-wallet-api (REST)
        ├─ Persist PENDING → MySQL
        ├─ Publish PENDING → Kafka (broker-wallet blocks balance)
        └─ Send to B3 → RabbitMQ

2. B3 processes, looks up price in Redis
        │
        ├─ FILLED/REJECTED → RabbitMQ → broker-order-api
        ├─ Update status → MySQL
        └─ Publish final status → Kafka (broker-wallet settles/refunds)

From the user's click to the updated wallet balance, through validation, price matching and eventual consistency.

What You'll Find in the Series

Each post documents the construction of a service, focusing on technical decisions, problems encountered and how they were solved:

Overview ← you are here
Infrastructure — Docker Compose with 12+ containers
broker-market-data-api — Python, Brapi, MongoDB, Kafka
b3-market-sync-api — Price synchronization to Redis
b3-matching-engine-api — The B3 execution engine
broker-asset-api — Asset catalog and cache
broker-wallet-api — Financial custody
broker-order-api — The order conductor
broker-identity-api — Authentication and JWT
broker-gateway-api — API Gateway

Feel free to leave feedback or questions in the comments!

About the Series

Series Index: Series Roadmap

Links:

Master-Class: Scaling Databases with Sharding, Partitioning, and Consistent Hashing

Sun, 19 Apr 2026 19:53:01 +0000

title: "Master-Class: Scaling Databases with Sharding, Partitioning, and Consistent Hashing"
published: true
description: "Learn the three pillars of horizontal scaling: Sharding, Partitioning, and Consistent Hashing, and how they relate to replication."

tags: database, systemdesign, backend, distributedsystems

Scaling a database is often the "final boss" of system design. While scaling your application layer is as simple as spinning up more containers, databases are stateful, making them significantly harder to distribute.

When vertical scaling hits a physical limit, you have no choice but to scale horizontally. In this guide, we break down the three pillars of horizontal scaling and how they intersect with replication strategies to build truly resilient systems.

Sharding vs. Partitioning: The Core Difference
Partitioning Strategies: Horizontal vs. Vertical
The "Modulo" Problem: Why Simple Hashing Fails
Consistent Hashing: The Secret to Rebalancing
The Hybrid Model: Relating Replication to Sharding
The Complexity Trade-offs

Sharding vs. Partitioning: The Core Difference

While often used interchangeably, they operate at different layers of the stack:

Partitioning: This is a logical split of your data. It’s about taking a giant dataset and breaking it into smaller, manageable chunks (e.g., splitting a 1TB table into ten 100GB chunks).
Sharding: This is a physical split across machines. It involves distributing those partitions across different database instances (shards) to distribute the CPU, RAM, and I/O load.

The Golden Rule: You shard the database and you partition the data.

Partitioning Strategies

How do you decide which data goes into which partition? It must be deterministic.

A. Horizontal Partitioning (Sharding)

You split the table by rows. For example, users with IDs 1–1000 go to Partition A, and 1001–2000 go to Partition B.

Best for: Distributing massive datasets across nodes for write/read throughput.

B. Vertical Partitioning

You split the table by columns. A User table might be split into User_Profile (Name, Email) and User_Metadata (Bio, Settings, Blob data).

Best for: Reducing I/O for queries that only need specific, frequently accessed columns.

The "Modulo" Problem: Why Simple Hashing Fails

When sharding, we need to map a key (like user_id) to a specific server. The simplest way is Modulo Hashing:

{% katex %}
server_index = hash(key) \pmod n
{% endkatex %}

Where {% katex inline %} n {% endkatex %} is the number of servers. This works until you need to scale. If you have 3 servers ({% katex inline %} n=3 {% endkatex %}) and you add a 4th ({% katex inline %} n=4 {% endkatex %}), the result of the modulo operation changes for almost every single key. This forces a massive data migration that can bring your system down.

Consistent Hashing: The Secret to Rebalancing

Consistent Hashing solves the "Modulo Problem" by decoupling the keys from the number of servers using a Hash Ring.

How it Works:

The Ring: Both servers and data keys are hashed onto the same ring (range {% katex inline %} 0 {% endkatex %} to {% katex inline %} 2^{32}-1 {% endkatex %}).
Clockwise Assignment: To find a key's server, you move clockwise on the ring until you hit the first server.
Minimal Migration: When you add a new server, it only takes over a small portion of the ring. Only the keys between the new server and its counter-clockwise neighbor need to move.

The Hybrid Model: Relating Replication to Sharding

Sharding handles capacity, but Replication handles availability. In a production system, every "Shard" is actually a "Replication Group."

A. Sharded + Single-Leader Replication

Each shard is a replica set with one Leader and multiple Followers.

The Workflow: Consistent hashing sends you to Shard 4. Within Shard 4, all writes go to the Leader, and reads can be distributed to Followers.
Example: Vitess (MySQL) or MongoDB.

B. Sharded + Multi-Leader Replication

Common in multi-region deployments. You shard by geography (e.g., EU users vs. US users), but within each shard, you have multiple leaders (e.g., London and Paris) to reduce local latency.

Example: Global Spanner configurations.

C. Sharded + Leaderless Replication (The "Dynamo" Way)

In systems like Cassandra, sharding and replication are unified on the ring.

The Workflow: A key is hashed to the ring. Instead of going to one node, it is replicated to the next N nodes clockwise.
The Quorum: You use {% katex inline %} W + R > N {% endkatex %} to ensure consistency across those sharded replicas.

The Complexity Trade-offs

Sharding is powerful, but it comes with a high "architectural tax":

Cross-Shard Joins: Joining data from Shard A and Shard B is incredibly expensive. You often have to perform the join in the application layer.
Hotspots: If one user (e.g., a celebrity) gets 100x more traffic than others, the shard they reside on will become a bottleneck.
Operational Overhead: Managing backups, monitoring, and failover for 100 shards is significantly more complex than a single instance.

Summary Comparison

Feature	Modulo Hashing	Consistent Hashing
Scaling Impact	~100% data migration	~$1/n$ data migration
Fault Tolerance	Low	High
Replication Choice	Usually Single-Leader	Often Leaderless / Multi-Leader

Final Thought

Sharding is how we find the "house" for our data; Replication is how we ensure that house doesn't burn down. Don't shard until you hit the physical limits of your hardware, but when you do, use Consistent Hashing to ensure your growth is sustainable.

{% link https://dev.to/piyush6348/master-class-understanding-database-replication-single-multi-and-leaderless-hhm %}

Master-Class: Understanding Database Replication (Single, Multi, and Leaderless)

Sun, 19 Apr 2026 19:52:19 +0000

title: "Master-Class: Understanding Database Replication (Single, Multi, and Leaderless)"
published: true
description: A deep dive into Single Leader, Multi-Leader, and Leaderless replication algorithms for distributed systems.

tags: systemdesign, backend, database, distributedsystems

Database Replication is the process of keeping a copy of the same data on multiple nodes. Whether you are aiming for high availability, reduced latency, or horizontal scalability, choosing the right replication algorithm is critical.

In this guide, we will explore the three primary algorithms used in modern distributed systems: Single Leader, Multi-Leader, and Leaderless.

Single Leader Replication
Multi-Leader Replication
Leaderless Replication
The Replication Lag Problem
Summary Comparison

1. Single Leader Replication

This is the most common approach (used by MySQL, PostgreSQL, and MongoDB). One node is designated as the leader (master), and all other nodes are followers (read replicas).

How it Works

Writes: All write requests must be sent to the leader. The leader writes the data locally and sends the change to all followers.
Reads: Clients can read from the leader or any follower.

Synchronous vs. Asynchronous

Synchronous: The leader waits for followers to confirm the write.
- Pros: Guaranteed consistency.
- Cons: High latency; if one node fails, the whole write pipeline blocks.
Asynchronous: The leader confirms the write immediately.
- Pros: High performance.
- Cons: Risk of data loss if the leader fails before followers sync.

Handling Failures

Follower Failure: A follower "catches up" by using its local log to request missing data from the leader.
Leader Failure (Failover): Requires detecting failure via timeouts, electing a new leader, and reconfiguring the system.

2. Multi-Leader Replication

In this setup, more than one node can accept writes. This is typically used for applications spread across multiple geographic data centers.

Use Cases

Multi-Data Center Operation: Users write to the nearest data center to reduce latency.
Offline Operation: Apps like calendars or note-taking tools act as local "leaders" that sync with a server later.

The Challenge: Conflict Resolution

If two users edit the same data in different data centers simultaneously, a conflict occurs.

Conflict Avoidance: Routing all writes for a specific record to the same leader.
Convergence: Using Last Write Wins (LWW) or Conflict-free Replicated Data Types (CRDTs) to merge changes.

3. Leaderless Replication

Popularized by Amazon’s Dynamo, this approach allows any node to accept writes and reads. Systems like Cassandra and Riak use this model.

Quorums ($n, w, r$)

To maintain consistency without a leader, these systems use quorums:

$n$: Total number of replicas.
$w$: Nodes that must confirm a write.
$r$: Nodes that must be queried for a read.
The Rule: For a successful read of the latest data, $w + r > n$.

Fixing Stale Data

Since nodes can go down, systems fix stale data via:

Read Repair: When a client detects an old version during a read, it pushes the newer value back to that node.
Anti-Entropy: A background process that constantly syncs data between replicas.

The Replication Lag Problem

Regardless of the algorithm, asynchronous replication often results in "replication lag." To maintain a good user experience, developers should implement:

Read-Your-Own-Writes: Ensures a user always sees the updates they just made.
Monotonic Reads: Ensures a user doesn't see data "disappear" when querying different replicas.
Consistent Prefix Reads: Guarantees that if writes happen in a specific order, they are read in that same order.

Summary Comparison

Algorithm	Best For	Main Downside
Single Leader	Read-heavy apps, general simplicity	Leader is a single point of failure for writes
Multi-Leader	Multi-region apps, offline capabilities	Extremely complex conflict resolution
Leaderless	High write throughput, high availability	Complexities in eventual consistency

Transaction Orchestration in Distributed Financial Systems: Coordination, Idempotency, and Eventual Consistency

Thu, 16 Apr 2026 15:38:23 +0000

Abstract

Distributed financial systems are composed of multiple subsystems, each responsible for enforcing a specific invariant. Ledger systems preserve correctness, custody systems enforce authority, compliance systems constrain allowed behavior, and smart contracts provide deterministic settlement.

However, real systems must coordinate these components under conditions of latency, partial failure, and inconsistent state visibility. This coordination problem is often underestimated and is responsible for many of the most subtle and dangerous production failures.

This article explores transaction orchestration in distributed financial systems, focusing on coordination strategies, idempotency guarantees, failure handling, and the realities of eventual consistency.

Correct components do not guarantee correct execution.

The illusion of a single transaction

When designing systems, it is tempting to think in terms of a single operation.

A withdrawal.
A transfer.
A settlement.

In reality, what appears as a single transaction is a sequence of distributed operations across multiple services.

A typical flow may involve:

ledger validation
compliance evaluation
risk checks
custody signing
settlement broadcast

Each step runs in a different context. Each step may fail independently.

The system does not execute one transaction.

It orchestrates a sequence of state transitions.

Coordination under uncertainty

Distributed systems do not operate under perfect conditions.

Messages may arrive late.
Services may retry operations.
Nodes may crash mid execution.

Two services may have different views of the same transaction at the same time.

This creates a fundamental challenge.

There is no global clock.
There is no perfectly synchronized state.

And yet, the system must behave as if there were.

Coordination is the mechanism that creates this illusion.

Idempotency as a safety guarantee

In financial systems, retries are inevitable.

If a request times out, it will be retried.
If a service crashes, operations may be replayed.

Without protection, this leads to duplication.

A withdrawal could be executed twice.
A settlement could be broadcast multiple times.

This is unacceptable.

Idempotency ensures that applying the same operation multiple times produces the same result.

text id=l7wq2r
apply(operation, state) multiple times
=> same final state

This property must exist across service boundaries, not just within a single component.

Eventual consistency and controlled divergence

Strong consistency across all components is rarely achievable in distributed systems.

Instead, systems operate under eventual consistency.

Different services may temporarily disagree on state.

The critical requirement is not immediate agreement.

It is bounded and controlled convergence.

The system must guarantee that all components eventually reach a consistent view of the transaction outcome.

Unbounded divergence leads to reconciliation problems and operational uncertainty.

Orchestration models

There are multiple ways to coordinate distributed transactions.

Centralized orchestration relies on a coordinator service that drives execution.

Choreography relies on event driven interaction between services.

Each model has tradeoffs.

Centralized orchestration simplifies reasoning but introduces a control dependency.
Choreography increases decoupling but makes reasoning about global state more complex.

Financial systems often use hybrid approaches, combining explicit coordination with event driven propagation.

Failure handling and partial execution

Failures rarely occur at clean boundaries.

A transaction may pass compliance and fail during custody signing.
A signature may be produced but not broadcast.
A broadcast may succeed but not be recorded internally.

The system must handle these partial states.

This requires:

clear state modeling
explicit transition tracking
safe retry mechanisms
reconciliation processes

Failure handling is not an edge case. It is the dominant execution path.

The danger of implicit sequencing

One of the most common sources of bugs is implicit sequencing.

Assuming that because step A happened before step B in code, it also happened before in the system.

In distributed environments, this assumption does not hold.

Messages can be reordered.
Events can be delayed.

Sequencing must be explicit.

Each step must validate that its preconditions are still valid at execution time.

Orchestration defines system behavior

Ledger enforces correctness.
Custody enforces authority.
Compliance enforces constraints.

But orchestration defines how the system behaves under real conditions.

It determines:

how failures propagate
how retries are handled
how state converges
how inconsistencies are resolved

This is where most production issues originate.

Conclusion

Distributed financial systems do not execute transactions in a single step. They orchestrate sequences of operations across multiple services, each with its own state and failure modes.

Correctness at the component level is necessary but insufficient. Systems must coordinate execution under uncertainty, ensuring that retries, delays, and partial failures do not violate global invariants.

Transaction orchestration is the layer that transforms correct components into a functioning system.

Without it, correctness remains theoretical.

Financial Systems as Composed State Machines: Correctness, Authority, and System Integrity

Thu, 16 Apr 2026 15:37:45 +0000

Abstract

Modern financial systems are often described in terms of individual components. Ledgers enforce correctness. Custody systems control asset authority. Compliance systems constrain behavior. Orchestration layers coordinate execution.

Each of these components can be designed correctly in isolation.

Yet production failures continue to occur in systems that are individually sound.

This article argues that financial infrastructure must be understood not as a collection of services, but as a composition of interacting state machines. We examine how correctness, authority, and policy enforcement interact across subsystem boundaries, and why system integrity emerges only when these components are composed under strict constraints.

Financial systems do not fail because components are incorrect. They fail because composition is undisciplined.

The illusion of correct systems

It is possible to build a ledger that enforces conservation of value.

It is possible to design custody systems that eliminate single key compromise through threshold cryptography.

It is possible to implement compliance engines that correctly evaluate regulatory constraints.

It is possible to orchestrate transactions across distributed services.

Each of these can be done correctly.

And yet, the system can still fail.

This is the point where most engineering intuition breaks.

Correctness at the component level does not imply correctness at the system level.

From services to state machines

The shift in perspective is subtle but critical.

Financial systems are not services exchanging messages.

They are state machines interacting through transitions.

Each subsystem can be modeled as:

S_i = (State_i, Transition_i)

Where:

State_i represents the internal state of subsystem i
Transition_i represents the allowed state transitions

For example:

Ledger enforces:

sum(entries(T)) = 0

Custody enforces:

signature(T) requires threshold participation

Compliance enforces:

policy(T, state) = allowed or rejected

Orchestration enforces:

execution(T) follows valid transition sequence

Individually, these systems define local invariants.

Composition defines global behavior

The full financial system can be modeled as:

S = Compose(S_ledger, S_custody, S_compliance, S_orchestration)

The critical property is:

Global correctness != sum of local correctness

Instead:

Global correctness = correctness of composition

This is where failures emerge.

If transitions across subsystems are not properly constrained, global invariants can be violated even when each subsystem behaves correctly.

Boundary violations and implicit coupling

Most real world failures originate at boundaries.

A custody system signs a transaction that was valid at evaluation time but invalid at execution time.

A compliance check passes based on stale state.

An orchestration layer retries an operation without ensuring idempotency across services.

A settlement completes externally but is not reflected internally.

Each subsystem behaves according to its own rules.

The failure occurs in how they interact.

This is not a bug in a component.

It is a failure of composition.

Temporal inconsistency and state divergence

Distributed systems introduce temporal uncertainty.

Different subsystems observe state transitions at different times.

This creates divergence.

A transaction may be:

valid in the ledger
approved by compliance
signed by custody

But still inconsistent globally because these decisions were made against different versions of state.

Without explicit coordination, the system operates on partially ordered events.

Correctness requires more than local validation.

It requires agreement on the context in which validation occurs.

The role of invariants across boundaries

Local invariants are necessary but insufficient.

Systems must enforce cross boundary invariants.

For example:

A transaction must not be signed if it violates compliance constraints at execution time.

A transaction must not be executed if its ledger state has changed since validation.

A transaction must not be retried if it has already been committed externally.

These invariants do not belong to a single subsystem.

They exist across the composition.

This is where most systems are weakest.

Failure is systemic, not local

Failures in financial systems are rarely isolated.

A retry in one service propagates to another.

A delayed message triggers inconsistent decisions.

A partial execution leads to duplicated operations.

The system fails as a whole.

This is why focusing only on component correctness is insufficient.

Engineers must reason about failure propagation across the entire system.

Compositional integrity

A system has compositional integrity when:

state boundaries are explicit
transitions are constrained across subsystems
sequencing is enforced
invariants are preserved globally

This requires discipline.

Not just in code, but in architecture.

Interfaces must encode constraints.

Events must carry sufficient context.

Operations must be idempotent across boundaries.

State must be versioned or validated at execution time.

Without these properties, composition becomes fragile.

Observability as proof of composition

One way to evaluate compositional integrity is through observability.

If a system cannot reconstruct the full lifecycle of a transaction across all subsystems, then its composition is not well defined.

Observability provides:

traceability across boundaries
visibility into sequencing
evidence of invariant preservation

It acts as a verification layer for system behavior.

The final model

We can describe a financial system as:

Global State S

Transitions:
  LedgerTransition
  CustodyTransition
  ComplianceTransition
  OrchestrationTransition

System integrity requires:

For all transitions T:
  invariants(S, T) are preserved across composition

This is not enforced by any single component.

It emerges from the architecture.

Conclusion

Financial systems are often engineered as collections of services, each responsible for a specific concern. This perspective is incomplete.

These systems are composed state machines whose correctness depends on the interaction of their components under real world conditions.

Ledger ensures correctness of value.
Custody ensures control of authority.
Compliance ensures validity of behavior.
Orchestration ensures coordination of execution.

But system integrity exists only when these are composed under strict constraints.

Without disciplined composition, even correct components produce incorrect systems.

Financial infrastructure is not defined by its parts.

It is defined by how those parts behave together.

Inside Airbnb’s Mussel v2: Rebuilding a Petabyte-Scale Key-Value Store

Tue, 14 Apr 2026 07:32:38 +0000

Airbnb recently published a detailed write-up on Mussel v2, the next generation of their internal key-value store for serving derived data. The system sits between offline data processing and online applications, turning large batch outputs and streaming updates into low-latency lookups used by systems like fraud detection, recommendations, and pricing.

The new architecture is a substantial redesign. It introduces a stateless dispatch layer, range-based sharding, stronger consistency controls, and a new storage backend designed to handle both large bulk loads and high-volume streaming updates in the same cluster. Migrating to it required moving more than a petabyte of data across thousands of tables, all while keeping production traffic online.

This article walks through both the original Mussel design and the new architecture, focusing on the engineering decisions behind the redesign.

What Mussel is - And Why it's a Hard Problem

Airbnb's fraud detection, recommendation engine, pricing models, and personalization systems all share a common need: take data computed offline - in warehouses and batch pipelines - and make it readable online at millisecond latency.

This is what engineers call the derived data problem. You can't query a data warehouse in real time. You can't fit 100TB into Redis. You need a system in the middle: one that can absorb large batch loads from offline compute, accept real-time streaming updates, and serve all of it under low-latency read paths.

Most teams end up with three separate systems for this: an object store for batch ingestion, a cache for reads, and a message queue for streaming writes. Each is optimized for its job, but keeping them in sync adds complexity and operational overhead.

By the time v1 was showing its limits, Mussel wasn't a small system. It was already serving over 4,000 client tables, managing roughly 130TB of data, handling 800k+ read QPS and 35k write QPS, with p95 read latency under 8ms. This was a mature, production-critical system - not a prototype being replaced by something grown-up.

What Was Breaking in v1

V1's problems weren't dramatic. They were the kind that build up slowly until they become a ceiling.

Operations were too manual. v1's storage layer was built on HRegion - the region server component from Apache HBase, powered by LSM trees, running locally on each Mussel storage node. HBase is capable, but it carries significant operational weight. Scaling v1 meant running multi-step Chef scripts on EC2 instances and carefully coordinating node additions in a system that wasn't designed for elastic scaling. None of this was automated. Every scaling event was a manual process that required institutional knowledge and took hours.

Hash partitioning created hotspots. v1 split data across 1024 logical shards by hashing primary keys. Apache Helix managed the mapping of those shards to physical nodes, and a leaderless replication model meant any replica of a shard could serve reads - which was good for read throughput. But hash partitioning gives you no control over where data actually lands. When access patterns aren't uniform - and they never are in production - certain nodes end up overloaded. Latency spikes follow. And fixing it requires rebalancing, which is more manual work.

Resource usage was invisible. v1 had no per-namespace quotas, no cost dashboards, no way for teams to see what their tables were consuming. At scale that becomes a reliability problem, not just a billing one. One team's runaway bulk load job can silently degrade the read latency of another team's fraud detection system.

These three problems - manual operations, uncontrollable hotspots, and opaque resource usage - aren't unique to Airbnb.

What v2 Changed and Why

Here's how the full v2 system is laid out:

┌──────────────────────────────────────────────────────┐
│                       Clients                        │
└──────────────────────────┬───────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────┐
│                Dispatcher (stateless)                │
│       routing · rate limiting · throttling           │
│   dual-write · shadow-read · per-table migration     │
└────────┬─────────────────┬──────────────────┬────────┘
         │                 │                  │
         ▼                 ▼                  ▼
  ┌─────────────┐   ┌─────────────┐   ┌───────────────────────┐
  │    Reads    │   │  Write path │   │      Bulk Load        │
  └──────┬──────┘   └──────┬──────┘   │  Airflow → S3         │
         │                 │          │  → StatefulSet Workers│
         │                 ▼          └──────────┬────────────┘
         │          ┌─────────────┐              │
         │          │    Kafka    │              │
         │          └──────┬──────┘              │
         │                 │                     │
         │                 ▼                     │
         │      ┌─────────────────────┐          │
         │      │  Replayer + Write   │          │
         │      │  Dispatcher         │          │
         │      └──────────┬──────────┘          │
         │                 │                     │
         └────────┬─────────┘                    │
                  │          ┌───────────────────┘
                  ▼          ▼
       ┌──────────────────────────────────────────┐
       │              NewSQL Backend              │
       │     range sharding · replicas · TTL      │
       └──────────────────────────────────────────┘

A Stateless Dispatcher as the Central Gateway

The most consequential architectural decision in v2 is the Dispatcher - a stateless, horizontally scalable Kubernetes service that sits between all clients and the storage backend. Every read and write goes through it.

Making the gateway stateless means you can scale it independently of the storage layer. Features like dual-write, shadow reads, rate limiting, and throttling all live in one place rather than being scattered across clients. And crucially, the team can control exactly where reads are served from - per table, dynamically - without changing a single line of code in any application that uses Mussel. That last capability turned out to be central to how the migration worked.

Kafka's Role - Old and New

One thing the v2 article doesn't make explicit: Kafka was already in v1. It wasn't a new addition. In v1, each storage node consumed from the Kafka partitions corresponding to its shards. This gave the system consistent write ordering across leaderless replicas without needing a leader node - nodes handling the same shard consume from the same partitions in the same order, so they all converge to the same state.

What changed in v2 is what Kafka is used for. In v1, it was primarily a replication mechanism between storage nodes. In v2, it's a durability and decoupling layer between the Dispatcher and the NewSQL backend. Writes land in Kafka first for durability, then a Replayer and Write Dispatcher apply them to the backend in order. This decoupling means the system can absorb write bursts without putting direct pressure on storage, and keeps writes safe even if the backend temporarily slows down. The tradeoff is that routing writes through Kafka adds some latency before they're visible.

Range-Based Sharding

Moving from hash-based to range-based partitioning is the foundational storage change in v2. With range sharding, each shard owns a known range of keys, which means tables can be pre-split before loading data into them. This distributes write pressure evenly across nodes and prevents hotspots during bulk ingestion - something hash partitioning gave you no control over. The tradeoff is that poorly chosen shard boundaries can still create hotspots. Airbnb's solution is to sample the source data before creating a table in v2, understand the key distribution, and pre-split accordingly. More work upfront, but it results in predictable, controllable data distribution.

TTL Without Relying on Compaction

Data expiration in v1 was handled by HRegion's compaction cycle. When HRegion's in-memory store fills up, it flushes to disk. Compaction periodically merges those files and cleans up expired records in the process. It works at moderate scale, but at petabyte scale the compaction process can't keep up - data accumulates faster than it gets cleaned.

v2 replaces this with a dedicated expiration service. It splits data namespaces into range-based subtasks and processes them concurrently across multiple workers, scanning and deleting expired records in parallel. Jobs are scheduled to avoid peak read traffic, and write-heavy tables get max-version enforcement with targeted deletes to prevent unbounded growth. The result is the same retention behavior as v1, but implemented as a purpose-built service rather than a side effect of storage compaction.

The Migration: Moving a Petabyte Without Downtime

Migrating data across storage engines while keeping production traffic running is where the interesting engineering happens. Airbnb's approach was methodical.

The Bootstrapping Challenge

The first obstacle: v1 didn't have table-level snapshots or Change Data Capture (CDC) streams. CDC is typically how you seed a new database during a migration - stream every change from the old system into the new one until they're in sync. Without it, Airbnb had to build a custom bootstrap pipeline for each table.

The process for each table: download v1 backup data, sample it to understand key distribution, create a pre-split v2 table based on that distribution, load the backup data into v2, verify via checksum, then replay any Kafka messages that accumulated during the load to catch up on writes that happened while bootstrapping was running.

For large tables, the bootstrap phase alone took hours or days. To handle interruptions gracefully, the team used Kubernetes StatefulSets - which provide persistent local state across pod restarts - to checkpoint progress periodically. If a job died partway through, it could resume from the last checkpoint rather than starting over.

The tradeoff is operational complexity - StatefulSets are harder to manage than stateless jobs. Each worker maintains local state that must be monitored, and failed pods need careful handling to avoid replaying already-completed work.

Four Stages, All Reversible

The migration moved through four stages:

Blue - all traffic on v1. The stable baseline while bootstrapping happens in the background.
Shadow - v1 continues serving traffic, but v2 processes reads and writes in parallel, silently. Responses from v2 are checked for correctness but never returned to callers. This lets the team verify v2's behavior without any risk to production.
Reverse - v2 takes over live traffic, with v1 kept on standby. Automatic circuit breakers monitor v2's error rate and replication lag; if either crosses a threshold, traffic snaps back to v1 instantly.
Cutover - v1 is decommissioned for that table.

Every stage could be reversed. And migration happened one table at a time, so each table could be at a different stage simultaneously. Lower-risk tables went first, building operational confidence before the team tackled high-criticality ones.


                         ┌──────────────┐
                         │   Clients    │
                         └──────┬───────┘
                                │
                                ▼
                     ┌────────────────────┐
                     │     Dispatcher     │
                     │--------------------│
                     │ • read routing     │
                     │ • dual-write ctrl  │
                     │ • stage config     │
                     └─────────┬──────────┘
                               │ writes
                               ▼
                        ┌───────────────┐
                        │     Kafka     │
                        │───────────────│
                        │ replayable    │
                        │ write log     │
                        └───────┬───────┘
                                │
                    ┌───────────┴───────────┐
                    │                       │
                    ▼                       ▼
            ┌───────────────┐       ┌───────────────┐
            │   Mussel v1   │       │   Mussel v2   │
            │  (HRegion)    │       │  (NewSQL)     │
            └───────────────┘       └───────────────┘
                    ▲                       ▲
                    └───────────┬───────────┘
                                │
                         read routing
                       (Dispatcher above
                        controls which
                        system responds)

* Each table is bootstrapped into v2 before entering Shadow stage.
┌──────────────────────────────────────────────────────────────────┐
│ Stage    │ Write Path              │ Read Path                      
├──────────┼─────────────────────────┼─────────────────────────────┤
│ BLUE     │ Kafka → v1              │ v1 serves                      
│          │                         │ v2 bootstrapping in backgroud 
├──────────┼─────────────────────────┼─────────────────────────────┤
│ SHADOW   │ Kafka → v1 + v2         │ v1 serves                      
│          │ (dual-write)            │ v2 responses checked,          
│          │                         │ not returned to callers        
├──────────┼─────────────────────────┼─────────────────────────────┤
│ REVERSE  │ Kafka → v1 + v2         │ v2 serves                      
│          │                         │ v1 on standby,                 
│          │                         │ circuit breaker active         
├──────────┼─────────────────────────┼─────────────────────────────┤
│ CUTOVER  │ Kafka → v2 only         │ v2 serves                      
│          │                         │ v1 decommissioned              
└──────────────────────────────────────────────────────────────────┘

Read Traffic Migration - The Underappreciated Half

Most migration stories focus on moving the data. Moving read traffic is a separate problem, and Airbnb's approach here is worth understanding on its own.

Once dual-write was established and data consistency confirmed, the Dispatcher handled read traffic migration in three phases. First, reads were served from v1 while shadow requests were simultaneously sent to v2 - responses compared but v2 never returned results to callers. Second, the roles flipped: v2 served reads while reverse shadow requests went to v1, providing a safety net to catch any instability. Third, full cutover with v1 decommissioned per table.

What makes read traffic migration more manageable than write migration is that reads are ephemeral. If a read returns a wrong result during shadowing, there's no lasting consequence - you catch it, investigate, and fix it. Writes are durable; if something goes wrong in a write path, you need a recovery story. Airbnb's pipeline handled these separately for this reason.

Running shadow reads effectively doubles the read load on both systems during the shadow and reverse phases. At an scale - where Mussel v1 already handles over 800k read QPS - this means the infrastructure must temporarily sustain roughly twice the total read traffic across both systems. That additional load is a non-trivial operational cost that must be accounted for during migration planning.

The Dispatcher made this fine-grained control possible. Because all read routing decisions lived in one stateless service, the team could change where any table's reads were served from - in real time, per table - without touching the applications consuming Mussel. Without a centralized routing layer, this would have required coordinating changes across every service using Mussel, which is a much harder problem across hundreds of tables.

Lessons Worth Taking Away

Consistency flexibility has a cost. v1 offered only eventual consistency. v2 introduced the option for stronger consistency guarantees per namespace. Airbnb had to build write deduplication, hotkey blocking, and lazy write repair to handle the edge cases that surfaced. If you're planning a similar migration, the storage layer change is usually the easier part. The surprises tend to show up in application behavior.

Presplitting isn't optional with range sharding. Migrating to a range-based backend without understanding your key distribution first will produce hotspots that look identical to the hash-partitioning hotspots you were trying to escape. Sampling the source data and pre-splitting tables accordingly isn't overhead - it's what makes the migration safe.

Architecture decisions today shape migration options tomorrow. Because Kafka was already the write path in v1, Airbnb could leverage the same stream for dual-write and catch-up replay during migration. This wasn't planned specifically for the migration - it was a property of v1's design that turned out to be the right foundation for v2. The decisions you make today directly affect how much flexibility you'll have when you need to change the system later.

What Mussel v2 Actually Delivers

At the end of their article, Airbnb makes a specific claim: Mussel v2 can simultaneously ingest tens of terabytes in bulk, sustain 100k+ streaming writes per second, and keep p99 reads under 25ms - all in the same cluster, with a single API for callers.

What makes this possible is that all three workloads share a common access contract: writes are eventually applied, reads are point or range lookups, and callers can tolerate a configurable staleness window. That shared contract is what lets one system serve all three patterns without them interfering with each other. Engineers no longer need to stitch together separate systems for each ingestion pattern and spend energy keeping them in sync - Mussel provides those guarantees out of the box.

What strengthens the claim most is that Mussel v2 has been running in production for a year at this scale. Engineering blog posts are full of capability claims. A year of production evidence at Airbnb's traffic levels is harder to dismiss.

What the Story is Really About

Pull back from the specific technology choices - the NewSQL backend, the Kubernetes manifests, the topology-aware TTL service - and the most transferable part of this story comes into focus.

Airbnb's ability to migrate a petabyte without downtime was shaped not just by what they built in v2, but by how v1 was designed years earlier. Kafka as the write path. Leaderless replication with topic-based ordering. Those decisions were made to solve different problems at the time. But the properties they created - a durable, replayable log at the center of every write, consistent ordering without a leader - turned out to be exactly what the v2 migration needed. The dual-write pipeline, the catch-up replay, the consistency guarantees during transition: all of it was more tractable because of how v1 worked.

Good infrastructure tends to have this quality. It solves the problem in front of you, but it also leaves options open for when the problem changes. The v1 engineers weren't building migration infrastructure. But the choices they made left those options available.

A useful question to ask about any storage system you own: not "can we migrate this?" but "if we needed to migrate this, what properties do we already have that would help?" Most teams only ask that question once a migration is already underway. Asking it earlier - and making deliberate architectural choices based on the answer - is what separates systems that are easy to evolve from systems that aren't.

References:[ Building a Next-Generation Key-Valu

CI/CD for Licensed Software You Don't Host

Fri, 10 Apr 2026 08:46:09 +0000

We are three weeks away from shipping NavEngine v4, an echo from the previous piece on Business Driven development.

I say "shipping" loosely. There is no deployment script I run, no SSH session I open and no Kubernetes rollout I watch. The software lives on infrastructure I have never seen, behind firewalls I cannot reach, on machines whose specs I do not know. Shipping, in this context, means pushing an image to a registry and trusting that a process running inside a container on a customer's server will eventually notice and do something about it.

That gap - between what I push and what the customer runs - is what this piece is about.

The Assumption Collapse

Every CI/CD tool I have ever used was built on a premise so foundational that nobody thought to state it: you control the deployment target. You own the server. You have the keys. Deployment, in the conventional sense, is just automation wrapped around access you already have.

NavEngine is quite the opposite. It exists as a custom image - qcow2 - shipped onto the customer's infrastructure. The customer owns the machine. I do not have SSH access unless I go through DWService and even then, that is a support channel, not a deployment one. Yet somehow, we need to continuously deliver updates to machines we cannot reach, across connections we cannot guarantee, without breaking software that is actively in use.

So the question became: if you cannot push, how do you deliver?

Indulge The moment you decide to ship software you don't host, you have made a decision with consequences that will follow you for the life of the product. Not just operationally. Architecturally. Every assumption your codebase makes about the environment it runs in now belongs to someone else's infrastructure. That is not a deployment problem. It is a design problem that shows up at deployment time.

The Answer is Pull

Watchtower. It runs as a container alongside the rest of the stack, polls the Docker registry on a configured interval, and when it detects a new image digest on the tag it is watching, it pulls and restarts the relevant containers. No webhook, no push, no SSH. The installation phones home for updates and takes what it finds.

The key design decision here was the floating tag. Every customer's Butane config ships with core:stable. Not core:v3.0.39. Not a digest pin. stable. When Watchtower polls the registry and sees that stable now resolves to a different digest than what is currently running, it pulls. What "stable" points to is entirely under my control, from the registry side, without touching anything on the customer's machine.

This sounds obvious once you say it. It took longer than I would like to admit to get there.

Two Registries, Two floating tags, One gate

Here is the full pipeline as it actually runs.

Every push to main triggers a build on the dev registry. The image gets tagged with a version identifier and a floating tag - staging. A staging environment - running the same Compose stack, same Butane configuration, same structure as a customer installation - pulls from staging. This is where the image lives until I am satisfied it works.

When staging looks good, I create a release. The production registry builds from that release, tags the image with the version (v3.0.40) and overwrites the floating stable tag. Customer installations, on their next Watchtower poll interval, see a new digest on stable and update.

The critical detail: stable is never overwritten by a push to main. Only by a release. The staging gate is the only thing standing between a broken image and a customer's running installation. There is no automated rollout percentage, no canary fleet, no gradual traffic shift. The gate is a human decision, made after watching staging run and deciding it is ready.

For a solo-operated product at this stage, that is the right call. Complexity in release infrastructure that you do not need is just surface area for things to go wrong.

Indulge A staging environment that does not accurately reflect production is a very expensive placebo. It gives you confidence without giving you information. The hardest thing about shipping self-hosted software is that your staging environment runs on infrastructure you understand, with data you created, on a network you control. Your customer's environment is none of those things. No pipeline fully closes that gap. The best you can do is know exactly where your confidence ends.

What Happens When stable is broken

It will happen. An image that passes staging will break in a customer environment for a reason that staging did not surface - a schema migration that assumed a clean database, a dependency that behaves differently on older hardware, a configuration value that was present in staging and absent in the field.

The recovery flow is: fix on main, watch it pass staging, cut a new release. v3.0.41 overwrites stable. Watchtower picks it up on the next poll interval. The customer, who may or may not have noticed anything, is now running the fixed image.

The window between the broken image landing and the fix arriving is real. Depending on how fast the hotfix moves through staging and how long the Watchtower poll interval is, a customer could be running broken software for anywhere from minutes to hours. There is no remote kill switch. There is no way to reach in and restart a service. There is DWService if the situation is bad enough to warrant it, but that is a support escalation, not a deployment tool.

This is the honest cost of not controlling the deployment target. You accept a recovery latency that you cannot compress below a certain floor. The mitigation is not a cleverer pipeline. It is investing deeply in staging fidelity and in making sure the image fails loudly rather than silently - health checks that surface problems immediately, startup validation that refuses to run on bad configuration rather than running badly.

A system that fails loudly is a system that can be fixed. A system that degrades quietly is a system that erodes trust before anyone knows there is a problem.

Enterprise and standard: different cadences, same pattern

How do we manage customers that diverge from the main product line with an enterprise license?

NavEngine has two license tiers. Enterprise customers are on a separate release cadence from standard customers. The mechanism is straightforward: separate floating tags on the production registry. core-enterprise:stable and core-standard:stable. The Butane config shipped to each customer points at the appropriate tag. Enterprise releases can go out on a different schedule, carry different feature sets, and move more cautiously than standard releases.

What prevents a standard customer from pointing Watchtower at the enterprise tag? Mostly friction. The Compose file is baked into the Butane config at provisioning time. There is no SSH access to change it. A customer would need console access and the motivation to go looking - unlikely for most, impossible to rule out for all.

The proper answer is registry-level access control: pull tokens scoped to the tags each customer is entitled to, issued at license activation and revoked at expiry. This means the registry enforces entitlement, not just the application. An expired license means an expired pull token means no updates, enforced at the point of delivery rather than after the fact.

This is on the roadmap. For v4, the answer is friction and trust.

Indulge License enforcement in self-hosted software is a negotiation between what you can technically control and what you have to trust. You cannot fully control what runs on a machine you do not own. At some point, a sufficiently motivated customer can circumvent almost any enforcement mechanism you build. The goal is not to make circumvention impossible. It is to make compliance easier than circumvention, and to make the value proposition strong enough that the question rarely comes up.

The license server is not in the update path

One decision I am glad we made early: the licensing server and the Docker registry are separate infrastructure. They do not share a failure domain.

Watchtower polls the registry. The license server is called from within one of the running containers as part of normal application operation. If the registry is unreachable, the software keeps running. If the license server is unreachable, the backend falls back to its last known state - persisted to disk, not held in memory, so it survives a container restart. The check runs periodically. The grace window is generous enough that a license server outage does not immediately affect customers, but not so generous that expired licenses can run indefinitely.

This matters because the failure modes compound. A product update that requires a license check to proceed has just made the license server a dependency of your deployment pipeline. Any outage that hits your license infrastructure also hits your ability to ship updates to paying customers. Keeping these paths separate means they fail independently, and independent failures are recoverable in ways that cascading ones are not.

What the pipeline actually looks like

The immutable version tags are not just for auditing. They are the rollback reference. If v3.0.40 breaks everything, v3.0.39 still exists in the registry. I can retag it as stable manually and customers will roll back on the next poll. This has not been needed yet. It is there for the day it is.

Indulge Most CI/CD writing treats deployment as the end of the story. Ship it, watch the metrics, move on. Self-hosted software inverts this. Deployment is the beginning of a period during which software you cannot reach is running in an environment you cannot see, on behalf of a customer whose experience you will only hear about if something goes wrong. The pipeline is not a delivery mechanism. It is a trust mechanism. Every decision in it is a decision about how much you trust the image before it leaves your hands.

Three Weeks Out

NavEngine v4 is three weeks away. The pipeline is running. Staging has held. The tags are in place.

None of that answers the only question that matters: what happens when the software leaves you?

It will run on machines you have never touched, against data you have never seen, in environments that do not care about your assumptions. By the time it fails, if it fails, it will already be someone else’s problem - and still entirely yours. The customer notices before you do. That is the thing. I am shit scared.

I suppose that this is the essence of these notes - to document real systems in real time. This is the inversion self-hosted systems force on you: treating deployment not as the end of control but as the beginning of its absence.

So you design for that absence.

You design for recovery over prevention.

For visibility over certainty and trust over control.

Everything else is just what it takes to make that possible.

Building a Production-Ready NDVI Pipeline: Normalization Fixes, WAF Resilience, and System Stability

Sun, 05 Apr 2026 11:21:57 +0000

From “It Works” to Production: Fixing NDVI Rendering, Surviving WAF Blocks, and Stabilizing a Failing Server

Most backend systems don’t fail because of obvious bugs.
They fail quietly—through misleading data, unstable infrastructure, and assumptions that don’t hold under load.

This week, I pushed an NDVI (Normalized Difference Vegetation Index) pipeline through a full transition:

from “it renders images” → to a production-grade, resilient system

This article breaks down what actually went wrong—and how each fix improved the system holistically.

1. The Illusion of Correctness: When “Green” Lies

The first issue looked harmless:
NDVI raster PNGs were rendering—but everything was green.

Root Cause

The normalization formula:

normalized = (ndvi + 1.0) / 2.0

This assumes NDVI values span [-1, 1].

But real-world farm data was tighter:

NDVI range: 0.42 → 0.63

After normalization:

0.71 → 0.81 → Entirely “green zone”

So the system was technically correct—but visually misleading.

2. The Fix: Dual Normalization Strategy

Instead of replacing one approach with another, I introduced two modes:

Histogram-Based Normalization (Default)

normalized = (ndvi - ndvi_min) / (ndvi_max - ndvi_min)

Expands contrast per image
Reveals actual variation
Best for analysis

Fixed Normalization (Legacy)

normalized = (ndvi + 1.0) / 2.0

Keeps consistency across images
Useful for comparisons

Why This Matters

This isn’t just a rendering fix—it’s a data integrity decision:

Never force a single interpretation when the data has multiple valid contexts.

3. External Systems Don’t Like You (STAC WAF Blocks)

The next issue wasn’t in my system—it was in how it behaved externally.

Symptom

Requests to Copernicus STAC API started failing
Response: empty HTML or “Request Rejected”

Classic WAF (Web Application Firewall) behavior.

Root Cause

The system behaved like a bot:

Rapid consecutive requests
Predictable timing patterns

The Fix: Controlled Request Behavior

sleep(base_interval + random.uniform(jitter_min, jitter_max))

Implemented:

Minimum delay between requests
Random jitter (1–5 seconds)
Graceful handling of invalid responses

Result

The system stopped being “blocked” and started being tolerated.

This is a key distributed systems lesson:

You don’t just integrate with APIs—you adapt to their defensive behavior.

4. Architecture Maturity: Preparing for Redis Streams

The NDVI pipeline originally used Celery for async jobs.

Instead of rewriting everything, I introduced a routing abstraction:

if settings.NDVI_QUEUE_BACKEND == "celery":
    dispatch_celery_job(...)
elif settings.NDVI_QUEUE_BACKEND == "stream":
    raise NotImplementedError

Why This Is Important

This small change enables:

Seamless migration to Redis Streams
Zero disruption to current workflows
Incremental architecture evolution

Principle

The best migrations don’t start with new systems—they start with abstractions.

5. The Real Problem: Server Crashes Every 10 Hours

This was the most critical issue.

Symptoms

System crashes after ~10 hours
No single obvious failure point

Root Cause

Unbounded resource usage across multiple services:

MySQL buffer pool growing unchecked
PHP-FPM workers accumulating memory
Redis using unlimited memory
Celery workers never restarting
Monitoring stack filling disk

6. The Fix: System-Wide Resource Governance

Instead of patching one component, I enforced limits everywhere:

Database

innodb_buffer_pool_size = 512M
max_connections = 50

PHP-FPM

pm.max_children = 15
pm.max_requests = 500

Redis

mem_limit: 512mb
maxmemory-policy: allkeys-lru

Celery

worker_max_tasks_per_child = 100
worker_max_memory_per_child = 512000  # KB

Observability Stack

Prometheus: 7-day retention
Loki: compaction + retention limits

Result

The system moved from eventual failure → controlled lifecycle stability

7. Observability and Documentation (The Often Ignored Layer)

Alongside code changes:

Added alerts for system health
Documented all NDVI configuration settings
Created deployment + rollback guides
Expanded test coverage

Why This Matters

Without this layer:

Fixes don’t scale
Bugs repeat
Systems become tribal knowledge

8. Security Mistake (Worth Mentioning)

During the process, a sudo password was exposed in plain text.

No system is “production-ready” if secrets are handled casually.

Immediate Takeaways

Never expose credentials in logs or chats
Rotate immediately if leaked
Move toward:
- environment-based configs
- secret management systems

Final Thoughts

This wasn’t about fixing a bug.

It was about addressing three deeper problems:

Data correctness (NDVI visualization)
System behavior under external constraints (WAF)
Infrastructure stability (resource limits)

The Real Transformation

Before:

“The system works.”

After:

“The system is predictable, resilient, and ready for scale.”

Key Lessons

Correct output is not always truthful output
External APIs require behavioral adaptation
Systems fail through unbounded growth, not sudden errors
Architecture evolves through abstraction, not rewrites
Stability is a system-wide property, not a single fix

If you’re building data pipelines or distributed systems, this pattern will repeat.

The earlier you design for it—the less painful it becomes.

Why Your System Fails on the Most Predictable Day of the Year

Fri, 03 Apr 2026 12:24:22 +0000

The Problem Hiding in Plain Sight

Most applications don't fail because of bad code. They fail because of bad architecture decisions made early that nobody questioned until it was too late.

Here's what actually breaks systems at scale:

Everything talking to everything (no clear boundaries)
The database doing work the application should do
Synchronous processing where async was needed
One giant service that owns too much responsibility
No separation between reads and writes under heavy load

None of these are framework problems. None of these are language problems. They are thinking problems.

To make this concrete, let's look at one example that millions of people have personally experienced and that engineers have failed to solve for decades.

The University Enrollment System

Think about course enrollment day at any university.

Every semester, thousands of students across every department flood the portal at the same time, each trying to register for their own offered courses. The load is not a surprise. The date is on the calendar. It happens every single semester like clockwork.

But the system was never designed for it.

Every request hits the same flow: check eligibility, check seat availability, write enrollment, update seat count, all synchronously, all at once. No queue. No cache. No separation between reads and writes. Just a database choking under the weight of an entirely predictable moment.

Students get timeout errors. Duplicate enrollments. Lost seats they were fully eligible for. Everyone refreshes in panic. And every semester, someone calls IT to "increase the server capacity" and nothing really changes.

The code isn't broken. The thinking is.

The 5 Architectural Problems Hiding Inside One Example

Most teams look at this scenario and see one problem: too much traffic. So they throw more servers at it. It helps a little, then fails again next semester.

The reality is there are 5 separate problems here, each requiring a different solution. Solving one without the others just moves the failure to a different place.

Problem 1: Performance, the Spike

Thousands of requests hitting the database simultaneously will bring any system to its knees regardless of how powerful the hardware is.

The fixes here are well understood but rarely implemented together:

A queue absorbs the spike so the database is not flooded all at once
A cache layer serves seat availability reads without hitting the database on every refresh
Separating reads from writes so checking availability and writing enrollment don't compete on the same database connection pool

This solves the crash. But it does not solve correctness. And that is where most teams stop.

Problem 2: Correctness, the Race Condition

Even after adding a queue, if multiple workers process requests in parallel, two students can still read "1 seat available" at the same time, both pass the eligibility check, and both get enrolled into the same last seat. The queue serialized the intake but not the processing.

This is a race condition, and a queue alone does not fix it.

The real solutions:

Pessimistic locking locks the seat row at the database level while one transaction is processing it, forcing all others to wait. It is safe and guarantees correctness but creates a bottleneck under extreme concurrent load and carries the risk of deadlocks if not implemented carefully.

Optimistic locking takes the opposite approach. Allow concurrent reads, but before writing the enrollment, check a version number on the seat record. If someone else already changed it, the transaction fails and must retry. Better for scale but generates retries under heavy conflict, which adds its own overhead.

Distributed locking with Redis places a temporary hold on a seat while it is being processed, similar to a reservation window on a ticket booking site, preventing any other request from touching that record at the same time. This is the approach production-grade booking systems actually use.

The right choice depends on your conflict rate, your scale, and your tolerance for complexity. But the wrong choice is to pick none of them and assume the queue is enough.

Problem 3: Idempotency, the Double Click

A student hits Enroll and the page is slow. They click again. Now two identical requests are in flight to the server.

Even if you have solved the race condition with locking, without idempotency both clicks can create two enrollment records for the same student in the same course. The lock protects the seat count. It does not protect against the same student submitting twice.

The fix is an idempotency key, a unique identifier generated by the client and attached to each enrollment request. When the server receives a request, it checks whether that key has already been processed. If yes, it returns the original result. If no, it processes it and stores the key. The second click gets the same response as the first without triggering a second enrollment.

This is not a database problem. It is an API design problem. And it is the one most teams never think about until they find duplicate records in production and cannot explain why.

Problem 4: Partial Failure, the Half-Enrolled Student

A successful enrollment is not one database write. It is a chain of operations: write the enrollment record, decrement the seat count, send a confirmation notification, update the academic record, generate a fee entry.

What happens if the system crashes after step 2 but before step 3?

The seat is taken. The student has no confirmation. The database says enrolled. The academic record disagrees. The student calls support. Support has no idea what state the system is in.

This is a partial failure and it is one of the hardest problems in distributed systems because it is invisible until a real person reports confusion.

Every multi-step operation needs either a fully atomic transaction that commits everything or rolls back completely, or a compensation strategy, a predefined plan for how to cleanly undo each completed step if something fails halfway. Designing for the happy path is easy. Designing for the middle of failure is architecture.

Problem 5: The Stale Cache Trap

Earlier we said to add a cache layer to serve seat availability reads. That is still correct. But it introduces a new failure mode that needs to be designed for explicitly.

The cache says 5 seats available. The database already has 0 after recent enrollments that have not yet invalidated the cache. Students attempt to enroll based on stale data, hit the lock, fail, and receive a confusing error that says the course is full, even though the portal showed availability 3 seconds ago.

The cache improved performance but silently introduced a trust problem. Students lose confidence in the system. Support tickets spike. And the engineering team is confused because the system is technically working correctly, the cache just has not caught up.

Cache invalidation strategy needs to be designed upfront: how quickly must the cache reflect reality, what triggers an invalidation, and what happens to users who are mid-flow when availability changes. These are not performance questions. They are product and architecture questions.

What This Means for How You Build

Five problems. Five different failure modes. Five different solutions. All hiding inside one scenario that happens on a date printed on the academic calendar months in advance.

This is why scaling is not about upgrading your server. The server was never the problem.

The engineers who scale systems well ask different questions from the start:

Where are my bottlenecks under 10x load?
What happens if this one service goes down?
Am I coupling things that should be independent?
Is my data model fighting my query patterns?
What happens in the middle of a failure, not just at the end?
Can the same request safely arrive twice?

None of these questions have anything to do with which framework you picked or which cloud provider you use. They are design questions. Thinking questions.

The engineers who ask them before traffic arrives are the ones who sleep well on enrollment day.

Final Thought

Architecture is not about knowing the right tools. It is about asking the right questions early enough that the answers still matter.

The skill that transfers everywhere, across languages, frameworks, companies, and industries, is the ability to look at a system and ask: what am I not designing for yet?

Everything else is just syntax.

When Redis Lies: Debugging a Schema That Wasn’t There

Thu, 26 Mar 2026 19:20:01 +0000

Nextcloud ↔ Django (DRF) ↔ Redis-backed distributed system

Introduction

There’s a particular class of bugs that doesn’t live in your code, your database, or your API.

It lives in the space between them.

Today’s issue looked simple on the surface:

“Unable to parse farm schema response.”

But the root cause wasn’t in the UI, nor in the API.

It was in the cache.

The System Setup

The architecture is fairly typical for a modern distributed backend:

Nextcloud App (UI + PHP backend)
Django REST Framework (DRF) API (/api/v1/...)
Redis (used for caching + token storage)
NDVI service feeding farm-state data

Redis had recently replaced APCu as the caching backend to support shared worker state and token reuse.

The Symptom

The Nextcloud admin UI failed to load farm schema with:

“Unable to parse farm schema response”

This error implied one thing:

The frontend expected valid JSON
It received something else

The Misleading Signals

Initial checks showed everything was “correct”:

✅ Schema endpoint URL was valid
/apps/weather_apis/api/v1/admin/farms/schema
✅ Route existed and matched
occ router:match confirmed it
✅ Redis was healthy
- redis-cli ping → OK
- PHP connection as www-data → OK
✅ Backend logic was intact

Yet the UI still failed.

The Trap

At this point, most debugging paths would focus on:

fixing the controller
adjusting serializers
checking permissions

But none of those were the problem.

The Reality

The issue was cache-induced inconsistency:

Redis had stored a stale response
The backend schema had evolved
The cached payload no longer matched the expected structure

So the system behaved like this:

Layer	State
Route	Correct
Backend	Correct
Cache	Outdated
UI	Broken

The UI wasn’t wrong.

The backend wasn’t wrong.

The cache was lying.

The Fix

The resolution was simple:

Clear Redis cache keys
Reload the UI

Immediately:

Schema endpoint returned valid JSON
UI parsing succeeded
Farms admin interface recovered

The Deeper Insight

Caches don’t just store data — they store assumptions.

When your system changes:

API shapes evolve
serializers update
authentication rules shift

But your cache:

continues serving old truths

Why This Matters

This issue highlights a key principle in distributed systems:

A system can be locally correct but globally inconsistent

Every component was working as designed.

But together, they produced failure.

Extending the System: Authentication Alignment

Alongside the fix, the farm state endpoint was improved:

Introduced Redis-backed authentication
Unified with existing observation pattern:
- JWT support
- API key authentication
- Integration tokens
Added:
- scope enforcement
- allowlisting
- OpenAPI documentation updates

Tests were expanded to cover:

API key access
integration token flows
proper rejection behavior (403 for unauthenticated)

Result:

7/7 tests passing
Consistent and secure API surface

Practical Takeaways

1. Always Suspect Cache

If you see:

incorrect response shapes
parsing errors
inconsistent behavior across layers

Check the cache first.

2. Introduce Cache Versioning

Avoid collisions between old and new schemas:

farms:schema:v1
farms:schema:v2

3. Use TTLs for Dynamic Data

Schema and metadata endpoints should not live forever in cache.

4. Add Observability

Track:

cache HIT / MISS
response origin (cache vs source)

5. Design Defensive APIs

Ensure endpoints:

always return valid JSON
even in failure scenarios

Final Thought

This wasn’t a UI bug.
It wasn’t an API bug.
It wasn’t even a Redis bug.

It was a system truth mismatch.

And those are the bugs that matter most—because solving them means you’re no longer just writing code.

You’re understanding the system.

“In distributed systems, the hardest bugs aren’t where things break.
They’re where things appear to work.”

Escaping Cache Fragmentation: How Misconfigured PHP Workers Flooded My Token System

Sun, 22 Mar 2026 18:02:00 +0000

The Symptom

I started noticing something strange in my observability stack:

Integration tokens were being minted repeatedly
My token endpoint showed activity even when no user interaction was happening
Metrics suggested constant “traffic” to an otherwise idle system

At first glance, it looked like:

A security issue
A rogue client
Or a broken API consumer

It was none of those.

The Root Cause

The issue came down to a subtle but critical architectural mistake:

I was using a non-shared cache in a multi-worker environment.

Stack involved:

PHP-FPM (2 workers)
APCu (in-memory cache)
Token-based integration between services

⚙️ What Went Wrong

APCu is process-local, not shared.

That means:

Worker A cache ≠ Worker B cache

Each PHP-FPM worker had its own isolated memory.

The Cascade Effect

My token logic was straightforward:

if token not in cache:
    mint_new_token()

But in reality, the system behaved like this:

Request hits Worker A → token exists → OK
Next request hits Worker B → cache miss → mint new token
Repeat across workers → continuous token regeneration

Why Observability Looked “Wrong”

From the outside, it looked like traffic was hitting the token endpoint.

But in reality:

The system was generating its own traffic due to cache inconsistency.

This is a key lesson:

Not all traffic is external
Some is emergent behavior from system design

✅ The Fix

I switched from APCu to:

Redis (shared cache)

Now:

All workers → same cache → consistent token state

Result:

Tokens minted once
Reused across all workers
Metrics stabilized instantly

Production Hardening (What I Added Next)

Fixing the cache wasn’t enough — I hardened the system further.

1. Distributed Locking

To prevent race conditions:

if token exists:
    return token

acquire lock
    re-check cache
    mint token if still missing
release lock

2. TTL Buffering

Avoid edge expiration issues:

cache_ttl = token_expiry - safety_margin

3. Observability Metrics

I added:

token_cache_hits
token_cache_misses
token_mint_count

Now anomalies show up immediately.

Key Takeaway

This wasn’t just a bug.

It was a distributed systems failure mode:

Cache locality + multi-worker architecture → inconsistent state → emergent traffic

⚡ Final Insight

If your system:

Runs multiple workers
Uses in-memory caching
Relies on shared state

Then this rule applies:

If your cache isn’t shared, your state isn’t real.

Closing

This issue reinforced something critical in my engineering journey:

You don’t debug systems by staring at code —
you debug them by understanding how state flows across boundaries.

If you're building distributed APIs, token systems, or high-concurrency services —
this is one edge case worth designing for early.

From 80-Second APIs to Sub-Second: Rebuilding a Geospatial Backend with Async Pipelines

Sun, 22 Mar 2026 11:15:19 +0000

From 80-Second APIs to Sub-Second: Fixing Latency with Async Pipelines (Django + Celery)

Introduction

At some point, every backend engineer hits this wall:

The API works perfectly… until it doesn’t.

I hit that wall with a farm analytics endpoint computing NDVI (Normalized Difference Vegetation Index) from satellite imagery. The system was correct, the logic was sound, and the results were accurate.

But the numbers told a different story:

P95 latency: 1.25 minutes

That’s not an API. That’s a blocking compute job pretending to be one.

This is the story of how I redesigned the system—from a synchronous request-driven model to an asynchronous data pipeline—and brought latency down to sub-second performance (P95 ≈ 725ms).

The Original Architecture (The Hidden Problem)

At first glance, the system looked clean:

[Client]
   ↓
[Django API]
   ↓
[STAC API → Satellite Data]
   ↓
[Raster Processing (NDVI)]
   ↓
[Response]

What happened on each request?

Query satellite imagery via STAC
Fetch raster bands (Red & NIR) from remote storage
Process NDVI using rasterio
Aggregate coverage
Return result

Why this seemed fine

It worked locally
It returned correct data
It followed a “pure API” mindset

But under the hood:

Remote I/O (S3-backed satellite data)
Heavy raster decoding (JPEG2000)
Sequential band reads
Full computation per request

The Breaking Point

Logs told the truth.

Each request looked like:

STAC request → ~5s
Raster read (B04) → ~5–10s
Raster read (B08) → ~5–10s
Processing → ~5s+
Total → ~80+ seconds

And the key realization:

I wasn’t building an API—I was executing a geospatial compute pipeline on every request.

The Core Insight

This is the shift that changes everything:

APIs should serve data, not compute it on demand.

The problem wasn’t Python.
The problem wasn’t Django.
The problem was architecture.

The New Architecture (Async Pipeline)

I redesigned the system around asynchronous computation + caching:

             (Scheduled / Triggered)
                    ↓
             [Celery Worker]
                    ↓
         [NDVI Computation Pipeline]
                    ↓
             [Redis / Database]
                    ↓
[Client] → [Django API] → [Cache Lookup]

Key changes

NDVI computation moved out of the request path
Results cached in Redis
Background jobs compute and refresh data
API returns instantly (no heavy compute)

Diagram 1 — Before vs After

Before (Request-driven)

Request
   ↓
STAC API
   ↓
Raster I/O
   ↓
NDVI Compute
   ↓
Response (80s)

After (Pipeline-driven)

Request → Cache → Response (~725ms P95)
              ↓ (miss)
         Async Task
              ↓
       Compute + Store

Implementation

1. Fast API Path (Non-blocking)

from django.core.cache import cache
from ndvi.tasks import compute_farm_state_coverage

def get_farm_state(farm_id: int) -> dict:
    cache_key = f"farm_state:{farm_id}"

    data = cache.get(cache_key)
    if data:
        return data

    compute_farm_state_coverage.delay(farm_id=farm_id)

    return {
        "coverage_pct": None,
        "status": "processing"
    }

2. Celery Task (Async Compute)

from celery import shared_task
from django.core.cache import cache

@shared_task(bind=True, autoretry_for=(Exception,), retry_backoff=True)
def compute_farm_state_coverage(self, farm_id: int) -> None:
    coverage = compute_ndvi_coverage(farm_id)

    cache.set(
        f"farm_state:{farm_id}",
        {
            "coverage_pct": coverage,
            "status": "ready"
        },
        timeout=60 * 60 * 6,
    )

3. Daily Backfill (Critical)

from celery import shared_task

@shared_task
def enqueue_daily_farm_state_coverage():
    farm_ids = get_active_farm_ids()

    for farm_id in farm_ids:
        compute_farm_state_coverage.delay(farm_id=farm_id)

Observability (The Real Upgrade)

Metrics added:

Task duration
Task success/failure
Queue depth

Metrics (Grafana Observations)

Grafana Screenshots

1. Latency Graph

Before

P95 latency: ~1.25 minutes

After

API latency: ~725ms (P95)
Background tasks: 60–90s

Before vs After Summary

Metric	Before	After
API latency	1.25 min	~725 ms (P95)
System type	Request-driven	Pipeline-driven
Scalability	Poor	Strong
Observability	Minimal	Improved

Final Thought

I stopped treating my API like a calculator and started treating my system like a data pipeline.

That’s when everything changed.

Decision Stack

Wed, 18 Mar 2026 06:29:23 +0000

✖ ✖ ✖ ✖ ✖ ✖ ✖

topic: decision_stack

Lately I’ve been following Cristiano Messina’s “Decision Stack” series.

A very interesting and professional perspective
on how modern systems should be designed.

The model is simple but powerful:

signals → context → objectives → constraints → execution → feedback

Where:

signals detect change
context defines meaning
objectives define what matters
constraints define boundaries
execution performs actions
feedback drives learning

✖

A question to practitioners:

Which of these layers
are already explicitly implemented
in your systems?

And in what form?

Are they:

centralized
scattered
implicit
formalized

✖

Curious how others structure this today.

✖ ✖ ✖ ✖ ✖ ✖ ✖

Modern Enterprise HR Is a Distributed System

Fri, 06 Feb 2026 08:59:50 +0000

A Systems Thinking Perspective

Most people see HR as administration, policies, forms, and approvals.

But inside large organizations, HR operations behave much more like distributed systems than administrative departments.

And if you don’t approach them architecturally, inefficiency becomes structural.

Enterprise Processes Are Systems

In software engineering, we think in terms of:

Inputs
Logic
Dependencies
Outputs
Failure points
Large-scale enterprise operations follow the same pattern.

An employee lifecycle workflow may appear simple: Onboarding → Payroll → Benefits → Exit

But under the surface, it involves:

Conditional logic
Policy variations
Data validation layers
Approval chains
Historical record dependencies

The complexity isn’t visible — but it’s real.

Where Enterprise Operations Usually Struggle

Across industries, common patterns emerge:

Hidden Dependencies
Processes rely on assumptions that aren’t formally documented.
Version Drift
Policies evolve, but legacy data doesn’t always align with updated logic.
Manual Overrides
Temporary fixes become permanent workflow fragments.
Data Fragmentation
Different business units may interpret similar rules differently.

These are not HR problems, They’re architecture problems.

The Systems Thinking Shift

When you stop treating enterprise operations as tasks and start viewing them as systems, optimization changes.

Instead of asking:

“How do we process this faster?”

You begin asking:

Where is the structural bottleneck?
What logic is implicit rather than explicit?
Which step creates cascading impact downstream?
Is automation accelerating inefficiency?

That shift transforms operational thinking.

Automation Is Not Optimization

Many organizations rush to digitize workflows; Digitizing a flawed structure doesn’t fix it, but scales it.

Optimization starts with:

End-to-end process mapping
Clear ownership definitions
Policy logic documentation
Dependency isolation
Data integrity validation
Only then does automation create value.

Why This Matters for Engineers

Software engineers are trained to think structurally.

Enterprise environments increasingly require that mindset — even outside traditional technical roles.

Whether the domain is:

HR
Finance
Supply chain
Sustainability

The core principle remains:
Systems thinking reduces complexity before technology reduces effort.

Final Thought

Enterprise efficiency is rarely limited by tools, it’s limited by architecture. And architecture exists far beyond code.

Connect with me
Via LinkedIn: https://www.linkedin.com/in/fadydesokysaeedabdelaziz
Via GitHub: https://github.com/fadydesoky

Building Research Infrastructure at Scale: Inside Globus's Hybrid SaaS Architecture

Mon, 26 Jan 2026 21:26:10 +0000

Research computing faces a unique architectural challenge: how do you build a platform that works across institutional boundaries, handles petabyte-scale data movement, supports diverse storage systems, and maintains fine-grained security controls, all without requiring users to be infrastructure experts?

Globus, a non-profit service from the University of Chicago, has spent nearly 30 years solving this problem. What started as the Globus Toolkit for grid computing has evolved into a comprehensive platform-as-a-service that now handles 2 petabytes of data transfer daily across 2,600+ institutions in 80+ countries.

The Hybrid Architecture

The key architectural insight behind Globus is its hybrid model. "We can't do that because all the institutions own their own resources," explains Rachana Ananthakrishnan, Executive Director of Globus. "We can't sort of take over those resources. We have to figure out how to add valuable services as a layer to that resource."

This means Globus doesn't centralize data—instead, it orchestrates movement between endpoints while data flows directly over high-speed research networks (often 100-400 Gbps). The architecture consists of three main components:

Local Agents run at institutional sites. The Globus Connect agent handles data operations with plugins for different storage types (POSIX, object stores, tape archives). The Globus Compute agent manages remote function execution on various schedulers (Slurm, PBS, Kubernetes). Action Providers allow integration with custom APIs.

Global Management Services runs as a multi-tenant SaaS. These services handle orchestration, heuristic-based optimization, reliability guarantees, and maintain the security fabric. But crucially, they maintain only control-plane connections—file data moves directly between endpoints.

Security Fabric uses OAuth and OIDC to federate authentication across 2,140+ identity providers. This allows researchers to use institutional credentials without creating new accounts, while maintaining fine-grained authorization across distributed resources.

Platform Capabilities

Globus offers six core capabilities that can be used independently or composed:

Managed Transfer provides fire-and-forget data movement with automatic retry, checksum verification, and progress tracking. "The protocol here continuously sends what are called markers," Ananthakrishnan explains. "So it says I've moved 5 bytes, 10 bytes, 20 bytes, and suddenly there's no marker. Then this keeps retrying with a backoff algorithm."

Data Sharing creates permission overlays on existing storage without staging data. Share with any identity, set time restrictions, and maintain audit logs, all without creating local accounts for collaborators.

Unified Access abstracts heterogeneous storage through an open connector ecosystem. The Community Connector Program allows vendors and users to build plugins for any storage system, while Globus handles authentication, translation, and rate limiting.

Search provides schema-agnostic metadata indexing with dynamic schema detection and fine-grained visibility controls. Think Elasticsearch optimized for research with built-in access control.

Compute brings function-as-a-service to on-prem HPC. Submit Python functions that run on remote systems with federated authentication. One COVID-era drug screening project used this to distribute ML workloads across diverse resources, completing training cycles in 31 seconds, compared with much longer traditional approaches.

Flows offers declarative workflow automation using JSON-based definitions. Event-driven execution with AWS Step Functions under the hood, but extended to securely call outside services and remote systems.

Real-World Performance

The Advanced Photon Source at Argonne National Laboratory demonstrates the platform's capabilities at scale. Multiple beamlines use Globus to automate data movement and processing. "These data services have taken the time to solve a structure from weeks to days and now to hours," notes beamline scientist Darren Sherrell.

For developers, Globus provides comprehensive REST APIs and SDKs. The platform is extensible—you're not locked into the web interface. The Franklin & Marshall College biology department even built custom flows that allow field volunteers to click a button, fill a questionnaire, and trigger automated processing. "It really increased the community participation," Ananthakrishnan notes. "In place of teaching all his volunteers how to do these 10 things, he defined a flow."

Compliance and Security

For protected data, Globus implements NIST 800-53 and 800-171 controls, signs BAAs for HIPAA, and complies with GDPR and data protection agreements. The architecture supports both personal data (visible only to the user) and shared data (visible across institutions) with appropriate access controls.

The security model is shared: Globus controls the platform layer, while institutions control their endpoints. "It's a hybrid SaaS, it's a shared security model," Ananthakrishnan explains. "We do some controls, they have to do some controls."

Getting Started

Globus offers a freemium model, basic features are free for non-profit research. Developers can start with the web interface or dive directly into the APIs. Documentation is available at docs.globus.org, and the platform includes a 90-day trial for qualified projects.

For architects building research platforms, Globus provides a reference architecture for handling distributed data at scale. The hybrid model, federated authentication, and connector ecosystem offer lessons applicable beyond scientific computing to any scenario requiring secure data movement across organizational boundaries.

Why AI Startups Need to Focus on Distribution Before Disruption

Sun, 21 Dec 2025 03:05:35 +0000

There’s a familiar story playing out across the AI startup world.

A founder builds something technically impressive.
The model works.
The demo looks sharp.
Early users are intrigued.

And then… nothing scales.

Not because the product isn’t good. But because disruption without distribution doesn’t go anywhere.

This is one of the most consistent mistakes I see AI startups make today.

The Myth: “If the Tech Is Good Enough, Adoption Will Follow”

This belief has deep roots in startup culture.

Build something meaningfully better.
Ship it.
Let the market discover it.

That logic worked in earlier software cycles, when:

competition was limited
attention was less fragmented
switching costs were lower
novelty carried more weight

In AI, that world no longer exists.

Today, technical capability is abundant.
Attention is scarce.

And distribution, not disruption, is the real bottleneck.

Why AI Products Are Especially Vulnerable Without Distribution

AI products face a unique challenge:

They often look similar from the outside.

To a user:

multiple tools claim “AI-powered”
outputs feel interchangeable
differentiation is hard to see quickly

That means adoption is driven less by technical depth and more by:

where people discover the product
how often they encounter it
how easily it fits into existing habits
how much they trust the source

Without intentional distribution, even strong AI products remain invisible.

Disruption Is a Product Story. Distribution Is a Behaviour Story.

Disruption answers:

What’s new?
What’s better?
What’s different?

Distribution answers:

Where do people already spend time?
How do they make decisions?
Who do they trust?
What triggers action?

Most AI startups obsess over the first set of questions and ignore the second.

That imbalance is fatal.

Because products don’t spread through features. They spread through behavioural alignment.

What Distribution Actually Means in the AI Era

Many founders hear “distribution” and immediately think:

ads
growth hacks
viral loops

That’s surface-level thinking.

Real distribution is about being present at the moment of need.

For AI startups, that often means:

embedding into existing workflows
integrating with tools people already use
educating the market consistently
becoming a trusted voice, not just a tool
showing up before the buying decision is made

In other words, distribution is not a channel. It’s a system.

Why Waiting to “Figure Out Distribution Later” Rarely Works

I often hear:
“Let’s build first. We’ll worry about distribution once it’s ready.”

In AI, that’s backwards.

Because:

users don’t know what they need yet
workflows are still forming
mental models are still unstable

The startups that win are the ones that:

shape the conversation early
educate their audience
define the problem before selling the solution

Distribution is not a post-launch activity. It’s part of product design.

The Quiet Advantage of Distribution-First Teams

Teams that lead with distribution tend to:

build clearer products
simplify messaging
focus on real use cases
receive better feedback
iterate in the right direction

Why?

Because when you’re close to users early, you don’t build in isolation.

You don’t guess what matters. You observe it.

This feedback loop often matters more than another model upgrade.

Disruption Without Distribution Creates Fragile Companies

AI startups that rely only on technical disruption often end up:

chasing constant feature upgrades
reacting to competitor releases
struggling with retention
burning trust through over-promising

In contrast, distribution-led companies build:

familiarity
trust
mindshare
long-term relevance

They don’t need to scream “disruption.” They become the default choice quietly.

What This Means for AI Founders Right Now

If you’re building in AI today, the strategic order matters.

Not:
Disrupt → Build → Distribute

But:
Distribute → Learn → Build → Deepen

This doesn’t mean lowering technical ambition. It means aligning it with reality.

The strongest AI companies of the next decade won’t be the ones with the most impressive demos.

They’ll be the ones that:

understood their audience early
earned attention before asking for adoption
built trust before asking for behaviour change

The Real Takeaway

Disruption gets headlines.
Distribution builds companies.

In the AI era, where intelligence is becoming cheaper and faster to access, attention, trust, and habit formation are the real moats.

If no one knows you exist or understands why you matter, your disruption doesn’t matter either.

Focus on distribution first.
The disruption will actually land.

Next Article:

“The Real Reason Most Developers Are Misusing Generative AI.”

Oracle migrations reveal hidden technical debt. Here's what developers need to know before moving.

Mon, 13 Oct 2025 21:25:28 +0000

The Real Cost of Oracle Technical Debt: What Developers Learn During Migration

Oracle databases have been around for 40 years. That's a lot of time to accumulate technical debt.

Spencer Kimball, CEO of Cockroach Labs, spends his days helping Fortune 100 companies migrate off Oracle. The technical challenges he sees are consistent: undocumented stored procedures, excessive use of legacy patterns, and architectural decisions made when networking was slow and security wasn't a priority.

"The people that wrote them are gone from our company," Kimball says of the legacy applications his customers want to migrate. "We want help solving this."

PostgreSQL is Eating the World

The first thing to understand about modern database migrations is that developers don't want Oracle's dialect anymore. They want PostgreSQL.

"Postgres is actively eating the world," Kimball says. When Cockroach Labs started 11 years ago, PostgreSQL wasn't the default choice. Now it is.

This shift shows up everywhere. AWS Aurora moved its center of gravity from MySQL to PostgreSQL. Even TiDB, which started as a MySQL engine, recently added PostgreSQL support. IBM's customers are asking for "a Postgres database for the enterprise."

The reason is straightforward: PostgreSQL is the standard for new development. If you're building something new, you're building it on Postgres.

Distributed vs Monolithic: Where Developers Get Surprised

The migration from Oracle to a distributed database like CockroachDB involves more than syntax changes. It requires a shift in thinking.

"Databases have grown up over the last 50 years with this perspective that the database is a tightly coupled monolithic system," Kimball explains. Oracle was built for InfiniBand networks and specialized hardware. Those requirements don't match cloud reality.

The most common surprise? Auto-incrementing IDs.

In a monolithic database, generating sequential IDs is fast because everything runs on one machine. In a distributed system with consensus requirements, that pattern creates bottlenecks.

The solution is UUID generation with node IDs embedded. These IDs are unique across the cluster, sort correctly, and create zero hot spots. It's a small change, but every developer on a migration team needs to understand it.

"Some of this is solved just by retraining developers," Kimball says. "Some of it is done by tooling that will look through a schema and say, 'Here's an auto-incrementing ID. We recommend you replace it with UUID generation.'"

AI is Unraveling Oracle Lock-In

Stored procedures represent Oracle's most effective lock-in mechanism. They were a reasonable architectural choice when network latency was high and database security was limited. DBAs pushed application logic into the database where it could be controlled and protected.

The result is mountains of undocumented code. Sometimes it's PL/SQL. Sometimes it's C routines with no unit tests and no documentation.

AI tools are changing the migration economics. Kimball says AI can analyze legacy stored procedures, generate documentation, create unit tests, and suggest complete migrations to PL/pgSQL (PostgreSQL's procedural language).

"The sharpest decrease in costs is going to come from AI," he says. Where migrations used to require 100+ consultants over three years at a cost of $10 million, AI-assisted approaches can reduce that significantly.

The tools from Cognition, Devin, and even Claude can handle much of the heavy lifting that previously required specialized Oracle knowledge.

Architecture Built for AI Scale

The architectural differences between distributed and monolithic databases matter more as AI workloads increase.

"You can't even get there without a distributed architecture," Kimball says about handling AI-scale traffic. The issue isn't just throughput. It's cost.

Cockroach Labs routinely runs 300-node test clusters for weeks at a time. These clusters contain petabytes of data. Three weeks used to cost $500,000.

Now they run these tests on spot instances and save 80%. "Could you run a traditional database across spot instances?" Kimball asks. "You'd be down most of the time."

A distributed database built for resilience can handle spot instances getting preempted. It replicates, rebalances, and keeps running through the chaos.

The company is targeting a 10x cost improvement through a combination of optimizations: spot instances, multi-tenancy (packing multiple workloads on the same physical database), layered storage (S3, locally attached SSDs, EBS), and continuous performance optimization.

They've already achieved 2x performance improvement on the same hardware this year.

Resilience Under Pressure

Performance benchmarks under ideal conditions don't tell you much about production reality. Cockroach Labs released a "Performance under Adversity" benchmark that measures how databases behave during chaos: pod failures, network partitions, disk stalls.

They tested CockroachDB against Oracle GDD 23ai (the first version of Oracle that can support GDD in a Raft configuration). The results are documented in detail, and the benchmark methodology is open so anyone can reproduce it.

CockroachDB encountered 56 errors resulting in 51 seconds of downtime across all chaos tests. Oracle GDD encountered 300 errors resulting in 369 seconds of downtime (over six minutes).

That's 5x more resilient.

The difference matters most during "grey failures"—those problems that don't immediately take the database offline but slowly degrade latency and breach SLAs.

The Migration Path Forward

If you're stuck on Oracle and planning a migration, here's what to focus on:

Audit your stored procedures. Use AI tools to document what you have before you start planning the migration. You probably have more technical debt than you think.

Understand distributed architecture patterns. Auto-incrementing IDs are just one example. Get your team trained on how distributed consensus works and where the architectural differences matter.

Test for resilience, not just performance. Peak performance under ideal conditions matters less than consistent performance under realistic failure scenarios.

Factor in the full cost. Don't just compare license fees. Include the cost of downtime, the complexity of running the database, and the total cost of ownership over time.

The cloud forced many companies to reconsider Oracle. AI is accelerating that shift. If you're going to modernize anyway—and you probably are—make sure you understand the technical implications before you start.

New York startup replaces three media tools with one platform using FUSE, S3, and custom AI models.

Mon, 13 Oct 2025 13:20:57 +0000

How Shade Built a File System That Actually Works for Remote Creative Teams

Media production teams have a storage problem. They use Lucid Link for file streaming, Frame.io for review and approval, and Iconik for asset management. That's three subscriptions, three login screens, and constant file transfers between systems.

Shade thinks this is ridiculous. The New York-based startup built a single platform that handles ingestion, storage, review, and delivery. After raising $5 million from General Catalyst and Signal Fire, they now serve 94 customers and project $10 million in revenue next year.

I recently met with Shade during the 64th IT Press Tour.

The Technical Stack

Shade's architecture sits on top of S3-compatible storage. Customers can bring their own Wasabi, AWS, or Google Cloud Storage buckets, or use Shade's managed storage. Either way, the raw files stay in object storage while Shade's metadata service runs separately on AWS.

The desktop application uses FUSE (Filesystem in Userspace) on Mac and Linux, with similar technology on Windows. When you mount a Shade workspace, it appears as a local drive. Behind the scenes, the FUSE layer intercepts file system calls and translates them into S3 requests.

Brandon Fan, Shade's co-founder and CEO, explained the approach: "We have a CDN infrastructure in front of S3 that speeds up access to various locations. We also have local upload capability—it uploads to the nearest server, then syncs to the origin server."

This architecture supports parallel chunk-based uploads and downloads. The result is fast enough that teams can stream 4K video without downloading entire files first.

Custom AI for Massive Scale

The AI components required more custom work than expected. Shade uses OpenAI's vision models for metadata generation—things like identifying shot types, lighting conditions, and visible logos. But the search engine is completely custom.

"We had to build our own semantic search," Fan said. "Most traditional models are too slow for millions of assets. We took open-source vision models and distilled them to run on CPU infrastructure instead of GPUs. This keeps costs down while letting users search through hundreds of thousands of assets."

The search works across images, videos, and documents. Type "people playing cards in a park" and it finds that exact scene. The system also handles transcription (via Assembly AI) and facial recognition.

For clustering and organizing large asset libraries, Shade built another custom algorithm. Standard clustering falls apart at scale, so they engineered a solution that handles the volume their customers generate.

File Locking and Conflicts

Creative applications like Premiere Pro have built-in project locking. For everything else, Shade creates conflict files when two users modify the same asset. If you rename a folder while someone saves a file into it, Shade preserves both actions and flags the conflict.

The system maintains a multi-tenant database that stores all metadata, permissions, and file relationships. When a client requests a file, it queries this service to get the S3 URL, then streams the data from object storage.

Integration Points

Shade offers webhook-based automation through a Zapier-style interface. Teams connect Shade to their project management tools, CRMs, or custom applications. The API exposes all metadata and file operations.

The platform also supports SSO via SAML, SCIM provisioning, and audit logging. Permissions work at three levels: workspace, individual users, and link-based sharing. You can share files with external collaborators through password-protected links that expire on a schedule.

Developer Background

Fan comes from an information retrieval research background, publishing papers on search technology before the transformer era. His co-founder Emerson, the CTO, is a roboticist who built the distributed file system infrastructure.

The team includes engineers from Lucid Link and Frame.io—people who know the competitive landscape and customer pain points firsthand. Most of the 15-person team works from the New York office to maintain tight collaboration between engineering and product.

Market Position

Frame.io sold to Adobe for $1.2 billion in 2021. Iconik raised $80 million. Shade raised $5 million and competes directly with both.

The cost advantage matters. A team using 10 seats and 10TB on Frame.io plus another 10TB on Lucid Link pays significantly more than the equivalent Shade setup. Fan claims Shade typically runs 30% cheaper than the combined stack it replaces.

Sales cycles run fast—usually a seven-day trial followed by contract signing. Procurement and legal reviews take longer, but the technical evaluation wraps quickly. Most deals close within a month of first contact.

The current average contract value sits around $10,000 to $15,000 annually for a 10-user, 25TB setup. Growth comes primarily through inbound traffic from Google searches and customer referrals.

What's Next

Shade plans to double the team this year, with most new hires in engineering. The roadmap includes sub-clipping (searching within video clips), expanded document workflows, and an app marketplace where third parties can build on the platform.

The vision echoes what Salesforce did for CRM and Notion did for documents—become the default system of record, but for files. Whether that happens depends on execution. But the technical foundation is solid, and the team has proven they can ship fast.

For engineering teams tired of duct-taping together file storage solutions, Shade offers a cleaner option. One platform, one API, one place where files live and teams collaborate.

AuriStor rebuilt the Andrew File System to fix decades of technical debt and performance issues.

Thu, 09 Oct 2025 23:57:38 +0000

AuriStor Tackles Distributed File System Performance With Protocol-Level Improvements

Most developers don't think much about distributed file systems until they hit a wall. You're moving files across continents, dealing with spotty connections, or trying to sync data for thousands of compute nodes. That's when you realize the infrastructure matters.

AuriStor spent the past decade rebuilding the Andrew File System. They forked from OpenAFS in 2012 and focused on what they call "paying off technical debt." The result is AuriStorFS, a commercial product that prioritizes performance and security over backward compatibility.

The company's approach is technical, not trendy. They identified bottlenecks in the RX remote procedure call protocol and systematically addressed them. Senior engineer Simon Wilkinson presented results showing a 10x performance improvement—the time to transfer 10GB of data dropped from 437,500 milliseconds to under 50,000 milliseconds using multiple threads.

Protocol improvements that matter

RX is a UDP-based RPC protocol. It's not new or exciting, but it works. AuriStor's team analyzed every aspect of how packets move through the system and found ways to optimize.

They increased the default window size from 32 packets to 8,192 packets. A larger window means more data in flight at once. On a 1ms round-trip link, this bumps theoretical throughput from 362 Mbit/sec to 92 Gbit/sec.

Window size alone doesn't solve everything. The team implemented RFC-compliant congestion control algorithms, including New Reno and SACK-based loss recovery. They added RACK-TLP for better tail loss detection. These aren't radical ideas—they're proven TCP techniques adapted for RX.

Path MTU discovery now works properly. The system starts with a conservative 1,200-byte MTU and probes upward. If your network supports jumbo frames, AuriStor uses them. This cuts packet overhead significantly.

Real-world performance gains

One government research organization moved its AFS deployment to the cloud and measured the results. Copying a 1GB file from local disk to AFS took 3 minutes and 11 seconds with OpenAFS servers. With AuriStor's 2021 release, it dropped to 1 minute. The latest release with path MTU discovery and proportional rate reduction cut it to 30 seconds.

These improvements scale. Production AuriStor file servers handle over 500,000 simultaneous connections from 40,000 cache managers. One deployment manages 1.7 million volumes, with shutdown times reduced from 30 minutes to 4 seconds.

A large financial institution (operating under NDA) runs 80 cells with 300 servers serving 175,000 clients. They deploy software updates continuously—new releases push out every few minutes. The infrastructure handles 1.5 million volumes across 180 to 200 regional cells globally, distributed across multiple cloud providers.

Security without the theater

The security model is straightforward. AuriStorFS uses GSS-API with Kerberos v5 for authentication and AES-256 for encryption. The YFS-RxGK security class supports combined identity authentication—both user and machine identities are verified together.

This matters for organizations that need to control where data gets accessed. You're not just "you"—you're "you on your phone," "you on your company laptop," "you on your personal laptop." You can grant permissions to a user only when they're on a specific machine.

Volume-level security policies let administrators enforce encryption requirements. Volumes can only move to servers with equal or stronger security policies. File servers can require specific authentication levels before serving data. Maximum access control lists prevent users from accidentally exposing data, even if they set overly permissive permissions on individual files.

The perpetual license model

AuriStor's pricing is unusual. Base cost is $21,000 per year per cell, covering up to 4 servers and 1,000 user or machine IDs. Additional servers cost $1,000 to $2,500 each. There's no charge based on storage capacity or data volume. As Jeffrey Altman, Founder and CEO
put it during the presentation to the 64th IT Press Tour, "We give you the first 100 exabytes for free."

The license is perpetual. Stop paying and you keep using the version you licensed. You just don't get updates or support. Security patches continue for at least two years after each release.

For developers working on large-scale deployments, this model is predictable. Your costs don't explode when you add storage.

Container integration happening now

AuriStor built a CSI driver for Kubernetes that works with Red Hat OpenShift. Their large financial customer is using it to transition 180,000 systems to containers without losing the benefits of AFS.

The challenge: container images for machine learning applications start at 40 gigabytes. You don't need all that data to run the application—only a fraction gets executed. Copying 40 gigabytes to expensive compute instances wastes time and money.

AuriStorFS caches only the regions of files that applications actually access. The CSI driver lets containers mount AFS volumes directly. Applications execute binaries from the AFS namespace instead of packaging them into container images.

One customer has software distribution trees that are hundreds of gigabytes. With updates every few minutes, pushing new container images to 50,000 clients is impossible. AuriStorFS replicates only the incremental changes.

The Linux kernel's built-in AFS client (kafs) supports AuriStorFS protocols. It's distributed in Fedora, Ubuntu, Debian, and RHEL 9.2 and later. Last night, the team successfully booted a Linux system from AFS for the first time using mainline kernel tools.

Technical trade-offs

AuriStorFS doesn't support server-side byte range locks yet. This means it's not suitable for databases or virtual machine images. That's on the roadmap but won't arrive before 2026.

The system works best for read-heavy workloads with occasional writes. Think software distribution, research data, and home directories. It handles files up to 16 exabytes, though real-world deployments use files in the 5TB to 50TB range regularly.

Mixed deployments with OpenAFS work but lose some features. You can run both, but you're limited to the lowest common denominator.

Small team, focused mission

AuriStor has a core team of five developers, the same people who worked on OpenAFS before the fork. They're augmented by about five part-time contractors with specialized skills.

The company is sub-$2 million in annual revenue but profitable. Altman didn't take a salary from 2007 to 2016. The company received SBIR grants from the Department of Energy, sponsored by SLAC and Fermilab.

More than 50% of their engineering effort goes into network performance. As Altman said, "If we can't make the network fast, it doesn't matter what anything else does."

Source code is walled

AuriStor keeps source code private. Customers can license access to repositories, test infrastructure, and participate in development. If AuriStor goes out of business, source licensees gain the right to create public forks.

This is a deliberate choice after years of watching OpenAFS struggle with the free rider problem. Organizations would use the software, but not fund development. The trade-off is commercial sustainability versus open collaboration. AuriStor chose sustainability.

For shops considering distributed file systems, AuriStorFS offers measurable improvements over OpenAFS. The question is whether the licensing model and closed source approach fit your organization. If you need global data access with predictable costs and don't want storage-based pricing, it's worth evaluating.