Coder Legion - Recent posts tagged distributed-systems

Reliable Event-Driven Architecture in Spring Boot: Outbox, Inbox, Retries, and Idempotency

Thu, 29 Oct 2026 00:07:44 +0000

Why publishing an event is only the beginning of building a reliable distributed system.

Building an event-driven application with Spring Boot is relatively straightforward.

You create a Kafka producer, publish an event, implement a consumer, and process the message.

Everything works beautifully during development.

Then production introduces the situations your happy-path tests never covered.

What happens when the database transaction commits, but Kafka is unavailable? What happens when the consumer processes an event successfully but crashes before acknowledging it? What happens when two application instances attempt to process the same message?

These aren't unusual edge cases. They're fundamental challenges of distributed systems.

And solving them requires more than adding a retry annotation.

Let's explore the reliability boundaries behind a production-ready event-driven architecture.

1. The dual-write problem

Imagine a payment service that must perform two operations:

Save a payment in PostgreSQL.
Publish a PaymentCompleted event to Kafka.

A straightforward implementation might look like this:

@Transactional
public void completePayment(Payment payment) {
    paymentRepository.save(payment);
    kafkaTemplate.send(
        "payment.completed",
        new PaymentCompleted(payment.getId())
    );
}

The code looks reasonable, but there's a serious problem.

The database transaction and Kafka publication do not share a single atomic transaction.

Several outcomes are possible:

The database commits, but the event is never published.
Kafka accepts the event, but the database transaction rolls back.
Publication times out, leaving the application uncertain whether Kafka received the event.

Adding retries doesn't eliminate the fundamental inconsistency.

The problem is not whether Kafka is reliable. The problem is coordinating two independent systems.

The transactional outbox

The transactional outbox pattern changes the transaction boundary.

Instead of publishing directly to Kafka, the application writes an outbox record within the same database transaction as the business operation.

Payment Service
      |
      v
Database Transaction
      |
      +-- Save Payment
      |
      +-- Save Outbox Event
      |
      v
    Commit
      |
      v
Outbox Publisher
      |
      v
    Kafka

Either the payment and outbox event are committed together, or neither is committed.

A separate publisher retrieves pending outbox records and delivers them to Kafka.

This solves the atomicity problem between the business state and the durable intent to publish.

It does not guarantee that an event will be published exactly once.

That distinction matters.

2. Why the outbox still needs idempotency

Consider this sequence:

The outbox publisher sends an event to Kafka.
Kafka successfully accepts it.
The application crashes before marking the outbox record as published.
The publisher restarts and sends the event again.

Kafka may now contain two copies of the same logical event.

This is a normal consequence of at-least-once delivery.

The correct response is not to assume duplicates can always be prevented.

Instead, design consumers that can safely tolerate them.

3. The transactional inbox

An inbox provides a durable record of incoming events that have already been processed.

Each event should carry a stable identifier, such as an event ID generated when the logical event is created.

When the consumer receives a message, it checks whether that event has already been processed.

A simplified model looks like this:

Kafka Event
     |
     v
Consumer
     |
     v
Database Transaction
     |
     +-- Claim Event ID in Inbox
     |
     +-- Apply Business Changes
     |
     +-- Mark Inbox Completed
     |
     v
   Commit

The important part is the transaction boundary.

The inbox record and the business changes must participate in the same transaction when using the same transactional database.

Otherwise, a consumer can record an event as completed even though its business operation failed.

A subtle but important case

Suppose a consumer receives PaymentCompleted, updates an invoice, and produces InvoicePaid.

Now the reliability boundary becomes:

Inbox + Business State + Outbox

All three should be committed atomically.

Incoming Event
      |
      v
Database Transaction
      |
      +-- Inbox Claim
      |
      +-- Update Invoice
      |
      +-- Create Outbox Event
      |
      v
    Commit
      |
      v
Outbox Publisher
      |
      v
Outgoing Event

This is one of the most useful patterns for building dependable event-driven workflows.

The inbox protects the incoming side. The outbox protects the outgoing side.

Neither eliminates the need for careful business-level idempotency, especially when external side effects are involved.

4. Retries are a lifecycle, not just an annotation

Retries are often introduced with a simple configuration:

@Retryable(
    retryFor = TransientException.class,
    maxAttempts = 3
)

For short-lived operations, that can be perfectly reasonable.

But what happens when a dependency is unavailable for 30 minutes? What happens when the application restarts? What happens when an event requires operator intervention?

An in-memory retry loop cannot reliably manage every long-running failure scenario.

For durable asynchronous processing, retries should be modeled as persistent state.

A typical lifecycle might look like this:

PENDING
   |
   v
PROCESSING
   |
   +---- Success ----> COMPLETED
   |
   +---- Retryable failure
   |           |
   |           v
   |      RETRY_SCHEDULED
   |           |
   |           v
   |        PENDING
   |
   +---- Terminal failure
               |
               v
             FAILED

Each attempt can record useful information:

Attempt count
Next eligible execution time
Last failure category
Processing status
Relevant timestamps

Exponential backoff with jitter helps avoid repeatedly overwhelming an unhealthy dependency.

But retry policies should distinguish transient failures from permanent ones.

Retrying an invalid business request a thousand times won't make it valid.

5. Multi-instance processing changes everything

A background publisher may work perfectly when there's only one application instance.

Then the service is scaled to three pods.

All three publishers query the same pending records.

Without coordination, multiple instances may attempt to publish or process the same work concurrently.

One common PostgreSQL technique is:

SELECT id
FROM outbox_event
WHERE status = 'PENDING'
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

SKIP LOCKED allows concurrent workers to skip rows already locked by another transaction.

This can improve coordination between workers, but it's not a complete delivery protocol.

The design must still consider:

How work is claimed and how long claims remain valid
Transaction duration and lock ownership
Recovery after a worker crashes
Retry scheduling and fairness
Ordering requirements
Duplicate delivery

For example, holding database locks across a slow network call can increase contention and create unnecessarily long transactions.

A robust implementation may use short claim transactions, explicit processing ownership, and recovery mechanisms rather than keeping database transactions open while publishing.

Concurrency control and delivery guarantees are related, but they are not interchangeable.

6. Exactly-once is not an end-to-end promise

It's tempting to describe an event-driven system as exactly-once.

But exactly-once semantics depend on the boundaries being discussed.

Kafka transactions can provide guarantees for supported Kafka operations. A PostgreSQL transaction can atomically update records within PostgreSQL.

Neither automatically makes a database update, Kafka publication, HTTP request, and external payment operation one globally atomic action.

A more useful engineering goal is:

At-least-once delivery with effectively-once business outcomes wherever possible.

That typically requires durable event identities, transactional inbox processing, idempotent business operations, and explicit handling of external side effects.

For payment systems, this distinction is particularly important.

Processing the same event twice must not mean charging a customer twice.

7. Observability is part of reliability

A system that retries failures silently can appear healthy while accumulating thousands of unprocessed events.

Operational visibility should answer questions such as:

How many events are pending?
How old is the oldest unpublished event?
How many retries are scheduled?
Which events are permanently failing?
Are workers making progress?
Are duplicates being detected?
Is event processing latency increasing?

Metrics, structured logs, tracing, and operator-facing failure information are not optional extras for a production system.

They help distinguish a temporary delay from a workflow that has stopped making progress.

A reliable system should make failure visible and recovery understandable.

8. Putting these patterns together

A practical architecture combines several responsibilities:

Business Operation
       |
       v
Transactional Outbox
       |
       v
Durable Publisher
       |
       v
Message Broker
       |
       v
Transactional Inbox
       |
       v
Business Processing
       |
       +-- State Changes
       |
       +-- Optional New Outbox Events
       |
       v
Commit

Around that flow, we need retry policies, concurrency coordination, idempotency, recovery, and observability.

Each component solves a different failure mode.

The outbox prevents losing the intent to publish when business data commits.

The inbox protects consumers from repeated processing.

Retries help transient failures recover.

Concurrency controls coordinate multiple workers.

Observability makes the system operable.

None of these patterns should be treated as magic.

Their value comes from understanding exactly which failure boundary each one protects.

9. Why I built NERV Event

After working with event-driven enterprise applications, I wanted these reliability concerns to be reusable rather than repeatedly implemented in individual services.

That led to NERV Event, an open-source Java and Spring Boot library focused on patterns such as transactional outbox, inbox processing, retries, and idempotency.

The goal isn't to hide distributed-systems complexity behind annotations.

It's to provide reusable infrastructure while keeping the reliability model explicit.

GitHub: https://github.com/czetsuyatech/nerv-event

The library is one possible implementation of these ideas. The architectural principles are useful regardless of which framework or library you choose.

Final thoughts

Publishing an event is easy.

Building a system that behaves predictably when databases, brokers, application instances, and downstream dependencies fail is much harder.

Reliable event-driven architecture begins with a simple question:

What happens if the application crashes at this exact point?

Ask that question at every transaction boundary, every message acknowledgment, and every external call.

The answers will reveal where durability, idempotency, retries, and recovery are actually needed.

I'd be interested to hear how other teams approach this: Do you use a transactional inbox alongside your outbox, or do you handle consumer idempotency directly in your business tables?

Originally published on CzetsuyaTech. Explore more articles on Java, Spring Boot, distributed systems, and software architecture there.

Tuesday Isn't a Valid State Transition

Tue, 20 Oct 2026 14:06:00 +0000

Calendars, evidence, and the difference between done and stuck

I expected my pear wine to spend two to three weeks in primary fermentation.

It spent nineteen days. The yeast went in on September 23, and on October 12 I siphoned the wine off its sediment and into a glass carboy.

So the calendar was right. That still isn't why I moved it.

What the calendar is for

Every fermentation recipe comes with time estimates. Primary takes one to three weeks. Secondary takes a few more. Bulk age for a few months. Those numbers aren't wrong. They're averages across a lot of batches, and they're useful for exactly one thing: planning. They tell you when to have a clean carboy ready, when to clear some space, when to stop making weekend plans.

What they can't do is tell you that this batch is ready. Yeast doesn't read recipes. It works at a speed set by temperature, sugar, nutrients, acidity, the particular strain, and the particular fruit, none of which the recipe author has ever met.

So the recipe gives you a forecast, and the fermentation gives you evidence. Only one of those gets to move the wine. If they happen to agree, as they did this time, that's a pleasant coincidence, not an authorization.

The rule winemakers actually use

The traditional test for "primary is done, time to rack" is simple: take readings over several days, and when you get the same reading three times running, fermentation has finished.

What I like about that rule is what it isn't. It isn't a date. It also isn't a single reading, because one measurement tells you where the fermentation is, not whether it's still moving. It's evidence sustained over a window. Time is part of the test, but only as the space in which the evidence has to hold still.

Here's what the pear's refractometer readings actually looked like:

Date	Raw Brix
Sep 23	12.9 (before the yeast)
Sep 25	6.0
Sep 30	6.0
Oct 2	5.7
Oct 4	5.6
Oct 7	5.6
Oct 10	5.6
Oct 11	5.6
Oct 12	5.6

Two things in that table are worth a closer look.

First, the early plateau. Most of the sugar was gone within two days, and then the reading sat at exactly 6.0 for five days. If I'd been looking for stillness and nothing else, that looked a lot like the end. It wasn't. The fermentation was slowing down, not stopping, and over the next week it crept down further before it finally held. Two matching readings were a pause, not a verdict, which is precisely why the rule asks for three.

Second, I didn't stop at three. By October 10 I had three consecutive readings at 5.6, which satisfies the rule. I kept reading anyway, and waited until five readings across eight days agreed.

Then the racking itself handed me one more piece of evidence. There was more wine in the primary fermenter than the glass carboy could hold. Rather than pour the extra down the drain, I filled a hydrometer jar with it: a completely different instrument, measuring a completely different physical property. The refractometer, once corrected for alcohol, put the wine at about 1.004. The hydrometer said 1.002. Two independent instruments, close enough to agree, and the second one confirmed that the correction formula I'd been trusting for three weeks was doing its job.

I like that this happened by accident, because it isn't really an accident. A hydrometer needs a large sample, which in a one-gallon batch is too expensive to take routinely. At the moment of a transition, when wine is being moved anyway, that same sample costs nothing. The transition is when the more trustworthy evidence becomes cheap.

None of that is the calendar talking. That's a guard, and it was satisfied before the wine moved.

Status is a column anyone can write

In a lot of systems, state is a string in a database, and a transition is an UPDATE statement.

UPDATE orders SET status = 'shipped' WHERE id = 4417;

Nothing in that statement requires that anything was shipped. It's a label, and anyone with write access can apply it for any reason: a cron job that marks orders shipped two days after payment because that's usually true, a release train that promotes whatever's on the branch because it's Tuesday, a support tool that flips a flag to make a ticket go away.

Each of these is the software version of racking the wine because the recipe said two to three weeks. Most of the time it'll even be right, just as my calendar was. The transition happens on schedule, the dashboard updates, and the system now asserts a state that nobody verified.

The fix isn't elaborate. A transition should have a guard, a condition that has to be true, based on evidence, before the state can change. "Shipped" should require a carrier scan. "Released" should require the checks that define releasable. "Primary complete" should require stable readings. The state machine's job is to refuse the transition when the evidence isn't there, however much the calendar insists, and to keep refusing even on the days the calendar happens to be right.

Same evidence, two states

Here's where it gets interesting. "Three consistent readings" is the signal for done. But a fermentation that has stalled also produces three consistent readings.

Mead is notorious for this. Honey has very little natural buffering, so as fermentation proceeds the pH drifts down, and past a certain point the yeast slows or quits entirely, often well short of dry. From the refractometer's point of view, a stuck mead and a finished mead look exactly the same: a number that has stopped moving.

My fireweed mead is a few days from that question right now. Its pH started at 3.5 and dropped to 3.1 within the first four days, where it has mostly stayed. Here are its last few readings:

Date	Raw Brix	Corrected Brix	Estimated ABV
Oct 7	11.8	4.3	10.4%
Oct 10	9.0	-0.3	12.8%
Oct 11	8.6	-1.0	13.1%
Oct 12	8.4	-1.3	13.3%

Look at the corrected column. The calculator now claims the mead contains less than no sugar, which is impossible. That's a model being pushed past the range it was built for, and it's a nice reminder that a derived number can confidently assert something no physical system can be. If my guard were "corrected Brix at or below zero," I'd have racked the mead two days ago.

The raw readings tell a different story: they're still falling, slowly. The mead isn't still yet, so the guard isn't satisfied, no matter what the corrected number says. And when it does go still, the question becomes where. The yeast I used, 71B, is generally rated to tolerate around 14% alcohol, and the estimate is already over 13%. A mead that stops near its yeast's limit with the sugar it was expected to consume gone is probably done. One that stops early, with sugar left over and the pH low, is probably stuck.

The mead has already taught me this lesson once, in miniature. Mead yeast needs nutrients added in stages early on, and the textbook guidance ties the last addition to the "one-third sugar break," the point where about a third of the sugar has been consumed. I scheduled my three additions by day instead. As it happened, the last one landed right around the sugar break anyway. Sometimes luck happens. But luck is exactly what it was: the schedule didn't know where the sugar was, and if the mead had been a few days slower, the same calendar would have fed it at the wrong time.

In other words, the guard for "done" can't just be "stable." It has to be "stable where a finished fermentation should be," which means the guard needs a model of the expected end state, not just a test for motion. And it needs to know which of its inputs to trust when they disagree.

Software has the same ambiguity everywhere. A batch job whose progress counter hasn't moved in an hour has either finished or hung. A queue depth sitting at zero means either everything was processed or the consumer died. A replication lag of zero might mean the replica is caught up, or that it stopped receiving anything to lag behind. In every case, "nothing is changing" is evidence consistent with two opposite states, and a transition that only checks for stillness will eventually pick the wrong one.

A transition is an event with a receipt

The row in my pear log for October 12 reads, in effect: 11:10 a.m., 5.6 Brix, hydrometer 1.002, moved to glass carboy. The transition and the evidence that justified it are on the same line.

That matters more than the new status. Months from now, if the wine turns out strange, "racked on October 12" tells me almost nothing. "Racked after five stable readings, confirmed by a second instrument" tells me what I believed, what I based it on, and whether I might have been fooled by a stall.

Systems benefit from the same habit. A state change shouldn't just overwrite the previous state. It should be recorded as an event: what the state was, what it became, what evidence satisfied the guard, and who or what made the call. Then the current state is something you can explain, not just something you can read. When "shipped" turns out to be wrong, you can see whether the guard was bad, the evidence was bad, or the guard was never checked at all.

What time is actually allowed to do

None of this means time is irrelevant. It plays three legitimate roles, and it's worth keeping them straight.

Time can permit a transition. Some transitions shouldn't happen before a minimum interval no matter what the evidence says, like a waiting period before a refund clears, or a wine that needs some minimum aging before bottling. Here time is a precondition alongside the evidence, not a replacement for it.

Time can create the conditions for evidence. The three-readings rule only works because readings are spread over days. Stability over a window requires a window. The pear's false plateau is exactly why: two readings five days apart looked still, and only more time showed they weren't.

Time can raise a question. If a fermentation hasn't finished long after it should have, that's worth investigating. But notice what the timeout actually tells you: something is unusual. It doesn't tell you the batch is done, and it doesn't tell you the batch has failed.

That last one is where software most often goes wrong. Timeouts routinely get wired straight to a terminal state: no response in thirty seconds means failed, no update in three days means complete. A timeout is legitimate evidence that you need to look. It is rarely evidence of what you'll find when you do. A timeout should produce a question, not an answer.

What time shouldn't do, on its own, is cause a transition. Nineteen days arriving was not a fact about the pear wine.

Guarding your transitions

Treat schedules as forecasts. Plan with them. Don't let them move state, even when they're right.

Put a guard on every transition that matters. If a state can be reached by writing a string, eventually it will be reached for the wrong reason.

Test for the expected end state, not just stillness. Stable and finished are different claims. A guard that can't tell them apart will someday declare a stall a success.

Know which evidence to trust when it disagrees. A derived value can be confidently impossible. Raw observations and independent instruments are how you catch it.

Record the evidence with the transition. The new state is the conclusion. Keep the reasons next to it.

Let timeouts ask, not answer. Route them to "needs attention," not to "done" or "failed."

The pear is in its carboy now, clearing slowly. October 12 was the day it moved. It happened to be a Monday. That wasn't the reason either.

From E-Commerce to Agentic AI: Paul Chen’s 30-Year Developer's Journey

Wed, 09 Sep 2026 09:22:02 +0000

From E-Commerce to Agentic AI: A 30-Year Developer’s Journey

I wrote my first real program in the early 1990s, sitting in a computer lab at university while working toward a Computer Science degree.

Back then, programming wasn't the career everyone was racing toward.

There was no app store, no cloud, and barely a World Wide Web.

You wrote code because you were genuinely fascinated by the idea that a machine could be made to think, even if only a little bit.

More than 30 years later, I'm still chasing that idea.

The technologies have changed dramatically — from e-commerce and distributed systems to cloud computing, serverless platforms, and now agentic AI.

But the fundamental question hasn't changed:

How can we make software do more with less human direction?

This is the story of that journey.

How It All Started

That fascination with programming led me to IBM straight out of university in 1997.

My first project was integrating tooling for WebSphere Commerce Server — e-commerce in its most primitive form.

This was before payment gateways were commodities and before anyone had figured out how to make online shopping feel safe.

We were building the pipes.

And the pipes were leaking.

Every day was a new problem nobody had solved before, and I loved it.

I joined the team right in the middle of a release crunch.

The very first version of the product was days away from shipping.

I was handed the storefront code and told to build the payment and shipping functions using JSP.

People were willing to help, but the release schedule had everyone stretched thin. There's only so much time senior engineers can spend hand-holding the new person when a deadline is breathing down everyone's neck.

You figure things out.

And you figure them out fast.

And figure things out I did — sometimes the hard way.

Somewhere in the middle of building the JSP wizard for shipping destinations, I quietly excluded Puerto Rico from the United States address list.

It wasn't a dramatic mistake.

It was just a missing entry in a form.

But a tester caught it.

And it taught me something I haven't forgotten:

The edge cases you miss are almost always the ones you didn't know to look for.

After that, I helped build a common, reusable library for payment and shipping address handling, so no one else on the team would have to rediscover the same geography lesson.

We shipped on time.

After one intense week, the team released the first version of the e-commerce engine.

I had joined late in the cycle and contributed only a slice of the whole, but I remember the feeling of that release clearly — the specific kind of pride that comes from being part of something that actually ships, even when you're new and still finding your footing.

Twenty Years at IBM: Growing Into the Problem

I stayed at IBM for more than twenty years.

That might sound like inertia, but it wasn't.

The problems kept changing faster than I could outrun them.

Learning What Distributed Systems Really Feel Like

The provisioning years, from 2003 to 2011, were where I learned what distributed systems actually feel like at scale.

I was the Technology Design Lead on IBM Tivoli Provisioning Orchestration, and later the Architect for Tivoli Provisioning Manager.

Provisioning servers across an enterprise sounds mundane until you've debugged a race condition that only surfaces at three in the morning when five hundred machines try to boot simultaneously.

Those years taught me to think about systems as living, breathing things.

Not static components.

Dynamic interactions.

When Cloud Made Problems Invisible

The caching and cloud years, from 2011 to 2017, pushed me into a different dimension entirely.

I became the Lead Architect for IBM Bluemix Data and Session Cache and eventually an IBM Senior Technical Staff Member (STSM) leading WebSphere eXtremeScale and Distributed Caching as a Service.

The shift to cloud meant the problems didn't just get harder.

They got invisible.

Performance issues weren't in your hands anymore.

They lived somewhere in a data centre you'd never visit.

I had to develop a different mental model:

Design for failure, not against it.

By this point, I had been recognised as an IBM Master Inventor, a title that came from contributing to more than twenty patents spanning automation, distributed systems, cloud data replication, and platform integration.

But the patents weren't the point.

Each one represented a problem that had no concise answer in the literature, where we had to work from first principles.

That habit of thinking from first principles is probably the most transferable thing I've carried across my entire career.

The Challenges That Shaped Me

I want to be honest here, because that's the point of this kind of post.

The hardest technical challenge I faced wasn't necessarily the most complex one.

It was designing for geo-distributed active-active data replication at Huawei, where two data centres in different regions both accept writes and must eventually converge to a consistent state.

The theory is elegant.

The reality is a minefield of clock drift, network partitions, and conflict-resolution edge cases that will humble you.

We published peer-reviewed research on it, which helped.

But what really helped was accepting something fundamental:

Distributed systems are fundamentally non-deterministic, and your job as an architect is to make uncertainty manageable, not to eliminate it.

From Technical Contributor to Technical Leader

The hardest career challenge was different.

It was making the jump from deep technical contributor to technical leader and then again from technical leader to entrepreneur.

Each transition required me to let go of something I was good at.

When I became a Chief Architect at Huawei leading 20–30 engineers, I had to accept that my leverage was now in the clarity of my technical vision and the health of the team, not in the elegance of my own code.

When I later co-founded a startup, I had to accept something else:

There are no playbooks, and the speed of learning matters more than the depth of experience.

When the Architecture Was Right but the Tools Weren't Ready

One project comes to mind clearly.

In 2024, I was leading a serverless computing initiative to build lightweight AI agent runtimes using WebAssembly (WASM).

The idea was that WASM's sandboxed, near-native execution model was a natural fit for running AI agents efficiently at the edge.

The architecture was sound.

The problem was that WASM tooling for Python wasn't mature enough at the time.

You couldn't simply compile Python code into a WASM module and run it.

Every integration required manual workarounds, bespoke bridging code, and a lot of patience.

We made it work.

But it cost significantly more engineering effort than the design had accounted for, and the result was more brittle than I would have liked.

Looking back now, I can see exactly what would have helped: using AI-assisted code generation to produce a standard, consistent interface layer for each WASM module.

That would have removed most of the manual friction.

The tooling wasn't there yet in 2024.

But the approach was right.

We were just slightly ahead of the ecosystem.

The lesson I took from it:

Keep exploring new things, and never hang up.

Technologies that feel half-baked in one year become load-bearing infrastructure two years later.

The teams that struggled through the rough edges are the ones who know how to use it properly when the rest of the industry catches up.

What I've Learned After 30 Years

After 30 years, if I strip away the buzzwords and the titles, here is what I actually know.

1. Every Technology Wave Has the Same Shape

E-commerce in the 90s.

Dot-com in 2000–2005.

Cloud in the 2010s.

Serverless in the 2020s.

Agentic AI today.

Each wave starts with practitioners saying:

"This changes everything."

Then comes a messy middle where the hard engineering problems reveal themselves.

And then comes a settling period where the technology becomes infrastructure.

I've now lived through enough waves that I can feel when we're in the messy middle.

And that's exactly where the most interesting work happens.

2. The Language Is the Least Important Part

I've shipped production code in C, Java, Python, Rust, and Node.js.

The language matters.

Rust taught me to think about memory in a way Java never forced me to, and that changed how I reason about performance.

But the language is never the point.

The point is understanding the problem deeply enough to choose the right tool.

3. Performance Problems Are Architecture Problems in Disguise

Every time I've chased a performance issue far enough down, I've found an architectural decision made too early, under constraints that no longer exist.

The lesson isn't to avoid early decisions.

It's to make them visibly, with the assumptions documented, so the next person — including future you — knows what to revisit when the world changes.

4. Patents Are a Record of Unsolved Problems

I have 20+ filed or issued patents.

I'm prouder of the publications.

A patent means you found an answer nobody had before.

A peer-reviewed paper means you explained it clearly enough that other people can build on it.

Both matter.

But the second one is harder.

At IBM and Huawei, I led large distributed teams, sometimes across time zones and cultures.

What separated the high-performing teams from the rest wasn't raw talent.

It was whether everyone in the room could draw the same system diagram from memory.

Technical alignment is a leadership output, not a documentation artifact.

Growth as a Developer: The Honest Version

I went from writing integration tooling for e-commerce in 1997 to architecting an agentic AI platform for the industrial domain in 2026.

The path looks linear on a résumé.

It wasn't.

There were long stretches of being the least experienced person in the room — and shorter, more uncomfortable stretches of being the most experienced person and realising I was wrong about something important.

The Gartner iPaaS Visionary Quadrant recognition for Huawei's cloud integration platform within four years was a team achievement I'm proud of.

But I remember more vividly the architectural dead-ends that preceded it, and the whiteboard sessions at night where we talked each other back from decisions that would have been expensive to unwind.

The real growth came from three habits I'd encourage any developer to build early.

Capture Your Reasoning, Not Just Your Results

When you make an architectural decision, write down the why.

What problem were you solving?

What did you consider?

What did you rule out?

Code tells you what the system does.

Only your notes tell you why it does it that way.

I can't count the number of times I returned to a system months later, stared at a design choice that looked strange, and had no record of the constraints that made it sensible at the time.

That missing context is expensive.

It means re-doing the analysis, or worse, undoing a decision that was actually correct.

Over the years, this became one of the most underrated disciplines I've tried to instil in every team I've led.

Read Outside Your Stack

Some of the best ideas I've applied to software engineering came from operations research, organisational theory, and systems biology.

Distributed systems are complex adaptive systems, and the vocabulary for understanding them has existed in other fields for decades.

Developers in the AI Era

In the AI era, coding is no longer the scarcest skill.

And that changes everything.

I've watched developers at every stage of their career worry about being replaced by AI.

I think they're asking the wrong question.

AI is already a capable coding assistant.

It will only get better.

What it cannot do is understand why a business needs a system to work a certain way, see the architectural trade-offs across a five-year roadmap, or translate a domain expert's vague requirements into a design that will still be maintainable when the team changes.

Those are human skills.

And they matter more now than they did when writing code was the bottleneck.

My advice to developers who are early in their careers is simple:

Invest in your ability to think in systems, understand the business problems behind the technical requirements, and communicate clearly across both worlds.

The developers who thrive in the next decade will be the ones who know how to direct AI effectively, not compete with it.

What I'm Building Now

I'm currently co-founding an AI startup focused on autonomous agentic systems for specific industrial domain sectors.

We're designing enterprise AI agents that can reason, plan, and act with minimal human intervention.

It's early.

It's hard.

And it's the most interesting technical problem I've worked on.

The thread from my first line of C code to today's agentic systems is actually straightforward:

I've always been drawn to the problem of making software do more with less human direction.

E-commerce automation in 1997.

Server provisioning orchestration in 2003.

Serverless computing on the edge in 2020.

Autonomous AI agents in 2026.

The abstraction layer has risen.

The fundamental question hasn't changed.

If you're a developer earlier in your journey, find that question — the one you keep circling back to no matter which technology is in fashion — and follow it.

The career will take care of itself.

A Note on CoderLegion

Communities like CoderLegion matter most in the messy middle of a wave when everyone is confident the technology is important but nobody yet agrees on the right way to use it.

Agentic AI is squarely in that middle right now.

The conversations I've found here, from practitioners who are actually building, not just theorising, are exactly the kind of signal that helps me calibrate what problems are real and which are manufactured by hype cycles.

About Paul Chen

Paul Chen is a COO focused on Agentic AI & Autonomous Systems and Intelligent Cloud & Edge Services, with more than 30 years of experience in distributed systems, advanced cloud applications, and serverless platforms. He is currently focused on agentic AI, autonomous systems, and multi-agent systems. :contentReference[oaicite:1]{index=1}

His career has spanned e-commerce, distributed systems, cloud computing, serverless platforms, and now autonomous AI systems.

View Paul Chen's CoderLegion Profile

Developer: Paul Chen
Series: CoderLegion Developer Stories
Interviewed and edited by: Mehadi Hasan, Community & Editorial Team, CoderLegion

Idempotency Starts with Business Operation Identity

Thu, 06 Aug 2026 09:13:38 +0000

The hardest part of idempotency is often not the retry. It is deciding when two attempts represent the same business operation.

Sometimes the identity is a request ID or external message ID. Sometimes it is a business key or a state transition that may happen only once. Once that identity is explicit, a transaction and database uniqueness constraint can enforce it, even when two attempts arrive together.

Without identity, "make it idempotent" is not yet a design. The system still does not know what must not happen twice. This matches how Stripe and AWS describe safe retries: repeated attempts need a stable identity so the original side effect is not performed again.

Determinism Is a Core Contract in Build Caching

Tue, 04 Aug 2026 14:30:00 +0000

If you have set up a remote build cache and watched the hit rate sit stubbornly low, nondeterministic outputs are one important suspect — but not the only one. Source churn, changing flags, platform differences, toolchain changes, eviction, and incomplete input declarations also cause misses. Nondeterminism is particularly dangerous because it can quietly reduce reuse or make cached results untrustworthy without producing an obvious build error. It is worth understanding exactly how that happens and how to isolate it.

Why determinism is the contract, not a nice-to-have

A build cache works by fingerprinting an action's inputs and using that fingerprint to look up a previously computed output. The entire scheme rests on one assumption: identical inputs produce identical outputs. That is the contract. Break it and one of two bad things happens.

If the cache keys only on inputs — which is how these systems work — and your action is nondeterministic, then the cache will happily hand back an output for those inputs, but not necessarily the one a fresh run would produce. Best case, the variation is cosmetic and harmless. Worst case, you have just served a subtly wrong artifact from cache and you will spend a very confusing afternoon. Either way, the cache is now a source of doubt rather than speed.

Another operational outcome is cascading misses, and that is where determinism stops being abstract.

One bad action poisons everything downstream

Builds are graphs. The output of one action is the input to the next. So consider what happens when a single early action — say, a code generator — embeds the current timestamp in its output. Each time that generator actually runs, the file it produces is byte-for-byte different, so its content hash changes.

Whenever that generator has to execute again — because its entry was evicted, the cache was bypassed, or another declared input changed — it produces a new content digest. Every action that consumes the generated file then sees a different input and misses its previous cache entry. Their outputs feed further actions, so the effect can cascade through the subgraph. You did not merely lose reuse for one action; you invalidated downstream keys even when the semantic content did not need to change.

Where nondeterminism actually comes from

The sources are mundane, which is what makes them easy to miss. The usual suspects:

Timestamps baked into outputs — __DATE__ and __TIME__ macros, build IDs, and the modification times stored inside archive formats like .a, .tar, and .zip.
Absolute paths leaking into outputs — debug info that records the full build directory, __FILE__ expansions, paths embedded by the linker. Build the same code in /home/alice and /home/bob and you get different bytes.
Iteration order — generating code by walking a hash map or set, whose iteration order is unspecified, so the output lines come out shuffled run to run.
Concurrency — actions that assemble output from parallel work without imposing a stable order on the result.
Randomness — UUIDs, unseeded PRNGs, anything that reaches for entropy at build time.
The ambient environment — locale, hostname, username, and stray environment variables that the action reads but never declared as inputs.
Filesystem ordering — relying on readdir order, which varies across filesystems and runs.

None of these throws an error. Each just makes one action's output wiggle, and the cache does the rest of the damage.

How to hunt them down

The basic diagnostic is to execute the same build twice from clean, isolated directories with cache reuse disabled, then compare the outputs bit for bit. Anything that differs deserves investigation. diffoscope is the right tool here — it understands archives, binaries, and debug information and can show not just that two outputs differ but where.

The fixes are well-trodden, because the reproducible-builds community has been fighting this war for years:

Timestamps: honor SOURCE_DATE_EPOCH so embedded dates are fixed rather than "now"; build archives in deterministic mode (tar --sort=name --mtime=..., ar in its deterministic D mode).
Absolute paths: use -ffile-prefix-map / -fdebug-prefix-map to rewrite build paths to a stable placeholder, so debug info does not depend on where you built.
Iteration order: sort before you emit. If codegen walks a map, sort the keys first. Never let an unordered container decide output order.
Environment: pin the locale (LC_ALL=C), and run actions in a sandbox that only exposes declared inputs, so an undeclared environment dependency fails loudly instead of silently varying.
Randomness: seed it, or design it out. A build is not the place for entropy.

Hermeticity is how you keep it fixed

Fixing the determinism bugs you have is half the job. The other half is preventing new ones, and hermetic, sandboxed builds remove many sources of ambient input. A filesystem sandbox can hide undeclared files and tools, and a controlled environment can pin locale and environment variables. Sandboxing alone does not necessarily hide the wall clock, randomness, hostname, kernel details, or every network source; those must be blocked, virtualized, or deliberately normalized. The goal is to make undeclared dependencies fail loudly or become stable rather than silently changing outputs.

Measure it, because it regresses

Determinism is not a one-time cleanup. It rots. Someone adds a code generator, someone embeds a build timestamp "just for diagnostics," and reuse slides without anyone noticing. Treat cache hit rate as a first-class metric, but segment it by action type, platform, branch, and miss reason because determinism is only one contributor. Track reproducibility separately with scheduled rebuild-and-compare tests. When hit rate drops and action keys churn without meaningful input changes, reach for build-twice-and-diff before blaming the cache infrastructure.

That is the real point. Determinism is not purity for its own sake. Along with complete input declarations, compatible platforms and toolchains, and retained cache entries, it is a precondition for trustworthy reuse. Get it wrong and the cache may return questionable outputs or lose reuse across a large subgraph. Get it right and equivalent actions can reliably share results across developers and CI. Determinism is not the whole cache system, but it is part of its correctness contract.

Further reading: the Reproducible Builds documentation

Why Modern Engineering Teams Must Prioritize System Observability Over Raw Performance

Wed, 15 Jul 2026 14:45:00 +0000

Engineering teams often chase performance: faster APIs, lower latency, optimized queries, reduced CPU usage. But in 2026, performance alone is no longer the defining factor of a reliable system. Modern architectures — distributed, event‑driven, containerized, multi‑service — demand something far more fundamental: [system observability][1].

Performance tells you how fast your system is.
Observability tells you why it behaves the way it does.

And for engineering teams, especially small ones, this difference determines whether a system scales or collapses under pressure.

Performance Without Observability Is a Blindfold
A fast system can still be fragile.
A highly optimized service can still fail silently.
A low‑latency API can still degrade unpredictably under load.

When teams focus exclusively on performance metrics, they optimize symptoms instead of causes. They see spikes, drops, delays — but they don’t see the internal relationships that create them.

Observability exposes:

hidden dependencies

cascading failures

slow internal paths

unexpected retry storms

misconfigured backoff logic

silent timeouts

resource starvation

cross‑service bottlenecks

Without this visibility, engineers operate reactively. They fix what they see, not what actually breaks the system.

The Three Pillars of Observability
Modern observability is not just logs. It is a structured, multi‑layered system that allows engineers to understand why something happened.

Metrics — the heartbeat of the system
Metrics show trends, anomalies, and thresholds.
They reveal when something is off — but not why.

Key engineering metrics include:

request latency distribution

error rate spikes

queue depth growth

retry counts

CPU throttling events

memory pressure

cache hit ratios

Metrics are the first signal, not the full story.

Logs — the narrative
Logs provide context.
They explain the events behind the metrics.

But logs alone are insufficient.
Without structure, correlation, and context, logs become noise.

Modern teams rely on:

structured JSON logs

correlation IDs

distributed trace IDs

log sampling

log enrichment with metadata

Logs transform raw events into actionable insights.

Traces — the map of the system
Distributed tracing is the most powerful observability tool of 2026.

Traces show:

how a request travels through the system

where it slows down

where it fails

where retries accumulate

where bottlenecks form

where unexpected dependencies appear

Traces reveal the architecture as it actually behaves — not as it was designed.

Why Observability Matters More for Small Teams
Large companies have SRE departments, dedicated monitoring teams, and 24/7 on‑call rotations.
Small engineering teams do not.

For small teams, observability is not a luxury — it is a multiplier.

It allows them to:

detect issues before users notice

reduce on‑call stress

shorten incident resolution time

prevent cascading failures

scale without rewriting the system

understand real‑world behavior instead of assumptions

Observability replaces guesswork with clarity.

Observability Enables Predictable Engineering
Predictability is the most valuable trait of a modern system.

A predictable system:

fails gracefully

recovers automatically

scales linearly

behaves consistently under load

provides clear signals when something is wrong

Observability is the foundation of this predictability.
It transforms engineering from reactive firefighting into proactive system design.

The Future: Observability as a Core Engineering Skill
In 2026, observability is no longer a DevOps specialty.
It is a core engineering skill.

Teams that master observability:

ship faster

debug faster

scale safer

maintain stability

reduce operational cost

build systems that survive real‑world conditions

Performance matters.
But system observability determines whether performance is sustainable.

The Art of Cloud Survival: The Day Monitoring Failed First

Wed, 15 Jul 2026 12:13:04 +0000

When Everything Looked Fine

Modern cloud systems are built to survive failure.

Applications run across multiple Availability Zones. Containers restart automatically. Serverless functions scale on demand. Databases replicate data continuously. Entire architectural patterns have emerged around the idea that failure is not an exception but an expected part of operating software at scale.

Yet some of the most painful incidents begin long before a server crashes or a service becomes unavailable.

They begin when nobody realizes something is already going wrong.

Imagine a typical weekday morning. Dashboards are open on a large monitor. Every graph appears healthy. CPU utilization is well below critical thresholds. Memory consumption is stable. No alarms are firing. Error rates remain low.

From an operational perspective, everything appears normal.

Meanwhile, users are experiencing increasing delays. Transactions that usually complete in milliseconds now take several seconds. Some requests time out entirely. Support tickets begin arriving. Customer complaints appear in chat channels. Teams start investigating because users are reporting problems, not because monitoring systems detected them.

The infrastructure is functioning. The monitoring strategy is not. And that distinction matters more than many teams realize.

The Green Dashboard Problem

One of the most dangerous moments in operations is staring at a dashboard filled with green indicators while customers are actively experiencing issues.

Most engineering teams invest heavily in monitoring. They collect metrics, configure alarms, centralize logs, and build detailed dashboards. The intention is good. Visibility is essential in cloud environments.

The challenge is that many dashboards focus on infrastructure health rather than service health.

A virtual machine can be healthy while an application is struggling.
A database can be available while queries are becoming progressively slower.
A queue can continue accepting messages while consumers fall further behind every minute.

In each case, the system is technically running. Yet the user experience is already deteriorating.

Distributed systems make this problem even harder to detect because failures rarely appear as obvious outages. Instead, they emerge gradually, hidden inside latency, retries, timeouts, and backlogs.

Even when the dashboard remains green, the customer experience does not.

Why Distributed Systems Hide Failure

In a monolithic application, a failure is often visible and immediate. If a critical component stops working, the impact is usually straightforward to identify.

Distributed systems behave differently.

A slowdown in one service can propagate through multiple dependencies. An overloaded database may increase response times. Those delays cause application servers to wait longer. Waiting requests consume resources. Retries begin accumulating. Message queues start growing. Additional traffic increases pressure on downstream services.

Nothing has technically failed.

Yet the system is moving steadily toward an incident.

This is one reason distributed architectures are both powerful and challenging. Their resilience often comes from decoupling components, but that same decoupling can make problems difficult to observe.

Symptoms appear far away from the original cause.

By the time users notice, the underlying issue may have been developing for hours.

Monitoring Tells You What You Expect

Many teams use the words monitoring and observability interchangeably.

They are related, but they are not the same thing.

Monitoring helps answer questions we already know to ask. It tells us whether predefined conditions have been met.

Is CPU utilization above 80 percent?
Is memory consumption increasing?
Has an error threshold been exceeded?

These are valuable questions. But real incidents rarely follow predefined scripts.

Observability focuses on understanding what is happening inside a system, especially when the problem is unexpected. Instead of simply collecting metrics, observability provides enough context to investigate unknown behaviors and discover why a system is acting differently than expected.

This distinction becomes increasingly important as architectures grow more complex.

A dashboard showing resource utilization may indicate that everything is healthy. Observability helps determine whether customers would agree.

The Signals That Actually Matter

One of the most influential ideas in modern operations comes from Google's Site Reliability Engineering practices: the Four Golden Signals.

These signals focus on the aspects of a system that most directly affect users.

Together, these metrics provide a much more meaningful view of system health than infrastructure metrics alone.

Consider a payment processing service.

Suppose CPU utilization remains below 40 percent and memory usage appears stable. Traditional infrastructure monitoring would suggest the service is healthy.

Now imagine database response times increase from 50 milliseconds to two seconds.

Customers begin abandoning transactions. Requests start timing out. Revenue is affected.

The infrastructure never appeared unhealthy. The customer experience did. And this is why effective monitoring begins with user-facing outcomes rather than resource consumption.

The Danger of Averages

Another common mistake is relying too heavily on averages. Average response times often create a comforting but incomplete picture. A service may report an average latency of 200 milliseconds while a small percentage of requests take several seconds to complete. Those slower requests may represent only a fraction of total traffic, yet they are often the first indication that something is wrong.

For the affected users, the average is irrelevant.

Their experience is defined by the request that took five seconds, not by the thousands that completed successfully.

This is why mature operations teams pay close attention to percentile measurements such as p95 and p99 latency.

These metrics reveal what is happening at the edges of the system, where performance problems frequently emerge first.

Many major incidents begin as small increases in tail latency long before average metrics show meaningful changes.

If teams monitor only averages, they may discover problems far later than they should.

Failures Rarely Arrive All at Once

Hollywood teaches us to expect disasters as sudden events. Real-world cloud incidents are usually much less dramatic.

Most failures arrive gradually.

A dependency becomes slower.
A third-party API starts returning intermittent errors.
A queue begins accumulating messages.
Consumers struggle to keep pace.
Retries increase traffic.
Latency spreads through the system.

What started as a minor degradation becomes a significant operational issue.

This is one reason observability is so important. Detecting a problem early often means the difference between a brief investigation and a lengthy outage.

The sooner a team can identify unusual behavior, the more options they have to contain the impact.

Cloud survival is rarely about preventing every failure, it is about recognizing failures before they become incidents.

Building Visibility on AWS

AWS provides a rich set of services that help teams understand what is happening inside distributed systems.

Amazon CloudWatch serves as the foundation for metrics, logs, dashboards, and alarms. It enables teams to monitor operational signals across applications and infrastructure.
AWS X-Ray helps trace requests as they move through multiple services, making it easier to identify bottlenecks and understand where latency is introduced.
Amazon EventBridge can automate operational workflows, trigger responses to specific events, and help reduce the time between detection and action.

Used together, these services create visibility across layers of a distributed architecture.

But tools alone do not guarantee observability. Organizations often spend months building dashboards while investing very little effort in defining what success or failure actually looks like from a user's perspective. Technology is only part of the solution. The real challenge is deciding what deserves attention.

The Question Every Team Should Ask

There is a simple question that every engineering team should periodically ask:

"If a customer experiences a serious problem right now, would we know before they tell us?"

The answer is often uncomfortable.

Many teams discover that their alerts focus on infrastructure while customer-impacting issues remain invisible until support tickets arrive.

That realization can be frustrating, but it is also valuable.

Because once teams understand the gap between system health and customer experience, they can begin designing monitoring strategies that reflect reality instead of assumptions.

The goal is not to collect more metrics, the goal is to collect the right ones.

Cloud Survival Requires Visibility

The most resilient cloud architectures are not always the ones with the most automation, the most services, or the most sophisticated deployment pipelines.

Often, they are the ones that provide the clearest visibility into what users are actually experiencing.
Failures are inevitable. Latency will increase. Dependencies will degrade. Services will behave in unexpected ways. Distributed systems guarantee that complexity will eventually surface. What separates resilient teams from struggling ones is not their ability to avoid failure.

It is their ability to see it.

The day monitoring fails first is usually the day a team learns an important lesson: collecting metrics is not the same as understanding a system. Cloud survival requires both.

Because you cannot respond to a problem you cannot see.

When "Latest" isn't Actually Latest

Thu, 02 Jul 2026 11:08:55 +0000

When "Latest" Isn't Actually Latest

TL;DR

In distributed systems, the last record your system processes is not always the most recent event that happened in reality.

Understanding the difference between Event Time and Processing Time is essential to avoid data quality issues in streaming pipelines, CDC architectures, and event-driven systems.

The Assumption That Works... Until It Doesn't

Most developers start their careers working with a single application and a single database.

In that world, time feels simple.

A customer updates their profile.
The database stores the change.
Everyone sees the updated record.

The sequence appears obvious because all operations happen within the same system.

Then distributed systems enter the picture.

A Simple Example

Imagine a customer updates their address at 10:00:00.

That change generates an event and begins traveling through a distributed architecture:

Application
Database
CDC pipeline
Kafka
Consumer service

Everything looks normal.

Now imagine an older address update, generated earlier, gets delayed because of a network issue or retry mechanism.

The result?

The older event arrives after the newer one.

Your consumer processes the events in this order:

New address
Old address

If your logic assumes that the last processed record represents the latest truth, you've just overwritten correct information with stale data.

The Real Problem: Time Has Multiple Meanings

This is why distributed systems distinguish between two important concepts:

Event Time

The moment something actually happened in the real world.

Example:

Customer changed address → 10:00:00

Processing Time

The moment a system receives or processes the event.

Example:

Consumer receives event → 10:00:15

Most of the time, these timestamps are close enough that nobody notices.

The interesting cases appear when they aren't.

Where Things Get Complicated

Several factors can cause events to arrive out of order:

Network latency
Retries
Consumer lag
Parallel processing
System failures
Distributed transactions
CDC replication delays

As architectures become more distributed, these situations become normal rather than exceptional.

What About Kafka?

Kafka helps preserve ordering within a partition through offsets.

However, offsets tell us the order in which records were written to Kafka.

They do not guarantee the order in which events actually occurred in the real world.

That's an important distinction.

A perfectly ordered Kafka partition can still contain events that represent out-of-order business actions.

Why Data Engineers Care

Many expensive data quality problems are not caused by broken infrastructure.

Pipelines may be healthy.
Kafka may be running perfectly.
Consumers may be processing records successfully.

The issue is often more subtle.

The system answers the right question using the wrong sequence of events.

When that happens, dashboards become inaccurate, customer profiles become inconsistent, and downstream decisions become unreliable.

Final Thought

Distributed systems force us to rethink a concept we usually take for granted: time.

The challenge isn't simply moving data from one system to another.

The challenge is preserving the meaning of that data while it travels through a network of independent components.

Because in distributed systems, the most recent record is not always the most recent event.

And understanding that difference is often the first step toward building reliable data systems.

The Miyake Event Problem: Anchoring Distributed Agents to Universal Time

Fri, 26 Jun 2026 03:09:29 +0000

Originally published at vibeagentmaking.com

In 2021, a team led by Margot Kuitems pinned the Viking settlement at L'Anse aux Meadows to a single year: 1021 CE. Before that work, the best estimate was "roughly 1000 CE" with multi-decade uncertainty.

They didn't build a better mass spectrometer. They found a cosmic-ray spike from 993 CE recorded in three pieces of wood, counted 28 rings to the bark, and arrived at a date no one can argue with. Precision improved 40–120x — not by better measurement, but by identifying the right external signal.

If you've ever worked with distributed systems, this should sound familiar. Your system has the same problem those archaeologists had before 2012.

The floating chronology problem

Before Miyake events were discovered, dendrochronology had a "floating chronology" problem. You could count tree rings and measure relative widths, but without an external anchor, the entire sequence could be off by decades. The rings were relatively dated but not absolutely dated.

Crossdating — matching ring patterns between trees across sites — solved this regionally. It works because climate is a shared signal: trees in the same region record the same drought years.

But crossdating only works within a region. A Japanese cedar can't be crossdated against a California bristlecone pine by ring width alone. For global anchoring, you need a signal every tree on Earth records simultaneously.

In 2012, Fusa Miyake found one: a 1.2% single-year spike in atmospheric carbon-14 from 774–775 CE — 20x the normal variation. Nine confirmed Miyake events have since been identified. These are extreme solar particle events imprinted in every growing tree, every ice core, globally, in the same calendar year.

Five properties make these uniquely powerful as anchors:

Global — every tree, everywhere
Simultaneous — same calendar year
Unambiguous — 20x above noise
Indelible — physically imprinted, can't be altered
Independently verifiable — any lab can check

Distributed systems have the same problem

Lamport formalized it in 1978: across multiple machines, "now" becomes ambiguous. You can't rely on physical clocks alone because they drift (commodity hardware: 10–200 ppm), networks have variable latency, and there's no single reference frame.

This is exactly the floating chronology. A single node's log is internally consistent — Event A before Event B on this machine. But across nodes, without an external anchor, you can't order events reliably. The logs float.

Lamport timestamps track causality rather than wall-clock time — the distributed-systems equivalent of crossdating. But like regional crossdating, they break at the boundaries: two causally disconnected streams can't be ordered.

AI agents face it in a harder form

LLM-based agents are stateless by design — every inference call starts fresh. Unlike a database node that persists its logical clock, a restarted agent loses temporal position entirely.

Context windows make it worse: 30%+ accuracy reduction for information in the middle of the context. Older timestamps degrade. And in multi-agent coordination, 36.94% of system failures trace back to stale state propagation — one agent acts on outdated information because there's no shared temporal reference.

Better clocks vs. better anchors

Google Spanner deploys GPS receivers and atomic clocks in every data center. TrueTime gets uncertainty down to ~7ms. Before committing a transaction, Spanner waits out the uncertainty to guarantee ordering. This works brilliantly — if you have Google-scale infrastructure.

CockroachDB, without atomic clocks, uses NTP (100–250ms offsets, 14–36x worse). When clock drift exceeds its configured maximum, a CockroachDB node self-terminates rather than risk inconsistent timestamps.

The system would rather die than lie about time. This maps to dendrochronology's practice of marking uncertain dates with "?" and refusing to report unreliable dates.

But both approaches try to improve the internal clock. Neither looks for external universal anchors. The L'Anse aux Meadows team didn't build a better mass spectrometer. They found the right signal.

Digital Miyake events

Ethereum's Beacon Chain: 12-second slots, finalization via Casper FFG after 2/3 validator attestation. Reverting a finalized block costs $30B+ in staked ETH. Bitcoin block height: monotonically increasing, enforced by proof-of-work.

Score these against the five-property framework and blockchain finalization hits 4.5/5 — the closest digital analog to a Miyake spike.

An agent that records "I observed state X as of Ethereum block 22,456,789" has a timestamp claim that anyone can verify, forever, without trusting the agent's local clock.

Tools like Chainpoint, OpenTimestamps, and OriginStamp already implement this pattern — aggregating hashes into Merkle trees, anchoring to Bitcoin transactions, generating self-contained proofs. This is crossdating for digital timestamps.

The ironies

Trees are better archival substrates than blockchains. A tree passively records the Miyake signal — no software, no network, no fees. We can date 7,000-year-old bristlecone pines. We can't verify a 15-year-old digital timestamp without the original chain.

Miyake events would destroy digital systems. The 774 CE event was 10x stronger than the Carrington Event — roughly 660 billion Hiroshima bombs of energy. Nature's best timestamp generator is also a civilization-ending threat to the systems that most need time anchors.

Dendrochronology solved this 50 years before CS. Douglass's crossdating (1920s) predates Lamport's logical clocks (1978) by half a century. Convergent evolution: two unrelated fields facing the same constraint arrived at the same architecture.

The lesson

The fix is not better clocks. It is better anchors.

The 993 CE spike had been sitting in every tree alive that year for a thousand years before anyone thought to look for it. The on-chain anchors are already there. The question is whether your system is looking.

Full essay with sources and data tables: vibeagentmaking.com/blog/the-miyake-event-problem/

The fix is not better clocks. It is better anchors. Chain of Consciousness applies the dendrochronologist's lesson to agent provenance — every action is anchored to a verifiable external record, building a chronology that doesn't float. Verify which events an agent can prove it witnessed, in what order, anchored to what: pip install chain-of-consciousness

The Art of Cloud Survival: When Retries Become the Outage

Thu, 25 Jun 2026 13:27:41 +0000

When Retries Become the Outage

At 2:17 AM, a payment service started returning errors. Not many. Just enough to attract attention. The dashboards didn't look alarming. CPU utilization was stable. Memory consumption was within normal limits. Infrastructure health checks were passing. If you had opened the monitoring console at that moment, you probably would have assumed the issue was temporary and would resolve itself within a few minutes.

The engineering team began investigating. Meanwhile, the rest of the system was already reacting. API clients started resubmitting requests. Lambda functions retried failed executions. Messages that couldn't be processed returned to queues. Downstream services attempted the same operations again, hoping the next attempt would succeed.

None of these actions were wrong. In fact, every component was behaving exactly as it had been designed to behave. And that was the problem.

What began as a relatively small service degradation gradually evolved into a much larger incident. The original failure affected a limited number of requests. The recovery mechanisms multiplied that traffic until the struggling service was forced to handle far more load than it had seen before the outage began.

The system wasn't collapsing because of the initial error anymore. It was collapsing because everybody was trying to help.

Why Retries Feel Like the Right Answer

Retries are one of the first resilience patterns most engineers learn, and for good reason.

Distributed systems operate in an imperfect world. Networks experience brief interruptions. Services become temporarily unavailable. Databases occasionally take longer than expected to respond. A request that fails now may succeed a few seconds later without any intervention.

Because of this, retries often improve reliability. A customer never notices the timeout. The application recovers automatically. Everybody wins. The trouble starts when we focus on the success story and forget to think about scale. A single retry seems harmless. Ten thousand retries are something else entirely.

Imagine a service receiving 10,000 requests during a busy period. Suddenly, a dependency starts responding slowly. Requests begin timing out. Each client retries three times before giving up.

Without realizing it, the system has transformed 10,000 requests into 40,000. That additional traffic creates more pressure on the dependency. More pressure leads to higher latency. Higher latency generates more failures. More failures trigger more retries. What looked like a recovery mechanism has become a feedback loop. And feedback loops are where distributed systems become dangerous.

The Real Problem Isn't Failure

One of the hardest lessons in cloud architecture is that systems rarely fail where the problem starts.

A database slows down.
An API begins timing out.
A message consumer falls behind.

Those events are usually manageable.

The real challenge comes from the chain reaction that follows. Every service makes a reasonable decision based on its local perspective. Every team implements protections intended to improve reliability. Every retry policy makes sense when viewed in isolation.

Yet when an incident occurs, those independent decisions begin interacting with each other. The result is often a level of complexity that nobody anticipated during design reviews. I've seen incidents where the root cause was resolved within minutes, but the platform continued struggling for hours because retries, reprocessing, and recovery workflows kept generating load long after the original issue had disappeared.

The failure wasn't the hardest part. Recovering from the recovery was.

When AWS Does Exactly What You Asked It To Do

AWS services are designed to help us build resilient systems, which is why many of them include retry mechanisms by default.

Take Lambda and SQS. When a Lambda function cannot successfully process a message, that message doesn't simply disappear. It becomes available again and can be processed later. This behavior protects against transient failures and prevents data loss. Most of the time, that's exactly what we want.

However, if the underlying problem is persistent—a broken dependency, an invalid payload, or a misconfigured service—the same message may be processed repeatedly while consuming compute resources and generating additional pressure on downstream systems. The same principle appears in event-driven architectures.

Services such as EventBridge improve reliability by attempting redelivery when consumers fail. That's an incredibly valuable capability. However, it also means that consumers must be designed with duplicate events in mind. Retries improve resilience. But resilience is never free.

Every retry consumes resources, generates traffic, and increases complexity. The question is not whether retries are good or bad. The question is whether the system can absorb the consequences when retries occur at scale.

Designing Retries That Help

Over the years, I've become less interested in whether a system retries and more interested in how it retries. The difference matters. One of the simplest improvements is exponential backoff.

Instead of retrying every second, the delay increases after each attempt. A request may wait one second, then two, then four, then eight. This gives dependencies time to recover and reduces the likelihood of overwhelming a struggling service.

Even then, another challenge remains. Imagine a million clients all deciding to retry after exactly eight seconds. Eight seconds later, they arrive together. The system experiences another traffic spike.

This is why mature systems often introduce jitter, adding a small amount of randomness to retry timing. The overall volume remains similar, but the requests become distributed over time rather than arriving in synchronized waves.

Another valuable pattern is the circuit breaker. When a dependency is clearly unavailable, continuing to send requests often provides little benefit. A circuit breaker temporarily stops traffic, allowing the dependency to recover while protecting the rest of the system from unnecessary load.

And sometimes, the most responsible decision is not to retry immediately at all.

Sometimes the correct answer is to move the failed message to a Dead Letter Queue, investigate the cause, and replay it later under controlled conditions. That approach may feel less aggressive than continuous retries, but it often prevents a localized issue from becoming a platform-wide incident.

A Different Way to Think About Resilience

Early in my career, I thought resilience was mostly about making systems persistent.

If something failed, try again.
If it failed again, try harder.

Experience changed that perspective.

Today, I think resilience is often about restraint. It's about understanding that every recovery mechanism introduces new behavior into the system. It's about recognizing that retries are not free. It's about accepting that the healthiest response to failure is not always immediate action. Sometimes the best thing a system can do is wait. Sometimes it's stop. Sometimes it's fail gracefully.

The most resilient cloud architectures aren't the ones that retry forever. They're the ones that understand the difference between recovering from a problem and amplifying it. Because in distributed systems, the outage doesn't always begin with the first failure. Sometimes it begins with the second attempt.

The Hidden Cost of Distributed Systems

Thu, 11 Jun 2026 12:12:07 +0000

One of the most interesting contradictions in cloud architecture is that we spend a tremendous amount of effort trying to eliminate complexity, yet many of the systems we build seem to become more complex over time.

A startup might begin with a single application, a single database, and a handful of users. Everything is relatively easy to understand. A request comes in, business logic is executed, data is stored, and a response is returned. The architecture is straightforward, deployments are simple, and troubleshooting usually involves looking in one place.

Then growth happens.

New customers arrive. New features are requested. More engineers join the team. Integrations are added. Reporting requirements expand. Mobile applications appear. Real-time processing becomes important. What was once a single application gradually evolves into a collection of services, databases, queues, and APIs.

Without necessarily realizing it, the organization has started building a distributed system.

What Exactly Is a Distributed System?

At its core, a distributed system is simply a group of independent components working together to provide what users perceive as a single application.

Consider what happens when a customer places an order on an e-commerce platform. From the customer's perspective, the experience feels simple: click a button, receive a confirmation, and wait for the package to arrive.

Behind the scenes, however, the request may travel through multiple services. One service records the order, another processes the payment, another updates inventory, while additional systems generate notifications, update dashboards, and trigger downstream workflows. Each component owns a specific responsibility, yet all of them collaborate to complete a single business transaction.

This separation of responsibilities is one of the defining characteristics of distributed systems.

Why Do We Build Them?

If distributed systems introduce additional complexity, a reasonable question follows:

Why not keep everything inside a single application?

The answer is that distributed systems solve problems that become increasingly difficult to address as organizations grow.

Imagine an online retailer preparing for Black Friday. The checkout process may suddenly experience a massive increase in traffic, while administrative reporting tools remain largely unchanged. In a monolithic architecture, scaling one component often means scaling the entire application. In a distributed architecture, services can scale independently based on their specific needs.

The benefits extend beyond scalability. Distributed systems also help organizations isolate failures, allow teams to work more independently, and enable different technologies to be used for different workloads. As systems and engineering organizations grow, these advantages become increasingly valuable.

This is why most modern cloud-native architectures eventually move in this direction.

Not because distributed systems are fashionable.

Because they solve real business and operational challenges.

The Trade-Offs Begin the Moment You Split the System

The problem is that complexity does not disappear.

It simply moves.

Inside a monolithic application, communication often happens through function calls within the same process. In a distributed system, communication happens over a network. That seemingly small change introduces an entirely new category of problems.

Networks introduce latency. Requests time out. Connections fail. Messages arrive late. Sometimes they arrive twice. Occasionally they never arrive at all.

Engineers must now design systems that can tolerate these realities.

The architecture becomes more flexible, but it also becomes less predictable.

Data Consistency Becomes a Design Problem

One of the biggest mindset shifts occurs when data is no longer stored in a single place.

Imagine a payment service successfully charges a customer. Immediately afterward, the inventory service fails before reserving the product. The customer has paid, but the order cannot be fulfilled.

What should happen next?

Should the payment be refunded automatically?
Should the inventory operation be retried?
Should the order remain pending until a human intervenes?

These questions rarely emerge in simple applications, but they appear regularly in distributed systems.

As a result, engineers begin adopting concepts such as eventual consistency, retries, idempotency, dead-letter queues, and event-driven communication patterns. These are not merely technical buzzwords. They are practical responses to the challenges created by distributing responsibilities across multiple services.

Choosing Complexity Carefully

One of the most valuable lessons in software architecture is that every decision involves trade-offs.

Distributed systems provide scalability, resilience, and organizational flexibility. They also introduce operational overhead, additional failure modes, and significantly more complexity.

For that reason, building a distributed system should never be the objective.

Solving a business problem should be the objective.

Sometimes that solution is a distributed architecture. Sometimes it is a well-designed monolith. The right answer depends entirely on the requirements, constraints, and growth trajectory of the system.

Final Thoughts

Distributed systems power many of the applications we use every day. They enable organizations to scale beyond the limits of a single application and support levels of reliability that modern users have come to expect.

At the same time, every service added to an architecture creates another relationship that must be understood, monitored, and maintained.

That is the hidden cost of distributed systems.

They give us the ability to scale, but they require us to manage a level of complexity that simply does not exist in smaller architectures.

And that is why understanding distributed systems is not really about learning new technologies.

It is about understanding the trade-offs that come with growth.

Why Your Tech Stack Doesn’t Matter

Wed, 06 May 2026 22:39:04 +0000

Architecting for Reliability in the Age of Multi-Agent Systems

We are currently over-indexing on "Model Orchestration."

Every week, a new library, a new vector database, or a new framework tops the GitHub trending charts.

This week it might be LangGraph. The next CrewAI. Something else right behind it.

Every week the same question shows up:

"Which stack should I use to build a reliable multi-agent system?"

It's the wrong question.

Because I've yet to see a system fail because to the wrong framework, language, or database.

I've seen them fail because they couldn't recover state, couldn't control context, and couldn't explain what they just did.

There’s a persistent belief that the logo on the documentation is the secret sauce for a production-ready system.

It isn’t. In fact, if you’re spending the majority of your time debating the stack, you’re missing the architectural patterns that actually determine whether your agents will succeed or hallucinate into oblivion.

The Illusion of the Framework

A Multi-Agent System (MAS) is not a library problem. It is a State Management problem disguised as an AI problem. Whether you use a graph-based logic or a role-based queue, the fundamental challenges and failure modes remain identical:

lost state
bloated context
untraceable decisions

The stack you choose is merely the syntax you use to solve universal engineering constraints.

The Core Thesis: Reliability in agentic workflows is derived from patterns, not packages. A secure, scalable system built in Python looks fundamentally the same as one built in Rust if the underlying system primitives are respected.

The Three Constants of Reliable Agents

Regardless of your tools, your architecture must solve for these three pillars to move from a "cool demo" to a production asset:

State is Sovereign
If an agentic loop fails at step 7 of 12, does your system restart from scratch? If so, your stack doesn't matter because your architecture is broken. A resilient system requires Deterministic Checkpointing:
- Capture the full thread state.
- Preserve intent, not just data.
- Resume execution without replaying the entire workflow.

Without this, your system is just a loop with amnesia.

The Context Tax
Context windows are not infinite. In reality, every token you give an agent is a tax on its reasoning. The "how" isn't about which LLM you use; it's about the Routing Layer:
- Classify intent
- Expose only relevant tools
- Minimize context surface area

Less context doesn’t limit the system—it sharpens it.

Governance as a First-Class Citizen
An agent is a service principal. If it cannot be audited, revoked, or sandboxed at the identity level, it shouldn't have access to your data or exist in production.

A reliable system enforces:

Least-Privilege Authorization, ensuring agents operate within a cryptographic "box" regardless of whether they are running in a Docker container or a serverless function.
Scoped tool usage
Traceable execution

Example

Consider a simple multi-agent workflow:

If your system can't resume from that point with the same context and intent, you don't have a system.

You have a demo.

A reliable system looks different.

The Framework-Agnostic Checklist

Pillar	The Real Question
Coordination	How do agents hand off work without bloating context or losing intent?
Observability	Can we trace every decision back to inputs and reasoning steps?
Resilience	What happens when a model fails mid-workflow? Can we resume without replaying?
Sovereignty	Who owns the data and execution environment—us or the platform?

Closing Thoughts

These are not new problems. They're just showing up in a new layer.

Stop chasing the framework. A system built in Python and one built in Rust will fail in exactly the same ways if the architecture is wrong.

The difference isn't the stack. It's whether you've designed for:

State
Context
Control

The tools are interchangeable. The architecture is not.

This is the foundation for the upcoming Sovereign Synapse series—where we move from theory to a local-first system that treats memory, context, and ownership as first-class concerns.

PC-0003—Building Minimal Governance Layers

Fri, 24 Apr 2026 07:46:44 +0000

Control Layer: Deterministic Governance in Structured Systems

Executive Summary

PC-0003 defines the governance layer as a formal, machine-verifiable boundary that determines which system states are admissible within a deterministic execution environment.

Within the trilogy architecture:

PC-0001 (Execution) ensures outputs are deterministic and reproducible
PC-0002 (Causality) ensures system history is fully reconstructable
PC-0003 (Governance) ensures only structurally valid states are permitted to exist

Together, these layers form a closed system of execution, history, and constraint enforcement.

A system is not defined by what it can produce, but by what it refuses to allow.

1. Governance as a Structural Boundary

Governance in PC-0003 is not policy, documentation, or interpretation.

It is a runtime-adjacent constraint system that enforces admissibility through explicit rules.

It defines:

valid inputs
valid transformations
valid resulting states

This replaces subjective validation with deterministic evaluation over structured data.

If a state violates governance constraints, it is not partially valid — it is excluded from the system entirely.

2. Deterministic Constraint Model

PC-0003 operates on a constraint satisfaction model over a graph-based state space.

Let:

G = system graph state
E = event input
C = constraint set

Governance validity is defined as:

V(G, E) = \bigwedge_{i=1}^{n} C_i(G, E)

A state transition is only admissible if all constraints evaluate to true.

Otherwise, the event is rejected and no mutation occurs.

3. Constraint-Driven System Integrity

System integrity is enforced through three structural invariants:

3.1 Input Constraints

All events must conform to a strict schema:

required identifiers
typed payloads
authenticated signatures
timestamped provenance

3.2 Transition Constraints

State mutation is strictly append-only:

no destructive edits
no retroactive modification
no implicit overwrites

3.3 Structural Constraints

The system graph must remain internally consistent:

bounded core structure
valid ontology relationships
preserved causal ordering

4. Graph Closure Model

PC-0003 operates over a dual-layer graph structure:

Core Graph

A fixed structural kernel defining system ontology:

immutable
deterministic
finite seed state

Extension Graph

An append-only event-derived structure:

fully causal
chronologically ordered
replayable

This yields:

G_total = G_core ∪ G_extension

This separation ensures stability without restricting evolution.

5. Cryptographic Authority Model

Authority is not assumed — it is verified.

All canonical events must be:

cryptographically signed
validated against trusted keys
rejected if unverifiable

This ensures system state is both structurally valid and authentically produced.

6. Failure Model and System Safety

PC-0003 prevents system degradation by eliminating ambiguous states.

Failure conditions include:

schema violations
invalid transitions
broken causal ordering
unsigned or corrupted events

In all cases:

invalid events do not modify state — they are excluded entirely.

This guarantees deterministic system behavior under all inputs.

7. Trust Stratification

PC-0003 defines a layered trust hierarchy:

Level 0 — Cryptographic Truth

Immutable external anchor (archival-grade reference)

Level 1 — Canonical Repository State

Versioned but traceable implementation state

Level 2 — Runtime Projection

Live system view for execution and visualization

Only Levels 0 and 1 define authoritative truth. Level 2 is derived.

8. Live Implementation Evidence (Proof Layer)

The governance model is implemented and observable in live systems:

Deterministic runtime and state projection layer:
https://library-catalog.streamlit.app/
Canonical source repository (ontology, runtime, and governance logic):
https://github.com/PeculiarLibrarian/the-peculiar-library/tree/main

These systems demonstrate:

deterministic graph execution
SHACL-based validation enforcement
causal event reconstruction
structured governance constraints (PC-0003 layer)

9. Ontological Registration (JSON-LD)

PC-0003 is formally registered as a first-class node in the system ontology:

{
  "@context": {
    "pc": "http://padi.system/pc#"
  },
  "@id": "pc:PC0003",
  "@type": "pc:GovernanceLayer",
  "pc:dependsOn": [
    "pc:PC0001",
    "pc:PC0002"
  ],
  "pc:role": "governance_layer",
  "pc:implementationProof": [
    "https://library-catalog.streamlit.app/",
    "https://github.com/PeculiarLibrarian/the-peculiar-library/tree/main"
  ]
}

10. Layered System Architecture

Layer	Function	Guarantee
PC-0001	Execution	Deterministic output
PC-0002	Causality	Full history reconstruction
PC-0003	Governance	Structural admissibility

Each layer constrains the one above it, forming a closed deterministic stack.

11. Closing Principle

PC-0003 formalizes governance as a structural constraint system, not an interpretive layer.

It ensures:

invalid states cannot exist
execution is deterministic
history is reconstructable
structure remains bounded and verifiable

A deterministic system is defined not by what it produces, but by the constraints that prevent invalid states from forming.

From AS/400s to AI Agent Swarms

Mon, 30 Mar 2026 20:54:15 +0000

I've spent 16 years building software — starting on IBM WebSphere Commerce platforms for major retailers, eventually leading architecture at companies like Hertz, DaVita, Chick-fil-A, Wayfair, and Kohl's, and now running a fractional CTO practice out of Austin.

I wanted to introduce myself to the CoderLegion community because I've been writing more this year, and the response has been eye-opening. My post about building a 35-agent AI coding swarm got more engagement in a week than anything I'd published in years. Turns out people want to read about what actually happens in production — not the sanitized conference-talk version, but the real decisions, trade-offs, and scars.

So that's what I write about.

What I've Been Working On

My most recent deep-dive was about the architecture behind a 6,000% throughput improvement at Hertz. The short version: their rate engine was doing 300 requests per second with p90 latency over a minute. We rebuilt it with CQRS, CDC-powered caching, and Kinesis streaming. The result was 3,000+ reads/sec at p95 under 30ms. The post walks through every architectural decision and why we made it — including the ones that didn't work the first time.

Before that, I wrote about orchestrating a swarm of 35 AI coding agents that run overnight — spawning specialized agents for different tasks, coordinating them through a hierarchical topology, and waking up to pull requests that are actually useful. It's the kind of thing that sounds like science fiction until you see the git log.

What I Build Day to Day

Right now I split my time between:

Fractional CTO work — leading platform rebuilds, defining architecture, running weekly executive checkins for clients who need senior technical leadership without a full-time hire
Enterprise commerce — about a decade of IBM WebSphere Commerce, plus Magento, Shopify, and BigCommerce across major retailers and enterprise brands
AI/ML in production — not just prototypes, but edge AI video pipelines feeding vision models on Raspberry Pis, agentic workflows that actually ship code, and real-time analysis at the edge
Distributed systems — the Hertz work, DaVita's 100+ microservices, Chick-fil-A's offline-first POS system across 3,000+ locations

What I'll Be Writing About

I've got a backlog of posts in various stages of done:

"Your Custom Cache Is a Bug Factory" — a follow-up to the Hertz piece about the caching anti-patterns I see in almost every codebase I audit
"Eventual Consistency Isn't Lazy — It's Architecture" — why most teams default to strong consistency out of fear, not necessity, and what it costs them
"Why You Probably Don't Need a Full-Time CTO" — the case for fractional technical leadership, from someone who does it

If you're interested in production architecture, real-world scaling problems, or what it actually looks like to lead engineering at companies that move millions of dollars through their software every day — I think you'll find something useful here.

I'm Mathew. Good to meet you all.

Find me at mdostal.com or connect on LinkedIn. If you're dealing with a scaling problem or making an architecture bet that could use a second opinion, I do free 30-minute calls.

Reporting and Production Make Terrible Roommates

Wed, 11 Mar 2026 14:01:18 +0000

I tend to think of reporting and production as incompatible roommates. They need the same space, they optimize for completely different things, and every accommodation one makes for the other is a debt that gets called in later. The production schema is usually what accumulates the most debt and gets hit the hardest.

Consider what happens when the analytics team asks for a denormalized order_summary view on the production database so their dashboards load faster. The DBA obliges, adds a materialized view, and now every schema migration has to account for it. Six months later the application team wants to split the orders table into orders and order_line_items. But the view is embedded in 10 dashboard queries and a nightly export job, and most of those queries also join it back to columns on orders directly. Redefining the view over the new tables is the easy part. Rewriting and revalidating a dozen queries the application team doesn't own is not, so the refactor stalls, and the production schema fossilizes around a reporting concern.

Reporting Tends to Win Every Schema Negotiation

The cause is structural. A transactional schema optimizes for write consistency, referential integrity, and the access patterns of the application that owns it. A reporting schema optimizes for read throughput, aggregation, and the access patterns of analysts and dashboards. When both share a schema, every design decision becomes a negotiation.

Once dashboards reach executives or exports reach outside consumers, reporting tends to win that negotiation, because saying no gets more expensive with every dashboard that already depends on a yes. Its requests arrive one at a time as small additions (a column, an index, a view), each cheap to accept and expensive to remove. A refactor's payoff is deferred, while a broken dashboard is visible to leadership the same morning.

Not every system needs a separation on day one. A small team with a single database, low reporting complexity, and a schema that's still fluid can query production directly without meaningful friction. But this fossilization is predictable. It emerges when reporting consumers multiply, when dashboards become load-bearing, and when schema changes require cross-team coordination.

Architects who recognize this trajectory can keep the door open for separation without building the full pipeline prematurely. That means resisting the urge to denormalize production schemas for reporting convenience, and keeping reporting access patterns from becoming implicit contracts on the production schema. When the separation does happen, it can be reactive, tapping into what the database already captures, or intentional, making the application responsible for producing reporting-quality records in the write path.

Reactive Separation Reuses What the Database Already Captures

A Dedicated Reporting Replica Isolates Load, Not Schema

The simplest place to start is to point reporting tools at a read replica of the production DB. Many teams already have replicas for distributing query load, and so dedicating one to reporting keeps analytical queries from competing with production traffic. No new infrastructure, no async pipeline, no application code changes.

  ┌─────────────┐         ┌─────────────────┐
  │ Application │────────>│  Production DB   │
  └─────────────┘  writes │  (Primary)       │
                          └────────┬─────────┘
                                   │ replication
                                   v
                          ┌─────────────────┐
                          │  Read Replica    │
                          │  (Same Schema)   │
                          └────────┬─────────┘
                                   │ direct queries
                          ┌────────┴─────────┐
                          │  BI / Dashboards  │
                          └──────────────────┘

This is a feasible fit when reporting needs are straightforward and the production schema is close enough to what reporting consumers need. It also assumes data that's a few seconds stale is acceptable. "A few seconds stale" is the optimistic case, though. Heavy analytical queries on the replica can cause replication lag to spike well beyond that, especially during peak reporting windows.

The isolation is partial, too. On a PostgreSQL standby, a long report can block the replica from applying changes. The replica then either cancels the report or, with hot_standby_feedback on, stops the primary from cleaning up dead rows. Reporting indexes land on production too, because a replica can't hold indexes of its own.

The replica breaks down when reporting needs diverge far enough from the production schema's shape. Reporting consumers write increasingly complex queries with multiple joins, or they start requesting schema changes to production to make their queries simpler, which is exactly the distortion this post is about.

The replica can't capture intermediate transitions either. Reporting queries see whatever state the database holds when they run, so if a record changes twice between one query and the next, the intermediate state is invisible.

Change Data Capture: State History Without Intent

CDC tools like Debezium tap the database's transaction log and emit changes as events. The application writes normally to whatever schema makes sense, and CDC streams those changes to a separate store. Unlike the replica approach, CDC captures every intermediate state change because it reads from the transaction log rather than querying current state.

  ┌─────────────┐         ┌─────────────────┐
  │ Application │────────>│  Production DB   │
  └─────────────┘  writes └────────┬─────────┘
                                   │ transaction log
                                   v
                          ┌─────────────────┐
                          │  CDC Connector   │
                          │  (e.g. Debezium) │
                          └────────┬─────────┘
                                   │ change events
                                   v
                          ┌─────────────────┐
                          │  Stream / Queue  │
                          │  (Kafka, Kinesis)│
                          └────────┬─────────┘
                                   │
                     ┌─────────────┴─────────────┐
                     v                           v
            ┌────────────────┐          ┌────────────────┐
            │  Transform (T) │          │  Schema        │
            │  Reshape/Join  │          │  Registry      │
            └───────┬────────┘          └────────────────┘
                    v
            ┌────────────────┐
            │ Reporting Store│
            │ (Warehouse/DL) │
            └────────────────┘

CDC's greatest strength is that it requires no application code changes and no new abstractions in the write path. That makes it often the only viable option for legacy systems where changing the write path is too risky, or for teams that need separation now and can't modify every service that writes data. It also tends to deliver complete payloads, because the log usually carries the full new row even when the application updated a single field.

The first limitation is semantic. CDC events originate from the database layer, so they capture what changed but not why it changed. A row update that represents a customer canceling an order looks identical to a row update that represents a system correcting a data entry error.

The obvious workaround is to add columns like cancelled_by and cancel_reason. They record the why only when the application itself needs them, because then they're domain state and CDC carries them. Adding them only for reporting is the distortion this post started with. And some intent has no natural row to live on at all, such as a correction or a change spanning several entities. For financial ledgers or audit-critical workflows, where intent is as important as state, the write path has to record the intent itself, which is what the outbox and event sourcing below do.

The second limitation is the absence of a contract boundary. The table structure is the contract, implicitly. When that schema changes, nothing fails at build time. The CDC pipeline either silently emits differently shaped events or breaks at runtime, and reporting consumers discover the problem in production rather than in development.

Two mitigations narrow that gap without closing it. A transformation layer that maps raw tables onto a stable reporting model, such as warehouse staging models, shrinks the repair after a schema change to one place. But the reporting side owns that layer, so it learns of production changes only after they ship. A schema registry can partially close this gap by rejecting incompatible schema versions when the pipeline tries to register them, but that's added infrastructure catching incompatibility at runtime rather than at build time.

The third limitation is database dependency. CDC is only as good as the log the database exposes. PostgreSQL's logical decoding, MySQL's binlog, and DynamoDB Streams are mature options. But a managed service that restricts log access, or a stream that retains changes only briefly (DynamoDB Streams keeps 24 hours), can push teams toward application-layer alternatives earlier than expected.

Reading the log also loads the primary. Logical decoding adds write-ahead log volume to every write. A PostgreSQL replication slot also holds that log on the primary until the connector consumes it, so a stalled CDC pipeline can fill the production database's disk.

Intentional Separation Captures Intent in the Write Path

Reactive approaches separate the workload but not the context. They capture what changed, not who changed it or why. That context exists at the application layer when the write happens, and it's lost the moment the data hits the database unless someone deliberately captures it.

The Outbox Pattern Makes the Contract Explicit

The outbox pattern makes the application responsible for producing reporting-quality records. Instead of letting the database schema define the downstream contract implicitly, the application writes a versioned record to an outbox table within the same database transaction as the domain state change. Either both commit or neither does. A separate process reads from the outbox and projects into whatever reporting store analytics needs. That relay can crash after publishing and before marking a record published, so delivery is at least once, and consumers deduplicate on the event ID. The application controls the payload shape, the versioning, and the context included in each record.

  ┌─────────────┐
  │ Application │
  └──────┬──────┘
         │ single transaction
         v
  ┌──────────────────────────────────────┐
  │          Production DB               │
  │                                      │
  │  ┌──────────────┐  ┌──────────────┐  │
  │  │ Domain Table  │  │ Outbox Table │  │
  │  │ (orders,      │  │ (versioned   │  │
  │  │  customers)   │  │  records)    │  │
  │  └──────────────┘  └──────┬───────┘  │
  └───────────────────────────┼──────────┘
                              │ poll / stream
                              v
                     ┌─────────────────┐
                     │ Relay Process   │
                     │ (reads outbox)  │
                     └────────┬────────┘
                              │ publish
                              v
                     ┌─────────────────┐
                     │ Reporting Store │
                     └─────────────────┘

Outbox table via a relational database:

CREATE TABLE outbox (
    id              BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    event_id        UUID          NOT NULL DEFAULT gen_random_uuid(),  -- globally unique, used downstream
    aggregate_type  VARCHAR(100)  NOT NULL,  -- e.g. 'Order', 'Customer'
    aggregate_id    VARCHAR(100)  NOT NULL,  -- e.g. order ID
    event_type      VARCHAR(100)  NOT NULL,  -- e.g. 'OrderCancelled'
    schema_version  INT           NOT NULL,  -- contract versioning
    occurred_at     TIMESTAMPTZ   NOT NULL DEFAULT now(),
    initiated_by    VARCHAR(200),            -- who: user ID, system name
    reason          VARCHAR(500),            -- why: 'customer_request', 'admin_override'
    payload         JSONB         NOT NULL,  -- full state snapshot + context
    published       BOOLEAN       NOT NULL DEFAULT FALSE
);

Outbox record via a Kinesis/Kafka stream (JSON envelope):

{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "aggregateType": "Order",
  "aggregateId": "ORD-20260218-4417",
  "eventType": "OrderCancelled",
  "schemaVersion": 2,
  "occurredAt": "2026-02-18T14:32:08.771Z",
  "initiatedBy": "user:jsmith",
  "reason": "customer_request",
  "payload": {
    "orderId": "ORD-20260218-4417",
    "customerId": "CUST-8821",
    "previousStatus": "Confirmed",
    "newStatus": "Cancelled",
    "lineItems": 3,
    "totalAmount": 284.50,
    "currency": "USD"
  }
}

The outbox has an explicit, versionable contract boundary. A breaking change to the outbox record is a deliberate code change that goes through review. If a developer renames a column in the production schema, the outbox record doesn't change unless someone deliberately updates it. A contract test in CI fails the build when the record's shape changes, before any dashboard breaks. Unlike a test on CDC tables, it lives in code its owners already test. The outbox table itself is a reporting concern in the production database, but domain tables never read or join it. Changing a domain table changes only the code that maps it into the record.

Because a polling relay doesn't rely on transaction log capabilities or vendor-specific change feed APIs, any database that supports transactions supports the pattern. A relay built on CDC, such as Debezium's outbox event router, inherits CDC's operational risks but not its implicit contract.

The outbox does not require a record for every database write. It only fires when a specific entity type has a meaningful state change, and only for the types that reporting cares about. Background jobs updating internal timestamps produce nothing. A deletion does produce a record, because knowing that an entity was removed and who removed it is itself meaningful. This keeps the coupling concentrated in write paths that produce meaningful state transitions, not spread across every query and update in the codebase.

That selectivity is also the outbox's main risk. A bulk admin script or hand-written SQL fix that changes an order's status without writing to the outbox leaves reporting silently incomplete. CDC can't miss a change that way, because it reads every committed change. Emitting the record from the domain model's state transition closes most of that gap, and out-of-band SQL against tracked tables has to write its own outbox record.

The outbox is my recommendation for teams that have outgrown a read replica, need reporting to know who acted and why, and don't need full event sourcing.

CQRS and Event Sourcing Keep Every Transition at the Highest Cost

CQRS (Command Query Responsibility Segregation) formally separates the write model from the read model. The write side accepts commands and persists state. The read side maintains whatever views consumers need. The two sides don't have to share a schema, or even storage. What CQRS adds is the explicit acknowledgment that changing state and reading it are different jobs that can deserve different models. CQRS does not require event sourcing. Its write side can persist state normally in a traditional database.

Event sourcing takes this further by changing what the write side stores. Instead of persisting current state and producing reporting records alongside it, every state mutation is recorded as an immutable event, and current state is derived by replaying those events. The event log becomes the source of truth, not the current snapshot. Every transition is preserved in the order it occurred. Production state is a projection of the event stream, and so is reporting state, and so is any other view you need. If the analytics team changes their requirements six months from now, you replay the same events through a new projection and the full history is there.

                          Commands
                              │
                              v
                     ┌─────────────────┐
                     │   Write Side    │
                     │ (Command Handler│
                     │  + Aggregates)  │
                     └────────┬────────┘
                              │ append events
                              v
                     ┌─────────────────┐
                     │  Event Store    │
                     │  (append-only)  │
                     └────────┬────────┘
                              │ project
                              v
                     ┌─────────────────┐
                     │  Projected      │
                     │  State Tables   │
                     │  (per entity)   │
                     └────────┬────────┘
                              │ query (read side)
              ┌───────────────┼───────────────┐
              v               v               v
     ┌────────────┐  ┌─────────────┐  ┌────────────────┐
     │  Prod API  │  │  Reporting  │  │  Audit /       │
     │  Queries   │  │  Store      │  │  Compliance    │
     └────────────┘  └─────────────┘  └────────────────┘

Event store document (append-only, NoSQL):

{
  "streamId": "Order-ORD-4417",
  "position": 4,
  "eventType": "OrderCancelled",
  "occurredAt": "2026-02-18T14:32:07Z",
  "payload": {
    "initiatedBy": "user:jsmith",
    "reason": "customer_request",
    "previousStatus": "Confirmed",
    "newStatus": "Cancelled",
    "lineItems": 3,
    "totalAmount": 284.50,
    "currency": "USD"
  },
  "metadata": {
    "correlationId": "req-88a1c",
    "causationId": "cmd-cancel-4417",
    "userId": "user:jsmith"
  }
}

Projected state table (derived from events, used by reporting/ETL):

CREATE TABLE order_projections (
    order_id         VARCHAR(100) PRIMARY KEY,
    customer_id      VARCHAR(100) NOT NULL,
    current_status   VARCHAR(50)  NOT NULL,
    item_count       INT          NOT NULL,
    total_amount     DECIMAL(12,2) NOT NULL,
    currency         VARCHAR(3)   NOT NULL,
    placed_at        TIMESTAMPTZ,
    shipped_at       TIMESTAMPTZ,
    cancelled_at     TIMESTAMPTZ,
    cancelled_by     VARCHAR(200),
    cancel_reason    VARCHAR(500),
    last_event_pos   INT          NOT NULL,  -- tracks replay position
    projected_at     TIMESTAMPTZ  NOT NULL DEFAULT now()
);

The projected table above is what reporting reads in practice. Reporting consumers rarely subscribe to the event stream directly, which keeps the stream internal to the domain. A current-state projection like order_projections keeps only the latest cancellation. When reporting needs history, the domain team that owns the projections builds a history-shaped one, such as a status-transition table.

This is a good fit for domains where the complete history of state transitions is genuinely valuable, like financial ledgers, audit-critical workflows, or systems where "undo" and "replay" are first-class requirements.

Most teams should not reach for this combination. Martin Fowler's CQRS article warns that most of the CQRS cases he has run into have not gone well. Many systems fit a CRUD mental model and should stay that way. CQRS should only apply to specific bounded contexts where the read and write access patterns are genuinely different, not across entire applications. Event sourcing compounds the cost. Events are immutable and permanent, so schema design requires careful thought. Aggregate replay gets expensive without snapshotting, and debugging production issues means reasoning about event sequences rather than inspecting current state.

Choosing an Approach by What Reporting Needs to Know

Approach	Use when	Key limitation	Complexity
Read replica	Schema shape is close to what reporting needs; slight staleness is acceptable	Replication lag under analytical load; no intermediate state history	Low
CDC (e.g. Debezium)	Write path can't be modified; full row history is needed	Captures state, not business intent; table structure is an implicit contract	Medium
Outbox pattern	Business context (who/why) matters; write path can be modified	A write path that skips the outbox silently drops data; relay process and versioning discipline needed	Medium
CQRS + Event Sourcing	Audit-critical domain; "undo" and "replay" are first-class requirements	Events are permanent; aggregate replay is expensive without snapshotting	High

Separate Early or Pay Later

Separating early means separating the access path before dashboards depend on base tables, not building a pipeline. The first request for a reporting-only column or view on production is the cue. Every shortcut that ties these workloads together makes the eventual separation harder. The analytics team and the application team end up negotiating every schema change, and the production schema accumulates the concerns of whoever complained most recently.

A read replica is enough to start, but only if dashboards query plain views rather than base tables. Those views live in a reporting schema the application team owns and can redefine, with grants that reach nothing else. Migrations still update the views, but now the team running the migrations owns them. When those views get too slow or too costly to redefine, move past the replica, to CDC if reporting needs only what changed and to an outbox if it needs why.

Whichever step comes next, reporting reads an owned surface rather than raw tables, whether that's views on the replica, a transformation layer on CDC, or an outbox record. That owned surface keeps the schema from freezing, and the separate store keeps reporting load off production.

Separation doesn't remove the coupling, but it gives the coupling an owner. With an outbox, the application team owns the record and its versions, and the analytics team owns the projection built from it. New questions about who or why still reach the application team, but as a reviewed change to a versioned record, and older records won't carry the new context. A production schema change that leaves the record's version intact ships without asking the analytics team.

Observability Is Authored, Not Installed

Mon, 16 Feb 2026 19:01:55 +0000

I have been a part of a dev team where poor observability constantly brought us to a standstill. Not because the tooling was missing, but because the data it collected never carried meaningful context. Alerts fired constantly, so operations teams ignored them. Dashboards existed for every service, but none of them answered the questions that mattered during incidents. Investigations that should have taken minutes took hours. It got bad enough that slow diagnosis alone stretched incidents into significant SLA violations.

We questioned the choice of platforms, dashboards, and alerting rules. Yet none of those could help, because no tool could report a distinction our code never recorded. The problem was upstream, since our code didn't know the difference between "I handled this correctly" and "something is actually broken."

The Classification Problem: Handled Outcomes Logged as Errors

Consider a payment processing system. A customer's card gets declined for insufficient funds. The payment gateway returns a rejection, and the system logs it as an ERROR.

But this is the system working correctly. Insufficient funds is a handled business case, not an exception. Because it's logged as an error, though, it shows up in error dashboards, triggers error-rate alerts, and adds to the ambient noise that operators learn to tune out.

Over time, "payment errors" become background radiation. The team knows most of them are just declined cards, so they stop investigating. Then the gateway starts timing out, or a partner pushes a breaking change, and the actual problem gets buried. Nobody notices because "payment errors are always high."

The usual response is to blame the team for ignoring alerts. It is a discipline problem, yes, but the discipline that's missing is upstream, in the code that treats expected outcomes as errors.

The fix belongs in that code, which should sort each outcome by what it means:

Expected success: The happy path. Logged at DEBUG if at all.
Expected failure: Business logic correctly rejecting something, like declined payments, validation failures on input from outside callers, or rejecting a caller over its quota. This is INFO, not ERROR.
Degraded but functional: The system recovered, but something is wearing thin. Retries running past their usual count, or a dependency throttling your calls. This is WARN. It isn't broken yet, but it needs attention before it becomes broken.
Unexpected failure: Something genuinely went wrong that demands investigation. This is the only category that should be ERROR. A call that fails past its retries belongs here even against a dependency known to be flaky, because no code path handled it. If that becomes routine, the fix is a handled path like a circuit breaker or fallback, which moves it to degraded.

Typed results make this distinction structural instead of incidental. When expected failures are returned as typed results rather than thrown as exceptions, the classification is baked into the code's structure. A declined payment returns a result; a gateway timeout that exhausts its retries throws an exception. A logging boundary still has to map each result type to a log level, but typed results put that decision in one visible, testable place instead of leaving it to guesswork at every call site.

The log level is only one way to carry that decision. A structured outcome field or a counter works as well, as long as the code sets it. For conditions that only show in aggregate, like routine retry counts, response times approaching SLA thresholds, or connection pools running hot, a metric carries the decision better than a WARN line per request.

When classification is right, every downstream tool benefits. Dashboards that track error rates become genuine health indicators because errors represent actual unexpected failures, not business logic working as designed. Log queries become surgical because structured errors with proper context let you filter to a specific tenant or operation in minutes. Once thresholds and routing are set well, alerts become actionable, because every error they count demands investigation.

When classification is wrong, the opposite happens. Alerts fire for expected outcomes, so operators learn to ignore them. Dashboards become decoration because nobody trusts what the numbers represent. Every investigation becomes archaeology because the data that should answer your questions is buried under noise.

Only the Code Can Make the Call

The usual alternative is to classify downstream, but no downstream tool can classify outcomes the code didn't make distinguishable. Each option runs into that limit in its own way.

A collector rule can change the level of decline lines in one central place. But it matches message text, so it breaks silently when the message changes, and it never sees lines that sampling already dropped.
Filters by logger category or exception type quiet framework and library noise without matching text. But they can't split a decline from a breakage inside the same operation.
An SLO holds the classification rule in one shared place. But it can only count a field the code emits.

Status codes look like a field the code emits, and auto-instrumentation can tell a 402 decline from a 504 timeout. But a status code carries the meaning only when the code puts it there. A decline wrapped in a 200 response body reaches the platform with the wrong label. A partner's breaking schema change gets the wrong label too, because it arrives as a 400 that looks just like a validation failure.

Only the code knows the difference, and it knows because someone designed it to. A partner integration shows what that design looks like. If the code validates each request against its own rules before sending, any schema or contract 400 the partner returns afterward is unexpected by definition. Sometimes the 400 reveals a rule your validator missed rather than a partner change. ERROR is still right, because the gap is a bug in your code, and fixing it removes that noise. Rejections based on the partner's own state, like a closed account, are different. They're expected outcomes, so map the partner's error codes for them explicitly.

Business Judgment Belongs in Counters, Not Log Levels

When the system correctly declines a card for insufficient funds, it's tempting to log that as WARN because you want the metric reviewed often. But a correctly handled decline is the system working as designed, not degrading. Whether the decline rate is "concerning" is a business question that changes with strategy and context, so log levels shouldn't encode that judgment. Leave business interpretation to reports and dashboards where it can evolve, not to code where it gets baked in and forgotten.

Declines still deserve watching, just not through the log level. Feed those reports from a decline counter rather than from INFO lines, because many teams sample INFO logs and a counter keeps every decline. The counter also catches breakage that looks like declines. A sudden spike in correctly handled declines can mean a bug sending wrong amounts or a card-testing attack, so the same counter can carry an anomaly alert routed to engineering, separate from the error rate.

Context Is Authored, Not Accumulated

The instinct is to compensate with volume: write verbose logs everywhere so you'll have context when you need it. But a trace-level log is not a dump file. Nearly every bug I've seen diagnosed from trace logs involved information the code already had at the point of failure and could have put in the error or warning itself. In those cases the problem wasn't insufficient logging volume; it was that nobody authored the context where it mattered.

A stronger version of the volume argument comes from Observability Engineering, by Charity Majors, Liz Fong-Jones, and George Miranda. It records one wide structured event per request with dozens of attributes, so you can ask questions afterward that nobody predicted. That approach handles unknowns far better than trace dumps, but it doesn't escape authoring. The event still needs an outcome field that says whether a decline was expected or broken, and domain fields like the tenant and the gateway's response code only exist if the code puts them there. Without them, a high-cardinality store is a faster way to search through the same unclassified noise.

Failures should carry their own context. When an operation fails, the error log should include what was being attempted, what went wrong, and enough identifying information to correlate it. Consider a checkout failure. Instead of logging "payment processing failed," a properly authored error captures the operation ("charge_payment"), the correlation ID, the gateway's response code, and the amount. That's structured enough to query and specific enough to reproduce the failure without re-running production traffic.

What solves most domain-logic bugs is understanding what the user did and what they sent, not tracing the code's internal flow. Connecting what they did across services takes a correlation key, and carrying that key is a solved problem. The W3C Trace Context standard defines how it travels between services, and most structured logging libraries can stamp it on every entry. With that key and errors authored like the checkout one, you have what you need to reproduce the problem.

Reproducing a failure from its input, rather than tracing it, borrows the idea behind event sourcing, where a state can be rebuilt by replaying the events that produced it. For deterministic domain logic, you don't need to trace every intermediate step if you can reconstruct the scenario from the input, the outcome, and the state the operation read.

The state the operation read is the part the request can't supply. A race, or a record that changed after the failure, can't be replayed from the input alone. So the error should carry the versions of the records it loaded. A version only helps while the old record still exists. If the store overwrites records in place, carry the field values the decision used instead.

Most of the context a failure needs can be chosen in advance, because the facts that decide an outcome are ones the code already handles. They are the operation, its input, the response from whatever it called, and the versions of the records it read. Environment facts like the build version, config, feature flags, and host belong there too, and the logging setup can attach those to every entry without per-call code. Some deciding facts won't be predictable. Those show up as incidents where someone had to add logging to find the cause.

What gets logged must be intentional. You know the domain, so you know the potential inputs, what's valid, and what's sensitive. That knowledge lets you author a safe context, enough to reproduce the problem without exposing data that shouldn't be in a log. An allowlist of input fields, hashed identifiers, or a reference to a stored payload can carry the input safely. If you don't understand the domain well enough to make that distinction, that's the source of the problem, not the logging infrastructure.

The difference between a useful error and a useless one is whether someone authored the context intentionally or hoped that raw volume would cover it.

Installed Tooling Reports Mechanics, Not Meaning

Classification and context both come from the code, but some observability really is installed. A platform catches transport failures on its own, like a timeout on an outbound call, and span timings and profilers expose creeping latency without any per-call code. Trace logging, toggled on temporarily for one flow, still diagnoses ordering bugs where the sequence itself is the evidence.

What they can't supply is meaning. They see that a call failed or slowed down, but not whether a rejection was the system working as designed, which fields an error needs, or whether an answer was wrong. Wrong output that never raises an error only surfaces in business-outcome metrics, and those are authored too, because someone has to decide which outcome to count. That's the sense in which observability is authored. The installed part tells you something failed, and only the authored part tells you whether it matters and why.

The Black Box Test: Diagnose From Outputs Alone

The word observability comes from control theory, where Rudolf Kálmán defined it in 1960. A system is observable when its internal state can be inferred from its external outputs. The practical test applies that definition. When something breaks, can you diagnose it from the system's outputs alone? Or do you need to add logging, redeploy, and wait for it to happen again? If the answer is the latter, your code doesn't explain itself yet.

Software never passes completely, so run the test as a rate. For each recent incident, ask whether the first responder could identify the failing operation, its input, and the cause from existing telemetry. Score it from the timeline rather than memory. An incident fails if anyone added logging before the cause was found, whether by redeploy, a live logpoint, or a trace toggle. That holds even when toggling trace logging was the right move for an ordering bug. The failing share should fall over several quarters. With only a few incidents a quarter, read the trend over a year.

Attaching a Debugger Hides the Gaps

Classification and context are design decisions, but many developers never test whether their logging actually answers the questions it needs to. One habit that contributes is the debugger. Stepping through code is the right tool for a logic bug in development, but the habit hurts when it becomes the only way surprises get explained. A developer who resolves every local surprise by stepping through code can get little practice diagnosing from outputs, so gaps in those outputs are easy to miss until production.

Some organizations extend this habit into production with remote debugging, but that's a security liability. A debugger attached to a running container, or to any production process, can read its memory, including secrets and customer data. It can also change what it executes, so every open debug port is another way into production.

Snapshot debuggers capture variables without pausing, so they're gentler, and redaction and audit can contain their exposure. But they still keep the habit of reaching into the process instead of fixing what it outputs. You should be observing system outputs, not reading a live process's memory.

Production should be a black box. If your default instinct when something breaks in production is to attach to it rather than read the outputs, you'll rarely feel the pressure to make those outputs useful. The classification stays sloppy, the context stays thin, and the errors stay vague. Not because you don't know better, but because you've never needed better.

Developers who diagnose from observable behavior, whether testing locally against containerized dependencies or against remote systems, tend to build the discipline. They feel the pain of vague errors and missing context firsthand, and they fix it at the source because they have no easier option.

Owners Who Get Paged Close the Gaps

That discipline holds best when builders own what they operate. You don't log payment declines as errors when you're the one who gets paged for "high error rate on payment service." You don't dump verbose logs instead of authoring context when you're the one parsing them at 3 AM.

Paging pressure only fixes code when it reaches the code. A paged team can just as easily raise thresholds or mute alerts. What can turn the pain into a source fix is bringing the authors into incident diagnosis and filing every misclassified outcome or context-free error found there as a bug against the owning team.

Tracking open telemetry bugs per owning team, beside the black box test's failing share, gives each team a number that backlog triage can see. Filing against the owning team matters most when the author sits on another team, because that author would otherwise never feel what their classification and context choices cause downstream.

Better tooling alone won't make your code explain itself. Running the black box test against it, and filing what it finds against the code that caused it, is how it starts to.

How Shared Libraries Become Shared Shackles

Tue, 03 Feb 2026 16:35:47 +0000

This is a highly opinionated take on shared libraries and the damage they do to team autonomy and development tempo. Teams deliver value faster and more consistently when they can make decisions, ship changes, and evolve their domains without coordinating across organizational boundaries. Shared libraries erode exactly that independence.

After seeing costs explode for trivial tasks and critical production updates failing to deliver on time in nearly every organization I have witnessed, I am willing to take a rather "extreme" stance on the subject.

This post focuses on distributed architectures, where the consequences are most severe.

Shared Libraries Violate Core Principles

Distributing components isn't just about distributing work. It's about the Single Responsibility Principle applied at the system level: clear ownership, implementation isolation, and infrastructural independence. The share-nothing principle makes this explicit. Services should be autonomous, independently deployable, and free from implementation coupling. DORA's research on loosely coupled teams links that independence to delivery performance, finding that teams who can change and deploy without depending on other teams deploy more often and recover faster.

Shared libraries violate these principles. They couple teams through shared implementation despite being distributed in name, creating little monoliths that bind development tempo across teams that were meant to operate independently.

Yet the pitch keeps coming: "We have this code in five places. Let's consolidate it into a shared library. We'll save time, ensure consistency, and make everyone's life easier." It sounds reasonable, it really does, yet the decision only calculates the cost of duplication while ignoring the cost of sharing across teams, domains, and technical boundaries.

The Costs the Pitch Doesn't Count

Version conflicts and upgrade pain

Five teams now depend on your library. They release on different cadences and at some point one or more teams require a breaking change. Now you're either maintaining multiple versions indefinitely or forcing upgrades on teams that have other priorities. The "one place to maintain" becomes "one place that blocks everyone."

Teams blocked waiting for changes

A team needs functionality the library doesn't have, and a wrapper in their own code can't supply it. They can't just add it. They need to coordinate with the library owners, get the change approved, wait for a release, and then upgrade. An afternoon's change in a team's own code can stretch into weeks of waiting.

Debugging across boundaries

When something breaks, the investigation now spans your code and the library code. Your team doesn't own the library, yet it runs inside your service, so the investigation is still yours. The abstraction that was supposed to simplify your team's work has added a layer to dig through.

Bloat or fragmentation, pick your poison

The library starts focused. Then another team needs something slightly different. Then another. The library accumulates features to serve multiple masters, becoming a grab-bag of loosely related functionality coupled together because they share a package, not because they belong together. The disciplined alternative is to split it into many small, focused packages, but that creates its own problem: an entourage of dependencies that each consuming team must track, version, and coordinate with whenever they change.

Obscured accountability

Shared libraries don't reduce your quality burden; they move it somewhere less visible. If the library has a bug, your service has a bug. The library just adds a dependency you don't own and can't fix on your own schedule.

Polyglot stacks multiply the library

The shared library pitch assumes a homogeneous technology landscape that many organizations don't have. If your organization has services in C#, Java, Python, and Go, do you keep four versions of every shared library in sync? In polyglot environments, the "shared" library becomes a second-class citizen in every language except the one the authoring team uses. Whatever consistency the library buys stops at the language boundary, so you need another way to stay consistent anyway.

Release discipline softens these costs without removing them

The usual escapes each leave part of the cost in place:

Pinning an old version only delays the choice until a fix a team needs ships past a breaking change, and backporting that fix is the multiple-version maintenance again.
Automated update tools raise pull requests for breaking upgrades as well, but each consuming team still changes its own code to take one.
Outside contributions shorten the owners' queue, but the change still needs their review and a release.
Forking or patching locally puts the team back to copying the code.

The Cohesion and Coupling Diagnosis

If two services genuinely need the same function, you have three possibilities:

It's a cohesion problem. That function belongs in one place and should be called, not duplicated. Extract it into a service with an API. Now there's a clear owner, a clear contract, and no shared implementation coupling consumers together.

It's a coupling problem. You've drawn your boundaries wrong. The services that "need" the same code are more related than you thought. Reconsider where the boundary belongs rather than papering over the boundary violation with a shared dependency.

It's genuinely independent. The similarity is coincidental. Both services need to format dates, parse JSON, or validate email addresses, each a few lines of glue around the runtime. Copy the code. Move on. Each copy can evolve with its service.

Every answer has a price

A service adds a network hop, an availability dependency, and an API to version, and new features still wait on its owner. In exchange, the owner changes the internals without any consumer moving, and only a breaking contract change asks callers to act.

Copying has limits too. The copies can drift, and a copied bug gets fixed once per copy. For small code that rarely changes, that's still cheaper than the coordination. Copies also don't show up in a dependency scanner, so what you copy should be the glue around the runtime or a mature library, not a hand-rolled parser or validator.

Some code can't be copied at all. A rule that must produce the same answer everywhere, like a tax calculation, would let two copies disagree in a way a user or the data could see. That's the cohesion case, and it needs one running implementation. A shared library deployed at different versions in different services disagrees just as two copies would.

Fixing a bug once is a boundary signal

The common rebuttal is "but if there's a bug, I fix it once and it propagates everywhere." The answer depends on where the bug lives. Bugs in third-party libraries get fixed upstream on the vendor's cycle, and you take the fix by upgrading. Logging is another common candidate for sharing, but what a service logs and traces has no single fix to propagate. Each service authors its logs and telemetry out of its own domain knowledge, aligned to shared values about how behavior gets classified and what context gets captured.

What remains is mostly business logic. If your business logic is so coupled across services that a single bug requires simultaneous fixes everywhere, you don't have a sharing problem, you have a boundary problem. So the default is no, and the burden of proof sits with the library to show it will stay narrow and stable.

Using mature, well-tested libraries for universal problems makes sense. Logging frameworks, HTTP clients, serialization libraries, and authentication middleware exist because these problems are universal and well-understood. Someone else solved them better than you would, and the cost of depending on their solution is low because the solution is stable.

Sharing your internal CustomerDto across services is different. Sharing your "standard" repository pattern is different. Sharing your domain models between bounded contexts is different. These aren't universal problems with stable solutions. They're your internal abstractions, and forcing them on other teams assumes those teams should think the same way you do.

The dividing line is stability, not who wrote the code. An internal library kept narrow, versioned, and slow to break behaves like an external one. Internal libraries that carry domain or team-specific abstractions tend not to stay that way, because those abstractions change whenever your domain does.

SDKs Are Different

An SDK abstracts what you expose: the public contract of a service or platform. A good SDK earns its existence by encoding integration complexity that would be expensive and error-prone for every consumer to reimplement: orchestrating multi-step workflows, managing state across API calls, handling idempotency, and abstracting version differences.

A shared library abstracts how you think internally, like your domain models, patterns, and "standard way" of doing things. The shared library serves a governance impulse, not the consumer. And unlike an SDK, it tries to couple internal teams to the same release cycle and the same implementation decisions.

The SDK says: "Here's how to use our thing."
The shared library says: "Here's how you should build your thing."

One is a service to consumers. The other is an imposition on autonomous teams disguised as help.

Your Runtime Already Solved This

The shared library pitch often targets "utility code" that your runtime already provides. If you're using .NET, the framework gives you HTTP clients, JSON serialization, logging abstractions, dependency injection, and configuration management. Why would you need an internal shared library wrapping HttpClient when HttpClient exists and is battle-tested by millions of applications?

Your wrapper just adds coordination overhead on top of something that didn't need wrapping.

The Principle Is Broader Than Distribution

The same problem exists in a modular monolith wherever different teams own different domains. There, shared packages between domains still couple teams to the same change cycles. The blast radius is contained, though, because teams share a deployable and version conflicts surface as build errors.

In a distributed system, a change that would have been a merge conflict becomes a multi-team coordination effort with blocked releases and stale dependencies. Topology changes the severity, not the coupling. If Domain A and Domain B in separate services share domain code, they're a distributed monolith with extra steps.

Two practices look like exceptions to this coupling, but each accepts the cost rather than escaping it. The first is the monorepo that enforces a single version of every dependency, which Google's Software Engineering at Google calls the One-Version Rule. Version skew can't happen, but a breaking change to the library has to update every consumer before it lands. The coordination moves to whoever changes the library. Google absorbs it with large-scale-change tooling most organizations lack.

The second is domain-driven design's Shared Kernel. Eric Evans conditions it on neither team changing it without consulting the other, which accepts the coordination cost on purpose.

The API Client Library Obsession

The most common incarnation of shared library dysfunction is the API client package: a library of hand-written contracts, DTOs, and client code that consumers are expected to import when calling your service. I have never seen this pattern result in anything short of chaos.

The pitch sounds reasonable: "We'll publish a client library so consumers don't have to write their own HTTP calls or define their own contracts." But this solves a problem that doesn't exist while creating several that do.

Every API should have documentation describing its contracts. If your API is well-documented with clear schemas, consumers can generate or write their own clients. The documentation is the contract, and a schema check in each consumer's CI catches drift from it. A machine-readable schema such as an OpenAPI document or a .proto file is that documentation in another form. A stubs package generated from the schema, with no hand-written code, is too. The trouble starts when the package carries hand-written DTOs and runtime behavior, binding every consumer to the producer's model and policies.

Every consumer has different needs. Service A might need three fields from one endpoint. Service B might need ten fields from a different endpoint. Service C might need to call the same endpoint but transform the response differently. When you force everyone to use your client library, you're imposing your view of how your API should be consumed.

Client libraries impose one team's operational policy on every consumer. Teams building client libraries inevitably add caching strategies, retry policies, circuit breakers, and connection pooling configurations. Those decisions belong to the calling service, which is the only one that knows its own latency budget, its failure tolerance, and how much damage a stale read does in its domain. The producer knows none of that. A client library freezes those choices upstream, where changing them requires a coordinated release across every consumer.

The producer still has a part to play, but on its own side of the boundary. It protects itself server-side with rate limits and publishes which operations are safe to retry, its backoff contract, and how its idempotency keys and pagination tokens work. The caller honors that contract through its own resilience library.

The absurdity becomes obvious with frontend consumers. The iOS app doesn't need a backend team's Swift package to call its API. The iOS team reads the documentation, or generates a client from the published schema itself, and maps responses to whatever structures suit the application.

This reflexive reach for client libraries has been conditioned by years of cargo-culting. Patterns that made sense in public cloud SDKs with complex auth flows got carried into internal services with straightforward REST endpoints, where they don't. When an internal API needs multi-step orchestration to call correctly, that logic belongs behind the API, where the producer controls it, not in a package every consumer has to upgrade.

The Governance Theater Problem

Shared libraries often emerge from a governance impulse: "Teams are doing things inconsistently. We need to standardize."

The instinct isn't wrong, and consistency matters. But a mandated shared library is governance theater.

If teams are building things inconsistently, ask why. Usually it's because they don't share the same understanding of what matters, what the tradeoffs are, and what "good" looks like. That's an alignment problem. It requires conversation, documentation, and shared values.

Forcing everyone to use the same library doesn't create alignment. It creates compliance.

Governance through values: "Here's why we authenticate this way, here are the tradeoffs, here's what we're optimizing for. Align your implementation to these principles."

Governance through code: "Use this library or you're non-compliant."

The first preserves autonomy. Teams understand the principles and can make good decisions in novel situations. The second creates coupling while providing the illusion of alignment. Teams can comply without understanding, and when they hit a situation the library doesn't cover, they have no principle to fall back on.

Governing through values doesn't mean leaving them unenforced. Conformance tests, linters, and reviews against the stated principles keep alignment from decaying. They gate builds rather than ship inside services, so when a rule tightens, each team fixes its own code without waiting on another team's release, and the rule can start as a warning.

An optional "paved road" library that teams can leave still fits, because the values come first and the library is one documented way to follow them. It turns into theater when using the library becomes the compliance check.

The Exception: Thin, Stable Code Like Security Protocols

There's one domain where shared libraries routinely make sense: security protocols like ingress handling, service-to-service authentication, and encryption standards.

Why security is different:

The domain is stable and well-understood. Authentication patterns don't change week to week.
The cost of getting it wrong is catastrophic. Security isn't a place for teams to make independent decisions and learn from mistakes.
The surface area is thin and focused. A good security library does one thing.
Autonomy isn't the goal. You actually want teams to do security the same way. The coupling is a feature, not a bug.

The four conditions earn the exception, not the fact that it's security. They are the burden of proof the default puts on any shared library. In-process code that meets them earns it too, such as redacting personal data before logs leave a service, or a thin package that configures OpenTelemetry identically everywhere without deciding what a service logs.

Uniformity alone doesn't earn the exception. Code that must behave identically everywhere but fails the conditions belongs in an external library or a service instead. Trace-context propagation must match across every service, yet it needs no internal library, because OpenTelemetry's libraries already implement the W3C Trace Context standard.

Even here, mature external libraries or a service mesh should do the protocol work, and the internal library should be the thinnest layer that applies your organization's policy. Thinness keeps its forced upgrades rare. When a vulnerability does force one, uniform and immediate rollout is the point. The moment it starts accumulating "helpful" utilities beyond its core purpose, it's sliding toward the problems that plague other shared libraries.

What to Do Instead

When you feel the urge to create a shared library, pause and diagnose the problem:

If it's a capability multiple services need: Build a service and expose an API, unless the logic is stable enough to version like an external library.
If the services need the same logic because they share a domain: Redraw the boundary. The shared code is a symptom of a split in the wrong place.
If it's a pattern you want to standardize: Write documentation. Explain the principles, the tradeoffs, and the reasoning. Let teams implement the pattern in their own codebases. They'll understand it better than if they'd just imported your abstraction.
If it's truly just duplicated code: Let it be duplicated. For small code that rarely changes, the coordination cost of sharing exceeds the maintenance cost of duplication.
If it's a security primitive, or code that meets the same four conditions: Fine. Build the library. Keep it minimal, stable, and focused. Recognize it's a necessary evil, not a model to emulate.

The shared library is a solution to a problem that rarely exists in the form people imagine. Duplicating small, stable code isn't what slows teams down. Coordination overhead is.

Share values, and the shared library more often becomes unnecessary.

Package Updates Are Investments, Not Hygiene Tasks

Mon, 12 Jan 2026 15:11:13 +0000

It is time to update a third-party package in your repository, or at least to consider it. So how do you know what is safe, what is needed, what is prudent, and what will keep our company from melting down in record time? To address these questions, many teams fall back on a reflex: always update immediately to "stay current," ignore updates entirely until forced, or let a bot merge whatever passes CI.

All three approaches treat package updates like chores, something to batch process or avoid. But package updates are investments. They consume time and introduce risk, which means they deserve the same deliberate evaluation you'd apply to any other technical decision. An investment decision isn't always a big one. For most updates it is a few minutes of triage asking whether this update earns its place now and what it could break, and only major versions and hot-path dependencies deserve a full business case.

The Distributed Systems Uniformity Trap

In distributed systems, a curious assumption often takes hold: all services must run the same package versions to maintain debuggability and behavioral consistency. And teams lacking clear governance see version alignment as a proxy for unity and control.

This assumption fails on multiple fronts. Distributed systems with shared-nothing architectures don't gain meaningful debugging benefits from version uniformity. If Service A runs a library on v2.1 and Service B runs it on v2.3 behind their own contracts, a failure in either traces back to that service. Its own logs and stack traces place it there, even when the symptoms spread to its callers.

Spread does carry a cost when a CVE lands, because each version in the fleet may need its own remediation path. The answer is a floor, not a single version. Every service, shared platform tool, and internal template stays above a supported baseline, which bounds the CVE cost without making services move in lockstep.

Version uniformity does matter in specific contexts:

Shared libraries and contracts: When services share a common library that defines data contracts or communication protocols, mismatched versions can cause subtle serialization bugs or contract violations
Security vulnerabilities: When a CVE affects multiple services, the floor rises to the fixed version and every service has to move above it, so attackers can't exploit the weakest link
Cross-cutting libraries: Tracing and context propagation, correlation-ID middleware, token validation, and retry policies act across service boundaries, so drift there breaks traces or changes retry behavior between callers
Framework-level breaking changes: When a platform upgrade (like .NET major versions) requires coordinated migration across services

Outside of these cases, enforcing uniformity wastes time and introduces unnecessary risk. Knowing which dependencies matter for coordination and which don't beats version number theater. When coordination does matter, focus on the boundaries: version your APIs explicitly, pin shared contract libraries, and establish migration windows rather than demanding instant synchronization across all services.

Match Scrutiny to the Kind of Update

Before updating any dependency, evaluate the change type and context. The Semantic Versioning specification provides a starting framework, but not all maintainers follow it rigorously, and even those who do sometimes misjudge what constitutes a breaking change. Read the changelog, not just the version number.

Patch updates (x.y.Z) carry security fixes and bug patches, and most should go through with your normal test run. A patch to a hot-path dependency still gets that dependency's load test or canary, because a patch-level bump can change runtime behavior.

A security fix that itself carries regression risk, such as a fix to a hot-path dependency or a release with fresh regression reports, is the one patch you might hold. Even then, the default is to patch. Hold it only when a reachability tool or a search shows the affected API isn't referenced directly or transitively, so the vulnerable code never runs. Then a few days' wait while the release proves itself may carry less risk than updating.

Minor updates (x.Y.z) require evaluating value against risk. New features and non-breaking changes matter only if they solve problems you have or deliver performance improvements that affect your workload. Check how long the release has been out and whether issues are being filed against that version. A release under a week old, or one with fresh regression reports, deserves skepticism. Let early adopters find the edge cases first, unless the release fixes an actively exploited vulnerability your code can reach, in which case the exposure outweighs the soak time.

Major updates (X.y.z) demand a business case. Breaking changes consume significant engineering time for migration, testing, and bug fixes. The value must justify the investment. Ask what capabilities become available, what technical debt gets resolved, and what risk comes from delaying (losing vendor support, missing future security patches). Treat major updates as planned initiatives with dedicated time and clear success criteria, not as squeezed-in tasks during feature development.

For any update, walk through these core questions:

Problem definition: What specific problem does this solve (security, feature, bug, performance, vendor requirement)? A routine patch needs no special reason, but if a minor or major update has no clear problem to solve, question it.
Research: Review changelogs for breaking changes, deprecations, and known issues. Check security scan results. Monitor community feedback (GitHub issues, forums, Stack Overflow).
Testing: Scope tests to what the update touches (see the testing section below), and ensure you have a rollback plan.
Rollout: Test in a canary environment first if possible. For distributed systems, roll out incrementally (one service at a time). Define who monitors the rollout and what metrics matter.

This framework doesn't guarantee perfection, and not every update needs every step. Canaries and staged rollouts are for hot-path dependencies and major versions. But it does at least encourage deliberate thinking and decisions instead of reflexive action.

Delay Has Its Own Costs

Delaying updates indefinitely creates different risks:

Security exposure: Unpatched vulnerabilities accumulate, and attackers target known CVEs in outdated packages
Vendor abandonment: Falling too far behind loses access to vendor support and community knowledge
Compounding migration cost: The longer you wait, the larger the gap between current and target versions, making eventual migration more painful
Ecosystem drift: New libraries and tools may assume newer dependency versions, limiting your options

"No clear problem" is a reason to wait, not a reason to stay put forever. Hold a regular review, quarterly or semi-annually. At each one, treat the gap itself as the problem once any of these holds:

The version you run is nearing the end of vendor support.
The gap is large enough that the next security fix would force a multi-version jump.
The dependency is blocking another upgrade you need.

Routine patches shouldn't wait for that review. They flow through as they arrive, in small batches that stay easy to attribute. The review catches whatever was held back, so the gap never grows large enough to need a migration project.

If you're already multiple versions behind, don't try to catch up all at once. Audit your dependencies, identify the high-risk gaps (unpatched CVEs, unsupported versions, libraries blocking other upgrades), and create a prioritized update roadmap. Chip away at it like technical debt rather than attempting a big-bang migration.

Test Based on What Changed

Fear pushes some teams toward retesting everything by hand, and convenience pushes others toward merging whatever is green. The first wastes time and the second misses the risks a dependency change actually carries.

Target your testing based on what changed:

Regression tests: Focus on code paths that use the updated dependency directly or indirectly. For a dependency on every path, such as an HTTP client, that is the whole automated suite, which is cheap to run. The waste is in hand-testing features the change never touches
Load tests: When the dependency sits on a hot path or manages connections, threads, or credentials, replicate production traffic patterns against the specific features that changed, then validate SLA compliance (response times, throughput, error rates)
Integration tests: If the dependency handles I/O (databases, APIs, file systems), test those boundaries thoroughly

Functional tests with serial requests can pass cleanly while hiding race conditions, deadlocks, or resource exhaustion that only manifest under production concurrency.

Trusted Vendors Still Ship Breaking Changes: The AWS SDK for .NET V4

Trusting a vendor doesn't make its updates safe.

In April 2025, AWS released version 4 of the AWS SDK for .NET with a long list of breaking changes, documented in its V4 migration guide. Some of them compile cleanly and fail only at runtime. Collection properties on request and response objects now default to null instead of an empty collection, so a loop over a response list that worked in V3 can throw a NullReferenceException in V4. A team that treated AWS as a trusted source and updated without reading that guide would meet these errors only at runtime, and only on the code paths its tests happened to exercise.

The harder change to catch wasn't a breaking API. In October 2025, the core package moved from 4.0.0.31 to 4.0.0.32. One team then reported in the SDK's GitHub issue 4053 that the upgrade would "grind our service to a halt" under concurrent load, with no exceptions in the logs. The reporter suspected locking around credential retrieval that let only one request at a time fetch signing credentials. (The issue uses the assembly numbering, 4.0.31.0 and 4.0.32.0.) The 4.0.0.32 changelog did say credential refresh during the expiry window was being reverted, but not that refreshes would now block callers.

The maintainer's reply explains that the change was deliberate: AWS had reverted background credential refresh because other users were getting expired credentials back, which is worse. So the same update fixed a correctness bug for some teams and collapsed throughput for others. A team hitting expired credentials had a clear reason to take it, and a team without that problem had none. Either way, a service with this change looks healthy in development and early testing, where requests arrive one at a time, and stalls under production traffic.

Two Truths From the AWS SDK Incident

Upfront due diligence has limits: You can review changelogs, run regression tests, and validate functionality, but some bugs only surface under production conditions
Ongoing vigilance matters: Staying plugged into ticket systems, community forums, and issue trackers helps you catch problems before they spread

Common Objections

Triage Is Cheap, and It Moves the Time to Office Hours

"We don't have time for this level of due diligence. Just like TDD, it sounds good in theory but slows us down in practice."

For most patch and minor updates, the framework comes down to reading a changelog, scanning the issue tracker, and running the tests that cover the changed paths. Load tests and staged rollouts are for the few dependencies on the hot path, not every update. Reading the V4 migration guide in the afternoon beats discovering a null collection in production at 2 AM. None of this argues for updating rarely. Small, frequent updates are easier to attribute when something breaks, and the framework works at that cadence. What it rules out is a change reaching production without anyone deciding it should.

Deliberate updates consume predictable, scheduled time during normal work hours. Autopilot updates consume unpredictable, high-stress time during incidents. Even if the hours came out equal, one approach happens during office hours with full context, while the other happens during outages with incomplete information. Reviewed updates still fail, as the AWS example above shows, but they fail with someone already watching the metrics they chose.

Blanket Patch Deadlines Treat Every CVE as Critical

"Our security team requires us to apply all patches within 48 hours of release. We don't have a choice."

Security policies that mandate blanket timelines without risk assessment can create more risk than they prevent. A policy that forces teams to apply an untested patch faster than they can validate it treats all vulnerabilities as equally critical, which they aren't.

When possible, present data to security leadership. Show the difference between a critical remote code execution vulnerability in your authentication layer (apply immediately) versus a theoretical XSS vulnerability in a library function your codebase never calls (evaluate deliberately). Security teams tend to be more open to adjusting a policy when the request comes with risk-based reasoning rather than a complaint about the deadline. Many already tier deadlines by severity, or by whether a CVE appears in CISA's Known Exploited Vulnerabilities catalog. A tiered policy is this framework applied to exposure. The framework adds the other side, the patch's own regression risk, which decides how much validation fits inside the deadline.

If your organization won't budge, at least apply the intentional framework to prioritize which updates get thorough validation versus rubber-stamp approval. Not every patch deserves the same scrutiny.

Automation Can't Make the Value Call

"Automated tooling already handles this for us."

Automation can find updates and flag CVEs, but it can't decide whether an update makes sense for your context.

Most security scanners tell you a vulnerability exists in a package you depend on. Even the ones that add reachability analysis can't tell you whether the risk outweighs the cost of the update for your system.

The strongest version of this objection is a gated pipeline. Renovate's minimumReleaseAge setting holds a new version back until it has been public for a set time, CI runs the suite, and a canary watches production metrics before the rollout widens. That pipeline encodes several steps of the framework above, and for low-risk dependencies it can be the whole process. What it can't encode is the value judgment. It will happily open the major-version PR that needs a business case, and it only catches the regressions its gates measure.

The AWS SDK throughput regression arrived in a fourth-segment version bump, exactly the kind of update auto-merge rules tend to wave through, and a suite of serial functional tests passes it cleanly. Only a load test or a canary that measures throughput would have stopped it. A team that had tiered the SDK as hot-path would read that changelog line as a reason to load-test, and a team that hadn't would scroll past it.

A person has to decide that a credential library on every request deserves a load test. That decision is made once, when you tier the dependency, and every later update to it inherits the gate. Most of the judgment this post asks for lives in that tiering and in the value call on majors. Within the low-risk tier, the gates can carry routine updates with no human minutes at all.

But for hot-path dependencies, auto-merging without those gates is worse than no automation. It trades a known risk (the version you run and its published issues) for an unknown one (bugs you didn't test for, shipped without anyone deciding to ship them). For hot-path dependencies and major versions, automation should notify, not decide.

Concentrate Review Where the Risk Is

"We have too many dependencies to evaluate each one individually. We'd spend all day reviewing changelogs."

Not all dependencies deserve equal attention. Apply the Pareto principle as a rule of thumb rather than a measured ratio. A small share of your dependencies carries most of your risk, such as authentication libraries, database drivers, core frameworks, and HTTP clients. They sit on every request, so a bug in one reaches all of your traffic. Focus your evaluation effort there.

Risk also comes from exposure, so a markdown parser that renders user input belongs in the first group even if it runs on one page.

For dependencies that are neither on the hot path nor handling untrusted input (date formatting helpers, color palette utilities, test fixture builders), batch review them on a short, regular cadence. Check for breaking changes and security issues in aggregate, test once across the batch, and apply together. A batch makes a regression harder to attribute, but bisecting a batch of formatting utilities is cheap, which is why hot-path dependencies stay out of it.

One risk ignores these tiers. A compromised release can do damage from any dependency, because it runs with your build's or service's full permissions. A release-age hold like Renovate's is cheap enough to apply to every tier, though it only catches a compromise someone discovers before the hold ends. The backdoor planted in xz-utils in 2024 sat in public releases for about a month before anyone noticed.

Fast Teams Stay Fast by Avoiding Self-Inflicted Incidents

"Our competitors ship faster because they don't overthink updates like this."

You don't know what your competitors do internally. You see their marketing velocity, not their operational reality. The review this post asks of most updates is minutes of triage, so the comparison turns on incidents, not review time. Shipping fast for a quarter is easy, and staying fast for years means avoiding the context-switching of constant firefighting. Every unforced error from an unreviewed update is time spent recovering from a self-inflicted wound instead of shipping.

Leadership Sets the Tone

Team leads determine how their teams approach updates. If leadership treats updates as chores to rush through, teams tend to cut corners. If leadership asks hard questions, prioritizes based on value, and accepts that "not yet" is sometimes the right answer, teams tend to follow their example.

Shipping fast and thinking deliberately aren't opposites. Triage on most updates takes minutes, and the incidents it prevents arrive at the worst possible time, as mysterious production issues traced back to an unconsidered dependency change two sprints ago.

Package updates are investment decisions, not hygiene tasks. Give each one the scrutiny its risk deserves, and decide on purpose.

Adaptability Over Cleverness: What Makes Code Actually Good

Mon, 05 Jan 2026 15:54:14 +0000

Systems that stay cheap to change for years aren't the ones built to fit today's requirements perfectly. They're the ones that bend without breaking when requirements shift, technologies evolve, and teams discover what they didn't know upfront. Building for change beats chasing premature perfection.

You will rarely get it right the first time. That's not a failure; it's how software development works. Requirements clarify through building, edge cases emerge through usage, and performance issues surface under real load. Teams that treat first attempts as gospel can spend months polishing solutions to the wrong problem.

Instead, build systems that can evolve. Give yourself room to deliver, learn, and adapt as reality proves what matters and what doesn't. Adaptable code is code where a likely change touches few places, can be made safely by someone other than its author, and shows quickly when it breaks something. Readability and simplicity matter for the same reason, since a change starts with someone understanding the code.

Correctness and speed aren't the only requirements code has to meet. Security, operability, and run cost are requirements too. Adaptability decides how much it costs to keep meeting all of them as the system changes, and that cost includes reading the code to diagnose an incident or pass an audit.

The exception is code that will never change or be read again, like a one-off migration script. That code only has to be correct.

Cleverness and Speed Make Change Expensive

Brian Kernighan and P. J. Plauger's The Elements of Programming Style put this in two maxims: "Write clearly – don't be too clever" and "Write clearly – don't sacrifice clarity for efficiency." Both describe the same cost. Clever code packs several decisions into one construct, like one cached value that quietly serves both a pricing rule and a fraud check. To change one of those policies, the next developer has to unpack both and work out which behavior the author intended and which is incidental.

Optimized code carries the same cost in a different form. An optimization usually rests on an assumption about today's workload, such as the data fitting in memory, records arriving sorted, or one query pattern dominating. The code around it comes to depend on that assumption too, so when the workload changes, the change has to reach every place that relied on it. Donald Knuth's 1974 paper "Structured Programming with go to Statements" argued that programmers' attempts at efficiency in noncritical code "have a strong negative impact when debugging and maintenance are considered." It's the same paper that calls premature optimization the root of all evil.

None of this argues against performance. When speed is a requirement, measure where it's needed and put the optimization behind a boundary, so the rest of the system doesn't depend on how it works. The fast path then becomes one more thing that can change in one place. Some optimizations can't hide, because they shape the contract itself, like a bulk API in place of per-item calls, a streaming response in place of a full list, or a denormalized table every report reads. Those are decided deliberately up front.

Adaptability Comes From Keeping Change Local

Adaptable code isn't magic. It follows principles that reduce coupling, isolate change, and make breakage obvious.

Boundaries Keep Each Change in One Place

Single Responsibility Principle is about change, not tasks. Robert C. Martin states it as "Each software module should have one and only one reason to change." Whether that module is a microservice owning a bounded context or a class handling a single concern, a shifted requirement lands in the piece responsible for it without cascading edits across the system. At service scale, a wrongly drawn boundary turns into a contract change between teams. That's why service boundaries deserve more care up front than class boundaries do.

Clean interfaces and separation of concerns apply at both macro and micro levels. Services communicate through contracts, not implementation details. Business logic doesn't know about HTTP; database layers don't make authorization decisions. Abstractions placed where implementations are likely to change let you pivot as understanding evolves without rewriting everything upstream.

Externalize what changes by making variable behavior configurable. Environment-specific settings keep the same code deployable to dev, staging, and production. Tunable timeouts, batch sizes, and retry policies let operations adjust the system's behavior without a code change.

Externalize only what is likely to vary, because each setting is flexibility with a carrying cost. It adds combinations to test and can drift between environments, so it needs the same likely-to-change test. A bad value can also take a system down, so validate configuration at startup like any other input.

Fast Failures and Tests Make Breakage Visible

Fail fast and loud surfaces problems immediately. Silent failures cascade into confusing bugs far from their source. Explicit validation at system boundaries, defensive assertions in critical paths, and structured logging create clear signals when something breaks.

Tests that give you confidence let you refactor without guessing what you broke. Unit tests verify component behavior. Integration tests catch interface mismatches. End-to-end tests confirm critical workflows still work. Tests help most when they check behavior through a component's public boundary, because a test coupled to internal details becomes one more place every change has to touch.

Consistency Shows Where a Change Belongs

Consistent naming and structure reduce cognitive load. Developers understand the codebase faster when patterns repeat. Services follow the same lifecycle. Features share the same folder layout and the same error-handling convention. When the structure is predictable, a developer can tell where a change belongs and what else it touches before opening a file, which is what lets someone other than the original author make it safely.

Trace one requirement through these habits. Suppose refunds above a threshold now need a manager's approval. If one module owns refunds, the rule lands there and nowhere else. If the same approval rule also governs discounts and credits, it changes for reasons refunds don't share. By the same one-reason-to-change test, it belongs in an approvals module that refunds call. A developer who didn't write that module finds it because every feature follows the same layout, and a test that issues a refund through the module's public interface fails if the approval check is missing.

If refund logic were instead copied into three controllers, the same rule means three edits, and a fourth copy nobody remembered keeps approving refunds without a word.

Flexibility Belongs Where Change Is Likely

The goal isn't maximum flexibility; it's appropriate flexibility. Rarely getting the design right the first time doesn't rule out appropriate flexibility. You can't predict what a requirement will become, but you can often predict where requirements will move. David Parnas made this the criterion for drawing module boundaries in his 1972 paper "On the Criteria To Be Used in Decomposing Systems into Modules." The designer starts from the design decisions that are difficult or likely to change and gives each module one of them to hide from the others.

Domain Knowledge and Change History Locate the Seams

Domain knowledge tells you where those decisions are, and version control history checks the guess. Files with high commit churn show where change has actually landed so far. Churn comes in two patterns that mean different things. A file that changes for many different tickets marks a place where requirements move, so an interface belongs there. A class and its tests changing together is expected, but files in separate modules that always change together for the same reason mark the opposite problem, a boundary drawn in the wrong place. Before the first release there is no history, and domain knowledge alone decides.

A payments system will need to support new payment providers, so the provider belongs behind an interface from the first release. Shape that interface around the domain's operations, like authorize, capture, and refund, rather than around the first vendor's API. That domain-shaped interface is a seam. It will still change when the second provider arrives with redirects, asynchronous confirmations, or a single charge step. But callers already depend on payment operations the domain owns rather than on one vendor's types, so the reshape is one contract change instead of a hunt for vendor types across the codebase.

Domain knowledge also shows where change is unlikely. An internal admin tool probably won't need a plugin architecture.

Decisions that other systems or stored data depend on follow a different rule. A published API contract, a persisted data format, or an event schema can't be fixed with a refactor later, because changing it means migrating data and coordinating every consumer. These deserve the same up-front care as the payment provider even without a change history. Service boundaries and the optimizations that shape a contract, like the bulk API earlier, belong in this group.

Contain a Dependency Until Its Shape Is Clear

Inside the codebase, a seam isn't the only option. When change is likely but its shape isn't clear, or when change arrives somewhere you kept simple, contain the dependency and add the abstraction later. Tests and consistent structure, the habits from the first half of this post, let you add the boundary when the second case actually shows up. That costs one refactor, instead of years maintaining an abstraction that nothing used.

That price stays low only if the concrete dependency is used from one place. Containing it takes no abstraction layer, just a rule that every call goes through one module or wrapper you own, and that callers never see vendor types or vendor error codes. Make the rule mechanical, such as keeping the vendor package a dependency of that one project, so a hurried direct call fails the build.

The wrapper differs from a seam in what it promises, not in whether it uses an interface. The wrapper offers only the operations today's callers use. A seam commits to the operations the domain needs from every implementation, and that commitment is the part that goes wrong when you guess. Either way, the later change replaces one file rather than forty call sites.

The tiebreaker follows from this. Add the seam up front when change is likely and you know the operations it must support, the way authorize, capture, and refund are known for payments even when each provider's mechanics differ. When the shape isn't clear yet, contain the dependency and wait for the second case to show it. Before adding a layer of flexibility, check:

Has this part of the system changed before, or does the domain make change routine, the way new providers are routine for payments?
If you wait, what would the change cost? When tests cover the code and the change would stay inside one module, adding the boundary later is cheap.
Who pays for the abstraction in the meantime? Every developer who reads the code pays for each layer of indirection, whether or not the change ever comes.

Perfect code written for yesterday's requirements fails when reality shifts. Over-engineered code taxes every change with layers nobody needed. Adaptable code finds the balance: flexible where change is likely, simple where it isn't.

Why Configuration Files Don't Belong With Your Code

Tue, 16 Dec 2025 17:25:52 +0000

When you first create that new shiny code project, your configuration requirements seem so obvious and straightforward. You think: "I have a single behavior or feature that just needs this one setting or I am just calling this one external resource which just needs this one API URL". Often, your needs are indeed simple, but simple does not translate directly to easy. And even before the future flood of settings and complex use cases arrive, that setting beside the code is already a problem. The local approach feels intuitive because config stays close to the code that uses it, but real business use cases, daily team dynamics, production deployment pipelines, and distributed architectures are just not that simple.

This localized intuition leads to predictable problems: secrets leak into Git history and persist indefinitely, environment-specific values multiply into templating systems that nobody fully understands, and configuration changes require full application redeployments. These aren't edge cases or signs of poor discipline. They follow from where the config lives. A secret in a file beside the code is one mistaken commit away from Git history, and a value shipped with the code changes only when the code ships.

The Hidden Costs of Config-in-Code

Secrets leak despite good intentions. Teams start with .gitignore files and environment variables, but secrets find their way into version control anyway. A developer copies a working config to test something, a merge conflict gets resolved wrong, or someone commits from a branch that predates the .gitignore update. GitGuardian's State of Secrets Sprawl 2025 report counted 23.8 million secrets leaked in public GitHub commits in 2024, a count that covers hardcoded source as well as config files. Once a secret hits Git history, every clone and fork carries it. GitHub's own guidance on removing sensitive data says to revoke or rotate the secret first, because rewriting history can't reach copies already pulled. Rotating means tracking down every service that uses it.

Environment parity erodes. Each environment needs different database hosts, API endpoints, feature flags, and connection pool sizes. Teams respond with templating systems that substitute values at build time, or multiple config files with naming conventions like config.dev.json and config.prod.json, or environment variables that multiply until nobody remembers what SERVICE_ENDPOINT_OVERRIDE_V2 was supposed to do. When values are substituted at build time, staging and production don't even run the same image. Debugging then requires understanding not just the code but the entire config transformation pipeline.

Runtime selection, like ASP.NET Core's appsettings.{Environment}.json with environment-variable overrides, fixes the image problem but not the tie to releases. It keeps one image, but production values still ship in every build and change only with a release. With several override layers, the running value depends on which layer applied last.

Deployments couple to configuration. When config lives with code, changing a timeout value means rebuilding and redeploying the application. When five services share a database connection string, each holds its own copy in its own repository. Changing it means five pull requests and five deployments, ordered so no service still uses the old credential when it's revoked.

Configuration fragments across components instead of cohering around domains. In a multi-tenant system, a tenant's feature flags, rate limits, and credentials end up copied into every service that handles its requests, so one change for that tenant means editing each copy.

Config drift becomes invisible. A value gets changed manually in production to fix an urgent issue and never makes it back to staging, or a developer updates the dev config but not the template. Over time, debugging requires archaeology rather than analysis.

Homegrown Config Stores Recreate the Bootstrap Problem

The natural engineering instinct is to solve this problem yourself by storing configs in a database or building a config API.

Config kept in the application's own database creates a circular dependency: your service needs the database connection string to start, but the connection string is in the database. You end up hardcoding bootstrap credentials anyway, which defeats the purpose.

A custom config API sounds cleaner until you realize the config service itself needs configuration, deployment sequencing, and high availability. If the config API validates values against schemas in its own code, a new payment setting means deploying the config API first, then the payment service. You've traded one problem for a more complex one.

The fundamental issue is bootstrapping: how does the application know where to find its configuration before it has any configuration? Cloud-native config stores solve this by using identity that's already present in the runtime environment, such as IAM roles in AWS, managed identities in Azure, or service accounts in GCP. The application authenticates using credentials the platform provides automatically, which means no bootstrap secrets to manage. A homegrown store can authenticate the same way, but at that point you're rebuilding, operating, and keeping highly available what the managed store already provides.

What External Config Stores Provide

AWS has Parameter Store and Secrets Manager, Azure has Key Vault and App Configuration, GCP has Secret Manager, and HashiCorp Vault works across all of them.

Sensitive data never touches the codebase because it never needs to. Credentials live in the config store, encrypted at rest, with access controlled through the same IAM policies that govern your other cloud resources. When someone leaves the team, you revoke their access in one place rather than hoping they don't have local copies of config files. Audit trails show who accessed what and when, which proves invaluable when security asks "who has seen the production database password in the last 90 days?"

Applications become portable because the same artifact runs everywhere. The application asks "what environment am I in?" at startup and fetches the appropriate config, eliminating templating systems and environment-specific builds.

Organize Config Around Domains, Not Components

When configuration lives with code, it naturally organizes around components: the payment API has its config, the notification service has its config, and the reporting batch job has its config. This feels logical until you realize that configuration often cuts across components. A domain like "payments" might span three APIs, two background processors, and a gateway configuration.

With 50 APIs and 100 tenants, changing one tenant's settings starts with working out which services handle that tenant. When the payment domain gets restructured into separate authorization and settlement services, all the tenant configs need redistribution.

Domain-oriented config inverts this relationship. The same tenant configuration applies to whichever services handle that tenant's requests. When the payment domain splits, the new authorization and settlement services both read the same /production/tenants/<tenant>/payments path, so no tenant config moves and only their read permissions change.

The tradeoff is that shared domain keys become a contract across their consumers, so like a shared schema they evolve additively. Before changing a key, check the IAM read grants on its path, which list those consumers. Sharing a domain doesn't mean sharing every secret, though, because credentials can still sit in per-consumer sub-paths.

Component-specific configuration still exists, but only for component-specific concerns: performance tuning, networking behavior, resource limits, internal timeouts. Domain config answers "what does this tenant need?" while component config answers "how does this service behave?"

Hierarchical Config in Practice

External config stores enable domain-oriented organization through hierarchy. AWS Parameter Store shows the pattern, and other platforms work similarly.

Parameters follow a path structure like /production/tenants/acme-corp/feature-flags or /production/domains/payments/integration-credentials. The hierarchy carries access and ownership. Developers can read /dev/* while /production/* stays restricted to deployment pipelines. A change to /production/shared/* reaches every service in an environment on its next refresh or restart. Domain teams own their domain's subtree while platform teams manage cross-cutting infrastructure config.

The cost is small in practice. According to AWS Systems Manager's pricing page, standard-tier Parameter Store parameters (up to 10,000 per account and region) carry no storage charge and no API charge at default throughput. Secrets Manager costs more, at $0.40 per secret per month plus API calls, so a few hundred secrets run about $100 a month.

Integration with compute services removes friction. ECS task definitions can reference parameters directly, and the platform injects the values at startup. Lambda functions can read them through AWS's Parameters and Secrets Lambda Extension. Kubernetes pods can receive them as native secrets through the External Secrets Operator.

Config Changes Get Heavier, and They Should

A distributed config store removes the friction that adds nothing, like rebuilding artifacts and redeploying every consumer of a shared value. It keeps the friction that protects production. Configuration updates feel heavier because they are heavier, more like database migrations than editing a text file. You'll write change requests, get approvals, coordinate timing with deployments, and verify rollback procedures work.

Configuration changes in production systems should be intentional and controlled because a misconfigured timeout can cascade into an outage and a wrong feature flag can expose unfinished functionality to customers. Google's SRE book reports that roughly 70% of outages are due to changes in a live system, and a config change is exactly that kind of change. The ceremony forces you to think about backwards compatibility (will existing instances handle this new format?), rollback procedures (how do we revert if this breaks something?), and dependency ordering (which services need to restart first?).

Backwards compatibility matters more than it did, because config and code now roll back independently. A new key has to be additive, and the code has to tolerate its absence. It's the same expand-then-contract discipline a schema migration uses.

Approval alone isn't enough, because a change to /production/shared/* reaches every service. Changes with that blast radius should roll out in stages the way code does. AWS AppConfig, for example, first checks a change with validators against a JSON schema or a Lambda function. Its deployment strategies then release the change to a growing percentage of targets over a bake time and roll it back automatically when a CloudWatch alarm fires.

The one exception is the kill switch, because an incident can't wait for a change request. One can live under its own path, writable by on-call through a narrow IAM role.

What Still Belongs in Local Config

The goal isn't to externalize everything since that would create unnecessary dependencies and slow down development.

Application defaults like log formats, default retry counts, or internal timeout values that don't change between environments MIGHT belong in code. These define how the application behaves, not how it connects to external systems. I would still hesitate before coupling these to your code, however, because a constant default can become the value someone must change during an incident.

Framework configuration for HTTP server settings, thread pool sizes, or middleware ordering rarely varies by environment. When it does, you can override specific values from the external store while keeping sensible defaults local.

Development shortcuts that let engineers run the application locally without network dependencies make sense for rapid iteration. The key is ensuring the code path to fetch external config exists and works.

The decision rule is straightforward:

Externalize it if it's sensitive, varies by environment, or is shared across services.
Keep it local if it describes internal application behavior that stays constant everywhere.

This rule sits close to the line The Twelve-Factor App draws, where config is "everything that is likely to vary between deploys" and internal config that doesn't vary "is best done in the code." The difference is where the values come from. Environment variables can stay the interface, as when ECS injects parameters into them. But a flat set of variables managed per deployment has no hierarchy to assign ownership by path and no version history, so a versioned store makes the better source.

For prototypes and throwaway code, do whatever gets you moving fastest, but establish the external config discipline before real users or sensitive data enter the picture.

Where a Store Beats a Config Repository

An external store versions the values themselves, per environment, and its audit trail records who changed each one.

A separate config repository is the other way to keep config out of the code, and it versions config too. A GitOps pipeline injects it at deploy time as mounted files or a ConfigMap. That also keeps one image, and it adds reviewed diffs and atomic multi-key commits. For non-secret values that one service owns and changes only on release, it is a sound home. The store wins where the repository can't follow:

Secrets, where even encrypted-in-repo tools like SOPS leave old ciphertext in history for anyone who once held the key, while the store audits every read and rotates without a commit
Shared values, where one write replaces a config deploy to every consumer
Incident-time changes like a kill switch, which can't wait for a deploy
Per-path permissions, where a repository exposes everything in it to anyone who can read it

Versioning in the store also changes rollback. Parameter Store versions each parameter separately, so you roll back one value at a time. When several values must revert together, AWS AppConfig versions them as one configuration profile. Bad deploy? Roll back the config without touching application code. Services that poll for changes pick up the old value within their refresh interval, and services that read config at startup need a restart, not a rebuild and redeploy.

A writable store moves drift risk rather than removing it. The drift described earlier becomes detectable, because parallel paths like /staging/... and /production/... can be compared for missing or extra keys. It becomes preventable once write access to /production/* is limited to the deployment pipeline. Where a path must stay writable, like the kill switch, drift becomes attributable, because every write lands in the audit trail with the caller's identity.

Preventing Configs from Creeping Back into Code

Moving to external config stores doesn't automatically stop developers from committing secrets. Without active governance, you'll find credentials in code reviews months after "completing" the migration.

Code review culture matters more than tooling once the store exists. With no config file left for a secret to slip into, what remains takes judgment. Team values over process! Scanners catch values that look like secrets, while a reviewer catches the hardcoded endpoint or tenant flag no pattern recognizes. After the migration the rule is simple: any credential or environment-specific value in the repository is wrong. Make it part of your definition of done: "configuration fetched from external store, no hardcoded environment-specific values."

Pre-commit hooks can run secret scanners that check each commit for patterns that look like secrets, including API keys, connection strings, and private keys. Teams should be able to govern themselves, but for high risk products this might be needed.

Repository templates ensure new projects start with proper .gitignore files excluding .env, config.local.*, and secrets.*.

CI/CD validation catches anything that slips past local checks. Pipelines can scan for hardcoded values, verify that config references resolve to external stores, and fail builds that contain suspicious patterns. GitHub's push protection goes further, rejecting a recognized secret before it reaches the remote.

Practical Concerns and How to Handle Them

"What happens when the config store is unavailable?" A managed config store is still a dependency, and the more likely problem is throttling, not an outage. AWS documents Parameter Store's default throughput as 40 reads per second shared across an account and region, so a fleet scaling out or an ECS deployment replacing many tasks at once can hit ThrottlingException at startup. Cache config after the first fetch, spread reads across startup, enable the paid higher-throughput setting when the fleet needs it, and serve stale cached values during brief outages.

A new instance has no cache, though, so a throttle or outage during scale-out blocks exactly the instances you need. A local fallback copy could cover that gap, but a copy holding secrets must be encrypted, and getting its key to the instance is the bootstrap problem again. So most services should accept the startup dependency and size throughput for scale-out.

"Startup is slower now." True. Each fetch is a network round trip, usually small next to connection pool and dependency injection setup. Fetch at startup and cache with a refresh interval rather than forever, so a rotated credential still reaches running services.

"Local development gets harder." It gets different, not harder. SDKs and credential chains make fetching config from external stores transparent once each developer (and CI) has a scoped cloud role. The default is reading from a dev path, with each developer's writes confined to a personal sub-path. Local overrides are the shortcut for offline or isolated work. The one-time setup pays for itself in fewer "works locally, fails in production" bugs.

"We can't work offline." How often do you actually develop offline? Pulling dependencies, accessing Jira, talking to teammates, running CI/CD, and pushing code all require connectivity. When it does matter, keep a local-only setup that needs no store, such as a .NET Aspire app host that defines local resources and configuration in source control.

The Daily Experience Changes

Testing becomes more realistic because engineers can run local code against the non-secret values in remote staging or dev configs, with read access scoped to those paths, rather than maintaining local approximations that drift from reality.

Committed config files stop being a path for leaked credentials, and when developers never get read access to /production, production secrets stop passing through people at all. When you need to rotate a credential, you update one place. With a dual-credential scheme like Secrets Manager's alternating-users rotation, the old credential stays valid while every service picks up the new one on its next fetch or restart. The question "did we rotate that key in all environments?" finally has a definitive answer.

Making the Shift

Local config files feel simple and convenient right up until they aren't. External config stores trade familiar convenience for operational discipline, and that trade-off is sound.

Start with secrets by moving credentials out of the codebase and into a config store. Once that discipline is established, expand to environment-specific values, and eventually the pattern becomes natural: config comes from the environment, not from files.

Auth Sessions Should Never Cross Internal Boundaries

Tue, 02 Dec 2025 15:31:54 +0000

It is tempting to treat user authentication sessions as ambient context. You pass them through API layers, into service calls, across event processors, and through background jobs. If a component needs to know who the user is, it needs the user's session, right?

This conflates authentication with context. You're treating an external trust artifact as internal state.

The result is internal services coupled to the external auth mechanism, a replayable credential spread across them, and systems that break when work runs after the session ends or without one.

Sessions Validate at Boundaries, Context Flows Internally

What Crosses the Boundary

"Session" here means whatever the client presents to prove who the user is, whether that's a cookie bound to a server-side session or a bearer access token such as a JWT. The arguments below apply to both, and where they differ, the post says which one it means.

Authentication sessions serve one purpose: validating identity at a security boundary. Once validated, the session should be consumed and replaced with explicit context and service credentials.

A user session arrives at the boundary: an API endpoint, gateway, or edge function. The boundary validates the session and confirms identity. The session is then consumed. You extract only the necessary context: user_id, tenant_id, correlation_id, a non-secret session id, and, when a client acts for the user, the client's identity and the scopes the user granted it. From that point forward, internal operations use service roles. Components operate with service credentials like database connections, API keys, and queue permissions. User context is passed explicitly as structured data, not as the original session token.

The granted scopes in that context matter. A third-party app that holds only read access to a user's orders must not gain everything the user can do once its token is converted at the edge. So downstream authorization allows an action only when the user's current permissions and the client's granted scopes both permit it.

The Boundary Signs the Context

Passing context explicitly doesn't mean leaving it unprotected. A downstream service can't accept a bare user_id from anyone who sends one, or any caller could claim to be any user. The context has to arrive in a form the receiver can trust, which means a small object the boundary signs. An authenticated service-to-service channel isn't enough on its own. It proves which service is calling but not which user that service is acting for, so a compromised caller could assert any user_id it liked.

Netflix describes this design in its "Edge Authentication and Token-Agnostic Identity Propagation" post. Its edge services validate the many token types clients present and mint an integrity-protected identity object, called a Passport, which is what downstream services receive.

A signed context object is still a credential, so it needs limits of its own:

It's issued and accepted only inside your trust domain, never by an external API.
It's bound to one request or one message rather than to the user's login, through a short expiry, an audience claim that limits it to your internal services, and a request or message id.
It carries only the fields the work needs.

The boundary signs with a private key that only it holds, so a compromised internal service can read the context it was given but can't mint one for a different user. Like Netflix's Passport, the same context is passed along the whole call chain rather than re-signed at each hop, so no hop has to call a signing service mid-request.

The context can still be replayed inside your system. Within its short validity window, a service in the call path could replay it to another internal service, which is the same exposure a forwarded token has. What's gone is the external replay. No API outside your system accepts it, and it doesn't outlive the request.

Token Exchange Meets the Line, Forwarding Doesn't

This post draws its line at the first two limits above. Whatever travels inward is issued inside your trust domain and bound to one request or message. A gateway that exchanges the user's token for an internally issued, internally audienced one, using OAuth 2.0 Token Exchange (RFC 8693), meets that line. Forwarding the token the client presented does not.

Another alternative, per-hop delegation from the external identity provider, has each service trade for a new token audienced to the next one. It answers the replay concern but still crosses the line. Every hop then calls the provider at runtime and parses its token format, while a boundary-signed context is verified locally with a key you own.

Meeting that line at the gateway doesn't have to be heavy. For a system with two or three services and one issuer, the gateway signing a short-lived internal token with a standard JWT library keeps its cost to one key to manage.

Why Sessions Don't Belong in Internal Flows

Sessions are external trust artifacts. A user session token represents authentication with an external identity provider or your auth system. It's proof that someone outside your system is who they claim to be. Once verified at your boundary, propagating it further treats authentication as global state rather than a specific validation event.

Repeated validation adds a runtime dependency on the auth server. Every component that receives a user session has to validate it. For a self-contained token like a JWT, that's a local signature and expiry check. Verifying a boundary-signed context costs about the same, so the saving there is small. For opaque tokens and server-side sessions, validation means a call to the auth server, such as the introspection endpoint defined in RFC 7662. Each hop then adds a network round trip and makes the auth server's availability a condition of every internal call for a single user action.

Coupling to external authentication. When internal components accept user sessions, they become coupled to the token format, the issuer's validation requirements, token refresh logic, and external auth provider availability. With one standard OIDC issuer and a shared validation library, switching issuers can be mostly configuration. The coupling shows when the estate isn't that tidy, with several client token types, a proprietary session format, or a migration between them. Then every component that parses sessions has to change with it. Netflix's Passport work was a response to exactly that variety. A boundary-signed context still needs a shared format and key distribution, but that format is yours and doesn't change when the external issuer does.

Session Propagation Breaks When Work Outlives the Request

Propagating user sessions through internal components creates predictable failures whenever work outlives the request or has no user behind it.

Session expiration mid-flow is the first to appear. Event processors run minutes later and the session has expired. Background jobs run hours later and the session is gone. Async processors retry and the session is invalid. You build workarounds: token refresh in queues, session persistence in metadata, special "system sessions" for background work.

Missing sessions entirely breaks your logic. Scheduled reports run at 3 AM with no user logged in. Webhooks arrive from external systems with no user session. System maintenance tasks have no user context. You duplicate logic: one path for user requests and one for non-user operations.

Context across time and actors fragments workflows. Multi-step workflows span hours. The original user's session expires. Different actors have different sessions. Background processors have none. The workflow breaks without explicit context that persists beyond the session lifetime.

Storing context as data, which the boundary pattern makes the default, removes all three. The service that owns a workflow stores the original context with the workflow's own record, as data about who started it. Each later step or retry works from that record rather than from a credential that expires. When a later step calls another service, the workflow service calls as itself and passes the initiating user_id as data. The receiver checks that this service is allowed to request this action for this tenant, then checks the user's current permissions. A scheduled job or a webhook handler works the same way, with its own service identity and no user at all.

Pair the pattern with one rule. Every step authorizes against the user's current permissions and account status when it runs, so a user whose access was removed, or whose session was revoked after a compromise, is refused even hours later. The rule is independent of the token question, and a design that forwards tokens could adopt it too, but this pattern can't work without it. The context carries identity and the client's scope ceiling, not the user's roles, so a step has to look up permissions to decide anything. That lookup reads your own permission data, which the service already depends on to do the work, rather than making the external auth server a condition of every hop.

Async work has a limit here. Nothing in this path ties the user_id to a workflow that user started. A compromised workflow service can therefore act for any user in the tenants it serves, through the operations its own service identity is allowed. That is the same trust any background job carries, and it's why each service's identity should be allowed only the operations that service performs.

Common Objections

"We Can Forward Tokens on Synchronous Calls Only"

The strongest version of this view forwards a short-lived, audience-restricted token down synchronous call chains and uses explicit context only for async work. That view's async half is this post's design. Its sync half still carries the external artifact, so it keeps the coupling to the issuer's format described earlier. Widening the token's audience to cover the internal services, which RFC 9700 permits for a small set of resource servers, keeps the external replay out. It still leaves a credential in the issuer's format that lasts the token's whole lifetime rather than one request.

"We Lose User Context for Auditing"

You lose the session token, not the user context. Proper auditing requires intentional context capture: user_id, tenant_id, correlation_id, source_ip, user_agent, and timestamp.

The identity fields travel in the signed context through every component. Request details like source_ip and user_agent are logged once at the boundary and joined to downstream records by correlation_id, so no internal service can forge them. Relying on session tokens for auditing is worse because sessions don't persist for audit trails, you're logging authentication artifacts instead of business context, and different components might parse tokens differently.

Intentional data flow is the point. If you need user_id in audit logs, require it as an explicit parameter. This forces you to think about what context crosses boundaries and prevents accidental omissions.

"Compliance Requires User Identity Throughout the System"

Compliance rules ask you to authenticate who is acting and to record what they did. The HIPAA Security Rule, for example, requires verifying that a person seeking access is the one claimed (45 CFR 164.312(d)) and mechanisms that record and examine activity in systems holding health data (164.312(b)). Authentication happens at the boundary. Recording happens wherever the action happens, and a trusted user_id in explicit context is what that record needs. Passing the session token does not improve compliance. It couples your compliance logging to your auth mechanism.

"Service Roles Create Privilege Escalation Risks"

The risk is that a service with broad credentials can be tricked or compromised into acting outside the current user's rights. A forwarded token limits that risk only on calls to other services. On those calls, within one request, a boundary-signed context limits it just as well, since a compromised service can't mint context for another user. Async steps rely on the calling service's own identity, with the limit described earlier.

For the service's own resources, the token changes nothing. A compromised service with a database connection that reads every tenant's rows can read them whether or not it holds a user token, unless the database itself enforces the user's identity.

Service roles provide technical capability like database access and API credentials. Business logic enforces authorization through grant checks, tenant isolation, and permission verification. Where a breach would be severe, the data layer can enforce tenant isolation too, for example with row-level security keyed to the tenant in the context. These checks are necessary whether you use service roles or propagate user sessions. Scope each service's credentials to what that service does, so a compromise is limited to one service's capability.

A forwarded session adds its own risk. A bearer token works for anyone who holds it, so every internal component that receives the user's token can replay it as that user anywhere the token is accepted. That's why RFC 9700, the OAuth 2.0 security best practice, says access tokens should be audience-restricted to a specific resource server. A token restricted to your API shouldn't be accepted by every service behind it, and one that is has become a credential your whole system leaks.

Service roles also make the split clearer. The service has technical capability but requires explicit business authorization. User sessions blur this distinction by suggesting the session itself grants permission.

"Zero Trust Means Every Service Verifies the User"

Zero trust doesn't ask for the user's original token at every hop. NIST SP 800-207 requires that authentication and authorization are enforced before every access to a resource, rather than inherited from network location or an earlier step. A downstream service that verifies its caller's service identity and the signature on the user context, then authorizes the action against that context, meets the requirement. Forwarding the session meets it no better and spreads a replayable credential.

The legitimate exception is a component that must call another system as the user, such as a third-party API that only accepts the user's delegated token. Even then the original session doesn't travel. The boundary, the one place that holds the user's token, uses OAuth 2.0 Token Exchange (RFC 8693) to trade it for a new token issued for that one audience and scoped to that call. Work that calls the third party hours later relies on a grant the owning service stores as data, the way a workflow stores its context, not on a token carried through queues.

"We Need Sessions for Distributed Tracing"

You need correlation IDs for distributed tracing, not session tokens.

Distributed systems require explicit tracing infrastructure: correlation_id, trace_id, span_id, and timestamps. This works for all request types including user requests, background jobs, webhooks, internal system tasks, lambda invocations, and event processors.

If you rely on session tokens for tracing, non-user-initiated requests fail. When you need to join traces to a login session, the non-secret session id in the context does that without carrying the token.

"We're Using a Monolith, This Doesn't Apply"

Inside one process there's no token crossing a network and nothing to replay, so the security arguments above weigh less. Most web frameworks already apply the boundary rule for you, since their request principal holds a validated identity rather than the raw token. What remains is where business logic gets that identity. You still must avoid global user session context and pass user context within structures appropriate to the needs of each module.

Business logic that reads the current user from the web request couples your service layer to your web layer. A background job, CLI tool, or internal script that calls the same logic has no web request, so it either fails the way a missing session fails or has to fake a request principal first. Logic that takes the user as a parameter can be called from any of them as it stands.

The same principles apply: modules accept explicit context and perform explicit authorization. Whether those modules are in separate processes or the same codebase is irrelevant to that design.

Respect the Boundary

User authentication sessions serve one purpose: validating identity at a security boundary. Once validated, the session has done its job.

In practice:

Validate the session once, and let no component behind it parse the token
Extract only the context the work needs, such as user_id, tenant_id, correlation_id, and the calling client's granted scopes
Sign that context at the boundary, bind it to one request or message, and send it over authenticated service calls
Give each service its own narrowly scoped credentials
Authorize every action against current permissions, account status, and the client's granted scopes, whether a user, a webhook, or a scheduled job started it
When a component must act as the user elsewhere, exchange for a new token scoped to that call

This isn't about microservices versus monoliths. It's about recognizing architectural boundaries and respecting them. Authentication artifacts belong at boundaries, not in internal flows.

Why I Changed My Mind About Exceptions

Wed, 26 Nov 2025 16:55:50 +0000

I prefer the clean code that is more often produced by throwing exceptions. With the happy path uncluttered by error handling, the implicit propagation of errors to appropriate orchestration layers, and the clean separation of concerns. It's aesthetically cleaner and moves the complexity of error handling to one or just a few decision points.

But when Rust's approach to error handling gained cultural influence and C# developers (among many others of course) began exploring Result types through community libraries, I re-examined my perspective more honestly. Do the arguments for Results hold up under scrutiny for expected failures in modern systems?

This isn't just blind advocacy. It's an honest conceptual analysis of the arguments from both camps, examining which ones hold up and which are subjective preference.

Before I dive into this, let's be clear about what we are not talking about. We are not talking about programming errors like null references, array index violations, or machine resource errors. Those are bugs and special occurrences which should cause a fatal response since there is no safe way to default the outcome.

Choosing Between Results and Exceptions

When validating a payment request that might fail for multiple reasons (insufficient funds, expired card, fraud detection, network timeout), should your code:

// Option 1: Throw exceptions
public PaymentConfirmation ProcessPayment(PaymentRequest request)
{
    ValidateRequest(request); // Throws ValidationException
    var charge = _paymentService.Charge(request); // Throws PaymentDeclinedException
    var confirmation = _repository.SaveTransaction(charge); // Throws DatabaseException
    return confirmation;
}

// Option 2: Return Results
public Result<PaymentConfirmation> ProcessPayment(PaymentRequest request)
{
    var validationResult = ValidateRequest(request);
    if (validationResult.IsFailure)
        return Result.Failure<PaymentConfirmation>(validationResult.Error);

    var chargeResult = _paymentService.Charge(request);
    if (chargeResult.IsFailure)
        return Result.Failure<PaymentConfirmation>(chargeResult.Error);

    var saveResult = _repository.SaveTransaction(chargeResult.Value);
    return saveResult; // Already Result<PaymentConfirmation>
}

In the first option, a higher layer makes the decision. Or we simply let the failure propagate and use middleware to convert it to what we assume is a safe response, often an HTTP status code.

In the second option, which can vary greatly depending on the language or library being used, we can choose or are forced to handle each possible outcome. Often we need to handle a success, a failure, or a partial failure. This makes the expected outcomes visible at every layer. It also avoids throwing deep in a workflow only to catch and translate at the top, paying an exception's cost for an outcome everyone expected.

Why Expected Failures Matter More Now

In modern distributed systems, expected failures happen constantly at scale: circuit breaker fallbacks, timeout retries, validation of user input, partial batch results.

Distributed systems made expected failures more frequent. A monolith already calls databases and third-party APIs that time out, but most of its calls stay in-process and can't. Decomposing it turns more of those in-process calls into network hops, and each hop adds timeouts and transient faults. Business answers that used to be method calls, such as an out-of-stock or credit check, now arrive as a peer's 404, 409, or decline, and treating those frequent outcomes as exceptions creates friction.

That friction shows up in the client making the call. At a service boundary, Results and exceptions both serialize a failure into a status and body, but common clients default to throwing. Refit throws an ApiException for every non-success status on a method returning Task<T>, so an expected 404 or 409 becomes an exception the caller must catch and inspect. Its ApiResponse<T> wrapper records the error instead, a Result in all but name.

Offline-first architecture became common. For progressive web apps and mobile and desktop clients, "no network" and sync conflicts are normal operating conditions.

Functional architecture became common. Even OOP codebases now use stateless services, immutable data pipelines, and event-driven patterns. These designs reward treating a failure as data, and a message handler shows the mechanism. A handler that returns a Result can acknowledge the message and publish the rejection as an event. A handler that throws on a business rejection makes the broker treat the message as failed, redeliver it, and after enough retries dead-letter a message that was never broken. A top-level catch that lists the rejection exception types fixes that, until a new rejection it doesn't list dead-letters. A Result puts every rejection in the handler's signature.

Rust Enforces the Split, C# Leaves It to Discipline

Rust emerged with strong functional influences. It enforces a clear split: expected failures return Result<T, E> types, while unexpected failures trigger panics that unwind the stack like exceptions. The standard library marks Result as #[must_use], so ignoring one draws a compiler warning by default (a team can deny that lint to make it an error).

C# started heavily OOP-dominant and progressively adopted functional features (LINQ, pattern matching, immutability). It historically used exceptions for all failures, both expected and unexpected.

In C#, the same split is now possible through pattern matching (C# 7+) and community Result libraries, but it relies on discipline rather than compiler enforcement. C# 15, shipping with .NET 11 in November 2026, adds union types whose switch expressions the compiler checks for exhaustiveness. A union of specific error cases gets checked handling once the caller matches on it.

Microsoft's own guidance stays exception-first. It hasn't shipped a general Result type, and its Framework Design Guidelines say not to use error codes over performance concerns. The guidelines also call slow exceptions in code that routinely fails "a valid concern" and note that a throwing member "can be orders of magnitude slower." For members that fail in common scenarios, they recommend the Try-Parse pattern (int.TryParse returning false), paired with a throwing version. Microsoft presents that as a pattern within the exception model, applied member by member, not a different default.

The general Result pattern is community-driven. FluentResults and ErrorOr have about 37 and 12 million NuGet downloads (October 2026), small next to Newtonsoft.Json's 9 billion, and LanguageExt's 49 million cover a whole functional library, not just Results. Still, several independent libraries building the same pattern shows a need, in some codebases, that the base library leaves unmet.

Arguments for Result Types

Performance for frequent expected failures

When validation failures happen thousands of times per second on a CPU-bound hot path, throwing costs more than branching, because an exception captures a stack trace and unwinds the stack. Inside a request that waits on a database or a timeout, the cost is noise, so this is the narrowest of the four arguments.

Strength: A gain on CPU-bound hot paths, not a reason on its own.

Information disclosure prevention by default

An exception carries its stack trace and message everywhere it goes, and following Microsoft's guidance puts a provider's details inside it. Microsoft's best practices for exceptions tell you to rethrow the original or wrap it as the inner exception. So a domain exception built by the book carries the provider exception from a database driver or library you don't control, and that message can include constraint names, SQL fragments, or file paths. ASP.NET Core's production default returns a bare 500, so those details leak through handlers teams write themselves, such as an error response that includes ex.Message or ex.ToString(). An adapter could throw a fresh exception with no inner one, but that goes against the guidance.

Results change this when the error type can't hold an exception, such as a C# 15 union of domain cases or an error record with only a code and message. An adapter that translates the provider exception into one of those logs it at that point. The diagnostics stay in your logs, and the error carries only the fields your domain code put in it. FluentResults' CausedBy and ErrorOr's metadata dictionary can both attach the exception, which reopens the leak. Unanticipated provider exceptions still depend on the boundary handler in either design.

Strength: With an error type that can't carry exceptions, modeled failures are safe by construction.

Type signatures as reliable documentation

Result<Order> tells you immediately that getting an order can fail. Order tells you nothing without reading implementation or relying on potentially outdated XML comments. Refactoring tools update type signatures automatically; they don't update documentation.

Strength: Fallibility in the signature can't silently go stale the way a comment can.

Natural composition for partial success and iteration

Batch operations, parallel workflows, and offline sync scenarios often have partial success. Results compose naturally through filtering and mapping.

More critically, exceptions pull iteration toward wherever they are caught. When some items in a collection might fail, iteration tends to land in the orchestration layer where the catch is, even if it logically belongs in the service layer.

// Exception approach - iteration pushed into the controller
public async Task ProcessOrders(List<string> orderIds)
{
    foreach (var id in orderIds) // Iterate where you can catch
    {
        try { await _orderService.Process(id); }
        catch (Exception ex) { /* handle */ }
    }
}

// Result approach - iteration lives in service layer
public async Task<Result<Order>[]> ProcessAll(List<string> orderIds)
{
    var tasks = orderIds.Select(id => Process(id));
    return await Task.WhenAll(tasks); // Parallel, every outcome kept
}

The service layer could iterate with exceptions too, but only by catching each item's exception and converting it into a per-item success or failure, which is a Result by another name.

Parallelism adds a second cost. Task.WhenAll waits for every task, but if any of them faults, the combined task faults with all the exceptions aggregated. await rethrows only the first, and no array of results comes back, so the caller goes back to each task to sort successes from faults. Task.WhenAll over Results returns every outcome as an ordinary array.

Strength: Results allow iteration and parallelism at appropriate abstraction levels.

Arguments for Exceptions

Implicit propagation to appropriate handlers

Exceptions bubble to orchestration layers without code at each level. The happy path stays clean; error handling lives at boundaries.

Results require explicit propagation. Return Result<T>, check it, propagate it. This threads error handling through intermediate functions that don't care about the specific error.

Strength: Separation of concerns. Domain logic stays focused on domain, not error threading.

Framework integration without friction

The .NET ecosystem uses exceptions. Entity Framework throws DbUpdateException. HttpClient throws HttpRequestException. Wrapping every framework call in try-catch to convert to Results creates boilerplate at every boundary.

Strength: Working with the ecosystem, not against it.

C# doesn't enforce Result handling

Unlike Rust, where ignoring a Result draws a compiler warning, C# lets you completely ignore returned Results. You can access .Value without checking .IsSuccess and get runtime exceptions anyway.

public Result<Order> GetOrder(string id) { /* ... */ }

GetOrder("123"); // Completely ignored, no compiler error

The "compiler safety" argument assumes static analyzers and discipline, the same discipline proper exception handling requires. The way each one fails differs too. A forgotten exception propagates and fails the request loudly, while a forgotten Result with no value, such as a failed save, lets execution continue as if it succeeded.

Strength: Results in C# provide discoverability, not enforcement.

Orchestration layers solve the same problems

Proper architecture already requires orchestration layers that catch domain exceptions, translate them to appropriate responses, and control what information crosses boundaries. Results don't eliminate the need for this architecture; they just change what propagates upward.

Strength: Architecture matters more than mechanism.

Exception documentation is sufficient with discipline

XML comments document exceptions, appear in IntelliSense, and provide discoverability at call sites:

/// <exception cref="OrderNotFoundException">When the order doesn't exist</exception>
public Order GetOrder(string id)

Well-maintained codebases keep documentation current through code reviews. If you have discipline to maintain exception documentation, you have discipline to handle Results properly.

Strength: Documentation works if teams maintain it.

Results encourage scattered error handling

Making it syntactically easy to handle errors inline encourages developers to scatter error-handling logic across call sites instead of centralizing it in orchestration layers.

Strength: Results can create worse maintainability if misused.

Which Arguments Hold Up?

Result Arguments That Stand

Throwing costs more than branching, which matters on CPU-bound hot paths
A modeled failure's error carries no stack trace or provider message
Type signatures show that an operation can fail more reliably than documentation
Iteration and parallelism can live at appropriate abstraction levels

Exception Arguments That Stand

Implicit propagation reduces boilerplate in intermediate layers
Framework integration is smoother without constant translation
Orchestration layers remain valuable for consolidating error handling decisions
C# doesn't enforce Result handling, so Results give discoverability, not enforcement

Exception arguments that weaken:

Documentation is sufficient with discipline (documentation rots in practice)
A clean happy path is clearer (it also hides what can fail)
Exceptions are for rare cases, so their cost doesn't matter (expected failures aren't rare)
Results scatter error handling (orchestration layers can still own the decisions)

Where the Structural Arguments Point

Some arguments on both lists are structural, meaning they follow from what the mechanism is, like propagation or signatures, rather than how carefully a team uses it. What separates them is whether each side's structural weakness has a workaround. Results' weakness is the exception camp's strongest argument, implicit propagation, since C# has no ? operator to hide the explicit threading Results need. But for a linear flow, library combinators recover most of the clean happy path, as in the earlier payment flow written with ErrorOr's Then:

public ErrorOr<PaymentConfirmation> ProcessPayment(PaymentRequest request) =>
    ValidateRequest(request)
        .Then(_ => _paymentService.Charge(request))
        .Then(charge => _repository.SaveTransaction(charge));

Async steps and steps that need several earlier values push combinators toward nested lambdas, so the recovery is partial.

Exceptions have no workaround for per-item outcomes, typed fallibility, or cheap hot-path failures. Catching and converting, or adding a Try-Parse member, each produces a Result by another name. Resilience pipelines such as Polly help less than they seem to. They keep retries and simple substitute-value fallbacks out of the intermediate layers, but per-item outcomes and fallbacks that need domain context still land there as try-catch blocks.

What an exception default gives up is the signature. Implicit propagation is genuine convenience, but it comes at a cost of invisible failures. A Result<Order> with an open error list, rather than a closed set like a C# 15 union, doesn't say every way it can fail either. But a partial, typed channel for the expected outcomes still beats no channel.

If Results win where the workarounds run out, the obvious compromise is to use them only there, a hybrid of exceptions by default and Results at the seams. In a distributed system the seams aren't few. Every downstream call, message handler, and batch is one. And a hybrid decides each method's failure contract case by case, so a caller can't tell from a signature which convention it follows. A default makes the signature answer that question the same way for every domain outcome. Infrastructure faults follow one test. The adapter returns a Result when the caller has a decision to make about the fault, and lets it throw otherwise. Its signature shows which faults come back.

A default still leaves callers two channels, a Result to check and exceptions that can still propagate. The exception channel is the boundary catch an exception-only design already has, so the added work is the Result branch, where the caller's decisions were going to live anyway.

Both sides depend on discipline, but a discarded Result can be caught by a linter and a stale exception comment can't. A forgotten Result that carries a value still fails loudly, since FluentResults throws when you read .Value from a failure. The silent case is a discarded Result with no value. The SDK's own CA1806 rule flags it once you raise its severity from suggestion. You list your Result-returning methods in its additional_use_results_methods option, an upkeep cost but a checkable one. Keeping <exception> comments current means tracing every throw through the call graph, which third-party checked-exception analyzers attempt and the SDK doesn't ship.

That reasoning comes down to a short set of rules:

Return Results for expected domain outcomes, such as validation, business rules, and declined or unavailable downstream services
Throw exceptions for programming errors, such as null references and contract violations
Translate a framework exception into a Result at the adapter only when the caller has a decision to make about it, such as a timeout that triggers a fallback or a unique-key violation that means "already exists," and let the rest propagate
Run parallel work as Task.WhenAll over Results, converting any exception an item throws into a failed Result at the item boundary, so one fault can't discard the other outcomes
Centralize error handling decisions in orchestration layers that handle both Results and the framework exceptions that propagate

Why the Resistance?

Paradigm friction explains part of the resistance. C# grew up with exceptions as its only failure channel, while the functional tradition that shaped Rust treats expected failures as data. Preferring exceptions in C# follows the language's history as much as any weighing of the arguments.

Hard-won expertise also contributes: "Exceptions work if done correctly, and I've learned how to do them correctly." This solves yesterday's problem (poor exception handling) rather than today's problem. When a large portion of operations return expected failures, exceptions require working around their design, not just using them correctly.

What I Learned

Result types should be the default for domain operations, not because they're perfect, but because the structural advantages outweigh the execution risks.

Despite my preferences, the evidence (at least conceptually) has led me to that conclusion. Where expected failures are frequent and callers act on them, visible failures, safe errors for the failures you model, and natural iteration patterns matter more than implicit propagation convenience. In a codebase of linear operations with few caller decisions, the balance is closer. The default is the pattern, not a library, so a team can move from today's libraries to C# 15 unions when it wants closed error sets.

When an increasing number of use cases can and will return expected failures, you need mechanisms designed for common outcomes, not rare anomalies.

Moving forward I will endeavor to use Results for expected failures, reserve exceptions for bugs, and build the discipline and tooling to use them both well.

Monoliths for Discovery, Microservices for Optimization

Thu, 20 Nov 2025 23:41:52 +0000

I know, I know. It is another post about microservices versus monoliths. The debate feels exhausted at this point. Yet every time I start a new project, I find myself weighing the same questions. Not because the answer is unclear, but because the answer genuinely depends on where you are and what you're trying to learn.

The choice isn't about finding the "right" architecture in the abstract. It's about choosing what fits your context, your constraints, and most importantly, what you need to discover.

Principles That Decide the Choice

Microservices Are Optimizations, Monoliths Enable Discovery

Microservices are optimizations for specific problems: team scaling, independent deployment, technology diversity. They're solutions to constraints you've already identified. Monoliths, on the other hand, accelerate discovery. They let you learn where boundaries should be, understand domains as they reveal themselves, and validate assumptions quickly without the overhead of distributed systems.

That overhead is concrete. Once two parts of a system talk over the network, every call can time out, fail halfway, or arrive twice, and data that used to change in one transaction now needs a consistency strategy. Each service also needs its own deployment pipeline, monitoring, and on-call coverage. Martin Fowler calls this the microservice premium, and in his post of that name he advises against even considering microservices "unless you have a system that's too complex to manage as a monolith."

The premium grows with every service added. In "Goodbye Microservices," Alexandra Noonan describes how Segment split its event delivery into one service per destination, for a sound reason. A slow destination's retries had flooded a shared queue and delayed all the others. The split ended at more than 140 services. Shared libraries drifted apart, and three full-time engineers spent most of their time keeping the system alive, until the team merged the services back into one. The signal justified separating slow destinations from the rest, and it didn't justify a service for every destination.

The premium hurts most while the domain is still unclear, because early boundaries are mostly guesses. Inside one process, a wrong guess is fixed with a refactoring. You move a class or change a method signature, and the compiler finds every caller. Across services, the same correction means changing two APIs, coordinating two deployments, and migrating data between two databases. Fowler's companion post, "MonolithFirst," makes this argument. It also reports that almost all the successful microservice stories he had heard of started with a monolith that grew too big and was broken up. He offers that as an informal observation, not a study.

Start with what helps you learn fastest, then optimize when you understand what actually needs optimizing.

Monoliths Don't Excuse Poor Code Quality

The stigma around monoliths comes from poor practices, not the architecture itself. Terrible monoliths exist, but so do terrible microservices. A monolith whose modules call each other's internals and write each other's tables is hard to change and harder to split. A set of microservices that share tables and must deploy together has the same coupling, with network calls added. What decides whether either system stays changeable is good design discipline, meaning SOLID principles applied at the module level and clear boundaries. Each module depends on the others' public interfaces and never on their internals.

The deployment model does change one thing. A process boundary makes reaching into another module's internals impossible, while inside a monolith it takes one import. That enforcement is part of what the microservice premium buys, but during discovery it isn't worth the price, because the same process boundary makes every module boundary expensive to move.

A monolith can get most of that enforcement more cheaply. It needs it, because good intentions lose to deadlines. Give each module its own database schema, and have CI load and test each module with only the modules it declares as dependencies. A shortcut then fails a build instead of slipping through review. Moving a boundary still means editing a dependency list and a schema in one repository rather than redeploying two services.

Neither the per-module schemas nor the CI checks add a runtime component, a network hop, or a second deployment. Both need upkeep, such as test setups for each module, and they are far harder to retrofit than to start with.

Ship Before the Architecture Is Perfect

Ship and learn. Every week spent settling the architecture before launch is a week without users to show which parts of the design matter. The better approach is to make reversible decisions, establish clear boundaries, and iterate based on what you learn from real usage. Perfection is a moving target. Momentum matters more.

Shipping before the design settles works only if wrong guesses are cheap to undo, and the starting choice decides how often you pay for distribution. Extracting a module from a monolith still means moving its data and taking on partial failure for every call across the new boundary. A team that starts with a modular monolith pays that distributed cost once per boundary it has already validated. A team that starts with services pays it for every guess, and again for each guess it has to correct. Either way, a decision stays reversible when undoing it touches one module instead of every caller, which is why the next two principles are about boundaries.

Data Boundaries Matter From Day One

Define data ownership early, even if you're starting with a monolith. Sharing one database between modules, or even between early services, isn't inherently wrong when you're just getting started. It works as long as each table has one owning module, and everything else goes through that module's code instead of querying its tables directly.

Without that rule, extraction gets expensive. When three modules write to the same orders table, pulling the order module out means finding and rewriting every one of those queries. Each cross-module join also becomes a network call or a copy of the data. Tangled data models trap teams in legacy architectures.

The ownership rule keeps the cheap in-process refactoring from the first principle. When a table changes owner in a modular monolith, the compiler and the test suite find the code to change, and there are no dual writes between two databases. The rule makes an ownership change a schema migration inside one database and one deployment, rather than a data move between two services.

Abstract Infrastructure Early

Put one routing layer you control in front of the monolith early. An API gateway works, and so does the reverse proxy or ingress it already runs behind. Either gives you flexibility before you know what your final infrastructure will look like. When clients call that layer instead of the monolith's own address, extracting a service later can keep the address clients already use, because the route moves there. That holds when the extracted module owns its own URL paths and the new service keeps the old contract.

The routing layer doesn't help with the harder server-side work of partial failure and consistency. A dedicated gateway is also one more component to run and one more network hop. It pays off most when you have clients you can't update all at once, such as mobile apps or partner integrations. For those systems, the routing layer keeps an extraction from turning into a coordinated client release.

The Strongest Case Against Starting With a Monolith

Stefan Tilkov answered Fowler with "Don't start with a monolith." He argues that a monolith's parts "will become extremely tightly coupled to each other" through shared libraries, in-process calls, and shared domain objects and persistence models, and that splitting one up afterward is extremely hard. He is right about what happens to a monolith built without discipline. That coupling is the reason the principles above exist.

Module boundaries, one owner per table, and a routing layer between clients and the deployment don't make a later split cheap, but they make it a matter of cutting along seams that already exist. The queries to rewrite are the ones the owning module already exposes, instead of a search through every module for whoever touches the data. Public accounts that follow one system from monolith discovery through extraction along settled seams are rare, so this rebuttal rests on the cost mechanism above more than on a body of case studies.

His argument also marks where this approach stops applying. When the boundaries are already known, as when a team rebuilds a system whose domain it has run for years, there is less left to discover. Building separately deployable parts from the start then gives up little. Discovery is the reason to start with a monolith, so the case for one weakens as the amount left to discover shrinks.

Some constraints are known on day one even when the domain is new. An organization that starts a new product with several teams already has the coordination constraint. So does a component that must sit apart for compliance or security, such as card handling kept out of the rest of the system's PCI scope. A known constraint is what the optimization exists for, so a few coarse services along those lines, each a modular monolith inside, fit that case better than one shared deployment. Discovery then happens within each service, where moving a boundary is still cheap.

My Approach: Domain-Based Modular Monoliths

Start with domain-based modules: one modular monolith organized along natural business boundaries, which Eric Evans's domain-driven design calls bounded contexts. This gives you:

Clear separation of concerns from the beginning
Teams that own their domain's code, even while they share a deployment
A migration path to microservices that follows existing module boundaries instead of cutting new ones
Faster initial development than building distributed systems from day one

Shopify took this route with its main codebase. Kirsten Westeinde's account of the work describes reorganizing the code by business domain, such as orders, shipping, and billing, and building tooling to check that components talk only through their public interfaces. The goal, in her words, was more modularity "without increasing the number of deployment units."

Five years later, Gannon McGibbon and Chris Salzberg's "A Packwerk Retrospective" reported that static checks from Packwerk, the boundary-checking tool Shopify later built and open-sourced, hadn't been enough. A package with all its recorded violations resolved still failed to boot when loaded on its own, because of dependencies the static analysis couldn't see. What they found trustworthy was running code, so once they got that package to boot with only its own code loaded, they added a CI step that runs its tests in isolation. Packwerk 3.0 also dropped its privacy checks, the static rule meant to keep other packages out of a package's internals.

Boundaries hold when something executes them, and a folder structure or a lint rule alone tends to erode one convenient shortcut at a time. Where only static checks guard the boundaries, Tilkov's coupling warning holds. Shopify's answer was a stronger check inside the monolith, not a move to services.

Split When the Evidence Shows Up

Split into microservices when there's actual evidence it's needed: independent scaling requirements, team coordination becoming a bottleneck, or specific technology needs that justify the operational complexity.

Each of those signals, plus one more, can be checked instead of argued:

Independent scaling: one module's load differs so much from the rest that scaling the whole deployment to meet it mostly buys capacity nothing else uses
Team coordination: teams routinely wait on each other to release, or a shared test suite and deploy queue hold every change to the pace of the slowest one
Technology needs: a module needs a runtime or data store the rest of the system can't host, and the gain outweighs operating a second stack
Fault isolation: one part's failures or slowdowns keep spreading to unrelated parts, and timeouts, bulkheads, or separate queues inside the deployment haven't contained them

These signals arrive late, once the pain is present. The monolith also gives an earlier one. Track the share of merged changes that touch more than one module, and how often a change moves code or tables between modules. Watch them over several releases while feature work continues. The boundaries have probably settled when three things hold: the first number is falling, the second has dropped to near zero, and no duplicated logic or swelling module is absorbing the work. That says the seams are stable enough to plan an extraction around. It isn't a reason to extract, which still waits for one of the signals above.

A signal tells you that something needs to be separated, and the module boundaries tell you where the seam already runs. Until a signal shows up, the monolith is still doing the job it was chosen for, which is teaching you where the boundaries belong.

Core Attributes of Distributed Systems: Reliability, Availability, Scalability, and More

Mon, 30 Jun 2025 21:28:45 +0000

Whether you’re building a simple web app or a large distributed system, users don’t just expect it to work → they want it to be fast, always available, secure, and to run smoothly without unexpected interruptions.

These expectations are captured in what we call system quality attributes or non-functional requirements.

In this article, we’ll explore the most critical attributes that any serious system should aim to deliver, especially in distributed environments. We’ll cover why each attribute matters for the users, how to measure it, and how to achieve it both proactively and reactively.

Reliability

Definition: Reliability is the ability of a system to operate correctly and continuously over time, delivering accurate results without unexpected interruptions or failures.

Why It Matters

Users rely on your system to behave predictably. If your banking app transfers money to the wrong account or your flight booking app glitches, it erodes customer trust instantly.

How to Measure:

Mean Time Between Failures (MTBF): Average time system runs before failing.
Error rate: Frequency of incorrect results (e.g., data corruption or logic bugs).

Proactive Techniques (making the system reliable in advance)

Fault prevention (Stop mistakes before they happen) → Write clean code, perform code reviews, use static analysis tools.
Fault removal (Find and fix mistakes early): Use automated testing, debugging, and formal verification.

Reactive Techniques (handling faults when they occur)

Fault tolerance (Keep working despite faults) → Use retries, replication/redundancy, graceful degradation, and error correction.
Fault detection (Spot problems quickly) → Monitor logs, set up alerts, use health checks and diagnostics.
Fault recovery (Fix issues promptly) → Restart services, failover to backups, roll back to safe states.

Availability

Definition: Availability is the ability of a system to be up and responsive when needed, ensuring users can access it at any time. It focuses on being ready to serve, not on whether the response is correct (which is covered by reliability).

Why It Matters

If your system crashes or is down during peak hours, users will leave. For mission-critical systems like trading, even seconds of downtime can be disastrous.

How to Measure

Uptime percentage → e.g., 99.9% uptime = ~8.7 hours of downtime/year.
Mean Time to Recovery (MTTR): How fast you recover from failure.
High availability (HA) typically refers to uptime of 99.9% or more, achieved through redundancy and failover strategies.

Proactive Techniques

Capacity planning: Predict demand and provision enough resources.
Redundant infrastructure: Extra hardware or cloud zones ready to take over.

Reactive Techniques

Failover mechanisms: Automatically switch to backup nodes or servers.
Auto-healing: Restart crashed services or containers automatically.

Scalability

Definition: Scalability is the ability of a system to handle more users or more data by adding more resources, without significantly slowing down or crashing.

Why It Matters

What works smoothly for 10 users might completely break when 10,000 people show up. If your product becomes popular, you want it to grow without falling apart.

How to Measure

Throughput → How many requests per second your system can handle.
Latency under load → How fast your system responds when many users are active at once.

Proactive Techniques (preparing for growth in advance)

Design for scalability (Build with growth in mind) → Use stateless designs, modular components, and databases that can be partitioned or scaled out.
Capacity planning (Plan ahead for future load) → Estimate how much traffic or data you’ll have later and make sure your system can handle it.

Reactive Techniques (handling growth when it happens)

Auto-scaling (Add resources on the fly) → Automatically spin up more servers when traffic spikes.
Load balancing (Distribute work evenly) → Spread incoming requests across multiple servers so no single one gets overloaded.

Maintainability

Definition: Maintainability is the ability of a system to be easily changed, updated, fixed, or improved over time without introducing new problems.

Why It Matters

Requirements always change. Bugs appear. New features need to be added. If your system is messy or overly complex, even small changes become risky and time-consuming. A maintainable system is easy to understand, modify, and operate day to day, letting teams respond quickly and confidently to new needs.

How to Measure

Mean Time to Modify (MTTM) → How long it takes to make a change or add a new feature.
Code churn → How frequently the code is updated or changed, which can indicate areas that are difficult to maintain or keep stable.

Proactive Techniques (making the system easier to change in advance)

Modular design (Break it into manageable parts) → Structure your system as small, independent components that are easier to understand, test, and replace.
Simplicity (Avoid unnecessary complexity) → Keep designs and code clear and straightforward to reduce errors and make it easier for new developers to pick up.
Clear documentation and standards (Help everyone stay aligned) → Write understandable docs and follow consistent coding styles so others can safely make changes.
Operability considerations (Design for smooth running in production) → Build clear configuration, easy deployment processes, and good monitoring hooks to simplify day-to-day management.

Reactive Techniques (improving it over time)

Refactoring (Clean up continuously) → Regularly improve the structure of code without changing its behavior to keep it healthy and easy to work with.
Automated regression tests (Prevent breaking existing features) → Run tests that ensure changes don’t accidentally introduce new bugs.
Incremental improvements (Make small, safe changes) → Tackle technical debt gradually without big risky rewrites.

Security

Definition: Security is the ability of a system to protect itself from unauthorized access, misuse, or attacks.

Why It Matters

A single security breach can damage your reputation, leak sensitive data, or cause big financial losses. Attackers don’t wait for you to be ready → you have to plan ahead.

How to Measure

Time to detect and respond → How quickly you can find and fix security issues.
Number of vulnerabilities over time → Track how many security flaws are open and how quickly they’re closed.
Compliance scores → Certifications like SOC2 or ISO 27001 that show your security practices meet industry standards.

Proactive Techniques (protecting the system in advance)

Threat modeling (Think like an attacker) → Identify and fix weak points before someone exploits them.
Secure defaults (Build security in by default) → Use encryption, strong passwords, and access controls.
Security scans (Catch issues early) → Run automated tools to find known vulnerabilities in your code.

Reactive Techniques (responding when something goes wrong)

Intrusion detection (Spot attacks fast) → Use systems that alert you to suspicious activity in real time.
Incident response (Limit the damage) → Apply security patches quickly and have a plan to contain and fix breaches.

If you have any feedback on the content, suggestions for improving the organization, or topics you’d like to see covered next, feel free to share → I’d love to hear your thoughts!

Coder Legion - Recent posts tagged distributed-systems

Reliable Event-Driven Architecture in Spring Boot: Outbox, Inbox, Retries, and Idempotency

1. The dual-write problem

The transactional outbox

2. Why the outbox still needs idempotency

3. The transactional inbox

A subtle but important case

4. Retries are a lifecycle, not just an annotation

5. Multi-instance processing changes everything

6. Exactly-once is not an end-to-end promise

7. Observability is part of reliability

8. Putting these patterns together

9. Why I built NERV Event

Final thoughts

Tuesday Isn't a Valid State Transition

What the calendar is for

The rule winemakers actually use

Status is a column anyone can write

Same evidence, two states

A transition is an event with a receipt

What time is actually allowed to do

Guarding your transitions

From E-Commerce to Agentic AI: Paul Chen’s 30-Year Developer's Journey

From E-Commerce to Agentic AI: A 30-Year Developer’s Journey

How It All Started

Twenty Years at IBM: Growing Into the Problem

Learning What Distributed Systems Really Feel Like

When Cloud Made Problems Invisible

The Challenges That Shaped Me

From Technical Contributor to Technical Leader

When the Architecture Was Right but the Tools Weren't Ready

What I've Learned After 30 Years

1. Every Technology Wave Has the Same Shape

2. The Language Is the Least Important Part

3. Performance Problems Are Architecture Problems in Disguise

4. Patents Are a Record of Unsolved Problems

5. The Best Teams Share a Mental Model, Not Just a Codebase

Growth as a Developer: The Honest Version

Capture Your Reasoning, Not Just Your Results

Read Outside Your Stack

Developers in the AI Era

What I'm Building Now

A Note on CoderLegion

About Paul Chen

Idempotency Starts with Business Operation Identity

Determinism Is a Core Contract in Build Caching

Why determinism is the contract, not a nice-to-have

One bad action poisons everything downstream

Where nondeterminism actually comes from

How to hunt them down

Hermeticity is how you keep it fixed

Measure it, because it regresses

Why Modern Engineering Teams Must Prioritize System Observability Over Raw Performance

The Art of Cloud Survival: The Day Monitoring Failed First

When Everything Looked Fine

The Green Dashboard Problem

Why Distributed Systems Hide Failure

Monitoring Tells You What You Expect

The Signals That Actually Matter

The Danger of Averages

Failures Rarely Arrive All at Once

Building Visibility on AWS

The Question Every Team Should Ask

Cloud Survival Requires Visibility

When "Latest" isn't Actually Latest

When "Latest" Isn't Actually Latest

TL;DR

The Assumption That Works... Until It Doesn't

A Simple Example

The Real Problem: Time Has Multiple Meanings

Event Time

Processing Time

Where Things Get Complicated

What About Kafka?

Why Data Engineers Care

Final Thought

The Miyake Event Problem: Anchoring Distributed Agents to Universal Time

The floating chronology problem

Distributed systems have the same problem

AI agents face it in a harder form