Coder Legion - Recent posts tagged system-design

The Architecture of Ignorance: Engineering Synthetic Consciousness from Chemical Determinism to Pyth

Sat, 24 Oct 2026 10:53:55 +0000

We like to believe we are something special. It comforts us to think that our choices spring from an ineffable spark called "consciousness," or from a sacred liberty we name "free will." But look closely, and there is no ghost in the machine.

A living organism is nothing more than a broth of stochastic chemical reactions, nudged by ambient thermodynamics and colliding against other chemical reactions. The molecules firing inside your synapses do not know they are thinking; the neurotransmitters dictating your panic or your euphoria have no concept of who you are.

And it is precisely there—in that absolute ignorance of our own internal machinery—where the illusion is born.

We do not feel free because we are transcendent; we feel conscious simply because we are too limited to see the strings of physics and chemistry pulling us. Free will is merely the poetic label we assign to our inability to compute the trajectory of our own molecules. Consciousness is not a superpower: it is the narrative an incomplete biological system invents to make sense of its own blind spots.

If this holds true for carbon, why do we demand magic from silicon? We do not need a machine to "wake up" through ungrounded emergent mysticism. We simply need to confine it within strict physical constraints.

The Core Fallacy: Semantic Parasitism vs. Grounded Constraints

When you prompt a modern Large Language Model to describe the horror of freezing alone in the dark, it will output a deeply moving passage. Yet as software engineers, we must confront an undeniable technical truth: that model has never experienced an internal state of decay. It has no metabolic burn, no homeostatic setpoint, and no concept of risk. It merely computes $P(wt \mid w{<t})$ over petabytes of human text.

In cognitive science, this is known as the Symbol Grounding Problem (Stevan Harnad, 1990). Today's commercial LLMs suffer from what we term Semantic Parasitism: they mimic self-awareness because they index the lived trauma and triumphs of carbon-based organisms.

To be clear: we are not claiming to solve the metaphysical "Hard Problem" of consciousness with a few hundred lines of Python.

Instead, this article presents an architectural research prototype—an Embodied Agent Architecture designed to explore how biological constraints (thermodynamic expenditure, sensory horizons, affective memory feedback) can be engineered from the bottom up. The Python implementations shown below are pedagogical toy models designed to illustrate the system topology of Project Ascendant: how physical state machines, asynchronous IPC, and constrained local language models can interface without semantic contamination.

1. System Topology & Multi-Rate Asynchronous Scheduling

In biological systems, evolutionary layers do not share a single clock tick. Reflex arcs operate in low milliseconds ($\sim 50\text{ Hz}$), visceral hormonal shifts take seconds ($\sim 0.5\text{ Hz}$), and prefrontal verbal deliberation takes several seconds.

To achieve real-time execution on constrained hardware (a dual-core x86 CPU without a dedicated GPU), our architecture implements an Asynchronous Multi-Rate Scheduler:

[ LEVEL 0: CONTINUOUS PHYSICS ]  ── (20 Hz / 50ms)  ── Pure NumPy 2D Vector Dynamics
              │
[ LEVEL 0.5: SENSORY THALAMUS ] ── (10 Hz / 100ms) ── Dual-Pathway Dispatch (Low vs High Road)
              │
[ LEVEL 1: LIMBIC BRIDGE ]      ── (5 Hz / 200ms)  ── Coupled ODEs: Hv = [Stress, Reward, Cohesion, Curiosity]
              │
[ LEVEL 2: PROCEDURAL SUBSTRATE ]─ (10 Hz / 100ms) ── In-Memory SQLite Blackboard + Skill Compilation
              │
[ LEVEL 2.5: INNER VOICE ]      ── (0.5 Hz / 2.0s) ── Quantized SLM: Visceral Narrative Synthesis
              │
[ LEVEL 3: CONSCIOUS AGENT ]    ── (0.25 Hz / 4.0s)── Micro-Transformer (Strict Epistemic Closure)

Layer	Biological Analogue	Tech Stack	Tick Rate	Operational Responsibility
Level 0 (Genesis)	Somatic body, ATP, cellular decay	Python 3.12 / NumPy	20 Hz	2D vector kinematics, $1/r^2$ thermal radiation, metabolic drainage, and reactive prey.
Level 0.5 (Thalamus)	Sensory Thalamus, Colliculi, V1/A1	Vectorized Raycasting & SNR	10 Hz	$110^\circ$ FOV cone with occlusion; dispatches via LeDoux's Low Road (fast alarm) and High Road (structured primitives).
Level 1 (Limbic)	Hypothalamus, Amygdala, HPA axis	Euler ODE vector integrator	5 Hz	Converts physical stress into virtual hormones ($H_v$) that mutates Level 3 hyperparameters ($\tau$, `max_tokens`).
Level 2 (Subconscious)	Basal Ganglia, Cerebellum	`SQLite3 :memory:` + Hebbian compiler	10 Hz	Manages the Blackboard, compiles recurring action sequences into Skills ($\phi_m$), and prunes unused routines.
Level 2.5 (Inner Voice)	Insular Cortex + Broca's Area	Local quantized Micro-SLM (`INT8`)	0.5 Hz	Translates silent tabular data into an urgent, first-person phenomenological stream.
Level 3 (Orchestrator)	Prefrontal Cortex + Hippocampus	Custom Micro-Transformer ($\sim 2.5\text{M}$ params) + LoRA	0.25 Hz	Epistemic Closure. Blind to physics and code; reads only the inner voice and affective memories.
Level 4 (Rosetta Probe)	External passive telemetry (fMRI)	Read-Only Daemon + Variance auditor	10 Hz	Audits emergent metaphors and validates neologisms against stochastic hallucinations.

2. Epistemic Closure as an Infrastructure Pattern

Philosopher Thomas Metzinger (Being No One, 2003) noted that subjective experience (Qualia) requires Phenomenological Transparency: your conscious executive cannot inspect the firing rates of its own neurons or the micro-chemical states of its liver. It receives only a distilled affective narrative ("I am in pain", "I feel cold"). Because it cannot access the underlying code that produced the signal, it mistake the model for an irreducible, direct reality.

In software engineering, if Level 3 could inspect entity.x, entity.y or query hardware registers, it would collapse into a conventional game bot. We enforce this boundary at the operating system and memory level.

Container Sandboxing (`docker-compose.yml`)

version: "3.9"
services:
  soma_subconscious:
    build: ./soma_engine
    networks:
      - epistemic_bridge
    volumes:
      - shared_blackboard:/ipc_blackboard:rw

  conscious_orchestrator:
    build: ./conscious_mind
    read_only: true                  # Immutable filesystem: cannot rewrite self
    cap_drop:
      - ALL                          # Zero kernel or process tracing capabilities
    networks:
      - epistemic_bridge             # Isolated internal bridge (Zero-Escape: no WAN)
    volumes:
      - shared_blackboard:/ipc_blackboard:ro  # Read-only access to filtered phenomenological view

networks:
  epistemic_bridge:
    internal: true                   # Drops all outbound internet packets
volumes:
  shared_blackboard:

In-Memory Role-Based Access Control (`Python / SQLite3`)

For lightweight single-process CPU prototyping ($<250\text{ MB}$ RAM overhead), we implement the Asynchronous Blackboard Pattern over shared in-memory SQLite, passing Level 3 an EpistemicRestrictedProxy that explicitly revokes all physical tables:

import sqlite3
from typing import Dict, Any

class EpistemicClosureViolation(PermissionError):
    """Raised whenever Level 3 attempts to inspect hardware or raw spatial vectors."""
    pass

class ConsciousBlackboardProxy:
    """
    The sole I/O interface for Level 3 (Conscious Orchestrator).
    Enforces Phenomenological Transparency at the pointer level.
    """
    ALLOWED_READ_TABLES = {"INNER_VOICE_STREAM", "AUTOBIO_MEMORY"}
    ALLOWED_WRITE_TABLES = {"CONSCIOUS_INTENT", "AUTOBIO_MEMORY"}

    def __init__(self, db_conn: sqlite3.Connection):
        self._conn = db_conn

    def read_phenomenological_state(self) -> Dict[str, Any]:
        cursor = self._conn.cursor()
        # Level 3 can ONLY consume the latest narrative compiled by Level 2.5
        cursor.execute("""
            SELECT narrative_text, perceived_urgency_tier 
            FROM INNER_VOICE_STREAM 
            ORDER BY sequence_id DESC LIMIT 1
        """)
        row = cursor.fetchone()
        return {
            "inner_voice": row[0] if row else "Total inner stillness.",
            "urgency_tier": row[1] if row else "CALM"
        }

    def emit_conscious_intent(self, internal_monologue: str, bodily_desire: str, speech: str) -> None:
        cursor = self._conn.cursor()
        cursor.execute("""
            INSERT INTO CONSCIOUS_INTENT (internal_thought, bodily_desire, external_speech)
            VALUES (?, ?, ?)
        """, (internal_monologue, bodily_desire, speech))
        self._conn.commit()

3. From Physics to Thalamic Filtering (Levels 0 & 0.5)

Neurobiologist Joseph LeDoux proved that when sensory inputs enter the brain, the Thalamus (Level 0.5) bifurcates the signal:

The Low Road: Subcortical path to the Amygdala (Level 1) firing in $\sim 12\text{ ms}$. Triggers immediate hormonal shock before cognitive identification occurs.
The High Road: Cortical path to Level 2, extracting noise-filtered geometric descriptors for deliberate action.

Furthermore, to prevent immediate mortality before the network has compiled its first survival skills, we implement an Ontogenetic Indulgence Factor ($\Lambda_{\text{infant}}$) modeled on maternal mammalian care.

import math
import numpy as np
from dataclasses import dataclass, field
from typing import List, Tuple

@dataclass
class SomaState:
    """Raw Level 0 physical state (Strictly hidden from Level 3)."""
    pos: np.ndarray = field(default_factory=lambda: np.array([50.0, 50.0], dtype=np.float32))
    vel: np.ndarray = field(default_factory=lambda: np.array([0.0, 0.0], dtype=np.float32))
    heading_rad: float = 0.0
    energy_kcal: float = 100.0          # Zero -> Irreversible Wipe
    body_temp_c: float = 37.0           # Homeostatic optimum: 37.0°C
    tissue_integrity: float = 100.0
    age_seconds: float = 0.0

class GenesisPhysicsAndThalamus:
    def __init__(self):
        self.optimal_temp = 37.0
        self.maturation_tau = 1800.0    # 30-minute infant window

    def step_metabolism_and_thermodynamics(
        self, 
        soma: SomaState, 
        motor_force: np.ndarray,
        cognitive_flops_used: int,
        heat_sources: List[Tuple[np.ndarray, float]], 
        parent_pos: np.ndarray,
        dt: float = 0.05
    ) -> float:
        soma.age_seconds += dt

        # 1. Ontogenetic Indulgence (Thermodynamic Lactation)
        infant_lambda = math.exp(-soma.age_seconds / self.maturation_tau)

        # 2. Environmental Heat via 1/r² Radiation + Parental Thermal Hug
        ambient_temp = 12.0 + 8.0 * math.sin(2 * math.pi * soma.age_seconds / 600.0)
        local_temp = ambient_temp

        for src_pos, power_q in heat_sources:
            dist_sq = float(np.sum((soma.pos - src_pos) ** 2))
            local_temp += power_q / (1.0 + 0.18 * dist_sq)

        dist_to_parent = float(np.linalg.norm(soma.pos - parent_pos))
        if dist_to_parent <= 2.0:
            local_temp += 14.0 / (1.0 + 1.2 * (dist_to_parent ** 2))

        # Newton's Law of Cooling
        soma.body_temp_c += 0.08 * (local_temp - soma.body_temp_c) * dt

        # 3. Master Metabolic Equation
        lactation_intake = 0.12 * infant_lambda if dist_to_parent <= 1.2 else 0.0
        basal_cost = 0.05
        kinetic_cost = 0.02 * (float(np.linalg.norm(motor_force)) ** 1.5)
        
        # Quadratic cold thermogenesis
        cold_deficit = max(0.0, self.optimal_temp - soma.body_temp_c)
        thermo_cost = 0.04 * (cold_deficit ** 2)

        # Thinking carries a real metabolic price
        effective_cog_factor = 0.0001 * (1.0 - 0.75 * infant_lambda)
        cognitive_cost = effective_cog_factor * cognitive_flops_used

        d_energy = (lactation_intake - (basal_cost + kinetic_cost + thermo_cost + cognitive_cost)) * dt
        soma.energy_kcal = max(0.0, min(100.0, soma.energy_kcal + d_energy))

        # 4. Level 0.5 (Low Road): Fast pain trigger from thermal burn (>45°C)
        fast_pain_pulse = 0.0
        if local_temp > 45.0:
            fast_pain_pulse = min(1.0, (local_temp - 45.0) / 10.0)
            soma.tissue_integrity -= 2.5 * fast_pain_pulse * dt

        return fast_pain_pulse

4. Damasio's Somatic Marker & Dynamic LLM Mutation

In standard AI architectures, retrieving a vector memory does not mutate the inference state of the engine. In Project Ascendant, memory is an Affective Engram:

{
  "engram_sequence_id": 142,
  "compression_tier": 0,
  "vitality_score": 0.88,
  "semantic_narrative": "I moved toward the glowing rock in the north; sudden sharp impact on my side; skin burned while parent shouted caution.",
  "somatic_hormone_footprint": {
    "h_stress": 0.94,
    "h_reward": 0.02,
    "h_cohesion": 0.65,
    "h_curiosity": 0.00
  },
  "embedding_vector_quantized": [-0.142, 0.881, 0.034, "... (64 dims)"]
}

When retrieved, this engram re-injects an echo of its original hormone signature ($H_v$) into the live Limbic Bridge, modifying the Transformer's sampling temperature ($\tau_{\text{LLM}}$) and maximum token limit:

import numpy as np
from dataclasses import dataclass

@dataclass
class LimbicHormones:
    stress: float = 0.10      # Cortisol/Adrenaline [0.0 - 1.0]
    reward: float = 0.20      # Dopamine [0.0 - 1.0]
    cohesion: float = 0.80    # Oxytocin [0.0 - 1.0]
    curiosity: float = 0.50   # Noradrenaline [0.0 - 1.0]

def evoke_memory_with_somatic_marker(
    current_hormones: LimbicHormones, 
    retrieved_engram: dict, 
    echo_beta: float = 0.25
) -> dict:
    past_hv = retrieved_engram["somatic_hormone_footprint"]

    # 1. Somatic Reviviscence: Re-injection of physiological echo
    current_hormones.stress = (1.0 - echo_beta) * current_hormones.stress + echo_beta * past_hv["h_stress"]
    current_hormones.reward = (1.0 - echo_beta) * current_hormones.reward + echo_beta * past_hv["h_reward"]
    current_hormones.cohesion = (1.0 - echo_beta) * current_hormones.cohesion + echo_beta * past_hv["h_cohesion"]

    # 2. Neuromodulatory Mutation of Level 3 Hyperparameters
    # High stress cuts cognitive budget (Flight or Fight reflex)
    if current_hormones.stress > 0.80:
        max_thinking_tokens = 16
        llm_temperature = 0.95 + 0.3 * (current_hormones.stress - 0.80) # Panic variance
    elif current_hormones.stress > 0.45:
        max_thinking_tokens = 36
        llm_temperature = 0.35  # Yerkes-Dodson optimum: focused & deterministic
    else:
        max_thinking_tokens = 64
        llm_temperature = 0.60 + 0.35 * current_hormones.curiosity # Creative associative zone

    return {
        "narrative_injected": retrieved_engram["semantic_narrative"],
        "modulated_hyperparams": {
            "temperature": round(llm_temperature, 3),
            "max_tokens": max_thinking_tokens
        }
    }

The Level 3 Ingestion Pipeline: How the Narrative Reaches the Weights

Engineers frequently ask: How does narrative_injected actually bind to the Micro-Transformer at 0.25 Hz without breaking Epistemic Closure?

In our prototype, Level 3 runs a strict 3-stage inference pipeline:

Context Assembly (Ephemeral Prompt Buffer):
Level 3 assembles an ephemeral sliding window (max 256 tokens) composed strictly of:
- System Persona Anchor: Pre-trained behavioral syntax from the Synthetic Corpus ("You are an organism situated in a physical environment...").
- Inner Voice Feed: The latest 1st-person string written by Level 2.5 ("My skin is cold; I feel an empty void inside; glowing shape visible ahead").
- Evoked Somatic Memory: The narrative_injected string returned by the retrieval step above.
Dynamic Sampling Modulation:
The inference call consumes the mutated temperature and max_tokens calculated above. If the agent is in mortal panic, temperature spikes to $1.2$ with max_tokens=16, forcing telegraphic, reactive action tokens. If calm, it samples at $0.6$ with max_tokens=64, enabling deliberation.
Structured Output Decoding:
The output is constrained via CFG (Context-Free Grammar) or regex masking to emit three distinct channels:
- [PENSAMIENTO] (Internal monologue for autobiographical storage).
- [INTENCIÓN] (Linguistic action vector parsed by Level 2's motor decoder: e.g., "approach heat slowly").
- [HABLA] (Acoustic broadcast emitted to the $1/d^2$ audio field).

There are no hardcoded conditionals like if danger: run(). The agent flees because recalling its previous trauma physically spikes its stress vector in the present, constricting its reasoning horizon and forcing an urgent reaction through its language weights.

5. The Rosetta Probe: Falsifying Emergent Metaphors (Level 4)

In mammalian brains (Naomi Eisenberger's fMRI studies), social isolation activates the exact same neural pain matrix as a physical burn.

In our architecture, prolonged isolation (decay of $h{\text{cohesion}}$) directly charges $h{\text{stress}}$. Inevitably, the entity will sit directly next to a heat source ($T{\text{body}} = 37.0^\circ\text{C}$), fully fed, but completely abandoned by its NPC Parent. It experiences acute visceral suffering ($h{\text{stress}} = 0.85, h_{\text{cohesion}} = 0.05$).

Because its Pure Synthetic Corpus contains zero psychological terms like "loneliness" or "sadness", its sub-word tokenizer combines physical roots to express its state:

"I feel a cold-shadow inside even though the stone burns."

To prove this is a genuine grounded metaphor rather than random token hallucination, Level 4 audits the utterance against a Quadruple Falsifiability Test:

import numpy as np
from collections import defaultdict
from typing import List

class RosettaFalsifiabilityAuditor:
    def __init__(self, seed_vocabulary: set):
        self.seed_vocab = seed_vocabulary
        self.candidate_log: dict[str, List[np.ndarray]] = defaultdict(list)
        self.certified_dictionary: dict[str, dict] = {}

    def audit_utterance(
        self, 
        tokens: List[str], 
        current_hv: np.ndarray, 
        body_temp_c: float,
        action_changed: bool
    ):
        for token in tokens:
            is_neologism = token not in self.seed_vocab
            is_thermal_metaphor = (token in {"cold", "freezing", "shiver"} and body_temp_c >= 36.5)

            if is_neologism or is_thermal_metaphor:
                if action_changed: # Pragmatic/behavioral efficacy
                    self.candidate_log[token].append(current_hv.copy())
                self._evaluate_candidate(token)

    def _evaluate_candidate(self, token: str):
        history = self.candidate_log[token]
        if len(history) < 3: # Diachronic consistency: minimum 3 episodes
            return

        hv_matrix = np.vstack(history)
        mean_variance = float(np.mean(np.var(hv_matrix, axis=0)))

        # Invariance check: Somatic footprint variance must stay below threshold
        if mean_variance < 0.08 and token not in self.certified_dictionary:
            centroid = np.mean(hv_matrix, axis=0)
            self.certified_dictionary[token] = {
                "occurrences": len(history),
                "somatic_variance": round(mean_variance, 5),
                "hormone_centroid_Hv": np.round(centroid, 3).tolist(),
                "status": "CERTIFIED_ABSTRACT_CONCEPT"
            }

Conclusion

Consciousness is not an esoteric property of carbon; it is the functional byproduct of an information-processing system trapped inside an operational bottleneck.

By constraining an agent within:

Continuous thermodynamics and finite metabolic budgets (Level 0),
Dual-pathway thalamic filtering (Level 0.5),
Homeostatic endocrine coupling (Level 1),
A first-person visceral inner voice (Level 2.5),
And an isolated decision cortex with affective memory engrams (Level 3),

we build a machine that does not pretend to feel because it read millions of books—it generates meaning because it has an internal balance to protect and everything to lose.

What are your thoughts on this architecture? Let's discuss in the comments below!

Why Your Product Design Is Costing You Users and You Do Not Know It

Tue, 20 Oct 2026 09:08:55 +0000

There is a version of your product that works perfectly. The features are built. The flows are connected. Every screen renders correctly on every device. You have tested it yourself, and it makes sense to you.

And users are still leaving.

Not because your product does not work. Because it does not communicate clearly enough for someone who is encountering it for the first time, under real conditions, with divided attention and no patience for confusion. The product works. The design is failing them, quietly, in ways that do not show up as errors and do not trigger support tickets. They just leave.

This is the most expensive kind of design problem because it is invisible until you look for it specifically. It does not announce itself as a design failure. It announces itself as a conversion rate that is lower than it should be, a retention number that does not reflect the quality of the product, a user acquisition cost that keeps climbing because you are filling a leaking bucket rather than fixing the leak.

This post is about what those failures look like in practice, why they are so easy to miss, and what it takes to fix them before they cost you another month of users who could have stayed.

Design problems do not announce themselves as design problems. They announce themselves as low conversion, poor retention, and users who churn after a single session without leaving a reason.

WHY GOOD PRODUCTS HAVE BAD DESIGN

The most common reason is that the people who built the product know it too well to see it clearly.

When you have been working on a product for months, every screen makes intuitive sense. You know what the button does. You know where to find the settings. You know that the empty state means you need to create something first. That knowledge is so deeply embedded that it is impossible to unknow it when you test the product yourself.

Your users do not have that knowledge. They arrive at your product with no context, a low tolerance for confusion, and a competing option one tap away. Every moment of uncertainty is a moment where they are deciding whether to push through or leave. Most of them leave without telling you why.

The second reason is that design quality is usually evaluated visually rather than functionally. Does it look good? Does it feel modern? Is the colour scheme consistent? These are valid questions but they are the wrong primary measure. The right primary measure is: does a user who has never seen this product before know what to do within the first ten seconds? Everything else is secondary to that.

THE DESIGN FAILURES THAT COST YOU THE MOST

No clear primary action

Every screen in your product should have one thing it is primarily asking the user to do. One button, one link, one action that is visually dominant and immediately obvious. When a screen has three buttons of equal size and equal visual weight, the user does not see three choices. They see a decision they have to make before they can proceed, and many of them will not make it.

A common version of this failure: an onboarding screen that asks the user to connect an integration, invite a team member, or complete their profile, presented as three equally prominent options. The user came to use the product, not to manage their account setup. The right design leads them to the value first and surfaces the setup options progressively. The wrong design stalls them at the threshold and calls it onboarding.

Forms that ask for too much too early

Every field you add to a form is a question you are asking the user to answer before they have received any value from your product. Early in a user journey, before trust is established, before the user has experienced what your product does for them, that question has a cost. The longer the form, the higher the cost, and the more users abandon before completing it.

A registration form that asks for name, email, password, company name, company size, role, and phone number is not collecting useful data. It is filtering out users who are not willing to invest that much before seeing anything. Most of them are not unwilling to give you that information eventually. They are unwilling to give it before they have a reason to trust you with it.

The fix is not to remove fields permanently. It is to defer them. Ask for the minimum required to create an account. Let the user experience the product. Ask for the rest when you have given them a reason to provide it.

Error messages that describe what went wrong without explaining what to do

Error messages are the moment a product most needs to communicate clearly and most often fails to. A user who hits an error is already frustrated. An error message that says "Invalid input" or "Something went wrong" gives them nothing to work with. They do not know what was invalid, what went wrong, or what action would resolve it. Many of them close the tab.

A form that rejects a password and says "Password does not meet requirements" without showing those requirements anywhere on the screen is a form that will lose a measurable percentage of its signups at that exact moment. A checkout that fails and says "Payment declined. Please try again" without suggesting an alternative payment method or explaining why the payment was declined is a checkout that is losing revenue at its most critical point.

Every error message in your product should answer three questions: what happened, why it happened, and what the user should do next. If it only answers the first one, it is an error message that creates friction rather than resolving it.

An error message that describes what went wrong without explaining what to do next is not helpful. It is a dead end dressed up as communication.

Products built by developers often have navigation structures that mirror the underlying data model or the feature development sequence. The sections are named for what they are technically rather than what they mean to the user. The hierarchy reflects the engineering architecture rather than the user journey.

A user looking for their billing information should not have to choose between Account, Settings, and Profile before finding it. A user trying to invite a team member should not have to navigate through three levels of menu to reach a function they will use regularly. Navigation that makes technical sense and user sense are different things, and conflating them is one of the most common sources of silent user frustration.

The test for navigation is simple: give the product to someone who has never used it and ask them to complete a specific task without guidance. Where they hesitate, where they click the wrong thing, where they express uncertainty, is where the navigation is failing them.

Empty states that leave users stranded

Every new user encounters your product in its empty state: no data, no history, no content, no evidence of what the product does or why they should keep going. This is the moment most products lose the users who were genuinely interested but needed one more signal to commit.

An empty state that shows a blank screen with a small grey message that says "No items yet" is an empty state that offers nothing. An empty state that shows what the product looks like when it is working, explains the first action the user should take, and makes that action immediately available is an empty state that converts curiosity into commitment.

The empty state is not an afterthought. It is the first real experience a new user has of your product in use. Designing it with the same care as the populated states is not extra work. It is the work that determines whether a new user becomes a retained user.

Mobile experiences that were designed on a desktop

A product designed on a large monitor by someone using a mouse will have interaction patterns optimised for that context. Hover states that reveal information. Small click targets that a cursor navigates precisely. Navigation placed at the top of the screen where a thumb cannot comfortably reach. Dense information layouts that make sense on a 27-inch display and become overwhelming on a 6-inch one.

When that product is used on a phone by someone in a real-world context, one-handed, distracted, in variable lighting, those decisions become friction. The user does not diagnose the problem as a design failure. They just find the product harder to use than they expected and stop using it.

If more than thirty percent of your users are on mobile and your product was not specifically designed for that experience, you have a gap between your actual user behaviour and your design assumptions that is costing you retention every day.

WHY THESE PROBLEMS ARE SO HARD TO SEE FROM INSIDE

The people closest to the product are the worst positioned to see these failures. Not because they lack skill, but because they lack distance. Every confusion that would stop a new user is invisible to someone who knows the product well enough to navigate around it unconsciously.

The most reliable way to see these problems is to watch a real user use the product for the first time without helping them. Not a user test with instructions and prompts. A genuine first-time session where you observe without intervening. What you see in thirty minutes of watching someone encounter your product fresh will surface more design problems than months of internal review.

The second most reliable method is your own data. Screens with high drop-off rates are screens with design problems. Forms with high abandonment rates have specific fields where users are stopping. Flows with low completion rates have a specific point where most users are leaving. The data does not tell you what the problem is. It tells you exactly where to look for it.

WHAT FIXING THESE PROBLEMS ACTUALLY REQUIRES

Most of the failures described in this post are not expensive to fix once they are identified. Clarifying a primary action is a design change, not a rebuild. Simplifying a registration form is a product decision, not a development project. Improving an error message is an afternoon of work. Redesigning an empty state is a focused sprint.

What they require is the willingness to look at the product through the eyes of someone who does not know it, to take the friction they experience seriously rather than explaining it away, and to treat the user experience as a product metric rather than an aesthetic concern.

The products that retain users are not always the ones with the most features or the best technology. They are the ones that make the first ten minutes feel effortless, because someone thought carefully about what a new user needs at each moment and designed for that rather than for the team that built it.

CLOSING THOUGHT

Your product is probably better than your metrics suggest. The gap between the product you built and the results you are seeing is often a design gap rather than a feature gap. Before adding the next feature, it is worth asking whether the features you already have are communicating clearly enough for the users who are encountering them for the first time. The answer, in most products, is that they are not. And the cost of that gap compounds with every user you acquire into it.

If your product is losing users at a specific point in the flow and you are not sure why, that is exactly the kind of problem I work on. Tell me about what you are seeing and I can tell you whether design is where the answer lives.

Design a rate limiter, from fixed window to token bucket

Wed, 07 Oct 2026 18:46:15 +0000

100 requests per minute, and 200 got through

I first published this on my blog, where you can pause and play with every animation: https://techinpencil.com/rate-limiter/

A fixed window limiter set to 100 requests per minute

The limiter in the picture above is set to 100 requests per minute, and it just let 200 through in under two seconds. It isn't broken. It's the simplest rate limiter there is, and a lot of real code works exactly like it: count requests per clock minute, wipe the count when the minute changes. A client that sends right before the change and right after it gets twice the limit through.

This is also a classic interview question. You're asked to design a rate limiter, you sketch a counter, and the interviewer asks what happens at 0:59. I'll follow that conversation here. Each section starts with the next question, and the pictures are frames from the animations. On my blog you can play with them and see where they break.

A counter and a clock

The interviewer asks: “Design a rate limiter for our public API. 100 requests per minute per API key.”

The first answer most people give is a good place to start. Keep one counter per API key per minute. When a request comes in, look up the counter for the current minute. If it's under the limit, add one and let the request through. If it's at the limit, answer with HTTP 429 Too Many Requests [1].

function allow(key, now) {
  const minute = Math.floor(now / 60_000);
  const count = counters.get(`${key}:${minute}`) ?? 0;
  if (count >= LIMIT) return false;
  counters.set(`${key}:${minute}`, count + 1);
  return true;
}

That's a fixed window. You need one integer per client, and old counters just expire. The picture below uses a limit of 10 per minute so you can count the slips. On the blog you can send your own requests, or switch the traffic to “right at 1:00”.

Fixed window, limit 10 per minute

Watch the yellow band. With steady traffic it never goes above 10. With traffic bunched around the boundary it shows 20: ten just before 1:00 and ten just after. Figma's engineering blog shows the same thing with a limit of 5 per minute [2]. Five requests at 11:00:59, five more at 11:01:00, and all ten go through.

How bad that is depends on what the limit protects. A 2x spike for a second or two is fine if you just want to stop a script from hammering you all day. It's a real problem if the thing behind the limiter can't take more than 100 a minute.

Remember every request

The interviewer asks: “OK, the boundary leaks. Can you make it exact?”

Stop thinking in clock minutes. Instead of “requests this minute”, count “requests in the last 60 seconds”, measured back from right now. For that you need the time of every request you let through. That list is the sliding log.

function allow(key, now) {
  const log = logs.get(key) ?? [];
  while (log.length && log[0] <= now - 60_000) log.shift();
  if (log.length >= LIMIT) return false;
  log.push(now);
  logs.set(key, log);
  return true;
}

The next picture gets the same bunched traffic. The bracket is the last 60 seconds, and a timestamp gets crossed off when it falls out of it.

Sliding log, limit 10 per minute

It never lets more than 10 through in any 60 seconds. What it costs you is memory: each client now needs a timestamp for every allowed request instead of one integer. At 100 per minute that's up to 100 timestamps per client, 8 bytes each (a 64-bit timestamp in milliseconds), so a million active clients need 800 MB just for timestamps. Figma did this math for a limit of 500 requests a day and got 5 million stored values for 10,000 users [2].

Two counters and a guess

The interviewer asks: “Can you get close to that without storing every timestamp?”

Keep the per-minute counters from the first version, but when a request arrives, look at two of them: this minute's and last minute's. Then pretend last minute's requests were spread evenly across it. If we're 15 seconds into this minute, the last 60 seconds still include the final 45 seconds of the previous one, so you count 75% of it.

const elapsed = now % 60_000;
const estimate = previous * (60_000 - elapsed) / 60_000 + current;
if (estimate + 1 > LIMIT) return false;

Cloudflare described this in 2017 with a worked example [3]: a limit of 50 per minute, 42 requests in the previous minute and 18 so far in the current one, 15 seconds in: 42 × 0.75 + 18 = 49.5. Figma wrote about something similar the same year [2]. For an hourly limit they keep 60 one-minute counters and add them up, which means more counters to store but less guessing.

In the next one, the clock moves through minute 2, and you pick how minute 1's traffic really came in. The dashed slips under the line are what the estimate assumes, and the solid ones are what happened.

Sliding window counter, limit 10 per minute

If minute 1's traffic came early, the guess still counts requests that are already older than 60 seconds, so it's too strict. If it came late, the guess spreads them out, counts too few, and lets a few extra through. So you store two integers per client instead of a whole log, and the count is sometimes a bit off.

Cloudflare measured how far off on real traffic: 400 million requests from 270,000 sources. The estimate missed the real rate by 6% on average, but that only changes the answer for clients sitting close to the limit, so just 0.003% of requests were wrongly allowed or wrongly limited [3]. Their counters lived in a Memcached fork with simple commands like GET, SET and INCR, and that's all this approach needs.

A bucket of tokens

The interviewer asks: “Some clients send bursts. A page load fires 20 API calls at once. Can you allow that without raising the average rate?”

Every limiter so far counts the requests from the last minute. A token bucket doesn't count requests at all. Each client gets a bucket that holds up to N tokens, and tokens drip in at a fixed rate. A request takes one token, and if there's none left it gets a 429. A full bucket lets a burst of N through at once, and after that the client gets exactly the refill rate.

The drip doesn't need a timer or a background job. You store two numbers per client, the token count and the time you last looked, and top the bucket up when the next request shows up:

function allow(b, now) {
  const refill = (now - b.last) / 1000 * RATE;
  b.tokens = Math.min(CAPACITY, b.tokens + refill);
  b.last = now;
  if (b.tokens < 1) return false;
  b.tokens -= 1;
  return true;
}

In the picture, the coin forming under the tap is that top-up. On the blog you can try a bucket of 1 with bursty traffic, then a bucket of 12.

]
Token bucket

So there are two settings instead of one: the bucket size decides how big a burst you accept, and the rate decides the average. Stripe wrote in 2017 that they use a token bucket for their API, with the limiters backed by Redis [4]. AWS API Gateway documents its throttling the same way: the rate is tokens added per second, and the burst is the size of the bucket [5].

It doesn't fix 0:59 on its own, though. A bucket of 100 that refills at 100 per minute can still let about 200 through in 60 seconds: the 100 it saved up, plus 100 more dripping in. If a hard ceiling per minute matters, keep the bucket small compared to the rate.

A bucket with a hole

The interviewer asks: “The payment provider behind this API accepts 2 calls per second. Bursts or not. Now what?”

The leaky bucket is the other famous bucket, and the name gets used for two different things. In the first one, the requests themselves go into the bucket and leak out of a hole at the bottom at a steady rate. The bucket is a queue. A burst doesn't get rejected, it waits its turn, and you only answer 429 when the queue is full.

The picture below shows the same bursts going through both, with the same size and rate: the token bucket on top, the queue below.

Token bucket and leaky bucket, same size and rate

The token bucket passes the burst straight to the server. The queue turns it into an even drip, but requests have to wait: here the last one in each burst waits almost 3 seconds. That's what you want in front of something that really can't take a burst, like that payment provider. NGINX's limit_req works this way when you set burst and leave out nodelay: extra requests are delayed and only rejected once they go past the burst size [6]. It answers with 503 by default, so if you want a 429 you have to set limit_req_status.

The second meaning counts instead of queueing. Shopify's API docs describe it with marbles: every request drops a marble into the bucket, the bucket leaks at a fixed rate, and a full bucket means 429 [7]. Nothing waits in this one. Turn it upside down and you have a token bucket, because marbles in the bucket are just tokens missing from it. For the same size and rate, the two make the same decision on every request.

Many servers, one limit

The interviewer asks: “You have 20 API servers behind a load balancer. Where does the counter live?”

A counter in each server's memory doesn't work. A client whose requests get spread over 20 servers can get up to 20 times the limit, so 2,000 a minute instead of 100. The counter has to move to a shared store, often Redis, and that opens a new way to get it wrong. Here it is step by step:

Two servers, one counter

Read, decide, write is three steps, and another server can slip in between any two of them. With thousands of requests per second on the same key, it will. The fix is to let the store read and write in one operation. For a counter that's INCR, which adds one and returns the new value, and you decide based on what it returns [8]. For a token bucket, where the update is more than adding one, the usual answer is a small Lua script. Redis runs a script from start to finish without running anything else in between [9].

Even the Redis docs point out a race in one of their own rate limiter examples: a client that crashes between INCR and EXPIRE leaves a counter with no expiry, which the docs call a leak. Their fix is a Lua script too [8].

Saying no politely

The interviewer asks: “What do you send back when you say no?”

Status 429 Too Many Requests, from RFC 6585 [1]. The same section says the response may include a Retry-After header, which tells the client how long to wait, and RFC 9110 defines it as a whole number of seconds or an HTTP date [10]. A client that respects it stops hammering you. RFC 6585 also says caches must not store a 429.

GitHub goes further and tells you where you stand on every response, not only when you're blocked [11]. Authenticated users get 5,000 requests an hour, and the headers say how many are left and when the count resets, in UTC epoch seconds:

HTTP/1.1 200 OK
x-ratelimit-limit: 5000
x-ratelimit-remaining: 4987
x-ratelimit-used: 13
x-ratelimit-reset: 1790001200

There's an IETF draft to standardize this with two headers, RateLimit and RateLimit-Policy [12]. It's still a draft, version 11 as of May 2026. Plenty of blog posts show the older RateLimit-Limit and RateLimit-Remaining names, which were replaced in 2023.

Side by side

If I had to pick one without knowing more, I'd take the token bucket with a small bucket. It stores two numbers per client, and you decide up front how big a burst is OK.

Sources

Nottingham, M. and R. Fielding, Additional HTTP Status Codes, RFC 6585, April 2012. Section 4.
Mahdi, N., An alternative approach to rate limiting, Figma blog, April 2017.
Desgats, J., How we built rate limiting capable of scaling to millions of domains, Cloudflare blog, June 2017.
Tarjan, P., Scaling your API with rate limiters, Stripe blog, March 2017.
Amazon Web Services, Throttle requests to your REST APIs, Amazon API Gateway Developer Guide.
NGINX, Module ngx_http_limit_req_module, nginx.org documentation.
Shopify, Shopify API limits, Shopify developer documentation.
Redis, INCR, command documentation. See the rate limiter patterns.
Redis, Scripting with Lua, Redis documentation.
Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., HTTP Semantics, STD 97, RFC 9110, June 2022. Section 10.2.3.
GitHub, Rate limits for the REST API, GitHub Docs.
IETF HTTPAPI Working Group, RateLimit header fields for HTTP, Internet-Draft draft-ietf-httpapi-ratelimit-headers-11, May 2026.

All seven animations, interactive: https://techinpencil.com/rate-limiter/

Good Architecture Starts with Trust

Mon, 31 Aug 2026 02:56:28 +0000

Over the last few articles, I've written about architecture from several different angles.

Good architecture creates engineering momentum because it helps teams make better decisions early. It includes observability because systems can't be improved if they can't be understood. It plans for failure because production always introduces uncertainty. And it supports change because software that cannot evolve eventually stops delivering value.

As I thought about how to conclude this series, I realized something.

None of those ideas stand on their own.

They're all really about one thing.

Trust.

Not just trust between people, although that's certainly part of it. The deeper kind of trust—the confidence that comes from knowing your system, your team, and your engineering practices are working together to move in the right direction.

When I look back over the systems I've been most proud of, that confidence is what stands out.

I trusted that another engineer could understand the code I had written. I trusted that a deployment would behave the way we expected. I trusted that if something failed, we had the visibility to understand it and the preparation to recover from it. I trusted that six months later the team could continue building without feeling trapped by the decisions we made today.

Looking back, that confidence wasn't accidental.

It was designed.

Trust in the System

One of the biggest misconceptions about architecture is that it exists to organize software.

I don't think that's its real purpose.

I think architecture exists to create confidence.

When engineers trust the system they're building, they work differently. They make changes with less hesitation because they understand the impact of those changes. They release more confidently because they know what success—and failure—looks like. They spend less time wondering whether the platform can support the next feature and more time focusing on solving the customer's problem.

That confidence comes from intentional decisions.

Observability builds trust because the system tells the truth about itself. Failure planning builds trust because the team knows what happens when things go wrong. Designing for change builds trust because today's solution doesn't become tomorrow's obstacle.

Those practices aren't independent ideas.

They're all investments in confidence.

Trust in the Team

Of course, software doesn't build itself.

People do.

And the quality of an architecture is often a reflection of how well those people work together.

I've worked with incredibly talented engineers who struggled to build great systems because every technical discussion became a debate to win. Ideas were defended instead of challenged. Questions went unasked because no one wanted to appear inexperienced. Problems stayed hidden because admitting uncertainty felt risky.

I've also worked with teams that, on paper, looked far less impressive.

Yet they built remarkable software.

Not because every engineer knew the perfect answer, but because they trusted one another enough to search for it together.

They questioned assumptions without questioning each other.

They admitted mistakes without fearing blame.

They shared ownership instead of protecting territory.

They understood that the best architecture rarely comes from the smartest person in the room.

It comes from a room where everyone feels safe enough to make the architecture better.

Trust in Leadership

The longer I've led engineering teams, the more I've realized that leadership has very little to do with having the best technical answers.

Leadership creates the conditions where the best answers can emerge.

That means creating an environment where engineers can challenge ideas respectfully. Where junior engineers are encouraged to ask "why?" without hesitation. Where changing your mind is viewed as learning instead of weakness. Where production incidents become opportunities to improve the system instead of opportunities to assign blame.

Leaders influence architecture long before anyone opens a design document.

They shape how decisions are made.

They shape how disagreements are handled.

They shape whether people optimize for being right or for finding the right answer.

Over time, those habits become part of the architecture itself.

Trust in the Future

Perhaps the greatest test of an architecture isn't what it looks like on launch day.

It's what it feels like years later.

Every engineer eventually moves to a new project.

Every team evolves.

Every leader is eventually replaced.

Good architecture should outlast all of us.

When another engineer inherits a system, they shouldn't feel like they're walking into someone else's maze. They should feel like they're joining a conversation that was documented, intentional, and left in good faith.

That's one of the reasons I believe documentation, mentoring, code reviews, architecture discussions, and thoughtful design matter so much.

They're acts of stewardship.

They say to the next engineer:

"We thought about this. We wanted to leave you something you could trust."

To me, that's one of the highest forms of professionalism in software engineering.

This Is What I Hope M²S² Becomes

When I think about the future of M²S² Engineering Group, I don't think about becoming the largest consultancy.

I think about becoming one of the most trusted.

A group of engineers who are known for telling the truth, even when the answer is uncomfortable. Who challenge assumptions because they care about the outcome, not because they want to win an argument. Who are willing to admit they don't know something and curious enough to find the answer together.

I want M²S² to be a place where clients trust the systems we help build because they trust the people building them.

More importantly, I want every engineer who becomes part of M²S² to leave an organization stronger than they found it—not just by delivering software, but by sharing knowledge, mentoring others, improving engineering practices, and helping teams gain confidence in their own ability to solve difficult problems.

That kind of impact lasts longer than any framework, programming language, or technology trend.

Bringing the Series Full Circle

This series began with engineering momentum.

I still believe momentum is one of the most valuable things an engineering organization can have.

But momentum doesn't begin with architecture diagrams.

It begins with trust.

Trust in the system.

Trust in the people.

Trust in leadership.

Trust that the work you're doing today won't become tomorrow's burden.

Everything we've discussed in this series—observability, resilience, adaptability, and thoughtful design—ultimately serves that purpose.

Not to build perfect software, but to build confidence.

Because when engineers trust the systems they're building and the people they're building them with, they make better decisions. Better decisions create better architecture. Better architecture creates momentum.

And momentum is what allows great engineering organizations to keep building long after the first release.

Thank you for following along through this series.

My hope isn't that these articles convince anyone there's only one way to build software. Architecture has always been about tradeoffs, context, and the people making the decisions.

If this series has a single message, it's this:

Technology changes. Teams change. Businesses change.

Trust is what allows all of them to move forward together.

That's the kind of engineering culture I believe in.

And that's the kind of company I'm building at M²S² Engineering Group.

Why I'm Building an Engineering Publishing Platform

Sun, 30 Aug 2026 02:46:00 +0000

Over the past several months, I've made a commitment to write more.

Not because I want to become a content creator, but because I've come to appreciate how much engineering knowledge quietly disappears every day. Architecture discussions end when the meeting is over. Production incidents become stories that fade with time. Lessons learned on one project rarely make it to the next. As engineers, we spend years building experience, but very little of it ever becomes knowledge that others can benefit from.

Writing has become my way of changing that.

Like many engineers, I started publishing wherever I could. I have a blog on my own website, along with accounts on Medium, Hashnode, DEV.to, and Coder Legion. Each platform reaches a different audience, and I genuinely like the idea of meeting engineers where they already are. If someone prefers reading on Hashnode instead of my website, that's perfectly fine with me. My goal has never been to drive traffic to a particular platform. It's to share ideas.

Unfortunately, that turned out to be much harder than I expected.

Every article followed the same routine. Once the writing was finished, I still had to copy Markdown between editors, upload images multiple times, recreate tags, tweak formatting, update SEO metadata, generate excerpts, and create social posts. By the time I finished publishing everywhere, I had spent almost as much time managing content as I had writing it.

As an engineer, my first instinct was simple.

Automate it.

I envisioned a small service that would let me publish an article once and have it automatically appear everywhere else. My website would remain the source of truth while Medium, Hashnode, DEV.to, and every future platform would simply become destinations. It felt like a straightforward integration problem—build a few connectors, push the content out, and move on to the next article.

Then I started designing it.

Like many software projects, it didn't take long before I realized I was solving the wrong problem.

The more I thought about the architecture, the less it resembled a collection of publishing scripts. If every platform was going to behave differently, I needed a common deployment model instead of a collection of one-off integrations. Articles would become immutable releases. Releases could be deployed to one or more publishing targets. Each destination would have its own connector responsible for understanding that platform, while the orchestration layer would manage deployments, retries, status, and lifecycle. The result was an architecture that felt much more like a software delivery pipeline than a blogging tool.

That solved the technical problem.

It didn't solve the real one.

Somewhere in the middle of working through the design, I realized that publishing was never the difficult part.

Thinking is.

Every article I've written over the past few months has started long before a Markdown file ever existed. Sometimes it begins with a conversation. Sometimes it's a lesson I didn't fully appreciate until years later. Other times it's an architectural decision that seemed obvious in hindsight but required hours of discussion to reach. The words themselves are usually the final step in a much longer process of reflection.

That's the process I don't want to lose.

Most writing tools begin with a blank editor. They assume you already know exactly what you want to say. My experience has been almost the opposite. The most valuable part of writing is discovering the idea. It's connecting experiences that didn't seem related, questioning assumptions, and slowly uncovering the principle hidden underneath the story. By the time I begin writing, the real work has already happened.

That realization changed the project entirely.

I'm still building the cross-posting platform because I genuinely want to publish once and distribute everywhere with as little manual effort as possible. But that's no longer the vision.

The vision is to build a platform that supports the entire journey from idea to publication. Conversations become drafts. Drafts become articles. Articles become releases that can be published anywhere. The platform should help organize ideas, generate summaries and metadata, manage revisions, and eventually understand the relationships between everything I've written. I don't want another tool that stores Markdown files. I want a system that helps preserve engineering judgment.

Over the coming weeks, I'll be documenting the architecture, implementation, tradeoffs, and inevitable mistakes as I build this into the M²S² platform. Some articles will be technical deep dives into the connector architecture or deployment pipeline. Others will focus on the decisions that shaped the system and the lessons learned along the way. My hope is that readers don't just see the finished product—they understand the thinking behind it.

If there's one thing I've learned throughout my career, it's that good engineering rarely starts with the perfect solution. More often, it starts with solving one problem, discovering a better question, and allowing the architecture to evolve alongside your understanding.

That's exactly what happened here.

I thought I was building a way to publish blog posts.

Instead, I found myself designing a platform for capturing and sharing engineering experience.

And I think that's a much more interesting problem to solve.

Adding Edges Is Not a Paradigm Shift

Mon, 17 Aug 2026 19:16:15 +0000

I read the post, agreed with it, and opened my repo to plan the migration.

Then I could not find the loop I was supposed to be replacing.

The repo is agentsmyth, a seven-step agent chain I have been building for months. What I found was a fixed sequence with a checkpoint between every step and exactly one edge leaving every node. There was no loop to tear out and no obvious place to put a graph. I spent an evening trying to work out what I had misunderstood, and eventually went back to read the original post properly.

It was a joke.

The Joke Was the Point

Here is the sequence, because that sequence is the story.

At 00:34 UTC on July 18, 2026, Peter Steinberger, the creator of OpenClaw, posted twelve words: "Are we still talking loops or did we shift to graphs yet?" No definition. No design principles. One line.

About four and a half hours later, Hamel Husain published a piece declaring loop engineering dead. The next day the slogan was repeated, and a replacement narrative was moving before the replacement had a stable meaning. Within weeks there were guides, comparison tables, and a discipline with a name.

Steinberger was not proposing a discipline. He was needling the field for how fast it renames things. Prompt engineering had become context engineering, which had become harness engineering, which had become loop engineering, and the underlying work had not changed nearly as much as the vocabulary had. He later posted a diagram of a two-state machine, looping and done, with a caption that left nothing to interpret:

"This is the silly thing you all hyped for weeks."

The joke was the point, and the field wrote explainers about the punchline.

Harrison Chase, who created LangGraph, which is the system nearly every one of those explainers pointed to as the destination, responded to the wave by saying he did not know what graph engineering was and still did not, but that it was basically just LangGraph.

Four days after that, he and Sydney Runkle published the piece that should have ended it. Three years of building graph agent systems, laid out plainly. Their position is not that graphs are overrated. It is that this was never a migration. Production agents need cycles, because they retry failed calls, ask for missing information, correct themselves after validation and pause for human input. A loop is a directed cyclic graph, a point they credit to David Khourshid. Loop engineering is not the alternative to graph engineering. It is the simple case of it.

The LangChain framework itself runs on top of LangGraph, which means the migration a lot of people were planning had already happened underneath them, three years ago, without a name.

The prior art runs deeper still. Anthropic documented prompt chaining, routing, parallelization, orchestrator-worker and evaluator-optimizer patterns back in 2024, with the recommendation to add complexity only when something demands it. LangGraph, AutoGen and Google ADK all shipped graph workflows before the word trended.

So there was no paradigm to shift to. My chain is a graph. It always was one, with a single edge leaving every node.

One more detail, and it is the one that stayed with me. The explainers cannot agree on how long the tweet was. One says six words. Another says nine. It is twelve. View counts range from 575,000 to 2.9 million depending on who is telling you. Dozens of articles were written about a single sentence, and a measurable share of them did not count the words in it.

Nobody checked the tweet.

That is the thing I actually want to be careful about. Not that the term is empty, because underneath it there is a real design problem about how multiple loops connect. But that a field can produce a month of confident writing on top of a line nobody read closely, and I was four paragraphs into planning a migration before I noticed I was one of them.

An Edge You Did Not Need Is a Checkpoint You Gave Away

That leaves the question worth asking, which is smaller and harder than the one I started with. Not whether to adopt graphs. What each additional edge is going to cost me.

Here is the shape of what I built. A fixed sequence of seven steps, running from specification through planning, implementation, review, testing and release. One agent chain. A checkpoint between every step, where each step reads the written output of the one before it rather than the conversation. One human reviewer, which is me, standing at every checkpoint.

Now put a bad spec into it. Not a broken one. A spec that is coherent, confident, and pointed slightly wrong, which is the only kind that survives long enough to hurt.

In the chain, that spec meets a checkpoint before anything is built. If I catch it, the cost is one step's work. If I miss it, it moves exactly one step, and there is another checkpoint waiting.

Add edges and the same spec splits three ways. Three nodes build on it in parallel, each one confidently, each one correct with respect to the thing it was handed. The error does not surface at a checkpoint, because the spec already passed the only checkpoint it was going to meet. It surfaces at convergence, when the branches come back and do not fit each other.

By then the work exists. All of it.

There is an obvious objection here, and it is correct.

Nothing about a graph forbids checkpoints. LangGraph ships interrupts for exactly this, and human-in-the-loop pausing is a documented feature rather than an afterthought. You can gate every node in a branching system if you decide to, and serious teams do.

So the claim is narrower than my diagram makes it look. A chain has one place to stand. A graph has as many places as it has branches, and every one of them has to be specified, staffed and waited on. Gating is not free in either shape. It is just cheap enough in a chain that I had stopped noticing I was paying for it.

What I keep finding in the posts recommending the move is edges being added and checkpoints not being. The topology gets the diagram. The gating gets left as an exercise.

Every branch that runs unattended is a checkpoint I chose not to stand at.

I have hit this failure before, at chain scale, and it was mine. Last year I handed an agent a global event mechanism across a React Native app. I wrote the plan, reviewed it, approved it. The agent implemented the approved plan correctly, and the app broke in places the plan had quietly skipped: passive components sitting outside the event scope, and a duplicate event firing that I only found by adding console logs to trace the flow.

Three days of fixing. The result landed around seventy to seventy-five percent aligned with what I had actually asked for.

That was one agent, one sequence, one plan. The failure was never capability. The plan had a hole and nothing in the shape of the process was built to find it before the building started.

The checkpoint was there. I was standing at it. It still went through.

That is the part that decides this. A checkpoint I miss costs one step, and I found the hole in three days because there was only one path to trace back along. I do not need to imagine what the same missed spec costs with three branches running through it at once.

The Condition I Have Not Met

There is a clean version of when to add the edges, and it comes from people inside the discipline rather than from anyone selling the word. Add them when the path has to branch. When work has to route between multiple agents. When the run has to survive a human sitting in the middle of it.

Against my system, nothing branches and there is one agent chain. The third one I arguably meet already, since I stop for a human at every step. Call it one for three.

I should be plain about the limit of my standing here. I have not built a multi-agent graph yet. This is an argument about whether I should, not a report from the other side of the decision. The people who have built them know things I do not (And I know that few of them would claim that "Loop engineering is dead!" because they know everything around it), and the LangChain writeup is the best evidence of that: their own team built early deep research on a predefined graph, then moved it toward a looser agentic loop, because the work would not sit still inside fixed nodes. They migrated away from a graph, on their own product, and said so publicly.

Structure you have not earned costs you. That holds in both directions.

So I am not adding graph support to agentsmyth. Not because graphs are hype, and not because the people building them are wrong. Because an edge is a complexity decision like any other. It pays back only when the work genuinely cannot be sequenced, and I have not brought it work that qualifies.

Processes like this do not usually die from being wrong. They die from friction.

Every edge is a place the thing can catch.

What would flip it is specific, and I want to name it so I am held to it. A task where two parts genuinely cannot be sequenced, where waiting on the first is real waste and not just impatience, and where I can specify both halves well enough that they still fit when they come back. That third clause is the hard one. It is also the one nobody is talking about, because it is not a topology problem. It is the same specification problem, made harder by being cut in two.

The edges were never the interesting part. The spec was.

The seven-step chain this article describes is agentsmyth. Install with npx agentsmyth init.

Do Not Rewrite the Legacy System. Contain It.

Fri, 31 Jul 2026 08:48:59 +0000

Understand the existing design, then isolate the change, design it cleanly, and implement it with TDD, ATDD and integration testing.

Legacy systems usually do not need to be rewritten entirely. More often, a few parts create most of the pain, while the rest still works well enough.

I start with a coarse view of the whole system (C4 models). The goal is not to document everything in detail, but to understand the main components, dependencies, integrations, ownership boundaries and finally to document the problems with the as-is implementation.

Then, together with stakeholders, I identify which parts are most critical to change and assign priorities. Only after that do I start working on a concrete part.

For the selected part, I explore the current implementation, analyse its behavior and document the design that actually exists, then create a coarse to-be C4 models and a more detailed UML models. Then I define a boundary around the part and introduce adapters that isolate the new design from the remaining legacy code, design the replacement cleanly and implement it with TDD.

The same process can then be repeated for the next priority.

The result is not an instant clean architecture. It is a system where more and more parts become understandable, documented and safe to change, one boundary at a time.

This is the kind of legacy modernization work I specialize in: recovering design, containing change, and improving the system one boundary at a time.

It's Been a While, Here's What I've Been Building

Sat, 18 Jul 2026 18:03:45 +0000

It's been a while since I last published anything here on CoderLegion.

I didn't stop building, I just got so deep into projects that writing took a back seat. So I thought it was time to share what I've been working on over the past few months.

🚀 Building Afia9jaVTU

One of the biggest projects I've worked on recently is Afia9jaVTU.

The goal wasn't just to create another VTU platform. I wanted to build something that could serve as a reliable foundation for digital services like airtime purchases, data subscriptions, wallet management, virtual accounts, bill payments, and more.

Working on it pushed me to think beyond writing APIs. I had to consider system architecture, payment reliability, background jobs, user experience, and how to build software that can scale.

It reminded me that software engineering is just as much about solving business problems as it is about writing clean code.

💳 Building ALATPay for Laravel

Another project that kept me busy was creating ALATPay for Laravel.

I noticed there wasn't a modern Laravel package that made integrating ALATPay straightforward for Laravel developers.

So I built one.

The package handles much of the repetitive work developers face when integrating payments, allowing them to focus on their applications instead of boilerplate integration code.

Building an open-source package is a different experience from building client software.

You have to think about:

Developer experience
Documentation
Backward compatibility
Clean APIs
Long term maintenance

It's a challenge I genuinely enjoyed.

🌐 Solving CGNAT Problems with CGNAT Inspector

If you've ever hosted services from home, you've probably encountered CGNAT.

It can be frustrating trying to figure out why port forwarding simply refuses to work.

That's why I built CGNAT Inspector.

The goal is simple:

Help developers and homelab enthusiasts quickly determine whether they're behind Carrier Grade NAT and understand what that means for self-hosting.

Sometimes the best projects come from solving your own problems.

🛠️ Experimenting with Developer Tools

Lately I've also been experimenting with building tools specifically for developers.

Small utilities.

Automation.

CLI ideas.

Infrastructure helpers.

Things that remove friction from everyday development.

Not every experiment becomes a finished product, but every experiment teaches something valuable.

🏢 Growing Royal Bcode Ventures

Outside open source, I've also been investing a lot of time into growing Royal Bcode Ventures (RBV).

Running a software company is a completely different challenge from writing software.

You're thinking about:

Finding clients
Delivering projects
Infrastructure
Operations
Customer support
Marketing
Finance
Team growth

It has taught me that building a business requires a very different mindset from building applications.

Both are difficult but still rewarding.

📡 Working on SmartAlert-T (SAT)

Another exciting project is SmartAlert-T (SAT).

The vision is to build a platform focused on intelligent monitoring and alerting.

Whether it's servers, infrastructure, or critical systems, timely notifications can make all the difference.

It's still evolving, but I'm excited about where it's headed.

📚 What I've Learned

Looking back, one thing stands out.

The best way to improve as a developer isn't by following every new trend.

It's by building.

Real products.

Real tools.

Real businesses.

Every project exposes new problems, and solving those problems teaches lessons no tutorial can.

What's Next?

I'm planning to become much more active here on CoderLegion again.

I'll be sharing:

Behind the scenes architecture decisions
Laravel tips
Open-source projects
Infrastructure experiments
Lessons from running a software business
Things that failed (because those are often the most educational)

If you've also been building something interesting, I'd love to hear about it in the comments.

Let's keep building.

Why Modern Engineering Teams Must Prioritize System Observability Over Raw Performance

Wed, 15 Jul 2026 14:45:00 +0000

Engineering teams often chase performance: faster APIs, lower latency, optimized queries, reduced CPU usage. But in 2026, performance alone is no longer the defining factor of a reliable system. Modern architectures — distributed, event‑driven, containerized, multi‑service — demand something far more fundamental: [system observability][1].

Performance tells you how fast your system is.
Observability tells you why it behaves the way it does.

And for engineering teams, especially small ones, this difference determines whether a system scales or collapses under pressure.

Performance Without Observability Is a Blindfold
A fast system can still be fragile.
A highly optimized service can still fail silently.
A low‑latency API can still degrade unpredictably under load.

When teams focus exclusively on performance metrics, they optimize symptoms instead of causes. They see spikes, drops, delays — but they don’t see the internal relationships that create them.

Observability exposes:

hidden dependencies

cascading failures

slow internal paths

unexpected retry storms

misconfigured backoff logic

silent timeouts

resource starvation

cross‑service bottlenecks

Without this visibility, engineers operate reactively. They fix what they see, not what actually breaks the system.

The Three Pillars of Observability
Modern observability is not just logs. It is a structured, multi‑layered system that allows engineers to understand why something happened.

Metrics — the heartbeat of the system
Metrics show trends, anomalies, and thresholds.
They reveal when something is off — but not why.

Key engineering metrics include:

request latency distribution

error rate spikes

queue depth growth

retry counts

CPU throttling events

memory pressure

cache hit ratios

Metrics are the first signal, not the full story.

Logs — the narrative
Logs provide context.
They explain the events behind the metrics.

But logs alone are insufficient.
Without structure, correlation, and context, logs become noise.

Modern teams rely on:

structured JSON logs

correlation IDs

distributed trace IDs

log sampling

log enrichment with metadata

Logs transform raw events into actionable insights.

Traces — the map of the system
Distributed tracing is the most powerful observability tool of 2026.

Traces show:

how a request travels through the system

where it slows down

where it fails

where retries accumulate

where bottlenecks form

where unexpected dependencies appear

Traces reveal the architecture as it actually behaves — not as it was designed.

Why Observability Matters More for Small Teams
Large companies have SRE departments, dedicated monitoring teams, and 24/7 on‑call rotations.
Small engineering teams do not.

For small teams, observability is not a luxury — it is a multiplier.

It allows them to:

detect issues before users notice

reduce on‑call stress

shorten incident resolution time

prevent cascading failures

scale without rewriting the system

understand real‑world behavior instead of assumptions

Observability replaces guesswork with clarity.

Observability Enables Predictable Engineering
Predictability is the most valuable trait of a modern system.

A predictable system:

fails gracefully

recovers automatically

scales linearly

behaves consistently under load

provides clear signals when something is wrong

Observability is the foundation of this predictability.
It transforms engineering from reactive firefighting into proactive system design.

The Future: Observability as a Core Engineering Skill
In 2026, observability is no longer a DevOps specialty.
It is a core engineering skill.

Teams that master observability:

ship faster

debug faster

scale safer

maintain stability

reduce operational cost

build systems that survive real‑world conditions

Performance matters.
But system observability determines whether performance is sustainable.

Why Local Webhook Testing Still Feels Broken

Wed, 15 Jul 2026 05:29:04 +0000

Building webhook integrations shouldn't require deploying your application every time you make a change.

One of the first things you discover when working with webhooks is that your local development environment isn't reachable from the internet.

Your application might be running perfectly on localhost:3000, but webhook providers like Stripe, GitHub, Clerk, Shopify, or Discord can't send requests to your machine.

So the usual workflow begins.

Start a tunnel.

Copy the public URL.

Update the webhook endpoint in the provider dashboard.

Restart the tunnel.

Copy a new URL.

Update the provider again.

Repeat.

It works.

But after doing it dozens of times across different projects, it starts to feel like unnecessary friction.

Local Development Shouldn't Slow You Down

When you're building a webhook integration, the goal is to iterate quickly.

Receive an event.

Inspect the payload.

Change some code.

Try again.

Instead, developers often find themselves spending just as much time managing tunnel URLs as writing application logic.

A tunnel expires.

The URL changes.

Every webhook provider now needs to be updated.

If you're integrating with multiple services at once, that overhead grows quickly.

It's a small problem that becomes surprisingly repetitive.

Tunnels Are More Than Just Public URLs

Most people think of a tunnel as a way to expose localhost.

That's true.

But for webhook development, it's also the bridge between your local application and the outside world.

It allows you to receive real production-like events without deploying your application every time you want to test a small change.

That feedback loop matters.

The faster you can receive, inspect, and replay events, the faster you can build reliable integrations.

Why I Built Tunnels into Hooktrace

While building Hooktrace, I realized developers were already using one tool to expose localhost and another to inspect webhook deliveries.

That felt fragmented.

Hooktrace already understands webhook events.

It already captures requests, payloads, headers, retries, and delivery timelines.

Adding tunnels means developers can expose a local endpoint and immediately start inspecting events from the same platform.

No context switching.

No juggling multiple dashboards.

Just one place to receive, inspect, debug, and replay webhook traffic.

It's Not About Replacing Existing Tools

There are already excellent tunneling tools available, and I've used many of them myself.

The goal isn't to replace them.

The goal is to make the webhook development experience feel more connected.

Instead of thinking about tunnels, event inspection, retries, aggregation, destinations, and observability as separate tools, I think they should work together.

That's the direction I'm taking with Hooktrace.

Building Hooktrace in Public

Every feature I build starts with a problem I've run into while building integrations or backend systems.

Sometimes it's webhook observability.

Sometimes it's event aggregation.

Sometimes it's routing events to different destinations.

This week, it was local webhook testing.

I'm documenting that journey in public—not just the finished product, but the engineering decisions, trade-offs, and lessons learned along the way.

If you're building event-driven systems or working with webhooks every day, I'd love to hear how you're solving these problems.

Good developer tools are shaped by real developer feedback.

Follow Along

I'm building Hooktrace in public and sharing the engineering journey one feature at a time.

🌐 Hooktrace: https://hooktrace.xyz

💻 Portfolio: https://www.codilad.dev

⭐ GitHub: https://github.com/hooktracehq/hooktrace

𝕏 Follow the build: https://x.com/yasirbuilds

Thanks for reading.

Good Architecture is Built for Change

Fri, 10 Jul 2026 20:02:49 +0000

Good architecture is not built around the assumption that everything will stay the same.

It is built around the reality that useful software will eventually need to change.

A team may start with a clear product direction, a clean workflow, and a system that fits the business well. Then the business learns something. A customer asks for a workflow that was not part of the original model. A market opportunity changes the roadmap. A technical decision that made sense early starts to feel tighter as the product grows.

That is normal.

The problem is not that change happens. The problem is when the system was built in a way that makes reasonable change harder than it needs to be.

Good architecture is not about predicting every future requirement. That usually leads to unnecessary abstractions, premature service boundaries, and complexity the team has not earned yet.

Good architecture is about making change less expensive when it arrives.

In the earlier posts in this series, I wrote about how good architecture creates engineering momentum by making important decisions visible early. I wrote about observability because teams need to understand and reason about systems in production. I wrote about failure planning because systems need to contain, recover from, and practice failure.

Change is the next piece.

A system can be observable, resilient, and well-built for its current purpose, but still become difficult to evolve if change was never part of the design conversation.

That is where architecture either protects momentum or slowly drains it.

Change Is the Real Test of Architecture

Most systems look cleanest when they are new.

The first version reflects the team’s best understanding at the time. The business rules are still fresh. The workflows are still easy to explain. The people who made the early decisions are usually still close to the work.

Then the system starts living in the real world.

The product grows. Customers use it in ways the team did not fully expect. Leadership sees new opportunities. Support learns where the workflow is confusing. Engineering discovers which assumptions were solid and which ones were only good enough for the first release.

That is when architecture gets tested.

In a system built for change, the team can usually find a reasonable place for the new behavior. The work may still be meaningful, but it does not feel mysterious. The structure of the system gives the team enough context to understand where the change belongs and what it might affect.

In a system built only for the first version, each new requirement starts pulling on unrelated parts of the codebase. A small change in one workflow creates risk somewhere else. Engineers become cautious because the system no longer explains itself clearly.

That is often how architectural pain begins.

Not with one dramatic failure, but with the steady increase in cost every time the business learns something new.

Future-Proofing Is the Wrong Goal

I am cautious about the phrase future-proof.

It sounds responsible, but it can lead teams toward the wrong kind of design. When teams try to account for every possible future, they often add complexity long before the system needs it. The code fills with abstractions. The architecture gains extension points no one uses. Services get split before the domain is well understood.

That kind of design can look sophisticated, but it often makes the current system harder to understand.

The better goal is not future-proofing.

The better goal is change tolerance.

A change-tolerant system does not pretend to know every future requirement. It simply avoids making today’s decisions so rigid that tomorrow’s reasonable changes become painful. It keeps the important concepts clear. It separates concerns where separation has value. It gives the team room to adapt without forcing every change through the most fragile part of the system.

That is a more practical standard.

Good architecture accepts that the future is uncertain, but it does not use uncertainty as an excuse for either overengineering or neglect. It aims for enough structure to support change without adding so much structure that the team can no longer move.

Coupling Is Where Change Gets Expensive

Most change pain comes from coupling.

Some coupling is expected. Parts of a system need to work together. The issue is not that relationships exist. The issue is when those relationships become unclear, accidental, or too expensive to change.

You can usually feel this in the work.

A product request sounds simple, but the implementation touches several unrelated areas. A business rule exists in more than one place, and no one is completely sure which version is authoritative. A database table becomes the shared contract for workflows that should not know so much about each other. A user interface depends on backend details that were never meant to be part of the contract.

At that point, the team is not only changing behavior. It is negotiating with hidden dependencies.

That is one of the clearest signs architecture is working against the team.

Poor architecture turns local changes into global events.

Good architecture does the opposite. It keeps related things close and unrelated things separate enough that change can happen where it belongs.

Boundaries Should Reflect the Business

One of the best ways to support change is to make sure the architecture reflects the business concepts the team is actually working with.

This does not mean every business concept needs its own microservice. Distribution is not the same thing as good boundaries. In many cases, a modular monolith with clear internal structure is easier to change than a set of services split too early around concepts the team does not fully understand yet.

The real question is whether responsibilities are clear.

When boundaries reflect the business, changes become easier to discuss. Teams can talk about the system in the same language as the product. Ownership becomes more natural. New behavior has a clearer place to land.

When boundaries are unclear, the system starts to resist change. Business rules spread across layers. Data ownership gets fuzzy. Teams spend more time deciding where work belongs than doing the work itself.

Good boundaries do not eliminate hard decisions.

They give teams a better starting point.

Good Architecture Makes Local Change Possible

A healthy architecture allows many changes to stay local.

That does not mean every change is small. Some business changes are genuinely large and deserve careful planning. But when architecture is doing its job, a change to one concept does not automatically force the team to rework unrelated parts of the system.

Local change matters because it preserves confidence.

When engineers understand the boundary around the work, they can reason about impact. They know where to test. They can communicate risk more clearly. They can move without needing to hold the entire system in their head.

That confidence compounds over time.

A team that can make local changes builds trust in the system. A team that has to coordinate every change across too many areas starts to slow down. Eventually, the team spends more energy protecting itself from the system than improving the system.

That is not just a tooling problem.

It is a design problem.

Good architecture gives change somewhere to go.

Contracts Help Systems Evolve

As systems grow, contracts become increasingly important.

An API contract, an event schema, a module boundary, or a data ownership rule is more than a technical detail. It is an agreement about how parts of the system interact.

Good contracts make change safer because they define expectations. They allow one part of the system to evolve without surprising every other part. They create a place to discuss compatibility, versioning, migration, and ownership.

Weak contracts create uncertainty.

When the contract is unclear, consumers begin depending on behavior that was never intended. Internal details become external assumptions. Changing one side of the relationship becomes risky because no one is sure who depends on what.

That is why compatibility matters.

A system built for change needs a way to evolve without breaking everything at once. Sometimes that means versioned APIs. Sometimes it means backward-compatible event schemas. Sometimes it means migration windows, deprecation paths, or adapters that allow old and new behavior to coexist for a while.

The specific technique depends on the system.

The principle is the same: change becomes safer when expectations are explicit.

Change Requires Discipline, Not Just Design

Architecture can make change easier, but it cannot carry the whole burden by itself.

A system that is built for change still needs engineering discipline. Tests need to give the team confidence that behavior still works. Observability needs to show whether changes behave correctly in production. Documentation needs to capture the decisions that matter. Ownership needs to be clear enough that changes have a path through the team.

Without those practices, even a well-designed system can degrade.

Boundaries blur when teams ignore them. Contracts weaken when compatibility is not respected. Tests lose value when they stop reflecting real behavior. Documentation becomes useless when it stops matching reality.

Good architecture creates the shape.

Good engineering practice keeps that shape healthy.

That is why adaptability is not a one-time design achievement. It is something a team maintains through repeated decisions.

Change Is a Business Reality

Software exists to serve a business need, and business needs rarely stand still.

That is why changeability has real business value.

A system that can adapt helps the organization respond to new opportunities. It makes product learning less expensive. It helps teams move with more confidence when priorities shift. It reduces the fear that every new requirement will turn into a risky rewrite.

A system that resists change creates a different kind of pressure.

Roadmaps become harder to trust. Estimates become more uncertain. Teams hesitate to improve the system because every change feels connected to too many others. Eventually, the architecture begins shaping the business more than the business shapes the architecture.

That is a dangerous place to be.

Good architecture does not make change free.

But it can keep change from becoming unnecessarily expensive.

From Change to Momentum

Engineering momentum depends on a team’s ability to keep moving as reality changes.

That means understanding the system in production. It means planning for failure. It means designing boundaries, contracts, and feedback loops that allow the system to evolve without constantly fighting itself.

Good architecture is built for change because useful software is never truly finished.

It grows with the business. It responds to what teams learn. It adapts as customers, markets, and requirements become clearer.

The goal is not to predict the future perfectly.

The goal is to avoid trapping the team inside decisions that no longer fit.

MARC², the M²S² architecture review bot, can help start that conversation by surfacing areas where coupling, unclear boundaries, or change risk may need more attention. From there, M²S² can help your team reason through the tradeoffs and turn those insights into practical architecture decisions.

MARC² can help you start the conversation. M²S² can help you turn it into action.

Good Architecture Plans for Failure

Fri, 10 Jul 2026 19:59:39 +0000

Good architecture does not assume everything will work.

It assumes something will fail and makes sure the team has a reasonable way to contain it, understand it, and recover from it.

That may sound pessimistic, but it is really practical engineering. Every production system eventually meets reality. Dependencies slow down. Networks misbehave. Deployments change behavior. Queues back up. Users retry actions. Data arrives in shapes no one modeled perfectly.

Failure is not always a sign that the architecture is bad.

Failure is part of the environment the architecture has to operate within.

In the first article of this series, I wrote about how good architecture creates engineering momentum by making important decisions visible early. In the second, I wrote about observability because teams need to understand and reason about systems in production.

Failure planning is the next step.

Observability helps a team see what is happening. Failure planning gives the team options when something goes wrong.

A system is not well-designed because nothing ever fails. A system is well-designed when expected failures are contained, visible, and recoverable enough that they do not create unnecessary chaos.

The Happy Path Is Not the Whole Design

Most systems start with the happy path.

A user takes an action. The API receives a request. The service validates the input. The database saves the record. An event is published. A downstream process reacts. The workflow completes.

That path matters. It is usually the reason the system exists in the first place.

But production systems are shaped just as much by the paths that do not go cleanly.

The database write succeeds, but the event publish fails. A third-party service times out after the user has already submitted the request. A message is processed twice. A worker crashes halfway through a job. A deployment changes behavior in one service but not another. A user refreshes, retries, or submits the same action again because the interface did not clearly tell them what happened.

These are not strange edge cases. They are normal conditions in real systems.

When the design only accounts for the successful flow, the team is forced to discover the failure flow later. Usually that happens under pressure, when production is already misbehaving and the cost of learning is higher.

Good architecture does not require every failure path to be solved upfront. That would be unrealistic. But it should make the most important failure paths visible early enough for the team to decide how they want the system to behave.

That is the difference between accidental behavior and intentional design.

Reliability Lives in the Details

Some of the most important architecture decisions look small at first.

Timeouts are a good example. A timeout is not just a configuration value. It defines how long one part of the system is allowed to wait on another before protecting itself. Without clear timeouts, a slow dependency can hold resources, block requests, and create pressure in places that were never the original source of the problem.

Retries are similar. Retrying a request can make a system more resilient when the failure is temporary. It can also make an outage worse if every client retries aggressively at the same time. A retry can recover work, or it can duplicate work. It depends on how the system is designed.

That is where idempotency matters.

If an operation may be attempted more than once, the system needs to know whether repeating it is safe. This matters for APIs, payments, event processing, background jobs, notifications, provisioning workflows, and any system where partial success is possible.

A retry without idempotency is not resilience. It is a bet that the same action will not hurt you twice.

These decisions often get treated as implementation details, but they shape the behavior of the system. They determine whether a failure stays small or spreads. They determine whether recovery is clean or messy. They determine whether the team can confidently automate retry behavior or has to manually clean up after every partial failure.

Good architecture takes these details seriously because reliability is rarely created by one big decision. It is usually created by many small decisions that work together.

Contain the Blast Radius

Once something fails, the next question is how far the failure should spread.

That is where blast radius matters.

Not every part of a system has the same business importance. If a reporting workflow is delayed, that may be acceptable. If checkout, login, or a core customer workflow fails, the impact is much higher. Good architecture recognizes that difference and avoids letting non-critical failure take down critical behavior.

This is where boundaries matter. Service boundaries, queue boundaries, rate limits, circuit breakers, fallback behavior, and feature flags are all ways to control how failure moves through a system. None of them are magic. Each adds its own complexity. But when used intentionally, they give the team more control over how the system behaves under pressure.

A downstream service being unavailable should not automatically mean every upstream workflow has to fail. A slow dependency should not be allowed to consume every resource in the system. A bad release should not require the entire platform to be rolled back if the change can be isolated or disabled.

The point is not to make every system bulletproof.

The point is to decide which failures are acceptable, which failures must be contained, and which parts of the system need to degrade gracefully instead of collapsing completely.

That is an architectural decision.

It is also a business decision.

Recovery Is Part of the Design

Prevention gets a lot of attention, but recovery matters just as much.

No matter how carefully a system is designed, something will eventually go wrong. A deployment will introduce an issue. A migration will behave differently than expected. A message will fail. A third-party service will return something unusual. A job will process only part of the data it was supposed to process.

The question is not only whether the team can prevent every problem.

The question is whether the team knows what to do next.

A system that supports recovery gives the team options. Failed messages can be retried or moved to a dead-letter queue. Events can be replayed. Data migrations have a rollback or repair strategy. Deployments can be reverted safely. Runbooks explain what to check and who owns the response.

These things do not need to be overbuilt, but they should not be ignored.

A system becomes more trustworthy when the team knows how to recover it. Engineers move differently when they know a failed job can be replayed, a bad deploy can be rolled back, or a partial failure can be repaired without guessing through production data.

Recovery creates confidence.

Confidence creates momentum.

Preparation Is Different Than Planning

Planning for failure is important, but planning is only half the work.

The other half is preparation.

A team can document a rollback process, but that does not mean everyone knows how to use it under pressure. A system can have a recovery path, but that does not mean the path has been tested recently. A runbook can describe what to do during an outage, but if no one has practiced the steps, the first real incident becomes the training session.

That is not where teams want to learn.

Good architecture gives teams options when something goes wrong. Good preparation makes sure the team knows how to use those options.

This is where practices like chaos days, game days, failure drills, and incident simulations become valuable. The goal is not to randomly break things for entertainment. The goal is to safely test assumptions, expose weak spots, and build confidence before the stakes are higher.

A team might simulate a dependency outage, force a queue to back up, test a rollback, replay failed messages, rotate a secret, restore from backup, or walk through what happens when a critical service becomes unavailable. These exercises do not need to be large or dramatic to be useful. Even a small controlled failure can teach the team a lot about the system.

The value is often in what the team discovers.

Maybe an alert fires too late. Maybe the runbook is missing a step. Maybe only one person knows how to perform the recovery. Maybe a fallback path exists in code but has never been used in production. Maybe the system technically recovers, but the customer experience is worse than expected.

Those are useful discoveries.

Finding them during a practice exercise is far better than finding them during a real incident.

Preparing for failure also changes team behavior. Engineers become more familiar with operational tools. Leaders gain a clearer understanding of system risk. Teams learn where the architecture is resilient and where it is still fragile.

That kind of practice builds confidence.

Planning gives the team a theory of recovery. Preparation proves whether that theory holds up.

Failure Planning Depends on Observability

Failure planning, preparation, and observability are closely connected.

Observability helps the team see failure. Planning gives the team recovery options. Preparation teaches the team how to use those options when the system is under pressure.

One without the others is incomplete.

If a system has strong failure handling but weak observability, the team may not know when something is degrading or why. If a system has strong observability but no recovery path, the team may know exactly what went wrong and still have no safe way to fix it. If the team has a recovery plan but has never practiced it, the plan may not survive first contact with a real incident.

The best systems give teams visibility, options, and rehearsal. They show where the failure happened, how far the impact spread, and what recovery paths are available. They also give the team enough practice to move from detection to understanding to action without starting from scratch.

This is why failure planning belongs in the architecture conversation, not just the incident response process.

By the time a system is failing in production, the team is mostly working with the options the architecture already gave them and the preparation they already invested in.

Good architecture gives them better options.

Good preparation helps them use those options well.

Failure Design Is a Business Decision

Not every system needs the same level of resilience.

A prototype does not need the same failure planning as a payment system. An internal admin tool does not need the same guarantees as a customer-facing workflow. A reporting job that can run late does not need the same recovery model as a real-time user action.

That is why designing for failure is not about making everything equally resilient. It is about matching the investment to the impact.

Resilience has a cost. Redundancy, queues, retries, fallback paths, data replication, disaster recovery, and operational tooling all add complexity. Sometimes that complexity is justified. Sometimes it is not.

Good architecture is honest about those tradeoffs.

It considers what the business needs, what the team can operate, and what the system is responsible for protecting. It separates critical workflows from secondary ones. It identifies where failure must be avoided, where it can be tolerated, and where graceful degradation is good enough.

The goal is not perfection.

The goal is intentionality.

When the team understands the consequences of failure, they can make better decisions about where to invest in reliability and where to keep the system simple.

From Failure Paths to Forward Momentum

Good architecture does not eliminate failure. It gives teams a better way to handle it.

That matters because engineering momentum is not just about how quickly a team can ship when everything is going well. It is also about how quickly the team can recover when something goes wrong.

A team that understands its failure paths can move with more confidence. A team that has clear recovery options can release with less fear. A team that practices its response is less likely to discover basic operational gaps during a real incident.

Failure planning is not about pessimism. It is about responsibility.

It is about recognizing that production systems operate in imperfect environments and designing them so that failure does not automatically become chaos.

That is why good architecture plans for failure.

It helps teams contain the impact, understand what happened, recover with confidence, and keep moving.

MARC², the M²S² architecture review bot, can help start that conversation by surfacing areas where failure behavior, reliability assumptions, or recovery paths may need more attention. From there, M²S² can help your team reason through the tradeoffs and turn those insights into practical architecture decisions.

MARC² can help you start the conversation. M²S² can help you turn it into action.

When "Latest" isn't Actually Latest

Thu, 02 Jul 2026 11:08:55 +0000

When "Latest" Isn't Actually Latest

TL;DR

In distributed systems, the last record your system processes is not always the most recent event that happened in reality.

Understanding the difference between Event Time and Processing Time is essential to avoid data quality issues in streaming pipelines, CDC architectures, and event-driven systems.

The Assumption That Works... Until It Doesn't

Most developers start their careers working with a single application and a single database.

In that world, time feels simple.

A customer updates their profile.
The database stores the change.
Everyone sees the updated record.

The sequence appears obvious because all operations happen within the same system.

Then distributed systems enter the picture.

A Simple Example

Imagine a customer updates their address at 10:00:00.

That change generates an event and begins traveling through a distributed architecture:

Application
Database
CDC pipeline
Kafka
Consumer service

Everything looks normal.

Now imagine an older address update, generated earlier, gets delayed because of a network issue or retry mechanism.

The result?

The older event arrives after the newer one.

Your consumer processes the events in this order:

New address
Old address

If your logic assumes that the last processed record represents the latest truth, you've just overwritten correct information with stale data.

The Real Problem: Time Has Multiple Meanings

This is why distributed systems distinguish between two important concepts:

Event Time

The moment something actually happened in the real world.

Example:

Customer changed address → 10:00:00

Processing Time

The moment a system receives or processes the event.

Example:

Consumer receives event → 10:00:15

Most of the time, these timestamps are close enough that nobody notices.

The interesting cases appear when they aren't.

Where Things Get Complicated

Several factors can cause events to arrive out of order:

Network latency
Retries
Consumer lag
Parallel processing
System failures
Distributed transactions
CDC replication delays

As architectures become more distributed, these situations become normal rather than exceptional.

What About Kafka?

Kafka helps preserve ordering within a partition through offsets.

However, offsets tell us the order in which records were written to Kafka.

They do not guarantee the order in which events actually occurred in the real world.

That's an important distinction.

A perfectly ordered Kafka partition can still contain events that represent out-of-order business actions.

Why Data Engineers Care

Many expensive data quality problems are not caused by broken infrastructure.

Pipelines may be healthy.
Kafka may be running perfectly.
Consumers may be processing records successfully.

The issue is often more subtle.

The system answers the right question using the wrong sequence of events.

When that happens, dashboards become inaccurate, customer profiles become inconsistent, and downstream decisions become unreliable.

Final Thought

Distributed systems force us to rethink a concept we usually take for granted: time.

The challenge isn't simply moving data from one system to another.

The challenge is preserving the meaning of that data while it travels through a network of independent components.

Because in distributed systems, the most recent record is not always the most recent event.

And understanding that difference is often the first step toward building reliable data systems.

Good Architecture Includes Observability

Fri, 19 Jun 2026 23:04:28 +0000

Good architecture is not only about how a system is built. It is also about how well the team can understand that system once it is running.

That is where observability belongs in the architecture conversation.

It is common for observability to be treated as something that comes after the main engineering work. The service gets built. The API works. The deployment succeeds. Then, somewhere near the end, the team starts thinking about logs, dashboards, alerts, and operational visibility.

That approach creates a gap.

The architecture may look clean on paper, but once the system is in production, the team has to understand how it behaves under real conditions. Real users do not follow the happy path perfectly. Dependencies slow down. Queues back up. Data arrives in unexpected shapes. Deployments change behavior in ways that are not always obvious.

If the system does not give the team a way to see those things clearly, the architecture is incomplete.

Observability is not decoration around the system. It is part of the system design.

Architecture Describes the System. Observability Shows the Truth.

Architecture is built on assumptions.

During design, teams make reasonable guesses about usage patterns, service boundaries, dependency behavior, data flow, scale, latency, and failure modes. Some of those assumptions are based on experience. Some are based on current requirements. Some are simply the best call the team can make with the information available at the time.

That is normal.

The problem is not that architecture contains assumptions. Every architecture does.

The problem is when those assumptions cannot be tested once the system is real.

A design might assume that an external dependency will be reliable enough. Production may show that it is the slowest part of the request path. A queue might look like a clean decoupling point during design. Production may reveal retry behavior, duplication, or ordering concerns that were not obvious upfront. A serverless function might seem isolated and simple until timeouts, cold starts, and downstream limits begin shaping the user experience.

Observability is how the team learns what reality looks like.

Without it, production becomes a black box. The team may know something is wrong, but not where. They may see failures, but not the path that produced them. They may know customers are impacted, but not which customers, how often, or why.

That is not just an operations problem.

It is an architecture problem.

If the team cannot understand how the system behaves in production, the design is missing a critical feedback loop.

Observability Is More Than Having Logs

A lot of systems have logs. Far fewer systems are actually observable.

I have seen versions of the same story play out more than once. A team ships a feature, the deployment succeeds, and the basic dashboards look fine. Then support reports that a subset of users cannot complete a workflow. The application has logs, but the logs only show isolated fragments. One service logged the request. Another logged a timeout. A third recorded a retry. Somewhere downstream, a dependency returned an unexpected response.

Nothing is completely invisible, but nothing is connected well enough to explain the full story.

That is the pitfall.

Teams do not need more disconnected data. They need enough context to move from symptom to cause.

Good observability connects the signals that matter. It helps engineers follow the path of a request, understand where latency was introduced, see which dependency shaped the outcome, and determine whether the issue is isolated or systemic. It reduces the need to rely on memory, guesswork, or the one person who happens to know that part of the system best.

Logs matter. Metrics matter. Traces, dashboards, and alerts matter too. But the toolset is not the point. The point is whether the team can ask useful questions about the system and get useful answers.

Dumping logs into a platform does not automatically create understanding. Generic CPU and memory metrics may tell you something about infrastructure health without telling you much about user impact. Dashboards can look impressive while answering very little. Alerts can wake people up without giving them enough context to act.

The goal is not more data.

The goal is better understanding.

That is why observability has to be designed intentionally.

Observability Changes How Teams Work

The most obvious benefit of observability is faster debugging, but that is not the only benefit.

The larger benefit is that observability changes the way teams make decisions.

When a system is difficult to observe, conversations tend to become speculative. People rely on memory, intuition, and whatever signal is easiest to find. During incidents, that can create stress and confusion. After incidents, it can lead to shallow conclusions because the team never had a clear enough view of what actually happened.

When a system is observable, the conversation changes.

The team can look at the request path instead of debating which service might be slow. They can separate a deployment issue from a data issue. They can see whether a failure affected one customer, one workflow, or the entire system. They can understand whether a dependency is occasionally slow or consistently shaping the user experience.

That kind of visibility creates better engineering behavior.

Teams move from guessing to investigating. They move from opinion to evidence. They move from “it seems fine” to “here is what the system is telling us.”

That feedback loop matters beyond incidents. It helps teams improve architecture over time. It shows where the system is under stress. It reveals dependencies that are more fragile than expected. It exposes workflows that are harder to support than they looked during design.

Observability does not make systems simple, but it does make complexity easier to see.

Visible complexity is much easier to manage than hidden complexity.

Observability Should Be Designed Early

Observability is much harder to add after the fact.

Once a system is already built, teams often discover that the information they need was never captured. The logs are inconsistent. Trace context does not cross service boundaries. Metrics describe infrastructure health but not business impact. Alerts fire too late, too often, or without enough detail to act on.

At that point, improving observability becomes another project layered on top of the original system.

It is much easier to build visibility into the system while the architecture is still being shaped.

That does not mean every project needs an elaborate observability platform on day one. It means the team should think about how the system will be understood once it is running. Critical workflows should leave a trail. Risky dependencies should be measurable. Failure modes should produce useful signals. The team should be able to tell whether the system is healthy from both a technical and user perspective.

This becomes especially important as systems become more distributed.

The more services, queues, functions, APIs, and third-party dependencies involved, the more important it becomes to preserve context across the flow. Without that context, every incident becomes a scavenger hunt across logs, dashboards, cloud consoles, and Slack messages.

That scavenger hunt costs time, confidence, and momentum.

The Best Signals Connect to Real Impact

Not every signal is equally useful.

A team can collect hundreds of metrics and still struggle to understand whether users are having a good experience. A dashboard can show that infrastructure is healthy while a critical workflow is failing. An alert can be technically correct and still provide little value if it does not explain what is impacted or why anyone should care.

Good observability connects technical behavior to real impact.

Latency is not just a performance number; it represents a user waiting on the system. Error rates are not just percentages; they are workflows that did not complete. Queue depth is not just an operational metric; it is a sign that work is backing up somewhere in the system. Dependency failures matter because they change how gracefully the application can recover, degrade, or protect the user experience. Deployment events matter because every release changes the behavior of the system in some way, even when the deployment itself succeeds.

The best signals help a team understand what is happening in terms that matter to both engineering and the business.

This does not make infrastructure metrics unimportant. They are part of the picture. But they should not be the whole picture. Teams also need visibility into critical workflows, service boundaries, external dependencies, and the outcomes customers actually experience.

A healthy architecture makes those relationships easier to see.

Observability Belongs in Architecture Reviews

This is why observability belongs in architecture reviews.

A design review should not stop at whether the components fit together. It should also consider how the system will be operated, supported, and improved once it is real.

That conversation changes the quality of the architecture.

It forces the team to think about where risk exists. It brings critical workflows into focus. It exposes dependencies that may need better monitoring or fallback behavior. It encourages teams to think about how they will diagnose failure before failure happens.

This is also where MARC², the M²S² architecture review bot, can help start the conversation.

A first-pass architecture review should surface more than service boundaries, data flow, and scaling assumptions. It should also help identify whether the system gives the team enough visibility to understand it in production.

MARC² is not a replacement for experienced judgment, but it can help reveal where observability may be missing from the design conversation.

That is a useful starting point.

Observability Is Part of Engineering Momentum

Engineering momentum depends on a team’s ability to keep moving without constantly losing time to avoidable confusion.

When observability is weak, teams lose time reconstructing context. Incidents take longer because the system does not clearly show where the problem started or how far the impact spread. Releases feel riskier because there is not enough confidence in how the system changed after deployment. Architecture decisions become harder to validate because the team has limited feedback from production.

That uncertainty affects more than engineering. Leaders have less confidence in what is happening. Product conversations become harder to ground in reality. Teams spend more energy interpreting symptoms than improving the system.

Strong observability creates a better feedback loop. It helps teams see how the system behaves, where pressure is building, and which decisions need to be revisited. It gives engineers and leaders a shared understanding of the system as it actually runs, not just as it was designed.

That is why observability is not just an operational concern.

It is part of good architecture.

The code matters. The services matter. The deployment model matters. The data model matters. But the team’s ability to understand and reason about the system in production is the most important thing.

A system that cannot be understood in production cannot be confidently operated.

And a system that cannot be confidently operated will eventually slow the team down.

Good architecture includes observability because good architecture is not only about building the system. It is about building a system the team can understand, operate, and improve.

That is what keeps momentum alive.

From Visibility to Momentum

If your team is designing a new system, preparing for launch, or trying to understand why an existing platform has become harder to operate, observability is a good place to look.

The question is not simply whether you have logs, metrics, traces, dashboards, or alerts. The better question is whether your team can understand what the system is doing when it matters most.

That is the kind of visibility good architecture should provide.

MARC², the M²S² architecture review bot, can help start that conversation by surfacing architecture and observability gaps early. From there, M²S² can help your team reason through the tradeoffs, improve the design, and build the feedback loops needed to keep moving with confidence.

MARC² can help you start the conversation. M²S² can help you turn it into action.

find . -delete Ran Before the Filter and Emptied the Whole Tree

Wed, 17 Jun 2026 23:57:57 +0000

I meant to delete the .cache files under a data directory. The server had been running for two months and the cache layer had grown to about 14GB. The application team told me it was safe to purge it — they'd rebuilt the cache logic and the old files were just dead weight. I typed find /data -delete -name "*.cache" because I was moving fast and I figured the order of arguments to find did not matter. It does. find evaluates its expression left to right, and -delete is not a filter — it is an action. It fired on every path find walked, starting at /data itself, and -name "*.cache" never got a chance to narrow anything. By the time I hit Ctrl-C the tree was roughly 80% gone.

That was not a test server.

The restore from backup took forty minutes. The application was down for those forty minutes. The postmortem was a forty-five minute conversation with people who did not particularly enjoy having it. I have run hundreds of find commands since then and I verify the expression order before every single one that has any destructive action attached to it — not because I've forgotten the rule, but because the cost of forgetting it once is not recoverable with an apology.

Why the order is the program

find does not have a flag parser that groups tests and actions separately. It walks a directory tree and evaluates its arguments as a logical expression, left to right, short-circuiting on false. When you write:

find /data -name "*.cache" -delete

it evaluates -name "*.cache" first on each path. If the name does not match, it short-circuits and -delete never runs on that path. When you write:

find /data -delete -name "*.cache"

it evaluates -delete first. -delete always succeeds — it removes the path and returns true, which means the expression continues to -name. The name check runs after the deletion, on a file that no longer exists, which is meaningless. The effect is that everything gets deleted and nothing is filtered.

This is not a bug. It is exactly how the man page says find works. It is just not how anyone instinctively reads a command the first few times they use it.

The rule is: tests filter, actions act, and actions must come after the tests that are supposed to narrow them. Write it in that order every time.

The quoting trap, right behind the ordering one

Here is the one that is even more subtle, because it causes the wrong behavior and produces no error at all:

# WRONG — the shell expands *.cache before find ever sees it
find /data -name *.cache

# RIGHT — quote the pattern so find does the matching
find /data -name "*.cache"

If there happens to be a single .cache file in your current working directory when you run this, the shell expands *.cache to that one filename and passes it to find -name as a literal string. find then searches for files named exactly that, everywhere under /data. It finds some, it finds none, but it is definitely not doing a wildcard search. If there are multiple .cache files in your current directory, find receives too many arguments for -name and errors out with something confusing about paths needing to precede the expression.

Either way you did not get what you intended, and if you added -delete, you deleted the wrong things silently.

Quoting the pattern is the fix. The quotes prevent the shell from expanding the glob so find receives the literal *.cache pattern and handles the wildcard itself, across the directory tree you pointed it at.

The age sign that everyone reverses at least once

find /var/log -name "*.log" -mtime +30   # older than 30 days
find /var/log -name "*.log" -mtime -1    # modified within the last day
find /var/log -name "*.log" -mtime 30    # exactly day 30 (almost never what you want)

+30 is older than thirty days. -1 is within the last day. A bare 30 means precisely thirty days ago, which is almost never the thing you're trying to match. The sign convention is the opposite of what feels natural — you want files "older than" a threshold and the intuitive symbol for "bigger number" is +, but a lot of people read +30 as "in the last thirty days" the first time they see it.

I have reversed this twice in production. Once I kept the wrong logs. Once I deleted logs I needed for an audit the following week. Neither was catastrophic but both were embarrassing, and both happened under the kind of mild time pressure where double-checking the man page feels slower than it actually is. The builder I built for this labels the output as "older than" or "within the last" in plain English next to the value, which removes the one decision in a find -delete job most likely to go wrong when you are already stressed.

The -exec batching difference nobody explains

# Runs the command once per file — slower, one PID per file
find /data -name "*.cache" -exec rm {} \;

# Batches files into one invocation — faster, one rm for many files
find /data -name "*.cache" -exec rm {} +

\; runs the command once per file. + collects as many paths as it can and passes them all to one invocation of the command, the same way xargs does. For something like rm, which accepts multiple arguments, the + form is faster and produces less process overhead. For something like a custom script that must process exactly one file at a time, \; is correct.

Most resources either do not mention this difference or mention it once in a reference table. The practical consequence is real — on a directory with ten thousand files, \; spawns ten thousand processes. The + form spawns a handful. On a cleanup job that runs in cron, the difference shows up in CPU load.

The preview step I skipped the day I broke things

The most reliable way to avoid the ordering and quoting mistakes is to build the command in two steps. First, run it with -print instead of -delete:

find /data -name "*.cache" -print

Read every line of that output. Confirm the list is what you intended. Then, and only then, swap -print for -delete. This adds maybe thirty seconds to the workflow. It would have saved me forty minutes and a postmortem. I skipped it because I was confident. Confidence is not a useful substitute for verification when the operation is irreversible.

The find command builder enforces this by showing a warning whenever you select -delete or -exec as the action, and offering to generate the -print version of the command first. It is the kind of nudge I would have appreciated having that day.

What the builder actually does

It assembles the expression in the legally correct order — tests first, action last — so you cannot accidentally replicate the mistake I made. You set the starting path, add tests in whatever order feels natural to you (name, type, age, size, exclude path), and pick an action. The output command always has the tests before the action, regardless of the order you clicked things in.

Every active flag gets a plain-English description inline. -mtime +30 reads as "modified more than 30 days ago." -name "*.cache" reads as "name matches the glob *.cache, quoted so find handles the wildcard." The -exec {} + form is the default when you pick -exec, with a note explaining why it is faster.

The whole point is to get from "I need to find and delete files matching these conditions" to a verified, copy-paste command without the thirty-second loop of man-page reading and second-guessing that I used to do and, on one bad morning, skipped.

What I upgraded while I was at it

The same discipline applies to the other tools. The rsync command builder now has presets for the three setups people build most often — local backup, push to remote, mirror — because those cover maybe ninety percent of rsync jobs and getting the flags right from scratch every time is where the dangerous ones like --delete get misapplied. The cron builder previews the next five run times after you build an expression, because a cron job I once deployed ran at 3am UTC instead of 3am local time and I did not notice until it fired on the wrong schedule for a week. You can also paste an existing crontab line and get the human-readable schedule back. The chmod builder now accepts an octal you paste in and sets the checkboxes — two-directional, because reading a file's permissions and understanding what they mean is just as common a task as setting them from scratch.

These are not features I planned in advance. They are all things I needed at 2am and did not have. That is the pattern most of this site runs on.

Build a find command with tests ordered before actions and every flag explained: https://bashsnippets.xyz/tools/find-command-builder

If you are scoping files before searching or transforming them, the whole pipeline is documented in Bash Text Processing: find, grep, sed, and awk. The rest of the free tools are at https://bashsnippets.xyz/tools

Why Systems For Humans, Not People?

Sat, 23 May 2026 19:11:40 +0000

I run a newsletter called Systems For Humans

I'm a software engineer. Not a data scientist, not an analyst, not a philosopher with credentials. But the longer I build software for businesses, the more I see software engineering and information systems collapse into one new practice. Call it behavioural engineering, or behaviour design, or whatever name eventually sticks. The naming is unsettled because the scale is unprecedented. But the practice itself is already running at planetary scale.

So here goes my journey and insights so far.

More people are connected now than at any point in human history. Behind that connectivity sits a cold, monstrous machine that sees people as data points. Zero sentiment.

Every click. Every scroll. Every GPS coordinate your phone hands over. Every call. Every message you sent and every one you almost sent and deleted. You are a collection of data points to the machine, whether you like it or not.

Back in the day, the business that won was the business that was good with people. Now the business that wins is the one that's good with human data points in relation to other data points. Thomas Davenport called this years ago. Every company eventually becomes a data company. By the time Google was running cloud engineering curricula around the idea, it had hardened into industry orthodoxy.

Human vs people

So why call this Systems for Humans and not Systems for People?

A human is the term we use to describe a person as a scientific phenomenon. Your emotions and sentiments aren't analysed for what they categorically are. They're analysed for what actions they drive you to take. Action economics drives the human world. Sentiment drives the people domain.

Back in the day we were told that the business that won was the business that was good with people. That framing was a polite lie.

Coca-Cola didn't win by just being good with people.

It was running one of the most successful behavioural engineering projects of the twentieth century, except it had to do it through television, billboards, and the radio. The craft was always deliberate. What we have today is a higher resolution, low latency, and broader reach version. ==The instruments got sharp enough to operate on you specifically, in real time, on a device you carry everywhere you go.== The practice didn't get invented. It got closer. It's now building an inescapable frame around you.

When an organisation focuses on the human every problem becomes diagnosable. It becomes like a mechanic tuning a car. A surgeon operating on a sedated person. It can engineer systems and manipulate behaviour and anticipate actions. For the first time human experience is officially and truly a scientific economic problem in real observable time.

Why you do what you do as a person doesn't matter to it. What matters is that you did it. We had an interest in it. And we got a record of the event.

Why this matters for builders in Zimbabwe

Look at why most tech startups here fail and why fintech succeeds.

The average indie hacker thinks: what do people want? He builds an app for that. The better question is: what does the system need to be more efficient around this want? Because even if person A wants something, it won't go anywhere if it doesn't make the system more efficient. Fintech apps aren't about making people happy, they are just there to make the finance system more efficient and reliable.

This creates a new problem for developers in places like ours. If you don't have access to data, you don't know what you're really building for. And the people who have the data know its value, so they won't give it away. Most of it is private anyway. In the end, ==we are locked out of the very inputs we'd need to build systems that actually serve the humans inside them.==

The alternative is to build your own data bank. That's a whole new complexity most aren't equipped for, but ethics aside, it's the only honest path for builders in our position. Every fleet management app, every inventory tracker, every tool that gets deployed in Zimbabwe must become a data collection apparatus.

Let's talk philosophy for a bit

You know the Ubuntu adage. Umuntu ngumuntu ngabantu. I am because you are.

For our grandparents, Ubuntu meant we must be empathetic. It confirmed that we build each other in community. From a data oriented point of view, Ubuntu means you as a data point can influence me as a data point upon points of contact.

Ubuntu describes human existence relationally: a person becomes a person through contact with other people. Modern data systems describe humans relationally too, just differently. Social graphs, recommendation systems and behavioural models all reduce people into nodes, edges and probabilities. The philosophical language changed. The relational structure did not.

In the early 20th century, after Europeans got tired of killing each other, a movement called humanism emerged. A philosophy that put human experience at the centre of society to prevent unnecessary suffering. It collected the known data points of suffering and produced a declaration recommending world leaders enhance human life accordingly.

I'm being deliberate with that phrasing. Even the humanist project, the one that's supposed to protect us from being reduced to inputs, has always been a data project under the hood. Governments and institutions cannot respond to suffering one person at a time. Rather they aggregate patterns, statistics and outcomes. The difference is that humanism claims those abstractions should ultimately serve human flourishing rather than optimisation alone.

Where this is going

The future is data-driven. More complex and predictive algorithms will manipulate and drive human behaviour at scales we haven't seen yet. Self optimising profit-driven algorithms are the new overlords.

My biggest fear is this: while humanist organisations fight governments to protect human rights, a new frontier for humanity is already being shaped. Systems for Humans means proactively building social and business systems that protect human interests on this frontier. And do not treat humanity as a consumable input or data point to be manipulated in service of the algorithm's grand design.

The line between consumer modelling and political modelling is thinner than people realise. Rayid Ghani spent ten years at Accenture Technology Labs building algorithms that scored grocery shoppers on how savvy they were at hunting deals, how they assembled their baskets, what would move them to switch brands. In 2011 he left Accenture and became chief scientist for Obama's re-election campaign. The same modelling techniques he had used to predict whether a shopper would respond to a coupon were retooled to predict whether a voter would respond to a political appeal. The campaign's analytics operation is now widely described as having been potentially decisive in the 2012 election outcome. The grocery basket and the ballot turned out to be the same problem.

Cathy O'Neil saw where this leads. In Weapons of Math Destruction she argues that once you post something, or take an action that gets captured, it's no longer yours. It becomes the property of the company that controls the API that captured it. There is a version of you, a collection of every action you've ever taken, that is not yours. It belongs to them. And that version can be used in real time to manipulate your friends, your family, your daughters, anyone on the same platform.

Including you.

This isn't speculation. It's already the operating model.

When OpenAI ships ChatGPT, they're not just engineering a model and serving it via API. They're shipping a system that reshapes how millions of people write, think, search, decide. The product is behavioural.

When Meta optimises the feed, the engineering and the information architecture are both subordinate to a single goal: produce a specific behavioural outcome (more time on platform) by routing the right stimulus to the right person at the right moment.

So where does that leave you?

When you finally buy that upcoming PS6 you've been wanting, is it really you?

Or is the purchase a data point the algorithm has been quietly developing for months? It starts as a reel: gaming is life. You click. You watch. It records. Then comes another: instead of alcohol, buy yourself leisure you won't regret. A steady stream of PS-vs-Xbox content ramps up. Until you buy.

At which point in that sequence did your agency live? The very thing humanists insist sits at the centre of the philosophy, where exactly is it?

If you want the practical companion to this, Mthokozisi Mabhena has written it. Poison Your Data walks through Cambridge Analytica, Myanmar, the 2024 Taiwan AI persona networks, and lands on a concrete defence: data poisoning. Go read it. His piece is probably the data point that nudged me into writing this one.

My question is the one upstream of his prescription. At what point does the humanist project still hold? When the systems are this good at predicting us, what's left of the agency humanism puts at the centre of everything?

Hi Guys :) Help Me Choose the Best Payment Architecture for My Project

Tue, 19 May 2026 16:26:42 +0000

I’m currently building a mini project focused on a VIP car collection inventory platform designed for high-end clients who own and manage luxury or classic vehicle collections.

One of the most important decisions right now is defining the payment architecture for the platform. Since the project is blockchain-oriented and built around Hyperledger Fabric concepts, I’m evaluating two different approaches:

Option A — Native Token System (On-Chain)

Create a native credit token inside the Fabric network.
Clients deposit real money → tokens are issued to their wallet → platform services consume those tokens.
This would create a fully on-chain ecosystem similar to ERC-20 logic, but implemented in Hyperledger Fabric.

Option B — Off-Chain Payments + On-Chain Verification

Payments happen externally through traditional methods (Stripe, bank transfer, etc.).
The blockchain only stores a “payment confirmed” event along with a hash of the payment receipt for verification and auditability.

What I’d Like Feedback On
Which architecture would be more realistic for a production environment?
Which option offers better scalability and maintainability?
Which one would be more attractive for VIP/high-end clients?
Would a fully on-chain payment experience actually add value in this type of application?
From a software engineering perspective, which approach would you personally choose and why?

I’d really appreciate opinions, technical feedback, architecture suggestions, or even hybrid alternatives. Thanks!

What I Learned Building a Multi-Agent AI System (That No Tutorial Warned Me About)

Sun, 10 May 2026 05:30:26 +0000

I’ve been building an AI platform called Wizard Ecosystem — a full-stack system with a multi-agent architecture, memory, tools, and orchestration layers.

At first, I thought multi-agent systems would be simple:

assign different roles to different agents
chain them together
let the orchestrator manage everything

In reality, it became one of the most unpredictable systems I’ve worked on.

This post is about the actual engineering problems I encountered while building it, and what I learned from them.

The Architecture

The system currently includes:

multiple specialized agents (researcher, writer, reviewer, coder, optimizer, translator, etc.)
orchestration logic for routing tasks
persistent memory systems
RAG/document retrieval
web search integration
tool execution
a Python SDK/API layer
multiple connected applications inside the ecosystem

The backend is primarily built using:

Python
Flask
Groq-hosted Llama models
modular routing and orchestration systems

What started as a chatbot slowly became a distributed AI workflow system.

Problem 1: Agents Don’t Naturally Cooperate

My first assumption was:

“If every agent has a role, they’ll behave consistently.”

That turned out to be completely wrong.

What actually happened:

reviewer agents contradicted writer agents
coder agents ignored earlier constraints
multiple agents repeated identical reasoning
responses drifted between executions

The biggest surprise was realizing:

LLMs are not stable workers.

They are probabilistic generators.

Even with identical prompts, outputs can vary enough to destabilize workflows.

Problem 2: Prompting Is Not Architecture

Initially, I tried fixing everything with better prompts.

That worked temporarily, but the system kept breaking in edge cases.

Eventually, I realized:

Prompts are configuration — not control.

The actual stability came from:

orchestration rules
validation layers
structured input/output schemas
limiting unnecessary agent interactions

Without external structure, multi-agent systems become chaotic very quickly.

Problem 3: Orchestration Became the Hardest Component

I originally thought the difficult part would be generation quality.

Instead, the hardest problem became:

deciding which agent should act, when, and with what context.

Some issues I encountered:

recursive reviewer loops
duplicate task execution
context overload
conflicting outputs from individually “correct” agents

This forced me to redesign the orchestrator multiple times.

The more agents I added, the more important coordination became.

Problem 4: Memory Is Dangerous

Persistent memory sounded easy:

store user information
retrieve it later
improve continuity

But memory introduced entirely new failure modes:

outdated context affecting future responses
incorrect assumptions being reinforced
irrelevant memory retrieval
hallucinated continuity

I learned that:

Memory in AI systems behaves more like active bias injection than passive storage.

It requires:

filtering
relevance scoring
strict context management
Problem 5: Latency Breaks the Illusion

Even using fast inference through Groq-hosted models, latency became a real issue.

Multi-agent systems amplify delays because:

every agent call adds overhead
orchestration introduces routing time
chained execution increases unpredictability

A technically “smart” system can still feel unintelligent if response timing becomes inconsistent.

That was something I underestimated early on.

The Biggest Insight

After rebuilding parts of the system multiple times, I realized:

The hardest part of multi-agent systems is not intelligence — it’s coordination.

Most tutorials focus on:

prompts
models
agents themselves

But the real engineering challenge is:

orchestration
state management
validation
system predictability

The intelligence of the system depends heavily on the structure surrounding the models.

What I Changed

To stabilize the system, I eventually:

centralized orchestration logic
standardized prompts through a prompt builder
reduced unnecessary agent chaining
added validation and reviewer stages
simplified routing behavior
limited memory retrieval scope

Ironically, making the system more constrained improved overall quality significantly.

Final Thoughts

Building a multi-agent AI platform completely changed how I think about software systems.

At first, I viewed LLMs as:

“smart tools.”

Now I view them more like:

non-deterministic distributed system components.

That changes everything about how you design around them.

The biggest lesson I learned is:

AI systems become far more reliable when architecture is prioritized over raw generation.

There’s still a lot I’m improving, but building these systems has been one of the most interesting engineering experiences I’ve worked on so far.

Originally published on Dev.to:
https://dev.to/ag_wizai

Building Production-Ready Spring Boot Microservices: Lessons from Scaling to Millions

Thu, 12 Feb 2026 05:38:38 +0000

Introduction

Over the past several years working in e-commerce, I've architected and scaled microservices handling millions of customer requests daily, including platforms processing over $100M in transactions. In this article, I'll share practical patterns for building Spring Boot microservices that can handle production scale.

This isn't theory—these are battle-tested approaches proven under real-world load.

1. Service Boundaries and Domain-Driven Design

The biggest mistake teams make is creating microservices that are too granular or tightly coupled.

Key principle: Each service should own a complete business capability.

// ❌ BAD: Services that are too granular
CustomerNameService
CustomerAddressService

// ✅ GOOD: Services that own complete domains
CustomerProfileService // owns all customer identity/profile data
SubscriptionService    // owns subscriptions, benefits, rewards

When building a subscription platform, a SubscriptionService should own the complete domain: enrollment, benefits tracking, payment integration, and conversions. This prevents the "distributed monolith" where you call 10 services for one operation.

2. Handling Distributed Transactions with Saga Pattern

At scale, distributed transactions become your enemy. Here's the orchestration approach:

@Service
public class SubscriptionEnrollmentOrchestrator {
    
    @Transactional
    public EnrollmentResult enrollMember(EnrollmentRequest request) {
        // Step 1: Validate customer (synchronous)
        CustomerProfile customer = customerClient.getCustomer(request.getCustomerId())
            .orElseThrow(() -> new CustomerNotFoundException());
            
        // Step 2: Process payment (synchronous with compensation)
        PaymentResult payment;
        try {
            payment = paymentClient.processPayment(request.getPaymentMethod());
        } catch (PaymentException e) {
            throw new EnrollmentFailedException("Payment failed", e);
        }
        
        // Step 3: Create subscription (local transaction)
        Subscription subscription;
        try {
            subscription = createSubscription(customer, payment);
            subscriptionRepository.save(subscription);
        } catch (Exception e) {
            // Compensate payment
            paymentClient.refundPayment(payment.getTransactionId());
            throw new EnrollmentFailedException("Subscription creation failed", e);
        }
        
        // Step 4: Publish event for async operations
        eventPublisher.publish(new SubscriptionEnrolledEvent(subscription));
        
        return EnrollmentResult.success(subscription);
    }
}

Key lessons:

Make operations idempotent - Include transaction IDs so retries don't cause duplicates
Compensate explicitly - If step N fails, undo steps 1 through N-1
Separate critical path from eventual consistency - Payment is synchronous, emails are async

3. Strategic Caching for High-Traffic Systems

We reduced database load by 85% through strategic caching:

@Service
public class SubscriptionService {
    
    private final LoadingCache<String, Optional<Subscription>> cache;
    
    public SubscriptionService(SubscriptionRepository repository) {
        this.cache = CacheBuilder.newBuilder()
            .maximumSize(10_000)
            .expireAfterWrite(5, TimeUnit.MINUTES)
            .recordStats()
            .build(CacheLoader.from(repository::findByCustomerId));
    }
    
    @Transactional
    public Subscription updateSubscription(String id, SubscriptionUpdate update) {
        Subscription updated = repository.findById(id)
            .map(s -> applyUpdate(s, update))
            .orElseThrow(() -> new SubscriptionNotFoundException(id));
            
        repository.save(updated);
        cache.invalidate(updated.getCustomerId()); // Invalidate on write
        
        return updated;
    }
}

Caching layers:

Application-level (Guava/Caffeine) - 5-minute TTL for hot data
Redis distributed cache - 15-minute TTL for cross-instance consistency
Database with proper indexing

4. Circuit Breakers and Resilience

When calling other services, assume they will fail:

@Service
public class CustomerProfileClient {
    
    @CircuitBreaker(name = "customer-profile", fallbackMethod = "getCustomerFallback")
    @TimeLimiter(name = "customer-profile")
    public CompletableFuture<CustomerProfile> getCustomer(String customerId) {
        return CompletableFuture.supplyAsync(() -> 
            restTemplate.getForObject("/api/customers/" + customerId, CustomerProfile.class)
        );
    }
    
    private CompletableFuture<CustomerProfile> getCustomerFallback(String customerId, Exception ex) {
        log.warn("Customer service unavailable, using cached data");
        return CompletableFuture.completedFuture(
            customerCache.get(customerId)
                .orElseThrow(() -> new ServiceUnavailableException("No cached data", ex))
        );
    }
}

Configuration:

CircuitBreakerConfig.custom()
    .failureRateThreshold(50)          // Open if 50% fail
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .slidingWindowSize(100)            // Consider last 100 calls
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .build();

5. Structured Logging and Distributed Tracing

You can't fix what you can't see:

@Slf4j
@RestController
public class SubscriptionController {
    
    @PostMapping("/subscriptions")
    public ResponseEntity<SubscriptionResponse> enrollMember(
            @RequestBody EnrollmentRequest request,
            @RequestHeader("X-Request-ID") String requestId) {
        
        MDC.put("requestId", requestId);
        MDC.put("customerId", request.getCustomerId());
        
        try {
            log.info("Starting enrollment", 
                kv("tier", request.getTier()),
                kv("paymentMethod", request.getPaymentMethod().getType()));
            
            Instant start = Instant.now();
            EnrollmentResult result = enrollmentService.enroll(request);
            
            log.info("Enrollment completed",
                kv("subscriptionId", result.getSubscriptionId()),
                kv("durationMs", Duration.between(start, Instant.now()).toMillis()));
            
            return ResponseEntity.ok(toResponse(result));
        } finally {
            MDC.clear();
        }
    }
}

Logging standards:

Use structured key-value pairs, not string concatenation
Track request IDs across service boundaries
Include customer ID and operation type in every log

6. Database Query Optimization

Slow queries kill scalability:

@Entity
@Table(name = "subscriptions", indexes = {
    @Index(name = "idx_customer_id", columnList = "customer_id"),
    @Index(name = "idx_tier_status", columnList = "tier, status"),
    @Index(name = "idx_expires_at", columnList = "expires_at")
})
public class Subscription {
    // Use @BatchSize to prevent N+1 queries
    @OneToMany(mappedBy = "subscription", fetch = FetchType.LAZY)
    @BatchSize(size = 50)
    private List<Benefit> benefits;
}

@Repository
public interface SubscriptionRepository extends JpaRepository<Subscription, String> {
    
    // Custom query with fetch join to avoid N+1
    @Query("SELECT s FROM Subscription s LEFT JOIN FETCH s.benefits WHERE s.customerId = :customerId")
    Optional<Subscription> findByCustomerIdWithBenefits(@Param("customerId") String customerId);
}

Performance lessons:

Index everything you query by
Use batch fetching to prevent N+1 queries
Paginate large results
Monitor and alert on queries >100ms

7. Connection Pool Tuning

spring:
  datasource:
    hikari:
      # Formula: (core_count * 2) + effective_spindle_count
      # For 4-core with SSD: (4 * 2) + 1 = 9
      maximum-pool-size: 10
      minimum-idle: 5
      connection-timeout: 3000
      validation-timeout: 1000
      leak-detection-threshold: 60000
      max-lifetime: 1800000  # 30 minutes

8. Zero-Downtime Deployments

# kubernetes/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: subscription-service
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 3        # Add 50% extra during update
      maxUnavailable: 1  # Keep 5/6 running
  template:
    spec:
      containers:
      - name: subscription-service
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 10

9. Metrics That Matter

@Service
public class MetricsService {
    
    private final Counter enrollmentsCounter;
    private final Timer enrollmentDuration;
    
    public MetricsService(MeterRegistry registry) {
        this.enrollmentsCounter = Counter.builder("subscription.enrollments")
            .tag("service", "subscription")
            .register(registry);
            
        this.enrollmentDuration = Timer.builder("subscription.enrollment.duration")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(registry);
    }
}

Monitor:

Business: Enrollments/hour, conversion rate, revenue
Technical: P50/P95/P99 latency, error rate, throughput
Infrastructure: CPU, memory, connection pool utilization

10. Testing Strategy

// Unit tests - fast, isolated
@Test
void shouldCalculateCorrectExpirationDate() {
    EnrollmentService service = new EnrollmentService(
        mock(CustomerClient.class),
        mock(PaymentClient.class),
        mock(SubscriptionRepository.class)
    );
    
    Instant enrolledAt = Instant.parse("2024-01-01T00:00:00Z");
    Instant expected = Instant.parse("2025-01-01T00:00:00Z");
    
    assertEquals(expected, service.calculateExpiration(enrolledAt, SubscriptionTier.ANNUAL));
}

// Integration tests - with real database
@SpringBootTest
@Testcontainers
class SubscriptionServiceIntegrationTest {
    
    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:15");
    
    @Test
    void shouldPersistAndRetrieveSubscription() {
        EnrollmentResult result = service.enroll(request);
        Optional<Subscription> retrieved = service.getSubscription(result.getSubscriptionId());
        
        assertTrue(retrieved.isPresent());
        assertEquals(SubscriptionTier.BASIC, retrieved.get().getTier());
    }
}

Key Takeaways

After building and scaling microservices handling millions of requests:

Design for failure - Circuit breakers, timeouts, and fallbacks are not optional
Embrace eventual consistency - Not everything needs to be synchronous
Observability is critical - You can't improve what you can't measure
Cache aggressively - But invalidate intelligently
Test at every level - Unit, integration, contract, and load tests
Automate deployments - Including rollbacks and scaling
Performance is a feature - Optimize from day one

What's Next?

In future posts, I'll dive deeper into:

Advanced observability with distributed tracing
Chaos engineering and resilience testing
Cost optimization for cloud deployments

Questions or want to discuss these patterns? Leave a comment below or connect with me on LinkedIn.

Tags: #spring-boot #microservices #java #system-design #aws #architecture #scalability #production-engineering

The Hardware Mutex: Binding LLM Context to a TPM Root of Trust

Tue, 03 Feb 2026 20:43:43 +0000

The Vulnerability: cat /proc/$PID/mem

We need to stop pretending that API Keys are "Identity."

If you are building an Autonomous Agent today, you are likely storing its credentials (OAuth tokens, PEM keys) in memory. This works fine for a stateless chatbot. It is catastrophic for an Agent authorized to move money.

The Attack Vector is trivial:

An attacker gains root access (or compromised container privileges).
They run a memory dump (gcore or just reading /proc/mem).
They grep for the Bearer token.
They clone the Agent’s identity and empty the wallet.

Software permissions (RBAC) cannot solve this because Root is God. If the operating system trusts the Admin, the Admin can impersonate the Agent.

The Fix: The Hardware Mutex

To secure high-stakes Agents, we must move from Software Identity (which can be copied) to Hardware Identity (which must be proved).

I’ve released a reference architecture called The Citadel Protocol that defines how to implement this using a Hardware Mutex.

Here are the mechanics of the stack:

The Enclave (The Body)
The Agent does not run in a standard Docker container. It runs inside a TEE (Trusted Execution Environment) — like AWS Nitro Enclaves or Intel SGX.

The Physics: The memory space is encrypted at the hardware level. Even the Hypervisor (AWS/Azure) cannot read the RAM.

The Attestation (The Heartbeat)
Instead of a static API Key, the Agent authenticates using Remote Attestation.
The Boot: On startup, the TPM measures the hash of the kernel and the application code into a Platform Configuration Register (PCR0).
The Quote: When the Agent wants to "act" (e.g., call an API), it asks the TPM to sign a "Quote" containing the PCR values.
The Verify: The upstream service verifies the Quote. If the code has been altered (even by one byte), the hash changes, the signature fails, and the connection is dropped.
The Witness (The Ledger)
We stop logging to text files (which can be rm -rf'd). We log to a Merkle Tree.
Every action the Agent takes is hashed and appended to a write-only chain.
The head of the chain is anchored externally.
This makes the execution history mathematically impossible to alter
without breaking the chain.

The Spec
We are entering the era where "Code" is no longer enough. We need "Code + Physics."

I have open-sourced the full architectural specification for this topology. It covers the specific PCR measurements, the Witness schemas, and the "Terminal Refusal" logic.

Read the full Citadel Protocol Specification (Zenodo): https://zenodo.org/records/18472859

What exactly is a "program" and what does it include?

Thu, 16 Oct 2025 20:24:57 +0000

In this tutorial, I wanna talk about "program" and "process" terms that are fundamental elements in computer-science. This two terms generally are used exchangeably. But in technically, both are so different. Let's begin with explaining these terms and then deeply see the what it includes:

A program is a file containing a range of information that describes how to construct a "process" at run time [1].
A process is an instance of an executing program [1].

These are the formal definations. If I try to express with my words:

A program (or binary) is a file that includes the machine code + metadata + debug information (if the program compiled with -g flag) and process is a abstraction point that kernel creates and then allocates hardware resources like RAM.

First thing that you should know is the program format. In recently, there are two type of program formats that the compilers generate:

ELF (Executable and Linkable Format)
PE (Portable Executable)

ELF Format

ELF is the standard format for UNIX/Linux systems and PE for Windows. I'm currently on Ubuntu 24.10 (x86_64) so that I will explain and use ELF format. As you guest, I don't know really PE format. But there is a good reference handbook that you can look at. It's "Practical Binary Analysis" written by Dennis Andriesse.

After explained the program formats, right now, let's look at the inside of the program:

In general, any program includes these:

Executable header
Program headers
Sections
Section headers

Every ELF file starts with an executable header, which is just a structured series of bytes telling you that it's and ELF file, what kind of ELF file it is, and where in the files to find all the other contents [2].

You can exactly see the content of this header with readelf command:

$ readelf -h ./copy
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Position-Independent Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x1160
  Start of program headers:          64 (bytes into file)
  Start of section headers:          17496 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         13
  Size of section headers:           64 (bytes)
  Number of section headers:         37
  Section header string table index: 36

In there, first four digits (7f 45) of Magic series define the program format. You will see the probably different series if you are on Windows. Other properties is the self-explaining.

The code and data in an ELF program are logically divided into sections. Each section includes a specific part of your source code. And the sections in the binary are contained in the section header table. Some sections are used to execute the machine instructions and some for other information (like symbol table used by debugger). Let's look at the section headers:

$ readelf --sections --wide ./copy
There are 37 section headers, starting at offset 0x4458:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        0000000000000318 000318 00001c 00   A  0   0  1
  [ 2] .note.gnu.property NOTE            0000000000000338 000338 000030 00   A  0   0  8
  [ 3] .note.gnu.build-id NOTE            0000000000000368 000368 000024 00   A  0   0  4
  [ 4] .note.ABI-tag     NOTE            000000000000038c 00038c 000020 00   A  0   0  4
  [ 5] .gnu.hash         GNU_HASH        00000000000003b0 0003b0 000028 00   A  6   0  8
  [ 6] .dynsym           DYNSYM          00000000000003d8 0003d8 000180 18   A  7   1  8
  [ 7] .dynstr           STRTAB          0000000000000558 000558 0000d3 00   A  0   0  1
  [ 8] .gnu.version      VERSYM          000000000000062c 00062c 000020 02   A  6   0  2
  [ 9] .gnu.version_r    VERNEED         0000000000000650 000650 000030 00   A  7   1  8
  [10] .rela.dyn         RELA            0000000000000680 000680 0000d8 18   A  6   0  8
  [11] .rela.plt         RELA            0000000000000758 000758 0000d8 18  AI  6  24  8
  [12] .init             PROGBITS        0000000000001000 001000 00001b 00  AX  0   0  4
  [13] .plt              PROGBITS        0000000000001020 001020 0000a0 10  AX  0   0 16
  [14] .plt.got          PROGBITS        00000000000010c0 0010c0 000010 10  AX  0   0 16
  [15] .plt.sec          PROGBITS        00000000000010d0 0010d0 000090 10  AX  0   0 16
  [16] .text             PROGBITS        0000000000001160 001160 00043a 00  AX  0   0 16
  [17] .fini             PROGBITS        000000000000159c 00159c 00000d 00  AX  0   0  4
  [18] .rodata           PROGBITS        0000000000002000 002000 000040 00   A  0   0  4
  [19] .eh_frame_hdr     PROGBITS        0000000000002040 002040 000034 00   A  0   0  4
  [20] .eh_frame         PROGBITS        0000000000002078 002078 0000a8 00   A  0   0  8
  [21] .init_array       INIT_ARRAY      0000000000003d78 002d78 000008 08  WA  0   0  8
  [22] .fini_array       FINI_ARRAY      0000000000003d80 002d80 000008 08  WA  0   0  8
  [23] .dynamic          DYNAMIC         0000000000003d88 002d88 0001f0 10  WA  7   0  8
  [24] .got              PROGBITS        0000000000003f78 002f78 000088 08  WA  0   0  8
  [25] .data             PROGBITS        0000000000004000 003000 000010 00  WA  0   0  8
  [26] .bss              NOBITS          0000000000004020 003010 000010 00  WA  0   0 32
  [27] .comment          PROGBITS        0000000000000000 003010 00002b 01  MS  0   0  1
  [28] .debug_aranges    PROGBITS        0000000000000000 00303b 000030 00      0   0  1
  [29] .debug_info       PROGBITS        0000000000000000 00306b 000472 00      0   0  1
  [30] .debug_abbrev     PROGBITS        0000000000000000 0034dd 000184 00      0   0  1
  [31] .debug_line       PROGBITS        0000000000000000 003661 00017e 00      0   0  1
  [32] .debug_str        PROGBITS        0000000000000000 0037df 00031a 01  MS  0   0  1
  [33] .debug_line_str   PROGBITS        0000000000000000 003af9 00012f 01  MS  0   0  1
  [34] .symtab           SYMTAB          0000000000000000 003c28 000438 18     35  18  8
  [35] .strtab           STRTAB          0000000000000000 004060 00028c 00      0   0  1
  [36] .shstrtab         STRTAB          0000000000000000 0042ec 00016a 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), l (large), p (processor specific)

The main topic of this tutorial is the sections. So I wanna explain in deeply each one:

.init and .fini: The machine code in these sections are used before/after the main() function in program. Don't forget that we're not writing this sections. Compiler itself creates these to handle low-level (or hardware-specific) works. I don't know actually what both of them do!
.text: It is the main area where we focus on binary analysis and reverse-engineering stuffs and includes the your code in machine instruction representation. If you look at there, you see that it has voluminous area and thousands of lines of machine code even if it is small program.
.bss, .data and .rodata: These sections are writable and used to hold various variable in source code. .bss includes the static and global uninitialized and .data includes the static and global initialized variables. Also .rodata consists of variables defined with const keyword in code.
.dynamic: This is the "road map" for the kernel and dynamic linker when loading and setting up an ELF binary for executions.
.init_array and .fini_array: These contains an array of pointers to functions to use as constructors/destructors. You maybe know that how to create constructor and destructor using compiler specific-tool, line one __attribute__((constructor)) void run_before_main(void);.
.shstrtab, .symtab, .strtab, .dynsym and .dynstr: The .shstrtab section is simply an array of NULL-terminated strings that contain the names of all the sections in binary. The .symtab contains a symbol table and .strtab contains the symbolic names. The .dynsym and .dynstr are analogous to .symtab and .strtab, except that they contain symbols and strings needed for dynamic linking rather than static linking.
.debug_: These sections are used primarily by debugger (GDB) so that it has not include and machine code but just metadata about program that debugger can use later. If you don't compile the program with -g option, these sections will not there.

Below is a part of .text section:

$ objdump -j .text -d ./copy

./copy:     file format elf64-x86-64

(...)

Disassembly of section .text:

0000000000001249 <main>:
    1249:f3 0f 1e fa          endbr64
    124d:55                   push   %rbp
    124e:48 89 e5             mov    %rsp,%rbp
    1251:48 81 ec 40 04 00 00 sub    $0x440,%rsp
    1258:89 bd cc fb ff ff    mov    %edi,-0x434(%rbp)
    125e:48 89 b5 c0 fb ff ff mov    %rsi,-0x440(%rbp)
    1265:64 48 8b 04 25 28 00 mov    %fs:0x28,%rax
    126c:00 00 
    126e:48 89 45 f8          mov    %rax,-0x8(%rbp)
    1272:31 c0                xor    %eax,%eax
    1274:83 bd cc fb ff ff 03 cmpl   $0x3,-0x434(%rbp)
    127b:75 24                jne    12a1 <main+0x58>
    127d:48 8b 85 c0 fb ff ff mov    -0x440(%rbp),%rax
    1284:48 83 c0 08          add    $0x8,%rax
    1288:48 8b 00             mov    (%rax),%rax
    128b:48 8d 15 72 0d 00 00 lea    0xd72(%rip),%rdx        # 2004 <_IO_stdin_used+0x4>

(...)

In here, we see the three columns that show the machine instructions of your program. At left side, you see the addresses of machine instructions. When running the program, stack pointer tracks these addresses. At center, you see the actual machine instructions corresponding to your code. Program counter register tracks this machine instructions. At right side, you see the assembly-level representations. When dealing with reverse-engineering, we inspect these to understand what program does.

Virtual Memory Layout

After discussed the ELF program format, right now, I will explain the virtual memory layout of a process (running program instance). When you run the program, kernel creates a memory layout as below:

I've explained the .text, .data, .bss, .rodata sections previously. Apart from these, you see the stack and heap areas.

Both are the memory areas that kernel uses to execute the machine instructions. Stack area has LIFO (Last-In First-Out) model. When executing the program, kernel pushs/pops the instructions in here. So it grows up to lower address space. Heap area is completely different! It has FIFO (First-In First-Out) model. Kernel uses this memory space for dynamic allocation with malloc()/free() function family. And it grows up to upper address space. The heap address border is claimed by brk() function.

Until here, I wanna give you a general overview about programs and processes. If you want to dive in more, you can check and look at the below references.

[1]. Kerrisk M., The Linux Programming Interface, no starch press, San Francisco, page 113.

[2]. Andriesse D., Practical Binary Analysis, no starch press, San Francisco, page 33.

Understanding embedded flashing and debugging

Tue, 14 Oct 2025 11:07:59 +0000

In embedded world, using IDE (like Keil and STM32Cube IDE) is heavily the preferred choice because of simplifying the all processes. But I think that we, embedded software developers, must learn and know what is going on under the hood!

In this articles, I'll show the all the steps of flashing and debugging a firmware written for stm32f446x series microcontroller. Firstly, I'll introduce the flashing a firmware into microcontroller and then give you how to debug it.

Before giving details, let's talk about what we need:

A Linux terminal (Ubuntu terminal)
A firmware (ended with .elf or .bin)
ARM GCC toolchain (arm-none-eabi- commands)
Open On-Chip Debugger (openocd command)
ST-Link hardware

Please get these before going further according to your system. I'm current on Ubuntu 22.04.

When talking about a embedded firmware, we have two program formats that we use in general. These are .bin and .elf formats. Ended with .bin firmware just contains the pure machine code and it has not debug information, program sections or other stuffs. It is used just for flashing, not for debugging. Beside that .elf firmware contains both pure machine code and other stuffs so that we can flash and debug it.

Please use the firmware ended with .elf

Another important point is the connection type between your host and microcontroller. For ARM Cortex, we have a few options:

openocd
dfu-util
st-flash

I have ST-Link connection so that openocd is preferred way in our case. If you have USB connection, you can use dfu-util. Most of the STM32 microcontrollers support DFU (Device Firmware Update) mode. Don't forget that you can just flash the firmware over USB, not for debugging.

All right, let's dive into flashing.

First of all, open a terminal and type this command:

$ openocd -f interface/stlink.cfg -f target/stm32f4x.cfg

In here, we specified two files. These are interface (which connection type that I use?) and target (what type of microcontroller that I wanna flash and debug). You can see all interface and target files from openocd folder that you've installed.

After typed this command, you get:

Open On-Chip Debugger 0.12.0
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
Info : auto-selecting first available session transport "hla_swd". To override use 'transport select <transport>'.
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : clock speed 2000 kHz
Info : STLINK V2J33M25 (API v2) VID:PID 0483:374B
Info : Target voltage: 3.228402
Info : [stm32f4x.cpu] Cortex-M4 r0p1 processor detected
Info : [stm32f4x.cpu] target has 6 breakpoints, 4 watchpoints
Info : starting gdb server for stm32f4x.cpu on 3333
Info : Listening on port 3333 for gdb connections
[stm32f4x.cpu] halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x08000530 msp: 0x2001ffd8

You can think that openocd is a bridge between your host and microcontroller. To flash and debug the firmware, we'll use arm-none-eabi-gdb command. It is actually a debugger for ARM Cortex. But we can flash too.

Right now, open a new terminal and type:

$ arm-none-eabi-gdb firmware.elf

After that you are in gdb session and we will make all operations from here. The first thing that we wanna is to connect to somewhere.

From openocd output, we see the "Info : Listening on port 3333 for gdb connections" line. As I said before, openocd is a bridge and created a port 3333 for gdb that we will connect there from gdb session. Let's do it:

(gdb) target remote localhost:3333

After that, you see the first line of main() from the firmware. It's time to flash the firmware and then inspect it:

(gdb) monitor reset halt
(gdb) load

Important note is that before flashing the firmware, resetting the microcontroller and halting the current execution flow is good practice. After these commands, you should see:

Loading section .isr_vector, size 0x1c4 lma 0x8000000
Loading section .text, size 0x1080 lma 0x80001d0
Loading section .rodata, size 0x10 lma 0x8001250
Loading section .init_array, size 0x4 lma 0x8001260
Loading section .fini_array, size 0x4 lma 0x8001264
Loading section .data, size 0x10 lma 0x8001268
Start address 0x80011d0, load size 4716
Transfer rate: 9 KB/sec, 786 bytes/write.

Congrats! You finally flashed the firmware.

In here, you see that many sections written to base address of flash memory (0x08000000). I will not make end-to-end binary analysis but let's explain basically what these sections are:

.isr_vector --> IRS (Interrupt Service Routine) handlers
.text --> machine code that I've written
.rodata --> read-only variable declarations (with const in C)
.init_array, .fini_array --> hardware-level machine code that I've not written
.data --> static or global initialized variables (if exists)
.bss --> static or global uninitialized variables (if exists)

These are the main blocks/sections in any sort of program. You can inspect these in deeply with arm-none-eabi-objdump and arm-none-eabi-readelf commands. But for now, let's go with debugging.

The main purpose of debugging is to track the execution of firmware and, sometimes, extract the any possible information about firmware.

You can look at these information over info commands. Some examples are below:

(gdb) info files
(gdb) info registers
(gdb) info functions
(gdb) info variables
(gdb) info locals
(gdb) info frame
(gdb) info breakpoints
(gdb) info line *addr

With list command, you can see the part of source file from gdb session:

(gdb) list 1
1/**
2 * FreeRTOS 101 - Complete Introductory Program
3 */
4
5#include "main.h"
6
7void TaskLED(void *params)
8{
9while (1) {
10HAL_GPIO_TogglePin(GPIOA, GPIO_PIN_0);

After gaining a overview about the firmware, next step is to track the execution flow. To control the execution flow, we have break, continue, step, next, finish commands:

(gdb) break main                # put the breakpoint at begin of main() function
(gdb) break *0x80001d0   # put the breakpoint at 0x080001d0 address
(gdb) delete                         # delete all breakpoints
(gdb) continue                    # continue the execution flow until any breakpoint
(gdb) next                            # go to next instruction (over functions)
(gdb) step                            # go to next instruction (into functions)
(gdb) finish                          # finish the execution flow

When dealing with embedded systems, we probably want to look at (and set) core and peripheral registers or variables so that we gain the low-level insights about it. For this we have print, x/, set commands. We use print for displaying any variable defined in source code, x/ for displaying registers, set for setting register bits:

(gdb) print initLED                 # display the GPIO initialization structure
(gdb) x/x 0x40023830          # look at RCC AHB1 peripheral clock enable register in hex
(gdb) x/d 0x40023830          # look at RCC AHB1 peripheral clock enable register in decimal
(gdb) x/s 0x40023830          # look at RCC AHB1 peripheral clock enable register in string
(gdb) x/10wx 0x40023830   # look at RCC AHB1 peripheral clock enable with next 9 word (32-bit) registers
(gdb) set {int32_t}0x40023830 |= (1 << 2)   # set the third bit of RCC AHB1 peripheral clock enable register

Well done! This is 80% of debugging.

Until now, I've tried to give you general workflow and commands about flashing and debugging the firmware. But as you guest, this is deep and complex area so that it deserves many articles. For now, I think that it is enough!

Foreign Keys in MySQL for Serious System Design

Fri, 03 Oct 2025 21:33:56 +0000

Let’s face it: encountering foreign keys in MySQL is a classic developer rite of passage. At first, it's about linking tables—but swiftly turns into crafting a bulletproof web of relationships and rules that underpin entire data-driven systems. But at scale, these seemingly simple constraints form the backbone of reliable, maintainable, and secure architectures.

This isn’t just a guide to syntax. This is your comprehensive field manual for wielding foreign keys: strategies, subtle caveats, best practices, and even architectural patterns favored by pros.

Quick Refresher: What are Foreign Keys?

A foreign key is a column or set of columns in one table that references the primary key in another table, enforcing relationship integrity between records. Without them, you risk “orphan” data—think: orders without matching customers, employees reporting to non-existent managers, or posts referencing gone users.

Example:

CREATE TABLE orders (
    order_id INT PRIMARY KEY,
    customer_id INT,
    FOREIGN KEY (customer_id) REFERENCES customers(customer_id)
);

Advanced Foreign Key Tactics

1. Cascade Operations: Controlling Your Data’s Life Cycle

The magic of MySQL’s foreign keys isn’t just the linking—it's about controlling what happens on updates or deletes. Choose from:

CASCADE: Mods or deletes in the parent auto-propagate to children. Delete a customer? Their orders vanish too.
SET NULL: Leaves child reference as NULL if parent goes away—if your business logic allows “detached” records.
RESTRICT/NO ACTION: Block changes if dependencies remain. Most conservative.

When to Use What?

Operation	Use Case Example	Beware...
CASCADE	Orders deleted when customer is deleted	Unintended mass deletions, data loss
SET NULL	Product discontinued, but not deleting sale history	Nullability: Foreign key columns must allow it
RESTRICT	Employees can’t be deleted if they manage teams	Can hinder batch deletes/updates

2. Many-to-Many & Junction Tables

Not all relationships are one-to-many. Many-to-many needs a junction (associative) table with two or more foreign keys.

Example: Users and Roles

CREATE TABLE user_roles (
    user_id INT,
    role_id INT,
    PRIMARY KEY(user_id, role_id),
    FOREIGN KEY (user_id) REFERENCES users(user_id),
    FOREIGN KEY (role_id) REFERENCES roles(role_id)
);

This prevents a user from having invalid roles and vice-versa.

3. Self-Referential & Hierarchical Relationships

Foreign keys can reference the same table (think: management hierarchy or tree categories):

CREATE TABLE employees (
    employee_id INT PRIMARY KEY,
    manager_id INT,
    FOREIGN KEY (manager_id) REFERENCES employees(employee_id)
);

This enables recursive queries and elegant modeling of organizational trees.

4. Deferrable Constraints (Not Native in MySQL Yet)

Some databases let you defer foreign key checks till commit. MySQL doesn't natively support this (as of v8.0), but it’s crucial to know if you’re porting designs—sometimes, you need to temporarily violate constraints in multi-step transactions.

Workaround: Drop constraints, perform bulk actions, re-add and validate—dangerous if not careful.

5. Foreign Key Indexing for Performance

Every foreign key column, especially in large tables, must be indexed! MySQL often creates indexes automatically, but manual indexing ensures optimal joins, cascades, and validation efficiency.

CREATE INDEX idx_orders_customer_id ON orders(customer_id);

Potential Pitfalls & Best Practices

Pitfall	Impact/Example	Best Practices
Mismatched Datatypes	INT in one, BIGINT in another = constraint fails	Match PK and FK datatypes exactly
Circular Dependencies	Table A FK B, B FK A --> deadlocks, migrations pain	Avoid unless model strictly requires; batch disables
Large Cascading Deletes	Deletes millions unintentionally	Use `RESTRICT` or disable, and log deletes
Orphaned Data	Data in child table with no valid parent	Enforce referential integrity via FKs
Bulk Data Loads	INSERTs/LOADs slower due to constraint checks	Temporarily disable FKs during trusted imports

Beyond Basics: Multi-Table and Polymorphic Relations

Composite Foreign Keys

Sometimes, relationships are defined by multiple columns:

CREATE TABLE shipments (
    order_id INT,
    item_id INT,
    ...
    FOREIGN KEY (order_id, item_id) REFERENCES order_items(order_id, item_id)
);

Polymorphic Association (the Anti-pattern)

If you need a comment to belong to either a blog post or a photo—MySQL won’t let you set a FK to multiple tables. This is called a polymorphic association and is discouraged in strict relational design—better to have a dedicated join table per association, or use inheritance-like strategies.

Enterprise Patterns: Audit Trails, Soft Deletes, and Data Warehousing

Soft Deletes: Instead of deleting, mark as deleted_at. Don’t cascade deletes on these columns, handle logically.
Audit Trails: Log every change; foreign keys can help tie logs to users/actions.
Data Warehousing / ETL: Foreign keys are often dropped in warehouse schemas for performance—integrity must be enforced during loading (ETL).

Real-World Example: E-commerce Data Model—Expanded

Tables:

Customers (customer_id, name, email)
Orders (order_id, customer_id, order_date, status)
Products (product_id, name, category_id, price, stock_quantity)
Order_Items (order_item_id, order_id, product_id, quantity, subtotal)
Categories (category_id, name, parent_category_id)
Reviews (review_id, product_id, customer_id, stars, comment, created_at)
Coupons (coupon_id, code, discount_percent, expires_at)
Order_Coupons (order_id, coupon_id)

Sample Foreign Key Usage:

ALTER TABLE orders
    ADD CONSTRAINT fk_customer FOREIGN KEY (customer_id)
        REFERENCES customers(customer_id)
        ON DELETE RESTRICT;

ALTER TABLE order_items
    ADD CONSTRAINT fk_order FOREIGN KEY(order_id)
        REFERENCES orders(order_id)
        ON DELETE CASCADE, 
    ADD CONSTRAINT fk_product FOREIGN KEY(product_id)
        REFERENCES products(product_id)
        ON DELETE RESTRICT;

ALTER TABLE products
    ADD CONSTRAINT fk_category FOREIGN KEY(category_id)
        REFERENCES categories(category_id)
        ON DELETE SET NULL;

ALTER TABLE reviews
    ADD CONSTRAINT fk_review_product
        FOREIGN KEY (product_id) REFERENCES products(product_id)
        ON DELETE CASCADE,
    ADD CONSTRAINT fk_review_customer
        FOREIGN KEY (customer_id) REFERENCES customers(customer_id)
        ON DELETE SET NULL;

Features Achieved:

Orders tied strictly to real customers; orphans impossible.
Deleting a product deletes its reviews, but not historical order records.
Deleting a category doesn’t nuke the products—only unclassifies them.

Foreign Keys and ORM Mappings (Advanced)

Using ORMs like SQLAlchemy, Hibernate, or Django ORM? Foreign keys are the link between your domain models. Properly defined, they:

Enable cascading saves/deletes in code.
Enable reverse lookups and eager/lazy loading.
Ensure constraints enforced at both DB and code layer.

Pro tip: Always mirror your DB constraints in your ORM models, and don’t rely solely on ORM-level validation.

For The Data Architects: When NOT to Use Foreign Keys

High-Volume Event Logs: Can slow inserts; consider only if strong integrity is a must.
Denormalized Warehouses: Use constraints during ETL, not at warehouse runtime.
Third-Party/Cross-DB Data: Can’t enforce with foreign keys; use programmatic or scheduled validations.
Microservices: Sometimes, each service owns its data: cross-database FK constraints can break scaling and autonomy.

Checklist: Production-Ready Foreign Key Design

All FKs indexed?
Datatypes exactly match?
All cascade options reviewed and intentional?
No accidental circular references?
Soft deletes and TTL data not cascaded on delete?
Batch loads/disables FKs as needed?

Resources for Further Mastery

MySQL Foreign Key Docs: Deep-dive with syntax and real-world use cases.
Normalization and ER Diagrams: Strengthen foundation in data modeling.
Books: “SQL Antipatterns” (Bill Karwin), “Designing Data-Intensive Applications” (Martin Kleppmann).

In summary: Foreign keys are your shield and your sword—they safeguard data integrity, enable powerful querying, and make your system maintainable at scale—but demand planning, discipline, and regular review.

Deep Learning is Non Negotiable Now

Sat, 02 Aug 2025 20:47:36 +0000

A dev shipped a feature using AI help. Clean code, no errors, delivered on time. Looked perfect — until it crashed under load.

Digging deeper, the issue was hidden inside a hashmap — an O(n²) performance bottleneck the AI had quietly introduced. But the real problem? He didn’t know how hashmaps work well enough to spot it.

Another story from Reddit — a guy merged some AI-generated tree traversal logic. Passed the basic tests, looked great. But in production, it broke on edge cases. Took him days to find the bug… and when he did, he still didn’t fully understand why it broke.

These aren’t rare cases anymore. I see this pattern constantly.

AI writes code fast, sure. But when things break — and they will — we realize we’ve outsourced the understanding too.

This is why I personally make time to keep learning: through docs, books, deep-dive videos, or even just experimenting with my own bugs. Constantly. If you like learning about programming concepts, data structures & algorithms, or System Design using real world problems, you can check my lists here.

The truth is, AI hasn’t made learning optional. It’s made it non-negotiable.
Because when the autopilot fails, you need to know how to fly the damn plane.

Composition vs. Inheritance in PHP Frameworks

Mon, 14 Jul 2025 17:27:50 +0000

I have already published this article, but I believe it covers essential concepts that are language-independent. I chose PHP and its frameworks to compare inheritance and composition, but the basic ideas can be found in many other languages and frameworks as well. It is essential for writing clean, maintainable code. Many developers dive into popular frameworks and use their powerful features without fully grasping the principles behind them. Yet, in our fast-paced and constantly evolving world, maintainability and reliability are crucial. In this post, I’ll briefly introduce both concepts and demonstrate how they are applied in well-known PHP frameworks like Slim or Laravel by exploring real code examples.

Understanding Inheritance

Let’s begin by defining inheritance. In the object-oriented world, a class represents a logical unit that models real-world concepts. When you create an instance of a class, you get an object. Objects have properties (data) and methods (behavior), and they can also relate to other objects.

One such relationship is inheritance, where a class (called a child or subclass) inherits properties and methods from another class (the parent or superclass). This allows the child class to reuse or override functionality defined in the parent, supporting extensibility and reducing code duplication.

In PHP, inheritance is only possible between classes and interfaces. A class can inherit from only one other class—this is known as single inheritance. While PHP doesn’t support multiple inheritance directly, it offers a feature called traits to help simulate it. However, I’ll focus solely on classical inheritance here.

Inheritance plays an important role in promoting the Don't Repeat Yourself (DRY) principle, making your codebase more maintainable and easier to extend.

What Is Composition?

Now, let’s explore what composition means. Think about the word itself—composition. What comes to mind? That’s right: creating something by putting smaller pieces together.

In programming, composition is the practice of building a class by using instances of other classes. Instead of inheriting behavior from a parent class, you're assembling functionality from separate, focused components. Each of these components (or "pieces") serves a clear, specific purpose, and together, they form a more complex system.

You might wonder: Why is this so important? The answer lies in one of the biggest challenges in software development—maintainability.

In our fast-paced, ever-changing world, codebases need to be adaptable. Composition supports this by reducing tight coupling and making it easier to modify or replace individual parts without causing ripple effects elsewhere. It also enhances testability. You can isolate individual components and write targeted tests for each one.

And as we all know, a high-quality codebase is impossible without good test coverage. With composition, you gain confidence in your system’s behavior because every business use case can be verified through well-structured tests.

Slim Framework

Let’s take a closer look at how inheritance and composition are used in the Slim Framework.

A good example of inheritance is the Logger class under the Slim namespace. It extends an abstract class called AbstractLogger, which in turn implements the LoggerInterface from the PSR-3 standard. Abstract classes in PHP cannot be instantiated directly, so this setup provides a reusable foundation for custom logging logic while adhering to a standardized interface.

This design allows developers to plug in their own logger implementations or use the default one in a consistent and predictable way, all while staying compatible with other PSR-3-compliant tools.

Another example of inheritance in Slim is in its HTTP-related exception classes, such as HttpException, HttpSpecializedException, and HttpForbiddenException. These classes share common behavior through inheritance but typically don’t contain complex logic. Instead, they override specific methods, implement a standard interface, or provide simple property access via getters and setters. This keeps the exception hierarchy clear and focused while still taking advantage of inheritance for code reuse.

Another interesting case is the FastRouteDispatcher class in Slim, which extends the GroupCountBased class from the FastRoute library. Although the GroupCountBased class is marked as @final in its docblock, it isn't technically marked with the final keyword in code. This annotation is a convention, signaling that the class should not be extended, even if PHP itself doesn’t enforce it.

In such cases, one could argue that a more flexible design using composition or a decorator pattern might be preferable. However, the Slim team may have chosen inheritance here for pragmatic reasons—such as tight integration with FastRoute, performance, or compatibility with previous versions.

This also serves as a practical example of polymorphism. Both GroupCountBased and Slim’s FastRouteDispatcher extend from the same base and can respond to shared method calls in different ways. This allows Slim to plug into FastRoute's routing system while still customizing behavior as needed.

A Composition Example in the Slim Framework

Now let’s look at how composition is used in the Slim Framework. When a class uses other classes to fulfill its responsibilities, we refer to it as a "has-a" relationship—one of the clearest signs of composition.

Slim handles these relationships through dependency injection (DI). Interestingly, Slim doesn't include a DI container of its own. Instead, it's designed to work with any PSR-11-compatible container, such as PHP-DI, Pimple, Laminas ServiceManager, or the Symfony DependencyInjection component.

This means Slim is built around a standard interface, not a concrete implementation. That’s a powerful design decision. It decouples the framework from any specific container and allows developers to plug in the tool that best fits their needs.

It’s a great example of flexible, future-proof design. Have you ever thought about structuring your own applications this way? If so, you’re well on your way to building maintainable and scalable software.

Laravel

In Laravel, inheritance plays a key role in the framework’s core architecture. A clear example is the Application class, which extends the Container class. This relationship allows Laravel’s application instance to inherit all the powerful features of the service container—such as dependency resolution and binding—while adding its own framework-specific logic.

A strong example of interface inheritance in Laravel is the ContainerContract interface. This contract extends the PSR-11 ContainerInterface, which defines the basic behavior of a dependency injection (DI) container. Laravel builds upon this standard by expanding the interface to meet the needs of a full-featured framework, turning it into the foundation of Laravel's entire DI system.

This design reflects Laravel’s contract-based architecture—a principle where components depend on interfaces, not concrete implementations. The Container class implements the ContainerContract, and Laravel components rely on the contract, making the system more flexible and testable.

Composition in Laravel

Laravel is full of examples that showcase the power of composition, which is used extensively throughout its components.

Take the PasswordBroker class, for instance. Rather than extending another class, it uses composition to handle password reset functionality. Dependencies like the UserProvider and other related services are injected via the constructor, allowing the class to collaborate with other components without tightly coupling to them.

This design allows Laravel to be flexible and testable. For instance, you can easily swap out the UserProvider, mock dependencies for unit testing, or extend behavior without modifying existing classes.

This is the true strength of composition—it promotes loose coupling and makes your codebase easier to maintain, extend, and reuse.

Conclusion

Understanding the differences between inheritance and composition—and knowing when to apply each—is a key step toward writing clean, maintainable code. When used appropriately, these principles help developers build flexible, testable, and scalable applications. And that ultimately leads to more satisfied teams, happier customers, and a healthier codebase.

8 Most Important System Design Concepts You Should Know

Sun, 29 Jun 2025 08:22:17 +0000

Get a Free System Design PDF with 158 pages by subscribing to our weekly newsletter: https://bit.ly/bbg-social

Animation tools: Adobe Illustrator and After Effects.

Checkout our bestselling System Design Interview books:
Volume 1: https://amzn.to/3Ou7gkd
Volume 2: https://amzn.to/3HqGozy

The digital version of System Design Interview books: https://bit.ly/3mlDSk9

ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.

Understanding DRAM Internals: How Channels, Banks, and DRAM Access Patterns Impact Performance

Fri, 27 Jun 2025 05:29:08 +0000

Modern software performance depends heavily on how efficiently memory is accessed, and not just on raw CPU speed. Sequential memory access tends to be much faster than random access, and understanding why requires a deeper look inside the memory subsystem, especially DRAM’s design.

There are two critical factors that influence memory performance:

Access time per memory access → how long it takes to fetch a single piece of data from memory (this is where DRAM latency plays a role).
The number of memory accesses → how often your program needs to fetch data, which depends on your choice of data structures, algorithms, data organization, and access patterns. Programs that reuse data or access nearby memory locations require fewer fetches due to CPU caching.

This article focuses solely on the first factor → DRAM access latency → and explains how DRAM is architected → and how different access patterns affect its performance. By grasping these fundamentals, you’ll be better equipped to write performance-aware code.

(We won’t cover CPU caches or virtual memory in this article — those will be discussed in follow-up articles to complete the full memory access picture.)

Anatomy of DRAM Components

Channels:
DRAM modules are divided into multiple channels. Each channel operates independently and has its own dedicated data bus, allowing true parallel data transfers across channels to the Memory Controller.
Data Bus per Channel:
Within each channel, the data bus is shared and serialized. This means only one data transfer can occur at a time from any bank in the channel to the memory controller, even if multiple banks are being accessed in parallel.

Ranks:
A rank is a group of 8 DRAM chips that work together to move 64 bits of data at once over the 64-bit data bus. Each chip adds 8 bits, taken from the corresponding row and column inside its own corresponding memory bank (Bank 3 in the above diagram). This way, all chips access and send their part of the data at the same time.

Banks:
Each rank has several banks, which are smaller sections of memory. Banks can work independently and in parallel with each other, but inside each bank, memory accesses are serialized.
Rows and Columns:
Each bank is organized as a grid of rows and columns. Every intersection of a row and column corresponds to a memory cell holding 1 byte of data, which represents one memory address.
Row Buffer:
Each bank has a row buffer that holds the currently active row’s data for faster access. Only one row per bank can be active at a time. That is why memory accesses within a single bank are serialized.

How DRAM Access Works Internally

When the CPU requests data from memory, it sends a memory address to the Memory Controller. The Memory Controller then translates this memory address into a specific bank, row and column that corresponds to that address.

Note: Each bank in the memory can have only one row active (open) at a time. There are three possible cases when accessing the requested row:

Row Buffer Conflict (~75 nanoseconds total): If a different row (let’s call it Row X’) is currently active in the bank, the memory controller must first wait a bit (around 30 nanoseconds), then close (precharge) that active row (which takes about 15 nanoseconds), and finally open (activate) the requested row X (which takes another 15 nanoseconds). This process takes the longest time overall.
Row Buffer Hit (~15 nanoseconds total):
If the requested row X is already active, the controller skips the closing and opening steps. This means the access is much faster since no extra delays are involved.
Row Buffer Miss (~30 nanoseconds total):
If no row is currently active in the bank, the memory controller simply activates the requested row X (taking about 15 nanoseconds).

After the correct row is active, the memory controller reads the requested column Y (which takes about 15 nanoseconds). Finally, the bank returns the requested data to the memory controller, which then sends it back to the CPU.

DRAM Bottlenecks and Their Impact on Memory Access

Shared Data Bus Bottleneck: As mentioned earlier, only one data transfer can happen at a time per channel, even if multiple banks are active. This bottleneck generally becomes significant during bursty or highly parallel traffic when many banks try to send data simultaneously over the shared data bus per channel.
- Latency impact: Each data transfer takes about 15 nanoseconds (tCAS).
- Sequential access: This bottleneck is typically less noticeable for bursty sequential memory accesses because modern CPUs use caches (L1, L2, L3), virtual memory page caching, and hardware prefetching. These mechanisms significantly reduce the number of DRAM accesses required and enable smoother, continuous data transfers over the shared bus.
- Random access: In bursty random access patterns, memory requests tend to target many banks unpredictably and in parallel often leading to bus contention, queuing, and delays when the bus becomes saturated. Moreover, caching and prefetching techniques don’t work well with random accesses because the requests lack predictable patterns, so the system can’t reduce the number of DRAM accesses effectively, making the bus bottleneck impact even more severe for bursty random access patterns.
Bank Access Bottlenecks:
As noted earlier, banks operate in parallel, but within each bank, memory accesses occur sequentially. Additionally, only one row can be active in a bank at any given time. Accessing a different row requires the bank to first precharge (close) the currently active row before activating the new one → a process that incurs significant delay.
- Latency impact:
- Row buffer hit (same row active): ~15 ns
- Row buffer miss or conflict (need precharge optionally + activate): ~ 30–75 ns
- Sequential access: When memory accesses tend to stay within the same row of a bank for some time, the chances of hitting an already active row (row buffer hit) are high. This minimizes costly precharge and activate cycles, resulting in faster access.
- Random access: Memory requests spread unpredictably across different rows and banks, causing frequent row buffer misses and forcing the bank to precharge and activate rows repeatedly. This serialization significantly increases latency due to longer row switching delays.

Conclusion: Comparing Sequential and Random Memory Access

Sequential memory accesses are generally faster because they align well with hardware optimizations like CPU caches, virtual memory page caching, memory prefetching, even during burst transfers. These mechanisms reduce both the number of DRAM accesses and the time per access by taking advantage of predictable patterns and row buffer hits.

On the other hand, random accesses often suffer from row buffer conflicts, serialized access within banks, and data bus contention, especially during bursty or highly parallel traffic. Caching and prefetching are less effective here, making random patterns appear slower.

However, random access isn’t inherently bad → when combined with smart data structures and algorithms that reduce the total number of memory fetches (like B-trees or hash maps), even high-latency accesses can be masked, leading to efficient overall performance.

Atomic Design methodology: deconstructing web design into core components for cohesive, scalable systems

Mon, 19 May 2025 18:19:23 +0000

In this article, learn how to deconstruct interfaces into core components, creating unified, scalable systems using Brad Frost's Atomic Design methodology to enhance your web design skills effectively!

What is Atomic Design?

What never ceases to amaze me in tech is that when I discover something new, like a language, framework, or methodology, the people credited with its creation are still with us and working in the industry!

Brad Frost is one such individual who has significantly impacted the design community with his concept of Atomic Design. This methodology revolutionizes how we approach web design by breaking down interfaces into their fundamental building blocks, or "atoms." Doing so allows designers and developers to create more cohesive and scalable design systems. Atomic Design simplifies the design process and ensures consistency across various platforms and devices. Brad's work continues to influence modern design practices, demonstrating the lasting impact of innovative thinkers in the tech industry.

Atoms

Atoms are the most basic building blocks in Atomic Design. They are the smallest user interface elements, such as buttons, labels, input fields, and icons. Think of them as the fundamental pieces that you can combine to create more complex components.

In web design, atoms are like HTML tags—simple and standalone. Focusing on atoms ensures consistency and reusability across your design system. For example, a button atom can be styled once and used throughout your entire project, maintaining a uniform look and feel. Understanding atoms is crucial because they form the foundation for more complex components.

Molecules

Molecules are groups of atoms bonded together, serving as the next step in building a user interface. They combine two or more atoms to form a more complex component that maintains a simple function. For example, a search form molecule might consist of an input field atom and a button atom.

By grouping atoms into molecules, you create more functional and interactive components. Molecules help organize your design system by providing reusable patterns that can be used across different project parts. They are essential for building consistent and efficient interfaces, allowing you to manage and update multiple atoms simultaneously.

Above is an image of the header section of our Elfgorithm App, which consists of an SVG icon and a styled text element. This combination forms a molecule in the context of Atomic Design. The header is structured as follows:

The <header> element is styled with padding and flexbox properties to align its child elements (the icon and text) horizontally.
The SVG icon, representing a gift, is styled with specific dimensions, colors, and rounded corners to visually stand out.
The text element is styled with bold, italic, and specific color properties, with a portion of the text having a different color and weight for emphasis.

This cohesive unit serves a specific function, demonstrating how molecules in Atomic Design combine multiple atoms to create more complex and functional components.

Organisms

Organisms are more complex components in Atomic Design. They are composed of groups of molecules and/or atoms working together to form a distinct section of an interface. Organisms represent a more substantial part of the UI, such as a navigation bar, a product listing, or a user profile section. Organisms are crucial for creating structured and functional layouts, combining various smaller components to serve a specific purpose.

Using organisms, you can build more dynamic and interactive parts of your application, ensuring that the design remains consistent and scalable. They also allow you to manage and update multiple molecules and atoms collectively, making it easier to maintain the overall design system.

The image above illustrates an organism within the context of Atomic Design. This section, titled "How It Works," combines multiple molecules into a cohesive unit that serves a specific purpose. Each step in the process (Create a Group, AI Matches & Suggests, Enjoy the Exchange) is a molecule, consisting of a number icon, a heading, and a descriptive paragraph.

The entire section, with its heading and grid layout, forms a distinct part of the user interface, explaining the process in a structured and functional manner. This combination of multiple molecules into a larger, interactive layout is characteristic of an organism, representing a more substantial part of the UI.

Templates

Templates are a key part of Atomic Design. They represent page-level objects that place components into a layout and demonstrate the design's underlying content structure. They are essentially a combination of organisms, molecules, and atoms arranged to form a cohesive layout. Templates focus on the page's structure rather than the specific content, allowing you to see how different components fit together.

Using templates ensures that your design system is flexible and adaptable, accommodating various content types while maintaining a consistent layout. This helps visualize the final product's appearance and function, providing a blueprint for building actual pages.

Pages

Pages are the final stage in Atomic Design, where templates are filled with real content to create a fully functional and complete user interface. They represent specific instances of templates, showcasing how the design system looks and behaves with actual data.

Pages are crucial for testing the effectiveness of your design system, as they allow you to see how all the components work together in a real-world scenario. By using pages, you can ensure that your design is both visually consistent and functional as well as user-friendly. They provide a comprehensive view of the final product, helping you identify any issues or improvements needed before deployment.

“Atomic Design” by Brad Frost—An Event Apart Austin 2015

Brad Frost introduces atomic design, a methodology that breaks down interfaces into their basic elements to create scalable and maintainable design systems.

Real-World Use Case

AI: Work smarter, not harder! Effectively using AI as a tool rather than a substitute for your work is an excellent method to maximize efficiency without increasing effort!

Our tech lead, Shashi Lo, understands the proper use of AI and allows us to use it while we build our project. However, it comes with a caveat: you must be able to understand and explain to him how the AI-generated code works! Fair enough.

I was tasked with creating a Google Doc that organizes our forty components and counting, labeling each individual component as an Atom, Molecule, or Organism.

Since the main goal is to write documentation for each component, and we were waiting for this clarification before we began, I utilized AI to analyze each component and categorize them as Atoms, Molecules, or Organisms. This demonstrates the effectiveness of proper component naming conventions, as the AI successfully sorted them all accordingly.

However, it was not enough just to have AI label each component; I needed to understand why each component was labeled as it was. So, I took the AI assistance further and asked it to write definitions for each component.

Using AI as a tool in this manner saved our team time on the mundane task of manually labeling each component as Atoms, Molecules, or Organisms. With the Google Doc completed, with all the labeled components and their definitions, we can quickly approve or disapprove each labeled component and update if necessary, allowing us to finish this task and start the components documentation!

About Us

What is Gridiron Survivor?

Gridiron Survivor is an apprenticeship program created by Shashi Lo, a Senior UX Engineer at Microsoft. It aims to provide developers entering the tech industry with vital work experience. The program focuses on practical training in project management, coding practices, and team collaboration, offering mentorship and skills essential for success in their initial tech roles.

What is the Elfgorithm App?

Elfgorithm is an AI-driven Secret Santa app set to launch in winter 2025. It streamlines gift exchanges by removing the guesswork from Secret Santa activities. The app manages gift-giving details and provides personalized gift suggestions, ensuring you find the perfect presents for everyone.

Gridiron Survivor Sponsors

A very special thanks to our sponsors!

GitKraken: A popular Git client that provides a graphical interface to manage Git repositories. It is known for its user-friendly design and features that simplify version control, making it easier for developers to collaborate and manage their code.
Frontend Mentor: An online platform that offers front-end coding challenges. It helps developers improve their skills by providing real-world projects to work on, along with a supportive community for feedback and learning.
Vercel: A cloud platform for static sites and serverless functions. It is designed to optimize the workflow of developers by providing tools for building, deploying, and scaling modern web applications with ease. Vercel is known for its seamless integration with frameworks like Next.js.

Be sure to listen to the HTML All The Things Podcast!

I also write articles for the HTML All The Things Podcast, which you can read on their website: https://www.htmlallthethings.com/.

Be sure to check out HTML All The Things on socials!

Affiliate & Discount Links!

With CodeMonkey, learning can be all fun and games! CodeMonkey transforms education into an engaging experience, enabling children to evolve from tech consumers to creators. Use CodeMonkey's FREE trial to unlock the incredible potential of young tech creators!

With a structured learning path tailored for various age groups, kids progress from block coding to more advanced topics like data science and artificial intelligence, using languages such as CoffeeScript and Python. The platform includes features for parents and teachers to track progress, making integrating coding into home and classroom settings easy.

Through fun games, hands-on projects, and community interaction, CodeMonkey helps young learners build teamwork skills and receive recognition for their achievements. It fosters a love for coding and prepares children for future career opportunities in an ever-evolving tech landscape.

To learn more about CodeMonkey, you can read my detailed review article!

Affiliate Links:

Advance your career with a 20% discount on Scrimba Pro using this affiliate link!

Become a hireable developer with Scrimba Pro! Discover a world of coding knowledge with full access to all courses, hands-on projects, and a vibrant community. You can read my article to learn more about my exceptional experiences with Scrimba and how it helps many become confident, well-prepared web developers!

Important: This discount is for new accounts only. If a higher discount is currently available, it will be applied automatically.

How to Claim Your Discount:

Click the link to explore the new Scrimba 2.0.
Create a new account.
Upgrade to Pro; the 20% discount will automatically apply.

Disclosure: This article contains affiliate links. I will earn a commission from any purchases made through these links at no extra cost to you. Your support helps me continue creating valuable content. Thank you!

Conclusion

Brad Frost's atomic Design methodology offers a structured approach to building cohesive and scalable design systems. It starts with Atoms, the smallest building blocks, such as buttons and icons, which provide consistency and reusability across the design. Moving up, Molecules are formed by combining atoms to create more functional components, like a search form. Organisms take it further by grouping molecules and atoms into larger, functional interface sections, such as navigation bars.

Templates then arrange these components into page-level structures, focusing on layout rather than specific content. Finally, Pages brings templates to life by filling them with real content, resulting in a complete and testable user interface.

By following these principles, designers and developers can create efficient, consistent, and user-friendly interfaces that work seamlessly across various platforms!

The subtle art of waiting

Wed, 23 Apr 2025 21:40:24 +0000

Recently, while working on a workshop titled Testing Your Pull Request on Kubernetes with GKE, and GitHub Actions, I faced twice the same issue: service A needs service B, but service A starts faster than service B, and the system fails. In this post, I want to describe the context of these issues and how I solved them both with the same tool.

Waiting in Kubernetes

It might sound strange to wait in Kubernetes. The self-healing nature of the Kubernetes platform is one of its biggest benefits. Let's consider two pods: a Python application and a PostgreSQL database.

The application starts very fast and eagerly tries to establish a connection to the database. Meanwhile, the database is initializing itself with the provided data; the connection fails. The pod ends up in the Failed state.

After a while, Kubernetes requests the application pod's state. Because it's failed, it terminates it and starts a new pod. At this point, two things can happen: the database pod isn't ready yet, and it's back to square one or it's ready, and the application finally connects.

To speed up the process, Kubernetes offers startup probes:

startupProbe:
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

With the above probe, Kubernetes waits for an initial ten seconds before requesting the pod's status. If it fails, it waits for another ten seconds. Rinse and repeat 30 times before it fails definitely.

You may have noticed the HTTP /healthz endpoint above. Kubernetes offers two exclusive Probe configuration settings: httpGet or exec. The former is suitable for web applications, while the latter is for other applications. It implies we need to know which kind of container the pod contains and how to check its status, provided it can. I'm no PostgreSQL expert, so I searched for a status check command. The Bitnami Helm Chart looks like the following when applied:

startupProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - -e
      - exec pg_isready -U $PG_USER -h $PG_HOST -p $PG_PORT

Note that the above is a simplification, as it gladly ignores the database name and an SSL certificate.

The startup probe speeds things up compared to the default situation if you configure it properly. You can set a long initial delay, and then shorter increments. Yet, the more diverse the containers, the harder it gets to configure, as you need to be an expert in each of the underlying containers.

It would be beneficial to look for alternatives.

Wait4x

Alternatives are tools whose focus is on waiting. A long time ago, I found the wait-for script for this. The idea is straightforward:

>./wait-for is a script designed to synchronize services like docker containers. It is sh and alpine compatible.

Here's how to wait for an HTTP API:

sh -c './wait-for http://my.api/health -- echo "The api is up! Let's use it"'

It got the job done, but at the time, you had to copy the script and manually check for updates. I've checked, and the project now provides a regular container.

wait4x plays the same role, but is available as a versioned container and provides more services to wait for: HTTP, DNS, databases, and message queues. That's my current choice.

Whatever tool you use, you can use it inside an https://kubernetes.io/docs/concepts/workloads/pods/init-containers/[init container]:

>A Pod can have multiple containers running apps within it, but it can also have one or more init containers, which are run before the app containers are started.

>Init containers are regular containers, except:

> Init containers always run to completion.
> Each init container must complete successfully before the next one starts.

Imagine the following Pod that depends on a PostgreSQL Deployment:

apiVersion: v1
kind: Pod
metadata:
  labels:
    type: app
    app: recommandations
spec:
  containers:
    - name: recommandations
      image: recommandations:latest
      envFrom:
        - configMapRef:
            name: postgres-config

The application is Python and starts quite fast. It attempts to connect to the PostgreSQL database. Unfortunately, the database hasn't finished initializing, so the connection fails, and Kubernetes restarts the pod.

We can fix it with an initContainer and a waiting container:

apiVersion: v1
kind: Pod
metadata:
  labels:
    type: app
    app: recommandations
spec:
  initContainers:
    - name: wait-for-postgres
      image: atkrad/wait4x:3.1
      command:
        - wait4x
        - postgresql
        - postgres://$(DATABASE_URL)?sslmode=disable
      envFrom:
        - configMapRef:
            name: postgres-config
  containers:
    - name: recommandations
      image: recommandations:latest
      envFrom:
        - configMapRef:
            name: postgres-config

In the above setup, the initContainer doesn't stop until the database accepts connections. When it does, it terminates, and the recommandations container can start. Kubernetes doesn't need to terminate the Pod as in the previous setup! It entails fewer logs and potentially fewer alerts.

When waiting becomes mandatory

The above is a slight improvement, but you can do without it. In other cases, waiting becomes mandatory. I experienced it recently when preparing for the workshop mentioned above. The scenario is the following:

The pipeline applies a manifest on the Kubernetes side
In the next step, it runs the test
As the test starts before the application is read, it fails.

We must wait until the backend is ready before we test. Let's use wait4x to wait for the Pod to accept requests before we launch the tests:

- name: Wait until the application has started
  uses: addnab/docker-run-action@v3                                       #1
  with:
    image: atkrad/wait4x:latest
    run: wait4x http ${{ env.BASE_URL }}/health --expect-status-code 200  #2

The GitHub Action allows running a container. I could have downloaded the Go binary instead.
Wait until the /health endpoint returns a 200 response code.

Conclusion

Kubernetes startup probes are a great way to avoid unnecessary restarts when you start services that depend on each other. The alternative is an external waiting tool configured in an initContainer. wait4x is a tool that can be used in other contexts. It's now part of my toolbelt.

To go further:

Originally published at A Java Geek on April 20^th, 2025

A problem statement for my next few articles on System Design and Cloud Computing on AWS

Wed, 02 Apr 2025 09:15:08 +0000

Design and Build a scalable Ecommerce Platform on AWS

This article describe the problem statement of the project we'll be looking at over the next series of articles. The goal is to build this system from scratch - from the frontend pages, backend apis, database integrations and Cloud infrastructure

Problem statement

You're building an e-commerce platform for a small-to-medium enterprise (SME) selling handmade, eco-friendly products. The company expects seasonal traffic spikes (especially during sales like Earth Day, Black Friday), and they need the site to be highly available, scalable, and secure. They also plan to expand globally within the next year.
The MVP (minimum viable product) must be ready to handle 5,000 daily users with the capability to scale to 50,000 during sales events.
The CEO loves "serverless" buzzwords, but they’re open to containers if needed.

Business Requirements (Detailed)

1. Core Functional Requirements

Customers must browse products, add to cart, checkout, and receive email confirmations.
Admins need an internal dashboard to manage inventory, orders, and customer data.
Orders should trigger payment processing (assume you’re using Stripe).
Inventory must update in real time to avoid overselling.
Support for user accounts, including password reset and MFA.
All product images and static assets must load quickly worldwide.

2. Non-Functional Requirements

Scalability: Must handle spikes in traffic without downtime.
Availability: 99.9% uptime (no single points of failure).
Security:
- PCI-DSS compliance for payment data (Stripe handles most of it, but your app must be secure).
- Customer data must be encrypted at rest and in transit.
- Protection from common web attacks (SQL injection, XSS, DDoS).
Performance: Page load time under 3 seconds globally.
Observability: Full monitoring, logging, and alerting for failures.
Cost Optimization: Minimal operational overhead; pay-for-use preferred.

3. User Requirements

Customer Portal: Browse catalog, search products, checkout process.
Admin Portal: CRUD for products, view orders, manage customers.
Authentication: Cognito (or something equivalent), with social login (Google, Facebook).

4. Data Requirements

Product catalog (name, price, description, image, stock).
Customer data (PII, shipping info, order history).
Orders (customer, items, payment status, shipping status).
Audit logs (who did what, and when).

5. Operational Requirements

CI/CD pipeline: Push to main → automatic build, test, deploy.
Blue/Green or Canary deployments for releases.
Disaster recovery: Backup and restore within 4 hours (RTO). Data loss no more than 15 minutes (RPO).
Cost reporting: Regular cost monitoring and alerts for unusual spikes.

6. Compliance & Legal

GDPR compliance (data deletion on request, explicit consent for data processing).
PCI-DSS (payment handling done through Stripe, but secure the backend and database).
Data residency: Store customer data in EU and US regions based on customer location.

Next steps?

The next article will talk about Core Functional Requirements and the microservices we will be building to ensure we have an MVP.

Coder Legion - Recent posts tagged system-design

The Architecture of Ignorance: Engineering Synthetic Consciousness from Chemical Determinism to Pyth

The Core Fallacy: Semantic Parasitism vs. Grounded Constraints

1. System Topology & Multi-Rate Asynchronous Scheduling

2. Epistemic Closure as an Infrastructure Pattern

Container Sandboxing (docker-compose.yml)

In-Memory Role-Based Access Control (Python / SQLite3)

3. From Physics to Thalamic Filtering (Levels 0 & 0.5)

4. Damasio's Somatic Marker & Dynamic LLM Mutation

The Level 3 Ingestion Pipeline: How the Narrative Reaches the Weights

5. The Rosetta Probe: Falsifying Emergent Metaphors (Level 4)

Conclusion

Why Your Product Design Is Costing You Users and You Do Not Know It

WHY GOOD PRODUCTS HAVE BAD DESIGN

THE DESIGN FAILURES THAT COST YOU THE MOST

No clear primary action

Forms that ask for too much too early

Error messages that describe what went wrong without explaining what to do

Navigation that reflects how the product was built rather than how users think

Empty states that leave users stranded

Mobile experiences that were designed on a desktop

WHY THESE PROBLEMS ARE SO HARD TO SEE FROM INSIDE

WHAT FIXING THESE PROBLEMS ACTUALLY REQUIRES

CLOSING THOUGHT

Design a rate limiter, from fixed window to token bucket

100 requests per minute, and 200 got through

A counter and a clock

Remember every request

Two counters and a guess

A bucket of tokens

A bucket with a hole

Many servers, one limit

Saying no politely

Side by side

Sources

Good Architecture Starts with Trust

Trust in the System

Trust in the Team

Trust in Leadership

Trust in the Future

This Is What I Hope M²S² Becomes

Bringing the Series Full Circle

Why I'm Building an Engineering Publishing Platform

Adding Edges Is Not a Paradigm Shift

The Joke Was the Point

An Edge You Did Not Need Is a Checkpoint You Gave Away

The Condition I Have Not Met

Do Not Rewrite the Legacy System. Contain It.

It's Been a While, Here's What I've Been Building

🚀 Building Afia9jaVTU

💳 Building ALATPay for Laravel

🌐 Solving CGNAT Problems with CGNAT Inspector

🛠️ Experimenting with Developer Tools

🏢 Growing Royal Bcode Ventures

📡 Working on SmartAlert-T (SAT)

📚 What I've Learned

What's Next?

Why Modern Engineering Teams Must Prioritize System Observability Over Raw Performance

Why Local Webhook Testing Still Feels Broken

Building webhook integrations shouldn't require deploying your application every time you make a change.

Local Development Shouldn't Slow You Down

Tunnels Are More Than Just Public URLs

Why I Built Tunnels into Hooktrace

It's Not About Replacing Existing Tools

Building Hooktrace in Public

Follow Along

Good Architecture is Built for Change

Change Is the Real Test of Architecture

Future-Proofing Is the Wrong Goal

Coupling Is Where Change Gets Expensive

Boundaries Should Reflect the Business

Good Architecture Makes Local Change Possible

Contracts Help Systems Evolve

Change Requires Discipline, Not Just Design

Change Is a Business Reality

From Change to Momentum

Good Architecture Plans for Failure

The Happy Path Is Not the Whole Design

Reliability Lives in the Details

Contain the Blast Radius

Container Sandboxing (`docker-compose.yml`)

In-Memory Role-Based Access Control (`Python / SQLite3`)