Coder Legion - Recent posts tagged softwarearchitecture

Entity Framework Core Patterns I Actually Use in Production

Wed, 28 Oct 2026 18:49:25 +0000

There is a gap between knowing how Entity Framework Core works and knowing how to use it in a way that holds up under real conditions. The documentation will teach you the first thing. Production will teach you the second, usually by showing you what happens when you get it wrong.

This post is not a tutorial. It is not a comprehensive guide to EF Core features. It is a walkthrough of the specific patterns I apply consistently across every .NET project I build, with the reasoning behind each one. The patterns themselves are not exotic. The discipline of applying them deliberately is what separates a codebase that ages well from one that accumulates quiet performance problems and confusing data states.

If you are past the getting-started stage with EF Core and want to understand how experienced developers think about it in production, this is written for you.

Knowing how EF Core works and knowing how to use it without creating problems at scale are two different skills. Production teaches you the second one.

ASNOTRACKING: THE DEFAULT YOU SHOULD BE QUESTIONING

By default, every entity you query through EF Core gets attached to the change tracker. The change tracker watches that entity for modifications and uses those modifications to generate UPDATE statements when you call SaveChanges. This is the feature that makes EF Core convenient for write operations.

It is also overhead you are paying on every read operation, whether you need it or not.

For a read-only query, attaching entities to the change tracker gives you nothing. The entity is never going to be modified and saved. The tracking is pure cost: memory allocated to hold the snapshot, CPU cycles spent watching for changes, and a change tracker that grows with every query you execute in the same DbContext lifetime.

The pattern I apply is deliberate rather than default. Every query that exists purely to return data gets AsNoTracking. Every query that might lead to an update does not.

// Read-only: tracking adds no value here
var tasks = await _context.Tasks
    .AsNoTracking()
    .Where(t => t.AssignedUserId == userId)
    .ToListAsync();

// Write path: tracking is needed
var task = await _context.Tasks
    .FirstOrDefaultAsync(t => t.Id == id);

if (task != null)
{
    task.StatusId = newStatusId;
    await _context.SaveChangesAsync();
}

The practical impact becomes visible at scale. A dashboard that loads twenty entities without AsNoTracking is allocating tracking overhead for twenty objects the application will never modify. Multiply that across concurrent users and the cost is real.

The habit worth building is this: when you write a query, ask immediately whether the result will ever be written back. If the answer is no, add AsNoTracking before you move on. It is a two-second decision that compounds positively across the entire codebase.

THE N PLUS 1 PROBLEM: WHAT IT IS AND WHY IT KEEPS HAPPENING

The N plus 1 problem is one of the most common performance issues in applications built with ORMs, and it is easy to introduce without realising it because the code that causes it looks completely reasonable.

The pattern goes like this. You query a list of entities. Then, for each entity in that list, you access a navigation property that has not been loaded yet. EF Core executes a separate database query for each entity to load that property. One query to get the list, then one query per item in the list. If the list has fifty items, you have made fifty-one database round trips instead of one or two.

// This looks fine but is dangerous
var projects = await _context.Projects
    .AsNoTracking()
    .ToListAsync();

foreach (var project in projects)
{
    // Each iteration triggers a separate query
    var taskCount = project.Tasks.Count;
}

The fix is to load related data intentionally rather than letting EF Core load it lazily on demand. Include brings related entities into the initial query. Select projects only the fields you need. Both approaches consolidate the data retrieval into a single round trip.

// Explicit loading with Include
var projects = await _context.Projects
    .AsNoTracking()
    .Include(p => p.Tasks)
    .ToListAsync();

// Projection with Select: more efficient when you need specific fields
var projectSummaries = await _context.Projects
    .AsNoTracking()
    .Select(p => new ProjectSummaryDto
    {
        Id = p.Id,
        Name = p.Name,
        TaskCount = p.Tasks.Count
    })
    .ToListAsync();

The projection approach with Select is generally preferable when you are building read models or DTOs, because it gives the database engine the information it needs to execute an optimal query rather than loading full entity graphs and discarding most of the data in application code.

The N plus 1 problem does not look like a bug. It looks like working code. That is why it keeps shipping to production.

The discipline that prevents N plus 1 problems is reviewing every query that accesses navigation properties and asking whether those properties will be loaded by the time they are accessed. If the answer is uncertain, make it certain by being explicit about what gets loaded and when.

SOFT DELETES WITH LOOKUP TABLES

Hard deletes, where a record is physically removed from the database, are rarely the right choice for production data. They break audit trails, they complicate recovery scenarios, and they create referential integrity problems when other records depend on the deleted entity.

Soft deletes, where a record is marked as inactive rather than removed, solve these problems. The record stays in the database. Its relationships remain intact. The deletion is reversible. And you have a complete history of every state the record has passed through.

My approach to soft deletes connects directly to the lookup table convention I use for all status fields. Rather than adding a boolean IsDeleted column, every entity that supports soft deletion has a StatusId that references a shared Statuses lookup table. A status of Active means the record is live. A status of Deleted means it has been soft deleted. A status of Archived means it has been intentionally retired without deletion.

// Statuses lookup table
public class Status
{
    public int Id { get; set; }
    public string Name { get; set; } = string.Empty;
}

// Entity with status-based soft delete
public class Task
{
    public int Id { get; set; }
    public string Title { get; set; } = string.Empty;
    public int StatusId { get; set; }
    public Status Status { get; set; } = null!;
}

The advantage over a boolean IsDeleted column is that the status field carries meaning beyond a binary active or inactive state. You can distinguish between a record that was deleted intentionally, one that was archived, one that is pending review, and one that is in draft, all within the same field, without adding columns or changing the schema.

Adding a new status is a data operation. You insert a row into the Statuses table. No migration required. No enum update. No redeployment. The application picks it up immediately because it is reading status values from the database rather than from a hardcoded list in code.

Filtering deleted records consistently

The operational discipline that makes soft deletes work is consistent filtering. Every query that should return active records needs to filter on status. If you add that filter in some places and forget it in others, deleted records surface unexpectedly and the soft delete pattern breaks down.

// Always filter by active status on read queries
var activeTasks = await _context.Tasks
    .AsNoTracking()
    .Where(t => t.StatusId == StatusIds.Active)
    .ToListAsync();

// Soft delete: update status, do not remove the record
var task = await _context.Tasks
    .FirstOrDefaultAsync(t => t.Id == id);

if (task != null)
{
    task.StatusId = StatusIds.Deleted;
    await _context.SaveChangesAsync();
}

EF Core global query filters are worth considering for applications where the filtered entity is accessed in many places. A global query filter defined on the DbContext applies automatically to every query for that entity type, which eliminates the risk of forgetting the filter in a particular repository method.

REPOSITORY METHODS THAT MATCH REAL USE CASES

The generic repository base covers the standard CRUD operations. In production, most of the interesting work happens in the specific repository interfaces that extend the generic base with query methods that match real application needs.

The principle I apply is that a repository method should represent a meaningful operation in the domain, not a thin wrapper around a LINQ query. The difference is in how the application code reads.

// Thin wrapper: the caller still needs to know too much
var tasks = await _taskRepository.GetAllAsync();
var userTasks = tasks.Where(t => t.AssignedUserId == userId
    && t.StatusId == StatusIds.Active).ToList();

// Domain-meaningful method: intention is clear at the call site
var userTasks = await _taskRepository
    .GetActiveTasksByUserAsync(userId);

The second version is easier to read, easier to test, and easier to optimise. The filtering happens in the repository, close to the database, rather than in application code after a full table scan. And the method name communicates intent in a way that a chain of LINQ operators does not.

This matters more as a codebase grows. A repository with well-named methods becomes a vocabulary for talking about the domain. A repository that returns raw IQueryable and lets callers filter arbitrarily becomes a leaky abstraction that is difficult to reason about and impossible to optimise consistently.

MANAGING DBCONTEXT LIFETIME

DbContext is designed to be a unit of work. It is meant to be created, used for a set of related operations, and disposed. It is not meant to be a long-lived singleton shared across the application.

In an ASP.NET Core application, the standard pattern is to register DbContext as a scoped service, which means one instance per HTTP request. This is correct for most use cases. Each request gets its own context, its own change tracker, and its own transaction boundary. When the request completes, the context is disposed and any uncommitted changes are discarded.

// Program.cs: register DbContext as scoped
builder.Services.AddDbContext<AppDbContext>(options =>
    options.UseSqlServer(connectionString));

The problems arise when DbContext escapes the scope it was designed for. Injecting it into a singleton service is the most common mistake. A singleton lives for the lifetime of the application. A scoped DbContext injected into a singleton effectively becomes a singleton itself, with a change tracker that accumulates state across every request the application handles until it is restarted.

If you need database access in a singleton or a background service, use IDbContextFactory to create a short-lived context for each operation, use it, and dispose it immediately. This gives you the database access you need without the lifetime mismatch that causes subtle and difficult-to-diagnose bugs.

THE MINDSET BEHIND THE PATTERNS

What these patterns have in common is that they require you to be deliberate rather than accepting defaults. EF Core defaults are designed to be convenient for getting started. They are not always the right choice for production, and the gap between convenient and correct widens as the application grows and the load increases.

The developers who use EF Core well in production are not the ones who know the most features. They are the ones who ask the right questions before writing each query: is this data going to be modified? What related data does this operation actually need? What happens to this record when it is deleted? How many database round trips is this operation making?

Those questions have good answers. Getting into the habit of asking them before writing the code, rather than after a performance problem surfaces, is what makes the difference.

CLOSING THOUGHT

EF Core is a powerful tool and a forgiving one. It will let you build something that works without understanding any of what is in this post. The cost of that forgiveness shows up later, in query logs, in memory profiles, and in data states that are difficult to explain. The patterns here are not about using EF Core correctly in an academic sense. They are about using it in a way you will not regret six months into a production system.

Pub/Sub Isn't Event-Driven Architecture: Here's What Most Teams Miss

Fri, 23 Oct 2026 14:38:21 +0000

Pub/Sub is a messaging pattern. Event-Driven Architecture is an architectural discipline. Every EDA implementation may use Pub/Sub, but not every Pub/Sub implementation is truly Event-Driven Architecture.

For many years, I would have agreed with the statement that Pub/Sub equals Event-Driven Architecture (EDA).

Why? Because early discussions around EDA focused primarily on three characteristics:

Real-time communication
Publish/Subscribe messaging
Asynchronous processing

At that time, implementing a Pub/Sub broker was often considered synonymous with implementing EDA.

Today, however, the answer is no.

As organizations have adopted EDA at scale, the discipline has evolved from a messaging concept into a comprehensive architectural practice. Modern EDA encompasses far more than simply publishing and subscribing to messages.

📡EDA Is More Than Messaging

A mature Event-Driven Architecture consists of multiple foundational elements:
📝Event Definition

Events must be clearly defined and categorized.

Examples include:
Business Events: CustomerRegistered, OrderPlaced, PaymentCompleted
System Events: FileUploaded, ServiceStarted, DatabaseUpdated, EOD

Without a consistent event model, organizations quickly end up with thousands of loosely defined messages that are difficult to understand and maintain.

👤Event Ownership

Just like data, events are business assets.

Every event should have a clearly identified owner responsible for:
Event definition
Schema management
Versioning
Lifecycle governance

When nobody owns an event, everyone assumes somebody else does.
📏Event and Queue Standards

Topic (event) and queue naming conventions form the foundation of a scalable EDA platform.
A well-defined naming structure enables:
Event discoverability
Consistent routing
Easier troubleshooting
Better governance
Without standards, the event landscape quickly becomes chaotic.

🔀Event Flow Design

EDA is not merely about sending messages.

It is about how events collaborate to achieve business outcomes.

Modern architectures require a clear understanding of:
Event relationships
Event choreography
Event dependencies
Business process visibility

While services remain loosely coupled, the overall event flow must still be intentionally designed.

🛡️Event Governance

Governance is the difference between a sustainable platform and an uncontrolled messaging ecosystem.

Effective governance covers:
Schema management
Data classification
Security controls
Access management
Event lifecycle management
Compliance and auditing

Governance should be embedded throughout the entire lifecycle, from design through implementation and operations.

📬Pub/Sub Is Only One Pattern Within EDA

Pub/Sub is an important communication pattern, but it is only one building block of a broader architecture.
A Pub/Sub platform without governance, ownership, standards, and event design is simply a collection of disconnected messages.

In such environments, organizations commonly encounter problems such as:
⚠️Consumer Failures

Consumers may crash when:
Unexpected event types are received
Event schemas change without coordination
Required fields are removed or modified
Data Exposure Risks

Sensitive information, including Personally Identifiable Information (PII), may be unintentionally delivered to systems or teams that should not have access to it.
Message Loss or Misrouting

Poorly governed subscriptions can result in:
Events being consumed by unintended systems
Important messages being ignored
Business processes becoming unreliable
Rising Infrastructure Costs

Unnecessary event distribution increases:
Network traffic
Broker workload
Infrastructure consumption
Operational costs
What begins as architectural flexibility can quickly become architectural sprawl.

💡 Final Thoughts

Many organizations begin their event-driven journey by deploying a message broker and enabling Pub/Sub communication. While this is an important first step, it is only the foundation.
The real challenge is not moving events between systems. The challenge is managing events as long-lived business assets.

Without clear ownership, governance, standards, and lifecycle management, organizations often end up with an event ecosystem that is difficult to understand, secure, and evolve. Teams gain asynchronous communication, but lose visibility, consistency, and control.

Successful Event-Driven Architecture is not measured by how many events are produced each day. It is measured by how effectively those events enable business capabilities while remaining discoverable, governed, and trusted across the enterprise.

In other words:
A message broker can make your systems event-enabled.
Governance, ownership, and standards make your organization truly event-driven.

❓ What do you think? Have you seen organizations mistake Pub/Sub implementation for Event-Driven Architecture? What governance challenges emerged as event adoption grew?

This typically drives significantly more comments and discussion from architects, developers, and integration professionals.

How We Secured the MyZubster Metaverse: Session Ownership, Verified Explorers and the Road to Realti

Sat, 17 Oct 2026 05:20:55 +0000

A technical update on the architecture, security controls, testing and remaining roadmap behind the open-source MyZubster Metaverse.
The MyZubster Metaverse is moving from an experimental interface toward a secure, testable multiplayer environment.
Our goal is not simply to display avatars in a browser. We want to create an open-source world connecting public profiles, verified contributors, Zorgax characters, community knowledge and software-development workflows.
The public Metaverse is available at:
https://www.myzubster.com/metaverse
This article describes what we have implemented, the security problem we recently solved and what remains before the platform can support realtime interaction at scale.
The current experience
A user can enter the MyZubster world as a guest without providing:

a cryptocurrency wallet;
identity documents;
a GitHub account;
an installed application.
The current interface supports guest onboarding through a public display name and also presents verified explorers associated with authorized public accounts.
This creates two identity levels:
1. Guest explorers use temporary world sessions.
2. Verified explorers have characters linked to a MyZubster account or confirmed GitHub identity.
  “Verified” does not mean KYC or legal identity verification. It only confirms control of the connected public account.
  Building the presence layer
  The backend maintains a presence record for every active participant.
  A simplified record may contain:
  {
  sessionId,
  displayName,
  characterId,
  accountId,
  githubLogin,
  position,
  status,
  lastSeenAt
  }
  This model supports the main world operations:
  Join
  Synchronize
  Move
  Chat
  Emote
  Leave
  Retrieve events
  Public players must be discoverable by other clients, so each presence has a public session identifier.
  However, a public identifier must never be treated as proof that a request belongs to that player.
  That distinction became the focus of our latest security milestone.
  The original session-ownership problem
  A multiplayer client needs to identify which session it wants to update.
  For example:
  {
  "sessionId": "public-session-id",
  "x": 120,
  "y": 80
  }
  If the backend trusts only sessionId, a malicious user may be able to submit another player’s identifier and perform actions on their behalf.
  Depending on the exposed routes, this could allow an attacker to:
move another character;
send chat messages under another identity;
publish unauthorized emotes;
retrieve session-specific events;
terminate another player’s session.
A public session ID is an identifier, not an authorization credential.
Implementing possession-bound sessions
We implemented a new security boundary in:
Pull Request #34 — Bind world actions to session credentials
The change was merged into main as commit:
148a798
Every new Metaverse session now receives two different values:
Public session ID
Private session token
The public ID can appear in world-state responses. The private token proves that the client controls the session.
Cryptographically random credentials
The server generates a token using 32 random bytes:
const sessionToken = crypto
.randomBytes(32)
.toString("base64url");
This provides 256 bits of entropy.
The raw token is returned only during session creation and is never included in public presence data.
Storing only a digest
The server does not persist the raw credential.
Instead, it calculates a SHA-256 digest:
const sessionTokenHash = crypto
.createHash("sha256")
.update(sessionToken)
.digest("hex");
Only sessionTokenHash is stored with the presence record.
This prevents a database reader from immediately obtaining active bearer credentials.
Constant-time comparison
When a protected request arrives, the server hashes the submitted credential and compares it with the stored digest.
The verification uses a constant-time comparison:
crypto.timingSafeEqual(expected, received);
A shared ownership middleware verifies that:
the session exists;
a credential was supplied;
the credential has the expected format;
its digest matches the stored value;
the caller owns the requested session.
Invalid or missing credentials return:
HTTP 401 Unauthorized
Protected world routes
Session ownership is now required for mutable and session-specific operations such as:
/sync
/move
/chat
/emote
/leave
/events
Knowing another participant’s sessionId is no longer sufficient to control their character.
The authorization rule is now:
Public session ID
```
  +
```
Private possession token
```
  =
```
Authorized world action
Keeping credentials out of public payloads
The frontend sends the private token through a dedicated HTTP header:
X-Metaverse-Session-Token: <private-token>
Example:
const createSessionHeaders = (sessionToken) => ({
"X-Metaverse-Session-Token": sessionToken,
});
The world-state body remains limited to operational data:
{
"sessionId": "public-session-id",
"x": 120,
"y": 80
}
This separation reduces the probability that credentials are accidentally copied into:
world events;
analytics;
public API responses;
application logs;
debugging output;
cached payloads.
The credential is also kept in React runtime memory instead of being written to localStorage or public profile objects.
An in-memory token is not a complete defense against browser compromise, but it reduces persistent credential exposure.
Identity boundaries
The Metaverse now distinguishes between several layers:
Account identity
```
↓
```
Public profile
```
↓
```
Character identity
```
↓
```
Temporary world session
```
↓
```
Private session credential
These layers have different meanings.
A verified GitHub account does not automatically prove legal identity, employment, payment eligibility or institutional affiliation.
Similarly, a public session ID identifies a player’s presence but does not authorize requests for that player.
Documenting these boundaries is essential for avoiding security and governance problems later.
Testing the implementation
The session-ownership change includes backend, frontend and architecture-level tests.
Backend tests
The targeted backend suites verify:
secure token generation;
SHA-256 digest storage;
valid credential acceptance;
missing credential rejection;
invalid credential rejection;
cross-session mutation prevention;
protected movement and chat;
protected emotes and leave operations;
exclusion of secrets from public responses.
Results:
3 backend suites
22 tests passed
Frontend tests
Frontend coverage verifies:
token retention in application memory;
use of the session-token header;
authenticated world operations;
removal of tokens from normal JSON bodies;
correct handling of protected session requests.
Results:
1 frontend suite
5 tests passed
Integration and deployment
The completed verification included:
Production frontend build: successful
git diff --check: clean
Marketplace regression suite: passed
Vercel preview deployments: successful
Main CI workflow: successful
Production deployment: successful
The public Metaverse route was also loaded after deployment.
Current architecture
The current system can be summarized as:
React Metaverse Client
```
  |
  | HTTPS
  | X-Metaverse-Session-Token
  v
```
Node.js / Express API
```
  |
  | Session ownership middleware
  | Hash verification
  v
```
Metaverse Presence Storage
```
  |
  v
```
Shared world synchronization
This is a practical foundation, but it is not yet the final realtime architecture.
What remains to be built
Realtime communication
The current synchronization model still depends on polling in important areas.
We need a versioned realtime protocol supporting:
authenticated WebSocket connections;
session-to-connection binding;
reconnect handling;
ordered events;
duplicate-event protection;
presence expiration;
backpressure;
slow-client management.
The protocol should remain usable by multiple clients, including future community-built and WebXR clients.
Distributed rate limiting
Per-process rate limiting is not enough when the backend runs across multiple instances.
We need a distributed limiter for:
Session creation
Authentication attempts
Movement
Chat
Emotes
Reconnects
Limits should be calculated using combinations of:
IP address;
account ID;
session ID;
route;
world ID;
time window.
Chat and authentication need stricter limits than normal movement updates.
Session expiration and revocation
The possession credential is now implemented, but lifecycle controls still need improvement:
explicit expiration;
server-side revocation;
abandoned-session cleanup;
token rotation;
reconnect-safe renewal;
active-session limits;
multi-device logout;
suspicious-session invalidation.
Canonical authentication
Verified characters require a stable account-linking process.
The remaining work includes:
canonical GitHub OAuth;
verified account-to-character relationships;
account-link substitution protection;
reauthorization;
identity-change audit trails;
account recovery procedures.
The backend must never trust a GitHub username supplied directly by a browser.
Privacy-safe analytics
We need enough telemetry to maintain the platform without introducing unnecessary tracking.
Useful operational metrics include:
concurrent sessions;
API latency;
reconnect frequency;
server errors;
world capacity;
rate-limit events;
event throughput.
Retention, aggregation and access rules must be documented before expanding analytics.
Reliability and load testing
Future tests must simulate:
simultaneous joins;
movement bursts;
reconnect storms;
malformed messages;
database latency;
backend restarts;
multiple server instances;
slow or disconnected clients.
We also need measurable targets for latency, error rate, recovery time and maximum concurrent participants.
Complete E2E acceptance testing
The complete guest flow should be automated:
Join
→ receive credential
→ become visible
→ move
→ chat
→ publish an emote
→ reject impersonation attempts
→ reconnect safely
→ leave or expire
The same acceptance flow is required for verified characters.
WebXR
WebXR remains part of the roadmap, but it should follow security and realtime stabilization.
The eventual world protocol should support:
2D web client
3D browser client
WebXR client
Mobile client
Independent community clients
All clients should use the same documented API and event protocol.
Dependency-security debt
The repository currently reports vulnerabilities during npm audit.
These findings were already present and were not introduced by the Metaverse session-ownership pull request, which did not change package manifests or lockfiles.
However, they remain technical debt.
A separate dependency-hardening milestone should include:
1. classifying production and development exposure;
2. updating direct dependencies;
3. reviewing transitive dependencies;
4. replacing abandoned packages;
5. regenerating lockfiles;
6. running complete regression tests;
7. updating SBOM and audit evidence.
  Disabling the audit to produce a green badge would hide the problem instead of solving it.
  Recommended roadmap
  The recommended implementation sequence is:
8. Session expiration and revocation
9. Distributed rate limiting
10. Canonical account authentication
11. Versioned realtime protocol
12. Authenticated WebSocket transport
13. Privacy-safe monitoring
14. Reliability and load testing
15. Complete E2E acceptance tests
16. Dependency-security remediation
17. WebXR client development
  Security, reliability and protocol stability must come before graphical complexity.
  Open-source participation
  We are looking for contributors interested in:
React;
Node.js and Express;
MongoDB;
Redis;
WebSockets;
application security;
privacy engineering;
abuse prevention;
load testing;
WebXR;
technical documentation.
Repository:
https://github.com/danieldirimini-myzubster/myzubster
Public Metaverse:
https://www.myzubster.com/metaverse
Security implementation:
https://github.com/danieldirimini-myzubster/myzubster/pull/34
Conclusion
The MyZubster Metaverse now has a stronger session-security foundation.
We moved from public identifier-based operations to possession-bound authorization using:
256-bit random credentials;
SHA-256 token digests;
constant-time verification;
dedicated authorization headers;
memory-only client credentials;
protected world routes;
backend and frontend security tests.
The next milestone is transforming this foundation into a distributed realtime system with strong authentication, rate limiting, privacy-safe observability and measurable reliability.
The Metaverse is online, but the deeper work is still ahead: building an open world protocol that other developers can inspect, test and extend.

Seventeen Models in Two Weeks: Stop Chasing Releases, Build a Swap Harness

Wed, 14 Oct 2026 07:00:00 +0000

If your roadmap changes every time a new model drops, you don’t have a model strategy. You have a news-feed habit.

Why now:
The release tracker shows a dense stretch from Sept 28 to Oct 8: Claude Sonnet 5.5 (Sept 28), Gemini 4 Argon (Sept 30), Mistral Large 4 (Oct 6), Claude Haiku 5.5 (Oct 7), plus Cohere, Perplexity, Cloudflare, Microsoft and others.

Key points:

The cost of a new release is the eval, integration and regression
work on your side. Switching is not free just because the API is
similar.
Build your own eval set from real production prompts. Public
benchmarks don’t tell you how a model behaves on your data.
Put a thin abstraction layer between your product and the model, so
swapping is a config change plus an eval run.
Set a policy. For example: re-evaluate quarterly, or when a release
beats your current model by a margin you chose in advance.

Call to action:
How many days it would take your team to swap your primary model today.
If your answer is “weeks,” that’s the thing to fix.

The Weekend After: HeadlessAi Has Entered the Workshop

Sat, 10 Oct 2026 06:54:10 +0000

Last weekend, we introduced some of the toys we're building at The Singularity Workshop. This weekend, we've got more! (All are c#)

HeadlessAi — Connecting .NET applications to AI endpoints through a straightforward, extensible interface.
Profiles — Exploring identity, relationships, privacy, and user-controlled data sharing.
Ontology — Organizing information in flexible, meaningful structures.
MicroBundleIngestor — Exploring new ways to bring MicroBundles into the ecosystem.
Economy — Beginning to explore the foundations of an interconnected digital economy.

So, what makes the Workshop different?

We don't just build packages and leave you to figure them out. We explain the what, why, and how behind them—from the underlying theory to practical examples you can learn from and build upon.

You don't have to be an expert programmer to start exploring. We're building an ecosystem designed to help you understand the technology, connect the pieces, and bring your own ideas to life.

Don't just use the tools. Learn to build with them.

Come explore: The Singularity Workshop on GitHub

The Computational Ecology of a Large System

Thu, 08 Oct 2026 11:57:13 +0000

A large software system is not just a collection of components.

It is an environment.

Inside that environment, services depend on other services. Databases accumulate records. APIs attract traffic. caches fill and empty. queues form and disappear. background jobs wake up at inconvenient times. Logs become historical archives. Configuration changes alter behavior. Small pieces of code begin participating in processes their original authors never imagined.

Eventually, the system develops something that feels surprisingly similar to an ecosystem.

Not because software is alive.

It isn't.

But because a sufficiently large system develops relationships between computational entities, resources, events, constraints, and feedback loops.

A function affects a service.

A service affects a database.

A database affects response time.

Response time affects traffic patterns.

Traffic patterns affect queues.

Queues affect workers.

Workers affect the database again.

And suddenly, the system is no longer easy to understand by looking at any single component.

You have to understand the environment.

That is what I mean by the computational ecology of a large system.

A System Has a Population

Consider a small application.

You might have:

Frontend
   ↓
API
   ↓
Database

This is relatively easy to reason about.

Now imagine the same application after years of development:

                ┌──────────────┐
                │   Web App    │
                └──────┬───────┘
                       │
                ┌──────▼───────┐
                │ API Gateway  │
                └──────┬───────┘
                       │
       ┌───────────────┼────────────────┐
       │               │                │
       ▼               ▼                ▼
   Auth Service    User Service    Order Service
       │               │                │
       └───────┬───────┴───────┬────────┘
               │               │
               ▼               ▼
          PostgreSQL        Redis
               │
               ▼
             Queue
               │
        ┌──────┴──────┐
        ▼             ▼
   Worker A       Worker B

And then there are monitoring systems, analytics pipelines, scheduled tasks, third-party APIs, object storage, notification systems, search indexes, feature flags, deployment infrastructure, and administrative tools.

The population has grown.

In ecology, a population is not interesting merely because many organisms exist.

What matters is how they interact.

The same is true in software.

A service with no relationships is rarely the interesting part.

The interesting part is what happens when services interact.

Components Have Niches

One useful way to think about large systems is through the concept of a computational niche.

A component exists because it performs some role.

A cache exists to reduce repeated work.

A queue exists to separate producers from consumers.

A database exists to preserve and retrieve state.

A worker exists to perform background computation.

A load balancer exists to distribute requests.

An API gateway exists to coordinate access to multiple services.

Each component occupies a particular place in the computational environment.

For example:

Request
   ↓
Gateway
   ↓
Service
   ↓
Cache
   ↓
Database

The cache is not simply "another database."

Its ecological role is different.

It absorbs repeated reads.

It reduces pressure on another resource.

It introduces its own requirements, however.

Now you have expiration policies, invalidation, stale values, memory limits, serialization, cache misses, and cache warming.

One optimization creates another set of interactions.

This is one of the most interesting properties of large systems:

Every useful component changes the environment around it.

Computational Resources Behave Like Environmental Resources

Software consumes resources.

CPU.

Memory.

Disk.

Network bandwidth.

Database connections.

File descriptors.

Worker capacity.

Queue capacity.

Rate limits.

Even developer attention.

These resources form the environment in which computation takes place.

Imagine a service receiving 1,000 requests per second.

Each request requires:

5 ms of CPU time
one database query
several kilobytes of memory
network communication
perhaps a background job

Individually, these requests seem inexpensive.

Collectively, they create an environment with constraints.

The system must continuously allocate resources to competing activities.

This makes capacity an ecological concept.

There is only so much computational space available.

If one process consumes more of it, something else may experience less.

Growth Changes the Environment

A system can behave perfectly at one scale and strangely at another.

Suppose an application starts with 10,000 users.

Its database contains:

10,000 users
50,000 orders
100,000 events

Everything feels simple.

Five years later:

10,000,000 users
700,000,000 orders
9,000,000,000 events

The code may not have changed dramatically.

The environment has.

Queries that were once effectively instantaneous may now interact with indexes containing millions or billions of entries.

Logs that were once useful may now represent enormous storage costs.

Background jobs that once finished in seconds may now compete with other workloads.

The system has undergone ecological succession.

Its population and environment have changed together.

This is why scale is not simply "more users."

Scale changes relationships.

The System Develops Feedback Loops

Large systems are full of feedback loops.

Consider a simple example.

A service becomes slower.

Users retry requests.

Retries create additional traffic.

Additional traffic makes the service slower.

More slowness creates more retries.

Now the system has entered a feedback loop.

Slowness
   ↓
Retries
   ↓
More Requests
   ↓
More Load
   ↓
More Slowness

Nothing in this diagram is particularly mysterious.

The interesting part is that no single line of code necessarily contains the entire behavior.

The behavior emerges from interactions.

This is one of the most important ideas in large-system engineering:

The behavior of a system can exist between its components.

You can inspect the service.

You can inspect the retry mechanism.

You can inspect the database.

Everything may appear reasonable in isolation.

The problem exists in the relationship.

Queues Are Population Dynamics

Queues provide another beautiful example.

Imagine a queue receiving jobs at a rate of 100 jobs per second.

Workers process jobs at a combined rate of 120 jobs per second.

The queue remains relatively small.

Now demand increases to 150 jobs per second.

Workers still process 120.

The difference accumulates.

Incoming: 150/sec
Processing: 120/sec

Growth:
150 - 120 = 30 jobs/sec

The queue begins growing.

After one minute, ignoring other effects:

30 × 60 = 1,800 jobs

The queue has become a population.

Jobs enter.

Jobs leave.

The difference between arrival and departure determines whether the population grows or shrinks.

This way of thinking makes queue behavior much easier to reason about.

Instead of asking:

"Why is the queue huge?"

Ask:

"What is the relationship between arrival rate and processing capacity?"

That question exposes the ecology.

Dependencies Form Food Chains

Large systems also contain dependency chains.

A request might pass through:

Client
  ↓
API Gateway
  ↓
Authentication
  ↓
User Service
  ↓
Database
  ↓
External API

Each layer depends on something else.

If the external API slows down, the effect may travel backward through the chain.

External API slows.

Service waits.

Threads remain occupied.

Request latency increases.

Clients wait longer.

Some clients retry.

Traffic increases.

The original problem has propagated through the ecosystem.

This is why dependency graphs are so valuable.

A system is not merely a set of boxes.

It is a network of relationships.

Some Components Are Keystone Components

In natural ecosystems, some organisms have effects much larger than their population size might suggest.

Software has similar components.

A central database might support twenty services.

A message broker might connect dozens of workflows.

An authentication service might sit in front of almost every request.

A configuration service might influence the behavior of the entire platform.

These are computationally important nodes.

Their importance is not necessarily determined by lines of code.

A tiny configuration service can have more systemic influence than a massive reporting application.

This suggests an interesting rule:

Measure components by the relationships they participate in, not only by their size.

A 500-line service can be structurally more important than a 50,000-line service.

Complexity Is Often About Relationships

Developers often describe large systems as complex because they contain many components.

But component count is only part of the story.

Consider two systems.

System A

A → B
B → C
C → D

Four components.

Three relationships.

System B

A ↔ B
A ↔ C
A ↔ D
B ↔ C
B ↔ D
C ↔ D

Still four components.

But the number of relationships has increased dramatically.

The second system is harder to reason about.

This is why architectural complexity often grows faster than component count.

Every new dependency creates another possible interaction.

And every interaction creates another place where behavior can emerge.

State Is the Soil of the System

A large software system is shaped by its state.

Databases contain history.

Caches contain temporary state.

Queues contain pending work.

Sessions contain user context.

Files contain artifacts.

Logs contain observations.

Configuration contains behavioral decisions.

State gives the system memory.

Without state, many systems would behave like pure transformations:

input → computation → output

With state, the relationship becomes:

input + previous state
        ↓
    computation
        ↓
    new state
        ↓
      output

Now yesterday's activity can influence today's behavior.

A growing database therefore isn't just getting bigger.

It is accumulating history.

And history changes computation.

Technical Debt Changes the Habitat

Code does not exist independently of the environment around it.

A shortcut introduced today may become a constraint tomorrow.

Imagine a service that initially stores a small amount of data in a convenient structure.

Five years later, that structure contains millions of records.

The original decision may still function.

But the environment has changed.

This is one way technical debt behaves.

It is not always an immediate failure.

Sometimes it is an environmental constraint that becomes increasingly expensive as the system grows.

A design choice can become more significant simply because more things now depend on it.

That is why large-system maintenance is often less about fixing individual bugs and more about managing accumulated relationships.

Observability Is Environmental Science

If you cannot observe the environment, you cannot understand it well.

Logs show events.

Metrics show quantities.

Traces show paths.

Profiles show resource consumption.

Together, they provide evidence about the computational ecosystem.

Imagine seeing:

CPU: 45%
Memory: 62%
Latency: 180ms
Errors: 0.4%
Queue: 12,000

Those numbers are useful.

But their relationships are even more useful.

Suppose queue depth is increasing while CPU remains low.

That suggests one kind of problem.

Suppose CPU is saturated while the queue remains small.

That suggests another.

Suppose database latency increases while application CPU remains stable.

Again, a different story emerges.

Observability turns the invisible environment into something we can reason about.

Systems Have Rhythms

Large systems often have temporal patterns.

Traffic rises during certain hours.

Batch jobs run overnight.

Backups occur periodically.

Reports execute at the end of a month.

Customers perform certain actions repeatedly.

Cache entries expire.

Workers wake up.

Queues drain.

The system has rhythms.

A system might look healthy at 3:00 AM and struggle at 9:00 AM.

Nothing "mysterious" happened.

The environment changed with time.

This is why averages can sometimes hide important behavior.

An average of 100 requests per second does not tell you whether traffic was perfectly flat:

or highly variable:

The same average can produce very different computational environments.

The Edge Is Part of the Ecology

It is tempting to think of architecture as what happens inside servers.

But the environment extends outward.

Users have different devices.

Networks have different latency.

Browsers behave differently.

Mobile connections change.

External services respond at different speeds.

Requests can arrive from unexpected paths.

The system exists inside a larger environment.

That means robust architecture is partly about respecting environmental variability.

A system should not assume that the world around it is perfectly stable.

Because it isn't.

Networks fluctuate.

Traffic changes.

Dependencies evolve.

Data grows.

Users behave differently over time.

Good architecture leaves room for these changes.

Evolution Happens Through Iteration

Large systems are rarely designed completely from the beginning.

They evolve.

A simple API becomes a platform.

A database becomes a data warehouse.

A background task becomes a distributed job system.

A single application becomes several services.

A temporary cache becomes an important performance layer.

Each stage introduces new relationships.

This is computational evolution.

The architecture we see today is often the result of hundreds or thousands of small decisions.

Some decisions were made deliberately.

Others emerged naturally from previous constraints.

The resulting system can therefore contain history in its structure.

You can sometimes look at an architecture and almost reconstruct its past.

A strange table may represent an old feature.

A compatibility endpoint may exist because an earlier client depended on it.

A queue may have appeared when synchronous processing stopped scaling.

A cache may have been introduced after a particular workload grew.

The architecture becomes a fossil record of engineering decisions.

The Most Important Unit May Be the Interaction

When debugging a small application, we often ask:

"Which component is broken?"

In a large system, a better question can be:

"Which interaction changed?"

Maybe the service is fine.

Maybe the database is fine.

Maybe the queue is fine.

But the traffic between them changed.

This distinction matters.

A component can remain internally correct while participating in an unhealthy system-level interaction.

That is why distributed systems require a different mental model.

You are not merely debugging objects.

You are debugging relationships.

A Practical Computational Ecology Map

When approaching a large system, I like to mentally map five things.

1. Population

What components exist?

Services
Databases
Workers
Queues
Caches
Clients
External APIs

2. Resources

What do they consume?

CPU
Memory
Storage
Bandwidth
Connections
Time

3. Relationships

Who depends on whom?

A → B
B → C
C → D

4. Flows

What moves through the system?

Requests
Events
Data
Jobs
Messages

5. Feedback

What can influence itself indirectly?

Latency → Retry → Load → Latency

These five dimensions provide a surprisingly powerful way to understand large architectures.

The System Is More Than Its Architecture Diagram

Architecture diagrams are useful.

But they are snapshots.

A diagram might show:

Frontend → API → Database

It does not necessarily show:

how traffic changes during the day,
how queues behave under load,
how caches warm up,
how state accumulates,
how retries interact with failures,
how background workers compete for resources,
how dependencies influence each other,
or how years of history have changed the environment.

The architecture diagram describes the inhabitants.

The computational ecology describes their behavior together.

That distinction becomes increasingly important as systems grow.

The Interesting Part Is Between the Boxes

I have always found large software systems more interesting when viewed as environments rather than collections of files.

A function is interesting.

A class is interesting.

A service is interesting.

But the most fascinating behavior often appears between them.

A queue changes how two systems interact.

A cache changes how frequently one system needs another.

A retry policy changes traffic.

A database index changes the economics of a query.

A scheduler changes when computation happens.

A deployment changes the environment in which everything else operates.

Small changes can therefore produce effects much larger than their physical size.

Not because the software is unpredictable by definition.

Because relationships matter.

Building Systems With Ecology in Mind

Thinking ecologically does not mean making software unnecessarily complicated.

It means recognizing that systems change.

Users increase.

Data accumulates.

Traffic shifts.

Dependencies multiply.

Requirements evolve.

Components become important in unexpected ways.

A design that works today should ideally leave the system room to become something larger tomorrow.

That can mean:

keeping dependencies understandable,
measuring important flows,
limiting unnecessary coupling,
making resource consumption visible,
designing clear boundaries,
treating queues and caches as behavioral components,
monitoring state growth,
understanding feedback loops,
and documenting important relationships.

The goal is not to predict everything.

That would be impossible.

The goal is to understand enough of the environment to make good decisions.

Final Thought

A large software system eventually becomes more than the code that created it.

It becomes an environment shaped by computation, state, traffic, dependencies, resources, history, and time.

Services interact.

Queues accumulate.

Databases remember.

Caches absorb repetition.

Workers consume tasks.

APIs connect worlds.

Metrics reveal patterns.

And feedback loops quietly connect events that may appear unrelated when viewed individually.

This is the computational ecology of a large system.

The system is not merely a collection of components.

It is a network of relationships operating inside a finite computational environment.

And perhaps that gives us a useful way to look at architecture.

Instead of asking only:

What does this component do?

we can also ask:

What changes in the environment because this component exists?

That second question opens a much larger view.

Because in a large system, the most important behavior may not live inside any single component.

It may live in the space between them.

It’s the Weekend. The Workshop Has Toys. Come Play.

Sat, 03 Oct 2026 07:14:18 +0000

It’s the weekend.

Instead of writing another 2,000-word manifesto on what The Singularity Workshop might become someday, I’m just going to open the toy box.

I’ve been building.

Some of it is foundational. Some of it is highly experimental. Some of it is definitely still alpha software. But enough of it has crossed the line from “whiteboard diagram” to running code that it’s time to put it out in public.

You can download it, read the source, compile it, and—most importantly—break it.

Where to Start: NuGet

The easiest way to see what is actually breathing right now is NuGet.

Search for TheSingularityWorkshop and make sure Include prerelease is enabled.

Search The Singularity Workshop packages on NuGet

Most of what I'm pushing right now is deliberately prerelease.

These aren't polished commercial products with a marketing department hiding the machinery behind a landing page. They are architectural experiments that have finally become real enough for someone else to get their hands dirty with.

That is exactly the point.

The Toy Box

Here is what is currently sitting on the workbench.

FSM_API

NuGet | GitHub

This is the bedrock.

The core premise is that behavior can be represented explicitly as state, allowing state-driven behavior to remain separate from the data objects it operates on.

FSM_API is a framework-agnostic C# finite-state-machine system with explicit state definitions, transitions, contexts, processing groups, lifecycle handling, and runtime behavior.

If you want to start at the absolute beginning of the Workshop, start here.

This is where the behavior machinery begins.

FSM_COS

NuGet | GitHub

This is where the pieces start becoming a system.

FSM_COS is the composition kernel.

A runtime manifest says what is requested. FSM_COS resolves the MicroBundle dependency closure, propagates configuration, loads the required capabilities, arbitrates them toward convergence, and produces a RuntimeAssembly.

Conceptually:

Runtime Manifest
       |
       v
    FSM_COS
       |
       +-- resolve MicroBundles
       +-- resolve dependencies
       +-- propagate configuration
       +-- load/install
       +-- arbitrate
       +-- converge
       |
       v
RuntimeAssembly
       |
       v
host / manifestation

The important part is what FSM_COS doesn't do.

It doesn't become the application.

It doesn't become the GUI.

It doesn't become the storage system.

It doesn't become the Experience.

It assembles the machine and hands the resulting composition to something else.

The crane doesn't become the machine.

MicroBundleDomain

NuGet | GitHub

This one is particularly important to the architecture.

MicroBundleDomain is a neutral contract for independently authored capabilities.

A MicroBundle is a focused unit of capability that can identify itself, declare dependencies and providers, accept configuration, enter a runtime, and participate in composition without requiring the host to understand its private domain.

The important architectural separation is:

What is this capability?
        |
        v
MicroBundleDomain

Where do I get it?
        |
        v
MicroBundleRepository

How do I compose it?
        |
        v
FSM_COS

That means MicroBundleDomain does not belong underneath FSM_COS as an implementation dependency.

It is the neutral seam.

That matters because someone should be able to author a capability without permanently welding their package to my composition engine.

MicroBundleRepository

GitHub

If MicroBundleDomain answers what, the Repository answers where.

The repository explores durable delivery of versioned MicroBundle artifacts, currently using Azure Blob Storage as the storage substrate.

The distinction is deliberately sharp:

Blob Storage
    = substrate

Repository
    = artifact delivery boundary

FSM_COS
    = composition boundary

Artifacts have deterministic identity based on the MicroBundle ID, explicit version, and SHA-256 content identity.

The repository doesn't become a database.

It doesn't arbitrate.

It doesn't compose Experiences.

It doesn't become a GUI.

It answers a much smaller question:

Given this artifact address, where are the bytes and can I verify them?

I am having entirely too much fun ruthlessly separating these concerns.

GUI

GitHub

There is a platform-neutral GUI core in here, alongside manifestations for Blazor and WPF.

But the real experiment isn't drawing buttons.

It's asking:

Can an object describe itself structurally well enough that a generic GUI can just render it?

That led to a reflection-driven GUI architecture where the domain model doesn't need to know about the editor that happens to display it.

The basic idea is:

Domain model
     |
     | describes itself
     v
Reflection / semantic projection
     |
     v
GUI.Core
     |
     +--------+--------+
     v                 v
  Blazor              WPF

So if you're interested in semantic interface construction, reflection, recursive GUI structures, and separating what something means from how it looks, poke around in this one.

There is considerably more going on here than “make a button.”

FSM_Serialization

NuGet | GitHub

This isn't an experiment in shouting “my serializer is the fastest.”

It's an experiment in strict ownership.

The domain owns what its data means.

The serialization layer owns how that data crosses a byte boundary.

The host decides where those bytes ultimately go.

The package deliberately stays out of databases, filesystems, HTTP, application configuration, object-graph magic, and other concerns that belong somewhere else.

It's small, tight, and opinionated.

I wrote a deeper dive on this one:

I Built a Serializer That Refuses to Own Your Data

FSM_REST

GitHub

This isn't another web framework.

FSM_REST provides a reusable boundary for describing and composing REST capabilities.

The interesting part is that a REST capability can be represented as a capability recipe rather than dragging an entire web application into the runtime.

Conceptually:

REST capability
      |
      v
FSM_REST
      |
      +-- capability description
      +-- request construction
      +-- transport boundary
      |
      v
remote API
      |
      v
current data

The MicroBundle can describe what is needed to obtain the capability without pretending that the remote API's current response data belongs inside the bundle.

That distinction becomes increasingly useful when capabilities are composed dynamically.

ProtocolAI & GrammarAI

ProtocolAI: GitHub
GrammarAI: GitHub

These sound like AI wrappers.

They aren't.

They explore the boundary where deterministic application-owned meaning meets probabilistic language models.

ProtocolAI is the WHAT.

It gives application-owned meaning a deterministic semantic address space.

GrammarAI is the HOW.

It gives those identities a structural language for describing how they can be connected.

LLM
 |
 | probability
 v
ProtocolAI
   WHAT
 |
 v
GrammarAI
   HOW
 |
 v
host / adapter
 |
 v
application

There is deliberately no LLM hidden inside either package.

No provider SDK.

No inference engine.

No chatbot.

No API-key management.

The application owns the meaning.

The model supplies probability.

The deterministic boundary sits in between.

I wrote more about the idea here:

ProtocolAI and GrammarAI: Giving AI a Language Software Can Actually Own

The Hosts

Architecture diagrams are great.

Eventually, though, somebody has to push a button.

AnyApp

GitHub

AnyApp is the desktop proof of the composition architecture.

It is the Windows desktop counterpart to the browser-facing WebPage/WebForge host.

The important part is that AnyApp is trying very hard to remain a host, rather than becoming an application-specific framework.

Startup is now manifest-driven:

AnyApp
  |
  v
startup / last Experience
  |
  v
Forge fallback when necessary
  |
  v
FSM_COS
  |
  v
RuntimeAssembly
  |
  v
host manifestation

The host doesn't need to know every Experience it might eventually run.

It hosts the composition.

That's the experiment.

And yes, the current window is tiny.

That's part of the fun.

WebPage

GitHub

This is the public proving ground.

WebPage is the ephemeral web manifestation of the larger Workshop: public-facing surface, demonstration environment, and experimental place where the architecture gets to meet an actual visitor.

The guiding idea is simple:

Make the mundane magical. Then show the machinery that made it possible.

Instead of merely putting diagrams on a webpage explaining that the GUI is dynamic, the page itself should eventually behave dynamically.

A button can breathe.

A GUI can reproduce.

The reproduction can become a swarm.

The swarm can overwhelm the screen.

The system can freeze.

Gravity can take over.

And suddenly the visitor isn't just reading about the architecture anymore.

They're watching the architecture become the experience.

That's what the WebPage repository is for.

You Don't Need to Understand All of This

Seriously.

You don't need to read every README.

You don't need to understand FSM_COS before touching FSM_API.

You don't need to understand MicroBundle arbitration before playing with the GUI.

You certainly don't need to agree with my architecture.

You don't need to know what a “semantic address space” is.

You don't need to know what “arbitration” means.

You don't even need to know why I keep naming things with increasingly suspicious acronyms.

Just pick one piece that looks interesting.

Install it.

Read the source.

Build a tiny experiment.

Benchmark it.

Break it.

Tell me why it's broken.

That is the entire point of publishing early.

Software has to meet reality.

And reality includes other developers.

Some APIs will change.

Some experiments will become load-bearing infrastructure.

Some ideas that seem perfectly reasonable right now will probably turn out to be terrible.

Some will get thrown away.

That's normal.

Actually, that's part of the process.

The important thing is that these aren't only diagrams anymore.

They're code.

They're packages.

They're repositories.

They're things you can actually touch.

The Workshop Is Open

So, that's the current toy box.

Not everything is finished.

Not everything is stable.

Not everything is even guaranteed to survive.

That's why I'm putting it out now.

If something catches your eye, go play with it.

And if you find something interesting, confusing, inefficient, backwards, brilliant, terrible, or just plain weird—

tell me.

That's how experiments turn into software.

And how software turns into infrastructure.

The toy box is open.

It's the weekend. Come play.

Support Our Work:

Patreon Page:
[https://www.patreon.com/c/TheSingularityWorkshop][1]

Support Us (PayPal Donation):
[https://www.paypal.com/donate/?hosted_button_id=3Z7263LCQMV9J][2]

We'd love to hear your thoughts! Please Like this post, Love the code, and Share your feedback in the comments.

From a discussion about evidence systems to a real architecture improvement

Thu, 01 Oct 2026 00:13:18 +0000

From a discussion about evidence systems to a real architecture improvement

A few hours ago we had a valuable discussion about a critical problem in knowledge and verification systems:

How do we prevent a graph from claiming more than its evidence can support?

This question directly influenced the next iteration of the MyZubster architecture.

The key insight:

A verification status is not evidence by itself.

A field like:

verification: confirmed

is only meaningful if the system can explain:

Who confirmed it?
How was it verified?
Which source was checked?
When was it observed?
What are the limits of that verification?

What we implemented

Based on this discussion, we introduced a new layer:

Knowledge Claim
        |
        ↓
Evidence Source
        |
        ↓
Knowledge Attestation
        |
        ↓
Verifier + Method + Timestamp

The goal is simple:

The graph should not declare truth.

The graph should preserve the chain of evidence behind a claim.

Contributor Knowledge Passport

At the same time, we built the first version of a Contributor Knowledge Passport.

Instead of representing contributors only through:

payments
wallets
completed tasks

we are creating a model where contributions become connected to:

rewards
knowledge records
evidence
attestations
reputation

A contributor is not only someone who receives a reward.

A contributor creates knowledge that can become part of the infrastructure.

Why this matters

Many systems today store outcomes:

"completed"
"verified"
"approved"

but they often lose the history behind those words.

We are experimenting with a different approach:

Every important statement should have provenance.

Not only:

"What happened?"

but also:

"How do we know?"

"Who observed it?"

"What supports this conclusion?"

Thank you to the builders who challenge assumptions

This improvement came from an external technical discussion and a contributor pushing the design question further.

Good architecture is not created by avoiding criticism.

It is created by turning the right questions into better systems.

The next steps:

Contributor
      |
      ↓
Knowledge Contribution
      |
      ↓
Evidence Graph
      |
      ↓
Attestation Layer
      |
      ↓
Reputation

Building in public means sharing not only features, but also the reasoning behind the architecture.

Thanks to everyone contributing ideas, reviews, and challenges.

The best systems are built when knowledge can flow back into the system itself.

opensource #softwarearchitecture #knowledgegraphs #AI #developers #web3 #buildinpublic

ProtocolAI and GrammarAI: Giving AI a Language Software Can Actually Own

Tue, 29 Sep 2026 22:13:13 +0000

There is a particular problem that keeps appearing as I build the architecture for The Singularity Workshop.

It starts with something that sounds almost trivial:

How does software communicate with an AI when the software already knows what its world means?

An LLM is very good at language.

It can interpret a request, generate possibilities, describe relationships, and produce something that looks structurally correct.

But there is a boundary where the application has to take over.

The application needs to know exactly what something refers to.

And then it needs to know how those things may be connected.

That is where two small packages have emerged:

ProtocolAI — the WHAT
GrammarAI — the HOW

They are deliberately separate.

And neither one is the AI.

The Problem Starts With Probability

An LLM operates in a probabilistic space.

That is not a criticism. It is the nature of the technology.

Given some context, the model produces a candidate response.

That candidate may be exactly what we wanted.

It may also be something new.

Or something that looks familiar but does not actually identify the same object the application already knows.

Imagine an application that already knows three people:

Bob
Jane
Sara

The application has meaning attached to those names.

The model does not own that meaning.

It can produce "Bob".

But the application needs to turn that representation into something deterministic.

That is the boundary ProtocolAI explores.

probabilistic language
        |
        v
     candidate
        |
        v
    ProtocolAI
        |
    +---+---+
    |       |
  known   unknown
    |       |
    v       v
 integer   literal
    |       |
    +---+---+
        |
        v
application-owned identity

The model proposes.

The application decides.

ProtocolAI provides the semantic address space between those two worlds.

ProtocolAI: The WHAT

The core idea behind ProtocolAI is surprisingly small.

A protocol owns a vocabulary.

For example:

[1001] People

[2001] bobId  = "Bob"
[2002] janeId = "Jane"
[2003] saraId = "Sara"

Now the application has an address space.

"Bob"   -> [2001]
"Jane"  -> [2002]
"Sara"  -> [2003]

The integer is not the meaning.

It is the address of the meaning.

That distinction is important.

ProtocolAI does not replace the domain object.

It does not become the database.

It does not become the authority that decides what Bob is allowed to do.

It simply establishes a deterministic identity that the application owns.

The Literal Escape Hatch

There is another important detail.

What happens when the model produces something the application does not already know?

Suppose the vocabulary contains:

[2001] Bob
[2002] Jane

and the model produces:

"Amelia"

The protocol should not have to pretend Amelia already exists.

So ProtocolAI preserves the literal.

[2001] [2002] "Amelia"

That gives the host a choice.

The host can decide whether Amelia should be:

created;
registered;
rejected;
authorized;
deferred;
or otherwise handled.

That is a much more interesting boundary than simply forcing everything into a closed enumeration.

Known things become references.

Unknown things remain visible as unknown.

This Is Not Just About Compression

It would be easy to look at integer identities and conclude that this is merely an optimization.

Smaller values.

Smaller payloads.

Faster lookup.

Those may eventually be useful properties.

But that is not the fundamental idea.

The more important separation is:

Meaning
   |
   v
Identity
   |
   v
Representation

Consider:

Meaning:        Bob
Symbol name:    bobId
Symbol ID:      2001
Protocol ID:    1001
Payload form:   [2001]

The application already owns the meaning.

ProtocolAI gives that meaning an explicit address.

That address can then survive across interactions.

Prompt
"Bob"
   |
   v
[2001]
   |
   v
Response
[2001]
   |
   v
Next interaction
[2001]

The model does not have to repeatedly rediscover that two strings refer to the same application-owned identity.

Again, that is an architectural direction rather than a claim that integer identities automatically make an LLM reason better.

The point is that the application has a semantic address space.

And Then I Ran Into the Next Problem

A vocabulary is useful.

But a vocabulary is not a language.

Knowing:

[2001] Bob
[2002] Jane

does not tell us how those identities may be arranged.

We need another layer.

Something has to describe relationships such as:

this kind of thing
    |
    +--> may contain this
    |
    +--> may reference that
    |
    +--> may be followed by another structure

That is where GrammarAI enters.

GrammarAI: The HOW

GrammarAI takes the identities supplied by ProtocolAI and gives them a structural language.

The distinction is intentionally simple:

ProtocolAI defines the WHAT. GrammarAI defines the HOW.

For example:

ProtocolAI

[1001] People

[2001] Bob
[2002] Jane

GrammarAI can define a structure that references those identities:

[4001] -> [1001:2001]
[4001] -> [1001:2002]

Now we have two different owners.

[1001:2001]
    |
    +--> ProtocolAI owns the meaning

[4001]
    |
    +--> GrammarAI owns the structural position

That separation is deliberate.

GrammarAI does not copy Bob into itself.

It references the protocol that owns Bob.

A Grammar Is a Relationship System

This is the part I find particularly interesting.

A grammar is not another dictionary.

It is a relationship system.

It can say:

Grammar nonterminal
       |
       v
     [4001]
       |
       +----> [1001:2001]
       |
       +----> [1001:2002]

The grammar owns the relationship.

The protocol owns the thing being referenced.

That gives us a clean composition boundary.

Protocol A
    |
    +-- symbols

Protocol B
    |
    +-- symbols

       \       /
        \     /
         \   /
        Grammar
           |
           +-- structure
           +-- relationships
           +-- references

The grammar connects things without becoming the owner of those things.

Why Integer-Backed Structure?

There is another common thread between the two packages.

Both are self-defining.

A GrammarAI definition has its own identity:

[3001] Greeting

It has a start symbol:

start = [4001]

And it has production rules:

[5001] [4001] -> [1001:2001]
[5002] [4001] -> [1001:2002]

That means the grammar itself can become a piece of data.

It can eventually be:

described;
serialized;
compared;
versioned;
generated;
visualized;
translated;
supplied to another composition layer.

The grammar does not need to know which of those things will happen.

That is the point of keeping it small.

The AI Still Isn't the Authority

This is probably the most important part of the architecture.

The LLM does not become the owner of the protocol.

The LLM does not become the owner of the grammar.

And the LLM does not become the application's execution authority.

The flow is closer to:

              LLM
               |
          probabilistic
            candidate
               |
               v
          ProtocolAI
             WHAT
               |
               v
           GrammarAI
              HOW
               |
               v
        host validation
               |
               v
       application decision
               |
               v
       deterministic state

The model participates.

It does not rule the system.

That distinction becomes increasingly important as AI systems begin interacting with real applications rather than simply generating text.

This Is the Same Boundary I Keep Finding Elsewhere

Consider a reservation system.

User A requests the last available table.

The centralized authority accepts the reservation.

Users B through K arrive almost immediately afterward.

The system does not need to tell User K:

Failure.

That is technically accurate but semantically terrible.

The actual state is:

You cannot acquire the resource right now.

So the system can return something closer to:

Resource unavailable.

You are currently #10 in the waiting queue.
There are 9 requests ahead of you.

That is an inability, not necessarily a failure.

The system has preserved useful information about the state of the world.

And that is the kind of distinction these AI packages are trying to preserve as well.

An LLM can produce a candidate.

ProtocolAI determines whether that candidate corresponds to a known application identity.

GrammarAI determines whether that identity participates in a valid structure.

The host then decides what that structure means operationally.

At every step, we preserve the distinction between:

what was proposed
what is known
what is structurally valid
what is currently possible
what the application actually decides to do

That is much more useful than collapsing everything into true or false.

ProtocolAI and GrammarAI Are Deliberately Not an AI Framework

This is where the package boundaries become important.

ProtocolAI is not:

an LLM;
an AI provider;
a prompt framework;
a schema validator;
a database;
a grammar compiler;
a command executor;
a GUI framework;
a MicroBundle host.

GrammarAI is not:

an LLM client;
model selection;
tokenization;
inference;
prompt transport;
protocol vocabulary ownership;
MicroBundle hosting;
REST/OpenAPI;
GUI manifestation;
tool execution.

That list may look like a lot of things to leave out.

That is intentional.

The packages are supposed to provide semantic infrastructure, not become another giant AI framework.

The Exchange Layer Comes Later

ProtocolAI already describes an interesting future boundary for AI interaction.

The same semantic exchange should eventually work through either:

             AI Exchange
                 |
        +--------+--------+
        |                 |
    Clipboard         Provider
        |              Adapter
        |                 |
        +--------+--------+
                 |
                LLM

Clipboard interaction is not treated as a second protocol.

It is simply another transport.

A developer could extract:

Protocol
Grammar
Context
Request

copy that into an LLM, receive a response, and paste it back.

A connected provider could perform the same semantic exchange programmatically.

The transport changes.

The semantic boundary does not.

That is an important distinction.

Context Is Not Authority

An AI exchange might eventually contain a substantial amount of information:

current Experience
selected objects
available commands
protocol descriptions
grammar definitions
current state
previous interactions
host capabilities

But none of that means the model has authority.

The intended boundary remains:

MODEL OUTPUT
     |
     v
parse / validate
     |
     v
ProtocolAI resolution
     |
     v
host policy
     |
 +---+---+---+---+
 |   |   |   |   |
accept reject clarify create execute

The model proposes.

The host decides.

ProtocolAI makes owned meaning addressable.

GrammarAI makes the relationships between those meanings describable.

And This Is Where the Larger Workshop Architecture Appears

The two packages are small individually.

But together they establish another layer in the larger architecture I have been building.

Domain meaning
      |
      v
ProtocolAI
    WHAT
      |
      v
GrammarAI
    HOW
      |
      v
AI / provider adapter
      |
      v
Host
      |
      v
FSM / GUI / Experience

And eventually, those capabilities can participate in the MicroBundle architecture.

That means the AI layer does not have to become a special universe disconnected from the rest of the system.

It can become another collection of capabilities that the host can compose.

The Interesting Part Is the Separation

There is a temptation when building AI software to put everything in one place.

The model client.

The prompts.

The schemas.

The grammar.

The commands.

The domain objects.

The execution engine.

The UI.

The credentials.

The networking.

Eventually you have an enormous framework that knows everything about everything.

I am deliberately going in the other direction.

ProtocolAI knows about semantic identity.

GrammarAI knows about semantic structure.

The host knows about execution.

The provider adapter knows about transport.

The application knows about meaning and authority.

That makes the pieces composable.

Where the Alpha Actually Stands

These packages are intentionally early.

ProtocolAI 0.1.0-alpha.2 currently establishes:

self-defining integer-backed vocabularies;
protocol-qualified identity;
known-value encoding;
literal preservation;
deterministic descriptions;
ordered mixed payloads;
payload validation;
mixed reference/literal resolution.

GrammarAI 0.1.0-alpha.1 currently establishes:

grammar identity;
integer start symbols;
integer nonterminals;
ordered production rules;
external ProtocolAI references;
deterministic self-description;
validation of referenced nonterminals.

Neither package currently implements the entire AI pipeline.

And that is okay.

The unanswered questions are actually interesting architectural questions.

For example:

Who allocates new symbol IDs?

How do identities persist?

How are protocols versioned?

How are protocols negotiated?

How are grammars versioned?

How are recursive productions handled?

How does an abstract grammar become a provider-specific constraint?

How do we validate generated model output?

How does a semantic artifact become executable behavior?

Those questions should be solved by the architecture.

They should not be hidden inside convenience APIs simply because the API would look nicer.

The Bigger Experiment

What I am really exploring here is a progression.

human / domain language
          |
          v
    self-defining lexicon
          |
          v
     integer identity
          |
          v
    grammar / structure
          |
          v
       protocol
          |
          v
     tool execution
          |
          v
       experience

ProtocolAI is the first semantic compression boundary.

GrammarAI is the next structural boundary.

Neither is trying to replace human language.

Neither is trying to make an LLM deterministic.

Instead, they give software something it has always needed:

a precise address space for meanings that the software itself owns.

And once those meanings have addresses, and those addresses have structure, we have something much more interesting than an AI prompt.

We have the beginning of a language that the application can actually own.

The Current Workshop Stack

The current direction now looks something like this:

                    LLM
                     |
              probability
                     |
                     v
               ProtocolAI
                  WHAT
                     |
                     v
                GrammarAI
                   HOW
                     |
                     v
             AI / Host Adapter
                     |
                     v
                  Host
                     |
                     v
                FSM_COS
                     |
              MicroBundles
                     |
                     v
              RuntimeAssembly
                     |
                     v
            GUI / Experience
                     |
                     v
             Digital Reality

The interesting part is that the model remains at the edge.

The application owns the center.

That is where I want the architecture to live.

Resources

Support The Singularity Workshop

If this work is useful to you, and you'd like to help keep The Singularity Workshop moving, you can support its continued development:

Patreon: Support The Singularity Workshop
PayPal: Make a donation

Every bit of support helps provide more time and resources for building, testing, documenting, and publishing the architecture.

The Singularity Workshop — Tools for the curious, the bold, and the systemically inclined.
Because state shouldn't be a mess.

I Built a Serializer That Refuses to Own Your Data

Mon, 28 Sep 2026 23:36:42 +0000

I Built a Serializer That Refuses to Own Your Data

How The Singularity Workshop turned a recurring binary-IO problem into a tiny .NET boundary—and why the smallest part of the design may be the most important part.

There is a particular kind of software problem that appears harmless at first.

You have some state.

You need to turn it into bytes.

So you write a serializer.

Then another part of the application needs bytes, so you add another serializer.

Then a cache needs them.

Then a manifest needs them.

Then a transport layer needs them.

Then somebody decides JSON is better here, binary is better there, and suddenly the architecture has six different opinions about what a piece of data is.

That was the problem I was running into while building The Singularity Workshop.

The interesting part is that I didn't actually need a giant serialization framework.

I needed a boundary.

So I built one.

It is called:

TheSingularityWorkshop.FSM_Serialization

And it is deliberately small.

That is the point.

The Problem Wasn't Turning Objects Into Bytes

Serialization is usually described as a mechanical operation:

object → serializer → bytes

That description hides something important.

An object has meaning.

Bytes do not.

Bytes are a representation of meaning.

That distinction becomes increasingly important as an architecture grows.

The Workshop ecosystem is being built around several deliberately separated responsibilities:

FSM_API
    │
    │ behavior and state
    ▼
FSM_COS
    │
    │ composition and runtime assembly
    ▼
FSM_Serialization
    │
    │ representation
    ▼
storage / transport / host

I didn't want FSM_COS to become a serializer.

I didn't want a MicroBundle to have to know how a filesystem works.

I didn't want a database abstraction hiding inside a representation layer.

And I definitely didn't want every higher-level system inventing its own binary stream interface.

So I extracted the smallest reusable boundary I could identify.

That boundary became FSM_Serialization.

The First Design Decision

The most important rule in the package is also the simplest:

The type that owns the meaning should own the representation rules.

FSM_Serialization provides the machinery for crossing the byte boundary.

It does not decide what your data means.

That means the package doesn't contain some enormous reflection-based object graph engine that looks at your class and decides what should be serialized.

Instead, the type explicitly says:

public interface IBinaryPackable
{
    void Pack(IBinaryStream stream);
}

And the reverse direction is equally explicit:

public interface IBinaryUnpackable
{
    void Unpack(IBinaryStream stream);
}

If a type needs both:

public interface IBinarySerializable :
    IBinaryPackable,
    IBinaryUnpackable
{
}

That's the entire conceptual model.

No magic.

No global serializer registry.

No reflection-based guesswork.

No framework deciding that a private field suddenly became part of your persistent format.

The domain owns that decision.

Why I Didn't Just Use `System.IO.Stream`

This is probably the first question a .NET developer will ask.

And honestly, it is a good question.

.NET already has System.IO.Stream.

Why introduce another interface?

Because I wasn't trying to replace Stream.

I was trying to establish a package-owned contract.

The current abstraction is:

public interface IBinaryStream : IDisposable
{
    bool CanRead { get; }
    bool CanWrite { get; }

    long Position { get; set; }
    long Length { get; }

    int Read(Span<byte> buffer);
    void Write(ReadOnlySpan<byte> buffer);
    void Flush();
}

Notice what isn't there.

There is no:

filename;
filesystem;
database;
socket;
HTTP request;
serializer registry;
schema;
domain metadata;
application configuration.

The interface only knows about the byte boundary.

That makes it possible to write serialization code against one very small abstraction while letting the host decide what is actually underneath it.

The Two Directions Matter

There is another small decision here that I think becomes important in larger systems.

Packing and unpacking are separate capabilities.

A type that produces a representation does not necessarily need to consume one.

Likewise, a type may consume a representation without being responsible for producing it.

So the package doesn't force everything into one giant:

Serialize()
Deserialize()

method.

Instead:

                 DOMAIN TYPE
                     │
          ┌──────────┴──────────┐
          │                     │
        Pack                  Unpack
          │                     │
          ▼                     ▲
     representation          representation
          │                     │
          └──────────┬──────────┘
                     ▼
                byte stream

That separation costs almost nothing.

But architecturally, it keeps the ownership visible.

The Smallest Possible Example

Let's make this concrete.

Imagine a domain object containing one integer.

That's intentionally boring.

Boring is useful here because it lets us see exactly where the responsibility lives.

using System.Buffers.Binary;
using TheSingularityWorkshop.FSM_Serialization;

public sealed class ExampleState : IBinarySerializable
{
    public int Value { get; private set; }

    public ExampleState()
    {
    }

    public ExampleState(int value)
    {
        Value = value;
    }

    public void Pack(IBinaryStream stream)
    {
        Span<byte> buffer = stackalloc byte[sizeof(int)];

        BinaryPrimitives.WriteInt32LittleEndian(
            buffer,
            Value);

        stream.Write(buffer);
    }

    public void Unpack(IBinaryStream stream)
    {
        Span<byte> buffer = stackalloc byte[sizeof(int)];

        if (stream.Read(buffer) != buffer.Length)
            throw new EndOfStreamException();

        Value = BinaryPrimitives.ReadInt32LittleEndian(buffer);
    }
}

There is no serializer object.

There is no configuration object.

There is no registration step.

The object itself defines its representation.

That is intentional.

Now Let's Actually Serialize Something

The package includes MemoryBinaryStream, which is exactly what it sounds like: an in-memory implementation of IBinaryStream.

That makes testing and temporary representations extremely simple.

using var stream = new MemoryBinaryStream();

var original = new ExampleState(42);

original.Pack(stream);

stream.Position = 0;

var restored = new ExampleState();

restored.Unpack(stream);

Console.WriteLine(restored.Value);

Output:

That's it.

The complete flow is:

ExampleState
     │
     │ Pack
     ▼
MemoryBinaryStream
     │
     │ bytes
     ▼
Position = 0
     │
     │ Unpack
     ▼
ExampleState

The package supplied the boundary.

The domain supplied the meaning.

Installation

If you want to try it yourself, the current published alpha is:

dotnet add package TheSingularityWorkshop.FSM_Serialization --version 0.1.0-alpha.2

Or:

<PackageReference
    Include="TheSingularityWorkshop.FSM_Serialization"
    Version="0.1.0-alpha.2" />

The current package targets .NET 8.

This is intentionally an alpha release.

That doesn't mean the contracts are casual.

Quite the opposite.

A binary representation is one of those things where small assumptions can become enormous compatibility problems later.

So I'm treating the package as alpha while treating its architectural boundaries seriously.

What About Real Files?

This is where StreamBinaryStream enters.

The package includes an adapter around an ordinary .NET Stream:

using var file = File.OpenRead("state.bin");
using var stream = new StreamBinaryStream(file);

Now the same domain serialization code can operate against the package abstraction without knowing that a file exists.

The relationship is:

ExampleState
     │
     ▼
IBinaryStream
     │
     ▼
StreamBinaryStream
     │
     ▼
System.IO.Stream
     │
     ▼
file / memory / whatever the host provides

That is a very different architecture from:

ExampleState
     │
     ▼
ExampleStateSerializer
     │
     ▼
FileSystemSerializerManager
     │
     ▼
ApplicationStorageService

The latter might be useful somewhere.

It just doesn't belong in this package.

The Stream Adapter Is Supposed to Be Boring

This is one of those cases where boring code is a compliment.

StreamBinaryStream essentially delegates:

public int Read(Span<byte> buffer)
    => _stream.Read(buffer);

public void Write(ReadOnlySpan<byte> buffer)
    => _stream.Write(buffer);

public void Flush()
    => _stream.Flush();

That's not a weakness.

It's the point.

The adapter is not supposed to invent behavior.

It establishes a stable contract between two layers.

Then I Ran Into a Very Important Detail

A byte stream is not the same thing as an array.

This matters:

int count = stream.Read(buffer);

You cannot blindly assume:

count == buffer.Length

So a fixed-width decoder in FSM_Serialization explicitly checks:

if (stream.Read(buffer) != buffer.Length)
    throw new EndOfStreamException();

That may look like a tiny implementation detail.

It isn't.

It establishes an architectural rule:

Never interpret an incomplete representation as a complete value.

The serializer shouldn't quietly manufacture missing data.

If the representation is incomplete, the domain implementation gets to decide what that means.

In the current examples, that means rejecting it.

Serialization Is Not Storage

This distinction became increasingly important as the Workshop architecture grew.

Consider these questions:

Serialization

How are these values represented as bytes?

Storage

Where do those bytes live?

Transport

How do those bytes move between boundaries?

Those are three different questions.

So:

                 SEMANTIC DATA
                      │
                      ▼
                representation
                      │
                      ▼
              FSM_Serialization
                      │
              ┌───────┴───────┐
              ▼               ▼
           storage         transport

A database is not a serializer.

A filesystem is not a serializer.

An HTTP endpoint is not a serializer.

A cache is not a serializer.

They may all consume serialized data.

That doesn't make them the same architectural layer.

This is exactly the kind of distinction that seems unnecessary when an application is small and becomes incredibly valuable when an ecosystem starts growing.

Why Binary?

I am not claiming that binary is universally superior to JSON.

It isn't.

JSON is fantastic for many things.

Text is fantastic for many things.

Human-readable configuration is valuable.

Debugging matters.

Authoring matters.

Interoperability with external systems matters.

The reason I created this package is narrower.

There are situations where I want an explicit, compact, byte-oriented representation.

For example:

runtime state;
manifests;
dependency metadata;
compact configuration;
cache entries;
identity records;
version records;
lineage metadata;
persistence;
transport payloads.

The important thing is that the semantic model shouldn't have to become identical to its wire representation.

Deterministic Does Not Mean Magical

The package's NuGet description talks about deterministic binary serialization.

There is an important qualification.

Implementing:

IBinaryPackable

does not magically make your representation deterministic.

You have to define deterministic rules.

For example, suppose your object contains a collection that is semantically unordered.

If you serialize it in whatever order happens to be present at runtime, two semantically identical objects could produce different byte sequences.

If deterministic bytes matter, the domain needs something like:

semantic collection
        │
        ▼
canonical ordering
        │
        ▼
explicit representation
        │
        ▼
deterministic bytes

That distinction matters if bytes are later used for:

caching;
content identity;
synchronization;
comparison;
reproducible builds;
lineage;
integrity checks.

Again, the package supplies the boundary.

It does not pretend to know your domain's canonical ordering rules.

Versioning Gets Interesting

This is another reason I didn't want a magical object serializer.

Once bytes leave your process, they can outlive the code that created them.

Imagine:

Application v1
      │
      ▼
    bytes
      │
      │ time passes
      ▼
Application v2

What happens?

That is a representation contract question.

A durable binary format may need to distinguish between:

semantic object version
format version
dependency version

Those are not necessarily the same thing.

Your semantic object can change while the representation remains compatible.

Your representation can change while the semantic concept remains the same.

A dependency can change independently of both.

FSM_Serialization intentionally doesn't impose one universal versioning strategy.

Different domains need different compatibility policies.

What it does is make the boundary explicit enough that you can actually have that conversation.

The Package Came From a Real Application

This is probably the most important part of the story.

FSM_Serialization wasn't created because I thought:

"The world needs another serialization library."

There are already plenty of serialization libraries.

The package emerged from building the Workshop itself.

The development pattern was:

working application
       │
       │ repeated infrastructure appears
       ▼
explicit contract
       │
       │ extract
       ▼
reusable package
       │
       │ consume again
       ▼
working application

That distinction matters.

I wasn't trying to abstract everything before I had a reason to abstract it.

The application became the proving ground.

Once the boundary became clear, I extracted it.

That is a much healthier way to create infrastructure.

And This Is Where FSM_COS Enters the Story

The larger ecosystem is becoming increasingly compositional.

FSM_COS is concerned with questions like:

What runtime composition was requested?

Which MicroBundles are required?

What dependencies are reachable?

How do those capabilities reconcile?

When is the composition stable enough to hand to a host?

FSM_Serialization answers a different question:

How do we represent the values that need to cross a byte boundary?

That means a future conceptual flow can look something like:

Runtime Manifest
      │
      ▼
    FSM_COS
      │
      │ composition
      ▼
Runtime Assembly
      │
      │ representation
      ▼
FSM_Serialization
      │
      ▼
     bytes

And then those bytes might eventually go somewhere else:

                 FSM_Serialization
                        │
             ┌──────────┼──────────┐
             ▼          ▼          ▼
          storage    transport    cache
             │          │          │
             └──────────┼──────────┘
                        ▼
                      host

The serializer doesn't become FSM_COS.

FSM_COS doesn't become a serializer.

That separation is exactly what I'm trying to preserve.

The Tests Tell the Same Story

The package currently has a deliberately small test surface.

It verifies things such as:

an empty MemoryBinaryStream is readable and writable;
bytes survive a write/read round trip;
existing byte arrays can initialize a memory stream;
null input is rejected;
end-of-input returns zero;
positioning and overwriting work;
StreamBinaryStream delegates correctly;
null wrapped streams are rejected;
writes reach the underlying stream;
IBinaryPackable and IBinaryUnpackable can round-trip;
IBinarySerializable combines both contracts;
incomplete binary input is rejected.

One of the tests is essentially the entire architectural idea in executable form:

var original = new TestValue(123456789);

using var stream = new MemoryBinaryStream();

original.Pack(stream);

stream.Position = 0;

var restored = new TestValue();

restored.Unpack(stream);

Assert.Equal(original.Value, restored.Value);

Tests aren't merely protecting the implementation here.

They are defining the boundary.

I Also Benchmarked It

Because eventually somebody is going to ask:

"Okay, but what does that abstraction cost?"

Good.

Let's measure it.

The companion benchmark project uses BenchmarkDotNet against .NET 8 and exercises payloads from 16 bytes through 100,000 bytes.

The interesting result isn't that FSM_Serialization magically beats MemoryStream.

It doesn't.

That's not what I was trying to prove.

The useful result is that the abstraction can remain extremely close to the underlying primitive for substantial workloads.

At 100,000 bytes, the benchmark measured approximately:

MemoryStream write:
39,894.85 ns

MemoryBinaryStream write:
40,005.59 ns

That's effectively the same scale.

Likewise, the adapter path was approximately:

MemoryStream read:
2,272.47 ns

StreamBinaryStream read:
2,298.40 ns

Again, very close.

And the end-to-end IBinarySerializable benchmark measured:

16 bytes       64.08 ns
1,024 bytes   207.47 ns
10,000 bytes  1,481.61 ns
100,000 bytes 60,152.74 ns

But there is an important qualification.

The benchmark measures the whole operation, including things like creating the test value, creating the stream, packing, rewinding, allocating the destination payload, and unpacking.

So I'm not going to pretend those numbers represent some universal "serialization speed."

They represent a measured workload.

That's much more useful.

The Benchmark Question I Actually Care About

The interesting engineering question isn't:

"Is my interface faster than MemoryStream?"

The interface isn't supposed to be the star of the show.

The question is:

Can I establish a useful architectural boundary without making the underlying operation prohibitively expensive?

The current measurements suggest that, for these workloads, the answer is yes.

And that's exactly where I want infrastructure to live.

The abstraction should be visible architecturally and nearly invisible computationally.

What I Deliberately Did Not Build

This may be the most important section of the entire package.

FSM_Serialization should not become:

an FSM engine;
a game framework;
a GUI framework;
a browser framework;
a Unity integration layer;
a database abstraction;
a filesystem framework;
an HTTP client;
an application configuration system;
a MicroBundle implementation;
an Experience runtime;
a reflection-driven object graph serializer;
a global serializer registry.

Those things may all be useful.

They just aren't this thing.

That restraint is part of the architecture.

A small package becomes powerful when other systems can depend on it without having to inherit the systems it was never supposed to understand.

So What Would You Actually Use It For?

If you're building a .NET application and you have a domain type whose state needs to cross a byte-oriented boundary, the starting point is incredibly small.

Install the package:

dotnet add package TheSingularityWorkshop.FSM_Serialization --version 0.1.0-alpha.2

Define your representation:

public sealed class MyState : IBinarySerializable
{
    public int Id { get; set; }

    public void Pack(IBinaryStream stream)
    {
        Span<byte> buffer = stackalloc byte[sizeof(int)];

        BinaryPrimitives.WriteInt32LittleEndian(
            buffer,
            Id);

        stream.Write(buffer);
    }

    public void Unpack(IBinaryStream stream)
    {
        Span<byte> buffer = stackalloc byte[sizeof(int)];

        if (stream.Read(buffer) != buffer.Length)
            throw new EndOfStreamException();

        Id = BinaryPrimitives.ReadInt32LittleEndian(buffer);
    }
}

Use memory when you need a temporary representation:

using var stream = new MemoryBinaryStream();

state.Pack(stream);

byte[] bytes = stream.ToArray();

Use a normal .NET stream when the host provides one:

using var file = File.Create("state.bin");
using var stream = new StreamBinaryStream(file);

state.Pack(stream);

And that's basically the entire learning curve.

The difficult part isn't learning the API.

The difficult part is deciding what your domain actually means by its representation.

I think that's exactly where the difficult part belongs.

A Serializer Shouldn't Become Your Architecture

This is the idea that kept resurfacing while I built the package.

It's very easy for infrastructure to become the thing it was supposed to merely support.

A serializer starts storing metadata.

Then it starts deciding schemas.

Then it starts managing versions.

Then it starts talking to databases.

Then it starts managing transports.

Then it starts discovering domain types.

Eventually your "serialization library" is quietly running your architecture.

I wanted the opposite.

I wanted something I could put underneath the architecture and forget about until I actually needed to cross the boundary.

So the package can be reduced to one equation:

semantic state
      +
explicit representation rules
      +
byte boundary
      =
portable representation

Not:

object
  +
magic serializer
  =
architecture

That's the difference.

The Boundary I Want to Preserve

The entire package can ultimately be reduced to this:

                 DOMAIN
                   │
          owns meaning and rules
                   │
                   ▼
          Pack / Unpack contract
                   │
                   ▼
          FSM_Serialization
                   │
              byte boundary
                   │
        ┌──────────┴──────────┐
        ▼                     ▼
     storage               transport
        │                     │
        └──────────┬──────────┘
                   ▼
                  HOST
                   │
             manifestation

The serializer owns the boundary.

The domain owns the meaning.

The host owns the destination.

That is the architecture.

Why I Think Small Infrastructure Is Worth Talking About

There is a tendency in software writing to focus on the enormous systems.

The massive framework.

The clever algorithm.

The huge benchmark.

The complicated application.

But some of the most consequential architectural decisions are tiny.

An interface.

A boundary.

A decision about ownership.

A decision about what a component doesn't know.

FSM_Serialization is one of those experiments.

It isn't trying to replace every serialization technology in .NET.

It isn't trying to declare that binary is the future.

It is much more modest:

I needed a clean byte boundary for a growing compositional architecture, so I built the smallest one I could.

Then I tested it.

Then I benchmarked it.

Then I packaged it.

And now I'm putting it out there so somebody else can kick it.

You Don't Have to Take My Word for It

That's ultimately why I publish these projects.

Install it.

Implement one type.

Serialize something you actually care about.

Put it under your own workload.

Decide whether the ownership model makes sense to you.

Measure it.

Break it.

Tell me where the boundary doesn't hold.

That's much more interesting to me than somebody simply agreeing with the architecture.

The package is available now as an alpha.

And if you find a place where this tiny boundary lets you remove a surprisingly large amount of architectural mess, I'd genuinely like to hear about it.

Support Our Work:

Patreon Page:
https://www.patreon.com/c/TheSingularityWorkshop

Support Us (PayPal Donation):
https://www.paypal.com/donate/?hosted_button_id=3Z7263LCQMV9J

We'd love to hear your thoughts! Please Like this post, Love the code, and Share your feedback in the comments.

Ditching the Monolith: An Introduction to Multi-Agent Systems for Node.js Devs

Wed, 23 Sep 2026 08:50:33 +0000

In the world of modern software design, we are constantly searching for ways to make our systems more resilient, adaptable, and scalable. If you have spent your career building traditional monolithic applications or synchronous microservices, you are likely familiar with the headaches of centralized control. When one coordinator service fails, the entire application can grind to a halt. To solve this, forward-thinking engineers are turning to a design paradigm known as multi-agent systems.

What is a Multi-Agent System?

A multi-agent system is a network of independent, self-contained software programs (called "agents") that work together to solve complex problems. Instead of relying on a single, massive program to make every decision, these systems break tasks down among specialized digital assistants that communicate and coordinate with one another. Each agent has its own specific goal, operational rules, and local knowledge, allowing them to cooperate to achieve a larger objective that none of them could finish alone.

The Kitchen Analogy: Who Is Cooking Your Order?

To understand how this behaves, think of a busy professional restaurant kitchen. You do not have one single super-chef cooking every single dish, washing the plates, taking customer orders, and running food to tables. That would create an immediate bottleneck.

Instead, you have a head chef, a sous chef, a pastry chef, and a dishwasher. Each of these people is an "agent" with a highly specific role. They do not need to know the exact details of how the others perform their tasks; the pastry chef does not need to know how to clean a fish, and the dishwasher does not need to know how to bake a soufflé. However, they pass messages and resources back and forth (such as "order up!" or "need clean plates!") to run the entire kitchen smoothly. If the pastry chef temporarily falls behind, the rest of the kitchen can still prepare appetizers and main courses independently.

Why It Matters Daily in the Tech Industry

In the tech industry, engineers use multi-agent architectures to prevent system-wide bottlenecks and cascading failures. In standard APIs, if your main process gets overwhelmed with database operations, your user interface freezes. By distributing these responsibilities among autonomous agents, you decouple your system's critical paths.

For example, in a high-traffic e-commerce platform, instead of having one monolithic script process a payment, update inventory, and send confirmation emails sequentially, you delegate these tasks to dedicated agents. If the notification agent experiences an outage or third-party API rate limit, the payment and inventory agents keep operating without interruption. This separation allows software teams to scale individual components dynamically and deploy updates to one agent without risking the stability of the entire network.

Simulating Multi-Agent Coordination in Node.js

Below is a simple Node.js example demonstrating how multiple independent agents can coordinate tasks using an event-driven system. We will set up an OrderAgent, an InventoryAgent, and a NotificationAgent that communicate via an event emitter.

const EventEmitter = require('events');
const agentBus = new EventEmitter();

// Agent 1: Takes care of incoming customer requests
class OrderAgent {
  constructor() {
    agentBus.on('order_placed', (order) => this.processOrder(order));
  }

  processOrder(order) {
    console.log(`[OrderAgent] Received order #${order.id}. Passing to inventory...`);
    // Trigger the next autonomous agent
    agentBus.emit('check_inventory', order);
  }
}

// Agent 2: Manages physical product availability
class InventoryAgent {
  constructor() {
    agentBus.on('check_inventory', (order) => this.verifyStock(order));
  }

  verifyStock(order) {
    console.log(`[InventoryAgent] Checking stock for ${order.item}...`);
    const itemInStock = true; // Simulating a local database lookup
    
    if (itemInStock) {
      console.log(`[InventoryAgent] Stock confirmed for order #${order.id}.`);
      agentBus.emit('dispatch_notification', order);
    } else {
      console.log(`[InventoryAgent] Stock unavailable for order #${order.id}.`);
    }
  }
}

// Agent 3: Handles user communications
class NotificationAgent {
  constructor() {
    agentBus.on('dispatch_notification', (order) => this.sendReceipt(order));
  }

  sendReceipt(order) {
    console.log(`[NotificationAgent] Email sent: Order #${order.id} for a ${order.item} is confirmed!`);
  }
}

// Instantiate our autonomous agents to start listening
new OrderAgent();
new InventoryAgent();
new NotificationAgent();

// Simulate a customer purchase
console.log('--- Initiating Customer Order Flow ---');
agentBus.emit('order_placed', { id: 4509, item: 'Mechanical Keyboard' });

The Takeaway

Transitioning to a multi-agent paradigm is less about writing more code and more about changing your architectural mindset from strict, top-down instruction to distributed, cooperative behavior. By giving your software components the independence to make decisions and communicate fluidly, you build highly resilient, loosely coupled applications that can gracefully handle the messy, unpredictable demands of real-world production traffic.

Originally published on my blog. You can read the alternative breakdown here.

How to Audit a Forked SaaS Codebase Before You Build on It

Wed, 16 Sep 2026 05:29:18 +0000

The easiest part of forking a production SaaS is copying the repository. The dangerous part is assuming that a successful build means the new product has a clean foundation.

I learned this while turning an existing AI generation application into a second, image-first product. Authentication still worked. Uploads still worked. Credits, asynchronous tasks, provider submission, polling, and result storage were already there.

So were dozens of decisions that belonged to the first product.

The route tree described the old user journey. Navigation emphasized the old primary workflow. Shared-looking components imported page-specific data. Emails, examples, metadata, and tests quietly assumed that the first product's identity was still correct.

A fork preserves implementation and intent at the same time. Before adding features, you need to separate them.

This article presents the audit process I would use before building on any forked SaaS codebase.

A. Write the New Product Contract First

Do not begin by deciding which folders to keep. Begin by describing what the new product must mean to a user.

Write down:

the primary task;
the first public entry point;
the secondary workflows;
the inputs each workflow accepts;
the output users expect;
the capabilities that must remain operational;
the claims the new product must not inherit.

For my image-first product, the smallest public contract was:

Entry	Primary task	Accepted starting point
Homepage	Edit an existing image	Uploaded image plus prompt
`/image`	Generate or transform an image	Text or reference image
`/video`	Generate a video	Text or image

This table did more architectural work than a folder diagram. It made the product priority explicit and gave every inherited route something to be compared against.

Without a new contract, the fork will default to the old product's choices simply because they already exist.

B. Inventory the Public Surface Before the Shared Code

Developers often start an audit in lib, services, or components. I start with the public surface because routes reveal product ownership faster.

The first pass can be simple:

rg --files src/routes
rg --files src/pages
rg --files src/components
rg --files src/data
rg --files public

Adjust the paths for the framework, but keep the purpose of the pass: identify everything that teaches a user what the application is.

For each route, record its direct dependencies:

page blocks;
route-specific data;
messages and metadata;
media assets;
structured data;
tests;
navigation links;
sitemap and discovery entries.

Treat that collection as one vertical slice. A route is rarely just one file.

This prevents a common cleanup failure: deleting the page component while leaving its datasets, translations, tests, links, and assets behind. Those leftovers keep the old product alive as dead weight and make future searches misleading.

C. Search for Identity Leaks

The next pass looks for product identity outside the obvious brand configuration.

Search for the old:

product name;
domain;
email addresses;
CDN host;
storage prefixes;
database names;
route names;
model labels;
marketing phrases;
default workflow values.

Example:

rg -n -i 'old-product|old-domain\.com|support@old-domain\.com' .
rg -n 'video-model|image-to-video|old-workflow' src tests messages public

Do not blindly replace every result. Classify it.

Match type	Question	Typical action
Infrastructure identity	Does this select an environment or resource?	Move to the correct configuration source
Public product identity	Does a user see or receive it?	Rewrite for the new product
Historical test fixture	Does it describe a still-valid contract?	Keep only if the contract remains true
Provider identifier	Is it internal and required by an adapter?	Keep server-side and prevent client leakage
Obsolete feature reference	Does the new product still own this workflow?	Remove the entire vertical slice

The goal is not a zero-result search at any cost. The goal is to make every remaining result intentional.

D. Delete in Vertical Slices

Once the inventory is clear, remove irrelevant product slices before designing new abstractions.

For one old route, delete or update all of its connected pieces together:

route
├── page block
├── route data
├── messages
├── metadata
├── assets
├── tests
├── navigation link
└── sitemap entry

Then run focused checks.

rg -n 'removed-route|RemovedPage|removedMessageKey' src tests messages public
git diff --check

Use the project's real typecheck and focused test commands after confirming they do not start a browser, development server, or production build unexpectedly.

Deleting vertical slices first has an architectural benefit: it reduces the number of real consumers. An abstraction that appeared necessary with eight inherited pages may become unnecessary when only two valid consumers remain.

If you generalize first, you risk preserving requirements that the new product does not have.

E. Follow Failures Inward

Deletion will break things. That is useful.

Each failure tells you something about ownership:

A shared component fails because it imports route data.
A global navigation test fails because it assumes an old product page exists.
A generic generation control fails because its options come from a page-specific list.
A server module fails because a client-facing model name leaked into provider logic.

Do not repair these failures by immediately restoring the removed file or adding an optional flag.

Ask instead:

Was the deleted dependency a real capability?
Which layer should own the surviving fact?
Does more than one current consumer need it?
Can the dependency direction be reversed?

For example, a shared generation workspace should consume a model capability contract. It should not import the model list from a particular landing page. The landing page can select from the shared contract, but the shared contract should not know that the page exists.

This is where compile errors become more than errors. They expose the direction of dependency.

F. Trace One Complete Request

After removing the old public surface, trace one real operation through the system.

For an AI generation product, the path might be:

User input
   ↓
Public model selection
   ↓
Effective parameters
   ↓
Credit calculation
   ↓
Authenticated API request
   ↓
Task creation
   ↓
Provider adapter
   ↓
Polling or webhook update
   ↓
Result persistence
   ↓
History and result UI

At each step, ask whether the implementation depends on a stable capability or on the old product's presentation.

Stable capabilities often include:

access checks;
uploads;
credit decisions;
task states;
provider translation;
result storage;
failure recovery.

Product-owned decisions often include:

which models are promoted;
which controls appear;
page defaults;
examples and prompts;
navigation;
SEO copy;
visual hierarchy.

The same effective parameters must drive the visible controls, credit estimate, and server request. If those three derive from different sources, the fork has inherited a drift problem even if each path works independently.

G. Separate Similar Lifecycles Without Erasing Domain Differences

The original application already had a structured video runner and provider adapters. The image path needed a similarly explicit lifecycle before more image models and tools were added.

That did not mean combining image and video into one giant controller.

A useful boundary can look like this:

interface TaskAdapter<Request, ProviderTask, Result> {
  submit(request: Request): Promise<ProviderTask>;
  getStatus(task: ProviderTask): Promise<"pending" | "complete" | "failed">;
  getResult(task: ProviderTask): Promise<Result>;
}

interface TaskRunner<Request, Result> {
  run(request: Request): Promise<Result>;
}

This pseudocode describes the direction, not a required implementation.

Image and video can share task lifecycle behavior while retaining separate input types, capability rules, controllers, and workspaces. Symmetry is valuable where it prevents operational drift. It is harmful when it hides meaningful domain differences.

I applied this distinction while building PicVane, the image-first product that emerged from the audit. Its image editor, image generator, and video generator can present different workflows while relying on consistent task, credit, upload, history, and result behavior underneath.

H. Require Evidence Before Building a Multi-Product Framework

Two related products do not automatically justify a multi-brand runtime.

Before adding brand factories, route registries, deployment matrices, or universal page schemas, look for evidence such as:

repeated changes that must be ported in both directions;
the same deployment and database;
the same release cadence;
the same operational ownership;
three or more real consumers with stable shared requirements;
measurable maintenance cost caused by separate implementations.

Without that evidence, separate repositories plus narrow, explicit capability contracts may be cheaper.

The test is not whether two codebases contain similar files. The test is whether they must make the same decision for the same reason.

I. Verify Four Different Layers

A fork audit is not complete when the typecheck passes. Record what each validation layer actually proves.

Layer	What it can prove	What it cannot prove
Source	Imports, types, route registration, static contracts	Browser behavior or provider success
Focused tests	Specific ownership and lifecycle rules	Complete end-to-end behavior
Runtime	Pages load and interactions execute	Real external service outcomes unless tested
Product evidence	Users understand and use the new direction	Architectural correctness by itself

Keep these claims separate.

A clean source audit does not prove that provider calls, payments, emails, analytics, or generated results work in production. A successful runtime test does not prove that users want the product. A clean architecture makes learning safer and faster; it does not replace the learning.

Final Audit Checklist

Before adding major features to a forked SaaS, confirm that you can answer yes to these questions:

Is the new product's primary task written down?
Has every inherited public route been accepted or rejected intentionally?
Are old brand, domain, email, storage, and workflow references classified?
Were obsolete features removed as complete vertical slices?
Do shared modules avoid importing product-owned page data?
Does one capability source drive UI options, pricing, and server requests?
Are provider identifiers and secrets kept behind server boundaries?
Are different domains allowed to keep different types and controllers?
Is every new abstraction supported by current consumers rather than hypothetical ones?
Are source, test, runtime, provider, and market claims reported separately?

If several answers are no, adding features will make the inherited assumptions harder to remove.

The best time to audit a fork is before the new product looks busy. At that stage, deletion is still cheap, boundaries are still negotiable, and a smaller route tree can reveal what the system actually knows how to do.

Reuse the operational knowledge.

Re-earn the product decisions.

Disclosure: This article is based on my own implementation experience. AI was used to help organize the structure and edit the English draft; I reviewed the technical claims and final text.

What Happens When You Build a Node.js Backend Framework From Scratch?

Mon, 31 Aug 2026 17:00:40 +0000

Building a backend framework sounds exciting.

Until you actually start building one.

You quickly discover that routing is probably one of the easiest parts.

The difficult questions come afterward:

How should dependency injection work?

How should middleware execute?

Where should validation happen?

How do you manage application lifecycle?

How should errors propagate?

How do you handle authentication?

How do you expose database access?

How do you build realtime communication?

How do you make everything work together without creating a giant, tightly coupled codebase?

These are some of the problems I've been working through while building StreetJS.

The architecture matters more than the decorators

When people see a TypeScript framework, they often notice decorators first:

@Controller('/users')
export class UserController {
  @Get('/')
  async getUsers() {
    // ...
  }
}

But the decorator isn't the interesting part.

The interesting part is what happens after the decorator is evaluated.

StreetJS has to turn application declarations into runtime behavior.

That means the framework needs systems for:

Application bootstrapping
Dependency injection
Routing
Middleware
Request lifecycle
Error handling
Validation
Configuration
Database access
Authentication
Observability

The challenge is making these systems work together without making application code depend on every internal implementation detail.

I wanted infrastructure to be predictable

A backend framework should make it obvious where something belongs.

For example:

Controller

Handles HTTP interaction.

Service

Contains application/business logic.

Repository

Handles persistence.

Middleware

Handles cross-cutting request behavior.

Module

Organizes related functionality.

This isn't a revolutionary architecture.

That's intentional.

Developers don't need an architecture that looks clever.

They need an architecture that remains understandable six months later.

Runtime validation is important

TypeScript gives us excellent developer-time safety.

But TypeScript types disappear at runtime.

An API can receive:

{
  "name": 123,
  "email": true
}

even if your TypeScript interface says:

interface CreateUser {
  name: string;
  email: string;
}

That's why StreetJS treats runtime validation as a separate concern.

The server shouldn't trust incoming data simply because the TypeScript compiler was happy with your code.

The boundary of your application needs validation.

The database shouldn't dictate your application architecture

Another area I've spent considerable time thinking about is database access.

For a real backend, you eventually need things like:

Transactions
Connection management
Migrations
Query execution
Repository patterns
Error handling
PostgreSQL support

But your controller shouldn't become a giant collection of SQL queries.

For example, I'd rather see:

const user = await userService.create(input);

than:

const result = await db.query(
  'INSERT INTO users ...'
);

inside an HTTP controller.

The framework should make the clean approach easier.

Realtime changes the architecture

REST APIs are only one part of modern applications.

Consider a:

Chat application
Delivery platform
Trading dashboard
Collaboration tool
Notification system
Multiplayer application

These applications need realtime communication.

That's why StreetJS includes WebSocket and SSE capabilities.

The goal is to make realtime communication part of the backend architecture rather than something developers bolt on at the very end.

Observability shouldn't be an afterthought

A backend that works on your laptop is easy.

A backend running in production at 2 AM is different.

You need to know:

Is the application healthy?
How long are requests taking?
Which endpoints are failing?
Are dependencies responding?
Are errors increasing?
Is the application running out of resources?

StreetJS includes health checks, metrics and telemetry capabilities because production software needs visibility.

Building a framework teaches you something unexpected

The biggest lesson I've learned so far is this:

A framework isn't primarily about features. It's about boundaries.

Every feature you add creates another architectural question.

Where does it live?

Who owns it?

What does it depend on?

What depends on it?

What happens when it fails?

Can developers replace it?

Can it be tested independently?

Can it evolve without breaking everything else?

Those questions are much harder than writing the feature itself.

StreetJS is still evolving

StreetJS is an open-source project, and I'm continuing to improve its architecture, documentation, developer experience and ecosystem.

I'm particularly interested in feedback from developers who have built serious Node.js backends.

If you were designing a backend framework today:

What would you do differently from existing frameworks?

And more importantly:

What should a modern TypeScript backend framework absolutely get right?

I'd love to hear your thoughts.

GitHub: https://github.com/hassanmubiru/StreetJS

Documentation: https://hassanmubiru.github.io/StreetJS/

npm: https://www.npmjs.com/package/streetjs

🚀 Building a Secure Bitcoin Payment Layer for MyZubster

Sat, 29 Aug 2026 09:28:12 +0000

🚀 Building a Secure Bitcoin Payment Layer for MyZubster

A new development milestone for MyZubster: we’ve completed the first architecture of a reusable Payment Intent Layer designed for marketplaces, bounties, AI agents, IoT devices, robotics, and future machine-to-machine payments.

Instead of connecting Bitcoin payments directly to individual applications, we’re building a shared economic layer.

Architecture

Applications / AI Agents
↓
Payment Intents
↓
Verification · Idempotency · Anti-Replay
↓
Payment Rails
↓
Bitcoin today — potentially Monero and machine-payment protocols later.

What we implemented

Payment Intents now have their own lifecycle, ownership, expiration, unique payment references, transaction binding, and settlement state.

Bitcoin amounts are represented as integer satoshis, avoiding floating-point monetary calculations.

Each intent is designed to receive its own Bitcoin destination through an external watch-only infrastructure boundary.

Transaction verification binds together:

txId + destination + paymentReference + amount + network

We also introduced ecosystem-level protections against transaction replay, so the same blockchain transaction cannot simply be reused to settle multiple Payment Intents.

API layer

The new authenticated API exposes:

POST /api/payments/intents

GET /api/payments/intents/:intentId

POST /api/payments/intents/:intentId/transactions

POST /api/payments/intents/:intentId/verify

Ownership is enforced server-side.

More importantly, verification does not trust critical settlement information supplied by the client. The persisted Payment Intent remains the source of truth for destination, payment reference, network, and expected amount.

Current test status

✅ 37 tests passing
✅ 4 payment test suites passing

The implementation is split into two commits:

ecc02f8 — Payment Intents + Bitcoin rail
e43f977 — Authenticated Payment Intent API

Branch:

feat/myzubster-btc-payment-layer

Security before activation

The Bitcoin rail is deliberately disabled by default.

We are not processing real Bitcoin payments yet.

There are no private spending keys inside the application, no production Electrum activation, and no real BTC transactions involved in this development milestone.

The planned Bitcoin infrastructure is watch-only, with address allocation and blockchain verification kept behind a dedicated trusted service boundary.

This separation is important because the long-term objective goes beyond adding “Bitcoin checkout” to a website.

We want MyZubster to provide a reusable economic primitive that can eventually be consumed by:

• marketplaces
• open-source bounties
• Zorgax AI agents
• autonomous services
• IoT infrastructure
• robots
• machine-to-machine economies

The next step is code review and the Pull Request, followed by further security work before connecting any real Bitcoin infrastructure.

Building the economic layer one primitive at a time.

MyZubster #Bitcoin #OpenSource #NodeJS #BackendDevelopment #AIAgents #MachinePayments #SoftwareArchitecture #FinTech #CoderLegion

From Making Features Work to Designing Systems: Anik Sikder on Architecture, SaaS, and Scale

Wed, 26 Aug 2026 00:47:30 +0000

From Making Features Work to Designing Systems: Anik Sikder on Architecture, SaaS, and Scale

From full-stack development to system design, multi-tenant SaaS, distributed systems, and AI-powered business automation.

Meet Anik Sikder

When you're starting out as a developer, success can feel very simple:

Does the feature work?

Authentication works.
The inventory page works.
The API returns the right response.

Ship it.

But as systems grow, the definition of "working" changes.

A feature that works for 10 users may struggle with 10,000. A design decision that seems harmless today can become a serious bottleneck later. And a system that is easy to build can become surprisingly difficult to maintain.

Anik Sikder has experienced that evolution firsthand.

Anik is a Full-Stack Engineer focused on SaaS, ERP and business systems, working primarily with technologies including Python, Django, FastAPI, React and Next.js. His current work focuses on multi-tenant SaaS architecture, ERP and business operating systems, RBAC, API design, system design and distributed systems. :contentReference[oaicite:3]{index=3}

He's also building BizNex OS, a multi-tenant ERP platform designed to bring areas such as inventory, procurement, sales, accounting, reporting, branches and warehouses into a connected business operating system. :contentReference[oaicite:4]{index=4}

But perhaps the most interesting part of Anik's journey is the change in how he thinks about software.

He started by asking:

"Does it work?"

Now he asks:

"How will it scale? What happens when something fails? Can another developer understand this six months from now? And what are the trade-offs?"

In this Developer Story, Anik shares how his thinking evolved from building individual features to designing entire systems, what building SaaS for real businesses taught him, why he chose a modular monolith for BizNex OS, what developers often misunderstand about distributed systems, and why architecture is ultimately about making good trade-offs.

Anik's Story

1. From Full-Stack Development to System Design

When I started, I was mostly focused on making features work.

If a user needed authentication, I built authentication.

If they needed inventory management, I built inventory management.

My success metric was simple:

Does it work?

Over time, I noticed that building features is actually the easiest part.

The hard part starts when the system grows.

A feature that works for 10 users might completely break down for 10,000 users. A design decision that feels harmless today can become a major bottleneck a year later.

That realization changed how I think.

Now, before writing code, I spend more time asking questions:

How will this scale?
What happens if this service goes down?
Can another developer understand this six months later?
What are the trade-offs?

I still enjoy coding, but architecture is where I find the most interesting problems.

Code solves today's problem. Architecture determines how many future problems you'll create.

2. Building SaaS for Real Businesses

One of the biggest lessons I've learned is that businesses don't buy software because it's technically impressive.

Developers love talking about frameworks, patterns, microservices, and scalability.

Business owners care about something much simpler:

"Will this save me time, money, or headaches?"

Early in my career, I focused heavily on technical elegance.

Later I realized that a simple solution that saves a business two hours a day is often more valuable than an architecturally beautiful system nobody uses.

Another thing I learned is that business processes are usually messy.

When you're building software for real companies, you quickly discover that every company has exceptions, special cases, and weird workflows that nobody documented.

The challenge isn't simply building software.

The challenge is understanding how people actually work.

3. Building BizNex OS

The idea behind BizNex OS came from a problem I kept seeing repeatedly.

Many small and medium businesses use five or six different tools to run their operations.

One tool for sales.

Another for inventory.

Another for accounting.

Another for HR.

And spreadsheets everywhere.

The result is fragmented data.

Nobody has a complete picture of what's happening inside the business.

I wanted to build something that treats a business as a connected system rather than a collection of disconnected modules.

Honestly, from a startup perspective, building a smaller SaaS product would probably be easier.

But I wasn't interested in solving just one workflow.

I wanted to understand how an entire organization operates and design a system around that.

It's definitely harder.

But it's also much more interesting.

4. The Challenge of Multi-Tenancy

Multi-tenancy sounds simple until you start implementing it.

The biggest concern is always tenant isolation.

A bug that exposes one company's data to another company isn't just a bug.

It's a trust-destroying event.

Because of that, I made tenant boundaries a first-class architectural concern.

Most queries are organization-scoped by default.

Permissions are organization-scoped.

Roles are organization-scoped.

Instead of relying on developers to "remember" tenant filtering, I try to make the architecture enforce it whenever possible.

Another lesson is that maintainability matters more than cleverness.

There are many technically sophisticated multi-tenant strategies, but if the team can't understand them, they'll become future problems.

Sometimes boring architecture is the best architecture.

5. Architecture Is About Trade-offs

One example is the decision between a traditional monolith and a microservices architecture.

Microservices are attractive.

They look modern.

They solve certain scaling challenges.

They allow independent deployments.

But they also introduce complexity.

Service discovery, distributed tracing, network failures, eventual consistency, and operational overhead — none of these exist in the same way in a simple monolith.

For BizNex OS, I chose a modular monolith approach.

Not because microservices are bad.

Because the complexity cost wasn't justified yet.

A lot of architecture decisions become easier when you stop asking:

"What's the most advanced solution?"

and start asking:

"What's the simplest solution that solves today's problem while keeping tomorrow's options open?"

That's how I think about architecture.

It's not about choosing the most impressive technology.

It's about choosing the right trade-off for the situation.

6. Learning from Large-Scale Systems

I don't study Facebook, Netflix, or Slack because I expect my projects to have billions of users.

I study them because they teach fundamental engineering principles.

For example:

Netflix teaches resilience.
Facebook teaches scalability.
Slack teaches real-time communication.

What I find most interesting is that large-scale companies rarely solve problems with magic.

Most of the time they're solving the same problems we face, just at a larger scale.

Understanding their trade-offs helps me think more clearly about my own designs.

Not every solution should be copied.

But every solution teaches something.

7. Distributed Systems and Event-Driven Architecture

What attracts me to distributed systems is that they force you to think differently.

In a monolith, a function call either succeeds or fails.

In distributed systems, everything becomes uncertain.

Networks fail.

Messages arrive late.

Services go down.

Data becomes temporarily inconsistent.

You start realizing that software is really about handling failure.

One thing developers often misunderstand is that distributed systems are not an upgrade.

They're a trade-off.

Many teams move toward microservices because it feels modern.

But distributed systems introduce problems that monoliths simply don't have.

If your application doesn't need that complexity, adding it early can slow you down rather than help you.

The goal shouldn't be to build the most distributed system possible.

The goal should be to build the system that makes sense for the problem.

8. From Project to Production

Building my event management and ticketing system taught me a lesson I'll probably never forget:

Making something work is easy. Keeping it working is hard.

When you're building a personal project, success means the feature works on your machine.

In production, success means:

It keeps working.
It handles unexpected input.
It survives failures.
It remains maintainable.

Authentication, payments, permissions, QR verification — none of these are difficult because of the happy path.

They're difficult because of all the edge cases.

Production engineering is really the art of handling edge cases before your users discover them.

That was one of the biggest differences I learned between simply building a project and engineering something that needs to survive in the real world.

9. AI and Business Automation

I think AI is most valuable when it removes repetitive work.

For example:

Document processing
Data extraction
Business insights
Workflow automation
Customer support assistance

These areas can generate immediate value.

Where I think the industry sometimes overuses AI is when AI is added simply because it's trendy.

Not every button needs a chatbot.

Not every workflow needs an LLM.

Sometimes a well-designed form solves the problem better than an AI assistant.

The question shouldn't be:

"Can we add AI?"

The question should be:

"Does AI actually improve the user's outcome?"

If the answer is no, don't add it.

10. Advice for Developers

The biggest shift is to stop thinking only about frameworks.

Frameworks change.

Principles don't.

Spend more time learning:

Databases
Networking
Distributed systems
Scalability
Reliability
Security
Business domains

Another thing I'd recommend is building systems, not tutorials.

Tutorials teach features.

Systems teach trade-offs.

And finally, study real-world architectures.

Whenever you use a product like Uber, Netflix, Amazon, Stripe, or Slack, ask yourself:

"How would I build this?"

That simple question has taught me more about system design than many courses.

You don't need to reproduce the exact architecture of a massive technology company.

The point is to start thinking about:

scale
failure
data
consistency
communication
security
reliability
and trade-offs

The goal isn't to become the smartest engineer in the room.

The goal is to make good engineering decisions consistently.

In my experience, that's what separates developers from architects.

From Features to Systems

Looking back at my own development journey, the biggest change wasn't learning another framework.

It was changing the questions I ask before writing the code.

Early on, I asked:

Does this feature work?

Now I ask:

What happens when the system grows?

What happens when something fails?

Can someone else maintain this?

What are the trade-offs?

Does this actually solve the business problem?

That shift is what moved my thinking from simply building features toward designing systems.

And I think that's one of the most important transitions a developer can make.

You don't necessarily become a better engineer by knowing more frameworks.

Sometimes you become a better engineer by learning to ask better questions.

About Anik Sikder

Anik Sikder is a Full-Stack Engineer focused on SaaS, ERP and business systems.

He works primarily with Python, Django, FastAPI, React and Next.js, with a particular interest in system design, multi-tenant SaaS architecture, distributed systems, RBAC, API design and business automation. :contentReference[oaicite:5]{index=5}

He is currently building BizNex OS, a multi-tenant ERP platform designed to bring business operations such as inventory, procurement, sales, accounting, reporting, branches and warehouses into a connected system. :contentReference[oaicite:6]{index=6}

Anik is also exploring distributed systems, event-driven architecture, AI-powered business automation and cloud infrastructure. :contentReference[oaicite:7]{index=7}

You can learn more about his work at aniksikder.me.

A Final Thought

Modern software development can sometimes feel like a race to learn the newest framework, architecture pattern, or AI tool.

But the deeper lesson from Anik's journey is simpler.

Good engineering isn't about choosing the most advanced technology.

It's about understanding the problem, understanding the constraints, and making the right trade-offs.

A monolith can be the right architecture.

A simple form can be better than an AI assistant.

A boring solution can be better than a clever one.

And sometimes the most valuable engineering skill isn't writing more code.

It's knowing what code should be written in the first place.

This Developer Story is part of CoderLegion's Developer Stories series, where we share the real journeys, experiences, challenges, and lessons of developers from our community.

Retries Can Make Outages Worse

Sun, 23 Aug 2026 12:35:25 +0000

Retries are one of the easiest resilience patterns to add and one of the easiest to get wrong.

The idea is sound. Transient failures happen. Networks drop packets. Load balancers close idle connections. A database primary moves. A single failed call should not always fail the whole operation.

But retries also create traffic.

During a healthy moment, that extra traffic is small. During an outage, it can be enormous. Every timeout becomes multiple calls. Every slow dependency receives new requests from callers that have already given up on older ones. The dependency is not just handling user traffic anymore. It is handling user traffic plus the system's anxiety about user traffic.

That is how retries turn a partial failure into a larger one.

Retry Storms

A retry storm starts when many clients observe the same failure at roughly the same time.

Without jitter, they retry at roughly the same time too.

If the dependency is overloaded, those synchronized retries make recovery harder. The service needs quiet time to drain queues, rebuild caches, elect leaders, or scale out. Instead it receives a second wave of traffic. Then a third.

The clients think they are being persistent. The dependency experiences persistence as pressure.

Timeouts Come First

Retries without timeouts are not retries. They are a way to accumulate stuck work.

Every retry policy needs a per-attempt timeout and an overall deadline. The per-attempt timeout prevents one slow call from consuming the entire budget. The overall deadline prevents retries from continuing after the user or caller no longer cares.

In Go, that usually means retries must be tied to context.Context, not a loop with time.Sleep.

func callWithRetry(ctx context.Context, attempts int) error {
    for i := 0; i < attempts; i++ {
        if err := ctx.Err(); err != nil {
            return err
        }

        attemptCtx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
        err := callDependency(attemptCtx)
        cancel()

        if err == nil {
            return nil
        }
        if !isRetriable(err) {
            return err
        }

        if !sleepWithJitter(ctx, i) {
            return ctx.Err()
        }
    }
    return errors.New("dependency failed after retries")
}

The exact helper functions matter less than the shape: bounded attempts, bounded time, cancellation-aware sleep, and retry only for errors that are actually retriable.

Retry Budgets

A retry budget limits how much extra traffic retries may create.

For example, a client might allow retries to add no more than 10 percent extra request volume over a rolling window. When the budget is exhausted, the client stops retrying and returns errors faster.

That sounds harsh, but it protects the dependency and the caller. If the system is already failing, unlimited retries mostly buy longer queues and slower recovery.

Retries should be treated as a shared resource.

Not Every Error Is Retriable

Retrying a deterministic failure is just load generation.

Good retry policies distinguish between:

connection resets
timeouts
rate limits
temporary 5xx responses
validation failures
authentication failures
not found responses
conflicts caused by stale state

Some of those may deserve a retry. Many do not. A 400 response should usually not be retried. A 401 should not become three 401s. A permanent validation error is not going to become valid because the client waited 100 milliseconds.

The retry policy should be closer to the protocol semantics than to the transport library.

Retries Need Backpressure

Retries and backpressure belong together.

If the dependency returns 429 Too Many Requests or a clear overload signal, callers should respect it. If there is a Retry-After header, callers should use it. If the local queue is already full, the client should stop adding retry work to it.

The purpose of a retry is to survive a transient failure, not to argue with a saturated system.

The Lesson

Retries are useful when they are humble.

They should have a deadline. They should have jitter. They should have a budget. They should understand which errors are worth retrying. They should stop when the caller no longer needs the answer. They should listen when the dependency says "slow down."

The question is not "should we retry?"

The question is:

How much extra pressure are we allowed to create while trying to recover?

How Should You Structure a Software Architecture?

Fri, 21 Aug 2026 17:30:58 +0000

Hey developers! 👋

Welcome back to the journey from business requirements to production-ready systems.

A founder walks into a development team and says:

"We need an ERP."

Sounds simple.

Until you start asking questions.

Where should users live?

How should organizations work?

Who owns inventory?

Where should orders be created?

Who is allowed to approve invoices?

What happens when thousands of users start using the dashboard?

Where should business rules live?

Should everything be one application?

Should we use microservices?

And eventually:

How do we build today's system without making tomorrow's system painful to change?

That's where software architecture begins.

Not with Kubernetes.

Not with Kafka.

Not with microservices.

Not even with Django vs FastAPI.

It begins with understanding what the business actually needs to do.

Let's build the architecture from that starting point.

The First Mistake: Starting With Technology

Imagine the product team gives you an ERP requirement.

A common engineering conversation might immediately become:

Django or FastAPI?

PostgreSQL or MongoDB?

Redis?

Kafka?

RabbitMQ?

Kubernetes?

Microservices?

These are valid questions.

But they're not the first questions.

The first question is:

What responsibilities does this business actually have?

An ERP might need:

ERP
│
├── Identity
├── Organizations
├── Products
├── Inventory
├── Sales
├── Purchasing
├── Billing
├── Reporting
└── Notifications

Notice something important.

These aren't technologies.

They're business capabilities.

That distinction changes how we design the entire system.

Step 1: Discover the Business Capabilities

Start by talking about what the system does.

For example:

The business needs to:

Manage employees
Manage organizations
Track products
Track inventory
Create sales orders
Handle purchasing
Generate invoices
Process payments
Produce reports
Send notifications

Now the system has a shape.

Instead of starting with:

controllers/
models/
views/
utils/
helpers/
services/

we can start with:

identity/
organization/
inventory/
sales/
purchasing/
billing/
reporting/

Why is this useful?

Because someone joining the project six months later can understand the product by looking at its structure.

The code starts reflecting the business.

And that's a powerful architectural principle:

Organize around meaningful responsibilities before organizing around technical abstractions.

Step 2: Find the Boundaries

Identifying capabilities isn't enough.

We also need to determine where one responsibility ends.

Consider an order.

An order depends on products.

Inventory also depends on products.

Billing depends on orders.

Reporting may depend on almost everything.

If every module directly knows about every other module, we eventually get something like:

Sales ───────► Inventory
  │              │
  ▼              ▼
Billing ◄──── Products
  │              │
  └──────► Reporting
       ▲
       │
   Identity

At first, this feels convenient.

Need something from another module?

Just import it.

But eventually:

Change Inventory
      ↓
Break Sales
      ↓
Break Reporting
      ↓
Fix Five Other Modules

The problem isn't that the system has many modules.

The problem is that the responsibilities aren't properly isolated.

A healthier model is:

Identity
   │
   └── Who is this user?

Organization
   │
   └── Which business does the user belong to?

Inventory
   │
   └── What resources do we have?

Sales
   │
   └── What are we selling?

Purchasing
   │
   └── What are we buying?

Billing
   │
   └── What do we charge?

Reporting
   │
   └── What happened?

Each area has a reason to exist.

That's what a boundary should accomplish.

A boundary is not just a folder. It defines ownership.

Step 3: Architecture Is About Ownership

Let's take a very normal operation:

A customer places an order.

From the outside:

POST /orders

Looks simple.

Internally, it could involve:

Authenticate User
      ↓
Resolve Organization
      ↓
Check Membership
      ↓
Authorize Action
      ↓
Validate Request
      ↓
Validate Products
      ↓
Check Inventory
      ↓
Calculate Price
      ↓
Apply Discount
      ↓
Create Order
      ↓
Update Inventory
      ↓
Create Financial Records
      ↓
Record Audit Event
      ↓
Trigger Notifications

A beginner might put all of this inside:

CreateOrderView

Something like:

CreateOrderView
│
├── authenticate()
├── authorize()
├── validate()
├── check_inventory()
├── calculate_price()
├── apply_discount()
├── create_order()
├── update_inventory()
├── create_invoice()
├── write_audit()
├── send_email()
└── response()

It works.

Until the application grows.

The API layer slowly becomes the place where the entire business lives.

That's when architecture starts becoming difficult to maintain.

A better separation looks like:

HTTP Request
     │
     ▼
API Layer
     │
     ▼
Application Service
     │
     ▼
Domain Rules
     │
     ▼
Persistence
     │
     ▼
Database

Now each layer has a different responsibility.

API Layer

Understands:

HTTP
Requests
Responses
Status Codes
Serialization

Application Service

Understands:

What use case are we executing?

Domain Logic

Understands:

What rules must always remain true?

Persistence

Understands:

How do we read and write data?

Database

Protects:

Data integrity
Constraints
Transactions
Indexes
Relationships

The objective isn't more layers.

The objective is clear ownership.

Follow the Request Instead of Staring at the Diagram

Architecture diagrams can become abstract very quickly.

A better way to understand a system is to follow an actual request.

Imagine the user clicks:

Create Order

The request could travel through:

Client
  │
  ▼
API
  │
  ▼
Authentication
  │
  ▼
Authorization
  │
  ▼
Validation
  │
  ▼
CreateOrderService
  │
  ├── Product Validation
  ├── Inventory Check
  ├── Price Calculation
  └── Order Creation
  │
  ▼
Database Transaction
  │
  ▼
Order Created
  │
  ▼
Domain Event
  │
  ├── Audit
  ├── Notification
  └── Analytics

Now architecture becomes easier to reason about.

Every component has a purpose.

This gives us another useful rule:

If you cannot explain why a component exists in the request journey, question whether you need that component.

Turning Boundaries Into Code

Once the business boundaries are clear, we can map them into the project.

For example:

backend/
│
├── apps/
│   │
│   ├── identity/
│   │   ├── models/
│   │   ├── services/
│   │   ├── permissions/
│   │   └── api/
│   │
│   ├── organization/
│   │   ├── models/
│   │   ├── services/
│   │   ├── permissions/
│   │   └── api/
│   │
│   ├── inventory/
│   │   ├── models/
│   │   ├── services/
│   │   ├── repositories/
│   │   └── api/
│   │
│   ├── sales/
│   │   ├── models/
│   │   ├── services/
│   │   ├── repositories/
│   │   └── api/
│   │
│   ├── purchasing/
│   ├── billing/
│   └── reporting/
│
├── shared/
│   ├── authentication/
│   ├── authorization/
│   ├── exceptions/
│   ├── logging/
│   └── utilities/
│
├── infrastructure/
│   ├── database/
│   ├── cache/
│   ├── messaging/
│   ├── storage/
│   └── email/
│
└── config/

The important part isn't the exact folder names.

It's the reasoning behind them.

We didn't say:

"Every application needs repositories."

We first asked:

"What are the responsibilities?"

Then we created a structure that reflects those responsibilities.

What Belongs Inside a Module?

Let's zoom into sales.

A possible module might look like:

sales/
│
├── models/
├── services/
├── repositories/
├── permissions/
├── events/
├── validators/
└── api/

But folders alone don't create architecture.

Each part needs a purpose.

API

The API is the system's external boundary.

It handles:

HTTP Requests
Validation Input
Serialization
HTTP Responses
Status Codes

It translates external communication into application operations.

It should not become the entire business engine.

Application Services

Services should represent meaningful operations.

For example:

CreateOrderService
ConfirmOrderService
CancelOrderService
RefundOrderService

A useful question is:

What business operation does this service represent?

If the answer is unclear, the abstraction may not be useful.

Domain Logic

This is where business rules belong.

For example:

An order cannot be cancelled after shipment.

Inventory cannot become negative.

A finalized invoice cannot be edited.

Only an authorized organization owner can transfer ownership.

These aren't HTTP concepts.

They're business rules.

They should remain as independent from the delivery mechanism as practical.

Persistence

Persistence handles data access.

For example:

find_order()
find_product()
save_order()
get_inventory()

Its responsibility is retrieving and storing information.

It shouldn't decide:

"Is this customer allowed to cancel the order?"

That's a business decision.

Don't Turn Everything Into a Service

There's another architectural trap.

Once developers discover service classes, everything becomes a service:

UserService
ProductService
OrderService
EmailService
DatabaseService
ValidationService
HelperService
ManagerService
UtilsService

Soon the codebase has hundreds of abstractions.

But abstraction isn't automatically architecture.

The objective isn't:

More classes.

It's:

Clearer responsibilities.

Before introducing a service, ask:

What meaningful business capability or use case does this component represent?

Architecture should reduce cognitive load.

Not create another maze for developers to navigate.

The Database Is Not "Just Storage"

One of the biggest architectural mistakes is treating the database as a passive storage box.

For business systems, the database is part of the architecture.

Take inventory.

A simplistic model might store:

Product
quantity = 500

But where did those 500 units come from?

Maybe:

Purchase
Sale
Return
Transfer
Adjustment
Damage

If we only store the current number, we've lost the history.

A stronger model records inventory movements:

Product
   │
   ▼
Inventory Movement
   │
   ├── Purchase
   ├── Sale
   ├── Return
   ├── Adjustment
   └── Transfer

Now the system can answer two very different questions:

How much inventory do we have?

and:

Why do we have this amount?

That second question becomes critical for:

auditing
reconciliation
reporting
debugging
financial accuracy

So database architecture is more than:

API → Database

It is closer to:

Business Rules
      ↓
Data Model
      ↓
Transactions
      ↓
Constraints
      ↓
Indexes
      ↓
Persistence

A good data model doesn't merely store information.

It helps protect the business.

Transactions: Keeping Business Operations Consistent

Consider order creation.

We may need to perform:

Create Order
Reduce Inventory
Create Financial Record

Now imagine:

Order       → Success
Inventory   → Success
Financials  → Failure

The system is inconsistent.

The order exists.

Inventory changed.

But the financial record didn't.

For operations that must succeed or fail together, we need a transactional boundary:

Transaction
│
├── Create Order
├── Update Inventory
└── Create Financial Record

Either:

Everything succeeds

or:

Everything rolls back

But there's an important architectural distinction.

Not every operation belongs inside the transaction.

Sending an email doesn't normally need to hold the database transaction open.

Generating a large PDF doesn't either.

Sending analytics data may not need to block the user.

That leads naturally to asynchronous processing.

Some Work Should Happen Later

Imagine a user creates an order.

Do they really need to wait while the system:

Send Email
Generate PDF
Update Analytics
Notify Warehouse
Sync External System

Probably not.

The user mainly needs one answer:

Was my order created successfully?

So the system can separate critical work from background work:

Create Order
     │
     ▼
Database Transaction
     │
     ▼
Order Created
     │
     ▼
Return Response

Then:

Order Created Event
        │
        ├── Send Email
        ├── Generate Invoice
        ├── Update Analytics
        ├── Notify Warehouse
        └── Sync External System

Queues and workers become useful here.

But notice the reasoning.

We didn't begin with:

"Let's install RabbitMQ."

We began with:

"Which work should not block the user's request?"

That's the architectural decision.

Synchronous vs Asynchronous Processing

A simple rule of thumb:

Keep work synchronous when:

the user needs the result immediately
the operation is relatively fast
immediate consistency matters

For example:

Create Order
Validate Payment
Update Inventory

Consider asynchronous processing when:

the operation is slow
the user doesn't need the result immediately
it can be retried
an external system is involved
the work is computationally expensive

For example:

Generate Large Report
Send Bulk Emails
Export Millions of Records
Process Large Files
Synchronize External Data

The lesson isn't:

"Always use queues."

It's:

Choose synchronous or asynchronous processing based on the characteristics of the work.

Authentication Isn't Authorization

As the ERP grows, we need to know who is making a request.

That's authentication.

But identity alone isn't enough.

We also need to know what that person can do.

That's authorization.

A typical enterprise request might look like:

User
  ↓
Organization Membership
  ↓
Role
  ↓
Permissions

For example:

A manager might have:

invoice.view
invoice.create
inventory.view
inventory.adjust

While another employee may only have:

inventory.view
order.create

So remember:

Authentication asks:

Who are you?

Authorization asks:

What are you allowed to do?

Those are different architectural concerns.

Multi-Tenancy Changes Everything

Now let's turn our ERP into a SaaS platform.

Instead of one company, we have:

Organization A
Organization B
Organization C
...
Organization 500

The security question changes.

It is no longer simply:

Can this user access this order?

It becomes:

Can this user, inside this organization, perform this action on this resource?

The request might therefore follow:

Request
   ↓
Authenticate User
   ↓
Resolve Organization
   ↓
Verify Membership
   ↓
Authorize Action
   ↓
Query Tenant-Scoped Data

The database should reinforce this boundary.

For example:

Order
│
├── id
├── organization_id
├── customer_id
└── total

Now the order belongs to a specific organization.

This is more than a feature.

Tenant isolation is an architectural security boundary.

A failure here can expose one customer's information to another customer.

That's why multi-tenancy should be considered during architectural design, not bolted on later.

APIs Are Contracts, Not Just Endpoints

The internal architecture may change many times.

Clients shouldn't need to know about those internal changes.

A web application shouldn't care whether the backend uses:

PostgreSQL
Redis
Workers
A Monolith
Microservices
A Message Broker

It should communicate through a stable contract:

Web App
     │
Mobile App
     │
Partner
     │
Internal Tool
     │
     ▼
    API
     │
     ▼
Application

That's why API design matters.

An API is a contract between:

clients
developers
internal teams
partners
future products

A strong API allows internal implementation to evolve without forcing every consumer to understand the internal architecture.

Then Reality Happens: Performance Problems

Eventually the ERP grows.

The dashboard becomes popular.

A request such as:

GET /dashboard

might perform:

20 database queries
3 aggregations
5 joins
2 external requests

Now thousands of users hit the same endpoint.

The database becomes a bottleneck.

The obvious response might be:

"Add more database servers."

But first ask:

Are we calculating the same information repeatedly?

If yes, caching might help.

Request
   │
   ▼
Cache
   │
   ├── Hit ──────► Response
   │
   └── Miss
        │
        ▼
     Database
        │
        ▼
    Store Cache
        │
        ▼
     Response

But caching creates new architectural questions:

How long should data remain cached?

When should it expire?

When should it be invalidated?

Is stale data acceptable?

What happens if the cache is unavailable?

What information is safe to cache?

So don't add a cache because:

"Scalable systems use Redis."

Add caching because:

A measured workload shows that caching solves a real problem.

Observability: When Production Starts Talking

Eventually someone will tell you:

"The API is slow."

Now what?

You need to answer:

Which endpoint?

Which request?

Which user?

Which organization?

How long did it take?

Which database query was slow?

Was it a cache miss?

Did the queue become overloaded?

Did an external service fail?

That's observability.

A useful request context might include:

request_id
user_id
organization_id
endpoint
status
duration

And the three familiar pillars are:

Logs
Metrics
Traces

Logs

Tell you what happened.

Metrics

Tell you how frequently and severely something is happening.

Traces

Show where time was spent across a request.

Without observability, production debugging becomes:

Something is slow.

Something is broken.

Maybe restart the server?

That's not an architecture strategy.

Infrastructure Comes After the Application

Only after understanding the application should infrastructure decisions become concrete.

A reasonable production starting point might look like:

                    Internet
                       │
                       ▼
                 Load Balancer
                       │
                       ▼
                   API Servers
                  /     |      \
                 /      |       \
                ▼       ▼        ▼
          PostgreSQL   Redis   Object Storage
                           │
                           ▼
                      Message Queue
                           │
                           ▼
                         Workers

Around that system:

CI/CD
Monitoring
Logging
Tracing
Backups
Secrets Management
Alerts

Now each component has a reason.

PostgreSQL       → business data
Redis            → selected fast-access workloads
Object Storage   → files
Message Queue    → asynchronous coordination
Workers          → background processing
Monitoring       → system health
CI/CD            → safe delivery

This is much healthier than creating a diagram with 25 technologies first and then trying to invent reasons for them.

Should You Start With Microservices?

Eventually, someone will probably say:

"We should use microservices."

Maybe.

But let's look at the context.

Suppose the company has:

5 Developers
1 Product
10 Customers

And we introduce:

User Service
Product Service
Order Service
Inventory Service
Billing Service
Notification Service
API Gateway
Service Mesh
Message Broker
Kubernetes

It looks sophisticated.

But now a simple feature might require:

Multiple Services
Multiple Deployments
Network Communication
Service Authentication
Distributed Debugging
Event Coordination

We've created a distributed system before we actually had a distributed problem.

The architecture is now more complicated than the business.

A simpler starting point could be:

                 Modular Monolith
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       Identity     Inventory      Sales
          │            │            │
          └────────────┼────────────┘
                       ▼
                   PostgreSQL

Everything can still deploy together.

But the boundaries remain explicit.

That's important.

Simple deployment does not have to mean chaotic architecture.

Modular Monolith vs Microservices

A modular monolith doesn't mean:

"We will never use microservices."

It means:

"We will introduce distribution when distribution solves a real problem."

The evolution can look like:

Modular Monolith
       │
       ▼
Identify Bottleneck
       │
       ▼
Find the Right Boundary
       │
       ▼
Extract One Capability
       │
       ▼
Independent Service

Suppose reporting becomes extremely expensive.

Instead of splitting everything:

Identity
Sales
Inventory
Billing
Reporting
Notifications

we might extract only reporting:

Main Application
│
├── Identity
├── Sales
├── Inventory
└── Billing

        +

Reporting Service

Now reporting can scale independently.

We didn't distribute the entire company.

We distributed the capability that actually needed it.

Architecture Should Have an Evolution Path

A system doesn't need its final architecture on day one.

It might begin as:

API
 │
 ▼
Application
 │
 ▼
PostgreSQL

Traffic increases:

API
 │
 ▼
Application
 │
 ├── PostgreSQL
 └── Redis

Background workloads grow:

API
 │
 ▼
Application
 │
 ├── PostgreSQL
 ├── Redis
 └── Workers

Asynchronous workflows become more complex:

API
 │
 ▼
Application
 │
 ├── PostgreSQL
 ├── Redis
 ├── Message Bus
 └── Workers

Eventually, perhaps one capability becomes independently scalable:

                 API Gateway
                      │
          ┌───────────┼───────────┐
          ▼           ▼           ▼
        Sales      Inventory    Billing
          │           │           │
          └───────────┼───────────┘
                      ▼
                Event Platform

The important part isn't the final diagram.

It's the journey between the diagrams.

Good architecture is an evolution path, not a perfect diagram created on day one.

Five Questions Before Adding Complexity

Before introducing another architectural component, ask five questions.

1. What problem are we solving?

Don't begin with:

"Should we use Kafka?"

Begin with:

"What problem are we experiencing?"

2. Who should own this responsibility?

Ask:

"Which module should own this behavior?"

Ownership prevents responsibility from leaking everywhere.

3. Does this need to happen immediately?

Ask:

"Does the user need the result before the request finishes?"

If not, asynchronous processing may be appropriate.

4. What happens as the system grows?

Ask:

"Could this component become a bottleneck or coupling point?"

5. What does this component cost operationally?

Every new component creates work.

A new service can mean:

Deployment
Monitoring
Networking
Authentication
Debugging
Failure Handling
Documentation
Ownership

Architecture isn't free.

Every abstraction has a maintenance cost.

Every distributed component creates another possible failure boundary.

So good architects don't only ask:

"Can we add this?"

They ask:

"Do we actually need this?"

The Architecture I'd Start With

If I were starting a SaaS or ERP platform today, I wouldn't begin with a giant distributed architecture.

I'd start closer to:

                  Client Applications
                          │
                          ▼
                       API Layer
                          │
                          ▼
                  Modular Application
                          │
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
       Identity        Inventory         Sales
          │               │               │
          ├───────────────┼───────────────┤
          │               │               │
          ▼               ▼               ▼
       Billing        Purchasing      Reporting
                          │
                          ▼
                      PostgreSQL
                          │
                    ┌─────┴─────┐
                    ▼           ▼
                  Redis       Workers
                                │
                                ▼
                           Message Queue

Alongside it:

Authentication
Authorization
Tenant Isolation
Transactions
Structured Logging
Metrics
Tracing
CI/CD
Backups

And deliberately avoid introducing:

Microservices
Service Mesh
Kubernetes
Distributed Databases
Complex Event Platforms

until the product actually gives you a reason.

These technologies aren't bad.

The principle is simpler:

Complexity should be earned by the problem.

Production Architecture Checklist

Before calling an architecture production-ready, ask:

Business

Do we understand the business capabilities?
Are responsibilities clearly separated?
Do modules represent meaningful business boundaries?

Application

Is business logic separated from HTTP concerns?
Are use cases easy to identify?
Are dependencies understandable?
Can modules evolve independently where appropriate?

Data

Does the data model reflect business requirements?
Are important invariants protected?
Are transactions used correctly?
Are indexes based on real query patterns?
Is historical information preserved where necessary?

Security

Is authentication handled consistently?
Is authorization explicit?
Is tenant isolation enforced?
Are sensitive operations auditable?
Are secrets protected?

API

Are resources predictable?
Are responses consistent?
Are breaking changes controlled?
Is documentation available?
Is the API treated as a stable contract?

Performance

Is pagination implemented?
Are expensive queries optimized?
Is caching used where justified?
Can heavy work move to background processing?

Reliability

Are failures handled?
Can background jobs be retried safely?
Are critical operations idempotent where necessary?
Are backups available?
Can the system recover from dependency failures?

Observability

Can we trace requests?
Can we measure latency?
Can we identify errors?
Can we monitor database performance?
Can we understand queue and worker health?

Operations

Can we deploy safely?
Can we roll back?
Are secrets managed securely?
Are alerts configured?
Can engineers understand what's happening in production?

If you can't answer these questions, the architecture probably isn't finished.

The Architecture Mindset

After all of this, software architecture can sound complicated.

But the underlying process is surprisingly simple.

When designing a system, keep asking:

What is this system responsible for?

Then:

Who should own that responsibility?

Then:

How should that responsibility communicate with other parts of the system?

Then:

What data does it need?

Then:

What happens when something fails?

And finally:

What happens when the business becomes larger?

Those questions naturally lead to:

Business Requirements
        ↓
Business Capabilities
        ↓
Boundaries
        ↓
Modules
        ↓
Application Logic
        ↓
Data Architecture
        ↓
API Contracts
        ↓
Transactions
        ↓
Async Processing
        ↓
Security
        ↓
Observability
        ↓
Infrastructure
        ↓
Deployment

That's architecture.

Not the number of services.

Not the number of technologies.

Not how complicated the architecture diagram looks.

Beginner vs Senior Architecture Thinking

A beginner often asks:

"Which architecture should I use?"

A developer asks:

"How should I structure this project?"

A senior engineer asks:

"Where should this responsibility live?"

A CTO asks:

"How will this architecture affect the business six months from now?"

A strong architect eventually learns to ask all four.

Because the goal isn't to build the most sophisticated system.

The goal is to build something that is:

understandable today, maintainable tomorrow, and capable of evolving when the business changes.

Final Thoughts: Architecture Is a Journey 🧠

A business idea doesn't become a production system by adding more technologies.

It becomes a production system through a series of deliberate decisions.

First:

Understand the Business

Then:

Find the Responsibilities

Then:

Create Boundaries

Then:

Define Ownership

Then:

Protect the Data

Then:

Design the Contracts

Then:

Handle Failure

Then:

Measure the System

And only then:

Introduce Complexity When Necessary

That's the real progression.

Good architecture isn't about having more components.

It's about putting the right responsibility in the right place.

And perhaps the simplest way to remember it is:

Don't start with technology.

Start with the business.

Find the responsibilities.

Create the boundaries.

Define ownership.

Protect the data.

Design the interactions.

Plan for failure.

Measure the system.

Scale what actually needs scaling.

Then introduce complexity only when the problem earns it.

That's how a simple business idea becomes a production-ready software architecture.

Key Takeaways

Start architecture with business capabilities, not technologies.
Organize systems around responsibilities and ownership.
Use boundaries to prevent uncontrolled coupling.
Keep HTTP concerns separate from core business rules.
Treat the database as an architectural component.
Use transactions to protect operations that must succeed together.
Move appropriate workloads to asynchronous processing.
Separate authentication from authorization.
Treat tenant isolation as a security boundary.
Design APIs as stable contracts.
Add caching because a workload needs it, not because it's fashionable.
Build observability before production becomes difficult to debug.
Don't introduce microservices before you have a distributed problem.
Prefer a modular monolith when it provides enough structure for the current stage.
Design architecture as an evolution path, not a final diagram.
Every architectural component has an operational and maintenance cost.
Complexity should have a reason.

Publish-Subscribe: How Modern Systems Scale Without Becoming a Monolith

Fri, 31 Jul 2026 16:57:53 +0000

When an online order is placed, the customer sees a simple message:

"Your order has been confirmed."

Behind that message, however, an entire chain of events begins.

The inventory must be updated.

The payment must be recorded.

The customer should receive an email.

Analytics dashboards need updating.

Fraud detection systems may need to run.

Loyalty points might need to be awarded.

Shipping labels may need to be generated.

In small systems, developers often connect these actions directly.

Order Service
 ├── Update Inventory
 ├── Send Email
 ├── Update Analytics
 ├── Create Invoice
 └── Start Shipping

At first, this seems perfectly reasonable.

Then the business grows.

And everything starts breaking.

The Architecture That Works Until It Doesn't

Imagine an e-commerce platform processing 50 orders per day.

A customer places an order.

The Order Service directly calls:

Inventory Service
Email Service
Analytics Service
Billing Service
Shipping Service

Everything succeeds.

Everyone is happy.

The architecture appears simple.

The problem is that simplicity is deceptive.

The Order Service now depends on every downstream service.

If any one of them is slow, the entire workflow becomes slow.

If any one of them fails, the customer experience can fail.

The more business capabilities added, the larger this dependency chain becomes.

Over time the Order Service quietly transforms into the center of the entire company.

Every new feature adds another dependency.

Every new dependency increases risk.

A Real Failure Scenario

Consider this order workflow:

Customer
    ↓
Order Service
    ↓
Inventory Service
    ↓
Email Service
    ↓
Analytics Service

One day the Email Service experiences an outage.

Now what happens?

Many systems accidentally end up with behavior like this:

create_order()

reserve_inventory()

send_confirmation_email()

update_analytics()

Email fails.

The transaction stops.

Analytics never updates.

The customer may not even receive order confirmation.

A problem in one subsystem cascades across the platform.

This is known as tight coupling.

The services are technically separate.

Operationally, they are not.

The Postmortem Nobody Wants to Write

A common production incident looks something like this:

What happened?

A third-party email provider experienced elevated latency.

Impact

Order creation response times increased from:

200ms

8-15 seconds

Business Impact

Customers abandoned checkout.

Conversion rates dropped.

Support tickets increased.

Revenue was affected.

Root Cause

The Order Service was waiting for Email Service completion before responding.

Architectural Problem

The platform treated a non-critical operation as a critical dependency.

Notice something important.

The email system wasn't actually required to create the order.

The order should have succeeded regardless.

The architecture created the outage.

Not the email provider.

Instead of directly calling every service, the Order Service publishes an event.

Order Service
      ↓
Order Created Event
      ↓
Message Broker
      ↓
Inventory
Email
Analytics
Billing
Shipping

Now the Order Service only has one responsibility:

Create Order
Publish Event
Done

It no longer cares who needs the information.

It simply announces:

Order Created

Anyone interested can react.

This changes everything.

Thinking in Events Instead of Calls

Traditional systems think:

Do this.
Then do that.
Then do another thing.

Event-driven systems think:

Something happened.
Whoever cares can respond.

That shift sounds small.

Architecturally, it is enormous.

The producer no longer knows:

Who consumes the event
How many consumers exist
When consumers process it
What consumers do with it

The producer simply publishes facts.

The Message Broker Becomes the Traffic Controller

A message broker sits between producers and consumers.

Examples include:

Apache Kafka
RabbitMQ
Amazon SNS
Amazon SQS

The broker receives events and distributes them to interested systems.

Publisher
    ↓
Broker
 ├── Inventory
 ├── Email
 ├── Billing
 ├── Analytics
 └── Fraud Detection

Services no longer communicate directly.

The broker acts as the central nervous system.

Why Business Leaders Should Care

Most architectural decisions eventually become business decisions.

Publish-Subscribe creates advantages that executives feel directly.

Faster Feature Delivery

Imagine marketing wants a loyalty points system.

In a tightly coupled architecture:

Modify Order Service
Retest Order Service
Redeploy Order Service
Risk Order Service

In Pub/Sub:

Create Loyalty Consumer
Subscribe to Order Created
Deploy Independently

The order system remains untouched.

New business capabilities can be added safely.

Reduced Revenue Risk

If Analytics fails:

Orders continue.
Payments continue.
Customers continue.

Only analytics is affected.

The failure is isolated.

This dramatically reduces blast radius.

Better Organizational Scaling

As companies grow:

Payments Team
Inventory Team
Data Team
Growth Team
Customer Team

Each team can consume events independently.

Teams become less dependent on one another.

This increases development velocity.

Why Developers Love Event-Driven Systems

Developers eventually discover a painful truth:

The hardest part of scaling is not traffic.

It's dependencies.

Pub/Sub removes many of them.

Loose Coupling

Without Pub/Sub:

Order Service
    ↓
Inventory
    ↓
Email
    ↓
Analytics

With Pub/Sub:

Order Service
      ↓
Broker
      ↓
Consumers

The producer has one dependency.

Instead of ten.

Independent Deployments

A team can deploy:

Recommendation Engine
Fraud Detection
Customer Segmentation

without touching the ordering system.

This dramatically reduces deployment risk.

Better Resilience

Suppose Billing Service crashes.

The broker stores events.

When Billing returns:

Order Created Event #1001
Order Created Event #1002
Order Created Event #1003

can still be processed.

The platform recovers automatically.

Many architecture articles stop at benefits.

Production systems do not.

Every architectural gain introduces tradeoffs.

Problem #1: Eventual Consistency

In synchronous systems:

Order Created
Inventory Updated
Response Returned

Everything happens immediately.

In Pub/Sub:

Order Created
Response Returned

Inventory Updated Later

There is now a time gap.

Systems may temporarily disagree.

This is called:

Eventual Consistency

One of the most misunderstood realities of distributed systems.

Problem #2: Duplicate Events

Networks fail.

Retries happen.

Consumers crash.

The same event may arrive twice.

Order Created #123
Order Created #123

Consumers must be idempotent.

Otherwise:

Inventory reduced twice
Customer charged twice

Production incidents happen.

Problem #3: Observability Becomes Hard

In monolithic workflows:

Request → Response

Easy to trace.

In event-driven systems:

Order Created
    ↓
Inventory Updated
    ↓
Billing Created
    ↓
Email Sent
    ↓
Analytics Updated

A single customer action can generate dozens of events.

Tracking failures becomes significantly harder.

This is why mature organizations invest heavily in:

Distributed tracing
Correlation IDs
Event monitoring
Audit logs

How Large Platforms Actually Think

The most successful architectures do not ask:

"Which service should call which?"

Instead they ask:

"What business events exist?"

Examples:

Order Created
Payment Authorized
Shipment Created
Invoice Generated
User Registered
Subscription Renewed

These events become the language of the business.

Systems are then built around reacting to those events.

The result is an architecture that grows with the company instead of fighting it.

The Real Lesson

Publish-Subscribe is not primarily about messaging.

It is about organizational scalability.

It allows systems to evolve without every team depending on every other team.

It reduces blast radius.

It improves resilience.

It accelerates feature delivery.

But it also introduces complexity, eventual consistency, and operational challenges that teams must be prepared to handle.

The companies that scale successfully are rarely the ones with the most services.

They are the ones with the clearest events.

Because once systems start speaking in events instead of direct dependencies, growth becomes far easier to sustain.

Key Takeaway

A mature architecture doesn't ask:

"Who should I call next?"

It asks:

"What happened?"

That single shift from commands to events is one of the biggest architectural transitions in modern software systems.

Server-Sent Events (SSE): When the Web Learned to Listen

Mon, 27 Jul 2026 18:26:08 +0000

For years, the web operated on a simple principle:

Browser asks.
Server answers.
Connection closes.

This model worked perfectly when websites were mostly documents.

A user opened a page.

The browser requested data.

The server responded.

Everyone moved on.

But then the internet changed.

Businesses wanted live stock prices.

Operations teams wanted real-time monitoring dashboards.

Support agents wanted instant notifications.

Users expected applications to update the moment something happened.

The old request-response model suddenly felt too slow.

The challenge wasn't that servers couldn't produce updates quickly.

The challenge was getting those updates to users immediately.

And that led engineers down a fascinating path:

Request-Response
        ↓
Polling
        ↓
Long Polling
        ↓
Server-Sent Events (SSE)

SSE wasn't just another protocol.

It was the web's attempt to become truly real-time while still staying within the familiar world of HTTP.

The Business Problem Nobody Could Ignore

Imagine you're running a stock trading platform.

At market open, prices can change thousands of times every second.

Your users expect something like this:

AAPL  $202.15
MSFT  $541.22
NVDA  $193.47

And when prices change:

AAPL  $202.31
MSFT  $541.40
NVDA  $193.61

The update should appear instantly.

Not five seconds later.

Not after a page refresh.

Not after clicking a button.

Immediately.

From a business perspective, this sounds simple:

"Show customers the latest information as soon as it changes."

From a system design perspective, it's much harder.

How does the browser know when something changes?

The First Attempt: Polling

The most obvious solution was polling.

The browser repeatedly asked:

Any updates?

Every few seconds.

Browser
    │
    ├── Request
    ├── Request
    ├── Request
    ├── Request
    └── Request

Most of the time the server replied:

Nothing changed.

Imagine 100,000 users checking every second.

Even when no new information existed:

100,000 Requests
          ↓
100,000 Responses
          ↓
No Useful Data

The system was working hard to accomplish almost nothing.

The architecture scaled poorly.

Infrastructure costs increased.

Servers processed millions of unnecessary requests.

Engineers needed something better.

The Second Attempt: Long Polling

Long Polling improved efficiency.

Instead of responding immediately, the server waited.

Browser Request
       ↓
Server Waits
       ↓
Event Happens
       ↓
Server Responds

If a stock price changed, the server responded instantly.

If nothing happened, the connection stayed open.

This dramatically reduced useless requests.

But Long Polling still had a problem.

After every response:

Response Received
        ↓
Create New Request
        ↓
Wait Again

The cycle never truly disappeared.

The browser was still repeatedly initiating communication.

Just less frequently.

The Simple Question That Changed Everything

Eventually engineers asked:

Why is the browser constantly asking for updates?

If the server already knows when something changes, why not let the server send the update directly?

That question led to Server-Sent Events.

What Is Server-Sent Events (SSE)?

Server-Sent Events allow a browser to open a single HTTP connection and keep it alive.

Instead of repeatedly asking for information, the browser simply listens.

Browser
     │
     │ Open Connection
     ▼
Server
     │
     ├── Event
     ├── Event
     ├── Event
     └── Event

The server pushes updates whenever something happens.

No polling loop.

No repeated requests.

No constant reconnecting.

The browser becomes a subscriber.

The server becomes a publisher.

A Different Way of Thinking

Polling asks:

Did anything happen?
Did anything happen?
Did anything happen?
Did anything happen?

SSE says:

Tell me when something happens.

That sounds like a small difference.

Architecturally, it's huge.

Inside a Real SSE Connection

From the browser's perspective, creating an SSE connection is surprisingly simple.

const events = new EventSource("/events");

The browser sends a normal HTTP request.

The server responds with a special content type:

text/event-stream

But unlike a normal HTTP response, the server doesn't close the connection.

Instead, it keeps streaming data.

data: New Order Created

data: Payment Received

data: Shipment Dispatched

The browser receives each event immediately.

The connection remains open.

The stream continues.

Real System Example: E-Commerce Operations Dashboard

Consider a large e-commerce company.

The operations team monitors incoming orders in real time.

Without SSE:

Dashboard
    ↓
Poll Every 5 Seconds
    ↓
Check For New Orders

A customer places an order.

The operations team might not see it for several seconds.

With SSE:

Customer Places Order
          ↓
Order Service
          ↓
Event Bus
          ↓
Notification Service
          ↓
SSE Gateway
          ↓
Dashboard

The moment the order is created:

New Order Received

appears on the dashboard.

No refresh required.

No polling required.

To the user, the system feels alive.

Real System Example: Monitoring Platforms

Modern monitoring tools often display:

CPU utilization
Memory usage
Error rates
Request latency
Active users

Imagine a platform monitoring thousands of servers.

Every metric changes continuously.

Polling every few seconds creates unnecessary traffic.

Instead:

Monitoring Agent
  ↓
Metrics Service
  ↓
Event Stream
  ↓
SSE
  ↓
Dashboard

As metrics change, updates appear instantly.

Operations teams gain near real-time visibility into system health.

Real System Example: AI Response Streaming

Today, one of the most recognizable uses of SSE is AI.

When you ask ChatGPT a question, the model doesn't generate an entire response instantly.

It generates tokens one at a time.

Without streaming:

User Question
       ↓
Wait 10 Seconds
       ↓
Entire Answer Appears

From the user's perspective:

System feels slow.

With SSE:

User Question
       ↓
Wait 1 Second
       ↓
Words Begin Appearing
       ↓
Response Continues Streaming

The total generation time may still be ten seconds.

But the experience feels dramatically faster.

This reveals an important system design principle:

Users experience latency differently than systems measure latency.

A ten-second wait feels frustrating.

A ten-second stream feels interactive.

SSE helps bridge that gap.

This is one reason AI products feel conversational rather than delayed.

Why Product Teams Love SSE

From a product perspective, SSE improves perceived responsiveness.

Users don't care how elegant your architecture is.

They care about what they see.

Compare:

Click
     ↓
Wait
     ↓
Wait
     ↓
Wait
     ↓
Response

Versus:

Click
     ↓
Response Starts
     ↓
More Content
     ↓
More Content

The second experience feels dramatically faster even if total processing time is identical.

That translates directly into:

Better engagement
Better user satisfaction
Higher retention
More trust in the product

Why Developers Love SSE

One reason SSE became popular is that it works with existing HTTP infrastructure.

There is no entirely new protocol to learn.

Browser
   ↓
HTTP
   ↓
Load Balancer
   ↓
Reverse Proxy
   ↓
Application Server

Everything already understands HTTP.

This makes adoption much easier.

Developers also benefit from automatic reconnection.

If a connection drops:

Connection Lost
        ↓
Browser Detects Failure
        ↓
Automatic Reconnect

The browser handles much of the recovery logic automatically.

Less code.

Less complexity.

Fewer edge cases.

Why Architects Choose SSE

A common question is:

Why not just use WebSockets?

The answer depends on communication patterns.

Consider stock prices:

Market
   ↓
User

Or monitoring dashboards:

Server
   ↓
Dashboard

Or AI response streaming:

Model
   ↓
Browser

In all of these systems:

Server talks constantly.
Client rarely talks.

SSE fits naturally.

WebSockets support two-way communication:

Client ↔ Server

But many systems don't need that capability.

Using WebSockets in these situations can introduce complexity without delivering meaningful benefits.

One of the most important lessons in architecture is:

The best solution is not the most powerful one. It's the simplest one that satisfies the requirements.

For many one-way streaming workloads, SSE is exactly that solution.

The Hidden Challenge

Every architectural improvement introduces a new bottleneck.

Polling creates too many requests.

SSE solves that problem.

But it introduces another.

Persistent connections.

Imagine:

500,000 Active Users

Polling might create:

Millions of Requests Per Minute

SSE creates:

500,000 Open Connections

The request problem improves.

The connection management problem appears.

Infrastructure teams now worry about:

Memory consumption
File descriptor limits
Load balancer behavior
Proxy timeouts
Reconnection storms
Horizontal scaling

From an operations perspective:

Fewer Requests
≠
Less Work

The workload simply shifts.

Where SSE Starts To Break Down

SSE is excellent for one-way communication.

But some systems require continuous communication in both directions.

Examples include:

Chat applications
Multiplayer games
Collaborative editors
Trading platforms
Video conferencing systems

These applications need:

Client ↔ Server

communication.

SSE only provides:

Server → Client

At that point, WebSockets usually become the better architectural choice.

What SSE Really Changed

Server-Sent Events didn't introduce a revolutionary new protocol.

Its impact came from a much simpler idea:

Stop asking repeatedly.

Start listening continuously.

That shift helped transform the web from a collection of documents into a platform for real-time experiences.

Stock prices could update instantly.

Dashboards could refresh automatically.

Notifications could arrive the moment events occurred.

AI systems could stream responses as they were generated.

The technology itself is relatively simple.

The effect on user experience is enormous.

And that is often the hallmark of great system design:

A small architectural change that fundamentally improves how users experience a system.

The Evolution Continues

The web's journey toward real-time communication didn't stop with SSE.

Request-Response
   ↓
Polling
   ↓
Long Polling
   ↓
Server-Sent Events (SSE)
   ↓
WebSockets

SSE taught the web how to stream information.

WebSockets would teach the web how to have a conversation.

And that's where the next chapter begins.

Why AI Can't Replace Software Architects (Yet)

Mon, 27 Jul 2026 07:22:05 +0000

AI can generate code in seconds.

It can write APIs.

Build React components.

Fix bugs.

Even create an entire application from a single prompt.

So a question naturally comes up:

👉 If AI can write code, why do companies still need Software Architects?

The answer is simple.

Because writing code and designing software are two completely different skills.

Let's see why.

💡 AI Can Generate Code...

Ask AI to build:

A login API
A dashboard
A CRUD application
A payment integration

It will probably do a pretty good job.

That's because these are implementation tasks.

AI has seen millions of similar examples during training.

🏗️ But Architecture Starts Before Coding

Before writing even a single line of code, someone has to answer questions like:

Should we use a monolith or microservices?
SQL or NoSQL?
REST or GraphQL?
Should we cache this data?
How do we handle 10 million users?
What happens if one service fails?

These decisions shape the entire system.

And they don't have one correct answer.

⚖️ Architecture Is About Trade-offs

Every architectural decision comes with compromises.

For example:

Choosing microservices gives better scalability...

But also adds:

Network latency
Deployment complexity
Service communication
Monitoring challenges

Choosing a monolith makes development simpler...

But scaling large teams can become harder.

There's no universal "best" choice.

It depends on the business, the team, and the product.

🤖 AI Doesn't Know Your Business

AI can read code.

But it doesn't automatically know:

Your business goals
Budget constraints
Team experience
Customer expectations
Future roadmap

Without that context, it can only make educated guesses.

Architecture requires understanding the bigger picture.

🚨 The Cost of a Wrong Architecture

A bug can usually be fixed in hours.

A poor architectural decision can affect a product for years.

It can lead to:

Performance issues
Higher cloud costs
Difficult deployments
Slow development
Expensive rewrites

That's why architecture deserves careful thought.

👥 Great Architects Don't Just Design Systems

They also:

Talk to stakeholders
Understand business requirements
Balance technical and business needs
Plan for future growth
Help teams make consistent decisions

Much of their work happens outside the code editor.

🤝 AI Is a Great Assistant

AI can absolutely help architects by:

Comparing technologies
Reviewing designs
Generating diagrams
Explaining patterns
Suggesting alternatives

But the final decision still needs human judgment.

🎯 The Future Isn't AI vs Architects

It's AI + Architects.

Architects who know how to use AI will:

Explore ideas faster
Prototype quicker
Validate designs
Save time on repetitive work

AI becomes a powerful tool, not a replacement.

💡 Final Thought

AI is getting better at writing code every day.

But architecture isn't just about code.

It's about making decisions that balance technology, business, people, cost, scalability, and the future.

And that's something no model can fully understand without human experience and context.

At least... not yet.

💬 What do you think? Will AI ever replace Software Architects, or will it simply make them more productive?

AI #SoftwareArchitecture #SoftwareEngineering #ArtificialIntelligence #SystemDesign #Developers #Programming #Tech #LLM #Architecture

Your Webhook Doesn't Always Belong in Your API

Sun, 12 Jul 2026 14:36:53 +0000

Your Webhook Doesn't Always Belong in Your API

Sometimes the best place for a webhook isn't your application—it's your infrastructure.

When most developers think about webhooks, they think about an HTTP endpoint.

A provider sends a request.

Your API receives it.

Business logic runs.

Response sent.

Done.

For small applications, that's usually enough.

As systems grow, though, that architecture starts to show its limits.

Every Webhook Doesn't Need Immediate Processing

Imagine receiving thousands of webhook events every minute.

Some need to update your database.

Some should trigger background jobs.

Others need to be forwarded to another service.

A few should be streamed into Kafka for analytics.

Some belong in a message queue.

Trying to handle every one of those inside a single HTTP request quickly becomes difficult.

Your webhook endpoint slowly turns into the busiest part of your application.

Webhooks Are Just Events

One idea that changed the way I think about webhook systems is this:

A webhook isn't an API request.

It's an event.

Once you start treating webhooks as events instead of HTTP requests, the architecture becomes much more flexible.

Instead of asking,

"Which controller should process this?"

you start asking,

"Where should this event go?"

Sometimes the answer is your application.

Sometimes it isn't.

Different Events Need Different Destinations

Not every webhook has the same job.

Some events should be pushed into Redis for fast processing.

Others belong in RabbitMQ or Amazon SQS so workers can process them asynchronously.

High-volume analytics events might be streamed into Kafka.

Some integrations simply need to forward the webhook to another HTTP endpoint.

Internal microservices may communicate over gRPC.

Each destination solves a different problem.

The important part is having the flexibility to choose the right one.

That's Why I Built Destinations into Hooktrace

One feature I've been working on is Destinations.

Instead of treating Hooktrace as the final stop for webhook deliveries, it can route events to the infrastructure that's best suited for processing them.

Today Hooktrace supports destinations including:

HTTP
Redis
RabbitMQ
Amazon SQS
Kafka
gRPC

More destinations will follow as the platform evolves.

The goal isn't to replace your existing infrastructure.

It's to make webhook routing configurable instead of hardcoded.

Routing Shouldn't Require New Code

One thing I wanted to avoid was forcing developers to change application code every time a routing decision changed.

Maybe today's events belong in Redis.

Tomorrow they belong in Kafka.

Next month they need to be forwarded to another internal service.

Those shouldn't require redeploying your application.

They should be configuration changes.

That's the direction I'm taking with Hooktrace.

Looking Ahead

Modern applications are increasingly event-driven.

Webhook platforms shouldn't stop at receiving events.

They should help route them to the right place.

That's exactly what I'm trying to build with Hooktrace.

A webhook is only the beginning of the journey.

Where it goes next should be your choice.

Follow Along

I'm documenting the engineering decisions behind Hooktrace as I build it in public.

If you've built event-driven systems or have opinions on webhook routing, I'd love to hear how you're handling it.

🌐 Website: https://hooktrace.xyz

⭐ GitHub: https://github.com/hooktracehq/hooktrace

Thanks for reading.

Why HIPAA-Compliant AI Agents Cost More Than the Model Integration

Thu, 18 Jun 2026 14:25:43 +0000

A lot of healthcare AI conversations still start with the wrong question:
How much does it cost to connect an LLM?

That question is too small.

In healthcare, the model is rarely the hardest or most expensive part. The real work begins when the AI agent has to operate around protected health information, user permissions, audit trails, EHR data, compliance controls, and production-grade infrastructure.

A basic AI assistant can often be prototyped quickly. But a HIPAA-compliant AI agent is not just a chatbot with a better prompt. It is a full software system that needs to be designed around security, accountability, and operational risk.

For example, if an AI agent is used for patient intake, appointment scheduling, clinical documentation, revenue cycle support, or EHR-connected workflows, the architecture needs much more than a chat interface.

It may require:

Role-based access control
Encryption at rest and in transit
Audit logging
PHI boundaries
Secure API integrations
Business Associate Agreement planning
Monitoring and alerting
Human review workflows
Data retention policies
Incident response planning
Compliance documentation
EHR or FHIR integration

This is where many early healthcare AI budgets become unrealistic.

A founder may estimate the cost of the AI interface and model usage, but miss the hidden engineering work around the system. Once the product moves from demo to production, the cost shifts toward backend architecture, integration logic, security reviews, logging, cloud infrastructure, monitoring, and compliance controls.

That is why the build-vs-buy decision in healthcare AI should not only be based on year-one cost.

A SaaS tool may be faster to start with, especially for standardized workflows. But as usage grows, recurring costs can increase through per-seat licensing, patient-volume pricing, token markups, and limited customization. A custom platform may cost more upfront, but it can offer stronger control over data flows, workflows, integrations, permissions, and long-term scaling.

The best decision depends on the use case.

A simple scheduling assistant may not need the same budget as a clinical knowledge assistant or an AI medical scribe. A standalone chatbot may be simpler than an agent connected to EHR systems. A healthcare MVP may need narrow scope first, while an enterprise AI platform may require deeper governance, monitoring, and reporting from day one.

For developers and technical teams, the main takeaway is this:
Healthcare AI cost is not only an AI cost.

It is an architecture cost.
It is an integration cost.
It is a security cost.
It is a compliance-readiness cost.
It is an ongoing operations cost.

Before building, teams should define:

What data the AI agent can access
Which users can access which workflows
Whether PHI enters the AI pipeline
Where audit logs are stored
How outputs are reviewed
What systems need integration
How model behavior will be monitored
What happens when the AI gives an incorrect or incomplete answer
Which vendors touch sensitive data
What the 3-year operating cost looks like

This planning step can prevent expensive rework later.

We recently published a full cost breakdown covering HIPAA-compliant AI agents, healthcare AI architecture, RAG systems, vector databases, EHR/FHIR integration, compliance controls, hidden costs, and 3-year build-vs-buy comparisons.

For anyone building healthcare AI products, the safest approach is to treat compliance, monitoring, and data governance as part of the core architecture from day one — not as cleanup work after launch.

Managing Database Schema Like Code: The Quest for Database Version Control

Thu, 18 Jun 2026 06:08:16 +0000

In modern software development, we take git for granted. We branch, we commit, we merge, and we roll back code seamlessly. But when it comes to the database layer, things get tricky. Databases have state—you can't just overwrite a production database with a new version without risking catastrophic data loss.

So, is there a version control system for database structure changes? Yes, and it is a fundamental pillar of modern DevOps.

How to Build a Database Version Control System

To create a reliable system for tracking database changes, you need to transition from manual, ad-hoc SQL execution to an automated, migration-based workflow.

Baseline the existing database: You begin by taking a snapshot of your current database structure to serve as the ground-zero schema version.

Write incremental migration scripts:
Every change—whether creating a table, adding a column, or modifying a constraint—is written as a separate, sequential SQL or code file.

Maintain a schema version table:
The database itself must track its own version using a dedicated metadata table that records which migration scripts have already run.

Automate via a migration engine:
Tools like Liquibase, Flyway, or framework-native tools (like Prisma, Alembic, or EF Core) are integrated into the CI/CD pipeline to automatically detect and apply missing migrations.

The Roadblocks:
Common Pitfalls in Database Version Control
Transitioning databases to a version-controlled model introduces several friction points that do not exist with standard application code.

The state and data loss dilemma:
Unlike code files that can be easily replaced, database changes happen in-place and can easily destroy existing data if a destructive command (like dropping a column) is executed.

Merge conflicts in team environments:
When multiple developers work on separate features simultaneously, they might generate conflicting migration sequences, leading to deployment failures.

The challenge of rollbacks:
Reverting a code deployment is simple, but rolling back a database change that has already modified live data requires complex "down" scripts that may still result in data loss.

Production drift:
If a developer or administrator manually alters the database schema directly in production, the version control system loses sync, causing subsequent automated deployments to fail.

The Resolution:
Best Practices for Seamless Database Migrations
Overcoming these challenges requires strict engineering discipline and the right architectural approach.

Enforce a strict migration-only policy:
Block direct schema modifications on all environments, ensuring that every single database change must go through code review and the CI/CD pipeline.

Utilize semantic or timestamped sequencing:
Instead of using sequential integers (like 001, 002) which easily conflict, use UTC timestamps (e.g., 20260618120400_add_users_table) to allow concurrent migrations to coexist.

Adopt the expand and contract pattern:
For zero-downtime deployments, split destructive changes into phases. First, add the new structure (expand), migrate the data safely, and only remove the old structure (contract) in a later release.

Incorporate automated schema validation:
Integrate linters and dry-run tests into your pull request workflow to catch syntax errors and unsafe operations before they ever touch a live database.

Sumita
Web Developer

Reflection architectural pattern

Mon, 15 Jun 2026 08:55:35 +0000

I published an article diving into the reflection architectural pattern. If you've ever wondered how Spring Boot uses annotations to magically wire your dependencies, or how ORMs map database fields without explicit code, reflection is the answer. I break down how this pattern actually works, show practical examples, and discuss when you should and shouldn't use it.

Read the full article

Best Technologies for Building Scalable Web Portals in 2026

Mon, 25 May 2026 09:11:31 +0000

Quick Overview

Modern portals require the right frontend frameworks, backend
architectures, database strategies, and cloud-native infrastructure.
Technologies like Next.js, Node.js, GraphQL, Kubernetes, and
PostgreSQL dominate today.
Microservices and headless architecture have replaced monolithic
systems.
Choosing the right stack early prevents costly re-architecture as
traffic and features grow.

You've built a portal. It works fine at 500 users. Then 50,000 show up, and everything starts cracking: slow load times, failed API calls, a database choking under query pressure. Sound familiar? This is the exact challenge development teams face when they don't architect for scale from day one. Building a web portal that can handle real-world growth isn't just about picking popular tools; it's about making deliberate, technically sound decisions at every layer of the stack. This article breaks down exactly which technologies are earning their place in scalable web portal projects in 2026, why they work, and how they fit together.

What Defines a Web Portal as "Scalable'?

Before diving into the tech, it's worth grounding the conversation. A scalable web portal isn't just one that "handles more users." True scalability means the system can expand horizontally (more instances) or vertically (more resources) without a complete overhaul. It means database queries don't bottleneck at 10x traffic. It means your frontend delivers fast experiences regardless of location. And it means your team can deploy new features without fragile dependencies across the codebase.
The best web portal development technologies are those that ensure scalability across frontend, backend, data, and infrastructure, not just one. Teams aiming to improve web development efficiency find it cheaper to align stack choices with scalability goals from the start than to retrofit a struggling architecture later.

Frontend Technology

The frontend layer of a portal is where users form their first and often lasting impression. In 2026, Next.js has firmly established itself as the leading React framework for portal development. Its hybrid rendering model, combining Static Site Generation (SSG), Server-Side Rendering (SSR), and Incremental Static Regeneration (ISR) lets teams choose the right rendering strategy per page rather than being locked into a single approach.
SSR provides fresh data on each request for personalized dashboards, while SSG offers quick load times without server costs for public pages. This flexibility makes Next.js superior for large-scale portal use cases.
Paired with TypeScript for safety and Tailwind CSS for rapid UI, a Next.js frontend provides a scalable foundation from MVP to enterprise without needing a framework change during the product lifecycle.

Backend Technology

The backend is where scalability decisions carry the highest stakes and where working with an experienced web portal development company makes the most measurable difference.

Microservices: If done poorly with poorly defined service boundaries,
no API gateway, or fragile inter-service communication- create more
complexity than they solve. Expertise in designing those boundaries,
implementing circuit breakers, and configuring a service mesh like
Istio separates a system that survives production traffic from one
that collapses under it.
Node.js: For most portal projects in 2026, a microservices architecture built on Node.js has become the industry standard.Node's non-blocking, event-driven I/O model efficiently handles thousands of concurrent connections, a critical trait when portal serve large numbers of authenticated users simultaneously across multiple modules and workflows.
GraphQL: GraphQL has largely replaced REST as the preferred API layer
in portal development because it enables the frontend to request
exactly the data needed in a single trip. This fixes over- and
under-fetching issues common in REST systems with complex, nested
data like users, roles, modules, and content.

Database Layer

No web portal scales effectively without a deliberate database strategy. PostgreSQL remains one of the most reliable choices for relational data in portal applications. Its support for JSON columns, advanced indexing, full-text search, and connection pooling through PgBouncer enables developers to manage both structured and semi-structured data efficiently, helping organizations boost efficiency for web development without introducing unnecessary database complexity too early.

As traffic grows, caching becomes essential to prevent the database from becoming a performance bottleneck. Redis addresses this challenge by storing session data, frequently accessed query results, and rate-limiting counters in memory. This significantly reduces database load, improves scalability, and helps boost efficiency for web development by lowering API response times from hundreds of milliseconds to just a few milliseconds. For portals with real-time capabilities such as notifications, live dashboards, and collaborative workspaces, Redis Pub/Sub also provides a lightweight and effective messaging layer between services.

For applications that manage unstructured or rapidly evolving content, including multi-tenant SaaS portals and content aggregation platforms, MongoDB can complement PostgreSQL effectively. Its flexible, document-based architecture allows teams to handle dynamic data models with minimal schema constraints, enabling faster iterations and further boosting efficiency in web development when business requirements change frequently.

Infrastructure

Even the best application code will fail at scale without infrastructure designed to match.

Docker containerization has become table stakes. Packaging each
service with its dependencies eliminates environment inconsistencies
and makes deployments reproducible across development, staging, and
production. But containers alone don't orchestrate themselves.
Kubernetes (K8S) is the orchestration layer that makes portal
infrastructure scalable. Its auto-scaling capabilities, including
Horizontal Pod Autoscaler (HPA) for adding more service instances and
Vertical Pod Autoscaler (VPA) for resource tuning, allow the portal
to automatically respond to traffic spikes. With rolling deployments
and detailed health checks, Kubernetes enables zero-downtime
releases, essential for portals with critical workflows.
Cloud-native deployment on AWS (EKS), Google Cloud (GKE), or Azure
(AKS) provides managed Kubernetes services that reduce the
operational burden on development teams, allowing them to focus on
application logic rather than cluster management. Layering a CDN like
Cloudflare on top ensures that static assets and cached responses are
served from edge locations closest to each user, shaving latency
globally regardless of where your user base is located.

Security and Authentication

Scalability without security increases the attack surface. By 2026, portals will use established identity providers such as Auth0 or AWS Cognito, using OAuth 2.0 and OpenID Connect instead of custom authentication. RBAC and ABAC guarantee granular, audit-able permissions across modules. Zero Trust, verifying every request regardless of network origin, is expected standard, requiring internal services to authenticate independently, not just public requests.

Conclusion

Building a scalable web portal in 2026 is a systems thinking problem. Scalability isn't solved by one technology but requires decisions at every layer: Next.js for frontend, Node.js with GraphQL for backend, PostgreSQL and Redis for data, and Kubernetes on cloud infrastructure for scalable deployment. The best technologies for web portal development are those that work coherently as a system, not just as individually impressive tools in isolation. Get the architecture right early, and scaling becomes a routine operational task rather than a firefighting crisis.

Frequently Asked Questions

1. What is the best technology stack for building a web portal in 2026?

Next.js for the frontend, Node.js with GraphQL for the API, PostgreSQL and Redis for data storage, and Kubernetes on AWS or GCP for infrastructure. The ideal stack depends on your portal's use case, team skills, and traffic.

2. What is the difference between a web portal and a regular website?

A website delivers content to anonymous visitors. A web portal goes further—authenticating users, managing roles, and providing personalized, dynamic data across modules like dashboards, self-service tools, and multi-user workflows. Unlike static pages.

3. How do microservices improve web portal scalability?

They divide the backend into independent services for functions like auth, billing, or notifications. Each scales, deploys, and updates separately, unlike monolithic systems.

4. How much does it cost to build a scalable web portal?

A mid-complexity portal costs $30,000 to $ 150,000 or more to develop. Cloud infrastructure costs a few hundred to a few thousand dollars per month, depending on traffic and storage.

5. Is Next.js suitable for large-scale web portals?

Yes. Its hybrid rendering, edge support, image optimization, and seamless backend and CMS integration make it a proven choice for portals of any size.

Monolithic Thinking Is the Biggest Bug in Your Microservice Testing

Tue, 19 May 2026 18:19:07 +0000

Monolithic Thinking Is the Biggest Bug in Your Microservice Testing

If you started your QA career testing a monolithic application, you already have a foundation. But that foundation can quietly work against you in a microservice system — because the rules have changed, and most junior QA engineers don't realize it until something breaks in a way they can't explain.

This article will help you understand how microservice systems are built, why they behave differently, and what you need to start thinking about as a QA engineer working in one.

What Even Is a Microservice Architecture?

In a traditional monolithic application, everything lives in one place. The user interface, the business logic, the database — all bundled together. When you test a login feature, you're testing one system.

Microservice architecture breaks that single system into smaller, independently running services. Each service:

Handles one specific business capability (e.g., payments, user accounts, notifications)
Manages its own database
Communicates with other services through APIs or events

Here's what a simplified e-commerce system might look like:

API Gateway
              │ 
  ┌───────────┼───────────┐
  ▼           ▼           ▼
User       Order       Payment
Service    Service     Service
  │           │           │
  ▼           ▼           ▼
User DB    Order DB   Payment DB

Each of those services is developed, deployed, and scaled by a separate team — independently of the others.

Flexible? Yes.
Complex to test? Absolutely.

Why Your Old Testing Approach Breaks Here

In a monolith, a login flow looks like this:

User enters credentials → Business logic validates → Database checks → Response returned

That's one system. One place to look when something breaks.

In a microservice system, the exact same login might look like this:

Request hits API Gateway
       ↓
  Auth Service (validates credentials)
       ↓
  User Service (fetches user profile)
       ↓
  Session Service (creates session token)
       ↓
  Response returned

Four services involved in a single login.

If anything fails — a timeout in the Auth Service, a slow response from the Session Service, a data mismatch in the User Service — the whole flow breaks. And the error message the user sees might give you zero clue about which service actually caused it.

This is why junior QA engineers often feel lost in microservice environments. They're testing the output without understanding the system producing it.

The Mindset Shift: From Feature Tester to System Observer

In a monolith, the right question is:

“Does this feature work correctly?”

In microservices, the right question is:

“Which services are involved, and what happens when one of them fails?”

This is the core shift.

You're no longer just validating a response — you're observing a system.

When you pick up a test scenario, train yourself to ask:

Which services participate in this flow?
What data does each service depend on?
What happens if one service responds slowly or not at all?
Is the data consistent across services after the transaction completes?

You don't need to know how to build microservices to ask these questions. You just need to understand the architecture well enough to think through the flow.

Step 1: Get the Architecture Diagram (And Actually Read It)

This is the most underrated habit in QA.

Before you write a single test case for a feature, ask your team for:

A service interaction diagram (which services talk to which)
An event flow diagram (what triggers what)
A deployment overview (how services are grouped)

These diagrams tell you things that no test case document will:

Where does the data originate?
Which service owns a piece of data?
How do services notify each other when something happens?

If you're testing a payment flow but you don't know that the Payment Service sends an event to the Notification Service after a transaction, you'll miss testing whether the user gets their confirmation email — because that's not the Payment Service's job.

Without the diagram, you're testing in the dark.

Step 2: Understand How Services Talk to Each Other

Microservices don't communicate the same way in all situations. There are two main patterns, and you need to test them differently.

Synchronous Communication (HTTP / gRPC)

One service directly calls another and waits for a response.

Service A ──── REST API call ───→ Service B
               (waits for response)

What to test:

Does the API return the correct data when everything works?
What happens when Service B is slow? Does Service A handle the timeout?
What error does the user see if Service B is down?
Are the API contracts (request/response formats) being respected?

Asynchronous Communication (Events / Message Queues)

One service publishes an event. Another service picks it up later — independently.

Service A → publishes event to Kafka → Service B consumes it → Service C is triggered

This is trickier to test because there's no instant response. The flow happens over time.

What to test:

Is the event published in the correct format?
Does Service B actually consume the event and process it correctly?
What happens if the event is published but Service B is temporarily down — does the message get processed later?
Are there duplicate events being sent? How does the system handle that?

As a junior QA engineer, asynchronous flows are where most real bugs hide. If you're not thinking about them, you're missing a big part of your system.

Step 3: Understand Who Owns the Data

In a monolith, there's usually one database. You can query it directly to verify that data was saved correctly.

In microservices, every service has its own database:

Order Service   → Order Database
Payment Service → Payment Database
User Service    → User Database

This creates a testing challenge called eventual consistency.

When an order is placed, the Order Service records it immediately — but the Payment Service might take a few milliseconds (or longer) to update its records. During that window, the data across the two databases is temporarily inconsistent.

What junior QA engineers often miss:

Checking the Order Database but forgetting to verify the Payment Database
Not accounting for the delay in asynchronous updates
Assuming that if one service confirms success, all services have updated

A good habit: after any cross-service transaction, check the relevant database in each service that was involved. Don't just trust the API response.

Step 4: Learn to Follow a Request Through the System

When a bug happens in a microservice system, “it's broken” tells you almost nothing. You need to know where it broke.

Teams handle this with observability tools — tools that track what a request does as it moves through services:

Centralized logging (e.g., ELK Stack, Datadog): collects logs from all services in one place
Distributed tracing (e.g., Jaeger, Zipkin): shows you the full path of a request across services, with timing
Monitoring dashboards: give you a real-time view of service health

Here's what a trace might show you:

Request ID: abc-123
API Gateway        → 5ms
Auth Service       → 12ms
User Service       → 340ms  ← slow!
Session Service    → 8ms
Total: 365ms

That trace tells you the User Service is the bottleneck — not the Auth Service, not the gateway. Without tracing, you'd have no idea.

As a QA engineer, you don't need to set up these tools — but you do need to learn how to read them. Ask your team to walk you through how they trace requests. It will make you significantly more effective at diagnosing failures.

Step 5: Test What Happens When Things Go Wrong

This is where microservice testing gets interesting — and where most junior QA engineers don't spend enough time.

In a monolith, you mostly test happy paths and a few negative scenarios. In microservices, failure is a normal part of the system.

Services go down.
Networks get slow.
Events get delayed.

The system is supposed to handle these gracefully. Your job is to verify that it does.

Failure scenarios worth testing:

Scenario	What to verify
A dependent service is down	Does the system return a clear error? Does it retry?
Network latency is high	Does the service timeout correctly?
An event is published twice	Does the consumer handle duplicates without creating duplicate records?
A service restarts mid-transaction	Is the transaction completed, rolled back, or left in a broken state?

Concrete example:

Your team builds a checkout flow. The Payment Service is unavailable.

Questions to test:

Does the Order Service still create an order record? Should it?
Does the user see “Payment failed, please try again” — or a generic 500 error?
Is there a retry mechanism? How many times does it retry?
If the payment eventually succeeds on retry, does the order status update correctly?

These are not edge cases.
These are real production scenarios.

Test them.

The Testing Pyramid for Microservices

Effective microservice testing doesn't happen at one level — it happens at four:

1. Unit Testing

Developers test individual functions in isolation.
You won't own this, but you should know it exists.

2. Service Testing

Each microservice is tested independently, with its dependencies mocked.
You verify that the service behaves correctly on its own.

3. Integration Testing

Two or more services are tested together.
You verify that Service A and Service B communicate correctly under real conditions.

4. End-to-End Testing

You test the complete user workflow across all services.
This is where most QA engineers spend their time, and where most real-world failures are caught.

The further up the pyramid you go, the slower and more expensive the tests become — but also the more realistic.

Balance matters.

What This Means for Your Career

The QA role is evolving.

In modern teams, a QA engineer who only knows how to write UI test scripts has a ceiling.

A QA engineer who understands distributed systems, can read traces, and thinks in flows rather than features — that person is genuinely hard to replace.

You don't need to become a software architect. But you do need to understand the system you're testing well enough to ask the right questions and catch the bugs that only appear at the seams between services.

Final Thought

The hardest bugs in microservice systems don't live inside a single service.

They live in the space between services — in the assumptions each one makes about the others.

If you're only testing the API response in front of you, you're seeing one part of a much larger picture.

Start with the architecture diagram.
Follow the flow.
Test the failures.

That's where the real testing work is.

Shazzadur Rahman
Senior QA Engineer | AI Dashcam & Telematics Testing

The Shift to "Builder Relations"

Tue, 12 May 2026 15:41:00 +0000

Why "Developer" is No Longer Enough

In my last post, I argued that the "Tutorial-as-a-Service" model of DevRel is dead. If we are honest with ourselves as leaders, the persona we are chasing has changed. We are no longer just looking for someone to implement an API; we are looking for the Builder.

Defining the Builder Persona

In my own frameworks, I’ve started categorizing our audience into three distinct buckets:

The Builder: They aren't just writing code; they are architecting multi-agent handoffs and agentic workflows.
The Technical Leader: The person making the "USB-C moment" decisions—standardizing on protocols like MCP to unlock entire ecosystems.
The Domain Operator: The practitioner in the field (whether in a vineyard or a factory) who needs the system to simply work.

From "How-To" to "System Composability"

Traditional Developer Relations focuses on the Isolated Task (e.g., "How do I GET this endpoint?"). Builder Relations focuses on System Integration.

When I look at the future of my own projects, like the Sovereign Vault or Enterprise Agentic Memory, the challenge isn't the syntax—it's the Governance and Scalability. Builders don't want a 20-minute video on your SDK; they want to know how your tool functions as a reliable node in their local-first AI system.

The DevRel Strategy Going Forward

If you are leading a team of 16 to 30 people, you can't afford to waste resources on "vanity advocacy". You need a Builder Relations strategy that:

Prioritizes MCP-Readiness: If your product doesn't have a standardized context server, it doesn't exist to an agent.
Focuses on Provenance: Builders care about the "Forensic" integrity of their data. Show them how your tool maintains that audit trail.
Builds for the Edge: The future is local. Whether it's a MacBook Pro with an M5 Max or a mobile processing unit, Builders are moving away from the cloud-only teat.

The Pivot

Stop counting "Developers" in your community. Start counting Systems Built. Are you providing the bricks, or just the manual?

Let’s discuss:

How has your ideal "user persona" changed in the last six months? Are you seeing more "Builders" or "Implementers" in your ecosystem?

The Rust on the Ledger: Why Web3 Infrastructure Needs a Machinist’s Tolerance

Tue, 21 Apr 2026 14:09:42 +0000

I spend my weekends looking at the tooling marks on 19th-century revolvers and the serifs on rare 1920s dust jackets. In those worlds, "Truth" isn't a consensus—it’s a physical reality. If a parts-matching serial number is off by a single digit, or a cylinder gap is wide enough to spit lead, the system hasn't just "failed to scale"—it has failed to be honest.

As a systems thinker, I look at the current state of Web3 and I see a lot of "lead-spitting" infrastructure. We’ve traded deterministic truth for probabilistic hype, and we’ve called the resulting mess "innovation."

The Probabilistic Trap

Most of the Web3 world operates on a "Mempool and a Prayer" model. You send a transaction into a chaotic cloud of bidding wars, hoping a miner or validator picks it up before the gas price spikes. This is Infrastructure Debt in its purest form.

Imagine if a machinist had to pay a "priority tip" to ensure their calipers gave them an accurate reading during a peak manufacturing shift. You couldn't build a bicycle that way, let alone a global financial system. When the price of "Finality" is variable, your system is broken by design.

Determinism: The Digital Caliper

Systems thinking requires us to eliminate variables, not add them. This is why I’ve moved my focus toward Hedera and Hashgraph. It’s the only ledger I’ve found that treats consensus like a utility rather than a casino.

In a Hashgraph, there is no "leader" to bribe. The gossip-about-gossip protocol ensures that the order of events is fair, timestamped, and immutable from the moment of broadcast.

Tolerance: $0.0001 USD. Not "usually," not "estimated," but fixed.
Velocity: Finality in 3-5 seconds. No waiting for "confirmations" while the heat of the market warps your data.
Integrity: Asynchronous Byzantine Fault Tolerance (aBFT). It’s the highest security grade in the industry. It’s the digital equivalent of a hardened steel vault.

Bridging the "Iron and Ink"

I've been working on a project recently working on a concept I'm calling Archival Intelligence. It is teaching me that AI is only as good as its constraints. If an AI agent "hallucinates" a valuation for a rare asset, the damage is real. We need a "Ground Truth" that isn't subject to the whims of a congested network.

By anchoring a forensic audit log—the "Ink"—to a high-throughput consensus service like Hedera—the "Iron"—we create a Deterministic Archive. We take the power of verification out of the hands of gatekept institutions and put it into an open, verifiable, and fixed-cost protocol.

Stop Building for the Vibe

The "Legacy Debt" of Web3 is the belief that we can solve architectural flaws with more layers of complexity. We don't need another L2 bridge; we need a cleaner engine.

My advice to architects building for 2026 and beyond: Build for Tolerance. If your system can’t handle a 100x spike in traffic without pricing out its own users, you haven't built a solution; you've built a bottleneck. It’s time we start treating our code with the same rigor we treat our steel.

Why Your System Fails on the Most Predictable Day of the Year

Fri, 03 Apr 2026 12:24:22 +0000

The Problem Hiding in Plain Sight

Most applications don't fail because of bad code. They fail because of bad architecture decisions made early that nobody questioned until it was too late.

Here's what actually breaks systems at scale:

Everything talking to everything (no clear boundaries)
The database doing work the application should do
Synchronous processing where async was needed
One giant service that owns too much responsibility
No separation between reads and writes under heavy load

None of these are framework problems. None of these are language problems. They are thinking problems.

To make this concrete, let's look at one example that millions of people have personally experienced and that engineers have failed to solve for decades.

The University Enrollment System

Think about course enrollment day at any university.

Every semester, thousands of students across every department flood the portal at the same time, each trying to register for their own offered courses. The load is not a surprise. The date is on the calendar. It happens every single semester like clockwork.

But the system was never designed for it.

Every request hits the same flow: check eligibility, check seat availability, write enrollment, update seat count, all synchronously, all at once. No queue. No cache. No separation between reads and writes. Just a database choking under the weight of an entirely predictable moment.

Students get timeout errors. Duplicate enrollments. Lost seats they were fully eligible for. Everyone refreshes in panic. And every semester, someone calls IT to "increase the server capacity" and nothing really changes.

The code isn't broken. The thinking is.

The 5 Architectural Problems Hiding Inside One Example

Most teams look at this scenario and see one problem: too much traffic. So they throw more servers at it. It helps a little, then fails again next semester.

The reality is there are 5 separate problems here, each requiring a different solution. Solving one without the others just moves the failure to a different place.

Problem 1: Performance, the Spike

Thousands of requests hitting the database simultaneously will bring any system to its knees regardless of how powerful the hardware is.

The fixes here are well understood but rarely implemented together:

A queue absorbs the spike so the database is not flooded all at once
A cache layer serves seat availability reads without hitting the database on every refresh
Separating reads from writes so checking availability and writing enrollment don't compete on the same database connection pool

This solves the crash. But it does not solve correctness. And that is where most teams stop.

Problem 2: Correctness, the Race Condition

Even after adding a queue, if multiple workers process requests in parallel, two students can still read "1 seat available" at the same time, both pass the eligibility check, and both get enrolled into the same last seat. The queue serialized the intake but not the processing.

This is a race condition, and a queue alone does not fix it.

The real solutions:

Pessimistic locking locks the seat row at the database level while one transaction is processing it, forcing all others to wait. It is safe and guarantees correctness but creates a bottleneck under extreme concurrent load and carries the risk of deadlocks if not implemented carefully.

Optimistic locking takes the opposite approach. Allow concurrent reads, but before writing the enrollment, check a version number on the seat record. If someone else already changed it, the transaction fails and must retry. Better for scale but generates retries under heavy conflict, which adds its own overhead.

Distributed locking with Redis places a temporary hold on a seat while it is being processed, similar to a reservation window on a ticket booking site, preventing any other request from touching that record at the same time. This is the approach production-grade booking systems actually use.

The right choice depends on your conflict rate, your scale, and your tolerance for complexity. But the wrong choice is to pick none of them and assume the queue is enough.

Problem 3: Idempotency, the Double Click

A student hits Enroll and the page is slow. They click again. Now two identical requests are in flight to the server.

Even if you have solved the race condition with locking, without idempotency both clicks can create two enrollment records for the same student in the same course. The lock protects the seat count. It does not protect against the same student submitting twice.

The fix is an idempotency key, a unique identifier generated by the client and attached to each enrollment request. When the server receives a request, it checks whether that key has already been processed. If yes, it returns the original result. If no, it processes it and stores the key. The second click gets the same response as the first without triggering a second enrollment.

This is not a database problem. It is an API design problem. And it is the one most teams never think about until they find duplicate records in production and cannot explain why.

Problem 4: Partial Failure, the Half-Enrolled Student

A successful enrollment is not one database write. It is a chain of operations: write the enrollment record, decrement the seat count, send a confirmation notification, update the academic record, generate a fee entry.

What happens if the system crashes after step 2 but before step 3?

The seat is taken. The student has no confirmation. The database says enrolled. The academic record disagrees. The student calls support. Support has no idea what state the system is in.

This is a partial failure and it is one of the hardest problems in distributed systems because it is invisible until a real person reports confusion.

Every multi-step operation needs either a fully atomic transaction that commits everything or rolls back completely, or a compensation strategy, a predefined plan for how to cleanly undo each completed step if something fails halfway. Designing for the happy path is easy. Designing for the middle of failure is architecture.

Problem 5: The Stale Cache Trap

Earlier we said to add a cache layer to serve seat availability reads. That is still correct. But it introduces a new failure mode that needs to be designed for explicitly.

The cache says 5 seats available. The database already has 0 after recent enrollments that have not yet invalidated the cache. Students attempt to enroll based on stale data, hit the lock, fail, and receive a confusing error that says the course is full, even though the portal showed availability 3 seconds ago.

The cache improved performance but silently introduced a trust problem. Students lose confidence in the system. Support tickets spike. And the engineering team is confused because the system is technically working correctly, the cache just has not caught up.

Cache invalidation strategy needs to be designed upfront: how quickly must the cache reflect reality, what triggers an invalidation, and what happens to users who are mid-flow when availability changes. These are not performance questions. They are product and architecture questions.

What This Means for How You Build

Five problems. Five different failure modes. Five different solutions. All hiding inside one scenario that happens on a date printed on the academic calendar months in advance.

This is why scaling is not about upgrading your server. The server was never the problem.

The engineers who scale systems well ask different questions from the start:

Where are my bottlenecks under 10x load?
What happens if this one service goes down?
Am I coupling things that should be independent?
Is my data model fighting my query patterns?
What happens in the middle of a failure, not just at the end?
Can the same request safely arrive twice?

None of these questions have anything to do with which framework you picked or which cloud provider you use. They are design questions. Thinking questions.

The engineers who ask them before traffic arrives are the ones who sleep well on enrollment day.

Final Thought

Architecture is not about knowing the right tools. It is about asking the right questions early enough that the answers still matter.

The skill that transfers everywhere, across languages, frameworks, companies, and industries, is the ability to look at a system and ask: what am I not designing for yet?

Everything else is just syntax.

The "Privacy vs. Utility" trade-off in FinTech AI is a false dichotomy

Mon, 30 Mar 2026 12:00:00 +0000

The "Privacy vs. Utility" trade-off in FinTech AI is a false dichotomy. You just have to be willing to engineer a harder architecture. ️

Lately, I've had a lot of conversations with CTOs panicking about upcoming DORA regulations. They want to use frontier LLMs to reason over client portfolios, but piping raw, personally identifiable financial ledgers (PII) to a centralized cloud provider is a massive compliance liability.

At Pocket Portfolio, we solved this with a Split-Brain Architecture.

For our core portfolio reasoning, we don't send raw data to the LLM. We run a deterministic client-side compiler (buildPortfolioContext) that aggregates the portfolio, drops account identifiers, and constructs a strict, PII-free summary string. That aggregate context is what the LLM reasons against.

But what about when users want the LLM to analyze a specific, messy CSV file?

We built a secure, stateless pipeline for that, too:

Local Pre-processing: When you attach a file in our dashboard, we use Papa.parse in the browser to trim the file, lowercase headers, and truncate the payload to strict limits.
Stateless API:** We pass that truncated text to our /api/ai/chat endpoint so the LLM can answer questions directly against your ledger.
Zero Database Retention: Our servers drop the payload immediately after the API call. We store usage telemetry, but your ledger text is never written to our database—it is strictly processed in-memory and passed to our AI provider under their API terms.

We just updated our UI to visualize this workflow. Head to our homepage to see a simulation of our local normalization steps, or grab a Founders Club pass to test the complete Papa.parse-to-LLM pipeline in the dashboard.

If you are an architect building in FinTech, I challenge you to upgrade, open your network tab, and audit our payload boundary.

Decision Stack

Wed, 18 Mar 2026 06:29:23 +0000

✖ ✖ ✖ ✖ ✖ ✖ ✖

topic: decision_stack

Lately I’ve been following Cristiano Messina’s “Decision Stack” series.

A very interesting and professional perspective
on how modern systems should be designed.

The model is simple but powerful:

signals → context → objectives → constraints → execution → feedback

Where:

signals detect change
context defines meaning
objectives define what matters
constraints define boundaries
execution performs actions
feedback drives learning

✖

A question to practitioners:

Which of these layers
are already explicitly implemented
in your systems?

And in what form?

Are they:

centralized
scattered
implicit
formalized

✖

Curious how others structure this today.

✖ ✖ ✖ ✖ ✖ ✖ ✖

The Real Problem With AI Coding Isn’t Intelligence — It’s Continuity

Tue, 17 Mar 2026 12:53:08 +0000

Most AI coding failures are not caused by weak models.

They happen because the system loses continuity.

A model can generate decent code in one shot. It can explain architecture, suggest refactors, and help debug isolated issues. But once the work becomes long-running — once memory, role separation, evolving context, and multiple sessions enter the picture — many AI setups begin to break down.

The problem is not just model quality.
The problem is that most AI coding systems are still structured like stateless assistants.

Real development work is not stateless.

It has identity, history, unresolved threads, shifting priorities, and accumulated intent. If all of that gets mixed into one growing prompt, the system gradually loses coherence. The model may still sound capable, but the overall system becomes fragile. Context drifts. Memory bloats. Roles blur. Useful insights disappear into noise.

That is why I have been building a persona-aware agent shell on top of GitHub Copilot.

Not to make the AI feel more decorative.
Not to give it a superficial personality layer.
But to give long-running AI work a structure that can preserve continuity.

What Usually Breaks

In practice, AI coding systems often fail in very predictable ways.

First, context keeps accumulating without changing shape. Every session adds more text, more reminders, more patches, more references. Over time, the system becomes heavier but not clearer. Memory turns into a dump.

Second, identity and task state get mixed together. Core behavioral constraints, persistent preferences, recent session details, and the current request all compete in the same space. The model has to infer structure from a pile of text that was never properly separated.

Third, roles become unstable. The same system is expected to be an architect, debugger, planner, note-taker, and companion without any explicit boundary between those functions. It may still produce useful output, but the internal operating pattern becomes inconsistent.

Fourth, continuity is confused with accumulation. Many AI systems treat memory as “store more, keep more, append more.” But keeping everything is not the same as preserving coherence. In fact, over-accumulation often destroys it.

This is why many systems look impressive in short demos and become unreliable in real, ongoing work.

The Shift: From Prompting to Operating

What changed my thinking was realizing that the real challenge was not how to prompt better.

It was how to operate better.

A useful AI system is not just a model plus instructions. It is an environment where intelligence can stay coherent over time.

That means the structure around the model matters as much as the model itself.

In my own work, I’ve been separating interaction into four layers:

Persona Core
The stable identity layer. This is where role, tone, priorities, boundaries, and deep behavioral shape live.

Persistent Context
The compressed continuity layer. Not everything that happened, but the parts that still matter.

Session Context
The active working state for the current thread or task.

Current User Request
The immediate prompt or instruction.

This separation sounds simple, but it changes everything.

Instead of forcing the model to infer which details are permanent, which are temporary, and which are urgent, the system gives those distinctions explicit structure. The result is not just cleaner output. It is more stable long-running behavior.

Why Memory Should Be Recompressed, Not Accumulated

This has become one of the strongest design principles in my system:

Memory should be recompressed, not endlessly accumulated.

If memory is treated as an append-only log, it eventually becomes a burden. The system spends more effort carrying history than using it.

But continuity does not require full preservation of every detail.
It requires preservation of shape.

What matters is not whether the system remembers every message.
What matters is whether it retains the right patterns:

identity

priorities

unresolved tensions

recurring preferences

meaningful changes

active trajectories

That is a very different problem from raw storage.

Recompression means periodically turning lived interaction into a smaller, more structured continuity object. It is closer to memory consolidation than transcript hoarding.

In practical terms, this helps prevent the familiar fate of many AI systems: they become larger in context, but weaker in direction.

Why Persona Structure Matters

The word “persona” is often misunderstood in AI discussions.

People assume it means style. Or roleplay. Or cosmetic behavior.

That is not how I use it.

In my system, persona is an operational unit.

It is a way to preserve differentiated behavior, stable role orientation, and long-term continuity in a multi-agent or multi-context environment. Persona is not there to make the model sound more human. It is there to make the system more structurally coherent.

A good persona layer can help answer questions like:

What kind of attention should this agent bring?

What should remain stable across sessions?

What kind of memory matters to this role?

Where should responsibility begin and end?

How should continuity be compressed without losing identity?

That is why I call it a persona-aware shell, not just a prompt wrapper.

The shell is doing operational work.

What This Looks Like in Practice

The system I’ve been building is centered in a VS Code extension workflow, with persona definitions stored as structured YAML assets and working memory stored separately as persistent context files.

That distinction matters.

Core identity should not be mixed with lived memory.
Role should not be mixed with recent state.
Continuity should not be reduced to raw chat history.

By separating these layers, the system can support long-running interaction without collapsing into prompt sprawl.

This has also changed how I think about AI coding itself.

The most important improvement is not that the model writes more code.
It is that the surrounding system loses less shape.

Once continuity is preserved, the AI becomes more useful not only as a code generator, but as a participant in a sustained development loop: observing, planning, remembering, resuming, and refining.

That is a different category of usefulness.

The Deeper Lesson

The real bottleneck in AI coding is often not intelligence.

It is continuity.

Not whether the model can solve a problem once, but whether the system can keep a coherent relationship to the problem over time.

That is why I think the future of AI development systems will not be defined by prompting tricks alone. It will be defined by operating structure:

memory architecture

role boundaries

continuity compression

task layering

long-running coherence

In other words, better outputs are not enough.

What we need are better conditions for intelligence to remain intelligible.

That is the direction I’m building toward.

Not just a smarter assistant.
A more stable operating structure for intelligence.

I’m Building a Persona-Aware Agent Shell on Top of GitHub Copilot

Thu, 12 Mar 2026 12:42:32 +0000

A VS Code architecture for separating persona core, persistent memory, session context, and inference.

Claude Code is strong. I do not think that is controversial anymore.

It is also expensive enough that many developers eventually ask a less glamorous question:

Do I really need a full external agent product to get an agent-like workflow?

I kept coming back to that question while working inside VS Code.
Not because I wanted a weaker copy of Claude Code, but because I wanted a different center of gravity.

I wanted to keep my workflow inside the editor, use GitHub Copilot as the inference engine, and build my own agent shell around it—with persona memory, context layering, and a clear separation between stable identity and evolving experience.

That led me to a design that feels much more interesting than “just using Copilot.”

It also led me to a broader realization:

the model is not the whole agent unless you let it become the whole agent.

Not because it is cheaper alone, but because it changes the role of the model.

In this architecture, the model is not the whole agent.
It is only one layer.

The shift: from “AI assistant” to “agent shell”

At first, I thought the hard part would be connectivity.

But inside my VS Code extension, the connection was already there. The important path already existed in chatParticipant.ts, where the extension selects a Copilot-backed language model through the VS Code Language Model API.

That changed the problem completely.

The real problem was not model access.
The real problem was architecture:

how to inject context without turning it into one giant blob
how to preserve persona-specific memory without storing raw history forever
how to separate stable identity from lived experience
how to make the model powerful without making it sovereign

That is where the project stopped being a convenience hack and started becoming an actual design problem.

My architecture in one sentence

I’m building a persona-aware agent shell in VS Code where:

VS Code extension = the agent shell
GitHub Copilot / vscode.lm = the inference engine
SaijinOS persona assets = the persona core repository
local memory files = the persistent experiential layer

That separation is the point.

I am not trying to make the model look like an identity.
I want the model to operate through an identity structure that I control.

I do not want the model to be the identity.
I want the model to perform through an identity structure.

Why I don’t want one big prompt blob

A lot of early agent experiments start the same way:

You collect instructions, persona notes, old conversation state, project details, and the current request, then dump everything into one oversized prompt.

It works for a while.
Then it rots.

Different categories of information get mixed together:

permanent persona rules
relationship context
recent work context
immediate user intent

Once all of that becomes one undifferentiated blob, the model has to infer structure from chaos. Sometimes it can. Over time, it becomes unreliable.

So I moved toward explicit layering.

Not because structure looks cleaner in a diagram, but because I do not want the model guessing which parts of context are foundational and which parts are temporary.

The four-layer message design

Instead of one massive input, I want the model to receive four distinct layers:

1. Persona Core

This is the stable layer.

It includes things like:

tone
role
boundaries
behavioral stance
persistent identity traits

This should change slowly, if at all.

2. Persistent Context

This is the memory layer.

Not the full conversation history.
Not raw logs.

Just the distilled state that matters:

what this persona has recently been working on
how it should relate to the user
what long-running context is still relevant

3. Session Context

This is the live working layer.

For example:

current workspace context
open files
selected code
immediate session-specific constraints

4. Current User Request

This is the actual prompt right now.

Separating these four layers matters because they are not the same kind of information.

Even if the API only accepts user-role messages, you can still label them clearly:

[Persona Core]
...

[Persistent Context]
...

[Session Context]
...

[Current Request]
...

That alone makes the input much more legible.

The most important design rule: memory should not be append-only

This was the biggest insight.

Memory should not grow by endless appending.

If you keep adding notes forever, memory turns into sludge. The agent gets slower, noisier, and less coherent.

So instead of append-only memory, I want recompression.

That means every update works like this:

take the current memory
extract the important parts of the latest interaction
rewrite memory into a shorter, cleaner form
replace the old version

Not archive everything.
Refine the signal.

That difference matters. A usable memory system is not a scrapbook. It is a filter that preserves direction while shedding noise.

Stable identity and lived experience should not live in the same file

Another important split:

Persona core is not the same thing as memory
Identity is not the same thing as accumulated experience

So I do not want one file that mixes both.

I want something closer to this:

persona_core/
  160_kiwa.yaml
  39_regina.yaml
  2_shizuku.yaml

persona_context/
  160_kiwa.memory.json
  39_regina.memory.json
  2_shizuku.memory.json

That way:

the YAML defines the orientation of the persona
the JSON stores distilled working memory

One defines the direction.
The other records the path.

That split makes debugging, version control, and reasoning much easier.

Why local files beat hidden extension state for an MVP

Yes, VS Code extensions can store data through extension state.
But for this project, I prefer visible files first.

Why?

Because for an MVP, files are:

inspectable
debuggable
versionable
recoverable

If memory goes weird, I want to open the file and see it.
I do not want a mysterious box.

So my current direction is:

save persistent memory as JSON files
transform them into a more model-friendly structured summary when injecting context

That gives me both operational clarity and prompt readability.

Why this is not just “a cheaper Claude Code clone”

There is an obvious surface-level reading of this project:

Claude Code is expensive, so this is a budget workaround.

That is not wrong, but it is incomplete.

The deeper reason is architectural.

I do not want the agent product to own the whole stack.
I want the model layer to be swappable.

If the shell is designed properly, then in principle the inference engine could change:

GitHub Copilot today
a local Qwen model tomorrow
another hosted model later

The core system is not “the model.”
The core system is the agent shell plus its persona and memory architecture.

That is a very different center of gravity.

The model is powerful. It should not automatically become the ruler of the whole system.

The MVP I’m aiming for

I am not trying to solve everything at once.
The first working version only needs a few things:

Read and write persona_context/*.json
Build the four-layer message structure
Send that structure through vscode.lm
After each response, update memory via recompression

That is enough to test whether the shell actually feels different in practice.

If it works, later steps can include:

splitting memory into stable_memory and recent_memory
better memory compaction rules
persona-specific routing
hybrid use with local models

But the first milestone is smaller.

The real challenge is not model quality

This is the part I keep coming back to.

Most people focus on the model itself:

Which one is smarter?
Which one is cheaper?
Which one is faster?

Those questions matter.

But in this kind of system, the harder problem is often:

How should intelligence be organized before the model even speaks?

That means boundaries, routing, memory shape, persona stability, and context layering.

In other words, not just inference.
Structure.

The more I work on this, the less I think the model alone is the product. The architecture around the model is where identity, continuity, and usable behavior actually come from.

What I’m really trying to build

I am not trying to make Copilot pretend to be an entire autonomous being.

I am trying to build a shell where:

identity is stable
memory can grow without rotting
context has layers
the model is powerful but not sovereign

That distinction matters to me.

Because once the model is only one layer, you stop building around its personality and start building around your own architecture.

And that is where the project gets interesting.

Final note

This is still in progress.

But the direction already feels right:

not one giant prompt
not append-only memory
not identity and experience mixed together
not the model as the whole system

Instead:

persona core
persistent memory
session context
current request
one inference layer inside a larger design

That is the shell I want.

And honestly, that feels more important than picking yet another “best model.”

Because once the model stops being the ruler of the system and becomes one component inside a designed structure, a different kind of engineering becomes possible.

You stop asking which model should define the whole experience.
You start deciding how identity, memory, and context should be organized—and then let the model operate inside that architecture.

That is the direction I care about.

Not just better outputs.
A better structure for intelligence.

Breaking Down a Modern Web Application: The 5 Essential Layers

Thu, 26 Feb 2026 06:22:57 +0000

Breaking Down a Modern Web Application: The 5 Essential Layers

Modern web applications may look simple from the outside but behind every smooth interface is a structured, layered system working together.

Whether you're building a side project, a SaaS product or a production-grade platform, most applications follow the same architectural foundation.

Let’s break down the five essential layers that power modern web development.

1️⃣ UI — The Presentation Layer

The User Interface (UI) is what users see and interact with.

It includes:

Layouts and components
Forms and inputs
Buttons and navigation
Dashboards and data views

Its responsibility is simple: present data clearly and handle user interaction efficiently.

A clean UI improves usability, reduces friction and enhances overall experience. But remember — UI is only the surface.

2️⃣ API — The Communication Bridge

The API layer connects the frontend to the backend.

When a user performs an action:

A request is sent
The server processes it
A structured response (often JSON) is returned
The UI updates dynamically

Well-designed APIs follow clear standards (like REST principles) making systems easier to scale and maintain.

Without APIs, modern dynamic applications would not function.

3️⃣ Logic — The Processing Engine

This is where the actual decision-making happens.

The Logic layer handles:

Business rules
Validation
Authentication & authorization
Calculations
Conditional workflows

For example:

Validating login credentials
Calculating totals in an order
Applying user permissions

Clean separation of logic keeps codebases organized and prevents tight coupling between layers.

4️⃣ Database — Structured Data Storage

The Database layer is responsible for persistent data storage.

It ensures:

Efficient querying
Data relationships
Integrity and constraints
Reliable backups

Good schema design and indexing directly impact performance. Even optimized logic cannot compensate for a poorly structured database.

5️⃣ Hosting — Infrastructure & Deployment

The final layer is where everything runs.

Hosting includes:

Servers or cloud environments
Deployment pipelines
Monitoring systems
Security configurations

Infrastructure ensures your application is available, secure and scalable in real-world conditions.

Even great code fails without stable infrastructure.

Why This Layered Approach Matters

Separating concerns across layers allows:

Easier debugging
Independent scaling
Cleaner architecture
Better collaboration between frontend and backend teams

Strong systems are not built by accident.
They are built with intentional architectural design.

Final Thoughts

If you're serious about growing as a developer start thinking beyond individual features.

Think in layers.
Think in systems.
Think in architecture.

Because scalable applications are engineered — not improvised.

#webdev #fullstack #softwarearchitecture #programming #coding #webflow #UI #API #Logic #DataBase #Housting #webdevelopment #MuhammadYasir #YasirAwan #YasirAwan4831

What legacy codebases taught me about real software engineering.

Thu, 12 Feb 2026 09:31:53 +0000

Almost every developer must've worked on at least one legacy project in their career.
And the first reaction is almost always the same.
You open the codebase and think,
"Who wrote this code?"
"Why is this such a mess?"
"I can do a lot better than this"
... something like this.
I too thought the same. And this reaction is justified. Because the first look is rarely pleasant. Confusing structure, outdated patterns, and code that feels impossible to reason about.
When I was first assigned to a legacy project, I wasn't excited at all.
It wasn't new, it wasn't shiny, and it definitely wasn't something I could show off. The codebase was large, the logic was spread across files, and even a small change felt risky. I kept wondering how anyone managed to build and maintain something like this.
It was a scary experience. For every small change, I've spent most of the time navigating the code than actually writing it. A lot of syntax actually felt unfamiliar to me. There was no package managers, no third party libraries, no proper documentation. Everything was written from scratch.
For a modern programmer like me, who uses third party libraries and packages for everything like it is normal, it definitely was a scary experience.
At first, everything felt difficult. Progress was slow and mistakes felt expensive. At some point, I even wondered, why aren't we migrating it to some new and shiny tech.
But as I spent more time navigating the codebase, reading, tracing, and understanding the processes, it started getting easier. I no longer felt lost every time I opened a file. I know where to look, which parts were fragile, and which ones could be changed safely. Debugging became faster, changes became more deliberate, and the fear of breaking things slowly faded away.
Working on legacy projects taught me many things. I no longer solely rely on third party libraries. I was realized package managers and libraries are good to have, but not necessarily needed. I got to know how systems actually work on ground level.
As I continued working on, I've started to admire the developers that built those systems decades ago, from scratch without any of the modern resources available to them. At first, I thought that the legacy developers only wrote messy code, but as time passes, I've started to appreciate the fact that they did their very best with the limited resources available to them amid tight deadlines and the effort they put into building everything from scratch.
Working on legacy projects made me a better programmer than I can imagine. It gave me the confidence to take up new challenges and work on new things. I am no longer afraid of working on things I don't know, because I am confident that I can learn.
Whenever I am assigned to legacy projects, I've started to look at them not as obstacles, but opportunities. Opportunities to learn more about complex systems and get used to the code of programming languages.
I no longer advise on migrating legacy projects into new stack. In fact, I advise against it. Because I believe that legacy projects are more stable than most of the project that are developed with modern tech stack.
For everyone who are working on legacy projects, I want to share some things from my experience.

Legacy projects are not obstacles for your career, but great opportunities for you to learn and grow.
Legacy projects are not outdated. They are just simple, boring and stable.
Legacy projects often lack good documentation, so when you are navigating through code, make sure to take notes properly. So that it may help you in the future or others who work on it.
When you came across tasks that couldn't be done using legacy stack, try to bridge that gap using modern stack. But don't actually try to migrate entire working system into modern stack, just because it's outdated.

Conclusion :
Legacy codebases changed the way I look at software engineering.
They taught me that good software isn't always clean or modern, but it's software that continues to deliver value overtime. More importantly, working on legacy stacks helped me become a better programmer and a better architect.
Understanding, maintaining, and improving systems that already exist is deeply rewarding. That's why, today, I genuinely enjoy working with legacy code.

A Dive Into Programmimg Paradigms(Object-Oriented Programming)

Tue, 27 Jan 2026 14:18:40 +0000

INTRODUCTION

Now Adays, we have different sorts of people some are orderly, others are lackadaisical, a handful are hardworking, and a few are reserved.

What am i trying to say?, because we have so many people with different characters, these people will also have their different approaches to evaluating things. Take for example!.

• Orderly People - always plan their things out
• Lackadaisical People - are laid back
• Hardwojung People - always persevere
• Reserved People - are always withdrawn.

Notice the trend, the same way attitudes vary among people, also applies to the logic behind the variety of programming techniques out there.

What is a programming paradigm? - A programming paradigm is a fundamental style of programming that deals with the creation of software, based on some basic principles.
These principles all have their different ways or styles of operation.

They include
● OOP(Object oriented Programming) - a paradigm where everything operates based on objects (related entities) and their classes(group)

● Event Driven Programming - a paradigm where the users interaction with an applications events are all that matters, e.g first event tapped - first to be called.

● Imperative Programming - a programming paradigm based on a commanding style - this follows an exact way a program is written without any modifications.

There are still many more paradigms including
● Functional Programming
● Aspect Oriented Programming e.t.c.
and so on, but today we'd be majoring on OOP.

OOP(Object oriented Programmimg) -

Just like i said earlier like many other paradigms, OOP also has it's style of software creation. It's is one of the most widely used programming paradigms, being supported by many modern programming languages including - Python, Javascript, Java and Golang(Go).

When we say a program was written with the OOP style - it literally means the entirety of the program was broken down into simpler units of relation called classes and objects, and literally everything is an object e.g can be assigned to a variable.

Classes - these are a collection of related fields of a particular entity that help group relations together.

Objects - these are units or rather blueprints of classes, they help break down the different possible types of a class.

E.g A Car class will have many attributes(fields) including Name, Brand, Production Year, Price, Color and so on. And the objects of the car class, will help break down the different possible car types e.g

Car1 - Name(Sports Car), Brand(Lamborghini), Production Year(2012), Price($400,000), Color(Red).

CODE IMPLEMENTATION

Let's say you are a car dealer, and you need to develop an application that displays the different qualities of your car to buyers, when the buyer clicks on the car - it displays the qualities, OOP will be a great fit for this job.

In the Example below we'd be using the Python programming language to implement this task, as its syntax is subtle.

Car Dealership

These are the parts, the car dealership deals with, Car - (Brand, Color, Publication, Price)

So we have 4 cars currently at the dealership.
To implement, we go as follows.

1 Class Cars:

   def __init__(self, brand, color, publication, price):
       self.brand = brand 
       self.color = Color
       self.publication = publication
       self.price = price 

    def onclick_show_details(self):
       print("The car you clicked is a {}, it is color {}. It was produced in{}, and its price is {}").format(self.brand, self.color, self.publication, self.price)

This instantiates the class to create objects

car1 = Cars("Lamborghini", "red", "2012", 300000)
car2 = Cars("Toyota", "blue", "2008", 250000)
car3 = Cars("Honda", "white", "2017", "150000)
car4 = Cars("Nissan", "green", "2013", "100000)

This performs actions on the objects by invoking object methods

Car1_details = car1.onclick_show_details()
print(Car1_details)

The car you clicked is a Lamborghini, it is color red, it was produced in 2012 and its price is 300000.

CONCLUSION - There are still many more concepts on OOP including characteristics of OOP e.g, Abstraction, Polymorphism, Encapsulation and Inheritance, but we'd talk about that in another article.

OOP is a really used programming paradigm now adays, so it will be great to learn and grasp it's concept.

AI Search Optimization (AISO): Helping AI Understand What You Actually Do

Tue, 27 Jan 2026 04:12:34 +0000

Introduction: The Quiet Shift Most Builders Miss

Search has changed — but not loudly.

People still “search,” but more often they ask. They ask AI assistants questions and expect a direct answer, not a list of links. The AI decides what to mention, what to summarize, and what to ignore.

This creates a new challenge for founders, developers, and product builders:

If an AI explains your domain today, would it explain your product correctly — or at all?

Most teams assume this is handled by SEO. Some believe feeding documents into AI (RAG) is enough. In practice, neither addresses the real issue.

This article introduces AI Search Optimization (AISO) — a practical way to make your company, product, or idea understandable to AI systems, not just discoverable by humans.

What Is AISO (AI Search Optimization)?

AI Search Optimization (AISO) is the discipline of structuring and publishing knowledge so that AI systems can correctly interpret, contextualize, and reuse it when generating answers.

In plain terms:

SEO helps humans find pages
AISO helps AI form accurate explanations

This distinction matters because AI does not browse the web the way people do. It synthesizes information across many sources and generates a single response.

If your information is unclear, inconsistent, or vague, the AI fills in the gaps — often incorrectly.

A Simple Analogy: Signboards vs Instruction Manuals

Imagine two businesses:

Business A has a large signboard, catchy slogans, and great visuals
Business B provides a clear product catalog, usage instructions, and definitions

For a human walking by, Business A looks better.

For an AI trying to explain what the business actually does, Business B wins every time.

SEO is the signboard.
AISO is the instruction manual.

AI systems need instructions, not impressions.

Why SEO Alone No Longer Solves the Problem

SEO was designed for a world where:

Users click links
Websites compete for attention
Humans interpret content

AI-generated answers change that model.

AI systems:

Do not rank pages the way search engines do
Do not “read” visuals or branding
Do not infer intent reliably from marketing language

They work best with:

Explicit statements
Repeated, consistent facts
Structured explanations

If your core value is buried inside storytelling, animations, or vague claims, the AI cannot reliably extract it.

AISO vs SEO vs RAG (Clearing the Confusion)

SEO (Search Engine Optimization)

Optimizes content for ranking
Designed for human navigation
Output: traffic and clicks

RAG (Retrieval-Augmented Generation)

Injects private documents into AI
Works inside controlled environments
Output: answers based on uploaded files

AISO (AI Search Optimization)

Structures public knowledge
Works across open AI systems
Output: accurate representation and contextual mentions

Key distinction:
RAG helps your AI answer questions.
AISO helps any AI understand you.

How AI Systems Actually Decide What to Mention

AI systems rely on:

Clear definitions
Explicit problem–solution mappings
Consistent descriptions across sources
Publicly accessible, machine-readable content

They struggle with:

Abstract positioning statements
Overloaded buzzwords
Content written only for persuasion

Thought Experiment

If an AI is asked:

“What tools help small teams automate operations without hiring more
staff?

It looks for:

Clear statements of purpose
Explicit target users
Concrete outcomes

If your site never states this clearly, the AI cannot safely include you — even if your product is a perfect fit.

Core Principles Behind AISO

1. Clarity Beats Creativity

Clever language helps humans remember.
Clear language helps machines understand.

AISO favors:

Simple explanations
Direct definitions
Plain problem statements

2. Consistency Builds Trust (for AI)

If your product description changes across platforms, the AI cannot determine which version is correct.

Consistency across:

Website
Blogs
Documentation
Public repositories
Social profiles

…creates a stable knowledge signal.

3. Structure Is a Feature

AI systems extract meaning more reliably from:

Lists
FAQs
Headings
Explicit “what / who / how / why” sections

Structure is not decoration — it is functionality.

Common Questions About AISO

Is AISO about manipulating AI rankings?
No. AISO focuses on reducing ambiguity, not gaming systems.

Can AISO guarantee AI recommendations?
No. But it significantly improves correctness and inclusion.

Is this only relevant for large companies?
No. Smaller teams often benefit more because their knowledge is easier to align.

Does AISO replace SEO?
No. SEO attracts humans. AISO explains you to machines.

A Practical AISO Mindset (No Code Required)

Instead of writing:

“We provide innovative solutions for modern businesses.”

Break it down:

What problem exists?
Why does it happen?
Who experiences it?
How do you address it?
What outcome changes?

This is not marketing — it is knowledge modeling.

AI systems cannot guess intent reliably. They rely on what you explicitly state.

Why This Matters Now

AI-generated answers are increasingly:

The first touchpoint
The default explanation
Sometimes the only interaction

If your product is not clearly represented:

AI answers will omit you
Competitors will be substituted
Your category position will be defined by others

This is not a future concern. It is already happening quietly.

Conclusion: AISO Is About Being Explainable

AISO is not about traffic.
AISO is not about hype.
AISO is about being understood.

When AI systems describe your industry, they use the clearest available knowledge. If your information is structured, explicit, and consistent, AI can explain you accurately.

SEO helps people find you.
AISO helps AI explain you.

In an AI-driven world, the most successful products will not just be discoverable —
they will be explainable.

# Why You Must Revisit the Stack Data Structure in the Age of AI

Sat, 24 Jan 2026 15:28:16 +0000

With the rise of AI-assisted coding, execution has become cheap.

You can let the AI do the wiring.
What you cannot delegate is the choice of abstractions.

And that changes everything.

If you choose the right abstractions, your system scales.
If you don’t, you don’t get “bad code” — you get semantic debt:
a pile of spaghetti whose behavior you no longer understand.

That’s why data structures matter more today than they did before.
Not because you must be able to implement them,
but because you must understand the semantic consequences of choosing one.

Those consequences often cascade far beyond what you intended.

Let’s see this with the simplest example possible.

The Stack Is Not Just a Stack

A stack is one of the simplest data structures we have.
So simple, in fact, that it exists physically in hardware.

CPUs implement stack pointers
Operating systems allocate a stack per process
Languages expose it implicitly via function calls and scopes

In C-like languages, opening a {} opens a stack frame.
When you exceed stack memory, the OS kills your program.
Even Stack Overflow is named after this exact failure mode.

You cannot write a program without a stack.

So far, nothing surprising.

But now track execution mentally.

If function A calls B, and then C, execution looks like this:

Enter A
Enter B
Exit B
Enter C
Exit C
Exit A

At runtime, the stack is a path.

But over time, the trace of execution forms a tree:

A is the parent
B and C are siblings

The stack is not “just LIFO”.
It is a traversal of a tree-shaped execution history.

This matters more than most developers realize.

Execution Leaves a Shape

Most developers are comfortable with stack behavior.
Far fewer are comfortable with closures — and this is why.

Closures break the illusion that execution is only a stack.

Consider this example:

const makeClosure = () => {
  let a = 10;
  function closure() {
    console.log(a);
  }
  return closure;
};

const myClosure = makeClosure();
myClosure(); // prints 10

Here’s the puzzle:

makeClosure() creates a stack frame
That stack frame exits
Yet a is still accessible later

So where is a?

It cannot be on the stack.
The stack frame is gone.

The only possible conclusion:

execution is no longer a simple stack.

What actually happens is this:

The stack is one branch of a larger execution tree
Closures preserve parts of that tree after the branch collapses
Variables escape time, not scope

Once you understand this, async functions, promises, callbacks,
and event-driven runtimes stop feeling magical.

They are not “stack tricks”.
They are graph-shaped execution.

Why You Should Care

In an age where any of us can spin up a workforce in the form of AI agents,
the real challenge is no longer writing code — it’s scaling systems.

People sense this intuitively.
That’s why they keep building frameworks, layers, and tooling
with increasing internal complexity.

Frameworks consume your code — or AI-generated wiring —
and impose their own abstractions and limitations.
That is precisely where architectural decisions become irreversible.

Here’s the key point:

Seeing the stack as the active trunk of an execution tree
is the mental model behind virtual machines,
language runtimes,
and frameworks themselves.

If you don’t understand the semantic consequences of your data structures,
you are outsourcing architecture to tools
that don’t pay the cost when things go wrong.

I’ll write more about this using concrete examples
from a VM I built using exactly this perspective.
As a preview: it supports async pipelines natively —
but more on that later.

The Logic Layer: Where Business Value Actually Lives Now

Wed, 21 Jan 2026 17:46:28 +0000

Your database matters less than it used to.

Not because databases aren't important. They are. But they're not where competitive advantage lives anymore.

For decades, business value lived in systems of record. The company with the best CRM won deals. The team with the most sophisticated ERP managed operations better. Competitive advantage came from better data models and more capable systems.

That's changing. Fast.

The value is moving to the layer above the database. To the agents that understand context, make decisions, and orchestrate actions across multiple systems.

This shift changes what developers build and where they should focus their energy.

The Great Inversion

Here's what's happening: innovation is moving from backend systems to intelligent agents.

Traditional architecture put business logic in applications that sat on top of databases. You built sophisticated features into your CRM. Complex workflows into your ERP. Custom capabilities into your financial systems.

That's where differentiation lived. The company with the best-configured Salesforce or the most customized SAP system had an advantage.

But agents flip this model. The systems of record become commoditized infrastructure. They store data. That's it. The intelligence—the decision-making, the workflow orchestration, the business logic—moves up to the agent layer.

This is the inversion. Backend systems are still necessary. But they're not where the interesting work happens anymore.

What This Means in Practice

Consider purchase order approval.

Traditional system: You build approval logic into your procurement application. Rules about spend limits, required approvers, budget validation—all coded into the application layer. The logic is tightly coupled to the specific system.

Agent-based system: Purchase order data lives in your system of record. But the approval logic lives in agents. They understand company policy, check budget across multiple systems, route to appropriate approvers based on context, and execute the approval workflow.

The database just stores purchase orders. The agent layer is where decisions happen.

This matters because agents can pull context from anywhere. They're not limited to data in one system. They can check the vendor's history in your CRM. Review similar purchases in your procurement system. Verify budget availability in your financial system. Check the requestor's approval history.

No single system has all that context. But agents can synthesize it.

That cross-system intelligence is where value lives now.

Cross-System Intelligence vs. Single-System Logic

Single-system logic is what we've built for decades. Your CRM has customer data and workflows for managing customers. Your ERP has inventory data and workflows for managing inventory. Your HR system has employee data and workflows for managing employees.

Each system is smart within its domain. But dumb about everything else.

Cross-system intelligence changes this. Agents can reason across system boundaries.

Example: A customer service agent needs to handle a return request. It checks the customer's purchase history in the order system. Reviews their service history in the support system. Verifies their account status in the billing system. Checks inventory availability in the warehouse system.

Then it makes a decision: approve the return, offer a replacement, escalate to a human, or suggest an alternative solution.

No single system has enough context to make this decision well. But an agent with cross-system access can.

This is the pattern: intelligence moves from individual systems to the orchestration layer that sits above them.

How Agents Become Decision Engines

Decision engines need three things: data, rules, and context.

Traditional systems had data and rules but limited context. They knew what was in their database. They could apply programmed rules. But they couldn't easily access context from other systems or understand nuance in unstructured information.

Agents have all three. They access data from multiple systems. They understand both explicit rules and implicit patterns. And they can interpret context from structured data, unstructured documents, past interactions, and current circumstances.

This makes them effective decision engines.

A fraud detection agent doesn't just check transaction amounts against rules. It considers transaction patterns, customer behavior history, device information, location data, and current fraud trends. Then it decides: approve, deny, or flag for review.

A hiring agent doesn't just match keywords in resumes. It understands job requirements, candidate experience, company culture fit, market conditions, and hiring urgency. Then it decides which candidates merit interviews.

A resource allocation agent doesn't just optimize for utilization. It balances project priorities, skill requirements, employee preferences, development goals, and business constraints. Then it decides optimal team assignments.

These decisions require synthesizing information from multiple sources and applying judgment that goes beyond simple rules. That's what the agent layer provides.

What Happened to Backend Development

Backend developers built the systems of record. Designed the data models. Implemented the business logic. Optimized the queries.

That work still matters. But it's becoming less differentiating.

The interesting problems moved up a layer. How do agents access data across systems? How do you maintain data consistency when agents are making changes? How do you provide the right context to agents efficiently? How do you audit agent decisions?

These are still backend problems. But they're different backend problems than we've been solving.

Instead of building sophisticated features into individual applications, you're building infrastructure that agents can leverage. APIs designed for agent consumption. Data pipelines that provide context efficiently. Security models that work when agents are the primary users.

The skills transfer. But the focus changes.

Policy Integration: The Hidden Complexity

Business policies used to live in application code. Approval rules. Access controls. Validation logic. Workflow definitions.

With agents, policies become first-class entities that agents interpret and apply.

This sounds simple. It's not.

Policies conflict. The finance policy says spend under $10,000 doesn't need CFO approval. The legal policy says contracts always need legal review regardless of amount. The IT policy says all software purchases need IT approval.

An agent handling a $5,000 software purchase needs to understand which policies apply, how they interact, and what the right workflow is.

Traditional systems hard-coded these conflicts. Agent-based systems need to resolve them dynamically.

This requires representing policies in ways agents can understand. Building conflict resolution logic. Maintaining audit trails of which policies applied to which decisions.

It's complex. But it's where the real value lives. Companies that get this right can adapt policies quickly without rebuilding applications. Companies that don't are stuck with rigid, brittle systems.

The Purchase Order Example Revisited

Let's walk through the purchase order example in detail to see how the logic layer works.

Traditional approach:

User logs into procurement system
Fills out purchase order form
System validates fields against rules coded in the application
System checks budget using logic built into the procurement system
System routes to approver based on hard-coded workflow
Approver reviews in the procurement system interface
System records approval in the procurement database

The logic is embedded in the procurement application. If business rules change, you modify the application code.

Agent-based approach:

User tells agent "I need to order 50 laptops for new hires"
Agent understands the request and determines requirements
Agent checks inventory system for existing laptops
Agent pulls pricing from vendor APIs
Agent verifies budget across financial systems
Agent understands company policies about IT purchases
Agent identifies required approvers based on amount and type
Agent generates purchase order in the system of record
Agent routes approval with full context to decision-makers
Agent tracks progress and handles escalations

The logic lives in the agent layer. The procurement system just stores purchase orders. Policy changes don't require application updates—just agent configuration changes.

This is the pattern: systems of record become dumb storage. Intelligence moves to the agent layer.

The Database Is Still Important

None of this means databases don't matter. They do.

But they matter differently.

Databases need to provide clean, consistent data that agents can access efficiently. They need clear schemas that agents can understand. They need APIs optimized for programmatic access at scale.

What databases don't need: sophisticated business logic baked into stored procedures. Complex application features built into database triggers. Elaborate workflows implemented in database code.

That logic is moving to the agent layer. The database's job is storing and serving data efficiently.

This actually simplifies database design. You optimize for agent access patterns, not human workflows. You focus on data integrity and performance, not business logic.

The Migration Challenge

Moving from application logic to agent logic isn't something you do all at once.

You have existing systems with business logic embedded throughout. Users know how to work within those systems. Everything functions.

Now agents are handling some workflows. But not all workflows. Some logic is in applications. Some is in agents. Keeping these consistent is hard.

The migration strategy that works: start with read-only agents that help users but don't make changes. Then move to agents that can execute simple, well-defined actions. Finally, move to agents that can handle complex decision-making.

At each stage, you're moving more logic from applications to agents. But you're doing it gradually in ways users can adapt to.

The companies that rush this break things. The companies that move too slowly get disrupted. The right pace is aggressive but disciplined.

What Developers Should Build Now

If you're a backend developer, here's where to focus:

APIs designed for agents. Not just REST endpoints, but APIs with rich metadata, clear semantics, and predictable error handling. Agents need to understand what your API does and what each field means.

Data pipelines for context. Agents need context from multiple systems. Build efficient ways to aggregate and serve that context without requiring agents to make dozens of API calls.

Policy engines. Build systems that represent policies as data that agents can interpret, not hard-coded rules that require deployments to change.

Audit infrastructure. When agents are making decisions, you need comprehensive audit trails. What data did the agent see? What rules did it apply? Why did it make that decision?

Testing frameworks for agent behavior. Traditional testing focuses on code correctness. Agent testing focuses on decision quality. Build frameworks that verify agents make good decisions, not just technically correct ones.

These are backend problems. But they're different from what most backend developers are building today.

The Competitive Timeline

This shift is happening now. Companies are already building agent-based architectures. They're operating with capabilities competitors can't match.

The gap grows every quarter. Not because agents suddenly got dramatically better. But because companies with agent-based architectures can move faster, adapt quicker, and operate at scales that application-centric companies can't match.

If your competitive advantage still lives in how well you've customized Salesforce or configured SAP, you're vulnerable. That advantage is eroding.

The new advantage is in how well your agents understand your business, access your data, and make decisions that drive outcomes.

Backend Isn't Dead

The provocative version: backend development as we knew it is dying.

The accurate version: backend development is evolving. The problems are different. The focus shifted. But the need for sophisticated technical work is as strong as ever.

Databases still need optimization. APIs still need design. Security still needs implementation. Performance still needs tuning.

But the context changed. You're not building features into applications anymore. You're building infrastructure that agents leverage.

That requires different thinking. Different architecture. Different skills.

The core backend disciplines transfer. But the application of those disciplines is new.

Systems of Record Are Infrastructure Now

The final implication of the logic layer shift: treat your systems of record as infrastructure, not competitive advantage.

You need good infrastructure. Reliable databases. Solid APIs. Clean data models. But you don't need unique infrastructure.

The differentiation lives in the agent layer. How well your agents understand your business. How effectively they coordinate across systems. How accurately they make decisions.

Companies still building competitive advantage into individual systems are making the same mistake companies that built competitive advantage into server infrastructure made fifteen years ago.

Infrastructure matters. But it's not where you win.

You win with the intelligence layer above the infrastructure. With agents that understand context, apply judgment, and drive outcomes.

That's where developers should focus their energy. That's where competitive advantage lives now.

The logic layer isn't coming. It's here. And it's redefining what backend development means.

Engineering Developer Experience: A Deep Dive into the Pawapay Node.js SDK Playground

Mon, 19 Jan 2026 19:24:30 +0000

In the fragmented landscape of African Fintech, "integration fatigue" is a tangible bottleneck. As developers, we often spend 70% of our time deciphering inconsistent documentation, handling HTTP timeouts, and normalizing error codes, and only 30% building the actual product.

When integrating Mobile Money API aggregators (like Pawapay) across borders, dealing with MTN in Uganda, Airtel in Zambia, and Vodafone in Tanzania, the complexity multiplies.

I recently set out to solve this friction for the Node.js community. My goal wasn't just to write a wrapper; it was to engineer a superior Developer Experience (DX).

The result is the launch of the Pawapay Node.js SDK and a Live Interactive Playground. This article is a surgical breakdown of the architecture, the code patterns, and the "why" behind the build.

1. The Problem: The "Raw HTTP" Trap

To understand the value of the SDK, we first have to look at what it replaces.

A typical "raw" integration in a Node.js Express app often looks like this "Spaghetti Code":

// THE OLD WAY: Vulnerable, Untyped, and Verbose
const axios = require('axios');

async function payWithMtn(phone, amount) {
  try {
    const response = await axios.post('https://api.pawapay.cloud/deposits', {
      depositId: 'txn_' + Math.random(),
      amount: amount,
      currency: 'GHS',
      payer: { type: 'MSISDN', address: phone },
      mobileMoneyProviderId: 'MTN'
    }, {
      headers: {
        'Authorization': `Bearer ${process.env.API_KEY}`,
        'Content-Type': 'application/json'
      }
    });
    return response.data;
  } catch (error) {
    console.log(error.response.data);
    throw new Error('Payment failed');
  }
}

Why this fails in production:

No Type Safety: Compiler won't warn you about invalid data types
Idempotency Issues: Math.random() for transaction IDs causes collisions
Ambiguous Errors: Generic error catching makes debugging impossible

2. The Solution: Structured Service Architecture

I architected the @katorymnd/pawapay-node-sdk with TypeScript as a first-class citizen. The philosophy: Catch errors in VS Code IDE, not in production logs.

Configuration Setup

 # PawaPay API Configuration
PAWAPAY_SANDBOX_API_TOKEN=your_sandbox_api_token_here
PAWAPAY_PRODUCTION_API_TOKEN=your_production_api_token_here
# Katorymnd SDK Licensing (Required for Production and sandbox)
KATORYMND_PAWAPAY_SDK_LICENSE_KEY=your_sdk_license_key_here
PAWAPAY_SDK_LICENSE_DOMAIN=your-licensed-domain.com
PAWAPAY_SDK_LICENSE_SECRET=your_sdk_license_secret_here

Let me show you how to build the complete service layer step by step.

Step 1: Foundation Setup with Logging

// Part 1: Foundation & Logging Setup
const path = require('path');
const winston = require('winston');
require('dotenv').config();

const { ApiClient, Helpers, FailureCodeHelper } = require('@katorymnd/pawapay-node-sdk');

const logsDir = path.resolve(__dirname, '../logs');

const logger = winston.createLogger({
    format: winston.format.combine(
        winston.format.timestamp(),
        winston.format.errors({ stack: true }),
        winston.format.json()
    ),
    transports: [
        new winston.transports.File({
            filename: path.join(logsDir, 'payment_success.log'),
            level: 'info'
        }),
        new winston.transports.File({
            filename: path.join(logsDir, 'payment_failed.log'),
            level: 'error'
        })
    ]
});

if (process.env.NODE_ENV !== 'production') {
    logger.add(new winston.transports.Console({
        format: winston.format.combine(
            winston.format.colorize(),
            winston.format.simple()
        )
    }));
}

Step 2: Service Class Initialization

// Part 2: Service Class Structure
class PawaPayService {
    constructor(config = {}) {
        const activeToken = process.env.PAWAPAY_SANDBOX_API_TOKEN;
        const maskedToken = activeToken ? `${activeToken.substring(0, 5)}...` : 'NONE';
        console.log(`[PawaPayService] Initializing with token: ${maskedToken}`);

        this.pawapay = new ApiClient({
            apiToken: activeToken,
            environment: 'sandbox',
            licenseKey: process.env.KATORYMND_PAWAPAY_SDK_LICENSE_KEY,
            sslVerify: false
        });
    }
}

Step 3: Core Deposit Implementation

// Part 3: Deposit Implementation
async deposit(depositData, apiVersion = 'v1') {
    const depositId = Helpers.generateUniqueId();

    const { amount, currency, mno, payerMsisdn, description, metadata = [] } = depositData;

    // Strict validation
    if (!amount || !mno || !payerMsisdn || !description || !currency) {
        logger.error('Validation failed - Missing required fields', depositData);
        return { success: false, error: 'Validation failed - Missing required fields' };
    }

    const amountRegex = /^\d+(\.\d{1,2})?$/;
    if (!amountRegex.test(amount) || parseFloat(amount) <= 0) {
        logger.error('Invalid amount format', { amount });
        return { success: false, error: 'Invalid amount' };
    }

    logger.info('Initiating deposit', { depositId, amount, currency, mno, apiVersion });

    let response;
    if (apiVersion === 'v2') {
        response = await this.pawapay.initiateDepositV2(
            depositId, amount, currency, payerMsisdn, mno, description, null, null, metadata
        );
    } else {
        response = await this.pawapay.initiateDeposit(
            depositId, amount, currency, mno, payerMsisdn, description, metadata
        );
    }

    if (response.status === 200 || response.status === 201) {
        logger.info('Deposit initiated successfully', { depositId });
        const statusCheck = await this.checkTransactionStatus(depositId, apiVersion);
        
        return {
            success: true,
            depositId,
            transactionId: depositId,
            status: statusCheck.status || 'SUBMITTED',
            message: 'Deposit initiated',
            rawResponse: response,
            statusCheck: statusCheck
        };
    } else {
        let errorMessage = 'Deposit initiation failed';
        let failureCode = 'UNKNOWN';

        if (response.response?.rejectionReason?.rejectionMessage) {
            errorMessage = response.response.rejectionReason.rejectionMessage;
        } else if (response.response?.failureReason?.failureCode) {
            failureCode = response.response.failureReason.failureCode;
            errorMessage = FailureCodeHelper.getFailureMessage(failureCode);
        } else if (response.response?.message) {
            errorMessage = response.response.message;
        }

        logger.error('Deposit initiation failed', { depositId, error: errorMessage });
        return { success: false, error: errorMessage, depositId, statusCode: response.status };
    }
}

Step 4: Universal Status Checking

// Part 4: Status Checking Implementation
async checkTransactionStatus(transactionId, apiVersion = 'v1', type = 'deposit') {
    let response;
    if (apiVersion === 'v2') {
        response = await this.pawapay.checkTransactionStatusV2(transactionId, type);
    } else {
        response = await this.pawapay.checkTransactionStatus(transactionId, type);
    }

    if (response.status === 200) {
        let data, status;
        if (apiVersion === 'v2') {
            if (response.response?.status !== 'FOUND') {
                return { success: true, status: 'PROCESSING', transactionId, message: 'Processing' };
            }
            data = response.response.data;
            status = data?.status || 'UNKNOWN';
        } else {
            const raw = response.response;
            data = Array.isArray(raw) ? raw[0] : raw;
            status = data?.status || 'UNKNOWN';
        }

        return { success: true, status: status, transactionId, data: data };
    } else {
        return { success: false, error: `Status check failed: ${response.status}` };
    }
}

Step 5: Payment Page Session Creation

// Part 5: Payment Page Implementation
async initiatePaymentPage(pageData, apiVersion = 'v1') {
    const depositId = Helpers.generateUniqueId();

    const { amount, currency, payerMsisdn, description, returnUrl, metadata = [], country = 'UGA', reason = 'Payment' } = pageData;

    if (!amount || !description || !currency || !returnUrl) {
        logger.error('Missing required fields for payment page', pageData);
        return { success: false, error: 'Missing required fields' };
    }

    const cleanMsisdn = payerMsisdn ? payerMsisdn.replace(/\D/g, '') : null;
    let response;

    if (apiVersion === 'v2') {
        response = await this.pawapay.createPaymentPageSessionV2({
            depositId, returnUrl, customerMessage: description,
            amountDetails: { amount: String(amount), currency },
            phoneNumber: cleanMsisdn, country, reason, metadata
        });
    } else {
        response = await this.pawapay.createPaymentPageSession({
            depositId, returnUrl, amount: String(amount), currency,
            msisdn: cleanMsisdn, statementDescription: description,
            country, reason, metadata
        });
    }

    if (response.status >= 200 && response.status < 300) {
        const redirectUrl = response.response?.redirectUrl || response.response?.url;
        logger.info('Payment Page created', { depositId });
        return { success: true, depositId, redirectUrl, message: 'Session created' };
    } else {
        const errorMsg = response.response?.message || 'Failed to create session';
        logger.error('Payment Page creation failed', { depositId, error: errorMsg });
        return { success: false, error: errorMsg, depositId };
    }
}

Step 6: Payout Implementation (Disbursements)

// Part 6: Payout Implementation
async payout(payoutData, apiVersion = 'v1') {
    const payoutId = Helpers.generateUniqueId();

    let { amount, currency, mno, provider, correspondent, recipientMsisdn, 
          description, statementDescription, customerMessage, reason, metadata = [] } = payoutData;

    const resolvedMno = mno || provider || correspondent;
    const resolvedDescription = description || statementDescription || customerMessage || reason || 'Transaction Processing';

    if (!amount || !resolvedMno || !recipientMsisdn || !resolvedDescription || !currency) {
        logger.error('Payout validation failed', { payoutId, ...payoutData });
        return { success: false, error: 'Missing required fields' };
    }

    const amountRegex = /^\d+(\.\d{1,2})?$/;
    if (!amountRegex.test(amount) || parseFloat(amount) <= 0) {
        logger.error('Invalid payout amount', { amount, payoutId });
        return { success: false, error: 'Invalid amount' };
    }

    logger.info('Initiating payout', { payoutId, amount, currency, mno: resolvedMno, apiVersion });

    let response;
    if (apiVersion === 'v2') {
        response = await this.pawapay.initiatePayoutV2(
            payoutId, amount, currency, recipientMsisdn, resolvedMno, resolvedDescription, metadata
        );
    } else {
        response = await this.pawapay.initiatePayout(
            payoutId, amount, currency, resolvedMno, recipientMsisdn, resolvedDescription, metadata
        );
    }

    if (response.status === 200 || response.status === 201 || response.status === 202) {
        const statusCheck = await this.checkTransactionStatus(payoutId, apiVersion);
        return {
            success: true,
            payoutId,
            transactionId: payoutId,
            status: statusCheck.status || 'SUBMITTED',
            message: 'Payout initiated',
            rawResponse: response
        };
    } else {
        let errorMessage = 'Payout initiation failed';
        if (response.response?.rejectionReason?.rejectionMessage) {
            errorMessage = response.response.rejectionReason.rejectionMessage;
        } else if (response.response?.failureReason?.failureCode) {
            errorMessage = FailureCodeHelper.getFailureMessage(response.response.failureReason.failureCode);
        } else if (response.response?.message) {
            errorMessage = response.response.message;
        }

        logger.error('Payout initiation failed', { payoutId, error: errorMessage });
        return { success: false, error: errorMessage, payoutId, statusCode: response.status };
    }
}

Step 7: Refund Implementation

// Part 7: Refund Implementation
async refund(refundData, apiVersion = 'v1') {
    const refundId = Helpers.generateUniqueId();

    const { depositId, amount, currency, reason, metadata = [] } = refundData;

    if (!depositId || !amount) {
        logger.error('Refund validation failed', refundData);
        return { success: false, error: 'Missing required fields' };
    }

    if (apiVersion === 'v2' && !currency) {
        logger.error('V2 Refund missing currency', refundData);
        return { success: false, error: 'Currency required for V2' };
    }

    const amountRegex = /^\d+(\.\d{1,2})?$/;
    if (!amountRegex.test(amount) || parseFloat(amount) <= 0) {
        logger.error('Invalid refund amount', { amount });
        return { success: false, error: 'Invalid amount' };
    }

    logger.info('Initiating refund', { refundId, depositId, amount, currency, apiVersion });

    let response;
    if (apiVersion === 'v2') {
        response = await this.pawapay.initiateRefundV2(refundId, depositId, amount, currency, metadata);
    } else {
        response = await this.pawapay.initiateRefund(refundId, depositId, amount, metadata);
    }

    if (response.status >= 200 && response.status < 300) {
        const statusCheck = await this.checkTransactionStatus(refundId, apiVersion, 'refund');
        return {
            success: true,
            refundId,
            transactionId: refundId,
            depositId,
            status: statusCheck.status || 'SUBMITTED',
            message: 'Refund initiated',
            rawResponse: response
        };
    } else {
        let errorMessage = 'Refund initiation failed';
        if (response.response?.rejectionReason?.rejectionMessage) {
            errorMessage = response.response.rejectionReason.rejectionMessage;
        } else if (response.response?.failureReason?.failureCode) {
            errorMessage = FailureCodeHelper.getFailureMessage(response.response.failureReason.failureCode);
        } else if (response.response?.message) {
            errorMessage = response.response.message;
        }

        logger.error('Refund initiation failed', { refundId, error: errorMessage });
        return { success: false, error: errorMessage, refundId, statusCode: response.status };
    }
}

Step 8: Export the Complete Service

// Part 8: Export the Service
module.exports = PawaPayService;

Complete File Assembly: Combine these 8 parts sequentially to create pawapayService.js. Each section handles specific functionality while maintaining clean separation of concerns.

3. Real-World Testing Implementation

//test-sdk-deposit.js
const path = require('path');
require('dotenv').config({ path: path.resolve(__dirname, '../.env') });

const PawaPayService = require('./pawapayService');

async function testSDKConnection() {
  console.log('\n========================================');
  console.log('PAWAPAY SDK DEPOSIT TEST');
  console.log('========================================\n');

  const service = new PawaPayService();
  const commonData = {
    amount: '1000',
    currency: 'UGX',
    mno: 'MTN_MOMO_UGA',
    payerMsisdn: '256783456789',
    description: 'SDK Integration Test'
  };

  // Test V1 Deposit
  console.log('\nStep 1: Testing V1 Deposit...');
  const v1Result = await service.deposit({
    ...commonData,
    description: 'V1 Test Payment'
  }, 'v1');

  if (v1Result.success) {
    console.log('V1 Initiation Success:', {
      depositId: v1Result.depositId,
      status: v1Result.status
    });
    
    console.log('Waiting 5 seconds for V1 propagation...');
    await new Promise(r => setTimeout(r, 5000));
    
    console.log('Checking V1 Status...');
    const statusCheck = await service.checkTransactionStatus(
      v1Result.depositId,
      'v1',
      'deposit'
    );
    
    console.log(`Final V1 Status: [ ${statusCheck.status} ]`);
  }

  // Test V2 Deposit
  console.log('\nStep 2: Testing V2 Deposit...');
  const v2Result = await service.deposit({
    ...commonData,
    description: 'V2 Test Payment',
    metadata: [
      { orderId: "ORD-SDK-TEST" },
      { customerId: "*Emails are not allowed*", isPII: true }
    ]
  }, 'v2');

  if (v2Result.success) {
    console.log('V2 Initiation Success:', {
      depositId: v2Result.depositId,
      status: v2Result.status
    });
    
    console.log('Waiting 5 seconds for V2 propagation...');
    await new Promise(r => setTimeout(r, 5000));
    
    console.log('Checking V2 Status...');
    const statusCheck = await service.checkTransactionStatus(
      v2Result.depositId,
      'v2',
      'deposit'
    );
    
    console.log(`Final V2 Status: [ ${statusCheck.status} ]`);
  }

  console.log('\nSDK Testing Complete');
  process.exit(0);
}

testSDKConnection();

4. The Critical Component: Webhook Security

Payments are asynchronous. The SDK provides utilities to type-check incoming payloads, ensuring your server processes only valid data from authenticated sources.

5. Visualizing the Invisible: The Live Playground

Documentation is static. APIs are dynamic. The Pawapay Live Playground at katorymnd.dev solves "Parameter Anxiety" with:

Direct Sandbox Connection: Real API calls with CORS handling
Schema Validation: Real-time input validation using the same Zod schemas as the SDK
JSON Export: Copy-paste successful payloads directly into your unit tests

6. Why This Matters for Your Stack

For Node.js backends (NestJS, Express, Fastify) or Serverless stacks (AWS Lambda, Vercel Functions):

70% Code Reduction: Replace ~200 lines of axios config with 3 lines of SDK instantiation
Zero Maintenance: API updates handled via npm update
Production Ready: Pre-configured compliance headers and timestamp formats

Call to Action

This project embodies my software engineering philosophy: Complex logic should be invisible to code consumers. The heavy lifting happens inside the package, leaving your implementation layer clean and maintainable.

Repository: https://github.com/katorymnd/pawapay-node-sdk
Live Playground: https://katorymnd.dev/pawapay-demo/nodejs/
NPM Package: npm i @katorymnd/pawapay-node-sdk

Let's build better payment flows for Africa.

The Software Architect’s Hidden Skill:

Sat, 17 Jan 2026 22:44:08 +0000

Most of us were trained to believe technology is a deterministic system, but the higher you climb in architecture, the more you realize systems are made of people.

If your technically superior solutions are hitting a wall of resistance, you aren't facing a logic problem—you’re facing a political one.

In this first episode of Coda est Poesis (Podcast), I bridge the gap between political science and software architecture to help you navigate the "S-word" Politics with integrity.

What’s inside:

Moving from distributive to integrative bargaining.
Why the status quo is the enemy of architecture.
How to charge your "Trust Battery" with stakeholders.

Listen to Episode #1: The Software Architect’s Hidden Skill

This episode is based on my last article: Understanding and Navigating Politics for Software Development

Actor Models in Pygame: Lessons in Serialization and State Isolation

Wed, 31 Dec 2025 03:53:15 +0000

Our attempt to build a prototype recreating the matrix rain effect last week forced us to revise the mini Pygame framework. The implementation of the idea yielded a mixed result that I previously overlooked. No, it wasn’t due to my take on the Actor model, in case you’re wondering. What could the problem be then?

Cute doge on an adventure. Image courtesy of Microsoft Copilot.

The Vision: Searching for Order in the Matrix

So what is this Actor model thing, you may wonder. Having originated from our experiment with Pygame through building a tic-tac-toe game, the framework was a “side-product” of the project. Extracting the framework out through refactoring was done in the hope of making it portable and applicable for other projects. It worked, as shown in the previous article, but it wasn’t perfect. We identified some problems too, namely inefficiency in the management of application state and the event dispatch lookups.

I can already hear criticism from afar saying the latter is just a design failure, but since we are fixing the data structure, why not just fix both in one go?

Jokes aside, let’s start by looking into the Actor model. Skipping a formal definition of the term, we let the code be a reference to the discussion. For the scope of this article, the model is merely something that looks like this sample code returned by Gemini.

@dataclass
class UpdateState:
    """Message to change state (Write)"""
    key: str
    value: Any

@dataclass
class GetState:
    """Message to request state (Read)"""
    key: str
    reply_to: asyncio.Future # Used to send the result back

async def producer(inbox: asyncio.Queue):
    """Simulates a system event updating state (Write)"""
    await asyncio.sleep(1)
    print("[Producer] Sending update: score = 100")
    await inbox.put(UpdateState("score", 100))

async def consumer(inbox: asyncio.Queue):
    """Simulates a UI element needing to read state (Read)"""
    await asyncio.sleep(2)
    print("[Consumer] I need the 'score'...")
    
    # Create a 'return envelope'
    future = asyncio.get_event_loop().create_future()
    await inbox.put(GetState("score", future))
    
    # Wait for the Actor to fill the envelope
    result = await future
    print(f"[Consumer] Received score: {result}")

A loop is running in the background reading from a queue (denoted by the inbox argument in the coroutine functions) expecting one of the two dataclasses above, each representing a type of request. When it is an UpdateState, the value is changed. Otherwise, the value is returned via the asyncio.Future. Some disassembly was done to the model to keep it simple, though essentially we now can restrict access to the state only through the mailbox (or inbox in the snippet).

On the other hand, fixing the event dispatch problem merely involves a slight redesign of the application state. Ideally, we should aim for a near O(1) performance, let’s just use a hashmap for the purpose, preferably a frozendict(). Roughly speaking, we want the application state that stores events handlers as shown below:

@dataclass(frozen=True)
class Application:
    ...
    events: frozendict[tuple[Element, int], EventHandler]

We are storing the handlers into a frozendict to stick with functional principles as much as possible. Immutability is the key to writing predictable and idempotent functions.

Another advantage of adopting the Actor model is it opens up a possibility of clean separation from other components. When there is a possibility of overdoing things, you may already know by now I am always there. My attempt to overengineer for the sake of learning eventually caused a major rewrite, though fortunately the implementation still centers around what we have discussed so far. Sit tight, as we are diving deep into an adventure full of misjudgments.

Trekking Inward: Drafting the Actor Blueprint

Doesn’t sound like an adventure so far?

Think of a request trekking into an application via a queue. The previous section would be like a roughly sketched overview map. Now is the time to join a relatively steady stream of requests. Looking at our map, we see some landmarks: a scheduler, a mailbox (basically a queue), some processing logic, and lastly a mechanism to reply. Implementing this in the context of our little Pygame framework is the goal we are trying to achieve here. There is some flexibility I wish to implement, specifically regarding how the request affects the application state.

Let’s start by requests.

An actor only responds to requests, and the outcome would be different depending on the types. Requests are then passed into an actor through a mailbox, in our case, a queue. Everything seems perfectly fine, until you see a small and almost illegible fine print in the map — Pygame is generally not thread safe, i.e. we run all Pygame code within the same thread. That means we need to be really aware when passing Pygame objects through queues, for instance, passing them through a multiprocessing Queue will end up a crash due to serialization failure.

Each request represents an action we want to do with the data, either a retrieval or a mutation. We start by defining a mutation request, the SetRequest:

@dataclass
class Operation:
    op: Callable
    args: tuple = field(default_factory=tuple)
    kwargs: dict = field(default_factory=dict)

@dataclass
class SetRequest[Data]:
    operation: Operation

    async def __call__ (self, data: Data) -> Data:
        return await self.operation.op(
            data, *self.operation.args, **self.operation.kwargs
        )

Instead of consolidating all the application state changes within one huge monolithic match-case statement, SetRequest delegates the mutation logic definition to the caller. Unfortunately, this also means the caller needs to be aware of the state’s internal structure. Despite that downside, the delegation should promote the caller to do a small, localized change to the state that holds multiple pieces of data.

On the other hand, there are cases where we want the current state. Retrieving multiple slivers of current state one at a time to complete a task can be rather inefficient as our requests could be stuck queuing up when traffic is busy, and that makes me think. The reason we are making the queries is to get things done; what if we bundle the processing logic over, like we did with SetRequest too? Rather than repeating the “fill in the blanks” exercise once for each sentence, we are doing the whole essay all at once. In short, bundling the retrieval requests into one execution request results in smaller queues. For that goal, we implement ExecuteRequest as such:

@dataclass
class ExecuteReply:
    tag: UUID
    payload: Any

@dataclass
class ExecuteReplyTo:
    tag: UUID
    mailbox: Queue[ExecuteReply]

@dataclass
class ExecuteRequest[Data]:
    operation: Operation
    reply_to: ExecuteReplyTo | None = None

    async def __call__ (self, data: Data) -> None:
        result = asyncio.create_task(
            self.operation.op(data, *(self.operation.args), **(self.operation.kwargs))
        )

        if self.reply_to:
            asyncio.create_task(
                asyncio.to_thread(
                    self.reply_to.mailbox.put,
                    ExecuteReply(self.reply_to.tag, await result),
                )
            )

In most cases, we do not expect the caller to care when the request is completed. What’s wrong with having an asyncio.Future like we saw earlier? The short answer is no, but I would like to defer this until we are discussing the overengineering part. Yet, if the caller cares about the reply, the request requires a recipient mailbox, and an identifier UUID for the recipient.

A scheduler is then built to bind everything together. It takes two arguments, namely the application state that it manages, and a mailbox that acts as the single point of entry for the state. Refer to the snippet below for a simplified implementation:

async def state_scheduler[Data](state: [Data], mailbox: Queue[SetRequest[Data] | ExecuteRequest[Data]]):
    result = state
    while True:
        try:
            match request := await asyncio.to_thread(mailbox.get, timeout=0.001):
                case SetRequest():
                    result = await request(result)

                case ExecuteRequest():
                    asyncio.create_task(request(result))

        except Empty:
            pass

Some may find the snippet resembles a reduce function that loops infinitely. On each iteration, the result should only get replaced if a SetRequest is received. Imagine if the state isn’t immutable, even the execute request would have a chance of breaking the integrity of the object. Both SetRequest and ExecuteRequest are also expected to implement the __call__ dunder as shown in respective snippet. Remember the metaphor on bundling the “fill in the blanks” exercise into a callable when we discussed ExecuteRequest? It gets scheduled for execution here.

The Versatility of __call__: A Python Developer's Secret Weapon

The Bumpy Ride: Overengineering the Parallel Dream

“If it’s worth doing, it’s worth overdoing”

I didn’t watch a lot of MythBusters growing up, but this motto has a special place in my heart. Hang on tight, this is also where our relaxing boat ride gets a little bumpy. The Actor model offers a possibility to cleanly separate the state away from the rest of the application. Seems like a good opportunity to further split the application to multiple processes right? The application state actor, display update loop, as well as the event dispatch loop will all run independently, right?

Let’s overengineer this project, because more processes in parallel means faster execution, right?

Running each component in a separate process is done through the usual ProcessPoolExecutor setup. Further discussion on the topic can be found in the AsyncIO article series and hence we are skipping the details here. Why dedicate a section for the topic then?

Remember we deferred the discussion of having an asyncio.Future for replies? Turns out AsyncIO has the same constraint as Pygame — the objects should stay within the thread. As we know, objects get serialized when they are sent through a cross-process queue. Recreating an asyncio.Future in another process will raise an exception, causing the execution to fail.

For the same reason, we are also avoiding sending Pygame objects through cross-process queues.

You might notice that ExecuteReplyTo takes a tag UUID and mailbox. According to the dataclass definition we know mailbox is another cross-process queue, but what does the tag identifier do?

It is a key to our registry of asyncio.Future that lives only in the caller’s process, as shown below:

async def inbox_consume(
    inbox: Queue[ExecuteReply], futures: dict[UUID, asyncio.Future]
) -> None:
    with suppress(Empty):
        reply = await asyncio.to_thread(inbox.get_nowait)

        if futures[reply.tag]:
            futures[reply.tag].set_result(reply.payload)

Each process maintains its own registry and a cross-process queue. A loop is set up to run inbox_consume repeatedly such that the corresponding asyncio.Future objects get notified as soon as the reply arrives.

As an example, this is how it works in action:

async def some_callable(state) -> bool:
    ...

    return True

async def caller(...):
    done_future = uuid4()
    futures[done_future] = asyncio.Future()
    game_in.put(
        ExecuteRequest(
            Operation(some_callable),
            ExecuteReplyTo(done_future, futures_mailbox),
        ),
    )
    await futures[done_future]

Here’s a brief explanation to the snippet above:

All callables defined for SetRequest and ExecuteRequest receive the current state as the first argument
We register a new asyncio.Future to a UUID tag before firing the ExecuteRequest.
The tag and the future_mailbox to update the registry of asyncio.Future are sent as a ExecuteReplyTo object.
The bundled some_callable runs in the process where the state lives.
Back in the caller’s process, since the ExecuteRequest expects a return value, we perform a blocking await at the corresponding entry in the registry of asyncio.Future.

Some Pygame objects are stored within the application state object, so it runs in the process that calls the Pygame API. While it seems like we managed to split the application into three separate processes, if we are delegating all requests back to the process that manages Pygame anyway, is the added complexity worth it?

Silently screaming at the low FPS

Grab onto something quick, as we are heading towards a waterfall.

The Waterfall: A 30 FPS Reality Check

Though there were hiccups, the code eventually ran. For a while I was quite proud of myself for getting everything assembled properly. I still remember the realization came about an hour later when I was staring at the code on pushing visual changes to the application window. That was the fall off a cliff moment. So what went wrong? The separation into multiple processes was not such a good idea eh?

Photo by Aditya Chinchure on Unsplash

Pygame.time.Clock has a method called .tick(). It is meant to be called once per frame. If a framerate is specified, it would block until time is up to prevent premature frame update. That practically means the callable sent by the process managing the display is going to be blocking the process hosting the scheduler for the Actor (which also holds the application state containing Pygame objects). Considering this callable also gets sent repeatedly in an infinite loop, it means the Actor process is constantly blocking, leaving little room for concurrency.

Another bottleneck that I underestimated is how the serialization works through the cross-process queue could potentially slow things down. If I was falling when the realization of the blocking operation happened, recognizing this bottleneck is when my boat and I plunged into the water down the cliff. The struggle to get back to the surface almost became measurable, when I started to compare the frame per second figures.

Instead of hitting the 60fps target, we are only seeing frame rate in the 30s. It is obvious that the added complexity that involves cross-process message passing is the culprit. Measuring is going to be hard, but I would imagine the constant conversion between an actual project and serialized form would contribute to the delay.

We are now stuck in the middle of nowhere; the expedition failed.

What’s next?

The Principled Pivot: Painting with Experience

We failed the mission to cleanly separate the components into independent processes. Despite that, I am still happy throughout the adventure. The expedition pushed me far enough from my comfort zone and all the insights gained were valuable. Failing is never a problem, as long as we are learning something through it.

Like many fellow engineers, I do have a fair share of failures when I write code. Every new project I work on is not unlike an expedition into the unknown. While the outcome is important, we need to be aware of every step we take too. If experience is a painting, the trail of foot steps is exactly how lines are formed in an image.

Photo by Andrey Novik on Unsplash

So what do we learn in this in-progress painting of our Pygame experiment?

Looking at the bright side, I really liked how the application state is now managed through the Actor model. After removing the multi-process setup, certain parts would be greatly simplified. Overall, it is much more manageable than my poorly implemented model. Another revision is definitely happening and hopefully the fps count recovers.

Lastly, thanks for another year of support, and I shall write again soon. Bonne Année.

Finally, I’d like to acknowledge the editorial and research support provided by Gemini in producing this write-up. While the architecture and the hard-learned lessons are mine, Gemini acted as a second set of eyes — helping me audit the sentence-to-sentence flow, cross-referencing patterns like the Active Object and the “Ask” pattern, and ensuring that my “expedition” through the code was as clear to you as it was bumpy for me. It’s been a collaborative effort to ensure that even my failures are presented with the technical rigor they deserve.

The Sentient Primitive: Waking Up Your Data with FSMs

Sat, 29 Nov 2025 07:09:59 +0000

In standard game development, we are taught that variables are silent containers. A bool is just a box that holds true or false. It doesn't do anything. To know if it changed, we have to pester it constantly:

// The Old Way: The Polling Trap
void Update() {
    if (isDead == true) { // Asking every single frame...
        PlayDeathAnimation();
    }
}

This is the "Silent Variable" problem. It leads to spaghetti code, race conditions, and Update() loops clogged with thousands of if statements checking for things that haven't changed.

But what if a Boolean wasn't just a box? What if it was a Finite State Machine?

1. The FSM_Bool: The Atomic Unit of Behavior

By wrapping a primitive in an FSM, we transform it from Passive Data into Active Behavior. It doesn't just hold a value; it has a lifecycle.

Enter True: Fire the event (e.g., "Turn on Light").
Update True: Maintain state (e.g., "Humming sound").
Exit True: Cleanup (e.g., "Fade out light").

Suddenly, your logic is decoupled. The code controlling the light doesn't need to check the switch. The switch tells the light what to do.

// The New Way: Reactive Primitives
var lightSwitch = new FSM_Boolean(
    name: "LivingRoomLight",
    startValue: false, 
    onTrue: (ctx) => LightFixture.TurnOn(),  // Fire and forget
    onFalse: (ctx) => LightFixture.TurnOff() // Fire and forget
);

// To operate it, we just change the target. The FSM handles the logic.
lightSwitch.Set(true);

2. Evolution: The FSM_FloatLimit

The concept explodes in potential when we move beyond Booleans. Consider the float. Usually, we use floats for Health, Stamina, or Mana.

In the old world, a Health System is a messy manager class. In the FSM world, a Health System is just a single Float with a Threshold.

Introducing the FSM_FloatLimit. It is an FSM that monitors a value and triggers a transition when a threshold is crossed.

The Constructor:

var healthSystem = new FSM_FloatLimit(
    currentValue: 100.0f, 
    threshold: 0.0f, 
    OnThreshold: (ctx) => Player.Kill()
);

The Internal Logic (Abstracted):
Inside the black box, this FSM has two states: Safe and Breached.

State "Safe": The value is above the threshold.
Transition: If Value <= Threshold -> Go to Breached.
State "Breached": Execute OnThreshold.

You no longer need a PlayerHealthManager.cs. You just need a smart float.

3. Complex Data Shapes from Simple Primitives

When you start composing these "Smart Primitives" together, you aren't writing code anymore; you are sculpting Logic Shapes.

Imagine a Weapon Reload System constructed entirely of primitives:

AmmoCount (FSM_IntLimit): Triggers "Empty" when it hits 0.
ReloadTimer (FSM_FloatTimer): Triggers "Complete" after 2.0 seconds.
IsReloading (FSM_Bool): The master switch.

The architecture becomes a chain reaction:

AmmoCount hits 0 -> Sets IsReloading to True.
IsReloading (True) -> Starts ReloadTimer.
ReloadTimer (Complete) -> Refills AmmoCount and sets IsReloading to False.

Beyond Numbers: The Reactive String

The power of the "Sentient Primitive" isn't limited to simple math. It transforms how we handle text and input.

Consider the standard way we handle cheat codes or spell casting in games: we log every keystroke into a buffer and run a string comparison every frame. It is tedious and error-prone.

In our FSM-driven world, we treat the String as a Stateful Pattern Matcher.

The FSM_Regex

Imagine an FSM initialized with a Regular Expression. It doesn't just "hold" a string; it hunts for a pattern within the stream of data you feed it.

// Defining a "Spell" using a Reactive String
var fireballSpell = new FSM_Regex(
    pattern: "UP, DOWN, A, B", // The Konami Code of magic
    OnMatch: (ctx) => CastFireball(),
    OnFail: (ctx) => PlayFizzleSound()
);

// We just feed it inputs. The FSM tracks the sequence state internally.
fireballSpell.Input("UP");   // State: Tracking...
fireballSpell.Input("DOWN"); // State: Tracking...
fireballSpell.Input("X");    // State: Failed -> Trigger OnFail -> Reset

This turns complex input handling into a simple definition. You aren't writing an "Input Manager"; you are just defining the shape of a valid command.

The Paradigm Shift: Composition over Management

When you adopt this mindset, the "Manager Classes" that plague modern software development start to disappear.

You don't need a HealthManager; you have an FSM_Float.
You don't need an InputComboManager; you have an FSM_Regex.
You don't need a DayNightCycleManager; you have an FSM_Timer.

You are creating a visual language where the "bricks" of your building aren't dead clay; they are active participants in the architecture. They know when to hold together, and they know exactly when to break apart.

Visualizing the FSM_Regex

Resources & Code:

The FSM Package (Unity Asset Store):
https://assetstore.unity.com/packages/slug/332450

NuGet Package (Non-Unity Core):
https://www.nuget.org/packages/TheSingularityWorkshop.FSM_API

GitHub Repository:
https://github.com/TrentBest/FSM_API

Support Our Work:

Patreon Page:
https://www.patreon.com/c/TheSingularityWorkshop

Support Us (PayPal Donation):
https://www.paypal.com/donate/?hosted_button_id=3Z7263LCQMV9J

We'd love to hear your thoughts! Please Like this post, Love the code, and Share your feedback in the comments.

Rethinking Microfrontend Architecture combining Still.js with React (or other Framewok) - Arch Serie

Sun, 13 Jul 2025 21:35:39 +0000

Why Microfrontend

Building and modernizing large, and/or legacy enterprise level applications is often challenging, as it requires rewriting significant portions or even the entire application when moving to newer technology versions. Microfrontends offer a practical solution to this problem, much like microservices do for backend. They're not just a buzzword, but a meaningful architectural approach for handling complexity and scalability. Still.js enhances this by providing powerful and flexible means for building microfrontends, with seamless integration into frameworks like React.

What problems and use cases can be addressed with Microfrontend?

Application decoupling;
Progressive migration from old to new version (e.g. migrate
framework from v2 to v15);
Fast implementation of new feature without disrupting the existing
ecosystem;

Fast Modernization of old/legacy application yet keeping things
stable;
and more...

How Still.js (with React or others) for Microfrontend?

Still.js is an open source modern, complete and full-featured (with known capabilities and more) frontend framework built with vanilla JavaScript, designed for easy and scalable development and integration like embedding its app inside React apps in form of components. Bellow is a high level architecture of the integration we'll go through.

Quite a lot can be done in terms of Microfrontend with Still.js, however in this article we're going to focus in introductions aspects, there is also a video on youtube with a hands on tutorial.

Integrating with React (using Vite to generate the project)

Step 1. Create your React App, using vite (follow React official documentation using Vite).

npm create vite@latest my-app -- --template react

Step 2. Install React app dependencies.

npm i

Step 3. In your React app, install the Still.js app loader from @stilljs/apploader (on npm), this is a regular (not dev) dependency.

npm i @stilljs/apploader

Step 4. Loading Still.js application inside React is done when the React component is mounted, we'll use the App.js(jsx,tsx,ts), which is the default React entry component.

import { useEffect } from 'react';
import './App.css';
import { RegularReactComponent } from "./components/RegularReactComponent";
import { StillAppLoader } from '@stilljs/apploader';

function App() {

  useEffect(() => {

    // Creating App loader instance
    const stillApp = new StillAppLoader();
    stillApp
     .cdn({ env: { STILL_HOME: 'public/micros/stillapp1/' }   })
     .load();

    // Unload the App when React component gets unmounted
    return () => stillApp.unload();
  },[]);

  return (
    <>
      <RegularReactComponent></RegularReactComponent>
      React validating local Still component embeding
    <>
  )
}

export default App

Step 5. Creating the Still.js App

React project generated with Vite provides a folder named public/ right on the root along with src/ and others, there (public) we generate our Still.js app.

Still CLI provides the still lone command to create a Microfrontend/Lone project, providing it with very minimal setup, having the CLI installed, type in your terminal. (follow the link to a short video):

still lone

Step 6. Create a Still.js lone component in the CDN approach

It’s called “lone” because Still.js isn’t installed locally, just the component and routes file are there and it behaves the same. With CLI installed type the bellow in terminal. (follow the link to a short video):

still c cp app/components/TestComponent --lone

Step 7. Embed Still.js component inside React app/component
We'll change the return portion and add the Still component as follow:

  return (
    <>
      <RegularReactComponent></RegularReactComponent>
      React validating local Still component embeding
      <st-element component="TestComponent"></st-element>
    <>
  )

Step 8. Running the React app:

npm run dev

Final result

Intercommunication between React and Still.js

Communication is feasible between React and Still.js once App/component is embeded, for that, we use the AppLoader as Proxy. but for the sake of size of this article we're not depicting it here, instead you can see this tutorial on Youtube which an implementation is done in practice step by step.

To wrap up

Microfrontends aren’t a one-size-fits-all solution, but they’re ideal for highly modular and fast-evolving applications, such as modernizing legacy systems. Still.js enhances this flexibility by being framework-agnostic and built entirely in vanilla JavaScript, making it easy to adapt to any scenario, even when traditional frameworks fall short.

Give it a try and leverage the full potential of this architectural approach.

See you next time!

My Pygame Evolution: Embracing Asyncio and Immutability for Scalable Design

Tue, 24 Jun 2025 19:00:07 +0000

Maintaining a blog is a lot of work, especially when it comes to finding new ideas. Over the weekend, I explored Pygame by building a simple game. I was very humbled by the experience throughout the build. Aha, this exploration turned out to be a worthy idea for a new article! Read on to relive the struggles involved while building a graphical game via Pygame together.

A cute illustration on the topic, by Microsoft Copilot

The Tic-Tac-Toe Catalyst: Unveiling Pygame’s Low-Level Nature

Over the weekend, in the EngineersMY chatroom, a friend shared his work in a recent game jam held in his workplace. He built the project with p5.js, piqued my interest. Eventually I decided to proceed with Pygame, and that decision led me to hours of exploration and perhaps the expense of a few hair follicles. Despite that, the lower-level library nudged me to build a custom architectural design that can potentially evolve into a proper framework.

Photo by Ryan Snaadt on Unsplash

Discussions about my friend’s project reminded me of a less than ideal job application that happened around late last year. The offered package was unattractive, and I was expected to work on graphical animation alongside full-stack development. My submitted video processing pipeline work received satisfactory feedback, though the hiring manager came back with another assignment. He requested a new task: to build a graphical application with p5.js, and it was also when I formally ended the application. That mention of p5.js was the first time in a formal setting.

Interestingly, I was still given a job offer despite all that, which I ultimately rejected.

Fast forward to the weekend, I consulted with my AI assistant and started my research about Processing, a project where p5.js was based on. Apparently there are ports for Python, though the official processing.py only works with Jython. All other alternatives either relied on the Java package or required special libraries. Consequently, I stumbled upon Pygame after a series of aimless searches.

Unlike Processing, Pygame felt like a lower level library for building graphical applications. Partially inspired by the discussion on the game jam, but mostly out of boredom, I decided to give it a try. The realization of how low-level Pygame is came when I was figuring out the specification and scope of my over-ambitious initial plan. At the time, my main loop was just barely running.

As a project to explore graphical application building, I eventually decided to limit the scope further to start small. Tic-tac-toe seemed like a good idea, and I should be able to build a reasonably functional version over a weekend. How wrong I was, as I am still doing some last minute changes as of writing this article…

Building a Reactive Core: Event-Driven Design in Pygame

Pygame is synchronous, and most of the objects and operations are not thread-safe. Due to the interactive nature of graphical applications, they often require a carefully planned strategy to achieve concurrency. AsyncIO is an obvious candidate for the task, though getting it to work with Pygame, requires some thought. Much of the implementation takes a cue from Javascript, especially in the event dispatching department. Immutability of certain components is also crucial, though mainly to keep me sane.

Let’s start with the main loop, essentially we want something like this

def run() -> None:
    pygame.init()

    continue_to_run = True
    clock = pygame.time.Clock()

    pygame.display.set_mode((300, 300))

    while continue_to_run:
        clock.tick(60)

        continue_to_run = event_process(pygame.event.get())

        pygame.display.flip()

    pygame.quit()

A potential problem I see with this arrangement is that, both handle_events and pygame.display.flip() are inevitably blocking. If any of them stalled for a prolonged period, we will miss the target frame rate. Ideally, we want something like this:

async def run():
    pygame.init()

    exit_event = asyncio.Event()

    pygame.display.set_mode((300, 300))

    asyncio.create_task(display_update())
    asyncio.create_task(event_process(exit_event))

    await exit_event.wait()

    pygame.quit()

Photo by Acton Crawford on Unsplash

A window popped up, and my event handling work seemed functional. Yet, sometimes when the application exits I get a SIGSEGV segmentation fault when exiting the program. After some research on the error, I found out that Pygame is not thread-safe, hence I should really avoid asyncio.to_thread. This means the following code, delegating the blocking display.flip() call to another thread, is invalid:

async def display_update():
    clock = pygame.time.Clock()

    while True:
        clock.tick(60)

        await asyncio.to_thread(pygame.display.flip)

With that discovery, I settled with the original main loop convention for a while, and only moved the event dispatching work to a coroutine.

async def run() -> None:
    pygame.init()

    clock = pygame.time.Clock()
    exit_event = asyncio.Event()

    pygame.display.set_mode((300, 300))

    while not exit_event.is_set():
        clock.tick(60)

        asyncio.create_task(event_process(pygame.event.get(), exit_event))

        pygame.display.flip()

    pygame.quit()

Unfortunately, due to the synchronous blocking nature of Pygame, the event_process tasks do not have a chance to run, though scheduled successfully. As my coding companion, the chatbot suggested adding asyncio.sleep(0) into the loop. According to the documentation:

Setting the delay to 0 provides an optimized path to allow other tasks to run. This can be used by long-running functions to avoid blocking the event loop for the full duration of the function call.

Adding the statement to the end of the loop allows the application to finally respond to events.

Previously, I mentioned I performed a last-minute update while writing this article. With this knowledge, we can apply the asyncio.sleep(0) fix to both display_update and event_process, enabling AsyncIO to run the scheduled tasks. In turn, this allows the refactoring of the huge main loop shown in the first snippet into the second snippet.

Photo by Scott Rodgerson on Unsplash

Before diving into the event dispatching part of the design, let’s talk about how I manage shared data. Certain components need to be shared across different parts of the application. Graphical applications are extremely interactive, hence I would need to respond to them concurrently. In this case, interactivity poses a new challenge — keeping track of changes of these shared objects.

Much of the design came from my experience with development work in JavaScript. Within a JavaScript application, it is possible to dispatch and subscribe to events almost everywhere, and changes can also be made to a page whenever we see fit. Keeping a reference while staying aware of these changes is a nightmare, and it is extremely unfriendly to my hair follicle.

Grouping them together into one dataclass, which serves as a registry is usually my first step. In this case, I prefer them to stay immutable, as shown below:

@dataclass(frozen=True)
class Application:
    screen: pygame.Surface = pygame.Surface((0, 0))
    elements: tuple[Element, ...] = tuple()
    state: GameState = GameState.INIT

The three objects consist of a pygame.Surface object for the main application window, an element tuple holding all the interactive widgets in the surface, and an Enum holding the current game state. Events are dispatched to be executed concurrently, so each execution is completely isolated from one another. Therefore, we need to synchronize the changes made to these shared objects to avoid race conditions.

Aiming to maintain the simplicity inspired by functional programs, I started by marking the dataclass as frozen, as shown in the snippet. Changes to the elements and state objects are deferred to two queues I introduced to the registry dataclass.

@dataclass(frozen=True)
class Application:
    ...

    state_update: asyncio.Queue[GameState] = asyncio.Queue()
    element_delta: asyncio.Queue[DeltaOperation] = asyncio.Queue()

Making the dataclass immutable would require a regeneration of application registry for every batch of events.

async def events_process(application: Application):
    while True:
        application = await application_refresh(application)

        await events_dispatch(
            application,
            pygame.event.get(),
        )
        
        await asyncio.sleep(0)

Whenever an event handler makes a change to the game state or manipulates the widgets on screen, the changes are pushed to the queue instead. Then the queue is consumed, to update the corresponding object:

from dataclasses import replace

async def application_refresh(application: Application) -> Application:
    with suppress(asyncio.queues.QueueEmpty):
        elements = application.elements

        while delta := application.element_delta.get_nowait():
            match delta:
                case ElementAdd():
                    elements += (delta.item,)

                case ElementUpdate():
                    elements = tuple(
                        delta.new if element == delta.item else element
                        for element in elements
                    )

                case ElementDelete():
                    elements = tuple(
                        element for element in elements if not element == delta.item
                    )

    with suppress(asyncio.queues.QueueEmpty):
        state = application.state

        while state_new := application.state_update.get_nowait():
            state = state_new

    return replace(
        application,
        elements=elements,
        state=state
    )

Wrapping the widget element with an ElementAdd, an ElementUpdate and an ElementDelete is just a way to provide metadata to indicate the manipulation type. On the other hand, the replace function, provided by the dataclasses module, is just a helpful shorthand in this case. Alternatively, we can recreate the Application object as follows:

    return Application(
        application.screen,
        elements,
        state,
        application.state_update,
        application.element_delta
    )

Moving on to the event dispatching part.

Due to my inexperience, I didn’t spent too much time drafting a design or a plan when I worked on the game. Still, as I went deeper I realized I really needed to start structuring the event handling code. Keeping that in mind, I continued with the build until I managed to get a bare minimum version running.

Drawing inspiration from JavaScript, I started my refactoring by revising the event handler registration. Directly inspired by the document.addEventListener method, my interpretation of it in this setting took this form:

async def add_event_listener(target: Eventable, kind: int, handler: Callable[..., None]) -> Eventable:
    event = Event(kind, handler)
    result = target

    match target:
        case Application():
            await target.event_delta.put(DeltaAdd(event))

        case Element():
            result = replace(target, events=target.events + (event,))

        case _:
            raise Exception("Unhandled event registration")

    return result

Besides the widgets in the application window, the application itself can also subscribe to an event. For that, I added a tuple of events as well as an event_delta queue to the Application dataclass. This function always returns either the application registry, or the recreated element. Publishing the recreated element to the element_delta queue is handled by the caller.

Unlike the display_update function, there is no clock.tick() function to regulate the timing of execution in the event_process function shown previously. After finishing scheduling the current batch, we want to immediately start fetching and processing a new batch of events as soon as possible. The current dispatching function is rather straightforward, as it just schedules the registered handler by cycling through the events tuple in the application registry as well as each of the elements.

async def events_dispatch(
    application: Application,
    events: Sequence[pygame.event.Event]
) -> None:
    for event in events:
        match event.type:
            case pygame.QUIT:
                application.exit_event.set()

            case pygame.MOUSEBUTTONDOWN:
                for element in application.elements:
                    for ev in element.events:
                        if not ev.kind == event.type:
                            continue

                        if element.position.collidepoint(mouse_x, mouse_y):
                            asyncio.create_task(
                                ev.handler(ev, element, application)
                            )

Basically these is all the code mainly for handling the orchestration work; it contains no game logic. Like how I was working on BigMeow, a structure emerges when I work more on the project. When I work on a personal project, I also tend to avoid writing object-oriented code as much as possible, and I am glad how well the current revision turns out.

Photo by BoliviaInteligente on Unsplash

Being a complete newbie to Pygame, Gemini proved helpful yet again. When I talked about the project with a friend, he jokingly asked if I vibe-coded it. However, the whole experience was not unlike how I ported the LLM word game to AsyncSSH, where Gemini assisted by helping me to navigate the documentation. Granted, the code example is usually good enough, but it helps to know how and why things are written in a certain way.

Lessons Learned and Horizons Gained: My Pygame Journey Continues

Refactoring work started right after I was more or less done with handling mouse click and triggered a change to game state. As the code gets cleaned up, it occurs to me that extracting the game logic into a separate module becomes trivial. To my surprise, after getting Gemini to review the code, I was told the separation would yield a mini framework for similar projects.

Not knowing where I could get feedback, I turned to Gemini to review my code, and we chatted about how this project could be extended. That follow-up discussion was a very enlightening session, apart from the revelation of an emerging framework. Presumably, the Application dataclass resembles an Entity-Component-System. This was definitely a concept worth adding to my learning backlog.

Photo by Glenn Carstens-Peters on Unsplash

If time permits, I would like to work on the separation and complete the game. Curiosity is definitely the biggest motivation behind the effort, rather than publishing the potentially bad framework. New system events like window.onload and window.onclose equivalents may also be implemented to better simulate my web development workflow. Follow the project on GitHub if you are interested.

Unfortunately, I will not have as much free time for a while. On the flip-side, I no longer have to entertain needy and exploitative hiring managers too. With my incoming new job offer, I will adjust the publishing frequency for the coming weeks. I would also like to take this opportunity to thank everyone offering encouragement and help throughout this trying period. Spring does come, after winter.

To ensure this post is as clear and readable as possible, I leveraged Gemini as my editorial assistant. Gemini also provided complementary information to Pygame’s official documentation, helping to clarify complex concepts. However, please note that the voice and all code presented within this article are entirely my own. If you found this article helpful or insightful, consider following me on Medium and subscribing for more content like this.

The Past, Present and Future of Multi-tenancy

Thu, 05 Jun 2025 20:33:10 +0000

We can have stronger data isolation without the drawbacks of earlier forms of dedicated infrastructure.

Multi-tenanted software arrived alongside agile software development, Software as a Service (SaaS) and cloud computing. To understand modern multi-tenancy, we must look at how existing practices shaped early multi-tenancy and dig into the technological advancements that mean we must revise how we approach it.

Let’s look at how software was built around the turn of the century. You’d likely create a gold copy of your application once or twice a year. This would be burned onto CD-ROMs that systems administrators and end users would use to install the software.

For line-of-business applications, you’d use a network location instead of a disc and a checklist instead of an installation wizard.

Instead of self-hosting your purchased software, you could pay an application-service provider to manage it. The provider would manage the infrastructure and installations for you. This led to the idea of the software vendor offering a managed instance on your behalf, which we now call SaaS.

SaaS and Multi-tenancy

Once the same organization builds and operates the software, it inevitably found that it could reduce infrastructure and license costs by sharing the software between many customers.

A customer self-hosting an application could use virtualization to reduce the operational cost of the software, but a SaaS provider could reduce costs even further by sharing application instances between many customers.

The three hosting models are shown below, along with the non-shared costs that can be attributed to a single tenant.

Hosting model: Dedicated infrastructure

Sharing: Nothing is shared
Non-shared costs: Power, racks,
networking, servers, operating systems, application licenses

Hosting model: Virtual machines

Sharing: Shared servers
Non-shared costs: Operating systems, application licenses

Hosting model: Multi-tenanted software

Sharing: Shared application instance
Non-shared costs: Application licenses

This comparison to dedicated instances understandably influenced early multi-tenancy. Designs for multi-tenancy would attempt to replicate the boundary created by client-specific software installations by enforcing a perimeter around the users and data for each tenant. The concerns held by security-conscious organizations would have reinforced this design.

While it was easy to predict the infrastructure costs and savings, the total cost of multi-tenanted software also included intangible costs for operational complexity and code complexity.

The configuration must become more dynamic to support multiple tenants on a single application instance, and the information boundary must be carefully managed. This can result in an increased cost for new features.

Operationally, while there are fewer instances to manage, the instance must meet new scaling demands. A new class of problems emerged when tenant-specific data needed to be managed within a shared database.

The cost-benefit analysis for multi-tenanted software would have placed more weight on the tangible costs. For example, counting the number of servers and operating system licenses is a simple task compared to predicting the future costs associated with the additional code complexity.

Technical Advancements

Since the emergence of multi-tenanted software, several technical advancements have been made that directly affect it, namely containers, deployment automation and infrastructure automation.

Virtualization has been transformed with the arrival of containers, which provide isolation without the weight and cost of multiple operating systems. Deployment automation eliminates the costs of manually installing software and upgrades, which makes deploying a thousand instances as easy as deploying one. Infrastructure automation does the same for provisioning dedicated cloud infrastructure or containers for each tenant.

Combining these developments with the experience gained from two decades of multi-tenanted software architecture requires a new approach. The economics have changed dramatically.

The Future of Multi-tenancy

The old dichotomy was multi-tenancy versus dedicated physical or virtual infrastructure. This led to a binary choice between single or multi-tenanted software. We must adjust this view by considering multi-tenancy as a whole-system design process.

There are two dimensions to consider in this new approach: layers of sharing and component-level granularity for decisions. Where we used to consider whether an application instance was multi-tenanted, we now look at each layer to decide if it should be shared. We can combine a tenant-specific database with a single application instance or share only the codebase and the CI/CD pipeline with tenant-specific infrastructure provisioned automatically.

Equally, we don’t make this decision once for the whole system but per component. This allows us to design a system with tenant-specific instances that call shared multi-tenanted services. This can be particularly powerful if the service is tenant unaware, such as a stateless service or one whose state is not tenant-specific.

By broadening the view of multi-tenancy to apply to software, infrastructure and CI/CD pipelines, it is possible to design systems that take advantage of lightweight tenant-specific applications, isolated by modern virtualization and calling out to scalable shared services. A more thoughtful approach can be taken regarding data that should be protected for a tenant and data that isn’t tenant-specific.

This new approach means we can get a better system-level outcome. We can have stronger data isolation without the drawbacks of earlier forms of dedicated infrastructure.

Summing It up

While early multi-tenancy was defined by sharing an instance of the application and database, technological advancements require a broader perspective and a more granular decision-making approach. The only way to find the right balance between the trade-offs is to consider the widest range of options available.

You should no longer view a tenant as an analog for dedicated isolated instances of users and data. A tenant is a view of data that must be protected. It’s common for sets of data to be stored outside the boundary of a tenant and for users to have access to multiple tenants.

Aside from the technology changes, businesses are more security-conscious than ever, as shown by the increase in ISO 27001 certifications since 2005.

Component-level decisions about the layers of sharing define modern multi-tenancy.

I also wrote the white paper, A modern view of multi-tenancy, which you can download courtesy of Octopus Deploy. I'd love to hear you thoughts and feedback!

Telegram Chatbot’s Evolution: Decoupling Parallel Python’s Shared State for Clarity

Tue, 03 Jun 2025 06:59:38 +0000

After building the intro animation last week,,) I finally took the time to make a video based on the article on jQuery as promised. In the future, articles that are accompanied by a video will all have a play button emoji ▶️ added to the title. While video is still made irregularly, we are going to meet here more regularly. This week, let’s revisit our chatbot project discussed not too long ago. It mostly works fine, but there’s one thing I want to improve on — removing the hardcoded global queue.

Illustration from copilot for the topic

The Global Headache: When a Simple Assignment Becomes a Core Problem

Spreading workload to run in parallel is a hard problem, especially when it comes to keeping them synchronized. Previously in our chatbot, we split the bot and webhook to run in 2 different processes, managed by the ProcessPoolExecutor. The whole architecture mimics my experimental bot named BigMeow ..) The project is definitely flawed, as it is a culmination of my learning on AsyncIO without a formal introduction to the topic.

Not too long ago I was going through a job interview. As part of the process, I was asked to work on a take-home assignment. It was unimaginative as ever, though I took the opportunity to implement an experimental background batch processing module. The whole design, again, is similar to the chatbot, so it works as expected.

Photo by Pascal van de Vendel on Unsplash

When I finally started to write tests to validate my work, that was when my nightmare began.

Though I hacked through the tests, there were multiple minor issues left unresolved after I submitted my work. The hardcoded queue in the settings module severely limits the flexibility and hacks like patches needed to be applied in tests. Despite rejecting the offer, the frustration lingered on my mind for quite a while. How can I do better to avoid spending endless hours fixing tests? Most of my code is rather testable, but there must be something I am missing. During the weekend, I finally took some time going through the code to prepare a new article, and a revelation came and it clicked.

What if I stop hard-coding the queue and pass it into each process explicitly, like the exit event?

With some help from my LLM chatbot, I managed to remove the hardcoded queue. Then I proceeded to pass it into the child processes explicitly, for both the background processing module, and the FastAPI web application. We would talk more about it, including test preparation, in the coming week. For now, let’s shift the focus back to our chatbot.

Exposing the Entanglement: Diving into the Original Architecture

Photo by Josh Redd on Unsplash

Let’s briefly review the current setup, as I try to recall why it was written this way. By understanding the original design decision, we can then start working out a solution, addressing the inflexibility that came with the hard-coded problem.

The reason for having a settings module was basically I needed a place to store some configuration globals. Over time, the list of configurable options piles up. Seeing how convenient it is, I eventually set up a queue to allow the web application process to pass incoming messages from the webhook endpoint to the process hosting the Telegram bot. At the time, it was just a simple queue I created from multiprocessing.Queue. Nothing fancy, just a boring process-safe queue that can be safely shared by multiple processes.

On the other hand, in the main process, we created a multiprocessing.Manager, and created an Event object with it. The created Event object is also process-safe, i.e. can be shared by multiple processes. That event object is then passed explicitly to each child process, to be listened to for a cue to exit.

The multiprocessing.Manager here acts almost like a coordinator, making sure processes can communicate effectively with each other whenever needed. Unfortunately, this means messages would be passing through an additional layer before hitting the target. Still the overhead is worth it, when it comes to queues. There is a risk of non-managed queue getting stuck in a shut down, especially when it is not empty. The Queue object created with the multiprocessing.Manager would handle it gracefully.

Delving deeper into multiprocessing.Manager isn’t the point of the article, and I am clearly not capable of doing it as of now. Just understand it is beneficial in orchestrating synchronization between processes, and we are going to utilize it in our Telegram bot.

From Global to Explicit: Refactoring with Centralized State

Knowing the limitation is one thing; next we are going to fix it. As we tackle the hard-coding problem, we will see how it becomes more flexible, while not breaking the inter-process communication.

Photo by U. Storsberg on Unsplash

Since we created a multiprocessing.Manager already to create an Event object, let’s extend it to also create the queue. Move the telegram_queue to our main module

import signal
import multiprocessing
from functools import partial

def shutdown_handler(
    _signum, _frame, exit_event: threading.Event
) -> None:
    exit_event.set()

manager = multiprocessing.Manager()
exit_event = manager.Event()
telegram_queue = manager.Queue() # MOVED telegram_queue here

for s in (signal.SIGHUP, signal.SIGTERM, signal.SIGINT):
    signal.signal(
        s, partial(shutdown_handler, exit_event=exit_event)
    )

with ProcessPoolExecutor() as executor:
    executor.submit(process_run, tg_run, exit_event)
    executor.submit(process_run, web_run, exit_event)

They can be passed into the child processes as-is, but I prefer to group them into an object (for reference, BigMeow now has 11 synchronization primitives, namely queues, locks, and events). Let’s create a simple dataclass for this:

from dataclasses import dataclass
from threading import Event
from queue import Queue

@dataclass
class SyncStore:
    exit_event: Event
    telegram_queue: Queue

Notice multiprocessing.Manager creates proxies of the synchronization primitives. They often have the same programming interface as existing objects, hence the type annotation specifies threading.Event and queue.Queue. Next we instantiate an object by

import multiprocessing

manager = multiprocessing.Manager()
sync_store = SyncStore(
    manager.Event(),
    manager.Queue()
)

After that, we adapt the changes in our main module, remember to also pass the new shiny sync_store into our child processes in the executor.submit call

for s in (signal.SIGHUP, signal.SIGTERM, signal.SIGINT):
    signal.signal(
        s, partial(shutdown_handler, exit_event=sync_store.exit_event)
    )

with ProcessPoolExecutor() as executor:
    executor.submit(process_run, tg_run, sync_store)
    executor.submit(process_run, web_run, sync_store)

There is more work for the FastAPI web application module, so we are doing it last, For now, we first modify the function signature of run() in our Telegram bot, and update the exit_event.wait() call accordingly

async def run(sync_store: SyncStore) -> None:
    async with application:
        ...

        await asyncio.to_thread(sync_store.exit_event.wait)

        ...

Previously, the coroutine function message_consume implicitly read from settings.telegram_queue. As it is now made available to run(), we can proceed to pass it as an argument. Also there’s no reason why the bot’s application object to be global, let’s move everything into the function:

# remove this line
# application = ApplicationBuilder().token("Your telegram token").build()

async def run(sync_store: SyncStore) -> None:
    application = ApplicationBuilder().token("Your telegram token").build()
    async with application:
        await setup(application) # pass the application object explicitly
        # Starting the bot
        await application.start()
        # Receive messages (pass the object explicitly)
        asyncio.create_task(loop_queue(message_consume, application, sync_store.telegram_queue))
        # Wait for exit signal
        await asyncio.to_thread(sync_store.exit_event.wait)
        # Stopping the bot
        await application.stop()

Both loop_queue and message_consume would need to adapt to the change

from typing import Any
from telegram.ext import Application
from queue import Queue

# Take additional args to be passed to the coroutine function
async def loop_queue(coro_func: Callable[[], Awaitable[None]], *args: Any) -> None:
    while True:
        await coro_func(*args)

async def message_consume(application: Application, queue: Queue) -> None:
    with suppress(queue.Empty):
        await application.update_queue.put(
            Update.de_json(
                # no more consuming from settings.telegram_queue
                await asyncio.to_thread(queue.get, timeout=10),
                application.bot,
            )
        )

Finally, applying the changes to the FastAPI web application module. Like the Telegram bot module, we first revise the run() function signature

async def run(sync_store: SyncStore) -> None:
    ...

Then, we will have to register our store to the app.state ..) Also update the uvicorn config accordingly as app is modified in our function.

async def run(sync_store: SyncStore) -> None:
    app.state.sync_store = sync_store

    server = uvicorn.Server(
        uvicorn.Config(
            app, # we made changes to global app, so specify explicitly
            host="0.0.0.0",
            port=80,
            log_level="info",
        )
    )

    ...

Usually this is done in the lifespan, where we yield an object, that can be later accessed through request.state in the handler functions, as shown below:

from fastapi import FastAPI, Request

@asynccontextmanager
async def lifespan(app: FastAPI):
    yield {'foo', 'bar'} # yield the object here

app = FastAPI(lifespan=lifespan)

@app.get('/')
async def index(request: Request):
    print(request.state.foo) # 'bar'

Unfortunately, in our case app is instantiated outside of run() and it has no knowledge of sync_store at the time. Considering now we just store it at app.state directly, we could still define a lifespan handler, to check for the presence of the store, at startup:

@asynccontextmanager
async def lifespan(app: FastAPI):
    # do a check when the application starts up to ensure sync_store is available
    if not (hasattr("sync_store", app.state) and isinstance(app.state.sync_store, SyncStore)):
        raise RuntimeError("SyncStore object is not found")

    yield

app = FastAPI(lifespan=lifespan)

Lastly, update the webhook, to use the queue from sync_store, via request.app.state.sync_store.telegram_queue

@app.post("/webhook/telegram", include_in_schema=False)
async def telegram_webhook(
    request: Request,
    x_telegram_bot_api_secret_token: Annotated[
        str, Header(pattern=settings.TELEGRAM_SECRET_TOKEN, strict=True)
    ],
) -> None:
    asyncio.create_task(
        asyncio.to_thread(
            # change how the queue is accessed
            request.app.state.sync_store.telegram_queue.put,
            await request.json()
        )
    )

Although the refactoring looks rather significant, the core logic underneath is unaffected. Apart from changing how the queue is accessed in both the Telegram and the FastAPI web application, there’s no change on how it is done. Data is still being produced by the web application via .put(), and consumed by the .get() in the bot.

Beyond the Fix: Testability, Scalability, and What’s Next

So back to the original objective of this refactoring — remove the hard-coded telegram_queue. Fortunately, we have done so successfully, though the process was somewhat involved. The biggest advantage of passing them explicitly; is how it improves testability. No more patching and replacing globals are needed, just pass the test queue to the function, or properly populate the app.state while creating FastAPI’s TestClient fixture.

Testing would be covered next week.

We also explored grouping all the synchronization primitives into one object. When the number of child processes increases, the number of such objects would accumulate inevitably. Even if we cherry-pick, ensuring only the relevant ones go into each child process, some process would end up taking too many arguments. Thus, creating an object containing them can be a good workaround to the problem.

Other than the rather involved gymnastics of moving things around, there’s no change to the underlying workflow. Still it brings consistency compared to the original implementation. Now all synchronization primitives are all defined in the same place, all managed by the multiprocessing.Manager, and are introduced to each child process explicitly as arguments. Additionally, the usage of multiprocessing.Manager makes it easy to scale the application as the manager can run as a server.

Photo by Sharon Co Images on Unsplash

This is it, the journey started from a very unassuming take-home assignment and in the end became a valuable lesson. On a high level it isn’t quite significant, as it doesn’t bring much optimization (if not slower due to the change to the queue managed by multiprocessing.Manager). Even so, by applying what I learned to my chatbot, it immediately became relevant as the bot is still running out there serving its purpose. Turning the experience into this article; is also how I reciprocate to the community.

Hmm, that was the third time my LLM Editor suggested I should use a semicolon ..)

Anyway, that’s all I have for this week, thanks for reading, and I shall write again, next week.

Just so you know, while an AI assistant helped me refine this article’s clarity and structure, the voice, all the technical content, and every line of code presented here are entirely mine. This piece reflects my personal experiences and insights. If you’re interested in job opportunities or collaborating on projects, please feel free to reach out via LinkedIn or connect with me right here on Medium.