Coder Legion - Recent posts tagged database

Flyway Migration: From Legacy Database to CI/CD

Thu, 29 Oct 2026 12:05:30 +0000

Most teams think of Flyway only when they need to manage ongoing database schema changes. But what if your project already has a complete, stable schema and you’re only now introducing version control? Or what if you want to incorporate your schema into a CI/CD pipeline for reliable, reproducible deployments?

In this guide, I’ll walk you through a real-world scenario where we used Flyway to baseline an existing MySQL database and then extended it to support version-controlled schema migrations in CI/CD, without disrupting production.

What is Flyway?

Flyway is a simple but powerful tool for managing database migrations. It applies versioned SQL scripts to your database and tracks them in a special table called flyway_schema_history.

Typical script names:

V1__initial_schema.sql
V2__add_orders_table.sql
V3__rename_column.sql

Flyway ensures these scripts run only once and in the correct order. It also integrates well with CI/CD tools, making it ideal for automated environments.

Use Case: Existing Schema with No Migrations (Yet)

In our project, we had:

A production MySQL database with dozens of tables
No migration history
No current need for structural changes
A desire to version control the database and enable future migrations via CI/CD

Step 1: Baselining an Existing Schema

Dump the schema

mysqldump -u root -p --no-data your_database > V1__baseline_schema.sql

Clean the output: remove auto-increments, user references, and environment-specific SQL.

Create directory structure

mkdir -p db/sql
mv V1__baseline_schema.sql db/sql/

Configure flyway.conf

flyway.url=jdbc:mysql://your-db-host:3306/your_database
flyway.user=db_user
flyway.password=secret_password
flyway.locations=filesystem:db/sql
flyway.baselineOnMigrate=true

Run baseline

flyway baseline

This creates the flyway_schema_history table and marks version 1 as the current state without running any script.

Step 2: Add a Real Migration

After baselining, we added a real change in a new script

-- db/sql/V2__add_status_to_orders.sql
ALTER TABLE orders ADD COLUMN status VARCHAR(20) DEFAULT 'pending';

Then we ran:

flyway migrate

Result: Flyway applied version 2, recorded it in history, and our database was now safely versioned.

⚙️ Step 3: Integrate Flyway into CI/CD

You can integrate Flyway with GitHub Actions, GitLab CI, Bitbucket Pipelines, or Jenkins.

✅ Example GitHub Actions Workflow

name: Flyway Migration
on:
  push:
    branches: [ main ]

jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Set up Java (Flyway needs it)
        uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '17'
      - name: Install Flyway
        run: |
          curl -L https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/10.8.1/flyway-commandline-10.8.1-linux-x64.tar.gz | tar xvz
          sudo ln -s $PWD/flyway-10.8.1/flyway /usr/local/bin/flyway
      - name: Run Flyway Migrations
        env:
          FLYWAY_URL: jdbc:mysql://your-host:3306/your_database
          FLYWAY_USER: ${{ secrets.DB_USER }}
          FLYWAY_PASSWORD: ${{ secrets.DB_PASSWORD }}
        run: |
          flyway -url=$FLYWAY_URL \
                 -user=$FLYWAY_USER \
                 -password=$FLYWAY_PASSWORD \
                 -locations=filesystem:./db/sql \
                 migrate

Benefits of This Approach

✅ You gain auditability for all database changes

✅ Developers can build local environments using the same scripts

✅ CI/CD ensures consistency across all environments

✅ Flyway’s validate command catches missing or changed scripts

✅ Schema management becomes part of your software delivery lifecycle

Potential Challenges

Problem → Fix

Flyway re-applies the baseline script
→ Use baselineOnMigrate=true and never modify a baseline
Passwords in config files
→ Use secrets and environment variables in CI/CD
Manual schema drift
→ Disable direct DB access in prod or run flyway validate periodically
Need rollback support
→ Write manual undo scripts or use point-in-time backups

Alternatives (and Why Flyway Wins)

Liquibase
✅ Pros: Advanced features, rollback support
❌ Cons: More complex, steeper learning curve

Django ORM / Alembic
✅ Pros: Built-in for app devs
❌ Cons: Not cross-language or DB-agnostic

Manual SQL or DB dumps
✅ Pros: Simple
❌ Cons: No versioning, no tracking, no automation

Flyway is a great middle-ground: powerful enough for teams, simple enough for solo devs, and perfect for CI/CD.

Final Thoughts

Flyway isn’t just for teams writing migrations every day. It’s also ideal for bringing an existing database under version control, giving your project a foundation for safe schema evolution.

We started with a baseline, added our first real migration, and then built it into our CI/CD pipeline — all in a day’s work.

If you’re managing an existing DB without schema tracking, now’s the time to fix that. Start with a baseline. Your future self will thank you.

Let’s Connect!

Have you tried Flyway for baseline-only use cases or integrated it into your CI/CD pipeline? I’d love to hear your experience in the comments.

If you found this helpful, follow me here on DEV for more DevOps and database engineering content.

Database Transactions - ACID Properties

Thu, 29 Oct 2026 01:49:48 +0000

Introduction

Yesterday, I went to a restaurant with my friends. We ordered some food, finished eating, and asked the waiter for the bill.

While the waiter was preparing the bill, I started thinking about how the billing system works behind the scenes.

When we place an order, the system has to do multiple things, like saving the order details, calculating the bill, and updating the payment status.

But what if something goes wrong in the middle? What if the system crashes before completing all these operations?

This is where database transactions come into the picture.

Recently, I was learning about database transactions, ACID properties, and isolation levels. In this article, I'll share what I understood, along with some simple examples.

What is a Database Transaction?

A database transaction is a group of one or more database operations that are treated as a single unit of work.

Let's take our restaurant example.

When a customer places an order, the system might need to perform these operations:

Create an order.
Save the ordered items.
Update the available stock.
Calculate and save the total bill.

All these operations are related. If one of them fails, we don't want the database to be left with incomplete information.

So, instead of executing them as separate independent operations, we can group them into a single transaction.

A transaction generally involves three important commands:

BEGIN — Starts a transaction.
COMMIT — Saves the changes made by the transaction.
ROLLBACK — Undoes the uncommitted changes if something goes wrong.

For example:

BEGIN;

INSERT INTO orders (order_id, table_id, total_amount)
VALUES (101, 5, 500);

INSERT INTO order_items (order_id, item_id, quantity)
VALUES (101, 10, 2);

UPDATE inventory
SET quantity = quantity - 2
WHERE item_id = 10;

COMMIT;

Here, all three database operations are part of one transaction.

If everything succeeds, we commit the transaction.

If an operation fails, we can roll back the transaction so that the database doesn't keep only part of the changes.

But what happens if the database itself crashes while executing a transaction?

A database can crash for several reasons, such as power failures, hardware problems, operating system failures, or software bugs.

Even in these situations, the database needs a way to recover without leaving incomplete transactions behind.

To handle these kinds of problems and maintain reliable data, database transactions follow four important properties, commonly called ACID properties.

ACID Properties

ACID stands for:

A — Atomicity
C — Consistency
I — Isolation
D — Durability

Each property solves a different problem.

Atomicity ensures that either all operations in a transaction happen or none of them happen.

Consistency ensures that a transaction preserves the rules and constraints defined in the database.

Isolation controls how multiple transactions running at the same time interact with each other.

Durability ensures that once a transaction is committed, its changes survive a crash, under the database's durability guarantees.

Let's understand each property with examples.

Atomicity

Atomicity means that either all the operations in a transaction should succeed, or none of them should happen.

Let's take our restaurant example.

Suppose a customer orders 2 pizzas, and each pizza costs ₹200.

When the waiter confirms the order, the system needs to perform two operations:

Create an order with a total amount of ₹400.
Reduce the available pizza stock by 2.

Now, let's say the order is created successfully, but the system crashes before updating the stock.

In this case, the order exists in the database, but the stock has not been updated. This leaves the data in an incorrect state.

To avoid this, we can execute both operations inside a transaction.

BEGIN;

INSERT INTO orders (order_id, amount)
VALUES (101, 400);

UPDATE inventory
SET quantity = quantity - 2
WHERE item_id = 1;

COMMIT;

If both operations succeed, the transaction is committed.

But if something goes wrong before the transaction is committed, the database rolls back the changes.

So, even if the order was inserted successfully, it won't remain in the database if the transaction fails.

This is what atomicity guarantees: All or nothing.

How does the database maintain atomicity?

Databases use transaction logs and recovery mechanisms to handle failures.

For example, PostgreSQL uses Write-Ahead Logging (WAL).

Before modified data pages are written to their final location on disk, PostgreSQL records the changes in WAL.

If the database crashes, it uses WAL and transaction status information during recovery to ensure that committed transactions are recovered and incomplete transactions do not become visible as committed changes.

We'll understand WAL in more detail when we discuss durability.

Consistency

Consistency means that a transaction should maintain the rules and constraints defined in the database.

While learning about consistency, I came across two different concepts: Data Consistency and Read Consistency.

1. Data Consistency (Correctness)

Data consistency is about keeping the data correct and making sure it follows the rules defined in the database.

Let's say we have three tables.

Blogs Table

blog_id	likes
1	5
2	1

Users Table

user_id	name
1	John
2	Edmon

Pictures Table

user_id	picture_id
1	1
1	2
2	3
2	4

Here, user_id in the Pictures table is a foreign key referencing the Users table.

Now suppose we try to insert a picture with user_id = 5, but there is no user with that ID in the Users table.

INSERT INTO pictures (user_id, picture_id)
VALUES (5, 5);

The database rejects this operation because it violates the foreign key constraint.

This is called referential integrity.

Similarly, databases maintain data consistency using constraints such as primary keys, foreign keys, UNIQUE, NOT NULL, and CHECK.

Apart from these constraints, we may also have business rules that need to be maintained by our application or database.

So, the main idea of ACID Consistency is that a transaction should move the database from one valid state to another valid state without breaking the defined rules.

In simple terms, it focuses on correctness of data.

2. Read Consistency (Availability and Freshness)

Now let's look at consistency from another perspective.

In real applications, we may have multiple copies of the same database, called replicas.

Why do we need replicas?

One reason is availability. If one database server goes down, another server may still be able to serve requests. Replicas can also help distribute read traffic.

Let's say we have one primary database and two replicas.

Initially, all three databases contain the same value.

Primary Database → 90
Replica 1 → 90
Replica 2 → 90

Now suppose a transaction updates the value from 90 to 100 in the primary database and commits.

But replication may take some time.

At that moment, the data might look like this:

Primary Database → 100
Replica 1 → 90
Replica 2 → 90

Now imagine two users trying to read that value.

T1:

User 1 reads from the primary database and gets 100.

T2:

User 2 reads from Replica 1 and gets 90.

Both users are reading the same record, but they are getting different values.

This happens because the replicas haven't received the latest update yet.

After some time, replication catches up.

Primary Database → 100
Replica 1 → 100
Replica 2 → 100

Now all three databases have the same value.

This is an example of Eventual Consistency.

It means different replicas may temporarily have different values, but eventually they become consistent if updates stop and replication completes successfully.

Availability vs. Consistency

Here, we have an interesting trade-off.

Suppose a replica hasn't received the latest update yet.

We have two possible approaches.

Option 1: Allow the read

We can allow the user to read from that replica, even though it has older data.

The user gets a response immediately, but the value might be stale.

This favors availability, although it doesn't guarantee that the user gets the latest data.

Option 2: Wait for the latest data

We can wait until the replica catches up, or route the request to a database that has the required version.

The user gets fresher data, but the request may take longer or become unavailable if the required data cannot be reached.

So, depending on the system's requirements, we may need to make trade-offs between availability and the consistency of reads, especially during network failures.

What's the Difference?

The main thing I understood is that these two concepts focus on different problems.

ACID Consistency → Correctness

It makes sure the database follows its defined rules and constraints. A transaction should not leave the data in an invalid state.

Read Consistency → Freshness of Data

It defines what values a reader is allowed to see, especially when data is replicated across multiple servers.

Availability is related to read consistency because allowing reads from replicas with slightly older data can help a system continue serving requests.

So, ACID consistency is mainly about keeping data correct, while consistency in distributed reads is about how up-to-date and consistent the data needs to be across different replicas.

Isolation

Until now, we have seen how transactions maintain atomicity and consistency.

But what happens when multiple transactions run at the same time?

Let's take our restaurant example.

Suppose two waiters are taking orders from different tables. Both are using the same billing system.

Now imagine only one pizza is left in stock, and both waiters are trying to place an order for it.

If both transactions read the available quantity at the same time, they might both think the pizza is available.

This is where Isolation comes in.

Isolation controls how multiple transactions running at the same time interact with each other.

Ideally, we want the result of concurrent transactions to be the same as some valid order of executing them one after another. But running every transaction one after another can affect performance.

So databases provide different isolation levels.

Before understanding them, let's look at four common problems that happen when transactions execute concurrently.

Dirty Read
Non-Repeatable Read
Phantom Read
Lost Update

1. Dirty Read

A dirty read happens when a transaction reads a change made by another transaction that hasn't committed yet.

Let's take a sales table.

pid	qnt	price
P1	10	5
P2	20	4

Suppose two transactions are running.

Time	TX1	TX2
T1	`BEGIN;`
T2	`SELECT pid, qnt * price FROM sales;` → P1: 50, P2: 80
T3		`BEGIN;`
T4		`UPDATE sales SET qnt = qnt + 5 WHERE pid = 'P1';`
T5	`SELECT SUM(qnt * price) FROM sales;` → 155
T6		`ROLLBACK;`
T7	`COMMIT;`

At T5, TX1 reads the updated value from TX2 even though TX2 hasn't committed yet.

Later, TX2 rolls back its changes, but TX1 has already used that uncommitted data.

This is called a Dirty Read.

PostgreSQL note: This example represents a database that allows dirty reads. PostgreSQL doesn't allow them, even with Read Uncommitted.

2. Non-Repeatable Read

A non-repeatable read happens when a transaction reads the same record twice and gets different values because another transaction committed an update in between.

Let's use the same Sales table.

Time	TX1	TX2
T1	`BEGIN;`
T2	`SELECT pid, qnt * price FROM sales;` → P1: 50, P2: 80
T3		`BEGIN;`
T4		`UPDATE sales SET qnt = qnt + 5 WHERE pid = 'P1';`
T5		`COMMIT;`
T6	`SELECT SUM(qnt * price) FROM sales;` → 155
T7	`COMMIT;`

Here, TX1 initially reads the sales values.

Meanwhile, TX2 updates the quantity and commits.

When TX1 later calculates the total, it sees the updated data.

So TX1 gets different results within the same transaction.

This demonstrates the effect of a non-repeatable read. Strictly speaking, the standard definition involves reading the same row twice; here, the second statement is an aggregation over the changed rows.

This can happen under PostgreSQL's default Read Committed isolation level.

3. Phantom Read

A phantom read happens when a transaction executes a query and later gets a different set of matching rows because another transaction inserts, deletes, or changes matching records.

Suppose we have this table:

id	ch
1	a
2	a
3	b
4	b

Time	TX1	TX2
T1	`BEGIN;`
T2	`SELECT * FROM test WHERE ch = 'b';` → 2 rows
T3		`BEGIN;`
T4		`INSERT INTO test (id, ch) VALUES (5, 'b');`
T5		`COMMIT;`
T6	`SELECT * FROM test WHERE ch = 'b';` → 3 rows
T7	`COMMIT;`

Initially, TX1 gets two records.

Meanwhile, TX2 inserts another record with ch = 'b' and commits.

When TX1 executes the same query again, it gets three records.

The newly appearing record is called a phantom row.

PostgreSQL's Repeatable Read prevents phantom reads because the transaction continues using the same snapshot.

4. Lost Update

A lost update happens when two transactions read the same value and update it independently, causing one transaction's change to be overwritten.

Suppose P1's current quantity is 10.

TX1 wants to increase it by 10, and TX2 wants to increase it by 5.

The expected final quantity is 25.

Time	TX1	TX2
T1	`BEGIN;`
T2	`SELECT qnt FROM sales WHERE pid = 'P1';` → 10
T3		`BEGIN;`
T4		`SELECT qnt FROM sales WHERE pid = 'P1';` → 10
T5	`UPDATE sales SET qnt = 20 WHERE pid = 'P1';`
T6		`UPDATE sales SET qnt = 15 WHERE pid = 'P1';` → Waits for TX1
T7	`COMMIT;`
T8		TX2's UPDATE completes
T9		`COMMIT;`

The final quantity becomes 15 instead of 25.

TX2 overwrites the value written by TX1, so TX1's update is lost.

This is called a Lost Update.

This can happen when an application calculates a new value using an earlier read and then writes that fixed value back.

One way to prevent it is by using SELECT FOR UPDATE.

BEGIN;

SELECT qnt
FROM sales
WHERE pid = 'P1'
FOR UPDATE;

UPDATE sales
SET qnt = qnt + 10
WHERE pid = 'P1';

COMMIT;

Here, PostgreSQL locks the selected row. Another transaction trying to update the same row must wait.

If we simply want to increment a value, we can also do it directly:

UPDATE sales
SET qnt = qnt + 5
WHERE pid = 'P1';

This avoids calculating the new value using an outdated application-side read.

Database Isolation Levels

Now that we have seen these problems, let's understand how databases prevent them.

SQL defines four common isolation levels:

Read Uncommitted
Read Committed
Repeatable Read
Serializable

1. Read Uncommitted

This is the weakest standard isolation level.

A transaction may read changes made by another transaction before they are committed.

So dirty reads, non-repeatable reads, and phantom reads are possible.

For example, in our Dirty Read scenario, TX1 could read the updated sales total even though TX2 later rolls back.

We can request this isolation level using:

BEGIN TRANSACTION
ISOLATION LEVEL READ UNCOMMITTED;

However, PostgreSQL treats Read Uncommitted as Read Committed.

So we cannot reproduce dirty reads in PostgreSQL.

2. Read Committed

Read Committed prevents dirty reads.

A transaction only reads committed data.

However, each statement gets a new snapshot, so the same query may return different results when executed twice.

Let's take our sales example.

Time	TX1	TX2
T1	`BEGIN ISOLATION LEVEL READ COMMITTED;`
T2	`SELECT pid, qnt * price FROM sales;` → P1: 50, P2: 80
T3		`BEGIN;`
T4		`UPDATE sales SET qnt = qnt + 5 WHERE pid = 'P1';`
T5		`COMMIT;`
T6	`SELECT SUM(qnt * price) FROM sales;` → 155
T7	`COMMIT;`

Even though TX1 hasn't finished, its next query sees the changes committed by TX2.

This is allowed because Read Committed creates a new snapshot for each statement.

Read Committed is PostgreSQL's default isolation level.

3. Repeatable Read

In Repeatable Read, a transaction continues reading from the same snapshot.

Let's use the same example.

Time	TX1	TX2
T1	`BEGIN ISOLATION LEVEL REPEATABLE READ;`
T2	`SELECT pid, qnt * price FROM sales;` → P1: 50, P2: 80
T3		`BEGIN;`
T4		`UPDATE sales SET qnt = qnt + 5 WHERE pid = 'P1';`
T5		`COMMIT;`
T6	`SELECT SUM(qnt * price) FROM sales;` → 130
T7	`COMMIT;`

Notice the difference.

Even though TX2 committed its update, TX1 still calculates the total using its original snapshot.

This happens because PostgreSQL uses a consistent snapshot established when the transaction first executes a statement that needs one.

So changes committed by other transactions after that snapshot aren't visible to TX1.

One interesting thing about PostgreSQL is that Repeatable Read also prevents phantom reads.

This is stronger than the minimum Repeatable Read guarantees described by the SQL standard.

However, Repeatable Read doesn't prevent every possible concurrency anomaly.

For example, write skew can still happen.

4. Serializable

Serializable is the strongest standard isolation level.

It ensures that concurrent transactions produce a result equivalent to executing those transactions one after another in some order.

This doesn't mean PostgreSQL actually runs every transaction sequentially.

Transactions can still run concurrently.

But PostgreSQL checks for dangerous dependencies that could produce a result that isn't equivalent to any serial execution.

PostgreSQL uses Serializable Snapshot Isolation (SSI).

Let's take an example.

Suppose we have two records.

Record	Value
A	100
B	100

Our business rule is that at least one record must always have a value of 100.

Two transactions are running at Serializable isolation.

Time	TX1	TX2
T1	`BEGIN ISOLATION LEVEL SERIALIZABLE;`
T2	`SELECT value FROM test WHERE id IN ('A', 'B');` → 100, 100
T3		`BEGIN ISOLATION LEVEL SERIALIZABLE;`
T4		`SELECT value FROM test WHERE id IN ('A', 'B');` → 100, 100
T5	`UPDATE test SET value = 0 WHERE id = 'A';`
T6		`UPDATE test SET value = 0 WHERE id = 'B';`
T7	`COMMIT;`
T8		`COMMIT;` → Serialization failure

Both transactions initially read A = 100 and B = 100.

TX1 changes A to 0.

TX2 changes B to 0.

If both transactions committed successfully, we would have:

A = 0
B = 0

This would violate our business rule.

This kind of anomaly is called Write Skew.

Under PostgreSQL Repeatable Read, both transactions may successfully commit because they update different rows.

But PostgreSQL Serializable can detect the dangerous dependency and abort one transaction.

We may see an error like:

ERROR: could not serialize access due to
read/write dependencies among transactions

When this happens, the application needs to retry the entire transaction.

Serializable is useful when we need stronger correctness guarantees, but we should also be prepared to handle transaction retries.

Pessimistic and Optimistic Concurrency Control

While learning about isolation, I also came across two ways of handling concurrent updates.

Pessimistic Concurrency Control

Here, we lock the data before changing it.

For example:

BEGIN;

SELECT qnt
FROM sales
WHERE pid = 'P1'
FOR UPDATE;

UPDATE sales
SET qnt = qnt + 10
WHERE pid = 'P1';

COMMIT;

The FOR UPDATE statement locks the selected row.

Another transaction trying to acquire a conflicting lock must wait until the first transaction releases it.

This is useful when conflicts are likely.

Optimistic Concurrency Control

In optimistic concurrency control, we allow transactions to proceed without locking the record exclusively in advance.

Before applying the update, we check whether another transaction has changed the data.

If a conflicting change is detected, the operation may fail and need to be retried.

This approach can be useful when conflicts are less frequent.

PostgreSQL's Serializable Snapshot Isolation also uses conflict detection, although its implementation is different from application-level version checking.

Comparison of Isolation Levels

Isolation Level	Dirty Read	Non-Repeatable Read	Phantom Read
Read Uncommitted	Possible	Possible	Possible
Read Committed	Prevented	Possible	Possible
Repeatable Read	Prevented	Prevented	Possible under SQL standard
Serializable	Prevented	Prevented	Prevented

This table describes the general SQL-standard guarantees.

PostgreSQL has two important differences:

Read Uncommitted behaves like Read Committed, so dirty reads are prevented.
Repeatable Read also prevents phantom reads.

Lost updates need separate consideration because their behavior depends on the type of update and the database's concurrency-control mechanisms.

What I Understood About Isolation

Isolation is not simply about making transactions run one after another.

It is about controlling how concurrent transactions interact.

Different isolation levels provide different guarantees, and the right choice depends on what our application needs.

For many applications, Read Committed is sufficient.

But when multiple queries need to see a consistent snapshot, Repeatable Read may be useful.

And when concurrent transactions must behave like some serial execution, Serializable provides the strongest guarantee.

The important thing is to understand what problems each isolation level prevents and choose accordingly.

Durability

Durability means that once a transaction is committed, its changes should not be lost, even if the database crashes or restarts.

Let's take our restaurant example.

Suppose a customer finishes eating, pays the bill, and the waiter confirms the payment in the billing system.

The database successfully commits the transaction.

But immediately after that, the database server crashes.

When the server restarts, the payment information should still be there.

This is what Durability guarantees.

How do databases maintain durability?

Databases use different techniques to make sure committed changes survive failures.

Some common techniques are:

Write-Ahead Logging (WAL)
OS Page Cache and fsync
Checkpoints
Asynchronous Snapshots
Append-Only Files (AOF)

1. Write-Ahead Logging (WAL)

Writing every change directly to the database files can be expensive.

For example, when we update a record, the database may need to modify multiple data pages and indexes.

Instead of writing all these changes to their final locations immediately, databases like PostgreSQL use Write-Ahead Logging (WAL).

The idea is simple.

Before modified data pages are written to disk, the database first records the changes in a log.

Let's see how PostgreSQL handles an UPDATE query.

BEGIN;

UPDATE orders
SET payment_status = 'Paid'
WHERE order_id = 101;

COMMIT;

When PostgreSQL executes this query, it modifies the data page in shared buffers and creates the required WAL records in WAL buffers.

Both are initially in RAM.

Under normal synchronous commit settings, PostgreSQL persists the required WAL before confirming the transaction.

The modified data pages can be written to disk later.

Here is how the process works.

Now suppose the database crashes after COMMIT, before the modified data pages are written to disk.

When PostgreSQL restarts, it can use the persisted WAL records to recover the committed changes.

This is why WAL plays an important role in durability.

2. OS Page Cache and fsync

One interesting thing I learned is that writing data to a file doesn't always mean it has reached the physical disk.

Usually, when a program writes to a file, the operating system may first keep that data in its page cache, which is part of RAM.

Why?

Because writing to RAM is much faster than writing directly to disk. The operating system can write the cached data to disk later.

But how is this related to database durability?

Let's take our PostgreSQL example.

When PostgreSQL executes an UPDATE query, it modifies the data in shared buffers and generates WAL records in WAL buffers.

Both are initially in RAM.

When PostgreSQL writes these WAL records to its WAL file, they may first go through the operating system's page cache.

So even though PostgreSQL has written the WAL to a file, the data might still be sitting in RAM.

The flow looks like this:

Now suppose the WAL has been written to the OS page cache, but it hasn't reached the disk yet.

If the machine suddenly loses power, the cached WAL data may be lost.

And if the modified data page was also only in RAM, PostgreSQL might lose that change.

This is where fsync() comes in.

fsync() is an operating system call that requests the modified data of a file to be synchronized to stable storage.

In simple terms, it helps make sure the data isn't just sitting in the OS page cache.

Under normal durability settings, PostgreSQL uses fsync() or other appropriate synchronization methods to persist the required WAL before confirming a synchronous commit.

So even if the database crashes immediately after COMMIT, PostgreSQL can use the persisted WAL to recover the changes.

The important difference is:

WAL records the changes needed for recovery, while fsync or a similar synchronization mechanism helps make those WAL records durable.

There is also a performance cost. Synchronizing data to storage takes time, so frequent synchronization can increase write latency.

PostgreSQL provides settings such as fsync and synchronous_commit to control this behavior.

3. Checkpoints

We have seen that PostgreSQL can commit a transaction before writing all the modified data pages to their final locations.

But those modified pages cannot remain only in memory forever.

They eventually need to be written to disk.

This is where Checkpoints come in.

A checkpoint is a process where PostgreSQL ensures that dirty data pages (modified in memory but not yet on disk) are written to storage and records a point from which crash recovery can proceed.

For example, suppose the database has been running for several hours and thousands of transactions have modified data.

If the database crashes, PostgreSQL needs to replay WAL records to recover changes that weren't already reflected in the database files.

Without checkpoints, the amount of WAL that needs to be processed during recovery could become very large.

By performing checkpoints periodically, PostgreSQL can reduce the amount of recovery work required.

So checkpoints mainly help with writing modified data pages to disk and controlling crash recovery time.

4. Asynchronous Snapshots

Some databases use snapshots as a persistence mechanism.

Instead of saving every change immediately, they periodically save a snapshot of their data.

For example, Redis supports RDB snapshots.

Suppose a database takes a snapshot at 10:00 AM.

After that, some new records are created at 10:05 AM.

Now imagine the server crashes at 10:06 AM, before the next snapshot is taken.

If snapshots are the only persistence mechanism, those recent changes may be lost.

So the durability guarantee depends on how frequently snapshots are taken and whether other persistence methods are enabled.

5. Append-Only Files (AOF)

Another persistence technique is an Append-Only File (AOF).

Redis supports this mechanism.

Instead of saving the entire dataset after every update, Redis records write operations in an append-only log.

When the server restarts, Redis can use the persisted AOF to rebuild its data.

However, durability depends on how frequently the AOF is synchronized to disk.

For example, Redis can be configured to synchronize after every write or approximately once per second.

So AOF provides another way to recover data, but recent changes may still be lost depending on the persistence settings.

Conclusion

When I started learning about database transactions, I thought it was mostly about BEGIN, COMMIT, and ROLLBACK.

But as I went deeper, I came across some interesting things, especially what happens when two transactions run at the same time and how PostgreSQL handles failures.

There is still a lot more to learn about how databases work internally, but understanding these concepts helped me see what actually happens behind a simple database query.

Your Pagination Is Showing Users the Same Item Twice (and Skipping Others). Here's Why Offset Breaks

Mon, 26 Oct 2026 11:11:24 +0000

A user scrolls through your feed. Page 1 ends with "Post 20". They tap "Next", and page 2 starts with... "Post 20" again.

Another user pages through an admin table and never sees order #4817. It exists. It is in the database. It just never showed up on any page.

Nobody changed any code. The data changed underneath the pagination, and your skip and limit could not cope.

The Code Everyone Writes

app.get('/api/posts', async (req, res) => {
  const page = Number(req.query.page) || 1;
  const limit = 20;

  const posts = await Post.find()
    .sort({ createdAt: -1 })
    .skip((page - 1) * limit)
    .limit(limit);

  res.json(posts);
});

It is simple, it works in development, and ?page=2 is easy to understand. That is why it is everywhere.

Why It Breaks

Offset pagination says "skip the first N rows, then give me the next 20". It counts positions, and positions move when the data does.

Duplicates. Someone publishes a new post while a reader is on page 1. Every existing post shifts down one position. When the reader requests page 2, the last item of page 1 is now the first item of page 2.

Skips. The reverse: a post is deleted while the reader is on page 1. Everything shifts up one, and the first item of what would have been page 2 slides into page 1's range. The reader never sees it.

Before:  [1 2 3 4 5 | 6 7 8 9 10]    page 1 = 1-5, page 2 = 6-10
New item arrives at the top:
After:   [0 1 2 3 4 | 5 6 7 8 9]     page 2 now starts with 5, which they already saw

The Second Problem: It Gets Slower on Deep Pages

skip(100000) does not jump to row 100,000. The database has to walk past 100,000 documents and throw them away before returning your 20. Page 1 is fast. Page 5,000 is slow, and the cost grows with the page number.

This is why a feed that feels fine with 1,000 rows slows down quietly at a million.

The Fix: Cursor Pagination

Instead of "skip N", say "give me the 20 items after this one". The anchor is a cursor: a value that identifies where the last page ended.

A new item arriving at the top cannot shift your place, because you are not counting positions. You are standing on a specific item.

Step 1: Sort by something stable, with a tie-breaker

createdAt alone is not enough, because two documents can share the same timestamp. Add _id as a tie-breaker so the order is always exact:

.sort({ createdAt: -1, _id: -1 })

Step 2: Support the query with an index

PostSchema.index({ createdAt: -1, _id: -1 });

With this index, the database jumps straight to the cursor and reads 20 documents, whether you are on page 1 or page 5,000.

Step 3: Query "after the cursor"

app.get('/api/posts', async (req, res) => {
  const limit = 20;
  const cursor = req.query.cursor ? decodeCursor(req.query.cursor) : null;

  const filter = cursor
    ? {
        $or: [
          { createdAt: { $lt: cursor.createdAt } },
          { createdAt: cursor.createdAt, _id: { $lt: cursor._id } },
        ],
      }
    : {};

  const rows = await Post.find(filter)
    .sort({ createdAt: -1, _id: -1 })
    .limit(limit + 1); // fetch one extra to know if there is a next page

  const hasMore = rows.length > limit;
  const items = hasMore ? rows.slice(0, limit) : rows;
  const last = items[items.length - 1];

  res.json({
    items,
    nextCursor: hasMore ? encodeCursor(last) : null,
  });
});

The $or says: "older than the cursor, or the same timestamp but a smaller _id". That second branch is what stops items with identical timestamps from being skipped or repeated.

Step 4: Encode the cursor so clients do not depend on it

const encodeCursor = (doc) =>
  Buffer.from(JSON.stringify({ createdAt: doc.createdAt, _id: doc._id })).toString('base64url');

const decodeCursor = (str) => {
  const { createdAt, _id } = JSON.parse(Buffer.from(str, 'base64url').toString());
  return { createdAt: new Date(createdAt), _id: new mongoose.Types.ObjectId(_id) };
};

Clients treat it as an opaque string and send it back as it is. You can change what is inside later without breaking anyone. In production, validate the decoded values and handle a malformed cursor with a 400, not a crash.

What You Give Up

Cursor pagination has a trade-off, and you should know it before switching:

No "jump to page 47". You can only go forward (or back, with a second cursor). For feeds and infinite scroll, that is perfect. For a numbered page bar, it is not.
No cheap total count. countDocuments on a large collection is not free either way, so many apps drop the "page 3 of 500" display.

When Offset Is Still Fine

Do not rewrite everything. Offset pagination is a reasonable choice for small, slow-changing tables, like an admin list with a few hundred rows, where users really do want page numbers.

Situation	Use
Feed, timeline, infinite scroll	Cursor
Large collection, deep pages	Cursor
Data changes while people browse	Cursor
Small admin table with page numbers	Offset is fine

The Rule

Offset pagination counts positions, and positions change. Cursor pagination anchors to a specific item, so it stays correct when data is added or removed. Always sort by a unique, indexed combination, and fetch one extra row to know whether there is a next page.

Quick audit: search your project for .skip(. For each hit, ask whether the collection can grow past a few thousand documents, or change while someone is paging through it.

If you work with MongoDB, I collect more patterns like this, around indexes and queries, in my MongoDB and Mongoose patterns guide, and you can browse my Next.js templates on pixelanas.com.

Get the templates: https://pixelanas.gumroad.com

Have you ever been bitten by duplicate or missing items in a paginated list? Tell me how you noticed in the comments.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

Multi-Tenant SaaS: Shared Database or One Per Tenant?

Wed, 21 Oct 2026 00:03:21 +0000

If you are building a SaaS product for more than one customer, you will have to answer one early question: how do you separate each customer's data? The choice between a shared database and a database per tenant shapes security, cost, performance and how painful your first enterprise customer will be.

I make this decision with founders as part of SaaS development work. This guide explains the three common models, when each fits, and the mistakes that are expensive to fix later.

What "multi-tenant" means

A tenant is one customer account, usually a company with its own users and data. A multi-tenant system serves many tenants from one codebase and one deployment. The alternative, running a separate copy of the whole app per customer, is single-tenant, and it gets hard to maintain quickly.

Multi-tenant design answers two questions:

Isolation: how do you make sure tenant A can never see tenant B's data?
Operations: how do you migrate, back up, scale and support many tenants without the work growing with every new customer?

The three data models

1. Shared database, shared schema

All tenants live in the same tables. Each row has a tenant_id column, and every query filters by it.

Strengths

Cheapest to run and simplest to start
One migration updates everyone
Easy to report across tenants

Weaknesses

One missed filter leaks data between customers
A very large tenant can slow everyone down ("noisy neighbour")
Restoring a single tenant from a backup is awkward

2. Shared database, separate schema per tenant

Tenants share a database server, but each gets its own schema, such as separate sets of tables.

Strengths

Stronger separation than a tenant column
Per-tenant backup and restore is easier

Weaknesses

Migrations must run once per tenant, which gets slow with many tenants
Tooling and connection handling are more complex

3. Separate database per tenant

Each tenant has its own database.

Strengths

Strongest isolation, which helps with security reviews and regulated customers
A tenant's data can be moved, backed up or deleted on its own
Heavy tenants don't affect the others' queries

Weaknesses

Highest operational cost and complexity
Migrations and monitoring multiply with each tenant
Cross-tenant reporting needs extra work

Microsoft's architecture guidance on tenancy models describes the same trade-off between isolation, cost and operational effort, and is worth reading alongside this.

How to choose: five questions

1. Who are your customers? Many small customers point toward a shared database. A few large or regulated customers who ask about data isolation point toward separate databases.

2. What are the compliance expectations? If customers need contractual guarantees about data separation, or data residency in specific regions, a separate database makes those promises easier to keep.

3. How uneven will tenants be? If one tenant could be a hundred times bigger than the average, plan for isolating heavy tenants early.

4. How big is the team? Every extra database is something to monitor, migrate and back up. A small team should choose the simplest model that meets the requirements.

5. How will you grow? Many products start shared and move their largest customers to dedicated databases later. A design that allows this is more valuable than a perfect choice on day one.

My default for early-stage SaaS

For most early products I start with a shared database and a tenant_id on every tenant-owned table, and I build the safeguards in from day one. This keeps cost and complexity low while the product finds its market, and it does not close the door on moving large customers out later.

The key is that "shared" must not mean "careless". The safeguards I add:

A global scope or middleware that applies the tenant filter automatically, so developers can't forget it
Database-level protection where the database supports it. PostgreSQL's row-level security can enforce tenant filtering even if application code has a bug.
Tenant-aware tests that fail if a query returns another tenant's rows
Composite unique keys that include the tenant, so two customers can use the same email or slug
Tenant context in logs and jobs, so background work never runs without knowing whose data it touches

If you use Laravel, packages such as Tenancy for Laravel support both single-database and multi-database approaches, which helps you start simple without being locked in.

Common mistakes

Forgetting background jobs. Queued jobs, scheduled tasks and webhooks run outside a web request, so the tenant context can be missing. Pass the tenant explicitly, as I describe in my guide to Laravel queue architecture.

Leaking through shared resources. Caches, search indexes and file storage need tenant-aware keys and paths, not only database rows. See secure multi-tenant file storage in Laravel.

Global unique constraints. Uniqueness rules that ignore the tenant make customers collide with each other.

No plan for super-admin access. Support staff need to view tenant data, and that access should be controlled and logged. My article on audit logging for SaaS compliance covers this.

Choosing separate databases too early. It feels safer but multiplies operations for a product that may only have a handful of customers.

Cross-tenant reporting as an afterthought. Analytics across tenants are easy in a shared schema and hard with many databases.

A real-world example

On Imperial Votes, a multi-tenant competition platform I built on Next.js and Prisma, organizers run their own competitions with branding tools, role-based permissions and dashboards, while super-admin controls sit above them. That mix of customers acting independently under one platform is exactly where the tenancy decision matters. The Imperial Votes case study shows how the product grew, and the matching write-up on building a multi-tenant voting SaaS goes deeper.

What to decide before you write code

Before the first migration, write down answers to these:

What is a tenant in our product (a company, a team, a workspace)?
Which tables belong to a tenant, and which are global?
How is the tenant identified on each request: subdomain, path or login?
How will background jobs and webhooks know the tenant?
What would force us to move a customer to a dedicated database?
How do support staff access tenant data safely?

Answering these costs an afternoon. Changing the model after launch costs months.

Moving a large tenant to its own database later

If you design for it, moving one customer out of the shared database is a manageable project rather than a rewrite. The usual steps:

Create the new database with the same schema and run the same migrations.
Copy the tenant's rows across, using the tenant_id to select exactly what belongs to them.
Verify counts and key records, and keep the old data until the checks pass.
Switch the tenant's connection so requests resolve to the new database.
Remove the tenant's rows from the shared database once you are confident.

This only works if every tenant-owned table carries the tenant identifier, files and caches are tenant-scoped, and your application resolves the database connection from the tenant. Those are the same safeguards that keep the shared model safe, which is why I add them from the start.

Key takeaways

Multi-tenancy is mostly about isolation and operational cost.
Shared database with a tenant column is the cheapest start. Separate databases give the strongest isolation.
Add safeguards such as global scopes, row-level security and tenant-aware tests so shared never means careless.
Plan for jobs, caches, files and support access, not only database tables.
Design so you can move large tenants out later.

If you are planning a SaaS product and want the architecture settled before you build, book a call and we can map the tenancy model to your customers and budget.

Originally published at aqibjavaid.site.

Demystifying MongoDB Write Concern and Read Concern in Node.js

Mon, 19 Oct 2026 09:22:10 +0000

What Are Write Concern and Read Concern?

In MongoDB, write concern and read concern are configuration settings that specify the level of guarantee you require when saving or retrieving data across a distributed cluster. A write concern determines how many database nodes must successfully record an update—and whether that data must be written to disk—before MongoDB reports success to your application. A read concern determines how fresh and consistent the data returned by a query must be, protecting your application from reading temporary data that might later be reversed.

Together, these two settings let developers choose where their application sits on the classic spectrum between maximum execution speed and rock-solid data safety.

The Real-World Analogy: Certified Mail vs. Kitchen Whiteboards

Imagine you run a bustling catering kitchen with multiple head chefs.

When taking an order, you have two choices for your write concern. You could scribble the customer's allergy request on a loose scrap of paper and immediately yell "Order placed!" (a fast write, but if you drop the paper, the order is lost). Alternatively, you could walk the order ticket over to the main order board, watch two sous-chefs sign off on it, and wait for them to file it in the metal lockbox before confirming the order with the customer. The second route takes several seconds longer, but you are guaranteed that the kitchen will never forget the order even if one chef walks out.

Now consider the read concern. If a waiter needs to know the soup of the day, they can either glance at a whiteboard where someone started drafting ideas with an erasable marker, or they can refuse to serve anything until they see the official menu stamped and approved by the head kitchen manager. Glancing at the draft whiteboard is instantaneous, but you might serve an experimental soup that the manager ultimately rejects five minutes later.

Why It Matters in Daily Software Engineering

Modern web architectures rarely run on a single database server. Instead, they use replica sets—groups of connected servers where one "primary" node handles writes and multiple "secondary" nodes replicate that data for redundancy.

If you leave MongoDB on default settings during high-traffic moments, subtle race conditions emerge:

Rollback Catastrophes: If an API endpoint confirms an e-commerce order after writing only to the primary node, and that primary node suddenly experiences a hardware failure before syncing to secondary nodes, the cluster elects a new primary. Your customer's order is wiped out, yet your payment processor already charged their credit card.
Dirty Reads: A background worker updates a bank balance. A microservice queries the secondary replica before the primary finishes replicating. If that transaction aborts or fails over, the microservice has operated on data that never officially existed.

By tuning write and read concerns, software engineers intentionally shield critical operations (payments, account transfers, credential updates) from data loss while letting low-risk operations (logging, view counters, user analytics) run with blazing speed.

Implementation in Node.js

Here is how you apply write and read concerns when working with the official MongoDB Node.js driver:

import { MongoClient } from 'mongodb';

const client = new MongoClient('mongodb://localhost:27017,localhost:27018/?replicaSet=rs0');

async function processPayment(orderId, paymentDetails) {
  await client.connect();
  const db = client.db('store');

  // Configure high-durability options for financial transactions
  const ordersCollection = db.collection('orders', {
    writeConcern: {
      w: 'majority',      // Must be acknowledged by more than half of the replica set nodes
      j: true,            // Must be written to the on-disk journal before confirming
      wtimeout: 5000      // Abort if replicas do not respond within 5 seconds
    },
    readConcern: {
      level: 'majority'   // Only read data committed by a majority of nodes
    }
  });

  // Step 1: Safely write with acknowledgment
  const writeResult = await ordersCollection.updateOne(
    { _id: orderId },
    { $set: { status: 'PAID', payment: paymentDetails } }
  );

  // Step 2: Read guaranteed, durable data
  const confirmedOrder = await ordersCollection.findOne({ _id: orderId });

  console.log('Payment processed safely:', confirmedOrder._id);
  return confirmedOrder;
}

In this example, specifying w: 'majority' and j: true ensures that the write cannot be rolled back if the primary server abruptly crashes. Setting level: 'majority' on the read ensures your application reads only finalized records.

Key Takeaway

Database speed and data reliability exist on a continuous slider: write concern dictates how hard your cluster works to preserve your updates, while read concern dictates how strictly your cluster verifies the stability of what you fetch. Mastering both allows you to protect financial transactions and vital data without unnecessarily throttling high-volume, low-risk endpoints across your stack.

Enjoyed this breakdown? Support my open-source work on Buy Me a Coffee or explore react-hook-lab!

Originally published on my blog. You can read the alternative breakdown here.

Database Migrations: Can Your Previous Release Still Run?

Mon, 19 Oct 2026 07:45:55 +0000

A deployment can roll back cleanly while the database stays incompatible with the old application. Before shipping a schema change, check whether the previous release can still read and write the data that the new release creates.

Consider a fictional account service. Version 1 reads a column called display_name. Version 2 expects public_name. Renaming the column and deploying the new code together looks tidy, until an old worker, a rolling-deployment instance, or an emergency rollback still asks for display_name.

The migration succeeded. The service can still fail.

Separate compatibility from cleanup

Danilo Sato’s explanation of parallel change describes three stages: expand, migrate, and contract. The useful idea is to introduce the new interface while the old one remains available, move consumers, then remove the old interface.

For a database rollout, turn that idea into explicit release gates:

Expand: introduce the new storage shape without removing what the currently deployed application needs.
Migrate: move reads and writes deliberately, checking compatibility during the transition.
Contract: remove the old shape only after its consumers and the intended rollback window are accounted for.

These stages are a planning model, not a guarantee of zero downtime. The exact migration still depends on the database, data volume, application behavior, and operational limits.

Work through the awkward middle

In the account-service example, adding public_name is only the beginning. Before copying values, decide which field is authoritative at each stage.

A possible transition is to deploy compatibility code that continues serving old readers while maintaining both values. If both columns live in the same row, update them in the same database transaction where practical. That still leaves an important question: can any old application instance or background job write only the original column?

Do not switch reads to the new column while those writers can silently make it stale. Either finish upgrading writers first or design a deliberate synchronization mechanism. A fallback read helps with missing values; it does not detect an outdated non-empty value.

Backfills need the same care. A naive copy can overwrite a newer value if it races with a live update. Define an update condition, concurrency strategy, and restart behavior before running the job. Record progress so an interrupted backfill can resume without blindly replaying the entire table.

Test the rollback with new data

A useful rehearsal includes more than deploying version 2 successfully:

Start with representative data, including empty and unusual values.
Apply the expansion and deploy the compatibility release.
Run the backfill alongside representative writes.
Create and edit records using the new release.
Restore the application version you actually intend to roll back to.
Verify that it can read and update those new records correctly.

Include background workers, scheduled jobs, exports, and direct database consumers in that exercise. A healthy web endpoint does not prove that every writer is compatible.

Use synthetic or appropriately protected test data. This checklist proposes evidence to collect; it does not claim that a production migration has been tested.

A compatible change can still block traffic

In PostgreSQL, ALTER TABLE generally acquires an ACCESS EXCLUSIVE lock unless the documentation specifies otherwise. Compatibility at the application level does not eliminate lock waits.

PostgreSQL also supports separating certain constraint additions from validation with NOT VALID and VALIDATE CONSTRAINT. The permitted constraint types and locking details matter. Check the documentation for your deployed version rather than applying the technique to every schema operation. See the PostgreSQL ALTER TABLE reference.

Before execution, define acceptable lock wait, query latency, replication lag, and backfill load. Decide who can stop the rollout and what the safe next step is if a threshold is crossed.

Make cleanup its own decision

Keep a short release note that answers four questions:

Which application versions can run against this schema?
Which data changes can the previous version understand?
What evidence allows the old column or interface to be removed?
After removal, does recovery mean application rollback, a forward fix, or data restoration?

Give the cleanup an owner. Leaving parallel paths indefinitely creates its own maintenance cost, but removing them too early destroys the compatibility you built.

The practical takeaway: rehearse rollback against data written by the new release before treating it as a recovery plan.

What is the most easily missed database consumer in your deployment process?

Waqas Ahmad | Software Engineer | Content Creator

Advanced SQL Analytics for Data Analysis.

Sun, 18 Oct 2026 13:54:35 +0000

Advanced SQL analytics for running a hospital database.

Introduction

Structured Query Language is essential and common in most industry sectors. In healthcare reporting, SQL plays a critical role in analytics. PostgreSQL lets data scientists take those transaction records and pull out operational insights, so you can make decisions. In this guide we go through the main SQL techniques, using a hospital database schema; patients , doctors, departments , wards, diagnoses, procedures, and visits.

First, before anything else we need to create our database schema. Connect to PostgerSQL DBMS and run a new sql script to start.

Create a schema hospital

create schema hospital;

Check to which schema you are connected.

show search_path;

If connected to a different schema, set connection to your hospital schema.

set search_path to hospital;

Now you are set to begin creating your tables. Run the following codes to create the different tables and insert data into the newly created tables:

Doctors Table

--create doctors table

create table hospital.doctors (
doctor_id int primary key,
doctor_name varchar(50) not null,
doctor_gender varchar(20),
doctor_specialty varchar(20),
doctor_experience_years int
);

select * from hospital.doctors;

-- insert data into doctors

insert into hospital.doctors (doctor_id, doctor_name, doctor_gender, doctor_specialty, doctor_experience_years)
values 
(101, 'DR.Brian Mwangi', 'Male', 'Cardiologist', 12),
(102, 'Dr. Sarah Wanjiku', 'Female', 'General Physician', 9), 
(103, 'Dr. Peter Otieno', 'Male', 'Orthopedic Surgeon', 15),
(104, 'Dr. Faith Chebet', 'Female', 'Pediatrician', 8);

Departments Table

-- create departments table

create table hospital.department (
department_id int primary key,
department_name varchar (50) not null
);


select * from hospital.department;

-- insert data into departments

insert into hospital.department (department_id, department_name)
values 
(10, 'Cardiology'),
(20, 'General Medicine'),
(30, 'Orthopedics'),
(40, 'Pediatrics');

Departments Table

-- create diagnosis table

create table hospital.diagnosis (
diagnosis_id int primary key,
diagnosis_name varchar(50),
diagnosis_category varchar(50)
);

select * from hospital.diagnosis;

--insert data into diagnosis table

insert into hospital.diagnosis (diagnosis_id, diagnosis_name, diagnosis_category)
values 
(501,'Hypertension', 'Chronic Disease'),
(502, 'Malaria', 'Infectious Disease'),
(503, 'Fracture', 'Injury'),
(504, 'Pneumonia', 'Respiratory Disease'),
(505, 'Diabetes', 'Chronic Disease'),
(506, 'Typhoid', 'Infectious'),
(507, 'Asthma', 'Respiratory Disease'),
(508, 'Back Pian', 'Musculoskeletal'),
(509, 'Gastritis', 'Digestive Disease'),
(510, 'Heart Failure', 'Chronic Disease'),
(511, 'Flu', 'Respiratoey Disease'),
(512, 'Kidney Infection', 'Infectious Disease'),
(513, 'Anaemia', 'Blood Disorder'),
(514, 'Sprain', 'Injury'),
(515, 'Gastroenteritis', 'Digestive Disease'),
(516, 'Dislocation', 'Injury'),
(517, 'Migraine', 'Neurological Disorder'),
(518, 'Chest Pain', 'Cardiovascular Disease');

Wards Table

-- create wards table 

create table hospital.wards (
ward_id int primary key, 
ward_name varchar (50)
);

-- insert data into wards
insert into hospital.wards (ward_id, ward_name)
values 
(100, 'Cardiac Care Unit'),
(200, 'General Ward'),
(300, 'Surgical Ward'),
(400, 'Childrens Ward');

select * from hospital.wards;

Patients Table

-- create patients table 
create table hospital.patients (
patient_id serial primary key, 
patient_name varchar(50) not null,
patient_gender varchar(20),
patient_age int,
county varchar(30),
sub_county varchar(30)
);

--insert data into patients
insert into hospital.patients (patient_name, patient_gender, patient_age, county, sub_county)
values 
('Amina Otieno', 'Female', 34, 'Nairobi', 'Westlands'),
('John Kariuki', 'Male', 42,' Kiambu', 'Thika'),
('Grace Naliaka', 'Female', 28, 'Kakamega', 'Lurambi'),
('David Mutua', 'Male', 55, 'Machakos', 'Athi River'),
('Esther Achieng', 'Female', 63, 'Kisumu', 'Kisumu Central'),
('Mohamed Ali', 'Male', 19, 'Mombasa', 'Nyali'),
('Lucy Wambui', 'Female', 7, 'Nakuru', 'Naivasha'),
('Samuel Kiptoo', 'Male', 39, 'Uasin Gishu', 'Eldoret East'),
('Mary Wairumu', 'Female', 31, 'Murang''a', 'Kiharu'),
('Peter Ouma', 'Male', 46, 'Siaya', 'Bondo'),
('Brian Omondi', 'Male', 24, 'Kisumu', 'Kisumu East'),
('Catherine Musyoka', 'Female', 52, 'Kitui', 'Kitui Central'),
('Daniel Maina', 'Male', 16, 'Nyeri', 'Othaya'),
('Eunice Moraa', 'Female', 44, 'Kisii', 'Bonchari'),
('Francis Njuguna', 'Male', 60, 'Laikipia', 'Nyanyuki'),
('Hellen Jepchirchir', 'Female', 37, 'Bomet', 'Sotik'),
('Isaac Kamau', 'Male', 29, 'Nairobi', 'Kasarani'),
('Jane Muthoni', 'Female', 22, 'Meru', 'Imenti North'),
('Kevin Barasa', 'Male', 48, 'Bungoma', 'Webuye'),
('Lilian Adhiambo', 'Female', 26, 'Homa Bay', 'Rangwe'),
('Moses Cheruiyot', 'Male', 33, 'Kericho', 'Ainamoi'),
('Nadia Hussein', 'Female', 5, 'Garissa', 'Dadaab'),
('Patrick Muli', 'Male', 51, 'Makueni', 'Wote'),
('Ruth Nekesa', 'Female', 41, 'Trans Nzoia', 'Kitale'),
('Josephine Wekesa', 'Female', 36, 'Busia', 'Matayos');

select * from hospital.patients;

Procedures Table

create table hospital.procedures (
procedure_id int primary key,
procedure_name varchar (50) not null,
procedure_category varchar (20)
);

--insert data into procedures
insert into hospital.procedures (procedure_id, procedure_name, procedure_category)
values
(701, 'Blood Pressure Monitoring', 'Diagnostic'),
(702, 'Malaria Test', 'Laboratory'),
(703, 'X-ray', 'Imaging'),
(704, 'Chest X-ray', 'Imaging'),
(705, 'Blood Sugar Test', 'Laboratory'),
(706, 'Full Blood Count', 'Laboratory'),
(707, 'Nebulization', 'Treatment'),
(708, 'Physiotherapy', 'Treatment'),
(709, 'Endoscopy', 'Diagnostic'),
(710, 'ECG', 'Diagnostic'),
(711, 'Consultation', 'Consultation'),
(712, 'Joint Support Bandage', 'Treatment'),
(713, 'IV Fluids', 'Treatment'),
(714, 'Closed Reduction', 'Treatment'),
(715, 'Pain Management', 'Treatment');


select * from hospital.procedures;

Visits Table

-- create visits table (Fact Table)

create table hospital.visits (
visit_id int primary key,
visit_date date not null,
patient_id int not null references hospital.patients(patient_id),
doctor_id int not null references hospital.doctors(doctor_id),
department_id int not null references hospital.department(department_id),
ward_id int references hospital.wards(ward_id),
diagnosis_id int references hospital.diagnosis(diagnosis_id),
procedure_id int references hospital.procedures(procedure_id),
admission_type varchar(20) not null
    check (admission_type in ('Outpatient', 'Inpatient', 'Emergency')),
length_of_stay_days int default 0
    check (length_of_stay_days >= 0),
triage_level varchar(10)
    check (triage_level in ('Red', 'Orange', 'Yellow', 'Green')),
insurance_provider varchar(50),
total_bill_kes numeric(10,2) not null
    check (total_bill_kes >= 0),
payment_method varchar(20),
follow_up_required varchar(3)
    check (follow_up_required in ('Yes', 'No')),
patient_satisfaction_score numeric(3,1)
    check (patient_satisfaction_score between 0 and 5)
);

--insert data into visits table
insert into hospital.visits (visit_id, visit_date, patient_id, doctor_id, department_id, ward_id, diagnosis_id,
procedure_id, admission_type, length_of_stay_days, triage_level, insurance_provider, total_bill_kes, payment_method,
follow_up_required, patient_satisfaction_score)
values
(1001, '2026-06-01', 1, 101, 10, 100, 501, 701, 'Outpatient', 0, 'Green', 'NHIF', 3500, 'Cash', 'Yes', 4.5),
(1002, '2026-06-01', 2, 102, 20, 200, 502, 702, 'Outpatient', 0, 'Yellow', 'Britam', 2500, 'Insurance', 'No', 4.1),
(1003, '2026-06-02', 3, 103, 30, 300, 503, 703, 'Inpatient', 3, 'Red', 'AAR', 28000, 'Insurance', 'Yes', 4.3),
(1004, '2026-06-02', 4, 104, 40, 400, 504, 704, 'Inpatient', 4, 'Red', 'None', 42000, 'Cash', 'Yes', 3.9),
(1005, '2026-06-03', 5, 101, 10, 100, 505, 705, 'Outpatient', 0, 'Yellow', 'CIC', 4000, 'Mobile Money', 'Yes', 4.7),
(1006, '2026-06-03', 6, 102, 20, 200, 506, 706, 'Outpatient', 0, 'Green', 'None', 3000, 'Cash', 'No', 4.0),
(1007, '2026-06-04', 7, 104, 40, 400, 507, 707, 'Outpatient', 0, 'Yellow', 'NHIF', 5500, 'Insurance', 'Yes', 4.6),
(1008, '2026-06-04', 8, 103, 30, 300, 508, 708, 'Outpatient', 0, 'Green', 'None', 6500, 'Cash', 'Yes', 4.2),
(1009, '2026-06-05', 9, 102, 20, 200, 509, 709, 'Outpatient', 0, 'Yellow', 'Jubilee', 15000, 'Insurance', 'No', 3.8),
(1010, '2026-06-05', 10, 101, 10, 100, 510, 710, 'Inpatient', 5, 'Red', 'AAR', 76000, 'Insurance', 'Yes', 4.4),
(1011, '2026-06-06', 1, 101, 10, 100, 505, 705, 'Outpatient', 0, 'Green', 'NHIF', 4200, 'Mobile Money', 'Yes', 4.8),
(1012, '2026-06-06', 11, 102, 20, 200, 502, 702, 'Outpatient', 0, 'Green', 'None', 2300, 'Cash', 'No', 4.0),
(1013, '2026-06-07', 12, 101, 10, 100, 501, 710, 'Outpatient', 0, 'Yellow', 'Britam', 7000, 'Insurance', 'Yes', 4.5),
(1014, '2026-06-07', 13, 104, 40, 400, 511, 711, 'Outpatient', 0, 'Green', 'None', 1800, 'Cash', 'No', 4.1),
(1015, '2026-06-08', 14, 103, 30, 300, 503, 703, 'Inpatient', 2, 'Red', 'CIC', 24000, 'Mobile Money', 'Yes', 4.6),
(1016, '2026-06-08', 15, 102, 20, 200, 512, 706, 'Inpatient', 3, 'Orange', 'Jubilee', 36000, 'Insurance', 'Yes', 3.7),
(1017, '2026-06-09', 16, 101, 10, 100, 510, 710, 'Inpatient', 6, 'Red', 'AAR', 82000, 'Insurance', 'Yes', 4.2),
(1018, '2026-06-09', 17, 102, 20, 200, 509, 711, 'Outpatient', 0, 'Green', 'None', 2000, 'Cash', 'No', 4.0),
(1019, '2026-06-10', 18, 104, 40, 400, 507, 707, 'Outpatient', 0, 'Yellow', 'NHIF', 5200, 'Mobile Money', 'Yes', 4.7),
(1020, '2026-06-10', 19, 103, 30, 300, 508, 708, 'Outpatient', 0, 'Green', 'Britam', 6500, 'Insurance', 'No', 4.1),
(1021, '2026-06-11', 2, 101, 10, 100, 501, 701, 'Outpatient', 0, 'Yellow', 'Britam', 3800, 'Insurance', 'Yes', 4.3),
(1022, '2026-06-11', 20, 102, 20, 200, 513, 706, 'Outpatient', 0, 'Green', 'None', 3200, 'Mobile Money', 'No', 4.4),
(1023, '2026-06-12', 21, 103, 30, 300, 514, 712, 'Outpatient', 0, 'Yellow', 'NHIF', 7200, 'Cash', 'Yes', 4.0),
(1024, '2026-06-12', 22, 104, 40, 400, 515, 713, 'Inpatient', 2, 'Orange', 'None', 18500, 'Cash', 'Yes', 4.5),
(1025, '2026-06-13', 5, 101, 10, 100, 501, 710, 'Outpatient', 0, 'Yellow', 'CIC', 7600, 'Insurance', 'Yes', 4.6),
(1026, '2026-06-13', 23, 102, 20, 200, 506, 706, 'Inpatient', 2, 'Orange', 'Jubilee', 29000, 'Insurance', 'Yes', 3.9),
(1027, '2026-06-14', 24, 103, 30, 300, 516, 714, 'Inpatient', 1, 'Red', 'AAR', 33000, 'Insurance', 'Yes', 4.2),
(1028, '2026-06-14', 7, 104, 40, 400, 511, 711, 'Outpatient', 0, 'Green', 'NHIF', 1900, 'Mobile Money', 'No', 4.8),
(1029, '2026-06-15', 25, 102, 20, 200, 517, 715, 'Outpatient', 0, 'Yellow', 'Britam', 4800, 'Cash', 'Yes', 4.1),
(1030, '2026-06-15', 10, 101, 10, 100, 518, 710, 'Emergency', 1, 'Red', 'AAR', 22000, 'Insurance', 'Yes', 4.4);

select * from hospital.visits;

After creating our database schema, we will begin modelling the data using different advanced SQL functions.

1. SQL Window Functions

Window functions do computations on related rows in a table while maintaining the individuality of each rows.

OVER, PARTITION BY, ORDER BY, and Window Frames

The OVER clause describes the rows for the window function operation.

PARTITION BY defines different partitions for each result set.
ORDER BY defines row order in each partition.
Window frames define different limits such asROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW for cumulative sums and ROWS BETWEEN 2 PRECEDING AND CURRENT ROW for moving averages.

Ranking Window Functions

Ranking functions provide numeric ordering according to specified criteria.

ROW_NUMBER provides a distinct numeric integer value to each row within a partition.
RANK provides identical rank values to ties, but skips other numbers.
DENSE_RANK provides identical rank values to ties, but does not skip numbers.
NTILE divides rows in partitions into buckets of equal size.

Question: How can hospital management rank inpatient visits by total bill within each admission type?

select 
    v.visit_id,
    p.patient_name,
    v.admission_type,
    v.total_bill_kes,
    row_number() over (partition by v.admission_type order by v.total_bill_kes desc) as row_num,
    rank() over (partition by v.admission_type order by v.total_bill_kes desc) as rnk,
    dense_rank() over (partition by v.admission_type order by v.total_bill_kes desc) as dense_rnk,
    ntile(2) over (partition by v.admission_type order by v.total_bill_kes desc) as bill_tier
from hospital.visits v
join hospital.patients p on v.patient_id = p.patient_id
where v.admission_type = 'Inpatient'
limit 5;

Expected Results

Explanation

ROW_NUMBER gives values from 1 to 5.
The values obtained by RANK and DENSE_RANK will be same as there are no duplicate bill values.
NTILE assigns the best bills to Tier 1.

Navigation functions retrieve values based on relative rows.

LAG retrieves values from previous row.
LEAD retrieves values from next row.
FIRST_VALUE gives the first value of the expression in the frame.
LAST_VALUE gives the last value of the expression in the frame bounds like ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING.

Question: How can we monitor patient billing process through multiple hospital visits?

select 
    v.visit_id,
    v.patient_id,
    v.visit_date,
    v.total_bill_kes,
    lag(v.total_bill_kes, 1, 0.00) over (partition by v.patient_id order by v.visit_date) as prev_bill,
    lead(v.total_bill_kes, 1, 0.00) over (partition by v.patient_id order by v.visit_date) as next_bill,
    first_value(v.total_bill_kes) over (partition by v.patient_id order by v.visit_date) as initial_bill,
    last_value(v.total_bill_kes) over (
        partition by v.patient_id 
        order by v.visit_date 
        rows between unbounded preceding and unbounded following
    ) as final_bill
from visits v
where v.patient_id in (1, 2)
order by v.patient_id, v.visit_date;

Expected Results

Explanation

LAG retrieves previous visit invoices and LEAD retrieves future visit invoices.
FIRST_VALUE and LAST_VALUE store the first and last visit invoices respectively.

Aggregate Window Functions

Aggregate functions SUM, AVG, COUNT, MIN, and MAX calculate moving sums and moving averages when combined with OVER.

Question: In what way do we determine cumulative income, rolling average billings, categories, and billings for departments on each visit?

select 
    v.visit_id,
    v.visit_date,
    v.total_bill_kes,
    sum(v.total_bill_kes) over (order by v.visit_date, v.visit_id) as cumulative_revenue,
    avg(v.total_bill_kes) over (
        order by v.visit_date 
        rows between 2 preceding and current row
    ) as rolling_avg_bill,
    count(*) over (partition by v.admission_type) as total_admission_type_count,
    min(v.total_bill_kes) over (partition by v.admission_type) as min_admission_bill,
    max(v.total_bill_kes) over (partition by v.admission_type) as max_admission_bill
from hospital.visits v
order by v.visit_date, v.visit_id
limit 5;

Expected Results

Explanation

SUM forms a running total.
AVG calculates a 3 row moving average.
COUNT, MIN, and MAX present group statistics without collapsing rows.

2. SQL Joins

Joins combine columns from multiple tables using related key columns.

INNER JOIN

INNER JOIN returns rows with matching key values in both tables.

Question: How do we generate a visit report combining patient names, attending doctors, and clinical departments?

select 
    v.visit_id,
    p.patient_name,
    d.doctor_name,
    dept.department_name,
    v.total_bill_kes
from hospital.visits v
inner join hospital.patients p on v.patient_id = p.patient_id
inner join hospital.doctors d on v.doctor_id = d.doctor_id
inner join hospital.department dept on v.department_id = dept.department_id
limit 5;

Expected Results

Explanation

INNER JOIN Matches keys across visits, patients, doctors, and department tables.

LEFT JOIN

The LEFT JOIN produces all the rows from the left table and the matched rows from the right table. Fields in the right table without a match produce NULL.

Question: How can we study the number of visits and expenses of patients without losing those that have no visits at all?

select 
    p.patient_id,
    p.patient_name,
    count(v.visit_id) as total_visits,
    coalesce(sum(v.total_bill_kes), 0.00) as total_spent_kes
from hospital.patients p
left join hospital.visits v on p.patient_id = v.patient_id
group by p.patient_id, p.patient_name
order by total_visits desc, p.patient_id
limit 5;

Expected Results

Explanation

All patient records are retained using LEFT JOIN.
Patients having no visits will have COUNT as 0 while COALESCE will convert NULL sums to 0.00.

RIGHT JOIN

RIGHT JOIN retrieves all records from right table and matching records from left table.

Question: How can hospital administration ensure that all physical wards are reflected in admission records?

select 
    w.ward_id,
    w.ward_name,
    v.visit_id,
    v.total_bill_kes
from hospital.visits v
right join hospital.wards w on v.ward_id = w.ward_id
where v.visit_id is null or v.visit_id <= 1003;

Expected Results

Explanation

All wards are kept in RIGHT JOIN.
The Childrens Ward will be with NULL visits if no match occurs.

FULL OUTER JOIN

FULL OUTER JOIN will display all records if there is a match in either left or right record.

Question: How do we perform auditing of doctor and department records for unassigned doctors or unmapped departments?

select 
    d.doctor_name,
    dept.department_name
from hospital.doctors d
full outer join hospital.department dept on d.doctor_id = dept.department_id;

Expected Results

Explanation

Null values will be returned in the rows where there is no relationship between doctor and department IDs.

CROSS JOIN

CROSS JOIN results in a Cartesian product where each row in the first table is paired up with each row of the second table.

Question: How can hospital managers get a complete list of all pairings of physicians with hospital wards for coverage planing?

select 
    d.doctor_name,
    w.ward_name
from hospital.doctors d
cross join hospital.wards w
order by d.doctor_name, w.ward_name
limit 5;

Expected Results

Explanation

Joining 4 doctors with 4 wards results in 16 combination rows.

SELF JOIN

SELF JOIN involves joining a table to itself for comparing rows in the same table.

Question: In what way can readmission of patients be identified by matching their first visit to their next visit?

select 
    v1.patient_id,
    v1.visit_id as original_visit,
    v1.visit_date as original_date,
    v2.visit_id as return_visit,
    v2.visit_date as return_date
from hospital.visits v1
inner join hospital.visits v2 on v1.patient_id = v2.patient_id and v1.visit_id < v2.visit_id
order by v1.patient_id
limit 5;

Expected Results

Explanation

Linking visits on patient ID where v1.visit_id is less than v2.visit_id matches distinct visits per patient.

Join Summary

INNER JOIN matches rows in both tables.
LEFT JOIN keeps all left table rows.
RIGHT JOIN keeps all right table rows.
FULL OUTER JOIN keeps all rows from both tables.
CROSS JOIN produces Cartesian pairs.
SELF JOIN compares rows within one table.

3. Common Table Expressions (CTEs)

A CTE is a named temporary table created within a WITH clause.

Simple CTEs

Question: What approach should be used to identify costly inpatient treatments and connect them with patients and their physicians?

with high_cost_visits as (
    select 
        visit_id,
        patient_id,
        doctor_id,
        total_bill_kes,
        admission_type
    from hospital.visits
    where total_bill_kes >= 35000.00
)
select 
    h.visit_id,
    p.patient_name,
    d.doctor_name,
    h.total_bill_kes
from high_cost_visits h
join hospital.patients p on h.patient_id = p.patient_id
join hospital.doctors d on h.doctor_id = d.doctor_id;

Expected Results

Explanation

CTE will filter out high cost visits first to keep the subsequent join clean.

Multiple Chained CTEs

Question: How do we find out department revenues, overall average, and top performing departments?

with dept_revenue as (
    select 
        department_id,
        sum(total_bill_kes) as total_revenue,
        count(visit_id) as total_visits
    from hospital.visits
    group by department_id
),
dept_benchmark as (
    select 
        avg(total_revenue) as benchmark_avg_revenue
    from dept_revenue
)
select 
    d.department_name,
    dr.total_revenue,
    dr.total_visits,
    round(db.benchmark_avg_revenue, 2) as benchmark_avg
from dept_revenue dr
join hospital.department d on dr.department_id = d.department_id
cross join dept_benchmark db
where dr.total_revenue > db.benchmark_avg_revenue;

Expected Results

Explanation

dept_revenue calculates the total revenue for each department, dept_benchmark calculates the average and the last SELECT statement filters departments whose revenues are greater than the average.

Recursive CTEs

Recursive CTEs create a sequence by means of an anchor query and recursive query linked by UNION ALL operator.

Question: How can the hospital management create a biweekly follow up schedule?

with recursive follow_up_schedule as (
    select 
        1 as cycle_number,
        cast('2026/07/01' as date) as scheduled_date
    union all
    select 
        cycle_number + 1,
        cast(scheduled_date + interval '14 days' as date)
    from follow_up_schedule
    where cycle_number < 4
)
select * from follow_up_schedule;

Expected Results

Explanation

Cycle number 1 starts from July 1 2026 as the anchor date, and the recursive query keeps adding 14 days till cycle number is 4.

CTEs vs. Subqueries

CTEs provide well-defined top-level modules, which can be used again and again, while subqueries within the WHERE or FROM clause tend to complicate code and cannot be executed recursively.

4. SQL Subqueries

A subquery is a query nested inside another SQL query.

Subquery Types

Scalar Subquery fetches one row and one column.
Non-correlatedSubquery executes only once and independently.
Correlated Subquery uses columns of outer query and executes for every row of outer query.
Nested Subquery uses subqueries in subqueries.

Subqueries in SELECT Clauses

Question: How can individual patient bills be shown along with the total hospital average bill?

select 
    visit_id,
    total_bill_kes,
    (select round(avg(total_bill_kes), 2) from hospital.visits) as overall_avg_bill
from hospital.visits
where visit_id between 1001 and 1005;

Expected Results

Explanation

Scalar subquery evaluates to average 16266.67 for all rows.

Using Subqueries in FROM Clauses (Derived Table)

Question: How do we query the pre-aggregated metrics of patients using an inline derived table?

select 
    patient_summary.patient_id,
    patient_summary.visit_count,
    patient_summary.total_spent
from (
    select 
        patient_id,
        count(visit_id) as visit_count,
        sum(total_bill_kes) as total_spent
    from hospital.visits
    group by patient_id
) as patient_summary
where patient_summary.visit_count > 1;

Expected Results

Explanation

The FROM subquery groups the number of visits per patient into table alias patient_summary, subject to WHERE.

Subqueries in WHERE Clause

Question: What is the correlated subquery to find visits where the total bill is greater than the average bill of that particular department?

select 
    v1.visit_id,
    v1.patient_id,
    v1.department_id,
    v1.total_bill_kes
from hospital.visits v1
where v1.total_bill_kes > (
    select avg(v2.total_bill_kes)
    from hospital.visits v2
    where v2.department_id = v1.department_id
)
limit 5;

Expected Results

Explanation

The inner query calculates the average billing amount for each department based on outer query department IDs.

Question: Which physicians have admitted any patients as inpatients?

select 
    d.doctor_id,
    d.doctor_name,
    d.doctor_specialty
from hospital.doctors d
where exists (
    select 1 
    from hospital.visits v 
    where v.doctor_id = d.doctor_id 
      and v.admission_type = 'inpatient'
);

Expected Results

Explanation

EXISTS returns true when there is a row for a visit that matches.

Question: How can we find visits where the billing value is greater than all the outpatient visit bills?

select 
    visit_id,
    patient_id,
    total_bill_kes,
    admission_type
from hospital.visits
where total_bill_kes > all (
    select total_bill_kes
    from hospital.visits
    where admission_type = 'outpatient'
)
limit 5;

Expected Results

Explanation

The internal query locates outpatient charges up to 15000.00. ALL filters visits more than 15000.00.

Conclusion

Understanding the concepts of window functions, joins, common table expressions, and subqueries helps in creating strong PostgreSQL queries that enable healthcare workers to review their operations and enhance financial efficiency. Such information enables healthcare providers to offer quality pateint care which meets patient outcomes.

GitHub

For the completed project visit my GitHub project Hospital Database

How to Back Up and Restore Self-Hosted n8n

Mon, 12 Oct 2026 15:50:00 +0000

A recoverable self-hosted n8n backup has four core parts: the database, the original encryption key, the deployment configuration and secrets, and any binary-data storage your executions still need. Back up the persistent .n8n volume even when PostgreSQL stores the main database. n8n documents that this directory can still contain the encryption key, logs, and source-control assets.

Partner disclosure. This guide includes a partner link to n8n Cloud. If you sign up through it, I may earn a commission at no extra cost to you.

Short answer: workflow JSON files alone are not enough. They do not recreate the complete instance state, and encrypted credentials are unusable without the key that encrypted them.

What must be in the backup

Asset	Why it matters	Typical location
Database	Workflows, encrypted credentials, users, projects, settings, and retained execution data	`~/.n8n/database.sqlite` or PostgreSQL
Encryption key	Decrypts the credentials stored in the database	`~/.n8n` settings or `N8N_ENCRYPTION_KEY`
Deployment config	Restores database connection, public URL, proxy, timezone, pruning, and execution behavior	Compose file, environment settings, secret references, proxy config
Binary data	Restores retained files handled by executions when filesystem or external storage is used	Persistent filesystem, database, or configured external store
Custom and community nodes	Allows restored workflows to load the same node types and versions	Package list, custom-node directory, container image, or build manifest

Do not put the raw encryption key, database password, OAuth secrets, or an unredacted environment file in a public repository. Keep a sanitized deployment definition in version control and protect the actual secrets in a restricted password manager, secret store, or encrypted backup.

Choose the database path before writing the runbook

Self-hosted n8n uses SQLite by default. The database file is ~/.n8n/database.sqlite. PostgreSQL is the other supported database option. The backup and restore commands must match the database your instance actually uses.

SQLite: capture a consistent persistent volume

The simplest small-instance method is a short maintenance window: stop the n8n application, archive its persistent volume, then start it again. Stopping writes avoids treating an arbitrary live file copy as a consistent database backup. For the standard Docker volume shown in n8n's installation guide, a generic sequence looks like this:

docker compose stop n8n
docker run --rm \
  -v n8n_data:/data:ro \
  -v "$PWD/backups:/backup" \
  alpine tar czf /backup/n8n-data-2026-07-18.tgz -C /data .
docker compose start n8n

Replace the service and volume names with the names in your deployment. If downtime is not acceptable, use a storage snapshot or SQLite-aware backup process that guarantees a consistent result. Do not assume that copying database.sqlite while executions are writing to it is safe.

PostgreSQL: use a database dump

PostgreSQL's official documentation recommends pg_dump for logical backups. A custom-format dump works with pg_restore and is practical for restoring into a fresh database. Adapt the connection values and run the tool from a host or database container that can reach PostgreSQL:

pg_dump --format=custom --file=n8n-2026-07-18.dump n8n

The dump does not replace the n8n persistent volume or explicit encryption key. It also does not capture the Compose file, environment settings, proxy configuration, or filesystem binary data. Store those as separate parts of the same dated backup set.

Add native n8n exports as a second recovery layer

n8n's Server CLI can export all database entity types. Its documentation positions this tooling for backups and migrations, including moves between SQLite and PostgreSQL. Execution-history data tables are excluded by default because they can be large, and can be included explicitly when your retention requirement needs them.

docker exec -u node n8n \
  n8n export:entities \
  --outputDir=/home/node/.n8n/cli-backup

Workflow and credential exports are also useful for selective recovery or versioned copies:

n8n export:workflow --backup --output=backups/workflows/
n8n export:credentials --backup --output=backups/credentials/

Avoid decrypted credential exports for routine backups. n8n supports a --decrypted flag for migrations to a different secret key, but the resulting files expose every sensitive value in plain text. If that exceptional path is necessary, isolate, encrypt, restrict, and delete the files after the migration.

A safe restore runbook

First restore into an isolated instance with no production webhooks, schedules, email sends, payment calls, or CRM writes. The first recovery test should never overwrite the only production copy.

Record the target. Use the same n8n version first, the same database type, and compatible custom-node versions. Upgrade only after the restored instance works.
Create fresh infrastructure. Prepare a new Docker volume or empty PostgreSQL database rather than clearing production.
Restore the original encryption key. Put the protected N8N_ENCRYPTION_KEY or original n8n settings in place before n8n reads the restored credentials.
Restore the database. Extract the stopped SQLite volume archive into the new volume, or restore the PostgreSQL dump into the new database.
Restore the surrounding state. Reapply deployment settings, binary storage, custom nodes, public URL settings, and reverse-proxy configuration.
Start without production traffic. Keep DNS, proxy routing, and outbound side effects isolated while checking startup and migration logs.
Verify the application. Confirm login, projects, workflows, credentials, node availability, execution history required by policy, and binary files that should still exist.
Run one safe test. Use test credentials or a non-destructive workflow and confirm that a credential can decrypt and authenticate.
Publish deliberately. Review which workflows should be published before switching traffic. Imported workflows can have different active-state behavior depending on the import path and deployment mode.

Restore a SQLite archive into a new volume

A new volume keeps the original recoverable while you inspect the restore. This example extracts the archive created above without deleting the production volume:

docker volume create n8n_restore_data
docker run --rm \
  -v n8n_restore_data:/data \
  -v "$PWD/backups:/backup:ro" \
  alpine tar xzf /backup/n8n-data-2026-07-18.tgz -C /data

Point a separate restore Compose file at n8n_restore_data. Do not bind it to the production hostname or webhook route until the checklist passes.

Restore PostgreSQL into a fresh database

createdb n8n_restore
pg_restore --dbname=n8n_restore n8n-2026-07-18.dump

Real deployments may need explicit host, user, role, ownership, TLS, and schema options. Test the exact command your operator will use and document it beside the backup job without embedding the password.

Restore-test checklist

The backup job has a timestamp, size, success state, and failure alert.
The database and encryption key come from the same recoverable setup.
The restore starts on a separate hostname, volume, and database.
Credential nodes open without decryption errors and one test authentication succeeds.
Required custom or community nodes load at the expected versions.
Published workflows, schedules, and webhook paths are reviewed before traffic moves.
Required retained binary files can be opened.
The restore time and missing manual steps are recorded for the next test.

Common backup failures

Mistake	What breaks during recovery
Only exporting workflow JSON	Users, projects, settings, retained executions, and complete credential state are not restored as one instance
Database without the original key	Credentials remain encrypted but cannot be used
Copying a live SQLite file casually	The backup may not represent one consistent database state
Skipping binary storage	Execution records may exist while retained documents or images are missing
Restoring straight over production	A bad archive or wrong configuration can remove the working recovery path
Never testing the restore	Missing keys, permissions, node packages, and undocumented steps appear during the incident

Common questions

Where does Docker store self-hosted n8n data?

n8n's standard Docker example mounts a named n8n_data volume at /home/node/.n8n. Your Compose file may use a different volume or bind mount, so inspect the deployment rather than assuming the example name.

Does exporting workflows back up credentials?

Not by itself. n8n provides separate workflow and credential export commands, plus the broader export:entities command. For same-instance disaster recovery, the database, original encryption key, config, and storage remain the stronger complete backup set.

Can I restore into a newer n8n version?

Restore the recorded version first when possible. This separates backup problems from application migrations. After the restored instance passes its checks, read the intervening release notes, take another backup, and perform the upgrade as a separate controlled change.

Sources checked

n8n partner link

Prefer a managed recovery path?

I may earn a commission if you sign up through this link, at no extra cost to you.

Try n8n Cloud

Need the recovery path documented?

Send the database type, deployment shape, storage locations, and recovery target. I will help map a backup set and an isolated restore test without exposing production secrets.

Map the restore

Originally published on FloxoLab.

Caching Strategies Demystified: Cache-Aside, Write-Through, and Write-Behind

Fri, 09 Oct 2026 10:55:06 +0000

In software development, caching is the practice of storing frequently used data in a high-speed, temporary storage layer so future requests can be served faster. Caching strategies are the architectural patterns that dictate how your application coordinates the movement of data between this high-speed temporary memory (the cache) and your permanent, slower storage system (the database). Choosing the right strategy ensures your application stays incredibly fast while keeping data accurate and consistent.

The Notebook and the Library Analogy

Imagine you are a researcher studying in a massive university library (the database) containing millions of books.

Cache-Aside (Lazy Loading): You keep a blank personal notebook (the cache) on your desk. When you need a fact, you look in your notebook first. If it is there (a cache hit), you read it instantly. If not (a cache miss), you walk into the library archives, search the shelves, find the fact, write it down in your notebook, and then read it.
Write-Through: Every time you discover a new piece of information, you write it in your personal notebook and immediately send a copy to the library's archiving desk to ensure both records are updated simultaneously before you proceed.
Write-Behind (Write-Back): When you are brainstorming rapidly, you scribble your notes exclusively in your notebook. At the end of the day, you hand your notebook to a library assistant who updates the main archives in the background while you sleep.

Why Caching Strategies Matter in the Tech Industry

Without these strategies, modern platforms would collapse. If a viral social media post forces millions of users to fetch the same user profile directly from a database, the database will exhaust its connection pool, slow down, and eventually crash.

Software engineers use these patterns strategically to balance speed and data integrity:

Cache-Aside prevents databases from being choked by heavy read traffic while keeping implementation simple.
Write-Through avoids data discrepancies, ensuring that critical data (like bank account balances) is never out of sync between the memory and the disk.
Write-Behind is utilized in write-heavy systems (like logging user activity or IoT sensor tracking) to group thousands of database writes into batch updates, protecting disk-based storage from wearing out or bottlenecking.

Implementing Cache-Aside in Node.js & Express

Here is how you can implement a standard Cache-Aside strategy using Express.js and MySQL. In this scenario, we check an in-memory cache object before querying our relational database.

const express = require('express');
const mysql = require('mysql2/promise');
const app = express();

// Mocking our fast memory cache (e.g., Redis)
const memoryCache = new Map();

// MySQL connection pool configuration
const dbPool = mysql.createPool({
  host: 'localhost',
  user: 'root',
  database: 'production_db'
});

// Route utilizing Cache-Aside
app.get('/api/users/:id', async (req, res) => {
  const userId = req.params.id;
  const cacheKey = `user:${userId}`;

  try {
    // 1. Check if the data exists in the high-speed cache
    if (memoryCache.has(cacheKey)) {
      console.log('Cache Hit! Serving from memory.');
      return res.json(memoryCache.get(cacheKey));
    }

    // 2. Cache Miss: Retrieve data from the slow MySQL database
    console.log('Cache Miss. Fetching from MySQL database...');
    const [rows] = await dbPool.query('SELECT id, name, email FROM users WHERE id = ?', [userId]);

    if (rows.length === 0) {
      return res.status(404).json({ error: 'User not found' });
    }

    const userData = rows[0];

    // 3. Write data to the cache so the next request is instant
    memoryCache.set(cacheKey, userData);

    return res.json(userData);
  } catch (error) {
    return res.status(500).json({ error: 'Internal Server Error' });
  }
});

app.listen(3000, () => console.log('Server running on port 3000'));

Key Takeaway

Caching is not a one-size-fits-all solution; it is a careful compromise between performance and accuracy. While caching can make your application feel instantaneous, selecting the wrong update strategy can lead to stale data displays or lost records, making it vital to match your architectural pattern to your application's specific read-and-write ratios.

Originally published on my blog. You can read the alternative breakdown here.

Instant at 10 Million Rows, Dead at a Billion: A Decision Framework for Billion-Row SQL

Fri, 09 Oct 2026 06:03:06 +0000

A query that returns in 40 milliseconds on ten million rows can time out completely on a billion. Same SQL, same schema, same database. I've watched a lot of teams hit that wall and go hunting for the one fix: a cleverer index, a bigger instance, whatever database was trending that month. There isn't one fix. There are two questions that decide almost everything. What engine class is this workload on — row-store, columnar, or distributed? And does the query's working set fit in memory, or does it spill to disk? Answer those honestly and most "billion-row problems" quietly resolve. Answer them wrong and no amount of tuning bails you out.

The query that was instant at 10 million rows and timed out at a billion

Here's the shape of it. It's a composite; I've seen it often enough that the specifics blur.

A team builds a dashboard on their main OLTP database. One query carries it: count distinct users, grouped by day, over a date range. Against a 10-million-row staging copy it comes back instantly, everyone signs off, ship it. Months later production crosses a billion rows — roughly the 1.1-billion-row scale in Mark Litwintschik's cross-engine benchmarks — and the identical query simply stops returning. On-call adds an index; nothing. A composite index; nothing. A bigger instance; somehow slightly worse. "Throw it at Spark," one person says. "Just use ClickHouse," says another. Nobody can explain why either would help. That's the real problem: everyone is arguing tactics with no framework for the decision.

The query didn't regress. It's doing exactly what it always did. What changed is that the workload crossed two lines at once — it outgrew the engine class that suited it at 10M, and its sorts and hashes stopped fitting in memory. Name those two lines and the mystery turns into a decision you can defend.

The two-axis framework

When a query dies at scale, the reflex is to index harder. Sometimes that's right. At a billion rows it's usually the wrong axis. Two levers actually decide whether a query survives.

Axis 1 — Engine class. Every database is built around one dominant access pattern, and that choice shapes everything downstream. A row-store OLTP engine (PostgreSQL, MySQL) keeps whole rows together and excels at "fetch or update this one record" and narrow ranges. A columnar/OLAP engine (ClickHouse, DuckDB, BigQuery) stores each column separately, so scanning and aggregating hundreds of millions of rows is the design point rather than the emergency. A distributed engine (Citus, Spark, Cassandra) spreads data across nodes for horizontal scale, trading query flexibility to get it. These are peers, not rungs on a ladder; none is "more advanced." Run a workload on the wrong one and you get a query that flies at 10M and dies at 1B.

Axis 2 — Memory budget. Independent of engine class, every non-trivial query builds intermediate structures: sort buffers, hash tables for GROUP BY and DISTINCT, join tables. Fit those in RAM and the operation runs at memory speed. Overflow the memory the engine is allowed, and it spills to disk and falls off a cliff. PostgreSQL ships shared_buffers at 128MB and work_mem at just 4MB; the docs are explicit that a sort or hash exceeding work_mem starts "writing to temporary disk files" (PostgreSQL 18 — Resource Consumption, as of 2026). Four megabytes is nothing at a billion rows. This is the axis most write-ups skip, and it's why the same query is fast one day and glacial the next. The SQL didn't change; the data just got big enough to spill.

The axes are orthogonal. You can pick the right engine class and still hit the memory cliff, or keep memory in check and still be on an engine that's wrong for the query's shape. So the move is: work out which quadrant you're actually in, then act on the axis that matches — change engine class, or change the memory budget — instead of adding another index to the axis that was never the problem.

Engine class 1: Row-store OLTP — great at point lookups, brutal on full-table aggregates

Row-stores are the workhorse for good reason. Ask for "order #48213 and its line items" or "the last 50 events for this user" and a row-store with the right index is hard to beat: it seeks to a handful of rows and returns. At 10M rows, the dashboard query was riding on that.

Aggregates over the whole table are a different story, and the reason is baked into how MVCC row-stores stay correct under concurrency. Because different transactions can see different versions of a row, the engine keeps no cached count of how many rows a table has. An exact count(*) therefore has to visit rows and check visibility — a sequential scan that grows linearly with the table.

The numbers are blunt. In Joe Nelson's Citus benchmark on counting (PostgreSQL 9.5.4, 2016 — treat the milliseconds as illustrative of the mechanism, which is unchanged today, not as current figures), an exact count(*) ran 85ms at 1M rows, 161ms at 2M, and 343ms at 4M: a straight line, with the scan about 88% of the cost. Extend that to a billion rows and a single count is tens of seconds before any filtering. count(1) doesn't rescue you; it measured slightly slower (99ms vs 85ms at 1M), because Postgres special-cases the argument-free count(*) while count(1) re-checks the argument on every row. And this isn't Postgres being difficult — MySQL's InnoDB shares the MVCC property and also has no O(1) table count.

When you don't need an exact number, stop asking for one. Every row-store keeps a maintained estimate for its planner, and you can read it directly. In PostgreSQL that's pg_class.reltuples (the Count-estimate pattern: SELECT reltuples::bigint FROM pg_class WHERE oid = 'schema.table'::regclass). In the same benchmark the estimate returned in about 0.3ms against 85ms for the exact count at 1M rows — roughly 280× faster, and effectively constant time at any size. The catch is accuracy: it's an estimate and can't apply a WHERE or DISTINCT. For "roughly how many rows" on a dashboard, that's the difference between a billion-row scan and reading one catalog row.

None of this means "Postgres is slow." It means a full-table aggregate is off-design for a row-store. Approximate counts, materialized rollups, and summary tables all soften it. But if scans and aggregates are your dominant pattern, you're fighting the engine class — and that's your first crossover signal.

Exact `count() vs the reltuples` estimate. Source: Citus / Joe Nelson, PostgreSQL 9.5.4, 2016 — illustrative of the mechanism, not current figures.*

Engine class 2: Columnar / OLAP — where scans and aggregates are the design point

If aggregates are killing your row-store, what's built for them, and how much faster is it really? This is where the framework gets counter-intuitive.

Columnar engines store each column contiguously instead of packing whole rows. A scan-and-aggregate query that touches 3 of 51 columns reads only those 3 off disk; the other 48 are never opened. And because a column holds one type with heavy repetition, it compresses hard, so there's less to read to begin with. Column pruning plus compression is why "scan a billion rows and aggregate" is routine here instead of an incident.

The headline number: Litwintschik's 1.1-billion-row taxi benchmark (1.1B rows, 51 columns, ~500GB uncompressed; page updated March 2024) runs one analytical query across dozens of engines. On Query 1, a single desktop ran ClickHouse in 0.088s and DuckDB in 0.498s. The same query took 2.36s on a 21-node Spark cluster, 3.54s on 21-node Presto, and 1–3s on BigQuery. Postgres with the cstore_fdw columnar extension took 152–368s across the four queries, and plain SQLite's format took 31,193s on Query 1 — about 8.7 hours, included to show what the wrong engine class actually costs.

A single desktop beat a 21-node cluster on a 1.1-billion-row scan by more than an order of magnitude. That one result is the whole framework in miniature. It isn't that ClickHouse or DuckDB is "the best database" — it's that for a scan-aggregate workload a columnar engine is in the right quadrant while the cluster pays coordination overhead it doesn't need at this size. (Caveats: one benchmark, one hardware generation, recent desktop CPUs for the fastest single-node runs. Trust the shape of the gap, not the exact milliseconds.)

Compression is the other half, and it's an economics lever as much as a speed one. ClickBench — the ClickHouse team's scan/aggregate benchmark across 60+ systems — publishes a live "Data Size" column so you can compare on-disk footprint on the same ~100M-row dataset. Columnar layouts routinely pack this kind of wide, repetitive data down by a large multiple versus a row-store, though the exact ratio moves with engine, codec, and data, so read the live figure for the systems you're weighing rather than trusting a headline number. (ClickBench is ClickHouse-run, so treat it as a scan-workload reference, not a neutral verdict.)

The honest counter-point: the same columnar engine that crushes the scan is the wrong tool for high-QPS single-row writes and point lookups. Insert a row and you touch every column store; update a field and compression works against you. The wide-column and key-value stores built for that job — Cassandra, ScyllaDB — sit at the opposite pole: excellent at point reads and writes on a known partition key, and structurally unfit for ad-hoc full-table aggregates. Cassandra's CQL docs say it plainly: CQL "only allows select queries that don't involve a full scan of all partitions," and forcing one needs ALLOW FILTERING, whose performance the docs call "unpredictable." A store modeled query-first around the partition key isn't a scan-aggregate engine, which is exactly why ClickBench lists Cassandra and ScyllaDB among the systems it couldn't benchmark on that suite. Right engine, wrong job, in both directions.

Same query, same 1.1B-row dataset, across engine classes. Source: Mark Litwintschik, 1.1 Billion Taxi Rides benchmarks, page updated March 2024.

Engine class 3: Distributed — horizontal scale, and the fan-out ceiling

When one node genuinely can't hold the data or serve the writes, distributed engines spread the work across many. The mental model most people carry — more nodes, more speed — holds right up until it doesn't, and knowing where that line sits is the difference between a defensible cluster and a single box with a network bill attached.

The line: does the query push down to the shards, or force a reshuffle? Distribute by tenant_id and group by tenant_id, and each node computes its slice while the coordinator stitches the results together. That's pushdown, and it parallelizes beautifully. But the moment a query needs data spread across nodes — a count(DISTINCT) on a non-distribution column, or a join across the distribution boundary — the engine has to reshuffle rows between workers first. That shuffle is the ceiling.

The Citus benchmark puts numbers on it (100M rows, 8 nodes, 32 shards; PG 9.5.4, 2016 — again mechanism-illustrative). A plain count(*) across the cluster ran about 1.2s. A count(DISTINCT) on the distribution column, which pushes down, ran about 3.4s. On a non-distribution column it can't push down, and pulling every row back to the coordinator is, in Nelson's words, "really no better than counting on a single database instance … plus high network overhead." An HLL approximate distinct ran 3.2–3.8s — the row-store's approximation trick again, now buying a way around the shuffle. This generalizes well past Citus: in Spark the expensive stage is almost always the shuffle, and distributed query design is mostly the craft of keeping the hot queries pushed down.

So distributed isn't a "scale" button. It's a trade — horizontal capacity for query flexibility and operational complexity. Align your access pattern with the distribution key and it's transformative. Keep needing cross-node data and you've bought nodes and network latency to arrive back at roughly single-box performance. That mismatch is its own crossover signal, and the one most likely to be discovered the hard way in production.

The memory / spill axis — the one most write-ups skip

Now the second axis, and the reason "just add an index" so often does nothing. A query can be on exactly the right engine class and still fall off a cliff, because its sorts and hash tables outgrew the memory the engine was allowed and spilled to disk. This is engine-independent: columnar and distributed engines build the same buffers and spill the same way when they overflow their limits. Postgres is just the easiest place to watch it, because its docs name the knob.

The spill cliff

At the default 4MB work_mem (PostgreSQL docs), a GROUP BY, DISTINCT, or ORDER BY that builds something larger switches from an in-memory hash or sort to an external merge that writes temp files to disk. The Citus benchmark shows the size of the drop (PG 9.5.4, 2016; mechanism unchanged per the current docs): at default work_mem, count(DISTINCT) on an integer column averaged 743ms, but on a text column 31,747ms — a ~43× gap — with Sort Method: external merge Disk in the plan. Give the operation enough work_mem to stay in RAM and the integer case drops to about 372ms as an in-memory HashAggregate. That in-RAM-versus-disk boundary is the axis. Stay on the right side and the query flies; cross it and it collapses.

This is exactly why indexing didn't help the composite team. An index changes how you find rows; it does nothing for a GROUP BY/DISTINCT whose result set is too big for work_mem. The lever there is memory, or a smaller working set — not another index.

When to re-architect instead of tune

work_mem is allocated per operation per query, and the docs warn total usage "could be many times" that value under load, so raising it globally just trades a spill problem for an out-of-memory one. Past a point the honest move is structural. PostgreSQL's partitioning guidance offers a clean rule: partition "when the size of the table [would] exceed the physical memory of the database server." Pruning then skips whole partitions — but it isn't free. The planner handles "up to a few thousand" partitions well only if pruning leaves few, since "each partition requires its metadata to be loaded into the local memory of each session that touches it." Thousands of poorly-pruned partitions is its own problem.

One more trap: deep pagination

LIMIT ... OFFSET n gets slower the further you page. As Markus Winand explains in "No Offset", the database "fetches and drops" all n preceding rows before returning the page, and results drift when rows are inserted between requests. A keyset (seek) query filters on the last-seen sort key instead and stays roughly constant-time however deep you go. Page latency that climbs with depth is a spill-adjacent signal, not a hardware one.

count(DISTINCT) at default vs raised work_mem. Source: Citus / Joe Nelson, PostgreSQL 9.5.4, 2016 — illustrative of the mechanism, not current figures.

The crossover-signal checklist: change engine class, don't add another index

This is the piece worth saving. A framework's job is to tell you when you've hit the boundary and it's time to move on an axis instead of tuning further. Eight signals I watch for, each tied to the axis it moves. Two or more firing means you're past tuning and into an engine-class or memory-budget decision.

Scans and aggregates dominate the workload (not point lookups). → Engine class, toward columnar. The row-store aggregate cliff; indexing won't fix an off-design pattern.
The working set exceeds server RAM. → Memory. Postgres's own partition threshold; partition, re-architect, or move to columnar/distributed.
Sorts and DISTINCTs keep spilling even after tuning work_mem. → Memory. You've hit the ceiling of per-query memory; the working set is the problem.
Pagination or scan latency rises with position. → Memory/scan; switch to keyset/seek before switching engines.
A single node has hit its write or IO ceiling. → Engine class, toward distributed. The one genuine "add nodes" signal.
Key DISTINCTs or joins need data from several nodes (a shuffle). → A warning on the distributed axis: without a distribution-key redesign, more nodes ≈ one box plus network.
You need thousands of partitions to stay fast. → Memory/metadata; the per-partition cost says a columnar or distributed store fits the data better.
You're keeping 500GB+ on one box for analytics. → Engine class plus economics; columnar compression and distributed storage costs start driving the decision.

What the list deliberately doesn't do is name a winning database. Each signal points to an axis and a direction, because the right specific engine depends on your query shapes, your team's ops maturity, and your budget. "Change engine class" is an architectural decision with a reason attached — something you can hand a skeptical peer or a budget owner instead of "the internet said use ClickHouse."

Cost and economics for the decision-makers

If you own the stack decision, latency is half the case; the half that gets budget approved is cost and defensibility. The benchmarks reframe cleanly as money.

A single big box can be cheaper and faster than a cluster. The Litwintschik result — one desktop-class node beating a 21-node cluster on a 1.1B-row scan — reads louder as a cost story than a speed one: 21 nodes carry ~21× the compute plus coordination, networking, and a standing ops burden, and still lose on this workload. For "big but not planetary" analytics, the first move is often the right engine class on one well-sized box, not a cluster. Distributed earns its cost when you truly exceed single-node capacity or need the availability.

Compression is a storage-cost lever. A columnar layout shrinks wide, repetitive data by a large multiple (measure it for your engines from ClickBench's "Data Size" column — it varies by engine and codec), which is a recurring cut in bytes stored and, in a warehouse, fewer dollars per query scanned.

Per-query billing cuts both ways. A serverless warehouse (BigQuery ran 1–3s per query here) fits spiky, ad-hoc analytics where an idle cluster would bleed money, and fits badly for a high-frequency dashboard hammering one aggregate, where a right-sized columnar box wins on total cost.

Every price and benchmark here will rotate — pricing shifts, hardware moves the single-node ceiling, vendor benchmarks get re-run. Don't anchor the decision to one dated figure. The framework doesn't rotate: two axes, eight signals. If a price moves, re-run the comparison.

Build it yourself

The framework is cheap until you feel it on your own data. Three projects, beginner to advanced. Each names a real tool (links verified reachable in 2026, but repos move — check before you lean on them).

Project 1 — Count your table three ways (~30 min). Feel the row-store aggregate cliff on real data. You'll need a Postgres or MySQL table with 10M+ rows and DuckDB. Run EXPLAIN ANALYZE SELECT count(*) FROM your_table; and note the time and the Seq Scan; then SELECT reltuples::bigint FROM pg_class WHERE oid = 'schema.your_table'::regclass;; then get a slice into columnar form — export CSV (COPY your_table TO 'slice.csv' (FORMAT csv)) and load it into DuckDB, or read the table directly with pg_duckdb — and count(*) there. Done when you can state your own numbers for all three and point to the Seq Scan. Stretch: add GROUP BY day.

Project 2 — Reproduce the spill cliff (~1–2 hrs). Watch a query cross the RAM→disk boundary. You'll need Postgres and a 1M+ row table with an integer and a text column (the NYC-taxi or ClickBench sets work). At default work_mem, run EXPLAIN ANALYZE SELECT count(DISTINCT int_col) FROM t; then the same on text_col, and confirm Sort Method: external merge Disk on the slow one. SET work_mem = '1GB';, re-run both. Done when you've captured your own before/after ratio and seen the plan flip to in-memory HashAggregate. Stretch: chart latency as you step work_mem from 4MB to 1GB and find your own cliff edge.

Project 3 — Run a slice of the 1.1B benchmark (~half a day). Reproduce the crossover on your hardware. You'll need the same big table in a row-store and a columnar engine, plus the ClickBench loaders (60+ DBMS setup scripts) and Litwintschik's posts for reference. Load an identical scan-aggregate table into each, time one representative query on both, and record the ratio and your hardware. Done when you reproduce a >100× gap — or can explain from the plans why yours differs. Stretch: add a distributed engine and build a query that forces a shuffle to find your own fan-out ceiling.

What to do Monday morning

The whole thing, compressed. When a query dies at scale, stop asking "which index?" and ask the two questions that decide it. Engine class: is the dominant shape point-lookups (row-store), scans and aggregates (columnar), or genuinely bigger than one node (distributed)? Memory budget: does the working set fit in RAM, or is it spilling? Then walk the eight signals; two or more means the move is on an axis, not another index. There's no single answer to "how do I handle a billion rows" — the answer is a decision you can write down and defend.

So go run it. Take your biggest table this week and do Project 1: count(*), then reltuples, then a columnar count in DuckDB. Watch where the plan shows a Seq Scan and how far apart the three numbers land. Want the full crossover? The ClickBench repo has loaders for 60+ engines, so you can time the same scan on a row-store and a columnar engine side by side. Bring back your own numbers — the framework only lands once the cliff is yours.

# Power BI Data Modelling: A Great Path to Great Analysis

Thu, 08 Oct 2026 03:59:58 +0000

Data Modeling in Power BI

Introduction

At a time when data is rapidly generated by different activities ranging from experiments to business, more often do we get data in different formats and structures. The data may or may not be related depending on the source. Related data in different formats and structure need to be re-organized to establish existing relationships for accurate analysis and reporting. To establish data model relationships and properties we need advanced tools that can connect to these sources. Then load the different datasets from source, and perform modelling to connect the links.

Data Modeling therefore, becomes very important. Data Modeling is the process of defining the data structure, properties, and relationships within a Data Model. Through organizing tables and columns, defining relationships between tables in the model, setting cardinality and cross-filter direction we achieve the best model which supports accurate calculations and reporting.

A Data Model is the logical representation of how data is structured and related within the data model.

Before modeling ensure that your data is clean and up to date, ready for modeling.

Why Data Modeling

Improved Performance and Scalability - join optimization, filters, and aggregations lead to a good model that is efficient and reliable. However, poorly designed models with complex relationships and redundant records cause delays, making queries slow.
Accuracy and Consistency - Guarantees the accuracy and consistency of the data used, making it reliable in reporting and drawing precise insights and improved decision making.
Supports Data Exploration - Users can use hierarchies and drill-down paths to support efficient data exploration.
Ease of Maintenance - Any changes in one model or report are reflected everywhere in real time enabling reusability and scalability, reducing the effort required to maintain reports and models as the business grows.

Components of Power BI Data Model

a. Tables and Sources

Tables are the raw building blocks of data models. They contain fields and records of particular data from different sources.
Sources, where we get our data from can vary depending on the context. We can get our data from SQL Database servers, Excel files, Data warehouse, SharePoint, and Power BI Dataflows.

Designing Source Tables

i. Fact Tables

Fact tables contain quantitative data, numerical data that can be aggregated and analyzed. Examples the Sales, Orders, or Invoices. This information is about specific events, business transaction, or measurements at the core of the data model.

Fact tables keeps one record for each transaction or event, includes only keys and metrics, and they avoid storing descriptive attributes.

Fact tables contain:

Foreign Key(FK) - a link to the Dimension table i.e. DateKey, ProductKey, e.t.c.
Measures(Numeric values) - Quantitative data you can aggregate. i.e. Quantity, UnitPrice, TotalAmount etc.
Degenerate dimensions - Descriptive data that does not warrant its dimension table. i.e. OrderNumber, InvoiceNumber.

ii. Dimension Tables

Dimension tables provide context or additional details to the data in Fact Tables. They contains fewer rows while fact tables can contain a large number of rows and they are on growing.

Relationship are used to establish links and connections between the fact and dimension tables in Power BI.

Dimension table contain:

Primary Key(PK) - Unique identifier e.g. ProductID
Descriptive attributes - Text, categories, hierarchies e.g. ProductName
Hierarchy Columns - Columns that form drill-down paths e.g. Country -> City -> Town.
Natural key - original identifier from the source system e.g. ProductCode

iii. Reference and Lookup Tables

They are small but critical tables in supporting the integrity and interpretation of data. They support additional mappings or categorizations. Example: DiscountTypes or StatusCodes.

Loading Data

Open Power BI
On the Ribbon Home tab, Click on Get Data from the Data group option.

From the drop down list, choose the source of your data. i.e. Text/CSV if your data is text or comma separated file.

Choose the path to your file i.e. C:/Users/User/Downloads/file.csv and click on Open.
Click on Connect on the Power Query editor window.
Your data is extracted to Power Query for cleaning. Click on Transform - Data to clean explore and clean it on the Power Query editor.

Finally, on the Power Query Editor Window Ribbon, Click on Close & Apply from the Home tab.

Changes are loaded and applied to Power BI. We will load all our Toy Store E-Commerce Database data set for modeling using the procedure above. The data is already clean, we are going to use it for demonstrating relationships in data modeling.

b. Relationships

Relationships are an important aspect of data modeling in Power BI. Relationships are for establishing connections and links between tables of a data model. They allow related tables to communicate with each other. They are the link between Fact Tables and Dimension Tables.

Relationships determine how data in the model is filtered, aggregated, combined, and visualized. They're critical to the accuracy and performance of queries in Power BI.

Cardinality

Cardinality type defines each relationship in a model. Cardinality refers to the number of unique values in one table related to the number of unique values in another.

Types of Cardinality in Relationships

One-to-One(1:1) - Every single record in one table is related to another single record in another table. Commonly used when splitting a large table for performance and security reasons.
One-to-Many(1:N) - Refers to when a single record in one table is related to many records in another table(s). Very common type of relationship in Power BI models. The Fact table is on the Many side while the Dimension table is on the One side when linking Fact tables and Dimension tables.
Many-to-Many(N:N) - Refers to when many records in the first table are related to many other records in the second table.

Cross Filter Directions

Used to determine the direction which filters propagate.

Single - When applied, only one of the tables in the relationship can filter data across the other table. Filtering data on the other table does not affect the data shown on the first table.

Bi-Directional - When applied, data can be filtered on either direction between the tables. Filtering data in one table affects data shown on the other table and vice versa. It is less often used and can impact performance negatively and create ambiguous filters.

Cross-filter options are largely dependent on the cardinality type.

Cardinality	Cross-Filter Options
1:1	Bi-Directional
1:*	Single or Bi-Directional
:	Single or Bi-Directional

Table 1: A table showing cardinality and their cross-filter options.

Creating a Relationship

To create relationships, first you need to:

Identify the matching columns(keys).
Choose the right cardinality.
Decide on the Filter direction.

The Model View Interface

To manage relationships in Power BI Desktop:

Click the Model icon in the left sidebar (the icon with three connected boxes)

You'll see all your tables displayed as boxes with their columns listed

Lines between tables represent relationships
The line style indicates cardinality: 1 on one end and * on the many end

Auto-Detect Relationships

When you load tables, Power BI automatically recognizes columns with same data type and name and creates relationships. However, you shoul always create relationships manually or review those created automatically.

Just like in our case, Power BI automatically creates relationships between tables in our model. We will review them.

When auto-detect works well:

Column names match exactly (e.g., ProductID in both tables)
Data types are compatible
Values actually correspond between the tables

Creating Relationships Manually

Using the Managing Relationship Dialog

On the Home tab → Manage Relationships

Click New

Select the first table and column, From Table
Select the second table and column, To Table
Configure cardinality and cross-filter direction

Click Save

Step by Step for Creating Relationships for our Toy Store E-Commerce Database

Load all tables:

 ```
     products
     orders
     order_items
     order_item_refunds
     website_sessions
     website_pageviews
 ````

Open Model view and delete any autodetected relationship.

To delete a relationship:

In Model view, right-click on the line joining the products and order_items tables
From the options displayed, Click on Delete

Save your work

Create the following relationships:

From Fact Table	To Dimension Table	Cardinality	Cross-Filter
products[product_id]	order_items[product_id]	One-to-Many	Single
orders[order_id]	order_items[order_id]	One-to-Many	Single
orders[order_id]	order_item_refunds[order_id]	One-to-Many	Single
website_sessions[website_session_id]	orders[website_session_id]	One-to-One	Bi-Direction
website_sessions[website_session_id]	website_pageviews[website_session_id]	One-to-Many	Single
order_items[order_item_id]	order_item_refunds[order_item_id]	One-to-one	Bi-Direction

Table 2: Table showing relationships to be created.

Creating a relationship between the products and order_items tables

On the Home tab → Manage Relationships
Click New
Select the products table and specify the column product_id
Select the order_items table and specify the column product_id
Set cardinality to One-to-Many and the cross-filter as Single

Click Save

Complete creating all the other relationships following the procedure. You will get something similar to:

Verify by checking that filters from any dimension table correctly filter the fact table.

Conclusion

Data modeling is very crucial to building powerful reports and visualizations. It is key to great analysis because it establishes relationships, links, and connections existing in a data model making it wholesome and complete. Further, it improves query performance increasing efficiency and reliability of queries. Correct data modeling can lead to accurate reporting and result reliability, thus informed decision making. To view the complete model and data for your practice on GitHub visit Toy Store E-Commerce.

¿Tu backup sobrevivirá a la próxima versión? La prueba que me faltaba

Wed, 07 Oct 2026 13:00:43 +0000

¿Tu backup sobrevivirá a la próxima versión? La prueba que me faltaba

Serie: Software auditable — Post 02.1
Read time: ~9 min · Etiquetas: python database testing opensource

Creí haber cerrado el problema

Hay una tranquilidad que no es ingeniería: es la sensación de haber terminado algo porque ya no duele. El Post 02 terminaba con una frase que me gustaba —recuperar es el trabajo— y con un test que restauraba una copia en una máquina limpia. Lo di por resuelto.

No lo estaba. Había confundido mi tranquilidad con una garantía.

Esto no va de arreglar un if. Va de un error de perspectiva: miré el sistema en el eje del espacio (¿funciona en otra máquina?) y me olvidé del eje del tiempo (¿funcionará cuando el software haya cambiado?). Un backup es, por definición, una promesa que se hace al futuro. Y yo la estaba validando solo en el presente.

El comentario que volvió a mirar donde yo no miraba

Mike Dabydeen, líder de la comunidad, leyó el post y escribió lo que yo no había querido ver. Su frase central: mi prueba demostraba que la copia de hoy se restaura con el código de hoy. Justo el caso que nadie necesita. Quien de verdad depende de un backup quiere restaurar una copia de hace un año con la versión que exista dentro de un año.

Cito sus dos ideas, porque las dos cambiaron el diseño:

"La prueba en máquina limpia demuestra que la copia de hoy se restaura con el código de hoy. Lo que todavía no cubre es que una copia hecha con la 1.3.0 se restaure con la versión que publiques dentro de un año, y ese es justo el caso de quien de verdad necesita su respaldo. Yo guardaría en el repo un paquete real generado por la 1.3.0 como accesorio y lo restauraría en CI en cada versión."

"Sobre el error del -wal, hay otra opción: restaurar con la misma API de copia de seguridad de SQLite, pero en sentido inverso... la copia mantiene abierta una transacción de escritura en el destino durante todo el proceso y la revierte si no termina, y el -wal que encuentra es el de esa misma base, así que no hay archivos huérfanos que borrar."

Tenía razón en las dos. Y la primera duele más, porque no es un bug: es una promesa sin testigo.

Acto I — El tiempo como testigo

La trampa del test que pasa

Mi test de recuperación era honesto y, aun así, insuficiente. Hacía esto:

# El test que probaba el presente
bundle = create_backup(source)        # se genera con el código actual
restore_backup(bundle, clean)         # y se restaura con el código actual
assert count_scans(clean) == 2

Generación y restauración usaban el mismo código. Un test cerrado sobre sí mismo: si mañana cambio el esquema y rompo la ruta de restauración, este test seguiría verde mientras las copias de los usuarios dejarían de restaurarse. La prueba de mi promesa era la promesa mirándose en un espejo.

Una promesa solo es real si sobrevive a quien la hizo.

La solución: una escalera de fixtures

Congelé un paquete real generado por la 1.3.0 y lo commité como golden file. En cada versión, CI debe restaurarlo en una máquina limpia:

tests/fixtures/backups/
├── aethernet-1.3.0-schema5.tar.gz          # el paquete real, ~5 KB
├── aethernet-1.3.0-schema5.expected.json   # su procedencia y las aserciones
└── README.md                               # cómo ampliar la escalera

El .expected.json guarda lo que hace verificable la prueba: sqlite_version, page_size, db_sha256 y los conteos que deben sobrevivir (escaneos, dispositivos, eventos). El test recorre todos los paquetes de la carpeta, no uno:

@pytest.mark.parametrize("bundle", sorted(FIXTURES.glob("*.tar.gz")), ids=lambda p: p.name)
def test_historical_backup_restores_on_clean_machine(bundle, tmp_path):
    expected = json.loads(_expected_path(bundle).read_text())
    result = restore_backup(bundle, _clean_paths(tmp_path))     # máquina limpia
    repo = Repository(Database(result.db_path))
    assert repo.counts()["wifi_scans"] == expected["counts"]["wifi_scans"]
    # ...informe regenerado, config restaurada, esquema migrado a v5

La regla de mantenimiento la dejó escrita el propio Mike: cuando cambie el esquema o el formato, añade un paquete nuevo y conserva los anteriores. Lo automaticé en un generador, scripts/gen_backup_fixture.py, que usa siempre el código actual y congela reloj y hostname para que el artefacto no filtre la máquina de quien lo creó.

Aquí está lo importante: MIGRATIONS es append-only; una copia del esquema v5 se migra sola hacia adelante. Así, la escalera deja de ser una promesa en un CHANGELOG y pasa a ser un test que falla el día que alguien rompa una migración.

Acto II — La elegancia de hacer menos

Del borrado defensivo a la primitiva correcta

En el Post 02 arreglé el -wal que vaciaba la base restaurada a base de borrar los ficheros auxiliares huérfanos:

_remove_sidecars(paths.db_path)     # fuera el -wal/-shm
os.replace(staged_db, paths.db_path)
_remove_sidecars(paths.db_path)     # y otra vez, por si acaso

Funciona, pero es una solución por fuerza bruta: limpio el desorden que otra cosa dejó. Mike propuso bajar un nivel y usar la misma primitiva de SQLite, en sentido inverso. En vez de renombrar un fichero y rezar por los sidecars, dejo que SQLite escriba la base preparada dentro de la base destino:

def _replace_db(staged_db: Path, target: Path) -> None:
    # Backup inverso: origen -> destino. SQLite abre una transacción de
    # escritura sobre el destino y la revierte si no termina.
    if _page_size(target) in (0, _page_size(staged_db)):
        try:
            _consistent_copy(staged_db, target)
            return
        except sqlite3.OperationalError as exc:
            if "locked" in str(exc).lower() or "busy" in str(exc).lower():
                raise BackupError("la base destino está en uso; detén el daemon antes de restaurar")
    # Fallback: reemplazo atómico con limpieza de sidecars
    _remove_sidecars(target)
    os.replace(staged_db, target)
    _remove_sidecars(target)

Las ventajas son las que él describió: no hay ventana en la que la base no exista, no quedan -wal/-shm huérfanos (el WAL que aparece es el de la propia base) y, si algo falla a mitad, el destino se revierte. Verifiqué el comportamiento con un experimento antes de escribir una línea: la copia inversa sobre una base WAL con conexión viva deja los datos correctos y no deja sidecar huérfano.

Hay elegancia en hacer menos y confiar en el motor que ya conoce sus invariantes.

La condición que no se puede ignorar

El backup inverso tiene un requisito real: si el destino está en modo WAL, ambos ficheros deben tener el mismo page_size. Si no coinciden, SQLite aborta con un OperationalError. Eso lo convierte en una restricción de diseño, no en una nota al pie:

Si los page_size coinciden → backup inverso (limpio, con rollback).
Si no coinciden → fallback atómico conservando la limpieza de sidecars.
Si el destino está bloqueado por un escritor vivo → se aborta con aviso, en lugar de competir por el fichero y corromperlo en silencio.

Ese último caso es el más importante para el usuario: restore --force sobre un daemon activo ya no es una ruleta. Si no puedo garantizar la transición, prefiero decir que no.

Lo que esto cambia (Takeaway)

Las dos observaciones de Mike son el mismo invariante visto desde dos ejes:

Temporal: bytes del pasado frente al código del futuro.
De concurrencia: el escritor del presente frente al restaurador.

Y hay una forma de leerlo que lo unifica: restaurar no es copiar ficheros, es una transición de estado. Cuando lo tratas como tal, el diseño se ordena solo — aparecen los guardarraíles, aparece el page_size, aparece el rechazo honesto — y el resto es escribir el test que lo demuestre.

Un test verde hoy es una promesa. Un test verde dentro de un año es una garantía.

Gracias

Otra vez, el mejor code review de Aethernet vino de la comunidad. Gracias, @Mike Dabydeen: no señalaste un bug, señalaste un punto ciego. Primero el tiempo, después la primitiva. Ambas correcciones están en la main por tu lectura.

Conclusión objetiva

La versión 1.3.0 de Aethernet incorpora ahora una garantía hacia adelante: un paquete real conservado como golden file que CI restaura en cada versión, un generador reproducible para ampliar la escalera y una restauración que usa el backup inverso de SQLite con fallback seguro. Todo con ruff + mypy --strict (0 errores) + vulture + 177 tests en verde (sin migración de esquema nueva: sigue en v5).

No es una promesa de marketing: es un test que puedes leer y un comando que puedes ejecutar.

# ampliar la escalera al cambiar el esquema o el formato
python scripts/gen_backup_fixture.py

# la garantía hacia adelante, corriendo en CI
pytest tests/test_backup_compat.py

# restaurar una copia vieja (aquí o en otra máquina)
aethernet restore aethernet-1.3.0-schema5.tar.gz

Y tú

¿Cómo verificas hoy que un backup de tu software seguirá restaurándose dentro de un año? ¿Confías en la compatibilidad hacia adelante de tus formatos o, como yo, creías tener un test que en realidad solo probaba el presente? ¿Y hasta dónde llevas la primitiva: ficheros, transacciones o capas más altas? Cuéntamelo en los comentarios; leo y respondo a todo.

_{English version below · El original está en español. If you're an English speaker, scroll down to the translation and tell me how you verify your backups in the discussion. I reply to everyone.}

Will your backup survive the next version? The test I was missing

Series: Auditable software — Post 02.1
Read time: ~9 min · Tags: python database testing opensource

I thought I had closed the problem

There's a kind of calm that isn't engineering: the feeling of having finished something because it no longer hurts. Post 02 ended with a line I liked —recovering is the job— and with a test that restored a copy on a clean machine. I called it done.

It wasn't. I had confused my calm with a guarantee.

This isn't about fixing an if. It's a failure of perspective: I looked at the system along the axis of space (does it work on another machine?) and forgot the axis of time (will it still work once the software has changed?). A backup is, by definition, a promise made to the future. And I was validating it only in the present.

The comment that looked where I didn't

Mike Dabydeen, a community leader, read the post and wrote what I hadn't wanted to see. His central point: my test proved that today's copy restores with today's code. Exactly the case nobody needs. Whoever truly depends on a backup wants to restore a copy from a year ago with the version that exists a year from now.

I quote his two ideas, because both changed the design:

"The clean-machine test proves that today's copy restores with today's code. What it still doesn't cover is that a copy made with 1.3.0 restores with the version you publish a year from now — and that is exactly the case of whoever truly needs their backup. I'd keep a real package generated by 1.3.0 in the repo as an artifact and restore it in CI on every version."

"About the -wal error, there's another option: restore using the same SQLite backup API, but in reverse... the copy keeps a write transaction open on the destination for the whole process and rolls it back if it doesn't finish, and the -wal it finds is that same database's, so there are no orphan files to delete."

He was right on both. And the first one stings more, because it isn't a bug: it's a promise with no witness.

Act I — Time as a witness

The trap of the test that passes

My recovery test was honest and still insufficient. It did this:

# The test that proved the present
bundle = create_backup(source)        # generated with today's code
restore_backup(bundle, clean)         # and restored with today's code
assert count_scans(clean) == 2

Generation and restore used the same code. A test closed in on itself: if tomorrow I change the schema and break the restore path, this test would stay green while users' copies stopped restoring. The proof of my promise was the promise looking at itself in a mirror.

A promise is only real if it outlives the one who made it.

The fix: a ladder of fixtures

I froze a real package generated by 1.3.0 and committed it as a golden file. On every version, CI must restore it on a clean machine:

tests/fixtures/backups/
├── aethernet-1.3.0-schema5.tar.gz          # the real package, ~5 KB
├── aethernet-1.3.0-schema5.expected.json   # its provenance and assertions
└── README.md                               # how to extend the ladder

The .expected.json stores what makes the test verifiable: sqlite_version, page_size, db_sha256 and the counts that must survive (scans, devices, events). The test walks every package in the folder, not one:

@pytest.mark.parametrize("bundle", sorted(FIXTURES.glob("*.tar.gz")), ids=lambda p: p.name)
def test_historical_backup_restores_on_clean_machine(bundle, tmp_path):
    expected = json.loads(_expected_path(bundle).read_text())
    result = restore_backup(bundle, _clean_paths(tmp_path))     # clean machine
    repo = Repository(Database(result.db_path))
    assert repo.counts()["wifi_scans"] == expected["counts"]["wifi_scans"]
    # ...report regenerated, config restored, schema migrated to v5

The maintenance rule was written by Mike himself: when the schema or the format changes, add a new package and keep the previous ones. I automated it in a generator, scripts/gen_backup_fixture.py, that always uses the current code and freezes clock and hostname so the artifact doesn't leak the machine that built it.

Here's the key part: MIGRATIONS is append-only, so a v5 copy migrates itself forward. The ladder stops being a promise in a CHANGELOG and becomes a test that fails the day someone breaks a migration.

Act II — The elegance of doing less

From defensive deletion to the right primitive

In Post 02 I fixed the -wal that wiped the restored database by deleting the orphan sidecar files:

_remove_sidecars(paths.db_path)     # drop the -wal/-shm
os.replace(staged_db, paths.db_path)
_remove_sidecars(paths.db_path)     # and again, just in case

It works, but it's brute force: I clean up the mess something else left. Mike suggested dropping one level and using SQLite's own primitive, in reverse. Instead of renaming a file and hoping about the sidecars, I let SQLite write the prepared database into the destination:

def _replace_db(staged_db: Path, target: Path) -> None:
    # Reverse backup: source -> destination. SQLite opens a write
    # transaction on the destination and rolls it back if it doesn't finish.
    if _page_size(target) in (0, _page_size(staged_db)):
        try:
            _consistent_copy(staged_db, target)
            return
        except sqlite3.OperationalError as exc:
            if "locked" in str(exc).lower() or "busy" in str(exc).lower():
                raise BackupError("destination database is in use; stop the daemon before restoring")
    # Fallback: atomic replace with sidecar cleanup
    _remove_sidecars(target)
    os.replace(staged_db, target)
    _remove_sidecars(target)

The advantages are the ones he described: there's no window where the database doesn't exist, no orphan -wal/-shm remain (the WAL that shows up belongs to the database itself), and if something fails halfway, the destination rolls back. I verified the behavior with an experiment before writing a single line: the reverse copy onto a WAL database with a live connection leaves the data correct and the orphan sidecar gone.

There's elegance in doing less and trusting the engine that already knows its own invariants.

The condition you can't ignore

The reverse backup has a real requirement: if the destination is in WAL mode, both files must share the same page_size. If they don't, SQLite aborts with an OperationalError. That turns it into a design constraint, not a footnote:

If the page_size matches → reverse backup (clean, with rollback).
If it doesn't match → atomic fallback keeping the sidecar cleanup.
If the destination is locked by a live writer → abort with a warning, instead of racing for the file and corrupting it silently.

That last case matters most to the user: restore --force over an active daemon is no longer a coin flip. If I can't guarantee the transition, I'd rather say no.

What this changes (Takeaway)

Mike's two observations are the same invariant seen from two axes:

Temporal: bytes from the past against the code of the future.
Concurrency: the writer of the present against the restorer.

And there's a way to read it that unifies both: restoring isn't copying files, it's a state transition. When you treat it as such, the design sorts itself out — the guardrails appear, the page_size appears, the honest refusal appears — and all that's left is to write the test that proves it.

A green test today is a promise. A green test a year from now is a guarantee.

Thanks

Once again, the best code review of Aethernet came from the community. Thank you, @Mike Dabydeen: you didn't point at a bug, you pointed at a blind spot. First time, then the primitive. Both fixes are on main because of your read.

Objective conclusion

Version 1.3.0 of Aethernet now ships a forward guarantee: a real package kept as a golden file that CI restores on every version, a reproducible generator to extend the ladder, and a restore that uses SQLite's reverse backup with a safe fallback. All of it with ruff + mypy --strict (0 errors) + vulture + 177 tests green (no new schema migration: still v5).

It isn't a marketing promise: it's a test you can read and a command you can run.

# extend the ladder when the schema or format changes
python scripts/gen_backup_fixture.py

# the forward guarantee, running in CI
pytest tests/test_backup_compat.py

# restore an old copy (here or on another machine)
aethernet restore aethernet-1.3.0-schema5.tar.gz

Over to you

How do you verify today that a backup of your software will still restore a year from now? Do you trust your formats' forward compatibility, or — like me — did you believe you had a test that was really only proving the present? And how far down do you take the primitive: files, transactions, or higher layers? Tell me in the comments; I read and reply to everyone.

_{Project license: MIT · No secrets or keys in this post. · Projects: github.com/yosoyignicion · Aethernet code: github.com/yosoyignicion/Aethernet · Release v1.3.0}

Tu backup no sirve si no puedes restaurarlo: el -wal que vació mi base

Wed, 07 Oct 2026 11:39:23 +0000

Tu backup no sirve si no puedes restaurarlo: el `-wal` que vació mi base

Serie: Software auditable — Post 02
Read time: ~10-12 · Etiquetas: python database testing opensource

La confesión: un botón que decía "RESPALDO" y no garantizaba nada

Tengo una regla que me ha costado aprender: desconfía de tus propias features hasta que puedas demostrarlas. Durante un tiempo, Aethernet tuvo un botón grande, verde y tranquilizador que decía RESPALDO .TAR.GZ. Lo pulsabas, salía un archivo, y te ibas a dormir tranquilo. No garantizaba absolutamente nada.

Ese botón es el punto de partida de este post. No porque estuviera roto —creaba un archivo, sí—, sino porque parecía una promesa y era una hipótesis sin comprobar: nunca nadie había restaurado esa copia en otro equipo. Yo no había hecho la única pregunta que importa: ¿funciona cuando la necesitas?

El feedback que lo puso por escrito

Un amigo de la comunidad me devolvió, con ayuda de su modelo agéntico como copiloto, un análisis que resumía el problema mejor que yo. Su idea central cabe en una frase: exportar no es recuperar. Y me propuso un criterio de aceptación concreto: una prueba de recuperación sin conexión, en una máquina limpia, sin acceso a la instalación original, en la que el usuario pueda examinar el historial y generar un informe. "La capacidad de exportación y la capacidad de recuperación son criterios de aceptación distintos."

Tenía razón, y además señalaba el hueco exacto de mi proyecto: no existía un procedimiento de recuperación documentado que cubriera base de datos, configuración y versión de esquema, ni claridad sobre qué claves había que guardar aparte.

Ese comentario convirtió una intuición en una lista de tareas verificables. Lo demás de este post es cómo lo cerré.

El problema, en el código

Mi respaldo era literalmente esto:

# El botón honesto y equivocado
target = paths.data_dir / f"aethernet-backup-{ts}.tar.gz"
with tarfile.open(target, "w:gz") as tar:
    tar.add(paths.db_path, arcname="aethernet.db")   # solo la base

Tiene cuatro problemas, y los cuatro importan:

Solo copia la base. La configuración (my_ssids, alias, retención…) se queda fuera.
No registra la versión del esquema. Al restaurar, no sabrías si la copia es más vieja o más nueva que la app.
Copia un fichero vivo. La base corre en modo WAL; copiar el .db a pelo mientras hay escrituras puede darte un estado incompleto.
No hay restore. Un backup sin restauración es un export con buenos modales.

	Exportar	Recuperar
Pregunta	¿se leen bien los datos?	¿arranco como si nada hubiera pasado?
Máquina	esta, con la app instalada	limpia, sin nada de la original
Prueba	abrir el informe	restaurar y volver a medir

La solución, en cuatro piezas

1) Un paquete que se puede inspeccionar

Un único .tar.gz con tres piezas y un manifiesto legible:

{
  "app": "aethernet",
  "format_version": 1,
  "app_version": "1.3.0",
  "schema_version": 5,
  "created_at": 1759800000.0,
  "db_sha256": "9f2c…",
  "members": ["aethernet.db", "config.toml", "manifest.json"],
  "secrets_required": []
}

Ese secrets_required: [] no es decoración: es una declaración de diseño. No hay claves irrecuperables que guardar aparte. El token de la API se regenera y nunca viaja en el paquete.

2) Copia consistente (y por qué no un `cp`)

Con WAL activo, copiar el fichero de la base es una apuesta. La forma correcta es dejar que SQLite haga la copia:

src = sqlite3.connect(str(db_path))
dst = sqlite3.connect(str(staged_db))
src.backup(dst)          # instantánea consistente, aunque haya un escritor vivo

Connection.backup() fusiona el WAL en un único fichero coherente. Es stdlib: cero dependencias nuevas.

3) Restaurar sin romper nada

El orden importa, y cada paso existe por un fallo real:

Validar antes de escribir — SHA-256 del manifiesto, miembros obligatorios y rutas seguras (nada de ../).
Rechazar el futuro — si la copia usa un esquema más nuevo que el binario, se aborta con un mensaje claro.
Resguardar lo actual — la base presente se copia a aethernet.db.bak-<fecha> antes de tocarla.
Migrar hacia adelante — las migraciones son append-only, así que una copia más antigua se actualiza sola al restaurar.

4) El criterio convertido en test

Aquí está el corazón del asunto. El criterio de aceptación del comentario dejó de ser una opinión y pasó a ser código que puede fallar:

def test_offline_recovery_on_clean_machine(tmp_path):
    # 1. origen: historial + ajustes
    source = _paths(tmp_path, "origin")
    ...  # se guardan escaneos y config
    bundle = create_backup(source, tmp_path / "backup.tar.gz")

    # 2. máquina limpia: otra instalación, sin nada de la original
    clean = _paths(tmp_path, "clean")
    restore_backup(bundle.path, clean)

    # 3. ¿se lee el historial y se genera un informe desde cero?
    repo = Repository(Database(clean.db_path))
    assert repo.wifi_scan_count() == 2
    report = build_report_data(repo, load_settings(clean))
    export(report, "md", clean.report_dir / "recovery.md")

Si este test se pone en rojo, mi backup no existe. No hay interpretación: o recupera, o no recupera.

El bug: el `-wal` que vació mi base restaurada

La primera versión del restore pasaba los tests unitarios… y fallaba el de recuperación real. Restauraba una copia con datos y la base aparecía vacía.

La causa raíz es una lección de SQLite que me llevé grabada: al reemplazar aethernet.db con un os.replace, los ficheros auxiliares aethernet.db-wal y aethernet.db-shm se quedan ahí. Al reabrir la base restaurada, SQLite encuentra un WAL que pertenece a otra base, lo reaplica y sobrescribe tus datos con el estado anterior.

El arreglo va en dos capas, porque el fallo tenía dos puertas:

_remove_sidecars(paths.db_path)          # fuera el -wal/-shm huérfano
os.replace(staged_db, paths.db_path)
_remove_sidecars(paths.db_path)          # y otra vez, por si acaso

…y en el CLI, cerrar la conexión del contexto antes de restaurar, porque no se restaura sobre una base abierta. Añadí un test que restaura sobre una conexión WAL viva para que no vuelva a pasar:

live = Database(paths.db_path); live.migrate()
restore_backup(bundle.path, paths)       # antes: 0 escaneos. ahora: 1.
assert Repository(Database(paths.db_path)).wifi_scan_count() == 1

El otro bug que no ves hasta que cambias de disco

Con el test en verde, probé en un disco distinto y apareció OSError: Invalid cross-device link. El staging vivía en /tmp (a menudo tmpfs) y el os.replace final apuntaba a ~/.local/share, en otro sistema de ficheros: un renombrado no puede cruzar discos.

La corrección es de una línea pero de fondo: el staging vive junto al destino, para que el reemplazo sea un renombrado atómico en el mismo sistema de ficheros. Para la configuración, un shutil.move (que sí sabe cruzar discos). Lo verifiqué con la base en tmpfs y el temporal en disco: recuperación correcta.

Lo que esto cambia (Takeaway)

Un criterio de aceptación no es una opinión: es un test que puede fallar. "Tengo backup" no significa nada hasta que has restaurado en una máquina limpia y has comprobado que el histórico y los informes siguen ahí. Y cuando el criterio se escribe como test, el diseño se ordena solo: aparece el manifiesto, aparece la copia consistente, aparecen los guardarraíles, y aparece el resto de la lista.

Exportar es fácil. Recuperar es el trabajo.

Gracias

Este post no existiría sin el análisis de un amigo de la comunidad, que con IA asistida convirtió una intuición mía en una lista de requisitos verificables. Su pregunta —"¿puedes restaurar en una máquina limpia?"— fue el mejor code review que ha recibido Aethernet. Gracias de verdad.

Conclusión objetiva

La versión 1.3.0 de Aethernet incorpora una recuperación offline demostrable: paquete con base consistente, configuración y manifiesto versionado; restore que valida, resguarda y migra; aethernet restore --verify para comprobar una copia sin escribir; y un test que reproduce la prueba en una instalación limpia. Todo con ruff + mypy --strict (0 errores) + vulture + 173 tests en verde y CI sobre Python 3.11 y 3.12 (sin migración de esquema nueva: el esquema sigue en v5).

No es una promesa de marketing: es un comando que puedes ejecutar.

aethernet backup                    # crea el paquete .tar.gz
aethernet restore --verify copia.tar.gz
aethernet restore copia.tar.gz      # recupérala en esta o en otra máquina

Y tú

¿Cómo verificas hoy que tus copias de seguridad sirven? ¿Alguna vez has restaurado un "backup" que no restauraba nada —y cómo lo descubriste? ¿Qué le exigirías a una herramienta para fiarte de su recuperación? Cuéntamelo en los comentarios; leo y respondo a todo.

_{English version below · El original está en español. If you're an English speaker, scroll down to the translation and tell me how you verify your backups in the discussion. I reply to everyone.}

Your backup is worthless if you can't restore it: the `-wal` that wiped my database

Series: Auditable software — Post 02
Read time: ~10-12 · Tags: python database testing opensource

The confession: a button that said "BACKUP" and guaranteed nothing

There's a rule I learned the hard way: distrust your own features until you can prove them. For a while, Aethernet had a big, green, reassuring button that said BACKUP .TAR.GZ. You pressed it, a file came out, and you went to sleep feeling safe. It guaranteed absolutely nothing.

That button is where this post begins. Not because it was broken —it did create a file— but because it looked like a promise and was really an untested hypothesis: no one had ever restored that copy on another machine. I hadn't asked the only question that matters: does it work when you need it?

The feedback that put it in writing

A friend from the community sent back an analysis —with an agentic model as co-pilot— that framed the problem better than I had. The core idea fits in one sentence: exporting is not recovering. And he proposed a concrete acceptance criterion: an offline recovery test on a clean machine, with no access to the original install, where the user can still inspect the history and generate a report.

His exact words:

"I'd add an offline recovery test: can a user restore a backup on a clean machine —with no access to the original install— and still inspect the history and generate a report? Export capability and recovery capability are distinct acceptance criteria."

He was right, and he also pointed at my project's exact gap: there was no documented recovery procedure covering the database, the configuration and the schema version, nor any clarity about which keys had to be handed over separately. That comment turned an intuition into a list of verifiable tasks. The rest of this post is how I closed it. Thank you, @lisarconnect, for the AI-assisted analysis.

The problem, in code

My backup was literally this:

# The honest, wrong button
target = paths.data_dir / f"aethernet-backup-{ts}.tar.gz"
with tarfile.open(target, "w:gz") as tar:
    tar.add(paths.db_path, arcname="aethernet.db")   # the database only

Four problems, and all four matter:

It only copies the database. The configuration (my_ssids, aliases, retention…) is left out.
It records no schema version. On restore, you wouldn't know if the copy is older or newer than the app.
It copies a live file. The database runs in WAL mode; copying the .db by hand while writes are happening can give you an incomplete state.
There's no restore. A backup without a restore is an export with good manners.

	Export	Recover
Question	do the data read back?	do I boot up as if nothing happened?
Machine	this one, app installed	clean, with nothing from the original
Test	open the report	restore and measure again

The solution, in four pieces

1) A package you can inspect

A single .tar.gz with three parts and a readable manifest:

{
  "app": "aethernet",
  "format_version": 1,
  "app_version": "1.3.0",
  "schema_version": 5,
  "created_at": 1759800000.0,
  "db_sha256": "9f2c…",
  "members": ["aethernet.db", "config.toml", "manifest.json"],
  "secrets_required": []
}

That secrets_required: [] isn't decoration: it's a design statement. There are no irreplaceable keys to hand over. The API token is regenerated and never travels in the package.

2) A consistent copy (and why not `cp`)

With WAL active, copying the database file is a gamble. The right way is to let SQLite make the copy:

src = sqlite3.connect(str(db_path))
dst = sqlite3.connect(str(staged_db))
src.backup(dst)          # consistent snapshot, even with a live writer

Connection.backup() folds the WAL into a single coherent file. It's stdlib: no new dependencies.

3) Restore without breaking anything

Order matters, and every step exists because of a real failure:

Validate before writing — manifest SHA-256, required members and safe paths (no ../).
Reject the future — if the copy uses a newer schema than the binary, it aborts with a clear message.
Protect the present — the current database is copied to aethernet.db.bak-<date> before touching it.
Migrate forward — migrations are append-only, so an older copy upgrades itself on restore.

4) The criterion turned into a test

This is the heart of it. The comment's acceptance criterion stopped being an opinion and became code that can fail:

def test_offline_recovery_on_clean_machine(tmp_path):
    # 1. origin: history + settings
    source = _paths(tmp_path, "origin")
    ...  # scans and config are stored
    bundle = create_backup(source, tmp_path / "backup.tar.gz")

    # 2. clean machine: another install, with nothing from the original
    clean = _paths(tmp_path, "clean")
    restore_backup(bundle.path, clean)

    # 3. does it read the history and generate a report from scratch?
    repo = Repository(Database(clean.db_path))
    assert repo.wifi_scan_count() == 2
    report = build_report_data(repo, load_settings(clean))
    export(report, "md", clean.report_dir / "recovery.md")

If this test goes red, my backup does not exist. No interpretation: either it recovers, or it doesn't.

The bug: the `-wal` that wiped my restored database

The first version of restore passed the unit tests… and failed the real recovery one. It restored a copy with data and the database came up empty.

The root cause is a SQLite lesson I won't forget: when you replace aethernet.db with os.replace, the sidecar files aethernet.db-wal and aethernet.db-shm stay behind. When SQLite reopens the restored database, it finds a WAL that belongs to another database, replays it, and overwrites your data with the previous state.

The fix comes in two layers, because the failure had two doors:

_remove_sidecars(paths.db_path)          # drop the orphan -wal/-shm
os.replace(staged_db, paths.db_path)
_remove_sidecars(paths.db_path)          # and again, just in case

…and in the CLI, closing the context connection before restoring, because you don't restore over an open database. I added a test that restores over a live WAL connection so it never happens again:

live = Database(paths.db_path); live.migrate()
restore_backup(bundle.path, paths)       # before: 0 scans. now: 1.
assert Repository(Database(paths.db_path)).wifi_scan_count() == 1

The other bug you only see when you change disks

With the test green, I tried another disk and hit OSError: Invalid cross-device link. The staging lived in /tmp (often tmpfs) and the final os.replace pointed at ~/.local/share, on a different filesystem: a rename can't cross disks.

The fix is a one-liner with real weight: staging lives next to the destination, so the replacement is an atomic rename on the same filesystem. For the config, a shutil.move (which does cross disks). I verified it with the database on tmpfs and the temp on disk: recovery works.

What this changes (Takeaway)

An acceptance criterion isn't an opinion: it's a test that can fail. "I have a backup" means nothing until you've restored it on a clean machine and checked that the history and the reports are still there. And once the criterion is written as a test, the design sorts itself out: the manifest appears, the consistent copy appears, the guardrails appear, and so does the rest of the list.

Exporting is easy. Recovering is the job.

Thanks

This post wouldn't exist without the analysis from a friend in the community, who —with AI assistance— turned an intuition of mine into a list of verifiable requirements. His question —"can you restore on a clean machine?"— was the best code review Aethernet has ever received. @lisarconnect, thank you, sincerely.

Objective conclusion

Version 1.3.0 of Aethernet ships a demonstrable offline recovery: a package with a consistent database, the configuration and a versioned manifest; a restore that validates, protects and migrates; aethernet restore --verify to check a copy without writing; and a test that reproduces the clean-machine trial. All of it with ruff + mypy --strict (0 errors) + vulture + 173 tests green and CI on Python 3.11 and 3.12 (no new schema migration: the schema is still v5).

It isn't a marketing promise: it's a command you can run.

aethernet backup                    # create the .tar.gz package
aethernet restore --verify copy.tar.gz
aethernet restore copy.tar.gz       # recover it here or on another machine

Over to you

How do you verify today that your backups actually work? Have you ever restored a "backup" that restored nothing —and how did you find out? What would you demand from a tool before trusting its recovery? Tell me in the comments; I read and reply to everyone.

_{Project license: MIT · No secrets or keys in this post. · Projects: github.com/yosoyignicion · Aethernet code: github.com/yosoyignicion/Aethernet · Release v1.3.0}

Building a High-Speed School Marketplace (Part 2): Multi-Role Security with Supabase RLS

Wed, 07 Oct 2026 04:15:37 +0000

This is Part 2 of our technical series on building a high-speed, lightweight school marketplace. In Part 1: \[Why We Ditched Frameworks for Vanilla JS & PWA,\]1 we discussed how we optimized the frontend for budget mobile devices. Today, we look at the backend architecture.

When planning the architecture for Aksara-Mart—our digital marketplace for vocational school canteens and cooperatives—we faced a fundamental infrastructure question:

Do we really need to build, maintain, and scale a traditional custom backend server (like Node.js/Express, Python, or Go) just to shuttle data between the browser and the database?
For a school environment, maintaining a traditional backend comes with real trade-offs:
1.Infrastructure & Hosting Overhead: A dedicated server needs 24/7 uptime monitoring, memory management, and scaling configuration for peak recess traffic.
2.Maintenance Burden: API routes, middleware controllers, and ORM abstractions require continuous patching and maintenance.
3.Unnecessary Latency: Every simple query (browsing the lunch menu, checking an order status) has to make an extra hop through an intermediate API server layer.
We wanted an architecture that could minimize backend server workload while keeping running costs close to zero.
That led us to Supabase. By allowing our frontend JavaScript to communicate directly with PostgreSQL via auto-generated APIs and WebSockets, we could offload the bulk of routine read queries directly to the client.
However, letting client-side JavaScript query a database directly sounds like a security nightmare—unless you have a bulletproof authorization layer built into the database itself.
That is where PostgreSQL Row Level Security (RLS) becomes the cornerstone of the entire system.

The Architecture: Direct Client Queries Without the Middleman
In a traditional web application, every user request passes through three layers:
```
[ Browser (Client) ] ──▶ [ Custom API Server (Express/Node) ] ──▶ [ Database ]
```
The API server spends most of its CPU cycles doing mundane tasks: authenticating tokens, verifying if a user is allowed to read a row, running the query, and formatting JSON.
With Supabase and PostgreSQL RLS, we simplified the architecture into a direct, high-performance two-tier model:
```
[ Browser (Vanilla JS Client) ] ──────────────▶ [ Supabase PostgreSQL + RLS ]
•Direct catalog SELECT                         - Kernel-level row authorization
•Direct order history lookup                   - Instant query evaluation (< 3ms)
•Real-time WebSocket subscriptions             - Atomic RPC stored procedures
```
The Advantages:
•Zero Backend Server Maintenance: No intermediate server processes to crash, restart, or run out of memory during a rush.
•Minimal Compute Overhead: PostgreSQL handles filtering and JSON serialization natively in C, which is vastly faster than parsing objects in Node.js.
•Direct Client Queries: Frontend JavaScript can fetch exactly the columns it needs, when it needs them, directly from the client.
The Core Problem: Securing Direct Database Access
If frontend JavaScript talks directly to the database using a public API key, what prevents a student from querying other users' private orders or inspecting canteen cost margins?
This is where Row Level Security (RLS) comes in.
Instead of writing authorization logic in backend middleware (if (req.user.id !== order.user_id) return res.status(403)), the authorization rules are compiled directly into PostgreSQL table definitions.
•Without RLS: A client query supabase.from('orders').select('*') returns every order in the database.
•With RLS: PostgreSQL automatically injects an internal filter into the query plan:
```
-- Automatically enforced by PostgreSQL on every query
WHERE email_customers = auth.jwt() ->> 'email'
```
Even if a user inspects the network tab and writes a custom script asking for all records, the database engine guarantees they only receive the rows they are explicitly permitted to see.
Managing Roles Across the School Ecosystem
A school marketplace serves several distinct groups with different permissions:
| Role | Access Scope | How Queries Are Handled |
| :--- | :--- | :--- |
| siswa (Students) | Browse active products, view own order history. | Direct client SELECT constrained by RLS. |
| guru (Teachers) | Same as students, with priority fulfillment flags. | Direct client SELECT constrained by RLS. |
| kasir (Cashiers) | View all active incoming orders, print receipts. | Direct client SELECT with staff privileges; no direct price/stock editing. |
| manajer (Managers) | Manage catalog items, update stock, view revenue. | Full CRUD permissions on product tables via RLS. |
| admin (System Admins) | System-wide configuration and role management. | Full administrative access. |
The admin_role Table
To keep role checks clean and reliable, we maintain a dedicated role table in PostgreSQL:
```
CREATE TABLE public.admin_role (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email TEXT UNIQUE NOT NULL,
role TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'active'
);
ALTER TABLE public.admin_role ENABLE ROW LEVEL SECURITY;
```
Practical Gotchas We Addressed:
4.Case-Insensitive Emails: Mobile keyboards often capitalize the first letter of an email (*Emails are not allowed*). Always use lower(trim(email)) in policies to prevent login lockouts.
5.Dual-Spelling Tolerance: In bilingual environments, staff may enter roles in English or Indonesian (manager vs manajer, cashier vs kasir). We check role aliases with IN ('manager', 'manajer', 'cashier', 'kasir') so minor spelling differences don't break authentication.
The Separation of Concerns: Direct Reads vs Server-Side Writes
Allowing frontend JavaScript to query the database directly works brilliantly for reads, but transactional writes require a different boundary.
A. Direct Client Reads (Safe via RLS)
Frontend JavaScript directly reads from the database for:
•The product catalog:
```
// Direct client query in skripmart.js
const { data: products } = await supabase
.from('product_list')
.select('id, name, price, stock, image_url')
.eq('status', 'active');
```
RLS ensures that wholesale costs (modal_price) and archived items are filtered out before data leaves PostgreSQL.
•Order tracking:
```
// Customers only receive their own orders
const { data: myOrders } = await supabase
.from('orders')
.select('*')
.order('created_at', { ascending: false });
```
B. Controlled Server-Side Writes (Via Database RPC)
While reading data directly from JavaScript is safe, creating orders and modifying stock should never happen via direct client INSERT or UPDATE:
```
// ❌ ANTI-PATTERN: Never allow direct client inserts for financial transactions
await supabase.from('orders').insert({
total_price: clientCalculatedTotal, // Client could tamper with this value!
items: cartItems
});
```
If we allowed direct client inserts on the orders table, a user could theoretically alter prices or bypass stock limits.
Instead, we follow a simple architectural rule:
Frontend JavaScript reads data directly using RLS.
Frontend JavaScript writes transactions via PostgreSQL Stored Procedures (RPC).
When a student checks out, the client invokes an RPC procedure (v2_create_order):
```
// ✅ SECURE: Transaction logic runs atomically inside PostgreSQL
const { data, error } = await supabase.rpc('v2_create_order', {
p_customer_name: userProfile.name,
p_delivery_type: deliveryOption,
p_items: cartPayload
});
```
Inside this procedure, the database recalculates prices from the official catalog, checks remaining stock, decrements inventory, and creates the order records in a single atomic transaction.
What We Gained from This Architecture
By offloading query processing directly to the client and database engine:
6.Zero Backend Maintenance: We do not maintain or pay for a custom Node.js/Express server layer.
7.Sub-3ms Authorization Overhead: RLS policy checks happen natively within PostgreSQL's query optimizer with virtually no latency penalty.
8.High Reliability During Traffic Spikes: When hundreds of students browse the menu simultaneously during recess, Supabase and Cloudflare serve the requests directly without choking an intermediate web server.
9.Complete Data Privacy: Every customer and staff member only receives data that belongs to their assigned role.

What's Next in Part 3?
With direct reads secured by RLS and client-side queries running smoothly, we still had to solve one critical concurrency problem:
During the 15-minute school break, dozens of students click "Checkout" at the exact same millisecond for the last few items in the canteen.
How do we prevent race conditions and overselling without a heavy application server?
In Part 3: Atomic Checkouts & Zero Overselling with PostgreSQL RPC, we will break down how we implemented pessimistic row locking (FOR UPDATE) inside PostgreSQL to handle high-concurrency checkout spikes in under 42 milliseconds!

Write C++ like it's SQL

Tue, 06 Oct 2026 12:59:20 +0000

A lot of C/C++ developers build complex data structures as if database theory never happened. And then they spend months maintaining pointer contracts that a schema could have made obvious.

Let’s take a graph structure as an example.

A common pattern looks like this:

struct Node;
struct Edge;

struct Node {
    uint32_t id;
    std::vector<Edge*> out_edges;
    void* payload;     // depends on node type
    uint32_t type;
};

struct Edge {
    Node* from;
    Node* to;
    void* payload;     // depends on edge type
    uint32_t type;
};

Looks flexible. In reality, it is a tiny undocumented database with manual referential integrity.Everyone must remember things like:

which Node::type maps to which payload struct
which Edge::type maps to which payload struct
whether payload == nullptr is allowed
who owns node payloads
who owns edge payloads
whether deleting a node invalidates edge payloads
how to serialize pointer graphs without losing meaning

And when requirements change, the whole thing turns into archaeology. Now let’s model the same problem like people who know schemas exist.

Start with identity:

using NodeId = uint32_t;
using EdgeId = uint32_t;

Then the core tables:

struct NodeRow {
    NodeId id;
};

struct EdgeRow {
    EdgeId id;
    NodeId from;
    NodeId to;
};

That alone already gives you something huge:

edges are explicitly relations between nodes
serialization is trivial
deletion rules can be formalized
validation is possible

Now add typed data associated with nodes:

struct TransformNode {
    NodeId node;
    float x, y, z;
};

struct LabelNode {
    NodeId node;
    std::string text;
};

struct ColorNode {
    NodeId node;
    uint8_t r, g, b, a;
};

And typed data associated with edges:

struct WeightEdge {
    EdgeId edge;
    float weight;
};

struct CapacityEdge {
    EdgeId edge;
    uint32_t max_flow;
};

struct StyleEdge {
    EdgeId edge;
    uint32_t line_style;
};

Now the model tells the truth.

A node may have a transform, or a label, or both, or neither. An edge may have a weight, a capacity, or some other attribute. No void*. No “if type == 3 cast to Foo”. No comment-based schema.

Need multiplicity rules?

Make them explicit:

one TransformNode row per node
zero or one WeightEdge row per edge
many LabelNode rows if your model allows multilingual labels
no EdgeRow may reference missing nodes

That is database thinking.

And it buys you real benefits even in plain C++:

strong typing instead of tribal knowledge
explicit foreign keys instead of raw pointers
easy serialization
easy validation
better data locality with arrays of records
queryable structure

Examples of useful queries:

find all edges whose to node is missing a transform
find all nodes with outgoing edges but no label
find all weighted edges crossing between two node groups
find dangling relations after a delete
find all data attached to a given node or edge

Try doing that cleanly in a pointer soup.

This is why I keep saying:

Many C/C++ codebases are just badly normalized databases with manual
memory management.

The irony is that game developers already stumbled into one good answer:

ECS. (or Entity-Component-System)

Entities are IDs. Components are typed tables. Systems are queries.

That idea is not limited to games. Graphs, CAD tools, compilers, pipelines, simulation systems, scene graphs, workflow engines — all of them can benefit from the same data-first thinking.

The real lesson is simple:

Before you design your structs like a C programmer, design your data
like a database engineer. Then make it fast like a systems programmer.

“Nice theory… but won’t this be slow?”

That’s the first reaction when you suggest modeling C++ data like a database. Fair question. Also mostly wrong.

Let’s take the graph model from the previous post and talk about how you actually make it fast.

First mistake people imagine:

“Tables + IDs = like SQL = slow joins”

No. You are not building a database engine. You are building arrays with structure.

Storage: SoA beats pointer soup
Instead of scattered heap objects:

Node* -> Edge* -> Node* -> payload*

You store dense arrays:

std::vector<NodeRow> nodes;
std::vector<EdgeRow> edges;
std::vector<TransformNode> transforms;
std::vector<WeightEdge> weights;

Now:

sequential memory
predictable access
prefetch-friendly
no pointer chasing

This alone often wins before you optimize anything else.

Aside of pure performance the inspection and debugging of such data organization is way simpler than chasing void* pointer soup.

IDs are cheaper than pointers (in practice)
A pointer graph:

random memory access
poor cache locality
hard to batch

An ID-based graph:

NodeId -> index lookup -> contiguous data

Yes, you pay an indirection. But you remove cache misses. On modern CPUs, that’s usually a net win.

“Joins” are just loops
Example:

find all edges with weights where from node has a transform

You don’t need SQL.

You do:

for (auto& w : weights) {
    auto& edge = edges[w.edge];
    if (has_transform[edge.from]) {
        // process
    }
}

That’s it.

no virtual dispatch
no pointer casting
no branching on “type”
no cache-miss lottery

Just tight loops over arrays. Eventually you may even come up with an idea of implementing a language integrated query (LINQ) framework.

Build indices where needed
If something is frequent, index it:

std::vector<std::vector<EdgeId>> edges_from_node;

or:

std::unordered_map<NodeId, TransformNode>

or even better:

sorted vectors + binary search
dense bitsets
sparse sets

You choose the structure per access pattern. Not everything needs to be O(1). Everything needs to be predictable.

Optional data without branching hell
Instead of:

if (node->payload && node->type == TRANSFORM)

You use:

presence in table
or a bitset mask

if (has_transform[node_id]) { ... }

Branch becomes predictable. Data becomes separate. Code becomes honest.

Memory management becomes trivial
Pointer model:

who owns what?
who deletes what?
what invalidates what?

Table model:

vectors own memory
IDs stay stable (or use handles/generations)
bulk free is trivial - serialization is dump-and-load, computing diffs is trivial (multiplayer game state synchronization anyone?)

You stop fighting lifetime bugs.

You can still go “full performance mode”
Nothing stops you from:

struct-of-arrays layouts
SIMD over columns
chunking (archetypes like ECS)
custom allocators / arenas
memory mapping
compression of columns

In fact, this model makes those optimizations easier, not harder.

The uncomfortable truth
Pointer-heavy designs feel “low level”. But they often produce:

worse cache behavior
harder reasoning
fragile invariants
slower real-world performance

Because modern performance is not about pointer flexibility. It’s about data layout and predictability.

The mental shift
Stop thinking:

“What does this object point to?”

Start thinking:

“Where does this fact live, and how do I iterate it efficiently?”

Database-style modeling is not about SQL.

It’s about:

explicit structure
normalized data
predictable access

And that’s exactly what high-performance C++ needs. Next time you reach for *void + enum** …ask yourself: Am I designing a data structure?
Or quietly building a database without admitting it?

Database-style modeling is powerful in C++. But it is not always the right tool. This matters, because every good idea becomes harmful when people turn it into a religion.
So when should you not model everything as tables + IDs?

When behavior matters more than data
Sometimes the central thing is not “what facts exist?”
Sometimes the central thing is:

“What operation should this object perform?”

Example:

struct Shape {
    virtual float area() const = 0;
    virtual void draw(Renderer&) const = 0;
};

If you have a small, stable set of objects with rich behavior, polymorphism may be cleaner than forcing everything into component tables.

A table model is great for querying data. An object model is often better for encapsulating behavior.

When the hierarchy is real and stable
If your domain naturally has a closed hierarchy:

Circle
Rectangle
Triangle

Then std::variant can be excellent:

using Shape = std::variant<Circle, Rectangle, Triangle>;

No void*. No fake database. No dynamic allocation required. Just a type-safe tagged union.
This is much better than pretending every tiny hierarchy needs an ID table.

When object identity is not important
Tables shine when identity and relationships matter. But not everything has identity.

A Vec3, Color, Matrix, Rect, Range, or BoundingBox is usually just a value. Do not give every value object an ID. That way lies madness.

Sometimes the right model is simply:

struct Color {
    uint8_t r, g, b, a;
};

Not every struct is secretly a table row.

When the data is tiny and local
If a structure is:

small
short-lived
private to one algorithm
not serialized
not shared across systems

Then simple direct structs are often best.
Example:

struct Candidate {
    float score;
    uint32_t index;
};

No schema needed. No registry. No component store.
Just code.

When traversal is the whole point
Some structures are naturally pointer-shaped.
Examples:

intrusive linked lists
trees with parent pointers
arena-allocated AST nodes
graph algorithms optimized around adjacency

A table representation can still work, but it may not be worth the extra ceremony.
If the dominant operation is “walk this exact shape,” direct references can be perfectly reasonable.
The mistake is not using pointers. The mistake is using pointers as an undocumented schema.

When lifetime is simple and ownership is obvious
If ownership is clear:

std::unique_ptr<Node> root;

and the whole structure dies together, then a pointer graph may be fine.
Especially with arenas.
A compiler AST is a good example:

build it
analyze it
maybe transform it
destroy it all at once

That can be simple and fast.

When the abstraction boundary is behavior-based
Sometimes the whole point is to hide the implementation.

Example:

struct StorageBackend {
    virtual void write(std::span<const std::byte>) = 0;
    virtual std::vector<std::byte> read(Key) = 0;
};

You do not want callers reaching into tables here.
You want an interface.
Data-first design is not an excuse to destroy encapsulation.

So what is the rule?
Use table/ID/database-style modeling when:

identity matters
relationships matter
serialization matters
validation matters
many systems touch the same data
the model evolves over time

Use object/value/polymorphic design when:

behavior matters more than relationships
data is local
hierarchy is small and stable
ownership is simple
encapsulation matters more than introspection

The real enemy is not OOP. The real enemy is informal contracts.
*void + enum + comment** is not a design pattern.
It is a schema that gave up.

Good C++ is not “always objects.” Good C++ is not “always tables.”
Good C++ is choosing the representation that makes illegal states hard, common operations fast, and future maintenance survivable.

The Entropy of a Growing Database

Tue, 06 Oct 2026 10:40:36 +0000

There is a strange moment in every software system.

At the beginning, the database feels almost too simple.

There are a few tables. A handful of records. A couple of indexes. The queries are obvious. The relationships make sense. You know where almost everything lives.

Then the application grows.

Users arrive. Features multiply. Logs accumulate. Transactions increase. New developers join the project. Old assumptions become new requirements. Temporary tables become permanent. Fields that were once optional become important. Queries written for one thousand records are suddenly running against ten million.

Nothing necessarily breaks.

That is what makes it interesting.

The database simply becomes different.

It develops history.

And with history comes something that engineers often experience long before they give it a name:

entropy.

In physics, entropy is associated with disorder and the number of possible states within a system. In software engineering, we can use the idea as a useful mental model.

A growing database does not necessarily become "disorganized" simply because it becomes large. Instead, it accumulates more possibilities, more relationships, more edge cases, more historical decisions, and more information about the past.

The database becomes a record not only of the application, but of the application's evolution.

Every column tells a story.

Every index represents a decision.

Every nullable field contains a possibility.

Every old record carries historical context.

Every duplicated value may be evidence of a decision someone once made.

The database is no longer just storage.

It becomes an archaeological site.

The First Database Is Usually Beautiful

Imagine creating a new application.

You might start with:

users
products
orders

The relationships are straightforward.

User
  |
  +---- Order
           |
           +---- Product

There is something satisfying about this stage.

The schema is small enough to fit inside your head.

You can open the database and understand it immediately.

A user has an ID.

An order belongs to a user.

An order contains products.

Everything feels intentional.

This is partly because the system has very little history.

You are designing the present.

There are no old migrations to consider.

No legacy records.

No abandoned features.

No compatibility requirements.

No strange values inserted by an earlier version of the application.

The database represents a relatively clean model of the current idea.

But software rarely stays in its first form.

The application evolves.

And the database evolves with it.

Every Feature Leaves a Trace

Suppose the application introduces subscriptions.

Now we add:

subscriptions
subscription_plans
payments

Later, someone adds discounts.

Then:

discounts
coupon_codes
order_discounts

Then refunds.

Then invoices.

Then payment attempts.

Then notifications.

Then audit records.

Then reporting tables.

None of these additions are inherently bad.

In fact, they may represent good engineering.

The important observation is that every feature expands the state space of the system.

The database now has more possible configurations.

An order can be:

created
paid
cancelled
refunded
partially_refunded

A subscription can be:

trial
active
paused
expired
cancelled

A payment can be:

pending
successful
failed
reversed

The system has become more expressive.

But it has also become more complex.

This is one way to think about database entropy.

Growth increases the number of relationships and states that the system must preserve correctly.

Rows Are History

A database row is not merely a piece of information.

It is a snapshot of a decision made at a particular point in time.

Consider a user record:

id: 42
name: Derek
email: *Emails are not allowed*
status: active

Six months later, the application changes its user model.

Maybe users can now have multiple email addresses.

Perhaps the system introduces organizations.

Maybe names can be changed.

Maybe accounts can be suspended.

The original row still exists.

The schema has changed, but the row carries the historical consequences of the old schema.

This is one reason migrations are more than technical operations.

A migration is often a negotiation between the present and the past.

You are asking an existing collection of data to live inside a new interpretation.

That is surprisingly profound.

The application changes quickly.

The data remembers slowly.

The Database Remembers Everything

Application code can be replaced.

A framework can be changed.

A frontend can be redesigned.

An API can be rewritten.

But the database often survives all of them.

This makes the database one of the longest-lived parts of many systems.

Imagine an application that has existed for ten years.

The current codebase may only reflect the latest two years of architectural decisions.

The database may reflect all ten.

It remembers the old customer model.

The old pricing model.

The old authentication system.

The old reporting system.

The old product structure.

The database becomes a kind of institutional memory.

Sometimes this memory is useful.

Sometimes it is inconvenient.

Often, it is both.

Null Is a Possibility

One of the most interesting forms of database entropy is the nullable field.

Consider:

phone_number VARCHAR(30) NULL

At first, this seems simple.

A user may or may not have a phone number.

But as the application grows, NULL can acquire different meanings.

Does it mean:

the user never provided a phone number?
the user declined to provide one?
the number has not been verified?
the information was imported from an older system?
the number was deleted?
the application does not know whether one exists?

A single database value can represent multiple conceptual states.

This is why mature systems sometimes require more explicit modeling.

Instead of:

phone_number = NULL

you may eventually have:

phone_number
phone_verified
phone_verified_at

Now the system can express more.

But expressing more also means managing more states.

The database becomes more precise.

And precision creates responsibility.

The Entropy of Meaning

Database complexity is not only about the number of tables.

It is also about meaning.

Imagine a column called:

status

What does it mean?

At first:

active
inactive

Later:

active
inactive
pending
suspended
archived

Then different parts of the application begin interpreting those values differently.

The frontend uses inactive to mean "not currently visible."

The API uses it to mean "account disabled."

The reporting system uses it to mean "customer has not purchased recently."

The same word has slowly accumulated multiple interpretations.

This is a different kind of entropy.

The structure may still look clean.

The meaning is what has become complicated.

Good database design therefore involves not only asking:

"What data do we store?"

but also:

"What exactly does this data mean?"

Indexes Are Memory With a Cost

As databases grow, indexes usually grow with them.

An index is an elegant idea.

Instead of searching every row, the database maintains additional structure that helps locate information efficiently.

For example:

CREATE INDEX idx_users_email
ON users(email);

Suddenly, finding a user by email becomes much easier.

But indexes are not free.

They consume storage.

They require maintenance.

Writes must update them.

Updates can become more expensive.

And eventually a large system may contain indexes created for queries that are no longer important.

This is another form of accumulated history.

An index may represent a query that mattered three years ago.

The query disappeared.

The index remained.

The database remembers a need that the application no longer has.

This does not mean indexes are bad.

It means that a mature database deserves periodic observation.

Sometimes optimization is not about adding another index.

Sometimes it is about asking whether an old one still belongs there.

Data Creates Gravity

There is a point in a system's life where data starts influencing architecture.

At the beginning, architecture determines the data model.

Later, the data model starts influencing architecture.

Suppose a table has:

500 rows

You can make significant changes easily.

Now imagine:

500 million rows

The same change has a completely different character.

A schema migration that takes seconds on a development machine may require careful planning in production.

A query that feels instantaneous on sample data may behave differently at scale.

A table that once felt small becomes an architectural component.

This is data gravity.

The more data accumulates, the more strongly it influences the decisions around it.

The database becomes difficult to move because the data has weight.

Not physical weight.

Architectural weight.

Time Is Another Dimension

A database does not merely contain objects.

It contains time.

Consider an order:

created_at
updated_at
paid_at
shipped_at
delivered_at

The record now describes a sequence.

The database becomes a timeline.

And once a system starts storing historical information, current state is no longer enough.

Suppose an account is currently:

active

That tells us what it is now.

It does not tell us whether it was suspended last year.

History requires additional structure.

This is where audit logs, event tables, history tables, and event-driven designs become valuable.

The system begins storing not only:

"What is true?"

but:

"What became true, and when?"

The second question creates much more information.

And much more information creates more complexity.

Entropy Is Not the Enemy

It would be tempting to conclude that database entropy is something engineers should eliminate.

But that would miss the point.

Some complexity is necessary.

A real system needs to represent real situations.

Customers change their information.

Orders change status.

Payments fail.

Products are discontinued.

Businesses change rules.

Users make mistakes.

Systems recover from failures.

History accumulates.

A database that perfectly represents a complex world will naturally contain complexity.

The goal is therefore not to create a database with zero entropy.

The goal is to keep complexity understood.

That distinction matters.

A complex schema can be healthy.

An apparently simple schema can be dangerous if its meanings are unclear.

The Importance of Constraints

One of the best ways to control database entropy is to make important rules explicit.

Consider:

email VARCHAR(255)

versus:

email VARCHAR(255) NOT NULL UNIQUE

The second schema communicates more.

It tells the database that every row must have an email and that duplicate emails are not allowed.

Constraints turn assumptions into rules.

Foreign keys do the same.

Check constraints do the same.

Unique indexes do the same.

A strong database does not rely entirely on application code to preserve its meaning.

It allows the database to protect important invariants.

In a growing system, this becomes increasingly valuable.

The more the system grows, the more places there are for assumptions to be violated.

Explicit rules reduce ambiguity.

Good Schemas Evolve

A common misconception is that good database design means creating a schema that never needs to change.

Real systems rarely work that way.

Good databases evolve.

The important question is whether the evolution is understandable.

A healthy schema might have:

clear naming;
explicit constraints;
documented relationships;
sensible indexes;
predictable migrations;
well-defined states;
historical information where necessary;
careful handling of deprecated fields.

The schema does not need to remain frozen.

It needs to remain legible.

Think about a city.

A city that grows does not remain identical to the village that existed before it.

Roads are added.

Buildings change.

Old structures are preserved.

New districts appear.

Infrastructure is upgraded.

The city becomes more complex.

But it can still be navigable.

A mature database should work similarly.

Cleaning Is Part of Engineering

Database maintenance is often associated with backups, indexes, and performance.

But cleanup deserves equal attention.

Unused columns can be investigated.

Deprecated records can be archived when appropriate.

Old indexes can be evaluated.

Duplicate data can be understood.

Historical tables can be separated from operational data.

Large datasets can be partitioned where useful.

The goal is not to delete information simply because it is old.

The goal is to understand what information the system has accumulated and why it exists.

Sometimes the correct answer is:

Keep it.

Sometimes:

Archive it.

Sometimes:

Transform it.

Sometimes:

Remove it safely.

Good cleanup is not about making the database look small.

It is about making its structure understandable.

The Database as a Map of the Product

One of the most interesting things about a mature database is that you can often reconstruct the history of the product by studying its schema.

You might find:

users
orders
subscriptions
legacy_orders
old_payments
payment_attempts
archived_products
audit_events

Even without reading the application code, these names tell a story.

The product started with users.

Then orders appeared.

Then subscriptions.

Payments became more sophisticated.

Older structures were replaced.

Historical information became important.

The database became a map of decisions.

This is why experienced developers sometimes develop a particular intuition when looking at schemas.

They are not only seeing tables.

They are seeing history.

Entropy and Developer Experience

There is another interesting consequence.

As database entropy increases, the importance of developer experience increases too.

A new developer joining a young project might understand the database in an afternoon.

A new developer joining a mature system may need weeks.

Not because they are less capable.

Because the system contains more history.

Documentation therefore becomes increasingly valuable as systems mature.

Schema diagrams help.

Migration histories help.

Naming conventions help.

Comments help.

Architecture documents help.

Good local development environments help.

The more complex the system becomes, the more valuable the map becomes.

The Quiet Intelligence of a Database

A database does something remarkable.

It turns the changing world into structure.

People make decisions.

Applications transform those decisions.

Databases preserve them.

Over time, millions of small events become rows, relationships, timestamps, indexes, constraints, and histories.

What began as:

users

can eventually become a representation of an entire ecosystem.

Customers.

Products.

Payments.

Messages.

Events.

Preferences.

Transactions.

Relationships.

History.

The database becomes a compressed model of reality.

Not reality itself.

But a carefully structured approximation of the parts of reality that the software needs to remember.

That is why database engineering is so interesting.

You are not merely storing information.

You are deciding what the system is allowed to remember.

Managing Entropy Instead of Fighting It

The most useful mindset is therefore not:

"How do I stop the database from becoming complicated?"

A better question is:

"How do I make increasing complexity remain understandable?"

That leads to practical habits.

Name things clearly.

Model important states explicitly.

Use constraints.

Review indexes.

Document unusual decisions.

Treat migrations carefully.

Understand historical data.

Separate operational and analytical workloads when appropriate.

Archive deliberately.

Monitor query behavior.

Measure before optimizing.

And most importantly, remember that every database is evolving.

There is no final schema.

There is only the current representation of the system.

Tomorrow may require another table.

Another relationship.

Another index.

Another migration.

Another idea.

That is normal.

The Database Has a Memory of Its Own

Eventually, a growing database becomes something more than storage.

It becomes memory.

It remembers what the system looked like.

It remembers what users did.

It remembers transactions.

It remembers changes.

It remembers relationships.

It remembers decisions that may no longer exist in the application code.

And this memory creates complexity.

But complexity is not necessarily failure.

Sometimes complexity is simply evidence that something has lived long enough to accumulate history.

A small database is like an empty notebook.

A mature database is like a library.

The challenge is not preventing the library from gaining books.

The challenge is making sure people can still find what they need.

That is the real engineering problem behind database entropy.

Not stopping growth.

Not fearing complexity.

Not endlessly simplifying everything.

But designing systems where complexity can grow without becoming invisible.

Because every growing database eventually tells a story.

The tables are its chapters.

The rows are its sentences.

The indexes are its shortcuts.

The constraints are its rules.

The migrations are its historical revisions.

And the schema is the architecture that holds the entire story together.

A database does not become interesting when it becomes large.

It becomes interesting when it becomes old enough to remember.

I Didn’t Want AI to Decide Fraud. I Wanted It to Investigate It.

Tue, 06 Oct 2026 09:50:36 +0000

How a painful backend, broken assumptions, graph debugging, evidence failures, and one very long final day became CaseVera — an evidence-first fraud investigation agent built on TigerGraph.

A fraud model can tell you that a transaction looks suspicious.

But would you block someone’s card because a model says it is 91.8% suspicious?

What if that customer has made hundreds of nearly identical purchases before?

What if the device is already familiar?

What if the region and product are completely normal?

What if a historical fraud case looks similar but has no direct connection to the current transaction?

And what if an AI generates a convincing explanation containing a piece of evidence that never actually existed?

That was the problem I kept running into while building my project for TigerGraph × Hacker House Goa 2026.

I initially thought the challenge was to build a better fraud detector.

It wasn’t.

The harder problem was building a system I would actually trust to investigate fraud.

That became CaseVera.

Investigate the case. Not just the score.

The problem with stopping at a fraud score

Most fraud pipelines eventually produce some variation of:

transaction → features → risk score → action

Risk scoring is extremely useful. But a score answers only one question:

How suspicious does this transaction look?

An investigator needs far more.

They need to know:

What actually happened?
What evidence supports fraud?
What evidence contradicts that conclusion?
What customers, cards, devices and transactions are connected?
Has something similar happened before?
Which policy applies?
Do we have enough evidence to act?
Should we ask for additional verification?
Who is allowed to approve the action?
Can every claim in the final decision be traced back to its source?

That is the gap CaseVera tries to address.

Instead of treating the risk model as the final answer, the risk score becomes the reason to begin an investigation.

What CaseVera does

CaseVera is a policy-constrained fraud investigation agent built around TigerGraph.

Its investigation loop is roughly:

Trigger → investigate the graph → retrieve context → assess evidence → handle uncertainty → request evidence if needed → apply policy → recommend the next best action → stop → persist case memory

The system combines:

TigerGraph + GSQL for structural relationships and graph evidence.

TigerGraph MCP as the graph-tool transport used by the runtime.

GraphRAG for retrieving relevant historical cases and policy/regulatory context.

A statistical fraud model as a suspicion prior.

A contradiction-aware evidence engine that separates:

fraud-supporting evidence,
legitimacy evidence,
ambiguous signals.

A deterministic policy layer for:

next-best actions,
approval routes,
SAR decisions,
stopping conditions.

And finally, a persistent case-memory layer that writes the completed investigation back into TigerGraph.

The decision that shaped the whole architecture

One decision may seem strange for an AI-agent project in 2026:

I deliberately kept an LLM out of the fraud decision loop.

I could have passed the transaction, graph context, previous cases and policies into Gemini or another general-purpose LLM and asked:

“Is this fraud? Explain why.”

It would probably return a very convincing answer.

That is exactly what worried me.

Fraud investigation is not a place where a convincing explanation is enough.

A generative model could accidentally:

mix evidence belonging to different entities,
interpret precedent as proof,
transform an assumption into an observation,
invent a relationship not present in the graph,
overstate certainty,
or recommend an action unsupported by policy.

In a casual chatbot, a hallucination may just be annoying.

In a financial investigation, it can contaminate the decision record.

So I split responsibility differently.

The model contributes suspicion.

TigerGraph contributes relationships and facts.

GraphRAG contributes relevant historical and policy context.

The evidence engine determines what the current facts actually support.

The policy engine determines which actions are allowed and who must approve them.

The LLM does not get to change the verdict, evidence, exposure, approval route or SAR decision.

That does not mean I think LLMs have no place in CaseVera.

Quite the opposite.

My next version would use them heavily at the boundaries:

turning free-form analyst questions into structured investigation requests,
letting analysts converse with a completed case,
generating concise summaries from already-verified evidence,
converting structured findings into more natural explanations,
improving analyst productivity.

But the underlying principle would remain:

Generation should improve the interface. It should never manufacture the evidence.

The case that validated the idea

One benchmark case became the best demonstration of why I built the architecture this way.

HHG-007.

The model suspicion was around 91.8%.

If the model score were allowed to dominate the decision, blocking the card would look completely reasonable.

CaseVera investigated further.

It found:

Fraud-supporting signals: 0

Legitimacy signals: 5

The amount was normal.

The region was normal.

The product behavior was normal.

And the customer had hundreds of previous charges at very similar amounts.

CaseVera did not declare the transaction legitimate either.

It stayed uncertain.

Its next-best action was:

Verify first.

Only if a simulated customer response later denied the transaction did a conditional block appear.

That case gave me one of the biggest lessons from this project:

Restraint is also intelligence.

A good fraud system should know how to act.

A good investigation system should also know when not to.

Why TigerGraph mattered

Some fraud patterns simply do not exist inside one transaction row.

Consider HHG-014.

If you inspect only the customer’s transaction, it does not tell the whole story.

Once TigerGraph follows the associated device fingerprint through the network, something much more interesting appears:

one device fingerprint → 24 customers → four connected cards

That structure becomes visible only after traversing relationships across accounts.

This became another principle behind CaseVera:

Fraud rarely lives in one row. It lives in relationships.

The graph contains customers, transactions, cards, device profiles, historical closed cases, investigation cases, evidence and document chunks.

GSQL queries retrieve things such as:

shared-device structures,
connected historical cases,
graph components,
customer context,
transaction context,
historical-case candidates.

GraphRAG then adds semantic retrieval over structurally relevant cases and policy documents.

Importantly:

Precedent is context. It is not proof.

A similar historical fraud case can help an investigator understand the situation.

It cannot prove the current customer committed fraud.

TigerGraph MCP — not just a checkbox

Near the end of the build, I realized something uncomfortable.

I had used TigerGraph MCP during development, but the investigation runtime was still calling the same installed GSQL queries directly through REST++.

The challenge explicitly required MCP.

I could have documented that MCP was used during development and moved on.

Instead, I rebuilt the graph transport boundary.

The important constraint was:

MCP integration was not allowed to change a single investigation decision.

So I introduced a separate transport adapter.

The investigator itself does not know whether the result came from REST++ or MCP.

Both transports return the same normalized graph results.

Then I tested them.

All 7 installed read queries returned identical normalized results over REST and MCP.

Then I ran the entire benchmark over both transports.

The final MCP run made:

96 graph calls

0 MCP failures

0 semantic differences between REST and MCP

0 semantic differences between MCP and the approved release

Different transport.

Same evidence.

Same decisions.

That was the point where MCP became a genuine part of the CaseVera runtime rather than a feature listed in a README.

The backend was the hardest part

The polished UI is probably the first thing people will notice.

The backend is where most of the project actually happened.

And it was painful.

More than once, I reached a stage where the system looked finished.

Then another assumption broke.

A relationship that seemed obvious from the data could not actually be proven.

Historical information had to obey strict point-in-time rules so an investigation could not learn from the future.

Customer-level card ownership could not automatically be treated as transaction-level card attribution.

Historical precedent had to remain context rather than current-case evidence.

Simulated customer replies had to remain hypothetical and could never silently become observed evidence.

Graph write-back had to be idempotent.

A successful write was not enough; the written state had to be independently read back and reconciled.

Even after the decision engine was finalized, integrating MCP briefly changed the release fingerprint because one Python file was re-saved with different Windows line endings.

The code behaved the same.

The bytes did not.

So the fingerprint failed.

I restored the original bytes before continuing.

At the time it was extremely frustrating.

In retrospect, it captured the entire philosophy of the project:

“Seems correct” was never enough. I wanted it proven.

The moment the system finally clicked

The project became much clearer once I stopped thinking of it as:

a fraud classifier

and started thinking of it as:

an investigation loop.

That changed everything.

The model was allowed to be wrong without destroying the system.

GraphRAG could provide useful context without becoming evidence.

Uncertainty became a legitimate outcome.

Human approval was designed into the workflow.

Evidence requests became part of the agent’s behavior.

And every investigation could end with a structured, auditable decision record.

The agent could now effectively say:

“I see something suspicious, but I do not yet have enough evidence.”

That is much more useful than forcing every case into fraud or legitimate.

Next-best action matters as much as the verdict

A fraud investigator does not only need:

“Fraud / not fraud.”

They need:

“What do we do now?”

CaseVera therefore produces both an initial and final next-best action.

Possible actions include:

allow,
monitor,
verify,
step-up authentication,
decline,
block a card,
escalate,
create a case,
file a report,
close with no fraud.

Actions also have approval routes:

auto

The agent can recommend restricted actions.

It cannot pretend a human-approved action has already happened.

SAR generation is also policy-controlled rather than triggered by a language model.

That separation became particularly important once I started thinking of CaseVera as something that could eventually operate in a real financial environment.

Case memory closes the loop

An investigation should not disappear once the decision is made.

CaseVera writes completed investigations back into TigerGraph as:

an InvestigationCase,
associated Evidence vertices,
relationships to transactions,
customers,
cards,
historical precedents,
documents,
devices.

For the benchmark, the 20 cases remain isolated from each other to maintain point-in-time fairness.

But the write side already establishes the foundation for a future system where newly completed investigations become usable institutional memory.

The final production write was independently reconciled:

20/20 investigation cases complete

273 evidence vertices

0 reconciliation divergences

Replay tests also verified that repeating writes did not create semantic growth.

The UI: show the investigation, not just the answer

I wanted the UI to feel like an analyst workstation rather than another fraud dashboard.

CaseVera includes:

case queue,
overview,
evidence graph,
evidence ledger,
policy decision view,
investigation replay,
policy corpus,
release audit.

The graph view shows connected entities.

The evidence ledger separates fraud, legitimacy and ambiguity.

The decision view explains which policy produced each recommendation.

Replay shows the sequence of graph retrieval, evidence gathering and decision formation.

The release audit exposes the boring but important stuff:

hashes, reconciliation, persistence and reproducibility.

The current browser UI is deliberately a verified read-only snapshot rather than pretending it is connected live to TigerGraph.

What was validated

The benchmark labels are hidden.

So I will not claim an accuracy number I cannot prove.

Instead, I validated what I could actually measure:

20/20 answer files produced and schema-validated

20/20 cases persisted and independently reconciled

273 Evidence vertices

7/7 graph queries identical over REST and MCP

96 MCP runtime calls, 0 failures

0 semantic differences between MCP and REST

0 semantic differences between MCP and the approved release

idempotent write replay verified

content-addressed scoring and persistence releases

To me, this matters more than inventing a benchmark score.

CaseVera is not perfect

There are several things I would build next.

First, real analyst and customer interaction instead of simulated responses.

Second, an LLM-powered interaction layer that lets analysts ask natural-language questions and receive explanations grounded only in verified case state.

Third, newly written InvestigationCase memory should become directly retrievable by later production investigations.

Fourth, stronger model calibration and additional fraud typologies.

Fifth, streaming triggers and richer analyst feedback loops.

And eventually, controlled dynamic tool selection — but only where the tool choices and evidence remain auditable.

The system I submitted is therefore not the end state.

It is the architecture I want to build that end state on.

Why I want to take CaseVera further

I do not think CaseVera should move forward because it has the flashiest AI model.

It does not.

I think its strength is that I treated the challenge as an investigation-engineering problem rather than an API demo.

I used TigerGraph where relationships genuinely mattered.

GraphRAG where retrieval genuinely mattered.

MCP as an actual runtime graph transport.

A statistical model where probabilistic suspicion was useful.

Deterministic controls where mistakes would be expensive.

And when the system contradicted one of my assumptions, I changed the assumption instead of forcing the output I expected.

There is still a lot I would like to improve.

But after spending most of this hackathon debugging edge cases, evidence provenance, graph relationships, temporal leakage, persistence and policy boundaries, the foundation now has one property I care about more than raw confidence:

traceability.

The biggest lesson I am taking away from this build is:

The most intelligent system is not always the one that answers fastest. Sometimes it is the one that knows when it does not know enough.

That is what I want CaseVera to become.

CaseVera — Investigate the case. Not just the score.

GitHub: https://github.com/BEAST04289/CaseVera

Built for TigerGraph × Hacker House Goa 2026.

One Rails App, Multiple Customers

Mon, 05 Oct 2026 08:00:48 +0000

Why Multi-Tenancy Matters

Imagine you are building a SaaS application.

You have 100 companies, but you don't want to deploy and maintain 100 separate applications.

That's where Multi-Tenancy comes in.

What is Multi-Tenancy?

Multi-tenancy allows a single application to serve multiple customers — called tenants — while keeping each tenant's data isolated.

For example:

                    Rails Application
                           │
          ┌────────────────┼────────────────┐
          ↓                ↓                ↓
       Tenant A         Tenant B         Tenant C
       Company A        Company B        Company C
          │                │                │
       Users/Data       Users/Data       Users/Data

Each company uses the same application, but should only be able to access its own data.

How can we implement it with Rails?

A common approach is adding a tenant_id to tenant-owned tables:

class User < ApplicationRecord
  belongs_to :tenant
end

Then queries are scoped to the current tenant:

current_tenant.users

For larger systems, another approach is database-per-tenant, where each customer gets a separate database.

The right strategy depends on your scale, security requirements, infrastructure, and operational complexity.

Why use Multi-Tenancy?

1️⃣ Lower Infrastructure Costs

One application can serve many customers instead of running separate deployments.

2️⃣ Easier Maintenance

Deploy features and bug fixes once instead of updating every customer environment.

3️⃣ Faster Scaling

You can scale application servers and background workers as the customer base grows.

4️⃣ Better SaaS Architecture

Multi-tenancy fits naturally with subscription-based products where many businesses share the same platform.

5️⃣ Centralized Development

One codebase means consistent features, security updates, and improvements across tenants.

⚠️ The important part

Multi-tenancy isn't simply adding a tenant_id.

You need to think about:

Data isolation
️ Authorization
️ Database architecture
⚡ Caching
Background jobs
Reporting
Tenant-aware integrations

A single missing tenant filter can become a serious data-isolation problem.

A Row Version Is a Cross-Layer Contract

Sat, 03 Oct 2026 06:24:18 +0000

A concurrency column can exist in the database and still be wrong.

The name may match. The migration may have run. Inserts may appear healthy. Yet optimistic concurrency can still be weaker than the application assumes because the runtime model, the physical schema, and the test environment disagree about what that column means.

The lesson is broader than one ORM feature: database-generated values are cross-layer contracts. Treating them as a property of only the entity class or only the table leaves a gap exactly where correctness matters.

What the contract actually contains

For a database-generated row version, at least five facts need to agree:

the physical type has true row-version semantics;
the value is non-nullable;
the database generates it on insert and update;
the ORM marks it as a concurrency token;
existing values remain intact during alignment or deployment.

A binary column of the expected length is not automatically equivalent. A nullable token changes the invariant. A model that knows about concurrency but not database generation can try to write a value the database owns. Conversely, a correctly generated database column provides little application protection if the runtime model never includes it in optimistic-concurrency checks.

The useful mental model is not “there is a column.” It is “every layer agrees who owns this value and how a stale write is detected.”

Prefer verification over destructive repair

Suppose the shipped database already has the correct non-null generated token, but the current EF Core model describes it incompletely. A tempting fix is to alter or recreate the column until the migration history looks tidy.

That can be the riskiest part of the change. Replacing a generated token can touch every row, invalidate values held by active writers, or disguise an unexpected schema. The safer move is often smaller:

align the runtime mapping with the physical contract;
add a forward-only migration that asserts the exact expected shape;
refuse to proceed when the real schema differs.

This kind of assertion migration changes no business data. It asks the database whether the table, column, type, length, nullability, and generation semantics match the known contract. Running it twice should be safe. Encountering a missing, nullable, or look-alike column should fail clearly.

Failing a rollout is inconvenient. Quietly weakening stale-write protection is worse. A fail-closed migration turns unknown drift into an explicit operational decision instead of an accidental data-integrity change.

Test at three different levels

No single test provider can prove the whole contract efficiently. A layered suite gives each test one honest job.

1. Model metadata

A cheap unit test can inspect EF Core metadata. It should verify that the property is required, marked as a concurrency token, and generated on add or update. This catches accidental mapping changes quickly, without starting a database.

Metadata tests do not prove the database behaves that way. They prove the application describes its expectation correctly.

2. Lightweight application flows

In-memory providers are useful for service and orchestration tests, but they do not automatically behave like a relational engine with generated row versions. Once a token becomes required, fixtures may fail because no value is produced.

A narrowly scoped, test-only save interceptor can generate a fresh fixed-length token for inserted or modified entities. That keeps fast tests representative of the required-value lifecycle. It should be described honestly as an emulator, not provider parity.

The emulator proves that application code can carry changing tokens through its flows. It cannot prove the SQL type, migration assertion, or actual stale-write mechanics.

3. A focused real-database proof

The decisive test uses the real provider. Create the shipped shape, insert a synthetic row, and retain its generated token. Apply the alignment migration—ideally twice—and verify that both the row and original token survive.

Then load the same row through two contexts. Save through the first and confirm its token changes. Attempt to save the stale copy through the second. The expected concurrency exception is the behavioural proof that the contract works end to end.

Negative cases matter too. A missing table, missing column, nullable token, or ordinary binary column should be refused. These cases prevent a permissive assertion from becoming ceremonial.

Put the expensive proof on the required path

A real-database test that developers can skip forever is documentation, not a gate. Register the small provider-faithful suite in the required CI job and include the relevant file scopes in change detection.

This does not mean every application test should use a full database. That would make feedback slower and fixtures harder to maintain. The trade-off is deliberate separation:

metadata tests are cheap and broad;
in-memory tests exercise application flows;
real-database tests prove database-owned behaviour.

The suite costs more than one happy-path unit test. In return, failures become easier to interpret because each layer states what it can and cannot guarantee.

A practical review checklist

When reviewing a database-generated concurrency token, ask:

Does the runtime model match the shipped type, nullability, and generation rules?
Does the migration preserve rows and existing token values?
Does unexpected schema fail closed?
Do fast fixtures represent the required token lifecycle without claiming provider equivalence?
Does a real-provider test prove token change and stale-write rejection?
Is that proof part of required CI?

The broader engineering habit is simple: when a value is owned by the database, verify the contract where each promise is made. Model tests describe intent. Fast tests protect application flow. Only the real provider can prove the final integrity boundary.

Demystifying ACID Transactions: How to Keep Your Database from Making a Mess

Thu, 01 Oct 2026 10:28:33 +0000

An ACID transaction is a set of four database safety rules—Atomicity, Consistency, Isolation, and Durability—that ensure database operations are executed reliably. It guarantees that even if your application crashes, loses power, or experiences thousands of concurrent requests, your data remains accurate and uncorrupted. Simply put, it ensures your database never ends up in a half-finished, broken state.

The Ticket Booth Analogy

Imagine buying a physical ticket at a busy movie theater box office. You hand the cashier a $15 bill, and they hand you the ticket. This exchange must happen as a single, indivisible unit. If you hand over your cash, but a sudden power outage occurs before the cashier can hand you the ticket, you would be furious if they kept your money. Conversely, the theater would lose out if you grabbed the ticket and walked away without paying.

The transaction is "all-or-nothing": either the cash changes hands and you get the ticket, or the entire deal is canceled, and you keep your money. ACID acts as this strict box-office coordinator for your software's database, ensuring that multi-step operations never stop halfway.

Why It Matters: Real-World Disasters

In daily software engineering, we rely on ACID transactions to prevent catastrophic data errors, especially when building systems like financial ledgers, e-commerce checkouts, or reservation platforms.

Without ACID, imagine a bank transfer system built on Node.js. When a user transfers $100, the system must perform two separate actions: deduct $100 from Account A, and add $100 to Account B. If the database server loses connection exactly halfway through this process, Account A loses money, but Account B never receives it. The $100 vanishes into the digital void.

Similarly, in a high-traffic retail store, if two customers try to buy the very last item in stock at the exact same millisecond, a non-ACID database might let both transactions complete. This leaves you with an oversold item and an angry customer. By grouping these operations inside an ACID transaction, developers ensure that if any step fails, the database automatically rolls back all changes, restoring everything to its original state.

Implementing ACID in Node.js and MySQL

Here is how we can implement a secure money transfer using Node.js and the standard mysql2/promise library. By wrapping our queries in a transaction block, we guarantee data safety.

const mysql = require('mysql2/promise');

async function transferFunds(fromAccountId, toAccountId, amount) {
  const connection = await mysql.createConnection({
    host: 'localhost',
    user: 'root',
    database: 'bank_db'
  });

  try {
    // Start the ACID transaction
    await connection.beginTransaction();

    // Step 1: Deduct from the sender's account
    const [deductResult] = await connection.execute(
      'UPDATE accounts SET balance = balance - ? WHERE id = ? AND balance >= ?',
      [amount, fromAccountId, amount]
    );

    if (deductResult.affectedRows === 0) {
      throw new Error('Insufficient funds or invalid sender account');
    }

    // Step 2: Add to the receiver's account
    const [creditResult] = await connection.execute(
      'UPDATE accounts SET balance = balance + ? WHERE id = ?',
      [amount, toAccountId]
    );

    if (creditResult.affectedRows === 0) {
      throw new Error('Invalid receiver account');
    }

    // If both steps succeed, permanently commit the changes to disk
    await connection.commit();
    console.log('Transaction completed successfully!');
  } catch (error) {
    // If anything fails, revert all changes immediately
    await connection.rollback();
    console.error('Transaction failed. Rolled back changes:', error.message);
  } finally {
    await connection.end();
  }
}

The Takeaway

ACID transactions are the ultimate insurance policy for your application's state. While they introduce a minor performance trade-off due to locking rows during updates, they completely eliminate the need to write complex, error-prone fallback code. Shifting the responsibility of data integrity to your database engine lets you sleep easy knowing your system will never tell a lie.

Originally published on my blog. You can read the alternative breakdown here.

Forensic Aviation Analytics: Auditing Fuel Bleeds & Delay Cascades on a Regional A320neo Network

Mon, 28 Sep 2026 11:41:49 +0000

When evaluating airline performance, typical analytics dashboards rely blindly on passenger-facing "On-Time" arrival percentages. However, from an operational and financial engineering standpoint, these high-level metrics can easily mask millions of dollars in silent cash destruction.

Due to strategic schedule padding, a short-haul flight can sit idling its engines on a congested tarmac for 20 minutes, yet still be registered as "On-Time" to the customer. The passenger is happy, but the airline’s finance department just watched pure margin vaporize into thin air.

To expose these hidden friction layers, I engineered LCC-Ops: an end-to-end data warehouse auditing over 120,000 regional transaction legs modeled around the single-aisle network performance of flynas across key GCC hubs.

Here is the technical architecture and the forensic database methodologies behind the pipeline.

The Tech Stack & Architecture

Database Layer: MySQL Server 8.0 (Advanced CTEs, Mathematical Window Functions, Schema Migrations)
Business Intelligence: Power BI Desktop (Data Type Optimization, Cross-Table Iterators, UI Design)
Core Fleet Constants: Engineered around the Airbus A320neo (CFM LEAP-1A26 engine) utilizing official ICAO Engine Emissions Databank metrics. At a standard 7% ground idle taxi thrust, this engine burns exactly 0.0940 kg of fuel per second, per engine

The Database Engineering Breakdown

Pillar 1: Isolating Tarmac Cash Destruction (Domain A)To find the true cost of airport ground bottlenecks, the ingestion engine isolates rows inside the transactions table (fact_flights) where the Out-Gate to Runway Takeoff latency (taxi_out) exceeded a standard 4-minute engine stabilization and warming buffer.

By pulling the real-world ICAO fuel coefficients down via relational keys, the calculation bypasses text-based padding and computes raw physical fuel destruction.

SELECT 
    f.origin AS departure_hub,
    COUNT(f.tail_number) AS total_monitored_flights,
    ROUND(AVG(f.taxi_out), 2) AS avg_taxi_out_minutes,
    -- Formula: (Actual Taxi Time - 4 Min Buffer) * 60 Sec * ICAO Idle Flow Rate * 2 Engines
    ROUND(SUM((f.taxi_out - 4) * 60 * e.fuel_flow_idle_kg_sec * 2), 2) AS total_wasted_fuel_kg
FROM fact_flights f
JOIN dim_aircraft a ON f.tail_number = a.tail_number
JOIN dim_engine e ON a.engine_identification = e.engine_identification
WHERE f.reporting_airline = 'XY' AND f.taxi_out > 4
GROUP f.origin;

Pillar 2: Tracking Cross-Border Delay Cascades (Domain D)
When an international flight arriving from an airport like Karachi (KHI) experiences customs or airspace delays, that latency risks cascading into the aircraft’s subsequent domestic legs (e.g., RUH ➔ JED).

To track this chronologically across individual daily airframe lifecycles, I utilized advanced LEAD() mathematical window functions

WITH ranked_daily_schedule AS (
    SELECT 
        flight_date, tail_number, origin, dest, carrier_delay AS leg_1_delay,
        LEAD(origin, 1) OVER (PARTITION BY tail_number, flight_date ORDER BY taxi_out DESC) AS leg_2_origin,
        LEAD(carrier_delay, 1) OVER (PARTITION BY tail_number, flight_date ORDER BY taxi_out DESC) AS leg_2_delay
    FROM fact_flights
    WHERE reporting_airline = 'XY'
)

Pillar 3: Menu Logistics & Hangar Risk Automation (Domains B & C)
Catering Spoilage Optimization: Short flight windows on regional tracks mean cabin crews lack the time to complete complex perishable sales loops. The data models validate flynas’s lean, shelf-stable buy-on-board menu model, proving that switching to fresh profiles would trigger an immediate 54% inventory spoilage cash bleed

Automated Hangar Reorder Points (ROP): The warehouse tracks rolling airframe hours against specific tail numbers (HZ-NS1 to HZ-NS5) [Query-relevant Context]. If a component approaches its high-wear safety limit, the system triggers automated database alerts to protect the airline against catastrophic $25,000/day Aircraft on Ground (AOG) hangar penalties

UI Engineering in Power BI

To present these data layers to corporate stakeholders, I built an interactive application-style interface directly inside Power BI using flynas’s signature branding tones.

The biggest challenge during BI ingestion was data type conversions—empty strings or trailing spaces in the raw logs caused math functions to drop blank values (--). By implementing upfront string transformations (Uppercase/Trim) and embedding data validation wrappers like ISNUMBER(VALUE()) directly into row-level DAX iterators, the front-end dynamically scales across massive transaction volumes without data loss.

The entire production code schema and scripts are live on my GitHub Link. I'd love to hear how other data engineers manage real-time fuel-burn parameters or multi-leg network routing in your pipelines!

Database Concurrency 101: Optimistic vs. Pessimistic Locking

Sun, 27 Sep 2026 09:32:31 +0000

In modern software development, we often have to handle multiple users interacting with our application at the exact same time. Optimistic and pessimistic locking are two distinct strategies used to manage concurrency, which is the technical term for when multiple users or systems try to update the exact same database record simultaneously. While optimistic locking assumes conflicts are rare and verifies changes only at the very end of an operation, pessimistic locking assumes conflicts are highly likely and preemptively blocks others from accessing the data until the current operation is finished.

A Relatable Everyday Analogy

To understand the difference, imagine a collaborative digital workspace where you and your colleagues edit text files.

Pessimistic locking is like an old-school file server where only one person can open a document for editing at a time. The moment you open the file, the server locks it. If your colleague tries to open it, they receive a "Read-Only" warning and are completely blocked from making edits until you save, close, and release your lock on the file.

Optimistic locking, on the other hand, is like a modern version control system like Git. Both you and your colleague can open, read, and edit the exact same document simultaneously. However, when you click "Save," the system checks if anyone else has saved a new version of the document since you opened it. If no one has, your save succeeds. If your colleague managed to save their changes first, the system blocks your save, alerts you to a "conflict," and asks you to review and merge the changes before trying again.

Why It Matters in the Tech Industry

In the tech industry, failing to implement a proper locking strategy leads to a highly destructive bug known as a "lost update" or a "race condition"—where the outcome of a process depends on the unpredictable sequence or timing of other events.

Imagine a financial application where two family members try to withdraw $50 from a shared $100 bank account at the exact same millisecond. Without locking, both transactions will read the balance as $100 simultaneously. Both transactions will approve the $50 withdrawal, and both will attempt to write the new balance of $50 back to the database. The bank is now down $100, but the database incorrectly shows a remaining balance of $50 because the second write silently overwrote the first one. By using database locking, software engineers prevent this double-spending, protect system integrity, and ensure transactions are processed safely.

Implementation in Node.js and MySQL

Here is a simple example of how to implement optimistic locking in a Node.js application using a MySQL database client. We do this by adding a version column to our database table and verifying it during our update query.

// A helper function to update a user's bank balance using Optimistic Locking
async function updateAccountBalance(connection, accountId, withdrawAmount, currentVersion) {
  // We attempt to update the balance and increment the version number,
  // but ONLY if the version in the database matches the version we originally read.
  const query = `
    UPDATE bank_accounts 
    SET balance = balance - ?, version = version + 1 
    WHERE id = ? AND version = ?
  `;

  const [result] = await connection.execute(query, [withdrawAmount, accountId, currentVersion]);

  // If no rows were affected, it means the version changed in the background
  // because another transaction updated it first. We must roll back and retry.
  if (result.affectedRows === 0) {
    throw new Error("Concurrency Conflict: The account was updated by another process. Please retry.");
  }

  return "Balance updated successfully!";
}

The Strategic Takeaway

Choosing between these two approaches is always a trade-off between speed and strict prevention. You should default to optimistic locking for applications with low-to-medium write conflicts because it keeps your application fast, highly available, and free of database bottlenecks. However, when dealing with highly critical operations where conflicts are frequent and data corruption is unacceptable—such as reservation systems or payment gateways—pessimistic locking is the superior choice, despite the performance hit of making users wait in a digital queue.

Originally published on my blog. You can read the alternative breakdown here.

Horizontally Scaling PostgreSQL: Database Sharding with Citus

Sat, 26 Sep 2026 10:35:35 +0000

A practical guide to scaling your database infrastructure horizontally. Learn how to shard your data, optimize query routing, and build a distributed multi-tenant architecture using Postgres, Citus and Docker.

Prerequisites

What do you need to follow along?

A working Laptop/PC?
Working knowledge of SQL or PostgreSQL.
To have at least eaten something today (preferably swallow related).

Database sharding is a strategy used to solve scalability issues in applications that have a massive amount of data. It involves splitting data into smaller chunks called shards. Each shard is stored on a separate database server instance, effectively spreading the data and the workload across multiple machines.

Why Shard Data?

As an application grows, the number of users and the amount of data it stores increase over time. At a certain limit, a single database server will become a bottleneck. The application slows down and affects the user experience.

Instead of upgrading a single server with more CPU or RAM, which will eventually hit a hardware ceiling, sharding lets you add more servers to your cluster. This allows applications to handle nearly infinite growth in data volume and user traffic.

Understanding Shards and Shard Keys

Before we start sharding data, let's first understand what shards and shard keys are.

Shards

The partitioned data chunks are called logical shards. The machine that stores the logical shard is called a physical shard or database node. A physical shard can contain multiple logical shards.

Shard Key

A shard key is used to determine how to partition the dataset. A shard key is a specific column in your dataset that determines which rows of data group together to form a shard.

Sharding Methods

There are a few ways to shard data depending on the application requirements.

Key-based sharding (hash sharding): Hash sharding uses a mathematical formula called a hash function. The hash function takes your shard key and generates a hash value, this hash value determines the exact physical shard where the row will be stored.
Range-based sharding: Range-based sharding, or dynamic sharding, splits database rows based on a range of values. Then the database designer assigns a shard key to the respective range.
Directory-based sharding: Directory sharding uses a lookup table to match database information to the corresponding physical shard. A lookup table is like a table on a spreadsheet that links a database column to a shard key.
Geo sharding: Geo sharding splits and stores database information according to geographical location.

For this guide, we are focusing exclusively on Hash Sharding. This is the industry standard for multi-tenant applications like the e-commerce platform we will be building in this tutorial.

Choosing Your Shard Key

If you pick the wrong shard key, your database will still crash under load. When choosing a shard key, you must ensure the data is distributed evenly across your nodes:

Cardinality: This refers to the number of unique values a key can have. You want a key with high cardinality like a User ID, not low cardinality like a Gender or State column, so the data can be split into many small, even chunks.
Frequency: You must avoid keys where one specific value appears too often. For example, if you shard by a City column and 80% of your users live in Ikeja, the Ikeja node will be overloaded while the others sit idle.
Monotonic change: This is the rate of change of the shard key. A monotonically increasing or decreasing shard key results in unbalanced shards. For example, consider a feedback database that is split into three different physical shards as follows:
- Shard A stores feedback from customers who have made 0 to 10 purchases.
- Shard B stores feedback from customers who have made 11 to 20 purchases.
- Shard C stores feedback from customers who have made 21 or more purchases.
  As the business grows, customers will make 21 or more purchases. The application stores their feedback in Shard C. This results in an unbalanced shard because Shard C contains more feedback records than the other shards.

Is Sharding Actually For You?

While sharding solves massive scale requirements, it introduces significant technical and operational tradeoffs. You should not shard your database unless a single, heavily provisioned Postgres instance can no longer handle your traffic.

If you do decide to shard your database, here are some of the biggest challenges you will face:

Data hotspots: Even with a good hash function, uneven traffic can create hotspots where one server becomes overloaded by a small group of highly active users.
Performance regression: Queries that do not use the shard key require the system to scatter and gather across every server. Cross-shard joins are computationally expensive because the application must pull and merge data from multiple physical locations over the network.
Operational complexity: Managing a sharded environment increases the difficulty of routine tasks. Schema updates, backups, and point-in-time recovery must be coordinated across multiple independent instances.

How Can We Shard Our Postgres Database: Introducing Citus

Citus is an open-source extension that transforms standard PostgreSQL into a distributed database. Instead of being limited to a single server, Citus allows your database to scale horizontally across multiple machines.

How Citus Works

The Coordinator Node: When you ask your database a question, you talk to the coordinator. The coordinator directs your request to the right servers, gathers the answers, and combines them for your application.
Worker Nodes: Citus breaks down large tables into smaller, manageable chunks (shards) and spreads them across these worker servers. They do the heavy lifting of storing the data and running the queries.

Start Early, Scale Later

Migrating a traditional, monolithic PostgreSQL database to a distributed, sharded database later in its life is incredibly difficult. You have to change your schema, rewrite complex queries, and alter how your application connects to the database.

To avoid this nightmare, you can start using Citus on a single node from day one. You design your tables with distribution keys immediately, and your application code is written to query a distributed database from the start.

The Transition: When your data outgrows that single machine, you won't have to rewrite your application. You simply spin up new worker machines, and Citus moves the existing shards over to the new hardware in the background while the application keeps running.

Running Citus Locally

We will use Docker to run Citus locally. Why use Docker instead of adding the extension directly to your host machine's Postgres installation?

Simulating a distributed database means running multiple Postgres instances simultaneously. Doing this natively forces you to manually manage conflicting ports and multiple data directories. Docker Compose solves this by orchestrating the entire cluster's networking with a single command, giving you a clean state that can be torn down and rebuilt in seconds.

Single Node vs Multi-Node Setup

While it is possible to run a single node Citus instance using a simple docker run command:

docker run -d --name citus_standalone -p 5432:5432 -e POSTGRES_PASSWORD=postgres citusdata/citus:14.1.0

This guide will focus exclusively on setting up a Multi-Node Cluster. If you are interested in exploring the single node setup further, you can check out the official Citus documentation here.

For our distributed, multi-node cluster, we will use the following Docker Compose configuration:

services:
  master:
    container_name: "${PROJECT_NAME:-citus}_master"
    image: "citusdata/citus:14.1.0"
    ports: ["${COORDINATOR_EXTERNAL_PORT:-5433}:5432"]
    labels: ["com.citusdata.role=Master"]
    environment: &AUTH
      POSTGRES_USER: "${POSTGRES_USER:-postgres}"
      POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
      PGUSER: "${POSTGRES_USER:-postgres}"
      PGPASSWORD: "${POSTGRES_PASSWORD}"
      POSTGRES_HOST_AUTH_METHOD: "${POSTGRES_HOST_AUTH_METHOD:-trust}"
  worker:
    image: "citusdata/citus:14.1.0"
    labels: ["com.citusdata.role=Worker"]
    environment: *AUTH

Citus has a built-in initialization script in its official Docker image. When launched via Docker Compose with these specific labels like com.citusdata.role, the coordinator automatically detects and registers the workers for you.

We can build and start the cluster in the background with:

docker compose up -d

We can also scale up dynamically. If you want to spin up a second or third worker machine on the fly, you do not even need to edit the compose file. Just run:

docker compose up -d --scale worker=3

Where 3 is the total number of workers you want to have running.

Simply spinning up the worker containers is not enough. After adding new worker nodes, we must register them with the coordinator so Citus knows they are active and ready to hold data. To do this, we will first connect to the interactive Postgres terminal on the master node:

docker compose exec master psql -U postgres

Now we can register our workers:

SELECT citus_add_node('db-sharding-worker-1', 5432);
SELECT citus_add_node('db-sharding-worker-2', 5432);
SELECT citus_add_node('db-sharding-worker-3', 5432);

So, if you scale up again to say 5 workers, you will have to run this command 2 more times to register the 2 new workers.

Notice the db-sharding- prefix in the hostnames. This is necessary because Docker Compose prefixes your project folder name to the container's network alias.

A Real World Scenario

In order to showcase the functionality of this database sharding setup, let's try to work with a real world scenario.

Consider an e-commerce multi-tenant application. It serves multiple independent companies that all share the same database infrastructure. To scale this effectively, we need to use a hash-based distribution with a well-chosen shard key. Let's assume by design, every tenant has a shared company_id, we will use that as our shard key.

By using company_id as the shard key across all operational tables, Citus guarantees that all data belonging to a specific company (like its users, its products, and its orders) is physically co-located on the exact same worker node.

Now that our workers are registered, we can create our core tables directly in the master node. Citus will automatically handle the underlying sharding for us.

Please note that the following schema is highly simplified to focus on the core concepts of sharding.

Run the following SQL commands in your coordinator's postgres interactive terminal:

-- 1. Create Tenant Table
CREATE TABLE companies (
    company_id INT PRIMARY KEY,
    company_name TEXT NOT NULL
);

-- 2. Create Tenant-Scoped Orders Table
CREATE TABLE orders (
    company_id INT NOT NULL,
    order_id INT NOT NULL,
    user_id INT NOT NULL,
    status_id INT NOT NULL, -- References our global order status table
    order_amount NUMERIC(10,2) NOT NULL,
    PRIMARY KEY (company_id, order_id)
);

-- 3. Create Global Order Reference Table
CREATE TABLE order_statuses (
    status_id INT PRIMARY KEY,
    status_name TEXT NOT NULL
);

-- 4. Create Tenant-Scoped Users Table
CREATE TABLE users (
    company_id INT NOT NULL,
    user_id INT NOT NULL,
    username TEXT NOT NULL,
    PRIMARY KEY (company_id, user_id)
);

-- 5. Create Tenant-Scoped Products Table
CREATE TABLE products (
    company_id INT NOT NULL,
    product_id INT NOT NULL,
    product_name TEXT NOT NULL,
    price NUMERIC(10,2) NOT NULL,
    PRIMARY KEY (company_id, product_id)
);

Telling Citus How to Shard and Replicate

Now that we have created our core tables, we have to tell Citus how to shard and replicate them across our worker nodes.

Reference Tables

First, we will set order_statuses as a reference table. We do this using create_reference_table.

Why do we create a reference table? A reference table is designed for relatively small, slowly-changing data like user roles, country codes, or order statuses.

When you mark a table as a reference table, Citus physically replicates a full copy of that entire table to every single worker node. This bypasses network overhead completely during JOIN operations. Because the worker node already has the data locally, it does not need to fetch it from another node to complete a query.

Distributed Tables

Next, we need to distribute our large tenant-data tables. We do this using create_distributed_table.

This function is the core of Citus's sharding mechanism. It tells the coordinator to partition the table based on a specific column (our shard key, company_id), and spread those partitions across the available worker nodes.

Because we use company_id across all these tables, any query filtering by a specific company will be routed directly to a single node, making it incredibly fast.

-- Replicate order_statuses to ALL nodes
SELECT create_reference_table('order_statuses');

-- Shard data based on the Hash of company_id
SELECT create_distributed_table('companies', 'company_id');
SELECT create_distributed_table('users', 'company_id');
SELECT create_distributed_table('products', 'company_id');
SELECT create_distributed_table('orders', 'company_id');

Multi-Tenant Seed Data

Now we will populate our cluster with some dummy companies along with some global order status entries, orders, and tenant users.

-- Global Order Lookups
INSERT INTO order_statuses VALUES (1, 'Pending'), (2, 'Shipped'), (3, 'Delivered');

-- Company Tenants
INSERT INTO companies VALUES (1, 'EbaMart'), (2, 'FufuHub'), (3, 'GymGear');

-- Multi-Tenant Users scattered across companies
INSERT INTO users (company_id, user_id, username) VALUES
(1, 101, 'Samuel'),   -- EbaMart Customer
(2, 201, 'Folashade'),-- FufuHub Customer
(1, 102, 'Umoh'),     -- EbaMart Customer
(3, 301, 'Oladejo'),  -- GymGear Customer
(2, 202, 'BigSam');   -- FufuHub Customer

-- Orders
INSERT INTO orders (company_id, order_id, user_id, status_id, order_amount) VALUES
(1, 5001, 101, 1, 150.00),  -- EbaMart order: Pending
(1, 5002, 102, 3, 45.00),   -- EbaMart order: Delivered
(2, 6001, 201, 2, 99.00),   -- FufuHub order: Shipped
(2, 6002, 202, 1, 250.00);  -- FufuHub order: Pending

Verify Layout and Node Placement

We can now verify the layout and node placement to see where they are added to and display it by running:

SELECT
    u.company_id,
    u.user_id,
    u.username,
    s.shardid,
    s.nodename
FROM users u
JOIN citus_shards s
  ON s.shardid = get_shard_id_for_distribution_column('users', u.company_id)
ORDER BY u.company_id;

Output:

company_id	user_id	username	shardid	nodename
1	101	Samuel	102042	db-sharding-worker-2
1	102	Umoh	102042	db-sharding-worker-2
2	201	Folashade	102065	db-sharding-worker-1
2	202	BigSam	102065	db-sharding-worker-1
3	301	Oladejo	102056	db-sharding-worker-1

As you can see, all data for company_id = 1 sits neatly on db-sharding-worker-2, while company_id = 2 sits on db-sharding-worker-1. This guarantees that queries for a single tenant will only hit one node.

We can also inspect the query plan for joining distributed tables with reference tables:

EXPLAIN (VERBOSE, COSTS OFF)
SELECT o.order_id, o.order_amount, s.status_name
FROM orders o
JOIN order_statuses s ON o.status_id = s.status_id
WHERE o.company_id = 1;

Output:

                                      QUERY PLAN
---------------------------------------------------------------------
---------------------------------------------------------------------
---------------------------------------------------------------------
------------------------
 Custom Scan (Citus Adaptive)
   Output: remote_scan.order_id, remote_scan.order_amount, remote_scan.status_name
   Task Count: 1
   Tasks Shown: All
   ->  Task
         Query: SELECT o.order_id, o.order_amount, s.status_name FROM (public.orders_102106 o JOIN public.order_statuses_102008 s ON ((o.status_id OPERATOR(pg_catalog.=) s.status_id))) WHERE (o.company_id OPERATOR(pg_catalog.=) 1)
         Node: host=db-sharding-worker-2 port=5432 dbname=postgres
         ->  Hash Join
               Output: o.order_id, o.order_amount, s.status_name
               Hash Cond: (s.status_id = o.status_id)
               ->  Seq Scan on public.order_statuses_102008 s
                     Output: s.status_id, s.status_name
               ->  Hash
                     Output: o.order_id, o.order_amount, o.status_id
                     ->  Bitmap Heap Scan on public.orders_102106 o
                           Output: o.order_id, o.order_amount, o.status_id
                           Recheck Cond: (o.company_id = 1)
                           ->  Bitmap Index Scan on orders_pkey_102106
                                 Index Cond: (o.company_id = 1)
(19 rows)

The query plan shows a Custom Scan (Citus Adaptive) delegating the query to the specific worker node holding company_id = 1.

We can check the active worker nodes with:

SELECT * FROM master_get_active_worker_nodes();

Output:

node_name	node_port
db-sharding-worker-3	5432
db-sharding-worker-1	5432
db-sharding-worker-2	5432

Rebalancing the Cluster

Anytime you dynamically scale the cluster and register a new worker node, you must run the Citus Shard Rebalancer:

SELECT citus_rebalance_start();

When new nodes are added, the coordinator does not automatically move existing data onto them. We have to explicitly balance the load. The rebalancer handles this by dynamically moving a subset of your tenants' data over to the new worker servers in the background, without taking the database offline.

Production Setup

The Docker Compose setup is perfect for local testing and development. However, running a distributed database in production across multiple physical servers or virtual machines requires a slightly different approach.

We must first change how network addressing and orchestration work. In a local setup, Docker Compose handles networking automatically because everything shares a single machine loopback. On production VMs, each container must be bound to the public or private IP address of its physical host machine, and you must enforce strict firewall security.

1. Spin up your VMs

Spin up your virtual machines on your preferred cloud provider. Assuming we provision 4 VMs, our network layout will look something like this:

VM 1 (Coordinator Node): Private IP 10.0.0.10
VM 2 (Worker Node 1): Private IP 10.0.0.11
VM 3 (Worker Node 2): Private IP 10.0.0.12
VM 4 (Worker Node 3): Private IP 10.0.0.13

I used random private IPs, but you should substitute them with your real VMs' private IPs.

2. Install Docker

Next, you will need to install Docker and Docker Compose on every VM. The installation commands will depend on your specific Linux distribution, so refer to the official Docker documentation for your OS.

You will no longer use a single master compose file for the cluster. Instead, you deploy an isolated, optimized compose layout directly on each corresponding VM instance.

3. The Coordinator Instance (VM 1)

Create a docker-compose.yml file specifically tailored to map the Postgres engine port directly out to the host VM network interface:

services:
  master:
    container_name: "citus_coordinator"
    image: "citusdata/citus:14.1.0"
    ports:
      - "5432:5432"
    restart: always
    environment:
      POSTGRES_USER: "citus_admin"
      POSTGRES_PASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      PGUSER: "citus_admin"
      PGPASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      POSTGRES_HOST_AUTH_METHOD: "scram-sha-256"
    volumes:
      - coordinator_data:/var/lib/postgresql/data

volumes:
  coordinator_data:

4. The Worker Instances (VMs 2, 3, and 4)

Every worker VM gets its own standalone compose file. Notice we expose port 5432 on the workers too, because the coordinator node needs a direct network pipe to talk to them:

services:
  worker:
    container_name: "citus_worker"
    image: "citusdata/citus:14.1.0"
    ports:
      - "5432:5432"
    restart: always
    environment:
      POSTGRES_USER: "citus_admin"
      POSTGRES_PASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      PGUSER: "citus_admin"
      PGPASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      POSTGRES_HOST_AUTH_METHOD: "scram-sha-256"
    volumes:
      - worker_data:/var/lib/postgresql/data

volumes:
  worker_data:

5. Configure Network Firewall Security

Since you are running across independent cloud instances, your cloud platform's Security Groups or Firewalls (like AWS Security Groups or ufw on Ubuntu) must be strictly configured to prevent public database scanning.

On the Coordinator VM (VM 1): Allow inbound TCP traffic on port 5432 from Anywhere (0.0.0.0/0) only if your external application servers need public access. If your application servers are in the same VPC, restrict inbound port 5432 access exclusively to your app servers' IP subnet range.
On the Worker VMs (VMs 2, 3, 4): Allow inbound TCP traffic on port 5432 EXCLUSIVELY from the Coordinator's Private IP (10.0.0.10). Block all other inbound network requests to the workers. They should never be exposed to the internet.

Next, log in to each VM and run the containers:

docker compose up -d

Once all containers are healthy, access the interactive postgres interface on VM 1 (the coordinator node):

docker compose exec master psql -U citus_admin

SELECT citus_add_node('10.0.0.11', 5432); -- Register Worker VM 2
SELECT citus_add_node('10.0.0.12', 5432); -- Register Worker VM 3
SELECT citus_add_node('10.0.0.13', 5432); -- Register Worker VM 4

Verify everything works:

SELECT * FROM master_get_active_worker_nodes();

6. Configure Production Password Authentication

Because we updated POSTGRES_HOST_AUTH_METHOD to scram-sha-256 for security, your instances will require passwords to authenticate cross-node.

We must add a Postgres password passfile on the coordinator container:

docker compose exec master bash -c "echo '*:*:*:citus_admin:YOUR_SUPER_SECURE_PRODUCTION_PASSWORD' >> /var/lib/postgresql/.pgpass && chmod 600 /var/lib/postgresql/.pgpass"

Surely you will know to replace YOUR_SUPER_SECURE_PRODUCTION_PASSWORD with your actual password.

Here is a detailed documentation of Citus multi-node installation and setup for reference: Citus Multi-Node Setup.

Adding Citus to an Existing Project

To add Citus to an already existing Postgres database project, the process follows these general steps:

Install the Extension: Install the Citus extension on your Postgres server. You will also need to add citus to the shared_preload_libraries in your postgresql.conf file and restart the database.
Create the Extension: Run CREATE EXTENSION citus; in your database.
Select a Shard Key: Identify the column that will serve as the shard key for your distributed tables, such as a company or tenant ID.
Convert Tables: Use create_distributed_table() to convert your existing tables into distributed tables. Use create_reference_table() for lookup tables that do not need to be sharded.
Migrate Data: If you already have data in the tables, it might need to be exported and loaded back in, depending on the volume of data and the specific requirements of the migration.

Conclusion

Database sharding is an incredibly powerful architectural pattern, but it comes with immense operational tradeoffs. By using tools like Citus, we can bridge the gap between standard relational SQL databases and massively distributed systems.

Whether you are starting a new multi-tenant application or scaling out an existing infrastructure, adopting a distributed mindset and orchestrating your local environments with tools like Citus and Docker Compose will save you countless hours of architectural nightmares down the road.

You can find the full Docker Compose source code for this tutorial on GitHub: umohsamuel/database-sharding.

References

Data Engineering Beyond "Pretty Charts": How I Built an Aviation Telemetry Leakage Engine

Wed, 23 Sep 2026 12:10:54 +0000

Stop Tracking "On-Time" Flights. Track Tarmac Fuel Destruction Instead.

Most aviation dashboards track gate-to-gate "On-Time" arrivals. The problem? It's a lagging indicator that completely masks why airlines silently hemorrhage millions. Padded flight schedules disguise a massive bottleneck: chronic tarmac fuel destruction occurring right before takeoff.

I built AeroOps, an end-to-end telemetry and multi-domain operations leakage engine, to expose exactly where systems drop value between the hangar and the runway.

Repo & Technical Report: [GitHub Link]
Stack: MySQL (Star Schema & CTEs) · Telemetry Data Modeling · Executive Dashboard Architecture · Power BI (DAX)

The Architectural & Engineering Challenge

Handling flight telemetry means dealing with high-volume, disjointed datasets. To make this actionable, the engineering required:

Star Schema Data Warehousing: Joined 513,000+ flight legs with precise technical burn parameters pulled directly from the ICAO Aircraft Emissions Databank.
Bypassing Relational Boundaries: Formulated deduplicated DAX lookup models to handle wide-body propulsion penalties under sub-second memory constraints.
Algorithmic Baseline Filters: Enforced a strict 4-minute engine warm-up buffer directly into the SQL logic. This stripped away safe, expected operational windows to isolate true, excess ground latency.

The Bottom-Line Impact Metrics

By querying raw transaction logs and user event data instead of building passive, "pretty" charts, the engine surfaced massive operational friction:

$42.69M in Tarmac Cash Destruction: Isolated and quantified the exact cost of ground traffic gridlocks at major hubs like Atlanta (ATL).
54% to 57% Short-Haul Menu Spoilage: Exposed a massive waste loop across major carriers (Delta/American) due to over-boarding. Transitioning the system to a 100% shelf-stable inventory model reclaimed $2.58M in simulated margin leaks.
$725K Legal Fine Exposure: Isolated a hidden 9% telemetry sync drop rate specifically among regional commuter airframes caused by corrupted logs.
Prevention of $1.50M Asset Groundings (AOG): Coded an automated Reorder Point (ROP) trigger tool to insulate warehouses against catastrophic out-of-stock asset scenarios.

The Takeaway

Data shouldn't just look nice in a presentation deck. If your queries aren't actively protecting business margins or exposing systemic infrastructure leaks, you're looking at the wrong metrics.

If your team is scaling fast and dealing with messy relational pipelines or unoptimized telemetry data, let's connect and fix your backend infrastructure loops.

Reverse-Engineering Messy Databases: My End-to-End Data Audits Across 17,000+ Relational Schema Logs

Wed, 23 Sep 2026 11:56:39 +0000

From Raw Logs to Business Strategy: How I Taught Myself Data Engineering & Analytics

I didn’t take the traditional path into data. Over the past 12 months, I bypassed the university lecture halls to teach myself relational database architecture, pipeline optimization, and growth analytics by diving straight into raw datasets.

I look at data through an engineering and business lens. Instead of building passive charts that just look pretty, I query transaction logs and user event data to find hidden operational friction, map out user behavior patterns, and isolate exactly where systems are dropping value. I build end-to-end data systems: raw SQL schemas, data sanitization scripts, interactive Power BI dashboards, and single-page executive reports.

Below are the architectural breakdowns, technical challenges, and business impact metrics of the four core data infrastructure projects I have engineered from scratch.

Project 1: StockPulse — Multi-Regional Supply Chain Audit

Technical Stack: MySQL (Mathematical Window Functions), Power BI, Advanced Excel
GitHub Codebase & Dashboard Layout: [LINK]

The Operational Bottleneck

Horizon Marketplace faced shrinking profit margins from slow-moving warehouse stock, while simultaneous supplier shipping delays caused high-demand items to go out of stock on our mobile application. Traditional procurement tracking models failed because they relied on simple historical averages, which immediately miscalculate when data hits high seasonal sales volatility spikes.

The Technical Implementation

Audited 17,646 operational and sales ledger transactions using MySQL window functions to calculate cumulative revenue contributions per SKU.
Coded a predictive restocking matrix script utilizing sales volatility standard deviations and rolling manufacturer lead timelines.
Formulated dynamic Safety Stock buffers and programmed automated database Reorder Point triggers to flag supplier order alerts before shelves hit zero.

The Business Impact Metrics

ABC inventory segmentation proved that exactly 23 Class A items drive 80% of total marketplace revenue, showing that the remaining catalog space was tying up critical capital.
Quantified a hidden revenue bleed of $63,897.26 caused by a single high-demand apparel product sitting entirely out of stock for 42 days due to upstream logistics bottlenecks.
Transitioned a multi-regional operation from manual, time-intensive spreadsheets to an automated, predictive data framework.

Project 2: FunnelOpt — HabitArcade B2C Telemetry Audit

Technical Stack: MySQL (Session Overrides), Funnel Analysis, User Lifecycle Mapping
GitHub Codebase & Database Logs: [LINK]

The Operational Bottleneck

A B2C application tracking engine suffered from severe data execution drops where product behavioral logs completely failed to map user activation patterns. The relational schema dropped child telemetry records because they were importing prior to the absolute database finalization of the parent account index logs.

The Technical Implementation

Audited full-funnel growth logs across 6 independent acquisition pipelines to analyze customer conversion latency and drop-offs.
Structured specific MySQL session overrides to temporarily bypass structural foreign key constraints.
Forced clean data synchronization across child records to completely restore parent-child index synchronization without corrupting historical system logs.

The Business Impact Metrics

Isolated a systemic “Streak-Breaking Crash Tax” where application sync errors wiped 30-day user milestones, triggering immediate 1-star reviews and an active 6% to 7% monthly subscriber bleed.
Exposed an “Inverted Value Curve” revealing that high-value annual checkout tiers underperformed monthly subscriptions due to silent gateway billing processor failures blocking top-priced subscription cash settlement.

Project 3: LeakStop — Olist B2B Revenue Leakage & Onboarding Audit

Technical Stack: MySQL (Schema Engineering), Power BI, Business Intelligence
GitHub Codebase & Analytical Readme: [LINK]

The Operational Bottleneck

Cross-vendor data duplication across digital marketplace ledgers was heavily skewing gross reporting metrics. Standard relational database joins double-counted total transactions when a customer order spanned multiple independent business vendors in a single checkout loop.

The Technical Implementation

Engineered robust MySQL ingestion and data sanitization scripts to enforce strict primary key integrity down to individual item vectors, removing all reporting bias.
Processed user account lifecycles to calculate onboarding completion latency across specific B2B verticals.
Programmed a real-time database filter to flag slipping user accounts precisely on Day 16 of transaction inactivity.

The Business Impact Metrics

Isolated an annual revenue leakage of R$ 600,600 (approximately $119,175 USD) stemming from 462 merchant accounts that stalled out completely during the onboarding phase.
Exposed an invisible latency anomaly where logistically complex business profiles stalled for an average of 112 days in domain routing and tax mapping phases.
Implemented a Day 16 early-warning database filter to flag at-risk accounts before they reached a terminal, zero-recovery retention cliff.

Project 4: CohortPace — SaaS Behavioral Retention Engine

Technical Stack: MySQL (Subqueries & Relational Joins), Cohort Modeling, Data Analytics
GitHub Codebase & Technical Report: [LINK]

The Operational Bottleneck

Initial application dashboards focused blindly on feature exploration depth, which completely masked why healthy-looking active user profiles were suddenly canceling subscriptions. Traditional active-user metrics acted as a lagging indicator, hiding friction because churning accounts remain active inside broken workflows right up to their exit timelines.

The Technical Implementation

Modeled simulated user cohort transactions across multi-platform SaaS behavioral profiles to analyze the core drivers of long-term Monthly Recurring Revenue (MRR).
Used advanced MySQL subqueries and relational joins to reconstruct full lifecycle interaction sequences.
Mapped user interaction velocity trends against subscription lifecycles to isolate clear behavioral churn signatures.

The Business Impact Metrics

Proved through data pacing sending signals that churning users maintain consistent interaction velocity close to their exit date, showing churn is driven by late-stage workflow utility limits rather than gradual disengagement.
Demonstrated that long-term user monetization is strictly dependent on habit formation within a single core workflow rather than broad, multi-feature platform exploration.

Conclusion: Turning Code Into Business Strategy

I don’t need a playbook to get started, and I don’t need a manager to hand me a static checklist. I find the question, ingest the data, clean the schemas, model the answer, and present the metric that directly protects business margins.

I am actively available for analytical remote contract roles, standalone data audits, and freelance pipeline optimization. If your startup is scaling fast and dealing with messy relational pipelines, let’s connect and fix your backend infrastructure loops.

Interactive Portfolio Hub: [LINK]
Contact: motiwala603@gmail.com
Connect with me: [LinkedIn]

Open to analyst remote roles across product, growth, and business analytics. Based in Pakistan, available worldwide.

Personal Expense Tracker V2 — Edit, Validation & Error Handling

Tue, 22 Sep 2026 17:19:39 +0000

I’ve completed V2 of my Personal Expense Tracker

This project started as a way to practice Python OOP, SQL, and SQLite by building something practical instead of only studying theory.

V1 focused on getting the core CRUD operations working.

With V2, I focused on making the application more reliable and user-friendly.

What I Added in V2

✏️ Edit Expenses

Users can now modify an existing expense.

The edit flow allows changing:

Amount
Category
Description
Date
Cancel the operation

This also gave me more practical experience working with SQL UPDATE statements and designing application flow around user choices.

️ Input Validation

I also added validation for user input.

The application now handles things such as:

Invalid menu choices
Invalid amount input
Negative or zero amounts
Invalid category selection
Empty descriptions
Invalid dates
Impossible dates such as 2026-02-30

For example, instead of allowing invalid input to crash the program, the application asks the user to provide a valid value.

⚠️ Error Handling

While implementing validation, I also started thinking more carefully about where errors can happen and how the application should respond.

One important lesson was that input validation is not just about preventing crashes.

It is about defining what the application considers valid input and controlling the flow when the user provides something unexpected.

What I Learned

V2 taught me more than just how to add an edit feature.

I learned about:

Designing validation rules before implementing them
Using try/except for user input
Separating application flow from business logic
Handling different types of invalid input
Thinking about edge cases
Designing better CLI user interaction
Keeping the project scope under control

One thing I learned during this version is that not every possible improvement needs to be implemented immediately.

There are still things I could add, such as more advanced database error handling and additional checks, but I decided to stop V2 at the scope I originally defined.

That is also part of learning software development: knowing when a version is complete.

️ Current Architecture

The project currently follows a simple separation of responsibilities:

main.py
   ↓
ExpenseManager
   ↓
Database / storage.py
   ↓
SQLite

main.py handles the CLI/application flow, ExpenseManager handles the application logic, and storage.py handles database operations.

V2 Status

V2 is now complete.

The project currently supports:

✅ Add expenses
✅ View expenses
✅ Edit expenses
✅ Delete expenses
✅ Category selection
✅ Input validation
✅ Date validation
✅ Basic error handling

The main goal of this version was not to build a huge application.

It was to take the working V1 and make it more reliable while strengthening my understanding of Python, OOP, SQL, SQLite, and application design.

Project

GitHub Repository: Personal Expense Tracker

I’ll continue improving the project in future versions when there is something meaningful to learn or add.

Thanks for reading!

Personal Expense Tracker — V1 Complete: Building the Core Working Stage

Tue, 22 Sep 2026 17:10:47 +0000

I’ve completed V1 of my Personal Expense Tracker, a small learning project built with Python and SQLite.

This project was created primarily to learn by building rather than following a tutorial step by step.

The goal of V1 was not to build a production-ready expense tracker.

The goal was:

Build the core application flow, understand the responsibilities of each layer, and make the fundamental functionality work end-to-end.

With that goal achieved, I consider V1 complete.

Why I Built This Project

I wanted to strengthen my understanding of:

Python OOP
Classes and objects
Object responsibilities
SQL and SQLite
CRUD operations
Database design
Separating application responsibilities
How different layers communicate with each other

Instead of learning these concepts separately, I decided to combine them into one small practical project.

That's why I consider this primarily a learning project.

The objective was not to create a feature-rich application immediately, but to understand the fundamentals by actually building something.

What I Built

The application is a command-line Personal Expense Tracker.

The core V1 functionality includes:

Adding an expense
Viewing expenses
Deleting an expense
Storing expenses in SQLite
Using SQLite-generated IDs
Separating the application into different responsibilities

The database currently contains an expenses table:

Column	Type
`id`	INTEGER PRIMARY KEY
`amount`	REAL
`category`	TEXT
`description`	TEXT
`date`	TEXT

Application Architecture

One of the main things I wanted to learn was how to organize responsibilities in an application.

I structured the project into three main layers:

main.py
   ↓
ExpenseManager
   ↓
Database
   ↓
SQLite

`main.py` — UI Layer

main.py is responsible for interacting with the user through the CLI.

Its responsibilities include:

Displaying the menu
Taking user input
Calling the appropriate operation
Displaying results

The UI does not directly handle SQL operations.

`expense_manager.py` — Application Layer

ExpenseManager acts as the middle layer between the UI and storage.

It coordinates application operations and communicates with the database object.

For example:

main.py
   ↓
ExpenseManager.add_expense()
   ↓
Database.insert_expense()

This helped me understand the difference between the application's operation and the actual persistence of data.

`storage.py` — Persistence Layer

The Database class handles communication with SQLite.

Its responsibilities include:

Creating the database connection
Initializing the database
Creating the table
Inserting expenses
Retrieving expenses
Deleting expenses

This keeps database-related responsibilities separate from the UI.

Understanding Object Relationships

Another important part of this project was understanding how objects can work together.

The application creates a database object and passes it to ExpenseManager.

Conceptually:

main.py
   │
   ├── creates Database
   │
   └── creates ExpenseManager
             │
             └── uses Database

The manager receives the database dependency instead of creating its own database internally.

This was one of the OOP concepts I wanted to understand through practical implementation.

Why V1 Is Complete

There is an important distinction between:

"V1 is complete"

and

"the project is finished."

V1 is complete because its intended scope has been achieved.

The core application flow works:

User
 ↓
CLI
 ↓
ExpenseManager
 ↓
Database
 ↓
SQLite

An expense can be added and stored, stored expenses can be retrieved and displayed, and expenses can be deleted.

The different layers can communicate with each other correctly.

That is the foundation I wanted to establish in V1.

What I Learned From V1

The biggest lesson wasn't simply learning SQLite syntax or Python syntax.

It was understanding responsibility.

I learned to think about the application as separate parts with different jobs:

UI
→ communicates with the user

Manager
→ coordinates application operations

Storage
→ communicates with the database

Along the way, I gained practical experience with:

Python classes
Constructors and attributes
Methods
Object relationships
Passing dependencies between objects
SQLite tables
Primary keys
CRUD operations
SQL queries
Multi-module Python projects
Debugging
Git and GitHub

What I Didn't Focus On in V1

I deliberately didn't try to make V1 handle every possible invalid input or edge case.

Comprehensive validation and robust error handling were not the main goals of this version.

Instead, I decided to make the core functionality work first and move robustness-related improvements into V2.

This gives the project a clearer progression:

V1
Core Working Stage
       ↓
V2
Robustness + Additional Core Operation

What's Coming in V2?

Now that the core working stage is complete, V2 will focus on improving the application's robustness and functionality.

The planned V2 work includes:

Validation

Validating user input
Validating expense IDs
Handling invalid values
Handling invalid menu input

Error Handling

Handling invalid user input gracefully
Handling invalid/non-existent expense IDs
Handling database-related errors
Providing better user-facing error messages

Edit Expense

V2 will also introduce an Edit Expense operation.

The core expense operations will then become:

Create
Read
Update
Delete

or simply:

CRUD

This will make the expense management functionality more complete.

V1 → V2

The project's progression can be summarized as:

V1
│
├── Add Expense
├── View Expenses
├── Delete Expense
├── SQLite persistence
├── Layered architecture
└── Core application flow
        │
        ▼
V2
│
├── Edit Expense
├── Input validation
├── ID validation
├── Error handling
└── CLI refinement

I intentionally want to keep these stages separate.

V1 proves that the foundation works.

V2 will make that foundation more robust and complete.

Final Thoughts

This project started as a learning exercise to understand Python OOP and SQLite by building something practical.

Completing V1 gave me more than just another project for my GitHub repository.

It helped me understand how a small application can be divided into responsibilities and how those components communicate with each other.

I'm treating V1 as a completed learning milestone, not as the final state of the application.

The core working stage is done.

Project

GitHub Repository: Personal Expense Tracker

Now it's time to build on that foundation.

V1: Complete.

Core working stage: Achieved.

Next: V2 — Validation, Error Handling & Edit Expense.

The Silent Performance Killer: Demystifying the N+1 Query Problem in Node.js

Mon, 21 Sep 2026 09:18:48 +0000

Have you ever noticed a feature that works beautifully on your local computer, only to crawl at a snail's pace once it gets deployed to production? One of the most common, yet invisible, reasons for this sudden slowdown is a database fetching behavior known as the N+1 query problem.

In short, the N+1 query problem occurs when an application makes one initial database query to fetch a list of records (the "1"), and then makes a separate database query for every single record returned in that list (the "N") to fetch related data. Instead of fetching all the required information in a single, efficient request, the application ends up chatting back and forth with the database dozens, hundreds, or thousands of times.

The Restaurant Server Analogy

To understand why this is so inefficient, imagine sitting down at a busy restaurant with a table of ten friends. It is time to order drinks.

An efficient server (who behaves like an optimized database query) would take everyone's drink orders on a notepad, walk to the kitchen once, load all ten glasses onto a single tray, and bring them back to the table in one trip. This requires just one round-trip to the kitchen.

Now imagine an N+1 server. This server walks to the table, asks the first person what they want, and walks all the way back to the kitchen to fetch that single drink. After serving it, they walk back to the table, ask the second person for their order, and make another complete round-trip to the kitchen. They repeat this individual process for all ten people. By the time everyone has a drink, the server has made eleven total trips (1 trip to inspect the table, plus N trips for the N guests). Your dining experience would be incredibly slow, and the poor server would be completely exhausted.

Why This Matters in Daily Engineering

In real-world software engineering, database round-trips are remarkably expensive. Every single query you run introduces network latency, database parsing overhead, and thread consumption in your database's connection pool.

If you are building a social media dashboard that displays 50 posts, and you load the comments for each post individually, you are executing 51 database queries. If 1,000 users visit that page simultaneously, your application will suddenly bombard your database with 51,000 queries. This easily exhausts your database connection pool, drives database CPU usage to 100%, and results in catastrophic timeouts for your users. Engineers must actively prevent this to keep application response times low and infrastructure costs manageable.

Seeing It in Action: Node.js and MySQL

Let's look at a typical Express.js endpoint where this mistake happens, followed by how we fix it.

Here is the problematic, naive approach:

// BAD: This triggers N+1 queries
app.get('/api/posts', async (req, res) => {
  try {
    // 1. Fetch all posts (The "1" query)
    const [posts] = await db.query('SELECT id, title, content FROM posts LIMIT 50');
    
    // 2. Loop through each post to fetch its comments (The "N" queries)
    for (let post of posts) {
      const [comments] = await db.query(
        'SELECT id, text FROM comments WHERE post_id = ?',
        [post.id]
      );
      post.comments = comments;
    }
    
    res.json(posts);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

Here is how we resolve this using a SQL JOIN to pull all the information in a single, efficient query:

// GOOD: Resolved using a single JOIN query
app.get('/api/posts', async (req, res) => {
  try {
    const query = `
      SELECT 
        p.id AS post_id, 
        p.title, 
        p.content, 
        c.id AS comment_id, 
        c.text AS comment_text
      FROM posts p
      LEFT JOIN comments c ON p.id = c.post_id
      LIMIT 50
    `;
    
    const [rows] = await db.query(query);
    
    // Group the flat SQL results into structured JSON
    const postsMap = {};
    for (const row of rows) {
      if (!postsMap[row.post_id]) {
        postsMap[row.post_id] = {
          id: row.post_id,
          title: row.title,
          content: row.content,
          comments: []
        };
      }
      if (row.comment_id) {
        postsMap[row.post_id].comments.push({
          id: row.comment_id,
          text: row.comment_text
        });
      }
    }
    
    res.json(Object.values(postsMap));
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

The Takeaway

The N+1 query problem highlights that how you fetch your data is just as important as the data itself. By switching from sequential, iterative loops to combined database operations like SQL JOINs, you minimize the highly expensive overhead of network round-trips. Designing your data fetching strategies proactively keeps your APIs incredibly fast and ensures your databases stay healthy, stable, and cost-efficient under high production loads.

Originally published on my blog. You can read the alternative breakdown here.

Rate Limiting Isn't One Layer: What LogicVisor and Titan Actually Do Differently

Sat, 19 Sep 2026 09:19:55 +0000

"Add rate limiting" sounds like one task. It's actually three separate layers, each defending against a different failure mode, with different trade-offs and different amounts of trust you can place in them. I only understood the distinction properly once I had to pick a strategy for two systems solving different problems: LogicVisor (a public AI tool anyone can hit anonymously) and Titan (a payments platform where the cost of a bad actor is different from the cost of someone burning through free AI credits).

1. Client-side: UX, not security

Debouncing a search input, greying out a submit button after the first click, backing off exponentially after a 429. All of this makes an app feel considerate. None of it stops anyone. A malicious actor skips your JavaScript entirely and hits the endpoint directly with curl. Client-side limiting is worth doing (it saves you real traffic and prevents accidental double-submits), but it is not a security control. If it's the only thing standing between your API and abuse, you don't have rate limiting, you have a polite suggestion.

2. Server-side: the layer that actually protects you

This is where LogicVisor and Titan diverge, because they're not defending against the same thing.

LogicVisor: several checks before a single AI token gets spent

LogicVisor is public. Anyone gets 3 free code reviews with no signup, which means the abuse surface is wide open by design. The submission route runs a stack of checks before it ever calls Gemini or Groq, because every AI call costs real money:

// 1. Check if this exact code has already been reviewed by this model
const cachedReview = await getCachedAIReview(preferred_model.id + "-" + canonicalHash);
if (cachedReview) {
  return NextResponse.json({ success: true, data: cachedReview }, { status: 201 });
}

// 2. Enforce the actual rate limit (IP + session based for anon users)
const rateLimitResult = await enforceAIRateLimit(user.id, request);

// 3. Slow down premium users intelligently instead of hard-blocking them
const throttleDelay = await getThrottleDelay(user.id);
if (throttleDelay > 0) {
  await new Promise((resolve) => setTimeout(resolve, throttleDelay));
}

// 4. Under heavy load, degrade gracefully instead of rejecting outright
const shouldDegrade = await shouldGracefullyDegrade(user.id, "ai_request");
if (shouldDegrade) {
  // return a basic, non-AI response instead of a 429
}

A few things worth naming separately, because they get lumped together under "rate limiting" but aren't the same mechanism:

Abuse prevention: anonymous sessions are capped at 3 reviews, tracked by a UUID session (7-day expiry) plus a browser fingerprint (User-Agent + Accept-Language + Accept-Encoding). This is the classic server-side layer, IP and fingerprint based, closest to the "token bucket per identity" pattern.
Cost control via deduplication: before rate limiting even runs, the code is canonicalized (AST-based) and hashed. If someone (or the same person twice) submits identical logic, they get the cached review instead of triggering a new paid AI call. This isn't rate limiting in the strict sense, it's closer to database-level duplicate detection, but it does the same job of protecting a scarce resource.
Graceful degradation over hard rejection: instead of a flat 429 when usage spikes, authenticated users can get a stripped-down, non-AI response. It costs nothing and still gives the user something, rather than a wall.

None of this is a single algorithm out of a textbook. It's several cheap, layered checks, ordered so the most expensive resource (the AI call) is the last thing hit, not the first.

Titan: NestJS Throttler backed by Redis

Titan's rate limiting is intentionally boring by comparison, and that's the correct choice for what it is. It's @nestjs/throttler wired up as a global guard, with Redis swapped in as the storage backend instead of the default in-memory store:

ThrottlerModule.forRootAsync({
  imports: [ConfigModule],
  inject: [ConfigService],
  useFactory: (configService: ConfigService) => ({
    throttlers: [
      {
        ttl: 60000,
        limit: 10,
      },
    ],
    storage: new ThrottlerStorageRedisService(
      configService.get<string>('REDIS_URL'),
    ),
  }),
}),

// registered alongside the JWT guard as a global APP_GUARD
{
  provide: APP_GUARD,
  useClass: ThrottlerGuard,
},

10 requests per 60 seconds, globally, enforced before a request even reaches a controller. No per-route tuning yet, no per-user tiers, just a flat ceiling applied everywhere via APP_GUARD.

The reason Redis matters here isn't exotic: an in-memory counter only knows about requests hitting that one process. The moment you run more than one instance behind a load balancer, in-memory rate limiting stops being true rate limiting, each instance is independently under-counting. Swapping the storage to ThrottlerStorageRedisService gives every instance a shared, consistent view of who's made how many requests, which is the actual requirement once you're not running a single box.

LogicVisor's layered, identity-aware approach makes sense for a system where the main threat is "someone is farming free AI reviews." Titan's flat, Redis-backed throttle makes sense for a system where the main threat is "someone is hammering an endpoint," and the priority is consistency across instances over nuance per user.

3. Database-level: the layer I'm deliberately not using yet

Postgres and Supabase both support rate limiting closer to the data layer (unlogged tables with triggers, PL/pgSQL token bucket functions, TTL-based counters in Mongo). I don't use this in either project, and I don't think I should yet. It's the right call for protecting specific high-value operations (financial writes, heavy analytical queries) where you need the database itself to be the source of truth and can't tolerate a race between the app layer and the DB. LogicVisor's canonical-hash cache lookup is adjacent to this idea (checking the data layer before doing expensive work), but it's a caching pattern, not a rate limiter, and I'd rather keep that distinction honest than dress it up as something it isn't.

What actually mattered, in order

Client-side limiting is UX. Treat it as UX. It's not a line item in your security review.
Server-side is where the real decision lives, and the right shape of it depends on what you're protecting: identity-based abuse (LogicVisor) versus flat request volume across a distributed system (Titan). They are not the same problem and don't need the same solution.
The most expensive resource should be the last thing your code touches. LogicVisor checks a cache, then a rate limit, then a throttle delay, before it ever calls an LLM. Ordering is a rate limiting decision too, even when it's not framed as one.
Database-level limiting is a tool for a specific job, not a default. I'm not reaching for it until there's an operation that actually needs that guarantee.

Cover photo by HsinKai Tai on Unsplash

How I Eliminated 19,494 Duplicate MongoDB Records (And What I'd Do Differently Today)

Sat, 19 Sep 2026 08:33:15 +0000

At a previous organization, I was handed ownership of a job platform serving 1,000+ users worldwide. Jobs on the platform could be scoped to a single city, a state, a whole country, or "available worldwide." Simple enough. At least, you'd think so.

When I joined, this was early in my career and I was a frontend dev. I was interested in backend work but hadn't done any real backend engineering yet — I wasn't battle-tested. Somewhere along the line, I got handed the whole platform. Frontend, backend, all of it. Bracing moment. I forced myself to slow down, actually understand the system end to end, and build a real plan before touching anything.

After talking with product management, I pulled up the backend task backlog. I learn best by doing, so I started at the bottom — smallest bugs first, working my way up. It went well for a while. Then I hit a ticket tied to core business logic.

The Wake-Up Call

Some endpoints were clocking API response times north of 7,000ms. I genuinely thought it was a typo. Then I remembered running into something similar on the frontend side before — so I patched the UX so users never felt the full weight of the delay. But the actual problem was still sitting there, untouched.

Before touching any code, I made a checklist. We were running MongoDB with NestJS:

What schemas was the endpoint touching?
What optimization had already been applied to those schemas?
How was the endpoint fetching data?
How many records did each related schema currently hold?

Checklist in hand, I went into the codebase.

My innocent dev brain was not prepared for the chaos I found.

What Was Actually Happening

Here's the setup: assume an empty jobs collection. A recruiter creates a job listing scoped to New York City. The backend takes the payload and creates one record with location set to New York City. Straightforward.

Now assume another empty collection, and this time the job is scoped to the United States instead of a single city. What do you think happens?

If you guessed "the same thing — one record, location set to United States" — I have bad news for you.

What the backend actually did: it looked at the payload, recognized "United States" as a country rather than a city, looked up every known city in the U.S. — and created a separate job record for each one.

Read that again.

Instead of one record, the platform generated one identical record per city, differing only in _id and location. And it got worse: a job marked "available worldwide" would spawn a record for every city on the planet.

This was a live product with real traffic. Querying that collection — even with indexes in place — was a nightmare.

For context on scale: a single job scoped to just the USA generated 19,495 records — one canonical listing plus 19,494 exact duplicates.


Database records (1 USA-wide job)	19,495
Exact duplicates among them	19,494
Query time on affected endpoints	~2.3s (worst cases exceeded 7s for worldwide-scoped jobs)

My first reaction was horrified confusion. After talking it over with management, I learned there was a reason for the original design — I won't get into it here, it's not the point of this piece. What mattered was that I had free rein to fix it if I could pull it off. Hold my Pokémon card.

The Constraints

Fixing the architecture wasn't the hard part conceptually — the hard part was fixing it under real constraints:

We still needed to resolve which cities a job was available in, for every existing feature that depended on that.
Whatever I built had to slot into the existing infrastructure without a rewrite of everything downstream.
I needed to show measurable, provable performance gains — not just "trust me, it's better."
None of this could disrupt daily active users. No maintenance windows, no downtime.

Designing the Fix

I started with the schema. The core question: how do I preserve full location fidelity while collapsing tens of thousands of records into one?

I thought back to basic compression concepts — take data, strip it to a minimal representation, and pair it with the instructions needed to reconstruct the original. That's essentially what I needed here: a compact way to represent "this job is available in X" without one document per X.

Two deliverables:

A new, efficient schema.
A migration path that collapsed the old duplicated records into the new schema.

For the new schema, I kept every existing field and replaced the flat location string with a structured locationObj:

interface LocationObj {
  country: string | string[];
  state: string | string[];
  city: string | string[];
}

The logic:

Single city → city is a string, state and country resolve to that city's parent state/country.
A specific group of cities → city becomes a string array.
An entire state → city becomes the sentinel value "all", state holds the actual state name.
An entire country → state and city both become "all".
Worldwide → country, state, and city all become "all".

So a job available across Berlin, Munich, and Hamburg looks like:

{
  title: "Marketing Manager",
  company: "TechCorp",
  locationObj: {
    country: "Germany",
    state: "all",
    city: ["Berlin", "Munich", "Hamburg"]
  }
}

And a job available anywhere in the world:

{
  title: "Software Developer",
  company: "TechCorp",
  locationObj: {
    country: "all",
    state: "all",
    city: "all"
  }
}

One document. Full location semantics preserved. No fan-out.

Migrating Without Downtime

The deduplication piece ran as a batch script against the old collection: group records that shared everything except location (same title, company, description, salary — the actual "identity" of a job), reconstruct the correct locationObj from the set of location values in that group, and write a single new document into the new collection.

Once the new collection was validated against the old one, I built new endpoints on top of the new schema, mirroring the contracts of the old ones, and prepared the frontend to point at them.

The cutover itself was a backfill with two catch-up passes, not a one-shot migration:

Run the migration script once against the live old collection.
Right before deploying the new endpoints, re-run it — this catches any jobs created while the first pass was running, since the old collection kept accepting writes the entire time.
Deploy the new endpoints and switch the frontend over.
Re-run the script once more right after — this catches anything created under the old schema during the deploy window itself, in case the rollout wasn't instantaneous.

I tested the full path — frontend, backend, and DB — on local and staging before any of this touched production, so by the time it shipped, the swap itself was just a normal deploy. From a user's perspective, they kept seeing exactly the jobs they were supposed to see, the whole way through.

Querying the New Schema

The query logic has to walk the hierarchy — country, then state, then city — because "all" at any level short-circuits the rest:

const query = {
  $or: [
    { "locationObj.country": "all" },
    { "locationObj.country": userCountry, "locationObj.state": "all" },
    { "locationObj.country": userCountry, "locationObj.state": userState, "locationObj.city": "all" },
    { "locationObj.country": userCountry, "locationObj.city": userCity },
    { "locationObj.country": userCountry, "locationObj.city": { $in: [userCity] } }
  ]
};

Combined with a compound index on locationObj.country, locationObj.state, and locationObj.city, this scales linearly with the number of actual jobs — not the number of cities they happen to be available in.

The Results

Metric	Before	After	Change
Database records (1 USA-wide job)	19,495	1	99.99% fewer
Query time	~2.3s	~0.12s	~95% faster
Storage footprint	847 MB	0.043 MB	99.99% smaller
Index size	156 MB	0.008 MB	99.99% smaller
Memory usage	~4.2 GB	~0.21 MB	99.99% less RAM
Exact duplicate documents	19,494	0	Eliminated

Management was, unsurprisingly, impressed.

How I'd Do It Now

Looking back at this with a few more years under my belt, the schema still holds up. What I'd change is the migration and verification process — at the time, "backfill, then re-run twice around the cutover" felt rigorous. It worked. But it also relied on timing more than I'd want to admit if I were reviewing this today.

The more rigid version of this — the one I'd actually reach for now — is the expand-contract pattern:

Expand: Deploy dual-writes behind a feature flag — every job creation or update writes to both the old and new schema at the same time.
Sync: Use MongoDB Change Streams to mirror any old-schema writes into the new schema in real time during the transition, instead of relying on timed reruns of a migration script. This closes the race-condition window entirely — there's no gap to miss, because nothing is waiting for a scheduled pass.
Contract: Once reads are fully on the new schema and stable, kill the old write path.

I'd also make the migration script itself idempotent and checkpointed — track the last processed _id so a crash mid-run means it resumes instead of reprocessing or silently skipping documents. Mine was a script I babysat through completion; a checkpointed version is one I could walk away from.

On verification — "compare the counts" was good enough to convince management, but it wasn't rigorous. I'd now do a real reconciliation pass: diff the reconstructed locationObj against the actual set of cities in the old schema, per job, and flag any mismatch automatically instead of trusting a spot check. And before flipping reads over, I'd run a dark-launch period — querying both schemas in production and logging discrepancies without serving the shadow result to users — so any gap shows up before it's user-facing, not after.

One more thing I'd change that isn't about the migration at all: I'd want this caught before it ever shipped. The root failure was that nothing stopped a single job-creation event from silently producing thousands of documents. A write-layer guardrail — "creating one job should never result in more than one document, alert if it does" — plus a five-line regression test asserting exactly that, would have caught this in review instead of in production months later.

Lessons Learned

Embedding vs. referencing isn't the real question — normalization is. The original design wasn't wrong because it embedded or referenced incorrectly; it was wrong because it materialized every possible resolution of a hierarchical value instead of storing the hierarchy itself. The general MongoDB advice of "embed for read-heavy, reference for write-heavy" didn't apply here — the fix was closer to a compression problem than a modeling problem.

Sentinel values are underrated. Using "all" as a stand-in for "every value at this level" let one field type (string | string[]) represent five distinct scopes without a single boolean flag or separate scope enum.

Get the actual scale of the problem before designing the fix. I didn't just need "make it faster" — I needed a number to prove it. Knowing it was 19,495 records for one job is what made the case to management, and it's what makes this story worth telling.

Fields, numbers, and code above are reconstructed from a project where I owned the full backend under an NDA — details are generalized where necessary, but the architecture, the failure mode, and the metrics are real.

Cover photo by Mika Baumeister on Unsplash

The Silent Database Killer: Understanding and Fixing the N+1 Query Problem

Thu, 17 Sep 2026 08:54:44 +0000

The N+1 query problem is a performance bottleneck that occurs when an application makes database queries inside a loop. Instead of fetching all the required data in a single, efficient query, the system executes one initial query to fetch a list of parent records, followed by "N" separate queries to fetch related child records for each item in that list. This creates an exponential number of database requests as your data grows.

The Grocery Store Analogy

Imagine you are baking a cake and need five different ingredients from your local grocery store. Instead of writing all five items on a single list and making one trip, you drive to the store, buy flour, and drive home. Then, you realize you need sugar, so you drive back to the store, buy sugar, and drive home. You repeat this entire round-trip for every single ingredient. This is highly inefficient, wasting time, fuel, and energy. In software, each trip to the database is like one of those painful, redundant car rides.

Why It Matters in the Tech Industry

On a daily basis, software engineers actively fight N+1 queries to prevent application servers and databases from crashing under heavy traffic. Each extra query introduces network latency and forces the database to parse, plan, and execute a new command. This wastes valuable connection pools, spikes CPU usage, and causes page load times to balloon. Fixing this issue is often the difference between a snappy, responsive web app and one that times out during peak business hours.

Spotting and Fixing the Problem in Node.js

Let's look at how this happens in a Node.js and Express application querying a MySQL database, and how to fix it.

// THE PROBLEM: The N+1 Query Way
app.get('/users-with-posts-bad', async (req, res) => {
  // 1. First query to get all users (The "1" in N+1)
  const [users] = await db.query('SELECT id, username FROM users');

  const usersWithPosts = [];

  // 2. Loop through each user and query their posts (The "N" queries)
  for (const user of users) {
    const [posts] = await db.query('SELECT id, title FROM posts WHERE user_id = ?', [user.id]);
    usersWithPosts.push({
      ...user,
      posts
    });
  }

  res.json(usersWithPosts);
});

// THE SOLUTION: The Efficient Way (Using a SQL JOIN)
app.get('/users-with-posts-good', async (req, res) => {
  // A single query fetches all users and their posts at once
  const query = `
    SELECT u.id AS userId, u.username, p.id AS postId, p.title
    FROM users u
    LEFT JOIN posts p ON u.id = p.user_id
  `;
  const [rows] = await db.query(query);

  // Format the flat SQL result into structured JSON
  const usersMap = {};
  for (const row of rows) {
    if (!usersMap[row.userId]) {
      usersMap[row.userId] = {
        id: row.userId,
        username: row.username,
        posts: []
      };
    }
    if (row.postId) {
      usersMap[row.userId].posts.push({
        id: row.postId,
        title: row.title
      });
    }
  }

  res.json(Object.values(usersMap));
});

The Takeaway

Database interactions are highly expensive operations. Always aim to retrieve data in bulk sets rather than looping over individual records. By shifting your mindset from iterative programming to relational database operations, you can transform a slow, unstable backend into a highly optimized, production-ready system that scales effortlessly.

Originally published on my blog. You can read the alternative breakdown here.

Your Reader Cannot Be Stricter Than Your Schema

Thu, 17 Sep 2026 08:20:48 +0000

SingleOrDefault looks like a small query choice. In practice, it is an assertion about your data model: for this predicate, zero or one row may exist.

That assertion is valuable when storage enforces the same rule. When the database permits duplicates, however, the reader is relying on an invariant that does not exist. One unexpected row can turn an ordinary lookup into an exception before validation, authorization, or other normal handling begins.

I recently reviewed a committed recovery for exactly that class of mismatch. The useful lesson was not the specific lookup. It was how to contain the exception without pretending the data had become correct.

A query can encode a hidden contract

Consider a generalized account table:

Rows permitted by storage: 0..many per normalized key
Rows expected by reader:    0..1 per normalized key

The read path might be perfectly reasonable in isolation:

var record = await records.SingleOrDefaultAsync(x => x.LookupKey == key);

But the method name does not create uniqueness. It only checks cardinality at read time. If two rows match, the query throws.

This is why I now treat Single and SingleOrDefault as executable invariant assertions. During review, I ask: what proves the invariant? A unique index? A constrained aggregate? A transactionally protected write path? Or only an earlier existence check that can race?

Containment is not repair

When duplicate data already exists, immediately adding a unique index may be unsafe. Migration can fail on the existing rows. Previously accepted imports or registrations may begin failing without a useful remediation path.

At the same time, leaving a critical read path to throw can make a narrow data defect disproportionately disruptive.

A staged response separates two goals:

Containment: provide a deterministic, non-crashing resolution path while preserving security and making the defect visible.
Correction: repair the existing data and enforce the intended rule for future writes.

The distinction matters. A tolerant reader is a recovery mechanism, not evidence that duplicates are acceptable.

Properties of a safe recovery path

A useful containment pattern keeps the normal case boring:

try
{
    return await ReadExpectedSingleAsync(key);
}
catch (InvalidOperationException)
{
    var matches = await ReadBoundedCandidatesAsync(key);
    if (matches.Count < 2)
        throw;

    ReportCollision(matches.Select(x => x.OpaqueId));
    return ChooseDeterministically(matches);
}

This sketch is intentionally incomplete, but it highlights several important properties.

Keep the fast path unchanged

Most lookups are not ambiguous. Do not make every request pay for a corruption-recovery query. Invoke the fallback only after the expected lookup demonstrates the precise failure you are prepared to handle.

Prove ambiguity before handling it

InvalidOperationException can come from many places. Catching it and returning “not found” would turn an unrelated store failure into a silent wrong answer. Re-query narrowly; if there are not actually multiple matches, rethrow the original fault.

Choose deterministically

Never accept database enumeration order as a resolution policy. Define explicit ranking from legitimate domain facts, then add a stable final tie-break. The same data must produce the same candidate on every attempt.

Determinism does not make the selected record correct. It prevents the recovery itself from becoming random.

Preserve downstream verification

Resolving one candidate must not grant access, approve a transaction, or bypass an eligibility rule. Credential checks, authorization, state validation, and other security decisions still run after resolution. Containment should make the lookup deterministic, not weaken the boundaries that follow it. A legitimate candidate may still lose the ranking, so some affected operations can remain unavailable until the data is repaired.

Bound and observe the fallback

A pathological key should not cause an unbounded scan during a critical request. Cap diagnostic reads, raise a high-signal operational event, and log only what responders need. Opaque record identifiers may support repair; the sensitive lookup value often does not belong in logs.

The stronger end state lives in storage

After containment, the work is not finished. The durable sequence is usually:

inventory collisions
-> decide merge or deletion policy
-> repair existing rows
-> reject or reconcile conflicting writes
-> add the storage constraint
-> remove or narrow temporary recovery code

For relational data, that often means a unique index aligned with the normalization and null semantics used by the application. The write path should return a useful domain result when the constraint rejects a conflict. A pre-insert lookup can improve the message, but the database remains the concurrency-safe authority.

There is a real trade-off. Immediate enforcement gives the cleanest future guarantee but can break deployment against historical data. Temporary tolerance contains the exception but adds policy, cannot guarantee later verification will succeed, and can become permanent by neglect. Treat it as migration work with an owner, telemetry, and an exit condition.

Test the invariant and the escape hatch

Focused tests should cover more than “duplicates do not throw.” I would pin:

the ordinary single-row path remains untouched;
selection is stable when candidates arrive in different orders;
inactive or retired records cannot outrank valid ones;
unrelated exceptions escape unchanged;
downstream verification still decides the outcome;
diagnostics contain repairable identifiers but no sensitive key;
the candidate read is bounded;
the future uniqueness constraint rejects a conflicting concurrent write.

The deeper lesson is architectural: reads, writes, and storage must agree on cardinality. If a reader says “exactly one,” make that promise explicit at the boundary able to enforce it. Until then, the code is not reading an invariant. It is discovering, at runtime, whether the hope was true.

Top Vector Databases for AI Agents in 2026: Qdrant vs Pinecone vs Weaviate vs PgVector vs Milvus

Thu, 17 Sep 2026 07:46:47 +0000

Top Vector Databases for AI Agents in 2026: Qdrant vs Pinecone vs Weaviate vs PgVector vs Milvus

Persistent memory is the foundation that turns a stateless LLM into a continuously improving, autonomous agent.

In 2026, selecting a vector database is no longer just about raw Approximate Nearest Neighbor (ANN) speed. For AI agents, the critical requirements have shifted to:

Payload & Metadata Filtering: Can you filter by tenant_id, user_id, and timestamp during vector graph traversal without sacrificing recall?
Hybrid Search (BM25 + Dense Vectors + Sparse SPLADE): Combining exact keyword matching (for code symbols and error codes) with semantic understanding.
Multi-Tenancy & Memory Namespacing: Safely isolating memory blocks across thousands of users and sessions.
Billion-Scale Quantization (Product Quantization & Scalar Quantization): Slashing RAM costs by 75–90% in production.

This guide provides a comprehensive architectural comparison of the top 5 vector databases for AI agents in 2026.

Head-to-Head Comparison Matrix

Feature / Metric	Qdrant	Pinecone (Serverless)	Weaviate	PgVector (PostgreSQL)	Milvus
Primary Architecture	Rust-native, disk-backed	Fully managed serverless	Go-native, modular RAG	PostgreSQL extension	Distributed cloud-native
Open Source	Yes (Apache 2.0)	Proprietary SaaS	Yes (BSD-3)	Yes (Open Source)	Yes (Apache 2.0)
Payload Filtering	Exceptional (HNSW custom payload indexing)	Good (Metadata filtering)	Strong (Inverted index + HNSW)	SQL WHERE clause	Strong (Partition keys)
Hybrid Search	Native (Dense + Sparse vectors)	Native hybrid	Native BM25 + Vector	SQL text search + pgvector	Native multi-vector
Quantization	Scalar & Product Quantization (Binary)	Automatic serverless compression	PQ, BQ, SQ	Halfvec, Binary Quantization	Scalar / Product Quantization
Best Fit	High-performance agent memory & self-hosted RAG	Zero-maintenance cloud SaaS	GraphQL & multi-modal search	Unified relational + vector apps	Ultra-large enterprise (100M+ vectors)

1. Qdrant: The Rust-Powered Standard for Agent Memory

Qdrant has emerged as the developer favorite for building agent memory systems (e.g. Mem0, LangChain, CrewAI).

Why Builders Choose Qdrant:

Rust Performance: Negligible latency overhead with predictable memory usage.
Fast Filtered Search: Indexes payload fields directly inside the HNSW graph, preventing the notorious "over-filtering" recall collapse.
Binary Quantization: Compresses embeddings by up to 32x, enabling in-memory vector search over millions of documents on standard hardware.

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, Filter, FieldCondition, MatchValue

client = QdrantClient(url="http://localhost:6333")

# Create multi-tenant collection for Agent Memory
client.create_collection(
    collection_name="agent_memories",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)

# Search strictly within user namespace
search_results = client.search(
    collection_name="agent_memories",
    query_vector=[0.05] * 1536,
    query_filter=Filter(
        must=[
            FieldCondition(key="user_id", match=MatchValue(value="user_12345")),
            FieldCondition(key="memory_type", match=MatchValue(value="preference"))
        ]
    ),
    limit=5
)

2. Pinecone Serverless: The Zero-DevOps Standard

If your team does not want to manage clusters, backups, or index sharding, Pinecone Serverless separates storage (S3/GCS) from compute (stateless query workers), delivering cost efficiency at variable agent traffic loads.

3. PgVector: Unified Relational + Vector Storage

For teams already running PostgreSQL, pgvector and pgvectorscale eliminate the complexity of running a secondary vector database. You can join relational customer data directly with vector embeddings in a single ACID transaction.

Architectural Recommendation in 2026

Choose Qdrant if you want top-tier filtered search, self-hosting flexibility, and efficient binary quantization for agent memory layers.
Choose Pinecone if you need fully managed serverless infrastructure with zero operational overhead.
Choose PgVector if your application is tightly coupled to relational PostgreSQL data and you want ACID guarantees.
Choose Milvus if your dataset exceeds 100M+ vectors across distributed Kubernetes clusters.

Compare all vector databases, benchmarks, and memory layers at AgDex.ai.

Stop Recreating Database Connections: An Illustrated Guide to Connection Pooling

Fri, 11 Sep 2026 02:43:51 +0000

If you have ever built an application that talks to a database, you have used a database connection. A database is simply a structured storage system for digital information, and a connection is the secure, active pipeline established between your application's code and that database.

However, constantly opening and closing these connections for every single user request is incredibly slow and resource-heavy. To solve this, developers use a technique called connection pooling. Connection pooling is a performance optimization method where a group (or "pool") of database connections is kept open and idle, waiting to be reused by whoever needs them, rather than being destroyed and recreated on demand.

The Analogy: The Pizza Delivery Fleet

To understand why connection pooling is so vital, imagine you run a busy, high-volume pizza restaurant. Every time an order comes in, you need to deliver it to a customer.

Imagine if, for every single delivery, you had to hire a brand-new driver, run a background check, sign a legal employment contract, set up their vehicle registration, and purchase a GPS. Once that single delivery was complete, you immediately fired them, canceled their insurance, and sold the car. It sounds absurd, right? The administrative overhead of hiring and firing would take three times longer than the actual delivery, and your restaurant would go bankrupt in a week.

Instead, you hire a permanent, dedicated fleet of five delivery drivers. They sit in the break room, fully certified and ready. When a pizza is boxed, the first available driver grabs the box, drives to the house, delivers it, and returns to wait in the break room for the next order.

In this analogy, the drivers are the "connection pool," the deliveries are your database queries (requests for information), and the hiring process is the network handshake required to open a new database connection.

Why It Matters in Daily Software Engineering

In real-world software, opening a database connection is highly expensive in terms of computing power. The application must perform a security handshake, authenticate its username and password, allocate memory on the database server, and establish a network route. This process can easily take 50 to 100 milliseconds.

If your website experiences a surge in traffic—say, 1,000 users clicking a button at the exact same moment—your server would try to open 1,000 separate connections simultaneously. The database server would quickly run out of memory, slow down to a crawl, and eventually crash, displaying a "database connection error" to your users.

Connection pooling prevents this complete system collapse. It sets a strict limit on the maximum number of connections allowed (e.g., capping it at 20). If 1,000 users arrive, they politely share those 20 open connections. Because each connection is already established, queries execute in 1 to 2 milliseconds. Once a query is done, the connection is instantly recycled for the next user in line, protecting your database from crashing.

A Simple Code Example in Node.js

Below is a simple JavaScript example using a popular PostgreSQL database driver. It demonstrates how we initialize a pool and query the database without manually opening and closing connections.

const { Pool } = require('pg');

// 1. Create a pool with a maximum capacity of 10 connections
const pool = new Pool({
  user: 'db_user',
  host: 'database.server.com',
  database: 'mydb',
  password: 'securepassword',
  port: 5432,
  max: 10, // Maximum connections in the pool
});

async function fetchUserData(userId) {
  // 2. Grab an already-open connection from the pool
  const client = await pool.connect();
  
  try {
    // 3. Execute the database query
    const result = await client.query('SELECT * FROM users WHERE id = $1', [userId]);
    return result.rows[0];
  } catch (err) {
    console.error('Database query error:', err);
  } finally {
    // 4. Crucial: Release the connection back to the pool
    // This does NOT close the connection; it just makes it available for others!
    client.release();
  }
}

The Takeaway

Connection pooling turns a heavy, repetitive administrative chore into a fast, shared utility. By keeping a smart, limited buffer of active connections ready to work, software systems can handle tens of thousands of requests smoothly, ensuring high-speed page loads while protecting underlying servers from sudden traffic spikes.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Speed Up Your Queries: A Plain-English Guide to Database Indexing

Fri, 11 Sep 2026 02:43:19 +0000

What is Database Indexing?

Database indexing is a technique used by software systems to speed up the retrieval of data from a database table. By creating a separate, organized pointer structure, the database can locate specific records without having to search through every single row of data. Think of it as a specialized shortcut map for your digital filing cabinet.

The Library Card Catalog Analogy

Imagine walking into a massive metropolitan library containing 100,000 books. If you want to find a book titled "The Midnight Mystery" and there is no catalog or organization system, you would have to start at the first shelf in the corner and look at every single book cover one by one until you find it. This slow, painful process is what a database does during a "full table scan."

Now, imagine the library has an alphabetical card catalog. You walk straight to the drawer marked "M," find the index card for "The Midnight Mystery," and read the card. It tells you exactly which aisle, shelf, and position the book is on. You walk directly there and grab it in seconds. In this scenario, the card catalog is the index, the books are the rows of data in your table, and the physical location is the memory address of the database record.

Why It Matters Daily in Tech

In the tech industry, engineers use database indexing to prevent application lag and catastrophic server crashes. Without indexes, as a company's database grows from thousands of rows to millions, simple daily actions like logging in, searching for an item, or pulling up a user profile would take seconds or even minutes instead of milliseconds.

Slow database queries hog the system's central processing unit (CPU) and memory, causing other requests to back up and eventually freeze the entire application. By adding appropriate indexes, developers keep application programming interfaces (APIs) lightning-fast, reduce cloud hosting bills (because servers do not have to work as hard), and ensure a smooth user experience. However, engineers must find a balance: every index takes up storage space and slows down write operations (like adding or updating data), because the database has to update both the main table and the index every single time a change is made.

How It Works in Practice

Here is a simple example in SQL (Structured Query Language). Imagine a system trying to look up a user by their email address.

-- Without an index, the database must scan every row in the users table
SELECT * FROM users WHERE email = 'developer@example.com';

-- To speed this query up, we create an index on the email column
CREATE INDEX idx_users_email ON users(email);

-- Now, the database uses the index to jump directly to the correct user
SELECT * FROM users WHERE email = 'developer@example.com';

The Takeaway

Database indexing is the ultimate balancing act of software performance optimization. It turns grueling, resource-heavy data searches into near-instant lookups, keeping applications snappy as they scale up to handle millions of users. Just remember that indexes are not free; treat them like directory signs in a physical store—use them where visitors frequently get lost, but do not clutter every wall with them, or you will slow down the inventory restocking process.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Alembic in Practice: Structure, Revisions, and the Traps SQLite Sets

Thu, 10 Sep 2026 03:50:15 +0000

What those hex filenames mean, why migrations silently applied to the wrong database for two months, and how to widen a CHECK constraint without quietly breaking it.

Alembic is the migration tool for SQLAlchemy. Most tutorials get you as far as
alembic revision --autogenerate and stop, which leaves you unprepared for the
two things that actually go wrong: SQLite's refusal to alter constraints, and
configuration that points migrations at a different database than your
application uses.

This post covers both, using real migrations from
FinOps Sentinel, an AWS cost
agent with four migrations, one of which took three attempts to get right.

The layout

alembic.ini                     # config; script location, logging
alembic/
├── env.py                      # runs on every alembic command
├── script.py.mako              # template for new revision files
└── versions/
    ├── c447645246ac_initial_schema.py
    ├── 08872b80af2e_notifications_and_remediations.py
    ├── a1c9f4d27b13_add_rds_and_s3_resource_types.py
    └── b7e2f81c4a90_add_spend_snapshots.py

Plus one table Alembic creates in your database:

CREATE TABLE alembic_version (version_num VARCHAR(32) NOT NULL);

One row. One value. That is the entirety of Alembic's state: which database is
at which revision. Everything else is derived.

Why the filenames look like keyboard mashing

This is the most common first question, and the answer explains the design.

c447645246ac_initial_schema.py
b7e2f81c4a90_add_spend_snapshots.py

The hex prefix is the revision ID: a 12-character slice of a UUID, generated
by alembic revision. The filename is <revision_id>_<your message slugified>.py.

Inside each file:

revision: str = "b7e2f81c4a90"
down_revision: Union[str, None] = "a1c9f4d27b13"

Those two fields form a linked list:

c447645246ac  initial_schema                  (down_revision = None)  ← root
      ↓
08872b80af2e  notifications_and_remediations
      ↓
a1c9f4d27b13  add_rds_and_s3_resource_types
      ↓
b7e2f81c4a90  add_spend_snapshots                                     ← head

Order comes from the pointers, never from filenames. Look at the project's
list again: b7e2f81c4a90 sorts alphabetically before c447645246ac, yet
c447645246ac is the first migration and b7e2f81c4a90 is the newest. If
Alembic sorted by filename, everything would run backwards.

Why not 0001, 0002, 0003?

Because sequential numbers collide the moment two people work in parallel.

main
 ├── feature/billing   → 0004_add_invoices.py
 └── feature/auth      → 0004_add_sessions.py

Both are valid. Both are 0004. Merging produces a conflict where the numbering
lies about order, and worse, the conflict is in the filename, and git resolves it
by keeping both, and now your migration history has two fourth steps.

With hex IDs there is no collision. And if both branches set down_revision to
the same parent, Alembic detects a genuine branch:

$ alembic upgrade head
ERROR: Multiple head revisions are present

It refuses to guess, and you resolve it explicitly with alembic merge. The
scary-looking IDs are what make that detection possible.

If you want readable filenames

Add to alembic.ini:

file_template = %%(year)d%%(month).2d%%(day).2d_%%(rev)s_%%(slug)s

Giving 20260728_b7e2f81c4a90_add_spend_snapshots.py: date-sortable, revision
ID still present because Alembic needs it. Existing files keep their names; only
new ones pick it up.

The configuration trap that cost two months

This is the bug I most want to warn people about, because it produces no
error message at all.

The default env.py reads the database URL from alembic.ini. FinOps Sentinel
needed it configurable, so early on it did the obvious thing:

# alembic/env.py: the version with the bug
def get_url():
    db_path = os.getenv("SENTINEL_DB_PATH", ".sentinel.db")
    return f"sqlite:///{db_path}"

Reasonable. Except the application reads the same setting through
pydantic-settings:

class Settings(BaseSettings):
    sentinel_db_path: str = ".sentinel.db"
    model_config = SettingsConfigDict(env_file=".env")   # ← reads .env too

os.getenv does not read .env. pydantic-settings does.

The documented setup puts SENTINEL_DB_PATH=data/sentinel.db in .env, not in
the shell. So:

Component	Reads from	Resolves to
The app	env var → `.env` → default	`data/sentinel.db`
Alembic	env var → default	`.sentinel.db`

Every migration ever run landed on .sentinel.db. The application used
data/sentinel.db. Two files, silently, for two months.

Why nothing broke for so long

The application also calls Base.metadata.create_all() in tests, and the first
three migrations only created tables that SQLAlchemy would have created anyway.
The schemas happened to agree.

Then came a migration whose table nothing else creates: spend_snapshots. The
next scan died:

alembic upgrade head
INFO  [alembic.runtime.migration] Running upgrade a1c9f4d27b13 -> b7e2f81c4a90

sentinel scan
OperationalError: (sqlite3.OperationalError) no such table: spend_snapshots

Alembic reported success. The app said the table didn't exist. Both were telling
the truth about different files.

The fix: one owner for the URL

# config.py: the single source of truth
def database_url() -> str:
    """The one place the findings database URL is built.

    Both the application (bootstrap.get_repository) and Alembic (alembic/env.py)
    call this. They used to build it independently, and drifted.
    """
    return f"sqlite:///{settings.sentinel_db_path}"

# alembic/env.py
from finops_sentinel.config import database_url

def get_url():
    return database_url()

# bootstrap.py
def get_repository() -> FindingsRepository:
    return SqlAlchemyRepository(db_url=database_url())

And alembic.ini is deliberately emptied:

# Deliberately empty. env.py overrides this with the application's own
# Settings.sentinel_db_path. A URL here would be dead config that reads as
# authoritative, which is exactly how this drifted before.
sqlalchemy.url =

Takeaway: if your app and your migrations resolve the database path
separately, they will eventually disagree, and nothing will tell you. Give the
URL exactly one owner and have both call it.

The project now pins this with tests:

def test_the_app_and_alembic_build_the_url_the_same_way():
    assert get_repository().db_url == database_url()

def test_alembic_ini_does_not_hardcode_a_database():
    parser = configparser.ConfigParser()
    parser.read(REPO_ROOT / "alembic.ini")
    assert parser.get("alembic", "sqlalchemy.url", fallback="").strip() == ""

def test_alembic_env_does_not_read_the_environment_itself():
    env_py = (REPO_ROOT / "alembic" / "env.py").read_text()
    assert "database_url" in env_py
    assert "os.getenv(" not in env_py

Autogenerate, and where to stop trusting it

alembic revision --autogenerate -m "add spend snapshots"

Alembic compares your SQLAlchemy models against the live database and writes the
diff. It needs target_metadata wired up in env.py:

from finops_sentinel.adapters.persistence.sqlalchemy_repo import Base
target_metadata = Base.metadata

It detects new/dropped tables and columns, nullability changes, most index
and unique-constraint changes, and some type changes.

It misses table and column renames (it sees a drop plus an add, and will
happily destroy your data), CHECK constraint changes, server defaults in many
cases, and anything requiring data transformation.

The generated file is a draft. The comment Alembic itself writes says so:

def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###

Always read it. For anything beyond adding a column, expect to hand-write.

Trap: SQLite cannot ALTER a constraint

Adding two values to an enum required widening a CHECK constraint:

class ResourceType(StrEnum):
    EBS_VOLUME = "ebs_volume"
    ELASTIC_IP = "elastic_ip"
    EC2_INSTANCE = "ec2_instance"
    EBS_SNAPSHOT = "ebs_snapshot"
    RDS_INSTANCE = "rds_instance"    # new
    S3_BUCKET = "s3_bucket"          # new

On Postgres: ALTER TABLE ... DROP CONSTRAINT ... ADD CONSTRAINT .... Done.

On SQLite: there is no ALTER for constraints. The only route is create a new
table, copy the data, drop the old, rename the new. Alembic wraps this in
batch_alter_table:

with op.batch_alter_table("resources", recreate="always") as batch_op:
    ...

Attempt 1: silently kept the old constraint

The first version passed table_args and ran without error:

with op.batch_alter_table(
    "resources",
    recreate="always",
    table_args=(
        sa.CheckConstraint(_resource_type_check(NEW_TYPES), name="check_resource_type"),
    ),
):
    pass

alembic upgrade head printed success. Inserting an rds_instance row still
failed.

Why: table_args adds to a reflected table definition rather than
replacing it. Alembic reflected the live table, including the old four-value
CHECK, then appended the new six-value one. The rebuilt table had both, and
the old one still rejected every new row.

Only dumping the schema revealed it:

sqlite3 data/sentinel.db ".schema resources"
# CONSTRAINT check_resource_type CHECK (resource_type IN ('ebs_volume', ...))   ← old
# CONSTRAINT check_resource_type CHECK (resource_type IN ('ebs_volume', ..., 's3_bucket'))

Attempt 2: `copy_from` to suppress reflection

The fix is copy_from, an explicit table definition that tells Alembic not
to reflect:

def _columns_only_resources() -> sa.Table:
    """The `resources` table as columns and primary key, with NO constraints.

    Handed to batch_alter_table as `copy_from`, which suppresses reflection of
    the live table. That suppression is the point: `table_args` adds to a
    reflected definition rather than replacing it.
    """
    return sa.Table(
        "resources",
        sa.MetaData(),
        sa.Column("id", sa.String(), primary_key=True, nullable=False),
        sa.Column("resource_id", sa.String(), nullable=False),
        sa.Column("resource_type", sa.String(), nullable=False),
        sa.Column("resource_arn", sa.String(), nullable=False),
        sa.Column("region", sa.String(), nullable=False),
        sa.Column("current_tags", sa.String(), nullable=False),
        sa.Column("lifecycle", sa.String(), nullable=False),
        sa.Column("first_seen_at", sa.DateTime(timezone=True), nullable=False),
        sa.Column("last_seen_at", sa.DateTime(timezone=True), nullable=False),
    )

Attempt 3: the indexes were gone

copy_from suppresses index reflection too. The rebuilt table lost its
unique index on resource_id, which the upsert logic depends on for
correctness. So it gets recreated by hand:

def _rebuild_resources(resource_types: tuple[str, ...]) -> None:
    with op.batch_alter_table(
        "resources",
        recreate="always",
        copy_from=_columns_only_resources(),
        table_args=(
            sa.CheckConstraint("lifecycle IN ('active', 'deleted')",
                               name="check_resource_lifecycle"),
            sa.CheckConstraint(_resource_type_check(resource_types),
                               name="check_resource_type"),
        ),
    ):
        pass  # the rebuild itself is the migration

    # copy_from also suppresses index reflection, so re-create by hand.
    bind = op.get_bind()
    existing = {index["name"] for index in sa.inspect(bind).get_indexes("resources")}
    if "ix_resources_resource_id" not in existing:
        op.create_index(op.f("ix_resources_resource_id"), "resources",
                        ["resource_id"], unique=True)

Rule for SQLite batch migrations: whatever the batch block omits is silently
absent from the rebuilt table. Every column, every constraint, every index has
to be re-declared. And "it ran without error" proves nothing. Dump the schema.

Downgrades that refuse to run

Most downgrade() functions are written on autopilot as the inverse of
upgrade(). Sometimes the honest inverse is refusing:

def downgrade() -> None:
    bind = op.get_bind()
    stranded = bind.execute(
        sa.text("SELECT COUNT(*) FROM resources "
                "WHERE resource_type IN ('rds_instance', 's3_bucket')")
    ).scalar_one()
    if stranded:
        raise RuntimeError(
            f"Cannot downgrade: {stranded} resource(s) use rds_instance or s3_bucket, "
            "which the previous CHECK constraint forbids. Delete those resources and "
            "the findings referencing them first if this downgrade is intended."
        )
    _rebuild_resources(OLD_RESOURCE_TYPES)

Narrowing the constraint with those rows present means deleting them, which
also orphans every finding referencing them. A downgrade that silently destroys
data to succeed is worse than one that stops and explains.

The docstring says so up front, so nobody is surprised at 2am:

downgrade() restores the four-value constraint and will FAIL if any
rds_instance or s3_bucket row exists by then, deliberately.

A simple migration, for contrast

Not every migration is a fight. Adding a table is exactly what you'd hope:

def upgrade() -> None:
    op.create_table(
        "spend_snapshots",
        sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True, nullable=False),
        sa.Column("snapshot_date", sa.Date(), nullable=False),
        # Stored as a string, matching SafeNumeric: SQLite has no Decimal, and
        # a float column would quietly round money.
        sa.Column("total_estimated_monthly_usd", sa.String(), nullable=False),
        sa.Column("open_findings", sa.Integer(), nullable=False),
        sa.Column("active_resources", sa.Integer(), nullable=False),
        sa.Column("captured_at", sa.DateTime(timezone=True), nullable=False),
    )
    op.create_index(
        op.f("ix_spend_snapshots_snapshot_date"),
        "spend_snapshots", ["snapshot_date"], unique=True,
    )

Note the docstring convention this project uses: the why, not the what:

snapshot_date is UNIQUE on purpose. The repository upserts by date, so two
scans on the same day collapse to one row. Otherwise a day that happened to
be scanned five times would weigh five times as much in the trailing mean, and
the z-score would measure scan cadence rather than spend.

Six months later, that paragraph is worth more than the code.

Keeping constraints from drifting

CHECK constraints duplicating enum values is a drift hazard. This project
generates them:

def _in_clause(column: str, values: type[StrEnum]) -> str:
    """Render `column IN ('a', 'b', ...)` straight from a StrEnum.

    Generated rather than hand-written so the constraint cannot drift from the
    enum: adding a ResourceType now updates this automatically, and only the
    Alembic migration has to be written by hand.
    """
    members = ", ".join(f"'{member.value}'" for member in values)
    return f"{column} IN ({members})"


class ResourceModel(Base):
    __tablename__ = "resources"
    ...
    __table_args__ = (
        CheckConstraint(_in_clause("lifecycle", ResourceLifecycle),
                        name="check_resource_lifecycle"),
        CheckConstraint(_in_clause("resource_type", ResourceType),
                        name="check_resource_type"),
    )

The model side can never drift from the enum. Only the migration is
hand-written, which is unavoidable, since a migration must describe a transition between
two states, and only one of them exists in your code.

Testing migrations

A green alembic upgrade head proves almost nothing (see attempt 1). Two things
worth doing:

1. Round-trip against a copy of a real database.

cp data/sentinel.db /tmp/mig-test.db
SENTINEL_DB_PATH=/tmp/mig-test.db alembic upgrade head
SENTINEL_DB_PATH=/tmp/mig-test.db alembic downgrade -1
SENTINEL_DB_PATH=/tmp/mig-test.db alembic upgrade head

sqlite3 /tmp/mig-test.db "select count(*) from resources; select count(*) from findings;"
# 274 / 187, same as before

Real data, real row counts, and it catches data loss a fresh database never
would.

2. Assert the error message is useful.

An un-migrated database should not produce a raw SQL traceback:

def upsert_resource(self, resource: Resource) -> None:
    try:
        self._upsert_resource(resource)
    except IntegrityError as exc:
        if "check_resource_type" not in str(exc):
            raise
        raise RuntimeError(
            f"The findings database does not allow resource type "
            f"'{resource.resource_type}'. It was created by an older "
            f"version of FinOps Sentinel. Run `alembic upgrade head` "
            f"against {self.db_url} and scan again."
        ) from exc

Failing hard is right: a half-written inventory is worse than a stopped run.
But the traceback told nobody what to do. Now it names the type, the database,
and the command.

Command reference

alembic revision -m "message"                  # empty migration, hand-written
alembic revision --autogenerate -m "message"   # diff models vs database

alembic upgrade head          # apply everything outstanding
alembic upgrade +1            # one step forward
alembic downgrade -1          # one step back
alembic downgrade base        # all the way down

alembic current               # what revision is this database at?
alembic history --verbose     # the full chain
alembic heads                 # >1 means you have a branch to merge
alembic merge -m "merge" a1b2 c3d4

alembic upgrade head --sql    # print SQL instead of executing (offline mode)

alembic current is the one to reach for when something is confusing. It tells
you where Alembic thinks the database is, and if that disagrees with the
database you expect, you have the bug from earlier in this post.

Checklist

Never edit a migration that has been applied anywhere but your laptop.
Write a new one. The old revision ID is a foreign key in every database that
ran it.
Read autogenerated migrations before committing. Especially for renames,
which autogenerate turns into drop-plus-add.
Write the "why" in the docstring. The SQL says what; only you can say why.
Test on a copy of real data. Fresh databases hide data-loss bugs.
Dump the schema after non-trivial migrations. "It ran" is not "it worked."
Give the database URL one owner that both the app and Alembic call.
Make downgrades honest, including refusing when the honest inverse is
data loss.

Migrations are the one part of a codebase where a mistake is not just wrong but
permanently wrong, replicated to every environment that ran it. Worth the
extra ten minutes.

Every example here is from github.com/boazleleina/finops-sentinel

Building a Real-Time Dashboard with FastAPI, WebSockets, and MySQL

Wed, 09 Sep 2026 08:42:41 +0000

I recently worked on a lightweight real-time dashboard built with FastAPI, WebSockets, and aiomysql.

The main goal was simple: retrieve live data from a MySQL database and push updates to the frontend in real time without relying on continuous polling.

This project provided a practical opportunity to explore asynchronous programming, WebSocket communication, and database integration using Python.

Project Overview

The application uses a FastAPI backend to communicate with a MySQL database and a WebSocket endpoint to deliver database updates to connected clients.

The overall architecture can be summarized as:

MySQL Database
      │
      ▼
   aiomysql
      │
      ▼
FastAPI Backend
      │
      ▼
  WebSocket
      │
      ▼
Frontend Dashboard

The WebSocket connection allows the server to push new data directly to the client, making the dashboard capable of displaying updates in real time.

What This Project Demonstrates

The project covers several important backend development concepts:

Setting up Python and the required dependencies
Building an asynchronous FastAPI backend
Connecting to MySQL using aiomysql
Implementing a WebSocket endpoint
Retrieving database records asynchronously
Sending live data to connected clients
Running the application with Uvicorn
Structuring a simple real-time backend application

Why Use WebSockets?

Traditional REST APIs generally follow a request/response model.

For example:

Client → Request → Server
Client ← Response ← Server

If the frontend needs updated information continuously, it may have to repeatedly send requests to the server.

This is commonly known as polling:

Client → Request
Server → Response

Client → Request
Server → Response

Client → Request
Server → Response

With WebSockets, the client establishes a persistent connection with the server:

Client ⇄ WebSocket Connection ⇄ Server

The server can then push new information to the client whenever it becomes available.

This makes WebSockets particularly useful for applications where data needs to be delivered quickly and continuously.

Potential Use Cases

This architecture can be useful for:

Real-time dashboards
Monitoring systems
Live administration panels
Student management systems
Record management applications
System status monitoring
Live data visualization
Notification systems

Documentation and Source Code

I created a complete guide covering the setup and implementation process, including:

Python installation
Required package installation
Project configuration
Server startup
Complete main.py implementation
FastAPI and WebSocket configuration

The GitHub repository also includes the documentation and source code for reference.

GitHub Repository

FastAPI WebSocket Real-Time Dashboard - GitHub Repository

The repository contains:

Complete Guide (PDF)
Source Code (main.py)

Key Technologies

Technology	Purpose
Python	Backend programming
FastAPI	Asynchronous web framework
WebSockets	Real-time client-server communication
aiomysql	Asynchronous MySQL connectivity
MySQL	Database
Uvicorn	ASGI server

Final Thoughts

This project was a useful exercise in combining asynchronous programming, database connectivity, and real-time communication into a practical backend application.

FastAPI and WebSockets provide a straightforward foundation for building applications where data needs to move between the server and client with minimal delay.

I look forward to extending this project with features such as authentication, multiple connected clients, automatic database change detection, improved error handling, and a more advanced frontend dashboard.

If you are working with FastAPI or real-time applications, I'd be interested to hear about your projects and approaches.

Also Published On

This article is also available on the following platforms:

LinkedIn: Read on LinkedIn
Medium: Read on Medium
AWS: Read on AWS
DEV Community: Read on DEV Community

Building database synchronization the boring way

Tue, 08 Sep 2026 21:13:56 +0000

TL;DR: iamroot.tech uses a custom-built, asynchronous MySQL database synchronization system handling ~10 million daily changes across several distributed nodes without consensus algorithms or distributed coordinators.
Key Architecture & Trade-offs:

Primary Keys: UUIDs to allow conflict-free row generation on any node.
Change Tracking: Database triggers log INSERT, UPDATE, and DELETE operations into a simplified change log.
Replication Mechanism: Asynchronous background workers process log entries.
Conflict Avoidance: Application-level single-writer authority per dataset
Fault Tolerance: Nodes catch up automatically after offline periods by replaying the simple change log backlog.

The setup - servers will disappear

iamroot.tech runs across numerous servers in different geographical locations. They are deliberately inexpensive machines, and an important assumption behind the entire architecture is that individual servers will - at some point in time - disappear, underperform or otherwise misbehave.

The goal has never been to make every server perfectly reliable. The goal is to make each server failure completely uninteresting.

That creates an interesting problem for the databases. A server needs access to shared data, but I don't want a write on server A to depend on server B being available. In fact, I don't really want server A to care about server B at all while serving a request.

So I ended up building my own synchronization system. It currently keeps roughly 50–60 tables synchronized across numerous servers and handles around 10 million changes on a normal day.

It isn't particularly clever. In fact, it's about as simple as it gets.

And that is rather the point.

A failing server should create a backlog, not an outage

The most important principle is that database writes are local operations. If the application running on server A needs to update something, it writes to the local MySQL database and continues.

Synchronization happens afterwards.

If server B happens to be slow, offline or has been gone for three days, that doesn't change what happens on A. It simply means B has more catching up to do when it eventually returns.

In practice, this gives iamroot.tech a distributed data architecture without trying to turn MySQL itself into a distributed database. Each server owns its local database and can operate independently, while relevant data is eventually synchronized between them.

Temporary differences between nodes are therefore expected. Eventual consistency isn't something the architecture reluctantly tolerates; it is part of the design.

That trade-off works because the application itself understands how the data is used.

The implementation is almost embarrassingly simple

Tables that need to be shared use UUIDs as their primary keys. That allows records to be created independently on different servers without coordinating ID ranges or worrying about primary key collisions later.

Each server then maintains its own local change log. Synchronized tables have triggers for INSERT, UPDATE and DELETE, and whenever something changes, a small entry containing the table, record ID, operation and timestamp is written to the log.

A background service processes those entries and sends the changes to whichever nodes should receive them. There is no central replication queue and no synchronization master.

For inserts and updates, I use MySQL's INSERT ... ON DUPLICATE KEY UPDATE. If a server somehow missed the original INSERT but later receives an UPDATE, the record simply gets created anyway.

Replicated writes bypass the change-log triggers for that database connection, while normal application writes still trigger them. Otherwise A would update B, B would send the same change back to A, and my wonderfully simple synchronization system would become a wonderfully effective infinite loop.

DELETEs work through the same mechanism. The DELETE entry stays in the change log until the relevant nodes have processed it, effectively making it a temporary replication tombstone. A server can therefore disappear for days and still learn which records were deleted while it was gone.

That's basically it.

Avoiding problems instead of solving them

This is not conflict-safe multi-master replication.

If two servers independently modify the same record at exactly the wrong moment, there is no clever distributed algorithm deciding which version represents universal truth.

Instead, I largely avoid the problem at the application level.

Individual records normally have one logical writer - one server or subsystem that is effectively king for that data. The code working with those records knows about that architecture.

That is an important distinction. I could add version vectors, distributed locking, automatic conflict resolution and considerably more machinery. Those are perfectly valid solutions to problems other systems have.

But if I can structure the application so the problem doesn't normally exist, I would rather do that.

The same philosophy applies elsewhere. Synchronization is at-least-once rather than exactly-once. Duplicate work happens, but the operations are idempotent, so that is mostly harmless. The change log isn't an event store either. If a record changes ten times, I don't need to reconstruct all ten historical versions. I care about getting the current state where it needs to go.

Recovery is deliberately boring

For every table and destination node, the synchronizer remembers how far through the local change log that node has progressed.

Important data is typically synchronized every 10 – 20 seconds, while less important tables may only be processed every ten minutes. If a destination disappears, its position simply stops advancing while everybody else continues.

When it returns, there is normally nothing for me to do. The synchronizer knows where it stopped and continues from there. Once all relevant nodes have moved beyond old change-log entries, those entries can be removed.

This isn't just how I hope it works. Servers have disappeared and returned plenty of times in production.

The recovery procedure is dead simple: sit back and wait.

I like recovery procedures like that.

Why not just use MySQL replication?

MySQL already provides very capable replication, and this isn't an attempt to build a better replacement.

I wanted slightly different properties.

Not every table needs to exist everywhere. Different data can have different synchronization intervals and destinations. More importantly, replication isn't part of the availability model for an individual application server. A slow destination creates work for the synchronizer, not latency for the application serving a user.

There are obvious trade-offs. The synchronizer doesn't automatically understand referential integrity between tables, although I can control their processing order. It doesn't negotiate schemas either; keeping compatible schemas across the relevant servers is my responsibility.

Those are not necessarily features waiting to be implemented. They are boundaries I know about and can design around.

Boring as a feature

Around 10 million changes pass through this setup every day, with the ASN database alone sometimes generating several million updates during a processing run.

Could I make the synchronization system considerably smarter? Absolutely.

But I didn't set out to build a distributed database. I wanted a multitude of servers that could operate independently, share the data they needed, tolerate each other disappearing, and recover without me getting involved.

The synchronization system is simply what fell out of those requirements.

There are plenty of situations where this design would be completely inappropriate. There are also plenty where sophisticated distributed database technology solves very real problems.

But complexity has a real cost.

For iamroot.tech, I'd rather have infrastructure I can explain on a single piece of paper, predict when something breaks, and largely leave alone when a server disappears.

Extremely simple, asynchronous and a tad inefficient in the actual operations, yes - but very efficient at solving the problem, very predictable in the way it works, and - most importantly - it does not require any human intervention at any point.

That is all I need.

How I Structured My Firestore Data Model for an Anti-Proxy Attendance System

Thu, 03 Sep 2026 04:12:28 +0000

The database architecture behind students, meetings, attendance, authentication, and controlling unnecessary Firestore reads.*

How I Structured My Firestore Data Model for an Anti-Proxy Attendance System

When I began building my attendance system, database architecture wasn't really my first concern.

I had one straightforward objective:

Make the MVP work.

I needed a place to store students.

I needed a way to create meetings.

I needed to save attendance records.

So I started creating Firestore data structures based on whatever the application needed at that moment.

Initially, that approach worked.

Then the application started expanding.

Authentication was introduced.

Dynamic QR authentication was introduced.

Device verification was introduced.

Attendance history was introduced.

Offline synchronization was introduced.

And suddenly, Firestore was doing much more than simply holding attendance records.

It was turning into the foundation that the rest of the application depended on.

That's when an important realization hit me:

A schema that is perfectly fine for an MVP may not remain suitable once the application starts growing.

The First Question: What Really Needs to Live in the Database?

An attendance system sounds straightforward until you actually list everything it needs to keep track of.

For every attendance event, I may need details such as:

The student
The meeting
The attendance status
The time the attendance was recorded
Authentication information
Device information
Synchronization state
Administrative information

The difficult part wasn't simply finding a place to store all of these things.

The harder problem was determining how all of these pieces should connect with one another.

I started looking at the database through the entities that actually existed inside the system.

Student
   │
   ├── Identity
   ├── Device
   └── Attendance
          │
          ▼
        Meeting
          │
          ├── QR Session
          └── Attendance Records

The database needed to represent these relationships without making the application repeatedly retrieve the same information.

Firestore Forces a Different Way of Thinking

Firestore doesn't work like a traditional relational database.

There are no conventional SQL joins where I can simply write something like:

SELECT ...
FROM students
JOIN attendance ...
JOIN meetings ...

Instead, I need to think about a different question:

Which pieces of data does the application need to access together?

That question became one of the most important considerations while designing the schema.

In Firestore, the structure of your data directly influences how much information the application has to retrieve.

And I had already experienced the consequences of getting that wrong.

I Still Remember the 50,000 Read Problem

Earlier during development, I accidentally burned through my Firestore free-tier reads while testing and debugging the application.

That experience completely changed the way I looked at database operations.

Before that happened, my thinking was basically:

"It's just one additional read."

Afterwards, my question became:

"Does the application actually need this read?"

That shift in mindset started influencing the database architecture as the project continued to evolve.

Designing Around Actual Access Patterns

Rather than beginning with:

"Which collections should I create?"

I started approaching the problem from the opposite direction:

"What information does the application actually need to retrieve?"

For example, when an attendance session is running, the system needs enough information to validate the request and then record the attendance correctly.

So the database has to support operations such as:

Create Meeting
      ↓
Generate Attendance Session
      ↓
Validate Student
      ↓
Validate Device
      ↓
Validate QR
      ↓
Record Attendance

The database should not become the thing slowing down every stage of the process.

Keeping Attendance as a Separate Record

One of the key decisions I made was to treat attendance as its own record rather than repeatedly changing a student's profile whenever they attend a meeting.

Conceptually, the relationship looks like this:

Student
   │
   └──────────────┐
                  │
                  ▼
              Attendance
                  │
                  ▼
                Meeting

This allows attendance history to exist separately from the student's core profile.

Instead of only asking:

"Is this student present?"

the system can answer a much more useful question:

"Was this student present for this particular meeting, and when exactly was that attendance recorded?"

That distinction is important.

Attendance represents an event, rather than simply being a permanent property attached to a student.

More Data Does Not Automatically Mean a Better Database

A common trap during database design is duplicating information everywhere because it makes individual queries feel easier.

At first, that can seem convenient.

The problem appears later when those duplicated values need to stay consistent.

For example, copying the same information into several documents may simplify one specific query, but now the application has to maintain multiple versions of the same data.

So I constantly found myself balancing:

Fast reads

against

Data duplication

against

Write complexity

against

Firestore usage

There isn't one universal answer that solves all of these concerns.

The right decision depends on how the application actually reads and writes the data.

What an Attendance Request Actually Contains

At a high level, an attendance request now carries several different pieces of context.

Attendance Request
       │
       ├── Student Identity
       │
       ├── Meeting
       │
       ├── QR Validation
       │
       ├── Device Verification
       │
       └── Attendance Record

So attendance isn't really just:

student → present

It's closer to:

student
   +
meeting
   +
valid session
   +
trusted device
   +
validated request
   =
attendance

That means the database is not merely a storage layer.

It becomes part of the security architecture as well.

Thinking Carefully About Reads and Writes

After experiencing the 50,000-read issue, I started paying much more attention to the difference between reads and writes.

A system can function correctly and still perform far more database operations than it actually needs.

For example:

Page Load
   ↓
Read Student
   ↓
Read Meeting
   ↓
Read Configuration
   ↓
Read Student Again
   ↓
Read Meeting Again

Nothing is technically failing here.

But the application is doing unnecessary work.

So I started looking for ways to:

Reuse information that the application already has
Eliminate repeated reads
Cache frequently accessed information
Minimize unnecessary writes
Fetch only the data that is actually required

This eventually became part of the larger optimization work I covered in Part 3.

Firestore Security Goes Beyond Security Rules

Another lesson from the project was that database security cannot be reduced to Firestore Security Rules alone.

Rules are important.

But they represent only one layer of the overall security model.

In my attendance system, validation happens across several stages:

User Authentication
        ↓
Dynamic QR Validation
        ↓
Device Verification
        ↓
Backend Validation
        ↓
Firestore
        ↓
Attendance Record

The database should never blindly trust whatever information the frontend submits.

The backend needs to validate the request before writing sensitive attendance information.

This separation became even more important as more anti-proxy mechanisms were introduced into the system.

Offline Support Changes the Data Lifecycle

Then another challenge appeared.

Offline support.

Once the system could record attendance without immediately communicating with Firestore, the data model needed to distinguish between two different states:

Attendance created locally

and

Attendance successfully synchronized

Conceptually, the lifecycle became:

Student
   ↓
Attendance Event
   ↓
Local Storage
   ↓
Pending Sync
   ↓
Firestore
   ↓
Synchronized

That meant the lifecycle of an attendance record was no longer simply:

Create → Save

It became:

Create
  ↓
Store locally
  ↓
Wait
  ↓
Synchronize
  ↓
Confirm

That changed the way I thought about reliability as well.

What I Would Avoid Doing

One of the clearest lessons from this project is that Firestore schemas should not be designed by blindly copying relational database patterns.

Firestore has its own strengths, limitations, and way of handling data.

Instead of asking:

"How would I structure this in MySQL?"

I found it much more useful to ask:

"What are the operations my application performs most often?"

Then I could design the data model around those access patterns.

That single shift in perspective made the database design much easier to reason about.

The Bigger Lesson From the Data Model

At the beginning, the database felt like infrastructure.

Something beneath the actual application.

But as the system became more complex, I realized that the data model affects almost every major part of the application:

Performance
Cost
Security
Offline synchronization
Backend architecture
Query complexity
Scalability

A poor data model doesn't necessarily cause an obvious failure on day one.

Sometimes the problem is much quieter.

It simply makes every new feature a little harder to build.

And that can become a much bigger problem over time.

What I Would Change Today

If I were starting this system again today, I would spend considerably more time thinking about the data model before implementing features.

I'd define:

Entities
   ↓
Relationships
   ↓
Access Patterns
   ↓
Read / Write Frequency
   ↓
Caching Strategy
   ↓
Security Boundaries

before writing the first database query.

I learned this lesson after the system was already functioning.

Building first showed me what the application genuinely needed.

But if I rebuilt it from scratch, I'd bring those lessons into the design phase much earlier.

Final Thoughts

The original requirement sounded simple:

"I just need somewhere to store attendance."

Over time, that requirement evolved into something much larger:

"I need a data architecture that can support authentication, dynamic QR validation, device verification, offline synchronization, and efficient reads and writes."

Those are two completely different problems.

And probably one of the biggest lessons this project gave me is that database design isn't simply about deciding where data should be stored.

It's about defining how the entire application communicates with that data.

The database isn't simply sitting underneath the architecture.

It is part of the architecture.

What's Next?

The database was only one part of the overall system.

The next challenge was infrastructure.

My backend was running on a free-tier service, and that introduced another constraint that became difficult to ignore:

cold starts.

Which raised another question:

How do you make a slow backend feel fast from the user's perspective?

That's the problem I'll explore next.

🔗 Project

GitHub:

https://github.com/siddarthpatelkama/UBA-veltech-attendance-system

💬 Your turn

If you've built a Firestore application before, what's one database decision you made early that later saved you from dealing with a much bigger problem?

I'd love to hear about it.

Layered Authorization: Network to Database

Wed, 02 Sep 2026 21:53:08 +0000

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

The previous post covered how Auth Service signs the mesh identity token that carries a user through the mesh. Once a request arrives with a verified identity, the next question is what that identity is allowed to do.

Authentication tells you who someone is. Authorization tells you what they're allowed to do. In this architecture, authorization runs at four granularities: network, request, resource, and field. Each operates at a different enforcement point with different information available, and each catches failures the others can't. A separate layer, PostgreSQL row-level security, sits at the database and backstops all of them when application code has a bug.

Why Four Granularities

A single policy engine can't efficiently handle every authorization decision. Some decisions require only the service identity (which pod is calling). Others require the user's role. Others require knowing the specific resource being accessed and the relationship between the user and that resource. And field visibility only makes sense at query time, when you know the actual data attributes.

Different information is available at different enforcement points:

At the sidecar: mTLS identity and JWT claims. No application context. This is where network-level authorization runs (Istio AuthorizationPolicy).
At the gateway: user role and route. No specific resource. This is where request-level authorization runs (ExtAuthz).
At the application: full request context, resource attributes, user-resource relationships. This is where Cerbos makes resource-level and field-level decisions.
At the database: transaction-level identity. No application logic. This is where row-level security runs. Strictly it's a data-access control rather than authorization, and it's the final enforcement point that backstops the others.

Trying to push all decisions to one point either means that point needs all context (which defeats the separation) or that some decisions can't be made at all.

Network (Sidecar): Istio AuthorizationPolicy

The first question: can this service even talk to that service?

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-ms1-to-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/zt-apps/sa/ms1-profile-aggregator-sa
      to:
        - operation:
            ports: ["8000"]
      when:
        - key: request.auth.claims[aud]
          values: ["ms2-employee-details"]
        - key: request.auth.claims[act][sub]
          values: ["ms1-profile-aggregator"]

This policy says: only Profile Aggregator's service account (ms1-profile-aggregator-sa) can reach Employee Records, and only with a token that has the correct audience and delegation claims. Everything else is denied.

Important nuance: Istio's default behavior when no AuthorizationPolicy targets a workload is to allow all traffic. Default-deny only kicks in once at least one ALLOW policy exists for that workload. This means every service that should be protected needs an explicit policy. A newly deployed service without a matching policy is open until you write one. In this architecture, every workload in zt-apps has at least one scoped ALLOW policy, which triggers deny-by-default for unmatched requests.

What this catches: lateral movement. If Holiday Calendar is compromised, the attacker cannot reach Employee Records. The connection is refused at the sidecar level before any application code processes the request. The attacker would need both the correct service account identity (the mTLS certificate) and a validly-signed token with the right claims.

The full boundary map:

Source	Can reach	Cannot reach
Ingress Gateway	Profile Aggregator, Holiday Calendar, Office Directory, Auth Service	Employee Records, Device Inventory, Vault, Postgres
Profile Aggregator	Employee Records, Device Inventory	Cerbos, Vault, Keycloak, Postgres
Employee Records, Device Inventory	Cerbos, Postgres (own schema only)	Profile Aggregator, Holiday Calendar, Office Directory, Vault
Holiday Calendar, Office Directory	Cerbos, Postgres (own schema only)	Profile Aggregator, Employee Records, Device Inventory, Vault
Auth Service	Vault, Postgres	Employee Records, Device Inventory, Holiday Calendar, Office Directory

Every cell in this table is an explicit decision. Unmatched traffic is denied because each workload has at least one ALLOW policy targeting it.

Request (Gateway): ExtAuthz

The second question: is this user, with this role, allowed to call this route?

Auth Service (auth-service) acts as an external authorization check at the gateway. Before minting a mesh token, it checks a route-role policy map:

Route	Method	Allowed Roles
`/api/profile/*`	any	employee, manager, hr_admin, it_admin
`/api/holidays`	GET	employee, manager, hr_admin, it_admin, public_data_admin, security_auditor
`/api/holidays`	POST/PUT/DELETE	hr_admin, public_data_admin
`/api/offices/*`	any	(bypasses ExtAuthz entirely; see below)

The system has six roles total (employee, manager, hr_admin, it_admin, public_data_admin, security_auditor). The core access model for the profile/HR flow is built around the first four. public_data_admin manages office and holiday data. security_auditor has read access to non-sensitive endpoints.

Office routes are the exception: the gateway's AuthorizationPolicy exempts /api/offices and /api/offices/* from ExtAuthz entirely. Reads are public, and write authorization (public_data_admin only) is enforced downstream by Cerbos at Office Directory, not at the gateway. Everything else routes through the map above.

A public_data_admin trying to access the profile endpoint gets a 403 at the gateway. They never hit Profile Aggregator. The token is never minted. Downstream services never see the request.

This is coarse authorization. It knows the user's role and the route they're hitting, but it knows nothing about which specific resource they want to access. A manager can access the profile endpoint, but that doesn't mean they should see every employee's salary; they should only see pay data for their own direct reports. That's a different layer's job.

The key decision here is what gets denied early (saving downstream resources and reducing attack surface) versus what gets deferred to finer-grained layers that have more context.

Resource and Field (Application): Cerbos

The third and fourth granularities: can this user access this specific resource? And which fields should they see?

Cerbos evaluates context-aware policies. It receives the principal (user ID, roles, attributes), the resource (type, ID, owner, attributes), and the action. It returns an allow/deny decision and a list of visible fields.

# cerbos/policies/employee_profile.yaml (abbreviated: view/list and hr_admin rules omitted)
apiVersion: api.cerbos.dev/v1
resourcePolicy:
  version: default
  resource: employee_profile
  rules:
    - actions: ["view_sensitive", "update"]
      effect: EFFECT_ALLOW
      roles: ["employee"]
      condition:
        match:
          expr: request.principal.id == request.resource.attr.id
      output:
        expr: |-
          {"visible_fields": ["name", "title", "department", "salary_band", "base_salary", "ssn"]}

    - actions: ["view_sensitive"]
      effect: EFFECT_ALLOW
      roles: ["manager"]
      condition:
        match:
          expr: request.principal.id == request.resource.attr.manager_id
      output:
        expr: |-
          {"visible_fields": ["name", "title", "department", "salary_band"]}

What this means concretely: Alice (employee) views her own profile and sees everything including salary and SSN. Bob (manager) views Alice's profile and sees the salary band but not the exact salary or SSN. Carol (another employee) views Alice's profile and sees only name, title, and department.

Same endpoint. Same authentication. Same route-level authorization. Different data, determined by the relationship between the user and the resource.

What Cerbos catches that the previous layers cannot: role alone isn't sufficient. A manager should see sensitive data for their direct reports, not for every employee in the company. This requires knowing who the resource belongs to and what the manager's relationship is to that person. The gateway doesn't have that information. Only the application, at query time, knows these relationships.

Field Masking

Cerbos doesn't filter data itself. It returns a list of visible fields, and the application code uses that list to shape the response. This is a contract between policy and application:

result = cerbos.check_resource(
    principal={"id": user_id, "roles": [role]},
    resource={"kind": "employee_profile", "id": employee_id, "attr": {"manager_id": ...}},
    actions=["view_sensitive"]
)
visible = result.outputs["visible_fields"]
# Application filters the response to only include these fields

The application must respect this output. If it doesn't, row-level security on the sensitive tables (PII and financials) still limits which rows the database returns, though the directory table is open and relies on Cerbos for field masking. Field-level masking specifically depends on correct application implementation. This is one place where the "security as infrastructure" model has a boundary. The mesh can't mask JSON response fields for you.

Database (Backstop): PostgreSQL Row-Level Security

The final question: even if everything above passes, should the database return this row?

Row-level security here is deliberately split. The employee directory table (hr.employees, holding name, title, department, manager) is readable by any authenticated principal, because the directory itself isn't sensitive and Cerbos handles field masking on top of it. The sensitive tables (hr.employee_pii, hr.employee_financials) carry the strict row-level policy:

CREATE POLICY employee_financials_visibility ON hr.employee_financials
  USING (
    current_setting('app.current_roles', true) LIKE '%hr_admin%'
    OR employee_id::text = current_setting('app.current_user_id', true)
    OR EXISTS (
      SELECT 1 FROM hr.employees e
      WHERE e.id = employee_id
        AND e.manager_id::text = current_setting('app.current_user_id', true)
    )
  );

ALTER TABLE hr.employee_financials ENABLE ROW LEVEL SECURITY;
ALTER TABLE hr.employee_financials FORCE ROW LEVEL SECURITY;

This policy says: you can see a financial record if you're an hr_admin, if it's your own, or if you manage that employee. Nothing else. A query with no WHERE clause still returns only the rows this user is allowed to see. The PII table (hr.employee_pii) uses the same shape, and it.hardware_assets uses self-or-it_admin.

RLS uses current_setting('app.current_user_id') and current_setting('app.current_roles'), which are set by the application at the start of each transaction from the sidecar-projected headers. The database has no knowledge of JWTs, sidecars, or mesh tokens. It just enforces visibility based on the transaction context it's given.

Why Cerbos and RLS

The two overlap, but they catch different failures:

Cerbos fails (policy bug, service misconfiguration, Cerbos unreachable): The services fail closed. If Cerbos is unreachable, the application returns 403 (the client catches all Cerbos errors and denies by default). If a policy bug returns an incorrect allow decision, RLS on the sensitive tables still drops the PII and financial rows the user can't see. The response might leak directory fields that should have been masked, but the sensitive records stay protected.

Application fails (developer forgets to call Cerbos, broken filter logic, new endpoint without an auth check): RLS still enforces on the sensitive tables. SELECT * FROM hr.employee_financials without any Cerbos check still returns only rows matching the RLS policy. The mistake doesn't become a financial-data breach.

RLS fails (misconfigured policy, wrong user ID set on transaction): The query order is: the service loads the employee record first (to obtain attributes like manager_id for the Cerbos policy), then calls Cerbos, then queries the sensitive tables only after Cerbos approves. That initial directory read is intentionally open, so Cerbos is what gates the sensitive query. A broken sensitive-table RLS policy widens which rows come back, but Cerbos already gated access to those tables. A broken Cerbos policy widens what's returned, but RLS still limits sensitive-row visibility.

Neither Cerbos nor RLS alone is sufficient. Together, an application developer's mistake stops short of full exposure of PII or financial data.

Schema Isolation

RLS is complemented by PostgreSQL GRANT-level isolation. Each service has a dedicated database role:

ms2_hr_role can only access hr.* tables (employees, PII, financials)
ms3_it_role can only access it.hardware_assets
ms4_public_readwrite_role can only access public_data.*

Even without RLS, Device Inventory physically cannot query hr.employee_financials. The GRANT isn't there. This is a separate enforcement layer from RLS. If Device Inventory is compromised and the attacker somehow bypasses Istio AuthorizationPolicy (they'd need the mTLS cert and a valid token), the database still won't serve them HR data because the connection role lacks the privilege.

How the Granularities Compose: A Concrete Example

Alice (employee) requests her own profile with GET /api/profile/{alice_id}:

Network: Gateway can reach Profile Aggregator. Profile Aggregator can reach Employee Records. ✓
Request: Alice has the employee role, /api/profile/* allows employees. Token minted. ✓
Resource and field (Cerbos): Alice is requesting her own record (principal.id == resource.attr.id). Allowed, with visible fields that include salary and SSN.
Database (RLS): When the app reads her financials and PII, app.current_user_id matches the employee_id on those rows, so they're returned.

Bob (employee) tries to see Alice's financial data. His request also comes in as GET /api/profile/{alice_id}, and Profile Aggregator fans out toward Alice's sensitive records:

Network: Same path. ✓
Request: Bob has the employee role, /api/profile/* allows employees. Token minted. ✓
Resource and field (Cerbos): When the app tries to read Alice's financials, it checks the view_sensitive action. Bob is not Alice and not Alice's manager. Denied. The /financials and /pii endpoints on Employee Records are all-or-nothing, so the response is a 403 with no partial data.
Database (RLS), if Cerbos had a bug and allowed it: The RLS policy on hr.employee_financials requires employee_id == current_user_id or that the caller manages the employee. Bob is neither. Zero rows returned. The bug at the resource layer is caught at the database.

Bob viewing Alice's basic profile is a different action on the directory: the view action is allowed for any employee, and Cerbos returns visible_fields: [name, title, department]. Same request path, separate actions, separate policies, and the directory row is readable while the sensitive tables are not.

What's Next

The final post covers the patterns and anti-patterns across all layers: what makes this architecture compose well, what operational pain points exist, and what you'd do differently at scale.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth

The Most Important Feature We’re Building Into Our AI Agent Is the Ability to Say “I Don’t Know

Wed, 02 Sep 2026 01:14:41 +0000

The Most Important Feature We’re Building Into Our AI Agent Is the Ability to Say “I Don’t Know”

AI agents are usually demonstrated by showing what they can do.

Write code.

Search the web.

Analyze documents.

Create plans.

Call APIs.

Automate workflows.

But while building Zorgax, the AI coordination layer inside MyZubster, I’ve started thinking that one of the most important capabilities of an agent might be the opposite:

knowing when it does not have enough evidence to act.

The dangerous part isn't hallucination alone

Developers already know about hallucinations.

A model can generate an answer that sounds perfectly reasonable and is completely wrong.

But once an AI system becomes an agent, the problem changes.

A hallucinated paragraph is annoying.

A hallucinated decision connected to an API can become an action.

Consider this:

LLM
 ↓
Confident answer
 ↓
API call
 ↓
Database write
 ↓
External action

Now uncertainty isn't just a language problem.

It's an infrastructure problem.

We started adding another possible output

In some Zorgax workflows, we're experimenting with explicit uncertainty states.

Instead of forcing the system to always produce:

YES

or:

NO

we allow:

MORE EVIDENCE REQUIRED

And sometimes:

TIE / HUMAN CHOICE

This sounds almost trivial.

It isn't.

A surprising amount of automation architecture assumes that every process eventually produces a machine-decidable answer.

Reality doesn't work that way.

Sometimes the available evidence genuinely isn't enough.

Confidence is not evidence

Imagine asking an AI agent:

Is this product idea validated?

A language model can easily produce ten paragraphs explaining why the idea looks promising.

But what evidence exists?

How many people were interviewed?

What problems did they report?

Were the answers independent?

Did anyone actually test the product?

Did they simply say “sounds cool”?

Would they use it?

Would they pay?

These are completely different claims.

So our preferred architecture is becoming:

QUESTION
   ↓
AVAILABLE EVIDENCE
   ↓
QUALITY CHECK
   ↓
ANALYSIS
   ↓
┌─────────────────────────────┐
│ Sufficient evidence?        │
└─────────────────────────────┘
        ↓              ↓
       YES             NO
        ↓              ↓
Recommendation    MORE EVIDENCE
                      REQUIRED

The AI isn't rewarded for always having an answer.

Sometimes the correct output is uncertainty.

This changes how you design agents

Once you accept that principle, several things change.

The agent needs to distinguish between:

KNOWN
INFERRED
ASSUMED
UNKNOWN

Those states should not be interchangeable.

For example:

KNOWN:
A test returned HTTP 201.

INFERRED:
The endpoint probably created the resource correctly.

UNKNOWN:
Whether the user finds the feature useful.

UNPROVEN:
Whether anyone will pay for it.

An AI system can help with all four categories.

But it shouldn't silently collapse them into:

FACT

The same problem appears in software engineering

Suppose Zorgax sees:

CI: green

What does that prove?

It proves the tests passed.

It does not prove:

the architecture is correct
the product is secure
users want the feature
production will never fail
the business will succeed

Green CI is evidence for a specific claim.

Nothing more.

I think this is an important mental model for AI systems:

Every piece of evidence has a scope.

And it becomes even more important with IoT

MyZubster is also exploring systems involving sensors and physical-world evidence.

Now imagine:

Sensor
  ↓
Data
  ↓
AI
  ↓
Decision

A sensor reports something unusual.

Should the AI immediately act?

Maybe.

But first:

Is the sensor healthy?

Is the reading within its calibration range?

Is the timestamp valid?

Is the measurement recent?

Do other sensors agree?

Is the data complete?

Was there a communication failure?

A better pipeline might be:

PHYSICAL EVENT
      ↓
SENSOR DATA
      ↓
PROVENANCE
      ↓
QUALITY CHECK
      ↓
ZORGAX
      ↓
CONFIDENCE / EVIDENCE GATE
      ↓
HUMAN DECISION
      ↓
ACTION
      ↓
MEASUREMENT

The AI becomes part of the evidence pipeline.

Not the source of truth.

“I don't know” needs architecture

There is another problem.

Simply prompting an LLM:

Say you don't know when uncertain.

isn't enough.

The surrounding software needs to understand uncertainty too.

For example:

const result = await zorgax.evaluate(evidence);

switch (result.status) {
  case "SUPPORTED":
    prepareRecommendation(result);
    break;

  case "INSUFFICIENT_EVIDENCE":
    requestMoreEvidence(result.missing);
    break;

  case "CONFLICTING_EVIDENCE":
    escalateToHuman(result);
    break;

  default:
    failClosed();
}

Now uncertainty becomes a system state.

That is much more interesting than treating it as conversational wording.

Fail closed instead of inventing certainty

Security engineering already uses this idea.

If authorization cannot be verified:

DO NOT GRANT ACCESS

If payment cannot be verified:

DO NOT ACTIVATE ENTITLEMENT

If identity cannot be verified:

DO NOT MARK VERIFIED

Why shouldn't AI reasoning use a similar principle?

If evidence cannot support the claim:

DO NOT INVENT CONFIDENCE

This doesn't mean doing nothing forever.

It means returning the missing information required to continue.

INSUFFICIENT_EVIDENCE

Missing:
- additional observations
- independent confirmation
- baseline measurement
- verified identity

Now uncertainty becomes actionable.

Human-in-the-loop becomes more precise

“Human-in-the-loop” is sometimes used as a vague safety phrase.

But the interesting question is:

When exactly does the human enter the loop?

Not every operation needs approval.

Reading a public document?

Probably not.

Calculating a metric?

Probably not.

Preparing a draft?

Probably not.

Sending money?

Yes.

Changing verified identity?

Yes.

Publishing a consequential claim?

Potentially.

Acting when evidence is contradictory?

Definitely.

So instead of:

AI → HUMAN → EVERYTHING

we are exploring:

LOW-RISK / REVERSIBLE
AI can prepare or execute within scope

HIGH-RISK / IRREVERSIBLE
Human approval required

INSUFFICIENT EVIDENCE
Stop and request evidence

That makes the boundary much easier to reason about.

Agents need epistemic permissions

We already think about permissions like:

READ
WRITE
DELETE
EXECUTE
ADMIN

Maybe AI systems also need something like epistemic permissions:

OBSERVE
INFER
RECOMMEND
CLAIM
ACT

An agent might be allowed to infer something internally without being allowed to publish it as fact.

It might be allowed to recommend an action without being allowed to execute it.

It might be allowed to prepare a database change without being allowed to commit it.

That separation is becoming increasingly important in how we're designing Zorgax.

The architecture we're moving toward

The larger loop currently looks like this:

KNOWLEDGE
    ↓
EVIDENCE
    ↓
REASONING
    ↓
UNCERTAINTY CHECK
    ↓
RECOMMENDATION
    ↓
HUMAN APPROVAL
    ↓
ACTION
    ↓
MEASUREMENT
    ↓
LEARNING

The uncertainty check might be one of the most important components in that diagram.

Because without it:

REASONING
   ↓
CONFIDENCE
   ↓
ACTION

can become dangerous very quickly.

The goal isn't an AI that always answers

The AI industry often measures capability by asking:

How many tasks can the model complete?

For agents interacting with real systems, I think another metric will become increasingly important:

How reliably does the system recognize when it should not proceed?

A useful agent should be capable.

But it should also understand boundaries.

It should distinguish evidence from inference.

It should expose uncertainty.

It should fail safely.

And sometimes it should produce the least impressive AI answer possible:

I DON'T KNOW YET.

MORE EVIDENCE REQUIRED.

That might actually be a feature.

I'm building MyZubster + Zorgax in public and experimenting with AI agents, open-source infrastructure, human approval, IoT and evidence-driven automation.

I'm curious how other developers are handling this:

Does your agent architecture have an explicit “insufficient evidence” state, or does the model always have to produce an answer?

The Identity Flow: From Gateway to Database

Tue, 01 Sep 2026 22:13:32 +0000

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

The previous post laid out the five security layers as a static architecture. This post follows a single request through them, down to the question that matters in practice: what does a service actually trust, and why is trusting it safe?

The system is an HR employee directory, built to be attacked. Five microservices sit behind the mesh, numbered ms1 through ms5: Profile Aggregator (ms1) is the gateway-facing orchestrator; Employee Records (ms2) holds PII and salary data and Device Inventory (ms3) holds hardware assignments, both internal-only; Holiday Calendar (ms4) and Office Directory (ms5) serve holiday and office-location data, most of it public. Auth Service mints a signed identity token, x-mesh-identity, that carries the user through the mesh, and Vault signs it. This post traces one concrete request: an employee, Alice, requests her own profile with GET /api/profile/{id}. That request reaches Profile Aggregator, which fans out to Employee Records and Device Inventory. I'll follow the ms1 to ms2 hop, Profile Aggregator calling Employee Records, so every ms2-* resource below is that running example, not a special case.

Employee Records reads the user's identity from a plain HTTP header, eg x-ms2-user. The header is named per service (ms2 reads x-ms2-user, ms3 reads x-ms3-user) so one service can never consume another's identity headers. Employee Records does no JWT parsing and makes no callback to an auth service. It reads that header and treats the value as the authenticated user. That should look fragile, because headers are trivially spoofable. Any external client can send x-ms2-user: alice, and so can any compromised pod inside the cluster.

It works because of everything that happens before a service reads that header. A single request moves through eight steps from the gateway to the database, each adding one guarantee:

Header stripping at the gateway.
Authentication of the caller's credentials.
Route-level authorization against a coarse policy map.
Token minting, signed via Vault.
Token validation and header projection at the destination sidecar.
AuthorizationPolicy, binding peer identity to token claims.
The service contract, the two headers the application actually reads.
Database identity context for row-level enforcement.

Six of those steps run in the mesh before the application sees the request. By the time Employee Records reads x-ms2-user, its value has been stripped of anything the caller sent and re-derived from the signed token the destination sidecar verified. The last two steps are the service's own contract and the database scoping it then sets up.

The Trust Boundary

The Istio Ingress Gateway is where external input is converted into internal identity. Everything before it is untrusted. An external client can send any header it wants, including headers that look exactly like internal identity assertions. The gateway's job is to strip those, authenticate the caller, and produce a cryptographically signed token that downstream services can verify independently.

After the gateway, a service trusts only the headers its own destination sidecar wrote. The sidecar does the verification itself: it strips any inbound copy of the projected headers (like x-ms2-user), validates the JWT signature, checks the issuer, and verifies expiry, then writes the claims into fresh headers. The application reads those headers directly and runs no verification of its own, because the sidecar already ran it.

That trust is narrowly scoped. It covers only the headers the sidecar itself wrote after validating the token. Anything that merely arrived on the request, from an external client or an internal pod, is ignored. So if a compromised internal pod sends a spoofed x-ms2-user header, the destination sidecar strips it before the JWT filter runs and re-projects the value from the verified token. The spoofed header never reaches the application.

This model has a clear contract: the gateway is responsible for identity creation, sidecars are responsible for identity verification, and services are responsible for business logic. If any of those three fail in isolation, the other two still limit the blast radius.

Step 1: Header Stripping

Before authentication, before routing, before anything else, a Lua EnvoyFilter removes every internal trust header from the incoming request:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-prestrip
  namespace: istio-system
spec:
  priority: -100
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.ext_authz"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua.prestrip
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              function envoy_on_request(request_handle)
                local headers_to_remove = {}
                for key, value in pairs(request_handle:headers()) do
                  local lower_key = string.lower(key)
                  if string.match(lower_key, "^x%-platform%-") or
                     string.match(lower_key, "^x%-role%-") or
                     lower_key == "x-mesh-identity" or
                     string.match(lower_key, "^x%-ms%d%-user") or
                     string.match(lower_key, "^x%-ms%d%-role") then
                    table.insert(headers_to_remove, lower_key)
                  end
                end
                for _, key in ipairs(headers_to_remove) do
                  request_handle:headers():remove(key)
                end
              end

The filter runs at priority -100 with INSERT_BEFORE the ExtAuthz filter. The order is load-bearing: if ExtAuthz ran first and injected x-mesh-identity based on a valid session, but a spoofed x-ms2-user header also survived from the client, the downstream service would read the attacker-controlled value.

The two-pass approach (collect keys, then remove) is required because mutating headers during iteration is undefined behavior in Envoy's Lua API.

What gets stripped: x-platform-*, x-role-*, x-mesh-identity, and all x-ms<N>-user / x-ms<N>-role headers. Case-insensitive matching ensures no variant slips through.

Step 2: Authentication (ExtAuthz)

After stripping, the request hits the ExtAuthz filter. This calls Auth Service (auth-service), which resolves the caller's identity from one of two sources:

Bearer token: Extracted from the Authorization header, validated against Keycloak's JWKS endpoint. Roles come from realm_access.roles in the Keycloak JWT.

Session cookie: An opaque zt_session cookie, looked up in PostgreSQL. The session record holds the user's roles from the original OIDC flow.

Both paths produce the same output: a subject identifier, username, email, roles, groups, and department. The subject is deterministic across both auth paths. It's computed as uuid5(NAMESPACE_URL, "istio-security://users/<username>"), which means the same user always gets the same sub regardless of whether they authenticated via cookie or bearer token. This is important for RLS (which filters by sub) and for audit trails.

Step 3: Route-Level Authorization

Before minting a token, Auth Service checks a coarse policy map. This is the first authorization gate:

# Coarse route/role policy map (ext_authz.py)
if path.startswith("/api/profile/"):
    allowed_roles = {"employee", "manager", "hr_admin", "it_admin"}
    decision["audience"] = ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"]
elif path.startswith("/api/holidays") and method == "GET":
    allowed_roles = {"employee", "manager", "hr_admin", "it_admin", "public_data_admin", "security_auditor"}
    decision["audience"] = ["ms4-holiday-calendar"]
elif path.startswith("/api/holidays") and method in ("POST", "PUT", "DELETE", "PATCH"):
    allowed_roles = {"public_data_admin", "hr_admin"}
    decision["audience"] = ["ms4-holiday-calendar"]
else:
    decision["reason"] = "Route not defined in coarse policy map"
    return decision

user_roles = set(decision["roles"])
if not allowed_roles.intersection(user_roles):
    decision["reason"] = "Unauthorized role for route"
    return decision

If the user's role doesn't match the route, the request is denied with 401. No token is minted. The audience field determines which services this token will be valid for. A token for the profile endpoint is valid at Profile Aggregator (ms1-profile-aggregator), Employee Records (ms2-employee-details), and Device Inventory (ms3-hardware-assets). A token for the holidays endpoint is valid only at Holiday Calendar (ms4-holiday-calendar). This prevents a valid token from one flow being replayed against a different service.

Any path not in the policy map is denied by default. GET /api/offices is deliberately absent, because those reads are public: the gateway's AuthorizationPolicy exempts them from ExtAuthz, so an unauthenticated read reaches Office Directory (ms5-office-locations) without a token and is treated as anonymous/public. Writes to /api/offices do go through the map above and require public_data_admin.

Step 4: Token Minting

If authentication and route authorization both pass, Auth Service mints a signed mesh token. The JWT is built manually (no library) and signed via Vault Transit:

header = {"alg": "RS256", "typ": "JWT", "kid": str(vault_key_version)}

payload = {
    "iss": "auth-service",
    "sub": subject_uuid,
    "preferred_username": username,
    "roles": ["employee", "manager"],
    "roles_csv": "employee,manager",
    "aud": ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"],
    "act": {"sub": "ms1-profile-aggregator"},
    "exp": now + 300,  # 5 minutes
    "jti": str(uuid4()),
    "request_id": request_id
}

The signing input (header_b64.payload_b64) is sent to Vault Transit's /v1/transit/sign/mesh-identity endpoint. Vault signs it with the RSA-2048 private key and returns the signature. Auth Service never holds the private key.

Key details:

roles_csv exists because Istio's outputClaimToHeaders can only project string claims, not arrays.
act (the delegation claim from RFC 8693) is set to ms1-profile-aggregator when Profile Aggregator is in the audience. Auth Service sets this at mint time because it already knows (from the route-policy map) that the profile flow will pass through Profile Aggregator to reach Employee Records and Device Inventory. The act claim asserts that Profile Aggregator is the authorized intermediary for this request. When Profile Aggregator forwards the token downstream, the receiving AuthorizationPolicy verifies both the mTLS identity (is this actually Profile Aggregator?) and the act claim (was Profile Aggregator designated as the intermediary?). If a different service somehow obtained this token, the mTLS check would fail.
kid is fetched live from Vault on every mint. When keys rotate, new tokens immediately reference the new version.
TTL is 5 minutes. Short enough to limit replay windows, long enough that a fan-out from Profile Aggregator to Employee Records and Device Inventory completes comfortably.
The JWT is assembled manually rather than via a library like PyJWT. This is because the signing step goes through Vault Transit's API (not a local key), and standard JWT libraries assume they hold the private key. Building the token by hand keeps the signing interface clean: encode header and payload, send bytes to Vault, attach the returned signature.

Step 5: Token Validation and Header Projection

The signed token travels as the x-mesh-identity header. The destination sidecar handles it in two stages before the application sees the request.

First, a Lua EnvoyFilter strips any inbound x-ms2-user / x-ms2-role headers. It runs at priority: 10 with INSERT_BEFORE the JWT authentication filter, so spoofed values are removed before any claim is projected:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: header-projection-ms2
  namespace: zt-apps
spec:
  priority: 10
  workloadSelector:
    labels:
      app: ms2-employee-details
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.jwt_authn"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua.projection
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              function envoy_on_request(request_handle)
                request_handle:headers():remove("x-ms2-user")
                request_handle:headers():remove("x-ms2-role")
              end

This is the destination-side counterpart to the gateway pre-strip described in The Trust Boundary: the same strip-then-project pattern, now enforced at the workload. Any inbound copy of these headers is removed before the JWT filter runs.

Then the RequestAuthentication resource validates the token and projects the verified claims:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: mesh-identity-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  jwtRules:
    - issuer: "auth-service"
      jwksUri: "http://auth-service.zt-apps.svc.cluster.local:8000/auth/jwks"
      forwardOriginalToken: false
      fromHeaders:
        - name: "x-mesh-identity"
      outputClaimToHeaders:
        - header: "x-ms2-user"
          claim: "sub"
        - header: "x-ms2-role"
          claim: "roles_csv"

The sidecar fetches public keys from the Auth Service JWKS endpoint, validates the JWT signature, checks iss and exp, and then projects sub into x-ms2-user and roles_csv into x-ms2-role. Because the strip filter already cleared any inbound copies, these headers can only carry claims from a validated token. These are the only headers the application reads.

One Istio nuance is easy to miss: RequestAuthentication validates a JWT when one is present. It does not, by itself, mean "a JWT is required." The requirement comes from the AuthorizationPolicy in the next step. That policy checks claims that only exist after successful validation, so a missing or invalid token has no matching rule and is denied.

Profile Aggregator has forwardOriginalToken: true because it needs to propagate x-mesh-identity to Employee Records and Device Inventory. All other services have it set to false since they're terminal and don't make downstream calls that need identity propagation.

Step 6: AuthorizationPolicy

After token validation, Istio's AuthorizationPolicy enforces token presence, token scope, and peer identity:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-ms1-to-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/zt-apps/sa/ms1-profile-aggregator-sa
      to:
        - operation:
            ports: ["8000"]
      when:
        - key: request.auth.claims[aud]
          values: ["ms2-employee-details"]
        - key: request.auth.claims[act][sub]
          values: ["ms1-profile-aggregator"]

Three things must be true simultaneously: the request must come from ms1-profile-aggregator-sa over mTLS, the token must be valid for ms2-employee-details, and the token must name ms1-profile-aggregator as the authorized intermediary. The peer identity comes from the client certificate. The aud and act values come from the signed JWT. These are independent cryptographic assertions. Compromising one doesn't give you the other.

The aud claim is a list because a single profile request legitimately reaches more than one service. Istio's claim matching checks whether the configured value is present in that claim, so a token minted for Profile Aggregator, Employee Records, and Device Inventory can match ms2-employee-details here. A token minted only for Holiday Calendar would not match.

This is confused-deputy prevention: stopping a privileged intermediary from being tricked into using its authority on behalf of an attacker. If Profile Aggregator's service account is somehow used by a different process, the process would still need a validly-signed token that authorizes Profile Aggregator to call Employee Records for this user flow. If an attacker replays a legitimate token from a different pod, the mTLS identity check fails. Both bindings must match.

The deny behavior depends on having an AuthorizationPolicy that targets the workload. Istio allows traffic by default when no policy applies. In this architecture, every protected workload has at least one scoped ALLOW policy, so unmatched traffic is denied. A request with no token, a bad signature, the wrong audience, or the wrong caller identity has no matching rule and never reaches the application.

Fine-grained authorization (can this user see this specific record, and which fields?) also runs at this point, via Cerbos. That's the resource and field-level layer, covered in a later post. This post stays on the identity flow.

Step 7: The Service Contract

After all of this, what does the service actually see? Two headers:

async def get_ms2_headers(
    x_ms2_user: str | None = Header(None),
    x_ms2_role: str | None = Header(None),
    x_request_id: str | None = Header(None),
):
    if not x_ms2_user or not x_ms2_role:
        raise HTTPException(status_code=401, detail="Missing required legacy headers")
    return {"user": x_ms2_user, "role": x_ms2_role, "request_id": x_request_id}

The service reads x-ms2-user and x-ms2-role. It returns 401 if they're missing. That's the entire identity contract. The 401 message calls them "legacy" headers, but the name only refers to this plain-header interface: the service reads simple headers as if it had no awareness of the mesh, and the mesh does the real verification upstream. The service does no JWT parsing, OIDC configuration, or Vault calls of its own. It trusts that if these headers are present, the sidecar has already done the verification.

Profile Aggregator forwards x-mesh-identity and x-request-id to downstream services. It explicitly does not forward x-ms2-user or x-ms2-role since the receiving sidecar projects those independently from the JWT. Profile Aggregator cannot influence what identity Employee Records sees.

Step 8: Database Identity Context

The final step. Before executing any query, the service sets the transaction context:

async def set_rls_context(session: AsyncSession, user_id: str, roles: str, request_id: str):
    await session.execute(
        text("SELECT set_config('app.current_user_id', :user_id, true)"),
        {"user_id": user_id},
    )
    await session.execute(
        text("SELECT set_config('app.current_roles', :roles, true)"),
        {"roles": roles},
    )
    if request_id:
        await session.execute(
            text("SELECT set_config('app.request_id', :request_id, true)"),
            {"request_id": request_id},
        )

PostgreSQL RLS policies read app.current_user_id and app.current_roles to filter rows; app.request_id is set for audit correlation. The true third argument makes each setting transaction-local, so it doesn't leak across pooled connections.

At this point, the database sees the same user identity that the sidecar projected from the verified token. In normal operation, the service is a pass-through for identity. It reads the sidecar-projected header and sets it on the transaction.

A fully compromised service process could call set_config with an arbitrary value. RLS doesn't protect against a malicious process controlling its own DB session. But the layers above constrain this: the compromised service only has its own scoped DB role (Employee Records can only access hr.* tables, Device Inventory only it.*), and AuthorizationPolicy prevents it from reaching other services' databases. The blast radius of a compromised Employee Records service is limited to the HR data it already has access to, scoped by RLS to the identities it actually receives requests for.

Failure Behavior

The secure path matters, but so does the failure path. A missing or invalid mesh token fails at AuthorizationPolicy because the required claims are absent. If Auth Service is down, ExtAuthz cannot approve the request and no mesh token is minted. If Vault is unreachable, Auth Service cannot sign a token. If the sidecar cannot refresh JWKS, it continues using its cached keys until the cache expires; after that, validation fails closed for tokens it cannot verify.

The one operational caveat is key rotation. Istio sidecars cache JWKS, so there is a short window where a sidecar may still verify tokens with a cached old public key. The next post covers that rotation window and why the token TTL is only 5 minutes.

The Complete Chain

Gateway strips external headers -> Auth Service validates credentials -> Auth Service checks route-level policy -> Auth Service mints a signed token -> sidecar validates the token and projects claims -> AuthorizationPolicy checks peer identity and token claims -> service reads projected headers -> database enforces row-level visibility.

Eight steps. By the time Employee Records reads x-ms2-user, that value has passed through header stripping, credential validation, route authorization, token signing, sidecar validation, and peer binding. The service receives a narrow identity contract and passes that identity into PostgreSQL for row-level enforcement.

What's Next

The next post covers the signing infrastructure: why Vault Transit instead of local keys, how key rotation works without downtime, and what the blast radius looks like when different components are compromised.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth

Demystifying Connection Pooling: How to Stop Crashing Your Database

Tue, 01 Sep 2026 12:57:32 +0000

Connection pooling is a performance-optimization technique used in software development to keep a cache of active database connections ready for reuse. Instead of establishing a brand-new connection and shutting it down every time an application needs to talk to a database, it borrows an active connection from a pre-established "pool." This significantly reduces the time, energy, and server overhead associated with opening and closing communication channels on the fly.

A Real-World Analogy: The Airport Taxi Stand

To understand how this works, imagine a busy taxi stand outside a bustling airport terminal. Instead of manufacturing a brand-new car every single time a passenger walks out of the terminal and then destroying that car once they reach their destination, a fleet of taxis is kept parked and waiting at the stand.

When a passenger arrives, they immediately hop into an available taxi, take their trip, and the taxi returns to the stand to wait for the next passenger. In this scenario, the taxi stand is the connection pool, the taxis are the active database connections, and the passengers are the individual data requests sent by your application. Keeping a fixed fleet of vehicles running is exponentially faster and less wasteful than creating and destroying cars for every single trip.

Why Connection Pooling Matters Daily in Tech

In the tech industry, connection pooling is critical for maintaining application speed and system stability. Creating a database connection is an "expensive" operation in computing terms because it requires performing complex network handshakes, validating security credentials, and allocating physical memory on both the application server and the database server.

Without pooling, if ten thousand users visit a popular website simultaneously, the server would try to open ten thousand unique database connections at the exact same moment. This behavior can quickly exhaust the database's hardware resources, causing the database to crash and leaving users with broken pages or infinite loading spinners. By utilizing a connection pool, software engineers cap the maximum number of active connections to a safe limit (like 20 or 50), protecting the database from overloading while dramatically speeding up response times for everyday users.

Connection Pooling in Action

Below is a simplified example in JavaScript using a PostgreSQL database library. It highlights the difference between manually managing individual client connections and utilizing an automatic connection pool.

// Scenario A: Without Connection Pooling (Slow and Fragile)
import { Client } from 'pg';

async function getUserWithoutPool(userId) {
  const client = new Client({ connectionString: 'postgresql://db' });
  await client.connect(); // Manually opens a brand-new connection
  
  const result = await client.query('SELECT * FROM users WHERE id = $1', [userId]);
  
  await client.end(); // Manually closes the connection
  return result.rows[0];
}

// Scenario B: With Connection Pooling (Fast and Robust)
import { Pool } from 'pg';
// Creates a reusable pool with a maximum limit of 10 connections
const pool = new Pool({ connectionString: 'postgresql://db', max: 10 });

async function getUserWithPool(userId) {
  // Automatically borrows a connection, queries the database, and
  // returns the connection back to the pool when finished
  const result = await pool.query('SELECT * FROM users WHERE id = $1', [userId]);
  return result.rows[0];
}

The Key Takeaway

Ultimately, connection pooling highlights a fundamental rule of scalable system architecture: recycling high-cost resources is almost always faster and safer than rebuilding them from scratch. By treating database connections as a shared, reusable fleet rather than single-use assets, engineering teams ensure their software remains highly resilient, cost-effective, and lightning-fast under pressure, preventing catastrophic database meltdowns during unexpected traffic spikes.

Originally published on my blog. You can read the alternative breakdown here.

Three Steps That Decide Whether Plain English Database Queries Actually Work

Mon, 31 Aug 2026 02:43:48 +0000

Most of us here have either built a text-to-SQL layer or been asked to. The demos always look great and the production version is the one that quietly returns the wrong number, so here is what actually separates the two.

The Schema Is The Feature, Not The Prompt

A natural language query layer is not a prompt engineering problem. When you connect a database, the model has to read the whole schema first: every table, every column, every data type, every foreign key. That is what lets it decide which tables to touch, which columns to filter on, and how to join them.

Skip that step and you still get SQL back. It runs, it returns rows, and it is wrong in a way nobody catches, because the join went through the wrong key. The systems that hold up are the ones that expose the generated SQL so a human can spot-check it, and that read relationships rather than guessing them from column names. There is a longer walkthrough of how that works in practice on asking your database questions in plain English.

Dirty Data Breaks It Before The Model Does

The second failure mode is upstream. Names stored three different ways, phone numbers in four formats, the same customer entered twice with a typo, blank fields where a required value should be. Your query is correct and your answer is still wrong, because COUNT DISTINCT on a column with fuzzy duplicates is not counting what you think.

This is the least glamorous part and it is where most of the accuracy lives. Fuzzy matching to find near-duplicate records, normalizing formats across the table, validating each field against the rules it should obey, flagging rows with missing required values instead of silently averaging around them. A practical rundown is in cleaning data with AI.

Let The Model Find What You Did Not Ask About

The third step is the one teams skip entirely. Every query you write encodes something you already suspected. Anomaly detection covers the rest: point anomalies like an order fifty times the average, contextual ones like revenue that is normal for a retailer and impossible for a two person consultancy, and collective ones where ten individually boring transactions land in the same minute.

That last category is the argument for running it at all, because no dashboard you would have thought to build catches it. Worth reading how AI anomaly detection actually runs if you are adding this to an existing stack.

The Takeaway

Plain English querying is not one feature, it is three: clean the data, give the model the real schema, then let it surface what your questions never covered. Ship it in that order and the demo and the production system finally agree with each other.

Demystifying the N+1 Query Problem: Why Your App is Secretly Slow

Fri, 28 Aug 2026 21:39:45 +0000

What is the N+1 Query Problem?

The N+1 query problem is a performance bottleneck that occurs when an application makes database requests in an inefficient loop. Instead of fetching all necessary data in a single, organized request, the application makes one initial request (the "1") followed by a separate request for every single item returned (the "N"). This results in a massive wave of unnecessary database communication that slows down applications.

A Relatable Analogy

Imagine you are a middle school teacher who needs to check if all 25 students in your classroom brought their signed field trip permission slips.

Instead of asking the entire class to raise their hands or collecting all the slips in one basket (a single, optimized request), you walk up to the first student's desk and ask, "Did you bring your slip?", and then walk all the way back to your desk to write down the answer. Then, you walk back out to the second student's desk, ask them, and walk back to your desk. You repeat this entire process 25 separate times.

The "1" is your initial step of looking at your student roster. The "N" (25) represents the individual, repetitive, time-consuming trips you make back and forth across the room to get the details. It is an exhausting, highly inefficient waste of physical effort and time.

Why It Matters Daily in Tech

In real-world software, every single trip to a database—a specialized system that stores and organizes app data—takes time. Data has to travel across a network, which introduces latency (the delay before data transfer begins).

When engineers build everyday features like social media feeds, e-commerce shopping carts, or analytics dashboards, they often use developer tools called Object-Relational Mappers (ORMs) to write code instead of raw database commands. If left unchecked, these tools automatically generate the N+1 query pattern. For example, if your feed displays 50 posts, the system will fire 1 query to get the posts, and then 50 separate, rapid-fire queries to look up the author for each individual post.

Software engineers must actively detect and prevent this because it overwhelms database servers, drives up CPU usage, increases cloud hosting bills, and makes web pages load painfully slowly for end-users, especially under heavy traffic.

The Concept in Code

Here is a simple example written in JavaScript, comparing the slow N+1 approach with the optimized solution. This example uses a mock database helper representing what happens under the hood of a typical web server.

// THE PROBLEM: The N+1 Query Approach
async function getBooksWithAuthorsBad() {
  // 1. Fetch all 50 books from the database (This is the "1" query)
  const books = await database.getBooks();

  for (const book of books) {
    // 2. Fetch the author details for EACH book individually (These are the "N" queries)
    // If there are 50 books, this block executes 50 separate times!
    book.author = await database.getAuthorById(book.authorId);
  } 
  return books;
}

// THE SOLUTION: Eager Loading (1 Query Total)
async function getBooksWithAuthorsGood() {
  // Fetch all books AND their corresponding authors in a single, combined database call
  // The database engine joins the tables together efficiently before sending data back
  const booksWithAuthors = await database.getBooks({ include: ['authors'] });
  return booksWithAuthors;
}

The Takeaway

Eliminating the N+1 query problem is not about writing faster code; it is about communicating with your data storage system more respectfully. By shifting from a repetitive loop-based mindset to a batch-based retrieval mindset, you stop wasting precious network cycles and processing power. This simple structural change keeps your applications lightning-fast, prevents database crashes, and ensures a seamless experience for your users as your platform grows.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

I Moved My Next.js Dashboard Logic Into Postgres. My Frontend Got Boring (And That's the Point).

Tue, 25 Aug 2026 16:16:35 +0000

My dashboard had a useMemo doing arithmetic it had no business doing.

It was a Pokémon TCG Pocket collection tracker, but that part doesn't matter. What matters is that the home page needed to show three things: overall completion, completion per set, and which set you were closest to finishing. The way I'd built it, the browser was fetching every card and every owned record, then grinding through the math on each render to figure all of that out.

It worked. It also got slower and harder to read every time the data grew or I added a metric. So I moved the aggregation out of React and into Postgres, and the surprising result was that my frontend got boring. Fewer hooks, less state, almost nothing left to break.

That's the whole argument of this post: aggregation belongs in the database, and when you put it there, the React code that's left over is the kind of boring you actually want in the layer your users touch.

What "fetching everything into React" actually looks like

Here's the shape of the original dashboard. Load all the cards, load the user's owned rows, then derive everything on the client.

const [cards, setCards] = useState<Card[]>([]);
const [owned, setOwned] = useState<OwnedCard[]>([]);

useEffect(() => {
  (async () => {
    const { data: allCards } = await supabase.from("cards").select("*");
    const { data: ownedRows } = await supabase
      .from("user_cards")
      .select("card_id");
    setCards(allCards ?? []);
    setOwned(ownedRows ?? []);
  })();
}, []);

const ownedIds = useMemo(() => new Set(owned.map((o) => o.card_id)), [owned]);

const perSet = useMemo(() => {
  const groups: Record<string, { total: number; have: number }> = {};
  for (const card of cards) {
    const g = (groups[card.set_id] ??= { total: 0, have: 0 });
    g.total += 1;
    if (ownedIds.has(card.id)) g.have += 1;
  }
  return groups;
}, [cards, ownedIds]);

const overall = useMemo(() => {
  const total = cards.length;
  const have = cards.filter((c) => ownedIds.has(c.id)).length;
  return total ? have / total : 0;
}, [cards, ownedIds]);

It's not bad code. But look at what it's costing:

The payload grows with the collection. I'm shipping every card row to the browser just to count them. The user never sees most of that data. It exists only to be reduced to a few numbers.
The logic can't be tested without mounting a component. The "closest set to finishing" rule lives inside hooks, tangled up with fetching and state.
Every new metric is more client state. Want a "rarest card you're missing" stat? That's another loop, another useMemo, another thing that re-runs on render.

I was using React as a database query engine. It's not one.

Why this is the wrong layer

Counting, grouping, and computing ratios across rows is a set operation. It's the thing relational databases were built to do, indexed and compiled, sitting right next to the data.

When I do it in React instead, I pay for the same work twice. Once to serialize every row and send it over the wire, and again to recompute the aggregates in JavaScript on the client. The database already knew the answer was "you own 41 of 86 cards in this set." I just declined to ask it that question and asked for all 86 rows instead.

This isn't really a performance micro-optimization. It's a question of which layer owns which job. Aggregation is a database job. I'd put it in the wrong place.

Moving it into an RPC function

Supabase lets you call Postgres functions directly with rpc(), so I wrote one that returns exactly the rows the dashboard needs: one row per set, with the counts already computed.

create or replace function public.get_home_set_completion_rows()
returns table (
  set_id text,
  set_name text,
  set_slug text,
  release_date date,
  image_url text,
  data_version text,
  total_cards integer,
  owned_cards integer,
  missing_cards integer
)
language sql
stable
security invoker
set search_path = public
as \(
  select
    sets.id as set_id,
    sets.name as set_name,
    sets.slug as set_slug,
    sets.release_date,
    sets.image_url,
    sets.data_version,
    count(cards.id)::integer as total_cards,
    count(user_cards.card_id)::integer as owned_cards,
    (count(cards.id) - count(user_cards.card_id))::integer as missing_cards
  from public.sets
  left join public.cards
    on cards.set_id = sets.id
  left join public.user_cards
    on user_cards.card_id = cards.id
    and user_cards.user_id = auth.uid()
  group by
    sets.id,
    sets.name,
    sets.slug,
    sets.release_date,
    sets.image_url,
    sets.data_version
  order by
    sets.release_date desc nulls last,
    sets.id;
\);

The left join against user_cards filtered by auth.uid() is doing the per-user work right inside the query. Sets the user has zero cards from still show up with owned = 0, which is what I want.

And the call site collapses to this:

const { data } = await supabase.rpc("get_home_set_completion_rows");

That's it. No fetching everything, no client-side grouping, no Set of owned IDs. The component receives compact, purpose-built rows and maps over them. Overall completion is a sum of two columns. "Closest to finishing" is a sort. Both are now trivial because the hard part already happened in the database.

The React file went from a pile of hooks to a Server Component that calls one function and renders the result. That's the "my frontend got boring" moment, and deleting all that state felt great.

When a view beats a function (and vice versa)

I reached for an RPC function here because of that auth.uid() filter, since the result depends on who's asking. But it's worth knowing when a plain view is the better tool:

Use a view when the aggregation is stable, reusable, and doesn't need parameters. A view reads like a table, composes into other queries, and is the simplest thing that works. Card counts per set that aren't user-specific would be a great view.
Use a function when you need arguments (a user ID, a set filter, a scope toggle) or multi-step logic that a single select can't express cleanly.

One footgun worth flagging: both views and functions interact with row-level security, and security definer versus security invoker decides whose permissions the code runs under. A security definer function bypasses the caller's RLS, which is occasionally what you want and frequently a way to leak other users' data by accident. If you're using RLS to protect per-user rows, and you should be, make sure your function respects it rather than quietly stepping around it.

The objections worth taking seriously.

A few reasonable complaints, because this trade isn't free.

"Now my logic is split across two languages and harder to grep." True. I moved complexity, I didn't delete it. The aggregation now lives in SQL instead of TypeScript, so there's a context switch to read the whole story. My argument is just that it lands in the right place, with the database owning database work, and that the SQL version is shorter and more declarative than the loops it replaced.

"SQL is harder to test." I'd push back on this one. The function is deterministic: give it fixture rows and it produces the same counts every time, with no component to mount. And separating it out made the logic that correctly stayed in TypeScript easier to test too. My pack-recommendation engine, which sums pull probabilities for missing cards and ranks packs by expected value, is real application logic, not aggregation, so it belongs in app code. Once it wasn't tangled up with fetching and dashboard math, I could unit-test it on its own. The point isn't "move everything into Postgres." It's "move the aggregation, keep the decisions."

"This is just more infrastructure to manage." Fair, but it's version-controlled SQL living next to the rest of the schema, not a magic black box. It migrates like everything else.

The rule of thumb

If you take one thing from this post, take this:

Push aggregation and filtering down to the database. Keep decisions and presentation in the app.

Or, more bluntly: if React is looping to count things, that loop probably wants to be a GROUP BY.

The frontend got boring, with smaller payloads, fewer hooks, and less state to corrupt, and boring is exactly what you want from the layer your users actually touch. The interesting, opinionated, easy-to-get-wrong logic moved to where it could be expressed simply and tested in isolation.

The genuinely contested part is where you draw the line. I keep ranking and business decisions in app code and push counting and filtering into the database, but I know people who'd move more in either direction. Where do you draw it?

(Full code, if you want to see the whole thing wired together: mwiginton/pocketdex-tracker. Or try the live app: pocketdex-tracker.vercel.app.)

Continuous Delivery Office Hours Ep.5: Delivering database changes

Mon, 24 Aug 2026 12:57:43 +0000

In the previous episode, we talked about different service design approaches. This time, we dive deeper into database changes.

Even for teams that can effortlessly deploy their application code, database changes can be more stressful. Changing a schema is a high-stakes operation, and there are many ways to do it badly.

Read on to find out why database deployments are different from application deployments, and what techniques you can use to make them worry-free.

Watch the episode

You can watch the episode below, or read on to find some of the key discussion points.

Watch Continuous Delivery Office Hours Ep.5

Why are databases different

When you change your application's code and discover an issue, it's trivial to revert to the previous version. In rare cases where a change affects your data, there may be cleanup to do.

You can use timestamps and a one-off process to fix those rare data-mess-up instances, but they hint at something fundamentally different about database updates. They have different levels and types of risk associated with them.

If you think backups will save you from a bad database deployment, you haven't yet tried it. Sure, they prevent total data loss, but between your backup and your fix, the data moved. Often by a lot.

There are techniques to apply transactions from the backup up to the point of failure, but if your change caused an issue you needed to roll back, you probably don't want to apply those transactions automatically. Welcome to the "data remediation project".

If you added a new column or table as part of your database change and you need to roll back, you have to decide what to do with any data in those tables. Do you forget it, or do you need to keep hold of it and reapply it later when you make a new attempt to extend the schema?

Modern software teams prefer fix-forward for application issues, which are reasonably easy to roll back. Databases take rollbacks to another level.

Crucial modernization steps

Imagine you met a friend for lunch and they told you they store they application's source code on a network share, rather than in version control. You'd think it was a joke, and when you realize it's not, you'd form a strong opinion about the kind of sloppy outfit they must be running.

When we meet for lunch, what are you going to tell me about your database schema and static data? Please tell me it's all in version control, not on a network share.

You should make all database changes by updating the files in version control and deploying them like you would your application code. You progress the change through environments to ensure it works, and you avoid embarrassment caused by the application failing because someone forgot to add the new column in production.

There are further choices to make, which we'll cover next, but failing to version-control your database is unforgivable.

State-based vs migration-based approaches

On to the first choice for your database project. Do you make state-based or migration-based updates?

State-based schemas describe the desired state of the database. It will list each table with its columns, indexes, and relationships. You use a model-based tool to deploy the database, which compares your current state with the desired state and applies the changes for you.

Some state-based tools convert the differences into standard database scripts, like ALTER TABLE... scripts. Others perform migrations by creating a new table with the changes and moving the data into it. This is important if you're using a technology like replication, which prevents the model migration mechanism from working.

The alternative to state-based database updates is migration-based updates, where you write your own ALTER TABLE scripts. You keep all your scripts in version control and use a tool that applies them in order and tracks when each was applied, preventing the same script from being applied twice to the same database.

The main difference between the two is procedural. You can code-review the migration scripts on demand, but you'll need to review state-based migration scripts after your tools generate the implementation plan, which can make the review task more of a large batch.

Tooling and automation

Whichever approach you use, tooling helps it work. Many of our customers use Redgate tools as part of their deployment process to manage database schema changes, and there's value in leaning on a tool written by folks who care deeply about the problem and how to solve it.

Crucially, you shouldn't make any database changes outside of tool-based automation.

Test data management

Once you've automated your database schema and static data, it's worth considering your test data. When automated or manual acceptance tests fail, it's usually because someone unwittingly messed up the data. Prior test runs often leave data in an inconsistent state, especially if a test failure halts the run.

You can resolve this issue by automating your test data setup. Not only does this make it easy to reset the data during your build and test cycle, but it also lets you provide a self-service runbook for the test team to reset the data in their test environment whenever they need to.

There's an up-front investment in this, but I can promise that it takes fewer hours than fixing your test data a few times.

Database refactoring patterns

The final thought to ponder concerns the steps you take in changing your database schema. When you're in the habit of deploying your database and application in the same release process, you start to depend on this change coordination.

You delete a column from the database and immediately deploy the application, with all references to the deleted column removed. It looks like it works smoothly, but it's a trap.

Imagine you had a critical bug in the application and had to redeploy the previous version. Now you can't, because the previous version will try to read from a column that doesn't exist. You no longer have a quick, easy back out plan, since you have to re-add the column, and you'll also need data to put in it.

This approach also prevents you from making seamless deployments, as even if you progressively roll out the application version, the old version will error out due to the database change. You may deploy in the opposite order, application first, then database. You'll discover the same problem with new columns that the latest version expects to find in the database.

You need to decouple database and application deployments, and there's a whole book on the topic, called Refactoring Databases (Ambler, Sadalage). You can start by following the expand/contract pattern, which splits updates into steps. The principle is that you don't delete a column until the production application has no reads or writes. You add a column and don't reference it in your code until you deploy to production.

This means you can run the current version and the new version of the software against the same database, which means you can progressively roll out the new version and redeploy the prior version without touching the database.

Databases are only as hard as you make them

The database is high-risk, which is why updating it can be scary. I hope you've found this post full of practical advice for making database deployments robust and stress-free.

Database deployments, like application deployments, should be a happy time. You should be celebrating the new features and enhancements you've delivered to your users, not biting your nails and worrying that something's about to go horribly wrong.

Happy deployments!

Continuous Delivery Office Hours is a series of conversations about software delivery, with Tony Kelly, Bob Walker, and Steve Fenton.

You can find more episodes on YouTube, Apple Podcasts, and Pocket Casts.

Parallel Processing Gone Wrong

Mon, 24 Aug 2026 10:38:39 +0000

⚡ "Let's Make It Faster!" (Famous Last Words)

The Scene: Sprint planning. The feature? Bulk data processing.
Me (naive): "Let's just run everything in parallel! It'll be 10x faster!"
Production (3 days later): catches fire
What I Thought Would Happen:
• 100 items to process
• Run them all in parallel
• Finish in 1/10th the time
• Get promoted
• Become CTO
What Actually Happened:
• 100 concurrent database connections
• Database connection pool exhausted
• ALL requests start failing (not just ours)
• 2 AM phone call from the CTO (not a promotion call)
• Emergency rollback at 3 AM
• Very awkward Slack conversation
The Broken Code:

// "I'll just parallelize everything!"
foreach (var item in items) {
    Task.Run(() => ProcessItem(item)); // YOLO
}

Why This Implodes:
❌ No limit on concurrent operations
❌ No shared state protection
❌ No error handling per item
❌ Overwhelms downstream services
❌ Database can't handle the connections
❌ Network ports get exhausted
❌ And my ego gets crushed
The Fix That Saved My Job:

int passCount = 0;
int failCount = 0;
var errors = new ConcurrentDictionary<string, string>();

// Key: MaxDegreeOfParallelism controls concurrency
Parallel.For(0, items.Count, 
    new ParallelOptions { 
        MaxDegreeOfParallelism = Environment.ProcessorCount // Usually 4-8
    }, 
    (i) => {
        try {
            ProcessItem(items[i]);
            
            // Thread-safe counter increment
            Interlocked.Increment(ref passCount);
        } 
        catch (Exception ex) {
            Interlocked.Increment(ref failCount);
            
            // Thread-safe error collection
            errors.TryAdd(i.ToString(), ex.Message);
        }
    });

// Now you can safely check results
Console.WriteLine($"Success: {passCount}, Failed: {failCount}");

The Golden Rules I Learned:
1️⃣ Parallelism ≠ Performance - Too much parallelism kills performance
2️⃣ Know Your Bottleneck - If it's I/O bound, 4-8 threads is usually optimal
3️⃣ Protect Shared State - Use Interlocked or ConcurrentCollections
4️⃣ Configure, Don't Hardcode - Make MaxDegreeOfParallelism environment-specific
5️⃣ Test Under Load - It works on my machine ≠ it works in production
The Results After Proper Implementation:
Controlled concurrency = predictable performance
✅ Database connections stay healthy
⚡ 6x speedup (not 100x, but sustainable)
Zero production incidents
Sleep peacefully without 3 AM alerts
The Hard Truth:
More threads ≠ More speed
It's like adding more cooks to a kitchen. At some point, they just bump into each other.
What's your parallel processing horror story? I can't be the only one who learned this the hard way

dotnet #parallelprocessing #performanceoptimization #productionissues #softwareengineering #threading #scalability #lessonslearned #debugging #csharp #techleadership