Coder Legion - Recent posts tagged postgres

Database Transactions - ACID Properties

Thu, 29 Oct 2026 01:49:48 +0000

Introduction

Yesterday, I went to a restaurant with my friends. We ordered some food, finished eating, and asked the waiter for the bill.

While the waiter was preparing the bill, I started thinking about how the billing system works behind the scenes.

When we place an order, the system has to do multiple things, like saving the order details, calculating the bill, and updating the payment status.

But what if something goes wrong in the middle? What if the system crashes before completing all these operations?

This is where database transactions come into the picture.

Recently, I was learning about database transactions, ACID properties, and isolation levels. In this article, I'll share what I understood, along with some simple examples.

What is a Database Transaction?

A database transaction is a group of one or more database operations that are treated as a single unit of work.

Let's take our restaurant example.

When a customer places an order, the system might need to perform these operations:

Create an order.
Save the ordered items.
Update the available stock.
Calculate and save the total bill.

All these operations are related. If one of them fails, we don't want the database to be left with incomplete information.

So, instead of executing them as separate independent operations, we can group them into a single transaction.

A transaction generally involves three important commands:

BEGIN — Starts a transaction.
COMMIT — Saves the changes made by the transaction.
ROLLBACK — Undoes the uncommitted changes if something goes wrong.

For example:

BEGIN;

INSERT INTO orders (order_id, table_id, total_amount)
VALUES (101, 5, 500);

INSERT INTO order_items (order_id, item_id, quantity)
VALUES (101, 10, 2);

UPDATE inventory
SET quantity = quantity - 2
WHERE item_id = 10;

COMMIT;

Here, all three database operations are part of one transaction.

If everything succeeds, we commit the transaction.

If an operation fails, we can roll back the transaction so that the database doesn't keep only part of the changes.

But what happens if the database itself crashes while executing a transaction?

A database can crash for several reasons, such as power failures, hardware problems, operating system failures, or software bugs.

Even in these situations, the database needs a way to recover without leaving incomplete transactions behind.

To handle these kinds of problems and maintain reliable data, database transactions follow four important properties, commonly called ACID properties.

ACID Properties

ACID stands for:

A — Atomicity
C — Consistency
I — Isolation
D — Durability

Each property solves a different problem.

Atomicity ensures that either all operations in a transaction happen or none of them happen.

Consistency ensures that a transaction preserves the rules and constraints defined in the database.

Isolation controls how multiple transactions running at the same time interact with each other.

Durability ensures that once a transaction is committed, its changes survive a crash, under the database's durability guarantees.

Let's understand each property with examples.

Atomicity

Atomicity means that either all the operations in a transaction should succeed, or none of them should happen.

Let's take our restaurant example.

Suppose a customer orders 2 pizzas, and each pizza costs ₹200.

When the waiter confirms the order, the system needs to perform two operations:

Create an order with a total amount of ₹400.
Reduce the available pizza stock by 2.

Now, let's say the order is created successfully, but the system crashes before updating the stock.

In this case, the order exists in the database, but the stock has not been updated. This leaves the data in an incorrect state.

To avoid this, we can execute both operations inside a transaction.

BEGIN;

INSERT INTO orders (order_id, amount)
VALUES (101, 400);

UPDATE inventory
SET quantity = quantity - 2
WHERE item_id = 1;

COMMIT;

If both operations succeed, the transaction is committed.

But if something goes wrong before the transaction is committed, the database rolls back the changes.

So, even if the order was inserted successfully, it won't remain in the database if the transaction fails.

This is what atomicity guarantees: All or nothing.

How does the database maintain atomicity?

Databases use transaction logs and recovery mechanisms to handle failures.

For example, PostgreSQL uses Write-Ahead Logging (WAL).

Before modified data pages are written to their final location on disk, PostgreSQL records the changes in WAL.

If the database crashes, it uses WAL and transaction status information during recovery to ensure that committed transactions are recovered and incomplete transactions do not become visible as committed changes.

We'll understand WAL in more detail when we discuss durability.

Consistency

Consistency means that a transaction should maintain the rules and constraints defined in the database.

While learning about consistency, I came across two different concepts: Data Consistency and Read Consistency.

1. Data Consistency (Correctness)

Data consistency is about keeping the data correct and making sure it follows the rules defined in the database.

Let's say we have three tables.

Blogs Table

blog_id	likes
1	5
2	1

Users Table

user_id	name
1	John
2	Edmon

Pictures Table

user_id	picture_id
1	1
1	2
2	3
2	4

Here, user_id in the Pictures table is a foreign key referencing the Users table.

Now suppose we try to insert a picture with user_id = 5, but there is no user with that ID in the Users table.

INSERT INTO pictures (user_id, picture_id)
VALUES (5, 5);

The database rejects this operation because it violates the foreign key constraint.

This is called referential integrity.

Similarly, databases maintain data consistency using constraints such as primary keys, foreign keys, UNIQUE, NOT NULL, and CHECK.

Apart from these constraints, we may also have business rules that need to be maintained by our application or database.

So, the main idea of ACID Consistency is that a transaction should move the database from one valid state to another valid state without breaking the defined rules.

In simple terms, it focuses on correctness of data.

2. Read Consistency (Availability and Freshness)

Now let's look at consistency from another perspective.

In real applications, we may have multiple copies of the same database, called replicas.

Why do we need replicas?

One reason is availability. If one database server goes down, another server may still be able to serve requests. Replicas can also help distribute read traffic.

Let's say we have one primary database and two replicas.

Initially, all three databases contain the same value.

Primary Database → 90
Replica 1 → 90
Replica 2 → 90

Now suppose a transaction updates the value from 90 to 100 in the primary database and commits.

But replication may take some time.

At that moment, the data might look like this:

Primary Database → 100
Replica 1 → 90
Replica 2 → 90

Now imagine two users trying to read that value.

T1:

User 1 reads from the primary database and gets 100.

T2:

User 2 reads from Replica 1 and gets 90.

Both users are reading the same record, but they are getting different values.

This happens because the replicas haven't received the latest update yet.

After some time, replication catches up.

Primary Database → 100
Replica 1 → 100
Replica 2 → 100

Now all three databases have the same value.

This is an example of Eventual Consistency.

It means different replicas may temporarily have different values, but eventually they become consistent if updates stop and replication completes successfully.

Availability vs. Consistency

Here, we have an interesting trade-off.

Suppose a replica hasn't received the latest update yet.

We have two possible approaches.

Option 1: Allow the read

We can allow the user to read from that replica, even though it has older data.

The user gets a response immediately, but the value might be stale.

This favors availability, although it doesn't guarantee that the user gets the latest data.

Option 2: Wait for the latest data

We can wait until the replica catches up, or route the request to a database that has the required version.

The user gets fresher data, but the request may take longer or become unavailable if the required data cannot be reached.

So, depending on the system's requirements, we may need to make trade-offs between availability and the consistency of reads, especially during network failures.

What's the Difference?

The main thing I understood is that these two concepts focus on different problems.

ACID Consistency → Correctness

It makes sure the database follows its defined rules and constraints. A transaction should not leave the data in an invalid state.

Read Consistency → Freshness of Data

It defines what values a reader is allowed to see, especially when data is replicated across multiple servers.

Availability is related to read consistency because allowing reads from replicas with slightly older data can help a system continue serving requests.

So, ACID consistency is mainly about keeping data correct, while consistency in distributed reads is about how up-to-date and consistent the data needs to be across different replicas.

Isolation

Until now, we have seen how transactions maintain atomicity and consistency.

But what happens when multiple transactions run at the same time?

Let's take our restaurant example.

Suppose two waiters are taking orders from different tables. Both are using the same billing system.

Now imagine only one pizza is left in stock, and both waiters are trying to place an order for it.

If both transactions read the available quantity at the same time, they might both think the pizza is available.

This is where Isolation comes in.

Isolation controls how multiple transactions running at the same time interact with each other.

Ideally, we want the result of concurrent transactions to be the same as some valid order of executing them one after another. But running every transaction one after another can affect performance.

So databases provide different isolation levels.

Before understanding them, let's look at four common problems that happen when transactions execute concurrently.

Dirty Read
Non-Repeatable Read
Phantom Read
Lost Update

1. Dirty Read

A dirty read happens when a transaction reads a change made by another transaction that hasn't committed yet.

Let's take a sales table.

pid	qnt	price
P1	10	5
P2	20	4

Suppose two transactions are running.

Time	TX1	TX2
T1	`BEGIN;`
T2	`SELECT pid, qnt * price FROM sales;` → P1: 50, P2: 80
T3		`BEGIN;`
T4		`UPDATE sales SET qnt = qnt + 5 WHERE pid = 'P1';`
T5	`SELECT SUM(qnt * price) FROM sales;` → 155
T6		`ROLLBACK;`
T7	`COMMIT;`

At T5, TX1 reads the updated value from TX2 even though TX2 hasn't committed yet.

Later, TX2 rolls back its changes, but TX1 has already used that uncommitted data.

This is called a Dirty Read.

PostgreSQL note: This example represents a database that allows dirty reads. PostgreSQL doesn't allow them, even with Read Uncommitted.

2. Non-Repeatable Read

A non-repeatable read happens when a transaction reads the same record twice and gets different values because another transaction committed an update in between.

Let's use the same Sales table.

Time	TX1	TX2
T1	`BEGIN;`
T2	`SELECT pid, qnt * price FROM sales;` → P1: 50, P2: 80
T3		`BEGIN;`
T4		`UPDATE sales SET qnt = qnt + 5 WHERE pid = 'P1';`
T5		`COMMIT;`
T6	`SELECT SUM(qnt * price) FROM sales;` → 155
T7	`COMMIT;`

Here, TX1 initially reads the sales values.

Meanwhile, TX2 updates the quantity and commits.

When TX1 later calculates the total, it sees the updated data.

So TX1 gets different results within the same transaction.

This demonstrates the effect of a non-repeatable read. Strictly speaking, the standard definition involves reading the same row twice; here, the second statement is an aggregation over the changed rows.

This can happen under PostgreSQL's default Read Committed isolation level.

3. Phantom Read

A phantom read happens when a transaction executes a query and later gets a different set of matching rows because another transaction inserts, deletes, or changes matching records.

Suppose we have this table:

id	ch
1	a
2	a
3	b
4	b

Time	TX1	TX2
T1	`BEGIN;`
T2	`SELECT * FROM test WHERE ch = 'b';` → 2 rows
T3		`BEGIN;`
T4		`INSERT INTO test (id, ch) VALUES (5, 'b');`
T5		`COMMIT;`
T6	`SELECT * FROM test WHERE ch = 'b';` → 3 rows
T7	`COMMIT;`

Initially, TX1 gets two records.

Meanwhile, TX2 inserts another record with ch = 'b' and commits.

When TX1 executes the same query again, it gets three records.

The newly appearing record is called a phantom row.

PostgreSQL's Repeatable Read prevents phantom reads because the transaction continues using the same snapshot.

4. Lost Update

A lost update happens when two transactions read the same value and update it independently, causing one transaction's change to be overwritten.

Suppose P1's current quantity is 10.

TX1 wants to increase it by 10, and TX2 wants to increase it by 5.

The expected final quantity is 25.

Time	TX1	TX2
T1	`BEGIN;`
T2	`SELECT qnt FROM sales WHERE pid = 'P1';` → 10
T3		`BEGIN;`
T4		`SELECT qnt FROM sales WHERE pid = 'P1';` → 10
T5	`UPDATE sales SET qnt = 20 WHERE pid = 'P1';`
T6		`UPDATE sales SET qnt = 15 WHERE pid = 'P1';` → Waits for TX1
T7	`COMMIT;`
T8		TX2's UPDATE completes
T9		`COMMIT;`

The final quantity becomes 15 instead of 25.

TX2 overwrites the value written by TX1, so TX1's update is lost.

This is called a Lost Update.

This can happen when an application calculates a new value using an earlier read and then writes that fixed value back.

One way to prevent it is by using SELECT FOR UPDATE.

BEGIN;

SELECT qnt
FROM sales
WHERE pid = 'P1'
FOR UPDATE;

UPDATE sales
SET qnt = qnt + 10
WHERE pid = 'P1';

COMMIT;

Here, PostgreSQL locks the selected row. Another transaction trying to update the same row must wait.

If we simply want to increment a value, we can also do it directly:

UPDATE sales
SET qnt = qnt + 5
WHERE pid = 'P1';

This avoids calculating the new value using an outdated application-side read.

Database Isolation Levels

Now that we have seen these problems, let's understand how databases prevent them.

SQL defines four common isolation levels:

Read Uncommitted
Read Committed
Repeatable Read
Serializable

1. Read Uncommitted

This is the weakest standard isolation level.

A transaction may read changes made by another transaction before they are committed.

So dirty reads, non-repeatable reads, and phantom reads are possible.

For example, in our Dirty Read scenario, TX1 could read the updated sales total even though TX2 later rolls back.

We can request this isolation level using:

BEGIN TRANSACTION
ISOLATION LEVEL READ UNCOMMITTED;

However, PostgreSQL treats Read Uncommitted as Read Committed.

So we cannot reproduce dirty reads in PostgreSQL.

2. Read Committed

Read Committed prevents dirty reads.

A transaction only reads committed data.

However, each statement gets a new snapshot, so the same query may return different results when executed twice.

Let's take our sales example.

Time	TX1	TX2
T1	`BEGIN ISOLATION LEVEL READ COMMITTED;`
T2	`SELECT pid, qnt * price FROM sales;` → P1: 50, P2: 80
T3		`BEGIN;`
T4		`UPDATE sales SET qnt = qnt + 5 WHERE pid = 'P1';`
T5		`COMMIT;`
T6	`SELECT SUM(qnt * price) FROM sales;` → 155
T7	`COMMIT;`

Even though TX1 hasn't finished, its next query sees the changes committed by TX2.

This is allowed because Read Committed creates a new snapshot for each statement.

Read Committed is PostgreSQL's default isolation level.

3. Repeatable Read

In Repeatable Read, a transaction continues reading from the same snapshot.

Let's use the same example.

Time	TX1	TX2
T1	`BEGIN ISOLATION LEVEL REPEATABLE READ;`
T2	`SELECT pid, qnt * price FROM sales;` → P1: 50, P2: 80
T3		`BEGIN;`
T4		`UPDATE sales SET qnt = qnt + 5 WHERE pid = 'P1';`
T5		`COMMIT;`
T6	`SELECT SUM(qnt * price) FROM sales;` → 130
T7	`COMMIT;`

Notice the difference.

Even though TX2 committed its update, TX1 still calculates the total using its original snapshot.

This happens because PostgreSQL uses a consistent snapshot established when the transaction first executes a statement that needs one.

So changes committed by other transactions after that snapshot aren't visible to TX1.

One interesting thing about PostgreSQL is that Repeatable Read also prevents phantom reads.

This is stronger than the minimum Repeatable Read guarantees described by the SQL standard.

However, Repeatable Read doesn't prevent every possible concurrency anomaly.

For example, write skew can still happen.

4. Serializable

Serializable is the strongest standard isolation level.

It ensures that concurrent transactions produce a result equivalent to executing those transactions one after another in some order.

This doesn't mean PostgreSQL actually runs every transaction sequentially.

Transactions can still run concurrently.

But PostgreSQL checks for dangerous dependencies that could produce a result that isn't equivalent to any serial execution.

PostgreSQL uses Serializable Snapshot Isolation (SSI).

Let's take an example.

Suppose we have two records.

Record	Value
A	100
B	100

Our business rule is that at least one record must always have a value of 100.

Two transactions are running at Serializable isolation.

Time	TX1	TX2
T1	`BEGIN ISOLATION LEVEL SERIALIZABLE;`
T2	`SELECT value FROM test WHERE id IN ('A', 'B');` → 100, 100
T3		`BEGIN ISOLATION LEVEL SERIALIZABLE;`
T4		`SELECT value FROM test WHERE id IN ('A', 'B');` → 100, 100
T5	`UPDATE test SET value = 0 WHERE id = 'A';`
T6		`UPDATE test SET value = 0 WHERE id = 'B';`
T7	`COMMIT;`
T8		`COMMIT;` → Serialization failure

Both transactions initially read A = 100 and B = 100.

TX1 changes A to 0.

TX2 changes B to 0.

If both transactions committed successfully, we would have:

A = 0
B = 0

This would violate our business rule.

This kind of anomaly is called Write Skew.

Under PostgreSQL Repeatable Read, both transactions may successfully commit because they update different rows.

But PostgreSQL Serializable can detect the dangerous dependency and abort one transaction.

We may see an error like:

ERROR: could not serialize access due to
read/write dependencies among transactions

When this happens, the application needs to retry the entire transaction.

Serializable is useful when we need stronger correctness guarantees, but we should also be prepared to handle transaction retries.

Pessimistic and Optimistic Concurrency Control

While learning about isolation, I also came across two ways of handling concurrent updates.

Pessimistic Concurrency Control

Here, we lock the data before changing it.

For example:

BEGIN;

SELECT qnt
FROM sales
WHERE pid = 'P1'
FOR UPDATE;

UPDATE sales
SET qnt = qnt + 10
WHERE pid = 'P1';

COMMIT;

The FOR UPDATE statement locks the selected row.

Another transaction trying to acquire a conflicting lock must wait until the first transaction releases it.

This is useful when conflicts are likely.

Optimistic Concurrency Control

In optimistic concurrency control, we allow transactions to proceed without locking the record exclusively in advance.

Before applying the update, we check whether another transaction has changed the data.

If a conflicting change is detected, the operation may fail and need to be retried.

This approach can be useful when conflicts are less frequent.

PostgreSQL's Serializable Snapshot Isolation also uses conflict detection, although its implementation is different from application-level version checking.

Comparison of Isolation Levels

Isolation Level	Dirty Read	Non-Repeatable Read	Phantom Read
Read Uncommitted	Possible	Possible	Possible
Read Committed	Prevented	Possible	Possible
Repeatable Read	Prevented	Prevented	Possible under SQL standard
Serializable	Prevented	Prevented	Prevented

This table describes the general SQL-standard guarantees.

PostgreSQL has two important differences:

Read Uncommitted behaves like Read Committed, so dirty reads are prevented.
Repeatable Read also prevents phantom reads.

Lost updates need separate consideration because their behavior depends on the type of update and the database's concurrency-control mechanisms.

What I Understood About Isolation

Isolation is not simply about making transactions run one after another.

It is about controlling how concurrent transactions interact.

Different isolation levels provide different guarantees, and the right choice depends on what our application needs.

For many applications, Read Committed is sufficient.

But when multiple queries need to see a consistent snapshot, Repeatable Read may be useful.

And when concurrent transactions must behave like some serial execution, Serializable provides the strongest guarantee.

The important thing is to understand what problems each isolation level prevents and choose accordingly.

Durability

Durability means that once a transaction is committed, its changes should not be lost, even if the database crashes or restarts.

Let's take our restaurant example.

Suppose a customer finishes eating, pays the bill, and the waiter confirms the payment in the billing system.

The database successfully commits the transaction.

But immediately after that, the database server crashes.

When the server restarts, the payment information should still be there.

This is what Durability guarantees.

How do databases maintain durability?

Databases use different techniques to make sure committed changes survive failures.

Some common techniques are:

Write-Ahead Logging (WAL)
OS Page Cache and fsync
Checkpoints
Asynchronous Snapshots
Append-Only Files (AOF)

1. Write-Ahead Logging (WAL)

Writing every change directly to the database files can be expensive.

For example, when we update a record, the database may need to modify multiple data pages and indexes.

Instead of writing all these changes to their final locations immediately, databases like PostgreSQL use Write-Ahead Logging (WAL).

The idea is simple.

Before modified data pages are written to disk, the database first records the changes in a log.

Let's see how PostgreSQL handles an UPDATE query.

BEGIN;

UPDATE orders
SET payment_status = 'Paid'
WHERE order_id = 101;

COMMIT;

When PostgreSQL executes this query, it modifies the data page in shared buffers and creates the required WAL records in WAL buffers.

Both are initially in RAM.

Under normal synchronous commit settings, PostgreSQL persists the required WAL before confirming the transaction.

The modified data pages can be written to disk later.

Here is how the process works.

Now suppose the database crashes after COMMIT, before the modified data pages are written to disk.

When PostgreSQL restarts, it can use the persisted WAL records to recover the committed changes.

This is why WAL plays an important role in durability.

2. OS Page Cache and fsync

One interesting thing I learned is that writing data to a file doesn't always mean it has reached the physical disk.

Usually, when a program writes to a file, the operating system may first keep that data in its page cache, which is part of RAM.

Why?

Because writing to RAM is much faster than writing directly to disk. The operating system can write the cached data to disk later.

But how is this related to database durability?

Let's take our PostgreSQL example.

When PostgreSQL executes an UPDATE query, it modifies the data in shared buffers and generates WAL records in WAL buffers.

Both are initially in RAM.

When PostgreSQL writes these WAL records to its WAL file, they may first go through the operating system's page cache.

So even though PostgreSQL has written the WAL to a file, the data might still be sitting in RAM.

The flow looks like this:

Now suppose the WAL has been written to the OS page cache, but it hasn't reached the disk yet.

If the machine suddenly loses power, the cached WAL data may be lost.

And if the modified data page was also only in RAM, PostgreSQL might lose that change.

This is where fsync() comes in.

fsync() is an operating system call that requests the modified data of a file to be synchronized to stable storage.

In simple terms, it helps make sure the data isn't just sitting in the OS page cache.

Under normal durability settings, PostgreSQL uses fsync() or other appropriate synchronization methods to persist the required WAL before confirming a synchronous commit.

So even if the database crashes immediately after COMMIT, PostgreSQL can use the persisted WAL to recover the changes.

The important difference is:

WAL records the changes needed for recovery, while fsync or a similar synchronization mechanism helps make those WAL records durable.

There is also a performance cost. Synchronizing data to storage takes time, so frequent synchronization can increase write latency.

PostgreSQL provides settings such as fsync and synchronous_commit to control this behavior.

3. Checkpoints

We have seen that PostgreSQL can commit a transaction before writing all the modified data pages to their final locations.

But those modified pages cannot remain only in memory forever.

They eventually need to be written to disk.

This is where Checkpoints come in.

A checkpoint is a process where PostgreSQL ensures that dirty data pages (modified in memory but not yet on disk) are written to storage and records a point from which crash recovery can proceed.

For example, suppose the database has been running for several hours and thousands of transactions have modified data.

If the database crashes, PostgreSQL needs to replay WAL records to recover changes that weren't already reflected in the database files.

Without checkpoints, the amount of WAL that needs to be processed during recovery could become very large.

By performing checkpoints periodically, PostgreSQL can reduce the amount of recovery work required.

So checkpoints mainly help with writing modified data pages to disk and controlling crash recovery time.

4. Asynchronous Snapshots

Some databases use snapshots as a persistence mechanism.

Instead of saving every change immediately, they periodically save a snapshot of their data.

For example, Redis supports RDB snapshots.

Suppose a database takes a snapshot at 10:00 AM.

After that, some new records are created at 10:05 AM.

Now imagine the server crashes at 10:06 AM, before the next snapshot is taken.

If snapshots are the only persistence mechanism, those recent changes may be lost.

So the durability guarantee depends on how frequently snapshots are taken and whether other persistence methods are enabled.

5. Append-Only Files (AOF)

Another persistence technique is an Append-Only File (AOF).

Redis supports this mechanism.

Instead of saving the entire dataset after every update, Redis records write operations in an append-only log.

When the server restarts, Redis can use the persisted AOF to rebuild its data.

However, durability depends on how frequently the AOF is synchronized to disk.

For example, Redis can be configured to synchronize after every write or approximately once per second.

So AOF provides another way to recover data, but recent changes may still be lost depending on the persistence settings.

Conclusion

When I started learning about database transactions, I thought it was mostly about BEGIN, COMMIT, and ROLLBACK.

But as I went deeper, I came across some interesting things, especially what happens when two transactions run at the same time and how PostgreSQL handles failures.

There is still a lot more to learn about how databases work internally, but understanding these concepts helped me see what actually happens behind a simple database query.

Cancel or send: one winner. An order flow on Prisma Postgres and Compute

Wed, 28 Oct 2026 21:30:34 +0000

Werbung / Disclosure: Prisma gave me a coupon for two months of a paid plan for free and said it will add free credits, so I could try Prisma Postgres and Compute for this project. All results below come from my own runs.

Summary

I built an order flow with a 10-minute grace period. Cancellation and dispatch must have exactly one winner in the database.
The winner is decided in Postgres, with a guarded update, a row lock and SKIP LOCKED. No clock and no timer decides it.
I broke the code on purpose, one line at a time, to check that my tests really fail. One of my race tests passed even without the lock.
On Compute, an idle app sleeps. My poller is a timer inside the server, so it only runs when someone calls the app. This changes the timing, not the correctness.

The problem

A customer orders a custom T-shirt. I promise a grace period of 10 minutes. During that time the customer can cancel. After that, the printer gets an email and the customer cannot cancel any more.

Two things can happen at almost the same moment. The customer presses cancel, and my poller wants to send. Exactly one of them must win. A cancelled order must never reach the printer, because an email cannot be recalled.

Design decisions

Claim first, then send

The poller claims an order by changing its status from PENDING to SENT before it sends the email. If the customer cancels first, the claim finds nothing. If the claim wins, the cancel is rejected with a 409.

Here, SENT means the order has been claimed for dispatch, not that the email has been delivered. If the server crashes after the claim, the printer may get no email. I set printerNotifiedAt only after Resend accepts it. A stuck-order query finds SENT orders without that timestamp, and a resend route lets me recover them.Sending the email before claiming the order could allow it to be sent even if the cancellation wins the database race.

A guarded transition

Every status change goes through one function. It checks that the transition is allowed for the actor. Then it writes with the old status in the where clause:

const { count } = await prisma.order.updateMany({
  where: { id, status: order.status },
  data: { status: to },
});
if (count === 0) throw new ConflictException('Order changed, please retry');

If someone changed the row between my read and my write, count is 0 and the caller gets a 409. This is the sixth time I used this read-then-write pattern across my projects.

A Postgres poller, not a queue

The hosted version has no Redis, so there is no BullMQ. The order and its deadline are written in one transaction. A timer loop claims due orders with one statement:

UPDATE "Order" SET status = 'SENT' -- plus updatedAt
WHERE id IN (
  SELECT id FROM "Order"
  WHERE status = 'PENDING' AND "sendAt" <= (now() AT TIME ZONE 'UTC')
  ORDER BY "sendAt" LIMIT 10
  FOR UPDATE SKIP LOCKED
)
RETURNING id

There is no enqueue gap, because the deadline is data. The costs are a polling delay and retry logic that I write myself. Today there is no automatic retry. The safety net is the stuck-order query and the resend route.

The server owns sendAt. A client-supplied value could skip the grace period, so the request schema rejects unknown keys.

A row lock on the design

An order points at a design. If the design changes after the order exists, the printer prints something the customer never ordered, and nobody sees an error. Creating an order, updating a design and deleting a design all take SELECT ... FOR UPDATE on the design row first. A design counts as locked when it has any order that is not cancelled.

Tests, and a test that proved nothing

I wrote integration tests against a real Postgres database. Then I broke the code on purpose, one line at a time, and checked that a test turned red.

I removed	What went red
the status in the `updateMany` guard	the cancel-versus-poller test, and all 30 printer/admin confirm iterations
`status = 'PENDING'` in the claim	a cancelled order was emailed, and two pollers sent two emails
`SKIP LOCKED`	the locked-row test (the pollers waited)
`FOR UPDATE` in order creation	only the deterministic lock test
`FOR UPDATE` in design update	only the deterministic lock test (65 passed, 1 failed)
`FOR UPDATE` in design removal	the remove-versus-create test: a foreign key error instead of a 409

The lock rows are the lesson. First I tested the lock with 30 free-running iterations of update versus create. They all passed with the lock removed. The broken interleaving ends in the same final database state as the harmless one, so no assertion on the final state can tell them apart. Only a test that pauses order creation after it takes the lock, and then checks whether the update can finish, sees the bug.

A test also found a timezone bug. I ran the integration session in America/Los_Angeles. Comparing sendAt with a bare now() failed because sendAt is a timestamp without time zone. The comparison now uses now() AT TIME ZONE 'UTC', assuming sendAt is stored as a UTC timestamp. This fixes the timezone interpretation, but not clock skew between the application and database; I discuss that remaining limitation below.

Prisma Postgres and Compute

Prisma ORM 8 (I write Prisma 8 for short) is a release candidate and Composer is in Early Access. Compute has been generally available since August 2026. Everything in this section was true on my machine in October 2026. Versions: Prisma CLI 8.0.0-rc.27, @prisma/orm-postgres 8.0.0-rc.20, Composer 0.34.0.

Why Prisma ORM 8 is in this project

The Console told me that my repository used prisma ^7.7.0, that deploys run on Prisma 8, and that I should upgrade first. So I installed Prisma 8 next to Prisma 7, inferred a contract from my database, and planned a baseline migration. The plan had 15 operations from an empty database: four enums, four tables, the (status, sendAt) index and four foreign keys.

Later, Martin from Prisma told me that a Prisma 7 app runs fine on Compute and that Prisma 8 is not required. I did not test a Prisma 7 only deploy myself, so I cannot confirm it, and I reported the Console banner to Prisma.

I kept Prisma 8 for one job: the deploy applies my baseline migration to the new database. My queries still run on Prisma 7 with the pg adapter. The docs also describe a plainer option, rawPostgres(), which hands your own client a db.url. With it, you migrate the database yourself. I did not use it.

I kept the queries on Prisma 7 for another reason. The release status page lists what Prisma 8 does not have yet, for example transaction isolation levels, atomic increment, most nested writes and P2002-style error codes. My locks use raw SQL and a guarded updateMany, and I did not try them on Prisma 8.

Two more facts for Prisma 7 users. First, npx prisma now runs the Prisma 8 CLI, which has different data commands (contract, db and migration). Pin prisma to ^7 and use npx prisma@7, or install an alias package like I did. Second, Prisma 8 is a release candidate, and the release status page expects general availability in October 2026.

Getting a NestJS 11 app onto Compute

Compute runs a single bundled file. My notes:

The deploy tool refused Node 22.23.4 and asked for Node 24.11.1 or newer. The Compute docs say Node 22.18 or newer on the 22 line is enough, so the docs and the tool disagree.
bcrypt is a native addon. Bundled into one file, it crashed at startup with __dirname is not defined. I switched to bcryptjs. Login and the tests still passed.
prisma dev is not supported on Windows yet, so I tested the bundle with plain node.
Composer needs type: module. I put the Composer files in a deploy folder with its own package.json, so nest build stays CommonJS. A small launcher reads the database URL that the platform provides and then starts the compiled app.
The Console's setup pull request failed at npm install with ERESOLVE. I could reproduce a conflict on my machine by adding alchemy next to Jest 30 (a Babel peer dependency clash). I could not see the Console's log, so I do not know if it is the same cause.

The first deploy created ten resources, including the database and the migration step, and returned a public URL. The docs say Compute runs deployed apps on Bun, and I built my bundle for Node. The app worked when I tried it. The poller is switched off on that deployment, so orders stay PENDING there.

The surprise: scale to zero versus my poller

My poller is a timer inside the server. I deployed a small test app with a counter that ticks every 5 seconds and shows its history. Then I left it alone overnight.

The startedAt value was the same after 15 hours, so the app was never restarted.
Only 2 ticks were recorded. The first gap was 54,940 seconds.
New ticks appeared at the moments I opened the page. Then they stopped again.

The Compute docs describe this: idle instances sleep with a memory snapshot and resume in milliseconds. A timer inside the process only runs when someone calls the app.

What still holds: one database claim wins, and a cancelled order cannot be claimed for dispatch through this flow. What breaks is the timing. The email goes out after the next request, not at minute 10. Exactly-once email delivery is not guaranteed; a crash during recovery can result in a duplicate.

Open question: where should the poller run?

Compute has no built-in cron scheduler. A Prisma engineer told me that the recommended patterns are options 2 and 3 below, and that keep-awake is meant for short, bounded work. The Compute limitations page and the Composer cron building block describe both. I have not chosen between them yet, and I would like to discuss it. These are the options I found:

Keep the instance awake. Prisma says keep-awake is meant for short, bounded work, not for an always-on loop. The docs add that it does not guarantee continuity through restarts or deployments. I have not measured what a permanently awake instance costs.
An external trigger (recommended by Prisma). A scheduler outside Compute calls a protected route. The route wakes the app, runs the same claim logic and returns. My tests show that extra triggers are safe: two pollers claim an order once. The timing precision depends on the scheduler, and the request must start responding within 60 seconds.
The Composer cron building block (recommended by Prisma). Its scheduler is a service that never scales to zero, so it costs one warm instance. Composer is in Early Access, and I did not find an example of mounting the cron runner in a NestJS app.

Questions that are still open:

Does the always-warm cron scheduler count against the Free plan, and how much memory does it use?
Has anyone mounted the cron runner inside a NestJS app?

If you solved this, I would like to hear how.

Raw output

These are trimmed outputs from my own runs.

The deploy:

t-shirt-backend-test
├─ database   postgres-database db_...
└─ app        compute-service cps_...
Stack deployed (10/10)
Deployed t-shirt-backend-test to production in 3m 1s.

The bundle with the native bcrypt addon, at startup:

ReferenceError: __dirname is not defined in ES module scope
    at ... require_node_gyp_build()(path.resolve(__dirname))

The test app's ticks, first call (15 hours after the deploy):

{"startedAt":"2026-10-25T16:36:23.113Z","now":"2026-10-26T07:59:53.581Z","count":2,
 "gaps":[{"from":"2026-10-25T16:42:26.838Z","to":"2026-10-26T07:58:06.462Z","seconds":54940}]}

The second call, about 10 minutes later (the first gap is shortened):

{"startedAt":"2026-10-25T16:36:23.113Z","now":"2026-10-26T08:10:05.860Z","count":4,
 "gaps":[{"seconds":54940},{"from":"2026-10-26T07:58:06.462Z","to":"2026-10-26T07:59:53.591Z","seconds":107}]}

Two failures from the mutation runs:

status guard removed, confirm race:   Expected length: 1   Received length: 2
design lock removed, lock-held test:  Expected: "rejected"  Received: "fulfilled"

What I did not verify

I did not run the poller on Compute under load, and I did not measure how late the emails are.
I have not fixed clock skew between the app server and the database. sendAt is written from the app clock and compared with the database clock, so skew changes the grace period silently.
There is no automatic retry for failed emails.
I did not port locks and transactions to Prisma 8. They stay on Prisma 7.

Takeaways

Let the database decide the winner. Do not let a timer or a clock decide it.
Break your code on purpose. A test that also passes without the guard proves nothing.
A free-running race loop can pass for the wrong reason. Force the interleaving.
Scale to zero changes what a timer inside your server means. Check it before you design around it.

I'm Mhamd, a backend engineer working with NestJS and TypeScript. Portfolio: mhamd021.github.io · GitHub: Mhamd021

A week of green runs, and the pipeline was waiting for me

Tue, 27 Oct 2026 16:02:11 +0000

On 1 September at 10:57 UTC the monthly ingest ran. It opened issue number 3 of the magazine, picked the vertical from the source registry, wrote 19 candidate sources into Postgres with stage = 'discovery', and opened a GitHub issue telling me it was my turn.

Every morning after that, the daily job ran and succeeded. Eight consecutive green runs, from the first of the month to the eighth. The article count for September stayed at zero the whole time.

The gate was doing its job

The daily job does not decide anything by itself. It asks the database which stage to run, and the answer depends on a view:

SELECT id, issue_id, source_url, tier, independent
FROM signals
WHERE stage = 'verify' AND (tier = 1 OR tier = 2 AND independent);

Nothing satisfies that view until I sit in Supabase Studio and tag the sources by hand. Tier 1 is a primary source, a regulator or the organisation that ran the thing. Tier 2 counts only when it is independent of whoever it describes, which rules out a consultancy writing up its own project as a case study.

That judgement is mine on purpose. It is the editorial bar of the magazine, and I did not want a model deciding what counts as proof of anything.

So the decision function found a draft issue with no verified signals, returned niente, and printed its reason to stderr:

advance: signal in attesa di verifica in Studio

Then it exited zero, which is correct. There was nothing for it to do.

What my alarms actually watch

I have three, and I was fairly pleased with them.

A cron that fails opens a GitHub issue on its own. A cron that never starts is caught by a daily watchman that asks the GitHub API when each schedule last fired, because a schedule that GitHub has quietly disabled produces no failure to report. And because a watchman inside the same failure domain is not a watchman, that job also checks in with a Sentry monitor that alarms on silence from outside GitHub.

Between them they cover a job that breaks and a job that never runs. Neither of those happened. What happened was a job that ran, succeeded, and did nothing, for seven days, because it was waiting for a person who had not noticed.

The information was already there

The part I find annoying is that nothing was hidden. The reason is written every single morning, in the run log, in Italian, on stderr. The job knows exactly why it is idle and says so out loud.

What it does not know is how long it has been saying it. niente on the second of the month and niente on the eighth are the same string, and the only thing that separates them is a week of my attention going somewhere else. No alarm reads that log line, and no counter turns it into a duration.

Green means the machine is healthy. I had been reading it as "the work is moving", and those are different claims.

How it ended

On the eighth I opened Studio and tagged 11 of the 19 sources: two from the regulator as tier 1, nine analyst reports as tier 2 and independent. I left out a vendor case study, which fails the independence test by construction, and seven security bulletins whose scraped body is only a title.

After that the pipeline did the rest in a few minutes. Generate wrote the piece from eight sources, embed rebuilt the chunks, export produced the two markdown files, and the judge on the pull request failed me for weak attribution, which is a story for another post.

The September issue went out a week late. Nothing had broken, and every run had been green.

So I do not need a fourth alarm. I need the daily job to count the mornings it has spent answering niente, and to say that number where I actually look.

I built a Supabase-style query client that talks directly to Postgres

Wed, 14 Oct 2026 22:57:08 +0000

I like the Supabase JS client. .from('todos').select('id, task').eq('done', false) reads like what I mean, and { data, error } means I'm not wrapping every query in try/catch.

What I don't always want is the stuff around it: a project URL, an anon key, and a PostgREST service sitting between my Node server and a Postgres I already control.

So I built pg-fluent: the postgrest-js query builder, executed directly against PostgreSQL through node-postgres.

import { createClient } from 'pg-fluent'

const db = createClient({ connectionString: process.env.DATABASE_URL })

const { data, error } = await db
  .from('todos')
  .select('id, task, done')
  .eq('done', false)
  .order('id')
  .limit(10)

If you've used supabase-js, that's already familiar. The difference is there's no HTTP request on the other end, just one parameterized SQL statement over a normal Postgres connection.

npm install pg-fluent

Repo: https://github.com/gerald-mathew/pg-fluent

Who it's for

People running long-lived servers (Express, Fastify, a VPS, containers) who like the Supabase query API but want to connect straight to Postgres, or who are moving off Supabase's hosted stack and don't want to rewrite every query. It's server-side only, more on that below.

How it works

I didn't reimplement the builder. pg-fluent uses the real @supabase/postgrest-js and hands it a custom fetch. When the builder "sends" a request, pg-fluent catches it, parses the URL, headers and body the way PostgREST would, plans one parameterized SQL statement, runs it through pg, and returns a response postgrest-js unpacks as if a server had answered.

Results are aggregated to JSON inside Postgres, so you get the same shape as Supabase. Embeds like select('id, title, author:users(name)') become subqueries that aggregate to JSON, with relationships resolved from the live schema (cached, with auto-refresh).

Since I'm intercepting what the builder emits, an upstream change could quietly change the SQL. So the dependency is pinned to an exact version, and docs/UPSTREAM.md records every bump with the date and reason. A bump that changes observable SQL or results is at least a minor release.

What you get beyond the basics

The whole postgrest-js surface works: filters, or()/not()/filter(), JSON paths like meta->>author, range(), single()/maybeSingle(), count, upsert, rpc, csv(), explain(), and embeds (many-to-one, one-to-many, many-to-many, !inner, aliases, spread). Then the things that only make sense with a direct connection:

See the SQL without running it:

const { text, values } = await db.from('todos').select('id').eq('done', false).toSQL()

Transactions with savepoints:

const user = await db.transaction(async (trx) => {
  const { data, error } = await trx.from('users').insert({ name: 'G' }).select().single()
  if (error) throw error
  await trx.from('profiles').insert({ user_id: data.id })
  return data
}, { isolationLevel: 'read committed' })

Raw SQL, safe by default. The tagged template always binds interpolations as parameters. The only way to run SQL text you built yourself is db.unsafe(), and the name is deliberate.

const { data } = await db.sql`select * from users where id = ${id}`

Streaming, COPY, LISTEN/NOTIFY:

for await (const row of db.from('events').select('*').stream()) {
  await handle(row)
}

await db.copyFrom('events', rows, { columns: ['id', 'kind'], format: 'csv' })

const sub = await db.listen('orders', (payload) => console.log(payload))

The listener gets its own non-pooled connection and reconnects with backoff.

Named connections and read replicas. Reads can go to replicas, while writes, RPC, transactions and LISTEN always hit the primary. Add .primary() when you need read-your-writes.

Type generation:

npx pg-fluent gen-types --connection "$DATABASE_URL" --schema public --out ./database.types.ts

The output is a Database type compatible with postgrest-js, so createClient<Database>(...) just works.

There are also logger and onQuery hooks and optional OpenTelemetry spans if @opentelemetry/api is installed. Parameter values are never logged unless you opt in.

Guardrails by default

Unfiltered UPDATE/DELETE is refused with PGFLUENT_UNFILTERED_MUTATION. Opt in with .allowUnfiltered() or a client option.
Reads are capped: maxRows defaults to 1000 when you set no limit or range.
Statements time out: statementTimeoutMs defaults to 30 seconds.
Retries are read-only. Transient connection failures retry with backoff for reads, and writes are never retried.

How I tested it

I didn't want to say "drop-in" on a hunch, so:

Differential tests against a real PostgREST server. The same requests go through both, and the payloads must match. One difference is intentional and documented: a reserved parameter name like columns or select used as a filter is rejected with PGFLUENT_INVALID_QUERY. PostgREST silently ignores it, which turns a scoped read into a full-table read.
Property-based injection tests with fast-check. Hostile strings go in as filter values, the SQL text must match the baseline, the values must land in parameters only, and a canary table must survive.
A real CI matrix: Node 22 and 24 against Postgres 15, 16, 17 and 18.

That's 200+ tests. There's also a benchmark harness against raw pg, but I'm not publishing performance claims. Network and pooling change the numbers too much, so it only reports methodology and variance for you to run yourself.

The RLS caveat (please read)

pg-fluent runs statements as the connection's own role, like psql or any other driver. It doesn't set request.jwt.claims and doesn't emulate Supabase's anon and authenticated roles.

So RLS policies written for Supabase's roles don't automatically protect you here. Connect as the table owner or a BYPASSRLS role and RLS is skipped entirely. Use a least-privilege role, keep this on the server, and never let database credentials reach a browser. For per-user enforcement, connect as a non-owner role and set SET ROLE and the JWT claims yourself. If you're migrating from supabase-js, treat this as step zero.

What's missing or rough

It's 0.2.x, so, honestly:

Embedding views isn't supported yet (you can read them, just not embed them).
No auth, storage or functions. It's a database client, not a BaaS SDK.
LISTEN/NOTIFY doesn't work through transaction-mode poolers like PgBouncer. Use session-mode or a direct connection.
Replica reads are asynchronous, and there's no lag measurement in v1.
Requires Node 22+ and Postgres 15 to 18.

Try it, break it

If you have a Supabase-style codebase and a Postgres you control, point pg-fluent at a test database and tell me what breaks. Issues and PRs are welcome at https://github.com/gerald-mathew/pg-fluent, and security reports go through the private channel in SECURITY.md.

I especially want to know which postgrest-js queries behave differently from what you expect. Real queries find things my fixtures won't.

Kafka Just Learned to Be a Queue. Postgres Has Been One All Along.

Fri, 09 Oct 2026 16:28:36 +0000

Originally published on DEV.to.

On February 17, 2026, Apache Kafka 4.2 shipped with this line in the release announcement: "Kafka Queues (Share Groups) is now production-ready."

That sentence quietly reopens one of backend engineering's oldest arguments. For years, "should my job queue be Postgres or Kafka?" had a lazy answer from both camps. The Kafka camp said Kafka scales. The Postgres camp said Kafka is not a queue: your consumer count is capped by your partition count, there is no per-message ack, and one poison message blocks everything behind it in its partition.

The second argument just lost most of its teeth. So this post rebuilds the comparison from primary sources and two small demos you can run in a few minutes: a Postgres SKIP LOCKED queue in Python, and a Kafka share group in Go.

The thesis

If a job is born inside a database transaction, Postgres is still the better queue in 2026.

Kafka share groups are real, well designed, and production-ready on the broker. But they answer a different question: "I already run Kafka, and now I also need queue semantics." They do not answer "I need a queue." Those two questions have different winners, and a team that does not already run Kafka would be adopting a whole platform to get one feature.

The antithesis, at full strength

Here is the best case against that thesis. It is a good one.

KIP-932 adds a new kind of group, the share group, and with it:

Consumers beyond partition count. "The number of consumers in a share group can exceed the number of partitions in a topic." The old trick of over-partitioning just to get parallelism goes away.
Per-record acknowledgement. A consumer can accept, release (retry), or reject (unprocessable) each record. Kafka 4.2 added a fourth option, RENEW, to extend the lock for slow work (KIP-1222).
Delivery counting with a poison-message limit. After the configured number of attempts (default 5, per the 4.3 broker configs), a record is archived instead of redelivered forever.
No queue depth limit and replay. In the KIP's own words: "queues done in a Kafka way, with no maximum queue depth and the ability to reset to a specific time for point-in-time recovery."
Many readers, one log. Several share groups can consume the same topic independently, next to ordinary consumer groups.

And the Postgres side has a well-known way to rot: MVCC. In the demo below, a single idle session holding an old snapshot made the median claim latency go from about 0.27 ms to about 15 ms on the same queue. That is not a theoretical concern. It is one forgotten BEGIN away.

So the antithesis is: Kafka now does queues properly, scales past one box, keeps history, and does not degrade because someone left a psql session open. The answer to that lives in the failure modes on both sides.

What actually shipped (facts, as of October 5, 2026)

	Kafka share groups (KIP-932)	Postgres `FOR UPDATE SKIP LOCKED` queue
Unit of work	Record in a topic partition	Row in a table
Claim	Broker-side acquisition lock, default 30 s (`share.record.lock.duration.ms`)	Row lock held by your transaction
Ack	ACCEPT / RELEASE / REJECT / RENEW	`COMMIT` (and delete or update the row)
Crash behavior	Lock expires (or the share session closes), record becomes available again	Transaction aborts, lock is released, row is visible again
Delivery guarantee	At-least-once (KIP-932)	At-least-once for external side effects; atomic with any writes in the same database
Ordering	Not guaranteed across batches, especially on redelivery	Whatever your `ORDER BY` gives you, best effort under concurrency
Poison messages	Delivery count limit, default 5	You write it (an `attempts` column)
Dead-letter queue	KIP-1191 listed as complete for 4.4.0, which is not released yet	You write it (a table)

Client support matters more than broker support, because your code talks to a client:

Java: KafkaShareConsumer is the reference client (javadoc).
Go: franz-go added "full support for KIP-932 share groups" in v1.21.0 (changelog). The demo below uses v1.22.1.
Python and C: librdkafka and confluent-kafka-python 2.15.0 (June 30, 2026) ship the share consumer as Preview, with an explicit note that it "should not be used in production environments" (confluent-kafka-python v2.15.0, librdkafka v2.15.0). Confluent's own GA post says non-Java client support is "targeted for the second half of 2026" (Confluent blog).

If your workers are Python on the mainstream Confluent client, Kafka queues are explicitly not for production today. That alone settles the question for a lot of data teams.

The Postgres queue is one SQL statement

Here is the entire claim step:

WITH next AS (
    SELECT id FROM jobs
    ORDER BY id
    LIMIT 50
    FOR UPDATE SKIP LOCKED
)
DELETE FROM jobs j USING next
WHERE j.id = next.id
RETURNING j.id, j.payload;

Run it inside a transaction, do the work, write the result, COMMIT. The commit is the ack. If the worker dies before committing, Postgres rolls the transaction back, the rows were never deleted, and another worker picks them up. That is the same contract as a share group's acquisition lock, except the "lock timeout" is "your connection died."

The Postgres docs are blunt about what SKIP LOCKED is for: "Skipping locked rows provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table" (SELECT docs).

Demo A: does SKIP LOCKED actually matter?

Eight worker threads drain 20,000 jobs in batches of 50. Each worker writes a row into a done table in the same transaction as the dequeue. The script asserts zero duplicates and zero lost jobs after every run. Each script execution does three alternating runs per cell and reports the median; the table shows the range of those medians across four executions:

Simulated work per batch (lock held)	`FOR UPDATE SKIP LOCKED`	plain `FOR UPDATE`
0 ms	14,400 to 17,600 jobs/s	13,200 to 32,800 jobs/s (about 32,000 in 3 of 4)
5 ms	14,000 to 17,600 jobs/s	7,100 to 7,200 jobs/s

Duplicates: 0. Lost: 0. In every run, for both variants.

The first row is the surprise: with near-zero lock hold time, plain FOR UPDATE was usually about twice as fast. A plausible explanation (interpretation, not measured): each SKIP LOCKED claim steps over the rows the other seven workers hold, while a blocked worker waits very briefly and gets its rows. The second row is the one that matters in production. Once workers hold the claim while doing real work, the blocking version collapses to about 7,000 jobs/s because everyone queues behind the head of the table, and SKIP LOCKED comes out 2 to 2.5 times ahead. Both variants lost nothing. SKIP LOCKED is about throughput under real work, not correctness.

One subtle thing the script surfaced: with SKIP LOCKED, a worker can get zero rows while jobs still exist, because everything left is claimed by someone else. It happened between 2 and 36 times per run. Treat an empty claim as "back off and poll again," never as "the queue is empty, exit."

Demo B: kill a worker mid-batch

A session claims 10 of 100 jobs and never commits. While it holds them, other sessions can claim 90. After pg_terminate_backend() on that session (the database-side equivalent of kill -9 on the worker), all 100 are claimable again. No lease column, no reaper job, as long as jobs are short (more on that below).

The real advantage: the job and the data commit together

This is the part Kafka cannot match without extra machinery. If your API inserts a user and enqueues "send welcome email" in the same transaction, the job exists if and only if the user exists. River, a Go job queue backed by Postgres, makes this its core argument (River docs): with a separate queue store you either enqueue after commit and lose jobs on a crash, or enqueue before commit and have workers race a row they cannot see yet.

The mainstream already voted on this. Solid Queue "is configured by default in new Rails 8 applications" and "leverages the FOR UPDATE SKIP LOCKED clause" (Solid Queue README).

Where the Postgres queue dies: one old snapshot

A queue table is the worst-case MVCC workload: every row is inserted once and deleted once. Postgres does not remove a deleted row immediately; the routine vacuuming docs explain that "the row version must not be deleted while it is still potentially visible to other transactions." If anything holds an old snapshot, vacuum cannot clean, and the dead rows pile up right where your claim query starts scanning.

Demo C: churn 200,000 jobs, then measure

The script pushes 200,000 jobs through the queue, enqueues 5,000 fresh ones, runs VACUUM (VERBOSE), and times 200 claims of 10 jobs each. Then it repeats the whole thing with one extra session sitting idle in transaction in REPEATABLE READ, which is exactly what a forgotten report query or a paused migration looks like.

	Median claim latency	`VACUUM VERBOSE` says
No old snapshot	0.269 ms / 0.272 ms	40,293 / 37,680 removed, 0 dead but not yet removable
One idle `REPEATABLE READ` session	15.754 ms / 14.961 ms	0 removed, 200,000 dead but not yet removable

(Two executions of the script, shown as first / second. In the no-snapshot case autovacuum had already cleaned most of the dead rows before the manual VACUUM ran.)

Same table, same query, same hardware. Roughly 55 to 58 times slower per claim, and the dead rows keep accumulating with every job processed while that session lives. Nothing errors. Your queue just gets slower until someone notices.

Things that hold old snapshots in real systems, all documented:

Long-running or idle-in-transaction sessions (vacuum docs tell you to find them via pg_stat_activity and backend_xmin).
Old replication slots (same page).
A hot standby with hot_standby_feedback on, which "can cause database bloat on the primary for some workloads" (replication config docs).

Mitigations that actually work:

Set idle_in_transaction_session_timeout for application and human roles, and transaction_timeout (added in PostgreSQL 17) where it is safe. Both are documented in client connection defaults, including the warning to be careful with connection poolers.
Alert on "dead but not yet removable" and on the age of the oldest backend_xmin, not just on queue depth.
Keep analytics off the primary. An extra test on the same box: an old snapshot held in a different database on the same cluster did not block cleanup of the queue table (50,000 removed), while the same snapshot in the same database blocked all of it (50,000 dead but not yet removable). That is a real escape hatch, but it has a price: a queue in its own database can no longer commit atomically with your application tables, which was the main reason to use Postgres in the first place.

The trap: your worker can be the long transaction

The claim pattern above holds the transaction open while the job runs. For a 50 ms job that is fine. For a 5 minute job, the worker itself is the old transaction. A quick check on the same box: a worker sitting idle in transaction after its DELETE ... RETURNING (plain READ COMMITTED, nothing exotic) left 50,000 freshly deleted rows in another table of the same database "dead but not yet removable." So long jobs need a lease instead: claim and commit in a short transaction, work outside any transaction, then ack only if you still own the lease.

-- claim: short transaction, commit right away
UPDATE jobs
SET locked_until = now() + interval '5 minutes',
    lease_token  = gen_random_uuid(),
    attempts     = attempts + 1
WHERE id IN (
    SELECT id FROM jobs
    WHERE locked_until IS NULL OR locked_until < now()
    ORDER BY id
    LIMIT 10
    FOR UPDATE SKIP LOCKED
)
RETURNING id, payload, lease_token, attempts;

-- ack: only the current lease holder can delete the job
DELETE FROM jobs WHERE id = $1 AND lease_token = $2;

Tested on the box: when a lease expired and a second worker re-claimed the job, the first worker's late ack deleted 0 rows and the second worker's ack deleted 1. Notice what you just built: an acquisition lock with a timeout and a delivery count. That is a share group, in SQL. You also gave up the "job commits with the data" property for the work itself, so this path needs idempotent jobs, exactly like Kafka.

A note on LISTEN/NOTIFY

If you use LISTEN/NOTIFY to wake workers, know the documented edges (NOTIFY docs): notifications are delivered only when the sending transaction commits, payloads must be shorter than 8000 bytes by default, and the notification queue (8 GB in a standard install) cannot be cleaned while a listening session sits in a long transaction. If that queue fills, "transactions calling NOTIFY will fail at commit." Use NOTIFY as a doorbell, never as the queue itself. PostgreSQL 19, with GA scheduled for October 29, 2026 (release dates), lists an improvement to "only wake up backends that are listening to specified notifications" in its draft release notes.

These are less about slow decay and more about semantics you have to design around.

1. The lock timeout is a duplicate-work timer. The default acquisition lock is 30 seconds. If your job takes 45, the broker hands the record to another consumer while the first one is still working. Use RENEW for long jobs, and make processing idempotent anyway. The franz-go kgo docs put it plainly: "Share group consumers are at-least-once regardless: any TCP hiccup can cause re-delivery, so user processing must be idempotent."

2. Ordering is gone. KIP-932: "The records in a share-partition can be delivered out of order to a consumer, in particular when redeliveries occur." If you need per-key ordering, you want a classic consumer group, not a share group.

3. A new share group starts at the end. share.auto.offset.reset defaults to latest (group configs). Create a share group on a topic that already has a backlog and, unless you set earliest, your workers will skip all of it.

4. Size-based retention can delete work you have not done. From the KIP: retention based on size "does potentially silently remove records that were eligible for delivery." A Postgres table does not delete your unprocessed jobs because the disk got full.

5. The unit of sharing is the record batch. The KIP says it directly: "the natural unit of sharing is the record batch." In the Go demo below, four consumers share one partition. With default fetch settings, one consumer got 949 of 1,000 jobs and two got nothing. With ShareMaxRecordsStrict() (the client side of the strict mode added by KIP-1206 in 4.2), the split was 246 / 251 / 252 / 251. Caveat: this ran against kfake, franz-go's in-process fake broker, not a real cluster, so treat the exact split as illustrative. The behavior matches what the KIP describes.

6. No atomicity with your database. Writing a row to Postgres and producing to Kafka are two systems. The standard fix is the transactional outbox: write the event to an outbox table in the same transaction, then let CDC (for example the Debezium outbox event router) publish it. Notice what that means: the reliable way to feed a Kafka queue from an OLTP app starts with a Postgres table.

7. In-flight work is capped per partition. group.share.partition.max.record.locks defaults to 2000 per share-partition (4.3 broker configs). Slow consumers holding locks can stall fetching on that partition until locks are acked or expire. And the delivery count "cannot be relied upon to be precise in all situations" (KIP-932), so do not build billing logic on it.

The core loop, using franz-go v1.22.1:

cl, err := kgo.NewClient(
    kgo.SeedBrokers(addrs...),
    kgo.ConsumeTopics("jobs"),
    kgo.ShareGroup("email-workers"),
    kgo.ShareMaxRecords(25),
    kgo.ShareMaxRecordsStrict(), // without this one member may get a whole record batch
)
if err != nil {
    panic(err)
}
defer cl.Close()

for {
    pollCtx, cancel := context.WithTimeout(ctx, time.Second)
    fetches := cl.PollFetches(pollCtx)
    cancel()
    fetches.EachRecord(func(r *kgo.Record) {
        if err := sendEmail(r.Value); err != nil {
            r.Ack(kgo.AckRelease) // transient: try again, delivery count goes up
            return
        }
        r.Ack(kgo.AckAccept)
    })
}

In the full demo, every job whose number ends in 0 is released on its first delivery to simulate a transient failure. Three runs in a row: 1,000 of 1,000 jobs accepted, 100 on a second delivery, 0 accepted twice, four consumers on one partition. The headline feature works.

The decision rule

Put it on one line first:

Tasks that belong to a transaction go in the database that owns the transaction. Facts that many systems read go in a log. Share groups let the log also hand out tasks, which is great once you already have the log.

Then the checklist.

Choose a Postgres queue when:

The job is created in the same transaction as the business data it depends on.
Your workers are in Python, or any language without a production share consumer yet.
Your measured peak fits your Postgres with your payloads. Demo A did 14,000 to 17,600 tiny jobs/s with SKIP LOCKED on 8 vCPUs; River claims "tens of thousands of jobs per second." Measure, do not inherit someone's number.
Jobs are short enough to run inside the claim transaction, or you use leases for the long ones.
You can enforce idle_in_transaction_session_timeout, keep analytics off the primary, and alert on dead-but-not-removable rows.

Choose Kafka share groups when:

Kafka is already the system of record for these events, and the queue is one more reader.
Several independent consumers need the same stream, or you need replay and reset-to-timestamp.
Your workers are on Java or Go (franz-go), processing is idempotent, and ordering does not matter.
You are fine designing around lock timeouts, retention, and share.auto.offset.reset.

Use both when the job starts in your database but the work fans out to other teams: outbox table in Postgres, CDC into Kafka, share groups downstream.

Red flags that you picked wrong: you are adding an outbox just to feed a Kafka queue that only one service reads (you wanted Postgres), or your Postgres queue needs a second database to stay fast (you are halfway to wanting a log).

Run it yourself

Hardware and versions for every number above: an 8 vCPU, 15 GB RAM Linux VM (kernel 6.12, Debian 13), PostgreSQL 17.11 with default settings except max_connections=200, client and server on the same machine over a Unix socket, Python 3.13.5 with psycopg 3.3.6, Go 1.26.0 with franz-go v1.22.1 and kfake. These are toy payloads on one box. They show mechanisms and ratios, not capacity for your system.

{% details pg_queue_demo.py (Python, needs a throwaway Postgres) %}

"""
Postgres-as-a-queue demo: SKIP LOCKED vs blocking FOR UPDATE, crash recovery, and the MVCC failure mode.
Needs a THROWAWAY Postgres (it drops tables named jobs and done).
  pip install "psycopg[binary]"
  DSN="host=/tmp port=54329 user=postgres dbname=postgres" python pg_queue_demo.py
"""
import os, time, threading, statistics
import psycopg

DSN = os.environ.get("DSN", "host=/tmp port=54329 user=postgres dbname=postgres")

CLAIM_SKIP = """
WITH next AS (
    SELECT id FROM jobs
    ORDER BY id
    LIMIT %(n)s
    FOR UPDATE SKIP LOCKED
)
DELETE FROM jobs j USING next
WHERE j.id = next.id
RETURNING j.id, j.payload
"""
CLAIM_BLOCKING = CLAIM_SKIP.replace("FOR UPDATE SKIP LOCKED", "FOR UPDATE")


def setup(conn):
    conn.execute("DROP TABLE IF EXISTS jobs, done")
    conn.execute("CREATE TABLE jobs (id bigserial PRIMARY KEY, payload jsonb NOT NULL)")
    # 'done' stands in for a side effect that lives in the SAME database as the queue
    conn.execute("CREATE TABLE done (job_id bigint NOT NULL, worker int NOT NULL)")
    conn.commit()


def enqueue(conn, n):
    conn.execute("INSERT INTO jobs (payload) SELECT jsonb_build_object('n', g) "
                 "FROM generate_series(1, %s) g", (n,))
    conn.commit()


def worker(wid, claim_sql, batch, stats, empties, work_ms):
    with psycopg.connect(DSN) as conn:
        while True:
            with conn.transaction():
                rows = conn.execute(claim_sql, {"n": batch}).fetchall()
                if rows:
                    if work_ms:
                        time.sleep(work_ms / 1000)  # "work" done while holding the claim
                    with conn.cursor() as cur:      # result commits atomically with the dequeue
                        cur.executemany("INSERT INTO done (job_id, worker) VALUES (%s, %s)",
                                        [(r[0], wid) for r in rows])
            if rows:
                stats[wid] += len(rows)
                continue
            # An empty claim does not always mean an empty queue (see the blocking variant).
            left = conn.execute("SELECT count(*) FROM jobs").fetchone()[0]
            conn.commit()
            if left == 0:
                return
            empties[wid] += 1


def run_workers(claim_sql, workers, batch, work_ms=0.0):
    stats, empties = [0] * workers, [0] * workers
    ts = [threading.Thread(target=worker, args=(i, claim_sql, batch, stats, empties, work_ms))
          for i in range(workers)]
    t0 = time.perf_counter()
    for t in ts:
        t.start()
    for t in ts:
        t.join()
    return time.perf_counter() - t0, sum(empties)


def experiment_throughput(n=20_000, workers=8, batch=50, repeats=3):
    for work_ms in (0.0, 5.0):
        print(f"\n== A. {workers} workers drain {n} jobs, batch={batch}, "
              f"simulated work per batch={work_ms} ms, {repeats} alternating runs")
        results = {"FOR UPDATE SKIP LOCKED": [], "FOR UPDATE (blocking)": []}
        for _ in range(repeats):
            for label, sql in (("FOR UPDATE SKIP LOCKED", CLAIM_SKIP),
                               ("FOR UPDATE (blocking)", CLAIM_BLOCKING)):
                with psycopg.connect(DSN) as conn:
                    setup(conn)
                    enqueue(conn, n)
                secs, empties = run_workers(sql, workers, batch, work_ms)
                with psycopg.connect(DSN) as conn:
                    total, distinct = conn.execute(
                        "SELECT count(*), count(DISTINCT job_id) FROM done").fetchone()
                    left = conn.execute("SELECT count(*) FROM jobs").fetchone()[0]
                assert total == distinct == n and left == 0, (total, distinct, left)
                results[label].append((n / secs, empties))
        for label, rs in results.items():
            print(f"{label:24s} median {statistics.median(r[0] for r in rs):7.0f} jobs/s | "
                  f"empty claims while work remained, per run: {[r[1] for r in rs]} | "
                  f"duplicates 0, lost 0")


def experiment_crash(n=100):
    print("\n== B. a worker dies mid-transaction")
    with psycopg.connect(DSN) as conn:
        setup(conn)
        enqueue(conn, n)
    victim = psycopg.connect(DSN)
    victim.execute(CLAIM_SKIP, {"n": 10}).fetchall()  # claimed 10 jobs, never commits
    with psycopg.connect(DSN, autocommit=True) as admin:
        q = "SELECT count(*) FROM (SELECT id FROM jobs FOR UPDATE SKIP LOCKED) s"
        print(f"victim holds its batch: claimable by others = {admin.execute(q).fetchone()[0]}")
        admin.execute("SELECT pg_terminate_backend(%s)", (victim.info.backend_pid,))
        time.sleep(0.5)
        print(f"victim session killed:  claimable by others = {admin.execute(q).fetchone()[0]}")
    try:
        victim.close()
    except Exception:
        pass


def median_claim_ms(rounds=200):
    lat = []
    with psycopg.connect(DSN) as conn:
        for _ in range(rounds):
            t0 = time.perf_counter()
            with conn.transaction():
                conn.execute(CLAIM_SKIP, {"n": 10}).fetchall()
            lat.append((time.perf_counter() - t0) * 1000)
    return statistics.median(lat)


def experiment_mvcc(churn=200_000):
    print(f"\n== C. churn {churn} jobs through the queue, then time claims, "
          f"with and without one old snapshot held open")
    for hold in (False, True):
        with psycopg.connect(DSN) as conn:
            setup(conn)
        holder = None
        if hold:
            # stands in for a long report query, a stuck migration, or an idle-in-transaction session
            holder = psycopg.connect(DSN, autocommit=True)
            holder.execute("BEGIN ISOLATION LEVEL REPEATABLE READ")
            holder.execute("SELECT 1").fetchone()  # first statement takes the snapshot
            with psycopg.connect(DSN, autocommit=True) as c2:
                st, xmin = c2.execute("SELECT state, backend_xmin FROM pg_stat_activity "
                                      "WHERE pid = %s", (holder.info.backend_pid,)).fetchone()
            print(f"   holder session: state={st!r}, backend_xmin={xmin}")
        with psycopg.connect(DSN) as conn:
            enqueue(conn, churn)
        run_workers(CLAIM_SKIP, 8, 100)            # drain everything, leaving dead rows behind
        with psycopg.connect(DSN) as conn:
            enqueue(conn, 5_000)                   # fresh work arrives behind the dead rows
        notes = []
        with psycopg.connect(DSN, autocommit=True) as conn:
            conn.add_notice_handler(lambda d: notes.append(d.message_primary or ""))
            conn.execute("VACUUM (VERBOSE) jobs")  # give vacuum every chance
        tuples_line = next((l.strip() for n in notes for l in n.splitlines() if "tuples:" in l), "")
        med = median_claim_ms()
        with psycopg.connect(DSN) as conn:
            size = conn.execute("SELECT pg_size_pretty(pg_total_relation_size('jobs'))").fetchone()[0]
        print(f"old snapshot held={str(hold):5s} median claim latency={med:8.3f} ms  "
              f"jobs table+index={size}\n   VACUUM VERBOSE: {tuples_line}")
        if holder:
            holder.execute("ROLLBACK")
            holder.close()


if __name__ == "__main__":
    with psycopg.connect(DSN) as c:
        print(c.execute("SELECT version()").fetchone()[0])
    experiment_throughput()
    experiment_crash()
    experiment_mvcc()

{% enddetails %}

{% details main.go (Go, franz-go share group against kfake) %}

// Share groups (KIP-932) in Go with franz-go: 4 consumers on ONE partition,
// per-record ack, and a transient failure that gets redelivered.
// Runs against kfake, franz-go's in-process fake broker, so no real cluster is needed.
package main

import (
"context"
"fmt"
"strconv"
"sync"
"time"

"github.com/twmb/franz-go/pkg/kfake"
"github.com/twmb/franz-go/pkg/kgo"
)

const (
topic   = "jobs"
group   = "email-workers"
total   = 1000
members = 4
)

func main() {
c, err := kfake.NewCluster(kfake.NumBrokers(1), kfake.SeedTopics(1, topic)) // ONE partition
if err != nil {
panic(err)
}
defer c.Close()
// Default is "latest": a brand-new share group would skip everything already in the topic.
c.SetGroupConfigs(group, map[string]string{"share.auto.offset.reset": "earliest"})

producer, err := kgo.NewClient(kgo.SeedBrokers(c.ListenAddrs()...), kgo.DefaultProduceTopic(topic))
if err != nil {
panic(err)
}
for i := 0; i < total; i++ {
producer.Produce(context.Background(), kgo.StringRecord(strconv.Itoa(i)), nil)
}
if err := producer.Flush(context.Background()); err != nil {
panic(err)
}
producer.Close()

var (
mu          sync.Mutex
accepted    = map[string]int{} // job -> times accepted
perMember   = make([]int, members)
redelivered int
)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

var wg sync.WaitGroup
for m := 0; m < members; m++ {
wg.Add(1)
go func(m int) {
defer wg.Done()
cl, err := kgo.NewClient(
kgo.SeedBrokers(c.ListenAddrs()...),
kgo.ConsumeTopics(topic),
kgo.ShareGroup(group),
kgo.ShareMaxRecords(25),
kgo.ShareMaxRecordsStrict(), // without this the broker may hand one member a whole record batch
)
if err != nil {
panic(err)
}
defer cl.Close()
for {
mu.Lock()
done := len(accepted) == total
mu.Unlock()
if done || ctx.Err() != nil {
_ = cl.FlushAcks(context.Background())
return
}
pollCtx, pollCancel := context.WithTimeout(ctx, time.Second)
fetches := cl.PollFetches(pollCtx)
pollCancel()
fetches.EachRecord(func(r *kgo.Record) {
n, _ := strconv.Atoi(string(r.Value))
if n%10 == 0 && r.DeliveryCount() == 1 {
r.Ack(kgo.AckRelease) // transient failure: give it back for another attempt
return
}
time.Sleep(2 * time.Millisecond) // pretend to send the email
r.Ack(kgo.AckAccept)
mu.Lock()
accepted[string(r.Value)]++
perMember[m]++
if r.DeliveryCount() > 1 {
redelivered++
}
mu.Unlock()
})
}
}(m)
}
wg.Wait()

dups := 0
for _, k := range accepted {
if k > 1 {
dups += k - 1
}
}
fmt.Printf("partitions=1 consumers=%d distinct jobs accepted=%d/%d\n", members, len(accepted), total)
fmt.Printf("accepted per consumer=%v\n", perMember)
fmt.Printf("accepted on a 2nd+ delivery (after RELEASE)=%d, accepted more than once=%d\n", redelivered, dups)
}

{% enddetails %}

For the Go demo: go mod init sharedemo && go get github.com/twmb/franz-go@v1.22.1 github.com/twmb/franz-go/pkg/kfake@v0.0.0-20260927204940-b5a45ccfdf7e && go run . (the kfake module requires Go 1.26, which the go command downloads automatically if your toolchain is older).

Takeaway

Kafka 4.2 fixed the strongest technical objection to "Kafka as a queue." It did not change where your jobs are born. If they are born in a Postgres transaction, keep them there, and spend your effort on the one thing that kills Postgres queues: old snapshots. When the events already live in Kafka and many teams read them, share groups are now a legitimate queue, on Java and Go today, and on Python once the client leaves Preview.

Sources

Apache Kafka 4.2.0 release announcement (Feb 17, 2026): https://kafka.apache.org/blog/2026/02/17/apache-kafka-4.2.0-release-announcement/
Apache Kafka 4.3.0 release announcement (May 22, 2026): https://kafka.apache.org/blog/2026/05/22/apache-kafka-4.3.0-release-announcement/
KIP-932, Queues for Kafka: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932: Queues for Kafka
Kafka 4.3 broker configs: https://kafka.apache.org/43/configuration/broker-configs/
Kafka 4.3 group configs: https://kafka.apache.org/43/configuration/group-configs/
KafkaShareConsumer javadoc (4.2): https://kafka.apache.org/42/javadoc/org/apache/kafka/clients/consumer/KafkaShareConsumer.html
Kafka 4.4.0 release plan (KIP-1191 status): https://cwiki.apache.org/confluence/spaces/KAFKA/pages/429064575/Release Plan 4.4.0
Confluent, Queues for Kafka GA: https://www.confluent.io/blog/kafka-queue-semantics-share-consumer-ga/
franz-go changelog: https://github.com/twmb/franz-go/blob/master/CHANGELOG.md
confluent-kafka-python v2.15.0: https://github.com/confluentinc/confluent-kafka-python/releases/tag/v2.15.0
librdkafka v2.15.0: https://github.com/confluentinc/librdkafka/releases/tag/v2.15.0
PostgreSQL SELECT (locking clause, SKIP LOCKED): https://www.postgresql.org/docs/current/sql-select.html
PostgreSQL routine vacuuming: https://www.postgresql.org/docs/current/routine-vacuuming.html
PostgreSQL NOTIFY: https://www.postgresql.org/docs/current/sql-notify.html
PostgreSQL client connection defaults (timeouts): https://www.postgresql.org/docs/current/runtime-config-client.html
PostgreSQL replication config (hot_standby_feedback): https://www.postgresql.org/docs/current/runtime-config-replication.html
PostgreSQL 17 release notes (transaction_timeout): https://www.postgresql.org/docs/17/release-17.html
PostgreSQL 19 draft release notes: https://www.postgresql.org/docs/19/release-19.html
PostgreSQL 19 RC1 and GA dates: https://www.postgresql.org/message-id/915517c4-d0ee-49f9-bbeb-c683ef2b9d3f@postgresql.org
River, transactional enqueueing: https://riverqueue.com/docs/transactional-enqueueing
Solid Queue README: https://github.com/rails/solid_queue
Debezium outbox event router: https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html

Building a Resilient Job Queue in Go: Postgres, Worker Pool and Graceful Shutdown

Thu, 01 Oct 2026 18:08:31 +0000

Building a Resilient Job Queue in Go: Postgres, Worker Pool and Graceful Shutdown

I am Pires Aurélio, Go Backend Engineer from Luanda, Angola. I build production systems in public.

For two years I built scrapers and simple REST APIs. I got tired of tutorials that only show Hello World. No jobs, no retries, no real production problems.

So I built a real one: A Production-Ready Background Job Queue in Go.

This is the production deep-dive companion to my DEV article. All links are in the first comment.

WHAT WE ARE BUILDING

A REST API where you POST a task, it goes to Postgres, and Go workers process it in background with retries and graceful shutdown. Like Sidekiq and Celery, but in pure Go.

Stack: Go + Gin + Postgres + Goroutines + Docker
Features: Clean Architecture, Postgres as queue, Worker Pool with 5 Goroutines, Graceful Shutdown, Exponential Backoff, Supervisor pattern.

ARCHITECTURE

Client -> POST /jobs -> API Gin -> Postgres jobs table -> Worker Pool -> Process -> Update Status

Why Postgres and not Redis? One less service and ACID guarantees. This proves you understand transactions and row-level locking.

THE PRODUCTION SCHEMA

Table jobs has: id UUID, payload JSONB, status pending processing done failed, attempts int, max_attempts int, last_error text, created_at and next_retry_at.

The most important part is the index on status and next_retry_at where status is pending. This makes the picker O log n with 1 million rows. See Image 1.

THE SAFE PICKER

The mistake in tutorials is SELECT then UPDATE. Two workers get the same job.

The solution is an atomic UPDATE where id comes from a sub-select with row-level locking and SKIP LOCKED. Only one worker gets a job. This is what separates toy from production. See Image 1.

WORKER POOL

Start with concurrency 5. For each worker start a goroutine. Loop select on ctx Done for shutdown, default picks a job and processes with retry.

If fails, increment attempts. If attempts >= max, mark failed. Else backoff = 2 power attempts seconds and set next_retry_at.

See Image 2.

SUPERVISOR - NEVER LOSE A JOB

If server crashes while processing, job stays processing forever.

On startup: Reset all jobs where status is processing and updated_at older than 5 minutes back to pending. This is what real companies do.

GRACEFUL SHUTDOWN - THE INTERVIEW QUESTION

Use signal NotifyContext for Interrupt and SIGTERM. Start workers with that context. Start HTTP server in goroutine. Block on ctx Done. On signal, create timeout context 10 seconds and call server Shutdown. Workers stop via ctx Done.

See Image 3. This proves you know context propagation.

BENCHMARK

Python + Celery + Redis: 120 jobs per sec, 180 MB RAM
Go + Postgres: 1850 jobs per sec, 18 MB RAM

15x faster, 10x less memory.

WHAT RECRUITERS SEE

You understand concurrency, you understand databases beyond SELECT, you think about failure, you can build infrastructure not just CRUD.

Full code with metrics is coming to my GitHub next week. Links in first comment.

Built from Luanda, Angola.

Horizontally Scaling PostgreSQL: Database Sharding with Citus

Sat, 26 Sep 2026 10:35:35 +0000

A practical guide to scaling your database infrastructure horizontally. Learn how to shard your data, optimize query routing, and build a distributed multi-tenant architecture using Postgres, Citus and Docker.

Prerequisites

What do you need to follow along?

A working Laptop/PC?
Working knowledge of SQL or PostgreSQL.
To have at least eaten something today (preferably swallow related).

Database sharding is a strategy used to solve scalability issues in applications that have a massive amount of data. It involves splitting data into smaller chunks called shards. Each shard is stored on a separate database server instance, effectively spreading the data and the workload across multiple machines.

Why Shard Data?

As an application grows, the number of users and the amount of data it stores increase over time. At a certain limit, a single database server will become a bottleneck. The application slows down and affects the user experience.

Instead of upgrading a single server with more CPU or RAM, which will eventually hit a hardware ceiling, sharding lets you add more servers to your cluster. This allows applications to handle nearly infinite growth in data volume and user traffic.

Understanding Shards and Shard Keys

Before we start sharding data, let's first understand what shards and shard keys are.

Shards

The partitioned data chunks are called logical shards. The machine that stores the logical shard is called a physical shard or database node. A physical shard can contain multiple logical shards.

Shard Key

A shard key is used to determine how to partition the dataset. A shard key is a specific column in your dataset that determines which rows of data group together to form a shard.

Sharding Methods

There are a few ways to shard data depending on the application requirements.

Key-based sharding (hash sharding): Hash sharding uses a mathematical formula called a hash function. The hash function takes your shard key and generates a hash value, this hash value determines the exact physical shard where the row will be stored.
Range-based sharding: Range-based sharding, or dynamic sharding, splits database rows based on a range of values. Then the database designer assigns a shard key to the respective range.
Directory-based sharding: Directory sharding uses a lookup table to match database information to the corresponding physical shard. A lookup table is like a table on a spreadsheet that links a database column to a shard key.
Geo sharding: Geo sharding splits and stores database information according to geographical location.

For this guide, we are focusing exclusively on Hash Sharding. This is the industry standard for multi-tenant applications like the e-commerce platform we will be building in this tutorial.

Choosing Your Shard Key

If you pick the wrong shard key, your database will still crash under load. When choosing a shard key, you must ensure the data is distributed evenly across your nodes:

Cardinality: This refers to the number of unique values a key can have. You want a key with high cardinality like a User ID, not low cardinality like a Gender or State column, so the data can be split into many small, even chunks.
Frequency: You must avoid keys where one specific value appears too often. For example, if you shard by a City column and 80% of your users live in Ikeja, the Ikeja node will be overloaded while the others sit idle.
Monotonic change: This is the rate of change of the shard key. A monotonically increasing or decreasing shard key results in unbalanced shards. For example, consider a feedback database that is split into three different physical shards as follows:
- Shard A stores feedback from customers who have made 0 to 10 purchases.
- Shard B stores feedback from customers who have made 11 to 20 purchases.
- Shard C stores feedback from customers who have made 21 or more purchases.
  As the business grows, customers will make 21 or more purchases. The application stores their feedback in Shard C. This results in an unbalanced shard because Shard C contains more feedback records than the other shards.

Is Sharding Actually For You?

While sharding solves massive scale requirements, it introduces significant technical and operational tradeoffs. You should not shard your database unless a single, heavily provisioned Postgres instance can no longer handle your traffic.

If you do decide to shard your database, here are some of the biggest challenges you will face:

Data hotspots: Even with a good hash function, uneven traffic can create hotspots where one server becomes overloaded by a small group of highly active users.
Performance regression: Queries that do not use the shard key require the system to scatter and gather across every server. Cross-shard joins are computationally expensive because the application must pull and merge data from multiple physical locations over the network.
Operational complexity: Managing a sharded environment increases the difficulty of routine tasks. Schema updates, backups, and point-in-time recovery must be coordinated across multiple independent instances.

How Can We Shard Our Postgres Database: Introducing Citus

Citus is an open-source extension that transforms standard PostgreSQL into a distributed database. Instead of being limited to a single server, Citus allows your database to scale horizontally across multiple machines.

How Citus Works

The Coordinator Node: When you ask your database a question, you talk to the coordinator. The coordinator directs your request to the right servers, gathers the answers, and combines them for your application.
Worker Nodes: Citus breaks down large tables into smaller, manageable chunks (shards) and spreads them across these worker servers. They do the heavy lifting of storing the data and running the queries.

Start Early, Scale Later

Migrating a traditional, monolithic PostgreSQL database to a distributed, sharded database later in its life is incredibly difficult. You have to change your schema, rewrite complex queries, and alter how your application connects to the database.

To avoid this nightmare, you can start using Citus on a single node from day one. You design your tables with distribution keys immediately, and your application code is written to query a distributed database from the start.

The Transition: When your data outgrows that single machine, you won't have to rewrite your application. You simply spin up new worker machines, and Citus moves the existing shards over to the new hardware in the background while the application keeps running.

Running Citus Locally

We will use Docker to run Citus locally. Why use Docker instead of adding the extension directly to your host machine's Postgres installation?

Simulating a distributed database means running multiple Postgres instances simultaneously. Doing this natively forces you to manually manage conflicting ports and multiple data directories. Docker Compose solves this by orchestrating the entire cluster's networking with a single command, giving you a clean state that can be torn down and rebuilt in seconds.

Single Node vs Multi-Node Setup

While it is possible to run a single node Citus instance using a simple docker run command:

docker run -d --name citus_standalone -p 5432:5432 -e POSTGRES_PASSWORD=postgres citusdata/citus:14.1.0

This guide will focus exclusively on setting up a Multi-Node Cluster. If you are interested in exploring the single node setup further, you can check out the official Citus documentation here.

For our distributed, multi-node cluster, we will use the following Docker Compose configuration:

services:
  master:
    container_name: "${PROJECT_NAME:-citus}_master"
    image: "citusdata/citus:14.1.0"
    ports: ["${COORDINATOR_EXTERNAL_PORT:-5433}:5432"]
    labels: ["com.citusdata.role=Master"]
    environment: &AUTH
      POSTGRES_USER: "${POSTGRES_USER:-postgres}"
      POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
      PGUSER: "${POSTGRES_USER:-postgres}"
      PGPASSWORD: "${POSTGRES_PASSWORD}"
      POSTGRES_HOST_AUTH_METHOD: "${POSTGRES_HOST_AUTH_METHOD:-trust}"
  worker:
    image: "citusdata/citus:14.1.0"
    labels: ["com.citusdata.role=Worker"]
    environment: *AUTH

Citus has a built-in initialization script in its official Docker image. When launched via Docker Compose with these specific labels like com.citusdata.role, the coordinator automatically detects and registers the workers for you.

We can build and start the cluster in the background with:

docker compose up -d

We can also scale up dynamically. If you want to spin up a second or third worker machine on the fly, you do not even need to edit the compose file. Just run:

docker compose up -d --scale worker=3

Where 3 is the total number of workers you want to have running.

Simply spinning up the worker containers is not enough. After adding new worker nodes, we must register them with the coordinator so Citus knows they are active and ready to hold data. To do this, we will first connect to the interactive Postgres terminal on the master node:

docker compose exec master psql -U postgres

Now we can register our workers:

SELECT citus_add_node('db-sharding-worker-1', 5432);
SELECT citus_add_node('db-sharding-worker-2', 5432);
SELECT citus_add_node('db-sharding-worker-3', 5432);

So, if you scale up again to say 5 workers, you will have to run this command 2 more times to register the 2 new workers.

Notice the db-sharding- prefix in the hostnames. This is necessary because Docker Compose prefixes your project folder name to the container's network alias.

A Real World Scenario

In order to showcase the functionality of this database sharding setup, let's try to work with a real world scenario.

Consider an e-commerce multi-tenant application. It serves multiple independent companies that all share the same database infrastructure. To scale this effectively, we need to use a hash-based distribution with a well-chosen shard key. Let's assume by design, every tenant has a shared company_id, we will use that as our shard key.

By using company_id as the shard key across all operational tables, Citus guarantees that all data belonging to a specific company (like its users, its products, and its orders) is physically co-located on the exact same worker node.

Now that our workers are registered, we can create our core tables directly in the master node. Citus will automatically handle the underlying sharding for us.

Please note that the following schema is highly simplified to focus on the core concepts of sharding.

Run the following SQL commands in your coordinator's postgres interactive terminal:

-- 1. Create Tenant Table
CREATE TABLE companies (
    company_id INT PRIMARY KEY,
    company_name TEXT NOT NULL
);

-- 2. Create Tenant-Scoped Orders Table
CREATE TABLE orders (
    company_id INT NOT NULL,
    order_id INT NOT NULL,
    user_id INT NOT NULL,
    status_id INT NOT NULL, -- References our global order status table
    order_amount NUMERIC(10,2) NOT NULL,
    PRIMARY KEY (company_id, order_id)
);

-- 3. Create Global Order Reference Table
CREATE TABLE order_statuses (
    status_id INT PRIMARY KEY,
    status_name TEXT NOT NULL
);

-- 4. Create Tenant-Scoped Users Table
CREATE TABLE users (
    company_id INT NOT NULL,
    user_id INT NOT NULL,
    username TEXT NOT NULL,
    PRIMARY KEY (company_id, user_id)
);

-- 5. Create Tenant-Scoped Products Table
CREATE TABLE products (
    company_id INT NOT NULL,
    product_id INT NOT NULL,
    product_name TEXT NOT NULL,
    price NUMERIC(10,2) NOT NULL,
    PRIMARY KEY (company_id, product_id)
);

Telling Citus How to Shard and Replicate

Now that we have created our core tables, we have to tell Citus how to shard and replicate them across our worker nodes.

Reference Tables

First, we will set order_statuses as a reference table. We do this using create_reference_table.

Why do we create a reference table? A reference table is designed for relatively small, slowly-changing data like user roles, country codes, or order statuses.

When you mark a table as a reference table, Citus physically replicates a full copy of that entire table to every single worker node. This bypasses network overhead completely during JOIN operations. Because the worker node already has the data locally, it does not need to fetch it from another node to complete a query.

Distributed Tables

Next, we need to distribute our large tenant-data tables. We do this using create_distributed_table.

This function is the core of Citus's sharding mechanism. It tells the coordinator to partition the table based on a specific column (our shard key, company_id), and spread those partitions across the available worker nodes.

Because we use company_id across all these tables, any query filtering by a specific company will be routed directly to a single node, making it incredibly fast.

-- Replicate order_statuses to ALL nodes
SELECT create_reference_table('order_statuses');

-- Shard data based on the Hash of company_id
SELECT create_distributed_table('companies', 'company_id');
SELECT create_distributed_table('users', 'company_id');
SELECT create_distributed_table('products', 'company_id');
SELECT create_distributed_table('orders', 'company_id');

Multi-Tenant Seed Data

Now we will populate our cluster with some dummy companies along with some global order status entries, orders, and tenant users.

-- Global Order Lookups
INSERT INTO order_statuses VALUES (1, 'Pending'), (2, 'Shipped'), (3, 'Delivered');

-- Company Tenants
INSERT INTO companies VALUES (1, 'EbaMart'), (2, 'FufuHub'), (3, 'GymGear');

-- Multi-Tenant Users scattered across companies
INSERT INTO users (company_id, user_id, username) VALUES
(1, 101, 'Samuel'),   -- EbaMart Customer
(2, 201, 'Folashade'),-- FufuHub Customer
(1, 102, 'Umoh'),     -- EbaMart Customer
(3, 301, 'Oladejo'),  -- GymGear Customer
(2, 202, 'BigSam');   -- FufuHub Customer

-- Orders
INSERT INTO orders (company_id, order_id, user_id, status_id, order_amount) VALUES
(1, 5001, 101, 1, 150.00),  -- EbaMart order: Pending
(1, 5002, 102, 3, 45.00),   -- EbaMart order: Delivered
(2, 6001, 201, 2, 99.00),   -- FufuHub order: Shipped
(2, 6002, 202, 1, 250.00);  -- FufuHub order: Pending

Verify Layout and Node Placement

We can now verify the layout and node placement to see where they are added to and display it by running:

SELECT
    u.company_id,
    u.user_id,
    u.username,
    s.shardid,
    s.nodename
FROM users u
JOIN citus_shards s
  ON s.shardid = get_shard_id_for_distribution_column('users', u.company_id)
ORDER BY u.company_id;

Output:

company_id	user_id	username	shardid	nodename
1	101	Samuel	102042	db-sharding-worker-2
1	102	Umoh	102042	db-sharding-worker-2
2	201	Folashade	102065	db-sharding-worker-1
2	202	BigSam	102065	db-sharding-worker-1
3	301	Oladejo	102056	db-sharding-worker-1

As you can see, all data for company_id = 1 sits neatly on db-sharding-worker-2, while company_id = 2 sits on db-sharding-worker-1. This guarantees that queries for a single tenant will only hit one node.

We can also inspect the query plan for joining distributed tables with reference tables:

EXPLAIN (VERBOSE, COSTS OFF)
SELECT o.order_id, o.order_amount, s.status_name
FROM orders o
JOIN order_statuses s ON o.status_id = s.status_id
WHERE o.company_id = 1;

Output:

                                      QUERY PLAN
---------------------------------------------------------------------
---------------------------------------------------------------------
---------------------------------------------------------------------
------------------------
 Custom Scan (Citus Adaptive)
   Output: remote_scan.order_id, remote_scan.order_amount, remote_scan.status_name
   Task Count: 1
   Tasks Shown: All
   ->  Task
         Query: SELECT o.order_id, o.order_amount, s.status_name FROM (public.orders_102106 o JOIN public.order_statuses_102008 s ON ((o.status_id OPERATOR(pg_catalog.=) s.status_id))) WHERE (o.company_id OPERATOR(pg_catalog.=) 1)
         Node: host=db-sharding-worker-2 port=5432 dbname=postgres
         ->  Hash Join
               Output: o.order_id, o.order_amount, s.status_name
               Hash Cond: (s.status_id = o.status_id)
               ->  Seq Scan on public.order_statuses_102008 s
                     Output: s.status_id, s.status_name
               ->  Hash
                     Output: o.order_id, o.order_amount, o.status_id
                     ->  Bitmap Heap Scan on public.orders_102106 o
                           Output: o.order_id, o.order_amount, o.status_id
                           Recheck Cond: (o.company_id = 1)
                           ->  Bitmap Index Scan on orders_pkey_102106
                                 Index Cond: (o.company_id = 1)
(19 rows)

The query plan shows a Custom Scan (Citus Adaptive) delegating the query to the specific worker node holding company_id = 1.

We can check the active worker nodes with:

SELECT * FROM master_get_active_worker_nodes();

Output:

node_name	node_port
db-sharding-worker-3	5432
db-sharding-worker-1	5432
db-sharding-worker-2	5432

Rebalancing the Cluster

Anytime you dynamically scale the cluster and register a new worker node, you must run the Citus Shard Rebalancer:

SELECT citus_rebalance_start();

When new nodes are added, the coordinator does not automatically move existing data onto them. We have to explicitly balance the load. The rebalancer handles this by dynamically moving a subset of your tenants' data over to the new worker servers in the background, without taking the database offline.

Production Setup

The Docker Compose setup is perfect for local testing and development. However, running a distributed database in production across multiple physical servers or virtual machines requires a slightly different approach.

We must first change how network addressing and orchestration work. In a local setup, Docker Compose handles networking automatically because everything shares a single machine loopback. On production VMs, each container must be bound to the public or private IP address of its physical host machine, and you must enforce strict firewall security.

1. Spin up your VMs

Spin up your virtual machines on your preferred cloud provider. Assuming we provision 4 VMs, our network layout will look something like this:

VM 1 (Coordinator Node): Private IP 10.0.0.10
VM 2 (Worker Node 1): Private IP 10.0.0.11
VM 3 (Worker Node 2): Private IP 10.0.0.12
VM 4 (Worker Node 3): Private IP 10.0.0.13

I used random private IPs, but you should substitute them with your real VMs' private IPs.

2. Install Docker

Next, you will need to install Docker and Docker Compose on every VM. The installation commands will depend on your specific Linux distribution, so refer to the official Docker documentation for your OS.

You will no longer use a single master compose file for the cluster. Instead, you deploy an isolated, optimized compose layout directly on each corresponding VM instance.

3. The Coordinator Instance (VM 1)

Create a docker-compose.yml file specifically tailored to map the Postgres engine port directly out to the host VM network interface:

services:
  master:
    container_name: "citus_coordinator"
    image: "citusdata/citus:14.1.0"
    ports:
      - "5432:5432"
    restart: always
    environment:
      POSTGRES_USER: "citus_admin"
      POSTGRES_PASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      PGUSER: "citus_admin"
      PGPASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      POSTGRES_HOST_AUTH_METHOD: "scram-sha-256"
    volumes:
      - coordinator_data:/var/lib/postgresql/data

volumes:
  coordinator_data:

4. The Worker Instances (VMs 2, 3, and 4)

Every worker VM gets its own standalone compose file. Notice we expose port 5432 on the workers too, because the coordinator node needs a direct network pipe to talk to them:

services:
  worker:
    container_name: "citus_worker"
    image: "citusdata/citus:14.1.0"
    ports:
      - "5432:5432"
    restart: always
    environment:
      POSTGRES_USER: "citus_admin"
      POSTGRES_PASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      PGUSER: "citus_admin"
      PGPASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      POSTGRES_HOST_AUTH_METHOD: "scram-sha-256"
    volumes:
      - worker_data:/var/lib/postgresql/data

volumes:
  worker_data:

5. Configure Network Firewall Security

Since you are running across independent cloud instances, your cloud platform's Security Groups or Firewalls (like AWS Security Groups or ufw on Ubuntu) must be strictly configured to prevent public database scanning.

On the Coordinator VM (VM 1): Allow inbound TCP traffic on port 5432 from Anywhere (0.0.0.0/0) only if your external application servers need public access. If your application servers are in the same VPC, restrict inbound port 5432 access exclusively to your app servers' IP subnet range.
On the Worker VMs (VMs 2, 3, 4): Allow inbound TCP traffic on port 5432 EXCLUSIVELY from the Coordinator's Private IP (10.0.0.10). Block all other inbound network requests to the workers. They should never be exposed to the internet.

Next, log in to each VM and run the containers:

docker compose up -d

Once all containers are healthy, access the interactive postgres interface on VM 1 (the coordinator node):

docker compose exec master psql -U citus_admin

SELECT citus_add_node('10.0.0.11', 5432); -- Register Worker VM 2
SELECT citus_add_node('10.0.0.12', 5432); -- Register Worker VM 3
SELECT citus_add_node('10.0.0.13', 5432); -- Register Worker VM 4

Verify everything works:

SELECT * FROM master_get_active_worker_nodes();

6. Configure Production Password Authentication

Because we updated POSTGRES_HOST_AUTH_METHOD to scram-sha-256 for security, your instances will require passwords to authenticate cross-node.

We must add a Postgres password passfile on the coordinator container:

docker compose exec master bash -c "echo '*:*:*:citus_admin:YOUR_SUPER_SECURE_PRODUCTION_PASSWORD' >> /var/lib/postgresql/.pgpass && chmod 600 /var/lib/postgresql/.pgpass"

Surely you will know to replace YOUR_SUPER_SECURE_PRODUCTION_PASSWORD with your actual password.

Here is a detailed documentation of Citus multi-node installation and setup for reference: Citus Multi-Node Setup.

Adding Citus to an Existing Project

To add Citus to an already existing Postgres database project, the process follows these general steps:

Install the Extension: Install the Citus extension on your Postgres server. You will also need to add citus to the shared_preload_libraries in your postgresql.conf file and restart the database.
Create the Extension: Run CREATE EXTENSION citus; in your database.
Select a Shard Key: Identify the column that will serve as the shard key for your distributed tables, such as a company or tenant ID.
Convert Tables: Use create_distributed_table() to convert your existing tables into distributed tables. Use create_reference_table() for lookup tables that do not need to be sharded.
Migrate Data: If you already have data in the tables, it might need to be exported and loaded back in, depending on the volume of data and the specific requirements of the migration.

Conclusion

Database sharding is an incredibly powerful architectural pattern, but it comes with immense operational tradeoffs. By using tools like Citus, we can bridge the gap between standard relational SQL databases and massively distributed systems.

Whether you are starting a new multi-tenant application or scaling out an existing infrastructure, adopting a distributed mindset and orchestrating your local environments with tools like Citus and Docker Compose will save you countless hours of architectural nightmares down the road.

You can find the full Docker Compose source code for this tutorial on GitHub: umohsamuel/database-sharding.

References

Next.js Full-Stack Auth Template

Wed, 09 Sep 2026 05:55:26 +0000

Setting up a complete, production-grade auth flow shouldn't take days.

I built an open-source starter template combining Next.js, Better-Auth, Drizzle ORM, and PostgreSQL to streamline the entire setup.

What it includes:

Email/Password & Social OAuth (Google & GitHub)
PostgreSQL schema management & migrations with Drizzle ORM
Server-side route protection via Next.js layouts
Full client and server type safety

I've shared the entire codebase and published a comprehensive architectural breakdown explaining the full setup step-by-step.

Live Demo: website-link
GitHub Repository: repo-link
Full Step-by-Step Guide: article-link
Video Demo: video-link

I Moved My Next.js Dashboard Logic Into Postgres. My Frontend Got Boring (And That's the Point).

Tue, 25 Aug 2026 16:16:35 +0000

My dashboard had a useMemo doing arithmetic it had no business doing.

It was a Pokémon TCG Pocket collection tracker, but that part doesn't matter. What matters is that the home page needed to show three things: overall completion, completion per set, and which set you were closest to finishing. The way I'd built it, the browser was fetching every card and every owned record, then grinding through the math on each render to figure all of that out.

It worked. It also got slower and harder to read every time the data grew or I added a metric. So I moved the aggregation out of React and into Postgres, and the surprising result was that my frontend got boring. Fewer hooks, less state, almost nothing left to break.

That's the whole argument of this post: aggregation belongs in the database, and when you put it there, the React code that's left over is the kind of boring you actually want in the layer your users touch.

What "fetching everything into React" actually looks like

Here's the shape of the original dashboard. Load all the cards, load the user's owned rows, then derive everything on the client.

const [cards, setCards] = useState<Card[]>([]);
const [owned, setOwned] = useState<OwnedCard[]>([]);

useEffect(() => {
  (async () => {
    const { data: allCards } = await supabase.from("cards").select("*");
    const { data: ownedRows } = await supabase
      .from("user_cards")
      .select("card_id");
    setCards(allCards ?? []);
    setOwned(ownedRows ?? []);
  })();
}, []);

const ownedIds = useMemo(() => new Set(owned.map((o) => o.card_id)), [owned]);

const perSet = useMemo(() => {
  const groups: Record<string, { total: number; have: number }> = {};
  for (const card of cards) {
    const g = (groups[card.set_id] ??= { total: 0, have: 0 });
    g.total += 1;
    if (ownedIds.has(card.id)) g.have += 1;
  }
  return groups;
}, [cards, ownedIds]);

const overall = useMemo(() => {
  const total = cards.length;
  const have = cards.filter((c) => ownedIds.has(c.id)).length;
  return total ? have / total : 0;
}, [cards, ownedIds]);

It's not bad code. But look at what it's costing:

The payload grows with the collection. I'm shipping every card row to the browser just to count them. The user never sees most of that data. It exists only to be reduced to a few numbers.
The logic can't be tested without mounting a component. The "closest set to finishing" rule lives inside hooks, tangled up with fetching and state.
Every new metric is more client state. Want a "rarest card you're missing" stat? That's another loop, another useMemo, another thing that re-runs on render.

I was using React as a database query engine. It's not one.

Why this is the wrong layer

Counting, grouping, and computing ratios across rows is a set operation. It's the thing relational databases were built to do, indexed and compiled, sitting right next to the data.

When I do it in React instead, I pay for the same work twice. Once to serialize every row and send it over the wire, and again to recompute the aggregates in JavaScript on the client. The database already knew the answer was "you own 41 of 86 cards in this set." I just declined to ask it that question and asked for all 86 rows instead.

This isn't really a performance micro-optimization. It's a question of which layer owns which job. Aggregation is a database job. I'd put it in the wrong place.

Moving it into an RPC function

Supabase lets you call Postgres functions directly with rpc(), so I wrote one that returns exactly the rows the dashboard needs: one row per set, with the counts already computed.

create or replace function public.get_home_set_completion_rows()
returns table (
  set_id text,
  set_name text,
  set_slug text,
  release_date date,
  image_url text,
  data_version text,
  total_cards integer,
  owned_cards integer,
  missing_cards integer
)
language sql
stable
security invoker
set search_path = public
as \(
  select
    sets.id as set_id,
    sets.name as set_name,
    sets.slug as set_slug,
    sets.release_date,
    sets.image_url,
    sets.data_version,
    count(cards.id)::integer as total_cards,
    count(user_cards.card_id)::integer as owned_cards,
    (count(cards.id) - count(user_cards.card_id))::integer as missing_cards
  from public.sets
  left join public.cards
    on cards.set_id = sets.id
  left join public.user_cards
    on user_cards.card_id = cards.id
    and user_cards.user_id = auth.uid()
  group by
    sets.id,
    sets.name,
    sets.slug,
    sets.release_date,
    sets.image_url,
    sets.data_version
  order by
    sets.release_date desc nulls last,
    sets.id;
\);

The left join against user_cards filtered by auth.uid() is doing the per-user work right inside the query. Sets the user has zero cards from still show up with owned = 0, which is what I want.

And the call site collapses to this:

const { data } = await supabase.rpc("get_home_set_completion_rows");

That's it. No fetching everything, no client-side grouping, no Set of owned IDs. The component receives compact, purpose-built rows and maps over them. Overall completion is a sum of two columns. "Closest to finishing" is a sort. Both are now trivial because the hard part already happened in the database.

The React file went from a pile of hooks to a Server Component that calls one function and renders the result. That's the "my frontend got boring" moment, and deleting all that state felt great.

When a view beats a function (and vice versa)

I reached for an RPC function here because of that auth.uid() filter, since the result depends on who's asking. But it's worth knowing when a plain view is the better tool:

Use a view when the aggregation is stable, reusable, and doesn't need parameters. A view reads like a table, composes into other queries, and is the simplest thing that works. Card counts per set that aren't user-specific would be a great view.
Use a function when you need arguments (a user ID, a set filter, a scope toggle) or multi-step logic that a single select can't express cleanly.

One footgun worth flagging: both views and functions interact with row-level security, and security definer versus security invoker decides whose permissions the code runs under. A security definer function bypasses the caller's RLS, which is occasionally what you want and frequently a way to leak other users' data by accident. If you're using RLS to protect per-user rows, and you should be, make sure your function respects it rather than quietly stepping around it.

The objections worth taking seriously.

A few reasonable complaints, because this trade isn't free.

"Now my logic is split across two languages and harder to grep." True. I moved complexity, I didn't delete it. The aggregation now lives in SQL instead of TypeScript, so there's a context switch to read the whole story. My argument is just that it lands in the right place, with the database owning database work, and that the SQL version is shorter and more declarative than the loops it replaced.

"SQL is harder to test." I'd push back on this one. The function is deterministic: give it fixture rows and it produces the same counts every time, with no component to mount. And separating it out made the logic that correctly stayed in TypeScript easier to test too. My pack-recommendation engine, which sums pull probabilities for missing cards and ranks packs by expected value, is real application logic, not aggregation, so it belongs in app code. Once it wasn't tangled up with fetching and dashboard math, I could unit-test it on its own. The point isn't "move everything into Postgres." It's "move the aggregation, keep the decisions."

"This is just more infrastructure to manage." Fair, but it's version-controlled SQL living next to the rest of the schema, not a magic black box. It migrates like everything else.

The rule of thumb

If you take one thing from this post, take this:

Push aggregation and filtering down to the database. Keep decisions and presentation in the app.

Or, more bluntly: if React is looping to count things, that loop probably wants to be a GROUP BY.

The frontend got boring, with smaller payloads, fewer hooks, and less state to corrupt, and boring is exactly what you want from the layer your users actually touch. The interesting, opinionated, easy-to-get-wrong logic moved to where it could be expressed simply and tested in isolation.

The genuinely contested part is where you draw the line. I keep ranking and business decisions in app code and push counting and filtering into the database, but I know people who'd move more in either direction. Where do you draw it?

(Full code, if you want to see the whole thing wired together: mwiginton/pocketdex-tracker. Or try the live app: pocketdex-tracker.vercel.app.)

Directus + Coolify: Should You Decouple Postgres & Redis?

Thu, 06 Aug 2026 21:39:55 +0000

This is Part 2 of the Directus + Coolify series. If you're new here, start with "Secure Your VPS Before Hackers Do" and the first Directus + Coolify post — the bundled, single-Compose-file setup — before following along with this one.

Introduction

In the first method, we coupled all of the services into one stack using a single Docker Compose file. The network between all services was created automatically, and we didn't have to start them up individually — which removes the risk of a race condition if service startup isn't handled properly.

If you're running a single app, that's genuinely the recommended way to set up Directus on a Coolify-managed VPS. Going in, I assumed there were several good reasons to split the services apart instead — more control over backups, monitoring, restarts, that kind of thing. So before recommending decoupling, I actually tested each of those assumptions on a live Coolify instance.

Most of them turned out to be wrong.

Myth 1: Restarting Directus Restarts the Whole Stack

I expected that restarting Directus inside the bundled Compose file would restart Redis and Postgres along with it. It doesn't. Coolify lets you restart each service in the stack independently — Directus, Database, and Cache each get their own Restart button, right there in the same view. No decoupling needed for this one.

Myth 2: You Need a Separate Database Resource for S3 Backups

Same story. Even with Postgres bundled inside the Directus Compose file, Coolify still gives it its own dedicated Backups option, S3 included. This isn't a separate-resource-only feature.

Myth 3: Scheduled Tasks Require Separate Services

Also not true. Coolify exposes a Scheduled Tasks tab per service, even inside a single bundled stack — complete with a Container name dropdown letting you target the cron job at just the database, or just Directus, without splitting anything apart.

What Actually Holds Up

Two things survived testing.

First: metrics. This one's confirmed directly in Coolify's own documentation — CPU and memory metrics collection is explicitly not available for Docker Compose–based deployments. If you want to see per-container resource usage through Coolify's built-in monitoring, the service needs to be created as its own standalone resource, not bundled inside a Compose file. This is a real, documented limitation of the bundled approach.

Second: sharing a database across multiple apps. This one isn't a Coolify feature at all — it's just how Docker networking works. A database defined inside one app's Compose file lives on that stack's own private network by default. A second, completely separate application can't reach it without deliberately bridging the two networks. If you've got a desktop app and a mobile app that both need to talk to the same Postgres instance, that database needs to exist as its own standalone resource from the start — it can't stay tucked inside one app's Compose file.

So this really comes down to two reasons to decouple, not five — one a genuine Coolify limitation, the other a structural fact about Docker networking. Let's set both of those up properly.

Adding the Resources

Adding the PostgreSQL Database

Dashboard → Add Project
+ Add Resource
Databases → PostgreSQL → Supabase PostgreSQL (with extensions)
Change the name to something human-friendly
Copy your username and password and save them somewhere — you'll need them shortly
Click Save
Click Start, and wait for the database to spin up (this can take a little while)

Once it's up, the status should read "Running (Healthy)."

💡 Enable metrics while you're here. In the sidebar, go to Servers → localhost → Metrics, and enable metrics. Back in your project, under Databases, click your Postgres database, then Metrics — you should now see live CPU/memory usage for it. This is the exact capability that isn't available on a bundled Compose deployment.

Adding Redis Cache

Dashboard → + Add Resource (from the project itself)
Databases → Redis
Rename it to something more convenient
Copy the Redis connection URL — you'll need it shortly
Save, then Start

Once both are running, SSH into your VPS and run:

docker ps

to confirm both containers are up.

The Docker Compose Config for Directus

This is what goes into the empty Compose file for the Directus resource:

services:
  directus:
    image: 'directus/directus:12.2.0'
    ports:
      - '8055:8055'
    volumes:
      - './uploads:/directus/uploads'
      - './extensions:/directus/extensions'
    healthcheck:
      test:
        - CMD-SHELL
        - 'wget --spider -q http://127.0.0.1:8055/server/ping || exit 1'
      interval: 10s
      timeout: 5s
      retries: 5
      start_interval: 5s
      start_period: 30s
    environment:
      SECRET: secretstring
      MARKETPLACE_TRUST: all
      DB_CLIENT: pg
      DB_HOST:
      DB_PORT: '5432'
      DB_DATABASE: postgres
      DB_USER: postgres
      DB_PASSWORD:
      CACHE_ENABLED: 'true'
      CACHE_STORE: redis
      CACHE_AUTO_PURGE: 'true'
      REDIS:
      ADMIN_EMAIL: *Emails are not allowed*
      ADMIN_PASSWORD: '1234567890'
      CORS_ENABLED: 'true'
      CORS_ORIGIN: 'true'
      CORS_CREDENTIALS: 'true'
      PUBLIC_URL:

Notice the healthcheck already uses 127.0.0.1 instead of localhost — that's the fix from Part 1. Carrying it forward here saves you from hitting the exact same "unhealthy" bug all over again.

Adding Directus

Dashboard → + Add Resource (from the project itself)
Applications → Docker Compose Empty
Paste in the Compose configuration above
Click Save
Network → Connect To Predefined Network → check the box
Services → Directus service → Settings
Add your Directus subdomain — remember to use https:// (e.g. https://directus.yourdomain.com)
Save

Connecting the Services Together

Variable	What it is
`SECRET`	A long, unguessable random string
`DB_HOST`	The name of your Postgres container — run `docker ps` on your VPS to find it
`DB_DATABASE`	The name of the database on your Postgres server (Coolify's default is usually `postgres`, but confirm it against your Postgres service)
`DB_USER`	The database username, from the Postgres service you created earlier
`DB_PASSWORD`	The password from that same Postgres service
`REDIS`	The Redis connection URL from the Redis service you created earlier
`PUBLIC_URL`	Your Directus subdomain

⚠️ Redis URL gotcha: the connection URL follows the format redis://username:password@host:port. Coolify's generated URL includes the username you set when creating the Redis resource — in my case, that username was also redis, so the URL looked like redis://redis:somelongvariable.... Directus doesn't need the username here, just the password, so strip that segment out: redis://:somelongvariable.... If you used a different username when creating your Redis resource, remove that value instead — not literally the word "redis."

Setting Up Environment Variables

Rather than hardcoding any of this directly into the Compose file, move it into Directus's environment variables:

Dashboard → Projects → Services / Directus
Environment Variables
Click + Add, enter the variable name in all caps, and its value
Save
Repeat for: SECRET, DB_HOST, DB_DATABASE, DB_USER, DB_PASSWORD, ADMIN_EMAIL, ADMIN_PASSWORD, and REDIS

Updating the Compose File

Now reference those variables instead of the raw values:

SECRET: '${SECRET}'
DB_HOST: '${DB_HOST}'

...and so on for each variable. Then:

Save
Restart

Launching Directus

Paste your Directus subdomain into the browser, and you should land on your running Directus instance. You can also click Links on the Directus service, then click the subdomain — it'll take you straight to your Directus Studio login.

Same as the last video: sign up for your free license, which arrives by email, and paste it into your Directus instance to unlock everything.

Any questions or hit a different result testing any of these yourself? Drop it in the comments — I'm genuinely curious whether this holds up across different Coolify versions and setups.

🚀 UUIDv4 vs UUIDv7 in PostgreSQL

Fri, 17 Jul 2026 17:21:52 +0000

Why Switching UUID Versions Can Boost Performance at Scale

Many teams choose UUIDs as primary keys to support distributed systems and avoid ID collisions.
But not all UUIDs are equal.

Some teams have seen 10×–50× performance improvements simply by switching from UUIDv4 to UUIDv7.

Sounds surprising?
Let’s break it down - step by step - with no hand-waving.

🎯 What This Blog Will Help You Decide

By the end, you’ll understand:

What UUIDv4 and UUIDv7 actually are
How PostgreSQL indexes work internally
Why random IDs hurt performance
Why UUIDv7 scales dramatically better
When you should (and shouldn’t) switch
How to benchmark this yourself

1️⃣ Quick Refresher: What Is a UUID?

A UUID (Universally Unique Identifier) is a 128-bit value designed to be globally unique.

Why developers use UUIDs:

Safe for distributed systems
Can be generated client-side
No central ID generator
Avoids ID collisions across services

PostgreSQL stores UUIDs efficiently (16 bytes), so UUIDs themselves are not slow.

👉 The problem is the insertion pattern, not the type.

2️⃣ UUIDv4 - Random by Design

How UUIDv4 Works

UUIDv4 is purely random (122 random bits).

Example:

9f1c0d3a-4b6e-4b8f-8f91-17e0a0c9a721
e2a31f8b-91f2-4df7-98cb-6e9fcae721aa
0d2c5f99-5d8c-4b6d-8a0e-2dbbfa17d441

Key properties:

No ordering
No timestamp
Completely scattered values

This randomness is great for uniqueness
…but terrible for database indexes.

3️⃣ UUIDv7 - Time-Ordered UUIDs

UUIDv7 is a newer standard designed specifically for modern databases.

How UUIDv7 Works

UUIDv7 combines:

A timestamp (milliseconds) in the most significant bits
Random bits for uniqueness

Example (simplified):

018f3b3a-9b5c-7d12-bc01-9a2cfe123456
018f3b3a-9b5d-7f44-a231-acde441298ab
018f3b3a-9b5e-82a1-9d22-acde112341aa

Key properties:

Mostly increasing over time
Still globally unique
Still safe for distributed systems

👉 This ordering changes everything.

4️⃣ How PostgreSQL Stores and Indexes Data (Critical Part)

PostgreSQL Table Storage

Rows are stored in a heap
Order in heap is not guaranteed

PostgreSQL Indexes

Primary keys use B-tree indexes by default.

5️⃣ What Is a B-Tree (In Simple Terms)

A B-tree:

Stores keys in sorted order
Is optimized for:
- Sequential inserts
- Range scans
- Cache-friendly access

Think of it like:

A well-organized book where new pages are best added at the end.

6️⃣ Why UUIDv4 Hurts PostgreSQL Performance

With UUIDv4:

Each new ID belongs somewhere random in the index
PostgreSQL must:
- Find the correct position
- Split pages frequently
- Rebalance the tree
- Touch random memory pages

Consequences:

Heavy page splits
Index fragmentation
Poor cache locality
Increased disk I/O
Slower inserts
Slower reads
Slower vacuum

This gets worse as the table grows.

7️⃣ Why UUIDv7 Is Fast

With UUIDv7:

New rows are mostly appended
Inserts hit the rightmost leaf page
Minimal page splits
Excellent cache locality
Sequential disk writes

This is very similar to:

BIGSERIAL
IDENTITY
Snowflake-style IDs

👉 PostgreSQL loves this pattern.

8️⃣ Why Teams See 10×–50× Improvements

At scale (millions of rows):

Metric	UUIDv4	UUIDv7
Insert latency	High	Low
Index size	Large	Smaller
Cache efficiency	Poor	Excellent
Page splits	Frequent	Rare
Vacuum cost	High	Lower

⚠️ Important note:

50× is workload-dependent
Typical gains are 5×–20×
Still very significant

9️⃣ Hands-On Benchmark (You Can Try This)

Table Setup

CREATE TABLE test_uuid_v4 (
  id UUID PRIMARY KEY,
  data TEXT
);

CREATE TABLE test_uuid_v7 (
  id UUID PRIMARY KEY,
  data TEXT
);

Insert UUIDv4

INSERT INTO test_uuid_v4
SELECT gen_random_uuid(), 'data'
FROM generate_series(1, 1000000);

Insert UUIDv7 (Postgres 16+ or extension)

INSERT INTO test_uuid_v7
SELECT uuidv7(), 'data'
FROM generate_series(1, 1000000);

UUIDv7 generation requires Postgres 18 (uuidv7()) or an extension such as pg_uuidv7 in earlier versions.

Observe:

Insert time
Index size
CPU usage
Disk writes

You’ll see:

UUIDv7 inserts are dramatically smoother
Index size is smaller
Less I/O pressure

🔍 Why This Matters in Real Systems

Backend Impact

Faster writes
Better read performance
Healthier indexes
Lower infrastructure cost

Frontend Impact

IDs sort naturally by creation time
Easier pagination
Better caching behavior
Cleaner URLs

⚠️ When UUIDv7 Is NOT the Right Choice

Be honest and balanced:

If you need fully random IDs
If timestamp leakage is a concern
If your dataset is small
If sequential IDs are unacceptable

UUIDv7 is a performance trade-off, not magic.

🏁 Final Thoughts

This performance boost isn’t accidental — it’s how databases work.

The takeaway:

PostgreSQL B-trees love ordered inserts
UUIDv4 is random → bad at scale
UUIDv7 is time-ordered → excellent at scale
Schema design decisions matter more than hardware

Choosing the right ID strategy early can save years of performance tuning later.

💬 Have you used UUIDv4 in production and faced scaling issues?
Or already switched to UUIDv7?

Let’s discuss — real-world experiences help everyone.

Chai aur SQL — A Beginner's Journey into Databases

Fri, 03 Jul 2026 04:56:02 +0000

title: Chai aur SQL — A Beginner's Journey into Databases
published: true
canonical_url: https://blog.abdulrdeveloper.me/chai-aur-sql-a-beginner-s-journey-into-databases

tags: sql, postgresql, database, backend

SQL stands for Structured Query Language. It is the language we use to talk to relational databases like MySQL, PostgreSQL, and SQLite.

What is a Server and Where Does the Database Live?

Most beginners think the database is somewhere floating in the "cloud." It is not.

A database is software running on a server. A server is just a powerful computer that runs 24/7 inside a data center — like Amazon AWS, Google Cloud, or DigitalOcean. When that software saves your data, it writes it to actual files on the server's SSD or hard drive.

MySQL and PostgreSQL save data as .ibd files or WAL (Write-Ahead Log) files. MongoDB saves as .bson files. These files sit on disk just like any file on your laptop.

When you write this in your backend code:

const pool = new Pool({ host: 'localhost', port: 5432 })

You are telling your backend to open a connection to the port where the database is listening. That is it. It is a TCP connection — like a phone call between your application and the database.

Why Docker for PostgreSQL?

Instead of installing PostgreSQL directly on my system, I used Docker. Docker creates an isolated environment (called a container) so I do not mess up my laptop's setup.

Here is the command I used to create a PostgreSQL container with persistent data:

Mac / Linux / WSL:

docker run -d --name sql_class \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=sql_class_01_db \
  -p 5433:5432 \
  -v "$(pwd)/pgdata:/var/lib/postgresql/data" \
  postgres

Windows (PowerShell):

docker run -d --name sql_class -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=sql_class_01_db -p 5433:5432 -v "${PWD}/pgdata:/var/lib/postgresql/data" postgres

The -v flag maps a pgdata folder inside your current directory to the container's storage. So even if the container is deleted, your data stays on your laptop.

To check if the container is running:

docker ps

To remove the container:

docker rm -f sql_class

Why Learn Raw SQL When We Have Prisma and Drizzle?

This was my first question. Why learn raw SQL when Prisma can generate the queries for me?

Here is what I found out:

ORMs like Prisma and Drizzle translate your JavaScript/TypeScript code into SQL strings and send them to the database. If you do not know SQL, you do not know what Prisma is actually doing behind the scene.

The N+1 problem is a real thing. ORMs sometimes generate 100 queries instead of 1 optimized query. You will only catch this if you understand SQL.

For complex queries — joining 6 tables, generating financial reports — ORM syntax breaks down. Developers fall back to raw SQL.

And SQL has not changed in 50 years. Sequelize came and went. Prisma came. Drizzle came. SQL is still the same.

DDL — How You Create Tables

DDL stands for Data Definition Language. These are the commands that define the structure of your database — not the data, just the structure.

The four main DDL commands are CREATE, ALTER, DROP, and TRUNCATE.

Here is a real table I created while practicing:

CREATE TABLE students (
    student_id SERIAL PRIMARY KEY,
    first_name VARCHAR(50) NOT NULL,
    last_name VARCHAR(50),
    email VARCHAR(100) UNIQUE NOT NULL,
    phone_number CHAR(10) UNIQUE,
    age INT CHECK (age > 12),
    current_status VARCHAR(20) DEFAULT 'active'
        CHECK (current_status IN ('active', 'graduated', 'dropped_out', 'on_leave')),
    has_joined_masterji BOOLEAN DEFAULT FALSE,
    current_score NUMERIC(5, 2) CHECK (current_score >= 0 AND current_score <= 100),
    enrollment_date DATE DEFAULT CURRENT_DATE
);

SERIAL means auto-incrementing integer — PostgreSQL handles the ID for you. NOT NULL means the column cannot be empty. UNIQUE means no two rows can have the same value. CHECK is a custom condition — here age must be greater than 12. DEFAULT sets a value automatically if nothing is provided.

To add a new column to an existing table later:

ALTER TABLE students ADD COLUMN batch_name VARCHAR(50) DEFAULT 'Web Dev 2026';

SELECT Queries — Reading Your Data

SELECT is the command to read data from a table.

-- Get everything
SELECT * FROM ipl_players;

-- Get specific columns
SELECT name, team, auction_price_crores FROM ipl_players;

Filtering with WHERE:

-- Basic condition
SELECT * FROM ipl_players WHERE team = 'Mumbai Indians';

-- Multiple conditions
SELECT * FROM ipl_players WHERE role = 'All-Rounder' AND wickets_taken > 10;

-- Either condition
SELECT * FROM ipl_players WHERE team = 'CSK' OR team = 'RCB';

-- Range
SELECT * FROM ipl_players WHERE auction_price_crores BETWEEN 5 AND 12;

-- From a list
SELECT * FROM ipl_players WHERE team NOT IN ('Mumbai Indians', 'CSK', 'RCB');

-- Pattern matching — names starting with R
SELECT * FROM ipl_players WHERE name LIKE 'R%';

-- Check for empty values
SELECT * FROM ipl_players WHERE team IS NULL;

% means zero or more characters. _ means exactly one character. LIKE is case-sensitive. ILIKE is case-insensitive (PostgreSQL only).

Sorting and limiting:

-- Sort by price, highest first
SELECT name, auction_price_crores FROM ipl_players ORDER BY auction_price_crores DESC;

-- Top 3 results
SELECT name, auction_price_crores FROM ipl_players ORDER BY auction_price_crores DESC LIMIT 3;

-- Skip first 3, get next 3 (pagination)
SELECT name, auction_price_crores FROM ipl_players ORDER BY auction_price_crores DESC LIMIT 3 OFFSET 3;

DML — Insert, Update, Delete

DML stands for Data Manipulation Language. This is how you actually work with the data inside the tables.

INSERT:

-- Single row
INSERT INTO canteen_menu (item_name, category, price)
VALUES ('Vada Pav', 'Snacks', 15);

-- Multiple rows at once
INSERT INTO canteen_menu (item_name, category, price) VALUES
('Masala Chai', 'Beverages', 10),
('Samosa', 'Snacks', 12),
('Maggi', 'Snacks', 25);

UPDATE:

UPDATE canteen_menu
SET price = 20
WHERE item_name = 'Vada Pav';

DELETE:

DELETE FROM canteen_menu WHERE item_name = 'Cold Coffee';

Always use WHERE with UPDATE and DELETE. If you forget the WHERE clause, SQL will update or delete every single row in the table. I learned this the hard way in practice.

Aggregate Functions — COUNT, SUM, AVG, MAX, MIN

Aggregate functions take a whole column and return one number.

-- How many rows?
SELECT COUNT(*) FROM smart_watch_sales;

-- Total revenue
SELECT SUM(units_sold * price_per_unit) AS total_revenue FROM smart_watch_sales;

-- Average price
SELECT AVG(price_per_unit) AS avg_price FROM smart_watch_sales;

-- Cheapest and most expensive
SELECT MIN(price_per_unit) AS cheapest, MAX(price_per_unit) AS costliest FROM smart_watch_sales;

GROUP BY lets you split these calculations by category:

-- Total units sold per brand
SELECT brand, SUM(units_sold) AS total_units_sold
FROM smart_watch_sales
GROUP BY brand
ORDER BY total_units_sold DESC;

HAVING filters the groups after they are created. WHERE cannot do this because WHERE runs before grouping:

-- Only show brands that sold more than 20 units total
SELECT brand, SUM(units_sold) AS total_units
FROM smart_watch_sales
GROUP BY brand
HAVING SUM(units_sold) > 20;

Joins — Connecting Multiple Tables

This is where relational databases get powerful.

We keep data in separate tables to avoid duplication. Students in one table, internships in another. Joins let you combine them in one query.

Foreign Key is what connects the tables. The internships table has a student_id column that references the student_id in the students table:

CREATE TABLE internships (
    internship_id SERIAL PRIMARY KEY,
    student_id INT REFERENCES students(student_id) ON DELETE SET NULL,
    company_name VARCHAR(100),
    role VARCHAR(50),
    stipend INT,
    status VARCHAR(20)
);

ON DELETE SET NULL means if a student is deleted, the internship record stays but student_id becomes NULL. ON DELETE CASCADE would delete the internship too.

INNER JOIN — only matching rows:

SELECT s.name, s.branch, i.company_name, i.role
FROM students s
INNER JOIN internships i ON s.student_id = i.student_id;

Students without internships will not show up here.

LEFT JOIN — all students, matched or not:

SELECT
    s.name,
    s.branch,
    COALESCE(i.company_name, 'No Internship') AS company_name,
    COALESCE(i.stipend, 0) AS stipend
FROM students s
LEFT JOIN internships i ON s.student_id = i.student_id;

COALESCE replaces NULL with a default value. Students without internships will appear with 'No Internship' instead of NULL.

To find students who have NOT applied anywhere:

SELECT s.name, s.email
FROM students s
LEFT JOIN internships i ON s.student_id = i.student_id
WHERE i.internship_id IS NULL;

RIGHT JOIN returns all rows from the right table. In practice, developers rarely use RIGHT JOIN — they just flip the table order and use LEFT JOIN instead. It reads more naturally.

FULL OUTER JOIN — everything from both tables:

SELECT s.name AS student_name, i.company_name
FROM students s
FULL OUTER JOIN internships i ON s.student_id = i.student_id;

This shows all students and all internships. Where there is no match on either side, you get NULL.

Indexes — Making Queries Fast

I inserted 1,000,000 rows into a table to test this.

Without an index, when you search by name, PostgreSQL scans every single row from top to bottom. This is called a Sequential Scan.

EXPLAIN ANALYZE SELECT marks FROM marks WHERE name = '809E15792322';

EXPLAIN ANALYZE shows you the query plan and execution time. Before adding an index, my query took around 40-50 milliseconds on 1 million rows.

After creating an index:

CREATE INDEX idx_name ON marks (name);

The same query dropped to under 1 millisecond. PostgreSQL now uses a B-Tree structure to find the row directly instead of scanning everything.

A covering index goes one step further. If your query asks for marks along with name, you can store both in the index itself:

CREATE INDEX idx_name ON marks (name) INCLUDE (marks);

Now PostgreSQL does not even need to visit the main table. It gets everything from the index directly. This is called an Index Only Scan.

Transactions and ACID

A transaction is a way to group multiple SQL operations into one unit of work. Either all of them succeed, or none of them do.

The classic example is a bank transfer:

BEGIN;

    UPDATE accounts SET balance = balance - 500 WHERE owner = 'Shubham';
    UPDATE accounts SET balance = balance + 500 WHERE owner = 'Hitesh';

COMMIT;

If the server crashes between the two UPDATE statements, the database will rollback both changes. The money will not disappear from Shubham's account and fail to reach Hitesh.

If something goes wrong, you can manually undo everything:

BEGIN;
    UPDATE accounts SET balance = balance - 5000 WHERE owner = 'Shubham';
ROLLBACK;

ACID is the set of guarantees that make this work:

Atomicity — All or nothing. If Step 2 fails, Step 1 is also undone. I tested this by putting SELECT 1/0 (division by zero error) in the middle of a transaction. The whole thing rolled back.

Consistency — The database only moves from one valid state to another. I set CHECK (balance >= 0) on the accounts table. When I tried to deduct 5000 from an account with 1000, PostgreSQL threw an error and refused the operation completely.

Isolation — Uncommitted transactions are invisible to other connections. I opened two terminal windows to test this. Terminal 1 started a transaction and updated a balance but did not COMMIT. Terminal 2 could still see the old balance. Only after Terminal 1 ran COMMIT did Terminal 2 see the new number.

Durability — Once committed, data is permanent. PostgreSQL writes committed transactions to the Write-Ahead Log (WAL) on disk before returning a success response. Even if the server crashes one second after the COMMIT, the data will be there when the server restarts.

All the code from this blog is in my GitHub repo: github.com/abdulrdeveloper/Learn-SQL

You can find more of my work at abdulrdeveloper.me
Read more posts at blog.abdulrdeveloper.me

When Your Database Is On Fire at 2 AM, Ellie Shows Up

Tue, 16 Jun 2026 00:57:09 +0000

Most database problems don't announce themselves in advance. They show up at 2 AM, when systems are down and the clock is running. And the DBA who knew this environment best left the company six months ago.

That's the reality Dave Page, CTO of pgEdge, has been watching play out for nearly 30 years in the Postgres community. It's also the problem he set out to fix with the pgEdge AI DBA Workbench, which moved to general availability last week.

The Staffing Problem Is the Database Problem

Ask most database teams where their biggest pain point is and you might expect the answer to be query performance, schema design, or index bloat. Page has a different take.

"Honestly, in my experience, it's finding staff that are experienced enough," he said.

That's the context behind the Workbench. Teams are being asked to manage more Postgres infrastructure with fewer qualified people. And the tools they've had available were built for a simpler era — one where you set manual thresholds and hoped you caught problems before users did.

The Workbench gives teams Ellie, an AI agent built directly into the monitoring system. She doesn't replace your DBA. But she works 24/7, and when something goes wrong, she can investigate 10 to 100 times faster than a human working through the problem manually.

"When your system's down and you're losing a million bucks an hour, Ellie helps you reduce that time," Page said.

What Ellie Actually Sees

The Workbench collects continuous snapshots of Postgres system views and stores them historically. If you have extensions like pg_stat_statements installed, Ellie has access to that data too — aggregate query stats across databases and users. Add the System Stats extension and she can also see operating system-level data: memory usage, CPU, disk mounts.

On top of that, she has access to a live RAG database of current product documentation, so she isn't limited by whatever the underlying model's training cutoff happens to be.

Every session is isolated to the individual user, so Ellie is working within your user context and permissions. Whatever Postgres exposes through system views for your session, she can see and query.

The result is an agent with genuine situational awareness — not a chatbot taking guesses based on a prompt, but something that can actually see the current state of your systems and correlate it with historical data.

Three-Tier Alerting, Anomaly Detection Included

One of the things Page said he always wanted to build but never had the technology for was anomaly-based alerting. With the Workbench, you don't need to manually configure thresholds for every metric. The system learns baselines and tells you when something is off.

"You don't need to do anything, just let it run," Page said.

The alerting runs across three tiers, with the final tier being an LLM that assesses cluster state and flags what needs attention. Every chart and graph in the system has a button that invokes an AI session to explain what you're looking at and suggest next steps.

Early on, the two most common issues the system has been surfacing are poorly indexed tables — showing up as queries running far longer than expected — and unexpectedly low cache hit ratios. Page noted the cache hit issue caught him off guard in frequency.

"Even on machines where you think, 'I've got plenty of memory on this box, it's going to be fine' — you look at the cache hit ratio and it's 8%," he said.

When Ellie identifies a fixable problem, she generates the SQL. You can execute it directly from the UI or copy it and run it yourself through your own change management process.

Built for Governance, Not Just Speed

One of the questions that comes up immediately with any AI agent touching production databases is: what are the guardrails?

The Workbench has an extensive RBAC system governing access not just to the system but to specific MCP tools and resources. LLM interactions are fully logged and traceable, which Page noted was built early for their own debugging purposes before becoming a governance feature.

And the system is explicit about human approval for any destructive action.

"We don't want Claude running amok," Page said.

The AI model layer is also flexible. The Workbench supports Anthropic, OpenAI, Voyage, Ollama, or any OpenAI API-compatible service. During the beta, the most common feedback was around proxy support — customers who route requests through a bearer-token proxy rather than connecting directly to an AI provider. That's now supported.

Free to Use, Open Source at the Core

The Workbench is available as a free download from GitHub and works with any Postgres environment running version 14 or above. pgEdge describes itself as a fiercely open-source company. The commercial play is straightforward: when larger teams are running the Workbench in production across hundreds of servers, they want a support contract and someone to call when something goes wrong.

Page has been building graphical management tools for Postgres since the early days of the community. The Workbench, he said, is the monitoring system he always wanted to build — finally made possible by technology that didn't exist until recently.

"It's all the lessons learned, things I've wanted to do for the past 20 years," he said.

Ben Fried, former CIO of Google and now a venture partner at Rally Ventures, put it more directly after seeing it: "I wish I'd had something like this years ago."

The Workbench is available at github.com/pgEdge/ai-dba-workbench.

Building LifeFast: A Solo Founder's Deep Dive into Fasting App Architecture

Wed, 03 Jun 2026 19:31:21 +0000

title: Building LifeFast: A Solo Founder's Deep Dive into Fasting App Architecture
published: true

tags: reactnative, postgres, expo, healthapp

There's a specific kind of frustration that comes from using a health app that feels like it was built by engineers who've never actually used it.

The timer shows a number. The number counts down. You earn a badge. That's it.

No context about what's happening inside your body. No sense of progression. No genuine reason to keep going beyond the guilt of breaking a streak. Just a countdown and a badge that means nothing by noon.

That frustration is what made me build LifeFast — a fasting tracker that treats users like intelligent adults who deserve to understand their own biology.

This is the story of how I built it. The technical decisions, the real challenges, the things I got completely wrong, and what I'd do differently. If you're building a health app, a mobile product, or just a solo-founder side project trying to become something real — I hope this saves you some time.

The Stack (And Why I Made These Choices)

Let me get the boring part out of the way first, because the choices here shaped everything downstream.

Frontend: React Native via Expo
Backend: Node.js + Express + TypeScript
Database: PostgreSQL
State management: Redux Toolkit + RTK Query
Build: EAS Build (Expo Application Services)
Email: Resend
Payments: Google Play Billing
Push notifications: Expo Push + custom job queue

I picked React Native over Flutter primarily because I'm a JavaScript developer at heart and the ecosystem felt more familiar. Expo specifically was the right call for a solo founder — the managed workflow gets you from idea to TestFlight equivalent without spending a week configuring native build chains. The EAS Build system in particular is genuinely excellent.

PostgreSQL was a non-negotiable. I've seen enough "just use Firestore" projects hit a wall the moment they needed anything resembling a JOIN, and for a health app with users, fasts, weights, water intake, community posts, and notification jobs all needing to relate to each other — a relational database was the only sane choice.

RTK Query for data fetching was one of the better decisions I made early. The automatic caching, invalidation, and optimistic updates it provides out of the box took a huge amount of complexity off the table on the client side. Pairing it with AsyncStorage for persistence meant the app works properly offline without any heroic effort.

The Hardest Part: Modelling Time

Fasting is fundamentally about time. But time in software is famously treacherous — especially when your users are in 40+ different timezones.

The core data model for a fast looks simple:

CREATE TABLE fasts (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id UUID NOT NULL REFERENCES users(id),
  status VARCHAR(20) NOT NULL DEFAULT 'SCHEDULED',
  timezone VARCHAR(100) NOT NULL,
  start_at TIMESTAMPTZ NOT NULL,
  end_at TIMESTAMPTZ NOT NULL,
  actual_start_at TIMESTAMPTZ,
  actual_end_at TIMESTAMPTZ,
  duration_target_mins INT NOT NULL,
  duration_actual_mins INT,
  ...
);

The trap I nearly fell into: storing times as local timestamps without timezone. I'd seen this mistake in other codebases and nearly made it myself anyway. TIMESTAMPTZ stores everything as UTC internally and lets PostgreSQL handle timezone conversion correctly. Combined with storing the user's timezone string ("Africa/Nairobi", "America/New_York", etc.) — this meant every query, every calculation, every scheduled notification fired at the right local time.

The server runs a cron job every 60 seconds:

setInterval(async () => {
  if (cronRunning) return;
  cronRunning = true;
  try {
    await autoStartDueFasts();
    await processDue();
  } catch (e: any) {
    console.error("[cron] tick error:", e?.message ?? e);
  } finally {
    cronRunning = false;
  }
}, 60_000);

autoStartDueFasts() transitions SCHEDULED fasts to ACTIVE when their start_at passes. processDue() fires push notifications from a job queue. The guard (if (cronRunning) return) prevents overlap on slow ticks — simple, and it works.

The Feature That Changed Everything: Metabolic Stages

This is where LifeFast became a different kind of fasting app.

Most fasting timers show you a percentage. 43%. 68%. 100%. These numbers don't mean anything experientially. What actually happens to your body during a fast follows a predictable, fascinating sequence — and if you show users that instead of just a countdown, everything changes.

I defined fasting stages based on published metabolic research:

Stage	Hours	What's Happening
Fed State	0–3h	Digesting, insulin elevated
Early Fast	3–8h	Glycogen depleting, fat burning begins
Fasting State	8–12h	Insulin low, fat oxidation accelerating
Fat Burning	12–18h	Liver glycogen depleted, ketone production starting
Ketosis	18–24h	Ketones measurable, metabolic switch active
Autophagy	24h+	Cellular self-cleaning in full swing

Each stage has a name, a description, a colour, and a start/end threshold. On the client, I compute which stage the user is in at render time based on elapsed hours:

function getFastingStage(elapsedHours: number): FastingStage {
  const stages = [
    { label: "Fed State", color: "#94a3b8", minHours: 0, maxHours: 3 },
    { label: "Early Fast", color: "#38bdf8", minHours: 3, maxHours: 8 },
    { label: "Fat Burning", color: "#fb923c", minHours: 8, maxHours: 18 },
    { label: "Ketosis", color: "#a78bfa", minHours: 18, maxHours: 24 },
    { label: "Autophagy", color: "#34d399", minHours: 24, maxHours: Infinity },
  ];
  return stages.findLast((s) => elapsedHours >= s.minHours) ?? stages[0];
}

Inside the app's circular timer, the active stage label shows in its colour — a glowing cyan or violet or green dot next to the stage name. Users started messaging me: "I just hit Ketosis for the first time!" — not because I told them to care, but because seeing it named and coloured in real time made it real.

That's the difference between showing someone a number and showing them a story about their own biology.

Building the Community Layer

About two months in, I realised I was building a productivity tool when I should have been building a social one. Fasting is hard in isolation. It's dramatically easier when you're doing it alongside people who understand the challenge.

So I built a full community layer: groups, posts, comments, nested replies, likes, and a moderation system.

The schema is relatively conventional, but a few decisions were non-obvious.

Soft deletes for moderation. When a user reports a post and an admin hides it, the content isn't deleted — it's flagged:

ALTER TABLE community_posts
  ADD COLUMN is_hidden BOOLEAN NOT NULL DEFAULT FALSE,
  ADD COLUMN hidden_reason TEXT,
  ADD COLUMN hidden_by UUID REFERENCES users(id),
  ADD COLUMN hidden_at TIMESTAMPTZ;

This means we can review decisions, reverse them, and maintain an audit trail. Hard deletes felt wrong for a moderation workflow.

Pinned posts per group. The query for listing posts handles pinning cleanly:

ORDER BY p.is_pinned DESC, p.created_at DESC

Two-token sort. Pin always wins; recency breaks ties. Simple and fast.

The notification fan-out problem. When a post in a popular group gets a comment, who gets notified? The author only. When a comment gets a reply, the parent comment's author gets notified. I built a lightweight notification_jobs table and a worker that processes it:

CREATE TABLE notification_jobs (
  id UUID PRIMARY KEY,
  user_id UUID NOT NULL REFERENCES users(id),
  kind VARCHAR(50) NOT NULL,
  scheduled_for TIMESTAMPTZ NOT NULL,
  payload JSONB,
  status VARCHAR(20) DEFAULT 'pending',
  attempts INT DEFAULT 0
);

The worker polls every 60 seconds, picks up pending jobs past their scheduled_for time, calls Expo's push notification API, and marks them as sent or failed. Dead simple, fully auditable, no external queue infrastructure needed at this scale.

The Billing Nightmare

I'm going to be honest: Google Play Billing is the worst API I have ever worked with professionally.

The documentation is spread across three different Google sites that contradict each other. The test environment behaves differently from production in ways that are not documented. The error codes are numerical and the mapping table is buried six pages deep in a section titled "Deprecated APIs".

Here's what eventually worked:

Use the google-play-billing library on the React Native side for purchase flow
On the backend, verify every purchase token against the Google Play Developer API before granting entitlements
Store the raw purchase_token and product_id — you'll need both for future verification and refund handling
Handle ITEM_ALREADY_OWNED (error code 7) gracefully — it means the user purchased on another account or device and is more common than you'd think

The verification endpoint on my backend:

router.post("/billing/google/verify", authenticate, async (req, res) => {
  const { purchaseToken, productId, packageName } = req.body;

  const verified = await verifyGooglePurchase(packageName, productId, purchaseToken);
  if (!verified) return res.status(400).json({ success: false, error: "Invalid purchase" });

  const entitlement = getEntitlementForProduct(productId);
  await billingRepo.recordPurchase({ userId: req.user.id, purchaseToken, productId, entitlement });
  await usersRepo.grantEntitlement(req.user.id, entitlement);

  res.json({ success: true, entitlement });
});

The lesson: never grant entitlements on the client side. Always verify on the server. This is not optional.

Shipping: EAS Build and Deployment

The CI/CD pipeline runs on GitHub Actions:

Push to main triggers the workflow
TypeScript compiles, then the Express app is bundled
SSH into the production DigitalOcean Droplet
Pull latest, rebuild Docker container, swap with zero-downtime restart

For the mobile app, EAS Build handles the actual Android build:

eas build --platform android --profile production

Then submit to Google Play via:

eas submit --platform android

EAS Submit directly uploads the AAB to the Play Store internal track. For a solo founder, this is a genuinely transformative improvement over the old "manually drag an AAB file into the Play Console" workflow.

One lesson I'd pass on: set up your production and staging environments early. I shipped several breaking changes to all users before I separated the two, and the user experience was awful. A staging environment you actually test on is not optional — it's the difference between professionalism and chaos.

What I Got Wrong

Email broadcast timing. I sent an announcement to my users about a new release. I wrote the batch-sending code to fire 50 emails in parallel. I had about 250 users at the time. The email API rate limits at 5 requests per second. You can do the math. Half my users got error responses instead of emails, and the 300ms sequential delay I needed was sitting there in the documentation the whole time.

Over-engineering notifications early. I spent a week building a sophisticated notification scheduling system before I had 50 users. The simpler version — a cron job and a jobs table — does the same job. I could have built it in a day and shipped the features users actually needed instead.

Not adding the progress percentage cap sooner. The fasting percentage can technically exceed 100% if someone continues fasting past their target window. For a while, my timer showed "127%" in a progress circle — which is both mathematically accurate and completely absurd. A one-line fix:

const percentage = Math.min(100, Math.max(0, Math.round(progressRatio * 100)));

Users notice things like this. They form impressions of quality from small details. Fix the small things before they become the big things.

The Product Is Alive

LifeFast is available now on Android.

It's free. It tracks your fasting window in real time, shows you your metabolic stage as you progress, logs your water intake, tracks your weight over time, lets you log progress photos, and connects you with a community of other fasters doing the same work.

I built this without a team, without investors, and without a roadmap handed to me by a product manager. Just a problem I cared about, a stack I understood, and enough stubbornness to keep shipping.

If you're building something similar — a health app, a behaviour-change product, anything where timing and user psychology overlap — feel free to reach out. I'd genuinely enjoy the conversation.

Full Sales Data Analysis: From Raw Data to Interactive Business Dashboard

Sun, 17 May 2026 08:03:49 +0000

Introduction

Anybody can be a data analyst,but what separates a good data analyst from a great one is what you can do and present.In this article i will be giving step by step guidelines from turning a raw data to an interactive business dashboards and giving insights.We will generate our sales data from Mockaroo,use Python to turn the JSON file to CSV,analyse with Sql and visualize with Power Bi.

1.Generating the sales data.

You will go to your browser and search Mockaroo.com.Once open,hit the generate data button and give AI a command to generate the data for you.You can select the number of rows that you want your data to have.Once its done,save the data as a json file.

a.Viewing the data

Once your data is generated and saved in your files in your machine,go to your Github account and add that file to your repo.

Once you open that file,hit the raw button and you will see something like this:

2.Importing using Python.

The work is not yet done.Copy the link from your raw data in github because we shall need it.Use the import requests and the url will be your link you copied from raw data in github.While pasting the url,ensure it has the either the double or single quotes at the start and at the end of the url link.The status code must always be 200.

The next step will be seeing the type of data that we have,either a list or dictionary.

We have created a data frame,viewed the first rows and saved the the data as csv.
You should be seing the sales data as an excel file in your files ready for analyzing.

SQL analysis

Since we already have our data as CSV,we will import it in any sql environment you use.I will be using DBeaver.First you create a schema called sales_data or any name of your choice,then creating the sales table.

1.creating the table

You create a schema called sales_data or any name of your choice,then creating the sales table.Every column name will be kept as texts.Why,to avoid data type errors during import and to handle missing values,weird formats or nulls with ease.

2.Importing the data to SQL environment

first,you go to your schema,refresh and search for your table name.Right click it and hit the import data button.Import the csv then press next,hit the browse button to select which file exactly that you want,then press next till you proceed to the next part.After importing the data,hit the refresh button on the table name below your schema to ensure the data is set.You can now write a query to see your data

b.cleaning the data

This is where most data analyst spend most of their time.With reference to my sales data,it's not so messy only few mistakes like fixing the total_price column.

1.fixing the total_price column

As i imported my data,the column total_price was full of error

I will fix it by step by step as shown below.

This SQL command fixes the total_price column by converting the text-based quantity values to integers using ::INTEGER,then multiplying each by a flat rate of $19.99.The WHERE clause targets only rows where total_price still contains the error message,leaving already-corrected rows untouched.

2.Trimming,capitalizing and removing special characters

Triming in SQL eliminates spaces before and after any word.Initcap is a special command that allows you to standardise the letters of words inside your columns.This is a similar command to 'proper' as used in My SQL and other SQL environment.

Now that our data is clean,we proceed to the next step which is answering business questions.Our data was not so messy so we used little times as possible.

c.Re-creating the clean table.

Since our data was stored as texts format,before analysis we should ensure everything is in its correct format as we did in cleaning.

Some of our columns are not in their corect format.We can change them manually by writing a simple query by altering the table.

d.Answering business questions

This part is so important since we shall use it in visualization.Answering business questions like:sales performance,product analysis,customer insights and operational questions
1.Finding top 10 sales rep

2.finding which method of payments produces more revenue

You can answer a lot of business questions as possible.I shall leave a link to my github for you to go through the business questions and answers.

Creating views

Views in SQL simplifies your work as a HR or a CEO since you don't have to go through every long queries.We shall also be using the views created in Power BI in creating visuals.We will be creating payment method analysis view,products perfromance views,sales perfomance analysis view and sales summary analysis view as our facttable.

The query worked well and now we have the sales view.
We have also created the sales rep perfomance view successfully.

The product perfomance views is also created successfully:

A payment method analysis view is also important we should have it as one of our views created:

Lastly,we must track down our customers so i created another view called customer analysis view and was succesfull

3Linking our sql environment to Power BI

since we have our views created,we shall be using them in Power BI.
1.Open Power BI and tap the home page,hit the import data from sql server then fill the required item:

2.Once you have already linked,select only the views we created in sql earlier as in our case.It will appear as shown:

4.Creating relationships.

Since we have 5 different mini-tables,we will create one called a fact table that will have a link to all the other tables.We will add a new column in each of the 5 different table called 'ID' column in relation with the name example 'product_id,sales_id'.After adding the new columns,we copy and paste them inside the fact table.

After adding the new column having the ID,we will merge queries them together to the fact table.Joining the common columns.

NOTE:You only select the common column(left outer join).

5.Creating dashboards

Once you have all the relationships,you can present your data in the forms of charts or any visual of choice.
This below is one of the examples i came up with:

You can have up to 5 maximum pages of different charts

6.Report and Insights

Open a new page called report and insights and select a text box since we shall be using texts a lot more than charts.
You can create a page called recommendation,executive report and so on.This can be presented to your boss with ease:

Conclusion

That one right there is a full data analyst project we did.We used tools like Python to import the data,SQL for analysis and creating visuals using Power BI.You can add it to your portfolio as one of the Projects and can help you land a job with ease since you have proof.
You can check more analysis i made using SQL :https://github.com/JaysonJob/full-sales-data-analysis/blob/main/Script-35.sql
I hope you found this article useful.Ciao

The Heart of B3: Building the Matching Engine with RabbitMQ, Redis and Spring Boot

Sun, 10 May 2026 21:29:07 +0000

Hello everyone!

Continuing the My Broker B3 series, we've reached one of the most anticipated components of the ecosystem: the B3 Matching Engine API.

In the previous post, we built b3-market-sync-api which synchronizes real prices from brapi.dev and stores them in Redis. Now it's time to use that data — this is the service that simulates the stock exchange itself, deciding whether a buy or sell order will be executed or rejected based on the prices we just put in cache.

️ What is the Matching Engine?

In the real world, when you place a buy order through your broker, it gets sent to B3, which checks whether there's a counterpart willing to trade at that price. This process is called matching.

In our simulator, the b3-matching-engine-api reproduces this logic in a simplified way:

Intake: Receives the order from the broker via RabbitMQ (queue mq-broker-to-b3)
Price: Fetches the current asset price from Redis (market:price:{TICKER}) — data injected by the Market Sync we built previously
Decision: Applies the matching rule
Persistence: Records the result in PostgreSQL
Notification: Returns the result to the broker via RabbitMQ (queue mq-b3-to-broker)

[Broker] ──RabbitMQ──▶ [mq-broker-to-b3] ──▶ [Matching Engine]
                                                      │
                                             Redis: market:price:{TICKER}
                                             (fed by Market Sync ⬆️)
                                                      │
                                         ┌────────────┴────────────┐
                                      FILLED                   REJECTED
                                         │                         │
                                   PostgreSQL                 PostgreSQL
                                         │                         │
                               [mq-b3-to-broker] ◀────────────────┘
                                         │
                                      [Broker]

MVP Focus

Before diving into the code, same disclaimer as previous posts: we're building the foundation. The goal is to have an end-to-end flow working with enough robustness to validate the POC.

In this phase, I prioritized:

End-to-end flow working correctly
Proper failure handling (no orders silently disappearing)
Dead Letter Queue for messages that fail during processing
REST API for querying execution history
Swagger documentation

️ Tech Stack

Technology	Usage
Java 21 + Spring Boot 3.5	Service core
Spring RabbitMQ	Order intake and result notification
Spring Data Redis	Real-time price lookup
Spring Data JPA + PostgreSQL	Execution persistence
Flyway	Database schema versioning
SpringDoc OpenAPI	Swagger UI documentation

️ Implementation Pillars

1. RabbitMQ Configuration

A critical lesson learned here: never rely on queues already existing in the broker. If the application starts before RabbitMQ has the queues created, the consumer fails on startup.

The solution is to declare all infrastructure beans directly in Spring — it ensures queues, exchanges and bindings exist before any message is processed:

@Bean
public DirectExchange exchange() {
    return new DirectExchange(exchangeName);
}

@Bean
public Queue queueIn() {
    return QueueBuilder.durable(queueIn)
            .withArgument("x-dead-letter-exchange", dlxName)
            .withArgument("x-dead-letter-routing-key", dlqName)
            .build();
}

@Bean
public Binding bindingQueueIn(Queue queueIn, DirectExchange exchange) {
    return BindingBuilder.bind(queueIn).to(exchange).with(routingKey);
}

Note the withArgument calls — they configure the Dead Letter Queue directly in the main queue declaration. Any message that fails processing is automatically redirected.

2. The Matching Logic

The heart of the service. The rule is straightforward:

BUY: If the price the user accepts to pay >= market price → FILLED
SELL: If the price the user wants to receive <= market price → FILLED
Otherwise → REJECTED

private static boolean isCanExecute(OrderEventDTO order, BigDecimal marketPrice) {
    if (SideStatus.BUY.name().equalsIgnoreCase(order.getSide())) {
        return order.getPrice().compareTo(marketPrice) >= 0;
    } else if (SideStatus.SELL.name().equalsIgnoreCase(order.getSide())) {
        return order.getPrice().compareTo(marketPrice) <= 0;
    }
    return false;
}

3. Failure Handling — No Silent Order Loss

A silent bug I identified: when the ticker wasn't in Redis, the method simply returned without doing anything. The broker would wait for a response that never came.

The fix was simple but critical — any failure must notify the broker:

if (marketDataOpt.isEmpty()) {
    log.warn("Price for ticker {} not found in Redis. Rejecting order {}",
             order.getTicker(), order.getOrderId());
    OrderResponseEvent response = new OrderResponseEvent(
        order.getOrderId(), ExecutionStatus.REJECTED.name(), BigDecimal.ZERO);
    orderProducer.sendToBroker(response);
    return;
}

This is exactly why we built b3-market-sync-api first. If Redis doesn't have the price, the order is immediately rejected — correct and predictable behavior.

4. Transactional Guarantee

Another critical point: persisting to PostgreSQL and publishing to RabbitMQ are two distinct operations. If the database saves but the publish fails, the execution is recorded but the broker is never notified — silent inconsistency.

The solution was adding @Transactional to the method that orchestrates both operations:

@Transactional
private void saveAndNotify(OrderEventDTO order, BigDecimal price, ExecutionStatus status) {
    // 1. Persists in PostgreSQL
    OrderExecution execution = OrderExecution.builder()
            .orderId(order.getOrderId())
            .ticker(order.getTicker())
            .side(SideStatus.valueOf(order.getSide()))
            .quantity(order.getQuantity())
            .executedPrice(price)
            .status(status)
            .build();
    repository.save(execution);

    // 2. Notifies the Broker
    OrderResponseEvent response = new OrderResponseEvent(
        order.getOrderId(), status.name(), price);
    orderProducer.sendToBroker(response);
}

5. Dead Letter Queue

For messages that fail in the consumer (unexpected processing error), we configured a full DLQ setup:

@Bean
public DirectExchange deadLetterExchange() {
    return new DirectExchange(dlxName);
}

@Bean
public Queue deadLetterQueue() {
    return QueueBuilder.durable(dlqName).build();
}

@Bean
public Binding bindingDlq(Queue deadLetterQueue, DirectExchange deadLetterExchange) {
    return BindingBuilder.bind(deadLetterQueue)
            .to(deadLetterExchange)
            .with(dlqName);
}

In the consumer, we rethrow the exception so RabbitMQ knows the message failed and routes it to the DLQ:

@RabbitListener(queues = "${app.rabbitmq.queue-in}")
public void receiveOrder(OrderEventDTO event) {
    log.info("New order received: ID {} | Ticker {} | Side {}",
            event.getOrderId(), event.getTicker(), event.getSide());
    try {
        matchingService.process(event);
    } catch (Exception e) {
        log.error("Failed to process order {}: {}", event.getOrderId(), e.getMessage(), e);
        throw e; // RabbitMQ routes to DLQ
    }
}

6. REST API + Swagger

Since the service exposes query endpoints, I configured Swagger UI from the start:

@Bean
public OpenAPI customOpenAPI() {
    return new OpenAPI()
            .info(new Info()
                    .title("B3 Matching Engine API")
                    .version("1.0.0")
                    .description("Simulates the B3 stock exchange matching engine..."));
}

Available endpoints:

Method	Endpoint	Description
GET	`/api/v1/executions`	List all executions
GET	`/api/v1/executions/order/{orderId}`	Find by order ID
GET	`/api/v1/executions/ticker/{ticker}`	Find by ticker
GET	`/api/v1/executions/status/{status}`	Find by status

✅ Validating the Execution

With the application running locally, all components started correctly:

✅ Flyway applied the order_executions table migration
✅ Hibernate validated the schema
✅ RabbitMQ connected with queues and DLQ declared
✅ Tomcat running on port 8091
✅ Swagger UI accessible at http://localhost:8091/swagger-ui.html

What's Next?

With b3-market-sync-api feeding Redis and b3-matching-engine-api ready to process orders, the entire B3 side is operational.

The next step is building the broker-order-api — the orchestrator on the broker side that will manage the complete order lifecycle:

Receive the buy/sell intent from the user
Validate the ticker via REST against broker-asset-api
Save the order as PENDING
Publish the event to Kafka
Send the order to the Matching Engine via RabbitMQ
Consume the feedback and update the final status

Once that service is ready, we'll have the first complete end-to-end flow running in the ecosystem.

Got any questions about the matching logic or RabbitMQ DLQ configuration? Drop them in the comments!

About the Series

⬅️ Previous Post: Syncing the Real Market: Consuming Brapi and Feeding Redis with Spring Boot

Series Index: Series Roadmap

Links:

Understanding Logic, Reusability and Integrity On SQL ; Procedures, Functions and Transactions.

Sun, 10 May 2026 16:23:47 +0000

SQL is widely known for data querying and manipulation but systems do grow; data becomes larger; processes become repetitive and operations become sensitive. SQL has some features which enables it to be considered a fully fledged programming language. Some of the features which I discuss in this article are procedures, functions and transactions. Each of these concepts serve distinct purposes.
Procedures execute operations, functions return values, and transactions ensure those operations are safe.

Stored Procedures

These are set of SQL statements stored in the database and executed as a unit which are used to perform tasks such as UPDATE, INSERT, DELETE etc.
They are triggered by calling them and passing the expected parameters by the procedure.
Here is an example of a procedure:

CREATE OR REPLACE PROCEDURE increase_salary(p_dept TEXT, p_percent NUMERIC)
LANGUAGE plpgsql
AS \(
BEGIN
    UPDATE employees
    SET salary = salary + (salary * p_percent / 100)
    WHERE department = p_dept;
END;
\);

Here is how a procedure is called:

CALL increase_salary('IT', 10);

Functions

This is a reusable logic block which can return values and can be used inside queries for data selection.
It can be used with SELECT, WHERE and is great for reusability.

CREATE OR REPLACE FUNCTION get_avg_salary(p_dept TEXT)
RETURNS NUMERIC
LANGUAGE plpgsql
AS \(
BEGIN
    RETURN (
        SELECT AVG(salary)
        FROM employees
        WHERE department = p_dept
    );
END;
\);

Here is an example of how the function can be used:

SELECT name, salary
FROM employees
WHERE salary > get_avg_salary(department);

Transactions

Transactions are used to group a set of code operations in which if any of it fails then the whole execution is aborted. This ensures that data is only changed when the full code execution structure is successful.
Transactions are best for data safety as they prevent partial updates.

BEGIN;

UPDATE accounts
SET balance = balance - 100
WHERE id = 1;

UPDATE accounts
SET balance = balance + 100
WHERE id = 2;

COMMIT;

If it fails:

ROLLBACK;

How They Work Together

It will not be complete to conclude the article without showing an example of how these concepts can be used together.
Here is a simple scenario showing the use of the three concepts interdependently.
Function:

CREATE OR REPLACE FUNCTION get_balance(acc_id INT)
RETURNS NUMERIC AS \(
BEGIN
    RETURN (SELECT balance FROM accounts WHERE id = acc_id);
END;
\) LANGUAGE plpgsql;

Procedure:

CREATE OR REPLACE PROCEDURE transfer_money(from_id INT, to_id INT, amount NUMERIC)
LANGUAGE plpgsql
AS $$
BEGIN
    IF get_balance(from_id) < amount THEN
        RAISE EXCEPTION 'Insufficient funds';
    END IF;

    UPDATE accounts SET balance = balance - amount WHERE id = from_id;
    UPDATE accounts SET balance = balance + amount WHERE id = to_id;
END;
$$;

Transaction:

BEGIN;
CALL transfer_money(1, 2, 100);
COMMIT;

Conclusion

SQL has a wide range of capabilities more than just being a querying language. Having features such as procedures, functions, transactions and many others helps it to be an efficient tool for use on data in whichever way needed. Mastering it comes a long way in helping the analysis of data directly from the database.

From Aiven to Railway: Our Database Migration

Tue, 05 May 2026 09:40:39 +0000

I didn't plan a migration. I just noticed we were paying for two databases.

This whole thing started with a very unglamorous realization:

We were paying for Aiven Postgres…
and also paying for Railway.

At some point I opened the Railway dashboard properly and thought:

"Wait — why aren't we using this for the database too?"

No incident. No outage. No scaling crisis.
Just mild financial confusion.

The decision wasn't really "should we migrate?"

It was more like:

"We're already using Railway for everything else. Why is the database emotionally living somewhere else?"

So I did what any developer does when something feels inconsistent: I turned it into a migration project.

The migration itself was weirdly easy

I expected:

scripts
downtime planning
CSV exports
at least one regretful late-night rollback

Instead, Railway just pointed me to their Postgres migrator:

Railway Postgres Migrator

The setup was:

Source: Aiven Postgres URL
Destination: Railway Postgres
Action: Deploy

That's it. I didn't trust it at first.

The one thing that actually tripped me up

Before the migrator worked, I hit a version mismatch. Railway's Postgres was running v18 — and the migrator requires source and destination to be on the same major version. Aiven was v17, so I had to upgrade it first before anything else would proceed.

Once versions matched, there was another quiet gotcha: two optional environment variables — NO_ACL and NO_OWNER — had to be set to true for the migration to complete cleanly.

Variable	What it does
`NO_ACL`	Skips restoring access privileges (GRANT/REVOKE commands)
`NO_OWNER`	Skips restoring object ownership (ALTER OWNER commands)

The Railway docs show their default as true, but in practice they were false — meaning the migrator was trying to restore ACLs and ownership from Aiven, hitting permission errors, and failing silently enough that it wasn't immediately obvious why.

Setting both to true explicitly is what actually made it work.

I kept waiting for something to break. Nothing did.

No data loss.
No partial sync weirdness.
No "why is production dead" moment.

Just a working database in a new place, which is almost more stressful than failure — because you keep checking if you missed something.

Railway vs Aiven (honest experience, not a benchmark)

Aiven Postgres:

Solid and reliable
More manual setup
External to your deployment ecosystem
SSL and config overhead you have to manage

Railway Postgres:

Lives inside your deployment environment
Internal networking by default
Less cognitive overhead on config
Everything in one dashboard

The biggest difference wasn't performance. It was mental overhead. Fewer places to check. Fewer credentials to rotate. One less tab permanently open.

If you're already on Railway and running an external Postgres for no strong reason — probably yes. The migrator genuinely works, the internal networking is a real quality-of-life upgrade, and the consolidation is worth it just to stop context-switching between dashboards.

If you have a strong reason for keeping the database external (compliance, specific Aiven features, multi-cloud setups) — that calculus changes. But "we started there and never questioned it" isn't a strong reason.

Sometimes the migration is just noticing you forgot to consolidate.

How to Connect Power BI to a SQL (PostgreSQL) Database and Build a Unified Dashboard

Mon, 04 May 2026 13:29:57 +0000

Introduction

Power BI is a business intelligence (BI) and data visualization tool from Microsoft. It enables analysts and business users to transform raw data into interactive dashboards and reports. Companies use Power BI to analyze sales, customer behavior, inventory trends, and other critical business metrics.

SQL databases, like PostgreSQL, are widely used to store and manage structured data. Connecting Power BI to SQL databases allows analysts to retrieve, clean, transform, and model data efficiently, creating a single source of truth for dashboards and KPIs.

In this article, we’ll walk through:

Connecting Power BI to local and cloud PostgreSQL databases
Loading raw data and applying transformations
Combining datasets from multiple sources
Adding a Date Dimension table (DimDate) for KPIs and filtering
Preparing a dashboard-ready model for GitHub and reporting

What You Need Before Connecting

Power BI Desktop installed
PostgreSQL database connection details (server, port, database, username, password)
For cloud connections: SSL certificate (CA certificate)
Model view showing relationships across cloud and local tables

Connecting to a Local PostgreSQL Database

Open Power BI Desktop.
Click Get Data → PostgreSQL database.

Enter your connection info:

Server:

localhost:5432 (replace 5432 with your PostgreSQL port)

Database: Name of your database

Enter your username and password.

Click OK to connect.

Note: The localhost:port syntax ensures Power BI connects to the correct PostgreSQL server port.

In the Navigator window, select tables to load, then click Load (or Transform Data).

Connecting to a Cloud PostgreSQL Database (Aiven or Similar)

Cloud databases require secure SSL connections.

Guide to Aiven Connection

From your cloud provider, obtain:

Host
Port
Database name
Username and password
SSL certificate (CA certificate)
Install the CA certificate on your system:
Open the downloaded .crt or .pem file.
Install it in the trusted root certificate store:

Windows: Right-click → Install Certificate → Place in “Trusted Root Certification Authorities”
Linux/macOS: Follow OS instructions to add to the system or user trust store

Installing the CA certificate allows Power BI to validate the server’s identity and establish a secure connection.

In Power BI Desktop:

Go to Get Data → PostgreSQL database
Enter Server

your-cloud-host:5432

and Database

Check Use SSL certificate and select the installed certificate
Enter your username and password

If successful, it opens your aiven cloud database as shown below
Choose tables from your database

Click OK and load the tables

Load Raw Data First

It’s important to load raw data before applying transformations:

Load all tables (customers, products, sales, inventory) first.
Review the raw data to assess missing values, incorrect or inconsistent formats and unnecessary columns
This helps you understand which fields need cleaning and transformation before building dashboards.

Filtering and Transformation

Use Power BI’s Power Query Editor for data preparation:

Remove unnecessary columns – keep only fields needed for analysis.
Rename columns – make names descriptive.
Filter rows – e.g., remove canceled sales or test records.
Change data types – ensure numeric and date fields are correct.
Create calculated columns – e.g., total sales = quantity × price.
Handle missing values – fill, replace, or remove nulls.

Creating Relationships Between Tables

Once data is cleaned:

Switch to Model View.
Create relationships:

`sales.customer_id → customers.customer_id

sales.product_id → products.product_id

inventory.product_id → products.product_id`

Validate relationships for correct cardinality and cross-filtering.

Relationships ensure Power BI can accurately summarize data across multiple tables.

Data Modeling Basics

Star schema
Fact table (sales) connected to dimension tables (customers, products, inventory).
Cardinality
One-to-many or many-to-one relationships.
Filter directions
Define single or bidirectional filters depending on your analysis needs.

Proper modeling ensures dashboards are interactive and accurate.

Why SQL Skills Matter for Power BI Analysts

Data extraction: Retrieve exactly the data you need.
Filtering & aggregation: Pre-process data for better performance.
Data cleaning & transformation: Prepare data before loading it into Power BI.
Performance optimization: Efficient queries improve dashboard responsiveness.

By combining SQL knowledge with Power BI, analysts can build powerful, reliable dashboards that drive business decisions.

Conclusion

Connecting PostgreSQL to Power BI empowers analysts to turn structured data into actionable insights. Whether using local or cloud databases, establishing secure and reliable connections ensures that the data you analyze is accurate and up-to-date. By first loading raw data, applying necessary transformations, and creating relationships, including a Date table for time-based analysis, you build a robust data model ready for interactive dashboards.

Mastering this workflow not only simplifies KPI calculations and trend analysis but also highlights the importance of SQL skills for preparing and managing data. Ultimately, integrating PostgreSQL with Power BI provides a seamless bridge between data storage and visualization, enabling smarter, data-driven business decisions.

Sub-queries vs Window Functions vs Common Table Expressions: Which Should You Use in SQL?

Tue, 28 Apr 2026 16:33:58 +0000

SQL is the most used tool for data manipulation but what happens when there are some concepts that seem to work towards the same ultimate outputs. As queries become more complex, developers often encounter different techniques that appear to produce similar results—namely sub-queries, common table expressions (CTEs), and window functions.
At first glance, these concepts can seem interchangeable. You might solve the same problem using any of them, which raises a natural question: which one is better?
The goal is not to choose one, but to understand when each is the right tool.
In this article I will discuss about sub-queries, window functions and common table expressions;

Sub-Queries
These are queries inside other queries and are executed either before or alongside the outer query. They can return scalar value, row or table.

SELECT *
FROM employees e
WHERE salary = (
    SELECT MAX(salary)
    FROM employees
    WHERE department = e.department
);

Window functions
These are used when you are trying to preserve your rows. Used together with components such as - OVER(),PARTITION BY,ORDER BY and common functions such as - SUM(),AVG(),RANK(),LAG(),LEAD()

SELECT employee_id,first_name, last_name,salary, job_title, department
FROM (
  SELECT *,
    RANK() OVER(PARTITION BY department ORDER BY salary DESC) AS rnk
  FROM employees
) t
WHERE rnk = 1;

Common Table Expressions (CTEs)
These are temporary results set that are initialized using -WITH. They are often used to improve readability and modularity of scripts.

WITH max_salary AS (
    SELECT department, MAX(salary) AS max_sal
    FROM employees
    GROUP BY department
)
SELECT e.*
FROM employees e
JOIN max_salary m
    ON e.department = m.department
WHERE e.salary = m.max_sal;

All these features of SQL are used to perform advanced data querying and transformations. They can all be tweaked to perform almost similar queries but it all depends on what the final output is expected to be.

When to use what
Each of the features has an instance when it is the most convenient over the others.

For instance when it is a simple filtering such as retaining all those above average marks then a sub-query is the best option.
When we are more focused on row level analytics then the window functions come in handy for that task.
In a situation where it is a multi-step logic and re-usability is a core factor the common table expressions are best suited for use.

Using in collaboration
The best thing about this features is that they can be used together for the best solution. This helps in the data transformation where there are steps within it in which either is considered to be superior over the other.
Here is an example where all of them are used together:

with ranked AS (
    SELECT 
        *,
        AVG(salary) OVER (PARTITION BY department) AS avg_salary
    FROM employees
)
SELECT *
FROM ranked
WHERE 
    salary > avg_salary
    AND salary > (
        SELECT AVG(salary) FROM employees
    );

Best Practices
These are some of the recommended practices when working with these SQL features:

It is best to avoid deep nesting of queries by using CTEs which help in the structuring of the script for easy readability.
When using CTEs it is necessary to name them well so that their purpose can be identified easily without having to read all the code snippets manually.
Window functions are often more efficient than sub queries in large datasets, so where possible it is best to use window functions over sub queries but in instances of small datasets any can be used depending on preference.

Conclusion
These are to be considered more as complementary tools than competing tools. In most advanced queries you will have to use them together or interchangeably to get to the results.
What makes a major difference is how you use them as they can heavily impact the performance. It is best to master all of them and understand the strength and weaknesses of each so that when it get to a point of picking one or more you do it from a point of information.

How I Survived My First Full stack App with Next.js

Fri, 18 Jul 2025 12:38:28 +0000

The story begins—as it always does—with a tutorial.

I was watching yet another video and thought: “Let’s build a full stack app.” I already had a background in React, and I’d made a few full stack projects before. So this time, I chose Next.js. It looked easy, especially with the new App Router.

Why Next.js? Because it’s React at the front, backend APIs included, and server actions baked right in. No need for separate Express servers or wiring up extra routers. Also, the assumption was simple: everything is a server component unless you slap a "use client" on top. I was in.

✨ Why I Fell in Love with the App Router

One of the best things about Next.js is the App Router. You don’t need to install React Router and define routes in App.jsx. Just drop a file in the /app folder, and boom—it’s a route. It's simple. It’s fast. It works.

First Battle: The UI Fear

Here’s my harsh truth—I suck at design.

My pages looked ugly. And that really messes with you. It triggers imposter syndrome. I’d open VS Code, freeze, and think: What do I even build here? The spiral begins: “Am I even cut out for frontend dev?”

Sound familiar? Been there. This time, I didn’t let it win.

Here's how I beat the block:

Took breaks when stuck.
Drew wireframes, even if they were ugly. Just enough to give me direction.
Used inspiration from other websites. Seriously, 90% of the internet uses the same layout tricks. Why not learn from them?
Tools like Google Stitch, v0.dev, and Bolt helped generate page ideas and inspiration.

Any progress is better than no progress. Remember that.

❤️ Server Actions

God bless Server Actions.

No more creating an API route or writing fetch calls and handling them manually. Need to read from the server? Just keep the page component as a server component, use a Server Action to grab the data, and pass it down to your client components. Done.

You can use server actions inside client components too, but there are caveats. I won’t pretend I know them all—this project was me figuring things out on the fly. So I won’t preach, but here’s what worked for me:

Keep data fetching in server components.
Use server actions to mutate or fetch data.
If using forms, yes—you can still call server actions from client components.

How I Wrote Ugly Code—Fast

Not gonna lie. I was a messy developer in the beginning.

Every time I made a new page/component, I didn't care about reusability. Things got chaotic fast.

But I followed one rule: put every component inside a well-named folder. For example: /components/dashboard/Table.tsx. Later, when I needed it somewhere else, I refactored it—added props, defined interfaces, added type safety—and made it reusable.

So... am I a great React developer? Maybe not.

Am I resourceful and fast? Yes.

Best case: smart developer.

Worst case: functional chaos.

The Email Saga

Now I understand why dev’s complain about bad DX (Developer Experience) on Twitter.

Until now, I had a chill life—React + Vite, everything worked like magic. But for this project, I needed to send emails inviting users to give feedback.

I initially tried Mailjet — looked promising, solid docs, decent limits. But then... everything broke after sending just 5 emails.

I got hit with a "temporary restriction" and no real explanation. Just a vague message and complete silence. I waited three days, hoping the ban would lift. Spoiler: it didn’t.

That’s when I realized how important good developer experience really is.

If I ever build a dev tool, I’m writing great docs, clear errors, and probably a fun YouTube guide.

So, I ditched Mailjet and moved to Resend — and this time, everything just worked. Fast setup, friendly DX, and no cryptic errors.

❤️ Huge shoutout to Resend.

???? No hard feelings, Mailjet — but that experience wasn’t it.

The Broken Database Horror

After surviving all that, I wanted to brag on LinkedIn: “Hey look! I integrated payments!” Even if they were test payments—still, it’s progress.

But the horror unfolded quickly.

Understanding how payments work end-to-end—from initiation to confirmation—is critical. If you don’t, you’ll keep breaking your schema with every little change.

What I learned:

If you're dealing with third-party APIs or external systems, always:

Track their status (pending, success, failure)
Use enum fields to reflect the state
Expect that some parts of the flow are out of your control

Know what you're storing and when. Or prepare to suffer. And yes, it will become a dev horror story.

Linting + TypeScript Woes

I used to think linting was just that one VS Code extension that makes code pretty.

Then I ran npm run build.

Boom. 10,000+ lines of errors. Why? It tried to lint the auto-generated code too.

So yeah, I had to:

Fix unused variables
Replace any types
Update configs to ignore generated folders

Lesson learned: don't skip linting. TypeScript isn’t just for safety—it’s a build-saver.

Deployment & Production Nightmares

I’d spent a month building this project. Now it was time to ship. Resume-ready, you know?

I used Vercel. Guess what?

Vercel kept trying to render server pages as static. My build kept failing.

Turns out: Vercel will try to statically generate everything unless you tell it not to.

Solution:

export const dynamic = "force-dynamic";

Boom. Now it knows your page is meant to run server-side only.

Oh, and Supabase—my free DB. Connecting it was hell.

There’s a pooled connection (for normal ops)
And a direct connection (for migrations)

But Prisma didn’t play well with pooled URLs. It caused weird, unexpected issues.

Fix:

Use Supabase’s "stable connection string" for local + migrations
Use Prisma Accelerate in production with the correct URL
Add the Accelerate URL to your .env.production

After that, things worked. Kinda.

Final Thoughts

You may not have learned any technical tricks from this post. But that’s the point.

Things will break.

Abstractions will leak.

You’ll be stuck. Frustrated. Confused.

But you’re a builder. And builders show up.

That’s what matters.

❤️ Takeaway

Everyone can teach you code. I want to teach you a mindset.

If you remember just two things from this post:

Break things. That’s how you learn.
Draw wireframes. Ugly ones. It gives you momentum.

That’s it. You’ll be 10x faster than me if you just start.

Follow My Journey

I write about my dev rants, lessons, and experiments—sometimes messy, sometimes insightful. If this resonated, follow me. I don’t just rant—I teach too.

Next up: how I broke and optimized my backend.

Till then,

Break. Fix. Weep. Shine.

— Love and peace,

Your fellow dev

Farewell & Little Sendoff Gifts for Curious Minds

Thanks for sticking around! If you're curious to poke around or steal some ideas (I won't tell), here are a few goodies:

Live App:

Create Next App

GitHub Repo: https://github.com/PRASHANTSWAROOP001/feedback-nextjs-app
Wireframes (Chaos Included): https://raw.githubusercontent.com/PRASHANTSWAROOP001/contact-page/refs/heads/main/wireframe2.jpg, https://github.com/PRASHANTSWAROOP001/contact-page/blob/main/wireframe1.jpg?raw=true

Coder Legion - Recent posts tagged postgres

Database Transactions - ACID Properties

Introduction

What is a Database Transaction?

ACID Properties

Atomicity

How does the database maintain atomicity?

Consistency

1. Data Consistency (Correctness)

2. Read Consistency (Availability and Freshness)

Availability vs. Consistency

What's the Difference?

Isolation

1. Dirty Read

2. Non-Repeatable Read

3. Phantom Read

4. Lost Update

Database Isolation Levels

1. Read Uncommitted

2. Read Committed

3. Repeatable Read

4. Serializable

Pessimistic and Optimistic Concurrency Control

Pessimistic Concurrency Control

Optimistic Concurrency Control

Comparison of Isolation Levels

What I Understood About Isolation

Durability

How do databases maintain durability?

1. Write-Ahead Logging (WAL)

2. OS Page Cache and fsync

3. Checkpoints

4. Asynchronous Snapshots

5. Append-Only Files (AOF)

Conclusion

Cancel or send: one winner. An order flow on Prisma Postgres and Compute

Summary

The problem

Design decisions

Claim first, then send

A guarded transition

A Postgres poller, not a queue

A row lock on the design

Tests, and a test that proved nothing

Prisma Postgres and Compute

Why Prisma ORM 8 is in this project

Getting a NestJS 11 app onto Compute

The surprise: scale to zero versus my poller

Open question: where should the poller run?

Raw output

What I did not verify

Takeaways

A week of green runs, and the pipeline was waiting for me

The gate was doing its job

What my alarms actually watch

The information was already there

How it ended

I built a Supabase-style query client that talks directly to Postgres

Who it's for

How it works

What you get beyond the basics

Guardrails by default

How I tested it

The RLS caveat (please read)

What's missing or rough

Try it, break it

Kafka Just Learned to Be a Queue. Postgres Has Been One All Along.

The thesis

The antithesis, at full strength

What actually shipped (facts, as of October 5, 2026)

The Postgres queue is one SQL statement

Demo A: does SKIP LOCKED actually matter?

Demo B: kill a worker mid-batch

The real advantage: the job and the data commit together

Where the Postgres queue dies: one old snapshot

Demo C: churn 200,000 jobs, then measure

The trap: your worker can be the long transaction

A note on LISTEN/NOTIFY

Where Kafka share groups bite

The Go side: a share group on one partition