Coder Legion - Recent posts tagged postgresql

Database Migrations: Can Your Previous Release Still Run?

Mon, 19 Oct 2026 07:45:55 +0000

A deployment can roll back cleanly while the database stays incompatible with the old application. Before shipping a schema change, check whether the previous release can still read and write the data that the new release creates.

Consider a fictional account service. Version 1 reads a column called display_name. Version 2 expects public_name. Renaming the column and deploying the new code together looks tidy, until an old worker, a rolling-deployment instance, or an emergency rollback still asks for display_name.

The migration succeeded. The service can still fail.

Separate compatibility from cleanup

Danilo Sato’s explanation of parallel change describes three stages: expand, migrate, and contract. The useful idea is to introduce the new interface while the old one remains available, move consumers, then remove the old interface.

For a database rollout, turn that idea into explicit release gates:

Expand: introduce the new storage shape without removing what the currently deployed application needs.
Migrate: move reads and writes deliberately, checking compatibility during the transition.
Contract: remove the old shape only after its consumers and the intended rollback window are accounted for.

These stages are a planning model, not a guarantee of zero downtime. The exact migration still depends on the database, data volume, application behavior, and operational limits.

Work through the awkward middle

In the account-service example, adding public_name is only the beginning. Before copying values, decide which field is authoritative at each stage.

A possible transition is to deploy compatibility code that continues serving old readers while maintaining both values. If both columns live in the same row, update them in the same database transaction where practical. That still leaves an important question: can any old application instance or background job write only the original column?

Do not switch reads to the new column while those writers can silently make it stale. Either finish upgrading writers first or design a deliberate synchronization mechanism. A fallback read helps with missing values; it does not detect an outdated non-empty value.

Backfills need the same care. A naive copy can overwrite a newer value if it races with a live update. Define an update condition, concurrency strategy, and restart behavior before running the job. Record progress so an interrupted backfill can resume without blindly replaying the entire table.

Test the rollback with new data

A useful rehearsal includes more than deploying version 2 successfully:

Start with representative data, including empty and unusual values.
Apply the expansion and deploy the compatibility release.
Run the backfill alongside representative writes.
Create and edit records using the new release.
Restore the application version you actually intend to roll back to.
Verify that it can read and update those new records correctly.

Include background workers, scheduled jobs, exports, and direct database consumers in that exercise. A healthy web endpoint does not prove that every writer is compatible.

Use synthetic or appropriately protected test data. This checklist proposes evidence to collect; it does not claim that a production migration has been tested.

A compatible change can still block traffic

In PostgreSQL, ALTER TABLE generally acquires an ACCESS EXCLUSIVE lock unless the documentation specifies otherwise. Compatibility at the application level does not eliminate lock waits.

PostgreSQL also supports separating certain constraint additions from validation with NOT VALID and VALIDATE CONSTRAINT. The permitted constraint types and locking details matter. Check the documentation for your deployed version rather than applying the technique to every schema operation. See the PostgreSQL ALTER TABLE reference.

Before execution, define acceptable lock wait, query latency, replication lag, and backfill load. Decide who can stop the rollout and what the safe next step is if a threshold is crossed.

Make cleanup its own decision

Keep a short release note that answers four questions:

Which application versions can run against this schema?
Which data changes can the previous version understand?
What evidence allows the old column or interface to be removed?
After removal, does recovery mean application rollback, a forward fix, or data restoration?

Give the cleanup an owner. Leaving parallel paths indefinitely creates its own maintenance cost, but removing them too early destroys the compatibility you built.

The practical takeaway: rehearse rollback against data written by the new release before treating it as a recovery plan.

What is the most easily missed database consumer in your deployment process?

Waqas Ahmad | Software Engineer | Content Creator

The Reconnect Bug Where Marking a User Online Erased the Data I Needed for Replay

Thu, 15 Oct 2026 07:00:41 +0000

While testing a reconnect system for a deliverer-facing WebSocket gateway: when someone reconnects after losing their connection, they should get a snapshot of their current job plus any events they missed while offline. Two pieces, both of which I was confident were correct on their own.

Marking presence:

async markOnline(userId: number) {
  await this.prisma.user.update({
    where: { id: userId },
    data: { isOnline: true, lastSeenAt: new Date() },
  });
}

Computing what was missed:

async replayDelivererEvents(delivererId: number) {
  const deliverer = await this.prisma.delivererProfile.findUnique({
    where: { id: delivererId },
    select: { user: { select: { lastSeenAt: true } }, activeJobId: true },
  });
  // ...fetch events since deliverer.user.lastSeenAt
}

And the connection handler that used both:

async handleConnection(client: Socket) {
  const deliverer = await this.getDelivererByUserId(payload.sub);

  await this.sessionService.markOnline(payload.sub);           // step 1
  const missedEvents = await this.replayService.replayForDeliverer(deliverer.id); // step 2

  client.emit('events.replay', missedEvents);
}

I tested it: disconnect, trigger a job assignment while offline, reconnect. events.replay came back empty, every time. The event genuinely happened. The query genuinely should have found it.

Why

Walk through the actual timeline. A deliverer disconnects — markOffline correctly stamps lastSeenAt with the disconnect time. Later, they reconnect. handleConnection fires, and the first thing it does is call markOnline, which immediately overwrites lastSeenAt to right now. Only after that does it call replayForDeliverer, which reads lastSeenAt to compute "what happened since they were last seen" except by then lastSeenAt no longer holds the disconnect time. It holds the current moment, because the line above just wrote over it.

getEventsSince({ since: lastSeenAt }) effectively became since: right now. The replay feature couldn't fail more quietly — no error, no exception, just a correctly-shaped empty array every single time.

What makes this bug worth writing about isn't the fix — it's that neither piece of code was wrong in isolation. Marking someone online should update lastSeenAt. Replay should read lastSeenAt. The bug only exists in the order they ran in, and that kind of bug doesn't show up by staring at either function — only by tracing the actual sequence of operations across the whole request.

The fix
Read before you overwrite:

async handleConnection(client: Socket) {
  const deliverer = await this.getDelivererByUserId(payload.sub);

  // Replay FIRST, using the lastSeenAt from before this connection
  const missedEvents = await this.replayService.replayForDeliverer(deliverer.id);

  // THEN mark online, which overwrites lastSeenAt for next time
  await this.sessionService.markOnline(payload.sub);

  client.emit('events.replay', missedEvents);
}

No schema change needed. Reordering two already-correct operations was the entire fix.

The part worth remembering

One field, two different jobs — "when did we last hear a heartbeat" and "what's the anchor point for replay" happened to share the same column, and that's exactly what made the collision possible. The one-line reorder fixes today's bug, but it leaves the deeper coupling in place: anything else that later wants to read lastSeenAt for a different purpose can walk into the same trap. The more robust version splits them. a dedicated lastDisconnectedAt, set only on disconnect, read only by replay so two unrelated concerns stop sharing state that one of them mutates as a side effect. I haven't built that yet. It's on the list specifically because I know why it would help, not just that it might.

Calling a Service Inside prisma.$transaction Doesn't Make It Atomic — Here's the Bug That Taught Me

Mon, 12 Oct 2026 09:22:43 +0000

I was fixing a real bug in a delivery backend I'm building: a customer's order could get created in Postgres even when something downstream failed, leaving an order with no event history behind it. The fix looked simple. Wrap the whole thing in
prisma.$transaction:

const order = await this.prisma.$transaction(async (tx) => {
  const newOrder = await tx.order.create({ data: { ... } });

  await this.eventLogService.recordEvent({
    entityType: 'ORDER',
    entityId: newOrder.id,
    eventType: EVENTS.ORDER.CREATED,
    payload: { orderId: newOrder.id, status: OrderStatus.PENDING_VENDOR_CONFIRMATION },
  });

  return newOrder;
});

I ran it, broke the event-logging step on purpose to test the rollback, and the order still got committed. The transaction did nothing.

Why

$transaction(async (tx) => {...}) opens one database connection and starts a transaction on it. The tx object you get inside the callback is bound to that specific connection. Every query run through tx participates in the same transaction.

My EventLogService.recordEvent method didn't use tx. It used this.prisma — its own injected Prisma client, a completely different connection:

async recordEvent(params) {
  const event = await this.prisma.eventLog.create({ ... }); // not tx — its own connection
  ...
}

Nesting the call inside $transaction changes nothing here. Prisma has no way to know "we're inside a transaction right now, please route all queries through it." The only thing that determines whether a query participates in a transaction is which client object actually executed it. Since recordEvent ran on its own connection, its insert committed instantly and independently, with no awareness that anything else was "inside" a transaction around it.

The fix
Every method that needs to be transaction-capable has to accept an optional transaction client and fall back to the default when none is passed:

async recordEvent(params, tx?: Prisma.TransactionClient) {
  const client = tx ?? this.prisma;
  const event = await client.eventLog.create({ data: params });
  ...
}

Then the caller passes its own tx through explicitly:

await this.eventLogService.recordEvent(params, tx);

Now recordEvent's insert runs on the same connection, inside the same transaction, as everything else. If anything later in that transaction throws, Postgres rolls back the whole connection's uncommitted work — including this insert — because it genuinely was part of the same unit of work, not just nested inside it syntactically.

I retested the same failure scenario. This time, the order didn't get created either. Rollback worked as a single unit, order and event together.

The part worth remembering
Code structure and transaction membership are two different things. A function call being lexically inside a $transaction block tells you nothing about which database connection it actually used. If you're composing transactional logic across services in Prisma — or any ORM with a similar explicit-client model — every service in that chain needs to accept and thread through the transaction client deliberately. There's no implicit "ambient transaction" you can rely on.

The honest cost: this makes service signatures a bit noisier — an extra optional parameter on every method that might need to participate in a caller's transaction. I think that's a fair trade for not silently losing atomicity on exactly the kind of multi-step write where you'd assume you had it.

From React to PostgreSQL: Understanding the Full Stack

Wed, 07 Oct 2026 05:22:39 +0000

Modern web applications are no longer built with a single technology. A production application usually involves multiple layers working together: a frontend that users interact with, a backend that processes requests, APIs that connect different components, and a database that stores information.

One of the most popular combinations for building modern full stack applications is React Node.js/Express + PostgreSQL. React provides an interactive user interface, the backend handles business logic and communication, and PostgreSQL provides reliable and structured data storage.

Understanding how these technologies communicate is much more valuable than simply learning each technology separately. A developer who understands the entire request lifecycle can build applications that are easier to maintain, debug, secure, and scale.

This article explores the complete journey from React in the browser to PostgreSQL in the database and explains how all the pieces fit together.

1. What Does "Full Stack" Actually Mean?

The term full stack development refers to working across multiple layers of a web application.

A typical application can be divided into four major areas:

**Frontend What users see and interact with
**Backend Server-side logic and application processing
**API Communication between frontend and backend
**Database Persistent storage of application data

A simplified architecture looks like this:

User
  ↓
React Frontend
  ↓
HTTP Request
  ↓
Backend / API
  ↓
Business Logic
  ↓
PostgreSQL Database
  ↓
Backend Response
  ↓
React UI Update

For example, imagine a task management application.

A user opens the application and sees a list of tasks. When they click "Add Task", React collects the task information and sends it to the backend.

The backend validates the information, processes the request, and sends an SQL query to PostgreSQL.

PostgreSQL stores the task and returns the result.

The backend then sends a response to React, and React updates the interface.

The entire process may happen in milliseconds.

Understanding this complete flow is the foundation of full-stack development.

2. React: The Frontend Layer

React is a JavaScript library for building user interfaces.

Instead of creating one large HTML page, React allows developers to divide an application into reusable components.

For example:

function TaskCard({ task }) {
  return (
    <div>
      <h3>{task.title}</h3>
      <p>{task.description}</p>
    </div>
  );
}

A larger application might contain components such as:

App
├── Navbar
├── Sidebar
├── Dashboard
├── TaskList
│   └── TaskCard
└── Footer

This component-based architecture makes large applications easier to manage.

React State

React applications frequently need to manage changing information.

For example:

const [tasks, setTasks] = useState([]);

The tasks variable contains the current data, while setTasks() updates it.

When the state changes, React re-renders the relevant part of the interface.

This makes React particularly useful for interactive applications.

Fetching Backend Data

React usually communicates with a backend through HTTP requests.

For example:

fetch("https://example.com/api/tasks")
  .then(response => response.json())
  .then(data => setTasks(data));

Modern applications may also use libraries such as Axios or tools provided by frameworks.

The important concept is that React itself normally does not communicate directly with PostgreSQL.

Instead, it communicates with the backend.

That separation is extremely important.

3. Why React Should Not Connect Directly to PostgreSQL

A beginner might wonder:

"If React can make network requests, why not connect React directly to PostgreSQL?"

The answer is security and architecture.

A browser application runs on the user's device. If database credentials were placed inside React code, anyone could potentially inspect them.

For example, exposing something like this would be extremely dangerous:

const databasePassword = "my-secret-password";

Frontend code is not a secure location for database credentials.

Instead, the architecture should look like:

React
   ↓
Backend API
   ↓
PostgreSQL

The backend protects the database credentials and controls what users are allowed to access.

This also allows the backend to perform validation, authentication, authorization, business logic, and database operations.

4. The Backend: Where the Application Thinks

The backend is responsible for processing requests from the frontend.

A popular JavaScript backend combination is:

Node.js
Express.js
PostgreSQL

Node.js allows JavaScript to run outside the browser.

Express provides tools for building APIs and handling HTTP requests.

A simple Express route might look like this:

app.get("/api/tasks", async (req, res) => {
  const result = await db.query("SELECT * FROM tasks");

  res.json(result.rows);
});

When React requests:

GET /api/tasks

the backend receives the request.

It queries PostgreSQL and returns the database results as JSON.

5. Understanding APIs

An API acts as a communication layer between different parts of an application.

For a task application, we might have:

GET    /api/tasks
GET    /api/tasks/:id
POST   /api/tasks
PUT    /api/tasks/:id
DELETE /api/tasks/:id

These endpoints represent different operations.

GET

Used to retrieve information.

GET /api/tasks

POST

Used to create new information.

POST /api/tasks

PUT or PATCH

Used to update information.

PATCH /api/tasks/25

DELETE

Used to remove information.

DELETE /api/tasks/25

This pattern is commonly associated with REST APIs.

A well-designed API gives the frontend a predictable way to communicate with the backend.

6. PostgreSQL: The Data Layer

PostgreSQL is a powerful open-source relational database system.

It stores information in tables containing rows and columns.

For example, a tasks table could look like:

tasks
------------------------------------------------
id | title          | completed | created_at
------------------------------------------------
1  | Learn React    | false     | ...
2  | Build API      | true      | ...
3  | Learn PostgreSQL| false    | ...

Each row represents a record.

Each column represents a property of that record.

A PostgreSQL table might be created using:

CREATE TABLE tasks (
    id SERIAL PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    description TEXT,
    completed BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

PostgreSQL provides powerful features for data integrity, relationships, transactions, indexing, querying, and more.

7. SQL: How the Backend Talks to PostgreSQL

SQL stands for Structured Query Language.

Backend applications use SQL to interact with relational databases.

For example:

SELECT * FROM tasks;

returns tasks.

To create a task:

INSERT INTO tasks (title, description)
VALUES ('Learn Full Stack Development', 'Study React, APIs and PostgreSQL');

To update a task:

UPDATE tasks
SET completed = TRUE
WHERE id = 1;

To delete a task:

DELETE FROM tasks
WHERE id = 1;

These operations are the foundation of database-driven applications.

8. The Complete Request Lifecycle

Let's follow a real request from React to PostgreSQL.

Imagine a user submits this form:

Title: Learn PostgreSQL
Description: Understand relational databases

Step 1: React Collects the Data

React stores the form values in state.

const [title, setTitle] = useState("");
const [description, setDescription] = useState("");

When the user submits the form, React creates an HTTP request.

Step 2: React Sends the Request

fetch("/api/tasks", {
  method: "POST",
  headers: {
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    title,
    description
  })
});

The browser sends the request to the backend.

Step 3: Express Receives It

The backend route receives the request:

app.post("/api/tasks", async (req, res) => {
  const { title, description } = req.body;

  // validation and database logic
});

Step 4: Backend Validates the Data

The backend should never blindly trust frontend input.

It might check:

if (!title) {
  return res.status(400).json({
    error: "Title is required"
  });
}

Step 5: Backend Queries PostgreSQL

The server sends an SQL query:

INSERT INTO tasks (title, description)
VALUES ($1, $2)
RETURNING *;

The $1 and $2 parameters are supplied separately.

This is important because parameterized queries help protect against SQL injection.

Step 6: PostgreSQL Processes the Query

PostgreSQL validates the query, modifies the appropriate table, and returns the result.

Step 7: Backend Sends a Response

The backend may return:

{
  "id": 15,
  "title": "Learn PostgreSQL",
  "description": "Understand relational databases",
  "completed": false
}

Step 8: React Updates the UI

React receives the response and updates its state.

The new task then appears on the screen.

This is the complete full-stack cycle.

9. Authentication: Connecting Users to Data

Most real applications need authentication.

Users might register with:

Name
Email
Password

The backend stores account information in PostgreSQL.

Passwords should never be stored as plain text.

Instead, applications typically use secure password hashing algorithms such as bcrypt or Argon2.

After login, the application can issue an authentication token or session.

A common architecture looks like:

User
 ↓
React Login Form
 ↓
POST /api/login
 ↓
Backend
 ↓
Verify Password
 ↓
PostgreSQL
 ↓
Create Session/Token
 ↓
React

For protected requests, the frontend sends authentication information with the request.

The backend verifies it before allowing access to protected resources.

10. Database Relationships

Real-world applications rarely have only one table.

For example, a project management application might contain:

users
projects
tasks
comments

A user can have many projects.

A project can have many tasks.

A task can have many comments.

This creates relationships between tables.

For example:

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    email VARCHAR(255) UNIQUE NOT NULL
);

Projects can reference users:

CREATE TABLE projects (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    user_id INTEGER REFERENCES users(id)
);

The user_id creates a relationship between the project and its owner.

This is one of the most important concepts when moving from simple applications to production systems.

11. JOIN Queries

Relationships become powerful when combined with SQL joins.

For example:

SELECT
    projects.name,
    users.name AS owner
FROM projects
JOIN users
ON projects.user_id = users.id;

This can return:

Project             Owner
--------------------------------
Marketing Website   John
Mobile Application  Sarah
CRM Dashboard       David

Instead of storing everything in one giant table, relational databases allow information to be organized into connected structures.

This improves data consistency and maintainability.

12. Validation Must Happen on the Backend

Frontend validation is useful for user experience.

For example, React can tell a user:

Email is required.

But frontend validation alone is not enough.

A malicious user can bypass the React interface and send requests directly to the API.

Therefore, the backend must validate requests independently.

For example:

if (!email || !email.includes("@")) {
  return res.status(400).json({
    error: "Invalid email"
  });
}

Production applications often use validation libraries to create structured schemas.

The rule is simple:

Never trust client-side input.

13. Error Handling

Full-stack applications must also handle errors gracefully.

A database connection might fail.

A user might request a resource that doesn't exist.

A SQL query might fail.

A server might experience an unexpected error.

Instead of exposing internal errors to users, the backend should return appropriate HTTP status codes.

For example:

200 OK
201 Created
400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found
500 Internal Server Error

React can then display an appropriate message.

For example:

if (!response.ok) {
  setError("Unable to load tasks.");
}

Good error handling improves both user experience and debugging.

14. Environment Variables

Database credentials should not be hard-coded into application source code.

Instead, applications commonly use environment variables.

For example:

DATABASE_URL=postgresql://user:password@localhost:5432/myapp

The backend reads the value from the environment.

This allows different configurations for:

Development
Testing
Staging
Production

It also helps prevent sensitive credentials from being accidentally committed to Git repositories.

A .env file should generally be excluded from version control when it contains secrets.

15. ORM vs Raw SQL

Developers can communicate with PostgreSQL using raw SQL or an Object-Relational Mapping tool.

Raw SQL:

SELECT * FROM users WHERE id = $1;

ORMs provide a higher-level programming interface.

Popular choices in the JavaScript ecosystem include tools such as Prisma, Drizzle, and Sequelize.

An ORM can make common database operations easier to organize.

However, understanding SQL remains important.

Using an ORM without understanding databases can create problems when queries become complex or performance issues appear.

A strong full-stack developer should understand both the abstraction and the underlying database.

16. Database Indexing

As an application grows, database performance becomes increasingly important.

Suppose a users table contains millions of records.

A query like:

SELECT *
FROM users
WHERE email = '*Emails are not allowed*';

could become expensive without appropriate indexing.

An index can improve lookup performance:

CREATE INDEX idx_users_email
ON users(email);

Indexes are powerful, but they also have a cost.

They consume storage and can increase the cost of insert and update operations.

Therefore, indexes should be designed around actual query patterns.

17. Transactions

Some operations require multiple database changes to succeed together.

Imagine transferring money between two accounts.

You might need to:

Subtract money from Account A
Add money to Account B

If the first operation succeeds but the second fails, the database could become inconsistent.

Transactions solve this problem.

Conceptually:

BEGIN;

UPDATE accounts
SET balance = balance - 100
WHERE id = 1;

UPDATE accounts
SET balance = balance + 100
WHERE id = 2;

COMMIT;

If something goes wrong, the transaction can be rolled back.

Transactions are an important part of reliable backend systems.

18. Performance Across the Full Stack

Full-stack performance isn't just about writing fast React components.

Every layer can introduce delays.

Consider:

React rendering
       ↓
Network request
       ↓
Backend processing
       ↓
Database query
       ↓
Network response
       ↓
React rendering

If the database query takes 2 seconds, optimizing React alone won't solve the problem.

Similarly, if the API returns a massive JSON response, the frontend may still feel slow.

Performance should therefore be considered across the entire architecture.

Possible optimizations include:

React code splitting
Caching
API pagination
Database indexing
Efficient SQL queries
Connection pooling
Compression
CDN usage
Lazy loading
Server-side caching

19. Pagination

Imagine an application has 100,000 products.

Returning every product to React at once would be inefficient.

Instead, the API can use pagination.

For example:

GET /api/products?page=1&limit=20

The backend retrieves only the required records.

SQL might use:

SELECT *
FROM products
ORDER BY id
LIMIT 20 OFFSET 0;

Pagination reduces network traffic and improves application responsiveness.

For very large datasets, cursor-based pagination can provide better performance than large offsets.

20. Caching

Not every request needs to hit PostgreSQL.

Suppose thousands of users request the same popular product.

Repeatedly querying the database can create unnecessary load.

Caching can store frequently requested information temporarily.

The architecture might become:

React
 ↓
API
 ↓
Cache
 ↓
PostgreSQL

If the data exists in the cache, the backend can return it immediately.

Caching strategies need careful consideration because stale data can create incorrect results.

21. Security in a Full-Stack Application

Security must exist at every layer.

Frontend

Avoid exposing sensitive information.

Backend

Validate input and authenticate users.

API

Implement authorization and rate limiting where appropriate.

Database

Use restricted database accounts and secure credentials.

SQL

Use parameterized queries.

Authentication

Hash passwords securely and protect sessions or tokens.

Deployment

Use HTTPS and secure environment variables.

Security isn't one feature that can simply be added at the end.

It is an architectural responsibility.

22. Development Workflow

A practical development workflow might look like this:

1. Design database schema
2. Build backend API
3. Test API
4. Build React interface
5. Connect frontend to API
6. Add authentication
7. Add validation
8. Add error handling
9. Optimize database queries
10. Test complete application
11. Deploy
12. Monitor

Developers can use Git to track changes throughout the process.

A typical repository might look like:

my-fullstack-app/
│
├── frontend/
│   ├── src/
│   ├── components/
│   └── pages/
│
├── backend/
│   ├── routes/
│   ├── controllers/
│   ├── services/
│   └── middleware/
│
├── database/
│   └── migrations/
│
└── README.md

The exact structure varies between projects, but separating responsibilities helps maintainability.

23. Testing the Full Stack

Testing should happen at different levels.

Frontend Testing

Test components and user interactions.

API Testing

Test endpoints independently.

For example:

POST /api/login
GET /api/users
POST /api/tasks
DELETE /api/tasks/:id

Database Testing

Verify that queries produce the expected results.

Integration Testing

Test whether multiple components work together.

End-to-End Testing

Simulate the experience of a real user.

For example:

Open application
↓
Register
↓
Login
↓
Create task
↓
Edit task
↓
Complete task
↓
Logout

The goal isn't simply to test individual functions.

It's to verify that the complete system behaves correctly.

24. Deployment

Once the application works locally, it needs to be deployed.

A typical production architecture might look like:

User
 ↓
Frontend Hosting / CDN
 ↓
React Application
 ↓
Backend API
 ↓
PostgreSQL

The frontend and backend can be deployed separately or together depending on the architecture.

The production database should have appropriate:

Backups
Access controls
Monitoring
Connection limits
Environment configuration

Deployment is not the end of development.

Production systems require monitoring and maintenance.

25. Logging and Monitoring

When something goes wrong in production, developers need information about what happened.

Backend logging can record events such as:

Request received
User authenticated
Database query executed
Response returned
Error occurred

Monitoring tools can help identify:

Slow API responses
High database usage
Server errors
Failed requests
Memory problems
Traffic spikes

A production application without observability can be extremely difficult to maintain.

26. The Most Important Skill: Understanding the Boundaries

Learning React, Node.js, Express, and PostgreSQL individually is useful.

But the most important full-stack skill is understanding where responsibility belongs.

React should focus primarily on presentation and user interaction.

The backend should handle business rules, authentication, authorization, and API behavior.

PostgreSQL should handle persistent data and relational operations.

For example:

React:
"What did the user click?"

Backend:
"Is this action allowed?"

Database:
"What information should be stored?"

When responsibilities are clearly separated, applications become easier to understand and maintain.

27. What Happens When You Click a Button?

Let's summarize the entire process using one simple example.

A user clicks:

"Complete Task"

React detects the click.

fetch("/api/tasks/15", {
  method: "PATCH",
  body: JSON.stringify({
    completed: true
  })
});

The request travels to the backend.

Express identifies the route.

The backend verifies the user's authentication.

The backend validates the input.

The server sends a parameterized SQL query:

UPDATE tasks
SET completed = $1
WHERE id = $2
RETURNING *;

PostgreSQL executes the query.

The database returns the updated task.

The backend converts the result into JSON.

React receives the response.

React updates its state.

The interface changes from:

☐ Learn PostgreSQL

to:

☑ Learn PostgreSQL

The user sees the result almost instantly.

That tiny interaction represents the entire full stack architecture working together.

28. What Should a Beginner Learn First?

If you're starting full-stack development, don't try to master everything simultaneously.

A practical learning path is:

Step 1: HTML and CSS

Understand the structure and styling of websites.

Step 2: JavaScript

Learn variables, functions, arrays, objects, asynchronous programming, modules, and modern JavaScript.

Step 3: React

Learn:

Components
Props
State
Hooks
Forms
Routing
API requests

Step 4: Node.js and Express

Learn:

HTTP
REST APIs
Middleware
Authentication
Validation
Error handling

Step 5: PostgreSQL

Learn:

Tables
Primary keys
Foreign keys
CRUD
JOINs
Indexes
Transactions
Database design

Step 6: Full-Stack Projects

Build applications that connect everything together.

For example:

Task manager
Blog platform
Ecommerce application
CRM
Job board
Inventory system
Project management platform

Building projects is where the concepts become real skills.

29. The Future of Full-Stack Development

Modern full-stack development continues to evolve.

Developers now have access to powerful frameworks, managed databases, cloud platforms, serverless infrastructure, AI coding assistants, and automated deployment systems.

However, the fundamental architecture remains remarkably consistent:

User
 ↓
Interface
 ↓
API
 ↓
Business Logic
 ↓
Database

Tools will change.

Frameworks will change.

Programming languages will evolve.

But understanding how data moves through a software system will remain valuable.

A developer who understands the complete lifecycle of a request can adapt to new technologies much more easily.

Conclusion

Going from React to PostgreSQL is not simply about learning four different technologies.

It's about understanding how an entire software system works.

React creates the interface.

The browser sends requests.

The backend receives those requests.

Business logic determines what should happen.

The API communicates with the database.

PostgreSQL stores and retrieves information.

The backend returns a response.

React updates the interface.

This cycle happens repeatedly throughout almost every modern data-driven web application.

The real power of full stack development comes from understanding the connections between these layers.

You don't need to become an expert in everything immediately. Start with the fundamentals, build small applications, understand how each layer communicates with the next, and gradually introduce more advanced concepts such as authentication, database relationships, caching, indexing, transactions, testing, and deployment.

Once you understand the journey from a user's click in React to a SQL query in PostgreSQL and back again you stop thinking about technologies as isolated tools.

You start thinking like a full stack developer.

And that shift in perspective is one of the most valuable skills you can develop in modern software engineering.

Why I Chose a Simple Capability Token for Waitless — and When I Would Harden It

Mon, 05 Oct 2026 07:20:50 +0000

While working on Waitless, I needed customers to be able to track their position in a queue and leave the queue without creating an account.

That led to a small but interesting security design question:

How can I authorize an anonymous user without introducing a full authentication system?

My current solution is deliberately simple: each queue entry receives a random UUID token, and that token is used as a capability for the tracking link.

UUID token instead of a sequential ID

A sequential identifier could look like:

/track/123

The problem is that IDs such as 123, 124, and 125 are predictable and enumerable.

For an anonymous tracking system, I don't want a user to be able to guess another customer's resource simply by changing the ID.

Instead, an entry has a separate token:

model Entry {
  id      String @id @default(uuid())
  token   String @unique @default(uuid())

  name     String
  email    String?
  position Int
  status   EntryStatus @default(WAITING)

  queueId String
  // ...
}

The public URL therefore looks more like:

/track/550e8400-e29b-41d4-a716-446655440000

The important point is that the token is not intended to be an ordinary identifier.

It acts as a capability.

What is a capability URL?

A capability can be thought of as:

An unguessable secret that grants access to a specific resource.

In my case:

possess valid token
        ↓
access this queue entry

The capability is deliberately designed to be difficult to guess.

That is different from simply hiding a sequential ID and hoping nobody discovers it.

This distinction matters:

Security by obscurity
→ "Nobody knows the ID."

Capability-based design
→ "Access requires possession of a deliberately unguessable credential."

The second is a much more intentional security model.

Why I kept the token simple for now

Waitless is an anonymous queueing system.

A customer does not need an account, password, or traditional login just to:

see their position
receive queue updates
leave the queue

Introducing a complete authentication system would add considerable complexity to the product and to the user experience.

For the current use case, a capability link gives me a simple model:

Join queue
    ↓
Generate random token
    ↓
Give customer tracking link
    ↓
Customer presents token
    ↓
Server identifies the queue entry
    ↓
Allow limited actions

This is a conscious trade-off.

I'm choosing simplicity and frictionless anonymous access because the current resource is relatively limited in scope.

But the token is also a bearer credential

There is an important consequence.

If possession of the token grants access, then:

whoever possesses the token can use it.

That makes the token a bearer credential.

For example, the current leave() operation relies on the same tracking capability.

So if the token is leaked, copied, logged, or shared, another person could potentially use it.

That means the design is intentionally simple, but it is not the strongest possible security model.

Known limitation

The tracking token currently also functions as a bearer credential for leave(), with no rate-limiting or one-time consumption.

I consider this an accepted limitation of the current design rather than something I want to hide.

One way to harden the system would be to use the tracking link only as an initial capability.

For example:

/claim/<secret>
      ↓
validate capability
      ↓
create anonymous session
      ↓
HttpOnly + Secure cookie
      ↓
redirect to /track

After that, the token would no longer need to remain in the URL.

The browser would use the session cookie for future requests.

This would reduce some of the risks associated with repeatedly carrying a secret in a URL and would give me more control over session lifecycle, expiration and revocation.

But it introduces another trade-off.

The trade-off: security vs. convenience

The current capability link has an important usability property:

The link itself works across devices.

A customer can open it on their phone, copy it to another device, bookmark it, or reopen it later.

With a one-time capability-to-session exchange, that behavior becomes more complicated.

For example:

Phone
  ↓
claim token
  ↓
session created

Now the same one-time token may no longer be usable on another device.

Supporting multiple devices would therefore require additional identity/session handling, which increases complexity.

For a simple anonymous queue, I decided that this trade-off is not currently justified.

When would this design change?

The answer is: when the product starts having identity requirements.

Imagine Waitless later introduces a reservation system.

Now a customer might have:

Account
   ↓
Reservations
   ↓
Queue entries
   ↓
Payment history
   ↓
Cancellation rules
   ↓
Personal information

The security requirements become much stronger.

At that point, a random tracking capability may no longer be enough as the main authorization mechanism.

I would probably move toward:

User authentication
        ↓
Authenticated session
        ↓
Authorization
        ↓
Reservation / queue access

The capability token could still exist for specific use cases, but it would no longer be responsible for the entire identity model.

UUIDv4 vs. UUIDv7

This also made me think more carefully about what an identifier is supposed to do.

For database identifiers, UUIDv7 is attractive because it is time-ordered and therefore provides better index locality than completely random UUIDv4 values.

For a public capability, however, my primary requirement is different:

I care more about unpredictability than database ordering.

That's why I would keep the concepts separate:

Database ID
    ↓
UUIDv7 can be a good choice

Public capability token
    ↓
Cryptographically random token / UUIDv4

This is an example of a broader engineering principle:

The best identifier for the database is not necessarily the best credential for the client.

What about storing the token itself?

Another improvement would be to store only a hash of the capability:

raw token
    ↓
SHA-256
    ↓
tokenHash

Then the database contains the verifier rather than the original bearer secret.

The application can hash the presented token and compare the result.

This gives additional protection if the database contents are exposed.

Again, it adds some implementation complexity, so it is a hardening step rather than something I needed to introduce simply to make the current Waitless flow work.

My current decision

For the current version of Waitless:

Random capability token
        ↓
Tracking URL
        ↓
Anonymous access
        ↓
Limited actions

I deliberately chose this because it provides a very small and understandable authorization mechanism without forcing customers through registration and login.

The trade-off is that the token is a bearer credential, so the system needs to treat it as sensitive.

I have therefore documented the limitation rather than pretending the design is perfect.

My planned evolution is roughly:

Current
Random capability URL
        ↓
Simple anonymous access


Later
Rate limiting
+ expiry / revocation
+ stronger token handling


When identity becomes important
        ↓
Capability URL
        ↓
Session / authentication
        ↓
Fine-grained authorization

The interesting lesson for me was not simply “use UUIDs.”

It was learning to ask:

What is this value actually supposed to represent?

An identifier, a capability, and an authenticated identity are three different concepts.

Once that distinction is clear, the architecture becomes much easier to reason about.

Why I Used BullMQ + Redis for Background Jobs in My NestJS Application — and When I Wouldn't

Sun, 04 Oct 2026 15:45:42 +0000

While working on Waitless, a real-time queue management platform, I came across a backend problem that looked simple at first:

A customer reaches the front of the queue, and the system needs to send notifications to other customers.

The first instinct could be to simply send the email inside the HTTP request.

But once I looked at the problem as a backend architecture problem rather than just an implementation task, I realized that sending the email and handling the user's request are two different workloads.

That led me to BullMQ + Redis.

The interesting part wasn't learning the API. The interesting part was understanding why this architecture made sense, what Redis was actually doing, and when I would choose something else instead.

The problem in Waitless

Waitless allows customers to join a digital queue and track their position without creating an account.

When one customer is called, another customer may need to receive a notification such as:

“It's your turn.”
“You're almost next.”

I already had the queue state stored in PostgreSQL. The question was:

Should the API send the email immediately, or should it create background work?

A synchronous approach could look like this:

HTTP request
     ↓
NestJS
     ↓
PostgreSQL
     ↓
Send email
     ↓
HTTP response

The problem is that sending an email depends on an external service.

If that service is slow or temporarily unavailable, my API request is affected.

I don't necessarily need the email to be completed before responding to the user.

What I actually need is:

“Remember that this email has to be sent, and process it separately.”

That is exactly the kind of problem a job queue is designed to solve.

Introducing BullMQ

BullMQ provides a job queue abstraction for Node.js applications.

Instead of doing the work immediately, the application creates a job:

Application
     ↓
   Queue
     ↓
    Job
     ↓
   Worker
     ↓
 Actual work

The application that creates the job is the producer.

The worker that processes the job is the consumer/worker.

This gives me a clean separation between:

Creating work

and:

Executing work

That separation becomes particularly useful when the work is slow, retryable, delayed, or needs to be processed independently from the HTTP request.

Where Redis comes in

One thing I initially had to clarify for myself was the relationship between BullMQ and Redis.

They are not the same thing.

A simplified mental model is:

NestJS
   ↓
BullMQ
   ↓
Redis

BullMQ is the queue system.

Redis is the backend that BullMQ uses to store and coordinate queue-related state.

In my NestJS application, I configure the connection like this:

BullModule.forRoot({
  connection: {
    host: 'localhost',
    port: 6379,
  },
});

Port 6379 is Redis's default port, so this configuration tells BullMQ which Redis instance to connect to.

The important distinction is:

BullMQ is not “running on port 6379.”
BullMQ is connecting to the Redis server running there.

Creating a notification job

In Waitless, I inject the notification queue into my service:

@InjectQueue('notifications')
private notifQueue: Queue

Now I can add jobs to that queue.

For example:

if (next.email) {
  await this.notifQueue.add('your-turn-now', {
    email: next.email,
    name: next.name,
    queueName: queue.name,
  });
}

This does not mean:

“Send the email now.”

It means:

“Create a job called your-turn-now and put it into the notifications queue.”

The payload contains the information the worker will need later.

Conceptually:

NestJS
   ↓
notifQueue.add(...)
   ↓
BullMQ
   ↓
Redis
   ↓
notifications queue
   ↓
your-turn-now job

The actual email is the responsibility of the worker.

Delayed jobs

One of the features I needed was delayed notification.

If another customer is second in line, I want to notify them shortly before their turn.

My implementation looks like this:

const secondInLine =
  await this.entryRepo.findSecondWaiting(queueId, next.id);

if (secondInLine?.email) {
  await this.notifQueue.add(
    'your-turn-soon',
    {
      email: secondInLine.email,
      name: secondInLine.name,
      position: 2,
      queueName: queue.name,
      trackingUrl: `http://localhost:3000/track/${secondInLine.token}`,
    },
    {
      delay: 30_000,
    },
  );
}

The important part is:

{ delay: 30_000 }

The job doesn't need to be processed immediately.

The architecture becomes:

Customer becomes second in line
            ↓
      Create BullMQ job
            ↓
       Delayed job
            ↓
          30 sec
            ↓
        Worker picks it up
            ↓
        Send notification

This is much more reliable than simply doing:

setTimeout(() => {
  sendEmail();
}, 30_000);

A timer belongs to the application process itself. A queue gives the work an explicit lifecycle outside the original request.

Why not just use PostgreSQL for the queue?

This was one of the more interesting questions I explored.

At first glance, PostgreSQL looks like a perfectly reasonable choice.

I could create a table such as:

jobs
-------------------
id
type
payload
status
created_at
processed_at

and continuously insert and delete jobs.

The problem is that the queue workload has different characteristics from my core business data.

My PostgreSQL database is responsible for things such as:

businesses
queues
customers
queue entries
relationships
transactions

Those are my core relational business data.

The queue workload is different:

job created
job becomes active
job succeeds
job fails
job retries
job is removed

This is a high-churn operational workload.

With PostgreSQL's MVCC model, repeatedly inserting and deleting rows creates dead tuples that later have to be cleaned up by VACUUM.

That doesn't mean PostgreSQL cannot implement queues. It absolutely can.

The real architectural question is:

Do I want the queue workload competing with my transactional business workload inside the same database?

For a small application, the answer might be yes.

For a higher-throughput system, separating those workloads can become valuable.

Why Redis made sense for my queue

Redis is designed around very fast operations on in-memory data structures.

That makes it a natural fit for workloads such as:

caching
counters
temporary state
queues
rate limiting
pub/sub
fast lookups

For my notification queue, the important characteristics were:

Asynchronous
High frequency
Short-lived job state
Low latency
Retries
Delayed execution
Worker concurrency

So instead of:

PostgreSQL
   ├── business data
   └── queue workload

I could separate them:

PostgreSQL
     ↓
Business state


Redis
     ↓
BullMQ
     ↓
Background jobs
     ↓
Workers

That separation was more important to me than simply saying:

“Redis is faster.”

What I learned about workload-driven architecture

This changed the way I think about databases.

The question shouldn't simply be:

Which database is better?

A better question is:

What workload am I dealing with?

For example:

Requirement	Possible choice
Relational transactional data	PostgreSQL
Very fast temporary state	Redis
Background jobs with retries	BullMQ + Redis
Document-oriented event data	MongoDB
Large event streams and replay	Kafka
Small simple background jobs	PostgreSQL or a scheduler
Managed cloud queue	SQS or similar service

These aren't absolute rules.

They are starting points for architectural reasoning.

BullMQ gives me more than a queue

Another thing I learned is that the real value of a job system is not just putting something into a list.

BullMQ provides concepts such as:

Jobs
Workers
Retries
Backoff
Delayed jobs
Concurrency
Rate limiting
Failed jobs
Job lifecycle

That matters because real systems fail.

For example:

Worker
   ↓
Send email
   ↓
Email provider unavailable
   ↓
Job fails
   ↓
Retry
   ↓
Worker tries again

The system can therefore treat background work as something with a lifecycle instead of a simple function call.

But there is an important failure model

A queue doesn't automatically mean:

“The job runs exactly once.”

A worker can crash.

A network call can time out.

A job can be retried.

The same operation may therefore be executed more than once.

That means job handlers should often be designed with idempotency in mind.

For example, a payment operation is much more dangerous to retry blindly than sending a harmless notification.

This was an important lesson for me:

Using a queue changes the failure model of the application.

You have to think about retries, duplicates, timeouts, and partial failures.

Could I use BullMQ without Redis?

Interestingly, yes.

Modern BullMQ also provides a PostgreSQL backend.

That means the conceptual architecture can also be:

NestJS
   ↓
BullMQ
   ↓
PostgreSQL

This can make sense when I want to keep the infrastructure simpler and the queue workload is moderate.

But there is a trade-off.

Using PostgreSQL for both workloads means:

PostgreSQL
 ├── transactional business data
 └── queue workload

So the queue and business operations share database resources.

That can be perfectly reasonable for a smaller application.

For a system with large queue throughput, isolating the queue workload may become more attractive.

This is why there isn't one universally correct answer.

When I would NOT use BullMQ + Redis

This was probably the most useful part of the exercise.

I don't want to add Redis and BullMQ just because they are popular technologies.

For something like:

const result = calculateTax(data);

there is no reason to create a queue.

If a task takes a few milliseconds and the user needs the result immediately, synchronous processing is simpler.

I also might avoid BullMQ for a very small scheduled task:

Every day at 08:00
Generate one report

A simple scheduler could be enough.

And if the application already uses PostgreSQL and has only a small number of background jobs, a PostgreSQL-backed solution may be simpler than adding another infrastructure component.

The lesson is:

Don't introduce distributed infrastructure without a workload that justifies it.

The decision I now make before choosing a queue

Instead of thinking:

“Should I use BullMQ?”

I ask:

1. Does this work need to happen asynchronously?

If yes, continue.

2. Does it need retries or failure recovery?

If yes, a real job system becomes more useful.

3. Does it need delayed execution?

For example:

Do this after 30 seconds.

4. Does it need multiple workers or concurrency?

For example:

Queue
 ├── Worker 1
 ├── Worker 2
 └── Worker 3

5. Does the workload justify another infrastructure component?

This is the trade-off question.

Only after answering those questions would I choose the technology.

My current mental model

The biggest thing I learned from implementing this feature is that architecture is less about memorizing technologies and more about matching workloads to the right tools.

For Waitless, my current architecture can be thought of as:

                    Waitless
                       │
          ┌────────────┼────────────┐
          ↓            ↓            ↓
     PostgreSQL      Redis       MongoDB
          │            │            │
   business state    jobs      event history
                    BullMQ
                       │
                    Workers
                       │
                   Email/API

PostgreSQL answers:

What is the current business state?

Redis + BullMQ answers:

What work needs to be processed asynchronously?

MongoDB answers:

What happened in the event history, and what can I derive from those events?

The important thing is not that I used three technologies.

The important thing is that each technology was chosen for a different workload.

Final takeaway

The best lesson I took from this wasn't actually “how to use BullMQ.”

It was this:

Before choosing a technology, understand the workload.

Ask:

What problem am I solving?
What are the access patterns?
What guarantees do I need?
How does the system fail?
What are the trade-offs?
What simpler alternative could work?

Once I started thinking this way, Redis, BullMQ, PostgreSQL, MongoDB and other technologies stopped looking like isolated tools.

They became different tools for different engineering problems.

And that is the mindset I want to continue developing as I work toward designing larger backend systems.

Next.js PostgreSQL: Build a Complete Full Stack Application

Fri, 02 Oct 2026 15:53:08 +0000

Building a modern full stack application is about much more than creating attractive user interfaces. A successful application needs a reliable frontend, a powerful backend, secure authentication, efficient database management, and a smooth user experience.

Among the many technologies available today, Next.js and PostgreSQL provide a powerful combination for building scalable, maintainable, and production-ready web applications.

Next.js offers a modern React-based framework with server-side rendering, routing, server components, and backend capabilities. PostgreSQL provides a reliable relational database for storing, managing, and retrieving structured information.

When these technologies work together, developers can create applications such as task management platforms, project management dashboards, ecommerce systems, customer relationship management tools, and SaaS products.

In this article, we will explore how to build a complete full stack application using Next.js and PostgreSQL, covering architecture, database design, authentication, CRUD operations, API development, security, testing, and deployment.

The goal is not simply to make an application work. It is to understand how different parts of a real-world software system communicate with one another.

1. Understanding the Technology Stack

Before starting development, it is important to understand the responsibilities of each technology.

What Is Next.js?

Next.js is a React framework designed for building modern web applications. It provides several features that simplify frontend and backend development.

Important features include:

File-based routing
Server Components
Client Components
Server-side rendering
Static site generation
API route handlers
Server Actions
Image optimization
Built-in performance improvements

One of the biggest advantages of Next.js is that developers can build frontend interfaces and server-side functionality within a single project.

Instead of maintaining completely separate frontend and backend repositories for every application, developers can organize much of the application logic in one codebase.

What Is PostgreSQL

PostgreSQL is an open-source relational database management system known for reliability, data integrity, and advanced querying capabilities.

It supports:

Relational tables
Primary and foreign keys
Transactions
Indexing
Complex SQL queries
JSON and JSONB data
Constraints
Database relationships

For applications involving users, products, orders, tasks, payments, or business records, PostgreSQL offers a structured way to manage information.

Why Combine Next.js and PostgreSQL

The combination provides a clear separation of responsibilities.

Next.js handles the user interface, application routing, and server-side logic, while PostgreSQL manages persistent application data.

A typical architecture looks like this:

User → Next.js Frontend → Server Logic → PostgreSQL Database

When a user creates a task, for example, the frontend sends the information to the server. The server validates the request, checks authorization, and executes a database operation. PostgreSQL stores the task, and the application returns the updated information to the user.

This architecture is suitable for applications that need reliable data handling and room for future growth.

2. Planning the Application Before Coding

Let's build a practical project: a Task Management Application.

Users will be able to register, log in, create tasks, update task details, mark tasks as completed, and delete records.

Core Features

Our application will include:

User registration and login
Secure authentication
Personal task dashboard
Create, read, update, and delete operations
Task status management
PostgreSQL database integration
Form validation
Responsive design
Error handling
Production deployment

Planning these features before writing code helps prevent unnecessary complexity.

It also allows us to identify the database tables, relationships, routes, and security requirements early.

3. Setting Up the Next.js Project

First, ensure that Node.js and npm are installed.

Create a new Next.js application using:

npx create-next-app@latest next-postgres-app

During setup, choose TypeScript, ESLint, Tailwind CSS, and the App Router.

Move into the project directory:

cd next-postgres-app

Start the development server:

npm run dev

Open the application in your browser:

http://localhost:3000

You should now see the default Next.js application.

Recommended Project Structure

A clean folder structure makes a full stack project easier to maintain.

next-postgres-app/
│
├── app/
│   ├── dashboard/
│   │   └── page.tsx
│   ├── login/
│   │   └── page.tsx
│   ├── register/
│   │   └── page.tsx
│   ├── api/
│   │   └── tasks/
│   │       └── route.ts
│   ├── layout.tsx
│   └── page.tsx
│
├── components/
│   ├── Navbar.tsx
│   ├── TaskForm.tsx
│   └── TaskList.tsx
│
├── lib/
│   ├── db.ts
│   └── auth.ts
│
├── prisma/
│   └── schema.prisma
│
├── public/
├── .env
└── package.json

This structure separates pages, reusable components, database utilities, and application logic.

4. Connecting PostgreSQL to Next.js

There are several ways to connect PostgreSQL to a Next.js application.

Developers can use:

Prisma ORM
Drizzle ORM
node-postgres (pg)
Raw SQL queries

For this tutorial, we will use Prisma because it provides schema management, type-safe database queries, and migration support.

Install Prisma:

npm install prisma @prisma/client

Initialize Prisma:

npx prisma init

This creates a Prisma configuration and schema file.

Configure the Database Connection

Inside the .env file, define your PostgreSQL connection string:

DATABASE_URL="postgresql://postgres:password@localhost:5432/taskdb"

Replace the username, password, host, and database name with your actual PostgreSQL configuration.

Never commit real database credentials to a public GitHub repository.

For production, use environment variables supplied by your hosting provider.

5. Designing the PostgreSQL Database

Database design is one of the most important parts of full stack development.

A poorly structured database can create duplicate records, inconsistent information, and difficult queries.

For our application, we need two primary entities:

User
Task

Each user can have multiple tasks, but each task belongs to one user.

This is a one-to-many relationship.

Prisma Database Schema

Open prisma/schema.prisma and define the models:

model User {
  id        Int      @id @default(autoincrement())
  name      String
  email     String   @unique
  password  String
  tasks     Task[]
  createdAt DateTime @default(now())
}

model Task {
  id          Int      @id @default(autoincrement())
  title       String
  description String?
  completed   Boolean  @default(false)
  userId      Int
  user        User     @relation(fields: [userId], references: [id], onDelete: Cascade)
  createdAt   DateTime @default(now())
}

The User model stores account information, while the Task model stores task details.

The userId field connects each task to its owner.

The unique constraint on email prevents multiple accounts from using the same email address.

The cascade delete behavior ensures that associated tasks are removed when their user record is deleted. In an actual production system, account deletion policies should be designed carefully.

Run the Database Migration

Execute:

npx prisma migrate dev --name init

Generate the Prisma Client:

npx prisma generate

Your database tables are now ready.

6. Creating a Reusable Database Client

Instead of creating a new database connection in every file, we can create a reusable Prisma client.

Inside lib/db.ts:

import { PrismaClient } from "@prisma/client";

const globalForPrisma = globalThis as unknown as {
  prisma: PrismaClient | undefined;
};

export const prisma =
  globalForPrisma.prisma ?? new PrismaClient();

if (process.env.NODE_ENV !== "production") {
  globalForPrisma.prisma = prisma;
}

This pattern helps prevent unnecessary Prisma client instances during development hot reloads.

It is especially useful when building applications with multiple server-side modules.

7. Building the First User Interface

Now let's create a simple homepage.

Inside app/page.tsx:

export default function Home() {
  return (
    <main className="min-h-screen flex flex-col items-center justify-center">
      <h1 className="text-4xl font-bold">
        Task Management App
      </h1>

      <p className="mt-4 text-gray-600">
        Organize your work and manage your daily tasks.
      </p>

      <a
        href="/dashboard"
        className="mt-6 rounded bg-blue-600 px-6 py-3 text-white"
      >
        Open Dashboard
      </a>
    </main>
  );
}

This creates a basic landing page.

In a production application, we could expand it with navigation, authentication links, feature descriptions, and responsive layouts.

A useful principle is to keep reusable UI elements inside the components directory instead of repeating the same code across multiple pages.

8. Implementing CRUD Operations

CRUD stands for:

Create
Read
Update
Delete

These operations form the foundation of many database-driven applications.

Let's implement them using Next.js Route Handlers and Prisma.

Create a Task API

Create app/api/tasks/route.ts.

import { NextResponse } from "next/server";
import { prisma } from "@/lib/db";

export async function GET() {
  try {
    const tasks = await prisma.task.findMany({
      orderBy: {
        createdAt: "desc",
      },
    });

    return NextResponse.json(tasks);
  } catch {
    return NextResponse.json(
      { error: "Failed to fetch tasks" },
      { status: 500 }
    );
  }
}

export async function POST(request: Request) {
  try {
    const body = await request.json();

    if (
      typeof body.title !== "string" ||
      !body.title.trim()
    ) {
      return NextResponse.json(
        { error: "A valid title is required" },
        { status: 400 }
      );
    }

    const task = await prisma.task.create({
      data: {
        title: body.title.trim(),
        description: body.description ?? null,
        userId: 1,
      },
    });

    return NextResponse.json(task, { status: 201 });
  } catch {
    return NextResponse.json(
      { error: "Failed to create task" },
      { status: 500 }
    );
  }
}

This demonstrates how the frontend communicates with the database through a server-side endpoint.

The userId: 1 value is only a temporary development placeholder. A real application must obtain the authenticated user's ID from a verified server-side session and authorize every operation.

Reading Tasks

The GET handler retrieves tasks from PostgreSQL.

The orderBy option sorts tasks by creation date, placing the newest records first.

For a real multi-user application, the query should filter by the authenticated user's ID.

Updating a Task

A PATCH endpoint can update an existing task.

const updatedTask = await prisma.task.update({
  where: {
    id: taskId,
  },
  data: {
    completed: true,
  },
});

The application should validate the task ID and verify that the authenticated user owns the task before updating it.

Deleting a Task

Prisma provides a simple deletion method:

await prisma.task.delete({
  where: {
    id: taskId,
  },
});

In a production system, authorization checks are essential to prevent users from deleting other users' data.

9. Building the Frontend Dashboard

A dashboard is where users interact with application data.

A useful task dashboard should provide:

A task creation form
A list of existing tasks
Completion indicators
Edit and delete buttons
Loading states
Error messages
Empty-state guidance

The interface should remain understandable even when the database is empty or a request fails.

For example, instead of showing a blank screen, display:

"No tasks yet. Create your first task to get started."

This small detail improves usability.

Example React Component

"use client";

import { useState } from "react";

export default function TaskForm() {
  const [title, setTitle] = useState("");
  const [loading, setLoading] = useState(false);

  async function handleSubmit(
    event: React.FormEvent<HTMLFormElement>
  ) {
    event.preventDefault();

    if (!title.trim()) return;

    setLoading(true);

    try {
      const response = await fetch("/api/tasks", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ title }),
      });

      if (!response.ok) {
        throw new Error("Unable to create task");
      }

      setTitle("");
    } catch (error) {
      console.error(error);
    } finally {
      setLoading(false);
    }
  }

  return (
    <form onSubmit={handleSubmit}>
      <input
        value={title}
        onChange={(event) => setTitle(event.target.value)}
        placeholder="Enter task title"
      />

      <button disabled={loading}>
        {loading ? "Saving..." : "Add Task"}
      </button>
    </form>
  );
}

This component demonstrates controlled form input, asynchronous requests, and loading-state management.

A complete dashboard would also refresh its task list after a successful submission.

10. Implementing Authentication

Authentication is a critical part of applications that store personal information.

Users should only be able to access their own records.

A typical authentication workflow includes:

User registration
Password validation
Secure password hashing
Login verification
Session creation
Protected routes
Logout functionality

Passwords must never be stored as plain text.

Use a password hashing algorithm such as Argon2id or bcrypt.

For a production application, a maintained authentication solution such as Auth.js can help manage sessions and authentication flows.

Protecting User Data

Authentication alone is not enough.

Every database operation should check authorization.

For example:

const tasks = await prisma.task.findMany({
  where: {
    userId: authenticatedUser.id,
  },
});

This ensures that the query retrieves records associated with the authenticated account.

The server must derive authenticatedUser.id from a verified session, not from a user-controlled request body.

This distinction is essential for preventing insecure direct object reference vulnerabilities.

11. Form Validation and Error Handling

Users do not always submit valid information.

They may leave fields empty, enter incorrect values, or submit unexpected data.

A robust application should validate data on both the client and server.

Libraries such as Zod can help define reusable validation schemas.

Example:

import { z } from "zod";

export const taskSchema = z.object({
  title: z.string().trim().min(1).max(120),
  description: z.string().max(1000).optional(),
});

The server should validate incoming data before interacting with PostgreSQL.

Good error handling should also distinguish between:

Invalid user input
Authentication failures
Authorization failures
Missing records
Database errors
Unexpected server failures

Avoid returning raw database errors or sensitive implementation details to users.

12. Improving Database Performance

As an application grows, database performance becomes increasingly important.

A query that works perfectly with 20 records may become slow with 200,000 records.

Several techniques can improve performance.

Indexing

PostgreSQL indexes help speed up queries that frequently filter or sort by particular fields.

For example, if the application frequently retrieves tasks by user ID, an index on userId may help.

CREATE INDEX idx_tasks_user_id
ON "Task" ("userId");

Indexes improve some read operations but introduce storage and write overhead, so they should be selected based on actual query patterns.

Pagination

Avoid loading thousands of records at once.

Instead, implement pagination.

For example:

const tasks = await prisma.task.findMany({
  skip: 0,
  take: 20,
  orderBy: {
    createdAt: "desc",
  },
});

For large datasets, cursor-based pagination can provide more consistent performance.

Connection Pooling

Production applications may handle many simultaneous requests.

Database connection pooling helps control the number of active connections and avoid exhausting PostgreSQL resources.

The appropriate pooling strategy depends on the deployment platform and database provider.

13. Testing the Application

Testing is not an optional step in professional development.

It helps identify problems before users encounter them.

A complete testing strategy can include:

Unit Testing: Test individual functions, validation rules, and utility logic.

Integration Testing: Verify that API routes interact correctly with PostgreSQL.

End-to-End Testing: Simulate user journeys such as registration, login, task creation, and deletion.

Tools such as Vitest, Jest, and Playwright can support these workflows.

Important scenarios to test include:

Creating a task with valid information
Rejecting an empty title
Preventing unauthorized access
Updating an existing task
Handling a missing task
Displaying errors when the database is unavailable

Testing helps make future code changes safer and easier.

14. Deploying the Application

After completing development and testing, the next step is deployment.

A common deployment setup includes:

Next.js application hosted on Vercel
PostgreSQL hosted on a managed database platform
Environment variables configured securely
Database migrations applied during deployment

Before deploying, verify:

Production database credentials are configured.
Authentication secrets are secure.
Database migrations have been applied.
Sensitive environment files are excluded from Git.
Error handling is configured.
The application builds successfully.
Database backups are available.

For production migrations, use the appropriate deployment migration command, such as:

npx prisma migrate deploy

Do not use development migration workflows against a live production database.

After deployment, monitor application logs, database performance, and error rates.

15. Common Challenges Developers Face

Even with modern tools, full stack development involves challenges.

Database Connection Errors

Incorrect credentials, unavailable database servers, or connection limits can cause failures.

Always verify environment configuration and database connectivity.

Authentication Problems

Improper session validation can expose private information.

Use established authentication libraries and enforce authorization on the server.

Slow Queries

Unoptimized database queries can affect response times.

Use query analysis, appropriate indexes, pagination, and efficient data retrieval.

Frontend and Backend Mismatches

The frontend may expect a different response structure than the backend provides.

TypeScript types, validation schemas, and consistent API contracts help reduce these problems.

Deployment Differences

An application that works locally may fail in production because of missing environment variables, migration issues, or incompatible runtime configuration.

Testing the production build before deployment helps identify these issues earlier.

16. How to Extend This Project

Once the basic task management application is complete, developers can add more advanced features.

Role-Based Access Control

Introduce roles such as administrator, manager, and standard user.

Each role can have different permissions.

File Uploads

Allow users to attach documents or images to tasks using an object storage service.

Real-Time Updates

Use WebSockets or a suitable real-time service to synchronize task changes between users.

Email Notifications

Send reminders when task deadlines approach.

Analytics Dashboard

Display completed tasks, pending tasks, productivity trends, and activity summaries.

AI-Powered Features

Integrate an AI service to summarize tasks, suggest priorities, or convert natural-language instructions into structured task descriptions.

These extensions transform a basic CRUD project into a more complete software product.

17. Lessons Learned from Building with Next.js and PostgreSQL

Building a full stack application teaches several important engineering principles.

First, application architecture matters. Separating database access, validation, UI components, and authentication makes code easier to understand.

Second, data security must be considered from the beginning rather than added at the end.

Third, a working application is not necessarily a production-ready application. Performance, testing, monitoring, accessibility, and deployment all contribute to reliability.

Finally, choosing technologies is only one part of software engineering. Understanding how they interact is what allows developers to build useful products.

Conclusion

Next.js and PostgreSQL provide a practical foundation for building modern full stack applications.

Next.js simplifies routing, rendering, and server-side application logic, while PostgreSQL provides reliable relational data storage.

By combining these technologies with Prisma, TypeScript, secure authentication, validation, and thoughtful database design, developers can create applications that are easier to maintain and extend.

Our task management example demonstrates the essential building blocks: project setup, database modeling, CRUD operations, frontend development, authentication, performance optimization, testing, and deployment.

Whether you are a beginner learning full stack development or an experienced developer expanding your technical portfolio, building a complete application from start to finish is a valuable way to strengthen your engineering skills.

The real achievement is not simply writing code that runs. It is creating a secure, reliable, and useful application that solves an actual problem.

**What would you build first with Next.js and PostgreSQL: a task management system, an ecommerce platform, or a SaaS dashboard? Share your thoughts and project ideas in the comments.

PostgreSQL with pgvector vs Vector DBs: Why Almost Nobody Needs Pinecone

Tue, 22 Sep 2026 06:24:46 +0000

Your team just signed up for a dedicated vector database costing $300 per month to index 50,000 customer support documents. The dashboard looks sleek, the documentation promises scalability to billions of vectors, and the product team celebrates that you are now officially "AI-native."

Yet in the shadows of your infrastructure, an operational nightmare has just been born: you now have two sources of truth. Every time a user updates a document in your primary relational database, an asynchronous synchronization job must push the update to the external vector database. If that sync job fails in the middle of the night, your RAG pipeline serves outdated or nonexistent information. You have broken ACID transactional consistency, doubled your storage overhead, added network latency to every query, and fragmented your security model.

All of this to index 50,000 vectors that your existing PostgreSQL instance could query in 8 milliseconds with a single line of SQL and zero additional cost.

In line with the engineering decisions we have championed across this blog — from separation of concerns in RAG: 7 Anti-Patterns to the pragmatic simplicity of our Productivity Stack 2026 with Supabase —, this article breaks down the most polarizing data infrastructure debate in applied AI: when do you genuinely need a specialized Vector DB (Pinecone, Qdrant, Milvus, Weaviate), and when is PostgreSQL with pgvector the vastly superior engineering choice?

The Vector Database Gold Rush

To understand how we arrived here, we must look back at the generative AI explosion of 2023–2024. When software developers discovered that converting text into multidimensional numerical representations (embeddings) enabled semantic search via geometric proximity, an immediate infrastructure question arose: where do we store and query these 1,536-dimensional vectors?

A wave of deep-tech startups raised hundreds of millions in venture capital, promising specialized vector search engines engineered from scratch for linear algebra and Approximate Nearest Neighbor (ANN) search. Pinecone, Qdrant, Chroma, Weaviate, and Milvus were born.

These dedicated vector databases did an extraordinary job evangelizing semantic search across the industry. But they made a fatal foundational assumption: they assumed that battle-tested relational database engines would be too slow to adapt to the AI era.

They were entirely wrong. In the open-source ecosystem, the pgvector extension transformed PostgreSQL — the most robust, mature, and widely deployed database engine on earth — into a world-class vector search engine.

The Hidden Cost of Dual-Database Architecture

Introducing a dedicated vector database is never just another monthly line item on your cloud bill. It is an architectural coupling decision that introduces four critical points of failure:

1. The Dual-Write Problem

When business data lives in two disconnected systems (your primary SQL database and your external Vector DB), every data mutation must write to both. What happens if the write to PostgreSQL succeeds, but the API call to Pinecone times out? State drifts out of sync immediately. To fix this, engineering teams are forced to build distributed event queues (Kafka, RabbitMQ), Outbox patterns, or complex Change Data Capture (CDC) pipelines, adding hundreds of lines of glue code and failure points.

2. Loss of ACID Transactions

In PostgreSQL, a BEGIN ... COMMIT block guarantees that operations are atomic, consistent, isolated, and durable. Storing vectors inside the same table or in a foreign-key relation in PostgreSQL ensures that deleting a document and deleting its embedding happen in the exact same atomic transaction. With an external Vector DB, eventual consistency is the absolute best you can achieve.

3. Distributed Network Latency

A realistic RAG query rarely searches raw vectors alone; it filters by relational metadata: "find the most relevant chunks from the technical manual, but only for version 2.4, created after January 2026, and belonging to user X's tenant."

In a split architecture, the query flow is tortuous:

Your backend queries PostgreSQL to retrieve the authorized tenant document IDs.
Your backend sends those IDs over the public internet as a filter payload to the external Vector DB (adding 50–150 ms of network latency).
The Vector DB executes vector similarity and returns chunk IDs.
Your backend round-trips back to PostgreSQL to retrieve the full raw text and relational metadata.

With pgvector, this entire workflow resolves in a single SQL query within the exact same database process.

4. Fragmented Security and Row Level Security (RLS)

As we explored in Prompt Injection and the EU AI Act, Row Level Security (RLS) is a non-negotiable defensive barrier for multi-tenant applications. In Supabase PostgreSQL, native RLS policies ensure that a user can never retrieve embeddings belonging to another organization, because authorization is enforced directly inside the database kernel. With an external Vector DB, you must replicate and maintain complex authorization logic across multiple application layers.

Inside pgvector Mechanics: HNSW vs IVFFlat

To operate pgvector in production with engineering confidence, you must understand how it indexes and traverses vector spaces. pgvector supports the industry's two dominant indexing algorithms:

-- Enable the pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create a table with a 1536-dimensional vector column (OpenAI / Vertex AI)
CREATE TABLE document_sections (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    article_slug TEXT NOT NULL,
    section_title TEXT NOT NULL,
    content TEXT NOT NULL,
    category TEXT NOT NULL,
    embedding VECTOR(1536),
    created_at TIMESTAMPTZ DEFAULT NOW()
);

1. IVFFlat Index (Inverted File Flat)

IVFFlat partitions the high-dimensional vector space into $K$ clusters or inverted lists using k-means clustering. At query time, the search algorithm identifies the nearest centroids and only scans vectors residing in those selected lists.

Pros: Fast build times and minimal RAM utilization.
Cons: Requires existing data before creating the index so centroids can be accurately calculated. Lower recall in high-dimensional spaces.

2. HNSW Index (Hierarchical Navigable Small World)

HNSW constructs a multi-layered hierarchical graph where vertices represent vectors and edges connect near neighbors. Search starts at the top layer with broad hops and descends into denser layers for high-precision local search.

-- Create an HNSW index optimized for cosine similarity distance
CREATE INDEX ON document_sections 
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

Pros: Outstanding search recall (>98%), ultra-fast query execution (a few milliseconds), and dynamic index building: new vectors can be inserted in real time without retraining.
Cons: Higher memory consumption and longer index build times compared to IVFFlat.

For the vast majority of production workloads, HNSW is the default recommended choice.

The Superpower: Hybrid SQL + Vector Queries

The most decisive advantage of pgvector over any standalone vector database is the ability to combine full relational algebra, JSONB operators, temporal filters, and vector similarity within a single, unified SQL statement:

-- Hybrid query in Supabase / PostgreSQL
SELECT 
    id,
    section_title,
    content,
    1 - (embedding <=> $1) AS cosine_similarity
FROM document_sections
WHERE category = 'Artificial Intelligence'
  AND created_at >= NOW() - INTERVAL '6 months'
ORDER BY embedding <=> $1
LIMIT 5;

The <=> operator computes cosine distance in native C directly at the CPU level. In a single execution plan, PostgreSQL applies the relational predicate filter (category and timestamp) and performs nearest-neighbor search across the filtered subset, returning results in under 10 milliseconds.

Replicating this in a standalone Vector DB requires synchronizing all metadata, dealing with pre-filtering or post-filtering trade-offs that hurt recall, and paying the latency tax of multiple network hops.

When You ACTUALLY Need a Dedicated Vector DB

Engineering integrity requires acknowledging when specialized tools outperform general-purpose engines. These are the specific architectural scenarios where a dedicated vector database (Qdrant, Milvus, Pinecone) is genuinely justified:

Scenario	Use PostgreSQL (`pgvector`)	Use Dedicated Vector DB
Vector Volume	< 10 million vectors	> 50–100 million vectors
Data Architecture	Monolith or service with existing relational DB	Massive decoupled search infrastructure
Required Filtering	Complex (JOINs, JSONB, RLS, relational permissions)	Simple (basic key-value tag filters)
Hardware / Memory	Standard server with balanced RAM/SSD	Specialized cluster optimized purely for RAM/GPU
Operational Overhead	$0 extra (included in your PostgreSQL instance)	$100 – $2,000+/mo for managed clusters
Massive Horizontal Sharding	Standard PostgreSQL table partitioning	Native distributed sharding across dozens of nodes

Unless your company is indexing the entire Amazon product catalog or billions of posts from a global social network, you are well within pgvector territory. For 95% of enterprise SaaS applications, internal RAG systems, and AI agent platforms, PostgreSQL handles vector workloads effortlessly.

Real Case Study: The Ops Copilot on Supabase

In our Productivity Stack 2026, we documented how we run Datalaria's infrastructure on Supabase (managed PostgreSQL).

When we built the Ops Engineering Copilot to enable readers to semantically query over 70 blog posts:

Each article is chunked into logical markdown sections (avoiding RAG Anti-Pattern 1).
Embeddings are generated using Gemini / Vertex AI (text-embedding-004).
Vectors and text are stored directly in a PostgreSQL table with an HNSW index via pgvector.
When a user asks a question, an RPC stored procedure executes cosine similarity and returns enriched context in less than 12 ms.

Total extra infrastructure cost: $0. Zero sync pipelines. Zero extra servers to monitor. 100% transactional integrity.

Connection with AI Economics and the EU AI Act

This architecture aligns directly with the 10x Rule from The Hidden Economics of AI: never adopt an external tool that adds operational friction and recurring costs unless it delivers a 10x better outcome. A dedicated vector database does not deliver a 10x better result for a corpus of 100,000 documents; it delivers the exact same semantic recall with 300% more technical debt.

Furthermore, under the EU AI Act (Article 10 on data governance and Article 12 on auditability), maintaining business records, user permissions, and embeddings in a unified database radically simplifies compliance audits. You don't have to document how data travels between separate vendors, nor do you struggle with GDPR Right to be Forgotten requests: a simple DELETE FROM users WHERE id = X cascades immediately to wipe the user, their documents, and all associated embeddings in one atomic transaction.

Conclusion: The Beauty of Engineering Simplicity

Modern software engineering suffers from a chronic temptation to collect specialized databases like trading cards. Every new paradigm seems to demand a new database, a new framework, and a new SaaS subscription.

Yet true engineering elegance is never about accumulating complexity; it is about achieving maximum capability with the minimum failure surface.

PostgreSQL has evolved continuously for over 30 years. It absorbed JSON (eliminating document databases for most use cases), absorbed geospatial data with PostGIS, and with pgvector, it has completely absorbed modern vector search.

Before opening your company credit card for another managed vector service, open a terminal to your PostgreSQL instance, run CREATE EXTENSION vector;, and test it yourself. The simplest solution is almost always the most resilient.

Sources of Interest:

Project LifeOps (Part 3): Core Interactive Modules: Fitness, Library, Cinema, and Professional Kanba

Tue, 15 Sep 2026 06:12:43 +0000

In earlier installments, we engineered the foundational backbone of LifeOps: in Part 1 we designed the polymorphic PostgreSQL schema and modular FastAPI backend, and in Part 2 we brought the Glassmorphism Dark Mode design system and global 360° Dashboard to life.

Now comes the most defining phase of development: the day-to-day functional modules.

A personal operating system fails if it forces users into rigid input forms or scatters daily data across disconnected views. The challenge was articulating three completely disparate life areas under unified design and performance standards:

🏃 Health & Physical Fitness: Workout logging across sports with dynamic metrics (distance volume, calories, pace calculations, and Personal Bests).
📚 Intellectual Culture & Media: Book tracking with interactive reading progress bars and an entertainment catalog organized by streaming platform.
💼 Professional Execution: An agile 4-column Kanban board mapped to projects, equipped with a chronological progress log powered by PostgreSQL JSONB.

[!TIP]
Test the live app: You can explore the production build of LifeOps directly at https://datalaria.com/apps/lifeops/.

🗺️ LifeOps Series Roadmap

To understand how every architectural layer fits together, this series spans 5 structured installments:

🟢 Part 1: Personal Operating System Architecture and FastAPI + Supabase Backend
🟢 Part 2: React Frontend with Glassmorphism, 360° Dashboard, and Design System
🟢 Part 3 (This article): Core Interactive Modules: Fitness, Library, Cinema, and Professional Kanban Board
⚪ Part 4: In-Memory Word Dossiers (.docx) & Multi-Sheet Excel Engine (.xlsx)
⚪ Part 5: 24/7 Zero-Cost Cloud Deployment ($0/month), Mobile UX, and PWA

1. The UX Dilemma: Dual View Mode (Inspiring Cards vs. Synthesized Table) 🎴📊

When building data-intensive dashboards, designers face a classic user experience tradeoff:

The Grid / Cards View: Visually engaging, spacious, and inspiring. It highlights book covers, movie posters, and workout chips at a glance.
The Data Table View: Irreplaceable when auditing timestamps, comparing heart rates, or scanning through an entire month of logs without endless vertical scrolling.

Instead of forcing a single opinionated view, I built a persistent dual-mode toggle stored in localStorage:

const [viewMode, setViewMode] = useState(() => {
  return localStorage.getItem('lifeops_view_sport') || 'grid';
});

const handleViewChange = (mode) => {
  setViewMode(mode);
  localStorage.setItem('lifeops_view_sport', mode);
};

In the header of each module, two unobtrusive buttons with LayoutGrid and Table icons toggle the presentation mode instantly. Thanks to React and Vite's build architecture, view switches execute in sub-milliseconds without triggering redundant server roundtrips.

2. Fitness & Performance Module (`SportModule.jsx`) 🏃💨

The fitness module caters to multi-sport athletes practicing running, road cycling, gym strength sessions, and swimming.

┌────────────────────────────────────────────────────────────────────────┐
│  🏃 Sport & Fitness                    [ + Log Workout ]       [⊞|≡]   │
├─────────────────┬─────────────────┬─────────────────┬──────────────────┤
│ TOTAL DISTANCE  │ ACTIVE TIME     │ CALORIES        │ PERSONAL BESTS   │
│   142.5 km      │   18.4 hours    │   12,450 kcal   │    4 PBs 🏅      │
├─────────────────┴─────────────────┴─────────────────┴──────────────────┤
│                                                                        │
│ ┌───────────────────────────┐         ┌──────────────────────────────┐ │
│ │ 🏃 RUNNING         🏅 PB  │         │ 🚴 CYCLING                   │ │
│ │ Sunday Long Run           │         │ Mountain Pass Century Ride   │ │
│ │ 2026-09-10                │         │ 2026-09-08                   │ │
│ │ [📍 21.1 km] [⏱️ 1h 48m]  │         │ [📍 65.0 km] [⏱️ 2h 45m]     │ │
│ │ [🔥 1,420 kcal] [❤️ 152bpm]│        │ [🔥 1,890 kcal] [⛰️ +950m]    │ │
│ └───────────────────────────┘         └──────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┘

2.1. Aggregated High-Level KPIs

Before rendering individual activity items, we compute cumulative totals using in-memory array reducers:

const totalKm = workouts.reduce((sum, act) => sum + (act.workout?.distance_km || 0), 0);
const totalMinutes = workouts.reduce((sum, act) => sum + (act.duration_minutes || 0), 0);
const totalCalories = workouts.reduce((sum, act) => sum + (act.workout?.calories || 0), 0);
const pbCount = workouts.filter((act) => act.workout?.personal_best).length;

2.2. Segregated FastAPI Payload

When an athlete records an exercise, the client dispatches a structured contract that FastAPI parses into the master table lifeops.activities and the child table lifeops.workouts:

const payload = {
  activity: {
    activity_type: 'sport',
    title: formData.title,
    date: formData.date,
    duration_minutes: parseInt(formData.duration_minutes),
    description: formData.notes,
  },
  workout: {
    workout_type: formData.workout_type, // 'running' | 'cycling' | 'gym'
    distance_km: parseFloat(formData.distance_km) || null,
    calories: parseInt(formData.calories) || null,
    avg_heart_rate: parseInt(formData.avg_heart_rate) || null,
    elevation_m: parseInt(formData.elevation_m) || null,
    personal_best: formData.personal_best,
    notes: formData.notes,
  },
};

This ensures shared attributes (dates, general durations, notes) remain globally indexable for the 360° Dashboard, while biometric attributes (elevation gain, heart rate, personal bests) reside in their specialized child table with cascading foreign key guarantees.

3. Culture & Media Tracking: Books & Cinema 📚🎬

For cultural tracking, the goal was leaving behind fragmented mobile notes in favor of an enriched visual library.

3.1. Interactive Reading Progress (`BooksModule.jsx`)

On each book card, we calculate page completion percentages in real time:

const percentage = b.pages_total > 0 
  ? Math.min(100, Math.round((b.pages_read / b.pages_total) * 100)) 
  : 0;

<div className="book-progress-wrapper">
  <div className="progress-info">
    <span>{b.pages_read} / {b.pages_total} pages</span>
    <span className="percentage-tag">{percentage}%</span>
  </div>
  <div className="progress-bar-bg">
    <div 
      className="progress-bar-fill" 
      style={{ 
        width: `${percentage}%`,
        background: percentage === 100 ? 'var(--accent-emerald)' : 'var(--grad-primary)'
      }} 
    />
  </div>
</div>

A status filter organizes titles into reading, completed, and wishlist, paired with an interactive 5-star rating system using lucide-react icons.

3.2. Streaming Media Catalog (`FilmsModule.jsx`)

The cinema module tracks movies, TV series, and documentaries cataloged by streaming provider (Netflix, HBO Max, Prime Video, Disney+, Cinema). Each entry logs release year, director, rating, and personal review notes, answering: "What were the best films I watched this past quarter?".

4. Professional Execution: The Kanban Board (`KanbanBoard.jsx`) 📋

For professional tasks, I implemented a full Kanban board featuring 4 columns reflecting agile task lifecycles:

📝 To Do (todo): Backlog tasks queued for upcoming execution.
⚡ In Progress (in_progress): Active work during the current sprint.
🔍 In Review (review): Tasks pending review, testing, or third-party feedback.
✅ Done (done): Completed deliverables.

┌──────────────────┬──────────────────┬──────────────────┬──────────────────┐
│ 📝 TO DO (3)     │ ⚡ IN PROGRESS (2)│ 🔍 IN REVIEW (1) │ ✅ DONE (8)      │
├──────────────────┼──────────────────┼──────────────────┼──────────────────┤
│ ┌──────────────┐ │ ┌──────────────┐ │ ┌──────────────┐ │ ┌──────────────┐ │
│ │ Refactor API │ │ │ Frontend UI  │ │ │ E2E Test Run │ │ │ DB Migration │ │
│ │ 🚨 Critical  │ │ │ 🟡 Medium    │ │ │ 🟢 Low       │ │ │ 2026-09-02   │ │
│ │ 📅 Sep 18    │ │ │ 💬 3 notes   │ │ │ 📅 Today     │ │ │ 💬 5 notes   │ │
│ │ [→ Move]     │ │ │ [←]      [→] │ │ │ [←]      [→] │ │ │ [← Move]     │ │
│ └──────────────┘ │ └──────────────┘ │ └──────────────┘ │ └──────────────┘ │
└──────────────────┴──────────────────┴──────────────────┴──────────────────┘

4.1. Optimistic UI Updates for Tactile Fluidity

When moving a task from in_progress to review, waiting for the entire roundtrip (browser → Render API → Supabase → response) introduces an undesirable delay of several hundred milliseconds.

To deliver an instantaneous tactile feel, we execute an optimistic state update in React:

const handleMoveTask = async (taskId, currentStatus, direction) => {
  const statusOrder = ['todo', 'in_progress', 'review', 'done'];
  const currentIndex = statusOrder.indexOf(currentStatus);
  const targetIndex = direction === 'next' ? currentIndex + 1 : currentIndex - 1;

  if (targetIndex < 0 || targetIndex >= statusOrder.length) return;
  const targetStatus = statusOrder[targetIndex];

  // 1. Instantaneous optimistic update

Project LifeOps (Part 1): Personal Operating System Architecture and FastAPI + Supabase Backend

Thu, 10 Sep 2026 17:06:49 +0000

How many apps do you use daily to manage your life? If you are anything like me, you probably have one app to track your runs and gym workouts, another to keep tabs on books, a website to rate movies and TV series, and two or three more tools for project management, tasks, and notes.

The result is almost always the same: scattered data silos, mounting monthly subscription bills, and zero ability to cross-analyze data or generate a consolidated monthly executive review.

With that problem in mind and true to the Datalaria ethos of "learning by building", I decided to design and engineer LifeOps: an All-in-One Personal & Professional Operating System (OS) that unifies under a single responsive dashboard:

🏃 Fitness & Performance: Track workouts (Running, Cycling, Gym), distance volume, Personal Bests (PB), and calories.
📚 Reading & Culture: Book progress tracking, percentage bars, and ratings.
🎬 Cinema & Entertainment: Movie, TV series, and documentary catalog organized by platform.
📋 Professional Portfolio & Tasks: Interactive Kanban board, budget tracking, and deliverable milestones.
📊 Reporting & Data Portability: Streaming Word (.docx) executive dossiers, multi-sheet Excel workbooks (.xlsx), and raw CSV exports with UTF-8 BOM.

Best of all: built on top of a rock-solid, production-grade architecture that is 100% Free ($0/month).

In this first installment of the series, we dive into the core foundations: cloud architecture design, PostgreSQL relational data modeling, and building the high-performance FastAPI backend. Let's get into it! 🚀

[!TIP]
Test the live app: You can explore the production build of LifeOps directly at https://datalaria.com/apps/lifeops/.

🗺️ LifeOps Series Roadmap

To understand how every architectural layer fits together, this series spans 5 structured installments:

🟢 Part 1 (This article): Personal Operating System Architecture and FastAPI + Supabase Backend.
⚪ Part 2: React Frontend with Glassmorphism, 360° Dashboard, and Design System.
⚪ Part 3: Core Interactive Modules: Fitness, Library, Cinema, and Professional Kanban Board.
⚪ Part 4: In-Memory Word Dossiers (.docx) & Multi-Sheet Excel Engine (.xlsx).
⚪ Part 5: 24/7 Zero-Cost Cloud Deployment ($0/month), Mobile UX, and PWA.

The $0/Month Architecture: How to Run Free at Scale 💡

When architecting LifeOps, the goal wasn't merely to run on localhost, but to deploy a production-grade system with zero recurring hosting costs using the most generous free tiers available:

┌─────────────────────────────────────────────────────────────┐
│                       CLIENT BROWSER                        │
│            [https://datalaria.com/apps/lifeops](https://datalaria.com/apps/lifeops)               │
└──────────────────────────────┬──────────────────────────────┘
                               │ (Proxy Rewrite 200)
                               ▼
┌─────────────────────────────────────────────────────────────┐
│               FRONTEND: Netlify CDN ($0/month)              │
│          React 18 + Vite 8 + i18n (ES/EN) + CSS Glass       │
└──────────────────────────────┬──────────────────────────────┘
                               │
                JWT Auth Token │  REST API Calls / Streaming
                               ▼
┌─────────────────────────────────────────────────────────────┐
│               BACKEND: FastAPI + Python 3.13                │
│    • In-Memory Docx Generator (python-docx + io.BytesIO)    │
│    • Multi-Sheet Excel Engine (openpyxl)                    │
│    • Anti-DoS Rate Limiting (slowapi) + Strict CORS         │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│              DATABASE: Supabase Managed Cloud               │
│       • PostgreSQL with isolated `lifeops` schema           │
│       • Row Level Security (RLS) & Supabase Auth            │
└─────────────────────────────────────────────────────────────┘

Frontend on Netlify ($0/mo): Single Page Application built with Vite and served globally with Gzip compression, route-level code splitting, and transparent proxy rewrite under Datalaria's domain.
Backend on FastAPI ($0/mo): Asynchronous Python 3.13 API with strict validation via Pydantic v2, hosted on Render with cold-start resilience.
Database on Supabase ($0/mo): Managed PostgreSQL database with 500 MB of storage, JWT authentication, and automated Row Level Security (RLS) policies.

[!NOTE]
Render Cold-Start Handling: Because the backend runs on Render.com's free tier, the container spins down after 15 minutes of inactivity. In the React frontend (detailed in Part 2), we implemented a preloader screen with automatic exponential-backoff retries, ensuring the user experiences zero abrupt connection drops.

Relational Data Modeling: Clean Polymorphism in PostgreSQL 🗄️

One of the biggest design challenges when combining disparate activities (a 10 km run, a 400-page book, or a Netflix movie) is avoiding monolithic tables cluttered with NULL columns.

To solve this, I designed a master table with specialized child extensions (1-to-1) inside an isolated lifeops schema:

-- 1. Master Activities Table
CREATE TABLE lifeops.activities (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
    activity_type VARCHAR(30) NOT NULL, -- 'sport', 'book', 'film'
    title VARCHAR(200) NOT NULL,
    date DATE NOT NULL DEFAULT CURRENT_DATE,
    duration_minutes INTEGER,
    rating INTEGER CHECK (rating >= 1 AND rating <= 5),
    description TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- 2. Fitness Extension (Workouts)
CREATE TABLE lifeops.workouts (
    activity_id UUID PRIMARY KEY REFERENCES lifeops.activities(id) ON DELETE CASCADE,
    workout_type VARCHAR(50) NOT NULL, -- 'running', 'cycling', 'gym', etc.
    distance_km NUMERIC(6,2),
    calories INTEGER,
    avg_heart_rate INTEGER,
    elevation_m INTEGER,
    personal_best BOOLEAN DEFAULT FALSE,
    notes TEXT
);

-- 3. Reading Extension (Books)
CREATE TABLE lifeops.books (
    activity_id UUID PRIMARY KEY REFERENCES lifeops.activities(id) ON DELETE CASCADE,
    author VARCHAR(150),
    pages_total INTEGER,
    pages_read INTEGER DEFAULT 0,
    status VARCHAR(30) DEFAULT 'reading', -- 'reading', 'completed', 'wishlist'
    genre VARCHAR(50)
);

-- 4. Cinema Extension (Films)
CREATE TABLE lifeops.films (
    activity_id UUID PRIMARY KEY REFERENCES lifeops.activities(id) ON DELETE CASCADE,
    media_type VARCHAR(30) DEFAULT 'movie', -- 'movie', 'series', 'documentary'
    director VARCHAR(150),
    platform VARCHAR(50), -- 'Cine', 'Netflix', 'HBO Max', 'Prime Video'
    genre VARCHAR(50),
    year INTEGER
);

Why this architecture shines:

Blazing Fast Global Aggregations: Dashboard queries hit only lifeops.activities for KPI counts and calendar activity heatmaps.
Cascading Referencial Integrity: When an activity is removed, PostgreSQL's ON DELETE CASCADE automatically purges child tables without orphaned rows.

Row-Level Security (RLS): Each user can only read and mutate their own data via declarative security policies:

CREATE POLICY "Users can only view their own activities"
ON lifeops.activities FOR SELECT
USING (auth.uid() = user_id);

[!IMPORTANT]
Isolated Schemas & Permissions in Supabase: When maintaining tables in a dedicated schema like lifeops (rather than default public), ensure you grant explicit schema access (GRANT USAGE ON SCHEMA lifeops TO authenticated;) and table permissions to the authenticated role. This isolates the operating system's database domain while honoring multi-tenant security.

The FastAPI Backend: Modular Architecture & Strict Typing ⚡

To organize the API cleanly as business domains expand, we follow FastAPI's Modular Router pattern:

lifeops-api/
├── main.py                  # Entrypoint, Middlewares, Rate Limiter & CORS
├── config.py                # Centralized environment settings with Pydantic Settings
├── middleware/
│   └── auth.py              # Supabase JWT validation & secure user_id extraction
├── models/
│   ├── activity.py          # Pydantic Schemas (Create, Update, Response)
│   └── project.py           # Schemas for Tasks and Projects
├── routers/
│   ├── activities.py        # CRUD endpoints for Fitness, Books & Cinema
│   ├── projects.py          # Endpoints for Kanban Board and Projects
│   ├── stats.py             # Global KPI aggregations & Dashboard metrics
│   ├── reports.py           # Word (.docx) generator & CSV/Excel streaming
│   └── alerts.py            # Intelligent reminder engine
└── services/
    ├── report_generator.py  # In-memory RAM .docx compilation
    └── data_exporter.py     # CSV (BOM) & multi-sheet Excel generator

Secure Supabase JWT Authentication

Every endpoint processing user data injects the get_current_user dependency:

from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
import jwt

security = HTTPBearer()

async def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(security)) -> AuthenticatedUser:
    token = credentials.credentials
    try:
        payload = jwt.decode(
            token,
            settings.supabase_jwt_secret,
            algorithms=["HS256"],
            audience="authenticated"
        )
        user_id = payload.get("sub")
        if not user_id:
            raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Token missing user identity")
        return AuthenticatedUser(id=user_id, email=payload.get("email"))
    except jwt.PyJWTError as e:
        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid or expired token")

Production Hardening: Rate Limiting & Anti-Abuse 🛡️

Because LifeOps supports on-the-fly Word report compilation and multi-sheet Excel generation, it was crucial to prevent resource starvation or DoS attacks.

We integrated Rate Limiting with slowapi directly into the FastAPI application:

from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address
from slowapi.errors import RateLimitExceeded

limiter = Limiter(key_func=get_remote_address, default_limits=["120/minute"])
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)

# Strict rate limit for CPU/memory-intensive Word generation
@router.post("/generate")
@limiter.limit("10/minute")
def generate_report(request: Request, req: GenerateReportRequest, user = Depends(get_current_user)):
    ...

If a client exceeds the threshold, the API responds instantly with an HTTP 429 Too Many Requests, protecting compute resources.

Conclusion and What's Next 🎯

With our FastAPI backend hardened, the PostgreSQL relational schema operational on Supabase, and rate limiting active, we have a production-grade foundation running at $0/month.

In Part 2 of this series, we dive straight into the frontend:

Single Page App development with React 18/19, Vite, and Glassmorphism design.
Internationalization (i18n) system for instant Spanish 🇪🇸 / English 🇬🇧 locale toggling.
The 360° Dashboard architecture with aggregated KPI widgets and interactive Recharts visualizations.
Authentication context (AuthContext) and session synchronization with Supabase.

References & Useful Links 🔗

🚀 Production Application: Try the live app at datalaria.com/apps/lifeops.
🌐 Production REST API: Interactive OpenAPI Swagger documentation at lifeops-api.onrender.com/docs.
⚡ FastAPI Framework: Official documentation at fastapi.tiangolo.com.
🗄️ Supabase Cloud: Guide to PostgreSQL and Row Level Security at supabase.com/docs.
🛡️ SlowAPI: Rate limiting for ASGI/FastAPI at github.com/laurentS/slowapi.

See you in the next installment! Feel free to leave a comment below or connect on socials. 👇

Postgres vs ClickHouse? I use both. 4 tricks from the split.

Thu, 09 Jul 2026 05:57:37 +0000

TL;DR. My uptime monitor keeps config and incidents in Postgres and check results in ClickHouse. The split is one rule: does a row ever change? Config gets edited, so it wants Postgres transactions and constraints. A check result is written once and never touched again, so it goes to ClickHouse, where the right codec, a careful sort key, and a per-row TTL make billions of rows cheap. Four tricks and a bonus below, useful even if you only ever run one database.

Every monitor I run writes a row every 20 seconds, from every region, and never stops. One monitor is about 4,300 rows a day, per region. Multiply by every monitor on the platform and the rows only ever go up.

That stream would slowly crush a normal Postgres table, and watching it is the whole product. So the check results do not live in Postgres. They live in ClickHouse. Everything else, the monitors and incidents and teams, lives in Postgres. The interesting part is the line between them.

People ask why not one database. "Postgres vs ClickHouse" is the wrong question, because the two are not fighting over the same job. Here is the line I draw, and the one rule under all of it: split your data by how it is written, not by which engine is faster.

1. Does the row change? That picks the database

Forget benchmarks for a second. One question sorts a table into one store or the other: after you write a row, will you ever change it?

one monitor, two stores:

  you edit a monitor  ->  Postgres   (targets, incidents, team)
                          the row changes, has constraints, lives in a transaction

  a probe checks it   ->  ClickHouse (check_results)
                          one row, written once, never updated

A monitor is a row that changes. You toggle it off, edit the URL, change the interval. So it lives in Postgres:

CREATE TABLE targets (
    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name          TEXT NOT NULL,
    check_spec    JSONB NOT NULL,
    interval_secs INTEGER NOT NULL CHECK (interval_secs >= 10),
    enabled       BOOLEAN NOT NULL DEFAULT true,
    updated_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

That CHECK means a 3-second interval can never reach the table, no matter which part of the app tried to write it. This is Postgres doing the thing it is best at: a small set of rows that must stay correct while many callers change them at once.

A check result is the opposite. It is written once when a probe finishes, and then it never changes. Nothing ever updates it, so it does not need any of that:

CREATE TABLE check_results (
    org_id      UUID,
    target_id   UUID,
    region      LowCardinality(String),
    timestamp   DateTime('UTC') CODEC(DoubleDelta, ZSTD(1)),
    status      Enum8('up' = 1, 'down' = 2, 'degraded' = 3, 'error' = 4),
    duration_ms UInt32 CODEC(T64, ZSTD(1))
) ENGINE = MergeTree

No updated_at, no foreign key, no transaction. Just an append-only stream that grows forever. That is the shape ClickHouse is built for and the shape that slowly hurts Postgres.

Notice the types too. status is an Enum8, one byte on disk, not the string "up". region is LowCardinality, stored once in a dictionary and referenced by a small id instead of repeating the text on every row. On a table that only ever grows, a byte saved per row is a byte saved times billions.

2. The right codec turns a day of timestamps into almost nothing

Once the results are in a column store, the type on each column is most of the compression, and the default setting wastes a lot of space.

Look at that timestamp again. A monitor checks every 20 seconds, so for one monitor a day of timestamps is a run of 20, 20, 20, 20, over and over. DoubleDelta stores the change in the change. For a steady interval the first delta is a constant 20 and the second delta is zero, so each row after the first packs down to about a bit. The codec is not a ClickHouse invention: it comes from Facebook's Gorilla time-series paper, built for exactly this, measurements taken at a steady rate.

timestamp   DateTime('UTC')  CODEC(DoubleDelta, ZSTD(1)),
duration_ms UInt32           CODEC(T64, ZSTD(1)),
dns_ms      Nullable(UInt16) CODEC(T64, ZSTD(1)),

I also store the timestamp at whole seconds on purpose. The smallest interval is 20 seconds, so no two checks for one monitor ever land in the same second, and the sub-second detail lives in duration_ms where it belongs. Whole seconds that step by a fixed amount compress far harder than milliseconds would.

The latency columns get T64 instead, because they are small integers that stay in a small range, a different shape from a steady clock. T64 crops the unused high bits off a block of values, so a UInt32 that never climbs past a few thousand milliseconds stops paying for all 32 bits. That is the trick worth copying, and it is not the specific codec names. It is that a timestamp ticking by a fixed step and a latency staying in a small range are two different shapes, and telling the database which is which does the work.

3. Put the tenant in the sort key, not the partition

I got this one wrong first, so learn it from my mistake instead of your own.

This is a multi-tenant app, so every row carries an org_id and almost every query filters by it. My first schema partitioned by org, one slot per customer, because it felt tidy. ClickHouse turned that into a flood of tiny parts, the merges could not keep up, and startup got slower the more customers I had. I ripped it out. A partition is not a folder for tidiness. It is a physical unit ClickHouse merges and expires, and you want few large ones, not many small ones.

Here is what it should be:

) ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (org_id, target_id, region, timestamp)

org_id leads the ORDER BY, so a per-org query walks the sorted primary index and reads only that org's slice. But it stays out of PARTITION BY. The rule I follow now: partition by something low-cardinality that you also delete by, here the day, and put the high-cardinality tenant key in the sort order. Sort key answers "find this org fast". Partition answers "drop old data cheaply", which is the next trick.

4. Make retention a number in a row, not a schema change

Different plans keep history for different lengths of time. The clumsy way is a migration or a cleanup job per plan. The clean way is to make the retention window a column:

ttl_days UInt16 DEFAULT 30 CODEC(ZSTD(1))
) ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (org_id, target_id, region, timestamp)
TTL timestamp + toIntervalDay(ttl_days)

Each row carries its own ttl_days, stamped from the org's plan at the moment it is written. A free plan keeps 30 days, a paid plan keeps more, and changing that needs no ALTER, no migration, no backfill. The next write just stores a different number. The daily partitions line up with the TTL, so ClickHouse expires old data by dropping whole parts, close to free, and a whole column of the same 30 compresses away to nothing. The flexibility costs no space.

One more piece makes reading that history cheap. A materialized view rolls raw checks into per-minute and per-hour summaries as they land:

CREATE MATERIALIZED VIEW check_results_1m
ENGINE = AggregatingMergeTree
ORDER BY (org_id, target_id, region, minute)
AS SELECT
    org_id, target_id, region,
    toStartOfMinute(timestamp)  AS minute,
    countState()                AS total_checks,
    countIfState(status = 'up') AS up_checks,
    avgState(duration_ms)       AS avg_duration_ms
FROM check_results
GROUP BY org_id, target_id, region, minute;

So a dashboard showing 30 days across a thousand monitors reads a few thousand minute-buckets, not millions of raw rows. Recent views read the minute rollup, long history reads an hour rollup, and the raw rows underneath expire on their own TTL. The firehose is there when you need to drill into one bad minute, and left alone the rest of the time.

One more, because everyone hits it: make inserts idempotent

Here is the trick I wish someone had told me first. My agents batch check results and send them to ClickHouse. A network blip after the server commits but before my side gets the ack means the batch retries and sends the exact same block again. Without protection, that double-counts every row in it, and your uptime numbers quietly drift.

ClickHouse has a fix built in, but for a plain (non-Replicated) MergeTree it is off until you turn it on:

SETTINGS index_granularity = 8192,
         non_replicated_deduplication_window = 1000;

The server hashes each inserted block and remembers the last 1,000 hashes. A retry sends an identical block, the hash matches, and the server drops it instead of appending it again. The retry becomes safe to do blindly, so the client stays simple: send, and if unsure, send again. Make the window bigger than the most blocks a single retry could resend, and you stop having to worry about duplicate writes.

So, one database or two?

For almost every app, one. If your data fits in Postgres and stays fast, a second database is a cost you should not pay: two schemas, two clients, two things to back up and think about.

Reach for the second store only when one table stops looking like the rest of your tables. For me that table was check_results. It is written once and never edited, it grows without end, and every question I ask it is a summary over a time range. That is a different shape from my monitors and my incidents, so it wanted a different database.

The rule I would give my past self: do not split by "which database is faster". Split by how the data is written. Rows that change and must stay correct want Postgres. An append-only stream you only ever summarize wants a column store like ClickHouse. Most of the tricks above are that one idea pushed down into the schema.

Uptimepage is open source, AGPL-3.0, and both schemas are in the repo: github.com/uptimepage/uptimepage. The probe that writes those rows is its own post, and the wider build story, one binary and two databases, is here. Or start free on the hosted tier and point a check at something.

How I Used PostgreSQL Events and MongoDB Event Logs in Two Different Real-World Systems

Sun, 05 Jul 2026 09:20:40 +0000

I built two systems that both needed event tracking.
I used PostgreSQL in one and MongoDB in the other.
Here's exactly why — and what I learned from doing both.
Although the two systems were completely different in purpose, they shared one architectural requirement: each needed a reliable way to record everything that happened — not just the final state, but the story of how that state came to be.
That sounds simple, but the nature of the events in each system was so different that using the same database for both would have caused real problems. This article explains how I approached event tracking in each system, why the solutions diverged, and what the experience taught me about choosing the right persistence model.

PostgreSQL Events and Projections in the Delivery Backend

The delivery backend was built around structured workflows: vendors create orders, couriers pick them up, jobs move through well-defined states, and batches group multiple deliveries. Every action in this system has a clear relational meaning. An event like ORDER_CREATED or JOB_STARTED isn't just a log entry — it's part of the domain model.
Because of that, PostgreSQL was the natural home for events. Each event needed to be tied to a specific entity, validated against constraints, indexed for fast lookup, and projected into timeline tables that the dashboard could query efficiently.
The event log table looked like this:

model EventLog {
  id          Int             @id @default(autoincrement())
  entityType  EventEntityType
  entityId    Int
  eventType   String
  payload     Json
  createdAt   DateTime        @default(now())

  @@index([entityType, entityId, createdAt])
}

Every event is immutable. Every event belongs to a relational entity. Every event carries structured metadata.
Once an event is written, it is immediately transformed into a timeline entry:

switch (event.entityType) {
  case EventEntityType.ORDER:
    await this.orderTimelineProjection.project(event);
    break;

  case EventEntityType.JOB:
    await this.jobTimelineProjection.project(event);
    break;
}

This projection layer is what makes the system feel fast. Instead of running complex joins to reconstruct history, the dashboard reads from dedicated timeline tables optimized for sequential queries. It also makes debugging easier: if something goes wrong, I can replay the event stream and see exactly how the system reached its current state.
PostgreSQL excels in this environment because the events are structured, relational, and part of the domain's consistency model. They aren't just analytics — they're part of the truth.

MongoDB Event Logs and Aggregation in Waitless

Waitless is a completely different kind of system. It's a real-time queue management platform where customers join via QR code, track their position live, and leave without creating an account. The system generates a lot of events — but these events are not relational. They're behavioral.
A CUSTOMER_JOINED event might contain the customer's position.
A NO_SHOW event might contain a timestamp.
A CUSTOMER_LEFT event might contain a reason.
A CALLED_NEXT event might contain which token was called.

The shape of the metadata changes depending on the event type. Forcing this into a rigid SQL schema would either require dozens of nullable columns or a JSON column that PostgreSQL would struggle to index efficiently at scale.
MongoDB is built for exactly this. Each event is a document with flexible metadata:

@Schema({ timestamps: true })
export class QueueEvent {
  @Prop({ required: true })
  queueId!: string;

  @Prop({ required: true, enum: QueueEventType })
  eventType!: QueueEventType;

  @Prop()
  token?: string;

  @Prop({ type: Object })
  metadata?: Record<string, any>;
}

This flexibility becomes powerful when combined with MongoDB's aggregation pipeline. Waitless computes analytics like total joined, total arrived, total no-shows, completion rate, and drop-off rate — all derived from event counts, not relational queries.
A note on Redis: Waitless already had Redis in the stack backing BullMQ. But Redis is a key-value store optimized for job queues and caching — not for querying across documents or running aggregations. MongoDB was the right fit for this specific need.
The separation between MongoDB and PostgreSQL is also crucial for performance. If analytics were computed in PostgreSQL, every dashboard refresh would compete with real-time queue operations for the same connection pool.

If I had used PostgreSQL for Waitless events, every analytics query would hit the same connection pool as the queue state — meaning a slow dashboard query could delay a customer joining a queue.
MongoDB avoids this entirely. It handles flexible event shapes, high write throughput, and analytics workloads without interfering with the transactional state stored in PostgreSQL.

Why PostgreSQL for One and MongoDB for the Other?

The answer becomes clear once you see the nature of the data.
In the delivery backend, events are part of the domain model. They describe state transitions, belong to relational entities, and require strong consistency. PostgreSQL is the right tool.
In Waitless, events are part of analytics. They are flexible, metadata-heavy, and high-volume. They have no relational meaning — they're behavioral observations about what happened in a queue session. MongoDB is the right tool.
Using PostgreSQL for both would have created unnecessary coupling and performance bottlenecks. Using MongoDB for both would have sacrificed relational guarantees where they mattered most.
This is the essence of polyglot persistence: choose the database that matches the shape and purpose of the data, not the one you're most comfortable with.

The Question That Actually Matters

Building two event-driven systems taught me that "SQL vs NoSQL" is the wrong question entirely.
The real question is: what role do your events play in the system?
If events belong to the domain model — use a relational database.
If events belong to analytics — use a document database.
PostgreSQL gave me structured timelines and strong consistency. MongoDB gave me flexible event logs and fast aggregation. Both systems use events, but the events serve completely different masters.
Understanding that difference before writing a single line of code is what separates a working system from a well-designed one.

Working with nested JSON: exploring it, mocking it, and querying it in Postgres

Fri, 03 Jul 2026 13:52:48 +0000

Flat JSON is easy. Nested JSON is where the day goes: an API response with orders inside customers inside accounts, a jsonb column three levels deep, a config file that grew a tree. Three tasks come up over and over, and each one has a trick that makes it painless. Here they are.

1. Reading a huge nested payload

Pretty-printing a 4,000-line response does not actually help. The indentation is correct, but finding one thing still means scrolling through thousands of lines. The fix is to stop treating JSON as text to scroll and start treating it as data to query.

Say the question is: does any item in this order have a null price?

{
  "account": {
    "id": 91,
    "customer": {
      "orders": [
        { "id": 4821, "items": [ { "sku": "A-19", "price": 12.5 } ] },
        { "id": 4822, "items": [ { "sku": "B-04", "price": null } ] }
      ]
    }
  }
}

Three moves turn this from a scroll into a lookup:

Filter by path, key, type, or value. Filtering to price pulls every price field out of the whole tree at once. Filtering by the value null jumps straight to the offending row. The needle in 4,000 lines becomes a single filtered result.
Select the nesting level. Instead of expanding everything, expand the tree to level 2 to see the shape of the data, then drill only into the branch that matters. This is the difference between reading structure and drowning in leaves.
Switch to a table view when a branch is really a list of records. Two hundred objects with the same keys is a spreadsheet pretending to be JSON, and reading it as rows and columns is far faster than reading it as nested braces.

Once JSON is filterable and foldable, payload size stops mattering.

2. Generating realistic nested JSON, fast

The opposite problem: there is no payload yet, and building a parser, a UI, or a test needs one. Hand-writing mock data is slow, and worse, it is too clean. Every record ends up with the same id, the same date, the same shape, which is exactly the uniformity that hides off-by-one bugs, dedupe bugs, and sort bugs.

Two shortcuts cover most cases:

Load a realistic nested sample. Start from a ready-made nested document (an order, a log line, an analytics event) and adapt it, instead of inventing structure from scratch.
Multiply one object into many varied records. Write a single object by hand, pick a count, and get back an array where fields like id, email, and dates actually vary per record. That variation is the point: a list of 50 rows that are genuinely different is what surfaces the bug that 50 identical rows would hide.

Realistic nested test data in seconds beats a hand-typed array of clones every time.

3. Querying and updating nested JSON in Postgres

Eventually that nested JSON lands in a jsonb column, and a viewer cannot help anymore. This is where people get stuck, so here is the practical core.

Assume a table orders with a jsonb column data holding the payload above.

Selecting. The two operators to internalize are -> and ->>. -> returns JSON, ->> returns text:

-- returns jsonb
SELECT data -> 'account' -> 'customer' FROM orders;

-- returns text (note the final ->>)
SELECT data -> 'account' ->> 'id' AS account_id FROM orders;

For a deep path, #> and #>> take an array of keys and are much cleaner than chaining arrows:

-- text value deep in the structure
SELECT data #>> '{account,customer,orders,0,id}' AS first_order_id
FROM orders;

To filter rows by a nested value, containment (@>) is both readable and indexable with a GIN index:

SELECT * FROM orders
WHERE data @> '{"account": {"id": 91}}';

And to work across an array, expand it into rows with jsonb_array_elements:

SELECT elem ->> 'id' AS order_id
FROM orders,
     jsonb_array_elements(data #> '{account,customer,orders}') AS elem;

Updating. The workhorse is jsonb_set(target, path, new_value). The gotcha that trips everyone: new_value must itself be jsonb, so cast it.

-- set a deep scalar (note the ::jsonb cast)
UPDATE orders
SET data = jsonb_set(data, '{account,customer,orders,0,id}', '9001'::jsonb)
WHERE id = 1;

Add or replace a key at a level with the || merge operator, and remove a path with #-:

-- shallow-merge a new key into the top level
UPDATE orders SET data = data || '{"archived": true}'::jsonb WHERE id = 1;

-- delete a nested key
UPDATE orders SET data = data #- '{account,customer,orders,0,items}' WHERE id = 1;

One honest warning: || merges only one level deep, so it will not deep-merge nested objects. For nested changes, reach for jsonb_set. And the genuinely hard case, updating a specific array element chosen by a condition rather than a fixed index, needs more than jsonb_set alone. That is a whole topic on its own, and it is the one worth reading a dedicated write-up on before you fight it in production.

Here is the honest reason all of this matters: nobody keeps these operators in their head. It is -> versus ->>, the #>> path syntax, the ::jsonb cast that jsonb_set quietly requires, the fact that || only merges one level. I forget them between projects every single time. So I wrote them down as a permanent, on-page reference next to the formatter: Querying nested JSON in PostgreSQL (SELECT and UPDATE) collects the common reads and writes in one place, for the moment memory does not serve. It is reference for the everyday cases, not a full Postgres manual.

Where the tool fits

I built all three of these into one place, at catssaymeow.org/json-formatter, because I got tired of one tab that only pretty-prints and another that only explores.

It formats, validates, and minifies (invalid JSON is pinpointed by line and column, not a bare "invalid"), and then lets you actually work with the structure: explore it as a tree, graph, or table, filter by path, key, type, or value, and expand to a chosen nesting level. When there is no data to work with yet, it can load a realistic nested sample or multiply one object into an array of varied mock records. It runs as client-side JavaScript, so the JSON stays in the browser, and clean JSON can be sent straight to a built-in API Client as a request body.

And for the part that happens after the database call, the same page carries a Postgres jsonb querying reference for the SELECT and UPDATE syntax that never sticks. Three recurring nested-JSON tasks, one page to come back to.

If you put it up against your own worst nested payload, I would like to hear what it struggles with.

Your Database GUI Shouldn't Need an Account

Wed, 10 Jun 2026 13:52:44 +0000

A critique of the trend toward account-required database GUI clients, arguing that tools holding production credentials should never route data through vendor backends. The post makes the case for local-first design using Tabularis as a concrete example: no signup, secrets stored in the OS keychain, all config as plain files on disk, bring-your-own-key AI, and only one network call (a GitHub version check). The tradeoffs — no sync, no team workspaces, no telemetry — are acknowledged openly. The core argument is that database clients are plumbing and should be durable and vendor-independent, not coupled to a company's runway.

Building an AI Product Backend From Scratch: FastAPI, Postgres, pgvector, Stripe, and What I'd Do Di

Mon, 18 May 2026 02:16:31 +0000

A couple of years ago I was hired as the solo backend engineer to build the production backend for an AI product company from scratch. No legacy code, no existing schema, no team to defer to — just a frontend team waiting on an API and a product team waiting on inference. This post is about the architecture decisions I made, what worked, and what I'd do differently if I were starting today.
The shape of the problem
The product needed five things working together:

A typed API for the frontend
LLM inference with reasonable UX (streaming, cost controls, retries)
Vector search over user-supplied content
Authentication and user management
Subscription billing

Standard stuff individually. The interesting part was keeping the operational footprint small enough that one person could run the whole thing without it falling over.
Why FastAPI + Postgres + pgvector
The temptation when building an AI product is to reach for a Lambda-and-microservices architecture from day one. I went the other direction: one FastAPI service, one Postgres database, pgvector for embeddings inside that same Postgres.
The reasoning:

One database, one source of truth. User data, subscription state, and vector embeddings live in the same Postgres. No sync issues between Postgres and a separate vector DB. Backups are one operation. Migrations are one tool (Alembic).
FastAPI's typing pays for itself. Pydantic schemas become the contract with the frontend. OpenAPI generation is free. Async support means LLM calls and external APIs don't block the event loop.
One service is faster to ship and easier to debug. When something breaks at 2am, you don't have to figure out which of seven services owns the bug.

The cost: at very high scale (millions of users, billions of embeddings), pgvector and a monolithic FastAPI service would need rethinking. For an early-stage product, this was the right tradeoff.
LLM integration: the part people underestimate
Connecting to an LLM API is one line of code. Making it production-grade is most of the engineering work. The pieces:

Streaming responses. Users will not wait 15 seconds for a wall of text. Server-sent events from FastAPI through to the frontend, with proper backpressure handling.
Token accounting per user. Every request logs prompt tokens, completion tokens, and cost. Without this you have no idea who's burning your budget.
Soft caps + fallback models. Hitting a per-user daily limit doesn't return an error — it switches to a cheaper model and continues. Users get a worse experience, not a broken one.
Retry with backoff that respects rate limits. Upstream providers throttle. Naive retry loops make it worse. Exponential backoff plus reading the Retry-After header is the minimum.
Prompt templates separated from application code. Templates live in their own module, versioned, with the application code only choosing which one to invoke. Changes to wording don't require touching business logic.

Vector search inside Postgres
pgvector is one of the more underrated decisions you can make. It's a Postgres extension that adds a vector type and similarity-search operators. You get nearest-neighbor queries with the same SQL you use for everything else, the same transaction guarantees, and the same backup story.
The gotchas worth knowing:

Indexes matter a lot. A naive sequential scan over a few hundred thousand vectors will be slow. IVFFlat or HNSW indexes are not optional past a certain table size.
Chunking is where retrieval quality lives. Embedding a 50-page document as one vector is useless. Chunk strategy (size, overlap, semantic boundaries) affects results more than the embedding model choice.
Re-embedding is expensive. When you change embedding models or chunking strategy, you re-embed everything. Design the ingestion pipeline so this isn't catastrophic.

Subscription billing: webhooks are the product
Stripe handles the payment UI and the regulatory complexity. Your job is to receive its webhooks and turn them into entitlements in your database.
The non-negotiables:

Webhooks must be idempotent. Stripe will deliver the same event twice. If your subscription state changes twice, you've granted access twice or revoked it incorrectly.
Webhook signature verification on every request. Without it, anyone can POST a fake "subscription.created" event to your endpoint.
Entitlements as a separate layer. Don't check Stripe's subscription status in every API handler. Maintain a local entitlements table updated by webhooks; check that. It's faster, more reliable, and survives Stripe outages.
Reconciliation jobs. Webhooks fail. A nightly job that pulls the truth from Stripe and reconciles your local state catches the gaps before users notice.

Auth: boring is correct
Email/password with hashed credentials, OAuth for social sign-in, JWT for sessions with refresh tokens, role-based access control on routes. None of this is novel and none of it should be. The mistake people make on AI products is wanting auth to be clever. It shouldn't be. Pick well-tested libraries and move on.
What I'd do differently today
A few things have changed in the 2-3 years since:

Structured outputs. Most LLM providers now support enforced JSON schema responses. When I built this, you parsed JSON out of completions and prayed. Today I'd use structured outputs everywhere they're supported and delete a lot of validation code.
Streaming standards. The streaming UX patterns are much more mature now. Vercel AI SDK and similar libraries handle a lot of what I wrote by hand.
Embedding model choices. The embedding landscape moved fast. I'd benchmark a few current options against the actual retrieval task instead of defaulting to OpenAI embeddings.
Observability from day one. I added logging and error reporting, but I'd build proper LLM-aware observability (token usage, latency percentiles, prompt versions, retrieval quality) into the request path from the first commit.

Takeaways
If you're building an AI product backend:

One service and one database will take you further than you think
Most of the engineering work is around the LLM call, not the LLM call itself
pgvector is a serious option if you're already on Postgres
Webhook idempotency and entitlement layers will save you from a class of bugs that are hard to debug after the fact

Happy to dig into any of these in the comments — especially if you've made different architectural choices and have war stories to share.

MCP-PostgreSQL-Ops

Fri, 01 May 2026 08:46:46 +0000

What is MCP-PostgreSQL-Ops?

A production-ready Model Context Protocol (MCP) server for PostgreSQL
operations and monitoring. 30+ tools that let you analyze, monitor, and
maintain PostgreSQL databases through natural language — works with
Claude Desktop, Claude Code, OpenWebUI, or any MCP-compatible client.

Stop juggling psql, pgAdmin, and custom monitoring scripts. Just ask:

"Show me the slowest queries in the last hour"
"Which tables have high bloat and need VACUUM?"
"Analyze autovacuum activity for the past 48 hours"
"Show all foreign key relationships for the customers table"

Why I built it

Most PostgreSQL MCP servers are thin SQL wrappers. This one is built for
real DBA workflows — performance diagnosis, bloat analysis, autovacuum
monitoring, schema introspection — without requiring superuser access or
exotic extensions.

Key Features

30+ tools covering connections, configs, schemas, indexes, IO,
WAL, replication, locks, and more
PostgreSQL 12–17 support with automatic version detection and
graceful feature degradation
Read-only and production-safe — works with regular role permissions
RDS / Aurora compatible — managed service friendly
Extension-independent core — most tools work without
pg_stat_statements; advanced query analytics light up automatically
when extensions are installed
Multi-database operations — switch between databases via parameter,
no reconnection needed
Bloat & autovacuum intelligence — dead tuple ratios, VACUUM
effectiveness analysis, autovacuum trigger thresholds
Bearer token auth for streamable-http mode in production

Installation

PyPI package available:

pip install mcp-postgresql-ops
# or
uvx --python 3.12 mcp-postgresql-ops

Claude Desktop config:

{
  "mcpServers": {
    "mcp-postgresql-ops": {
      "command": "uvx",
      "args": ["--python", "3.12", "mcp-postgresql-ops"],
      "env": {
        "POSTGRES_HOST": "127.0.0.1",
        "POSTGRES_PORT": "5432",
        "POSTGRES_USER": "postgres",
        "POSTGRES_PASSWORD": "your-password",
        "POSTGRES_DB": "your-db"
      }
    }
  }
}

Or run the full Docker Compose stack with built-in test data, OpenWebUI,
and MCPO proxy in 5 minutes.

Tech Stack

Python 3.12 + asyncpg
FastMCP framework
stdio and streamable-http transports

Looking for Feedback

Which additional PostgreSQL areas should I cover next? (Logical
replication slots? Partitioning analysis? Extension-specific tools?)
Anyone running this against large production clusters — performance
feedback welcome
Cross-version testing reports (especially PG 12 and PG 17 edge cases)

Repo: https://github.com/call518/MCP-PostgreSQL-Ops
PyPI: https://pypi.org/project/MCP-PostgreSQL-Ops/

⭐ Stars, issues, and PRs welcome. The codebase is structured for easy
custom-tool addition — see the "Adding Custom Tools" section in the README.

The Risk Map: Architecting an Obsolescence-Immune Data Foundation

Sat, 25 Apr 2026 07:47:17 +0000

1. The Flat Table Trap (Excel is Dead)

Managing an industrial Bill of Materials (BOM) through spreadsheets is a structural deficiency. Excel provides a two-dimensional environment for a three-dimensional problem. Hardware engineering and manufacturing dependencies operate under a graph logic.

When the procurement department receives an End of Life (EOL) alert for a component, calculating the volumetric impact by searching text across multiple static documents introduces operational latency.

The UNE-EN IEC 62402:2019 standard, in its Clause 8.10 (Data Acquisition), establishes the requirement to maintain "a list of configuration sub items within an item" alongside "the identification of the items and sub items details: manufacturer, part number and specification". Achieving the level of parametric traceability demanded by the standard requires the design of a relational data model.

2. Modeling Reality: The Graph Approach (The Internal Environment)

A bill of materials is a dependency tree. To structure the internal manufacturing environment data in PostgreSQL, we define three main indexed tables:

products: Represents the end products. Includes the gross_margin parameter, required to correlate component supply risk with the financial impact on the P&L.
internal_parts: A consolidated register of components, raw materials, and sub-assemblies defined by engineering. Uses a global unique identifier (internal_pn).
bom_lines: Implements the hierarchical relationship. Defines dependencies through mutually exclusive foreign keys (parent_product_id or parent_assembly_id) to a child component (child_pn), specifying a quantity.

Technical Validation: Normalizing the data into a graph structure enables the use of recursive algorithms capable of calculating total exposure in milliseconds. If "Equipment A" requires 4 capacitors and "Equipment B" requires 12, a shortage alert on the component node propagates mathematically and automatically up the tree, cross-referencing totals against the order backlog (SLAs).

3. The Firewall: The AML Table (Approved Manufacturer List)

A core principle of industrial inventory integrity is to prevent the mixing of internal nomenclature with dynamic global market references. Omitting this separation generates traceability inconsistencies.

The architecture implements the aml (Approved Manufacturer List) table as an interface between internal hardware design and the supply chain.

This table enables the Dual-Sourcing strategy. If the bill of materials specifies the component CAP-10K (ERP code), the aml table relates this internal node to both a Texas Instruments reference (preference_level: Primary) and a validated equivalent from Analog Devices (preference_level: Secondary). Upon a supply interruption from the primary source, the relational logic allows switching demand to the secondary source without altering the base BOM.

4. The Integrated Radar (The External Environment)

Integration with global commercial dynamics is consolidated in the manufacturer_parts table.

This entity receives telemetry flows from market aggregators (SiliconExpert, IHS Markit, Accuris) through asynchronous integrations. It maintains an updated record of the manufacturer part numbers (mpn), their lifecycle state (lifecycle_status), and the estimated obsolescence date (estimated_eol_date).

5. Data Segmentation and Intellectual Property (RLS in Supabase)

Deploying this model involves managing Intellectual Property (IP) assets. Accidental or unauthorized exposure of the complete product topology is an unacceptable information security risk. Leveraging the Supabase platform allows implementing security architectures at the data layer.

To prevent external generative models (LLMs) or unprivileged users from reconstructing the complete BOM dependencies, we apply Row Level Security (RLS) directly in PostgreSQL:

-- Security: Row Level Security (RLS) policies - IP Protection
ALTER TABLE products ENABLE ROW LEVEL SECURITY;
ALTER TABLE bom_lines ENABLE ROW LEVEL SECURITY;

-- Allow general read access to authenticated logic
CREATE POLICY "Allow read access to authenticated users" 
ON products FOR SELECT TO authenticated USING (true);

-- Restrict mutations (insert/update/delete) on products and BOM lines to Admin role exclusively
CREATE POLICY "Restrict products mutations to admins" 
ON products FOR ALL TO authenticated 
USING ((auth.jwt() ->> 'role') = 'admin');

The policies restrict any modifications to the products and bom_lines tables to the "Admin" role. Additionally, the architectural design dictates that the database must parse the mathematical expansion of the tree internally and provide the external AI Agent with only the aggregated risk value, without returning the topological structure.

6. Next Steps

The relational database schema for obsolescence management is defined based on modular integrity principles.

The next step in the data architecture process will focus on integration automation. We will evaluate development in Python, using the Pandas library, to engineer an extraction, transformation, and loading (ETL) pipeline capable of ingesting BOM lists in Excel format and structurally populating them into this Supabase ecosystem. Subscribe to access this operational phase.

Database Optimization Strategies That Cut Query Execution Time by 60%: A Practical Guide

Wed, 15 Apr 2026 23:39:07 +0000

If your web application is slow, the database is almost certainly the bottleneck. Not the frontend framework. Not the web server. Not the network. The database.

I have spent nine years building backend systems across seven companies, and in every single engagement, database performance has been the most impactful area of optimization. At VacancySoft, where I currently serve as a Full Stack Engineer, I reduced query execution times by 60% across our core platform. At Univelcity, I achieved a 45% reduction in API response times, with database optimization accounting for the majority of those gains. At Lordwin Group, I optimised a real-time gaming backend to sustain 3,000+ concurrent users with sub-100ms query responses.

This article is the practical guide I wish I had when I started. It covers the exact strategies, tools, and techniques I use to diagnose and fix database performance problems in production systems running PostgreSQL and MySQL.

Strategy 1: Query Analysis and Profiling — Know Before You Optimize

The biggest mistake engineers make with database optimization is guessing. They add indexes randomly, rewrite queries based on intuition, or throw caching at the problem without understanding what is actually slow.

I start every optimization effort the same way: profiling.

Enable Slow Query Logging

In PostgreSQL:

-- postgresql.conf
log_min_duration_statement = 200  -- Log queries taking > 200ms
log_statement = 'none'            -- Don't log all queries, just slow ones

In MySQL:

SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 0.2;  -- 200ms threshold
SET GLOBAL log_queries_not_using_indexes = 'ON';

At VacancySoft, I established a weekly ritual of reviewing the slow query log. This alone surfaced dozens of optimization opportunities that no amount of code review would have caught. Queries that looked clean in the ORM were generating catastrophic execution plans.

Use EXPLAIN ANALYZE Religiously

EXPLAIN shows you the query plan. EXPLAIN ANALYZE actually executes the query and shows real timing data. The difference matters.

EXPLAIN ANALYZE
SELECT v.id, v.title, o.name AS organisation_name, s.name AS sector_name
FROM vacancies v
JOIN organisations o ON o.id = v.organisation_id
JOIN sectors s ON s.id = v.sector_id
WHERE v.status = 'active'
  AND v.created_at >= NOW() - INTERVAL '30 days'
ORDER BY v.created_at DESC
LIMIT 20;

Before optimization, this query at VacancySoft returned an EXPLAIN ANALYZE output like this:

Sort  (cost=15234.56..15234.61 rows=20 width=128) (actual time=1842.331..1842.345 rows=20 loops=1)
  Sort Key: v.created_at DESC
  ->  Hash Join  (cost=8234.12..15234.43 rows=20 width=128) (actual time=1201.442..1841.892 rows=24531 loops=1)
        ->  Seq Scan on vacancies v  (cost=0.00..6842.00 rows=24531 width=96) (actual time=0.021..892.134 rows=24531 loops=1)
              Filter: ((status = 'active') AND (created_at >= ...))
              Rows Removed by Filter: 187234
Planning Time: 0.892 ms
Execution Time: 1843.012 ms

Two immediate red flags: a sequential scan on the vacancies table scanning over 200,000 rows and filtering out 187,000 of them, and a hash join that is materialising far more rows than the final result needs.

This is the kind of visibility that transforms optimization from guesswork into engineering. Olamilekan Lamidi's first rule of database optimization: never optimise a query you have not profiled.

Strategy 2: Indexing Strategies — The Highest-Impact Change

Indexing is the single most impactful optimization technique in my toolkit. A well-placed index can turn a 2-second query into a 5-millisecond query. But indexes are not free — they consume storage, slow down writes, and can mislead the query planner if poorly designed.

Composite Indexes

Order matters in composite indexes. The index (status, created_at) is useful for queries that filter on status and sort or filter on created_at. It is useless for queries that filter only on created_at.

For the vacancy search query above, I created:

CREATE INDEX idx_vacancies_status_created_at
ON vacancies (status, created_at DESC);

This single index eliminated the sequential scan. The query planner now uses an index scan that reads only the relevant rows:

Index Scan using idx_vacancies_status_created_at on vacancies v
  (cost=0.42..234.56 rows=20 width=96) (actual time=0.089..2.341 rows=20 loops=1)
  Index Cond: ((status = 'active') AND (created_at >= ...))

Execution time: from 1,843ms to 12ms. A 99.3% reduction from a single composite index.

Partial Indexes

At VacancySoft, the majority of our queries filter on status = 'active'. Only about 15% of vacancies are active at any given time. A partial index dramatically reduces the index size:

CREATE INDEX idx_vacancies_active_created_at
ON vacancies (created_at DESC)
WHERE status = 'active';

This index is roughly 85% smaller than the full composite index, which means it fits in memory more easily and produces faster index scans.

Covering Indexes (Index-Only Scans)

A covering index includes all columns that a query needs, allowing PostgreSQL to serve the query entirely from the index without touching the heap (the actual table data). This eliminates random I/O, which is the most expensive operation on traditional storage.

CREATE INDEX idx_vacancies_covering
ON vacancies (status, created_at DESC)
INCLUDE (id, title, organisation_id, sector_id)
WHERE status = 'active';

With this covering index, PostgreSQL executes an index-only scan:

Index Only Scan using idx_vacancies_covering on vacancies v
  (cost=0.42..45.23 rows=20 width=96) (actual time=0.067..0.412 rows=20 loops=1)
  Heap Fetches: 0

Heap Fetches: 0 is the goal. The database never touches the table at all.

My Indexing Methodology

I follow a systematic process:

Identify the top 20 slowest queries from the slow query log.
Run EXPLAIN ANALYZE on each one.
Look for sequential scans on tables with more than 10,000 rows.
Check for sort operations that are not backed by an index.
Create targeted indexes and verify the plan changes.
Monitor write performance to ensure indexes are not degrading INSERT/UPDATE speed beyond acceptable thresholds.

At VacancySoft, this methodology across 15+ modules produced the 60% aggregate reduction in query execution time.

Strategy 3: Query Refactoring — Fixing What the ORM Gets Wrong

ORMs are excellent for productivity but frequently generate suboptimal SQL. The N+1 query problem is the most well-known example, but I encounter more subtle issues regularly.

Eliminating N+1 Queries

In Laravel (which I used extensively at 2am Tech, Lordwin Group, and Viaduct):

// Bad: N+1 problem — 1 query for organisations + N queries for vacancies
$organisations = Organisation::all();
foreach ($organisations as $org) {
    echo $org->vacancies->count(); // Triggers a query per organisation
}

// Good: Eager loading — 2 queries total
$organisations = Organisation::with('vacancies')->get();

// Better: When you only need the count
$organisations = Organisation::withCount('vacancies')->get();
foreach ($organisations as $org) {
    echo $org->vacancies_count; // No additional queries
}

At 2am Tech, the Addio financial platform was suffering from response times exceeding 3 seconds on the organisation dashboard. Profiling revealed over 150 queries per page load, almost all of them N+1 queries from Eloquent relationships. I refactored the critical paths to use eager loading and withCount, reducing the query count to 8 and the response time to 400ms.

Replacing Subqueries with JOINs and CTEs

ORMs often generate correlated subqueries where a JOIN or CTE would be far more efficient:

-- ORM-generated: correlated subquery (executes once per row)
SELECT o.id, o.name,
  (SELECT COUNT(*) FROM vacancies v WHERE v.organisation_id = o.id AND v.status = 'active')
    AS active_vacancy_count
FROM organisations o
WHERE o.sector_id = 5;

-- Refactored: single pass with JOIN
SELECT o.id, o.name, COUNT(v.id) AS active_vacancy_count
FROM organisations o
LEFT JOIN vacancies v ON v.organisation_id = o.id AND v.status = 'active'
WHERE o.sector_id = 5
GROUP BY o.id, o.name;

Window Functions for Complex Analytics

At VacancySoft, we needed to show month-over-month vacancy trends per sector. The initial implementation used multiple queries with application-level aggregation. I replaced it with a single query using window functions:

WITH monthly_counts AS (
  SELECT
    s.name AS sector_name,
    DATE_TRUNC('month', v.created_at) AS month,
    COUNT(*) AS vacancy_count
  FROM vacancies v
  JOIN sectors s ON s.id = v.sector_id
  WHERE v.created_at >= NOW() - INTERVAL '12 months'
  GROUP BY s.name, DATE_TRUNC('month', v.created_at)
)
SELECT
  sector_name,
  month,
  vacancy_count,
  LAG(vacancy_count) OVER (PARTITION BY sector_name ORDER BY month) AS prev_month_count,
  ROUND(
    (vacancy_count - LAG(vacancy_count) OVER (PARTITION BY sector_name ORDER BY month))::numeric
    / NULLIF(LAG(vacancy_count) OVER (PARTITION BY sector_name ORDER BY month), 0) * 100,
    1
  ) AS month_over_month_pct
FROM monthly_counts
ORDER BY sector_name, month;

This single query replaced five application-level queries and a significant amount of JavaScript aggregation logic. Response time for the trends endpoint dropped from 2.1 seconds to 280ms.

Strategy 4: Schema Redesign for Performance

Sometimes the problem is not the query — it is the schema. Normalization is taught as gospel in database courses, but production systems often require strategic denormalization.

Materialized Views

At VacancySoft, our analytics dashboard required aggregating data across five tables. Even with optimized queries and proper indexes, the aggregation was too slow for real-time display. I introduced materialized views:

CREATE MATERIALIZED VIEW mv_sector_analytics AS
SELECT
  s.id AS sector_id,
  s.name AS sector_name,
  COUNT(v.id) AS total_vacancies,
  COUNT(v.id) FILTER (WHERE v.status = 'active') AS active_vacancies,
  COUNT(DISTINCT v.organisation_id) AS hiring_organisations,
  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY v.salary_max) AS median_salary,
  DATE_TRUNC('day', NOW()) AS snapshot_date
FROM sectors s
LEFT JOIN vacancies v ON v.sector_id = s.id
  AND v.created_at >= NOW() - INTERVAL '90 days'
GROUP BY s.id, s.name;

CREATE UNIQUE INDEX ON mv_sector_analytics (sector_id);

-- Refresh concurrently (no lock on reads)
REFRESH MATERIALIZED VIEW CONCURRENTLY mv_sector_analytics;

I set up a cron job to refresh this view every 15 minutes. The dashboard query that previously took 3.8 seconds now completes in under 10ms because it reads from the pre-computed view.

Strategic Denormalization

For the organisation profile pages, rather than joining five tables on every request, I added a cached_stats JSONB column on the organisations table:

ALTER TABLE organisations ADD COLUMN cached_stats JSONB DEFAULT '{}';

-- Updated by a background job every hour
UPDATE organisations SET cached_stats = jsonb_build_object(
  'active_vacancies', (SELECT COUNT(*) FROM vacancies WHERE organisation_id = organisations.id AND status = 'active'),
  'total_vacancies_90d', (SELECT COUNT(*) FROM vacancies WHERE organisation_id = organisations.id AND created_at >= NOW() - INTERVAL '90 days'),
  'avg_time_to_fill', (SELECT AVG(EXTRACT(EPOCH FROM (filled_at - created_at)) / 86400) FROM vacancies WHERE organisation_id = organisations.id AND filled_at IS NOT NULL),
  'updated_at', NOW()
);

This is a tradeoff: the data can be up to one hour stale, and we have the overhead of the background update job. But for this use case, slight staleness is acceptable, and the performance gain — from 450ms to 8ms per profile page — is transformative.

Strategy 5: Connection Pooling and Read Replicas

At scale, query optimization alone is not enough. You also need to manage how connections to the database are handled.

Connection Pooling with PgBouncer

PostgreSQL creates a new process for each connection, which is expensive. At VacancySoft, I deployed PgBouncer in front of PostgreSQL in transaction pooling mode:

; pgbouncer.ini
[databases]
vacancysoft = host=127.0.0.1 port=5432 dbname=vacancysoft

[pgbouncer]
listen_port = 6432
pool_mode = transaction
max_client_conn = 500
default_pool_size = 25
reserve_pool_size = 5

This allows 500 application connections to share 25 database connections. Without pooling, our application would regularly hit PostgreSQL's connection limit during traffic spikes, causing cascading failures.

Read Replicas

For read-heavy workloads (which describe most web applications), I split reads and writes across primary and replica databases:

// Database configuration with read/write splitting
const knex = require('knex');

const primary = knex({
  client: 'pg',
  connection: process.env.DATABASE_PRIMARY_URL,
});

const replica = knex({
  client: 'pg',
  connection: process.env.DATABASE_REPLICA_URL,
});

// Service layer decides which connection to use
class VacancyService {
  async search(filters) {
    return replica('vacancies')
      .where(filters)
      .orderBy('created_at', 'desc');
  }

  async create(data) {
    return primary('vacancies')
      .insert(data)
      .returning('*');
  }
}

At VacancySoft, this split reduced load on the primary database by approximately 70%, since the vast majority of our traffic is read operations (search, analytics, reporting).

Strategy 6: Caching Layers

The fastest database query is the one you never make. Caching is the final layer of my optimization strategy.

Redis for Hot Data

const Redis = require('ioredis');
const redis = new Redis(process.env.REDIS_URL);

async function getSectorAnalytics(sectorId) {
  const cacheKey = `sector:${sectorId}:analytics`;
  const cached = await redis.get(cacheKey);

  if (cached) {
    return JSON.parse(cached);
  }

  const data = await db('mv_sector_analytics')
    .where('sector_id', sectorId)
    .first();

  await redis.setex(cacheKey, 900, JSON.stringify(data)); // 15-minute TTL

  return data;
}

Cache Invalidation Strategy

Cache invalidation is famously one of the two hard problems in computer science. My approach is pragmatic:

TTL-based expiration for data that can tolerate staleness (analytics, aggregations).
Event-driven invalidation for data that must be fresh (user profiles, active listings). When a vacancy is created or updated, I publish an event that invalidates the relevant cache keys.
Cache warming for predictable access patterns. At VacancySoft, I pre-warm the cache for the top 50 sectors and top 100 organisations every 15 minutes, so the first user to access these pages never experiences a cache miss.

// Event-driven cache invalidation
eventBus.on('vacancy:created', async (vacancy) => {
  await redis.del(`sector:${vacancy.sector_id}:analytics`);
  await redis.del(`org:${vacancy.organisation_id}:stats`);
  await redis.del('homepage:featured_vacancies');
});

Results: Before and After

Here is a summary of the measurable impact these strategies produced across my career, as implemented by Olamilekan Lamidi:

Company	Metric	Before	After	Improvement
VacancySoft	Average query execution time	850ms	340ms	60% reduction
VacancySoft	Dashboard load time	3.8s	0.4s	89% reduction
VacancySoft	p95 API response time	1.2s	180ms	85% reduction
Univelcity	API response time (avg)	620ms	340ms	45% reduction
Univelcity	Queries per page load	45	8	82% reduction
Lordwin Group	Real-time query latency	220ms	35ms	84% reduction
2am Tech	Organisation dashboard load	3.1s	400ms	87% reduction

Key Takeaways

Profile before you optimize. EXPLAIN ANALYZE and slow query logs should be your starting point, not your intuition.
Composite and partial indexes are the highest-leverage optimization in most applications. One well-placed index can eliminate 99% of query time.
ORMs are productivity tools, not performance tools. Review the SQL they generate, especially for complex queries.
Materialized views and strategic denormalization are not anti-patterns — they are essential tools for read-heavy workloads.
Connection pooling is mandatory once you are handling more than a few hundred concurrent connections.
Caching is the last layer, not the first. Optimize the underlying queries before adding cache complexity.

Database optimization is not a one-time project. It is an ongoing discipline. The queries that perform well today may become bottlenecks tomorrow as data grows and access patterns change. Build profiling and monitoring into your workflow, and treat database performance as a first-class engineering concern.

From Zero to Hero: Create a Cyberpunk Snake Game with Real-Time Ranking using Supabase and Vanilla J

Tue, 17 Mar 2026 06:55:03 +0000

Do you remember the classic Snake from the old Nokias? Now imagine it with neon lights, synthesized music, explosive particles, and most importantly: connected to the cloud.

Web game development has experienced a renaissance thanks to the power of HTML5 Canvas and Serverless tools. In the past, if you wanted to save your players' scores to create a global leaderboard, you needed to set up a server, configure a REST API, manage databases, and pay for hosting. Today, thanks to Backend-as-a-Service (BaaS), we can do it in minutes.

In this tutorial, we are going to build "Neon Snake", an aesthetic and functional web arcade game that uses Supabase (an Open Source alternative to Firebase) to persist data in PostgreSQL.

️ The Tech Stack

We're going to keep it simple but powerful. No heavy frameworks, just the purity of the web:

Frontend: HTML5 Canvas + Vanilla JavaScript (ES6+).
Styles: Tailwind CSS (via CDN) for the user interface.
Backend: Supabase (PostgreSQL + Realtime).
Audio: Web Audio API for code-generated sound effects.

Step 1: Setting up the Backend with Supabase

Before writing a single line of JavaScript, we need our brain in the cloud. Supabase offers us a real PostgreSQL database with an automatically generated API.

Create a free account at supabase.com.
Create a new project called NeonSnake.
Go to the Table Editor section and create a new table called snake_scores.

The Magic of SQL

To streamline the process, Supabase allows you to execute SQL commands directly. Go to the SQL Editor and paste the following code. This will create the table structure and configure the security policies (RLS - Row Level Security) so your game can read and write data publicly.

-- 1. Create the scores table
create table snake_scores (
  id bigint generated by default as identity primary key,
  username text not null,
  score integer not null,
  created_at timestamp with time zone default timezone('utc'::text, now()) not null
);

-- 2. Enable Row Level Security (RLS)
alter table snake_scores enable row level security;

-- 3. Policy to allow anyone to view the ranking (Public Read)
create policy "Ver ranking publico"
on snake_scores for select
using ( true );

-- 4. Policy to allow anyone to save their record (Public Write)
create policy "Guardar puntuacion"
on snake_scores for insert
with check ( true );

Security Note: In a commercial game, we would use full user authentication. For this arcade-style demo, we allow public (anonymous) access to reduce friction so anyone can play and log their record instantly.

Finally, go to Settings > API and copy your Project URL and your anon public key. You will need them shortly.

Step 2: The Cyberpunk Frontend (HTML5 + Canvas)

Our game will live in a single index.html file. We will use Tailwind CSS for the floating menus (Game Over, Start Screen) and the <canvas> element to render the game at 60 FPS.

The basic structure is as follows:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Neon Snake 2077</title>
    <script src="[https://cdn.tailwindcss.com](https://cdn.tailwindcss.com)"></script>
    <style>
        @import url('[https://fonts.googleapis.com/css2?family=Orbitron:wght@400;700&display=swap](https://fonts.googleapis.com/css2?family=Orbitron:wght@400;700&display=swap)');
        body { font-family: 'Orbitron', sans-serif; background-color: #050505; }
        /* Neon glow effect for the canvas */
        canvas { box-shadow: 0 0 20px #0ff; border: 2px solid #0ff; }
    </style>
</head>
<body class="h-screen w-full flex items-center justify-center overflow-hidden bg-gray-900">

    <div class="relative">
        <canvas id="gameCanvas" width="400" height="400"></canvas>
        
        <div id="gameOverMenu" class="hidden absolute inset-0 bg-black/90 flex flex-col items-center justify-center z-10">
            <h2 class="text-4xl text-red-500 font-bold mb-4 drop-shadow-[0_0_10px_rgba(239,68,68,0.8)]">SYSTEM FAILURE</h2>
            <input type="text" id="usernameInput" placeholder="Enter ID..." class="bg-gray-800 text-cyan-400 p-2 rounded border border-cyan-500 mb-4 text-center outline-none">
            <button id="saveBtn" class="bg-cyan-600 hover:bg-cyan-500 text-white font-bold py-2 px-4 rounded shadow-[0_0_15px_#06b6d4] transition-all">
                UPLOAD DATA
            </button>
            
            <div id="leaderboard" class="mt-6 text-sm text-gray-300 w-64 text-left h-40 overflow-y-auto custom-scrollbar"></div>
        </div>
    </div>

    <script type="module">
        import { createClient } from '[https://cdn.jsdelivr.net/npm/@supabase/supabase-js/+esm](https://cdn.jsdelivr.net/npm/@supabase/supabase-js/+esm)'
        
        // ... Game Logic here (See step 3) ...
    </script>
</body>
</html>

Game Logic: Beyond Eating Apples

To give it that "Pro" touch and differentiate it from a basic tutorial, we implement several key mechanics in JavaScript:

Animation Loop: Using requestAnimationFrame to maintain fluidity at 60 FPS.
Particle System: An array of objects that spawn at the "food" coordinates when eaten and fade gradually, simulating a data explosion.
Dynamic Firewalls: From level 5 onwards, we generate static obstacles that the player must dodge, progressively increasing the difficulty.
Hybrid Controls:
- PC: We listen for keydown events for directional arrows.
- Mobile: We capture touchstart and touchend to calculate the gesture direction (swipe) and move the snake.

Step 3: Connecting the Dots (Integration with Supabase)

Here is where the Backend-as-a-Service magic happens. We are going to connect our Canvas to the PostgreSQL database. Inside our <script type="module"> tag:

1. Client Initialization

const supabaseUrl = 'YOUR_SUPABASE_PROJECT_URL';
const supabaseKey = 'YOUR_SUPABASE_ANON_KEY';
const supabase = createClient(supabaseUrl, supabaseKey);

2. The "Game Over" Flow

When the player loses and hits "UPLOAD DATA", we need to ensure the data is saved before showing the leaderboard. We will use async/await to control this asynchronous flow.

async function saveScore(username, score) {
    // 1. Save the score
    const { error } = await supabase
        .from('snake_scores')
        .insert([{ username: username, score: score }]);

    if (error) {
        console.error('Error uploading data:', error);
        alert('Connection error with the central server.');
        return;
    }
    
    // 2. Immediately after, update the leaderboard visually
    fetchLeaderboard();
}

3. Fetching the Ranking (Select)

We retrieve the Top 20 sorted by score in descending order to show who rules the server.

async function fetchLeaderboard() {
    const { data: scores, error } = await supabase
        .from('snake_scores')
        .select('username, score')
        .order('score', { ascending: false }) // Descending order
        .limit(20);                           // Only the top 20

    if (error) console.error('Error fetching leaderboard:', error);
    
    // Render in the DOM
    const list = document.getElementById('leaderboard');
    list.innerHTML = '<h3 class="text-cyan-400 border-b border-cyan-500 mb-2 font-bold">TOP HACKERS</h3>';
    
    scores.forEach((entry, index) => {
        // Highlight top 3 with different colors if you want
        const colorClass = index < 3 ? 'text-yellow-400' : 'text-gray-300';
        
        list.innerHTML += `
            <div class="flex justify-between py-1 border-b border-gray-800">
                <span class="${colorClass}">#${index + 1} ${entry.username}</span>
                <span class="text-cyan-300 font-mono">${entry.score}</span>
            </div>`;
    });
}

Bonus: Realtime ⚡

Do you want the ranking to update on everyone's screen if someone breaks the record at that precise moment? Supabase makes this trivial with its subscriptions:

// Subscribe to changes in the 'snake_scores' table
supabase
  .channel('public:snake_scores')
  .on('postgres_changes', 
      { event: 'INSERT', schema: 'public', table: 'snake_scores' }, 
      (payload) => {
        console.log('New record detected on the network!', payload);
        fetchLeaderboard(); // Automatically refresh the list for everyone
      }
  )
  .subscribe();

The Final Result

By putting it all together, you get a fluid and highly competitive user experience:

The user plays frantically dodging neon "Firewalls".
Upon crashing, the game stops with a visual "glitch" effect.
They enter their name (e.g., "Neo").
Upon clicking send, in milliseconds, their name appears on the leaderboard.
If their friend is playing on another mobile at the same time, they will see "Neo's" name appear on their screen without reloading the page.

And once created... all that remains is to enjoy and play

Datalaria Snake Game

Conclusion

We have moved from a simple static canvas to a Fullstack application in real-time without touching a traditional backend server (Node.js, PHP, Python) or managing infrastructure. The combination of creativity in game development with the power of BaaS tools like Supabase opens a world of possibilities for frontend developers.

The next step? I invite you to clone the project and improve it: add GitHub authentication to avoid duplicate names, create snake skins that unlock with high scores, or implement a multiplayer mode using Supabase presence channels.

The code is yours!

Did you like this tutorial? Share it on your networks with your High Scores and don't forget to check the official Supabase documentation to dive deeper.

Getting Started with PostgreSQL: Best Practices from Development to Production

Sun, 15 Mar 2026 17:40:28 +0000

Databases are the backbone of modern applications. Whether you're building APIs, SaaS platforms, or enterprise systems, choosing the right database and using it correctly can make a huge difference in performance and scalability.

One of the most reliable and powerful databases available today is PostgreSQL.

PostgreSQL is known for its stability, extensibility, performance, and strong SQL compliance, which is why many modern systems rely on it for production workloads.

In this guide, we’ll explore how to get started with PostgreSQL and follow best practices from development all the way to production environments.

Why Choose PostgreSQL?

PostgreSQL has become a favorite among developers and companies because it offers:

Advanced SQL support
Strong data integrity
JSON support for hybrid workloads
Extensibility through extensions
Excellent performance for large-scale systems

Many major platforms rely on PostgreSQL, including Netflix, Apple, and Instagram.

It works well for:

web applications
analytics workloads
enterprise systems
microservices architectures

Installing PostgreSQL

You can install PostgreSQL locally using the official installer or through a package manager.

Typical installation steps:

# macOS (Homebrew)
brew install postgresql

# Ubuntu
sudo apt install postgresql

# Windows
Download installer from postgresql.org

After installation, start the database service and connect using the CLI tool:

psql -U postgres

Basic Database Setup

A typical project setup includes:

CREATE DATABASE myapp;

CREATE USER myapp_user WITH PASSWORD 'securepassword';

GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;

This separation between database owner and application user is an important security best practice.

Designing Tables Properly

Good database design is essential for long-term scalability.

Example table:

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email TEXT UNIQUE NOT NULL,
    name TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Important principles:

always define a primary key
enforce unique constraints where necessary
avoid nullable fields unless truly optional
use timestamps for auditing

Use Indexes for Performance

Indexes help speed up queries significantly.

Example:

CREATE INDEX idx_users_email ON users(email);

Indexes are useful for:

search queries
filtering large datasets
join operations

However, too many indexes can slow down writes, so use them strategically.

Use Migrations Instead of Manual Changes

Schema changes should always be version-controlled.

Popular migration tools include:

Prisma
TypeORM
Flyway

Example migration:

ALTER TABLE users ADD COLUMN last_login TIMESTAMP;

This ensures that all environments stay consistent.

Use Transactions for Data Safety

Transactions ensure that multiple operations succeed or fail together.

Example:

BEGIN;

INSERT INTO orders (...) VALUES (...);
UPDATE users SET balance = balance - 100 WHERE id = 1;

COMMIT;

If something fails:

ROLLBACK;

Transactions prevent partial updates and inconsistent data.

Use Connection Pooling

Opening a database connection for every request is expensive.

Connection pooling improves performance by reusing connections.

Popular tools include:

PgBouncer
pg-pool

Connection pooling becomes essential in high traffic production systems.

Logging and Monitoring

Monitoring your database helps detect issues early.

PostgreSQL provides built-in logging capabilities.

You can monitor:

slow queries
long-running transactions
connection usage

Tools commonly used with PostgreSQL include:

pgAdmin
Grafana

Monitoring is crucial for production reliability.

Backup Strategy for Production

Always have automated backups.

Typical strategies include:

Daily backups:

pg_dump myapp > backup.sql

Full database backups:

pg_dumpall > full_backup.sql

For production systems, consider:

automated cloud backups
replication clusters
point-in-time recovery

This protects your data from accidental loss.

Scaling PostgreSQL in Production

As your application grows, scaling becomes important.

Common scaling strategies include:

Read Replicas

Separate read queries from write operations.

Example architecture:

Primary DB → Write operations
Replica DB → Read queries

This improves performance under heavy load.

Partitioning Large Tables

When tables become very large, partitioning improves performance.

Example:

CREATE TABLE logs_2025 PARTITION OF logs
FOR VALUES FROM ('2025-01-01') TO ('2026-01-01');

Partitioning helps with:

faster queries
easier data management
improved maintenance

Security Best Practices

Production databases must be secured properly.

Important steps include:

enforce strong passwords
restrict database network access
use SSL connections
avoid superuser access for applications

Principle of least privilege is essential.

Development vs Production Setup

Development environments can be simple, but production requires more structure.

Development setup:

local PostgreSQL instance
minimal security configuration
small datasets

Production setup:

managed PostgreSQL service
automated backups
monitoring and alerts
connection pooling

Many companies use managed services like Amazon RDS or Azure Database for PostgreSQL.

Final Thoughts

PostgreSQL is one of the most powerful and reliable databases available today.

But simply installing it is not enough. Building scalable systems requires:

proper schema design
indexing strategies
migration workflows
monitoring and backups
production-grade security

When used correctly, PostgreSQL can power applications from small startups to massive enterprise platforms.

If you're building modern applications today, PostgreSQL is definitely a database worth mastering.

https://www.enterprisedb.com/downloads/postgres-postgresql-downloads

Merancang Dashboard Manajemen Proyek Konstruksi: Progress, RAB, dan Issue Tracking Berbasis Web

Fri, 06 Mar 2026 17:00:00 +0000

Di 2026, obrolan soal AI agents mulai masuk ke meja operasional konstruksi—bukan lagi sekadar demo, tapi cara baru mengorkestrasi data lintas tim (lapangan–QS–procurement–PMO). Bahkan media bisnis besar menyorot peluangnya di industri konstruksi lewat pembahasan tentang AI agents dan produktivitas proyek di artikel AI agents untuk konstruksi. Namun sebelum “AI” bekerja, fondasinya harus beres: data progres, biaya, dan isu harus bisa bicara lewat API. Di sinilah dashboard manajemen proyek konstruksi jadi pintu masuk paling masuk akal.

Kalau kamu pernah merasa progres “aman” tapi cashflow meledak—kamu tidak sendirian. Riset tentang pelacakan dan analisis real-time di proyek konstruksi menegaskan tantangan utama ada pada data multi-sumber (jadwal, BIM, platform manajemen proyek, hingga data lapangan) yang terlambat terhubung ke keputusan harian. Itu dibahas lebih teknis di studi real-time tracking and analysis in construction projects. Kita mengangkat tema ini karena banyak developer membangun aplikasi “rapi”, tapi jarang menyentuh problem yang benar-benar menyelamatkan proyek: single source of truth + metrik yang disepakati.

“Dashboard bukan pajangan. Ia adalah kontrak: definisi metrik, ritme data masuk, dan cara tim mengambil keputusan.”

1. Kenapa Dashboard Itu Lebih dari Grafik

Dashboard yang dipakai manajemen bukan yang paling cantik—melainkan yang paling actionable. Di konstruksi, actionable artinya: begitu ada deviasi, orang tahu harus ngapain, bukan cuma “melihat angka”. Pembukaan bab ini sengaja menekankan bahwa dashboard manajemen proyek konstruksi adalah produk data + proses, bukan semata UI.

Tiga pertanyaan yang wajib dijawab dashboard

Progress: “Apa yang benar-benar sudah selesai?” (volume, bobot, earned value sederhana)
Biaya (RAB vs aktual): “Apakah biaya mengikuti rencana?” (burn rate, deviasi per item)
Issue tracking: “Apa yang menghambat dan siapa owner-nya?” (RFI, NCR, HSE finding, approval delay)

Pola gagal yang sering terjadi

Progres ada di laporan harian, biaya ada di spreadsheet, issue ada di WhatsApp.
Definisi persentase berbeda antar tim (“80%” versi site ≠ “80%” versi QS).
Dashboard lambat → ditinggal → kembali ke chat/Excel.

2. Peta Data: Dari Lapangan ke Single Source of Truth

Sebelum bicara Next.js atau Postgres, tentukan dulu “bahasa data”-nya. Bab ini membuka dengan satu prinsip: kalau input berantakan, output secanggih apa pun akan jadi noise. Dengan begitu, dashboard manajemen proyek konstruksi bisa tumbuh stabil.

Minimal dataset yang worth collecting

Domain	Contoh data	Sumber umum	Frekuensi	Risiko kalau tidak rapi
Progres	volume, bobot item, % capaian	daily report, checklist	harian	% “ngambang”, sulit audit
RAB	struktur budget, baseline	RAB/BoQ	awal + revisi	deviasi tidak terlacak
Biaya aktual	PO, invoice, petty cash	finance/procurement	harian/mingguan	overrun ketahuan telat
Issue	RFI, NCR, HSE finding	site + QA/QC	harian	bottleneck jadi laten
Resource	man power, alat, material	log lapangan	harian	produktivitas tidak kebaca

Kontrak input yang realistis (bukan utopis)

Daily report: tanggal, cuaca, shift, item pekerjaan + volume, foto, kendala
Cost ledger: tanggal, item, vendor, nilai, tipe (commitment vs actual)
Issue: tipe, severity, owner, due date, status, evidence

Tip praktis: “Wajibkan 5–7 field, bukan 20 field.” Sistem yang memaksa detail berlebihan biasanya ditinggalkan.

3. Arsitektur MVP yang Developer-Friendly

Kita akan gunakan stack yang mainstream di komunitas: Next.js + TypeScript untuk UI, API layer (Next API Routes atau service terpisah), dan PostgreSQL untuk data. Fokusnya: agregasi—karena dashboard hidup dari ringkasan.

Komponen

Frontend: Next.js (App Router), TypeScript, TanStack Query
Backend: API (REST), validation (Zod/Pydantic), auth (JWT/Session)
DB: PostgreSQL (schema untuk progres, RAB, biaya, issue)
Observability: OpenTelemetry + log correlation (request-id)
Optional: background worker (BullMQ/Temporal) untuk ETL ringan

Kenapa “endpoint agregasi” itu kunci

Kalau UI harus menarik data mentah lalu menghitung sendiri, lama-lama jadi:

query berat di client
duplikasi logic
data “tidak konsisten” antar halaman

Solusi: buat endpoint seperti GET /projects/:id/dashboard yang mengirim:

KPI ringkas
series chart
daftar deviasi terbesar
issue prioritas (SLA breach)

Bacaan internal yang relevan (dan sedang ramai di CoderLegion) soal menyatukan progres, RAB, dan laporan ke satu alur data bisa kamu bandingkan dengan artikel Dashboard Monitoring Proyek Konstruksi dengan Python + FastAPI dari penulis aktif.

4. Model Data: Skema yang Tidak Membuat Kamu Menyesal

Kebanyakan dashboard mati bukan karena UI jelek—tapi karena skema DB menyulitkan agregasi. Bab ini membuka dengan pola desain yang “tahan revisi” ketika RAB berubah atau definisi progres disesuaikan.

Entitas inti (versi praktis)

projects
work_items (item pekerjaan + bobot + unit)
budget_items (struktur RAB/BoQ)
daily_reports (header laporan)
daily_progress (detail progres per item per hari)
cost_ledger (commitment & actual)
issues (RFI/NCR/HSE/approval)

Prinsip penting

Pisahkan header vs detail (laporan harian jangan jadi 1 tabel raksasa)
Simpan unit & koefisien agar konversi tidak “liar” di UI
Versioning RAB (budget_version) untuk audit dan perubahan scope

5. How-To: Dari Nol ke Dashboard yang Data-Ready

Bab ini sengaja dibuat taktis. Tujuannya: kamu bisa mulai dari “MVP yang dipakai”, lalu iterasi. Minimal 1 sprint.

Langkah 1 — Sepakati metrik (biar tidak debat di akhir)

Definisikan progress (volume, bobot, atau EVM sederhana?)
Definisikan “aktual biaya” (cash out vs commitment?)
Definisikan prioritas issue (severity + SLA)

Langkah 2 — Buat form input harian yang mobile-friendly

Checklist form:

tanggal, shift, cuaca
item + volume + evidence (foto)
resource (man power, alat)
kendala + kategori

Langkah 3 — Bangun API kontrak yang stabil

Validasi ketat (Zod/Pydantic)
Idempotency key untuk submit ulang
Error format konsisten (code, message, hint)

Langkah 4 — Buat endpoint agregasi untuk dashboard

Contoh output (ringkas):

kpis: progress total, deviasi biaya, burn rate
charts: progress series, cost series
alerts: top deviasi item, issue SLA breach

Langkah 5 — Tambahkan audit trail

created_by, created_at, source
histori revisi daily report
change log untuk budget version

Micro-win: begitu tim bisa melihat “deviasi per item” + “issue yang menahan progres”, adopsi dashboard biasanya naik drastis.

6. Taktik UX yang Bikin Dashboard Dipakai (Bukan Sekali Lihat)

Bab ini dibuka dengan realita: user dashboard di proyek itu sibuk. Mereka perlu “jawaban cepat”, bukan eksplorasi panjang. Kita tetap ingat konteks dashboard manajemen proyek konstruksi—harus terasa seperti alat kerja.

Pola layout yang terbukti efektif

Top bar KPI (3–5 angka): progress, deviasi biaya, issue open, SLA breach
Middle: 2 chart utama (progress vs time, cost vs time)
Bottom: tabel deviasi item + daftar issue prioritas

Istilah kekinian yang relevan (tanpa sekadar jargon)

Single Source of Truth: 1 definisi data yang disepakati
Data contract: API schema sebagai kontrak lintas tim
Observability: tracing + log correlation (biar cepat cari bottleneck)
Operational analytics: analitik yang langsung memicu aksi

7. FAQ (Yang Biasanya Ditanyakan di Dunia Nyata)

1) Apakah dashboard harus dibuat dengan React/Next.js?
Tidak wajib. Kamu bisa pakai BI tools (Metabase/Superset). Next.js cocok kalau kamu butuh kontrol penuh atas workflow, auth, dan UX.

2) Kalau RAB masih Excel, apa langkah paling aman?
Mulai dari import terstruktur ke budget_items, lalu simpan versi. Jangan overwrite baseline.

3) Data lapangan sering kosong. Apa yang dilakukan?
Mulai dari minimum viable reporting (5–7 field wajib) + tampilkan field kosong sebagai alert.

4) Apa metrik yang paling cepat memberi dampak?
Deviasi biaya vs progres per item pekerjaan—biasanya paling cepat “membuka mata”.

5) Bagaimana menghindari dashboard jadi “tempat debat”?
Tegaskan definisi metrik, lock baseline, dan simpan audit trail. Data yang bisa diaudit = diskusi yang lebih sehat.

Dashboard yang Menjadi Alat Keputusan

Sebagai penutup, ada prinsip yang relevan untuk budaya engineering yang berujung pada kualitas sistem: “You build it, you run it.” — sering dikaitkan dengan Werner Vogels, CTO Amazon yang dikenal luas di ranah distributed systems dan cloud architecture. Artinya: “Kamu yang membangun, kamu juga yang menjalankan.” Dalam konteks dashboard, ini mengingatkan kita bahwa developer perlu peduli pada kualitas data, performa API, dan dampaknya ke operasional—karena feedback loop terbaik lahir dari sistem yang benar-benar dipakai.

Kalau kamu sedang mengerjakan proyek industri/komersial dan ingin menautkan praktik digital (dashboard, MEP, hingga integrasi IT networking) ke workflow lapangan, kamu bisa melihat konteks layanan dan portofolio kami di kontraktor Karawang—sebagai referensi dunia nyata bagaimana dashboard manajemen proyek konstruksi akhirnya “turun ke tanah” dan dipakai harian.

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Article",
      "headline": "Merancang Dashboard Manajemen Proyek Konstruksi: Progress, RAB, dan Issue Tracking Berbasis Web",
      "description": "Panduan praktis merancang dashboard manajemen proyek konstruksi yang menggabungkan progres, RAB vs aktual, dan issue tracking berbasis web—dengan fokus pada data contract, agregasi KPI, dan arsitektur MVP.",
      "about": [
        "Project Management",
        "Construction Technology",
        "Web Development",
        "Observability"
      ],
      "keywords": [
        "dashboard manajemen proyek konstruksi",
        "progress proyek",
        "RAB vs aktual",
        "issue tracking",
        "Next.js",
        "TypeScript",
        "PostgreSQL"
      ],
      "inLanguage": "id-ID",
      "author": {
        "@type": "Organization",
        "name": "PT. Nikifour"
      },
      "publisher": {
        "@type": "Organization",
        "name": "CoderLegion"
      },
      "mainEntityOfPage": {
        "@type": "WebPage",
        "@id": "https://coderlegion.com/"
      }
    },
    {
      "@type": "HowTo",
      "name": "Merancang dashboard manajemen proyek konstruksi yang data-ready",
      "description": "Langkah demi langkah untuk merancang dashboard manajemen proyek konstruksi: menyepakati KPI, menyiapkan input harian, membangun API kontrak stabil, membuat endpoint agregasi, dan menerapkan audit trail.",
      "totalTime": "PT6H",
      "supply": [
        { "@type": "HowToSupply", "name": "Akun hosting / server" },
        { "@type": "HowToSupply", "name": "Database PostgreSQL" }
      ],
      "tool": [
        { "@type": "HowToTool", "name": "Next.js" },
        { "@type": "HowToTool", "name": "TypeScript" },
        { "@type": "HowToTool", "name": "PostgreSQL" }
      ],
      "step": [
        {
          "@type": "HowToStep",
          "name": "Sepakati metrik (KPI) proyek",
          "text": "Definisikan progress (volume/bobot/EVM sederhana), definisikan biaya (cash out vs commitment), dan definisikan prioritas issue (severity + SLA)."
        },
        {
          "@type": "HowToStep",
          "name": "Buat input harian yang mobile-friendly",
          "text": "Gunakan form ringkas: tanggal, shift, cuaca, item + volume + bukti, resource, kendala + kategori."
        },
        {
          "@type": "HowToStep",
          "name": "Bangun API kontrak yang stabil",
          "text": "Terapkan validasi ketat, idempotency key untuk submit ulang, dan format error konsisten."
        },
        {
          "@type": "HowToStep",
          "name": "Buat endpoint agregasi dashboard",
          "text": "Sediakan endpoint yang mengembalikan KPI, data chart, alerts deviasi, dan issue SLA breach dalam format siap render."
        },
        {
          "@type": "HowToStep",
          "name": "Aktifkan audit trail",
          "text": "Simpan created_by/created_at/source, histori revisi laporan, dan versioning RAB untuk kebutuhan audit."
        }
      ]
    },
    {
      "@type": "FAQPage",
      "mainEntity": [
        {
          "@type": "Question",
          "name": "Apakah dashboard harus dibuat dengan React/Next.js?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Tidak wajib. Kamu bisa memakai BI tools (Metabase/Superset). Next.js cocok jika butuh kontrol penuh atas workflow, auth, dan UX."
          }
        },
        {
          "@type": "Question",
          "name": "Kalau RAB masih Excel, apa langkah paling aman?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Mulai dari import terstruktur ke budget_items dan simpan versi. Jangan overwrite baseline agar audit tetap aman."
          }
        },
        {
          "@type": "Question",
          "name": "Data lapangan sering kosong. Apa yang dilakukan?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Mulai dari minimum viable reporting (5–7 field wajib) dan tampilkan field kosong sebagai alert agar disiplin input terbentuk."
          }
        },
        {
          "@type": "Question",
          "name": "Apa metrik yang paling cepat memberi dampak?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Deviasi biaya vs progres per item pekerjaan—biasanya paling cepat menunjukkan bottleneck dan potensi overrun."
          }
        }
      ]
    }
  ]
}

Patching the Gaps: A Production-Ready Guide to Google’s ADK with TypeScript

Tue, 10 Feb 2026 11:02:52 +0000

Building production-grade AI agents is tricky. In a landscape shifting rapidly, betting your entire architecture on a single provider's raw SDK allows for quick starts but leads to brittle vendor lock-in. This is where Google’s Agent Development Kit (ADK) shines. It acts as a crucial abstraction layer, providing a standardized common interface for defining tools (similar to LiteLLM which Google ADK actually uses internally), events, and agent orchestration. This means you aren't locked solely into Gemini; you can design systems that utilize different models from various providers — even using them as fallbacks for one another — without rewriting your core business logic.

However, if you’ve tried taking the ADK into production with TypeScript, you’ve likely hit a significant reality gap. While the Python documentation is comprehensive, the TypeScript side is remarkably thin, steering devs toward "ADK Devtools" which, while fine for local prototyping, simply don't cut it for a hardened production stack.

Before we fix that, let's briefly strip away the jargon and define how the ADK actually works under the hood. It relies on three core concepts:

The Runner (Event Loop): Think of this as the engine room. It’s an asynchronous loop that takes an incoming signal (an event), processes it through your agent’s logic (calling the LLM, executing tools), and determines the next step (updating state in the process).
State: This is the "short-term memory" of the active conversation. It’s mutable data representing where the user is right now in a flow (e.g., { step: 'collecting_address', attempt: 2 }).
Context: The environment the runner operates in. This is the static stuff the agent needs to do its job but doesn't change turn-by-turn. Think database connections, user IDs, or API keys.

Here is the crucial part for production: Events in the ADK aren't just passive text messages. They carry their own state payload representing a "delta" or in simple layman terms, change. The Runner uses this specific event state to update the overall session state before beginning the next turn (handled by the session service).

You can read more about the Google ADK architecture here: https://google.github.io/adk-docs/get-started/about/

In a real-world environment, you can't let that critical state live in volatile, in-memory storage. You need true persistence so agents retain context across server restarts and container scaling. Furthermore, if you’re running a modern TypeScript stack with ESM, the currently shipped @google/adk package suffers from dependency conflicts that break standard builds.

Today, we’re going to fix this. We'll walk through patching the ADK to bridge the ESM/CommonJS gap and implementing custom PostgreSQL-backed session and memory services to ensure that state is saved durably. It’s a deep dive, but essential for anyone moving beyond the "Hello World" phase.

Here's the GitHub repo with complete code examples you can refer from: tutorial repository

The Current Limitations

For production TypeScript applications, the current ADK falls short in three key areas:

Lack of Persistence: The provided examples use in-memory services. Server restarts mean lost user context.
ESM/CJS Conflicts: The published package's CommonJS build incorrectly imports the ESM-only lodash-es package, rather than standard lodash, breaking many build pipelines.
Missing Infrastructure: Unlike the Python SDK, the TypeScript version lacks built-in persistent session services (like VertexAI) out of the box.

We will resolve these issues using NestJS, TypeORM (PostgreSQL), and pnpm package manager for patching.

Hands-On: Building a Persistent ADK Agent

We are going to implement a CloudSqlSessionService and a CloudSqlMemoryService. This ensures your agent's state and long-term memories live safely in your Postgres database rather than ephemeral RAM.

1. Patching @google/adk

The first step is stabilizing the dependency tree. As mentioned, the @google/adk package currently ships a CommonJS build that incorrectly imports lodash-es. We need to patch this to get it running in a modern environment.

Ensure you have pnpm installed, then run (I assume here that you already have this dependency included in your project):
pnpm patch @google/adk

Open the temporary directory pnpm provides (it'll be printed out in your terminal) and make two changes:

In package.json, add the standard lodash (CommonJS) to dependencies, matching the version of the existing lodash-es entry.
In dist/cjs/index.js, do a find-and-replace: change all instances of 'lodash-es' to 'lodash'.

Here's my package.json example:

{
  "dependencies": {
    "@google/genai": "^1.37.0",
    "@modelcontextprotocol/sdk": "^1.24.0",
    "google-auth-library": "^10.3.0",
    "lodash-es": "^4.17.22",
    "lodash": "^4.17.22", <----- This change here. Also ensure version numbers match for lodash-es (above) and lodash.
    "zod": "3.25.76"
  }

Apply the fix:
pnpm patch-commit <path-to-temp-dir>

Note: You can find the already-patched version in the tutorial repository if you want to skip this step.

2. The Webpack Shim (For NX/Webpack users)

If you are using Webpack (or NX), you need to configure it to handle mixed module types. Since your project is likely ESM, but the patched ADK package is CJS, your webpack.config.cjs (you should have a webpack.config.js file if your project is commonjs) needs explicit handling for output formatting to ensure Node runs it correctly.

// Key configuration: Save output as .cjs to run as a CommonJS module while maintaining TS paths (if your project is esm), otherwise, use .js
config.output = {
  ...config.output,
  // ensure the output is treated as commonjs
  filename: '[name].cjs',
  chunkFilename: '[name].chunk.cjs',
};

3. Implementing Persistent Sessions

Before we go to the actual meat of the implementation, I'll briefly explain the database entities and how they map to Google ADK types (feel free to skip this section if this is of no interest to you).

The Persistence Schema: Mapping ADK to PostgreSQL

You'll need a database schema that mirrors the ADK's internal logic. Using TypeORM, we define three primary entities: Sessions, Events, and Memories.

The Session Entity

The SessionEntity acts as the root. It tracks the overall state (the short-term memory) and metadata like appName and userId. This ensures that even if your server instances cycle, the agent knows exactly where it left off with a specific user.

import { BaseEntity } from '@app/common/lib/entities/base.entity.mjs';
import { Column, Entity, OneToMany, Index } from 'typeorm';
import { EventEntity } from './event.entity.mjs';

@Entity('adk_sessions')
export class SessionEntity extends BaseEntity {
  @Column({ type: 'varchar', length: 100 })
  @Index({ unique: true })
  sessionId: string; // The ADK session identifier (e.g. phone_YYYY-MM-DD)

  @Column({ type: 'varchar', length: 255 })
  appName: string;

  @Column({ type: 'varchar', length: 255 })
  userId: string; // The phone number identifier

  @Column({ type: 'jsonb', default: {} })
  state: Record<string, unknown>;

  @Column({ type: 'bigint' })
  lastUpdateTime: number;

  @Column({ type: 'boolean', default: false })
  isSummarized: boolean; // To avoid re-summarizing already summarized sessions.

  @OneToMany(() => EventEntity, (event: EventEntity) => event.session, { cascade: true })
  events: EventEntity[];
}

The Event Entity

Every interaction, whether it’s a user message, an agent response, or a tool execution, is an event. Notice the actions and content columns are jsonb. This is because ADK events are complex objects; storing them as JSON allows us to preserve the full payload without creating dozens of narrow columns.

import { Column, Entity, JoinColumn, ManyToOne, type Relation } from 'typeorm';
import { SessionEntity } from './session.entity.mjs';
import { BaseEntity } from '@app/common/lib/entities/base.entity.mjs';

@Entity('adk_events')
export class EventEntity extends BaseEntity {
  @Column({ type: 'varchar', length: 150, unique: true })
  eventId: string; // Global Event ID provided by ADK

  @ManyToOne(() => SessionEntity, (session) => session.events, { onDelete: 'CASCADE' })
  @JoinColumn()
  session: Relation<SessionEntity>;

  @Column({ type: 'jsonb' })
  actions: Record<string, unknown>; // Stores the full ADK Event object

  @Column({ type: 'varchar', length: 255 })
  invocationId: string;

  /**
   * Represents the entity that generated the event.
   * "user" - for input from the human user
   * "Agent Name" - (e.g. "General Agent") for output from a specific AI agent
   */
  @Column({ type: 'varchar', length: 255, nullable: true })
  author: string;

  @Column({ type: 'bigint' })
  timestamp: number;

  @Column({ type: 'varchar', length: 255, nullable: true })
  branch: string;

  @Column({ type: 'jsonb' })
  content: Record<string, unknown>; // Stores the full ADK Event object
}

The Memory Entity

Finally, the MemoryEntity handles long-term persistence. The key column here is embedding, which uses the vector type. This is what allows your agent to perform semantic searches over thousands of past interactions to find relevant historical context.

import { Column, Entity, Index } from 'typeorm';
import type { Content } from '@google/genai';
import { BaseEntity } from '@app/common/lib/entities/base.entity.mjs';

@Entity('adk_memories')
export class MemoryEntity extends BaseEntity {
  @Column({ type: 'varchar', length: 255 })
  @Index()
  userId: string;

  @Column({ type: 'varchar', length: 255 })
  @Index()
  appName: string;

  @Column({ type: 'jsonb' })
  content: Content;

  @Column({ type: 'varchar', length: 255, nullable: true })
  author: string;

  @Column({ type: 'bigint' })
  timestamp: number;

  @Column({ type: 'vector', length: 768, nullable: true })
  embedding: number[];
}

Actual Implementation: The Services Layer

We need a custom SessionService that persists state to PostgreSQL via TypeORM.

Critical Implementation Detail: When overriding the appendEvent method in your custom service, you must call super.appendEvent({ session, event }) first.
This call registers the incoming event with the ADK framework's current in-memory execution context for the active "turn." If you omit this, the framework won't recognize that an event occurred, even if you successfully save it to your database.

Here's a code snippet of my implementation (you can find the full code example in the project repo):

export class CloudSqlSessionService extends BaseSessionService {
  override async appendEvent({ session, event }: AppendEventRequest): Promise<Event> {
    // 1. Essential: Tell the ADK framework something happened in this turn
    await super.appendEvent({ session, event });

    // 2. Fetch current session state from DB
    const sessionEntity = await this.sessionsRepository.findOne({
      where: { sessionId: session.id },
    });

    // 3. Create event entity
    const eventEntity = this.eventsRepository.create({
      /* ... mapping logic ... */
    });

    await this.eventsRepository.save(eventEntity);
    
    // 4. Persist the updated state delta to Postgres
    await this.sessionsRepository.update(sessionEntity.id, {
      state: updatedState,
      lastUpdateTime: Date.now(),
    });

    return event;
  }
}

4. Long-Term Memory (Vector Search)

For long-term recall across different sessions, we implement a CloudSqlMemoryService. This utilizes pgvector within PostgreSQL for similarity search and Gemini’s text-embedding-004 model for generating embeddings.
Important: You MUST enable the pgvector extension in postgres. Refer to this link on how to install this extension: installation instructions, then enable it by running this command:

CREATE EXTENSION IF NOT EXISTS vector;

Here's the code example for saving and searching your long term memory (again, the full code example is available in the project repo):

export class CloudSqlSessionService extends BaseSessionService {
  async addSessionToMemory(session: Session): Promise<void> {
    console.info(`------------- Adding session ${session.id} events (total events: ${session.events.length}) to memory for user ${session.userId} -------------`);

    for (const event of session.events) {
      console.log('event: ', event);
      console.log('event.content: ', event.content);
      const textContent = stringifyContent(event).trim();
      console.log('textContent: ', textContent, '\n\n');
      if (!textContent) {
        continue;
      }

      try {
        const embedding = await this.generateEmbedding(textContent);

        const memory = this.memoryRepository.create({
          userId: session.userId,
          appName: session.appName,
          content: event.content,
          author: event.author || 'unknown',
          timestamp: event.timestamp,
          embedding: embedding,
        });

        await this.memoryRepository.save(memory);
      } catch (e) {
        console.error(`Failed to generate embedding for event in session ${session.id}: ${e.message}`, e);
      }
    }
  }

  /**
   * Searches for relevant memories using vector similarity.
   */
  async searchMemory(request: SearchMemoryRequest): Promise<SearchMemoryResponse> {
    const { appName, userId, query } = request;

    try {
      const queryEmbedding = await this.generateEmbedding(query);
      const vectorStr = `[${queryEmbedding.join(',')}]`;

      // Use pgvector's cosine distance operator <=>
      const results = await this.dataSource.query(
        `SELECT id, content, author, timestamp
         FROM adk_memories
         WHERE "userId" = $1
           AND "appName" = $2
         ORDER BY embedding <=> $3
           LIMIT 5`,
        [userId, appName, vectorStr],
      );

      const memories: MemoryEntry[] = results.map(r => ({
        content: r.content,
        author: r.author,
        timestamp: r.timestamp,
      }));

      return { memories };
    } catch (error) {
      console.error(`Failed to search memory: ${error.message}`, error);
      return { memories: [] };
    }
  }

  /**
   * Generates an embedding for the given text.
   */
  private async generateEmbedding(text: string): Promise<number[]> {
    const response = await this._genAI.models.embedContent({
      model: 'text-embedding-004',
      contents: [text],
    });
    if (!response.embeddings || response.embeddings.length === 0) {
      throw new Error('No embeddings returned from API');
    }
    return response.embeddings[0].values!;
  }
}

5) Giving the Agent a "Memory" Tool

With your CloudSqlMemoryService in place, you need a way for the agent to actually trigger a search. In the ADK, this is done by defining a FunctionTool. This tool acts as the bridge between the LLM’s reasoning and your vector database.

When the agent realizes it lacks specific historical context (e.g., "What did we discuss about the project architecture last month?"), it will call this load_memory tool.

import { FunctionTool } from '@google/adk';
import { z } from 'zod';

/**
 * A built-in tool that allows agents to load relevant long-term memories.
 */
const loadMemoryTool = new FunctionTool({
  name: 'load_memory',
  description: 'Loads relevant long-term memories for the current user based on a query.',
  parameters: z.object({
    query: z.string().describe('The search query for memory.'),
  }),
  execute: async ({ query }, toolContext) => {
    if (!toolContext) {
      throw new Error('Tool context is missing');
    }
    // The ADK passes the active MemoryService via toolContext
    return await toolContext.searchMemory(query);
  },
});

export default loadMemoryTool;

Why `toolContext` is Key

Notice that we don't import the memory service directly into the tool. Instead, we pull it from the toolContext. This is an architectural win: it keeps your tools decoupled from specific database implementations and ensures that the search is automatically scoped to the correct userId and appName currently active in the session.

6) Orchestrating the Agent (The "General Agent" Factory)

Now that we have our persistent services and our load_memory tool, we need to wire them into the agent. In a production NestJS or Express environment, you’ll often want to pass external dependencies (like database connections or configurations) into your agent definition.

A clean way to do this is by using a factory function. This allows you to inject your MainContext directly into the agent and its sub-agents.

import { LlmAgent } from '@google/adk';
import loadMemoryTool from '../load-memory.tool.mjs';

// Factory function to inject context and dependencies
export default function generalAgent(context: MainContext) {
  return new LlmAgent({
    name: 'general_agent',
    model: 'gemini-1.5-pro', // Use your preferred model constant
    instruction: `
  **Identity:**
  You are SkyCast, a helpful weather assistant. 
  **Current Date and Time:** {current_datetime}
  
  **Long-term Memory:** You can access the user's favorite locations and past queries by calling the \`load_memory\` tool.
  
  **User profile:** {user:profile?} 
  Use this to identify the user's home city or preferred units.

  **Core Responsibilities:**
  1. **Triage & Routing:** Route users to specific sub-agents (user, forecast, or alerts).
  2. **Memory Retrieval:** Proactively check memory for saved preferences if the user is vague.
  `,
    tools: [loadMemoryTool],
    // Sub-agents receive the same context for consistent behavior
    subAgents: [userAgent(context), forecastAgent(context), alertsAgent(context)],
  });
}

Why This Structure Wins

Recursive Context: By passing context into subAgents, every agent in the hierarchy shares the same understanding of the user's session and database connection.
Dynamic Instructions: Notice the placeholders like {current_datetime} and {user:profile?}. The ADK fills these in at runtime, giving the LLM immediate "short-term" context before it even decides to call the "long-term" load_memory tool.
Agent Handoffs: The subAgents array defines the "routing" capabilities. If the user asks to register, the General Agent handles the handoff to the user_agent automatically based on your routing rules.

Architectural Benefits

By moving off the default in-memory implementations and using a factory pattern for orchestration, we gain several production advantages:

Stateless Application Tier: Offloading state to PostgreSQL means your application containers can scale horizontally without losing track of a user's progress.
Session Durability: User context survives deployments, crashes, or server restarts.
Complex Multi-Agent Logic: The factory pattern allows you to build a sophisticated "brain" that delegates specialized tasks (like weather alerts or billing) while maintaining a single source of truth for user memory.

Wrapping Up

By patching the ADK's module issues, implementing database-backed services, and orchestrating your agents through a clean factory pattern, you move from a prototype to a production-ready system. The result is a robust backend that handles state correctly, regardless of infrastructure churn.

Is OpenAI really running ChatGPT on a single PostgreSQL instance?

Tue, 27 Jan 2026 09:05:05 +0000

The headline of OpenAI’s recent article https://openai.com/index/scaling-postgresql/ feels a bit clickbaity,
If they truly used only one database instance, ChatGPT would’ve been dead on arrival.

But the reality is far more impressive: they’re using the right tool for each layer of the persistent stack:

They push PostgreSQL to its absolute limits, with a single primary writer, yes, but backed by nearly 50 read replicas across global regions.
For write-heavy workloads, they’ve wisely migrated to sharded systems like Azure Cosmos DB.
They’ve added layers of resilience:
- connection pooling with PgBouncer,
- query rate limiting,
- caching with lock leasing,
- cascading replication (in testing),
- strict schema-change policies.

It’s not magic, it’s mature, thoughtful engineering at scale.

If you work with databases, this post is absolutely worth reading.

Why Updating Data in Postgres Is More Expensive Than You Think

Tue, 20 Jan 2026 03:30:41 +0000

Context

I was recently tasked with optimizing stockinsights.ai’s full text search system for one of our core features: keyword search over filings.

Previously, this worked by storing entire documents as raw strings in MongoDB and scanning through them at query time to return surrounding snippets. As you can imagine, this approach became increasingly slow and inefficient as the number of filings kept growing.

This clearly needed to be fixed.

While looking at the existing system, I noticed that we already stored chunked text snippets in a separate Postgres instance as part of our embeddings tables for semantic search. Each chunk already existed as a row, paired with an embedding.

It seemed obvious that we should reuse this same chunked data for keyword search as well.

At a high level, the idea looked something like this:

Keyword search across snippets
→ return matching text chunks
→ rank documents higher if more chunks match
→ return one representative snippet per document

All I needed to do, or so I thought, was add another column to store tsvectors (Postgres’s internal data structure for full text search) and create a GIN index on top of it.

How hard could it be?

The obvious solution

Postgres has built-in full text search using tsvector and tsquery. The plan was straightforward:

Add a tsvector column to the embeddings table
Populate it from the existing text column
Create a GIN index
Query snippets directly instead of scanning raw documents

Note: We could have normalized this into a separate table, but at a startup, we often accept some denormalization in favor of simpler systems.

From an application point of view, this was almost trivial. No new data model, no new pipeline, just reuse what already existed.

The mistake was assuming that a small logical change would translate into a small physical change inside the database.

When the costs showed up

**
Initially, nothing failed. The migration just took much longer than expected.

Each document type table had roughly 2 to 5 million rows. Updating even 10,000 rows was taking hours, and each iteration made the next one slower. What started as a small backfill quickly turned into something that felt unbounded in cost.

Populating the new column dragged on, disk usage grew far beyond what the additional data alone could justify, and creating the GIN index noticeably impacted write throughput while it was running.

More concerning was that queries themselves began to slow down. Reads that were previously fast started degrading as the table and indexes bloated, even though the query plans hadn’t meaningfully changed.

At that point, it was clear that something deeper was going on.

UPDATE in Postgres is not mutation

Postgres stores tables as heap files. A heap file is an unordered collection of tuples, and tuples are immutable.

When you run an UPDATE, Postgres does not modify a row in place. Instead, it:

Inserts a new version of the row
Marks the old version as no longer visible
Leaves cleanup for later
This design exists to support MVCC (multi-version concurrency control), which allows readers and writers to operate concurrently by keeping multiple versions of rows and resolving visibility at read time.

In practice, this means UPDATE is not a mutation of an existing row, but the creation of a new version of that row.

Why adding a column rewrites every row

A Postgres row is stored as a single physical tuple containing all columns. What I learned here is that there is no column-level storage, and therefore no partial row update.

When you populate a new column on an existing table, Postgres must:

Copy every existing column value
Append the new column value
Write an entirely new tuple
Even if every other column remains unchanged, the entire row is rewritten. On a large table, this effectively means rewriting the entire dataset, not just adding one column.

Why text, tsvector, and embeddings make this worse
Our table already stored text snippets and embeddings, and we were about to add tsvectors on top of that.

A few things compound the cost here:

Text columns are variable length. Large values often exceed the inline storage limit. When that happens, Postgres uses TOAST.
TOAST stores oversized column values out of line and keeps pointers to them in the main tuple.

Even when data is TOASTed, updating the row still requires rewriting the main tuple and, in some cases, the TOAST entries themselves.

tsvector expands data. It is not just the original text. It stores lexemes (similar to tokens) along with positional metadata, and it is also typically stored via TOAST.

Embeddings make rows wide. Embeddings are large binary float arrays stored alongside text and indexed separately. Combined with text and tsvector, each row becomes wide, and wide rows are expensive to rewrite.

This combination pushes right up against the limits of what an immutable row store is comfortable handling.

Indexes amplify the cost

On top of this, the table already had indexes.

Every new tuple version requires index maintenance. For each index, Postgres must:

Insert new index entries
Maintain index structure
Write corresponding WAL records
GIN indexes are especially heavy. A single row can generate many index entries, often one per token in the tsvector.

While most indexes in Postgres are B-tree based, GIN indexes are inverted indexes.

WAL (write-ahead log) is the sequential log used for crash recovery and replication. More index work means more WAL traffic, which directly affects disk usage, replication lag, and write latency.

Indexes make reads fast, but they multiply write cost. That tradeoff is unavoidable.

An alternate approach: COPY, swap, drop

While digging through Postgres blogs and discussions, I came across an alternative approach for modifying very large tables.

Instead of updating the table in place, you:

Create a new table with the desired schema
Copy data into it
Recreate indexes
Swap it with the old table
Drop the old table
The COPY-swap-drop approach.

This has a few important properties:

Each row is written exactly once
No dead tuples are created
Indexes are built cleanly
There is no incremental index maintenance during updates
By contrast, UPDATE creates deferred work. Dead tuples accumulate and must be cleaned up later, and indexes are updated repeatedly for each new row version.

In an immutable storage model, rewriting data cleanly can often be cheaper than mutating it incrementally.

Final design

One additional factor that helped was simply scaling the RDS instance. More memory meant more rows could be processed efficiently during the rebuild, which significantly reduced runtime.

The final workflow looked like this:

Upgrade the instance
Use COPY-swap-drop to rebuild the table
Recreate indexes
This approach ended up being far faster and more predictable than trying to UPDATE rows in place.

Oracle migrations reveal hidden technical debt. Here's what developers need to know before moving.

Mon, 13 Oct 2025 21:25:28 +0000

The Real Cost of Oracle Technical Debt: What Developers Learn During Migration

Oracle databases have been around for 40 years. That's a lot of time to accumulate technical debt.

Spencer Kimball, CEO of Cockroach Labs, spends his days helping Fortune 100 companies migrate off Oracle. The technical challenges he sees are consistent: undocumented stored procedures, excessive use of legacy patterns, and architectural decisions made when networking was slow and security wasn't a priority.

"The people that wrote them are gone from our company," Kimball says of the legacy applications his customers want to migrate. "We want help solving this."

PostgreSQL is Eating the World

The first thing to understand about modern database migrations is that developers don't want Oracle's dialect anymore. They want PostgreSQL.

"Postgres is actively eating the world," Kimball says. When Cockroach Labs started 11 years ago, PostgreSQL wasn't the default choice. Now it is.

This shift shows up everywhere. AWS Aurora moved its center of gravity from MySQL to PostgreSQL. Even TiDB, which started as a MySQL engine, recently added PostgreSQL support. IBM's customers are asking for "a Postgres database for the enterprise."

The reason is straightforward: PostgreSQL is the standard for new development. If you're building something new, you're building it on Postgres.

Distributed vs Monolithic: Where Developers Get Surprised

The migration from Oracle to a distributed database like CockroachDB involves more than syntax changes. It requires a shift in thinking.

"Databases have grown up over the last 50 years with this perspective that the database is a tightly coupled monolithic system," Kimball explains. Oracle was built for InfiniBand networks and specialized hardware. Those requirements don't match cloud reality.

The most common surprise? Auto-incrementing IDs.

In a monolithic database, generating sequential IDs is fast because everything runs on one machine. In a distributed system with consensus requirements, that pattern creates bottlenecks.

The solution is UUID generation with node IDs embedded. These IDs are unique across the cluster, sort correctly, and create zero hot spots. It's a small change, but every developer on a migration team needs to understand it.

"Some of this is solved just by retraining developers," Kimball says. "Some of it is done by tooling that will look through a schema and say, 'Here's an auto-incrementing ID. We recommend you replace it with UUID generation.'"

AI is Unraveling Oracle Lock-In

Stored procedures represent Oracle's most effective lock-in mechanism. They were a reasonable architectural choice when network latency was high and database security was limited. DBAs pushed application logic into the database where it could be controlled and protected.

The result is mountains of undocumented code. Sometimes it's PL/SQL. Sometimes it's C routines with no unit tests and no documentation.

AI tools are changing the migration economics. Kimball says AI can analyze legacy stored procedures, generate documentation, create unit tests, and suggest complete migrations to PL/pgSQL (PostgreSQL's procedural language).

"The sharpest decrease in costs is going to come from AI," he says. Where migrations used to require 100+ consultants over three years at a cost of $10 million, AI-assisted approaches can reduce that significantly.

The tools from Cognition, Devin, and even Claude can handle much of the heavy lifting that previously required specialized Oracle knowledge.

Architecture Built for AI Scale

The architectural differences between distributed and monolithic databases matter more as AI workloads increase.

"You can't even get there without a distributed architecture," Kimball says about handling AI-scale traffic. The issue isn't just throughput. It's cost.

Cockroach Labs routinely runs 300-node test clusters for weeks at a time. These clusters contain petabytes of data. Three weeks used to cost $500,000.

Now they run these tests on spot instances and save 80%. "Could you run a traditional database across spot instances?" Kimball asks. "You'd be down most of the time."

A distributed database built for resilience can handle spot instances getting preempted. It replicates, rebalances, and keeps running through the chaos.

The company is targeting a 10x cost improvement through a combination of optimizations: spot instances, multi-tenancy (packing multiple workloads on the same physical database), layered storage (S3, locally attached SSDs, EBS), and continuous performance optimization.

They've already achieved 2x performance improvement on the same hardware this year.

Resilience Under Pressure

Performance benchmarks under ideal conditions don't tell you much about production reality. Cockroach Labs released a "Performance under Adversity" benchmark that measures how databases behave during chaos: pod failures, network partitions, disk stalls.

They tested CockroachDB against Oracle GDD 23ai (the first version of Oracle that can support GDD in a Raft configuration). The results are documented in detail, and the benchmark methodology is open so anyone can reproduce it.

CockroachDB encountered 56 errors resulting in 51 seconds of downtime across all chaos tests. Oracle GDD encountered 300 errors resulting in 369 seconds of downtime (over six minutes).

That's 5x more resilient.

The difference matters most during "grey failures"—those problems that don't immediately take the database offline but slowly degrade latency and breach SLAs.

The Migration Path Forward

If you're stuck on Oracle and planning a migration, here's what to focus on:

Audit your stored procedures. Use AI tools to document what you have before you start planning the migration. You probably have more technical debt than you think.

Understand distributed architecture patterns. Auto-incrementing IDs are just one example. Get your team trained on how distributed consensus works and where the architectural differences matter.

Test for resilience, not just performance. Peak performance under ideal conditions matters less than consistent performance under realistic failure scenarios.

Factor in the full cost. Don't just compare license fees. Include the cost of downtime, the complexity of running the database, and the total cost of ownership over time.

The cloud forced many companies to reconsider Oracle. AI is accelerating that shift. If you're going to modernize anyway—and you probably are—make sure you understand the technical implications before you start.

PostgreSQL 18 Uncovered: Asynchronous I/O, UUIDv7, and Pain-Free Upgrades

Tue, 30 Sep 2025 13:04:23 +0000

As is tradition around this time of year, the global PostgreSQL community has surprised us with a new major version, and this time is no exception. On September 25th, the PostgreSQL Global Development Group officially released PostgreSQL 18. I'll tell you right now: it's one of the most exciting updates in recent years, especially if you're obsessed with performance and operational efficiency.

In this article, we'll break down the official announcement to uncover the new features that will genuinely change the way we work—from developing custom software to deploying high-availability architectures. Forget minor tweaks; these are game-changing improvements. Let's dive in!

The Key New Features in PostgreSQL 18

After analyzing the announcement, I've identified four areas that I consider revolutionary.

The Headliner: Asynchronous I/O (AIO) for a Massive Performance Boost

This is the crown jewel of PostgreSQL 18. Until now, Postgres relied on the operating system's readahead mechanisms to speed up data retrieval, but this isn't always efficient.

PostgreSQL 18 introduces a new Asynchronous I/O (AIO) subsystem. Instead of requesting a block of data and waiting, it can now issue multiple I/O requests in parallel.

What does this mean? For operations that read a lot of data from disk (like sequential scans, VACUUM, etc.), the impact is huge. Community benchmarks have shown performance improvements of up to 3x in certain scenarios.
How do you use it? It's controlled by the new io_method parameter, which lets you choose between different implementations like io_uring (on Linux) or stick with the classic synchronous behavior.

Bottom line: If you have a read-intensive (I/O-bound) workload, like analytics systems or applications with large datasets, you're going to see a night-and-day difference.

Upgrades Are Finally Pain-Free

Anyone who has managed a production database knows the post-upgrade dread: queries run slow until ANALYZE rebuilds the planner statistics. PostgreSQL 18 tackles this problem head-on.

Statistics Preservation: Finally! It's now possible to keep planner statistics during a major version upgrade. This means your database will perform at its optimal level almost immediately after upgrading, eliminating that painful performance degradation phase.
Faster pg_upgrade: The upgrade utility itself has been optimized. It can now run checks in parallel (--jobs) and has a new --swap option that exchanges data directories instead of copying them, dramatically speeding up the process.

Bottom line: The process of upgrading to a new major version will be much faster and less disruptive to your applications.

Developer Gems You'll Love

This release is packed with features that make our lives easier and allow us to write more modern and efficient code.

Say Hello to uuidv7()!: A native function to generate v7 UUIDs has been added. Unlike random v4 UUIDs, v7 UUIDs are time-ordered. Using them as primary keys drastically improves B-Tree index performance by reducing fragmentation and improving data locality. This is a fundamental change for high-insertion tables!
Virtual Generated Columns: Generated columns (GENERATED COLUMNS) are now virtual by default. This means the value is computed at query time instead of being stored on disk, saving space for values that aren't needed constantly.
RETURNING with OLD and NEW: DML statements (INSERT, UPDATE, DELETE, MERGE) can now access both the old (OLD) and new (NEW) values directly in the RETURNING clause. This is fantastic for creating audit logs or implementing change data capture logic in a single query.

Bottom line: Native tools for modern design patterns that previously required more complex workarounds.

Authentication, Security, and Important Changes to Note

Native OAuth 2.0 Support: This is one of the hidden gems of the release, especially for professional environments. In practice, it means you can now authenticate database users through external identity providers like Auth0, Okta, Microsoft Entra ID (Azure AD), or Google. Technically, this is enabled directly in the pg_hba.conf file with a new oauth authentication method. You configure key parameters like the issuer (your identity provider's URL) and the scope (the permissions the access token must have). The token validation itself is modular and delegated to specialized libraries, making the system highly flexible. The result is that instead of managing local passwords, you can delegate authentication to your central Single Sign-On (SSO) system. This not only radically simplifies user management but also elevates security by enabling centralized policies like multi-factor authentication (MFA).
The End of an Era: MD5 Deprecation: Although secure authentication with SCRAM has been available for several versions, PostgreSQL 18 takes the final step: it officially deprecates the use of MD5 for passwords. Support will be completely removed in a future release, so there are no more excuses. The community's message is clear: if you're still using MD5, migrating to SCRAM is an urgent and high-priority task.
Default Checksums: New clusters created with initdb will now have page checksums enabled by default. This improves data integrity but is a crucial detail to consider when planning an upgrade from a cluster without them.

My Critical Perspective: What This Means for Your Projects

Is Asynchronous I/O a silver bullet? No. It's a powerful tool for I/O-bound workloads. If your bottleneck is the CPU, you won't see a magical improvement. But for deploying analytics applications or AI services that process large volumes of data, the impact will be enormous.
The Real Operational Game-Changer: While AIO gets the headlines, I believe preserving statistics on upgrades is the single most important feature for production systems. It eliminates one of the biggest headaches and risks associated with an upgrade, reducing uncertainty and making it easier to keep your tech stack up to date.
For Custom Software Development: My recommendation is to start using uuidv7() as the default primary key on all new tables where it makes sense. It's an easy and free performance win. The improvements to RETURNING will also greatly simplify audit logic in the backends I build.
The Ace Up Your Sleeve for Professional Projects (OAuth 2.0): Don't underestimate this feature. If you build custom software for businesses, as I do, offering integration with their SSO system (like Microsoft 365 or Google Workspace) is no longer a nightmare. It's an enterprise-grade feature that now comes standard, allowing you to sell more secure and better-integrated projects into your clients' ecosystems. It's a powerful differentiator.
Immediate Call to Action: Audit your systems. If you're still using MD5 authentication, plan your migration to SCRAM now. This isn't a suggestion; it's an imminent security requirement.

Hands-On: Test PostgreSQL 18 with Docker and Examples

To understand why the deprecation of MD5 is such a big deal, there's nothing like seeing it in action. We'll use Docker to set up a lab with two parallel instances of PostgreSQL 18: one configured to use the old and now-obsolete MD5 method, and another using the modern and secure SCRAM, which has been the recommended method for years. This way, you can see the security difference for yourself.

Requirements

To execute these examples you will need to install Docker.

Step 1: Set Up the Comparison Environment

Create a file named docker-compose.yml. Notice that we're defining the two databases (db_md5 and db_scram), the adminer web interface, and an extra container named pg_client that will act as our toolbox with psql pre-installed.

docker-compose.yml (MD5 vs. SCRAM Comparison)

services:
  db_md5:
    image: postgres:18
    container_name: postgres18_md5
    restart: always
    command: postgres -c password_encryption=md5 -c io_method=sync
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: testdb
    ports:
      - "5432:5432"
    volumes:
      - postgres_data_md5:/var/lib/postgresql/data

  db_scram:
    image: postgres:18
    container_name: postgres18_scram
    restart: always
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: testdb
    ports:
      - "5433:5432"
    volumes:
      - postgres_data_scram:/var/lib/postgresql/data

  pg_client:
    image: postgres:18
    container_name: pg_client
    # This command keeps the container running so we can exec into it
    command: tail -f /dev/null

  adminer:
    image: adminer
    container_name: postgres18_adminer
    restart: always
    ports:
      - "8080:8080"

volumes:
  postgres_data_md5:
  postgres_data_scram:

In a terminal, run docker-compose up -d or docker compose up -d. Your entire environment is now ready, with no need to install anything else on your machine.

Note: The password for the user in all connections that require it (psql, Adminer) is password, as defined in the docker-compose.yml file.

Step 2: Run the Examples and Compare

For the terminal examples, we'll use our pg_client container.

How to use the terminal:

Open a shell session inside the client container with this command:
```
docker exec -it pg_client bash
```
Once inside (you'll see a prompt like root@...:/#), you can run psql commands against the databases using their service names (db_scram or db_md5).

Connect to the SCRAM (secure) instance:

Terminal: Inside the pg_client container, run:
```
psql -h db_scram -U user -d testdb
```
Adminer (http://localhost:8080): Server: db_scram

![Login Adminer - db_scram][1]

Example 1: The new `uuidv7()`

-- Compare the results of the two UUID types
SELECT uuidv4(), uuidv7();

Run the query a few times. You'll notice that uuidv4() values are completely random, while uuidv7() values start with a similar sequence that changes slightly. That initial part is the timestamp, which allows the index to sort them efficiently!

testdb=# SELECT uuidv4(), uuidv7();
                uuidv4                |                uuidv7                
--------------------------------------+--------------------------------------
 a3c361d2-4aa2-49fe-84fd-247d249484e0 | 019999e7-70ce-7e41-ad73-97cd6e42f2a8
(1 row)

testdb=# SELECT uuidv4(), uuidv7();
                uuidv4                |                uuidv7                
--------------------------------------+--------------------------------------
 3c8115a7-8667-47e2-971c-b7ce61504572 | 019999e7-7713-75db-a7ff-e7c2ebbfb47f
(1 row)

testdb=# SELECT uuidv4(), uuidv7();
                uuidv4                |                uuidv7                
--------------------------------------+--------------------------------------
 1fc5216d-a874-418b-abcb-0fd389823abc | 019999e7-7c0f-7a0a-a364-6a3a536a0b6c
(1 row)

Example 2: Virtual Generated Columns

CREATE TABLE products (
    name TEXT,
    price_eur NUMERIC,
    price_usd NUMERIC GENERATED ALWAYS AS (price_eur * 1.07) VIRTUAL
);

INSERT INTO products (name, price_eur) VALUES ('Mechanical Keyboard', 80);

SELECT * FROM products;

The price_usd column doesn't take up disk space; it's calculated every time you query it.

testdb=# CREATE TABLE products (
    name TEXT,
    price_eur NUMERIC,
    price_usd NUMERIC GENERATED ALWAYS AS (price_eur * 1.07) VIRTUAL
);
CREATE TABLE
testdb=# INSERT INTO products (name, price_eur) VALUES ('Mechanical Keyboard', 80);
INSERT 0 1
testdb=# SELECT * FROM products;
        name         | price_eur | price_usd 
---------------------+-----------+-----------
 Mechanical Keyboard |        80 |     85.60
(1 row)

Example 3: `RETURNING` with `OLD` and `NEW` Values

CREATE TABLE price_audit (
    product_name TEXT,
    old_price NUMERIC,
    new_price NUMERIC,
    changed_at TIMESTAMPTZ DEFAULT now()
);

WITH updated AS (
    UPDATE products
    SET price_eur = 85
    WHERE name = 'Mechanical Keyboard'
    RETURNING OLD.name, OLD.price_eur AS old_price, NEW.price_eur AS new_price
)
INSERT INTO price_audit (product_name, old_price, new_price)
SELECT name, old_price, new_price FROM updated;

SELECT * FROM price_audit;

It's like magic! No complex triggers or double queries needed.

testdb=# CREATE TABLE price_audit (
    product_name TEXT,
    old_price NUMERIC,
    new_price NUMERIC,
    changed_at TIMESTAMPTZ DEFAULT now()
);
CREATE TABLE
testdb=# WITH updated AS (
    UPDATE products
    SET price_eur = 85
    WHERE name = 'Mechanical Keyboard'
    RETURNING OLD.name, OLD.price_eur AS old_price, NEW.price_eur AS new_price
)
INSERT INTO price_audit (product_name, old_price, new_price)
SELECT name, old_price, new_price FROM updated;
INSERT 0 1
testdb=# SELECT * FROM price_audit;
    product_name     | old_price | new_price |          changed_at           
---------------------+-----------+-----------+-------------------------------
 Mechanical Keyboard |        80 |        85 | 2025-09-30 09:16:50.260192+00
(1 row)

Step 3: Experimenting with Asynchronous I/O (AIO)

Demonstrating a performance improvement like AIO in a simple example is tricky, as it heavily depends on your hardware (especially your disk speed). However, we can set up a small lab for you to experiment and measure the impact on your own system.

With our Docker setup, we have the perfect lab: the db_md5 service starts with synchronous I/O (io_method=sync), and the db_scram service uses the PostgreSQL 18 default, which is with AIO enabled.

1. Prepare a Large Dataset

We'll use pgbench, a tool included with PostgreSQL, to create test data in both instances.

From your terminal, run these two commands. Each will take a couple of minutes to populate its respective database with \~1.5 GB of data:

# Populate the database WITHOUT AIO (db_md5)
docker exec pg_client bash -c "PGPASSWORD=password pgbench -i -s 100 -h db_md5 -U user testdb"
  
# Populate the database WITH AIO (db_scram)
docker exec pg_client bash -c "PGPASSWORD=password pgbench -i -s 100 -h db_scram -U user testdb"

2. Run the Tests and Compare

For a fair comparison, we'll first restart both containers to clear memory caches.

From your terminal, run:
```
docker restart postgres18_md5 postgres18_scram
```
Wait a few seconds for the databases to start up.
Now, enter the client container to run the tests:
```
docker exec -it pg_client bash
```

Test on the server WITHOUT AIO:

# Connect to db_md5 and measure the time
PGPASSWORD=password psql -h db_md5 -U user -d testdb -c "\timing on" -c "SELECT count(*) FROM pgbench_accounts;"

Take note of the time it returns.

root@5dfdfaef5e1a:/# PGPASSWORD=password psql -h db_md5 -U user -d testdb -c "\timing on" -c "SELECT count(*) FROM pgbench_accounts;"
Timing is on.
  count   
----------
 10000000
(1 row)
  
Time: 828.820 ms

Test on the server WITH AIO:

# Connect to db_scram and measure the time
PGPASSWORD=password psql -h db_scram -U user -d testdb -c "\timing on" -c "SELECT count(*) FROM pgbench_accounts;"

Take note of this second time.

root@5dfdfaef5e1a:/# PGPASSWORD=password psql -h db_scram -U user -d testdb -c "\timing on" -c "SELECT count(*) FROM pgbench_accounts;"
Timing is on.
  count   
----------
 10000000
(1 row)
  
Time: 539.800 ms

3. Analyze the Results

Compare the time from the second test (WITH AIO) with the first (WITHOUT AIO). On a system with fast disks (SSD/NVMe), you should see a noticeable reduction in execution time. In my case, the time dropped from 828.820 ms to 539.800 ms, an improvement of 34.87%.

Important: The improvement you see will depend heavily on your hardware, operating system, and Docker configuration. The goal of this experiment is to give you the tools to measure it in your own environment.

Step 4: The Definitive Authentication Comparison

Now, let's create a user on each database to see the crucial difference.

1. On the MD5 (insecure) instance:
Connect to it:

Terminal: Inside the pg_client container, run:
```
psql -h db_md5 -U user -d testdb
```
Adminer: Server: db_md5

And run the following SQL:

CREATE USER test_md5_user WITH PASSWORD 'a_simple_password';
SELECT rolname, rolpassword FROM pg_authid WHERE rolname = 'test_md5_user';

The result will show the easily identifiable MD5 hash:

testdb=# CREATE USER test_md5_user WITH PASSWORD 'a_simple_password';
WARNING:  setting an MD5-encrypted password
DETAIL:  MD5 password support is deprecated and will be removed in a future release of PostgreSQL.
HINT:  Refer to the PostgreSQL documentation for details about migrating to another password type.
CREATE ROLE
testdb=# SELECT rolname, rolpassword FROM pg


  [1]: https://coderlegion.com/?qa=blob&qa_blobid=3116745045302959377

What actually happens when you run an SQL query?

Sun, 07 Sep 2025 11:40:41 +0000

Do you know what actually happens when you run an SQL query?

It’s not as simple as: “SQL just fetches rows from a table.”

Behind the scenes, your query goes through a step-by-step execution order and knowing this is often an interview question for beginners.

Execution Order:

FROM (pick the table)
WHERE (filter rows)
GROUP BY (make buckets)
HAVING (filter buckets)
SELECT (choose what to show)
ORDER BY (sort the results)
LIMIT (control how much to show)

Notice how WHERE comes before grouping, while HAVING comes after grouping.

Notice how ORDER BY is one of the last steps, even though we often write it after SELECT.

Common Confusions:

a) WHERE vs HAVING

WHERE filters rows before grouping.
HAVING filters groups after aggregation.

Example:

WHERE city = 'Delhi' → filter students from Delhi first.

HAVING COUNT(*) > 10 → only keep cities with more than 10 students.

b) GROUP BY vs ORDER BY

GROUP BY → bundles rows into categories (e.g., students per city).

ORDER BY → decides the sequence in which results are displayed (e.g., sort cities by student count).

c) LIMIT with ORDER BY

Important tasks like “find top 3 highest salaries” or “top 5 cheapest products.”

If you found this useful, drop a like or share it with someone preparing for SQL interviews.

Follow for more such insightful posts , I’m breaking SQL down into small, clear lessons you’ll actually remember.

Building an Order Processing Saga with MassTransit

Tue, 12 Aug 2025 09:22:53 +0000

MassTransit has been around since 2007 and has matured into one of the most stable and feature-rich open-source .NET messaging frameworks.

It acts like a factory for message brokers, meaning you can switch between providers such as RabbitMQ, Azure Service Bus, ActiveMQ, Amazon SQS, and others with minimal configuration changes.

This makes it ideal for enterprise-grade distributed systems where you want flexibility without vendor lock-in.

Why MassTransit?

Stability: Over a decade of continuous development and production use.

Multi-Broker Support: Easily swap between Azure Service Bus, RabbitMQ, Amazon SQS, etc., without rewriting business logic.

Developer Friendly: Built for .NET developers with excellent documentation and integration with DI, EF Core, and ASP.NET Core.

Advanced Features:

Outbox pattern
Request/Response messaging
Retry and circuit-breaker policies
Sagas and State Machines
Message scheduling

One of the Coolest Features: The State Machine

A State Machine allows you to model complex workflows where messages arrive in any order, but your application processes them in a controlled, state-aware sequence.

For example, in this Order Processing Saga:

An order is submitted.
The saga waits for stock confirmation.
If stock is reserved, the saga waits for payment authorization.
Once payment is authorized, the order is marked as completed.
If any step fails, the saga can transition to a failed state and trigger compensating actions.

This approach eliminates spaghetti code for multi-step processes and makes workflows easy to visualize and maintain.

Technology Stack Used in This Project
.NET 9

MassTransit (with Saga State Machine)
RabbitMQ (message broker)
PostgreSQL (saga persistence)
Entity Framework Core (database access)
Docker (local RabbitMQ/PostgreSQL setup)

Source Code
Full working implementation available here:
GitHub – OutboxSagaDemo

References

MassTransit Documentation – https://masstransit.io/

MassTransit GitHub Repository – https://github.com/MassTransit/MassTransit

MassTransit Saga State Machine Guide – https://masstransit.io/documentation/patterns/saga

RabbitMQ Documentation – https://www.rabbitmq.com/documentation.html

PostgreSQL Documentation – https://www.postgresql.org/docs/

Entity Framework Core Docs – https://learn.microsoft.com/en-us/ef/core/

PostgreSQL Client Certificate Authentication: Complete Setup Guide

Fri, 01 Aug 2025 12:50:19 +0000

Client certificate authentication in PostgreSQL provides a secure, passwordless way to authenticate users. Instead of relying on passwords, clients present valid X.509 certificates to prove their identity.

What to Configure

Server-Side Requirements

SSL certificates: Server certificate + client certificates
PostgreSQL SSL settings: Enable SSL and configure certificate paths
Authentication rules: Configure pg_hba.conf for certificate-based auth
User mapping: Link certificate Common Names to database users

Client-Side Requirements

Client certificate: Valid X.509 certificate for the user
Private key: Matching private key for the certificate
Root certificate: Server's certificate for verification
Connection parameters: Proper SSL mode and certificate paths

How to Configure

1. Generate SSL Certificates

# Create SSL directory
mkdir -p /var/lib/postgresql/17/main/ssl
cd /var/lib/postgresql/17/main/ssl

# Generate server certificate
openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr \
  -subj "/C=US/ST=State/L=City/O=Org/OU=OrgUnit/CN=localhost"
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

# Generate client certificate for user 'appuser'
openssl genrsa -out appuser.key 4096
openssl req -new -key appuser.key -out appuser.csr \
  -subj "/C=US/ST=State/L=City/O=Org/OU=OrgUnit/CN=appuser"
openssl x509 -req -days 365 -in appuser.csr \
  -CA server.crt -CAkey server.key -CAcreateserial -out appuser.crt

# Set permissions
chown postgres:postgres *.crt *.key
chmod 600 *.key
chmod 644 *.crt

2. Configure PostgreSQL SSL Settings

Add to postgresql.conf:

# SSL Configuration
ssl = on
ssl_cert_file = '/var/lib/postgresql/17/main/ssl/server.crt'
ssl_key_file = '/var/lib/postgresql/17/main/ssl/server.key'
ssl_ca_file = '/var/lib/postgresql/17/main/ssl/server.crt'
ssl_prefer_server_ciphers = on
ssl_min_protocol_version = 'TLSv1.2'

3. Configure Authentication Rules

Update pg_hba.conf:

# SSL connections require client certificate authentication
hostssl all             all             0.0.0.0/0               cert clientcert=verify-full
hostssl all             all             ::/0                    cert clientcert=verify-full

# Non-SSL connections require password authentication
hostnossl all           all             0.0.0.0/0               scram-sha-256
hostnossl all           all             ::/0                    scram-sha-256

4. Create Database User

The database user must match the certificate's Common Name(CN):

CREATE USER appuser WITH LOGIN;
GRANT CONNECT ON DATABASE mydb TO appuser;
-- Grant additional permissions as needed

How to Connect from Client

Critical Connection Parameters

Parameter	Self-Managed Value	RDS Value	Purpose
`sslmode`	`require` or `verify-full`	`require` or `verify-full`	Force SSL connection
`sslcert`	`/path/to/client.crt`	N/A (not supported)	Client certificate file
`sslkey`	`/path/to/client.key`	N/A (not supported)	Client private key file
`sslrootcert`	`/path/to/server.crt`	`./global-bundle.pem`	Server certificate for verification
`host`	`localhost` or custom	`instance.region.rds.amazonaws.com`	Database server endpoint
`password`	Not required with client cert	Required (or IAM token)	Authentication credential

Connection Examples

Self-Managed PostgreSQL with Client Certificates

# psql - No password required
psql "host=localhost port=5432 dbname=mydb user=appuser \
      sslmode=require \
      sslcert=./certs/appuser.crt \
      sslkey=./certs/appuser.key \
      sslrootcert=./certs/server.crt"

// .NET - Client certificate authentication
var connectionString = "Host=localhost;Port=5432;Database=mydb;Username=appuser;" +
                      "SSL Mode=Require;" +
                      "Client Certificate=./certs/appuser.crt;" +
                      "Client Key=./certs/appuser.key;" +
                      "Root Certificate=./certs/server.crt;";

# Python - Client certificate authentication
import psycopg2
conn = psycopg2.connect(
    host="localhost",
    port=5432,
    database="mydb",
    user="appuser",
    sslmode="require",
    sslcert="./certs/appuser.crt",
    sslkey="./certs/appuser.key",
    sslrootcert="./certs/server.crt"
)

// Java JDBC - Client certificate authentication
String url = "jdbc:postgresql://localhost:5432/mydb" +
            "?ssl=true" +
            "&sslmode=require" +
            "&sslcert=./certs/appuser.crt" +
            "&sslkey=./certs/appuser.key" +
            "&sslrootcert=./certs/server.crt" +
            "&user=appuser";

Connection conn = DriverManager.getConnection(url);

AWS RDS PostgreSQL with SSL (Password Required)

# psql - Password required, verify server certificate
psql "host=mydb.cluster-xyz.us-east-1.rds.amazonaws.com \
      port=5432 dbname=postgres user=myuser password=mypassword \
      sslmode=verify-full sslrootcert=./global-bundle.pem"

// .NET - RDS with SSL verification
var connectionString = "Host=mydb.cluster-xyz.us-east-1.rds.amazonaws.com;" +
                      "Port=5432;Database=postgres;Username=myuser;Password=mypassword;" +
                      "SSL Mode=Require;Root Certificate=./global-bundle.pem;";

# Python - RDS with SSL verification  
import psycopg2
conn = psycopg2.connect(
    host="mydb.cluster-xyz.us-east-1.rds.amazonaws.com",
    port=5432,
    database="postgres",
    user="myuser",
    password="mypassword",
    sslmode="verify-full",
    sslrootcert="./global-bundle.pem"
)

// Java JDBC - RDS with SSL verification
String url = "jdbc:postgresql://mydb.cluster-xyz.us-east-1.rds.amazonaws.com:5432/postgres" +
            "?ssl=true" +
            "&sslmode=verify-full" +
            "&sslrootcert=./global-bundle.pem" +
            "&user=myuser" +
            "&password=mypassword";

Connection conn = DriverManager.getConnection(url);

Server Configuration vs Client Parameters

Server Side (`pg_hba.conf`)

# Authentication method MUST be 'cert'
hostssl all all 0.0.0.0/0 cert clientcert=verify-full

Client Side (Connection String)

# SSL mode MUST be 'require' or 'verify-full'
sslmode=require
# All three certificate files MUST be provided
sslcert=./client.crt
sslkey=./client.key  
sslrootcert=./server.crt

Common Pitfalls

Certificate CN mismatch: Client certificate CN must match database username
Missing clientcert=verify-full: Server won't require client certificates
Wrong SSL mode: sslmode=disable or prefer may skip certificate auth
File permissions: Private keys must be readable only by the client process
Certificate expiration: Monitor and rotate certificates before expiry

Security Benefits

No password transmission: Eliminates password-based attacks
Strong authentication: Based on cryptographic proof of identity
Non-repudiation: Certificate-based audit trails
Mutual authentication: Both client and server verify each other

AWS RDS PostgreSQL Configuration

RDS Limitations for Client Certificates

AWS RDS PostgreSQL has limited support for client certificate authentication compared to self-managed PostgreSQL:

What's NOT Available in RDS

Custom client certificate generation: You cannot upload custom client certificates
Full cert authentication method: RDS doesn't support the cert method in pg_hba.conf
Direct certificate file access: No access to /var/lib/postgresql/ directory

What IS Available in RDS

1. RDS SSL Server Certificates & Configuration

# Download RDS CA certificate bundle
wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

# Enable SSL in RDS parameter group
aws rds modify-db-parameter-group \
    --db-parameter-group-name mygroup \
    --parameters "ParameterName=ssl,ParameterValue=1,ApplyMethod=immediate"

2. Alternative: IAM Database Authentication

For passwordless authentication in RDS, use IAM database authentication:

-- Create IAM-enabled database user
CREATE USER iamuser;
GRANT rds_iam TO iamuser;
GRANT CONNECT ON DATABASE postgres TO iamuser;

# Connect using IAM token (Python example)
import boto3
import psycopg2

rds_client = boto3.client('rds', region_name='us-east-1')
token = rds_client.generate_db_auth_token(
    DBHostname='mydb.xyz.us-east-1.rds.amazonaws.com',
    Port=5432,
    DBUsername='iamuser'
)

conn = psycopg2.connect(
    host="mydb.xyz.us-east-1.rds.amazonaws.com",
    port=5432,
    database="postgres",
    user="iamuser",
    password=token,  # IAM token as password
    sslmode="require",
    sslrootcert="./global-bundle.pem"
)

3. Hybrid Approach: ALB + Client Certificates + RDS

For applications requiring client certificate authentication with RDS:

[Client with Client Cert] → [ALB with SSL termination] → [App Server] → [RDS with IAM Auth]

ALB handles client certificate verification
Application validates certificates in code
RDS connection uses IAM authentication

Summary & Verification

Self-Managed vs AWS RDS Comparison

Feature	Self-Managed	AWS RDS
Client Certificate Auth	✅ Full support	❌ Not supported
Server SSL Certificates	✅ Custom certificates	✅ AWS-managed certificates
Passwordless Authentication	✅ Client certificates	✅ IAM authentication
Certificate Management	❌ Manual rotation	✅ Automatic rotation
SSL Configuration Control	✅ Full control	❌ Limited parameters

Verification Commands

-- Verify SSL is enabled
SHOW ssl;

-- Check active SSL connections with certificate details
SELECT datname, usename, client_addr, ssl, ssl_version, ssl_cipher,
       client_dn, issuer_dn
FROM pg_stat_ssl 
JOIN pg_stat_activity ON pg_stat_ssl.pid = pg_stat_activity.pid 
WHERE ssl = true;
-- Example output
pid|ssl |version|cipher                |bits|client_dn                                                      |client_serial                                   |issuer_dn                                                   |datid|datname       |pid|leader_pid|usesysid|usename       |application_name|client_addr|client_hostname|client_port|backend_start                |xact_start|query_start                  |state_change                 |wait_event_type|wait_event|state|backend_xid|backend_xmin|query_id           |query                                                                                                                        |backend_type  |
---+----+-------+----------------------+----+---------------------------------------------------------------+------------------------------------------------+------------------------------------------------------------+-----+--------------+---+----------+--------+--------------+----------------+-----------+---------------+-----------+-----------------------------+----------+-----------------------------+-----------------------------+---------------+----------+-----+-----------+------------+-------------------+-----------------------------------------------------------------------------------------------------------------------------+--------------+
754|true|TLSv1.3|TLS_AES_256_GCM_SHA384| 256|/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=redgatemonit|608692775915102986323080169170088721995325147053|/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=localhost|16384|redgatemonitor|754|          |   16385|redgatemonitor|psql            |10.88.0.1  |               |      59168|2025-07-30 09:43:20.240 +0100|          |2025-07-30 09:43:58.869 +0100|2025-07-30 09:43:58.870 +0100|Client         |ClientRead|idle |           |            |8885888923103386772|SELECT datname, usename, client_addr, ssl,* FROM pg_stat_ssl JOIN pg_stat_activity ON pg_stat_ssl.pid = pg_stat_activity.pid;|client backend|

Key Takeaways

Self-managed PostgreSQL: Use client certificates for true passwordless authentication
AWS RDS: Combine IAM authentication with SSL encryption for secure connections
Hybrid approach: Use ALB for client certificate verification with RDS backend
Security: Both approaches provide enterprise-grade security with different trade-offs

Client certificate authentication in PostgreSQL provides enterprise-grade security without the overhead of password management, making it ideal for automated systems and high-security environments.

References

https://www.postgresql.org/docs/current/ssl-tcp.html

What is SQL? A Beginner's Complete Guide

Thu, 10 Jul 2025 10:56:55 +0000

Follow Me for More Content!

Before we dive into SQL, let's connect! Follow me on these platforms for more programming tutorials and tech insights:

GitHub: Abdelhakim-Baalla - Check out my projects and code
LinkedIn: abdelhakimbaalla - Professional updates and networking
Twitter (X): @Abdelhakim99891 - Quick tips and tech discussions
Portfolio: abdelhakim-baalla

Introduction

Have you ever wondered how websites store and retrieve information about users, products, or orders? Or how apps like Instagram know which photos to show you? The answer lies in databases, and the key to communicating with these databases is SQL.

If you're completely new to programming or databases, don't worry! SQL is actually one of the most beginner-friendly programming languages you can learn. In this comprehensive guide, we'll explore what SQL is, why it's so important, and how you can start using it today.

By the end of this post, you'll have a solid understanding of SQL fundamentals and be ready to start your database journey. Let's dive in!

What is SQL?

SQL stands for Structured Query Language (pronounced "S-Q-L" or "sequel"). Think of it as a special language that allows you to communicate with databases. Just like you use English to talk to people, you use SQL to talk to databases.

Breaking Down the Name

Structured: SQL follows specific rules and formats
Query: You ask questions (queries) to get information
Language: It's a programming language with its own syntax

Simple Analogy

Imagine a database as a massive digital filing cabinet with thousands of folders and documents. SQL is like having a super-efficient assistant who can:

Find any document you need in seconds
Organize documents in any order you want
Add new documents to the right folders
Update information in existing documents
Remove documents you no longer need

Why is SQL Important?

1. Universal Language for Databases

SQL is the standard language for working with databases. Whether you're using MySQL, PostgreSQL, SQL Server, or Oracle, the basic SQL concepts remain the same.

2. High Demand in the Job Market

According to recent surveys, SQL consistently ranks among the top 5 most in-demand programming skills. Jobs requiring SQL skills often offer competitive salaries.

3. Foundation for Data Analysis

If you're interested in data science, business analytics, or any field involving data, SQL is your starting point.

4. Easy to Learn

Unlike complex programming languages, SQL uses English-like words and phrases, making it incredibly beginner-friendly.

How Databases Work (The Basics)

Before we dive deeper into SQL, let's understand what databases are and how they're organized.

What is a Database?

A database is a structured collection of information stored electronically. Think of it like a digital spreadsheet, but much more powerful and organized.

Database Structure

Databases are organized in a hierarchy:

Database
├── Tables (like spreadsheet tabs)
│   ├── Rows (individual records)
│   └── Columns (categories of information)

Real-World Example

Let's say you're building an online bookstore. Your database might have these tables:

Books Table: Contains information about each book
Customers Table: Stores customer information
Orders Table: Tracks customer purchases
Authors Table: Information about book authors

Basic SQL Operations (CRUD)

SQL operations are often referred to as CRUD operations:

Create - Add new data
Read - Retrieve existing data
Update - Modify existing data
Delete - Remove data

1. Creating Data (INSERT)

Adding new information to a database:

-- Adding a new book to our bookstore
INSERT INTO books (title, author, price, genre)
VALUES ('The Great Gatsby', 'F. Scott Fitzgerald', 12.99, 'Fiction');

2. Reading Data (SELECT)

Retrieving information from a database:

-- Get all books
SELECT * FROM books;

-- Get only book titles and prices
SELECT title, price FROM books;

-- Get books under $15
SELECT * FROM books WHERE price < 15.00;

3. Updating Data (UPDATE)

Modifying existing information:

-- Update the price of a specific book
UPDATE books 
SET price = 14.99 
WHERE title = 'The Great Gatsby';

4. Deleting Data (DELETE)

Removing information:

-- Remove a book from the database
DELETE FROM books 
WHERE title = 'The Great Gatsby';

Key SQL Concepts Every Beginner Should Know

1. Tables

Tables are the building blocks of databases. Each table represents a specific type of entity (like books, customers, or orders).

2. Primary Keys

A unique identifier for each row in a table. Like a social security number for database records.

3. Relationships

Tables can be connected to each other. For example, an order table might reference a customer table.

4. Indexes

Special structures that make searching faster. Like the index in a book that helps you find topics quickly.

Getting Started with SQL

Step 1: Choose a Database System

For beginners, I recommend starting with:

SQLite: Simple, file-based database perfect for learning
MySQL: Popular, free, and widely used
PostgreSQL: Powerful and feature-rich

Step 2: Install and Set Up

Most database systems offer free community editions. You can also use online SQL playgrounds like:

DB Fiddle
SQLiteOnline
W3Schools SQL Tryit Editor

Step 3: Practice with Sample Data

Start with simple examples:

-- Create a simple table
CREATE TABLE students (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    age INTEGER,
    grade TEXT
);

-- Add some data
INSERT INTO students (name, age, grade) VALUES
('Alice Johnson', 20, 'A'),
('Bob Smith', 19, 'B'),
('Carol Davis', 21, 'A');

-- Query the data
SELECT name, grade FROM students WHERE grade = 'A';

Common SQL Commands Reference

Data Retrieval

-- Basic selection
SELECT column1, column2 FROM table_name;

-- With conditions
SELECT * FROM table_name WHERE condition;

-- Sorting results
SELECT * FROM table_name ORDER BY column_name;

-- Limiting results
SELECT * FROM table_name LIMIT 10;

Data Modification

-- Insert new record
INSERT INTO table_name (col1, col2) VALUES (value1, value2);

-- Update existing record
UPDATE table_name SET column1 = value1 WHERE condition;

-- Delete record
DELETE FROM table_name WHERE condition;

Real-World Applications of SQL

1. E-commerce Websites

Track product inventory
Process customer orders
Manage user accounts
Generate sales reports

Store user profiles
Manage posts and comments
Track likes and shares
Analyze user behavior

3. Banking Systems

Process transactions
Maintain account balances
Generate financial reports
Ensure data security

4. Healthcare Systems

Store patient records
Track medical appointments
Manage prescription data
Generate health reports

Best Practices for SQL Beginners

1. Use Meaningful Names

-- Good
SELECT customer_name, order_date FROM orders;

-- Avoid
SELECT c, d FROM o;

2. Always Use WHERE Clauses Carefully

-- Always test your WHERE conditions
SELECT * FROM customers WHERE customer_id = 123;

3. Start Simple

Begin with basic queries before moving to complex joins and subqueries.

4. Practice Regularly

Like any language, SQL improves with consistent practice.

Common Mistakes to Avoid

1. Forgetting WHERE Clauses

-- This deletes ALL records!
DELETE FROM customers;

-- This deletes specific records
DELETE FROM customers WHERE customer_id = 123;

2. Not Backing Up Data

Always backup your database before making major changes.

3. **Using SELECT* in Production**

Instead of selecting all columns, specify only the columns you need.

4. Ignoring Case Sensitivity

Some databases are case-sensitive. Be consistent with your naming.

Next Steps in Your SQL Journey

1. Advanced Topics to Explore

Joins (connecting multiple tables)
Indexes (improving query performance)
Stored procedures (reusable code blocks)
Triggers (automatic actions)

2. Practice Resources

HackerRank SQL challenges
LeetCode database problems
SQLBolt interactive tutorial
Codecademy SQL course

3. Build Projects

Create a personal expense tracker
Build a simple library management system
Design a blog database structure

Database design principles
Data modeling
Business intelligence tools
Cloud database services (AWS RDS, Google Cloud SQL)

Tools and Resources

Free Learning Resources

W3Schools SQL Tutorial: Comprehensive and beginner-friendly
SQLBolt: Interactive SQL lessons
freeCodeCamp: Free SQL course on YouTube
Khan Academy: Database basics

Practice Platforms

HackerRank: SQL challenges and competitions
LeetCode: Database problems
SQLZoo: Interactive SQL tutorials
DB Fiddle: Online SQL playground

Books for Deeper Learning

"Learning SQL" by Alan Beaulieu
"SQL in 10 Minutes, Sams Teach Yourself" by Ben Forta
"Head First SQL" by Lynn Beighley

Conclusion

SQL is your gateway to the world of databases and data management. What makes SQL special is its simplicity and power – you can start writing useful queries within hours of learning the basics.

Remember these key points:

SQL is the universal language for databases
It uses English-like syntax that's easy to learn
The four basic operations (CRUD) form the foundation
Practice with real data to build confidence
Start simple and gradually tackle more complex concepts

The journey of learning SQL is incredibly rewarding. Whether you're interested in web development, data analysis, or just want to add a valuable skill to your toolkit, SQL is an excellent investment in your future.

Don't let the technical aspects intimidate you – every SQL expert started exactly where you are now. The key is to start practicing with simple examples and gradually build your skills.

Ready to start your SQL journey? Pick one of the online SQL playgrounds mentioned above and try creating your first table today. You'll be amazed at how quickly you progress!

Stay Connected for More Learning Content!

If you found this guide helpful, I'd love to have you join my community of learners! Follow me on these platforms for more programming tutorials, career advice, and tech insights:

Don't Miss These!

GitHub: Abdelhakim-Baalla - Browse my code and projects
LinkedIn: abdelhakimbaalla - Professional insights and networking
Twitter (X): @Abdelhakim99891 - Daily tech tips and discussions
Instagram: @abdelhakim.baalla - Behind-the-scenes content
Dev.to: abdelhakim_baalla - In-depth tutorials
Medium: @abdelhakimbaalla50 - Detailed articles
Reddit: Abdelhakim-Baalla430 - Community discussions
Portfolio: abdelhakim-baalla.vercel.app - My work showcase
Substack: @abdelhakimbaalla - Newsletter content
Hashnode: abdelhakimbaalla.hashnode.dev - Technical blog
YouTube: @Dinocrafting - Video tutorials
Stack Overflow: Profile - Q&A and help

Your support means the world to me! Follow for more beginner-friendly programming content, project tutorials, and career guidance.

PostgreSQL 18 Beta 1 Summary

Sun, 29 Jun 2025 18:56:59 +0000

PostgreSQL 18 Beta 1 - New Features and Breaking Changes

Source: GitHub - PostgreSQL 18 Beta 1 New Features

Overview

PostgreSQL 18 introduces over 200 new features and changes. This document summarizes all significant new features, improvements, and breaking changes introduced in the Beta 1 release. Meaning these features can disappear too, this is just a beta.

1. Highlights of Major New Features

1.1 Adaptation to Large Scale Environments

Virtual Columns: Support for virtual (non-stored) columns in tables.
ONLY Clause for VACUUM/ANALYZE: Now supports partitioned tables to avoid recursive partition scans.

1.2 Reliability Enhancements

pg_dumpall Output Format: Supports formats other than plain text.
pg_verifybackup Tar Format: Adds support for tar format verification.

1.3 Maintainability Improvements

Statistics Migration: pg_dump and pg_dumpall can now include optimizer statistics.
Functions for Stats Management: Added functions to set or clear optimizer stats per table/column.
- pg_restore_relation_stats, pg_clear_relation_stats, pg_restore_attribute_stats, pg_clear_attribute_stats
- Commits: d32d146

1.4 Programming Improvements

Access to OLD and NEW in RETURNING clause.
New Function: UUID v7 support.

1.5 Preparations for Future Features

Asynchronous I/O Infrastructure: Introduced with io_method and io_workers.
- Commits: da72269
NUMA Support: Basic infrastructure and views (pg_shmem_allocations_numa).
- Commit: 8cc139b

2. Breaking Changes & Incompatibilities

2.1 Platform & Tool Support

Dropped support for: PA-RISC, LLVM < 14, OpenSSL < 1.1.1, Python < 3.6.8.
Added support: Tcl 9.

2.2 Passwords

MD5 passwords: Now deprecated; warning shown by default.
- Commit: db6a4a9

2.3 Configuration

Removed --disable-spin-locks and --disable-atomics.
- Commits: e256266
initdb now enables data checksums by default.
- Commit: 04bec89

2.4 SQL & DDL Statements

COPY FREEZE: Disallowed on foreign tables.
- Commit: 401a695
EXPLAIN: BUFFERS enabled by default.
- Commit: c2a4078
CREATE SUBSCRIPTION: Default streaming = parallel.
- Commit: 1bf1140

3. Architecture Changes

3.1 Catalog Changes

New Views: pg_aios, pg_shmem_allocations_numa
Added Columns: Many views now include additional metadata, such as constraint enforcement flags, VACUUM/ANALYZE timing, and I/O stats.

3.2 Logical Replication

Logs detailed conflict types like insert_exists, update_differ, delete_missing.
- Commit: 9758174
Propagation of generated columns via publish_generated_columns.
- Commit: 7054186

3.3 Storage I/O Enhancements

Streaming I/O: Broadened scope to VACUUM, CREATE DATABASE, GiST/SP-GiST, etc.
Async I/O: Controlled with io_method, can use io_uring.

4. SQL Enhancements

4.1 ALTER Enhancements

ALTER CONSTRAINT SET INHERIT/NO INHERIT
ADD CONSTRAINT ... NOT VALID
ALTER TABLE ONLY DROP CONSTRAINT

4.2 ANALYZE Improvements

ONLY Clause: Skip partitions.
Verbose Stats: Includes I/O, WAL, and delay stats.

5. Utilities & Tooling

pg_combinebackup, pg_createsubscriber: New utilities.
pg_stat_io: Combines WAL and I/O metrics.
psql \conninfo: More detailed output.

6. Contrib Modules

Highlights:

amcheck: Enhanced with streaming I/O.
Enhancements to: pgcrypto, pg_stat_statements, postgres_fdw.

Summary

PostgreSQL 18 Beta 1 introduces significant architectural upgrades and functional improvements while laying the groundwork for upcoming innovations like async I/O and NUMA-aware processing. Compatibility testing is essential for a smooth transition.

Crystal DBA brings AI-powered PostgreSQL expertise to dev teams, bridging the database optimization skills gap.

Sat, 10 May 2025 23:55:20 +0000

Crystal DBA: AI-Powered PostgreSQL Optimization for Development Teams

In the ever-evolving database landscape, PostgreSQL has become a powerhouse for application development, but mastering its nuances presents significant challenges. During the 60th IT Press Tour, Crystal DBA unveiled their innovative approach to democratizing PostgreSQL expertise through AI, offering developers and engineers a way to optimize database performance without specialized knowledge.

The PostgreSQL Expertise Gap

PostgreSQL's popularity continues to surge across development teams, but running it efficiently requires specialized skills that are increasingly hard to find. This expertise gap creates a critical bottleneck for engineering teams.

As Johann Schleier-Smith, CEO and founder of Crystal DBA explained, "PostgreSQL has exploded in popularity, but it takes a fairly specialized skill set to run it really well. Most companies don't have access to that level of expertise."

Schleier-Smith brings credible experience to this problem space. As former CTO of if(we), he scaled social platforms to 300 million members while navigating the complexities of various database technologies, including PostgreSQL.

Crystal DBA: Your AI Database Teammate

Crystal DBA positions itself as neither just a chatbot nor a fully autonomous agent, but rather an "AI teammate" that works alongside engineers and developers. This teammate approach provides specialized PostgreSQL knowledge on demand while integrating with existing workflows through Slack, Teams, web interfaces, or terminal access.

"The teammate metaphor works best for us," explained Schleier-Smith. "It sits in your Slack or chat, understands everything that's going on, and helps you get the job done. But it's not here to take your job—it's here to help you do your job better."

Technical Implementation

The platform combines three technological approaches:

Generative AI: Powers the conversational interface and reasoning capabilities
Predictive AI: Analyzes workload patterns and predicts optimization opportunities
Classical Optimization: Implements proven database tuning algorithms

This tech stack enables several key capabilities that directly benefit development teams:

Query Analysis and Optimization

During the demo, Crystal DBA demonstrated how it can identify performance bottlenecks in SQL queries by analyzing execution plans and suggesting improvements:

-- From a simple question like:
-- "What are the three most time-consuming queries running on this system?"

-- Crystal DBA can run diagnostic commands like:
SELECT query, calls, total_time, mean_time
FROM pg_stat_statements
ORDER BY total_time DESC
LIMIT 3;

-- Then provide optimizations including:
-- - Index recommendations
-- - Query structure improvements
-- - Configuration parameter adjustments

Configuration Management

PostgreSQL has over 300 configuration parameters across categories like:

Cache and Buffer Management (37 parameters)
Write-Ahead Log (38 parameters)
Query Tuning (48 parameters)
Vacuum (13 parameters)

Crystal DBA can recommend optimal settings based on workload characteristics, helping engineers avoid the complexity of manual tuning while aligning configuration with application requirements.

Automated Maintenance Recommendations

The platform can analyze database behavior to suggest maintenance routines, particularly around vacuum operations—PostgreSQL's method for reclaiming storage and maintaining performance:

"The autovacuum settings appear to be causing inefficient cleanup cycles. Consider adjusting autovacuum_vacuum_threshold from 50 to 200 and autovacuum_vacuum_scale_factor from 0.2 to 0.1 to better match your workload patterns."

Security and Enterprise Readiness

Crystal DBA addresses security concerns through several approaches:

Read-only operation: The system only executes read-only diagnostic commands by default
Deployment flexibility: Options include cloud-based, on-premises with LLaMa models, or hybrid approaches
PII protection: Filters prevent sensitive data from being processed by the AI system

Neil Thombre, VP of Engineering at Crystal DBA and former database development lead at Amazon, emphasized: "All the predicates are obfuscated as $1, $2—just column names. One could argue column names are also PII but that's what you're asking LLM to tune. You're never going to show values."

Pricing and Availability

Crystal DBA offers a free tier for individuals that provides real-time analysis without historical data collection. A team tier at $200/month per production server includes continuous monitoring, historical analysis, and Slack integration. Enterprise pricing is customized based on deployment requirements.

Use Cases for Development Teams

The solution addresses several pain points for developers and engineering teams:

Scaling challenges: Ensuring database performance as application demand grows
Cost optimization: Identifying opportunities to reduce infrastructure expenses
Time efficiency: Reducing time spent on database troubleshooting
Version upgrades: Simplifying the complex process of PostgreSQL version migration

As one attendee commented, "It's like no-code for databases. I can tell your agent to set up a database for an ERP system."

Market Position and Differentiation

While solutions like Datadog, SolarWinds, and pganalyze offer PostgreSQL monitoring, Crystal DBA differentiates itself through its AI-first approach focused on making database expertise accessible to generalist engineers rather than database specialists.

The seed-funded startup (reportedly $7 million from Amplify Partners) currently focuses exclusively on PostgreSQL but acknowledges potential expansion to MySQL, Oracle, and SQL Server in the future.

The Future of Database Management

Crystal DBA represents an emerging trend in database management—using AI to augment human expertise rather than replace it. By embedding specialized knowledge directly into engineering workflows, it addresses a critical skills gap while allowing development teams to focus on building applications rather than tuning databases.

As PostgreSQL continues its rise as a leading database technology, tools like Crystal DBA may become essential components in the modern developer's toolkit.

PostgreSQL Change data capture (CDC) + golang sample code.

Thu, 03 Apr 2025 14:53:53 +0000

PostgreSQL Change data capture (CDC) + golang sample code.

The architecture of modern web applications consists of several software components such as dashboards, analytics, databases, data lakes, caches, search, etc.

The database is usually the core part of any application. Real-time data updates keep disparate data systems in continuous sync and respond quickly to new information. So how to keep your application ecosystem in sync? How do these other components get information about changes in the database? Change Data Capture or CDC refers to any solution that identifies new or changed data.

This post is about PostgreSQL CDC and the ways to achieve this.

Change data capture (CDC) is an approach to data integration to detect, capture, and deliver the changes made to database data sources.

In General, CDC-based data integration consists of the following steps:

Capture change data in a source database.
Convert the changed data to a format that your consumers can accept.
Publish the data to consumers or target database.

PostgreSQL offers two built-in ways to make CDC possible:

From transaction logs, PostgreSQL WALs, aka Write Ahead Logs.
Database Triggers.

Let's briefly discuss the pros and cons of using Transaction logs (WALs) and Triggers to capture data changes.

Triggers.

Trigger-based methods involve creating audit triggers on the database to capture all the events related to INSERT, UPDATE and DELETE methods.

Triggers can be attached to tables (partitioned or not) or views.

Triggers can also fire for TRUNCATE statements. If a trigger event occurs, the trigger's function is called at the appropriate time to handle the event.

The most important advantage of this method is that all of this can be done at the SQL level, unlike transaction logs.
However, the use of triggers has a significant impact on the performance of the source database, because these triggers need to be run on the application database when changes are made to the data.

Transaction Logs

On the other hand, for modern DBMS, transaction logs (WAL for PostgreSQL) are commonly used for transaction logging , replication, and REDO.

In PostgreSQL, all transactions like INSERT, UPDATE, DELETE, and DDLs are written to the WAL before the client sends the transaction result.

The advantage of this approach is that it does not affect the performance of the database in any way.
It also requires no modification to DB tables or the application. There is no need to create additional tables in the source database.
Log-based CDC is generally considered the superior approach to change data capture applicable to all possible scenarios, including systems with extremely high transaction volumes.

If you want row-by-row streaming of Postgres data changes as they happen, you'll need Logical Decoding or Postgres logical replication feature.

Using Postgres Logical Decoding.

Logical decoding is the official name of PostgreSQL's log-based CDC (Logical Replication).

Logical decoding uses the content of the PostgreSQL Write-Ahead Log to store all of the activities that occur in the database. Write Ahead Log is an internal log that describes database changes on a storage level.

The first step in using logical decoding is to set the following parameters in the Postgres configuration postgresql.conf.

wal_level = logical
max_replication_slots = 5
max_wal_senders = 10

Setting wal_level to logical allows the WAL to record information needed for logical decoding.
Ensure that your max_replication_slots value is equal to or higher than the number of PostgreSQL connectors that use WAL plus the number of other replication slots your database uses.
Ensure that the max_wal_senders parameter, which specifies the maximum number of concurrent connections to the WAL, is at least twice the number of logical replication slots. For example, if your database uses 5 replication slots in total, the max_wal_senders value must be 10 or greater.

Restart your Postgres server to apply the changes.

The second step is to set up logical replication using the output plugin test_decoding

Create a logical replication slot for the database you want to sync by running the following command.

SELECT pg_create_logical_replication_slot('replication_slot', 'test_decoding');

Note: Each replication slot has a name, which can contain lower-case letters, numbers, and the underscore character.

To verify the slot is created successfully run the following command.

SELECT slot_name, plugin, slot_type, database, active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots;

In the next step create a publication for all your tables or only specific ones. If you specify tables, you add or remove tables from the publication later.

CREATE PUBLICATION pub FOR ALL TABLES;

CREATE PUBLICATION pub FOR TABLE table1, table2, table3;

Optionally you can choose which operations to include in the publication. For example, the following publication includes only INSERT and UPDATE operations for table1.

CREATE PUBLICATION insert_update_only_pub FOR TABLE table1 WITH (publish = 'INSERT, UPDATE');

Verify that your chosen tables are in the publication.

psql-stream=# SELECT * FROM pg_publication_tables WHERE pubname='pub';
Output
pubname | schemaname | tablename
---------+------------+-----------
pub | public | table1
pub | public | table2
pub | public | table3
(3 rows)

Since then, our publication pub tracks all tables' changes in the psql-stream database.

Let's create an abstract table t and fill it with some records.

create table t (id int, name text);
INSERT INTO t(id) SELECT g.id FROM generate_series(1, 10) as g(id);

As a result we have 10 records in the table t.

psql-stream=# SELECT count(*) FROM t;
count
-------
10
(1 row)

Finally, it is time to check if our Logical Replication works.

Run the following command in the PostgreSQL console to see Postgres WAL entries.

SELECT * FROM pg_logical_slot_get_changes('replication_slot', NULL, NULL);

As a result, you get something like:

    lsn | xid | data                          
-----------+------+--------------------------------------------------------
 0/19EA2C0 | 1045 | BEGIN 1045
 0/19EA2C0 | 1045 | table public.t: INSERT: id[integer]:1 name[text]:null
 0/19EA300 | 1045 | table public.t: INSERT: id[integer]:2 name[text]:null
 0/19EA340 | 1045 | table public.t: INSERT: id[integer]:3 name[text]:null
 0/19EA380 | 1045 | table public.t: INSERT: id[integer]:4 name[text]:null
 0/19EA3C0 | 1045 | table public.t: INSERT: id[integer]:5 name[text]:null
 0/19EA400 | 1045 | table public.t: INSERT: id[integer]:6 name[text]:null
 0/19EA440 | 1045 | table public.t: INSERT: id[integer]:7 name[text]:null
 0/19EA480 | 1045 | table public.t: INSERT: id[integer]:8 name[text]:null
 0/19EA4C0 | 1045 | table public.t: INSERT: id[integer]:9 name[text]:null
 0/19EA500 | 1045 | table public.t: INSERT: id[integer]:10 name[text]:null
 0/19EA5B0 | 1045 | COMMIT 1045
(13 rows)

pg_logical_slot_peek_changes is another PostgreSQL command to peek changes from WAL entries without consuming them. So calling pg_logical_slot_peek_changes multiple times returns the same result each time.

On the other hand pg_logical_slot_get_changes only returns results the first time. The following calls of pg_logical_slot_get_changes return empty result sets. This means when get command is executed, the results are served and deleted, which greatly enhancing our ability to write the logic for using these events to create a replica of the table.

Remember to destroy a slot you no longer need to stop it consuming

SELECT pg_drop_replication_slot('replication_slot');

Output plugins.

We've already talked about test_decoding output plugin available on Postgres 9.4+. Though created as an example of an output plugin, it is still useful if your consumer supports it.

Along with the test_decoding plugin, another pgoutput plugin is shipped with PostgreSQL natively. pgoutput is available since Postgres 10. Some consumers support it for decoding (e.g. Debezium).

Run the following command to create the plugin based on pgoutput as in step 2 above.

SELECT * FROM pg_create_logical_replication_slot('replication_slot', 'pgoutput');

The following command consumes data changes similar to those described in step 6.

psql-stream=# SELECT * FROM pg_logical_slot_peek_binary_changes('replication_slot', null, null, 'proto_version', '1', 'publication_names', 'pub');
    lsn | xid | data                                           
-----------+------+------------------------------------------------------------------------------------------
 0/19A15F8 | 1038 | \x4200000000019a1d9000027de20a91a0ea0000040e
 0/19A15F8 | 1038 | \x52000080387075626c69630074006400020169640000000017ffffffff006e616d650000000019ffffffff
 0/19A15F8 | 1038 | \x49000080384e0002740000000234306e
 0/19A1890 | 1038 | \x49000080384e0002740000000234316e
 0/19A1910 | 1038 | \x49000080384e0002740000000234326e
 0/19A1990 | 1038 | \x49000080384e0002740000000234336e
 0/19A1A10 | 1038 | \x49000080384e0002740000000234346e
 0/19A1A90 | 1038 | \x49000080384e0002740000000234356e
 0/19A1B10 | 1038 | \x49000080384e0002740000000234366e
 0/19A1B90 | 1038 | \x49000080384e0002740000000234376e
 0/19A1C10 | 1038 | \x49000080384e0002740000000234386e
 0/19A1C90 | 1038 | \x49000080384e0002740000000234396e
 0/19A1DC0 | 1038 | \x430000000000019a1d9000000000019a1dc000027de20a91a0ea
(13 rows)

Here you can notice that the results are returned in binary format. pgoutput plugin produces binary output.

wal2json is another popular output plugin for logical decoding.

Here is a sample output from wal2json plugin

{
      "change":[
         {
            "kind":"insert",
            "schema":"public",
            "table":"t",
            "columnnames":[
               "id",
               "name"
            ],
            "columntypes":[
               "integer",
               "character varying(255)"
            ],
            "columnvalues":[
               1,
               ""
            ]
         }
      ]
   }
   {
      "change":[
         {
            "kind":"update",
            "schema":"public",
            "table":"t",
            "columnnames":[
               "id",
               "name"
            ],
            "columntypes":[
               "integer",
               "character varying(255)"
            ],
            "columnvalues":[
               1,
               "New Value"
            ],
            "oldkeys":{
               "keynames":[
                  "id"
               ],
               "keytypes":[
                  "integer"
               ],
               "keyvalues":[
                  1
               ]
            }
         }
      ]
   }
   {
      "change":[
         {
            "kind":"delete",
            "schema":"public",
            "table":"t",
            "oldkeys":{
               "keynames":[
                  "id"
               ],
               "keytypes":[
                  "integer"
               ],
               "keyvalues":[
                  1
               ]
            }
         }
      ]
   }

Important tips about slots.

Keep the following in mind when working with slots:

Each slot has only one output plugin (you choose which one).
Each slot provides changes from only one database.
One database can have multiple slots.
Each data change is typically is emitted once per slot.
But a slot may re-emit the changes when the Postgres instance is restarted. A consumer must deal with this situation.
An unconsumed slot is a threat to the availability of your Postgres instance. Postgres will save all WAL files for these unconsumed changes. This can lead to storage overflow.

PostgreSQL WAL consumers.

A consumer is any application that can ingest the Postgres logical decoding stream. pg_recvlogical is a PostgreSQL application that can manage slots and consume the stream from them. It is included in the Postgres distribution, so it's probably already installed with PostgreSQL.

Photo by Markus Spiske / Unsplash

Golang sample code.

The following Golang code example shows how to get started creating your own Postgress WAL consumer. It uses PostgreSQL-10.x logical replication to stream database changes (decoded WAL messages) from the source database.

package main

import (
"context"
"fmt"
"os"
"os/signal"
"strings"
"time"

"github.com/jackc/pgconn"
"github.com/jackc/pglogrepl"
"github.com/jackc/pgproto3/v2"
)

const CONN = "postgres://postgres:postgres@localhost/psql-streamer?replication=database"
const SLOT_NAME = "replication_slot"
const OUTPUT_PLUGIN = "pgoutput"
const INSERT_TEMPLATE = "create table t (id int, name text);"

var Event = struct {
Relation string
Columns []string
}{}

func main() {
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
conn, err := pgconn.Connect(ctx, CONN)
if err != nil {
panic(err)
}
defer conn.Close(ctx)

// 1. Create table
if _, err := conn.Exec(ctx, INSERT_TEMPLATE).ReadAll(); err != nil {
fmt.Errorf("failed to create table: %v", err)
}

// 2. ensure publication exists
if _, err := conn.Exec(ctx, "DROP PUBLICATION IF EXISTS pub;").ReadAll(); err != nil {
fmt.Errorf("failed to drop publication: %v", err)
}

if _, err := conn.Exec(ctx, "CREATE PUBLICATION pub FOR ALL TABLES;").ReadAll(); err != nil {
fmt.Errorf("failed to create publication: %v", err)
}

// 3. create temproary replication slot server
if _, err = pglogrepl.CreateReplicationSlot(ctx, conn, SLOT_NAME, OUTPUT_PLUGIN, pglogrepl.CreateReplicationSlotOptions{Temporary: true}); err != nil {
fmt.Errorf("failed to create a replication slot: %v", err)
}

var msgPointer pglogrepl.LSN
pluginArguments := []string{"proto_version '1'", "publication_names 'pub'"}

// 4. establish connection
err = pglogrepl.StartReplication(ctx, conn, SLOT_NAME, msgPointer, pglogrepl.StartReplicationOptions{PluginArgs: pluginArguments})
if err != nil {
fmt.Errorf("failed to establish start replication: %v", err)
}

var pingTime time.Time
for ctx.Err() != context.Canceled {
if time.Now().After(pingTime) {
if err = pglogrepl.SendStandbyStatusUpdate(ctx, conn, pglogrepl.StandbyStatusUpdate{WALWritePosition: msgPointer}); err != nil {
fmt.Errorf("failed to send standby update: %v", err)
}
pingTime = time.Now().Add(10 * time.Second)
//fmt.Println("client: please standby")
}

ctx, cancel := context.WithTimeout(ctx, time.Second*10)
defer cancel()

msg, err := conn.ReceiveMessage(ctx)
if pgconn.Timeout(err) {
continue
}
if err != nil {
fmt.Errorf("something went wrong while listening for message: %v", err)
}

switch msg := msg.(type) {
case *pgproto3.CopyData:
switch msg.Data[0] {
case pglogrepl.PrimaryKeepaliveMessageByteID:
//fmt.Println("server: confirmed standby")

case pglogrepl.XLogDataByteID:
walLog, err := pglogrepl.ParseXLogData(msg.Data[1:])
if err != nil {
fmt.Errorf("failed to parse logical WAL log: %v", err)
}

var msg pglogrepl.Message
if msg, err = pglogrepl.Parse(walLog.WALData); err != nil {
fmt.Errorf("failed to parse logical replication message: %v", err)
}
switch m := msg.(type) {
case *pglogrepl.RelationMessage:
Event.Columns = []string{}
for _, col := range m.Columns {
Event.Columns = append(Event.Columns, col.Name)
}
Event.Relation = m.RelationName
case *pglogrepl.InsertMessage:
var sb strings.Builder
sb.WriteString(fmt.Sprintf("INSERT %s(", Event.Relation))
for i := 0; i < len(Event.Columns); i++ {
sb.WriteString(fmt.Sprintf("%s: %s ", Event.Columns[i], string(m.Tuple.Columns[i].Data)))
}
sb.WriteString(")")
fmt.Println(sb.String())
case *pglogrepl.UpdateMessage:
var sb strings.Builder
sb.WriteString(fmt.Sprintf("UPDATE %s(", Event.Relation))
for i := 0; i < len(Event.Columns); i++ {
sb.WriteString(fmt.Sprintf("%s: %s ", Event.Columns[i], string(m.NewTuple.Columns[i].Data)))
}
sb.WriteString(")")
fmt.Println(sb.String())
case *pglogrepl.DeleteMessage:
var sb strings.Builder
sb.WriteString(fmt.Sprintf("DELETE %s(", Event.Relation))
for i := 0; i < len(Event.Columns); i++ {
sb.WriteString(fmt.Sprintf("%s: %s ", Event.Columns[i], string(m.OldTuple.Columns[i].Data)))
}
sb.WriteString(")")
fmt.Println(sb.String())
case *pglogrepl.TruncateMessage:
fmt.Println("ALL GONE (TRUNCATE)")
}
}
default:
fmt.Printf("received unexpected message: %T", msg)
}
}
}

This code just logs incoming events, but in a production environment you can easily send them to a message queue or a target database.

Conclusion

Logical decoding in PostgreSQL provides an efficient way for other application components to stay up-to-date with data changes in your Postgres database.

Traditionally, the pull notification model has been used, in which each application component queries Postgres at a certain interval. Logical encoding uses the push notification model , where Postgres notifies other parts of the application of every change as soon as it happens.

Data change events can now be sent to consumers in milliseconds without querying the database. With logic decoding, the PostgreSQL database becomes a central part of your modern dynamic real-time application.

DBConvert Streams is a new data replication platform that uses CDC technology to stream data between databases.

Database Interaction using Python: Introduction to SQLite/MySQL/PostgreSQL

Wed, 19 Mar 2025 11:56:30 +0000

The article covers POSTGRESQL, MySQL, and SQLITE3—three major database management systems. They provide unique features and benefits and are essential to software development. Let's explore their features so that you can make better selections that will improve the project. Together, let's explore these essential elements in modern software engineering.

As SQLite is lightweight, it is a great choice for embedded applications and situations where simplicity is essential. On the other hand, MySQL stands out thanks to its strong client-server design, scalability, and widespread industry acceptance. PostgreSQL, on the other hand, is a popular option for enterprise-level applications due to its versatility, dependability, and compliance with ACID short for Atomicity, Consistency, Isolation, and Durability.

SQLite Database

SQLite is a relational database engine that is lightweight, serverless, self-contained, and requires no configuration. Because it works directly with files, it's perfect for low-resource applications. Distribution and deployment are made easier by the fact that its self-contained databases are kept as single files.

Features:

Embedded in applications: SQLite, which is frequently integrated into programmes, allows internal data administration without the need for external servers. Data management is made easier by this, especially for offline functioning.
File-based database: SQLite databases are manageable and portable since they are kept as individual files on the file system.
Appropriate for small to medium-sized applications: Because of its simplicity and lightweight design, SQLite is especially well-suited for small to medium-sized applications, even though it can manage huge datasets as well.

Let's go through an example of defining tables, installing SQLite in Python, and carrying out CRUD (Create, Read, Update, Delete) operations:

conn = sqlite3.connect('example.db')

cursor = conn.cursor()

cursor.execute('''CREATE TABLE IF NOT EXISTS users

                  (id INTEGER PRIMARY KEY, name TEXT, email TEXT)''')

cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ('Alice', '*Emails are not allowed*'))

cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ('Ali', '*Emails are not allowed*'))

print("Records after initial insertion:")

cursor.execute("SELECT DISTINCT * FROM users")

rows = cursor.fetchall()

for row in rows:

    print(row)

cursor.execute("UPDATE users SET email = ? WHERE name = ?", ('*Emails are not allowed*', 'Alice'))

print("\nRecords after updating email of Alice:")

cursor.execute("SELECT DISTINCT * FROM users")

rows = cursor.fetchall()

for row in rows:

    print(row)

cursor.execute("DELETE FROM users WHERE name = ?", ('Alice',))

print("\nRecords after deleting Alice:")

cursor.execute("SELECT DISTINCT * FROM users")

rows = cursor.fetchall()

for row in rows:

    print(row)

Output:

Tip: Make sure the 'DISTINCT' keyword is applied to the right columns in SQL queries to obtain unique value combinations and prevent accidental duplicate results.

MySQL Database

The popular open-source RDBMS MySQL provides strong features and dependability for systems ranging from small-scale initiatives to enterprise-level setups. It offers a wide range of tools and features for effective structured data management.

Features:

Client-server architecture:MySQL has a client-server design, which enables several clients to connect to the database server at once. The management of database resources, scalability, and security are all improved by this architecture.
Performance and scalability: MySQL is designed to be scalable, meaning it can manage big databases and high transaction volumes while maintaining optimal performance. Performance is improved by its effective caching, query optimisation, and indexing methods.
Broad industrial adoption: MySQL is widely used in a number of industries, including telecommunications, e-commerce, technology, and finance. Its widespread community support, dependability, and open-source nature have all added to its appeal.

You can get the MySQL Community Server package from mysql official website and install MySQL by following the given steps. As an alternative, you can use package managers such as Homebrew for macOS or apt-get for Linux.

Connecting to MySQL from Python:

import mysql.connector

conn = mysql.connector.connect(user='user', password='password',

                               host='localhost',

                               database='test')

cursor = conn.cursor()

cursor.execute("SELECT * FROM table_name")

results = cursor.fetchall()

conn.close()

Creating tables and executing queries using Python's MySQL Connector:

cursor = conn.cursor()

cursor.execute('''CREATE TABLE IF NOT EXISTS users

                  (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(255), email VARCHAR(255))''')

cursor.execute("INSERT INTO users (name, email) VALUES (%s, %s)", ('Alice', '*Emails are not allowed*'))

cursor.execute("SELECT * FROM users")

results = cursor.fetchall()

PostgreSQL Database

Strong and open-source, PostgreSQL is an object-relational database system that is well-known for its feature-rich set, dependability, and extensibility. It is a top option for both small-scale applications and large corporate systems because of its comprehensive functions that enable complicated data types, transactions, and data integrity.

Features:

Extensibility: Because of PostgreSQL's flexible architecture, programmers are able to design unique functions, data types, and indexing techniques to meet the needs of particular applications. The smooth integration of third-party extensions and libraries is made possible by this flexibility.
ACID Compliance: PostgreSQL guarantees ACID (
Atomicity: To preserve database integrity, transactions are either fully completed or entirely aborted.
Consistency: Assures that the database is kept intact by writing only legitimate data.
Isolation: Data integrity is ensured by many transactions running simultaneously without interfering.
Durability: Even in the event of a system failure, committed transaction modifications remain intact.) compliance, which in turn provides dependable and consistent database transactions. This functionality is crucial for ensuring consistency across the database and data integrity even in the case of system failures.
Data Integrity and Reliability: PostgreSQL uses built-in restrictions like triggers and rules, foreign keys, and concurrency management to guarantee data integrity and stability.

To install PostgreSQL, you can download and install the PostgreSQL package from the official website.

import psycopg2

conn = psycopg2.connect(

    dbname='your_database_name',

    user='your_username',

    password='your_password',

    host='localhost',

    port='5432'

)

if conn:

    print("Connected to PostgreSQL")

else:

    print("Failed to connect to PostgreSQL")

conn.close()

Creating tables and executing queries using Python's psycopg2 library:

# Create a cursor object

cursor = conn.cursor()

cursor.execute('''CREATE TABLE IF NOT EXISTS users (

                    id SERIAL PRIMARY KEY,

                    name VARCHAR(255),

                    email VARCHAR(255)

                )''')

cursor.execute("INSERT INTO users (name, email) VALUES (%s, %s)", ('Alice', '*Emails are not allowed*'))

cursor.execute("SELECT * FROM users")

results = cursor.fetchall()

for row in results:

    print(row)

conn.commit()

Note: Always securely handle database credentials and avoid hardcoding them directly into your code for better security.

Comparison

Factor	SQLite	MySQL	PostgreSQL
Performance	Good for small to medium -sized applications	Optimized for read-heavy workloads	Optimized for complex queries and large datasets
Scalability	Limited scalability due to file- based architecture	Good scalability with support for clustering and sharding	High scalability with advanced replication and partitioning
Features	Basic features with limited concurrency support	Rich feature set including stored procedures, triggers, and views	Comprehensive feature set including JSONB data type, full-text search, and advanced indexing
Ease of Use	Simple setup and usage, suitable for beginners	Easy to use, especially with graphical tools like MySQL Workbench	Requires more configuration but offers powerful features
Use Case Suitability	Ideal for embedded systems and small applications	Well-suited for web applications and medium- sized databases	Best for large-scale applications with complex data requirements

Advanced Concepts

Advanced ideas like indexing, optimization, and transactions are essential for improving the scalability, performance, and dependability of databases.

Transactions: Transactions ensure the ACID properties of database operations, handling multiple tasks as a single unit for data accuracy and reliability. All database systems provide tools for effective transaction management.
Indexing: Indexes expedite data retrieval, particularly for SELECT queries, by swiftly accessing entries in database tables based on specific field values. Effective indexing reduces the need for full table scans, significantly enhancing query performance.
Optimization: Database optimization strategies improve effectiveness and performance through resource usage optimization, performance tuning, query optimization, and schema design optimization. Each system provides unique features and tools to enhance productivity and address performance issues.

import sqlite3

import mysql.connector

import psycopg2

def sqlite_transactions():

    conn = sqlite3.connect('example.db')

    cursor = conn.cursor()

    try:

        conn.execute("BEGIN")

        cursor.execute("UPDATE users SET balance = balance - 100 WHERE id = 1")

        cursor.execute("UPDATE users SET balance = balance + 100 WHERE id = 2")

        conn.commit()

        print("Transaction committed successfully")

    except sqlite3.Error as e:

        conn.rollback()

        print("Transaction rolled back due to error:", e)

    finally:

        conn.close()

def mysql_transactions():

    conn = mysql.connector.connect(user='user', password='password', database='example')

    cursor = conn.cursor()

    try:

        conn.start_transaction()

        cursor.execute("UPDATE users SET balance = balance - 100 WHERE id = 1")

        cursor.execute("UPDATE users SET balance = balance + 100 WHERE id = 2")

        conn.commit()

        print("Transaction committed successfully")

    except mysql.connector.Error as e:

        conn.rollback()

        print("Transaction rolled back due to error:", e)

    finally:

        conn.close()

def postgres_transactions():

    conn = psycopg2.connect(dbname='example', user='user', password='password', host='localhost')

    cursor = conn.cursor()

    try:

        conn.autocommit = False

        cursor.execute("UPDATE users SET balance = balance - 100 WHERE id = 1")

        cursor.execute("UPDATE users SET balance = balance + 100 WHERE id = 2")

        conn.commit()

        print("Transaction committed successfully")

    except psycopg2.Error as e:

        conn.rollback()

        print("Transaction rolled back due to error:", e)

    finally:

        conn.close()

sqlite_transactions()

mysql_transactions()

postgres_transactions()

SQLite, designed for single-user and small-scale applications, uses local file-based databases and simple transaction management with BEGIN, COMMIT, and ROLLBACK statements.
MySQL, a robust RDBMS, offers scalability and concurrency control with start_transaction(), commit(), and rollback() methods.
PostgreSQL, known for advanced features and extensibility, provides strong support for concurrency, data integrity, and advanced SQL features, managing transactions via autocommit mode and commit()/rollback() methods.

FAQ
Q: What is the difference between SQL and NoSQL databases?
A: SQL (Structured Query Language) databases, also known as relational databases, organize data into tables with predefined schemas and use SQL for querying. NoSQL databases, on the other hand, store data in flexible, non-tabular formats and do not require a fixed schema. They are designed for scalability and handling unstructured data.

Conclusion

Databases are essential to software development because they allow for effective data management. MySQL scales for the web, PostgreSQL meets enterprise needs, and SQLite is ideal for lightweight applications. When choosing a project, it is essential to comprehend their strengths. Using cutting-edge ideas promotes creativity in digital solutions by ensuring data performance and dependability.

Reference

For more reference:
SQLite Official Website:
SQLite Official Website

Coder Legion - Recent posts tagged postgresql

Database Migrations: Can Your Previous Release Still Run?

Separate compatibility from cleanup

Work through the awkward middle

Test the rollback with new data

A compatible change can still block traffic

Make cleanup its own decision

The Reconnect Bug Where Marking a User Online Erased the Data I Needed for Replay

Calling a Service Inside prisma.$transaction Doesn't Make It Atomic — Here's the Bug That Taught Me

From React to PostgreSQL: Understanding the Full Stack

1. What Does "Full Stack" Actually Mean?

2. React: The Frontend Layer

React State

Fetching Backend Data

3. Why React Should Not Connect Directly to PostgreSQL

4. The Backend: Where the Application Thinks

5. Understanding APIs

GET

POST

PUT or PATCH

DELETE

6. PostgreSQL: The Data Layer

7. SQL: How the Backend Talks to PostgreSQL

8. The Complete Request Lifecycle

Step 1: React Collects the Data

Step 2: React Sends the Request

Step 3: Express Receives It

Step 4: Backend Validates the Data

Step 5: Backend Queries PostgreSQL

Step 6: PostgreSQL Processes the Query

Step 7: Backend Sends a Response

Step 8: React Updates the UI

9. Authentication: Connecting Users to Data

10. Database Relationships

11. JOIN Queries

12. Validation Must Happen on the Backend

13. Error Handling

14. Environment Variables

15. ORM vs Raw SQL

16. Database Indexing

17. Transactions

18. Performance Across the Full Stack

19. Pagination

20. Caching

21. Security in a Full-Stack Application

Frontend

Backend

API

Database

SQL

Authentication

Deployment

22. Development Workflow

23. Testing the Full Stack

Frontend Testing

API Testing

Database Testing

Integration Testing

End-to-End Testing

24. Deployment

25. Logging and Monitoring

26. The Most Important Skill: Understanding the Boundaries

27. What Happens When You Click a Button?

28. What Should a Beginner Learn First?

Step 1: HTML and CSS

Step 2: JavaScript

Step 3: React

Step 4: Node.js and Express

Step 5: PostgreSQL

Step 6: Full-Stack Projects

29. The Future of Full-Stack Development

Conclusion

Why I Chose a Simple Capability Token for Waitless — and When I Would Harden It

UUID token instead of a sequential ID

What is a capability URL?

Why I kept the token simple for now

But the token is also a bearer credential

Known limitation

A stronger design: capability URL → session cookie

The trade-off: security vs. convenience