Coder Legion - Recent posts tagged mysql

Flyway Migration: From Legacy Database to CI/CD

Thu, 29 Oct 2026 12:05:30 +0000

Most teams think of Flyway only when they need to manage ongoing database schema changes. But what if your project already has a complete, stable schema and you’re only now introducing version control? Or what if you want to incorporate your schema into a CI/CD pipeline for reliable, reproducible deployments?

In this guide, I’ll walk you through a real-world scenario where we used Flyway to baseline an existing MySQL database and then extended it to support version-controlled schema migrations in CI/CD, without disrupting production.

What is Flyway?

Flyway is a simple but powerful tool for managing database migrations. It applies versioned SQL scripts to your database and tracks them in a special table called flyway_schema_history.

Typical script names:

V1__initial_schema.sql
V2__add_orders_table.sql
V3__rename_column.sql

Flyway ensures these scripts run only once and in the correct order. It also integrates well with CI/CD tools, making it ideal for automated environments.

Use Case: Existing Schema with No Migrations (Yet)

In our project, we had:

A production MySQL database with dozens of tables
No migration history
No current need for structural changes
A desire to version control the database and enable future migrations via CI/CD

Step 1: Baselining an Existing Schema

Dump the schema

mysqldump -u root -p --no-data your_database > V1__baseline_schema.sql

Clean the output: remove auto-increments, user references, and environment-specific SQL.

Create directory structure

mkdir -p db/sql
mv V1__baseline_schema.sql db/sql/

Configure flyway.conf

flyway.url=jdbc:mysql://your-db-host:3306/your_database
flyway.user=db_user
flyway.password=secret_password
flyway.locations=filesystem:db/sql
flyway.baselineOnMigrate=true

Run baseline

flyway baseline

This creates the flyway_schema_history table and marks version 1 as the current state without running any script.

Step 2: Add a Real Migration

After baselining, we added a real change in a new script

-- db/sql/V2__add_status_to_orders.sql
ALTER TABLE orders ADD COLUMN status VARCHAR(20) DEFAULT 'pending';

Then we ran:

flyway migrate

Result: Flyway applied version 2, recorded it in history, and our database was now safely versioned.

⚙️ Step 3: Integrate Flyway into CI/CD

You can integrate Flyway with GitHub Actions, GitLab CI, Bitbucket Pipelines, or Jenkins.

✅ Example GitHub Actions Workflow

name: Flyway Migration
on:
  push:
    branches: [ main ]

jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Set up Java (Flyway needs it)
        uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '17'
      - name: Install Flyway
        run: |
          curl -L https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/10.8.1/flyway-commandline-10.8.1-linux-x64.tar.gz | tar xvz
          sudo ln -s $PWD/flyway-10.8.1/flyway /usr/local/bin/flyway
      - name: Run Flyway Migrations
        env:
          FLYWAY_URL: jdbc:mysql://your-host:3306/your_database
          FLYWAY_USER: ${{ secrets.DB_USER }}
          FLYWAY_PASSWORD: ${{ secrets.DB_PASSWORD }}
        run: |
          flyway -url=$FLYWAY_URL \
                 -user=$FLYWAY_USER \
                 -password=$FLYWAY_PASSWORD \
                 -locations=filesystem:./db/sql \
                 migrate

Benefits of This Approach

✅ You gain auditability for all database changes

✅ Developers can build local environments using the same scripts

✅ CI/CD ensures consistency across all environments

✅ Flyway’s validate command catches missing or changed scripts

✅ Schema management becomes part of your software delivery lifecycle

Potential Challenges

Problem → Fix

Flyway re-applies the baseline script
→ Use baselineOnMigrate=true and never modify a baseline
Passwords in config files
→ Use secrets and environment variables in CI/CD
Manual schema drift
→ Disable direct DB access in prod or run flyway validate periodically
Need rollback support
→ Write manual undo scripts or use point-in-time backups

Alternatives (and Why Flyway Wins)

Liquibase
✅ Pros: Advanced features, rollback support
❌ Cons: More complex, steeper learning curve

Django ORM / Alembic
✅ Pros: Built-in for app devs
❌ Cons: Not cross-language or DB-agnostic

Manual SQL or DB dumps
✅ Pros: Simple
❌ Cons: No versioning, no tracking, no automation

Flyway is a great middle-ground: powerful enough for teams, simple enough for solo devs, and perfect for CI/CD.

Final Thoughts

Flyway isn’t just for teams writing migrations every day. It’s also ideal for bringing an existing database under version control, giving your project a foundation for safe schema evolution.

We started with a baseline, added our first real migration, and then built it into our CI/CD pipeline — all in a day’s work.

If you’re managing an existing DB without schema tracking, now’s the time to fix that. Start with a baseline. Your future self will thank you.

Let’s Connect!

Have you tried Flyway for baseline-only use cases or integrated it into your CI/CD pipeline? I’d love to hear your experience in the comments.

If you found this helpful, follow me here on DEV for more DevOps and database engineering content.

Understanding Database Replication: How to Keep Your Apps Fast and Reliable

Tue, 27 Oct 2026 09:21:52 +0000

What is Database Replication?

Database replication is the automated process of keeping multiple exact copies of a database synchronized across different server instances. When data is added, updated, or removed in the primary database, those changes are copied to one or more replica databases in near real-time. This setup ensures that if one database fails or experiences high traffic, the overall application remains operational, responsive, and secure against data loss.

A Real-World Analogy: The Restaurant Kitchen Whiteboard

Imagine a high-volume restaurant kitchen with a single Head Chef and five Line Cooks. Every time a new order arrives or a special menu item changes, the Head Chef writes it down on a master whiteboard in the center of the kitchen.

If all five Line Cooks had to walk up to that single whiteboard every few seconds to read the latest order, a massive bottleneck would form around the Head Chef. Cooking would grind to a halt because everyone is waiting in line just to read information.

To solve this, the kitchen hires a runner whose sole job is to copy whatever the Head Chef writes on the master whiteboard onto five individual clipboards stationed at each Line Cook's workstation. The Head Chef remains the only person writing new orders (the Primary database), while the Line Cooks read from their dedicated station clipboards (the Read Replicas). Orders flow instantly, cooks stay productive, and the head chef is never overwhelmed.

Why Database Replication Matters in Modern Engineering

In real-world web applications, user activity is overwhelmingly biased toward reading data rather than writing it. For example, millions of people scroll through social media posts or view e-commerce products for every one person who creates a new post or completes a purchase. If every read request hit the exact same database server handling write operations, that single database would quickly exhaust its memory and processing power.

Engineers use database replication to solve several critical challenges:

High Availability and Fault Tolerance: A hardware failure or crash on the primary database server does not bring down the entire website. System administrators can quickly promote a replica to become the new primary writer with minimal downtime.
Performance Scaling: Read traffic is distributed across multiple read-only replica servers. This practice, known as load balancing, prevents any single server from becoming a CPU bottleneck.
Heavy Reporting Offloading: Running complex analytical queries (like generating monthly revenue reports) takes immense computing resources. Performing these queries on a replica ensures transactional write operations for live customers remain fast and unaffected.

Implementing Read/Write Splitting in Node.js

To take advantage of database replication, backend application code must route write commands (INSERT, UPDATE, DELETE) to the primary database, while directing read queries (SELECT) to replica nodes. A common database driver used in Node.js applications is mysql2.

Here is how you can set up dual connections in a Node.js module to execute read/write splitting:

// db.js - Database connection module for Primary and Replica instances
const mysql = require('mysql2/promise');

// Primary connection reserved strictly for write operations
const primaryPool = mysql.createPool({
  host: process.env.DB_PRIMARY_HOST || 'primary.db.example.com',
  user: 'admin',
  password: 'securepassword',
  database: 'store_db',
  connectionLimit: 10
});

// Replica connection pool used for read-only queries
const replicaPool = mysql.createPool({
  host: process.env.DB_REPLICA_HOST || 'replica-1.db.example.com',
  user: 'read_only_user',
  password: 'readpassword',
  database: 'store_db',
  connectionLimit: 20
});

// Function to execute write operations on the primary DB
async function executeWrite(sqlQuery, params) {
  const [result] = await primaryPool.execute(sqlQuery, params);
  return result;
}

// Function to execute read operations on the replica DB
async function executeRead(sqlQuery, params) {
  const [rows] = await replicaPool.execute(sqlQuery, params);
  return rows;
}

module.exports = {
  executeWrite,
  executeRead
};

In this example, calling executeWrite() executes the SQL query on the primary database node, ensuring that structural data modifications land safely on the source authority. Calling executeRead() queries the replica node, taking performance stress off the writer server.

Key Takeaway

Database replication shifts application architecture from a fragile single point of failure into a redundant, horizontally scalable infrastructure. By separating write transactions from high-volume read requests, development teams guarantee that their software stays online and lightning-fast, even during massive user traffic surges.

Enjoyed this breakdown? Support my open-source work on Buy Me a Coffee or explore react-hook-lab!

Originally published on my blog. You can read the alternative breakdown here.

Demystifying ACID Transactions: How to Keep Your Database from Making a Mess

Thu, 01 Oct 2026 10:28:33 +0000

An ACID transaction is a set of four database safety rules—Atomicity, Consistency, Isolation, and Durability—that ensure database operations are executed reliably. It guarantees that even if your application crashes, loses power, or experiences thousands of concurrent requests, your data remains accurate and uncorrupted. Simply put, it ensures your database never ends up in a half-finished, broken state.

The Ticket Booth Analogy

Imagine buying a physical ticket at a busy movie theater box office. You hand the cashier a $15 bill, and they hand you the ticket. This exchange must happen as a single, indivisible unit. If you hand over your cash, but a sudden power outage occurs before the cashier can hand you the ticket, you would be furious if they kept your money. Conversely, the theater would lose out if you grabbed the ticket and walked away without paying.

The transaction is "all-or-nothing": either the cash changes hands and you get the ticket, or the entire deal is canceled, and you keep your money. ACID acts as this strict box-office coordinator for your software's database, ensuring that multi-step operations never stop halfway.

Why It Matters: Real-World Disasters

In daily software engineering, we rely on ACID transactions to prevent catastrophic data errors, especially when building systems like financial ledgers, e-commerce checkouts, or reservation platforms.

Without ACID, imagine a bank transfer system built on Node.js. When a user transfers $100, the system must perform two separate actions: deduct $100 from Account A, and add $100 to Account B. If the database server loses connection exactly halfway through this process, Account A loses money, but Account B never receives it. The $100 vanishes into the digital void.

Similarly, in a high-traffic retail store, if two customers try to buy the very last item in stock at the exact same millisecond, a non-ACID database might let both transactions complete. This leaves you with an oversold item and an angry customer. By grouping these operations inside an ACID transaction, developers ensure that if any step fails, the database automatically rolls back all changes, restoring everything to its original state.

Implementing ACID in Node.js and MySQL

Here is how we can implement a secure money transfer using Node.js and the standard mysql2/promise library. By wrapping our queries in a transaction block, we guarantee data safety.

const mysql = require('mysql2/promise');

async function transferFunds(fromAccountId, toAccountId, amount) {
  const connection = await mysql.createConnection({
    host: 'localhost',
    user: 'root',
    database: 'bank_db'
  });

  try {
    // Start the ACID transaction
    await connection.beginTransaction();

    // Step 1: Deduct from the sender's account
    const [deductResult] = await connection.execute(
      'UPDATE accounts SET balance = balance - ? WHERE id = ? AND balance >= ?',
      [amount, fromAccountId, amount]
    );

    if (deductResult.affectedRows === 0) {
      throw new Error('Insufficient funds or invalid sender account');
    }

    // Step 2: Add to the receiver's account
    const [creditResult] = await connection.execute(
      'UPDATE accounts SET balance = balance + ? WHERE id = ?',
      [amount, toAccountId]
    );

    if (creditResult.affectedRows === 0) {
      throw new Error('Invalid receiver account');
    }

    // If both steps succeed, permanently commit the changes to disk
    await connection.commit();
    console.log('Transaction completed successfully!');
  } catch (error) {
    // If anything fails, revert all changes immediately
    await connection.rollback();
    console.error('Transaction failed. Rolled back changes:', error.message);
  } finally {
    await connection.end();
  }
}

The Takeaway

ACID transactions are the ultimate insurance policy for your application's state. While they introduce a minor performance trade-off due to locking rows during updates, they completely eliminate the need to write complex, error-prone fallback code. Shifting the responsibility of data integrity to your database engine lets you sleep easy knowing your system will never tell a lie.

Originally published on my blog. You can read the alternative breakdown here.

Database Concurrency 101: Optimistic vs. Pessimistic Locking

Sun, 27 Sep 2026 09:32:31 +0000

In modern software development, we often have to handle multiple users interacting with our application at the exact same time. Optimistic and pessimistic locking are two distinct strategies used to manage concurrency, which is the technical term for when multiple users or systems try to update the exact same database record simultaneously. While optimistic locking assumes conflicts are rare and verifies changes only at the very end of an operation, pessimistic locking assumes conflicts are highly likely and preemptively blocks others from accessing the data until the current operation is finished.

A Relatable Everyday Analogy

To understand the difference, imagine a collaborative digital workspace where you and your colleagues edit text files.

Pessimistic locking is like an old-school file server where only one person can open a document for editing at a time. The moment you open the file, the server locks it. If your colleague tries to open it, they receive a "Read-Only" warning and are completely blocked from making edits until you save, close, and release your lock on the file.

Optimistic locking, on the other hand, is like a modern version control system like Git. Both you and your colleague can open, read, and edit the exact same document simultaneously. However, when you click "Save," the system checks if anyone else has saved a new version of the document since you opened it. If no one has, your save succeeds. If your colleague managed to save their changes first, the system blocks your save, alerts you to a "conflict," and asks you to review and merge the changes before trying again.

Why It Matters in the Tech Industry

In the tech industry, failing to implement a proper locking strategy leads to a highly destructive bug known as a "lost update" or a "race condition"—where the outcome of a process depends on the unpredictable sequence or timing of other events.

Imagine a financial application where two family members try to withdraw $50 from a shared $100 bank account at the exact same millisecond. Without locking, both transactions will read the balance as $100 simultaneously. Both transactions will approve the $50 withdrawal, and both will attempt to write the new balance of $50 back to the database. The bank is now down $100, but the database incorrectly shows a remaining balance of $50 because the second write silently overwrote the first one. By using database locking, software engineers prevent this double-spending, protect system integrity, and ensure transactions are processed safely.

Implementation in Node.js and MySQL

Here is a simple example of how to implement optimistic locking in a Node.js application using a MySQL database client. We do this by adding a version column to our database table and verifying it during our update query.

// A helper function to update a user's bank balance using Optimistic Locking
async function updateAccountBalance(connection, accountId, withdrawAmount, currentVersion) {
  // We attempt to update the balance and increment the version number,
  // but ONLY if the version in the database matches the version we originally read.
  const query = `
    UPDATE bank_accounts 
    SET balance = balance - ?, version = version + 1 
    WHERE id = ? AND version = ?
  `;

  const [result] = await connection.execute(query, [withdrawAmount, accountId, currentVersion]);

  // If no rows were affected, it means the version changed in the background
  // because another transaction updated it first. We must roll back and retry.
  if (result.affectedRows === 0) {
    throw new Error("Concurrency Conflict: The account was updated by another process. Please retry.");
  }

  return "Balance updated successfully!";
}

The Strategic Takeaway

Choosing between these two approaches is always a trade-off between speed and strict prevention. You should default to optimistic locking for applications with low-to-medium write conflicts because it keeps your application fast, highly available, and free of database bottlenecks. However, when dealing with highly critical operations where conflicts are frequent and data corruption is unacceptable—such as reservation systems or payment gateways—pessimistic locking is the superior choice, despite the performance hit of making users wait in a digital queue.

Originally published on my blog. You can read the alternative breakdown here.

Data Engineering Beyond "Pretty Charts": How I Built an Aviation Telemetry Leakage Engine

Wed, 23 Sep 2026 12:10:54 +0000

Stop Tracking "On-Time" Flights. Track Tarmac Fuel Destruction Instead.

Most aviation dashboards track gate-to-gate "On-Time" arrivals. The problem? It's a lagging indicator that completely masks why airlines silently hemorrhage millions. Padded flight schedules disguise a massive bottleneck: chronic tarmac fuel destruction occurring right before takeoff.

I built AeroOps, an end-to-end telemetry and multi-domain operations leakage engine, to expose exactly where systems drop value between the hangar and the runway.

Repo & Technical Report: [GitHub Link]
Stack: MySQL (Star Schema & CTEs) · Telemetry Data Modeling · Executive Dashboard Architecture · Power BI (DAX)

The Architectural & Engineering Challenge

Handling flight telemetry means dealing with high-volume, disjointed datasets. To make this actionable, the engineering required:

Star Schema Data Warehousing: Joined 513,000+ flight legs with precise technical burn parameters pulled directly from the ICAO Aircraft Emissions Databank.
Bypassing Relational Boundaries: Formulated deduplicated DAX lookup models to handle wide-body propulsion penalties under sub-second memory constraints.
Algorithmic Baseline Filters: Enforced a strict 4-minute engine warm-up buffer directly into the SQL logic. This stripped away safe, expected operational windows to isolate true, excess ground latency.

The Bottom-Line Impact Metrics

By querying raw transaction logs and user event data instead of building passive, "pretty" charts, the engine surfaced massive operational friction:

$42.69M in Tarmac Cash Destruction: Isolated and quantified the exact cost of ground traffic gridlocks at major hubs like Atlanta (ATL).
54% to 57% Short-Haul Menu Spoilage: Exposed a massive waste loop across major carriers (Delta/American) due to over-boarding. Transitioning the system to a 100% shelf-stable inventory model reclaimed $2.58M in simulated margin leaks.
$725K Legal Fine Exposure: Isolated a hidden 9% telemetry sync drop rate specifically among regional commuter airframes caused by corrupted logs.
Prevention of $1.50M Asset Groundings (AOG): Coded an automated Reorder Point (ROP) trigger tool to insulate warehouses against catastrophic out-of-stock asset scenarios.

The Takeaway

Data shouldn't just look nice in a presentation deck. If your queries aren't actively protecting business margins or exposing systemic infrastructure leaks, you're looking at the wrong metrics.

If your team is scaling fast and dealing with messy relational pipelines or unoptimized telemetry data, let's connect and fix your backend infrastructure loops.

The 200-byte trap: why WordPress core updates break Arabic URLs

Tue, 22 Sep 2026 09:22:50 +0000

You update WordPress on a quiet afternoon, a routine release, the kind you've installed a hundred times. The dashboard says everything went fine. Then the 404s start: not a handful, but every long-headlined article in your archive, all at once, all in Arabic.

Nothing in the update log mentions it. No plugin changed. And the cruel part: the data that made those URLs work is already gone, shaved off inside a database-upgrade routine that ran for a few milliseconds and reported success. This isn't a freak accident or a broken plugin. It's three separate assumptions baked into WordPress core, each hard-coding the same number, 200, and Arabic sites are almost uniquely exposed to all three. We hit this running WordPress for Arabic newsrooms, the same high-traffic publishing we've written about surviving breaking-news spikes. We traced it to the exact lines in core and built a fix that survives every future update. Here's the whole story.

Why Arabic URLs hit a wall English never does

WordPress stores a post's slug in the post_name column of wp_posts, and a category or tag slug in the slug column of wp_terms. Both are VARCHAR(200) by default: room for 200 characters, which for an English headline is generous. "Everything you need to know about our new pricing" is barely fifty. You'd have to write a paragraph to run out.

Arabic is a different arithmetic, because of what WordPress actually stores in that column. It doesn't keep the raw Arabic text; it stores the percent-encoded form, the same %XX sequence that travels in the URL. Take a single word:

الذكاء  →  %d8%a7%d9%84%d8%b0%d9%83%d8%a7%d8%a1

Every Arabic letter is two bytes in UTF-8, and every byte becomes a three-character %XX token. So one Arabic character costs about six characters of column space. Do the division: VARCHAR(200) holds roughly 33 Arabic characters. A normal news headline, "القبض على المتهمين في قضية الاحتيال الإلكتروني", blows past that before it's halfway done.

So Arabic publishers learn early to widen the columns:

ALTER TABLE wp_posts MODIFY post_name VARCHAR(1024) NOT NULL DEFAULT '';
ALTER TABLE wp_terms MODIFY slug      VARCHAR(1024) NOT NULL DEFAULT '';

VARCHAR(1024) holds about 170 Arabic characters, comfortable. Permalinks resolve, the newsroom publishes, and the problem looks solved.

It isn't. It's about one-third solved, and the other two-thirds are waiting for the next core update. (This isn't only an Arabic problem, either: any non-Latin script that percent-encodes (Persian, Urdu, Hebrew, Greek) hits the same wall. Arabic publishers just hit it first; scripts that take three bytes a character, such as Hindi, Thai or Chinese, hit it harder still, at about 22 characters to Arabic's 33.)

The number 200 lives in three places, not one

Here's the insight that turns this from a recurring mystery into something you can actually close out. WordPress hard-codes "200" in three independent places, and widening the column only touches the first:

Layer	What it caps at 200	Does widening the column fix it?
1, Storage	the column that physically stores the slug	Yes, but a core update silently undoes it
2, Generation	every new slug, the moment it is created	No, it's PHP; needs a code override
3, De-duplication	slugs that collide and need a numeric suffix	No, but this only fires on collisions

Your ALTER fixes Layer 1's storage. But Layer 2 means that even with a 1024-wide column, every new slug is still cut to 200 bytes before it's ever saved, unless something on your site already overrode it. (If long Arabic slugs save correctly for you today, a plugin, your theme, or a past developer already patched Layer 2, which means you have an undocumented dependency you didn't know you were relying on.) And Layer 1, the storage you so carefully widened, is exactly the one a core update quietly reverts. Let's take each in turn.

Layer 1: how a core update erases your column width

WordPress keeps a canonical description of its own database schema, and on every major update it reconciles the live database against that description. The chain is short and deterministic:

A new release bumps $wp_db_version in version.php.
On the next admin request, wp_upgrade() notices the bump and runs the database upgrade.
That calls make_db_current_silent(), which calls dbDelta( wp_get_db_schema() ).
wp_get_db_schema() is the canonical schema, and it says, verbatim:

post_name varchar(200) NOT NULL default '',
...
slug varchar(200) NOT NULL default '',

dbDelta() then runs a DESCRIBE on every live table, compares each column to the canonical definition, and for any mismatch emits an ALTER TABLE … CHANGE COLUMN to "correct" it. Your post_name is VARCHAR(1024); the canonical says VARCHAR(200); they don't match, so dbDelta dutifully issues a CHANGE COLUMN back to 200, MySQL rebuilds the table, and every byte past the 200th is discarded.

Here's the detail that makes it unrecoverable. Years ago, core added downsize protection to dbDelta, but only for TEXT and BLOB columns (it was Trac #36748, born from exactly this kind of data loss). VARCHAR got no such guard. So a TEXT column is safe from accidental shrinking; your VARCHAR(1024) slug column is not. dbDelta shrinks it without hesitation, and restoring the column to 1024 afterwards brings back the width but not the data; those bytes were handed to MySQL and dropped.

That's why your fix keeps coming undone: always right after an update, always silently. It isn't a bug aimed at you; it's core doing exactly what it was designed to do, to a column you were never expected to change.

For accuracy: a separate routine, pre_schema_upgrade(), also runs hand-written
ALTERs, but those are version-gated to historic migrations and won't touch a
modern site's post_name. The reversion you're seeing is dbDelta.

The fix: don't repair the damage, prevent the shrink

The instinct is to write something that detects the reverted column and re-widens it after each update. Resist it. By the time you can detect the shrink, the data is already gone; you'd just be putting an empty wider column back over truncated slugs. The only fix that actually saves the data is one that stops the shrink from ever happening.

And core hands you the seam to do it. dbDelta exposes a filter, dbdelta_create_queries, that receives the canonical CREATE TABLE statements before they're compared to the live database. Rewrite the canonical post_name varchar(200) to varchar(1024) in that filter, and dbDelta's idea of "correct" now matches what's actually on disk. It sees no mismatch. It emits no CHANGE COLUMN. There is nothing to truncate.

<?php
/**
 * Plugin Name: Arabic Slug Schema Guard (must-use)
 * Description: Keeps wp_posts.post_name and wp_terms.slug at VARCHAR(1024) across
 *              core DB upgrades, so long Arabic slugs are never truncated.
 *              Prevention-first: dbDelta never even tries to shrink them.
 */
defined( 'ABSPATH' ) || exit;

const SLUG_COLUMN_LEN = 1024;

// Rewrite the canonical schema dbDelta diffs against, so "desired" already
// equals the live 1024-wide column, so no CHANGE COLUMN is ever emitted.
add_filter( 'dbdelta_create_queries', function ( $queries ) {
    global $wpdb;

    foreach ( array( $wpdb->posts => 'post_name', $wpdb->terms => 'slug' ) as $table => $column ) {
        if ( isset( $queries[ $table ] ) ) {
            $queries[ $table ] = preg_replace(
                '/(\b' . $column . '\s+varchar\()\s*200\s*(\))/i',
                '${1}' . SLUG_COLUMN_LEN . '${2}',
                $queries[ $table ]
            );
        }
    }
    return $queries;
} );

That single filter is the whole prevention story for Layer 1, and it's robust in a way an after-the-fact repair can never be: it runs inside dbDelta, so it covers every path that triggers a schema reconcile: the admin "database update required" screen, background auto-updates, and wp core update-db on the CLI alike.

Where this code lives matters as much as what it does. Put it in a must-use plugin (a single .php file in wp-content/mu-plugins/), not a regular plugin. MU-plugins always load, can't be deactivated from the dashboard, and load before the upgrade routine runs. And because a core update only ever replaces wp-admin, wp-includes and the root PHP files (it never touches wp-content), your guard is guaranteed to still be in place at the exact moment the upgrade fires. A regular plugin could be deactivated the one time it mattered; don't bet your archive on a toggle.

Layer 2: stop new slugs truncating at birth

Prevention on the column keeps your existing slugs safe. But Layer 2 is still cutting every new slug to 200 bytes at the moment of creation, inside sanitize_title_with_dashes(). The relevant line, unchanged in current core, is:

$title = utf8_uri_encode( $title, 200 );

That function is attached to the sanitize_title filter, so you can cleanly swap it for an identical copy that uses a larger byte budget:

// Layer 2: replace core's slug generator with a byte-for-byte copy whose only
// change is the encoding cap. The copy self-tests against core (see below), so a
// future core rewrite of this function is caught instead of drifting silently.
remove_filter( 'sanitize_title', 'sanitize_title_with_dashes' );
add_filter( 'sanitize_title', 'asg_sanitize_title_with_dashes', 10, 3 );

function asg_sanitize_title_with_dashes( $title, $raw_title = '', $context = 'display' ) {
    // … the body is line-for-line core, with one change:
    //     utf8_uri_encode( $title, 200 )  →  utf8_uri_encode( $title, 1000 )
    // 1000 leaves headroom under the 1024 column for a "-2" collision suffix.
}

Pick a sane budget: around 1000 bytes (≈ 160 Arabic characters) leaves room under the column for the numeric suffix WordPress adds to resolve duplicates. You don't want unlimited slugs; you want enough.

Layer 3 (_truncate_post_slug()) caps at 200 too, but it only runs when a slug collides and needs a -2 suffix. For unique news headlines that almost never fires, and it isn't filterable, so most sites can leave it alone. If you genuinely need more than 200 bytes even on collisions, you take over uniqueness through the pre_wp_unique_post_slug filter, an advanced case worth a footnote, not a paragraph.

Keeping the fork honest

Copying a core function buys a maintenance debt: the day WordPress rewrites sanitize_title_with_dashes(), your copy diverges, silently, which is the exact failure mode we're trying to kill. Pinning the copy to a core version and re-diffing by hand works right up until the release someone forgets to check, which on a long-lived site is when, not if. So don't rely on remembering. Make the plugin re-diff itself.

The lever: remove_filter() only unhooks the function; sanitize_title_with_dashes() stays callable. So you can use core's own current code as a live oracle and assert your fork still agrees with it on short inputs, where neither byte cap fires and the only thing left that could differ is the cleanup logic:

// Run once per core version. Short fixtures => neither length cap engages,
// so any difference is pure logic drift, not the byte budget.
foreach ( array( 'الذكاء الاصطناعي', 'عاجل: «مهم» اليوم', 'foo/bar — baz' ) as $f ) {
    if ( sanitize_title_with_dashes( $f, $f, 'save' )
         !== asg_sanitize_title_with_dashes( $f, $f, 'save' ) ) {
        // core changed the cleanup logic: log + email ops, and (optionally) fall
        // back to core's generator so slugs cap at 200 again: degraded, but safe
        // and known, never a malformed slug from a stale copy.
    }
}

This beats a source-hash check on two counts. It's semantic, not textual: it stays quiet when core only touches the length handling (which you override anyway) and fires only when a change actually breaks your output, so the alerts that do fire get acted on. And it degrades safely: on drift you fall back to core's exact behaviour: worst case, new slugs cap at 200 until you re-sync, never a half-encoded slug from a stale copy. Loud-and-safe beats silent-and-wrong. The full guard (with the per-core-version cache and the optional fail-safe switch) is in the complete plugin below.

A tripwire, because silent failure is the real enemy

The filter prevents the known incident. But the failure mode that actually hurt you was that it happened silently, so the last piece is making sure you'd never again learn about it from a drop in search traffic. The same MU-plugin can verify the column widths after every update and shout if anything is wrong:

Hook upgrader_process_complete to flag that a core update just ran.
On the next admin request, check the real column widths in information_schema and, if either has shrunk, write to the error log and email ops.
Expose a WP-CLI command for cron-based monitoring:

# Nightly: verify the slug columns, email ops if either has reverted.
0 3 * * *  cd /var/www/site && wp asg verify | grep -q REVERTED \
           && wp asg verify | mail -s "WP slug schema reverted on $(hostname)" *Emails are not allowed*

One discipline to internalise: this tripwire restores confidence, not data. If it ever reports a reverted column, treat it as an incident, not a self-healing event; the slugs longer than the new width are already gone and must come back from a backup. Prevention (the filter) and backups are your safety net; the tripwire just tells you when to reach for them.

"Isn't VARCHAR(1024) dangerous?": no, and the reason is reassuring

Widening these columns sounds risky (indexes, utf8mb4, InnoDB limits), and it's the part people most often get scared off by. It's almost entirely a non-issue, and one fact explains why:

The index on these columns is a fixed 191-character prefix, independent of the column's declared length. Core defines KEY post_name (post_name(191)); it indexes the first 191 characters whether the column is 200 wide or 1024. Widening the column doesn't enlarge the index by a single byte.

That dissolves the usual worries:

Index / InnoDB prefix limits. The old InnoDB prefix ceiling is 767 bytes; WordPress chose 191 because 191 × 4 (utf8mb4's worst case) = 764, just under it. Because the prefix stays at 191 no matter how wide the column grows, you never move towards that limit. (A quiet irony: slugs are stored as percent-encoded ASCII, so they were never going to hit four bytes per character anyway.)
Row size. VARCHAR stores the actual length of each value, not its declared maximum; a 30-character slug takes ~30 bytes in a VARCHAR(1024) exactly as it would in a VARCHAR(200). Rows don't get bigger because the ceiling moved up.
Query plans. Lookups use the same 191-prefix index to narrow, then verify the full value. Identical behaviour at 200 or 1024.
Plugins. SEO plugins, caches and CDNs read post_name and emit longer URLs without complaint.

The genuine risks aren't in steady state; they're tools that recreate the schema from an old definition: a migration plugin that runs its own CREATE TABLE, or, most commonly, restoring a SQL dump taken before you widened the columns. A dump of the current database preserves VARCHAR(1024) correctly; importing an old structure silently puts you back at 200. Write that one down in your runbook.

Doing it safely on a site with millions of posts

There's one operational gotcha, and it's about the migration itself, not the result. Going from VARCHAR(200) to VARCHAR(1024) cannot be an instant, in-place change. InnoDB only keeps a VARCHAR length change "in place" while both lengths stay in the same size class: values under 256 use one length byte, values 256 and over use two. Crossing 255 forces ALGORITHM=COPY: a full table rebuild. On a wp_posts with millions of rows, a raw ALTER means a long operation and a problematic lock.

So on a large live site, do the initial widening with an online schema-change tool, not a bare ALTER:

pt-online-schema-change \
  --alter "MODIFY post_name VARCHAR(1024) NOT NULL DEFAULT ''" \
  --chunk-time=0.5 --max-load Threads_running=50 \
  --execute D=wordpress,t=wp_posts

(gh-ost is an equally good triggerless alternative.) wp_terms is tiny and takes a plain ALTER without ceremony; wp_posts is the one that needs the online tool. The rollout that's bitten no one:

Back up, and confirm the dump shows VARCHAR(1024) afterwards.
Clone to staging, deploy the MU-plugin, and run a real major core update there. Confirm no CHANGE COLUMN appears and the verify command reports OK.
Deploy the MU-plugin to production before the next core update: prevention has to be in place ahead of the event it prevents.
Widen the production columns off-peak with pt-osc / gh-ost.
Add a nightly verify cron and a 404 monitor, so anything unexpected surfaces in hours, not in a month of lost rankings.
Keep the MU-plugin in version control, and never import a pre-widening structure dump.

The deeper fix: stop betting your URLs on a column width

Everything above makes Arabic slugs safe. But there's a more permanent way to think about it, and for a large news archive it's worth the shift: don't let the URL depend on the slug surviving at all.

Anchor your permalinks on the numeric post ID, and keep the Arabic slug as a descriptive (but cosmetic) suffix:

https://example.com/123456/الذكاء-الاصطناعي

Resolve the post by its ID, and the slug becomes decoration. If it's ever truncated, malformed, or edited, the URL still resolves to the right article instead of 404-ing, and WordPress can 301 to the canonical form. You keep the Arabic keywords in the URL for search relevance and you become structurally immune to the entire class of problem. (If you want to stop touching core tables altogether, you can go one step further and store the long slug in your own indexed routing table, which dbDelta has no opinion about, but for most teams, ID-anchored permalinks are the sweet spot.)

That's the difference between patching a bug and designing it out.

The complete plugin

Everything above, assembled into one must-use plugin. Drop it in wp-content/mu-plugins/wp-arabic-slug-schema-guard.php, widen the columns once with an online schema-change tool, and all three layers, plus the tripwire, are handled in a single place. The slug generator is a copy of core's sanitize_title_with_dashes(); only the byte cap differs.

📦 Prefer it ready-made? The plugin is open-source on GitHub as mantekio/wp-arabic-slug-schema-guard, or composer require mantekio/wp-arabic-slug-schema-guard. Star it, file an issue, or lift it whole.

<?php
/**
 * Plugin Name: Arabic Slug Schema Guard (must-use)
 * Description: Keeps wp_posts.post_name and wp_terms.slug at VARCHAR(1024) across
 *              core DB upgrades, lets core generate long URL-encoded Arabic slugs,
 *              and verifies + alerts after every update. Prevention-first:
 *              dbDelta never truncates, because the canonical schema it diffs
 *              against already says 1024. The slug fork self-tests against core
 *              on each update and can fail safe.
 */
defined( 'ABSPATH' ) || exit;

const ASG_COLUMN_LEN = 1024;  // physical column width (bytes)
const ASG_SLUG_BYTES = 1000;  // max generated slug length, under the column width
const ASG_SCHEMA_SIG = 'post_name:1024;slug:1024';
// define( 'ASG_ALERT_EMAIL', '*Emails are not allowed*' );  // optional, set in wp-config.php
// define( 'ASG_L2_FAILSAFE', true );               // optional: on Layer-2 drift, fall back to core (slugs cap at 200) until you re-sync

/* LAYER 1: PREVENTION
 * Rewrite the canonical CREATE TABLE dbDelta diffs against, so "desired" already
 * equals the live 1024-wide column and no destructive CHANGE COLUMN is emitted. */
add_filter( 'dbdelta_create_queries', function ( $queries ) {
    global $wpdb;
    foreach ( array( $wpdb->posts => 'post_name', $wpdb->terms => 'slug' ) as $table => $column ) {
        if ( isset( $queries[ $table ] ) ) {
            $queries[ $table ] = preg_replace(
                '/(\b' . $column . '\s+varchar\()\s*200\s*(\))/i',
                '${1}' . ASG_COLUMN_LEN . '${2}',
                $queries[ $table ]
            );
        }
    }
    return $queries;
} );

/* LAYER 2: GENERATION
 * Core caps new slugs at 200 bytes via utf8_uri_encode($title, 200). Swap in a
 * copy of sanitize_title_with_dashes() whose only change is the byte budget.
 * Installed conditionally: if the drift guard (below) flagged THIS core version
 * and ASG_L2_FAILSAFE is on, stay on core's default (caps at 200, degraded but
 * safe and known) rather than run a fork we know has drifted. */
if ( get_option( 'asg_l2_status' ) === 'drift:' . $GLOBALS['wp_version']
     && defined( 'ASG_L2_FAILSAFE' ) && ASG_L2_FAILSAFE ) {
    // Known-drifted on this core version → leave core's generator in place.
} else {
    remove_filter( 'sanitize_title', 'sanitize_title_with_dashes' );
    add_filter( 'sanitize_title', 'asg_sanitize_title_with_dashes', 10, 3 );
}

function asg_sanitize_title_with_dashes( $title, $raw_title = '', $context = 'display' ) {
    $title = strip_tags( $title );
    // Preserve already-encoded octets through the cleanup below.
    $title = preg_replace( '|%([a-fA-F0-9][a-fA-F0-9])|', '---$1---', $title );
    $title = str_replace( '%', '', $title );
    $title = preg_replace( '|---([a-fA-F0-9][a-fA-F0-9])---|', '%$1', $title );

    if ( seems_utf8( $title ) ) {
        if ( function_exists( 'mb_strtolower' ) ) {
            $title = mb_strtolower( $title, 'UTF-8' );
        }
        $title = utf8_uri_encode( $title, ASG_SLUG_BYTES );  // core hard-codes 200 here
    }

    $title = strtolower( $title );

    if ( 'save' === $context ) {
        // EVERYTHING here is copied verbatim from core. The ONLY sanctioned difference
        // between this fork and core is the byte budget above. Drop a line and you have
        // drifted, see the note under the drift guard below.

        // Turn non-breaking spaces, en/em dashes and slashes into hyphens.
        $title = str_replace( array( '%c2%a0', '%e2%80%93', '%e2%80%94' ), '-', $title );
        $title = str_replace( array( ' ', ' ', '–', '–', '—', '—' ), '-', $title );
        $title = str_replace( '/', '-', $title );

        // Strip these characters entirely. Note %c2%ab / %c2%bb: the guillemets Arabic
        // uses as quotation marks. And %e2%80%8e / %e2%80%8f: the LRM and RTL marks.
        $title = str_replace(
            array(
                '%c2%ad',                                            // soft hyphen
                '%c2%a1', '%c2%bf',                                  // iexcl, iquest
                '%c2%ab', '%c2%bb', '%e2%80%b9', '%e2%80%ba',        // angle quotes
                '%e2%80%98', '%e2%80%99', '%e2%80%9c', '%e2%80%9d',  // curly quotes
                '%e2%80%9a', '%e2%80%9b', '%e2%80%9e', '%e2%80%9f',
                '%e2%80%a2',                                         // bullet
                '%c2%a9', '%c2%ae', '%c2%b0', '%e2%80%a6', '%e2%84%a2',
                '%c2%b4', '%cb%8a', '%cc%81', '%cd%81',              // acute accents
                '%cc%80', '%cc%84', '%cc%8c',                        // grave, macron, caron
                '%e2%80%8b', '%e2%80%8c', '%e2%80%8d',               // zero-width chars
                '%e2%80%8e', '%e2%80%8f',                            // LRM, RTL mark
                '%e2%80%aa', '%e2%80%ab', '%e2%80%ac',
                '%e2%80%ad', '%e2%80%ae',
                '%ef%bb%bf',                                         // byte order mark
                '%ef%bf%bc',                                         // object replacement
            ),
            '',
            $title
        );

        // Convert × to 'x'.
        $title = str_replace( '%c3%97', 'x', $title );
    }

    $title = preg_replace( '/&.+?;/', '', $title );          // strip remaining entities
    $title = str_replace( '.', '-', $title );
    $title = preg_replace( '/[^%a-z0-9 _-]/', '', $title );  // keep only slug-safe chars
    $title = preg_replace( '/\s+/', '-', $title );
    $title = preg_replace( '|-+|', '-', $title );
    return trim( $title, '-' );
}

/* LAYER 2 DRIFT GUARD: self-test the fork against core's live implementation.
 * remove_filter() only unhooks sanitize_title_with_dashes(); the function stays
 * callable, so we use core's CURRENT code as a live oracle and assert our fork
 * still agrees on SHORT inputs (where neither byte cap engages, the only thing
 * the fork changed). Any divergence == core altered the cleanup logic and our
 * copy has drifted. Runs once per core version; shouts, and can fail safe. */
add_action( 'admin_init', 'asg_guard_layer2_drift' );

function asg_guard_layer2_drift() {
    $ok      = 'ok:' . $GLOBALS['wp_version'];
    $drifted = 'drift:' . $GLOBALS['wp_version'];
    if ( in_array( get_option( 'asg_l2_status' ), array( $ok, $drifted ), true ) ) {
        return;  // already settled for this core version
    }
    if ( ! function_exists( 'sanitize_title_with_dashes' ) ) {
        return;  // core refactored it away: no oracle to diff against; the fork
                 // still runs as a standalone copy (it needs no removed function).
    }

    // Deliberately SHORT fixtures so neither length cap fires; any difference is
    // pure cleanup-logic drift, not the byte budget.
    $fixtures = array(
        'الذكاء الاصطناعي',
        'عاجل: تطورات «مهمة» اليوم',
        'Mixed عربي + English — dash',
        'foo/bar — baz',
        'UPPER   multiple   spaces',
        '%d8%a7 pre-encoded octet',
    );

    $drift = array();
    foreach ( $fixtures as $f ) {
        $core = sanitize_title_with_dashes( $f, $f, 'save' );
        $ours = asg_sanitize_title_with_dashes( $f, $f, 'save' );
        if ( $core !== $ours ) {
            $drift[] = "in=[{$f}] core=[{$core}] ours=[{$ours}]";
        }
    }

    if ( $drift ) {
        $msg = "[Arabic Slug Schema Guard] Layer-2 DRIFT on WP {$GLOBALS['wp_version']}: core "
             . "sanitize_title_with_dashes() no longer matches our fork; re-sync the copy.\n"
             . implode( "\n", $drift );
        error_log( $msg );
        if ( defined( 'ASG_ALERT_EMAIL' ) && function_exists( 'wp_mail' ) ) {
            wp_mail( ASG_ALERT_EMAIL, 'WP slug Layer-2 drift: ' . wp_parse_url( home_url(), PHP_URL_HOST ), $msg );
        }
        update_option( 'asg_l2_status', $drifted, true );  // autoload: the fail-safe reads it every request
        return;  // do NOT cache "ok" while drifted
    }
    update_option( 'asg_l2_status', $ok, true );
}

/* LAYER 3: DE-DUPLICATION (optional)
 * _truncate_post_slug() also caps at 200, but only when a slug COLLIDES and needs
 * a "-2" suffix, rare for unique headlines, and it isn't filterable. If you need
 * >200 bytes even on collisions, take over uniqueness here. Most sites skip this.
 *
 * add_filter( 'pre_wp_unique_post_slug', 'asg_unique_post_slug', 10, 6 );
 */

/* TRIPWIRE: verify + alert after every core update
 * This only detects and alerts: it restores the COLUMN, never bytes already
 * truncated. Treat any revert as an incident: restore from backup. */
add_action( 'upgrader_process_complete', function ( $upgrader, $extra ) {
    if ( isset( $extra['type'] ) && 'core' === $extra['type'] ) {
        delete_option( 'asg_schema_sig' );  // force a re-check on the next admin_init
    }
}, 10, 2 );

add_action( 'admin_init', function () {
    if ( get_option( 'asg_schema_sig' ) === ASG_SCHEMA_SIG ) {
        return;  // fast path: already verified since the last update
    }
    $reverted = asg_reverted_columns();
    if ( $reverted ) {
        $msg = '[Arabic Slug Schema Guard] Column(s) reverted: ' . implode( ', ', $reverted )
             . '. Long slugs were truncated: restore from backup and check 404 logs.';
        error_log( $msg );
        if ( defined( 'ASG_ALERT_EMAIL' ) ) {
            wp_mail( ASG_ALERT_EMAIL, 'WP slug schema reverted: ' . wp_parse_url( home_url(), PHP_URL_HOST ), $msg );
        }
        return;  // do NOT cache "ok" while broken
    }
    update_option( 'asg_schema_sig', ASG_SCHEMA_SIG, false );
} );

function asg_reverted_columns() {
    global $wpdb;
    $reverted = array();
    foreach ( array( array( $wpdb->posts, 'post_name' ), array( $wpdb->terms, 'slug' ) ) as list( $table, $column ) ) {
        $len = (int) $wpdb->get_var( $wpdb->prepare(
            "SELECT CHARACTER_MAXIMUM_LENGTH FROM information_schema.COLUMNS
              WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s AND COLUMN_NAME = %s",
            DB_NAME, $table, $column
        ) );
        if ( $len && $len < ASG_COLUMN_LEN ) {
            $reverted[] = "{$table}.{$column} (now {$len})";
        }
    }
    return $reverted;
}

/* WP-CLI: `wp asg verify` for cron-based monitoring */
if ( defined( 'WP_CLI' ) && WP_CLI ) {
    WP_CLI::add_command( 'asg verify', function () {
        foreach ( asg_reverted_columns() ?: array( 'ok' ) as $row ) {
            WP_CLI::log( 'ok' === $row ? 'Schema OK: both columns at VARCHAR(1024).' : "REVERTED: {$row}" );
        }
    } );
}

A postscript: the drift guard caught us

The argument above is that a fork of a core function drifts silently, so the guard matters more than the copy. That turned out to be true in the least comfortable way.

Versions 1.0.0 through 1.2.0 of the released plugin shipped a fork that was missing core's entire "strip these characters entirely" block, along with the × to x mapping. The consequence lands exactly on the readers this post is written for: Arabic guillemets (« ») and the LRM/RTL direction marks survived into slugs as %c2%ab, %c2%bb, %e2%80%8f. The character filter at the end cannot remove them, because %c2%ab is built entirely from permitted characters.

The drift guard printed above would have caught it on the first admin_init. Its second fixture is 'عاجل: تطورات «مهمة» اليوم'. It never fired, because the guard is bound to admin_init and nobody had opened wp-admin on an install running that fork. A guard on a hook that never fires is a comment.

Both are fixed in v1.3.0: the fork is re-synced (the block above is now complete and verified against core by an offline oracle on every build), and wp asg verify evaluates the drift oracle itself rather than waiting for an admin page load. If you lifted this code before July 2026, take the str_replace block above.

This is the kind of problem that never shows up in a tutorial. It lives where WordPress internals, MySQL's storage rules, Arabic character encoding and SEO all overlap, and it only reveals itself at the worst possible moment. It's also the territory we work in every day: keeping Arabic publishing fast, multilingual and unbreakable on WordPress, right through to AWS at scale.

If you run a WordPress publication in Arabic and want it to survive its own updates, let's talk.

Stop Killing Your Database: The Power of Connection Pooling

Tue, 22 Sep 2026 08:51:06 +0000

Connection pooling is a software development technique used to maintain a cache of database connections that can be reused for future requests. Instead of opening and closing a brand-new connection to the database every time a user wants to read or write data, your application keeps a "pool" of active connections ready to go. This drastically reduces the overhead of establishing new database handshakes over and over again.

The Pizza Delivery Analogy

Imagine you run a busy pizza shop. Every time a delivery order comes in, instead of sending a driver who is already waiting at the shop, you post a job listing online, interview a candidate, hire them, buy them a car, have them deliver the pizza, and then fire them and sell the car immediately after.

It sounds completely insane, right? Yet, that is exactly what your server does when it creates a brand-new database connection for every single HTTP request, only to destroy it a millisecond later.

Connection pooling is like hiring a dedicated team of five delivery drivers who hang out at the shop. When an order is ready, one driver grabs it, delivers it, and immediately returns to the shop to wait for the next order.

Why Connection Pooling Matters Daily

In production environments, creating a database connection is an expensive operation that involves cryptographic handshakes, network latency, and authentication checks. If your Node.js application receives 100 concurrent API requests and tries to spin up 100 separate database connections simultaneously, your MySQL database will quickly run out of memory or reject connections altogether, throwing the dreaded "Too many connections" error.

By implementing a connection pool, you cap the maximum number of active database connections, protect your database from crashing under sudden traffic spikes, and shave off up to 90% of your API response latency because the connection is already warmed up and ready to run queries.

Implementation in Node.js & MySQL

Here is how simple it is to set up a connection pool in an Express.js backend using the popular mysql2 driver:

const express = require('express');
const mysql = require('mysql2/promise');

const app = express();

// Create a connection pool instead of a single connection
const pool = mysql.createPool({
  host: 'localhost',
  user: 'root',
  database: 'store_db',
  waitForConnections: true,
  connectionLimit: 10, // Max 10 active connections kept in the pool
  queueLimit: 0
});

app.get('/products', async (req, res) => {
  try {
    // The pool automatically rents out a connection, runs the query,
    // and returns the connection back to the pool when finished.
    const [rows] = await pool.query('SELECT * FROM products LIMIT 10');
    res.json(rows);
  } catch (error) {
    console.error('Database query error:', error);
    res.status(500).send('Database error');
  }
});

app.listen(3000, () => {
  console.log('Server running on port 3000');
});

The Takeaway

Optimizing your backend performance isn't always about writing faster algorithms; often, it's about managing your external resources wisely. By treating database connections as a reusable, limited public utility rather than disposable single-use items, you build highly resilient APIs capable of weathering massive traffic spikes without sweating.

Originally published on my blog. You can read the alternative breakdown here.

Why Your Node.js App Crashes Under Traffic: The Magic of Connection Pooling

Sun, 20 Sep 2026 08:56:50 +0000

Imagine launching your shiny new web application. During development, everything runs smoothly. But the moment you launch on social media and a few hundred users hit your API at the exact same time, your server grinds to a halt and starts throwing nasty database connection timeout errors. This common disaster is almost always caused by a lack of database connection pooling.

What is Connection Pooling?

Database connection pooling is a performance optimization technique where a set of active database connections is kept open and shared among multiple requests. Instead of creating and destroying a new database connection every single time an application needs to run a query, the application borrows an existing connection from a pre-allocated pool, executes the query, and immediately returns the connection back to the pool. This prevents the costly overhead of starting new communication channels from scratch.

The Rental Car Analogy

To understand this concept, think of a rental car agency at an airport. Imagine if every time a traveler landed at the airport, the rental agency had to buy a brand-new car from the factory, register it with the DMV, customize it for the driver, and then completely crush and destroy the car at the end of the traveler's weekend trip. That would be an absurdly slow and expensive process.

Instead, the agency maintains a "pool" of 50 pre-registered, fully fueled, ready-to-drive cars parked in a lot. When a traveler arrives, they borrow a car from the lot, use it for their trip, and then return it to the lot. The car is washed and immediately made available for the next traveler. This is connection pooling: instead of building and destroying connections on the fly, you manage a recycled fleet of them.

Why It Matters Daily in the Tech Industry

Opening a database connection is an incredibly resource-intensive operation. To establish a new connection, your backend application and database server must complete a network handshake, negotiate security protocols (like TLS/SSL), authenticate user credentials, and allocate dedicated RAM and CPU threads to handle that specific session.

If you build an application where every user request opens a brand-new connection, a sudden spike in web traffic will force your Node.js server to attempt to open hundreds of database connections simultaneously. Your database server (like MySQL) has a built-in safety limit of maximum allowable concurrent connections. Once that limit is hit, the database will aggressively reject all incoming connection requests, throwing "Too many connections" errors, causing API endpoints to return 500 errors, and crashing your platform. Connection pooling solves this by capping the total number of open database connections and keeping extra queries safely queued up in memory until a connection becomes available.

Connection Pooling in Action (Node.js & MySQL)

Here is how you implement connection pooling using Express and the standard mysql2/promise library in a Node.js environment:

const express = require('express');
const mysql = require('mysql2/promise');
const app = express();

// 1. Create a global connection pool with configuration limits
const pool = mysql.createPool({
  host: 'localhost',
  user: 'root',
  password: 'secure_password',
  database: 'ecommerce_db',
  waitForConnections: true,    // Queue queries if no connections are free
  connectionLimit: 10,         // Maximum number of open connections
  queueLimit: 0                // No limit on the number of queued queries
});

app.get('/api/products', async (req, res) => {
  try {
    // 2. pool.query automatically borrows an idle connection, 
    // executes the query, and returns it to the pool.
    const [products] = await pool.query('SELECT id, name, price FROM products LIMIT 20');
    res.json(products);
  } catch (error) {
    console.error('Database failure:', error);
    res.status(500).json({ error: 'Internal Server Error' });
  }
});

app.listen(3000, () => {
  console.log('Server is running on port 3000');
});

The Takeaway

Software scalability is not just about writing clean logic or optimizing loops; it is about protecting and sharing finite hardware resources under heavy load. By implementing connection pooling, you protect your database from crashing, reduce API response latency, and build backend systems capable of handling traffic spikes gracefully without costing a fortune in extra server infrastructure.

Originally published on my blog. You can read the alternative breakdown here.

Supercharge Your Node.js App: Why You Need Database Connection Pooling

Sat, 19 Sep 2026 08:17:43 +0000

What is Connection Pooling?

Database connection pooling is a technique where a cache of database connections is kept open and shared among multiple queries, rather than opening and closing a brand-new connection for every single query. When your application needs to talk to the database, it borrows an existing connection from the pool, runs the query, and immediately returns it. This avoids the heavy performance penalty of creating and destroying connections constantly.

The Pizza Scooter Analogy

Imagine a busy local pizza shop. Every time a customer orders a pizza, if the shop had to buy a brand-new delivery scooter from the dealership, register it, insure it, deliver the pizza, and then scrap or sell the scooter immediately afterward, the business would go bankrupt in an hour.

Instead, the shop maintains a stable fleet of five delivery scooters parked in the back. When a delivery is ready, a driver grabs an available scooter from the fleet, completes the delivery, and parks it back in the lot for the next driver to use. Connection pooling is that stable fleet of scooters.

Why Connection Pooling Matters Daily

In backend development, establishing a connection to a database like MySQL is an incredibly heavy operation. It requires several steps:

A TCP network handshake.
SSL/TLS security negotiation.
User authentication and privilege verification.
Memory allocation on the database server.

If your Express.js server handles 1,000 requests per second and you open a new connection for each one, your database server's CPU will spike to 100% just managing handshakes. Eventually, the database will run out of resources and throw a dreaded "Too many connections" error, crashing your app. By using a connection pool, you bypass this startup overhead. Your queries run almost instantly because the connection is already active and authenticated.

Simple Code Example

Let's see the difference in a MERN stack application using Node.js, Express, and the mysql2 package.

The Bad Way (New Connection per Request)

const express = require('express');
const mysql = require('mysql2/promise');
const app = express();

const dbConfig = { host: 'localhost', user: 'root', database: 'store' };

app.get('/products', async (req, res) => {
  try {
    // Slow and resource-intensive: creating a connection on every request
    const connection = await mysql.createConnection(dbConfig);
    const [rows] = await connection.execute('SELECT * FROM products LIMIT 10');

    // Closing it manually
    await connection.end();
    res.json(rows);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.listen(3000);

The Good Way (Using a Connection Pool)

const express = require('express');
const mysql = require('mysql2/promise');
const app = express();

const dbConfig = { host: 'localhost', user: 'root', database: 'store' };

// Create a pool of up to 10 reusable connections
const pool = mysql.createPool({
  ...dbConfig,
  connectionLimit: 10,
  waitForConnections: true,
  queueLimit: 0
});

app.get('/products', async (req, res) => {
  try {
    // Automatically borrows an active connection and returns it when done
    const [rows] = await pool.execute('SELECT * FROM products LIMIT 10');
    res.json(rows);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.listen(3000);

Key Takeaway

Database connection pooling is not just a performance optimization; it is a fundamental survival mechanism for production databases. It turns chaotic database traffic into a predictable, queued stream, ensuring your server remains responsive and your database stays healthy under heavy load.

Originally published on my blog. You can read the alternative breakdown here.

The Silent Database Killer: Understanding and Fixing the N+1 Query Problem

Thu, 17 Sep 2026 08:54:44 +0000

The N+1 query problem is a performance bottleneck that occurs when an application makes database queries inside a loop. Instead of fetching all the required data in a single, efficient query, the system executes one initial query to fetch a list of parent records, followed by "N" separate queries to fetch related child records for each item in that list. This creates an exponential number of database requests as your data grows.

The Grocery Store Analogy

Imagine you are baking a cake and need five different ingredients from your local grocery store. Instead of writing all five items on a single list and making one trip, you drive to the store, buy flour, and drive home. Then, you realize you need sugar, so you drive back to the store, buy sugar, and drive home. You repeat this entire round-trip for every single ingredient. This is highly inefficient, wasting time, fuel, and energy. In software, each trip to the database is like one of those painful, redundant car rides.

Why It Matters in the Tech Industry

On a daily basis, software engineers actively fight N+1 queries to prevent application servers and databases from crashing under heavy traffic. Each extra query introduces network latency and forces the database to parse, plan, and execute a new command. This wastes valuable connection pools, spikes CPU usage, and causes page load times to balloon. Fixing this issue is often the difference between a snappy, responsive web app and one that times out during peak business hours.

Spotting and Fixing the Problem in Node.js

Let's look at how this happens in a Node.js and Express application querying a MySQL database, and how to fix it.

// THE PROBLEM: The N+1 Query Way
app.get('/users-with-posts-bad', async (req, res) => {
  // 1. First query to get all users (The "1" in N+1)
  const [users] = await db.query('SELECT id, username FROM users');

  const usersWithPosts = [];

  // 2. Loop through each user and query their posts (The "N" queries)
  for (const user of users) {
    const [posts] = await db.query('SELECT id, title FROM posts WHERE user_id = ?', [user.id]);
    usersWithPosts.push({
      ...user,
      posts
    });
  }

  res.json(usersWithPosts);
});

// THE SOLUTION: The Efficient Way (Using a SQL JOIN)
app.get('/users-with-posts-good', async (req, res) => {
  // A single query fetches all users and their posts at once
  const query = `
    SELECT u.id AS userId, u.username, p.id AS postId, p.title
    FROM users u
    LEFT JOIN posts p ON u.id = p.user_id
  `;
  const [rows] = await db.query(query);

  // Format the flat SQL result into structured JSON
  const usersMap = {};
  for (const row of rows) {
    if (!usersMap[row.userId]) {
      usersMap[row.userId] = {
        id: row.userId,
        username: row.username,
        posts: []
      };
    }
    if (row.postId) {
      usersMap[row.userId].posts.push({
        id: row.postId,
        title: row.title
      });
    }
  }

  res.json(Object.values(usersMap));
});

The Takeaway

Database interactions are highly expensive operations. Always aim to retrieve data in bulk sets rather than looping over individual records. By shifting your mindset from iterative programming to relational database operations, you can transform a slow, unstable backend into a highly optimized, production-ready system that scales effortlessly.

Originally published on my blog. You can read the alternative breakdown here.

Stop Crashing Your Database: A Developer's Guide to Connection Pooling

Wed, 16 Sep 2026 08:51:01 +0000

Database connections are the lifeblood of any web application, but they are also surprisingly expensive to create and destroy. When your application struggles under sudden traffic spikes, the database is often the first thing to break. This is where connection pooling comes in.

Connection pooling is a software development technique that creates and maintains a cache of active database connections, keeping them ready for immediate reuse. Instead of initiating and closing a brand-new connection every time a user requests data, your application borrows an already-established connection from this pre-allocated "pool." Once the query is complete, the connection is returned to the pool for the next request to use, rather than being discarded.

The Taxi Stand Analogy

To understand this clearly, think of a busy taxi stand outside a major hotel. If every guest who wanted to travel into town had to call a car dealership, buy a brand-new car, register it with the city, drive to their destination, and then crush the vehicle at a scrap yard when they arrived, transportation would be impossibly slow, expensive, and wasteful.

Instead, the hotel relies on a taxi stand. A small fleet of vehicles sits idling in the driveway. When a guest steps outside, they instantly hop into an available taxi, take their trip, and the driver returns to the stand to wait for the next passenger. The taxi is never destroyed; it is constantly reused. In this analogy, the guests are your incoming API requests, the taxis are your database connections, and the stand is your connection pool.

Why Connection Pooling Matters Daily

In real-world software engineering, establishing a database connection is not a simple task. It requires a multi-step process: initiating a TCP handshake over the network, negotiating security protocols (TLS/SSL), authenticating the database user's credentials, allocating memory on the database server, and setting up session variables. Doing this on every single HTTP request adds massive latency to your API responses.

Furthermore, databases have strict physical limits. Every active connection consumes a slice of the database server's RAM and CPU. If your Express server experiences a sudden burst of 500 concurrent users and tries to open 500 individual connections simultaneously, your MySQL database will quickly run out of memory. This leads to the dreaded "Too many connections" error, bringing your entire application to a grinding halt. Connection pooling caps the maximum number of connections, forcing extra requests to wait in a fast queue rather than crashing your database server.

Putting It into Code

Here is how you implement connection pooling in a Node.js application using the popular mysql2 driver. Instead of using createConnection, we use createPool to manage our resources efficiently.

const mysql = require('mysql2/promise');

// Configure and create the connection pool
const pool = mysql.createPool({
  host: 'localhost',
  user: 'root',
  password: 'your_secure_password',
  database: 'production_db',
  waitForConnections: true,    // Queue requests when pool is full
  connectionLimit: 10,         // Max number of simultaneous active connections
  queueLimit: 0                // No limit on waiting requests (0 means infinite)
});

// Helper function to fetch a user profile
async function getUserProfile(userId) {
  try {
    // The pool automatically checks out a connection, runs the query, and returns it
    const [rows] = await pool.execute(
      'SELECT id, username, email FROM users WHERE id = ?',
      [userId]
    );
    return rows[0];
  } catch (error) {
    console.error('Database query error:', error);
    throw error;
  }
}

// Example execution
getUserProfile(42).then(user => console.log('User retrieved:', user));

The Takeaway

Connection pooling is a quiet champion of system architecture, transforming database communication from a slow bottleneck into a reliable, high-performance system utility. By setting up a robust pool inside your application, you prevent database crashes, lower response times, and ensure that your backend can handle high-concurrency traffic without breaking a sweat.

Originally published on my blog. You can read the alternative breakdown here.

A Complete Guide to Install XAMPP, Configure Apache & MySQL Services, and Deploy WordPress Using Pow

Wed, 09 Sep 2026 10:21:11 +0000

Setting up a reliable local web development environment is one of the first steps for anyone working with WordPress, PHP, MySQL, or Apache. While XAMPP makes this process easier by bundling the required components into a single package, configuring the environment properly, especially through the command line, can still be challenging.

Many online tutorials focus only on the graphical interface and often skip essential configuration steps, troubleshooting techniques, or service management. As a result, beginners frequently encounter issues with Apache, MySQL, PHP compatibility, phpMyAdmin, or WordPress installation.

To address these challenges, I created a comprehensive guide that demonstrates how to install, configure, and manage an entire XAMPP and WordPress development environment using Windows PowerShell.

Whether you're a student learning web technologies, an educator preparing laboratory sessions, or a developer looking for a repeatable deployment process, this guide provides a structured approach from start to finish.

Why This Guide?

A properly configured local development environment saves hours of troubleshooting and creates a consistent workflow for development, testing, and learning.

Instead of relying solely on mouse clicks, this guide introduces a command-driven approach that helps users understand what happens behind the scenes while also making the installation process easier to reproduce across multiple machines.

The guide follows a complete workflow covering installation, service management, WordPress deployment, version upgrades, and troubleshooting.

What You'll Learn

The guide covers every major step required to build a local WordPress development environment.

1. XAMPP Installation

Learn how to install XAMPP using PowerShell commands, verify the installation, and safely uninstall and reinstall it when necessary.

Topics include:

Downloading XAMPP
Silent installation
Installation verification
Troubleshooting incomplete installations
Safe uninstallation

2. Apache Service Configuration

Apache is configured as a Windows Service, allowing it to be managed directly through PowerShell.

You'll learn how to:

Install Apache as a service
Verify service status
Start and stop Apache
Manage services efficiently
Troubleshoot service-related issues

3. MySQL Service Installation

Database configuration is an essential part of every WordPress deployment.

This section explains how to:

Install MySQL as a Windows Service
Verify successful installation
Start and stop the service
Prepare the database environment for WordPress

4. WordPress Deployment on XAMPP

One of the largest sections of the guide walks through deploying WordPress on a local server.

Topics include:

Creating WordPress databases
Configuring users and permissions
Downloading WordPress
Extracting and organizing files
Running the installation
Configuring the initial website
Accessing the WordPress dashboard

5. Upgrading PHP

Modern WordPress versions require newer PHP releases.

The guide explains how to safely:

Replace outdated PHP versions
Configure php.ini
Enable required PHP extensions
Update Apache configuration
Verify the installation

6. Upgrading phpMyAdmin

Many bundled XAMPP installations include older versions of phpMyAdmin.

The guide demonstrates:

Installing the latest release
Preserving existing configurations
Migrating settings safely
Verifying successful upgrades

7. Daily Service Management

Instead of opening the XAMPP Control Panel every time, the guide shows how to manage services directly from PowerShell.

This includes:

Starting Apache
Starting MySQL
Opening WordPress
Accessing phpMyAdmin
Stopping services after development

Why Use PowerShell Instead of Only the XAMPP Control Panel?

Learning to manage services through PowerShell provides several long-term advantages.

Faster setup
Easier automation
Better troubleshooting
Repeatable deployments
Improved understanding of Windows service management
Practical DevOps and system administration skills

These skills become increasingly valuable when working with larger development environments or automating deployments.

Who Should Read This Guide?

This guide is suitable for:

Web Developers
WordPress Developers
PHP Developers
Computer Science Students
Information Technology Students
Software Engineering Undergraduates
Lecturers and Instructors
System Administrators
Beginners learning local web development

It is especially useful in academic environments where students need to deploy multiple local development environments using consistent procedures.

GitHub Repository

The complete guide and supporting resources are available on GitHub.

GitHub Repository:

XAMPP-WordPress-PowerShell-Guide

If you find this resource useful, consider giving the repository a star. It helps increase visibility and encourages further improvements.

Also Published On

This article is also available on the following platforms:

LinkedIn: Read on LinkedIn
Medium: Read on Medium
AWS: Read on AWS
DEV Community: Read on DEV Community

Acknowledgements

I would like to express my sincere gratitude to Mr. Madusanka Mithrananda for his invaluable guidance, encouragement, and continuous support throughout the development of this guide and the article. His expertise, constructive feedback, and mentorship were instrumental in shaping this work and deepening my understanding of XAMPP, PowerShell, and WordPress deployment.

I truly appreciate his dedication to teaching, willingness to share knowledge, and the encouragement provided throughout this journey. His guidance has been an invaluable part of the successful completion of this guide.

Final Thoughts

Documentation is one of the most valuable yet overlooked contributions to the developer community. A clear, repeatable guide can save countless hours for students, developers, and educators trying to build reliable development environments.

This project was created with the goal of simplifying the setup process while encouraging developers to understand the underlying configuration instead of simply following GUI-based installation steps.

I hope it serves as a practical reference for anyone working with XAMPP, Apache, MySQL, PHP, PowerShell, and WordPress.

If you have suggestions, discover improvements, or would like to contribute, feel free to open an issue or submit a pull request on GitHub.

Happy coding!

Building a Real-Time Payment Gateway Integration with PHP, PayHere, ngrok, and MySQL

Wed, 09 Sep 2026 09:49:02 +0000

Recently, I built a complete real-time payment gateway integration using PHP, PayHere Sandbox, ngrok, and MySQL to simulate and test online payments in a local development environment.

This project covered:

Secure PayHere checkout integration
Real-time payment callback handling through ngrok
Transaction logging and MySQL database storage
Email notifications using PHPMailer
PDF receipt generation using mPDF

One of the most interesting parts of the project was using ngrok to expose localhost securely, allowing PayHere to communicate with local callback URLs such as notify_url, return_url, and cancel_url.

The project structure was designed to keep the workflow clean and modular, with separate files for checkout processing, notifications, verification, configuration, and database handling.

This implementation was a great hands-on experience in understanding how real-world payment systems work behind the scenes, from hash generation and callback verification to preventing duplicate transactions and securely handling payment confirmations.

Repository

GitHub Repository: https://github.com/NithilaLD/Building-a-Real-Time-Payment-Gateway-Integration-with-PHP-PayHere-ngrok-and-MySQL/

The repository includes the complete project resources, including the full implementation guide in PDF format, source code files for the payment gateway integration, and database setup scripts required to configure and run the application locally.

Tech Stack

Also Published On

This article is also available on the following platforms:

LinkedIn: Read on LinkedIn
Medium: Read on Medium
AWS: Read on AWS
DEV Community: Read on DEV Community

Building database synchronization the boring way

Tue, 08 Sep 2026 21:13:56 +0000

TL;DR: iamroot.tech uses a custom-built, asynchronous MySQL database synchronization system handling ~10 million daily changes across several distributed nodes without consensus algorithms or distributed coordinators.
Key Architecture & Trade-offs:

Primary Keys: UUIDs to allow conflict-free row generation on any node.
Change Tracking: Database triggers log INSERT, UPDATE, and DELETE operations into a simplified change log.
Replication Mechanism: Asynchronous background workers process log entries.
Conflict Avoidance: Application-level single-writer authority per dataset
Fault Tolerance: Nodes catch up automatically after offline periods by replaying the simple change log backlog.

The setup - servers will disappear

iamroot.tech runs across numerous servers in different geographical locations. They are deliberately inexpensive machines, and an important assumption behind the entire architecture is that individual servers will - at some point in time - disappear, underperform or otherwise misbehave.

The goal has never been to make every server perfectly reliable. The goal is to make each server failure completely uninteresting.

That creates an interesting problem for the databases. A server needs access to shared data, but I don't want a write on server A to depend on server B being available. In fact, I don't really want server A to care about server B at all while serving a request.

So I ended up building my own synchronization system. It currently keeps roughly 50–60 tables synchronized across numerous servers and handles around 10 million changes on a normal day.

It isn't particularly clever. In fact, it's about as simple as it gets.

And that is rather the point.

A failing server should create a backlog, not an outage

The most important principle is that database writes are local operations. If the application running on server A needs to update something, it writes to the local MySQL database and continues.

Synchronization happens afterwards.

If server B happens to be slow, offline or has been gone for three days, that doesn't change what happens on A. It simply means B has more catching up to do when it eventually returns.

In practice, this gives iamroot.tech a distributed data architecture without trying to turn MySQL itself into a distributed database. Each server owns its local database and can operate independently, while relevant data is eventually synchronized between them.

Temporary differences between nodes are therefore expected. Eventual consistency isn't something the architecture reluctantly tolerates; it is part of the design.

That trade-off works because the application itself understands how the data is used.

The implementation is almost embarrassingly simple

Tables that need to be shared use UUIDs as their primary keys. That allows records to be created independently on different servers without coordinating ID ranges or worrying about primary key collisions later.

Each server then maintains its own local change log. Synchronized tables have triggers for INSERT, UPDATE and DELETE, and whenever something changes, a small entry containing the table, record ID, operation and timestamp is written to the log.

A background service processes those entries and sends the changes to whichever nodes should receive them. There is no central replication queue and no synchronization master.

For inserts and updates, I use MySQL's INSERT ... ON DUPLICATE KEY UPDATE. If a server somehow missed the original INSERT but later receives an UPDATE, the record simply gets created anyway.

Replicated writes bypass the change-log triggers for that database connection, while normal application writes still trigger them. Otherwise A would update B, B would send the same change back to A, and my wonderfully simple synchronization system would become a wonderfully effective infinite loop.

DELETEs work through the same mechanism. The DELETE entry stays in the change log until the relevant nodes have processed it, effectively making it a temporary replication tombstone. A server can therefore disappear for days and still learn which records were deleted while it was gone.

That's basically it.

Avoiding problems instead of solving them

This is not conflict-safe multi-master replication.

If two servers independently modify the same record at exactly the wrong moment, there is no clever distributed algorithm deciding which version represents universal truth.

Instead, I largely avoid the problem at the application level.

Individual records normally have one logical writer - one server or subsystem that is effectively king for that data. The code working with those records knows about that architecture.

That is an important distinction. I could add version vectors, distributed locking, automatic conflict resolution and considerably more machinery. Those are perfectly valid solutions to problems other systems have.

But if I can structure the application so the problem doesn't normally exist, I would rather do that.

The same philosophy applies elsewhere. Synchronization is at-least-once rather than exactly-once. Duplicate work happens, but the operations are idempotent, so that is mostly harmless. The change log isn't an event store either. If a record changes ten times, I don't need to reconstruct all ten historical versions. I care about getting the current state where it needs to go.

Recovery is deliberately boring

For every table and destination node, the synchronizer remembers how far through the local change log that node has progressed.

Important data is typically synchronized every 10 – 20 seconds, while less important tables may only be processed every ten minutes. If a destination disappears, its position simply stops advancing while everybody else continues.

When it returns, there is normally nothing for me to do. The synchronizer knows where it stopped and continues from there. Once all relevant nodes have moved beyond old change-log entries, those entries can be removed.

This isn't just how I hope it works. Servers have disappeared and returned plenty of times in production.

The recovery procedure is dead simple: sit back and wait.

I like recovery procedures like that.

Why not just use MySQL replication?

MySQL already provides very capable replication, and this isn't an attempt to build a better replacement.

I wanted slightly different properties.

Not every table needs to exist everywhere. Different data can have different synchronization intervals and destinations. More importantly, replication isn't part of the availability model for an individual application server. A slow destination creates work for the synchronizer, not latency for the application serving a user.

There are obvious trade-offs. The synchronizer doesn't automatically understand referential integrity between tables, although I can control their processing order. It doesn't negotiate schemas either; keeping compatible schemas across the relevant servers is my responsibility.

Those are not necessarily features waiting to be implemented. They are boundaries I know about and can design around.

Boring as a feature

Around 10 million changes pass through this setup every day, with the ASN database alone sometimes generating several million updates during a processing run.

Could I make the synchronization system considerably smarter? Absolutely.

But I didn't set out to build a distributed database. I wanted a multitude of servers that could operate independently, share the data they needed, tolerate each other disappearing, and recover without me getting involved.

The synchronization system is simply what fell out of those requirements.

There are plenty of situations where this design would be completely inappropriate. There are also plenty where sophisticated distributed database technology solves very real problems.

But complexity has a real cost.

For iamroot.tech, I'd rather have infrastructure I can explain on a single piece of paper, predict when something breaks, and largely leave alone when a server disappears.

Extremely simple, asynchronous and a tad inefficient in the actual operations, yes - but very efficient at solving the problem, very predictable in the way it works, and - most importantly - it does not require any human intervention at any point.

That is all I need.

Arabic search in WordPress: the matches it silently misses

Fri, 24 Jul 2026 23:41:33 +0000

Two hundred and fifty thousand articles, and the thing editors complained about was the search box. Not the accuracy of it. The wait. Type a name into the newsroom's admin search, go and do something else, come back.

That is the easy complaint, because it arrives. Someone tells you. What never arrives is the other one, and it took me far too long to notice it was missing: nobody has ever told me the search came back empty for a story we definitely published. Nobody ever will. A search that returns nothing does not look like a bug. It looks like proof you never covered it.

So we fixed the loud one, and left the quiet one running for years.

It is the same seam as an earlier piece on Arabic slugs: Arabic quietly breaking an assumption WordPress never knew it was making. Last time it was the URL. This time it is the search box.

What WordPress actually does with an Arabic query

Type a term into WordPress search and WP_Query builds this, near enough:

WHERE post_title   LIKE '%term%'
   OR post_excerpt LIKE '%term%'
   OR post_content LIKE '%term%'

That is the whole mechanism. Not a search engine, not an index, a substring test run three times per row. It knows nothing about words, and nothing at all about Arabic.

We measured what that means on a throwaway WordPress 7.0.2 and MySQL 8.0.35: one post per spelling, then the obvious searches.

you search	you get back
محمد	only the plain post
مُحَمَّد (with tashkeel)	only the tashkeel post
محمـــد (with tatweel)	only the tatweel post
أحمد (hamza alef)	only the hamza post
احمد (bare alef)	only the bare alef post
مصطفى (alef maqsura)	only the maqsura post
مصطفي (yaa)	only the yaa post
مكتبه (haa for taa marbuta)	nothing at all

Read that right-hand column again. Every search returns only its own exact spelling, and the last returns nothing at all, even though every term on the left is the same word to a human being. A reader who types a name without diacritics never sees the article that has them. Someone who writes أحمد never finds احمد. Someone who ends a word with ه instead of ة gets an empty page.

The one result that looks like a success is the exception that proves the point. Searching كتاب does return الكتاب. That is not the search understanding the definite article, it is the letters ك ت ا ب happening to sit inside الكتاب as a literal substring. Search وكتاب, the same word with a prefixed و, and you get nothing, because the bare word is no longer embedded in it.

The admin post list behaves identically to the front end, which is the part that matters, because that is where your editors live.

The same word, written three ways, and a reader means one thing by all of them. WordPress compares raw bytes with LIKE, so the three stay three different strings and none finds the others. Fold them with the same normaliser and they collapse to a single token that matches every time.

The database already knows. WordPress never asks it.

This is the part that genuinely surprised me.

WordPress does not inherit your server's collation. It forces its own onto its tables, utf8mb4_unicode_520_ci, whatever MySQL's default happens to be. And that collation is not naive about Arabic. Ask MySQL directly, with an equality test:

SELECT 'مُحَمَّد' = 'محمد' COLLATE utf8mb4_unicode_520_ci;   -- 1

One. True. The collation folds the diacritics away and says these are the same string. On MySQL 8's utf8mb4_0900_ai_ci it folds tatweel as well.

Now ask it the way WordPress asks it:

SELECT 'مُحَمَّد' LIKE '%محمد%' COLLATE utf8mb4_unicode_520_ci;   -- 0

Zero. Same data, same collation, same server, opposite answer. Wildcard pattern matching does not apply the collation's character folding, so every piece of Arabic knowledge sitting in your database is thrown away the moment you wrap the term in %. We tested four collations. Under =, the good ones fold. Under LIKE '%…%', all four return zero.

The capability was already in the building. The search architecture walks straight past it.

The identical pair, the diacritic-marked spelling and the plain one, under WordPress's own collation. Compared with =, the collation folds the tashkeel away and returns 1: they are equal. Compared with LIKE and a wildcard, that folding is skipped and it returns 0. Same data, same collation, opposite answer, and WordPress search asks with LIKE.

Even reaching it would not be enough

It is tempting to stop here and go looking for a way to make WordPress compare with =. It would not save you, and the measurements show why.

A collation folds accents. Tashkeel are accents, combining marks that sit on top of a letter, which is exactly why they fold.

Hamza is not an accent. أ and ا are different letters. So are ى and ي. So are ة and ه. None of them fold, in any collation we tested, under any comparison. And these are not edge cases in Arabic, they are the ordinary daily variation: the same name spelled أحمد by one editor and احمد by another, the same word ending ى in one headline and ي in the next.

So the database will hand you diacritics for free if you ask it correctly, and it will never hand you the rest. Anything that actually works has to normalise the text itself, deliberately, on both sides of the comparison.

What it costs, and why the broken searches are the expensive ones

The other half of this is the half your editors do complain about.

A LIKE with a leading wildcard cannot use an index. Not "is slow to use one", cannot use one, ever. EXPLAIN on the query WordPress generates says it plainly:

type: ALL     key: NULL     rows: 50013

Full scan, no key, every row. We seeded 50,000 realistic Arabic posts and timed it with a fresh PHP process per run, because repeating it inside one process makes WordPress's in-request cache report a fraction of a millisecond, which is a lie:

a term matching one post: about 215 ms
a term matching nothing: about 225 ms

The search that finds nothing is the slowest, and that is not a rounding artefact. LIKE '%term%' gives up on a row the instant it finds the substring, so a row that matches is cheap: often the title matches and the long body is never read at all. A row that does not match has to be read to the last byte of every field before MySQL can be certain the word is absent. Proving absence is the expensive case. And in Arabic, the searches that come back empty are precisely the variant spellings in the table above.

You might expect LIMIT 10 to rescue this. It cannot, and MySQL says so in plain words. WordPress needs a total for pagination, so it asks for SQL_CALC_FOUND_ROWS, and the query plan reports the consequence itself:

Limit: 10 row(s) (no early end due to SQL_CALC_FOUND_ROWS)

Ten results on the page, every matching row enumerated to produce them.

The scan cost tracks table size, so a 250,000-article archive lands somewhere around a second per search, per concurrent searcher, with no index able to help. (That figure is an extrapolation from the 50k measurement, not a measurement of a 250k table.)

Now put the two halves together. The searches that silently return nothing are the same searches that scan your entire archive in order to return nothing. In Arabic, those are every variant spelling your readers and editors actually type. The complaint you get and the complaint you never get are the same query.

What we built, and what it fixed

We did the obvious right thing, and it worked.

We built a table outside WordPress, on AWS, one row per article: id, title, body, author, categories. Every time a post was published, updated or deleted, that row was kept in step. Search stopped asking the WordPress database to scan a quarter of a million rows and started asking a store built for the job. Public search had already been handed off to a Google Custom Search Engine, so between the two, search stopped being the thing anyone raised in a meeting.

(That table later grew into something much larger, a recommendation engine that learns what a reader is interested in and chooses their next article. That is a story for another day.)

By every measure we were watching, we had fixed search.

What it did not fix

We never normalised the Arabic.

The new table was faster. It was still storing and matching the same raw strings, with the same orthographic variation, in the same literal way. A reader typing محمد still missed مُحَمَّد. أحمد still missed احمد. مكتبه still found nothing. We had rebuilt the entire search path, moved it to another cloud, and put a proper store underneath it, and every one of those searches failed exactly as it had before.

That is the lesson, and it took us years to see it: the speed problem was an infrastructure problem, and the matching problem never was. Scale, indexes and a store designed for querying will fix a scan. None of them will turn two spellings of the same word into the same token. Normalisation is a text problem, and if you do not solve it, it follows you to AWS.

And because nobody ever complains about a search that returns nothing, nothing ever forced the issue. The loud problem got an entire re-architecture. The quiet one just carried on.

The layer we were missing

The fix is small, and it is the same fix wherever your index lives.

Normalise both sides. Keep a normalised copy of the searchable text, index that, and put every incoming query through the identical normaliser before matching. Both halves matter: normalise only the stored text and a query carrying diacritics still misses, normalise only the query and it misses the stored variants.

What the normaliser has to do:

Strip the diacritics. Tashkeel are combining marks (U+064B to U+0652 and friends). Remove them entirely.
Strip tatweel (U+0640). It is decoration and carries no meaning.
Unify the alef forms. أ إ آ ٱ all become ا.
Unify ى to ي, and decide about ة to ه. These are conventions, not truths. Pick one, apply it to both sides, write it down.
Strip the definite article الـ, so الكتاب and كتاب fold to one token. Guard it, so it only comes off when enough of the word is left to still be a word, otherwise ال on the front of a short word gets eaten too. And stop there. It is tempting to strip the single-letter prefixes و ف ب ك ل as well, but those letters begin ordinary words, so بيت would lose its ب and فكرة would become كرة, a genuinely different word. So وكتاب will not fold to كتاب, and that conservative choice is the correct one.
Leave Latin alone. Newsroom copy is full of WordPress, AWS, S3 and half-transliterated terms. A normaliser that mangles those breaks more than it fixes.

Do that and the table from earlier collapses into a single row: every spelling of محمد, however it was typed, becomes one token that matches all the others. And because the normalised copy is matched through an index instead of a leading-wildcard scan, the query that used to take a second stops scanning the table at all.

And none of this needs a cloud. The story above happened to run through a table we put on AWS, but that was our answer to scale, not to matching. On an ordinary single-server WordPress, the same normalised copy in the same database, with the query folded the same way, fixes both the misses and the scan with nothing external at all. That is the layer we were missing, and it would have been missing just as badly on one server as on twenty.

One change, both complaints.

The plugin

The whole fix is one file. It goes in wp-content/mu-plugins/, and in its default mode it needs nothing else: no cloud, no service, no configuration.

composer require mantekio/wp-arabic-search

On save it folds everything searchable about a post, title, excerpt and body, into one normalised column in a shadow table with a real FULLTEXT index, and rewrites WordPress search to match against that column through the identical normaliser. The public site and the wp-admin post list both run through WP_Query, so both are fixed at once. Build the index over what you already have with one command:

wp arabic-search reindex

Media gets the same treatment, which turned out to matter more than expected. An attachment keeps its text in four places: the title, the caption, the alt text and the filename. All four go into the same column, so a picture captioned in Arabic is findable by that caption, and صورة-بيروت.jpg is findable by its own name. WordPress does not search alt text at all, so on that one the plugin is not restoring parity, it is adding something core never had. This changes what the Media Library can find, not what appears anywhere else: the plugin decides which rows match the text, and WordPress still decides which post types and statuses are allowed in the results.

It ships with two modes, because the normaliser is the reusable part and the index is not. The default, index, is the self-contained one just described. The other, normaliser, maintains nothing and rewrites nothing: it exposes the normalising function and fires an action carrying the normalised text on every save, so a pipeline that indexes somewhere else (the AWS table we built, OpenSearch, anything) can fold its stored text and its queries with the same code. That is the guarantee that actually matters, and it is the one we did not have for years: both sides normalised by identical code.

define( 'WPAS_MODE', 'normaliser' );

One thing it deliberately refuses to do. InnoDB does not index words shorter than three characters, which in Arabic covers من, في, ما and عن: real words, typed constantly. An index that has never seen a word cannot find it, so requiring one would return an empty page for a search plain WordPress answers perfectly well. That is exactly the failure this article is about, and it would have been ours. So the plugin drops those tokens rather than requiring them, and where a search contains nothing else it hands the query back to core's LIKE untouched. Search a two-letter word and you get precisely what WordPress would have given you; search two words where only one clears the threshold and the long one still matches. The rule it follows is that where it cannot help, it stands aside rather than answer worse than the search it replaced. There is nothing to configure for this, and no reason to go changing innodb_ft_min_token_size on your server: it needs a MySQL restart and a rebuild of every full-text index on the box, and it buys you nothing here.

Which raises the question any search tool eventually gets asked, usually by an editor who is certain the article exists: why did this not match? Answering it used to mean reading the source, so there is now a command that just tells you.

$ wp arabic-search explain "من المكتبة"
normalised:  من مكتبه

+--------+------+--------------------------------------------------+
| token  | used | reason                                           |
+--------+------+--------------------------------------------------+
| من     | no   | shorter than innodb_ft_min_token_size (3)        |
| مكتبه  | yes  | indexable                                        |
+--------+------+--------------------------------------------------+

matches with: MATCH(search_text) AGAINST ('+مكتبه' IN BOOLEAN MODE)

The folded form, every token with whether it was used and why not, and the exact query that runs. It is the smallest feature in the plugin and the one I would miss most, because "search is broken" and "search is working exactly as designed on a word the index cannot hold" look identical from the outside, and this is the difference between them.

We eventually ran it against the real thing: a quarter of a million Arabic articles, plus half a million media items, on a copy of a live newsroom archive. One search settles the whole argument.

searching مُحَمَّد across 250,000 articles	results	time
WordPress	1	4.8 s
with the plugin	28,238	0.55 s

One. Out of twenty-eight thousand articles that contain that name, plain WordPress found a single one, because only one of them happened to spell it with exactly those diacritics. The other 28,237 were in the database the whole time, and no one searching for that name would ever have seen them.

The speed is the part you would have put in the ticket. The other column is the part nobody was ever going to report.

The rest of the numbers come from a controlled 50,000-post rig, where the match count can be dialled precisely, before and after:

matches, out of 50,000	default WordPress	with the plugin
1	about 215 ms	5 ms
500 (1%)	about 212 ms	10 ms
5,000 (10%)	about 230 ms	24 ms
25,000 (50%)	about 250 ms	115 ms
50,000 (100%)	about 310 ms	about 210 ms

EXPLAIN tells the same story as the clock. Before, every search was the full scan:

type: ALL        key: NULL        rows: 50013

After, the index drives it:

type: fulltext   key: search_ft   rows: 1

We had that bottom row wrong for a while, and the way it was wrong is worth repeating, because it is a trap set for anyone who benchmarks a text search. Our first measurement said the index lost on a term matching everything. It did not. The word we had seeded happened to sit near the start of every document, and since LIKE abandons a row the moment it finds a match, we were timing the scan at its luckiest: first few bytes, done, next row. Move the same word to the end of the same documents, and the result inverts.

a term matching all 50,000 rows	default WordPress	with the plugin
the word sits at the start of every document	140 ms	208 ms
the word sits at the end of every document	300 ms	205 ms

Read the plugin column first: it barely moves, because an index does not care where in the text the word happens to sit. Now read the other one, which doubles. We had benchmarked the scan's single luckiest case and called it a fair fight.

So the honest shape is a crossover, not a caveat. Against a term the scan actually has to read, the index wins across the entire range in the table above. Against a term positioned for the scan's best possible case, the index still wins until roughly 70% of the archive matches. A query that matches seventy per cent of a newsroom archive is not a search, it is a SELECT * with extra steps.

And the case that used to hurt most is now the cheapest. The old worst case was the search that came back empty, because proving absence meant reading every byte of every row. An index that finds nothing has nothing to read.

One honest word on where it is in its life. It is new, and it ships as 0.9.3. That it is already on its fourth release says something worth saying plainly: six rounds of adversarial testing found a bug every single time, and not one of them was found by reading the code.

The worst was in the first release. Search a two-letter Arabic word, من or في, and it returned nothing at all. The check meant to drop words too short for the index counted bytes, and an Arabic letter is two bytes, so every one of those words sailed through and became a required term the index had never held. A plugin whose entire argument is that Arabic search silently returns nothing had shipped a way to make Arabic search silently return nothing. Others followed: a health check that could not detect the condition it existed to detect, and a reindex command that announced success having written not one row.

They are all fixed, and every fix is in the changelog, but the pattern is the point. Each one was invisible from the code and obvious the moment something actually ran it. If this piece has a moral it is that one, and it applies to your search too: the failure that returns nothing looks exactly like the absence of anything to find.

What it has not yet had is months in a live newsroom, so it is out in the open under the GPL, on GitHub and Packagist, being hardened in daylight. The repository is github.com/mantekio/wp-arabic-search. Same loop as the pieces before it: the article explains the thinking, the repository is the plugin, the package is the one-line install, and each points at the other two.

Known limits

Being clear about what a tool does not do is part of the tool.

Single-letter prefixes are deliberately not stripped. Folding the definite article is safe behind a length guard; folding و ف ب ك ل is not, because they also start ordinary words, so it would turn بيت into يت and فكرة into كرة. The cost of that safety is that وكتاب does not fold to كتاب. Real prefix handling needs morphology, not a character map.
Folding is not stemming. It will not connect كتاب to its broken plural كتب, because Arabic plurals rewrite the word rather than append to it. That needs a root dictionary, and that is a different project.
The results are only as complete as the index. Nothing walks your archive when you install it, so run wp arabic-search reindex once. After that each save keeps its own row current, and wp arabic-search status exits non-zero the moment the index falls behind, goes stale, or carries rows for posts that no longer exist. Drift is something you get told about rather than something a reader discovers for you.
It costs disk, inside the database. A second normalised copy of your searchable text is still a copy. On that quarter-million-article archive the table came to about 1.1 GB, roughly 3 KB per article, though the FULLTEXT index on top of it was only 17 MB. Media is nearly free: attachments were two thirds of the rows and a sixteenth of the bytes.
Words the index cannot hold are correct, not fast. A search that falls back gets core's behaviour entirely, including core's speed, because it is core's query. On that same archive a two-letter word took about three seconds, exactly as it always had, while an indexed search of the same corpus took well under one. Nothing is worse than before. Nothing is better either.
Normalisation is lossy on purpose. Folding ة into ه, or ى into ي, will occasionally merge two words that genuinely differ. In an archive search that trade is nearly always worth it, but it is a trade: decide it deliberately and document it, rather than discovering it later.
Past a certain size, use a real search engine. OpenSearch ships an arabic analyser that does normalisation, stemming and stopwords properly, and gives you relevance ranking a LIKE will never have. If the archive is large and search matters, that is the destination. You still need the normaliser, because the same text problem travels with you. Ours did.

A quarter of a million articles, and the only thing anyone ever told us about search was that it was slow. So we made it fast. We built the table, moved the queries to AWS, kept it in sync on every post, the complaints stopped, and for a long time we took that to mean we had finished.

The searches that returned nothing never produced a single complaint, because a search that returns nothing looks exactly like a subject you never covered. The archive was full of articles nobody could find, and the silence around them read as success.

That is the part worth keeping, whatever your stack: complaints tell you about the loud failures and nothing at all about the quiet ones. Someone has to go and look.

That is how we run WordPress on AWS for newsrooms: fix what people report, then go hunting for what they cannot. If your archive is in Arabic and you have never tested what your search does with a hamza, let's talk.

How to Create a Trigger That Prevents an Update

Thu, 11 Jun 2026 05:23:15 +0000

To stop a table update in modern MySQL, you use the SIGNAL statement. This statement allows you to return an error state and a custom message to the application, which immediately aborts the triggering operation and rolls back the uncommitted transaction.

The Code Implementation
SQL
DELIMITER $$

CREATE TRIGGER before_employee_update
BEFORE UPDATE ON employees
FOR EACH ROW
BEGIN

-- Check if the salary is being decreased
IF NEW.salary < OLD.salary THEN
    SIGNAL SQLSTATE '45000' 
    SET MESSAGE_TEXT = 'Error: Employee salary cannot be decreased.';
END IF;

END$$

DELIMITER ;
Key Trigger Mechanics
BEFORE UPDATE ensures the validation happens before any changes are actually written to the disk, allowing for a clean interception.

NEW vs OLD modifiers let you compare the incoming updated value against the existing historical value.

SQLSTATE '45000' is a generic state that represents an "unhandled user-defined exception," making it perfect for custom business rule enforcement.

The Potential Problems
While this approach is highly effective, relying solely on database-level triggers to block updates introduces a few specific challenges in real-world applications.

Silent Application Failures: If your backend code isn't explicitly wrapped in robust try-catch blocks, a trigger-enforced database error can crash your API route or leave end-users looking at a generic, unhelpful 500 error page.

Hidden Business Logic: Moving critical validation rules exclusively into MySQL means your application logic becomes split. Developers reviewing backend code might not realize why an update is failing if the rule lives entirely inside a database trigger.

Performance Overhead: For massive bulk updates, executing conditional checks and firing state signals for every single row (FOR EACH ROW) can noticeably degrade database throughput.

How to Solve These Problems
To handle these issues gracefully, you need a strategy that bridges the gap between your database and your application layer.

Graceful Exception Handling: Always catch the SQL exception in your backend application and map the 45000 custom error message directly to a user-friendly frontend alert.

Dual-Layer Validation: Mirror the trigger logic inside your application code. Use the database trigger as a bulletproof final line of defense, while using application validation to provide immediate, low-latency feedback to users.

Clear Database Documentation: Ensure all custom triggers are heavily documented in your codebase repository so team members maintain full visibility over automated data restrictions.

Sumita
Web Developer

Your Database GUI Shouldn't Need an Account

Wed, 10 Jun 2026 13:52:44 +0000

A critique of the trend toward account-required database GUI clients, arguing that tools holding production credentials should never route data through vendor backends. The post makes the case for local-first design using Tabularis as a concrete example: no signup, secrets stored in the OS keychain, all config as plain files on disk, bring-your-own-key AI, and only one network call (a GitHub version check). The tradeoffs — no sync, no team workspaces, no telemetry — are acknowledged openly. The core argument is that database clients are plumbing and should be durable and vendor-independent, not coupled to a company's runway.

Public key retrieval is not allowed / Needs improvement

Mon, 25 May 2026 04:13:39 +0000

Today, in this article we will discuss the error message "Public key retrieval is not allowed". This error message is indicating that the system is configured to disallow the retrieval of a public key. This is likely a security measure to prevent unauthorized access to sensitive information. It is recommended to check with the administrator or technical support for further assistance.

What does error "Public key retrieval is not allowed" mean? #

The error message "Public key retrieval is not allowed" usually occurs when an application or service is trying to access a public key from the server it is trying to connect to, but the operation is not allowed due to security or access control measures that prevent the retrieval of the public key.

In cryptography, public keys are used for encryption and digital signatures, while private keys are used for decryption and signing. When you attempt to connect to a server, the system must first retrieve the server's public key to establish a secure connection. If the server's public key cannot be retrieved, the system may not be able to establish a secure connection, and the error message "Public key retrieval is not allowed" will be displayed.

The cause of this error can vary. This may happen be due to a firewall, network restriction, or a policy set by the system administrator. To resolve this issue, you may need to contact your system administrator or network administrator to request access to the public key or check the firewall or network configurations.

Why is the Error Generated? #

The "Public key retrieval is not allowed" error message can occur due to various causes. Some of the most common causes are:

Firewall restriction: Your system may not be able to retrieve the server's public key if the connection is being blocked by a firewall. The firewall may be configured to block connections to the server or to block the retrieval of the public key.
For example; you are trying to connect to a server over a secure connection using SSL/TLS, but the firewall is blocking the connection.
Network restriction: The system may not be able to retrieve the server's public key if there are restrictions in place on the network that prevent the retrieval. This could be due to a policy set by the network administrator or due to a misconfiguration of the network.
Say, you try to retrieve the public key of a server that is located in a different country, but the connection is blocked by the network administrator due to security policies. Then you will receives the error message "Public key retrieval is not allowed."
Access control measures: The server may have security measures in place that prevent unauthorized access to the public key. This could be due to a policy set by the system administrator or due to security software installed on the server e.g. you are trying to connect to a server that has security measures in place to prevent unauthorized access to the public key.
```
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;

public class MySQLConnectionExample {

  public static void main(String[] args) {
    try {
      // Load the MySQL Connector/J driver
      Class.forName("com.mysql.cj.jdbc.Driver");

      // Establish a connection to the MySQL server
      Connection conn = DriverManager.getConnection(
        "jdbc:mysql://localhost:3306/test_db?useSSL=true",
        "username",
        "password"
      );

      System.out.println("Connected to the MySQL server successfully!");

      // Close the connection
      conn.close();
    } catch (SQLException | ClassNotFoundException e) {
      System.out.println("An error occurred while connecting to the MySQL server: " + e.getMessage());
    }
  }
}
```
In this example, we first load the MySQL Connector/J driver using the Class.forName method. Then, we use the DriverManager.getConnection method to establish a connection to the MySQL server. The connection URL includes the useSSL=true parameter, which indicates that we want to use SSL/TLS to secure the connection. We also specify the username and password to use when connecting to the server.
Here when the client is unable to retrieve the public key an error occurs, an exception will be thrown and an error message will be displayed.
IP/domain restrictions: The server may be configured to only allow public key retrieval from specific IP addresses or domains. If you are not on the approved list, it may not be able to retrieve the public key.
An example is that you are trying to connect to a server that is configured to only allow public key retrieval from specific IP addresses or domains and your system is not on the approved list, so the will receive the error.
Incorrect server configuration: The server may be misconfigured, which could prevent the retrieval of the public key. This could be due to an error in the server's settings or due to an incorrect configuration of the server's software. For example, you try to connect to a server that is misconfigured, which prevents the retrieval of the public key.
Certificate issues: The server may have an expired or revoked certificate, which could prevent the retrieval of the public key. In this case, the system may not be able to establish a secure connection with the server. For example, you will try to connect to a server that has an expired or revoked certificate, which prevents the retrieval of the public key.

Note It is important to understand the cause of the "Public key retrieval is not allowed" error in order to resolve it effectively. Depending on the cause, you may need to make apply the appropiate solution.

How to Solve the Error? #

The possible solutions for the causes of the "Public key retrieval is not allowed" error message are as follows:

Firewall restriction: To resolve this issue, you can check the firewall configuration and ensure that the connection to the server is not being blocked. You may also need to open specific ports or add an exception to the firewall to allow the connection.
Network restriction: To resolve this issue, you may need to contact the network administrator and request that the restriction be lifted or that the necessary changes be made to the network configuration.

Access control measures: To resolve this issue, you may need to request access to the public key from the system administrator or the security software provider. The system administrator may need to change the security measures in place to allow you to retrieve the public key. Like, the error in previous coding example occurs due to useSSL=true parameter. Therefore, we change it to useSSL=false as done below.

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;

public class MySQLConnectionExample {

  public static void main(String[] args) {
    try {
      // Load the MySQL Connector/J driver
      Class.forName("com.mysql.cj.jdbc.Driver");

      // Establish a connection to the MySQL server
      Connection conn = DriverManager.getConnection(
        "jdbc:mysql://localhost:3306/test_db?useSSL=false",
        "username",
        "password"
      );

      System.out.println("Connected to the MySQL server successfully!");

      // Close the connection
      conn.close();
    } catch (SQLException | ClassNotFoundException e) {
      System.out.println("An error occurred while connecting to the MySQL server: " + e.getMessage());
    }
  }
}

If you change the value of useSSL in the connection string from true to false, then the connection will no longer attempt to use SSL encryption. This change can resolve the "Public key retrieval is not allowed" error if the issue was caused by a problem with the server's SSL certificate.

IP/domain restrictions: To resolve this issue, one may need to check the server configuration and ensure that your IP address or domain is on the approved list. If the IP address or domain is not on the approved list, you may need to request that it be added.
Incorrect server configuration: To resolve this issue, you may need to check the server configuration and make any necessary changes to ensure that the server is configured correctly. You may also need to check the server software and ensure that it is installed and configured correctly.
Certificate issues: To resolve this issue, you may need to check the server certificate and ensure that it is valid and up-to-date. If the certificate is expired or revoked, you may need to request a new certificate from a trusted certificate authority.

The Conclusion

In conclusion, the "Public key retrieval is not allowed" error message indicates that the your are unable to retrieve the server's public key, which is required to establish a secure connection using SSL/TLS. This error can be caused by various factors, including firewall restrictions, network restrictions, access control measures, IP/domain restrictions, incorrect server configuration, and certificate issues. To resolve the issue you may need to check for the cause of the error in your specific situation and apply the suitable solution.

The References

Some resources for reference or further information are given below:

MySQL Connector/J documentation:
https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-using-ssl.html
Java API documentation:
https://docs.oracle.com/en/java/javase/14/docs/api/java.sql/java/sql/Driver.html
MySQL official forums:
https://forums.mysql.com/

I Lost a Client's Database on a $5 VPS. Here's the 12-Line Script That Would Have Saved It.

Wed, 20 May 2026 17:42:54 +0000

It was a WordPress site on a $5 DigitalOcean droplet. Client's small business. Nothing fancy.

I had SSH access. I had root. I had every tool I needed to set up automated backups. I just... didn't. Because it was a small site. Because I'd "get to it later." Because the database was only 40MB and what could go wrong?

What went wrong was a botched plugin update that corrupted the wp_options table. No backup. No snapshot. No dump. Just a client on the phone asking why their site shows a white screen and me trying to explain that their content might be gone.

I rebuilt it from a cached Google version and a Wayback Machine scrape. Took two days. Billed zero.

That was the last time I ran a database without automated backups.

The Script

This is what I use now. Every server, every database, no exceptions.

#!/bin/bash

CHECK="✓"
CROSS="✗"

# --- Configuration ---
DB_USER="root"
DB_PASS="your_password"          # Or use ~/.my.cnf for security
DB_NAME="your_database"          # Change to your DB name
BACKUP_DIR="/var/backups/mysql"
KEEP_DAYS=7                       # Delete backups older than this
DATE=$(date '+%Y-%m-%d_%H-%M-%S')
FILENAME="${DB_NAME}_${DATE}.sql.gz"

# --- Create backup directory if it doesn't exist ---
mkdir -p "$BACKUP_DIR"

# --- Run the backup ---
echo "Backing up database: $DB_NAME..."

if mysqldump -u"$DB_USER" -p"$DB_PASS" "$DB_NAME" | gzip > "${BACKUP_DIR}/${FILENAME}"; then
  echo "$CHECK Backup created: ${BACKUP_DIR}/${FILENAME}"
  echo "$CHECK Size: $(du -sh "${BACKUP_DIR}/${FILENAME}" | cut -f1)"
else
  echo "$CROSS Backup FAILED for $DB_NAME — check credentials and database name"
  exit 1
fi

# --- Delete old backups ---
echo "Removing backups older than $KEEP_DAYS days..."
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +"$KEEP_DAYS" -delete
echo "$CHECK Cleanup complete"

# --- Show current backups ---
echo "Current backups:"
ls -lh "$BACKUP_DIR"/*.sql.gz 2>/dev/null || echo "No backups found"

That's it. mysqldump → gzip → timestamp → auto-cleanup. Runs in under a second on most databases.

What Each Piece Does

mysqldump -u"$DB_USER" -p"$DB_PASS" "$DB_NAME" — dumps every table, every row, every schema definition into a single SQL file. This is the standard MySQL backup tool and it's been reliable for decades.

| gzip — pipes the dump straight into compression without ever writing an uncompressed file to disk. A 200MB database compresses to maybe 15MB. On a cheap VPS with limited storage, this matters.

date '+%Y-%m-%d_%H-%M-%S' — timestamps every backup file. You get filenames like mydb_2026-05-20_03-00-01.sql.gz. When something breaks, you can see exactly which point in time each backup represents.

find ... -mtime +"$KEEP_DAYS" -delete — the line most people forget. Without this, your backup folder grows forever. -mtime +7 means "modified more than 7 days ago." Adjust KEEP_DAYS based on how much disk you have.

Stop Hardcoding Passwords

The script above works, but DB_PASS="your_password" sitting in a plain text file is a liability. In production, use a ~/.my.cnf file instead:

[client]
user=root
password=yourpassword

Then chmod 600 ~/.my.cnf and remove the -u and -p flags from the mysqldump command entirely. MySQL reads credentials from that file automatically. Nobody sees the password in ps aux output, nobody finds it in your bash history.

Schedule It

Open crontab -e and add:

0 3 * * * /home/user/mysql-backup.sh >> /var/log/mysql-backup.log 2>&1

That runs the backup every night at 3 AM and logs the output. If you want it every 6 hours instead:

0 */6 * * * /home/user/mysql-backup.sh >> /var/log/mysql-backup.log 2>&1

If you're not comfortable with cron syntax, I built a free cron job builder that generates the line for you — click the schedule you want, copy the output.

Variations I Actually Use

Back up ALL databases on the server:

mysqldump --all-databases | gzip > "${BACKUP_DIR}/all_dbs_${DATE}.sql.gz"

Copy the backup to a remote server (so it survives a full disk failure):

scp "${BACKUP_DIR}/${FILENAME}" backupuser@remote-server:/backups/

Send yourself an email when the backup runs (or fails):

Pair this with my email alert script and you'll get a notification either way.

The Part Nobody Mentions

Test your restores. A backup you've never restored is a backup that might not work. Once a month, spin up a test database and run:

gunzip < your_backup.sql.gz | mysql -u root -p test_database

If that command works, your backups are real. If it doesn't, you've been collecting dead files.

The full script with the line-by-line breakdown, the ~/.my.cnf setup walkthrough, and three more variations is at:

bashsnippets.xyz/snippets/mysql-database-backup.html

If you're running a database on a VPS without automated backups, take 90 seconds and set this up today. Future you will be grateful.

CAP Theorem Explained: What I Learned Setting Up MySQL Replication

Mon, 11 May 2026 11:37:29 +0000

title: CAP Theorem Explained: What I Learned Setting Up MySQL Replication
published: true

tags: systemdesign, mysql, distributedsystems, beginners

I recently started my journey into system design and data-intensive applications. This experience has completely changed how I look at software — especially around the hard tradeoffs of consistency, availability, and what happens when things go wrong at scale.

Let me start with a scenario that surprised me when I first encountered it:

Imagine you just inserted a record in your app. A millisecond later, a read on a different server returns the old data. How? Welcome to distributed systems.

What is CAP Theorem?

CAP stands for Consistency, Availability, and Partition Tolerance. The theorem states that a distributed system can only guarantee 2 out of these 3 properties at any given time.

Consistency — Every read returns the most recent write, or an error. All nodes in the system see the same data at the same time.

Availability — Every request gets a response (not an error), even if some nodes are down. The system is always up.

Partition Tolerance — The system keeps running even when network communication between nodes is lost or delayed.

The Real Choice: CP vs AP

Here is the most important thing to understand: you cannot actually drop Partition Tolerance.

Network partitions are not a design choice — they are a reality. Hardware fails, packets drop, data centers lose connectivity. Any system that runs across multiple machines will eventually face a partition.

So the real decision every distributed system makes is: when a partition happens, do you stay consistent or stay available?

A Simple Analogy: The Bank Branch

Imagine a bank with its HQ in New York and a branch in Boston. The branch always calls HQ before confirming any transaction.

A customer in Boston deposits $500. Right at that moment, the network between Boston and New York goes down.

The Boston branch now has two options:

Option 1 — CP (Consistency + Partition Tolerance):
The branch refuses to process any transactions until it can reach HQ. The data stays consistent, but the service is unavailable during the outage. If you walk up to the counter, you get turned away.

Option 2 — AP (Availability + Partition Tolerance):
The branch processes the deposit locally and syncs with HQ once the network recovers. The service stays available, but if someone checks the account balance at the New York branch before the sync, they see the old amount — temporarily inconsistent data.

Neither option is wrong. The right choice depends entirely on what your application can tolerate.

Hands-On: MySQL Replication as an AP System

To make this concrete, I built a small master-slave replication setup using Docker to see these tradeoffs in action.

You can find the full setup in this repository.

How it works:

One MySQL container acts as the master — it accepts all writes
A second container acts as the slave — it monitors the master and replays every change
The master records every write to a binary log (binlog)
The slave runs two threads: an IO thread that pulls the binlog from the master, and a SQL thread that replays those events locally

What I observed:

When I inserted rows on the master, they showed up on the slave within milliseconds under normal conditions. But when I inserted 10,000 rows rapidly, I could watch the Seconds_Behind_Master value in SHOW SLAVE STATUS grow and then shrink back to zero as the slave caught up.

That gap — replication lag — is the tradeoff in action. During those seconds, a read on the slave returns stale data. The slave is always available to serve reads, but it cannot guarantee it has the latest write.

This makes the setup an AP system: Available and Partition Tolerant, but not strongly Consistent.

If you wanted CP behaviour, you would switch to synchronous replication — the master waits for the slave to confirm every write before acknowledging it to the client. You get consistency, but every write now pays the latency cost of that round-trip.

When to Choose CP vs AP

Prefer CP when a wrong answer is worse than no answer:

Bank balances and financial transactions
Seat reservation systems (cinema, flights)
Distributed locks and coordination

Prefer AP when brief staleness is acceptable:

Social media feeds
Product catalogs and shopping carts
DNS resolution
Caching layers

Beyond CAP: Consistency is a Spectrum

CAP theorem is a useful mental model, but it is a simplification. In practice, consistency is not binary — there is a whole spectrum:

Linearizability — the strongest guarantee; reads always reflect the latest write
Sequential consistency — operations appear in the same order across all nodes, but may lag
Eventual consistency — all nodes will converge to the same value, given enough time (this is what MySQL async replication gives you)

If you want to go deeper, Designing Data-Intensive Applications by Martin Kleppmann covers replication in Chapter 5 and consistency models in Chapter 9. It is genuinely the best resource I have found on this topic.

Conclusion

CAP is not a theorem you apply once and move on — it is a lens for thinking about tradeoffs. Every distributed system you design makes a CAP choice, whether you make it explicitly or not. Understanding that choice, and being able to justify it, is what separates a system that works in production from one that only works in theory.

From Stream to Database: Processing Market Data with Spring Boot, Redis, and Flyway

Fri, 08 May 2026 20:51:06 +0000

Hey folks!

Continuing the My Broker B3 series, today we build trading-broker-asset — the Java service that consumes the quotes published to Kafka by trading-broker-market-data and transforms them into structured data in MySQL, with high-performance caching in Redis.

If the Python service is the collector, this one is the processor.

️ This Service's Role in the Ecosystem

[trading-broker-market-data]
         │
         │ Kafka: trading-assets-market-data-v1
         ▼
[trading-broker-asset]
         │
    ┌────┴────┐
    ▼         ▼
  MySQL     Redis
(assets)  market:price:{TICKER}
              │
    [b3-matching-engine]   (reads from here)
    [trading-broker-order] (validates ticker from here)

Two critical consumers depend on the data this service maintains: the Matching Engine (for price decisions) and the Order API (to validate whether a ticker exists and is active).

MVP Focus

In this phase, I prioritized:

Robust Kafka consumer with error handling
Asset upsert: creates on first event, updates on subsequent ones
Redis cache with 10-minute TTL
REST API for catalog queries
Status filter: only ACTIVE assets are returned

️ Tech Stack

Technology	Usage
Java 21 + Spring Boot 3.5.11	Service core
Spring Kafka	Quote event consumption
MySQL + Flyway	Versioned asset catalog
Spring Data Redis	Real-time price cache
SpringDoc OpenAPI	Swagger UI documentation

️ Implementation Pillars

1. The Database Schema (Flyway)

CREATE TABLE assets (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    ticker VARCHAR(20) NOT NULL UNIQUE,
    name VARCHAR(200),
    current_price DECIMAL(19, 4),
    last_update DATETIME,
    status ENUM('ACTIVE', 'INACTIVE') NOT NULL DEFAULT 'ACTIVE',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Why Flyway? Schema versioning ensures that any environment (local, CI, production) has exactly the same database structure, without surprises.

2. The Kafka Consumer

@KafkaListener(
        topics = "trading-assets-market-data-v1",
        groupId = "trading-broker-asset"
)
public void consume(AssetMarketDataDTO dto) {
    log.info("Market data received for ticker: {}", dto.getTicker());
    try {
        assetService.updateAsset(dto);
    } catch (Exception e) {
        log.error("Failed to process market data for ticker {}: {}",
                dto.getTicker(), e.getMessage(), e);
        // Rethrow so Kafka can apply retry policy instead of silently discarding
        throw e;
    }
}

Why rethrow the exception? If processing fails silently, the offset is committed and the message is discarded — the asset is never updated. Rethrowing lets Kafka apply its retry policy.

3. The Asset Upsert

@Transactional
public void updateAsset(AssetMarketDataDTO dto) {
    Asset asset = assetRepository.findByTicker(dto.getTicker())
            .orElse(Asset.builder()
                    .ticker(dto.getTicker())
                    .status(AssetStatus.ACTIVE)
                    .build());

    asset.setName(dto.getName());
    asset.setCurrentPrice(dto.getPrice());
    // Uses real market timestamp from event — not LocalDateTime.now()
    asset.setLastUpdate(dto.getUpdatedAt() != null ? dto.getUpdatedAt() : LocalDateTime.now());

    assetRepository.save(asset);

    // Update Redis cache after persisting
    marketPriceCacheService.updatePrice(dto.getTicker(), dto.getPrice());
}

Important decision: we use the real updatedAt from the Kafka event, not LocalDateTime.now(). This ensures lastUpdate in the database reflects when the price was generated by the market, not when it was processed by the service.

4. Redis Cache with TTL

private static final Duration CACHE_TTL = Duration.ofMinutes(10);

public void updatePrice(String ticker, BigDecimal price) {
    String key = "market:price:" + ticker;
    // TTL of 10 minutes — prevents stale prices for inactive or delisted assets
    redisTemplate.opsForValue().set(key, price.toString(), CACHE_TTL);
}

Why TTL? If an asset is deactivated or delisted, its price will expire from Redis in 10 minutes. Without TTL, the Matching Engine could use prices for assets that no longer exist.

5. Status Filter in Queries

public List<AssetDTO> findAllActive() {
    // Query directly by status — not findAll() + filter in memory
    return assetRepository.findAllByStatus(AssetStatus.ACTIVE).stream()
            .map(AssetDTO::fromEntity)
            .toList();
}

public Optional<AssetDTO> findByTicker(String ticker) {
    // Only returns ACTIVE assets — inactive returns 404
    return assetRepository.findByTickerAndStatus(ticker.toUpperCase(), AssetStatus.ACTIVE)
            .map(AssetDTO::fromEntity);
}

When trading-broker-order validates a ticker before creating an order, it calls GET /api/v1/assets/{ticker}. If the asset is inactive, it gets a 404 and the order is rejected. Correct and predictable.

REST API

Method	Endpoint	Description
GET	`/api/v1/assets`	List all active assets
GET	`/api/v1/assets/{ticker}`	Find asset by ticker (404 if inactive)

Swagger UI: http://localhost:8083/swagger-ui.html

✅ Validating the Execution

With the application running and trading-broker-market-data publishing to Kafka:

✅ Consumer connected to the trading-assets-market-data-v1 topic
✅ Assets being created/updated in MySQL in real time
✅ Redis being updated with TTL after each event
✅ Logs showing Updating ticker PETR4 with price R$ 47.37

What's Next?

With the asset catalog working, the next step is b3-market-sync-api — which synchronizes prices directly to the B3's Redis — followed by b3-matching-engine-api, which uses those prices to execute orders.

About the Series

⬅️ Previous Post: Tooling Tips: MongoDB and Kafka

➡️ Next Post: Syncing the Real Market: Consuming Brapi and Feeding Redis with Spring Boot

Series Index: Series Roadmap

Links:

Database Optimization Strategies That Cut Query Execution Time by 60%: A Practical Guide

Wed, 15 Apr 2026 23:39:07 +0000

If your web application is slow, the database is almost certainly the bottleneck. Not the frontend framework. Not the web server. Not the network. The database.

I have spent nine years building backend systems across seven companies, and in every single engagement, database performance has been the most impactful area of optimization. At VacancySoft, where I currently serve as a Full Stack Engineer, I reduced query execution times by 60% across our core platform. At Univelcity, I achieved a 45% reduction in API response times, with database optimization accounting for the majority of those gains. At Lordwin Group, I optimised a real-time gaming backend to sustain 3,000+ concurrent users with sub-100ms query responses.

This article is the practical guide I wish I had when I started. It covers the exact strategies, tools, and techniques I use to diagnose and fix database performance problems in production systems running PostgreSQL and MySQL.

Strategy 1: Query Analysis and Profiling — Know Before You Optimize

The biggest mistake engineers make with database optimization is guessing. They add indexes randomly, rewrite queries based on intuition, or throw caching at the problem without understanding what is actually slow.

I start every optimization effort the same way: profiling.

Enable Slow Query Logging

In PostgreSQL:

-- postgresql.conf
log_min_duration_statement = 200  -- Log queries taking > 200ms
log_statement = 'none'            -- Don't log all queries, just slow ones

In MySQL:

SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 0.2;  -- 200ms threshold
SET GLOBAL log_queries_not_using_indexes = 'ON';

At VacancySoft, I established a weekly ritual of reviewing the slow query log. This alone surfaced dozens of optimization opportunities that no amount of code review would have caught. Queries that looked clean in the ORM were generating catastrophic execution plans.

Use EXPLAIN ANALYZE Religiously

EXPLAIN shows you the query plan. EXPLAIN ANALYZE actually executes the query and shows real timing data. The difference matters.

EXPLAIN ANALYZE
SELECT v.id, v.title, o.name AS organisation_name, s.name AS sector_name
FROM vacancies v
JOIN organisations o ON o.id = v.organisation_id
JOIN sectors s ON s.id = v.sector_id
WHERE v.status = 'active'
  AND v.created_at >= NOW() - INTERVAL '30 days'
ORDER BY v.created_at DESC
LIMIT 20;

Before optimization, this query at VacancySoft returned an EXPLAIN ANALYZE output like this:

Sort  (cost=15234.56..15234.61 rows=20 width=128) (actual time=1842.331..1842.345 rows=20 loops=1)
  Sort Key: v.created_at DESC
  ->  Hash Join  (cost=8234.12..15234.43 rows=20 width=128) (actual time=1201.442..1841.892 rows=24531 loops=1)
        ->  Seq Scan on vacancies v  (cost=0.00..6842.00 rows=24531 width=96) (actual time=0.021..892.134 rows=24531 loops=1)
              Filter: ((status = 'active') AND (created_at >= ...))
              Rows Removed by Filter: 187234
Planning Time: 0.892 ms
Execution Time: 1843.012 ms

Two immediate red flags: a sequential scan on the vacancies table scanning over 200,000 rows and filtering out 187,000 of them, and a hash join that is materialising far more rows than the final result needs.

This is the kind of visibility that transforms optimization from guesswork into engineering. Olamilekan Lamidi's first rule of database optimization: never optimise a query you have not profiled.

Strategy 2: Indexing Strategies — The Highest-Impact Change

Indexing is the single most impactful optimization technique in my toolkit. A well-placed index can turn a 2-second query into a 5-millisecond query. But indexes are not free — they consume storage, slow down writes, and can mislead the query planner if poorly designed.

Composite Indexes

Order matters in composite indexes. The index (status, created_at) is useful for queries that filter on status and sort or filter on created_at. It is useless for queries that filter only on created_at.

For the vacancy search query above, I created:

CREATE INDEX idx_vacancies_status_created_at
ON vacancies (status, created_at DESC);

This single index eliminated the sequential scan. The query planner now uses an index scan that reads only the relevant rows:

Index Scan using idx_vacancies_status_created_at on vacancies v
  (cost=0.42..234.56 rows=20 width=96) (actual time=0.089..2.341 rows=20 loops=1)
  Index Cond: ((status = 'active') AND (created_at >= ...))

Execution time: from 1,843ms to 12ms. A 99.3% reduction from a single composite index.

Partial Indexes

At VacancySoft, the majority of our queries filter on status = 'active'. Only about 15% of vacancies are active at any given time. A partial index dramatically reduces the index size:

CREATE INDEX idx_vacancies_active_created_at
ON vacancies (created_at DESC)
WHERE status = 'active';

This index is roughly 85% smaller than the full composite index, which means it fits in memory more easily and produces faster index scans.

Covering Indexes (Index-Only Scans)

A covering index includes all columns that a query needs, allowing PostgreSQL to serve the query entirely from the index without touching the heap (the actual table data). This eliminates random I/O, which is the most expensive operation on traditional storage.

CREATE INDEX idx_vacancies_covering
ON vacancies (status, created_at DESC)
INCLUDE (id, title, organisation_id, sector_id)
WHERE status = 'active';

With this covering index, PostgreSQL executes an index-only scan:

Index Only Scan using idx_vacancies_covering on vacancies v
  (cost=0.42..45.23 rows=20 width=96) (actual time=0.067..0.412 rows=20 loops=1)
  Heap Fetches: 0

Heap Fetches: 0 is the goal. The database never touches the table at all.

My Indexing Methodology

I follow a systematic process:

Identify the top 20 slowest queries from the slow query log.
Run EXPLAIN ANALYZE on each one.
Look for sequential scans on tables with more than 10,000 rows.
Check for sort operations that are not backed by an index.
Create targeted indexes and verify the plan changes.
Monitor write performance to ensure indexes are not degrading INSERT/UPDATE speed beyond acceptable thresholds.

At VacancySoft, this methodology across 15+ modules produced the 60% aggregate reduction in query execution time.

Strategy 3: Query Refactoring — Fixing What the ORM Gets Wrong

ORMs are excellent for productivity but frequently generate suboptimal SQL. The N+1 query problem is the most well-known example, but I encounter more subtle issues regularly.

Eliminating N+1 Queries

In Laravel (which I used extensively at 2am Tech, Lordwin Group, and Viaduct):

// Bad: N+1 problem — 1 query for organisations + N queries for vacancies
$organisations = Organisation::all();
foreach ($organisations as $org) {
    echo $org->vacancies->count(); // Triggers a query per organisation
}

// Good: Eager loading — 2 queries total
$organisations = Organisation::with('vacancies')->get();

// Better: When you only need the count
$organisations = Organisation::withCount('vacancies')->get();
foreach ($organisations as $org) {
    echo $org->vacancies_count; // No additional queries
}

At 2am Tech, the Addio financial platform was suffering from response times exceeding 3 seconds on the organisation dashboard. Profiling revealed over 150 queries per page load, almost all of them N+1 queries from Eloquent relationships. I refactored the critical paths to use eager loading and withCount, reducing the query count to 8 and the response time to 400ms.

Replacing Subqueries with JOINs and CTEs

ORMs often generate correlated subqueries where a JOIN or CTE would be far more efficient:

-- ORM-generated: correlated subquery (executes once per row)
SELECT o.id, o.name,
  (SELECT COUNT(*) FROM vacancies v WHERE v.organisation_id = o.id AND v.status = 'active')
    AS active_vacancy_count
FROM organisations o
WHERE o.sector_id = 5;

-- Refactored: single pass with JOIN
SELECT o.id, o.name, COUNT(v.id) AS active_vacancy_count
FROM organisations o
LEFT JOIN vacancies v ON v.organisation_id = o.id AND v.status = 'active'
WHERE o.sector_id = 5
GROUP BY o.id, o.name;

Window Functions for Complex Analytics

At VacancySoft, we needed to show month-over-month vacancy trends per sector. The initial implementation used multiple queries with application-level aggregation. I replaced it with a single query using window functions:

WITH monthly_counts AS (
  SELECT
    s.name AS sector_name,
    DATE_TRUNC('month', v.created_at) AS month,
    COUNT(*) AS vacancy_count
  FROM vacancies v
  JOIN sectors s ON s.id = v.sector_id
  WHERE v.created_at >= NOW() - INTERVAL '12 months'
  GROUP BY s.name, DATE_TRUNC('month', v.created_at)
)
SELECT
  sector_name,
  month,
  vacancy_count,
  LAG(vacancy_count) OVER (PARTITION BY sector_name ORDER BY month) AS prev_month_count,
  ROUND(
    (vacancy_count - LAG(vacancy_count) OVER (PARTITION BY sector_name ORDER BY month))::numeric
    / NULLIF(LAG(vacancy_count) OVER (PARTITION BY sector_name ORDER BY month), 0) * 100,
    1
  ) AS month_over_month_pct
FROM monthly_counts
ORDER BY sector_name, month;

This single query replaced five application-level queries and a significant amount of JavaScript aggregation logic. Response time for the trends endpoint dropped from 2.1 seconds to 280ms.

Strategy 4: Schema Redesign for Performance

Sometimes the problem is not the query — it is the schema. Normalization is taught as gospel in database courses, but production systems often require strategic denormalization.

Materialized Views

At VacancySoft, our analytics dashboard required aggregating data across five tables. Even with optimized queries and proper indexes, the aggregation was too slow for real-time display. I introduced materialized views:

CREATE MATERIALIZED VIEW mv_sector_analytics AS
SELECT
  s.id AS sector_id,
  s.name AS sector_name,
  COUNT(v.id) AS total_vacancies,
  COUNT(v.id) FILTER (WHERE v.status = 'active') AS active_vacancies,
  COUNT(DISTINCT v.organisation_id) AS hiring_organisations,
  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY v.salary_max) AS median_salary,
  DATE_TRUNC('day', NOW()) AS snapshot_date
FROM sectors s
LEFT JOIN vacancies v ON v.sector_id = s.id
  AND v.created_at >= NOW() - INTERVAL '90 days'
GROUP BY s.id, s.name;

CREATE UNIQUE INDEX ON mv_sector_analytics (sector_id);

-- Refresh concurrently (no lock on reads)
REFRESH MATERIALIZED VIEW CONCURRENTLY mv_sector_analytics;

I set up a cron job to refresh this view every 15 minutes. The dashboard query that previously took 3.8 seconds now completes in under 10ms because it reads from the pre-computed view.

Strategic Denormalization

For the organisation profile pages, rather than joining five tables on every request, I added a cached_stats JSONB column on the organisations table:

ALTER TABLE organisations ADD COLUMN cached_stats JSONB DEFAULT '{}';

-- Updated by a background job every hour
UPDATE organisations SET cached_stats = jsonb_build_object(
  'active_vacancies', (SELECT COUNT(*) FROM vacancies WHERE organisation_id = organisations.id AND status = 'active'),
  'total_vacancies_90d', (SELECT COUNT(*) FROM vacancies WHERE organisation_id = organisations.id AND created_at >= NOW() - INTERVAL '90 days'),
  'avg_time_to_fill', (SELECT AVG(EXTRACT(EPOCH FROM (filled_at - created_at)) / 86400) FROM vacancies WHERE organisation_id = organisations.id AND filled_at IS NOT NULL),
  'updated_at', NOW()
);

This is a tradeoff: the data can be up to one hour stale, and we have the overhead of the background update job. But for this use case, slight staleness is acceptable, and the performance gain — from 450ms to 8ms per profile page — is transformative.

Strategy 5: Connection Pooling and Read Replicas

At scale, query optimization alone is not enough. You also need to manage how connections to the database are handled.

Connection Pooling with PgBouncer

PostgreSQL creates a new process for each connection, which is expensive. At VacancySoft, I deployed PgBouncer in front of PostgreSQL in transaction pooling mode:

; pgbouncer.ini
[databases]
vacancysoft = host=127.0.0.1 port=5432 dbname=vacancysoft

[pgbouncer]
listen_port = 6432
pool_mode = transaction
max_client_conn = 500
default_pool_size = 25
reserve_pool_size = 5

This allows 500 application connections to share 25 database connections. Without pooling, our application would regularly hit PostgreSQL's connection limit during traffic spikes, causing cascading failures.

Read Replicas

For read-heavy workloads (which describe most web applications), I split reads and writes across primary and replica databases:

// Database configuration with read/write splitting
const knex = require('knex');

const primary = knex({
  client: 'pg',
  connection: process.env.DATABASE_PRIMARY_URL,
});

const replica = knex({
  client: 'pg',
  connection: process.env.DATABASE_REPLICA_URL,
});

// Service layer decides which connection to use
class VacancyService {
  async search(filters) {
    return replica('vacancies')
      .where(filters)
      .orderBy('created_at', 'desc');
  }

  async create(data) {
    return primary('vacancies')
      .insert(data)
      .returning('*');
  }
}

At VacancySoft, this split reduced load on the primary database by approximately 70%, since the vast majority of our traffic is read operations (search, analytics, reporting).

Strategy 6: Caching Layers

The fastest database query is the one you never make. Caching is the final layer of my optimization strategy.

Redis for Hot Data

const Redis = require('ioredis');
const redis = new Redis(process.env.REDIS_URL);

async function getSectorAnalytics(sectorId) {
  const cacheKey = `sector:${sectorId}:analytics`;
  const cached = await redis.get(cacheKey);

  if (cached) {
    return JSON.parse(cached);
  }

  const data = await db('mv_sector_analytics')
    .where('sector_id', sectorId)
    .first();

  await redis.setex(cacheKey, 900, JSON.stringify(data)); // 15-minute TTL

  return data;
}

Cache Invalidation Strategy

Cache invalidation is famously one of the two hard problems in computer science. My approach is pragmatic:

TTL-based expiration for data that can tolerate staleness (analytics, aggregations).
Event-driven invalidation for data that must be fresh (user profiles, active listings). When a vacancy is created or updated, I publish an event that invalidates the relevant cache keys.
Cache warming for predictable access patterns. At VacancySoft, I pre-warm the cache for the top 50 sectors and top 100 organisations every 15 minutes, so the first user to access these pages never experiences a cache miss.

// Event-driven cache invalidation
eventBus.on('vacancy:created', async (vacancy) => {
  await redis.del(`sector:${vacancy.sector_id}:analytics`);
  await redis.del(`org:${vacancy.organisation_id}:stats`);
  await redis.del('homepage:featured_vacancies');
});

Results: Before and After

Here is a summary of the measurable impact these strategies produced across my career, as implemented by Olamilekan Lamidi:

Company	Metric	Before	After	Improvement
VacancySoft	Average query execution time	850ms	340ms	60% reduction
VacancySoft	Dashboard load time	3.8s	0.4s	89% reduction
VacancySoft	p95 API response time	1.2s	180ms	85% reduction
Univelcity	API response time (avg)	620ms	340ms	45% reduction
Univelcity	Queries per page load	45	8	82% reduction
Lordwin Group	Real-time query latency	220ms	35ms	84% reduction
2am Tech	Organisation dashboard load	3.1s	400ms	87% reduction

Key Takeaways

Profile before you optimize. EXPLAIN ANALYZE and slow query logs should be your starting point, not your intuition.
Composite and partial indexes are the highest-leverage optimization in most applications. One well-placed index can eliminate 99% of query time.
ORMs are productivity tools, not performance tools. Review the SQL they generate, especially for complex queries.
Materialized views and strategic denormalization are not anti-patterns — they are essential tools for read-heavy workloads.
Connection pooling is mandatory once you are handling more than a few hundred concurrent connections.
Caching is the last layer, not the first. Optimize the underlying queries before adding cache complexity.

Database optimization is not a one-time project. It is an ongoing discipline. The queries that perform well today may become bottlenecks tomorrow as data grows and access patterns change. Build profiling and monitoring into your workflow, and treat database performance as a first-class engineering concern.

My SQL Mistakes Almost Broke Me… But I Learned a Lot

Thu, 16 Oct 2025 21:45:41 +0000

Introduction

Let me be honest... when I first started working with SQL, I made a LOT of mistakes. From messing up queries to accidentally deleting test data, it was exhausting. But each error taught me something valuable about coding, debugging, and patience.

My Biggest SQL Oops Moments

1️⃣ The Missing WHERE Clause
I once ran a DELETE query without a WHERE condition… and yes, I wiped out an entire table. Lesson: Always double-check before hitting execute.

2️⃣ Incorrect Joins
I joined two tables and got a result set that made no sense. After some digging, I realized I was joining on the wrong column. Lesson: Know your schema before writing joins.

3️⃣ Data Type Confusion
I tried comparing numbers stored as strings and got unexpected results. Lesson: Always know your data types.

4️⃣ Installation Woes
At first, I was super excited to install MySQL manually in VS Code. But I hit one stubborn error after another. Even ChatGPT couldn’t save me! Lesson: Tech can be frustrating — patience is key.

What I Learned

SQL mistakes aren’t failures .. they’re learning opportunities.
Test your queries on small datasets first.
Keep backups or work in safe environments.
Document what works and what doesn’t ..it saves future headaches.
Tech journeys are full of errors, tears, and breakthroughs ..and that’s okay!

Advice to Beginners

If you’re new to SQL or databases, know that everyone struggles at first. Don’t give up. Mistakes are part of learning. Patience, persistence, and community support are your best friends.

Let’s Connect

I’d love to hear your SQL horror stories or tips for avoiding mistakes! Share them with me on:
dev.to • GitHub • LinkedIn • X/Twitter

Signature
The Duchess of Hackers
Full-Stack Developer | Digital Marketer

What is SQL? A Beginner's Complete Guide

Thu, 10 Jul 2025 10:56:55 +0000

Follow Me for More Content!

Before we dive into SQL, let's connect! Follow me on these platforms for more programming tutorials and tech insights:

GitHub: Abdelhakim-Baalla - Check out my projects and code
LinkedIn: abdelhakimbaalla - Professional updates and networking
Twitter (X): @Abdelhakim99891 - Quick tips and tech discussions
Portfolio: abdelhakim-baalla

Introduction

Have you ever wondered how websites store and retrieve information about users, products, or orders? Or how apps like Instagram know which photos to show you? The answer lies in databases, and the key to communicating with these databases is SQL.

If you're completely new to programming or databases, don't worry! SQL is actually one of the most beginner-friendly programming languages you can learn. In this comprehensive guide, we'll explore what SQL is, why it's so important, and how you can start using it today.

By the end of this post, you'll have a solid understanding of SQL fundamentals and be ready to start your database journey. Let's dive in!

What is SQL?

SQL stands for Structured Query Language (pronounced "S-Q-L" or "sequel"). Think of it as a special language that allows you to communicate with databases. Just like you use English to talk to people, you use SQL to talk to databases.

Breaking Down the Name

Structured: SQL follows specific rules and formats
Query: You ask questions (queries) to get information
Language: It's a programming language with its own syntax

Simple Analogy

Imagine a database as a massive digital filing cabinet with thousands of folders and documents. SQL is like having a super-efficient assistant who can:

Find any document you need in seconds
Organize documents in any order you want
Add new documents to the right folders
Update information in existing documents
Remove documents you no longer need

Why is SQL Important?

1. Universal Language for Databases

SQL is the standard language for working with databases. Whether you're using MySQL, PostgreSQL, SQL Server, or Oracle, the basic SQL concepts remain the same.

2. High Demand in the Job Market

According to recent surveys, SQL consistently ranks among the top 5 most in-demand programming skills. Jobs requiring SQL skills often offer competitive salaries.

3. Foundation for Data Analysis

If you're interested in data science, business analytics, or any field involving data, SQL is your starting point.

4. Easy to Learn

Unlike complex programming languages, SQL uses English-like words and phrases, making it incredibly beginner-friendly.

How Databases Work (The Basics)

Before we dive deeper into SQL, let's understand what databases are and how they're organized.

What is a Database?

A database is a structured collection of information stored electronically. Think of it like a digital spreadsheet, but much more powerful and organized.

Database Structure

Databases are organized in a hierarchy:

Database
├── Tables (like spreadsheet tabs)
│   ├── Rows (individual records)
│   └── Columns (categories of information)

Real-World Example

Let's say you're building an online bookstore. Your database might have these tables:

Books Table: Contains information about each book
Customers Table: Stores customer information
Orders Table: Tracks customer purchases
Authors Table: Information about book authors

Basic SQL Operations (CRUD)

SQL operations are often referred to as CRUD operations:

Create - Add new data
Read - Retrieve existing data
Update - Modify existing data
Delete - Remove data

1. Creating Data (INSERT)

Adding new information to a database:

-- Adding a new book to our bookstore
INSERT INTO books (title, author, price, genre)
VALUES ('The Great Gatsby', 'F. Scott Fitzgerald', 12.99, 'Fiction');

2. Reading Data (SELECT)

Retrieving information from a database:

-- Get all books
SELECT * FROM books;

-- Get only book titles and prices
SELECT title, price FROM books;

-- Get books under $15
SELECT * FROM books WHERE price < 15.00;

3. Updating Data (UPDATE)

Modifying existing information:

-- Update the price of a specific book
UPDATE books 
SET price = 14.99 
WHERE title = 'The Great Gatsby';

4. Deleting Data (DELETE)

Removing information:

-- Remove a book from the database
DELETE FROM books 
WHERE title = 'The Great Gatsby';

Key SQL Concepts Every Beginner Should Know

1. Tables

Tables are the building blocks of databases. Each table represents a specific type of entity (like books, customers, or orders).

2. Primary Keys

A unique identifier for each row in a table. Like a social security number for database records.

3. Relationships

Tables can be connected to each other. For example, an order table might reference a customer table.

4. Indexes

Special structures that make searching faster. Like the index in a book that helps you find topics quickly.

Getting Started with SQL

Step 1: Choose a Database System

For beginners, I recommend starting with:

SQLite: Simple, file-based database perfect for learning
MySQL: Popular, free, and widely used
PostgreSQL: Powerful and feature-rich

Step 2: Install and Set Up

Most database systems offer free community editions. You can also use online SQL playgrounds like:

DB Fiddle
SQLiteOnline
W3Schools SQL Tryit Editor

Step 3: Practice with Sample Data

Start with simple examples:

-- Create a simple table
CREATE TABLE students (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    age INTEGER,
    grade TEXT
);

-- Add some data
INSERT INTO students (name, age, grade) VALUES
('Alice Johnson', 20, 'A'),
('Bob Smith', 19, 'B'),
('Carol Davis', 21, 'A');

-- Query the data
SELECT name, grade FROM students WHERE grade = 'A';

Common SQL Commands Reference

Data Retrieval

-- Basic selection
SELECT column1, column2 FROM table_name;

-- With conditions
SELECT * FROM table_name WHERE condition;

-- Sorting results
SELECT * FROM table_name ORDER BY column_name;

-- Limiting results
SELECT * FROM table_name LIMIT 10;

Data Modification

-- Insert new record
INSERT INTO table_name (col1, col2) VALUES (value1, value2);

-- Update existing record
UPDATE table_name SET column1 = value1 WHERE condition;

-- Delete record
DELETE FROM table_name WHERE condition;

Real-World Applications of SQL

1. E-commerce Websites

Track product inventory
Process customer orders
Manage user accounts
Generate sales reports

Store user profiles
Manage posts and comments
Track likes and shares
Analyze user behavior

3. Banking Systems

Process transactions
Maintain account balances
Generate financial reports
Ensure data security

4. Healthcare Systems

Store patient records
Track medical appointments
Manage prescription data
Generate health reports

Best Practices for SQL Beginners

1. Use Meaningful Names

-- Good
SELECT customer_name, order_date FROM orders;

-- Avoid
SELECT c, d FROM o;

2. Always Use WHERE Clauses Carefully

-- Always test your WHERE conditions
SELECT * FROM customers WHERE customer_id = 123;

3. Start Simple

Begin with basic queries before moving to complex joins and subqueries.

4. Practice Regularly

Like any language, SQL improves with consistent practice.

Common Mistakes to Avoid

1. Forgetting WHERE Clauses

-- This deletes ALL records!
DELETE FROM customers;

-- This deletes specific records
DELETE FROM customers WHERE customer_id = 123;

2. Not Backing Up Data

Always backup your database before making major changes.

3. **Using SELECT* in Production**

Instead of selecting all columns, specify only the columns you need.

4. Ignoring Case Sensitivity

Some databases are case-sensitive. Be consistent with your naming.

Next Steps in Your SQL Journey

1. Advanced Topics to Explore

Joins (connecting multiple tables)
Indexes (improving query performance)
Stored procedures (reusable code blocks)
Triggers (automatic actions)

2. Practice Resources

HackerRank SQL challenges
LeetCode database problems
SQLBolt interactive tutorial
Codecademy SQL course

3. Build Projects

Create a personal expense tracker
Build a simple library management system
Design a blog database structure

Database design principles
Data modeling
Business intelligence tools
Cloud database services (AWS RDS, Google Cloud SQL)

Tools and Resources

Free Learning Resources

W3Schools SQL Tutorial: Comprehensive and beginner-friendly
SQLBolt: Interactive SQL lessons
freeCodeCamp: Free SQL course on YouTube
Khan Academy: Database basics

Practice Platforms

HackerRank: SQL challenges and competitions
LeetCode: Database problems
SQLZoo: Interactive SQL tutorials
DB Fiddle: Online SQL playground

Books for Deeper Learning

"Learning SQL" by Alan Beaulieu
"SQL in 10 Minutes, Sams Teach Yourself" by Ben Forta
"Head First SQL" by Lynn Beighley

Conclusion

SQL is your gateway to the world of databases and data management. What makes SQL special is its simplicity and power – you can start writing useful queries within hours of learning the basics.

Remember these key points:

SQL is the universal language for databases
It uses English-like syntax that's easy to learn
The four basic operations (CRUD) form the foundation
Practice with real data to build confidence
Start simple and gradually tackle more complex concepts

The journey of learning SQL is incredibly rewarding. Whether you're interested in web development, data analysis, or just want to add a valuable skill to your toolkit, SQL is an excellent investment in your future.

Don't let the technical aspects intimidate you – every SQL expert started exactly where you are now. The key is to start practicing with simple examples and gradually build your skills.

Ready to start your SQL journey? Pick one of the online SQL playgrounds mentioned above and try creating your first table today. You'll be amazed at how quickly you progress!

Stay Connected for More Learning Content!

If you found this guide helpful, I'd love to have you join my community of learners! Follow me on these platforms for more programming tutorials, career advice, and tech insights:

Don't Miss These!

GitHub: Abdelhakim-Baalla - Browse my code and projects
LinkedIn: abdelhakimbaalla - Professional insights and networking
Twitter (X): @Abdelhakim99891 - Daily tech tips and discussions
Instagram: @abdelhakim.baalla - Behind-the-scenes content
Dev.to: abdelhakim_baalla - In-depth tutorials
Medium: @abdelhakimbaalla50 - Detailed articles
Reddit: Abdelhakim-Baalla430 - Community discussions
Portfolio: abdelhakim-baalla.vercel.app - My work showcase
Substack: @abdelhakimbaalla - Newsletter content
Hashnode: abdelhakimbaalla.hashnode.dev - Technical blog
YouTube: @Dinocrafting - Video tutorials
Stack Overflow: Profile - Q&A and help

Your support means the world to me! Follow for more beginner-friendly programming content, project tutorials, and career guidance.

Automating MySQL Backups on a VPS with Telegram Alerts Part 2

Sat, 22 Mar 2025 22:01:43 +0000

Backing up MySQL databases is essential for data security and disaster recovery. In this article, we continue from our previous post and provide a script to automate MySQL database backups from a VPS to a local machine. This script ensures your backups are safe and up-to-date.

Prerequisites

A Debian-based VPS
MySQL installed and running
SSH access to your VPS
A local machine with enough storage for backups

MySQL Backup Script

The following script automates MySQL database backups by securely transferring them from a VPS to a local folder. It also sends notifications via Telegram when a backup succeeds or fails.

Script Overview

Uses rsync to transfer backup files from the VPS.
Logs all actions for debugging purposes.
Sends a Telegram notification on success or failure.
Cleans up old backup files on the VPS after a successful transfer.

Backup Script

#!/bin/bash
# MySQL Backup Script for VPS
# Author: Gift Balogun

# Configuration
REMOTE_HOST="your_vps_ip"
REMOTE_USER="root"
REMOTE_PORT=22
LOCAL_PATH="/WebsiteBackup"
REMOTE_PATH="/root/backup/mysql"
LOG_FILE="/var/log/mysql_backup.log"

# Telegram Notification
TELEGRAM_BOT_TOKEN="your_bot_token"
TELEGRAM_CHAT_ID="your_chat_id"

# Logging function
log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}

log "Starting MySQL backup from $REMOTE_HOST..."

# Ensure host key is added
ssh-keyscan -p $REMOTE_PORT $REMOTE_HOST >> ~/.ssh/known_hosts 2>/dev/null

# Perform backup
rsync -e "ssh -o StrictHostKeyChecking=no -p $REMOTE_PORT" -avzP "$REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH" "$LOCAL_PATH" 2>> "$LOG_FILE"

if [ $? -eq 0 ]; then
    log "Backup completed successfully."
    curl -s -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage" \
         -d chat_id="$TELEGRAM_CHAT_ID" \
         -d text="Backup completed successfully from $REMOTE_HOST to $LOCAL_PATH." >> "$LOG_FILE"
    
    # Cleanup remote backup folder
    ssh "$REMOTE_USER@$REMOTE_HOST" -p "$REMOTE_PORT" "rm -rf $REMOTE_PATH/*" 2>> "$LOG_FILE"
    log "Remote files deleted successfully."
else
    log "Backup failed. Check logs for details."
    curl -s -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendDocument" \
         -F chat_id="$TELEGRAM_CHAT_ID" \
         -F document=@"$LOG_FILE"
fi

Explanation

SSH and rsync Setup:
- The script ensures that the VPS host key is added to known_hosts to prevent SSH authentication issues.
- rsync securely transfers MySQL backups from the VPS to the local machine.
Logging and Notifications:
- The script logs each action to /var/log/mysql_backup.log.
- If the backup succeeds, a success message is sent via Telegram.
- If it fails, the log file is sent to Telegram for troubleshooting.
Cleanup:
- After a successful backup, the remote backup folder is cleared to save space.

Running the Script

Save the script as backup.sh.
Make it executable: chmod +x backup.sh
Run it manually: ./backup.sh
Schedule it with cron: crontab -e
```
0 2 * * * /path/to/backup.sh
```
This runs the script every day at 2 AM.

Conclusion

This script simplifies MySQL database backups and ensures data safety. You can find the full script on GitHub:
https://github.com/giftbalogun/localmysqlbackup

If you haven't read Part 1, check it out here:
https://coderlegion.com/1487/automate-your-vps-database-backups-like-a-pro-a-complete-mysql-backup-script

Database Interaction using Python: Introduction to SQLite/MySQL/PostgreSQL

Wed, 19 Mar 2025 11:56:30 +0000

The article covers POSTGRESQL, MySQL, and SQLITE3—three major database management systems. They provide unique features and benefits and are essential to software development. Let's explore their features so that you can make better selections that will improve the project. Together, let's explore these essential elements in modern software engineering.

As SQLite is lightweight, it is a great choice for embedded applications and situations where simplicity is essential. On the other hand, MySQL stands out thanks to its strong client-server design, scalability, and widespread industry acceptance. PostgreSQL, on the other hand, is a popular option for enterprise-level applications due to its versatility, dependability, and compliance with ACID short for Atomicity, Consistency, Isolation, and Durability.

SQLite Database

SQLite is a relational database engine that is lightweight, serverless, self-contained, and requires no configuration. Because it works directly with files, it's perfect for low-resource applications. Distribution and deployment are made easier by the fact that its self-contained databases are kept as single files.

Features:

Embedded in applications: SQLite, which is frequently integrated into programmes, allows internal data administration without the need for external servers. Data management is made easier by this, especially for offline functioning.
File-based database: SQLite databases are manageable and portable since they are kept as individual files on the file system.
Appropriate for small to medium-sized applications: Because of its simplicity and lightweight design, SQLite is especially well-suited for small to medium-sized applications, even though it can manage huge datasets as well.

Let's go through an example of defining tables, installing SQLite in Python, and carrying out CRUD (Create, Read, Update, Delete) operations:

conn = sqlite3.connect('example.db')

cursor = conn.cursor()

cursor.execute('''CREATE TABLE IF NOT EXISTS users

                  (id INTEGER PRIMARY KEY, name TEXT, email TEXT)''')

cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ('Alice', '*Emails are not allowed*'))

cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ('Ali', '*Emails are not allowed*'))

print("Records after initial insertion:")

cursor.execute("SELECT DISTINCT * FROM users")

rows = cursor.fetchall()

for row in rows:

    print(row)

cursor.execute("UPDATE users SET email = ? WHERE name = ?", ('*Emails are not allowed*', 'Alice'))

print("\nRecords after updating email of Alice:")

cursor.execute("SELECT DISTINCT * FROM users")

rows = cursor.fetchall()

for row in rows:

    print(row)

cursor.execute("DELETE FROM users WHERE name = ?", ('Alice',))

print("\nRecords after deleting Alice:")

cursor.execute("SELECT DISTINCT * FROM users")

rows = cursor.fetchall()

for row in rows:

    print(row)

Output:

Tip: Make sure the 'DISTINCT' keyword is applied to the right columns in SQL queries to obtain unique value combinations and prevent accidental duplicate results.

MySQL Database

The popular open-source RDBMS MySQL provides strong features and dependability for systems ranging from small-scale initiatives to enterprise-level setups. It offers a wide range of tools and features for effective structured data management.

Features:

Client-server architecture:MySQL has a client-server design, which enables several clients to connect to the database server at once. The management of database resources, scalability, and security are all improved by this architecture.
Performance and scalability: MySQL is designed to be scalable, meaning it can manage big databases and high transaction volumes while maintaining optimal performance. Performance is improved by its effective caching, query optimisation, and indexing methods.
Broad industrial adoption: MySQL is widely used in a number of industries, including telecommunications, e-commerce, technology, and finance. Its widespread community support, dependability, and open-source nature have all added to its appeal.

You can get the MySQL Community Server package from mysql official website and install MySQL by following the given steps. As an alternative, you can use package managers such as Homebrew for macOS or apt-get for Linux.

Connecting to MySQL from Python:

import mysql.connector

conn = mysql.connector.connect(user='user', password='password',

                               host='localhost',

                               database='test')

cursor = conn.cursor()

cursor.execute("SELECT * FROM table_name")

results = cursor.fetchall()

conn.close()

Creating tables and executing queries using Python's MySQL Connector:

cursor = conn.cursor()

cursor.execute('''CREATE TABLE IF NOT EXISTS users

                  (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(255), email VARCHAR(255))''')

cursor.execute("INSERT INTO users (name, email) VALUES (%s, %s)", ('Alice', '*Emails are not allowed*'))

cursor.execute("SELECT * FROM users")

results = cursor.fetchall()

PostgreSQL Database

Strong and open-source, PostgreSQL is an object-relational database system that is well-known for its feature-rich set, dependability, and extensibility. It is a top option for both small-scale applications and large corporate systems because of its comprehensive functions that enable complicated data types, transactions, and data integrity.

Features:

Extensibility: Because of PostgreSQL's flexible architecture, programmers are able to design unique functions, data types, and indexing techniques to meet the needs of particular applications. The smooth integration of third-party extensions and libraries is made possible by this flexibility.
ACID Compliance: PostgreSQL guarantees ACID (
Atomicity: To preserve database integrity, transactions are either fully completed or entirely aborted.
Consistency: Assures that the database is kept intact by writing only legitimate data.
Isolation: Data integrity is ensured by many transactions running simultaneously without interfering.
Durability: Even in the event of a system failure, committed transaction modifications remain intact.) compliance, which in turn provides dependable and consistent database transactions. This functionality is crucial for ensuring consistency across the database and data integrity even in the case of system failures.
Data Integrity and Reliability: PostgreSQL uses built-in restrictions like triggers and rules, foreign keys, and concurrency management to guarantee data integrity and stability.

To install PostgreSQL, you can download and install the PostgreSQL package from the official website.

import psycopg2

conn = psycopg2.connect(

    dbname='your_database_name',

    user='your_username',

    password='your_password',

    host='localhost',

    port='5432'

)

if conn:

    print("Connected to PostgreSQL")

else:

    print("Failed to connect to PostgreSQL")

conn.close()

Creating tables and executing queries using Python's psycopg2 library:

# Create a cursor object

cursor = conn.cursor()

cursor.execute('''CREATE TABLE IF NOT EXISTS users (

                    id SERIAL PRIMARY KEY,

                    name VARCHAR(255),

                    email VARCHAR(255)

                )''')

cursor.execute("INSERT INTO users (name, email) VALUES (%s, %s)", ('Alice', '*Emails are not allowed*'))

cursor.execute("SELECT * FROM users")

results = cursor.fetchall()

for row in results:

    print(row)

conn.commit()

Note: Always securely handle database credentials and avoid hardcoding them directly into your code for better security.

Comparison

Factor	SQLite	MySQL	PostgreSQL
Performance	Good for small to medium -sized applications	Optimized for read-heavy workloads	Optimized for complex queries and large datasets
Scalability	Limited scalability due to file- based architecture	Good scalability with support for clustering and sharding	High scalability with advanced replication and partitioning
Features	Basic features with limited concurrency support	Rich feature set including stored procedures, triggers, and views	Comprehensive feature set including JSONB data type, full-text search, and advanced indexing
Ease of Use	Simple setup and usage, suitable for beginners	Easy to use, especially with graphical tools like MySQL Workbench	Requires more configuration but offers powerful features
Use Case Suitability	Ideal for embedded systems and small applications	Well-suited for web applications and medium- sized databases	Best for large-scale applications with complex data requirements

Advanced Concepts

Advanced ideas like indexing, optimization, and transactions are essential for improving the scalability, performance, and dependability of databases.

Transactions: Transactions ensure the ACID properties of database operations, handling multiple tasks as a single unit for data accuracy and reliability. All database systems provide tools for effective transaction management.
Indexing: Indexes expedite data retrieval, particularly for SELECT queries, by swiftly accessing entries in database tables based on specific field values. Effective indexing reduces the need for full table scans, significantly enhancing query performance.
Optimization: Database optimization strategies improve effectiveness and performance through resource usage optimization, performance tuning, query optimization, and schema design optimization. Each system provides unique features and tools to enhance productivity and address performance issues.

import sqlite3

import mysql.connector

import psycopg2

def sqlite_transactions():

    conn = sqlite3.connect('example.db')

    cursor = conn.cursor()

    try:

        conn.execute("BEGIN")

        cursor.execute("UPDATE users SET balance = balance - 100 WHERE id = 1")

        cursor.execute("UPDATE users SET balance = balance + 100 WHERE id = 2")

        conn.commit()

        print("Transaction committed successfully")

    except sqlite3.Error as e:

        conn.rollback()

        print("Transaction rolled back due to error:", e)

    finally:

        conn.close()

def mysql_transactions():

    conn = mysql.connector.connect(user='user', password='password', database='example')

    cursor = conn.cursor()

    try:

        conn.start_transaction()

        cursor.execute("UPDATE users SET balance = balance - 100 WHERE id = 1")

        cursor.execute("UPDATE users SET balance = balance + 100 WHERE id = 2")

        conn.commit()

        print("Transaction committed successfully")

    except mysql.connector.Error as e:

        conn.rollback()

        print("Transaction rolled back due to error:", e)

    finally:

        conn.close()

def postgres_transactions():

    conn = psycopg2.connect(dbname='example', user='user', password='password', host='localhost')

    cursor = conn.cursor()

    try:

        conn.autocommit = False

        cursor.execute("UPDATE users SET balance = balance - 100 WHERE id = 1")

        cursor.execute("UPDATE users SET balance = balance + 100 WHERE id = 2")

        conn.commit()

        print("Transaction committed successfully")

    except psycopg2.Error as e:

        conn.rollback()

        print("Transaction rolled back due to error:", e)

    finally:

        conn.close()

sqlite_transactions()

mysql_transactions()

postgres_transactions()

SQLite, designed for single-user and small-scale applications, uses local file-based databases and simple transaction management with BEGIN, COMMIT, and ROLLBACK statements.
MySQL, a robust RDBMS, offers scalability and concurrency control with start_transaction(), commit(), and rollback() methods.
PostgreSQL, known for advanced features and extensibility, provides strong support for concurrency, data integrity, and advanced SQL features, managing transactions via autocommit mode and commit()/rollback() methods.

FAQ
Q: What is the difference between SQL and NoSQL databases?
A: SQL (Structured Query Language) databases, also known as relational databases, organize data into tables with predefined schemas and use SQL for querying. NoSQL databases, on the other hand, store data in flexible, non-tabular formats and do not require a fixed schema. They are designed for scalability and handling unstructured data.

Conclusion

Databases are essential to software development because they allow for effective data management. MySQL scales for the web, PostgreSQL meets enterprise needs, and SQLite is ideal for lightweight applications. When choosing a project, it is essential to comprehend their strengths. Using cutting-edge ideas promotes creativity in digital solutions by ensuring data performance and dependability.

Reference

For more reference:
SQLite Official Website:
SQLite Official Website

Automate Your VPS Database Backups Like a Pro: A Complete MySQL Backup Script

Sat, 15 Mar 2025 08:51:52 +0000

Managing a VPS (Virtual Private Server) with a MySQL database? Then you know how important **regular backups** are!

A single accidental command, server crash, or unexpected failure can **wipe out critical data** in seconds. That’s why automating backups is a must!

In this guide, you’ll learn how to:

✅ Automate MySQL backups on your VPS
✅ Compress backups to save space
✅ Automatically delete old backups
✅ Schedule backups using cron jobs

By the end of this guide, you’ll have a **fully automated, hands-free backup solution** running on your server!

Why You Need Automated MySQL Backups

Relying on **manual backups** is risky and inefficient. Here's why automation is essential:

❌ **Mistakes happen** – A single DROP DATABASE command can erase years of data.
❌ **Downtime is expensive** – Losing customer data can damage your reputation and business.
❌ **Disk failures are unpredictable** – Outdated backups won’t help if your server crashes.
✅ **Automation ensures regular backups** – Set it once, and let your VPS handle the rest!

️ Prerequisites

Before we start, make sure you have:

✔️ A VPS running **Debian/Ubuntu**
✔️ MySQL/MariaDB installed
✔️ Sudo privileges

Step 1: Download the MySQL Backup Script

Clone the backup script from GitHub:

git clone https://github.com/giftbalogun/vpsMySQLBackup.git
cd vpsMySQLBackup

⚙️ Step 2: Configure the Backup Script

Open the script and edit your MySQL credentials and backup settings:

nano backup_mysql.sh

Modify these lines:

MYSQL_USER="your_mysql_username"
MYSQL_PASSWORD="your_mysql_password"
DAYS_TO_KEEP=5        # Set how many days to keep backups (0 to keep forever)
GZIP=1                # Set to 1 for compressed backups, 0 for raw SQL
BACKUP_PATH="/root/backup/mysql"

**Tip:** Make sure your MySQL user has the correct privileges!

Step 3: Make the Script Executable & Run It

Give the script execution permission:

chmod +x backup_mysql.sh

Run the script manually to test it:

sudo ./backup_mysql.sh

✅ If everything is set up correctly, you’ll see output like this:

--------------------------------------------------------
      MySQL Backup Script - Copyright © 2024 Gift Balogun
--------------------------------------------------------
[INFO] Backing up database: my_database (compressed)
[SUCCESS] Backup completed: my_database
[INFO] Removing backups older than 5 days...
[INFO] Cleanup completed.
[INFO] MySQL Backup Process Completed Successfully!

Step 4: Automate Backups with Cron Jobs

To ensure backups run automatically, we’ll use **cron jobs**.

1️⃣ Open the cron job editor:

crontab -e

2️⃣ Add this line at the bottom:

0 2 * * * /path/to/backup_mysql.sh >> /var/log/mysql_backup.log 2>&1

This schedules the backup **every day at 2 AM** and logs the output.

3️⃣ Save and exit (CTRL + X, then Y and ENTER).

Done! Your server now automatically backs up MySQL databases daily.

️ Step 5: Restore a Backup (When Needed)

In case of data loss, restoring a backup is simple:

1️⃣ Find your backup file:

ls -lh /root/backup/mysql/

2️⃣ Restore it to MySQL:

mysql -u your_mysql_username -p your_database_name < /root/backup/mysql/your_backup.sql

If your backup is compressed (.sql.gz), use:

gunzip -c /root/backup/mysql/your_backup.sql.gz | mysql -u your_mysql_username -p your_database_name

️ Troubleshooting

**Permission Denied?** Run the script with sudo or check file permissions.
**Crontab Not Running?** Ensure the script has execute permission and use the full path in crontab -e.
**Backup Files Missing?** Double-check the BACKUP_PATH directory and DAYS_TO_KEEP settings.

Final Thoughts

Automating MySQL backups **saves time**, **prevents data loss**, and ensures **peace of mind**.

Want the full script? **Get it on GitHub:** View Full Code Here

Need Help or Have Questions? Drop a comment below!

Database Interaction: Connecting to Databases

Fri, 26 Jul 2024 10:22:07 +0000

In this article, we will address the key issue of database connections in software development. Large volumes of data are effectively stored and managed via databases, which form the foundation of many applications. Building scalable and reliable apps requires developers to have a solid understanding of database connections.
In software development, database connectivity is essential for maintaining data integrity, scalability, and system performance. Relational databases like MySQL and PostgreSQL offer structured storage and sophisticated querying, whereas NoSQL databases like MongoDB and Redis offer flexibility for unstructured data. Comprehending these distinctions is essential for developers to make well-informed decisions.

Selecting the Right Database

This tutorial offers information on different kinds of databases and things to think about.
Different Types of Databases:

Relational Databases: These databases arrange information into structured tables that have pre-established connections between them. PostgreSQL, SQL Server, and MySQL are a few examples.
NoSQL database: Large amounts of unstructured or semi-structured data are easily handled by NoSQL databases because they provide a more flexible data format. Redis, Cassandra, and MongoDB are a few examples.

Factors to Consider When Choosing a Database:

Data Structure and Requirements: NoSQL databases are better for unstructured or schema-less data, while relational databases are better for highly structured data that requires complex querying.
Scalability: Relational databases scale vertically by adding resources to a single server; NoSQL databases scale horizontally by sharing data across servers.
Performance: Relational databases perform better in ACID transactions, however NoSQL databases place a higher priority on low latency and high throughput.
Requirements for Consistency and Integrity: Strict consistency and transactional integrity are guaranteed for accurate data by relational databases. NoSQL databases may provide models for eventual consistency, putting high availability ahead of instant consistency.

from pymongo import MongoClient

client = MongoClient('mongodb://localhost:27017/')

db = client['my_database']

collection = db['my_collection']

collection.insert_one({'name': 'XYZ', 'age': 30, 'email': '*Emails are not allowed*'})

# Query documents

result = collection.find_one({'name': 'XYZ'})

print(result)

Tip: Prototype with different databases to assess performance, scalability, and suitability for your project before making a final decision.

Python's Common Database Connection Methods

There are numerous ways to connect to databases in Python, including:

1. Using built-in libraries:

Direct database connections without the need for additional dependencies are made possible by Python's built-in modules, such as sqlite3 for SQLite.

import sqlite3

conn = sqlite3.connect('example.db')

cursor = conn.cursor()

cursor.execute('SELECT * FROM table_name')

results = cursor.fetchall()

for x in results:

  print(x)

2. Third-party libraries:

External libraries provide additional functionality and optimizations for particular database systems, such as psycopg2 for PostgreSQL and mysql-connector-python for MySQL. These packages offer simple-to-use APIs for establishing database connections and running queries.

import mysql.connector

mydb= mysql.connector.connect(

    host='hostname',

    user='username',

    database='database_name'

)

mycursor = mydb.cursor()

mycursor.execute('SELECT * FROM table_name')

myresult = mycursor.fetchall()

for x in myresult:

  print(x)

3. ORM (Object-Relational Mapping) frameworks:

By abstracting database interactions, ORM frameworks such as SQLAlchemy save developers from writing raw SQL queries and enable them to work with Python objects. These frameworks offer strong capabilities for manipulating and managing databases.

from sqlalchemy import create_engine, MetaData, Table

engine = create_engine('postgresql://username:password@localhost/database_name')

metadata = MetaData()

table = Table('table_name', metadata, autoload=True, autoload_with=engine)

results = engine.execute(table.select()).fetchall()

Connecting to Relational Database

There are multiple processes involved in setting up connections to widely used relational databases such as SQLite, MySQL, and PostgreSQL. Here's a thorough explanation with examples to help clarify:

1. Setting up connections with each database:

A. SQLite:

Data is stored in a single file with the lightweight serverless database engine SQLite. With Python's built-in sqlite3 package, connecting to SQLite is simple.

import sqlite3

conn = sqlite3.connect('example.db')

B. MySQL:

The mysql-connector-python library can be used to create a connection for MySQL. Make sure pip install mysql-connector-python is installed on your computer.

import mysql.connector

conn = mysql.connector.connect(

    host='localhost',

    user='username',

    database='database_name'

)

C. PostgreSQL:

You can use the psycopg2 library to establish a connection to PostgreSQL. Install it (pip install psycopg2), please.

import psycopg2

conn = psycopg2.connect(

    host='localhost',

    user='username',

    password='password',

    database='database_name'

)

2. Handling connection errors and exceptions:

For your application to be robust, it is imperative that you handle exceptions and connection problems in a polite manner. Try-except blocks are a useful tool for handling and catching exceptions.

try:

    conn = psycopg2.connect(

        host='localhost',

        user='username',

        database='database_name'

    )

except psycopg2.Error as e:

    print(f"Error connecting to PostgreSQL: {e}")

3. Executing SQL queries and retrieving results:

Once connected, you may use cursor objects to run SQL queries and get the appropriate responses.

cursor = conn.cursor()

cursor.execute('SELECT * FROM table_name')

results = cursor.fetchall()

for row in results:

    print(row)

cursor.close()

Connecting to NoSQL Databases

In Python, connecting to well-known NoSQL databases such as MongoDB and Redis requires simple steps for database operations, error handling, and setup:

1. Setting up connections with each database:

A. MongoDB:

To create a connection for MongoDB, use the pymongo package.

from pymongo import MongoClient

client = MongoClient('mongodb://localhost:27017/')

B. Redis:

Redis can be accessed by using the redis library.

import redis

r = redis.Redis(host='localhost', port=6379, db=0)

2. Handling connection errors and exceptions:

Make sure to use try-except blocks to properly handle connection problems.

try:

    client = MongoClient('mongodb://localhost:27017/')

except Exception as e:

  print(f"Error connecting to MongoDB: {e}")

// Redis

try:

    r = redis.Redis(host='localhost', port=6379, db=0)

except Exception as e:

    print(f"Error connecting to Redis: {e}")

3. Executing database Operations:

Execute database operations like delete, update, and insert.

MongoDB:

db = client['my_database']

collection = db['my_collection']

collection.insert_one({'name': 'John', 'age': 30})

collection.update_one({'name': 'John'}, {'$set': {'age': 31}})

collection.delete_one({'name': 'John'})

Redis:

r.set('key', 'value')

value = r.get('key')

r.delete('key')

Error Handling

Error handling in database connections is crucial for program reliability and robustness, preventing crashes and enhancing user experience by addressing unforeseen database errors and connection issues.

Using try-except blocks: Try-except blocks can be used to catch and manage exceptions that are raised during database operations. This guarantees that your program can continue running even after encountering failures.

import psycopg2

try:

    conn = psycopg2.connect(

        host='localhost',

        user='username',

        database='database_name'

    )

except psycopg2.Error as e:

    print(f"Error connecting to PostgreSQL: {e}")

Handling specific error types: To improve error management precision, classify errors such as query problems or connection failures and provide customised error messages and actions.

import psycopg2

try:

    conn = psycopg2.connect(

        host='localhost',

        user='username',

        password='password',

        database='database_name'

    )

except psycopg2.OperationalError as e:

    print(f"Operational error connecting to PostgreSQL: {e}")

except psycopg2.DatabaseError as e:

    print(f"Database error connecting to PostgreSQL: {e}")

Implementing retry mechanisms: Retry techniques automatically retry database operations during temporary errors or network issues, enhancing application robustness, especially in unstable contexts.

import psycopg2

import time

max_retries = 3

retry_delay = 1  # seconds

for retry_count in range(max_retries):

    try:

        conn = psycopg2.connect(

            host='localhost',

            user='username',

            database='database_name'

        )

        break

    except psycopg2.Error as e:

        print(f"Error connecting to PostgreSQL: {e}")

        if retry_count < max_retries - 1:

            print(f"Retrying in {retry_delay} seconds...")

            time.sleep(retry_delay)

        else:

            print("Max retries exceeded, unable to connect.")

          break

Best Practices and Tips

By adhering to these best practices, Python programs may guarantee stable database connections:

To effectively manage database connections, make use of connection pooling.
In order to reduce overhead and boost efficiency, reuse connections.
To liberate resources, always properly close connections.
Handle connection credentials securely to avoid unwanted access.

from sqlalchemy import create_engine, Pool

# Create an engine with connection pooling

engine = create_engine('database://user:password@host:port/db_name', pool_size=5, max_overflow=10)

# Use the engine to connect to the database

connection = engine.connect()

# Perform database operations...

# Close the connection

connection.close()

FAQ
Q: Why is connection pooling important in database applications?
A: Connection pooling helps manage and reuse database connections, reducing the overhead of establishing new connections for each database operation. This improves performance and scalability, especially in applications with high traffic.

Conclusion

To sum up, this article highlights how important database connections are to software development. Comprehending the subtle differences between relational and NoSQL database types is essential for achieving the best possible project performance. Python provides flexible ways to provide error handling and make database connections. Through careful consideration of database selection and adherence to industry best practices, developers can construct resilient systems that effectively handle data and satisfy the requirements of contemporary software development.

Reference

For more reference:

PyMongo
PyMongo - MongoDB Python driver

Coder Legion - Recent posts tagged mysql

Flyway Migration: From Legacy Database to CI/CD

What is Flyway?

Use Case: Existing Schema with No Migrations (Yet)

Step 1: Baselining an Existing Schema

Dump the schema

Create directory structure

Configure flyway.conf

Run baseline

Step 2: Add a Real Migration

⚙️ Step 3: Integrate Flyway into CI/CD

✅ Example GitHub Actions Workflow

Benefits of This Approach

Potential Challenges

Alternatives (and Why Flyway Wins)

Final Thoughts

Let’s Connect!

Understanding Database Replication: How to Keep Your Apps Fast and Reliable

What is Database Replication?

A Real-World Analogy: The Restaurant Kitchen Whiteboard

Why Database Replication Matters in Modern Engineering

Implementing Read/Write Splitting in Node.js

Key Takeaway

Demystifying ACID Transactions: How to Keep Your Database from Making a Mess

The Ticket Booth Analogy

Why It Matters: Real-World Disasters

Implementing ACID in Node.js and MySQL

The Takeaway

Database Concurrency 101: Optimistic vs. Pessimistic Locking

A Relatable Everyday Analogy

Why It Matters in the Tech Industry

Implementation in Node.js and MySQL

The Strategic Takeaway

Data Engineering Beyond "Pretty Charts": How I Built an Aviation Telemetry Leakage Engine

Stop Tracking "On-Time" Flights. Track Tarmac Fuel Destruction Instead.

The Architectural & Engineering Challenge

The Bottom-Line Impact Metrics

The Takeaway

The 200-byte trap: why WordPress core updates break Arabic URLs

Why Arabic URLs hit a wall English never does

The number 200 lives in three places, not one

Layer 1: how a core update erases your column width

The fix: don't repair the damage, prevent the shrink

Layer 2: stop new slugs truncating at birth

Keeping the fork honest

A tripwire, because silent failure is the real enemy

"Isn't VARCHAR(1024) dangerous?": no, and the reason is reassuring

Doing it safely on a site with millions of posts

The deeper fix: stop betting your URLs on a column width

The complete plugin

A postscript: the drift guard caught us

Further reading

Stop Killing Your Database: The Power of Connection Pooling

The Pizza Delivery Analogy

Why Connection Pooling Matters Daily

Implementation in Node.js & MySQL

The Takeaway

Why Your Node.js App Crashes Under Traffic: The Magic of Connection Pooling

What is Connection Pooling?

The Rental Car Analogy

Why It Matters Daily in the Tech Industry

Connection Pooling in Action (Node.js & MySQL)

The Takeaway

Supercharge Your Node.js App: Why You Need Database Connection Pooling

What is Connection Pooling?

The Pizza Scooter Analogy

Why Connection Pooling Matters Daily

Simple Code Example

The Bad Way (New Connection per Request)

The Good Way (Using a Connection Pool)

Key Takeaway

The Silent Database Killer: Understanding and Fixing the N+1 Query Problem

The Grocery Store Analogy

Why It Matters in the Tech Industry

Spotting and Fixing the Problem in Node.js

The Takeaway

Stop Crashing Your Database: A Developer's Guide to Connection Pooling

The Taxi Stand Analogy

Why Connection Pooling Matters Daily

Putting It into Code