Coder Legion - Recent posts tagged sql

How to Format SQL Queries in Python: Best Practices, Gotchas, and Real-World Examples

Thu, 29 Oct 2026 16:30:33 +0000

Stop writing SQL strings that look like a ransom note. Here's how to write queries that are readable, safe, and maintainable.

The Problem With "Good Enough" SQL Formatting

Most Python developers start here:

user_id = 5
query = "SELECT * FROM users WHERE id = " + str(user_id)
cursor.execute(query)

It works. Until it doesn't — and when it breaks, it breaks badly: SQL injection, cryptic errors from mismatched types, and queries that take 45 minutes to debug at 2am. Let's fix that, permanently.

1. Never Concatenate User Input — Use Parameterized Queries

This is rule #1 and it's non-negotiable.

❌ The Wrong Way (SQL Injection Waiting to Happen)

username = request.args.get("username")  # Could be: ' OR '1'='1
query = f"SELECT * FROM users WHERE username = '{username}'"
cursor.execute(query)

If username is ' OR '1'='1, your entire users table just got exposed.

✅ The Right Way: Parameterized Queries

username = request.args.get("username")

# psycopg2 (PostgreSQL)
cursor.execute("SELECT * FROM users WHERE username = %s", (username,))

# sqlite3
cursor.execute("SELECT * FROM users WHERE username = ?", (username,))

# SQLAlchemy Core
from sqlalchemy import text
result = conn.execute(text("SELECT * FROM users WHERE username = :name"), {"name": username})

The database driver handles escaping. You never touch it. This pattern is immune to SQL injection by design.

Gotcha: Note the trailing comma in (username,). Without it, Python treats the string as an iterable and passes each character as a separate parameter. This is one of the most common beginner bugs.

# 💥 Bug: passes ('a', 'l', 'i', 'c', 'e') instead of ('alice',)
cursor.execute("SELECT * FROM users WHERE username = %s", (username))

# ✅ Correct: single-element tuple
cursor.execute("SELECT * FROM users WHERE username = %s", (username,))

2. Multi-Line Queries: Triple Quotes + Consistent Indentation

For anything longer than one clause, use triple-quoted strings. They're readable, diffable, and don't require escape characters.

❌ String Concatenation Chaos

query = "SELECT u.id, u.name, o.total " + \
        "FROM users u " + \
        "JOIN orders o ON u.id = o.user_id " + \
        "WHERE o.total > 100"

✅ Triple-Quoted SQL

query = """
    SELECT
        u.id,
        u.name,
        u.email,
        o.total,
        o.created_at
    FROM users u
    JOIN orders o
        ON u.id = o.user_id
    WHERE
        o.total > 100
        AND u.active = TRUE
    ORDER BY o.created_at DESC
    LIMIT 50
"""

cursor.execute(query)

Style conventions that matter:

SQL keywords in UPPERCASE
One clause per line (SELECT, FROM, WHERE, ORDER BY)
Indent column lists by 4 spaces
Put ON conditions on their own line, indented under JOIN

3. Dynamic Queries: Building Them Safely

Real applications need dynamic filters. Here's how to do it without string concatenation.

Scenario: A search endpoint with optional filters

def get_orders(user_id: int, status: str = None, min_total: float = None):
    base_query = """
        SELECT
            o.id,
            o.status,
            o.total,
            o.created_at
        FROM orders o
        WHERE o.user_id = %s
    """
    params = [user_id]
    conditions = []

    if status:
        conditions.append("AND o.status = %s")
        params.append(status)

    if min_total is not None:
        conditions.append("AND o.total >= %s")
        params.append(min_total)

    full_query = base_query + "\n        ".join(conditions)

    cursor.execute(full_query, params)
    return cursor.fetchall()

This is the accumulator pattern: build a list of clauses and params in parallel, then join them. The number of %s placeholders always matches len(params).

Gotcha: Never dynamically interpolate column names or table names with parameters — the DB driver only parameterizes values, not identifiers. If you need a dynamic column name, whitelist it explicitly:

ALLOWED_SORT_COLUMNS = {"created_at", "total", "status"}

def get_orders_sorted(sort_by: str):
    if sort_by not in ALLOWED_SORT_COLUMNS:
        raise ValueError(f"Invalid sort column: {sort_by}")

    # Now safe to interpolate — it came from our whitelist, not user input
    query = f"""
        SELECT id, status, total, created_at
        FROM orders
        ORDER BY {sort_by} DESC
    """
    cursor.execute(query)

4. IN Clauses: The Tricky One

Parameterizing IN (...) trips up almost everyone the first time.

❌ This Won't Work

ids = [1, 2, 3, 4]
cursor.execute("SELECT * FROM users WHERE id IN (%s)", (ids,))
# Error: query expects 1 placeholder, got a list

✅ Generate Placeholders Dynamically

ids = [1, 2, 3, 4]

# sqlite3
placeholders = ", ".join("?" * len(ids))
query = f"SELECT * FROM users WHERE id IN ({placeholders})"
cursor.execute(query, ids)

# psycopg2 — uses %s for each
placeholders = ", ".join(["%s"] * len(ids))
query = f"SELECT * FROM users WHERE id IN ({placeholders})"
cursor.execute(query, ids)

✅ SQLAlchemy makes this trivial

from sqlalchemy import select, bindparam

stmt = select(users_table).where(users_table.c.id.in_([1, 2, 3, 4]))
result = conn.execute(stmt)

5. Use SQLAlchemy Core for Complex Queries

When your raw SQL starts looking like a maze, SQLAlchemy Core gives you composable, Pythonic query building without the full ORM overhead.

from sqlalchemy import create_engine, select, func, and_, desc
from sqlalchemy import Table, MetaData

engine = create_engine("postgresql://user:pass@localhost/mydb")
meta = MetaData()
meta.reflect(bind=engine)

orders = meta.tables["orders"]
users = meta.tables["users"]

stmt = (
    select(
        users.c.id,
        users.c.name,
        func.sum(orders.c.total).label("lifetime_value"),
        func.count(orders.c.id).label("order_count"),
    )
    .join(orders, users.c.id == orders.c.user_id)
    .where(
        and_(
            orders.c.status == "completed",
            orders.c.created_at >= "2024-01-01",
        )
    )
    .group_by(users.c.id, users.c.name)
    .having(func.sum(orders.c.total) > 500)
    .order_by(desc("lifetime_value"))
    .limit(20)
)

with engine.connect() as conn:
    results = conn.execute(stmt).fetchall()

This compiles to safe, parameterized SQL automatically. You can also print the generated SQL for debugging:

print(stmt.compile(dialect=engine.dialect, compile_kwargs={"literal_binds": True}))

6. Formatting for Logging and Debugging

When you need to log a query (for debugging, audit trails, etc.), format it cleanly — but never log with real user data embedded.

import logging
import re

logger = logging.getLogger(__name__)

def log_query(query: str, params: tuple):
    # Normalize whitespace for clean log output
    clean_query = re.sub(r'\s+', ' ', query.strip())
    logger.debug("Executing SQL: %s | Params: %s", clean_query, params)

log_query("""
    SELECT id, name
    FROM users
    WHERE active = %s
""", (True,))

# Output: Executing SQL: SELECT id, name FROM users WHERE active = %s | Params: (True,)

Gotcha: Don't use cursor.mogrify() (psycopg2) to log queries with real values in production — it defeats the purpose of parameterization from an audit standpoint, and if logs are exfiltrated, you've exposed real data.

7. Query Constants: SQL as Module-Level Variables

For queries used across your codebase, define them as constants at the top of your module. This makes them findable, testable, and easy to update.

# queries.py

GET_USER_BY_ID = """
    SELECT id, name, email, created_at
    FROM users
    WHERE id = %s
      AND active = TRUE
"""

GET_USER_ORDERS = """
    SELECT
        o.id,
        o.status,
        o.total,
        o.created_at
    FROM orders o
    WHERE o.user_id = %s
    ORDER BY o.created_at DESC
"""

INSERT_ORDER = """
    INSERT INTO orders (user_id, total, status, created_at)
    VALUES (%s, %s, %s, NOW())
    RETURNING id
"""

# Usage in your service layer
from queries import GET_USER_BY_ID, GET_USER_ORDERS

def get_user_with_orders(user_id: int):
    cursor.execute(GET_USER_BY_ID, (user_id,))
    user = cursor.fetchone()

    cursor.execute(GET_USER_ORDERS, (user_id,))
    orders = cursor.fetchall()

    return user, orders

This makes SQL grep-able, version-controllable, and reviewable in PRs.

8. Gotchas Quick Reference

Gotcha	What Goes Wrong	Fix
`(value)` vs `(value,)`	String unpacked as characters	Always use trailing comma for single-element tuples
f-string with user input	SQL injection	Use parameterized queries
Dynamic `IN (...)`	Wrong number of placeholders	Generate placeholders programmatically
Dynamic column names	Parameters don't work for identifiers	Whitelist allowed column names
Logging with real values	Data exposure in logs	Log query template + params separately
`None` in params	NULL comparison breaks	Use `IS NULL` in SQL, not `= NULL`

# None/NULL gotcha
user_role = None

# ❌ Won't match any rows — SQL NULL != NULL
cursor.execute("SELECT * FROM users WHERE role = %s", (user_role,))

# ✅ Use IS NULL when the value might be None
if user_role is None:
    cursor.execute("SELECT * FROM users WHERE role IS NULL")
else:
    cursor.execute("SELECT * FROM users WHERE role = %s", (user_role,))

Summary

Situation	Recommended Approach
Simple parameterized query	`cursor.execute(sql, (val,))`
Multi-line query	Triple-quoted string with clause-per-line formatting
Dynamic filters	Accumulator pattern (list of clauses + params)
`IN` clause	Generate placeholders: `", ".join(["%s"] * len(ids))`
Complex joins/aggregates	SQLAlchemy Core
Reusable queries	Module-level string constants
Dynamic column names	Whitelist validation + f-string

The core rule is simple: SQL structure is yours to format; SQL values are the driver's job to escape. Internalize that boundary and most of these best practices follow naturally.

Found a gotcha I missed? Drop it in the comments — SQL has a way of surprising everyone eventually.

Advanced SQL Analytics for Data Analysis.

Sun, 18 Oct 2026 13:54:35 +0000

Advanced SQL analytics for running a hospital database.

Introduction

Structured Query Language is essential and common in most industry sectors. In healthcare reporting, SQL plays a critical role in analytics. PostgreSQL lets data scientists take those transaction records and pull out operational insights, so you can make decisions. In this guide we go through the main SQL techniques, using a hospital database schema; patients , doctors, departments , wards, diagnoses, procedures, and visits.

First, before anything else we need to create our database schema. Connect to PostgerSQL DBMS and run a new sql script to start.

Create a schema hospital

create schema hospital;

Check to which schema you are connected.

show search_path;

If connected to a different schema, set connection to your hospital schema.

set search_path to hospital;

Now you are set to begin creating your tables. Run the following codes to create the different tables and insert data into the newly created tables:

Doctors Table

--create doctors table

create table hospital.doctors (
doctor_id int primary key,
doctor_name varchar(50) not null,
doctor_gender varchar(20),
doctor_specialty varchar(20),
doctor_experience_years int
);

select * from hospital.doctors;

-- insert data into doctors

insert into hospital.doctors (doctor_id, doctor_name, doctor_gender, doctor_specialty, doctor_experience_years)
values 
(101, 'DR.Brian Mwangi', 'Male', 'Cardiologist', 12),
(102, 'Dr. Sarah Wanjiku', 'Female', 'General Physician', 9), 
(103, 'Dr. Peter Otieno', 'Male', 'Orthopedic Surgeon', 15),
(104, 'Dr. Faith Chebet', 'Female', 'Pediatrician', 8);

Departments Table

-- create departments table

create table hospital.department (
department_id int primary key,
department_name varchar (50) not null
);


select * from hospital.department;

-- insert data into departments

insert into hospital.department (department_id, department_name)
values 
(10, 'Cardiology'),
(20, 'General Medicine'),
(30, 'Orthopedics'),
(40, 'Pediatrics');

Departments Table

-- create diagnosis table

create table hospital.diagnosis (
diagnosis_id int primary key,
diagnosis_name varchar(50),
diagnosis_category varchar(50)
);

select * from hospital.diagnosis;

--insert data into diagnosis table

insert into hospital.diagnosis (diagnosis_id, diagnosis_name, diagnosis_category)
values 
(501,'Hypertension', 'Chronic Disease'),
(502, 'Malaria', 'Infectious Disease'),
(503, 'Fracture', 'Injury'),
(504, 'Pneumonia', 'Respiratory Disease'),
(505, 'Diabetes', 'Chronic Disease'),
(506, 'Typhoid', 'Infectious'),
(507, 'Asthma', 'Respiratory Disease'),
(508, 'Back Pian', 'Musculoskeletal'),
(509, 'Gastritis', 'Digestive Disease'),
(510, 'Heart Failure', 'Chronic Disease'),
(511, 'Flu', 'Respiratoey Disease'),
(512, 'Kidney Infection', 'Infectious Disease'),
(513, 'Anaemia', 'Blood Disorder'),
(514, 'Sprain', 'Injury'),
(515, 'Gastroenteritis', 'Digestive Disease'),
(516, 'Dislocation', 'Injury'),
(517, 'Migraine', 'Neurological Disorder'),
(518, 'Chest Pain', 'Cardiovascular Disease');

Wards Table

-- create wards table 

create table hospital.wards (
ward_id int primary key, 
ward_name varchar (50)
);

-- insert data into wards
insert into hospital.wards (ward_id, ward_name)
values 
(100, 'Cardiac Care Unit'),
(200, 'General Ward'),
(300, 'Surgical Ward'),
(400, 'Childrens Ward');

select * from hospital.wards;

Patients Table

-- create patients table 
create table hospital.patients (
patient_id serial primary key, 
patient_name varchar(50) not null,
patient_gender varchar(20),
patient_age int,
county varchar(30),
sub_county varchar(30)
);

--insert data into patients
insert into hospital.patients (patient_name, patient_gender, patient_age, county, sub_county)
values 
('Amina Otieno', 'Female', 34, 'Nairobi', 'Westlands'),
('John Kariuki', 'Male', 42,' Kiambu', 'Thika'),
('Grace Naliaka', 'Female', 28, 'Kakamega', 'Lurambi'),
('David Mutua', 'Male', 55, 'Machakos', 'Athi River'),
('Esther Achieng', 'Female', 63, 'Kisumu', 'Kisumu Central'),
('Mohamed Ali', 'Male', 19, 'Mombasa', 'Nyali'),
('Lucy Wambui', 'Female', 7, 'Nakuru', 'Naivasha'),
('Samuel Kiptoo', 'Male', 39, 'Uasin Gishu', 'Eldoret East'),
('Mary Wairumu', 'Female', 31, 'Murang''a', 'Kiharu'),
('Peter Ouma', 'Male', 46, 'Siaya', 'Bondo'),
('Brian Omondi', 'Male', 24, 'Kisumu', 'Kisumu East'),
('Catherine Musyoka', 'Female', 52, 'Kitui', 'Kitui Central'),
('Daniel Maina', 'Male', 16, 'Nyeri', 'Othaya'),
('Eunice Moraa', 'Female', 44, 'Kisii', 'Bonchari'),
('Francis Njuguna', 'Male', 60, 'Laikipia', 'Nyanyuki'),
('Hellen Jepchirchir', 'Female', 37, 'Bomet', 'Sotik'),
('Isaac Kamau', 'Male', 29, 'Nairobi', 'Kasarani'),
('Jane Muthoni', 'Female', 22, 'Meru', 'Imenti North'),
('Kevin Barasa', 'Male', 48, 'Bungoma', 'Webuye'),
('Lilian Adhiambo', 'Female', 26, 'Homa Bay', 'Rangwe'),
('Moses Cheruiyot', 'Male', 33, 'Kericho', 'Ainamoi'),
('Nadia Hussein', 'Female', 5, 'Garissa', 'Dadaab'),
('Patrick Muli', 'Male', 51, 'Makueni', 'Wote'),
('Ruth Nekesa', 'Female', 41, 'Trans Nzoia', 'Kitale'),
('Josephine Wekesa', 'Female', 36, 'Busia', 'Matayos');

select * from hospital.patients;

Procedures Table

create table hospital.procedures (
procedure_id int primary key,
procedure_name varchar (50) not null,
procedure_category varchar (20)
);

--insert data into procedures
insert into hospital.procedures (procedure_id, procedure_name, procedure_category)
values
(701, 'Blood Pressure Monitoring', 'Diagnostic'),
(702, 'Malaria Test', 'Laboratory'),
(703, 'X-ray', 'Imaging'),
(704, 'Chest X-ray', 'Imaging'),
(705, 'Blood Sugar Test', 'Laboratory'),
(706, 'Full Blood Count', 'Laboratory'),
(707, 'Nebulization', 'Treatment'),
(708, 'Physiotherapy', 'Treatment'),
(709, 'Endoscopy', 'Diagnostic'),
(710, 'ECG', 'Diagnostic'),
(711, 'Consultation', 'Consultation'),
(712, 'Joint Support Bandage', 'Treatment'),
(713, 'IV Fluids', 'Treatment'),
(714, 'Closed Reduction', 'Treatment'),
(715, 'Pain Management', 'Treatment');


select * from hospital.procedures;

Visits Table

-- create visits table (Fact Table)

create table hospital.visits (
visit_id int primary key,
visit_date date not null,
patient_id int not null references hospital.patients(patient_id),
doctor_id int not null references hospital.doctors(doctor_id),
department_id int not null references hospital.department(department_id),
ward_id int references hospital.wards(ward_id),
diagnosis_id int references hospital.diagnosis(diagnosis_id),
procedure_id int references hospital.procedures(procedure_id),
admission_type varchar(20) not null
    check (admission_type in ('Outpatient', 'Inpatient', 'Emergency')),
length_of_stay_days int default 0
    check (length_of_stay_days >= 0),
triage_level varchar(10)
    check (triage_level in ('Red', 'Orange', 'Yellow', 'Green')),
insurance_provider varchar(50),
total_bill_kes numeric(10,2) not null
    check (total_bill_kes >= 0),
payment_method varchar(20),
follow_up_required varchar(3)
    check (follow_up_required in ('Yes', 'No')),
patient_satisfaction_score numeric(3,1)
    check (patient_satisfaction_score between 0 and 5)
);

--insert data into visits table
insert into hospital.visits (visit_id, visit_date, patient_id, doctor_id, department_id, ward_id, diagnosis_id,
procedure_id, admission_type, length_of_stay_days, triage_level, insurance_provider, total_bill_kes, payment_method,
follow_up_required, patient_satisfaction_score)
values
(1001, '2026-06-01', 1, 101, 10, 100, 501, 701, 'Outpatient', 0, 'Green', 'NHIF', 3500, 'Cash', 'Yes', 4.5),
(1002, '2026-06-01', 2, 102, 20, 200, 502, 702, 'Outpatient', 0, 'Yellow', 'Britam', 2500, 'Insurance', 'No', 4.1),
(1003, '2026-06-02', 3, 103, 30, 300, 503, 703, 'Inpatient', 3, 'Red', 'AAR', 28000, 'Insurance', 'Yes', 4.3),
(1004, '2026-06-02', 4, 104, 40, 400, 504, 704, 'Inpatient', 4, 'Red', 'None', 42000, 'Cash', 'Yes', 3.9),
(1005, '2026-06-03', 5, 101, 10, 100, 505, 705, 'Outpatient', 0, 'Yellow', 'CIC', 4000, 'Mobile Money', 'Yes', 4.7),
(1006, '2026-06-03', 6, 102, 20, 200, 506, 706, 'Outpatient', 0, 'Green', 'None', 3000, 'Cash', 'No', 4.0),
(1007, '2026-06-04', 7, 104, 40, 400, 507, 707, 'Outpatient', 0, 'Yellow', 'NHIF', 5500, 'Insurance', 'Yes', 4.6),
(1008, '2026-06-04', 8, 103, 30, 300, 508, 708, 'Outpatient', 0, 'Green', 'None', 6500, 'Cash', 'Yes', 4.2),
(1009, '2026-06-05', 9, 102, 20, 200, 509, 709, 'Outpatient', 0, 'Yellow', 'Jubilee', 15000, 'Insurance', 'No', 3.8),
(1010, '2026-06-05', 10, 101, 10, 100, 510, 710, 'Inpatient', 5, 'Red', 'AAR', 76000, 'Insurance', 'Yes', 4.4),
(1011, '2026-06-06', 1, 101, 10, 100, 505, 705, 'Outpatient', 0, 'Green', 'NHIF', 4200, 'Mobile Money', 'Yes', 4.8),
(1012, '2026-06-06', 11, 102, 20, 200, 502, 702, 'Outpatient', 0, 'Green', 'None', 2300, 'Cash', 'No', 4.0),
(1013, '2026-06-07', 12, 101, 10, 100, 501, 710, 'Outpatient', 0, 'Yellow', 'Britam', 7000, 'Insurance', 'Yes', 4.5),
(1014, '2026-06-07', 13, 104, 40, 400, 511, 711, 'Outpatient', 0, 'Green', 'None', 1800, 'Cash', 'No', 4.1),
(1015, '2026-06-08', 14, 103, 30, 300, 503, 703, 'Inpatient', 2, 'Red', 'CIC', 24000, 'Mobile Money', 'Yes', 4.6),
(1016, '2026-06-08', 15, 102, 20, 200, 512, 706, 'Inpatient', 3, 'Orange', 'Jubilee', 36000, 'Insurance', 'Yes', 3.7),
(1017, '2026-06-09', 16, 101, 10, 100, 510, 710, 'Inpatient', 6, 'Red', 'AAR', 82000, 'Insurance', 'Yes', 4.2),
(1018, '2026-06-09', 17, 102, 20, 200, 509, 711, 'Outpatient', 0, 'Green', 'None', 2000, 'Cash', 'No', 4.0),
(1019, '2026-06-10', 18, 104, 40, 400, 507, 707, 'Outpatient', 0, 'Yellow', 'NHIF', 5200, 'Mobile Money', 'Yes', 4.7),
(1020, '2026-06-10', 19, 103, 30, 300, 508, 708, 'Outpatient', 0, 'Green', 'Britam', 6500, 'Insurance', 'No', 4.1),
(1021, '2026-06-11', 2, 101, 10, 100, 501, 701, 'Outpatient', 0, 'Yellow', 'Britam', 3800, 'Insurance', 'Yes', 4.3),
(1022, '2026-06-11', 20, 102, 20, 200, 513, 706, 'Outpatient', 0, 'Green', 'None', 3200, 'Mobile Money', 'No', 4.4),
(1023, '2026-06-12', 21, 103, 30, 300, 514, 712, 'Outpatient', 0, 'Yellow', 'NHIF', 7200, 'Cash', 'Yes', 4.0),
(1024, '2026-06-12', 22, 104, 40, 400, 515, 713, 'Inpatient', 2, 'Orange', 'None', 18500, 'Cash', 'Yes', 4.5),
(1025, '2026-06-13', 5, 101, 10, 100, 501, 710, 'Outpatient', 0, 'Yellow', 'CIC', 7600, 'Insurance', 'Yes', 4.6),
(1026, '2026-06-13', 23, 102, 20, 200, 506, 706, 'Inpatient', 2, 'Orange', 'Jubilee', 29000, 'Insurance', 'Yes', 3.9),
(1027, '2026-06-14', 24, 103, 30, 300, 516, 714, 'Inpatient', 1, 'Red', 'AAR', 33000, 'Insurance', 'Yes', 4.2),
(1028, '2026-06-14', 7, 104, 40, 400, 511, 711, 'Outpatient', 0, 'Green', 'NHIF', 1900, 'Mobile Money', 'No', 4.8),
(1029, '2026-06-15', 25, 102, 20, 200, 517, 715, 'Outpatient', 0, 'Yellow', 'Britam', 4800, 'Cash', 'Yes', 4.1),
(1030, '2026-06-15', 10, 101, 10, 100, 518, 710, 'Emergency', 1, 'Red', 'AAR', 22000, 'Insurance', 'Yes', 4.4);

select * from hospital.visits;

After creating our database schema, we will begin modelling the data using different advanced SQL functions.

1. SQL Window Functions

Window functions do computations on related rows in a table while maintaining the individuality of each rows.

OVER, PARTITION BY, ORDER BY, and Window Frames

The OVER clause describes the rows for the window function operation.

PARTITION BY defines different partitions for each result set.
ORDER BY defines row order in each partition.
Window frames define different limits such asROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW for cumulative sums and ROWS BETWEEN 2 PRECEDING AND CURRENT ROW for moving averages.

Ranking Window Functions

Ranking functions provide numeric ordering according to specified criteria.

ROW_NUMBER provides a distinct numeric integer value to each row within a partition.
RANK provides identical rank values to ties, but skips other numbers.
DENSE_RANK provides identical rank values to ties, but does not skip numbers.
NTILE divides rows in partitions into buckets of equal size.

Question: How can hospital management rank inpatient visits by total bill within each admission type?

select 
    v.visit_id,
    p.patient_name,
    v.admission_type,
    v.total_bill_kes,
    row_number() over (partition by v.admission_type order by v.total_bill_kes desc) as row_num,
    rank() over (partition by v.admission_type order by v.total_bill_kes desc) as rnk,
    dense_rank() over (partition by v.admission_type order by v.total_bill_kes desc) as dense_rnk,
    ntile(2) over (partition by v.admission_type order by v.total_bill_kes desc) as bill_tier
from hospital.visits v
join hospital.patients p on v.patient_id = p.patient_id
where v.admission_type = 'Inpatient'
limit 5;

Expected Results

Explanation

ROW_NUMBER gives values from 1 to 5.
The values obtained by RANK and DENSE_RANK will be same as there are no duplicate bill values.
NTILE assigns the best bills to Tier 1.

Navigation functions retrieve values based on relative rows.

LAG retrieves values from previous row.
LEAD retrieves values from next row.
FIRST_VALUE gives the first value of the expression in the frame.
LAST_VALUE gives the last value of the expression in the frame bounds like ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING.

Question: How can we monitor patient billing process through multiple hospital visits?

select 
    v.visit_id,
    v.patient_id,
    v.visit_date,
    v.total_bill_kes,
    lag(v.total_bill_kes, 1, 0.00) over (partition by v.patient_id order by v.visit_date) as prev_bill,
    lead(v.total_bill_kes, 1, 0.00) over (partition by v.patient_id order by v.visit_date) as next_bill,
    first_value(v.total_bill_kes) over (partition by v.patient_id order by v.visit_date) as initial_bill,
    last_value(v.total_bill_kes) over (
        partition by v.patient_id 
        order by v.visit_date 
        rows between unbounded preceding and unbounded following
    ) as final_bill
from visits v
where v.patient_id in (1, 2)
order by v.patient_id, v.visit_date;

Expected Results

Explanation

LAG retrieves previous visit invoices and LEAD retrieves future visit invoices.
FIRST_VALUE and LAST_VALUE store the first and last visit invoices respectively.

Aggregate Window Functions

Aggregate functions SUM, AVG, COUNT, MIN, and MAX calculate moving sums and moving averages when combined with OVER.

Question: In what way do we determine cumulative income, rolling average billings, categories, and billings for departments on each visit?

select 
    v.visit_id,
    v.visit_date,
    v.total_bill_kes,
    sum(v.total_bill_kes) over (order by v.visit_date, v.visit_id) as cumulative_revenue,
    avg(v.total_bill_kes) over (
        order by v.visit_date 
        rows between 2 preceding and current row
    ) as rolling_avg_bill,
    count(*) over (partition by v.admission_type) as total_admission_type_count,
    min(v.total_bill_kes) over (partition by v.admission_type) as min_admission_bill,
    max(v.total_bill_kes) over (partition by v.admission_type) as max_admission_bill
from hospital.visits v
order by v.visit_date, v.visit_id
limit 5;

Expected Results

Explanation

SUM forms a running total.
AVG calculates a 3 row moving average.
COUNT, MIN, and MAX present group statistics without collapsing rows.

2. SQL Joins

Joins combine columns from multiple tables using related key columns.

INNER JOIN

INNER JOIN returns rows with matching key values in both tables.

Question: How do we generate a visit report combining patient names, attending doctors, and clinical departments?

select 
    v.visit_id,
    p.patient_name,
    d.doctor_name,
    dept.department_name,
    v.total_bill_kes
from hospital.visits v
inner join hospital.patients p on v.patient_id = p.patient_id
inner join hospital.doctors d on v.doctor_id = d.doctor_id
inner join hospital.department dept on v.department_id = dept.department_id
limit 5;

Expected Results

Explanation

INNER JOIN Matches keys across visits, patients, doctors, and department tables.

LEFT JOIN

The LEFT JOIN produces all the rows from the left table and the matched rows from the right table. Fields in the right table without a match produce NULL.

Question: How can we study the number of visits and expenses of patients without losing those that have no visits at all?

select 
    p.patient_id,
    p.patient_name,
    count(v.visit_id) as total_visits,
    coalesce(sum(v.total_bill_kes), 0.00) as total_spent_kes
from hospital.patients p
left join hospital.visits v on p.patient_id = v.patient_id
group by p.patient_id, p.patient_name
order by total_visits desc, p.patient_id
limit 5;

Expected Results

Explanation

All patient records are retained using LEFT JOIN.
Patients having no visits will have COUNT as 0 while COALESCE will convert NULL sums to 0.00.

RIGHT JOIN

RIGHT JOIN retrieves all records from right table and matching records from left table.

Question: How can hospital administration ensure that all physical wards are reflected in admission records?

select 
    w.ward_id,
    w.ward_name,
    v.visit_id,
    v.total_bill_kes
from hospital.visits v
right join hospital.wards w on v.ward_id = w.ward_id
where v.visit_id is null or v.visit_id <= 1003;

Expected Results

Explanation

All wards are kept in RIGHT JOIN.
The Childrens Ward will be with NULL visits if no match occurs.

FULL OUTER JOIN

FULL OUTER JOIN will display all records if there is a match in either left or right record.

Question: How do we perform auditing of doctor and department records for unassigned doctors or unmapped departments?

select 
    d.doctor_name,
    dept.department_name
from hospital.doctors d
full outer join hospital.department dept on d.doctor_id = dept.department_id;

Expected Results

Explanation

Null values will be returned in the rows where there is no relationship between doctor and department IDs.

CROSS JOIN

CROSS JOIN results in a Cartesian product where each row in the first table is paired up with each row of the second table.

Question: How can hospital managers get a complete list of all pairings of physicians with hospital wards for coverage planing?

select 
    d.doctor_name,
    w.ward_name
from hospital.doctors d
cross join hospital.wards w
order by d.doctor_name, w.ward_name
limit 5;

Expected Results

Explanation

Joining 4 doctors with 4 wards results in 16 combination rows.

SELF JOIN

SELF JOIN involves joining a table to itself for comparing rows in the same table.

Question: In what way can readmission of patients be identified by matching their first visit to their next visit?

select 
    v1.patient_id,
    v1.visit_id as original_visit,
    v1.visit_date as original_date,
    v2.visit_id as return_visit,
    v2.visit_date as return_date
from hospital.visits v1
inner join hospital.visits v2 on v1.patient_id = v2.patient_id and v1.visit_id < v2.visit_id
order by v1.patient_id
limit 5;

Expected Results

Explanation

Linking visits on patient ID where v1.visit_id is less than v2.visit_id matches distinct visits per patient.

Join Summary

INNER JOIN matches rows in both tables.
LEFT JOIN keeps all left table rows.
RIGHT JOIN keeps all right table rows.
FULL OUTER JOIN keeps all rows from both tables.
CROSS JOIN produces Cartesian pairs.
SELF JOIN compares rows within one table.

3. Common Table Expressions (CTEs)

A CTE is a named temporary table created within a WITH clause.

Simple CTEs

Question: What approach should be used to identify costly inpatient treatments and connect them with patients and their physicians?

with high_cost_visits as (
    select 
        visit_id,
        patient_id,
        doctor_id,
        total_bill_kes,
        admission_type
    from hospital.visits
    where total_bill_kes >= 35000.00
)
select 
    h.visit_id,
    p.patient_name,
    d.doctor_name,
    h.total_bill_kes
from high_cost_visits h
join hospital.patients p on h.patient_id = p.patient_id
join hospital.doctors d on h.doctor_id = d.doctor_id;

Expected Results

Explanation

CTE will filter out high cost visits first to keep the subsequent join clean.

Multiple Chained CTEs

Question: How do we find out department revenues, overall average, and top performing departments?

with dept_revenue as (
    select 
        department_id,
        sum(total_bill_kes) as total_revenue,
        count(visit_id) as total_visits
    from hospital.visits
    group by department_id
),
dept_benchmark as (
    select 
        avg(total_revenue) as benchmark_avg_revenue
    from dept_revenue
)
select 
    d.department_name,
    dr.total_revenue,
    dr.total_visits,
    round(db.benchmark_avg_revenue, 2) as benchmark_avg
from dept_revenue dr
join hospital.department d on dr.department_id = d.department_id
cross join dept_benchmark db
where dr.total_revenue > db.benchmark_avg_revenue;

Expected Results

Explanation

dept_revenue calculates the total revenue for each department, dept_benchmark calculates the average and the last SELECT statement filters departments whose revenues are greater than the average.

Recursive CTEs

Recursive CTEs create a sequence by means of an anchor query and recursive query linked by UNION ALL operator.

Question: How can the hospital management create a biweekly follow up schedule?

with recursive follow_up_schedule as (
    select 
        1 as cycle_number,
        cast('2026/07/01' as date) as scheduled_date
    union all
    select 
        cycle_number + 1,
        cast(scheduled_date + interval '14 days' as date)
    from follow_up_schedule
    where cycle_number < 4
)
select * from follow_up_schedule;

Expected Results

Explanation

Cycle number 1 starts from July 1 2026 as the anchor date, and the recursive query keeps adding 14 days till cycle number is 4.

CTEs vs. Subqueries

CTEs provide well-defined top-level modules, which can be used again and again, while subqueries within the WHERE or FROM clause tend to complicate code and cannot be executed recursively.

4. SQL Subqueries

A subquery is a query nested inside another SQL query.

Subquery Types

Scalar Subquery fetches one row and one column.
Non-correlatedSubquery executes only once and independently.
Correlated Subquery uses columns of outer query and executes for every row of outer query.
Nested Subquery uses subqueries in subqueries.

Subqueries in SELECT Clauses

Question: How can individual patient bills be shown along with the total hospital average bill?

select 
    visit_id,
    total_bill_kes,
    (select round(avg(total_bill_kes), 2) from hospital.visits) as overall_avg_bill
from hospital.visits
where visit_id between 1001 and 1005;

Expected Results

Explanation

Scalar subquery evaluates to average 16266.67 for all rows.

Using Subqueries in FROM Clauses (Derived Table)

Question: How do we query the pre-aggregated metrics of patients using an inline derived table?

select 
    patient_summary.patient_id,
    patient_summary.visit_count,
    patient_summary.total_spent
from (
    select 
        patient_id,
        count(visit_id) as visit_count,
        sum(total_bill_kes) as total_spent
    from hospital.visits
    group by patient_id
) as patient_summary
where patient_summary.visit_count > 1;

Expected Results

Explanation

The FROM subquery groups the number of visits per patient into table alias patient_summary, subject to WHERE.

Subqueries in WHERE Clause

Question: What is the correlated subquery to find visits where the total bill is greater than the average bill of that particular department?

select 
    v1.visit_id,
    v1.patient_id,
    v1.department_id,
    v1.total_bill_kes
from hospital.visits v1
where v1.total_bill_kes > (
    select avg(v2.total_bill_kes)
    from hospital.visits v2
    where v2.department_id = v1.department_id
)
limit 5;

Expected Results

Explanation

The inner query calculates the average billing amount for each department based on outer query department IDs.

Question: Which physicians have admitted any patients as inpatients?

select 
    d.doctor_id,
    d.doctor_name,
    d.doctor_specialty
from hospital.doctors d
where exists (
    select 1 
    from hospital.visits v 
    where v.doctor_id = d.doctor_id 
      and v.admission_type = 'inpatient'
);

Expected Results

Explanation

EXISTS returns true when there is a row for a visit that matches.

Question: How can we find visits where the billing value is greater than all the outpatient visit bills?

select 
    visit_id,
    patient_id,
    total_bill_kes,
    admission_type
from hospital.visits
where total_bill_kes > all (
    select total_bill_kes
    from hospital.visits
    where admission_type = 'outpatient'
)
limit 5;

Expected Results

Explanation

The internal query locates outpatient charges up to 15000.00. ALL filters visits more than 15000.00.

Conclusion

Understanding the concepts of window functions, joins, common table expressions, and subqueries helps in creating strong PostgreSQL queries that enable healthcare workers to review their operations and enhance financial efficiency. Such information enables healthcare providers to offer quality pateint care which meets patient outcomes.

GitHub

For the completed project visit my GitHub project Hospital Database

Instant at 10 Million Rows, Dead at a Billion: A Decision Framework for Billion-Row SQL

Fri, 09 Oct 2026 06:03:06 +0000

A query that returns in 40 milliseconds on ten million rows can time out completely on a billion. Same SQL, same schema, same database. I've watched a lot of teams hit that wall and go hunting for the one fix: a cleverer index, a bigger instance, whatever database was trending that month. There isn't one fix. There are two questions that decide almost everything. What engine class is this workload on — row-store, columnar, or distributed? And does the query's working set fit in memory, or does it spill to disk? Answer those honestly and most "billion-row problems" quietly resolve. Answer them wrong and no amount of tuning bails you out.

The query that was instant at 10 million rows and timed out at a billion

Here's the shape of it. It's a composite; I've seen it often enough that the specifics blur.

A team builds a dashboard on their main OLTP database. One query carries it: count distinct users, grouped by day, over a date range. Against a 10-million-row staging copy it comes back instantly, everyone signs off, ship it. Months later production crosses a billion rows — roughly the 1.1-billion-row scale in Mark Litwintschik's cross-engine benchmarks — and the identical query simply stops returning. On-call adds an index; nothing. A composite index; nothing. A bigger instance; somehow slightly worse. "Throw it at Spark," one person says. "Just use ClickHouse," says another. Nobody can explain why either would help. That's the real problem: everyone is arguing tactics with no framework for the decision.

The query didn't regress. It's doing exactly what it always did. What changed is that the workload crossed two lines at once — it outgrew the engine class that suited it at 10M, and its sorts and hashes stopped fitting in memory. Name those two lines and the mystery turns into a decision you can defend.

The two-axis framework

When a query dies at scale, the reflex is to index harder. Sometimes that's right. At a billion rows it's usually the wrong axis. Two levers actually decide whether a query survives.

Axis 1 — Engine class. Every database is built around one dominant access pattern, and that choice shapes everything downstream. A row-store OLTP engine (PostgreSQL, MySQL) keeps whole rows together and excels at "fetch or update this one record" and narrow ranges. A columnar/OLAP engine (ClickHouse, DuckDB, BigQuery) stores each column separately, so scanning and aggregating hundreds of millions of rows is the design point rather than the emergency. A distributed engine (Citus, Spark, Cassandra) spreads data across nodes for horizontal scale, trading query flexibility to get it. These are peers, not rungs on a ladder; none is "more advanced." Run a workload on the wrong one and you get a query that flies at 10M and dies at 1B.

Axis 2 — Memory budget. Independent of engine class, every non-trivial query builds intermediate structures: sort buffers, hash tables for GROUP BY and DISTINCT, join tables. Fit those in RAM and the operation runs at memory speed. Overflow the memory the engine is allowed, and it spills to disk and falls off a cliff. PostgreSQL ships shared_buffers at 128MB and work_mem at just 4MB; the docs are explicit that a sort or hash exceeding work_mem starts "writing to temporary disk files" (PostgreSQL 18 — Resource Consumption, as of 2026). Four megabytes is nothing at a billion rows. This is the axis most write-ups skip, and it's why the same query is fast one day and glacial the next. The SQL didn't change; the data just got big enough to spill.

The axes are orthogonal. You can pick the right engine class and still hit the memory cliff, or keep memory in check and still be on an engine that's wrong for the query's shape. So the move is: work out which quadrant you're actually in, then act on the axis that matches — change engine class, or change the memory budget — instead of adding another index to the axis that was never the problem.

Engine class 1: Row-store OLTP — great at point lookups, brutal on full-table aggregates

Row-stores are the workhorse for good reason. Ask for "order #48213 and its line items" or "the last 50 events for this user" and a row-store with the right index is hard to beat: it seeks to a handful of rows and returns. At 10M rows, the dashboard query was riding on that.

Aggregates over the whole table are a different story, and the reason is baked into how MVCC row-stores stay correct under concurrency. Because different transactions can see different versions of a row, the engine keeps no cached count of how many rows a table has. An exact count(*) therefore has to visit rows and check visibility — a sequential scan that grows linearly with the table.

The numbers are blunt. In Joe Nelson's Citus benchmark on counting (PostgreSQL 9.5.4, 2016 — treat the milliseconds as illustrative of the mechanism, which is unchanged today, not as current figures), an exact count(*) ran 85ms at 1M rows, 161ms at 2M, and 343ms at 4M: a straight line, with the scan about 88% of the cost. Extend that to a billion rows and a single count is tens of seconds before any filtering. count(1) doesn't rescue you; it measured slightly slower (99ms vs 85ms at 1M), because Postgres special-cases the argument-free count(*) while count(1) re-checks the argument on every row. And this isn't Postgres being difficult — MySQL's InnoDB shares the MVCC property and also has no O(1) table count.

When you don't need an exact number, stop asking for one. Every row-store keeps a maintained estimate for its planner, and you can read it directly. In PostgreSQL that's pg_class.reltuples (the Count-estimate pattern: SELECT reltuples::bigint FROM pg_class WHERE oid = 'schema.table'::regclass). In the same benchmark the estimate returned in about 0.3ms against 85ms for the exact count at 1M rows — roughly 280× faster, and effectively constant time at any size. The catch is accuracy: it's an estimate and can't apply a WHERE or DISTINCT. For "roughly how many rows" on a dashboard, that's the difference between a billion-row scan and reading one catalog row.

None of this means "Postgres is slow." It means a full-table aggregate is off-design for a row-store. Approximate counts, materialized rollups, and summary tables all soften it. But if scans and aggregates are your dominant pattern, you're fighting the engine class — and that's your first crossover signal.

Exact `count() vs the reltuples` estimate. Source: Citus / Joe Nelson, PostgreSQL 9.5.4, 2016 — illustrative of the mechanism, not current figures.*

Engine class 2: Columnar / OLAP — where scans and aggregates are the design point

If aggregates are killing your row-store, what's built for them, and how much faster is it really? This is where the framework gets counter-intuitive.

Columnar engines store each column contiguously instead of packing whole rows. A scan-and-aggregate query that touches 3 of 51 columns reads only those 3 off disk; the other 48 are never opened. And because a column holds one type with heavy repetition, it compresses hard, so there's less to read to begin with. Column pruning plus compression is why "scan a billion rows and aggregate" is routine here instead of an incident.

The headline number: Litwintschik's 1.1-billion-row taxi benchmark (1.1B rows, 51 columns, ~500GB uncompressed; page updated March 2024) runs one analytical query across dozens of engines. On Query 1, a single desktop ran ClickHouse in 0.088s and DuckDB in 0.498s. The same query took 2.36s on a 21-node Spark cluster, 3.54s on 21-node Presto, and 1–3s on BigQuery. Postgres with the cstore_fdw columnar extension took 152–368s across the four queries, and plain SQLite's format took 31,193s on Query 1 — about 8.7 hours, included to show what the wrong engine class actually costs.

A single desktop beat a 21-node cluster on a 1.1-billion-row scan by more than an order of magnitude. That one result is the whole framework in miniature. It isn't that ClickHouse or DuckDB is "the best database" — it's that for a scan-aggregate workload a columnar engine is in the right quadrant while the cluster pays coordination overhead it doesn't need at this size. (Caveats: one benchmark, one hardware generation, recent desktop CPUs for the fastest single-node runs. Trust the shape of the gap, not the exact milliseconds.)

Compression is the other half, and it's an economics lever as much as a speed one. ClickBench — the ClickHouse team's scan/aggregate benchmark across 60+ systems — publishes a live "Data Size" column so you can compare on-disk footprint on the same ~100M-row dataset. Columnar layouts routinely pack this kind of wide, repetitive data down by a large multiple versus a row-store, though the exact ratio moves with engine, codec, and data, so read the live figure for the systems you're weighing rather than trusting a headline number. (ClickBench is ClickHouse-run, so treat it as a scan-workload reference, not a neutral verdict.)

The honest counter-point: the same columnar engine that crushes the scan is the wrong tool for high-QPS single-row writes and point lookups. Insert a row and you touch every column store; update a field and compression works against you. The wide-column and key-value stores built for that job — Cassandra, ScyllaDB — sit at the opposite pole: excellent at point reads and writes on a known partition key, and structurally unfit for ad-hoc full-table aggregates. Cassandra's CQL docs say it plainly: CQL "only allows select queries that don't involve a full scan of all partitions," and forcing one needs ALLOW FILTERING, whose performance the docs call "unpredictable." A store modeled query-first around the partition key isn't a scan-aggregate engine, which is exactly why ClickBench lists Cassandra and ScyllaDB among the systems it couldn't benchmark on that suite. Right engine, wrong job, in both directions.

Same query, same 1.1B-row dataset, across engine classes. Source: Mark Litwintschik, 1.1 Billion Taxi Rides benchmarks, page updated March 2024.

Engine class 3: Distributed — horizontal scale, and the fan-out ceiling

When one node genuinely can't hold the data or serve the writes, distributed engines spread the work across many. The mental model most people carry — more nodes, more speed — holds right up until it doesn't, and knowing where that line sits is the difference between a defensible cluster and a single box with a network bill attached.

The line: does the query push down to the shards, or force a reshuffle? Distribute by tenant_id and group by tenant_id, and each node computes its slice while the coordinator stitches the results together. That's pushdown, and it parallelizes beautifully. But the moment a query needs data spread across nodes — a count(DISTINCT) on a non-distribution column, or a join across the distribution boundary — the engine has to reshuffle rows between workers first. That shuffle is the ceiling.

The Citus benchmark puts numbers on it (100M rows, 8 nodes, 32 shards; PG 9.5.4, 2016 — again mechanism-illustrative). A plain count(*) across the cluster ran about 1.2s. A count(DISTINCT) on the distribution column, which pushes down, ran about 3.4s. On a non-distribution column it can't push down, and pulling every row back to the coordinator is, in Nelson's words, "really no better than counting on a single database instance … plus high network overhead." An HLL approximate distinct ran 3.2–3.8s — the row-store's approximation trick again, now buying a way around the shuffle. This generalizes well past Citus: in Spark the expensive stage is almost always the shuffle, and distributed query design is mostly the craft of keeping the hot queries pushed down.

So distributed isn't a "scale" button. It's a trade — horizontal capacity for query flexibility and operational complexity. Align your access pattern with the distribution key and it's transformative. Keep needing cross-node data and you've bought nodes and network latency to arrive back at roughly single-box performance. That mismatch is its own crossover signal, and the one most likely to be discovered the hard way in production.

The memory / spill axis — the one most write-ups skip

Now the second axis, and the reason "just add an index" so often does nothing. A query can be on exactly the right engine class and still fall off a cliff, because its sorts and hash tables outgrew the memory the engine was allowed and spilled to disk. This is engine-independent: columnar and distributed engines build the same buffers and spill the same way when they overflow their limits. Postgres is just the easiest place to watch it, because its docs name the knob.

The spill cliff

At the default 4MB work_mem (PostgreSQL docs), a GROUP BY, DISTINCT, or ORDER BY that builds something larger switches from an in-memory hash or sort to an external merge that writes temp files to disk. The Citus benchmark shows the size of the drop (PG 9.5.4, 2016; mechanism unchanged per the current docs): at default work_mem, count(DISTINCT) on an integer column averaged 743ms, but on a text column 31,747ms — a ~43× gap — with Sort Method: external merge Disk in the plan. Give the operation enough work_mem to stay in RAM and the integer case drops to about 372ms as an in-memory HashAggregate. That in-RAM-versus-disk boundary is the axis. Stay on the right side and the query flies; cross it and it collapses.

This is exactly why indexing didn't help the composite team. An index changes how you find rows; it does nothing for a GROUP BY/DISTINCT whose result set is too big for work_mem. The lever there is memory, or a smaller working set — not another index.

When to re-architect instead of tune

work_mem is allocated per operation per query, and the docs warn total usage "could be many times" that value under load, so raising it globally just trades a spill problem for an out-of-memory one. Past a point the honest move is structural. PostgreSQL's partitioning guidance offers a clean rule: partition "when the size of the table [would] exceed the physical memory of the database server." Pruning then skips whole partitions — but it isn't free. The planner handles "up to a few thousand" partitions well only if pruning leaves few, since "each partition requires its metadata to be loaded into the local memory of each session that touches it." Thousands of poorly-pruned partitions is its own problem.

One more trap: deep pagination

LIMIT ... OFFSET n gets slower the further you page. As Markus Winand explains in "No Offset", the database "fetches and drops" all n preceding rows before returning the page, and results drift when rows are inserted between requests. A keyset (seek) query filters on the last-seen sort key instead and stays roughly constant-time however deep you go. Page latency that climbs with depth is a spill-adjacent signal, not a hardware one.

count(DISTINCT) at default vs raised work_mem. Source: Citus / Joe Nelson, PostgreSQL 9.5.4, 2016 — illustrative of the mechanism, not current figures.

The crossover-signal checklist: change engine class, don't add another index

This is the piece worth saving. A framework's job is to tell you when you've hit the boundary and it's time to move on an axis instead of tuning further. Eight signals I watch for, each tied to the axis it moves. Two or more firing means you're past tuning and into an engine-class or memory-budget decision.

Scans and aggregates dominate the workload (not point lookups). → Engine class, toward columnar. The row-store aggregate cliff; indexing won't fix an off-design pattern.
The working set exceeds server RAM. → Memory. Postgres's own partition threshold; partition, re-architect, or move to columnar/distributed.
Sorts and DISTINCTs keep spilling even after tuning work_mem. → Memory. You've hit the ceiling of per-query memory; the working set is the problem.
Pagination or scan latency rises with position. → Memory/scan; switch to keyset/seek before switching engines.
A single node has hit its write or IO ceiling. → Engine class, toward distributed. The one genuine "add nodes" signal.
Key DISTINCTs or joins need data from several nodes (a shuffle). → A warning on the distributed axis: without a distribution-key redesign, more nodes ≈ one box plus network.
You need thousands of partitions to stay fast. → Memory/metadata; the per-partition cost says a columnar or distributed store fits the data better.
You're keeping 500GB+ on one box for analytics. → Engine class plus economics; columnar compression and distributed storage costs start driving the decision.

What the list deliberately doesn't do is name a winning database. Each signal points to an axis and a direction, because the right specific engine depends on your query shapes, your team's ops maturity, and your budget. "Change engine class" is an architectural decision with a reason attached — something you can hand a skeptical peer or a budget owner instead of "the internet said use ClickHouse."

Cost and economics for the decision-makers

If you own the stack decision, latency is half the case; the half that gets budget approved is cost and defensibility. The benchmarks reframe cleanly as money.

A single big box can be cheaper and faster than a cluster. The Litwintschik result — one desktop-class node beating a 21-node cluster on a 1.1B-row scan — reads louder as a cost story than a speed one: 21 nodes carry ~21× the compute plus coordination, networking, and a standing ops burden, and still lose on this workload. For "big but not planetary" analytics, the first move is often the right engine class on one well-sized box, not a cluster. Distributed earns its cost when you truly exceed single-node capacity or need the availability.

Compression is a storage-cost lever. A columnar layout shrinks wide, repetitive data by a large multiple (measure it for your engines from ClickBench's "Data Size" column — it varies by engine and codec), which is a recurring cut in bytes stored and, in a warehouse, fewer dollars per query scanned.

Per-query billing cuts both ways. A serverless warehouse (BigQuery ran 1–3s per query here) fits spiky, ad-hoc analytics where an idle cluster would bleed money, and fits badly for a high-frequency dashboard hammering one aggregate, where a right-sized columnar box wins on total cost.

Every price and benchmark here will rotate — pricing shifts, hardware moves the single-node ceiling, vendor benchmarks get re-run. Don't anchor the decision to one dated figure. The framework doesn't rotate: two axes, eight signals. If a price moves, re-run the comparison.

Build it yourself

The framework is cheap until you feel it on your own data. Three projects, beginner to advanced. Each names a real tool (links verified reachable in 2026, but repos move — check before you lean on them).

Project 1 — Count your table three ways (~30 min). Feel the row-store aggregate cliff on real data. You'll need a Postgres or MySQL table with 10M+ rows and DuckDB. Run EXPLAIN ANALYZE SELECT count(*) FROM your_table; and note the time and the Seq Scan; then SELECT reltuples::bigint FROM pg_class WHERE oid = 'schema.your_table'::regclass;; then get a slice into columnar form — export CSV (COPY your_table TO 'slice.csv' (FORMAT csv)) and load it into DuckDB, or read the table directly with pg_duckdb — and count(*) there. Done when you can state your own numbers for all three and point to the Seq Scan. Stretch: add GROUP BY day.

Project 2 — Reproduce the spill cliff (~1–2 hrs). Watch a query cross the RAM→disk boundary. You'll need Postgres and a 1M+ row table with an integer and a text column (the NYC-taxi or ClickBench sets work). At default work_mem, run EXPLAIN ANALYZE SELECT count(DISTINCT int_col) FROM t; then the same on text_col, and confirm Sort Method: external merge Disk on the slow one. SET work_mem = '1GB';, re-run both. Done when you've captured your own before/after ratio and seen the plan flip to in-memory HashAggregate. Stretch: chart latency as you step work_mem from 4MB to 1GB and find your own cliff edge.

Project 3 — Run a slice of the 1.1B benchmark (~half a day). Reproduce the crossover on your hardware. You'll need the same big table in a row-store and a columnar engine, plus the ClickBench loaders (60+ DBMS setup scripts) and Litwintschik's posts for reference. Load an identical scan-aggregate table into each, time one representative query on both, and record the ratio and your hardware. Done when you reproduce a >100× gap — or can explain from the plans why yours differs. Stretch: add a distributed engine and build a query that forces a shuffle to find your own fan-out ceiling.

What to do Monday morning

The whole thing, compressed. When a query dies at scale, stop asking "which index?" and ask the two questions that decide it. Engine class: is the dominant shape point-lookups (row-store), scans and aggregates (columnar), or genuinely bigger than one node (distributed)? Memory budget: does the working set fit in RAM, or is it spilling? Then walk the eight signals; two or more means the move is on an axis, not another index. There's no single answer to "how do I handle a billion rows" — the answer is a decision you can write down and defend.

So go run it. Take your biggest table this week and do Project 1: count(*), then reltuples, then a columnar count in DuckDB. Watch where the plan shows a Seq Scan and how far apart the three numbers land. Want the full crossover? The ClickBench repo has loaders for 60+ engines, so you can time the same scan on a row-store and a columnar engine side by side. Bring back your own numbers — the framework only lands once the cliff is yours.

Inside the Database Stack

Wed, 07 Oct 2026 13:42:37 +0000

Think of a restaurant. Your Django code is the waiter, PostgreSQL is the kitchen, and the disk is the pantry. When dinner is slow, people blame the waiter's handwriting (the ORM). Far more often the waiter is making 200 separate trips to the kitchen, or the cook is checking every shelf in the pantry because nobody labeled them.

This article follows one order from the waiter's pad to the pantry shelf and back, and ends with what actually makes databases slow.

Part I: The Waiter (Python & Django ORM)

Django open-sourced its ORM in 2005. Its core idea is the QuerySet, a description of the data you want rather than the data itself.

orders = Order.objects.filter(status="paid")   # no SQL has run
orders = orders.filter(total__gt=100)          # still nothing
print(orders.query)                            # shows the SQL it *would* run

This is lazy evaluation. The waiter writes the order on the pad but doesn't walk to the kitchen until you actually need food: when you loop over the results, call list(), slice with a step, or ask for len(). Each .filter() returns a new QuerySet, and Django compiles the final chain into a single SQL statement only at the last moment.

That laziness is a benefit. You can build queries across functions without touching the database. It is also a trap, because a harmless-looking attribute access inside a loop can silently send the waiter running.

ORM vs raw SQL. The ORM buys you safety (parameterized queries that block SQL injection), portability, and readable code. It costs you some CPU: building the query objects, compiling SQL, and turning every row into a full Python model instance. For a few hundred rows, this overhead is invisible. For a hundred thousand rows, .values(), .only(), or .iterator() can shrink it considerably.

Part II: Too Many Trips (Query Performance)

The classic failure is the N+1 query problem:

for order in Order.objects.all():          # 1 query
    print(order.customer.name)             # +1 query per order

Show 500 orders and you've sent 501 queries. It is like a waiter walking to the kitchen 500 times for 500 glasses of water. Each trip is fast, but the trips add up, especially when the database sits across a network.

Django has two fixes:

select_related("customer") performs a SQL JOIN and fetches the related row in the same query. Use it for single-valued relations (foreign keys, one-to-one).
prefetch_related("items") runs a second query and stitches the results together in Python. Use it for many-valued relations, where a join would duplicate parent rows.

Order.objects.select_related("customer").prefetch_related("items")

Two metrics matter here, and people often confuse them. Query count is how many trips you made. Query latency is how long each one took. A page with 400 fast queries and a page with 3 slow ones need different cures. Tools like django-debug-toolbar or CaptureQueriesContext show both.

Beyond N+1, the usual bottlenecks are fetching columns you never use, calling len(qs) when qs.count() or qs.exists() would do, saving rows one at a time instead of bulk_create(), and filtering in Python what the database could filter itself.

Part III: The Pantry (Database Fundamentals)

PostgreSQL is a process-per-connection system. Each client gets its own backend process, and all of them share memory buffers and a write-ahead log (WAL), which records changes before they hit the main files so a crash can't lose them.

Rows live in a table stored as fixed-size pages, 8 KB each by default. Imagine a warehouse of unlabeled boxes. Finding "all orders from customer 42" means opening every box. That is a sequential scan, and it is the default when nothing smarter is available.

An index is the label system. The standard type is the B-tree, a balanced, sorted tree dating back to 1970. Like a phone book, it lets Postgres jump to the right place in a handful of steps instead of reading everything.

A composite index covers several columns, and order matters:

CREATE INDEX idx_orders_cust_date ON orders (customer_id, created_at);

This helps queries filtering on customer_id, or on customer_id plus created_at. It barely helps a query filtering on created_at alone, just as a phone book sorted by surname is useless for finding everyone named "Ahmed."

The key idea is selectivity: how well a value narrows the search. An index on customer_id (many distinct values) is sharp. An index on a boolean is_active where 98% of rows are true is nearly worthless, because the database would still read most of the table.

Indexes hurt, too. Every insert, update, or delete must also update every index. They consume disk and memory, and they slow writes. An index is a trade, not a gift.

Part IV: The Kitchen's Brain (Query Planning)

When SQL arrives, PostgreSQL's optimizer doesn't just run it. It considers many possible strategies, estimates the cost of each using table statistics, and picks the cheapest. Think of a GPS comparing routes.

You can ask to see the route:

EXPLAIN SELECT * FROM orders WHERE customer_id = 42;

This shows the plan and the estimated costs without running anything. Add ANALYZE and Postgres actually executes the query and reports real times and row counts:

EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE customer_id = 42;

Careful: EXPLAIN ANALYZE really runs the statement. For an UPDATE or DELETE, wrap it in a transaction and ROLLBACK.

The plan names the access method:

Sequential scan: read the whole table. Fine for small tables, or when you need most rows anyway.
Index scan: walk the index, then fetch each matching row. Great for a few rows.
Bitmap scan: collect matching locations from the index, sort them, then read the pages in order. This is the middle path for "more than a few, fewer than most."

For joins, three algorithms compete: nested loop (for each row on the left, look up the right, good for small inputs), hash join (build a lookup table in memory, good for large unsorted inputs), and merge join (zip two sorted lists together).

The most useful habit is comparing the planner's estimated rows with the actual rows. A big gap usually means stale statistics, and ANALYZE table_name; often fixes it. The optimizer is only as good as its guesses.

Part V: Many Diners, One Kitchen (Transactions & Concurrency)

A transaction bundles steps into one all-or-nothing unit. Databases promise ACID, a term formalized in 1983:

Atomicity: all steps happen, or none do.
Consistency: rules (constraints) always hold.
Isolation: concurrent transactions don't trample each other.
Durability: once committed, it survives a crash.

PostgreSQL delivers isolation through MVCC (multi-version concurrency control). Instead of making readers wait for writers, it keeps multiple versions of a row. Readers see a consistent snapshot while writers create new versions. Readers don't block writers, and writers don't block readers. The price is cleanup: old versions pile up until VACUUM removes them.

Postgres defaults to Read Committed isolation, where each statement sees data committed before it began. Repeatable Read and Serializable are stricter, and Serializable can abort transactions with errors you must retry.

Snapshots don't stop every conflict. Take a classic race condition:

acct = Account.objects.get(pk=1)
acct.balance -= 50      # two requests both read 100, both write 50
acct.save()             # one withdrawal vanishes

The fix is to lock the row:

from django.db import transaction

with transaction.atomic():
    acct = Account.objects.select_for_update().get(pk=1)
    acct.balance -= 50
    acct.save()

select_for_update() takes a row lock, so the second request waits its turn. For simple arithmetic, an F("balance") - 50 expression pushes the math into the database itself and is even safer.

Locks bring the risk of deadlocks: transaction A holds row 1 and wants row 2, while B holds row 2 and wants row 1. It is two people in a narrow hallway, each waiting for the other to step aside. Postgres detects this (after deadlock_timeout, 1 second by default) and kills one transaction. The cure is to lock rows in a consistent order everywhere.

Part VI: How to Measure Honestly

I'm deliberately not printing benchmark numbers here. Figures from my machine would be meaningless on yours, and invented ones would be worse. What matters is the method:

Fix the environment. Same hardware, same Postgres version, same config, warm cache and cold cache runs.
Scale the dataset. Test at 1K, 100K, and 10M rows. Many problems only appear at scale, and a missing index looks harmless on a small table.
Measure several things: latency (report p50 and p95, not just the average), throughput (queries per second under concurrency), memory (tracemalloc), and query count.
Compare like with like: ORM vs equivalent hand-written SQL, and indexed vs unindexed, changing one variable at a time.

import time
from django.db import connection
from django.test.utils import CaptureQueriesContext

with CaptureQueriesContext(connection) as ctx:
    start = time.perf_counter()
    list(Order.objects.select_related("customer")[:1000])
    elapsed = time.perf_counter() - start

print(len(ctx), "queries,", round(elapsed * 1000, 1), "ms")

Run it many times, discard the first (cold) run, and keep the distribution.

Part VII: Findings

What actually causes slowness. In most web applications, the usual suspects are, in rough order of how often they appear: too many queries (N+1), missing or unusable indexes on filtered columns, fetching far more rows or columns than needed, and long-held locks. Rarely is the culprit that Python is "too slow" at building SQL.

Where the ORM is expensive. Materializing huge result sets into model instances, hidden per-row queries, and ORM-generated queries nobody has ever inspected.

Where ORM optimizations matter. select_related, prefetch_related, only()/defer(), values(), bulk_create(), exists() and iterator(). These change the shape of the work, and that is where the big wins live.

Where indexes matter. On columns used in WHERE, JOIN, and ORDER BY, in large tables, with high selectivity. Always confirm with EXPLAIN that the index is actually used.

When raw SQL is justified. Complex reporting, window functions or CTEs the ORM expresses awkwardly, or a measured hot path where the ORM's generated query is demonstrably poor. Use parameterized queries, never string formatting.

The trade-offs. Readability and safety against fine control. Fast reads against slower writes (indexes). Strict isolation against throughput.

Myths worth retiring:

"The ORM is slow." Usually the queries are slow, not the ORM.
"Raw SQL is always faster." Not if it has the same plan and the same number of round trips.
"More indexes, more speed." Each one taxes every write.
"An index will always be used." The planner may rightly ignore it.
_"select_related everywhere."_ Joining tables you never read just makes the kitchen work harder.

The one-sentence takeaway: measure first, count your queries, read your plans, and only then optimize.

References

Find me across the web:

Portfolio: ahmershah.dev
Crunchbase: @syed-ahmer-shah
Crunchbase Company: @syedahmershah
Clutch: @syed-ahmer-shah
Tech Behemoth: @syed-ahmer-shah
Design Rush: @syed-ahmer-shah
Edverise: @syed-ahmer-shah
Trust Pilot: @ahmershah.dev
LinkedIn: Syed Ahmer Shah
GitHub: @ahmershahdev
AWS Builder Profile: @syedahmershah
DEV: @syedahmershah
Medium: @syedahmershah
Hashnode: @syedahmershah
Substack: @syedahmershah
HackerNoon: @syedahmershah
Substack: @syedahmershah
Facebook: @ahmershahdev
Linkedin Page: @syedahmershah
YouTube: @ahmershahdev
Instagram: @ahmershahdev
TikTok: @ahmershahdev

Reverse-Engineering Messy Databases: My End-to-End Data Audits Across 17,000+ Relational Schema Logs

Wed, 23 Sep 2026 11:56:39 +0000

From Raw Logs to Business Strategy: How I Taught Myself Data Engineering & Analytics

I didn’t take the traditional path into data. Over the past 12 months, I bypassed the university lecture halls to teach myself relational database architecture, pipeline optimization, and growth analytics by diving straight into raw datasets.

I look at data through an engineering and business lens. Instead of building passive charts that just look pretty, I query transaction logs and user event data to find hidden operational friction, map out user behavior patterns, and isolate exactly where systems are dropping value. I build end-to-end data systems: raw SQL schemas, data sanitization scripts, interactive Power BI dashboards, and single-page executive reports.

Below are the architectural breakdowns, technical challenges, and business impact metrics of the four core data infrastructure projects I have engineered from scratch.

Project 1: StockPulse — Multi-Regional Supply Chain Audit

Technical Stack: MySQL (Mathematical Window Functions), Power BI, Advanced Excel
GitHub Codebase & Dashboard Layout: [LINK]

The Operational Bottleneck

Horizon Marketplace faced shrinking profit margins from slow-moving warehouse stock, while simultaneous supplier shipping delays caused high-demand items to go out of stock on our mobile application. Traditional procurement tracking models failed because they relied on simple historical averages, which immediately miscalculate when data hits high seasonal sales volatility spikes.

The Technical Implementation

Audited 17,646 operational and sales ledger transactions using MySQL window functions to calculate cumulative revenue contributions per SKU.
Coded a predictive restocking matrix script utilizing sales volatility standard deviations and rolling manufacturer lead timelines.
Formulated dynamic Safety Stock buffers and programmed automated database Reorder Point triggers to flag supplier order alerts before shelves hit zero.

The Business Impact Metrics

ABC inventory segmentation proved that exactly 23 Class A items drive 80% of total marketplace revenue, showing that the remaining catalog space was tying up critical capital.
Quantified a hidden revenue bleed of $63,897.26 caused by a single high-demand apparel product sitting entirely out of stock for 42 days due to upstream logistics bottlenecks.
Transitioned a multi-regional operation from manual, time-intensive spreadsheets to an automated, predictive data framework.

Project 2: FunnelOpt — HabitArcade B2C Telemetry Audit

Technical Stack: MySQL (Session Overrides), Funnel Analysis, User Lifecycle Mapping
GitHub Codebase & Database Logs: [LINK]

The Operational Bottleneck

A B2C application tracking engine suffered from severe data execution drops where product behavioral logs completely failed to map user activation patterns. The relational schema dropped child telemetry records because they were importing prior to the absolute database finalization of the parent account index logs.

The Technical Implementation

Audited full-funnel growth logs across 6 independent acquisition pipelines to analyze customer conversion latency and drop-offs.
Structured specific MySQL session overrides to temporarily bypass structural foreign key constraints.
Forced clean data synchronization across child records to completely restore parent-child index synchronization without corrupting historical system logs.

The Business Impact Metrics

Isolated a systemic “Streak-Breaking Crash Tax” where application sync errors wiped 30-day user milestones, triggering immediate 1-star reviews and an active 6% to 7% monthly subscriber bleed.
Exposed an “Inverted Value Curve” revealing that high-value annual checkout tiers underperformed monthly subscriptions due to silent gateway billing processor failures blocking top-priced subscription cash settlement.

Project 3: LeakStop — Olist B2B Revenue Leakage & Onboarding Audit

Technical Stack: MySQL (Schema Engineering), Power BI, Business Intelligence
GitHub Codebase & Analytical Readme: [LINK]

The Operational Bottleneck

Cross-vendor data duplication across digital marketplace ledgers was heavily skewing gross reporting metrics. Standard relational database joins double-counted total transactions when a customer order spanned multiple independent business vendors in a single checkout loop.

The Technical Implementation

Engineered robust MySQL ingestion and data sanitization scripts to enforce strict primary key integrity down to individual item vectors, removing all reporting bias.
Processed user account lifecycles to calculate onboarding completion latency across specific B2B verticals.
Programmed a real-time database filter to flag slipping user accounts precisely on Day 16 of transaction inactivity.

The Business Impact Metrics

Isolated an annual revenue leakage of R$ 600,600 (approximately $119,175 USD) stemming from 462 merchant accounts that stalled out completely during the onboarding phase.
Exposed an invisible latency anomaly where logistically complex business profiles stalled for an average of 112 days in domain routing and tax mapping phases.
Implemented a Day 16 early-warning database filter to flag at-risk accounts before they reached a terminal, zero-recovery retention cliff.

Project 4: CohortPace — SaaS Behavioral Retention Engine

Technical Stack: MySQL (Subqueries & Relational Joins), Cohort Modeling, Data Analytics
GitHub Codebase & Technical Report: [LINK]

The Operational Bottleneck

Initial application dashboards focused blindly on feature exploration depth, which completely masked why healthy-looking active user profiles were suddenly canceling subscriptions. Traditional active-user metrics acted as a lagging indicator, hiding friction because churning accounts remain active inside broken workflows right up to their exit timelines.

The Technical Implementation

Modeled simulated user cohort transactions across multi-platform SaaS behavioral profiles to analyze the core drivers of long-term Monthly Recurring Revenue (MRR).
Used advanced MySQL subqueries and relational joins to reconstruct full lifecycle interaction sequences.
Mapped user interaction velocity trends against subscription lifecycles to isolate clear behavioral churn signatures.

The Business Impact Metrics

Proved through data pacing sending signals that churning users maintain consistent interaction velocity close to their exit date, showing churn is driven by late-stage workflow utility limits rather than gradual disengagement.
Demonstrated that long-term user monetization is strictly dependent on habit formation within a single core workflow rather than broad, multi-feature platform exploration.

Conclusion: Turning Code Into Business Strategy

I don’t need a playbook to get started, and I don’t need a manager to hand me a static checklist. I find the question, ingest the data, clean the schemas, model the answer, and present the metric that directly protects business margins.

I am actively available for analytical remote contract roles, standalone data audits, and freelance pipeline optimization. If your startup is scaling fast and dealing with messy relational pipelines, let’s connect and fix your backend infrastructure loops.

Interactive Portfolio Hub: [LINK]
Contact: motiwala603@gmail.com
Connect with me: [LinkedIn]

Open to analyst remote roles across product, growth, and business analytics. Based in Pakistan, available worldwide.

Speed Up Your Queries: A Plain-English Guide to Database Indexing

Fri, 11 Sep 2026 02:43:19 +0000

What is Database Indexing?

Database indexing is a technique used by software systems to speed up the retrieval of data from a database table. By creating a separate, organized pointer structure, the database can locate specific records without having to search through every single row of data. Think of it as a specialized shortcut map for your digital filing cabinet.

The Library Card Catalog Analogy

Imagine walking into a massive metropolitan library containing 100,000 books. If you want to find a book titled "The Midnight Mystery" and there is no catalog or organization system, you would have to start at the first shelf in the corner and look at every single book cover one by one until you find it. This slow, painful process is what a database does during a "full table scan."

Now, imagine the library has an alphabetical card catalog. You walk straight to the drawer marked "M," find the index card for "The Midnight Mystery," and read the card. It tells you exactly which aisle, shelf, and position the book is on. You walk directly there and grab it in seconds. In this scenario, the card catalog is the index, the books are the rows of data in your table, and the physical location is the memory address of the database record.

Why It Matters Daily in Tech

In the tech industry, engineers use database indexing to prevent application lag and catastrophic server crashes. Without indexes, as a company's database grows from thousands of rows to millions, simple daily actions like logging in, searching for an item, or pulling up a user profile would take seconds or even minutes instead of milliseconds.

Slow database queries hog the system's central processing unit (CPU) and memory, causing other requests to back up and eventually freeze the entire application. By adding appropriate indexes, developers keep application programming interfaces (APIs) lightning-fast, reduce cloud hosting bills (because servers do not have to work as hard), and ensure a smooth user experience. However, engineers must find a balance: every index takes up storage space and slows down write operations (like adding or updating data), because the database has to update both the main table and the index every single time a change is made.

How It Works in Practice

Here is a simple example in SQL (Structured Query Language). Imagine a system trying to look up a user by their email address.

-- Without an index, the database must scan every row in the users table
SELECT * FROM users WHERE email = 'developer@example.com';

-- To speed this query up, we create an index on the email column
CREATE INDEX idx_users_email ON users(email);

-- Now, the database uses the index to jump directly to the correct user
SELECT * FROM users WHERE email = 'developer@example.com';

The Takeaway

Database indexing is the ultimate balancing act of software performance optimization. It turns grueling, resource-heavy data searches into near-instant lookups, keeping applications snappy as they scale up to handle millions of users. Just remember that indexes are not free; treat them like directory signs in a physical store—use them where visitors frequently get lost, but do not clutter every wall with them, or you will slow down the inventory restocking process.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Three Steps That Decide Whether Plain English Database Queries Actually Work

Mon, 31 Aug 2026 02:43:48 +0000

Most of us here have either built a text-to-SQL layer or been asked to. The demos always look great and the production version is the one that quietly returns the wrong number, so here is what actually separates the two.

The Schema Is The Feature, Not The Prompt

A natural language query layer is not a prompt engineering problem. When you connect a database, the model has to read the whole schema first: every table, every column, every data type, every foreign key. That is what lets it decide which tables to touch, which columns to filter on, and how to join them.

Skip that step and you still get SQL back. It runs, it returns rows, and it is wrong in a way nobody catches, because the join went through the wrong key. The systems that hold up are the ones that expose the generated SQL so a human can spot-check it, and that read relationships rather than guessing them from column names. There is a longer walkthrough of how that works in practice on asking your database questions in plain English.

Dirty Data Breaks It Before The Model Does

The second failure mode is upstream. Names stored three different ways, phone numbers in four formats, the same customer entered twice with a typo, blank fields where a required value should be. Your query is correct and your answer is still wrong, because COUNT DISTINCT on a column with fuzzy duplicates is not counting what you think.

This is the least glamorous part and it is where most of the accuracy lives. Fuzzy matching to find near-duplicate records, normalizing formats across the table, validating each field against the rules it should obey, flagging rows with missing required values instead of silently averaging around them. A practical rundown is in cleaning data with AI.

Let The Model Find What You Did Not Ask About

The third step is the one teams skip entirely. Every query you write encodes something you already suspected. Anomaly detection covers the rest: point anomalies like an order fifty times the average, contextual ones like revenue that is normal for a retailer and impossible for a two person consultancy, and collective ones where ten individually boring transactions land in the same minute.

That last category is the argument for running it at all, because no dashboard you would have thought to build catches it. Worth reading how AI anomaly detection actually runs if you are adding this to an existing stack.

The Takeaway

Plain English querying is not one feature, it is three: clean the data, give the model the real schema, then let it surface what your questions never covered. Ship it in that order and the demo and the production system finally agree with each other.

A SQL interview is a translation test, not a syntax test

Sat, 22 Aug 2026 14:27:55 +0000

Four-Leaf's SQL Interview Guide

Key takeaways

A SQL interview is a translation test. The syntax is the easy half, and most candidates who fail were fluent in SQL. What sinks them is starting to type before the question is pinned down, or writing a correct query that answers something nobody asked. The round scores three things: whether you clarify before you write, whether your query is correct on the edge cases, and whether you can explain your reasoning while your hands are moving. It also looks meaningfully different depending on whether you're interviewing as a data analyst, a data scientist, or an engineer.

What does a SQL interview test?

The round tests whether you can turn an underspecified business question into a query that survives contact with real data. Interviewers rarely hand you a clean specification, because the job never does either.

SQL is worth preparing for on volume alone. The Stack Overflow 2025 Developer Survey found 58.6% of developers had used SQL in the past year, making it the third most-used language behind JavaScript at 66% and HTML/CSS at 61.9%, and ahead of Python at 57.9%. It shows up in loops for roles that aren't nominally data roles at all, which is part of why candidates underprepare for it.

The tell that separates a strong candidate is what happens in the first sixty seconds. Weak candidates read the prompt and start typing. Strong candidates restate the question, ask what counts as an active user or a completed order, confirm whether the answer should include rows with nulls, and only then write. That opening exchange is often worth more to the interviewer than the query itself, because it's the part of the job that can't be looked up.

What kinds of SQL questions come up?

Three shapes cover most of what gets asked, and they escalate in a predictable order.

The first is a join and aggregation question. Given two or three tables, produce a count or a sum grouped by something. These look trivial and catch people on join type, because an inner join silently drops the users who never ordered, and the question usually wanted them counted as zero.

The second is a window function question. Rank purchases per customer, find each user's second transaction, compute a running total, or calculate a month-over-month change. This is where a lot of loops separate candidates, because window functions are the boundary between people who write SQL occasionally and people who use it as a primary tool.

The third is an open business question against a schema you've just been shown. "Tell me whether retention improved after the March release." There's no single correct query. The interviewer is watching how you decompose the question, what you decide retention means, and whether you notice that the March cohort has less time to churn than the February one.

How is a SQL interview scored?

Three dimensions, and they map to those question shapes. It helps to run each one against the questions the round keeps returning to: give me a count by group, rank something per user, and answer this vague business question.

Do you clarify before you write? A passing answer asks one question about the schema. A strong answer names the ambiguity that would change the query and resolves it. "Does an active user mean any event in the window, or a purchase?" is the difference between two very different numbers, and interviewers plant that ambiguity on purpose.

Is your query correct on the edges? A passing answer runs. A strong answer accounts for nulls, duplicates, ties in a ranking, and rows on the boundary of a date range. Say the edge case out loud when you handle it, because a silent LEFT JOIN looks identical to a lucky one.

Can you narrate while you write? A passing answer goes quiet and produces a query. A strong answer talks through the plan first, writes, then checks the result against a rough expectation out loud. Interviewers score reasoning they can hear, and a correct query delivered in silence gets a weaker write-up than a slightly imperfect one that was explained.

How does a SQL interview differ for a data analyst versus a data scientist?

Same language, meaningfully different round, and mixing them up is the most common preparation mistake in the function.

Data analyst. The emphasis is breadth and business translation. Expect more questions, less depth per question, and heavy weighting on whether your numbers would hold up in front of a stakeholder. Definitional precision matters more than query elegance. You're likely to be asked what a metric should mean before you're asked to compute it, and an answer that flags a misleading denominator scores higher than one that optimizes a join.

Data scientist. The emphasis shifts toward experiment and cohort logic. Expect fewer questions with more depth, and expect at least one that touches an A/B test readout, a cohort definition, or a sampling problem hiding inside the SQL. A question about whether a March cohort had less time to churn than a February one is a data scientist question. Correctness under a statistical framing is what's being scored, not query volume.

Analytics engineer. The round moves toward modeling. Expect questions about how you'd structure the model rather than write a one-off answer, including idempotency, incremental logic, and what happens when the query is rerun.

Software engineer. SQL usually appears inside a broader technical round rather than on its own, and the framing is performance and correctness at scale. Expect indexing, query plans, and the consequences of a full table scan rather than business definitions.

Product management. Increasingly common and usually light. The bar is whether you can pull your own numbers without asking an analyst, so expect one aggregation question and no window functions.

Four-Leaf covers the wider data loop in how to prepare for a data science interview.

What is overrated

Memorizing exotic syntax. Recursive CTEs and pivot tricks appear in practice sets far more often than in interviews, and time spent there is time not spent on window functions, which appear constantly.

Speed is the other overrated thing. Candidates rush because they assume the clock is the test, then miss a join type. The interviewer is almost always more interested in whether you caught the null case than in whether you finished ninety seconds early.

Using an AI assistant to get through this round is a losing trade. CodeSignal reported in February 2026 that cheating and fraud attempts on proctored assessments more than doubled, from 16% in 2024 to 35% in 2025. Interviewers noticed. A 2025 interviewing.io survey of 67 of them, 52 at FAANG companies, found 81% suspected candidates of using AI and about a third had caught someone at it. The visible response has been more live, narrated rounds, which is precisely the format where a candidate who can't explain their own query falls apart.

A five-step playbook for a SQL round

Drill window functions until they're automatic. ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, and a running sum. These carry more interview weight per hour of study than anything else in SQL.

Practice restating the question out loud before writing. Give yourself a rule that you don't type until you've named one ambiguity.

Build a null and duplicate checklist. Before you call a query done, ask what happens to rows with nulls, rows that appear twice, and rows exactly on the boundary date.

Do at least three problems on a whiteboard or plain text editor. Autocomplete hides gaps that a bare editor exposes, and many live rounds use a bare editor.

Narrate a solved problem end to end. Explaining a query you already understand is a separate skill from writing it, and it's the one being scored. A voice mock interview is a reasonable way to rehearse the narration, since the failure mode is going quiet under pressure rather than not knowing the syntax.

Where this is heading

SQL rounds are getting more conversational, and the reason is the same force reshaping every technical loop. As assistants make it trivial to produce a syntactically correct query, interviewers move the bar to the part that's harder to fake, which is deciding what to compute and defending the choice. Four-Leaf's AI-Era Hiring Index, a study of 3,502 open roles at 16 top tech employers, found LLM or foundation-model experience listed in 57% of data and machine learning job descriptions against 21% of engineering ones, so the expectation that data candidates work alongside these tools is already written into the postings.

That direction rewards the candidate who understood the question. The syntax was never the hard part, and it's about to matter even less. Where this round sits in the wider sequence is covered in how an onsite interview loop actually works. To rehearse it out loud rather than read about it, start with eight coding interview platforms, compared.

We caught a SQL injection with an offline AI security scanner — here's the exact query it found

Sat, 22 Aug 2026 07:06:23 +0000

https://youtu.be/GknBmsqaFk4

The bug

Here's a login endpoint from a small Express demo app
(scan-target-demo-apps/apps/01-sql-injection):

app.post('/login', (req, res) => {
  const { username = '', password = '' } = req.body;
  const query = `SELECT id, username, email, is_admin FROM users
                  WHERE username = '${username}' AND password = '${password}'`;
  const result = db.exec(query);
  // ...
});

You've seen this shape before. username and password go straight into the SQL string — no
parameter binding, no escaping. Submit admin' -- as the username and anything as the password,
and the query becomes:

SELECT id, username, email, is_admin FROM users
WHERE username = 'admin' -- ' AND password = 'anything'

-- comments out the rest of the line. The password check never runs. {"ok": true}, logged in
as admin.

Classic CWE-89, OWASP A03:2021. Nothing novel here — the interesting part is what caught it.

What actually caught it

We ran this target through AI Security Studio, an offline security research platform we've
been building. No cloud calls, no code leaves the machine — everything you're about to read
happened on a laptop with no network access to anywhere but localhost:3001.

The important design decision: the LLM never sees raw source code or raw HTTP traffic.
The pipeline looks like this:

Parser → Rule Engine → Knowledge Retrieval → Summarization → LLM → Reasoning → Finding → Report

A deterministic parser and rule engine do the actual finding. The local LLM only explains
already-discovered evidence — it's not asked to eyeball a codebase and guess where the bugs
might be. If the evidence isn't there, the platform is built to say "Needs Manual Verification"
instead of inventing a conclusion.

For this run, that meant:

Point the scanner at http://localhost:3001 as scope.
Let it run a real passive crawl — no exploitation, just discovery.
Run a source scan against the target's own directory.
The rule engine flags the string-concatenated query pattern in server.js:56 and the missing
auth check on the state-changing POST /login route.

The generated report includes the actual vulnerable line as evidence, not a generic "possible SQL
injection" label:

Source preview.
  app.post('/login', (req, res) => {
    const { username = '', password = '' } = req.body;
    const query = `SELECT id, username, email, is_admin FROM users WHERE username = '${username}' AND password = '${password}'`;

Risk: High. Root cause: CWE-89 (and a separate CWE-306 finding for the missing authorization
check). Suggested fix: parameterized queries. Confidence explicitly marked — some findings from
this pass are "Confirmed," others are flagged "Needs Manual Verification" rather than asserted,
because header/config-level evidence alone doesn't prove exploitability the way a manual
Active Test / Repeater run does.

Why "offline" is the actual point

It's easy to gloss over "runs locally" as a checkbox feature. In practice it means:

You can point it at unreleased/internal code without a vendor's cloud pipeline ever touching it.
The rule engine — not the LLM — does secret detection, JWT/header/cookie analysis, and pattern
matching, so results are reproducible and don't drift between runs.
The LLM's job shrinks to something it's actually reliable at: turning structured findings into a
clear explanation, not hunting for vulnerabilities in raw text.

Try it yourself

The demo app is intentionally vulnerable and open source — safe to point any scanner at, including
this one:

Demo target: https://github.com/sendwavehub/scan-target-demo-apps
AI Security Studio: https://apps.microsoft.com/store/detail/9PJ0J7BK1M27?cid=DevShareMCLPCB
{% embed https://sendwavehub.tech/th/apps/ai-security-studio-4 %}

Built for security researchers, pentesters, and AppSec teams doing work they're authorized to do.

Next up in this series: stored XSS — one comment field, every visitor who loads the page.

I added DuckDB to my SQL playground, and it turned out to be a sandboxing problem

Thu, 20 Aug 2026 12:38:29 +0000

I added DuckDB to my SQL playground, and it turned out to be a sandboxing problem

Sharing a small build-in-public update: sqlize.online, a free tool for testing SQL queries, checking JOINs, and sharing reproducible examples, just got DuckDB support alongside the ~30 engines it already had (MySQL, Postgres, SQL Server, Oracle, Firebird, ClickHouse, SQLite...).

I'd wanted to add DuckDB for a while — I use it a lot in my own projects and like having local OLAP around. The holdup was boring: the whole thing runs on PHP, and PHP's PDO talks to MySQL, Postgres, and SQLite out of the box, but had nothing for DuckDB.

That changed when I found pdo_duckdb by Ilia Alshanetsky — a real PDO driver for it. That's the piece that made this worth doing.

Why DuckDB specifically

Short version: it's "SQLite for analytics." Vectorized engine, built for heavy aggregations on a single core, not for transactions.

What actually got me was that it reads Parquet on the fly, from disk or over HTTP, with no separate load step. For a playground, that's great — instead of a two-row toy table, you can hand people something real. So I preloaded NYC's Yellow Taxi trip data for January 2024, about three million rows, sitting in a table called yellow_tripdata:

SELECT
    PULocationID,
    COUNT(*) AS total_trips,
    ROUND(AVG(total_amount), 2) AS avg_fare
FROM yellow_tripdata
GROUP BY PULocationID
ORDER BY total_trips DESC
LIMIT 10;

That's a table, not a file path — more on why that matters below.

Building the driver, plus an unrelated Debian surprise

pdo_duckdb needs PHP 8.1+. My setup runs several PHP versions side by side for different engines, so that already cuts out a couple of pools. Worse, the latest release only ships prebuilt binaries for PHP 8.4/8.5 — on 8.1, where this feature currently lives, I had to build from source: phpize && ./configure --with-pdo-duckdb=... && make install, linked against libduckdb.so. Not exotic, but not a one-command install either — about half the engines in this project get built by hand at this point.

While I was at it I hit a completely unrelated bug: bumped that PHP pool to the latest patch release, and the image build broke for no obvious reason. Turned out the base image had quietly moved from Debian bookworm to trixie, which renamed libaio1 to libaio1t64. Rolled the base image back. Cost me about forty minutes of confusion for something that had nothing to do with DuckDB at all.

The actual problem: DuckDB runs inside the PHP process

MySQL and Postgres live in their own containers and can't see the PHP worker's filesystem at all. DuckDB is embedded — it opens up right inside PHP-FPM. Good for speed, no sockets or IPC in the way. Bad because by default it can reach anything the process can reach: COPY ... TO '/etc/whatever', read_csv from any path, read_parquet from any URL, INSTALL for arbitrary extensions.

Not something you want available to arbitrary user SQL.

First attempt was the obvious one — set it in SQL directly:

SET enable_external_access = false;
SET allow_community_extensions = false;
SET lock_configuration = true;

This genuinely works, and lock_configuration = true really does stop the same session from undoing it later. But the driver has its own separate protection: it flatly refuses a chunk of security-sensitive settings — paths, extension autoload — when they're passed via the DSN or PDO::DUCKDB_ATTR_CONFIG at connect time. On purpose, so the app using the driver can't accidentally weaken its own config.

The real mechanism turned out to be PHP's own open_basedir. Set it — even at runtime, with ini_set() — and the driver shuts off DuckDB's entire external-access surface: read_csv, read_parquet, COPY, ATTACH, httpfs, INSTALL, all of it, no matter which path gets requested. Regular CREATE TABLE/INSERT/SELECT against the session's own file keep working fine.

ini_set('open_basedir', '/code:/tmp/sessions:/tmp/databases');
$pdo = new PDO("duckdb:{$sessionFile}");

I tried stacking both — open_basedir plus the SET statements on top, just for extra safety. Didn't work: once open_basedir is active, the driver rejects allow_community_extensions outright ("not allowed when open_basedir is set") and the connection breaks. Had to pick one. Kept open_basedir — it covers everything on its own.

And yes, there was a bug. I was applying open_basedir right before running a query, but not before handing a raw PDO connection to user-written PHP code — which happens when someone runs PHP and SQL together and picks DuckDB as the engine. So: submit a PHP script with no SQL in the separate field, get back a completely unrestricted $pdo, and $pdo->exec("COPY (SELECT 1) TO '/tmp/whatever.csv'") just works. Caught it myself before it shipped anywhere public.

Why the dataset isn't loaded on every request

Three million rows over HTTPS on every session would be brutal — a cold build takes anywhere from 5 to 35 seconds. So it's built once at deploy time, by a separate script using a plain connection with no open_basedir set (downloading the file there is a legitimate job for that script, not for user SQL), and cached to a shared file. Every session after that is just a copy() — a fraction of a second.

Try to break it

If you find a way to read a server file or get around open_basedir, I'd rather hear about it directly than see the service go down. No bug bounty budget, but I'll credit you publicly if you find something real.

Try it here: sqlize.online — pick "DuckDB" or "DuckDB Preloaded" as the SQL version. Happy to answer questions about how any of this is built.

From GitHub Repositories to a Professional Data Analytics Portfolio Website

Sat, 08 Aug 2026 07:35:02 +0000

Building More Than Just Projects — Building My Professional Identity

Completing my Data Analyst Internship at Logic Stack was a significant milestone. Over four weeks, I worked on real-world business problems using Excel, Power BI, Python and SQL, successfully completing four practical analytics projects.

Like many learners, I initially thought that uploading my projects to GitHub would be enough.

After all, GitHub is the standard platform for developers and data professionals.

But then I asked myself a simple question:

If a recruiter visits my profile, will they truly understand my journey just by opening four separate repositories?

The answer was No.

That realization inspired me to build something much bigger than another GitHub repository.

It inspired me to build my own Production-Level Data Analytics Portfolio Website.

Why GitHub Alone Wasn't Enough

GitHub is excellent for storing source code and documentation.

However, recruiters usually spend only a few minutes reviewing a candidate's profile.

Opening multiple repositories one by one makes it difficult to understand:

The overall learning journey
Project progression
Skills gained
Business impact
Technical growth

I wanted to create a place where everything could be explored from a single professional dashboard.

The Idea

Instead of sharing multiple independent repositories, I designed a portfolio website that presents my internship as one complete journey.

Rather than saying,

"Here are my projects."

I wanted the website to tell a story:

How I started, what I learned, which challenges I faced, how I solved them and what I built by the end of the internship.

That became the foundation of the entire portfolio.

Designing the Portfolio

The goal was simple:

Create a clean, modern, recruiter-friendly website that highlights both my Data Analytics skills and my Web Development experience.

The portfolio includes carefully designed sections that guide visitors through my internship journey.

Home

A professional landing page introducing my internship, background, and career focus.

Projects

A dedicated section for all four internship projects.

Each project contains:

Project overview
Technologies used
GitHub Repository
LinkedIn post
YouTube explanation video

Everything is connected from one location.

Skills Learned

Instead of simply listing technologies, I organized everything I learned during the internship into one dedicated section.

The portfolio highlights skills including:

Microsoft Excel
Power BI
Python
SQL
Data Cleaning
Exploratory Data Analysis
KPI Development
Business Intelligence
Dashboard Design
Documentation

This gives recruiters a quick overview of my technical capabilities.

Challenges Solved

One of my favourite sections is the Challenges Solved page.

Instead of hiding the problems I encountered, I documented them professionally.

Examples include:

Python environment setup
Library installation issues
SQL query debugging
Data cleaning problems
Documentation challenges
GitHub repository organization

Each challenge is paired with the solution I implemented.

I believe problem-solving demonstrates professional growth more effectively than simply showing successful projects.

Certificate & Internship Documents

The portfolio also includes my internship certificate and professional documents.

To keep them organized, I implemented a simple password-protected section that allows authorized viewers to access these files when needed.

This keeps the homepage clean while still making important documents available.

Building with Modern Technologies

This project also became an opportunity to strengthen my web development skills.

The website was built using modern technologies including:

Next.js
TypeScript
Tailwind CSS
Framer Motion
Responsive Design
SEO Optimization
Structured Data (JSON-LD)
Robots.txt
Sitemap.xml

By combining these technologies, I transformed a simple portfolio into a production-ready web application.

SEO Matters Too

I didn't want the website to look professional only for people who already had the link.

I wanted search engines to understand it as well.

To improve visibility, I implemented:

Metadata optimization
Open Graph tags
Structured Data
Robots configuration
XML Sitemap
Performance optimization

Achieving high Lighthouse scores became another goal during development, ensuring the website follows modern web standards.

Combining Two Different Skill Sets

This project became much more than a portfolio.

It combined two different fields I enjoy:

Data Analytics

Business insights
SQL
Power BI
Python
Excel

Web Development

Next.js
TypeScript
UI Design
Responsive Layouts
SEO
Performance Optimization

Bringing these two disciplines together allowed me to create something that represents my complete technical profile.

Lessons Beyond the Internship

Building this website taught me several important lessons.

A portfolio is not just about displaying work.

It is about presenting your journey professionally.

I learned that:

Presentation matters.
Documentation matters.
User experience matters.
Personal branding matters.
Professional storytelling matters.

A recruiter should be able to understand who you are within minutes.

That became the guiding principle behind this project.

Final Thoughts

Looking back, I'm glad I didn't stop after uploading four repositories.

Creating this portfolio allowed me to organize my internship experience into one professional platform that showcases both my technical skills and my continuous learning journey.

This website represents more than projects.

It represents persistence.

It represents growth.

It represents every challenge solved throughout the internship.

Most importantly, it reminds me that learning never truly ends.

Every project becomes a stepping stone toward the next opportunity.

Explore My Complete Portfolio

🌐 Live Portfolio:
https://yasirawaninfo-data.vercel.app/

💻 GitHub:
https://github.com/YasirAwan4831

Thank you for taking the time to explore my journey. I hope it inspires other learners to build not only projects—but also a professional identity around their work.

#DataAnalytics #SQL #SQLite #PowerBI #Python #Panda #Numpy #Matplotlib #Excel #BusinessIntelligence #DataVisualization #Dashboard #DataCleaning #JupyterNotebook #ProblemSolving #GitHub #WebDevelopment #LearningInPublic #Portfolio #CareerGrowth ##MuhammadYasir #YasirAwan4831

Continuing My Data Analyst Journey: Week 3, 4 and Challenges I Faced During My Data Analyst Intern

Fri, 07 Aug 2026 19:22:22 +0000

In my previous blog, I shared my journey through Week 1 and Week 2, where I worked on Excel-based data cleaning, sales analysis, Pivot Tables and Power BI dashboards.

As the internship progressed, the learning curve became much steeper. The focus shifted from spreadsheet analytics to programming, databases and solving real-world business problems using professional data analysis tools.

Weeks 3 and 4 introduced completely new technologies, larger datasets and more technical challenges that pushed me far beyond my comfort zone.

These two weeks transformed my internship from learning business reporting into building complete end-to-end data analytics solutions.

Week 3: Supply Chain Analytics using Python & Power BI

Week 3 marked my first serious experience with Python for Data Analytics.

Unlike previous weeks, this project required setting up a complete Python development environment before writing a single line of analysis. I worked extensively with Pandas, NumPy, Matplotlib and Jupyter Notebook to clean, analyse and visualize a real-world supply chain dataset containing thousands of records.

After completing the Exploratory Ddata Analysis (EDA), I integrated the insights into Power BI, creating an interactive dashboard that highlighted sales performance, shipment modes, product groups, vendors and country-wise business metrics.

This project taught me that successful data analysis begins with clean, reliable data—and that Python is one of the most powerful tools for transforming raw datasets into meaningful business insights.

GitHub Repository: https://github.com/YasirAwan4831/week-3-python-powerbi-supply-chain-analytics

YouTube Video: https://youtu.be/SQk41XxxpsY?si=ElhxePDt916bh_VJ

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_eda-dataabranalyst-fullabrstackabrdevel

Week 4: Funnel & Revenue Analysis using SQL & Power BI

The final week introduced one of the most important skills in modern data analytics: SQL.

For this project, I worked with SQLite to analyse customer funnel performance, revenue generation and conversion metrics directly from a relational database. Instead of relying on spreadsheets, I learned how businesses retrieve insights using SQL queries, aggregate functions, filtering, grouping and KPI calculations.

The extracted data was then connected to Power BI, where I built an interactive business dashboard presenting conversion rates, revenue trends and overall funnel performance.

This project completed my internship journey by combining database querying, business intelligence and visualization into one complete analytics workflow.

GitHub Repository: https://github.com/YasirAwan4831/week-4-sql-powerbi-funnel-analysis

YouTube Video: https://youtu.be/sdBYOKqT6YM?si=HfNw1CrgG71dHIzi

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_sqlite-sqlabrbased-dataabranalys

From Learning Tools to Solving Problems

By the end of Week 4, I had worked with:
Microsoft Excel
Power BI
Python
SQL
Business Analytics
Data Cleaning
Exploratory Data Analysis (EDA)
KPI Dashboard Development
Funnel Analysis
GitHub Documentation

However, learning these tools was only part of the journey.
The real growth came from overcoming technical challenges, debugging errors, researching solutions and continuously improving every project.

10 Real Challenges I Faced During My Data Analyst Internship (And How I Solved Them)

Every Internship Teaches Skills — But Real Growth Comes From Solving Problems

When people look at a completed project, they usually see the dashboard, the charts, the SQL queries, or the final GitHub repository.

What they don't see are the countless hours spent fixing errors, learning unfamiliar tools, debugging environments, researching documentation and solving problems that never appeared in any tutorial.

During my one-month Data Analyst Internship at Logic Stack, I completed four industry-based projects covering Excel, Power BI, Python, SQL and Business Analytics.

However, the most valuable outcome wasn't just the projects themselves.

It was learning how to solve problems independently.

This blog shares the biggest technical and practical challenges I faced during my internship—and how each one made me a better Data Analyst.

Week 3 Project

Supply Chain Analytics using Python & Power BI

By Week 3, the internship shifted from Excel into Python-based analytics.

This project focused on cleaning, analysing and visualizing a real-world Supply Chain dataset using Python (Pandas, NumPy, Matplotlib) alongside Power BI.

Unlike previous weeks, this project introduced a completely new development environment, making it both exciting and challenging.

Week 4 Project

Funnel & Revenue Analysis using SQL & Power BI

The final week introduced SQL for business analytics.

The objective was to analyse conversion funnels, revenue performance and customer behaviour using SQL queries before presenting business insights inside Power BI dashboards.

This project simulated real-world reporting workflows where raw database queries become executive dashboards.

Challenge 1

Starting With Zero Practical Experience

Challenge

Although I had experience in web development, I had almost no practical exposure to professional Data Analytics workflows.

Everything—from Excel reporting to SQL analytics—was new.

Solution

Instead of trying to learn everything at once, I followed a structured approach:

Learning one technology at a time helped me build confidence while creating portfolio-ready projects every week.

Challenge 2

Building a Professional Excel Workflow

Challenge

Creating clean spreadsheets is easy.

Creating professional business reports is not.

I had to learn:

Structured Tables
Pivot Tables
Professional Formatting
Multiple Worksheets
Formula Organization

Solution

I researched Excel best practices and redesigned my workbook several times until it followed professional reporting standards.

The result was an organized multi-sheet Excel project that later became the foundation for my Power BI dashboard.

Challenge 3

Learning Power BI Dashboard Design

Challenge

Power BI wasn't difficult because of the software.

It was difficult because dashboards must answer business questions—not simply display charts.

Solution

I learned:

KPI Cards
DAX Basics
Dashboard Layout Principles
Visual Hierarchy
Business Storytelling

Rather than creating colourful visuals, I focused on dashboards that communicate insights clearly.

Challenge 4

Configuring the Python Development Environment

Challenge

This was one of the toughest technical challenges.

Issues included:

VS Code Python interpreter problems
Jupyter Kernel connection failures
Package installation errors
Environment configuration
Version conflicts

These problems had nothing to do with analytics—but they completely blocked development.

Solution

Instead of giving up, I researched official documentation, debugging guides and community discussions.

Eventually I:

Fixed interpreter paths
Installed required libraries
Connected Jupyter successfully
Stabilized the development environment

This experience taught me that environment setup is a real engineering skill.

Challenge 5

Cleaning a Real-World Dataset

Challenge

The Supply Chain dataset contained:

Missing values
Mixed data types
Invalid numeric values
Text inside number columns

Real business datasets are rarely clean.

Solution

Using Pandas, I:

Converted invalid values safely
Filled missing values
Removed inconsistencies
Validated cleaned data

Only after proper cleaning did visualization become meaningful.

Challenge 6

Installing SQLite & Building SQL Workflows

Challenge

Week 4 introduced SQL.

Before writing queries, I first had to configure SQLite and import datasets correctly.

Solution

I installed DB Browser for SQLite, verified database structure, imported CSV datasets correctly and prepared the environment before beginning SQL analysis.

This preparation prevented many future errors.

Challenge 7

Writing Complex SQL Queries

Challenge

Simple SELECT statements were easy.

Business reporting required:

GROUP BY
Aggregate Functions
Filtering
Revenue Calculations
Conversion Metrics

Solution

Instead of memorizing syntax, I practiced writing increasingly complex queries while documenting each command inside my repository.

This improved both my SQL knowledge and documentation skills.

Challenge 8

Turning Data Into Business Insights

Challenge

Anyone can generate numbers.

A Data Analyst must explain what those numbers actually mean.

Solution

For every project I focused on answering business questions:

Which products perform best?
Where is revenue growing?
Which funnel stage loses customers?
What actions should businesses take?

This mindset transformed technical work into business analysis.

Challenge 9

Professional GitHub Documentation

Challenge

Writing code is only half the job.

Presenting projects professionally is equally important.

Solution

I created consistent repositories with:

Detailed README files
Professional documentation
SVG animations
Shields.io badges
Folder organization
Screenshots
Business explanations

These repositories became portfolio-quality projects rather than simple assignments.

Challenge 10

Building My Own Data Analyst Portfolio Website

Challenge

Instead of simply uploading four GitHub repositories, I wanted to create something unique.

Solution

I designed and developed a complete interactive portfolio website using Next.js and TypeScript.

The website includes:

Internship Overview
Four Complete Projects
GitHub Repositories
LinkedIn Posts
YouTube Project Videos
Skills Learned
Challenges Solved
Password-Protected Certificate Section
Resume Download
Modern Responsive UI
SEO Optimization

This project combined my web development background with my newly acquired Data Analytics skills, creating a portfolio that reflects both technical implementation and professional presentation.

What This Internship Really Taught Me

This internship wasn't just about learning Excel, Python, SQL, or Power BI.

It taught me:

How to research independently.
How to debug unfamiliar problems.
How to organize professional projects.
How to document technical work.
How to communicate business insights.
How to keep learning when solutions aren't immediately obvious.

These lessons will remain valuable long after the internship ends.

Final Thoughts

Looking back, the dashboards and repositories represent only the visible outcome.

The real achievement was overcoming every obstacle that appeared along the way.

Every error message, every failed installation, every debugging session and every rewritten query contributed to my growth as a Data Analyst.

The internship has ended—but the learning journey continues.

Thank you, Logic Stack, for providing a practical environment where I could build real projects, solve real problems and gain valuable industry experience.

Connect With Me

🌐 Portfolio
https://yasirawaninfo-data.vercel.app/

💻 GitHub
https://github.com/YasirAwan4831

💼 LinkedIn
https://linkedin.com/in/yasirawan4831

Thank you for reading. If this journey inspires someone starting their own Data Analytics path, then sharing these experiences has been worthwhile.

#DataAnalytics #SQL #SQLite #PowerBI #BusinessIntelligence #DataVisualization #Dashboard #GitHub #Internship #LogicStack #DataAnalyst #LearningJourney #OpenToWork #Portfolio #YasirAwan4831

My Data Analyst Internship Journey: From Learning the Fundamentals to Building Production-Ready Anal

Fri, 07 Aug 2026 06:12:48 +0000

Introduction

Every learning journey begins with a single decision.

For me, that decision was to step into the world of Data Analytics—a field that combines problem-solving, business understanding, and technology to transform raw data into meaningful insights.

When I joined the Logic Stack Data Analyst Internship, I wasn't simply looking to complete another internship. My goal was much bigger: I wanted to build practical skills, work on real-world projects, document every step professionally and create a portfolio that genuinely reflected my growth.

Over the course of one month, I worked on multiple projects involving Excel, Python, SQL, Power BI, GitHub, technical documentation, and portfolio development. Every week introduced new concepts, new challenges and new opportunities to improve.

This blog shares my complete journey—from the first day of the internship to the completion of the final project. Rather than focusing only on the outcomes, I want to share the process, the lessons I learned and how each project helped me grow as an aspiring Data Analyst.

Why I Chose This Internship

The field of Data Analytics has become one of the fastest-growing domains in technology. Organizations across every industry rely on data to make better decisions, optimize operations, and understand customer behaviour.

I wanted hands-on experience with the tools and workflows used by professionals instead of learning only through theory.

This internship provided exactly that opportunity.

It focused on:

Practical business datasets
Real analytical workflows
Professional documentation
Dashboard development
Industry-standard tools
Portfolio building

Instead of completing isolated exercises, every week contributed to a larger learning journey.

About the Logic Stack Internship

The internship was designed as a structured four-week program where every week introduced a new challenge.

Rather than repeating similar exercises, each project focused on solving a different business problem using different analytical techniques.

Throughout the internship I worked with:

Microsoft Excel
Python
Pandas
NumPy
SQL (SQLite)
Power BI
GitHub
Technical Documentation

Every project required not only completing the technical work but also presenting it professionally through GitHub repositories, documentation, dashboards, screenshots and project reports.

My Goals Before Starting

Before beginning the internship, I set a few personal goals for myself:

Learn modern Data Analytics tools through practical work.
Improve my analytical thinking.
Build projects suitable for a professional portfolio.
Learn proper GitHub documentation.
Create dashboards that communicate business insights effectively.
Gain confidence working with real datasets.

Looking back now, these goals helped me stay focused throughout the internship.

Week 1 – Learning the Foundations

The first week focused on building a strong analytical foundation.

Instead of jumping directly into advanced visualization, the emphasis was placed on understanding datasets, identifying business questions, cleaning information and extracting useful insights.

This project taught me that successful data analysis begins long before dashboards are created.

Some of the concepts I worked on included:

Understanding structured datasets
Data cleaning
Exploratory analysis
Identifying patterns
Basic business insights
Professional project documentation

More importantly, I learned how to organize a project professionally from the beginning—a habit that became extremely valuable during the following weeks.

GitHub Repository: https://github.com/YasirAwan4831/week-1-retail-sales-excel-analysis

YouTube Video: https://youtube.com/shorts/-vCaOvUj-DA?si=4DzXsE1v3Idt84xs

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_excel-logicstack-dataanalyst-activity-

Week 2 – Building Analytical Confidence

The second week pushed my skills further.

The datasets became larger, the questions became more analytical and the expectations shifted from simply calculating values to explaining business performance.

This project strengthened my ability to:

Interpret business metrics
Compare trends
Identify opportunities
Present findings clearly
Think from a business perspective instead of only a technical perspective

One important lesson from this week was that data alone has very little value unless it is converted into actionable insights.

That mindset completely changed how I approached the remaining internship projects.

GitHub Repository: https://github.com/YasirAwan4831/week-2-excel-powerbi-sales-dashboard

YouTube Video: https://youtube.com/shorts/JLg-B-rs0gE?si=kCWG7KHxdxw2d-Pa

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_weekabr2-excel-powerabrbi-activity-

Continuing the Journey...

By the end of the first two weeks, I had gained confidence working with datasets, understanding business requirements, and documenting projects professionally.

However, the most challenging—and most rewarding—part of the internship was still ahead.

In the next part of this journey, I'll cover:

Week 3: Supply Chain Analytics with Python & Power BI
Week 4: SQL + Power BI Funnel Analysis
Building production-quality GitHub repositories
Creating a modern Data Analytics Portfolio Website
Earning my internship completion certificate
Key lessons and reflections from the entire experience

The second half of this journey represents the point where I transitioned from completing internship tasks to building projects that reflect my professional capabilities as a Data Analyst.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ---------------------------------------------------
#DataAnalytics #Portfolio #SQL #PowerBI #Python #Excel #GitHub #WebDevelopment #LearningInPublic #CareerGrowth #OpenToWork Muhammad Yasir #MuhammadYasir #YasirAwan4831

How to get body of the query using aiosql module? is that enough?

Tue, 14 Jul 2026 02:42:44 +0000

In Python development, keeping your database queries clean often leads us to use aiosql, which lets us write pure SQL in .sql files and load them directly as callable Python methods. However, when building APIs or debugging, you often need to inspect, log, or manipulate the actual raw SQL query string from inside your application. Here is how you can access the underlying SQL query body dynamically, the hurdles you will face, and how to elegantly solve them.

Extracting the Query Body DirectlyTo retrieve the raw SQL string of any loaded query, you do not need to read the physical files again. The aiosql library attaches metadata directly to your query objects once they are loaded. Every query loaded onto your queries namespace has a .sql attribute containing the raw SQL body.

You can access metadata like the .operation type to see if it is a SELECT, INSERT, or UPDATE statement. Pythonimport aiosql

Load queries from your local SQL directory

queries = aiosql.from_path("./queries", "sqlite3")

Fetch the raw SQL query body directly from the method

raw_sql = queries.get_user_by_id.sql
print(raw_sql)

Real-World Hurdles You Will EncounterWhile fetching the raw string is simple, relying on it for runtime debugging brings a few modern development challenges.Parameter placeholder mismatch: The raw .sql string still holds the un-interpolated named placeholders like :user_id or ?, meaning it cannot be copied directly into a database GUI for manual testing.Driver-specific syntax shifts: Depending on the driver you bind (such as asyncpg or sqlite3), the way parameters are prepared can alter how the underlying SQL engine interprets the query. Missing runtime values: The raw query string itself does not carry the actual live parameters that were passed during the execution context.

The SolutionsTo get around these limitations and use your query bodies effectively for logging, auditing, or dry-running, you can implement these clean workarounds.Inspect metadata side-by-side: Combine the .sql attribute with the .parameters attribute to map variables to their placeholders programmatically before execution.Intercept with context managers: Write a lightweight wrapper to capture the query state dynamically, making it incredibly easy to log exact executed query structures.Integrate dry-run formatting: Build a helper function that swaps out :placeholder keys with mock values for easier diagnostic testing.Python# Extract parameters alongside the query text
query_func = queries.get_user_by_id
print("SQL Body:", query_func.sql)
print("Expected Parameters:", query_func.parameters)
By leveraging these built-in attributes, you can easily inspect your database interactions without cluttering your clean SQL file structure.

Sumita
Web Developer

SQL Syntax Highlighter

Thu, 09 Jul 2026 12:51:06 +0000

Lightweight SQL Syntax Highlighter for the Browser: Why I Built sql-highlighter

If you need a lightweight SQL syntax highlighter for docs, tutorials, or training platforms, this article explains a practical approach that works without heavy dependencies.

My name is Slava Rozhnev. I build free SQL learning projects sqlize.online and sqltest.online, where users practice SQL on real examples.

In this post, I share:

why I created my own browser SQL highlighter
how it works internally
how to support modern SQL keywords and functions
how to extend highlighted tokens without changing the library itself

TL;DR

I built sql-highlighter, a small dependency-free JavaScript SQL highlighter, because standard configurations in larger libraries did not fully cover modern, dialect-specific SQL used in educational content.

The implementation uses ordered regex passes with safe handling of strings/comments, supports light and dark themes, and allows extending keywords/functions via SQLHighlighter.extend(...) without updating the core script.

Why I Built a Custom SQL Syntax Highlighter

On sqltest.online, I initially used highlight.js for SQL code highlighting.

In practice, many modern or dialect-specific SQL words were not highlighted. I forked the SQL definition, added missing keywords, and submitted a pull request, but it was rejected.

The reason: those additions were not part of standard SQL.

That is reasonable for a universal library. But for a learning platform, I needed practical coverage of real-world SQL syntax. So I built a focused highlighter designed for this exact use case.

What sql-highlighter Is

sql-highlighter is a dependency-free SQL syntax highlighter for the browser.

It is designed to be simple to adopt:

no framework required
no mandatory build step
works with regular script and CSS includes
also available via npm

Repository and docs: README.md

What Gets Highlighted

The library highlights the SQL tokens that matter most in real queries and training material:

SQL keywords
built-in SQL functions
data types
string literals
comments
numbers

Function names are highlighted only when used as function calls, which helps avoid false positives when the same word is used as an identifier.

How the Highlighter Works

The architecture is intentionally small: dictionaries + regular expressions + sequential text passes.

A key detail is processing order. Multi-word constructs such as SELECT DISTINCT and GROUP BY are processed before single-word keywords. This prevents partial or repeated matching after a longer construct has already been highlighted.

Algorithm:

Find strings and comments and replace them with temporary placeholders.
Highlight keywords, functions, constants, and types.
Highlight numbers while avoiding matches inside identifiers.
Restore placeholders as .comment and .string spans.

The goal is not full SQL parsing. The goal is fast, reliable SQL highlighting for web pages.

Why Regex Instead of a Full SQL Parser

For this product, a full parser would be too heavy:

more maintenance complexity
larger frontend footprint
more dependencies and parser-specific edge cases

With careful ordering and token isolation, regex gives an effective balance of quality, speed, and bundle size.

Extend Keywords and Functions Without Updating the Script

This is one of the most useful capabilities.

You can extend supported SQL keywords/functions for your dialect directly in your project, without forking or modifying the base library.

SQLHighlighter.extend({
  keyWords: ['MERGE', 'PIVOT'],
  functions: ['ARRAY_AGG', 'STRING_AGG'],
  constants: ['CURRENT_ROLE'],
  types: ['JSONB', 'UUID']
});

SQLHighlighter.highlightCodeBlocks();

This keeps the core script stable while allowing product-level customization for specific SQL courses or database dialects.

Styling, Theming, and UX

The project uses token classes and CSS variables, so it is easy to integrate into existing design systems.

It also supports both light and dark themes, which is useful for modern docs, tutorials, and developer portals.

Who Should Use sql-highlighter

Use this tool if you need to:

highlight SQL in learning content or technical documentation
add SQL code highlighting to a site quickly
avoid heavy dependencies for a simple frontend feature
customize highlighted SQL keywords/functions by dialect

Development Notes

I used Claude for parts of the development workflow to speed up repetitive implementation tasks and quickly validate ideas.

Architecture and behavior decisions were driven by real requirements and user feedback from my SQL learning platforms.

Roadmap

Next improvements include:

broader SQL dialect coverage
expanded automated test coverage
improved theming and UX integration

If you build SQL education tools or SQL-heavy docs, feedback and contributions are welcome.

FAQ

Is this a full SQL parser?

No. It is a regex-based highlighter designed for speed and practical browser usage.

Can I add custom SQL keywords?

Yes. Use SQLHighlighter.extend(...) to add custom keywords, functions, constants, and types.

Do I need to fork the project to support my SQL dialect?

No. Dialect-specific extensions can be added in your own code without changing the core library.

Is it suitable for production websites?

Yes, especially when you need fast, lightweight SQL highlighting for docs, tutorials, and educational content.

Repository: sql-highlighter

Thanks for reading.

How to Format SQL Queries for Better Readability

Thu, 09 Jul 2026 12:43:06 +0000

You know the feeling. Someone shares a query in Slack, or you open a .sql file left behind by a developer who's long gone, and it's just... one giant wall of text. Lowercase here, uppercase there, no line breaks, five joins crammed into a single sentence. You end up scrolling sideways just to figure out what table a column came from.

That's not a "them" problem. Every SQL developer has written a query like that under deadline pressure. But queries get read far more often than they get written, and a badly formatted one costs everyone — including future you — real time.

This guide walks through practical, no-nonsense SQL formatting best practices you can start using today. Every rule comes with a before-and-after example, so you can see exactly why the "after" version is easier to work with.

Why SQL Readability Matters

SQL isn't a private notebook. It's a shared language between developers, analysts, and database administrators — often people who didn't write the original query and have no context for it.

Readable SQL matters because:

You'll debug faster. When a report breaks at 2 AM, you don't want to spend twenty minutes just figuring out what a query does before you can even start fixing it.
Code reviews go quicker. A reviewer can catch a logic error in seconds when the structure is clear. They can't catch anything in a 400-character single line.
Onboarding is smoother. New team members inherit your queries. Clean formatting is a small kindness that saves them hours.
Mistakes get caught earlier. Misplaced AND/OR logic, missing join conditions, and typo'd column names all hide more easily in dense, unformatted SQL.

Here's a quick taste of what a difference formatting makes, even for a simple SELECT:

Before:

select id,name,email,created_at from users where status='active' and created_at>'2023-01-01' order by created_at desc;

After:

SELECT
    id,
    name,
    email,
    created_at
FROM users
WHERE status = 'active'
    AND created_at > '2023-01-01'
ORDER BY created_at DESC;

Same query, same result set — but the second version tells you what's happening at a glance. Each column has its own line, each clause starts on its own line, and the keywords stand out from the data. That's the whole game.

Why Poorly Formatted SQL Becomes Hard to Maintain

Unformatted SQL doesn't just look messy — it actively works against you over time.

Git diffs turn into noise. If your whole query lives on one line, changing a single condition rewrites the entire line in version control, making it impossible to see what actually changed.
Logic errors hide in plain sight. A missing parenthesis or a misplaced OR is easy to miss when everything blends together.
Nested queries lose their shape. Without indentation, you can't tell which SELECT belongs to which level of nesting.
Every edit gets riskier. When you can't quickly understand a query, you're more likely to introduce a bug while changing it.

Common Mistake: Mixing tabs and spaces in the same file. It looks fine in your editor but turns into a jagged mess in anyone else's, since editors render tab width differently.

The Real Benefits of Consistent SQL Formatting

Consistent formatting compounds over time. A few concrete payoffs:

Faster reviews. Reviewers spend their attention on logic, not on decoding structure.
Predictable structure. Once you know the pattern, you can scan any query on the team and know where to look for the WHERE clause, the joins, or the aggregation.
Cleaner diffs. One clause or column per line means version control shows exactly what changed — nothing more.
Fewer "wait, what does this do?" moments during incident response, when you least want to be puzzling over syntax.

Common SQL Formatting Mistakes Developers Make

Before we get into the fixes, here's what tends to go wrong most often:

Writing the entire query on a single line.
Random, inconsistent indentation from line to line.
Inconsistent capitalization — Select, SELECT, and select all in the same file.
Vague, meaningless aliases like a, b, t1, t2.
No spacing around operators (total>100 instead of total > 100).
Mixing comma-first and comma-last styles in the same codebase.
Deeply nested subqueries with no indentation to show the hierarchy.
Skipping comments on non-obvious business logic.
Lines so long you have to scroll horizontally to read them.
Join conditions that aren't aligned or grouped clearly.

If a few of those look familiar, don't worry — the rest of this guide fixes each one.

Proper Indentation Techniques

Indentation is the backbone of readable SQL. The rule is simple: subordinate clauses and conditions get indented under the keyword they belong to.

Pick either 2 or 4 spaces (never tabs, since they render inconsistently across editors and terminals) and stick with it everywhere.

Before:

select * from orders where (status='pending' or status='processing') and customer_id in (select id from customers where region='west' and active=1);

After:

SELECT *
FROM orders
WHERE (status = 'pending' OR status = 'processing')
    AND customer_id IN (
        SELECT id
        FROM customers
        WHERE region = 'west'
            AND active = 1
    );

Pro Tip: Indent each subquery one level deeper than its parent. That way, nesting depth is something you can see, not something you have to mentally track.

Formatting SELECT Statements

Once you have more than two or three columns, put each one on its own line. It's easier to scan, and it means adding or removing a column later only touches one line in your diff.

You'll see two common styles for commas:

Comma-last (most common, reads like prose):

SELECT
    id,
    name,
    email
FROM users;

Comma-first (makes missing or trailing commas easy to spot):

SELECT
    id
    , name
    , email
FROM users;

Both are valid. What matters is picking one and using it consistently across your team — mixing the two in the same codebase is what actually causes confusion.

Pro Tip: Avoid SELECT * outside of quick, exploratory queries. An explicit column list is easier to read, easier to review, and won't silently break your application if someone adds a column to the table later.

Formatting WHERE Clauses

Every condition gets its own line. Boolean operators (AND, OR) go at the start of the line, indented under WHERE.

Before:

SELECT * FROM orders WHERE status='shipped' AND total>100 AND customer_id IS NOT NULL AND created_at BETWEEN '2024-01-01' AND '2024-12-31' AND region IN ('west','east','central');

After:

SELECT *
FROM orders
WHERE status = 'shipped'
    AND total > 100
    AND customer_id IS NOT NULL
    AND created_at BETWEEN '2024-01-01' AND '2024-12-31'
    AND region IN ('west', 'east', 'central');

Now you can scan down the left edge and read the filter logic like a checklist, instead of parsing one long run-on sentence.

Pro Tip: When you're mixing AND and OR in the same clause, always use parentheses — even when operator precedence would technically get it right without them. It removes any doubt for the next reader.

Formatting JOINs

Always use explicit JOIN ... ON syntax instead of old-style comma joins. Put each join on its own line, with the ON condition right after it.

Before:

select o.id, c.name, p.title, o.total from orders o, customers c, products p, order_items oi where o.customer_id=c.id and oi.order_id=o.id and oi.product_id=p.id and o.status='completed';

After:

SELECT
    o.id,
    c.name,
    p.title,
    o.total
FROM orders o
INNER JOIN customers c ON c.id = o.customer_id
INNER JOIN order_items oi ON oi.order_id = o.id
INNER JOIN products p ON p.id = oi.product_id
WHERE o.status = 'completed';

Explicit joins separate "how tables connect" from "what rows I want," which makes the whole query easier to trace top to bottom. You can also add or remove a join without touching the WHERE clause at all.

Common Mistake: Comma joins put join logic and filter logic in the same place. Forget one join condition, and you've accidentally created a cartesian product — a silent bug that can quietly multiply your row count.

Formatting GROUP BY, ORDER BY, and HAVING

Keep every major clause — SELECT, FROM, WHERE, GROUP BY, HAVING, ORDER BY — starting at the same indentation level (column zero). This "clause skeleton" lets you see the shape of a query in about two seconds.

Before:

select customer_id, count(*) as order_count, sum(total) as total_spent from orders where status='completed' group by customer_id having sum(total)>500 order by total_spent desc;

After:

SELECT
    customer_id,
    COUNT(*) AS order_count,
    SUM(total) AS total_spent
FROM orders
WHERE status = 'completed'
GROUP BY customer_id
HAVING SUM(total) > 500
ORDER BY total_spent DESC;

If you're grouping or ordering by several columns, break those out too:

GROUP BY
    customer_id,
    region,
    order_year

Formatting Nested Queries and Subqueries

Each level of nesting gets its own indentation level. This is where formatting discipline pays off the most, because nested logic is exactly where bugs like to hide.

Before:

select name from products where id in (select product_id from order_items where order_id in (select id from orders where customer_id=42));

After:

SELECT name
FROM products
WHERE id IN (
    SELECT product_id
    FROM order_items
    WHERE order_id IN (
        SELECT id
        FROM orders
        WHERE customer_id = 42
    )
);

Pro Tip: If you find yourself nesting more than two or three levels deep, that's usually a sign to reach for a CTE instead. It's the same logic, but far easier to read top to bottom.

Formatting Common Table Expressions (CTEs)

CTEs turn nested, hard-to-follow subqueries into named, sequential steps. Here's the same logic from above, rewritten as a CTE:

WITH customer_orders AS (
    SELECT id
    FROM orders
    WHERE customer_id = 42
),
ordered_items AS (
    SELECT product_id
    FROM order_items
    WHERE order_id IN (SELECT id FROM customer_orders)
)
SELECT name
FROM products
WHERE id IN (SELECT product_id FROM ordered_items);

A few formatting habits that make CTEs easier to work with:

Name each CTE for what it represents, not cte1, cte2.
Indent the body of each CTE one level in, and keep the final SELECT unindented so it's obvious that's the main query.
Put a blank line between CTEs when you have more than one — it gives your eyes a place to rest between logical steps.

Aliasing Tables and Columns Clearly

Once you're joining more than one table, aliases stop being optional in practice. Use short, meaningful aliases — o for orders, c for customers — rather than arbitrary letters like a and b that force the reader to scroll back up to the FROM clause to remember what's what.

Also alias every computed or aggregated column with AS, so the result set is self-explanatory:

SELECT
    o.id,
    COUNT(oi.id) AS item_count,
    SUM(oi.price * oi.quantity) AS order_total
FROM orders o
INNER JOIN order_items oi ON oi.order_id = o.id
GROUP BY o.id;

Without AS order_total, that column would show up as some generated, meaningless expression in your output.

Consistent Capitalization Strategies

The most common convention: uppercase for SQL keywords, lowercase or snake_case for identifiers.

SELECT customer_id, order_date
FROM orders
WHERE status = 'completed';

Some teams — particularly in the modern data/analytics engineering world — prefer everything lowercase, including keywords:

select customer_id, order_date
from orders
where status = 'completed';

Both are legitimate SQL style guide choices. What actually matters is picking one for your team and applying it everywhere, so readers don't have to context-switch between styles from file to file.

Spacing Conventions

Small, consistent spacing habits go a long way:

Space around comparison and arithmetic operators: total > 100, not total>100.
Space after every comma: id, name, email.
No space between a function name and its parenthesis: COUNT(*), not COUNT (*).
One space before and after AS.

-- Avoid
SELECT id,name,COUNT (*) AS cnt FROM orders WHERE total>=100;

-- Prefer
SELECT id, name, COUNT(*) AS cnt
FROM orders
WHERE total >= 100;

Line Breaks

Use a new line for:

Every major clause (SELECT, FROM, WHERE, JOIN, GROUP BY, ORDER BY, HAVING).
Every column, once you have more than two or three.
Every JOIN.
Each CTE, with a blank line separating them for visual breathing room.

The goal isn't "more line breaks are always better" — it's giving each logical piece of the query its own visual space so your eyes can jump straight to the part you need.

Formatting CASE Statements

CASE expressions get unreadable fast when crammed onto one line. Break each WHEN onto its own line, indented under CASE, with END aligned back at the CASE level.

Before:

select id, case when total>1000 then 'high' when total>500 then 'medium' else 'low' end as tier from orders;

After:

SELECT
    id,
    CASE
        WHEN total > 1000 THEN 'high'
        WHEN total > 500 THEN 'medium'
        ELSE 'low'
    END AS tier
FROM orders;

This layout makes it trivial to scan the conditions in order and confirm the logic is doing what you expect — especially useful when there are five or six WHEN branches instead of two.

Formatting Window Functions (Briefly)

Window functions can get long fast once you add PARTITION BY and ORDER BY inside the OVER() clause. Break those onto their own indented lines so the window definition doesn't blend into the rest of the SELECT list:

SELECT
    customer_id,
    order_date,
    total,
    SUM(total) OVER (
        PARTITION BY customer_id
        ORDER BY order_date
    ) AS running_total
FROM orders;

This keeps the "what am I calculating" (SUM(total)) visually separate from the "how is it grouped and ordered" (the OVER() clause) — two different ideas that shouldn't be squeezed into one dense line.

Commenting SQL Code Effectively

Good comments explain why, not what. The SQL itself already tells the reader what's happening; your comments should fill in context that isn't obvious from the syntax alone.

-- Excludes test accounts flagged by the fraud team in March 2024
SELECT *
FROM users
WHERE is_test_account = 0
    AND created_at > '2024-03-01';

Use -- for short, single-line notes and /* ... */ for longer explanations above a complex section of logic.

Common Mistake: Leaving a strange-looking filter or condition uncommented. Six months from now, nobody — including you — will remember why region != 'archived' needed to be there, and someone will waste an hour rediscovering it (or worse, delete it and break something).

Team Coding Standards and SQL Style Guides

Formatting debates burn a surprising amount of time in code review — "tabs vs. spaces," "comma-first vs. comma-last," "uppercase vs. lowercase." A short, documented SQL style guide settles these once, instead of relitigating them in every pull request.

A useful team style guide covers:

Capitalization rules for keywords and identifiers.
Indentation size (2 or 4 spaces).
Comma placement.
Alias naming conventions.
Table and column naming conventions (snake_case is the most common default).
How and when to use CTEs versus subqueries.

Keep it short — a single STYLE.md file in your repository is often enough. The goal isn't a rulebook; it's a shared reference so reviewers can say "see the style guide" instead of relitigating formatting from scratch every time.

When Automatic Formatting Is Useful

Manually reformatting a long legacy query, or something a teammate pasted in from a spreadsheet export, is tedious — and it's easy to introduce a typo while you're at it.

That's where automatic formatting tools earn their keep. An online SQL Formatter can take a dense, single-line query and instantly restructure the indentation, casing, and spacing for you. It's especially handy for:

Cleaning up inherited or legacy scripts you didn't write.
Standardizing formatting across a team without manual nitpicking in every review.
Quickly making sense of a query someone pasted into a chat message.

Automatic formatting is a great starting point, but it's still worth understanding the underlying rules yourself. That way you can fine-tune the output, format queries on the fly at the terminal, and actually catch logic issues instead of just relying on a tool to make things look tidy.

Common Formatting Conventions Followed by Professional Developers

If you want a quick checklist to build habits around, most professional SQL style guides agree on:

Uppercase keywords, lowercase snake_case identifiers.
2 or 4 space indentation — never tabs.
One clause per line.
One column per line once you're past two or three columns.
Explicit JOIN syntax, never comma joins.
Meaningful, consistent table aliases.
AS for every derived or aggregated column.
Parentheses around mixed AND/OR logic, even when not strictly required.
Trailing semicolons on every statement.
Comments that explain business context, not syntax.

Frequently Asked Questions

Does SQL formatting affect query performance?
No. Whitespace, line breaks, and capitalization don't change how the database engine parses or executes a query. Formatting is entirely for the benefit of the humans reading the code.

Should SQL keywords always be uppercase?
It's not required, but it's a widely used convention because it visually separates syntax (SELECT, WHERE, JOIN) from your actual data — table and column names. Pick uppercase or lowercase and apply it consistently.

What's the difference between comma-first and comma-last formatting?
Comma-last (the comma at the end of a line) reads more like natural prose. Comma-first (the comma at the start of the next line) makes it easier to spot a missing or accidental trailing comma at a glance. Both are valid — consistency across your team matters more than which one you choose.

How many spaces should I use for indentation in SQL?
Two or four spaces are both common choices. Four spaces makes nested subqueries more visually distinct; two spaces keeps deeply nested queries from drifting too far to the right. Avoid tabs, since they render at different widths across editors.

Should I always alias tables, even in single-table queries?
It's not necessary for a single-table query, but it becomes important as soon as you join a second table, since it removes any ambiguity about which table a column belongs to.

*Is it okay to use SELECT in formatted SQL?**
It works fine for quick, exploratory queries, but it's generally discouraged in production code or reports. An explicit column list is easier to review and won't silently change if someone adds a column to the underlying table later.

How do I format SQL in a way that's friendly to version control?
Keep each clause and each column on its own line. That way, a small change — like adding one more column — shows up as a single-line diff instead of rewriting an entire block, which makes reviews much faster.

Can I mix tabs and spaces in the same SQL file?
No. Mixed tabs and spaces render inconsistently across different editors and terminals, and it's one of the most common causes of formatting drift on a team. Pick one and configure your editor to enforce it.

Do CTEs perform better than subqueries?
Not inherently — performance depends on the specific database engine and how it optimizes the query plan. CTEs are primarily a readability tool that lets you break complex logic into named, sequential, easy-to-follow steps.

What's the best way to enforce consistent formatting across a team?
Write down your conventions in a short style guide, use an automatic formatter as a shared starting point, and check formatting during code review (or with a pre-commit hook) so it doesn't slip through.

Wrapping Up

SQL formatting isn't about being precious over style — it's a maintenance investment that pays off every time someone (including you) has to read, debug, or extend a query later.

A few habits carry most of the value:

One clause per line, one column per line.
Consistent capitalization and indentation.
Explicit joins with clear, meaningful aliases.
Comments that explain the "why," not the "what."
Parentheses whenever AND and OR mix.

Build these into your daily workflow and your queries — and your team's — get easier to review, debug, and hand off over time. And when you're short on time or inheriting a messy legacy query, running it through an online SQL Formatter first gives you a clean, consistent starting point you can fine-tune by hand from there.

Working with nested JSON: exploring it, mocking it, and querying it in Postgres

Fri, 03 Jul 2026 13:52:48 +0000

Flat JSON is easy. Nested JSON is where the day goes: an API response with orders inside customers inside accounts, a jsonb column three levels deep, a config file that grew a tree. Three tasks come up over and over, and each one has a trick that makes it painless. Here they are.

1. Reading a huge nested payload

Pretty-printing a 4,000-line response does not actually help. The indentation is correct, but finding one thing still means scrolling through thousands of lines. The fix is to stop treating JSON as text to scroll and start treating it as data to query.

Say the question is: does any item in this order have a null price?

{
  "account": {
    "id": 91,
    "customer": {
      "orders": [
        { "id": 4821, "items": [ { "sku": "A-19", "price": 12.5 } ] },
        { "id": 4822, "items": [ { "sku": "B-04", "price": null } ] }
      ]
    }
  }
}

Three moves turn this from a scroll into a lookup:

Filter by path, key, type, or value. Filtering to price pulls every price field out of the whole tree at once. Filtering by the value null jumps straight to the offending row. The needle in 4,000 lines becomes a single filtered result.
Select the nesting level. Instead of expanding everything, expand the tree to level 2 to see the shape of the data, then drill only into the branch that matters. This is the difference between reading structure and drowning in leaves.
Switch to a table view when a branch is really a list of records. Two hundred objects with the same keys is a spreadsheet pretending to be JSON, and reading it as rows and columns is far faster than reading it as nested braces.

Once JSON is filterable and foldable, payload size stops mattering.

2. Generating realistic nested JSON, fast

The opposite problem: there is no payload yet, and building a parser, a UI, or a test needs one. Hand-writing mock data is slow, and worse, it is too clean. Every record ends up with the same id, the same date, the same shape, which is exactly the uniformity that hides off-by-one bugs, dedupe bugs, and sort bugs.

Two shortcuts cover most cases:

Load a realistic nested sample. Start from a ready-made nested document (an order, a log line, an analytics event) and adapt it, instead of inventing structure from scratch.
Multiply one object into many varied records. Write a single object by hand, pick a count, and get back an array where fields like id, email, and dates actually vary per record. That variation is the point: a list of 50 rows that are genuinely different is what surfaces the bug that 50 identical rows would hide.

Realistic nested test data in seconds beats a hand-typed array of clones every time.

3. Querying and updating nested JSON in Postgres

Eventually that nested JSON lands in a jsonb column, and a viewer cannot help anymore. This is where people get stuck, so here is the practical core.

Assume a table orders with a jsonb column data holding the payload above.

Selecting. The two operators to internalize are -> and ->>. -> returns JSON, ->> returns text:

-- returns jsonb
SELECT data -> 'account' -> 'customer' FROM orders;

-- returns text (note the final ->>)
SELECT data -> 'account' ->> 'id' AS account_id FROM orders;

For a deep path, #> and #>> take an array of keys and are much cleaner than chaining arrows:

-- text value deep in the structure
SELECT data #>> '{account,customer,orders,0,id}' AS first_order_id
FROM orders;

To filter rows by a nested value, containment (@>) is both readable and indexable with a GIN index:

SELECT * FROM orders
WHERE data @> '{"account": {"id": 91}}';

And to work across an array, expand it into rows with jsonb_array_elements:

SELECT elem ->> 'id' AS order_id
FROM orders,
     jsonb_array_elements(data #> '{account,customer,orders}') AS elem;

Updating. The workhorse is jsonb_set(target, path, new_value). The gotcha that trips everyone: new_value must itself be jsonb, so cast it.

-- set a deep scalar (note the ::jsonb cast)
UPDATE orders
SET data = jsonb_set(data, '{account,customer,orders,0,id}', '9001'::jsonb)
WHERE id = 1;

Add or replace a key at a level with the || merge operator, and remove a path with #-:

-- shallow-merge a new key into the top level
UPDATE orders SET data = data || '{"archived": true}'::jsonb WHERE id = 1;

-- delete a nested key
UPDATE orders SET data = data #- '{account,customer,orders,0,items}' WHERE id = 1;

One honest warning: || merges only one level deep, so it will not deep-merge nested objects. For nested changes, reach for jsonb_set. And the genuinely hard case, updating a specific array element chosen by a condition rather than a fixed index, needs more than jsonb_set alone. That is a whole topic on its own, and it is the one worth reading a dedicated write-up on before you fight it in production.

Here is the honest reason all of this matters: nobody keeps these operators in their head. It is -> versus ->>, the #>> path syntax, the ::jsonb cast that jsonb_set quietly requires, the fact that || only merges one level. I forget them between projects every single time. So I wrote them down as a permanent, on-page reference next to the formatter: Querying nested JSON in PostgreSQL (SELECT and UPDATE) collects the common reads and writes in one place, for the moment memory does not serve. It is reference for the everyday cases, not a full Postgres manual.

Where the tool fits

I built all three of these into one place, at catssaymeow.org/json-formatter, because I got tired of one tab that only pretty-prints and another that only explores.

It formats, validates, and minifies (invalid JSON is pinpointed by line and column, not a bare "invalid"), and then lets you actually work with the structure: explore it as a tree, graph, or table, filter by path, key, type, or value, and expand to a chosen nesting level. When there is no data to work with yet, it can load a realistic nested sample or multiply one object into an array of varied mock records. It runs as client-side JavaScript, so the JSON stays in the browser, and clean JSON can be sent straight to a built-in API Client as a request body.

And for the part that happens after the database call, the same page carries a Postgres jsonb querying reference for the SELECT and UPDATE syntax that never sticks. Three recurring nested-JSON tasks, one page to come back to.

If you put it up against your own worst nested payload, I would like to hear what it struggles with.

Chai aur SQL — A Beginner's Journey into Databases

Fri, 03 Jul 2026 04:56:02 +0000

title: Chai aur SQL — A Beginner's Journey into Databases
published: true
canonical_url: https://blog.abdulrdeveloper.me/chai-aur-sql-a-beginner-s-journey-into-databases

tags: sql, postgresql, database, backend

SQL stands for Structured Query Language. It is the language we use to talk to relational databases like MySQL, PostgreSQL, and SQLite.

What is a Server and Where Does the Database Live?

Most beginners think the database is somewhere floating in the "cloud." It is not.

A database is software running on a server. A server is just a powerful computer that runs 24/7 inside a data center — like Amazon AWS, Google Cloud, or DigitalOcean. When that software saves your data, it writes it to actual files on the server's SSD or hard drive.

MySQL and PostgreSQL save data as .ibd files or WAL (Write-Ahead Log) files. MongoDB saves as .bson files. These files sit on disk just like any file on your laptop.

When you write this in your backend code:

const pool = new Pool({ host: 'localhost', port: 5432 })

You are telling your backend to open a connection to the port where the database is listening. That is it. It is a TCP connection — like a phone call between your application and the database.

Why Docker for PostgreSQL?

Instead of installing PostgreSQL directly on my system, I used Docker. Docker creates an isolated environment (called a container) so I do not mess up my laptop's setup.

Here is the command I used to create a PostgreSQL container with persistent data:

Mac / Linux / WSL:

docker run -d --name sql_class \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=sql_class_01_db \
  -p 5433:5432 \
  -v "$(pwd)/pgdata:/var/lib/postgresql/data" \
  postgres

Windows (PowerShell):

docker run -d --name sql_class -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=sql_class_01_db -p 5433:5432 -v "${PWD}/pgdata:/var/lib/postgresql/data" postgres

The -v flag maps a pgdata folder inside your current directory to the container's storage. So even if the container is deleted, your data stays on your laptop.

To check if the container is running:

docker ps

To remove the container:

docker rm -f sql_class

Why Learn Raw SQL When We Have Prisma and Drizzle?

This was my first question. Why learn raw SQL when Prisma can generate the queries for me?

Here is what I found out:

ORMs like Prisma and Drizzle translate your JavaScript/TypeScript code into SQL strings and send them to the database. If you do not know SQL, you do not know what Prisma is actually doing behind the scene.

The N+1 problem is a real thing. ORMs sometimes generate 100 queries instead of 1 optimized query. You will only catch this if you understand SQL.

For complex queries — joining 6 tables, generating financial reports — ORM syntax breaks down. Developers fall back to raw SQL.

And SQL has not changed in 50 years. Sequelize came and went. Prisma came. Drizzle came. SQL is still the same.

DDL — How You Create Tables

DDL stands for Data Definition Language. These are the commands that define the structure of your database — not the data, just the structure.

The four main DDL commands are CREATE, ALTER, DROP, and TRUNCATE.

Here is a real table I created while practicing:

CREATE TABLE students (
    student_id SERIAL PRIMARY KEY,
    first_name VARCHAR(50) NOT NULL,
    last_name VARCHAR(50),
    email VARCHAR(100) UNIQUE NOT NULL,
    phone_number CHAR(10) UNIQUE,
    age INT CHECK (age > 12),
    current_status VARCHAR(20) DEFAULT 'active'
        CHECK (current_status IN ('active', 'graduated', 'dropped_out', 'on_leave')),
    has_joined_masterji BOOLEAN DEFAULT FALSE,
    current_score NUMERIC(5, 2) CHECK (current_score >= 0 AND current_score <= 100),
    enrollment_date DATE DEFAULT CURRENT_DATE
);

SERIAL means auto-incrementing integer — PostgreSQL handles the ID for you. NOT NULL means the column cannot be empty. UNIQUE means no two rows can have the same value. CHECK is a custom condition — here age must be greater than 12. DEFAULT sets a value automatically if nothing is provided.

To add a new column to an existing table later:

ALTER TABLE students ADD COLUMN batch_name VARCHAR(50) DEFAULT 'Web Dev 2026';

SELECT Queries — Reading Your Data

SELECT is the command to read data from a table.

-- Get everything
SELECT * FROM ipl_players;

-- Get specific columns
SELECT name, team, auction_price_crores FROM ipl_players;

Filtering with WHERE:

-- Basic condition
SELECT * FROM ipl_players WHERE team = 'Mumbai Indians';

-- Multiple conditions
SELECT * FROM ipl_players WHERE role = 'All-Rounder' AND wickets_taken > 10;

-- Either condition
SELECT * FROM ipl_players WHERE team = 'CSK' OR team = 'RCB';

-- Range
SELECT * FROM ipl_players WHERE auction_price_crores BETWEEN 5 AND 12;

-- From a list
SELECT * FROM ipl_players WHERE team NOT IN ('Mumbai Indians', 'CSK', 'RCB');

-- Pattern matching — names starting with R
SELECT * FROM ipl_players WHERE name LIKE 'R%';

-- Check for empty values
SELECT * FROM ipl_players WHERE team IS NULL;

% means zero or more characters. _ means exactly one character. LIKE is case-sensitive. ILIKE is case-insensitive (PostgreSQL only).

Sorting and limiting:

-- Sort by price, highest first
SELECT name, auction_price_crores FROM ipl_players ORDER BY auction_price_crores DESC;

-- Top 3 results
SELECT name, auction_price_crores FROM ipl_players ORDER BY auction_price_crores DESC LIMIT 3;

-- Skip first 3, get next 3 (pagination)
SELECT name, auction_price_crores FROM ipl_players ORDER BY auction_price_crores DESC LIMIT 3 OFFSET 3;

DML — Insert, Update, Delete

DML stands for Data Manipulation Language. This is how you actually work with the data inside the tables.

INSERT:

-- Single row
INSERT INTO canteen_menu (item_name, category, price)
VALUES ('Vada Pav', 'Snacks', 15);

-- Multiple rows at once
INSERT INTO canteen_menu (item_name, category, price) VALUES
('Masala Chai', 'Beverages', 10),
('Samosa', 'Snacks', 12),
('Maggi', 'Snacks', 25);

UPDATE:

UPDATE canteen_menu
SET price = 20
WHERE item_name = 'Vada Pav';

DELETE:

DELETE FROM canteen_menu WHERE item_name = 'Cold Coffee';

Always use WHERE with UPDATE and DELETE. If you forget the WHERE clause, SQL will update or delete every single row in the table. I learned this the hard way in practice.

Aggregate Functions — COUNT, SUM, AVG, MAX, MIN

Aggregate functions take a whole column and return one number.

-- How many rows?
SELECT COUNT(*) FROM smart_watch_sales;

-- Total revenue
SELECT SUM(units_sold * price_per_unit) AS total_revenue FROM smart_watch_sales;

-- Average price
SELECT AVG(price_per_unit) AS avg_price FROM smart_watch_sales;

-- Cheapest and most expensive
SELECT MIN(price_per_unit) AS cheapest, MAX(price_per_unit) AS costliest FROM smart_watch_sales;

GROUP BY lets you split these calculations by category:

-- Total units sold per brand
SELECT brand, SUM(units_sold) AS total_units_sold
FROM smart_watch_sales
GROUP BY brand
ORDER BY total_units_sold DESC;

HAVING filters the groups after they are created. WHERE cannot do this because WHERE runs before grouping:

-- Only show brands that sold more than 20 units total
SELECT brand, SUM(units_sold) AS total_units
FROM smart_watch_sales
GROUP BY brand
HAVING SUM(units_sold) > 20;

Joins — Connecting Multiple Tables

This is where relational databases get powerful.

We keep data in separate tables to avoid duplication. Students in one table, internships in another. Joins let you combine them in one query.

Foreign Key is what connects the tables. The internships table has a student_id column that references the student_id in the students table:

CREATE TABLE internships (
    internship_id SERIAL PRIMARY KEY,
    student_id INT REFERENCES students(student_id) ON DELETE SET NULL,
    company_name VARCHAR(100),
    role VARCHAR(50),
    stipend INT,
    status VARCHAR(20)
);

ON DELETE SET NULL means if a student is deleted, the internship record stays but student_id becomes NULL. ON DELETE CASCADE would delete the internship too.

INNER JOIN — only matching rows:

SELECT s.name, s.branch, i.company_name, i.role
FROM students s
INNER JOIN internships i ON s.student_id = i.student_id;

Students without internships will not show up here.

LEFT JOIN — all students, matched or not:

SELECT
    s.name,
    s.branch,
    COALESCE(i.company_name, 'No Internship') AS company_name,
    COALESCE(i.stipend, 0) AS stipend
FROM students s
LEFT JOIN internships i ON s.student_id = i.student_id;

COALESCE replaces NULL with a default value. Students without internships will appear with 'No Internship' instead of NULL.

To find students who have NOT applied anywhere:

SELECT s.name, s.email
FROM students s
LEFT JOIN internships i ON s.student_id = i.student_id
WHERE i.internship_id IS NULL;

RIGHT JOIN returns all rows from the right table. In practice, developers rarely use RIGHT JOIN — they just flip the table order and use LEFT JOIN instead. It reads more naturally.

FULL OUTER JOIN — everything from both tables:

SELECT s.name AS student_name, i.company_name
FROM students s
FULL OUTER JOIN internships i ON s.student_id = i.student_id;

This shows all students and all internships. Where there is no match on either side, you get NULL.

Indexes — Making Queries Fast

I inserted 1,000,000 rows into a table to test this.

Without an index, when you search by name, PostgreSQL scans every single row from top to bottom. This is called a Sequential Scan.

EXPLAIN ANALYZE SELECT marks FROM marks WHERE name = '809E15792322';

EXPLAIN ANALYZE shows you the query plan and execution time. Before adding an index, my query took around 40-50 milliseconds on 1 million rows.

After creating an index:

CREATE INDEX idx_name ON marks (name);

The same query dropped to under 1 millisecond. PostgreSQL now uses a B-Tree structure to find the row directly instead of scanning everything.

A covering index goes one step further. If your query asks for marks along with name, you can store both in the index itself:

CREATE INDEX idx_name ON marks (name) INCLUDE (marks);

Now PostgreSQL does not even need to visit the main table. It gets everything from the index directly. This is called an Index Only Scan.

Transactions and ACID

A transaction is a way to group multiple SQL operations into one unit of work. Either all of them succeed, or none of them do.

The classic example is a bank transfer:

BEGIN;

    UPDATE accounts SET balance = balance - 500 WHERE owner = 'Shubham';
    UPDATE accounts SET balance = balance + 500 WHERE owner = 'Hitesh';

COMMIT;

If the server crashes between the two UPDATE statements, the database will rollback both changes. The money will not disappear from Shubham's account and fail to reach Hitesh.

If something goes wrong, you can manually undo everything:

BEGIN;
    UPDATE accounts SET balance = balance - 5000 WHERE owner = 'Shubham';
ROLLBACK;

ACID is the set of guarantees that make this work:

Atomicity — All or nothing. If Step 2 fails, Step 1 is also undone. I tested this by putting SELECT 1/0 (division by zero error) in the middle of a transaction. The whole thing rolled back.

Consistency — The database only moves from one valid state to another. I set CHECK (balance >= 0) on the accounts table. When I tried to deduct 5000 from an account with 1000, PostgreSQL threw an error and refused the operation completely.

Isolation — Uncommitted transactions are invisible to other connections. I opened two terminal windows to test this. Terminal 1 started a transaction and updated a balance but did not COMMIT. Terminal 2 could still see the old balance. Only after Terminal 1 ran COMMIT did Terminal 2 see the new number.

Durability — Once committed, data is permanent. PostgreSQL writes committed transactions to the Write-Ahead Log (WAL) on disk before returning a success response. Even if the server crashes one second after the COMMIT, the data will be there when the server restarts.

All the code from this blog is in my GitHub repo: github.com/abdulrdeveloper/Learn-SQL

You can find more of my work at abdulrdeveloper.me
Read more posts at blog.abdulrdeveloper.me

Why not Sakila? Building a Modern SQL Learning Database for MariaDB

Fri, 26 Jun 2026 20:42:52 +0000

If you've taught SQL or learned it from a course, you've almost certainly met Sakila. The little DVD rental database has been the go-to sample schema for MySQL and MariaDB tutorials for nearly two decades. It's clean, well-normalized, and comes pre-loaded with enough data to write interesting queries.

But here's the thing: Sakila was designed in 2006 for MySQL 5.0.

A lot has changed since then.

What Sakila can't show you

Open the Sakila schema today and count the column types: INT, VARCHAR, TEXT, ENUM, DATETIME, DECIMAL, TINYINT. That's it. No JSON. No FULLTEXT search beyond a basic demo. No SET type. And absolutely no VECTOR.

Meanwhile, MariaDB 11.7 ships with:

Native VECTOR(N) type for AI-era similarity search
Rich JSON functions (JSON_TABLE, JSON_VALUE, JSON_EXTRACT)
Window functions (RANK, LAG, LEAD, running aggregates)
Recursive CTEs
FULLTEXT with boolean mode and relevance scoring

Teaching someone SQL with Sakila in 2026 is like teaching someone to drive in a car with no GPS, no reversing camera, and a manual choke. The fundamentals still apply — but they're missing a huge chunk of what the tool can actually do.

There's also the domain problem. DVD rentals. If you're under 35, you may never have set foot in a video rental shop. The mental model is unfamiliar and the business rules feel arbitrary. What's the difference between a film, an inventory item, and a rental? Why does payment exist independently of rental? Explaining the schema takes time that should go into explaining SQL.

The domain that everyone understands

I needed something universal. Something with:

Obvious entities and relationships
Multiple natural hierarchies (good for recursive CTEs)
A mix of small lookup tables and large transactional tables
A clear reason for JSON (semi-structured data)
An excuse to use vectors (semantic search is everywhere now)

A university fits perfectly. Students, courses, faculty, enrollments, grades — everyone has lived inside this system. The relationships are intuitive. And the domain naturally produces the data shapes I needed:

Need	University equivalent
Big analytic table	`grade_events` — 120 000+ rows of scored items
Hierarchy for recursive CTE	Departments (Faculty → Department → Sub-dept) and course prerequisites
JSON for semi-structured data	Faculty office hours, student emergency contacts, grant funding details
FULLTEXT search	Course descriptions, publication abstracts
VECTOR search	Course semantic embeddings for "find similar courses"
Audit trail	Every enrollment and grade change logged with JSON diffs

What University DB looks like

The schema has 16 tables in four tiers:

Lookup (tiny, < 100 rows): semesters, rooms, scholarships

Domain (small, up to 2 000 rows): departments, faculty, students, courses, course_prerequisites, sections

Transactional (medium): enrollments, student_scholarships, research_projects, publications, project_members

Analytic (big): grade_events (~120k rows) and audit_log (~60k rows, populated by triggers)

Every significant MariaDB datatype appears at least once:

-- VECTOR on courses — semantic embeddings for similarity search
embedding  VECTOR(1536) NULL

-- JSON on faculty — semi-structured office hours
office_hours JSON NULL
-- [{"day":"Mon","start":"10:00","end":"12:00"}, ...]

-- SET on publications — multi-value keyword tags
keywords SET('AI','ML','Databases','Security','Bioinformatics', ...) 

-- FULLTEXT on courses and publications
FULLTEXT KEY ft_course (title, description)

The schema ships with 7 views, 6 stored procedures, and 7 triggers — including one that blocks enrollment when a section is full, and three that write JSON diffs to the audit_log table.

Four levels of example queries

I structured the example queries into four files so the database works for everyone from first-day learners to DBAs:

Level 1 — Basics: SELECT, WHERE, GROUP BY, single-table aggregation

Level 2 — Intermediate: multi-table JOIN (up to 5 tables), correlated subqueries, FULLTEXT search, JSON_VALUE

Level 3 — Advanced: window functions, CTEs, recursive CTEs, JSON_TABLE, SET/FIND_IN_SET, VEC_Distance similarity search

Level 4 — DBA/Developer: EXPLAIN ANALYZE, index strategy, stored procedure authoring, transaction isolation levels, audit log forensics

A level 3 query to find the full prerequisite chain for a course looks like this:

WITH RECURSIVE prereq_chain AS (
    SELECT cp.prerequisite_id,
           p.code   AS prereq_code,
           p.title  AS prereq_title,
           1        AS depth
    FROM   course_prerequisites cp
    JOIN   courses p ON p.course_id = cp.prerequisite_id
    WHERE  cp.course_id = (SELECT course_id FROM courses WHERE code = 'CS300')

    UNION ALL

    SELECT cp2.prerequisite_id,
           p2.code,
           p2.title,
           pc.depth + 1
    FROM   course_prerequisites cp2
    JOIN   prereq_chain         pc ON pc.prerequisite_id = cp2.course_id
    JOIN   courses              p2 ON p2.course_id = cp2.prerequisite_id
    WHERE  pc.depth < 10
)
SELECT DISTINCT depth, prereq_code, prereq_title
FROM   prereq_chain
ORDER  BY depth, prereq_code;

And a vector similarity search to find courses related to a given one:

SELECT c.code, c.title,
       VEC_Distance(ref.embedding, c.embedding) AS distance
FROM   courses ref
JOIN   courses c ON c.course_id <> ref.course_id
WHERE  ref.code = 'CS101'
ORDER  BY distance
LIMIT  5;

Try it right now

You don't need to install anything to explore the schema.

You can run queries against University DB directly in sqlize.online — my online SQL editor that supports MariaDB 11.7. Paste any query from the example files and see results immediately.

If you want a more structured learning experience with exercises and instant feedback, check out sqltest.online — designed for exactly this kind of hands-on SQL practice.

Get the database

Everything is open-source under the MIT license:

👉 github.com/rozhnev/university-db

The repository includes:

01_schema.sql — all 16 tables
02_objects.sql — views, procedures, triggers
03_seed_small.sql — static seed data
generate_data.py — Python/Faker script to populate ~130 000 rows
docker-compose.yml — one command to get a running database
queries/level1.sql through level4.sql — 50+ example queries

git clone https://github.com/rozhnev/university-db.git
cd university-db
cp .env.example .env
docker compose up --build

How to contribute

The project is open source and contributions of any kind are welcome.

Found a bug? — Open an issue on GitHub. Mistakes in the schema, incorrect queries, typos in comments — all worth reporting. The more specific the description, the faster it gets fixed.

Have an interesting query? — Send a pull request to queries/. Level 3–4 examples are especially valuable: window functions, recursive CTEs, JSON forensics, vector search.

Want to improve the data generator? — generate_data.py is intentionally kept simple. More realistic grade distributions, additional data scenarios, or faster bulk-insert batching are all good targets.

Need support for another environment? — A Kubernetes manifest, Helm chart, or a setup script for a cloud managed service (RDS, Cloud SQL, PlanetScale) would be a useful addition.

Educational materials? — Exercises with solutions, workshop slides, or Jupyter notebooks built on this schema are all welcome.

Fork the repository and send a pull request — code review within a few days.

Is Sakila dead?

Not at all. It's still a perfectly valid database for learning basic SQL, and its portability (it runs on any MySQL 5.x+ install) is a genuine advantage. But as a primary teaching tool for modern MariaDB, it's showing its age.

University DB fills the gap for anyone who wants to teach or learn the full surface area of what MariaDB 11.7 can do — from a first SELECT to a vector similarity search in a recursive CTE inside a stored procedure.

I hope it's useful. Feedback, issues, and pull requests are very welcome.

Slava Rozhnev — sqlize.online · sqltest.online · GitHub

INNER JOIN vs Correlated Subquery in SQL: What Is Actually Faster?

Mon, 22 Jun 2026 20:57:21 +0000

Most SQL developers hear the same rule early: always prefer INNER JOIN over correlated subqueries, because subqueries create an N+1 performance problem.

That rule is useful, but it is not always true.

I tested this on four engines (MySQL, Oracle, PostgreSQL, SQLite) and got mixed results. In some cases, the correlated subquery was faster than the join query.

If you want to validate the same patterns yourself, you can run these examples in live sandboxes and practice similar SQL tasks on sqltest.online.

What You Will Learn

what INNER JOIN does best;
when a correlated subquery can be competitive;
why execution plans matter more than "golden rules";
how to benchmark both approaches correctly.

What Is INNER JOIN in SQL?

INNER JOIN returns only rows that match in both tables.

Why it matters: it is usually the cleanest set-based way to combine data, and often gives the optimizer strong options.

SELECT
    c.customer_id,
    c.name,
    o.order_id
FROM customers c
INNER JOIN orders o
    ON c.customer_id = o.customer_id;

In many workloads, this is exactly what you want. But performance still depends on table sizes, indexes, and optimizer decisions.

Benchmark Scenario

I used a small outer table and a larger inner table:

customers: 25 rows
orders: 1000 rows
index on orders(customer_id)

Task: return each customer with the number of their orders.

Two Query Patterns

Pattern A: JOIN + GROUP BY

SELECT
    c.customer_id,
    COUNT(o.order_id) AS orders_count
FROM customers c
LEFT JOIN orders o
    ON c.customer_id = o.customer_id
GROUP BY c.customer_id;

Note: I used LEFT JOIN here to keep customers with zero orders. If you only need matched rows, use INNER JOIN.

Pattern B: Correlated Subquery

SELECT
    c.customer_id,
    (
        SELECT COUNT(o.order_id)
        FROM orders o
        WHERE o.customer_id = c.customer_id
    ) AS orders_count
FROM customers c;

The second form looks like classic N+1, but an index can make each lookup very cheap.

How to Inspect the Real Cost

Use execution plans, not assumptions:

MySQL / PostgreSQL: EXPLAIN ANALYZE
SQLite: EXPLAIN QUERY PLAN
Oracle: EXPLAIN PLAN FOR ... + DBMS_XPLAN.DISPLAY

Results Summary Across Engines

MySQL 8 and Oracle 23c

In my tests, the correlated subquery often won.

Why: with a small outer table (25 customers), the optimizer could do very fast indexed lookups on orders(customer_id) per customer. That was cheaper than building larger join/group intermediate work.

PostgreSQL 16

JOIN + GROUP BY won in my test.

Why: PostgreSQL produced an efficient set-based plan and handled the join/aggregate path very well.

SQLite

Practical tie.

Both variants ended with very similar plans and near-identical timing on this dataset.

Why the INNER JOIN Rule Sometimes Fails

The "always use INNER JOIN" advice fails when it ignores three variables:

Database engine optimizer behavior
Data distribution and table size ratio
Index availability and quality

SQL performance is not about syntax preference. It is about the physical plan your engine chooses.

Common Mistakes

assuming INNER JOIN is always faster without checking plans;
forgetting that INNER JOIN removes non-matching rows;
testing once without warm-up runs;
benchmarking without a supporting index;
comparing query text instead of comparing actual execution plans.

Practical Checklist

Before choosing INNER JOIN or correlated subquery:

Add the right index (here: orders(customer_id)).
Run both query variants.
Capture execution plans.
Compare timing over several runs.
Pick the query that is both correct and fast for your engine.

Final Takeaway

INNER JOIN is a strong default and often the right choice, but it is not a universal performance guarantee.

The best query is the one that produces the best execution plan on your actual database, with your actual data.

SQL Formatting Best Practices: Writing Readable, Maintainable Queries

Fri, 19 Jun 2026 10:07:57 +0000

Messy SQL is a productivity killer. A query that took 20 minutes to write can take 2 hours to debug if it's unformatted. Good SQL formatting isn't about aesthetics — it's about reducing cognitive load when you're six levels deep in a JOIN and trying to figure out where the bug is.

The Core Conventions

Every SQL style guide agrees on the fundamentals. These apply across MySQL, PostgreSQL, SQL Server, BigQuery, and SQLite.

1. Uppercase keywords

``sql

-- Good

SELECT id, name, email

FROM users

WHERE status = 'active'

ORDER BY created_at DESC;

-- Bad

select id, name, email from users where status = 'active' order by created_at desc;

Uppercase keywords make the structure of a query immediately scannable. Most formatters apply this automatically.

2. One clause per line

Each major clause — SELECT, FROM, WHERE, JOIN, GROUP BY, ORDER BY, HAVING — should start on its own line. This makes it easy to add, remove, or comment out individual clauses during development.

3. Indent continuation lines

Sub-expressions, JOIN conditions, and WHERE conditions should be indented:

<code>sql SELECT u.id, u.name, o.total, p.name AS product_name FROM users u INNER JOIN orders o ON o.user_id = u.id INNER JOIN order_items oi ON oi.order_id = o.id INNER JOIN products p ON p.id = oi.product_id WHERE u.status = 'active' AND o.created_at >= '2024-01-01' AND o.total > 100 ORDER BY o.created_at DESC; </code>

4. Leading AND/OR in WHERE clauses

Put AND and OR at the beginning of a condition line, not the end. This makes it trivial to comment out a single condition without breaking the query:

<code>sql -- Easy to comment out individual conditions: WHERE status = 'active' AND created_at >= '2024-01-01' -- AND deleted_at IS NULL ← just add -- to disable AND role IN ('admin', 'editor') </code>

If you put AND at the end of each line, commenting out one line breaks the syntax.

5. Meaningful aliases

Aliases should clarify, not obscure:

`sql -- Good

FROM users u

INNER JOIN orders o ON o.user_id = u.id

-- Bad

FROM users a

INNER JOIN orders b ON b.a_id = a.id

Single-letter aliases are fine when the table name is obvious (u for users), but avoid arbitrary letters that don't relate to the table name.

Naming Conventions

Consistent naming is part of readable SQL. The most widely adopted conventions:

Tables: snake_case, plural nouns — users, order_items, product_categories
Columns: snake_case — user_id, created_at, is_active
Primary keys: id or {table}_id (e.g. user_id in the users table)
Foreign keys: {referenced_table}_id (e.g. user_id in orders)
Booleans: is_ or has_ prefix — is_active, has_paid
Timestamps: created_at, updated_at, deleted_at

Avoid CamelCase — most databases are case-insensitive by default, so UserID and userid are the same identifier unless quoted.

CTEs Over Subqueries

Common Table Expressions (CTEs) make complex queries readable by naming intermediate results:

`sql -- Hard to read: nested subqueries

SELECT u.name, order_summary.total

FROM users u

INNER JOIN (

SELECT user_id, SUM(total) AS total

FROM orders

WHERE status = 'completed'

GROUP BY user_id

) order_summary ON order_summary.user_id = u.id;

-- Easy to read: CTE

WITH completed_orders AS (

SELECT user_id, SUM(total) AS total

FROM orders

WHERE status = 'completed'

GROUP BY user_id

)

SELECT u.name, co.total

FROM users u

INNER JOIN completed_orders co ON co.user_id = u.id;

CTEs also make it easy to debug intermediate steps — you can run the CTE in isolation to verify its output before using it in the main query.

Long WHERE Clauses

When a WHERE clause has many conditions, group related conditions together and add a blank line between groups:

<code>sql WHERE -- User filters u.status = 'active' AND u.role IN ('admin', 'editor') -- Date range AND o.created_at >= '2024-01-01' AND o.created_at < '2025-01-01' -- Amount filter AND o.total BETWEEN 50 AND 500 </code>

Use parentheses whenever you mix AND and OR — operator precedence rules can cause subtle bugs:

`sql -- BUG: this is parsed as: status = 'active' AND (role = 'admin' OR role = 'editor' AND deleted_at IS NULL)

WHERE status = 'active' AND role = 'admin' OR role = 'editor' AND deleted_at IS NULL

-- CORRECT: explicit grouping

WHERE status = 'active'

AND (role = 'admin' OR role = 'editor')

AND deleted_at IS NULL

Automating SQL Formatting

Manual formatting is not sustainable across a team. Automate it:

VS Code: Install the "SQL Formatter" extension — format on save with Ctrl+Shift+P → Format Document
Pre-commit hooks: Use sql-formatter npm package with husky + lint-staged to format .sql files before every commit
Python projects: sqlparse library — sqlparse.format(sql, reindent=True, keyword_case='upper')
CI/CD: sqlfluff` supports format checking and enforcement in pipelines

For one-off formatting without installing anything, this free SQL formatter runs in your browser, supports 19+ dialects (MySQL, PostgreSQL, BigQuery, Snowflake, T-SQL, SQLite, and more), and never sends your queries to a server.

Summary

Good SQL formatting comes down to a few consistent habits:

Uppercase keywords, lowercase identifiers
One clause per line, indented continuation lines
Leading AND/OR for easy debugging
Meaningful aliases that relate to table names
CTEs over nested subqueries for anything complex
Explicit parentheses when mixing AND/OR
Automate it — formatters remove the decision overhead entirely

The goal is SQL that a colleague can read and understand at a glance — both when you're writing it and when you're debugging it six months later.

Format SQL instantly in your browser at snappytools.app/sql-formatter-beautifier — supports 19+ dialects, keyword case options, and runs entirely client-side.

Why SQL Still Rules the Data World: Surprising Truths from the Relational Revolution

Sun, 14 Jun 2026 14:39:21 +0000

In the hyper-accelerated world of software development, where programming languages often have the shelf life of a smartphone, SQL stands as a defiant outlier. With roots stretching back to the early 1970s, it has not only survived the rise and fall of countless "modern" competitors but has thrived.

As we struggle to navigate an unprecedented deluge of digital information, it is a fascinating paradox: this "ancient" tool, born in the era of magnetic tapes, remains the industry’s most sophisticated way to manage the complexities of data. To understand why, we have to look past the syntax and into the architectural soul of the relational model.

The "High-Power Zoom Lens" of Data

The enduring success of relational databases lies in their unique ability to handle massive datasets without losing focus. Many alternative database management systems break down under heavy loads.

The reason is usually architectural: their focus is too narrow. They are designed to do one thing well, but they lack versatility. In these systems, the “lens” is effectively stuck on maximum zoom, making it impossible to see the forest for the trees.

SQL, by contrast, functions like a high-precision instrument for data. As Alan Beaulieu notes in Learning SQL:

"SQL is akin to one of those snazzy digital cameras with the
high-power zoom lens in that you can use SQL to look at large sets of
data, or you can zoom in on individual rows (or anywhere in between)."

This versatility allows us to act as both microscopes and telescopes. We can perform macro-scale analysis—identifying multi-year trends across millions of records—and then instantly zoom in to inspect a single micro-scale transaction. It is this specific ability to scale perspective that has allowed the relational model to defeat almost every attempt to dethrone it.

The Zen of Giving Up Control

For developers raised on Java, C++, or Python, the hardest part of mastering SQL isn’t the keywords—it’s the fundamental shift in mindset. Most languages are procedural; the programmer is the micromanager, defining both the desired result and the specific process to achieve it. SQL is a nonprocedural language, which requires a "Zen-like" surrender of control to an external agent: the Optimizer.

The "What" (SQL): You define the inputs and the desired output (e.g., "Show me all customers in Alexandria with active accounts").
The "How" (Procedural): The Optimizer analyzes your query, evaluates table configurations and available indexes, and then picks an Execution Plan, which the server uses to actually retrieve the data.

Trusting the Optimizer to generate this Execution Plan is a rite of passage. While we can occasionally influence these decisions with "hints," the database engine is almost always better at navigating the physical complexities of the data than a human writing manual loops and pointers.

The "Four-Letter Word" That Isn't What You Think

In the realm of data integrity, there is one four-letter word that causes more bugs for beginners than any other: NULL.

To a junior developer, NULL looks like a zero or an empty string. To an architect, it is the absence of value. According to the source context, NULL specifically represents data that is:

Not applicable
Unknown
An empty set

A senior practitioner will tell you that a zero is a number and an empty string is a character, but a NULL is a question mark.

Pro-Tip: You cannot use the equality operator (=) with NULL. To find non-terminated employees, for example, a Senior Architect would warn you to use the specific syntax WHERE end_date IS NULL. Attempting to use WHERE end_date = NULL will return nothing, as the absence of a value cannot be "equal" to anything.

Why "Redundant" Data Is Actually a Clean Design

It is counter-intuitive, but the relational model’s use of "redundant" data—specifically foreign keys—is the secret to its reliability. This is the heart of Normalization. We refine a design to ensure every independent piece of information exists in exactly one place.

Consider a "Person Table." A common beginner mistake is to create a "compound object" column, like a name column containing both first and last names, or an address column containing street, city, and zip. This is a violation of normalization. Why? Because as the source material warns:

"...otherwise, the data might be changed in one place but not another,
causing the data in the database to be unreliable."

By ensuring a "single source of truth," we prevent data corruption. We store a person's name once in the person table and use their unique ID (the foreign key) everywhere else. If their name changes, we update one row in one table, and the change is logically reflected across the entire system.

The Evolution from Pointers to Tables

To appreciate SQL, one must understand the chaos it replaced. Before Dr. E.F. Codd proposed the Relational Model in 1970, we relied on Hierarchical and Network systems. These were rigid structures where data was connected by physical "pointers." To find a record, you had to manually traverse these links like a navigator in a dark forest.

Codd’s revolution replaced these physical links with logical joins. This transition was reflected in the language's own evolution. It began as DSL/Alpha, was simplified into SQUARE, and was then refined into SEQUEL (Structured English Query Language) before finally becoming the standard SQL we use today. By moving from "pointers" to "sets of tables," Codd made data accessible to the user rather than just the specialist, allowing us to link information dynamically based on values rather than hard-coded physical paths.

Conclusion

The enduring nature of SQL is a testament to the elegant efficiency of the relational model. In an era where the industry is frequently obsessed with "new" tools and NoSQL alternatives, the fundamentals established in the 1970s are more relevant than ever.

We now manage terabytes of data spread across fast-access drives, with tens of gigabytes held in high-speed memory, yet the core problems of retrieval and integrity remain the same. SQL has entered "middle age," but its future is bright because it solves these problems with a rigor that modern alternatives often lack. We must ask ourselves: in our obsession with "new" tools, have we overlooked the sophisticated power of the relational model in favor of modern tool-bloat? Mastering these "old" fundamentals is not just a history lesson—it is the only way to truly master the data-driven world of tomorrow.

Reference Material

SQL Formatting and Beautification: Standards, Tools, and Best Practices

Sat, 06 Jun 2026 23:48:06 +0000

Unformatted SQL is a maintenance problem. A 200-character single-line query is hard to review in a PR, harder to debug under pressure, and inconsiderate to hand off to a teammate. SQL formatting costs nothing and pays back immediately.

What a SQL Formatter Does

A formatter restructures query text into a consistent, indented layout — without changing what the query does. The database ignores whitespace; formatting is entirely for humans.

Before:

`<code>sql select u.id,u.name,o.total from users u inner join orders o on u.id=o.user_id where o.status='paid' order by o.total desc limit 100; </code>

After:
<code>sql SELECT u.id, u.name, o.total FROM users u INNER JOIN orders o ON u.id = o.user_id WHERE o.status = 'paid' ORDER BY o.total DESC LIMIT 100; </code>

Same query. Dramatically more readable.

Key Formatting Decisions

Keyword case — SQL keywords are case-insensitive. SELECT vs select. Uppercase is the most common convention (keywords stand out visually from identifiers). Choose one and stick to it across your codebase.

Clause alignment — Each major clause (SELECT, FROM, WHERE, GROUP BY, ORDER BY) starts a new line. Conditions within a clause are indented.

Leading vs trailing commas:
`sql -- Trailing (most common)

SELECT

id,

name,

-- Leading (easier to add/remove first item)

SELECT

,name

,email

AND/OR placement:
<code>sql -- AND at start of line (easy to comment out one condition) WHERE status = 'active' AND created_at > '2026-01-01' AND amount > 100 </code>

SQL Dialects

SQL is not one standard. Each database has extensions:

| Database | Notes |

|---|---|

| PostgreSQL | ILIKE, :: casting, $$ dollar quoting, CTEs everywhere |

| MySQL | Backtick identifiers, LIMIT x,y, AUTO_INCREMENT |

| SQLite | Minimal syntax, no stored procedures |

| SQL Server (T-SQL) | TOP, NOLOCK, GETDATE(), GO |

| BigQuery | STRUCT, ARRAY_AGG, UNNEST, standard SQL mode |

A good formatter knows which dialect you're using. SnappyTools' SQL formatter supports all major dialects via a dropdown selector.

Beautification: Compressing SQL Back

The reverse of formatting — collapsing multi-line SQL to a single line — is useful for:

Embedding SQL in JSON configuration
Storing queries in environment variables
Code generation output
Sending queries in HTTP request bodies

Both directions are available at snappytools.app/sql-formatter-beautifier/.

Automating SQL Formatting in CI

For teams, use sqlfluff:

`bash pip install sqlfluff

Lint

sqlfluff lint queries/ --dialect postgres

Auto-fix

sqlfluff fix queries/ --dialect postgres

.sqlfluffconfig:<code>ini [sqlfluff] dialect = postgres indent_unit = space tab_space_size = 2 </code>

For pre-commit hooks:

<code>yaml <h1>.pre-commit-config.yaml</h1> repos: - repo: https://github.com/sqlfluff/sqlfluff rev: 3.0.0 hooks: - id: sqlfluff-lint args: [--dialect, postgres] </code>

Anti-Patterns to Avoid

Implicit joins (hard to see the join condition):
`sql -- Avoid

SELECT * FROM orders, users WHERE orders.user_id = users.id;

-- Better

SELECT * FROM orders INNER JOIN users ON orders.user_id = users.id;

SELECT in production — Not formatting, but worth mentioning: always name columns explicitly. SELECT breaks when the schema changes.

Inconsistent keyword case within one query:
`sql -- Bad

select id FROM users WHERE status='active';

-- Good

SELECT id FROM users WHERE status = 'active';

Summary

SQL formatting is about maintainability and team consistency, not performance. Consistent uppercase keywords, clause-per-line layout, and proper indentation make queries reviewable and debuggable. For one-off formatting of queries from APIs, schema dumps, or colleagues: snappytools.app/sql-formatter-beautifier/.

Originally published at https://snappytools.app/sql-formatter-beautifier/

How to Format SQL for Readability: A Practical Style Guide

Sat, 06 Jun 2026 10:03:48 +0000

SQL doesn't enforce any whitespace rules — the engine treats a one-liner and a 40-line query exactly the same. But your teammates and future self will thank you for the 40-line version.

Here's a concise SQL formatting guide that works across MySQL, PostgreSQL, SQLite, and SQL Server.

Core rules

1. Uppercase keywords, lowercase names

This single convention does the most work:

``sql

SELECT u.name, o.total

FROM users u

JOIN orders o ON u.id = o.user_id

WHERE o.status = 'completed'

ORDER BY o.total DESC;

SQL keywords stand out visually; your table and column names don't compete for attention.

2. One clause per line

Each major clause (SELECT, FROM, WHERE, JOIN, GROUP BY, HAVING, ORDER BY, LIMIT) starts on a new line at the left margin.

3. Indent continuation items

Items in a SELECT list or multiple WHERE conditions are indented one level:

<code>sql SELECT u.name, u.email, o.total, o.created_at FROM users u WHERE o.status = 'completed' AND o.created_at > '2024-01-01'; </code>

4. Alias everything consistently

If you alias users as u, use u.column everywhere in that query. Mixing users.id and u.name in the same query causes unnecessary cognitive load.

5. Use CTEs for complex queries

Named CTEs (WITH ... AS (...)) replace nested subqueries with human-readable steps:

<code>sql WITH active_users AS ( SELECT id, name FROM users WHERE last_login > NOW() - INTERVAL '30 days' ), their_orders AS ( SELECT user_id, COUNT(*) AS order_count FROM orders GROUP BY user_id ) SELECT u.name, t.order_count FROM active_users u JOIN their_orders t ON u.id = t.user_id ORDER BY t.order_count DESC; </code>

Each CTE reads like a named step. The final SELECT is obvious.

Formatting inherited SQL

When you receive SQL from an ORM, export tool, or legacy codebase, it often arrives as one long string. The SQL Formatter on SnappyTools takes minified or unformatted SQL and outputs properly indented, keyword-uppercased SQL for MySQL, PostgreSQL, SQLite, or T-SQL. Paste in one query or a full migration file — it runs in your browser with no upload.

What to avoid

Mixed case keywords (Select, WHERE, from)
Comma-first style (debated — choose one and stick to it)
Deeply nested subqueries when a CTE would be clearer
Aliases that are cryptic abbreviations (u2, x, t3`)

SQL style is team infrastructure. Establish a convention in your team's contributing guide, enforce it with a formatter in CI, and you'll spend less time decoding queries and more time understanding what they do.

SQL Formatting Best Practices: Make Your Queries Readable and Maintainable

Fri, 05 Jun 2026 23:53:21 +0000

SQL is one of the few languages where formatting is genuinely optional — the engine doesn't care about whitespace, indentation, or casing. But your teammates do. And so does future-you.

The Core Rules

1. Uppercase keywords

Most style guides recommend uppercase SQL keywords and lowercase table/column names. It creates visual hierarchy:

``sql

SELECT id, name, email

FROM users

WHERE is_active = true

ORDER BY name ASC;

vs. the unformatted version:

<code>sql select id,name,email from users where is_active=true order by name asc; </code>

2. One clause per line

Each major clause starts on a new line. Conditions are indented:

<code>sql SELECT u.id, u.name, COUNT(o.id) AS order_count FROM users u LEFT JOIN orders o ON o.user_id = u.id WHERE u.is_active = true AND u.created_at > '2026-01-01' GROUP BY u.id, u.name ORDER BY order_count DESC LIMIT 100; </code>

3. Indent subqueries

<code>sql SELECT * FROM ( SELECT user_id, SUM(amount) AS total FROM orders WHERE status = 'completed' GROUP BY user_id ) AS spending WHERE spending.total > 1000; </code>

4. Use CTEs for complex logic

Common Table Expressions make multi-step queries readable:

<code>sql WITH active_users AS ( SELECT id, name FROM users WHERE is_active = true ), high_spenders AS ( SELECT user_id, SUM(amount) AS total FROM orders GROUP BY user_id HAVING SUM(amount) > 1000 ) SELECT au.name, hs.total FROM active_users au JOIN high_spenders hs ON hs.user_id = au.id ORDER BY hs.total DESC; </code>`

Formatting Reference

| Element | Convention |

|---|---|

| SQL keywords | UPPERCASE |

| Table/column names | lowercase_snake_case |

| Aliases | Explicit AS keyword |

| Indentation | 4 spaces |

| Max line length | ~100 characters |

| Semicolons | Always close statements |

Why This Matters in Code Review

Badly formatted SQL creates two problems during review:

The reviewer can't spot logic errors because they're fighting the formatting
The author can't catch their own bugs because nesting obscures the query structure

With consistent formatting, a five-level JOIN query becomes scannable. Without it, you're debugging character-by-character.

Format Without Thinking About It

The SnappyTools SQL Formatter & Beautifier formats any SQL query in one paste. It handles indentation, keyword casing, and clause alignment. Takes 5 seconds and saves everyone who reads the code 5 minutes.

Runs in-browser, no upload, no account.

Does your team have a SQL style guide? Or does formatting vary by developer?

SQL Formatting and Beautification: Standards, Tools, and Best Practices

Sun, 31 May 2026 14:37:02 +0000

What a SQL Formatter Does

A formatter restructures query text into a consistent, indented layout — without changing what the query does. The database ignores whitespace; formatting is entirely for humans.

Before:

`<code>sql select u.id,u.name,o.total from users u inner join orders o on u.id=o.user_id where o.status='paid' order by o.total desc limit 100; </code>

Same query. Dramatically more readable.

Key Formatting Decisions

Clause alignment — Each major clause (SELECT, FROM, WHERE, GROUP BY, ORDER BY) starts a new line. Conditions within a clause are indented.

Leading vs trailing commas:
`sql -- Trailing (most common)

SELECT

id,

name,

-- Leading (easier to add/remove first item)

SELECT

,name

,email

SQL Dialects

SQL is not one standard. Each database has extensions:

| Database | Notes |

|---|---|

| PostgreSQL | ILIKE, :: casting, $$ dollar quoting, CTEs everywhere |

| MySQL | Backtick identifiers, LIMIT x,y, AUTO_INCREMENT |

| SQLite | Minimal syntax, no stored procedures |

| SQL Server (T-SQL) | TOP, NOLOCK, GETDATE(), GO |

| BigQuery | STRUCT, ARRAY_AGG, UNNEST, standard SQL mode |

A good formatter knows which dialect you're using. SnappyTools' SQL formatter supports all major dialects via a dropdown selector.

Beautification: Compressing SQL Back

The reverse of formatting — collapsing multi-line SQL to a single line — is useful for:

Embedding SQL in JSON configuration
Storing queries in environment variables
Code generation output
Sending queries in HTTP request bodies

Both directions are available at snappytools.app/sql-formatter-beautifier/.

Automating SQL Formatting in CI

For teams, use sqlfluff:

`bash pip install sqlfluff

Lint

sqlfluff lint queries/ --dialect postgres

Auto-fix

sqlfluff fix queries/ --dialect postgres

.sqlfluffconfig:<code>ini [sqlfluff] dialect = postgres indent_unit = space tab_space_size = 2 </code>

For pre-commit hooks:

Anti-Patterns to Avoid

Implicit joins (hard to see the join condition):
`sql -- Avoid

SELECT * FROM orders, users WHERE orders.user_id = users.id;

-- Better

SELECT * FROM orders INNER JOIN users ON orders.user_id = users.id;

SELECT in production — Not formatting, but worth mentioning: always name columns explicitly. SELECT breaks when the schema changes.

Inconsistent keyword case within one query:
`sql -- Bad

select id FROM users WHERE status='active';

-- Good

SELECT id FROM users WHERE status = 'active';

Summary

Originally published at https://snappytools.app/sql-formatter-beautifier/

5 SQL Habits That Make Your Queries Unmaintainable

Wed, 27 May 2026 10:09:37 +0000

Most SQL problems aren't about syntax. They're about habits. The same bad patterns show up in every codebase, written by developers who knew better but were in a hurry. Here are five to stop immediately.

1. Single-line queries

``sql

SELECT u.id, u.name, u.email, o.total FROM users u INNER JOIN orders o ON u.id = o.user_id WHERE u.active = 1 AND o.total > 100 ORDER BY o.created_at DESC LIMIT 50

This works. Nothing will break. But when you have to debug it at 2am — or when a colleague has to touch it — you'll regret every character on that line.

Fix: one clause per line, always.

<code>sql SELECT u.id, u.name, u.email, o.total FROM users u INNER JOIN orders o ON u.id = o.user_id WHERE u.active = 1 AND o.total > 100 ORDER BY o.created_at DESC LIMIT 50 </code>

Same query. Completely different to work with.

2. SELECT *

<code>sql SELECT * FROM users </code>

I know. It's fast to write. But it creates invisible dependencies — your application code may break silently when someone adds or removes a column. You also fetch data you don't need, which hits performance at scale.

Always name your columns. Yes, even for quick exploratory queries in development — the habit transfers to production code.

3. Implicit joins

<code>sql -- Old style SELECT u.name, o.total FROM users u, orders o WHERE u.id = o.user_id </code>

This is valid SQL from the 1980s. It still runs. But mixing data relationships (the u.id = o.user_id filter) with data filters (business logic in WHERE) is a recipe for accidental Cartesian products and confused reviewers.

Explicit JOIN syntax is unambiguous:

<code>sql SELECT u.name, o.total FROM users u INNER JOIN orders o ON u.id = o.user_id </code>

4. Magic numbers in conditions

<code>sql WHERE status = 2 WHERE type = 7 WHERE permission_level >= 4 </code>

What is status 2? What is type 7? You might know today. In three months, neither you nor anyone else will.

Use comments, constants in your application layer, or at minimum leave a comment in the SQL:

<code>sql WHERE status = 2 -- 2 = active WHERE type = 7 -- 7 = premium_user </code>`

Better yet, if your database supports it, use an ENUM or a lookup table.

5. No formatting before committing

Raw, unformatted SQL in migrations, stored procedures, and ORM raw queries gets committed and stays there forever. Once it's in the codebase, nobody reformats it — touching it means potentially breaking it.

The fix takes 10 seconds. Before you commit any SQL, paste it into a formatter. SnappyTools SQL Formatter handles 19 dialects (PostgreSQL, MySQL, SQL Server, SQLite, BigQuery, and more). Pick your dialect, choose your indent style, format, and copy. It runs in your browser — no setup, no signup.

Good SQL is readable SQL. These aren't academic style preferences — they directly affect how quickly bugs get caught in reviews, how fast new team members can understand the codebase, and how much time you spend debugging vs. shipping.

Pick the habit that's costing you the most time right now and fix it first.

Format SQL in seconds: SnappyTools SQL Formatter — 19 dialects, browser-based, no signup.

Originally published at https://snappytools.app/sql-formatter-beautifier/</a>

SQL Formatting: Why It Matters and How to Do It Fast

Sun, 24 May 2026 10:12:27 +0000

SQL is the language that powers most of the world's data — and it is also one of the most commonly written in the least readable way. A query that runs correctly often looks like this in the wild:

``sql

SELECT u.id,u.name,o.total FROM users u JOIN orders o ON u.id=o.user_id WHERE o.created_at>'2024-01-01' AND o.status='completed' ORDER BY o.total DESC LIMIT 50

Nobody reviews that in a pull request. Nobody debugs that at 2am. Formatting is not just aesthetic — it is a correctness tool.

What SQL Formatting Actually Does

A SQL formatter applies consistent rules to your query structure:

Uppercase keywords — SELECT, FROM, WHERE, JOIN, GROUP BY stand out visually from column names and values
Indentation — each clause starts on a new line; nested subqueries are indented further
One clause per line — makes it easy to add, remove, or comment out individual conditions
Aligned expressions — long SELECT lists and WHERE conditions line up vertically

The same query formatted:

<code>sql SELECT u.id, u.name, o.total FROM users u JOIN orders o ON u.id = o.user_id WHERE o.created_at > '2024-01-01' AND o.status = 'completed' ORDER BY o.total DESC LIMIT 50 </code>

The logic is identical. The readability is not.

Dialect Differences Matter

SQL has a standard, but every major database deviates from it. Formatting should reflect the dialect your team uses:

| Database | Notable formatting differences |

|----------|-------------------------------|

| MySQL | BACKTICK identifiers, LIMIT n, m syntax |

| PostgreSQL | $$ dollar-quoting, :: cast syntax |

| SQL Server / T-SQL | [BRACKET] identifiers, TOP n instead of LIMIT |

| BigQuery | Backtick identifiers, STRUCT and ARRAY types |

| SQLite | Minimal type system, AUTOINCREMENT quirks |

A good formatter understands which keywords are dialect-specific and won't flag them as errors.

When to Format

Code review — unformatted SQL in a migration or stored procedure is a review blocker. Format before you commit.

Debugging — when a query returns wrong results, formatting it makes the logical structure visible. Missing parentheses, wrong JOIN conditions, and misplaced AND/OR groupings reveal themselves immediately.

Refactoring — adding a new JOIN or changing a WHERE clause is far easier on formatted code. You can see exactly where to insert the new clause.

Sharing with a DBA — formatted queries communicate intent. A DBA who can't parse your query can't help you optimise it.

Legacy code review — inherited scripts often look like one long line. Format them first, understand them second.

Inline vs. Standalone Formatting

For day-to-day development: most IDEs have SQL formatting built in (VS Code SQLTools extension, DataGrip's built-in formatter, DBeaver's Ctrl+Shift+F). Set a standard style in your team's .editorconfig or IDE settings and format on save.

For quick ad-hoc formatting (a query from a log, a schema from Stack Overflow, a vendor-supplied migration): a browser-based tool is fastest. No configuration, no install, paste and go.

Building a Team SQL Style Guide

Consistent formatting is more important than which convention you choose. A short style guide prevents formatting debates in code review and makes automated formatting enforceable:

Keyword case: UPPERCASE keywords (near-universal convention)
Identifier case: lowercase table and column names
Indentation: 2 or 4 spaces (pick one, enforce it)
Join style: explicit JOIN always (never implicit comma-joins)
Trailing comma vs leading comma: leading commas ( , column_name) help with git diffs — controversial but worth a team discussion

Document the guide in your repo's README or a CONTRIBUTING.md and point your formatter configuration at it.

SQL Formatting in CI

For teams serious about consistency, format checking can run in CI:

sqlfluff — Python-based SQL linter + formatter with dialect support and CI integration
pgFormatter — PostgreSQL-specific, used in pre-commit hooks
prettier-plugin-sql — Prettier plugin for SQL files

A CI gate that rejects unformatted SQL is the same pattern as eslint or black` for JavaScript or Python. It removes the "formatting nit" category from code reviews entirely.

Try It in Your Browser

If you need to quickly format a query without installing anything, SnappyTools provides free, browser-based tools for developers. All processing happens in your browser — no code is sent to a server.

Clean queries are debuggable queries. Format early, format consistently, and your future self (and your teammates) will thank you.

Originally published at https://snappytools.app/

Understanding Logic, Reusability and Integrity On SQL ; Procedures, Functions and Transactions.

Sun, 10 May 2026 16:23:47 +0000

SQL is widely known for data querying and manipulation but systems do grow; data becomes larger; processes become repetitive and operations become sensitive. SQL has some features which enables it to be considered a fully fledged programming language. Some of the features which I discuss in this article are procedures, functions and transactions. Each of these concepts serve distinct purposes.
Procedures execute operations, functions return values, and transactions ensure those operations are safe.

Stored Procedures

These are set of SQL statements stored in the database and executed as a unit which are used to perform tasks such as UPDATE, INSERT, DELETE etc.
They are triggered by calling them and passing the expected parameters by the procedure.
Here is an example of a procedure:

CREATE OR REPLACE PROCEDURE increase_salary(p_dept TEXT, p_percent NUMERIC)
LANGUAGE plpgsql
AS \(
BEGIN
    UPDATE employees
    SET salary = salary + (salary * p_percent / 100)
    WHERE department = p_dept;
END;
\);

Here is how a procedure is called:

CALL increase_salary('IT', 10);

Functions

This is a reusable logic block which can return values and can be used inside queries for data selection.
It can be used with SELECT, WHERE and is great for reusability.

CREATE OR REPLACE FUNCTION get_avg_salary(p_dept TEXT)
RETURNS NUMERIC
LANGUAGE plpgsql
AS \(
BEGIN
    RETURN (
        SELECT AVG(salary)
        FROM employees
        WHERE department = p_dept
    );
END;
\);

Here is an example of how the function can be used:

SELECT name, salary
FROM employees
WHERE salary > get_avg_salary(department);

Transactions

Transactions are used to group a set of code operations in which if any of it fails then the whole execution is aborted. This ensures that data is only changed when the full code execution structure is successful.
Transactions are best for data safety as they prevent partial updates.

BEGIN;

UPDATE accounts
SET balance = balance - 100
WHERE id = 1;

UPDATE accounts
SET balance = balance + 100
WHERE id = 2;

COMMIT;

If it fails:

ROLLBACK;

How They Work Together

It will not be complete to conclude the article without showing an example of how these concepts can be used together.
Here is a simple scenario showing the use of the three concepts interdependently.
Function:

CREATE OR REPLACE FUNCTION get_balance(acc_id INT)
RETURNS NUMERIC AS \(
BEGIN
    RETURN (SELECT balance FROM accounts WHERE id = acc_id);
END;
\) LANGUAGE plpgsql;

Procedure:

CREATE OR REPLACE PROCEDURE transfer_money(from_id INT, to_id INT, amount NUMERIC)
LANGUAGE plpgsql
AS $$
BEGIN
    IF get_balance(from_id) < amount THEN
        RAISE EXCEPTION 'Insufficient funds';
    END IF;

    UPDATE accounts SET balance = balance - amount WHERE id = from_id;
    UPDATE accounts SET balance = balance + amount WHERE id = to_id;
END;
$$;

Transaction:

BEGIN;
CALL transfer_money(1, 2, 100);
COMMIT;

Conclusion

SQL has a wide range of capabilities more than just being a querying language. Having features such as procedures, functions, transactions and many others helps it to be an efficient tool for use on data in whichever way needed. Mastering it comes a long way in helping the analysis of data directly from the database.

Sub-queries vs Window Functions vs Common Table Expressions: Which Should You Use in SQL?

Tue, 28 Apr 2026 16:33:58 +0000

SQL is the most used tool for data manipulation but what happens when there are some concepts that seem to work towards the same ultimate outputs. As queries become more complex, developers often encounter different techniques that appear to produce similar results—namely sub-queries, common table expressions (CTEs), and window functions.
At first glance, these concepts can seem interchangeable. You might solve the same problem using any of them, which raises a natural question: which one is better?
The goal is not to choose one, but to understand when each is the right tool.
In this article I will discuss about sub-queries, window functions and common table expressions;

Sub-Queries
These are queries inside other queries and are executed either before or alongside the outer query. They can return scalar value, row or table.

SELECT *
FROM employees e
WHERE salary = (
    SELECT MAX(salary)
    FROM employees
    WHERE department = e.department
);

Window functions
These are used when you are trying to preserve your rows. Used together with components such as - OVER(),PARTITION BY,ORDER BY and common functions such as - SUM(),AVG(),RANK(),LAG(),LEAD()

SELECT employee_id,first_name, last_name,salary, job_title, department
FROM (
  SELECT *,
    RANK() OVER(PARTITION BY department ORDER BY salary DESC) AS rnk
  FROM employees
) t
WHERE rnk = 1;

Common Table Expressions (CTEs)
These are temporary results set that are initialized using -WITH. They are often used to improve readability and modularity of scripts.

WITH max_salary AS (
    SELECT department, MAX(salary) AS max_sal
    FROM employees
    GROUP BY department
)
SELECT e.*
FROM employees e
JOIN max_salary m
    ON e.department = m.department
WHERE e.salary = m.max_sal;

All these features of SQL are used to perform advanced data querying and transformations. They can all be tweaked to perform almost similar queries but it all depends on what the final output is expected to be.

When to use what
Each of the features has an instance when it is the most convenient over the others.

For instance when it is a simple filtering such as retaining all those above average marks then a sub-query is the best option.
When we are more focused on row level analytics then the window functions come in handy for that task.
In a situation where it is a multi-step logic and re-usability is a core factor the common table expressions are best suited for use.

Using in collaboration
The best thing about this features is that they can be used together for the best solution. This helps in the data transformation where there are steps within it in which either is considered to be superior over the other.
Here is an example where all of them are used together:

with ranked AS (
    SELECT 
        *,
        AVG(salary) OVER (PARTITION BY department) AS avg_salary
    FROM employees
)
SELECT *
FROM ranked
WHERE 
    salary > avg_salary
    AND salary > (
        SELECT AVG(salary) FROM employees
    );

Best Practices
These are some of the recommended practices when working with these SQL features:

It is best to avoid deep nesting of queries by using CTEs which help in the structuring of the script for easy readability.
When using CTEs it is necessary to name them well so that their purpose can be identified easily without having to read all the code snippets manually.
Window functions are often more efficient than sub queries in large datasets, so where possible it is best to use window functions over sub queries but in instances of small datasets any can be used depending on preference.

Conclusion
These are to be considered more as complementary tools than competing tools. In most advanced queries you will have to use them together or interchangeably to get to the results.
What makes a major difference is how you use them as they can heavily impact the performance. It is best to master all of them and understand the strength and weaknesses of each so that when it get to a point of picking one or more you do it from a point of information.

The 5 Biggest Traps When Writing Your Own SQL Injection Filter

Wed, 22 Apr 2026 07:54:55 +0000

SQL injection is one of those vulnerabilities every developer thinks they understand — until their handcrafted “secure” filter gets bypassed by a payload they never anticipated.

The pattern repeats constantly:

A project launches fast.
Someone adds a few blacklist rules.
A regex grows into a monster.
Edge cases accumulate.
Attackers eventually walk straight through it.

Despite two decades of awareness, SQL injection remains part of the OWASP Top 10 because many defenses are still fundamentally flawed at the architectural level.

This article breaks down five common traps teams fall into when building their own SQL injection filtering logic, along with real-world bypass techniques that continue to work in production systems.

Why DIY SQL Injection Filters Keep Failing

Most custom filters are based on a false assumption:

“Malicious SQL can be reliably identified through string matching.”

That assumption collapses immediately under real attacker behavior.

Modern SQL injection payloads are not static strings. They are:

Context-aware
Encoding-aware
Database-specific
Parser-dependent
Often multi-stage

Attackers are not fighting your regex. They are fighting the SQL parser behind your application.

That distinction matters.

Trap #1: Keyword Blacklists Are Fundamentally Fragile

The classic beginner defense looks like this:

blocked = ["select", "union", "drop", "--"]for keyword in blocked:    if keyword in user_input.lower():        reject()

This fails almost instantly.

Attackers can mutate payloads endlessly while preserving SQL semantics.

Examples:

SeLeCtUN/**/IONUNI%0aON/*!50000SELECT*/

MySQL comments alone create countless bypass opportunities:

SELECT/**/password/**/FROM/**/users

Or:

UNIunionON SELECT

If your filter removes union, the payload may still parse successfully depending on reconstruction behavior.

The core issue is simple:

SQL is not a plain text format. It is a grammar.

Trying to secure a grammar using substring matching is structurally weak.

Trap #2: Encoding and Charset Edge Cases

This is where many “works in testing” filters completely collapse in production.

A famous example is the wide-byte injection problem in older MySQL + PHP environments using GBK encoding.

Example payload:

%bf%27 OR 1=1 --

Under certain multibyte encodings:

%bf%27 becomes a valid multibyte character
The quote escaping breaks
The trailing SQL becomes executable

The application believes the quote was escaped safely.

The database parser disagrees.

That parser mismatch is exactly what attackers exploit.

Other common encoding tricks include:

Double URL encoding
UTF-16 transformations
Unicode normalization
Mixed charset confusion
Overlong UTF-8 sequences

Most homemade filters never even inspect requests at this layer.

Trap #3: Second-Order SQL Injection

Many teams only inspect incoming requests.

That misses one of the nastiest categories of SQL injection entirely.

Second-order SQL injection happens when:

Malicious input is stored safely
The payload appears harmless initially
Another backend process later reuses the stored data unsafely
Injection triggers downstream

Example:

A registration form stores this username safely:

admin'--

No immediate damage occurs.

Months later:

query = f"SELECT * FROM audit WHERE user = '{stored_username}'"

Now the payload executes.

This is why “input filtering” alone is insufficient.

The dangerous moment is often not ingestion. It is reuse.

Second-order injection is especially common in:

Admin dashboards
Reporting systems
Legacy migration scripts
Analytics tooling
Scheduled jobs
Internal APIs

Teams frequently assume internal data is trustworthy. Attackers rely on that assumption.

Trap #4: Database Dialect Differences

A filter tested against MySQL may fail completely against PostgreSQL or MSSQL.

Each database engine has:

Different syntax
Different comments
Different casting behavior
Different concatenation operators
Different escaping semantics

Examples:

MySQL:

SELECT @@version

PostgreSQL:

SELECT version()

MSSQL:

SELECT @@version

String concatenation differences:

'a' + 'b'

'a' || 'b'

A blacklist that blocks one dialect often misses another.

Even worse, many modern applications support multiple database backends simultaneously through ORM abstraction layers.

Your filter may not even know which parser ultimately executes the query.

Trap #5: Regex Maintenance Becomes an Operational Nightmare

This is the part many engineering teams underestimate.

The first regex seems manageable:

(select|union|drop|insert)

Six months later:

(?i)(union(\s+all)?\s+select|select.+from|benchmark\(|sleep\(|load_file\()

A year later:

False positives explode
Legitimate requests break
Performance degrades
Nobody fully understands the rules anymore

Now every framework upgrade becomes risky.

Every new database feature expands attack surface.

Every edge case requires another patch.

Eventually the “simple filter” evolves into an unmaintainable shadow-WAF hidden inside business code.

At that point, your application team is spending engineering time rebuilding a lower-quality security engine.

That is usually the wrong tradeoff.

The Real Problem: SQL Injection Defense Is a Parsing Problem

The central mistake behind most DIY defenses is treating SQL injection as a text-matching problem.

It is not.

It is a parser interpretation problem.

Attackers succeed by creating discrepancies between:

Application-layer assumptions
Middleware transformations
Database parser behavior

That is why modern defenses increasingly rely on:

AST analysis
Semantic inspection
Behavioral modeling
Context-aware parsing
Multi-layer decoding

Rather than brittle keyword matching.

What Actually Works

Prepared statements and parameterized queries remain the foundation.

But real production systems are messy:

Legacy code exists
Dynamic query builders exist
Third-party plugins exist
Internal tools exist
Old endpoints survive for years

Which is why runtime protection still matters.

A modern WAF should not simply scan for suspicious words.

It should understand:

SQL grammar structure
Payload intent
Encoding transformations
Obfuscation techniques
Parser behavior

That is a fundamentally different level of analysis.

Solutions like SafeLine take this approach by using semantic analysis instead of relying purely on manually maintained blacklist rules. The practical advantage is operational: developers stop wasting time maintaining endless regex patches inside application code and can focus on actual business logic instead.

That division of responsibility is usually the sane architecture.

Application code should implement secure query practices.

Traffic inspection engines should handle hostile payload analysis at scale.

Final Thoughts

The history of SQL injection defense is full of teams attempting to out-regex attackers.

It rarely ends well.

Attackers only need one parser discrepancy.

Defenders maintaining blacklist logic need to correctly anticipate all of them.

That asymmetry is why handcrafted SQL injection filters keep failing — even in mature systems.

The deeper lesson is architectural:

If your security model depends on continuously guessing every future payload mutation attackers might invent, the model is already losing.

SafeLine Live Demo: https://demo.waf.chaitin.com:9443/statistics
Website: https://safepoint.cloud/landing/safeline
Docs: https://docs.waf.chaitin.com/en/home
GitHub: https://github.com/chaitin/SafeLine

5 Things This Playwright SQL Fixture Does So You Don't Have To

Mon, 13 Apr 2026 07:53:00 +0000

5 Things This Playwright SQL Fixture Does So You Don't Have To

If you write Playwright tests that touch a database, you know the boilerplate: connect, seed, clean up, close. Repeat for every file. Hope nothing leaks.

@playwright-labs/fixture-sql handles that boilerplate. Here is exactly what it takes off your plate.

1. Opens and closes connections automatically

Without the fixture, every test file that needs a database connection looks like this:

let client: SqlClient;
beforeEach(async () => { client = await createConnection(); });
afterEach(async () => { await client.close(); });

With the fixture, it is one line:

test.use({ sqlAdapter: sqliteAdapter(':memory:') });

test('my test', async ({ sql: db }) => {
  // db is ready — no setup needed
  // db.close() is called by the fixture after this function, even if it throws
});

The connection lifecycle is entirely handled by Playwright's fixture system.

2. Prevents connection leaks when tests fail

When a test throws an error, code in afterEach might not run correctly depending on how the error propagates. Fixtures are different — Playwright guarantees their teardown phase runs regardless of test outcome.

Every connection opened through sql or useSql is tracked and closed in teardown. You cannot leak a connection through a failing test.

3. Catches SQL errors at compile time

The fixture integrates with @playwright-labs/sql-core's compile-time SQL validator. Write your queries with sql("…") instead of a raw string:

// ❌ runtime error
await db.query('SELECT * WHERE id = ?', [1]);

// ✅ compile error — TypeScript catches missing FROM before tests run
await db.query(sql("SELECT * WHERE id = ?"), [1]);
//             ~~~~~~~~~~~~~~~~~~~~~~~~~~~ never

The type system models SQL grammar as a finite-state machine. Invalid structure → never. Wrong number of params → compile error.

const q = sql("SELECT name FROM users WHERE id = ?");
// typeof q → SqlStatement<[unknown]>

await db.query(q, [1]);        // ✅
await db.query(q);             // ❌ params required
await db.query(q, [1, 2, 3]); // ❌ too many params

4. Handles multiple connections with one call

Need to test replication, or talk to two databases in one test? useSql opens additional connections and registers them all for teardown:

test('primary → replica', async ({ useSql }) => {
  const primary = await useSql(pgAdapter(process.env.PRIMARY_URL!));
  const replica  = await useSql(pgAdapter(process.env.REPLICA_URL!));

  await primary.execute(sql("INSERT INTO events (type) VALUES ($1)"), ['login']);

  const { rows } = await replica.query(sql("SELECT * FROM events WHERE type = $1"), ['login']);
  expect(rows).toHaveLength(1);
});
// both connections closed automatically

No manual cleanup. No try/finally.

5. Works with PostgreSQL, MySQL, and SQLite out of the box

Same API, three drivers:

// SQLite — in-memory, zero infrastructure
import { sqliteAdapter } from '@playwright-labs/fixture-sql/sqlite';
test.use({ sqlAdapter: sqliteAdapter(':memory:') });

// PostgreSQL
import { pgAdapter } from '@playwright-labs/fixture-sql/pg';
test.use({ sqlAdapter: pgAdapter(process.env.DATABASE_URL!) });

// MySQL / MariaDB
import { mysqlAdapter } from '@playwright-labs/fixture-sql/mysql';
test.use({ sqlAdapter: mysqlAdapter(process.env.MYSQL_URL!) });

Drivers are optional peer dependencies — install only the one you use.

Bonus: editor autocomplete

Pair with @playwright-labs/ts-plugin-sql and add a schema file to get table/column autocomplete and diagnostics inside sql templates directly in your editor.

# generate schema types from your live DB
pnpm playwright-labs-sql-core pull --adapter pg --url postgresql://localhost/mydb --out ./src/db-types.ts

// tsconfig.json
{
  "compilerOptions": {
    "plugins": [{ "name": "@playwright-labs/ts-plugin-sql", "schemaFile": "./src/db-types.ts" }]
  }
}

Get started

pnpm add -D @playwright-labs/fixture-sql
pnpm add -D better-sqlite3   # or pg / mysql2

Source: github.com/vitalics/playwright-labs — MIT license.

Design Data Model for Multi-Tenant RBAC

Mon, 13 Apr 2026 04:29:26 +0000

Part 1: Identity Foundation

Behavior Description

Assume we are building a KPI management platform. Multiple companies can create accounts (tenants) and let their employees access the platform.

A user can be assigned to one or several roles. Their permissions are the sum of those roles' permissions.
A user cannot access features outside their permissions. Even if they know the API specs and bypass the UI, they will still be blocked at the API layer.
Admins can change a user's roles or a role's permissions at any time. The only inconvenience is that the affected user needs to log out and log back in for the changes to take effect.

Sounds straightforward, right? Let's look at what happens under the hood.

Deriving the Data Model from Use Cases

Before implementing, I take time to think about what the system needs to do and what data structures are required to support those use cases.

The first thing we hear in the spec is "multiple tenants." A system that serves multiple companies and their users. That gives us at least two objects to start with: tenants and users.

Note: I use the plural convention for table names throughout this post.

Defining the Relationship Between Tenants and Users

Now we have the foundation, but the relationship between tenants and users is not yet defined. To define it, we need to find the right constraint. There are two cases:

Case 1: One tenant can have many users. One user must use a unique email per tenant.
Case 2: One tenant can have many users. One user can use the same email across different tenants.

Consider this scenario: you are a contractor working for two different companies that both use this platform. If we go with Case 1, the best you can do is maintain two separate emails for two separate tenants on the same system. That is a real inconvenience.

Notice I used the word person instead of user here. Technically, every tenant is independent. There is no good reason to force a person to use a different email just to become a user of another tenant.

Case 2 is the preferred design.

This raises a natural question: if a user can log in with the same email across multiple tenants, how do we know which tenant they belong to?

The answer is that when a user logs in, we need to identify the tenant. There are a few approaches:

Ask the user to type a tenant code in the login form. Bad idea.
Provide a typeahead field to help the user find their tenant code. Still a bad idea.
Give each tenant a subdomain or sub-path, such as tenant1.ourplatform.com or ourplatform.com/tenant1. This is the widely accepted approach, especially the subdomain variant. The only tradeoff is that instead of a standard SSL certificate, you need a wildcard SSL certificate.

The Identity Table

Another natural question follows: if users share an email across tenants, can they also share a password?

No. Password policies can differ per tenant, so sharing passwords across tenants is not acceptable. This leads us to a third object: the identities table. An identity represents a user's membership in a specific tenant, containing tenant's specific information.

The relationship is clear: one user can have many identities, and each identity belongs to exactly one tenant. With this simple three-table model - tenants, users, and identities - we satisfy the core constraint cleanly: a person can use the same email across multiple independent tenants.

Recognizing the Many-to-Many Relationship

Look closely at the diagram. Both the N (of 1:N) ends from users and tenants point to the identities table. This is a sign that users and tenants have an N:N relationship - a user can belong to multiple tenants, and a tenant can have multiple users. The identities table is effectively the join table.

With that insight, we can express the data model more precisely:

In Part 2, we will build on this foundation and design the roles and permissions model.

How to Handle NULLs in Snowflake: A Complete Guide!

Tue, 07 Apr 2026 14:39:38 +0000

Dear Readers,

NULL in data is like a weed, it can mess up your analysis and calculations, and ultimately bleed the value out of your analytics work. In this article, we’ll dive into what NULL is, why it exists, when it appears, and how to deal with it in Snowflake ?

So, without further ado, let’s meet this culprit.

What NULL?

In SQL, NULL is basically the database equivalent of a shrug.It is a special marker that represents a missing or unknown value. It's not the same as an empty string ('') or the number zero (0). It simply means “I don’t know” or “there’s no value here at all.”

Why NULL Exists?

NULL is essential because sometimes your data just doesn’t exist yet, or never will. NULL allows the database to acknowledge the gap without faking a value.

Eg:

A customer doesn’t have a middle name.
A product doesn’t have a discount yet.
A delivery date is unknown at the time of record insertion.

When NULL Appears?

NULL occurs anytime there is a lack of value from the user.

Common reasons include:

Optional fields with no provided value.
Data not yet available (future delivery dates, pending approvals)
Inapplicable values (someone’s “date of divorce” who never married)

How to Deal with NULL in Snowflake?

Snowflake provides multiple functions and methods to deal with NULL dataset aka unknown data similar to other databases. I have listed some of them below:

Use IS NULL / IS NOT NULL to identify the NULL data.
Use COALESCE(col, default) or IFNULL(col, default) to replace them in select , joins, aggregates etc
Use explicit filter on WHERE clauses to include NULL rows as it can can drop NULL rows unexpectedly.
Use NVL, CASE WHEN, or similar functions to handle them in calculations or strings.
Comparisons with NULL return UNKNOWN in SQL’s three-valued logic. Use COALESCE or IS NULL checks to avoid surprises.

Eg:

==Using NULL/NOT NULL

-- Finds all employees without a manager
SELECT name FROM employees WHERE manager_id IS NULL;

-- Finds all employees with a manager
SELECT name FROM employees WHERE manager_id IS NOT NULL;

==Using COALESCE

-- Calculates total compensation, treating a NULL bonus as 0

SELECT salary + COALESCE(bonus, 0) AS total_compensation FROM employees

The Trouble NULL Causes:

1. Comparison Traps

NULL comparisons are a common pitfall because standard operators like = or != don’t work as expected. Any comparison involving NULL returns UNKNOWN, not TRUE or FALSE. Use IS NULL or IS NOT NULL to handle the traps.

Eg:

--Evaluates to UNKNOWN when NULL = NULL

SELECT *
FROM employees
WHERE manager_id = NULL;

-- The correct way to find employees with a NULL manager_id is:
SELECT *
FROM employees
WHERE manager_id IS NULL;

2. Math & Concatenation Failures:

NULL acts like a virus in arithmetic and string concatenation operations. Any operation involving NULL will result in NULL.

Eg:

--Return of NULL total+compensation when bonus is NULL:

SELECT salary + bonus AS total_compensation
FROM employees;

--The correct way to add bonus is use COALESCE :

SELECT salary + COALESCE(bonus, 0) AS total_compensation
FROM employees;

3. Aggregate Surprises:

Aggregate functions like SUM() and AVG() ignore NULL values. This can lead to misleading results if you’re not aware of it.

Eg:

--Excludes NULL rows

SELECT AVG(salary) AS avg_salary, SUM(salary) AS total_salary
FROM employees;

--Use COALESCE to treat NULL salaries as 0 to get more accurate company-wide average

SELECT AVG(COALESCE(salary, 0)) FROM employees;

4. Join Issues:

NULL values in join keys never match each other. If an employee has a NULL in manager_id, that row won’t link to any id in the employees table even if other employees also have NULL as manager_id.

Use a LEFT JOIN + COALESCE to tackle the NULL issue.In following demonstrated way, you keep all employees in the output, and NULL managers are clearly labeled without risking false matches.

Eg:

--This INNER JOIN excludes employees with no manager_id

SELECT

   E.name AS employee,
   M.name AS manager

FROM employees AS E
INNER JOIN
employees AS M
ON E.manager_id = M.id;

--Use COALESCE to include employees without a manager

SELECT
E.name AS employee,
COALESCE(M.name, 'No Manager') AS manager
FROM employees E
LEFT JOIN employees M
ON E.manager_id = M.id;

5.Logic Surprises

In SQL’s three-valued logic (TRUE, FALSE, UNKNOWN), an UNKNOWN result in a WHERE clause acts like FALSE and discards the row. However, UNKNOWN can have different effects in other clauses such as CASE.

Eg:

--Logic Mishandling for NULL data.

SELECT
CASE

WHEN bonus > 10000 THEN 'High Bonus'
WHEN bonus > 5000 THEN 'Mid Bonus'
ELSE 'Low or No Bonus'

END AS bonus_category
FROM employees;

-- Explicitly checks for NULL first

SELECT
CASE

WHEN bonus IS NULL THEN 'No Bonus'
WHEN bonus > 10000 THEN 'High Bonus'
WHEN bonus > 5000 THEN 'Mid Bonus'
ELSE 'Low Bonus'

END AS bonus_category
FROM employees;
Closing thoughts,

NULLs aren’t a bug, they’re part of SQL’s design.NULL is tricky. The safest approach is to scan your dataset for every attribute that could be NULL, then apply handling strategies suited to your business context. In Snowflake, a few well-placed COALESCE, IS NULL, and CASE statements can save you from major data quality headaches.

Hope you enjoyed the article! If you like my writing, feel free to connect with me on Linkedin and follow for more such interesting reads.

The Data Engineer’s Sandbox: Architecting for Autonomy with LXD

Wed, 25 Mar 2026 14:21:54 +0000

As Data Engineers, we often live in the cloud. Our daily lives are governed by S3 buckets, Glue jobs, and Kafka clusters. But there is a silent pressure that every senior professional feels: the cost of experimentation.

How many times have you hesitated to spin up a resource just to validate a simple schema or a transformation logic? We've all been there—balancing the need for agility with the responsibility of cost-awareness.

I’ve always believed that autonomy is a superpower. True technical mastery comes when you can prototype "blindly," without an internet connection, directly on your machine, and with zero impact on your company's AWS bill.

Why I moved away from heavy VMs

For years, the choice was either a heavy Virtual Machine (VM) that drained my laptop’s RAM or a Docker container that, while fast, sometimes lacked the persistence and system-level isolation I needed for a long-term practice environment.

When I started working with Ubuntu 24.04, I found the perfect middle ground: LXD. It offers the "system feel" of a VM with the "instant speed" of a container. It became my go-to for building local SQL sandboxes where I can fail fast, learn deep, and spend zero.

The Blueprint: Setting up your Local Engine

The goal is simple: a functional database server running in minutes.

The Foundation

We start by initializing the hypervisor to manage our internal networking.

sudo snap install lxd
sudo lxd init --auto

Launching the Instance

We choose a lightweight Ubuntu image as our dedicated server.

lxc launch ubuntu:24.04 sql-practice

The "Hardware" Setup

We enter the container and transform that empty shell into a robust PostgreSQL engine.

lxc exec sql-practice -- bash
apt update && apt install -y postgresql postgresql-contrib

From Infrastructure to Impact

Setting up the "pipes" is only half the battle. As engineers, we care about the data. In my experience building event-driven architectures and streaming pipelines with Kafka, I’ve learned that a local sink is the best place to test idempotency and schema evolution before touching a production environment.

Using the COPY command for massive local ingestion or pushing CSV files directly into the container (lxc file push) are the types of "unsexy" but highly efficient workflows that define a professional.

Closing thoughts: Design first

My philosophy is simple: Systems must be resilient and observable from day one. That includes our local dev environments. Complexity is a cost, and simplicity scales better.

By mastering these local sandboxes, we aren't just saving a few dollars on an AWS bill; we are reclaiming our creative freedom as builders.

About the Author

I am Rocío Baigorria, a Senior Data Engineer and co-leader of AWS Girls Argentina. I specialize in building "institutional-grade" data platforms where reliability, cost-efficiency, and business impact are core design principles.

Connect with me on LinkedIn: https://www.linkedin.com/in/rociobaigorria/

Explore my Resources: https://tr.ee/2t-m587551

Getting Started with PostgreSQL: Best Practices from Development to Production

Sun, 15 Mar 2026 17:40:28 +0000

Databases are the backbone of modern applications. Whether you're building APIs, SaaS platforms, or enterprise systems, choosing the right database and using it correctly can make a huge difference in performance and scalability.

One of the most reliable and powerful databases available today is PostgreSQL.

PostgreSQL is known for its stability, extensibility, performance, and strong SQL compliance, which is why many modern systems rely on it for production workloads.

In this guide, we’ll explore how to get started with PostgreSQL and follow best practices from development all the way to production environments.

Why Choose PostgreSQL?

PostgreSQL has become a favorite among developers and companies because it offers:

Advanced SQL support
Strong data integrity
JSON support for hybrid workloads
Extensibility through extensions
Excellent performance for large-scale systems

Many major platforms rely on PostgreSQL, including Netflix, Apple, and Instagram.

It works well for:

web applications
analytics workloads
enterprise systems
microservices architectures

Installing PostgreSQL

You can install PostgreSQL locally using the official installer or through a package manager.

Typical installation steps:

# macOS (Homebrew)
brew install postgresql

# Ubuntu
sudo apt install postgresql

# Windows
Download installer from postgresql.org

After installation, start the database service and connect using the CLI tool:

psql -U postgres

Basic Database Setup

A typical project setup includes:

CREATE DATABASE myapp;

CREATE USER myapp_user WITH PASSWORD 'securepassword';

GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;

This separation between database owner and application user is an important security best practice.

Designing Tables Properly

Good database design is essential for long-term scalability.

Example table:

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email TEXT UNIQUE NOT NULL,
    name TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Important principles:

always define a primary key
enforce unique constraints where necessary
avoid nullable fields unless truly optional
use timestamps for auditing

Use Indexes for Performance

Indexes help speed up queries significantly.

Example:

CREATE INDEX idx_users_email ON users(email);

Indexes are useful for:

search queries
filtering large datasets
join operations

However, too many indexes can slow down writes, so use them strategically.

Use Migrations Instead of Manual Changes

Schema changes should always be version-controlled.

Popular migration tools include:

Prisma
TypeORM
Flyway

Example migration:

ALTER TABLE users ADD COLUMN last_login TIMESTAMP;

This ensures that all environments stay consistent.

Use Transactions for Data Safety

Transactions ensure that multiple operations succeed or fail together.

Example:

BEGIN;

INSERT INTO orders (...) VALUES (...);
UPDATE users SET balance = balance - 100 WHERE id = 1;

COMMIT;

If something fails:

ROLLBACK;

Transactions prevent partial updates and inconsistent data.

Use Connection Pooling

Opening a database connection for every request is expensive.

Connection pooling improves performance by reusing connections.

Popular tools include:

PgBouncer
pg-pool

Connection pooling becomes essential in high traffic production systems.

Logging and Monitoring

Monitoring your database helps detect issues early.

PostgreSQL provides built-in logging capabilities.

You can monitor:

slow queries
long-running transactions
connection usage

Tools commonly used with PostgreSQL include:

pgAdmin
Grafana

Monitoring is crucial for production reliability.

Backup Strategy for Production

Always have automated backups.

Typical strategies include:

Daily backups:

pg_dump myapp > backup.sql

Full database backups:

pg_dumpall > full_backup.sql

For production systems, consider:

automated cloud backups
replication clusters
point-in-time recovery

This protects your data from accidental loss.

Scaling PostgreSQL in Production

As your application grows, scaling becomes important.

Common scaling strategies include:

Read Replicas

Separate read queries from write operations.

Example architecture:

Primary DB → Write operations
Replica DB → Read queries

This improves performance under heavy load.

Partitioning Large Tables

When tables become very large, partitioning improves performance.

Example:

CREATE TABLE logs_2025 PARTITION OF logs
FOR VALUES FROM ('2025-01-01') TO ('2026-01-01');

Partitioning helps with:

faster queries
easier data management
improved maintenance

Security Best Practices

Production databases must be secured properly.

Important steps include:

enforce strong passwords
restrict database network access
use SSL connections
avoid superuser access for applications

Principle of least privilege is essential.

Development vs Production Setup

Development environments can be simple, but production requires more structure.

Development setup:

local PostgreSQL instance
minimal security configuration
small datasets

Production setup:

managed PostgreSQL service
automated backups
monitoring and alerts
connection pooling

Many companies use managed services like Amazon RDS or Azure Database for PostgreSQL.

Final Thoughts

PostgreSQL is one of the most powerful and reliable databases available today.

But simply installing it is not enough. Building scalable systems requires:

proper schema design
indexing strategies
migration workflows
monitoring and backups
production-grade security

When used correctly, PostgreSQL can power applications from small startups to massive enterprise platforms.

If you're building modern applications today, PostgreSQL is definitely a database worth mastering.

https://www.enterprisedb.com/downloads/postgres-postgresql-downloads

Gnoke Dataforge

Sat, 14 Mar 2026 06:32:54 +0000

The Community

Vibe developers — people who ship fast with AI assistants like Claude, Cursor, or ChatGPT.

One thing I've learned after years of building: when you hand an AI assistant a well-structured schema upfront, it hallucinates less, asks fewer clarifying questions, and produces far better code. But there was no lightweight, offline-first tool to quickly model a table and export it in AI-ready formats. So I built one.

What I Built

Gnoke DataForge — a browser-based relational schema modeler that runs 100% offline.

You open it, design your tables visually, define column types and constraints (PK, NOT NULL, UNIQUE), and export in formats your AI assistant can immediately use:

.json — flat data format
.entity.json — structured entity format ideal for pasting into AI prompts
.sql — CREATE TABLE + INSERT statements
.sqlite — a real binary SQLite database file
.csv — universal fallback

Projects are saved locally via IndexedDB. No account. No server. No telemetry.

Demo

???? Live: https://edmundsparrow.github.io/gnoke-dataforge

Open it, edit the demo table, switch to SQLite mode to define types and constraints, then hit Export — all without leaving the browser.

Code

{% embed https://github.com/edmundsparrow/gnoke-dataforge %}

How I Built It

The entire app is vanilla HTML, CSS, and JavaScript — no framework, no build step.

SQLite in the browser via sql.js (WebAssembly) — handles .sqlite export and import
IndexedDB for local project persistence — each saved table is stored as a real SQLite binary blob
A custom JSON normalizer that accepts multiple JSON shapes (array of objects, entity format, flat arrays) so importing from AI-generated output just works
Light/dark theme, mobile drawer, and a skip-intro toggle for repeat visits

The design prioritizes speed: open → build → export → paste into AI. That's the whole loop.

Beyond the Basics: 5 Game-Changing Secrets of SQL Joins and Window Functions

Mon, 02 Mar 2026 07:40:25 +0000

1. Introduction: The Data Relationships Hook

Think of a database as a digital filing cabinet. In this architecture, information is organized into drawers—or schemas—such as HR, Finance, or Sales. Within these drawers sit the documents, which represent our individual tables.

While this structure ensures clean organization, real-world business intelligence rarely lives in a single drawer. To answer strategic questions, you must bridge the gaps between disparate tables without creating a "data mess". Whether you are joining sales records with inventory levels or HR data with finance budgets, mastering these relationships is what separates a basic query-writer from a Senior Architect. This guide distills complex SQL operations into high-impact takeaways, focusing on maintaining relational integrity while scaling your analysis.

2. The Horizontal vs. Vertical Divide: Joins vs. Unions

The most fundamental distinction in data architecture is how you choose to expand your result set.

Joins combine tables horizontally. They add columns based on a shared key (like a Primary or Foreign Key), making the result set "wider".
Unions combine tables vertically. They stack rows from one dataset on top of another, making the result set "longer".

For a UNION to be architecturally sound, the query must meet three strict structural requirements: the columns must be in the same order, have the same count, and—most importantly—possess matching data types. As an architect, you must remember that the database engine is indifferent to your column aliases but ruthless regarding data types.

"The database doesn't care about naming but cares about data types in Unions... If a VARCHAR is matched with an INT, the database will throw a mismatch error." — Data with Baraa

The Mental Model: Distinct vs. All From a strategy perspective, think of UNION as a DISTINCT operation for stacked rows; it automatically removes duplicates. If you need the "data as is" without the overhead of deduplication, UNION ALL is your preferred tool.

Sample Query: Vertical Stacking

``-- Combining customer and employee names into a single master list
SELECT first_name, last_name FROM customers
UNION
SELECT first_name, last_name FROM employees
LIMIT 100;

``

3. The "Workhorse" Joins: Why Left is Often Better than Right

In relational logic, the Inner Join is the default state. When you use the JOIN keyword, SQL is explicitly looking for matching values in the ON statement. If a record in the first table—for example, a customer who hasn't placed an order yet—has no match in the second table, that record "disappears" from the final result set.

For high-level reporting where data integrity is paramount, the Left Join is the industry workhorse. It preserves every record from the left (first) table and fills missing data from the right table with NULL values. This ensures you don't accidentally drop crucial business entities (like customers or products) simply because they lack transaction history.

Architectural Insight: The Right Join is functionally redundant. Any Right Join can be reframed as a Left Join by simply reordering the tables. In production environments, Left Joins are the standard because they align with a left-to-right reading flow, making queries significantly easier to audit and maintain.

Sample Query: Production-Ready Left Join
``-- Retrieving all customers and any associated event data
SELECT
c.customer_id,
c.first_name,
e.event_name
FROM customers AS c
LEFT JOIN events AS e
ON c.customer_id = e.customer_id
WHERE c.customer_id IS NOT NULL
LIMIT 500;

``

4. The Cartesian Chaos: Understanding the Cross Join

A Cross Join represents the "Cartesian Product" of two tables. Unlike other joins, it ignores matching values entirely, attaching every row of the second table to every row of the first. If you join a 1,000-row table with a 500-row table, you will generate a massive result set of 500,000 rows.

In standard relational reporting, Cross Joins are often avoided because they lack the primary/foreign key bond that defines logical relationships. Joining a car table and an item table by a shared color attribute "wouldn't make any sense" for data integrity. However, from a strategic standpoint, Cross Joins are powerful for generating permutations—such as creating a master grid of every possible product-color combination for an inventory audit.

5. Window Functions: Grouping Without the "Collapse"

The true "game-changer" for intermediate learners is the OVER() clause. Traditional GROUP BY operations "collapse" your data, rolling multiple rows into a single aggregate level. While useful for summaries, this loses the individual row detail.

Window Functions allow you to perform calculations across a result set while keeping every unique row intact. This allows you to view an individual's salary right next to the department average, or calculate a Rolling Total.

Strategic Use Case: The Rolling Total In finance and healthcare, tracking cumulative sums is vital. By adding an ORDER BY clause inside the OVER() window, you transform a static sum into a running balance.

Sample Query: Rolling Totals and Averages
-- Calculating a cumulative salary sum and a static department average SELECT first_name, gender, salary, SUM(salary) OVER(PARTITION BY gender ORDER BY employee_id) AS rolling_total, AVG(salary) OVER(PARTITION BY gender) AS avg_dept_salary FROM employees;

6. The Ranking Trio: Row Number, Rank, and Dense Rank

Sequencing data is a core requirement for leaderboards and performance tracking. SQL offers three nuances for handling ties within a window:

- Row Number: Assigns a unique, sequential integer to every row (1, 2, 3, 4). Even in the event of a tie, the numbers will not repeat.
**

- Rank: Assigns the same number to ties but skips the next position based on the count of duplicates. If two rows tie for 1st, the next is 3rd (1, 1, 3). This is "positional" ranking.
**-

- Dense Rank: Assigns the same number to ties but keeps the next number sequential (1, 1, 2).

This is "numerical" ranking.
Architectural Preference: DENSE_RANK is typically preferred for professional reporting. It ensures there are no "gaps" in your leaderboard, maintaining a clean hierarchy regardless of how many entities share the same value.7.

Conclusion: Levelling Up Your Query Game

Moving from a beginner to an advanced analyst is less about learning syntax and more about understanding the logic of relationships. Every time you approach a new dataset, you must make a strategic choice: do you need to collapse your data into an aggregate summary with a Join or Group By, or do you need to partition it with a Window Function to maintain row-level detail?

The absolute grounding rule of SQL is that it is a language of relationships. By mastering the nuances of how data stacks vertically via Unions or expands horizontally via Joins, you ensure the relational integrity of your architecture.

Cool SQL Server Features You Can Use in Real World.

Fri, 27 Feb 2026 08:17:57 +0000

If you ever had to “fix a batch” of invoices, reset workflow statuses, or reprocess failed documents, you know the pain, one mistake and you either update too much, or you update the right rows but lose the evidence of what changed.

SQL Server has a few features that make these scripts safer, easier to review, and much easier to debug later. Below is a clean demo that you can paste in SSMS and use in your daily work.

SET NOCOUNT ON;
SET XACT_ABORT ON;

BEGIN TRY
    BEGIN TRAN;

    /* 1) Demo table (drop + create) */
    IF OBJECT_ID('dbo.DocumentRepositoryDemo', 'U') IS NOT NULL
        DROP TABLE dbo.DocumentRepositoryDemo;

    CREATE TABLE dbo.DocumentRepositoryDemo
    (
        DocumentRepositoryID bigint IDENTITY(1,1) NOT NULL CONSTRAINT PK_DocumentRepositoryDemo PRIMARY KEY,
        OriginalInvoiceID    varchar(50) NOT NULL,
        WorkflowStatus       int NOT NULL,
        HasResponse          bit NOT NULL,
        IsForwardedTP        bit NOT NULL,
        ModifiedAtUtc        datetime2(3) NULL,
        ModifiedBy           sysname NULL
    );

    /* 2) Seed some fake data */
    INSERT dbo.DocumentRepositoryDemo
        (OriginalInvoiceID, WorkflowStatus, HasResponse, IsForwardedTP, ModifiedAtUtc, ModifiedBy)
    VALUES
        ('7601002721400-318242', 2, 1, 1, SYSUTCDATETIME(), SUSER_SNAME()),
        ('7601002721400-318161', 3, 1, 0, SYSUTCDATETIME(), SUSER_SNAME()),
        ('7601001392038-318127', 4, 0, 1, SYSUTCDATETIME(), SUSER_SNAME()),
        ('7601000777447-967851', 9, 0, 0, SYSUTCDATETIME(), SUSER_SNAME()),
        ('7601000777447-968159', 1, 1, 1, SYSUTCDATETIME(), SUSER_SNAME());

    /* 3) Target list (table variable) */
    DECLARE @InvoiceIds TABLE
    (
        OriginalInvoiceID varchar(50) NOT NULL PRIMARY KEY
    );

    INSERT INTO @InvoiceIds (OriginalInvoiceID)
    VALUES
        ('7601002721400-318242'),
        ('7601002721400-318161'),
        ('7601000777447-968159');

    /* 4) Change tracking (audit table variable) */
    DECLARE @Changed TABLE
    (
        DocumentRepositoryID bigint,
        OriginalInvoiceID    varchar(50),
        OldWorkflowStatus    int,
        NewWorkflowStatus    int,
        OldHasResponse       bit,
        NewHasResponse       bit,
        OldIsForwardedTP     bit,
        NewIsForwardedTP     bit,
        ChangedAtUtc         datetime2(3),
        ChangedBy            sysname
    );

    /* 5) Update + capture before/after with OUTPUT */
    UPDATE dr
        SET dr.WorkflowStatus = 5,
            dr.HasResponse = 0,
            dr.IsForwardedTP = 0,
            dr.ModifiedAtUtc = SYSUTCDATETIME(),
            dr.ModifiedBy = SUSER_SNAME()
    OUTPUT
        deleted.DocumentRepositoryID,
        deleted.OriginalInvoiceID,
        deleted.WorkflowStatus, inserted.WorkflowStatus,
        deleted.HasResponse, inserted.HasResponse,
        deleted.IsForwardedTP, inserted.IsForwardedTP,
        inserted.ModifiedAtUtc,
        inserted.ModifiedBy
    INTO @Changed
    (
        DocumentRepositoryID,
        OriginalInvoiceID,
        OldWorkflowStatus, NewWorkflowStatus,
        OldHasResponse, NewHasResponse,
        OldIsForwardedTP, NewIsForwardedTP,
        ChangedAtUtc,
        ChangedBy
    )
    FROM dbo.DocumentRepositoryDemo dr
    INNER JOIN @InvoiceIds i
        ON i.OriginalInvoiceID = dr.OriginalInvoiceID
    WHERE dr.WorkflowStatus <> 5
       OR dr.HasResponse <> 0
       OR dr.IsForwardedTP <> 0;

    /* 6) Results */
    SELECT *
    FROM @Changed
    ORDER BY OriginalInvoiceID, DocumentRepositoryID;

    SELECT COUNT(*) AS RowsUpdated
    FROM @Changed;

    COMMIT TRAN
END TRY
BEGIN CATCH
    IF XACT_STATE() <> 0
        ROLLBACK TRAN
    THROW
END CATCH

What makes this “SQL Server cool” (and why it matters)

SQL Server normally prints messages like “(1 row affected)” after every statement. That is fine in ad hoc work, but noisy in scripts that do multiple steps.

With SET NOCOUNT ON:

result sets stay clean (only the output you intentionally select),
less chatter between server and client,
stored procedures often become easier to consume from apps and jobs.

SET XACT_ABORT ON

With XACT_ABORT ON, when a runtime error happens (conversion, constraint violation, etc.), SQL Server forces the transaction into a state where it cannot silently continue. In practice, this reduces the chance you end up with a transaction left open, or partially executed logic that you forget to undo.

It pairs perfectly with TRY/CATCH because it pushes you toward “all or nothing”.

3) TRY/CATCH + XACT_STATE(), predictable rollback logic

XACT_STATE() helps because it tells you the transaction state:

1 there is an active, committable transaction
-1 the transaction is uncommittable, you must rollback
0 no active transaction

That is why this pattern is so common:

IF XACT_STATE() <> 0
    ROLLBACK TRAN;

4) Table variables as “scoped in memory tables”

This script uses two table variables:

@InvoiceIds is a safe target list

@Changed is an audit collector

Why that’s useful:

you avoid long IN (...) lists inside the update,
your target list can be validated and deduped (the PRIMARY KEY
prevents duplicates),
you keep the update set based, readable, and reviewable.

Important detail for accuracy: table variables are often called “in memory”, but they can spill to tempdb depending on size and query plan. Still, they are great for small and medium sets.

5) OUTPUT deleted, inserted, instant before/after audit

This is one of the most underrated features for operational work.

OUTPUT gives you:

deleted.* values before the update
inserted.* values after the update

So you can:

show exactly what changed,
store it into @Changed, optionally persist
it into a real audit table.

This is the difference between “I ran a script” and “I can prove exactly what the script did”.

Takeaways :

NOCOUNT keeps scripts readable and reduces noise.
XACT_ABORT makes failure behavior safer.
TRY/CATCH plus XACT_STATE() gives you consistent rollback behavior.
Table variables make targeting and audit capture clean and scoped.
OUTPUT gives you real auditing with almost no extra effort.
Avoiding unnecessary updates is a performance win and a safety win.

Corrupted‑to‑Gold

Sat, 31 Jan 2026 17:35:50 +0000

Corrupted‑to‑Gold: MDM Foundations

This build takes intentionally corrupted CSV files and transforms them into clean, trusted, analytics‑ready Delta tables. The pipeline demonstrates real‑world data engineering constraints: inconsistent delimiters, malformed dates, broken encodings, duplicate entities, and multi‑format ingestion.

The solution includes:

✔ RAW → BRONZE ingestion

Normalized delimiters

   Removed non‑ASCII noise
   Safe splitting logic
   Header detection
   Schema validation

✔ SILVER cleansing

Email, phone, country, and status normalization

   Multi‑format date parsing
   Null handling and type correction

✔ Survivorship scoring (MDM‑light)

Email validity

  Phone completeness
  Date reliability
  Status correctness
  Row‑level scoring + best‑record selection

✔ GOLD outputs

Clean, deduped, resequenced customer entities

   Delta‑optimized storage
   Ready for downstream MDM and ADF Mapping Data Flows

This project sets the foundation for the full MDM pipeline (matching, clustering, survivorship, and golden‑record creation) and the upcoming ADF MDF suite demonstrating schema drift, parameterization, joins, window functions, branching, SCD‑light, partitioning, and KPI generation.

If you want the full code, architecture, or notebooks, the repo is here:

github.com/usman19zafar/Project_corrupted-to-gold-data-journey

Is OpenAI really running ChatGPT on a single PostgreSQL instance?

Tue, 27 Jan 2026 09:05:05 +0000

The headline of OpenAI’s recent article https://openai.com/index/scaling-postgresql/ feels a bit clickbaity,
If they truly used only one database instance, ChatGPT would’ve been dead on arrival.

But the reality is far more impressive: they’re using the right tool for each layer of the persistent stack:

They push PostgreSQL to its absolute limits, with a single primary writer, yes, but backed by nearly 50 read replicas across global regions.
For write-heavy workloads, they’ve wisely migrated to sharded systems like Azure Cosmos DB.
They’ve added layers of resilience:
- connection pooling with PgBouncer,
- query rate limiting,
- caching with lock leasing,
- cascading replication (in testing),
- strict schema-change policies.

It’s not magic, it’s mature, thoughtful engineering at scale.

If you work with databases, this post is absolutely worth reading.

Mini Ingestion Framework (MIF-1)

Fri, 23 Jan 2026 11:37:17 +0000

I built the Mini Ingestion Framework (MIF) — a deterministic, SOP‑driven future engine that turns simple user requirements into a complete Spark ingestion pipeline.

MIF uses a numbered SOP library to generate:

a minimal SOP subset

ordered ingestion steps

deterministic Spark code

ASCII documentation

a validated Parquet output

No guesswork. No drift. Just reproducible ingestion.

How it works:

User provides requirements (rename, drop, metadata, schema, output).

Rules Engine evaluates them against the SOP library.

Minimal SOP subset is selected.

Spark code is generated step‑by‑step.

Output is a ready‑to‑run ingestion notebook.

Example input:

Code
rename:
circuited: circuit_id
drop:

url
metadata:
ingestion_date
output: parquet

MIF returns:

SOP steps → 7B, 7A, 7C, 9B, 9J

Spark code for each step

Final ingestion notebook

Why it matters: (Need & Utility)

Most ingestion pipelines are inconsistent and hard to audit.

MIF introduces a repeatable, governed, standards‑based ingestion
pattern for real data engineering teams.

Roadmap:

Automated CLI

Silver/Gold SOP subsets

Schema registry integration

Notebook generator

https://github.com/usman19zafar/Standardizing-Apache-Spark-

Author:
Usman Zafar — Foundational Architect & Cognitive Architect -- Rookie Data Engineer!

Case study for Stored Procedures.

Sat, 17 Jan 2026 10:07:57 +0000

Just wrapped up a real‑world debugging arc inside Azure Synapse — the kind that reminds you data engineering is built on investigation, not just pipelines.

A simple task (“load 18 months of NYC Taxi data into the Silver
layer”) turned into a full breakdown of:

schema drift (ehail_fee missing in Bronze)

CTAS behavior in distributed compute

why Synapse writes 1 file for some months and 3 for others

how two stored procedures (instructor vs. corrected) produce
different outputs

and how to validate a lake house pipeline end‑to‑end

I documented the entire journey — including the side‑by‑side stored procedure comparison, ASCII diagrams, and the debugging narrative — so others can see what real data engineering looks like when things don’t go as planned.

If you want to see the technical breakdown and the final working solution, check it out here:

https://github.com/usman19zafar/Azure-Synapse-Project-/blob/main/Data Ingestion/Stored Procedures 2,CaseStudy.md

This is the part of the craft that makes you a better engineer — understanding why systems behave the way they do, not just how to make them run.

Why learning to build your own ORM is worth it (even if you use EF Core or Dapper)

Fri, 19 Dec 2025 13:53:30 +0000

Why learning to build your own ORM is worth it (even if you use EF Core or Dapper)

Most developers meet ORMs the same way they meet electricity, you flip a switch and things work. That’s great, until something gets slow, a query behaves oddly, or you hit a transaction or concurrency edge case and you suddenly need to know what’s happening under the hood.

Building a small ORM, even a deliberately limited one, is one of the fastest ways to close the gap between “I can use EF Core or Dapper” and “I understand data access.”

To keep it grounded, I’ll anchor this around my tiny demo ORM (MicroOrm), a .NET 8 sample using SQLite, simple POCOs, a lightweight Include for eager loading, and CancellationToken propagation end to end. (GitHub)

1) You stop treating the database like a black box

When you write mapping yourself, you’re forced to answer questions full ORMs often hide:

What SQL is actually executed?
How are parameters created and bound?
How do nulls map into value types vs nullable types?
What happens when schema changes, column names drift, or types don’t match?
How do you retrieve identities safely after inserts?

This is where “I know EF” turns into “I can debug a data issue under pressure.”

2) Reflection becomes a tool, not a mystery

A tiny ORM almost always starts with reflection.

You inspect a type, find its attributes, discover the key property, build a column list, and map rows back into objects. In MicroOrm’s demo, entities like Blog and Post use familiar attributes like [Table], [Key], [ForeignKey], and [InverseProperty], which is exactly the kind of metadata you typically read via reflection. (GitHub)

The payoff is immediate because you hit two truths fast:

Reflection is powerful, it removes a ton of repetitive glue code.
Reflection is not free, so you learn to cache metadata and keep reflection out of hot paths.

Those instincts carry straight into EF Core too.

3) Generics give you type safety and a clean API surface

Generics are the other superpower of a micro ORM.

They let you write APIs that feel natural in C# and stay strongly typed:

GetById<T>(id) returns a T, not object
GetAll<T>() returns List<T>, not dictionaries
One set of operations can work across your entities without repetition

In practice, building a micro ORM becomes a “generics + reflection” exercise done right, compile-time safety on the outside, runtime discovery on the inside.

4) You finally learn ADO.NET for real (SqlConnection, SqlCommand, and friends)

EF Core and Dapper sit on top of ADO.NET. When you build your own ORM, you stop skipping the fundamentals:

Connections (SqlConnection, SqliteConnection, or DbConnection)
You learn what “open late, close early” actually looks like, why using or await using matters, and how connection pooling changes the way you should think about lifetime.
Commands (SqlCommand, DbCommand)
You get comfortable with parameterized queries (and why they’re non negotiable), command timeouts, and how provider differences show up.
Readers (DbDataReader)
You learn what streaming results means, how mapping happens row by row, and where conversions and null handling can bite you.
Transactions (DbTransaction)
You start treating transaction boundaries as a deliberate design choice, not an incidental side effect of the ORM.

MicroOrm’s demo explicitly highlights cancellation support across opening connections and executing commands, which is exactly the sort of detail you only appreciate once you’re directly using these classes. (GitHub)

5) You learn the real price of convenience (and why loading strategy matters)

MicroOrm’s Include supports both references and collections, implemented as extra queries per entity. It’s intentionally simple, and extremely educational because you can feel the tradeoff immediately: clearer code and predictable behavior, plus a very real risk of N+1 query explosions if you’re not careful. (GitHub)

After you build even a basic Include, you start spotting inefficient loading patterns everywhere, including in mature ORMs.

6) You learn portability the honest way

“Database agnostic” always has seams.

The sample calls it out plainly: switching from SQLite to SQL Server means swapping the provider and adjusting identity retrieval in SqlBuilder.Insert to use SELECT SCOPE_IDENTITY();. That single detail teaches a big lesson, database behavior differences are real and they show up fast. (GitHub)

The takeaway

Writing a tiny ORM is not about shipping a competitor to EF Core.

It’s about gaining X-ray vision for what’s underneath, reflection, generics, ADO.NET primitives like SqlConnection and SqlCommand, query cost, round trips, and transactions. Then, when you choose EF Core or Dapper, you’re not just using them, you’re using them with intent.

TextQL rewrote their codebase seven times to solve natural language querying at petabyte scale.

Fri, 10 Oct 2025 14:43:01 +0000

TextQL Built a Rosetta Stone for Enterprise Data — and It Actually Works

Most natural language query tools fail when you move beyond your laptop. TextQL built theirs to work with data that doesn't fit on a single computer.

The company spent three years teaching AI how to work with tables at enterprise scale. Not the tables you find in a Google Doc or Excel file. The ones with hundreds of thousands of tables, trillions of rows, and petabytes of data spread across different systems.

Ethan Ding, TextQL's CEO, explained their approach during the 64th IT Press Tour. "First generation AI products were designed for documents that fit on your computer," he said. "Try to put more than 10 gigabytes of data in an Excel file and your computer starts to fall apart."

The Real Problem

Enterprise data lives in data warehouses, ERP systems, CRM platforms, and ITSM tools. Each platform has its own query language, table format, and way of storing data. Moving data between these systems costs millions of dollars and takes months.

The expense isn't accidental. These platforms deliberately make it expensive to leave. They charge 10 to 20 times markup on the underlying infrastructure costs. A CFO who wants to move business logic from SAP to NetSuite faces a contract with Accenture that runs $50 million over five years — and only moves 10% of the actual business logic.

"If your landlord knew it cost you $50 million to move apartments, they could charge whatever they wanted," Ding said. "That's why these platforms have really high lock-in."

The Technical Challenge

TextQL built what Ding calls a "Rosetta Stone system." Instead of translating between every possible combination of query languages and table formats (an n-by-n complexity problem), they created an intermediary layer that makes it a one-by-n problem.

The system can connect to any database and translate queries on the fly. It doesn't require data migration or lengthy setup. You connect your database, ask a question in natural language, and the AI agent writes code, checks results, and tries different approaches when queries fail.

During the demo, Ding showed how the system handles a complex query about flight routes and passenger data. The AI wrote SQL queries, caught execution errors, rewrote the queries, and eventually produced visualizations on a satellite map. All without manual intervention.

"The AI is problem-solving the same way a human data scientist would," Ding explained. "You try something, it fails, you keep working through it."

What Makes It Different

Most natural language query tools work fine with documents and small datasets. TextQL focused on the hard stuff. They can handle petabytes of data across multiple systems with different schemas. The system reconciles when a customer appears as "Fred" in one database, "Fred Frankel" in another, and "Emails are not allowed" in a third.

The company uses Claude and GPT models but runs their own infrastructure layer. "We can spin up a really big computer on the fly," Ding said. "Historically, you had to do this work where the database lives. We can do it on the fly, which means you only pay for minutes in transit."

Their pricing model reflects this approach. They charge based on compute usage, not on tokens or seats. Customers commit to an amount and burn it down as they use the system.

The Hard Road

TextQL started in December 2022. They rewrote their entire codebase seven times. The first version didn't work. Neither did the second. Or the third.

"We lost every customer until January of this year," Ding admitted. "All the code is new as of January. We haven't lost a single customer or pilot since then."

The 15-person team is mostly senior engineers — dropouts from high-frequency trading firms, former researchers, and people with Master's and PhDs. The CTO was the youngest senior engineer ever at Facebook, working on a team led by Andrew Bosworth (the person who got the famous billion-dollar offer from Facebook).

Current State

TextQL raised $5 million and projects $7 to 7.5 million in revenue. They started serious customer conversations in June and closed their first major contracts. Sales cycles in this space typically run 12 months. TextQL shortened theirs to six.

Their target customers are CFO offices, financial services firms, and healthcare organizations. The team recently hired experienced enterprise sales executives who've sold to Fortune 500 companies for decades.

"Put us in a room next to competing software," Ding challenged. "Put us in a one-to-one comparison. I will bet on our system every time."

The company's approach works because they focused on the hardest problems first. When data is spread across multiple systems, when queries need to run against trillions of rows, when the cost of vendor lock-in reaches millions of dollars — that's where TextQL operates.

Most companies promise natural language querying. TextQL built a system that can actually deliver it at enterprise scale. That's a meaningful difference.

What actually happens when you run an SQL query?

Sun, 07 Sep 2025 11:40:41 +0000

Do you know what actually happens when you run an SQL query?

It’s not as simple as: “SQL just fetches rows from a table.”

Behind the scenes, your query goes through a step-by-step execution order and knowing this is often an interview question for beginners.

Execution Order:

FROM (pick the table)
WHERE (filter rows)
GROUP BY (make buckets)
HAVING (filter buckets)
SELECT (choose what to show)
ORDER BY (sort the results)
LIMIT (control how much to show)

Notice how WHERE comes before grouping, while HAVING comes after grouping.

Notice how ORDER BY is one of the last steps, even though we often write it after SELECT.

Common Confusions:

a) WHERE vs HAVING

WHERE filters rows before grouping.
HAVING filters groups after aggregation.

Example:

WHERE city = 'Delhi' → filter students from Delhi first.

HAVING COUNT(*) > 10 → only keep cities with more than 10 students.

b) GROUP BY vs ORDER BY

GROUP BY → bundles rows into categories (e.g., students per city).

ORDER BY → decides the sequence in which results are displayed (e.g., sort cities by student count).

c) LIMIT with ORDER BY

Important tasks like “find top 3 highest salaries” or “top 5 cheapest products.”

If you found this useful, drop a like or share it with someone preparing for SQL interviews.

Follow for more such insightful posts , I’m breaking SQL down into small, clear lessons you’ll actually remember.

SQL vs NoSQL: Choosing the Right Database Before It Chooses Your Fate

Sat, 23 Aug 2025 11:52:31 +0000

SQL or NoSQL - the debate isn’t just about databases, it’s about how your application will think, grow, and scale.

Pick the wrong one, and you might end up fighting your database instead of building your product.

What if the database you choose today becomes the reason your app slows down, fails to scale, or costs you thousands to fix later?

Let’s make sure that never happens.

Choosing between SQL and NoSQL is one of the most important decisions you’ll make when designing a data-driven application.

Both have their strengths, weaknesses, and ideal use cases and the choice can shape your system’s performance, scalability, and even your development speed.

In this guide, we’ll break down what SQL and NoSQL are, where each shines, their pros and cons, and how to decide which one’s right for you.

What is SQL?

SQL (Structured Query Language) is a standardized, domain-specific programming language used to access, manipulate, and manage data within relational database management systems (RDBMS).
Data is organized into tables with rows and columns, enforcing a strict schema. Examples of SQL operations include SELECT, INSERT, UPDATE, DELETE, and advanced queries involving JOINs.

What is NoSQL?

NoSQL (“Not Only SQL”) refers to a broad category of non-relational databases that store and retrieve data differently from traditional table-based databases.
NoSQL databases come in many types: key-value, document, wide-column, and graph. They allow for flexible schemas and are designed for high scalability, performance, and handling unstructured or semi-structured data.

Uses of SQL and NoSQL Databases

SQL

Suitable for applications needing consistent, structured data: finance systems, ERP, CRM, e-commerce, and any domain where ACID transactions and complex queries are crucial.
Commonly found in analytical workloads, data warehousing, and transactional systems.

NoSQL

Used when flexibility and scalability are necessary: big data analytics, social networks, content management, IoT, and mobile applications.
Ideal for handling rapidly evolving data models, storing large amounts of unstructured data, and supporting horizontal scaling across multiple servers.

When Should You Use SQL vs NoSQL?

SQL is best when:

Your data is structured and fits well into tables (tabular model).
You need strong data consistency (e.g., financial transactions).
Your application requires complex queries and joins.

NoSQL is preferable when:

The data is unstructured, semi-structured, or changing frequently.
Scalability and distributed architectures are important (e.g., web-scale apps, IoT, streaming data).
You need high write/read throughput, flexible schema, or must store large datasets with minimal overhead.

Popular SQL and NoSQL Databases

Pros and Cons

SQL Databases

Pros:

Strong data consistency (ACID properties).
Powerful query capabilities, including JOINs and aggregations.
Mature ecosystem, well-supported in enterprise environments.
Standardized language and strong transactional support.

Cons:

Rigid schema; structure changes are complex.
Vertical scaling (limited by hardware upgrades).
Can become resource-intensive with large volumes of data.
Less ideal for unstructured or semi-structured data.

NoSQL Databases

Pros:

Highly scalable via horizontal distribution across servers.
Flexible schema; dynamic data models.
Good performance under heavy load or with big data volumes.
Suitable for rapid development and changing requirements.

Cons:

May sacrifice strong consistency for performance (eventual consistency).
Limited support for complex queries and joins.
Diverse models (key-value, document, graph, etc.) – learning curve can
be steeper.
ACID guarantees may be absent or limited.

Now let’s dive deep into the Pros and Cons

What Are ACID Properties?

ACID is a set of four guarantees that make transactions in a database reliable:

How SQL Databases Support ACID

Traditional relational databases like MySQL, PostgreSQL, Oracle, SQL Server are designed around ACID.

How they do it:

Atomicity → Transactions can be started with BEGIN and rolled back with ROLLBACK if anything fails.
Consistency → Enforced using:
- Data types
- Constraints (PRIMARY KEY, FOREIGN KEY, CHECK)
- Triggers & rules
Isolation → Achieved using locking and isolation levels (READ COMMITTED, SERIALIZABLE, etc.).
Durability → Achieved through:
- Write-ahead logging (WAL) -changes are logged before applying.
- Data replication and backups.

Example:

BEGIN;
UPDATE accounts SET balance = balance - 500 WHERE id = 1;
UPDATE accounts SET balance = balance + 500 WHERE id = 2;
COMMIT;

If any update fails, SQL ensures the whole operation rolls back.

Why Many NoSQL Databases Don’t Fully Support ACID

Many NoSQL databases (e.g., MongoDB, Cassandra, Couchbase) prioritize scalability and availability over strict ACID — especially when distributed across many servers.

Challenges:

Distributed nature → Keeping strong ACID across multiple nodes is expensive and slow.
Instead, many follow BASE:
- Basically Available → System is always responsive.
- Soft state → Data may change over time (eventual consistency).
- Eventual consistency → All replicas will agree eventually, but not instantly.
Many NoSQL systems relax Isolation and Consistency to gain:
- Faster writes
- Easier horizontal scaling
- High availability in case of network failures

❗Important Note

Not all NoSQL is non-ACID, some have partial or optional ACID support.
- MongoDB → ACID transactions for multi-document operations (since v4.0), but with performance trade-offs.
- Cassandra → Lightweight transactions with limited isolation.
But by default, most NoSQL systems lean towards BASE for performance and scalability.

SQL databases are less ideal for unstructured or semi-structured data

SQL databases are less ideal for unstructured or semi-structured data mainly because of how they’re designed at their core.

1. SQL Databases Expect a Fixed Schema

In SQL, you must define a schema (tables, columns, data types) before inserting data.
Every row in a table must follow this schema.
Problem with unstructured data:
- Unstructured (e.g., images, videos, free text) and semi-structured (e.g., JSON, XML) data doesn’t fit neatly into rigid tables.
- If the shape of the data changes often, you’d need to alter the schema repeatedly which is expensive and disruptive.

2. Data Fits Poorly into Rows and Columns

SQL databases are row-and-column oriented.
Unstructured data often has:
- Varying fields per record.
- Optional or nested attributes.
Trying to store this in SQL means:
- Many NULL values for unused columns.
- Complex join tables to represent nested relationships.
- Reduced performance.

3. Complex Storage for Nested or Variable Data

Semi-structured data like JSON can be stored in modern SQL databases (e.g., PostgreSQL’s jsonb), but:
- It loses many of the query optimizations of relational tables.
- Indexing and searching within JSON fields is slower and more resource-intensive.
NoSQL document databases (like MongoDB) handle nested, variable fields natively, making them faster for such use cases.

4. Scaling and Flexibility Issues

Unstructured/semi-structured data grows in unpredictable ways.
SQL’s strict schema + vertical scaling approach makes it harder to adapt.
NoSQL’s flexible schema + horizontal scaling is a better fit for changing and large-scale unstructured datasets.

NoSQL databases support a flexible schema

When we say NoSQL databases support a flexible schema, we mean:

1. No Predefined Table Structure

In SQL, you must define all columns and their types before inserting data.
In NoSQL (e.g., MongoDB, Cassandra, DynamoDB), you don’t have to predefine all fields.
You can insert a record with certain fields and another record with completely different fields in the same collection/table.

2. Different Records Can Have Different Structures

Example in MongoDB (Document Database):

// Document 1
{
  "name": "Vignesh",
  "email": "*Emails are not allowed*"
}

// Document 2
{
  "name": "Raj",
  "phone": "9876543210",
  "address": {
    "city": "Chennai",
    "zip": "600001"
  }
}

Here:

Vigneshhas an emailfield but no phone.
Raj has a phone and nested addressbut no email.
Both are stored in the same collection without schema changes.

3. Easy to Evolve

You can add, remove, or rename fields anytime without altering the entire database structure.
This is especially useful when:
- Your data model changes often.
- You’re storing unstructured or semi-structured data (e.g., JSON, XML).
- You have to handle diverse data sources with different formats.

4. Schema-on-Read vs Schema-on-Write

SQL uses schema-on-write → structure is enforced when data is inserted.
NoSQL often uses schema-on-read → structure is applied when data is retrieved/processed.
This means you can store raw, irregular data and shape it later.

What is Vertical Scaling (Scale Up)?

Meaning: Increasing the capacity of a single machine (server) to handle more load.
How it’s done: Add more CPU, RAM, storage, or faster disks to the same server.
Analogy: Like buying a bigger, faster laptop instead of having multiple laptops.

SQL Databases and Vertical Scaling

Traditional SQL databases (like MySQL, PostgreSQL, Oracle) store data in a structured, relational format.
They are often designed to run on a single powerful machine for consistency and complex queries.
Scaling horizontally (across multiple servers) is harder for SQL because:
- They rely heavily on transactions (ACID compliance).
- Data is often interrelated (joins across multiple tables).
- Splitting (sharding) the data while preserving relationships is complex.
So the simpler approach has been:

Make the one machine stronger → Vertical scaling.

What is Horizontal Scaling (Scale Out)?

Meaning: Adding more machines (servers) to distribute the load.
How it’s done: Use multiple servers that share the workload, often with data spread across them.
Analogy: Instead of buying a supercomputer, you buy 10 normal computers and make them work together.

NoSQL Databases and Horizontal Scaling

NoSQL databases (like MongoDB, Cassandra, Couchbase) store data in a non-relational way (documents, key-value pairs, wide-columns, etc.).
They are built with distribution in mind from the start:
- Data can be sharded across multiple servers easily.
- Each server handles a portion of the data.
- This allows handling huge datasets and high traffic by just adding more machines.
So the common scaling method is:

Add more servers to share the work → Horizontal scaling.

Conclusion

Choosing between SQL and NoSQL isn’t about which one is better in a vacuum. It’s about which one aligns with your data model, growth plans, and application needs.

Think of it like building a house: you wouldn’t choose the same foundation for a skyscraper as you would for a cabin.

If you need structure, consistency, and reliability, SQL databases offer proven stability.
If you need flexibility, speed at scale, and evolving data models, NoSQL can give you the agility you need.

The most successful projects start with clarity, not guesswork.

Pick the right database today, and you won’t just avoid costly mistakes, you’ll set your application up to scale gracefully for years to come.

Follow me on Linkedin | GitHub | Twitter

Subscribe to my newsletter and get articles delivered straight to your inbox.

Introduction to Subqueries, an advanced technique in SQL

Thu, 14 Aug 2025 19:11:51 +0000

Advanced SQL : Mastering Subqueries for High-Performance Data Analysis

In the realm of relational database management, SQL subqueries are essential tools for building powerful, scalable, and efficient queries. Subqueries, also known as nested queries or inner queries, allow us to perform dynamic filtering, conditional logic, and data aggregation with elegance and precision. This comprehensive guide explores advanced subquery techniques, equipping you with the practical knowledge to build performant SQL code that handles complex data requirements.

What Is a Subquery in SQL?

A subquery is a SQL query embedded within another SQL query. It provides an intermediate result set that the outer query can use. Subqueries can be placed in several clauses, including:

SELECT
FROM
WHERE
HAVING

Subqueries may return scalar values, rows, or even entire result sets, depending on how they are written and where they are used.

Types of Subqueries in SQL

1. Scalar Subqueries

A scalar subquery returns a single value. These are typically used in the SELECT or WHERE clause to perform comparisons or calculations against a single derived value.

SELECT employee_name, salary
FROM employees
WHERE salary > (SELECT AVG(salary) FROM employees);

This query returns all employees whose salaries exceed the company-wide average salary.

2. Row Subqueries

Row subqueries return a single row with multiple columns and are often used in WHERE or FROM clauses.

SELECT department_name
FROM departments
WHERE (manager_id, location_id) = 
      (SELECT manager_id, location_id 
       FROM departments 
       WHERE department_id = 50);

This example filters for departments that share the same manager and location as department 50.

3. Column Subqueries

Column subqueries return multiple values in a single column, useful with operators like IN, ANY, ALL, or EXISTS.

SELECT customer_id, customer_name
FROM customers
WHERE customer_id IN 
      (SELECT customer_id FROM orders WHERE total_amount > 5000);

Here, we filter customers who have made high-value purchases.

4. Correlated Subqueries

A correlated subquery references columns from the outer query, meaning it executes once for each row in the outer query. This technique is powerful but can be less performant if not optimized.

SELECT e.employee_name, e.salary
FROM employees e
WHERE salary > 
      (SELECT AVG(salary) 
       FROM employees 
       WHERE department_id = e.department_id);

This query compares each employee’s salary to the average salary of their own department.

5. Nested Subqueries

SQL allows multiple levels of nesting. These deeply nested subqueries are useful for solving hierarchical or layered logic problems.

SELECT product_name
FROM products
WHERE product_id IN (
    SELECT product_id
    FROM order_details
    WHERE order_id IN (
        SELECT order_id
        FROM orders
        WHERE customer_id = 'C101'
    )
);

This query retrieves all products purchased by a specific customer through multiple levels of nesting.

Using Subqueries in Different Clauses

Subqueries in SELECT

Useful for creating calculated fields on a per-row basis.

SELECT employee_id,
       (SELECT COUNT(*) FROM tasks WHERE tasks.assigned_to = e.employee_id) AS task_count
FROM employees e;

Subqueries in FROM

Known as inline views, subqueries in the FROM clause allow us to define temporary result sets used as tables in joins or filters.

SELECT department_id, avg_salary
FROM (
    SELECT department_id, AVG(salary) AS avg_salary
    FROM employees
    GROUP BY department_id
) dept_stats
WHERE avg_salary > 70000;

Subqueries in WHERE

Most common use-case for filtering data conditionally.

SELECT product_id, product_name
FROM products
WHERE category_id = (
    SELECT category_id
    FROM categories
    WHERE category_name = 'Electronics'
);

Subqueries in HAVING

Allows filtering of aggregate results after GROUP BY.

SELECT manager_id, COUNT(*) AS team_size
FROM employees
GROUP BY manager_id
HAVING COUNT(*) > 
      (SELECT AVG(team_count)
       FROM (
           SELECT manager_id, COUNT(*) AS team_count
           FROM employees
           GROUP BY manager_id
       ) temp);

This finds managers leading teams larger than the average.

Performance Optimization for Subqueries

When working with subqueries, especially correlated subqueries, performance can degrade on large datasets. Here’s how we can optimize them:

Use JOINs instead of correlated subqueries when possible.
Index all columns used in subquery filters or joins.
Replace complex subqueries with Common Table Expressions (CTEs) when appropriate.
Avoid repeated subquery execution by caching results in temporary tables or views.

Advanced Use-Cases of Subqueries

Top-N Records per Group

SELECT *
FROM sales s
WHERE sales_amount = (
    SELECT MAX(sales_amount)
    FROM sales
    WHERE region = s.region
);

Retrieves the highest sale per region using a correlated subquery.

Detecting Missing Data

SELECT product_name
FROM products
WHERE product_id NOT IN (
    SELECT DISTINCT product_id
    FROM order_details
);

Finds products that have never been ordered.

Ranking Using Subqueries

SELECT employee_id, salary,
       (SELECT COUNT(*) FROM employees e2 WHERE e2.salary > e1.salary) + 1 AS rank
FROM employees e1
ORDER BY rank;

Implements a ranking system without window functions.

Conditional Aggregation

SELECT department_id,
       (SELECT COUNT(*) FROM employees e2 WHERE e2.department_id = d.department_id AND e2.gender = 'F') AS female_count
FROM departments d;

Calculates the number of female employees in each department using correlated subqueries.

Best Practices When Writing Subqueries

Always test subqueries independently to verify their result sets.
Use meaningful aliases to keep complex queries readable.
Avoid deep nesting unless absolutely necessary—refactor into CTEs.
Use EXPLAIN plans to inspect query cost and identify bottlenecks.
Be cautious with NULLs when using IN or NOT IN subqueries.

Conclusion: Subqueries as the Backbone of Advanced SQL

Subqueries form the core of advanced SQL querying techniques. They allow us to decompose complex logic, leverage relational data, and build dynamic, reusable queries. When mastered, subqueries provide a clean and structured approach to querying that simplifies development and improves data retrieval performance.

The effective use of scalar, correlated, and inline subqueries enables analysts and developers to create intelligent, high-performance SQL solutions that scale with data complexity.

Coder Legion - Recent posts tagged sql

How to Format SQL Queries in Python: Best Practices, Gotchas, and Real-World Examples

The Problem With "Good Enough" SQL Formatting

1. Never Concatenate User Input — Use Parameterized Queries

❌ The Wrong Way (SQL Injection Waiting to Happen)

✅ The Right Way: Parameterized Queries

2. Multi-Line Queries: Triple Quotes + Consistent Indentation

❌ String Concatenation Chaos

✅ Triple-Quoted SQL

3. Dynamic Queries: Building Them Safely

Scenario: A search endpoint with optional filters

4. IN Clauses: The Tricky One

❌ This Won't Work

✅ Generate Placeholders Dynamically

✅ SQLAlchemy makes this trivial

5. Use SQLAlchemy Core for Complex Queries

6. Formatting for Logging and Debugging

7. Query Constants: SQL as Module-Level Variables

8. Gotchas Quick Reference

Summary

Advanced SQL Analytics for Data Analysis.

Advanced SQL analytics for running a hospital database.

Introduction

1. SQL Window Functions

OVER, PARTITION BY, ORDER BY, and Window Frames

Ranking Window Functions

Value & Navigation Window Function

Aggregate Window Functions

2. SQL Joins

INNER JOIN

LEFT JOIN

RIGHT JOIN

FULL OUTER JOIN

CROSS JOIN

SELF JOIN

Join Summary

3. Common Table Expressions (CTEs)

Simple CTEs

Multiple Chained CTEs

Recursive CTEs

CTEs vs. Subqueries

4. SQL Subqueries

Subquery Types

Subqueries in SELECT Clauses

Using Subqueries in FROM Clauses (Derived Table)

Subqueries in WHERE Clause

Conclusion

GitHub

Instant at 10 Million Rows, Dead at a Billion: A Decision Framework for Billion-Row SQL

The query that was instant at 10 million rows and timed out at a billion

The two-axis framework

Engine class 1: Row-store OLTP — great at point lookups, brutal on full-table aggregates

Engine class 2: Columnar / OLAP — where scans and aggregates are the design point

Engine class 3: Distributed — horizontal scale, and the fan-out ceiling

The memory / spill axis — the one most write-ups skip

The spill cliff

When to re-architect instead of tune

One more trap: deep pagination

The crossover-signal checklist: change engine class, don't add another index

Cost and economics for the decision-makers

Build it yourself

What to do Monday morning

Inside the Database Stack

Part I: The Waiter (Python & Django ORM)

Part II: Too Many Trips (Query Performance)

Part III: The Pantry (Database Fundamentals)

Part IV: The Kitchen's Brain (Query Planning)

Part V: Many Diners, One Kitchen (Transactions & Concurrency)

Part VI: How to Measure Honestly

Part VII: Findings

References

Find me across the web:

Reverse-Engineering Messy Databases: My End-to-End Data Audits Across 17,000+ Relational Schema Logs

From Raw Logs to Business Strategy: How I Taught Myself Data Engineering & Analytics

Project 1: StockPulse — Multi-Regional Supply Chain Audit

The Operational Bottleneck

The Technical Implementation

The Business Impact Metrics

Project 2: FunnelOpt — HabitArcade B2C Telemetry Audit

The Operational Bottleneck