Coder Legion - Recent posts tagged dataengineering

DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap

Thu, 22 Oct 2026 08:52:00 +0000

Originally published on DEV.to: https://dev.to/firfircelik/duckdb-20-vs-polars-vs-spark-on-one-machine-150m-rows-under-a-45-gib-cap-h38

DuckDB 2.0.0 shipped on October 21, 2026 (announcement, PyPI). The next morning I ran the same single-machine benchmark I had already run on DuckDB 1.5.6: three engines, one 8 vCPU VM, Parquet in and Parquet out, a hard memory cap, and four queries that look like real data engineering work.

The thesis I started with, and that the numbers support:

DuckDB beats Spark-on-one-box on memory-bound DE work. It finished every run, at every size, 5 to 12 times faster than local PySpark at 150M rows, and it still finished the heavy queries with a 1 GB memory limit.
Polars hits a wall on window dedup. At 150M rows the ROW_NUMBER() style "latest row per customer" query never finished under the cap. Not at 4.5 GiB, not with an out-of-core budget, not at 9 GiB.
Polars is not slow. On filters, joins and a high-cardinality group by it ties DuckDB or beats it, when it fits.

Everything below is measured on that one box. Where I interpret or give an opinion, I say so.

The contenders

	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Model	In-process SQL engine	In-process DataFrame engine, lazy + streaming	JVM engine, `local[8]`
How I drove it	`COPY (SQL) TO 'out.parquet'`	`LazyFrame.sink_parquet()`	`spark.sql(...).write.parquet()`
Memory knob	`SET memory_limit = '3GB'`	default (plus one run with an out-of-core budget)	`spark.driver.memory = 3g`
Threads	8	8 (`POLARS_MAX_THREADS`)	8

Polars 2.0.0 was published on PyPI on October 6, 2026, PySpark 4.2.0 on July 14, 2026. DuckDB 2.0.0 wheels landed on PyPI on October 21, the same day as the release calendar entry.

Setup

Hardware and versions

VM: 8 vCPU Intel Xeon, 15 GiB RAM, no swap, Linux 6.12. It is a shared box, so other processes were running. Every run waited until at least 5 GiB was free and CPU busy was under 30% over a 3 second window.
Python 3.13.5, OpenJDK 21.0.12, pyarrow 26.0.0.
DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0. The "before" numbers are DuckDB 1.5.6 on the same box and data, run on October 19.

Data. A seeded synthetic star schema written with pyarrow: a sales fact table and a 50,000-row products dimension. Three sizes: 10M rows (213 MB), 50M (1.1 GB) and 150M (3.2 GB of snappy Parquet, 30 files of 5M rows). About 10 orders per customer, so 150M rows means roughly 15M customers.

Queries. The same SQL for DuckDB and Spark, the same logic as Polars expressions:

q1_filter_agg: filter on a timestamp and status, group by store and status.
q2_join_agg: fact-to-dimension join, group by category and brand.
q3_window_latest: latest order per customer with ROW_NUMBER(). The classic dedup and SCD pattern.
q4_highcard_agg: one row per customer (about 15M groups at 150M).

-- q3_window_latest
SELECT customer_id, order_id, ts, qty * price AS amount
FROM (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                 ORDER BY ts DESC, order_id DESC) AS rn
    FROM sales
) t
WHERE rn = 1

Method.

Every run is a fresh process: interpreter start, import, session, read Parquet, compute, write Parquet. So no warm caches inside the engine, and Spark pays its JVM start every time.
Engines run strictly one at a time. The engine order rotates per repetition so no engine always goes first.
5 repetitions at 10M, 3 at 50M and 150M. I report medians.
A watchdog samples the RSS of the whole process tree every 50 ms and kills the run if it goes above 4.5 GiB. That is the cap. It is deliberately tighter than the box, because "fits in a small container" is the question I care about.
Wall is the whole process. Query is read, compute and write only. The difference is setup.
Correctness: a validation script compares row counts and rounded checksums of every output. Where an engine produced output, all three engines agreed, for every query and size.

Results at 150M rows

Query	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Filter + aggregate	0.87 s (3/3)	0.96 s (3/3)	10.77 s (3/3)
Join + aggregate	1.47 s (3/3)	1.46 s (3/3)	16.10 s (3/3)
Window dedup	7.92 s (3/3)	killed (0/3)	46.29 s (2/3)
High-cardinality group by	7.91 s (3/3)	4.21 s (1/3)	39.70 s (3/3)

Median wall seconds, runs that finished under the 4.5 GiB cap in brackets.

Peak RSS (median of finished runs) tells the other half:

Query	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Filter + aggregate	0.77 GiB	0.94 GiB	1.85 GiB
Join + aggregate	0.42 GiB	1.96 GiB	1.61 GiB
Window dedup	3.01 GiB	above 4.5 (killed)	3.74 GiB, one run hit 7.51
High-cardinality group by	3.01 GiB	4.49 GiB	4.40 GiB

Three things stand out.

Spark on one box pays twice. About 5 seconds of every Spark run is setup (median 5.0 to 5.8 s, JVM plus session) before it reads a byte. Then the query itself is slower: query-only time was 7.4x DuckDB's on the filter, 7.9x on the join, 5.2x on the window dedup and 4.3x on the group by. In wall time that is 5x to 12x. And spark.driver.memory = 3g did not keep the process at 3 GiB: one window dedup run reached 7.51 GiB of process-tree RSS and was killed.

Polars is fast right up to the cap. It tied DuckDB on filter and join and was the fastest engine on the high-cardinality group by (4.21 s against 7.91 s). But only 1 of 3 runs finished: that one peaked at 4.49 GiB, and the other two crossed the cap at 4.50 and 4.55 GiB. Fast and close to the edge is a fine place for a notebook and a bad place for a scheduled job.

The window dedup is where Polars stops. At 150M rows it was killed in all 3 runs, within a few seconds of starting, at 4.52 to 4.66 GiB.

Polars and the window wall

I gave Polars two more chances on the 150M window dedup:

Polars 2.0.0, 150M window dedup	Result
Default settings, 4.5 GiB cap (Oct 22)	0/3, killed at 4.52 to 4.66 GiB
`POLARS_OOC_MEMORY_BUDGET_MB=3072`, 4.5 GiB cap (Oct 22)	0/3, killed at 4.57 to 4.66 GiB
Default settings, 9 GiB cap (Oct 19, same Polars 2.0.0)	0/3, killed at 9.04 to 9.54 GiB

The out-of-core budget variable is one I found in the Polars 2.0.0 source, not a documented knob, so treat that row as "I tried", not as guidance. The same variable did help the group by: with it, Polars finished 3/3 at 4.67 s and 4.18 to 4.30 GiB.

At smaller sizes the dedup works, but the shape is already visible:

Window dedup	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
10M rows	0.60 s, 0.65 GiB	1.83 s, 1.20 GiB	12.74 s, 2.94 GiB
50M rows	2.33 s, 2.19 GiB	10.36 s, 4.46 GiB	29.06 s, 3.77 GiB
150M rows	7.92 s, 3.01 GiB	killed	46.29 s, 3.74 GiB

At 50M Polars already used 4.46 GiB for a query DuckDB did in 2.19 GiB, and took 4.4x longer.

This is the Polars expression I used. It is the direct translation of the SQL:

import polars as pl

amount = pl.col("qty").cast(pl.Float64) * pl.col("price")
rn = pl.int_range(pl.len()).over(
    "customer_id", order_by=["ts", "order_id"], descending=True
)
(
    pl.scan_parquet("data/sales_150m/*.parquet")
    .filter(rn == 0)
    .select("customer_id", "order_id", "ts", amount.alias("amount"))
    .sink_parquet("out/polars/150m_q3_window_latest.parquet")
)

Fair caveat: I did not test other Polars formulations of the same dedup (a sort followed by unique(keep="first"), or a group_by().agg(...) with sort_by().last()). One of them may stream better. The point stands for the common case, though: the first thing a team writes is the window, and that one did not fit.

Update (Oct 22): after this went out, Victor Cid (@Vleyked) tested exactly that in engine-bench PR #1. His sort + unique(keep="first") version drops the in-memory window fallback from the plan and finished 3/3 at 150M rows under the same 4.5 GiB cap, peaking at 2.50 to 2.74 GiB, but only with POLARS_OOC_MEMORY_BUDGET_MB=1024. With default settings or a 3072 MB budget it still went over the cap. The price is speed: a 90.58 s median against 17.56 s for DuckDB on the same 4 vCPU runner, so roughly 5x slower. His output matched DuckDB row for row (14,999,347 rows). So Polars can do this dedup under the cap if you rewrite it and tune the spill budget, and with a smarter query it gets close to DuckDB (see below). His runs were on different hardware, so they are not in the tables above.

Second update: in engine-bench PR #2 he filtered each customer down to their max timestamp first and only sorted those rows to break ties. That brought the median down to 14.23 s at 150M rows with default Polars settings, peaking at 4.04 GiB under the same cap, with output matching DuckDB row for row. On his runner DuckDB's reference was a single 18.50 s run, so I'm not calling that a repeated cross-engine win. The real lesson is that the first way you write a Polars dedup can decide whether it finishes at all.

Third update (Oct 23): Ritchie Vink, the creator of Polars, pointed out that only DuckDB was told about the memory limit, so Polars did not know it had to stay under 4 GiB, and suggested setting POLARS_OOC_MEMORY_BUDGET_MB for a fairer comparison. I reran every Polars query at 150M rows on the same box under the same 4.5 GiB cap with that budget, next to a DuckDB rerun in the same session: the high-cardinality group by finished 3/3 at 6.17 s and 3.46 GiB with a 3072 MB budget, against 9.56 s for DuckDB (the box was busier than on Oct 22 and DuckDB ran slower too, so compare within this session, not with the tables above; the Oct 22 run with the same budget gave 4.67 s). The window dedup with over(order_by) still went over the cap in every run at 3072, 1024 and 512 MB, Victor's sort plus unique version fit only at 1024 MB (31.28 s, 2.29 GiB), and the max-timestamp rewrite finished 3/3 on default settings (5.61 s, 4.02 GiB). Ritchie says Polars main now spills the group by correctly and that this ships in the next release, which was not on PyPI yet when I ran this, so I will rerun when it ships. Raw results and output checks are in engine-bench PR #3 and in results/polars_ooc_budget_2026-10-23 on the polars-ooc-budget-2026-10-23 branch.

Fourth update (Oct 26): after the third update I reported the window dedup to Polars as pola-rs/polars#30270: the over(order_by) window inside filter() ignored POLARS_OOC_MEMORY_BUDGET_MB and ran in an in-memory fallback node. Ritchie Vink fixed it (#30272, #30282 and #30280), and the fixes shipped in Polars 2.1.0. I reran the 150M benchmark on the official 2.1.0 wheel on the same box, with the same harness and the same 4.5 GiB cap, next to a DuckDB rerun in the same session. With a budget set, the window dedup now finishes 3/3 at every budget I tried: 18.78 s and 3.34 GiB at 3072 MB, 20.11 s and 1.29 GiB at 1024 MB, 43.01 s and 2.15 GiB at 512 MB. On 2.0.0 it failed at all three. Memory now follows the budget closely: at 512 MB, Victor's sort plus unique version peaked at 0.69 GiB (73.85 s) and the high-cardinality group by at 0.74 GiB (9.43 s). DuckDB still ran the window dedup faster in the same session, 8.07 s, about 2.3x quicker than Polars at 3072 MB, and the max-timestamp rewrite remains the fastest Polars dedup (5.12 s on default settings). Every finished output matched DuckDB row for row (14,999,347 rows). Two caveats. Without a budget, the window dedup and sort plus unique still went over the cap. My cap is a process watchdog, not a cgroup limit, so the new cgroup-aware default in 2.1.0 had nothing to detect and is untested here; set the budget explicitly. One sort plus unique run at 3072 MB took 285.64 s for no clear reason, so I report medians of 3 runs. Raw results, plans and output checks: engine-bench PR #4 and the 2.1.0 results README.

DuckDB under pressure: memory_limit = 1 GB

The interesting DuckDB question is not "how fast with room to spare" but "what happens when the box is small". Same 150M rows, memory_limit cut to 1 GB:

150M, `memory_limit = 1GB`	Wall (median)	Peak RSS	Finished
Window dedup	14.97 s	1.12 GiB	3/3
High-cardinality group by	13.94 s	1.08 GiB	3/3
Window dedup, `preserve_insertion_order = false`	14.94 s	1.12 GiB	3/3
Group by, `preserve_insertion_order = false`	13.88 s	1.08 GiB	3/3

With a third of the memory, DuckDB spilled to disk and still beat Spark's unconstrained run by about 3x on both queries, at a quarter to a third of Spark's peak RSS. The DuckDB docs list window functions with PARTITION BY ... ORDER BY among the blocking operators it can process larger than memory (How to Tune Workloads). This run is that sentence, measured.

The setup is three lines:

import duckdb

con = duckdb.connect()
con.execute("SET memory_limit = '1GB'")
con.execute("SET threads = 8")
con.execute("SET temp_directory = '/mnt/fast-local-disk/duckdb_tmp'")  # where spills go
con.execute("""
    COPY (
        SELECT customer_id, order_id, ts, qty * price AS amount
        FROM (
            SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                         ORDER BY ts DESC, order_id DESC) AS rn
            FROM read_parquet('data/sales_150m/*.parquet')
        )
        WHERE rn = 1
    ) TO 'out/latest_per_customer.parquet' (FORMAT parquet)
""")

DuckDB 1.5.6 vs 2.0.0

Same box, same data, same scripts. 1.5.6 ran on October 19, 2.0.0 on October 22. Median wall seconds at 150M:

Setting	Query	1.5.6	2.0.0	Change
`memory_limit = 3GB`	Window dedup	9.09	7.92	-13%
`memory_limit = 3GB`	Group by	9.61	7.91	-18%
`memory_limit = 1GB`	Window dedup	17.79	14.97	-16%
`memory_limit = 1GB`	Group by	22.06	13.94	-37%
1GB, no insertion order	Window dedup	14.85	14.94	+1%
1GB, no insertion order	Group by	15.15	13.88	-8%

The useful finding is the last four rows together. On 1.5.6, SET preserve_insertion_order = false was worth 16% on the dedup and 31% on the group by under a 1 GB limit. On 2.0.0 the default already lands where the tuned setting did, and the flag changes almost nothing for these two queries. If you carry that setting in job configs as a performance trick, re-measure after upgrading. It still matters for what it was designed for: the docs recommend it when large imports or exports run out of memory, and it lets DuckDB reorder rows without an ORDER BY.

Not everything improved. The 50M window dedup was slower on 2.0.0 (1.94 s to 2.33 s), and peak RSS for the light queries went up (150M filter: 0.26 to 0.77 GiB, join: 0.23 to 0.42 GiB), while the heavy queries went down (window dedup: 3.47 to 3.01 GiB). I have not investigated why. The two runs were also three days apart on a shared box, so treat differences under about 10 to 15% as noise. The 1 GB group by drop is far outside that.

The release itself is mostly a feature release: Quack and CONNECT for client-server use, asynchronous I/O, a first-class VARIANT, triggers, a PEG parser by default (announcement). It is an LTS line supported until at least October 2027. One migration note matters for pipelines: database files written by 2.0 cannot be read by versions before 2.0. My benchmark only reads and writes Parquet, so it never touched that.

Where Spark still wins

None of this says "replace Spark". It says Spark on one box is the wrong comparison for small and medium jobs. Spark still wins when:

The data does not fit one node's disk and memory, or the deadline does not fit one node's cores. This test stops at 3.2 GB of Parquet on purpose. DuckDB spills well, but one machine is still one machine.
The platform is built around it. On Databricks and Fabric, Spark is where Unity Catalog or OneLake governance, Delta writes with concurrent writers, lineage, job orchestration and streaming already live. A DuckDB step outside that boundary is a step you have to secure, monitor and explain.
The cluster is already running. The 5 second JVM start I measured per run disappears in a warm interactive cluster or a long job. The query-only gap (4x to 8x here) does not.
The team is a Spark team. One engine everyone can debug at 02:00 is worth more than a faster engine one person understands.

My opinion as a Lead DE on Azure, Databricks and Fabric: the line is not the engine, it is the size of the job. Most jobs I see are tens of millions of rows dressed up as big data and started on a cluster. For those (file-to-file transforms before landing, validation and reconciliation checks, small dimension builds, CI tests for SQL logic) a single-node engine with a memory limit is the cheaper and more predictable choice. Anything that writes governed Delta tables stays on the platform.

Failure modes I hit, so you don't have to

Engine memory settings are not process memory. Spark's 3g driver heap still produced a 7.51 GiB process. DuckDB's whole process stayed close to its memory_limit: about 3.0 GiB peak at a 3 GB limit, 1.1 GiB at 1 GB. Size containers on measured RSS, not on the setting.
"Streaming" does not mean bounded. Polars streamed the filter, join and group by fine. The window dedup with an over(order_by=...) did not stay bounded, even with 9 GiB.
Close to the cap is a failure that hasn't happened yet. Polars' group by finished in 1 of 3 runs, peaking at 4.49 GiB against a 4.5 GiB cap. Same query, same data, same version. Schedule that and it fails some nights.
Shared hosts lie. One Spark run at 10M was killed because free memory on the whole box fell under my 1 GiB safety floor while Spark itself was at 1.6 GiB. That was the neighbours, not Spark. Gate each run on free memory and CPU, and record both.
Validate outputs, not just timings. Every engine produced the same row counts and checksums where it finished. Without that check, a fast wrong answer looks like a win.

Reproduce it

Everything is public at firfircelik/engine-bench: scripts, raw results, pinned versions and the query plans. Here are the parts that matter. The data generator is seeded (SEED = 20261019) and writes 5M-row chunks:

rng = np.random.default_rng([SEED, chunk_index])
t = pa.table({
    "order_id": np.arange(lo, lo + n, dtype=np.int64),
    "ts": pa.array(start + rng.integers(0, 270 * 24 * 3600, n), pa.timestamp("s")),
    "customer_id": rng.integers(0, rows // 10, n, dtype=np.int32),
    "product_id": rng.integers(0, 50_000, n, dtype=np.int32),
    "store_id": rng.integers(0, 500, n, dtype=np.int16),
    "qty": rng.integers(1, 10, n, dtype=np.int16),
    "price": np.round(rng.uniform(1.0, 500.0, n), 2),
    "status": rng.choice(STATUSES, n, p=[0.55, 0.20, 0.12, 0.08, 0.05]),
})
pq.write_table(t, f"sales_{rows // 1_000_000}m/part-{chunk_index:04d}.parquet",
               row_group_size=1_000_000, compression="snappy")

Spark session, the only non-default settings:

spark = (SparkSession.builder.master("local[8]")
         .config("spark.driver.memory", "3g")
         .config("spark.local.dir", TMP)
         .config("spark.ui.enabled", "false")
         .config("spark.sql.session.timeZone", "UTC")
         .getOrCreate())

And the watchdog, which is the part I would copy into any benchmark of your own. It measures the whole process tree, because JVMs and thread pools do not show up in the parent's RSS:

import psutil, subprocess, time

GIB = 1024 ** 3

def tree_rss(p):
    total = 0
    for q in [p] + p.children(recursive=True):
        try:
            total += q.memory_info().rss
        except psutil.NoSuchProcess:
            pass
    return total

proc = subprocess.Popen(cmd)
p, peak = psutil.Process(proc.pid), 0
while proc.poll() is None:
    rss = tree_rss(p)
    peak = max(peak, rss)
    if rss > 4.5 * GIB:
        for q in p.children(recursive=True) + [p]:
            q.kill()
        break
    time.sleep(0.05)

The takeaway

On one 8 vCPU machine with a 4.5 GiB cap, DuckDB 2.0.0 was the only engine that finished every run of every query at every size, and it did the heavy ones with 1 GB. Polars matched or beat it on the light queries and the group by, and could not finish the window dedup at 150M even with 9 GiB. Local Spark finished almost everything, 5x to 12x slower, and its memory setting did not bound its memory.

If your job fits on one machine, measure it on one machine before you start a cluster for it.

Sources

Announcing DuckDB 2.0.0 LTS "Cinnamon Teal", DuckDB team, October 21, 2026
DuckDB release calendar (2.0.0 LTS on 2026-10-21, 1.5.6 on 2026-09-28)
duckdb 2.0.0 on PyPI, polars 2.0.0 on PyPI, pyspark 4.2.0 on PyPI
DuckDB: How to Tune Workloads (larger-than-memory operators, preserve_insertion_order)
DuckDB: Order Preservation
All timings and memory numbers: my own runs on October 19 (DuckDB 1.5.6, and the 9 GiB Polars and Spark runs) and October 22, 2026 (DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0).

Advanced SQL Analytics for Data Analysis.

Sun, 18 Oct 2026 13:54:35 +0000

Advanced SQL analytics for running a hospital database.

Introduction

Structured Query Language is essential and common in most industry sectors. In healthcare reporting, SQL plays a critical role in analytics. PostgreSQL lets data scientists take those transaction records and pull out operational insights, so you can make decisions. In this guide we go through the main SQL techniques, using a hospital database schema; patients , doctors, departments , wards, diagnoses, procedures, and visits.

First, before anything else we need to create our database schema. Connect to PostgerSQL DBMS and run a new sql script to start.

Create a schema hospital

create schema hospital;

Check to which schema you are connected.

show search_path;

If connected to a different schema, set connection to your hospital schema.

set search_path to hospital;

Now you are set to begin creating your tables. Run the following codes to create the different tables and insert data into the newly created tables:

Doctors Table

--create doctors table

create table hospital.doctors (
doctor_id int primary key,
doctor_name varchar(50) not null,
doctor_gender varchar(20),
doctor_specialty varchar(20),
doctor_experience_years int
);

select * from hospital.doctors;

-- insert data into doctors

insert into hospital.doctors (doctor_id, doctor_name, doctor_gender, doctor_specialty, doctor_experience_years)
values 
(101, 'DR.Brian Mwangi', 'Male', 'Cardiologist', 12),
(102, 'Dr. Sarah Wanjiku', 'Female', 'General Physician', 9), 
(103, 'Dr. Peter Otieno', 'Male', 'Orthopedic Surgeon', 15),
(104, 'Dr. Faith Chebet', 'Female', 'Pediatrician', 8);

Departments Table

-- create departments table

create table hospital.department (
department_id int primary key,
department_name varchar (50) not null
);


select * from hospital.department;

-- insert data into departments

insert into hospital.department (department_id, department_name)
values 
(10, 'Cardiology'),
(20, 'General Medicine'),
(30, 'Orthopedics'),
(40, 'Pediatrics');

Departments Table

-- create diagnosis table

create table hospital.diagnosis (
diagnosis_id int primary key,
diagnosis_name varchar(50),
diagnosis_category varchar(50)
);

select * from hospital.diagnosis;

--insert data into diagnosis table

insert into hospital.diagnosis (diagnosis_id, diagnosis_name, diagnosis_category)
values 
(501,'Hypertension', 'Chronic Disease'),
(502, 'Malaria', 'Infectious Disease'),
(503, 'Fracture', 'Injury'),
(504, 'Pneumonia', 'Respiratory Disease'),
(505, 'Diabetes', 'Chronic Disease'),
(506, 'Typhoid', 'Infectious'),
(507, 'Asthma', 'Respiratory Disease'),
(508, 'Back Pian', 'Musculoskeletal'),
(509, 'Gastritis', 'Digestive Disease'),
(510, 'Heart Failure', 'Chronic Disease'),
(511, 'Flu', 'Respiratoey Disease'),
(512, 'Kidney Infection', 'Infectious Disease'),
(513, 'Anaemia', 'Blood Disorder'),
(514, 'Sprain', 'Injury'),
(515, 'Gastroenteritis', 'Digestive Disease'),
(516, 'Dislocation', 'Injury'),
(517, 'Migraine', 'Neurological Disorder'),
(518, 'Chest Pain', 'Cardiovascular Disease');

Wards Table

-- create wards table 

create table hospital.wards (
ward_id int primary key, 
ward_name varchar (50)
);

-- insert data into wards
insert into hospital.wards (ward_id, ward_name)
values 
(100, 'Cardiac Care Unit'),
(200, 'General Ward'),
(300, 'Surgical Ward'),
(400, 'Childrens Ward');

select * from hospital.wards;

Patients Table

-- create patients table 
create table hospital.patients (
patient_id serial primary key, 
patient_name varchar(50) not null,
patient_gender varchar(20),
patient_age int,
county varchar(30),
sub_county varchar(30)
);

--insert data into patients
insert into hospital.patients (patient_name, patient_gender, patient_age, county, sub_county)
values 
('Amina Otieno', 'Female', 34, 'Nairobi', 'Westlands'),
('John Kariuki', 'Male', 42,' Kiambu', 'Thika'),
('Grace Naliaka', 'Female', 28, 'Kakamega', 'Lurambi'),
('David Mutua', 'Male', 55, 'Machakos', 'Athi River'),
('Esther Achieng', 'Female', 63, 'Kisumu', 'Kisumu Central'),
('Mohamed Ali', 'Male', 19, 'Mombasa', 'Nyali'),
('Lucy Wambui', 'Female', 7, 'Nakuru', 'Naivasha'),
('Samuel Kiptoo', 'Male', 39, 'Uasin Gishu', 'Eldoret East'),
('Mary Wairumu', 'Female', 31, 'Murang''a', 'Kiharu'),
('Peter Ouma', 'Male', 46, 'Siaya', 'Bondo'),
('Brian Omondi', 'Male', 24, 'Kisumu', 'Kisumu East'),
('Catherine Musyoka', 'Female', 52, 'Kitui', 'Kitui Central'),
('Daniel Maina', 'Male', 16, 'Nyeri', 'Othaya'),
('Eunice Moraa', 'Female', 44, 'Kisii', 'Bonchari'),
('Francis Njuguna', 'Male', 60, 'Laikipia', 'Nyanyuki'),
('Hellen Jepchirchir', 'Female', 37, 'Bomet', 'Sotik'),
('Isaac Kamau', 'Male', 29, 'Nairobi', 'Kasarani'),
('Jane Muthoni', 'Female', 22, 'Meru', 'Imenti North'),
('Kevin Barasa', 'Male', 48, 'Bungoma', 'Webuye'),
('Lilian Adhiambo', 'Female', 26, 'Homa Bay', 'Rangwe'),
('Moses Cheruiyot', 'Male', 33, 'Kericho', 'Ainamoi'),
('Nadia Hussein', 'Female', 5, 'Garissa', 'Dadaab'),
('Patrick Muli', 'Male', 51, 'Makueni', 'Wote'),
('Ruth Nekesa', 'Female', 41, 'Trans Nzoia', 'Kitale'),
('Josephine Wekesa', 'Female', 36, 'Busia', 'Matayos');

select * from hospital.patients;

Procedures Table

create table hospital.procedures (
procedure_id int primary key,
procedure_name varchar (50) not null,
procedure_category varchar (20)
);

--insert data into procedures
insert into hospital.procedures (procedure_id, procedure_name, procedure_category)
values
(701, 'Blood Pressure Monitoring', 'Diagnostic'),
(702, 'Malaria Test', 'Laboratory'),
(703, 'X-ray', 'Imaging'),
(704, 'Chest X-ray', 'Imaging'),
(705, 'Blood Sugar Test', 'Laboratory'),
(706, 'Full Blood Count', 'Laboratory'),
(707, 'Nebulization', 'Treatment'),
(708, 'Physiotherapy', 'Treatment'),
(709, 'Endoscopy', 'Diagnostic'),
(710, 'ECG', 'Diagnostic'),
(711, 'Consultation', 'Consultation'),
(712, 'Joint Support Bandage', 'Treatment'),
(713, 'IV Fluids', 'Treatment'),
(714, 'Closed Reduction', 'Treatment'),
(715, 'Pain Management', 'Treatment');


select * from hospital.procedures;

Visits Table

-- create visits table (Fact Table)

create table hospital.visits (
visit_id int primary key,
visit_date date not null,
patient_id int not null references hospital.patients(patient_id),
doctor_id int not null references hospital.doctors(doctor_id),
department_id int not null references hospital.department(department_id),
ward_id int references hospital.wards(ward_id),
diagnosis_id int references hospital.diagnosis(diagnosis_id),
procedure_id int references hospital.procedures(procedure_id),
admission_type varchar(20) not null
    check (admission_type in ('Outpatient', 'Inpatient', 'Emergency')),
length_of_stay_days int default 0
    check (length_of_stay_days >= 0),
triage_level varchar(10)
    check (triage_level in ('Red', 'Orange', 'Yellow', 'Green')),
insurance_provider varchar(50),
total_bill_kes numeric(10,2) not null
    check (total_bill_kes >= 0),
payment_method varchar(20),
follow_up_required varchar(3)
    check (follow_up_required in ('Yes', 'No')),
patient_satisfaction_score numeric(3,1)
    check (patient_satisfaction_score between 0 and 5)
);

--insert data into visits table
insert into hospital.visits (visit_id, visit_date, patient_id, doctor_id, department_id, ward_id, diagnosis_id,
procedure_id, admission_type, length_of_stay_days, triage_level, insurance_provider, total_bill_kes, payment_method,
follow_up_required, patient_satisfaction_score)
values
(1001, '2026-06-01', 1, 101, 10, 100, 501, 701, 'Outpatient', 0, 'Green', 'NHIF', 3500, 'Cash', 'Yes', 4.5),
(1002, '2026-06-01', 2, 102, 20, 200, 502, 702, 'Outpatient', 0, 'Yellow', 'Britam', 2500, 'Insurance', 'No', 4.1),
(1003, '2026-06-02', 3, 103, 30, 300, 503, 703, 'Inpatient', 3, 'Red', 'AAR', 28000, 'Insurance', 'Yes', 4.3),
(1004, '2026-06-02', 4, 104, 40, 400, 504, 704, 'Inpatient', 4, 'Red', 'None', 42000, 'Cash', 'Yes', 3.9),
(1005, '2026-06-03', 5, 101, 10, 100, 505, 705, 'Outpatient', 0, 'Yellow', 'CIC', 4000, 'Mobile Money', 'Yes', 4.7),
(1006, '2026-06-03', 6, 102, 20, 200, 506, 706, 'Outpatient', 0, 'Green', 'None', 3000, 'Cash', 'No', 4.0),
(1007, '2026-06-04', 7, 104, 40, 400, 507, 707, 'Outpatient', 0, 'Yellow', 'NHIF', 5500, 'Insurance', 'Yes', 4.6),
(1008, '2026-06-04', 8, 103, 30, 300, 508, 708, 'Outpatient', 0, 'Green', 'None', 6500, 'Cash', 'Yes', 4.2),
(1009, '2026-06-05', 9, 102, 20, 200, 509, 709, 'Outpatient', 0, 'Yellow', 'Jubilee', 15000, 'Insurance', 'No', 3.8),
(1010, '2026-06-05', 10, 101, 10, 100, 510, 710, 'Inpatient', 5, 'Red', 'AAR', 76000, 'Insurance', 'Yes', 4.4),
(1011, '2026-06-06', 1, 101, 10, 100, 505, 705, 'Outpatient', 0, 'Green', 'NHIF', 4200, 'Mobile Money', 'Yes', 4.8),
(1012, '2026-06-06', 11, 102, 20, 200, 502, 702, 'Outpatient', 0, 'Green', 'None', 2300, 'Cash', 'No', 4.0),
(1013, '2026-06-07', 12, 101, 10, 100, 501, 710, 'Outpatient', 0, 'Yellow', 'Britam', 7000, 'Insurance', 'Yes', 4.5),
(1014, '2026-06-07', 13, 104, 40, 400, 511, 711, 'Outpatient', 0, 'Green', 'None', 1800, 'Cash', 'No', 4.1),
(1015, '2026-06-08', 14, 103, 30, 300, 503, 703, 'Inpatient', 2, 'Red', 'CIC', 24000, 'Mobile Money', 'Yes', 4.6),
(1016, '2026-06-08', 15, 102, 20, 200, 512, 706, 'Inpatient', 3, 'Orange', 'Jubilee', 36000, 'Insurance', 'Yes', 3.7),
(1017, '2026-06-09', 16, 101, 10, 100, 510, 710, 'Inpatient', 6, 'Red', 'AAR', 82000, 'Insurance', 'Yes', 4.2),
(1018, '2026-06-09', 17, 102, 20, 200, 509, 711, 'Outpatient', 0, 'Green', 'None', 2000, 'Cash', 'No', 4.0),
(1019, '2026-06-10', 18, 104, 40, 400, 507, 707, 'Outpatient', 0, 'Yellow', 'NHIF', 5200, 'Mobile Money', 'Yes', 4.7),
(1020, '2026-06-10', 19, 103, 30, 300, 508, 708, 'Outpatient', 0, 'Green', 'Britam', 6500, 'Insurance', 'No', 4.1),
(1021, '2026-06-11', 2, 101, 10, 100, 501, 701, 'Outpatient', 0, 'Yellow', 'Britam', 3800, 'Insurance', 'Yes', 4.3),
(1022, '2026-06-11', 20, 102, 20, 200, 513, 706, 'Outpatient', 0, 'Green', 'None', 3200, 'Mobile Money', 'No', 4.4),
(1023, '2026-06-12', 21, 103, 30, 300, 514, 712, 'Outpatient', 0, 'Yellow', 'NHIF', 7200, 'Cash', 'Yes', 4.0),
(1024, '2026-06-12', 22, 104, 40, 400, 515, 713, 'Inpatient', 2, 'Orange', 'None', 18500, 'Cash', 'Yes', 4.5),
(1025, '2026-06-13', 5, 101, 10, 100, 501, 710, 'Outpatient', 0, 'Yellow', 'CIC', 7600, 'Insurance', 'Yes', 4.6),
(1026, '2026-06-13', 23, 102, 20, 200, 506, 706, 'Inpatient', 2, 'Orange', 'Jubilee', 29000, 'Insurance', 'Yes', 3.9),
(1027, '2026-06-14', 24, 103, 30, 300, 516, 714, 'Inpatient', 1, 'Red', 'AAR', 33000, 'Insurance', 'Yes', 4.2),
(1028, '2026-06-14', 7, 104, 40, 400, 511, 711, 'Outpatient', 0, 'Green', 'NHIF', 1900, 'Mobile Money', 'No', 4.8),
(1029, '2026-06-15', 25, 102, 20, 200, 517, 715, 'Outpatient', 0, 'Yellow', 'Britam', 4800, 'Cash', 'Yes', 4.1),
(1030, '2026-06-15', 10, 101, 10, 100, 518, 710, 'Emergency', 1, 'Red', 'AAR', 22000, 'Insurance', 'Yes', 4.4);

select * from hospital.visits;

After creating our database schema, we will begin modelling the data using different advanced SQL functions.

1. SQL Window Functions

Window functions do computations on related rows in a table while maintaining the individuality of each rows.

OVER, PARTITION BY, ORDER BY, and Window Frames

The OVER clause describes the rows for the window function operation.

PARTITION BY defines different partitions for each result set.
ORDER BY defines row order in each partition.
Window frames define different limits such asROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW for cumulative sums and ROWS BETWEEN 2 PRECEDING AND CURRENT ROW for moving averages.

Ranking Window Functions

Ranking functions provide numeric ordering according to specified criteria.

ROW_NUMBER provides a distinct numeric integer value to each row within a partition.
RANK provides identical rank values to ties, but skips other numbers.
DENSE_RANK provides identical rank values to ties, but does not skip numbers.
NTILE divides rows in partitions into buckets of equal size.

Question: How can hospital management rank inpatient visits by total bill within each admission type?

select 
    v.visit_id,
    p.patient_name,
    v.admission_type,
    v.total_bill_kes,
    row_number() over (partition by v.admission_type order by v.total_bill_kes desc) as row_num,
    rank() over (partition by v.admission_type order by v.total_bill_kes desc) as rnk,
    dense_rank() over (partition by v.admission_type order by v.total_bill_kes desc) as dense_rnk,
    ntile(2) over (partition by v.admission_type order by v.total_bill_kes desc) as bill_tier
from hospital.visits v
join hospital.patients p on v.patient_id = p.patient_id
where v.admission_type = 'Inpatient'
limit 5;

Expected Results

Explanation

ROW_NUMBER gives values from 1 to 5.
The values obtained by RANK and DENSE_RANK will be same as there are no duplicate bill values.
NTILE assigns the best bills to Tier 1.

Navigation functions retrieve values based on relative rows.

LAG retrieves values from previous row.
LEAD retrieves values from next row.
FIRST_VALUE gives the first value of the expression in the frame.
LAST_VALUE gives the last value of the expression in the frame bounds like ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING.

Question: How can we monitor patient billing process through multiple hospital visits?

select 
    v.visit_id,
    v.patient_id,
    v.visit_date,
    v.total_bill_kes,
    lag(v.total_bill_kes, 1, 0.00) over (partition by v.patient_id order by v.visit_date) as prev_bill,
    lead(v.total_bill_kes, 1, 0.00) over (partition by v.patient_id order by v.visit_date) as next_bill,
    first_value(v.total_bill_kes) over (partition by v.patient_id order by v.visit_date) as initial_bill,
    last_value(v.total_bill_kes) over (
        partition by v.patient_id 
        order by v.visit_date 
        rows between unbounded preceding and unbounded following
    ) as final_bill
from visits v
where v.patient_id in (1, 2)
order by v.patient_id, v.visit_date;

Expected Results

Explanation

LAG retrieves previous visit invoices and LEAD retrieves future visit invoices.
FIRST_VALUE and LAST_VALUE store the first and last visit invoices respectively.

Aggregate Window Functions

Aggregate functions SUM, AVG, COUNT, MIN, and MAX calculate moving sums and moving averages when combined with OVER.

Question: In what way do we determine cumulative income, rolling average billings, categories, and billings for departments on each visit?

select 
    v.visit_id,
    v.visit_date,
    v.total_bill_kes,
    sum(v.total_bill_kes) over (order by v.visit_date, v.visit_id) as cumulative_revenue,
    avg(v.total_bill_kes) over (
        order by v.visit_date 
        rows between 2 preceding and current row
    ) as rolling_avg_bill,
    count(*) over (partition by v.admission_type) as total_admission_type_count,
    min(v.total_bill_kes) over (partition by v.admission_type) as min_admission_bill,
    max(v.total_bill_kes) over (partition by v.admission_type) as max_admission_bill
from hospital.visits v
order by v.visit_date, v.visit_id
limit 5;

Expected Results

Explanation

SUM forms a running total.
AVG calculates a 3 row moving average.
COUNT, MIN, and MAX present group statistics without collapsing rows.

2. SQL Joins

Joins combine columns from multiple tables using related key columns.

INNER JOIN

INNER JOIN returns rows with matching key values in both tables.

Question: How do we generate a visit report combining patient names, attending doctors, and clinical departments?

select 
    v.visit_id,
    p.patient_name,
    d.doctor_name,
    dept.department_name,
    v.total_bill_kes
from hospital.visits v
inner join hospital.patients p on v.patient_id = p.patient_id
inner join hospital.doctors d on v.doctor_id = d.doctor_id
inner join hospital.department dept on v.department_id = dept.department_id
limit 5;

Expected Results

Explanation

INNER JOIN Matches keys across visits, patients, doctors, and department tables.

LEFT JOIN

The LEFT JOIN produces all the rows from the left table and the matched rows from the right table. Fields in the right table without a match produce NULL.

Question: How can we study the number of visits and expenses of patients without losing those that have no visits at all?

select 
    p.patient_id,
    p.patient_name,
    count(v.visit_id) as total_visits,
    coalesce(sum(v.total_bill_kes), 0.00) as total_spent_kes
from hospital.patients p
left join hospital.visits v on p.patient_id = v.patient_id
group by p.patient_id, p.patient_name
order by total_visits desc, p.patient_id
limit 5;

Expected Results

Explanation

All patient records are retained using LEFT JOIN.
Patients having no visits will have COUNT as 0 while COALESCE will convert NULL sums to 0.00.

RIGHT JOIN

RIGHT JOIN retrieves all records from right table and matching records from left table.

Question: How can hospital administration ensure that all physical wards are reflected in admission records?

select 
    w.ward_id,
    w.ward_name,
    v.visit_id,
    v.total_bill_kes
from hospital.visits v
right join hospital.wards w on v.ward_id = w.ward_id
where v.visit_id is null or v.visit_id <= 1003;

Expected Results

Explanation

All wards are kept in RIGHT JOIN.
The Childrens Ward will be with NULL visits if no match occurs.

FULL OUTER JOIN

FULL OUTER JOIN will display all records if there is a match in either left or right record.

Question: How do we perform auditing of doctor and department records for unassigned doctors or unmapped departments?

select 
    d.doctor_name,
    dept.department_name
from hospital.doctors d
full outer join hospital.department dept on d.doctor_id = dept.department_id;

Expected Results

Explanation

Null values will be returned in the rows where there is no relationship between doctor and department IDs.

CROSS JOIN

CROSS JOIN results in a Cartesian product where each row in the first table is paired up with each row of the second table.

Question: How can hospital managers get a complete list of all pairings of physicians with hospital wards for coverage planing?

select 
    d.doctor_name,
    w.ward_name
from hospital.doctors d
cross join hospital.wards w
order by d.doctor_name, w.ward_name
limit 5;

Expected Results

Explanation

Joining 4 doctors with 4 wards results in 16 combination rows.

SELF JOIN

SELF JOIN involves joining a table to itself for comparing rows in the same table.

Question: In what way can readmission of patients be identified by matching their first visit to their next visit?

select 
    v1.patient_id,
    v1.visit_id as original_visit,
    v1.visit_date as original_date,
    v2.visit_id as return_visit,
    v2.visit_date as return_date
from hospital.visits v1
inner join hospital.visits v2 on v1.patient_id = v2.patient_id and v1.visit_id < v2.visit_id
order by v1.patient_id
limit 5;

Expected Results

Explanation

Linking visits on patient ID where v1.visit_id is less than v2.visit_id matches distinct visits per patient.

Join Summary

INNER JOIN matches rows in both tables.
LEFT JOIN keeps all left table rows.
RIGHT JOIN keeps all right table rows.
FULL OUTER JOIN keeps all rows from both tables.
CROSS JOIN produces Cartesian pairs.
SELF JOIN compares rows within one table.

3. Common Table Expressions (CTEs)

A CTE is a named temporary table created within a WITH clause.

Simple CTEs

Question: What approach should be used to identify costly inpatient treatments and connect them with patients and their physicians?

with high_cost_visits as (
    select 
        visit_id,
        patient_id,
        doctor_id,
        total_bill_kes,
        admission_type
    from hospital.visits
    where total_bill_kes >= 35000.00
)
select 
    h.visit_id,
    p.patient_name,
    d.doctor_name,
    h.total_bill_kes
from high_cost_visits h
join hospital.patients p on h.patient_id = p.patient_id
join hospital.doctors d on h.doctor_id = d.doctor_id;

Expected Results

Explanation

CTE will filter out high cost visits first to keep the subsequent join clean.

Multiple Chained CTEs

Question: How do we find out department revenues, overall average, and top performing departments?

with dept_revenue as (
    select 
        department_id,
        sum(total_bill_kes) as total_revenue,
        count(visit_id) as total_visits
    from hospital.visits
    group by department_id
),
dept_benchmark as (
    select 
        avg(total_revenue) as benchmark_avg_revenue
    from dept_revenue
)
select 
    d.department_name,
    dr.total_revenue,
    dr.total_visits,
    round(db.benchmark_avg_revenue, 2) as benchmark_avg
from dept_revenue dr
join hospital.department d on dr.department_id = d.department_id
cross join dept_benchmark db
where dr.total_revenue > db.benchmark_avg_revenue;

Expected Results

Explanation

dept_revenue calculates the total revenue for each department, dept_benchmark calculates the average and the last SELECT statement filters departments whose revenues are greater than the average.

Recursive CTEs

Recursive CTEs create a sequence by means of an anchor query and recursive query linked by UNION ALL operator.

Question: How can the hospital management create a biweekly follow up schedule?

with recursive follow_up_schedule as (
    select 
        1 as cycle_number,
        cast('2026/07/01' as date) as scheduled_date
    union all
    select 
        cycle_number + 1,
        cast(scheduled_date + interval '14 days' as date)
    from follow_up_schedule
    where cycle_number < 4
)
select * from follow_up_schedule;

Expected Results

Explanation

Cycle number 1 starts from July 1 2026 as the anchor date, and the recursive query keeps adding 14 days till cycle number is 4.

CTEs vs. Subqueries

CTEs provide well-defined top-level modules, which can be used again and again, while subqueries within the WHERE or FROM clause tend to complicate code and cannot be executed recursively.

4. SQL Subqueries

A subquery is a query nested inside another SQL query.

Subquery Types

Scalar Subquery fetches one row and one column.
Non-correlatedSubquery executes only once and independently.
Correlated Subquery uses columns of outer query and executes for every row of outer query.
Nested Subquery uses subqueries in subqueries.

Subqueries in SELECT Clauses

Question: How can individual patient bills be shown along with the total hospital average bill?

select 
    visit_id,
    total_bill_kes,
    (select round(avg(total_bill_kes), 2) from hospital.visits) as overall_avg_bill
from hospital.visits
where visit_id between 1001 and 1005;

Expected Results

Explanation

Scalar subquery evaluates to average 16266.67 for all rows.

Using Subqueries in FROM Clauses (Derived Table)

Question: How do we query the pre-aggregated metrics of patients using an inline derived table?

select 
    patient_summary.patient_id,
    patient_summary.visit_count,
    patient_summary.total_spent
from (
    select 
        patient_id,
        count(visit_id) as visit_count,
        sum(total_bill_kes) as total_spent
    from hospital.visits
    group by patient_id
) as patient_summary
where patient_summary.visit_count > 1;

Expected Results

Explanation

The FROM subquery groups the number of visits per patient into table alias patient_summary, subject to WHERE.

Subqueries in WHERE Clause

Question: What is the correlated subquery to find visits where the total bill is greater than the average bill of that particular department?

select 
    v1.visit_id,
    v1.patient_id,
    v1.department_id,
    v1.total_bill_kes
from hospital.visits v1
where v1.total_bill_kes > (
    select avg(v2.total_bill_kes)
    from hospital.visits v2
    where v2.department_id = v1.department_id
)
limit 5;

Expected Results

Explanation

The inner query calculates the average billing amount for each department based on outer query department IDs.

Question: Which physicians have admitted any patients as inpatients?

select 
    d.doctor_id,
    d.doctor_name,
    d.doctor_specialty
from hospital.doctors d
where exists (
    select 1 
    from hospital.visits v 
    where v.doctor_id = d.doctor_id 
      and v.admission_type = 'inpatient'
);

Expected Results

Explanation

EXISTS returns true when there is a row for a visit that matches.

Question: How can we find visits where the billing value is greater than all the outpatient visit bills?

select 
    visit_id,
    patient_id,
    total_bill_kes,
    admission_type
from hospital.visits
where total_bill_kes > all (
    select total_bill_kes
    from hospital.visits
    where admission_type = 'outpatient'
)
limit 5;

Expected Results

Explanation

The internal query locates outpatient charges up to 15000.00. ALL filters visits more than 15000.00.

Conclusion

Understanding the concepts of window functions, joins, common table expressions, and subqueries helps in creating strong PostgreSQL queries that enable healthcare workers to review their operations and enhance financial efficiency. Such information enables healthcare providers to offer quality pateint care which meets patient outcomes.

GitHub

For the completed project visit my GitHub project Hospital Database

Python 3.15 Is Not a Speed Release for Data Teams. It Is an Operations Release.

Mon, 12 Oct 2026 07:13:52 +0000

Originally published on DEV.to: https://dev.to/firfircelik/python-315-is-not-a-speed-release-for-data-teams-it-is-an-operations-release-5fbo

Python 3.15.0 shipped on October 9, 2026 (release post). The headline number most people will repeat is in the JIT section of What's New: a "7-8% geometric mean performance improvement" over the standard interpreter on x86-64 Linux, and 11-12% on AArch64 macOS.

For a data team, the tempting reading is "upgrade the job images, pipelines get faster." I installed 3.15.0 on a VM, ran a small pandas/pyarrow workload with the JIT off and on, and then spent most of my time on three features that do not touch throughput at all. Those three are the ones that change how you run Python next week.

The thesis

For data teams, Python 3.15 is an operations release, not a speed release. The features that matter are about startup (PEP 810 lazy imports), observability (the Tachyon sampling profiler and frame pointers on by default) and supply chain (PEP 829 .start files replacing import lines in .pth files).

Each of them also has a trap that the release notes do not spell out. I hit all of them on the VM, and they are documented below with the commands to reproduce them.

The antithesis, at full strength

The best counter-case is not silly. It goes like this:

The JIT is real now. What's New lists a new tracing frontend, basic register allocation, more constant propagation and reference count elimination. The pyperformance range is wide, "from roughly 15% slowdown to over 100% speedup", but the geometric mean is positive on both platforms.
Data engineers write more Python-level loops than they admit. Row-wise apply, per-record parsing, UDFs, validation code. That is exactly the code a JIT can see.
Free-threading got a stable ABI. PEP 803 adds abi3t: extensions that target the Stable ABI can now be compiled so they are compatible with free-threaded builds. That is the long road to real multi-core Python for CPU-bound pipeline code.

So the antithesis is: the startup and tooling features are nice, but the throughput work is what will show up on the cloud bill.

Here is what that looked like on the VM.

Setup

Everything below ran on one shared VM: 8 vCPU Intel Xeon, 15 GiB RAM, Debian 13, kernel 6.12. Python 3.15.0 is the uv-managed build (Clang 22, tail-calling interpreter, JIT compiled in but off by default, turned on with PYTHON_JIT=1). Libraries: numpy 2.5.4, pandas 3.0.6, pyarrow 26.0.0, all installed from cp315 wheels three days after the release. The VM is shared with other workloads, so I report medians of repeated fresh processes and repeat whole rounds.

One thing I did not expect: the VM had uv 0.12.24, and uv python install 3.15 installed 3.15.0rc3, not the final release. uv 0.13.0 installed 3.15.0. If your images resolve "3.15" through an older tool, check python -VV before you trust anything. That matters again in the profiler section.

JIT off vs on

Three workloads, timed in-process, 7 fresh processes per setting, two rounds:

Workload	JIT off (median)	JIT on (median)	Change
Pure-Python parsing loop, 1M rows	337.6 / 332.4 ms	309.7 / 304.6 ms	about 8% faster
pandas row-wise `apply`, 100k rows	1176.9 / 1183.6 ms	1126.8 / 1144.3 ms	about 3 to 4% faster
pandas vectorized `groupby().sum()`, 2M rows	63.5 / 76.3 ms	63.3 / 71.4 ms	within noise (single runs ranged 60 to 103 ms)

jit_bench.py (the three workloads)

"""Time three data-team-shaped workloads in-process. Run with PYTHON_JIT=0 or 1."""
import sys
import time

import numpy as np
import pandas as pd


def pure_python_parse(n=1_000_000):
    total = 0
    for i in range(n):
        line = f"{i},{i % 97},{i * 3}"
        a, b, c = line.split(",")
        if int(b) > 50:
            total += int(c)
    return total


def vectorized_groupby(n=2_000_000):
    rng = np.random.default_rng(42)
    df = pd.DataFrame({"k": rng.integers(0, 50_000, n), "v": rng.random(n)})
    return df.groupby("k")["v"].sum().iloc[0]


def rowwise_apply(n=100_000):
    df = pd.DataFrame({"email": [f" User{i}+*Emails are not allowed* " for i in range(n)]})
    return df.apply(lambda r: r["email"].strip().lower().replace("+spam", ""), axis=1).iloc[0]


WORKLOADS = {"pure_python_parse": pure_python_parse,
             "vectorized_groupby": vectorized_groupby,
             "rowwise_apply": rowwise_apply}

fn = WORKLOADS[sys.argv[1]]
fn(10_000)  # warm-up
t0 = time.perf_counter()
fn()
print(f"{(time.perf_counter() - t0) * 1000:.1f}")

Run it as PYTHON_JIT=0 python jit_bench.py pure_python_parse and PYTHON_JIT=1 ..., several times each.

What the numbers support (interpretation): the JIT does what the release notes say for Python-level code. The pure-Python loop landed right in the 7-8% band. But a pipeline whose hot path is already in numpy, pandas or Arrow does not get faster, because the JIT never sees that code. And the row-wise apply is the honest middle: 3 to 4% is real, but the fix for a 1.2 second apply is to vectorize it, not to wait for a JIT.

The free-threading ABI is a packaging milestone. What's New says moving an extension to abi3t "usually requires some non-trivial changes to the source code", and it pays off only when the extensions you depend on ship those wheels and you run a free-threaded build (cp315t). That is worth tracking, not something you collect by bumping a base image.

So the antithesis is partly right: there is a free win for Python-level code. It is small, it is opt-in, and it is experimental. The three features below change your runbooks this week.

1. Lazy imports fix startup, not throughput

The wrong assumption: "lazy imports make Python faster."

The mechanics: lazy import pandas as pd binds pd to a lightweight proxy. The module is loaded the first time the name is used, a step PEP 810 calls reification. If loading fails, the error is raised at first use, and the traceback shows both places. Lazy imports are only allowed at module scope: inside a function, a class body or a try/except they are a SyntaxError. I confirmed the try case on 3.15.0, which rules out the classic try: import orjson as json fallback.

I wrote a toy pipeline CLI, pipectl, with the imports most data tools have at the top: numpy, pandas, pyarrow, requests. It has a version command, a check command that validates a JSON config, and a run command that aggregates 2M rows and writes Parquet. The lazy variant changes only the import lines:

lazy import json
import argparse
import sys

lazy import numpy as np
lazy import pandas as pd
lazy import pyarrow as pa
lazy import pyarrow.parquet as pq
lazy import requests

Each command was run 30 times in a fresh process after warm-up, in three clean rounds:

Command	Eager (median per round)	Lazy (median per round)
`python -c pass` (floor)	9.3 ms
`pipectl version`	351.4 / 381.2 / 344.4 ms	20.0 / 19.6 / 19.9 ms
`pipectl check cfg.json`	352.1 / 342.7 / 352.1 ms	21.2 / 21.1 / 20.8 ms
`pipectl run` (2M rows)	453.2 / 441.0 / 450.8 ms	400.5 / 392.8 / 400.5 ms

-X importtime shows where the eager 350 ms goes: pandas about 180 ms cumulative, numpy about 40 ms, requests about 35 ms (medians of three runs, and importtime adds its own overhead).

The version and check rows are the point of the feature: about 94% less wall time, because those commands never touch pandas. That is more than the "50-70% in practice" that PEP 810 cites for command-line tools, and it should be: my CLI is the extreme case where the cheap commands use none of the heavy imports.

The run row is the one to read twice. Lazy saved about 50 ms, so did it make the work faster? No. I ran a third variant, eager but with the import requests line deleted:

`pipectl run`, 20 runs each	Median
eager	454.9 ms
eager without `import requests`	410.4 ms
lazy	413.3 ms

The whole gain is the import that the run path never uses. Lazy imports remove the imports you don't use. They don't make the ones you do use any cheaper. If your job imports pandas and uses pandas, it pays the full 180 ms either way.

Where this pays off for data teams (interpretation): CLIs, --help in CI, pre-commit hooks, tests that import a package to run one function, notebook kernels, and anything that gets parsed often. The Airflow docs already warn that "import statements also count as top-level code" in Dag files and that expensive imports there hurt the scheduler (Airflow best practices). The lazy keyword looks built for that pattern. I did not test it inside Airflow, and you need an Airflow version that supports 3.15 first.

Trap 1: the global switch broke pandas

3.15 also has an interpreter-wide mode: -X lazy_imports=all or PYTHON_LAZY_IMPORTS=all makes every module-level import in every module potentially lazy, third-party code included. It is tempting to flip it in a job image and collect the startup win without touching code. On 3.15.0 with pandas 3.0.6 and numpy 2.5.4, that failed 5 out of 5 times:

$ PYTHON_LAZY_IMPORTS=all python -c "import pandas as pd; pd.DataFrame({'a': [1]})"
  File ".../pandas/_typing.py", line 401, in <module>
    TakeIndexer: TypeAlias = Sequence[int] | Sequence[np.integer] | npt.NDArray[np.integer]
                                                                    ~~~~~~~~~~~^^^^^^^^^^^^
TypeError: 'lazy_import' object is not subscriptable

The minimal trigger is npt.NDArray[np.integer] with numpy.typing imported under the global mode. numpy alone and requests alone worked. A tiny pure-Python package with the same re-export chain worked too, so this looks specific to how numpy.typing (which defines a module-level __getattr__) interacts with global laziness. I did not chase it further. The lesson does not depend on the root cause: the global mode changes the import semantics of code you did not write, and the lazy keyword in your own entry point does not.

Trap 2: side-effect imports silently stop happening

Plenty of data code imports a module only for what it does at import time: registering a pandas accessor, a SQLAlchemy dialect, a plugin. Linters flag these as unused, and they are the first thing someone will mark lazy:

lazy import pandas as pd
lazy import dq_accessors  # noqa: F401  (registers df.dq at import time)

df = pd.DataFrame({"a": [1, None, 3], "b": [None, None, 1]})
print(df.dq.null_ratio())

AttributeError: 'DataFrame' object has no attribute 'dq'. Did you mean '.eq' instead of '.dq'?

The name dq_accessors is never used, so it is never reified, and the registration never runs. The eager version prints {'a': 0.333, 'b': 0.667}. PEP 810 says this directly: side effects are "deferred until first use", and it recommends explicit initialization functions instead of import-time registration.

Trap 3: the PEP's kill switch does not exist in 3.15.0

PEP 810 describes a third mode, "none", to force every import eager. The shipped 3.15.0 accepts only normal and all, which matches the sys docs. Copying the PEP is not harmless:

$ PYTHON_LAZY_IMPORTS=none python -c pass
Fatal Python error: config_init_lazy_imports: PYTHON_LAZY_IMPORTS: invalid value; expected 'all' or 'normal'

Exit code 1, before any of your code runs. sys.set_lazy_imports("none") raises ValueError. The PEP page itself now says it is "a historical document". If you want an emergency "everything eager" switch, a filter that returns False does it (verified: json loads at the lazy import line):

import sys

sys.set_lazy_imports_filter(lambda importer, name, fromlist: False)

Runbook: lazy imports

Measure first: python -X importtime -m yourtool --help 2> imports.txt, then sort by the cumulative column.
Mark only the heavy imports in your own entry point lazy. Leave third-party internals alone.
Keep side-effect imports eager, or replace them with an explicit register() call.
Do not set PYTHON_LAZY_IMPORTS=all in a job image unless the full test suite passes under it.
If you must support 3.14 too, use __lazy_modules__ = ["pandas", ...] with normal import statements. 3.15 treats those imports as lazy, and older versions just see a module variable and import eagerly (checked on 3.15.0 and 3.13.5 with json).

2. Tachyon: attach to the stuck job by PID, and read the error rate

The wrong assumption: "a sampling profiler is free and always right."

The mechanics: profiling.sampling (PEP 799, named Tachyon) reads the target process's memory from outside. It can run a script, attach to a PID for a time window, or dump a single traceback-style snapshot of a live process. Output formats include pstats tables, collapsed stacks, flame graphs, Firefox Profiler and a live TUI. The docs list the constraints that matter in production:

Same Python minor version for profiler and target. If either side is a pre-release, both must be the exact same version.
Permissions. On Linux, root, CAP_SYS_PTRACE, or a Yama ptrace_scope of 0. The docs note the default ptrace_scope of 1 restricts ptrace to parent processes.
Non-blocking mode (the default) never stops the target, but "can occasionally produce incomplete or inconsistent stack traces". --blocking pauses the target for each sample.

I wrote the classic slow nightly job: a pandas row-wise apply with a per-row Python function, looping forever, and pretended I did not know why it was slow.

slow_job.py

"""A 'stuck' nightly job: a row-wise pandas apply that nobody profiled."""
import time

import numpy as np
import pandas as pd


def normalize_email(row):
    return row["email"].strip().lower().replace("+spam", "")


def enrich(df):
    df["email_norm"] = df.apply(normalize_email, axis=1)
    return df


def main():
    rng = np.random.default_rng(0)
    n = 200_000
    df = pd.DataFrame({
        "id": np.arange(n),
        "email": [f" User{i}+*Emails are not allowed* " for i in rng.integers(0, 10**6, n)],
    })
    while True:
        enrich(df.copy())
        time.sleep(0.01)


if __name__ == "__main__":
    main()

The first thing you want during an incident is "what is it doing right now":

$ python -m profiling.sampling dump --blocking 440208
Stack dump for PID 440208, thread 440208 (main thread, has GIL; most recent call last):
  File "slow_job.py", line 31, in <module>
    main()
  File "slow_job.py", line 26, in main
    enrich(df.copy())
  File "slow_job.py", line 14, in enrich
    df["email_norm"] = df.apply(normalize_email, axis=1)
  ...
  File "slow_job.py", line 10, in normalize_email
    return row["email"].strip().lower().replace("+spam", "")
  File ".venv/lib/python3.15/site-packages/pandas/core/series.py", line 959, in Series.__getitem__
    return self._get_value(key)

No restart, no code change, and the culprit is on screen. On this kernel there is no Yama module (/proc/sys/kernel/yama/ptrace_scope does not exist), so attaching as the same unprivileged user just worked. On a host with the default ptrace_scope of 1, plan for root or CAP_SYS_PTRACE before the incident, not during it.

Then the part I did not expect. Without --blocking, dump failed 8 times out of 20 with RuntimeError: Failed to parse initial frame in chain. With --blocking it succeeded 20 out of 20. For attach, the profiler prints an Error rate line, the percentage of samples it could not read. I measured it on a fixed workload (median of six apply rounds per run, 1 kHz default rate, three repetitions):

Mode	Target round time	Error rate
No profiler	1170 / 1150 / 1186 ms
`attach` (non-blocking, default)	1179 / 1180 / 1150 ms	67.8% / 62.0% / 63.7%
`attach --blocking`	1606 / 1760 / 1657 ms	1.5% / 1.6% / 1.3%

Both halves of the docs are true at once. Non-blocking mode had no overhead I could measure on the target. And on this VM, 62 to 68% of its samples were unreadable in this test (46 to 50% in two earlier 5-second runs against slow_job.py, 38% against a pure-Python recursive function). Blocking mode was clean, and it made the job 37 to 53% slower while attached.

Interpretation: treat a non-blocking profile like a survey with a response rate. If 60% of samples failed, the remaining 40% may still point at the right function (here the row-wise apply dominated either way), but check the error rate before you trust percentages. Use dump --blocking for a one-shot snapshot, where a single pause costs nothing, and keep --blocking profiles short on anything user-facing.

And the version rule bit me right away. A job started on the 3.15.0rc3 that the older uv had installed could not be inspected from 3.15.0:

Can't attach to a pre-release Python interpreter from a process running a different Python version

From rc3 to rc3 it worked. Pin the profiler to the job image's exact interpreter, which in practice means running it from inside that image.

The quieter observability change is PEP 831: CPython is now built with frame pointers by default. What's New says this makes native stack unwinding "faster and more reliable" for system profilers, debuggers, crash analysis tools and eBPF-based tools. The uv build I used has -fno-omit-frame-pointer in its CFLAGS. The catch, in the release notes' words: "A single native component built without frame pointers can break stack unwinding for the whole Python process." Your own Rust or C extensions need the same flags if you want mixed native and Python profiles.

Runbook: profiling a stuck job

python -m profiling.sampling dump --blocking PID first. One snapshot, one short pause.
Then python -m profiling.sampling attach -d 30 --flamegraph -o job.html PID, and read Error rate before the flame graph.
Run the profiler with the job's exact interpreter. A pre-release on either side means exact versions only.
Decide now who is allowed CAP_SYS_PTRACE on worker hosts, and where.

3. `.pth` import lines become `.start` files: auditable, not safe

The wrong assumption: "a package can only run code when I import it."

The mechanics: at startup the site module reads every .pth file in site-packages. Lines are paths to add to sys.path, except lines that start with import, which are executed with exec(). Every Python process in that environment runs them: python -c pass, pip, pytest, the notebook kernel, the scheduler. PEP 829 calls this "a broad attack surface".

This is not theoretical. The advisory for the compromised litellm 1.82.7 and 1.82.8 releases (PYSEC-2026-2, published March 24, 2026) describes credential-harvesting malware, and its references point to a litellm_init.pth file inside the 1.82.8 package.

And it is not only malware. setuptools 84.0.0, installed fresh on the VM, still ships distutils-precedence.pth, whose single line starts with import os; ....

What PEP 829 and the site docs change in 3.15:

A new name.start file lists entry points as pkg.mod:callable. They run after all .pth path lines are applied.
If name.start exists next to name.pth, the import lines in that .pth are ignored.
Without a matching .start, import lines still run in 3.15, and the deprecation message appears only with -v. The docs say they will be "silently ignored in Python 3.18 and 3.19", with a warning in 3.20.
Errors on one line no longer abort the rest of the file. A failing entry point prints a traceback and startup continues.

On the VM, with a demo zz_demo.pth containing a path line and an import line:

$ python -c "import sys; print('/tmp/extra_path_dir' in sys.path)"
[zz_demo.pth] import line executed at startup
True

python -v shows the new deprecation notice for both the demo file and setuptools' file:

import lines in .../site-packages/distutils-precedence.pth are deprecated, use entry points in a .../site-packages/distutils-precedence.start file instead.

After adding zz_demo.start with one line, zz_demo_boot:init, the import line is suppressed and the entry point runs:

import lines in .../zz_demo.pth are suppressed due to matching .../zz_demo.start file.
Executing entry point: zz_demo_boot:init from .../zz_demo.start
[zz_demo.start] entry point zz_demo_boot:init called

Here is the part to be precise about. A .start file still runs code at every interpreter start. A malicious wheel can ship one just as easily as a .pth. The PEP says so: "The overall pre-start code execution attack surface is not eliminated by this PEP." What 3.15 gives you is a narrower, auditable shape: startup code is a named callable in an importable module, it goes through the import system so audit hooks (PEP 578) can see it, and "you can tell by listing the files in the site-dir, exactly where arbitrary code execution occurs."

That listing is the useful part, and you can do it today on any Python version:

startup_audit.py (runs on Python 3.9+)

"""List everything that can run code at interpreter start in this environment.

Run it with the interpreter you want to audit:
    .venv/bin/python startup_audit.py
"""
import site
import sys
from pathlib import Path

dirs = list(dict.fromkeys(site.getsitepackages() + [site.getusersitepackages()]))
found = 0
for d in map(Path, dirs):
    if not d.is_dir():
        continue
    for f in sorted(d.glob("*.pth")):
        lines = f.read_text(encoding="utf-8-sig", errors="replace").splitlines()
        code = [l for l in lines if l.startswith(("import ", "import\t"))]
        if code:
            found += 1
            shadow = f.with_suffix(".start").exists()
            note = "skipped on 3.15+ (matching .start), runs on older Pythons" if shadow else "RUNS on every start"
            print(f"{f}  [{note}]")
            for l in code:
                print(f"    {l[:110]}")
    for f in sorted(d.glob("*.start")):
        found += 1
        print(f"{f}  [.start entry points]")
        for l in f.read_text(encoding="utf-8-sig").splitlines():
            if l.strip() and not l.lstrip().startswith("#"):
                print(f"    {l.strip()}")
    for name in ("sitecustomize.py", "usercustomize.py"):
        if (d / name).exists():
            found += 1
            print(f"{d / name}  [runs on every start]")
print(f"{sys.version.split()[0]}: {found} startup code hook(s) found")

Output in the demo environment (with setuptools installed):

.../site-packages/distutils-precedence.pth  [RUNS on every start]
    import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __
.../site-packages/zz_demo.pth  [skipped on 3.15+ (matching .start), runs on older Pythons]
    import sys; sys.stderr.write("[zz_demo.pth] import line executed at startup\n")
.../site-packages/zz_demo.start  [.start entry points]
    zz_demo_boot:init
3.15.0: 3 startup code hook(s) found

Most .pth files are harmless. An editable install of a src-layout package with setuptools wrote a path-only .pth on the VM, with no code at all. The ones that execute code are few enough to put on an allowlist.

python -S disables both mechanisms (I checked), but it also skips site-packages entirely, so it is a debugging tool, not a production setting.

Runbook: startup code

Run the audit in your image build and fail the build on any startup hook that is not on an allowlist.
If you maintain a package with a .pth import line, move the code into a zero-argument function and add a matching .start file. Keep the .pth line as import pkg.mod; pkg.mod.callable() while you still support 3.14.
Treat a new .start file in a dependency diff like a new postinstall script in npm: a reason to read the release.

What to do this week

If you run...	Do this	Skip this
Python CLIs, CI tooling, notebook kernels	Measure with `-X importtime`, add `lazy` to your entry point's heavy imports	`PYTHON_LAZY_IMPORTS=all` in images
Long batch jobs on VMs or Kubernetes	Install the exact job interpreter in the image, rehearse `dump --blocking` and `attach` on a staging job, sort out `CAP_SYS_PTRACE`	Trusting a profile without reading `Error rate`
Anything that installs from PyPI	Add the startup audit to the image build, review new `.pth` and `.start` files in dependency updates	Assuming `.start` files make startup code safe
Python-heavy loops or row-wise UDFs	Try `PYTHON_JIT=1` in a benchmark, expect single digits	Counting on the JIT to fix a row-wise `apply`

The JIT will show up in your benchmarks as a few percent. Lazy imports, Tachyon and .start files will show up in your incident timeline: the CLI that starts in 20 ms instead of 350, the stuck job you can inspect without a restart, and the list of files that run code before your main() does. Upgrade for those, and measure the rest.

Instant at 10 Million Rows, Dead at a Billion: A Decision Framework for Billion-Row SQL

Fri, 09 Oct 2026 06:03:06 +0000

A query that returns in 40 milliseconds on ten million rows can time out completely on a billion. Same SQL, same schema, same database. I've watched a lot of teams hit that wall and go hunting for the one fix: a cleverer index, a bigger instance, whatever database was trending that month. There isn't one fix. There are two questions that decide almost everything. What engine class is this workload on — row-store, columnar, or distributed? And does the query's working set fit in memory, or does it spill to disk? Answer those honestly and most "billion-row problems" quietly resolve. Answer them wrong and no amount of tuning bails you out.

The query that was instant at 10 million rows and timed out at a billion

Here's the shape of it. It's a composite; I've seen it often enough that the specifics blur.

A team builds a dashboard on their main OLTP database. One query carries it: count distinct users, grouped by day, over a date range. Against a 10-million-row staging copy it comes back instantly, everyone signs off, ship it. Months later production crosses a billion rows — roughly the 1.1-billion-row scale in Mark Litwintschik's cross-engine benchmarks — and the identical query simply stops returning. On-call adds an index; nothing. A composite index; nothing. A bigger instance; somehow slightly worse. "Throw it at Spark," one person says. "Just use ClickHouse," says another. Nobody can explain why either would help. That's the real problem: everyone is arguing tactics with no framework for the decision.

The query didn't regress. It's doing exactly what it always did. What changed is that the workload crossed two lines at once — it outgrew the engine class that suited it at 10M, and its sorts and hashes stopped fitting in memory. Name those two lines and the mystery turns into a decision you can defend.

The two-axis framework

When a query dies at scale, the reflex is to index harder. Sometimes that's right. At a billion rows it's usually the wrong axis. Two levers actually decide whether a query survives.

Axis 1 — Engine class. Every database is built around one dominant access pattern, and that choice shapes everything downstream. A row-store OLTP engine (PostgreSQL, MySQL) keeps whole rows together and excels at "fetch or update this one record" and narrow ranges. A columnar/OLAP engine (ClickHouse, DuckDB, BigQuery) stores each column separately, so scanning and aggregating hundreds of millions of rows is the design point rather than the emergency. A distributed engine (Citus, Spark, Cassandra) spreads data across nodes for horizontal scale, trading query flexibility to get it. These are peers, not rungs on a ladder; none is "more advanced." Run a workload on the wrong one and you get a query that flies at 10M and dies at 1B.

Axis 2 — Memory budget. Independent of engine class, every non-trivial query builds intermediate structures: sort buffers, hash tables for GROUP BY and DISTINCT, join tables. Fit those in RAM and the operation runs at memory speed. Overflow the memory the engine is allowed, and it spills to disk and falls off a cliff. PostgreSQL ships shared_buffers at 128MB and work_mem at just 4MB; the docs are explicit that a sort or hash exceeding work_mem starts "writing to temporary disk files" (PostgreSQL 18 — Resource Consumption, as of 2026). Four megabytes is nothing at a billion rows. This is the axis most write-ups skip, and it's why the same query is fast one day and glacial the next. The SQL didn't change; the data just got big enough to spill.

The axes are orthogonal. You can pick the right engine class and still hit the memory cliff, or keep memory in check and still be on an engine that's wrong for the query's shape. So the move is: work out which quadrant you're actually in, then act on the axis that matches — change engine class, or change the memory budget — instead of adding another index to the axis that was never the problem.

Engine class 1: Row-store OLTP — great at point lookups, brutal on full-table aggregates

Row-stores are the workhorse for good reason. Ask for "order #48213 and its line items" or "the last 50 events for this user" and a row-store with the right index is hard to beat: it seeks to a handful of rows and returns. At 10M rows, the dashboard query was riding on that.

Aggregates over the whole table are a different story, and the reason is baked into how MVCC row-stores stay correct under concurrency. Because different transactions can see different versions of a row, the engine keeps no cached count of how many rows a table has. An exact count(*) therefore has to visit rows and check visibility — a sequential scan that grows linearly with the table.

The numbers are blunt. In Joe Nelson's Citus benchmark on counting (PostgreSQL 9.5.4, 2016 — treat the milliseconds as illustrative of the mechanism, which is unchanged today, not as current figures), an exact count(*) ran 85ms at 1M rows, 161ms at 2M, and 343ms at 4M: a straight line, with the scan about 88% of the cost. Extend that to a billion rows and a single count is tens of seconds before any filtering. count(1) doesn't rescue you; it measured slightly slower (99ms vs 85ms at 1M), because Postgres special-cases the argument-free count(*) while count(1) re-checks the argument on every row. And this isn't Postgres being difficult — MySQL's InnoDB shares the MVCC property and also has no O(1) table count.

When you don't need an exact number, stop asking for one. Every row-store keeps a maintained estimate for its planner, and you can read it directly. In PostgreSQL that's pg_class.reltuples (the Count-estimate pattern: SELECT reltuples::bigint FROM pg_class WHERE oid = 'schema.table'::regclass). In the same benchmark the estimate returned in about 0.3ms against 85ms for the exact count at 1M rows — roughly 280× faster, and effectively constant time at any size. The catch is accuracy: it's an estimate and can't apply a WHERE or DISTINCT. For "roughly how many rows" on a dashboard, that's the difference between a billion-row scan and reading one catalog row.

None of this means "Postgres is slow." It means a full-table aggregate is off-design for a row-store. Approximate counts, materialized rollups, and summary tables all soften it. But if scans and aggregates are your dominant pattern, you're fighting the engine class — and that's your first crossover signal.

Exact `count() vs the reltuples` estimate. Source: Citus / Joe Nelson, PostgreSQL 9.5.4, 2016 — illustrative of the mechanism, not current figures.*

Engine class 2: Columnar / OLAP — where scans and aggregates are the design point

If aggregates are killing your row-store, what's built for them, and how much faster is it really? This is where the framework gets counter-intuitive.

Columnar engines store each column contiguously instead of packing whole rows. A scan-and-aggregate query that touches 3 of 51 columns reads only those 3 off disk; the other 48 are never opened. And because a column holds one type with heavy repetition, it compresses hard, so there's less to read to begin with. Column pruning plus compression is why "scan a billion rows and aggregate" is routine here instead of an incident.

The headline number: Litwintschik's 1.1-billion-row taxi benchmark (1.1B rows, 51 columns, ~500GB uncompressed; page updated March 2024) runs one analytical query across dozens of engines. On Query 1, a single desktop ran ClickHouse in 0.088s and DuckDB in 0.498s. The same query took 2.36s on a 21-node Spark cluster, 3.54s on 21-node Presto, and 1–3s on BigQuery. Postgres with the cstore_fdw columnar extension took 152–368s across the four queries, and plain SQLite's format took 31,193s on Query 1 — about 8.7 hours, included to show what the wrong engine class actually costs.

A single desktop beat a 21-node cluster on a 1.1-billion-row scan by more than an order of magnitude. That one result is the whole framework in miniature. It isn't that ClickHouse or DuckDB is "the best database" — it's that for a scan-aggregate workload a columnar engine is in the right quadrant while the cluster pays coordination overhead it doesn't need at this size. (Caveats: one benchmark, one hardware generation, recent desktop CPUs for the fastest single-node runs. Trust the shape of the gap, not the exact milliseconds.)

Compression is the other half, and it's an economics lever as much as a speed one. ClickBench — the ClickHouse team's scan/aggregate benchmark across 60+ systems — publishes a live "Data Size" column so you can compare on-disk footprint on the same ~100M-row dataset. Columnar layouts routinely pack this kind of wide, repetitive data down by a large multiple versus a row-store, though the exact ratio moves with engine, codec, and data, so read the live figure for the systems you're weighing rather than trusting a headline number. (ClickBench is ClickHouse-run, so treat it as a scan-workload reference, not a neutral verdict.)

The honest counter-point: the same columnar engine that crushes the scan is the wrong tool for high-QPS single-row writes and point lookups. Insert a row and you touch every column store; update a field and compression works against you. The wide-column and key-value stores built for that job — Cassandra, ScyllaDB — sit at the opposite pole: excellent at point reads and writes on a known partition key, and structurally unfit for ad-hoc full-table aggregates. Cassandra's CQL docs say it plainly: CQL "only allows select queries that don't involve a full scan of all partitions," and forcing one needs ALLOW FILTERING, whose performance the docs call "unpredictable." A store modeled query-first around the partition key isn't a scan-aggregate engine, which is exactly why ClickBench lists Cassandra and ScyllaDB among the systems it couldn't benchmark on that suite. Right engine, wrong job, in both directions.

Same query, same 1.1B-row dataset, across engine classes. Source: Mark Litwintschik, 1.1 Billion Taxi Rides benchmarks, page updated March 2024.

Engine class 3: Distributed — horizontal scale, and the fan-out ceiling

When one node genuinely can't hold the data or serve the writes, distributed engines spread the work across many. The mental model most people carry — more nodes, more speed — holds right up until it doesn't, and knowing where that line sits is the difference between a defensible cluster and a single box with a network bill attached.

The line: does the query push down to the shards, or force a reshuffle? Distribute by tenant_id and group by tenant_id, and each node computes its slice while the coordinator stitches the results together. That's pushdown, and it parallelizes beautifully. But the moment a query needs data spread across nodes — a count(DISTINCT) on a non-distribution column, or a join across the distribution boundary — the engine has to reshuffle rows between workers first. That shuffle is the ceiling.

The Citus benchmark puts numbers on it (100M rows, 8 nodes, 32 shards; PG 9.5.4, 2016 — again mechanism-illustrative). A plain count(*) across the cluster ran about 1.2s. A count(DISTINCT) on the distribution column, which pushes down, ran about 3.4s. On a non-distribution column it can't push down, and pulling every row back to the coordinator is, in Nelson's words, "really no better than counting on a single database instance … plus high network overhead." An HLL approximate distinct ran 3.2–3.8s — the row-store's approximation trick again, now buying a way around the shuffle. This generalizes well past Citus: in Spark the expensive stage is almost always the shuffle, and distributed query design is mostly the craft of keeping the hot queries pushed down.

So distributed isn't a "scale" button. It's a trade — horizontal capacity for query flexibility and operational complexity. Align your access pattern with the distribution key and it's transformative. Keep needing cross-node data and you've bought nodes and network latency to arrive back at roughly single-box performance. That mismatch is its own crossover signal, and the one most likely to be discovered the hard way in production.

The memory / spill axis — the one most write-ups skip

Now the second axis, and the reason "just add an index" so often does nothing. A query can be on exactly the right engine class and still fall off a cliff, because its sorts and hash tables outgrew the memory the engine was allowed and spilled to disk. This is engine-independent: columnar and distributed engines build the same buffers and spill the same way when they overflow their limits. Postgres is just the easiest place to watch it, because its docs name the knob.

The spill cliff

At the default 4MB work_mem (PostgreSQL docs), a GROUP BY, DISTINCT, or ORDER BY that builds something larger switches from an in-memory hash or sort to an external merge that writes temp files to disk. The Citus benchmark shows the size of the drop (PG 9.5.4, 2016; mechanism unchanged per the current docs): at default work_mem, count(DISTINCT) on an integer column averaged 743ms, but on a text column 31,747ms — a ~43× gap — with Sort Method: external merge Disk in the plan. Give the operation enough work_mem to stay in RAM and the integer case drops to about 372ms as an in-memory HashAggregate. That in-RAM-versus-disk boundary is the axis. Stay on the right side and the query flies; cross it and it collapses.

This is exactly why indexing didn't help the composite team. An index changes how you find rows; it does nothing for a GROUP BY/DISTINCT whose result set is too big for work_mem. The lever there is memory, or a smaller working set — not another index.

When to re-architect instead of tune

work_mem is allocated per operation per query, and the docs warn total usage "could be many times" that value under load, so raising it globally just trades a spill problem for an out-of-memory one. Past a point the honest move is structural. PostgreSQL's partitioning guidance offers a clean rule: partition "when the size of the table [would] exceed the physical memory of the database server." Pruning then skips whole partitions — but it isn't free. The planner handles "up to a few thousand" partitions well only if pruning leaves few, since "each partition requires its metadata to be loaded into the local memory of each session that touches it." Thousands of poorly-pruned partitions is its own problem.

One more trap: deep pagination

LIMIT ... OFFSET n gets slower the further you page. As Markus Winand explains in "No Offset", the database "fetches and drops" all n preceding rows before returning the page, and results drift when rows are inserted between requests. A keyset (seek) query filters on the last-seen sort key instead and stays roughly constant-time however deep you go. Page latency that climbs with depth is a spill-adjacent signal, not a hardware one.

count(DISTINCT) at default vs raised work_mem. Source: Citus / Joe Nelson, PostgreSQL 9.5.4, 2016 — illustrative of the mechanism, not current figures.

The crossover-signal checklist: change engine class, don't add another index

This is the piece worth saving. A framework's job is to tell you when you've hit the boundary and it's time to move on an axis instead of tuning further. Eight signals I watch for, each tied to the axis it moves. Two or more firing means you're past tuning and into an engine-class or memory-budget decision.

Scans and aggregates dominate the workload (not point lookups). → Engine class, toward columnar. The row-store aggregate cliff; indexing won't fix an off-design pattern.
The working set exceeds server RAM. → Memory. Postgres's own partition threshold; partition, re-architect, or move to columnar/distributed.
Sorts and DISTINCTs keep spilling even after tuning work_mem. → Memory. You've hit the ceiling of per-query memory; the working set is the problem.
Pagination or scan latency rises with position. → Memory/scan; switch to keyset/seek before switching engines.
A single node has hit its write or IO ceiling. → Engine class, toward distributed. The one genuine "add nodes" signal.
Key DISTINCTs or joins need data from several nodes (a shuffle). → A warning on the distributed axis: without a distribution-key redesign, more nodes ≈ one box plus network.
You need thousands of partitions to stay fast. → Memory/metadata; the per-partition cost says a columnar or distributed store fits the data better.
You're keeping 500GB+ on one box for analytics. → Engine class plus economics; columnar compression and distributed storage costs start driving the decision.

What the list deliberately doesn't do is name a winning database. Each signal points to an axis and a direction, because the right specific engine depends on your query shapes, your team's ops maturity, and your budget. "Change engine class" is an architectural decision with a reason attached — something you can hand a skeptical peer or a budget owner instead of "the internet said use ClickHouse."

Cost and economics for the decision-makers

If you own the stack decision, latency is half the case; the half that gets budget approved is cost and defensibility. The benchmarks reframe cleanly as money.

A single big box can be cheaper and faster than a cluster. The Litwintschik result — one desktop-class node beating a 21-node cluster on a 1.1B-row scan — reads louder as a cost story than a speed one: 21 nodes carry ~21× the compute plus coordination, networking, and a standing ops burden, and still lose on this workload. For "big but not planetary" analytics, the first move is often the right engine class on one well-sized box, not a cluster. Distributed earns its cost when you truly exceed single-node capacity or need the availability.

Compression is a storage-cost lever. A columnar layout shrinks wide, repetitive data by a large multiple (measure it for your engines from ClickBench's "Data Size" column — it varies by engine and codec), which is a recurring cut in bytes stored and, in a warehouse, fewer dollars per query scanned.

Per-query billing cuts both ways. A serverless warehouse (BigQuery ran 1–3s per query here) fits spiky, ad-hoc analytics where an idle cluster would bleed money, and fits badly for a high-frequency dashboard hammering one aggregate, where a right-sized columnar box wins on total cost.

Every price and benchmark here will rotate — pricing shifts, hardware moves the single-node ceiling, vendor benchmarks get re-run. Don't anchor the decision to one dated figure. The framework doesn't rotate: two axes, eight signals. If a price moves, re-run the comparison.

Build it yourself

The framework is cheap until you feel it on your own data. Three projects, beginner to advanced. Each names a real tool (links verified reachable in 2026, but repos move — check before you lean on them).

Project 1 — Count your table three ways (~30 min). Feel the row-store aggregate cliff on real data. You'll need a Postgres or MySQL table with 10M+ rows and DuckDB. Run EXPLAIN ANALYZE SELECT count(*) FROM your_table; and note the time and the Seq Scan; then SELECT reltuples::bigint FROM pg_class WHERE oid = 'schema.your_table'::regclass;; then get a slice into columnar form — export CSV (COPY your_table TO 'slice.csv' (FORMAT csv)) and load it into DuckDB, or read the table directly with pg_duckdb — and count(*) there. Done when you can state your own numbers for all three and point to the Seq Scan. Stretch: add GROUP BY day.

Project 2 — Reproduce the spill cliff (~1–2 hrs). Watch a query cross the RAM→disk boundary. You'll need Postgres and a 1M+ row table with an integer and a text column (the NYC-taxi or ClickBench sets work). At default work_mem, run EXPLAIN ANALYZE SELECT count(DISTINCT int_col) FROM t; then the same on text_col, and confirm Sort Method: external merge Disk on the slow one. SET work_mem = '1GB';, re-run both. Done when you've captured your own before/after ratio and seen the plan flip to in-memory HashAggregate. Stretch: chart latency as you step work_mem from 4MB to 1GB and find your own cliff edge.

Project 3 — Run a slice of the 1.1B benchmark (~half a day). Reproduce the crossover on your hardware. You'll need the same big table in a row-store and a columnar engine, plus the ClickBench loaders (60+ DBMS setup scripts) and Litwintschik's posts for reference. Load an identical scan-aggregate table into each, time one representative query on both, and record the ratio and your hardware. Done when you reproduce a >100× gap — or can explain from the plans why yours differs. Stretch: add a distributed engine and build a query that forces a shuffle to find your own fan-out ceiling.

What to do Monday morning

The whole thing, compressed. When a query dies at scale, stop asking "which index?" and ask the two questions that decide it. Engine class: is the dominant shape point-lookups (row-store), scans and aggregates (columnar), or genuinely bigger than one node (distributed)? Memory budget: does the working set fit in RAM, or is it spilling? Then walk the eight signals; two or more means the move is on an axis, not another index. There's no single answer to "how do I handle a billion rows" — the answer is a decision you can write down and defend.

So go run it. Take your biggest table this week and do Project 1: count(*), then reltuples, then a columnar count in DuckDB. Watch where the plan shows a Seq Scan and how far apart the three numbers land. Want the full crossover? The ClickBench repo has loaders for 60+ engines, so you can time the same scan on a row-store and a columnar engine side by side. Bring back your own numbers — the framework only lands once the cliff is yours.

Understanding Parquet File Format

Thu, 08 Oct 2026 11:42:51 +0000

What is a File Format?

A file format defines how data is structured and stored so that software can read and write it consistently.

Think of it as the container for your data. Just like music comes in MP3, WAV, or FLAC, data files also come in different formats depending on the use case.

Common data file formats:

CSV – Simple text-based, rows separated by commas (easy to use, but
large in size and no compression).
JSON – Flexible, hierarchical (good for APIs but verbose).
Avro – Row-based binary format, compact, supports schema evolution
(common in streaming).
ORC – Columnar format optimized for Hadoop and Hive.
Parquet – Columnar format designed for analytics and compression
efficiency.

Introduction to Parquet

If you work with data, you have probably heard the term Parquet come up in conversations about data lakes, analytics, or cloud storage. But what exactly is it, and why does everyone seem to love it?

The Everyday Analogy: Organizing Your Groceries

Imagine you run a grocery delivery service, and you have to store all the items customers ordered.

You could store them in two ways:

Row-based storage (like CSV):

You put one of each item from an order into a bag: milk, bread, apples, and pasta all together. To find all the bread you have sold, you have to open every bag and dig around to see whether it is inside.

Column-based storage (like Parquet):
Instead of mixing, you group similar items together:

One box for all the milk
One box for all the bread
One box for all the apples

If you need all the bread you have sold, you simply open the bread box, no need to touch anything else.

That is Parquet in action, organized storage that makes retrieval faster and more efficient.

The Technical View

Parquet is an open-source, columnar storage format developed by Twitter and Cloudera, now an Apache project. It is:

Self-describing – Metadata in the file footer stores schema, number
of rows, and statistics.
Highly compressed – Supports Snappy, Gzip, ZSTD, and more.
Structured internally into:
Row Groups → subsets of rows (horizontal partitions).
Column Chunks → data for each column inside a row group (vertical partitions).
Pages → the smallest unit of storage, making reads efficient.
Magic Number (PAR1) – Ensures the file is recognized as Parquet.

This hybrid layout balances efficient columnar reads with manageable write performance.

Why Choose Parquet Over Other Formats?

Now that we understand Parquet, let us see why it stands out compared to other formats.

CSV – Human-readable but inefficient; no compression or schema
support.
JSON – Flexible for nested data, but verbose and large in size.
Avro – Great for row-based streaming but not optimized for analytics.
ORC – Similar to Parquet, optimized for Hadoop and Hive workloads.

Parquet vs the rest:

Much smaller file sizes due to efficient compression
Faster query performance with column pruning and predicate pushdown
Widely supported across Spark, Hive, Presto, Trino, BigQuery,
Snowflake, AWS Athena, and more
Ideal for data lakes and analytical workloads

How Data is Written in Parquet

When writing data into a Parquet file, the process follows a layered structure:

Rows are collected into Row Groups

Instead of writing each record one by one (like CSV), Parquet groups
multiple rows together into a Row Group (horizontal partition).

Within each Row Group, data is split into Column Chunks

Each column’s values are stored together → Column A values go into
Column Chunk A, Column B values into Column Chunk B, and so on.
This preserves the columnar storage advantage.

Column Chunks are broken into Pages

Pages are the smallest unit of storage in Parquet.
Data Pages → contain the actual column data
Dictionary Pages → optionally store dictionaries used for dictionary encoding
Optional Page Indexes can provide additional information for
efficiently locating and skipping pages:
Column Index → stores statistics such as min/max values for pages
Offset Index → stores the locations and sizes of pages

Metadata is written into the File Footer

Contains schema, row count, statistics (min/max per column), and row
group metadata.
Makes Parquet a self-describing format.

Magic Number (PAR1) is added

At the start and end of the file → ensures the file can be validated
as a Parquet file.

How Data is Read from a Parquet File

File Footer Read First

Parquet does not scan the entire file right away.
It jumps directly to the footer, where all the metadata is stored.
Metadata tells the engine:
Schema (what columns exist)
Number of rows
Statistics for each column in each row group (min/max)
Row group and column chunk offsets (where the data lives in the file)

➡️ This helps the engine decide which parts of the file to actually read.

Predicate Pushdown & Column Pruning

If the query only needs certain columns (SELECT name, age), only
those column chunks are accessed.
If the query has filters (WHERE age > 30), Parquet uses min/max stats
in metadata to skip entire row groups that don’t match.

➡️ This reduces I/O drastically compared to row-based formats.

Reading Row Groups

The engine opens only the relevant row groups.
Inside each row group, it locates the column chunks needed for the
query.

Decompression & Decoding Pages

Column chunks are split into pages.
Each page is decompressed (Snappy, Gzip, ZSTD, etc.) and decoded
(Dictionary, RLE, Delta encoding).
Pages are read into memory one at a time, which is efficient.

Reconstructing Results

Once the engine reads, decompresses, and decodes the relevant column
data, it reconstructs the requested rows and columns.
The engine materializes the final output as a result set for the
query.

Parquet File Size: Small vs Large

The size of each Parquet file matters a lot for performance.

Too Small Files (< 50 MB each)

Too much metadata and file-open operations.
Increases overhead for query engines (Spark, Presto, Hive).
Known as the small files problem.

Too Large Files (> 1 GB each)

Slows down reads and reduces parallelism.

💡 Ideal File Size:

256 MB to 1 GB for most big data frameworks.
Large enough to compress well, small enough for parallelism.

How Data Types Affect Compression (and File Size)

Parquet’s efficiency is heavily influenced by data type selection:

Strings vs Integers: Integers compress far better than storing
numbers as strings.
Decimals vs Floats: Decimals (fixed precision) often compress better.
Categorical Data: Repeated values (like “USA”) compress extremely
well with dictionary encoding.
Timestamps: Storing as INT64 (epoch values) is more efficient than strings.

Tip: Choose the most precise and smallest data type possible to save space and improve query speed.

Encoding Strategies used in Parquet

Run Length Encoding (RLE)

How it works: Stores consecutive repeated values as a single value + a count.

Best for: Columns with long runs of the same value (e.g., flags, booleans, country/state codes).

Benefit: Extremely efficient for repetitive data, reduces size drastically.

Example:

Raw → A A A A B B C C C
RLE → (A,4), (B,2), (C,3)

Dictionary Encoding

How it works: Builds a dictionary of unique values in a column and replaces each value with a small index reference.

Best for: Low-cardinality columns (few unique values, but repeated often).

Benefit: Saves space when values repeat often, lookups are fast.

Example:

Raw → USA, USA, CAN, USA, CAN, MEX
Dictionary → {0: USA, 1: CAN, 2: MEX}
Encoded → 0, 0, 1, 0, 1, 2

Delta Encoding

How it works: Stores the difference (delta) between consecutive values instead of storing full values.

Best for: Sequential/numeric data like timestamps, IDs, counters.

Benefit: Great for compressing increasing sequences, works well with integers and dates.

Example:

Raw → 100, 105, 110, 115
Delta → 100, +5, +5, +5

Parquet Optimization & Best Practices

Predicate Pushdown – Enable in your query engine to filter row groups
at scan time.
Compression Codecs
Snappy: Fast, decent ratio
Gzip: Higher compression, slower
ZSTD: Good balance
Avoid Small Files – Merge into larger files to reduce metadata
overhead.
Row Group Size – Balance between compression (bigger row groups) and
parallelism (smaller ones).
Encoding Strategies
Dictionary Encoding for low-cardinality strings
Run-Length Encoding for repeated values
Delta Encoding for incremental numbers like timestamps
Sort Data – Sorting on commonly filtered columns improves compression
and predicate pushdown.
ACID Transactions – Parquet itself does not provide ACID, use table formats like Delta Lake, Iceberg, or Hudi for transactional
guarantees.

Conclusion

Parquet has become a default choice in the modern data stack because it offers the best of both worlds like compact storage and fast analytics.

By understanding how it works internally (row groups, column chunks, pages) and following best practices (ideal file sizes, data types, compression, and encodings), you can save costs, improve query speeds, and build scalable pipelines.

# Power BI Data Modelling: A Great Path to Great Analysis

Thu, 08 Oct 2026 03:59:58 +0000

Data Modeling in Power BI

Introduction

At a time when data is rapidly generated by different activities ranging from experiments to business, more often do we get data in different formats and structures. The data may or may not be related depending on the source. Related data in different formats and structure need to be re-organized to establish existing relationships for accurate analysis and reporting. To establish data model relationships and properties we need advanced tools that can connect to these sources. Then load the different datasets from source, and perform modelling to connect the links.

Data Modeling therefore, becomes very important. Data Modeling is the process of defining the data structure, properties, and relationships within a Data Model. Through organizing tables and columns, defining relationships between tables in the model, setting cardinality and cross-filter direction we achieve the best model which supports accurate calculations and reporting.

A Data Model is the logical representation of how data is structured and related within the data model.

Before modeling ensure that your data is clean and up to date, ready for modeling.

Why Data Modeling

Improved Performance and Scalability - join optimization, filters, and aggregations lead to a good model that is efficient and reliable. However, poorly designed models with complex relationships and redundant records cause delays, making queries slow.
Accuracy and Consistency - Guarantees the accuracy and consistency of the data used, making it reliable in reporting and drawing precise insights and improved decision making.
Supports Data Exploration - Users can use hierarchies and drill-down paths to support efficient data exploration.
Ease of Maintenance - Any changes in one model or report are reflected everywhere in real time enabling reusability and scalability, reducing the effort required to maintain reports and models as the business grows.

Components of Power BI Data Model

a. Tables and Sources

Tables are the raw building blocks of data models. They contain fields and records of particular data from different sources.
Sources, where we get our data from can vary depending on the context. We can get our data from SQL Database servers, Excel files, Data warehouse, SharePoint, and Power BI Dataflows.

Designing Source Tables

i. Fact Tables

Fact tables contain quantitative data, numerical data that can be aggregated and analyzed. Examples the Sales, Orders, or Invoices. This information is about specific events, business transaction, or measurements at the core of the data model.

Fact tables keeps one record for each transaction or event, includes only keys and metrics, and they avoid storing descriptive attributes.

Fact tables contain:

Foreign Key(FK) - a link to the Dimension table i.e. DateKey, ProductKey, e.t.c.
Measures(Numeric values) - Quantitative data you can aggregate. i.e. Quantity, UnitPrice, TotalAmount etc.
Degenerate dimensions - Descriptive data that does not warrant its dimension table. i.e. OrderNumber, InvoiceNumber.

ii. Dimension Tables

Dimension tables provide context or additional details to the data in Fact Tables. They contains fewer rows while fact tables can contain a large number of rows and they are on growing.

Relationship are used to establish links and connections between the fact and dimension tables in Power BI.

Dimension table contain:

Primary Key(PK) - Unique identifier e.g. ProductID
Descriptive attributes - Text, categories, hierarchies e.g. ProductName
Hierarchy Columns - Columns that form drill-down paths e.g. Country -> City -> Town.
Natural key - original identifier from the source system e.g. ProductCode

iii. Reference and Lookup Tables

They are small but critical tables in supporting the integrity and interpretation of data. They support additional mappings or categorizations. Example: DiscountTypes or StatusCodes.

Loading Data

Open Power BI
On the Ribbon Home tab, Click on Get Data from the Data group option.

From the drop down list, choose the source of your data. i.e. Text/CSV if your data is text or comma separated file.

Choose the path to your file i.e. C:/Users/User/Downloads/file.csv and click on Open.
Click on Connect on the Power Query editor window.
Your data is extracted to Power Query for cleaning. Click on Transform - Data to clean explore and clean it on the Power Query editor.

Finally, on the Power Query Editor Window Ribbon, Click on Close & Apply from the Home tab.

Changes are loaded and applied to Power BI. We will load all our Toy Store E-Commerce Database data set for modeling using the procedure above. The data is already clean, we are going to use it for demonstrating relationships in data modeling.

b. Relationships

Relationships are an important aspect of data modeling in Power BI. Relationships are for establishing connections and links between tables of a data model. They allow related tables to communicate with each other. They are the link between Fact Tables and Dimension Tables.

Relationships determine how data in the model is filtered, aggregated, combined, and visualized. They're critical to the accuracy and performance of queries in Power BI.

Cardinality

Cardinality type defines each relationship in a model. Cardinality refers to the number of unique values in one table related to the number of unique values in another.

Types of Cardinality in Relationships

One-to-One(1:1) - Every single record in one table is related to another single record in another table. Commonly used when splitting a large table for performance and security reasons.
One-to-Many(1:N) - Refers to when a single record in one table is related to many records in another table(s). Very common type of relationship in Power BI models. The Fact table is on the Many side while the Dimension table is on the One side when linking Fact tables and Dimension tables.
Many-to-Many(N:N) - Refers to when many records in the first table are related to many other records in the second table.

Cross Filter Directions

Used to determine the direction which filters propagate.

Single - When applied, only one of the tables in the relationship can filter data across the other table. Filtering data on the other table does not affect the data shown on the first table.

Bi-Directional - When applied, data can be filtered on either direction between the tables. Filtering data in one table affects data shown on the other table and vice versa. It is less often used and can impact performance negatively and create ambiguous filters.

Cross-filter options are largely dependent on the cardinality type.

Cardinality	Cross-Filter Options
1:1	Bi-Directional
1:*	Single or Bi-Directional
:	Single or Bi-Directional

Table 1: A table showing cardinality and their cross-filter options.

Creating a Relationship

To create relationships, first you need to:

Identify the matching columns(keys).
Choose the right cardinality.
Decide on the Filter direction.

The Model View Interface

To manage relationships in Power BI Desktop:

Click the Model icon in the left sidebar (the icon with three connected boxes)

You'll see all your tables displayed as boxes with their columns listed

Lines between tables represent relationships
The line style indicates cardinality: 1 on one end and * on the many end

Auto-Detect Relationships

When you load tables, Power BI automatically recognizes columns with same data type and name and creates relationships. However, you shoul always create relationships manually or review those created automatically.

Just like in our case, Power BI automatically creates relationships between tables in our model. We will review them.

When auto-detect works well:

Column names match exactly (e.g., ProductID in both tables)
Data types are compatible
Values actually correspond between the tables

Creating Relationships Manually

Using the Managing Relationship Dialog

On the Home tab → Manage Relationships

Click New

Select the first table and column, From Table
Select the second table and column, To Table
Configure cardinality and cross-filter direction

Click Save

Step by Step for Creating Relationships for our Toy Store E-Commerce Database

Load all tables:

 ```
     products
     orders
     order_items
     order_item_refunds
     website_sessions
     website_pageviews
 ````

Open Model view and delete any autodetected relationship.

To delete a relationship:

In Model view, right-click on the line joining the products and order_items tables
From the options displayed, Click on Delete

Save your work

Create the following relationships:

From Fact Table	To Dimension Table	Cardinality	Cross-Filter
products[product_id]	order_items[product_id]	One-to-Many	Single
orders[order_id]	order_items[order_id]	One-to-Many	Single
orders[order_id]	order_item_refunds[order_id]	One-to-Many	Single
website_sessions[website_session_id]	orders[website_session_id]	One-to-One	Bi-Direction
website_sessions[website_session_id]	website_pageviews[website_session_id]	One-to-Many	Single
order_items[order_item_id]	order_item_refunds[order_item_id]	One-to-one	Bi-Direction

Table 2: Table showing relationships to be created.

Creating a relationship between the products and order_items tables

On the Home tab → Manage Relationships
Click New
Select the products table and specify the column product_id
Select the order_items table and specify the column product_id
Set cardinality to One-to-Many and the cross-filter as Single

Click Save

Complete creating all the other relationships following the procedure. You will get something similar to:

Verify by checking that filters from any dimension table correctly filter the fact table.

Conclusion

Data modeling is very crucial to building powerful reports and visualizations. It is key to great analysis because it establishes relationships, links, and connections existing in a data model making it wholesome and complete. Further, it improves query performance increasing efficiency and reliability of queries. Correct data modeling can lead to accurate reporting and result reliability, thus informed decision making. To view the complete model and data for your practice on GitHub visit Toy Store E-Commerce.

Data Preprocessing for Machine Learning

Sat, 26 Sep 2026 07:51:45 +0000

Introduction

Most ML tutorials jump straight into models and algorithms. But here is what no one tells beginners: the model isn't where the magic happens. The data preparation is.

Think of it this way. You wouldn't hand a toddler a jigsaw puzzle with missing pieces, warped edges, and pieces from three different puzzles mixed, and then blame the toddler for not solving it. That's exactly what happens when we feed raw, messy data into a machine learning model and wonder why it performs terribly.

Data preprocessing is the act of preparing raw data so a model can actually make sense of it. It covers everything from creating useful features to handling gaps in the data to making sure numbers play fair with each other.

In this article, we will walk through four pillars of preprocessing:

Feature Engineering
Handling Missing Values
Scaling
Normalization

By the end, you won’t just know what these terms mean. You will understand why they exist, when they matter, and what breaks when you skip them.

Why Preprocessing Matters

A machine learning model is only as good as the data it receives.

Real-world data is messy. It has gaps. It has inconsistencies. Some numbers are in the thousands, others are tiny decimals. Some information is buried inside other information and needs to be extracted before it becomes useful.

Preprocessing bridges raw chaos and structured learning.

The Four Pillars of Data Preprocessing

Each pillar addresses a unique issue in your data. Neglecting any one of them will compromise your model's performance.

1. Feature Engineering:

Teaching the Machine What to Notice

The Child: A child walks into a pet store. They see animals everywhere. But their parents point and say, “Look at the size. Look at the fur. Does it have a tail?” The parent isn’t changing the animals; they are teaching the child what to pay attention to.

The Machine: Raw data often contains information, but not in a form the model can use directly. Feature engineering is the process of creating, transforming, or selecting the right input variables (features) so the model can actually find patterns.

Why it matters: A column named “date_of_birth” is not useful for a model predicting insurance risk. However, a new column called “age,” derived from that date, provides valuable information for the model.

What Feature Engineering Looks Like in Practice

Creating new features from existing ones: Given a “timestamp” column, derive “hour_of_day,” “day_of_week,” and “is_weekend.” This provides a model predicting restaurant traffic with useful signals, rather than a raw timestamp it cannot interpret.

Combining Features: You have “house_length” and “house_width.” Neither feature alone provides a complete picture. Multiply them to derive “house_area,” a single feature that captures information not conveyed by either feature individually.

Encoding Categories: A column contains the values “Red,” “Blue,” and “Green.” Since a model cannot interpret these labels directly, they are converted into numerical representations that the model can process. One-hot encoding, for example, represents “Red” as [1, 0, 0], “Blue” as [0, 1, 0], and so on.

The Takeaway: Feature engineering is the art of translating human knowledge into a language the model can learn from. The model finds patterns, but you decide what it gets to look at.

2. Handling Missing Values:

Filling in the Gaps

The Child: A child is reading a picture book, and one page is torn out. They don’t throw away the entire book. They might guess what happened on that page based on the story before and after. Or they skip it and keep reading.

The Machine: Real-world datasets almost always have missing values. Sensors fail. People skip survey questions. Records get corrupted. The model needs a strategy: fill the gap, remove the gap, or flag the gap.

Why it matters: Most ML algorithms cannot process empty cells. If you don’t handle missing values, your model either crashes or silently learns the wrong thing.

The Three Strategies for Missing Data

Strategy 1 — Remove it (Drop): If only a tiny fraction of your data has gaps, sometimes the simplest move is to drop those rows. Like a child skipping the torn page, you lose a little, but the rest of the story still makes sense. Risk: If too many rows have gaps, you lose valuable data.

Strategy 2 — Fill it (Impute): Replace the missing value with something reasonable. Common approaches include using the mean (average) for numerical data, the mode (most frequent value) for categorical data, or the median if your data has extreme outliers. Like the child guessing what happened on the missing page based on context.

Strategy 3 — Flag it (Indicator): Create a new column that says “this value was missing” (1 or 0). This way, the model knows the gap existed and can learn whether the absence of data is itself a signal. Sometimes, a patient skipping a question on a health survey is more informative than any answer they could have given.

The Takeaway: Missing data is not necessarily a problem; rather, it presents a decision point. The appropriate approach depends on the extent of the missing data, the reasons for its absence, and the requirements of the model.

3. Feature Scaling:

Making the Numbers Play Fair

The Child: Imagine two children comparing their collections. One child has 3 seashells. The other has 3,000 stickers. If you ask, “Who has the bigger collection?” the answer seems obvious, but is it fair? The scales are completely different. To compare meaningfully, you would need to put both collections on the same measuring system.

The Machine: ML models that calculate distances or gradients (like KNN, SVM, or neural networks) are heavily influenced by the magnitude of numbers. A feature ranging from 0 to 1,000,000 will dominate a feature ranging from 0 to 1, even if the smaller feature matters more.

Why it matters: Without scaling, large-magnitude features can overwhelm smaller ones and make them irrelevant. The model doesn’t know that “salary in dollars” and “years of experience” should carry equal weight; it only sees big numbers and small numbers.

The Two Main Scaling Methods

Min-Max Scaling (Normalization to a range): Scales all values into a fixed range, typically 0 to 1. The formula: (value - min) / (max - min). It’s like converting each child’s collection into a percentage of their personal maximum. It works well when you need bounded values and your data doesn’t have extreme outliers.

Standardization (Z-score Scaling): Centers data at 0 and scales it to a standard deviation of 1. Formula: (value - mean) / standard_deviation. Like grading on a curve, each student’s score shows how far it is from average. This works well with outliers or when an algorithm assumes normally distributed data.

The Takeaway: Scaling does not alter the information conveyed by your data. Instead, it adjusts each feature's relative influence, ensuring no single feature disproportionately dominates the others.

4. Normalization:

Reshaping the Distribution

The Child: A teacher asks the class to rate how much they liked a movie on a scale of 1 to 10. One child always gives everything a 9 or 10. Another child’s ratings spread evenly from 1 to 10. To compare their opinions fairly, you would need to adjust for each child’s personal rating style and their distribution.

The Machine: Normalization transforms the shape of your data’s distribution. Even after scaling, your data might be heavily skewed, with most values clustered on one side with a long tail. Many ML algorithms perform better when the data follows a more symmetric, bell-curve-like distribution.

Why it matters: Skewed distributions can mislead models. If 95% of your income data clusters between $20,000 and $80,000 but a few values shoot up to $200,000, the model might overfit to those extremes or underperform on most of the data.

Common Normalization Techniques

Log Transformation: Take the logarithm of each value. This compresses the long tail and spreads out the clustered values. Extremely effective for income data, population data, or anything with exponential growth patterns.

Box-Cox Transformation: A more flexible approach that identifies the optimal power transformation to make the data as closely resemble a normal distribution as possible. The method automatically determines the most appropriate transformation.

Quantile Transformation: Maps values to their percentile ranks to transform the data into a specified distribution, typically uniform or normal. This highly aggressive approach ensures the desired output distribution but may distort relationships among nearby values.

The Takeaway: Normalization concerns the shape of your data, not the scale. It ensures your data’s distribution doesn’t secretly undermine your model’s assumptions.

Scaling vs. Normalization:

These two terms get mixed up constantly,

Scaling adjusts the range or magnitude of your features. It answers: “How big are these numbers relative to each other?”

Normalization adjusts the distribution shape of your features. It answers: “What does the spread of these numbers look like?”

Both are often necessary. First, scale the features to comparable ranges; then, normalize them if the distribution is skewed.

Think of it this way: scaling sets the volume; normalization tunes the equalizer.

The Preprocessing Decision Tree

How do you decide what preprocessing to apply?

Conclusion

Data preprocessing is rarely celebrated or featured in headlines. Nevertheless, it is, without exaggeration, where most substantive machine learning work takes place.

Feature engineering decides what the model sees.
Missing value handling decides how gaps are managed.
Scaling ensures no feature unfairly dominates.
Normalization reshapes distributions so algorithms can work as
designed.

These are essential steps and should not be omitted. They form the foundation for every model, prediction, and result, making thorough data preparation critical.

If you step back, the pattern is strikingly familiar: whether it’s a child learning to organize their toy box or a model learning to classify images, learning quality depends entirely on preparation quality.

Ensure the data is accurate, and the model may exceed expectations. Without adequate preparation, no algorithm can compensate.

How I Used Wren AI to Let My Marketing Team Query a Star Schema Without Writing SQL

Thu, 17 Sep 2026 07:45:12 +0000

If you've worked in data engineering long enough, you know the drill.

Someone from marketing pings you on a Monday morning: "Hey, can you pull campaign spend by ad set for last week?"

You run the query. Send it. Five minutes later: "Can we also add impressions?"

You run it again.

This loop doesn't end. It just gets longer.

At my previous job, I decided to break it. Here's exactly how I did it — using Microsoft Fabric, PostgreSQL, and Wren AI — and what I learned along the way.

The Stack

Before I get into the how, here's what I was working with:

Source: Meta Ads data in Microsoft Fabric (Delta tables, Gold layer)
Target: Local PostgreSQL (localhost:5432, schema: gold)
Semantic layer: Wren AI
Goal: Let business users ask natural language questions against a proper star schema — no SQL required
Step 1: Getting the Data Out of Fabric

My first instinct was a direct Fabric → Postgres connection via Spark JDBC. Clean, automated, elegant.

Didn't work. Fabric is cloud. My Postgres was localhost. Cloud can't reach your laptop. That's just how networking works.

So I went with the practical option: export as CSV, import locally.

Here's what the Gold layer looked like:

Standard star schema. Clean. Well structured. Ready to go.

Step 2: Loading into PostgreSQL

I created a gold schema in Postgres and defined tables for each CSV — all columns as text for the initial load (you can cast later, don't overthink the first load).

sql
CREATE SCHEMA IF NOT EXISTS gold;

Where I got stuck: the COPY command

My first attempt was server-side COPY:

sql
COPY gold.dim_campaign FROM 'C:/Users/myuser/Downloads/dim_campaign.csv'
WITH (FORMAT csv, HEADER true);

Got a Permission denied error immediately. The PostgreSQL service account can't read files from your Windows user profile folder. Makes sense from a security standpoint — still annoying at 11pm.

The fix: Use pgAdmin's client-side Import/Export Data instead.

Right-click the table → Import/Export Data → Import → select your CSV → Format: csv, Header: Yes, Encoding: UTF8.

Slower than bulk COPY but it works every time.

Another thing that caught me: duplicate primary keys

When I tried adding a primary key to dim_adset, Postgres threw this:

ERROR: could not create unique index "dim_adset_pkey"
DETAIL: Key (adset_sk)=(909) is duplicated.

Fix — deduplicate before adding the constraint:

sql
DELETE FROM gold.dim_adset a
USING gold.dim_adset b
WHERE a.ctid < b.ctid
AND a.adset_sk = b.adset_sk;

Always worth running a quick dedup check before adding PKs on imported data.

Step 3: The Star Schema

Standard stuff if you've built a warehouse before. Same conceptual model as what I had in Fabric/Power BI.

Step 4: Connecting Wren AI

This is where it gets interesting.

Wren AI is an open-source semantic layer with a natural language interface. You connect it to your database, define your data model (relationships, metrics, descriptions), and business users can ask plain English questions and get SQL-backed answers.

Here's the workflow I followed:

Connect data source Point Wren AI at localhost:5432, database postgres, schema gold.
Select tables Chose all 8 core tables from the gold schema.
Define relationships In the "Define relationships" screen, I created Many-to-One links using the _sk and date_key columns — matching the star schema above.
Deploy This publishes your model to the Wren engine. The Deploy button was greyed out initially — that usually means there's a validation error in your relationships. Check for incomplete or circular links.
Ask questions Once deployed, anyone can type a question and get an answer.

The Bug That Wasted My Afternoon

After setting everything up, I asked:

"Show total impressions, clicks, and spend by campaign name"

Result: empty.

No error. No warning. Just nothing.

I went to pgAdmin and ran a quick validation:

sql
SELECT 'fact_ad_performance_daily' t, COUNT() c
FROM gold.fact_ad_performance_daily
UNION ALL
SELECT 'dim_campaign', COUNT() FROM gold.dim_campaign
UNION ALL
SELECT 'dim_ad', COUNT(*) FROM gold.dim_ad;

Output:

fact_ad_performance_daily0
dim_campaign1069
dim_ad2813

I had loaded every single dimension table and completely forgot to import the fact table. 39,000 rows. Sitting in a CSV. Not in the database.

Always validate row counts before you start building the semantic layer. This is the kind of mistake that's obvious in hindsight and invisible in the moment.

Once I imported daily csv with the correct ~39k rows, the same question returned a proper breakdown. Campaign names, spend, impressions, clicks — exactly what the marketing team needed.

What Wren AI Actually Does Well

After going through all of this, here's my honest take:

It respects your existing data model. You're not reinventing your schema for Wren AI. It sits on top of what you already have. The relationships you define mirror what you'd build in any BI tool — it just adds a natural language layer on top.

The semantic layer is engineer-controlled. Business users ask questions, but they're asking against a model you built and you validate. The data engineer still owns the semantics. That matters for accuracy.

It genuinely removes the middleman. Once it's working, a non-technical user can type a question and get a real answer in seconds. That's not hype — it actually works, and it's a meaningful shift in how fast insight moves through a team.

Final Thought

The actual data engineering here wasn't complicated. Star schema, surrogate keys, foreign key relationships — standard stuff. The interesting part was watching what happened after the plumbing was in place.

The marketing team stopped waiting on me. They started getting answers in seconds. The data was finally reaching the people who needed it, at the speed they needed it.

That's the goal, right? The pipeline isn't the point. The pipeline is what makes the point possible.

Have you used Wren AI or a similar semantic layer on top of a warehouse? I'd love to hear how it went drop a comment below.

The Core Paradigms of Machine Learning

Tue, 15 Sep 2026 08:11:53 +0000

Introduction

Most machine learning explanations start with algorithms. In this blog, let's start with a simpler question: how does anything learn?

At its core, machine learning isn’t magic; it’s a set of strategies for turning data into behavior. Sometimes we guide the system with explicit answers. Sometimes we let it uncover structure on its own. And sometimes we let it learn through feedback, improving with every decision it makes.

In this article, we will walk through the four core learning paradigms: supervised, unsupervised, self-supervised, and reinforcement learning, not as abstract theory, but as intuitive learning patterns you have already seen in the real world. By the end, you won’t just recognize the terms; you will understand why these approaches exist, when they matter, and how they shape modern AI systems.

Defining the Paradigms

Learning paradigms are the fundamental ways an algorithm processes data to find patterns. They are not just mathematical functions; they are distinct philosophical approaches to problem-solving.

The Four paths to pattern recognition

Each paradigm dictates exactly what kind of data the machine needs and how independently it can operate.

1. Supervised Learning: The Flashcard Method

The Child: You show a child a card with a picture of a 🍎 and the
word “Apple” written on it. You repeat this until they can see the
shape and say the word.
The Machine: We give the model a Dataset (the pictures) and Labels
(the word “Apple”). The model calculates a mathematical function to
map the input to the correct output.
Use case: This is used for Classification (Is this email spam or
not?) and Regression (Predicting the exact price of a house).

Ground Truth: The model relies entirely on the “Teacher” being 100% correct on the labels provided.

2. Unsupervised Learning: Discovering on Your Own

The Child: A child is given a bucket of colored blocks 🟦🧱🟡.
Without being told to, they might start putting all the red ones
together and all the blue ones together.
The Machine: The model looks for Features (color, shape, size) and
calculates the “distance” between data points. Things that are
“close” together get grouped into Clusters.
Use case: This is used for Clustering (Grouping similar news articles
together) and Anomaly Detection (Finding a credit card transaction
that looks "weird" or "different" from your usual spending).

Clustering: Finding underlying structure in chaotic data when you don’t know what you are actually looking for yet.

3. Self-Supervised Learning: The Context Clue

The Child: You show a child a sentence with a word missing: “The cat
sat on the ___.” Based on their past experience, they guess “mat” or
“floor.”
The Machine: We take a massive amount of unlabeled data (like the
entire internet) and hide parts of it. The model tries to predict the
missing piece.
Use case: This is the "secret sauce" behind Large Language Models
(LLMs) like ChatGPT. It allows machines to learn from massive amounts
of data without humans having to label every single thing.

Pre-training: This is how "Brainy" models like GPT are built before they are taught specific tasks.

Reinforcement Learning: Learning from Consequences
The Child: A child tries to ride a bike 🚲. They wobble (penalty),
adjust their balance, and eventually move forward (reward).
The Machine: An Agent lives in an environment. It takes an Action,
and the environment gives it a Score (Reward). The machine’s only
goal is to maximize that score over time.
Use case: This is used for Robotics and Game AI (like AlphaGo), where
the machine needs to make a sequence of decisions to reach a goal.

Trial and Error: The model doesn't need to be told the "right" answer upfront, just whether its actions are moving it closer to the ultimate goal.

The Architect’s Decision Tree

How to choose the right paradigm for your specific problem?

Conclusion

Machine learning, at its core, is not about models; it’s about how learning is structured.

Supervised learning shows us the power of guidance. Unsupervised learning reveals how structure can emerge without labels. Self-supervised learning bridges the gap, turning raw data into its own teacher. And reinforcement learning pushes systems to learn through interaction, feedback, and long-term consequences.

These paradigms aren’t competing ideas; they are complementary lenses. Modern AI systems often blend them, moving fluidly from labeled data to pattern discovery to autonomous decision-making. Understanding this progression is what separates surface-level familiarity from true intuition.

If you step back, the pattern is strikingly simple: whether in machines or humans, learning evolves from instruction → exploration → self-discovery → adaptation.

And once you see machine learning this way, every model, every system, and every breakthrough becomes easier to reason about, not as complexity, but as a variation of these fundamental ways of learning.

The Engineering Debt Trap: Why "Working Code" Is Killing Modern Software Development

Mon, 31 Aug 2026 14:09:03 +0000

There is a quiet crisis unfolding in modern software architecture. On the surface, features are shipping faster than ever. Continuous Integration and Continuous Deployment (CI/CD) pipelines push updates dozens of times a day. Artificial intelligence tools auto-complete complex logic in milliseconds. Production dashboards gleam with green metrics.

Yet, behind this illusion of hyper-productivity, developer velocity across the industry is hitting a structural wall.

Teams that once built entire platforms in months now take quarters to roll out basic modifications. Simple bug fixes cascade into unintended regression outages. Senior engineers spend up to 70% of their time acting as archeologists—digging through layers of undocumented abstractions, legacy workarounds, and implicit dependencies just to understand where a state change occurs.

We have built a software ecosystem that prioritizes immediate delivery over structural longevity. We call this "technical debt," but that standard financial analogy fails to capture its true nature. Debt can be leveraged strategically to generate revenue. What most software teams accumulate isn't strategic debt; it is structural decay—a steady, compounding operational tax that eventually bankrupts engineer morale and system scalability alike.

If we want to build software that outlasts the quarterly planning cycle, we must radically rethink how we write, structure, and maintain complex codebases.

1. The Fallacy of "Working Code"

The foundational axiom taught in introductory computer science and bootcamp programs is simple: Does the program produce the expected output for the given input?

If the tests pass, the code is considered functional. If it is functional, it gets merged.

This framing is dangerous. Writing software that executes correctly on a machine is the easiest part of software engineering. The true challenge—the discipline that separates junior coders from staff architects—is writing software that remains understandable and adaptable to human minds over years of continuous mutation.

+-----------------------------------------------------------------+
|                        THE CODE LIFECYCLE                       |
|                                                                 |
|   [ Written Once ]  --->  [ Read 100x ]  --->  [ Modified 20x ]  |
|          |                                            |         |
|          v                                            v         |
|   Machine Execution                           Human Cognition   |
|   (Cheap & Fast)                              (Expensive & Slow)|
+-----------------------------------------------------------------+

Code is read far more often than it is written. When an engineer writes a complex nested conditional block or relies on implicit side effects to save twenty lines of code today, they save twenty minutes of their own time while burning hundreds of hours of collective cognitive bandwidth for every developer who touches that module in the future.

The Cost of Cognitive Overhead

Human working memory is strictly constrained. Cognitive science shows that the average human mind can hold roughly 4 to 7 operational chunks of information in working memory at any given moment.

When reading code, an engineer must construct a mental model of the system’s execution context:

What is the current state of this object?
What implicit global variables exist?
Which threads or async event loops can mutate this state concurrently?
What side effects will invoking this function trigger down the dependency chain?

If understanding a single function requires keeping 12 different environmental variables in mind simultaneously, the engineer's working memory overflows. To compensate, they slow down, make educated guesses, or accidentally introduce bugs.

Unmaintainable code isn't code that fails to run; it is code that overloads human cognitive capacity.

2. Microservices, Abstraction Layers, and Premature Architecture

Over the last decade, the industry attempted to solve codebase complexity through architectural segregation: enter the microservices paradigm and heavy abstraction patterns.

Instead of one monolithic codebase, organizations split their logic across dozens—or hundreds—of isolated micro-repositories. The promise was alluring: independent deployments, isolated failure domains, and small, manageable codebases.

In reality, many engineering organizations simply traded internal code complexity for distributed network complexity.

MONOLITHIC COMPLEXITY                  DISTRIBUTED COMPLEXITY
+-----------------------+              +------+   HTTP   +------+
| Class A -> Class B    |              | Svc A| -------->| Svc B|
|   (Direct Call,       |              +------+          +------+
|    Compile Time)      |                 |                 |
|                       |                 v Async Event     v Database
| Class C -> Class D    |              +------------------------+
+-----------------------+              |   Kafka / RabbitMQ     |
                                       +------------------------+

The Microservice Mirage

When you break a monolith into microservices without clear bounded contexts, you do not eliminate complexity; you export it onto the network interface.

Function calls become network RPCs subject to latency, jitter, and partial network partitions.
In-memory ACID transactions are replaced by saga patterns, eventual consistency, and complex distributed rollback mechanics.
Debugging a single trace requires aggregating logs across twenty disparate observability tools instead of setting a breakpoint in an IDE.

If two microservices must always be deployed together, or if changing an API contract in Service A requires synchronized, breaking code updates in Service B, you do not have microservices. You have a distributed monolith—combining all the performance overhead and operational friction of microservices with the tight coupling of a legacy monolith.

The Abstraction Tax

A similar failure mode occurs at the code level through premature abstraction. Driven by principles like DRY (Don't Repeat Yourself), developers routinely create generic helper functions, base classes, and abstraction layers before fully understanding the problem space.

Duplicate code is far cheaper than the wrong abstraction.

When you prematurely abstract two pieces of logic that look identical today but evolve for different business reasons tomorrow, you force those distinct domains to share an increasingly convoluted set of if/else flags within the generic abstraction. Over time, the abstraction becomes a brittle monstrosity that nobody dares to touch.

3. The Architecture of Maintainability: Principles for Long-Term Value

How do we construct software systems that resist decay? Maintainability is not an accident; it is the deliberate result of specific design trade-offs.

+-----------------------------------------------------------------+
|                    SUSTAINABLE CODE FOUNDATIONS                 |
|                                                                 |
|  [ Explicit > Implicit ]    [ Localized State ]   [ Low Coupling ]|
|             \                      |                     /      |
|              v                     v                    v       |
|               HIGH-VELOCITY, LONG-LIVED ARCHITECTURE            |
+-----------------------------------------------------------------+

I. Explicit Over Implicit

Avoid magic. Reflection, auto-wiring dependency injection frameworks that obscure implementation locations, global dynamic state, and monkey-patching make code visually brief at the cost of operational clarity.

Code should tell a direct, clear story. A reader should be able to trace execution paths by following explicit function calls and types without relying on runtime surprises or framework magic.

II. Localize State and Embrace Immutability

Shared mutable state is the primary root cause of concurrency bugs, race conditions, and unpredictable side effects.

When state changes arbitrarily across an application, reasoning about system behavior requires reading every single line of code that could potentially modify that state.

Prefer Immutable Data Structures: Treat data as facts that cannot be retroactively altered. When state changes, produce a new representation of the data rather than mutating the existing object in place.
Isolate Side Effects: Keep your core business logic pure (inputs produce outputs with zero external side effects) and push I/O operations (database writes, API network requests, disk access) to the outer edges of your application architecture.

III. High Cohesion and Low Coupling (Real Bounded Contexts)

Modules should be grouped by business domain capability, not by technical abstraction layers.

Bad Layered Architecture: Placing all Controllers in one folder, all Services in another, and all Models in a third. Modifying a single feature requires jumping across three distinct directory trees.
Good Domain-Driven Package Structure: Grouping all logic related to Payments inside a self-contained module. The Payments module exposes a narrow, public API interface while keeping its internal storage schemas, helper utilities, and processing steps strictly internal and private.

DOMAINS OVER LAYERS

  Layer-First (Fragmented)             Domain-First (Cohesive)
  ├── controllers/                     ├── payments/
  │   ├── UserController.js            │   ├── PaymentService.js
  │   └── PaymentController.js         │   ├── PaymentRepository.js
  ├── services/                        │   └── PaymentApi.js
  │   ├── UserService.js               └── users/
  │   └── PaymentService.js                ├── UserService.js
  └── models/                          └── UserRepository.js
      ├── UserModel.js
      └── PaymentModel.js

4. Culture Shift: Refactoring as a Daily Discipline

You cannot fix structural code rot with a dedicated "refactoring sprint" once a year. Technical debt is like interest on financial debt; if left untended, the interest payments swallow your entire operational capacity.

Engineering organizations that maintain high velocity over decades view refactoring not as a project stage, but as a continuous habit integrated into daily development.

The Boy Scout Rule

Leave the codebase cleaner than you found it. If you open a source file to add a feature or fix a bug, perform one small cleanup task before submitting your pull request:

Rename an ambiguous variable.
Break a 100-line function into three small, pure helper methods.
Add a missing test case covering an edge condition.
Delete dead code or outdated comments.

These micro-improvements take minutes, but over the course of thousands of pull requests, they compound into massive structural maintenance gains.

Reframing Engineering Priorities to Leadership

Product managers and business executives rarely push back on code quality out of malice; they push back because engineers frame quality in technical terms rather than business metrics.

When advocating for architectural investments:

Don't say: "We need to rewrite this module using modern reactive patterns because the current code is spaghetti."
Do say: "The current architecture causes 12 hours of weekly developer friction and accounts for 40% of our production regression bugs. Restructuring this domain will reduce feature delivery timelines for upcoming roadmap items by an estimated 25%."

Conclusion: The Craftsman's Mindset

Software engineering is fundamentally an iterative discipline. The systems we design today will inevitably face pressures, scale constraints, and business requirements that we cannot foresee.

We cannot predict the future, but we can design for change.

Writing sustainable code requires stepping away from short-term shortcuts and rejecting the false metric of raw volume production. It demands that we view our code not merely as instructions executed by machines, but as a living human communication medium.

When you write your next line of code, don't ask yourself simply "Does this work?"

Ask yourself: "When a colleague opens this file at 2:00 AM three years from now to fix a critical production outage, will this code guide them to a clear solution, or trap them in an avoidable maze?"

Building great software isn't just about solving problems today. It is about ensuring the system remains empowered to solve the problems of tomorrow.

What are your biggest pain points with technical debt in your current codebase? How does your team balance feature velocity with long-term code quality? Share your experiences in the comments below!

How Open Embedding Models Can Match GPT-Class Retrieval Quality

Thu, 06 Aug 2026 09:09:21 +0000

RAG (Retrieval-Augmented Generation - a pattern where you fetch relevant documents before generating an answer) lives or dies on retrieval quality. A cheaper retrieval stack isn't always a compromise.

The Retrieval Quality Gap Is Often in the Chunking, Not the Model

Most teams assume the embedding model is the bottleneck when retrieval underperforms. The real culprit is usually upstream: how documents are split before they're embedded. Overlapping chunks with a recursive character splitter, combined with a well-tuned open embedding model like bge-large-en-v1.5 or nomic-embed-text, can close a surprising amount of the gap against expensive proprietary embeddings - often without touching the LLM layer at all.

The pattern that tends to work: chunk at a meaningful boundary (paragraph or sentence, not fixed character count), add a small overlap window (10 - 15% of chunk size), then embed with a model that scores well on the MTEB benchmark (Massive Text Embedding Benchmark - a standardized leaderboard for comparing retrieval and semantic similarity models). Many open models in the 1B parameter range sit competitively on MTEB against models costing 100x more per token to call via API.

Real Example

Here's a minimal retrieval setup using LangChain with a local open embedding model via Ollama:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512, chunk_overlap=64, separators=["\n\n", "\n", ". "]
)
docs = splitter.create_documents([your_text])

embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(docs, embeddings)

results = vectorstore.similarity_search("your query here", k=4)

Swapping nomic-embed-text for bge-large-en-v1.5 (via HuggingFace) takes one line change. Running both against your actual dataset on a held-out set of queries - not a synthetic benchmark - is the fastest way to know which one fits your domain.

The cost difference is real: embedding a million tokens locally via Ollama is effectively free after the initial model pull, versus measurable per-call costs at API scale.

Key Takeaways

Chunking strategy (boundary type + overlap) affects retrieval quality at least as much as the embedding model choice - fix this before swapping models.
Open embedding models in the 1B range score competitively on MTEB and run locally or on modest GPU instances.
Evaluate retrieval quality on your own queries, not just published benchmarks - domain-specific retrieval patterns differ from general benchmarks.

If you've swapped embedding models in a production RAG pipeline, which metric did you actually use to confirm the quality change - cosine similarity threshold, end-task accuracy, or something else?

Sources referenced: HackerNews discussion - "Beating GPT-5.6 Sol on retrieval with 100x cheaper open models", MTEB Leaderboard (Hugging Face)

🚀 Open Sourcing etl-pipeline-mcp: A Local-First, Decoupled Data Pipeline for AI Agents

Tue, 28 Jul 2026 15:09:11 +0000

Hey CoderLegion community,
When connecting LLMs or AI agents directly to production databases or third-party APIs (like HubSpot, Stripe, or GitHub), two major problems usually pop up:
Unsanitized PII/PHI leaks right into the model's context.
Fragmented identities when events come from multiple webhooks and CRMs.
To solve this, I’ve built and released etl-pipeline-mcp — a local-first Model Context Protocol (MCP) server designed to handle high-throughput extraction, transformation, identity resolution, and data warehouse loading without relying on third-party SaaS middle layers.
⚙️ Key Architecture & Features
Decoupled Hot/Cold Storage:
Hot Tier (RAM): High-speed in-memory SQLite buffer to handle high-velocity, real-time webhooks with backpressure throttling.
Cold Tier (Disk): Persistent, on-disk SQLite archive for long-term storage and analytical queries.
Automated PII Redaction & Compliance:
Hashes sensitive fields (emails, IP addresses, customer IDs) using SHA-256 before saving to disk or exporting.
Built-in alignment for GDPR, CCPA, and HIPAA requirements.
Identity Resolution (ID Stitching):
Stitches 5+ platform-specific IDs (Stripe Customer ID, GitHub User ID, HubSpot Lead ID, Social Handles, Email) into a single, unified mcp_root_id.
Multi-Warehouse & Data Lake Exports:
Formats and exports clean records directly to Snowflake, ClickHouse, or BigQuery.
Generates columnar JSON-Parquet buffers for local DuckDB or S3 Data Lake ingestion.
🛠️ Installation & Usage
You can install it directly via PyPI or run it with uv:

Install via PyPI

pip install etl-pipeline-mcp

Or connect it to Claude Desktop or Cursor in claude_desktop_config.json:
{
"mcpServers": {

"seosiri-etl-pipeline": {
  "command": "uv",
  "args": [
    "run",
    "--github",
    "SEOSiri-Official/etl-pipeline-mcp",
    "src/main_server.py"
  ]
}

}
}

🔗 Links & Resources
PyPI Package: https://pypi.org/project/etl-pipeline-mcp/
GitHub Repository: https://github.com/SEOSiri-Official/etl-pipeline-mcp
Full Architecture Guide: https://www.seosiri.com/2026/07/etl-pipeline-mcp.html
Would love to hear your feedback or ideas for extending the transformation layer!

Deploying My First Ubuntu Virtual Machine on Microsoft Azure: SSH Access, Apache2 Installation, and Web Server Configuration"

Thu, 16 Jul 2026 12:51:23 +0000

Introduction

Microsoft Azure Virtual Machines (VMs) provide scalable cloud computing resources that allow users to deploy and manage operating systems in the cloud. In this project, I created an Ubuntu 22.04 Virtual Machine in Azure, connected to it using SSH from both Git Bash through the Visual Studio Code, updated the server, installed Apache2, configured network security settings, and deployed a custom web page accessible through a public IP address.

This hands-on lab helped me gain practical experience with Azure Infrastructure as a Service (IaaS), Linux administration, SSH connectivity, and web server deployment.

Project Objectives
Create an Ubuntu 22.04 Azure VM
Download and secure a .pem private key
Connect to the VM using Git Bash
Update and upgrade Ubuntu packages
Install and verify Apache2
Create a custom web page
Access the website through a browser

Prerequisites
Active Azure Subscription
Azure Portal access
Git Bash installed
Internet connection

Step 1. Log In to Azure Portal

On your web browser, head over to https://portal.azure.com and log in with your Microsoft account. If you don't have an account, create or sign up and provide the required information so your Microsoft account can be activated.

A successful login bring you to this page below,

__Step 2: Create a Resources Group
__
We need to create our Resources Group first before we go ahead to create our virtual machine
. Search for Resource Group in the search bar at the top of your portal

page and Select/Click on Resource Group.

Click on Create to create Resource Group

Type the Resource group name, select the region and click Next
Apply tags to your resources to logically organize them by categories and click Next

Review the resources group and click on Create

Click on Resource Group on the left to bring out the resource group you created

Step 2. Create a New Virtual Machine
Type "Virtual Machines" in the search bar at the top. Once you see virtual machine display click on the virtual machine

Once you click on the virtual machine, your screen will be displayed as shown below. Click on CREATE

When you click on "Create" button, it will display a drop down as shown below. You will then click on the Azure virtual machine highlighted in red arrow. This will take you to the project details where you will begin to fill the basic configuration for you to start creating your virtual machine.

Create virtual Machine under the BASIC CONFIGURATIONS by filling the following tabs as follows:

_Subscription: Select your subscription(Azure Subscription 1)
Resource Group: Use an existing or create a resource group (FlackernProj-rg)
Virtual machine name: flackern-vm or use your desired username
Region: Select a nearby or preferred location ((Africa) South Africa North)
Availability options: Leave it as default (Availability Zone)
Availability Zone: Zone 1 (For this demo we select only one zone)
Security type: Standard
Image: Ubuntu Server 24.04 LTS - x64 Gen2
VM architecture: x64
Size: Standard_B2as_v2 - 2 vcpus, 8 GiB memory ($71.32)
Authentication type: SSH public key
Username: Use your desired username
SSH public key source: Generate new key pair
SSH Key Type: RSA SSH Format
Key pair name: flackern-vmkey or use your desired username
Public inbound ports: Allow selected ports
Select inbound ports: HTTP (80) and SSH (22)

Step 3. Configure Disks

Click on Next : Disks >, this leads you to on the next page, you can use the default OS disk size, use the Image default (30 GiB) and for the OS disk type, use the default settings Premium SSD (locally-redundant storage) unless you have unique configuration requirements.

Step 4. Networking Configuration

On the bottom of the last page in Step 3, Click on Next : Networking >, this leads you to the Network interface page below:

Create your virtual network and subnet or make use of the one you have created before

Management, monitoring, Advanced, Tags: These pages usually displayed as default. Find their default screenshots below. However, you may wish to skip them and proceed to “Review+Create”.

Step 5. Generate SSH private Key
After a successful Linux VM deployment process, Azure will validate the configuration

_Then Click "Review + create". This will show validation passed as highlighted below.

_Then click “Create” . This will take you to the next page to generate new key pair. Then, you will click on “Download private key and create resource” as shown below. The new key pair will then be downloaded on your PC, while Azure will also provision your VM.
__

Download private key and create resource, afterwards, the deployment will start.

Wait for Deployment: Azure will now provision your virtual machine. This process may take a few minutes. Then the next page will show that your deployment is complete as shown below, then you will click on “Go to Resource”

Next, click on “Connect” and then “Connect”. This will take you to a page with other options to connect your VM.

Retrieve Public IP Address: Once the virtual machine is deployed, note down the Public IP address on Azure portal overview page.

Note: Also go to your download and save the downloaded key pair into a file for later use when connecting via SSH.

Step 6: SSH into VM Using Git Bash -Connect via SSH Key Authentication:
Open your Command Line Interface on Windows or Terminal on your Visual studio code using gitbash or power shell
Move the .pem file into a secure folder.
Set permissions:
chmod 600 flackern-vm_key.pem

Step 7: SSH into VM Using Azure Cloud Shell
Connect:
ssh -i flackern-vm_key.pem flackern-vm@102.37.144.183

Step 8: Update Ubuntu Packages
Using sudo apt update and click enter.This will update our virtual machine

Step 9: Install Apache2
Instal apache2 on our VM and click enter
Install:
sudo apt install apache2 -y

Now let us confirm if apache2! Was actually installed on the virtual machine, we can do this by going to your Azure site and copy the ip address and paste it on the web page and see the result.

Copy ip address

Test Your Web Server
Open a web browser and enter the public IP address you obtained earlier. If apache2 is running correctly, you should see a default welcome page.
Congratulations! You have successfully created a Linux virtual machine on Microsoft Azure and installed the Apache2 web server. This is a foundational step for hosting web applications or deploying other web services.

When "Latest" isn't Actually Latest

Thu, 02 Jul 2026 11:08:55 +0000

When "Latest" Isn't Actually Latest

TL;DR

In distributed systems, the last record your system processes is not always the most recent event that happened in reality.

Understanding the difference between Event Time and Processing Time is essential to avoid data quality issues in streaming pipelines, CDC architectures, and event-driven systems.

The Assumption That Works... Until It Doesn't

Most developers start their careers working with a single application and a single database.

In that world, time feels simple.

A customer updates their profile.
The database stores the change.
Everyone sees the updated record.

The sequence appears obvious because all operations happen within the same system.

Then distributed systems enter the picture.

A Simple Example

Imagine a customer updates their address at 10:00:00.

That change generates an event and begins traveling through a distributed architecture:

Application
Database
CDC pipeline
Kafka
Consumer service

Everything looks normal.

Now imagine an older address update, generated earlier, gets delayed because of a network issue or retry mechanism.

The result?

The older event arrives after the newer one.

Your consumer processes the events in this order:

New address
Old address

If your logic assumes that the last processed record represents the latest truth, you've just overwritten correct information with stale data.

The Real Problem: Time Has Multiple Meanings

This is why distributed systems distinguish between two important concepts:

Event Time

The moment something actually happened in the real world.

Example:

Customer changed address → 10:00:00

Processing Time

The moment a system receives or processes the event.

Example:

Consumer receives event → 10:00:15

Most of the time, these timestamps are close enough that nobody notices.

The interesting cases appear when they aren't.

Where Things Get Complicated

Several factors can cause events to arrive out of order:

Network latency
Retries
Consumer lag
Parallel processing
System failures
Distributed transactions
CDC replication delays

As architectures become more distributed, these situations become normal rather than exceptional.

What About Kafka?

Kafka helps preserve ordering within a partition through offsets.

However, offsets tell us the order in which records were written to Kafka.

They do not guarantee the order in which events actually occurred in the real world.

That's an important distinction.

A perfectly ordered Kafka partition can still contain events that represent out-of-order business actions.

Why Data Engineers Care

Many expensive data quality problems are not caused by broken infrastructure.

Pipelines may be healthy.
Kafka may be running perfectly.
Consumers may be processing records successfully.

The issue is often more subtle.

The system answers the right question using the wrong sequence of events.

When that happens, dashboards become inaccurate, customer profiles become inconsistent, and downstream decisions become unreliable.

Final Thought

Distributed systems force us to rethink a concept we usually take for granted: time.

The challenge isn't simply moving data from one system to another.

The challenge is preserving the meaning of that data while it travels through a network of independent components.

Because in distributed systems, the most recent record is not always the most recent event.

And understanding that difference is often the first step toward building reliable data systems.

The Data Engineer Roadmap for 2026 (in an AI-Native World)

Sat, 27 Jun 2026 09:09:07 +0000

This is the narrated version of our free, interactive Data Engineer Roadmap. Same areas, same order, with a focus on the one thing each layer asks of you that AI can't do for you.

Every data engineer roadmap written before early 2025 made the same quiet assumption: that the hard part was writing the code. Learn SQL. Learn Python. Wire up a pipeline in Airflow. Ship it. Congratulations, you're a data engineer.

That assumption is dead. AI writes the SQL now. It writes the DAG, the PySpark job, the dbt model, the masking policy - and it writes them faster than you, at 2am, without complaining. If your roadmap is a checklist of tools to learn so you can produce that code, you're training for a race that has already been run.

So a 2026 roadmap has to be a different shape. Not "what do I learn so I can write a pipeline," but "what do I understand so I can tell whether the AI-written pipeline is right, and fix it when it isn't." That is a map of depth, not a list of tools.

The one idea that makes the whole map work

Most roadmaps draw becoming-senior as new areas appearing: the junior does SQL and dbt, the senior does Kafka and Spark and Kubernetes. That is not how it works.

A senior engineer works the same areas a junior does. The difference is how far into each one they go.

A junior knows Parquet is "the fast columnar format" and can partition a table. A senior reasons about row groups, page statistics, dictionary encoding, and why a scan cost what it cost. A junior writes a Spark job. A senior debugs its shuffle and its skew. Same topic, different altitude.

That matters more now than ever, because AI raises the floor to roughly the junior line. It reliably gets you the partitioned table and the working Spark job. The depth above that line is exactly the part it can't reason about for you, and exactly where your career value now lives.

So as we walk the areas, watch for the pattern: AI does the surface; you own the depth.

Foundations and SQL

Joins, window functions, CTEs, Python, the command line, Git, ETL vs ELT.

AI writes almost all of this now. That doesn't make SQL optional, it makes it table stakes. You learn it not to produce it, but to catch when the generated query is quietly wrong: the join that fans out and double-counts revenue, the WHERE that silently drops NULLs, the window frame that is off by one row. The senior depth is reading an EXPLAIN plan and knowing why a query is slow. AI hands you the query; understanding it is still yours.

Data modeling and transformation

Dimensional modeling, star and snowflake schemas, fact vs dimension tables, dbt models and tests. Then the depth: Slowly Changing Dimensions, grain, conformed dimensions, the One Big Table pattern, Data Vault.

AI drafts the model. What it can't do is the judgement calls: what is the grain of this fact table, what does "one customer" mean across three source systems, which dimension is conformed across marts. The classic trap is Slowly Changing Dimensions - everyone can recite the types, almost nobody internalizes which version of a dimension their facts join to. Get it wrong and "revenue by region last quarter" reports a number that was never true.

Replay a change timeline yourself in the free, in-browser SCD Playground, then practice the area in the Dimensional Data Modeling track.

Orchestration and pipelines

Airflow DAGs, scheduling, sensors, backfills, retries, idempotency. Senior: scheduler and executor internals, data-aware scheduling, lineage, freshness SLAs, being on-call.

AI generates the DAG, and it is good at it. What it doesn't generate is the understanding of failure modes the job actually requires, because the real work here isn't the happy path, it's the 3am page. Why did this task hang? Why did the backfill double-write? Is this retry safe, or did it just send the same email twice? Idempotency is a property you reason about, not a snippet AI sprinkles in. See the Orchestration and Pipelines track.

Storage and file formats

Parquet, row vs columnar, compression, object storage, partitioning. Senior: row groups, page statistics, predicate pushdown, encoding, the small-file problem, the internals of ORC, Avro and Arrow.

This is where AI is least useful and depth pays the most, because why a scan costs what it costs is a property of the bytes on disk, not the query text. AI reads and writes Parquet fine. It can't tell you why two files with identical rows differ tenfold in scan cost - that is row group sizing, encoding choice, and whether min/max statistics let the engine skip pages.

Point the free Parquet Viewer at your own files (100% in-browser, nothing is uploaded) to see the row groups and statistics yourself. Track: Storage and File Formats.

Data lakes and table formats

Lake vs warehouse vs lakehouse, Iceberg and Delta, time travel, schema evolution. Senior: ACID and snapshot isolation internals, compaction, catalogs, the Iceberg-vs-Delta-vs-Hudi tradeoffs.

AI scaffolds the table operations happily. The part that bites, and that it won't warn you about, is what happens when two writers commit at once. Snapshot isolation, optimistic concurrency, conflict resolution, compaction fighting your ingest job: that is distributed-systems reasoning, not autocomplete. Track: Open Table Formats.

Ingestion and streaming

Batch ingestion, Kafka basics, producers and consumers, event time vs processing time. Senior: exactly-once semantics, consumer group rebalancing, Change Data Capture, stream processing in Flink or Kafka Streams.

AI writes the producer and the consumer. Where it goes quiet is where data-quality bugs are actually born: the gap between event time and processing time that makes your windowed aggregates wrong, the rebalance that reprocessed a batch, the "exactly-once" guarantee that was only ever at-least-once because of how you committed offsets. Track: Ingestion and Transport.

Distributed compute

Spark DataFrames, transformations vs actions, lazy evaluation. Senior: shuffle and partitioning, broadcast joins and data skew, Catalyst and codegen, memory and fault tolerance.

AI writes the transformation. It cannot tune the execution. Why did this job spill to disk? Why is one task taking 40 times longer than the other 199 (hello, data skew)? Should this join broadcast or shuffle? That reasoning, about how a logical DataFrame becomes physical work across a cluster, is squarely yours. Track: Compute Engines.

Query engines and OLAP

What OLAP is, warehouse vs query engine, ClickHouse, Trino. Senior: MergeTree and projections, federation and pushdown, execution models, cost-based optimization, real-time OLAP, EXPLAIN literacy.

AI writes the SQL the dashboard runs. Why that dashboard is slow, and how to fix it at the engine rather than by rewriting the query, is senior work. It lives in how the engine sorts and merges data, what it can push down, and what its optimizer chose. Track: Query Engines and OLAP.

Semantic and metrics layer

Metrics and dashboards, the semantic layer, data-quality tests. Senior: data contracts, schema registries, metric governance, reverse ETL.

AI drafts a metric definition. What it can't do is the organizational work of making "revenue" mean exactly one thing across finance, sales and product. That is a human contract - negotiated, governed, enforced - and it is the layer where data finally becomes shared business language instead of seven conflicting spreadsheets.

Governance, quality and cloud

PII basics, GDPR and CCPA, cloud, CI/CD for data. Senior: masking and tokenization, row and column access control, right-to-erasure across a lakehouse, infrastructure as code, data observability at scale.

AI flags the obvious PII column. What it can't design is right-to-erasure across a lakehouse with time travel and immutable snapshots - that is architecture, not autocomplete. The masking itself is full of guarantee-breaking gotchas: an unsalted hash is a lookup table, a redacted ZIP that keeps five digits still re-identifies people. Generate the DDL with the free PII Masking Policy Generator. Track: PII and Data Governance.

So, will AI replace data engineers?

It raises the floor and moves the value up.

AI now does the old junior checklist well: queries, DAGs, glue code, boilerplate pipelines. What's left for you is the durable part - reasoning about the system. Why a scan costs what it costs. What happens when two writers commit. Why a job spilled to disk.

AI doesn't replace the engineer who understands that depth, it gives them leverage. They direct the AI through the surface work and spend their judgement on the part it can't reach. The engineer who only knew the surface is the one under pressure now, because the surface is free.

That is the whole premise of the map. You touch every area early - junior and senior work the same areas. What stretches out over a career is how deep you go into each, and the deep end is precisely the part AI can't shortcut for you.

Open the full interactive Data Engineer Roadmap to see every topic on a single timeline, with a "Going senior" toggle that reveals the depth in each layer. Then if you want to practice that depth on real engines instead of slideware, that is what the curriculum and the free in-browser tools are for.

Originally published on the Petascale Labs blog.

The Story Behind Apache Iceberg's Format-Version: v1 to v4

Sat, 27 Jun 2026 09:08:20 +0000

Most engineers meet Apache Iceberg as a one-line answer: "it's the thing that
gives you ACID transactions and time travel on object storage." That's true, and
it's also where most people stop. But Iceberg has a version dial baked into every
table - a single integer called format-version, currently 1 through 4 - and
each turn of that dial is a chapter in a single, surprisingly coherent story.

It's a story about pushing transactional behavior down through layers. v1 made a
table atomic on top of immutable files, but it could only append and overwrite
whole files. v2 pushed correctness down to the row. v3 pushed identity down to
the row, so a row keeps the same id across compactions. v4 turned around and
refactored the metadata itself so it can scale and move without rewrites.

To follow that arc you need the one mental model the rest of this post hangs
on. So we start there, then walk the four versions in order, keeping the parts
that actually matter for understanding what's on disk.

The primer: an Iceberg table is a tree of files, not a folder

The instinct everyone brings from Hive is that a table is a directory: point at
a path, list the files under it, that's your table. Iceberg breaks that instinct
on purpose. An Iceberg table is not a directory of data files. It is a
tree of immutable metadata files rooted at a single pointer held in a
catalog.

Every commit appends new files to that tree and atomically moves the pointer.
Nothing is ever mutated in place. That single design choice - immutable files,
one movable pointer - is what makes everything else possible.

The tree has five layers:

Read it top to bottom and you have the whole format:

Layer 0 - the catalog. It holds one fact per table: "the current pointer
for prod.db.sales is this metadata file." The catalog is the only mutable
thing in the entire system, and it lives outside the format spec (REST
catalog, Hive Metastore, Glue, JDBC, and so on).
Layer 1 - the table metadata JSON. A new one is written on every commit.
It holds the schemas, partition specs, sort orders, table properties, the list
of snapshots, and named references. This is the file the catalog points at.
Layer 2 - the manifest list. One per snapshot. An Avro file listing the
manifests that make up that snapshot, each row carrying a partition summary so a
scan can skip whole manifests it doesn't need.
Layer 3 - the manifests. Avro files, each an index over a set of data or
delete files, with per-column statistics for each file.
Layer 4 - the data. The actual Parquet/ORC/Avro files, plus delete files
and deletion-vector blobs.

Why build a table this way? Two reasons, and they're the whole pitch:

Atomic commits without coordination. Because everything below the catalog
is content-addressed by path and never mutated, readers and writers can build
files independently with no locking. The only contended operation in the
entire system is the catalog pointer swap. One compare-and-set, and the commit
is either visible or it isn't.
Cheap reads. A scan loads the manifest list (roughly 100 KB even for a
huge table), prunes manifests by partition, and only then opens any data file.
You never list a directory; you walk an index.

On disk, a table looks roughly like this:

s3://warehouse/db/sales/                 <- table location
  metadata/
    00001-....metadata.json              <- Layer 1: root JSON, one per commit
    00002-....metadata.json
    00042-....metadata.json              <- currently pointed at by the catalog
    snap-....avro                        <- Layer 2: a manifest list per snapshot
    8ef6-....avro                        <- Layer 3: manifests
    9b21-....avro
    stats-....puffin                     <- optional: table stats (NDV sketches)
  data/
    year=2025/month=06/00000-0-....parquet   <- Layer 4: data
    year=2025/month=06/00001-0-....parquet
    dv-....puffin                            <- Layer 4: deletion vectors

Hold that picture. Every version below is a change to one or more of these five
layers, and the format-version integer in Layer 1 is what tells a reader which
rules apply. A reader that supports up to version N will refuse to open a table
whose format-version is higher than N rather than silently misread it.

Now the four chapters.

v1: a table that's atomic, but only at the file

v1 is the foundation, and it nails the hard part: snapshot isolation and atomic
commits on immutable files. A v1 table already has schemas, partition specs,
snapshots, and time travel. If all you ever do is append data and occasionally
overwrite whole files, v1 is a complete, correct table format.

Its limits are exactly the things the later versions go after.

It can't delete or update a single row. The smallest unit v1 can change is a
file. Want to delete one row? Rewrite the entire file without it, then swap the
file in a new snapshot. This is copy-on-write, and on a wide table it means
rewriting gigabytes to remove a handful of rows.

Its metadata carries a few quirks that v2 had to clean up:

The schema and partition spec were stored as singular fields - one schema,
one spec - rather than a list of historical ones. A v1 metadata file has a
schema and a partition-spec; both are deprecated from v2 onward in favor of
schemas[] and partition-specs[].
Partition field IDs were not tracked explicitly. Implementations just
assigned them sequentially starting at 1000. That caused real ambiguity when
the same logical field got a different transform across specs - there was no
stable identity to tie them together.
A snapshot could embed its manifests inline as a manifests: [path, ...]
list instead of pointing at a separate manifest-list file. Convenient, but it
meant the per-snapshot bookkeeping that later inheritance rules depend on had
nowhere to live.
A few data_file fields existed that nobody needed: block_size_in_bytes,
file_ordinal, sort_columns. All removed in v2.

None of this makes v1 wrong. It makes v1 append-shaped. The entire push of v2
is to make the table mutable at the row, and to add the bookkeeping that makes
that safe.

v2: correctness pushed down to the row

v2 is the version most production tables ran on for years, because it's the one
that turns Iceberg from an append log into a real transactional table. Three big
ideas arrive together: row-level deletes, sequence numbers, and named
references.

Row-level deletes: merge-on-read

Instead of rewriting a file to remove rows, v2 lets you write a small delete
file that says "these rows in that data file are gone." The reader applies
deletes on the fly at scan time. This is merge-on-read, and it comes in two
flavors:

Position deletes - a list of (file_path, position) tuples: "row 42 and
row 1007 of this file are deleted." Precise, cheap to write, used by DELETE
and MERGE.
Equality deletes - a predicate on column values: "any row where
id = 12345 is deleted." These don't need to know where the row lives, which
makes them ideal for streaming upserts where you delete-then-insert without
reading the old file first.

To make this work, the format needed a way to say whether a file holds data or
deletes, and a way to order deletes against data. Both arrived in v2.

The `content` discriminator and sequence numbers

A v2 manifest, and each file it lists, now carries a content field. On the
manifest list it's 0 = data or 1 = deletes; a single manifest holds either
data files or delete files, never both, so scan planning can load all the
delete manifests first. On the data file itself, content is 0 = DATA,
1 = POSITION_DELETES, 2 = EQUALITY_DELETES.

The ordering problem is subtler. If a delete file says "delete id = 12345,"
which inserts of that id does it kill - the ones before it, or also the ones
after? v2 answers this with a monotonic sequence number assigned at commit
time and threaded through every layer:

The table metadata tracks last-sequence-number, bumped on each commit.
Each snapshot records its sequence-number.
The manifest list records each manifest's sequence_number and a
min_sequence_number (the smallest data sequence number among live files in
it).
Each manifest entry carries the file's sequence_number.

The rule that falls out: an equality delete applies to a data file only when
the delete's sequence number is greater than the file's (and they share a
partition). A position delete applies at equal-or-greater sequence. That's how
"delete then insert" does the right thing - the new insert has a higher sequence
number than the delete, so it survives.

Inheritance: why this is cheap

Here's the piece that surprises people. A manifest entry can leave its
snapshot_id, sequence_number, and file_sequence_number null in the
file, and the reader fills them in from the manifest list. Why bother? Because
it lets the same manifest file be reused across optimistic-retry attempts. When
a commit loses the compare-and-set race and has to retry with a new sequence
number, only the small manifest list needs rewriting - the manifests and data
files it points at are untouched.

Named references: branches and tags

v2 adds a refs map to the table metadata - named branches and tags pointing at
snapshots:

"refs": {
  "main":        { "snapshot-id": 8392648, "type": "branch" },
  "audit":       { "snapshot-id": 8000000, "type": "branch",
                   "min-snapshots-to-keep": 10, "max-snapshot-age-ms": 604800000 },
  "prod_release":{ "snapshot-id": 7281001, "type": "tag",
                   "max-ref-age-ms": 2592000000 }
}

main always exists; if refs is empty it implicitly points at
current-snapshot-id. Branches let you stage and validate writes off to the side
(write-audit-publish); tags pin a snapshot so expiration won't garbage-collect
it. Branches carry their own retention floor (min-snapshots-to-keep,
max-snapshot-age-ms); tags and non-main branches carry max-ref-age-ms. main
never expires.

The v2 metadata cleanup

v2 also formalized a lot of Layer 1. These fields became required:
last-sequence-number, current-schema-id, schemas, default-spec-id,
partition-specs, last-partition-id, default-sort-order-id, sort-orders,
and table-uuid (a stable identity generated at create time, used as a
refresh-time integrity check). The singular schema and partition-spec are
deprecated, the inline snapshot manifests list is gone, and partition field IDs
are now explicit and unique across all specs - fixing the v1 ambiguity.

One nice compatibility property: a v1 file reads cleanly as v2. A missing
sequence_number is read as 0, and a missing content is read as 0 (data). So
upgrading is a metadata-only operation; nothing has to be rewritten on day one.

By the end of v2, Iceberg is a full transactional table: insert, delete, update,
upsert, branch, tag, time-travel. So what's left for v3?

v3: identity, efficient deletes, and richer data

If v2 made the table transactional, v3 makes the row a first-class citizen.
It adds three things that don't fit neatly into v2's model: a stable identity for
every row, a far more efficient delete mechanism, and a richer type and security
surface.

Row lineage: every row gets a stable id

This is the headline feature, and it's genuinely clever because it touches three
layers at once without storing an id per row anywhere. v3 mandates that every row
has a stable _row_id that survives compaction - so you can track a row across
rewrites, build change feeds, and reason about lineage. It works by seeding,
not storing:

Layer	Field	Set when	Used for
1 (table metadata)	`next-row-id`	bumped per commit	seeds the next snapshot's first row id
2 (snapshot)	`first-row-id`, `added-rows`	at commit	starting `_row_id` for the manifest list
2 (manifest list)	`first_row_id` per manifest	at commit	starting `_row_id` for files in that manifest
3 (manifest entry)	`data_file.first_row_id`	at commit	starting `_row_id` for rows in that file
4 (data file)	reserved fields `_row_id`, `_last_updated_sequence_number`	inherited at read	stable identity across compactions

The reader computes a row's id with one formula:

_row_id = data_file.first_row_id + row_position_in_file

No per-row storage; the id is derived from where the row sits. If first_row_id
is null - say a v2-era file in a table that was upgraded to v3 - then _row_id
reads as null for those rows, which is exactly the honest answer. Equality
deletes deliberately break lineage: an equality-delete update never reads the old
row, so the replacement gets a fresh _row_id rather than inheriting one it
can't prove.

Deletion vectors: position deletes, done right

Position delete files worked, but they had a scaling problem: lots of tiny
delete files, each needing to be opened and merged. v3 replaces them with
deletion vectors (DVs) - a single compressed bitmap per data file, stored as
a blob inside a Puffin file. One bitmap, one referenced data file, looked up by
byte offset.

The manifest entry for a DV reuses the position-delete content code but adds
three fields:

manifest_entry {
  status = 1
  data_file {
    content              = 1                              // shares position-delete code
    file_path            = "s3://warehouse/db/sales/data/dv-....puffin"
    file_format          = "puffin"
    partition            = { country: "IN", ts_day: 2025-06-15 }  // same partition as target
    record_count         = 3                              // number of deleted positions
    referenced_data_file = "s3://.../00000-abc.parquet"   // v3 NEW: which data file this DV covers
    content_offset       = 4                              // v3 NEW: byte offset of the blob in the Puffin file
    content_size_in_bytes= 23                             // v3 NEW: blob length
  }
}

referenced_data_file, content_offset, and content_size_in_bytes are the new
fields that let the reader jump straight to one bitmap. Position delete files are
deprecated in v3: writers can't create new ones, and existing ones get merged
into DVs over time. The result is one delete artifact per data file instead of a
pile of small files.

New types and column defaults

v3 broadens what a column can hold. New primitive types: variant (semi-
structured), geometry and geography (spatial), unknown (a column whose type
isn't known yet), and nanosecond timestamps timestamp_ns / timestamptz_ns.

It also adds column defaults, which finally make adding a non-null column
sane:

{
  "id": 5, "name": "country", "type": "string", "required": false,
  "initial-default": "IN",   // value for rows in files written before this column existed
  "write-default": "IN"      // value to fill when a writer omits the column
}

initial-default is the value that existing rows get for a freshly added
column, with no file rewrite - the reader synthesizes it. write-default is what
new writes use when the column is omitted. Together they make schema evolution a
metadata change instead of a backfill.

The type-promotion rules that keep schema evolution safe also expand in v3:

From	v1 / v2 promotion	v3+ adds
`unknown`	-	promotable to any type
`int`	`long`	`long`
`date`	-	`timestamp`, `timestamp_ns` (not the tz variants)
`float`	`double`	`double`
`decimal(P, S)`	`decimal(P', S)` with `P' > P`	same

Partition transforms get multi-argument

v3 adds source-ids (plural) on partition fields, so a transform can take more
than one source column. Single-argument transforms still write the old
source-id. The full set of allowed transforms is identity, bucket[N],
truncate[W], year, month, day, hour, and void. And a forward-
compatibility rule lands: v3 readers must tolerate an unknown transform and
simply skip filter pushdown on it, rather than refusing to read. (v1/v2 only
should.) Writers, of course, still can't commit a transform they don't
understand.

Encryption arrives

v3 adds table-level encryption with a three-place key model:

"encryption-keys": [
  { "key-id": "k1",
    "encrypted-key-metadata": "BASE64...",   // KMS-wrapped data encryption key
    "encrypted-by-id": "kms-master-2025" }   // the logical key-encryption-key id
]

The table JSON holds encryption-keys[] - data encryption keys (DEKs) each
wrapped by a KMS-resident key-encryption-key (KEK).
Each snapshot carries a key-id naming which DEK protects that snapshot's
manifest-list key metadata.
Each file can carry per-file key_metadata (this field already existed in
v1/v2 on data files, but without a central registry).

The DEK-to-KEK chain is opaque to the format; implementations plug into AWS KMS,
GCP KMS, Vault, and so on via the wrapped bytes.

By the end of v3, a row has an identity, deletes are a single bitmap, columns can
default and hold variant or spatial data, and the table can be encrypted. The
user-visible feature set is essentially complete. Which is why v4 looks
different from everything before it.

v4: the refactor for scale and portability

v4 introduces no new user-visible types and no new delete mechanisms. It is a
metadata refactor aimed at three things: performance, portability, and richer
per-file statistics. The changes are quieter, but two of them matter a lot in
production.

Relative paths: move a table without rewriting it

This is the biggest invisible change in the whole format. In v1 through v3,
every path stored inside metadata - file_path, manifest_path, the manifest
list, metadata-file, statistics paths - had to be absolute, complete with a
URI scheme like s3:// or hdfs://. That meant the moment you wanted to move a
table to a different bucket, every one of those absolute paths was wrong, and you
had to rewrite the entire metadata tree to fix them.

v4 allows paths to be relative to the table location. The resolution rule is
simple:

Format	Table location	Stored path	Resolves to
v4	`s3://bucket/db/table`	`data/00000.parquet`	`s3://bucket/db/table/data/00000.parquet`
v4	`s3://bucket/db/table`	`hdfs://wh/...`	`hdfs://wh/...` (absolute, used as-is)
v3 and earlier	`s3://bucket/db/table`	`s3://bucket/db/table/data/00000.parquet`	unchanged

If a stored path has a URI scheme, it's absolute and used as-is. If it doesn't,
the reader resolves it as table_location + "/" + path. The writer rule:
default to relative for files under the table location, use absolute for files
outside it (say, a backfill from another bucket). Because location is now what
ties relative paths together, location in the table metadata JSON becomes
optional - the catalog can supply it.

The operational payoff is the headline: moving a table from
s3://bucket-a/db/sales/ to s3://bucket-b/db/sales/ needs only a catalog
pointer update and, optionally, a new metadata.json with the new location. No
manifest list, no manifest, no data file gets rewritten. Pre-v4 the same move
required a full metadata rewrite.

Typed `content_stats`: five maps become one struct

In v3 and earlier, per-column statistics on a data file were five parallel
maps keyed by field id: value_counts, null_value_counts,
nan_value_counts, lower_bounds, upper_bounds (plus the on-disk
column_sizes). Five maps to keep in sync, all loosely typed (bounds were raw
binary-encoded bytes).

v4 replaces them with one typed struct, content_stats, whose layout is
generated from the table schema itself. Each column reserves a block of ids and
gets a typed sub-struct:

146: optional struct content_stats {
  10_400: optional struct id (default null) {        // stats for table field 2 (int)
    10_401: optional int     lower_bound;
    10_402: optional int     upper_bound;
    10_403: optional boolean tight_bounds;
    10_404: optional long    value_count;
  }
  10_600: optional struct data (default null) {       // stats for table field 3 (string)
    10_601: optional string  lower_bound;
    10_602: optional string  upper_bound;
    10_603: optional boolean tight_bounds;            // v4 NEW: exact vs truncated min/max
    10_604: optional long    value_count;
    10_605: optional long    null_value_count;
    10_607: optional int     avg_value_size_in_bytes; // v4 NEW: variable-length sizing
  }
}

The id assignment is mechanical - each column reserves 200 ids - and two genuinely
new pieces of information appear: tight_bounds, a flag saying whether the
min/max are exact or truncated (truncated bounds still prune, but you have to scan
to confirm a match), and avg_value_size_in_bytes for variable-length columns,
which helps the planner estimate read cost. Spatial columns use typed
geo_lower / geo_upper structs instead of opaque WKB bytes.

The reassuring part: v3 and v4 statistics are equivalent. A missing map key in
v3 is the same as a missing-or-null sub-struct in v4. Nothing is lost in the
translation; it's the same information, typed and consolidated.

The file-system catalog is gone

v1 through v3 allowed a "file-system table": sequential metadata filenames
(v1.metadata.json, v2.metadata.json, ...) where a commit was an atomic file
rename. That only ever worked safely on HDFS, because object stores like S3
don't offer atomic rename. v4 removes it entirely. Every v4 table uses a real
catalog (the metastore model), where a commit is a compare-and-set on the catalog
pointer. This closes a long-standing source of silent corruption on object
storage.

That's the whole of v4: relative paths, optional location, typed
content_stats with tight_bounds and average value size, and the death of the
file-system catalog. A refactor, not a feature release - and exactly the kind of
change a format makes once its feature surface has settled.

How the layers earn their keep: scan planning

The reason all this structure exists is to make a scan cheap, so it's worth
watching a query actually use it. Given the current snapshot, a scan does
three-level pruning without ever needing a separate planner index:

Open the manifest list. One read, roughly 100 KB.
Layer-2 pruning. Drop any manifest whose partition summary can't match the
query predicate. Whole manifests skipped without opening them.
Open the surviving delete manifests first, then the data manifests. Delete
manifests come first so the reader knows which deletes are in play before it
decides what to emit.
Layer-3 pruning. For each data file, check its lower_bounds /
upper_bounds; if they rule the file out, skip it. Otherwise emit it as a scan
task.
Pair each scan task with the deletes that apply, using the sequence-number
rules from v2:
- a deletion vector applies when its referenced_data_file matches and its
  sequence is greater-or-equal, in the same partition;
- a position delete applies by the same rule, but only when no DV is present;
- an equality delete applies when its sequence is strictly greater than the
  data file's, in the same partition (or globally if unpartitioned).

Manifest list, then manifest, then file. Three reads narrow a petabyte table to
the handful of files a query actually needs. That funnel is the entire reason the
tree-of-files design beats a directory listing.

Reading across versions

A practical note that saves real debugging time: the format is designed so older
files read correctly under newer rules.

v1 read as v2: a missing sequence_number is 0; a missing content is 0
(data). Upgrading v1 to v2 is metadata-only.
v2 read as v3: files without first_row_id simply report _row_id as null;
position delete files keep working but can't be created anew.
v3 read as v4: a missing stats map key equals a null typed sub-struct; absolute
paths keep working unchanged alongside new relative ones.

And the one hard rule in the other direction: a reader refuses to open a table
whose format-version is higher than the reader supports. The version integer is
a contract, not a hint.

The whole format on one page

Here is the entire metadata surface, top to bottom, with the version each piece
arrived in:

catalog -> metadata.json
  |-- table level: format-version, schemas[], partition-specs[], sort-orders[],
  |               properties{}, last-*-id, last-sequence-number, encryption-keys[] (v3+)
  |-- refs{} -> main / branches / tags -> snapshot-id                            (v2+)
  |-- snapshots[] (whole history; expired entries pruned)
        |-- one snapshot:
              snapshot-id, parent-snapshot-id, sequence-number, summary{op,...},
              first-row-id + added-rows (v3+), key-id (v3+)
                |-- manifest list (Avro): [ manifest_file x N ]
                      manifest_file: path, len, spec-id, content (data|deletes),  (v2+)
                                     seq#, min_seq#, counts, partitions[],
                                     first_row_id (v3+)
                        |-- manifest (Avro): header{schema, spec, content} + [ entry x N ]
                              entry: status, snapshot_id, seq#, file_seq#,
                                     data_file {
                                       content, file_path, format, partition,
                                       record_count, size, sort_order_id,
                                       metrics maps (v1-v3)  -- OR --  content_stats struct (v4),
                                       equality_ids (v2+),
                                       referenced_data_file / content_offset / size (v3+),
                                       first_row_id (v3+), key_metadata
                                     }
                                |-- data file / delete file / Puffin DV blob

And the version-by-version cheat sheet:

	v1	v2	v3	v4
Core model	atomic snapshots on immutable files	+ row-level deletes, sequence numbers	+ stable row identity	metadata refactor only
Deletes	rewrite whole file (copy-on-write)	position + equality delete files	deletion vectors (position deletes deprecated)	unchanged
Schema/spec	singular `schema` / `partition-spec`	lists, explicit partition field ids	column defaults, new types, `source-ids`	unchanged
References	-	`refs`: branches + tags	unchanged	unchanged
Row lineage	-	-	`next-row-id` / `first-row-id` / `_row_id`	unchanged
Types	base	+ sort orders	variant, geometry, geography, unknown, ns timestamps	unchanged
Encryption	per-file key metadata only	per-file key metadata only	`encryption-keys`, snapshot `key-id`	unchanged
Statistics	metrics maps	metrics maps	metrics maps	typed `content_stats` + `tight_bounds`
Paths	absolute only	absolute only	absolute only	relative or absolute; `location` optional
Catalog	file-system or metastore	file-system or metastore	file-system or metastore	metastore only

Every commit appends a new metadata.json plus new manifest-list, manifest, and
data files. The old tree stays reachable through the metadata log for rollback
and time travel, until snapshot expiration garbage-collects it. Nothing is ever
mutated; the whole format is an append-only tree with one movable pointer at the
root.

The throughline

Read the four versions back to back and the arc is clean. v1 made a table
atomic on immutable files. v2 pushed correctness down to the row with deletes
and sequence numbers, and added branches. v3 pushed identity down to the row,
made deletes a single bitmap, and broadened types and security. v4 turned inward
and refactored the metadata so it can move and scale without rewrites.

It's the same instinct applied at finer and finer grain: make the unit of change smaller, and make the metadata that tracks it cheaper. That's why a format that started as an append-only snapshot log can now back a streaming upsert table with row-level lineage across petabytes - without ever giving up the one property it started with, the single atomic pointer swap.

If you want to build this understanding from the ground up - why data lakes broke,how snapshots and manifests really work, and the hands-on mechanics of deletes,branches, and compaction - that's the
Iceberg Foundations track,
part of the broader open table formats
curriculum. The format rewards reading it as a story, because that's how it was written.

Not Worthless. Just Not Found Yet. - On verification work that arrives before anyone asks for it

Tue, 23 Jun 2026 08:00:00 +0000

The shape of the work right now

Zenodo views: 1. Downloads: 0. No external citations to the ledger to date.

The code still runs.

That is the uncomfortable shape of this work right now. A full Flamehaven cycle can take three weeks, sometimes six months or more: theory, repo, verification, result artifact, paper, ledger. No client asked for it. No deadline forced it. No market signal arrived to confirm that it mattered.

So the question becomes simple, and not very flattering:

Is this verification work, or just elaborate work done in a room nobody enters?

My answer is that those are different questions. Visibility asks whether people are watching. Provenance asks whether the work survives being checked later, by anyone, including a future version of us. In AI-era research, I think that second question is becoming the scarce one.

This work is not valuable because someone is watching it.

It is valuable because, when trust in AI-generated output becomes scarce, already-verifiable work becomes rare. Not worthless. Not yet found. Those are not the same thing, and the rest of this is about why the difference matters.

How one cycle actually runs

From the outside it looks like "wrote a paper." In practice, six stages chain together in sequence.

Theory — Fix the boundary of the claim before anything else: what you will assert and, just as importantly, what you will not.
Repo — Translate theory into code. The code doesn't prove the theory; it renders the theory into something falsifiable and checkable.
Verify — The harness checks whether the code is internally self-consistent and actually respects the boundary defined upstream.
Result — Numbers, pass/fail counts, degraded-pass flags, dropped into a file. Not a hand-written summary. An artifact produced by execution.
Paper — Interprets the result. Does not stand in for it.
Ledger — Timestamps and keeps the trace of the entire cycle. When someone wants to re-verify the same theory later, the repo, result, and paper from that moment are still there, unchanged.

The reason these six stages run in strict order every time is simple: skip one, and the next stage proceeds unverified. Build the repo before fixing theory, and you end up writing code without knowing what it's supposed to test. Write the paper before running verify, and the paper ends up describing what the code was supposed to do rather than what it actually did. Skip ledger, and the next cycle has no way to audit the one before it.

This scaffold isn't rebuilt from scratch for every project.

The same skeleton gets reused. QSOT2 is one instance of that skeleton applied to the quantum-channel domain.

The two-axis problem this cycle is built against

A result.json that took days to build, and nobody opens it. A claim_boundary field designed with real care, and nobody who'd recognize what it's doing. The time was real. Almost none of it registers as value from the outside.

Answering that doubt requires reframing the question. Not who's watching, but whether this is the kind of problem worth watching in the first place. The evidence splits cleanly into two axes.

The first is output contamination. In May 2026, arXiv began enforcing a one-year submission ban for papers found to contain incontrovertible evidence of unchecked generative AI use, including hallucinated references — leftover meta-commentary addressed to or from the model counts as well. [1]

The scale behind that policy is documented: the rate of papers containing at least one fabricated reference went from roughly 1 in 2,828 in 2023 to 1 in 458 in 2025 to 1 in 277 in the first seven weeks of 2026, close to a twelvefold rise. A study co-authored by arXiv founder Paul Ginsparg audited 111 million references across arXiv, bioRxiv, SSRN, and PubMed Central, and found a conservative estimate of 146,932 hallucinated citations in 2025 alone. [2]

Copy-pasted prompt residue, citations to papers that don't exist — this isn't an edge case anymore. It's a statistically detectable frequency.

The second is reproduction failure. OpenAI's PaperBench evaluated AI agents on the task of replicating 20 ICML 2024 papers from scratch, graded against 8,316 rubric items. [3] The best-performing tested agent scored 21.0%. Human ML PhDs, given 48 hours on a comparable subset, reached 41.4%.

That doesn't mean AI agents are useless for research. It means fluent generation and successful reconstruction are still very different capabilities. On this benchmark, the strongest tested agent reached roughly half the human baseline on the comparable subset, and that gap matters precisely because the output can look competent before anyone re-runs it.

What the cycle answers back with

These are two stages of the same failure. The first is that submitted output often carries contamination straight through, unchecked. The second is that even output which clears that bar doesn't reproduce when someone actually re-runs it. Embedding claim_boundary directly in code is one response. Letting DEGRADED_PASS fire automatically is another. Together, they turn the claim into something the repo has to keep surviving, not something written once and trusted forever.

This isn't a concern unique to us. The broader research ecosystem is already circling the same problem from different angles: provenance, transparent reporting, dataset documentation, model cards, correction mechanisms, reproducibility.

One academic paper argues that universities and research communities need to build provenance infrastructure that anchors AI-era scholarship, since generative AI has made it trivial to produce fluent, polished, but unsubstantiated output. [4]

Elsevier's 2025 Researcher of the Future report found that 58% of researchers already use AI tools for work, while only 22% rate current GenAI tools as trustworthy. [5][6] 85% agreed that corrections and retractions help preserve the integrity of the scholarly record. Adoption and distrust are rising together, not trading off against each other.

That is the gap Flamehaven is trying to work inside. Not as an institution. Not as a standard-setting body. Just as a small project asking one narrow question repeatedly:

Can the claim, code, result, paper, and ledger still point to each other after the attention has moved on?

Not a system. A small pond.

Looked at in isolation, QSOT2 can seem like overkill — this much scaffolding for a toy model with no new physics in it. That objection is fair. It's also the point. QSOT2 isn't presented as new physics. It's a controlled pond for testing whether the verification skeleton stays clear.

claim_boundary, the DEGRADED_PASS verdict logic, flagging patterns like riemann_is_toy — none of this is QSOT2-specific. Wrap the same harness around a different phenomenological model and the same tier of verification carries over unchanged. BioAI auditing, RExSyn-Nexus, CR-EP all share the same skeleton.

Taken one at a time, each output looks worthless. Underneath, it isn't code rebuilt from zero each time. It's a structure built once and reused continuously — the important part is that the same skeleton is being grafted onto new domains, not redesigned for each one.

Calling Flamehaven a system would overstate it.

Right now it's closer to a small pond fed by a creek. The water is clear enough that you can see the bottom, and anything happening in it can be checked by anyone who looks. Murky water isn't a default we accept. Exposing riemann_is_toy instead of quietly dropping it is the same choice: keep the water clear rather than cloud it to look more impressive.

A pond isn't a grand thing on its own. But a pond that stays clear and stable, over time, tends to attract life.

Frogs lay eggs. Pondweed takes root. Birds come to drink. None of this was designed top-down as an ecosystem from day one — it's what happens when one body of water is kept clear long enough that a small ecosystem accretes around it on its own.

What the clarity is actually worth

The pond metaphor only works if “clear” stops being a mood and becomes something an outsider can inspect. In this context, clear means fewer hidden dependencies, fewer interpretive layers, and fewer places where the paper can make the result look cleaner than the execution actually was.

That is why the technical details matter.

Machine-epsilon-level CPTP verification runs on nothing but NumPy. 31 of the 35 core checks complete using pure NumPy, no PyTorch, no GPU. This does not make the work important by itself. It makes the work easier to re-run, easier to question, and harder to hide behind infrastructure.

So QSOT2's view count and download count aren't the metric that matters here. The metric is how many other domains this skeleton has been grafted onto, and whether each graft required redesigning from scratch. That answer is already accumulating, one instance at a time.

The useful test for any of this isn't whether someone agrees with the paper.

Agreement is cheap. The useful test is whether they can clone the repo, inspect the claim boundary, re-run the verification harness, compare result.json, and see the same ledgered output that existed before they arrived. That's the level where this work wants to be judged.

Living with worthlessness, rather than defeating it

The problem is that this is not the level where the visible market usually judges work.

The visible market still rewards velocity and virality more than verification scaffolding. This isn't work the market has learned how to price yet, and that's a different problem than the work being wrong.

So the task is not to defeat worthlessness with a louder story.

It is to tolerate a state that looks worthless while refusing to cloud the water before a signal arrives confirming that judgment was wrong. Recognition arriving late and recognition never arriving are different outcomes. There's no way to distinguish between them right now. What's been available, so far, is to hold the form steady and wait.

We have been waiting.

Not for Flamehaven to be noticed. That would be the wrong story. What we were waiting for, although we could not have named it cleanly at the time, was for the pressure itself to become public: the sense that proof-shaped output can outrun disclosure, that claims can separate from the artifacts meant to hold them, and that human responsibility can become decorative after the machine has already produced something convincing.

It arrived not as recognition, but as language.

When the outside world starts naming the same problem

In June 2026, a working group of sixteen researchers from fifteen universities published the Leiden Declaration on Artificial Intelligence and Mathematics, drawn from a Lorentz Center workshop the previous September that had brought together around sixty researchers from ten countries. [7]

The declaration is now officially endorsed by the International Mathematical Union. It names, in the language of mathematics, the same family of failure modes this piece has been circling. Automated techniques can produce results that look convincing but contain almost invisible errors, difficult to distinguish from genuine mathematical proofs, putting traditional standards of correctness, transparency, and independent verifiability under pressure.

Results communicated through press releases or blog posts, ahead of peer review and without the disclosure needed for scientific evaluation, are named as a direct threat to proper evaluation.

The Declaration's first recommendation for individual researchers is to disclose tool use explicitly, in a dedicated section, and to retain exclusive human responsibility for the correctness of any automated result.

That does not mean the Leiden Declaration validates Flamehaven. It means something narrower, and more useful: the failure mode Flamehaven was built around is no longer just an internal concern. Plausible-but-unreliable automated output, weak disclosure, missing provenance, and human accountability drifting away from machine-generated claims are now being named by the mathematical community itself, with institutional backing behind it.

claim_boundary and DEGRADED_PASS are not the Declaration in code. They are one small attempt to operationalize the same family of concern at the artifact level, built before that concern reached us as a public declaration with institutional backing.

Not because anyone here anticipated that working group, but because the underlying problem was already visible at small scale, one quiet result.json at a time.

A working alignment note mapping the Flamehaven verification cycle against each of the Declaration's recommendations is now on the ledger, section by section, including the places where the cycle currently falls short of what the Declaration asks for. [8]

This kind of naming has happened before

Once a field begins naming its own failure mode, the next question is not whether the language is elegant. The question is whether anyone changes behavior before an external authority forces them to.

That is why the historical comparison matters.

In 1975, recombinant DNA researchers gathered at Asilomar after parts of the field had already entered a voluntary pause, worried that the technology had outpaced anyone's ability to assess its risk. [9]

Roughly 150 scientists debated guidelines from scratch over four days, and the NIH later used those recommendations as a template for its official rules on recombinant DNA research, shaping biosafety regulation for decades after.

The point is not that AI mathematics is recombinant DNA. The point is narrower: sometimes a field begins to govern itself before formal institutions know what to do with the risk.

Asilomar did not begin as regulation. It began as scientists deciding that some kinds of power should not be allowed to outrun the practices that make them inspectable.

Small ponds, one current

This doesn't make the pond important on its own.

It changes something smaller, but more useful: it changes what the pond can now be compared against.

A year ago, a verification harness around toy quantum models could look like private overbuilding — too much claim_boundary, too many execution artifacts, too much ledger discipline for work almost nobody had asked to inspect.

After the Leiden Declaration, the same habit sits inside a larger problem statement. The question is no longer only whether this particular pond matters. It is whether research claims, once AI makes them cheap to produce, can remain tied to artifacts sturdy enough to inspect.

The Declaration is a mathematics-community document, not a software-verification framework. That boundary matters. Still, its core concern — automated output outrunning disclosure, review, and human responsibility — is the same pressure Flamehaven has been trying to make operational at the artifact level.

Asilomar named a related pattern fifty years earlier, in a different field, with a different danger at stake: a technical capability began moving faster than the practices built to govern it, and the field responded before formal institutions had settled the rules.

That is the only comparison I want to make.

A declaration names the pressure from above. A ledger answers from below by keeping each claim tied to the code, result, paper, and timestamped record that supported it. Neither changes the whole field alone. But a watershed was never one current to begin with. It is what becomes visible after enough separate sources of clean water keep draining toward the same low ground.

The trajectory still splits two ways. The trust gap around AI-generated research provenance could close on its own, market-driven and indifferent to anything built here. Or this kind of self-imposed, bottom-up correction — scientists and builders setting their own terms before regulation arrives — could become one of the ways it closes.

Which branch is correct can't be known from here. The second branch is no longer just a hope, though. It now has a name, an institutional endorsement, a publication date, and a fifty-year precedent showing that field-led restraint can shape formal governance.

That still does not make the work valuable in the market yet. The code still runs. The ledger still holds. The result still reproduces.

We are not the ocean. We are not even the current. But a pond does not need to know it is part of a watershed to matter. It only has to stay clear, drain in the same direction as other clear water, and let the geography do the rest.

References

[1] Nature, "arXiv to ban researchers for a year if they don't disclose AI use," May 2026

[2] Ginsparg et al., "Hallucinated References in Preprint Servers," arXiv, 2026

[3] OpenAI, "PaperBench: Evaluating AI's Ability to Replicate AI Research"

[4] "Provenance Infrastructure for AI-Era Scholarship," arXiv, 2026

[5] Elsevier, "Shaping the Researcher of the Future: Confidence, Collaboration and AI," 2025

[6] Elsevier, "Researcher of the Future — Confidence in Research," 2025

[7] Leiden Declaration on Artificial Intelligence and Mathematics, June 2026

[7a] Leiden University, "Leiden Declaration: AI is challenging the core values of mathematics," June 2026

[8] Flamehaven Verification Ledger, "Leiden Alignment Note"

[9] Embryo Project Encyclopedia, "Asilomar Conference (1975)"

Running Apache Airflow + Docker for Free Using GitHub Codespaces

Tue, 23 Jun 2026 07:22:54 +0000

While building my ETL pipeline project, I ran into a common beginner problem:

Running Apache Airflow locally on Windows with Docker was painful.

Problems included:

Low disk space
Docker setup issues
Linux compatibility problems
Environment debugging overhead

With only ~17GB free on my laptop, running multiple Airflow containers locally became difficult.

So I moved the entire setup to GitHub Codespaces.

What Codespaces Provided

Out of the box:

Ubuntu Linux environment
Docker pre-installed
VS Code in browser
Auto-cloned GitHub repo
Port forwarding for Airflow UI

Workflow

docker-compose up

Then:

Open Airflow UI
Trigger ETL DAG
Verify successful execution ✔️

Airflow was running in ~90 seconds.

Important Security Lesson

I accidentally committed my GCP service account key once.

GitHub Secret Scanning blocked the push automatically.

Immediately added:

*.json
.env

Never commit cloud credentials.

Why This Setup Helps Beginners

Codespaces removes a huge amount of local environment friction and lets you focus more on:

Airflow orchestration
ETL pipelines
Docker workflows
Cloud integrations

If you'd like:

Beginner-friendly walkthrough → check Medium
Engineering-focused breakdown → check Hashnode

Medium: Link
Hashnode: Link

Project Repo:ETL Pipeline

DataEngineering #Docker #ApacheAirflow #GitHubCodespaces

Why I Stopped Treating Job Applications as My Only Career Strategy

Tue, 23 Jun 2026 07:22:20 +0000

Like many engineers, I started my job search with a simple idea:

Apply to enough roles and eventually something will work out.

The reality was more complicated.

Some positions were already filled.
Some never responded.
Some required significantly more experience.
Some disappeared before interviews even started.

After a while, I realized something important:

Applications are necessary, but they are not the only mechanism for creating opportunities.

A Simple Probability Problem

Imagine sending 100 applications.

If the response rate is 2%, the expected number of responses is:

100 × 0.02 = 2

Now imagine spending part of that effort on:

Building projects
Writing technical articles
Creating a portfolio
Participating in engineering discussions

None of these guarantee opportunities.

But they increase the number of ways someone can discover your work.

What I Decided to Build

Instead of focusing exclusively on applications, I started working on:

Payment Gateway Design

Understanding transactions, idempotency, retries, and failure handling.

Schema Design Portfolio

Documenting database designs and architectural decisions.

Data Engineering Journey

Exploring Kafka, Spark, Airflow, and distributed systems.

Technical Writing

Sharing lessons learned while studying and building.

The Hard Part Nobody Talks About

The internet often makes personal branding sound easy.

Reality looks more like this:

Writing articles nobody reads.
Publishing posts that get little engagement.
Maintaining projects after the excitement wears off.
Spending months before seeing meaningful results.

There is no shortcut.

The value comes from consistency.

Final Thought

I'm not abandoning job applications.

I'm simply trying to build assets that continue working even when I'm not actively applying.

Applications create opportunities one submission at a time.

Projects and writing create opportunities that can compound over time.

I'm curious how other engineers balance these two approaches.

Managing Database Schema Like Code: The Quest for Database Version Control

Thu, 18 Jun 2026 06:08:16 +0000

In modern software development, we take git for granted. We branch, we commit, we merge, and we roll back code seamlessly. But when it comes to the database layer, things get tricky. Databases have state—you can't just overwrite a production database with a new version without risking catastrophic data loss.

So, is there a version control system for database structure changes? Yes, and it is a fundamental pillar of modern DevOps.

How to Build a Database Version Control System

To create a reliable system for tracking database changes, you need to transition from manual, ad-hoc SQL execution to an automated, migration-based workflow.

Baseline the existing database: You begin by taking a snapshot of your current database structure to serve as the ground-zero schema version.

Write incremental migration scripts:
Every change—whether creating a table, adding a column, or modifying a constraint—is written as a separate, sequential SQL or code file.

Maintain a schema version table:
The database itself must track its own version using a dedicated metadata table that records which migration scripts have already run.

Automate via a migration engine:
Tools like Liquibase, Flyway, or framework-native tools (like Prisma, Alembic, or EF Core) are integrated into the CI/CD pipeline to automatically detect and apply missing migrations.

The Roadblocks:
Common Pitfalls in Database Version Control
Transitioning databases to a version-controlled model introduces several friction points that do not exist with standard application code.

The state and data loss dilemma:
Unlike code files that can be easily replaced, database changes happen in-place and can easily destroy existing data if a destructive command (like dropping a column) is executed.

Merge conflicts in team environments:
When multiple developers work on separate features simultaneously, they might generate conflicting migration sequences, leading to deployment failures.

The challenge of rollbacks:
Reverting a code deployment is simple, but rolling back a database change that has already modified live data requires complex "down" scripts that may still result in data loss.

Production drift:
If a developer or administrator manually alters the database schema directly in production, the version control system loses sync, causing subsequent automated deployments to fail.

The Resolution:
Best Practices for Seamless Database Migrations
Overcoming these challenges requires strict engineering discipline and the right architectural approach.

Enforce a strict migration-only policy:
Block direct schema modifications on all environments, ensuring that every single database change must go through code review and the CI/CD pipeline.

Utilize semantic or timestamped sequencing:
Instead of using sequential integers (like 001, 002) which easily conflict, use UTC timestamps (e.g., 20260618120400_add_users_table) to allow concurrent migrations to coexist.

Adopt the expand and contract pattern:
For zero-downtime deployments, split destructive changes into phases. First, add the new structure (expand), migrate the data safely, and only remove the old structure (contract) in a later release.

Incorporate automated schema validation:
Integrate linters and dry-run tests into your pull request workflow to catch syntax errors and unsafe operations before they ever touch a live database.

Sumita
Web Developer

Supply Chain Sense: Merging Gemini AI and Math for Smart Retail Inventory

Sat, 13 Jun 2026 05:51:18 +0000

In neighbourhood retail markets, local Kirana stores, and hyper-local fulfilment centres, inventory management isn't an administrative task—it's a high-stakes daily tightrope walk. If a shop owner over-orders fresh dairy, poultry, or vegetables, the items rot on the shelves, expiration windows slam shut, and cold-storage electricity bills quickly wipe out their slim profit margins.

Conversely, if they under-order to play it safe, modern quick-commerce delivery grids and retail distributors issue immediate algorithmic visibility penalties, dropping their store placement and instantly handing premium shelf space to competitors. Traditional inventory tracking relies entirely on static, backward-looking spreadsheets that fail the moment a real-world disruption strikes.

To solve this, I engineered Supply Chain Sense: an open-source, AI-driven inventory optimization suite that bridges the gap between unstructured real-world context and deterministic operations mathematics. By coupling classical Continuous Review Inventory Models with real-time risk contextualization powered by Google Gemini 2.5 Flash, the system translates unpredictable disruption narratives into live, mathematically precise supply chain parameters.

Here is a look at the foundational constraints that inspired this project, the technical architecture, the cross-category mathematics, and how the application works end-to-end.

The Genesis: The Small Business Inventory Paradox

When designing an inventory management system specifically for small business retailers and neighbourhood mini-marts, you quickly realize they operate on an incredibly tight rope. Unlike massive retail conglomerates with deep capital reserves, a small business manages a highly diverse product portfolio spanning everything from short-shelf-life dairy to slow-moving electronics. This diversity creates a brutal, high-stakes dual threat:

The Risk of Over-Stocking: Spoilage vs. Frozen Capital

If a merchant over-orders fresh goods or raw materials, the consequences are immediate. Items rot on shelves, expiration windows slam shut, and the compounding overhead costs of cold-storage electricity and disposal completely obliterate their slim profit margins. On the other hand, if they over-stock high-unit-cost non-perishables (like electronic accessories or cosmetics), items don't rot, but their vital working capital is frozen in dead inventory that sits on a shelf for six months. In this sector, excess inventory is an active financial drain.

The Risk of Under-Stocking: Brutal Platform Penalties

Conversely, under-ordering is equally catastrophic. Modern quick-commerce apps, digital delivery grids, and large distributors enforce strict service-level agreements. If a merchant experiences a stockout on basic items, they face brutal immediate financial penalties, sudden downgrades in platform algorithm visibility, and immediate loss of premium shelf space to competitors. Customers who encounter empty shelves don't just wait for a backorder; they migrate to another provider forever.

Four Key Design Decisions for a Real Product Mix

We started this project on a core set of principles tailored specifically to handle a realistic convenience store ledger, shaping how our AI layer reasons about risk across different product categories:

The Temporal Risk Window:
The ultimate question for fresh inventory isn't just "how much volume do we order?" but "when does our risk window close?" Demand calculation must be incredibly time-sensitive, because a 3-day miscalculation on fresh goods is catastrophic, resulting in a total inventory write-off.

Asymmetric Risk Mapping:
The engine recognizes that not all stockouts are created equal. Running out of critical daily staples (like fresh food and beverages) causes severe reputational damage and drops a store's customer retention rate instantly. In contrast, over-stocking a few extra shelf-stable household items is an acceptable, manageable holding cost. The system intentionally treats core staples with an asymmetric safety bias.

Hyper-Sensitivity to Weather & Seasonal Signals:
Fast-moving consumer goods are hyper-reactive to external environmental shifts. Sudden monsoon deluges break transportation corridors while spiking grocery demand, local cultural festivals shift consumption patterns within hours, and high summer temperature spikes require instantaneous adjustments to cold-chain logistics capacity.

The "Virtual COO" Directive:
Small business owners do not have the time to decrypt complex academic models or interpret abstract data science jargon. The core output of the engine needed to act as an automated, domain-aware Chief Operating Officer—delivering tactical, empathetic, and category-specific advice that reads like real operational guidance, rather than generic corporate-speak.

Grounding the Operations in Classic Math

To maintain absolute structural integrity, the engine completely avoids arbitrary adjustments. Every initialized constraint is directly mapped to established operations research and financial risk theory using a continuous review model framework.

Target Service Levels (SL) and the Z-Score Matrix

The Target Service Level represents the mathematical probability that a business will successfully fulfil customer orders without experiencing a stockout during a single vendor replenishment cycle. To convert this percentage into an active warehouse cushion, the code maps it onto a standard normal distribution curve using the Inverse Cumulative Distribution Function (also known as the Percent Point Function or Z-score function):

Z = PPF(SL)

The application initializes distinct service levels based on the financial and reputational cost of a stockout:

Food and Beverages (99% Service Level, Z approx. 2.33): High protection threshold. If an essential grocery staple is unavailable, customers migrate immediately, inducing harsh penalties.

Electronic Accessories / Home & Lifestyle (85% Service Level, Z approx. 1.04):
Lower protection threshold. These high-value, slow-moving items tie up too much capital if over-stocked, and consumers are typically willing to tolerate minor backorder wait times.

Annual Carrying Cost Rates (I) and Daily Holding Costs (H)

Storing items involves overhead (rent, climate control, insurance, and spoilage). The Annual Carrying Cost Rate (I) represents this physical expense as a percentage of the item's unit price. The engine derives the Daily Holding Cost (H) per item using the following calculation:

H = (Unit Price x I) / 365

The category rates directly correspond to product characteristics:

Food and Beverages (35% Carrying Rate): High rate driven by extreme perishability, tight shelf-life limitations, and the steep energy costs required for refrigerated storage.

Fashion Accessories (25% Carrying Rate): High rate driven by rapid style obsolescence and seasonal demand expiration.

Electronic Accessories (15% Carrying Rate): Low rate due to high material stability, compact shelf volume, and long physical shelf-life profiles.

Elevating the Square Root of Lead Time

The Core Mathematics: Why Variance Scales with sqrt(Lead Time)

A common mistake in basic inventory systems is multiplying demand volatility (std_dev) linearly by the supplier lead time duration. Supply Chain Sense strictly avoids this by implementing standard statistical operations theory: standard deviation does not scale linearly over time; variance does.

Because daily customer demand behaves like an independent random variable, demand variance aggregates linearly across successive days (Var = std_dev^2 x Lead Time). To return this back into the physical unit dimension of a standard deviation cushion, we must extract the square root of the temporal window:

Adjusted Volatility Buffer = std_dev x sqrt(Lead Time adjusted)

This mathematical choice is critical for risk exposure. If a supply chain transit window stretches from 2 days to 8 days (a 4x increase in delay), the actual risk exposure and required safety stock cushion do not quadruple—they double (2x). Utilizing a linear multiplier would cause small businesses to massively over-order, locking up thousands of dollars of vital working capital in redundant safety stock.

System Architecture & Fail-Safe Design

To ensure true production-grade resilience for small businesses, the application handles data extraction and API calling defensively through two distinct mechanisms:

Deterministic Edge-Computed Heuristic Fallback: In real-world retail environments, you cannot assume 100% cloud uptime. If a neighborhood merchant is facing a severe regional crisis—like a monsoon or a transport strike—cellular networks often experience heavy traffic spikes or blackouts. If our software relied entirely on a stable cloud connection to reach Google Gemini, a network drop would completely freeze the store's inventory pipeline.

To prevent this single point of failure, the architecture includes a deterministic backup engine that executes completely offline on the edge, instantly parsing text logs using keyword tracking and lookahead loops if the cloud API becomes unreachable.

def intelligent_local_fallback(text_input, unique_categories):
    """
    Edge-computed deterministic backup engine to handle cloud network traffic spikes.
    """
    text = text_input.lower()
    detected_delay = 0
    words = text.split()
    
    # Extract temporal delay window from unstructured text
    for i, word in enumerate(words):
        if word in ['day', 'days', '-day'] and i > 0:
            clean_num = words[i-1].replace('-', '').strip()
            if clean_num.isdigit():
                detected_delay = int(clean_num)
                break

    # Default baseline delay handling for critical supply disruptions
    if detected_delay == 0 and any(k in text for k in ['strike', 'block', 'flood', 'storm', 'delay', 'cyclone']):
        detected_delay = 3

    food_shock = 1.00
    others_shock = 1.00
    
    # Asymmetric risk classification based on macro events
    if any(k in text for k in ['strike', 'block', 'protest', 'highway']):
        food_shock = 1.15
    elif any(k in text for k in ['cyclone', 'flood', 'storm', 'monsoon', 'weather']):
        food_shock = 1.60
        others_shock = 0.40
    elif any(k in text for k in ['holiday', 'festival', 'celebration', 'christmas']):
        food_shock = 1.30
        others_shock = 1.10

    adjustments = {}
    for cat in unique_categories:
        if any(f_word in cat.lower() for f_word in ['food', 'beverage', 'grocery', 'fresh', 'eat']):
            adjustments[cat] = { "Demand_Shock_Factor": food_shock, "Lead_Time_Delay_Days": detected_delay }
        else:
            adjustments[cat] = { "Demand_Shock_Factor": others_shock, "Lead_Time_Delay_Days": detected_delay }

    return {
        "disruption_summary": f"Local Fail-Safe Active: Inferred operational shifts with an estimated +{detected_delay} day transit bottleneck.",
        "department_adjustments": adjustments
    }

Let's break down exactly how this fallback engine isolates critical data points without an external AI model:

Text Tokenization & Temporal Search: The code normalizes the user's input with .lower() and splits it into individual words. It then searches for anchor terms like ['day', 'days', '-day']. When found, it utilizes a relative index lookahead (words[i-1]) to extract the preceding character, strips out hyphens, and runs .isdigit() to safely convert natural language mentions of transit timelines (e.g., "5-day delay") into an integer.

Keyword Risk Mapping: If no explicit day count is found but critical disruption keywords (like strike, flood, or cyclone) are present, the algorithm applies a defensive baseline delay of 3 days to keep safety parameters conservative.

Hardcoded Asymmetric Shocks: The function mimics our primary Gemini AI layer by applying uneven demand modifiers based on the crisis profile. For instance, if weather keywords like monsoon or flood are detected, it applies an aggressive 1.60 demand acceleration factor to essential groceries (food_shock) due to panic buying patterns, while deliberately dialing down discretionary categories to 0.40 (others_shock) to preserve store capital.

Schema Matching: Finally, it iterates through the active store channels and structures its return payload as a dictionary that precisely mimics the JSON schema generated by Gemini. This allows the core pandas continuous review loop to ingest the data seamlessly without knowing whether the parameters came from the cloud or the local edge.

Schema-Enforced Structural Reliability: By forcing the model configuration parameters to return exact, structured JSON maps (response_mime_type="application/json"), the system bypasses unpredictable conversational AI filler text, mapping narrative updates straight into pandas data manipulation streams without any risk of execution structural runtime crashes.

Real-World Simulation: Asymmetric Trade-Offs in Action
Let's evaluate how the system handles an intense multi-channel crisis scenario. Imagine an operator inputs the following unstructured update into the live feed:

"Heavy storms and flash floods have blocked major highway entry points, delaying incoming cargo trucks by roughly 5 days. Local news is predicting intense panic buying on household groceries, while clothing and luxury retail spots expect massive drops in foot traffic."

The platform routes this alert to the cloud, executes the optimization models, and produces asymmetric parameter overrides:

Case A: The High-Risk Channel (Food & Beverages)

AI Extraction: Gemini reads the text and extracts a Demand_Shock_Factor of 1.5 (50% velocity acceleration) and a Lead_Time_Delay_Days of +5.

The Mathematics: Total lead time expands from 2 days to 7 days. Simultaneously, average daily consumption (avg_demand) and standard deviation (std_dev) are scaled up by 1.5x to reflect panic-buying. Because consumption velocity and transit delays escalate together, the mathematical engine drives the Optimized Reorder Point upward significantly. This immediate surge creates a massive inventory buffer, keeping essential items on shelves throughout the storm and avoiding platform penalties.

Fig 1: The High-Risk Channel (Food & Beverages)

Case B: The Low-Risk Channel (Electronic Accessories or Fashion)

AI Extraction: Gemini recognizes that discretionary segments will experience a steep decline in foot traffic as consumers focus purely on survival essentials, assigning a compressed Demand_Shock_Factor of 0.5.

The Mathematics: Even though the regional transit delay pushes the supplier lead time window to 7 days, consumer buying activity drops by half. The engine balances these opposing metrics and drops the Optimized Reorder Point down safely.

Fig 2: The Low-Risk Channel (Electronic Accessories or Fashion)

The Strategic Value

This defensive suppression is where the system shines. By scaling down ordering triggers for durable, discretionary goods, the application safeguards the merchant's working capital. The money saved by not ordering slow-moving items is automatically redirected to finance the critical safety stock required for high-demand groceries.

The system then delivers customized "Virtual COO" textual guidance, explaining the exact logic of these changes in clear, operational terms before generating a downloadable blueprint (.CSV) that can be fed directly into automated procurement software.

Fig 3: Downloadable Tailored Inventory Blueprint (.csv) available after analysis

Enterprise-Grade Security: Simple Cloud Key Management

When deploying an open-source data application to public repositories like GitHub, security is a paramount concern. Hardcoding API keys or committing credential files (.env, .json) directly into a repository is an immediate security vulnerability that invites token theft and exploitation.

However, forcing small business owners to set up complex local environment variables manually defeats the purpose of building a lean, accessible application. To solve this, the architecture abstracts credential handling completely away from the source code, deploying an encrypted, cloud-native runtime boundary check that integrates seamlessly with hosted environment secret managers.

if "GEMINI_API_KEY" in st.secrets:
    os.environ["GEMINI_API_KEY"] = st.secrets["GEMINI_API_KEY"]
    try:
        client = genai.Client()
    except Exception as e:
        st.error(f"Failed to initialize Gemini Client: {e}")
else:
    st.error("**Security Token Missing!** Please add your `GEMINI_API_KEY` to your Streamlit Advanced Secrets panel.")
    st.stop()

Encrypted Sandbox Introspection: The application uses Streamlit's native dictionary abstraction, st.secrets, to check for the presence of the authentication token in the cloud's secure memory space. This completely removes the need to track, store, or manage sensitive string files within your Git tree.

Environment Variable Synchronization: Once identified, the token is bound directly into the operating system's local memory environment using os.environ["GEMINI_API_KEY"]. This ensures the downstream Google GenAI SDK can implicitly fetch the credential without requiring explicit, plain-text string handling inside individual function parameters.

Defensive Try-Except Wrapping: The instantiation call genai.Client() is safely wrapped inside an execution monitoring block. If a network configuration issue, invalid credential structure, or handshake error occurs, the system catches the failure gracefully with except Exception as e and prints a clean user-facing error message without causing a total application crash.

Assertive Streamlit Halting: If the secret token is missing entirely from the cloud environment, the code displays a clear markdown warning message and immediately invokes st.stop(). This acts as an assertive architectural boundary, completely freezing execution to prevent the optimization engine from crashing or attempting unauthorized operations further down the line.

Conclusion: Grounding Agri-Retail in Real-World Context

By anchoring the natural language comprehension of Large Language Models directly within rigorous statistical inventory control equations, we can build applications that are highly intuitive yet mathematically sound. Supply Chain Sense demonstrates its true value when dropped into highly volatile, localized ecosystems like the Indian Agri-retail corridor.

In a sector where a sudden monsoon deluge halts transport logistics overnight, local wholesale mandis face sudden price shocks, and regional festivals cause hyper-local spikes in demand for dairy and perishables within hours, static software fails.

Generative AI serves as a real-time logical reasoning layer that reads unstructured news alerts, interprets local environmental context, and immediately updates physical warehouse thresholds.

This architecture proves that AI's greatest utility isn't generating open-ended conversational filler, but acting as a dynamic data-engineering bridge—shielding small merchants from stockouts, eliminating food spoilage, and protecting thin margins in an unpredictable world.

The full repository is completely open-source, features a seamless downloadable blueprint generator for automated purchasing setups, and is ready for use.

Check out the code and deploy it for your business here: github.com/Ishaa2004
Let's connect on LinkedIn: linkedin.com/in/isha-gupta12Open to freelance projects: Need a custom dashboard, inventory model, or data pipeline for your business? I take on select freelance projects. [View my gigs on Fiverr][3] or reach out directly on LinkedIn.

https://www.fiverr.com/ishagupta102

Monitoring the Shift: How to Effectively Track SQL Server Table Changes

Fri, 12 Jun 2026 11:32:51 +0000

In modern data environments, knowing exactly when, how, and who changed your data is paramount for auditing, synchronization, and debugging. Implementing a reliable tracking system requires a solid blueprint, an understanding of potential roadblocks, and modern architectural solutions.

The Initial Blueprint: How to Build It
To begin tracking modifications in an SQL Server database, developers traditionally lean on a few standard database-level mechanisms.

Audit Columns and Triggers: The classic approach involves adding LastModifiedDate and ModifiedBy columns directly to your tables. You then deploy AFTER UPDATE and AFTER INSERT triggers to automatically populate these timestamps whenever a row is modified.

Temporal Tables (System-Versioned): A more modern built-in approach where SQL Server automatically manages a parallel history table. Every time a row is updated or deleted, the old version is seamlessly archived with a validity period.

The Real-World Roadblocks: What Will Go Wrong
While the initial setup seems straightforward, deploying these methods in high-traffic production environments introduces significant engineering challenges.

Severe Performance Degradation: Heavy use of synchronous triggers introduces massive transactional overhead. Every write operation is forced to wait for the trigger logic to execute, causing blocking, deadlocks, and slowed application response times.

Storage Explosion: Tracking every single micro-change means your database size will balloon rapidly. Without an automated data-retention and purging strategy, you will quickly run out of disk space.

Schema Evolution Fragility: Traditional tracking triggers are tightly coupled to the table structure. The moment a schema change occurs (like adding or dropping a column), your tracking scripts can break completely, leading to silent data loss or application crashes.

The Modern Resolution: How We Solve These Problems
To bypass the performance and maintenance nightmares of triggers, we must pivot toward asynchronous, engine-level features native to SQL Server.

Implementing Change Data Capture (CDC): Instead of using triggers that intercept active queries, CDC reads the transaction log asynchronously. This completely eliminates runtime performance overhead on user queries while capturing full INSERT, UPDATE, and DELETE histories.

Utilizing Change Tracking (CT): If you only need to know which rows changed (and not the historical values themselves), Change Tracking is the lightweight solution. It answers the question "Has this row changed since version X?" with minimal storage footprint, making it ideal for cache synchronization.

Automated Partitioning and Offloading: To solve the storage crisis, history tables should be partitioned by date. Older audit logs should be automatically archived to cheaper cold storage (like Azure Blob Storage or an external data lake) to keep the primary transactional database lean and fast.

Sumita
Web Developer

Your Data Strategy Is Backwards. Here's the Pattern That Actually Works.

Thu, 11 Jun 2026 16:42:39 +0000

The numbers are hard to ignore. Eighty-seven percent of organizations are investing in data and analytics. Seventy-four percent are struggling to achieve and scale real value from it. The average Chief Data Officer lasts just 2.5 years in the role before moving on.

Something is structurally wrong — and according to Nysa Zaran Evagelinos, Research Director at Info-Tech Research Group, the problem starts with how organizations think about data products.

"Why, given that we have so much data, given that we all do things with this data, is it always so challenging?" she asked, opening the final day keynote at Info-Tech LIVE 2026 in Las Vegas. "It's not a niche discipline. There's so much advice out there. Where do we keep falling short?"

The Data Product Problem

The current industry answer to that question is the data product — owned, trusted, reusable, discoverable. Vendors and analysts have been promoting the concept as the path to data-driven value for several years now.

But the results tell a different story. According to a 2026 study of data engineering organizations, 70% of data product initiatives stall after 12 to 18 months.

The reason, Zaran argued, is that most data products solve the wrong problem. They deliver real value — but only within the CIO and CDO portfolio. Faster time-to-incident-detection. Better time-to-resolution. Reduced data downtime. Quicker analytics deployment. These are efficiency gains for the data team, not enterprise value drivers.

"The results are limited to running a more efficient shop," she said. "But the mission is to drive broader transformation — shift the culture, make a difference to the organization. Not just run a more efficient pipeline."

The Data Value Chain

The framework Zaran introduced reframes the problem entirely. Instead of starting with data, she starts with value — and works backward.

The Data Value Chain runs from left to right: Data → Insight → Decision → Action → Value. AI maps directly onto this chain, perceiving and learning on the left side, reasoning and planning in the middle, creating and acting on the right.

Most organizations have invested heavily in the left side of that chain. They've built data infrastructure, established pipelines, created dashboards, and produced insights. What they've largely failed to do is own the right side — the leap from Decision to Action, and from Action to measurable Value. That leap, Zaran noted, is where data culture either exists or it doesn't.

"The business is left to own the rest of the chain," she said. "And the gap to value remains."

Four Companies That Closed the Gap

To illustrate what it looks like to own the full chain, Zaran walked through four case studies — each representing a different entry point on the value chain.

Bloomberg Terminal anchored the left side. With 10 billion data points published daily across 8,000 curated datasets and 40,000 data fields, Bloomberg's product goal was straightforward: connect decision-makers to exceptional data they couldn't get anywhere else. The result is roughly $15 billion in annual revenue.

Netflix moved further right. Rather than selling data, Netflix built decision intelligence — a feedback loop that uses viewing behavior to drive personalization, content strategy, and a low-friction experience. The system continuously collects data, predicts preferences, serves content, and refines itself. Revenue: $45.18 billion, with $3.4 billion invested annually in technology and development.

UPS ORION pushed into action. The route optimization system — a copilot for drivers and logistics planners — took 10 years and $250 million to build. The payoff: $300 to $400 million in annual savings and 100,000 tonnes of carbon emissions avoided. The hardest part wasn't the algorithm. It was getting drivers to trust it enough to follow it.

Waymo went all the way to autonomy. Five hundred thousand rides per week, 200 million miles logged, operating in 10 US cities. The data product isn't a dashboard or a recommendation engine — it's a system that perceives, decides, and acts without human intervention. Governance, Zaran noted, means running billions of simulations so the system never encounters a scenario it hasn't already seen.

"Each of these companies figured out a different way to use the data value chain," she said. "They each stopped at a different point — and in doing so, they each had different problems to solve."

The Winning Pattern

What separates these companies from the 74% struggling to scale isn't better data infrastructure. It's where they started.

Most organizations start on the left: collect data, build pipelines, create products, and hope value follows. The winning pattern does the opposite — start on the right and backtrack.

Four questions drive the approach: What outcomes does the organization actually care about? What actions and decisions drive those outcomes? Can data move the needle on those decisions? And do we have the data, or can we get it?

"A data product is almost a byproduct," Zaran said. "It's a byproduct of the discipline of running an outcome end-to-end using data. You measure it by its purpose. You measure it by the downstream change it creates."

The implication for AI is direct. Organizations racing to deploy AI tools, agents, and copilots face the same trap that data products fell into — investing heavily in capabilities without anchoring them to a specific outcome the organization cares about.

"This is how AI becomes a tool," Zaran said. "Not your destination."

The Hidden Cost of Distributed Systems

Thu, 11 Jun 2026 12:12:07 +0000

One of the most interesting contradictions in cloud architecture is that we spend a tremendous amount of effort trying to eliminate complexity, yet many of the systems we build seem to become more complex over time.

A startup might begin with a single application, a single database, and a handful of users. Everything is relatively easy to understand. A request comes in, business logic is executed, data is stored, and a response is returned. The architecture is straightforward, deployments are simple, and troubleshooting usually involves looking in one place.

Then growth happens.

New customers arrive. New features are requested. More engineers join the team. Integrations are added. Reporting requirements expand. Mobile applications appear. Real-time processing becomes important. What was once a single application gradually evolves into a collection of services, databases, queues, and APIs.

Without necessarily realizing it, the organization has started building a distributed system.

What Exactly Is a Distributed System?

At its core, a distributed system is simply a group of independent components working together to provide what users perceive as a single application.

Consider what happens when a customer places an order on an e-commerce platform. From the customer's perspective, the experience feels simple: click a button, receive a confirmation, and wait for the package to arrive.

Behind the scenes, however, the request may travel through multiple services. One service records the order, another processes the payment, another updates inventory, while additional systems generate notifications, update dashboards, and trigger downstream workflows. Each component owns a specific responsibility, yet all of them collaborate to complete a single business transaction.

This separation of responsibilities is one of the defining characteristics of distributed systems.

Why Do We Build Them?

If distributed systems introduce additional complexity, a reasonable question follows:

Why not keep everything inside a single application?

The answer is that distributed systems solve problems that become increasingly difficult to address as organizations grow.

Imagine an online retailer preparing for Black Friday. The checkout process may suddenly experience a massive increase in traffic, while administrative reporting tools remain largely unchanged. In a monolithic architecture, scaling one component often means scaling the entire application. In a distributed architecture, services can scale independently based on their specific needs.

The benefits extend beyond scalability. Distributed systems also help organizations isolate failures, allow teams to work more independently, and enable different technologies to be used for different workloads. As systems and engineering organizations grow, these advantages become increasingly valuable.

This is why most modern cloud-native architectures eventually move in this direction.

Not because distributed systems are fashionable.

Because they solve real business and operational challenges.

The Trade-Offs Begin the Moment You Split the System

The problem is that complexity does not disappear.

It simply moves.

Inside a monolithic application, communication often happens through function calls within the same process. In a distributed system, communication happens over a network. That seemingly small change introduces an entirely new category of problems.

Networks introduce latency. Requests time out. Connections fail. Messages arrive late. Sometimes they arrive twice. Occasionally they never arrive at all.

Engineers must now design systems that can tolerate these realities.

The architecture becomes more flexible, but it also becomes less predictable.

Data Consistency Becomes a Design Problem

One of the biggest mindset shifts occurs when data is no longer stored in a single place.

Imagine a payment service successfully charges a customer. Immediately afterward, the inventory service fails before reserving the product. The customer has paid, but the order cannot be fulfilled.

What should happen next?

Should the payment be refunded automatically?
Should the inventory operation be retried?
Should the order remain pending until a human intervenes?

These questions rarely emerge in simple applications, but they appear regularly in distributed systems.

As a result, engineers begin adopting concepts such as eventual consistency, retries, idempotency, dead-letter queues, and event-driven communication patterns. These are not merely technical buzzwords. They are practical responses to the challenges created by distributing responsibilities across multiple services.

Choosing Complexity Carefully

One of the most valuable lessons in software architecture is that every decision involves trade-offs.

Distributed systems provide scalability, resilience, and organizational flexibility. They also introduce operational overhead, additional failure modes, and significantly more complexity.

For that reason, building a distributed system should never be the objective.

Solving a business problem should be the objective.

Sometimes that solution is a distributed architecture. Sometimes it is a well-designed monolith. The right answer depends entirely on the requirements, constraints, and growth trajectory of the system.

Final Thoughts

Distributed systems power many of the applications we use every day. They enable organizations to scale beyond the limits of a single application and support levels of reliability that modern users have come to expect.

At the same time, every service added to an architecture creates another relationship that must be understood, monitored, and maintained.

That is the hidden cost of distributed systems.

They give us the ability to scale, but they require us to manage a level of complexity that simply does not exist in smaller architectures.

And that is why understanding distributed systems is not really about learning new technologies.

It is about understanding the trade-offs that come with growth.

go-intake v0.1.0: A Small Go Library for Messy Data Intake

Tue, 09 Jun 2026 10:05:39 +0000

go-intake: Go-Native Streaming Data Ingestion Toolkit

Executive Summary

go-intake is a minimalist, streaming-first ETL toolkit for Go developers that transforms messy data into validated, record-oriented output. With zero third-party dependencies and a focus on composition over configuration, it provides enterprise-grade data pipeline capabilities in a single, embeddable library.

Architecture Overview

Core Abstraction Layers

Component	Interface	Responsibility	Pattern
Source	3-method interface (`Open`, `Read`, `Close`)	Data ingestion from CSV, JSONL	Stream until EOF
Transformer	`Apply(ctx, Record) (Record, error)`	Field normalization, type parsing	Immutable, returns copy
Validator	`Validate(ctx, Record) error`	Schema enforcement, business rules	Read-only
Quarantine	`Write(ctx, InvalidRecord)`	Rejected record capture with context	Structured error logging
Sink	3-method interface (`Open`, `Write`, `Close`)	Output to CSV, JSONL	Streaming write

Design Principles

┌─────────────┐    ┌──────────────┐    ┌─────────────┐
│   Source    │───▶│ Transformers │───▶│ Validators  │
└─────────────┘    └──────────────┘    └─────────────┘
                                            │
                               ┌────────────┴──────────┐
                               ▼                       ▼
                         ┌──────────┐           ┌───────────┐
                         │   Sink   │           │Quarantine  │
                         └──────────┘           └───────────┘

Memory Usage Comparison

Memory (MB)
   500 │                      
       │    ●●●●●●●●●●●●●●●●●●  go-intake (constant)
       │                      
   100 │                      
       │                      
    50 │    ████ goflow grows with dataset
       │    ██████            
    25 │    ████████████      
       │    ████████████████  streamz grows
       │                       
     0 └──────────────────────
        0K    50K   100K   150K
           Records Processed

Performance Benchmarks

Throughput Analysis

Dataset Size	Records Processed	Time Elapsed	Throughput
Small (10K)	10,000	~40ms	~250,000/sec
Medium (100K)	100,000	~410ms	~243,902/sec
Large (161K)	161,568	643ms	251,000/sec

Memory Efficiency: The streaming model ensures constant memory usage regardless of dataset size. Only one record exists in memory at any point during processing.

Comparative Performance

graph LR
    A[go-intake<br/>251K/s] --> B[Memory: O(1)]
    C[Gopherize<br/>125K/s] --> D[Memory: O(n)]
    E[goflow<br/>180K/s] --> F[Memory: O(n)]

Performance Scaling Chart

Records/sec
   300K │
        │
   250K │    ● (go-intake: 251K)
        │   ╱
   200K │  ╱
        │ ╱
   150K │● (goflow: 180K)
        │
   100K │● (streamz)
        │
    50K │● (Gopherize: 125K)
        │
     0K └─────────────────────

Feature Matrix vs Competition

Feature	go-intake	Gopherize	goflow	streamz
✅ Zero Dependencies	✓	✗	✗	✗
✅ Streaming Model	✓	Partial	✓	✓
✅ Structured Errors	✓	✗	Partial	✗
✅ Multi-validation Errors	✓	✗	✗	✗
✅ Schema Discovery	✓	✗	✗	✗
✅ Header Normalization	✓	✓	✗	✗
✅ Quarantine Sinks	✓	✗	✗	✗
✅ Context Cancellation	✓	✗	✓	✓
Lines of Code	~2,300	~8,500	~12,000	~6,200

Bundled Components

Sources

CSVSource: RFC 4180 compliant, configurable delimiter, empty line skipping
JSONLSource: Newline-delimited JSON, 16MB line buffer support

Sinks

CSVSink: Automatic header generation, deterministic column ordering
JSONLSink: HTML-escaping disabled, one object per line

Transformers

Function	Purpose	Use Case
`NormalizeHeaders(style)`	Case normalization	Column standardization
`TrimStrings()`	Whitespace removal	Data cleaning
`ParseFloat/Int/Bool/Date`	Type conversion	Schema enforcement
`Rename/Drop/Keep/Copy`	Schema evolution	ETL pipeline stages
`AddField`	Constant injection	Metadata enrichment
`MapField`	Custom transformations	Business logic

Validators

Function	Type	Error Aggregation
`Required(field)`	Presence	✅
`Min/Max/Between`	Numeric range	✅
`Regex(field, pattern)`	Pattern matching	✅
`Enum(field, values...)`	Allowlist	✅
`Email/URL`	Format validation	✅
`NotFuture(field)`	Temporal constraint	✅

Real-World Validation Results

Dataset 1: COVID-19 Aggregated Data (161,568 records)

Field Profiling Results:

Field	Type	Confidence	Null %
Confirmed	int	100%	0%
Country	string	100%	0%
Date	date	100%	0%
Deaths	int	100%	0%
Recovered	int	100%	0%

Pipeline Results:

Read: 161,568
Written: 161,568
Invalid: 0
Failed: 0
Runtime: 643ms

Dataset 2: Synthetic Messy Data (5,000 records with 2-5% error injection)

Statistical Validation:

Metric	Value	Visualization
Total Records	5,000	██████████
Valid Output	4,772 (95%)	████████░░
Quarantined	228 (4.56%)	██░░░░░░░░
Error Detection	100%	██████████

Pipeline Accuracy
100% │ ██████████
     │ ██████████  (100% records accounted for)
 95% │ ████████░░  (95% valid)
     │ ████░░░░░░  (4.56% quarantined)
  0% └────────────────
     0%       100%  Accuracy

Error Distribution in Quarantine:

Error Type	Count	Description	Distribution
required	96	Missing required fields	████░░░░░░ (29%)
min	58	Negative numeric values	██░░░░░░░░ (18%)
regex	172	Email format mismatch	██████░░░░ (53%)
Total	326	Captured validation errors	██████████ (100%)

Use Cases

1. Data Migration Pipelines

p := intake.New().
    From(source.CSV("legacy_export.csv")).
    Transform(transform.NormalizeHeaders(transform.SnakeCase)).
    Validate(validate.Required("id")).
    To(sink.JSONL("normalized.jsonl"))

2. API Data Validation

p := intake.New().
    From(source.JSONL("api_incoming.jsonl")).
    Transform(transform.TrimStrings()).
    Validate(
        validate.Email("email"),
        validate.URL("website"),
        validate.Min("age", 18),
    ).
    OnInvalid(quarantine.JSONL("rejected.jsonl")).
    To(sink.CSV("clean.csv"))

3. ETL for Analytics

p := intake.New().
    From(source.CSV("raw_events.csv")).
    Transform(
        transform.NormalizeHeaders(transform.SnakeCase),
        transform.ParseDate("timestamp", time.RFC3339),
        transform.ParseFloat("value"),
    ).
    Validate(validate.Required("event_id")).
    OnInvalid(quarantine.JSONL("bad_events.jsonl")).
    To(sink.JSONL("analytics_ready.jsonl"))

4. Data Quality Inspection

src := source.CSV("unknown_dataset.csv")
profile, err := discover.InspectSource(ctx, src, discover.Options{
    SampleSize: 10000,
})

for _, f := range profile.Fields {
    fmt.Printf("%s: %s (confidence %.2f)\n", 
        f.Name, f.Type, f.TypeConfidence)
}
for _, issue := range profile.Issues {
    fmt.Printf("[%s] %s\n", issue.Severity, issue.Message)
}

Data Transformation Flow

    ┌─────────────────┐
    │  Raw CSV/JSONL  │
    │   (Messy Data)  │
    └────────┬────────┘
             │
             ▼
    ┌─────────────────┐     ┌───────────────┐
    │   Discover      │────▶│ Field types   │
    │   Inspect       │     │ Null ratios   │
    └─────────────────┘     └───────────────┘
             │
             ▼
    ┌─────────────────┐
    │  Normalize      │
    │  Headers        │
    │  Trim Strings   │
    └────────┬────────┘
             │
             ▼
    ┌─────────────────┐
    │   Parse Types   │
    │  (int, float,  │
    │   date, bool)   │
    └────────┬────────┘
             │
             ▼
    ┌─────────────────┐     ┌───────────────┐
    │   Validate      │────▶│ Valid Output  │
    │   Required      │     │   (Sink)      │
    │   Min/Max       │     └───────────────┘
    │   Regex/Enum    │             │
    └────────┬────────┘             │
             │                      │
             ▼                      ▼
    ┌─────────────────┐     ┌───────────────┐
    │   Quarantine    │     │   Stats:      │
    │   (errors +    │     │ Read/Written/ │
    │   record data)  │     │ Invalid/Failed│
    └─────────────────┘     └───────────────┘

Competitor Comparison Matrix

Toolkit	Dependencies	Streaming	Discovery	Quarantine	Lines of Code
go-intake	Zero	✅	✅	✅	~2,300
Gopherize	12	Partial	❌	❌	~8,500
goflow	18	✅	❌	❌	~12,000
streamz	8	✅	❌	❌	~6,200
etl-go	5	❌	❌	❌	~4,500

Technical Specifications

Specification	Value
Go Version	1.23+
License	MIT
Dependencies	Zero (stdlib only)
Memory Model	Streaming (O(1))
Concurrency	Context-aware cancellation
Test Coverage	100% on public API
Race Detector	Clean

Non-Goals (Explicit Design Decisions)

❌ CLI/REPL interface (library-first)
❌ DAG engine or scheduler
❌ Distributed execution
❌ DataFrame abstraction
❌ Airflow/Airbyte clone
❌ Connector marketplace
❌ PDF/ML features

Installation

go get github.com/firfircelik/go-intake

Quality Gate

go test -count=1 ./...
go vet ./...
go test -race -count=1 ./...

All tests pass. Race detector clean. Zero external dependencies.

Snowflake Summit 26: What Developers, Engineers, and Architects Need to Know

Tue, 02 Jun 2026 13:00:00 +0000

Snowflake's biggest user conference yet kicked off June 1 in San Francisco, and the company arrived with a full slate of product announcements aimed squarely at builders. Here's what matters most if you're writing code, managing data infrastructure, or designing systems on Snowflake.

First, Two New Names

Before getting into the announcements, note two product rebranding moves. Cortex Code is now officially Snowflake Coco. The informal name had already taken hold with customers, so Snowflake made it official. Snowflake Intelligence is now Snowflake Cowork — a rename that reflects how the product has grown from a next-gen BI tool into a broader productivity platform for knowledge workers.

Coco Gets a Lot More Surface Area

The biggest news for developers is how many places Coco now lives. Snowflake is introducing a VS Code extension, an Excel plugin, Slack integration, and a standalone Coco Desktop app. Claude Code support is also on the list — useful for any team already using Anthropic's CLI in their workflow.

On the capability side, Coco can now handle long-running autonomous tasks via Cloud Agents, create reusable workflows through Skills, and support automations for recurring monitoring and validation tasks. New integrations with Retool, Superblocks, and Vercel let developers turn a prompt into a deployed app on Snowflake's App Runtime.

Bala Kasiviswanathan, Snowflake's VP of Developer and AI Experiences, framed the shift this way: "A year ago, building on enterprise data meant writing code, managing infrastructure, and stitching together disconnected tools for orchestration, streaming, governance, and deployment. Today, Snowflake CoCo is collapsing that complexity into a single AI-native development experience."

CEO Sridhar Ramaswamy added that Coco can already outperform frontier models on operations within Snowflake, and that partners are shifting from time-and-material billing to outcome-based engagements because of it.

The customer numbers are concrete. Fanatics reduced pipeline troubleshooting from days to hours. Thomson Reuters modernized legacy systems and delivered insights in days instead of weeks across more than 37,500 governed tables.

Coco also works across data sources beyond Snowflake — including AWS Glue, dbt, Apache Airflow, Databricks, and Postgres.

Snowflake Datastream: Managed Streaming, Built In

For data engineers managing streaming infrastructure, Snowflake introduced Datastream — a fully managed, Kafka wire protocol compatible streaming service built natively into the platform. Existing Kafka clients and applications can point to it without rewriting integrations. Under the hood, it's pure Snowflake technology, not a Kafka wrapper.

The practical appeal: streaming data routes directly into Snowflake tables with low latency, and it automatically inherits existing governance, access controls, masking policies, and lineage. No separate broker to manage. Datastream enters private preview shortly after Summit.

Apache Iceberg v3 Goes GA — and Open Data Gets Serious

Snowflake announced several moves on the open data front. Apache Iceberg v3 support is now generally available, with the broadest feature set on the market — expanded data types, cross-system change tracking, and improved semi-structured data performance.

Snowflake Storage for Apache Iceberg Tables is also GA, with bi-directional read/write access to Snowflake-managed Iceberg tables from external engines via Horizon Catalog, powered by Apache Polaris. The Iceberg REST Scan Plan API ensures security and masking policies travel with data even when accessed by external tools.

Zero-copy integrations for SAP, Salesforce, and Workday mean direct access to that data without replication. The architectural goal: work on a single, governed copy of data wherever it lives, without moving it.

Two new context layers sit on top of all of this. Horizon Context brings in explicit, customer-defined metadata — including from the Select Star acquisition, which ingests metadata from Postgres, SQL Server, Tableau, and Power BI. Cortex Sense builds implicit context automatically from usage patterns, feeding both Coco and Cowork without requiring manual configuration.

Natoma Acquisition: Governing What Agents Actually Do

This is the announcement that architects should pay the most attention to. Snowflake announced its intent to acquire Natoma, an enterprise Model Context Protocol (MCP) platform. The significance: Natoma extends governance beyond data access to cover AI agent actions — sending emails, summarizing Slack threads, updating Jira tickets, checking calendars — all from within Snowflake's governed environment with policy enforcement and full audit trails.

For anyone designing agentic systems, this signals where Snowflake is heading. The goal is to be the control plane not just for what data agents can see, but for what they can do across enterprise systems.

AI Security: Three New Layers

Four security announcements are worth noting for teams building agentic workloads:

Agent Identity gives every agent a verified identity with role-based permissions and a full audit trail. Prompt Injection Protection uses ML-driven detection to block jailbreak attempts. The Trust Center is getting an AI security package with codified best practices for agent configuration, credential usage, and data access. And multi-party authorization requires a second approver for sensitive operations — protecting against both insider threats and runaway agents.

A Few More Things Worth Noting

Snowflake Apps now supports native React development alongside AI-powered workflows, moving from prompt to a production-ready app deployed inside the security perimeter.

Adaptive Compute goes GA at Summit. It automatically determines the right mix of compute resources in real time — no manual warehouse sizing — with reported gains of up to 3.5x over first-generation warehouses.

Cowork is adding automations, background agents, and a next-generation artifacts and dashboards layer. For architects: Cowork is the governed, managed experience for everyone in the organization. Coco is the power tool for technical users who know what they're doing.

Standardizing the Ingestion Interface: OpenBrokerCSV and the MIT Importer

Fri, 29 May 2026 08:00:00 +0000

Standardizing the Ingestion Interface: OpenBrokerCSV and the MIT Importer

We open-source a financial interaction workflow — not a bag of UI widgets. The concrete artifact is @pocket-portfolio/importer (packages/importer, MIT on npm). It normalises broker-specific chaos into a Trade model the app and SDK consumers can reason about.

Package reality

From packages/importer/package.json:

Name: @pocket-portfolio/importer
License: MIT
Surface: TypeScript adapters (Robinhood, Fidelity, eToro, IBKR Flex, Koinly, TurboTax, Ghostfolio, Sharesight, and more) + universal inference for unknown CSV shapes.

Schema SSOT in-repo: packages/importer/README.md, packages/importer/src/universal/types.ts, adapter registry in src/registry.ts. A dedicated SCHEMA.md is on the roadmap; until then, cite TypeScript types and README — do not invent a phantom file path in external copy.

Client-side parse by design

Broker CSV parsing runs in the browser for the product architecture we ship — the file does not need to land on an app-server disk for transformation. That supports:

Inspectable transforms (clone the repo, read the adapter),
Reduced central warehouse temptation for raw exports,
Adversarial testing in src/universal/adversarial.ts without multi-tenant DB blast radius.

Universal mapping: truncated preview only

For unknown columns, docs/IP-TECHNICAL-MECHANISMS.md documents truncated payload for mapping assistance: headers + sample rows — full file stays client-side. This mirrors the Ask AI instinct (Part 2): send the minimum structure needed for the next step.

OpenBrokerCSV narrative

Open Portfolio surfaces OpenBrokerCSV as the procurement-facing name for the normalised interchange (app/open/openbrokercsv/page.tsx). Pocket exercises the same package under retail UX pressure — that is the harness relationship (Part 6).

What we do not promise

A plugin marketplace runtime in app/ — not shipped.
“Zero server” after import for signed-in users — Firebase still holds authoritative trades (Part 4).

Do point engineers to GitHub, Discussions, and npm — the real contributor loop (README-OSS-WORKFLOW.md).

Part 5 of Sovereign Ingestion & Stateless Inference.

Read the full Sovereign Intelligence book, explore Open Portfolio, or try Pocket Portfolio.

AI-Powered DLQ Triage with Amazon Bedrock

Mon, 11 May 2026 23:56:56 +0000

AI-Assisted Operational Intelligence for SQS DLQ Triage with Amazon Bedrock

Overview

bedrock-dlq-triage turns a dead-letter queue from a passive failure sink into an operational intelligence layer. When a message lands in an SQS DLQ, an AWS Lambda triage processor sends the payload to Amazon Bedrock, classifies the failure, assigns a severity, suggests a remediation path, and stores the result in DynamoDB for operator review. A separate replay flow lets operators re-inject only approved messages back into the source queue with idempotency safeguards.

This pattern is useful when teams handle high message volumes, heterogeneous failure modes, and repetitive investigation work. Instead of reading raw payloads one by one, operators get a structured record that highlights likely root cause, urgency, and next action. The system also creates an audit trail that supports compliance, incident review, and safer operational replay.

Problem Statement

DLQs are excellent at preserving failed messages, but they do not explain why a message failed. In real production systems, a single DLQ may contain transient integration errors, schema mismatches, downstream timeouts, poison messages, and malformed payloads, all mixed together. That forces engineers to spend time triaging failures manually before they can decide whether to fix code, replay data, or quarantine the message.

The goal of this design is to reduce mean time to understanding by using generative AI as a triage assistant. The model does not replace engineering judgment; it standardizes the first pass, applies a consistent rubric, and produces structured output that downstream workflows can consume. That makes DLQ handling faster, more repeatable, and easier to monitor.

Architecture

The core flow is simple: SQS DLQ events invoke a Lambda function named triage_processor, which calls Amazon Bedrock for classification and recommendation generation. The Lambda then writes a normalized triage record into DynamoDB, emits structured logs and custom metrics to CloudWatch, and archives prompt artifacts and raw message snapshots in S3 when needed. A DynamoDB stream triggers a second Lambda, replay_handler, which performs controlled replays back into the source SQS queue.

SQS DLQ ──► Lambda (triage_processor)
                │
                ├──► Amazon Bedrock (Claude / Titan)
                │         └── classification + severity + remediation
                │
                ├──► DynamoDB (triage_results)
                │         └── audit trail, TTL, idempotency key
                │
                └──► CloudWatch (structured logs + custom metrics)

DynamoDB Stream ──► Lambda (replay_handler)
                         └──► Source SQS Queue (controlled replay)

All compute runs inside a VPC, IAM follows least privilege, secrets live in AWS Secrets Manager, and data at rest is protected with AWS KMS across SQS, DynamoDB, and S3. DynamoDB Streams provides near-real-time change propagation for replay workflows and retains stream records for up to 24 hours. [page:1]

How Triage Works

The triage Lambda should normalize every message into a consistent envelope before calling Bedrock. A good prompt usually includes the message body, metadata such as queue name and receive count, recent error context if available, and a strict output schema like JSON. Amazon Bedrock supports Claude text completion and messages APIs, which makes it suitable for structured inference calls from Lambda. [page:2]

A practical output schema might include failure_type, severity, confidence, recommended_action, replay_safe, and operator_notes. The severity score can be rule-based, model-based, or a hybrid of both; for example, authentication failures may be marked high severity, while temporary downstream throttling may be low or medium severity. The most useful implementations also include a compact rationale so operators understand why the model reached its conclusion.

To keep results actionable, the Lambda should validate the model output before writing to DynamoDB. If the response is malformed, the function can fall back to a deterministic classification path or mark the record as needs_human_review. That keeps the system resilient even when the model output is incomplete or ambiguous.

Replay Flow

Replay should be separated from triage so that analysis and action do not happen in the same execution path. When a triage result is approved, the DynamoDB stream emits the change event, and the replay Lambda can check the replay state, idempotency key, and operator decision before sending the original payload back to the source queue. DynamoDB Streams is designed for this kind of near-real-time change capture and preserves item-level changes in order within the stream. [page:1]

The replay path should be intentionally conservative. Only messages that meet explicit criteria should be eligible for re-injection, such as replay_safe = true, a matching approved status, and a message-level deduplication key that prevents duplicates. This design is especially important when the same DLQ message may be inspected, edited, and replayed multiple times by different operators.

Data Model

A DynamoDB table such as triage_results can store the operational state of each failure. A useful partition key is the DLQ message ID or a deterministic hash of message body plus queue metadata, while a sort key can represent the triage attempt or replay version. TTL can be used to automatically expire stale records after the operational retention window, which helps keep the table small and reduces noise.

Attribute	Purpose
`message_id`	Primary identifier for the failed message.
`idempotency_key`	Prevents duplicate triage and replay actions.
`failure_type`	Model-generated or rule-assisted classification.
`severity`	Operational urgency, such as low, medium, high, or critical.
`recommended_action`	Suggested remediation step for operators.
`replay_safe`	Indicates whether controlled replay is allowed.
`status`	Triage state, such as new, reviewed, approved, replayed, or closed.
`ttl`	Expiration time for automatic cleanup.

Because DynamoDB Streams can emit the full before-and-after image of a record, it works well for audit and replay triggers when you need traceability around operator decisions. [page:1]

Observability

CloudWatch should capture both machine-readable logs and operational metrics. Useful metrics include triage volume, Bedrock invocation latency, classification confidence, replay attempts, replay success rate, and the number of messages escalated for human review. Alarms can trigger when triage failures spike, when Bedrock latency degrades, or when replay activity exceeds a safe threshold.

Structured logs are especially valuable because they let engineers correlate one DLQ message with its classification, Bedrock prompt version, model response, and replay outcome. That observability makes the system debuggable and also gives teams a feedback loop for improving prompt design and classification rules over time.

Security And Governance

The system should use least-privilege IAM roles for each Lambda function, and Bedrock permissions should be scoped to the specific model ARNs in use. Secrets and tokens belong in AWS Secrets Manager rather than environment variables, and KMS should protect data in SQS, DynamoDB, and S3. Keeping the workload inside a VPC can help align the design with enterprise network controls and inspection requirements.

Governance also benefits from storing prompt templates, model versions, and replay approvals alongside the triage record. That gives teams a full audit trail for incident response and makes it easier to reproduce a decision later. In regulated environments, this history is often as important as the classification itself.

Implementation Notes

A strong implementation starts with a narrow prompt and a fixed response schema. The model should be asked to classify the failure, assign severity, recommend an action, and state whether replay is safe, all in machine-readable JSON. The Lambda should retry cautiously, enforce timeouts, and reject responses that do not match the expected schema.

Idempotency is critical for both triage and replay. The safest approach is to derive an idempotency key from stable message attributes and record processing state in DynamoDB before any replay action occurs. That protects the system from duplicate deliveries, operator retries, and stream reprocessing.

Why It Matters

This architecture changes DLQ handling from manual troubleshooting into an AI-assisted operational workflow. Engineers spend less time reading raw failure payloads and more time addressing the actual root cause. Operators also gain a consistent, auditable decision trail for every failure that enters the DLQ.

The best result is not just automation, but better judgment at scale. Bedrock supplies the first-pass reasoning, Lambda orchestrates the workflow, DynamoDB preserves state, and CloudWatch keeps the system observable.

Eradicating Operational Drag: Architecting a Resilient Data Ingestion Pipeline

Wed, 06 May 2026 12:24:56 +0000

In the lifecycle of a scaling technology company, manual data entry is a silent killer of engineering velocity.

I was recently tasked with solving a massive operational bottleneck for a mobile telecommunications client. Their core product relied on a perpetually up-to-date, deeply categorized database of thousands of mobile devices encompassing highly specific hardware specs, varying image URLs, and user metadata.

Historically, this required human operators spending days manually copying data from centralized industry hubs. It was slow, wildly prone to human error, and completely unscalable. Every time a manufacturer announced a new lineup, the client's operations team was paralyzed by manual data entry.

The directive was clear: automate it. However, as a senior engineer, you know that writing a quick Python or Node.js scraping script is easy; building a resilient, scalable data ingestion pipeline that survives network timeouts, DOM mutations, and rate limits is a complex architectural challenge.

Here is a case study on my thought process, the engineering hurdles, and how I architected a robust ingestion engine that permanently eliminated this operational drag.

The Architectural Challenge: The Web is Hostile

When transitioning from manual collection to automated ingestion, you are not just writing a script you are building a distributed system that must interact with external, highly volatile environments.

Before writing any logic, I mapped out the three primary failure vectors of web-based data extraction:

The Fragility of the DOM: HTML is not a reliable API. Websites redesign their UIs, class names change, and tables nest unpredictably. A brittle parser will crash the entire pipeline the moment a target website updates its CSS.
Network Hostility & Rate Limiting: Target servers do not want you scraping thousands of pages concurrently. If you run a massive for-loop of HTTP requests, your IP will be aggressively banned within seconds.
State Management & Idempotency: If the pipeline runs twice, or crashes halfway through, it cannot create duplicate database entries or corrupt existing data. The system must be deterministic.

Phase 1: Engineering a Resilient Parsing Matrix

The first problem to solve was the extraction of unstructured data. Mobile phone specifications are often buried inside deeply nested, inconsistent HTML tables.

Instead of hardcoding DOM traversal logic directly into the execution flow, I separated the "Fetching" from the "Parsing." I designed the system to fetch the raw HTML payload and pass it to an isolated, stateless parsing module using Cheerio (a high-performance, server-side implementation of core jQuery logic).

The Thought Process: To protect against UI changes, I avoided tying the parser to brittle visual CSS classes (like .red-text-bold). Instead, I targeted semantic data attributes (e.g., span[data-spec=battery]). I structured the parser as a "Configuration Matrix", a simple mapping dictionary that linked our internal database schema fields to specific HTML selectors.

If the target website updated its UI, the core engine wouldn't break; we simply updated a single line in the configuration matrix. This transformed a fragile web scraper into a robust, maintainable data transformer.

Phase 2: Respectful Concurrency and the Worker Queue

The most common mistake junior engineers make with automation is aggressive concurrency. Firing 5,000 HTTP GET requests simultaneously will trigger DDoS protection (like Cloudflare) on the target server, resulting in immediate IP blacklisting.

The Thought Process:
To ensure high availability and prevent network bans, the ingestion engine had to be "polite" but persistent. I architected the flow using an asynchronous queue system.

The Indexer: The master process performs a single, lightweight pass over the target's brand directory, identifying pagination limits and gathering the specific URLs for all 5,000+ devices.
The Queue: Instead of fetching these URLs immediately, the indexer pushes them into a task queue.
Throttled Workers: A fleet of worker nodes pulls URLs from the queue at a strictly controlled rate. I implemented a Token Bucket algorithm to enforce a maximum outbound request limit (e.g., 5 requests per second).
Exponential Backoff: If a worker receives a 429 Too Many Requests or 503 Service Unavailable error, it doesn't crash. It relies on exponential backoff waiting 2 seconds, then 4, then 8 before retrying.

This respectful, queued concurrency guaranteed that we could ingest massive amounts of data continuously without ever triggering hostile network defenses.

Phase 3: Guaranteeing Idempotent Database Writes

The final, and arguably most critical, piece of the architecture was ensuring data integrity. The ingestion pipeline runs as a scheduled cron job (e.g., every 24 hours) to catch newly released devices.

If the pipeline grabs data for a phone that already exists in our database, performing a blind INSERT operation would result in thousands of duplicated rows.

The Thought Process:
I engineered the final stage of the pipeline to be strictly idempotent.

Before committing data, the worker generates a unique hash based on the device's brand and model name. It queries our primary database for this hash.

If the hash does not exist, it executes an INSERT.
If the hash exists, it runs a differential check. It compares the newly scraped specifications against the existing database record. If the target website updated a specification or added a new image URL, the worker executes a surgical UPDATE on only the modified fields.

This delta-update logic drastically reduced database write-load and guaranteed that running the crawler 100 times yielded the exact same perfect state as running it once.

The Engineering Impact

The true value of software engineering is measured in business leverage. By replacing a human workflow with a structurally sound, event-driven ingestion microservice, the impact was profound.

Eradicated Operational Drag: A process that previously paralyzed the operations team for days was reduced to a silent background job that autonomously updates the platform while everyone sleeps.
Flawless Data Integrity: We eliminated human transcription errors entirely. The database became a mathematically accurate reflection of the source data.
Architectural Scalability: Because the engine decoupled fetching, parsing, and state management, onboarding a completely new data source simply requires creating a new CSS parsing matrix, rather than rewriting the core infrastructure.

Automation is not merely writing scripts to mimic human clicks; it is about architecting resilient, deterministic pipelines that empower humans to stop acting like machines and focus entirely on high-value business logic.

Dealing with massive data orchestration bottlenecks or looking to architect resilient automated pipelines? Let's Connect!
I am Ankit Jaiswal, a Senior Full Stack AI Engineer specializing in system design, distributed architectures, and building robust, cloud-agnostic SaaS platforms.

How to Connect Power BI to a SQL (PostgreSQL) Database and Build a Unified Dashboard

Mon, 04 May 2026 13:29:57 +0000

Introduction

Power BI is a business intelligence (BI) and data visualization tool from Microsoft. It enables analysts and business users to transform raw data into interactive dashboards and reports. Companies use Power BI to analyze sales, customer behavior, inventory trends, and other critical business metrics.

SQL databases, like PostgreSQL, are widely used to store and manage structured data. Connecting Power BI to SQL databases allows analysts to retrieve, clean, transform, and model data efficiently, creating a single source of truth for dashboards and KPIs.

In this article, we’ll walk through:

Connecting Power BI to local and cloud PostgreSQL databases
Loading raw data and applying transformations
Combining datasets from multiple sources
Adding a Date Dimension table (DimDate) for KPIs and filtering
Preparing a dashboard-ready model for GitHub and reporting

What You Need Before Connecting

Power BI Desktop installed
PostgreSQL database connection details (server, port, database, username, password)
For cloud connections: SSL certificate (CA certificate)
Model view showing relationships across cloud and local tables

Connecting to a Local PostgreSQL Database

Open Power BI Desktop.
Click Get Data → PostgreSQL database.

Enter your connection info:

Server:

localhost:5432 (replace 5432 with your PostgreSQL port)

Database: Name of your database

Enter your username and password.

Click OK to connect.

Note: The localhost:port syntax ensures Power BI connects to the correct PostgreSQL server port.

In the Navigator window, select tables to load, then click Load (or Transform Data).

Connecting to a Cloud PostgreSQL Database (Aiven or Similar)

Cloud databases require secure SSL connections.

Guide to Aiven Connection

From your cloud provider, obtain:

Host
Port
Database name
Username and password
SSL certificate (CA certificate)
Install the CA certificate on your system:
Open the downloaded .crt or .pem file.
Install it in the trusted root certificate store:

Windows: Right-click → Install Certificate → Place in “Trusted Root Certification Authorities”
Linux/macOS: Follow OS instructions to add to the system or user trust store

Installing the CA certificate allows Power BI to validate the server’s identity and establish a secure connection.

In Power BI Desktop:

Go to Get Data → PostgreSQL database
Enter Server

your-cloud-host:5432

and Database

Check Use SSL certificate and select the installed certificate
Enter your username and password

If successful, it opens your aiven cloud database as shown below
Choose tables from your database

Click OK and load the tables

Load Raw Data First

It’s important to load raw data before applying transformations:

Load all tables (customers, products, sales, inventory) first.
Review the raw data to assess missing values, incorrect or inconsistent formats and unnecessary columns
This helps you understand which fields need cleaning and transformation before building dashboards.

Filtering and Transformation

Use Power BI’s Power Query Editor for data preparation:

Remove unnecessary columns – keep only fields needed for analysis.
Rename columns – make names descriptive.
Filter rows – e.g., remove canceled sales or test records.
Change data types – ensure numeric and date fields are correct.
Create calculated columns – e.g., total sales = quantity × price.
Handle missing values – fill, replace, or remove nulls.

Creating Relationships Between Tables

Once data is cleaned:

Switch to Model View.
Create relationships:

`sales.customer_id → customers.customer_id

sales.product_id → products.product_id

inventory.product_id → products.product_id`

Validate relationships for correct cardinality and cross-filtering.

Relationships ensure Power BI can accurately summarize data across multiple tables.

Data Modeling Basics

Star schema
Fact table (sales) connected to dimension tables (customers, products, inventory).
Cardinality
One-to-many or many-to-one relationships.
Filter directions
Define single or bidirectional filters depending on your analysis needs.

Proper modeling ensures dashboards are interactive and accurate.

Why SQL Skills Matter for Power BI Analysts

Data extraction: Retrieve exactly the data you need.
Filtering & aggregation: Pre-process data for better performance.
Data cleaning & transformation: Prepare data before loading it into Power BI.
Performance optimization: Efficient queries improve dashboard responsiveness.

By combining SQL knowledge with Power BI, analysts can build powerful, reliable dashboards that drive business decisions.

Conclusion

Connecting PostgreSQL to Power BI empowers analysts to turn structured data into actionable insights. Whether using local or cloud databases, establishing secure and reliable connections ensures that the data you analyze is accurate and up-to-date. By first loading raw data, applying necessary transformations, and creating relationships, including a Date table for time-based analysis, you build a robust data model ready for interactive dashboards.

Mastering this workflow not only simplifies KPI calculations and trend analysis but also highlights the importance of SQL skills for preparing and managing data. Ultimately, integrating PostgreSQL with Power BI provides a seamless bridge between data storage and visualization, enabling smarter, data-driven business decisions.

Market Data Integrator: Consuming Real-Time Data with Python, MongoDB, and Kafka

Sun, 03 May 2026 20:33:47 +0000

Hey folks!

Continuing the My Broker B3 series, today we'll build the first microservice in the ecosystem: the trading-broker-market-data.

This service has a simple and well-defined responsibility: fetch real quotes for Brazilian assets from the Brapi API and distribute them to the ecosystem — persisting the history in MongoDB and publishing updated prices to Kafka for other services to consume.

️ This Service's Role in the Ecosystem

[brapi.dev] ◀── HTTP ── [broker-market-data-api]
                                  │
                    ┌─────────────┴──────────────┐
                    ▼                             ▼
             MongoDB                           Kafka
        (price_history)          (trading-assets-market-data-v1)
                                               │
                                    [broker-asset-api] (consumer)

It's the entry point for real data into the system. Without it, the ecosystem operates on static prices.

MVP Focus

The goal here is clear: have real quotes available in Kafka for other services to react to. In this phase, I prioritized:

Scheduled ingestion of 50 assets (Blue Chips, REITs and ETFs)
Historical persistence in MongoDB for audit
Kafka event publishing for downstream consumers
Rate limit protection for the Brapi API (free plan)
Immediate execution on service startup

️ Tech Stack

Technology	Usage
Python 3.12	Service runtime
requests	HTTP client for Brapi API
pymongo	MongoDB persistence
confluent-kafka	Kafka producer
schedule	Periodic scheduling
python-dotenv	Environment variables

️ The Implementation

The Ingestion Loop

The heart of the service is the run_ingestion() function, which iterates over the watchlist, fetches each asset's price and distributes the data:

def run_ingestion():
    """Fetches market data from Brapi, persists to MongoDB and publishes to Kafka."""
    logger.info("Market data ingestion job started")

    token = os.getenv("BRAPI_TOKEN")
    if not token:
        logger.error("BRAPI_TOKEN not set. Aborting.")
        return

    # Producer created per run to avoid crash on import if Kafka is unavailable
    producer = Producer({
        'bootstrap.servers': os.getenv("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092"),
        'client.id': 'broker-market-data-api'
    })

    mongo = MongoRepository()
    success_count = 0
    error_count = 0

    for ticker in WATCHLIST:
        try:
            url = f"https://brapi.dev/api/quote/{ticker}?token={token}"
            response = requests.get(url, timeout=10)
            response.raise_for_status()

            result = response.json()["results"][0]
            payload = {
                "ticker": result.get("symbol"),
                "name": result.get("longName"),
                "short_name": result.get("shortName"),
                "price": result.get("regularMarketPrice"),
                "volume": result.get("regularMarketVolume"),
                "updated_at": result.get("regularMarketTime")
            }

            # 1. Persist historical record in MongoDB
            mongo.save(payload.copy())

            # 2. Publish event to Kafka
            producer.produce(
                topic=TOPIC_NAME,
                key=ticker,
                value=json.dumps(payload).encode('utf-8'),
                callback=delivery_report
            )
            producer.poll(0)

            success_count += 1
            time.sleep(0.5)  # Rate limiting — respect Brapi free plan

        except Exception as e:
            error_count += 1
            logger.error(f"Error processing ticker {ticker}: {e}")

    producer.flush()
    logger.info(f"Ingestion completed: {success_count} succeeded, {error_count} failed")

Important design decisions:

Why create the Producer inside the function? If it were at module level, an unavailable Kafka at import time would crash the script before executing any logic. Creating it inside the function means a Kafka failure doesn't prevent execution — only that run fails.

Why try/catch per ticker? If one asset fails (timeout, invalid ticker), the loop continues for the rest. The error_count at the end shows how many failed without stopping the work.

The time.sleep(0.5) — respects the rate limit of Brapi's free plan. For 50 assets, the full run takes ~25 seconds.

The Scheduling

if __name__ == "__main__":
    logger.info(f"Starting — sync every {SYNC_INTERVAL_MINUTES} minutes")

    # Run immediately on startup, then on schedule
    run_ingestion()

    schedule.every(SYNC_INTERVAL_MINUTES).minutes.do(run_ingestion)

    while True:
        schedule.run_pending()
        time.sleep(1)

Why run immediately on startup? If the service restarts, we don't want to wait 30 minutes for data in Kafka. The first execution is immediate, subsequent ones follow the schedule.

The Kafka Payload

{
  "ticker": "PETR4",
  "name": "Petróleo Brasileiro S.A. - Petrobras",
  "short_name": "PETROBRAS PN",
  "price": 35.50,
  "volume": 15234567,
  "updated_at": 1714234567
}

Topic: trading-assets-market-data-v1
Key: asset symbol (e.g. PETR4) — ensures events for the same asset go to the same partition

The Watchlist

50 assets covering the main B3 segments:

WATCHLIST = [
    # Blue Chips
    "PETR4", "PETR3", "VALE3", "ITUB4", "BBDC4", "BBAS3", ...
    # REITs / FIIs
    "MXRF11", "HGLG11", "KNIP11", "XPLG11", ...
    # ETFs
    "BOVA11", "IVVB11"
]

Security: Token Out of Code

# .env — never committed
BRAPI_TOKEN=your_token_here

# application — no fallback, required variable
token = os.getenv("BRAPI_TOKEN")
if not token:
    logger.error("BRAPI_TOKEN not set. Aborting.")
    return

Never expose credentials as default values in public repositories.

✅ Validating the Execution

With the service running locally:

✅ First run executes immediately on startup
✅ MongoDB receives documents with created_at automatically
✅ Kafka receives messages with key = ticker
✅ Scheduler waits 30 minutes for the next round

What's Next?

With broker-market-data-api publishing quotes to Kafka, the next step is b3-market-sync-api — the Java service that consumes these events and synchronizes prices to Redis, feeding the Matching Engine with real-time data.

About the Series

⬅️ Previous Post: Infrastructure as Code

➡️ Next Post: Syncing the Real Market: Consuming Brapi and Feeding Redis with Spring Boot

Series Index: Series Roadmap

Links:

How to Build a Serverless Data Lake Foundation with AWS Glue

Sat, 02 May 2026 12:41:31 +0000

1. Introduction

Welcome to this comprehensive tutorial on building a Serverless Data Lake Foundation using AWS Glue. By the end of this guide, you will be able to design and implement a robust, automated pipeline that extracts raw data from Amazon S3, transforms it into an optimized analytics-ready format, and makes it available for querying via Amazon Athena. This architectural pattern is highly useful because it completely eliminates the need to manage underlying servers, clusters, or infrastructure, allowing you to focus entirely on your core data logic. Furthermore, it automatically scales out alongside your data volume, ensuring long-term cost-effectiveness since you only pay for the exact compute resources consumed during the active transformation process. Whether you are dealing with daily batch processing operations or aggregating vast amounts of historical data, mastering this foundational architectural pattern is a critical and necessary milestone in modern data engineering.

2. Prerequisites

Before starting this step-by-step implementation, ensure you have the following tools and foundational knowledge prepared:

AWS Account: An active Amazon Web Services account with administrative or sufficient permissions to create S3 buckets, configure IAM roles, and deploy AWS Glue jobs.
Basic Python or PySpark Knowledge: Familiarity with basic data manipulation concepts, variables, and data structures using Python or Apache Spark.
Understanding of Cloud Storage: General knowledge of how object storage systems function, specifically Amazon S3 concepts including buckets and prefixes.
IAM Fundamentals: A foundational understanding of Identity and Access Management principles to allow distinct AWS services to communicate securely.

3. Step-by-Step

Step 1: Architecting the Serverless Data Lake Foundation

What to do: Review the end-to-end data flow and understand the specific roles of each AWS service before provisioning any cloud resources. The architecture requires Amazon S3 for storage, Amazon EventBridge for orchestration, AWS Glue for processing and metadata management, and Amazon Athena for consumption.

Why do it: Establishing a clear mental model and a visual blueprint prevents configuration errors later in the process. It ensures that all AWS services are logically connected, network paths are understood, and the separation between storage and compute is maintained throughout the pipeline design.

Example:

Step 2: Configuring Amazon S3 Storage Zones

What to do: Access the Amazon S3 console and create a new S3 bucket with a unique name. Inside this bucket, create two distinct folders (prefixes). Name the first folder raw/ to represent the landing zone for your incoming, unprocessed data. Name the second folder curated/ to represent the destination for your clean, transformed data.

Why do it: Segregating data by its processing state is a fundamental best practice in data engineering. It prevents accidental overwrites of source data, simplifies access control policies, and optimizes query performance by keeping raw, unoptimized files strictly separated from the analytical query engine.

Example:

# 1. Define the core S3 Data Lake Bucket
resource "aws_s3_bucket" "datalake_production" {
  bucket = "enterprise-datalake-production"

  tags = {
    Environment = "Production"
    Purpose     = "Serverless Data Lake Foundation"
    ManagedBy   = "Terraform"
  }
}

# 2. Enable Bucket Versioning (Recommended Data Lake Best Practice)
resource "aws_s3_bucket_versioning" "datalake_versioning" {
  bucket = aws_s3_bucket.datalake_production.id
  
  versioning_configuration {
    status = "Enabled"
  }
}

# 3. Create the 'raw' subdirectory (Landing Zone)
resource "aws_s3_object" "raw_zone" {
  bucket = aws_s3_bucket.datalake_production.id
  key    = "raw/"
  
  # Defines the object as a directory rather than a standard file
  content_type = "application/x-directory"
}

# 4. Create the 'curated' subdirectory (Gold/Analytics Zone)
resource "aws_s3_object" "curated_zone" {
  bucket = aws_s3_bucket.datalake_production.id
  key    = "curated/"
  
  content_type = "application/x-directory"
}

Step 3: Setting Up IAM Permissions

What to do: Navigate to the AWS Identity and Access Management console and create a new Service Role specifically assigned to AWS Glue. Attach the AWS managed policy named AWSGlueServiceRole. Additionally, create and attach an inline policy that explicitly grants s3:GetObject and s3:PutObject permissions targeted strictly at the S3 bucket you created in the previous step.

Why do it: AWS services operate under the strict principle of least privilege. AWS Glue cannot access your raw data, write processed files to S3, or publish execution logs to Amazon CloudWatch without explicit, cryptographically secure permissions defined by an IAM Role.

Example:

# 1. Define the Trust Relationship (Assume Role Policy) for AWS Glue
data "aws_iam_policy_document" "glue_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    
    principals {
      type        = "Service"
      identifiers = ["glue.amazonaws.com"]
    }
  }
}

# 2. Create the IAM Role
resource "aws_iam_role" "glue_execution_role" {
  name               = "GlueDataLakeExecutionRole"
  assume_role_policy = data.aws_iam_policy_document.glue_assume_role_policy.json

  tags = {
    Environment = "Production"
    Purpose     = "Serverless Data Lake Foundation"
    ManagedBy   = "Terraform"
  }
}

# 3. Attach the core AWS Managed Policy for Glue Service operations
# This grants necessary permissions for CloudWatch logging, ENI creation (if VPC bounded), etc.
resource "aws_iam_role_policy_attachment" "glue_service_managed_policy" {
  role       = aws_iam_role.glue_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}

# 4. Define a restrictive inline policy for the specific Data Lake S3 bucket
data "aws_iam_policy_document" "s3_datalake_access_policy" {
  statement {
    sid    = "AllowDataLakeReadWrite"
    effect = "Allow"
    
    actions = [
      "s3:GetObject",
      "s3:PutObject"
    ]
    
    resources = [
      # Restrict access exclusively to the specific bucket and defined prefixes
      "arn:aws:s3:::enterprise-datalake-production/raw/*",
      "arn:aws:s3:::enterprise-datalake-production/curated/*"
    ]
  }
}

# 5. Create and attach the inline S3 policy to the IAM Role
resource "aws_iam_role_policy" "s3_datalake_inline_policy" {
  name   = "RestrictedS3DataLakeAccess"
  role   = aws_iam_role.glue_execution_role.id
  policy = data.aws_iam_policy_document.s3_datalake_access_policy.json
}

Step 4: Orchestrating the Pipeline Execution Flow

What to do: Define the exact sequence of automated events that will trigger and execute your ETL process. Configure an Amazon EventBridge rule that listens for an ObjectCreated event in your S3 raw/ prefix. Set the target of this rule to invoke your AWS Glue Job automatically whenever a new file finishes uploading.

Why do it: Automation entirely removes the need for manual intervention and scheduling guesswork. By structurally mapping the sequence, you ensure that compute resources are only utilized when new data is actually present, maximizing efficiency and minimizing idle costs.

Example:

Step 5: Authoring the AWS Glue Transformation Job

What to do: Open the AWS Glue Studio visual editor and create a new Spark ETL job. Configure the source node to read from your S3 raw/ path. Add a transformation node to clean the data, such as dropping null values, mapping column names to standard conventions, and casting data types correctly. Finally, configure the target node to write the data back to your S3 curated/ path, explicitly selecting Parquet as the output format.

Why do it: The transformation phase is where raw information becomes valuable data. Writing the output in Apache Parquet format is critical. Parquet is a highly optimized, columnar storage format that compresses data heavily, dramatically reduces long-term S3 storage costs, and exponentially speeds up subsequent analytical queries by allowing engines to skip irrelevant columns.

Example (Terraform):

resource "aws_glue_job" "datalake_curation_job" {
  name     = "raw-to-curated-parquet-job"
  role_arn = aws_iam_role.glue_execution_role.arn # From the previous IAM step

  # Defines the execution engine and script location
  command {
    name            = "glueetl"
    script_location = "s3://enterprise-datalake-production/scripts/etl_job.py"
    python_version  = "3"
  }

  default_arguments = {
    "--job-language"                     = "python"
    "--enable-metrics"                   = "true"
    "--enable-continuous-cloudwatch-log" = "true"
  }

  # Cost and performance optimization
  glue_version      = "4.0"
  worker_type       = "G.1X"
  number_of_workers = 2

  tags = {
    Environment = "Production"
    Purpose     = "Serverless Data Lake Foundation"
    ManagedBy   = "Terraform"
  }
}

Example (Glue Job):

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)

# ==============================================================================
# Node 1: S3 Source (Reads raw CSV data)
# ==============================================================================
S3Source_node1 = glueContext.create_dynamic_frame.from_options(
    format_options={"quoteChar": '"', "withHeader": True, "separator": ","},
    connection_type="s3",
    format="csv",
    connection_options={
        "paths": ["s3://enterprise-datalake-production/raw/"], 
        "recurse": True
    },
    transformation_ctx="S3Source_node1"
)

# ==============================================================================
# Node 2: Transform - ApplyMapping (Cleans and maps data types)
# ==============================================================================
ApplyMapping_node2 = ApplyMapping.apply(
    frame=S3Source_node1,
    mappings=[
        ("id", "string", "user_id", "string"),
        ("name", "string", "full_name", "string"),
        ("timestamp", "string", "event_time", "timestamp")
    ],
    transformation_ctx="ApplyMapping_node2"
)

# ==============================================================================
# Node 3: S3 Target (Writes optimized Parquet data)
# ==============================================================================
S3Target_node3 = glueContext.write_dynamic_frame.from_options(
    frame=ApplyMapping_node2,
    connection_type="s3",
    format="glueparquet",
    connection_options={
        "path": "s3://enterprise-datalake-production/curated/", 
        "partitionKeys": []
    },
    format_options={"compression": "snappy"},
    transformation_ctx="S3Target_node3"
)

job.commit()

Example (Glue Studio):

Step 6: Cataloging the Data with AWS Glue Crawlers

What to do: Navigate to the Crawlers section within the AWS Glue console. Create a new crawler and configure its target repository to point directly at your curated/ S3 folder. Assign the IAM role created in Step 3 to the crawler. Set the crawler to run on a daily schedule or to be triggered immediately after the AWS Glue transformation job completes successfully.

Why do it: Data sitting in S3 is useless to business analysts without structural context. The Crawler automatically inspects your Parquet files, infers the underlying schema including column names and data types, and registers this structural definition as a logical table within the centralized AWS Glue Data Catalog.

Step 7: Querying the Transformed Data with Amazon Athena

What to do: Navigate to the Amazon Athena console. Ensure you have set up an Athena query result location in a separate S3 bucket. Select the database created by your Glue Crawler from the dropdown menu. Write and execute standard SQL queries against your newly cataloged table to analyze the data.

Why do it: This final step validates the structural integrity of your entire pipeline. Athena uses the metadata schema stored in the Data Catalog to seamlessly execute SQL queries directly against the compressed Parquet files residing in S3. This proves that your serverless architecture is fully operational and ready for business intelligence workloads.

4. Common Troubleshooting

Problem 1: Out of Memory Errors in Glue Jobs
This issue frequently occurs due to data skew, where one worker node attempts to process significantly more data than others, or when attempting to process thousands of excessively small files simultaneously.
Solution: Resolve this by navigating to the Glue Job details and increasing the worker type from the standard configuration to G.1X or G.2X, providing more memory per executor. Alternatively, utilize Spark's .repartition() command within your script to evenly distribute the workload across the entire serverless cluster before executing wide transformations.

Problem 2: Glue Crawler Creates Multiple Unwanted Tables
If your curated S3 data utilizes a deeply nested, highly partitioned, or logically inconsistent folder structure, the AWS Glue Crawler might misinterpret the subfolders as entirely separate tables instead of partitions of a single dataset.
Solution: To rectify this, edit the Crawler configuration settings. Locate the section labeled "Grouping behavior for S3 data" and explicitly check the option to "Create a single schema for each S3 path". This forces the crawler to treat all files under the root prefix as part of the same table.

Problem 3: Amazon Athena Queries Return Zero Records
If the table successfully appears in the Data Catalog but standard SELECT queries return absolutely empty results, the S3 data is highly likely partitioned, but the Data Catalog remains unaware of the new partition directories.
Solution: Execute the MSCK REPAIR TABLE your_table_name; command directly within the Athena query editor. This command scans the S3 path, loads the missing partition metadata into the Data Catalog, and immediately makes the data queryable.

5. Conclusion

Congratulations on successfully architecting and understanding the Serverless Data Lake Foundation. You have learned how to decouple storage from compute layers, automate data transformations using event-driven principles, and seamlessly bridge raw information into a highly optimized, queryable state. By mastering these core components, you now possess the foundational engineering skills required to build scalable, enterprise-grade data platforms. As a logical next step, consider exploring AWS Step Functions to orchestrate more complex, multi-job dependencies, or experiment with Apache Iceberg within your AWS Glue jobs to introduce robust transactional capabilities and time-travel querying to your data lake.

Data's Expiration Date

Thu, 30 Apr 2026 01:42:18 +0000

Here's something nobody tells you when you start in data engineering: data goes bad.

Not like "error 500" bad. More like "left the milk out" bad. Silent. Subtle. Dangerous.

I learned this the hard way. Built a beautiful pipeline once: clean transformations, perfect schema, the works. Proud of myself. Real proud. Then, two weeks later, predictions started drifting. Turns out the upstream API had quietly started serving cached responses after an update. My data looked fresh. Smelled fresh. But it was last week's data wearing a new timestamp like a cheap disguise.

That's the thing about data. It has a shelf life. And we keep pretending it doesn't.

Your customer segmentation from Q2? Adorable. Your users moved on in Q3. That fraud detection model from last year? Cute. Fraudsters iterate faster than your sprint cycle.

So now I check expiration dates like I'm grocery shopping. When was this actually generated? Has the schema shifted? Is the distribution what my model expects? Can I trace it back to source?

Because here's the truth: data without a valid date isn't data. It's a story you're telling yourself. And in production, self-deception is the most expensive bug you'll ship.

Stop Treating Your Data Pipeline Like a Script - Treat It Like a Product

Mon, 27 Apr 2026 19:04:09 +0000

I learned this the hard way after watching a simple ETL job torch our weekend.

When I started in data engineering, I thought my job was writing scripts that moved data from A to B. Clean, logical, done. I was wrong.

The difference between a pipeline that works and one that survives? Three things nobody told me:

Observability first, logic second - If you can't see what's happening inside your pipeline, you're flying blind. Dashboards aren't optional; they're infrastructure.

Data contracts over hope - Assume your upstream source will silently betray you. Schema changes, null explosions, timestamp format switches at 2am. Code defensively or suffer.

Idempotency is non-negotiable - Rerunning yesterday's job shouldn't duplicate records or corrupt state. Build for reruns, not just first runs.

The mindset shift: Your pipeline isn't finished when it runs. It's finished when it runs reliably while you're sleeping.

What's one lesson you learned after your first production failure? Drop it below.

I Built a Real-Time Crypto Analytics Pipeline for $0.01/Month — Here's the Full Architecture

Mon, 27 Apr 2026 09:35:03 +0000

How I combined Apache Flink, Redpanda, Airflow, dbt Cloud, and Grafana to track Bitcoin, Ethereum, Solana, BNB, and Cardano in real time — all running on Google Cloud for less than a cup of coffee per month.

If you've been learning data engineering, you've probably built pipelines that move CSV files from A to B. Nothing wrong with that — but real data engineering interviews are starting to ask a different question: Can you handle data that never stops arriving?

This article walks through CoinPulse, a project I built from scratch to answer that question. It's a hybrid streaming and batch crypto analytics pipeline that:

Consumes live trade events from Binance's WebSocket feed at ~1 tick per second per coin
Processes them with Apache Flink using 1-minute tumbling windows
Enriches the streaming data daily with CoinGecko market metadata via Airflow
Transforms everything with dbt Cloud
Serves a live, auto-refreshing Grafana dashboard — publicly accessible

Total cloud cost: ~$0.01/month.

Here's how I built it.

The Problem I Was Solving

Crypto market data has two distinct tempos:

Real-time (milliseconds to seconds): Price ticks from exchanges. You need stream processing here — you can't batch-load ticks after the fact and call it "live." The data volume is too high to dump raw into a warehouse row by row.

Daily (slow-moving context): Market cap, trading volume, rankings, OHLC candlesticks. These change once a day. You don't need Flink for this — a simple scheduled API call works fine.

The interesting engineering challenge is: how do you join these two tempos cheaply and reliably?

The naive answer — stream everything into BigQuery using the Streaming Insert API — costs money at scale. The smarter answer is what I built: stream to GCS first, batch-load to BigQuery for free, and join the two lanes in dbt.

Architecture Overview

Before diving in, here's the full picture:

STREAMING LANE
Binance WebSocket → Python Producer → Redpanda → PyFlink → GCS → BigQuery
                    (real-time ticks)  (Kafka)  (windows)  (JSONL) (free load)

BATCH LANE  
CoinGecko API → Airflow DAG (7 tasks) → GCS → BigQuery
                (daily @ 06:00 UTC)    (Parquet)  (free load)

TRANSFORMATION LAYER
BigQuery → dbt Cloud → crypto_staging.* → crypto_mart.*
           (daily @ 07:00 UTC)

VISUALIZATION
BigQuery → Grafana Cloud → 6 panels, auto-refresh 30s
           (BigQuery plugin, free)

Everything runs locally in Docker except the data warehouse (BigQuery) and transformation/visualization layers (dbt Cloud + Grafana Cloud). This is the "local-to-cloud" pattern — keep compute on your machine, use GCP only for storage and analytics.

The Streaming Lane: Binance → Redpanda → PyFlink → GCS

Why Binance Instead of CoinCap?

I originally planned to use CoinCap's WebSocket feed. Halfway through the build, I discovered CoinCap now requires an API key even for the free tier. Binance's WebSocket, however, is completely open — no account, no key, no rate limits on the trade stream.

The URL pattern is:

wss://stream.binance.com:9443/stream?streams=btcusdt@trade/ethusdt@trade/solusdt@trade/bnbusdt@trade/adausdt@trade

Each message looks like:

{
  "stream": "btcusdt@trade",
  "data": {
    "s": "BTCUSDT",
    "p": "76034.12",
    "q": "0.00150000",
    "T": 1776875894886
  }
}

The Python Producer

The producer is a Docker container running websockets + confluent-kafka. It connects to Binance, maps Binance pair names to readable coin names (BTCUSDT → bitcoin), and publishes JSON messages to a Redpanda topic:

message = {
    "symbol": "bitcoin",
    "price_usd": 76034.12,
    "event_timestamp": "2026-04-23T18:31:34+00:00"
}
producer.produce(topic="crypto-prices", key="bitcoin", value=json.dumps(message))

Why Redpanda Instead of Kafka?

Kafka requires Zookeeper. In a local Docker setup, that's three containers just for the message broker (Zookeeper, Kafka broker, Schema Registry if you want it). Redpanda is Kafka-API compatible — meaning PyFlink's Kafka connector works with it unchanged — but runs as a single binary, no Zookeeper. It starts in under 3 seconds and uses ~512MB RAM instead of Kafka's 2-3GB.

The PyFlink Job

The Flink job does the heavy lifting. It consumes from Redpanda, applies a 1-minute tumbling event-time window per coin symbol, and computes:

avg_price — average price within the window
min_price / max_price — price range
price_stddev — standard deviation (volatility proxy)
open_price / close_price — first and last tick
record_count — number of ticks received

Event-time windows (as opposed to processing-time) are important here. They use the timestamp embedded in the Binance message, so late-arriving messages (within a 10-second watermark) land in the correct window rather than the current one.

The output of each completed window is a JSONL file written to GCS:

gs://coinpulse-data-lake/streaming/2026/04/23/18/stream_bitcoin_20260423_183134.jsonl

A scheduled BigQuery Load Job (free) then loads these files into crypto_raw.stream_prices, partitioned by HOUR on event_timestamp and clustered by symbol.

The key insight here: I never use BigQuery's Streaming Insert API, which charges ~$0.01 per 200MB. Load Jobs are completely free. The data is 1 minute old in the warehouse instead of 0 seconds old — but for a dashboard refreshing every 30 seconds, nobody notices.

The Batch Lane: CoinGecko → Airflow → GCS → BigQuery

The Airflow DAG

The batch pipeline runs as a 7-task Airflow DAG at 06:00 UTC daily with two parallel branches:

fetch_coingecko → transform_to_parquet → upload_to_gcs → load_to_bigquery
fetch_ohlc      → upload_ohlc_to_gcs   → load_ohlc_to_bigquery

Markets branch hits CoinGecko's /coins/markets endpoint for all 5 coins in a single call, getting current price, market cap, 24h volume, price change percentage, rank, and fully diluted valuation.

OHLC branch hits /coins/{id}/ohlc?days=1 for each coin, getting candlestick arrays [timestamp, open, high, low, close].

Both branches serialize data to typed Parquet (using explicit schema definitions, not autodetect — more on why this matters below) and upload to GCS before loading to BigQuery.

The Schema War (And How I Won It)

This took me longer than I'd like to admit. The root cause: PyArrow (used by Pandas .to_parquet()) infers Python datetime objects as TIMESTAMP in Parquet metadata. But when BigQuery's autodetect reads a datetime.isoformat() string, it creates the column as STRING. When the same column arrives as a proper datetime object in the next run, BigQuery rejects the load because TIMESTAMP ≠ STRING.

The permanent fix: define explicit schemas in the BigQuery Load Job config and force all string columns to str dtype in Pandas before writing Parquet:

df["snapshot_date"] = df["snapshot_date"].astype(str)
df["ingested_at"]   = df["ingested_at"].astype(str)

job_config = bigquery.LoadJobConfig(
    source_format=bigquery.SourceFormat.PARQUET,
    schema=MARKETS_SCHEMA,  # explicit, no autodetect
    write_disposition=bigquery.WriteDisposition.WRITE_APPEND,
)

Once I stopped fighting autodetect and defined schemas explicitly, the pipeline ran clean for weeks.

The Transformation Layer: dbt Cloud

Why dbt Cloud Over dbt Core

I already had Redpanda, Flink, Airflow, and PostgreSQL running locally — approximately 13GB RAM. Adding a dbt Core Docker container would push that higher for no functional gain. dbt Cloud's free developer plan gives a managed scheduler, visual lineage graph, and test runner at zero cost and zero local RAM.

The Model Structure

Staging layer (views, zero storage cost):

stg_stream_prices — cleans and type-casts streaming data, filters nulls and zero prices
stg_coingecko_markets — adds market_cap_category (large/mid/small/micro cap) derived column
stg_ohlc_candles — computes candle_range, candle_change_pct, and candle_direction (bullish/bearish)

Mart layer (incremental tables, insert_overwrite):

mart_crypto_prices — joins streaming aggregations with latest daily market metadata
mart_volatility — computes hourly composite_volatility_score combining streaming stddev with OHLC candle metrics

The mart models use insert_overwrite with hour-level partitioning and an incremental filter that scans only the last 2 hours of streaming data per run. This means dbt never full-scans the streaming table — it touches only the partitions it needs to update.

The dbt Deploy Job

A scheduled dbt Cloud production environment job runs daily at 07:00 UTC — one hour after Airflow — ensuring mart tables always have fresh data before the dashboard refreshes.

After some initial failures (a common dbt Fusion syntax issue with accepted_values requiring arguments: nesting), the deploy job has run cleanly every day since April 19, 2026. Each run completes in under 45 seconds.

The Dashboard: Grafana Cloud

Why Grafana Over Looker Studio

Looker Studio is the natural choice for BigQuery — it has a native connector and no setup friction. But it has one fatal flaw for streaming analytics: it doesn't auto-refresh.

Grafana Cloud's free tier includes the BigQuery plugin (one-click install, no credit card), supports 30-second auto-refresh, has proper time series panels with logarithmic Y-axis (essential when plotting BTC at $76K alongside ADA at $0.25 on the same chart), and produces significantly more polished visualizations.

My prior projects (Sovereign Debt Observatory, Tech Ecosystem Observatory) both used Looker Studio, so using Grafana here also demonstrates range across tooling.

The 6 Dashboard Panels

Panel 1 — Live Crypto Prices (Streaming)
A treemap showing the latest price for each coin from crypto_raw.stream_prices, colour-coded green/red by price direction. BTC at $76K, ETH at $2.3K, SOL at $86, BNB at $630, ADA at $0.249.

Panel 2 — Price Trend Over Time (Streaming)
A time series chart with logarithmic Y-axis showing avg_price per 1-minute window for all coins. The log scale handles the 300,000x price spread between BTC and ADA.

Panel 3 — Hourly Volatility Score (Streaming)
A gradient bar chart (green → red) showing the composite_volatility_score per coin from mart_volatility. Cardano leads at 0.566 (most volatile relative to its price), Binance Coin lowest at 0.264.

Panel 4 — Market Cap Rankings (Batch)
BTC dominates at $1.57T, ETH at $289B. The scale difference is immediately visible and gives context to the streaming data.

Panel 5 — Daily OHLC Candles (Batch)
A table panel showing raw candle data from coingecko_ohlc_candles with candle_direction colour-coded — bearish rows highlighted in red.

Panel 6 — 24h Price Change % (Batch)
ADA leads at +0.563%, ETH is the only negative at -0.132%.

Public URL: https://derrickryangiggs.grafana.net/public-dashboards/81560968e15140f08f65b52d78a4b252

Infrastructure: Terraform for Everything

All GCP resources are defined in Terraform — nothing clicked in the console. Running terraform apply provisions:

GCP project with billing account linked
Required APIs (BigQuery, BigQuery Storage, Cloud Storage, Resource Manager)
Service account with least-privilege IAM roles (BQ Data Editor, BQ Job User, GCS Object Admin)
GCS bucket with 90-day lifecycle rule
BigQuery datasets and tables with explicit schemas, partitioning, and clustering

Nothing is hardcoded. All values flow from .env via TF_VAR_ prefixed variables. This is non-negotiable — hardcoded project IDs in .tf files are a code review failure waiting to happen.

The Cost Breakdown (For Real)

Service	Details	Monthly Cost
Redpanda	Local Docker	$0.00
PyFlink	Local Docker	$0.00
Apache Airflow	Local Docker	$0.00
Google Cloud Storage	~200MB of Parquet + JSONL	~$0.01
BigQuery	<1GB data, Load Jobs only	~$0.00
dbt Cloud	Developer plan	$0.00
Grafana Cloud	Free tier	$0.00
Total		~$0.01/month

The $0.01 is literally the GCS storage bill. Everything else is free.

The three decisions that make this possible:

Use GCS + Load Jobs instead of BigQuery Streaming Inserts. Streaming Inserts charge per row. Load Jobs are free. The trade-off is 1-minute data freshness instead of real-time — acceptable for a dashboard refreshing every 30 seconds.
Run compute locally. Cloud Composer (managed Airflow) starts at $300/month. Dataproc (managed Flink) is similarly expensive. Running on your own machine costs nothing except electricity.
Use free tiers everywhere else. dbt Cloud developer plan, Grafana Cloud free tier, CoinGecko Demo API free tier, Binance WebSocket (no auth required). Every external service in this pipeline has a genuinely free tier sufficient for this workload.

Hard-Won Lessons

1. Flink's WindowFunction.apply() signature changed in 2.2.0
In PyFlink 1.18, apply(self, key, window, inputs, collector) uses a collector argument. In 2.2.0, the signature is apply(self, key, window, inputs) and you use yield instead. This isn't documented prominently. I lost an afternoon to this.

2. Read env vars inside functions, not at module import time
My Airflow DAG originally read GCS_BUCKET_NAME = os.environ["GCS_BUCKET_NAME"] at the module level. When Airflow's scheduler reloads the module after a container restart, GCS_BUCKET_NAME is empty — the env var isn't available at import time in the scheduler process. Moving all os.environ reads inside the task functions fixed this permanently.

3. Never use autodetect for BigQuery Load Jobs if you care about schema stability
autodetect=True infers schema from the first file it sees. If that file has a datetime object, BQ creates a TIMESTAMP column. If the next file has an .isoformat() string, BQ rejects it as a STRING / TIMESTAMP mismatch. Define explicit schemas in the LoadJobConfig and cast DataFrame columns explicitly before writing Parquet.

4. The Flink TaskManager won't connect if jobmanager.rpc.address in config.yaml doesn't match your Docker container name
The baked-in config.yaml in the PyFlink image has jobmanager.rpc.address: jobmanager. My container is named coinpulse-flink-jobmanager. The FLINK_PROPERTIES env var doesn't always override config.yaml. The fix: add a sed command in the Dockerfile to rewrite the address in config.yaml directly.

5. Redpanda's rpk topic create command arguments must be on one line
Multi-line bash in Docker Compose entrypoint with > YAML block scalar doesn't handle line continuations the way you'd expect. The --brokers, --partitions, and --replicas flags ended up being interpreted as separate commands. Fix: use ["bash", "-c", "single long command string"] format instead.

What I'd Add Next (The "Circuit Breaker" Enhancements)

After sharing the project, I got feedback suggesting three production-grade improvements. Here's my assessment:

1. dbt-expectations data contracts
Add guardrails so the pipeline fails loudly if bad data arrives — negative prices, null coin symbols, impossible price spikes. In dbt, this is two lines per test in your schema YAML. The dbt-expectations package brings expect_column_values_to_be_between and similar assertions. Highly recommended, zero cost.

2. BigQuery ML ARIMA_PLUS price forecasting
Train a time series model directly in BigQuery SQL to forecast the next 7 days of prices per coin. The first 10GB of CREATE MODEL data processed monthly is free — both projects' datasets are well under that threshold, making this effectively free. It turns the dashboard from "here's what happened" into "here's what might happen."

3. GitHub Actions CI/CD for Terraform
A terraform plan on every PR to the repo, ensuring infrastructure changes are reviewed before applying. Standard practice in production environments, takes about 20 lines of GitHub Actions YAML to implement.

The Full Stack, Summarised

Layer	Tool	Why
IaC	Terraform	Reproducible, version-controlled infrastructure
Message Broker	Redpanda	Kafka-compatible, no Zookeeper, runs in one container
Stream Processing	PyFlink 2.2.0	Stateful windowed aggregations, event-time semantics
Orchestration	Apache Airflow	Full end-to-end DAG, retry logic, monitoring
Object Storage	GCS	Staging layer between compute and warehouse
Data Warehouse	BigQuery	Serverless, free Load Jobs, columnar, partitioned
Transformations	dbt Cloud	Incremental models, lineage graph, scheduled runs
Visualization	Grafana Cloud	Auto-refresh, BigQuery plugin free on all tiers

Try It Yourself

The full project is open source:

GitHub: https://github.com/Derrick-Ryan-Giggs/coinpulse

Live Dashboard: https://derrickryangiggs.grafana.net/public-dashboards/81560968e15140f08f65b52d78a4b252

The README has step-by-step reproduction instructions — clone, fill in .env, run terraform apply, start the Docker stacks, and you have a running pipeline within about 20 minutes. All you need is a GCP account (free tier works), a CoinGecko Demo API key (free, no card), dbt Cloud account (free developer plan), and Grafana Cloud account (free forever).

What This Project Taught Me

Building this reinforced something I suspected but didn't fully appreciate: the hardest problems in data engineering aren't the algorithms — they're the plumbing.

Getting PyFlink to connect to a Redpanda broker with the right container hostname was a day of debugging. Getting the Airflow DAG to read env vars correctly on scheduled runs (not just manual triggers) was another. Getting BigQuery to accept the same schema consistently across runs required understanding how PyArrow serialises Python types to Parquet column metadata.

None of these problems are glamorous. But solving them is what separates a "pipeline that worked once on my laptop" from a "pipeline that runs reliably every day."

That's what CoinPulse is — a pipeline that runs reliably every day, for $0.01/month.

If you found this useful or have questions about any part of the implementation, feel free to reach out. I write regularly about data engineering, cloud infrastructure, and GCP on Medium, Dev.to, and Hashnode.

Your Serverless Data Lake is Lying to You: Add Observability or Lose Data (AWS)

Mon, 20 Apr 2026 12:55:14 +0000

TL;DR

Serverless Data Lakes are scalable and cost-efficient—but they can fail silently
Without observability, you risk making decisions on incomplete data
Adding a Dead Letter Queue (DLQ) ensures no data is lost
Monitoring with Amazon CloudWatch + alerts via Amazon SNS gives you real visibility
Trade-off: slightly more complexity for significantly higher reliability
The Moment Everything “Worked”… But Was Wrong

It was 2 AM.

The pipeline had completed successfully.
Amazon Athena was returning results.

Dashboards were updating.

And yet… the numbers didn’t match.

Walking through logs in Amazon CloudWatch, I found the issue:
messages sitting in a queue that no one was monitoring.

No alarms.
No failures reported.
Just silently missing data.

That’s the uncomfortable truth about serverless systems:

They don’t crash. They drift.

The Typical “Clean” Architecture

Most Serverless Data Lakes follow a familiar pattern:

Amazon S3 as the storage layer
AWS Glue for schema and transformation
Amazon Athena for analytics

It’s elegant:

No servers
Pay-per-use
Infinite scale

But it assumes something dangerous:

That success = correctness

What Was Missing: Observability

The issue wasn’t the architecture.

It was the lack of visibility into:

What failed
What was retried
What never got processed

Without that, your data lake becomes a black box.

And black boxes are risky in data engineering.

The Fix: Designing for Failure (On Purpose)

For an E-commerce analytics workshop, I reworked the architecture with a different mindset:

Assume everything will fail—and make it visible when it does.

1. Decoupling Ingestion

Instead of triggering jobs directly from S3 events:

Amazon S3 emits events
Amazon SQS captures and buffers them

This introduces control:

Retry mechanisms
Backpressure handling
Event durability

2. Dead Letter Queue (DLQ): No Data Left Behind

Failures are inevitable. Silent failures are optional.

By adding a DLQ:

Failed messages are isolated after retries
Nothing is lost
You can inspect and replay events

This turns “unknown issues” into actionable signals.

3. Lightweight Orchestration
AWS Lambda polls SQS
Triggers AWS Glue jobs

Simple, decoupled, and reactive.

4. Optimized Storage for Analytics
Raw data lands in S3 (CSV/JSON)
Transformed into Parquet
Partitioned by date

This reduces cost and improves query performance in Amazon Athena.

Observability: Where Things Get Real

This is the layer most diagrams ignore—and the one that matters most in production.

Metrics That Tell the Truth

Using Amazon CloudWatch:

Queue depth (are messages piling up?)
DLQ size (are failures accumulating?)
Glue job status
Lambda error rates
Alerts That Don’t Wait

With Amazon SNS:

DLQ > 0 → immediate alert
Glue job failure → alert
Pipeline inactivity → alert

No need to “check dashboards.”
You get notified when something breaks.

A Simple Rule

If your system fails and you don’t know immediately:

You don’t have observability.

Trade-offs (Because Nothing Is Free)

Adding observability and DLQs improves reliability—but it’s not without cost.

What You Gain

Data reliability
Failure traceability
Faster incident response
Confidence in your analytics

What You Pay

More components (SQS, DLQ, Lambda)
Slightly higher operational complexity
Additional monitoring configuration
Small increase in cost (messages, metrics, alerts)

The Real Trade-off

You’re not choosing between “simple vs complex.”

You’re choosing between:

Simple system that hides failures
Slightly more complex system that exposes them

In production, that’s an easy decision.

Final Thought

Serverless is powerful. It removes infrastructure, reduces costs, and accelerates development.

But it also removes friction—and friction is often what makes failures visible.

If you don’t design for observability, your system will fail quietly.
And quiet failures are the most expensive ones.

How are you handling failures in your data pipelines today?

Are you capturing them… or hoping they don’t happen?

Oracle GoldenGate 23ai: Powering Distributed AI with Real-Time Data Replication

Sat, 18 Apr 2026 11:13:01 +0000

The database world has always needed a reliable bridge between operational systems and analytical or AI workloads. For decades, Oracle GoldenGate has served that role — silently moving terabytes of transactional changes across databases, clouds, and continents with minimal latency and near-zero impact on source systems. With the release of GoldenGate 23ai in 2024, that bridge now carries a new payload: AI vector embeddings.

This post covers GoldenGate's core use cases, the Microservices Architecture that underpins modern deployments, and the powerful new distributed AI capabilities that make GoldenGate 23ai a critical component of enterprise GenAI pipelines.

What Is Oracle GoldenGate?

At its heart, GoldenGate is a Change Data Capture (CDC) and real-time replication engine. Rather than querying the source database directly — which would impose load — GoldenGate reads the database's redo/transaction logs, captures only the changed deltas (inserts, updates, deletes), and delivers them to one or more target systems with sub-second latency.

This log-based approach makes GoldenGate extremely low-impact on production systems while supporting highly demanding replication topologies across heterogeneous environments: Oracle to PostgreSQL, Oracle to Kafka, MySQL to BigQuery, and many more.

Common GoldenGate Use Cases

GoldenGate's flexibility makes it applicable across a broad range of enterprise scenarios:

1. Multi-Active, High Availability, and Cross-Region Deployments

GoldenGate enables active-active (bidirectional) replication, where multiple database instances in different regions accept both reads and writes simultaneously. Built-in conflict detection and resolution ensures data consistency across sites. This architecture powers mission-critical applications that require zero planned downtime and regional failover capabilities — with GoldenGate's ExaPortMon-style awareness ensuring continuous replication even under network disruptions.

2. Data Offloading and Data Hub

Rather than running analytical queries against production OLTP databases, GoldenGate streams only the changed data to a dedicated analytics database, data warehouse, or data lake in real time. This eliminates the traditional "nightly CSV batch" pattern and gives analysts access to data that is seconds, not hours, behind production.

3. Migrations and Upgrades

GoldenGate is the gold standard for zero-downtime database migrations — moving from on-premises to cloud, upgrading Oracle Database versions, or switching from Oracle to PostgreSQL. The approach: synchronize old and new systems in parallel using GoldenGate, then cut over the application connection once replication lag reaches zero. Even migrations of hundreds of terabytes are reduced to a switchover window measured in minutes.

4. Analytics Data Feeds

GoldenGate feeds real-time change streams into analytics platforms — data warehouses, Apache Kafka topics, Oracle Streaming Service, Confluent Cloud, and cloud object stores. This underpins event-driven architectures and real-time dashboards that would be impossible with batch-based ETL.

5. Stream Analytics

With GoldenGate Stream Analytics, change events flowing through GoldenGate can be enriched, filtered, and analyzed in motion — before they land in a target system. Integrations with Oracle Machine Learning (OML) and ONNX (Open Neural Network Exchange) enable actionable AI/ML directly from streaming pipelines, turning raw CDC events into predictive signals.

GoldenGate Microservices Architecture

When Oracle introduced the Microservices Architecture (MA) in GoldenGate 12.3 (August 2017), it represented a fundamental modernization of how GoldenGate is deployed, managed, and integrated. The classic command-line-driven architecture (GGSCI) was complemented by a fully API-driven, web-based platform.

Five Core Components

The Microservices Architecture is built around five independently manageable services:

Service Manager — acts as a watchdog, managing and monitoring all other services within a deployment
Administration Server — the central control entity for managing Extract, Replicat, and all replication processes
Distribution Server — a high-performance data distribution agent that routes trail files to one or more targets over HTTPS/WebSockets
Receiver Server — handles all incoming trail files from remote Distribution Servers
Performance Metrics Server — collects and exposes real-time performance data for all GoldenGate processes via REST API or JSON

Key Benefits of Microservices Architecture

REST API-driven management: Every GoldenGate operation — starting a replicat, checking lag, configuring a pipeline — is exposed via a standardized REST API, enabling automation with Python, Terraform, and CI/CD pipelines.
Renewed web UI in 23ai: GoldenGate 23ai ships with a completely redesigned graphical interface, with more logical, guided assistant steps for setting up integrations — a marked improvement over the previous look and feel.
TLS encryption and OAuth 2.0 authentication out of the box, with integration into external identity providers.
Simpler patching and upgrades: Each microservice can be updated with minimal disruption. Upgrading from GoldenGate 21c to 23ai, for example, requires only downloading the new software and updating the OGG_HOME directory parameters.
Deployment flexibility: Runs on-premises, as a fully managed service in OCI (OCI GoldenGate), or on third-party clouds including AWS, Azure, and GCP.

GoldenGate Version Upgrade Path

For teams still running older GoldenGate versions, the supported upgrade sequence through the modern microservices era is:

12.1 → 12.2 → 12.3 (Microservices introduced) → 18c → 19c → 21c → 23ai → 26ai (next LTS)

Note: Classic Architecture support for Oracle Database is deprecated in 23ai. All new deployments should use Microservices Architecture. GoldenGate 26ai has been announced as the next Long-Term Support (LTS) release, with further AI and Data Mesh enhancements planned.

Distributed AI Processing with Vector Replication

This is where GoldenGate 23ai breaks genuinely new ground. The release adds first-class support for the Oracle Database 23ai VECTOR data type — the foundational building block of AI Vector Search and RAG (Retrieval-Augmented Generation) pipelines.

What Is a Vector Embedding?

When AI models process documents, images, or other unstructured data, they convert that content into vector embeddings — high-dimensional numerical representations that encode semantic meaning. These vectors are stored in vector databases and searched using similarity algorithms (cosine distance, dot product, Euclidean distance) to retrieve contextually relevant content.

Keeping vector stores synchronized with the operational databases that hold the source data is a significant engineering challenge — and exactly what GoldenGate 23ai solves.

Key Vector Replication Capabilities

Migrate Vectors into Oracle Vector Database Without Downtime

GoldenGate can migrate vector embeddings stored in one database into Oracle Database 23ai (or Oracle AI Database 26ai) without any service interruption. This enables seamless transitions from on-premises environments to OCI, or between cloud providers, while keeping AI-powered applications running continuously.

Replicate and Consolidate Vector Changes in Real Time

As source data changes — a customer record is updated, a product description is revised — GoldenGate captures those changes and propagates updated vectors to all target vector stores in real time. This ensures that RAG pipelines are always querying fresh, consistent embeddings rather than stale snapshots.

Heterogeneous Vector Replication

GoldenGate 23ai can replicate vectors both homogeneously (Oracle to Oracle) and heterogeneously across different vector stores, provided the embedding algorithm, type, and dimension are consistent across source and target. If the source and target use different embedding algorithms, GoldenGate can replicate the raw source data instead, allowing the target system to re-vectorize it with its own model.

Support extends beyond Oracle: GoldenGate 23ai also supports capture and delivery of the pgvector extension for PostgreSQL and its derivatives, as well as the VECTOR datatype for MySQL 9.0+.

Multi-Cloud, Multi-Active Oracle Vector Database

GoldenGate enables active-active vector database architectures spanning multiple cloud regions or providers. AI applications in different regions can write to and read from local vector stores while GoldenGate keeps all instances synchronized — enabling both low-latency AI inference and high availability for vector workloads.

Stream Changes to Search Engines

GoldenGate can stream vector and document changes in real time to Elasticsearch or OpenSearch compatible search indexes, enabling hybrid semantic + keyword search architectures without manual synchronization pipelines.

GoldenGate Data Streams: Pub/Sub for Database Events

A major new capability in GoldenGate 23ai is the Data Streams service, which provides simplified, event-driven access to database change records via AsyncAPI over WebSocket connections. This enables developers to subscribe to database change events — including vector updates — in a pub/sub pattern, without needing to manage trail files or configure traditional Replicat processes. Combined with Oracle Database 23ai's JSON Relational Duality views, this means document-side changes can be captured and propagated as structured events in real time.

Generative AI with Your Own Business Data

GoldenGate 23ai directly enables three patterns for building AI applications on private enterprise data:

1. From-Scratch LLMs (Custom Foundation Models)

Organizations with truly unique proprietary datasets — medical records, legal case files, financial transaction histories — may opt to train custom Large Language Models from the ground up on that data. GoldenGate ensures the training datasets fed into these models are continuously fresh and complete, sourced in real time from operational databases rather than periodic exports.

This is an advanced, resource-intensive approach suited to well-resourced organizations with data assets that are sufficiently distinctive to justify the investment.

2. Fine-Tuned LLMs (Domain Adaptation)

A more practical approach for most enterprises: take a pre-trained foundational LLM (such as Llama, Mistral, or an OCI Generative AI model) and fine-tune it on a private domain dataset. GoldenGate replicates the relevant operational data into a training pipeline, keeping the fine-tuned model aligned with current business reality as data evolves over time.

3. RAG (Retrieval-Augmented Generation) — The Most Common Pattern

RAG is currently the dominant architecture for enterprise GenAI. Instead of baking private knowledge into model weights (which requires expensive retraining), RAG retrieves relevant documents at query time and injects them into the LLM's prompt as context.

GoldenGate 23ai is purpose-built to power better RAG pipelines:

Keeps vector stores current: As source data changes, GoldenGate propagates those changes to the vector index within seconds — not hours.
Enables a consolidated vector hub: Rather than maintaining separate, potentially inconsistent vector stores for each application, GoldenGate can consolidate vectors from multiple source databases into a single Oracle AI Database vector hub.
Reduces hallucination risk: Stale vector indexes are one of the primary causes of LLM hallucinations in RAG systems. Fresh data = more accurate, grounded responses.
Supports heterogeneous sources: GoldenGate can pull data from Oracle, PostgreSQL, MySQL, SQL Server, DB2, and more — vectorize it, and deliver it to Oracle AI Database, all in a unified pipeline.

Why GoldenGate 23ai for AI: The Summary

Capability	Impact
Real-time vector replication	RAG pipelines query fresh embeddings, not stale snapshots
Zero-downtime vector migration	Move AI workloads to cloud without service interruption
Multi-cloud active-active vectors	Low-latency AI inference globally with high availability
Heterogeneous source support	Unify vectors from Oracle, PostgreSQL, MySQL, and more
Data Streams (AsyncAPI)	Event-driven AI pipelines without complex trail file management
Stream to search engines	Elasticsearch/OpenSearch integration for hybrid search
Fine-tuning data pipelines	Keep domain-adapted LLMs current with live business data

The underlying thesis is the same one Oracle argues for Exadata AI Storage: bring AI to the data, not data to the AI. GoldenGate 23ai extends this philosophy to the replication layer — ensuring that wherever your AI application runs, it has access to consistent, real-time data from your authoritative operational systems.

As enterprise AI moves from experimentation to production, the quality of data pipelines will increasingly determine the quality of AI outcomes. GoldenGate 23ai is Oracle's answer to that challenge.

Are you using GoldenGate for AI pipelines or RAG architectures? What challenges have you run into keeping vector stores synchronized? Share in the comments below.

Building the Sovereign Debt Observatory: An End-to-End ELT Pipeline on World Bank Debt Data for Low

Thu, 09 Apr 2026 10:02:56 +0000

Introduction

Global sovereign debt is one of the most consequential datasets in existence. It shapes foreign policy, determines credit ratings, drives IMF bailout decisions, and affects the daily lives of billions of people in developing countries. The World Bank publishes this data openly — 130+ countries, 27 years of history, updated quarterly — yet there is no ready-made analytical layer on top of it.

If you want to answer a question like "which African countries have the highest ratio of private nonguaranteed debt to total external debt, and how has that changed since 2010?", you have to manually download Excel files from multiple World Bank portals, clean inconsistent column names, handle missing values, and stitch everything together in a spreadsheet. Every time the data updates, you do it again.

That is the problem this project solves.

The Sovereign Debt Observatory is an end-to-end ELT pipeline that ingests World Bank external debt data, lands it in a cloud data lake, transforms it in BigQuery using dbt Cloud, orchestrates everything quarterly with Apache Airflow, and surfaces the answers in a Looker Studio dashboard.

This article walks through every layer of the pipeline — the architecture decisions, the technical challenges, and how I solved them.

The Five Questions This Pipeline Answers

Before writing a single line of code, I defined the analytical questions the pipeline needed to answer. This kept every decision grounded in purpose rather than technology for its own sake.

How is gross external debt distributed across public, publicly guaranteed, private nonguaranteed, and multilateral sectors per country?
Which countries carry the highest short-term external debt exposure and how has that changed since 2010?
What share of external debt is foreign-currency denominated and where is that ratio worsening?
How has regional external debt stock evolved from 1998 to 2025 across Africa, Latin America, East Asia, South Asia, Europe and Central Asia, and the Middle East?
Which countries face the heaviest debt service pressure relative to their total debt position?

Architecture Overview

The pipeline follows a modern ELT pattern — extract and load first, transform inside the warehouse.

World Bank API (IDS source 2)
        |
        v
PySpark job (Docker container)
        |
        v
Google Cloud Storage — raw Parquet, partitioned by extracted_date
        |
        v
BigQuery external tables (raw dataset)
        |
        v
dbt Cloud — staging views + mart tables
        |
        v
Looker Studio dashboard
        |
Orchestrated by Apache Airflow on Docker Compose
Infrastructure provisioned by Terraform

Why ELT and not ETL?

In ETL, transformation happens before loading — your Spark job does the cleaning, aggregation, and business logic before writing to the warehouse. In ELT, raw data lands untransformed and the warehouse does all the heavy lifting.

For this project, ELT is the right choice for three reasons. First, raw data is preserved in GCS indefinitely — if analytical requirements change six months from now, I just write a new dbt model without re-running the ingestion layer. Second, BigQuery is optimized for analytical SQL transformations at scale — it is far better at this than PySpark running in a Docker container on a local machine. Third, dbt gives us version-controlled, tested, documented transformations that are readable by anyone with SQL knowledge.

PySpark's job in this pipeline is purely mechanical: hit the API, paginate, write Parquet. No business logic. No aggregations. Pure extract and load.

Data Sources

International Debt Statistics (IDS) — World Bank source 2

The IDS database is the flagship World Bank debt dataset. It covers external debt stocks and flows for low and middle income countries, with annual data going back to 1998. I access it through the wbgapi Python library, which wraps the World Bank Indicators API v2.

I ingest nine series:

Series Code	Description
DT.DOD.DECT.CD	Total external debt stocks
DT.DOD.DLXF.CD	Long-term external debt
DT.DOD.DPNG.CD	Private nonguaranteed debt
DT.DOD.MIBR.CD	PPG IBRD loans
DT.DOD.DPPG.CD	Public and publicly guaranteed debt
DT.DOD.DIMF.CD	IMF credit
DT.DOD.PVLX.CD	Present value of external debt
DT.DOD.MWBG.CD	IBRD loans and IDA credits
DT.DOD.MIDA.CD	PPG IDA loans

One important lesson from building this: the World Bank has multiple API source databases, and not all of them support the standard wbgapi query format. Sources 22 (QEDS SDDS), 23 (QEDS GDDS), and 54 (JEDH) all return JSON decode errors when queried programmatically — they use a separate DataBank backend. Only source 2 (IDS) reliably supports the Indicators API. This cost me several hours of debugging.

Quarterly External Debt Statistics SDDS (QEDS)

QEDS provides quarterly debt payment schedule data broken down by sector and maturity. Unlike IDS, QEDS does not support programmatic API access in the standard format. The World Bank provides bulk Excel downloads for each supplementary table instead.

I download five Excel files directly:

Table 1.5 — Net external debt position by sector
Table 3 — Debt service payment schedule by sector
Table 3.2 — Debt service by sector and instrument
Table 2.1 — Foreign currency and domestic currency debt
Table 1.6 — Reconciliation of positions and flows

Infrastructure as Code with Terraform

Every GCP resource in this project is provisioned by Terraform. The three core resources are a GCS bucket for the data lake and three BigQuery datasets — raw, staging, and mart.

resource "google_storage_bucket" "data_lake" {
  name          = var.gcs_bucket_name
  location      = var.region
  force_destroy = true

  lifecycle_rule {
    action { type = "Delete" }
    condition { age = 90 }
  }

  versioning { enabled = true }
}

resource "google_bigquery_dataset" "raw" {
  dataset_id                 = "raw"
  location                   = var.location
  delete_contents_on_destroy = true
}

The 90-day lifecycle rule on GCS automatically deletes old partitions, keeping storage costs near zero. The entire GCP footprint for this project costs less than $0.05 per month — BigQuery's free tier covers 1 TB of queries and 10 GB of storage, which is far more than this dataset requires.

A setup_gcp.sh script handles the one-time bootstrap — creating the GCP project, enabling APIs, creating the service account, and granting IAM roles. The billing account ID is passed as an environment variable so it never appears in version-controlled files.

BILLING_ACCOUNT=your-billing-id bash scripts/setup_gcp.sh

Ingestion Layer — PySpark on Docker

JEDH / IDS Ingestion

The IDS ingestion script uses wbgapi to fetch all nine series across all available countries from 1998 to 2025. The API returns data in wide format — one row per country per series, with year columns as separate fields. PySpark writes this to GCS as Snappy-compressed Parquet, partitioned by extraction date.

One critical issue I hit: the World Bank API returns year columns as bare integers (1998, 1999, etc.). BigQuery rejects column names that start with numbers. The fix was to prefix all year columns with year_ before writing:

combined.columns = [
    f"year_{col}" if str(col).isdigit() else col
    for col in combined.columns
]

This produces clean column names like year_1998, year_1999 that BigQuery accepts without complaint.

QEDS Ingestion

The QEDS ingestion downloads five Excel files from World Bank DataBank, reads all sheets from each file using pandas.read_excel, and concatenates them into a single DataFrame.

The Excel files have messy column names — spaces, brackets, special characters. A clean_column_name function normalizes everything:

def clean_column_name(col):
    col = str(col)
    col = re.sub(r'\s*\[.*?\]', '', col)  # remove [YR2021Q4] suffixes
    col = col.strip()
    col = re.sub(r'[^a-zA-Z0-9_]', '_', col)
    col = re.sub(r'_+', '_', col)
    col = col.strip('_')
    if col and col[0].isdigit():
        col = 'q_' + col  # prefix quarter columns: q_2021q4
    return col.lower()

I also hit an OOM error trying to write the QEDS data through PySpark — 76,835 rows with 90+ columns across 214 sheets was too much for the JVM heap in the Docker container. The fix was to bypass Spark entirely for QEDS and write directly to GCS using the google-cloud-storage Python client:

buffer = io.BytesIO()
df.to_parquet(buffer, index=False, engine="pyarrow")
buffer.seek(0)

client = storage.Client()
bucket = client.bucket(GCS_BUCKET)
blob = bucket.blob(output_path)
blob.upload_from_file(buffer, content_type="application/octet-stream")

No JVM, no Spark executor, no OOM. The lesson: use the right tool for the job. PySpark is excellent for large distributed datasets. For a 76K-row DataFrame from an Excel file, plain pandas and the GCS Python client is simpler and more reliable.

Docker Image

The ingestion image is built on eclipse-temurin:17-jdk-jammy rather than a plain Python image. This is because PySpark requires Java, and the python:3.11-slim base image uses Debian Trixie which does not carry openjdk-17-jdk in its default repositories. The Temurin image ships Java 17 out of the box, which is exactly what Spark 3.5.1 needs.

The GCS connector JAR is downloaded at build time:

RUN curl --progress-bar -L \
    https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar \
    -o ${SPARK_HOME}/jars/gcs-connector-hadoop3-latest.jar

Credentials are never baked into the image. They are mounted at runtime as a volume:

docker run --rm \
  -v /path/to/key.json:/app/credentials/key.json \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials/key.json \
  sovereign-debt-ingestion:v1 python3 extract_jedh.py

Orchestration with Apache Airflow

Airflow runs on Docker Compose using the official apache/airflow:2.9.2 image. The stack includes a Celery executor with Redis as the message broker and Postgres as the metadata database.

The DAG runs quarterly — on the 1st of January, April, July, and October at 06:00 UTC:

schedule_interval="0 6 1 1,4,7,10 *"

Task flow:

extract_load_jedh >> extract_load_qeds

Both tasks use the DockerOperator to spin up the ingestion container on the host machine, which means Airflow itself does not need PySpark, Java, or any data dependencies — it just tells Docker to run the job.

The Docker socket problem

The DockerOperator communicates with the host Docker daemon through the Unix socket at /var/run/docker.sock. This requires two things: the socket must be mounted into the Airflow worker container, and the worker must have permission to use it.

The socket mount goes in docker-compose.yml under the common volumes section:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

The permission fix requires adding the Docker group ID to the Airflow worker. On my machine the Docker group ID is 984:

group_add:
  - "984"

Without the group add, the worker can see the socket but gets PermissionError(13, 'Permission denied'). This is a common Airflow + Docker-in-Docker gotcha that took several debugging sessions to resolve.

Transformation with dbt Cloud

Staging Layer

The staging models are materialized as views — no storage cost, no latency, just a SQL lens over the raw external tables.

stg_jedh does the heavy lifting: it unpivots the wide year-column format into long format using BigQuery's UNPIVOT operator:

unpivot(
    value for year in (
        year_1998, year_1999, year_2000, ..., year_2025
    )
)

Then extracts the year integer from the column name:

cast(replace(year, 'year_', '') as int64) as year

And adds a human-readable series description via a CASE statement, turning DT.DOD.DECT.CD into Total external debt stocks.

The result is a clean long-format table: one row per country per series per year.

stg_qeds is simpler — it selects the clean columns, uses SAFE_CAST to convert quarter values to float64, and filters out null countries and series codes. SAFE_CAST is preferable to CAST here because it returns NULL on failure rather than throwing an error, which is the right behaviour for messy Excel data.

Mart Layer

The mart models are materialized as tables with partitioning and clustering for query efficiency.

mart_debt_stocks is partitioned by year and clustered by country_code and series_code. It enriches the staging data with YoY percentage change and debt as a percentage of total external debt per country per year — both computed using window functions:

lag(b.debt_value_usd) over (
    partition by b.country_code, b.series_code
    order by b.year
) as prev_year_value,

safe_divide(
    b.debt_value_usd - lag(b.debt_value_usd) over (
        partition by b.country_code, b.series_code
        order by b.year
    ),
    lag(b.debt_value_usd) over (
        partition by b.country_code, b.series_code
        order by b.year
    )
) * 100 as yoy_change_pct

mart_regional_debt assigns countries to six World Bank regions using a CASE statement on ISO3 country codes, then aggregates total, average, max, and min debt stocks per region per series per year. This powers the regional trajectory time series on the dashboard.

mart_debt_service computes total annual debt payments and average quarterly payments per country from the QEDS payment schedule data, enabling debt service pressure analysis.

dbt Tests

All models have not_null tests on primary dimension columns. Running dbt test after every model change ensures data quality is enforced at the transformation layer rather than discovered downstream in the dashboard.

The Dashboard

The Looker Studio dashboard has two pages, both connected directly to the BigQuery mart tables.

Page 1 — Global Debt Overview answers Q4 at a glance. A time series chart using mart_regional_debt shows six regional debt trajectories from 1998 to 2024. Africa's trajectory is notably steeper post-2010. A bar chart shows the top 20 countries by total external debt for the selected year. A scorecard shows total global external debt.

Page 2 — Country Deep-Dive answers Q1, Q2, and Q5. A stacked bar chart shows debt composition by sector over time for any selected country. A line chart shows short-term vulnerability trends. A table sorted by total 2021 payments shows which countries face the most acute debt service pressure.

Live dashboard: https://lookerstudio.google.com/reporting/7fc18e9e-a5c6-4616-b920-b5b4bddf2264

/home/derrrick-ryan-giggs/Pictures/Screenshots/Screenshot From 2026-04-09 12-08-05.png

/home/derrrick-ryan-giggs/Pictures/Screenshots/Screenshot From 2026-04-09 12-08-28.png

/home/derrrick-ryan-giggs/Pictures/Screenshots/Screenshot From 2026-04-09 12-08-51.png

/home/derrrick-ryan-giggs/Pictures/Screenshots/Screenshot From 2026-04-09 12-09-33.png

Key Technical Lessons

1. Know which World Bank API sources support programmatic access. Only source 2 (IDS) reliably supports the Indicators API v2 via wbgapi. Sources 22, 23, and 54 use a different DataBank backend and return JSON decode errors. For QEDS data, use the bulk Excel downloads instead.

2. BigQuery rejects column names starting with numbers. Prefix them before writing Parquet. A simple list comprehension handles this: f"year_{col}" if str(col).isdigit() else col.

3. Use the right tool for the data size. PySpark is excellent for large datasets but overkill for a 76K-row Excel file. The GCS Python client with pandas and PyArrow is simpler, faster, and doesn't OOM.

4. Docker-in-Docker with Airflow requires explicit socket mounting and group permissions. Mount /var/run/docker.sock and add the Docker group ID to the worker container. Without the group add, you get a silent permission error.

5. ELT separates concerns cleanly. When the analytical questions evolved during development, I only needed to update dbt models — never the ingestion layer. This separation is the most valuable architectural decision in the project.

What I Would Do Differently

If I were building this again, I would use the World Bank DataBank bulk download API to get historical QEDS time series data instead of point-in-time Excel files. The current QEDS data only has a handful of quarters because the Excel files are snapshots of the latest publication. A proper time series would require downloading and archiving each quarterly release.

I would also add a load_to_bigquery step that loads Parquet directly into native BigQuery tables rather than using external tables. External tables work well but they require manually updating the source URI list each time a new partition is added. A native table with partitioning handles this automatically.

Conclusion

The Sovereign Debt Observatory took the World Bank's raw debt data from scattered Excel files and API endpoints to a fully automated, tested, and documented analytical pipeline. Every component is reproducible — Terraform provisions the infrastructure, Docker packages the ingestion environment, Airflow schedules the runs, and dbt documents and tests the transformations.

The full source code is on GitHub: https://github.com/Derrick-Ryan-Giggs/sovereign-debt-observatory

If you have questions about any part of the implementation, drop them in the comments. I am happy to go deeper on any layer.

Ryan Derrick Giggs is a data engineering practitioner and technical writer based in Nairobi, Kenya. He is currently completing the DataTalksClub Data Engineering Zoomcamp 2026.

LinkedIn: https://linkedin.com/in/ryan-giggs-a19330265
GitHub: https://github.com/Derrick-Ryan-Giggs

Pandas 3.0's PyArrow String Revolution: A Deep Dive into Memory and Performance

Wed, 08 Apr 2026 06:01:21 +0000

Introduction

Pandas 3.0 made a game-changing decision: PyArrow-backed strings are now the default. Instead of storing strings as Python objects (the old object dtype), pandas now uses Apache Arrow's columnar format with the new string[pyarrow] dtype.

But here's the question that matters: How much does this new string dtype actually improve performance and memory usage in real-world scenarios?

To find out, I ran comprehensive benchmarks across diverse datasets and common string operations. The results? 51.8% memory savings on average, with operations running 2-27x faster.

This isn't a theoretical improvement, it's a fundamental shift in how pandas handles string data.

The Results: Summary Dashboard

Let me start with the headline numbers, then we'll dive into how I got them.

The Four Key Metrics

1. 51.8% Memory Savings

Across all test datasets, the new PyArrow string dtype used half the memory of the old object dtype. This isn't a marginal improvement, it's transformative for memory-constrained environments.

2. 6.17x Average Operation Speedup

String operations aren't just more memory-efficient: they're dramatically faster. On average, operations like str.lower(), str.contains(), and str.len() run 6x faster with PyArrow strings.

Some operations are even more impressive:

str.len(): 27x faster
str.startswith(): 16x faster
str.endswith(): 15x faster

3. 889 MB Total Memory Saved

Across our test datasets (totaling 645 MB on disk), we saved nearly 1 GB of RAM in memory. For a real data pipeline processing dozens of datasets, this compounds quickly.

4. Memory Overhead: The Game Changer

The bottom chart reveals something crucial about how pandas handles strings:

Old string dtype (object):

CSV files on disk: 645 MB
Loaded into pandas: 1,714 MB
Memory overhead: 165.7% (more than doubles!)

New string dtype (PyArrow):

CSV files on disk: 645 MB
Loaded into pandas: 825 MB
Memory overhead: 27.9% (minimal overhead)

What does this mean?

When pandas reads a CSV file, it doesn't just store the raw bytes: it creates in-memory data structures for fast operations. The old object dtype was incredibly inefficient, essentially duplicating string data multiple times. The new PyArrow string dtype keeps overhead minimal with a smarter memory layout.

This is the difference between pandas 2's Python-object approach and pandas 3's columnar Arrow approach.

The Methodology: Why 5 Different Datasets?

Now that you've seen the results, let me explain how I tested this. Real-world data comes in many shapes and sizes. A single benchmark on one type of data wouldn't tell the whole story.

That's why I created 5 distinct datasets, each representing common patterns you'll encounter in production:

1. Low Cardinality Dataset (1M rows)

What it is: Repeated categorical values like product categories, status codes, regions, and priorities.

Why it matters: This is typical of business data - think order statuses, customer segments, or department codes. The same values repeat millions of times.

Example columns:

category: "Electronics", "Clothing", "Food" (10 unique values)
status: "pending", "completed", "failed" (4 unique values)

2. High Cardinality Dataset (1M rows)

What it is: Mostly unique strings like user IDs, email addresses, and session tokens.

Why it matters: When every row is different (like customer emails or transaction IDs), pandas can't use simple optimizations. This tests worst-case scenarios.

Example columns:

user_id: "USER_00000001", "USER_00000002"... (1M unique)
email: "user123@example45.com" (1M unique)

3. Mixed String Lengths Dataset (1M rows)

What it is: A combination of short codes (2-5 chars), medium names (20-50 chars), and long descriptions (100-300 chars).

Why it matters: Real data isn't uniform. You might have product codes next to customer addresses next to order notes. This tests how pandas handles variable-length strings.

4. Dataset With Nulls (1M rows)

What it is: Data with missing values (10-33% nulls in different columns).

Why it matters: Messy data is reality. How does pandas 3.0 handle missing string data compared to pandas 2?

5. Large Dataset (10M rows)

What it is: A scaled-up version to test performance at scale.

Why it matters: Memory savings that look good at 1M rows might behave differently at 10M rows. This validates the findings scale linearly.

Memory Savings by Dataset Type

The memory savings from PyArrow strings vary significantly by dataset characteristics:

Best Case: Low Cardinality Data (-71.6%)

When data has repeated values (like categories), PyArrow strings shine:

Object dtype: 219 MB
PyArrow string dtype: 62 MB
Savings: 71.6%

Worst Case: Mixed String Lengths (-30.6%)

Variable-length strings see smaller (but still significant) savings:

Object dtype: 383 MB
PyArrow string dtype: 266 MB
Savings: 30.6%

The Pattern

Notice how savings correlate with data characteristics:

Repeated values (low cardinality) → Best savings (64-72%)
Unique values (high cardinality) → Good savings (53-55%)
Variable length (mixed sizes) → Moderate savings (31%)

Takeaway: PyArrow strings help everywhere, but they're especially powerful for categorical-like data.

Performance: Operation-Specific Speedups

This heatmap shows how much faster PyArrow strings are compared to object dtype for common string operations (values > 1.0 mean PyArrow is faster).

The Fastest Operations

str.len(): 10-27x faster
str.startswith() and str.endswith(): 11-18x faster
str.contains(): 3-5x faster
str.split(): 1-8x faster

The Pattern

Read operations (like len(), startswith()) → Massive speedups (10-27x)

These operations just examine existing data without modification

Transform operations (like replace(), split()) → Good speedups (2-5x)

These operations create new data, which limits the performance gains

The Trade-off: CSV Loading Time

There's no such thing as a free lunch. While PyArrow strings save memory and run operations faster, loading CSV files is 9%-61% slower.

Why the Slowdown?

When pandas reads a CSV with PyArrow strings enabled:

It parses the text (same as before)
It converts strings to PyArrow's columnar format (extra step)
This conversion involves building dictionary encodings and optimized memory structures

Pandas is doing more work upfront to enable better performance downstream.

Real-world impact: On our 10M row dataset, the difference is 1.63s vs 2.02s, an extra 0.4 seconds for 10 million rows. For many data pipelines, this upfront cost might be negligible compared to the 2-27x speedup in subsequent operations.

Pros and Cons: Should You Adopt PyArrow Strings?

Benefits of PyArrow String Dtype

Massive Memory Savings (30-72%)
Dramatically Faster String Operations (2-27x)
Minimal Memory Overhead (28% vs 166%)
Modern Data Ecosystem Integration

Trade-offs to Consider

Slower CSV Loading (9-61% slower)
- Initial data ingestion takes longer
- May impact workflows that repeatedly load small files
- The trade-off: slower start, much faster operations
Behavioral Changes
- String dtype behaves differently from object dtype in edge cases
- Need to update code that explicitly checks for object dtype
- Testing required for migration

The Recommendation

For most data workflows, PyArrow strings are a clear win. The memory and performance benefits far outweigh the trade-offs.

Consider staying with object dtype if:

You rarely work with string columns
Your datasets easily fit in memory
Load time is critical and you rarely perform string operations
You have legacy code that's deeply coupled to object dtype behavior

Definitely adopt PyArrow strings if:

You process large datasets with text data
String operations are a significant part of your workflow
Memory is a constraint in your environment
You're building production data pipelines
You work with modern data tools (Parquet, Arrow, DuckDB, etc.)

Conclusion

Our comprehensive analysis across 5 diverse datasets and 15+ string operations conclusively shows that PyArrow-backed strings deliver transformative improvements:

51.8% average memory savings across all dataset types
6.17x average operation speedup for string operations
Minimal memory overhead (28% vs 166% with Python objects)

PyArrow strings aren't just an incremental improvement, they're a fundamental reimagining of how pandas handles text data. By adopting Apache Arrow's proven columnar format, pandas has joined the modern data ecosystem while delivering massive performance and memory improvements.

For most data practitioners working with text data, the question isn't "Should I use PyArrow strings?" but rather "How quickly can I migrate?"

Questions or feedback? Feel free to open an issue or contribute to this analysis! The code we used in this analysis has been uploaded to this repo.

Stop Managing Clusters: Building a Scalable Serverless Data Lake on AWS

Mon, 06 Apr 2026 20:52:19 +0000

Building data pipelines shouldn't feel like babysitting servers. If you’ve ever managed a dedicated cluster just to run a few SQL queries, you know the pain: capacity planning, idle costs, and the "fun" of scaling infrastructure at 3 AM.

As a Data Engineer, I always follow a simple mantra: Design, then exist. (Or in this case: Design serverless, then relax.)

Today, we’re breaking down how to centralize your fragmented data into a Serverless Data Lake using the "Big Three" of AWS: S3, Glue, and Athena.

⚠️ The Problem: The "Data Swamps"
When an organization grows, its data ends up in silos—CSV reports in one bucket, application logs in another, and transactional exports scattered everywhere.

The result? Data is hard to find, and your analytical queries require heavy ETL before you can even start.

️ The Architecture: Storage ≠ Compute

The beauty of a serverless approach is the decoupling of storage from compute. You only pay for what you store and what you process.

Amazon S3 (The Backbone): This is your central repository. We don't just "dump" data here; we organize it into Layers:
Raw Layer: The "Source of Truth." Data exactly as it arrived.
Curated Layer: Cleaned, partitioned, and optimized data (think Parquet format).
AWS Glue (The Librarian): You don't want to manually define schemas. Glue Crawlers scan your S3 buckets, infer the schema, and populate the Glue Data Catalog.
Amazon Athena (The Engine): A serverless query engine that lets you run standard SQL directly against your files in S3. No clusters to spin up.

️ Mini-Demo: From S3 to SQL in 3 Steps

Ingest: Drop your dataset (CSV or JSON) into your raw S3 bucket.
Catalog: Run a Glue Crawler. It will "detect" your columns and create a table in the metadata catalog.
Query: Head over to the Athena Console and run a query like this:

SQL
-- Aggregating sales data directly from S3 files
SELECT

region, 
SUM(amount) as total_sales

FROM "data_lake_db"."sales_curated"
GROUP BY region
ORDER BY total_sales DESC;

Pro-Tips for the Community
If you're building this for production, keep these two things in mind:

Friends don't let friends use CSV for Analytics: Convert your data to Apache Parquet. Because it’s a columnar format, Athena only reads the columns you need. This can reduce your query costs by up to 90%.
Partitioning is King: Organize your S3 paths by date (e.g., s3://my-bucket/year=2026/month=04/). This limits the amount of data Athena has to scan, making your queries lightning-fast.

Final Thoughts
Serverless Data Lakes allow us to experiment fast. You can build a proof-of-concept in an afternoon and scale it to petabytes without ever touching a Linux terminal.

I’m curious—what’s your biggest headache when managing data at scale? Are you team "Serverless All The Way" or do you still prefer dedicated clusters for specific workloads?

Let’s discuss in the comments!

Tech Ecosystem Observatory: How I Built a Cloud-Native Data Pipeline to Track Global Tech Layoffs

Mon, 30 Mar 2026 07:55:31 +0000

Just completed my DEZ Zoomcamp 2026 capstone project — the Tech Ecosystem Observatory

Built a full cloud-native batch data pipeline from scratch that answers: which industries are shedding the most jobs, and how does that correlate with YC startup activity?

Here's what the pipeline looks like end to end:

✅ Terraform — provisioned GCS bucket and BigQuery datasets as infrastructure as code

✅ Docker — containerized all ingestion scripts into a portable image

✅ Kestra — orchestrated a 4-task batch DAG running weekly every Monday 6AM UTC

✅ Google Cloud Storage — raw JSONL data lake storing layoffs and YC company data

✅ BigQuery — partitioned by date (monthly) and clustered by industry/country for optimized queries

✅ dbt Cloud — built staging views and mart tables (mart_monthly_layoffs + mart_tech_ecosystem)

✅ Looker Studio — 2-page interactive dashboard with layoff trends, geo maps, ecosystem stress ratios

Data: 4,317 layoff events (2023–2024) + 5,690 YC-backed companies

Live dashboard: https://lookerstudio.google.com/reporting/b1620cae-97cb-4911-82b8-dd0c46ee8acb

GitHub: https://github.com/Derrick-Ryan-Giggs/tech-ecosystem-observatory

Huge thanks to @DataTalksClub and Alexey Grigorev for building and maintaining this incredible free course. If you're serious about data engineering, this is where you start

https://github.com/DataTalksClub/data-engineering-zoomcamp/

Who else is building data pipelines? Drop your projects below

Streaming with PyFlink and Redpanda

Mon, 16 Mar 2026 13:34:43 +0000

Week 7 of Data Engineering Zoomcamp by @DataTalksClub complete

Just finished Module 7 - Streaming with PyFlink. Learned how to:

Set up Redpanda as a Kafka replacement
Build Kafka producers and consumers in Python
Create tumbling and session windows in Flink
Analyze real-time taxi trip data with stream processing

Here's my homework solution: https://github.com/Derrick-Ryan-Giggs/Streaming-Homework

You can sign up here: https://github.com/DataTalksClub/data-engineering-zoomcamp/

Batch Processing with Apache Spark

Sat, 07 Mar 2026 13:04:56 +0000

Week 6 of Data Engineering Zoomcamp by @DataTalksClub complete

Just finished Module 6 - Batch Processing with Spark. Learned how to:

✅ Set up PySpark and create Spark sessions

✅ Read and process Parquet files at scale

✅ Repartition data for optimal performance

✅ Analyze millions of taxi trips with DataFrames

✅ Use Spark UI for monitoring jobs

Processing 4M+ taxi trips with Spark - distributed computing is powerful

Here's my homework solution: https://github.com/Derrick-Ryan-Giggs/pyspark-homework

Following along with this amazing free course - who else is learning data engineering?

You can sign up here: https://github.com/DataTalksClub/data-engineering-zoomcamp/

Why Most Data Projects Fail Before the First Model Is Built

Thu, 05 Mar 2026 18:45:16 +0000

Many organizations invest in AI, analytics, and dashboards — yet most data projects fail before the first model is even built.

When people think about data projects, they often imagine machine learning models, predictive algorithms, and complex pipelines.

But in reality, most data initiatives fail long before any model is trained.

Not because the algorithms are weak.

But because the foundation is broken.

The Data Illusion

Organizations today generate enormous amounts of data.

They store logs, transactions, operational records, and performance metrics.

On paper, everything looks ready for analytics.

But when teams actually start working with the data, they quickly encounter problems:

Missing values
Inconsistent formats
Conflicting sources
Undefined metrics
Poor documentation

Suddenly, the project shifts from analysis to data archaeology.

Data Science Starts with Data Reliability

Before any meaningful analysis can begin, teams must answer fundamental questions:

What is the source of truth?
Who owns the dataset?
How frequently is it updated?
What transformations are applied?
Are definitions consistent across departments?

Without clear answers, even the most advanced models produce misleading insights.

The Hidden Cost of Poor Data Foundations

Many organizations invest heavily in analytics tools, dashboards, and AI platforms.

But without strong data foundations, these investments create an illusion of intelligence.

Dashboards become visually impressive but operationally misleading.

Models generate predictions, but the inputs themselves are unstable.

This leads to one of the most dangerous outcomes in data work:

False confidence.

Decisions start relying on numbers that appear precise but are fundamentally unreliable.

Data Engineering Is the Real Backbone

In practice, the majority of effort in data projects is not modeling.

It is:

Data cleaning
Validation
Schema alignment
Pipeline reliability
Monitoring data quality

That is why experienced teams often say:

“80% of data science is data preparation.”

And the better the data infrastructure, the faster meaningful insights appear.

A Simple Rule for Data Teams

Before building any model, ask three questions:

Is the data trustworthy?
Is the definition of the metric consistent?
Can the pipeline reproduce the same dataset tomorrow?

If the answer to any of these is unclear, the problem is not analytical.

It is architectural.

Final Thought

Good data teams do not start with models.

They start with reliability.

Because in data systems, accuracy is not created by algorithms.

It is created by architecture.

If you're interested in systems thinking, data architecture, and enterprise optimization, feel free to connect.

LinkedIn: https://www.linkedin.com/in/fadydesokysaeedabdelaziz

GitHub: https://github.com/fadydesoky

From APIs to Warehouses: AI-Assisted Data Ingestion with dlt

Sun, 01 Mar 2026 11:45:47 +0000

dlt Workshop of Data Engineering Zoomcamp by @DataTalksClub complete

Just finished the Data Ingestion workshop with @dltHub. Learned how to:
✅ Build REST API data pipelines with dlt
✅ Use AI-assisted development with dlt MCP Server
✅ Load paginated API data into DuckDB
✅ Inspect pipeline data with dlt Dashboard and marimo notebooks

Built a full NYC taxi data pipeline from a custom API - AI-assisted data engineering is the future

Here's my homework solution: https://github.com/Derrick-Ryan-Giggs/-my-dlt-taxi-pipeline

Following along with this amazing free course - who else is learning data engineering?

You can sign up here: https://github.com/DataTalksClub/data-engineering-zoomcamp/

The Day BI Died in Real Time: What MCP Integration With a Semantic Layer Means for Data Engineers

Fri, 27 Feb 2026 18:32:35 +0000

On December 12, 2025, Strategy CPO Saurabh Abhyankar watched Claude Code connect to Mosaic through the Model Context Protocol (MCP) and build a complete analytics application. No dashboard tool. No BI platform. Just an AI coding agent querying a semantic layer and generating a working app.

At Strategy World 2026, Abhyankar told a room of industry analysts and practitioners that this was the moment he knew BI was dead. Not dying. Dead. Twenty years of building BI tools, he said — billions of dollars of engineering — and an AI coding agent replicated it in minutes.

That's a bold claim from a CPO whose company sells BI tools. But the technical architecture underneath it is worth understanding, regardless of whether you buy the marketing. Because what's actually happening at the protocol level — MCP connecting AI agents to governed semantic layers — changes how data engineers and developers will build analytics for the foreseeable future.

The Abstraction Gap That Explains Everything

Abhyankar's keynote built on a technical argument that developers and data engineers should find familiar: the history of productivity in software is the history of abstraction layers.

Physical circuits gave way to assembly language, which delivered roughly a 50x productivity improvement. High-level languages and eventually Python and AI-assisted coding have pushed that number to approximately 22,000x over six decades. Each layer of abstraction let engineers work at a higher level while the toolchain handled the complexity underneath.

Data engineering followed a different trajectory. The move from flat files to relational databases and SQL delivered about a 600x improvement. And then it stopped. The Big Data era actually regressed — MapReduce and Java-based processing were less productive than SQL for most analytical workloads. Modern lakehouse platforms like Snowflake and Databricks abstracted compute from storage, which was a meaningful architectural advance, but the query interface is still SQL. The abstraction layer didn't move up.

That's the gap. Software engineering productivity improved 22,000x. Data engineering productivity improved 600x and stalled. Abhyankar's argument is that this gap explains why data engineering projects take so long relative to software development, why data teams are perpetually backlogged, and why the BI industry has been building incrementally better versions of the same thing for 30 years.

Mosaic, in this framing, is the next abstraction layer. Instead of working at the SQL and table level, data engineers work at the business semantics level — defining what entities, metrics, and relationships mean. The platform handles the SQL generation, table creation, and query optimization underneath, the same way a compiler handles the conversion from high-level code to machine instructions.

How MCP Changes the Architecture

The Model Context Protocol is what makes the December 12 demo more than a parlor trick. MCP provides a standardized way for AI agents to discover and interact with external tools and data sources. When Claude Code connected to Mosaic via MCP, it didn't need to know the underlying database schema, the SQL dialect, or the table relationships. It queried the semantic layer, which translated business-level questions into the correct technical operations.

This is architecturally significant for three reasons.

First, the AI agent works at the same level of abstraction as a business user. It asks for revenue by region or inventory by store, and the semantic layer handles the joins, filters, and aggregations. The agent doesn't generate raw SQL against source tables. It generates queries against governed business definitions. That means the output is consistent regardless of which agent, which LLM, or which interface is asking.

Second, governance travels with the data. Because the agent queries through the semantic layer, every access control, metric definition, and business rule applies automatically. An AI agent building an inventory management app gets the same governed data as a dashboard, a notebook, or a human asking questions through conversational BI. The governance isn't bolted on after the fact. It's baked into the query path.

Third, the application layer becomes disposable. If an AI coding tool can generate a working analytics app from a requirements document in minutes, the traditional build cycle — requirements gathering, development, testing, deployment, maintenance — collapses. The app itself stops being the valuable artifact. The semantic layer underneath it becomes the durable asset.

Abhyankar demonstrated this live on stage with three different AI platforms. Gemini pulled data from Mosaic to create a product management workflow across CRM and ticketing systems. Claude generated multi-sheet Excel analyses with visualizations and built executive presentations from the same governed data source. OpenAI's Codex took a requirements document and built a complete inventory management application that worked on iPad and iPhone — from specification to working app in minutes.

The common thread: all three used Mosaic as the data layer through MCP. None of them needed to know anything about the underlying databases.

Open Semantic Interchange and Git Integration: What Matters to Engineers

Abhyankar announced that Strategy has joined the Open Semantic Interchange initiative. The practical implication: semantic models defined in Mosaic can be exported and imported in a common YAML format. On paper, this means enterprises aren't locked into a single semantic layer vendor. They can move definitions between tools or run multiple semantic layers that stay synchronized.

More immediately useful for engineering teams: Mosaic now supports Git integration. Semantic models can be version-controlled, run through CI/CD pipelines, and managed with standard diff and merge workflows. If you're a data engineer who's been maintaining metric definitions in documentation or tribal knowledge, this is the shift toward treating semantic models as code — versioned, tested, reviewed, and deployed through the same pipelines as application code.

Abhyankar was candid that enterprises will end up with multiple semantic layers. Companies acquire other companies, different teams adopt different tools, and the notion of a single universal layer across a large enterprise is aspirational. The OSI standard and Git integration are Strategy's answer to that reality: if you're going to have multiple semantic layers, at least make them interoperable and manageable through standard engineering workflows.

From Descriptive Semantics to Business Ontology

The most technically ambitious part of the Mosaic roadmap is the expansion from a semantic layer into a full business ontology. Traditional semantic layers are descriptive: they define what entities and metrics mean. Mosaic is adding verbs, rules, subtypes, workflows, and operational descriptions.

In concrete terms, that means encoding not just that a "customer" entity exists, but that a customer buys products, returns products, and must buy before returning. Premium customers inherit from the base customer entity with additional properties. Stores operate according to defined workflows. Factory material flows follow specific sequences.

During the analyst briefing, one industry researcher drew the distinction between descriptive and prescriptive semantics. A descriptive layer tells you what your data looks like. A prescriptive layer tells agents what to do with it. That distinction matters for the AI use case. If you want an agent to take action on behalf of the business — not just answer questions but execute workflows — it needs prescriptive rules, not just descriptive metadata.

Abhyankar described a system where Mosaic can suggest relationships automatically when it sees entities. If it encounters a customer entity and a product entity, it can propose that the relationship is "customer buys product" based on common patterns. Companies that have existing business process documentation can feed that into Mosaic to accelerate the ontology build. And over time, metadata from existing Strategy deployments across industries can be aggregated into vertical templates — a default semantic layer for financial services, retail, healthcare, and so on.

This is where the vision gets ambitious and the execution risk increases. Encoding an entire business into a semantic ontology is a massive undertaking. It's also where AI has real limitations.

What Production Looks Like Today

At the same conference, Sachin Bhatta, BI Director at the North Texas Tollway Authority, described what it's actually like to run Mosaic in production. NTTA processes over 300 billion toll transactions and moves 1.7 terabytes of data daily. They were one of the earliest Mosaic pilot customers and are now live in dark mode alongside their existing infrastructure.

Bhatta's experience offers useful data points for engineering teams evaluating the platform.

On the data warehouse question: Bhatta is keeping his BigQuery environment. He sees the direction Mosaic is heading, but he's waiting for the technology to prove out at his data volumes before migrating. For new projects and new data sources, he starts in Mosaic. For existing infrastructure, he runs both in parallel.

On AI limitations: Bhatta offered a practical corrective that developers need to hear. AI handles textual queries well, but it still struggles with mathematical calculations and computations. That's why the metrics layer in Mosaic matters separately from the LLM integration. Calculated metrics need to be defined and computed by the engine, not generated by a language model that might hallucinate the math. Mosaic addresses this with two types of agents — one that operates on the semantic model and one that operates on dashboards — keeping computation separate from natural language interpretation.

On integration with existing Strategy environments: this is the gap Bhatta flagged most directly. Organizations that have invested years in building semantic layers with thousands of attributes and metrics on Strategy's traditional Intelligence Server platform need a bridge to Mosaic. That bridge doesn't fully exist yet. New projects work well in Mosaic. Migrating legacy semantic models is still a work in progress.

On preparatory work: Bhatta was blunt about what has to happen before Mosaic or AI delivers value. Your data dictionary, data catalog, and naming conventions need to be in order. Mosaic reads your metadata. If the metadata is inconsistent, the LLM answers will be inconsistent. No amount of AI sophistication fixes dirty metadata.

NTTA also provided a useful cost comparison. Before Mosaic, Bhatta's team ran customer clustering through Google Vertex AI — 32 separate processes involving knowledge graphs and ML models on BigQuery. Mosaic's built-in clustering feature could have handled much of that work within a single tool at significantly lower cost. For engineering teams doing cost-benefit analysis, that's a concrete data point.

What Data Engineers Should Pay Attention To

Strip away the conference marketing and three technical developments from Strategy World 2026 are worth tracking.

The first is MCP as the standard interface between AI agents and governed data layers. This isn't specific to Strategy. Any semantic layer or data platform that exposes its capabilities through MCP becomes accessible to any AI agent that supports the protocol. The competitive question isn't which semantic layer is best. It's which one your AI toolchain can talk to most effectively.

The second is semantic models as code. Git integration, CI/CD pipelines, version control, diff and merge — these are the workflows that data engineers already use for everything else. Bringing semantic model management into the same toolchain removes a category of manual governance work and makes semantic definitions testable and auditable in the same way application code is.

The third is the shift from descriptive to prescriptive semantics. If your organization is planning for agentic AI — agents that don't just answer questions but execute business workflows — your semantic layer needs to express business rules, relationships, and constraints, not just metric definitions. That's a significantly larger modeling effort than what most organizations have undertaken, and it's worth starting the planning now even if the tooling isn't fully mature.

Abhyankar gave the room a timeline for the data warehouse transition: some elements in six months, substantial capability within five years. That's an honest range from a CPO who watched his own product category get replicated by an AI agent in real time.

For data engineers, the practical move is the same one NTTA is already making: start new projects in the semantic layer, keep existing infrastructure running, invest in metadata quality, and watch how MCP integration with your AI toolchain evolves over the next two quarters. The architecture is shifting. The timeline depends on your data volumes, your governance maturity, and how much of your current pipeline is adding value versus just moving bytes.

Data Platforms with Bruin

Tue, 24 Feb 2026 09:04:57 +0000

Week 5 of Data Engineering Zoomcamp by @DataTalksClub complete

Just finished Module 5 - Data Platforms with Bruin. Learned how to:

✅ Build end-to-end ELT pipelines with Bruin
✅ Configure environments and connections
✅ Use materialization strategies for incremental processing
✅ Add data quality checks to ensure data integrity
✅ Deploy pipelines from local to cloud (BigQuery)

Modern data platforms in a single CLI tool - no vendor lock-in

Here's my homework solution: https://github.com/Derrick-Ryan-Giggs/bruin-taxi-pipeline-homework

Following along with this amazing free course - who else is learning data engineering?

You can sign up here: https://github.com/DataTalksClub/data-engineering-zoomcamp/

Analytics Engineering

Sun, 15 Feb 2026 11:51:43 +0000

Week 4 of Data Engineering Zoomcamp by @DataTalksClub complete

Just finished Module 4 - Analytics Engineering with dbt. Learned how to:

✅ Build transformation models with dbt
✅ Create staging, intermediate, and fact tables
✅ Write tests to ensure data quality
✅ Understand lineage and model dependencies
✅ Analyze revenue patterns across NYC zones

Transforming raw data into analytics-ready models - the T in ELT

Here's my homework solution: https://github.com/Derrick-Ryan-Giggs/Analytics-Engineering

Following along with this amazing free course - who else is learning data engineering?

You can sign up here: https://github.com/DataTalksClub/data-engineering-zoomcamp/

Data Is Not a Department — It’s a Decision Architecture

Fri, 13 Feb 2026 12:47:57 +0000

Most organizations say they want to be “data-driven.”

They invest in dashboards.
They build reporting systems.
They hire analysts.

Yet somehow, decisions still rely on intuition, hierarchy, or urgency.

The issue is rarely the absence of data.

It’s the absence of decision architecture.

The Illusion: “We Need More Data"

In many enterprise environments, the default reaction to underperformance is:

“We need better data.”

But when you look closely, the organization often already has:

Historical performance metrics
Operational KPIs
Real-time dashboards
Financial summaries
Compliance reports

The problem isn’t collection.

It’s translation.

Data exists — but decision flow does not.

Where Data Quietly Fails

Over time, I’ve noticed a recurring pattern in large systems.

Reports are generated. Dashboards are circulated. Metrics are reviewed.

But very few questions are clearly defined:

Who owns this metric?
What threshold triggers action?
What happens when it deviates?
Who is accountable for response?
Is there feedback after the decision?

Without these layers, data becomes informative — not operational.

And informative systems do not scale well.

Data Without Structure Becomes Noise

Imagine a KPI dashboard that shows:

Turnaround time
Error rate
Processing backlog
Resource utilization

All useful.

But unless the organization defines:

A decision threshold
A response protocol
An escalation path
A measurable follow-up

The dashboard becomes a passive artifact.

It looks analytical. It feels analytical. But it doesn’t change behavior.

That’s not a data problem.

That’s an architecture problem.

What Decision Architecture Actually Means

Decision architecture is not a tool.

It’s a structured design that connects:

Metrics
Ownership
Triggers
Actions
Feedback loops

In mature systems, every critical metric answers:

When does this require action?
Who initiates the action?
What is the defined response?
How is impact measured afterward?

That is when data becomes operational.

A Practical Enterprise Lens

In large organizations, complexity multiplies quickly:

Multiple regions
Multiple policy versions
Multiple approval layers
Multiple reporting lines

Without structured decision architecture, each layer interprets the same data differently.

Over time, this creates:

Fragmented responses
Delayed interventions
Manual overrides
Invisible technical debt

And eventually, performance issues are treated as isolated incidents instead of systemic misalignments.

The Human Element

There’s also a subtle human factor.

When decision rules are unclear:

Teams hesitate.
Accountability blurs.
Escalation becomes personal instead of structural.

Clear architecture removes ambiguity. It replaces emotion with logic. And it reduces friction across teams.

Data alone cannot do that.

Design can.

From Data Collection to Data Responsibility

Organizations that truly operate on data do something different.

They don’t just measure.

They define:

Ownership boundaries
Trigger conditions
Decision latency limits
Review cycles
Iteration mechanisms

They treat data as infrastructure — not decoration.

Final Thought

Good organizations collect data.

Mature organizations analyze it.

High-performing organizations design decisions around it.

Data is not a department.

It is an architectural responsibility.

And when that architecture is clear, performance becomes a byproduct — not a struggle.

Finally, If you’re interested in systems thinking, enterprise optimization, and designing responsible decision frameworks, feel free to connect.

LinkedIn: https://www.linkedin.com/in/fadydesokysaeedabdelaziz
GitHub: https://github.com/fadydesoky

Data Warehousing & BigQuery

Sun, 08 Feb 2026 11:13:57 +0000

Week 3 of Data Engineering Zoomcamp by @DataTalksClub complete!
Just finished Module 3 - Data Warehousing with BigQuery. Learned how to:
Create external tables from GCS bucket data
Build materialized tables in BigQuery
Partition and cluster tables for performance
Understand columnar storage and query optimization
Analyze NYC taxi data at scale
Working with 20M+ records and learning how partitioning reduces query costs!

Here's my homework solution: https://github.com/Derrick-Ryan-Giggs/Module-3-Data-Warehousing-BigQuery-homework

Following along with this amazing free course - who else is learning data engineering?
You can sign up here: https://github.com/DataTalksClub/data-engineering-zoomcamp/

Coder Legion - Recent posts tagged dataengineering

DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap

The contenders

Setup

Results at 150M rows

Polars and the window wall

DuckDB under pressure: memory_limit = 1 GB

DuckDB 1.5.6 vs 2.0.0

Where Spark still wins

Failure modes I hit, so you don't have to

Reproduce it

The takeaway

Sources

Advanced SQL Analytics for Data Analysis.

Advanced SQL analytics for running a hospital database.

Introduction

1. SQL Window Functions

OVER, PARTITION BY, ORDER BY, and Window Frames

Ranking Window Functions

Value & Navigation Window Function

Aggregate Window Functions

2. SQL Joins

INNER JOIN

LEFT JOIN

RIGHT JOIN

FULL OUTER JOIN

CROSS JOIN

SELF JOIN

Join Summary

3. Common Table Expressions (CTEs)

Simple CTEs

Multiple Chained CTEs

Recursive CTEs

CTEs vs. Subqueries

4. SQL Subqueries

Subquery Types

Subqueries in SELECT Clauses

Using Subqueries in FROM Clauses (Derived Table)

Subqueries in WHERE Clause

Conclusion

GitHub

Python 3.15 Is Not a Speed Release for Data Teams. It Is an Operations Release.

The thesis

The antithesis, at full strength

Setup

JIT off vs on

1. Lazy imports fix startup, not throughput

Trap 1: the global switch broke pandas

Trap 2: side-effect imports silently stop happening

Trap 3: the PEP's kill switch does not exist in 3.15.0

Runbook: lazy imports

2. Tachyon: attach to the stuck job by PID, and read the error rate

Runbook: profiling a stuck job

3. .pth import lines become .start files: auditable, not safe

Runbook: startup code

What to do this week

Instant at 10 Million Rows, Dead at a Billion: A Decision Framework for Billion-Row SQL

The query that was instant at 10 million rows and timed out at a billion

The two-axis framework

Engine class 1: Row-store OLTP — great at point lookups, brutal on full-table aggregates

Engine class 2: Columnar / OLAP — where scans and aggregates are the design point

Engine class 3: Distributed — horizontal scale, and the fan-out ceiling

The memory / spill axis — the one most write-ups skip

The spill cliff

When to re-architect instead of tune

One more trap: deep pagination

The crossover-signal checklist: change engine class, don't add another index

Cost and economics for the decision-makers

Build it yourself

What to do Monday morning

Understanding Parquet File Format

What is a File Format?

Introduction to Parquet

The Everyday Analogy: Organizing Your Groceries

The Technical View

Why Choose Parquet Over Other Formats?

How Data is Written in Parquet

How Data is Read from a Parquet File

Parquet File Size: Small vs Large

How Data Types Affect Compression (and File Size)

3. `.pth` import lines become `.start` files: auditable, not safe