Coder Legion - Recent posts tagged duckdb

DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap

Thu, 22 Oct 2026 08:52:00 +0000

Originally published on DEV.to: https://dev.to/firfircelik/duckdb-20-vs-polars-vs-spark-on-one-machine-150m-rows-under-a-45-gib-cap-h38

DuckDB 2.0.0 shipped on October 21, 2026 (announcement, PyPI). The next morning I ran the same single-machine benchmark I had already run on DuckDB 1.5.6: three engines, one 8 vCPU VM, Parquet in and Parquet out, a hard memory cap, and four queries that look like real data engineering work.

The thesis I started with, and that the numbers support:

DuckDB beats Spark-on-one-box on memory-bound DE work. It finished every run, at every size, 5 to 12 times faster than local PySpark at 150M rows, and it still finished the heavy queries with a 1 GB memory limit.
Polars hits a wall on window dedup. At 150M rows the ROW_NUMBER() style "latest row per customer" query never finished under the cap. Not at 4.5 GiB, not with an out-of-core budget, not at 9 GiB.
Polars is not slow. On filters, joins and a high-cardinality group by it ties DuckDB or beats it, when it fits.

Everything below is measured on that one box. Where I interpret or give an opinion, I say so.

The contenders

	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Model	In-process SQL engine	In-process DataFrame engine, lazy + streaming	JVM engine, `local[8]`
How I drove it	`COPY (SQL) TO 'out.parquet'`	`LazyFrame.sink_parquet()`	`spark.sql(...).write.parquet()`
Memory knob	`SET memory_limit = '3GB'`	default (plus one run with an out-of-core budget)	`spark.driver.memory = 3g`
Threads	8	8 (`POLARS_MAX_THREADS`)	8

Polars 2.0.0 was published on PyPI on October 6, 2026, PySpark 4.2.0 on July 14, 2026. DuckDB 2.0.0 wheels landed on PyPI on October 21, the same day as the release calendar entry.

Setup

Hardware and versions

VM: 8 vCPU Intel Xeon, 15 GiB RAM, no swap, Linux 6.12. It is a shared box, so other processes were running. Every run waited until at least 5 GiB was free and CPU busy was under 30% over a 3 second window.
Python 3.13.5, OpenJDK 21.0.12, pyarrow 26.0.0.
DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0. The "before" numbers are DuckDB 1.5.6 on the same box and data, run on October 19.

Data. A seeded synthetic star schema written with pyarrow: a sales fact table and a 50,000-row products dimension. Three sizes: 10M rows (213 MB), 50M (1.1 GB) and 150M (3.2 GB of snappy Parquet, 30 files of 5M rows). About 10 orders per customer, so 150M rows means roughly 15M customers.

Queries. The same SQL for DuckDB and Spark, the same logic as Polars expressions:

q1_filter_agg: filter on a timestamp and status, group by store and status.
q2_join_agg: fact-to-dimension join, group by category and brand.
q3_window_latest: latest order per customer with ROW_NUMBER(). The classic dedup and SCD pattern.
q4_highcard_agg: one row per customer (about 15M groups at 150M).

-- q3_window_latest
SELECT customer_id, order_id, ts, qty * price AS amount
FROM (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                 ORDER BY ts DESC, order_id DESC) AS rn
    FROM sales
) t
WHERE rn = 1

Method.

Every run is a fresh process: interpreter start, import, session, read Parquet, compute, write Parquet. So no warm caches inside the engine, and Spark pays its JVM start every time.
Engines run strictly one at a time. The engine order rotates per repetition so no engine always goes first.
5 repetitions at 10M, 3 at 50M and 150M. I report medians.
A watchdog samples the RSS of the whole process tree every 50 ms and kills the run if it goes above 4.5 GiB. That is the cap. It is deliberately tighter than the box, because "fits in a small container" is the question I care about.
Wall is the whole process. Query is read, compute and write only. The difference is setup.
Correctness: a validation script compares row counts and rounded checksums of every output. Where an engine produced output, all three engines agreed, for every query and size.

Results at 150M rows

Query	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Filter + aggregate	0.87 s (3/3)	0.96 s (3/3)	10.77 s (3/3)
Join + aggregate	1.47 s (3/3)	1.46 s (3/3)	16.10 s (3/3)
Window dedup	7.92 s (3/3)	killed (0/3)	46.29 s (2/3)
High-cardinality group by	7.91 s (3/3)	4.21 s (1/3)	39.70 s (3/3)

Median wall seconds, runs that finished under the 4.5 GiB cap in brackets.

Peak RSS (median of finished runs) tells the other half:

Query	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Filter + aggregate	0.77 GiB	0.94 GiB	1.85 GiB
Join + aggregate	0.42 GiB	1.96 GiB	1.61 GiB
Window dedup	3.01 GiB	above 4.5 (killed)	3.74 GiB, one run hit 7.51
High-cardinality group by	3.01 GiB	4.49 GiB	4.40 GiB

Three things stand out.

Spark on one box pays twice. About 5 seconds of every Spark run is setup (median 5.0 to 5.8 s, JVM plus session) before it reads a byte. Then the query itself is slower: query-only time was 7.4x DuckDB's on the filter, 7.9x on the join, 5.2x on the window dedup and 4.3x on the group by. In wall time that is 5x to 12x. And spark.driver.memory = 3g did not keep the process at 3 GiB: one window dedup run reached 7.51 GiB of process-tree RSS and was killed.

Polars is fast right up to the cap. It tied DuckDB on filter and join and was the fastest engine on the high-cardinality group by (4.21 s against 7.91 s). But only 1 of 3 runs finished: that one peaked at 4.49 GiB, and the other two crossed the cap at 4.50 and 4.55 GiB. Fast and close to the edge is a fine place for a notebook and a bad place for a scheduled job.

The window dedup is where Polars stops. At 150M rows it was killed in all 3 runs, within a few seconds of starting, at 4.52 to 4.66 GiB.

Polars and the window wall

I gave Polars two more chances on the 150M window dedup:

Polars 2.0.0, 150M window dedup	Result
Default settings, 4.5 GiB cap (Oct 22)	0/3, killed at 4.52 to 4.66 GiB
`POLARS_OOC_MEMORY_BUDGET_MB=3072`, 4.5 GiB cap (Oct 22)	0/3, killed at 4.57 to 4.66 GiB
Default settings, 9 GiB cap (Oct 19, same Polars 2.0.0)	0/3, killed at 9.04 to 9.54 GiB

The out-of-core budget variable is one I found in the Polars 2.0.0 source, not a documented knob, so treat that row as "I tried", not as guidance. The same variable did help the group by: with it, Polars finished 3/3 at 4.67 s and 4.18 to 4.30 GiB.

At smaller sizes the dedup works, but the shape is already visible:

Window dedup	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
10M rows	0.60 s, 0.65 GiB	1.83 s, 1.20 GiB	12.74 s, 2.94 GiB
50M rows	2.33 s, 2.19 GiB	10.36 s, 4.46 GiB	29.06 s, 3.77 GiB
150M rows	7.92 s, 3.01 GiB	killed	46.29 s, 3.74 GiB

At 50M Polars already used 4.46 GiB for a query DuckDB did in 2.19 GiB, and took 4.4x longer.

This is the Polars expression I used. It is the direct translation of the SQL:

import polars as pl

amount = pl.col("qty").cast(pl.Float64) * pl.col("price")
rn = pl.int_range(pl.len()).over(
    "customer_id", order_by=["ts", "order_id"], descending=True
)
(
    pl.scan_parquet("data/sales_150m/*.parquet")
    .filter(rn == 0)
    .select("customer_id", "order_id", "ts", amount.alias("amount"))
    .sink_parquet("out/polars/150m_q3_window_latest.parquet")
)

Fair caveat: I did not test other Polars formulations of the same dedup (a sort followed by unique(keep="first"), or a group_by().agg(...) with sort_by().last()). One of them may stream better. The point stands for the common case, though: the first thing a team writes is the window, and that one did not fit.

Update (Oct 22): after this went out, Victor Cid (@Vleyked) tested exactly that in engine-bench PR #1. His sort + unique(keep="first") version drops the in-memory window fallback from the plan and finished 3/3 at 150M rows under the same 4.5 GiB cap, peaking at 2.50 to 2.74 GiB, but only with POLARS_OOC_MEMORY_BUDGET_MB=1024. With default settings or a 3072 MB budget it still went over the cap. The price is speed: a 90.58 s median against 17.56 s for DuckDB on the same 4 vCPU runner, so roughly 5x slower. His output matched DuckDB row for row (14,999,347 rows). So Polars can do this dedup under the cap if you rewrite it and tune the spill budget, and with a smarter query it gets close to DuckDB (see below). His runs were on different hardware, so they are not in the tables above.

Second update: in engine-bench PR #2 he filtered each customer down to their max timestamp first and only sorted those rows to break ties. That brought the median down to 14.23 s at 150M rows with default Polars settings, peaking at 4.04 GiB under the same cap, with output matching DuckDB row for row. On his runner DuckDB's reference was a single 18.50 s run, so I'm not calling that a repeated cross-engine win. The real lesson is that the first way you write a Polars dedup can decide whether it finishes at all.

Third update (Oct 23): Ritchie Vink, the creator of Polars, pointed out that only DuckDB was told about the memory limit, so Polars did not know it had to stay under 4 GiB, and suggested setting POLARS_OOC_MEMORY_BUDGET_MB for a fairer comparison. I reran every Polars query at 150M rows on the same box under the same 4.5 GiB cap with that budget, next to a DuckDB rerun in the same session: the high-cardinality group by finished 3/3 at 6.17 s and 3.46 GiB with a 3072 MB budget, against 9.56 s for DuckDB (the box was busier than on Oct 22 and DuckDB ran slower too, so compare within this session, not with the tables above; the Oct 22 run with the same budget gave 4.67 s). The window dedup with over(order_by) still went over the cap in every run at 3072, 1024 and 512 MB, Victor's sort plus unique version fit only at 1024 MB (31.28 s, 2.29 GiB), and the max-timestamp rewrite finished 3/3 on default settings (5.61 s, 4.02 GiB). Ritchie says Polars main now spills the group by correctly and that this ships in the next release, which was not on PyPI yet when I ran this, so I will rerun when it ships. Raw results and output checks are in engine-bench PR #3 and in results/polars_ooc_budget_2026-10-23 on the polars-ooc-budget-2026-10-23 branch.

Fourth update (Oct 26): after the third update I reported the window dedup to Polars as pola-rs/polars#30270: the over(order_by) window inside filter() ignored POLARS_OOC_MEMORY_BUDGET_MB and ran in an in-memory fallback node. Ritchie Vink fixed it (#30272, #30282 and #30280), and the fixes shipped in Polars 2.1.0. I reran the 150M benchmark on the official 2.1.0 wheel on the same box, with the same harness and the same 4.5 GiB cap, next to a DuckDB rerun in the same session. With a budget set, the window dedup now finishes 3/3 at every budget I tried: 18.78 s and 3.34 GiB at 3072 MB, 20.11 s and 1.29 GiB at 1024 MB, 43.01 s and 2.15 GiB at 512 MB. On 2.0.0 it failed at all three. Memory now follows the budget closely: at 512 MB, Victor's sort plus unique version peaked at 0.69 GiB (73.85 s) and the high-cardinality group by at 0.74 GiB (9.43 s). DuckDB still ran the window dedup faster in the same session, 8.07 s, about 2.3x quicker than Polars at 3072 MB, and the max-timestamp rewrite remains the fastest Polars dedup (5.12 s on default settings). Every finished output matched DuckDB row for row (14,999,347 rows). Two caveats. Without a budget, the window dedup and sort plus unique still went over the cap. My cap is a process watchdog, not a cgroup limit, so the new cgroup-aware default in 2.1.0 had nothing to detect and is untested here; set the budget explicitly. One sort plus unique run at 3072 MB took 285.64 s for no clear reason, so I report medians of 3 runs. Raw results, plans and output checks: engine-bench PR #4 and the 2.1.0 results README.

DuckDB under pressure: memory_limit = 1 GB

The interesting DuckDB question is not "how fast with room to spare" but "what happens when the box is small". Same 150M rows, memory_limit cut to 1 GB:

150M, `memory_limit = 1GB`	Wall (median)	Peak RSS	Finished
Window dedup	14.97 s	1.12 GiB	3/3
High-cardinality group by	13.94 s	1.08 GiB	3/3
Window dedup, `preserve_insertion_order = false`	14.94 s	1.12 GiB	3/3
Group by, `preserve_insertion_order = false`	13.88 s	1.08 GiB	3/3

With a third of the memory, DuckDB spilled to disk and still beat Spark's unconstrained run by about 3x on both queries, at a quarter to a third of Spark's peak RSS. The DuckDB docs list window functions with PARTITION BY ... ORDER BY among the blocking operators it can process larger than memory (How to Tune Workloads). This run is that sentence, measured.

The setup is three lines:

import duckdb

con = duckdb.connect()
con.execute("SET memory_limit = '1GB'")
con.execute("SET threads = 8")
con.execute("SET temp_directory = '/mnt/fast-local-disk/duckdb_tmp'")  # where spills go
con.execute("""
    COPY (
        SELECT customer_id, order_id, ts, qty * price AS amount
        FROM (
            SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                         ORDER BY ts DESC, order_id DESC) AS rn
            FROM read_parquet('data/sales_150m/*.parquet')
        )
        WHERE rn = 1
    ) TO 'out/latest_per_customer.parquet' (FORMAT parquet)
""")

DuckDB 1.5.6 vs 2.0.0

Same box, same data, same scripts. 1.5.6 ran on October 19, 2.0.0 on October 22. Median wall seconds at 150M:

Setting	Query	1.5.6	2.0.0	Change
`memory_limit = 3GB`	Window dedup	9.09	7.92	-13%
`memory_limit = 3GB`	Group by	9.61	7.91	-18%
`memory_limit = 1GB`	Window dedup	17.79	14.97	-16%
`memory_limit = 1GB`	Group by	22.06	13.94	-37%
1GB, no insertion order	Window dedup	14.85	14.94	+1%
1GB, no insertion order	Group by	15.15	13.88	-8%

The useful finding is the last four rows together. On 1.5.6, SET preserve_insertion_order = false was worth 16% on the dedup and 31% on the group by under a 1 GB limit. On 2.0.0 the default already lands where the tuned setting did, and the flag changes almost nothing for these two queries. If you carry that setting in job configs as a performance trick, re-measure after upgrading. It still matters for what it was designed for: the docs recommend it when large imports or exports run out of memory, and it lets DuckDB reorder rows without an ORDER BY.

Not everything improved. The 50M window dedup was slower on 2.0.0 (1.94 s to 2.33 s), and peak RSS for the light queries went up (150M filter: 0.26 to 0.77 GiB, join: 0.23 to 0.42 GiB), while the heavy queries went down (window dedup: 3.47 to 3.01 GiB). I have not investigated why. The two runs were also three days apart on a shared box, so treat differences under about 10 to 15% as noise. The 1 GB group by drop is far outside that.

The release itself is mostly a feature release: Quack and CONNECT for client-server use, asynchronous I/O, a first-class VARIANT, triggers, a PEG parser by default (announcement). It is an LTS line supported until at least October 2027. One migration note matters for pipelines: database files written by 2.0 cannot be read by versions before 2.0. My benchmark only reads and writes Parquet, so it never touched that.

Where Spark still wins

None of this says "replace Spark". It says Spark on one box is the wrong comparison for small and medium jobs. Spark still wins when:

The data does not fit one node's disk and memory, or the deadline does not fit one node's cores. This test stops at 3.2 GB of Parquet on purpose. DuckDB spills well, but one machine is still one machine.
The platform is built around it. On Databricks and Fabric, Spark is where Unity Catalog or OneLake governance, Delta writes with concurrent writers, lineage, job orchestration and streaming already live. A DuckDB step outside that boundary is a step you have to secure, monitor and explain.
The cluster is already running. The 5 second JVM start I measured per run disappears in a warm interactive cluster or a long job. The query-only gap (4x to 8x here) does not.
The team is a Spark team. One engine everyone can debug at 02:00 is worth more than a faster engine one person understands.

My opinion as a Lead DE on Azure, Databricks and Fabric: the line is not the engine, it is the size of the job. Most jobs I see are tens of millions of rows dressed up as big data and started on a cluster. For those (file-to-file transforms before landing, validation and reconciliation checks, small dimension builds, CI tests for SQL logic) a single-node engine with a memory limit is the cheaper and more predictable choice. Anything that writes governed Delta tables stays on the platform.

Failure modes I hit, so you don't have to

Engine memory settings are not process memory. Spark's 3g driver heap still produced a 7.51 GiB process. DuckDB's whole process stayed close to its memory_limit: about 3.0 GiB peak at a 3 GB limit, 1.1 GiB at 1 GB. Size containers on measured RSS, not on the setting.
"Streaming" does not mean bounded. Polars streamed the filter, join and group by fine. The window dedup with an over(order_by=...) did not stay bounded, even with 9 GiB.
Close to the cap is a failure that hasn't happened yet. Polars' group by finished in 1 of 3 runs, peaking at 4.49 GiB against a 4.5 GiB cap. Same query, same data, same version. Schedule that and it fails some nights.
Shared hosts lie. One Spark run at 10M was killed because free memory on the whole box fell under my 1 GiB safety floor while Spark itself was at 1.6 GiB. That was the neighbours, not Spark. Gate each run on free memory and CPU, and record both.
Validate outputs, not just timings. Every engine produced the same row counts and checksums where it finished. Without that check, a fast wrong answer looks like a win.

Reproduce it

Everything is public at firfircelik/engine-bench: scripts, raw results, pinned versions and the query plans. Here are the parts that matter. The data generator is seeded (SEED = 20261019) and writes 5M-row chunks:

rng = np.random.default_rng([SEED, chunk_index])
t = pa.table({
    "order_id": np.arange(lo, lo + n, dtype=np.int64),
    "ts": pa.array(start + rng.integers(0, 270 * 24 * 3600, n), pa.timestamp("s")),
    "customer_id": rng.integers(0, rows // 10, n, dtype=np.int32),
    "product_id": rng.integers(0, 50_000, n, dtype=np.int32),
    "store_id": rng.integers(0, 500, n, dtype=np.int16),
    "qty": rng.integers(1, 10, n, dtype=np.int16),
    "price": np.round(rng.uniform(1.0, 500.0, n), 2),
    "status": rng.choice(STATUSES, n, p=[0.55, 0.20, 0.12, 0.08, 0.05]),
})
pq.write_table(t, f"sales_{rows // 1_000_000}m/part-{chunk_index:04d}.parquet",
               row_group_size=1_000_000, compression="snappy")

Spark session, the only non-default settings:

spark = (SparkSession.builder.master("local[8]")
         .config("spark.driver.memory", "3g")
         .config("spark.local.dir", TMP)
         .config("spark.ui.enabled", "false")
         .config("spark.sql.session.timeZone", "UTC")
         .getOrCreate())

And the watchdog, which is the part I would copy into any benchmark of your own. It measures the whole process tree, because JVMs and thread pools do not show up in the parent's RSS:

import psutil, subprocess, time

GIB = 1024 ** 3

def tree_rss(p):
    total = 0
    for q in [p] + p.children(recursive=True):
        try:
            total += q.memory_info().rss
        except psutil.NoSuchProcess:
            pass
    return total

proc = subprocess.Popen(cmd)
p, peak = psutil.Process(proc.pid), 0
while proc.poll() is None:
    rss = tree_rss(p)
    peak = max(peak, rss)
    if rss > 4.5 * GIB:
        for q in p.children(recursive=True) + [p]:
            q.kill()
        break
    time.sleep(0.05)

The takeaway

On one 8 vCPU machine with a 4.5 GiB cap, DuckDB 2.0.0 was the only engine that finished every run of every query at every size, and it did the heavy ones with 1 GB. Polars matched or beat it on the light queries and the group by, and could not finish the window dedup at 150M even with 9 GiB. Local Spark finished almost everything, 5x to 12x slower, and its memory setting did not bound its memory.

If your job fits on one machine, measure it on one machine before you start a cluster for it.

Sources

Announcing DuckDB 2.0.0 LTS "Cinnamon Teal", DuckDB team, October 21, 2026
DuckDB release calendar (2.0.0 LTS on 2026-10-21, 1.5.6 on 2026-09-28)
duckdb 2.0.0 on PyPI, polars 2.0.0 on PyPI, pyspark 4.2.0 on PyPI
DuckDB: How to Tune Workloads (larger-than-memory operators, preserve_insertion_order)
DuckDB: Order Preservation
All timings and memory numbers: my own runs on October 19 (DuckDB 1.5.6, and the 9 GiB Polars and Spark runs) and October 22, 2026 (DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0).

Building a Zero-Telemetry Production Log Triage Engine with Python, DuckDB, and Gemini

Thu, 08 Oct 2026 18:06:58 +0000

During a Sev-1 incident, every minute spent waiting for an enterprise observability platform's search indexing—or watching a junior SRE run unindexed grep and awk commands across 20GB raw .jsonl dumps—directly translates to business downtime.

Most commercial log analytics vendors charge astronomical data ingestion fees, only to provide rigid dashboards, slow text queries, and generic LLM summaries that hallucinate stack trace causality.

In this guide, we'll build a Production Log Triage & Root-Cause Engine that runs entirely locally, processes millions of raw log entries in seconds using DuckDB, clusters redundant stack traces deterministically into semantic fault signatures, and leverages Google Gemini 1.5 Flash / Pro to pinpoint root causes and generate zero-telemetry incident post-mortems.

1. The Bottleneck: Why Centralized Log Indexing Fails Incident Triage

Centralized observability stacks (Datadog, Splunk, Elastic) are designed for long-term retention and metrics aggregation, not real-time ad-hoc crash triage. When services experience cascading failure:

Ingestion Throttling & Lag: Log volume spikes 100x during a cascade. Centralized collectors throttle or lag by 5 to 15 minutes right when you need sub-second visibility.
The High-Cardinality Trap: Indexing UUIDs, dynamic user IDs, and raw exception messages creates massive index bloat. Querying raw text strings across unindexed columns over multi-gigabyte partitions leads to timeouts.
Context Window Exhaustion: Dumping 100,000 raw lines into an LLM context window burns tokens instantly, triggers rate limits, and results in context degradation where the model misses the primary causal trigger buried in repetitive retry loops.

To solve this, we decouple triage from long-term storage using a local, three-stage pipeline:

┌─────────────────┐       ┌────────────────────────┐       ┌─────────────────────────┐       ┌────────────────────┐
│  Raw Log Dumps  │ ----> │  DuckDB SQL Engine     │ ----> │  Semantic Fingerprint   │ ----> │  Gemini 1.5 Triage │
│ (.jsonl/syslog) │       │ (Multi-GB in-memory)   │       │ & Error Clustering      │       │ (Root Cause & PR)  │
└─────────────────┘       └────────────────────────┘       └─────────────────────────┘       └────────────────────┘

2. The Architecture

The engine operates entirely within an ephemeral execution environment or CLI runner:

In-Process Analytical Querying (DuckDB): Queries raw compressed .jsonl.gz, .csv, or Syslog files directly on disk using vectorized execution without spinning up an external database daemon.
Deterministic Masking & Fingerprinting: Strips transient parameters (UUIDs, memory addresses, timestamps, numeric IDs, IP addresses) from stack traces to generate a deterministic SHA-256 cluster key.
Volumetric Anomaly Scoring: Ranks fault signatures by frequency, blast radius (unique services impacted), and temporal proximity to the incident start.
Structured Gemini Root-Cause Extraction: Feeds the top unique signatures, complete with clean contextual stack traces and surrounding debug lines, into Gemini 1.5 Pro/Flash to produce actionable root causes and git diff patches.

3. The Code & Implementation

Step 1: High-Speed Log Filtering with DuckDB

DuckDB allows us to query gigabytes of JSONL logs directly from disk using standard SQL, outperforming raw Python parsing by orders of magnitude.

import duckdb

def extract_fault_candidates(log_file_path: str, min_level: str = "ERROR"):
    """
    Directly reads JSONL from disk, filters error-tier rows,
    and extracts stack traces, timestamps, and service tags.
    """
    con = duckdb.connect(database=":memory:")
    
    # Configure DuckDB for high-throughput disk scan
    con.execute("SET threads TO 4;")
    con.execute("SET preserve_insertion_order = false;")
    
    query = f"""
        SELECT 
            timestamp,
            service_name,
            level,
            message,
            COALESCE(stack_trace, '') as stack_trace
        FROM read_json_auto('{log_file_path}')
        WHERE level IN ('{min_level}', 'CRITICAL', 'FATAL')
        ORDER BY timestamp ASC
    """
    
    df = con.execute(query).df()
    con.close()
    return df

Step 2: Semantic Log Normalization & Fingerprinting

To compress 100,000 errors into single-digit unique signatures, we mask high-cardinality noise using deterministic regex rules before generating a cluster hash.

import re
import hashlib
import pandas as pd

class LogNormalizer:
    def __init__(self):
        self.patterns = [
            (re.compile(r'[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}'), '<UUID>'),
            (re.compile(r'0x[0-9a-fA-F]+'), '<HEX_ADDR>'),
            (re.compile(r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b'), '<IP>'),
            (re.compile(r'\b\d+\b'), '<INT>'),
            (re.compile(r'(https?://\S+)'), '<URL>'),
        ]

    def mask(self, text: str) -> str:
        for pattern, replacement in self.patterns:
            text = pattern.sub(replacement, text)
        return text

def cluster_errors(df: pd.DataFrame) -> pd.DataFrame:
    normalizer = LogNormalizer()
    
    # Normalize messages and stack traces
    df['normalized_signature'] = df.apply(
        lambda row: normalizer.mask(f"{row['service_name']}::{row['message']}::{row['stack_trace'][:200]}"),
        axis=1
    )
    
    # Generate deterministic hash
    df['signature_hash'] = df['normalized_signature'].apply(
        lambda sig: hashlib.sha256(sig.encode('utf-8')).hexdigest()[:12]
    )
    
    # Aggregate clusters
    clusters = df.groupby('signature_hash').agg(
        occurrence_count=('timestamp', 'count'),
        first_seen=('timestamp', 'min'),
        last_seen=('timestamp', 'max'),
        sample_service=('service_name', 'first'),
        sample_message=('message', 'first'),
        sample_stack_trace=('stack_trace', 'first')
    ).reset_index().sort_values(by='occurrence_count', ascending=False)
    
    return clusters

Step 3: Targeted Root-Cause Analysis with Gemini 1.5

Now that thousands of noisy lines are clustered into top signatures, we pass the critical clusters to Gemini 1.5 using structured outputs to avoid generic summaries.

import os
from google import genai
from google.genai import types
from pydantic import BaseModel, Field

class RootCauseAnalysis(BaseModel):
    fault_signature_id: str
    root_cause_summary: str = Field(description="Concise explanation of the underlying software or infrastructure bug.")
    failure_domain: str = Field(description="e.g., Database, Network, Concurrency, Logic, Memory")
    severity_impact: str = Field(description="Low, Medium, High, Critical")
    remediation_patch: str = Field(description="Code snippet, configuration change, or SQL mitigation")

def analyze_signature_with_gemini(cluster_row: dict) -> RootCauseAnalysis:
    client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
    
    prompt = f"""
    Analyze this production log fault cluster to isolate the absolute root cause.
    
    OCCURRENCE COUNT: {cluster_row['occurrence_count']}
    SERVICE: {cluster_row['sample_service']}
    FIRST SEEN: {cluster_row['first_seen']}
    LAST SEEN: {cluster_row['last_seen']}
    ERROR MESSAGE: {cluster_row['sample_message']}
    
    RAW STACK TRACE:
    {cluster_row['sample_stack_trace']}
    
    Provide precise diagnostic details and a code-level remediation fix.
    """
    
    response = client.models.generate_content(
        model='gemini-1.5-flash',
        contents=prompt,
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            response_schema=RootCauseAnalysis,
            temperature=0.1
        )
    )
    
    return RootCauseAnalysis.model_validate_json(response.text)

4. Deployment, Performance & Zero-Telemetry Execution

Handling Large Log Dumps (>50GB)

By default, DuckDB's streaming parser streams rows out-of-core. If you are operating under tight memory constraints (e.g., a triage agent running in a 2GB RAM container):

PRAGMA max_memory = '1.5GB';
PRAGMA temp_directory = '/tmp/duckdb_spill';

DuckDB will transparently spill intermediate aggregations to disk, preventing Out-Of-Memory (OOM) crashes even when sorting massive volumes.

Rate Limiting & Token Optimization

Passing full stack traces can quickly eat into context windows if uncapped. We apply three safeguards:

Truncation Threshold: Limit raw stack traces to the bottom 15 frames where the crash occurred.
Concurrency Limiting: Wrap Gemini API calls in an asyncio.Semaphore(5) to prevent 429 quota exhaustion.
Model Tiering: Use gemini-1.5-flash for high-volume clusters (>95% of cases), switching to gemini-1.5-pro only for multi-service cascade anomalies with complex distributed traces.

Exporting Structured Post-Mortems

After analysis, the engine compiles a structured Markdown post-mortem document directly into the incident repository:

def export_markdown_report(analyses: list[RootCauseAnalysis], output_path: str = "incident_report.md"):
    with open(output_path, "w") as f:
        f.write("# Automated Incident Triage & Root Cause Report\n\n")
        for item in analyses:
            f.write(f"## Fault Signature: `{item.fault_signature_id}`\n")
            f.write(f"- **Domain**: {item.failure_domain}\n")
            f.write(f"- **Severity**: {item.severity_impact}\n\n")
            f.write(f"### Root Cause\n{item.root_cause_summary}\n\n")
            f.write(f"### Suggested Remediation\n```\n{item.remediation_patch}\n```\n\n---\n")

5. Conclusion & Ready-to-Use Package

By combining DuckDB's in-process analytical speed with deterministic regex fingerprinting, you compress 100,000+ chaotic log entries into an actionable set of signatures, reducing LLM token consumption by over 98% while eliminating ingestion costs.

You can implement this architecture using the code snippets above to build an in-house triage CLI for your engineering team.

If you prefer a pre-built, production-tested CLI tool with automated format detection (Syslog, Docker JSON, Morgan, Winston), multi-threaded clustering, and enterprise Markdown/JSON post-mortem exporters, you can get the turnkey package here:

Instant Access on Whop: Production Log Triage Engine
Direct Download on Gumroad: Gumroad Download (Use promo code EARLYBIRD for 20% off)

Building an Autonomous CI/CD Log Triage & Self-Healing PR Engine with Python and DuckDB

Wed, 07 Oct 2026 17:14:08 +0000

Building an Autonomous CI/CD Log Triage & Self-Healing PR Engine with Python and DuckDB

Every growing engineering team hits the same wall: CI pipelines fail, developers ignore cryptic 200MB console outputs, and senior engineers spend 15+ hours every week parsing raw text to fix the same 4 recurring issues—broken lockfiles, misconfigured test fixtures, missing environment variables, or dependency drift.

Observability vendors sell multi-thousand-dollar enterprise suites to ingest these logs, yet they stop at alerting. You still have to open the console, copy the stack trace, diagnose the problem, create a branch, write the fix, and submit a PR.

In this guide, we'll build a zero-infrastructure, self-healing pipeline that runs natively inside your CI environment (GitHub Actions, GitLab CI, or n8n runner). It parses multi-gigabyte build artifacts using DuckDB, clusters stack traces deterministically without bloated vector stores, prompts an LLM with compact context, and automatically submits self-healing Pull Requests via the GitHub REST API.

1. The Bottleneck: Why Traditional Log Triage Fails

Most CI log monitoring tools fail for three architectural reasons:

Data Transfer Tax & Latency: Pushing gigabytes of raw container logs over the wire to external aggregators creates latency and ballooning bandwidth bills.
Vector DB Overhead: Vector embeddings for error logs are overkill. Logs are semi-structured text where 95% of lines are identical runtime noise. Embedding entire raw logs produces polluted semantic spaces that fail to isolate exact trace boundaries.
Lack of Actionability: Alerting on Slack is not a resolution. If a failure stems from a deprecated dependency version or a missing mock key, the remediation step is deterministic.

By executing triage locally on the CI runner, we eliminate network egress, process logs in memory via columnar storage, and resolve routine failures without human intervention.

2. System Architecture

The engine follows a strict four-stage pipeline running directly on your CI execution container:

[ Raw CI Log Files (*.log, *.json) ]
              │
              ▼
┌───────────────────────────────┐
│ DuckDB High-Throughput Ingest │ ──► Columnar filtering of timestamps,
│ & Regex Pattern Clustering    │     log levels, and stack frames
└──────────────┬────────────────┘
              │
              ▼
┌───────────────────────────────┐
│ Deterministic Triage Engine   │ ──► Isolates exact stack trace and
│ (Root-Cause Classifier)       │     queries target source file
└──────────────┬────────────────┘
              │
              ▼
┌───────────────────────────────┐
│ Structured LLM Patch Synthesizer │ ──► Generates unified diff
└──────────────┬────────────────┘
              │
              ▼
[ GitHub REST API: Creates Git Branch + Opens Self-Healing PR ]

DuckDB: Reads gigabyte-scale logs directly from disk or gzipped archives using zero-copy SQL queries.
Pattern Clustering: Groups repeated log lines using tokenized signature matching.
Autonomous Git Engine: Validates the proposed diff locally, branches out, commits the patch, and triggers a PR.

3. The Code & Implementation

Step 1: Sub-Second Log Parsing with DuckDB

Instead of loading logs line-by-line in pure Python, we leverage DuckDB's native regex parsing engine to query the raw log file as a relational table.

import duckdb
import os

def extract_critical_failures(log_path: str) -> list[dict]:
    """
    Parses uncompressed or gzipped logs and extracts error frames
    without loading the entire file into Python memory.
    """
    con = duckdb.connect(database=":memory:")
    
    query = """
    WITH raw_lines AS (
        SELECT 
            row_number() OVER () AS line_num,
            column0 AS content
        FROM read_csv(?, header=False, sep='\n', auto_detect=False)
    ),
    filtered_errors AS (
        SELECT 
            line_num, 
            content,
            regexp_extract(content, '(?i)(error|fatal|exception|failed):?\s*(.*)', 2) AS error_msg
        FROM raw_lines
        WHERE regexp_matches(content, '(?i)(ERROR|FATAL|Exception|FAILED)')
    )
    SELECT 
        line_num, 
        content, 
        error_msg
    FROM filtered_errors
    LIMIT 25;
    """
    
    results = con.execute(query, [log_path]).fetchall()
    con.close()
    
    return [{"line": row[0], "raw": row[1], "error": row[2]} for row in results]

Step 2: Extracting Context Around the Root Failure

Once DuckDB pinpoints the fatal line number, we extract the local stack trace window and the affected source code file to pass to our LLM generator.

import re
from pathlib import Path

def isolate_trace_context(log_path: str, error_line: int, window: int = 15) -> str:
    """Extracts preceding and succeeding context around an error line."""
    start = max(1, error_line - window)
    end = error_line + window
    
    con = duckdb.connect(database=":memory:")
    query = """
    SELECT list(column0)
    FROM (
        SELECT row_number() OVER () AS rn, column0
        FROM read_csv(?, header=False, sep='\n', auto_detect=False)
    )
    WHERE rn BETWEEN ? AND ?;
    """
    lines = con.execute(query, [log_path, start, end]).fetchone()[0]
    con.close()
    return "\n".join(lines) if lines else ""

def locate_offending_file(trace_text: str) -> tuple[str, str]:
    """Locates local source code file and extracts code content."""
    file_match = re.search(r'File "([a-zA-Z0-9_\-\./]+\.py)", line (\d+)', trace_text)
    if not file_match:
        return "", ""
    
    file_path, line_no = file_match.group(1), int(file_match.group(2))
    if Path(file_path).exists():
        return file_path, Path(file_path).read_text()
    return "", ""

Step 3: LLM Patch Synthesizer & GitHub PR Automation

We feed the trace and target source code into a structured LLM prompt that enforces raw unified diff output, then push the patch directly through the GitHub API.

import json
import urllib.request
import base64

def generate_patch_with_llm(trace: str, source_code: str, file_path: str, api_key: str) -> str:
    """Generates a strict unified diff to fix the identified CI breakage."""
    system_prompt = (
        "You are an automated CI remediation agent. Analyze the provided test failure "
        "and source code. Output ONLY a valid unified diff (git patch) that fixes the failure. "
        "Do not output markdown code blocks, explanations, or commentary."
    )
    user_prompt = f"TARGET FILE: {file_path}\n\nSTACK TRACE:\n{trace}\n\nSOURCE CODE:\n{source_code}"
    
    payload = json.dumps({
        "model": "gpt-4o-mini",
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt}
        ],
        "temperature": 0.1
    }).encode("utf-8")
    
    req = urllib.request.Request(
        "https://api.openai.com/v1/chat/completions",
        data=payload,
        headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
    )
    
    with urllib.request.urlopen(req) as resp:
        data = json.loads(resp.read().decode("utf-8"))
        return data["choices"][0]["message"]["content"].strip()

def create_self_healing_pr(repo: str, branch: str, file_path: str, updated_content: str, token: str):
    """Applies remediation by opening a Git branch and PR via GitHub REST API."""
    headers = {
        "Authorization": f"Bearer {token}",
        "Accept": "application/vnd.github.v3+json"
    }
    base_url = f"https://api.github.com/repos/{repo}"
    
    # 1. Fetch default branch SHA
    req = urllib.request.Request(f"{base_url}/git/ref/heads/main", headers=headers)
    with urllib.request.urlopen(req) as resp:
        sha = json.loads(resp.read().decode())["object"]["sha"]
        
    # 2. Create target branch
    branch_payload = json.dumps({"ref": f"refs/heads/{branch}", "sha": sha}).encode()
    req = urllib.request.Request(f"{base_url}/git/refs", data=branch_payload, headers=headers)
    urllib.request.urlopen(req)
    
    # 3. Fetch file SHA
    req = urllib.request.Request(f"{base_url}/contents/{file_path}?ref={branch}", headers=headers)
    with urllib.request.urlopen(req) as resp:
        file_sha = json.loads(resp.read().decode())["sha"]
        
    # 4. Commit updated file
    encoded_content = base64.b64encode(updated_content.encode()).decode()
    commit_payload = json.dumps({
        "message": "fix(ci): autonomous self-healing patch for test failure",
        "content": encoded_content,
        "sha": file_sha,
        "branch": branch
    }).encode()
    req = urllib.request.Request(f"{base_url}/contents/{file_path}", data=commit_payload, headers=headers, method="PUT")
    urllib.request.urlopen(req)
    
    # 5. Open Pull Request
    pr_payload = json.dumps({
        "title": "🤖 [Self-Healing] Fix CI Failure Regression",
        "head": branch,
        "base": "main",
        "body": "This Pull Request was automatically generated by the Autonomous CI Log Triage Engine after detecting deterministic failures in the pipeline."
    }).encode()
    req = urllib.request.Request(f"{base_url}/pulls", data=pr_payload, headers=headers)
    urllib.request.urlopen(req)

4. Deployment, Timeouts, and Rate Limits

To run this in production without adding latency to green runs:

Conditional Triggers: Only execute the triage step on if: failure() in GitHub Actions or when: on_failure in GitLab CI.
Run-Time Budgets: Restrict the DuckDB scan window to the last 5,000 lines. DuckDB handles this in under 200ms.
LLM Cost Guardrails: Strip ANSI escape codes (\x1b\[[0-9;]*m) before sending payloads to LLMs to prevent token inflation. Truncate context to include only the stack trace boundary and the active function block.
Prevent PR Flooding: Cache the hash of the error signature (MD5(trace_text)) in your build cache. If an identical PR is already pending, skip recreation.

5. Conclusion & Ready-to-Use Workflow

You can implement this architecture using the code snippets above or integrate it into an automated n8n webhook listener. By replacing third-party SaaS log aggregators with embedded DuckDB queries, you gain deterministic triage while keeping logs completely inside your security perimeter.

If you prefer a plug-and-play solution, you can get the complete turnkey package containing production GitHub Action workflows, Docker container configurations, synthetic log test fixtures, and pre-built n8n triage workflows:

Instant Access on Whop
Direct Download on Gumroad (Use promo code EARLYBIRD for 20% off)

I added DuckDB to my SQL playground, and it turned out to be a sandboxing problem

Thu, 20 Aug 2026 12:38:29 +0000

I added DuckDB to my SQL playground, and it turned out to be a sandboxing problem

Sharing a small build-in-public update: sqlize.online, a free tool for testing SQL queries, checking JOINs, and sharing reproducible examples, just got DuckDB support alongside the ~30 engines it already had (MySQL, Postgres, SQL Server, Oracle, Firebird, ClickHouse, SQLite...).

I'd wanted to add DuckDB for a while — I use it a lot in my own projects and like having local OLAP around. The holdup was boring: the whole thing runs on PHP, and PHP's PDO talks to MySQL, Postgres, and SQLite out of the box, but had nothing for DuckDB.

That changed when I found pdo_duckdb by Ilia Alshanetsky — a real PDO driver for it. That's the piece that made this worth doing.

Why DuckDB specifically

Short version: it's "SQLite for analytics." Vectorized engine, built for heavy aggregations on a single core, not for transactions.

What actually got me was that it reads Parquet on the fly, from disk or over HTTP, with no separate load step. For a playground, that's great — instead of a two-row toy table, you can hand people something real. So I preloaded NYC's Yellow Taxi trip data for January 2024, about three million rows, sitting in a table called yellow_tripdata:

SELECT
    PULocationID,
    COUNT(*) AS total_trips,
    ROUND(AVG(total_amount), 2) AS avg_fare
FROM yellow_tripdata
GROUP BY PULocationID
ORDER BY total_trips DESC
LIMIT 10;

That's a table, not a file path — more on why that matters below.

Building the driver, plus an unrelated Debian surprise

pdo_duckdb needs PHP 8.1+. My setup runs several PHP versions side by side for different engines, so that already cuts out a couple of pools. Worse, the latest release only ships prebuilt binaries for PHP 8.4/8.5 — on 8.1, where this feature currently lives, I had to build from source: phpize && ./configure --with-pdo-duckdb=... && make install, linked against libduckdb.so. Not exotic, but not a one-command install either — about half the engines in this project get built by hand at this point.

While I was at it I hit a completely unrelated bug: bumped that PHP pool to the latest patch release, and the image build broke for no obvious reason. Turned out the base image had quietly moved from Debian bookworm to trixie, which renamed libaio1 to libaio1t64. Rolled the base image back. Cost me about forty minutes of confusion for something that had nothing to do with DuckDB at all.

The actual problem: DuckDB runs inside the PHP process

MySQL and Postgres live in their own containers and can't see the PHP worker's filesystem at all. DuckDB is embedded — it opens up right inside PHP-FPM. Good for speed, no sockets or IPC in the way. Bad because by default it can reach anything the process can reach: COPY ... TO '/etc/whatever', read_csv from any path, read_parquet from any URL, INSTALL for arbitrary extensions.

Not something you want available to arbitrary user SQL.

First attempt was the obvious one — set it in SQL directly:

SET enable_external_access = false;
SET allow_community_extensions = false;
SET lock_configuration = true;

This genuinely works, and lock_configuration = true really does stop the same session from undoing it later. But the driver has its own separate protection: it flatly refuses a chunk of security-sensitive settings — paths, extension autoload — when they're passed via the DSN or PDO::DUCKDB_ATTR_CONFIG at connect time. On purpose, so the app using the driver can't accidentally weaken its own config.

The real mechanism turned out to be PHP's own open_basedir. Set it — even at runtime, with ini_set() — and the driver shuts off DuckDB's entire external-access surface: read_csv, read_parquet, COPY, ATTACH, httpfs, INSTALL, all of it, no matter which path gets requested. Regular CREATE TABLE/INSERT/SELECT against the session's own file keep working fine.

ini_set('open_basedir', '/code:/tmp/sessions:/tmp/databases');
$pdo = new PDO("duckdb:{$sessionFile}");

I tried stacking both — open_basedir plus the SET statements on top, just for extra safety. Didn't work: once open_basedir is active, the driver rejects allow_community_extensions outright ("not allowed when open_basedir is set") and the connection breaks. Had to pick one. Kept open_basedir — it covers everything on its own.

And yes, there was a bug. I was applying open_basedir right before running a query, but not before handing a raw PDO connection to user-written PHP code — which happens when someone runs PHP and SQL together and picks DuckDB as the engine. So: submit a PHP script with no SQL in the separate field, get back a completely unrestricted $pdo, and $pdo->exec("COPY (SELECT 1) TO '/tmp/whatever.csv'") just works. Caught it myself before it shipped anywhere public.

Why the dataset isn't loaded on every request

Three million rows over HTTPS on every session would be brutal — a cold build takes anywhere from 5 to 35 seconds. So it's built once at deploy time, by a separate script using a plain connection with no open_basedir set (downloading the file there is a legitimate job for that script, not for user SQL), and cached to a shared file. Every session after that is just a copy() — a fraction of a second.

Try to break it

If you find a way to read a server file or get around open_basedir, I'd rather hear about it directly than see the service go down. No bug bounty budget, but I'll credit you publicly if you find something real.

Try it here: sqlize.online — pick "DuckDB" or "DuckDB Preloaded" as the SQL version. Happy to answer questions about how any of this is built.

pdo_duckdb: DuckDB for PHP, Behind the PDO API You Already Know

Fri, 19 Jun 2026 18:52:42 +0000

DuckDB is the closest thing the analytics world has to SQLite. It runs in-process, needs no server, reads and writes a single file, and chews through columnar aggregate queries that would make a row-store sweat. PHP has shipped PDO_SQLite in core for twenty years. Until now it had no equivalent for DuckDB.

So I wrote one. pdo_duckdb is a native PDO driver. You open a DuckDB database with a DSN, prepare statements, bind parameters, and iterate results through the same PDO API you already use for SQLite, MySQL, and PostgreSQL.

$db = new PDO('duckdb:/path/to/analytics.duckdb');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

$stmt = $db->prepare('SELECT region, SUM(amount) AS total FROM sales WHERE year = ? GROUP BY region');
$stmt->execute([2026]);
foreach ($stmt as $row) {
    printf("%s: %s\n", $row['region'], $row['total']);
}

This is also the first PDO driver I've written in over fifteen years. I was one of the original authors of PDO back in the PHP 5.1 days, so coming back to that extension surface was equal parts muscle memory and culture shock.

Why a PDO Driver and Not Another FFI Binding

You can already reach DuckDB from PHP today. It has a C API you can drive through FFI, and there's always shelling out to the CLI. Both work. Both also hand you a brand-new API surface, manual lifetime management, and a data layer that looks nothing like the rest of your application.

PDO is the abstraction PHP developers already have in their fingers. A PDO driver means prepared statements, parameter binding, the exception error mode, fetch modes, and transactions all behave the way they do for every other PDO backend. Your repository classes and query builders transfer unchanged. The driver is written in C against DuckDB's C API, so there is no FFI boundary at runtime and no per-call marshalling happening up in PHP.

To be fair to FFI, it has its place. If you need a DuckDB capability the PDO surface doesn't expose, FFI hands you the entire C API and gets out of the way. pdo_duckdb makes the opposite trade on purpose: less breadth, in exchange for fitting into code you've already written.

DuckDB in One Paragraph, If You Haven't Met It

DuckDB is an in-process analytical (OLAP) database. The relationship to ClickHouse or Snowflake is roughly what SQLite is to Postgres: same SQL you know, no server to run, embedded directly in your process. It is columnar and vectorized, built for scans and aggregations over a lot of rows rather than single-row OLTP traffic. The DSN tells the driver which database to open:

duckdb:/path/to/database.duckdb   # file-backed database
duckdb::memory:                   # in-memory database
duckdb:                           # in-memory database (empty path)

An in-memory database is scratch space that vanishes when the connection closes; a file-backed one persists to disk. DuckDB's own extensions load through ordinary SQL, with no special driver API to learn:

$db->exec('LOAD json');                     // bundled extensions load offline
$db->exec('INSTALL httpfs; LOAD httpfs;');  // downloadable extensions

Reading Parquet, querying a CSV in place, or pulling a file over HTTP is a LOAD away, all through the same $db->exec() you would use for any statement. The driver doesn't wrap or re-expose any of it. DuckDB's SQL surface is the API.

Bulk Loading: The Appender

Row-by-row INSERT is the wrong way to push a lot of data into a columnar store, and DuckDB knows it. The driver exposes DuckDB's native appender through PDO::duckdbAppender(), which streams rows into a table far faster than a loop of prepared INSERT statements.

$db->exec('CREATE TABLE events (id INTEGER, name VARCHAR, ts TIMESTAMP)');

$app = $db->duckdbAppender('events');      // optional 2nd arg: schema name
foreach ($rows as $r) {
    $app->appendRow($r['id'], $r['name'], $r['ts']);
}
$app->flush();                              // or $app->close() to finalize

appendRow(...$values) takes one argument per column, left to right, and returns the appender so calls can chain. PHP null, bool, int, float, and string map to DuckDB values and get cast to the target column types. This is the one place the driver steps outside the standard PDO surface, and it earns the exception. Bulk ingestion is exactly where the row-at-a-time PDO model leaves the most performance on the table, so it's the one spot worth a driver-specific method.

The Parts of PDO That Don't Map Cleanly

This is the honest section. DuckDB is not MySQL, and a faithful driver has to refuse a few things rather than fake them.

lastInsertId() is not supported. DuckDB has no implicit rowid, so there is nothing honest to return. If you need generated keys, use a sequence and currval().

setAttribute(PDO::ATTR_AUTOCOMMIT, false) is rejected. DuckDB is autocommit-by-default with no session-level toggle, so instead of pretending otherwise, the driver points you at beginTransaction() for explicit transactions. From there commit() and rollBack() map onto DuckDB's COMMIT and ROLLBACK.

Placeholders get rewritten. Positional ? and named :name parameters are both supported, and PDO rewrites them to DuckDB's native $N form. That creates one sharp edge: because : introduces a named placeholder, an inline STRUCT or MAP literal inside a prepared query needs a space after the colon. Write {'k': 1}, not {'k':1}, or the parser reads :1} as a parameter and gets confused.

Types come back predictably, with one rule worth internalizing. Integers up to 64-bit signed arrive as PHP int, FLOAT and DOUBLE as float, and BLOB as a binary string. Everything else, VARCHAR, the DATE and TIME and TIMESTAMP family, DECIMAL, the wide HUGEINT and UBIGINT integers, and DuckDB's nested types, comes back in its canonical string form. That keeps values lossless when they wouldn't fit a native PHP scalar, at the cost of a cast on your side when you want a number out of a HUGEINT column.

Results are materialized, for now. execute() buffers the full result set in memory before PDO begins fetching. For the analytical queries DuckDB is built for that is usually fine, but a SELECT that returns millions of rows is bounded by available memory rather than streamed row by row. True streaming through DuckDB's pending-result API is the next thing on the list.

One behavior goes the other way, in your favor. When open_basedir is set, the driver disables DuckDB's SQL-level external file access: read_csv, COPY, ATTACH, httpfs, and the rest. Without that, a single COPY ... TO '/somewhere/outside' would punch straight through the sandbox open_basedir is supposed to enforce. The restriction holds at the SQL layer, not just for the path of the database file itself.

What Writing a PDO Driver Looks Like in 2026

The PDO driver model itself has barely moved since I last touched it. You fill in a handful of method tables, hang your connection state off the driver object, and PDO drives the rest. What changed is everything around it.

On PHP 8.4 and up, PDO::connect('duckdb:...') returns a Pdo\Duckdb subclass instance, and duckdbAppender() lives on that subclass. With new PDO('duckdb:...'), and on PHP 8.1 through 8.3, the method is reachable on the base object directly. PHP 8.5 starts emitting a deprecation when you call a driver-specific method on the base PDO class, so on 8.4 and newer you'll want PDO::connect(). The driver supports both paths; the subclass is just the cleaner one going forward.

The part that actually cost me a release was the link line. DuckDB ships as a C++ library with a C API bolted on top. The driver is C. The prebuilt Linux binary in 0.2.0 loaded fine on my machine and then died on a clean host with undefined symbol: _ZTVN10__cxxabiv120__function_type_infoE, the vtable for a C++ ABI type that was supposed to be baked into the binary. The bundled DuckDB C++ runtime was never getting statically linked, because the gcc C driver silently ignores -static-libstdc++, a flag it treats as g++-only. So the module quietly depended on a libstdc++ the target box didn't ship. 0.2.1 links the static libstdc++ and libgcc_eh in explicitly, and the Linux binary is genuinely self-contained now. macOS and Windows were never affected, which is its own special way to lose an afternoon chasing the wrong platform.

Distribution is the other thing that improved while I was away. On Linux x86_64 and arm64, macOS on Apple Silicon, and Windows x64, PIE downloads a self-contained prebuilt binary with no DuckDB install and no build toolchain required:

pie install iliaal/pdo_duckdb

On anything else it falls back to a source build that needs libduckdb and duckdb.h. The last time I shipped a PDO driver, this would have been a PECL package and a paragraph of phpize instructions for the reader to follow. Prebuilt binaries as the default install path is the single biggest quality-of-life change in PHP extension distribution since I started writing them.

Where It Stands

pdo_duckdb is an early release, and I'd rather say so than oversell it. The query path, the appender, parameter binding, transactions, and type decoding all work and are covered by tests. Streaming is the main thing still on the bench, and the type-mapping surface will keep growing as people push real workloads through it. I haven't published benchmarks, because the speed here is DuckDB's and the driver's job is to stay out of its way.

If you've wanted DuckDB's analytical engine reachable from PHP without learning a new client API or hand-rolling FFI, this is that, sitting behind the PDO surface you already know.

pie install iliaal/pdo_duckdb

github.com/iliaal/pdo_duckdb