Coder Legion - Recent posts tagged spark

DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap

Thu, 22 Oct 2026 08:52:00 +0000

Originally published on DEV.to: https://dev.to/firfircelik/duckdb-20-vs-polars-vs-spark-on-one-machine-150m-rows-under-a-45-gib-cap-h38

DuckDB 2.0.0 shipped on October 21, 2026 (announcement, PyPI). The next morning I ran the same single-machine benchmark I had already run on DuckDB 1.5.6: three engines, one 8 vCPU VM, Parquet in and Parquet out, a hard memory cap, and four queries that look like real data engineering work.

The thesis I started with, and that the numbers support:

DuckDB beats Spark-on-one-box on memory-bound DE work. It finished every run, at every size, 5 to 12 times faster than local PySpark at 150M rows, and it still finished the heavy queries with a 1 GB memory limit.
Polars hits a wall on window dedup. At 150M rows the ROW_NUMBER() style "latest row per customer" query never finished under the cap. Not at 4.5 GiB, not with an out-of-core budget, not at 9 GiB.
Polars is not slow. On filters, joins and a high-cardinality group by it ties DuckDB or beats it, when it fits.

Everything below is measured on that one box. Where I interpret or give an opinion, I say so.

The contenders

	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Model	In-process SQL engine	In-process DataFrame engine, lazy + streaming	JVM engine, `local[8]`
How I drove it	`COPY (SQL) TO 'out.parquet'`	`LazyFrame.sink_parquet()`	`spark.sql(...).write.parquet()`
Memory knob	`SET memory_limit = '3GB'`	default (plus one run with an out-of-core budget)	`spark.driver.memory = 3g`
Threads	8	8 (`POLARS_MAX_THREADS`)	8

Polars 2.0.0 was published on PyPI on October 6, 2026, PySpark 4.2.0 on July 14, 2026. DuckDB 2.0.0 wheels landed on PyPI on October 21, the same day as the release calendar entry.

Setup

Hardware and versions

VM: 8 vCPU Intel Xeon, 15 GiB RAM, no swap, Linux 6.12. It is a shared box, so other processes were running. Every run waited until at least 5 GiB was free and CPU busy was under 30% over a 3 second window.
Python 3.13.5, OpenJDK 21.0.12, pyarrow 26.0.0.
DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0. The "before" numbers are DuckDB 1.5.6 on the same box and data, run on October 19.

Data. A seeded synthetic star schema written with pyarrow: a sales fact table and a 50,000-row products dimension. Three sizes: 10M rows (213 MB), 50M (1.1 GB) and 150M (3.2 GB of snappy Parquet, 30 files of 5M rows). About 10 orders per customer, so 150M rows means roughly 15M customers.

Queries. The same SQL for DuckDB and Spark, the same logic as Polars expressions:

q1_filter_agg: filter on a timestamp and status, group by store and status.
q2_join_agg: fact-to-dimension join, group by category and brand.
q3_window_latest: latest order per customer with ROW_NUMBER(). The classic dedup and SCD pattern.
q4_highcard_agg: one row per customer (about 15M groups at 150M).

-- q3_window_latest
SELECT customer_id, order_id, ts, qty * price AS amount
FROM (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                 ORDER BY ts DESC, order_id DESC) AS rn
    FROM sales
) t
WHERE rn = 1

Method.

Every run is a fresh process: interpreter start, import, session, read Parquet, compute, write Parquet. So no warm caches inside the engine, and Spark pays its JVM start every time.
Engines run strictly one at a time. The engine order rotates per repetition so no engine always goes first.
5 repetitions at 10M, 3 at 50M and 150M. I report medians.
A watchdog samples the RSS of the whole process tree every 50 ms and kills the run if it goes above 4.5 GiB. That is the cap. It is deliberately tighter than the box, because "fits in a small container" is the question I care about.
Wall is the whole process. Query is read, compute and write only. The difference is setup.
Correctness: a validation script compares row counts and rounded checksums of every output. Where an engine produced output, all three engines agreed, for every query and size.

Results at 150M rows

Query	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Filter + aggregate	0.87 s (3/3)	0.96 s (3/3)	10.77 s (3/3)
Join + aggregate	1.47 s (3/3)	1.46 s (3/3)	16.10 s (3/3)
Window dedup	7.92 s (3/3)	killed (0/3)	46.29 s (2/3)
High-cardinality group by	7.91 s (3/3)	4.21 s (1/3)	39.70 s (3/3)

Median wall seconds, runs that finished under the 4.5 GiB cap in brackets.

Peak RSS (median of finished runs) tells the other half:

Query	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Filter + aggregate	0.77 GiB	0.94 GiB	1.85 GiB
Join + aggregate	0.42 GiB	1.96 GiB	1.61 GiB
Window dedup	3.01 GiB	above 4.5 (killed)	3.74 GiB, one run hit 7.51
High-cardinality group by	3.01 GiB	4.49 GiB	4.40 GiB

Three things stand out.

Spark on one box pays twice. About 5 seconds of every Spark run is setup (median 5.0 to 5.8 s, JVM plus session) before it reads a byte. Then the query itself is slower: query-only time was 7.4x DuckDB's on the filter, 7.9x on the join, 5.2x on the window dedup and 4.3x on the group by. In wall time that is 5x to 12x. And spark.driver.memory = 3g did not keep the process at 3 GiB: one window dedup run reached 7.51 GiB of process-tree RSS and was killed.

Polars is fast right up to the cap. It tied DuckDB on filter and join and was the fastest engine on the high-cardinality group by (4.21 s against 7.91 s). But only 1 of 3 runs finished: that one peaked at 4.49 GiB, and the other two crossed the cap at 4.50 and 4.55 GiB. Fast and close to the edge is a fine place for a notebook and a bad place for a scheduled job.

The window dedup is where Polars stops. At 150M rows it was killed in all 3 runs, within a few seconds of starting, at 4.52 to 4.66 GiB.

Polars and the window wall

I gave Polars two more chances on the 150M window dedup:

Polars 2.0.0, 150M window dedup	Result
Default settings, 4.5 GiB cap (Oct 22)	0/3, killed at 4.52 to 4.66 GiB
`POLARS_OOC_MEMORY_BUDGET_MB=3072`, 4.5 GiB cap (Oct 22)	0/3, killed at 4.57 to 4.66 GiB
Default settings, 9 GiB cap (Oct 19, same Polars 2.0.0)	0/3, killed at 9.04 to 9.54 GiB

The out-of-core budget variable is one I found in the Polars 2.0.0 source, not a documented knob, so treat that row as "I tried", not as guidance. The same variable did help the group by: with it, Polars finished 3/3 at 4.67 s and 4.18 to 4.30 GiB.

At smaller sizes the dedup works, but the shape is already visible:

Window dedup	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
10M rows	0.60 s, 0.65 GiB	1.83 s, 1.20 GiB	12.74 s, 2.94 GiB
50M rows	2.33 s, 2.19 GiB	10.36 s, 4.46 GiB	29.06 s, 3.77 GiB
150M rows	7.92 s, 3.01 GiB	killed	46.29 s, 3.74 GiB

At 50M Polars already used 4.46 GiB for a query DuckDB did in 2.19 GiB, and took 4.4x longer.

This is the Polars expression I used. It is the direct translation of the SQL:

import polars as pl

amount = pl.col("qty").cast(pl.Float64) * pl.col("price")
rn = pl.int_range(pl.len()).over(
    "customer_id", order_by=["ts", "order_id"], descending=True
)
(
    pl.scan_parquet("data/sales_150m/*.parquet")
    .filter(rn == 0)
    .select("customer_id", "order_id", "ts", amount.alias("amount"))
    .sink_parquet("out/polars/150m_q3_window_latest.parquet")
)

Fair caveat: I did not test other Polars formulations of the same dedup (a sort followed by unique(keep="first"), or a group_by().agg(...) with sort_by().last()). One of them may stream better. The point stands for the common case, though: the first thing a team writes is the window, and that one did not fit.

Update (Oct 22): after this went out, Victor Cid (@Vleyked) tested exactly that in engine-bench PR #1. His sort + unique(keep="first") version drops the in-memory window fallback from the plan and finished 3/3 at 150M rows under the same 4.5 GiB cap, peaking at 2.50 to 2.74 GiB, but only with POLARS_OOC_MEMORY_BUDGET_MB=1024. With default settings or a 3072 MB budget it still went over the cap. The price is speed: a 90.58 s median against 17.56 s for DuckDB on the same 4 vCPU runner, so roughly 5x slower. His output matched DuckDB row for row (14,999,347 rows). So Polars can do this dedup under the cap if you rewrite it and tune the spill budget, and with a smarter query it gets close to DuckDB (see below). His runs were on different hardware, so they are not in the tables above.

Second update: in engine-bench PR #2 he filtered each customer down to their max timestamp first and only sorted those rows to break ties. That brought the median down to 14.23 s at 150M rows with default Polars settings, peaking at 4.04 GiB under the same cap, with output matching DuckDB row for row. On his runner DuckDB's reference was a single 18.50 s run, so I'm not calling that a repeated cross-engine win. The real lesson is that the first way you write a Polars dedup can decide whether it finishes at all.

Third update (Oct 23): Ritchie Vink, the creator of Polars, pointed out that only DuckDB was told about the memory limit, so Polars did not know it had to stay under 4 GiB, and suggested setting POLARS_OOC_MEMORY_BUDGET_MB for a fairer comparison. I reran every Polars query at 150M rows on the same box under the same 4.5 GiB cap with that budget, next to a DuckDB rerun in the same session: the high-cardinality group by finished 3/3 at 6.17 s and 3.46 GiB with a 3072 MB budget, against 9.56 s for DuckDB (the box was busier than on Oct 22 and DuckDB ran slower too, so compare within this session, not with the tables above; the Oct 22 run with the same budget gave 4.67 s). The window dedup with over(order_by) still went over the cap in every run at 3072, 1024 and 512 MB, Victor's sort plus unique version fit only at 1024 MB (31.28 s, 2.29 GiB), and the max-timestamp rewrite finished 3/3 on default settings (5.61 s, 4.02 GiB). Ritchie says Polars main now spills the group by correctly and that this ships in the next release, which was not on PyPI yet when I ran this, so I will rerun when it ships. Raw results and output checks are in engine-bench PR #3 and in results/polars_ooc_budget_2026-10-23 on the polars-ooc-budget-2026-10-23 branch.

Fourth update (Oct 26): after the third update I reported the window dedup to Polars as pola-rs/polars#30270: the over(order_by) window inside filter() ignored POLARS_OOC_MEMORY_BUDGET_MB and ran in an in-memory fallback node. Ritchie Vink fixed it (#30272, #30282 and #30280), and the fixes shipped in Polars 2.1.0. I reran the 150M benchmark on the official 2.1.0 wheel on the same box, with the same harness and the same 4.5 GiB cap, next to a DuckDB rerun in the same session. With a budget set, the window dedup now finishes 3/3 at every budget I tried: 18.78 s and 3.34 GiB at 3072 MB, 20.11 s and 1.29 GiB at 1024 MB, 43.01 s and 2.15 GiB at 512 MB. On 2.0.0 it failed at all three. Memory now follows the budget closely: at 512 MB, Victor's sort plus unique version peaked at 0.69 GiB (73.85 s) and the high-cardinality group by at 0.74 GiB (9.43 s). DuckDB still ran the window dedup faster in the same session, 8.07 s, about 2.3x quicker than Polars at 3072 MB, and the max-timestamp rewrite remains the fastest Polars dedup (5.12 s on default settings). Every finished output matched DuckDB row for row (14,999,347 rows). Two caveats. Without a budget, the window dedup and sort plus unique still went over the cap. My cap is a process watchdog, not a cgroup limit, so the new cgroup-aware default in 2.1.0 had nothing to detect and is untested here; set the budget explicitly. One sort plus unique run at 3072 MB took 285.64 s for no clear reason, so I report medians of 3 runs. Raw results, plans and output checks: engine-bench PR #4 and the 2.1.0 results README.

DuckDB under pressure: memory_limit = 1 GB

The interesting DuckDB question is not "how fast with room to spare" but "what happens when the box is small". Same 150M rows, memory_limit cut to 1 GB:

150M, `memory_limit = 1GB`	Wall (median)	Peak RSS	Finished
Window dedup	14.97 s	1.12 GiB	3/3
High-cardinality group by	13.94 s	1.08 GiB	3/3
Window dedup, `preserve_insertion_order = false`	14.94 s	1.12 GiB	3/3
Group by, `preserve_insertion_order = false`	13.88 s	1.08 GiB	3/3

With a third of the memory, DuckDB spilled to disk and still beat Spark's unconstrained run by about 3x on both queries, at a quarter to a third of Spark's peak RSS. The DuckDB docs list window functions with PARTITION BY ... ORDER BY among the blocking operators it can process larger than memory (How to Tune Workloads). This run is that sentence, measured.

The setup is three lines:

import duckdb

con = duckdb.connect()
con.execute("SET memory_limit = '1GB'")
con.execute("SET threads = 8")
con.execute("SET temp_directory = '/mnt/fast-local-disk/duckdb_tmp'")  # where spills go
con.execute("""
    COPY (
        SELECT customer_id, order_id, ts, qty * price AS amount
        FROM (
            SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                         ORDER BY ts DESC, order_id DESC) AS rn
            FROM read_parquet('data/sales_150m/*.parquet')
        )
        WHERE rn = 1
    ) TO 'out/latest_per_customer.parquet' (FORMAT parquet)
""")

DuckDB 1.5.6 vs 2.0.0

Same box, same data, same scripts. 1.5.6 ran on October 19, 2.0.0 on October 22. Median wall seconds at 150M:

Setting	Query	1.5.6	2.0.0	Change
`memory_limit = 3GB`	Window dedup	9.09	7.92	-13%
`memory_limit = 3GB`	Group by	9.61	7.91	-18%
`memory_limit = 1GB`	Window dedup	17.79	14.97	-16%
`memory_limit = 1GB`	Group by	22.06	13.94	-37%
1GB, no insertion order	Window dedup	14.85	14.94	+1%
1GB, no insertion order	Group by	15.15	13.88	-8%

The useful finding is the last four rows together. On 1.5.6, SET preserve_insertion_order = false was worth 16% on the dedup and 31% on the group by under a 1 GB limit. On 2.0.0 the default already lands where the tuned setting did, and the flag changes almost nothing for these two queries. If you carry that setting in job configs as a performance trick, re-measure after upgrading. It still matters for what it was designed for: the docs recommend it when large imports or exports run out of memory, and it lets DuckDB reorder rows without an ORDER BY.

Not everything improved. The 50M window dedup was slower on 2.0.0 (1.94 s to 2.33 s), and peak RSS for the light queries went up (150M filter: 0.26 to 0.77 GiB, join: 0.23 to 0.42 GiB), while the heavy queries went down (window dedup: 3.47 to 3.01 GiB). I have not investigated why. The two runs were also three days apart on a shared box, so treat differences under about 10 to 15% as noise. The 1 GB group by drop is far outside that.

The release itself is mostly a feature release: Quack and CONNECT for client-server use, asynchronous I/O, a first-class VARIANT, triggers, a PEG parser by default (announcement). It is an LTS line supported until at least October 2027. One migration note matters for pipelines: database files written by 2.0 cannot be read by versions before 2.0. My benchmark only reads and writes Parquet, so it never touched that.

Where Spark still wins

None of this says "replace Spark". It says Spark on one box is the wrong comparison for small and medium jobs. Spark still wins when:

The data does not fit one node's disk and memory, or the deadline does not fit one node's cores. This test stops at 3.2 GB of Parquet on purpose. DuckDB spills well, but one machine is still one machine.
The platform is built around it. On Databricks and Fabric, Spark is where Unity Catalog or OneLake governance, Delta writes with concurrent writers, lineage, job orchestration and streaming already live. A DuckDB step outside that boundary is a step you have to secure, monitor and explain.
The cluster is already running. The 5 second JVM start I measured per run disappears in a warm interactive cluster or a long job. The query-only gap (4x to 8x here) does not.
The team is a Spark team. One engine everyone can debug at 02:00 is worth more than a faster engine one person understands.

My opinion as a Lead DE on Azure, Databricks and Fabric: the line is not the engine, it is the size of the job. Most jobs I see are tens of millions of rows dressed up as big data and started on a cluster. For those (file-to-file transforms before landing, validation and reconciliation checks, small dimension builds, CI tests for SQL logic) a single-node engine with a memory limit is the cheaper and more predictable choice. Anything that writes governed Delta tables stays on the platform.

Failure modes I hit, so you don't have to

Engine memory settings are not process memory. Spark's 3g driver heap still produced a 7.51 GiB process. DuckDB's whole process stayed close to its memory_limit: about 3.0 GiB peak at a 3 GB limit, 1.1 GiB at 1 GB. Size containers on measured RSS, not on the setting.
"Streaming" does not mean bounded. Polars streamed the filter, join and group by fine. The window dedup with an over(order_by=...) did not stay bounded, even with 9 GiB.
Close to the cap is a failure that hasn't happened yet. Polars' group by finished in 1 of 3 runs, peaking at 4.49 GiB against a 4.5 GiB cap. Same query, same data, same version. Schedule that and it fails some nights.
Shared hosts lie. One Spark run at 10M was killed because free memory on the whole box fell under my 1 GiB safety floor while Spark itself was at 1.6 GiB. That was the neighbours, not Spark. Gate each run on free memory and CPU, and record both.
Validate outputs, not just timings. Every engine produced the same row counts and checksums where it finished. Without that check, a fast wrong answer looks like a win.

Reproduce it

Everything is public at firfircelik/engine-bench: scripts, raw results, pinned versions and the query plans. Here are the parts that matter. The data generator is seeded (SEED = 20261019) and writes 5M-row chunks:

rng = np.random.default_rng([SEED, chunk_index])
t = pa.table({
    "order_id": np.arange(lo, lo + n, dtype=np.int64),
    "ts": pa.array(start + rng.integers(0, 270 * 24 * 3600, n), pa.timestamp("s")),
    "customer_id": rng.integers(0, rows // 10, n, dtype=np.int32),
    "product_id": rng.integers(0, 50_000, n, dtype=np.int32),
    "store_id": rng.integers(0, 500, n, dtype=np.int16),
    "qty": rng.integers(1, 10, n, dtype=np.int16),
    "price": np.round(rng.uniform(1.0, 500.0, n), 2),
    "status": rng.choice(STATUSES, n, p=[0.55, 0.20, 0.12, 0.08, 0.05]),
})
pq.write_table(t, f"sales_{rows // 1_000_000}m/part-{chunk_index:04d}.parquet",
               row_group_size=1_000_000, compression="snappy")

Spark session, the only non-default settings:

spark = (SparkSession.builder.master("local[8]")
         .config("spark.driver.memory", "3g")
         .config("spark.local.dir", TMP)
         .config("spark.ui.enabled", "false")
         .config("spark.sql.session.timeZone", "UTC")
         .getOrCreate())

And the watchdog, which is the part I would copy into any benchmark of your own. It measures the whole process tree, because JVMs and thread pools do not show up in the parent's RSS:

import psutil, subprocess, time

GIB = 1024 ** 3

def tree_rss(p):
    total = 0
    for q in [p] + p.children(recursive=True):
        try:
            total += q.memory_info().rss
        except psutil.NoSuchProcess:
            pass
    return total

proc = subprocess.Popen(cmd)
p, peak = psutil.Process(proc.pid), 0
while proc.poll() is None:
    rss = tree_rss(p)
    peak = max(peak, rss)
    if rss > 4.5 * GIB:
        for q in p.children(recursive=True) + [p]:
            q.kill()
        break
    time.sleep(0.05)

The takeaway

On one 8 vCPU machine with a 4.5 GiB cap, DuckDB 2.0.0 was the only engine that finished every run of every query at every size, and it did the heavy ones with 1 GB. Polars matched or beat it on the light queries and the group by, and could not finish the window dedup at 150M even with 9 GiB. Local Spark finished almost everything, 5x to 12x slower, and its memory setting did not bound its memory.

If your job fits on one machine, measure it on one machine before you start a cluster for it.

Sources

Announcing DuckDB 2.0.0 LTS "Cinnamon Teal", DuckDB team, October 21, 2026
DuckDB release calendar (2.0.0 LTS on 2026-10-21, 1.5.6 on 2026-09-28)
duckdb 2.0.0 on PyPI, polars 2.0.0 on PyPI, pyspark 4.2.0 on PyPI
DuckDB: How to Tune Workloads (larger-than-memory operators, preserve_insertion_order)
DuckDB: Order Preservation
All timings and memory numbers: my own runs on October 19 (DuckDB 1.5.6, and the 9 GiB Polars and Spark runs) and October 22, 2026 (DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0).

Fabric Runtime 2.0 is becoming the default. Here is what actually breaks.

Fri, 09 Oct 2026 13:20:30 +0000

Originally published on DEV.to.

Fabric Runtime 2.0 is becoming the default. Here is what actually breaks.

Late September 2026 is when Fabric Runtime 2.0 stops being an opt-in experiment and becomes the path of least resistance for new workspaces and environment items. If you treat that as a release note, you will discover the breakages in production.

I work as a Lead Data Engineer on Azure, Fabric, Databricks, and Spark. This is the migration checklist I wish teams ran before the default flipped under them.

Docs to keep open while you read:

1) Hook: the default is the migration

Microsoft's Runtime 2.0 page is blunt: GA and ready for production, but not the default yet. You opt in. The stated plan is to make Runtime 2.0 the default UX selection and the default for new workspaces and new environment items in late September 2026.

Today is 28 September 2026. That window is now.

Existing workspaces, jobs, and Environment items stay on whatever you configured. The trap is quieter. Every new workspace for a pilot, every Environment someone creates because "defaults are fine," starts landing on Spark 4.1 / Delta 4.2 / Python 3.13 once the plan executes.

Treat the flip as an incident rehearsal date. Pin what you care about. Validate 2.0 on a scoped Environment. Do not discover Python wheel breakage through a failed Monday job.

Runtime 1.3 still has runway. Secondary writeups put Microsoft's LTS window for 1.3 roughly from early October 2026 through March 2027. Verify against current lifecycle docs before you commit a program date. LTS means supported, not "ignore migration."

2) Wrong assumption: GA means notebooks just run

Runtime 1.3 to Runtime 2.0 is not a patch bump. It is a stack jump on nearly every component:

Component	Runtime 1.3 (typical)	Runtime 2.0
Apache Spark	3.5	4.1
Delta Lake	3.2	4.2
Python	3.11	3.13
Java	11	21
Scala	2.12	2.13
OS	(prior Fabric OS)	Azure Linux 3.0 (Mariner 3.0)

Industry writeups also call out Delta protocol defaults moving higher (reader/writer versions, Deletion Vectors in the 2.0 path). Treat protocol as a first-class migration risk, not a footnote. See Microsoft's Delta Lake table format interoperability guidance for what cross-experience readers can actually open.

Practical fallout I care about:

Python 3.11 -> 3.13. Environment-level libraries may migrate automatically. Validation does not. Custom wheels and pinned packages are your problem. Session-scoped %pip install and notebook Resources are a different blast radius than Environment items.
Scala 2.12 -> 2.13 / Java 11 -> 21. JARs and connectors that "worked forever" on 1.3 deserve a dual-run, not hope.
SparkR is deprecated in Spark 4.x and may be removed later. If you still have SparkR notebooks, the clock is already running.

GA means Microsoft ships it for production use. It does not mean your Environment's library graph survives the jump without you watching.

3) Wrong assumption: NEE = free 6x

Runtime 2.0 includes the Native Execution Engine (NEE). Microsoft's docs describe it as a vectorized path based on Apache Gluten (offload middle layer) and Velox (C++ engine from Meta). Supported operators leave the JVM and run columnar with SIMD-style acceleration on Parquet, Delta, and (now) CSV. Microsoft cites up to six times faster vs open-source Spark on TPC-DS. That is a vendor benchmark on a specific workload family. It is a ceiling story, not your median job.

How you enable it:

Environment -> Spark compute -> Acceleration -> enable native execution engine (preferred UX path), or
Spark conf: spark.native.enabled (also usable per notebook / SJD via %%configure)

Hard limits from the docs that matter in production:

Structured streaming is not supported. Use streaming? Plan for JVM (or keep NEE off for those jobs).
Unsupported ops fall back to classic Spark. Fallback is automatic. Your job keeps running. Your SLA may not.
Every boundary between native columnar and JVM row-oriented execution pays a conversion tax. That is the part marketing slides skip.
Fabric Spark Advisor can surface fallback alerts in notebook cell output. Use them. Do not ignore green "job succeeded" when Advisor is screaming about offload misses.

NEE also preserves AQE, cost-based rewrites, column pruning, and predicate pushdown when operators do offload. Good. That does not help if half the plan is bouncing in and out of Velox.

Blunt take: a vectorized engine that falls back on half your plan is not acceleration. It is a tax you opted into because a dashboard said "up to 6x."

4) How to see the truth in the plan

Do not argue with anecdotes. Read the plan.

In Spark UI / History Server, look for node names ending in:

*Transformer (for example ProjectExecTransformer, BroadcastHashJoinExecTransformer, RollUpHashAggregateTransformer)
*NativeFileScan
VeloxColumnarToRowExec

Those suffixes mean native offload happened for that node. Light-blue JVM nodes next to green native nodes (Microsoft's Gluten SQL / DataFrame tab uses that color language) are your fallback map. Every VeloxColumnarToRowExec (and the reverse path) is a place where you paid for format conversion.

Also useful: df.explain(), Fabric Spark Advisor fallback alerts, and the Gluten SQL / DataFrame tab's native-vs-JVM node counts.

Surgical disable for one cell (unsupported op, correctness edge, or A/B):

spark.conf.set('spark.native.enabled', 'false')

Re-enable afterward if later cells should use NEE again. Spark runs cells in order. Conf sticks.

5) Delta 4.x trap for multi-workload tables

This is the silent one.

Microsoft is explicit: Delta Lake 4.2-specific features are experimental and only work on Spark experiences (notebooks and Spark Job Definitions). If the same Delta tables must be used across multiple Fabric workloads, do not enable those features.

Why it hurts: SQL analytics endpoint, Power BI Direct Lake, and other non-Spark readers do not all speak the same protocol / feature set. Protocol upgrades (higher reader/writer versions, Deletion Vectors, experimental checkpoints) are effectively one-way doors. Recreate the table is the recovery path, not "uncheck the box." Shared bronze/silver tables that feed Spark and Direct Lake and SQL endpoint are exactly where someone enables a shiny 4.x feature on a Friday and breaks Monday's semantic model.

Rule I use: default to the most constrained reader in the estate. If Direct Lake or the SQL endpoint must read it, keep features inside what those readers support. Save experimental Delta 4.x toys for Spark-only sandboxes.

6) Practical migration pattern

This is the runbook shape I actually use.

1. Inventory

Workspaces and Environment items: which runtime are they on?
Jobs that create new Environments without pinning
Libraries: Environment wheels vs session %pip
Tables read by more than one Fabric experience
Streaming jobs (NEE: leave off or isolate)
SparkR notebooks (debt clock)

2. Baseline on 1.3

Duration, shuffle bytes, spill, failure rate, data quality checks
Capture plans for critical queries before you change anything
Note current Delta protocol / table features on shared tables

3. Dual-run on 2.0

Scoped Environment item first (surgical), not whole workspace
Same input snapshot, same checks, compare plans and SLA
Turn NEE on only after the JVM path is green on 2.0
Kill vanity actions that force row materialization (yes, that includes habit .show() in hot paths during timing runs)

4. Rollback in the runbook before you need it

Runtime: pin Environment / workspace back to 1.3 while LTS covers you
NEE only:

spark.conf.set('spark.native.enabled', 'false')

Put both in the on-call doc. A rollback you invent at 02:00 is not a rollback.

7) CI/CD angle: pin the runtime like a toolchain

You would not let python float to whatever the image shipped last Tuesday without a lockfile. Treat Fabric runtime the same way.

Pin runtime version on Environment items that production notebooks and SJDs attach to
Fail the pipeline if an Environment publishes without an explicit runtime
Gate promotion: 1.3 baseline metrics vs 2.0 dual-run metrics must be compared, not vibes
Separate "NEE enabled" as its own promotion flag. Runtime upgrade and native offload are two changes. Ship them as two changes.
Use Fabric runtime release channels (preview) to rehearse upcoming defaults on custom pools. Do not park production there.

If CI deploys notebook JSON and never asserts runtime + Acceleration settings, you are continuous-delivering entropy.

8) Close: benchmarks sell the ceiling, production pays the median

Workload type	Enable NEE?	Watch for...
Large Parquet/Delta batch ETL, heavy agg/join	Yes, after dual-run	Fallback density, conversion boundaries, Advisor alerts
Interactive lakehouse SQL / notebook exploration	Often yes	Date type mismatches on filters, wide UDF usage
Structured streaming	No (unsupported)	Accidental enable via Environment inheritance
JSON/XML heavy paths	Limited / often no	Format falls back to JVM; measure before celebrating
Shared Delta tables (Spark + SQL endpoint / Direct Lake)	NEE optional; Delta 4.x experimental features: no	Protocol / feature lockout of non-Spark readers
SparkR or exotic Scala libs	Maybe later	Language deprecation and binary incompat first

Takeaways

Pin Runtime 1.3 on anything you cannot afford to surprise, then opt into 2.0 on a scoped Environment before the default teaches your newest workspace the hard way.
Treat NEE as a plan shape problem, not a toggle. *Transformer and VeloxColumnarToRowExec tell the truth; "up to 6x" does not.
Never enable experimental Delta 4.x features on multi-experience tables. Shared tables inherit the weakest reader's constraints.

Benchmarks sell the ceiling. Production pays the median. Migrate like you believe that.

What is your scarier unpaid bill right now: Python 3.13 libraries, NEE fallback thrash, or a Delta feature someone enabled on a table Direct Lake still needs to read?

Sources

Runtime 2.0 in Fabric (Microsoft Learn)
Native execution engine for Fabric Data Engineering (Microsoft Learn)
Context (not a substitute for the docs): Fabric Runtime 2.0 two breakages (Rajaniesh Kaushikk, 2026-09-23)

Cost & Performance Comparison of Data Orchestration on Azure; Data Analysis of Data!

Tue, 03 Feb 2026 03:58:35 +0000

New Project: Cost & Performance Comparison of Data Orchestration on Azure

I recently completed an experimental study comparing runtime and cost efficiency across three common data engineering orchestration approaches on Azure:

• Databricks Notebook Clusters
• Azure Data Factory → Databricks
• Databricks Workflows (multi-notebook pipelines)

Instead of theoretical claims, I used measured execution times, a fixed cluster configuration (Standard_D4s_v3), and then scaled the results to 1 TB of data and 100 hours of processing using clearly stated assumptions.

Key findings:

✅ For small datasets, orchestration overhead dominates (ADF & Workflows are slower)
✅ At TB-scale, all approaches converge in runtime — architecture matters more than speed
✅ Cost differences are marginal at scale, but orchestration choices matter for long-running workloads
✅ Databricks Workflows are ideal for Spark-heavy pipelines
✅ ADF still makes sense when orchestrating multiple Azure services

This project helped me better understand where architectural decisions actually matter — beyond “ADF vs Databricks” debates.

GitHub repo with experimental results, tables, and screenshots:
https://github.com/usman19zafar/Data-Engineering-Cost-Performance-Comparison

Happy to hear thoughts from other data engineers — especially how you decide orchestration strategy in production systems.

Corrupted‑to‑Gold

Sat, 31 Jan 2026 17:35:50 +0000

Corrupted‑to‑Gold: MDM Foundations

This build takes intentionally corrupted CSV files and transforms them into clean, trusted, analytics‑ready Delta tables. The pipeline demonstrates real‑world data engineering constraints: inconsistent delimiters, malformed dates, broken encodings, duplicate entities, and multi‑format ingestion.

The solution includes:

✔ RAW → BRONZE ingestion

Normalized delimiters

   Removed non‑ASCII noise
   Safe splitting logic
   Header detection
   Schema validation

✔ SILVER cleansing

Email, phone, country, and status normalization

   Multi‑format date parsing
   Null handling and type correction

✔ Survivorship scoring (MDM‑light)

Email validity

  Phone completeness
  Date reliability
  Status correctness
  Row‑level scoring + best‑record selection

✔ GOLD outputs

Clean, deduped, resequenced customer entities

   Delta‑optimized storage
   Ready for downstream MDM and ADF Mapping Data Flows

This project sets the foundation for the full MDM pipeline (matching, clustering, survivorship, and golden‑record creation) and the upcoming ADF MDF suite demonstrating schema drift, parameterization, joins, window functions, branching, SCD‑light, partitioning, and KPI generation.

If you want the full code, architecture, or notebooks, the repo is here:

github.com/usman19zafar/Project_corrupted-to-gold-data-journey