Coder Legion - Recent posts tagged databricks

Auto CDF Moved the CDC Tax From Writes to Reads: Delta CDF vs Auto CDF vs Spark 4.2 CHANGES

Mon, 26 Oct 2026 07:12:17 +0000

Originally published on DEV.to: https://dev.to/firfircelik/auto-cdf-moved-the-cdc-tax-from-writes-to-reads-delta-cdf-vs-auto-cdf-vs-spark-42-changes-9bb

On September 1, 2026, Databricks made automatic change data feed (Auto CDF) generally available. The release note says it "removes write-time overhead and makes MERGE and UPDATE operations about 15% faster on tables that you query for changes." Seven weeks earlier, Apache Spark 4.2.0 (July 14) added a SQL CHANGES clause, so reading row-level changes is now a first-class part of the engine, not a Delta-only option.

Most of the takes I have seen stop at "free CDC, faster MERGE, turn it on." My thesis is narrower:

Auto CDF does not remove the cost of change data. It moves it from the writer, who pays once per commit, to every reader, who pays on every read.
Its prerequisite, row tracking, has a write-time cost of its own that nobody puts on the slide.
Spark 4.2's CHANGES is a contract, not an implementation. Whether it works depends on your connector, and on open-source Delta 4.4.1 it does not work out of the box.

So the real question is not "legacy or auto." It is: how many times is each commit read, and how much data does each commit rewrite?

Three ways to get the changes

	Legacy CDF (write time)	Auto CDF (read time)	Spark 4.2 `CHANGES`
Where the work happens	Writer writes change files next to the data	Reader computes changes from row tracking metadata	Connector supplies a changelog, Spark post-processes
Opt in	`delta.enableChangeDataFeed = true` per table	Automatic when requirements are met	Catalog must implement `loadChangelog`
Who can read it	Any Delta reader that implements the `cdc` action of the protocol	Databricks readers only (DBR 19+)	Any Spark 4.2 client, if the connector supports it
Formats	Delta only	Delta with row tracking, Iceberg v3	Whatever the connector exposes
Read API	`readChangeFeed`, `table_changes()`	Same APIs	`SELECT ... CHANGES FROM VERSION`, `.changes()`

FACT (Databricks docs): legacy CDF "materializes changes during table writes", Auto CDF "computes changes during table reads using row lineage metadata", and the two cannot be used together on one table. Migration is a single statement: ALTER TABLE t UNSET TBLPROPERTIES ('delta.enableChangeDataFeed').

The mechanism, in two paragraphs

Legacy CDF. The Delta protocol has a cdc action. On UPDATE, DELETE and MERGE the writer puts the changed rows, with _change_type, into files under _change_data/. The protocol tells readers to use those files when they exist and skip the add and remove actions for that version. Insert-only commits and full partition deletes produce no change files, because the log already says what happened. So the reader of a commit touches roughly the changed rows, nothing more.

Auto CDF. Row tracking gives each row a stable row ID and a row commit version. The Delta protocol explains how: for inserts, the IDs are derived from a baseRowId per file plus the row position, which costs almost nothing. But when a writer copies a row into a new file (the "copy" in copy-on-write), it must preserve the stable ID, so it materializes the row ID and commit version into hidden columns in the data file. At read time, the engine compares row IDs and commit versions across the files a commit removed and added to work out which rows were inserted, updated or deleted. That is real computation over real files, and it happens on every read.

What I measured

I cannot run Databricks' Auto CDF outside Databricks, and I am not going to pretend otherwise. What I can run is open-source Delta Lake 4.4.1 on Spark 4.2.0, which has both halves of the mechanism: legacy CDF, and row tracking with _metadata.row_id and _metadata.row_commit_version.

Setup:

A 10M-row Delta table (about 507 MB of Parquet), deletion vectors off, so MERGE is pure copy-on-write.
Each round: a MERGE that updates about 100,000 rows (1%, keys hashed across the whole id range) and inserts 10,000 new ones. 5 rounds per variant, medians reported, after a discarded warm-up.
Three variants: plain, legacy CDF, row tracking only.
Read side: read the changes of the last commit. For legacy CDF that is readChangeFeed for one version. For row tracking it is a reconstruction I wrote: read the files the commit removed (at version v-1) and added (at version v), and join on row_id and row_commit_version.

I checked the reconstruction against legacy CDF on a table with both features enabled: for a MERGE (200 inserts, 1,986 pre-images, 1,986 post-images) and for a DELETE (100 deletes) the counts matched exactly.

The core of the reconstruction:

after = (spark.read.format("delta").option("versionAsOf", v).load(path)
         .select("*", F.col("_metadata.row_id").alias("rid"),
                 F.col("_metadata.row_commit_version").alias("rcv"),
                 F.col("_metadata.file_path").alias("fp"))
         .where(F.col("fp").isin(added_files)))
changed = after.where(F.col("rcv") == v)          # inserts + post-images
# pre-images: rows in removed files whose rid appears in `changed`
# deletes:    rows in removed files whose rid appears nowhere in `after`

Results, 10M rows, scattered 1% upsert, median of 5 rounds:

Variant	MERGE time	Bytes written per MERGE	Change files per MERGE	Read one commit's changes
Plain	6.03 s (4.75 to 7.69)	505.8 MB	none	n/a
Legacy CDF	6.52 s (6.19 to 7.10)	506.0 MB	10.8 MB	0.21 s
Row tracking only	6.16 s (5.92 to 7.69)	547.6 MB	none	4.73 s (my reconstruction)

A second, partial run of the plain and legacy CDF variants gave 5.64 s and 6.89 s for the MERGE, and 0.18 s for the CDF read.

What the numbers say, and what they do not

1. The write tax of legacy CDF is real but small here, and smaller than my noise. Legacy CDF was slower in both runs (6.52 vs 6.03 s, and 6.89 vs 5.64 s), but the spread inside one variant (4.75 to 7.69 s) is wider than the gap. Databricks measured about 15% on its runtime; my numbers are compatible with that and cannot confirm it. The byte count is solid: 10.8 MB of change files per MERGE, about 2% of what the MERGE rewrote anyway.

2. Row tracking wrote 8% more data on every rewrite. 547.6 MB versus 505.8 MB per MERGE, because every copied row now carries its stable row ID and commit version in hidden columns. That is about four times the size of the CDF change files in this workload. This is the part of the "free" read-time CDF that you pay at write time.

3. Reading changes at read time costs what the commit rewrote, not what it changed. The 1% scattered upsert touched every file, so the commit removed and re-added the whole table. Legacy CDF read 10.8 MB of change files in 0.21 s. My reconstruction had to scan both sides of a 500 MB rewrite to find 100,000 changed rows: 4.73 s, about 22 times slower. On a smaller test (200,000 rows, 8 files) the same scattered upsert rewrote 8 of 8 files, while an upsert clustered on the newest ids rewrote 1 of 8. The layout of your updates decides how expensive read-time CDF is.

What I cannot claim: that Databricks' implementation is as slow as mine. It is very likely smarter, and with deletion vectors a MERGE writes only the changed rows to new files and marks the old ones, which shrinks the set of rows a reader has to compare. My test is the copy-on-write case, which is the worst case for read-time CDF. The mechanism still holds: the read-time cost scales with what a commit touched, and you pay it once per reader per read.

4. CHANGES on open-source Delta 4.4.1 fails by default. With the stock DeltaCatalog, all three of SELECT * FROM t CHANGES FROM VERSION 1, the path-based form, and spark.read.changes("t") returned:

[UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE] The feature is not supported:
Catalog spark_catalog does not support Change Data Capture (CDC). SQLSTATE: 0A000

That matches the source. Delta PR #6794 (merged May 2026) adds catalog-driven read-time CDF behind an internal flag that is off by default, requires row tracking, and leaves the streaming side of the new path as a follow-up. I did not test the flag; it is internal and I would not run production on it. Spark 4.2 defines the shape of change data. Your connector decides whether you get any.

Gotchas to check before you run UNSET

All FACT, from the Databricks and Spark docs:

Auto CDF is not supported on tables with row filters or column masks. If your sensitive tables are governed with ABAC or masks, they are exactly the ones that cannot move.
Only Databricks readers can query Auto CDF on Delta, and external Iceberg clients cannot query it on Iceberg v3. Legacy CDF lives in the open Delta protocol. If anything outside Databricks reads your change feed, check before migrating.
It needs Databricks Runtime 19 or above and a table registered in Unity Catalog with row tracking enabled (or Iceberg v3).
Enabling row tracking on an existing table is not free. It assigns IDs to every existing row, can create several new table versions, and concurrent writers fail with MetadataChangedException while it runs. Pause streaming writers first. It also upgrades the writer protocol, so older external Delta clients that do not support those features can no longer write to the table.
Multi-statement transactions: Auto CDF is not supported if the source table was modified during the transaction.
_metadata.row_id and row_commit_version cannot be read while reading the change feed. If your downstream logic wants both, plan for it.
Spark 4.2 streaming netChanges holds back the latest commit. Rows touched only in the latest commit are not emitted until a later commit moves the watermark, and streaming net changes can differ from batch net changes over the same range. Fine for a pipeline, surprising for a low-latency consumer.

Decision table

OPINION, built on the mechanism and my numbers:

Your table looks like	Better choice	Why
Write-heavy, changes read rarely or by one pipeline	Auto CDF	You stop paying on every write for a feed few people read
One commit read by many consumers, often	Legacy CDF, or Auto CDF plus one stream that lands changes in a table	Read-time cost multiplies by readers times reads
Scattered updates on a copy-on-write table	Legacy CDF, or enable deletion vectors and measure	Each commit rewrites most files, so read-time diffing is expensive
Row filters or column masks	Legacy CDF	Auto CDF is not supported there
Change feed read outside Databricks	Legacy CDF	Auto CDF is Databricks-readers only
Iceberg v3 tables on Databricks	Auto CDF	Legacy CDF does not support Iceberg

What I would do this week

Inventory who reads each change feed. Count consumers and their trigger intervals per table. That count is the multiplier on read-time cost, and nobody has it written down.
Look at the update layout. In table history, compare numTargetRowsUpdated against files rewritten for your MERGE operations. A 1% update that rewrites 100% of files is a bad candidate for read-time CDF.
Migrate the boring tables first: write-heavy, one reader, no masks, nothing outside Databricks. Keep legacy CDF on the hot, many-reader tables until you have measured both on your own runtime.
If you fan out, fan out once. One stream reads the change feed and writes a changes table; the other consumers read that. The Databricks docs already show this archive pattern for permanent history.

Auto CDF is a good default for most tables, and a quiet bill for the ones read by five pipelines. CDC is still a cost. The release moved it to the reader, and the bill now goes up with every pipeline that reads the feed.

Hardware & versions

Box: 8 vCPU Intel Xeon VM, 16 GB RAM, Linux 6.12, OpenJDK 21.0.12.
PySpark 4.2.0, delta-spark 4.4.1 (delta-spark_4.2_2.13), local[8], driver memory 8 GB (5 GB in the partial second run), 16 shuffle partitions.
10M rows, 5 columns, about 507 MB Parquet, deletion vectors off. 5 rounds per variant, medians, warm-up discarded.
Same box also hosts another long-running benchmark of mine. During the main run its runner was paused waiting for an idle host, but its services stayed in memory. The partial second run overlapped that benchmark's active phase, so I quote it only as a cross-check. Treat timings as indicative. The byte counts and the correctness check are deterministic.
Measured October 26, 2026. Script, raw logs and pinned versions: cdf_bench gist.

Sources

Databricks September 2026 release notes, "Automatic change data feed is now generally available" (Sep 1, 2026): https://docs.databricks.com/aws/en/release-notes/product/2026/september
Databricks docs, Use change data feed (Auto CDF requirements, migration, limitations): https://docs.databricks.com/aws/en/delta/delta-change-data-feed
Azure Databricks docs, same page: https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed
Databricks docs, Row tracking: https://docs.databricks.com/aws/en/delta/row-tracking
Delta Lake protocol v4.4.1 (Add CDC File, Row Tracking, materialized Row IDs): https://github.com/delta-io/delta/blob/v4.4.1/PROTOCOL.md
Delta Lake docs, Change data feed: https://docs.delta.io/delta-change-data-feed/
Delta Lake docs, Row tracking: https://docs.delta.io/delta-row-tracking/
Delta Lake 4.4.0 release (Spark 4.2 support): https://github.com/delta-io/delta/releases/tag/v4.4.0
Delta PR #6794, Add Changelog support: https://github.com/delta-io/delta/pull/6794
Apache Spark 4.2.0 release notes (Change Data Capture, SPARK-55668): https://spark.apache.org/releases/spark-release-4-2-0.html
Apache Spark news, Spark 4.2.0 released (July 14, 2026): https://spark.apache.org/news/
Spark PR, SPARK-55948 DSv2 CDC connector API and SQL CHANGES clause: https://github.com/apache/spark/pull/54738
Spark 4.2.0 API docs, Changelog: https://spark.apache.org/docs/4.2.0/api/scala/org/apache/spark/sql/connector/catalog/Changelog.html
Spark 4.2.0 API docs, DataStreamReader.changes (streaming netChanges behavior): https://spark.apache.org/docs/4.2.0/api/scala/org/apache/spark/sql/streaming/DataStreamReader.html
My measurements: OSS Delta 4.4.1 on Spark 4.2.0, October 26, 2026 (setup above).

I post shorter breakdowns daily on LinkedIn.

Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew

Fri, 09 Oct 2026 13:20:49 +0000

Originally published on DEV.to.

Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a job restarted and silver row counts jumped, or a hot customer key OOMed a worker while the rest of the cluster sat idle.

Neither bug is exotic. Both are documented contracts that happy-path snippets skip.

This article covers two production surfaces Staff and Lead data engineers own on Databricks / PySpark:

Streaming upserts via foreachBatch: at-least-once writes, watermark dedupe state, Delta MERGE idempotency (logical, not txnAppId), empty batches, and why you must fully consume a batch DataFrame.
Pandas Function APIs: groupBy().applyInPandas() loads an entire group into memory and does not honor spark.sql.execution.arrow.maxRecordsPerBatch; mapInPandas / mapInArrow do.

I will mark FACT (from primary docs) vs interpretation (engineering judgment). No invented latency percentages. No "exactly-once foreachBatch" fairy tales.

Primary sources: Databricks foreachBatch, Delta MERGE / streaming upsert, watermarks, state store reader, idempotent Delta writes, pandas function APIs, Apache Arrow in PySpark.

Part 1. Design for the restart: `foreachBatch`, watermarks, and MERGE

Contract 1: `foreachBatch` is at-least-once

FACT (Databricks): foreachBatch() provides only at-least-once write guarantees. You can use the batchId to deduplicate and reason toward exactly-once yourself. You still own end-to-end semantics.

Interpretation: A green micro-batch in the UI is not a guarantee that a retry will be a no-op. Treat every handler as if Spark will call it again with the same batch_id.

APP_ID = "silver-orders-upsert-v1"  # change if you reset the checkpoint


def process_batch(batch_df, batch_id: int) -> None:
    # FACT: Delta sources can deliver empty micro-batches (e.g. OPTIMIZE with
    # nothing to process, or predicate pruning). Code must tolerate them.
    if batch_df.isEmpty():
        return

    valid = batch_df.filter("order_amount > 0 AND customer_id IS NOT NULL")
    invalid = batch_df.filter(
        "NOT (order_amount > 0 AND customer_id IS NOT NULL)"
    )

    # FACT: txnAppId + txnVersion on DataFrameWriter make appends idempotent
    # for retries of the same batch_id. Binding txnVersion to batch_id is the
    # documented pattern.
    (
        valid.write.format("delta")
        .mode("append")
        .option("txnAppId", APP_ID)
        .option("txnVersion", batch_id)
        .saveAsTable("main.silver.orders")
    )

    if not invalid.isEmpty():
        (
            invalid.write.format("delta")
            .mode("append")
            .option("txnAppId", APP_ID)
            .option("txnVersion", batch_id)
            .saveAsTable("main.silver.orders_dlq")
        )


(
    spark.readStream.table("main.bronze.raw_orders")
    .writeStream.foreachBatch(process_batch)
    .option("checkpointLocation", "/Volumes/main/ops/checkpoints/orders")
    .start()
)

FACT (Databricks Delta streaming): If you delete the streaming checkpoint and restart with a new checkpoint, you must provide a different txnAppId. New checkpoints start at batch ID 0; reusing the old app id can make Delta skip writes you intended to redo.

FACT (Databricks): Prefer letting the query fail and letting Lakeflow Jobs / Airflow retry the batch, rather than building silent local retry loops that can duplicate or drop data.

Contract 2: watermark dedupe has a state lease (and you must finish the batch)

FACT (Databricks watermarks): dropDuplicatesWithinWatermark always deduplicates events that arrive within the watermark delay. Events outside the delay might still be deduped, but that is not guaranteed. To guarantee dropping all duplicates, set the watermark delay greater than the maximum timestamp distance between duplicate events.

FACT (Databricks foreachBatch): When you use stateful operators such as dropDuplicatesWithinWatermark, each foreachBatch iteration must completely consume the batch DataFrame. If you only peek (show(2), take a few rows) and leave rows unread, the streaming query can fail on the next batch.

from pyspark.sql.functions import col

deduped = (
    spark.readStream.table("main.bronze.events")
    .withWatermark("event_time", "10 hours")  # > max duplicate lag (FACT guidance)
    .dropDuplicatesWithinWatermark(["event_id"])
)


def bad_handler(batch_df, batch_id: int) -> None:
    # Anti-pattern: partial consume after a stateful operator
    batch_df.show(2)
    # remaining rows never consumed -> next micro-batch can fail


def good_handler(batch_df, batch_id: int) -> None:
    if batch_df.isEmpty():
        return
    # Full action that consumes the plan (write, count, foreach, etc.)
    (
        batch_df.write.format("delta")
        .mode("append")
        .option("txnAppId", "events-dedupe-v1")
        .option("txnVersion", batch_id)
        .saveAsTable("main.silver.events")
    )


(
    deduped.writeStream.foreachBatch(good_handler)
    .option("checkpointLocation", "/Volumes/main/ops/checkpoints/events")
    .start()
)

Debugging tip (FACT): You can inspect Structured Streaming state with the statestore reader. Metadata can show an operator named dedupeWithinWatermark; query that operatorId when you need to see what the lease still holds. See Read Structured Streaming state information.

Contract 3: MERGE inside `foreachBatch` is a logical idempotency problem

FACT (Delta Lake): You must use foreachBatch for Delta Lake merge operations from Structured Streaming. Restarts can apply the same batch multiple times, so your merge statement inside foreachBatch must be idempotent.

FACT (Databricks / Delta writer options): txnAppId and txnVersion are DataFrameWriter options for idempotent writes (append / save). Do not assume they wrap DeltaTable.merge(...). Design MERGE so re-running the same micro-batch yields the same table state.

FACT (Delta): A merge can fail if multiple source rows match the same target row (ambiguous update). For CDC, preprocess to keep the latest change per key before merging.

FACT (Delta): MERGE often reads the source multiple times. That can inflate reported streaming input rates. Cache the micro-batch DataFrame before merge and unpersist after if this becomes a bottleneck.

from delta.tables import DeltaTable
from pyspark.sql import functions as F

target = DeltaTable.forName(spark, "main.silver.customers")


def upsert_cdc(batch_df, batch_id: int) -> None:
    if batch_df.isEmpty():
        return

    # Collapse to latest change per business key (FACT: avoid multi-match MERGE)
    latest = (
        batch_df.select(
            "customer_id",
            F.struct("event_time", "payload", "deleted").alias("chg"),
        )
        .groupBy("customer_id")
        .agg(F.max("chg").alias("chg"))
        .select("customer_id", "chg.*")
    )

    # FACT: MERGE may scan the source more than once -> cache the batch
    latest = latest.cache()
    try:
        (
            target.alias("t")
            .merge(latest.alias("s"), "t.customer_id = s.customer_id")
            .whenMatchedDelete(condition="s.deleted = true")
            .whenMatchedUpdate(
                set={"payload": "s.payload", "updated_at": "s.event_time"}
            )
            .whenNotMatchedInsert(
                condition="s.deleted = false",
                values={
                    "customer_id": "s.customer_id",
                    "payload": "s.payload",
                    "updated_at": "s.event_time",
                },
            )
            .execute()
        )
    finally:
        latest.unpersist()


(
    spark.readStream.table("main.bronze.customer_cdc")
    .writeStream.foreachBatch(upsert_cdc)
    .option("checkpointLocation", "/Volumes/main/ops/checkpoints/customer_cdc")
    .start()
)

Interpretation: "Idempotent MERGE" here means latest-wins on a stable key (and deletes keyed the same way), not "Spark will skip the merge via txn metadata." If the same CDC events replay, the table should converge to the same rows.

Managed alternative (context, not a product pitch): Databricks processing-guarantees docs note that managed Delta-to-Delta flows and AUTO CDC can give stronger defaults; a custom foreachBatch edge is still at-least-once unless you add explicit idempotent write or dedupe logic. Use AUTO CDC when it fits; when you hand-roll MERGE, you own the restart contract above.

Part 1 checklist (print this next to the PR)

Checkpoint	Why
Empty-batch safe	Delta sources can emit empty micro-batches
Full consume after stateful ops	Partial `show`/`take` can fail the next batch
Watermark > max duplicate lag	Out-of-watermark dedupe is not guaranteed
Append path uses `txnAppId` + `txnVersion=batch_id`	Writer-level idempotency on retry
New checkpoint => new `txnAppId`	Else batch 0 collisions skip writes
MERGE is latest-per-key / safe to re-run	txn options do not wrap `DeltaTable.merge`
Cache around MERGE when metrics / cost hurt	MERGE multi-pass over the micro-batch
Fail fast to the orchestrator	Local retry loops are how you invent exactly-once bugs

Part 2. Arrow is not a free pass: `applyInPandas` vs `mapInPandas` / `mapInArrow`

Python DE craft is not only streaming. The other Tuesday failure mode is "we used pandas because Arrow is fast" and then a skewed group killed the executor.

Three APIs, three memory contracts

FACT (Databricks pandas function APIs / Apache Spark):

API	What you write	Memory / batching contract
`groupBy().applyInPandas(f, schema)`	One pandas DataFrame per group	All rows for the group load into memory before `f` runs. `spark.sql.execution.arrow.maxRecordsPerBatch` is not applied. Skew => OOM risk.
`mapInPandas(f, schema)`	Iterator of pandas DataFrames (batches)	Batch size controlled by `spark.sql.execution.arrow.maxRecordsPerBatch` (default 10,000 rows in Spark docs). Output length can differ from input.
`mapInArrow(f, schema)`	Iterator of `pyarrow.RecordBatch`	Same Arrow batch sizing idea; stay in Arrow without pandas round-trip when you can.

Cogrouped applyInPandas has the same whole-cogroup-in-memory warning.

Skew turns "elegant" into OOM

# Looks clean. Under a hot customer_id it is a bomb.
def per_customer_features(pdf):
    # pdf = ALL rows for one customer_id
    pdf = pdf.sort_values("event_time")
    pdf["rolling_amt"] = pdf["amount"].rolling(50, min_periods=1).sum()
    return pdf


# FACT: entire group must fit in worker memory; maxRecordsPerBatch does NOT help
out = (
    events.groupBy("customer_id")
    .applyInPandas(
        per_customer_features,
        schema="customer_id string, event_time timestamp, amount double, rolling_amt double",
    )
)

Interpretation: Use applyInPandas when the group is a true unit of work and you can bound group size (filter, sample hot keys, pre-aggregate, salt, or split whales). Do not use it as a generic "run pandas on Spark" hammer.

Prefer batch maps when you do not need whole-group semantics

spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "5000")


def score_batches(iterator):
    # init once per Python worker task
    model = load_light_model()  # your code; keep it picklable / available on workers
    for pdf in iterator:
        pdf = pdf.copy()
        pdf["score"] = model.predict(pdf[["f1", "f2"]])
        yield pdf[["id", "score"]]


scored = events.mapInPandas(
    score_batches,
    schema="id long, score double",
)

Or stay in Arrow when pandas is unnecessary:

import pyarrow as pa
import pyarrow.compute as pc


def filter_arrow(iterator):
    for batch in iterator:
        mask = pc.greater(batch.column("amount"), 0)
        yield batch.filter(mask)


positive = events.mapInArrow(filter_arrow, schema=events.schema)

Decision table

Situation	Prefer	Watch out
Need whole-group window / asof / custom group ML	`applyInPandas`	Hot keys; no Arrow batch limit; cogroup same issue
Row/batch transform, arbitrary output length	`mapInPandas`	Tune `maxRecordsPerBatch` down if wide rows
CPU vector ops, avoid pandas copies	`mapInArrow`	Arrow type coverage / compute familiarity
Series-to-series column expression	classic `pandas_udf`	Output length must match input for many UDF shapes

FACT (Spark Arrow guide): Lower maxRecordsPerBatch when column count is large; Arrow batches can pressure JVM memory during conversion.

How the two themes connect

Both failures are the same Staff DE mistake: assuming the platform's happy path equals your failure domain.

Streaming: you assumed merge + checkpoint = exactly-once. Docs say at-least-once + you make MERGE idempotent; writer txn options are a different tool.
Pandas APIs: you assumed "Arrow optimized" = bounded memory. Docs say group maps ignore the batch limit.

Own the contract in the PR template, not in the postmortem.

Optional diagram (for a cover or inline figure)

Two panels:

Restart timeline: micro-batch N fails mid-MERGE -> orchestrator retries batch N -> arrows labeled "append: txn skip" vs "MERGE: must be latest-wins safe".
Skew funnel: many small customer_id groups sail through applyInPandas; one whale group expands to "all rows in RAM" with a red X; side path routes whales to salted / pre-aggregated / mapInPandas path.

References

I'm Firat Celik, Lead Data Engineer. I write about production data platforms, PySpark/Delta, and the boring contracts that keep pipelines honest.

Azure Synapse to Databricks: A Migration Playbook That Survives Production

Fri, 09 Oct 2026 13:19:33 +0000

Originally published on DEV.to.

Azure Synapse → Databricks: A Migration Playbook That Survives Production

A field-style guide for Lead / Senior Data Engineers moving Dedicated SQL Pools, pipelines, and BI off Synapse without burning trust, budget, or the on-call team.

Why this migration is harder than the slide deck
What you are actually migrating
Program shape: discovery → waves
Four parallel workstreams
Data movement: skip the JDBC tax
Code conversion: syntax is easy, semantics bite
Dimensional patterns on Delta (SCD2)
Reconciliation & cutover
Power BI / consumption cutover
30 / 60 / 90 plan
Anti-patterns I keep seeing
Field rules I will not compromise on
FAQ
Sources

1) Why this migration is harder than the slide deck

If you have lived in Azure Synapse for years, the marketing line sounds simple:

Move to Databricks. Unity Catalog. Lakehouse. Done.

In production programs, that is not the hard part.

The hard part is that Synapse is not one system. Under one brand you usually own years of Dedicated SQL Pool logic, ADF/Synapse Pipelines, serverless SQL over the lake, Spark pools, permission models, and Power BI datasets wired to Synapse endpoints.

Treat that estate as one workstream with one deadline, and the schedule slips. Every time.

I work as a Lead Data Engineer across Azure Synapse, Microsoft Fabric, Azure Data Lake, Databricks/Spark, SQL/T-SQL, and Power BI. This playbook is the version I want in the room before the first converted stored procedure lands in production.

Thesis: Migration success is not “all notebooks on Databricks.”
It is decision-ready data, trusted KPIs, and a platform your team can operate.

2) What you are actually migrating

Synapse piece	Typical complexity	What “done” looks like on Databricks
Dedicated SQL Pools	Highest — procs, distributions, CCI, years of tuning	Databricks SQL / Spark jobs, Delta tables, Liquid Clustering / predictive optimization
Serverless SQL	Medium — views & external tables over ADLS	Views / tables in Unity Catalog over the same lake paths (often simpler)
Spark pools	Lower — already Spark-shaped	Databricks Runtime notebooks / jobs (often few changes)
ADF / Synapse Pipelines	High — orchestration + connectors	Lakeflow Connect / Lakeflow Jobs / Workflows
Governance	High — SQL perms + Purview stitching	Unity Catalog (and stop dual-governing forever)
Power BI / semantic models	High politically	Databricks SQL Warehouse endpoints + dataset remaps

Dedicated SQL Pools usually consume most of the calendar. Orchestration and BI consume most of the organizational risk.

3) Program shape: discovery → waves

Do not start with “transpile everything.” Start with a program.

flowchart LR
  A[1 Discovery<br>Lakebridge Profiler] --> B[2 Assessment<br>Analyzer / T-SQL]
  B --> C[3 Design<br>Hybrid lift + modernize]
  C --> D[4 Pilot<br>Lighthouse E2E]
  D --> E[5 Waves<br>Parallel workstreams]

Phase A — Discovery

Inventory what actually runs:

pools, databases, schemas, object counts
top queries / heavy procs by CPU & IO
pipeline inventory (ADF / Synapse)
downstream BI datasets and owners
unused objects (your cheapest scope cut)

Lakebridge Profiler (Databricks Labs) is built for Synapse estate profiling — metadata, utilisation, query patterns, baselines for a TCO conversation.

Phase B — Assessment

Classify the T-SQL estate:

low / medium / high complexity
unsupported constructs
dependency graphs (what breaks if this view moves)

Lakebridge Analyzer helps here. Start waves with low/medium complexity and clear business owners — not the ugliest proc on day one.

Phase C — Design (usually hybrid)

Three strategies show up in every steering committee:

Approach	When it works	Failure mode
Lift-and-shift	Hard deadline off Synapse	You recreate Synapse-shaped pain on a new bill
Big-bang rewrite	Greenfield team, tiny estate	Timeline fantasy
Hybrid	Most real enterprises	Needs discipline: automate bulk convert, modernize in waves

For most Synapse migrations, hybrid wins: automated conversion gets you off Synapse on a schedule; modernization (Delta patterns, Liquid Clustering, Lakeflow, UC) happens after workloads are stable.

Phase D — Pilot (lighthouse)

Pick one end-to-end slice:

source → ingest → transform → gold mart → Power BI

Prove:

storage + UC layout
job standards
reconciliation pack
BI remap
on-call runbook

The pilot’s job is not heroics. It is producing reusable assets for later waves.

Phase E — Waves

Each wave should ship a visible business win, not just “another schema moved.”

4) Four parallel workstreams

After the lighthouse, scale on four tracks together. Sequential “ETL first, BI last” is how stakeholders only feel pain.

flowchart TB
  P[Lighthouse Pilot]
  P --> I[Ingestion<br>ADF/Synapse Pipelines → Lakeflow Connect]
  P --> T[Transformation<br>T-SQL / procs → Databricks SQL / Spark]
  P --> O[Orchestration<br>Schedules → Lakeflow Jobs]
  P --> C[Consumption<br>Power BI → Databricks SQL Warehouse]

Track	From	To
Ingestion	ADF / Synapse Pipelines	Lakeflow Connect / Jobs (or open ingest into Delta)
Transformation	T-SQL / procs / views	Databricks SQL / Spark with clear contracts
Orchestration	Synapse schedules & deps	Lakeflow Jobs / Workflows
Consumption	Power BI / semantic models	Databricks SQL Warehouses

Sequencing tip that builds trust

A BI-aware sequence often works better than pure ETL-first:

Land or expose business-facing marts early (even interim / federation patterns where appropriate).
Put a trusted KPI dataset in front of users.
Modernize the uglier ETL underneath while the business already sees progress.

Enablement is a fifth unofficial track. Synapse-native teams underinvest here and then call the platform “not ready.”

5) Data movement: skip the JDBC tax

For large Dedicated SQL Pool tables, JDBC Synapse → Databricks is how you burn cluster hours and patience.

Prefer:

Synapse CETAS → Parquet on ADLS Gen2 → Databricks Spark read → Delta in Unity Catalog

flowchart LR
  S[Synapse Dedicated SQL Pool] -->|CETAS| L[ADLS Gen2 Parquet]
  L -->|Spark read / Delta write| D[Unity Catalog Delta]
  S -.->|JDBC slow path ❌| D

Why this path wins

Synapse writes columnar Parquet in parallel (CETAS).
ADLS is the shared landing zone both engines already understand.
Databricks reads Parquet with strong scan performance, then writes Delta (ACID, time travel, UC governance).
You avoid row-by-row JDBC serialization.

Example — CETAS from Synapse Dedicated SQL Pool

-- File format
CREATE EXTERNAL FILE FORMAT ParquetFF
WITH (
    FORMAT_TYPE = PARQUET,
    DATA_COMPRESSION = 'org.apache.hadoop.io.compress.SnappyCodec'
);

-- Landing zone on ADLS Gen2
CREATE EXTERNAL DATA SOURCE MigLanding
WITH (
    LOCATION = 'https://<storage>.dfs.core.windows.net/<container>',
    TYPE = HADOOP
);

-- Export a bounded slice (always partition/filter on first runs)
CREATE EXTERNAL TABLE ext.fact_sales_export
WITH (
    LOCATION = '/migration/fact_sales/ds=2026-09-03/',
    DATA_SOURCE = MigLanding,
    FILE_FORMAT = ParquetFF
)
AS
SELECT
    sales_id,
    customer_nk,
    product_nk,
    business_date,
    net_amount,
    quantity
FROM dbo.fact_sales
WHERE business_date >= '2024-01-01';

Example — land as Delta on Databricks

from pyspark.sql import functions as F

src = "abfss://<container>@<storage>.dfs.core.windows.net/migration/fact_sales/"

df = (
    spark.read.parquet(src)
    .withColumn("ingested_at", F.current_timestamp())
)

(
    df.write
    .format("delta")
    .mode("overwrite")
    .option("overwriteSchema", "true")
    .saveAsTable("sales_uc.gold.fact_sales")
)

Physical design: do not cargo-cult Synapse

In Synapse you spent years on:

DISTRIBUTION = HASH(...)
ROUND_ROBIN / REPLICATE
clustered columnstore choices

On Databricks / Delta, those directives do not travel. Mapping HASH(customer_id) to a Delta partition is a classic self-own: high-cardinality keys create tiny-file nightmares.

Prefer:

Liquid Clustering / CLUSTER BY AUTO
predictive optimization / compaction as an operating model
partitions only for coarse, stable predicates (e.g. business_date) when they truly help

Rule: Drop physical Synapse directives. Re-tune with lakehouse primitives after you have real query telemetry — not before.

6) Code conversion: syntax is easy, semantics bite

Automated conversion (Lakebridge transpile and friends) often handles a large share of syntax — commonly cited in the ~80–90% range for bulk translation. Your engineers live in the remaining 10–20%: cursors, dynamic SQL, weird error handling, and semantic mismatches.

Common remaps

T-SQL	Databricks SQL
`GETDATE()`	`CURRENT_TIMESTAMP()`
`ISNULL(a, b)`	`COALESCE(a, b)` / `IFNULL(a, b)`
`LEN(s)`	`LENGTH(s)`
`CHARINDEX(sub, str)`	`LOCATE(sub, str)`
`SELECT TOP 10 ...`	`SELECT ... LIMIT 10`
`CONVERT(INT, col)`	`CAST(col AS INT)`

Semantic trap #1 — case sensitivity

Dedicated SQL Pools are often case-insensitive by collation. Databricks SQL is case-sensitive by default.

-- May silently change results after migration
WHERE status = 'active'

-- Make intent explicit
WHERE LOWER(status) = 'active'

Build this into reconciliation: pick string dimensions and compare distinct counts before you argue about revenue.

Semantic trap #2 — stored procedures

Migrate structure first (parameters, control flow, set-based DML). Optimize second.

Rewrite candidates (do not pretend transpile “finished” them):

cursors / row-by-row loops
heavy dynamic SQL
Synapse-only performance hints
temp-table choreography that exists only to fight distribution skew

Semantic trap #3 — TRY/CATCH vs workflow design

Simple audit logging often stays SQL-shaped. Multi-step compensation and downstream signaling usually belong in Lakeflow Jobs (retries, quarantine tasks, alerting) rather than a 2,000-line proc that tries to be an orchestrator.

7) Dimensional patterns on Delta (SCD2)

Slowly Changing Dimensions are where Synapse estates accumulate snowflake uniqueness. Your goal is not to clone the old proc line-for-line. Preserve the business rule: history + current row queryability.

Example pattern — expire then insert (illustrative)

-- 1) Expire changed current rows
MERGE INTO dim_customer AS t
USING staging_customer AS s
ON t.customer_nk = s.customer_nk AND t.is_current = true
WHEN MATCHED AND (
     t.segment <> s.segment
  OR t.country <> s.country
) THEN UPDATE SET
  t.is_current = false,
  t.valid_to   = current_timestamp();

-- 2) Insert new current versions
INSERT INTO dim_customer (
  customer_nk, segment, country, valid_from, valid_to, is_current
)
SELECT
  s.customer_nk,
  s.segment,
  s.country,
  current_timestamp(),
  CAST(NULL AS TIMESTAMP),
  true
FROM staging_customer s
LEFT ANTI JOIN dim_customer t
  ON t.customer_nk = s.customer_nk
 AND t.is_current = true;

Delta’s ACID behavior makes multi-step SCD flows safer than “hope the warehouse lock ordering holds.” Still: wrap critical loads in jobs with clear quarantine paths.

8) Reconciliation & cutover

Budget reality: validation often costs more calendar time than conversion.

Minimum reconcile pack (every wave)

Row counts by business date / partition
Aggregates — revenue, quantity, distinct business keys
Hash / checksum compares on ordered business-key sets
Tolerance checks for floats / FX / rounded measures
String-dimension sanity (case folding surprises)

Lakebridge Reconcile supports structured comparison patterns across these dimensions.

Cutover pattern that keeps executives calm

Parallel run (Synapse + Databricks) for the lighthouse mart
Side-by-side Power BI validation with the business owner
Flip the dataset connection
Pause Synapse compute (do not delete yet)
Keep rollback for a defined soak window
Only then decommission

Early decommissioning is how “successful migrations” become weekend rollbacks.

9) Power BI / consumption cutover

Technical remap is usually:

Synapse endpoint → Databricks SQL Warehouse
refresh credentials / gateway assumptions
validate DAX measures against reconcile packs

Political remap is harder:

who signs off KPI parity?
which report is the source of truth during dual-run?
what is the communication plan when numbers differ by 0.3% due to case folding or timezone?

Treat consumption as a first-class workstream with named business owners, not a Friday afternoon connection string change.

10) 30 / 60 / 90 plan

Days 1–30 — foundations

[ ] Run Lakebridge Profiler + Analyzer on Dedicated pools
[ ] Cut dead objects from scope (document why)
[ ] Choose one lighthouse: pipeline → gold mart → one Power BI dataset
[ ] Stand up Unity Catalog layout + storage conventions
[ ] Define job naming, environments, and CI basics
[ ] Write the reconcile notebook/job v0

Days 31–60 — prove the path

[ ] CETAS → ADLS → Delta for large facts (no JDBC heroics)
[ ] Transpile + hand-fix the top procs for the lighthouse
[ ] Repoint one BI workspace to Databricks SQL Warehouse
[ ] Parallel run + signed business acceptance
[ ] Capture runbooks: failure modes, quarantine, reruns

Days 61–90 — scale without drama

[ ] Wave 2/3 domains with the same pack
[ ] Orchestration parity on Lakeflow Jobs
[ ] Enablement plan for Synapse-native engineers
[ ] Cost review (warehouse sizing vs old pool bills)
[ ] Only then: formal Synapse decommission proposal

11) Anti-patterns I keep seeing

Transpile-first, assess-never — migrating abandoned procs with confidence.
JDBC for multi-billion-row facts — slow, expensive, fragile.
HASH key → Delta partition — tiny files, sad Spark UI.
Big-bang BI cutover — no dual-run, no owner, no soak.
Recreating every Synapse knob — preserving constraints without preserving reasons.
Skipping enablement — platform is fine; team still thinks in distributions.
Decommission on day one of “green” — optimism is not a rollback strategy.

12) Field rules I will not compromise on

Hybrid > pure lift-and-shift > big-bang rewrite
Assessment before conversion
ADLS + Parquet/Delta over JDBC for big moves
Do not map Synapse HASH keys to partitions
Reconcile every wave
Do not decommission Synapse early
Enablement is a workstream, not a lunch-and-learn
Migrate outcomes (trusted KPIs), not just objects

13) FAQ

Q: Should we move to Fabric instead of Databricks?
A: Different target, overlapping Azure story. This article assumes a Databricks Lakehouse landing zone. If Fabric is in play, treat that as an explicit architecture decision — do not accidentally run two migrations under one slogan.

Q: Can we keep Synapse Serverless for “just a few views”?
A: Sometimes as a bridge. Long-term dual query planes usually recreate the governance tax you are trying to escape.

Q: How perfect must automated conversion be?
A: Good enough to move bulk syntax. Your senior engineers should own semantic diffs, SCD logic, and orchestration.

Q: What is the first metric of a healthy pilot?
A: Not “job duration.” It is signed KPI parity with a business owner plus a rerun runbook a junior can execute.

14) Sources

Databricks — Navigating a Synapse Migration to Databricks
https://www.databricks.com/blog/navigating-synapse-migration-databricks
Databricks — Introducing Lakebridge
https://www.databricks.com/blog/introducing-lakebridge-free-open-data-migration-databricks-sql
Lakebridge docs — Synapse Profiler
https://databrickslabs.github.io/lakebridge/docs/assessment/profiler/synapse/
Lakebridge docs — Transpile
https://databrickslabs.github.io/lakebridge/docs/transpile/
Practical large-table movement discussion (CETAS → ADLS → Delta)
https://medium.com/@prabhakarankanniappan/copy-3-5-billion-rows-from-synapse-to-databricks-in-30-minutes-fda15e0ffca0

ETL vs ELT — Why Notebook Placement Matters Body / Content: Yes — the placement of your Databricks

Fri, 06 Feb 2026 14:09:12 +0000

ETL vs ELT — Why Notebook Placement Matters

Body / Content:

The placement of your Databricks notebooks determines whether your pipeline behaves like ETL or ELT.

If Databricks runs after loading data into the lake, it’s ELT.
If Databricks runs

before loading transformed data, it’s ETL.

Business Analogy:

Clean vegetables before putting them in the fridge → ETL

Dump everything in the fridge and clean later → ELT

Architecture Mapping:

RAW → BRONZE in ADF = Extract + Load (EL)
SILVER → GOLD in Data-bricks = Transform (T)

✅ This makes your pipeline ELT, the modern standard:

Cheap storage Scalable compute Delta Lake supports late
transformations

Notebooks preserve raw data and give flexibility

Cost & Performance Comparison of Data Orchestration on Azure; Data Analysis of Data!

Tue, 03 Feb 2026 03:58:35 +0000

New Project: Cost & Performance Comparison of Data Orchestration on Azure

I recently completed an experimental study comparing runtime and cost efficiency across three common data engineering orchestration approaches on Azure:

• Databricks Notebook Clusters
• Azure Data Factory → Databricks
• Databricks Workflows (multi-notebook pipelines)

Instead of theoretical claims, I used measured execution times, a fixed cluster configuration (Standard_D4s_v3), and then scaled the results to 1 TB of data and 100 hours of processing using clearly stated assumptions.

Key findings:

✅ For small datasets, orchestration overhead dominates (ADF & Workflows are slower)
✅ At TB-scale, all approaches converge in runtime — architecture matters more than speed
✅ Cost differences are marginal at scale, but orchestration choices matter for long-running workloads
✅ Databricks Workflows are ideal for Spark-heavy pipelines
✅ ADF still makes sense when orchestrating multiple Azure services

This project helped me better understand where architectural decisions actually matter — beyond “ADF vs Databricks” debates.

GitHub repo with experimental results, tables, and screenshots:
https://github.com/usman19zafar/Data-Engineering-Cost-Performance-Comparison

Happy to hear thoughts from other data engineers — especially how you decide orchestration strategy in production systems.

Coder Legion - Recent posts tagged databricks

Auto CDF Moved the CDC Tax From Writes to Reads: Delta CDF vs Auto CDF vs Spark 4.2 CHANGES

Three ways to get the changes

The mechanism, in two paragraphs

What I measured

What the numbers say, and what they do not

Gotchas to check before you run UNSET

Decision table

What I would do this week

Hardware & versions

Sources

Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew

Part 1. Design for the restart: foreachBatch, watermarks, and MERGE

Contract 1: foreachBatch is at-least-once

Contract 2: watermark dedupe has a state lease (and you must finish the batch)

Contract 3: MERGE inside foreachBatch is a logical idempotency problem

Part 1 checklist (print this next to the PR)

Part 2. Arrow is not a free pass: applyInPandas vs mapInPandas / mapInArrow

Three APIs, three memory contracts

Skew turns "elegant" into OOM

Prefer batch maps when you do not need whole-group semantics

Decision table

How the two themes connect

Optional diagram (for a cover or inline figure)

References

Azure Synapse to Databricks: A Migration Playbook That Survives Production

Azure Synapse → Databricks: A Migration Playbook That Survives Production

Table of contents

1) Why this migration is harder than the slide deck

2) What you are actually migrating

3) Program shape: discovery → waves

Phase A — Discovery

Phase B — Assessment

Phase C — Design (usually hybrid)

Phase D — Pilot (lighthouse)

Phase E — Waves

4) Four parallel workstreams

Sequencing tip that builds trust

5) Data movement: skip the JDBC tax

Why this path wins

Example — CETAS from Synapse Dedicated SQL Pool

Example — land as Delta on Databricks

Physical design: do not cargo-cult Synapse

6) Code conversion: syntax is easy, semantics bite

Common remaps

Semantic trap #1 — case sensitivity

Semantic trap #2 — stored procedures

Semantic trap #3 — TRY/CATCH vs workflow design

7) Dimensional patterns on Delta (SCD2)

Example pattern — expire then insert (illustrative)

8) Reconciliation & cutover

Minimum reconcile pack (every wave)

Cutover pattern that keeps executives calm

9) Power BI / consumption cutover

10) 30 / 60 / 90 plan

Days 1–30 — foundations

Days 31–60 — prove the path

Days 61–90 — scale without drama

11) Anti-patterns I keep seeing

12) Field rules I will not compromise on

13) FAQ

14) Sources

Suggested LinkedIn teaser (after you publish on DEV.to)

ETL vs ELT — Why Notebook Placement Matters Body / Content: Yes — the placement of your Databricks

ETL vs ELT — Why Notebook Placement Matters

Body / Content:

Cost & Performance Comparison of Data Orchestration on Azure; Data Analysis of Data!

New Project: Cost & Performance Comparison of Data Orchestration on Azure

Part 1. Design for the restart: `foreachBatch`, watermarks, and MERGE

Contract 1: `foreachBatch` is at-least-once

Contract 3: MERGE inside `foreachBatch` is a logical idempotency problem

Part 2. Arrow is not a free pass: `applyInPandas` vs `mapInPandas` / `mapInArrow`