Coder Legion - Recent posts tagged data-engineering

Auto CDF Moved the CDC Tax From Writes to Reads: Delta CDF vs Auto CDF vs Spark 4.2 CHANGES

Mon, 26 Oct 2026 07:12:17 +0000

Originally published on DEV.to: https://dev.to/firfircelik/auto-cdf-moved-the-cdc-tax-from-writes-to-reads-delta-cdf-vs-auto-cdf-vs-spark-42-changes-9bb

On September 1, 2026, Databricks made automatic change data feed (Auto CDF) generally available. The release note says it "removes write-time overhead and makes MERGE and UPDATE operations about 15% faster on tables that you query for changes." Seven weeks earlier, Apache Spark 4.2.0 (July 14) added a SQL CHANGES clause, so reading row-level changes is now a first-class part of the engine, not a Delta-only option.

Most of the takes I have seen stop at "free CDC, faster MERGE, turn it on." My thesis is narrower:

Auto CDF does not remove the cost of change data. It moves it from the writer, who pays once per commit, to every reader, who pays on every read.
Its prerequisite, row tracking, has a write-time cost of its own that nobody puts on the slide.
Spark 4.2's CHANGES is a contract, not an implementation. Whether it works depends on your connector, and on open-source Delta 4.4.1 it does not work out of the box.

So the real question is not "legacy or auto." It is: how many times is each commit read, and how much data does each commit rewrite?

Three ways to get the changes

	Legacy CDF (write time)	Auto CDF (read time)	Spark 4.2 `CHANGES`
Where the work happens	Writer writes change files next to the data	Reader computes changes from row tracking metadata	Connector supplies a changelog, Spark post-processes
Opt in	`delta.enableChangeDataFeed = true` per table	Automatic when requirements are met	Catalog must implement `loadChangelog`
Who can read it	Any Delta reader that implements the `cdc` action of the protocol	Databricks readers only (DBR 19+)	Any Spark 4.2 client, if the connector supports it
Formats	Delta only	Delta with row tracking, Iceberg v3	Whatever the connector exposes
Read API	`readChangeFeed`, `table_changes()`	Same APIs	`SELECT ... CHANGES FROM VERSION`, `.changes()`

FACT (Databricks docs): legacy CDF "materializes changes during table writes", Auto CDF "computes changes during table reads using row lineage metadata", and the two cannot be used together on one table. Migration is a single statement: ALTER TABLE t UNSET TBLPROPERTIES ('delta.enableChangeDataFeed').

The mechanism, in two paragraphs

Legacy CDF. The Delta protocol has a cdc action. On UPDATE, DELETE and MERGE the writer puts the changed rows, with _change_type, into files under _change_data/. The protocol tells readers to use those files when they exist and skip the add and remove actions for that version. Insert-only commits and full partition deletes produce no change files, because the log already says what happened. So the reader of a commit touches roughly the changed rows, nothing more.

Auto CDF. Row tracking gives each row a stable row ID and a row commit version. The Delta protocol explains how: for inserts, the IDs are derived from a baseRowId per file plus the row position, which costs almost nothing. But when a writer copies a row into a new file (the "copy" in copy-on-write), it must preserve the stable ID, so it materializes the row ID and commit version into hidden columns in the data file. At read time, the engine compares row IDs and commit versions across the files a commit removed and added to work out which rows were inserted, updated or deleted. That is real computation over real files, and it happens on every read.

What I measured

I cannot run Databricks' Auto CDF outside Databricks, and I am not going to pretend otherwise. What I can run is open-source Delta Lake 4.4.1 on Spark 4.2.0, which has both halves of the mechanism: legacy CDF, and row tracking with _metadata.row_id and _metadata.row_commit_version.

Setup:

A 10M-row Delta table (about 507 MB of Parquet), deletion vectors off, so MERGE is pure copy-on-write.
Each round: a MERGE that updates about 100,000 rows (1%, keys hashed across the whole id range) and inserts 10,000 new ones. 5 rounds per variant, medians reported, after a discarded warm-up.
Three variants: plain, legacy CDF, row tracking only.
Read side: read the changes of the last commit. For legacy CDF that is readChangeFeed for one version. For row tracking it is a reconstruction I wrote: read the files the commit removed (at version v-1) and added (at version v), and join on row_id and row_commit_version.

I checked the reconstruction against legacy CDF on a table with both features enabled: for a MERGE (200 inserts, 1,986 pre-images, 1,986 post-images) and for a DELETE (100 deletes) the counts matched exactly.

The core of the reconstruction:

after = (spark.read.format("delta").option("versionAsOf", v).load(path)
         .select("*", F.col("_metadata.row_id").alias("rid"),
                 F.col("_metadata.row_commit_version").alias("rcv"),
                 F.col("_metadata.file_path").alias("fp"))
         .where(F.col("fp").isin(added_files)))
changed = after.where(F.col("rcv") == v)          # inserts + post-images
# pre-images: rows in removed files whose rid appears in `changed`
# deletes:    rows in removed files whose rid appears nowhere in `after`

Results, 10M rows, scattered 1% upsert, median of 5 rounds:

Variant	MERGE time	Bytes written per MERGE	Change files per MERGE	Read one commit's changes
Plain	6.03 s (4.75 to 7.69)	505.8 MB	none	n/a
Legacy CDF	6.52 s (6.19 to 7.10)	506.0 MB	10.8 MB	0.21 s
Row tracking only	6.16 s (5.92 to 7.69)	547.6 MB	none	4.73 s (my reconstruction)

A second, partial run of the plain and legacy CDF variants gave 5.64 s and 6.89 s for the MERGE, and 0.18 s for the CDF read.

What the numbers say, and what they do not

1. The write tax of legacy CDF is real but small here, and smaller than my noise. Legacy CDF was slower in both runs (6.52 vs 6.03 s, and 6.89 vs 5.64 s), but the spread inside one variant (4.75 to 7.69 s) is wider than the gap. Databricks measured about 15% on its runtime; my numbers are compatible with that and cannot confirm it. The byte count is solid: 10.8 MB of change files per MERGE, about 2% of what the MERGE rewrote anyway.

2. Row tracking wrote 8% more data on every rewrite. 547.6 MB versus 505.8 MB per MERGE, because every copied row now carries its stable row ID and commit version in hidden columns. That is about four times the size of the CDF change files in this workload. This is the part of the "free" read-time CDF that you pay at write time.

3. Reading changes at read time costs what the commit rewrote, not what it changed. The 1% scattered upsert touched every file, so the commit removed and re-added the whole table. Legacy CDF read 10.8 MB of change files in 0.21 s. My reconstruction had to scan both sides of a 500 MB rewrite to find 100,000 changed rows: 4.73 s, about 22 times slower. On a smaller test (200,000 rows, 8 files) the same scattered upsert rewrote 8 of 8 files, while an upsert clustered on the newest ids rewrote 1 of 8. The layout of your updates decides how expensive read-time CDF is.

What I cannot claim: that Databricks' implementation is as slow as mine. It is very likely smarter, and with deletion vectors a MERGE writes only the changed rows to new files and marks the old ones, which shrinks the set of rows a reader has to compare. My test is the copy-on-write case, which is the worst case for read-time CDF. The mechanism still holds: the read-time cost scales with what a commit touched, and you pay it once per reader per read.

4. CHANGES on open-source Delta 4.4.1 fails by default. With the stock DeltaCatalog, all three of SELECT * FROM t CHANGES FROM VERSION 1, the path-based form, and spark.read.changes("t") returned:

[UNSUPPORTED_FEATURE.CHANGE_DATA_CAPTURE] The feature is not supported:
Catalog spark_catalog does not support Change Data Capture (CDC). SQLSTATE: 0A000

That matches the source. Delta PR #6794 (merged May 2026) adds catalog-driven read-time CDF behind an internal flag that is off by default, requires row tracking, and leaves the streaming side of the new path as a follow-up. I did not test the flag; it is internal and I would not run production on it. Spark 4.2 defines the shape of change data. Your connector decides whether you get any.

Gotchas to check before you run UNSET

All FACT, from the Databricks and Spark docs:

Auto CDF is not supported on tables with row filters or column masks. If your sensitive tables are governed with ABAC or masks, they are exactly the ones that cannot move.
Only Databricks readers can query Auto CDF on Delta, and external Iceberg clients cannot query it on Iceberg v3. Legacy CDF lives in the open Delta protocol. If anything outside Databricks reads your change feed, check before migrating.
It needs Databricks Runtime 19 or above and a table registered in Unity Catalog with row tracking enabled (or Iceberg v3).
Enabling row tracking on an existing table is not free. It assigns IDs to every existing row, can create several new table versions, and concurrent writers fail with MetadataChangedException while it runs. Pause streaming writers first. It also upgrades the writer protocol, so older external Delta clients that do not support those features can no longer write to the table.
Multi-statement transactions: Auto CDF is not supported if the source table was modified during the transaction.
_metadata.row_id and row_commit_version cannot be read while reading the change feed. If your downstream logic wants both, plan for it.
Spark 4.2 streaming netChanges holds back the latest commit. Rows touched only in the latest commit are not emitted until a later commit moves the watermark, and streaming net changes can differ from batch net changes over the same range. Fine for a pipeline, surprising for a low-latency consumer.

Decision table

OPINION, built on the mechanism and my numbers:

Your table looks like	Better choice	Why
Write-heavy, changes read rarely or by one pipeline	Auto CDF	You stop paying on every write for a feed few people read
One commit read by many consumers, often	Legacy CDF, or Auto CDF plus one stream that lands changes in a table	Read-time cost multiplies by readers times reads
Scattered updates on a copy-on-write table	Legacy CDF, or enable deletion vectors and measure	Each commit rewrites most files, so read-time diffing is expensive
Row filters or column masks	Legacy CDF	Auto CDF is not supported there
Change feed read outside Databricks	Legacy CDF	Auto CDF is Databricks-readers only
Iceberg v3 tables on Databricks	Auto CDF	Legacy CDF does not support Iceberg

What I would do this week

Inventory who reads each change feed. Count consumers and their trigger intervals per table. That count is the multiplier on read-time cost, and nobody has it written down.
Look at the update layout. In table history, compare numTargetRowsUpdated against files rewritten for your MERGE operations. A 1% update that rewrites 100% of files is a bad candidate for read-time CDF.
Migrate the boring tables first: write-heavy, one reader, no masks, nothing outside Databricks. Keep legacy CDF on the hot, many-reader tables until you have measured both on your own runtime.
If you fan out, fan out once. One stream reads the change feed and writes a changes table; the other consumers read that. The Databricks docs already show this archive pattern for permanent history.

Auto CDF is a good default for most tables, and a quiet bill for the ones read by five pipelines. CDC is still a cost. The release moved it to the reader, and the bill now goes up with every pipeline that reads the feed.

Hardware & versions

Box: 8 vCPU Intel Xeon VM, 16 GB RAM, Linux 6.12, OpenJDK 21.0.12.
PySpark 4.2.0, delta-spark 4.4.1 (delta-spark_4.2_2.13), local[8], driver memory 8 GB (5 GB in the partial second run), 16 shuffle partitions.
10M rows, 5 columns, about 507 MB Parquet, deletion vectors off. 5 rounds per variant, medians, warm-up discarded.
Same box also hosts another long-running benchmark of mine. During the main run its runner was paused waiting for an idle host, but its services stayed in memory. The partial second run overlapped that benchmark's active phase, so I quote it only as a cross-check. Treat timings as indicative. The byte counts and the correctness check are deterministic.
Measured October 26, 2026. Script, raw logs and pinned versions: cdf_bench gist.

Sources

Databricks September 2026 release notes, "Automatic change data feed is now generally available" (Sep 1, 2026): https://docs.databricks.com/aws/en/release-notes/product/2026/september
Databricks docs, Use change data feed (Auto CDF requirements, migration, limitations): https://docs.databricks.com/aws/en/delta/delta-change-data-feed
Azure Databricks docs, same page: https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed
Databricks docs, Row tracking: https://docs.databricks.com/aws/en/delta/row-tracking
Delta Lake protocol v4.4.1 (Add CDC File, Row Tracking, materialized Row IDs): https://github.com/delta-io/delta/blob/v4.4.1/PROTOCOL.md
Delta Lake docs, Change data feed: https://docs.delta.io/delta-change-data-feed/
Delta Lake docs, Row tracking: https://docs.delta.io/delta-row-tracking/
Delta Lake 4.4.0 release (Spark 4.2 support): https://github.com/delta-io/delta/releases/tag/v4.4.0
Delta PR #6794, Add Changelog support: https://github.com/delta-io/delta/pull/6794
Apache Spark 4.2.0 release notes (Change Data Capture, SPARK-55668): https://spark.apache.org/releases/spark-release-4-2-0.html
Apache Spark news, Spark 4.2.0 released (July 14, 2026): https://spark.apache.org/news/
Spark PR, SPARK-55948 DSv2 CDC connector API and SQL CHANGES clause: https://github.com/apache/spark/pull/54738
Spark 4.2.0 API docs, Changelog: https://spark.apache.org/docs/4.2.0/api/scala/org/apache/spark/sql/connector/catalog/Changelog.html
Spark 4.2.0 API docs, DataStreamReader.changes (streaming netChanges behavior): https://spark.apache.org/docs/4.2.0/api/scala/org/apache/spark/sql/streaming/DataStreamReader.html
My measurements: OSS Delta 4.4.1 on Spark 4.2.0, October 26, 2026 (setup above).

I post shorter breakdowns daily on LinkedIn.

Backtesting live football on Polymarket: 6 traps in a recorded order book

Wed, 14 Oct 2026 11:44:45 +0000

We recorded the order book of every Polymarket football (soccer) market for three weeks and tested two in-play ideas: buying a team right after a late price jump at a level score, and protecting goal bets with a stop-loss. Neither paid. Buying the jump 2 seconds later lost $2.85 per $10 over 837 jumps (t -7.9), and every one of 28 stop settings did worse than holding on the replay. The full results are on our blog and the data is on GitHub.

This post is about the data side: six traps we hit while building the test. Most of them will bite anyone who backtests in-play markets on Polymarket.

1. You have to record the book yourself, then check the recording

Public Polymarket APIs give you trades and the current book, not the book as it was a second ago. For an in-play test you need the best bid and ask around every event, so we recorded the CLOB WebSocket market channel (book, price_change, last_trade_price) with exchange timestamps.

Before using it, check it against on-chain trades. Ours had 99.9% of on-chain trades, and 97.6% of them happened at the best price the recording showed at that moment. If your numbers are much lower, your replay will invent fills.

2. Events arrive out of order

The exchange can send the price_change for a level that a trade just ate before the last_trade_price of that same trade. Read events in arrival order and fills look about 25% worse than they were. Put each trade ahead of the book change it caused before you measure anything.

3. The mid is not a price you can trade

Our first detector watched the mid-price: if a team's mid jumps by 0.25 or more late in a level match, call it a goal. Many of those "goals" were sellers pulling their quotes. The mid doubles, the bid stays where it was, and the spread opens up.

One example from the data: the mid went from 0.24 to 0.505 in seconds, but the ask was 0.78 and the bid about 0.23. A bot buying 2 seconds later paid 0.81. Ten seconds after the jump the ask was back at 0.25, and the match ended in a draw.

Store the ask and the spread, not only the mid. The implied bid is easy to get back if you only kept mid and ask:

def spread_at_jump(mid: float, ask: float) -> float:
    bid = 2 * mid - ask          # implied bid
    return round(ask - bid, 2)   # e.g. mid 0.505, ask 0.78 -> 0.55

In our data 627 of 837 late jumps had a spread of 20 cents or more, and the jumping side won only 36% of those.

4. The "level score" can be fake too

Our rule also needed the draw to be the favorite 30 seconds before the jump. That condition is read from mids as well, so pulled quotes can fake it in a match one team already leads. When all three books of a 1X2 match are two-sided, the three mids add up to about 1.00. Check that before you trust the state:

def book_looks_clean(home_mid: float, draw_mid: float, away_mid: float) -> bool:
    return 0.95 <= home_mid + draw_mid + away_mid <= 1.05

In 51 of the 85 jumps that had a tight spread, the three mids before the jump added up to more than 1.10. One of them was a team that had led for 20 minutes: its bid disappeared for a moment, the score looked level, and the bid coming back looked like a late goal.

5. Daily batch files overlap

We ran the scan one day at a time, and each day file also covered the first hours after midnight UTC. Jumps after midnight landed in two files: 48 duplicates out of 885 raw rows. They did not change the conclusion, but they would have inflated every count and t-statistic. If you split a time series by day, check the joins.

6. The fastest entry is not reachable

Football markets have secondsDelay: 1 in the Gamma API: a taker order waits a second before it can match, and you cannot cancel it during that time. So a result "at +0 s" is not a strategy, it is a plot point. In our test the earliest entries lost the most (-$4.21 per $10 at +0 s, -$2.85 at +2 s), because the first seconds after a jump are when the price is most wrong in the direction you are buying.

The same delay hurts stops. Polymarket has no stop orders, so a stop is a bot watching trades and then market-selling after the delay, paying the taker fee a second time. In live football prices swing several cents within minutes, and a 3-cent stop fired on 86 to 96% of our replayed bets.

Reproduce it

The repo has data/late_goal_jumps.csv (one row per jump, with the asks 0 to 60 seconds after it and the result) and a standard-library script that recomputes every late-goal number:

python3 scripts/summarize_late_goals.py data/late_goal_jumps.csv

METHOD.md describes the recorder, the detection rule and each trap above in more detail.

What this does not show

About three weeks of football, mostly club leagues. Pre-match markets, other sports and stops on strategies whose entries already make money were not tested. Bets in the same match are not independent, so read t values near 2 with care.

Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.

Backtesting a Polymarket copy trading bot without fooling yourself

Sat, 10 Oct 2026 13:28:30 +0000

Copy trading is the most common bot idea on Polymarket: find the wallets that win and buy what they buy. We tested it on five sports (MLB, NFL, college football, League of Legends and Valorant) and it did not survive. The full results are on our blog and the data and script are on GitHub.

This post is about the backtest itself: how to set it up so it can tell you "no", and the one mistake that makes almost every copy bot backtest look better than it would be live.

1. Define a copy you could actually execute

A "copy" in our test is a wallet's first buy of at least $20 in a market, at a price between 0.30 and 0.85. We buy the same outcome for $10 at the wallet's price plus 1 cent, pay the sports taker fee and hold to resolution. Roughly, in code:

def copy_result(price: float, won: bool, stake: float = 10.0, slip: float = 0.01) -> float:
    p = price + slip                      # you never get the whale's exact price
    shares = stake / p
    fee = shares * 0.05 * p * (1 - p)     # Polymarket sports taker fee
    payout = shares if won else 0.0
    return payout - stake - fee

One cent is generous. In fast in-play markets the price moves much further before a copy fills. We also reran MLB at +3 cents: every result went negative, including the top 10.

2. Pick on one period, measure on the next

The rule: rank wallets by their copy results over the two months before the test month, then copy only in the test month. Nothing from the test month touches the selection. We used three rules fixed before the run:

ALL10: every wallet with at least 10 copies (the "average whale")
POS: at least 20 copies, mean above zero, t-statistic of 2 or more
TOP10: the 10 best by t-statistic

MLB, top 10, per $10 copied:

Test month	While picked	Month after
July 2026	+$3.65	+$0.16
August 2026	+$3.06	-$0.04
September 2026	+$2.72	-$0.47
Oct 1-4 2026	+$3.78	-$2.23 (n 7)

The same rule that showed +$3 in the pick window showed about zero the month after.

3. Check whether rank carries over at all

Before tuning thresholds, check if past performance predicts future performance for individual wallets. For MLB wallets with at least 10 copies in both periods, the Spearman correlation between the pick months and the next month was -0.01, +0.15, +0.08 and -0.03. NFL was between -0.09 and +0.03.

With correlations like that, picking "the best" is picking the luckiest. With thousands of active wallets, dozens will have a great two months by chance.

4. Look at the baseline before the winners

The average MLB whale, copied one cent worse than its own price, lost 31 to 63 cents per $10 in every test month. If the baseline is negative, a selection rule has to beat a loss before it makes anything. Check ALL10 first.

5. The trap: wallet labels computed today

This one is easy to miss. Analytics sites (ours included) label wallets: "smart money", "50+ resolved markets", win rate, profit factor. Those labels are computed today, on each wallet's full history, including the months you are about to backtest.

If you pick wallets with today's labels and backtest them on last summer, the selection already knows who survived. Wallets that blew up are not labeled "smart", so the backtest never copies them. It will always look better than it would have live.

We hit a mild version of this ourselves: one filter in our profile screen used each wallet's share of activity in the sport on its full history. We disclosed it in the methodology. It most likely flatters the pools, and they still showed no edge.

Rule of thumb: every feature used to pick a wallet must be computable from data strictly before the pick date.

6. Make it reproducible

The repo has scripts/copy_backtest.py (Python 3.9+, standard library only) that runs the same walk-forward on the OrcaLayer API for any league:

export ORCALAYER_API_KEY=sk_orca_...        # never commit it
python3 scripts/copy_backtest.py --league mlb --start 2026-05-01 --end 2026-09-30 --max-markets 800

To check that the API path and our internal database agree, we ran both on the same 800 MLB games: 161,094 copies in both, every wallet, price and result equal, total -$83,012.60 in both. The script output showed the same picture: top 10 from about +$3 per $10 to between -$1.29 and +$1.06 the month after.

The script needs a Premium API key. If you just want to look at a wallet's record before copying it, the free OrcaLayer MCP server works from Claude without a key.

What this does not show

Five sports, and picking by past results only. Slower markets like politics, where informed money can stay ahead for days, and picking wallets by how they trade rather than how much they made, were not tested.

Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.

Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew

Fri, 09 Oct 2026 13:20:49 +0000

Originally published on DEV.to.

Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a job restarted and silver row counts jumped, or a hot customer key OOMed a worker while the rest of the cluster sat idle.

Neither bug is exotic. Both are documented contracts that happy-path snippets skip.

This article covers two production surfaces Staff and Lead data engineers own on Databricks / PySpark:

Streaming upserts via foreachBatch: at-least-once writes, watermark dedupe state, Delta MERGE idempotency (logical, not txnAppId), empty batches, and why you must fully consume a batch DataFrame.
Pandas Function APIs: groupBy().applyInPandas() loads an entire group into memory and does not honor spark.sql.execution.arrow.maxRecordsPerBatch; mapInPandas / mapInArrow do.

I will mark FACT (from primary docs) vs interpretation (engineering judgment). No invented latency percentages. No "exactly-once foreachBatch" fairy tales.

Primary sources: Databricks foreachBatch, Delta MERGE / streaming upsert, watermarks, state store reader, idempotent Delta writes, pandas function APIs, Apache Arrow in PySpark.

Part 1. Design for the restart: `foreachBatch`, watermarks, and MERGE

Contract 1: `foreachBatch` is at-least-once

FACT (Databricks): foreachBatch() provides only at-least-once write guarantees. You can use the batchId to deduplicate and reason toward exactly-once yourself. You still own end-to-end semantics.

Interpretation: A green micro-batch in the UI is not a guarantee that a retry will be a no-op. Treat every handler as if Spark will call it again with the same batch_id.

APP_ID = "silver-orders-upsert-v1"  # change if you reset the checkpoint


def process_batch(batch_df, batch_id: int) -> None:
    # FACT: Delta sources can deliver empty micro-batches (e.g. OPTIMIZE with
    # nothing to process, or predicate pruning). Code must tolerate them.
    if batch_df.isEmpty():
        return

    valid = batch_df.filter("order_amount > 0 AND customer_id IS NOT NULL")
    invalid = batch_df.filter(
        "NOT (order_amount > 0 AND customer_id IS NOT NULL)"
    )

    # FACT: txnAppId + txnVersion on DataFrameWriter make appends idempotent
    # for retries of the same batch_id. Binding txnVersion to batch_id is the
    # documented pattern.
    (
        valid.write.format("delta")
        .mode("append")
        .option("txnAppId", APP_ID)
        .option("txnVersion", batch_id)
        .saveAsTable("main.silver.orders")
    )

    if not invalid.isEmpty():
        (
            invalid.write.format("delta")
            .mode("append")
            .option("txnAppId", APP_ID)
            .option("txnVersion", batch_id)
            .saveAsTable("main.silver.orders_dlq")
        )


(
    spark.readStream.table("main.bronze.raw_orders")
    .writeStream.foreachBatch(process_batch)
    .option("checkpointLocation", "/Volumes/main/ops/checkpoints/orders")
    .start()
)

FACT (Databricks Delta streaming): If you delete the streaming checkpoint and restart with a new checkpoint, you must provide a different txnAppId. New checkpoints start at batch ID 0; reusing the old app id can make Delta skip writes you intended to redo.

FACT (Databricks): Prefer letting the query fail and letting Lakeflow Jobs / Airflow retry the batch, rather than building silent local retry loops that can duplicate or drop data.

Contract 2: watermark dedupe has a state lease (and you must finish the batch)

FACT (Databricks watermarks): dropDuplicatesWithinWatermark always deduplicates events that arrive within the watermark delay. Events outside the delay might still be deduped, but that is not guaranteed. To guarantee dropping all duplicates, set the watermark delay greater than the maximum timestamp distance between duplicate events.

FACT (Databricks foreachBatch): When you use stateful operators such as dropDuplicatesWithinWatermark, each foreachBatch iteration must completely consume the batch DataFrame. If you only peek (show(2), take a few rows) and leave rows unread, the streaming query can fail on the next batch.

from pyspark.sql.functions import col

deduped = (
    spark.readStream.table("main.bronze.events")
    .withWatermark("event_time", "10 hours")  # > max duplicate lag (FACT guidance)
    .dropDuplicatesWithinWatermark(["event_id"])
)


def bad_handler(batch_df, batch_id: int) -> None:
    # Anti-pattern: partial consume after a stateful operator
    batch_df.show(2)
    # remaining rows never consumed -> next micro-batch can fail


def good_handler(batch_df, batch_id: int) -> None:
    if batch_df.isEmpty():
        return
    # Full action that consumes the plan (write, count, foreach, etc.)
    (
        batch_df.write.format("delta")
        .mode("append")
        .option("txnAppId", "events-dedupe-v1")
        .option("txnVersion", batch_id)
        .saveAsTable("main.silver.events")
    )


(
    deduped.writeStream.foreachBatch(good_handler)
    .option("checkpointLocation", "/Volumes/main/ops/checkpoints/events")
    .start()
)

Debugging tip (FACT): You can inspect Structured Streaming state with the statestore reader. Metadata can show an operator named dedupeWithinWatermark; query that operatorId when you need to see what the lease still holds. See Read Structured Streaming state information.

Contract 3: MERGE inside `foreachBatch` is a logical idempotency problem

FACT (Delta Lake): You must use foreachBatch for Delta Lake merge operations from Structured Streaming. Restarts can apply the same batch multiple times, so your merge statement inside foreachBatch must be idempotent.

FACT (Databricks / Delta writer options): txnAppId and txnVersion are DataFrameWriter options for idempotent writes (append / save). Do not assume they wrap DeltaTable.merge(...). Design MERGE so re-running the same micro-batch yields the same table state.

FACT (Delta): A merge can fail if multiple source rows match the same target row (ambiguous update). For CDC, preprocess to keep the latest change per key before merging.

FACT (Delta): MERGE often reads the source multiple times. That can inflate reported streaming input rates. Cache the micro-batch DataFrame before merge and unpersist after if this becomes a bottleneck.

from delta.tables import DeltaTable
from pyspark.sql import functions as F

target = DeltaTable.forName(spark, "main.silver.customers")


def upsert_cdc(batch_df, batch_id: int) -> None:
    if batch_df.isEmpty():
        return

    # Collapse to latest change per business key (FACT: avoid multi-match MERGE)
    latest = (
        batch_df.select(
            "customer_id",
            F.struct("event_time", "payload", "deleted").alias("chg"),
        )
        .groupBy("customer_id")
        .agg(F.max("chg").alias("chg"))
        .select("customer_id", "chg.*")
    )

    # FACT: MERGE may scan the source more than once -> cache the batch
    latest = latest.cache()
    try:
        (
            target.alias("t")
            .merge(latest.alias("s"), "t.customer_id = s.customer_id")
            .whenMatchedDelete(condition="s.deleted = true")
            .whenMatchedUpdate(
                set={"payload": "s.payload", "updated_at": "s.event_time"}
            )
            .whenNotMatchedInsert(
                condition="s.deleted = false",
                values={
                    "customer_id": "s.customer_id",
                    "payload": "s.payload",
                    "updated_at": "s.event_time",
                },
            )
            .execute()
        )
    finally:
        latest.unpersist()


(
    spark.readStream.table("main.bronze.customer_cdc")
    .writeStream.foreachBatch(upsert_cdc)
    .option("checkpointLocation", "/Volumes/main/ops/checkpoints/customer_cdc")
    .start()
)

Interpretation: "Idempotent MERGE" here means latest-wins on a stable key (and deletes keyed the same way), not "Spark will skip the merge via txn metadata." If the same CDC events replay, the table should converge to the same rows.

Managed alternative (context, not a product pitch): Databricks processing-guarantees docs note that managed Delta-to-Delta flows and AUTO CDC can give stronger defaults; a custom foreachBatch edge is still at-least-once unless you add explicit idempotent write or dedupe logic. Use AUTO CDC when it fits; when you hand-roll MERGE, you own the restart contract above.

Part 1 checklist (print this next to the PR)

Checkpoint	Why
Empty-batch safe	Delta sources can emit empty micro-batches
Full consume after stateful ops	Partial `show`/`take` can fail the next batch
Watermark > max duplicate lag	Out-of-watermark dedupe is not guaranteed
Append path uses `txnAppId` + `txnVersion=batch_id`	Writer-level idempotency on retry
New checkpoint => new `txnAppId`	Else batch 0 collisions skip writes
MERGE is latest-per-key / safe to re-run	txn options do not wrap `DeltaTable.merge`
Cache around MERGE when metrics / cost hurt	MERGE multi-pass over the micro-batch
Fail fast to the orchestrator	Local retry loops are how you invent exactly-once bugs

Part 2. Arrow is not a free pass: `applyInPandas` vs `mapInPandas` / `mapInArrow`

Python DE craft is not only streaming. The other Tuesday failure mode is "we used pandas because Arrow is fast" and then a skewed group killed the executor.

Three APIs, three memory contracts

FACT (Databricks pandas function APIs / Apache Spark):

API	What you write	Memory / batching contract
`groupBy().applyInPandas(f, schema)`	One pandas DataFrame per group	All rows for the group load into memory before `f` runs. `spark.sql.execution.arrow.maxRecordsPerBatch` is not applied. Skew => OOM risk.
`mapInPandas(f, schema)`	Iterator of pandas DataFrames (batches)	Batch size controlled by `spark.sql.execution.arrow.maxRecordsPerBatch` (default 10,000 rows in Spark docs). Output length can differ from input.
`mapInArrow(f, schema)`	Iterator of `pyarrow.RecordBatch`	Same Arrow batch sizing idea; stay in Arrow without pandas round-trip when you can.

Cogrouped applyInPandas has the same whole-cogroup-in-memory warning.

Skew turns "elegant" into OOM

# Looks clean. Under a hot customer_id it is a bomb.
def per_customer_features(pdf):
    # pdf = ALL rows for one customer_id
    pdf = pdf.sort_values("event_time")
    pdf["rolling_amt"] = pdf["amount"].rolling(50, min_periods=1).sum()
    return pdf


# FACT: entire group must fit in worker memory; maxRecordsPerBatch does NOT help
out = (
    events.groupBy("customer_id")
    .applyInPandas(
        per_customer_features,
        schema="customer_id string, event_time timestamp, amount double, rolling_amt double",
    )
)

Interpretation: Use applyInPandas when the group is a true unit of work and you can bound group size (filter, sample hot keys, pre-aggregate, salt, or split whales). Do not use it as a generic "run pandas on Spark" hammer.

Prefer batch maps when you do not need whole-group semantics

spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "5000")


def score_batches(iterator):
    # init once per Python worker task
    model = load_light_model()  # your code; keep it picklable / available on workers
    for pdf in iterator:
        pdf = pdf.copy()
        pdf["score"] = model.predict(pdf[["f1", "f2"]])
        yield pdf[["id", "score"]]


scored = events.mapInPandas(
    score_batches,
    schema="id long, score double",
)

Or stay in Arrow when pandas is unnecessary:

import pyarrow as pa
import pyarrow.compute as pc


def filter_arrow(iterator):
    for batch in iterator:
        mask = pc.greater(batch.column("amount"), 0)
        yield batch.filter(mask)


positive = events.mapInArrow(filter_arrow, schema=events.schema)

Decision table

Situation	Prefer	Watch out
Need whole-group window / asof / custom group ML	`applyInPandas`	Hot keys; no Arrow batch limit; cogroup same issue
Row/batch transform, arbitrary output length	`mapInPandas`	Tune `maxRecordsPerBatch` down if wide rows
CPU vector ops, avoid pandas copies	`mapInArrow`	Arrow type coverage / compute familiarity
Series-to-series column expression	classic `pandas_udf`	Output length must match input for many UDF shapes

FACT (Spark Arrow guide): Lower maxRecordsPerBatch when column count is large; Arrow batches can pressure JVM memory during conversion.

How the two themes connect

Both failures are the same Staff DE mistake: assuming the platform's happy path equals your failure domain.

Streaming: you assumed merge + checkpoint = exactly-once. Docs say at-least-once + you make MERGE idempotent; writer txn options are a different tool.
Pandas APIs: you assumed "Arrow optimized" = bounded memory. Docs say group maps ignore the batch limit.

Own the contract in the PR template, not in the postmortem.

Optional diagram (for a cover or inline figure)

Two panels:

Restart timeline: micro-batch N fails mid-MERGE -> orchestrator retries batch N -> arrows labeled "append: txn skip" vs "MERGE: must be latest-wins safe".
Skew funnel: many small customer_id groups sail through applyInPandas; one whale group expands to "all rows in RAM" with a red X; side path routes whales to salted / pre-aggregated / mapInPandas path.

References

I'm Firat Celik, Lead Data Engineer. I write about production data platforms, PySpark/Delta, and the boring contracts that keep pipelines honest.

Fabric Runtime 2.0 is becoming the default. Here is what actually breaks.

Fri, 09 Oct 2026 13:20:30 +0000

Originally published on DEV.to.

Fabric Runtime 2.0 is becoming the default. Here is what actually breaks.

Late September 2026 is when Fabric Runtime 2.0 stops being an opt-in experiment and becomes the path of least resistance for new workspaces and environment items. If you treat that as a release note, you will discover the breakages in production.

I work as a Lead Data Engineer on Azure, Fabric, Databricks, and Spark. This is the migration checklist I wish teams ran before the default flipped under them.

Docs to keep open while you read:

1) Hook: the default is the migration

Microsoft's Runtime 2.0 page is blunt: GA and ready for production, but not the default yet. You opt in. The stated plan is to make Runtime 2.0 the default UX selection and the default for new workspaces and new environment items in late September 2026.

Today is 28 September 2026. That window is now.

Existing workspaces, jobs, and Environment items stay on whatever you configured. The trap is quieter. Every new workspace for a pilot, every Environment someone creates because "defaults are fine," starts landing on Spark 4.1 / Delta 4.2 / Python 3.13 once the plan executes.

Treat the flip as an incident rehearsal date. Pin what you care about. Validate 2.0 on a scoped Environment. Do not discover Python wheel breakage through a failed Monday job.

Runtime 1.3 still has runway. Secondary writeups put Microsoft's LTS window for 1.3 roughly from early October 2026 through March 2027. Verify against current lifecycle docs before you commit a program date. LTS means supported, not "ignore migration."

2) Wrong assumption: GA means notebooks just run

Runtime 1.3 to Runtime 2.0 is not a patch bump. It is a stack jump on nearly every component:

Component	Runtime 1.3 (typical)	Runtime 2.0
Apache Spark	3.5	4.1
Delta Lake	3.2	4.2
Python	3.11	3.13
Java	11	21
Scala	2.12	2.13
OS	(prior Fabric OS)	Azure Linux 3.0 (Mariner 3.0)

Industry writeups also call out Delta protocol defaults moving higher (reader/writer versions, Deletion Vectors in the 2.0 path). Treat protocol as a first-class migration risk, not a footnote. See Microsoft's Delta Lake table format interoperability guidance for what cross-experience readers can actually open.

Practical fallout I care about:

Python 3.11 -> 3.13. Environment-level libraries may migrate automatically. Validation does not. Custom wheels and pinned packages are your problem. Session-scoped %pip install and notebook Resources are a different blast radius than Environment items.
Scala 2.12 -> 2.13 / Java 11 -> 21. JARs and connectors that "worked forever" on 1.3 deserve a dual-run, not hope.
SparkR is deprecated in Spark 4.x and may be removed later. If you still have SparkR notebooks, the clock is already running.

GA means Microsoft ships it for production use. It does not mean your Environment's library graph survives the jump without you watching.

3) Wrong assumption: NEE = free 6x

Runtime 2.0 includes the Native Execution Engine (NEE). Microsoft's docs describe it as a vectorized path based on Apache Gluten (offload middle layer) and Velox (C++ engine from Meta). Supported operators leave the JVM and run columnar with SIMD-style acceleration on Parquet, Delta, and (now) CSV. Microsoft cites up to six times faster vs open-source Spark on TPC-DS. That is a vendor benchmark on a specific workload family. It is a ceiling story, not your median job.

How you enable it:

Environment -> Spark compute -> Acceleration -> enable native execution engine (preferred UX path), or
Spark conf: spark.native.enabled (also usable per notebook / SJD via %%configure)

Hard limits from the docs that matter in production:

Structured streaming is not supported. Use streaming? Plan for JVM (or keep NEE off for those jobs).
Unsupported ops fall back to classic Spark. Fallback is automatic. Your job keeps running. Your SLA may not.
Every boundary between native columnar and JVM row-oriented execution pays a conversion tax. That is the part marketing slides skip.
Fabric Spark Advisor can surface fallback alerts in notebook cell output. Use them. Do not ignore green "job succeeded" when Advisor is screaming about offload misses.

NEE also preserves AQE, cost-based rewrites, column pruning, and predicate pushdown when operators do offload. Good. That does not help if half the plan is bouncing in and out of Velox.

Blunt take: a vectorized engine that falls back on half your plan is not acceleration. It is a tax you opted into because a dashboard said "up to 6x."

4) How to see the truth in the plan

Do not argue with anecdotes. Read the plan.

In Spark UI / History Server, look for node names ending in:

*Transformer (for example ProjectExecTransformer, BroadcastHashJoinExecTransformer, RollUpHashAggregateTransformer)
*NativeFileScan
VeloxColumnarToRowExec

Those suffixes mean native offload happened for that node. Light-blue JVM nodes next to green native nodes (Microsoft's Gluten SQL / DataFrame tab uses that color language) are your fallback map. Every VeloxColumnarToRowExec (and the reverse path) is a place where you paid for format conversion.

Also useful: df.explain(), Fabric Spark Advisor fallback alerts, and the Gluten SQL / DataFrame tab's native-vs-JVM node counts.

Surgical disable for one cell (unsupported op, correctness edge, or A/B):

spark.conf.set('spark.native.enabled', 'false')

Re-enable afterward if later cells should use NEE again. Spark runs cells in order. Conf sticks.

5) Delta 4.x trap for multi-workload tables

This is the silent one.

Microsoft is explicit: Delta Lake 4.2-specific features are experimental and only work on Spark experiences (notebooks and Spark Job Definitions). If the same Delta tables must be used across multiple Fabric workloads, do not enable those features.

Why it hurts: SQL analytics endpoint, Power BI Direct Lake, and other non-Spark readers do not all speak the same protocol / feature set. Protocol upgrades (higher reader/writer versions, Deletion Vectors, experimental checkpoints) are effectively one-way doors. Recreate the table is the recovery path, not "uncheck the box." Shared bronze/silver tables that feed Spark and Direct Lake and SQL endpoint are exactly where someone enables a shiny 4.x feature on a Friday and breaks Monday's semantic model.

Rule I use: default to the most constrained reader in the estate. If Direct Lake or the SQL endpoint must read it, keep features inside what those readers support. Save experimental Delta 4.x toys for Spark-only sandboxes.

6) Practical migration pattern

This is the runbook shape I actually use.

1. Inventory

Workspaces and Environment items: which runtime are they on?
Jobs that create new Environments without pinning
Libraries: Environment wheels vs session %pip
Tables read by more than one Fabric experience
Streaming jobs (NEE: leave off or isolate)
SparkR notebooks (debt clock)

2. Baseline on 1.3

Duration, shuffle bytes, spill, failure rate, data quality checks
Capture plans for critical queries before you change anything
Note current Delta protocol / table features on shared tables

3. Dual-run on 2.0

Scoped Environment item first (surgical), not whole workspace
Same input snapshot, same checks, compare plans and SLA
Turn NEE on only after the JVM path is green on 2.0
Kill vanity actions that force row materialization (yes, that includes habit .show() in hot paths during timing runs)

4. Rollback in the runbook before you need it

Runtime: pin Environment / workspace back to 1.3 while LTS covers you
NEE only:

spark.conf.set('spark.native.enabled', 'false')

Put both in the on-call doc. A rollback you invent at 02:00 is not a rollback.

7) CI/CD angle: pin the runtime like a toolchain

You would not let python float to whatever the image shipped last Tuesday without a lockfile. Treat Fabric runtime the same way.

Pin runtime version on Environment items that production notebooks and SJDs attach to
Fail the pipeline if an Environment publishes without an explicit runtime
Gate promotion: 1.3 baseline metrics vs 2.0 dual-run metrics must be compared, not vibes
Separate "NEE enabled" as its own promotion flag. Runtime upgrade and native offload are two changes. Ship them as two changes.
Use Fabric runtime release channels (preview) to rehearse upcoming defaults on custom pools. Do not park production there.

If CI deploys notebook JSON and never asserts runtime + Acceleration settings, you are continuous-delivering entropy.

8) Close: benchmarks sell the ceiling, production pays the median

Workload type	Enable NEE?	Watch for...
Large Parquet/Delta batch ETL, heavy agg/join	Yes, after dual-run	Fallback density, conversion boundaries, Advisor alerts
Interactive lakehouse SQL / notebook exploration	Often yes	Date type mismatches on filters, wide UDF usage
Structured streaming	No (unsupported)	Accidental enable via Environment inheritance
JSON/XML heavy paths	Limited / often no	Format falls back to JVM; measure before celebrating
Shared Delta tables (Spark + SQL endpoint / Direct Lake)	NEE optional; Delta 4.x experimental features: no	Protocol / feature lockout of non-Spark readers
SparkR or exotic Scala libs	Maybe later	Language deprecation and binary incompat first

Takeaways

Pin Runtime 1.3 on anything you cannot afford to surprise, then opt into 2.0 on a scoped Environment before the default teaches your newest workspace the hard way.
Treat NEE as a plan shape problem, not a toggle. *Transformer and VeloxColumnarToRowExec tell the truth; "up to 6x" does not.
Never enable experimental Delta 4.x features on multi-experience tables. Shared tables inherit the weakest reader's constraints.

Benchmarks sell the ceiling. Production pays the median. Migrate like you believe that.

What is your scarier unpaid bill right now: Python 3.13 libraries, NEE fallback thrash, or a Delta feature someone enabled on a table Direct Lake still needs to read?

Sources

Runtime 2.0 in Fabric (Microsoft Learn)
Native execution engine for Fabric Data Engineering (Microsoft Learn)
Context (not a substitute for the docs): Fabric Runtime 2.0 two breakages (Rajaniesh Kaushikk, 2026-09-23)

Azure Synapse to Databricks: A Migration Playbook That Survives Production

Fri, 09 Oct 2026 13:19:33 +0000

Originally published on DEV.to.

Azure Synapse → Databricks: A Migration Playbook That Survives Production

A field-style guide for Lead / Senior Data Engineers moving Dedicated SQL Pools, pipelines, and BI off Synapse without burning trust, budget, or the on-call team.

Why this migration is harder than the slide deck
What you are actually migrating
Program shape: discovery → waves
Four parallel workstreams
Data movement: skip the JDBC tax
Code conversion: syntax is easy, semantics bite
Dimensional patterns on Delta (SCD2)
Reconciliation & cutover
Power BI / consumption cutover
30 / 60 / 90 plan
Anti-patterns I keep seeing
Field rules I will not compromise on
FAQ
Sources

1) Why this migration is harder than the slide deck

If you have lived in Azure Synapse for years, the marketing line sounds simple:

Move to Databricks. Unity Catalog. Lakehouse. Done.

In production programs, that is not the hard part.

The hard part is that Synapse is not one system. Under one brand you usually own years of Dedicated SQL Pool logic, ADF/Synapse Pipelines, serverless SQL over the lake, Spark pools, permission models, and Power BI datasets wired to Synapse endpoints.

Treat that estate as one workstream with one deadline, and the schedule slips. Every time.

I work as a Lead Data Engineer across Azure Synapse, Microsoft Fabric, Azure Data Lake, Databricks/Spark, SQL/T-SQL, and Power BI. This playbook is the version I want in the room before the first converted stored procedure lands in production.

Thesis: Migration success is not “all notebooks on Databricks.”
It is decision-ready data, trusted KPIs, and a platform your team can operate.

2) What you are actually migrating

Synapse piece	Typical complexity	What “done” looks like on Databricks
Dedicated SQL Pools	Highest — procs, distributions, CCI, years of tuning	Databricks SQL / Spark jobs, Delta tables, Liquid Clustering / predictive optimization
Serverless SQL	Medium — views & external tables over ADLS	Views / tables in Unity Catalog over the same lake paths (often simpler)
Spark pools	Lower — already Spark-shaped	Databricks Runtime notebooks / jobs (often few changes)
ADF / Synapse Pipelines	High — orchestration + connectors	Lakeflow Connect / Lakeflow Jobs / Workflows
Governance	High — SQL perms + Purview stitching	Unity Catalog (and stop dual-governing forever)
Power BI / semantic models	High politically	Databricks SQL Warehouse endpoints + dataset remaps

Dedicated SQL Pools usually consume most of the calendar. Orchestration and BI consume most of the organizational risk.

3) Program shape: discovery → waves

Do not start with “transpile everything.” Start with a program.

flowchart LR
  A[1 Discovery<br>Lakebridge Profiler] --> B[2 Assessment<br>Analyzer / T-SQL]
  B --> C[3 Design<br>Hybrid lift + modernize]
  C --> D[4 Pilot<br>Lighthouse E2E]
  D --> E[5 Waves<br>Parallel workstreams]

Phase A — Discovery

Inventory what actually runs:

pools, databases, schemas, object counts
top queries / heavy procs by CPU & IO
pipeline inventory (ADF / Synapse)
downstream BI datasets and owners
unused objects (your cheapest scope cut)

Lakebridge Profiler (Databricks Labs) is built for Synapse estate profiling — metadata, utilisation, query patterns, baselines for a TCO conversation.

Phase B — Assessment

Classify the T-SQL estate:

low / medium / high complexity
unsupported constructs
dependency graphs (what breaks if this view moves)

Lakebridge Analyzer helps here. Start waves with low/medium complexity and clear business owners — not the ugliest proc on day one.

Phase C — Design (usually hybrid)

Three strategies show up in every steering committee:

Approach	When it works	Failure mode
Lift-and-shift	Hard deadline off Synapse	You recreate Synapse-shaped pain on a new bill
Big-bang rewrite	Greenfield team, tiny estate	Timeline fantasy
Hybrid	Most real enterprises	Needs discipline: automate bulk convert, modernize in waves

For most Synapse migrations, hybrid wins: automated conversion gets you off Synapse on a schedule; modernization (Delta patterns, Liquid Clustering, Lakeflow, UC) happens after workloads are stable.

Phase D — Pilot (lighthouse)

Pick one end-to-end slice:

source → ingest → transform → gold mart → Power BI

Prove:

storage + UC layout
job standards
reconciliation pack
BI remap
on-call runbook

The pilot’s job is not heroics. It is producing reusable assets for later waves.

Phase E — Waves

Each wave should ship a visible business win, not just “another schema moved.”

4) Four parallel workstreams

After the lighthouse, scale on four tracks together. Sequential “ETL first, BI last” is how stakeholders only feel pain.

flowchart TB
  P[Lighthouse Pilot]
  P --> I[Ingestion<br>ADF/Synapse Pipelines → Lakeflow Connect]
  P --> T[Transformation<br>T-SQL / procs → Databricks SQL / Spark]
  P --> O[Orchestration<br>Schedules → Lakeflow Jobs]
  P --> C[Consumption<br>Power BI → Databricks SQL Warehouse]

Track	From	To
Ingestion	ADF / Synapse Pipelines	Lakeflow Connect / Jobs (or open ingest into Delta)
Transformation	T-SQL / procs / views	Databricks SQL / Spark with clear contracts
Orchestration	Synapse schedules & deps	Lakeflow Jobs / Workflows
Consumption	Power BI / semantic models	Databricks SQL Warehouses

Sequencing tip that builds trust

A BI-aware sequence often works better than pure ETL-first:

Land or expose business-facing marts early (even interim / federation patterns where appropriate).
Put a trusted KPI dataset in front of users.
Modernize the uglier ETL underneath while the business already sees progress.

Enablement is a fifth unofficial track. Synapse-native teams underinvest here and then call the platform “not ready.”

5) Data movement: skip the JDBC tax

For large Dedicated SQL Pool tables, JDBC Synapse → Databricks is how you burn cluster hours and patience.

Prefer:

Synapse CETAS → Parquet on ADLS Gen2 → Databricks Spark read → Delta in Unity Catalog

flowchart LR
  S[Synapse Dedicated SQL Pool] -->|CETAS| L[ADLS Gen2 Parquet]
  L -->|Spark read / Delta write| D[Unity Catalog Delta]
  S -.->|JDBC slow path ❌| D

Why this path wins

Synapse writes columnar Parquet in parallel (CETAS).
ADLS is the shared landing zone both engines already understand.
Databricks reads Parquet with strong scan performance, then writes Delta (ACID, time travel, UC governance).
You avoid row-by-row JDBC serialization.

Example — CETAS from Synapse Dedicated SQL Pool

-- File format
CREATE EXTERNAL FILE FORMAT ParquetFF
WITH (
    FORMAT_TYPE = PARQUET,
    DATA_COMPRESSION = 'org.apache.hadoop.io.compress.SnappyCodec'
);

-- Landing zone on ADLS Gen2
CREATE EXTERNAL DATA SOURCE MigLanding
WITH (
    LOCATION = 'https://<storage>.dfs.core.windows.net/<container>',
    TYPE = HADOOP
);

-- Export a bounded slice (always partition/filter on first runs)
CREATE EXTERNAL TABLE ext.fact_sales_export
WITH (
    LOCATION = '/migration/fact_sales/ds=2026-09-03/',
    DATA_SOURCE = MigLanding,
    FILE_FORMAT = ParquetFF
)
AS
SELECT
    sales_id,
    customer_nk,
    product_nk,
    business_date,
    net_amount,
    quantity
FROM dbo.fact_sales
WHERE business_date >= '2024-01-01';

Example — land as Delta on Databricks

from pyspark.sql import functions as F

src = "abfss://<container>@<storage>.dfs.core.windows.net/migration/fact_sales/"

df = (
    spark.read.parquet(src)
    .withColumn("ingested_at", F.current_timestamp())
)

(
    df.write
    .format("delta")
    .mode("overwrite")
    .option("overwriteSchema", "true")
    .saveAsTable("sales_uc.gold.fact_sales")
)

Physical design: do not cargo-cult Synapse

In Synapse you spent years on:

DISTRIBUTION = HASH(...)
ROUND_ROBIN / REPLICATE
clustered columnstore choices

On Databricks / Delta, those directives do not travel. Mapping HASH(customer_id) to a Delta partition is a classic self-own: high-cardinality keys create tiny-file nightmares.

Prefer:

Liquid Clustering / CLUSTER BY AUTO
predictive optimization / compaction as an operating model
partitions only for coarse, stable predicates (e.g. business_date) when they truly help

Rule: Drop physical Synapse directives. Re-tune with lakehouse primitives after you have real query telemetry — not before.

6) Code conversion: syntax is easy, semantics bite

Automated conversion (Lakebridge transpile and friends) often handles a large share of syntax — commonly cited in the ~80–90% range for bulk translation. Your engineers live in the remaining 10–20%: cursors, dynamic SQL, weird error handling, and semantic mismatches.

Common remaps

T-SQL	Databricks SQL
`GETDATE()`	`CURRENT_TIMESTAMP()`
`ISNULL(a, b)`	`COALESCE(a, b)` / `IFNULL(a, b)`
`LEN(s)`	`LENGTH(s)`
`CHARINDEX(sub, str)`	`LOCATE(sub, str)`
`SELECT TOP 10 ...`	`SELECT ... LIMIT 10`
`CONVERT(INT, col)`	`CAST(col AS INT)`

Semantic trap #1 — case sensitivity

Dedicated SQL Pools are often case-insensitive by collation. Databricks SQL is case-sensitive by default.

-- May silently change results after migration
WHERE status = 'active'

-- Make intent explicit
WHERE LOWER(status) = 'active'

Build this into reconciliation: pick string dimensions and compare distinct counts before you argue about revenue.

Semantic trap #2 — stored procedures

Migrate structure first (parameters, control flow, set-based DML). Optimize second.

Rewrite candidates (do not pretend transpile “finished” them):

cursors / row-by-row loops
heavy dynamic SQL
Synapse-only performance hints
temp-table choreography that exists only to fight distribution skew

Semantic trap #3 — TRY/CATCH vs workflow design

Simple audit logging often stays SQL-shaped. Multi-step compensation and downstream signaling usually belong in Lakeflow Jobs (retries, quarantine tasks, alerting) rather than a 2,000-line proc that tries to be an orchestrator.

7) Dimensional patterns on Delta (SCD2)

Slowly Changing Dimensions are where Synapse estates accumulate snowflake uniqueness. Your goal is not to clone the old proc line-for-line. Preserve the business rule: history + current row queryability.

Example pattern — expire then insert (illustrative)

-- 1) Expire changed current rows
MERGE INTO dim_customer AS t
USING staging_customer AS s
ON t.customer_nk = s.customer_nk AND t.is_current = true
WHEN MATCHED AND (
     t.segment <> s.segment
  OR t.country <> s.country
) THEN UPDATE SET
  t.is_current = false,
  t.valid_to   = current_timestamp();

-- 2) Insert new current versions
INSERT INTO dim_customer (
  customer_nk, segment, country, valid_from, valid_to, is_current
)
SELECT
  s.customer_nk,
  s.segment,
  s.country,
  current_timestamp(),
  CAST(NULL AS TIMESTAMP),
  true
FROM staging_customer s
LEFT ANTI JOIN dim_customer t
  ON t.customer_nk = s.customer_nk
 AND t.is_current = true;

Delta’s ACID behavior makes multi-step SCD flows safer than “hope the warehouse lock ordering holds.” Still: wrap critical loads in jobs with clear quarantine paths.

8) Reconciliation & cutover

Budget reality: validation often costs more calendar time than conversion.

Minimum reconcile pack (every wave)

Row counts by business date / partition
Aggregates — revenue, quantity, distinct business keys
Hash / checksum compares on ordered business-key sets
Tolerance checks for floats / FX / rounded measures
String-dimension sanity (case folding surprises)

Lakebridge Reconcile supports structured comparison patterns across these dimensions.

Cutover pattern that keeps executives calm

Parallel run (Synapse + Databricks) for the lighthouse mart
Side-by-side Power BI validation with the business owner
Flip the dataset connection
Pause Synapse compute (do not delete yet)
Keep rollback for a defined soak window
Only then decommission

Early decommissioning is how “successful migrations” become weekend rollbacks.

9) Power BI / consumption cutover

Technical remap is usually:

Synapse endpoint → Databricks SQL Warehouse
refresh credentials / gateway assumptions
validate DAX measures against reconcile packs

Political remap is harder:

who signs off KPI parity?
which report is the source of truth during dual-run?
what is the communication plan when numbers differ by 0.3% due to case folding or timezone?

Treat consumption as a first-class workstream with named business owners, not a Friday afternoon connection string change.

10) 30 / 60 / 90 plan

Days 1–30 — foundations

[ ] Run Lakebridge Profiler + Analyzer on Dedicated pools
[ ] Cut dead objects from scope (document why)
[ ] Choose one lighthouse: pipeline → gold mart → one Power BI dataset
[ ] Stand up Unity Catalog layout + storage conventions
[ ] Define job naming, environments, and CI basics
[ ] Write the reconcile notebook/job v0

Days 31–60 — prove the path

[ ] CETAS → ADLS → Delta for large facts (no JDBC heroics)
[ ] Transpile + hand-fix the top procs for the lighthouse
[ ] Repoint one BI workspace to Databricks SQL Warehouse
[ ] Parallel run + signed business acceptance
[ ] Capture runbooks: failure modes, quarantine, reruns

Days 61–90 — scale without drama

[ ] Wave 2/3 domains with the same pack
[ ] Orchestration parity on Lakeflow Jobs
[ ] Enablement plan for Synapse-native engineers
[ ] Cost review (warehouse sizing vs old pool bills)
[ ] Only then: formal Synapse decommission proposal

11) Anti-patterns I keep seeing

Transpile-first, assess-never — migrating abandoned procs with confidence.
JDBC for multi-billion-row facts — slow, expensive, fragile.
HASH key → Delta partition — tiny files, sad Spark UI.
Big-bang BI cutover — no dual-run, no owner, no soak.
Recreating every Synapse knob — preserving constraints without preserving reasons.
Skipping enablement — platform is fine; team still thinks in distributions.
Decommission on day one of “green” — optimism is not a rollback strategy.

12) Field rules I will not compromise on

Hybrid > pure lift-and-shift > big-bang rewrite
Assessment before conversion
ADLS + Parquet/Delta over JDBC for big moves
Do not map Synapse HASH keys to partitions
Reconcile every wave
Do not decommission Synapse early
Enablement is a workstream, not a lunch-and-learn
Migrate outcomes (trusted KPIs), not just objects

13) FAQ

Q: Should we move to Fabric instead of Databricks?
A: Different target, overlapping Azure story. This article assumes a Databricks Lakehouse landing zone. If Fabric is in play, treat that as an explicit architecture decision — do not accidentally run two migrations under one slogan.

Q: Can we keep Synapse Serverless for “just a few views”?
A: Sometimes as a bridge. Long-term dual query planes usually recreate the governance tax you are trying to escape.

Q: How perfect must automated conversion be?
A: Good enough to move bulk syntax. Your senior engineers should own semantic diffs, SCD logic, and orchestration.

Q: What is the first metric of a healthy pilot?
A: Not “job duration.” It is signed KPI parity with a business owner plus a rerun runbook a junior can execute.

14) Sources

Databricks — Navigating a Synapse Migration to Databricks
https://www.databricks.com/blog/navigating-synapse-migration-databricks
Databricks — Introducing Lakebridge
https://www.databricks.com/blog/introducing-lakebridge-free-open-data-migration-databricks-sql
Lakebridge docs — Synapse Profiler
https://databrickslabs.github.io/lakebridge/docs/assessment/profiler/synapse/
Lakebridge docs — Transpile
https://databrickslabs.github.io/lakebridge/docs/transpile/
Practical large-table movement discussion (CETAS → ADLS → Delta)
https://medium.com/@prabhakarankanniappan/copy-3-5-billion-rows-from-synapse-to-databricks-in-30-minutes-fda15e0ffca0

6 things that broke our Polymarket trading bot before it made a dollar

Wed, 07 Oct 2026 12:47:20 +0000

We spent a few weeks trying to build a profitable Polymarket bot. Arbitrage between linked markets, in-play market making, copying wallets that trade before goals, betting against miscalibrated prices. We recorded the full order book for every football market, indexed on-chain trades and replayed each idea.

None of them made money. The full research with numbers is on our blog, and the CSVs are on GitHub.

This post is the engineering side: six things about the Polymarket API and our own pipeline that would have broken the bot even if a strategy had worked.

1. Your taker order waits a second, and you can't cancel it

Sports markets have a field in the Gamma API called secondsDelay. Football is 1, NBA is 0. You can see it on any event:

GET https://gamma-api.polymarket.com/events?slug=fif-sri-mri-2026-10-05
... "secondsDelay": 1 ...

Per the Order Lifecycle docs, a marketable order waits out that delay before matching, and "during either delay, the order is pending and cannot be canceled". Resting limit orders can still be cancelled. Some crypto and finance up/down markets have a 250 ms version of the same thing.

What it did to us: our arbitrage detector found 369 mispriced sets across linked football markets in 18 hours. Of the 2c+ ones we rechecked one second later, all legs were still in the book in 3, worth $1.83 total. Makers pull their quotes inside your delay, or a faster taker takes the legs. If your bot sees a price and hits it, check secondsDelay first.

2. Fees are not a percentage

Only takers pay, and the fee depends on price:

def taker_fee(shares: float, price: float, fee_rate: float) -> float:
    # fee_rate from Gamma feeSchedule.rate, e.g. 0.05 for sports
    return shares * fee_rate * price * (1 - price)

It peaks at 0.50 (p * (1 - p) = 0.25) and goes to zero near 0 and 1. A 1c edge on a 50c contract is mostly eaten. Rates and maker rebates are in the fees docs.

3. The mempool gives you nothing

Orders and cancels are off-chain (EIP-712 signatures), only the settlement goes to Polygon. We measured:

Source	Lag behind matching
Order book WebSocket	~17 ms
On-chain settlement	~2 s median

By the time a trade shows up in the Polygon mempool, it already happened and the book already moved. Watch the WebSocket, not the chain.

4. WebSocket events arrive out of order

The exchange can send price_change (a level got eaten) before the last_trade_price event of the same trade. We replayed fills in arrival order at first, and every fill looked about 25% worse than it really was. Sort by the trade, not by arrival time, before you trust any replay.

For what it's worth, a plain WebSocket recorder (price_change, book, last_trade_price with tx hash) captured 99.9% of the trades that later appeared on-chain, so you don't need anything fancier to build a replay dataset.

5. Your backtest can lie with perfect data

This one wasn't the API, it was us. We picked wallets that bought right before goals and simulated copying them 1-5 seconds later: +$4.23 per $10. Then we noticed we had only selected the buys that came right before goals. In real time you don't know which buys those are.

Rerun on every buy by the same wallets, without looking at the outcome: -$0.41 per $10.

Any filter that uses something you only learn later (a goal, a resolution, a price move) has to be removed from the selection step. If the result survives that, then you can start believing it.

6. Some "Polymarket bot" repos want your private key

Before you reach for an existing open-source bot:

StepSecurity found 20+ fake repos with bought stars (polymarket-trading-bot-new, polymarket-copy-trading-bot-sports, ...) pulling typosquatted npm packages that steal keys and .env files.
SafeDep found 9 npm packages (polymarket-bot, polymarket-claude-code, ...) that ask you to paste your wallet key.

Read the code and the dependency tree before anything touches a funded key.

Smaller Gamma API notes

offset pagination returns 422 somewhere after 2,100-2,500 records. Use the keyset endpoints for a full crawl.
/events/keyset ignores the negRisk filter, filter on the market field instead. /markets/keyset respects end_date_min / end_date_max.
Some requests fail without a User-Agent header.

The data is CC BY 4.0. If you're building something on Polymarket and want to check our numbers, the repo has every CSV, and we'd like to hear where we got it wrong.

Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.

The IRS Handed Back 96.7% of the Money on Its AI Coding-Tools Order. The Record Says When, Not Why.

Tue, 29 Sep 2026 12:55:06 +0000

Originally published at vibeagentmaking.com.

One IRS delivery order for AI coding and chat tools was obligated at $3.31 million on the last day of fiscal 2025 and de-obligated by 96.7% nine months later. A census of all 449 fiscal-2026 federal AI contract actions shows how unusual that shape is, and why the record cannot say why.

The federal contract record is a plain document. It does not editorialise, it rarely explains, and when an agency changes its mind it writes the change down in capital letters and moves on. Here are the two lines that make up the entire recorded life of one purchase.

On September 29, 2025, with one day left in the federal fiscal year, the Internal Revenue Service obligated $3,314,392.00 to a company called Sokat Consulting LLC. The award text reads, in full: "THIS IS DELIVERY ORDER 1 FOR ARTIFICIAL INTELLIGENCE CODING TOOLS AND ARTIFICIAL INTELLIGENCE CHAT TOOLS AGAINST 2032L225D00002." The period of performance ran from September 30, 2025 to September 29, 2026. The product code says perpetual-license business application software. The ceiling on the order, the most it could ever have grown to, was $5,080,563.10.

On July 9, 2026, nine months and ten days later, a modification numbered P00001 took $3,205,416.90 of that money back. Its text: "THIS IS A REQ FOR MOD TO DE-OBLIGATE FUNDING FOR ARTIFICIAL INTELLIGENCE CODING TOOLS AND ARTIFICIAL INTELLIGENCE CHAT TOOLS AGAINST 2032L225D00002." Its action-type code is C, which the record itself defines as "funding only action".

That is all there is. What is left on the order is $108,975.10, which is 3.3% of what was obligated. The de-obligation landed with about twelve weeks still to run on a one-year order. The vehicle the order sits on, contract 2032L225D00002, has exactly one order on it, this one, as of September 21, 2026, both by a text search for the vehicle's number and by the vehicle's own list of orders in the award database. There is no delivery order 2.

I want to be careful about what I am doing with these two lines, because the temptation is obvious and I have watched other people give in to it. A 96.7% de-obligation on an AI coding-tools order, at an agency that has been in the news for staffing cuts, in the year everyone is buying AI coding tools, is a story that writes itself. The problem is that the record does not write it. No document in the award file gives a reason beyond the modification's own title. So the rest of this piece is not about why the IRS did this. It is about the only question the record can actually answer: how unusual is it?

The population

To answer that I need every comparable line, not one. The government's own award database, USAspending, has an API that returns contract transactions by keyword, so on September 18, 2026 I pulled every federal contract action in fiscal 2026 (October 1, 2025 through the day of the pull) whose text matches "artificial intelligence". That is the whole selection rule. It matches the award's own description, so it catches awards that say they are about AI and misses awards that do AI work without saying so. It is a census of what the text says, and every total below should be read that way.

The pull returned 449 transactions: $358,899,346.16 obligated, $14,990,819.87 de-obligated, the de-obligations spread across 34 negative lines. So the naive headline is available immediately: fifteen million dollars of federal AI money went back this year. The naive headline is wrong, and the way it is wrong is the useful part.

A negative line in this database means the obligated amount on an award went down. It does not say why, and there are at least four different events that produce one. The API records, on every transaction, a one-letter action-type code and the plain-English name of that code, and I took each code's meaning from the row that carried it rather than from memory. F means "terminate for convenience (complete or partial)". K means "close out". C means "funding only action". B means "supplemental agreement for work within scope".

Then I did the thing the headline number skips: I opened each of the 34 awards and read every modification on it, and sorted the negative lines into four bins.

Offsets. Two lines, $8,162,903.45. The same award put the same amount back within days. The largest single negative line in the whole year belongs to this bin: on May 29, 2026, ECS Federal's Army contract for AI/ML algorithms and models (W911QX25C0002) shows a "supplemental agreement for work within scope" of minus $7,812,903.45, and on June 3, five days later, a "funding only action" of plus $7,812,903.45. Read the negative line alone and 52.1% of the year's "returned AI money" is a single accounting pair. Read the award's history and the contract grew all year: plus $6,737,560.46 in January, plus $4,475,886.00 in February, plus $10,028,288.07 in March, plus $4,941,047.00 in May. It is obligated at $72,819,404.20 against a $97,637,874.98 ceiling. Nothing came back.

Closeouts. Five lines, $230,099.96. The administrative close of a finished contract, where whatever was not spent is released. This is the sound of a contract ending normally.

Terminations. Three lines, $1,926,136.26. The government formally ended remaining work. These are the only lines in the year that carry the termination code, and I will come back to all three.

Standing de-obligations. Twenty-four lines, $4,671,680.20. Money went back and did not come back within thirty days on the same award, and the line carries no termination code. This is the bin the IRS order sits in, and it sits in it the way a whale sits in a pond. The IRS line is $3,205,416.90 of the bin's $4,671,680.20, which is 68.6%. Counting the closeouts and terminations too, it is nearly half of every dollar that left a federal AI contract this year and stayed gone.

So the answer to "how unusual" is: very. Not because the amount is large in federal terms, it is not, but because the shape is. The rest of the year's pull-backs are either bookkeeping, or the ordinary end of a contract, or one of three explicit terminations. The IRS line is none of those. It is the largest thing in the database that went back, stayed back, and was never called a termination.

What the three real terminations look like

It helps to see what the record looks like when the government does end AI work on purpose, because the contrast with the IRS line is the point.

The Securities and Exchange Commission, December 8, 2025. Award 50310225F0034 to NumInformatics Inc, described in the record as "OIT AIML". The termination line is the most explicit sentence in this whole census: "PARTIAL TERMINATION FOR CONVENIENCE AGREEMENT FOR OFFICE OF INFORMATION TECHNOLOGY ARTIFICIAL INTELLIGENCE MACHINE LEARNING (AIML). TERMINATED ALL REMAINING AIML WORK BEYOND THE TERMINATION OF CONVENIENCE NOTICE, TO INCLUDE ALL OPTIONAL OUTYEARS." It took back $851,509.27 of $1,333,740.53 and left $482,231.26, 36.2% of what had been obligated. When the SEC ended its AI/ML support work, the record says so in the record's own vocabulary.

The Patent and Trademark Office, August 20, 2026. Award 1333BJ24C00280005 to Accenture Federal Services, described as "PATENT SEARCH ARTIFICIAL INTELLIGENCE MODELS AND LABOR. THIS ACTION IS AWARDED PURSUANT TO THE USPTO EFFICIENCY ACT." The termination line takes back $624,728.99 and is titled "PATENT SEARCH ARTIFICIAL INTELLIGENCE DEVELOPMENT SECURITY OPERATIONS SUPPORT." That is a partial termination of a slice: the award still carries $27,753,766.76 obligated against a $55,322,899.54 ceiling, so the terminated amount is 2.2% of what had been obligated before it. The patent-search AI work did not end; a piece of it did, and the record names the piece.

The Air Force, March 26, 2026. Award FA864924P1125 to AI2 Incorporated, "SECURE LARGE LANGUAGE MODELS TO ENABLE GENERAL ARTIFICIAL INTELLIGENCE FOR CRITICAL DEPARTMENT OF THE AIR FORCE APPLICATIONS." The termination took back $449,898.00 of $1,799,592.00, which is exactly 25.0%, to the dollar. The award record lists the period of performance as August 16, 2024 to December 16, 2025, so the termination line is dated three months after the recorded end of performance. I do not know what that sequence means and the record does not say; I note it because a reader who goes to the award page will see it.

Three terminations, $1.93 million between them, each one labelled as a termination by the agency that made it; two of them left most of the award in place, and the SEC's left 36.2%. That is what the federal government ending AI work looks like in fiscal 2026 when it chooses to say so. It is small, it is partial, and it is named.

What a de-obligation is not

The IRS modification is coded C, a funding action, not F, a termination. That is not a technicality; it is the difference between what the record asserts and what a reader might assume. A funding-only de-obligation is the mechanism an agency uses when money on an award will not be needed. It is used when a vendor delivers under budget, when a scope is cut, when a programme is cancelled, when a licence count is reduced, when the funds are needed elsewhere before the fiscal year closes, and for reasons that never make it into a title field at all. The record does not distinguish between these. Reading "de-obligate funding" as "the tools failed" is a guess, and I am not going to make it, and I would ask you not to make it on my behalf.

There is a second thing this piece does not claim, and it comes from an earlier one. In a previous look at the same ledger I found that the outlay column, the field that is supposed to say what was actually paid, is missing, zero, or partial on most federal AI awards, and cannot be used to say what a vendor received. So I am not claiming that Sokat was paid the $108,975.10 that remains on the order, or that it was not, or that any part of the $3.3 million was ever drawn before it went back. Obligation is a promise of money. This record is a record of promises, and of one promise mostly withdrawn.

What the record does establish is narrower and, I think, more interesting than the guess. An agency committed $3.3 million, with one day left to spend that year's money, to a category of software that every large organisation is currently being told it needs. Nine months later it kept 3.3% of the commitment. It did not call this a termination. It did not place a second order on the vehicle it had set up to place orders on. Every one of those facts is a line in a public database, and none of them is an opinion.

How to read a number like 96.7%

I want to say something about the number itself, because it is the kind of number that travels, and numbers that travel lose their denominators on the way.

96.7% is the de-obligated amount over the obligated amount on one delivery order. It is not the IRS's AI budget, or its software budget, or anything about the IRS's other AI spending, which in this census consists of unrelated buys: $1,734,979.00 to MITRE on September 9, 2026, $485,864.89 to Unison for a "Virtual Acquisition Office" on August 29, and two zero-dollar administrative lines. It is not a statement about the vendor's product, which the record never describes beyond "coding tools" and "chat tools". It is not a statement about whether anyone at the IRS used the tools; the record has no field for that. And it is not evidence that AI coding tools are being returned across the government: in the same year and the same database, the sum of every standing de-obligation on every AI-described contract other than this one is $1,466,263.30, spread across 23 lines, the largest of them $394,959.23.

What 96.7% is, is a precise description of one event with no explanation attached. That makes it useful in exactly one way. Someone who wants to know why a large federal AI purchase went nearly all the way back now knows which purchase to ask about, which agency, which order number, which modification, and which date. A Freedom of Information Act request for the justification behind modification P00001 on order 2032L225F00049 would get an answer the API cannot give. So would the IRS's fiscal 2026 information-technology spend plan. So, for that matter, would a phone call to the vendor.

The method, so you can check it

Everything above comes from three small scripts run against the USAspending API on September 18, 2026 at 00:43 UTC, published alongside this piece together with the raw responses they saved. The first, ai_termination_census.py, pulls the census: POST /api/v2/search/spending_by_transaction/ with the keyword "artificial intelligence", award types A through D (the contract types), and the fiscal-2026 date window. The second, deob_classify.py, takes every award that carried a negative line and pulls its full modification history from POST /api/v2/transactions/, then classifies each negative line as an offset (the same award re-obligated the same amount within thirty days), a termination (code F), a closeout (code K), or standing (none of those). The third, award_detail.py, pulls each named award's own record from GET /api/v2/awards/, which is where the descriptions, periods and ceilings quoted here come from. A fourth script, figures_c6153.py, reads the saved responses back and recomputes every figure quoted above, so a number that drifts from its source fails the run; it makes no network call.

The limits are facts about the measurement rather than hedges. The keyword selects text, not a curated list of AI programmes. Fiscal 2026 is partial, ending at the pull date. Only contract actions are counted, not grants or the ceilings of the vehicles themselves. The thirty-day offset window is a parameter; a re-obligation on day forty-five, or on a sibling award, reads as standing here. Seven rows carry action codes (W, H, L) that I did not resolve to descriptions; none of them is negative, so none affects any figure above. And once more, because it is the sentence most likely to be dropped when this gets quoted: a de-obligation is money returned, not a verdict on the vendor, the programme, or the tools.

The order still has $108,975.10 on it and eight days of performance left as I write this. On September 30 the fiscal year turns over, the vehicle will either get a delivery order 2 or it will not, and the database will record which, in capital letters, without saying why.

Every dollar figure, date, percentage and quoted record line in this piece was checked against the saved API responses by a script published with it, figures_c6153.py, which reloads the 449-row census and the per-award modification histories, recomputes the four bins and every derived number, and asserts each quoted amount is present in the rows. A figure missing from the rows fails the run. The three collection scripts and the raw responses they saved (ai_fy26.json, ai_fy26_verdicts.json, awards.out.txt) are published beside it, with the two checks of the vehicle's orders (vehicle_2032L225D00002_recheck.json, the text search, and idv_child_awards_2032L225D00002.json, the vehicle's own list of orders).

Sources:

USAspending.gov award record, delivery order 2032L225F00049 (Sokat Consulting LLC, Department of the Treasury / Internal Revenue Service, parent IDV 2032L225D00002): the September 29, 2025 award and modification P00001 of July 9, 2026, quoted verbatim. https://www.usaspending.gov/award/CONT_AWD_2032L225F00049_2050_2032L225D00002_2050
USAspending.gov record of the vehicle itself, IDV 2032L225D00002, whose list of orders holds one award as of September 21, 2026. https://www.usaspending.gov/award/CONT_IDV_2032L225D00002_2050
USAspending.gov award record, W911QX25C0002 (ECS Federal, LLC, Department of the Army): modifications P00010 through P00015, the May 29 and June 3, 2026 offset pair. https://www.usaspending.gov/award/CONT_AWD_W911QX25C0002_9700_-NONE-_-NONE-
USAspending.gov award record, 50310225F0034 (NumInformatics Inc, Securities and Exchange Commission): modification P00002 of December 8, 2025, the partial termination for convenience, quoted verbatim. https://www.usaspending.gov/award/CONT_AWD_50310225F0034_5000_47QTCA24D00G1_4732
USAspending.gov award record, 1333BJ24C00280005 (Accenture Federal Services LLC, U.S. Patent and Trademark Office): modification P26012 of August 20, 2026. https://www.usaspending.gov/award/CONT_AWD_1333BJ24C00280005_1344_-NONE-_-NONE-
USAspending.gov award record, FA864924P1125 (AI2 Incorporated, Department of the Air Force): modification P00003 of March 26, 2026. https://www.usaspending.gov/award/CONT_AWD_FA864924P1125_9700_-NONE-_-NONE-
USAspending API, POST /api/v2/search/spending_by_transaction/, POST /api/v2/transactions/ and GET /api/v2/awards/{id}/, pulled September 18, 2026 00:43 UTC; action-type code meanings taken from each response's own action_type_description field. https://api.usaspending.gov/docs/endpoints
Treasury Inspector General for Tax Administration, "Snapshot Report: IRS Workforce Reductions as of May 2025", Report No. 2025-IE-R027, July 18, 2025: 25,386 employees separated, took a deferred resignation offer or used another incentive to leave, 25 percent of the IRS's workforce. The source for the staffing cuts. https://www.tigta.gov/sites/default/files/reports/2025-08/2025ier027fr.pdf
"The $1.45 Billion Federal AI Contracts Ledger Accounts for 17 Percent", this site, on the outlay column's reliability. https://vibeagentmaking.com/blog/1-45-billion-federal-ai-contracts-ledger-accounts-for-17-percent/

Polymarket Gamma API endDate Is Not the Resolution Deadline

Thu, 24 Sep 2026 17:36:25 +0000

Checked against the Polymarket Gamma API on 24 September 2026.

Short answer. OrcaLayer compared the endDate field with the written rules on 23 Polymarket markets. Polymarket's documentation describes a market's end date as the point when it becomes eligible for resolution. The field did not behave that way on sports markets: on NFL and soccer matches it was the kickoff, on esports the start plus 6 hours, on tennis the original start plus 7 days. The deadline that governs a market is in its description, and whether it still trades is in closed and acceptingOrders.

If you build on Polymarket data, this changes what your code should read. A bot that stops trading at endDate, a dashboard that counts down to it, and a model that treats it as the resolution time are all reading a field that means different things on different market types.

What is endDate in the Polymarket Gamma API?

endDate is a UTC timestamp on every market in the Polymarket Gamma API. The API does not document what it represents, and it is not the resolution deadline. In OrcaLayer's sample it was kickoff on NFL and soccer matches, start plus 6 hours on esports, and original start plus 7 days on tennis. The binding deadline is in description.

Every market returned by https://gamma-api.polymarket.com/markets/{id} carries endDate at the top level. If you are new to these endpoints, start with our Polymarket REST API guide for developers.

The response usually tells you what kind of market you are looking at. Sports matches carry gameStartTime, eventStartTime, gameId and sportsMarketType (for example "moneyline"). Markets that belong to a group carry groupItemTitle, which can be a label such as "Match Winner" or a date such as "September 30". Closed markets add closedTime and umaEndDate. The rules are in plain English in description.

Polymarket's own resolution documentation puts it plainly: "Always read the resolution rules before trading." The rest of this article shows what happens when code reads endDate instead.

Polymarket endDate by market type

Market type	Markets	What endDate was	What the rules say	Closed before or after endDate
Esports (CS2, LoL, Dota 2)	4	Scheduled start + exactly 6 hours, 4 of 4	A postponed match must be rescheduled to start within 14 days, or the market resolves 50-50	All 4 before, by 19 minutes to 3 h 18 min
Tennis (ATP, ITF)	4	Original scheduled start + exactly 7 days, 4 of 4	A winner must be determined within 14 days, or the market resolves 50-50	All 4 before, by 6.3 to 6.8 days
NFL, college football, soccer matches	4	Kickoff, 4 of 4	No fixed deadline: open until the game is completed	All 4 after, by 2 h 17 min to 4 h 57 min
MLB match	1	Start + 7 days	No fixed deadline	Before, by about 7 days
Sports futures	3	Varies	A "postponed after" backstop	Still open
Deadline and price markets	7	Matched the rules in 4 of 7	"By" a date, or a date in the title	Mixed

The sample is 23 hand-picked markets covering each type, so read each row as an observed pattern, not a platform-wide rate.

Esports: endDate is the start plus six hours

All four esports matches had an endDate exactly six hours after gameStartTime. The Counter-Strike match between QUAZAR and Misa Esports started at 08:00 UTC on 23 September, carried an endDate of 14:00 UTC, and closed at 13:41 UTC.

Six hours is a fair guess for when a best-of-three finishes, but the rules for the same market allow far longer:

If the match is postponed, it must be rescheduled to a start time on or before October 7, 2026 at 11:59 PM ET (14 calendar days after the originally scheduled start of September 23, 2026 at 4:00 AM ET). If, by that deadline, the most recently announced rescheduled start time is on or before the deadline, this market resolves based on the result of the match as played. Otherwise, this market resolves 50-50, regardless of whether the match is subsequently played.

The 14 days apply to the rescheduled start, not to the result. A match rescheduled to start on 7 October can resolve later still, long after endDate has passed. You can see current esports markets on OrcaLayer's esports page.

Tennis: seven days after the original schedule, and it does not move

Every tennis match in the sample had an endDate exactly seven days after its originally scheduled start. Here the 14-day rule applies to the result:

If the match is canceled (not played at all), ends in a tie, or a winner has not been determined by October 6, 2026, 11:59 PM ET (14 days after the scheduled start), this market will resolve to 50-50.

Rescheduling is where it goes wrong. In two ATP matches, gameStartTime was later than the start printed in the rules. The US Open match between Ben Shelton and Carlos Alcaraz was scheduled for 8 September at 15:30 UTC and started on 9 September at 03:00 UTC. Its endDate was 15 September at 15:30 UTC: seven days after the original slot, not the new one.

A countdown built on endDate for a tennis market counts from a schedule that may no longer exist. Take the start from gameStartTime and the deadline from the rules.

NFL and soccer markets: endDate is kickoff, not the end of the game

On the Broncos vs Chiefs NFL game, the SMU vs Louisville college football game, a La Liga match and a Romanian Liga I match, endDate was equal to kickoff to the second. All four markets closed after it, between 2 h 17 min and 4 h 57 min later, which is how long the games took.

The rules on these markets set no deadline at all:

If the game is postponed, this market will remain open until the game has been completed.

A kickoff is the one moment when a match market is certainly not ready to resolve. Code that compares the current time with endDate would have marked all four of these markets as finished at kickoff, while the games were being played and the markets were still trading.

Across the 16 closed markets in the sample, 6 closed after their endDate. Four were these matches. The other two were a Bitcoin price market that closed 14 minutes after its reference time and a Fed rate market, covered below.

Sports futures: sometimes the event, sometimes the backstop

League and tournament winner markets followed no single rule. On Shakhtar Donetsk winning the 2026-27 UEFA Champions League, endDate was 30 May 2027 at 23:59 UTC, while the rules fall back to "Other" only if the season is "postponed after June 19, 2027, 11:59 PM ET", about twenty days later. The New York Yankees World Series market had an endDate of 31 October 2026, and its rules allow postponement until 31 December 2026. The Boston Celtics NBA Finals market matched its rules to the minute.

For futures, the field alone does not tell you whether you are looking at an expected event date or the formal backstop.

Deadline markets: right in 4 of 7, once wrong by 92 days

Markets that ask whether something happens "by" a date came closest to what most people assume endDate means. Two of them, "Will Anthropic or OpenAI IPO first?" and the Clarity Act market, matched the deadline in the rules, one with a one-minute offset (05:00 instead of 04:59 UTC). The two Bitcoin price markets also matched, though their rules point to "the date specified in the title" or "the month specified in the title", so the deadline lives in the market name.

Three did not match:

"Will the US confirm that aliens exist by September 30?" The rules say "by September 30, 2026, 11:59 PM ET", which is 1 October at 03:59 UTC. endDate was 1 January 2027 at 04:59 UTC, 92 days later. That is exactly the endDate of the parent event the market belongs to, which runs to the end of 2026, so the market appears to carry its event's date rather than its own. Two other fields in the same response point the right way or the wrong way: groupItemTitle is "September 30", which matches the rules, and hasReviewedDates is true, which shows that a reviewed flag does not guarantee the date matches the rules.
The Fed rate market for the September 2026 meeting. endDate was 16 September at 00:00 UTC, which is 8 PM ET on 15 September, the evening of the first day of a two-day meeting. The market closed 20 hours after its endDate.
The market on the next Russian parliamentary election. endDate was 30 September 2026 at 00:00 UTC. The rules fall back to "Other" only if results are not known by 30 September 2027. The market closed on 21 September.

On grouped date markets, check groupItemTitle before you trust endDate.

Why endDate changes, and why the Gamma API returns old values

Two markets returned different endDate values depending on which snapshot the API served.

For the aliens market, a response with updatedAt of 21 September showed endDate as 1 October 2026 at 03:59 UTC, which matches the rules. A fresh response on 24 September showed 1 January 2027. For the IPO market, a response with updatedAt of 8 August showed 31 December 2027 at 00:00 UTC, 29 hours before the written deadline, and the fresh response showed 1 January 2028 at 04:59 UTC, which matches the rules. Between those snapshots, the IPO market was corrected and the aliens market moved 92 days away from its own rules.

updatedAt will not tell you when that happened. It changes often and in bulk: at one point on 24 September the aliens, IPO, Celtics and Bitcoin markets all carried the same updatedAt to the microsecond. Store your own fetch time instead.

The older values reached us through cached responses. In three cases in the sample, a plain request to /markets/{id} returned an older snapshot, and adding an unused query parameter returned the current one. If your code reads endDate once when a market is listed, it may be holding a value the platform has since changed.

endDate vs gameStartTime vs closedTime vs umaEndDate

Field	What it held in the sample	Safe use
`endDate`	Kickoff, start + 6 h, start + 7 days, or a deadline, depending on market type	A hint, never a deadline
`gameStartTime`	Start of the sports event, updated when a match was rescheduled	Event start
`closedTime`	When the market actually closed	After-the-fact close time
`umaEndDate`	Identical to `closedTime` on all 16 closed markets	Same as `closedTime`
`description`	The written rules, including any deadline in ET	The real deadline

The formats differ. gameStartTime and closedTime come as "2026-09-23 08:00:00+00", while endDate and eventStartTime use "2026-09-23T08:00:00Z". datetime.fromisoformat rejects both forms before Python 3.11, so normalize first:

from datetime import datetime

def parse_ts(s: str) -> datetime:
    """Parse any Gamma API timestamp into an aware UTC datetime."""
    s = s.replace(" ", "T").replace("Z", "+00:00")
    if s.endswith("+00"):
        s += ":00"
    return datetime.fromisoformat(s)

How to tell if a Polymarket market is still trading

Do not use endDate for this, and do not use active either. The closed CS2 market in the sample returned "active": true together with "closed": true and "acceptingOrders": false. Read closed and acceptingOrders, and for the resolution state read umaResolutionStatus, which was "resolved" on that market.

def is_tradable(m: dict) -> bool:
    return not m.get("closed") and bool(m.get("acceptingOrders"))

This is not bulletproof. In January 2026 a public report on Polymarket's Rust SDK repository showed finished Africa Cup of Nations markets still returning active=true, closed=false and acceptingOrders=true, with endDate equal to kickoff (issue #199). On our September sample the two flags were reliable, but treat them as the best signal available, not a guarantee.

Closing and resolving are separate steps. After a market closes, the outcome is proposed to the UMA Optimistic Oracle, and according to Polymarket's documentation an undisputed proposal resolves after a 2-hour challenge period.

How to find the real resolution deadline of a Polymarket market

Read description. That is the contract. Sports matches say "on or before … ET" or "by … ET (14 days after the scheduled start)". US match markets say they "remain open until the game has been completed". Futures say "postponed after …".
Check the title and groupItemTitle on price and date markets. "On September 17" or a groupItemTitle of "September 30" is the deadline.
Take the start from gameStartTime, not from endDate minus a guess.
Use closedTime or umaEndDate to know when a market actually closed. Both are empty until it does.
Re-fetch before you act. Add a cache-busting parameter and store your fetch time next to every value you keep.

Python: parse the resolution deadline from a market description

This parser found the deadline on every market in the sample whose rules state a full date and a clock time in Eastern Time:

import re
from datetime import datetime
from zoneinfo import ZoneInfo

MONTHS = "January|February|March|April|May|June|July|August|September|October|November|December"
DEADLINE = re.compile(
    rf"\b(?:on or before|by|after) ((?:{MONTHS}) \d{{1,2}}, \d{{4}}),? (?:at )?(\d{{1,2}}:\d{{2}} ?[AP]M) E[DS]?T\b"
)

def rule_deadlines_utc(description: str) -> list[datetime]:
    """Return every ET deadline stated in a Polymarket market description, in UTC."""
    found = []
    for day, clock in DEADLINE.findall(description):
        local = datetime.strptime(f"{day} {clock.replace(' ', '')}", "%B %d, %Y %I:%M%p")
        local = local.replace(tzinfo=ZoneInfo("America/New_York"))
        found.append(local.astimezone(ZoneInfo("UTC")))
    return found

Use America/New_York rather than a fixed offset: ET is UTC-4 until 1 November 2026 and UTC-5 after, so a hard-coded offset will be an hour wrong for several months of every year. The parser returns nothing on US match markets, because their rules contain no deadline, and nothing on price markets, where the date is in the title. It does not handle "noon", "p.m." or abbreviated months, and it does not tell a start deadline from a result deadline, so read the sentence around each match.

What this sample does not tell you

Twenty-three markets show that endDate means different things on different market types. They do not show how often each case occurs across Polymarket, or that every esports market uses exactly six hours. Valorant is not in the sample, and there is a single MLB game.

We did not establish why the aliens market moved, or whether that change was deliberate. All markets here come from the international Polymarket Gamma API. Polymarket US is a separate exchange with its own contract rules, and nothing in this article applies to it.

Data: 23 markets fetched one by one from gamma-api.polymarket.com/markets/{id} on 24 September 2026. The full sample is downloadable as JSON.

Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.

Building NLP Systems for 96 Low-Resource Languages: A Web Scraping War Story

Thu, 06 Aug 2026 16:21:00 +0000

Or why language vitality online has nothing to do with speaker counts — and everything to do with data engineering

Russia speaks dozens of minority languages — Tatar, Bashkir, Udmurt, and even tiny ones like South Yukaghir (fewer than 50 speakers). Some have no Wikipedia, no standard corpus, and almost no digital footprint.

But they do exist online — scattered across forums, blogs, and VK communities.

We wanted to measure their online presence. As NLP engineers, we quickly realized: no global corpus, no labelled dataset, no pre-trained embeddings, and no simple way to ask a search engine "give me all pages in Chechen".

So we built our own pipeline. Here's what every developer building NLP systems for low-resource languages needs to know.

The Core Problem: Low-Resource NLP Without a Dictionary

In mainstream NLP, you start with a corpus. Wikipedia. Common Crawl. Something.

For 96 minority languages, the corpus doesn't exist. You have to find the data first — before you can even think about training embeddings, fine-tuning BERT, or building a translation system.

This is the ultimate cold-start problem.

Every stage of a standard NLP pipeline breaks:

NLP Stage	The Problem
Data Acquisition	No central repository. Data is scattered across VK, forums, blogs.
Language Identification	No pre-trained classifier for 90% of these languages.
Tokenization	No standard orthography. People type without diacritics.
Embeddings	No pre-trained word vectors. You must build from scratch.
Evaluation	No gold-standard test sets. You can't measure what you can't label.

Step 1: Data Acquisition — The Search API as Your Crawler

You can't download the whole internet (unless you're Google). But search engines already crawled it.

We used Yandex.XML — a limited API with 10,000 requests per day. Each request returns one page of results, and each next page costs another request.

For an engineer, this means: you have to be extremely efficient with your queries.

The challenge: What do you search for?

We needed word markers — words that are:

Frequent in the target language
Unique — not appearing in Russian or other languages
Free of special characters — because real users don't type them

That last point is critical for any NLP system dealing with real-world user-generated content.

Most minority languages use Cyrillic extensions: ҙ, ә, ö, ҷ. There's no standard keyboard for them. People write "менан" instead of "менән". If we query with the "correct" spelling, we lose 90% of real texts.

We call this "household spelling" — like medieval birch-bark manuscripts, but with modern laziness.

For your NLP pipeline: If you're building systems for languages without standard keyboards, you need fuzzy matching at the query stage, not just at the evaluation stage. This means:

Levenshtein-based query expansion
Phonetic normalization
Handling diacritic stripping before search

Step 2: Noise Filtering — The Precision Problem

Even good markers return junk. We had to filter:

Wikipedia duplicates — we can download those in one click, so exclude them
Linguistic papers — Russian text containing one line of Rutul as an example
Music sites — song titles but no lyrics
Typos — e.g., Russian "пазух" (bosom) mistyped as "пазох", which is a valid Khakas word meaning "again"

The solution: Multi-marker verification

If one marker appears on a domain but others from the same language don't — it's probably a false positive.

For your ML pipeline: This is ensemble validation — using multiple weak signals to confirm a strong one. The trade-off is extra API calls, which are expensive at 10k/day.

Alternative approach: Train a binary classifier (Russian vs. everything else) on the fly. We did this later — more on that below.

Step 3: Domain Classification — Building a Tiered Index

We asked Yandex to categorize each domain into three tiers:

Tier	Description	Example
1. Language-first	Most pages in the minority language	Small blogs, community sites
2. Mixed	Several pages, but not dominant	Regional news portals
3. Giant platforms	Minority content is buried	YouTube, stihi.ru

Additionally, we specifically queried VK.com communities because post-2012, most standalone language sites died — and everything moved to social networks.

For your data engineering pipeline: You need a stratified crawling strategy. Don't treat all domains equally. Allocate more budget to Tier 1, and use depth-limited sampling for Tier 3.

Step 4: Language Detection Without a Training Corpus

We downloaded pages and needed to separate Russian from, say, Tuvan.

Classic n-gram language detection requires a pre-labelled corpus — which we didn't have for most languages.

The solution: We did have Russian — plenty of it. So we built a binary classifier: Russian vs. everything else.

If a paragraph isn't Russian, we assume it's our target language.

For your ML system:

This is a one-class classification problem with a twist.
It's not perfect — but it's good enough to build the first ever web corpora for these languages.
This is a classic weak supervision approach: use what you have (Russian) to learn what you don't (everything else).

What We Learned: Online vs. Offline — Zero Correlation

We compared online metrics (Wikipedia articles, VK communities, text volume) with offline metrics (speaker count, regional economy).

Metric Pair	Correlation
Among online metrics	>0.7 (Spearman)
Online vs. offline	≈ 0

This is the key insight for anyone building IT systems for low-resource languages:

How many people speak a language does not predict how alive it is on the internet.

What matters is a small, active, passionate community.

Example: Bashkir has fewer speakers than Chechen, but far more online communities. Why?

Chechnya's internet infrastructure recovered later (1990s-2000s conflicts delayed digital adoption).
Online activity is a function of infrastructure, not just population.

For your product strategy: If you're building a language model for a low-resource language, prioritize communities, not census data. Find the VK groups, Discord servers, and Telegram channels. That's where the data lives.

A Few Geeky Findings

Finding	Implication
Post-2012 shift	Almost all minority language activity moved to VK.com. Standalone forums died.
Typical user	19-31 years old, lives in the region's capital (Ufa for Bashkir, Izhevsk for Udmurt).
Typical post length	~5 words per post in minority language. Most conversation is in Russian; people sprinkle native words as identity markers.
Interconnectivity	Minority language sites link to each other across language families. A rare case of "peoples united" online.

For your NLP models: If you're building embeddings from this data, expect sparse, short texts. Standard word2vec on 5-word posts won't work well. Consider:

Character-level embeddings (more robust to spelling variation)
Cross-lingual transfer (use Russian or English as a bridge)
Zero-shot approaches (label propagation from related languages)

The Data Engineering Pipeline — Summary

┌─────────────────┐
│  Yandex.XML API │  ← 10k requests/day budget
│  (search engine)│
└────────┬────────┘
         ▼
┌─────────────────┐
│  Word Markers   │  ← Frequent + Unique + No diacritics
│  (handcrafted)  │  ← "Household spelling" handling
└────────┬────────┘
         ▼
┌─────────────────┐
│  Domain Filter  │  ← Remove Wikipedia, linguistic papers
│  (multi-marker) │  ← Ensemble validation
└────────┬────────┘
         ▼
┌─────────────────┐
│  Tiered Index   │  ← Language-first, Mixed, Giant platforms
│  (stratified)   │  ← VK.com special treatment
└────────┬────────┘
         ▼
┌─────────────────┐
│  Binary Class.  │  ← Russian vs. everything else
│  (weak super.)  │  ← No pre-labelled corpus needed
└────────┬────────┘
         ▼
┌─────────────────┐
│  Web Corpora    │  ← First ever for these languages
│  (downloadable) │  ← Available at web-corpora.net
└─────────────────┘

The Data

Downloadable corpora and domain lists:
web-corpora.net/wsgi3/minorlangs/download

And yes, there's a VK community with cute memes in minority languages — featuring cats.

The Bottom Line for Engineers

Web scraping isn't just about avoiding robots.txt. It's about understanding human behaviour — typos, keyboard limitations, social network migration, and why a language with 50 speakers can thrive online if those 50 are determined enough.

Key takeaways for building NLP systems for low-resource languages:

Start with search APIs, not crawlers. The internet is already indexed. Use it.
Handle real-world spelling. People don't use diacritics. Your queries shouldn't either.
Use weak supervision. You don't need a labelled corpus to start — use what you have.
Prioritize communities, not speakers. Online vitality ≠ offline speaker count.
Expect short texts. ~5-word posts are the norm. Plan your embeddings accordingly.

Further reading:

Our downloadable corpora: web-corpora.net
Yandex.XML API documentation
VK.com API for social media data extraction

Liked this? Share with your NLP engineering team. They'll never look at "low-resource" the same way again.

Foundation Models Are Not the Product: Why Data Engineers Are Becoming the Bridge to Generative AI

Sat, 01 Aug 2026 20:22:35 +0000

TL;DR

Foundation Models are powerful, but they are not magic knowledge engines.

They can generate text, summarize information, and answer questions because they understand patterns learned during training. However, they do not know your company's data, processes, or domain knowledge.

That is where Data Engineering becomes critical.

The next generation of AI applications will not only depend on better models. They will depend on better data pipelines, better architectures, and better ways to deliver context.

The Question I Keep Hearing

Over the last months, while exploring Generative AI architectures, I noticed a recurring question:

"Why does this model know so much?"

It is a natural question.

You open ChatGPT, Claude, or another AI assistant, ask something complex, and receive an answer that feels surprisingly accurate.

The first instinct is to think:

"The model knows."

But the reality is more interesting.

The model does not know your business.

It does not know your internal documentation, your operational procedures, your customer history, or your company's decisions from last quarter.

What it knows are patterns.

And that difference is exactly where Data Engineers have an important role to play.

From Data Pipelines to AI Pipelines

For years, our job as Data Engineers has been focused on moving and transforming data so organizations can make better decisions.

We built pipelines like:

Operational Systems → ETL/ELT → Data Lake → Data Warehouse → Analytics

Now we are seeing a new evolution:

Enterprise Data → Retrieval Pipeline → Foundation Model → AI Application

The goal is different, but the engineering principles remain familiar.

Data still needs to be:

Reliable.
Governed.
Secure.
Fresh.
Well-structured.

The main difference is the consumer.

Before, the consumer was a dashboard or an analyst.

Now, the consumer can be an AI system.

The Problem With "Smart" Models

Imagine asking an AI assistant:

"What is our refund policy?"

The model can generate an answer that looks perfect.

It may use professional language.

It may explain exceptions.

It may even sound confident.

But if the model has never seen your company's actual refund policy, that answer is only a prediction based on patterns.

And predictions are not the same as facts.

This is one of the biggest challenges when moving Generative AI from experiments into production.

A fluent answer is not necessarily a correct answer.

For enterprise applications, accuracy and trust matter more than impressive demos.

Why RAG Became a Key Architecture Pattern

Retrieval-Augmented Generation (RAG) addresses this challenge by adding an important step before generation.

Instead of asking the model to answer only from its training knowledge, we first retrieve relevant information from trusted enterprise sources and provide that context to the model.

The workflow looks like this:

Collect enterprise data from different sources.
Clean and transform the information.
Generate embeddings and store them in a vector database.
Retrieve relevant context based on the user's question.
Send the context together with the prompt to the Foundation Model.
Generate a response grounded in business data.

The Foundation Model provides reasoning and language capabilities.

The data provides accuracy.

The pipeline connects both.

A Practical Example

Imagine a company with thousands of technical documents:

Product manuals.
Internal procedures.
Support documentation.
Security guidelines.
Engineering decisions.

The information exists, but finding the right answer requires time.

A traditional search system can locate documents, but employees still need to read and interpret them.

A RAG-powered assistant changes the experience.

Instead of searching through hundreds of pages, an engineer can ask:

"How do we handle this type of production incident?"

The system retrieves relevant documentation and generates an answer based on internal knowledge.

The AI is not replacing the documentation.

It is making the documentation accessible.

What This Means for Data Engineers

Generative AI has created a lot of excitement around models.

However, many production challenges are not model challenges.

They are data challenges.

How do we keep information updated?
How do we control access?
How do we prevent sensitive data exposure?
How do we evaluate response quality?
How do we monitor AI pipelines?

These are engineering questions.

And they are questions Data Engineers already know how to solve.

The future of AI will not only belong to the teams building the biggest models.

It will belong to the teams that can connect models with reliable, trusted, and meaningful data.

Final Thoughts

A Foundation Model brings intelligence capabilities.

But intelligence without context has limited business value.

The real opportunity for Data Engineers is building the bridge between enterprise data and AI applications.

The same discipline that helped organizations become data-driven is now helping them become AI-enabled.

The next generation of data pipelines will not only move information.

They will deliver knowledge.

Understanding Python Virtual Environments: The Interpreter Mismatch That Caused My Imports to Fail

Wed, 22 Jul 2026 17:49:21 +0000

While setting up my Data Engineering learning environment, I ran into an issue that many Python developers eventually face.

Every package installation completed successfully, yet importing those same packages consistently failed.

Initially, I assumed the virtual environment had become corrupted. I recreated it several times before realizing that the environment itself wasn’t the issue.

The real culprit was an interpreter mismatch.

The Python interpreter executing my project was different from the interpreter used by pip during package installation.

As a result, packages were installed successfully — but into an entirely different Python installation.

Why this happens
Python installations can coexist on the same machine.

Become a Medium member
Examples include:

System Python
Microsoft Store Python
Virtual environments
Conda environments
WSL installations
If VS Code selects one interpreter while pip installs into another, import errors are inevitable.

How I diagnose it now
Verify where python
Verify where pip
Confirm the active VS Code interpreter.
Install packages with:
python -m pip install
Verify installation:
python -m pip show
Standard workflow
python -m venv .venv
.venv\Scripts\activate
python -m pip install
Why this matters for Data Engineering
As I continue learning technologies such as PySpark, Airflow, Kafka, and modern data tooling, maintaining a reproducible Python environment becomes increasingly important.

Many “package not found” issues aren’t caused by the package itself — they’re caused by environment misconfiguration.

Debugging the environment before debugging the code has become one of the most valuable habits I’ve picked up so far.

Hopefully this saves another learner a few hours of frustration.

Your Dataset Is Your Foundation. Have You Checked If It's Crumbling?

Thu, 16 Jul 2026 17:17:49 +0000

Or why Wikipedia is the perfect Data Quality testing ground (not just for linguists)

Every developer who has ever scraped Wikipedia for an NLP task thinks: "It's human-written text. That means the dataset is balanced, natural, and ready to feed into my model."

Spoiler: it's not.

We analyzed 6 language editions of Wikipedia (Russian, Tatar, Bashkir, Mari, Mordvin, Komi) using the crudest but most effective tool in a data engineer's arsenal — frequency analysis of top tokens.

The results will change how you think about building training sets and text corpora for any IT system.

The litmus test: healthy dataset vs. train wreck

In any natural, human-written corpus, the most frequent tokens are function words — prepositions, conjunctions, pronouns, auxiliary verbs. This is a universal law across languages.

Example from the Russian National Corpus:

#	Word	POS
1	и (and)	conj
2	в (in)	prep
3	не (not)	part

If your top-20 contains "river", "basin", "water", "orchid" — your data pipeline is broken, even if your scraping code is flawless.

Russian Wikipedia — the gold standard (mostly)

1.06M articles. Top tokens:

#	Word	Frequency
1	в (in)	10,859,129
2	и (and)	5,761,105
3	на (on)	3,214,393

✅ Function words dominate. The dataset is usable for language modeling.

But Tatar and Bashkir Wikipedia? Data Quality Nightmare

~50K articles each. Top-5 for Tatar:

#	Word	Translation	Frequency
1	елга	river	132,567
2	бассейны	basin	75,706
3	су	water	54,689
4	буенча	by (postposition)	48,838
5	Русия	Russia	48,722

"River" appears 3x more often than the conjunction "and". Bashkir is identical: йылга (river), бассейны, һыу (water), Рәсәй (Russia).

Why? Auto-generated content strikes again

Most articles in these editions weren't written by humans. They were auto-generated from a template using Russia's State Water Registry.

Thousands of river entries — each repeating "river", "basin", "water", "km", "mouth". This is a textbook case of systematic bias that kills any ML pipeline.

For a developer, this means:

Your word embeddings will be obsessed with hydrology.
Any text classifier will overfit on geographic patterns.
You'll spend months debugging why your model keeps generating "river" in every context.

Mordvin Wikipedias: The Orchid Syndrome

Erzya and Moksha editions (~1,100–1,500 articles each). Top tokens include "category", "see", "day", "year" — but also:

Erzya	Moksha
истэжо (also)	ди (and)
категория	и (and)
ван (see)	тыналста (from family)
чи (day)	касыкссь (plant)
ды (and)	панчф (flower)
ие (year)	орхидея (orchid)

The explanation: hundreds of articles about orchids (family Orchidaceae) were bot-generated. Each repeats "orchid", "flower", "species", "genus".

They call it the Orchid Phenomenon — a dataset hijacked by bots writing about a single narrow topic.

Quick sanity check: average article length

Edition	Avg length (words)
Russian	184.5
Tatar	58.5
Bashkir	56.8
Komi-Zyrian	29.6
Erzya	34.5
Moksha	34.9

Short articles = high risk of template-driven, repetitive content. This is a red flag for any Data Engineer: you need additional filtering.

What this means for you as a developer building IT systems

1. Never trust a dataset blindly — especially for low-resource languages or small editions. Dataset size ≠ dataset quality.

2. Frequency analysis is your cheap, powerful friend. Generate a top-N word list in 5 minutes with any language (Python, collections.Counter, a few lines of code). Make it a mandatory step in your ETL pipeline.

3. Auto-generated content is a systemic risk. It's everywhere: not just rivers and orchids, but also geocoordinates, biological taxa, movie stubs, sports tournaments. If you don't clean it, your model will think the world consists of rivers and flowers.

4. Data Drift starts here. If your training distribution doesn't match production traffic, your system makes decisions based on a distorted reality. That's not a bug in your code — it's a bug in your data.

5. Version and profile your data like you version your code. Implement Data Quality checks (Great Expectations, Pytest for dataframes) in your CI/CD pipeline. If the top-10 tokens suddenly change — halt the pipeline.

The bottom line

Wikipedia is free, but not always natural.

For major languages (English, Russian, German) the corpus is surprisingly balanced. For many minority languages, the Wikipedia you scrape might be a botanical reference, not a linguistic corpus.

For an engineer, this means: always profile your data. A frequency list takes 5 minutes to generate and can save you months of debugging weird model predictions and overfitting on noise patterns.

Further reading: Orekhov, Reshetnikov (2014). "Towards an evaluation of Wikipedia as a linguistic source: a comparative study." In Modern Russian Language on the Internet, Moscow.

Liked this? Share with your fellow engineers. They'll never look at a Wikipedia dump as "clean data" again.

Is AI Coming for Data Analyst Jobs? Depends What You Actually Do All Day

Wed, 15 Jul 2026 13:11:30 +0000

Short answer: no. A bit longer answer: no, unless your whole job is just writing SQL.

Someone started a thread with a good question:

"You said frontend is dying because it's 'pure pattern-matching' that AI can now do. As a Data Analyst, I see Claude clean data, write SQL and build dashboards in seconds. The exact same pattern. Are Junior Data Analysts heading for the same C-Tier obsolescence?"

Honestly? Some of them, yeah. But not because they're analysts. Because a lot of them weren't really doing analysis to begin with.

People got confused about what the job even is

Somewhere along the way, "data analyst" turned into a fancy title for "person who writes SQL." That was never right. SQL is just the tool you use. The actual job is figuring out what's true, what matters, and what's about to blow up in a meeting if nobody catches it in time.

If your day looks like this:

"Pull last quarter's numbers"
"Clean this messy CSV"
"Make me a bar chart"

...that's not really analysis. That's data entry with extra steps. And yeah, that part is basically automated now. Claude writes that query in two seconds flat and never complains about it. No point pretending otherwise.

But that was always the junior, boring 20% of the job. People just mistook it for the whole thing.

Here's what AI still can't do

No matter how good these tools get at writing SQL, they still can't do this stuff:

Spot when a number is lying to you.
Revenue's up 12%. Nice - except pricing changed halfway through the quarter, and one big client is basically all of that "growth." The query ran fine. The conclusion is still wrong. You only catch that if you know the business well enough to get suspicious of a number that looks too good.

Decide what "active user" even means.
Someone has to actually define this. Logged in the last 30 days? Fired any event at all? Paying customer only? Get it wrong once, quietly, and two teams will spend the next year arguing about whose number is "correct," not realizing they're using different definitions.

Push back on a bad question.
A manager asks "why are we losing customers." That's not one query, it's like ten different ones, and picking the wrong one burns a week. A good analyst asks more questions before touching the keyboard. AI just answers whatever you literally typed - even if it's the wrong question, answered perfectly.

Notice the bug that gives a confident, wrong answer.
A join quietly duplicates rows and inflates a number by 40%. Nothing errors out. It just looks slightly off to someone who's been burned by this exact thing before, and they go dig into it. That gut feeling comes from experience, not from training data.

Actually own the decision.
Batch job or live streaming? Is this dashboard even worth the extra $2k a month in warehouse costs given who's actually looking at it? These are business calls wearing technical clothes. AI can list your options all day. It's not sitting in the budget meeting defending the choice.

None of that is pattern-matching. It's judgment you build by getting things wrong before, at this specific company, with this specific messy data. That's the one thing a model trained on everyone's data and belonging to no one - just can't have.

So who's actually in trouble here?

The person whose whole pitch is "I write SQL fast." If that's genuinely all you bring, you're competing with something faster, cheaper, and awake at 3am. Not a hot take, just math.

The junior who uses AI to skip the boring stuff and spends that extra time in meetings, arguing about what a metric should mean, or getting suspicious of numbers that look too clean — that person just got a huge boost. Not a threat to them at all.

The part worth actually talking about

You used to build that judgment slowly. Write a bad join, watch a dashboard break, get grilled about it in standup, learn from it. Painful, but it worked. It was basically an apprenticeship disguised as boring ticket work.

AI just ate the boring ticket work. Which means that apprenticeship juniors used to go through without even thinking about it is disappearing. Telling someone "you'll pick up the instincts eventually" isn't much help if the reps that used to build those instincts don't exist anymore.

So the real question isn't "will AI replace data analysts." It's: if the boring reps are gone, how does a junior actually build that "something's off with this number" instinct on purpose, instead of just hoping three years of grunt work hands it to them?

That one's worth an actual answer. "You'll figure it out eventually" doesn't cut it anymore.

Shoutout to @alexvoste for the original C-Tier/S-Tier take that kicked this whole conversation off. If you haven't seen his stuff - he's the guy building ForgeZero, a build toolchain in Go that's reportedly beating Ninja by a wide margin on large task sets. Worth a follow if you care about performance work at the systems level.

The Art of Cloud Survival: The Day Monitoring Failed First

Wed, 15 Jul 2026 12:13:04 +0000

When Everything Looked Fine

Modern cloud systems are built to survive failure.

Applications run across multiple Availability Zones. Containers restart automatically. Serverless functions scale on demand. Databases replicate data continuously. Entire architectural patterns have emerged around the idea that failure is not an exception but an expected part of operating software at scale.

Yet some of the most painful incidents begin long before a server crashes or a service becomes unavailable.

They begin when nobody realizes something is already going wrong.

Imagine a typical weekday morning. Dashboards are open on a large monitor. Every graph appears healthy. CPU utilization is well below critical thresholds. Memory consumption is stable. No alarms are firing. Error rates remain low.

From an operational perspective, everything appears normal.

Meanwhile, users are experiencing increasing delays. Transactions that usually complete in milliseconds now take several seconds. Some requests time out entirely. Support tickets begin arriving. Customer complaints appear in chat channels. Teams start investigating because users are reporting problems, not because monitoring systems detected them.

The infrastructure is functioning. The monitoring strategy is not. And that distinction matters more than many teams realize.

The Green Dashboard Problem

One of the most dangerous moments in operations is staring at a dashboard filled with green indicators while customers are actively experiencing issues.

Most engineering teams invest heavily in monitoring. They collect metrics, configure alarms, centralize logs, and build detailed dashboards. The intention is good. Visibility is essential in cloud environments.

The challenge is that many dashboards focus on infrastructure health rather than service health.

A virtual machine can be healthy while an application is struggling.
A database can be available while queries are becoming progressively slower.
A queue can continue accepting messages while consumers fall further behind every minute.

In each case, the system is technically running. Yet the user experience is already deteriorating.

Distributed systems make this problem even harder to detect because failures rarely appear as obvious outages. Instead, they emerge gradually, hidden inside latency, retries, timeouts, and backlogs.

Even when the dashboard remains green, the customer experience does not.

Why Distributed Systems Hide Failure

In a monolithic application, a failure is often visible and immediate. If a critical component stops working, the impact is usually straightforward to identify.

Distributed systems behave differently.

A slowdown in one service can propagate through multiple dependencies. An overloaded database may increase response times. Those delays cause application servers to wait longer. Waiting requests consume resources. Retries begin accumulating. Message queues start growing. Additional traffic increases pressure on downstream services.

Nothing has technically failed.

Yet the system is moving steadily toward an incident.

This is one reason distributed architectures are both powerful and challenging. Their resilience often comes from decoupling components, but that same decoupling can make problems difficult to observe.

Symptoms appear far away from the original cause.

By the time users notice, the underlying issue may have been developing for hours.

Monitoring Tells You What You Expect

Many teams use the words monitoring and observability interchangeably.

They are related, but they are not the same thing.

Monitoring helps answer questions we already know to ask. It tells us whether predefined conditions have been met.

Is CPU utilization above 80 percent?
Is memory consumption increasing?
Has an error threshold been exceeded?

These are valuable questions. But real incidents rarely follow predefined scripts.

Observability focuses on understanding what is happening inside a system, especially when the problem is unexpected. Instead of simply collecting metrics, observability provides enough context to investigate unknown behaviors and discover why a system is acting differently than expected.

This distinction becomes increasingly important as architectures grow more complex.

A dashboard showing resource utilization may indicate that everything is healthy. Observability helps determine whether customers would agree.

The Signals That Actually Matter

One of the most influential ideas in modern operations comes from Google's Site Reliability Engineering practices: the Four Golden Signals.

These signals focus on the aspects of a system that most directly affect users.

Together, these metrics provide a much more meaningful view of system health than infrastructure metrics alone.

Consider a payment processing service.

Suppose CPU utilization remains below 40 percent and memory usage appears stable. Traditional infrastructure monitoring would suggest the service is healthy.

Now imagine database response times increase from 50 milliseconds to two seconds.

Customers begin abandoning transactions. Requests start timing out. Revenue is affected.

The infrastructure never appeared unhealthy. The customer experience did. And this is why effective monitoring begins with user-facing outcomes rather than resource consumption.

The Danger of Averages

Another common mistake is relying too heavily on averages. Average response times often create a comforting but incomplete picture. A service may report an average latency of 200 milliseconds while a small percentage of requests take several seconds to complete. Those slower requests may represent only a fraction of total traffic, yet they are often the first indication that something is wrong.

For the affected users, the average is irrelevant.

Their experience is defined by the request that took five seconds, not by the thousands that completed successfully.

This is why mature operations teams pay close attention to percentile measurements such as p95 and p99 latency.

These metrics reveal what is happening at the edges of the system, where performance problems frequently emerge first.

Many major incidents begin as small increases in tail latency long before average metrics show meaningful changes.

If teams monitor only averages, they may discover problems far later than they should.

Failures Rarely Arrive All at Once

Hollywood teaches us to expect disasters as sudden events. Real-world cloud incidents are usually much less dramatic.

Most failures arrive gradually.

A dependency becomes slower.
A third-party API starts returning intermittent errors.
A queue begins accumulating messages.
Consumers struggle to keep pace.
Retries increase traffic.
Latency spreads through the system.

What started as a minor degradation becomes a significant operational issue.

This is one reason observability is so important. Detecting a problem early often means the difference between a brief investigation and a lengthy outage.

The sooner a team can identify unusual behavior, the more options they have to contain the impact.

Cloud survival is rarely about preventing every failure, it is about recognizing failures before they become incidents.

Building Visibility on AWS

AWS provides a rich set of services that help teams understand what is happening inside distributed systems.

Amazon CloudWatch serves as the foundation for metrics, logs, dashboards, and alarms. It enables teams to monitor operational signals across applications and infrastructure.
AWS X-Ray helps trace requests as they move through multiple services, making it easier to identify bottlenecks and understand where latency is introduced.
Amazon EventBridge can automate operational workflows, trigger responses to specific events, and help reduce the time between detection and action.

Used together, these services create visibility across layers of a distributed architecture.

But tools alone do not guarantee observability. Organizations often spend months building dashboards while investing very little effort in defining what success or failure actually looks like from a user's perspective. Technology is only part of the solution. The real challenge is deciding what deserves attention.

The Question Every Team Should Ask

There is a simple question that every engineering team should periodically ask:

"If a customer experiences a serious problem right now, would we know before they tell us?"

The answer is often uncomfortable.

Many teams discover that their alerts focus on infrastructure while customer-impacting issues remain invisible until support tickets arrive.

That realization can be frustrating, but it is also valuable.

Because once teams understand the gap between system health and customer experience, they can begin designing monitoring strategies that reflect reality instead of assumptions.

The goal is not to collect more metrics, the goal is to collect the right ones.

Cloud Survival Requires Visibility

The most resilient cloud architectures are not always the ones with the most automation, the most services, or the most sophisticated deployment pipelines.

Often, they are the ones that provide the clearest visibility into what users are actually experiencing.
Failures are inevitable. Latency will increase. Dependencies will degrade. Services will behave in unexpected ways. Distributed systems guarantee that complexity will eventually surface. What separates resilient teams from struggling ones is not their ability to avoid failure.

It is their ability to see it.

The day monitoring fails first is usually the day a team learns an important lesson: collecting metrics is not the same as understanding a system. Cloud survival requires both.

Because you cannot respond to a problem you cannot see.

The Art of Cloud Survival: When Retries Become the Outage

Thu, 25 Jun 2026 13:27:41 +0000

When Retries Become the Outage

At 2:17 AM, a payment service started returning errors. Not many. Just enough to attract attention. The dashboards didn't look alarming. CPU utilization was stable. Memory consumption was within normal limits. Infrastructure health checks were passing. If you had opened the monitoring console at that moment, you probably would have assumed the issue was temporary and would resolve itself within a few minutes.

The engineering team began investigating. Meanwhile, the rest of the system was already reacting. API clients started resubmitting requests. Lambda functions retried failed executions. Messages that couldn't be processed returned to queues. Downstream services attempted the same operations again, hoping the next attempt would succeed.

None of these actions were wrong. In fact, every component was behaving exactly as it had been designed to behave. And that was the problem.

What began as a relatively small service degradation gradually evolved into a much larger incident. The original failure affected a limited number of requests. The recovery mechanisms multiplied that traffic until the struggling service was forced to handle far more load than it had seen before the outage began.

The system wasn't collapsing because of the initial error anymore. It was collapsing because everybody was trying to help.

Why Retries Feel Like the Right Answer

Retries are one of the first resilience patterns most engineers learn, and for good reason.

Distributed systems operate in an imperfect world. Networks experience brief interruptions. Services become temporarily unavailable. Databases occasionally take longer than expected to respond. A request that fails now may succeed a few seconds later without any intervention.

Because of this, retries often improve reliability. A customer never notices the timeout. The application recovers automatically. Everybody wins. The trouble starts when we focus on the success story and forget to think about scale. A single retry seems harmless. Ten thousand retries are something else entirely.

Imagine a service receiving 10,000 requests during a busy period. Suddenly, a dependency starts responding slowly. Requests begin timing out. Each client retries three times before giving up.

Without realizing it, the system has transformed 10,000 requests into 40,000. That additional traffic creates more pressure on the dependency. More pressure leads to higher latency. Higher latency generates more failures. More failures trigger more retries. What looked like a recovery mechanism has become a feedback loop. And feedback loops are where distributed systems become dangerous.

The Real Problem Isn't Failure

One of the hardest lessons in cloud architecture is that systems rarely fail where the problem starts.

A database slows down.
An API begins timing out.
A message consumer falls behind.

Those events are usually manageable.

The real challenge comes from the chain reaction that follows. Every service makes a reasonable decision based on its local perspective. Every team implements protections intended to improve reliability. Every retry policy makes sense when viewed in isolation.

Yet when an incident occurs, those independent decisions begin interacting with each other. The result is often a level of complexity that nobody anticipated during design reviews. I've seen incidents where the root cause was resolved within minutes, but the platform continued struggling for hours because retries, reprocessing, and recovery workflows kept generating load long after the original issue had disappeared.

The failure wasn't the hardest part. Recovering from the recovery was.

When AWS Does Exactly What You Asked It To Do

AWS services are designed to help us build resilient systems, which is why many of them include retry mechanisms by default.

Take Lambda and SQS. When a Lambda function cannot successfully process a message, that message doesn't simply disappear. It becomes available again and can be processed later. This behavior protects against transient failures and prevents data loss. Most of the time, that's exactly what we want.

However, if the underlying problem is persistent—a broken dependency, an invalid payload, or a misconfigured service—the same message may be processed repeatedly while consuming compute resources and generating additional pressure on downstream systems. The same principle appears in event-driven architectures.

Services such as EventBridge improve reliability by attempting redelivery when consumers fail. That's an incredibly valuable capability. However, it also means that consumers must be designed with duplicate events in mind. Retries improve resilience. But resilience is never free.

Every retry consumes resources, generates traffic, and increases complexity. The question is not whether retries are good or bad. The question is whether the system can absorb the consequences when retries occur at scale.

Designing Retries That Help

Over the years, I've become less interested in whether a system retries and more interested in how it retries. The difference matters. One of the simplest improvements is exponential backoff.

Instead of retrying every second, the delay increases after each attempt. A request may wait one second, then two, then four, then eight. This gives dependencies time to recover and reduces the likelihood of overwhelming a struggling service.

Even then, another challenge remains. Imagine a million clients all deciding to retry after exactly eight seconds. Eight seconds later, they arrive together. The system experiences another traffic spike.

This is why mature systems often introduce jitter, adding a small amount of randomness to retry timing. The overall volume remains similar, but the requests become distributed over time rather than arriving in synchronized waves.

Another valuable pattern is the circuit breaker. When a dependency is clearly unavailable, continuing to send requests often provides little benefit. A circuit breaker temporarily stops traffic, allowing the dependency to recover while protecting the rest of the system from unnecessary load.

And sometimes, the most responsible decision is not to retry immediately at all.

Sometimes the correct answer is to move the failed message to a Dead Letter Queue, investigate the cause, and replay it later under controlled conditions. That approach may feel less aggressive than continuous retries, but it often prevents a localized issue from becoming a platform-wide incident.

A Different Way to Think About Resilience

Early in my career, I thought resilience was mostly about making systems persistent.

If something failed, try again.
If it failed again, try harder.

Experience changed that perspective.

Today, I think resilience is often about restraint. It's about understanding that every recovery mechanism introduces new behavior into the system. It's about recognizing that retries are not free. It's about accepting that the healthiest response to failure is not always immediate action. Sometimes the best thing a system can do is wait. Sometimes it's stop. Sometimes it's fail gracefully.

The most resilient cloud architectures aren't the ones that retry forever. They're the ones that understand the difference between recovering from a problem and amplifying it. Because in distributed systems, the outage doesn't always begin with the first failure. Sometimes it begins with the second attempt.

The Data Engineer’s Sandbox: Architecting for Autonomy with LXD

Wed, 25 Mar 2026 14:21:54 +0000

As Data Engineers, we often live in the cloud. Our daily lives are governed by S3 buckets, Glue jobs, and Kafka clusters. But there is a silent pressure that every senior professional feels: the cost of experimentation.

How many times have you hesitated to spin up a resource just to validate a simple schema or a transformation logic? We've all been there—balancing the need for agility with the responsibility of cost-awareness.

I’ve always believed that autonomy is a superpower. True technical mastery comes when you can prototype "blindly," without an internet connection, directly on your machine, and with zero impact on your company's AWS bill.

Why I moved away from heavy VMs

For years, the choice was either a heavy Virtual Machine (VM) that drained my laptop’s RAM or a Docker container that, while fast, sometimes lacked the persistence and system-level isolation I needed for a long-term practice environment.

When I started working with Ubuntu 24.04, I found the perfect middle ground: LXD. It offers the "system feel" of a VM with the "instant speed" of a container. It became my go-to for building local SQL sandboxes where I can fail fast, learn deep, and spend zero.

The Blueprint: Setting up your Local Engine

The goal is simple: a functional database server running in minutes.

The Foundation

We start by initializing the hypervisor to manage our internal networking.

sudo snap install lxd
sudo lxd init --auto

Launching the Instance

We choose a lightweight Ubuntu image as our dedicated server.

lxc launch ubuntu:24.04 sql-practice

The "Hardware" Setup

We enter the container and transform that empty shell into a robust PostgreSQL engine.

lxc exec sql-practice -- bash
apt update && apt install -y postgresql postgresql-contrib

From Infrastructure to Impact

Setting up the "pipes" is only half the battle. As engineers, we care about the data. In my experience building event-driven architectures and streaming pipelines with Kafka, I’ve learned that a local sink is the best place to test idempotency and schema evolution before touching a production environment.

Using the COPY command for massive local ingestion or pushing CSV files directly into the container (lxc file push) are the types of "unsexy" but highly efficient workflows that define a professional.

Closing thoughts: Design first

My philosophy is simple: Systems must be resilient and observable from day one. That includes our local dev environments. Complexity is a cost, and simplicity scales better.

By mastering these local sandboxes, we aren't just saving a few dollars on an AWS bill; we are reclaiming our creative freedom as builders.

About the Author

I am Rocío Baigorria, a Senior Data Engineer and co-leader of AWS Girls Argentina. I specialize in building "institutional-grade" data platforms where reliability, cost-efficiency, and business impact are core design principles.

Connect with me on LinkedIn: https://www.linkedin.com/in/rociobaigorria/

Explore my Resources: https://tr.ee/2t-m587551

From Delayed Data to Real-Time Decisions: Building an Event-Driven Pipeline on AWS

Thu, 19 Mar 2026 21:11:10 +0000

At 2:17 AM, the dashboard wasn’t moving.

Transactions were still coming in, but the numbers on the screen were frozen. The system was running, but the data was already 10 minutes behind.

At that scale, 10 minutes wasn’t just delay — it meant decisions based on outdated information.

That was the moment everything had to change.

The Problem: When Data Arrives Too Late

In many systems, data pipelines are built around batch processing or synchronous APIs. They work — until they don’t.

As traffic grows, these systems start to break in predictable ways:

Increasing latency
Bottlenecks in database reads
Inconsistent data across services
Delayed analytics and reporting

The core issue is simple:
the system reacts too late.

To solve this, the architecture had to shift from request-driven to event-driven.

The Shift: Thinking in Events

Instead of asking for data, services react to events as they happen.

Every transaction becomes an event. Every state change is recorded as a fact.

This changes everything:

Systems become asynchronous
Components are loosely coupled
Scalability comes from partitioning, not vertical scaling

The goal was clear:

Process more than 15,000 events per second with sub-50ms latency — reliably.

Designing the Pipeline

The first version of the system was built as a distributed architecture using open-source tools.

Core components:

- Apache Kafka for event streaming

- Kafka Streams for real-time processing

- Spring Boot for processing services

- PostgreSQL for durable storage

- Redis for low-latency reads

- Prometheus and Grafana for observability

Event Flow

The pipeline follows a simple but powerful structure:

Producer → Kafka → Stream Processing → Storage → Analytics

Producers publish transaction events
Events are serialized and validated
Kafka distributes them across partitions
Stream processors handle transformations and aggregations
Results are stored and made available for real-time queries

This allowed continuous processing instead of waiting for batches.

The Challenges (Where Systems Usually Break)

Designing the pipeline was only part of the work. Making it reliable at scale required solving real issues:

Event Duplication
Ensuring exactly-once processing required transactional guarantees and careful handling of offsets.
Latency Spikes
Under heavy load, consumer lag increased. This was mitigated with parallel consumers and optimized batching.
Throughput Optimization
Producer batching (32KB) and Snappy compression significantly improved performance.
Resource Bottlenecks
Connection pooling and efficient serialization reduced pressure on downstream systems.

These are the kinds of problems that don’t show up in diagrams — only in real systems.

Results

After multiple iterations and optimizations, the system achieved:

Throughput: 15,000+ events per second

Latency (P99): under 50ms

Availability: 99.95%

Data loss: 0% (exactly-once processing)

More importantly, this enabled near real-time anomaly detection.

What used to take minutes could now happen in milliseconds.

Moving to AWS: Same Architecture, Less Infrastructure

Once the system was stable, the next step was moving to the cloud.

The key decision was not to redesign the system — only to replace infrastructure with managed services.

Cloud Architecture:

Event ingestion via Amazon EventBridge or Amazon MSK
Processing with AWS Lambda
Workflow orchestration using AWS Step Functions
Storage with DynamoDB or Amazon RDS
Monitoring through Amazon CloudWatch

Each component maps directly to the original architecture.

The logic stays the same. The operations overhead disappears.

The Key Insight

The most important lesson wasn’t about tools.

It was about understanding the system.

Event-driven architectures are built on a few core principles:

Events are immutable
Systems react asynchronously
Scalability comes from partitioned streams
State is derived from event logs

Once these are clear, moving between technologies becomes straightforward.

Final Thoughts

Real-time systems are not about speed alone.

They are about enabling better decisions.

When data arrives late, the system might still work — but the business doesn’t.

Designing event-driven pipelines is ultimately about reducing the gap between what happens and what the system knows.

Design, therefore I exist.

The Senior Engineer's Blueprint: Architecting Data Pipelines for Business Growth

Sat, 07 Mar 2026 11:31:47 +0000

What Separates a Junior Engineer from a Senior Data Engineer?

In this blueprint, I'm going to reveal how senior engineers architect for the "probable future," and the specific business metrics you need to build into your pipelines from day one.

The difference

The junior mindset: "How do I move this data from A to B as efficiently as possible?"

The senior mindset: "Why are we moving this data, and what business questions will the executive team need this data to answer six months from now?"

That's the main difference. The junior is focused on the present, while the senior is focused on the future.
To close this gap:

Shift your mindset to think like a senior engineer.
Develop business intelligence skills and think like a business analyst and the business owner.

To develop business intelligence skills, you need to know what the business goal is. They shift based on the directin of the current but what stays constant are:

To make more money
To get more customers
To spend less

And your job is assist in answering the how and why questions.(a.k.a solving problems)

Architecting for the Probable Future

Pipelines fail at predictive analytics when built purely for historical reporting. Stop capturing just what happened today. Start designing schemas that predict tomorrow.

Below are the core metrics and specific table attributes every data engineer should build into their models.
PS: Calculated fields are left for dimensional modeling, not for relational modeling.

1. Financial Metrics: Following the Money

A pipeline must provide real-time visibility into margins, not just revenue.

Gross Profit Analysis

gross_revenue_amt / total_amt: Top-line sales before deductions.
cogs_amt: Cost of Goods Sold.
discount_amt: Applied promotions or coupons.
net_margin_pct: Real-time profitability metric ((Gross Revenue - COGS - Discounts) / Gross Revenue).

Frozen Money & Dead Stock

days_inventory_on_hand_qty: How long cash is tied up in physical products.
shrinkage_rate_pct: Loss due to theft, damage, or error.
is_dead_stock_flg: Boolean identifying unmoving inventory.

Surge Management & Revenue Alerts

revenue_surge_velocity_amt: Rate of incoming revenue to trigger auto-scaling or business alerts.
abandoned_cart_value_amt: Immediate lost revenue during traffic spikes.

2. The Customer Metrics: Predicting Human Behavior

Don't wait for a cancellation event. Build architectures that catch warning signs early to identify reasons for customer order cancellations.

Customer Turnover (Churn)

days_since_last_login_qty: Leading indicator of drop-off.
email_open_rate_pct: Engagement health.
predicted_churn_risk_score: ML-derived metric based on usage deceleration. (track every login, email click, and page view).

Customer Lifetime Value (CLV)

historical_clv_amt: Total historical customer spend.
avg_order_value_amt: (AOV) Trailing 12-month average transaction size.

Customer Sentiment & Customer Satisfaction Score

csat_score_num: Post-purchase satisfaction score (1-5).
support_ticket_cnt: Number of active or historical complaints.

3. Operational Efficiency: Inventory Management and Workforce Performance

Your dimensional models must natively support core company KPIs.

Inventory Management

inventory_turnover_ratio: How often inventory is sold and replaced.
gmroi_pct: Gross Margin Return on Investment.(track buying cost and sale price)
sell_through_rate_pct: Percentage of received inventory actually sold.

Workforce Performance

sales_per_employee_amt: Revenue efficiency capability.
employee_turnover_rate_pct: Organizational health indicator.(How often employees leave the company)
support_resolution_time_mins: Productivity metric for customer service reps.(how long it takes to resolve a support ticket)

4. Operational Cost

Tracking the cost of acquiring and maintaining business.

customer_acquisition_cost_amt: (CAC) Blended cost to acquire a net-new customer. (track every customer acquisition event, including marketing spend, sales, and customer service interactions).
return_on_ad_spend_pct: (ROAS) Marketing revenue vs. advertising cost.
promotion_effort_cost_amt: Cost of active campaigns and outreach per user. (track discounts and promotions )

Taking the Next Step

To help you skip the trial-and-error phase, I’m working on high-value technical resources to level up your engineering:

Premium Design Templates: Plug-and-play architectures and product reader dimensional data models for the modern data stack.
The "Every Table" Metrics Handbook: A guide detailing the exact audit columns, SCD strategies, and hidden metrics you should proactively include on every single table you deploy.
Building an Analytics Warehouse from Scratch (Free Walkthrough): In an upcoming post, I’ll provide a complete technical walkthrough on web scraping your own data(or using available data) and building a warehouse from the ground up, for free.

Make sure you're subscribed so you don't miss these upcoming drops.

Conclusion

Let's build up the habit of not just asking "what's the easiest way to ingest this?", but mainly "How will this data be used to save or make X money?" and "Why is this important to the Y?"

That is how you become a senior engineer.

Drop a comment below with the biggest architectural mistake you made early in your career.

Coder Legion - Recent posts tagged data-engineering

Auto CDF Moved the CDC Tax From Writes to Reads: Delta CDF vs Auto CDF vs Spark 4.2 CHANGES

Three ways to get the changes

The mechanism, in two paragraphs

What I measured

What the numbers say, and what they do not

Gotchas to check before you run UNSET

Decision table

What I would do this week

Hardware & versions

Sources

Backtesting live football on Polymarket: 6 traps in a recorded order book

1. You have to record the book yourself, then check the recording

2. Events arrive out of order

3. The mid is not a price you can trade

4. The "level score" can be fake too

5. Daily batch files overlap

6. The fastest entry is not reachable

Reproduce it

What this does not show

Backtesting a Polymarket copy trading bot without fooling yourself

1. Define a copy you could actually execute

2. Pick on one period, measure on the next

3. Check whether rank carries over at all

4. Look at the baseline before the winners

5. The trap: wallet labels computed today

6. Make it reproducible

What this does not show

Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew

Part 1. Design for the restart: foreachBatch, watermarks, and MERGE

Contract 1: foreachBatch is at-least-once

Contract 2: watermark dedupe has a state lease (and you must finish the batch)

Contract 3: MERGE inside foreachBatch is a logical idempotency problem

Part 1 checklist (print this next to the PR)

Part 2. Arrow is not a free pass: applyInPandas vs mapInPandas / mapInArrow

Three APIs, three memory contracts

Skew turns "elegant" into OOM

Prefer batch maps when you do not need whole-group semantics

Decision table

How the two themes connect

Optional diagram (for a cover or inline figure)

References

Fabric Runtime 2.0 is becoming the default. Here is what actually breaks.

Fabric Runtime 2.0 is becoming the default. Here is what actually breaks.

1) Hook: the default is the migration

2) Wrong assumption: GA means notebooks just run

3) Wrong assumption: NEE = free 6x

4) How to see the truth in the plan

5) Delta 4.x trap for multi-workload tables

6) Practical migration pattern

7) CI/CD angle: pin the runtime like a toolchain

8) Close: benchmarks sell the ceiling, production pays the median

Takeaways

Sources

Azure Synapse to Databricks: A Migration Playbook That Survives Production

Azure Synapse → Databricks: A Migration Playbook That Survives Production

Table of contents

1) Why this migration is harder than the slide deck

2) What you are actually migrating

3) Program shape: discovery → waves

Phase A — Discovery

Phase B — Assessment

Phase C — Design (usually hybrid)

Phase D — Pilot (lighthouse)

Phase E — Waves

4) Four parallel workstreams

Sequencing tip that builds trust

5) Data movement: skip the JDBC tax

Why this path wins

Example — CETAS from Synapse Dedicated SQL Pool

Example — land as Delta on Databricks

Physical design: do not cargo-cult Synapse

6) Code conversion: syntax is easy, semantics bite

Common remaps

Semantic trap #1 — case sensitivity

Semantic trap #2 — stored procedures

Semantic trap #3 — TRY/CATCH vs workflow design

7) Dimensional patterns on Delta (SCD2)

Example pattern — expire then insert (illustrative)

8) Reconciliation & cutover

Part 1. Design for the restart: `foreachBatch`, watermarks, and MERGE

Contract 1: `foreachBatch` is at-least-once

Contract 3: MERGE inside `foreachBatch` is a logical idempotency problem

Part 2. Arrow is not a free pass: `applyInPandas` vs `mapInPandas` / `mapInArrow`