menu

Recent posts tagged data-engineering

Level UI • 17 hours • in Articles • 9 min read
An MCP server is not a new API. It is a thin layer over the one you already have
Somewhere in the last year, "do we need an MCP server?" joined the list of questions that arrive in a brief before anyone has said what the thing is for. It is usually framed as MCP vs API, as though the Model Context Protocol were a newer kind of AP...
post-cover-33074
  • ai
  • api
  • webdev
  • security
7 Reactions 7 Comments
firatcelik • 4 days • in Articles • 9 min read
Auto CDF Moved the CDC Tax From Writes to Reads: Delta CDF vs Auto CDF vs Spark 4.2 CHANGES
Originally published on DEV.to: https://dev.to/firfircelik/auto-cdf-moved-the-cdc-tax-from-writes-to-reads-delta-cdf-vs-auto-cdf-vs-spark-42-changes-9bb On September 1, 2026, Databricks made automatic change data feed Auto CDF generally available. T...
post-cover-32478
  • data-engineering
  • databricks
  • apache-spark
  • delta-lake
0 Reactions 0 Comments
orcalayer • Oct 14 • in Articles • 2 min read
Backtesting live football on Polymarket: 6 traps in a recorded order book
We recorded the order book of every Polymarket football soccer market for three weeks and tested two in-play ideas: buying a team right after a late price jump at a level score, and protecting goal bets with a stop-loss. Neither paid. Buying the jump...
post-cover-31052
  • polymarket
  • python
  • backtesting
  • data-engineering
6 Reactions 2 Comments
orcalayer • Oct 10 • in Articles • 3 min read
Backtesting a Polymarket copy trading bot without fooling yourself
Copy trading is the most common bot idea on Polymarket: find the wallets that win and buy what they buy. We tested it on five sports MLB, NFL, college football, League of Legends and Valorant and it did not survive. The full results are on our bloght...
post-cover-30590
  • polymarket
  • python
  • backtesting
  • data-engineering
1 Reactions 0 Comments
firatcelik • Oct 9 • in Articles • 8 min read
Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew
Originally published on DEV.tohttps://dev.to/firfircelik/two-pyspark-contracts-staff-des-learn-on-a-bad-tuesday-foreachbatch-restarts-and-applyinpandas-skew-39pa. Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked...
post-cover-30476
  • databricks
  • pyspark
  • python
  • data-engineering
0 Reactions 0 Comments
firatcelik • Oct 9 • in Articles • 7 min read
Fabric Runtime 2.0 is becoming the default. Here is what actually breaks.
Originally published on DEV.tohttps://dev.to/firfircelik/fabric-runtime-20-is-becoming-the-default-here-is-what-actually-breaks-12i9. Fabric Runtime 2.0 is becoming the default. Here is what actually breaks. Late September 2026 is when Fabric Runti...
post-cover-30475
  • fabric
  • spark
  • data-engineering
  • azure
0 Reactions 0 Comments
firatcelik • Oct 9 • in Articles • 10 min read
Azure Synapse to Databricks: A Migration Playbook That Survives Production
Originally published on DEV.tohttps://dev.to/firfircelik/azure-synapse-databricks-a-migration-playbook-that-survives-production-nge. Azure Synapse → Databricks: A Migration Playbook That Survives Production A field-style guide for Lead / Senior Dat...
post-cover-30474
  • databricks
  • azure
  • data-engineering
  • etl
2 Reactions 1 Comments
orcalayer • Oct 7 • in Articles • 3 min read
6 things that broke our Polymarket trading bot before it made a dollar
We spent a few weeks trying to build a profitable Polymarket bot. Arbitrage between linked markets, in-play market making, copying wallets that trade before goals, betting against miscalibrated prices. We recorded the full order book for every footba...
post-cover-30137
  • polymarket
  • python
  • api
  • data-engineering
3 Reactions 1 Comments
Tom Smithverified • 14 hours • in Articles • 9 min read
Developer Weekly Briefing — October 30, 2026
Security dominated this week, but not in the usual way. Two active campaigns turned GitHub configuration files into attack surfaces. Researchers demonstrated that the built-in browsers inside AI coding tools can read your SSH keys. And at Oracle AI ...
post-cover-33104
  • agenticai
  • devsecops
  • github
  • localai
6 Reactions 3 Comments
Alex • Sep 29 • in Articles • 12 min read
The IRS Handed Back 96.7% of the Money on Its AI Coding-Tools Order. The Record Says When, Not Why.
> Originally published at vibeagentmaking.comhttps://vibeagentmaking.com/blog/the-irs-handed-back-96-7-of-the-money-on-its-ai-coding-tools/. One IRS delivery order for AI coding and chat tools was obligated at $3.31 million on the last day of fiscal...
  • government
  • ai
  • data-engineering
  • procurement
0 Reactions 0 Comments
orcalayer • Sep 24 • in Articles • 10 min read
Polymarket Gamma API endDate Is Not the Resolution Deadline
Checked against the Polymarket Gamma API on 24 September 2026. Short answer. OrcaLayer compared the endDate field with the written rules on 23 Polymarket markets. Polymarket's documentation describes a market's end date as the point when it becomes ...
post-cover-28532
  • polymarket
  • python
  • api
  • data-engineering
10 Reactions 2 Comments
nevmenandr • Aug 6 • in Articles • 6 min read
Building NLP Systems for 96 Low-Resource Languages: A Web Scraping War Story
Or why language vitality online has nothing to do with speaker counts — and everything to do with data engineering Russia speaks dozens of minority languages — Tatar, Bashkir, Udmurt, and even tiny ones like South Yukaghir fewer than 50 speakers. ...
post-cover-22837
  • low-resource-nlp
  • web-scraping
  • data-engineering
  • language-models
2 Reactions 2 Comments
tuni56 • Aug 1 • in Articles • 3 min read
Foundation Models Are Not the Product: Why Data Engineers Are Becoming the Bridge to Generative AI
TL;DR Foundation Models are powerful, but they are not magic knowledge engines. They can generate text, summarize information, and answer questions because they understand patterns learned during training. However, they do not know your company's ...
post-cover-23952
  • gen-ai
  • artificialintelligence
  • data-engineering
  • data
4 Reactions 1 Comments
Tanmay Rajesh Bhurkunde • Jul 22 • in Articles • 1 min read
Understanding Python Virtual Environments: The Interpreter Mismatch That Caused My Imports to Fail
While setting up my Data Engineering learning environment, I ran into an issue that many Python developers eventually face. Every package installation completed successfully, yet importing those same packages consistently failed. Initially, I assum...
post-cover-23274
  • data-engineering
2 Reactions 1 Comments
nevmenandr • Jul 16 • in Articles • 3 min read
Your Dataset Is Your Foundation. Have You Checked If It's Crumbling?
Or why Wikipedia is the perfect Data Quality testing ground not just for linguists Every developer who has ever scraped Wikipedia for an NLP task thinks: "It's human-written text. That means the dataset is balanced, natural, and ready to feed into ...
post-cover-22826
  • data-engineering
  • machine-learning
  • data-quality
  • nlp-pipelines
3 Reactions 1 Comments
Jayson kibet • Jul 15 • in Articles • 4 min read
Is AI Coming for Data Analyst Jobs? Depends What You Actually Do All Day
Short answer: no. A bit longer answer: no, unless your whole job is just writing SQL. Someone started a thread with a good question: > "You said frontend is dying because it's 'pure pattern-matching' that AI can now do. As a Data Analyst, I see C...
  • data-analysis
  • data-science
  • data-engineering
  • ai
5 Reactions 4 Comments
tuni56 • Jul 15 • in Articles • 6 min read
The Art of Cloud Survival: The Day Monitoring Failed First
When Everything Looked Fine Modern cloud systems are built to survive failure. Applications run across multiple Availability Zones. Containers restart automatically. Serverless functions scale on demand. Databases replicate data continuously. Enti...
post-cover-22753
  • distributed-systems
  • cloud-architecture
  • data-engineering
3 Reactions 1 Comments
tuni56 • Jun 25 • in Articles • 5 min read
The Art of Cloud Survival: When Retries Become the Outage
When Retries Become the Outage At 2:17 AM, a payment service started returning errors. Not many. Just enough to attract attention. The dashboards didn't look alarming. CPU utilization was stable. Memory consumption was within normal limits. Infrastr...
post-cover-21369
  • distributed-systems
  • cloud-architecture
  • data-engineering
3 Reactions 2 Comments
tuni56 • Mar 25 • in Articles • 2 min read
The Data Engineer’s Sandbox: Architecting for Autonomy with LXD
As Data Engineers, we often live in the cloud. Our daily lives are governed by S3 buckets, Glue jobs, and Kafka clusters. But there is a silent pressure that every senior professional feels: the cost of experimentation. How many times have you hesit...
post-cover-13481
  • data-engineering
  • sandbox
  • sql
2 Reactions 0 Comments
tuni56 • Mar 19 • in Articles • 3 min read
From Delayed Data to Real-Time Decisions: Building an Event-Driven Pipeline on AWS
At 2:17 AM, the dashboard wasn’t moving. Transactions were still coming in, but the numbers on the screen were frozen. The system was running, but the data was already 10 minutes behind. At that scale, 10 minutes wasn’t just delay — it meant decisi...
post-cover-13228
  • data-engineering
  • aws
  • kafka
  • eventdriven
6 Reactions 2 Comments
Clifford .O. Potter • Mar 7 • in Articles • 3 min read
The Senior Engineer's Blueprint: Architecting Data Pipelines for Business Growth
What Separates a Junior Engineer from a Senior Data Engineer? In this blueprint, I'm going to reveal how senior engineers architect for the "probable future," and the specific business metrics you need to build into your pipelines from day one. The...
post-cover-12452
  • data-engineering
3 Reactions 1 Comments
To see more, click for the full list of posts or popular tags.
chevron_left
🔥 19 posts • 👩‍💻 8 active writers
✨ Want to write in this topic?
Share your learning. Get discovered. Build your profile.
Join CoderLegion
💡 Why developers use CoderLegion
Track learning • Share posts • Get ranked publicly
Create your profile