menu

Recent posts tagged dataengineering

Level UI • 18 hours • in Articles • 9 min read
An MCP server is not a new API. It is a thin layer over the one you already have
Somewhere in the last year, "do we need an MCP server?" joined the list of questions that arrive in a brief before anyone has said what the thing is for. It is usually framed as MCP vs API, as though the Model Context Protocol were a newer kind of AP...
post-cover-33074
  • ai
  • api
  • webdev
  • security
7 Reactions 7 Comments
firatcelik • Oct 22 • in Articles • 14 min read
DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap
Originally published on DEV.to: https://dev.to/firfircelik/duckdb-20-vs-polars-vs-spark-on-one-machine-150m-rows-under-a-45-gib-cap-h38 DuckDB 2.0.0 shipped on October 21, 2026 announcementhttps://duckdb.org/2026/10/21/announcing-duckdb-200, PyPIhtt...
post-cover-32051
  • duckdb
  • dataengineering
  • python
  • spark
3 Reactions 1 Comments
Nyachaki • Oct 18 • in Articles • 13 min read
Advanced SQL Analytics for Data Analysis.
Advanced SQL analytics for running a hospital database. Introduction Structured Query Language is essential and common in most industry sectors. In healthcare reporting, SQL plays a critical role in analytics. PostgreSQL lets data scientists take ...
post-cover-31618
  • database
  • datascience
  • dataengineering
  • sql
2 Reactions 1 Comments
firatcelik • Oct 12 • in Articles • 16 min read
Python 3.15 Is Not a Speed Release for Data Teams. It Is an Operations Release.
Originally published on DEV.to: https://dev.to/firfircelik/python-315-is-not-a-speed-release-for-data-teams-it-is-an-operations-release-5fbo Python 3.15.0 shipped on October 9, 2026 release posthttps://blog.python.org/2026/10/python-3150-final-is-he...
post-cover-30776
  • python
  • dataengineering
  • performance
  • security
1 Reactions 0 Comments
Shailesh • Oct 9 • in Articles • 16 min read
Instant at 10 Million Rows, Dead at a Billion: A Decision Framework for Billion-Row SQL
!Abstract brand-toned data pathway backdrop for the billion-row SQL decision framework.https://unrivaled-starlight-d655be.netlify.app/blog/visuals/billion-row-sql-hero.png?v=2 A query that returns in 40 milliseconds on ten million rows can time out...
post-cover-30403
  • database
  • sql
  • dataengineering
  • architecture
2 Reactions 0 Comments
Lakshmi Mahabaleshwara • Oct 8 • in Articles • 6 min read
Understanding Parquet File Format
What is a File Format? A file format defines how data is structured and stored so that software can read and write it consistently. Think of it as the container for your data. Just like music comes in MP3, WAV, or FLAC, data files also come in dif...
post-cover-30266
  • dataengineering
  • datalakehouse
  • apache-parquet
  • bigdata
2 Reactions 2 Comments
Nyachaki • Oct 7 • in Articles • 8 min read
# Power BI Data Modelling: A Great Path to Great Analysis
Data Modeling in Power BI Introduction At a time when data is rapidly generated by different activities ranging from experiments to business, more often do we get data in different formats and structures. The data may or may not be related dependi...
post-cover-30210
  • powerfuldevs
  • datascience
  • dataengineering
  • database
2 Reactions 0 Comments
Lakshmi Mahabaleshwara • Sep 26 • in Articles • 7 min read
Data Preprocessing for Machine Learning
Introduction Most ML tutorials jump straight into models and algorithms. But here is what no one tells beginners: the model isn't where the magic happens. The data preparation is. Think of it this way. You wouldn't hand a toddler a jigsaw puzzle w...
post-cover-28690
  • machinelearning
  • dataengineering
  • artificialintelligence
8 Reactions 4 Comments
Jayasree • Sep 17 • in Articles • 4 min read
How I Used Wren AI to Let My Marketing Team Query a Star Schema Without Writing SQL
If you've worked in data engineering long enough, you know the drill. Someone from marketing pings you on a Monday morning: "Hey, can you pull campaign spend by ad set for last week?" You run the query. Send it. Five minutes later: "Can we also add...
post-cover-27479
  • data
  • dataengineering
6 Reactions 2 Comments
Lakshmi Mahabaleshwara • Sep 15 • in Articles • 4 min read
The Core Paradigms of Machine Learning
Introduction Most machine learning explanations start with algorithms. In this blog, let's start with a simpler question: how does anything learn? At its core, machine learning isn’t magic; it’s a set of strategies for turning data into behavior. ...
post-cover-27243
  • machinelearning
  • ai
  • aimodels
  • dataengineering
3 Reactions 1 Comments
Tom Smithverified • 15 hours • in Articles • 9 min read
Developer Weekly Briefing — October 30, 2026
Security dominated this week, but not in the usual way. Two active campaigns turned GitHub configuration files into attack surfaces. Researchers demonstrated that the built-in browsers inside AI coding tools can read your SSH keys. And at Oracle AI ...
post-cover-33104
  • agenticai
  • devsecops
  • github
  • localai
6 Reactions 3 Comments
SuMiTa • Aug 31 • in Articles • 7 min read
The Engineering Debt Trap: Why "Working Code" Is Killing Modern Software Development
There is a quiet crisis unfolding in modern software architecture. On the surface, features are shipping faster than ever. Continuous Integration and Continuous Deployment CI/CD pipelines push updates dozens of times a day. Artificial intelligence to...
  • dataengineering
  • coding
0 Reactions 0 Comments
Basavaraj-Shepur • Aug 6 • in Articles • 2 min read
How Open Embedding Models Can Match GPT-Class Retrieval Quality
RAG Retrieval-Augmented Generation - a pattern where you fetch relevant documents before generating an answer lives or dies on retrieval quality. A cheaper retrieval stack isn't always a compromise. The Retrieval Quality Gap Is Often in the Chunking...
post-cover-24206
  • llm
  • rag
  • agenticai
  • dataengineering
1 Reactions 1 Comments
seosiri • Jul 28 • in Articles • 1 min read
🚀 Open Sourcing etl-pipeline-mcp: A Local-First, Decoupled Data Pipeline for AI Agents
Hey CoderLegion community, When connecting LLMs or AI agents directly to production databases or third-party APIs like HubSpot, Stripe, or GitHub, two major problems usually pop up: Unsanitized PII/PHI leaks right into the model's context. Fragmented...
post-cover-23678
  • mcp
  • dataengineering
  • database
  • data
0 Reactions 0 Comments
Makansgreat • Jul 16 • in Articles • 7 min read
Deploying My First Ubuntu Virtual Machine on Microsoft Azure: SSH Access, Apache2 Installation, and Web Server Configuration"
Introduction Microsoft Azure Virtual Machines VMs provide scalable cloud computing resources that allow users to deploy and manage operating systems in the cloud. In this project, I created an Ubuntu 22.04 Virtual Machine in Azure, connected to it u...
post-cover-22810
  • beginners
  • linux
  • azure
  • dataengineering
2 Reactions 1 Comments
tuni56 • Jul 2 • in Articles • 2 min read
When "Latest" isn't Actually Latest
When "Latest" Isn't Actually Latest TL;DR In distributed systems, the last record your system processes is not always the most recent event that happened in reality. Understanding the difference between Event Time and Processing Time is essential...
post-cover-21755
  • dataengineering
  • distributed-systems
  • system-design
  • kafka
1 Reactions 1 Comments
petascalelabs • Jun 27 • in Articles • 7 min read
The Data Engineer Roadmap for 2026 (in an AI-Native World)
> This is the narrated version of our free, interactive Data Engineer Roadmaphttps://petascalelabs.com/data-engineer-roadmap. Same areas, same order, with a focus on the one thing each layer asks of you that AI can't do for you. Every data engineer ...
post-cover-21446
  • dataengineering
  • career
  • roadmap
  • ai
0 Reactions 0 Comments
petascalelabs • Jun 27 • in Articles • 3 min read
The Story Behind Apache Iceberg's Format-Version: v1 to v4
Most engineers meet Apache Iceberg as a one-line answer: "it's the thing that gives you ACID transactions and time travel on object storage." That's true, and it's also where most people stop. But Iceberg has a version dial baked into every table - a...
  • dataengineering
  • interview
  • bigdata
  • opensource
3 Reactions 1 Comments
Flamehaven • Jun 23 • in Articles • 12 min read
Not Worthless. Just Not Found Yet. - On verification work that arrives before anyone asks for it
The shape of the work right now !1https://coderlegion.com/?qa=blob&qablobid=14808488737358335165 Zenodo views: 1. Downloads: 0. No external citations to the ledger to date. The code still runs. That is the uncomfortable shape of this work right ...
post-cover-21189
  • computerscience
  • dataengineering
  • governance
  • data-science
2 Reactions 1 Comments
Tanmay Rajesh Bhurkunde • Jun 23 • in Articles • 1 min read
Running Apache Airflow + Docker for Free Using GitHub Codespaces
While building my ETL pipeline project, I ran into a common beginner problem: Running Apache Airflow locally on Windows with Docker was painful. Problems included: Low disk space Docker setup issues Linux compatibility problems Environment deb...
post-cover-21185
  • dataengineering
  • docker
  • python
  • apacheairflow
2 Reactions 1 Comments
Tanmay Rajesh Bhurkunde • Jun 23 • in Articles • 1 min read
Why I Stopped Treating Job Applications as My Only Career Strategy
Like many engineers, I started my job search with a simple idea: Apply to enough roles and eventually something will work out. The reality was more complicated. Some positions were already filled. Some never responded. Some required significantly ...
post-cover-21184
  • softwareengineering
  • learninginpublic
  • careerdevelopment
  • dataengineering
1 Reactions 0 Comments
SuMiTa • Jun 18 • in Articles • 2 min read
Managing Database Schema Like Code: The Quest for Database Version Control
In modern software development, we take git for granted. We branch, we commit, we merge, and we roll back code seamlessly. But when it comes to the database layer, things get tricky. Databases have state—you can't just overwrite a production database...
post-cover-20877
  • databasedevops
  • dataengineering
  • webdevelopment
  • softwarearchitecture
15 Reactions 3 Comments
Page:
chevron_left
🔥 62 posts • 👩‍💻 30 active writers
✨ Want to write in this topic?
Share your learning. Get discovered. Build your profile.
Join CoderLegion
💡 Why developers use CoderLegion
Track learning • Share posts • Get ranked publicly
Create your profile