Coder Legion - Recent posts tagged roadmap

The Data Engineer Roadmap for 2026 (in an AI-Native World)

Sat, 27 Jun 2026 09:09:07 +0000

This is the narrated version of our free, interactive Data Engineer Roadmap. Same areas, same order, with a focus on the one thing each layer asks of you that AI can't do for you.

Every data engineer roadmap written before early 2025 made the same quiet assumption: that the hard part was writing the code. Learn SQL. Learn Python. Wire up a pipeline in Airflow. Ship it. Congratulations, you're a data engineer.

That assumption is dead. AI writes the SQL now. It writes the DAG, the PySpark job, the dbt model, the masking policy - and it writes them faster than you, at 2am, without complaining. If your roadmap is a checklist of tools to learn so you can produce that code, you're training for a race that has already been run.

So a 2026 roadmap has to be a different shape. Not "what do I learn so I can write a pipeline," but "what do I understand so I can tell whether the AI-written pipeline is right, and fix it when it isn't." That is a map of depth, not a list of tools.

The one idea that makes the whole map work

Most roadmaps draw becoming-senior as new areas appearing: the junior does SQL and dbt, the senior does Kafka and Spark and Kubernetes. That is not how it works.

A senior engineer works the same areas a junior does. The difference is how far into each one they go.

A junior knows Parquet is "the fast columnar format" and can partition a table. A senior reasons about row groups, page statistics, dictionary encoding, and why a scan cost what it cost. A junior writes a Spark job. A senior debugs its shuffle and its skew. Same topic, different altitude.

That matters more now than ever, because AI raises the floor to roughly the junior line. It reliably gets you the partitioned table and the working Spark job. The depth above that line is exactly the part it can't reason about for you, and exactly where your career value now lives.

So as we walk the areas, watch for the pattern: AI does the surface; you own the depth.

Foundations and SQL

Joins, window functions, CTEs, Python, the command line, Git, ETL vs ELT.

AI writes almost all of this now. That doesn't make SQL optional, it makes it table stakes. You learn it not to produce it, but to catch when the generated query is quietly wrong: the join that fans out and double-counts revenue, the WHERE that silently drops NULLs, the window frame that is off by one row. The senior depth is reading an EXPLAIN plan and knowing why a query is slow. AI hands you the query; understanding it is still yours.

Data modeling and transformation

Dimensional modeling, star and snowflake schemas, fact vs dimension tables, dbt models and tests. Then the depth: Slowly Changing Dimensions, grain, conformed dimensions, the One Big Table pattern, Data Vault.

AI drafts the model. What it can't do is the judgement calls: what is the grain of this fact table, what does "one customer" mean across three source systems, which dimension is conformed across marts. The classic trap is Slowly Changing Dimensions - everyone can recite the types, almost nobody internalizes which version of a dimension their facts join to. Get it wrong and "revenue by region last quarter" reports a number that was never true.

Replay a change timeline yourself in the free, in-browser SCD Playground, then practice the area in the Dimensional Data Modeling track.

Orchestration and pipelines

Airflow DAGs, scheduling, sensors, backfills, retries, idempotency. Senior: scheduler and executor internals, data-aware scheduling, lineage, freshness SLAs, being on-call.

AI generates the DAG, and it is good at it. What it doesn't generate is the understanding of failure modes the job actually requires, because the real work here isn't the happy path, it's the 3am page. Why did this task hang? Why did the backfill double-write? Is this retry safe, or did it just send the same email twice? Idempotency is a property you reason about, not a snippet AI sprinkles in. See the Orchestration and Pipelines track.

Storage and file formats

Parquet, row vs columnar, compression, object storage, partitioning. Senior: row groups, page statistics, predicate pushdown, encoding, the small-file problem, the internals of ORC, Avro and Arrow.

This is where AI is least useful and depth pays the most, because why a scan costs what it costs is a property of the bytes on disk, not the query text. AI reads and writes Parquet fine. It can't tell you why two files with identical rows differ tenfold in scan cost - that is row group sizing, encoding choice, and whether min/max statistics let the engine skip pages.

Point the free Parquet Viewer at your own files (100% in-browser, nothing is uploaded) to see the row groups and statistics yourself. Track: Storage and File Formats.

Data lakes and table formats

Lake vs warehouse vs lakehouse, Iceberg and Delta, time travel, schema evolution. Senior: ACID and snapshot isolation internals, compaction, catalogs, the Iceberg-vs-Delta-vs-Hudi tradeoffs.

AI scaffolds the table operations happily. The part that bites, and that it won't warn you about, is what happens when two writers commit at once. Snapshot isolation, optimistic concurrency, conflict resolution, compaction fighting your ingest job: that is distributed-systems reasoning, not autocomplete. Track: Open Table Formats.

Ingestion and streaming

Batch ingestion, Kafka basics, producers and consumers, event time vs processing time. Senior: exactly-once semantics, consumer group rebalancing, Change Data Capture, stream processing in Flink or Kafka Streams.

AI writes the producer and the consumer. Where it goes quiet is where data-quality bugs are actually born: the gap between event time and processing time that makes your windowed aggregates wrong, the rebalance that reprocessed a batch, the "exactly-once" guarantee that was only ever at-least-once because of how you committed offsets. Track: Ingestion and Transport.

Distributed compute

Spark DataFrames, transformations vs actions, lazy evaluation. Senior: shuffle and partitioning, broadcast joins and data skew, Catalyst and codegen, memory and fault tolerance.

AI writes the transformation. It cannot tune the execution. Why did this job spill to disk? Why is one task taking 40 times longer than the other 199 (hello, data skew)? Should this join broadcast or shuffle? That reasoning, about how a logical DataFrame becomes physical work across a cluster, is squarely yours. Track: Compute Engines.

Query engines and OLAP

What OLAP is, warehouse vs query engine, ClickHouse, Trino. Senior: MergeTree and projections, federation and pushdown, execution models, cost-based optimization, real-time OLAP, EXPLAIN literacy.

AI writes the SQL the dashboard runs. Why that dashboard is slow, and how to fix it at the engine rather than by rewriting the query, is senior work. It lives in how the engine sorts and merges data, what it can push down, and what its optimizer chose. Track: Query Engines and OLAP.

Semantic and metrics layer

Metrics and dashboards, the semantic layer, data-quality tests. Senior: data contracts, schema registries, metric governance, reverse ETL.

AI drafts a metric definition. What it can't do is the organizational work of making "revenue" mean exactly one thing across finance, sales and product. That is a human contract - negotiated, governed, enforced - and it is the layer where data finally becomes shared business language instead of seven conflicting spreadsheets.

Governance, quality and cloud

PII basics, GDPR and CCPA, cloud, CI/CD for data. Senior: masking and tokenization, row and column access control, right-to-erasure across a lakehouse, infrastructure as code, data observability at scale.

AI flags the obvious PII column. What it can't design is right-to-erasure across a lakehouse with time travel and immutable snapshots - that is architecture, not autocomplete. The masking itself is full of guarantee-breaking gotchas: an unsalted hash is a lookup table, a redacted ZIP that keeps five digits still re-identifies people. Generate the DDL with the free PII Masking Policy Generator. Track: PII and Data Governance.

So, will AI replace data engineers?

It raises the floor and moves the value up.

AI now does the old junior checklist well: queries, DAGs, glue code, boilerplate pipelines. What's left for you is the durable part - reasoning about the system. Why a scan costs what it costs. What happens when two writers commit. Why a job spilled to disk.

AI doesn't replace the engineer who understands that depth, it gives them leverage. They direct the AI through the surface work and spend their judgement on the part it can't reach. The engineer who only knew the surface is the one under pressure now, because the surface is free.

That is the whole premise of the map. You touch every area early - junior and senior work the same areas. What stretches out over a career is how deep you go into each, and the deep end is precisely the part AI can't shortcut for you.

Open the full interactive Data Engineer Roadmap to see every topic on a single timeline, with a "Going senior" toggle that reveals the depth in each layer. Then if you want to practice that depth on real engines instead of slideware, that is what the curriculum and the free in-browser tools are for.

Originally published on the Petascale Labs blog.

[ROADMAP] Beyond the CSV: Statements as a Product Surface

Wed, 06 May 2026 11:33:00 +0000

Beyond the CSV: Statements as a Product Surface

Part 10 looks past today’s CSV importers at broker statements (often PDF). The architecture below is directional: we are spelling out constraints and a credible pipeline, not announcing a finished PDF-to-portfolio feature in production.

What you can use today

Broker adapters and universal CSV — importer package and in-app CSV flow. Heavy parsing stays in the browser; where cloud-assisted mapping exists, only bounded previews go upstream (for example headers plus a small sample of rows).
Ask AI context — always the buildPortfolioContext string (Parts 2–3), no matter how trades entered the app.

Why PDF is harder than CSV

Layout — multi-column tables, repeated headers, page breaks.
Noise — footnotes, disclaimers, mixed fonts.
Scans — OCR errors break deterministic parsers.
Trust — people should confirm extracted rows before anything merges into a ledger—the same bar we set on universal CSV mapping.

Target pipeline (when we ship it)

Extract — client-side or trusted worker → row-like text or tables.
Map — the same schema discipline as CSV (still bounded if the cloud helps).
Parse deterministically — Trade[] in the same spirit as today’s genericParse.
Review — explicit user confirmation before merge.

Scope. This installment is engineering intent and product reasoning. End-to-end PDF statement import is not described here as something you can do in the live app today; when we ship it, we will say so in release notes and product copy, not only in a serial essay.

Part 10 of Sovereign Engineering.

Read the full Sovereign Intelligence book or try the app.

[Article] Divooka, Where We Are

Sat, 16 Aug 2025 19:42:13 +0000

Many years from now, when power users, researchers, and historians look at this article, they will see how it all started and glimpse the vision behind our work.

Overview

Ah, vision - a fascinating subject, and a quality not appreciated by all. Like the ability to dream, to imagine, and to dwell in the realm of mathematics and technicality. For imaginative minds, it's dangerous to wander too far; without standing on solid ground, one can hardly make real progress. Often, it pays to be hands-on.

Divooka is (or is becoming) a complete, general-purpose, high-level, multi-paradigm, flexibly typed, modern visual programming language/system, complete with package management, standard libraries, application frameworks, graph editors, IDE, CLI executor, code generator & compiler, derived frontend apps, examples, documentation, and accompanying tutorials and training materials. We've achieved the majority of this already, so it's not an exaggeration to list them now. What's more, there are still ideas I haven't put into words.

It's hard to do one thing well, and to achieve all of this with limited resources, we need a good strategy. Of course, the strategy itself is a secret I won't share - it's fairly simple, really, but requires determination and execution power. After a year since incorporation last July, it's useful to review and recap what we've done and where we are.

In this article, I'll focus on the period from January to August 2025, but to give a complete picture, I find it instructive to look back and highlight outcomes from last year and key lessons learned.

Early 2024

2024 was about the modernized inception of what Divooka is today.

Neo was reviewed a third time, and we managed to trim down most legacy code. It was time to think more broadly about the concept of "general-purpose visual programming," rather than just a demonstration for data analytics. A new repository was created for modern development. Gospel was conceived.

Talks with investors and prospective entrepreneurs mostly confirmed that this subject is hard, and we learned there are countless low-code platforms in China. That is good.

A disagreement with a co-creator helped me settle the fact that we would definitely go desktop-first and worry about the web later.

The conception of Quadrinity and the flexible office software framework inspired me to quit my job and move forward.

July–Sep 2024

During initial incorporation, development was highly focused, and Divooka, though very simplistic, was a well-defined, easy-to-perceive single tool with only a handful of nodes. With Haopu's help on plotting, it quickly looked like a handy little utility for data analytics - a subject matter we should continue to explore and focus on in the future.

At the same time, to prepare for public preview and an angel investor meeting, I spent a month making prototypes for many potential application areas, including experimental libraries and web use. Due to lack of sophisticated integration and completeness, I doubt these prototypes made as strong an impression as I hoped.

On the marketing side, the concept of a case competition was pitched during this time, but it never materialized later in 2024 due to slow progress on software completion and the dashboard library.

Sep–Dec 2024

With the promise of investor funding, I decided to expand operations and start hiring contractors to work on various aspects of the overall Divooka platform, including GUI design, website improvements, and preparation for first practical use. In a previous article, I mentioned that end users were unlikely to use Divooka directly during initial adoption. To elaborate:

Lack of comprehensive DSL libraries at this stage.
Overall complexity and learning curve as a general-purpose programming language.

While we could have focused on DSL, the second point made it unlikely to gain traction (given how unknown we were) and not commercially feasible any time soon. Thus, we needed more specific, tool-like, highly focused, direct end-user GUI applications.

For that reason, we published Fantasy Planet Painter. We also planned to publish Arcadia, but timeline issues prevented it.

It was during this period that we conceived what was previously known as "ZFA," while "fighting" for the trademark of "Zora" in Canada. Due to delays with investment funding, personnel management, and other corporate matters, we didn't make significant progress on the modern Morpheus framework or ZFA at that time.

In conclusion, during these busy four months, there was not much I would call technical progress. Most efforts were miscellaneous but time-consuming. I also learned it was better to take more matters into my own hands.

Jan–Aug 2025

January to August has been much more fruitful and productive, despite major distractions in January, May, and July. With help from Aaron, Allan, and ChatGPT - and without the communication burden of contractors and external collaborators - progress has been slow but steady. More importantly, I've had time to investigate highly technical matters and build foundational work.

The Web Summit in May and internal training from April to July helped us greatly polish the Neo graph editor, stabilize Neo's internal serialization format, and establish Divooka for general-purpose use.

Ecosystem-wise, I've done plenty of investigation and prototype work on Python integration, build automation, code refactoring, and preliminary work on online services.

There has been extensive work on Neo and Neo-first features, including general GUI improvements, bug fixes, plugin framework enhancements, subgraph work, generics, lambda support, multilingual framework setup, procedural context, and other experimental work. Related to this is the improved Neo serialization format, which is becoming more robust and stable. We haven't yet worked on package management. Although we are trying to make it DiDS-conformant, it's still far from achieving a single final format.

DSL-wise, there has been initial work on Ol'ista, Glaze!, Slide Present, and Novella. Many utilities were also created for ChatGPT and GenAI services, including interfaces for Nvidia NIM, ComfyUI, and Hugging Face inference services. Different users have different demands, and it's hard to focus on any specific library without a pilot project. While we do have internal pilot projects, they are large in scope and still require groundwork, so Divooka hasn't yet played a significant role. I do want to seek external collaboration on this - but at the moment resources are constrained.

On the plotting front, as of early August, Aaron has made significant progress on the mid-level plotting API, making Divooka's plotting library almost "self-sustainable," able to meet a wide range of demands.

Although data analytics is arguably not our only focus, use case investigations have improved in-memory DB handling, refactoring, and preliminary libraries for PostgreSQL ingestion. One architectural challenge is expression interpretation: we decided long ago that, at least for data analytics, it's not efficient to rely entirely on node interfaces - some coding is desirable. The main focus is SQL snippets through our in-memory DB library, though at times LINQ expressions and in-place C# scripting have been useful. However, concerns with the latter are robustness, error tolerance, predictability, increased distribution size, and future program export compatibility.

The data analytics API also faces the challenge of whether to focus on OOP or tabular data as intermediate formats. Sometimes one is more convenient than the other, which increases development costs to maintain both. Eventually, we will likely keep both but prefer exposing the tabular format in Divooka.

A more complete picture of "data analytics" in Divooka would extend into data science and machine learning. There's plenty of research to do here, including choosing a foundational library, gradually building our own infrastructure, and developing graph-friendly, visual-programming-first APIs. This will have to wait until next year.

Summary

The most visible outcome is that Neo is receiving significant updates. On the programming language front, Divooka's function libraries are growing as the language becomes more capable, flexible, and production-ready. Once we start working on package management, it will be more prepared for practical day-to-day use. However, package management requires a fully functional online service first.

Meanwhile, we are actively exploring more "direct-user-facing" applications, which I'll discuss in future posts.

We could use a lot of help on DSL work, and that's the plan as the foundations solidify and architectural decisions become clearer.

References

Wiki Entries on related subjects:

DevLogs on Divooka/Neo progress:

The Introductory to Visual Programming with Divooka Textbook:

Access it online here!

Additional references:

Glossary

DSL: Domain-specific Library
DiDS: Divooka Document Specification
DiOS: Divooka Open Standards
Gospel: The new, revived, modern incarnation of Parcel, gateway into Divooka; Now a new graph editor frontend for Divooka
Neo: The Windows graph editor for Divooka
Morpheus: The cross-platform GUI framework and graph editor architecture for Divooka
Quadrinity: A way to perceive office suite software as a single framework consisting of multiple views, like Blender's workspaces
ZFA: Zora Frontier Architecture