Coder Legion - Recent posts tagged etl

Unlocking Data Pipeline Success Through Core ETL vs ELT differences

Mon, 19 Oct 2026 21:28:34 +0000

Introduction

In the modern landscape of enterprise analytics, data engineers face a critical decision when constructing scalable architecture: determining how raw information flows from transactional systems into central repositories. The traditional approach of Extract, Transform, and Load has served organizations for decades, yet the rise of high-performance cloud warehouses introduced Extract, Load, and Transform as a formidable alternative. Understanding ETL vs ELT differences serves as the foundation for optimizing data processing efficiency, managing cloud infrastructure expenses, and accelerating time to actionable business intelligence. Choosing the right pipeline design dictates how quickly raw records become meaningful metrics, affecting every decision maker who relies on reliable metrics.

Architectural Evolution of Data Pipelines

The classic pipeline model emerged during an era when storage space and compute power were expensive commodities. Legacy databases required structured inputs, which forced engineers to scrub, format, and reshape incoming records on specialized secondary servers prior to ingestion. This intermediate processing layer kept target databases from suffering extreme performance degradation during query execution.

As cloud storage costs plummeted and distributed cloud computing systems matured, modern architecture underwent a massive shift. High-performance cloud warehouses empowered data teams to ingest vast streams of unrefined information instantly. Instead of spending time preparing datasets externally, organizations leverage the raw, elastic power of modern analytical storage to execute queries directly on raw data inside the destination environment.

Technical Mechanics and Execution Order

The essential ETL vs ELT differences stem from where and when processing operations occur within the pipeline stages.

In the classic workflow, extraction occurs first by gathering records from disparate sources like application logs, operational databases, or third-party web services. Next, an intermediate staging engine processes these inputs, applying strict business logic, data masking, and deduplication rules. Only after every record matches the destination schema does the system write the clean datasets into the target repository.

The modern framework alters this sequential chain by swapping the secondary and tertiary operations. Raw files are extracted and written directly into the cloud warehouse without prior cleanup. Once inside the cloud repository, data scientists and engineers utilize SQL or specialized transformation platforms to shape, aggregate, and enrich datasets as needed for downstream analytical dashboards.

Performance, Scalability, and Speed Dynamics

System performance remains one of the primary ETL vs ELT differences evaluating cloud suitability. Legacy workflows create operational bottlenecks when handling massive volumes because external staging servers must process every row before storage happens. Large incoming batches extend processing windows, occasionally delaying reporting availability until late nightly updates complete.
Conversely, the modern framework delivers superior loading speed because it bypasses pre-storage processing entirely. By landing raw records into highly scalable distributed repositories, transformations occur in parallel using native computing power. This direct loading model enables near-real-time analytical views and accommodates sudden spikes in data ingestion without causing pipeline stalls.

Handling Structured and Unstructured Data Types

Data diversity further highlights key ETL vs ELT differences across enterprise applications. Traditional workflows excel when operating on structured relational inputs that follow rigid, predictable schema definitions. Because every column must be defined before ingestion, unstructured or semi-structured files require extensive pre-processing and custom code before they can land safely in legacy storage.
Modern ingestion models handle structured, semi-structured, and completely unstructured information with equal ease. Sensor outputs, JSON payloads, server logs, and document stores land in cloud storage without immediate formatting requirements. Data teams retain full flexibility to extract value from these diverse formats whenever new analytical requirements emerge.

Cost Structures and Resource Management

Financial considerations influence how engineering teams evaluate ETL vs ELT differences across infrastructure deployments. Traditional workflows require ongoing investments in specialized transformation hardware and dedicated processing engines. Maintenance overhead grows as schemas change, demanding continuous developer effort to update rigid transformation rules before new data fields can be loaded.

Modern cloud pipelines shift resource allocation toward elastic cloud computing. While raw storage costs remain minimal, query costs inside cloud data platforms can escalate if transformation queries are unoptimized. However, eliminating dedicated intermediate servers simplifies data stack architecture and reduces initial setup expenses significantly.

Security, Compliance, and Data Governance

Data protection protocols introduce nuanced ETL vs ELT differences that governance teams must review closely. Traditional pipelines mask sensitive information, remove personally identifiable details, and apply compliance rules before data reaches destination storage. This pre-load sanitization prevents unauthorized personnel from accessing sensitive raw records.

The modern approach loads raw, unmasked datasets directly into central storage, requiring robust role-based access controls, column-level security, and automated encryption inside the cloud warehouse. While this provides flexibility for auditing raw records, it demands strict governance frameworks to ensure privacy compliance across all analytical environments.

Making the Right Choice for Your Engineering Team

Selecting the optimal pattern depends entirely on your current technology stack, team skill sets, and analytical speed requirements. Organizations dealing with strict regulatory demands, legacy systems, or stable structured metrics often benefit from traditional pipeline control. Teams scaling rapid cloud infrastructure, analyzing semi-structured event streams, or requiring real-time insights will find modern ingestion strategies far superior. Evaluating these fundamental trade-offs ensures your data architecture scales effortlessly alongside your growing operational needs.

Azure Synapse to Databricks: A Migration Playbook That Survives Production

Fri, 09 Oct 2026 13:19:33 +0000

Originally published on DEV.to.

Azure Synapse → Databricks: A Migration Playbook That Survives Production

A field-style guide for Lead / Senior Data Engineers moving Dedicated SQL Pools, pipelines, and BI off Synapse without burning trust, budget, or the on-call team.

Why this migration is harder than the slide deck
What you are actually migrating
Program shape: discovery → waves
Four parallel workstreams
Data movement: skip the JDBC tax
Code conversion: syntax is easy, semantics bite
Dimensional patterns on Delta (SCD2)
Reconciliation & cutover
Power BI / consumption cutover
30 / 60 / 90 plan
Anti-patterns I keep seeing
Field rules I will not compromise on
FAQ
Sources

1) Why this migration is harder than the slide deck

If you have lived in Azure Synapse for years, the marketing line sounds simple:

Move to Databricks. Unity Catalog. Lakehouse. Done.

In production programs, that is not the hard part.

The hard part is that Synapse is not one system. Under one brand you usually own years of Dedicated SQL Pool logic, ADF/Synapse Pipelines, serverless SQL over the lake, Spark pools, permission models, and Power BI datasets wired to Synapse endpoints.

Treat that estate as one workstream with one deadline, and the schedule slips. Every time.

I work as a Lead Data Engineer across Azure Synapse, Microsoft Fabric, Azure Data Lake, Databricks/Spark, SQL/T-SQL, and Power BI. This playbook is the version I want in the room before the first converted stored procedure lands in production.

Thesis: Migration success is not “all notebooks on Databricks.”
It is decision-ready data, trusted KPIs, and a platform your team can operate.

2) What you are actually migrating

Synapse piece	Typical complexity	What “done” looks like on Databricks
Dedicated SQL Pools	Highest — procs, distributions, CCI, years of tuning	Databricks SQL / Spark jobs, Delta tables, Liquid Clustering / predictive optimization
Serverless SQL	Medium — views & external tables over ADLS	Views / tables in Unity Catalog over the same lake paths (often simpler)
Spark pools	Lower — already Spark-shaped	Databricks Runtime notebooks / jobs (often few changes)
ADF / Synapse Pipelines	High — orchestration + connectors	Lakeflow Connect / Lakeflow Jobs / Workflows
Governance	High — SQL perms + Purview stitching	Unity Catalog (and stop dual-governing forever)
Power BI / semantic models	High politically	Databricks SQL Warehouse endpoints + dataset remaps

Dedicated SQL Pools usually consume most of the calendar. Orchestration and BI consume most of the organizational risk.

3) Program shape: discovery → waves

Do not start with “transpile everything.” Start with a program.

flowchart LR
  A[1 Discovery<br>Lakebridge Profiler] --> B[2 Assessment<br>Analyzer / T-SQL]
  B --> C[3 Design<br>Hybrid lift + modernize]
  C --> D[4 Pilot<br>Lighthouse E2E]
  D --> E[5 Waves<br>Parallel workstreams]

Phase A — Discovery

Inventory what actually runs:

pools, databases, schemas, object counts
top queries / heavy procs by CPU & IO
pipeline inventory (ADF / Synapse)
downstream BI datasets and owners
unused objects (your cheapest scope cut)

Lakebridge Profiler (Databricks Labs) is built for Synapse estate profiling — metadata, utilisation, query patterns, baselines for a TCO conversation.

Phase B — Assessment

Classify the T-SQL estate:

low / medium / high complexity
unsupported constructs
dependency graphs (what breaks if this view moves)

Lakebridge Analyzer helps here. Start waves with low/medium complexity and clear business owners — not the ugliest proc on day one.

Phase C — Design (usually hybrid)

Three strategies show up in every steering committee:

Approach	When it works	Failure mode
Lift-and-shift	Hard deadline off Synapse	You recreate Synapse-shaped pain on a new bill
Big-bang rewrite	Greenfield team, tiny estate	Timeline fantasy
Hybrid	Most real enterprises	Needs discipline: automate bulk convert, modernize in waves

For most Synapse migrations, hybrid wins: automated conversion gets you off Synapse on a schedule; modernization (Delta patterns, Liquid Clustering, Lakeflow, UC) happens after workloads are stable.

Phase D — Pilot (lighthouse)

Pick one end-to-end slice:

source → ingest → transform → gold mart → Power BI

Prove:

storage + UC layout
job standards
reconciliation pack
BI remap
on-call runbook

The pilot’s job is not heroics. It is producing reusable assets for later waves.

Phase E — Waves

Each wave should ship a visible business win, not just “another schema moved.”

4) Four parallel workstreams

After the lighthouse, scale on four tracks together. Sequential “ETL first, BI last” is how stakeholders only feel pain.

flowchart TB
  P[Lighthouse Pilot]
  P --> I[Ingestion<br>ADF/Synapse Pipelines → Lakeflow Connect]
  P --> T[Transformation<br>T-SQL / procs → Databricks SQL / Spark]
  P --> O[Orchestration<br>Schedules → Lakeflow Jobs]
  P --> C[Consumption<br>Power BI → Databricks SQL Warehouse]

Track	From	To
Ingestion	ADF / Synapse Pipelines	Lakeflow Connect / Jobs (or open ingest into Delta)
Transformation	T-SQL / procs / views	Databricks SQL / Spark with clear contracts
Orchestration	Synapse schedules & deps	Lakeflow Jobs / Workflows
Consumption	Power BI / semantic models	Databricks SQL Warehouses

Sequencing tip that builds trust

A BI-aware sequence often works better than pure ETL-first:

Land or expose business-facing marts early (even interim / federation patterns where appropriate).
Put a trusted KPI dataset in front of users.
Modernize the uglier ETL underneath while the business already sees progress.

Enablement is a fifth unofficial track. Synapse-native teams underinvest here and then call the platform “not ready.”

5) Data movement: skip the JDBC tax

For large Dedicated SQL Pool tables, JDBC Synapse → Databricks is how you burn cluster hours and patience.

Prefer:

Synapse CETAS → Parquet on ADLS Gen2 → Databricks Spark read → Delta in Unity Catalog

flowchart LR
  S[Synapse Dedicated SQL Pool] -->|CETAS| L[ADLS Gen2 Parquet]
  L -->|Spark read / Delta write| D[Unity Catalog Delta]
  S -.->|JDBC slow path ❌| D

Why this path wins

Synapse writes columnar Parquet in parallel (CETAS).
ADLS is the shared landing zone both engines already understand.
Databricks reads Parquet with strong scan performance, then writes Delta (ACID, time travel, UC governance).
You avoid row-by-row JDBC serialization.

Example — CETAS from Synapse Dedicated SQL Pool

-- File format
CREATE EXTERNAL FILE FORMAT ParquetFF
WITH (
    FORMAT_TYPE = PARQUET,
    DATA_COMPRESSION = 'org.apache.hadoop.io.compress.SnappyCodec'
);

-- Landing zone on ADLS Gen2
CREATE EXTERNAL DATA SOURCE MigLanding
WITH (
    LOCATION = 'https://<storage>.dfs.core.windows.net/<container>',
    TYPE = HADOOP
);

-- Export a bounded slice (always partition/filter on first runs)
CREATE EXTERNAL TABLE ext.fact_sales_export
WITH (
    LOCATION = '/migration/fact_sales/ds=2026-09-03/',
    DATA_SOURCE = MigLanding,
    FILE_FORMAT = ParquetFF
)
AS
SELECT
    sales_id,
    customer_nk,
    product_nk,
    business_date,
    net_amount,
    quantity
FROM dbo.fact_sales
WHERE business_date >= '2024-01-01';

Example — land as Delta on Databricks

from pyspark.sql import functions as F

src = "abfss://<container>@<storage>.dfs.core.windows.net/migration/fact_sales/"

df = (
    spark.read.parquet(src)
    .withColumn("ingested_at", F.current_timestamp())
)

(
    df.write
    .format("delta")
    .mode("overwrite")
    .option("overwriteSchema", "true")
    .saveAsTable("sales_uc.gold.fact_sales")
)

Physical design: do not cargo-cult Synapse

In Synapse you spent years on:

DISTRIBUTION = HASH(...)
ROUND_ROBIN / REPLICATE
clustered columnstore choices

On Databricks / Delta, those directives do not travel. Mapping HASH(customer_id) to a Delta partition is a classic self-own: high-cardinality keys create tiny-file nightmares.

Prefer:

Liquid Clustering / CLUSTER BY AUTO
predictive optimization / compaction as an operating model
partitions only for coarse, stable predicates (e.g. business_date) when they truly help

Rule: Drop physical Synapse directives. Re-tune with lakehouse primitives after you have real query telemetry — not before.

6) Code conversion: syntax is easy, semantics bite

Automated conversion (Lakebridge transpile and friends) often handles a large share of syntax — commonly cited in the ~80–90% range for bulk translation. Your engineers live in the remaining 10–20%: cursors, dynamic SQL, weird error handling, and semantic mismatches.

Common remaps

T-SQL	Databricks SQL
`GETDATE()`	`CURRENT_TIMESTAMP()`
`ISNULL(a, b)`	`COALESCE(a, b)` / `IFNULL(a, b)`
`LEN(s)`	`LENGTH(s)`
`CHARINDEX(sub, str)`	`LOCATE(sub, str)`
`SELECT TOP 10 ...`	`SELECT ... LIMIT 10`
`CONVERT(INT, col)`	`CAST(col AS INT)`

Semantic trap #1 — case sensitivity

Dedicated SQL Pools are often case-insensitive by collation. Databricks SQL is case-sensitive by default.

-- May silently change results after migration
WHERE status = 'active'

-- Make intent explicit
WHERE LOWER(status) = 'active'

Build this into reconciliation: pick string dimensions and compare distinct counts before you argue about revenue.

Semantic trap #2 — stored procedures

Migrate structure first (parameters, control flow, set-based DML). Optimize second.

Rewrite candidates (do not pretend transpile “finished” them):

cursors / row-by-row loops
heavy dynamic SQL
Synapse-only performance hints
temp-table choreography that exists only to fight distribution skew

Semantic trap #3 — TRY/CATCH vs workflow design

Simple audit logging often stays SQL-shaped. Multi-step compensation and downstream signaling usually belong in Lakeflow Jobs (retries, quarantine tasks, alerting) rather than a 2,000-line proc that tries to be an orchestrator.

7) Dimensional patterns on Delta (SCD2)

Slowly Changing Dimensions are where Synapse estates accumulate snowflake uniqueness. Your goal is not to clone the old proc line-for-line. Preserve the business rule: history + current row queryability.

Example pattern — expire then insert (illustrative)

-- 1) Expire changed current rows
MERGE INTO dim_customer AS t
USING staging_customer AS s
ON t.customer_nk = s.customer_nk AND t.is_current = true
WHEN MATCHED AND (
     t.segment <> s.segment
  OR t.country <> s.country
) THEN UPDATE SET
  t.is_current = false,
  t.valid_to   = current_timestamp();

-- 2) Insert new current versions
INSERT INTO dim_customer (
  customer_nk, segment, country, valid_from, valid_to, is_current
)
SELECT
  s.customer_nk,
  s.segment,
  s.country,
  current_timestamp(),
  CAST(NULL AS TIMESTAMP),
  true
FROM staging_customer s
LEFT ANTI JOIN dim_customer t
  ON t.customer_nk = s.customer_nk
 AND t.is_current = true;

Delta’s ACID behavior makes multi-step SCD flows safer than “hope the warehouse lock ordering holds.” Still: wrap critical loads in jobs with clear quarantine paths.

8) Reconciliation & cutover

Budget reality: validation often costs more calendar time than conversion.

Minimum reconcile pack (every wave)

Row counts by business date / partition
Aggregates — revenue, quantity, distinct business keys
Hash / checksum compares on ordered business-key sets
Tolerance checks for floats / FX / rounded measures
String-dimension sanity (case folding surprises)

Lakebridge Reconcile supports structured comparison patterns across these dimensions.

Cutover pattern that keeps executives calm

Parallel run (Synapse + Databricks) for the lighthouse mart
Side-by-side Power BI validation with the business owner
Flip the dataset connection
Pause Synapse compute (do not delete yet)
Keep rollback for a defined soak window
Only then decommission

Early decommissioning is how “successful migrations” become weekend rollbacks.

9) Power BI / consumption cutover

Technical remap is usually:

Synapse endpoint → Databricks SQL Warehouse
refresh credentials / gateway assumptions
validate DAX measures against reconcile packs

Political remap is harder:

who signs off KPI parity?
which report is the source of truth during dual-run?
what is the communication plan when numbers differ by 0.3% due to case folding or timezone?

Treat consumption as a first-class workstream with named business owners, not a Friday afternoon connection string change.

10) 30 / 60 / 90 plan

Days 1–30 — foundations

[ ] Run Lakebridge Profiler + Analyzer on Dedicated pools
[ ] Cut dead objects from scope (document why)
[ ] Choose one lighthouse: pipeline → gold mart → one Power BI dataset
[ ] Stand up Unity Catalog layout + storage conventions
[ ] Define job naming, environments, and CI basics
[ ] Write the reconcile notebook/job v0

Days 31–60 — prove the path

[ ] CETAS → ADLS → Delta for large facts (no JDBC heroics)
[ ] Transpile + hand-fix the top procs for the lighthouse
[ ] Repoint one BI workspace to Databricks SQL Warehouse
[ ] Parallel run + signed business acceptance
[ ] Capture runbooks: failure modes, quarantine, reruns

Days 61–90 — scale without drama

[ ] Wave 2/3 domains with the same pack
[ ] Orchestration parity on Lakeflow Jobs
[ ] Enablement plan for Synapse-native engineers
[ ] Cost review (warehouse sizing vs old pool bills)
[ ] Only then: formal Synapse decommission proposal

11) Anti-patterns I keep seeing

Transpile-first, assess-never — migrating abandoned procs with confidence.
JDBC for multi-billion-row facts — slow, expensive, fragile.
HASH key → Delta partition — tiny files, sad Spark UI.
Big-bang BI cutover — no dual-run, no owner, no soak.
Recreating every Synapse knob — preserving constraints without preserving reasons.
Skipping enablement — platform is fine; team still thinks in distributions.
Decommission on day one of “green” — optimism is not a rollback strategy.

12) Field rules I will not compromise on

Hybrid > pure lift-and-shift > big-bang rewrite
Assessment before conversion
ADLS + Parquet/Delta over JDBC for big moves
Do not map Synapse HASH keys to partitions
Reconcile every wave
Do not decommission Synapse early
Enablement is a workstream, not a lunch-and-learn
Migrate outcomes (trusted KPIs), not just objects

13) FAQ

Q: Should we move to Fabric instead of Databricks?
A: Different target, overlapping Azure story. This article assumes a Databricks Lakehouse landing zone. If Fabric is in play, treat that as an explicit architecture decision — do not accidentally run two migrations under one slogan.

Q: Can we keep Synapse Serverless for “just a few views”?
A: Sometimes as a bridge. Long-term dual query planes usually recreate the governance tax you are trying to escape.

Q: How perfect must automated conversion be?
A: Good enough to move bulk syntax. Your senior engineers should own semantic diffs, SCD logic, and orchestration.

Q: What is the first metric of a healthy pilot?
A: Not “job duration.” It is signed KPI parity with a business owner plus a rerun runbook a junior can execute.

14) Sources

Databricks — Navigating a Synapse Migration to Databricks
https://www.databricks.com/blog/navigating-synapse-migration-databricks
Databricks — Introducing Lakebridge
https://www.databricks.com/blog/introducing-lakebridge-free-open-data-migration-databricks-sql
Lakebridge docs — Synapse Profiler
https://databrickslabs.github.io/lakebridge/docs/assessment/profiler/synapse/
Lakebridge docs — Transpile
https://databrickslabs.github.io/lakebridge/docs/transpile/
Practical large-table movement discussion (CETAS → ADLS → Delta)
https://medium.com/@prabhakarankanniappan/copy-3-5-billion-rows-from-synapse-to-databricks-in-30-minutes-fda15e0ffca0

go-intake v0.1.0: A Small Go Library for Messy Data Intake

Tue, 09 Jun 2026 10:05:39 +0000

go-intake: Go-Native Streaming Data Ingestion Toolkit

Executive Summary

go-intake is a minimalist, streaming-first ETL toolkit for Go developers that transforms messy data into validated, record-oriented output. With zero third-party dependencies and a focus on composition over configuration, it provides enterprise-grade data pipeline capabilities in a single, embeddable library.

Architecture Overview

Core Abstraction Layers

Component	Interface	Responsibility	Pattern
Source	3-method interface (`Open`, `Read`, `Close`)	Data ingestion from CSV, JSONL	Stream until EOF
Transformer	`Apply(ctx, Record) (Record, error)`	Field normalization, type parsing	Immutable, returns copy
Validator	`Validate(ctx, Record) error`	Schema enforcement, business rules	Read-only
Quarantine	`Write(ctx, InvalidRecord)`	Rejected record capture with context	Structured error logging
Sink	3-method interface (`Open`, `Write`, `Close`)	Output to CSV, JSONL	Streaming write

Design Principles

┌─────────────┐    ┌──────────────┐    ┌─────────────┐
│   Source    │───▶│ Transformers │───▶│ Validators  │
└─────────────┘    └──────────────┘    └─────────────┘
                                            │
                               ┌────────────┴──────────┐
                               ▼                       ▼
                         ┌──────────┐           ┌───────────┐
                         │   Sink   │           │Quarantine  │
                         └──────────┘           └───────────┘

Memory Usage Comparison

Memory (MB)
   500 │                      
       │    ●●●●●●●●●●●●●●●●●●  go-intake (constant)
       │                      
   100 │                      
       │                      
    50 │    ████ goflow grows with dataset
       │    ██████            
    25 │    ████████████      
       │    ████████████████  streamz grows
       │                       
     0 └──────────────────────
        0K    50K   100K   150K
           Records Processed

Performance Benchmarks

Throughput Analysis

Dataset Size	Records Processed	Time Elapsed	Throughput
Small (10K)	10,000	~40ms	~250,000/sec
Medium (100K)	100,000	~410ms	~243,902/sec
Large (161K)	161,568	643ms	251,000/sec

Memory Efficiency: The streaming model ensures constant memory usage regardless of dataset size. Only one record exists in memory at any point during processing.

Comparative Performance

graph LR
    A[go-intake<br/>251K/s] --> B[Memory: O(1)]
    C[Gopherize<br/>125K/s] --> D[Memory: O(n)]
    E[goflow<br/>180K/s] --> F[Memory: O(n)]

Performance Scaling Chart

Records/sec
   300K │
        │
   250K │    ● (go-intake: 251K)
        │   ╱
   200K │  ╱
        │ ╱
   150K │● (goflow: 180K)
        │
   100K │● (streamz)
        │
    50K │● (Gopherize: 125K)
        │
     0K └─────────────────────

Feature Matrix vs Competition

Feature	go-intake	Gopherize	goflow	streamz
✅ Zero Dependencies	✓	✗	✗	✗
✅ Streaming Model	✓	Partial	✓	✓
✅ Structured Errors	✓	✗	Partial	✗
✅ Multi-validation Errors	✓	✗	✗	✗
✅ Schema Discovery	✓	✗	✗	✗
✅ Header Normalization	✓	✓	✗	✗
✅ Quarantine Sinks	✓	✗	✗	✗
✅ Context Cancellation	✓	✗	✓	✓
Lines of Code	~2,300	~8,500	~12,000	~6,200

Bundled Components

Sources

CSVSource: RFC 4180 compliant, configurable delimiter, empty line skipping
JSONLSource: Newline-delimited JSON, 16MB line buffer support

Sinks

CSVSink: Automatic header generation, deterministic column ordering
JSONLSink: HTML-escaping disabled, one object per line

Transformers

Function	Purpose	Use Case
`NormalizeHeaders(style)`	Case normalization	Column standardization
`TrimStrings()`	Whitespace removal	Data cleaning
`ParseFloat/Int/Bool/Date`	Type conversion	Schema enforcement
`Rename/Drop/Keep/Copy`	Schema evolution	ETL pipeline stages
`AddField`	Constant injection	Metadata enrichment
`MapField`	Custom transformations	Business logic

Validators

Function	Type	Error Aggregation
`Required(field)`	Presence	✅
`Min/Max/Between`	Numeric range	✅
`Regex(field, pattern)`	Pattern matching	✅
`Enum(field, values...)`	Allowlist	✅
`Email/URL`	Format validation	✅
`NotFuture(field)`	Temporal constraint	✅

Real-World Validation Results

Dataset 1: COVID-19 Aggregated Data (161,568 records)

Field Profiling Results:

Field	Type	Confidence	Null %
Confirmed	int	100%	0%
Country	string	100%	0%
Date	date	100%	0%
Deaths	int	100%	0%
Recovered	int	100%	0%

Pipeline Results:

Read: 161,568
Written: 161,568
Invalid: 0
Failed: 0
Runtime: 643ms

Dataset 2: Synthetic Messy Data (5,000 records with 2-5% error injection)

Statistical Validation:

Metric	Value	Visualization
Total Records	5,000	██████████
Valid Output	4,772 (95%)	████████░░
Quarantined	228 (4.56%)	██░░░░░░░░
Error Detection	100%	██████████

Pipeline Accuracy
100% │ ██████████
     │ ██████████  (100% records accounted for)
 95% │ ████████░░  (95% valid)
     │ ████░░░░░░  (4.56% quarantined)
  0% └────────────────
     0%       100%  Accuracy

Error Distribution in Quarantine:

Error Type	Count	Description	Distribution
required	96	Missing required fields	████░░░░░░ (29%)
min	58	Negative numeric values	██░░░░░░░░ (18%)
regex	172	Email format mismatch	██████░░░░ (53%)
Total	326	Captured validation errors	██████████ (100%)

Use Cases

1. Data Migration Pipelines

p := intake.New().
    From(source.CSV("legacy_export.csv")).
    Transform(transform.NormalizeHeaders(transform.SnakeCase)).
    Validate(validate.Required("id")).
    To(sink.JSONL("normalized.jsonl"))

2. API Data Validation

p := intake.New().
    From(source.JSONL("api_incoming.jsonl")).
    Transform(transform.TrimStrings()).
    Validate(
        validate.Email("email"),
        validate.URL("website"),
        validate.Min("age", 18),
    ).
    OnInvalid(quarantine.JSONL("rejected.jsonl")).
    To(sink.CSV("clean.csv"))

3. ETL for Analytics

p := intake.New().
    From(source.CSV("raw_events.csv")).
    Transform(
        transform.NormalizeHeaders(transform.SnakeCase),
        transform.ParseDate("timestamp", time.RFC3339),
        transform.ParseFloat("value"),
    ).
    Validate(validate.Required("event_id")).
    OnInvalid(quarantine.JSONL("bad_events.jsonl")).
    To(sink.JSONL("analytics_ready.jsonl"))

4. Data Quality Inspection

src := source.CSV("unknown_dataset.csv")
profile, err := discover.InspectSource(ctx, src, discover.Options{
    SampleSize: 10000,
})

for _, f := range profile.Fields {
    fmt.Printf("%s: %s (confidence %.2f)\n", 
        f.Name, f.Type, f.TypeConfidence)
}
for _, issue := range profile.Issues {
    fmt.Printf("[%s] %s\n", issue.Severity, issue.Message)
}

Data Transformation Flow

    ┌─────────────────┐
    │  Raw CSV/JSONL  │
    │   (Messy Data)  │
    └────────┬────────┘
             │
             ▼
    ┌─────────────────┐     ┌───────────────┐
    │   Discover      │────▶│ Field types   │
    │   Inspect       │     │ Null ratios   │
    └─────────────────┘     └───────────────┘
             │
             ▼
    ┌─────────────────┐
    │  Normalize      │
    │  Headers        │
    │  Trim Strings   │
    └────────┬────────┘
             │
             ▼
    ┌─────────────────┐
    │   Parse Types   │
    │  (int, float,  │
    │   date, bool)   │
    └────────┬────────┘
             │
             ▼
    ┌─────────────────┐     ┌───────────────┐
    │   Validate      │────▶│ Valid Output  │
    │   Required      │     │   (Sink)      │
    │   Min/Max       │     └───────────────┘
    │   Regex/Enum    │             │
    └────────┬────────┘             │
             │                      │
             ▼                      ▼
    ┌─────────────────┐     ┌───────────────┐
    │   Quarantine    │     │   Stats:      │
    │   (errors +    │     │ Read/Written/ │
    │   record data)  │     │ Invalid/Failed│
    └─────────────────┘     └───────────────┘

Competitor Comparison Matrix

Toolkit	Dependencies	Streaming	Discovery	Quarantine	Lines of Code
go-intake	Zero	✅	✅	✅	~2,300
Gopherize	12	Partial	❌	❌	~8,500
goflow	18	✅	❌	❌	~12,000
streamz	8	✅	❌	❌	~6,200
etl-go	5	❌	❌	❌	~4,500

Technical Specifications

Specification	Value
Go Version	1.23+
License	MIT
Dependencies	Zero (stdlib only)
Memory Model	Streaming (O(1))
Concurrency	Context-aware cancellation
Test Coverage	100% on public API
Race Detector	Clean

Non-Goals (Explicit Design Decisions)

❌ CLI/REPL interface (library-first)
❌ DAG engine or scheduler
❌ Distributed execution
❌ DataFrame abstraction
❌ Airflow/Airbyte clone
❌ Connector marketplace
❌ PDF/ML features

Installation

go get github.com/firfircelik/go-intake

Quality Gate

go test -count=1 ./...
go vet ./...
go test -race -count=1 ./...

All tests pass. Race detector clean. Zero external dependencies.

ETL vs ELT — Why Notebook Placement Matters Body / Content: Yes — the placement of your Databricks

Fri, 06 Feb 2026 14:09:12 +0000

ETL vs ELT — Why Notebook Placement Matters

Body / Content:

The placement of your Databricks notebooks determines whether your pipeline behaves like ETL or ELT.

If Databricks runs after loading data into the lake, it’s ELT.
If Databricks runs

before loading transformed data, it’s ETL.

Business Analogy:

Clean vegetables before putting them in the fridge → ETL

Dump everything in the fridge and clean later → ELT

Architecture Mapping:

RAW → BRONZE in ADF = Extract + Load (EL)
SILVER → GOLD in Data-bricks = Transform (T)

✅ This makes your pipeline ELT, the modern standard:

Cheap storage Scalable compute Delta Lake supports late
transformations

Notebooks preserve raw data and give flexibility