Coder Legion - Recent posts tagged mdm

DNS: The phonebook of the internet.

Wed, 11 Feb 2026 12:14:10 +0000

One‑line: DNS = the phonebook of the internet.

Two‑words: Name‑resolution.

Business analogy: Instead of memorizing everyone’s phone number, you look up their name in a directory — DNS is that directory for computers.

Mechanical definition (tight, clean)
DNS (Domain Name System) converts human‑friendly names into machine‑friendly IP addresses.

Examples:

google.com → 142.250.72.14
kafka.svc.cluster.local → the pod’s internal cluster IP
elasticsearch.default.svc.cluster.local → the StatefulSet’s service IP

You type the name → DNS returns the IP → your app connects.

Why it matters for Docker vs Kubernetes

Docker (single host)

Containers can cheat and use localhost
DNS barely matters because everything shares one machine

Kubernetes (multi‑node)

Pods move
IPs change
Nodes change
Restarts happen

So the only stable way to reach anything is through DNS names, not IPs or localhost.

Examples:

kafka:9092
zookeeper:2181
elasticsearch:9200

or in full cluster form:

kafka.svc.cluster.local:9092

The rule for distributed systems
If you can’t reach it by DNS, you don’t have a real distributed system yet.

One‑line: Here are the exact DNS names and the Docker‑side simulation.

Two‑words: Cluster realism.

Business analogy: Teach your app to use street addresses (DNS) even when it’s still living in your garage (Docker).

Exact DNS Names for Kafka / ELK / MDM in Kubernetes
These are the canonical, production‑safe names every pod should use.

Kafka

kafka.default.svc.cluster.local:9092

If using a dedicated namespace:

kafka.data.svc.cluster.local:9092

Zookeeper

zookeeper.default.svc.cluster.local:2181

Kafka Connect

connect.default.svc.cluster.local:8083

Elasticsearch

elasticsearch.default.svc.cluster.local:9200

If using a StatefulSet with headless service:

elasticsearch-0.elasticsearch.default.svc.cluster.local:9200

Kibana

kibana.default.svc.cluster.local:5601

MDM ETL Jobs
Jobs should reference Kafka and Elasticsearch via DNS:

kafka.default.svc.cluster.local:9092
elasticsearch.default.svc.cluster.local:9200
Bronze / Silver / Gold Storage
If using object storage:

minio.default.svc.cluster.local:9000
If using HDFS‑style:

namenode.default.svc.cluster.local:8020
2. How Docker Compose Can Simulate Kubernetes‑Style DNS
This is the trick that makes your local dev behave like a cluster.

Rule: Use service names, not localhost
Docker Compose automatically creates an internal DNS zone where each service name resolves like a Kubernetes Service.

Example docker‑compose.yml
yaml

services:

  kafka:
    image: confluentinc/cp-kafka
    ports:
      - "9092:9092"

  zookeeper:
    image: confluentinc/cp-zookeeper
    ports:
      - "2181:2181"

  connect:
    image: confluentinc/cp-kafka-connect
    depends_on:
      - kafka
      - zookeeper

Inside the network, these DNS names work:

kafka:9092
zookeeper:2181
connect:8083
elasticsearch:9200
kibana:5601

Exactly like Kubernetes:

kafka.default.svc.cluster.local:9092

3. The Golden Rule for Dev → CI/CD → Kubernetes

Never use localhost.
Always use service names.

Docker:

kafka:9092
elasticsearch:9200

Kubernetes:

kafka.default.svc.cluster.local:9092
elasticsearch.default.svc.cluster.local:9200

Your code never changes — only the environment does.

One‑line: Day zero.

Two‑words: DNS first.

Business analogy: If production uses street addresses, you don’t train your team on “my house, my room, my desk.”

Direct answer (tight, mechanical, architect‑grade)
Teams should switch to service‑style DNS from the very first local prototype, long before CI/CD or Kubernetes enter the picture.

Because the moment you allow localhost:9092 into the code path, you’ve already baked in a failure mode:

CI/CD → breaks

Multi‑node cluster → breaks

Sidecar networking → breaks

Jobs/CronJobs → break

Scaling → breaks

Canary deploys → break

Localhost is a lie in distributed systems.
It only works on a laptop because Docker Compose simulates a fake single‑machine universe.

The correct development sequence

1. Dev environment (Docker)

Use:

kafka:9092
elasticsearch:9200
zookeeper:2181

Never:

localhost:9092
127.0.0.1:9200

2. CI/CD

Use the same names:

kafka:9092
elasticsearch:9200

Your pipeline becomes cluster‑ready without changing a single line.

3. Kubernetes

DNS becomes:

kafka.svc.cluster.local:9092
elasticsearch.svc.cluster.local:9200
Same pattern, zero rewrites.

Why “DNS from day zero” is the only safe rule
Eliminates environment drift

Prevents pipeline failures

Makes ETL jobs portable

Avoids rewriting configs during migration

Forces distributed‑system thinking early

Protects MDM lineage and auditability

If you wait until Kubernetes to fix localhost assumptions, you’re already in trouble — because your entire pipeline has been trained to believe in a topology that doesn’t exist in production.

Docker‑to‑Kubernetes failure points: Where Docker Assumptions Break in Kubernetes

Tue, 10 Feb 2026 14:52:23 +0000

Docker vs Kubernetes for Kafka + ELK + MDM

Docker assumptions

Assumptions Docker makes here:

Single machine → IPs are predictable
localhost can reach all services via host ports
Startup order partially controlled by depends_on
No strict health checks → app assumes services are ready
Volumes persist across container restarts, host storage available
latest images work fine
Ports do not collide (developer responsibility)
CI/CD not enforced → dev environment is “safe”

What can break in CI/CD / Kubernetes:

localhost:9092 in Connect fails → Kafka unreachable
Startup race → pipelines try to read before Kafka is ready
Host port collisions if multiple builds run simultaneously
Volumes may not exist or behave differently → Bronze/Silver/Gold data lost

Kubernetes assumptions

Kubernetes assumptions enforced here:

Pods ephemeral → localhost cannot reach other services
IPs change → must use service DNS
Startup order not guaranteed → readiness/liveness probes required
Pods can fail → apps must retry / be idempotent
Volumes are externalized via PVC → pods assume statelessness
No shared host ports → dynamic service ports or ClusterIP used
Scaling / replicas expected → apps must handle multiple consumers
Secrets/configs injected → apps cannot rely on local .env
Potential failures if Docker assumptions remain:
Connect tries localhost:9092 → fails
ETL Jobs start before Kafka is ready → missing data
MDM pipeline loses state if volume not properly configured → corrupted Bronze/Silver/Gold
Kibana cannot reach Elasticsearch → wrong host/IP
Scaling Kafka / Connect fails → race conditions

Summary Table: Where things break

╔═════════════════════════════╗
║ LAYER ║
╠═════════════════════════════╣
║ Networking ║
║ Docker: localhost works ║
║ Kubernetes: localhost isolated; DNS required ║
║ Break: Kafka Connect cannot reach Kafka ║
╚═════════════════════════════╝

╔═════════════════════════════╗
║ Startup order ║
╠═════════════════════════════╣
║ Docker: depends_on works ║
║ Kubernetes: startup unordered ║
║ Break: ETL jobs fail on unavailable source ║
╚═════════════════════════════╝

╔═════════════════════════════╗
║ State ║
╠═════════════════════════════╣
║ Docker: local volumes ║
║ Kubernetes: pods ephemeral; use PVs ║
║ Break: Bronze/Silver/Gold lost on restart ║
╚═════════════════════════════╝

╔═════════════════════════════╗
║ Port binding ║
╠═════════════════════════════╣
║ Docker: host ports available ║
║ Kubernetes: dynamic ports; cluster network ║
║ Break: port collision or misrouting ║
╚═════════════════════════════╝

╔═════════════════════════════╗
║ Scaling ║
╠═════════════════════════════╣
║ Docker: manual / optional ║
║ Kubernetes: replicas normal ║
║ Break: consumer duplication / missed msgs ║
╚═════════════════════════════╝

╔═════════════════════════════╗
║ Config ║
╠═════════════════════════════╣
║ Docker: .env files ║
║ Kubernetes: ConfigMaps / Secrets ║
║ Break: jobs fail; app misconfigured ║
╚═════════════════════════════╝

╔═════════════════════════════╗
║ Health ║
╠═════════════════════════════╣
║ Docker: optional ║
║ Kubernetes: readiness/liveness required ║
║ Break: app starts too early → errors ║
╚═════════════════════════════╝

Mental picture

Docker Compose (dev)

Single host
localhost:9092 → Kafka
depends_on → partial order
Volumes → local paths
Ports → host mapped
App assumes success
Key insight

Kubernetes (prod)

Multi-node cluster
kafka.svc.cluster.local:9092
readiness_Probe + retries
Persistent_Volumes / Stateful_Sets
Ports abstracted via Services
App assumes failure / retries

Closing Remarks

Docker Compose hides bad assumptions: “works on my laptop”
Kubernetes exposes them: “does your pipeline really work anywhere?”

Warning:

In an enterprise MDM pipeline, ignoring these differences = data loss, compliance violations, or audit failures.

ETL vs ELT — Why Notebook Placement Matters Body / Content: Yes — the placement of your Databricks

Fri, 06 Feb 2026 14:09:12 +0000

ETL vs ELT — Why Notebook Placement Matters

Body / Content:

The placement of your Databricks notebooks determines whether your pipeline behaves like ETL or ELT.

If Databricks runs after loading data into the lake, it’s ELT.
If Databricks runs

before loading transformed data, it’s ETL.

Business Analogy:

Clean vegetables before putting them in the fridge → ETL

Dump everything in the fridge and clean later → ELT

Architecture Mapping:

RAW → BRONZE in ADF = Extract + Load (EL)
SILVER → GOLD in Data-bricks = Transform (T)

✅ This makes your pipeline ELT, the modern standard:

Cheap storage Scalable compute Delta Lake supports late
transformations

Notebooks preserve raw data and give flexibility