Coder Legion - Recent posts tagged cloud-computing

From Big Data to the Cloud, Part 1 — Why One Machine Is Never Enough

Sun, 25 Oct 2026 05:46:00 +0000

A calm, sequential guide for anyone entering the field. The article is divided into smaller chapters to make it easier to understand.

This series grows out of the Cloud Computing labs I teach to fourth-year Computer Science students at the University of Hilla. The labs move quickly, because a lab has ninety minutes. The articles can afford to slow down. Here we take the same path the labs take, but we stop at every step and ask the question a careful student always asks: why this, and not something simpler?

The path itself is short to state. Data has grown past the point where one machine can store it, process it, or analyse it efficiently. The answer is not a bigger machine but many ordinary machines working together. And the framework that manages those machines is Apache Hadoop. Each of those three sentences hides a chain of reasoning, and the purpose of this part is to lay that chain out link by link — so that by the end, the cluster does not look like a fashionable choice but like the answer the problem forces on us.

The destination of this series is the cloud. But we will not begin there, because a definition of the cloud means little until you have felt the problem it was built to solve. So the series takes the long road on purpose: first the problem (this part), then the cluster that answered it and the burden that answer carried (Part 2), and only then the cloud — defined in Part 3, and explored service by service in the parts after it.

We will not touch a configuration file in this part. What we build here is the problem — precisely enough that when Part 2 opens Hadoop up and looks at HDFS, MapReduce, and YARN from the inside, each component reads as the answer to a question we have already asked.

Chapter 1: Where All This Data Comes From

Every Action Leaves a Record
Before we define anything, it is worth noticing how ordinary the sources of big data are. Every click, every purchase, every email, every chat message, and every social post creates data. None of these events is large on its own.

But the sources do not stop at people typing. Sensors and IoT devices report what they measure. Banks record every transaction. Hospitals keep patient records. Streaming services log what their users watch. Each of these produces records continuously, whether or not anything interesting is happening.

The individual record is small. What makes the situation new is the multiplication: small records, produced by enormous numbers of people and devices, without end. Data now reaches billions of records, and petabytes every day. The important words there are every day. The data does not arrive once. It keeps arriving.

Chapter 2: What Big Data Actually Means

A Definition About Tools, Not Size
The most useful definition of big data is not a number. It is this: big data is datasets so large and complex that traditional tools — spreadsheets, a single database server — cannot store, process, or analyse them efficiently.

It is worth being exact about why the definition is phrased this way. Students often ask how large data must be before it “counts” as big. The definition answers by pointing at the tools rather than at a size: data becomes big at the moment the ordinary tool stops coping with it. Notice also that it names three separate jobs — store, process, analyse — and a tool can fail at any one of them.

The Goal Is Insight, Not Storage
A second point is easy to miss: the goal is not only storing data. The goal is extracting insights from it — turning a mountain of records into a decision someone can act on.

The classic example is an online store. It already keeps every purchase as a matter of bookkeeping. The value appears only when it analyses customer habits: what people buy, when, and together with what. With that analysis, the store can make targeted offers and discounts, which increases revenue. The data was there all along; what matters is the ability to read it. Hold onto this, because a system that can store huge data but cannot analyze it has solved only half the problem.

Three Shapes of Data
Data does not only come in different sizes. It comes in different shapes, and the shape decides which tools can handle it.

Structured data is the familiar case: every row has the same columns. A student table with a name, a department, and an age is structured.
Semi-structured data carries its own labels — tags in XML and HTML, keys in JSON — but does not promise that every record has the same fields. The keys tell you what each value means, yet there is no rigid schema forcing every record into one shape.
Unstructured data has no predefined format at all. An email body, a social post, an image, a video: the meaning is in there, but no column tells you where.

And here is the uncomfortable fact: most of today’s data is unstructured, and that is exactly what traditional databases handle worst. They were built around the first row of that table. The world moved to the third.

Chapter 3: The Five V’s

A Vocabulary for “Why It Is Hard”
“Too large and too complex” is a good definition, but it is too blunt to design a system against. We need to name the separate ways data can be difficult, because each one places a different demand on the machine that handles it. The standard vocabulary for this is the five V’s.

One at a Time
01 — Volume: how much. The huge amount of data, from terabytes to petabytes. Volume is the V that breaks storage first.

02 — Velocity: how fast. How fast data is created and must be processed — think of stock trading or live streams. The second half of that sentence is the important one: these systems need answers quickly, not just storage quickly. Velocity is the V that breaks time.

03 — Variety: how many shapes. Many types and formats of data — the structured, semi-structured, and unstructured shapes from Chapter 2, often mixed together. Variety is the V that breaks fixed schemas.

04 — Veracity: how trustworthy. How accurate and trustworthy the data is. Real data is noisy, messy, and has missing values, so cleaning and checking quality stands between you and a conclusion you can defend.

05 — Value: what we get out of it. The useful insight we finally get out of the data. This is the V that justifies the other four — the point Chapter 2 already made, now given its own name.

Notice the split: the first three describe the data itself, while the last two describe whether we can trust it and whether it was worth the effort.

Chapter 4: Why Traditional Systems Fail
What “Traditional” Means Here
By a traditional system we mean one built for structured data on a single powerful machine. All the data lives on that machine; all the work runs on it. This design is good at what it was built for. The point of this chapter is that big data asks it to do something it was never built for — and it breaks in five places.

The Five Breaking Points

It was built for structured data on one machine. Its whole design assumes fixed rows and columns, and one computer doing all the work.
It cannot keep up with the volume and speed of modern data. This is Volume and Velocity together: too many bytes, arriving too fast, for one machine to absorb.
It has difficulty handling unstructured data — images, video, text. This is Variety. And since most of today’s data is unstructured, this is not an edge case; it is most of the data.
Scaling up means buying a bigger, very expensive server — and there is always a limit. When one server is not enough, the traditional move is to scale up: replace it with a bigger one. That works for a while, but each step costs more, and eventually there is no bigger machine to buy.
It is slow to extract insights from huge datasets. Put the first four together and the last failure follows on its own. This is the failure that matters most, because it is a failure of Value.

Three Challenges, Stated Plainly
All five breaking points reduce to three questions any big data system must answer.

Storage — where do we keep data that keeps growing?
Processing — how do we analyse it fast enough to be useful?
Scalability — how do we grow (or shrink) resources as data changes?

Read the third question twice, because it hides a detail that will matter later in this series. It does not only ask how to grow. It asks how to grow or shrink — how to have exactly as much capacity as the data needs, no more and no less. Keep that small word “shrink” in your pocket; we will come back for it.

The single powerful machine answers none of them well once the data is big enough. So the question becomes: what architecture answers all three?

Chapter 5: The Distributed Idea

Many Ordinary Machines, Working Together
The answer is to stop making one machine bigger. Instead of one big machine (monolithic), we use many ordinary machines working together, called a cluster. Each machine in the cluster is a node. Both the data and the processing are split across the nodes. A cluster can range from a few nodes to thousands — and cheap commodity hardware is enough: normal PCs and servers, not specialised machines.

This strategy is called scaling out: when you need more capacity, you do not replace the machine — you add more machines.

The Price of Distribution
Scaling out is not free. The moment we spread data across many machines, we inherit three problems the single machine never had:

Which data lives on which node? With one machine, a file is simply on its disk. With many, something must remember where every piece went.
How do nodes communicate and coordinate? Many machines working on the same job is only useful if the work is divided and the results are brought back together.
What happens when a machine crashes? On one server, a crash is a rare event. In large clusters, failures are normal.
That last point changes how the whole system must be designed. If failure is the normal state of affairs, the system cannot treat it as an emergency. It must expect machines to fail and keep working anyway, without losing data.

Stated as a General Rule
Scaling out trades one hard problem — a machine that cannot grow past its limit — for three new ones: tracking where data lives, coordinating the nodes, and surviving failure. The trade is worth it only if software solves those three problems, so that the people using the cluster do not have to. What the field needed was a framework that solves them once, for everyone.

Chapter 6: Enter Hadoop

What Hadoop Is
Apache Hadoop is an open-source framework for distributed storage and parallel processing of big data. It runs on clusters of commodity hardware — exactly the cluster Chapter 5 described. It was created around 2005, when the volume of web data, from sites like Facebook and YouTube, exploded. It is free to download from Apache, and commercial distributions also exist, such as Cloudera, Hortonworks, and MapR.

Its Core Components, at a Glance
The cleanest way to understand Hadoop’s design is to hold it up against the three challenges from Chapter 4. Each core component answers one of them.

We deliberately stay at this altitude for now. How HDFS splits and tracks blocks, why MapReduce runs the program where the data already is, and how YARN hands out CPU and memory are the subject of Part 2.

What Hadoop Buys

Scalable — add more nodes as data grows. (Notice: as data grows. The “shrink” half of the Scalability question is still waiting.)
Fault tolerant — keeps copies of data on different machines, so a hardware failure does not lose data. This is the answer to “what happens when a machine crashes?”
Cost-effective — uses cheap commodity machines.
Flexible — handles structured and unstructured data. This is the answer to Variety.
Parallel — many nodes work at the same time.

What Hadoop Costs
An honest account includes the other column.

It is designed for batch processing, not real time. For real-time work, Apache Spark is better.
It is complex to configure and maintain — which is exactly what the labs practise.
Replication uses extra storage space. Keeping copies on different machines is what makes Hadoop fault tolerant, but every copy takes up disk.

Where It Is Used
Healthcare: patient records, medical images, early disease prediction.
Banking: fraud detection, transaction analysis.
Social media: sentiment analysis of posts.
E-commerce: recommendations, customer behaviour, inventory management — the online store from Chapter 2, now at full scale.

What Comes in Part 2

We now have the problem in full, and the shape of the answer. We know why one machine is never enough, why a cluster is the way out, and which new problems the cluster creates. We have met Hadoop’s components by name. What we do not yet have is the inside view.

Part 2 opens the components one at a time. We follow a 400 MB file as HDFS splits it into 128 MB blocks, and see why, with three copies of each block, that file ends up stored as twelve blocks across the cluster. We meet the NameNode that keeps track of which block is on which machine, and the DataNodes that send it a heartbeat to say “I’m alive” — and we ask what happens when one of them stops. We walk through MapReduce by hand, on a word count and on an average-age example, from Map through Shuffle & Sort to Reduce. And we watch YARN take a submitted job, start an ApplicationMaster for it, and request containers on the nodes that hold the data.

The single principle running through all of it: Hadoop moves the work to the data, not the data to the work.

And Part 2 will end where the rest of this series begins. Hadoop solves storage, processing, and growth — but it leaves us holding a cluster: machines to buy, a framework that is complex to configure and maintain, and capacity that can grow but not easily shrink. That remaining burden is exactly what the cloud takes off our hands, and it is where Part 3 picks up the story.

References

Deshpande, T., Karanth, S., & Turkington, G. Hadoop: Data Processing and Modelling (Learning Path). Packt Publishing.
Oracle. What Is Big Data? https://www.oracle.com/big-data/what-is-big-data/
Big Data — Understanding Hadoop: Architecture, Features, Challenges & Applications. Medium. https://medium.com/@rp99452/big-data-understanding-hadoop-architecturefeatures-challenges-applications-c7ab7011e375
Mehta, C. B. (2022). A Comprehensive Review of Hadoop Ecosystem for Big Data Processing and Analytics. IJNRD, 7(10).

From “Why Is This Pod Pending?” to `HTTP/1.1 200 OK`: My NGINX Gateway Fabric Lab

Thu, 15 Oct 2026 15:15:48 +0000

A beginner-friendly Kubernetes networking story about Gateway API, Services, HTTPRoutes, DNS, a few misleading errors, and the joy of finally seeing a successful response.

The mission

The goal of this exercise was simple to say and slightly less simple to do: deploy NGINX Gateway Fabric in my Kubernetes cluster, create a small NGINX web application, and make the application accessible through a Kubernetes Gateway and HTTPRoute.

The intended request path was:

My Kali terminal
      |
      | HTTP request to NodePort :30080
      v
NGINX Gateway
      |
      | HTTPRoute matches path /
      v
Kubernetes Service: frontend-svc:80
      |
      v
Pod: frontend-app (NGINX listens on port 80)
      |
      v
HTTP/1.1 200 OK

Think of it like sending a parcel. The Gateway is the front desk, the HTTPRoute is the instruction that says where the parcel should go, the Service is the stable address for the destination, and the Pod is the actual person receiving it.

This post walks through what I did, the problems I hit, and how I worked through them. The useful lesson was not just getting the page to load. It was learning to test one layer at a time instead of randomly changing things until Kubernetes takes pity on me.

1. Installing the Gateway API building blocks

I started in my nginx-gateway-fabric working directory and applied the standard Gateway API CRDs:

kubectl apply -k config/crd/gateway-api/standard

CRD means CustomResourceDefinition. Kubernetes comes with built-in resources such as Pods, Deployments, and Services. CRDs teach Kubernetes about additional resource types. The Gateway API CRDs add resources such as GatewayClass, Gateway, and HTTPRoute.

I then installed the NGINX Gateway Fabric-specific CRDs and the NodePort deployment for version 1.6.1:

kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.1/deploy/crds.yaml

kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.1/deploy/nodeport/deploy.yaml

The NodePort deployment gave me a way to reach the Gateway using a port on a Kubernetes node. In this lab, HTTP was exposed on port 30080.

Problem 1: Kustomize timed out while fetching the CRDs

My first attempt to fetch the Gateway API CRDs directly from GitHub using kubectl kustomize failed with a Git fetch timeout. Because the command failed to produce the manifests, the following kubectl apply command reported that no objects had been passed to it.

That second message was a consequence of the first failure, not a separate mystery.

What I did: I worked with the repository locally and applied the Kustomize directory from the checked-out source:

kubectl apply -k config/crd/gateway-api/standard

Lesson: When a command downloads configuration and then applies it, check the download/rendering step first. If it produces nothing, kubectl apply cannot magically apply it.

2. The Gateway Pod was Pending

After applying the NGINX Gateway Fabric deployment, I checked the Pod:

kubectl get pods -n nginx-gateway

At first, it showed Pending. My first attempt to describe it failed too:

kubectl describe pod nginx-gateway-55f96886f7-j4kzq

The Pod was not found because I had left out its namespace. It lived in nginx-gateway, not the default namespace.

The corrected command was:

kubectl describe pod nginx-gateway-55f96886f7-j4kzq -n nginx-gateway

The important event was:

0/2 nodes are available: 2 node(s) had untolerated taint(s).

What does that mean?

A taint is a rule on a Kubernetes node that says, in effect, “Don't schedule Pods here unless they tolerate this condition.” A toleration is a matching permission in a Pod's specification.

The Gateway Pod was not necessarily broken. Kubernetes was saying it could not find a node where the Pod was allowed to run. I also saw that k8s-worker1 briefly became NotReady before returning to Ready.

What I learned: Check the node status and the Pod's scheduling events before changing the deployment. Taints may be protecting the cluster for a reason, so don't remove them blindly just to make a Pod start.

Useful commands:

kubectl get nodes -o wide
kubectl describe pod nginx-gateway-55f96886f7-j4kzq -n nginx-gateway
kubectl describe node k8s-worker1

Eventually, the Gateway Pod was scheduled to k8s-worker1 and progressed toward running.

3. Creating the frontend application

Next, I created a simple NGINX Pod:

kubectl run frontend-app --image=nginx --port=8080

One subtle detail caused a later problem: --port=8080 does not reconfigure NGINX to listen on port 8080. It records container-port metadata in the Pod specification. The NGINX image still listens on port 80 by default.

I exposed the Pod as a Service:

kubectl expose pod frontend-app \
  --name=frontend-svc \
  --port=80 \
  --target-port=8080

A Service gives a stable network address to a group of Pods. This is useful because Pods can be replaced, and their IP addresses can change. Clients can keep talking to the Service instead of tracking individual Pod IPs.

I checked the Service and its endpoints:

kubectl describe svc frontend-svc -n default
kubectl get endpoints frontend-svc -n default

Initially, the endpoint showed port 8080. That looked consistent with the Service configuration, but it was not consistent with the port NGINX actually used.

Problem 2: The Service pointed to the wrong port

I tested NGINX from inside its own Pod:

kubectl exec frontend-app -n default -- \
  curl -I --max-time 5 http://127.0.0.1:8080/

It failed:

curl: (7) Failed to connect to 127.0.0.1 port 8080

Instead of guessing, I tested the default NGINX port:

kubectl exec frontend-app -n default -- \
  curl -I --max-time 5 http://127.0.0.1:80/

That returned:

HTTP/1.1 200 OK
Server: nginx/1.31.7

There was the clue. NGINX was alive; I had simply sent it to the wrong port.

Fix: I changed the Service's targetPort to 80:

kubectl patch svc frontend-svc -n default \
  -p '{"spec":{"ports":[{"port":80,"targetPort":80}]}}'

Then I checked it again:

kubectl describe svc frontend-svc -n default

The endpoint now pointed to the Pod on port 80.

Lesson: The Service's port is the port clients use to reach the Service. The targetPort is the port on the backend Pod that receives the traffic. Those values can be different, but targetPort must match the application that is actually listening.

4. Building the HTTPRoute

The next step was to tell the Gateway where to send incoming HTTP requests. I created an HTTPRoute called frontend-route.

The important parts of the route were:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: frontend-route
  namespace: default
spec:
  parentRefs:
    - name: nginx-gateway
      namespace: nginx-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: frontend-svc
          port: 80

In plain English:

Attach this route to the nginx-gateway Gateway in the nginx-gateway namespace.
Match requests whose path starts with /.
Forward matching requests to the Service named frontend-svc on port 80.

I checked the result with:

kubectl describe httproute frontend-route -n default

The good news was:

Type:    Accepted
Status:  True

Type:    ResolvedRefs
Status:  True
Message: All references are resolved

These conditions mean the Gateway accepted the route and Kubernetes could resolve its references. They are useful signals, but I still needed to test actual traffic.

Problem 3: The HTTPRoute could not find the backend Service

My first route was in the nginx-gateway namespace and referred to frontend-svc. Kubernetes returned:

BackendNotFound
spec.rules[0].backendRefs[0].name: Not found: "frontend-svc"

I had created frontend-svc in the default namespace. A Service reference in an HTTPRoute is normally resolved in the route's own namespace. Giving the Service the correct name was not enough; the namespace mattered too.

I edited the route YAML and placed the working frontend-route in default, while keeping its parentRefs pointing to the Gateway in nginx-gateway.

That fixed the reference. The route then reported both Accepted=True and ResolvedRefs=True.

Lesson: In Kubernetes, always check both the resource's name and namespace. Two resources can share a name in different namespaces, and a route can be accepted while still having an unresolved backend reference.

5. Testing the Service before testing the Gateway

Before troubleshooting the Gateway, I tested the Service directly. First, I checked that CoreDNS was running:

kubectl get pods -n kube-system
kubectl get svc -n kube-system kube-dns

Then I created a short-lived diagnostic Pod:

kubectl run dns-test --image=busybox:1.36 \
  --restart=Never --command -- sleep 300

kubectl wait --for=condition=Ready pod/dns-test \
  -n default --timeout=60s

I inspected its DNS settings:

kubectl exec dns-test -- cat /etc/resolv.conf

The file showed the cluster DNS Service IP, 10.96.0.10. I then tested DNS resolution:

kubectl exec dns-test -- \
  nslookup kubernetes.default.svc.cluster.local

kubectl exec dns-test -- \
  nslookup frontend-svc.default.svc.cluster.local

Both names resolved. The frontend Service resolved to 10.104.158.51.

Problem 4: One curl test could not resolve the Service name

An earlier temporary curl Pod returned:

curl: (6) Could not resolve host: frontend-svc.default.svc.cluster.local

That was confusing because the DNS Service and CoreDNS Pods looked healthy.

Rather than restarting DNS immediately, I used a separate diagnostic Pod to test the basics. Its nslookup commands successfully resolved both the Kubernetes API Service and frontend-svc. Then I tested the Service itself:

kubectl exec dns-test -- wget -S -O- \
  http://frontend-svc.default.svc.cluster.local/

This returned the default NGINX welcome page and an HTTP/1.1 200 OK response.

What I can conclude: DNS resolution and Service connectivity worked from the diagnostic Pod. The earlier curl test failed, but the evidence did not establish its exact cause. The right move was to reproduce the test and inspect the layers separately rather than assume CoreDNS was broken.

6. The Gateway was programmed, but my first request returned a 500

I checked the Gateway and its Service:

kubectl get svc -n nginx-gateway
kubectl describe gateway nginx-gateway -n nginx-gateway

The Service showed:

TYPE       NodePort
PORT(S)    80:30080/TCP,443:30081/TCP

The Gateway status showed Accepted=True and Programmed=True. That meant the Gateway configuration had been accepted and programmed, but an end-to-end test was still needed.

I checked the route in nginx-gateway and discovered that a second route with the same name existed there. Unlike the working route in default, this duplicate route still referenced a backend Service that was not in its namespace. Its status included BackendNotFound.

I deleted the broken duplicate route:

kubectl delete httproute frontend-route -n nginx-gateway

I kept the working route in default.

My first request to the NodePort returned 500 Internal Server Error. After removing the broken duplicate route, I tried again. The request succeeded.

The logs show that the bad duplicate route existed and that the response changed after I removed it. That makes the duplicate route a strong suspect in the earlier failure, although the response alone does not prove it was the only cause.

Problem 5: Two routes, one name, different namespaces

Kubernetes permits resources with the same name in different namespaces. That is not automatically an error, but it made this troubleshooting session confusing: one frontend-route was correctly configured, while the other had an unresolved backend.

Lesson: When a route behaves unexpectedly, check all routes across namespaces:

kubectl get httproute -A

Don't check only the namespace where you expect the route to live.

7. The final test: success through the NodePort

I checked the node addresses:

kubectl get nodes -o wide

The worker node's internal IP was 10.0.2.4. The Gateway's HTTP NodePort was 30080, so I tested:

curl -i --max-time 10 http://10.0.2.4:30080/

At first, I mistakenly typed the placeholder literally:

curl -i --max-time 10 http://WORKER_IP:30080/

The terminal returned Could not resolve host: WORKER_IP. The shell did exactly what I asked: it treated WORKER_IP as a hostname. Placeholders must be replaced with actual values before running a command.

The final request returned:

HTTP/1.1 200 OK
Server: nginx
Content-Length: 896

And, below the headers, the familiar “Welcome to nginx!” HTML page appeared.

That was the end-to-end confirmation I wanted: a request from my Kali terminal entered through the node's NodePort, reached the NGINX Gateway, matched the HTTPRoute, and was forwarded to the frontend Service and Pod.

How a Kubernetes Cluster Works | Control Plane & Worker Nodes

Tue, 29 Sep 2026 12:00:00 +0000

How does a Kubernetes cluster actually work?

I’ve been learning Kubernetes and realized that it’s easy to memorize terms like Control Plane, Worker Node, API Server, Scheduler, and kubelet without really understanding how they fit together.

So I put together a video explaining how a Kubernetes cluster works, including:

Control Plane and Worker Nodes
API Server
etcd
Scheduler
Controller Manager
Pods
kubelet
Container Runtime
kube-proxy
What happens when you run a kubectl command?
Different ways to create and operate a Kubernetes cluster

My goal was to understand the flow, not just memorize component definitions.

🎥 Watch here:
How a Kubernetes Cluster Works | Control Plane & Worker Nodes

I’d love to hear from other Kubernetes learners like me: which part of the cluster architecture took you the longest to understand?

How Kubernetes Works | Kubernetes Explained Simply for Beginners

Mon, 14 Sep 2026 14:41:36 +0000

Kubernetes can feel confusing when you first start learning it.

Pods. Nodes. Services. Deployments. Scaling.

There are so many new terms that it is easy to focus on memorizing definitions instead of understanding how everything connects.

I recently created a simple animated explanation of how Kubernetes works, using a small application and a real-world-style scenario to walk through the concepts.

If you're currently learning Kubernetes, this might help make the pieces click.

What Kubernetes concept took you the longest to understand?

Kubernetes #DevOps #CloudComputing #K8s #CNCF #LFX

From `kubectl` to `jq`: A Beginner's Detective Story Through Kubernetes JSON Paths

Thu, 27 Aug 2026 11:36:35 +0000

Or: How I Learned to Stop Worrying and Love the Dot-Notation

The Setup: Two Nodes, One Mystery

Picture this: you've just spun up a Kubernetes cluster on your laptop. You've got two nodes talking to each other:

NAME          STATUS   ROLES           AGE   VERSION
k8s-worker1   Ready    <none>          9d    v1.36.3
osboxes       Ready    control-plane   9d    v1.36.3

Everything looks calm. But beneath the surface, one node is about to go rogue, and you're about to learn more about JSON paths than you ever thought possible.

This is the story of how a simple assignment—"Check the number of schedulable nodes excluding tainted (NoSchedule) and write the number to a file"—turned into an epic journey through Kubernetes internals, JSONPath syntax, jq filters, and the subtle art of handling null.

Chapter 1: Peeking Under the Hood with JSON

When you run kubectl get pods, you get a nice table. But Kubernetes is holding so much more information than fits in a table. To see everything, you ask for JSON:

k get po -o json

What comes back is a massive JSON object with nested objects, arrays, and maps. It's intimidating at first, but think of it like a Russian nesting doll:

{
  "apiVersion": "v1",
  "items": [
    {
      "kind": "Pod",
      "metadata": {
        "name": "nginx",
        "labels": { "run": "nginx" }
      },
      "spec": {
        "containers": [
          {
            "name": "nginx",
            "image": "nginx"
          }
        ]
      }
    }
  ]
}

The key insight: Every Kubernetes object follows this pattern. There's always a top-level structure, and inside .items[] lives the actual object (Pod, Node, Service, etc.).

Chapter 2: JSONPath — Kubernetes' Built-in X-Ray Vision

JSONPath is like giving kubectl a treasure map. Instead of dumping the entire JSON, you say: "Give me exactly this field."

Success: Extracting Pod Labels

k get pods -o=jsonpath='{.items[0].metadata.labels}'
# Output: {"run":"nginx"}

Boom! We navigated: .items → first item [0] → .metadata → .labels.

Success: Finding the Container Image

k get pods -o=jsonpath='{.items[0].spec.containers[0].image}'
# Output: nginx

Notice how we chain deeper: .spec → .containers[0] → .image. It's like a file path, but for JSON.

Success: Volume Mounts

k get pods -o=jsonpath='{.items[0].spec.containers[0].volumeMounts[0].mountPath}'
# Output: /var/run/secrets/kubernetes.io/serviceaccount

We're getting good at this! But wait...

Chapter 3: The Node Expedition — Where Things Get Interesting

Pods were practice. Now let's look at nodes.

Experiment 1: Getting Node Names

k get nodes -o=jsonpath='{.items[*].metadata.name}'
# Output: k8s-worker1 osboxes

The [*] wildcard says: "For every item in the array, give me this field." It's the JSONPath way of saying "all of them."

Experiment 2: Adding a Newline

k get nodes -o=jsonpath='{.items[*].metadata.name}{"\n"}'
# Output: k8s-worker1 osboxes
# (now with a proper newline!)

JSONPath can inject literal strings. Handy for formatting.

Chapter 4: Custom Columns — Making Your Own Tables

JSONPath is great for single values. But what if you want a table with only the columns you care about?

Enter custom-columns.

Attempt 1: The Syntax Trap

k get nodes -o='custom-columns=Node:{.items[*].metadata.name}'
# Output:
# Node
# <none>
# <none>

What went wrong? With custom-columns, Kubernetes already iterates over .items for you. Using {.items[*]...} is redundant—it expects a path relative to a single item, not the whole list.

The Fix: Drop the `.items[*]`

k get nodes -o='custom-columns=Node:{.metadata.name}'
# Output:
# Node
# k8s-worker1
# osboxes

Aha moment: custom-columns iterates automatically. You write the path as if you're looking at one node, and Kubernetes repeats it for all nodes.

Attempt 2: The Unclosed Brace

k get nodes -o='custom-columns=Node:{.metadata.name},IP:{.status.addresses[*].address'
# error: unexpected path string...

What went wrong? Missing closing } brace. JSONPath expressions in custom-columns must be fully enclosed in {}.

The Fix: Close Your Braces!

k get nodes -o='custom-columns=Node:{.metadata.name},IP:{.status.addresses[*].address}'
# Output:
# Node         IP
# k8s-worker1  10.0.2.4,k8s-worker1
# osboxes      10.0.2.15,osboxes

It works! But notice the IP column has two values per node (InternalIP and Hostname). That's because [*] grabs everything in the array.

Attempt 3: Maps Are Not Arrays

k get nodes -o='custom-columns=Node:{.metadata.name},Capacity:{.status.capacity[*]}'
# error: map[string]interface {} is not array or slice

What went wrong? .status.capacity is a map (key-value pairs: cpu: 2, memory: 2018648Ki), not an array. You can't use [*] on a map.

Attempt 4: The Map Dump

k get nodes -o='custom-columns=Node:{.metadata.name},CPUCapacity:{.status.capacity}'
# Output:
# Node         CPUCapacity
# k8s-worker1  map[cpu:2 ephemeral-storage:242470332Ki hugepages-2Mi:0 memory:2018648Ki pods:110]
# osboxes      map[cpu:2 ephemeral-storage:515464096Ki hugepages-2Mi:0 memory:4011360Ki pods:110]

It works, but it's ugly. We wanted just the CPU number, not the whole map printed as a Go-style map[...].

The Fix: Drill Into the Map

k get nodes -o='custom-columns=Node:{.metadata.name},CPUCapacity:{.status.capacity.cpu}'
# Output:
# Node         CPUCapacity
# k8s-worker1  2
# osboxes      2

Aha moment: Maps are accessed by key, just like objects. .status.capacity.cpu says: "Go into capacity, then get the value for key 'cpu'."

Chapter 5: The Real Boss Fight — Counting Schedulable Nodes

Now for the main event. The assignment sounds simple: count schedulable nodes, excluding those with a NoSchedule taint.

But this is where we descend into the underworld.

Mistake 1: Wrong Resource Type

kubectl get pods -o json | jq -r '.items[] | select(.spec.taints[].effect == "NoSchedule")'
# (empty output)

What went wrong? We queried pods. Pods don't have .spec.taints—nodes do! This is like looking for a car's engine in the trunk.

Mistake 2: The Null Iteration Bomb

kubectl get nodes -o json | jq -r '.items[] | select(.spec.taints[].effect == "NoSchedule")'
# jq: error: Cannot iterate over null (null)

What went wrong? We switched to nodes, but k8s-worker1 has no taints at all. In JSON, a missing field is null. jq tried to iterate over null with [], and jq hates that.

Think of it like this: you asked jq to "open every box in this room," but one of the boxes doesn't exist. jq throws its hands up.

Attempt 3: Filtering for Null Taints

kubectl get nodes -o json | jq -r '.items[] | select(.spec.taints == null)' > schedulable-nodes.txt

This works for finding nodes with no taints, but it writes the entire JSON object of k8s-worker1 to the file. The assignment asked for a number, not a novel.

Also, what if a node has taints, but none of them are NoSchedule? This query would miss it.

The Working Solution: The Full `jq` Pipeline

After much trial and error, here is the command that actually solves the problem:

kubectl get nodes -o json | jq '[.items[] | select(.spec.taints == null or ([.spec.taints[] | select(.effect == "NoSchedule")] | length == 0))] | length' > schedulable-nodes.txt

Let's dissect this beast, piece by piece.

Chapter 6: The `jq` Command Autopsy

Stage 1: Get the Data

kubectl get nodes -o json

Produces the full JSON list of nodes.

Stage 2: Enter `jq`

| jq '...'

jq is a command-line JSON processor. Think of it as grep and awk had a baby that only speaks JSON.

Stage 3: Iterate Over Items

.items[]

This says: "Take the items array and process each element one by one."

Stage 4: The `select()` Filter

select(.spec.taints == null or (...))

select() keeps an item only if the condition is true. We have two conditions joined by or:

Condition A: .spec.taints == null

Does this node have no taints at all?
For k8s-worker1 (when healthy): true → keep it.

Condition B: ([.spec.taints[] | select(.effect == "NoSchedule")] | length == 0)
This is the sophisticated part. Let's break it down:

.spec.taints[] — "Go through each taint on this node."
| select(.effect == "NoSchedule") — "Keep only the taints whose effect is NoSchedule."
[ ... ] — "Put those matching taints into an array."
| length — "Count how many items are in that array."
== 0 — "Is that count zero?"

If a node has taints but zero of them are NoSchedule, this returns true.

Why both conditions? Because if a node has no taints, .spec.taints is null, and you can't iterate over null. Condition A catches the "no taints" case. Condition B catches the "has taints, but none are NoSchedule" case.

Stage 5: Collect and Count

[ ... ] | length

The outer [ ... ] gathers all the nodes that survived the filter into a new array. Then | length counts them.

In our case:

k8s-worker1 (no NoSchedule taint) → KEPT
osboxes (has node-role.kubernetes.io/control-plane:NoSchedule) → DISCARDED

Result: 1

Stage 6: Write to File

> schedulable-nodes.txt

Shell redirection writes the output (just the number 1) to the file.

Chapter 7: The Plot Twist — A Node Goes Dark

Remember when we said k8s-worker1 was healthy? At one point in the session, it wasn't.

The node JSON revealed:

"conditions": [
  {
    "type": "Ready",
    "status": "Unknown",
    "reason": "NodeStatusUnknown",
    "message": "Kubelet stopped posting node status."
  }
],
"spec": {
  "taints": [
    {
      "effect": "NoSchedule",
      "key": "node.kubernetes.io/unreachable"
    },
    {
      "effect": "NoExecute",
      "key": "node.kubernetes.io/unreachable"
    }
  ]
}

The kubelet on k8s-worker1 stopped sending heartbeats. After ~55 minutes of silence, the control plane declared it unreachable and slapped NoSchedule and NoExecute taints on it.

This is why JSON exploration matters. If you only looked at kubectl get nodes, you'd see NotReady or Unknown. But the JSON tells you why: the kubelet died. Maybe the VM crashed. Maybe the battery died on the laptop hosting it. The JSON is the detective's notebook.

Later, the node recovered—its taints disappeared and it started reporting healthy status again. But for a while, the schedulable node count dropped to zero (because both nodes had NoSchedule taints: k8s-worker1 from being unreachable, and osboxes from being the control plane).

Chapter 8: Key Lessons — The JSONPath Survival Guide

Concept	JSONPath Example	What It Does
Single item	`{.items[0].metadata.name}`	First item's name
All items	`{.items[*].metadata.name}`	All names, space-separated
Wildcard in maps	`{.status.capacity.cpu}`	Value for key `cpu`
Wildcard in arrays	`{.spec.containers[*].image}`	All container images
Literal string	`{.metadata.name}{"\n"}`	Add a newline
Custom columns	`-o='custom-columns=NAME:{.metadata.name}'`	Build your own table

The Golden Rules

custom-columns iterates for you. Don't use .items[*] inside it. Use .metadata.name, not .items[*].metadata.name.
Maps ≠ Arrays. You can't use [*] on a map like .status.capacity. Use .status.capacity.cpu instead.
jq hates null iteration. Always guard against null when using []. Use == null or // [] (default empty array) as safety nets.
Know your resource anatomy. Pods have .spec.nodeName and .spec.tolerations. Nodes have .spec.taints and .status.conditions. Don't mix them up!
Count at the end. If the assignment asks for a number, wrap your filter in [ ... ] | length before writing to the file.

The Final File

cat schedulable-nodes.txt

One file. One number. Hours of learning compressed into a single digit.

Epilogue: Why This Matters

Kubernetes is an API-driven system. Everything you see in kubectl is a pretty-printed version of JSON traveling over HTTP. When you learn JSONPath and jq, you stop being a tourist and start being an explorer. You can:

Build monitoring scripts that extract exact metrics
Write admission controllers that inspect object specs
Debug scheduling failures by reading taints and conditions
Automate cluster reports without parsing human-readable tables

Internet-Free Infrastructure: How Two Servers Talk Without Using the Internet

Tue, 25 Aug 2026 17:21:06 +0000

Imagine you're building a SaaS platform.

Nothing unusual.

A web application.

A PostgreSQL database.

A Redis cache.

Everything runs in the cloud.

The architecture looks something like this:

Users
  ↓
Application Server
  ↓
PostgreSQL

Simple.

A user sends a request.

The application queries the database.

The database responds.

The application returns data.

Feature shipped.

Customers happy.

Everyone moves on.

For a while.

The Question Nobody Asks

Most engineers focus on:

Application Logic

Database Queries

API Design

But infrastructure engineers eventually ask a different question:

How does the application server actually reach the database server?

Not conceptually.

Literally.

How does one machine know where another machine is?

How does a packet travel from one server to another?

And perhaps the most surprising question:

If both servers live in the cloud, why doesn't their communication go through the Internet?

The Common Misconception

Many people imagine cloud infrastructure like this:

Application Server
        ↓
     Internet
        ↓
Database Server

It seems reasonable.

After all, both servers exist somewhere inside AWS, Azure, or Google Cloud.

Surely the traffic goes through the Internet.

Right?

Actually, no.

In most production systems, internal communication never touches the public Internet.

Instead, it looks like this:

Application Server
        ↓
 Private Network
        ↓
Database Server

No public Internet.

No ISP.

No external routers.

No public IP addresses.

Just private infrastructure.

This distinction matters because it changes everything about:

Security
Performance
Reliability
Cost
Scalability

To understand why, we need to start with the first building block.

The VPC.

The Cloud Is A Giant City

Imagine AWS as a massive country.

Inside that country:

Netflix

Airbnb

Stripe

Shopify

Your Startup

Thousands of Others

All of them are running servers.

All of them are storing data.

All of them are deploying applications.

Now imagine there were no boundaries.

No isolation.

No separation.

Every server could see every other server.

That would be a security nightmare.

Cloud providers solve this problem using something called a:

Virtual Private Cloud
(VPC)

Think of a VPC as:

Your own private city
inside a giant country.

The cloud provider owns the country.

You own the city.

Inside that city:

Your servers can communicate
Your databases can communicate
Your caches can communicate
Other companies cannot access them

The first thing cloud infrastructure gives you is not a server.

It's a private network.

Why Infrastructure Starts With A Network

New engineers often think infrastructure begins here:

EC2 Instance

Virtual Machine

Server

It doesn't.

Infrastructure usually begins here:

VPC

Because servers without networks are useless.

Imagine buying a house.

But there are:

No roads

No addresses

No transportation

The house exists.

But nobody can reach it.

Servers have the same problem.

Before machines can communicate, they need a network.

The VPC creates that network.

The Next Problem

Suppose your company grows.

Now you have:

Application Servers

Database Servers

Redis

Background Workers

Message Queues

Monitoring Systems

Admin Tools

Everything lives inside one VPC.

At first this sounds convenient.

Until security enters the conversation.

Should a database be reachable from everywhere?

Probably not.

Should every server have Internet access?

Probably not.

Should internal systems be exposed publicly?

Definitely not.

This is where subnets appear.

Subnets Are Neighborhoods

If a VPC is a city:

VPC = City

Then:

Subnet = Neighborhood

A company might create:

VPC
│
├── Public Subnet
│
├── Application Subnet
│
└── Database Subnet

Each subnet serves a purpose.

For example:

Public Subnet

Load Balancer

Private Application Subnet

App Server 1
App Server 2
App Server 3

Private Database Subnet

PostgreSQL
Redis

Notice something important.

The database sits deeper inside the infrastructure.

Further away from the Internet.

This is intentional.

Because good infrastructure is built around reducing exposure.

The Database Doesn't Need The Internet

Many founders are surprised by this.

A database usually doesn't need users.

A database needs applications.

That's a completely different requirement.

Consider:

Customer
    ↓
Application
    ↓
Database

The customer never talks directly to PostgreSQL.

Only the application does.

So exposing the database publicly creates risk without creating value.

This is why production databases often live in private subnets.

The database can still be reached.

But only by trusted systems.

Every Server Gets A Private Address

Now imagine an application server wants to reach PostgreSQL.

How does it find the database?

The same way you find a building.

Addresses.

Every server receives an IP address.

Example:

Application

10.0.1.10

Database

10.0.2.20

These are private IP addresses.

Not public ones.

Nobody on the Internet can reach them.

But machines inside the VPC can.

Now the application knows:

Database

10.0.2.20

Problem solved?

Not quite.

Knowing an address isn't enough.

Knowing The Address Doesn't Create A Path

Imagine someone gives you:

221B Baker Street

Great.

You know the destination.

But how do you get there?

You need roads.

Networks have the same problem.

An IP address tells you:

Where

But not:

How

That responsibility belongs to routing.

Routing Is The GPS Of The Cloud

Every network contains routing rules.

These rules answer one question:

Where should this packet go next?

A route table might contain:

Destination      Target

10.0.0.0/16      Local

Meaning:

Any traffic
inside this VPC
stays inside this VPC

Now imagine:

Application

10.0.1.10

sending a request to:

Database

10.0.2.20

The network checks the routing table.

Destination?

10.0.2.20

Result:

Inside VPC

Therefore:

Deliver Locally

The packet never leaves the cloud network.

The Internet is never involved.

The communication happens entirely inside private infrastructure.

The Request's Actual Journey

When the application queries PostgreSQL:

SELECT * FROM users;

What really happens is:

Application
   ↓
Private IP
   ↓
Route Table
   ↓
Database Subnet
   ↓
Database Server

No Internet.

No DNS lookup to the public Internet.

No external networking providers.

Just internal routing.

This is why internal communication is usually:

Faster
More secure
More predictable
Less exposed to external failures

Why This Matters To Founders

At first glance this sounds like technical trivia.

It's not.

Because infrastructure decisions directly affect business outcomes.

Imagine two architectures.

Architecture A

Application
     ↓
Internet
     ↓
Database

Architecture B

Application
     ↓
Private Network
     ↓
Database

The second architecture provides:

Better security
Lower attack surface
Lower latency
Fewer compliance concerns
Better scalability

The customer never notices.

But the company benefits every day.

This is one of the reasons mature SaaS platforms invest heavily in infrastructure design.

Not because customers ask for VPCs.

Because customers expect reliability.

The Most Important Lesson

When most people think about cloud infrastructure, they think about servers.

Experienced engineers think about networks.

Because before applications communicate:

Networks must exist.

Before databases respond:

Routes must exist.

Before systems scale:

Infrastructure boundaries must exist.

The Internet is only one network.

Most production systems spend far more time communicating inside private networks than across the public Internet.

And understanding that idea is the first step toward understanding modern cloud infrastructure.

Frequently Asked Questions

What is a VPC?

A Virtual Private Cloud (VPC) is an isolated private network inside a cloud provider where your infrastructure resources can communicate securely.

What is a subnet?

A subnet is a smaller network segment inside a VPC used to organize and isolate infrastructure resources.

Why do servers use private IP addresses?

Private IP addresses allow internal communication without exposing systems directly to the public Internet.

Does traffic between servers always go through the Internet?

No.

In most cloud architectures, communication between internal services stays inside the provider's private network.

What is a route table?

A route table contains rules that determine where network traffic should be sent.

Why are databases usually placed in private subnets?

Because databases typically only need to communicate with application servers, not with public users.

Key Takeaways

Most production server-to-server communication never touches the public Internet.
A VPC creates an isolated private network inside the cloud.
Subnets divide a network into smaller security and operational boundaries.
Servers communicate using private IP addresses.
Route tables determine how packets travel through the network.
Security starts with network design, not application code.
Cloud infrastructure is fundamentally a networking problem before it becomes a server problem.
Understanding VPCs, subnets, private IPs, and routes is the foundation of modern cloud architecture.

Confidential Computing

Tue, 25 Aug 2026 12:39:44 +0000

Trust systems have long been a contentious topic in cloud computing. Standard cloud models operate on a passive assumption that you trust your cloud provider and you trust the underlying host OS, and more importantly, you trust the hypervisor not to peek at your running data.

For highly-regulated industries like healthcare, FinTech, and defence contractors, that assumption is ultimately non-compliant. Between nation-state threat models, sophisticated insider risks, and multi-tenant cloud environments, data cannot be left vulnerable while it’s still active. So this is where Confidential Computing was created to address that critical trust problem. Unfortunately, while it solves many of these critical privacy problems, it also introduces some operational trade-offs that might still prevent your security teams from adopting it.

Historically, encryption covered two states:

Data at-rest (on disk)
Data in-transit (over the network).

Confidential Computing completes the trifecta by introducing Data in-use protection. Confidential Computing specifically relies on Trusted Execution Environments (TEEs).

Data at-rest --> Encrypted on disk (AES-256)
Data in-transit --> Encrypted over network (TLS)
Data in-use --> Encrypted within memory (TEEs)

TEEs are hardware-level enclaves embedded directly into the CPU. Memory pages inside a TEE are cryptographically isolated so that neither other tenants, host operating systems, nor hypervisors can read or manipulate the running code. Through remote attestation, an application can cryptographically prove to a remote verifier that it is running inside a legitimate, uncompromised hardware enclave before any sensitive data is sent.

For compliance-heavy sectors, Confidential Computing unlocks capabilities that were previously off-limits in shared public clouds. Multiple hospital networks, as an example, could pool proprietary patient data to train large language models or diagnostic algorithms without exposing Protected Health Information (PHI data) to each other or the cloud vendor. Likewise, banks can perform real-time cross-institutional fraud detection on combined transaction feeds while strictly maintaining individual consumer privacy. Finally, any and all organisations could choose to simplify compliance under strict regimes like the GDPR or any newly-emerging AI governance frameworks by maintaining mathematical and cryptographic proof of data privacy during computation.

While TEEs offer powerful hardware guarantees, they are often misunderstood as a complete security cure-all. A clear look at the threat model reveals significant operational boundaries. For example, those hardware TEEs protect the workload from the hypervisor, but they do not protect the hypervisor or any neighbouring workloads from a malicious or compromised application inside the enclave. Similarly, TEEs require specialised hardware instances. These carry encrypted memory limits and in many cases can force organisations into vendor lock-in or complex application refactoring to keep the Trusted Computing Base (TCB) small. And last but not least, these TEEs unfortunately inevitably contain hardware-level vulnerabilities. Firmware updates and side-channel attacks such as CVE-2024-56161 and BadRAM prove that hardware is not infallible. Patching hardware bugs can mean waiting on microcode updates or replacing silicon entirely.

For full transparency, I work at a cloud-native security company called Edera looking to solve this issue through a zero-trust lens. Since TEEs focus on protecting against a compromised hypervisor, Edera compliments this by focusing specifically on protecting against a compromised workload. Most modern multi-tenant environments need both. Edera successfully bridges the existing blind spots between hardware-bound TEEs and Kubernetes, ensuring businesses don't simply have to choose between having strict isolation and improved dev agility.

When paired with physical TEEs, Edera runs within encrypted memory pages. This delivers complete bidirectional security such as data-in-use encryption isolating your app from the cloud provider. Combined with Edera’s hypervisor-level isolation, you can keep your infrastructure safe from untrusted workload code. Unfortunately, it’s worth stating that not every cloud region, edge device, or Kubernetes cluster has access to specialised TEE hardware required for Confidential Computing. For these specific environments, there’s Edera Confidential Core. If you are not familiar with Edera Confidential Core, it’s a feature of Edera’s ultra-slim Xen-based hypervisor architecture that provides hypervisor-enforced memory encryption and SPIFFE identity-driven remote attestation. It isolates the tenant memory on page faults and provides the core benefits that we’ve come to expect from confidential computing such as encryption in memory as well as strict workload boundary protection, without requiring any specialised hardware or forcing disruptive application rewrites.

Securing data in the cloud shouldn’t be about picking a single security term, instead you should think about how you are going to define your threat model. For example, if your threat model demands protection against a cloud provider or hypervisor peeking at your data, Confidential Computing with TEEs is absolutely ESSENTIAL. If your threat model requires protecting host infrastructure from compromised, user-facing container workloads, hypervisor isolation via Edera becomes the right solution for that specific problem. However, if you need a comprehensive zero-trust isolation without being locked into expensive hardware dependencies, I hope I made the case that Edera Confidential Core is capable of bridging the gap. Cloud security should never be about forcing trade-offs between hardware availability, app performance, and data privacy. By bringing workload isolation into harmony with memory confidentiality, enterprises can run their most sensitive workloads anywhere safely.

Troubleshooting Kubernetes Application Failures: A Real Debugging Session

Tue, 18 Aug 2026 16:09:53 +0000

Here's a scenario every Kubernetes learner eventually runs into: your Pods are all Running, your Deployments say 1/1 or 3/3 AVAILABLE, everything looks healthy in kubectl get... and yet when you try to actually hit your app, you get nothing but silence or a flat-out connection refusal.

That's exactly the situation I found myself in on my two-node cluster (one control-plane node, one worker) while running the classic Example Voting App — vote, redis, worker, db, and result. This post walks through the actual debugging session, the dead ends I hit, and the three separate root causes I eventually tracked down. If you're new to Kubernetes networking, this is the kind of thing you'll hit constantly, so it's worth understanding the pattern rather than just memorizing the fix.

The Setup

A simple 5-component app running across 2 nodes:

$ k get nodes
NAME          STATUS   ROLES           AGE   VERSION
k8s-worker1   Ready    <none>          23h   v1.36.3
osboxes       Ready    control-plane   24h   v1.36.3

$ k get deploy
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
db       1/1     1            1           113m
redis    1/1     1            1           113m
result   1/1     1            1           113m
vote     1/1     1            1           113m
worker   3/3     3            3           113m

Everything's READY. Every Pod is Running. By every surface-level signal, this cluster is healthy. So when curl to the vote service just hung and then refused the connection, it was a good reminder that "Running" only tells you the container started — it says nothing about whether traffic can actually reach it.

The Architecture

Before diving into the debugging, it helps to see the full request path this app takes. Five components, two different NodePorts for user-facing traffic, and three internal ClusterIP hops in between:

The user hits the vote frontend (Python) from outside the cluster, through a NodePort service on 31000.
vote writes each vote into Redis, reached internally via a ClusterIP service — Redis never needs to be exposed outside the cluster.
The worker (.NET) consumes votes off Redis as they come in.
worker writes the processed result into Postgres (the db component), again over ClusterIP.
The result frontend (Node.js) pulls the current tally from Postgres.
The user checks the live results through a second NodePort, 31001.

This is a useful diagram to keep next to your terminal while debugging, because it tells you exactly which Service type to expect at each hop: vote and result are the only two components that should ever be reachable via NodePort — everything else (redis, worker, db) should only ever be reachable ClusterIP-to-ClusterIP, inside the cluster. That distinction is what made the three bugs below easy to isolate: I knew from this diagram alone that if vote (step 1) wasn't reachable, the problem had to be in the vote Service or Pod itself — and if redis couldn't be reached by worker (step 3), the problem had to be either a Service selector or a NetworkPolicy sitting between them.

Where To Even Start

When an app "isn't working" in Kubernetes, the traffic path you're debugging usually looks like this:

Client → NodePort/Service → Endpoints → Pod IP:containerPort → Your app

A failure can happen at any link in that chain, so the debugging flow is basically peeling back one layer at a time:

k get pods -o wide          # Are the pods actually running, and where?
k get svc                   # Do the services exist, with the right type/ports?
k describe svc <name>       # What's the selector, targetPort, and endpoint?
k get ep                    # Does the service actually have any endpoints?
k get po --show-labels      # Do the pod labels match what the service expects?

That last one ended up being the key to almost everything below. Let's go through what actually broke.

Issue #1: NodePort Connects... to Nothing

First symptom — trying to hit the vote app from outside the cluster just failed outright:

$ curl http://10.0.2.4:31000
curl: (7) Failed to connect to 10.0.2.4:31000 after 4 ms: Could not connect to server

The service existed, and it had an endpoint, so my first instinct (checking for missing endpoints) was a dead end:

$ k describe svc vote
Name:                     vote
Selector:                 app=vote
Type:                     NodePort
Port:                     vote-service  5000/TCP
TargetPort:               8080/TCP
NodePort:                 vote-service  31000/TCP
Endpoints:                192.168.194.74:8080

That all looks fine at a glance — there is an endpoint. So I went a layer deeper and checked what port the container inside the Pod was actually bound to:

$ kubectl get pod vote-6c95bd6c8-hjz4s -o jsonpath='{.spec.containers[*].ports}'
[{"containerPort":80,"name":"vote","protocol":"TCP"}]

$ kubectl logs vote-6c95bd6c8-hjz4s
[2026-08-18 12:42:10 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)

And there's the mismatch: the container was listening on port 80, but the Service's targetPort was set to 8080. The Service was faithfully forwarding traffic to a port nothing was listening on — which from the outside just looks like a dead connection.

The fix was a one-line edit to the Service spec:

$ k edit svc vote
# change targetPort from 8080 to 80
service/vote edited

$ kubectl describe svc vote
TargetPort:               80/TCP
Endpoints:                192.168.194.74:80

$ curl http://10.0.2.4:31000
<!DOCTYPE html>
<html>
  ...
  <h3>Cats vs Dogs!</h3>
  ...

Lesson: targetPort on a Service has to match the actual containerPort your app is listening on inside the Pod — not the port you think it should be, and not the port field on the Service itself (those are two different numbers doing two different jobs). Always check kubectl logs or the container's declared port when a Service "has an endpoint" but still won't respond.

Issue #2: A Service With Zero Endpoints At All

Next up, the result service. This one was worse — it didn't even have a broken endpoint, it had no endpoint whatsoever:

$ k describe svc result
Name:                     result
Selector:                 app=results
Type:                     NodePort
Port:                     result-service  5001/TCP
TargetPort:               80/TCP
NodePort:                 result-service  31001/TCP
Endpoints:

An empty Endpoints: field is one of the most common Kubernetes networking bugs, and it always means the same thing: the Service's selector doesn't match any Pod's labels. Kubernetes isn't "broken" here — it's doing exactly what it's told, which is "find me Pods with these exact labels." If nothing matches, it just quietly gives you nothing.

So I checked the actual Pod labels:

$ k get po --show-labels
NAME                     LABELS
result-58c64976b-7qqh4   app=result,pod-template-hash=58c64976b

Found it. The Service selector was looking for app=results (plural), but the Pod was labeled app=result (singular). One character, zero traffic. This is an incredibly easy typo to make and an incredibly hard one to see just by glancing at YAML — describe output puts it right in front of you.

Fix: edit the Service selector to match the Pod's real label.

$ k edit svc result
service/result edited

$ k get ep
NAME     ENDPOINTS
result   192.168.194.83:80

Endpoint populated immediately — no Pod restart needed, since the Service just watches for matching labels live.

Issue #3: NetworkPolicies Fail Silently Too

At this point vote and result were both reachable, but vote still couldn't talk to redis internally — a NetworkPolicy was involved:

$ k describe netpol access-redis
Name:         access-redis
Spec:
  PodSelector:     app=redis
  Allowing ingress traffic:
    From:
      PodSelector: app=frontend
  Policy Types: Ingress

This policy says: "Only allow ingress to redis Pods from Pods labeled app=frontend." Sounds reasonable — except nothing in this cluster was labeled app=frontend. The actual frontend-facing app was labeled app=vote:

$ k get po --show-labels | grep vote
vote-6c95bd6c8-hjz4s   app=vote,pod-template-hash=6c95bd6c8

Same root cause as Issue #2 — a selector referencing a label that doesn't exist anywhere in the cluster — just applied to a NetworkPolicy instead of a Service. That's worth internalizing: NetworkPolicies are also just label selectors under the hood, and they fail the exact same silent way. There's no error, no event, no warning. Traffic is simply dropped, and unless you specifically go look at k get netpol you might never think to check it.

$ k edit netpol access-redis
# change podSelector under "From" to app=vote
networkpolicy.networking.k8s.io/access-redis edited

A Quick Debugging Checklist

If your app is deployed, Pods are Running, but traffic isn't flowing, walk this list roughly top to bottom:

k get pods -o wide                              # Pods running? Which node/IP?
k get svc                                        # Service exists, right type/ports?
k describe svc <svc-name>                        # selector, targetPort, endpoints
k get ep                                         # any endpoints at all?
k get po --show-labels                           # do labels actually match the selector?
kubectl logs <pod>                               # is the app listening where you expect?
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].ports}'  # real containerPort
k get netpol                                     # any policies restricting traffic?
k describe netpol <name>                         # does its selector match real Pods?

A couple of things I ran into along the way that are worth a quick mention: some minimal app images don't ship with ss, netstat, or even much of a shell (exec: "ss": executable file not found in $PATH was a fun one), so don't assume you can always debug from inside the container — sometimes kubectl logs and the Pod's declared ports are more reliable than trying to exec in. Also, kubectl get endpoints is deprecated in favor of discovery.k8s.io/v1 EndpointSlice from v1.33 onward, so kubectl get endpointslice -l kubernetes.io/service-name=<svc> is the more future-proof command going forward.

None of these were exotic bugs — a wrong port number, a pluralized label, and a stale selector in a policy. But that's honestly the nature of most Kubernetes networking issues: small, easy-to-miss mismatches between YAML files that Kubernetes will never flag as an "error," because as far as the control plane is concerned, you asked for exactly what you got. Learning to methodically walk the Service → Endpoints → Pod chain is what turns "my app doesn't work and I have no idea why" into a two-minute fix.

Kubernetes ETCD Backup And Restore — Part 2: Restoring etcd and Recovering the Cluster

Mon, 17 Aug 2026 10:56:09 +0000

In Part 1, we created and validated an etcd snapshot while an nginx Deployment existed. Now we deliberately remove that workload, restore the snapshot, reconfigure the static etcd Pod, troubleshoot the problems that appear, and bring etcd back to a healthy state.

This is Part 2 of a two-part series.

The restore workflow covered:

deleting the test nginx Deployment;
making and diagnosing an incorrect restore command;
restoring the snapshot into a separate data directory;
understanding why the restored data should initially be kept separate;
handling filesystem and sudo mistakes;
changing the static etcd manifest;
understanding the impact of the static Pod manifest directory;
troubleshooting namespace and RBAC errors;
investigating an etcd Pod stuck in Pending;
checking kubelet;
restarting kubelet to reconcile the static Pod;
confirming that etcd returned to 1/1 Running;
documenting the final recovery workflow and lessons learned.

The important distinction is that the terminal evidence explicitly confirms successful snapshot restoration and recovery of the etcd Pod. It does not include a final k get all proving that the nginx Deployment reappeared after recovery, so this article does not claim that as independently verified.

17. Delete nginx

I deleted the Deployment:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k delete deploy nginx
deployment.apps "nginx" deleted from default namespace

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m

The nginx Deployment was gone.

This gave us the desired recovery scenario:

BEFORE BACKUP
    |
    +-- nginx Deployment exists
    |
    v
TAKE ETCD SNAPSHOT
    |
    v
DELETE nginx
    |
    v
nginx no longer exists
    |
    v
RESTORE SNAPSHOT
    |
    v
expect previous cluster state

18. First restore mistake: wrong snapshot filename

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --data-dir /opt/etcd/backup snapshot restore snapshot.db

The restore process immediately told me:

Error: open snapshot.db: no such file or directory

Again, the filename was wrong.

The actual snapshot was:

/opt/etcd-backup.db

This is a very common command-line mistake: creating a file with one name and later trying to restore a similarly named file.

The first restore attempt was therefore not an etcd failure. It was simply a path/filename error.

19. Restore the snapshot into a separate data directory

Instead of overwriting the active etcd data directory immediately, I restored the snapshot into a separate directory:

/var/lib/etcd-restore-from-backup

The command used was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restore-from-backup

The output included:

Deprecated: Use `etcdutl snapshot restore` instead.

and then:

restoring snapshot

followed by:

Trimming membership information from the backend...

and:

added member

Finally:

restored snapshot

This was the actual successful restore of the snapshot into the new data directory.

20. Why use a separate restore directory?

The original etcd data directory was:

/var/lib/etcd

The restored copy was:

/var/lib/etcd-restore-from-backup

This distinction is important.

It allows the restored data to be prepared without immediately destroying the existing data directory.

For a beginner, think of it like restoring a database backup into a new folder first:

Original:
    /var/lib/etcd

Restored:
    /var/lib/etcd-restore-from-backup

Only after the restored data exists do we configure etcd to use it.

21. More path mistakes while inspecting the restored data

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cd /var/lib/data
bash: cd: /var/lib/data: No such file or directory

Again, the directory did not exist.

The restore directory was:

/var/lib/etcd-restore-from-backup

I also tried:

sudo cd /var/lib/etcd-restore-from-backup

and got:

sudo: cd: command not found
sudo: "cd" is a shell built-in command, it cannot be run directly.

This error is worth understanding.

cd is a shell built-in command. sudo normally executes an external command, so:

sudo cd ...

does not work.

Instead, because the directory required elevated permissions, I used:

sudo ls /var/lib/etcd-restore-from-backup

which showed:

member

That confirmed that the restore had created the expected etcd member directory structure.

22. Editing the static etcd manifest

The next step was to configure the static etcd pod to use the restored data directory.

The original manifest used:

--data-dir=/var/lib/etcd

The modified configuration used:

--data-dir=/var/lib/etcd-restore-from-backup

The corresponding hostPath was also changed to:

/var/lib/etcd-restore-from-backup

The resulting Pod description later confirmed:

--data-dir=/var/lib/etcd-restore-from-backup

and:

Path: /var/lib/etcd-restore-from-backup

This is how the restored database was connected to the etcd process.

23. Why moving the manifest files caused disruption

During the troubleshooting process, I temporarily moved the static pod manifests out of the manifest directory:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv * /tmp

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 0

The directory was empty.

I then restored the manifests:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv /tmp/*.yaml .

and confirmed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 16
-rw------- 1 root root 3939 Aug 13 11:07 kube-apiserver.yaml
-rw------- 1 root root 3230 Aug 13 11:07 kube-controller-manager.yaml
-rw------- 1 root root 1726 Aug 13 11:07 kube-scheduler.yaml
-rw------- 1 root root 2596 Aug 13 12:37 etcd.yaml

This demonstrates how sensitive the static pod manifest directory is.

When the kubelet stops seeing a static pod manifest, the corresponding static pod can disappear.

For this reason, moving files in /etc/kubernetes/manifests should be done carefully.

24. Another small typo

I accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get oi
error: the server doesn't have a resource type "oi"

The intended command was clearly not oi.

The correct command for Pods was:

k get po

This is a harmless error, but it illustrates another important habit:

When kubectl reports that a resource type does not exist, first check the command spelling before assuming the Kubernetes API is broken.

25. The RBAC error after the manifest changes

After the static pod manipulation, this happened:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po
Error from server (Forbidden): pods is forbidden: User "kubernetes-admin" cannot list resource "pods" in API group "" in the namespace "default"

This was different from the earlier errors.

This time, the Kubernetes API server was responding, but the authenticated user was being denied access.

The command:

sudo k get po

was also not a solution:

sudo: k: command not found

The reason is that k was a shell alias, and aliases normally are not available to sudo in that way.

More importantly, sudo does not automatically solve Kubernetes RBAC problems.

I then checked the system namespace explicitly:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system

This returned the control-plane Pods.

That helped separate the problem from the actual etcd state.

26. etcd was now Pending

The output showed:

etcd-osboxes   0/1   Pending

I first tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po etcd-osboxes
Error from server (NotFound): pods "etcd-osboxes" not found

Again, the namespace mattered.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes

The description showed the important configuration:

Status: Pending

but also:

Image: registry.k8s.io/etcd:3.6.8-0

and:

--data-dir=/var/lib/etcd-restore-from-backup

The Pod also showed the restored directory mounted:

/var/lib/etcd-restore-from-backup from etcd-data

and:

Path: /var/lib/etcd-restore-from-backup

The kubelet events showed:

Normal  Pulled   ...
Normal  Created  ...
Normal  Started  ...

So the container had actually been created and started even though the Kubernetes status remained Pending.

This is an important troubleshooting clue:

Do not rely on one status field alone. Read the full Pod description and events.

27. Trying to read etcd logs

I tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs etcd-osboxes -n kube-system
Error from server (BadRequest): container "etcd" in pod "etcd-osboxes" is not available

I also accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs

which returned:

error: expected 'logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]'.
POD or TYPE/NAME is a required argument for the logs command

The second error was simply because kubectl logs requires a Pod or workload name.

The first error was more interesting: Kubernetes did not consider the container available for normal log retrieval.

I then checked the Pod in wide format:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get pod etcd-osboxes -n kube-system -o wide
NAME           READY   STATUS    RESTARTS   AGE     IP       NODE      NOMINATED NODE   READINESS GATES
etcd-osboxes   0/1     Pending   0          4m58s   <none>   osboxes   <none>           <none>

The lack of a Pod IP was another indication that the Pod was not fully ready.

28. Inspecting kubelet

Since etcd was a static Pod, the kubelet was a key component to inspect.

I ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl status kubelet --no-pager

The result showed:

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; preset: disabled)
     Active: active (running)

The kubelet was running.

The logs also showed repeated activity around the time of the problem.

At this point, rather than making more changes to the etcd manifest, I restarted kubelet:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl restart kubelet

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ systemctl daemon-reload

29. The recovery

After restarting kubelet, I checked the system namespace again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-74c68c8864-fjtnv   1/1     Running   5 (7m47s ago)   95m
calico-node-dzlcp                          1/1     Running   0               95m
coredns-589f44dc88-l522w                   1/1     Running   0               101m
coredns-589f44dc88-nzsh9                   1/1     Running   0               101m
etcd-osboxes                               1/1     Running   0               6m51s
kube-apiserver-osboxes                     1/1     Running   0               101m
kube-controller-manager-osboxes            1/1     Running   1 (7m32s ago)   101m
kube-proxy-d8krd                           1/1     Running   0               101m
kube-scheduler-osboxes                     1/1     Running   1               101m

This was the desired state.

Most importantly:

etcd-osboxes   1/1   Running

30. Confirming the restored etcd configuration

I then described the etcd Pod again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes

The important portion was:

Status: Running
IP: 10.0.2.15

The etcd container was:

State: Running
Ready: True
Restart Count: 0

And the command still showed:

--data-dir=/var/lib/etcd-restore-from-backup

The mounted volume was:

/var/lib/etcd-restore-from-backup from etcd-data

with the host path:

/var/lib/etcd-restore-from-backup

This confirmed that etcd was now running using the restored data directory.

31. What happened to the nginx Deployment?

The exercise was designed around the snapshot that was taken while nginx existed.

The important sequence was:

1. Create nginx Deployment
2. Wait for nginx to become Running
3. Take etcd snapshot
4. Delete nginx
5. Restore the snapshot
6. Point etcd at restored data
7. Restart kubelet
8. Confirm etcd is Running

The backup validation showed a snapshot revision of:

with:

918 total keys
3.9 MB total size

The restore operation successfully rebuilt an etcd data directory.

The final terminal state confirmed that etcd was successfully running from:

/var/lib/etcd-restore-from-backup

The terminal session provided here does not contain a final k get all after the successful etcd recovery showing nginx reappearing, so I would not claim that nginx restoration was independently verified from the terminal evidence.

That distinction matters.

The exercise successfully demonstrated:

snapshot creation;
snapshot validation;
snapshot restoration;
switching the static etcd Pod to restored data;
recovery of the etcd Pod.

The source terminal output does not provide a final explicit post-recovery k get all proving that the nginx Deployment was recreated from the restored snapshot.

Reference: Validate the Backup Before Restoring

The session showed that etcdctl snapshot status is deprecated and recommended etcdutl.

The working validation command was:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Example output from the session:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

This is useful because a backup file existing on disk does not, by itself, prove that it is a valid etcd snapshot.

Reference: The Restore Command

The restore performed successfully with:

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

The terminal explicitly reported:

Deprecated: Use `etcdutl snapshot restore` instead.

So for a newer workflow, prefer the modern etcdutl snapshot restore command rather than copying the deprecated command blindly.

Troubleshooting Lessons From the Lab

Error 1: Wrong directory

cd /etc/kkubenetes/manifests

and:

cd /etc/kubenetes/manifests

failed because of spelling mistakes.

Fix

cd /etc/kubernetes/manifests

Error 2: Docker container not found

docker exec -it osboxes bash

returned:

No such container: osboxes

Lesson

A Kubernetes node name is not automatically a Docker container name.

Error 3: Permission denied reading etcd files

cat etcd.yaml

returned:

Permission denied

Fix

sudo cat etcd.yaml

The same principle applied to the protected etcd private key.

Error 4: Invalid etcdctl flag

---endpoint=...

returned:

Error: bad flag syntax

Fix

Use the correctly spelled flag:

--endpoints=...

Error 5: Private key permission denied

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied

Fix

Run the etcd client command with sufficient privileges:

sudo etcdctl ...

Error 6: Wrong snapshot filename

snapshot.db

did not exist.

The actual file was:

/opt/etcd-backup.db

Lesson

Always verify the exact backup path before restoring.

Error 7: `etcdctl snapshot status` deprecated

The client reported:

Deprecated: Use `etcdutl snapshot status` instead.

Fix

Install/use etcdutl and run:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Error 8: `etcdutl` not installed

The command:

etcdutl

was initially unavailable.

Fix

The system suggested installing etcd-server, which provided the utility in this environment.

Error 9: `sudo cd`

This failed:

sudo cd /var/lib/etcd-restore-from-backup

with:

sudo: cd: command not found

Lesson

cd is a shell built-in.

Use commands such as:

sudo ls /var/lib/etcd-restore-from-backup

or start a privileged shell if you genuinely need one.

Error 10: Pod name not found

This:

k describe po etcd-osboxes

returned:

pods "etcd-osboxes" not found

because the command defaulted to the default namespace.

Fix

k describe po -n=kube-system etcd-osboxes

Error 11: `kubectl logs` without a Pod

k logs

returned:

POD or TYPE/NAME is a required argument for the logs command

Fix

Provide the resource:

k logs <pod-name>

and specify the namespace when needed.

Error 12: `kubectl` Forbidden

After the control-plane changes:

Error from server (Forbidden): pods is forbidden:
User "kubernetes-admin" cannot list resource "pods"

This was an authorization/RBAC problem rather than a shell permission problem.

sudo did not solve it because Kubernetes authorization is separate from Linux sudo.

Error 13: etcd remained Pending

The etcd Pod stayed:

0/1 Pending

even though the container had been created and started according to the events.

The investigation included:

k describe po -n=kube-system etcd-osboxes

and:

sudo systemctl status kubelet --no-pager

The practical recovery step that resolved the state in this lab was:

sudo systemctl restart kubelet

After that:

etcd-osboxes   1/1   Running

Why Static Pods Matter in This Exercise

The etcd Pod was not created with a normal Deployment such as:

kind: Deployment

Instead, it was defined by:

/etc/kubernetes/manifests/etcd.yaml

That makes it a static Pod.

The kubelet watches the manifest directory and manages the Pod locally.

This explains several things that happened during the exercise:

changing etcd.yaml affected etcd;
temporarily moving manifests out of the directory affected the control plane;
restarting kubelet caused it to re-read and reconcile the manifests;
etcd could be controlled without creating a normal Kubernetes Deployment.

For kubeadm clusters, understanding static Pods is essential when troubleshooting the control plane.

A Safer Mental Model for etcd Restore

Do not think of restore as:

"copy backup file over etcd"

A better mental model is:

ETCD SNAPSHOT
     |
     v
restore snapshot
     |
     v
new etcd data directory
     |
     v
configure etcd to use restored directory
     |
     v
kubelet recreates/restarts static etcd Pod
     |
     v
etcd becomes healthy
     |
     v
Kubernetes API becomes available against restored state

This is why the exercise used:

/var/lib/etcd-restore-from-backup

instead of immediately destroying:

/var/lib/etcd

What I Would Do Differently Next Time

The lab worked, but it also exposed several areas where the workflow could be cleaner.

1. Check the exact paths first

Before typing a long command:

ls -l /etc/kubernetes/pki/etcd/
ls -l /opt/etcd-backup.db

This avoids guessing filenames and permissions.

2. Inspect the manifest before changing it

First:

sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.backup

Then edit the manifest.

That gives you a known-good copy if something goes wrong.

3. Prefer the current etcd utilities

The terminal explicitly showed that:

etcdctl snapshot status

is deprecated and recommended:

etcdutl snapshot status

Likewise, the restore output recommended:

etcdutl snapshot restore

4. Avoid unnecessary movement of all manifests

This command:

sudo mv * /tmp

is powerful and potentially disruptive.

Moving all control-plane manifests affects more than etcd.

A safer approach is to modify only the required manifest and keep backups of the original.

5. Verify each stage

After every major step:

k get pods -A

or:

k get pods -n kube-system

and inspect:

k describe pod <pod> -n kube-system

if anything looks abnormal.

Compact Recovery Checklist

For a kubeadm single-control-plane lab similar to this one:

Identify etcd

k get pods -n kube-system | grep etcd

Inspect the manifest

cd /etc/kubernetes/manifests
sudo cat etcd.yaml

Identify:

--data-dir
--listen-client-urls
--cert-file
--key-file
--trusted-ca-file

Create snapshot

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

Validate snapshot

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Restore to a separate directory

sudo etcdutl snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

Configure etcd to use the restored directory

Update:

--data-dir=/var/lib/etcd-restore-from-backup

and the corresponding hostPath.

Restart/reconcile kubelet if required

sudo systemctl restart kubelet

Verify

k get pods -n kube-system

Look for:

etcd-osboxes   1/1   Running

What This Lab Actually Taught Me

The most valuable part of this exercise was not memorizing one command.

It was learning how the pieces connect:

Kubernetes
    |
    +-- kube-apiserver
    |
    +-- kube-controller-manager
    |
    +-- kube-scheduler
    |
    +-- kubelet
            |
            +-- static Pod manifests
                    |
                    +-- etcd
                            |
                            +-- /var/lib/etcd

The backup process interacted directly with etcd:

etcd
  |
  +-- TLS certificates
  |
  +-- etcdctl
  |
  +-- snapshot
  |
  +-- /opt/etcd-backup.db

The restore process then created:

/var/lib/etcd-restore-from-backup

and the static Pod was changed to consume that restored data.

Finally, kubelet reconciled the static Pod and etcd returned to:

1/1 Running

Final Takeaway

An etcd backup is not just another Kubernetes YAML export.

The core workflow is:

1. Verify etcd is healthy.
2. Find the etcd endpoint and TLS certificates.
3. Take an etcd snapshot.
4. Validate the snapshot.
5. Restore the snapshot into a separate data directory.
6. Configure the etcd static Pod to use the restored data.
7. Let kubelet reconcile the static Pod.
8. Verify etcd and the Kubernetes control plane.

The terminal session also demonstrated something that documentation often hides: the commands do not always work on the first attempt.

There were:

spelling mistakes;
incorrect flags;
permission problems;
wrong filenames;
deprecated commands;
missing utilities;
namespace mistakes;
RBAC errors;
attempts to use sudo where it did not apply;
a static etcd Pod that remained Pending;
and finally a kubelet restart that brought etcd back to Running.

Those failures were not separate from the learning experience. They were the learning experience.

The most useful troubleshooting habit from this lab is simple:

When Kubernetes gives you an error, do not immediately change random things. Read the error, identify which layer produced it, inspect the relevant component, make the smallest reasonable correction, and verify the result.

That is the difference between memorizing Kubernetes commands and actually learning how the control plane works.

Terminal command reference from the lab

Cluster health

k get pods -A
k get pods -A -w
k get nodes
k get all
k get all -A

Static Pod manifests

cd /etc/kubernetes/manifests
ls
sudo cat etcd.yaml

Install tooling

sudo apt install etcd-client
sudo apt install etcd-server

Configure etcdctl

export ETCDCTL_API=3

Backup

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

Validate

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Test workload

k create deploy nginx --image=nginx
k get po
k get po -w
k describe po nginx-7f8fbb96d-9f7wm
k get all

Delete workload

k delete deploy nginx

Restore

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

Troubleshoot etcd

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes
k get pod etcd-osboxes -n kube-system -o wide
k logs etcd-osboxes -n kube-system
sudo systemctl status kubelet --no-pager
sudo systemctl restart kubelet

Final verification

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes

Closing note

This was a lab on a single-node kubeadm cluster. In a production environment, etcd backup and restore requires much stricter operational controls, including secure off-host backup storage, tested restore procedures, careful handling of TLS credentials, cluster topology awareness, and a clearly defined disaster-recovery procedure.

A backup is only useful if you can successfully restore it.

Take the backup. Validate it. Practice the restore.

Kubernetes ETCD Backup And Restore — Part 1: Creating and Validating the Backup

Mon, 17 Aug 2026 10:54:12 +0000

This is Part 1 of a two-part series.

In this part, the focus is entirely on the backup side of the disaster-recovery exercise:

understanding what etcd does;
verifying the Kubernetes control plane;
locating the static etcd manifest;
understanding the etcd endpoint and TLS certificates;
installing etcdctl and etcdutl;
dealing with command-line mistakes and permission errors;
creating a valid etcd snapshot;
validating the snapshot;
creating an nginx Deployment;
taking a second snapshot containing that workload;
preparing the cluster for the restore exercise in Part 2.

The actual restoration, static Pod reconfiguration, kubelet troubleshooting, and recovery are covered in Part 2.

A beginner-friendly, hands-on walkthrough of backing up and restoring etcd in a kubeadm-based Kubernetes cluster — including the mistakes, errors, debugging steps, and the terminal commands that actually happened during the lab.

Introduction

If you are learning Kubernetes, you will eventually hear a simple but important statement:

etcd is the database of a Kubernetes cluster.

That description is short, but it has a major consequence: if the etcd data is lost or corrupted, the Kubernetes control plane can lose the information it needs to know what exists in the cluster.

In this lab, I practiced an end-to-end etcd backup and restore workflow on a single-node kubeadm cluster running on a machine named osboxes.

The exercise covered:

identifying the etcd pod and confirming that the control plane was healthy;
locating the static etcd manifest;
inspecting the etcd configuration;
installing etcdctl and etcdutl;
creating an etcd snapshot;
validating the snapshot;
deliberately creating an nginx Deployment so there was something meaningful to restore;
deleting the Deployment;
restoring the etcd snapshot into a separate data directory;
changing the static etcd manifest to use the restored data;
troubleshooting an etcd pod that remained Pending;
restarting kubelet;
confirming that etcd eventually returned to Running.

The important part of this exercise was not simply getting a successful backup command. It was understanding what each error meant and how to reason from the evidence in the terminal.

The terminal session showed the etcd pod running before the backup, with the control plane components also healthy. Later, after changing the etcd static pod configuration, etcd became Pending; the investigation eventually led to restarting kubelet, after which etcd became 1/1 Running. The original terminal output is the source for the commands and results reproduced throughout this article.

1. First: What is etcd?

Before touching commands, it helps to understand what we are backing up.

Kubernetes has many components, but the important relationship for this exercise is roughly:

kubectl
   |
   v
kube-apiserver
   |
   v
  etcd
   |
   +---- Kubernetes cluster state

The API server is the main interface through which Kubernetes clients interact with the cluster. etcd is the persistent key-value store used by Kubernetes to store cluster state.

That state includes information such as:

Kubernetes objects;
Deployments;
Pods and their specifications;
Services;
ConfigMaps;
Secrets;
cluster configuration and metadata.

A useful mental model for a beginner is:

The Kubernetes API is how you ask Kubernetes about the cluster; etcd is where much of that cluster state is persisted.

This is why backing up etcd is fundamentally different from simply exporting Kubernetes YAML.

2. Verify that the cluster is healthy first

Before taking a backup, I first checked the cluster.

Terminal

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-74c68c8864-fjtnv   0/1     ContainerCreating   0          3m10s
kube-system   calico-node-dzlcp                          0/1     Init:2/3             0          3m10s
kube-system   coredns-589f44dc88-l522w                   0/1     ContainerCreating   0          8m56s
kube-system   coredns-589f44dc88-nzsh9                   0/1     ContainerCreating   0          8m56s
kube-system   etcd-osboxes                               1/1     Running             0          9m6s
kube-system   kube-apiserver-osboxes                     1/1     Running             0          9m9s
kube-system   kube-controller-manager-osboxes            1/1     Running             0          9m5s
kube-system   kube-proxy-d8krd                           1/1     Running             0          8m56s
kube-system   kube-scheduler-osboxes                     1/1     Running             0          9m8s

The most important observation here is:

etcd-osboxes  1/1  Running

The API server, controller manager, scheduler and kube-proxy were also running.

Some networking components were still starting, so I watched the pods:

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A -w

Eventually CoreDNS and Calico became ready.

I then checked the node:

┌──(osboxes㉿osboxes)-[~]
└─$ k get nodes
NAME      STATUS   ROLES           AGE   VERSION
osboxes   Ready    control-plane   10m   v1.36.3

At this point the single control-plane node was Ready.

The initial cluster state is important because a backup should be taken from a functioning etcd instance rather than blindly assuming that etcd is healthy.

3. An important distinction: Kubernetes YAML backup vs etcd backup

Before working directly with etcd, I also ran:

┌──(osboxes㉿osboxes)-[~]
└─$ k get all -A -o yaml > backup.yaml

This produced:

┌──(osboxes㉿osboxes)-[~]
└─$ ls
backup.yaml  Documents  kubectl  Pictures  Public  Videos
Desktop      Downloads  Music    Projects  Templates  VM-Share

This is useful, but it is not the same thing as an etcd snapshot.

The command:

k get all -A -o yaml > backup.yaml

exports Kubernetes resources returned by kubectl get all.

An etcd snapshot, on the other hand, is a snapshot of the etcd backend itself.

For an etcd disaster-recovery exercise, the etcd snapshot is the important artifact.

4. Finding the etcd static pod

This cluster was created with kubeadm, so etcd was running as a static pod.

I initially made a typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kkubenetes/manifests
bash: cd: /etc/kkubenetes/manifests: No such file or directory

The problem was simply the spelling:

kkubenetes

instead of:

kubernetes

I then made another typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubenetes/manifests
bash: cd: /etc/kubenetes/manifests: No such file or directory

Again, the correct directory is:

/etc/kubernetes/manifests

Eventually:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubernetes/manifests

and:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml

This directory is extremely important on a kubeadm control-plane node.

The files in /etc/kubernetes/manifests are static pod manifests. The kubelet watches this directory and makes sure the described control-plane containers are running.

That means editing etcd.yaml is effectively changing how the control-plane etcd process is launched.

5. Another wrong assumption: trying to use Docker

I also tried:

┌──(osboxes㉿osboxes)-[~]
└─$ docker exec -it osboxes bash
Error response from daemon: No such container: osboxes

This failed because the Kubernetes node name osboxes is not necessarily a Docker container name.

In this setup, the node was running Kubernetes with containerd, and the etcd pod later showed a container ID beginning with:

containerd://

So the important lesson is:

Do not assume that a Kubernetes node or Pod name is a Docker container name.

If you need to inspect a Kubernetes container, first determine which container runtime the node is using.

6. Inspecting the etcd manifest

The first attempt to read the manifest without elevated privileges also failed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cat etcd.yaml
cat: etcd.yaml: Permission denied

The solution was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo cat etcd.yaml

The manifest revealed several important values.

For example:

- --advertise-client-urls=https://10.0.2.15:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379,https://10.0.2.15:2379

It also showed that the etcd data directory was originally:

/var/lib/etcd

and the TLS certificates were under:

/etc/kubernetes/pki/etcd/

This information is essential because etcdctl needs to connect to the correct endpoint and authenticate with the appropriate certificates.

7. Install etcdctl

The machine did not initially have the etcd client installed, so I installed it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo apt install etcd-client

The package installed successfully:

Installing:
  etcd-client

Summary:
  Upgrading: 0, Installing: 1, Removing: 0, Not Upgrading: 205

I then checked the snapshot command:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ETCDCTL_API=3 etcdctl snapshot

The output showed:

COMMANDS:
    restore  Restores an etcd member snapshot to an etcd directory
    save     Stores an etcd node backend snapshot to a given file
    status   [deprecated] Gets backend snapshot status of a given file

The key command for the backup is:

etcdctl snapshot save

8. ETCDCTL_API=3

I also explicitly exported:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ export ETCDCTL_API=3

For the lab, this made it explicit that etcdctl should use the v3 API.

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl snapshot

again showed the snapshot subcommands.

For a beginner, the important idea is:

ETCDCTL_API=3

selects the etcd API version used by the client.

9. Creating the first etcd snapshot — and making a typo

I attempted the backup with:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379 \
> --cacert=/etc/kubernetes/pki/etcd/ca.crt \
> --cert=/etc/kubernetes/pki/etcd/server.crt \
> --key=/etc/kubernetes/pki/etcd/server.key \
> snapshot save /opt/etcd-backup.db

The error was:

Error: bad flag syntax: ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379

The mistake was the number of hyphens.

I typed:

---endpoint

instead of:

--endpoints

There were actually two problems worth noticing:

There were three hyphens.
The correct flag is --endpoints, plural.

This is a classic CLI problem: the command can look almost correct to a human while being completely invalid to the parser.

10. The next error: permission denied

I corrected the flag and tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

The command reached the certificate files but failed with:

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied

This made sense.

The etcd private key is protected, and the regular user did not have permission to read it.

The solution was to run the command with sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

This time the backup succeeded.

11. The successful etcd backup

The successful command produced:

{"level":"info","ts":"2026-08-13T12:11:59.649233-0400","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/opt/etcd-backup.db.part"}
{"level":"info","ts":"2026-08-13T12:11:59.660384-0400","logger":"client","caller":"v3/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-08-13T12:11:59.660467-0400","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-08-13T12:11:59.831074-0400","logger":"client","caller":"v3/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-08-13T12:11:59.894020-0400","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.9 MB","took":"now"}
{"level":"info","ts":"2026-08-13T12:11:59.894158-0400","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/opt/etcd-backup.db"}
Snapshot saved at /opt/etcd-backup.db

The resulting backup file was:

/opt/etcd-backup.db

The snapshot was approximately:

3.9 MB

This was the first major milestone.

12. Validating the snapshot

I initially tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --write-out=table snapshot status snapshot.db

The client responded:

Deprecated: Use `etcdutl snapshot status` instead.

Error: stat snapshot.db: no such file or directory

There were two lessons here.

Lesson 1: The filename was wrong

The backup was saved as:

/opt/etcd-backup.db

not:

snapshot.db

Lesson 2: The command was deprecated

The output explicitly recommended:

etcdutl snapshot status

So I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db

but etcdutl was not installed.

The system suggested:

sudo apt install etcd-server

I installed the package.

After installation, I still used the wrong filename once:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db
Error: stat snapshot.db: no such file or directory

Then I used the correct path:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status /opt/etcd-backup.db

This produced:

Error: open /opt/etcd-backup.db: permission denied

Again, the issue was file permissions.

So I used sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

The result was:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ a633855e │     5973 │       1298 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

This confirmed that the snapshot was readable and contained etcd data.

13. Why create an nginx Deployment?

A backup is much more meaningful if we have something we can verify after restoring it.

At this point, I created an nginx Deployment.

My first attempt was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx
error: required flag(s) "image" not set

The command was missing the container image.

I then tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy --image=nginx
error: exactly one NAME is required, got 0
See 'kubectl create deployment -h' for help and examples

Now the image was present, but the Deployment name was missing.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx --image=nginx
deployment.apps/nginx created

This is a good beginner lesson:

kubectl create deployment <NAME> --image=<IMAGE>

So:

kubectl create deployment nginx --image=nginx

means:

create a Deployment;
call it nginx;
use the nginx container image.

14. Waiting for the nginx Pod

Immediately after creation:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS              RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   0/1     ContainerCreating   0          4s

The Pod remained in:

ContainerCreating

I watched it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -w

After waiting, I checked again.

The Pod eventually became:

nginx-7f8fbb96d-9f7wm   1/1   Running   0   3m16s

The describe output explained why it initially took time:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ---   ----               -------
  Normal  Scheduled  83s   default-scheduler  Successfully assigned default/nginx-7f8fbb96d-9f7wm to osboxes
  Normal  Pulling    82s   kubelet            spec.containers{nginx}: Pulling image "nginx"

The Pod had been scheduled successfully. The kubelet was simply pulling the image.

This is an important troubleshooting principle:

ContainerCreating does not automatically mean something is broken. Check the Pod events before changing anything.

Eventually:

k get po
NAME                    READY   STATUS    RESTARTS   AGE
nginx-7f8fbb96d-9f7wm   1/1     Running   0          3m16s

15. Take another backup after the nginx Deployment exists

This step is important because the first snapshot was taken before the nginx Deployment was created.

After nginx was running, I created another snapshot:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

The snapshot was again:

3.9 MB

and the command ended with:

Snapshot saved at /opt/etcd-backup.db

I validated it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Result:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

Notice that the revision changed.

The backup now represented a later state of the cluster.

16. Verify the state we want to recover

At this point:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   1/1     Running   0          4m1s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1            4m2s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-7f8fbb96d   1         1         1       4m2s

The important object is:

deployment.apps/nginx

It was healthy.

Now the test was simple:

If I delete nginx, can I use the etcd snapshot to recover the cluster state that existed when the snapshot was taken?

Debugging a Broken Ingress Setup: A Step-by-Step Walkthrough

Fri, 24 Jul 2026 10:05:45 +0000

This post documents a real debugging session that started with a simple symptom — example.com wouldn't load — and ended up uncovering several separate, unrelated problems layered on top of each other. It's written to walk through the actual diagnostic process, not just the final fix, since the process is usually the more useful part to learn from.

Starting point: an Ingress setup (from a previous post) was working, then stopped working after some changes were made mid-session, including installing a different CNI (Calico) on top of the cluster's existing one. This is the story of tracking that down.

The Symptom

❯ curl example.com
curl: (7) Failed to connect to example.com port 80 after 21007 ms: Could not connect to server

A connection timeout like this can mean a lot of different things: DNS pointing to the wrong place, a firewall silently dropping packets, a service that's down, or a machine that simply can't reach the target network at all. The rest of this post is the process of ruling each of those out, one at a time.

Step 1: Confirm What the Hostname Actually Resolves To

Before assuming anything about networking, the first thing to check is whether example.com is even resolving to the IP it's supposed to:

❯ getent hosts example.com
10.96.101.127   example.com

This confirmed the local /etc/hosts entry was working correctly — example.com resolved to 10.96.101.127, matching the Ingress controller's IP from earlier setup. So the problem wasn't DNS/hosts resolution. The IP itself was the next thing to check.

Step 2: Test the IP Directly, Bypassing the Hostname

If the hostname resolves correctly but the connection still fails, the next step is to test the raw IP directly — this removes DNS/hosts from the equation entirely and asks: "is this address reachable at all?"

❯ curl -v http://10.96.101.127
*   Trying 10.96.101.127:80...
* connect to 10.96.101.127 port 80 from 10.0.2.15 port 57576 failed: Connection refused
* Failed to connect to 10.96.101.127 port 80 after 21003 ms: Could not connect to server

Two useful details came out of this:

Connection refused, not a timeout. This is a meaningful difference. A timeout usually means packets are going out and nothing is answering. "Refused" happens quickly and means something did respond — but actively said "no."
The source IP was 10.0.2.15. This is VirtualBox's default address for a guest machine in NAT networking mode, which confirmed this was being tested from inside a VM.

The actual explanation: 10.96.101.127 is a Kubernetes ClusterIP. A ClusterIP isn't a real address that exists anywhere on the network — it's a virtual IP that only works because of routing rules (iptables/IPVS) that Kubernetes' kube-proxy installs inside the specific machine running the cluster. No other machine — not a VM, not a laptop on the same network — can ever reach a ClusterIP directly, because that address doesn't exist anywhere outside that one machine's own routing table.

This meant the original approach (pointing /etc/hosts at the ClusterIP) was never going to work from a separate machine, regardless of any other fix.

Step 3: Find a Real, Reachable Address Instead

Since ClusterIPs don't work across machines, the next step was finding an address that does — specifically, the IP of one of the actual Docker containers backing the kind cluster's nodes, combined with the Ingress controller's NodePort (a real port opened on the node itself, unlike a ClusterIP).

❯ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED      STATUS          PORTS                       NAMES
691fe58f2715   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes   127.0.0.1:38079->6443/tcp   cka-new-control-plane
aae68ca53b7b   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes                               cka-new-worker2
b51bce23b84f   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes                               cka-new-worker
75634b82514e   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes                               cka-new-worker3

None of the worker nodes had any ports published to the host — only the control-plane's Kubernetes API port was mapped. So the node's actual IP address on Docker's internal network was needed instead:

❯ docker inspect cka-new-worker | grep -A 15 '"kind"'
...
"IPAddress": "172.18.0.5",
...

A discrepancy showed up here worth noting: this IP (172.18.0.5) didn't match what kubectl get nodes -o wide had shown earlier in the session for this same node (172.18.0.3). Docker assigns container IPs dynamically, and since these containers showed Up 18 minutes (meaning they had recently restarted), the IP had simply changed. This is a good general reminder that hardcoding container IPs is fragile — they aren't guaranteed to stay the same across restarts.

Testing the NodePort against this IP, however, still failed:

❯ curl -v http://172.18.0.5:31003 -H "Host: example.com"
*   Trying 172.18.0.5:31003...
* connect to 172.18.0.5 port 31003 from 172.18.0.1 port 53748 failed: Connection timed out

So even with a real, reachable-in-theory IP and the correct port, nothing answered. This meant the problem was something deeper than just "wrong IP" — worth investigating the cluster's internal health next.

Step 4: Ruling Out the Obvious Culprits

At this point, several standard checks were run to narrow down where the actual fault was.

Was the Ingress controller pod even healthy?

❯ kubectl get pods -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS      AGE   IP           NODE
ingress-nginx-controller-7d65c586d6-kvn9s   1/1     Running   3 (20m ago)   16h   10.244.1.2   cka-new-worker2

It was Running, but had restarted 3 times recently — worth remembering, but not conclusive on its own.

Was kube-proxy (the component responsible for NodePort/ClusterIP routing) healthy on every node?

❯ kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide
NAME               READY   STATUS    RESTARTS      AGE     NODE
kube-proxy-2mlbd   1/1     Running   4 (25m ago)   2d18h   cka-new-worker
kube-proxy-jb2rt   1/1     Running   4             2d18h   cka-new-worker2
kube-proxy-m2vq9   1/1     Running   4             2d18h   cka-new-control-plane
kube-proxy-rnlwc   1/1     Running   4             2d18h   cka-new-worker3

All Running, though cka-new-worker had also restarted recently.

Was there a conflict between two CNI plugins (since Calico had been installed on top of the cluster's default kindnet networking earlier)?

❯ docker exec cka-new-worker2 ls /etc/cni/net.d/
10-kindnet.conflist

Only one CNI config was present — no conflict there, at least on this node.

Did basic pod-to-pod networking inside the cluster even work at all?

❯ kubectl run curl --rm -it --image=curlimages/curl -- curl http://hello-world
curl: (6) Could not resolve host: hello-world

This was the first real breakthrough. This same command had worked earlier in the setup process — now it couldn't even resolve the Service's name. This shifted the investigation toward DNS specifically.

Step 5: Investigating the DNS Failure

First, check whether restarting kube-proxy (to clear out any stale routing rules) fixes it:

❯ kubectl delete pods -n kube-system -l k8s-app=kube-proxy
pod "kube-proxy-2mlbd" deleted from kube-system namespace
pod "kube-proxy-jb2rt" deleted from kube-system namespace
pod "kube-proxy-m2vq9" deleted from kube-system namespace
pod "kube-proxy-rnlwc" deleted from kube-system namespace

After waiting for the new pods to come back Running:

❯ kubectl run curl --rm -it --image=curlimages/curl -- curl http://hello-world
curl: (6) Could not resolve host: hello-world

No change. This ruled out stale kube-proxy rules as the cause.

Check whether the cluster's internal DNS service (CoreDNS) has healthy endpoints behind it:

❯ kubectl get endpoints -n kube-system kube-dns
NAME       ENDPOINTS                                                   AGE
kube-dns   10.244.0.2:9153,10.244.0.3:9153,10.244.0.2:53 + 3 more...   2d18h

This looked healthy — CoreDNS pods were properly registered.

Check whether a real pod is even configured to use the right DNS server:

❯ kubectl exec -it hello-world-c8b6f8dbd-9t9n6 -- sh
# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

This was also correct — 10.96.0.10 is the standard internal DNS Service IP.

So far: correct DNS configuration, healthy-looking CoreDNS, no stale kube-proxy rules — yet DNS lookups still failed. The next step was testing whether this was actually a DNS-specific issue, or something more fundamental with networking between nodes.

Step 6: Isolating DNS vs. General Networking

Test a plain HTTP connection to a Service, bypassing DNS entirely (using the raw ClusterIP):

# python3 -c "import urllib.request; print(urllib.request.urlopen('http://10.96.74.127').read())"
b'Hello, World!'

This worked. So general Service routing (HTTP, TCP) was fine — the problem was specific to DNS.

Test raw connectivity to the DNS port itself (port 53), both against the Service IP and a CoreDNS pod's IP directly:

# python3 -c "
import socket
try:
    s = socket.create_connection(('10.96.0.10', 53), timeout=3)
    print('connected')
    s.close()
except Exception as e:
    print('failed:', e)
"
failed: timed out

# python3 -c "
import socket
try:
    s = socket.create_connection(('10.244.0.2', 53), timeout=3)
    print('connected')
    s.close()
except Exception as e:
    print('failed:', e)
"
failed: timed out

Both failed — even a direct connection attempt to a CoreDNS pod's own IP, with no DNS resolution or Service routing involved at all, still timed out.

The key detail that explained this: checking which node each pod involved was running on showed that the working test (10.96.74.127, resolving to the hello-world pod) happened to be a pod on the same node the test was run from. The failing tests (both CoreDNS IPs) were both on the control-plane node — a different node entirely. This pointed toward a cross-node networking problem, not a DNS-specific one.

Step 7: Confirming Cross-Node Networking Was Broken

Checked whether the Ingress controller's NodePort Service had a networking policy that might explain uneven behavior across nodes:

❯ kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.spec.externalTrafficPolicy}'
Local

This turned out to be a real, separate finding: externalTrafficPolicy: Local means kube-proxy will only forward NodePort traffic on a node if that node happens to be running a copy of the target pod locally — every other node will actively drop the traffic rather than forward it elsewhere. This fully explained the earlier NodePort timeout, since the Ingress controller pod only ran on cka-new-worker2, and the test had hit a different node.

Confirmed by checking iptables on the failing node:

❯ docker exec cka-new-worker3 iptables-save | grep -i "53\|drop" | head -30
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "ingress-nginx/ingress-nginx-controller:http has no local endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 31003 -j DROP

This line confirms it directly: any NodePort traffic to port 31003 on this node is explicitly dropped, because this node has no local copy of the Ingress controller pod.

Fixing the NodePort routing issue was straightforward — hit the correct node directly:

❯ curl -v http://172.18.0.4:31003 -H "Host: example.com"
* Connected to 172.18.0.4 (172.18.0.4) port 31003
< HTTP/1.1 504 Gateway Time-out

The connection succeeded this time — but nginx itself returned a 504 Gateway Time-out. This meant the request reached the Ingress controller fine, but the Ingress controller couldn't reach its own backend (the hello-world Service). And, notably, the Ingress controller pod and the hello-world pod were running on two different nodes — another cross-node hop, timing out in exactly the same pattern as the DNS failures.

This was the real conclusion: same-node pod-to-pod traffic worked consistently throughout this session. Cross-node pod-to-pod traffic (DNS lookups to CoreDNS, and now the Ingress controller reaching its backend) consistently failed. Along the way, other explanations were ruled out — no competing CNI configs, no blocking NetworkPolicy, correctly configured DNS, correct iptables NAT rules, and Calico's own control-plane pods weren't even running:

❯ kubectl get pods -n calico-system -o wide
No resources found in calico-system namespace.

The most likely explanation: installing Calico mid-way through the cluster's life (on top of the existing kindnet CNI, without removing it) had left the routes between nodes in a broken, inconsistent state — even after Calico itself was no longer running. Untangling that kind of mixed-CNI networking state by hand is generally not worth the time; the standard, reliable fix is to rebuild the cluster from scratch.

Step 8: Rebuilding the Cluster

❯ kind delete cluster --name cka-new

To make future testing easier, a kind config file was created to forward the Ingress controller's expected ports straight from the container to localhost on the host machine — removing the need to look up container IPs at all going forward:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: cka-new
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 31003
        hostPort: 31003
        protocol: TCP
      - containerPort: 30437
        hostPort: 30437
        protocol: TCP
  - role: worker
  - role: worker
  - role: worker

❯ kind create cluster --name cka-new --config kind-config.yaml

This time, Calico was intentionally left out — the default kindnet CNI had been working fine before Calico was introduced mid-session, and nothing about the setup actually required Calico's features.

Reapplying the application and Ingress manifests:

❯ kubectl apply -f deployment.yaml -f service.yaml -f ingress.yaml
❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/aws/deploy.yaml

Step 9: A Slow Pod Isn't Always a Stuck Pod

While waiting for the Ingress controller to come up, it sat in ContainerCreating for several minutes, which raised the question of whether something was actually wrong:

❯ kubectl get pods -n ingress-nginx -w
ingress-nginx-controller-7d65c586d6-fz54r   0/1   ContainerCreating   0   2m59s

Checking the pod's events showed the actual cause:

❯ kubectl describe pod -n ingress-nginx ingress-nginx-controller-7d65c586d6-fz54r
...
Events:
  Warning  FailedMount  4m41s (x8 over 5m45s)  kubelet  MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      3m30s                  kubelet  Pulling image "registry.k8s.io/ingress-nginx/controller:v1.15.1@sha256:..."

This looked alarming at first, but both issues were self-resolving, expected steps of the install process rather than actual problems:

The FailedMount warnings happened because two setup Jobs (ingress-nginx-admission-create and ingress-nginx-admission-patch) needed to finish first — they're responsible for generating the certificate secret the controller pod was waiting to mount. Until those Jobs completed, that secret simply didn't exist yet.
The image pull itself was just slow — checking kubectl get events showed the two setup Jobs' images had each taken over a minute to pull, which suggested a slow registry pull rather than anything broken locally.

Waiting it out confirmed this:

❯ kubectl get pods -n ingress-nginx
ingress-nginx-controller-7d65c586d6-fz54r   1/1   Running   0   7m34s

Lesson here: not every ContainerCreating or warning event means something is broken — checking the actual Events section (via kubectl describe pod) before assuming failure saves a lot of unnecessary troubleshooting.

Step 10: Fixing the Service Configuration Correctly

With the controller running, the externalTrafficPolicy and Service type still needed to be set correctly, to avoid re-hitting the earlier Local-policy and LoadBalancer-pending issues.

The first attempt patched only externalTrafficPolicy:

❯ kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
❯ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)
ingress-nginx-controller   LoadBalancer   10.96.61.252   <pending>     80:32087/TCP,443:32290/TCP

This revealed two remaining issues: the Service was still type: LoadBalancer (stuck <pending>, same problem as before), and the NodePorts had landed on different numbers (32087/32290) than what kind-config.yaml was forwarding (31003/30437).

Fixing both at once, by explicitly setting the type to NodePort and pinning the exact port numbers:

❯ kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","protocol":"TCP","nodePort":31003},{"name":"https","port":443,"targetPort":"https","protocol":"TCP","nodePort":30437}]}}'

❯ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME                       TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)
ingress-nginx-controller   NodePort   10.96.61.252   <none>        80:31003/TCP,443:30437/TCP

This matched the port forwarding configured in kind-config.yaml.

Step 11: Confirming Everything Works

❯ curl -v http://localhost:31003 -H "Host: example.com"
*   Trying [::1]:31003...
* connect to ::1 port 31003 from ::1 port 49216 failed: Connection refused
*   Trying 127.0.0.1:31003...
* Connected to localhost (127.0.0.1) port 31003
< HTTP/1.1 200 OK
Hello, World!

This confirmed everything was working correctly: the port forwarding in kind-config.yaml, the Cluster traffic policy correctly routing regardless of which node the request landed on, and the Ingress controller successfully reaching the hello-world backend (no more 504 Gateway Time-out).

The ::1 (IPv6) connection attempt failing before falling back to IPv4 is expected — kind-config.yaml only forwards the port over IPv4, so curl's dual-stack attempt simply skips it and retries over IPv4 instead. Not an error to fix.

Final step — update /etc/hosts, this time pointing at 127.0.0.1 instead of any container IP, since port forwarding now handles the routing:

127.0.0.1   example.com

❯ curl http://example.com:31003
Hello, World!

Why the Port Still Has to Be Specified

Running curl http://example.com without a port defaults to port 80 — but nothing is mapped to host port 80 in this setup, only 31003 and 30437. Without a port, the connection fails immediately:

curl: (7) Failed to connect to example.com port 80 after ...: Connection refused

This is expected: the Ingress controller is exposed via a NodePort Service, and NodePorts are, by design, high-numbered ports (typically in the 30000–32767 range) — they were never going to answer on plain port 80 without an explicit mapping.

To make curl http://example.com work without a port number, kind-config.yaml could instead map host port 80 directly to the NodePort:

extraPortMappings:
  - containerPort: 31003
    hostPort: 80
    protocol: TCP

This requires a cluster rebuild to take effect, and on Linux, binding to port 80 may require elevated (sudo) privileges since ports below 1024 are privileged. For local testing and practice environments, simply including the NodePort in the URL is generally the more common and simpler approach — production clusters solve the "no port number" problem differently anyway, using a real external load balancer or a bare-metal solution like MetalLB in front of the cluster.

Setting Up Ingress for a Flask App

Fri, 24 Jul 2026 08:53:23 +0000

Today's goal: take a simple Flask app running in Kubernetes and expose it to the outside world using Ingress, instead of relying on NodePort or LoadBalancer services. This post walks through the setup, the errors I hit along the way, and how each one was resolved.

Why Use Ingress Instead of NodePort or LoadBalancer?

Kubernetes gives you a few ways to expose a Service outside the cluster:

NodePort — opens a fixed port (usually in the 30000–32767 range) on every node. Works, but the port numbers are ugly and it doesn't scale well for multiple applications.
LoadBalancer — provisions a cloud load balancer per Service. This has real drawbacks:
- Cost — each LoadBalancer Service typically means a new cloud load balancer, billed separately.
- Cloud lock-in — it depends on your cloud provider's integration (AWS, GCP, Azure, etc.).
- No advanced routing — no path-based or host-based routing, and no built-in SSL termination.

Ingress solves this with two separate pieces:

Ingress Resource — a YAML manifest where you declare routing rules (e.g., "requests for example.com/ should go to the hello-world Service").
Ingress Controller — the component that actually watches the API server for Ingress Resources and configures a real proxy (in this case, NGINX) to implement those rules.

You write the rules; the controller enforces them. Without a running Ingress Controller, an Ingress Resource does nothing on its own.

Step 0: Building and Pushing the Docker Image

Before any of the Kubernetes manifests would work, the Flask app first had to be containerized and pushed to a registry, Kubernetes only pulls and runs images, not raw source code. This step happens before everything else, even though it isn't captured in the terminal history below.

app.py — a minimal Flask app:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=80)

Dockerfile:

# Use the official image as a parent image
FROM python:3.7-slim

# Set the working directory in the container
WORKDIR /app

# Copy the dependencies file to the working directory
COPY requirements.txt .

# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt

# Copy the content of the local src directory to the working directory
COPY . .

# Run app.py when the container launches
CMD ["python", "app.py"]

requirements.txt:

Flask==2.2.5

This image is built and pushed to Docker Hub as oyebobs/ingress-demo:v1 — the same tag referenced in deployment.yaml below. If this step is skipped, kubectl will still accept the Deployment manifest, but the Pod will get stuck in ImagePullBackOff because there's nothing to pull.

Step 1: The Kubernetes Manifests

Three manifests, each with a distinct job.

deployment.yaml — defines the Pod and keeps it running:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  labels:
    app: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: oyebobs/ingress-demo:v1
        ports:
        - containerPort: 80

service.yaml — gives the Pod(s) a stable internal address:

apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

ingress.yaml — defines the routing rule for external traffic:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: "example.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80

The ingressClassName: nginx field matters: in a cluster with more than one Ingress Controller installed, this tells Kubernetes which controller should handle this particular Ingress Resource.

After applying the Deployment and Service, both looked healthy:

❯ k get po
NAME                          READY   STATUS    RESTARTS      AGE
curl-test                     1/1     Running   1 (44s ago)   3m43s
hello-world-c8b6f8dbd-9t9n6   1/1     Running   1             33m

❯ k get svc
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
hello-world   ClusterIP   10.96.74.127   <none>        80/TCP    26m

Step 2: Debugging Internal Connectivity

A curl-test Pod was already running to test connectivity to the hello-world Service. The first attempt failed:

❯ k exec -it curl-test -- curl 10.96.74.127
command terminated with exit code 137

Exit code 137 means the container was OOM-killed (killed for exceeding its memory limit) — this is not a networking error. Running kubectl describe pod curl-test confirmed repeated restarts with Last State: Terminated, Reason: Error.

To check whether the app itself was actually working, I shelled directly into the hello-world Pod:

❯ kubectl exec -it hello-world-c8b6f8dbd-9t9n6 -- curl localhost:80
error: ... exec: "curl": executable file not found in $PATH

The python:3.7-slim base image doesn't include curl — "slim" images intentionally leave out extras like this. As a workaround, I used Python's built-in urllib module instead:

# python3 -c "import urllib.request; print(urllib.request.urlopen('http://localhost:80').read())"
b'Hello, World!'

This confirmed the app itself was working correctly inside its own Pod. The issue was isolated to the curl-test Pod repeatedly getting OOM-killed, not the app or the network.

Fix: rather than continuing to debug a resource-constrained, long-running Pod, I switched to a disposable Pod using kubectl run --rm -it, which creates a temporary Pod, opens a shell, and deletes the Pod automatically on exit:

❯ kubectl run curl-test --image=curlimages/curl -it --rm -- curl -v 10.244.3.2:80
...
< HTTP/1.1 200 OK
< Server: Werkzeug/2.2.3 Python/3.7.17
...
Hello, World!

The same worked against the Service's ClusterIP directly:

❯ kubectl run curl-test --image=curlimages/curl -it --rm -- curl -v 10.96.74.127:80
< HTTP/1.1 200 OK
Hello, World!

Takeaway: internal networking and the app were fine the entire time. The failures were caused by one Pod being killed for memory usage, not by a cluster-wide issue. When only one Pod misbehaves and everything else works, the Pod itself is usually the place to look first.

Step 3: Applying the Ingress — Address Stays Empty

With internal connectivity confirmed, I applied the Ingress resource:

❯ k apply -f ingress.yaml
ingress.networking.k8s.io/hello-world created

❯ k get ingress
NAME          CLASS   HOSTS         ADDRESS   PORTS   AGE
hello-world   nginx   example.com             80      8s

The ADDRESS column stayed empty. This is expected at this stage: the Ingress Resource exists, but no Ingress Controller is running yet to act on it.

Step 4: Installing the NGINX Ingress Controller

❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/aws/deploy.yaml
namespace/ingress-nginx created
...
deployment.apps/ingress-nginx-controller created
...
ingressclass.networking.k8s.io/nginx created

This manifest creates a dedicated namespace, RBAC rules, the controller Deployment, and admission webhooks. The controller Pod came up successfully:

❯ k get po -A | grep nginx
ingress-nginx   ingress-nginx-controller-7d65c586d6-kvn9s   1/1   Running   0   7m24s

Its logs confirmed it detected the Ingress resource and reloaded its configuration:

I0723 16:19:18.803001 ... "Found valid IngressClass" ingress="default/hello-world" ingressclass="nginx"
I0723 16:19:19.839235 ... "Backend successfully reloaded"

The controller was running and had picked up the routing rule. The next question was why the Ingress still had no address.

Step 5: LoadBalancer Stuck on `<pending>`

❯ k get svc -n ingress-nginx
NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)
ingress-nginx-controller    LoadBalancer   10.96.101.127   <pending>     80:31003/TCP,443:30437/TCP

EXTERNAL-IP stayed on <pending> indefinitely. This happens because a type: LoadBalancer Service depends on a Cloud Controller Manager to provision an actual external load balancer and assign it an IP. On a self-managed kind/kubeadm cluster with no cloud provider integration, there's nothing available to fulfill that request, so it never resolves.

Fix: change the Service type to NodePort, which doesn't require any cloud integration:

❯ k edit svc ingress-nginx-controller -n ingress-nginx
service/ingress-nginx-controller edited

❯ k get svc -n ingress-nginx
NAME                        TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)
ingress-nginx-controller    NodePort   10.96.101.127   <none>        80:31003/TCP,443:30437/TCP

After deleting and re-applying ingress.yaml, the ADDRESS field populated:

❯ k get ing -w
NAME          CLASS   HOSTS         ADDRESS         PORTS   AGE
hello-world   nginx   example.com   10.96.101.127   80      44s

Testing directly from the local machine, however, still failed:

❯ curl 10.96.101.127
curl: (7) Failed to connect to 10.96.101.127 port 80 after 21024 ms: Could not connect to server

Step 6: Investigating the CNI (Calico Installation)

While troubleshooting the connection timeout, I also installed Calico as the cluster's CNI (Container Network Interface) plugin, replacing the default kindnet networking that kind clusters ship with. Calico is a common choice in real-world clusters because of its more detailed network policy support and easier debugging tools.

❯ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/tigera-operator.yaml
namespace/tigera-operator created
...
deployment.apps/tigera-operator created

❯ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/custom-resources.yaml
resource mapping not found for name: "default" ... no matches for kind "Installation" in version "operator.tigera.io/v1"
ensure CRDs are installed first

This error occurs when the Calico custom resources are applied before the tigera-operator has finished installing its Custom Resource Definitions (CRDs) — the CRDs need to exist before Kubernetes can accept objects of that kind. Fix: wait for the operator Pod to reach Running, then re-apply:

❯ kubectl get pods -n tigera-operator -w
tigera-operator-75c7c596d9-tsmbz   1/1   Running   0   2m4s

❯ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/custom-resources.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created
...

Calico's components came up correctly:

❯ k get po -A -n calico-system | grep calico
calico-apiserver   calico-apiserver-64cf87458d-qrwnb   1/1   Running   0   3m58s
calico-apiserver   calico-apiserver-64cf87458d-tqw44   1/1   Running   0   3m58s

Even with Calico running, curling the Ingress IP from the local machine still timed out:

❯ curl 10.96.101.127
curl: (7) Failed to connect to 10.96.101.127 port 80 after 21007 ms: Could not connect to server

Important clarification: 10.96.101.127 is a ClusterIP, which is only routable inside the cluster's network — it is not reachable from an external machine like a laptop, regardless of which CNI is installed. Swapping the CNI plugin did not (and could not) resolve this particular timeout, since the address itself is not meant to be reachable externally. The correct way to test was to send traffic from inside the cluster, or through the exposed NodePort.

Step 7: Testing From Inside the Cluster

To test correctly, I ran a temporary Pod inside the cluster and issued requests from there, which reflects how real traffic would actually reach the Service:

❯ kubectl run curl --rm -it --image=curlimages/curl -- sh
~ $ curl http://hello-world
Hello, World!

This confirmed internal DNS-based service discovery was working. Next, hitting the Ingress Service's ClusterIP directly by IP, without specifying a hostname:

~ $ curl http://10.96.101.127
<html>
<head><title>404 Not Found</title></head>
<body><center><h1>404 Not Found</h1></center>
<hr><center>nginx</center></body>
</html>

This 404 is expected behavior, not an error. The Ingress rule only matches requests where the Host header is example.com. Curling the raw IP sends no Host header, so NGINX has no matching rule and returns a 404.

To simulate a request with the correct hostname, I used curl's --resolve flag to manually map example.com to the Ingress IP. The first attempt used the wrong syntax:

~ $ curl example.com --resolve example.com:80 http://10.96.101.127
curl: (49) Could not parse CURLOPT_RESOLVE entry 'example.com:80'

--resolve expects the format host:port:ip, not a separate URL. Corrected:

~ $ curl example.com --resolve example.com:80:10.96.101.127
Hello, World!

~ $ curl http://example.com
Hello, World!

This confirmed host-based routing was working as configured: the same Ingress IP serves different responses depending on the requested hostname — the core capability that distinguishes Ingress from a plain NodePort or LoadBalancer setup.

Step 8: Mapping example.com in /etc/hosts

--resolve is convenient for one-off curl requests, but it only applies to that single command — it doesn't help if you want to open http://example.com in a browser or run multiple tools against the same hostname. For that, the standard approach is to edit /etc/hosts directly, which tells the local machine to resolve example.com to the Ingress IP for every request, not just one curl call.

❯ sudo nano /etc/hosts

Inside the file, a line was added mapping the hostname example.com to the Ingress controller's IP address:

10.96.101.127   example.com

After saving the file, any tool on the machine — curl, a browser, etc. — resolves example.com to that IP without needing --resolve or any other flag:

❯ curl http://example.com
Hello, World!

This simulates what a real DNS record would do in production: pointing a domain name at the Ingress controller's IP. In production, you'd create an actual DNS A record for example.com pointing at the Ingress controller's external IP; /etc/hosts is the local-machine equivalent for testing without needing a real, registered domain.

One thing worth noting: this entry only affects DNS resolution on the local machine — it doesn't change how anyone else resolves the domain. It's also worth removing once testing is done, so it doesn't cause confusion later if the Ingress IP changes.

How a Football Match Helped Me Understand Kubernetes: Desired State vs Actual State

Sat, 18 Jul 2026 10:00:00 +0000

Have you ever wondered what Desired State and Actual State mean in Kubernetes?

I used to hear those terms all the time, but they didn't truly make sense until I watched a football match.

In this video, Ada explains one of Kubernetes' most important concepts using a simple football analogy. Instead of complicated diagrams and technical jargon, you'll see how a football coach, substitutions, and an unexpected injury perfectly explain how Kubernetes constantly works to keep your applications running.

Whether you're preparing for the KCNA, CKA, or CKAD, learning Kubernetes for DevOps, or just getting started with cloud-native technologies, this real-world story will help you understand one of Kubernetes' core ideas.

What you'll learn

What Desired State means in Kubernetes
What Actual State means
How Kubernetes compares both states
What reconciliation is
Why Kubernetes automatically creates new Pods
Why this makes Kubernetes reliable

🎯 Watch next

What a Busy Restaurant Can Teach You About Kubernetes Services

🚀 Want to practice Kubernetes yourself?

Spin up a managed Kubernetes cluster on DigitalOcean

You can follow my work and projects on Twitter, LinkedIn, GitHub, and YouTube, where I share what I am building and learning along the way.

If this video helped you understand Kubernetes better, consider subscribing for more real-life cloud and DevOps stories.

Thank you for reading.

Understanding PersistentVolumes (PV) and PersistentVolumeClaims (PVC)

Thu, 16 Jul 2026 16:23:41 +0000

Today I learned one of the most confusing Kubernetes concepts for beginners: PersistentVolumes (PV) and PersistentVolumeClaims (PVC).

At first, my PVC refused to bind and stayed in the Pending state. After some troubleshooting, I finally got everything working. And honestly? Breaking it first helped me understand it much better.

Why do we even need PVs?

Imagine you're living in a rented apartment.

Pod = You.
Container filesystem = Your temporary room.
PersistentVolume (PV) = A storage unit that exists independently of you.
PersistentVolumeClaim (PVC) = The rental agreement saying, "I'd like to use a storage unit."

If your Pod dies, everything inside the container normally disappears.

A PV lets your data survive even when the Pod is deleted.

The Relationship

Pod
 │
 ▼
PersistentVolumeClaim (PVC)
 │
 ▼
PersistentVolume (PV)
 │
 ▼
Actual storage (hostPath, NFS, cloud disk, etc.)

The Pod never talks directly to the PV.

Instead it says:

"Hey PVC, can I borrow some storage?"

The PVC then finds a suitable PV.

Step 1 — Create a PersistentVolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-demo
  labels:
    type: local
spec:
  storageClassName: ""
  capacity:
    storage: 512Mi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/data/config"

This creates 512Mi of storage backed by a directory on the node.

Step 2 — Create a PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi

Notice that the claim only requests 256Mi.

Kubernetes looks for a PV that satisfies the request and binds them together.

My First Roadblock

After creating the PVC:

kubectl apply -f pvc.yaml
kubectl get pvc

I saw:

STATUS
Pending

Why Was My PVC Stuck in `Pending`?

This was the part that confused me the most.

I had already created my PersistentVolume (PV), so I expected my PersistentVolumeClaim (PVC) to bind to it immediately. Instead, every time I checked, the PVC was still in the Pending state.

kubectl get pvc

Output:

NAME       STATUS
pvc-demo   Pending

To investigate, I ran:

kubectl describe pvc pvc-demo

The Events section showed:

FailedBinding
no persistent volumes available for this claim and no storage class is set

At first, I thought Kubernetes couldn't see my PersistentVolume at all. That wasn't actually the problem.

The issue was that my PVC and PV didn't agree on the access mode.

My PVC requested:

accessModes:
  - ReadWriteOnce

while my PV was configured with:

accessModes:
  - ReadWriteMany

Because of this difference, Kubernetes couldn't match the claim to the volume, so the PVC stayed in the Pending state.

After deleting the old resources, correcting the configuration, and recreating both the PV and PVC, Kubernetes successfully matched them and their status changed to Bound.

The biggest lesson I learned is that whenever a PVC is stuck in Pending, the first command to run should be:

kubectl describe pvc pvc-demo

The Events section usually contains the clue you need to figure out why the claim couldn't be bound.

Step 3 — Mount the PVC inside a Pod

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: pv-container
      image: nginx:latest
      volumeMounts:
        - name: pv-storage
          mountPath: /var/app/config
      ports:
        - containerPort: 80
          name: http-server
  volumes:
    - name: pv-storage
      persistentVolumeClaim:
        claimName: pvc-demo

Think of this as plugging an external hard drive into your laptop.

The Pod can now read and write files through the PVC.

Step 4 — Test It

kubectl exec -it app -- bash

Inside the container:

cd /var/app/config

echo "This is a test" > test.html

cat test.html

Output:

This is a test

Success!

I accidentally tried:

curl test.html

which failed because curl fetches URLs, not local files.

Lesson learned.

The Big Idea

Many beginners confuse PVs and PVCs.

Remember this:

PersistentVolume (PV) = The actual storage.
PersistentVolumeClaim (PVC) = The request for storage.
Pod = The application that uses the claim.

A simple analogy:

🍕 PV = The pizza.
🧾 PVC = The order receipt.
😋 Pod = The hungry person.

You don't grab the pizza straight from the oven—you present your receipt first!

Commands I Used

kubectl apply -f pv.yaml
kubectl get pv

kubectl apply -f pvc.yaml
kubectl get pvc
kubectl describe pvc pvc-demo

kubectl apply -f pod.yaml
kubectl get pods

kubectl exec -it app -- bash

What I Learned

Persistent storage survives Pod restarts.
Pods use PVCs, not PVs, directly.
A PVC must successfully bind before a Pod can use it.
kubectl describe is one of the best debugging tools when something is stuck.

If you're just getting started, think of AWS, GCP, and Azure as cloud platforms

Sat, 11 Jul 2026 11:52:48 +0000

What is Cloud Computing?

Traditionally, if you wanted to run an application, you had to buy:

Physical servers
Storage devices
Networking equipment
Cooling systems
Backup power
Security infrastructure

Cloud providers own this infrastructure and let you use it on demand.

The Three Major Cloud Providers

                    **1. Amazon Web Services (AWS)

Released: 2006

Company: Amazon Web Services

AWS is the world's largest cloud platform.

                   ****Common Services

EC2 (Virtual Servers)
S3 (Object Storage)
RDS (Managed Databases)
Lambda (Serverless Functions)
IAM (Identity & Access Management)
CloudWatch (Monitoring)
SageMaker (Machine Learning)
Bedrock (Generative AI)
```
                   **Used by
```
Netflix
Airbnb
NASA
Samsung
BMW
```
                 **Strengths
```
Largest service ecosystem
Very mature platform
Excellent scalability

Strong AI and ML offerings

         **2. Google Cloud Platform (GCP)**

Released: 2008
Company: Google Cloud
Google Cloud is known for AI, machine learning, big data, and Kubernetes.

                 **Common Services

Compute Engine
Cloud Storage
Cloud SQL
BigQuery
Cloud Run
Vertex AI
Kubernetes Engine (GKE)
```
                **Used by
```
Spotify
PayPal
Snapchat
Twitter/X
```
              **Strengths
```
Excellent AI ecosystem
Big data analytics
High-performance networking
```
        **3. Microsoft Azure
```

Released: 2010
Company: Microsoft Azure
Azure is widely adopted by enterprises and organizations already using Microsoft technologies.

             **Common Services

Virtual Machines
Blob Storage
Azure SQL Database
Azure Kubernetes Service (AKS)
Azure OpenAI Service
Azure AI Search
```
             **Used by
```
Walmart
Adobe
LinkedIn
Many government organizations
Large enterprise businesses
```
            **Strengths
```
Enterprise integration
Strong business adoption

Optimizing CPU Bounds: Maximizing Python Throughput in 2026

Fri, 10 Jul 2026 15:37:48 +0000

When application scalability depends entirely on processing efficiency, software teams must look beyond standard code structures. We analyze how modern Python minimizes time-to-market and processing latency under intensive production workloads.

The High-Throughput Implementation Plan:
The blueprint relies on keeping infrastructure slim while taking total advantage of modern multi-core processors.

Configure container instances with free-threaded execution modules to unlock absolute physical CPU capabilities.
Integrate lightning-fast system package managers to reduce continuous deployment image build times significantly.
Structure background workers to leverage shared memory pools to entirely prevent cross-process messaging overhead.

The Failure Points of Scale:
Without careful optimization, intense thread utilization and data volumes can degrade platform stability.
Race conditions multiply across unshielded variables when utilizing true multi-threaded thread parallelism.
Heavy initial compilation times degrade user experience if code paths aren't warmed up effectively before launch. Stale information structures plague execution pipelines if caching layers lack automated invalidation flags.

The Architectural Remedies We guarantee stability by decoupling operational paths and establishing clear performance guardrails.
Inject atomic variable locks around critical memory vectors to ensure complete safety across concurrent threads.Utilize adaptive specialized bytecode compilation to optimize active paths dynamically based on data types.
Offload heavy mathematical workloads to specialized underlying arrays written in highly optimized native languages.

Visit our official site: www.nextbigcreative.com

Kubernetes Network Policies Explained (and How I Actually Made Them Work)

Thu, 09 Jul 2026 16:50:22 +0000

If you've ever wondered, "How do I stop my pods from randomly talking to everything in my cluster?" – this is the post for you.

Today, I dove into Kubernetes Network Policies – the way you control who can talk to whom inside your cluster. I started with weavenet, then switched to Calico, and I'm excited to walk you through how it all came together.

What Even Is a Network Policy?

A NetworkPolicy in Kubernetes is like a firewall rule for your pods. It lets you:

Control inbound traffic (who can talk to a pod)
Control outbound traffic (where a pod can talk to)

For example:

You can create a deny-all policy that blocks all incoming traffic to a pod.
Or an allow-only policy that says: "Only the backend pods can talk to the mysql pod on port 3306."

Without network policies, by default, everything can talk to everything in your cluster. That's fine for demos, terrible for real security.

Why I Needed a Different CNI

Kubernetes itself doesn't implement network policies. It just defines them. The actual work is done by your CNI plugin (Container Network Interface).

I started with weavenet, which supports basic network policies. But as I dug deeper, I realized I wanted something more powerful and production-friendly. That's when I switched to Calico, which is:

Fast
Scalable
Great for complex network policies
Works well with kind clusters (as long as you configure it correctly)

If you're just experimenting, weavenet is fine. If you want to level up your cluster networking, Calico is a great next step.

My Test Cluster: A Kind Setup with Calico

To test network policies, I created a simple kind cluster with:

1 control-plane node
2 worker nodes
Disabled default CNI (disableDefaultCNI: true)
Custom pod subnet: 192.168.0.0/16

kind Cluster YAML

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30001
        hostPort: 30001
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true
  podSubnet: "192.168.0.0/16"

Creating the Cluster

With the YAML saved as kind-config.yaml, I created the cluster like this:

kind create cluster --name day26 --config kind-config.yaml

Then I checked the nodes:

kubectl get nodes

And here's the part that was a bit confusing at first:

Until I installed a CNI (like Calico), the worker nodes stayed in a NotReady state.

That's because Kubernetes needs a CNI to:

Assign IPs to pods
Set up networking between nodes
Enable traffic to flow across the cluster

Without a CNI, the control-plane node can be Ready, but the worker nodes will remain NotReady until networking is configured.

Only after installing Calico did all nodes show as Ready:

kubectl get nodes
# NAME           STATUS   ROLES           AGE   VERSION
# day26-control-plane   Ready    control-plane   ...
# day26-worker          Ready    worker          ...
# day26-worker2         Ready    worker          ...

Once Calico was running, my cluster was ready for real network policy experiments.

The App I Built: Frontend → Backend → MySQL

To make network policies meaningful, I created a tiny 3-tier app:

frontend pod + service
backend pod + service
mysql pod + service (db service)

Pods & Services

apiVersion: v1
kind: Pod
metadata:
  name: frontend
  labels:
    role: frontend
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - name: http
          containerPort: 80
          protocol: TCP
***
apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    role: frontend
spec:
  selector:
    role: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
***
apiVersion: v1
kind: Pod
metadata:
  name: backend
  labels:
    role: backend
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - name: http
          containerPort: 80
          protocol: TCP
***
apiVersion: v1
kind: Service
metadata:
  name: backend
  labels:
    role: backend
spec:
  selector:
    role: backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
***
apiVersion: v1
kind: Service
metadata:
  name: db
  labels:
    name: mysql
spec:
  selector:
    name: mysql
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
***
apiVersion: v1
kind: Pod
metadata:
  name: mysql
  labels:
    name: mysql
spec:
  containers:
    - name: mysql
      image: mysql:latest
      env:
        - name: "MYSQL_USER"
          value: "mysql"
        - name: "MYSQL_PASSWORD"
          value: "mysql"
        - name: "MYSQL_DATABASE"
          value: "testdb"
        - name: "MYSQL_ROOT_PASSWORD"
          value: "verysecure"
      ports:
        - name: http
          containerPort: 3306
          protocol: TCP

At this point, everything could talk to everything:

frontend could reach backend
backend could reach mysql
frontend could also reach mysql directly (which we probably don't want)

Now it was time to lock it down.

My First Real Network Policy: Only Backend → MySQL

The goal:

mysql should only accept traffic from pods with label role: backend
On port 3306
No one else (including frontend) should touch the database

NetworkPolicy Manifest

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-test
spec:
  podSelector:
    matchLabels:
      name: mysql
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              role: backend
      ports:
        - port: 3306

Key parts:

podSelector: targets the mysql pod (name: mysql)
policyTypes: [Ingress]: we're controlling inbound traffic
ingress.from.podSelector: only pods with role: backend are allowed
ports: only port 3306

After applying this:

backend → mysql works
frontend → mysql blocked
mysql → anywhere (outbound) still allowed (we didn't restrict Egress)

How I Tested It

I used a simple approach:

Run a temporary pod with curl or mysql client.
Try to reach services from different pods.
Observe which connections succeed or fail.

Example:

# From frontend pod – should fail
kubectl exec -it frontend -- curl -s http://db:3306

# From backend pod – should succeed
kubectl exec -it backend -- curl -s http://db:3306

Seeing the first one fail and the second one succeed was the "aha!" moment: the policy is actually working.

Wrapping Up

Network Policies are one of those things that feel abstract until you actually apply them and see connections blocked in real time. Once that happens, you suddenly understand:

Why security matters
Why "everything can talk to everything" is not a strategy
How Kubernetes gives you granular control over your cluster's traffic

Learning Cloud My Own Way

Wed, 08 Jul 2026 10:00:00 +0000

Disclosure: Some of the links in this article are affiliate links. This means I may earn a small commission if you click on them. I only recommend tools that I have used.

Learning cloud and DevOps takes patience, discipline, and consistency. There are days when everything clicks, and there are days when nothing seems to make sense. That is completely normal.

One thing I have discovered is that everyone learns differently.

Some people love following a detailed roadmap from beginning to end. Others prefer structured courses with a fixed schedule. For me, learning is more spontaneous. I follow my curiosity. When I find a topic interesting, I dive deep into it. No amount of advice or motivation can make me learn something if I genuinely have no interest in it.

Recently, I have been learning Kubernetes, and it has been one of the most interesting technologies I have studied so far. Instead of only reading documentation or watching tutorials, I decided to explain what I was learning by creating videos.

Here are two videos I created while learning Kubernetes:

Tools That Have Helped Me Learn

Along the way, I have used a few tools that have made learning easier.

DigitalOcean has been a great platform for experimenting with cloud services. If you are a startup or someone learning cloud on a budget, it is worth checking out because getting basic services running is straightforward.

I have also used Gamma to create presentations and portfolio content, and Lovable to build portfolio projects more quickly.

If you would like to explore these tools, here are the links:

Do You Really Need AWS to Learn Cloud?

One question I see often is whether you need AWS before you can start learning cloud.

My answer might surprise you.

Other cloud providers are friendly and can help you build practical skills before moving into AWS, like DigitalOcean.

There is no single right way to learn cloud computing.

Find a learning style that works for you.

If you enjoy reflections like this, you can subscribe to the newsletter.

You can follow my work and projects on Twitter, LinkedIn, GitHub, and YouTube, where I share what I am building and learning along the way.

Thank you for reading.

Understanding Kubernetes Service Accounts (The Identity Every Pod Uses)

Tue, 07 Jul 2026 15:58:52 +0000

Most of us learn about Kubernetes users before we learn about Service Accounts. That's actually where the confusion starts Service Accounts end up looking like "just another kind of user," and you file them away without really getting what makes them different.

They're not just another user. They solve a completely different problem.

Today I learned that human users authenticate from outside the cluster — you, sitting at your laptop, running kubectl with your certificate or your login. But Pods run inside the cluster, and they often need to talk to the Kubernetes API too — to list other pods, read a ConfigMap, watch for changes, whatever. A Pod can't exactly type in a username and password. It needs its own identity.

That identity is called a Service Account.

User vs. Service Account

Before touching a single command, it's worth seeing the two side by side:

User	Service Account
Used by humans	Used by Pods
External to Kubernetes	Managed by Kubernetes
Usually authenticated using certificates	Authenticated using tokens
You create it outside the cluster	Kubernetes creates and manages it

Keep that table in your head — everything below is really just this table playing out in the terminal.

Step 1 — Checking Existing Service Accounts

❯ k get sa
NAME      AGE
default   38d

Notice I didn't create this. It was already there, 38 days old — as old as the namespace itself.

That's the first thing to internalize: every namespace automatically gets a Service Account called default. You don't ask for it. If you create a Pod and never say which Service Account it should use, Kubernetes quietly attaches this one for you. Keep that in mind — it comes back to bite (in a good way) later in this post.

Step 2 — Creating My Own Service Account

❯ k create sa build-sa
serviceaccount/build-sa created

❯ k describe sa build-sa
Name:                build-sa
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Events:              <none>

Simple enough — I now have a second identity, build-sa, sitting in the default namespace, ready to be attached to Pods later.

But here's the question that trips a lot of beginners up: why doesn't it have a token?

If you learned Kubernetes a few years ago, or you're reading an older tutorial, you'll expect describe sa to show a Tokens: field pointing at an auto-generated Secret. Not anymore. Newer versions of Kubernetes stopped automatically minting long-lived Secret-based tokens for every Service Account, mainly for security reasons — a token that never expires and just sits around as a Secret is a standing risk if it leaks. Instead, Kubernetes now prefers short-lived, automatically-rotated tokens issued on demand.

Since I specifically wanted a durable token I could inspect and use for --as testing, I had to create it myself.

Step 3 — Creating a ServiceAccount Token Secret

I wrote a secret.yaml that, at minimum, needs this annotation:

metadata:
  annotations:
    kubernetes.io/service-account.name: build-sa

This one line is doing all the work. It tells the Kubernetes controller: "This Secret isn't just a random Opaque blob — it's a token for the build-sa Service Account." Because of that annotation, a built-in controller notices the Secret, generates a signed token for build-sa, and populates the Secret with it automatically.

❯ k apply -f secret.yaml
secret/build-robot-secret created

❯ k get secret
NAME                 TYPE                                  DATA   AGE
backend-user         Opaque                                1      5d3h
build-robot-secret   kubernetes.io/service-account-token   3      14s

See the TYPE column — kubernetes.io/service-account-token, not Opaque. That type is exactly what tells Kubernetes "treat this Secret specially and fill it with SA credentials."

Inspecting the Secret

❯ k describe secret build-robot-secret
Name:         build-robot-secret
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: build-sa
              kubernetes.io/service-account.uid: d065d358-58fa-44a6-9656-8263e0fa05eb

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1107 bytes
namespace:  7 bytes
token:      eyJhbGciOiJSUzI1NiIs... (truncated)

Three fields, each with a job:

ca.crt — the cluster's certificate authority. Whoever holds this Secret can verify they're really talking to the real API server, not an impersonator.
namespace — just the namespace this identity belongs to (default), so any client using this Secret knows where it "lives."
token — the important one. This is a JWT (JSON Web Token), signed by the cluster. This is what a Pod, script, or client hands to the API server to say "I am build-sa, and here's proof." I've truncated the token above — treat these things like passwords, not like log output, even in a lab cluster.

If you decode a Service Account JWT (there's nothing secret about the structure, just don't paste your actual token online), you'll find claims like the issuer, the namespace, and a sub field that looks like system:serviceaccount:default:build-sa. That string is the Service Account's real, canonical identity inside Kubernetes — remember it, it matters in a minute.

Testing Permissions

Now the interesting part. I wanted to see what build-sa could actually do, without creating a Pod yet:

❯ k get pods --as build-sa
Error from server (Forbidden): pods is forbidden: User "build-sa" cannot list resource "pods" in API group "" in the namespace "default"

Two things to unpack here.

First — what does --as actually do? It's easy to assume you're "logging in" as build-sa. You're not. --as is impersonation. You're still using your own admin credentials to talk to the API server, but you're telling it: "Evaluate this request as if it came from build-sa instead of me." The API server checks whether your account is allowed to impersonate build-sa (admin usually is), and if so, it re-runs the authorization check as that identity. It's a testing and debugging superpower — you get to try on someone else's permissions without actually needing their credentials.

Second — why Forbidden? This is the single most important lesson in this whole exercise: the Service Account existed. The token existed. Authentication succeeded. Authorization failed.

Those are two separate gates in Kubernetes:

Authentication — "Who are you?" build-sa proved this fine, via impersonation.
Authorization — "What are you allowed to do?" Nothing had granted build-sa any permissions at all, so RBAC said no.

A lot of beginners collapse these into one concept. They're not the same thing, and separating them in your head will save you hours of confused debugging later.

Creating the Role

❯ k create role build-role \
--verb=list,get,watch \
--resource=pod
role.rbac.authorization.k8s.io/build-role created

Breaking down the flags:

role build-role — I'm creating a Role, which is a namespaced set of permissions (as opposed to a ClusterRole, which applies cluster-wide).
--verb=list,get,watch — the specific actions this Role allows. Not create, not delete — just read-style operations.
--resource=pod — the Kubernetes resource type these verbs apply to.

On its own, a Role does nothing. It's a permission template sitting unused until something binds an identity to it.

Creating the RoleBinding

❯ k create rolebinding rb \
--role=build-role \
--user=build-sa
rolebinding.rbac.authorization.k8s.io/rb created

This is the glue: it binds the build-role permissions to the identity build-sa.

One honest caveat worth flagging, since I want this post to be technically accurate and not just "it worked so it's fine": I used --user=build-sa here, and it worked in this exercise because I was testing purely through impersonation with --as build-sa, and --as treats whatever string you give it as a plain username.

But that's not actually how a real Service Account identifies itself to the cluster. Its canonical identity is system:serviceaccount:<namespace>:<name> — in my case, system:serviceaccount:default:build-sa. In production, the standard, correct way to bind a RoleBinding to a Service Account is:

--serviceaccount=default:build-sa

If I'd actually mounted this token inside a real client and tried to use it (rather than testing through admin impersonation), a binding made with --user=build-sa would not match that Service Account's real identity, and the request would still be Forbidden. So: my command worked for this specific impersonation test, but if you're setting this up for a real workload, use --serviceaccount=<namespace>:<name> instead. Good to know the difference exists before it costs you a debugging session.

Testing Again

❯ k get pods --as build-sa
No resources found in default namespace.

No error this time — just an empty result, because there genuinely were no pods yet. Authentication and authorization both passed. That's the fix landing.

Creating a Deployment

❯ k create deploy test --image=nginx
deployment.apps/test created

Worth pointing out deliberately: I created this Deployment as myself (admin), not as build-sa. build-sa only has list, get, and watch on pods — nowhere near enough permission to create a Deployment. If I'd tried k create deploy test --image=nginx --as build-sa, it would have been Forbidden too.

Watching the Pod

❯ k get po -w --as build-sa
NAME                    READY   STATUS    RESTARTS   AGE
test-56848fd9dc-fhdjv   1/1     Running   0          89s

The -w flag means watch — instead of printing the pod state once and exiting, kubectl keeps the connection open and streams updates as the pod's status changes. Combined with --as build-sa, this also confirms build-sa's watch verb from the Role is working, not just list and get.

Going Inside the Pod

This is where it got genuinely interesting.

❯ k exec -it test-56848fd9dc-fhdjv bash
error: exec [POD] [COMMAND] is not supported anymore. Use exec [POD] -- [COMMAND] instead

I hit this error a few times before it clicked. Older kubectl versions let you run kubectl exec pod bash and it would figure out where your flags ended and the container's command began. Modern kubectl refuses to guess — it wants an explicit -- to separate "arguments meant for kubectl" from "the command to run inside the container." So:

❯ k exec -it test-56848fd9dc-fhdjv -- bash
root@test-56848fd9dc-fhdjv:/#

That extra -- is the whole fix. Everything after it belongs to the container, not to kubectl.

The Most Interesting Discovery

Once inside, I went looking for exactly the kind of thing this Pod would need to talk to the API server — and there it was:

root@test-56848fd9dc-fhdjv:/var/run/secrets/kubernetes.io/serviceaccount# ls -lrt
total 0
lrwxrwxrwx 1 root root 12 Jul  7 14:38 token -> ..data/token
lrwxrwxrwx 1 root root 16 Jul  7 14:38 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 13 Jul  7 14:38 ca.crt -> ..data/ca.crt

Same three files as the Secret I inspected earlier — token, namespace, ca.crt — automatically mounted at /var/run/secrets/kubernetes.io/serviceaccount, with no configuration from me at all. Kubernetes does this for every Pod by default, via a projected volume, so that any process running inside the container can immediately authenticate to the API server if it needs to (this is how things like controllers and operators, running as Pods themselves, talk back to Kubernetes).

Now the obvious beginner question, and the one I asked myself immediately: "Wait — I created build-sa. Why does the token in here say default?"

Sure enough, decoding what's inside shows a sub claim of system:serviceaccount:default:default — the default Service Account, not build-sa.

The answer is almost anticlimactic once you see it: my Deployment spec never included serviceAccountName: build-sa. I never told Kubernetes which Service Account this Pod should use — so it fell back to the one thing every namespace already has, the one from Step 1. The default Service Account.

This is, genuinely, the most valuable lesson from the whole exercise: creating a Service Account doesn't automatically attach it to anything. You have to explicitly wire it into a Pod (or Deployment, or Job, etc.) with serviceAccountName — or Kubernetes will silently keep using default on your behalf, and you won't get an error telling you. You'll just quietly have the wrong permissions, or, worse in production, the default Service Account might have more access than you meant to grant that workload.

Key Takeaways

Service Accounts are identities for Pods — not humans.
Every namespace automatically gets a default Service Account.
Service Accounts authenticate using tokens (JWTs), not certificates.
Newer Kubernetes versions don't auto-create long-lived token Secrets anymore — you create them explicitly when you need one.
RBAC still governs what a Service Account is allowed to do, exactly like it does for users.
--as impersonates another identity for testing — it doesn't log you in as them.
Authentication and authorization are two separate checks: proving who you are is not the same as being allowed to do something.
Bind RoleBindings to Service Accounts with --serviceaccount=<namespace>:<name> in real setups, not --user=<name>.
Pods automatically get their Service Account's credentials mounted at /var/run/secrets/kubernetes.io/serviceaccount — no extra config needed.
If you don't set serviceAccountName on a Pod, Kubernetes silently uses default. Always check.

Understanding Kubernetes ClusterRoles and ClusterRoleBindings

Tue, 07 Jul 2026 15:03:06 +0000

Yesterday I learned about Roles and RoleBindings. Today I explored ClusterRoles and ClusterRoleBindings for cluster-scoped resources like Nodes.

Why a Role Wasn't Enough

kubectl auth can-i get nodes

Output:

Warning: resource 'nodes' is not namespace scoped
no

Nodes are cluster-scoped resources, so a namespace Role cannot grant access.

Creating a ClusterRole

kubectl create clusterrole node-reader --verb=get,list,watch --resource=nodes

Verify:

kubectl get clusterrole | grep node-reader

Creating a ClusterRoleBinding

I first made a mistake:

kubectl create clusterrolebinding reader-binding --clusterrole=cluster-reader --user=adeoye

Checking permissions:

kubectl auth can-i get nodes --as adeoye

returned:

no - RBAC: clusterrole.rbac.authorization.k8s.io "cluster-reader" not found

The binding referenced a ClusterRole that didn't exist.

I fixed it by deleting and recreating the binding:

kubectl delete clusterrolebinding reader-binding
kubectl create clusterrolebinding reader-binding --clusterrole=node-reader --user=adeoye

Now:

kubectl auth can-i get nodes --as adeoye

returned:

yes

Testing as adeoye

kubectl config use-context adeoye
kubectl get nodes

Listing nodes worked.

Describing a node:

kubectl describe node cka-cluster3-worker

showed most information, but RBAC blocked access to leases and pods because the ClusterRole only granted access to nodes.

Trying to delete a node:

kubectl delete node cka-cluster3-worker

returned Forbidden because the ClusterRole only included get, list, and watch.

Key Takeaways

Roles are namespace-scoped.
ClusterRoles are cluster-scoped.
ClusterRoleBindings grant cluster-wide permissions.
RBAC permissions are resource-specific.
Kubernetes follows the principle of least privilege.

Understanding Kubernetes RBAC: Roles, RoleBindings, and and Client Certificates

Mon, 06 Jul 2026 14:15:05 +0000

One thing I discovered today is that before Kubernetes can decide what you're allowed to do, it first needs to know who you are.

That means there are two separate steps:

Authentication – Proving your identity using a client certificate.
Authorization – Deciding what you're allowed to do using RBAC.

In this lab, I created my own Kubernetes user named adeoye, authenticated the user with a certificate, and then used RBAC to control what that user could access.

Step 1 – Generate a Private Key

openssl genrsa -out adeoye.key 2048

This generates a 2048-bit RSA private key.

Think of this as your digital identity. It should never be shared.

Step 2 – Generate a Certificate Signing Request (CSR)

openssl req -new -key adeoye.key -out adeoye.csr -subj "/CN=adeoye"

A Certificate Signing Request asks Kubernetes to issue a client certificate.

The important part is:

/CN=adeoye

The Common Name (CN) becomes the Kubernetes username.

Step 3 – Base64 Encode the CSR

cat adeoye.csr | base64 | tr -d '\n'

Kubernetes expects the CSR to be embedded inside a YAML manifest as Base64-encoded text. The tr -d '\n' removes line breaks so it becomes a single continuous string.

Step 4 – Submit the CSR

After inserting the encoded CSR into csr.yaml, I submitted it.

kubectl apply -f csr.yaml

Initially, the request was in the Pending state so I had to approve the request.

Step 5 – Approve the Certificate

kubectl certificate approve adeoye

Approving the request tells Kubernetes to trust the user and issue a signed client certificate.

Step 6 – Extract the Signed Certificate

kubectl get csr adeoye -o jsonpath='{.status.certificate}' | base64 -d > adeoye.crt

Now I had:

adeoye.key
adeoye.crt

The key proves my identity, while the certificate proves Kubernetes trusts that identity.

Step 7 – Configure kubectl

kubectl config set-credentials adeoye \
--client-key=adeoye.key \
--client-certificate=adeoye.crt

Then I created a context.

kubectl config set-context adeoye \
--cluster=kind-cka-cluster3 \
--user=adeoye

A mistake I made: I initially typed lind-cka-cluster3 instead of kind-cka-cluster3, which caused API errors until I corrected the cluster name.

Finally, I switched to my new user.

kubectl config use-context adeoye

At this point I was authenticated, but I still didn't have permissions to perform actions.

RBAC Time!

Authentication tells Kubernetes who I am.

RBAC tells Kubernetes what I'm allowed to do.

Creating the Role

role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader

rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs:
    - get
    - watch
    - list

This Role only allows reading Pods.

Apply it:

kubectl apply -f role.yaml

Creating the RoleBinding

binding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default

subjects:
- kind: User
  name: adeoye
  apiGroup: rbac.authorization.k8s.io

roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

This binds the adeoye user to the pod-reader Role.

kubectl apply -f binding.yaml

Testing the Permissions

Verify permission:

kubectl auth can-i get pods

Output:

yes

List Pods:

kubectl get pods

Output:

No resources found in default namespace.

That means the command succeeded, but there were simply no Pods.

Next, I tried listing Deployments.

kubectl get deploy

Output:

Error from server (Forbidden): deployments.apps is forbidden

Why Could I Read Pods but Not Deployments?

The answer is in the Role.

It only grants access to:

resources:
- pods

It does not include Deployments.

Deployments belong to the apps API group, while Pods belong to the core API group.

Since my RoleBinding connects adeoye to the pod-reader Role, Kubernetes only allows the permissions defined there.

No Deployment permissions were granted, so Kubernetes correctly denied access.

This is the Principle of Least Privilege in action.

What I Learned

Kubernetes first authenticates users using client certificates.
The certificate's Common Name becomes the Kubernetes username.
A Role defines permissions.
A RoleBinding assigns those permissions to users.
Authentication and authorization are separate processes.
RBAC only grants explicitly defined permissions.
Pods and Deployments are different resources, so access to one doesn't imply access to the other.

Final Thoughts

Today's lab tied together authentication and authorization. Creating my own client certificate, configuring a new kubectl context, and then restricting that user with RBAC made the concepts much easier to understand.

Security in Kubernetes isn't about giving users full access,but about giving them exactly the permissions they need, nothing more.

Have you ever left a tech meetup feeling like everyone else understood the conversation except you?

Thu, 02 Jul 2026 10:00:00 +0000

Have you ever left a tech meetup feeling like everyone else understood the conversation except you?

I've been there.

I enjoyed the networking, met some amazing people, and even looked forward to the food. But there were times I got home wondering if I'd actually learned anything or just spent my time, energy, and money.

That experience reminded me how overwhelming Kubernetes can feel when it's explained with too much technical jargon.

So I created a short video where Ada explains Kubernetes to Tunde during a walk to the bus stop. The goal was simple: make Kubernetes easier to understand without assuming any prior knowledge.

If you're learning Kubernetes or know someone who's just getting started, I'd love to hear your thoughts.

🎥 https://youtu.be/NCcXfrbzKz0

What's one tech topic you wish had been explained more simply when you were starting out?

Kubernetes Autoscaling: HPA vs VPA Explained With Hands-On Practice

Wed, 01 Jul 2026 12:25:29 +0000

If you've ever wondered how Kubernetes knows when to spin up more pods or give a pod more memory, that's autoscaling — and it's one of those concepts that sounds intimidating until you actually do it yourself. Day 17 of the #40DaysOfKubernetes challenge is where it clicked for me.

What is Autoscaling in Kubernetes?

At its core, autoscaling means Kubernetes adjusts resources automatically based on demand. You don't manually intervene every time traffic spikes. There are two main types:

HPA (Horizontal Pod Autoscaler) — adds or removes pods based on CPU/memory usage
VPA (Vertical Pod Autoscaler) — adjusts the resources (CPU/memory) of existing pods

Think of HPA as hiring more staff when the shop gets busy. VPA is more like giving one staff member more tools to handle the workload alone.

What I Did — Setting Up HPA

First I deployed the sample php-apache app with defined CPU requests and limits:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
spec:
  selector:
    matchLabels:
      run: php-apache
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - name: php-apache
        image: registry.k8s.io/hpa-example
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 500m
          requests:
            cpu: 200m

apiVersion: v1
kind: Service
metadata:
  name: php-apache
  labels:
    run: php-apache
spec:
  ports:
  - port: 80
  selector:
    run: php-apache

The key part is setting resources.requests.cpu — HPA needs this to calculate utilization. Without it, the autoscaler has nothing to measure against.

Then I created the HPA object targeting 50% average CPU utilization, with a minimum of 1 pod and maximum of 10. This is the declarative method:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

While this is the imperative method:

kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10

Generating Load to Watch It Scale

This is the fun part. I ran a load generator in a separate pod — basically a loop hammering the apache service with requests:

kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- \
  /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

Then watched the HPA respond in real time:

kubectl get hpa php-apache --watch

Watching the replica count climb from 1 to several pods as CPU utilization crossed 50% made the whole concept land in a way that reading documentation never does.

HPA vs VPA — When Do You Use Which?

	HPA	VPA
Scales	Number of pods	Pod resource limits
Best for	Stateless apps with variable traffic	Apps where sizing is hard to predict upfront
Works with	CPU, memory, custom metrics	CPU and memory

In practice, most production workloads use HPA. VPA is useful during early deployment when you're still figuring out the right resource requests for an app.

Key Takeaway

Don't skip setting resources.requests in your deployment spec. HPA is blind without it. That one line is what connects your workload to the autoscaler.

The more I learn about AWS, the more I respect fundamentals.

Wed, 10 Jun 2026 09:00:00 +0000

One thing I have noticed lately while learning cloud is how easy it is to feel behind, especially online.

You open LinkedIn or YouTube and see people talking about Kubernetes, advanced architectures, multi-cloud setups, and dozens of AWS services, as if everybody already understands everything.

Meanwhile, some people are still trying to fully understand what a VPC even does.

Many people quietly feel pressured to skip fundamentals because everyone else seems “ahead.”

I used to feel that way too.

At first, I thought learning AWS was mostly about memorising services and launching servers.

But once I started learning AWS more deeply, I realized the cloud is much bigger than that.

Things like VPCs, subnets, routing tables, NAT gateways, and security groups felt overwhelming at first.
Not because they were impossible to learn, but because many explanations made them sound overly technical.

At some point, I decided to create a video explaining VPCs to myself using a simple neighborhood analogy.

And that changed a lot for me.

For the first time, things genuinely started making sense.

The VPC became the neighborhood itself.

Public subnets felt like major roads anyone could access publicly.

Private subnets became quieter streets where sensitive resources like databases stay protected.

Even security groups stopped feeling like random rules and started making sense as gates controlling who can enter specific buildings.

That small shift changed the way I understood networking. I stopped trying to memorize definitions and started understanding why companies structure cloud environments the way they do.

That is the difference between passive learning and real understanding.

Watching tutorials feels productive, but troubleshooting teaches deeper lessons. I even wrote about that in my previous newsletter.

For example, I understood security groups much better after accidentally blocking traffic to my EC2 instance and trying to figure out why my application suddenly stopped responding.

That frustration taught me more than multiple videos combined.

Many people underestimate how normal confusion is in cloud computing.

Some concepts take time. Sometimes you revisit the same topic three or four times before it finally clicks.

Even experienced engineers still go back to fundamentals once in a while.

You do not need to understand everything immediately.

The internet sometimes makes tech careers look very linear.
Study. Get certified. Get hired.
But real learning usually looks messier than that.

There are moments of confusion. There are things you break accidentally. And there are concepts that make no sense today but suddenly click weeks later.

And that is probably how it is supposed to be.

So these days, I am trying to focus less on looking impressive online and more on actually understanding what I am learning or building.

The deeper I go into AWS, the more I realize strong fundamentals make advanced concepts less intimidating.

If you enjoy reflections like this, you can subscribe to the newsletter.

You can follow my work and projects on Twitter, LinkedIn, GitHub, and YouTube, where I share what I am building and learning along the way.

Thank you for reading.

Developer Journar Day5: How to Launch and Configure an Amazon EC2 Instance (From Scratch)

Sat, 30 May 2026 07:38:57 +0000

When you start working with cloud computing on AWS, one of the first major milestones is understanding how EC2 instances work. In simple terms, it’s like spinning up your own server on the internet in just a few minutes.

In this article, I’ll walk you through how to create and launch an EC2 instance that works as a basic web server.

What is an EC2 instance?

An instance in Amazon Web Services, specifically within the Amazon EC2 service, is a virtual machine that you can configure based on your needs: CPU, memory, storage, and operating system.

Think of it as renting a computer in the cloud that you can start, stop, and customize anytime.

⚙️ Step 1: Create the instance
The first step is to go to the EC2 dashboard and click “Launch Instance.”

Here you define some key settings:

️ Instance name

You can assign a descriptive name such as:

“My Web Server”

This helps you easily identify it later.

Step 2: Choose the AMI

An Amazon Machine Image is basically the base image used to create your server.

In this case, we use Amazon Linux, a lightweight and optimized OS for cloud environments.

️ Step 3: Instance type

This defines the computing power of your virtual machine.
For this example:

t3.micro
1 vCPU
1 GB RAM
Free Tier eligible This is perfect for learning and small projects.

Step 4: Key pair (SSH access)

To securely connect to your instance, you need a key pair:

Public key: stored in the instance
Private key: kept on your machine
This enables secure SSH access.

Step 5: Network settings

Here you control how your instance is accessed.

In this setup:

HTTPS traffic is allowed from the internet
A Security Group is used to manage inbound rules

This is what makes your server accessible from a browser.

Step 6: Storage configuration

You assign virtual disk storage (EBS):

8 GB
gp3 volume type Enough for a basic web server setup.

Step 7: User Data script (bootstrapping the server)

The base AMI only includes the operating system, so no web server is installed by default.

To fix this, we use User Data, which allows us to run a script at launch time that:

Installs a web server (like Apache or Nginx)
Starts and enables the service automatically
Prepares the instance to serve web content

Step 8: Launch the instance

Once everything is configured:
Click “Launch Instance”

AWS handles the rest: provisioning, setup, and booting your virtual machine.

Step 9: Access your server

After a couple of minutes, the instance will be in a running state.

Then:

Copy the public IP address
Open a browser
Paste the IP

And you’ll see your web server running

Conclusion
Launching an EC2 instance is one of the first essential steps in understanding cloud computing on AWS.

With just a few clicks, you can deploy a fully functional virtual server ready to host applications, APIs, or websites.

If you want, I can also help you next with:

adding AWS architecture diagrams
explaining VPC, subnets, and security groups visually
or turning this into a “Cloud Engineer roadmap series” post

Just tell me

10 Tools Every Gen Z Developer Should Know About (2026 Edition)

Sat, 14 Mar 2026 12:32:58 +0000

10 Tools Every Gen Z Developer Should Know About (2026 Edition)

If you’re a Gen Z developer trying to survive the chaotic tech world, knowing the right tools isn’t optional — it’s your superpower. The right tools can save hours of work, make your projects look professional, and even help you land your dream job.

Here are 10 tools that will level up your coding, design, and productivity game — fast.

1. AI Coding Assistants – ChatGPT & GitHub Copilot

AI is no longer sci-fi. These assistants can write code snippets, debug, and even explain complex algorithms in plain English.

Why it matters: Speeds up coding and learning simultaneously.
Pro tip: Ask ChatGPT to explain your own code — it’s like having a tutor 24/7.

2. Modern IDEs & Editors – VS Code & JetBrains Fleet

Your development environment can make or break your workflow.

Why it matters: Extensions and smart suggestions turn small projects into pro-level apps.
Pro tip: Use VS Code extensions like Live Share to collaborate in real-time.

3. Cloud Development Platforms – Replit & CodeSandbox

Set up a dev environment in seconds, no heavy installations needed.

Why it matters: Instantly share your projects with anyone online.
Pro tip: Experiment with new frameworks without messing up your local machine.

4. Version Control Tools – GitHub & GitLab

Version control isn’t just for pros — it’s how developers keep their work organized.

Why it matters: Collaboration, portfolio building, and job-ready skills.
Pro tip: Use GitHub Copilot integration to save time on pull requests and commits.

5. Design & Prototyping Tools – Figma, Canva, & MidJourney

Modern developers aren’t just coders — design matters.

Why it matters: Build prototypes, mockups, or AI-generated graphics for your projects.
Pro tip: Use Figma community templates to speed up your UI design process.

6. API & Backend Tools – Postman, Supabase & Firebase

You don’t need to be a backend wizard anymore.

Why it matters: Build full-stack apps quickly with minimal server setup.
Pro tip: Connect Firebase authentication and database in under 30 minutes.

7. Dev Automation / Productivity Tools – Notion, Obsidian & Linear

Managing projects can be messy. These tools make it clean and automated.

Why it matters: Organize your projects, track progress, and automate repetitive tasks.
Pro tip: Use Notion templates to manage your coding projects like a startup.

8. Learning & Experimentation Platforms – LeetCode, Kaggle & HackerRank

Practice makes perfect. These platforms turn learning into a challenge.

Why it matters: Sharpen skills, prepare for interviews, or compete globally.
Pro tip: Kaggle competitions are perfect for building a real portfolio.

9. AI-Powered Design & Writing Tools – ChatGPT, Runway, Copymatic

Content, docs, and even marketing materials can be handled with AI.

Why it matters: Helps solo developers focus on coding while AI handles the rest.
Pro tip: Generate README files or documentation instantly — makes your project look pro.

10. Emerging Tech Tools – Web3 Dev, LangChain & Quantum SDKs

Stay ahead of the curve. The future belongs to developers who innovate.

Why it matters: Skills in emerging tech give you an edge in the next 5–10 years.
Pro tip: Experiment with LangChain for AI-powered apps — it’s the next big trend.

Closing Thoughts

These 10 tools won’t just make you a better developer — they’ll make you unstoppable. The trick is to pick a few, master them, and build projects that showcase your skills.

Which of these tools have you tried? And which one are you starting with today? Drop a comment and share this with your coding squad!

Docker key terms simplified

Sat, 07 Mar 2026 10:28:38 +0000

Containerization is a must-have skill for DevOps engineers and software developers alike. It is essential for creating robust, deployable applications. In this article, I will explain the Docker concepts every developer needs to know and provide a hands-on example.

Prerequisites
What is Docker?
Why is Docker Important
Idea of How Docker Works
What is a Docker Image?
Dockerfile Commands Explanation
Running the Image
What is a Docker Container?
How to Run a Docker Container
Docker Volumes
Final Thoughts

Prerequisites:

to follow along this article be sure to

Download and install Docker Desktop.
Ensure Docker is running (especially on macOS or Windows) while practicing.

What is docker:

Docker is a software platform that helps package and run applications on any machine with the exact environment you have configured.

Teams often face the "it works on my machine" issue, where an application runs locally but crashes when a teammate tries to run it. This happens due to:

Mismatched environment or library versions.
Differences in Operating Systems (OS) that don't support specific features.

Configuring every environment to match can be time-consuming and slows down development. Docker solves this by "containerizing" the environment.

Why is docker important:

As a developer, you shouldn't waste time troubleshooting environment setups. Docker allows you to:

Configure the environment once and reuse it throughout the development process.
Run multiple copies of the same application with different environments without conflicts.
Eliminate library version mismatches.

Idea of how docker works:

Imagine you have a minivan that you use as a home, a business venture, and transport all at once. You can drive anywhere, and everything you need is already inside. You don't need to find a new bed or kitchen every time you move; it's all in the van.

That is the idea of Docker. It contains all the essentials needed to run an application in a single container. This container can be used by anyone, anywhere, without further configuration.

What is a docker image:

Docker image is a blueprint that describes how our application environment will look like and how it will be built from the ground up and after it is created it cannot be modified.

with docker image we describe how docker should allocate and arrange our application environment this process happens in a file called Dockerfile.

A Dockerfile defines the layers (stages) required to run the container, starting from the OS up to your codebase.

#this creates a node runtime of version 18 and OS of alpine
From node:18-alpine
 
#this specifies the path where docker will store you application codes this could be any name of path you want
WORKDIR /app 

#copies the package file to docker files directory so that when npm can have packages to install
COPY package.json 

#this will run and install all the packages you are using in your package file
RUN npm run install 

#copies all the codebase in your directory to the docker files
COPY . . 

#define the port that will be used when testing the application

EXPOSE 3000

# if you are using nodemon you write a command to start your application like this
CMD ["npm", "ru", "dev"]

# if you are using node to execute the code you use
CMD ["node", "name of the file.js"]

Docker file commands Explanation

From specifies which kind of runtime you are using in my case I am using JavaScript, but you can use whatever based on your project setup and the colon helps in specifying the version and the hyphen to specify the OS system the system will be built on top of.

WORKDIR this command specifies the directory in which docker will store your application codes generally you can specify any path you want but /app is the convenient one.

COPY this command copies a specific file from your project files to docker

EXPOSE this command specifies the port in which you are application is running on and also helps to define a port your application will run on when starting a container

CMD this command is the one that runs you application after it have been initialized.

Note: for window users and macOS when running your application using nodemon package inside your run script file in package.json file place -L flag before your file for it to run, without it your application won’t get started unless you are a Linux user

Running the image

After setting your docker file you can create an image by running the build command in your terminal using the docker command

docker build -t my-first-docker-image:version-one .

The build command is the one used to build the image for your docker, and the -t flag us used to specify the tag of the image.

Image tag helps to differentiate images with the same name, docker image cannot be modified after they are built to edit it you would need to create a new image, but you may want to keep the same image but change its versions so you can track your changes and version over time and to do that the -t flag helps create a tag that you can refer as a version of the image, to specify the tag or version of your image you use double colon(: )

e.g.

docker build -t my-first-docker-image:version-two .

If you didn’t define the tag name by default docker will assign it the latest as the default tag name

e.g.

docker build -t my-first-docker-image .

without -t flag it would hard to track down the changes of the image when we are constantly making changes and rebuilding the image

. tells docker where to look for files to copy

What is docker container:

Docker container is the implementation of a plan created in docker image, now all of the specifications you’ve specified inside your Dockerfile will be executed one by one and create a container of your application codes and also create the underlying OS which is alpine and a runtime environment of node.js below is the command to run and create a container.

How to run a docker container

docker run --name container_name -p  image-name

Let’s start with knowing each command step by step

run command is used to start a new container.

--name flag specifies the name for your container, the name you would like your container to have

-p flag is referred as port mapping this should be used if you have specified EXPOSE command in your Dockerfile it helps map any port you specify to the exposed port in docker file you put you preferred port on the left and the exposed port on the right and separate them using a colon.

e.g.

-p prefferedPort:exposedPort

image-name and then you have to specify the name of image in which you want to create a container from.

running a container from the image we’ve created above we can something like this

docker run --name container_one -p 3000:3000 my-first-docker-image:version-one

Docker Volumes

Docker images are "read-only." If you change your code on your computer, the running container won't see those changes unless you rebuild the image. Docker Volumes solve this.

Volumes act as a bridge between your local development environment and the running container.

docker run --name container_one -p 3000:3000 -v /our apps location:./our docker specified path

the command is the one we’ve been using to create a container before but The -v flag binds your local folder to the folder inside the container (e.g., /app). Now, any changes you make in your code editor will instantly reflect inside the running container!

So, the real command can look something like this:

docker run --name container_one -p 3000:3000 -v /C:\Users\arnol\OneDrive\Documentos\newContent\client:./app

Final Thoughts

Docker makes development faster and smoother. Whether you are aiming to be a DevOps engineer or a Full-stack developer, understanding these basics is essential for modern software engineering.

Thank you for reading! If you learned something new or have suggestions, please let me know in the comments below.

OCI Generative AI Security: Dedicated GPUs, RDMA Networks, and Enterprise-Grade Data Protection

Sat, 17 Jan 2026 18:12:41 +0000

Security and privacy of customer workloads is an essential design tenet for Oracle Cloud Infrastructure (OCI) Generative AI. In an era where data breaches and AI security vulnerabilities dominate headlines, OCI has architected a comprehensive security framework that ensures enterprise-grade protection for AI workloads. This deep dive explores the multi-layered security approach that makes OCI Generative AI one of the most secure platforms for enterprise AI deployments.

The Foundation: GPU Isolation and Dedicated Infrastructure

Dedicated GPU Allocation

GPUs allocated for a customer's generative AI tasks are pooled within a dedicated RDMA network, ensuring they are exclusively allocated to a single customer and not shared with others. This fundamental isolation guarantees that customer data remains secure and inaccessible to unauthorized parties.

Key Isolation Characteristics:

Physical Separation: Each customer's GPU cluster operates on dedicated hardware, preventing any cross-contamination or data leakage between tenants.

Network Isolation: RDMA cluster networks with less than 10 microsecond latency provide ultra-high-bandwidth connections (1.6 TB/sec internode bandwidth) while maintaining complete isolation from other customers' workloads.

Compute Isolation: OCI Supercluster can deploy up to 32,768 GPUs per cluster, all dedicated to a single customer with no sharing of computational resources.

RDMA Network Architecture

Remote Direct Memory Access (RDMA) is a critical component of OCI's AI infrastructure, providing both performance and security benefits.

RDMA Technology Overview:

RDMA allows for low-latency connections between nodes and access to GPU memory without involving the CPU. This technology enables:

Ultra-Low Latency: Less than 10 microseconds between nodes
High Bandwidth: 1.6 TB/sec internode bandwidth for massive data transfers
CPU Offload: Direct memory-to-memory transfers without CPU intervention
Security: Dedicated network fabric preventing cross-tenant access

Network Types:

OCI's cluster network uses RDMA over Converged Ethernet Version 2 (RoCE v2) on top of NVIDIA ConnectX-7 network interface cards (NICs) to support high-throughput and latency-sensitive workloads.

For advanced deployments, OCI utilizes NVIDIA Quantum InfiniBand or NVIDIA Spectrum-X Ethernet with RDMA over Converged Ethernet (RoCE) configuration.

Scale and Performance:

OCI enables customers to cluster up to 4,096 bare metal nodes, each with 8 GPUs, totaling up to 32,768 GPUs. The latest infrastructure scales even further—zettascale OCI Supercluster scales up to 131,072 GPUs, making it the largest hyperscale AI supercomputer in the cloud.

Hardware Security Modules (HSM)

Keys are stored on Hardware Security Modules (HSMs) that meet Federal Information Processing Standards (FIPS) 140-2 Security Level 3 security certification.

HSM Protection Modes:

Software Protection: Keys stored and processed on servers, recommended for most use cases with strong encryption.

HSM Protection: Keys stored on HSMs meeting FIPS 140-2 Security Level 3 certification, recommended for stringent compliance requirements like financial services and healthcare.

Model Endpoint Security

Single-Customer Model Endpoints

For strong data privacy and security, a dedicated GPU cluster only handles fine-tuned models for a single customer. This architectural decision ensures:

Data Isolation: No model from one customer processes data from another customer
Performance Isolation: Resources allocated exclusively for your workloads
Compliance: Easier to meet regulatory requirements with clear tenant boundaries
Security Boundaries: Physical and logical separation of processing environments

Endpoint Access Control

Model endpoints implement multiple layers of access control:

Authentication: API keys, OAuth tokens, or IAM credentials required for all requests
Authorization: Role-Based Access Control (RBAC) determines what actions authenticated users can perform
Network Security: Private endpoints within Virtual Cloud Networks (VCNs), with optional public access through controlled ingress
Rate Limiting: Configurable throttling to prevent abuse and ensure fair resource allocation

Customer Data and Model Isolation

Tenancy-Level Isolation

Customer data access is restricted within a customer's tenancy so that one customer's data cannot be seen by another customer. This fundamental principle manifests across multiple dimensions:

Compartment Isolation:

OCI uses compartments to organize and isolate resources. The architecture implements compartmentalization and private subnets to isolate different operational environments.

Best Practice: Create separate compartments for:

Development environments
Staging/testing
Production workloads
Different business units or projects

Data Separation:

Training data, fine-tuned model weights, embeddings, and inference logs are all stored within customer-controlled storage with strict access controls preventing cross-tenant access.

Workload Isolation:

A single NVIDIA GB200 NVL72 rack can be configured to launch multiple NVLink groupings for smaller workloads and provide strong isolation for efficient workload distribution.

Network-Level Isolation

Private Subnets:

Resources deployed in private subnets within VCNs, preventing direct internet access unless explicitly configured.

Security Lists and Network Security Groups (NSGs):

Stateful firewalls controlling inbound and outbound traffic at the subnet and network interface level.

Service Gateway and NAT Gateway:

Controlled access to Oracle services and internet resources without exposing instances to inbound internet traffic.

Integrated OCI Security Services

OCI Generative AI leverages Oracle's comprehensive security ecosystem to provide defense-in-depth.

OCI Identity and Access Management (IAM)

IAM provides the foundation for authentication and authorization across all OCI services.

Key IAM Capabilities:

User and Group Management: Create users, organize them into groups, and assign permissions based on job functions.

Policies: Stringent access controls using OCI IAM policies enforce least-privilege access.

Example IAM Policy for Generative AI:

Allow group GenAI-Engineers to manage generative-ai-family in compartment GenAI-Prod
Allow group GenAI-Engineers to use virtual-network-family in compartment GenAI-Networking
Allow group Data-Scientists to read generative-ai-endpoints in compartment GenAI-Prod

Identity Providers: Integration with corporate identity systems (SAML, SCIM) for single sign-on.

Multi-Factor Authentication (MFA): Additional security layer requiring multiple forms of verification.

OCI Key Management Service (Vault)

The Key Management Service is critical for protecting data at rest and managing encryption keys.

Vault Architecture:

Oracle Vault is a logical grouping of keys. There are two types of vaults: Private and Virtual, which have different levels of isolation, pricing, and computing.

Key Features:

Centralized Key Management: OCI Key Management provides centralized management of the encryption of your data.

Customer-Managed Keys (CMK): Customers create and control their own encryption keys, maintaining full ownership.

Oracle-Managed Keys: Default encryption with Oracle-managed keys that are automatically rotated.

Key Rotation: Each master encryption key (MEK) is automatically assigned a key version. When you rotate a key, the Vault service generates a new key version.

Integration with Generative AI:

OCI Generative AI uses Key Management Service to:

Store Base Model Keys Securely: Encryption keys for foundational models are managed through Vault
Encrypt Fine-Tuned Models: Custom model weights encrypted with customer-controlled keys
Protect Training Data: Datasets used for fine-tuning encrypted before storage
Secure Inference Data: Input and output data encrypted in transit and at rest

OCI Object Storage Security

Object Storage buckets store critical AI assets including training data, model weights, and embeddings.

Default Encryption:

OCI encrypts all objects by default with Oracle-managed keys, which are periodically rotated. This ensures data is protected even if customers don't explicitly configure encryption.

Customer-Managed Encryption:

OCI provides options for customers to use their own keys for more security. Using IAM policies that authenticate users performing tasks, a best practice is creating and periodically rotating encryption keys in the Vault service to protect resources in Object Storage.

Encryption at Multiple Layers:

Object-Level Encryption: Each object encrypted with a unique Data Encryption Key (DEK)
Bucket-Level Encryption: Entire buckets can be encrypted with a specific Master Encryption Key (MEK)
Client-Side Encryption: Customers can encrypt data before sending it to Oracle Object Storage for additional control

Required IAM Policies:

To use customer-managed keys with Object Storage, create a policy like:

allow service objectstorage-<region_name> to use keys in compartment <compartment_name>

Access Control:

Pre-Authenticated Requests (PARs): Time-bound access to objects without requiring IAM credentials
Bucket Visibility: Private by default, with explicit configuration required for public access
Versioning: Track and restore previous versions of objects

Security Best Practices:

Restrict BUCKET_UPDATE permission to a minimal set of IAM groups to minimize the possibility of existing buckets being made public inadvertently or maliciously.

Cloud Guard: Active Threat Detection

Comprehensive logging and audit trails maintained for all significant operations.

Cloud Guard continuously monitors your OCI environment for security weaknesses and threats.

Detector Rules for Object Storage:

Cloud Guard includes detector rules for Object Storage to identify public buckets, unencrypted data, and other security issues.

Capabilities:

Automated Detection: Identifies misconfigurations and security threats in real-time
Responder Actions: Automatically remediate issues or notify administrators
Problem Dashboard: Centralized view of all detected security problems
Integration: Works across all OCI services including Generative AI resources

Data Protection Throughout the AI Lifecycle

Training Data Security

Ingestion:

Data uploaded to Object Storage with automatic encryption
Customer-managed keys available for additional control
Access restricted via IAM policies

Processing:

Training occurs on dedicated GPUs isolated within customer's RDMA network
No data sharing between customers
Temporary data purged after training completion

Storage:

Fine-tuned model weights stored in Object Storage buckets
Encrypted by default and managed by Key Management Service
Version control for model iterations

Inference Data Security

Input Processing:

Requests authenticated via IAM or API keys
Data encrypted in transit using TLS 1.2+
Processed on dedicated endpoints with no cross-tenant access

Output Generation:

Responses encrypted in transit
Optional logging with encryption at rest
Retention policies configurable per use case

Monitoring:

Continuous monitoring of resource usage to proactively mitigate contention
Audit logs for all API calls
Integration with OCI Logging Analytics

Advanced Security Features

Security Zones

Security Zones provide compartment-associated Oracle-defined recipes of security policies based on best practices.

Enforced Policies:

No Public Access: Resources cannot be accessible from public internet
Customer-Managed Keys: Data encryption enforced using Customer-Managed Keys for block volumes, boot volumes, and Object Storage buckets
Deny Public Buckets: Public access to Object Storage buckets is denied to prevent accidental exposure

Use Case:

Deploy Generative AI workloads in Security Zones to automatically enforce security best practices and maintain compliance.

Compliance and Certifications

OCI maintains extensive compliance certifications relevant to AI workloads:

Regional Compliance:

GDPR: European data protection requirements
HIPAA: Healthcare data in the United States
FedRAMP: U.S. government cloud security
ISO 27001/27017/27018: International security standards

Industry-Specific:

PCI DSS: Payment card industry
SOC 1/2/3: Service organization controls
FIPS 140-2 Level 3: Cryptographic module security

Sovereign Cloud Options

Oracle's distributed cloud, AI infrastructure, and generative AI services enable governments and enterprises to deploy AI factories that run cloud services locally and within a country's secure premises with operational controls supporting sovereign goals.

Available Options:

OCI Dedicated Region: Entire cloud region operated exclusively for a single customer
Oracle Alloy: Cloud platform for partners to deliver sovereign cloud
Oracle EU Sovereign Cloud: European data residency and operational sovereignty
Oracle Government Cloud: FedRAMP-authorized regions for U.S. government

Best Practices for Secure Generative AI Deployments

1. Implement Least-Privilege Access

Principle: Grant only the minimum permissions necessary for each role.

Implementation:

# Development team - read-only on production
Allow group GenAI-Developers to read generative-ai-family in compartment GenAI-Prod

# Production team - full management
Allow group GenAI-Production to manage generative-ai-family in compartment GenAI-Prod

2. Use Customer-Managed Keys

Benefit: Full control over encryption key lifecycle and usage.

Steps:

Create Vault in Key Management Service
Generate or import master encryption key
Create IAM policy allowing services to use key
Configure buckets and resources to use customer key

3. Enable Comprehensive Logging

Audit Logs: Enable audit logging for all API calls to Generative AI services
Flow Logs: Monitor network traffic to/from GPU clusters
Application Logs: Capture inference requests and responses for analysis

4. Leverage Network Isolation

Private Endpoints: Deploy model endpoints in private subnets
Service Gateway: Access Oracle services without internet exposure
Bastion Hosts: Secure administrative access to resources

5. Implement Defense-in-Depth

Multiple Layers:

Network security (NSGs, Security Lists)
Identity security (IAM, MFA)
Data security (encryption at rest and in transit)
Application security (input validation, rate limiting)
Monitoring security (Cloud Guard, Logging Analytics)

6. Regular Security Reviews

Periodic Assessments:

Review IAM policies quarterly
Rotate encryption keys according to policy
Audit user access and remove unused accounts
Update network security rules based on traffic patterns

7. Secure Development Practices

The architecture includes a CI/CD pipeline for promoting models from the playground environment to production.

Separation of Environments:

Development: Unrestricted experimentation with synthetic data
Staging: Production-like with access controls
Production: Maximum security with customer data

Monitoring and Incident Response

Real-Time Monitoring

Metrics to Track:

API request rates and latency
Authentication failures
Unusual access patterns
Resource utilization anomalies

Alerting:
Configure Cloud Events to trigger notifications for:

IAM policy changes
Network security group modifications
Encryption key access
Dedicated cluster scaling events

Incident Response

Preparation:

Define incident response procedures
Establish communication channels
Designate response team members

Detection:

Cloud Guard active monitoring
Log analysis for anomalies
User-reported issues

Containment:

Isolate affected resources
Revoke compromised credentials
Block suspicious network traffic

Recovery:

Restore from backups if necessary
Rotate encryption keys
Update security configurations

Lessons Learned:

Document incident timeline
Identify root causes
Update procedures and controls

Future Security Enhancements

Oracle continues investing in security capabilities:

AI-Powered Security:
Machine learning models detecting anomalous behavior in API usage patterns and access patterns.

Zero Trust Architecture:
Moving toward continuous verification of all access requests regardless of source.

Confidential Computing:
NVIDIA BlueField-3 DPUs accelerate networking, storage, and security workloads, enabling hardware-based isolation for sensitive computations.

Enhanced Compliance:
Expanding certifications and attestations for emerging regulatory requirements.

OCI Generative AI security is built on a comprehensive, multi-layered approach that addresses the unique challenges of enterprise AI deployments:

Infrastructure Security:

Dedicated GPUs pooled within RDMA networks exclusively for single customers
Dedicated GPU clusters handling only customer's base and fine-tuned models
Ultra-low latency RDMA networking with complete tenant isolation

Data Security:

Customer data access restricted within tenancy
Object Storage encrypted by default with periodic key rotation
Customer-managed keys for maximum control

Identity and Access:

Comprehensive IAM with fine-grained policies
Integration with enterprise identity providers
Multi-factor authentication support

Compliance:

FIPS 140-2 Level 3 certified HSMs
Sovereign cloud options for data residency
Extensive regulatory certifications

Security and privacy of customer workloads is truly an essential design tenet at OCI. By combining dedicated infrastructure, comprehensive encryption, identity controls, and continuous monitoring, OCI Generative AI provides enterprise-grade security that enables organizations to confidently deploy AI at scale.

How does your organization approach AI security? What security features are most critical for your use cases? Share your experiences in the comments

OCI Generative AI Service: Enterprise-Grade LLMs on Oracle Cloud

Fri, 12 Dec 2025 09:55:53 +0000

Oracle Cloud Infrastructure (OCI) Generative AI is a fully managed service providing enterprises with access to state-of-the-art, customizable large language models through a comprehensive API. OCI positions itself as a neutral, enterprise-focused platform offering unprecedented choice and flexibility in the generative AI space.

What is OCI Generative AI?

OCI Generative AI helps enterprises seamlessly integrate advanced language comprehension capabilities into applications, providing a complete end-to-end platform for building, customizing, and deploying LLM-powered applications at scale.

Key Capabilities:

Access to pretrained foundational models from multiple leading AI providers
Flexible fine-tuning with custom datasets on dedicated infrastructure
Enterprise-grade security, compliance, and data sovereignty
Integration with Oracle's broader AI ecosystem
Support for both on-demand usage and dedicated hosting

Available Models (2025)

Cohere Models:

Command A (03-2025): 256K token context, excels at tool use, agents, RAG, and multilingual tasks
Command R/R+: 128K context for RAG and enterprise use cases
Embed models: English v3.0, Multilingual v3.0, Embed 4

Meta Llama Models:

Llama 4 Maverick: 17B active from ~400B total (mixture-of-experts)
Llama 4 Scout: 17B active from ~109B total
Llama 3.3 (70B), Llama 3.2 Vision (90B, 11B), Llama 3.1 (405B, 70B)

Google Gemini Models (Coming Soon):
Oracle will be the only hyperscaler aside from Google Cloud to offer Gemini as a managed service.

Gemini 2.5 Pro/Flash/Flash-Lite

xAI Grok Models:

Grok 4/Grok 4 Fast, Grok 3 series, Grok Code Fast 1

OpenAI Models:

gpt-oss-120b and gpt-oss-20b

Core Features

1. Pretrained Models

Chat Models: Multi-turn conversational AI
Text Generation: Content creation, code generation, documentation
Embedding Models: Vector representations for semantic search (384-1024 dimensions)
Rerank Models: Relevance scoring for search results

2. Fine-Tuning

Strategies:

T-Few & Vanilla for Cohere models (with layer-specific optimization)
LoRA for Llama 3 models (efficient parameter adaptation)

Customization:

Training epochs, learning rate, batch size
Early stopping controls
JSONL format with prompt/completion pairs

3. Dedicated AI Clusters

GPU-based compute resources exclusive to your tenancy:

Isolated infrastructure for fine-tuning
Private GPUs for hosting with zero-downtime scaling
Multiple cluster sizes (Small/Large/Large Generic 2/4 for 405B models)

4. Deployment Options

On-Demand: Pay-per-character, ideal for experimentation
Dedicated Clusters: Predictable costs for production workloads

5. OCI Generative AI Agents (2024-2025)

Fully managed RAG service combining LLMs with enterprise search.

Agent Hub Features (March 2025):

SQL Tool: Self-correction, multi-dialect support (Oracle SQL, SQLite)
Enhanced RAG Tool: Hybrid search, multi-modal parsing, multi-lingual support (7+ languages)
Integration with OCI Object Storage, OpenSearch, Oracle Database 23ai

Use Cases

Text Generation: Blog posts, marketing copy, documentation
Semantic Search: Intent-based search, recommendations, document retrieval
Document Summarization: Executive summaries, support tickets, research papers
Classification: Support ticket routing, sentiment analysis, intent detection
Question Answering: Intelligent responses from documents and knowledge bases
Enterprise Knowledge Management: RAG-powered customer support

Developer Experience

Access Methods:

OCI Console Playground
REST API
OCI CLI
SDKs (Python, Java, TypeScript, Node.js)

Framework Integration:

LangChain: Prompt templating, memory, chain-of-thought
LlamaIndex: Context-augmented applications, RAG solutions

Tool Use: Chat models can integrate with external tools and APIs for complex queries requiring external data.

Security & Compliance

Key Features:

Data isolation within your tenancy
Encryption at rest and in transit
Fine-grained IAM policies and RBAC
Comprehensive audit trails
Private endpoints and VPN connectivity

Regional Availability:

US (Chicago, Phoenix, Ashburn)
Europe (Frankfurt, London, Amsterdam)
Asia Pacific (Tokyo, Mumbai, Seoul)
Middle East (Dubai, Jeddah)
Latin America (São Paulo)
Sovereign Cloud: Oracle EU Sovereign Cloud for data residency

Advanced Features

Configurable Parameters:

Temperature (0.0-1.0)
Top-k and top-p sampling
Frequency/presence penalties
Seed parameter for reproducibility

Performance Benchmarks:
RAG scenarios with 2,000-token prompts and 200-token responses benchmarked across cluster types.

Pricing

Free Tier: $300 credits for trial
On-Demand: Pay-per-character (input/output)
Dedicated Clusters: Predictable monthly costs

Oracle's AI Ecosystem

Oracle Database 23ai:

Native AI Vector Search
In-database LLM integration
Support for RAG workflows

MySQL HeatWave:

In-database LLMs (HeatWave GenAI)
Automated vector store

Oracle Fusion Applications:
Generative AI embedded across ERP, HCM, SCM, and CX applications.

OCI Data Science:
No-code access to open-source LLMs (Meta, Mistral AI) with Hugging Face Transformers and PyTorch.

Certification

OCI 2025 Generative AI Professional Certification covers:

LLM Fundamentals & Transformer Architecture
Prompt Engineering
Fine-Tuning Techniques
RAG Workflows & Vector Databases
Agent Development

Resources: Free tutorials, Coursera courses, Oracle MyLearn platform

Competitive Advantages

Model Choice: Access to Cohere, Meta, Google, xAI, OpenAI through unified platform
Enterprise-First: Security, compliance, data sovereignty
Cost-Effective: Better price-performance with transparent pricing
Database Integration: Unique integration with Oracle Database 23ai and MySQL HeatWave
Sovereign Cloud: Data residency guarantees

Getting Started

Create OCI account (free tier available)
Navigate to Analytics & AI → Generative AI
Choose approach: Playground, API/SDK, or Fine-tuning
Select model and configure parameters
Start building

Example: Building a Support Chatbot

Ingest documentation into Oracle Database 23ai vector store
Create embeddings using Cohere Embed models
Deploy RAG Agent
Implement chat interface with LangChain
Enable multi-turn conversations with tool calling

OCI Generative AI offers comprehensive, enterprise-focused LLM capabilities addressing real-world needs: security, sovereignty, choice, and integration. Whether building chatbots, implementing semantic search, or developing multi-agent systems, OCI provides the foundation for enterprise-grade AI applications.

Are you using OCI Generative AI? Share your experiences below

Overcoming Key Challenges to Machine Learning Adoption

Tue, 14 Oct 2025 21:01:11 +0000

Introduction
Machine Learning (ML) promises transformative insights and automation, but adopting it at scale is far from simple. Organizations often underestimate the complexity of operationalizing ML models. This article explores the four major challenges to ML adoption—and how to address them effectively.

1. Data Preparation

High-quality data is the foundation of any ML initiative.

Yet, most projects stall here because:

Access Issues: Data silos and compliance restrictions make it hard to gather complete datasets.

Pipeline Management: Cleaning, labeling, and maintaining data pipelines over time is resource-intensive.

✅ Actionable Solutions

Centralize Data Access: Implement a data lake or warehouse with role-based access controls.
Automate Data Quality Checks: Use tools like Great Expectations or Deequ for continuous validation.
Version Control for Data: Adopt DVC (Data Version Control) to track changes and maintain reproducibility.

Pro Tip: Invest in automated data validation tools and establish clear data governance policies early.

2. Model Development

Building accurate and interpretable models is harder than it looks:

Model Selection & Training: Choosing the right algorithm and tuning hyperparameters requires expertise. Selecting the

Deployment at Scale: Rolling out multiple models across environments introduces complexity.

✅ Actionable Solutions

Leverage AutoML: Tools like H2O.ai or Google AutoML can accelerate model selection and tuning.
Adopt MLOps Frameworks: Use Kubeflow or MLflow for streamlined training, deployment, and monitoring.
Containerize Models: Package models in Docker containers for portability and scalability.

Pro Tip: Use MLOps frameworks like Kubeflow or MLflow to streamline training and deployment.

3. Model Governance & Security

Once deployed, models need continuous oversight:

Bias & Drift Monitoring: Models degrade over time as data changes.

Adversarial Attacks: ML systems can be manipulated if not properly secured.

✅ Actionable Solutions

Bias Audits: Integrate fairness metrics using libraries like AIF360 or Fairlearn.
Drift Detection: Automate drift alerts with Evidently AI or WhyLabs.
Security Hardening: Apply adversarial testing and encrypt model endpoints using TLS.

Pro Tip: Implement automated drift detection and adversarial testing as part of your CI/CD pipeline.

4. Access to Compute Resources

Training ML models demands significant compute power:

GPU Availability: Limited GPU resources can bottleneck training.
Workload Scheduling: Efficient orchestration is essential for cost control.

✅ Actionable Solutions

Cloud GPU Clusters: Use AWS EC2 GPU instances or Azure Machine Learning Compute for elasticity.
Autoscaling: Implement Kubernetes autoscaling for dynamic resource allocation.
Spot Instances: Reduce costs by leveraging spot pricing for non-critical workloads.

Pro Tip: Explore cloud-based GPU clusters and autoscaling strategies to optimize resource usage.

5. Talent: The Hidden Barrier

Scarcity of specialized talent remains one of the biggest hurdles. Upskilling your team and leveraging community resources can bridge this gap.

✅ Actionable Solutions

Internal Bootcamps: Organize ML enablement sessions for engineers and analysts.
Mentorship Programs: Pair junior staff with experienced ML practitioners.
Community Engagement: Encourage participation in open-source projects and forums.

Conclusion

ML adoption isn’t just about algorithms—it’s about data readiness, governance, infrastructure, and talent. By addressing these challenges proactively, organizations can unlock the full potential of machine learning.

What is Edge Computing, and Why Should Developers Care?

Thu, 25 Sep 2025 08:40:26 +0000

What is Edge Computing, and Why Should Developers Care?

Introduction

You’ve probably noticed it before: even cloud apps running on top-tier internet sometimes lag. Now imagine that lag happening in a self-driving car that needs to make a decision in milliseconds—or in a medical device monitoring live patient data. That “half‑a‑second delay” suddenly feels like forever.

And that’s exactly why edge computing exists. Instead of sending all your data on a round trip to the cloud and back, edge computing shifts the processing power closer to where the data is born—your phone, your IoT device, or even the microcontroller inside a factory sensor.

For us developers, this isn’t just another infrastructure buzzword. Edge computing flips the playbook, giving us new ways to build apps that are insanely fast, smarter, and more resilient. Let’s unpack what it is—and why you should definitely have it on your radar.

Understanding Edge Computing

What Actually Is Edge Computing?

At its simplest, edge computing means pushing computation closer to the data source.

The old cloud model (traditional):
Data → sent to a centralized cloud → processed → response sent back to the device.

The edge model (new):
Data → processed locally at the “edge” → only essential info goes to the cloud.

Think of it as turning “dumb” endpoints into smart, decision‑making nodes. No more waiting around for a cloud server thousands of miles away.

Real‑world use cases include:

AR/VR apps that need frame‑perfect, real‑time performance.
Smart cameras that can detect faces or threats on‑device.
Autonomous vehicles reacting in milliseconds without asking permission from the cloud.

Why Edge Computing Rocks (Benefits That Matter)

Here’s why this isn’t hype:

⚡ Low latency: Real‑time feels truly real‑time.
Bandwidth savings: You’re not shooting terabytes of raw data to the cloud.
Higher reliability: Systems keep working even if connectivity drops.
Better security: Sensitive data can stay local instead of traveling across networks.

For apps, this translates directly into a smoother, stronger user experience—and for devs, that’s a competitive edge.

Why Developers Should Care

New Playground, New Problems to Solve

Edge isn’t just an infra tweak—it’s opening entirely new categories of software. Right now, people are already building:

IoT ecosystems for smart cars, homes, and wearables.
Drones and robots that drive themselves.
Immersive AR/VR apps that break the latency barrier.
Fintech and healthcare solutions where milliseconds literally matter.

Guess what? Every single one of those needs developers who understand edge.

Skills Worth Leveling Up

If you want to future‑proof your skill set, here’s where to start hacking:

Distributed systems thinking: Split workloads between cloud and edge.
Optimization for resource‑constrained devices: Write lightweight, efficient code that still packs a punch.
Hands‑on with frameworks: AWS IoT Greengrass, Azure IoT Edge, Google Distributed Cloud Edge, etc.
Design for hybrid: Decide what must run locally and what makes sense in the cloud.

This isn’t just “learn another cloud service.” It’s about rethinking where your app’s intelligence actually lives.

Watch Out for These Challenges

Not everything is shiny—there are some headaches you’ll run into:

Device chaos: Every device is different. → Fix: Use containers and lightweight Kubernetes (K3s) to standardize deployments.
Mass deployment hell: Updating thousands of devices isn’t fun. → Fix: Leverage DevOps pipelines and orchestration tooling.
Cost vs. performance juggling: Some tasks are better in the cloud; some must stay at the edge. → Fix: Design hybrid strategies from day one.

As with any paradigm shift, knowing the pitfalls upfront helps you build smarter.

Conclusion

Let’s be real: edge computing isn’t just hype—it’s happening. By crunching data closer to where it originates, it brings blazing speed, improved reliability, and opens the door to the kind of apps we’ve only daydreamed about.

For developers, this means opportunities everywhere: from IoT gadgets to autonomous vehicles to smart city systems. But it also means you’ll need to adapt—sharpen your distributed systems mindset, get comfortable with hybrid architectures, and start experimenting with edge frameworks.

Next step? Don’t just read about it. Spin up a Raspberry Pi, push some logic to the edge, deploy a tiny ML model, or mess around with an IoT sensor. The future of computing is already here—waiting at the edge.

Understanding OCI Infrastructure: Regions, Availability Domains, and Fault Domains

Fri, 22 Aug 2025 20:29:15 +0000

Oracle Cloud Infrastructure (OCI) Physical Architecture: A Complete Guide

Oracle Cloud Infrastructure (OCI) is built on a robust physical architecture designed to deliver high availability, fault tolerance, and optimal performance. Understanding this architecture is crucial for designing resilient cloud solutions. Let's dive into the key components that make up OCI's infrastructure.

Core Infrastructure Components

OCI's physical architecture is organized into three main levels:

Regions - Geographic locations
Availability Domains (ADs) - Data centers within regions
Fault Domains (FDs) - Hardware groupings within availability domains

Regions: Your Geographic Foundation

A region is a localized geographical area where Oracle has deployed OCI infrastructure. Think of regions as Oracle's data center hubs strategically placed around the world. For example, Phoenix serves as a major region on the US West Coast.

As of August 2024, Oracle Cloud Infrastructure (OCI) boasts more than 45 active regions and over 100 Availability Domains globally, making it comparable to major cloud providers like AWS and Microsoft Azure.

Choosing the Right Region

The golden rule for region selection is simple: choose the region closest to your users. This ensures:

Lowest network latency
Best application performance
Optimal user experience

Data sovereignty and compliance requirements may also influence your region choice, especially for regulated industries.

Availability Domains: Isolated Data Centers

An Availability Domain (AD) consists of one or more fault-tolerant data centers located within a single OCI region. Here's what makes them special:

Key Characteristics

Availability domains are isolated from each other, fault tolerant, and very unlikely to fail simultaneously
Connected by low-latency, high-bandwidth networks
No shared physical infrastructure (power, cooling, or internal networks)
Traffic between availability domains and between regions is encrypted

Availability Domain Distribution

Every Region can have up to 3 Availability Domains
Some regions may have fewer ADs depending on their size and scope
Each AD operates independently to prevent cascading failures

Fault Domains: Hardware-Level Resilience

Fault Domains (FDs) represent the finest level of isolation in OCI's architecture. They're groupings of hardware and infrastructure within an availability domain.

Fault Domain Structure

Each region has at least three fault domains
Every availability domain contains exactly 3 fault domains
Software deployments across multiple fault domains are staggered

Protection Benefits

Fault domains provide anti-affinity protection against:

Unexpected hardware failures - Server crashes, network issues, storage problems
Planned maintenance - Scheduled updates and infrastructure work
Power and cooling issues - Localized infrastructure problems

Oracle Maximum Availability Architecture (MAA)

Oracle Maximum Availability Architecture (MAA) is a set of best practices developed by Oracle engineers over many years for the integrated use of Oracle High Availability, data protection, and disaster recovery technologies.

MAA Components

The MAA blueprint includes these critical technologies:

Oracle Flashback - optimizes logical failure repair
Backup and Recovery - Comprehensive data protection strategies
Multitenancy - Efficient resource sharing and isolation
Data Guard - Database disaster recovery and high availability
Real Application Clusters (RAC) - allow customers to run a single Oracle Database across multiple servers to maximize availability and enable horizontal scalability
Application Continuity - Seamless failover for database connections

Integrated Security

MAA implements a defense-in-depth security approach covering:

Data encryption and protection
Identity and access management
Network security controls
Compute infrastructure security
OCI tenancy-level protections

Organizational Structure: Tenancy and Compartments

Tenancy: Your OCI Account

Each OCI account receives one tenancy (root compartment)
Think of tenancy as your organization's master container
Provides the highest level of resource organization

Compartments: Logical Organization

Compartments are logical containers that help you organize and isolate cloud resources:

Key Features

Can be nested up to 6 levels deep
Inherit policies from parent compartments
Tenancy-wide availability - accessible across all subscribed regions
Resource mobility - most resources can be moved between compartments

Best Practices

Use compartments to separate environments (dev, test, prod)
Align compartments with your organizational structure
Implement least-privilege access through compartment-based policies

Virtual Cloud Network (VCN): Your Software-Defined Network

A Virtual Cloud Network (VCN) is your private software-defined network in OCI, functioning like a traditional physical network but with cloud flexibility.

VCN Characteristics

Regional scope - Each VCN exists within a single OCI region
CIDR support - Supports both IPv4 and IPv6 addressing
Complete networking stack - Includes subnets, route tables, and security lists

Subnet Types

Public Subnets (Internet Gateway)

Public subnets enable direct internet connectivity:

Allow ingress traffic from the internet
Allow egress traffic to the internet
Support public IP addresses for resources
Ideal for web servers, load balancers, and public-facing services

Private Subnets (NAT Gateway)

Private subnets provide secure, isolated environments:

No direct ingress from the internet
Controlled egress through NAT Gateway
No public IP addresses assigned
Perfect for databases, application servers, and sensitive workloads

Networking Components

Route Tables - Define traffic routing rules
Security Lists - Act as subnet-level firewalls
Gateways - Provide connectivity options (Internet, NAT, Service)

Identity and Access Management (IAM) Policies

IAM policies form the backbone of OCI security, controlling who can access what resources and when.

Policy Hierarchy

Fleet administrators can set IAM policies at multiple levels:

Tenancy level - Organization-wide policies
Compartment level - Environment or project-specific access
User level - Individual user permissions
Group level - Role-based access control

Policy Capabilities

IAM policies enable you to:

Create, manage, and delete cloud resources
Control access to specific services and features
Implement compliance requirements
Audit and monitor resource usage

Best Practices

Follow the principle of least privilege
Use groups instead of individual user assignments
Regularly review and audit policy assignments
Implement separation of duties for critical operations

Conclusion

Oracle Cloud Infrastructure's physical architecture provides a solid foundation for building highly available, scalable, and secure cloud applications. By understanding regions, availability domains, and fault domains, you can design resilient architectures that withstand various failure scenarios.

The combination of MAA best practices, flexible compartment organization, robust VCN networking, and comprehensive IAM policies gives you the tools needed to build enterprise-grade cloud solutions.

Whether you're migrating existing workloads or building new cloud-native applications, OCI's architecture provides the reliability and performance your business demands.

Ready to start your OCI journey? Consider your region selection carefully, plan your compartment strategy, and always design for high availability across multiple fault domains.

Kubernetes Predictions Were Wrong

Tue, 08 Jul 2025 08:56:45 +0000

Why has the solution to the Kubernetes complexity problem proven so elusive?

In 2020, people were predicting that Kubernetes would disappear within a year. They believed someone would create a service that would reduce the adjacent choices and make Kubernetes the easy default. Everyone would use Kubernetes, but few people would work at the nuts-and-bolts level.

I checked this morning, and it’s still here — and so are the nuts and bolts. So, why has the solution to the Kubernetes complexity problem proven so elusive?

An Ecosystem of Choices

In isolation, Kubernetes isn’t particularly complicated, but a broad ecosystem of tools, options and decisions surrounds it. You can run Kubernetes anywhere, from a local machine to the cloud. You can run on a single low-powered machine or thousands of hyperscale nodes. You can use it to run one monolith or thousands of microservices.

There isn’t one opinionated way to use Kubernetes. It’s designed to be highly flexible. That means you get to make all the decisions. You choose exactly how to configure Kubernetes and which tools to use alongside it.

It’s the sheer volume of decisions you have to make to get started that people are talking about when they say things are hard, not the complexity of Kubernetes itself.

And this is before you consider the software itself. If you’ve just written a microservice in Go that runs in a container, it’s pretty easy to run it in Kubernetes. If you have a heritage monolith written under the assumption it would run on a virtual machine, there’s a great deal of work to bend it to work in a container.

It’s no wonder a developer responsible for delivering software that solves business problems reaches overload when faced with so many options along the way, especially when seen in aggregate across the different choices to be made for containers, infrastructure automation and container registries, with the various file formats and conventions used.

If you want to cycle to work, selecting a bicycle is a research-heavy task, so imagine buying a frame, wheels, handlebars, brakes and gearing separately. Some riders will want this level of control, but it’s overwhelming when your job is just getting to a destination.

Cognitive Overload and Platform Teams

The view that Kubernetes would settle into quiet utility and effectively disappear while also running all our workloads failed to materialize. Nobody managed to create a single opinionated path for Kubernetes that would take care of all these choices.

The simple reason for this is that the mythical one true way wouldn’t work for most applications and services. It’s impossible to create a single simple path without acknowledging the context of the application and organization.

This is why platform engineering has gained traction. While there’s little chance of creating an industrywide path of simplified choices, creating one within an organization is perfectly feasible.

A minimal viable platform could be a wiki page listing pre-baked decisions and providing a standard example for each configuration file. This might evolve into a facade that allows developers to specify what they need along a simple dimension, such as “size,” with the platform taking care of the details behind the flag.

Platforms should provide simplified ways to do the right thing while letting expert developers peel back the layers when the standard approach isn’t suitable.

Fewer Options Without Limiting Teams

The lack of a single opinionated path for Kubernetes underscores the importance of contextual factors for the application and organization. While creating a universal, simplified approach is impractical, platform engineering is one way to reduce the many decisions that need to be made along the way, especially when such decisions aren’t crucial.

Despite the perceived complexity issues, Kubernetes has done nothing but grow. Clearly, the benefits outweigh the adoption pain. So, imagine what would happen if Kubernetes and the associated moving parts were easier to adopt?

Platform teams could be the crucial factor in creating an opinionated Kubernetes setup that still respects the local context. Perhaps the predictions aren’t wrong, just delayed.

Power Up Your App with Server-Sent Push Notifications

Sun, 08 Jun 2025 05:36:54 +0000

You will need a server-side implementation to send push notifications. Here’s an example using Node.js with the web-push library.

Install `web-push`

npm install web-push

Server Code Example

server.js

const webPush = require('web-push');

// Replace with your own VAPID keys
const vapidKeys = webPush.generateVAPIDKeys();

webPush.setVapidDetails(
  'mailto:*Emails are not allowed*',
  vapidKeys.publicKey,
  vapidKeys.privateKey
);

// Example subscription object (should be obtained from the client)
const subscription = {
  endpoint: 'USER_SUBSCRIPTION_ENDPOINT',
  keys: {
    auth: 'USER_AUTH_KEY',
    p256dh: 'USER_P256DH_KEY',
  },
};

// Send a push notification
const payload = JSON.stringify({
  title: 'Hello!',
  body: 'This is a push notification from Angular!',
});

webPush.sendNotification(subscription, payload)
  .then(response => console.log('Notification sent successfully:', response))
  .catch(error => console.error('Error sending notification:', error));

Running the Application

Serve your Angular application:

   ng serve --prod

Open your browser and navigate to your application.
Accept the notification permissions.
Trigger a push notification from your server.

Summary

Setting up Angular service workers for push notifications.
Requesting notification permissions in the Angular app.
Sending push notifications from a Node.js server.

Make sure to replace YOUR_PUBLIC_VAPID_KEY with your actual VAPID key from your server-side implementation.

To see the frontend implementation, check out this post.
https://coderlegion.com/3273/leveraging-service-workers-to-enhance-web-app-performance-and-reliability

How to login to your remote server like Digital Ocean using SSH key on window

Thu, 22 May 2025 12:25:58 +0000

SSH stands for Secure Shell. it is a cryptographic network protocol used to securely assess and manage remote computers over the internet.
There are several use cases of SSH, which include:
a.Secure data transfer
b.Passwordless login
c.Secure file transfer
d.Remote login
e.Port forwarding
This article is focused on how to login to your remote server using ssh key on windows, so e will not be talking about the other use case of ssh.
To log in to your remote server using Windows, you need to be sure your Windows has openSSH installed. To confirm this,
•Go to the window search bar and type “PowerShell”, this will bring it out, click it to open the PowerShell
•In the PowerShell, type ‘ssh’ and click the Enter key on your keyboard
You are expected to see the usage instructions like this

`PS C:\Users\USER> ssh
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b bind_address]

       [-c cipher_spec] [-D [bind_address:]port] [-E log_file]
       [-e escape_char] [-F configfile] [-I pkcs11] [-i identity_file]
       [-J destination] [-L address] [-l login_name] [-m mac_spec]
       [-O ctl_cmd] [-o option] [-P tag] [-p port] [-Q query_option]
       [-R address] [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
       destination [command [argument ...]]`

If you did not see this, then you will have to install OpenSSH, which can be found in the settings of your Windows OS.
•The next step is to create the SSH key on the PowerShell by typing the command ‘ssh-keygen’
This will prompt you to input the file name you want to save the key into; if you do not input this, it will save the key into a default file name.
You will also be prompted to type in a password twice, which will act as the passphrase for the key
After this, you will see a message like this:

Your identification has been saved in ozoSSh Your public key has been saved in ozoSSh.pub The key fingerprint is: SHA256:SRxDqca9xn0U64HewqfjHcuKJSw0llIhVDPIa0oj+ms user@DESKTOP-EB25MV0 The key's randomart image is: +--[ED25519 256]--+ | ooo=.+. | | o. =.o . | | o.oo . o | |. o o.+o... + | |.o +..=.S= + . | |. . + o+ = = | | . ..o .=. | | E . +oo o | | ... ..oo+ | +----[SHA256]-----+

This signifies that the keys have been created.
•Type the ‘ls’ command to display the files in that directory. Search for the file you saved the key in, you will see two files with the same file name, just that one of them has ‘.pub’ attached to it as a suffix
•Type “cat root@”
This will prompt you to type in your passphrase used to set up the SSH key before you are logged in.
The logic here is that as the SSH key pair is created, the public key is saved in your remote server; you need the corresponding private key, which the server will compare with that of the public key in order to grant you access to the remote server. If, for any reason, the private key does not match the public key, you will be denied access to the server.

How Kubernetes Simplifies Cloud Application Deployment, Scaling, and Management

Fri, 14 Mar 2025 04:29:22 +0000

As cloud-native applications continue to gain momentum, the need for efficient, scalable, and automated management of containers becomes more essential. Kubernetes has emerged as the industry-standard platform for automating the deployment, scaling, and management of containerized applications across various cloud environments. This article dives deep into how Kubernetes transforms the way organizations handle their cloud application lifecycle, providing advanced features such as auto-scaling, self-healing, and high availability.

What is Kubernetes?

Kubernetes, often abbreviated as K8s, is an open-source container orchestration platform designed to automate the deployment, scaling, and management of containerized applications. Originally developed by Google, Kubernetes has become the go-to solution for organizations looking to streamline their cloud infrastructure management. By abstracting away the complexity of infrastructure management, Kubernetes allows developers to focus on application code while leaving operational concerns to the platform.

In a modern cloud environment, Kubernetes enables seamless container management across on-premises data centers, public clouds, and hybrid environments. This abstraction makes it possible to run applications in a consistent manner, regardless of the underlying hardware or cloud provider.

Automating Application Deployment with Kubernetes

One of the most powerful features of Kubernetes is its ability to automate the deployment of applications. Traditionally, deploying an application to a cloud environment involves manually provisioning servers, configuring networking, and installing dependencies. Kubernetes eliminates much of this manual effort by using containers as the foundation for deploying applications. Containers encapsulate an application and its dependencies, ensuring that the application runs consistently across various environments, from local development to production in the cloud.

In Kubernetes, deployments are defined using declarative configuration files, typically written in YAML or JSON. These configuration files describe the desired state of the application, including the number of replicas, storage requirements, and networking policies. Kubernetes then works to ensure that the application is deployed according to the defined specifications, automatically handling the provisioning of resources and setting up the necessary services to make the application accessible to users.

The declarative nature of Kubernetes means that developers can version control their application’s infrastructure. If an update is made to the configuration, Kubernetes automatically adjusts the system to match the desired state, whether that’s adding more replicas or rolling back to a previous version. This results in faster, more reliable deployments.

Scaling Applications Automatically with Kubernetes

Scaling is one of the key challenges faced when managing cloud applications, especially as the number of users fluctuates. Kubernetes excels in automatic scaling, providing both horizontal and vertical scaling mechanisms for applications. Horizontal scaling is the process of increasing or decreasing the number of application instances (pods) to match the demand. Kubernetes can scale your application based on a variety of metrics, such as CPU usage, memory usage, or custom metrics that you define.

The Kubernetes Horizontal Pod Autoscaler (HPA) is a built-in feature that automatically adjusts the number of pods running in a deployment based on real-time metrics. For example, if the CPU usage of a pod exceeds a threshold, the HPA can spin up additional replicas to distribute the load. Conversely, when demand decreases, Kubernetes will scale down the number of replicas to save resources. This elasticity ensures that resources are used efficiently, while maintaining optimal performance under varying traffic loads.

In addition to HPA, Kubernetes also supports **Cluster Autoscaling**. This feature allows Kubernetes to automatically scale the underlying infrastructure by adding or removing nodes (virtual machines) in response to resource demands. This is particularly useful when running workloads in a cloud environment, where infrastructure scaling can be automated based on predefined policies. Kubernetes can add more nodes when there are too many pending pods or remove nodes when there is excess capacity.

Self-Healing and Fault Tolerance in Kubernetes

Cloud applications need to be highly available and resilient to failures. Kubernetes helps ensure that applications are self-healing and fault-tolerant by continuously monitoring the health of deployed services. When an application or service experiences an issue, Kubernetes automatically takes corrective action to restore the desired state.

For example, if a pod crashes due to an error or becomes unresponsive, Kubernetes will automatically restart it or replace it with a new pod. This process is transparent to users, who experience minimal disruptions. Additionally, Kubernetes supports the concept of “replicas” where multiple instances of an application are run simultaneously across different nodes, ensuring that the application remains available even if one of the instances fails.

Kubernetes also performs regular health checks using readiness and liveness probes. A readiness probe checks if a container is ready to accept traffic, while a liveness probe checks if the container is still running. If a liveness probe fails, Kubernetes will automatically restart the container, ensuring that unhealthy services are quickly replaced with healthy ones.

Service Discovery and Load Balancing in Kubernetes

In modern cloud applications, services often need to communicate with one another. Kubernetes simplifies service discovery by providing an internal DNS-based service discovery system. Every service within a Kubernetes cluster is automatically assigned a DNS name, allowing other services to connect to it by using its service name, rather than hardcoding IP addresses. This dynamic service discovery makes it easy to update or replace services without disrupting the entire application.

Kubernetes also provides built-in load balancing. When users or other services send traffic to an application, Kubernetes automatically distributes that traffic across all the healthy instances (pods) of the service, ensuring that no single instance is overloaded. This load balancing mechanism helps achieve high availability and ensures that resources are used effectively, providing a smooth user experience even under heavy traffic conditions.

Configuring and Managing Cloud Environments with Kubernetes

Managing different environments in the cloud can be a complex task, especially as the number of services and applications grows. Kubernetes simplifies this by using namespaces, which allow you to isolate resources within a cluster. This makes it easy to manage multiple environments, such as development, staging, and production, all within a single cluster. Each environment can have its own set of resources, configurations, and access controls, while still sharing the underlying infrastructure.

Additionally, Kubernetes provides the ability to manage configuration data and sensitive information using ConfigMaps and Secrets. ConfigMaps store non-sensitive configuration data, such as environment variables or configuration files, while Secrets store sensitive data, such as passwords or API keys. These resources can be injected into applications at runtime, making it easier to manage environment-specific configurations without having to modify application code.

Integrating Kubernetes with CI/CD Pipelines

Continuous integration and continuous deployment (CI/CD) practices are critical for modern software development. Kubernetes integrates seamlessly with CI/CD pipelines, enabling automated deployment of containerized applications as new code is pushed to version control systems.

By integrating Kubernetes with CI/CD tools like Jenkins, GitLab CI, or CircleCI, developers can create automated workflows that deploy new versions of an application to the Kubernetes cluster whenever new changes are made. This enables faster feedback loops, improved code quality, and more frequent releases. Kubernetes helps automate the entire process, from building containers to deploying them to production, ensuring consistency and reliability in application delivery.

Conclusion

Kubernetes is a powerful tool for automating the deployment, scaling, and management of containerized applications in the cloud. By leveraging its robust features, such as self-healing, auto-scaling, service discovery, and CI/CD integration, organizations can achieve greater efficiency, reliability, and flexibility in their cloud environments. Kubernetes empowers developers to focus on building applications while leaving the complexities of infrastructure management to the platform.

Whether you're deploying simple microservices or large-scale cloud applications, Kubernetes helps ensure that your applications are always available, scalable, and resilient, providing a smooth and reliable experience for both developers and users alike.

Important Concept In Cloud Computing.

Wed, 12 Mar 2025 09:06:32 +0000

Important Concept In Cloud Computing

What is Cloud Computing
Cloud computing is providing computing services which include servers, storage, software, networking over the internet to provide faster innovation and flexible resources.
Instead of storing files on a storage device or hard drive, a user can save them on the cloud making it possible to access the files from anywhere as long as there is access to the internet.

Models of Cloud Computing

Public cloud: this cloud computing is open to the general public and is owned by a third party provider.

Private cloud: this cloud computing is provisioned and managed within an organizations premises.

Hybrid cloud: this cloud computing combines public and private cloud services which allows data and application to be shared between them.

What is virtualization
Virtualization in cloud computing is a technology that allows multiple virtual machines (VMs) to run on a single physical machine or host by simulating hardware components like processors, memory and storage.

Types of Virtualization

Server Virtualization: this is also known as hardware virtualization and is most common type of virtualization in cloud computing. It can also partition physical server into multiple virtual machines.

Desktop virtualization: it allows users simulate a workstaion load to access a desktop from a connected device.

Network virtualization: can be assigned to a specific device or server and can also combine network resources into separate channels.

What is scalability

Scalability in cloud computing is the ability of a cloud system to increase or decrease its resources and capacity to match changing business needs. Cloud scalability can either be flexible (it allows users to quickly add or remove resources as needed, providing adaptability to fluctuating workloads) or cost efficiency (it help optimize costs by only paying for the resource actually used).

What is Agility

Agility in cloud computing is the ability of an organization to quickly respond to changing business needs, opportunities by rapidly deploying, scaling and managing cloud based resources and applications. It also means being able to respond fast to new challenges through the use of cloud technology.

Services that Enables Agility

Infrastructure as a service(IaaS) provides scalable infrastructure resources such as servers, storage, and networking.

Platform as a Service(PaaS) offers scalable platform for developing, running and managing applications.

Software as a Service(SaaS) provides scalable software applications with automatic updates and maintenance.

What is High Availability

High availability in cloud computing refers to the ability of a cloud system to ensure that applications, services and data are accessible and usable at all times with minimal downtime or disruption. It aims to maintain a high level of uptime by distributing resources across multiple locations and automatically switching to backups in case of failures.

Benefits of High Availability in Cloud Computing

Improves user experience: users can access applications and services at all times with minimal disruption.

Increase productivity: businesses and companies can maintain operations and productivity even in the event of hardware or software failures.

Enhanced reputation: high availability demonstrates a commitment to reliability and customer satisfaction.

What is Fault Tolerant

Fault tolerance in cloud computing is the ability of a cloud system to detect, recover from and continue operating despite hardware or software failures or disruption. Essentially the system can withstand failures and keep running smoothly.
High availability, reduce downtime, improve reliability and increase customer satisfaction are all benefits of fault tolerance in cloud computing.

What is Global Reach

Global reach in cloud computing refers to the ability of a cloud provider to deliver cloud services and applications to users anywhere in the world regardless of geographical location. Essentially, it means a cloud service can be accessed from anywhere with an internet connection.

Key Characteristics of Global Reach in Cloud Computing

Worldwide availability: cloud service are available to users in multiple regions and countries.
High availability: cloud services are designed to be always on with minimal downtime or disruption.

Scalability: cloud services can scale to meet the needs of a global user base.

Low latency: cloud services are delivered with minimal delay ensuring fast and responsive performance.

What is the Difference Between Elasticity and Scalability

In cloud computing, elasticity means the ability of a cloud system to automatically adjust its resources to match changing workload demands without manual intervention while scalability means the capacity to increase or decrease resources (e. g, computer power, storage or networking) to match changing workload demands.

Securing ASGI Server with AWS Cognito

Thu, 20 Feb 2025 19:02:49 +0000

Introduction

I published an article on hash node a couple weeks ago titled: [Deploying Complete Serverless API with any containerised Python ASGI Framework on AWS Lambda, API][1]

I figured I needed to add a logical next step to securing the api. I was also partly inspired by the course by Steven Marek towards AWS Solutions Architect Certification, which I'm studying.

/src_code

The GitHub code repo for this project is here: https://github.com/652-Animashaun/blog-serverless.git

Its written in Python - FastAPI, deployed to AWS Lambda, then put an API Gateway in front to route client requests to appropriate endpoints. That's it you're all caught up.

While code example used here is Fast API framework, I implore you the implementation discussed is python framework agnostic. Come on I'll show you.

What is AWS Cognito?

I must admit the AWS Cognito documentation gave me the run around. I spent way more time than I thought I should implementing this mechanism. I'm not all that new to JWT authentication. I've implemented mechanisms using the OAuth before.
Anyways I’ve done the work, gone through the belly of the beast that is the AWS documentation, so that you won’t have to. I’ll try to cut out the fluff.

Amazon Cognito is an identity platform for web and mobile apps. It’s a
user directory, an authentication server, and an authorization service
for OAuth 2.0 access tokens and AWS credentials. With Amazon Cognito,
you can authenticate and authorize users from the built-in user
directory, from your enterprise directory, and from consumer identity
providers like Google and Facebook.

So off the bat, we can already see several use cases for the AWS Cognito service. For now we want a simple authentication process, i.e user is served a login page to enter credentials, redirected to homepage on successful login, or presented a register page to create an account get OTP via provided email to verify ID. They get a token (JWT) which is presented to the server’s gate keeper for some resource access, in this case an API endpoint.
What is happening under the hood is simple enough too. Well relatively, if you compare to implementing the mechanism from scratch.

User authentication with Cognito user pools

Login & Register Pages - Cognito

While you can build and customise to your liking, but AWS Cognito comes with a ready-to-use login and register pages where you can just redirect users to, you’ll also provide a redirect url where the returned token can be processed to authorize user.

Sending & Verifying OTP - Cognito

Usually one would be inclined to use a service like SES or SNS on AWS or Twilio, but you’ll have to manually call or implement the mechanism from the server side. AWS cognito takes care of this out of the box.

Encoding & Signing Tokens - Cognito

Cognito also takes care of encoding tokens. RS256 or HS256 base64encoding, also add features like expiry, a refresh token if you want. A Json Web Token contains 3 parts header, payload and a signature. Payload is where the user identity like email, username are returned. Cognito returns 3 tokens, the access_token, id_token and a refresh_token, more about that later.

Decoding and Granting Access - Server

When a token is returned to the server, the receiver of the JWT verifies the signature using the secret key or the public key. And maybe go a step further and include it in request header so that subsequent requests can bear the authorisation token.

Look, point is Cognito takes a lot of manual implementation out of the developers hand. I’m sold!.

Let's dive right into the AWS Cognito dashboard create a user pool in 3 steps.

Creating and configuring a user pool

Select Create user pool from the User pools menu, or select Get started for free in less than five minutes.

Under Define your application, I’m choosing “traditional web application” you can choose the Application type that best fits the application scenario that you want to create authentication and authorization services for.

In Name your application, enter a descriptive name or proceed with the default name.

You must make some basic choices under Configure options that support settings that you can't change after you create your user pool. For this use case I’m selecting username, email and full name.

Add a return url on the server that handles payload receipts and process the payload. in this case mine is http://localhost:8000/api/v1/users/token for development and testing purposes. You can add more redirect url options. Once you're ready to update lambda function with local changes, we'll be sure to add another return url , this time with the API Gateway stage endpoint. Else we get some funny error client_id redirect_uri mismatch. Hit create.

Web App Server

Lets take a look in the user pool thats just been created:

Under recommendations you should go ahead and click “setup your web app”.

AWS was kind enough to show some code examples for different languages.

Selecting python will give you a quick preview how things would look like:

As shown, the framework in Cognitgo example is Flask, but it doesnt matter, that about how we’ll be implementing ours. I can already see that to start with I’ll need a /login endpoints and an /authorize endpoints at a minimum.

/app.api.enpoints.users.py

import os
import jwt
from pprint import pprint
from fastapi import APIRouter, HTTPException, status, Depends
from starlette.requests import Request
from app.utils.auth import oauth, verify_token
from starlette.responses import RedirectResponse

router = APIRouter()

@router.get("/login")
async def login(request: Request):
    return await oauth.oidc.authorize_redirect(request, "http://localhost:8000/api/v1/users/token")

@router.get("/token")
async def authorize(request: Request):
    payload = await oauth.oidc.authorize_access_token(request)
    token = payload["id_token"]
    res = await verify_token(token)
    user = payload["userinfo"]
    return token

These endpoints aren’t doing much heavy lifting. /login simply calls on oidc.authorize_redirect() provided by the oauth import from utils where OAuth was instanced and passed to oauth, more on that subsequently. It takes a request and a redirect url to process response.

async def authorize(request) will receive a get request on signing completion on cognito, with a json payload that looks like:

{ "id_token": "eyJraWQiOiJ3ekxUT1lrNVR....", "access_token": "eyJraWQiOiI3b0pqeFhY...", "refresh_token": "eyJjdHkiOiJKV1QiLC....", "expires_in": 3600, "token_type": "Bearer", "expires_at": 1738976151, "userinfo": { "at_hash": "aYWSJq...", "sub": "b4d8e428-f091-7040-80a5-f2", "email_verified": true, "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_ftTN1aEr4", "cognito:username": "shaun", "nonce": "SYqZdIip5wBjjI10983", "origin_jti": "edc6ytb3c-4870-4979-8672-7d9a261ff471", "aud": "47re0a7e8l43ok719rtqcqnjih", "token_use": "id", "auth_time": 1738972551, "exp": 1738976151, "iat": 1738972551, "jti": "f20cf2a4-5c3f-44f9-8d0b-919816d41f38", "email": "*Emails are not allowed*" } }

As seen above we can already grab userinfo and quickly see the user ID. But theoretically some skilled man in the middle can provide their own ID, I dont know how it's done, the main thing is it is possible. And I like to preempt that, I want to do a quick verify.

import os
import jwt
from jwt import PyJWKClient
from authlib.integrations.starlette_client import OAuth
from fastapi import APIRouter, HTTPException, status, Depends, Security, HTTPException
from datetime import date, datetime, time, timedelta, timezone
from typing import Annotated
from app.models.users import User
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials, OAuth2PasswordBearer
from starlette.responses import RedirectResponse
from starlette.requests import Request


signin_url = 'https://cognito-idp.us-east-1.amazonaws.com/us-east-1_ftTN1aEr4/.well-known/jwks.json'
security = HTTPBearer(auto_error=False)

CLIENT_SECRET = os.environ["CLIENT_SECRET"]
CLIENT_ID = os.environ["CLIENT_ID"]
ISSUER = "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_ftTN1aEr4"


oauth = OAuth()
oauth.register(
  name='oidc',
  authority=ISSUER,
  client_id=CLIENT_ID,
  client_secret=CLIENT_SECRET,
  server_metadata_url='https://cognito-idp.us-east-1.amazonaws.com/us-east-1_ftTN1aEr4/.well-known/openid-configuration',
  client_kwargs={'scope': 'email openid phone'}
)

async def verify_token(token:str)-> User:
jwks_client = PyJWKClient(signin_url)

try:
signing_key = jwks_client.get_signing_key_from_jwt(token)
payload = jwt.decode(
token,
signing_key,
audience=CLIENT_ID,
options={"verify_exp": True},
algorithms=["RS256"],
)
user = User(username=payload["cognito:username"], email=payload["email"], access_token=token)
return User
except jwt.ExpiredSignatureError:
raise HTTPException(status_code=401, detail="Token has expired")
except jwt.InvalidTokenError as e:
raise HTTPException(status_code=403, detail="Invalid token")

async def get_current_user(credentials: Annotated[HTTPAuthorizationCredentials, Security(security)]):
if not credentials:
return False
try:
return await verify_token(credentials.credentials)
except HTTPException:
return False

Lets do a quick run through the code:

The singing_url is provided by congnito in the userpool dashboard, this an endpoint provided by Cognito. At the back of it is a json_body of keys used to sign the token. More on that subsequently.

In FastApi using Oauth2 as dependency is through the Security() class provided. You can read more about the stuff here. TLDR fastapi.security.HTTPBearer is one of the tools provided to use called as a dependable during visit of protect endpoints. It’s use case is for bearer token authentication. auto_error=False because as per docs, by default, if the HTTP Bearer token is not provided (in an Authorization header), it'll automatically cancel the request and send the client an error. I don’t want that, I want to catch that and redirect to login so user can try challenge again.

HTTPBearer returns an object HTTPAuthorizationCredentials which I believe is received via the request header and provides it to the application.

Then we just shamelessly copied the boiler plate code on cognito recommendation page to instantiate Oauth.register().

async def verify_token() will take in a token, first it want’s to retrieve signing keys from the signing_url using the python module PyJWKClient class provided by JWT module. Next it calls jwt.decode(), takes the token, signing_key and a couple other keyword arguments. The decoded payload contains a dictionary with several user identifying information, but it’ll go ahead, returning username and email.

In async def get_current_user() where the first injection takes place. I’ve provided the HTTPAuthorizationCredentials as HTTPBearer security which would try to retrieve the automaticall from the incoming request header the Bearer Authorization this is where the token is expected to be for an authenticated request.

/app.api.endpoints.posts.py

To show the authentication mechanism actually works we have to create an endpoint that is protected:

from fastapi import APIRouter, Depends, HTTPException, status
from typing_extensions import Annotated
from app.utils.auth import oauth, verify_token, get_current_user
from app.models.users import User
from starlette.requests import Request
from starlette.responses import RedirectResponse

router = APIRouter()

@router.get("/")
async def all_posts(current_user: Annotated[User, Depends(get_current_user)]):
    if not current_user:
        return RedirectResponse(url="/api/v1/users/login")
    print("current_user", current_user)
    return {"Welcome": "Shaun's Blog", "user": current_user}

Pretty straightforward right? async def all_posts depends on get_current_user which in turns returns a user object or None, in which case user will be redirected to /login.

Oh did I mention and AWS Cognito is free for 10K monthly active users. Check it out:

Amazon Cognito Essentials and Lite have a free tier. The free tier
does not automatically expire at the end of your 12-month AWS Free
Tier term, and it is available to both existing and new AWS customers
indefinitely. Please note - the free tier pricing isn’t available in
the AWS GovCloud (US-West) region.

For users who sign in directly via Amazon Cognito or through a social identity provider, Amazon Cognito user pools has a free tier of
10,000 monthly active user (MAU) per month per account or per AWS
organization. This free tier is applicable for customers that
configure their user pools to either the Lite or Essentials tier.
There is no free tier for the Plus tier.

Stick around next one up is integrating social ID providers like google social media accounts. Would be fun.

One more thing, remember when you push changes to lambda, add the api gateway endpoint version of your /token (authorise) to the list of return urls in the Cognito UserPool dashboard. This might be the cause of client_id redirect_uri mismatch error, watch out.

References:

https://docs.aws.amazon.com/cognito/latest/developerguide/getting-started-user-pools-application.html

https://pyjwt.readthedocs.io/en/stable/installation.html

Cloud Fundamentals

Mon, 17 Feb 2025 10:08:44 +0000

What is Cloud Computing

Cloud computing is a method of delivering computing services over the Internet. This enables faster computing, flexible resources, and easy scaling.

Cloud Models

Public Cloud

Private Cloud

Hybrid Cloud

1. Public Cloud

The public cloud is owned by cloud service providers. This cloud service provider provides resources and services to multiple users and organizations. These users or organizations access the network over the internet.

There are no deep technical skills required to operate the public cloud. As this is a public cloud there services are available to everyone(public).

No capital expenditure is required to scale up the resources when required.

Applications can be quickly provisioned and de-provisioned.

Users or Organizations pay only for what they use.

2. Private Cloud

The private cloud is owned by an organization. Organizations create a cloud environment in their datacenter.

The organization is responsible for everything from purchasing the hardware to the updates to networking and security. The users outside of the organization cannot access the data or services of the private cloud.

Hardware must be purchased for start-up and maintenance.

Organizations have complete control over resources and security.

Organizations are responsible for hardware maintenance and updates.

3. Hybrid Cloud

As you might guess, a Hybrid cloud is the combination of public and private cloud to allow applications to run in the most appropriate location.

The organization decides what resources to put in the public cloud and what resources to put in the private cloud.

Deep technical skills are required to maintain a private cloud and also ensure both clouds can operate together efficiently.

There is flexibility about what to be in public and what to be in a private cloud.

Provides the most flexibility to the organization.

Organizations determine where to run their applications.

Organizations control security, compliance, or legal requirements.

Benefits of Cloud

The following are the biggest reasons "Why customers are moving to the cloud?"

High availability

Fault tolerance

Scalability

Elasticity

Global reach

Customer latency capabilities

Agility

Predictive cost considerations

1. High availability

It is the availability to keep the resources up and running for long periods of time with very little downtime (negligible).

2. Fault tolerance

The ability to remain up and running, if a component or service has a failure and is no longer working. For example, some hardware failure in the machine.

3. Scalability

There are two types of scaling. They are: Horizontal Scaling and Vertical Scaling

Horizontal Scaling (also called scaling out) is adding resources or servers or virtual machines as needed.

Vertical Scaling (also called scaling up) is adding computing power, that is adding toexisting resources by adding more CPU, adding more memory.

4. Elasticity

The ability to automatically increase or decrease resources as needed.

5. Global reach

Accessing resources and adding resources around the world. like creating resources from anywhere to anywhere.

6. Customer latency capabilities

User can be as close to our customers as possible to reduce latency for accessingtheir resources.

Now with the cloud, with the global reach, we can put resources around the world. putting them as close to our customers as possible.

7. Agility

The ability to quickly react to workload demands. like a sudden spike in traffic, allocate more resources. like down in traffic, deallocating resources.

8. Predictive cost considerations

Almost every cloud service providers provide a calculator to calculate the estimated cost of the resources they what to use.

So no surprise in the bills.

Cloud Expenditure models

Capital Expenditure (CapEx)

Operational Expenditure (OpEx)

1. Capital Expenditure (CapEx)

The up-front sending of money on physical infrastructure. That is when we what to create our own datacenter, we have to get budgets approved, buy or rent a space, get the physical resources, and network them together to build our own datacenter.

The cost spent on Capital Expenditure reduces over time. As physical resources performance may reduce, may damage or become outdated.

In the cloud, we don't need to do that.

2. Operational Expenditure (OpEx)

We spend as we needed. The spending and billing of services or products as needed. The Expenses are deducted in the same year and we are paying monthly or yearly for those expenses.

Consumption based model

Cloud service providers operate on the consumption-based model, which means the end-users only pay for the resources that they use.
Think of it as our utility bill, we have a meter on it and we pay by what we actually used. Same thing for the cloud. We pay by what we actually use.

Advantages

Better cost prediction

Prices for individual resources and services are provided

Billing is based on actual usage

Cloud Services

Infrastructure-as-a-Service (IaaS)

Platform-as-a-Service (PaaS)

Software-as-a-Service (SaaS)

1. Infrastructure-as-a-Service (IaaS)

Build pay-as-you-go IT infrastructure by renting servers, virtual machines, storage, networks, and operating systems from a cloud provider.

IaaS is the most flexible cloud service. We can configure and manage hardware for our application.

This flexibility helps in optimizing the application as per our needs as we have full access to hardware resources.

User Manage: Data & Access, Application, Runtime, Operating System, Virtual Machines

Cloud Service Provider Manage: Compute, Networking, Storage

Example: Virtual Machines

2. Platform-as-a-Service (PaaS)

Provides an environment for building, testing, and deploying software applications without focusing on managing underlying hardware/ infrastructure.

They are responsible for managing servers, infrastructure, operating systems, Networks, Network firewalls, Network security, and even service configuration.

This makes organizations more efficient and productive.

Here we can focus on our application development rather than spending more time in hardware configuration, software configurations, etc.,

User Manage: Data & Access, Application

Cloud Service Provider Manage: Runtime, Operating System, Virtual Machines, Compute, Networking, Storage

Example: App services or containers

3. Software-as-a-Service (SaaS)

Users connect to and use cloud-based apps over the internet. for example Google Suite, Google Drive, OneDrive, Google Docs, Google Sheets, Google Slides, etc.,

It is basically like renting software monthly or yearly.

The Cloud service providers manage everything else, the Servers, Networks, Operating Systems, Physical buildings, Infrastructure, Development Tools, and even hosted applications and apps.

Users pay for the software they use on a subscription model.

User Manage: Data & Access

Cloud Service Provider Manage: Application, Runtime, Operating System, Virtual Machines, Compute, Networking, Storage

Example: Microsoft Teams, Microsoft 365, Google Suite, Google Drive, OneDrive, Google Docs, Google Sheets, Google Slides, etc.,

NOTE: WE CAN USE ANY COMBINATION OF THESE INFRASTRUCTURES

Linkedin: https://www.linkedin.com/in/khushi-dubey-6036a2305

Thank you!

Using Elastic Container Registry (ECR) as an image repository.

Mon, 03 Feb 2025 09:07:04 +0000

Introduction

This article discusses using Elastic Container Registry (ECR) as an image repository. AWS Elastic Container Registry is a fully managed container image registry service that Amazon Web Services (AWS) provides. It is used to store, manage, and deploy container images (Docker images) securely, making it an essential component of your containerized application development workflow.

Why Use AWS ECR?

Before diving into the setup, let’s understand why AWS ECR is a preferred choice over other container registries like Docker Hub, Google Container Registry (GCR), and Azure Container Registry (ACR).

Key Benefits of AWS ECR

✔️ Fully Managed – No need to set up and maintain your own container registry. AWS handles scaling, security, and maintenance.

✔️ High Security – Supports IAM-based access controls and AWS KMS encryption for protecting images.

✔️ Deep Integration – Works seamlessly with AWS services like ECS, EKS, Lambda, and CodePipeline.

✔️ Automated Image Scanning – Identifies vulnerabilities in container images.

✔️ Lifecycle Policies – Automatically deletes old or unused images to save storage costs.

Prerequisite

Create an AWS account
Install docker on your local computer
Install AWS CLI
Create an IAM user with an Access Key and Secret Access Key

Pushing a Docker Image to ECR

After creating the repository, follow these steps to push your Docker image to AWS ECR.

1. Authenticate Docker with AWS ECR

Run the following command to authenticate Docker with AWS ECR:

bash

aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <aws-account-id>.dkr.ecr.<your-region>.amazonaws.com

Replace <your-region> (e.g., us-east-1) and <aws-account-id> with your actual AWS credentials.

Note: If authentication fails, ensure your IAM user has the AmazonEC2ContainerRegistryFullAccess policy attached.

2. Build the Docker Image

Navigate to your application’s directory and build a Docker image:

bash

docker build -t my-app .

This command will create a Docker image with the tag my-app.

3. Tag the Image for AWS ECR

Tag the Docker image to match your AWS ECR repository URL:

bash

docker tag my-app:latest <aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/my-app-repo:latest

4. Push the Image to ECR

Finally, push the tagged image to the ECR repository:

bash

docker push <aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/my-app-repo:latest

Once completed, your image will be available in AWS ECR for deployment.

Setup

Log in to your AWS dashboard, search for ECR, and click on it
Create a repository -- note that the default repository for ECR is private.
Provide a name for the repository, and click Create.

Once the repository is created, click on the "view push command"
Follow the commands to push your image to the created repository.

Authenticate the docker client

Build the image

Tag the image

Push the image

Troubleshooting Common Issues

❌ Error: no basic auth credentials
Solution: Run aws ecr get-login-password and ensure your IAM user has AmazonEC2ContainerRegistryFullAccess.

❌ Error: AccessDeniedException
Solution: Verify that your IAM role has the correct ECR permissions (ecr:PutImage, ecr:GetAuthorizationToken).

❌ Docker Push Fails with 403 Error
Solution: Ensure your AWS region and repository URL are correct.

Conclusion

AWS Elastic Container Registry (ECR) provides a secure, scalable, and fully managed solution for storing and managing container images. By following this guide, you can:

✅ Set up an ECR repository

✅ Push and manage Docker images

✅ Integrate ECR with ECS, EKS, and AWS Lambda

✅ Automate deployments using AWS CodePipeline

With AWS ECR, managing containerized applications becomes seamless and efficient.

Differentiating Network Architecture: An Overview of 2-Tier and 3-Tier Network Architectures

Sun, 19 Jan 2025 10:01:32 +0000

Hi Everyone, Today, we will explore network architectures, how companies utilize them, and the significance of these architectures in today's digital landscape. We will also highlight some major companies that implement these architectures, discuss the differences and similarities between traditional and modern architectures, and delve into the various types of network architectures.

What is a Network?

A network is a collection of interconnected devices that communicate with each other to share resources and information. These devices can include computers, servers, switches, routers, and other hardware components.

What is Network Architecture?

Network architecture refers to the structural design of a network, defining how devices, systems, and protocols interact to facilitate data transfer. It acts as a blueprint for organizing components like switches, routers, and servers to ensure efficient communication. Effective network architecture is vital for:

Scalability: Supporting growing demands.
Performance: Ensuring low latency and high throughput.
Security: Protecting data during transmission.

The 2-Tier Architecture: A Simple Cafe Setup

Imagine walking into a cafe where everything is straightforward. You place your order at the counter (the client), and the person behind the counter(the server) prepares your drink. This is the essence of 2-Tier Architecture, which consists of two layers: the Access Layer (client tier) and the Core/Distribution Layer (server tier).

Two-tier architecture, or client-server architecture, divides an application into two layers: the Client Tier and the Server Tier. The client tier handles the user interface and sends requests, while the server tier processes these requests, executes business logic, and interacts with the database. Communication occurs over a network using protocols like HTTP or TCP/IP.

Advantages of 2-Tier Architecture

Simplicity: Easy to design, implement, and manage due to fewer components.
Cost-Effectiveness: Requires less hardware, ideal for small businesses.
Low Latency: Direct communication between clients and servers reduces delays.

Limitations of 2-Tier Architecture

Scalability Issues: Performance degrades as more clients connect to the server.
Single Point of Failure: If the server fails, all client requests are disrupted.
Limited Traffic Management: Lacks advanced features like VLAN routing or QoS.

The 3-Tier Architecture: A Corporate Office Setup

Imagine you entered a corporate office where tasks are divided among departments. You approach the front desk to make a request (the Access Layer), the management team processes your request and coordinates the work (the Distribution Layer), and all critical data is securely stored in the archive room (the Core Layer). This is the essence of 3-Tier Architecture, which consists of three layers: the Access Layer, the Distribution Layer, and the Core Layer.

The Three Layers of 3-Tier Architecture

Access Layer: This layer is the entry point for users to interact with the system, much like the front desk in an office. In networking, this layer connects end-user devices to the network using switches or wireless access points.
Distribution Layer: Acting as the management team, this layer processes requests received from the Access Layer and determines how to handle them. In networking terms, it uses multi-layer switches or routers to manage traffic and apply filtering or routing rules.
Core Layer: The Core Layer is like the archive room where all important data is stored securely. It handles high-speed data transfers and ensures reliable access to databases or storage systems.

Advantages of 3-Tier Architecture

Separation of Responsibilities: Each layer has a distinct role, making it easier to maintain and troubleshoot.
Scalability: Each layer can be scaled independently based on demand, ensuring smooth performance during high traffic.
Efficiency: The Distribution Layer optimizes traffic flow and reduces bottlenecks by managing requests intelligently.

Limitations of 3-Tier Architecture

Complexity: Designing and managing three separate layers requires expertise and careful planning.
Higher Costs: Additional infrastructure for each layer increases hardware and operational costs.

2-Tier vs 3-Tier

While 2-Tier Architecture directly connects clients to servers (or devices to a network), 3-Tier Architecture introduces a middle Distribution Layer that optimizes performance and scalability. For example:
In a corporate network, if many users in the Access Layer try to communicate directly with the Core Layer (as in 2-Tier), it can lead to congestion.
With 3-Tier Architecture, the Distribution Layer efficiently manages traffic between users (Access) and data storage (Core), ensuring smooth operations even during peak usage.

As we wrap with 2-tier and 3-tier architectures, it's important to highlight the growing adoption of a third model—Spine-Leaf Design. This architecture is particularly well-suited for data centers that manage high volumes of east-west traffic, which is becoming increasingly prevalent in today's digital landscape.
Currently, around 80% of companies are leveraging some form of this innovative architecture, enabling them to achieve greater scalability, reduced latency, and more efficient traffic management compared to traditional models. The Spine-Leaf architecture simplifies network design by utilizing two layers: the Spine layer, which acts as the backbone connecting all Leaf switches, and the Leaf layer, which connects directly to servers and endpoints.

Spline Leaf Design

Spine-Leaf Architecture is a modern two-layer network topology designed to optimize data center performance by addressing the limitations of traditional three-tier architectures. It consists of two main layers: the Spine Layer, which acts as the backbone of the network, and the Leaf Layer, which connects directly to servers and end-user devices. In this architecture, every leaf switch is interconnected with every spine switch in a full mesh topology, allowing for efficient east-west traffic flow—server-to-server communication within the same data center—while minimizing latency and reducing hop counts to a maximum of two. This design enhances scalability, as new spine or leaf switches can be added without disrupting existing connections, and improves redundancy by allowing multiple active links between switches, thus preventing bottlenecks. However, it does require more cabling due to the extensive interconnections and may have limitations on the number of hosts supported based on spine port counts. Overall, this architecture is increasingly favored in modern data centers for its ability to handle high volumes of traffic efficiently.

Conclusion

At the end, understanding 2-Tier, 3-Tier, and Spine-Leaf architectures is essential for designing efficient and scalable networks tailored to organizational needs. 2-Tier Architecture offers simplicity and cost-effectiveness, making it ideal for smaller setups, while 3-Tier Architecture provides enhanced scalability, security, and flexibility for larger enterprises. Meanwhile, the modern Spine-Leaf Architecture is revolutionizing data centers by efficiently handling high volumes of east-west traffic with reduced latency and improved scalability. Each model serves a unique purpose, and choosing the right architecture depends on factors like application size, traffic demands, and future growth.

Thank you for making it to the end! If you enjoyed the content, please consider sharing it with someone who might benefit from it.
Feel free to reach out to me on Twitter.

Thankyou Once Again!!

Check out this article for beginners on Kubernetes

Fri, 17 Jan 2025 19:34:28 +0000

Getting Started With Kubernetes: A beginner's guide

Introduction

In today’s world of software development, applications are no longer created as one single large program. Instead, they’re broken into smaller pieces or units called microservices, which run inside containers. But as you start running more containers, managing them can quickly become challenging. That’s where Kubernetes comes in.

This article introduces you to Kubernetes, covering what it is, explains why it’s essential, and how to get started with it!

Prerequisites

A basic understanding on Containers and Docker basics, for this, you can check out my article on Understanding Docker: A beginner's guide to containerization
Basic Command Line Knowledge, you should be comfortable using the terminal or command line interface (CLI)
YAML basics and that's because Kubernetes configurations are written in YAML.
Basic Networking Concepts. You should understand simple networking ideas, what an IP address is, and how ports work.
A Willingness to Learn. Kubernetes introduces some new concepts, like pods, services, and deployments so having patience and a mindset to experiment will make the learning process enjoyable.

Now, let's dive right in!

What Is Kubernetes?

Kubernetes, often abbreviated as K8s, is an open-source container orchestration platform developed by Google. It's designed to automate the deployment, scaling, and management of containerized applications. With Kubernetes, developers can focus on writing code, while the platform handles the complexities of deploying and managing those applications in production.

Why Use Kubernetes?

As applications become more complex and microservices-based architectures gain popularity, the need for efficient management tools has increased. Here are some reasons to consider using Kubernetes:

Scalability: Kubernetes provides powerful scaling options. You can easily scale your application up or down based on traffic or resource demand.
Load Balancing: It automatically distributes network traffic across multiple containers, ensuring no single container is overloaded.
Self-Healing: Kubernetes detects when a container fails and automatically restarts it or schedules it on a different node, ensuring high availability.
Service Discovery: It simplifies the process of managing communication between services, allowing them to find each other easily.
Seamless Updates: Kubernetes allows you to update your app with zero downtime.

Key Terms and Components in Kubernetes

When understanding Kubernetes, you're required to be familiar with these terms or core components. Here’s a breakdown of its key concepts:

1. Cluster

A Kubernetes cluster is a set of machines (called nodes) that run containerized applications. Each cluster has at least one master node and multiple worker nodes.

2. Node

A node is a physical or virtual machine in the cluster that runs applications. There are two types of nodes:

Master Node: Manages the cluster, coordinating tasks and distributing workloads to worker nodes.
Worker Node: Executes the applications and services.

3. Pod

A pod is the smallest deployable unit in Kubernetes. It can contain one or more containers that share storage and network resources. Pods are typically used to run tightly coupled applications.

4. Deployment

A deployment is a Kubernetes resource that manages the lifecycle of your application. It defines how many replicas (copies) of a pod should be running at any given time and ensures that the desired state of the application is maintained.

5. Service

A service is an abstract way to expose an application running on a set of pods. It provides a stable endpoint for users to access the application and can handle load balancing and failover.

6. ConfigMap and Secret

ConfigMap: Used to store non-sensitive configuration data for your application.
Secret: Used to store sensitive information, such as passwords or API keys, ensuring they are securely managed.

How Kubernetes Works

Let’s walk through an example:

You have a web app and want to run it in 3 containers.

You create a deployment in Kubernetes, specifying the app’s configuration and that you want 3 replicas.

Kubernetes starts 3 pods (each containing your app’s container) on the available worker nodes.

A service is created to expose your app to users.

If one pod fails, Kubernetes automatically starts a new one to maintain 3 replicas.

Getting Started With Kubernetes

There are various ways to install Kubernetes, depending on your environment and requirements. Here are a few popular options for beginners:

1. Minikube

Minikube is a tool that allows you to run Kubernetes locally on your machine. It’s ideal for development and testing.

To install Minikube:

Install VirtualBox or another compatible virtualization tool.
Install Minikube following the instructions on Minikube's official site.
Start your local Kubernetes cluster with minikube start.

2. Kubectl

Kubectl is the command-line interface for interacting with your Kubernetes cluster. You can install it by following the instructions on the Kubernetes documentation site.

3. Cloud Providers

If you want to use Kubernetes in production, consider cloud providers that offer managed Kubernetes services, like:

Google Kubernetes Engine (GKE)
Amazon Elastic Kubernetes Service (EKS)
Azure Kubernetes Service (AKS)

Deploying Your First Application

Now that you have a basic understanding of Kubernetes, let’s deploy a simple application. We’ll use a basic Nginx web server as an example.

Create a Deployment

Create a file named nginx-deployment.yaml:

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: nginx-deployment  
spec:  
  replicas: 2  
  selector:  
    matchLabels:  
      app: nginx  
  template:  
    metadata:  
      labels:  
        app: nginx  
    spec:  
      containers:  
      - name: nginx  
        image: nginx:latest  
        ports:  
        - containerPort: 80

Apply the Deployment

Run the following command to create the deployment in your cluster:

kubectl apply -f nginx-deployment.yaml

Expose the Deployment as a Service

To make the Nginx deployment accessible, create a service:

kubectl expose deployment nginx-deployment --type=NodePort --port=80

Access Your Application

Get the URL to access your application:

minikube service nginx-deployment --url

Open this URL in your web browser, and you should see the default Nginx welcome page!

For further learning, here is a list of Kubernetes resources;
✨ Kubernetes Documentation
Kubernetes Docs

✨ Kubernetes Basics Interactive Tutorial
Kubernetes Basics

✨ Katacoda Kubernetes Scenarios
Katacoda Kubernetes

✨ Play with Kubernetes
Play with Kubernetes

✨ Kubernetes YouTube Channel
Kubernetes YouTube

✨ TechWorld with Nana
TechWorld with Nana

✨ Google Cloud Tech
Google Cloud Tech

✨ Coursera - Introduction to Kubernetes
Coursera Kubernetes

✨ Udemy - Kubernetes for Absolute Beginners

Conclusion

In this article, you've learned what Kubernetes is, why we use it, its key terms and concepts, how to install and get started with Kubernetes, and finally, how to deploy your first application on it.

Kubernetes is a powerful tool for managing containerized applications, offering numerous features for scaling, load balancing, and self-healing. While the learning curve may difficult at first, there are plenty of resources and a supportive community to help you along the way.

Embrace the journey, and happy coding! If you have any questions or need further details on specific topics, feel free to ask or comment your thoughts!

The Ultimate Beginner's Guide to Azure AI Fundamentals (AI-900)

Wed, 01 Jan 2025 22:59:07 +0000

Introduction

The AI-900 certification is your entry ticket into the world of AI and Azure. As businesses embrace AI across industries, understanding Azure AI opens up diverse job opportunities. In this blog, we’ll break down tips, best practices, and how Azure AI Fundamentals can help you land exciting career prospects.

1. What is Azure AI Fundamentals (AI-900)?

Overview: The AI-900 is an entry-level certification offered by Microsoft, designed to provide foundational knowledge of AI concepts and how they are implemented using Azure AI services.

Topics Covered:

AI Concepts: Overview of AI, machine learning, deep learning.
Azure AI Services: Cognitive Services, Azure Machine Learning, Azure Bot Services.
AI Workloads: Types of AI workloads and solutions.
Responsible AI: Ethical considerations in AI development.

2. Best Practices to Prepare for the AI-900 Exam

Start with the Basics:

Familiarize yourself with AI fundamentals before diving into Azure-specific tools.
Learn about machine learning, deep learning, and AI model training.

Utilize Free Learning Resources:

Microsoft Learn provides structured, free courses aligned with the AI-900 exam.
Hands-on labs to experiment with Azure AI services like Computer Vision and Language Understanding.

Break Down the Exam Topics:

Focus on AI workloads: understand different AI models and use cases.
Master the core Azure AI tools: Cognitive Services, Azure ML, and Bot Services.

Practice with Real-World Examples:

Experiment with Azure AI tools using a free Azure account.
Try building simple AI models using Azure Machine Learning Studio.

Take Practice Exams:

Use practice tests to familiarize yourself with the exam format and question types.
Identify areas where you need more practice.

Practice Sets | AI 900: Microsoft Azure AI Fundamentals Exam https://courses.kloudsaga.com/courses/azure-ai-fundamentals

3. Key Tips for Success in AI-900

Stay Hands-On: AI is best learned through experience. Try implementing what you learn using Azure's free-tier services.

Focus on Real-World Applications: Understand how Azure AI tools solve business problems in real industries such as healthcare, finance, and customer service.

Study the Ethical Aspects: The AI-900 exam covers responsible AI practices, including fairness, transparency, and accountability—don’t skip this section.

Time Management: The AI-900 exam is time-constrained. Practice managing your time during the practice exams to ensure you can complete the questions within the allotted time.

4. Azure AI Career Opportunities

The AI-900 certification is not just a credential; it unlocks a wide range of exciting career opportunities in AI and cloud computing.

Here are some roles you can explore:

AI Developer:
Skills Needed: Knowledge of machine learning models, programming (Python, C#), and Azure Bot Services.
Roles: Building AI applications, developing chatbots, and integrating Azure AI into existing solutions.
Average Salary: $80,000 - $110,000.

Data Scientist/AI Engineer:
Skills Needed: Experience with Azure Machine Learning, data analysis, and predictive modeling.
Roles: Analyzing data to create predictive models and using Azure to deploy machine learning solutions.
Average Salary: $100,000 - $130,000.

Cloud Solutions Architect:
Skills Needed: Understanding cloud architecture, AI solutions, and Azure services.
Roles: Designing scalable AI solutions using Azure cloud infrastructure for enterprise clients.
Average Salary: $120,000 - $160,000.

AI Consultant:
Skills Needed: Familiarity with Azure AI tools, client communication, and project management.
Roles: Advising companies on how to implement AI-powered solutions and adopt Azure AI for their specific needs.
Average Salary: $90,000 - $130,000.

5. Best Practices to Grow Your AI Career After AI-900

Build a Portfolio: Start creating personal projects using Azure AI. Showcase chatbots, image recognition, or natural language processing models.

Stay Updated: AI is evolving rapidly. Subscribe to AI-related blogs, attend webinars, and stay on top of new features and tools in Azure.

Network with Professionals: Join AI communities, attend events like Microsoft Ignite, or engage in forums like Stack Overflow and LinkedIn groups.

Explore Further Certifications: After completing AI-900, you can pursue advanced certifications such as Azure AI Engineer (AI-102) or Microsoft Certified: Azure Data Scientist.

6. Key Takeaways

The AI-900 certification is an excellent starting point for anyone
looking to break into the world of AI and cloud computing with Azure.
Hands-on practice is critical.
Azure offers free tools and trial accounts to help you apply what you learn in real-world scenarios. After passing the AI-900 exam, explore various career roles,
including AI developer, data scientist, and cloud architect.
Networking and continuous learning will help you stay competitive in
the fast-growing field of AI.

Conclusion

The AI-900 certification is intiative into the world of artificial intelligence and Azure cloud services. By following the best practices, tips, and strategies outlined above, you’ll be well on your way to mastering Azure AI and unlocking new career opportunities in this fast-evolving field.

References:

https://learn.microsoft.com/en-us/training/courses/ai-900t00
https://kloudsaga.com

Cloud Computing Simplified: Models, Services, and the Future of Technology

Mon, 09 Dec 2024 23:37:48 +0000

Cloud computing has changed how we store, access, and manage data. It offers flexible, affordable, and powerful solutions that help businesses and individuals focus on what matters most. As more organizations adopt cloud technologies, understanding the basics is becoming essential.

In this guide, I’ll explain cloud computing, break down its key models, introduce its services, and highlight how it’s shaping industries and innovation.

Table of Contents

Introduction
Why Use Cloud Computing?
Cloud Deployment Models
- On-Premise Computing
- Public Cloud
- Private Cloud
- Hybrid Cloud
- Community Cloud
- Multi-Cloud
Cloud Services
- IaaS
- PaaS
- SaaS
- DaaS
- STaaS
- FaaS
Conclusion

Introduction

Cloud computing lets you access computing power over the internet instead of relying on expensive, on-site hardware. Whether you’re a business owner or a tech enthusiast, the cloud helps reduce costs, increase flexibility, and speed up innovation.

Let’s explore how it works, step by step.

Why Use Cloud Computing?

Here are five reasons why businesses are embracing cloud computing:

Cost Savings: You only pay for what you use—no need to buy expensive hardware upfront.
Scalability: Easily adjust resources as your needs grow or shrink.
Accessibility: Work from anywhere with an internet connection.
Security: Top providers offer advanced protections against cyber threats.
Speed: Quickly develop, test, and deploy projects to stay ahead.

Cloud computing simplifies IT so you can focus on building ideas instead of managing hardware.

Cloud Deployment Models

Cloud computing can be set up in different ways, depending on your needs:

1. On-Premise Computing
This is the traditional approach where everything is managed in-house.

Pros: Full control over data; suitable for strict security needs.
Cons: Expensive and hard to scale.

2. Public Cloud
Public clouds are shared by multiple users and managed by providers like AWS or Google Cloud.

Pros: Cost-effective and scalable.
Cons: Less customization; potential security concerns.

3. Private Cloud
A private cloud is dedicated to one organization, offering more control.

Pros: Highly secure and customizable.
Cons: Expensive and complex to maintain.

4. Hybrid Cloud
Mixing private and public clouds for flexibility.

Pros: Combines local security with public cloud scalability.

5. Community Cloud
A community cloud is a shared cloud infrastructure tailored for a specific group of users or organizations with shared concerns, such as compliance, security, or mission-critical goals.

Pros Costs are shared among participating organizations, enabling secure collaboration within the community, with access limited to members, making it more secure than public clouds.
Cons: Managing shared resources requires coordination among all parties and community clouds offer less scalability compared to public cloud solutions. Use Cases include healthcare with hospitals sharing patient data while ensuring HIPAA compliance, government agencies collaborating on secure joint initiatives, and education with universities sharing research infrastructure and tools.

6. Multi-Cloud
Using services from multiple cloud providers.

Pros: Using services from multiple cloud providers avoids reliance on a single provider while offering increased reliability through fault tolerance, improved disaster recovery, and higher system availability.

Cons: Managing multiple cloud providers can introduce complexity, lead to higher operational costs, and increase security risks due to varying security protocols across providers.

Cloud Services

Cloud computing isn’t just about storing data. Here are its main types of services:

1. Infrastructure as a Service (IaaS)
Think of IaaS as renting virtual servers and storage online.

Examples: Amazon EC2, Azure Virtual Machines.
Uses: Hosting websites, running applications.

2. Platform as a Service (PaaS)
PaaS provides tools to build and launch apps without managing the hardware.

Examples: Heroku, Google App Engine.
Uses: Streamlining app development.

3. Software as a Service (SaaS)
SaaS delivers ready-to-use apps over the internet, often subscription-based.

Examples: Google Workspace, Salesforce.
Uses: Productivity tools, CRM software.

4. Desktop as a Service (DaaS)
DaaS provides virtual desktops that can be accessed from any device.

Examples: Amazon WorkSpaces, Citrix.
Uses: Remote work setups.

5. Storage as a Service (STaaS)
STaaS offers flexible cloud storage for businesses and individuals.

Examples: Amazon S3, Google Cloud Storage.
Uses: Backups, disaster recovery.

6. Function as a Service (FaaS)
FaaS is about running code without managing servers, also known as serverless computing.

Examples: AWS Lambda, Azure Functions.
Uses: Event-driven apps like chatbots.

Conclusion

Cloud computing is transforming industries by making technology accessible, affordable, and efficient. From IaaS to SaaS, these services empower businesses to innovate faster and smarter.

The future of computing is in the cloud. Whether you’re just starting or looking to dive deeper, there’s no better time to explore its potential.

Coder Legion - Recent posts tagged cloud-computing

From Big Data to the Cloud, Part 1 — Why One Machine Is Never Enough

Chapter 1: Where All This Data Comes From

Chapter 2: What Big Data Actually Means

Chapter 3: The Five V’s

Chapter 5: The Distributed Idea

Chapter 6: Enter Hadoop

What Comes in Part 2

References

From “Why Is This Pod Pending?” to `HTTP/1.1 200 OK`: My NGINX Gateway Fabric Lab

The mission

1. Installing the Gateway API building blocks

Problem 1: Kustomize timed out while fetching the CRDs

2. The Gateway Pod was Pending

What does that mean?

3. Creating the frontend application

Problem 2: The Service pointed to the wrong port

4. Building the HTTPRoute

Problem 3: The HTTPRoute could not find the backend Service

5. Testing the Service before testing the Gateway

Problem 4: One curl test could not resolve the Service name

6. The Gateway was programmed, but my first request returned a 500

Problem 5: Two routes, one name, different namespaces

7. The final test: success through the NodePort

How a Kubernetes Cluster Works | Control Plane & Worker Nodes

How does a Kubernetes cluster actually work?

How Kubernetes Works | Kubernetes Explained Simply for Beginners

Kubernetes #DevOps #CloudComputing #K8s #CNCF #LFX

From `kubectl` to `jq`: A Beginner's Detective Story Through Kubernetes JSON Paths

The Setup: Two Nodes, One Mystery

Chapter 1: Peeking Under the Hood with JSON

Chapter 2: JSONPath — Kubernetes' Built-in X-Ray Vision

Success: Extracting Pod Labels

Success: Finding the Container Image

Success: Volume Mounts

Chapter 3: The Node Expedition — Where Things Get Interesting

Experiment 1: Getting Node Names

Experiment 2: Adding a Newline

Chapter 4: Custom Columns — Making Your Own Tables

Attempt 1: The Syntax Trap

The Fix: Drop the .items[*]

Attempt 2: The Unclosed Brace

The Fix: Close Your Braces!

Attempt 3: Maps Are Not Arrays

Attempt 4: The Map Dump

The Fix: Drill Into the Map

Chapter 5: The Real Boss Fight — Counting Schedulable Nodes

Mistake 1: Wrong Resource Type

Mistake 2: The Null Iteration Bomb

Attempt 3: Filtering for Null Taints

The Working Solution: The Full jq Pipeline

Chapter 6: The jq Command Autopsy

Stage 1: Get the Data

Stage 2: Enter jq

Stage 3: Iterate Over Items

Stage 4: The select() Filter

Stage 5: Collect and Count

Stage 6: Write to File

Chapter 7: The Plot Twist — A Node Goes Dark

Chapter 8: Key Lessons — The JSONPath Survival Guide

The Golden Rules

The Final File

Epilogue: Why This Matters

Internet-Free Infrastructure: How Two Servers Talk Without Using the Internet

The Question Nobody Asks

The Common Misconception

The Cloud Is A Giant City

Why Infrastructure Starts With A Network

The Next Problem

Subnets Are Neighborhoods

The Database Doesn't Need The Internet

Every Server Gets A Private Address

Knowing The Address Doesn't Create A Path

Routing Is The GPS Of The Cloud

The Request's Actual Journey

Why This Matters To Founders

Architecture A

Architecture B

The Most Important Lesson

Frequently Asked Questions

The Fix: Drop the `.items[*]`

The Working Solution: The Full `jq` Pipeline

Chapter 6: The `jq` Command Autopsy

Stage 2: Enter `jq`

Stage 4: The `select()` Filter

Error 7: `etcdctl snapshot status` deprecated

Error 8: `etcdutl` not installed

Error 9: `sudo cd`

Error 11: `kubectl logs` without a Pod

Error 12: `kubectl` Forbidden