Coder Legion - Recent posts tagged network

From Big Data to the Cloud, Part 1 — Why One Machine Is Never Enough

Sun, 25 Oct 2026 05:46:00 +0000

A calm, sequential guide for anyone entering the field. The article is divided into smaller chapters to make it easier to understand.

This series grows out of the Cloud Computing labs I teach to fourth-year Computer Science students at the University of Hilla. The labs move quickly, because a lab has ninety minutes. The articles can afford to slow down. Here we take the same path the labs take, but we stop at every step and ask the question a careful student always asks: why this, and not something simpler?

The path itself is short to state. Data has grown past the point where one machine can store it, process it, or analyse it efficiently. The answer is not a bigger machine but many ordinary machines working together. And the framework that manages those machines is Apache Hadoop. Each of those three sentences hides a chain of reasoning, and the purpose of this part is to lay that chain out link by link — so that by the end, the cluster does not look like a fashionable choice but like the answer the problem forces on us.

The destination of this series is the cloud. But we will not begin there, because a definition of the cloud means little until you have felt the problem it was built to solve. So the series takes the long road on purpose: first the problem (this part), then the cluster that answered it and the burden that answer carried (Part 2), and only then the cloud — defined in Part 3, and explored service by service in the parts after it.

We will not touch a configuration file in this part. What we build here is the problem — precisely enough that when Part 2 opens Hadoop up and looks at HDFS, MapReduce, and YARN from the inside, each component reads as the answer to a question we have already asked.

Chapter 1: Where All This Data Comes From

Every Action Leaves a Record
Before we define anything, it is worth noticing how ordinary the sources of big data are. Every click, every purchase, every email, every chat message, and every social post creates data. None of these events is large on its own.

But the sources do not stop at people typing. Sensors and IoT devices report what they measure. Banks record every transaction. Hospitals keep patient records. Streaming services log what their users watch. Each of these produces records continuously, whether or not anything interesting is happening.

The individual record is small. What makes the situation new is the multiplication: small records, produced by enormous numbers of people and devices, without end. Data now reaches billions of records, and petabytes every day. The important words there are every day. The data does not arrive once. It keeps arriving.

Chapter 2: What Big Data Actually Means

A Definition About Tools, Not Size
The most useful definition of big data is not a number. It is this: big data is datasets so large and complex that traditional tools — spreadsheets, a single database server — cannot store, process, or analyse them efficiently.

It is worth being exact about why the definition is phrased this way. Students often ask how large data must be before it “counts” as big. The definition answers by pointing at the tools rather than at a size: data becomes big at the moment the ordinary tool stops coping with it. Notice also that it names three separate jobs — store, process, analyse — and a tool can fail at any one of them.

The Goal Is Insight, Not Storage
A second point is easy to miss: the goal is not only storing data. The goal is extracting insights from it — turning a mountain of records into a decision someone can act on.

The classic example is an online store. It already keeps every purchase as a matter of bookkeeping. The value appears only when it analyses customer habits: what people buy, when, and together with what. With that analysis, the store can make targeted offers and discounts, which increases revenue. The data was there all along; what matters is the ability to read it. Hold onto this, because a system that can store huge data but cannot analyze it has solved only half the problem.

Three Shapes of Data
Data does not only come in different sizes. It comes in different shapes, and the shape decides which tools can handle it.

Structured data is the familiar case: every row has the same columns. A student table with a name, a department, and an age is structured.
Semi-structured data carries its own labels — tags in XML and HTML, keys in JSON — but does not promise that every record has the same fields. The keys tell you what each value means, yet there is no rigid schema forcing every record into one shape.
Unstructured data has no predefined format at all. An email body, a social post, an image, a video: the meaning is in there, but no column tells you where.

And here is the uncomfortable fact: most of today’s data is unstructured, and that is exactly what traditional databases handle worst. They were built around the first row of that table. The world moved to the third.

Chapter 3: The Five V’s

A Vocabulary for “Why It Is Hard”
“Too large and too complex” is a good definition, but it is too blunt to design a system against. We need to name the separate ways data can be difficult, because each one places a different demand on the machine that handles it. The standard vocabulary for this is the five V’s.

One at a Time
01 — Volume: how much. The huge amount of data, from terabytes to petabytes. Volume is the V that breaks storage first.

02 — Velocity: how fast. How fast data is created and must be processed — think of stock trading or live streams. The second half of that sentence is the important one: these systems need answers quickly, not just storage quickly. Velocity is the V that breaks time.

03 — Variety: how many shapes. Many types and formats of data — the structured, semi-structured, and unstructured shapes from Chapter 2, often mixed together. Variety is the V that breaks fixed schemas.

04 — Veracity: how trustworthy. How accurate and trustworthy the data is. Real data is noisy, messy, and has missing values, so cleaning and checking quality stands between you and a conclusion you can defend.

05 — Value: what we get out of it. The useful insight we finally get out of the data. This is the V that justifies the other four — the point Chapter 2 already made, now given its own name.

Notice the split: the first three describe the data itself, while the last two describe whether we can trust it and whether it was worth the effort.

Chapter 4: Why Traditional Systems Fail
What “Traditional” Means Here
By a traditional system we mean one built for structured data on a single powerful machine. All the data lives on that machine; all the work runs on it. This design is good at what it was built for. The point of this chapter is that big data asks it to do something it was never built for — and it breaks in five places.

The Five Breaking Points

It was built for structured data on one machine. Its whole design assumes fixed rows and columns, and one computer doing all the work.
It cannot keep up with the volume and speed of modern data. This is Volume and Velocity together: too many bytes, arriving too fast, for one machine to absorb.
It has difficulty handling unstructured data — images, video, text. This is Variety. And since most of today’s data is unstructured, this is not an edge case; it is most of the data.
Scaling up means buying a bigger, very expensive server — and there is always a limit. When one server is not enough, the traditional move is to scale up: replace it with a bigger one. That works for a while, but each step costs more, and eventually there is no bigger machine to buy.
It is slow to extract insights from huge datasets. Put the first four together and the last failure follows on its own. This is the failure that matters most, because it is a failure of Value.

Three Challenges, Stated Plainly
All five breaking points reduce to three questions any big data system must answer.

Storage — where do we keep data that keeps growing?
Processing — how do we analyse it fast enough to be useful?
Scalability — how do we grow (or shrink) resources as data changes?

Read the third question twice, because it hides a detail that will matter later in this series. It does not only ask how to grow. It asks how to grow or shrink — how to have exactly as much capacity as the data needs, no more and no less. Keep that small word “shrink” in your pocket; we will come back for it.

The single powerful machine answers none of them well once the data is big enough. So the question becomes: what architecture answers all three?

Chapter 5: The Distributed Idea

Many Ordinary Machines, Working Together
The answer is to stop making one machine bigger. Instead of one big machine (monolithic), we use many ordinary machines working together, called a cluster. Each machine in the cluster is a node. Both the data and the processing are split across the nodes. A cluster can range from a few nodes to thousands — and cheap commodity hardware is enough: normal PCs and servers, not specialised machines.

This strategy is called scaling out: when you need more capacity, you do not replace the machine — you add more machines.

The Price of Distribution
Scaling out is not free. The moment we spread data across many machines, we inherit three problems the single machine never had:

Which data lives on which node? With one machine, a file is simply on its disk. With many, something must remember where every piece went.
How do nodes communicate and coordinate? Many machines working on the same job is only useful if the work is divided and the results are brought back together.
What happens when a machine crashes? On one server, a crash is a rare event. In large clusters, failures are normal.
That last point changes how the whole system must be designed. If failure is the normal state of affairs, the system cannot treat it as an emergency. It must expect machines to fail and keep working anyway, without losing data.

Stated as a General Rule
Scaling out trades one hard problem — a machine that cannot grow past its limit — for three new ones: tracking where data lives, coordinating the nodes, and surviving failure. The trade is worth it only if software solves those three problems, so that the people using the cluster do not have to. What the field needed was a framework that solves them once, for everyone.

Chapter 6: Enter Hadoop

What Hadoop Is
Apache Hadoop is an open-source framework for distributed storage and parallel processing of big data. It runs on clusters of commodity hardware — exactly the cluster Chapter 5 described. It was created around 2005, when the volume of web data, from sites like Facebook and YouTube, exploded. It is free to download from Apache, and commercial distributions also exist, such as Cloudera, Hortonworks, and MapR.

Its Core Components, at a Glance
The cleanest way to understand Hadoop’s design is to hold it up against the three challenges from Chapter 4. Each core component answers one of them.

We deliberately stay at this altitude for now. How HDFS splits and tracks blocks, why MapReduce runs the program where the data already is, and how YARN hands out CPU and memory are the subject of Part 2.

What Hadoop Buys

Scalable — add more nodes as data grows. (Notice: as data grows. The “shrink” half of the Scalability question is still waiting.)
Fault tolerant — keeps copies of data on different machines, so a hardware failure does not lose data. This is the answer to “what happens when a machine crashes?”
Cost-effective — uses cheap commodity machines.
Flexible — handles structured and unstructured data. This is the answer to Variety.
Parallel — many nodes work at the same time.

What Hadoop Costs
An honest account includes the other column.

It is designed for batch processing, not real time. For real-time work, Apache Spark is better.
It is complex to configure and maintain — which is exactly what the labs practise.
Replication uses extra storage space. Keeping copies on different machines is what makes Hadoop fault tolerant, but every copy takes up disk.

Where It Is Used
Healthcare: patient records, medical images, early disease prediction.
Banking: fraud detection, transaction analysis.
Social media: sentiment analysis of posts.
E-commerce: recommendations, customer behaviour, inventory management — the online store from Chapter 2, now at full scale.

What Comes in Part 2

We now have the problem in full, and the shape of the answer. We know why one machine is never enough, why a cluster is the way out, and which new problems the cluster creates. We have met Hadoop’s components by name. What we do not yet have is the inside view.

Part 2 opens the components one at a time. We follow a 400 MB file as HDFS splits it into 128 MB blocks, and see why, with three copies of each block, that file ends up stored as twelve blocks across the cluster. We meet the NameNode that keeps track of which block is on which machine, and the DataNodes that send it a heartbeat to say “I’m alive” — and we ask what happens when one of them stops. We walk through MapReduce by hand, on a word count and on an average-age example, from Map through Shuffle & Sort to Reduce. And we watch YARN take a submitted job, start an ApplicationMaster for it, and request containers on the nodes that hold the data.

The single principle running through all of it: Hadoop moves the work to the data, not the data to the work.

And Part 2 will end where the rest of this series begins. Hadoop solves storage, processing, and growth — but it leaves us holding a cluster: machines to buy, a framework that is complex to configure and maintain, and capacity that can grow but not easily shrink. That remaining burden is exactly what the cloud takes off our hands, and it is where Part 3 picks up the story.

References

Deshpande, T., Karanth, S., & Turkington, G. Hadoop: Data Processing and Modelling (Learning Path). Packt Publishing.
Oracle. What Is Big Data? https://www.oracle.com/big-data/what-is-big-data/
Big Data — Understanding Hadoop: Architecture, Features, Challenges & Applications. Medium. https://medium.com/@rp99452/big-data-understanding-hadoop-architecturefeatures-challenges-applications-c7ab7011e375
Mehta, C. B. (2022). A Comprehensive Review of Hadoop Ecosystem for Big Data Processing and Analytics. IJNRD, 7(10).

Can You Belong to a Community Without Finding Your Audience?

Thu, 22 Oct 2026 19:18:09 +0000

Lately, I have been questioning whether CoderLegion is the right platform for the content I create.

As a game developer sharing the progress of Life of Alex, I write about building mechanics, designing environments, solving problems, and learning through development. These are subjects I care about, and documenting them is part of how I contribute to the community.

However, the traffic reaching that work has been limited. That has made me consider a question that might be familiar to other creators: how do you recognize when a platform is valuable to you, but might not be the best fit for your content?

My experience with CoderLegion makes that question particularly interesting. I am a leader here, I have become a recognized voice, and have had the opportunity to share my career journey through an interview. Those things matter to me. They are meaningful reasons to remain involved.

At the same time, recognition within a community and readership of your work are different things. You can feel welcomed and appreciated while still wondering whether what you publish is reaching people interested in the subject.

I do not expect a role or a badge to guarantee an audience. Readers choose where to spend their time, and contributors have to earn that attention. But when you regularly share work and see limited traffic, it is reasonable to assess whether the subject, format, and audience are connecting.

Game development can sit in an unusual position within broader technical communities. A development update might involve visual design, player experience, animation, or the gradual construction of an environment. Its value may be harder to communicate through a headline than a tutorial that answers a specific technical question.

That is one possible explanation, rather than a conclusion about CoderLegion’s audience. Presentation, discoverability, and my own approach could also play a part. Low traffic tells me that something is not connecting; it does not tell me exactly why.

There is also a practical limit to how far I want to adapt. I am happy to improve how I explain my work, provide more context, and make an article more useful to readers. However, I still want the subject to reflect the game I am actually building. If every post becomes a search for a more popular topic, the development journal gradually loses its purpose.

For now, I want to remain part of CoderLegion while being realistic about what publishing here brings. A platform can have value through its relationships, conversations, and opportunities, even when it contributes little traffic. The question is how much time to invest in each of those benefits, particularly when that time also comes out of development.

I would be interested to hear how other contributors approach this. Have you found an audience by changing how you present your work, or discovered that a community you enjoy simply has limited interest in your particular subject?

I have tried everything under the sun: taking interest and contributing to other people's passions or careers, I created discussions over different topics, i posted more often and more rare, tried changing tags... I am at a loss...

What helps you decide whether to keep adapting, publish less frequently, or put more effort into another platform?

My Network Engineering Home Lab setup

Sun, 18 Oct 2026 07:57:18 +0000

Introduction

A while ago, I built a home lab to gain hands-on experience with Linux server administration and network engineering. Using some old hardware, a ThinkCentre M81 desktop and a Tenda router, I set up a private LAN, configured a static IP and DHCP reservations, and deployed an SSH-accessible Linux server running an Nginx web server.

This project allowed me to apply concepts I had learned in college in a real-world scenario, practice network setup and IP management, and explore server configuration beyond the classroom. It’s also an example of repurposing older hardware to create a functional learning environment.

Below, I’ve documented the full technical process, step by step, so you can see exactly how I built this setup and replicate it if you want to create your own home lab.

Tenda Router Setup

The router I used is the Tenda AC1200 router: https://www.tendacn.com/product/ac6.html.

Turn on the router.
Connect to the router’s LAN port (1 or 2) via ethernet from a PC you are able to use.
Visit 192.168.0.1, the default router IP (written in the back of your router) from a web browser.
Enter the default login password (written in the back of your router).
Go to system settings and change the login password
Go to Wi-Fi settings and change the Wi-Fi name and password.
Go to system settings and upgrade the firmware if needed.

The router is ready for connection. Disconnect the ethernet from the router, you can now access it via Wi-Fi.

ThinkCentre M81 Server setup

Installing Linux Mint

Connect the ethernet port of the CPU to any the Tenda router’s LAN ports, and install linuxmint normally: https://linuxmint-installation-guide.readthedocs.io/en/latest/

But during the installation process, choose to erase the whole disk and install linux mint while asked about partitions.

Do not select to set up an encrypted LVM (full disk encryption) in the Advanced features. Using an LVM is okay but encrypted LVM for full disk encryption requires someone to type the password in the keyboard connected to the server every time it boots, and we want it to automatically boot and be ready for ssh login so it is not recommended.

And the authorized ssh keys stay in the home folder of the user, so if you encrypt it, logging in to the user account will be required for key based ssh to work. Logging in remotely strictly using ssh keys will not work then. So do not encrypt the home folder.

After the installation is complete, login and set the root password.

sudo passwd

Turning the desktop into a server

After that, we need to turn this into a server from a desktop.

Install openssh-server and configure it: https://wiki.debian.org/SSH#Installation_of_the_server
[Not required but highly recommended] Make the ssh server run on a port different from 22, for security (to prevent automated brute force scripts).
[Not required but highly recommended] Turn on firewall and allow the ssh. When the firewall is turned off, every port open on the system is open to the network. So turn it on to make sure only the ports you allow are accessible from the outside.
```
sudo ufw allow <ssh_port_number>
sudo ufw enable
```
Now we need static IP for the server instead of DHCP from the router. We will be using systemd-networkd for this.
```
sudo ifconfig
```
And remember the name of the ethernet port connecting to the router (usually eno1)
Now stop and disable NetworkManager, it is bound to the display manager so if we use primarily Network Manager, the server will not have networking when started headless.
```
sudo systemctl stop NetworkManager
sudo systemctl disable NetworkManager
```
Now enable systemd-networkd on boot
```
sudo systemctl enable systemd-networkd
```
Create a file /etc/systemd/network/10-eno1.network and put in the contents
```
[Match]
Name=eno1
```

[Network]
Address=192.168.0.2/24
Gateway=192.168.0.1
DNS=192.168.0.1


Where `192.168.0.1` is the IP address of your router and `192.168.0.2` is the IP address of the server, and `eno1` is the name of the network interface of the ethernet port.
8. Reload systemd-networkd for the changes to take effect

sudo systemctl restart systemd-networkd

9. Now run `sudo ifconfig` again and the ip address should have changed on the ethernet port.
10. Now that we have static IP, go to router login page -> System Settings -> DHCP Reservation
![tenda-dhcp-reserv](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c4stq9hz8n3tpht6i728.png)
And manually bind the static IP to the MAC address of the server.

We have turned the desktop into a server. Now ssh into it from your PC to test if it works.

ssh -p username@192.168.0.2


## Making server headless

Once you have logged in to the server via ssh, you can easily turn it headless. You simply have to stop the display manager and disable it on boot so it always runs headless.

(after ssh into server)
sudo systemctl stop lightdm
sudo systemctl disable lightdm


After this, the monitor screen connected to the desktop goes blank. This is expected. If you are not sure if the system hanged or if the graphical manager has stopped, press Ctrl+Alt+F1 to enter virtual terminal 1. If you see the virtual terminal, it is working properly. Now you can safely remove the monitor and keyboard and mouse from the server’s CPU as you won’t need it anymore and will always access it by ssh.

## Configuring Web Server
While you are still ssh-ed into the server, run

sudo apt install nginx
sudo systemctl start nginx
sudo systemctl enable nginx

To install, start and enable the web server on boot. It runs on port 80, the default port for HTTP. So allow it on the firewall for the website to be accessed from the LAN.

sudo ufw allow 80

You can leave it as it is to get the default nginx server page or create a file `/var/www/html/index.html` with your own website on the `index.html` file, or even host a react app by copying the build files into the `/var/www/html` folder.


## Accessing the website

Connect to the Wi-Fi network of the Tenda router from any device that has Wi-Fi.
![wifi-con](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o16p8noin4w1xe8417wt.png)

And then open the IP address of the server in a web browser. You will get a no HTTPS warning but you can ignore it and continue since we are in a controlled environment.

![website](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/438xncg43o7z8v4zo6ff.png)


## SUCCESS
That’s it. The home LAB has been set up successfully. You have a LAN on Wi-Fi and a server that you can ssh into and use as backup storage, and a web server you can host websites on your LAN.

MyZubster: Building a Provenance Network for Knowledge, Skills, and Real-World Exchange

Thu, 17 Sep 2026 04:40:03 +0000

MyZubster: Building a Provenance Network for Knowledge, Skills, and Real-World Exchange

We started MyZubster with a simple real-world question:

What happens if I give someone my kefir culture, they reproduce it at home, document what they learn, and eventually pass both the culture and the knowledge to someone else?

That small experiment is evolving into something much larger.

We're building MyZubster as an open-source system for representing how resources, knowledge, and skills move between people while preserving provenance and uncertainty.

The core loop is simple:

Share → Try → Observe → Improve → Share again
From a kefir culture to a provenance graph

Milk kefir is our first real-world prototype.

Instead of storing only:

Daniel gave kefir to Nicola

we want the system to represent the actual lifecycle:

Daniel's culture → Marketplace listing → Nicola accepts → Hand exchange → Nicola confirms receipt → Nicola maintains his culture → Observations → Future donation

Every transition is a separate event.

If something hasn't happened yet, the system shouldn't pretend it has.

If a quantity, temperature, timestamp, outcome, or condition wasn't recorded, it stays:

UNKNOWN

This principle has become one of the foundations of the MyZubster Knowledge Protocol.
Knowledge is another transferable resource

Then we realized the same model can describe learning.

Daniel has been helping Nicola work with GitHub, Docker, and the MyZubster ecosystem.

That creates another graph:

Daniel → Nicola learns → Nicola attempts → Observable artifact → Nicola can teach another person

But there is an important rule:

Evidence is not inherited.

If Nicola teaches another participant, that participant must create their own evidence.

So the graph becomes:

Daniel → Nicola → Person B → Person C

while every edge has independent provenance.
From fermentation to a general knowledge network

The repository has now expanded beyond fermentation.

Current and newly introduced domains include:

KF- — Fermentation
SND- — Sound systems
DEV- — Programming
UNI- — University/research
PET- — Animals
PERM- — Permaculture
MUS- — Music
ART- — Art
SPT- — Sport
MA- — Martial arts

This allows very different communities to use the same underlying protocol.

A tekno musician could teach someone how to operate or assemble part of a sound system.

A musician could teach another person DJing, production, rhythm, or an instrument.

An artist could teach drawing or comics.

A calisthenics practitioner could document teaching a particular progression.

A Thai-boxing practitioner or coach could document a bounded training session.

The domain changes.

The provenance model doesn't.
Turning skills into marketplace objects

We're now extending the MyZubster marketplace concept so that an offer doesn't necessarily represent a physical object.

It could represent:

a resource, a lesson, mentoring, a workshop, a demonstration, or a collaboration request.

A skill exchange can therefore follow:

Teacher → Skill Listing → Learner Accepts → Session → Participation Confirmation → Attempt → Evidence → MyZubster Record

And eventually:

A teaches B → B reproduces → B teaches C → C reproduces

This creates a distributed graph of practical knowledge.
Physical and knowledge graphs can intersect

The interesting part is that MyZubster can maintain two separate graphs.

A physical-resource graph:

Daniel → Kefir → Nicola → New culture → Future recipient

And a knowledge graph:

Daniel → Nicola → Reproduced skill → New learner

They can reference each other without becoming the same thing.

Receiving kefir doesn't prove that someone knows how to maintain it.

Reading a Knowledge Card doesn't prove that someone performed the procedure.

Teaching a skill doesn't prove that the learner mastered it.

Each statement needs its own evidence.
Where blockchain fits

Blockchain is intended as an optional provenance layer.

We're deliberately avoiding the shortcut:

blockchain = truth

It doesn't.

An on-chain record can establish that a particular commitment or declaration was recorded.

For MyZubster, we therefore distinguish states such as:

RECORDED

from:

ONCHAIN_RECORDED

The latter should only exist when we have an actual network and verifiable transaction/record identifier.

A transaction hash doesn't automatically prove competence, lesson quality, physical delivery, professional qualifications, or the truth of every statement committed to it.

Those remain evidence problems.
Evidence states

MyZubster currently works with a deliberately small evidence vocabulary:

PERSONAL_PRACTICE
TRADITIONAL_PRACTICE
OBSERVATION
EXTERNAL_SOURCE
VERIFIED_GUIDANCE

And there is another important constraint:

More observations do not automatically produce VERIFIED_GUIDANCE.

Verification requires an explicit review step.

That prevents a community voting mechanism or simple repetition counter from silently transforming popular claims into verified knowledge.
The implementation

This isn't only a concept document.

The repository already contains tooling for:

canonical Knowledge Cards;

domain validation;

evidence records;

claims and sources;

knowledge relations;

reproductions;

knowledge versions;

collaboration requests;

verification reviews;

integrity checking;

graph persistence;

transactional record/graph updates;

recovery logic;

a read-only Knowledge API;

a browser-based Knowledge Explorer.

We've also started creating real cross-domain records.

For example:

DEV-001 — Nicola's observable MyZubster/Docker/comics project artifacts

DEV-002 — Nicola's reported learning/onboarding path

ART-001 — Daniel's comics practice

SPT-001 — Daniel's calisthenics practice

MA-001 — Daniel's Thai-boxing practice

For claims where supporting evidence hasn't yet been attached, the system explicitly preserves that limitation rather than manufacturing provenance.
Current development status

Our last confirmed complete local run was:

110 tests / 110 passing

That result predates the newest domain and Knowledge Card additions, so we're not claiming those latest changes as tested until the next complete run.

That's another principle we're trying to follow throughout the project:

don't convert an expected result into an observed result.

The public Knowledge Explorer is available here:

MyZubster Knowledge Explorer

The source code and current protocol development are here:

MyZubster on GitHub

At the moment, GitHub contains newer work than the production Knowledge Explorer deployment, so the repository is the source of truth for the latest development.
What's next

The next major step is moving the new Skill Marketplace model from protocol documentation into structured schemas and runtime records.

The target lifecycle is:

Listing → Acceptance → Session/Exchange → Evidence → Knowledge Graph → Optional on-chain commitment

After that, we can start testing the protocol with more real people and very different types of knowledge.

Kefir is just the first physical experiment.

The broader question behind MyZubster is:

Can we build an open provenance graph showing not just what people say they know, but how knowledge and resources actually move, get reproduced, change, and get passed to the next person?

That's what we're trying to find out.

Two Racing Lines: Annotations and the DSL in NetFlow 0.7.0

Mon, 14 Sep 2026 08:00:00 +0000

I have been building NetFlow, my Kotlin Multiplatform networking library, since it was an Android-only thing called Communication. I wrote about that migration and why I took it on in Why I Took the Leap from Android-Only to Kotlin Multiplatform, so I will not repeat it here. Version 0.6 I added a transform parameter. The compiler forces you to provide whenever your DTO and your domain model are different types. Forget it and the build fails. No runtime surprise, no half-mapped object reaching the UI.

Then I started a new data layer in a side project, opened an empty file, and typed @GET.

Into my own library. Which did not have annotations.

That is a small, stupid moment, and it told me something I had been avoiding. NetFlow has three stars on GitHub and, as far as I know, no users outside my own projects, so I cannot tell you what evaluators said about it. I can tell you what my own hands did when I was not paying attention.

The reflex

Years of Retrofit built a motor pattern into every Android developer I know, including me. You want an endpoint, you write an interface, you put @GET("users/{id}") on a method. It is not a decision. It is closer to reaching for the indicator stalk.

Here is what fetching a list looked like in NetFlow 0.6:

// commonMain
val todos = client.call {
    path = "todos"
    method = HttpMethod.Get
}.responseListFlow<TodoDto, Todo>(transform = { it.toModel() })

Read that with fresh eyes. It says exactly what it does, it is one expression, and the mapping is compiler-checked. It is also nothing like what muscle memory reaches for.

A developer evaluating a networking library gives it about ninety seconds of README. In that window, "does this look like the thing I already know" beats "is this better" almost every time. I don't think that is laziness. Familiarity is a real cost saving when you are picking one of five libraries that all do roughly the same job.

So the 0.7.0 question was not "annotations or DSL". It was: can I put a familiar entry on the front without weakening the thing behind it?

What came out is a ladder with three rungs. The bottom rung is an annotated interface and covers most of what you write in a day. The middle rung annotates the request and hands the response back to you. The top rung is the DSL, unchanged, for the calls that need everything. You climb only as far as the call in front of you requires.

Rung one: the return type picks the strategy

The annotation layer lives in two new modules, netflow-annotations and netflow-ksp. You annotate an interface, KSP generates the implementation, and a factory extension hands it to you.

// commonMain
@NetFlowApi
interface TodoApi {

    @GET("todos")
    suspend fun getTodos(@Query completed: Boolean?): AsyncState<List<TodoDto>>

    @GET("todos/{id}")
    fun observeTodo(@Path id: Int): Flow<ResultState<TodoDto>>

    @Headers("Accept: application/json", "X-Client: netflow")
    @POST("todos")
    suspend fun create(@Body request: CreateTodoRequest): AsyncState<TodoDto>

    @DELETE("todos/{id}")
    suspend fun delete(@Path id: Int): AsyncState<Unit>

    @GET("todos")
    fun pagedTodos(): Flow<PagingData<TodoDto>>
}

val api = client.createTodoApi()

The part I want to point at is that there is no @Strategy annotation anywhere. The return type does that job.

AsyncState<T> on a suspend function means a one-shot call. Flow<ResultState<T>> on a non-suspend function means an observable call with loading and error states. Flow<PagingData<T>> means a Paging 3 flow. List<T> inside any of those routes to the list variants. The processor reads the shape and picks responseAsync, responseListAsync, responseFlow, responseListFlow, or responsePaginated accordingly.

This is not a clever trick, it is consistency. Flow<ResultState<T>> already self-describes in the hand-written DSL. Making the annotation layer read the same signal means there is one fact to learn instead of two.

The processor is strict about mismatches, and the errors are compile-time:

Function 'getTodos' returns AsyncState and must be suspend.
Function 'observeTodo' path declares '{id}' but there is no @Path parameter named 'id'.
Function 'create' has more than one @Body; at most one @Body is allowed.

@Path and @Query and @Header take their wire name from the parameter name unless you override it, a nullable @Query or @Header is omitted from the request when it is null, and at most one @Body is allowed per function.

What the processor actually writes

This matters more than it sounds, so here is the generated file for two of those methods, copied out of build/generated:

// _TodoApiImpl.kt, generated
internal class _TodoApiImpl(
  private val client: NetFlowClient,
) : TodoApi {
  override suspend fun getTodos(completed: Boolean?): AsyncState<List<TodoDto>> = client.call {
    method = HttpMethod.Get
    path = "todos"
    if (completed != null) parameter("completed" to completed)
  }
  .responseListAsync<TodoDto>()

  override fun observeTodo(id: Int): Flow<ResultState<TodoDto>> = client.call {
    method = HttpMethod.Get
    path = "todos/" + id
  }
  .responseFlow<TodoDto>()
}

public fun NetFlowClient.createTodoApi(): TodoApi = _TodoApiImpl(this)

That is the 0.6 DSL. Byte for byte, it's the code you would have written by hand. The annotations are not a parallel implementation, they are a text expansion into the existing one. Nothing new runs at runtime, there is no reflection, and there is no second code path to keep in sync when the DSL changes.

It also means you can open the file and read it.

The decision that matters: no transform in annotations

Now the constraint that shaped everything else.

An annotation cannot hold a lambda. Kotlin annotation parameters are limited to compile-time constants, so transform = { it.toModel() } has nowhere to live. Neither does onNetworkSuccess { db.insert(it) }, or local { observe { db.todos() } }, or a paging RemoteMediator.

The mapping problem has known workarounds and I looked at all of them. @MappedBy(TodoMapper::class) pointing at an object with a map function. A converter registry you populate at client construction. An interface the DTO implements. Every one of them takes a relationship the 0.6 compiler verifies and moves it somewhere the compiler cannot see, resolved by KSP at best and by a map lookup at worst. Forgetting a mapper would stop being a build error and start being a runtime cast exception on a device.

I was not going to make the convenience layer less safe than the thing it is a convenience for. So annotated methods return the DTO. Full stop.

The mapping moves one layer out, into the repository, using the .map helpers NetFlow already ships. Their KDoc has said the same thing since 0.1: usually used in the repository to map the dto to the model.

// commonMain
class TodoApiRepositoryImpl(client: NetFlowClient) {

    private val api = client.createTodoApi()

    suspend fun getTodos(): AsyncState<List<Todo>> =
        api.getTodos(completed = null).map { dtos -> dtos.map { it.toModel() } }

    fun observeTodo(id: Int): Flow<ResultState<Todo>> =
        api.observeTodo(id).map { state -> state.map { it.toModel() } }

    fun pagedTodos(): Flow<PagingData<Todo>> =
        api.pagedTodos().map { page -> page.map { it.toModel() } }
}

The mapping is still explicit, still typed, still checked by the compiler. It moved. It did not weaken.

If there is one transferable idea in this whole article, it's that one. When you build a convenience layer over a stricter API, the layer is allowed to be less capable. It's not allowed to be less safe. The moment it is, every user of the convenience layer is quietly worse off than the users who ignored it, and you have built a trap with good ergonomics.

The paged endpoint is one line

If there is a single place where this layering pays for itself, it's paging.

An annotation library that only speaks HTTP gets you as far as the DTO. After that you write a PagingSource: a load that does the key arithmetic, a getRefreshKey that reads state.anchorPosition and reaches for the closest page, a LoadResult branch for success and one for failure, then a Pager and a PagingConfig to assemble it. That is fifty lines you have written before, and fifty more for the next paged endpoint, because none of this is about your data.

In NetFlow you only need a PagingData return type.

// commonMain
@GET("todos")
fun pagedTodos(): Flow<PagingData<TodoDto>>

@Paginated(pageSize = 15)
@GET("todos")
fun pagedTodosSmall(): Flow<PagingData<TodoDto>>

@Paginated is optional, and it carries only the two things: the query parameter name for the page number, and the page size. Leave it off and you get page and 20.

The DTO extends PagingModel, which is where the page number and the last-updated timestamp live:

@Serializable
data class TodoDto(
    val userId: Int,
    val id: Int,
    val title: String,
    val completed: Boolean
) : PagingModel()

Forget it and the build stops with Flow<PagingData<T>> requires T to extend PagingModel. Put @Paginated on a method that returns something else and the build stops there too. Neither is a runtime discovery.

The repository maps exactly like every other annotated call, and the ViewModel does what it always does:

// commonMain
fun pagedTodos(): Flow<PagingData<Todo>> =
    api.pagedTodos().map { page -> page.map { it.toModel() } }

val todos = repository.pagedTodos().cachedIn(viewModelScope)

That flow is declared in commonMain, and this is the part with no Ktorfit equivalent at all. On Android you collect it with collectAsLazyPagingItems(). On iOS, netflow-paging ships PagingCollectionViewController, a Kotlin class exposing loadStateFlow, onPagesUpdatedFlow, getItems() and loadNextPage() to Swift through SKIE, so a SwiftUI list drives the same pager. One annotated method, both platforms, no per-platform paging code.

Now the boundary. An annotated paged method is network-only. The generated call says so:

// generated
.responsePaginated<TodoDto, TodoDto> {
  onlyApiCall = true
  defaultPageSize = 15
}

Remote plus local paging needs a PagingSource factory, an insertAll block, and a timestamp lambda. Three lambdas. So it stays where lambdas live, and getting there is the next rung.

Rung two: NetFlowCall

That leaves an obvious hole. What about an endpoint whose request is pure boilerplate, path and method and three query params, but whose response needs a local cache?

Under a strict reading of rung one you drop back to the full DSL and hand-write the request too, which means the annotation layer is useless for exactly the calls that have the most typing in them.

0.7.0 splits the call in half. An annotated method can return NetFlowCall, which is the request with no response strategy attached:

// commonMain
@GET("todos")
fun todosCall(): NetFlowCall

The generated implementation stops at the request:

// generated
override fun todosCall(): NetFlowCall = client.prepareCall {
  method = HttpMethod.Get
  path = "todos"
}

And the caller composes the response side with the whole DSL available, transform requirement included:

// commonMain
fun getTodos(): Flow<ResultState<List<Todo>>> =
    api.todosCall().responseListFlow<TodoDto, Todo>(transform = { it.toModel() }) {

        onNetworkSuccess { dtos ->
            database.todoQueries.transaction {
                dtos.forEach { database.todoQueries.insertTodo(it.toEntity()) }
            }
        }

        local(
            { observe { database.todoQueries.selectTodos() } },
            transform = { entities -> entities.map { it.toModel() } }
        )
    }

The annotation owns what annotations are good at: path, method, query parameters, headers, body. The repository owns what only a lambda can express.

Because the response strategy now belongs to the caller, @Wrapped on a NetFlowCall method is a compile error, with a message that says why:

Function 'todosCall' returns NetFlowCall — @Wrapped is the caller's
response-strategy choice, not the interface's.

@Paginated fails the same way, since it requires a Flow<PagingData<T>> return type it will never see. I would rather refuse the annotation than silently pick a strategy on someone's behalf.

prepareCall { } is public, so hand-written DSL users get the same split without touching annotations.

Rung three: the call {} DSL

The top rung is unchanged. Remote plus local paging, localQuery with its count and items and invalidation lambdas, retry policies with a retryOn predicate, per-request headers computed at call time, onlyLocalCall for offline reads. The paged call from two sections ago, with a database under it, looks like this:

// commonMain
@OptIn(ExperimentalPagingApi::class)
override fun getTodos(): Flow<PagingData<Todo>> = client.call {
    path = "todos"
}.responsePaginated<TodoDto, Todo> {
    localSource(
        pagingSource = { TodoPagingSource(database) },
        transform = { it.toModel() }
    )

    deleteOnRefresh = false
    insertAll(transform = { it.toEntity() }) { todos ->
        database.todoQueries.transaction {
            database.todoQueries.deleteTodos()
            todos.forEach { database.todoQueries.insertTodo(it) }
        }
    }

    firstItemDatabase(
        itemDatabase = { database.todoQueries.getFirstTodo().executeAsOneOrNull() },
        timestamp = { it.lastUpdatedTimestamp }
    )
}

There is no annotation on earth that expresses that, and I did not try to invent one. The top of the ladder is supposed to look like this.

Three rungs, or three libraries

The alternative to layering is what most projects already do. Ktorfit for the HTTP calls, Store or something hand-rolled for the cache, Paging wired up separately with its own RemoteMediator. It works. I have shipped it.

What it costs is coherence. Three libraries mean three sets of state types that don't compose, three failure models, and three test setups. Your Ktorfit call throws, your cache layer returns a sealed class, your paging layer emits LoadState, and the repository is the place where you write the glue that reconciles all of it. Then you write a MockEngine for one, a fake for another, and an in-memory database for the third.

In NetFlow the three rungs return the same ResultState, AsyncState, and PagingData types. They run on the same NetFlowClient. And they are tested with the same MockNetFlowClient, which intercepts requests without any network at all:

@Test
fun `delete removes item from local database`() = runTest {
    val mockClient = MockNetFlowClient { _ -> NetFlowMockResponse.success() }
    val repository = TodoRepositoryImpl(mockClient, database)

    repository.deleteTodo(id = 1)

    mockClient.assertCalled("todos/1", HttpMethod.Delete)
}

That test doesn't care which rung the repository used. Swap getTodos from an annotated method to a call {} block and the test still passes, because the boundary is the client, not the call style.

Now the honest part. If you just want to use annotations, Ktorfit beats this comfortably. It has years of production use, multipart and form-data, streaming, response converters, a much larger vocabulary, and a community that has already hit the bugs you are about to hit. NetFlow 0.7.0 has a first pass and a roadmap item that still says multipart. If HTTP is genuinely all you need, use Ktorfit.

The exception is the one I spent a whole section on. No annotation layer hands you a Flow<PagingData<T>> from commonMain, because that requires the paging integration to already be part of the same library. That is not me out-annotating Ktorfit, it's the stack underneath the annotations showing through, which is the only bet I am actually making.

Generating into commonMain with KSP

This is the part I found least documented, so here it is in full.

The processor emits one file per annotated interface into the common metadata source set, and every target then compiles that file as ordinary common code. On the consumer side it is four pieces of wiring:

// build.gradle.kts
plugins {
    alias(libs.plugins.ksp)
}

kotlin {
    sourceSets {
        commonMain {
            kotlin.srcDir("build/generated/ksp/metadata/commonMain/kotlin")
        }
        commonMain.dependencies {
            implementation("io.github.kmpbits:netflow-annotations:<latest_version>")
        }
    }
}

dependencies {
    add("kspCommonMainMetadata", "io.github.kmpbits:netflow-ksp:<latest_version>")
}

tasks.withType<KotlinCompilationTask<*>>().configureEach {
    if (name != "kspCommonMainKotlinMetadata") {
        dependsOn("kspCommonMainKotlinMetadata")
    }
}

The srcDir line and the dependsOn block are the two that bite. Without srcDir, the generated file exists on disk and no source set can see it. Without the task dependency, your Android compilation happily starts before KSP has produced anything, and you get an unresolved reference to createTodoApi that disappears on the second build. That intermittent-failure flavour is the worst kind to debug, which is exactly why it is worth ten lines of Gradle.

One detail worth mention: netflow-ksp has no dependency on netflow-paging. The processor emits responsePaginated as a fully qualified name string and lets the consumer's own classpath resolve it. If you never return Flow<PagingData<T>>, you never pull in Paging 3, and the annotation module stays a pure Kotlin Multiplatform artifact with no Android-flavoured transitive weight.

I picked KSP over a compiler plugin for one reason: the output is a plain .kt file you can open. When a generated call sends the wrong query parameter, you go look at the generated line and see it. A compiler plugin gives you IR, a stack trace, and a bad afternoon. Ktorfit itself ran on KSP for years and only moved to a compiler-plugin-only model after it had the user base to absorb that opacity. For a library at 0.7.0 with three stars, readable output is worth more than whatever the plugin route buys.

The line I hold

Adding a familiar surface to an unfamiliar API is not a compromise, as long as the surface refuses to lie about what is underneath it. Annotations in NetFlow do not wrap the DSL, they generate it, they return the same types, they run on the same client, they break at compile time when you ask them for something they cannot do, and they stop at the exact line where a lambda becomes necessary. That stopping point is the design, not a gap in it.

A layer that quietly did less checking so it could cover more cases would be a worse library with a better README.

This is 0.7.0, and the honest framing is that I have no external users yet. The annotation surface may well shift in 0.8 or 0.9 once somebody who is not me tries to model an API with it. Read the code samples here as 0.7.0, not as a promise.

But the shape I would defend regardless of version: put the familiar entry on the front, keep the harder API reachable from it without changing cars, and be loud about where one ends and the other begins.

Pit lane or full circuit, it is the same car. 🏁

The library is available on GitHub.

Cloud Computing Is Just npm install for Infrastructure

Sun, 16 Aug 2026 08:44:28 +0000

Hello, I'm Ganesh. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star git-lrc on GitHub to help more developers discover the project. Do give it a try and share your feedback for improving the product.

Every time you pull in a third-party package instead of writing something yourself, you make a trade: you stop owning the internals, but you're still on the hook for how you use it, what it costs you at scale, and what happens the day it breaks in production.

Cloud computing runs on the exact same trade: you're just importing servers, storage, and networking instead of a library.

That mental model answers the questions that actually matter once you move past the marketing pitch: If I don't manage the OS anymore, what am I still on the hook for? Do I self-host, use a managed service, or mix both? Why does my bill move even when my code didn't change? And what happens when the dependency I'm relying on goes down, gets slow, or degrades under load?

In this article, I'll walk through the 5 concepts that answer those questions.

1. The Shared Responsibility Model

While cloud providers manage a significant portion of the infrastructure, customers still have important security and operational responsibilities.

This concept is known as the Shared Responsibility Model, and the division of duties changes depending on whether you're running workloads On-Premises, or using IaaS, PaaS, or SaaS.

On-Premises: You Manage Everything

When applications run in your own data center, every component—data, devices, accounts, IAM, applications, network controls, operating systems, physical servers, networking, and the datacenter itself is your responsibility.

Since there's no cloud provider involved, there's nothing to share.

Infrastructure as a Service (IaaS)

Examples include virtual machines such as Amazon EC2, Azure Virtual Machines, and Google Compute Engine.

The provider supplies the physical infrastructure, hosts, networking, and datacenters, while you manage data, devices, accounts, IAM, applications, network controls, and operating systems.

IaaS offers the most flexibility of the cloud models but also requires the most operational management.

Platform as a Service (PaaS)

Examples include Azure App Service, AWS Elastic Beanstalk, Google App Engine, and Heroku.

PaaS removes the burden of managing operating systems and infrastructure. Customers still manage data, devices, and accounts; IAM, applications, and network controls are shared; and the provider manages the OS, infrastructure, hosts, networking, and datacenters.

Software as a Service (SaaS)

Examples include Microsoft 365, Salesforce, Google Workspace, and Dropbox.

Customers manage data, devices, and accounts, with IAM as a shared responsibility.

The provider manages everything else—applications, network controls, OS, infrastructure, hosts, networking, and datacenters making SaaS the model with the lowest operational overhead.

Shared Responsibility at a Glance

Service Model	Customer Manages	Shared	Provider Manages
On-Premises	Everything	None	Nothing
IaaS	OS, applications, IAM, network controls, data	None	Infrastructure & physical hardware
PaaS	Data, devices, accounts	IAM, applications, network controls	OS, infrastructure & physical hardware
SaaS	Data, devices, accounts	IAM	Applications, OS, infrastructure & physical hardware

No matter which model you choose, your data, devices, and user accounts remain your responsibility.

As you move from On-Premises → IaaS → PaaS → SaaS, the provider takes on progressively more of the operational burden but protecting users, managing access, and securing data always stay in your hands.

2. Cloud Deployment Models: Public, Private, and Hybrid

A cloud deployment model defines where your infrastructure is hosted, who owns it, and who can access it.

The three primary models are:

Public
Private
Hybrid

each offer different levels of flexibility, security, and control.

Public Cloud

Owned and operated by a provider such as Microsoft Azure, AWS, or Google Cloud Platform, with infrastructure shared among multiple customers.

Characteristics: no hardware to maintain, rapid provisioning, pay-as-you-go pricing, high scalability, global availability.
Advantages: lower upfront costs, rapid deployment, automatic maintenance, high availability ideal for startups and growing businesses.
Limitations: less control, shared environment, potential compliance restrictions.
Best for: web applications, mobile backends, dev/test environments, AI/ML workloads, disaster recovery.

Private Cloud

Dedicated to a single organization, hosted either in-house or by a third-party provider, with no resource sharing across organizations.

Characteristics: dedicated infrastructure, greater administrative control, enhanced security, custom networking.
Advantages: better compliance support, increased data privacy, full infrastructure control, custom security policies.
Limitations: higher costs, requires skilled administrators, slower scaling.
Best for: government, healthcare, banking and finance, and other heavily regulated enterprises.

Hybrid Cloud

Combines public and private environments, letting workloads and data move between them as needed This keeps sensitive workloads private while using the public cloud for scale.

Characteristics: combines both models, flexible workload placement, supports gradual migration, optimizes cost and performance.
Advantages: flexibility, business continuity, disaster recovery, cost optimization, easier migration from on-premises.
Limitations: more complex architecture, requires strong networking and identity management, more governance overhead.
Best for: large enterprises, seasonal traffic spikes, backup/DR, organizations migrating to the cloud.

Quick Comparison

Feature	Public Cloud	Private Cloud	Hybrid Cloud
Ownership	Cloud provider	Single organization	Both
Cost	Low upfront	Higher	Moderate
Scalability	Excellent	Limited by hardware	Excellent
Security	Good	Very High	High
Maintenance	Provider	Organization	Shared
Flexibility	High	Moderate	Very High

Example: An e-commerce company might run its customer-facing website on the public cloud to handle traffic spikes during sales events, store payment records in a private cloud for compliance, and connect both through a hybrid architecture for secure data exchange balancing security, performance, and cost.

3. The Consumption-Based Pricing Model

Instead of purchasing expensive servers and networking equipment upfront, the cloud lets businesses pay only for what they use, an approach known as the consumption-based model, or pay-as-you-go pricing. Think of it like an electricity bill: you pay for the units you consume, not a fixed monthly amount.

Traditional Infrastructure vs. Cloud Consumption

Traditional Infrastructure	Consumption-Based Cloud
Large upfront hardware investment	No upfront infrastructure purchase
Pay even when servers are idle	Pay only for resources you use
Capacity planning required	Scale resources up or down instantly
Hardware maintenance is your responsibility	Cloud provider manages infrastructure

Key Benefits

No upfront costs : start using services immediately without buying hardware or licenses.
Pay only for what you use : compute hours, storage, database transactions, data transfer, and function executions are all billed based on actual consumption.
Scale when needed : resources grow during demand spikes and shrink afterward, an elasticity that's expensive to replicate on-premises.
Better cost optimization : usage-based billing encourages practices like shutting down unused VMs, right-sizing instances, using autoscaling, and monitoring spend regularly.

Consumption-Based vs. Subscription-Based Pricing

Consumption-Based	Subscription-Based
Pay for actual usage	Fixed monthly or yearly fee
Flexible costs	Predictable costs
Ideal for changing workloads	Best for consistent workloads
Easy to scale	Usually includes predefined resource limits

An e-commerce site running a consumption-based model keeps only a few virtual machines running on normal days, automatically adds instances during festival sales, and scales back down once traffic normalizes, paying only for the extra capacity while it's actually in use.

To avoid cost surprises, it's worth setting spending budgets, enabling billing alerts, monitoring usage, and reviewing bills regularly.

4. High Availability vs. Scalability

Two qualities determine whether users get a smooth experience in the cloud: high availability and scalability. They're often mentioned together, but they solve different problems.

High Availability

High availability (HA) is a system's ability to remain operational even when failures occur. Rather than preventing every failure, cloud platforms detect problems and automatically shift workloads to healthy resources. Providers achieve this through redundant infrastructure, multiple data centers and Availability Zones, automatic failover, and SLAs that define expected uptime.

Example: if a server crashes during an online banking transaction, traffic is redirected to a healthy instance so customers don't lose access to their accounts.

Scalability

Scalability is the ability to handle increasing or decreasing workloads by adjusting resources—a matter of capacity rather than uptime.

Vertical scaling (scale up): add more CPU, RAM, or faster storage to an existing machine—simple, but limited by hardware ceilings.
Horizontal scaling (scale out): add more instances and distribute traffic across them preferred for cloud-native applications, and often automated based on CPU, memory, or request volume.

High Availability vs. Scalability

High Availability	Scalability
Keeps services running during failures	Handles increasing workloads efficiently
Focuses on minimizing downtime	Focuses on increasing capacity
Uses redundancy and failover	Uses additional computing resources
Improves reliability	Improves performance under load

A movie ticket booking platform sees a traffic surge when tickets for a blockbuster release, the application automatically adds servers to handle demand (scalability), and if one server fails mid-rush, requests are redirected to healthy servers without interrupting users (high availability).

A well-designed cloud application aims for both.

5. Reliability and Predictability: Building Trust Beyond Uptime

Hosting an application in the cloud doesn't, by itself, guarantee a great experience. Users expect applications to stay available, recover quickly from failures, and perform consistently regardless of workload—expectations built on two more fundamental principles: reliability and predictability.

Understanding Reliability

Reliability is a system's ability to continue operating, or recover quickly, when failures occur. Hardware failures, software bugs, network outages, and even data center outages are inevitable in distributed systems, so rather than trying to eliminate every failure, cloud platforms are designed to detect, isolate, and recover from them automatically, distributing applications across multiple VMs, Availability Zones, or regions so that if one component goes down, another takes over with minimal disruption.

For example, if a virtual machine crashes unexpectedly, the platform can automatically provision a replacement and restore service without manual intervention.

Understanding Predictability

Predictability is about delivering consistent application behavior, performance, and costs over time, giving teams confidence that infrastructure will respond as expected under both normal and peak workloads. Providers support this through tools that monitor infrastructure health, analyze workload patterns, estimate future resource needs, and automatically adjust capacity. The result:

Stable application response times
Consistent system behavior
Predictable infrastructure costs
Better resource planning
Improved operational efficiency

Reliability vs. Predictability

Reliability	Predictability
Focuses on recovering from failures	Focuses on maintaining consistent performance and behavior
Ensures applications remain available	Ensures applications perform consistently over time
Achieved through redundancy, failover, and fault tolerance	Achieved through monitoring, autoscaling, analytics, and capacity planning
Measured by availability, resilience, and recovery	Measured by performance consistency, scalability, and cost stability

Reliability answers: "Will the application continue running if something fails?" Predictability answers: "Will the application continue performing as expected as demand changes?"<

Cloud-Based Network Monitoring: How It Works, Key Metrics, and Best Practices

Thu, 06 Aug 2026 15:01:53 +0000

Cloud-based network monitoring enables organizations to monitor, manage, and optimize their network infrastructure remotely through cloud-hosted platforms. It provides real-time visibility into network performance, device health, traffic patterns, and security events, allowing IT teams to quickly identify and resolve issues before they impact business operations. Compared to traditional on-premises monitoring, cloud-based solutions offer greater scalability, flexibility, centralized management, and reduced infrastructure costs, making them well suited for modern hybrid and distributed environments.

Successful cloud-based network monitoring depends on selecting the right monitoring platform, maintaining comprehensive visibility across cloud and on-premises resources, configuring real-time alerts, and continuously analyzing performance data. By adopting proactive monitoring practices, organizations can improve network reliability, strengthen security, optimize resource utilization, and ensure consistent performance as their infrastructure grows and evolves.

Source: This summary is based on the blog "Cloud-Based Network Monitoring" by PathSolutions.

Blog URL: https://www.pathsolutions.com/blog/cloud-based-network-monitoring

Understanding Multiple Input and Output Neural Network

Wed, 15 Jul 2026 13:17:23 +0000

In the previous article, we discussed how ReLU activation function works. Now let's see how multiple input and multiple output neural networks work.

How Multiple Input and Output Neural Network works

Until now, we worked with a neural network that had a single input and a single output. In real-world problems, we usually have multiple inputs and multiple outputs.

In this article, we will build a neural network with:

2 inputs (input1 and input2)
1 hidden layer with 2 neurons
3 outputs (output1, output2, output3)
ReLU as the activation function

Network Structure

input1 ──┐
         ├──► hidden_neuron1 ──┬──► output1
input2 ──┤                     ├──► output2
         └──► hidden_neuron2 ──┴──► output3

Each input is connected to each hidden neuron, and each hidden neuron is connected to each output neuron.

Forward Pass Equations

All weights and biases are assigned based on normal distribution.
Hidden Layer Calculation
For hidden neuron 1:

x1 = (input1 * w1) + (input2 * w2) + b1
y1 = ReLU(x1) = max(0, x1)

For hidden neuron 2:

x2 = (input1 * w3) + (input2 * w4) + b2
y2 = ReLU(x2) = max(0, x2)

Output Layer Calculation

For output1:

output1 = (y1 * w5) + (y2 * w6) + b3

For output2:

output2 = (y1 * w7) + (y2 * w8) + b4

For output3:

output3 = (y1 * w9) + (y2 * w10) + b5

Example with Numbers

Let's work through a concrete example.

Given inputs:

input1 = 2
input2 = 3

Assume the following initial weights and biases:

Hidden layer weights:
  w1 = 0.5,  w2 = -0.3,  b1 = 0.1
  w3 = -0.4, w4 = 0.8,   b2 = 0.2

Output layer weights:
  w5 = 0.6,  w6 = 0.7,  b3 = 0.1
  w7 = -0.5, w8 = 0.4,  b4 = 0.2
  w9 = 0.3,  w10 = -0.6, b5 = 0.0

Calculate hidden neuron values

Hidden neuron 1:

x1 = (2 * 0.5) + (3 * -0.3) + 0.1
   = 1.0 - 0.9 + 0.1
   = 0.2

y1 = ReLU(0.2) = max(0, 0.2) = 0.2

Hidden neuron 2:

x2 = (2 * -0.4) + (3 * 0.8) + 0.2
   = -0.8 + 2.4 + 0.2
   = 1.8

y2 = ReLU(1.8) = max(0, 1.8) = 1.8

Calculate outputs

Output 1:

output1 = (0.2 * 0.6) + (1.8 * 0.7) + 0.1
        = 0.12 + 1.26 + 0.1
        = 1.48

Output 2:

output2 = (0.2 * -0.5) + (1.8 * 0.4) + 0.2
        = -0.10 + 0.72 + 0.2
        = 0.82

Output 3:

output3 = (0.2 * 0.3) + (1.8 * -0.6) + 0.0
        = 0.06 - 1.08 + 0.0
        = -1.02

Final Results

output1 = 1.48
output2 = 0.82
output3 = -1.02

Why ReLU works here

Notice that x1 = 0.2 and x2 = 1.8 — both are positive, so ReLU passes them through unchanged.

If any x value were negative (say x1 = -0.5), then:

y1 = ReLU(-0.5) = max(0, -0.5) = 0

That neuron would contribute nothing to the outputs, effectively "turning off" and making the network sparse and efficient.

Matrix Representation

You can think of all the weights as a matrix of connections:

Hidden Layer (2x2 weight matrix + 2 biases):

  w1   w2   b1        w3   w4   b2
[ 0.5  -0.3  0.1 ]  [ -0.4  0.8  0.2 ]


Output Layer (3x2 weight matrix + 3 biases):

  w5   w6   b3
[ 0.6  0.7  0.1 ]   → output1

  w7   w8   b4
[-0.5  0.4  0.2 ]   → output2

  w9  w10   b5
[ 0.3 -0.6  0.0 ]   → output3

Each output neuron learns a different combination of the hidden neuron outputs, allowing the network to produce multiple independent predictions.

Conclusion

We now understand how a neural network with 2 inputs and 3 outputs works step by step using the ReLU activation function:

Each hidden neuron receives all inputs, computes a weighted sum plus bias, and applies ReLU.
Each output neuron receives all hidden neuron outputs and computes its own weighted sum plus bias.
The network can produce multiple distinct outputs simultaneously from the same inputs.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

⭐ Star git-lrc on GitHub

Understanding Backpropagation: How Neural Networks Learn from Their Mistakes

Sun, 05 Jul 2026 07:23:45 +0000

From Linear Regression to Gradient Descent

In previous parts we discussed about linear regression, gradient descent and how to calculate the optimal slope and intercept using Gradient Descent.

In this article, we'll build an intuition for backpropagation, understand why it is necessary, and explore how the chain rule and gradient descent work together to enable neural network learning.

What is Backpropagation?

Backpropagation is an algorithm used to train neural networks.

It is a way to adjust the weights and biases of a neural network to reduce the error between the predicted output and the actual output.

In simple terms, it is a way for the neural network to learn from its mistakes.

This learning process is powered by an algorithm called backpropagation, one of the most important concepts in machine learning. Backpropagation provides a systematic way for a neural network to determine which internal parameters contributed to an error and how those parameters should be updated to reduce future mistakes.

Imagine teaching a student to solve math problems.

After each attempt, you compare the student's answer with the correct one, identify where mistakes occurred, and provide feedback. Over time, the student adjusts their approach and improves. Backpropagation works in a similar way: the network makes a prediction, calculates the error, traces that error backward through the network, and updates its parameters accordingly.

At its core, backpropagation combines two fundamental mathematical ideas:

The Chain Rule from calculus, which helps determine how changes in one part of the network affect the final error.
Gradient Descent, an optimization technique that uses those calculations to update the network's parameters in the direction that reduces error.

By repeatedly answering these questions across thousands or millions of training examples, neural networks gradually learn patterns hidden within data and become increasingly accurate.

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.

⭐ Star git-lrc on GitHub

Introduction to Neural Networks

Sun, 31 May 2026 18:12:12 +0000

Hello, I'm Ganesh. I'm building git-lrc, a free, micro AI code review that runs on commit. It is free, and source-available on GitHub. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

In this series of articles, we will explore the world of neural networks and how they work.

In this article, we will discuss the fundamental idea of neural networks.

What is a Neural Network?

A neural network is a computational model that is inspired by the structure and function of the human brain.

It is widely used in many fields like image recognition, natural language processing, etc.
It is most popular machine learning algorithm.

Simplest image of a Neural Network is :

What Neural Networks Manily Do?

It tries to approximate the function of the given input and map it to the output.

In simple terms, it tries to learn the relationship between the input and output. It approximates the output based on the input data.

This will be be done with basic mathematical functions which totaly changes based on the data we provided.

For example:

Let's take example of Drug which can cure illness.

Now let's split the gruop of people based on dosage of drug they consume.

Low dosage: Group A
Medium dosage: Group B
High dosage: Group C

Most Commonly Gruop results may be ploted as this in the graph.

To predict the efficacy of the dosage of drug we can't simply use simple straight line or normal distribution.

As for example I ploted this graph for understaning on how it is done.

This is where nerual network will help.

If we have data points in different location it will find and draw a line to predict the values between the points.

By above graph we can able to understand what neural network can do.

Conclusion

Neural Network is the best tool to fit the data points which are scattered with curve which we can use it to predict the values in between the data points.

We got to know with basic example what it actualy do.

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.
⭐ Star it on GitHub: https://github.com/HexmosTech/git-lrc

Spectral Stability and Amplification Propagation in Interacting Agent Networks: (CAIT Part=4)

Sat, 16 May 2026 09:08:32 +0000

I donot want to bore anyone with technical details. IN short i can say THIS IS A FATHER of ALL RESEARCH PAPERS EVER PRODUCED! People generally do this much in their life time, Dedicated to all who are connected to me in any capacity. Thank you!

1. Global Operator Model

Meaning:
A single mathematical engine that describes how all agents (humans + AI) influence each other.

Business usefulness:
Lets you model organization‑wide behavior (e.g., productivity, risk, misinformation spread) as one system instead of isolated departments.

2. Kronecker Jacobian (Network × Local Dynamics)

Meaning:
Separates what comes from individual behavior vs. what comes from the network structure.

Business usefulness:
Helps leaders see whether a problem is caused by:

individual performance

team structure

communication patterns

AI‑human interaction loops

This is the math behind root‑cause analysis at scale.

3. Spectral Stability Law

Meaning:
Stability = whether small problems die out or grow.

Business usefulness:
Predicts when:

a rumor becomes a crisis

a productivity issue becomes systemic

a small AI error cascades across teams

This is early‑warning detection for organizational instability.

4. Critical Coupling Law

Meaning:
There is a threshold where interactions become “too connected” and the system destabilizes.

Business usefulness:
Shows when:

too much collaboration slows the company

too much AI‑automation creates fragility

too many dependencies create bottlenecks

This is the math behind optimal team size, optimal communication density, and optimal AI integration.

5. Fixed‑Point Equilibrium Theory

Meaning:
Describes the long‑run steady state of the organization.

Business usefulness:
Predicts:

the stable productivity level

the stable inequality level

the stable AI‑human workload distribution

This is the math behind forecasting and scenario planning.

6. Heterogeneity Propagation

Meaning:
Small differences between people or teams get amplified by the network.

Business usefulness:
Explains why:

small skill gaps become large performance gaps

minor misinformation becomes major divergence

small AI‑tool differences create large outcome differences

This is the math behind inequality, polarization, and performance divergence.

7. Topology‑Driven Bifurcation

Meaning:
The network can suddenly shift from “everyone similar” to “teams behaving very differently.”

Business usefulness:
Predicts tipping points such as:

sudden culture fragmentation

sudden productivity divergence

sudden AI‑adoption gaps

This is the math behind organizational tipping points.

8. Perturbation Expansions

Meaning:
You can compute how small changes affect the whole system.

Business usefulness:
Allows sensitivity analysis:

“If we increase AI usage by 5%, what happens?”

“If we reduce communication load by 10%, does stability improve?”

This is the math behind policy simulation and optimization.

9. Spectral Shift Formula

Meaning:
Quantifies exactly how network structure changes stability.

Business usefulness:
Lets you redesign:

reporting lines

communication flows

AI‑tool integration patterns

to make the organization more stable.

10. Inequality Curvature Functional

Meaning:
Measures how inequality grows as the system becomes unstable.

Business usefulness:
Predicts:

when skill inequality will accelerate

when AI‑tool inequality will widen

when performance gaps will explode

This is the math behind DEI risk, talent management, and fairness analysis.

11. Stability Margin Reduction Law

Meaning:
Networks always reduce stability unless carefully designed.

Business usefulness:
Explains why:

fast‑moving companies become fragile

hyper‑connected teams burn out

AI‑heavy workflows collapse under stress

This is the math behind organizational resilience.

12. Universal Spectral Law

Meaning:
All phenomena — stability, inequality, bifurcation, synchronization — come from one spectral mechanism.

Business usefulness:
Gives leaders a single unifying framework to understand:

productivity

risk

inequality

AI‑human interaction

organizational design

This is the math behind enterprise‑wide decision intelligence.

Ultra‑Short Version (MBA‑level)
Your paper gives a business leader the ability to:

predict when systems become unstable

detect early warning signals

understand how small issues scale

design better teams and workflows

manage AI‑human interactions safely

prevent inequality and performance divergence

optimize communication and collaboration

simulate organizational scenarios

It is essentially a mathematical engine for organizational strategy, risk, and AI governance.

https://explore.openaire.eu/search/result?pid=10.5281/zenodo.20228867

How a Browser Works: A Beginner-Friendly Guide to Browser Internals

Fri, 13 Mar 2026 12:34:31 +0000

Introduction: A Simple Action, a Complex Story

You type a URL.
You press Enter.
A webpage appears.

It feels instant. Effortless.

Nothing special, right?!!

But under the hood, that single key press triggers a carefully coordinated chain of events involving networking, parsing, layout calculations, and rendering logic — all happening in milliseconds or maybe less than that.

Most beginners assume a browser’s job is simple: “It opens websites.”

That’s not wrong — but it’s wildly incomplete.

A browser is not just a viewer. It’s a system (or maybe we can say equivalent to an Operating System itself). The one that fetches files, understands languages like HTML and CSS, converts text into structured data, calculates layouts, and finally paints pixels onto your screen.

In this guide, we’ll slow down that moment.

We won’t dive into specifications.
We won’t memorize components.
We’ll follow a story — from typing a URL to seeing the pixels.

When you press Enter, the browser doesn’t jump straight to drawing the page.
It follows a pipeline — a sequence of steps — where each stage prepares data for the next.

Let’s break that system down, and by the end, browsers will feel less like magic and more like well-designed machines doing exactly what they’re supposed to do.

What Is a Browser, Really?

At a high level, a browser is a platform that does four core jobs:

Provides a user interface (tabs, address bar, buttons)
Fetches resources from the network
Understands web languages (HTML, CSS, JavaScript)
Converts those instructions into pixels on your screen

Think of a browser like a restaurant:

The menu and tables are what you interact with
The kitchen processes raw ingredients.
The chefs follow instructions.
The final dish is what gets served.

The important takeaway:
A browser is a software application, a collection of cooperating components that is used to translate HTML into visual data, not a single program.

Main Parts of a Browser

Here are the major players within the browsers:

User Interface

Whatever you see in the browser except the window where the web page is rendered, is known as the UI of browser. The address bar, tabs, refresh button, back button — these are part of the browser’s control panel.

What’s important to understand is this:

The UI doesn’t render websites.

It simply accepts your input and hands off the real work to other components.

When you type a URL and press Enter, the UI says:

“Here’s the request. Handle it.”

Browser engine:

Browsers engine is the brain of the browser. It manages:

Navigation: Back & Forth, the reloads
Communication: Managing the flow between the UI layer & disk storage.
Session management: it keeps track of sessions (from start to end), state across different tabs.

Example: Trident(Internet Explorer, Edge), Blink(Chrome, Opera, Brave), Webkit(Apple's Safari), Gecko(Firefox) etc.

Rendering engine

The rendering engine is responsible to make HTML content visible.
Rendering engine does the most heavy tasks like creating the DOM, CSSOM using parser, render tree, reflows, paint.

It listen the commands of the browser engine like reload, refresh etc.

Networking

Once the browser engine receives your URL, it asks the networking layer to fetch resources. It handles the DNS resolutions, request data from server using http/https

Then, the browser requests for:

HTML files
CSS files
Images, fonts, and more

At this stage, the browser doesn’t care how anything looks.
It’s just collecting raw materials.

JS Interpreter

This is the main layer that is responnsible to interpret and parse the JavaScript codes. Some JS engines are V8(used by NodeJS), SpiderMonkey(used by Mozilla Foundation; Firefox), JavaScriptCore(used by Apple, Bun), Chakra(Used by Microsoft Edge) etc.

UI backend

UI backend does the actual work of displaying the HTML content in the browser by lighting up the pixels and setting up the respective backends like what will happen after clicking on "About" button will be set up by thhis llayer.

Rendering engine creates layout (reflow) and then in the paint process creates instructions and give it to UI backend.

Disk API

It is used to save data into Browser cache, cookies, local storage, session storage etc.
It also writes data into your computer's browser folder using some internal method.

This is the high level overview of the browser components that what each component is responsible for.

How Browser Actually works?

Now, let's deep dive into how the components work together to render a website. I will again start with the same question: what happens when we write an URL on the address bar and Enter?

As you press Enter after entering URL:

The browser engine commands the networking layer for DNS resolution.
The networking layer finds the IP address of the server, then a handshake connection is established. After connecting to the server, networking layer requests data using HTTP or HTTPs.
As the first packet of data is recieved, networking layer sends the packet to the rendering engine.
Now the rendering engine sends HTML codes to tokenizer to covert codes to tokens and then parse the HTML codes.

Now, it’s clear how data is fetched to browser after entering the URL. Now we are going to deep dive into how the HTML and CSS are getting passed, how the reflux is generated, and ultimately how that display comes in front of us.

HTML Parsing: From Text to DOM

HTML arrives as plain text.
The browser doesn’t see “headings” or “sections” yet — it sees characters.

So it parses the HTML:

Reads it token by token
Understands tags and nesting
Builds a structured representation called the DOM

What Is the DOM?

The Document Object Model (DOM) is a tree-like structure where:

Each HTML element becomes a node
Parent-child relationships are preserved

Think of the DOM as a family tree of elements.

CSS Parsing: From Styles to CSSOM

CSS also starts as text.
The browser parses CSS into another structure called the CSSOM (CSS Object Model).
This structure stores:

Selectors
Properties
Values
Cascading and inheritance rules

Important note:

The CSSOM doesn’t decide positions — it only defines how things should look.

Content Sink / DOM mutation observer

You can think content sink as the collector and dispatcher. As the DOM nodes are parsed, it gets the reference of that node and sends that to the render engine to build the render tree.
Content sink is also known as DOM Mutation Observer because as the DOM gets updates, it observes that and the render tree is built again.

DOM + CSSOM → Render Tree

Now the browser has:

Structure (DOM)
Styling rules (CSSOM)

It merges them into the Render Tree / Visual Tree. The render tree:

Contains only visible elements
Includes computed styles
Is optimized for layout and painting

You can think of it as:

The DOM dressed in CSS, ready to be drawn.

Now we have completed an important checkpoint of creating a render tree. Now, we are in the final step where the browser will use render tree nodes and display them in the browser. Here are the further steps:

Frame Construction

After the render tree is created, the browser constructs frames. Frames do not calculate the final size or position of elements. Instead, they define how each render tree node participates in layout—whether it behaves as a block, inline, flex item, or grid item, and how it interacts with surrounding elements.

The browser must "re-construct" this frame up to 60 times per second so that scrolling, animations, and interactions feel smooth rather than static.

Reflow (Frame Tree)

This is the step where the browser calculates the width, length and the positions (x-axis & y-axis), margin, padding etc. This is the highest computational task performed by a browser in this process.

It takes the frames, viewport size as input and enables geometry to each frame. In simple terms, it creates the layout of website.

Paint

This is the second last step of rendering the website. It translate frames layout into paint instructions. In simple words, it creates the instruction for the UI backend.

UI Backend

Now at the end, UI Backend comes into the picture and talk with your OS graphics API to lights up the pixels in the screen and decides how the behind-the-scenes should actually go on.

Conclusion: Browsers Aren’t Magic — They’re Engineering

Browsers feel effortless because they’re incredibly well designed.

They hide complexity behind clean interfaces, parallel processes, and optimized pipelines. What looks like “instant loading” is actually thousands of small, precise decisions executed at high speed.

As a beginner, you don’t need to remember every component or step.
What matters is understanding that:

Browsers parse, they don’t guess
They build structures before drawing
Rendering is a process, not a single action

The next time you press Enter, you’ll know:
how a system just went to work — and did exactly what it was built to do.

Follow for more beginner-friendly breakdowns of core software engineering concepts.

DNS Explained: How the Internet Finds Websites

Fri, 13 Mar 2026 12:29:05 +0000

Introduction: How Does a Browser Know Where a Website Lives?

When you type google.com into your browser, something interesting happens behind the scenes.
Your browser doesn’t magically know where Google lives on the internet. It doesn’t even understand names like google.com at all.
What it understands are numbers—IP addresses like 142.250.190.14. So the real question becomes:

How does a human-friendly name turn into a machine-friendly address?

All these translation job is handled by DNS.

What is this DNS?

The Domain Name System (DNS) is the phonebook of the Internet. What happens in phonebook? By this you remember people by names, phones connect using numbers. Humans access information online through domain names, like google.com or chaicode.com. Web browsers interact through Internet Protocol (IP) addresses. DNS translates domain names (example.com) to IP addresses (ex: 93.184.216.34) so browsers can load Internet resources.

DNS works using the distributed hierarchical structure of servers.
Now the question is:

How does it actually work in our fingertips??

Let's find out this in our next section.

How does DNS work?

When we search something in the browser's address bar, the browser does not know where the requested content exists. What do we do when we lose our way in a new place to reach our destination? We ask questions. Browsers do exactly that. They ask someone the address of the requested content, and that someone is DNS and the request is called a DNS Query. This DNS query can be broken down into several steps, ensuring that users can access websites by simply typing a domain name into their browser. The DNS resolution works recursively.
The main motto of DNS is:

Who should I talk to next?

Everything in DNS is working based on this question. You are going to findout soon.

User Input: You enter a website address (for example, chaicode.com) into your web browser's address bar.
Local Cache Check: Your browser first checks its local cache to see if it has recently looked up the domain. It gets stored due to the DNS caching; DNS caching is a mechanism that stores DNS records locally to avoid querying external DNS servers repeatedly for the same information. This speeds up the browsing experience and reduces network traffic. If it finds the corresponding IP address, it uses that directly without querying external servers. And if it doesn't find it then it will move to the next step.
DNS Resolver Query: If the IP address isn’t in the local cache, your browser sends the request to the DNS resolver. The resolver is typically provided by your Internet Service Provider (ISP) or by your Router.
Root DNS Server: The DNS resolver sends the request to a root DNS server. There are in total 13 root Name Server around the world, but thanks to Anycast IP Addressing, which allows multiple physical severs to share the same IP, the total instances of Root Servers are more than 1700. The root server doesn’t know the exact IP address for chaicode.com but knows which Top-Level Domain (TLD) server to query based on the domain’s extension.
TLD Server: TLD stands for Top-Level-Domain, these servers manage domain extensions like .com, .org, .net, .edu, .gov and others. The TLD server for .com directs the resolver to the authoritative DNS server for chaicode.com.
Authoritative DNS Server: This server holds the actual DNS records for chaicode.com, including the IP address of the website’s server. There are many popular Authoritative DNS Servers around the world like: Cloudflare, Vercel, Netlify, Digital Ocean, Hostinger, Namecheap, Godaddy etc. It sends this IP address back to the resolver.
Final Response: The DNS resolver sends the IP address to your browser, allowing it to connect to the website’s server and load the page. Besides that for future reference, the IP address gets stored in the local cache.

So here we can see that every server is actually telling the DNS resolver that who should it talk to next to get more information. That's why the resolver is often referred as Recursive DNS Resolver.
If this feels like overwhelming, that’s okay. The key idea to remember is simple: DNS keeps asking the next right server until it finds the final answer.

What and Why the DNS Records

DNS records (aka zone files) are instructions that live in authoritative DNS servers and provide information about a domain including what IP address is associated with that domain and how to handle requests for that domain. These records consist of a series of text files written in what is known as DNS syntax. DNS syntax is just a string of characters used as commands that tell the DNS server what to do. All DNS records also have a ‘TTL’, which stands for time-to-live, which means how long a DNS record is cached before it expires. When the TTL expires, the cache is cleared and a fresh DNS query must be made.
But before diving into individual DNS records, let’s look at how they all work together for a single website.

Most common types of DNS records

DNS management relies on DNS server connections. Knowing how your DNS servers interact with each other via DNS records will make managing your DNS a less daunting task.

The most common types of DNS records are:

A Record

Address records, or A records, are the most common DNS records used. They create a direct connection between an IPv4 address and a domain name. IPv4 addresses have the following format: 93.184.216.34. This is the simplest record and this is the last cycle of a DNS query means the resolver looks for this record as its target. A record solves one simple problem: once DNS is done asking questions, this is the final answer it needs to reach the website. This is the most important IP also, because depending on this all the traffics would be sent to the server's machine. Example:

Domain	Record	Value	TTL
example.com	A	192.0.2.1	14400

AAAA Record

Same job as A records, but this type of record connects domain names to IPv6 addresses (8 groups of 4 hexadecimal digits). IPv6 addresses have more numerals than IPv4 address and are becoming more common as options for IPv4 addresses are running out. Example:

Domain	Record	Value	TTL
@	AAAA	2606:2800:220:1:248:1893:25c8:1946	14400

The "@" symbol in this example indicates that this is a record for the root domain.

CNAME Record

Canonical name records, or CNAME records, direct an alias domain to a canonical domain. This means that the CNAME record stores another domain and the A record of that domain would be the value of the former. This type of record is used to link subdomains to domain A or AAAA records. It answers the question:

“This name is just an alias—what is the real name?”

For example, instead of creating two A records for www.example.com and product.example.com, you could link product.example.com & www.example.com to a CNAME record that is then linked to an A record for example.com. The value is that if the IP address changes for the root domain, only the A record will have to be updated and the CNAME will update accordingly.
For example:

Record	Host	Value	TTL
CNAME	blog.piyushgarg.dev	hashnode.network	10400
A	hashnode.network	76.76.21.21	10400

NS Record

The most important DNS record from the list. Nameserver, or NS records, show which DNS server is responsible for managing the DNS Queries and acting as the authoritative nameserver for your domain. Authoritative nameservers contain the final information about a specific domain and its corresponding IP address. An NS record points to all of the different records your domain holds. A domain often has multiple NS records which can indicate primary and secondary nameservers for that domain. Without NS records, users will not be able to access your website.
By querying the NS record value, we will get the A record of a DNS server i.e, an IP of that DNS server which is storing our needed website details and after querying that server's IP we will get the IP address of the desired website.
NS records don’t point to your website. They point to the DNS servers that know the answers about your website.

Here is an example of an NS record:

example.com	Record	Value	TTL
@	NS	pns1.exampleserver.com	20400

Note that NS records can never point to a canonical name (CNAME) record.

MX Record

The Mail Exchanger record is used for email. This simply points to the server where the emails should be delivered for that domain name. These records, along with an email server, allow for the creation of individual email accounts, such as Emails are not allowed, that are linked to the domain (example.com) and it should be running on SMTP. The Mail Transfer Agent will query the MX record for that domain and will look for the email server, and then DNS will respond with the server details with whom the MTA should send the email.
There are in general two email servers are used with different priorities. The 'priority' numbers before the domains for the MX records indicate preference; the lower 'priority' value is preferred. The server will always try mailhost1 first because 10 is lower than 20. In the result of a message send failure, the server will default to mailhost2.

example.com	Record	Priority	Value	TTL
@	MX	10	mailhost1.example.com	40400
@	MX	20	mailhost2.example.com	40400

TXT Record

Text, or TXT records, store miscellaneous textual information related to domains and subdomains. The TXT record was originally intended as a place for human-readable notes. However, now it is also possible to put some machine-readable data into TXT records. One domain can have many TXT records.
Spammers often try to fake or forge the domains from which they send their email messages. TXT records are a key component of several different email authentication methods (Like: SPF, DKIM, DMARC) that help an email server determine if a message is from a trusted source. By configuring these records, domain operators can make it more difficult for spammers to spoof their domains and can track attempts to do so.

SPF records: SPF TXT records list all the servers that are authorized to send email messages from a domain.
DKIM records: DKIM works with digital signature - each email using a public-private key pair. This helps verify that the email is actually from the domain it claims to be from.
DMARC records: A DMARC TXT record references the domain's SPF and DKIM policies.

Conclusion: DNS Is a System, Not a Set of Memorized Records

At first glance, DNS can feel overwhelming—root servers, name servers, records with strange names, and numbers everywhere. But once you step back, a simple pattern emerges.

DNS is not magic.
DNS is a question-and-answer system.

Each DNS record exists to solve one specific problem:

NS records answer who is responsible for this domain
A and AAAA records answer where the website lives
CNAME records answer is this name just an alias
MX records answer where emails should be delivered
TXT records answer how this domain can prove or describe itself

Together, these records form a complete identity for a domain on the internet.

So the next time you type a website into your browser, remember: your browser isn’t guessing. It’s calmly asking the internet the right questions, in the right order, until it gets a clear, final answer.
And now—you know exactly how that happens.

Follow for more beginner-friendly breakdowns of core software engineering concepts.

Inside TCP: Establishment, Transfer, Reliability, and Connection Teardown

Wed, 11 Mar 2026 09:14:47 +0000

The Internet Without Rules: A Thought Experiment

Imagine two people trying to have a conversation over a call but it's a bit noisy phone line. Words can get dropped, repeated, or misunderstood — and there’s even no guarantee that the other side even heard you. That chaotic experience resembles what happens on a raw network without rules and guarantees.

Computers also face the same challenges when transferring data over networks. Without a structured communicating mechanisms, information can be lost, arrive out of order, can be duplicated or be corrupted while getting transmitted. This is where TCP (Transmission Control Protocol) comes in: a set of rules and processes that turns an unreliable network into a reliable, ordered, and correct data delivery channel.

In this blog, we go beyond definitions. We have already discussed a vast portion of the TCP protocol in my previous blog. We’ll unpack how TCP actually works — from establishing a connection to transfer data and gracefully closing it.

What TCP Solves — The Big Picture

At its core, TCP is designed to solve four fundamental problems networks introduce:

Unreliable delivery: Packets may be dropped or never arrive.
Out-of-order packets: They may arrive in a different sequence than they were sent.
Corrupted data: Bits can flip/manipulated during transmission due to noise.
data duplicacy: Data packets can be duplicated through out the transmission.

TCP addresses these with:

Connection establishment (3-way handshake)
Sequence numbering and acknowledgements
Retransmission of lost data
Flow control and error checking

This transforms an unreliable network into reliable for HTTP, email, file transfer, and interactive remote sessions.

The 3-Way Handshake: Starting a Conversation

Before meaningful data can transmit between two endpoints, TCP performs a 3-way handshake — an exchange that ensures both sides are ready for the communication.

Here’s an easy real-world analogy:

Person A says: “Hello, Can you hear me? Are you ready to talk?”
Person B replies: “Yes you are audible, I’m ready.”
Person A confirms: “Great — let’s begin.”

This three-step back-and-forth is precisely what TCP does to establish a reliable connection.

The 3-way handshake can be both sides actually, if the client issues a request then that would be Active Open, on the other hand, if the server tells that its TCP i.e, it's ready to accepts the connection, than that would be Passive Open.

Step 1 — SYN

The client sends a packet with the SYN (synchronize) flag set. This segment is for synchronization of Sequence numbers. It consumes one sequence number.
This packet includes an initial sequence number — the first byte of data the client plans to send and every time that would be incremented by 1.

Client → Server: SYN (Seq = x)

Step 2 — SYN-ACK

The server responds with a segment that has 2 flag bits set:

SYN set — to synchronize its own sequence number
ACK set — acknowledging the client’s SYN

It sends its own initial sequence number and consumes one sequence number.

Server → Client: SYN-ACK (Seq = y, Ack = x+1)

Step 3 — ACK

The client sends a final acknowledgement segment back. It's just the acknowledgement of the second segment sent from the server-side and it consumes one acknowledge number field; doesn't consume any sequence number.

Client → Server: ACK (Ack = y+1)

Once this exchange completes, both sides have synchronized sequence numbers and the connection transitions into the ESTABLISHED state. Once the handshake completes, TCP shifts from coordination to actual data delivery.

How Data Transfer Works

Once the connection is established, data is transferred in segments, each carrying:

A sequence number — to track ordering of bytes
An acknowledgement number — confirming receipt up to a certain byte
A window size — letting the other side know how much data it can send without overflow. It's byte-oriented.

Here’s how TCP manages this:

Sender sends a segment with a sequence number.
Receiver sends back an ACK indicating the next expected byte.
Sender slides its window forward and sends more data.

This process continues until all data is sent and acknowledged.

Ensuring Reliability and Correctness

TCP achieves reliability through several interlocking mechanisms:

Sequence Numbers: Every byte in a TCP stream gets a monotonically increasing sequence number. This enables receivers to reorder out-of-order packets and detect missing data.
Acknowledgements: Receivers send ACKs to confirm receipt. If an expected ACK does not arrive within a timeout, the sender assumes packet loss and retransmits.
Sliding Windows: Used to make transitions more efficient as well as to control the flow of data, so that the destination doesn't get overwhelmed by the data.
Retransmission: When data is lost (inferred by missing ACKs), TCP automatically resends the missing segments.
Catching Duplication: If a data gets tranmitted again i.e, if duplication happens then it also helps to catch it during the transmission from the receiver side.
Flow Control: By negotiating a “window size”, TCP ensures senders don’t overwhelm receivers.

Together, these mechanisms make TCP a reliable, ordered, and error-checked transport protocol.

Connection Termination: Ending the Conversation

Any of the two parties who are exchanging data, can close the connection, although in most of the cases, the termination is initiated from the client side. Most implementations allow 3-way handshaking for connection-termination.

After the transmission gets completed, the client side sends the first FIN segment to the receiver side. FIN flag set may include the last chunk of data sent by the client or it can be the control segment. It consumes one sequence number if it doesn't carry data.
The server after receiving the FIN segment informs its process about the situation and sends the second segment, a FIN + ACK segment to confirm the receipt of the FIN flagset which is sent from the client-side and at the same time to announce the closing of the connection at server. This segment can also contain last chunk of data from the server, and if it doesn't carry that, it's gonna consume one sequence number.
The client sends the last segment, an ACK segment to confirm the receipt of FIN segment that is sent from the server. It basically consumes the acknowledge number, which is 1 plus the sequence number received in FIN segment of the server. It doesn't carry data nor any sequence number.

Conclusion

TCP doesn’t just send bits. It ensures a disciplined exchange that turns the unpredictable real world of networks into a dependable communication channel. Through:

The 3-way handshake
Sequencing and ACKs
Retransmission and flow control

TCP guarantees that data arrives reliably, in order, and free of corruption.

Armed with this understanding, you’ll not only write better networking code — you’ll be able to architect systems that depend on predictable, resilient communications.

TCP vs UDP: When to Use What, and How TCP Relates to HTTP

Wed, 11 Mar 2026 09:10:19 +0000

Introduction: The Internet Runs on Rules

The internet is not magic. It’s a carefully negotiated contract between machines.

Every email you send, every video you stream, every API call your backend makes—none of it happens randomly. Data moves because rules exist that define how information is packetized, how it is delivered, and what happens when things go wrong.

At the heart of this contract sit two transport protocols: TCP and UDP.

They solve the same problem—moving data between systems—but with different philosophies. One prioritizes reliability and order, the other prioritizes speed and minimal overhead. Understanding when to use each is not academic trivia; it’s a core architectural decision that impacts performance, scalability, and user experience.

And layered on top of this is HTTP, often misunderstood, frequently misattributed, and routinely confused with TCP itself.

Let’s clear the fog in this blog!!

What Are TCP and UDP? (High-Level View)

At a very high level, TCP and UDP are transport-layer protocols. Their job is simple in theory:

Move data from one machine to another.

How they choose to do that is where everything changes.

UDP — The Fast Announcement System:

The User Datagram Protocol is also called connectionless, unreliable transport protocol.

There are no host-to-host communication, It relies on Process-to-Process communication (based on Ports). Unlike TCP, UDP does not establish a connection before sending data, nor does it guarantee delivery, order, or error correction.

Now the questions arise, "If there are no reliability, no error-handling, no ordering, then why to use it??"

Well with every disadvantages, there also comes some advantages. UDP is a simple protocol that use minimum overhead. If a process wants to send a message but doesn't care whether it's reached or not, it can use UDP. Sending a message by using UDP takes much less interaction between sender and receiver than using DCCP, SCTP, TCP making it ideal for time-sensitive applications where speed is prioritised over reliability..

UDP Header: UDP packets, also called as User Datagrams, have a fixed size header of 8 bytes.

Field	Size	Description
Source Port	16 bits	Sender’s port number.
Destination Port	16 bits	Receiver’s port number.
Length	16 bits	Total length of UDP header and data.
Checksum	16 bits	Used for error detection (optional in IPv4, mandatory in IPv6).

Unlike TCP, checksum in UDP is optional. If checksum is not calculated then the field will be filled with 1s.

Features:

Less-Reliable
No Error control & Flow control
No Retransmission
Fast
No ordering
No Congestion

Common Use Cases of UDP:

Broadcasting
Video Conferencing
Voice over IP (Services like WhatsApp voice calls, Wi-Fi calling)
Online Gamings
Live Video Streaming
Music Streaming
DNS Query/Resolution

TCP — The Reliable Courier

Transmission Control Protocol is a Connection-oriented, Reliable transport protocol.

It's a core protocol of the Internet Protocol suite (TCP/IP), responsible for reliable, ordered, and error-checked delivery of data between applications over a network. As it's a Process-to-Process protocol like UDP, therefore it uses the Port numbers. Developed in the 1970s by Vint Cerf and Bob Kahn, TCP operates at the Transport Layer (Layer 4) of the OSI model and ensures that data sent from one device reaches another correctly and in the same order it was sent.

Now a question should arise "How this Protocol can achieve the Reliability, Error Handling, Order Checking??"

A reliable connection is established between Client and Server via a Three-way Handshake before data transfer even begins. It ensures that both sides are synchronised and ready to communicate using Positive Acknowledgement with Retransmission (PAR):

Each segment sent must be acknowledged.
If a segment is lost or corrupted (detected using the checksum), the receiver discards it, and the sender retransmits.

But before knowing about 3-way handshake in detail, first we need to know about "What is Segment??"

A TCP segment consists of data bytes to be sent and a header that is added to the data by TCP as shown:

The header of a TCP segment can range from 20-60 bytes. 40 bytes are for options. If there are no options, a header is 20 bytes else it can be of upmost 60 bytes. Header fields:

Source Port / Destination Port (16 bits each): Identify sending and receiving applications.
Sequence Number (32 bits): Position of the first byte in this segment, used for ordering.
Acknowledgement Number (32 bits): Next byte expected by the receiver (confirms data received).
Header Length (HLEN): Size of the header (5–15 words, i.e., 20–60 bytes).
Window Size: Receiver’s buffer size (for flow control).
Checksum: Error detection (mandatory).
Urgent Pointer: Position of urgent data (if URG flag is set).
Control Flags (1 bit each):

URG: Urgent data
ACK: Acknowledgement valid
PSH: Push data immediately
RST: Reset connection
SYN: Synchronize sequence numbers
FIN: Terminate connection

So what actually happens in 3-way handshake is stated below step-by-step:

Step 1 (SYN): In the first step, the client wants to establish a connection with a server, so it sends a segment with SYN (Synchronize Sequence Number) which informs the server that the client is likely to start communication.
Step 2 (SYN + ACK): Server responds to the client request with SYN-ACK signal bits set. Acknowledgement(ACK) signifies the response of the segment it received and SYN signifies with what sequence number it is likely to start the segments with.
Step 3 (ACK): The client acknowledges the response of the server and they both establish a reliable connection with which they will start the actual data transfer.

Like the establishemt of the connection, Three-way Handshake is also used in connection termination. Though there's also another option available which is Four-way Handshake with half-close, but in most of the modern case implementation, 3-way handshake is used.

Features:

Reliable
Error control & Loss-detection
Flow control
Retransmission
Slow
Ordering
Congestion Control

Common Use Cases of TCP:

Web-Browsing (HTTP, HTTPS)
Text-Communication
Email (SMTP, IMAP)
File Transfer (FTP, SFTP)

What Is HTTP—and Where It Fits

The Hypertext Transfer Protocol (HTTP) is an Application level protocol that is mainly used to access the data that are on the World Wide Web (www). Basically it enables the connection in between the server and the clients (Web Browsers).
It operates on a request-response model, where a client sends a request to a server, which then responds with the requested resource, such as an HTML page(texts, image, or video). HTTP is stateless, meaning each request is independent. It runs over TCP and has evolved through versions like HTTP/1.1, HTTP/2, and HTTP/3, with each improving performance, security, and efficiency.
HTTP functions as a combination of FTP and SMTP.
HTTP uses the services of TCP on the port 80.

Understanding HTTP Request and Response

HTTP Request

An HTTP request is how your browser asks the server for something. It includes:

HTTP Version: The version of HTTP (like HTTP/1.1 or HTTP/2) that is being used.
URL: The specific address of the resource (e.g., https://www.example.com/about).
HTTP Method: The type of action being requested (e.g., GET to retrieve information, POST to send data, PUT to send document from server etc).
HTTP Request Headers: Extra information (metadata) about the request, like what kind of browser you're using or what kind of content you’re expecting.
HTTP Request Body: In some cases, the request will include a body that contains data (e.g., when you submit a form).

HTTP Response

An HTTP response is the server’s answer to your request. It includes:

HTTP Status Code: A number that tells you if the request was successful or not (e.g., 200 OK means everything is fine, 404 Not Found means the requested page doesn’t exist, 500 Internal Server Error means server site related error might be crash).
Response Headers: Information about the response (Metadata), like what kind of data is being sent (e.g., Content-Type: text/html means it’s an HTML page).
Response Body: The content that the server sends back (e.g., HTML code that the browser will use to display the webpage).

More about HTTP Methods

An HTTP method, sometimes referred to as an HTTP verb, indicates the action that the HTTP request expects from the queried server. For example,

Method	Action
GET	Client requests from the Server
HEAD	Request information about document but the not the document itself
POST	Sends information from Client to Server
PUT	Sends document from Server to Client
DELETE	Request the Sever to delete the specified resource
OPTION	Inquiry about available options

More about HTTP Status Codes

HTTP status codes are standardized three-digit numbers returned by a server in response to a client's request. They are grouped into five classes based on the first digit:

1xx (Informational): The request was received, and the process is continuing.
2xx (Success): The request was successfully received, understood, and accepted.
3xx (Redirection): Further action is required to complete the request.
4xx (Client Error): The request contains bad syntax or cannot be fulfilled.
5xx (Server Error): The server failed to fulfill an apparently valid request.

✅ Note: The most commonly encountered codes are 200 (OK), 404 (Not Found), 400 (Bad Request), 500 (Internal Server Error), and 301/302 (Redirects).

You don’t need to memorize these flags—but understanding why they exist explains how TCP guarantees order and reliability.

The Relationship Between TCP and HTTP (Deep Dive)

The Critical Clarification: HTTP Runs on Top of TCP

At a fundamental level, HTTP and TCP operate at different layers and solve different classes of problems.

TCP operates at the transport layer
HTTP operates at the application layer

This separation is intentional and foundational to how the internet scales.

When we say “HTTP runs on top of TCP”, we mean this literally in terms of layering:

HTTP does not send data directly over the network.
It relies on TCP to carry its data reliably from one endpoint to another.

HTTP is completely unconcerned with:

Packet loss
Retransmissions
Ordering of bytes
Congestion control

All of that responsibility is delegated to TCP.

Step-by-Step: What Actually Happens During an HTTP Request

When you type a URL into a browser and hit Enter, the following sequence occurs (simplified but accurate):

a. TCP Connection Establishment

The client and server perform a TCP handshake.
This creates a reliable, bidirectional communication channel.
From this point on, both sides agree on sequence numbers, window sizes, and delivery guarantees.

b. HTTP Constructs the Request

HTTP formats the request as structured text (method, headers, body).
Example: GET /api/users HTTP/1.1
HTTP does not worry about packet boundaries or fragmentation.

c. TCP Transmits the Data

TCP breaks the HTTP message into segments.
Each segment is numbered, tracked, and retransmitted if lost.
TCP ensures all bytes arrive, in order, exactly once.

d. HTTP Interprets the Response

Once TCP delivers the byte stream, HTTP parses it.
Status codes, headers, and body are interpreted at the application level.

Why HTTP Does Not (and Cannot) Replace TCP

A common misconception is assuming that HTTP could somehow “handle everything” on its own. This is architecturally incorrect.

Different Responsibilities, Different Layers

HTTP defines semantics

What does this request mean?
Is this a GET or POST?
What does this response mean?
How should the response be interpreted?

TCP defines mechanics

How do bytes move across unreliable networks?
What happens when packets are lost?
Are the packets sent in order?
How is congestion managed?

HTTP is deliberately designed without transport logic. This is not a limitation—it is a strength.

Conclusion: Protocols Are Design Decisions, Not Just Theory

TCP, UDP, and HTTP are not competing technologies—they are collaborating layers, each solving a different class of problems.

TCP exists to make unreliable networks usable
UDP exists to make real-time systems possible
HTTP exists to standardize communication on the web

Choosing between TCP and UDP is not about which protocol is “better”.
It’s about what your system values more—reliability or latency, correctness or responsiveness.
HTTP does not replace TCP.
It depends on it.
Once you understand this layering, networking stops feeling abstract. You stop memorizing protocols and start designing systems intentionally.
And that is the real goal—not passing an exam, but making informed architectural choices in real-world software.

Understanding Network Devices & their Functionalities

Wed, 11 Mar 2026 09:00:54 +0000

Introduction

Every time we connect to Wi-Fi, open any website, or use an online app, a silent conversation starts between multiple network devices. Data moves from our device, through cables and radio waves across networks we never see, and comes back in milliseconds or maybe less than that!! This entire process often feels invisible and confusing.

In this blog, we are going to try to overcome this confusion.

We will start from the very basics and build a clear mental model of how the internet actually reaches our home or office. Step by step, we will explore common network devices, such as modems, routers, switches, hubs, firewalls, and finally, we will learn about the wireless network concept that we use in our daily lives, i.e., Wi-Fi, and how the internet actually works.

Instead of memorizing definitions, we will understand why each device exists, what problem it solves, and how all of them work together in real-world systems.

By the end of this blog, we will be able to look at any network and understand what is happening behind the scenes. Let's dive into it.

Modem: Your Network’s Gateway to the Internet

This is the main thing that you need to connect your home to the internet. Without a modem, it's not possible at all. A modem establishes and maintains a dedicated internet connection to the Internet Service Provider.

A modem is mainly responsible for signal modulation or demodulation. That is, the analog signal gets converted into the digital signal, and also the digital signal gets converted into the analog signal. That's why its name is "Modular Demodulator," which is in short form "Modem." As our computer only reads digital signals and the internet uses analog signals, a modem demodulates the incoming analog signal into a digital signal and modulates the outgoing digital signal into an analog signal. That's why a modem is really very important to establish the internet connection in our home or office.

There are different types of modems in real life, and the structure and the port structure are also different. But mainly there are commonly three ports in a modem:

One power port to have continuous power flow.
The Ethernet port is used to connect the Ethernet cable which is connected with the router.
One is used for internet input, where we use optical fiber cable or coaxial cable or telephone line.

Router: The Traffic Director of the Network

The main thing that comes after the modem is Router, whose sole target is to allow multiple electronic devices in our home to share the same internet connection.

As we know that the internet service provider is providing your home with an internet connection but via a single wire only!! So how can the other devices in our home can have access of the internet?!!

And here the ROUTER comes into the picture!! The router decides where traffic goes.

Router allows internet connection in two different ways.

The Antennas through which the router provides the wireless internet access or Wifi to our home.
Using the Ethernet ports through which we can have the wired connection to the internet.
When an electronic device gets connected to the Router, the router sends a private dynamic IP address to distinguish it from other connected devices. This happens only due to the DHCP (Dynamic Host Configuration Protocol) server which also stores the table of MAC addresses of those devices that are already connected to the internet. By this IP address and the MAC address, router can decide on which device as well as at which application, the response should be sent.

There's also a table that is being used within the Router which maintains a list of known connected devices and the best path to reach them (which is calculated by different routing algorithms: Dijkstra's Algorithm, Bellman-Ford Algorithm, Floyd-Warshall Algorithm etc.). This table is known as the Routing Table.

There's also another table is also being used which is NAT (Network Address Translation) Table, which maps internal private IPs to external public IPs for internet access as our devices gets the private IPs but the internet only sees one public IP.

Modem vs. Router: What’s the Difference?

Feature	Modem	Router
Purpose	Connects to ISP	Connects multiple devices to the modem
Function	Converts analog ↔ digital signals	Distributes internet via Wi-Fi or Ethernet
Internet Access	Required for internet	Not sufficient alone
Example Use	Needed to get online	Needed to share internet across devices

Many modern devices combine both functions into a modem-router combo.

Hub vs Switch: How Local Networks Really Work

Hub is a device which is used to connect all of our network devices together on an internal network by multiple Ethernet connection. Hub is not intelligent as it only knows whether any devices are connected to any of its Ports, so it doesn't filter any data or has any intelligence as to where to send the data. So when a data packet arrives at one of the ports, it is copied to all other ports, thus all the devices that are connected to the hub receives the data packets. As it is not only a matter of security concerns but also creates unnecessary traffic in the network wasting the bandwidth, that's why in modern world, Hub is not that much used (obsolete).

In the other hand, Switch is very similar to Hub i.e, a device that has multiple ports that accepts Ethernet connections from network devices, but unlike a hub, Switch is intelligent as it actually stores the IP addresses (maps within the ARP table) as well as the physical addresses of the devices (MAC address) that are connected to the switch (maintains the CAM table). So when a data packet is sent to a switch, it's only directed to the intended destination port. Thus switches are more preferable over hubs in modern world because there will not be any unnecessary traffics over the network.

Both hubs and switches are used to exchange data within a LAN, not used to exchange data outside their own network.

To exchange data outside their own network there should be a device that needs to be able to read the IP addresses (because both switch and hub can't read IP addresses).

So here comes the Router. Router acts as the gateway of a network. If data packet is received by a router, the router detects the IP of the data packet and determines if the data packet was meant for its own network or another.

So here's the main difference between Hub & Switch:

Hub only detects that a device is physically connected to it.
But Switch can detect the specific devices that are connected to it because of the records of the MAC addresses of those devices.

Firewall: Where Security Actually Lives

A firewall is a system that is designed to prevent the unauthorised access from entering a private network by just filtering the data packets that comes in from the internet i.e, to create a safety barrier between a private network and the public internet. As we already know, in the internet there are always going to be hackers or any malicious software that may try to penetrate into our private network and that's why a firewall is needed to prevent them. Besides our home, a firewall is very much important for the large organisations or businesses as they would obviously need to protect their secured data from the internet. It works by filtering the incoming network by checking the Access Control List where the sender's IP addresses, the protocols that's being used for the data transmission, the destination's IP address, Domain names, Protocols, Ports are stored. this access control list is designed by the Network Administrator and by this it can be decided what can enter the local network and what can not. A firewall can be either software or can be hardware or it also can be the mixture of both. Firewalls are basically of two types.

Host-based Firewall: This is a software based firewall and it's installed in the computer. It's prime focus is to protect the computer only nothing else. For example: In Windows, there are already pre-installed firewall that are used. Besides that there are also some 3rd party software based firewall that can be used to protect the computer. Besides that, there are a lot of antivirus programs that have built-in host based firewall.
Network-based Firewall: The hardware based or combination of both hardware and software based firewall that are used to protect the entire network; not only focused to only one computer. It works upon the network layer and operates in between the private network and internet. As we have already mentioned, the Network-based firewall can be a stand-alone hardware device or it can also be implemented within the Router.

In the large organisations, they mostly use both the network-based and host-based firewall to ensure maximum protection.

What Is Wi-Fi — And What Actually Happens Internally

WIFI, aka IEEE 802.11, is simply a wireless communication method that allows electronic devices to talk to a local network using radio waves.

Let's discuss what actually happens.

When we switch on wifi in our phone, it shows nearby networks, which are actually the SSIDs of the networks. Now the question arises that, what is this SSID?!! Don't be overwhelmed by the fancy name; it's just the name of the network, a unique identifier for that wireless network. Most routers broadcast their SSID by default, making the network visible to nearby devices.

It seems simple that we just click on the network name we need to connect to with our phone, and then it gets connected automatically after the password verification. But it's NOT THAT SIMPLE!!

When we connect, the phone joins the network, the password gets verified & then an IP address is sent to the phone by the router. But here's a question that should arise in your mind that "How does a router get to know to which device it needs to send data or from which device it needs to receive data?"

The answer we have already discussed previously in the router section, each device gets a unique private IP address (e.g., 192.168.1.5) from the router via DHCP server. The router uses this to determine where to send incoming or outgoing data packets. Also, there are some tables that are used, to track which IP addresses are assigned to which MAC addresses(devices) (DHCP Lease Table) and a list of known networks and the best path to reach them (Routing Table).

Now let's assume you want to watch a YouTube video. From your phone, you send a request to access that YouTube video. What actually happens is that your request is at first getting divided into several data packets that are in the bits format (0/1). At first the switch checks the other connected devices for the request, if it's available in other devices then it will be fetched from there, and if not then the request will be forwarded to the Router. The router then will forward the data request to the Internet Service Provider. The Internet Service Provider will then send that data request to the YouTube server. Then the YouTube server will send the response (in data packets and bits format) to the Internet service provider. The Internet service provider will send that response to the router. The router will send that to the switch, and according to the MAC address and the port, the switch will send the response to the device that has requested for it. The device that has requested for the video will merge that data packets according to the ordering and protocols and convert that bitstream into the video format so that the user who has requested it can watch it. And when the data is moving in between the LAN and the WAN, there is always the firewall that maintains the records and checks whether the response is sent by the hackers or the malicious software. All this scenario is happening within, like, some milliseconds or less than that!!

Conclusion

Networking often feels complex not because it is magic, but because most of its work happens silently in the background. Every click, every video, and every message depends on multiple devices working together in perfect coordination — each with a very specific responsibility.

In this blog, we started from the basics and gradually built a clear picture of how data moves from our device to the internet and back. We saw how the modem connects us to the outside world, how the router manages and directs traffic, how switches and hubs handle local communication, how firewalls protect our networks, and how Wi-Fi enables wireless access to all of this infrastructure.

The key takeaway is simple: networks are layered systems. No single device does everything. Each one solves a focused problem, and together they form a reliable, scalable, and secure communication system.

This is just the foundation. With this mental model, you are now better prepared to explore deeper topics like OSI model, TCP vs UDP, DNS, cloud networking, and real-world production architectures.

And that’s where things start to get really interesting.

The Discreet Lives of Apps

Fri, 05 Sep 2025 00:25:21 +0000

Ever wonder how your phone streams Netflix, checks email, refreshes social media, and syncs your weather app—all at the same time—without mixing everything up? It feels like magic. But under the hood, your operating system is quietly running a well-organized show with something called ports.

Think of apps like secret agents living in the same embassy (your phone). Every time they “phone home” across the internet, they need a unique badge number that proves who they are. That badge number? It’s the port.

1. The Secret Identities of Your Apps

When an app sends data out into the internet, it doesn’t just say, “Hey, I’m Instagram!” Instead, it uses a port number, like 80 or 443, to signal how the data should be handled.

Port 80: the old-school door for unencrypted web traffic (HTTP).
Port 443: the secure, encrypted door for HTTPS traffic.

Without ports, it would be like a hundred spies all whispering in the same room—pure chaos. With ports, everyone knows whose message is whose.

Fun Facts

Port 443 handles over 80% of all web traffic today—that’s like being the busiest checkout lane in a store that never closes.
Tech culture nicknames ports too: Port 666 is sometimes called “doom” (after the video game), and 1337 spells leet in hacker speak.

2. The OS: Master Connection Handler

Behind the curtain, your operating system (OS) is the spymaster. It hands out ports, tracks who’s using what, and makes sure your Netflix stream doesn’t collide with your WhatsApp call.

This coordination happens through sockets—temporary pipelines that connect apps to the network.

Imagine the OS as a phone operator in the 1950s, manually plugging cables into switchboards. Only today, the switchboard is digital, lightning-fast, and juggling thousands of lines at once.

Fun Facts

Your phone can theoretically handle 65,000+ simultaneous connections—basically, holding a conversation with an entire stadium of people.
Even in normal use, your phone might juggle 50–100 ports just for background refreshes, social media, and emails.

3. Why Port 443 Took Over the Internet

Port 443 isn’t just another door; it’s the secure door. It became popular because it supports SSL/TLS encryption, the tech behind HTTPS.

Why does that matter? Well, the difference between browsing over Port 80 (HTTP) and Port 443 (HTTPS) is like whispering your bank password loudly in public versus using a sealed envelope with your name on it.

Fun Facts

The “s” in https:// was such a game-changer that Google actually ranks HTTPS sites higher in search results.
Today, Netflix alone probably generates more 443 traffic in a day than the entire internet did in 1995.

4. Ephemeral Ports: The Mall Parking Spaces

Here’s where things get clever. Your phone doesn’t just use ports like 80 and 443. It also creates temporary ports, called ephemeral ports, to handle unique conversations.

Think of them as parking spots in a busy shopping mall. You drive in, park for a while, leave, and the spot becomes available for someone else.

So when your browser connects to https://example.com on port 443, your OS might assign your app a temporary local port (say, 49152). The site knows you by this parking spot, while the OS ensures you don’t collide with another app’s connection.

Fun Facts

Your computer can create and destroy thousands of ephemeral ports per minute without you noticing.
It’s one of the quietest but most important juggling acts your device performs.

5. Local vs. Internet Ports: What’s the Difference?

Not all ports are global superstars like 80 and 443. Some are local ports, used by developers and apps running just on your machine.

For example:

8080: A common “backup” web port, often used for local testing.
5555: Frequently used by Android devices for debugging.
25565: Used by Minecraft servers so friends can connect.

These local ports are like service entrances in a building—different from the front doors (like 80 and 443) that face the internet. They help apps communicate internally or set up test servers without colliding with the main network highways.

Fun Facts

Port 8080 is so common it’s basically the internet’s “Plan B” when 80 is busy.
Gamers literally “open ports” to create digital doorways for friends to join matches.

6. Why All This Matters

From your perspective, it’s seamless. You just tap Instagram, join a Zoom call, or watch Netflix, and it all “just works.” But under the hood, your OS is running a high-stakes logistics game: handing out ports, securing them, and reusing them efficiently.

And while it feels invisible, understanding ports is the first step to understanding how the internet truly works—and why security and efficiency go hand in hand.

Bonus Fun Facts

Hackers often scan for open ports like burglars checking for unlocked windows—that’s why firewalls are critical.
If Port 443 traffic were a highway, it would need about 50 lanes to handle rush-hour internet traffic.

Want to Learn More?

Kernel TLS, NIC Offload, and Socket Sharding: What’s New and Who Uses It?

Sun, 17 Aug 2025 22:00:14 +0000

Modern servers are expected to push hundreds of gigabits per second while keeping latency low and CPU use manageable. Two kernel-level innovations—socket sharding and kernel TLS (kTLS)—help make that possible. When paired with NIC TLS offload, the gains are even bigger.

Socket Sharding

Traditionally, only one process could accept() connections from a TCP socket.
With Linux 4.5 (2016) and SO_REUSEPORT enhancements, multiple processes can share a listening socket. This “socket sharding” lets the kernel distribute connections efficiently across worker processes.

Adopted in NGINX and HAProxy.
FreeBSD has similar support.
Windows does not provide this functionality.

✅ Result: Better scalability on multi-core systems with no user-space load balancer needed.

Kernel TLS (kTLS)

Linux introduced kTLS in 4.13 (2017) (send path first, later receive).
Instead of copying encrypted data between user space and kernel space, the kernel itself handles TLS record encryption/decryption.

Avoids extra copies and syscalls.
Yields 20–40% CPU savings in many workloads.

Benchmarks

FreeBSD users: throughput improved from 25–30 Gb/s → 100+ Gb/s with dual-100G NICs.
NGINX with kTLS + SSL_sendfile(): roughly 2× throughput when serving static files compared to user-space TLS.

NIC TLS Offload

Some NICs (Mellanox/NVIDIA ConnectX, Chelsio T6, Intel QAT) can offload encryption entirely.
The kernel sets up the session keys, while the NIC performs the crypto inline.

Netflix (FreeBSD + Mellanox CX6-DX):
- In production, serving 400–800 Gb/s per server.
- Reported ~50% reduction in memory bandwidth use vs CPU crypto.
NVIDIA/Mellanox (netdev conf):
- Inline TLS offload gave ~2× throughput vs software kTLS.
- Example: 8.8 Gb/s vs 4.4 Gb/s in test environments.
Chelsio T6:
- Vendor benchmarks show ~44.8 Gb/s TLS throughput with very low CPU use.
Academic 2024 SmartNIC study:
- Inline mode: ~9.3 Gb/s throughput under parallel load.
- Coprocessor mode: less throughput but lowest CPU cost.

Windows Support?

Unlike Linux and FreeBSD, Windows does not support kernel TLS.
TLS remains in user space via SChannel or libraries like OpenSSL/BoringSSL, with no NIC offload integration.

Summary

Socket sharding (Linux 4.5+) spreads load across processes → higher scalability.
kTLS (Linux 4.13+, FreeBSD 13+) avoids copies → 20–40% CPU savings, ~2× throughput for NGINX.
NIC TLS offload (Netflix, NVIDIA, Chelsio):
- 2× throughput, huge CPU savings,
- 400–800 Gb/s per server possible in production.
Windows: No kernel TLS or NIC offload support today.

These optimizations matter most at hyperscaler scale, but are increasingly available to anyone running NGINX, HAProxy, or workloads on modern NICs.

References

Broadcast devices' name on the local network

Thu, 03 Jul 2025 20:55:48 +0000

With years, I accumulated devices on my local network, which in general run on Linux. I meticulously added them to my /etc/hosts/ file, so as not to remember their IP. Something puzzled me, though: my Synology NAS was readily available as nas.local on the network, without doing anything. I have close to zero skills in system administration, so here are my findings.

The `.local` domain

We can learn more about .local domain from Wikipedia.

>The domain name .local is a special-use domain name reserved by the Internet Engineering Task Force (IETF) so that it may not be installed as a top-level domain in the Domain Name System (DNS) of the Internet. As such it is similar to the other special domain names, such as .localhost. However, .local has since been designated for use in link-local networking, in applications of multicast DNS (mDNS) and zero-configuration networking (zeroconf) so that DNS service may be established without local installations of conventional DNS infrastructure on local area networks.

>-- .local

The next step is obviously checking the Multicast DNS definition:

>Multicast DNS (mDNS) is a computer networking protocol that resolves hostnames to IP addresses within small networks that do not include a local name server. It is a zero-configuration service, using essentially the same programming interfaces, packet formats and operating semantics as unicast Domain Name System (DNS). It was designed to work as either a stand-alone protocol or compatible with standard DNS servers.
>It uses IP multicast User Datagram Protocol (UDP) packets and is implemented by the Apple Bonjour and open-source Avahi software packages, included in most Linux distributions.
>Although the Windows 10 implementation was limited to discovering networked printers, subsequent releases resolved hostnames as well. mDNS can work in conjunction with DNS Service Discovery (DNS-SD), a companion zero-configuration networking technique specified separately in RFC 6763.

>-- Multicast DNS

In short, when you send a request to a .local domain, you sent it to a special multicast IP–224.0.0.251. Devices can join the multicast group associated to this IP. From this point on, the router forwards the request to all devices in the group.The one with the corresponding host will respond, others will ignore it.

Here's the result of pinging the above IP on my local network:

PING 224.0.0.251 (224.0.0.251): 56 data bytes
64 bytes from 192.168.1.73: icmp_seq=0 ttl=64 time=0.251 ms
64 bytes from 224.0.0.251: icmp_seq=0 ttl=255 time=13.876 ms
64 bytes from 192.168.1.36: icmp_seq=0 ttl=64 time=127.181 ms
64 bytes from 192.168.1.71: icmp_seq=0 ttl=64 time=128.267 ms
64 bytes from 192.168.1.80: icmp_seq=0 ttl=64 time=128.588 ms

Notice that I get responses from other IPs.

With this approach, local networks can avoid the burden of setting up an internal DNS and the configuration that goes with it on each device.

Setting up a `.local` domain

For my own local network, I prefer the .local domain to a full-fledged DNS and configuring every device. As I mentioned above, my Synology NAS does it automatically.

Let's set it up for my Raspberry Pi. It works for any Linux-based distribution. I'll use Avahi, which implements mDNS.

sudo apt-get update
sudo apt-get install avahi-daemon

We also need to update the /etc/nsswitch.conf to plugin in into the host resolution system.

...
hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4
...

The above configuration tries to resolve host in order:

Look in /etc/hosts
Use Avahi

If the host hasn't been resolved at this point, return. Note that options after this stage are for documentation only: you can remove them with no side-effect, but it allows admins what options are available.

Since I started using Ansible to manage my Pi, here's the corresponding playbook:

- name: Configure Raspberry Pi
  hosts: 192.168.1.144
  vars:
    ansible_user: johndoe
  tasks:
    - name: Update apt cache
      become: true
      apt:
        update_cache: yes
    - name: Broadcast device as pi.local
      block:
        - name: Set hostname to pi
          become: true
          hostname:
            name: pi
        - name: Update /etc/hosts with new hostname
          become: true
          lineinfile:
            path: /etc/hosts
            regexp: '^127\.0\.1\.1\s+'
            line: '127.0.1.1       pi'
        - name: Install avahi-daemon
          become: true
          apt:
            name: avahi-daemon
            state: present
        - name: Ensure avahi-daemon is enabled and running
          service:
            name: avahi-daemon
            state: started
            enabled: true
        - name: Set mdns in /etc/nsswitch.conf
          become: true
          lineinfile:
            path: /etc/nsswitch.conf
            regexp: '^hosts:'
            line: 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4'
            backup: true

Conclusion

If you operate a small local network that only a few trusted people have access to, you can benefit from multicast and the .local domain instead of full-fledged internal DNS server. We only had to install the Avahi daemon locally and tweak the /etc/nsswitch.conf file. For Infrastructure-as-Code purposes, I used an Ansible playbook.

To go further:

Originally published at A Java Geek on June 29^th, 2025