Coder Legion - Recent posts tagged file system

I Built a Semantic File System in Rust to Eradicate the Directory Tree

Sat, 02 May 2026 12:21:15 +0000

We write autonomous AI agents, compile at the speed of light, and deploy globally in milliseconds. Yet, at the core of our OS, we still organize data using a digital metaphor from 1964: the hierarchical directory tree.

You are forced to memorize arbitrary pathways (/src/utils/auth/keys) and exact filenames just to retrieve your own code. grep and find are band-aids applied to a fundamentally broken paradigm.

You do not care where a file is. You care what it means.

That is why I built OmitFS—a production-grade, 100% local, zero-dependency semantic file system written entirely in Rust.

Search your files the way you think about them:

You type	OmitFS finds instantly
`"the production API config"`	`config/prod.env`
`"meeting notes from last sprint"`	`2024-03-15_standup.docx`
`"calculus assignment"`	`Math_HW_Chapter4_Final.pdf`

No cloud. No API limits. Forever offline.

The Architecture: Bare-Metal Speed & Air-Gapped Privacy

If you are building infrastructure for developers, it cannot be a fragile Python wrapper around an OpenAI API call. It needs to be fast, memory-safe, and uncompromisingly secure.

I built OmitFS completely in Rust. It compiles down to a single binary with zero background Docker containers or external databases required.

Here is the internal pipeline:

Code snippet

graph TD
    A["Files dropped into raw/"] --> B{OmitFS Daemon}
    B --> C["Text Extraction (PDF, DOCX, Code)"]
    C --> J["Chunker (200-word windows)"]
    J --> K{"SHA-256 Manifest (Skip unchanged)"}
    K -->|Changed| L["EmbeddingEngine (all-MiniLM-L6-v2)"]
    L --> O[("LanceDB Vector Database")]
    O --> P{Two-Stage Search}
    P --> Q["1. ANN Vector Search (Cosine)"]
    Q --> R["2. BM25 Re-ranker (TF-IDF)"]
    R --> S["Results"]

The Core Stack:

The FUSE Kernel Bridge: Using the fuser crate, OmitFS intercepts standard POSIX system calls. Virtual folders mirror standard UNIX permissions and byte-sizes, ensuring zero friction with existing CLI tools like vim, cat, and grep.
The Local Inference Engine: Powered by the Rust ML framework candle, OmitFS runs a quantized embedding model (all-MiniLM-L6-v2) locally on your CPU/GPU. It generates 768-dimensional vectors from text payloads in ~5ms.
The Vector Store: LanceDB runs natively within the Rust process. It executes sub-millisecond Approximate Nearest Neighbor (ANN) searches, followed by a BM25 keyword re-ranker for absolute precision.
The Extractor: Natively extracts text from .pdf, .docx, .xlsx, and source code. If Tesseract or Whisper are on your $PATH, it seamlessly OCRs images and transcribes audio.

Beyond Search: RAG and MCP Integration

A semantic file system is powerful, but it is also the perfect foundational memory layer for local AI workflows.

1. Ask AI (Local RAG)

By hooking OmitFS up to a local LLM runner like Ollama, you can perform Retrieval-Augmented Generation (RAG) over your entire hard drive without passing your data to the cloud.

Bash

$ omitfs ask "What integral technique did I use in problem 4?"

OmitFS retrieves the exact mathematical chunk from your PDF, feeds it to llama3 or phi4 as context, and streams the answer back to your terminal.

2. Native MCP Integration (Cursor / Claude)

OmitFS ships with native Model Context Protocol (MCP) server capabilities. You can instantly plug it into Claude Desktop or Cursor IDE. Your AI agent can autonomously query OmitFS to find the exact documentation or architectural decisions hidden across your local machine to give you better code generations.

The Benchmarks

Rust ensures OmitFS stays entirely out of your way. Because of the SHA-256 manifest, the daemon never re-indexes files that haven't changed.

Operation	CPU (i7-12th gen)	GPU (RTX 3080)
Embed one chunk (~200 words)	~25 ms	~2 ms
Index 1,000 text files	~4 min	~20 sec
Search (query → results)	~60 ms	~10 ms
RAG answer (Llama3 8B)	~8 sec	~0.8 sec
Daemon restart (unchanged)	instant	instant

One-Line Install

The installation is completely frictionless. A single command downloads the pre-built binary, patches your PATH, and sets up the daemon.

macOS / Linux:

Bash

curl -sSf https://raw.githubusercontent.com/Panav-Payappagoudar/OmitFS/main/install.sh | sh

Windows PowerShell:

PowerShell

irm https://raw.githubusercontent.com/Panav-Payappagoudar/OmitFS/main/install.ps1 | iex

Join the Resistance

We spend too much time managing the "containers" of our data rather than interacting with the substance itself. OmitFS strips away the UI clutter, the nested folders, and the mental overhead.

If you are a Rustacean, an AI engineer, or just someone tired of looking for lost files, I would love your feedback.

Check out the full source code on GitHub:

Panav-Payappagoudar/OmitFS on GitHub

If this project saves you time, dropping a ⭐ on the repo helps massively.

Beyond the Terminal: My Hunt Through the Linux File System

Sat, 25 Apr 2026 05:07:24 +0000

At first, Linux felt like magic. You open the terminal, type a command… and things just work. No questions asked. But after a while, that “magic” started bothering me.

Because I kept thinking:

Where is all of this actually happening?
How does the system know what to do?

Like… when I connect to the internet, where is that configuration stored?
When a user logs in, where is that data coming from?
When something breaks, where does Linux keep track of it?

These weren’t just random questions anymore. They were pulling me deeper.

So instead of just using Linux… I decided to explore it. Not like a normal user.
But like someone trying to understand what’s going on behind the scenes.

Digging into folders.
Opening system files.
Trying to connect the dots.

And what I discovered?

It completely changed how I look at Linux. Because behind every command…
there’s a system quietly doing a lot more than we realize.

`/etc` - The Place Where System Behavior Lives

The first place I explored was /etc, because I kept hearing that it’s important.

So I went inside:

cd /etc
ls

At first, it just looked like a collection of random files. But the more I opened them, the more I realized something important.

/etc is not just a folder.

It is where Linux stores all configuration that controls system behavior.

When I opened files like:

cat /etc/hosts
cat /etc/passwd

I could literally see how my system is configured.

This made me realize why this folder exists. Instead of hardcoding system behavior deep inside programs, Linux stores configurations as editable files. This means the system becomes flexible you can change behavior without touching the core system.

The problem this solves is huge. Imagine if you had to recompile the OS just to change a setting. That would be impossible for daily use.

But with /etc, Linux separates configuration from execution.

And the insight that hit me was this:
My system is not controlled by hidden magic. It’s controlled by plain text files that I can read and understand.

`/etc/resolv.conf` - How My System Finds Websites

One thing I never really questioned was typing something like:

ping google.com

It just works. But then I stopped and thought:

How does my system even understand what “google.com” is?
Because at the end of the day, computers don’t understand names, they understand IP addresses.

So I decided to dig into it.

I ran:

cat /etc/resolv.conf

And I saw this:

nameserver 10.0.2.3

At first, this confused me.

I was expecting something like 8.8.8.8 (Google DNS).
But this… looked different.

So what is this `10.0.2.3`

It turns out, this is not a public DNS server. It’s a local DNS resolver provided by my virtual machine environment

Since I’m running Linux inside a VM, my system is not directly talking to the internet.

Instead, it works like this:

My Linux system → asks 10.0.2.3 → which then forwards the request to actual DNS servers

So basically, 10.0.2.3 is acting like a middleman.

Why does this file exist?

This file tells the system: “If you don’t understand a domain name, ask this nameserver.”

Without this, the system wouldn’t know where to send DNS queries.

What problem does it solve?

Humans use names like:

But systems use IP addresses. So DNS acts as a translator between human-friendly names and machine-friendly IPs.

And /etc/resolv.conf tells Linux where that translator lives.

What I found interesting

The biggest realization for me was this: My system isn’t always directly using public DNS.
In my case, it’s using a local resolver (10.0.2.3), which then handles everything behind the scenes.

That means even something as simple as opening a website involves:

local system configuration
virtual networking layers
DNS forwarding

Small but powerful insight

Just by changing this file, I can control how my system resolves domains.

Which means I can switch to a faster DNS, block certain domains, & control network behavior.

And honestly… Something as simple as opening a website suddenly felt way deeper than I thought.

Routing Table - How Linux Decides Where Data Goes

Next, I wanted to understand how my system sends data.

So I ran:

ip route

What I saw was something like:

This is the routing table.

At a basic level, this tells the system: “If you don’t know where to send data, send it to this gateway.”

In my case, that gateway is 10.0.2.2.

This exists because your system doesn’t just randomly send data. It needs a structured way to decide where packets go.

The problem it solves is routing confusion. Without this, your system wouldn’t know whether to send data locally or over the internet.

The second line (10.0.2.0/24) tells the system that anything in this local network range can be accessed directly, without going through the gateway.

The insight here was surprising: My system is not just sending data it’s making routing decisions like a mini network controller.

And since I’m using a virtual machine, 10.0.2.2 is actually a virtual gateway, not a physical router.

I always thought networking happens “somewhere outside”. But Linux is actively involved in deciding paths. And in this case, it’s even routing traffic through a virtual layer before it reaches the real network.

`/var/log` - The Silent Observer of Everything

Then I explored logs:

cd /var/log
ls

And opened files like:

cat messages
cat secure

This folder is basically the memory of the system.

Everything that happens:

logins
errors
services
system events

…it all gets recorded here. This exists because systems need a way to track what’s happening internally.
The problem it solves is debugging and monitoring. Without logs, if something breaks, you’re blind.

But what really stood out to me was this:

Linux doesn’t interrupt you with errors all the time.
It quietly writes everything down.

It’s like the system is constantly saying:
“I’ll tell you everything… but only if you come and look.”

`/etc/passwd` & `/etc/shadow` - How Linux Handles Users Securely

To understand users, I ran:

cat /etc/passwd
sudo cat /etc/shadow

What I saw was interesting.

/etc/passwd contains user details.

But passwords are not there.

They’re stored in /etc/shadow, which requires special permissions.

This exists for security reasons. If everything was stored in one place, anyone could read sensitive data.

The problem it solves is protecting user credentials.

But the insight here was deeper:

Linux separates identity from authentication.

/etc/passwd → who you are
/etc/shadow → how you prove it

That separation makes the system more secure by design.

Permissions - Built-In Security, Not an Add-On

When I ran:

ls -l

I saw things like:

-rwxr-xr--

At first, it looked confusing. But then i understood: this is how Linux controls access to every file.

Permissions exist because not every user should have full control over everything.

The problem it solves is unauthorized access and system damage.

But what really stood out was:

Security in Linux is not something extra.
It is built directly into the file system itself.

Every file carries its own rules.

`/proc` - Where I Saw the System Live

This was the most surprising part

I explored:

cd /proc
ls

And then:

cat /proc/cpuinfo
cat /proc/meminfo

What I realized is: These are not real files stored on disk. They are generated in real time by the kernel. This exists so users and programs can access live system data.
The problem it solves is system monitoring.

But the insight here blew my mind:

When I read these files, I’m not reading stored data.
I’m interacting with the system live.

It felt like I was directly talking to Linux.

`/dev` - Where Hardware Becomes Files

Then I explored:

ls /dev

I saw entries for disks, terminals, and devices.

This exists because Linux treats hardware as files.

The problem it solves is complexity. Instead of learning different ways to interact with devices, everything becomes file operations.

And the insight here was simple but powerful: In Linux, everything is unified under one idea: files.

That consistency makes the system elegant.

`/boot` - The Starting Point of Everything

Finally, I checked:

ls /boot

This folder contains the kernel and bootloader files.

This exists because the system needs instructions to start itself. The problem it solves is system initialization.
And the insight: Without /boot, Linux cannot even begin to run.

It’s literally where everything starts.

`/var/log/secure` - Who Tried to Enter My System?

While exploring logs, I went deeper:

So I ran:

sudo tail -f /var/log/secure

And what I saw was honestly eye-opening.

This file logs things like:

login attempts
sudo usage
authentication failures

Basically… anything related to who is trying to access the system.

This exists because systems need to keep track of access.

Who logged in
Who tried and failed
Who used admin privileges

Without something like this, there would be no way to monitor system security.

The problem it solves is pretty serious. If something suspicious happens, like:

repeated failed login attempts
unauthorized access
misuse of sudo

you need a way to trace it. And that’s exactly what this file does. The insight here really changed how I see system logs.

Linux doesn’t constantly warn you about everything.
Instead, it quietly records everything in the background.

And this file becomes like a security audit trail. If someone tries to break into your system… the evidence is already there.

`/dev/null` - The Black Hole of Linux

This one was honestly one of the coolest discoveries.

I started with something simple:

ls /dev/null

Then I tried this:

echo "hello" > /dev/null

And… it disappeared.

No output. Nothing showed up anywhere. That’s when I realized what was happening.

/dev/null is a special file that discards anything written to it. This exists because sometimes, you don’t care about output. Maybe a command is producing too much noise, or you just want to run something silently.

The problem it solves is simple but important. Without /dev/null:

your terminal would get cluttered
scripts would print unnecessary output
things would become harder to manage

The insight here was actually pretty cool.

/dev/null is like a black hole. Anything goes in… and nothing comes out. And suddenly, this made sense:

command > /dev/null 2>&1

This basically tells the system:

Run the command, but don’t show me anything.

And now I finally understood why developers use it so often.

`systemd` - The Hidden Manager Running Everything

Then I got curious about what’s running behind the scenes.

So I explored:

cd /etc/systemd/system
ls

At first, it just looked like a bunch of configuration files.

But then it clicked. This is where system services are defined and managed.

systemd is basically the component that handles:

services
startup processes
background tasks

It’s the reason your system feels “alive” even when you’re not doing anything.

This exists because modern systems need automation. You don’t manually start things like:

networking
logging
system monitoring

Every time you boot your system.

The problem it solves is huge. Without something like systemd you would have to manually start every service after boot . Which is clearly not practical. The insight here was interesting. Even when I’m doing nothing…
Linux is still running multiple services silently in the background. Things just work because systemd is managing everything for me.

`/boot/grub2/grub.cfg` - The First Instructions Linux Follows

This one felt powerful.

I ran:

cat /boot/grub2/grub.cfg

And I was looking at the configuration used by GRUB the bootloader.

This file tells the system:

which kernel to load
how to start Linux

This exists because before Linux even starts, something needs to:

initialize the system
load the kernel
hand over control to the OS

And that’s exactly what the bootloader does.

The problem it solves is fundamental. Without this configuration:
the system wouldn’t even know how to boot. No startup. No OS. Nothing.

The insight here made me realize how critical this file is. This is literally where the system’s startup behavior is defined. If this file gets corrupted Linux won’t even start
And that’s when it hit me… This isn’t just another config file. This is the starting point of everything.

After exploring all these parts, one thing became very clear to me:

Linux is not just an operating system
It is a collection of well-structured systems connected through files

From booting → to users → to networking → to logs

Everything is transparent, accessible, & understandable.
And once you start exploring it like this… you stop guessing and start understanding.

Linux is not hiding anything from you.
Everything is there in files, waiting to be explored.

The moment you stop just using Linux… and start reading it… that’s when things finally start clicking.

Coder Legion - Recent posts tagged file system

I Built a Semantic File System in Rust to Eradicate the Directory Tree

The Architecture: Bare-Metal Speed & Air-Gapped Privacy

The Core Stack:

Beyond Search: RAG and MCP Integration

The Benchmarks

One-Line Install

Join the Resistance

Beyond the Terminal: My Hunt Through the Linux File System

/etc - The Place Where System Behavior Lives

/etc/resolv.conf - How My System Finds Websites

So what is this 10.0.2.3

Why does this file exist?

What problem does it solve?

What I found interesting

Small but powerful insight

Routing Table - How Linux Decides Where Data Goes

/var/log - The Silent Observer of Everything

/etc/passwd & /etc/shadow - How Linux Handles Users Securely

Permissions - Built-In Security, Not an Add-On

/proc - Where I Saw the System Live

/dev - Where Hardware Becomes Files

/boot - The Starting Point of Everything

/var/log/secure - Who Tried to Enter My System?

/dev/null - The Black Hole of Linux

systemd - The Hidden Manager Running Everything

/boot/grub2/grub.cfg - The First Instructions Linux Follows