Coder Legion - Recent posts tagged server management

Linux Fundamentals for DevOps: Filesystem

Mon, 27 Apr 2026 16:45:41 +0000

When you SSH into a server to fix a broken application or install a new tool, the absolute worst thing you can do is guess where files belong. Linux does not put files in random places; it follows a strict, predictable map.

Understanding this map means you know exactly where to look when an application crashes, when a disk fills up, or when you need to configure a container.

If you run ls / on your server, you'll see a massive wall of folders.

As a DevOps engineer, you don't need to memorize every single one. Here are the crucial directories you will actually use, what they do, and the proof of why you should care about them.

`/etc` — The Control Room (Configurations)

You need to change the port your web server listens on, block a user from SSHing into the server, or configure a systemd service.

Historically, /etc stood for "et cetera," but today it is best understood as Editable Text Configurations. Every server, daemon, and background service stores its settings here.

Real scenario: Change the port Nginx listens on
Your app is running on port 80 but you need it on 8080 for a staging environment.

# Step 1: Find the config
cat /etc/nginx/nginx.conf

# Step 2: Edit it
nano /etc/nginx/nginx.conf
# Change: listen 80  →  listen 8080

# Step 3: Reload (no downtime!)
systemctl reload nginx

`/var` — The Growing Data Hub

This directory stands for "variable." It is designed to hold data that actively grows and changes while the system is running. If your server suddenly triggers a "Disk Space 100%" alert, /var is almost always the culprit.

We split this into two critical sub-directories:

/var/log (System and Application Logs): Your web application returned a 500 Error, or a database failed to start. This is the very first place you check.

# Watch errors as they happen, live
tail -f /var/log/nginx/error.log

# Search last 100 lines for "error" keyword
tail -n 100 /var/log/syslog | grep -i error

# See how much log data has accumulated
du -sh /var/log/

/var/lib (State Information and Databases): This is where applications save their actual heavy data. It is where PostgreSQL stores its database files, and where Docker stores its images and containers.

# "Why is my disk 80% full after installing Docker?"
du -sh /var/lib/docker

# Prune unused Docker images to reclaim space
docker system prune -a

# Where PostgreSQL stores its actual database files
ls /var/lib/postgresql/

NOTE: On modern Linux (Ubuntu 20.04+, Debian 10+), /lib is not a real directory anymore. You can see /lib and /usr/lib has common files

`/home` and `/root` — User Territories

You need to grant a new developer SSH access to a server, or you need to authenticate your server to pull code from a private Git repository.

These directories hold user-specific files. /home contains folders for standard users (e.g., /home/saeed), while /root is the isolated home directory for the superuser.

The DevOps importance: configuration files starting with a dot that control how tools behave per-user. Your .bashrc (shell settings), .kube/config (Kubernetes cluster access credentials), and .ssh/ (SSH keys) are all stored here.

# Grant a new developer SSH access
# They send you their public key. You do this:
cat dev-alice-key.pub >> ~/.ssh/authorized_keys

# Verify it was added
tail -n 3 ~/.ssh/authorized_keys

# Alice can now SSH in — no password needed

`/proc` — The Matrix (A Window Into the Kernel)

You are running a Docker container and need to know exactly how much memory the kernel is allowing it to use, or you need to see the exact environment variables a running process received. Every running process gets a numbered folder here (its Process ID, or PID) containing its live runtime data.

/proc The files aren't stored on disk — they don't exist until you read them. It is a "virtual filesystem" created in memory by the Linux kernel. . The moment you open /proc/1234/status, the kernel generates the content on the fly. It's a live feed into what's actually happening right now.

# Find the PID of your container's main process
docker inspect --format='{{.State.Pid}}' my-container

# See every env variable it received at startup
cat /proc/1234/environ | tr '\0' '\n'

# See what files/sockets it has open right now
ls -la /proc/1234/fd

# See exact memory usage the kernel is tracking
cat /proc/1234/status | grep VmRSS

# How long has this server been running?
cat /proc/uptime

# CPU info — useful in CI/CD pipelines
cat /proc/cpuinfo | grep "model name" | head -1

# Current memory usage
cat /proc/meminfo

`/dev` — Everything is a File

Linux's philosophy is: "if it can be read from or written to, it's a file." Your hard drive? A file. A USB drive? A file. Even nothingness and randomness are files. /dev is where all these hardware-and-virtual devices live as file objects. Your hard drives show up here (like /dev/sda or /dev/nvme0n1). But it also holds special virtual devices that are incredibly useful for scripting.

You are writing a cron job and want to completely hide its output so it doesn't spam your logs, or you are attaching a new block storage volume to your cloud instance.

#/dev/null — The Black Hole                                                                                                                                                                                                                                                                                                                                                        
# This cron job spams your mail. Make it silent:
# The "2>&1" part also redirects error output to null

./noisy-backup-script.sh > /dev/null 2>&1

# Useful breakdown:
#   >           redirects stdout
#   /dev/null   the black hole
#   2>&1        also send stderr to wherever stdout goes

`/usr` — The Real Home of Installed Software

You downloaded a pre-compiled binary (like terraform or helm) and want to install it so that anyone on the server can use it.

/usr (Unix System Resources) is where the vast majority of your installed applications, libraries, and tools actually live.

The DevOps importance: When you manually download and install a binary, you should place it in /usr/local/bin. This specific folder is meant for tools managed by the system administrator, ensuring they don't get overwritten when the system package manager (like apt or yum) runs its automated updates.

# Move a downloaded binary to the standard location for user-installed tools
mv terraform /usr/local/bin/terraform

`/opt` — Third-Party Add-ons

You are installing a heavy monitoring agent like Datadog, or a proprietary vendor application.

Linux reserves /opt (Optional software) for self-contained, third-party applications. Instead of scattering files across /usr and /etc, these monolithic tools keep everything neatly sandboxed inside their own dedicated folder here.

`/tmp` — The Throwaway Folder

You need to download a zip file, extract a shell script, run it, and never think about it again.

The /tmp directory is for temporary files. It is completely wiped clean every time the server reboots, so you should never put persistent or important data here. It also usually has open write permissions, making it the perfect staging ground for quick tasks.

Blog Summary

This guide breaks down the essential directories every DevOps engineer needs to know—like /etc for configurations, /var for logs, and /proc for live system processes. Complete with real-world use cases and simulated terminal outputs, this post serves as a practical map for quickly troubleshooting crashes, managing servers, and finding exactly what you need without getting lost.

Linux Fundamentals For DevOps: Linux core mechanism

Mon, 27 Apr 2026 16:45:13 +0000

Knowing where files live is only half the battle. To be an effective DevOps engineer, you need to know how Linux actually executes programs, manages their lifecycles, and secures them.
The crucial concepts you need to troubleshoot effectively.

The `$PATH` Variable

Linux does not search your entire hard drive when you type a command. It only checks a specific, prioritized list of folders. This list is saved in an environment variable called $PATH.

You download a new binary tool (like kubectl or terraform). You type its name in the terminal, but Linux replies with the dreaded command not found. To fix this, you must either move the binary into a folder that is already in your $PATH, or add your custom folder to the $PATH.

# Command: View the list of directories Linux currently checks for commands

echo $PATH

# Output Proof (Directories are separated by colons):
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Command: Add your custom scripts folder to the PATH for the current session

export PATH="/home/saeed/custom-scripts:$PATH"

How Binaries Work (Shared Libraries)

When you run a command like nginx or python, you are executing a binary file. In Linux, executables are typically formatted as ELF (Executable and Linkable Format) files. However, most binaries are not standalone; they rely on external pieces of code called shared libraries (.so files, similar to .dll files in Windows).

You compile a Go application or migrate a tool into a minimal Docker container (like Alpine Linux). When you run it, it crashes silently or throws a confusing no such file or directory error—even though you can see the binary sitting right there. Usually, the binary isn't missing; its required libraries are missing.

You can use the ldd command to print the shared libraries required by any program.

# Command: Verify exactly which shared libraries Nginx needs to run
ldd /usr/sbin/nginx

# Output Proof:
        linux-vdso.so.1 (0x00007ffe3b1f4000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f3a8b411000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f3a8b39c000)
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007f3a8b2f6000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3a8b0cd000)

File Permission

Linux strict permissions dictate exactly who can Read, Write, or eXecute a file. This is often represented by numbers (like 777 or 644).

You are trying to connect to a cloud server via SSH using a private key (.pem file), but the connection is instantly rejected. If you set a .pem file to 777 (meaning everyone on the system can read, write, and execute it), SSH will intentionally reject it for being too exposed. Private keys must be set to 400—meaning absolutely no one except the owner is allowed to read it.

# Command: Attempting to SSH with a bad permission key
ssh -i my-key.pem ubuntu@10.0.0.5

# Output Proof:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0777 for 'my-key.pem' are too open.

# Command: Secure the key so only the owner can read it (4), and no one can write (0) or execute (0)

chmod 400 my-key.pem

Processes and Signals

Every running program is a process assigned a unique Process ID (PID). The operating system communicates with and controls these processes using "Signals."

An application is hogging 100% of the CPU and needs to be stopped, or Kubernetes is trying to scale down a pod without dropping active user connections. You can send a SIGTERM (Signal 15), which politely asks the program to save its data and exit cleanly. Alternatively, a SIGKILL (Signal 9) forcefully terminates it at the hardware level, bypassing cleanup and potentially causing data corruption.

# Command: Find the Process ID of an application
pgrep -la nginx

# Output Proof:
1234 nginx: master process
1235 nginx: worker process

# Command: Politely ask the master process to shut down and clean up
kill -15 1234

File Descriptors

Remember concept , "Everything is a File"? That rule has a dark side.

Because Linux treats network connections as files, every time a user connects to your web server, the operating system opens a "File Descriptor" to track that connection. To protect itself from crashing, the Linux kernel sets strict limits on how many file descriptors a single process can open at once.

Your Nginx web server or Node.js application is suddenly dropping user connections during a high-traffic spike. You check the CPU and RAM, and both are perfectly fine. You check the Nginx error logs and see the dreaded worker_connections are not enough or Too many open files error. The operating system has forcibly stopped your app from accepting new traffic.

# Command: Show the maximum number of open file descriptors allowed for this user
ulimit -n

# Output Proof:
1024

# Command: Check how many file descriptors the entire system is currently using
cat /proc/sys/fs/file-nr

# Output Proof:
1205   0   9223372036854775807

Blog Summary

Understanding Linux goes beyond knowing where files live; it’s about understanding how the operating system executes code, secures data, and manages its own limits. This guide breaks down the invisible mechanics of Linux that every DevOps engineer must master to troubleshoot effectively.

You'll learn why binaries silently fail due to missing shared libraries, how to quickly fix "command not found" errors using the $PATH variable, and the strict permission rules required to secure SSH keys. We also cover how to safely manage and terminate running processes using signals, and finally, uncover the hidden trap of File Descriptors—explaining why servers suddenly drop connections with "Too many open files" errors and how to raise those limits.

Coder Legion - Recent posts tagged server management

Linux Fundamentals for DevOps: Filesystem

/etc — The Control Room (Configurations)

/var — The Growing Data Hub

/home and /root — User Territories

/proc — The Matrix (A Window Into the Kernel)

/dev — Everything is a File

/usr — The Real Home of Installed Software

/opt — Third-Party Add-ons

/tmp — The Throwaway Folder

Blog Summary

Linux Fundamentals For DevOps: Linux core mechanism

The $PATH Variable

How Binaries Work (Shared Libraries)

File Permission

Processes and Signals

File Descriptors

Blog Summary

`/etc` — The Control Room (Configurations)

`/var` — The Growing Data Hub

`/home` and `/root` — User Territories

`/proc` — The Matrix (A Window Into the Kernel)

`/dev` — Everything is a File

`/usr` — The Real Home of Installed Software

`/opt` — Third-Party Add-ons

`/tmp` — The Throwaway Folder

The `$PATH` Variable