Coder Legion - Recent posts tagged sysadmin

Finding the world-writable files on your server before someone else does

Thu, 29 Oct 2026 13:00:00 +0000

A world-writable file can be rewritten by every account on the machine — including the service account your web server runs as. Code executing as www-data through a compromised plugin doesn't need root: any chmod 777 you left behind during past debugging is a file it can overwrite or replace outright.

Two details break most homegrown audits. First, find -perm 777 (no dash) matches mode 777 exactly — a 666 file is equally world-writable and slips through. The dash form uses mask semantics and catches every file with the other-write bit set. Second, never fix findings with chmod -R 644: on directories, the execute bit means "may be entered," and stripping it recursively seals off the whole tree. Split the fix by type instead:

find /var/www -type f -perm -o+w 2>/dev/null
find /var/www -type f -exec chmod 644 {} \;
find /var/www -type d -exec chmod 755 {} \;

SSH keys and secrets go tighter, to 600 — OpenSSH refuses group- or world-readable private keys outright.

I keep the longer version as a script with a saved report file, a SUID scan, and a recommended-modes reference: full file-permissions security audit script.

The log was deleted. Its 48M was still allocated.

Wed, 28 Oct 2026 13:00:00 +0000

A 64M scratch tmpfs, a worker process holding its log open on file descriptor 3, and the usual reflex when space runs low: rm the log. du on the directory dropped from 48M to 0. df stayed at 48M used. The next 32M write failed with No space left on device, and df read 100% while du found only the 16M that had been written.

rm removes a name. The kernel frees the blocks only when no names are left and no process has the file open, and the worker still had it open. lsof +L1 lists exactly those files:

$ lsof -w +L1 /srv
COMMAND    PID USER FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
sleep   608657 root 3w   REG   0,78 50331648     0    3 /srv/app/worker.log (deleted)
exit=0

NLINK 0, (deleted), 48M held for writing by PID 608657. Truncating through the process's own descriptor, : > /proc/608657/fd/3, took df back to 25% while the process kept running.

Next time, empty a live log with : > file instead of deleting it. With an append-mode writer (>>) that is clean. A writer that opened with > keeps its old offset: in the same test, its log came back as a 17M sparse file with 1.0M on disk.

Finding which process holds a port or a busy mount, plus a script that groups deleted-but-open files and totals them, is on the lsof command examples page.

Patch These Four Linux Kernel Local Root Vulns Now: DirtyAH6, PPPoEject, TUNderflow, and DiagSpill

Sun, 25 Oct 2026 23:46:09 +0000

Originally published at codewithromi.com.
Four Linux kernel vulnerabilities went public on 18 September 2026, and they are the kind you should not sit on. DirtyAH6 (CVE-2026-80844), PPPoEject (CVE-2026-68121), TUNderflow (CVE-2026-81000) and DiagSpill (CVE-2026-74469) are all local privilege escalation bugs, meaning an attacker or malicious process with a foothold on your machine can escalate to root. Working proof of concept exploits for all four are public. If you are running Proxmox, a Raspberry Pi, or any self-hosted Linux node, this is a patching weekend.

Security researcher Asim Manizada reported the four bugs in mid July 2026 and published a technical write-up plus working exploits on 18 September 2026, after a coordinated hold so distributions could ship fixes first. The oss-security post and the accompanying write-up carry the per-CVE detail. Upstream, the earliest stable releases that contain all four fixes are 5.10.270, 5.15.221, 6.1.188, 6.6.157, 6.12.109, 6.18.50 and 7.2.4. Note that the 7.0 series reached end of life on 27 June 2026, so a 7.0 kernel only carries these fixes if your vendor backported them.

💡 Why This Matters for Homelabbers

Local root exploits are often dismissed as "low risk" because they require existing access. That framing misses the threat model for homelabs and self-hosted infrastructure. If any service you run gets compromised, a web app, an LXC container with a shared kernel, or even a rogue package, a local root bug turns that into full node ownership.

Proxmox LXC containers share the host kernel. A container breakout combined with a local priv-esc is a realistic attack chain. Raspberry Pi nodes running exposed services are in the same boat. Patch the kernel, then verify.

🧰 Prerequisites

SSH or console access to each node
sudo or root privileges
A maintenance window, since patching requires a reboot
Proxmox nodes: access to the Proxmox web UI or CLI for VM/container shutdown sequencing

🔧 Step-by-Step: Patching Your Nodes

Step 1: Check your current kernel version on every node so you know what you are starting from.

uname -r

Note this output. You will compare it after the update to confirm the new kernel is running.

Step 2: Update your package index and apply all available security updates. On Debian and Ubuntu based systems, including Proxmox and Raspberry Pi OS:

sudo apt update && sudo apt full-upgrade -y

On Proxmox specifically, you also want to pull from the Proxmox repositories. If you are on the no-subscription repo, this is already covered by the command above. Run pveversion -v afterward to confirm the Proxmox packages are current too.

Step 3: If you are running Raspberry Pi OS, check the kernel package by its current name. On Bullseye and earlier the kernel shipped in the raspberrypi-kernel package, but Raspberry Pi OS moved to Debian style linux-image-rpi-* packages with Bookworm, so on a current install you want linux-image-rpi-v8 or linux-image-rpi-2712 depending on the board. The apt full-upgrade command above will still catch it, but confirm with:

On Bookworm and newer:

apt-cache policy linux-image-rpi-v8

On Bullseye and older:

apt-cache policy raspberrypi-kernel

The "Installed" and "Candidate" versions should match after your upgrade. If they do not, run the upgrade again.

Step 4: Reboot to load the new kernel. This is not optional. The running kernel in memory is still the old one until you reboot.

sudo reboot

For Proxmox hosts, shut down or migrate your VMs and containers before rebooting the host. Use the Proxmox web UI, or list what is running:

qm list
pct list

Then shut down each VM or container by its ID:

qm shutdown [vmid]
pct shutdown [ctid]

Step 5: After the reboot, confirm the new kernel is loaded:

uname -r

The version string should be newer than what you recorded in Step 1. Cross-reference it against your distro's security advisory to confirm the patched version is in place. For Debian, check security-tracker.debian.org. For Ubuntu, check ubuntu.com/security/notices. For Raspberry Pi OS, check the Raspberry Pi blog and release notes.

🛡️ Patch Status, Checked 29 September 2026

This is the part of the post that goes stale fastest, so here is a dated snapshot. Everything below was checked against the Debian security tracker, the Ubuntu CVE tracker and the Proxmox kernel changelog on 29 September 2026. Re-check before you decide you are covered.

Debian 13 trixie: all four first fixed in linux 6.12.111-1, shipped as DSA-6528-1.
Debian 12 bookworm: DirtyAH6, PPPoEject and DiagSpill first fixed in linux 6.1.187-1 from bookworm-security. TUNderflow is the exception. The Debian tracker still lists bookworm as vulnerable, 6.1.187-1 included, so there is no fix for CVE-2026-81000 on bookworm yet.
Ubuntu 22.04, 24.04 and 26.04: at the time of this check the Ubuntu CVE tracker showed no released kernel for any of the four on the main linux packages. They sit at needed or pending. Watch ubuntu.com/security/notices instead of assuming an upgrade fixed it.
Proxmox VE: the Proxmox kernel changelog names only TUNderflow, first fixed in proxmox-kernel-7.0 version 7.0.14-19, released 18 September 2026. The newest build at the time of checking was 7.0.14-20 from 24 September 2026. The other three CVEs are not named anywhere in that changelog, and the Proxmox 7.0 kernel follows the Ubuntu kernel, which has not released them, so do not assume DirtyAH6, PPPoEject and DiagSpill are closed on a Proxmox host. Run pveversion -v and compare against the changelog.
Raspberry Pi OS: no Raspberry Pi OS kernel build carrying all four fixes could be confirmed at the time of this check. On the 6.12 branch you need 6.12.109 or newer for the full set, so check the installed package version rather than assuming.

🛡️ What Each Vulnerability Affects

Here is what each bug actually is, taken from the disclosure and the researcher's write-up. Three of the four need CAP_NET_ADMIN in a network namespace the attacker controls, which an ordinary local user obtains by creating an unprivileged user namespace first. DiagSpill is the exception and needs no special privileges at all.

DirtyAH6 (CVE-2026-80844): The IPv6 Authentication Header code in the IPsec and XFRM path does not check that a routing header's segments_left field is no larger than the number of addresses actually present, so pointer arithmetic in ipv6_rearrange_rthdr() walks out of bounds and an oversized memmove() corrupts kernel memory. It needs AH6 and XFRM support plus CAP_NET_ADMIN and CAP_NET_RAW in a namespace the attacker controls. The researcher also reports a remote crash, and in his own lab remote root, against a host acting as an IPv6 router that applies AH in transport mode.
PPPoEject (CVE-2026-68121): A use after free in pppoe_sendmsg(). The function holds a pointer into the socket buffer head across a dev_hard_header() callback that can reallocate that head, then writes six bytes through the stale pointer. It needs a lower device whose header callback expands the buffer head, not merely a loadable PPPoE module, which makes it the narrowest of the four in practice.
TUNderflow (CVE-2026-81000): The TUN/TAP driver stores a receive headroom value without bounding it. A device path that propagates oversized headroom (the proof of concept chains a netkit device through Open vSwitch to a raw TUN port) underflows a size calculation and leaves packet data 64 bytes past the end of the allocation. Simply having a TUN or TAP interface is not enough, you need that headroom propagation path. Worth correcting a common assumption here: Docker's default bridge networking uses veth pairs rather than TUN/TAP, and the in kernel WireGuard module is its own device type, so the usual homelab exposure is QEMU tap interfaces and OpenVPN style TUN devices, not containers.
DiagSpill (CVE-2026-74469): An SCTP association can hold up to 65,536 peer transports but the counter is 16 bits wide, so it wraps to zero at the limit. sctp_diag then reserves buffer space based on the wrapped count and copies the full transport list anyway, writing roughly 8 MiB past the end of the response buffer. This one needs no capabilities and no user namespace, only SCTP and sctp_diag support, which is what makes it the priority of the four.

DiagSpill is the one to move fastest on if you are triaging. It is the only one of the four that an ordinary local user can reach without creating a user namespace, which also means the usual advice to restrict unprivileged user namespaces does nothing for it. If you do not need SCTP, keep the module out of the kernel.

🛡️ Hardening Beyond the Patch

Patching is the fix, but a few additional controls reduce your exposure, which matters more than usual here because some distributions have not shipped all four yet. Restricting unprivileged user namespaces closes the ordinary user path to three of the four. It does nothing for DiagSpill.

Restrict unprivileged user namespaces: Three of these four bugs need a user namespace for an ordinary local user to reach them. Ubuntu introduced AppArmor based restrictions in 23.10 and enabled them by default from 24.04, controlled by kernel.apparmor_restrict_unprivileged_userns, so 24.04 and 26.04 are covered out of the box. Debian is not. The Debian and Ubuntu specific kernel.unprivileged_userns_clone knob has been discussed for deprecation, so confirm it exists on your kernel with sysctl -a | grep userns before relying on it; the portable upstream equivalent is sysctl -w user.max_user_namespaces=0. Either way you will break things that legitimately use user namespaces, including Flatpak, rootless containers and browser sandboxes.
Use AppArmor or SELinux profiles: Proxmox ships with AppArmor. Make sure it is enforcing for your LXC containers. Unprivileged containers with AppArmor confinement are significantly harder to break out of.
Limit local access: Do not run untrusted code or expose shell access to untrusted users on these nodes. Local privesc only matters if someone or something local can trigger it.
Unload unused kernel modules: If you are not using PPPoE or SCTP, keep both out. Add blacklist pppoe and blacklist sctp to a file in /etc/modprobe.d/ and run update-initramfs -u. Blacklisting stops the on demand autoload that a socket call would trigger, and for DiagSpill that is the only mitigation short of patching, since restricting user namespaces does not touch it. Verify with lsmod | grep -E 'pppoe|sctp' afterwards.

🔧 Troubleshooting

Kernel version did not change after reboot: Your package manager may have the patched package available but it was not installed. Run apt list --upgradable | grep linux-image to check, then install explicitly with sudo apt install linux-image-[version] matching your architecture.

Proxmox node is stuck on an older kernel: Proxmox sometimes pins the kernel for stability. Check /etc/default/grub and the GRUB menu to ensure the new kernel is selected as default. Run update-grub after confirming the package is installed.

Raspberry Pi will not boot after upgrade: This is rare but can happen if the bootloader and kernel fall out of sync. Boot from a backup SD card or use rpi-update with caution, and check the Raspberry Pi OS forums for your specific board and OS version before using it.

✅ Wrap Up

Four local root bugs in one disclosure batch is a bad day for Linux security. Updating and rebooting is the right move, but as of 29 September 2026 it is not the whole answer: Debian trixie has all four, Debian bookworm is still missing the TUNderflow fix, and Ubuntu has not released kernels for any of them. Prioritize DiagSpill, since it is the one an unprivileged local user can reach without a user namespace, and check the tracker for your distribution instead of assuming a fresh upgrade covered you.

After you patch, take five minutes to check your other nodes. It is easy to forget a Raspberry Pi tucked behind the TV or a secondary Proxmox node you set up months ago. Cross-reference the patched kernel version against your distro's security tracker to make sure you are actually covered, not just up to date on packages.

Stay subscribed to oss-sec if you want these disclosures in your inbox before they make the rounds on aggregators.

bash ran a path that no longer existed, and exited 127

Wed, 21 Oct 2026 13:00:00 +0000

A demo script called report, two directories on PATH, and one mv between them, all on a scratch tmpfs:

$ report
report 2.1: 14 jobs ok
exit=0
$ mv /opt/jobs/bin/report /opt/jobs/newbin/report
$ ls -l /opt/jobs/newbin/report
-rwxrwxr-x 1 root root 42 Oct  3 18:26 /opt/jobs/newbin/report
$ report
session.sh: line 5: /opt/jobs/bin/report: No such file or directory
exit=127
$ type -a report
report is /opt/jobs/newbin/report
$ hash -t report
/opt/jobs/bin/report
$ command -v report
/opt/jobs/bin/report

The file is there, on PATH, and bash ran the old path anyway. The first time it finds a command through a PATH search, it stores the full path in a hash table and skips the search after that. When the file moves, it executes the remembered path and reports No such file or directory with exit 127, the same code a typo gets.

The confusing part is the end of that output. type -a searches PATH fresh and finds the new location. command -v reports what bash would run, which is the stale one. hash -r clears the table, a new terminal starts empty, and shopt -s checkhash makes bash re-search when a hashed path has gone: with it set, the same sequence printed the tool's output and exit 0.

The stale hash is one of six causes behind command not found. The others, from a cron PATH to a CRLF shebang, each reproduced with a script that names the cause, are in command not found in bash, six causes.

Every disk-alert script I had piped into a command this machine does not have

Thu, 15 Oct 2026 13:00:00 +0000

The standard bash alerting pattern ends the same way everywhere you read it: pipe the finding into mail. So I checked what my own box would actually do with that line.

$ type -a mail mailx sendmail
mail not found
mailx not found
sendmail not found

Nothing. No MTA, no mailutils, no fallback. Any alert script written the usual way would have run its check correctly, found the problem correctly, piped the message into a command that does not exist, and told me nothing.

This fails quieter than it sounds. Under set -euo pipefail you at least get a non-zero exit somewhere a cron wrapper might notice. Without it, the pipeline's exit status is the last command's, cron mails you nothing because cron's mail also needs an MTA, and the script looks like it ran fine.

Check for the binary before you rely on it, not after an incident. And on a box that does have mail, exit 0 still only means the message reached a local MTA — on a host with no outbound relay it lands in a spool file under /var/mail that nobody reads. Confirm real delivery once by tailing /var/log/mail.log. Cron's minimal PATH is a second reason to test from inside cron rather than your login shell.

The msmtp relay config and the rate-limiting guard are in the bash email alert walkthrough.

One empty variable, one error, one wrong answer, exit 0

Wed, 14 Oct 2026 13:00:00 +0000

I wrote four lines to capture a bash error for a guide. One empty variable, tested twice:

$ cat queue-check.sh
#!/bin/bash
count=""
if [ $count -eq 0 ]; then echo "queue empty"; fi
if [[ $count -eq 0 ]]; then echo "queue empty ([[ ]])"; fi

$ bash queue-check.sh
queue-check.sh: line 3: [: -eq: unary operator expected
queue empty ([[ ]])
exit=0

The message blames -eq. The bug is the unquoted $count before it: word splitting removed the empty value, so [ received -eq 0 and failed with status 2, which an if treats as "not true". The [[ ]] line does no splitting, read the empty string as 0 in an arithmetic test, and printed a confident wrong answer. Then the script exited 0.

Strict mode does not help here. count="" is set, so set -u stays quiet, and set -e ignores failures inside an if condition. What stopped it was validating the value first: [[ $count =~ ^[0-9]+$ ]] || exit 1.

Most bash errors work this way. The message names the check that failed, not the cause: a syntax error reported on line 4 for a missing semicolon on line 2, Permission denied after chmod +x on a noexec mount. Nine of them, each reproduced with the check that names the cause, are in Bash Error Messages, Decoded.

My 30-day log cleanup kept a file that was 30 and a half days old

Thu, 08 Oct 2026 13:00:00 +0000

I set three log files to known ages with touch -d — exactly 30 days, 30 days and 12 hours, and 31 days — then ran what I thought was a 30-day retention policy:

find . -name '*.log' -mtime +30

It returned one file: the 31-day-old one. The 30.5-day file stayed. So did the 30-day file.

-mtime counts whole 24-hour periods and throws away the remainder, and +30 means strictly greater than 30 of them. A file aged 30 days and 12 hours counts as 30, and 30 is not greater than 30. A real 30-day policy is -mtime +29. Re-running with +29 returned all three, which is what I wanted in the first place.

Two more things that bit during the same session. Piping to xargs rm split app v2.log on the space — rm reported cannot remove 'ws/app' and cannot remove 'v2.log', and the file survived. And -delete is an action that returns true, so putting it before your filter is catastrophic: find ord -delete -name '*.log' removed a file called keep.txt and then the directory itself. Preview with -print, keep -delete last.

The multi-directory loop, the .gz variant, and the cron lines are in the find -mtime log retention walkthrough.

Catch a filling disk weeks before it becomes an outage

Tue, 06 Oct 2026 13:00:00 +0000

A disk almost never fails loudly. It fills a percent at a time until writes stop, and at 100% nothing reports "disk full" — the database throws write errors about its own files, apps crash-loop, and logging (a write itself) stops recording at the exact moment you need evidence. You end up debugging five fake problems instead of one real one.

The prevention costs one line on a daily cron:

df -h | awk 'NR>1 && $5+0 >= 80 {print $0}'

NR>1 skips the header. $5+0 is the trick: df's Use% column is a string like 82%, and awk's numeric coercion keeps the leading digits and drops the %, so the threshold comparison works without cut or sed. Pipe any output to mail -s and you get an email the day a filesystem crosses 80.

Why 80? At 80% a disk growing one percent a day leaves you weeks to investigate with du -ah | sort -rh | head -20. At 95% you're deleting files under pressure; at 100% every service on the box misbehaves at once.

This check is #1 in a set of 25 cron-ready bash scripts for sysadmins covering disk monitoring, backups, service watchdogs, SSL expiry, and port audits.

sed 's/port/listen_port/g' rewrote four lines of a four-line config

Thu, 01 Oct 2026 13:00:00 +0000

A four-line YAML file (port: 8080, support_email, transport: tcp, report_dir: /var/reports) and the rename most people type first: sed -i 's/port/listen_port/g' app.yml. GNU sed 4.9 exited 0 and printed nothing, because -i never does. Afterwards the file held suplisten_port_email, translisten_port and /var/relisten_ports. One line was meant to change. All four did.

sed matches characters, not words. The letters p-o-r-t sit inside support, transport and report, and g replaced every occurrence. GNU sed's \b word boundary fixes the match, and one pipe shows the damage before anything is written:

sed 's/\bport\b/listen_port/g' app.yml | diff app.yml -

Without -i the file on disk stays intact, and diff prints the lines that would change. It exits 1 when it finds differences, so the same check works inside a script. Only after the diff reads right does -i go back on.

Two caveats before trusting \b. Underscore counts as a word character, so db_port stays untouched. And \b is a GNU extension: BSD sed on macOS spells word boundaries [[:<:]] and [[:>:]], and its -i requires a suffix argument (sed -i '') where GNU's is optional.

The replacement side bites too. An unescaped & means the whole match, so s/CEO/Smith & Sons/ produces Smith CEO Sons.

The bulk-replace script, with a grep -rl file list, per-file diff and an --apply gate that fails toward reading, is on the sed find-and-replace snippet page.

What a Linux Sysadmin Actually Does All Day (Spoiler: It's Not Just `sudo apt update`)

Fri, 25 Sep 2026 15:27:27 +0000

Somewhere out there is a job posting that describes a "Linux System Administrator" as someone who "maintains servers." That description is technically true the way "doctors deal with bodies" is technically true. It tells you nothing, it undersells the job by an order of magnitude, and it's why half the people who land in this role are shocked, three weeks in, to discover they're now responsible for network routing, certificate rotation, backup verification, and being the calm voice at 3 a.m. when a disk fills up and takes production with it.

Let's actually map the territory, without the job-posting euphemisms.

Everything Except Bending the Metal

If it isn't literally designing server hardware, it's probably yours. The realistic scope of the role covers the entire operating system lifecycle — installation, configuration, patching, the accounts and permissions layered on top of all of it — plus every service that depends on that OS staying upright: web servers, databases, file storage, whatever the business actually makes money from. Add the network layer on top: interfaces, firewalls, routing at the host level, VPNs. Add backups, and not just "having" them — configuring them correctly, testing restores on a schedule instead of assuming they'll work when it actually matters, and snapshotting filesystems and databases before you're the reason someone needs them. Add security on top of all of that: access rights, certificates, vulnerability monitoring, log auditing, and standing between the box and whatever's currently trying to get into it.

That's not a job description. That's five job descriptions wearing a trench coat, and the fact that it's usually billed as one role is either a testament to how capable good sysadmins are or a quiet admission that most companies would rather understaff this than budget for it properly. I'll let you guess which one it usually is.

Prevention Beats Heroics, Every Single Time

The single most underrated skill in this job isn't fixing things fast. It's noticing them slowly enough in advance that "fixing" was never required.

Monitor before it's an emergency. Zabbix and Prometheus exist to tell you a disk is filling up while you still have options, not after the pager goes off and options have narrowed to "explain this in the incident channel." Prometheus's pull-based, time-series model has become the default assumption for anything running in Kubernetes, while Zabbix still earns its keep in more traditional, heterogeneous fleets where you need broad out-of-the-box coverage without writing your own exporters for everything. Neither is objectively "better" — they're solving slightly different shapes of the same problem, and plenty of shops run both without embarrassment.

Visualize, don't just collect. Raw metrics are where good intentions go to die unread. Grafana turns a wall of numbers nobody checks into a dashboard people actually glance at on the way to their coffee, which is the entire difference between "we had the data" and "we noticed the problem."

Automate the repetitive stuff before it automates a mistake into your fleet instead. This is where the tooling landscape has genuinely shifted, and it's worth being current about it instead of reciting the same three names out of habit. Puppet and Chef both still exist — Chef now lives under Progress Software, Puppet under Perforce — and both remain entirely legitimate choices in shops that already standardized on their agent-based, declarative model years ago. But the gravitational centre of new automation work has moved decisively toward Ansible's agentless, SSH-based approach, precisely because it doesn't require you to pre-install anything on a target before you can manage it. Centralizing configuration this way isn't about looking modern. It's about making sure the fifty servers behind you were actually configured the way you meant, instead of the way you remembered meaning it three deploys ago.

Document like someone else — possibly a more tired version of you — will have to clean up after this change at 2 a.m. What changed, why, and what actually happened as a result. A wiki without an update discipline is just an elaborate way of lying to your future self, and version-controlling your configuration alongside that documentation is the difference between "we can see exactly what changed" and "we're diffing memories."

Treat backups as a product, not a checkbox. A backup job that's never been test-restored isn't a backup. It's a belief system, and belief systems have a terrible track record during actual outages.

The OS Is the Whole Foundation, So Know What You're Standing On

Everything above assumes a server operating system doing its actual job underneath: managing hardware resources, running services, handling networking, enforcing security boundaries and access separation. Get comfortable with the fact that which OS family you're standing on changes almost everything else about how you'll do this job.

The Linux Families, and What Actually Distinguishes Them

Debian-based — Debian itself, Ubuntu, Mint. The reputation for stability and thorough documentation is earned, not marketing. Ubuntu Server in particular remains the default on-ramp for small and mid-sized shops precisely because the learning curve is forgiving and the community answer to almost any problem already exists somewhere.

RHEL-based — Red Hat Enterprise Linux, AlmaLinux, Rocky Linux, and CentOS, which deserves a specific note because its story isn't finished the way older tutorials imply. CentOS as the free, bit-for-bit downstream rebuild of RHEL is functionally gone — Red Hat shifted it to CentOS Stream, a rolling preview that sits upstream of RHEL rather than mirroring it, and then tightened access to RHEL's own source in a way that made several downstream projects' lives noticeably harder. AlmaLinux and Rocky Linux exist specifically because that gap needed filling, and both have become the practical default wherever a shop needs RHEL-compatible stability without a RHEL support contract. This is still the corner of the Linux world enterprises reach for when "boring and well-supported for a decade" beats "exciting."

Arch-based — Arch Linux, Manjaro. The pitch here isn't convenience, it's comprehension: a rolling-release model and a famously minimal starting point that force you to understand what you're installing instead of inheriting someone else's defaults. Fewer production servers run it than you'd guess from how often it comes up in conversation, which tells you something honest about the gap between "educational" and "operationally boring in the best way."

Unix Is the Ancestor, Not a Competitor Wearing a Costume

FreeBSD and OpenBSD carry the Unix lineage forward as genuinely open, still-maintained systems — FreeBSD as a long-standing web-serving workhorse, OpenBSD as the option people reach for specifically when "we obsess over security defaults" needs to be true rather than aspirational, which is why it shows up disproportionately often inside routers and firewalls.

The distinction that actually matters, though, isn't "which is better." It's understanding that Linux isn't a Unix clone — it's an independent implementation built in the spirit of Unix's ideas and conventions: the filesystem model, process management, the permissions structure, the whole shape of working in a terminal. When someone says "Linux and other Unix-like systems," that's not sloppy phrasing. That's the actual, accurate relationship.

Where the Proprietary Options Still Fit

Windows Server remains the correct call the moment Active Directory, Exchange, or deep integration with the Microsoft stack is the actual requirement rather than a default nobody questioned. macOS, meanwhile, quietly stopped being a distinct server product entirely — Apple folded macOS Server's functionality back into the base OS years ago, and what's left shows up almost exclusively inside small, Apple-native shops rather than anything resembling general-purpose server infrastructure. It also remains the one entry on this list you can't actually buy on its own — it comes bundled with the hardware, whether you wanted the bundle or not.

Why This Is Still Worth Learning, Bluntly

If you're deciding whether Linux administration deserves real time investment, the honest case doesn't need embellishment. Depending on which measurement you trust, Linux runs somewhere between roughly half and the overwhelming majority of the world's servers, and among sites where the underlying OS can actually be identified, the Unix-family total — Linux very much included — sits comfortably above nine in ten. Virtually every top-1-million web server and every single one of the world's fastest supercomputers has been running it for years now, and every major public cloud treats Linux as the load-bearing default rather than a supported alternative.

Beyond the numbers, the actual reasons to learn it hold up on their own merits: it scales cleanly from a five-dollar VPS to an entire data centre without switching mental models, the multi-user permissions architecture was security-conscious from its foundations rather than bolted on afterward, and — the reason I'd actually put first — it's the rare piece of foundational infrastructure you're genuinely allowed to open up and understand completely. You can read the source, modify the kernel, rebuild a component that annoys you, and nobody's licensing department is going to stop you. That's not a minor philosophical footnote. It's the entire reason this operating system family ended up running the internet instead of just participating in it.

Why I prefer systemd timers over cronjobs

Thu, 24 Sep 2026 13:39:29 +0000

Most users would schedule their tasks using a cronjob, and that's perfectly reasonable

Introduction

I will assume you are running on a linux machine with systemd (which you should be if you're running on a server), other init systems such as runit or openRC are not taken into consideration

The content is mainly targeted for servers, however you can apply these concepts to your personal machines

What is a cronjob

Cronjobs are essentially used for scheduling tasks at certain intervals or exact time(s)

The task could be a shell script, a command, a call, anything you may need

Example Usage

Creating a cronjob is very easy

Run crontab -e

Paste this:

*/1 * * * * echo "$(date -Iminutes) Running" >> /tmp/cronjob_running.txt

This would echo the current datetime in ISO format to file /tmp/cronjob_running.txt every minute

Simple, easy and straightforward

Problem

This works fine and well for personal, simple or non important tasks.

But let's take this job:

0 0 * * * /usr/local/sbin/postgresql_backup.sh >> /var/log/postgres/backups/backup.log 2>&1

This job would run the postgresql_backup.sh script every day at 00:00 and output the logs AND errors to /var/log/postgres/backups/backup.log

What happens when the machine goes down at 00:00? or something even more probable, cron's systemd unit (like cronie) stops or does not start because someone forgot to (re)enable it.

Answer: The job never runs

Systemd timers

What is systemd

It's the core init unit of your linux system, when your machine boots up, the kernel will load it with a pid of 1

A systemd timer is, as the name suggests, a timer that is managed by systemd

So basically, the solution is already integrated into your Operating System, you're just not using it!

What's more, they have more precision than cronjobs do and they support much finer scheduling granularity.

How to create these timers

Now I'll be honest, creating them is a bit of a pain and not as quick as cronjobs, but we'll talk about the differences at the end!

For the sake of simplicity I'll keep the example very straightforward, but note you can really get a bit more complex with these

First, you create a user systemd unit by running:

systemctl --user edit --force --full example-unit.service

We're creating a user scoped timer for this example, but for more serious work like a database backup, use sudo and omit user, like this: sudo systemctl edit --force --full example-unit.service

The --force --full flags are used to create the unit if it doesn't exist, I use the systemctl edit command so that if we exit with an empty file, we won't have issues, plus it does create a backup for us which is handy

Then put this into the content

[Unit]
Description=Our Example systemd Unit

[Service]
ExecStart=/usr/bin/echo "Hello There!"

[Install]
WantedBy=default.target

Save and exit
Reload your systemd daemon:

systemctl --user daemon-reload
Run the unit

systemctl --user start example-unit.service

Nothing happens, why? where's my output?

You inspect your service like any other systemd unit.. with status subcommand

Run this

systemctl --user status example-unit.service

You get something similar to this:

But where's the schedule here?
Well, this is why I said

creating them is a bit of a pain and not as quick as cronjobs

The Timer

Finally, it's time we create the actual timer unit

Run this command:

systemctl --user edit --force --full example-unit.timer

The same goes here, omit the --user flag and add sudo if doing a system wide timer!

Add this

[Unit]
Description=Running Our Example Unit regularly

[Timer]
OnCalendar=*:0/1
# This here is our magic flag
Persistent=true
Unit=example-unit.service

[Install]
WantedBy=timers.target

Save and exit
Run this command

systemctl --user start example-unit.timer

NOTE: We're only starting the timer here, if we reboot the machine the timer will not persist
For the timer to remain enabled, you do systemctl --user enable --now example-unit.timer

Here's what you get:

This will run our unit every minute, and with our magic Persistent=true even if your machine goes down, or for whatever reason it gets missed due to some random bit flipping, your timer WILL run when it gets the chance

It stores the last execution timestamp on disk. When the timer becomes active, systemd checks this file, and if the target trigger time passed while the system was powered off it immediately triggers the service.

Comparing systemd timers to other solutions

Now for more informed readers, you may know of anacron which is a perfectly reasonable solution however there are some differences you need to be aware of.

Here's a comparison to systemd timers, cronjobs and anacron

Solution	Simple to create	Flexible Scheduling	Runs on missed tasks
Systemd	❌	✅	✅
Anacron	✅	❌	✅
Cronjob	✅	✅	❌

anacron doesn't have that much flexibility, they have daily basis checkup, which you may or may not want, so beware of that

Systemd timers Scheduling

You may have noticed that we used the OnCalendar keyword in our timer, it's not the only one! there are more things you can configure

Monotonic Timers

These basically run relative to something, it can be relative to system boot, it can be time since the unit was activated!

Here's a small table of keywords and schedules you can have that I borrowed from this great Article

Keyword	Meaning
OnActiveSec	Schedule the task relative to the time when the timer unit itself is activated
OnBootSec	Schedule task relative to the system boot time
OnStartupSec	Schedule the task relative to the time when systemd started
OnUnitActiveSec	Schedule the task relative to the last time the service unit was active
OnUnitInactiveSec	Schedule the task relative to the last time the service unit was inactive

Seconds are used as the default unit of time. We can specify a different unit after the value (e.g. 15m for fifteen minutes)

Realtime Timers

These are the ones that use the OnCalendar keyword

The basic syntax is:

OnCalendar=Mon *-*-* 12:30:00
           │   │ │ │ │  │  │
           │   │ │ │ │  │  └── SECOND
           │   │ │ │ │  └───── MINUTE
           │   │ │ │ └──────── HOUR
           │   │ │ └────────── DAY OF MONTH
           │   │ └──────────── MONTH
           │   └────────────── YEAR
           └────────────────── WEEKDAY

Conclusion

Systemd Timers are great for scheduling important tasks or interacting with a daemon that you already installed.
So are they the solution for everything? No, they're cumbersome to create and honestly I don't like the way it's handled.

Use the right tool for the job, if it's a business critical backup that runs twice a day? a very important data collection task? Use systemd timers, if it's simply another task that would not harm if it was missed once, use cronjobs, they're much easier to create

There are many things that I have not mentioned in this article, and things I didn't go into depth into but I hope this article helps future engineers with their journey in automating and scheduling

I sent my report script SIGTERM mid-write. Yesterday's output survived intact.

Thu, 24 Sep 2026 13:00:00 +0000

A script that writes straight to the path a consumer reads can die halfway and leave a torn file that everything downstream loads as complete. No error appears anywhere, because the only process that failed is gone. I tested the fix the unfriendly way on bash 5.3.9: a CSV generator with a sleep 30 planted between rows, killed with SIGTERM one second in.

TMP_FILE=$(mktemp)
cleanup() { local code=$?; rm -f "$TMP_FILE"; exit "$code"; }
trap cleanup EXIT
generate_rows > "$TMP_FILE"
mv "$TMP_FILE" "$FINAL_PATH"

Exit 143, which is 128 plus signal 15. The temp file was gone, and out.csv still held the previous run's three complete lines. Stale is something monitoring can see. Torn is not.

EXIT fires on normal completion, an explicit exit 1, a set -e abort and after a handled signal, which is why this is five lines instead of twenty. And because mv on one filesystem is atomic, being half-written and being at the published path are never true at once. Even kill -9, which no trap catches, cannot publish a torn file.

The ordering is the part people move. I ran four handler variants against a script ending in exit 3, and only one exited 0: rm -f first, then local code=$?. It captured rm's success and laundered the failure. Capture first, clean second, exit last.

The complete script, subshell scoping rules and a line-by-line breakdown are on the trap cleanup page for temp files and atomic writes.

Who owns that port? One ss flag answers it without sudo

Wed, 16 Sep 2026 17:29:08 +0000

Who owns that port? One ss flag answers it without sudo

For months my dev server started on 3001 because something already held 3000, and I kept stepping around it. When I finally asked, ss -ltn 'sport = :3000' showed the port listening on every interface — with an empty Process column, because ss -p only names sockets owned by your own user. Without root, it looked like nobody owned it.

One flag closes most of that gap:

ss -ltnpe 'sport = :3000'

-e prints the socket's cgroup, and on a systemd machine the end of that path is the unit that opened it. Mine ended in docker.service. docker ps then named it: a docker-proxy publishing a container I had started in the spring and forgotten, reachable from the whole network.

Two things worth carrying away. The host only ever sees Docker's proxy, never the application behind it, so a host-side port list tells you what is exposed, not what is running. And "list open ports" is really three questions: what is listening, who owns it, and whether it is reachable. They need three different commands.

The full write-up covers all three — with and without root, without netstat, inside containers, and an audit script that diffs the listener list between runs: how to list open ports on Linux with the owning process.

Why I run my home lab like enterprise infrastructure

Sat, 22 Aug 2026 20:19:51 +0000

Originally published on DEV Community in July 2026. This version is updated with the current state of the lab and a new section on what happened when I finally tested my backups.

I'm an IT Systems Administrator / Service Desk Team Lead L2 in Torino, Italy. During the day I manage endpoints and infrastructure for a large public-sector IT provider. At night I run PolarisCore, a home lab built with the same standards I use at work: VLAN segmentation, a real reverse proxy, internal DNS, ZFS storage with redundancy, and monitoring. Not a pile of Docker containers on a single box.

Here's the current state, pulled live via SSH an hour before publishing.

The compute layer — Proxmox

When I wrote the original version of this article, the lab ran on a single Proxmox node with 7.5 GiB of RAM. I had assigned about 10 GiB across the containers. Linux lets you over-commit memory, so it worked until it didn't, and the OOM killer took a container roughly once a month.

A mainboard swap fixed that this week:

pve-manager/9.2.11 (kernel 7.0.14-12-pve)
Host: HP EliteDesk 800 G5, i7-8700T (12 threads)
Uptime: 4 hours (fresh install)
RAM: 12 / 31 GiB used
ZFS rpool (NVMe mirror): 74.3 GB / 236 GB (31%)
Guests: 14 LXC running / 17 total, plus 1 VM

Those containers handle a Caddy reverse proxy, internal DNS (PowerDNS + Pi-hole), a self-built deployment tool, a Next.js search frontend, a job scheduler, an asset inventory, and a handful of smaller services. VLANs separate management, services, IoT, gaming, guest and VPN traffic, with the services VLAN carrying most of what faces the internet.

One detail worth stealing: the node has a single physical NIC. Every VLAN reaches it through one VLAN-aware Linux bridge with 802.1Q tagging, and the switch port is a trunk.

iface nic0 inet manual

auto vmbr0
    bridge-ports nic0
    bridge-vlan-aware yes
    bridge-vids 2-4094

I spent a day convinced I needed a second network card before I checked how the old node was actually wired and found it had been doing exactly this all along. Read your own configs before you buy hardware.

The storage layer — TrueNAS SCALE

TrueNAS 25.10.6
Uptime: 9 hours
Pool DATA (RAIDZ1, 4x8TB): 14.1 TB / 21.8 TB used (64%)
Pool APPS (mirror): 93.8 GB / 460 GB used (20%)
Pool boot: 10.2 GB / 236 GB used (4%)
All pools: ONLINE

22 TB isn't there for the sake of it. It holds a media library (Jellyfin and the *arr stack, running as native TrueNAS apps rather than VMs), project data, and a dedicated encrypted share. RAIDZ1 across four drives gives me a single-drive failure buffer, which for a setup holding real data matters more to me than raw speed.

The part I got wrong: backups

This is the section that wasn't in the original article, and it's the one worth reading.

While preparing the container migration to the new node, I opened the backup storage to confirm I had a rollback path. The most recent archive was dated 19 April. It covered 4 containers out of 14.

For four months I believed I had a safety net. It covered less than a third of the machines and was four months stale. Nothing had alerted me. No email, no red icon in the interface, no failing job. The backup job was configured against an explicit list of container IDs, that list had been written once, and the lab had grown past it. A job that backs up four containers successfully reports success.

So I ran a full backup of everything. It failed on all seventeen guests with the same error:

tar: failed to return to initial working directory: Permission denied

The cause is a two-layer UID translation. Unprivileged LXC containers map their internal users to a host range starting at 100000. The NFS server the backups were written to applies its own remapping on top. The result is a tar process whose working directory becomes unreachable to it partway through, so it can't chdir back when it finishes.

The fix is one flag:

vzdump --all 1 --storage <target> --mode snapshot --tmpdir /var/tmp/vzdump

--tmpdir keeps the working directory on local disk instead of the network share. The archive still lands on the share; only the scratch path moves.

Seventeen archives, then seventeen restores onto the new node. Sixteen of sixteen running containers came back on the first attempt, and the seventeenth was already powered off by design. That was the first time those archives had ever been read by anyone.

The lesson isn't "make backups." Everyone makes backups. The lesson is that a backup job reports on what you told it to do, not on what you needed it to do, and the only way to learn the difference is to restore.

The backup layer — Proxmox Backup Server

The old node didn't get retired. It became the backup target:

Proxmox Backup Server 4.2.5
Host: HP EliteDesk 800 G4, i5-8500T (6 cores), 7.53 GiB RAM
Datastore backup-ct: 40.20 GB / 245.55 GB used (16.37%)
Backup count: 17 CT groups, 1 VM group — 18 snapshots

The first full run finished in ten minutes for eighteen guests. It now runs nightly at 02:30 against every guest, with deduplication, a weekly integrity verification job, and a retention policy that keeps 7 daily, 4 weekly and 6 monthly snapshots.

Three properties make this better than what I had before:

It backs up --all, not a list. A new container is protected the night it's created, with no config change. That single flag is the direct fix for the failure above.
Deduplication is content-addressed at the chunk level. Seventeen Debian containers share most of their filesystem, so they cost far less than seventeen times one container.
Verification is a separate scheduled job. Restoring is still the only real test, but a weekly checksum pass catches silent corruption without me remembering to look.

One thing I have not solved: the backup server sits in the same cabinet as the machine it backs up. That protects me against disk failure and my own mistakes. It does nothing against fire or theft. An off-site copy of the critical subset is the next job on the list, and I'd rather say that out loud than let the diagram imply I'm covered.

The alerting trap, again

I configured email notifications and sent a test. It bounced:

550-5.7.1 Messages missing a valid Message-ID header are not accepted

Gmail doesn't accept mail sent directly from a residential IP address. The fix was routing through an authenticated SMTP relay on port 587 with an app password instead of delivering directly.

Notice the shape of this. If I hadn't sent that test, the only channel that tells me a backup failed would have been silently discarding every message. I would have rebuilt exactly the situation I had just spent a day fixing: a system that looks healthy because nothing is reporting.

Test the alerting path, not only the thing being alerted on.

The network layer

FTTH 10 Gbps from TIM into a UniFi Cloud Gateway Max, distributed over a 2.5 GbE backbone to switches and clients. The 10 Gbps WAN isn't about single-stream throughput to the internet. It's headroom, so the internal backbone never bottlenecks against the uplink when backups, streaming and syncs run in parallel. TLS is a wildcard certificate issued via DNS-01, auto-renewing, with zero inbound ports opened for issuance.

A debugging note from this week that cost me an hour. The new node negotiated its link at 100 Mb/s while everything else ran at 1000 or 2500. My first theory was a bad cable, and I was wrong: the cable was a perfectly good Cat 7. ethtool showed the real answer, that the machine advertised 1000baseT/Full while the link partner advertised only up to 100baseT/Full. Autonegotiation had settled on 100 during one of the many power cycles that day and never retried, because it doesn't. Forcing the speed once and then re-enabling autoneg fixed it, and a small boot service now re-negotiates if it ever finds itself at 100 again.

ethtool <iface> | grep -A2 "Link partner advertised link modes"

That one command separates "the cable is broken" from "the link is stuck", and they have completely different fixes.

The lightweight cloud piece — Oracle Always Free

Not everything lives on-prem. Site analytics run on Umami, self-hosted, replacing GA4 across five sites, on an Oracle Cloud Always Free VM:

Ubuntu 20.04.6 LTS
VM.Standard.A1.Flex — 1 OCPU / 6 GB RAM
RAM used: 694 MiB (11%)
Uptime: 6 weeks
Stack: Umami + Postgres 15 + Caddy, Docker Compose

Six weeks of uptime and it barely registers on CPU. For a low-traffic analytics backend the free tier is more than enough, and there's no reason to pay for a VPS to do this.

A side project that came out of daily frustration

The one piece of software here that isn't infrastructure glue is NovaSCM, an open-source self-hosted alternative to Microsoft SCCM for zero-touch Windows deployment: PXE boot, autounattend generation, network scanning, WiFi 802.1X certificate management, and workflow-based software deployment. It's MIT-licensed, lives on GitHub, and runs as one of the containers above.

Why bother with all this at home

Building it the enterprise way, with segmentation, real certificate automation, redundant storage and a reverse proxy instead of exposed ports, is the best way I know to understand those patterns rather than read about them.

The backup story above is the argument in miniature. I have written backup procedures at work. I have reviewed them. I still had a four-month gap in my own lab, because reviewing a procedure and restoring from it are different activities, and only one of them tells you the truth. When something breaks at 11 pm on my own network, nobody is going to open a ticket about it, and there's no second line to escalate to. That's a different kind of pressure, and it teaches differently.

More on the project and the services I haven't written up yet at polariscore.it.

The 4 things every cronjob needs that most tutorials skip

Wed, 06 May 2026 21:23:10 +0000

Cron job tutorials have a problem. They teach you the five-field syntax, show you a couple of examples like `0 2 * * `, and send you on your way. What they don't cover is everything that needs to go around* the expression for the job to actually work reliably in production.

Here's what I mean.

Problem 1: Cron's PATH is not your PATH

When you run a command in your terminal, your shell has a full PATH — all the binary directories your distro configured, plus whatever you've added. When cron runs, it uses a stripped-down default PATH. Binaries that are available to you interactively may not be visible to cron.

The fix is to set PATH explicitly in your crontab or at the top of your wrapper script:

PATH=/usr/local/bin:/usr/bin:/bin

This single line prevents the most common class of "cron job runs but does nothing" failures.

Problem 2: Cron's default shell is sh, not bash

If your script uses any bash-specific syntax — arrays, `[[ ]]` conditionals, `$()` substitution styles — and cron is running it with sh, you'll get silent failures or unexpected behavior.

SHELL=/bin/bash

Add this to the top of your crontab. One line.

Problem 3: No logging means no debugging

If a cron job fails, where does the error go? By default: nowhere. It vanishes. Adding log redirection to every crontab entry is the fix:

0 2 * * * /path/to/script.sh >> /var/log/mybackup.log 2>&1

The `2>&1` captures stderr alongside stdout so you see command errors, not just your script's output.

Problem 4: Long-running jobs can stack on top of themselves

If you have a job that runs every 5 minutes but sometimes takes 8 minutes, a second instance starts before the first one finishes. Depending on what the script is doing (writing to a file, syncing a directory, running a database query), this can corrupt output.

Lock files via `flock` prevent this:

/usr/bin/flock -n /tmp/myjob.lock /path/to/script.sh

The second instance sees the lock is held and exits immediately.

All of this is in a free tool

I built Cron Job Builder to handle all four of these automatically. You pick your schedule from dropdowns, check the options you want (logging, PATH, SHELL, MAILTO, lock file), and it generates the complete crontab entry — expression + wrappers + bash header — ready to paste into `crontab -e`.

It also shows you the next 10 scheduled run times so you can verify your expression is actually doing what you intend before you deploy it.

There are quick presets for the schedules you use constantly: daily 2am, every 6 hours, weekdays 9am, monthly on the 1st, every 15 minutes.

→ https://bashsnippets.xyz/tools/cron-job-builder.html

The rest of the library — free bash scripts for backup, log cleanup, disk monitoring, uptime checking — lives at https://bashsnippets.xyz.

Linux Fundamentals For DevOps: Linux core mechanism

Mon, 27 Apr 2026 16:45:13 +0000

Knowing where files live is only half the battle. To be an effective DevOps engineer, you need to know how Linux actually executes programs, manages their lifecycles, and secures them.
The crucial concepts you need to troubleshoot effectively.

The `$PATH` Variable

Linux does not search your entire hard drive when you type a command. It only checks a specific, prioritized list of folders. This list is saved in an environment variable called $PATH.

You download a new binary tool (like kubectl or terraform). You type its name in the terminal, but Linux replies with the dreaded command not found. To fix this, you must either move the binary into a folder that is already in your $PATH, or add your custom folder to the $PATH.

# Command: View the list of directories Linux currently checks for commands

echo $PATH

# Output Proof (Directories are separated by colons):
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Command: Add your custom scripts folder to the PATH for the current session

export PATH="/home/saeed/custom-scripts:$PATH"

How Binaries Work (Shared Libraries)

When you run a command like nginx or python, you are executing a binary file. In Linux, executables are typically formatted as ELF (Executable and Linkable Format) files. However, most binaries are not standalone; they rely on external pieces of code called shared libraries (.so files, similar to .dll files in Windows).

You compile a Go application or migrate a tool into a minimal Docker container (like Alpine Linux). When you run it, it crashes silently or throws a confusing no such file or directory error—even though you can see the binary sitting right there. Usually, the binary isn't missing; its required libraries are missing.

You can use the ldd command to print the shared libraries required by any program.

# Command: Verify exactly which shared libraries Nginx needs to run
ldd /usr/sbin/nginx

# Output Proof:
        linux-vdso.so.1 (0x00007ffe3b1f4000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f3a8b411000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f3a8b39c000)
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007f3a8b2f6000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3a8b0cd000)

File Permission

Linux strict permissions dictate exactly who can Read, Write, or eXecute a file. This is often represented by numbers (like 777 or 644).

You are trying to connect to a cloud server via SSH using a private key (.pem file), but the connection is instantly rejected. If you set a .pem file to 777 (meaning everyone on the system can read, write, and execute it), SSH will intentionally reject it for being too exposed. Private keys must be set to 400—meaning absolutely no one except the owner is allowed to read it.

# Command: Attempting to SSH with a bad permission key
ssh -i my-key.pem ubuntu@10.0.0.5

# Output Proof:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0777 for 'my-key.pem' are too open.

# Command: Secure the key so only the owner can read it (4), and no one can write (0) or execute (0)

chmod 400 my-key.pem

Processes and Signals

Every running program is a process assigned a unique Process ID (PID). The operating system communicates with and controls these processes using "Signals."

An application is hogging 100% of the CPU and needs to be stopped, or Kubernetes is trying to scale down a pod without dropping active user connections. You can send a SIGTERM (Signal 15), which politely asks the program to save its data and exit cleanly. Alternatively, a SIGKILL (Signal 9) forcefully terminates it at the hardware level, bypassing cleanup and potentially causing data corruption.

# Command: Find the Process ID of an application
pgrep -la nginx

# Output Proof:
1234 nginx: master process
1235 nginx: worker process

# Command: Politely ask the master process to shut down and clean up
kill -15 1234

File Descriptors

Remember concept , "Everything is a File"? That rule has a dark side.

Because Linux treats network connections as files, every time a user connects to your web server, the operating system opens a "File Descriptor" to track that connection. To protect itself from crashing, the Linux kernel sets strict limits on how many file descriptors a single process can open at once.

Your Nginx web server or Node.js application is suddenly dropping user connections during a high-traffic spike. You check the CPU and RAM, and both are perfectly fine. You check the Nginx error logs and see the dreaded worker_connections are not enough or Too many open files error. The operating system has forcibly stopped your app from accepting new traffic.

# Command: Show the maximum number of open file descriptors allowed for this user
ulimit -n

# Output Proof:
1024

# Command: Check how many file descriptors the entire system is currently using
cat /proc/sys/fs/file-nr

# Output Proof:
1205   0   9223372036854775807

Blog Summary

Understanding Linux goes beyond knowing where files live; it’s about understanding how the operating system executes code, secures data, and manages its own limits. This guide breaks down the invisible mechanics of Linux that every DevOps engineer must master to troubleshoot effectively.

You'll learn why binaries silently fail due to missing shared libraries, how to quickly fix "command not found" errors using the $PATH variable, and the strict permission rules required to secure SSH keys. We also cover how to safely manage and terminate running processes using signals, and finally, uncover the hidden trap of File Descriptors—explaining why servers suddenly drop connections with "Too many open files" errors and how to raise those limits.

Coder Legion - Recent posts tagged sysadmin

Finding the world-writable files on your server before someone else does

The log was deleted. Its 48M was still allocated.

Patch These Four Linux Kernel Local Root Vulns Now: DirtyAH6, PPPoEject, TUNderflow, and DiagSpill

💡 Why This Matters for Homelabbers

🧰 Prerequisites

🔧 Step-by-Step: Patching Your Nodes

🛡️ Patch Status, Checked 29 September 2026

🛡️ What Each Vulnerability Affects

🛡️ Hardening Beyond the Patch

🔧 Troubleshooting

✅ Wrap Up

Related from the lab

bash ran a path that no longer existed, and exited 127

Every disk-alert script I had piped into a command this machine does not have

One empty variable, one error, one wrong answer, exit 0

My 30-day log cleanup kept a file that was 30 and a half days old

Catch a filling disk weeks before it becomes an outage

sed 's/port/listen_port/g' rewrote four lines of a four-line config

What a Linux Sysadmin Actually Does All Day (Spoiler: It's Not Just `sudo apt update`)

Everything Except Bending the Metal

Prevention Beats Heroics, Every Single Time

The OS Is the Whole Foundation, So Know What You're Standing On

The Linux Families, and What Actually Distinguishes Them

Unix Is the Ancestor, Not a Competitor Wearing a Costume

Where the Proprietary Options Still Fit

Why This Is Still Worth Learning, Bluntly

Why I prefer systemd timers over cronjobs

Introduction

What is a cronjob

Example Usage

Problem

Systemd timers

What is systemd

How to create these timers

The Timer

Comparing systemd timers to other solutions

Systemd timers Scheduling

Monotonic Timers

Realtime Timers

Conclusion

I sent my report script SIGTERM mid-write. Yesterday's output survived intact.

Who owns that port? One ss flag answers it without sudo

Who owns that port? One ss flag answers it without sudo

Why I run my home lab like enterprise infrastructure

The compute layer — Proxmox

The storage layer — TrueNAS SCALE

The part I got wrong: backups

The backup layer — Proxmox Backup Server

The alerting trap, again

The network layer

The lightweight cloud piece — Oracle Always Free

A side project that came out of daily frustration

Why bother with all this at home

The 4 things every cronjob needs that most tutorials skip

Cron job tutorials have a problem. They teach you the five-field syntax, show you a couple of examples like 0 2 * * *, and send you on your way. What they don't cover is everything that needs to go around the expression for the job to actually work reliably in production.

Here's what I mean.

When you run a command in your terminal, your shell has a full PATH — all the binary directories your distro configured, plus whatever you've added. When cron runs, it uses a stripped-down default PATH. Binaries that are available to you interactively may not be visible to cron.

The fix is to set PATH explicitly in your crontab or at the top of your wrapper script:

This single line prevents the most common class of "cron job runs but does nothing" failures.

If your script uses any bash-specific syntax — arrays, [[ ]] conditionals, $() substitution styles — and cron is running it with sh, you'll get silent failures or unexpected behavior.

Add this to the top of your crontab. One line.

If a cron job fails, where does the error go? By default: nowhere. It vanishes. Adding log redirection to every crontab entry is the fix:

The 2>&1 captures stderr alongside stdout so you see command errors, not just your script's output.

If you have a job that runs every 5 minutes but sometimes takes 8 minutes, a second instance starts before the first one finishes. Depending on what the script is doing (writing to a file, syncing a directory, running a database query), this can corrupt output.

Lock files via flock prevent this:

The second instance sees the lock is held and exits immediately.

I built Cron Job Builder to handle all four of these automatically. You pick your schedule from dropdowns, check the options you want (logging, PATH, SHELL, MAILTO, lock file), and it generates the complete crontab entry — expression + wrappers + bash header — ready to paste into crontab -e.

It also shows you the next 10 scheduled run times so you can verify your expression is actually doing what you intend before you deploy it.

There are quick presets for the schedules you use constantly: daily 2am, every 6 hours, weekdays 9am, monthly on the 1st, every 15 minutes.

No login, no account, runs entirely in the browser. Free.

The rest of the library — free bash scripts for backup, log cleanup, disk monitoring, uptime checking — lives at https://bashsnippets.xyz.

Linux Fundamentals For DevOps: Linux core mechanism

The $PATH Variable

How Binaries Work (Shared Libraries)

File Permission

Processes and Signals

File Descriptors

Blog Summary

Cron job tutorials have a problem. They teach you the five-field syntax, show you a couple of examples like `0 2 * * `, and send you on your way. What they don't cover is everything that needs to go around* the expression for the job to actually work reliably in production.

If your script uses any bash-specific syntax — arrays, `[[ ]]` conditionals, `$()` substitution styles — and cron is running it with sh, you'll get silent failures or unexpected behavior.

The `2>&1` captures stderr alongside stdout so you see command errors, not just your script's output.

Lock files via `flock` prevent this:

I built Cron Job Builder to handle all four of these automatically. You pick your schedule from dropdowns, check the options you want (logging, PATH, SHELL, MAILTO, lock file), and it generates the complete crontab entry — expression + wrappers + bash header — ready to paste into `crontab -e`.

The `$PATH` Variable