Coder Legion - Recent posts tagged bash

Finding the world-writable files on your server before someone else does

Thu, 29 Oct 2026 13:00:00 +0000

A world-writable file can be rewritten by every account on the machine — including the service account your web server runs as. Code executing as www-data through a compromised plugin doesn't need root: any chmod 777 you left behind during past debugging is a file it can overwrite or replace outright.

Two details break most homegrown audits. First, find -perm 777 (no dash) matches mode 777 exactly — a 666 file is equally world-writable and slips through. The dash form uses mask semantics and catches every file with the other-write bit set. Second, never fix findings with chmod -R 644: on directories, the execute bit means "may be entered," and stripping it recursively seals off the whole tree. Split the fix by type instead:

find /var/www -type f -perm -o+w 2>/dev/null
find /var/www -type f -exec chmod 644 {} \;
find /var/www -type d -exec chmod 755 {} \;

SSH keys and secrets go tighter, to 600 — OpenSSH refuses group- or world-readable private keys outright.

I keep the longer version as a script with a saved report file, a SUID scan, and a recommended-modes reference: full file-permissions security audit script.

The log was deleted. Its 48M was still allocated.

Wed, 28 Oct 2026 13:00:00 +0000

A 64M scratch tmpfs, a worker process holding its log open on file descriptor 3, and the usual reflex when space runs low: rm the log. du on the directory dropped from 48M to 0. df stayed at 48M used. The next 32M write failed with No space left on device, and df read 100% while du found only the 16M that had been written.

rm removes a name. The kernel frees the blocks only when no names are left and no process has the file open, and the worker still had it open. lsof +L1 lists exactly those files:

$ lsof -w +L1 /srv
COMMAND    PID USER FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
sleep   608657 root 3w   REG   0,78 50331648     0    3 /srv/app/worker.log (deleted)
exit=0

NLINK 0, (deleted), 48M held for writing by PID 608657. Truncating through the process's own descriptor, : > /proc/608657/fd/3, took df back to 25% while the process kept running.

Next time, empty a live log with : > file instead of deleting it. With an append-mode writer (>>) that is clean. A writer that opened with > keeps its old offset: in the same test, its log came back as a 17M sparse file with 1.0M on disk.

Finding which process holds a port or a busy mount, plus a script that groups deleted-but-open files and totals them, is on the lsof command examples page.

TruffleHog Secret Scanner Automation

Tue, 27 Oct 2026 16:34:53 +0000

Context and Problem

An AWS EKS Cluster was attacked, and its secrets were compromised. The Kubernetes Secrets were not encrypted, anyone who works with Kubernetes know they are only base64-encoded by default.

Task

I had a mission to scan 115 repositories across GitHub and Azure DevOps for secrets exposed anywhere in the Git history.

The goal was simple: identify exposed secrets and provide actionable reports for the engineering team to fix them.

A colleague recommended a tool called TruffleHog, which became the foundation of the workflow.

TruffleHog

"TruffleHog is a secrets scanning tool that digs deep into your code repositories to find secrets, passwords, and sensitive keys."

— TruffleHog official website

My Workflow

Read the TruffleHog documentation;
Ran a test in a personal GitHub repository to see how the tool works;
Check the total amount of repositories to scan, 115;
Designed how I was going to complete the Task:

- Decided to automate the process using Bash and jq.
- The script would receive the repository URL as a parameter.
- Used jq to filter and mask findings.
- Used jq to generate separate JSON files for active and inactive secrets.
- Generated standardized Notion reports to support remediation workflows.
- Used the generated JSON data with Gemini to create the reports.

Report Example

Lessons Learned

- Standardizing reports made remediation much faster.
- Active and inactive secrets should be separated to improve prioritization.
- Automation becomes essential when dealing with dozens of repositories.

Why Bash and jq?

Keep the workflow simple and easy to run.

jq made it easy to:

- filter findings
- mask sensitive values
- split active and inactive secrets
- generate structured outputs

Check the code

https://github.com/ja-lourenco/trufflehog-scan

Three ways a jq one-liner lies to your bash script

Tue, 27 Oct 2026 13:00:00 +0000

Most "the filter looks right but the script misbehaves" bugs come down to three silent failure modes.

First, a missing -r. jq outputs JSON by default, so a string arrives with its quotes attached — your variable holds "web-01", quotes included, and a comparison against web-01 never matches. Any value headed into a shell variable, filename, or test wants -r.

Second, missing keys don't error. jq emits null, which lands in bash as a literal four-character string — non-empty, so [ -n ] guards pass and backup-null.tar.gz gets written. The // operator supplies a real fallback.

Third, select() compares exactly. A value that is almost right matches nothing, and nothing is not an error:

jq -r '.items[] | select(.active == true) | .name'

If the API sends "active": "true" as a string, that item is skipped. If you match "Web-01" against "web-01", zero elements come back. Either way jq prints nothing and exits 0, so the silence reads as success.

All three share the same fingerprint: exit code zero, plausible output, wrong result. The fastest way I've found to catch them before they ship is building the filter against the actual response: this interactive jq filter builder with a live in-browser preview lets you click through your real JSON to build the path, handles the -r, the // fallback and select() quoting for you, and shows exactly what the filter emits before anything runs.

Limpando ~20 GB de caches de Terraform e Node.js com um único comando Bash

Mon, 26 Oct 2026 18:03:24 +0000

Mantenho todos os meus repositórios em uma única pasta — cerca de 150, entre trabalho para clientes, templates e experimentos. Cada um deles é uma pequena armadilha: um node_modules dentro de app/ e um .terraform dentro de infra/terraform/. Nenhum dos dois é versionado, ambos são reconstruídos em segundos (bem, minutos), e juntos tinham comido mais de 20 GB do meu disco sem que eu percebesse.

O gatilho foi banal: df -h mostrando 94% de uso numa partição de 233 GB, com 10 GB livres. A correção também deveria ser banal. Em vez de rodar um find ... -exec rm -rf do qual eu me arrependeria, escrevi um comando pequeno em que eu pudesse confiar — e mantive.

Este artigo traz o script, o raciocínio por trás de cada proteção e o que aprendi ao rodá-lo.

O objetivo

Um comando, de qualquer lugar, que:

Varra ~/Documentos/github (em qualquer profundidade — incluindo monorepos e pastas agrupadas).
Encontre apenas dois tipos de diretório:
- */infra/terraform/.terraform
- */app/node_modules
Informe o tamanho de cada um e os totais.
Delete esses diretórios — mas somente quando isso for pedido explicitamente.

Decisões de projeto

1. Dry-run como padrão: apagar cache de build é reversível, mas executar a operação em 30 diretórios distribuídos por 150 repositórios exige cautela. O parâmetro --delete é obrigatório para efetivar as exclusões.

2. find -prune em vez de du recursivo: evita percorrer diretórios como .git, node_modules e .terraform, mantendo a varredura eficiente mesmo com milhares de arquivos.

3. Whitelist de caminhos antes de apagar: o rm -rf só é executado em caminhos que correspondem exatamente aos sufixos permitidos e estão dentro da raiz varrida. Qualquer caminho fora dessas regras é ignorado e reportado.

4. Medição de tamanho em paralelo: executa o du em paralelo usando xargs -P "$(nproc)", aproveitando os núcleos disponíveis e reduzindo o tempo de execução em árvores grandes.

5. Saída legível, cálculos precisos: os tamanhos são somados em bytes e formatados com numfmt, evitando divergências causadas pela soma de valores arredondados.

6. Espaço livre antes e depois: compara o espaço disponível no disco antes e depois da limpeza para verificar o ganho real obtido.

Um detalhe que vale destacar: .terraform.lock.hcl é um arquivo e não é tocado. Ele fixa as versões dos providers e deve ser versionado. Apenas o diretório .terraform/ (cache de providers e módulos) é removido.

O script

Salve como limpar-cache, dê permissão de execução e coloque em /usr/local/bin:

#!/usr/bin/env bash
#
# limpar-cache — varre um workspace e levanta/remove:
#   * <projeto>/infra/terraform/.terraform
#   * <projeto>/app/node_modules
#
# Uso:
#   limpar-cache                 # só relatório (dry-run, nada é apagado)
#   limpar-cache --delete        # apaga, pedindo confirmação
#   limpar-cache --delete --yes  # apaga sem confirmação
#   limpar-cache --only node     # apenas node_modules
#   limpar-cache --only terraform
#   limpar-cache --base ~/outro/dir
#

set -uo pipefail

BASE="${HOME}/Documentos/github"
ONLY="all"
DO_DELETE=0
ASSUME_YES=0
JOBS="$(nproc 2>/dev/null || echo 4)"

usage() {
  cat <<'EOF'
limpar-cache — relatório e limpeza de caches em ~/Documentos/github

Alvos:
  */infra/terraform/.terraform
  */app/node_modules

Opções:
  -d, --delete         remove os diretórios encontrados (com confirmação)
  -y, --yes            não pede confirmação (implica --delete)
  -n, --dry-run        apenas relatório (padrão)
      --only TIPO      all (padrão) | terraform | node
      --base DIR       raiz a varrer (padrão: ~/Documentos/github)
  -h, --help           esta ajuda

Exemplos:
  limpar-cache
  limpar-cache --only terraform
  limpar-cache -d -y
EOF
}

while (($#)); do
  case "$1" in
    -d | --delete) DO_DELETE=1 ;;
    -y | --yes) ASSUME_YES=1 ;;
    -n | --dry-run) DO_DELETE=0 ;;
    --only)
      ONLY="${2:-}"
      shift
      ;;
    --base)
      BASE="${2:-}"
      shift
      ;;
    -h | --help)
      usage
      exit 0
      ;;
    *)
      printf 'limpar-cache: opção inválida: %s\n\n' "$1" >&2
      usage >&2
      exit 2
      ;;
  esac
  shift
done

case "$ONLY" in
  all | terraform | node) ;;
  *)
    printf 'limpar-cache: --only deve ser all, terraform ou node\n' >&2
    exit 2
    ;;
esac

((ASSUME_YES)) && DO_DELETE=1

if [[ ! -d "$BASE" ]]; then
  printf 'limpar-cache: diretório não encontrado: %s\n' "$BASE" >&2
  exit 1
fi

BASE="$(realpath -m -- "$BASE")"

human() { numfmt --to=iec-i --suffix=B --format='%.1f' "$1" 2>/dev/null || printf '%sB' "$1"; }

avail_bytes() { df -B1 --output=avail -- "$BASE" 2>/dev/null | tail -1 | tr -d ' '; }

# ---------------------------------------------------------------- coleta -----
# -prune evita descer em .git, node_modules e .terraform (rápido mesmo com 150 repos)
TF_DIRS=()
NODE_DIRS=()
while IFS= read -r -d '' d; do
  case "$d" in
    */infra/terraform/.terraform) TF_DIRS+=("$d") ;;
    */app/node_modules) NODE_DIRS+=("$d") ;;
  esac
done < <(find "$BASE" -mindepth 2 \( -name .git -o -name node_modules -o -name .terraform \) -prune -print0 2>/dev/null)

[[ "$ONLY" == "node" ]] && TF_DIRS=()
[[ "$ONLY" == "terraform" ]] && NODE_DIRS=()

if ((${#TF_DIRS[@]} == 0 && ${#NODE_DIRS[@]} == 0)); then
  printf 'Nada encontrado em %s (filtro: %s).\n' "$BASE" "$ONLY"
  exit 0
fi

TMP="$(mktemp)"
trap 'rm -f "$TMP"' EXIT

# tamanho aparente em paralelo: "tipo<TAB>bytes<TAB>caminho"
sizes_of() {
  local kind="$1"
  shift
  (($#)) || return 0
  printf '%s\0' "$@" |
    xargs -0 -r -n1 -P "$JOBS" bash -c '
      p="$1"
      s="$(du -sb -- "$p" 2>/dev/null | cut -f1)"
      [[ -n "$s" ]] || s=0
      printf "%s\t%s\t%s\n" "$0" "$s" "$p"' "$kind"
}

{
  sizes_of terraform "${TF_DIRS[@]+"${TF_DIRS[@]}"}"
  sizes_of node "${NODE_DIRS[@]+"${NODE_DIRS[@]}"}"
} >"$TMP"

sort -t $'\t' -k2,2nr -o "$TMP" "$TMP"

total_tf="$(awk -F'\t' '$1=="terraform"{s+=$2} END{print s+0}' "$TMP")"
total_node="$(awk -F'\t' '$1=="node"{s+=$2} END{print s+0}' "$TMP")"
total_all="$((total_tf + total_node))"
count_tf="${#TF_DIRS[@]}"
count_node="${#NODE_DIRS[@]}"

# ------------------------------------------------------------- relatório -----
printf '\nRaiz: %s\n' "$BASE"
printf '%-9s %12s  %s\n' "TIPO" "TAMANHO" "CAMINHO"
printf '%-9s %12s  %s\n' "--------" "-----------" "----------------------------------------"
while IFS=$'\t' read -r kind bytes path; do
  printf '%-9s %12s  %s\n' "$kind" "$(human "$bytes")" "${path#"$BASE"/}"
done <"$TMP"

printf '\nResumo\n'
printf '  terraform/.terraform : %3d diretório(s)  %10s\n' "$count_tf" "$(human "$total_tf")"
printf '  app/node_modules     : %3d diretório(s)  %10s\n' "$count_node" "$(human "$total_node")"
printf '  TOTAL                : %3d diretório(s)  %10s\n' "$((count_tf + count_node))" "$(human "$total_all")"
printf '  Livre em disco agora : %10s\n' "$(human "$(avail_bytes)")"

if ((DO_DELETE == 0)); then
  printf '\nModo relatório (nada foi apagado). Para remover: %s --delete\n\n' "$(basename "$0")"
  exit 0
fi

# --------------------------------------------------------------- remoção -----
if ((ASSUME_YES == 0)); then
  printf '\nRemover %d diretório(s), liberando ~%s? [s/N] ' "$((count_tf + count_node))" "$(human "$total_all")"
  read -r reply
  case "$reply" in
    s | S | y | Y | sim | SIM) ;;
    *)
      printf 'Cancelado.\n\n'
      exit 1
      ;;
  esac
fi

is_safe() {
  local p="$1" real
  [[ -n "$p" ]] || return 1
  [[ "$p" != "/" && "$p" != "$HOME" ]] || return 1
  case "$p" in
    */infra/terraform/.terraform | */app/node_modules) ;;
    *) return 1 ;;
  esac
  real="$(realpath -m -- "$p" 2>/dev/null)" || return 1
  [[ "$real" == "$BASE"/* ]] || return 1
  [[ "$real" != "$BASE" ]] || return 1
  return 0
}

avail_before="$(avail_bytes)"
removed=0
failed=0

while IFS=$'\t' read -r kind bytes path; do
  if ! is_safe "$path"; then
    printf '  IGNORADO (fora do padrão/raiz): %s\n' "$path"
    ((failed++))
    continue
  fi
  if rm -rf -- "$path" 2>/dev/null && [[ ! -e "$path" ]]; then
    printf '  removido  %10s  %s\n' "$(human "$bytes")" "${path#"$BASE"/}"
    ((removed++))
  else
    printf '  FALHOU    %10s  %s\n' "$(human "$bytes")" "${path#"$BASE"/}" >&2
    ((failed++))
  fi
done <"$TMP"

avail_after="$(avail_bytes)"
freed=$((avail_after - avail_before))
((freed < 0)) && freed=0

printf '\nConcluído: %d removido(s), %d falha(s).\n' "$removed" "$failed"
printf 'Espaço livre: %s -> %s (~%s liberados)\n\n' \
  "$(human "$avail_before")" "$(human "$avail_after")" "$(human "$freed")"

((failed == 0)) || exit 1
exit 0

Instale e deixe disponível em todo shell:

install -m 755 limpar-cache /usr/local/bin/limpar-cache

E no ~/.bashrc:

export PATH="/usr/local/bin:$PATH"
alias lcache='limpar-cache'

A linha explícita de PATH é redundante na maioria das distribuições, mas não custa nada e deixa a configuração à prova de shells de login esquisitos. (Ela já me salvou uma vez — veja a última lição.)

Rodando

Relatório primeiro, sempre:

$ lcache

Raiz: /home/user/Documentos/github
TIPO           TAMANHO  CAMINHO
--------   -----------  ----------------------------------------
node            1,4GiB  projeto-alfa/app/node_modules
node            1,1GiB  projeto-beta/app/node_modules
terraform     870,7MiB  projeto-alfa/infra/terraform/.terraform
terraform     865,2MiB  projeto-gama/infra/terraform/.terraform
node          771,2MiB  projeto-delta/app/node_modules
...

Depois, quando a lista parecer certa:

lcache --delete          # pergunta: Remover 30 diretório(s), liberando ~20,7GiB? [s/N]
lcache -d -y --only node # apenas node_modules, sem prompt

Uma execução na minha máquina:

Resumo
  terraform/.terraform :   9 diretório(s)      7,1GiB
  app/node_modules     :  21 diretório(s)     13,6GiB
  TOTAL                :  30 diretório(s)     20,7GiB
  Livre em disco agora :      9,9GiB

30 diretórios, ~20,7 GB. O disco saiu de 94% para cerca de 86% de uso — e cada um desses repositórios é reconstruível com npm ci ou terraform init.

Lições aprendidas

1. -prune é o truque inteiro. Um find ~/Documentos/github -name node_modules ingênuo desce em todo node_modules procurando outros aninhados. Em 150 repositórios isso são minutos de I/O inútil. Podar no momento do match transforma tudo em uma única varredura rápida.

2. Tamanho aparente ≠ uso de disco. du -sb dá bytes aparentes (rápido, comparável), enquanto du -s dá blocos alocados (o que o df reporta). Para um relatório de limpeza, o tamanho aparente é o número honesto e previsível; para "quanto eu vou ganhar de verdade", use --block-size=1 sem -b. Eu mostro df --output=avail antes/depois justamente porque os dois diferem.

3. O page cache frio domina o tempo de execução. A primeira execução mediu ~2m26s; a segunda foi bem mais rápida. Nada de errado com o script — são 30 árvores de diretório sendo lidas do disco pela primeira vez. Paralelismo com xargs -P ajuda, mas não vence a física.

4. Duas cópias de um script é um bug esperando para acontecer. Eu tinha instalado em ~/.local/bin primeiro e depois "movi" para /usr/local/bin — mas ~/.local/bin vem antes no PATH, então a cópia antiga sombreava a nova silenciosamente. Correção: mover de verdade (install + rm), não copiar. E conferir com command -v, não com which (que é outro binário, com outra semântica).

5. Proteções são baratas; erros de rm -rf não. A função is_safe() tem ~12 linhas e rejeita /, $HOME, qualquer coisa fora da raiz varrida e qualquer caminho que não termine nos dois sufixos esperados. Ela roda em cada caminho, mesmo os vindos do meu próprio find. Essa redundância é o ponto: o custo é microssegundos, o modo de falha é "apaguei meu diretório home".

6. Whitelist ganha de blacklist. Eu deliberadamente não apago dist/, .next/, .venv/ ou coverage/. Esses têm custos de reconstrução muito diferentes e alguns são artefatos de deploy. Se eu precisar deles depois, cada um ganha seu sufixo explícito no case — nunca uma regra de "apague tudo que pareça cache".

Checklist de recuperação

Depois de uma limpeza dessas, cada repositório precisa de uma coisa:

# Projetos Node
cd app && npm ci          # ou pnpm install / yarn install

# Projetos Terraform
cd infra/terraform && terraform init

O terraform init baixa os providers novamente; se você usa cache de plugins (TF_PLUGIN_CACHE_DIR), será bem mais rápido da próxima vez — vale configurar se costuma limpar .terraform com frequência.

Considerações finais

O script tem cerca de 200 linhas, a maioria delas printf. O que o torna seguro não é sofisticação, e sim um punhado de escolhas entediantes: dry-run primeiro, podar cedo, whitelist de caminhos, confirmar antes de apagar e imprimir o que realmente aconteceu.

Vinte gigabytes não é muito em termos absolutos, mas era 60% do espaço livre que eu tinha. Mais importante: o comando agora existe. Na próxima vez que o disco apertar, é uma linha em vez de uma tarde de du e dúvidas.

Se você mantém muitos repositórios num só lugar, o padrão se generaliza — troque os dois sufixos pelos seus próprios artefatos de build, mantenha o dry-run como padrão e nunca deixe o script adivinhar o que deve apagar.

Why grep "error|failed" finds nothing in a log full of failures

Thu, 22 Oct 2026 13:00:00 +0000

If grep returns empty on a file you know contains matches, suspect the pattern before the file. Default grep uses Basic Regular Expressions (BRE), where |, +, ?, and () are literal characters unless backslash-escaped. So grep "error|failed" app.log searches for the literal twelve-character string error|failed — pipe included — which no logger ever writes. Alternation needs Extended Regular Expressions via -E:

grep "error|failed" app.log     # BRE: literal string, matches nothing
grep -E "error|failed" app.log  # ERE: matches either word

The dangerous part is the exit behavior. An impossible pattern is not a syntax error, so grep exits 1 — the same code as "no matches in a clean file" — and prints nothing. A wrong question and a healthy log look identical, which is how this bug hides inside monitoring checks and cron pipelines for days.

Two related traps: -P (Perl regex, \d, lookaheads) is GNU-only and fails on macOS/BSD grep, and bare grep aimed at a directory refuses with "Is a directory" unless -r is set.

When I'm unsure what a pattern will match, I assemble the command in this grep command builder with a live match tester — paste a sample line you know should match, and a zero count tells you the pattern is the bug before the empty result gets believed.

bash ran a path that no longer existed, and exited 127

Wed, 21 Oct 2026 13:00:00 +0000

A demo script called report, two directories on PATH, and one mv between them, all on a scratch tmpfs:

$ report
report 2.1: 14 jobs ok
exit=0
$ mv /opt/jobs/bin/report /opt/jobs/newbin/report
$ ls -l /opt/jobs/newbin/report
-rwxrwxr-x 1 root root 42 Oct  3 18:26 /opt/jobs/newbin/report
$ report
session.sh: line 5: /opt/jobs/bin/report: No such file or directory
exit=127
$ type -a report
report is /opt/jobs/newbin/report
$ hash -t report
/opt/jobs/bin/report
$ command -v report
/opt/jobs/bin/report

The file is there, on PATH, and bash ran the old path anyway. The first time it finds a command through a PATH search, it stores the full path in a hash table and skips the search after that. When the file moves, it executes the remembered path and reports No such file or directory with exit 127, the same code a typo gets.

The confusing part is the end of that output. type -a searches PATH fresh and finds the new location. command -v reports what bash would run, which is the stale one. hash -r clears the table, a new terminal starts empty, and shopt -s checkhash makes bash re-search when a hashed path has gone: with it set, the same sequence printed the tool's output and exit 0.

The stale hash is one of six causes behind command not found. The others, from a cron PATH to a CRLF shebang, each reproduced with a script that names the cause, are in command not found in bash, six causes.

rsync's dangerous flags don't fail. They do a different job and exit 0.

Tue, 20 Oct 2026 13:00:00 +0000

The trap with rsync is that its dangerous flag combinations don't fail — they complete a different operation than the one you intended, and exit 0. --delete makes the destination match the source, so a mistyped or freshly moved source directory means rsync removes everything at the destination and reports success. Your cron log shows a clean run while the backup empties itself. A trailing slash changes the meaning too: /data/ syncs the directory's contents, while /data copies the directory itself into the destination — and on a mirror job, that difference decides what gets compared and what gets deleted.

The defense is previewing every destructive sync before running it live:

rsync -av --delete --dry-run /srv/www/ user@backup:/srv/backup/www/

Read the deleting … lines it prints; only when that list matches your expectations do you remove --dry-run. Note also that excludes take one pattern per flag — --exclude='*.log' --exclude='.git' — because a comma-separated list inside a single flag silently matches nothing.

If you'd rather compose the command with those guardrails enforced — a red warning when --delete is enabled without a dry run, per-pattern quoted excludes, and a live preview of the finished command — I built a visual rsync command builder with delete-flag guardrails that runs entirely in the browser.

One Bash Script to Audit a Polyglot Codebase (Python, TS, Java, Go, C, HTML)

Mon, 19 Oct 2026 11:43:05 +0000

If your project mixes languages, one terminal command can run each language's own tests and security checks and fail the build when anything is red.

The script (1) counts files per language, (2) runs build and type checks, (3) runs tests and SAST, (4) scans dependencies and secrets, (5) runs a quick concurrency check, and (6) writes a checksummed report.

I tested it on a small demo project: python compile passed, missing scanners showed as SKIP, an invalid HTML file produced a real FAIL, and the run exited with code 1.

Full guide with per-language settings, a troubleshooting table and a CI example: https://www.seosiri.com/2026/10/polyglot-code-audit-bash-testing-pipeline.html#conclusion

Which scanner would you add first?

Every disk-alert script I had piped into a command this machine does not have

Thu, 15 Oct 2026 13:00:00 +0000

The standard bash alerting pattern ends the same way everywhere you read it: pipe the finding into mail. So I checked what my own box would actually do with that line.

$ type -a mail mailx sendmail
mail not found
mailx not found
sendmail not found

Nothing. No MTA, no mailutils, no fallback. Any alert script written the usual way would have run its check correctly, found the problem correctly, piped the message into a command that does not exist, and told me nothing.

This fails quieter than it sounds. Under set -euo pipefail you at least get a non-zero exit somewhere a cron wrapper might notice. Without it, the pipeline's exit status is the last command's, cron mails you nothing because cron's mail also needs an MTA, and the script looks like it ran fine.

Check for the binary before you rely on it, not after an incident. And on a box that does have mail, exit 0 still only means the message reached a local MTA — on a host with no outbound relay it lands in a spool file under /var/mail that nobody reads. Confirm real delivery once by tailing /var/log/mail.log. Cron's minimal PATH is a second reason to test from inside cron rather than your login shell.

The msmtp relay config and the rate-limiting guard are in the bash email alert walkthrough.

One empty variable, one error, one wrong answer, exit 0

Wed, 14 Oct 2026 13:00:00 +0000

I wrote four lines to capture a bash error for a guide. One empty variable, tested twice:

$ cat queue-check.sh
#!/bin/bash
count=""
if [ $count -eq 0 ]; then echo "queue empty"; fi
if [[ $count -eq 0 ]]; then echo "queue empty ([[ ]])"; fi

$ bash queue-check.sh
queue-check.sh: line 3: [: -eq: unary operator expected
queue empty ([[ ]])
exit=0

The message blames -eq. The bug is the unquoted $count before it: word splitting removed the empty value, so [ received -eq 0 and failed with status 2, which an if treats as "not true". The [[ ]] line does no splitting, read the empty string as 0 in an arithmetic test, and printed a confident wrong answer. Then the script exited 0.

Strict mode does not help here. count="" is set, so set -u stays quiet, and set -e ignores failures inside an if condition. What stopped it was validating the value first: [[ $count =~ ^[0-9]+$ ]] || exit 1.

Most bash errors work this way. The message names the check that failed, not the cause: a syntax error reported on line 4 for a missing semicolon on line 2, Permission denied after chmod +x on a noexec mount. Nine of them, each reproduced with the check that names the cause, are in Bash Error Messages, Decoded.

I escaped the dot correctly and sed still rewrote a file marked DO NOT CHANGE

Tue, 13 Oct 2026 13:00:00 +0000

Four config files, one hostname to rename. I knew the dot in api.internal is a regex wildcard, so I escaped it. grep -rn 'api.internal' unescaped matched a fourth file containing apiXinternal — a typo somebody deliberately kept. Escaped, api\.internal matched three. Good.

Then I ran the rename, and two of those three should never have changed: a line ending in # DO NOT CHANGE, and a commented-out sample. My -not -path "*/samples/*" exclusion did nothing, because the file was called sample.conf and was not inside a samples/ directory.

That is the real lesson, and escaping was never the whole answer. The dot is the trap everyone warns about; the trap that actually got me was assuming my exclusion pattern matched what I pictured. The fix is an order, not a flag — make grep -l print the file list while everything is still read-only, and read it:

find /etc/myapp -type f -name "*.conf" -exec grep -l "api\.internal" {} +

If that list surprises you, the surprise is the bug, and you found it before anything was written. Only then pipe it to sed -i.bak, so every file keeps a rollback copy — and diff one against its .bak before trusting the run. sed -i exits 0 whether it changed four files or none.

The awk summarizing stage and the incident-triage pipeline are in the find, grep, sed and awk ordering guide.

My 30-day log cleanup kept a file that was 30 and a half days old

Thu, 08 Oct 2026 13:00:00 +0000

I set three log files to known ages with touch -d — exactly 30 days, 30 days and 12 hours, and 31 days — then ran what I thought was a 30-day retention policy:

find . -name '*.log' -mtime +30

It returned one file: the 31-day-old one. The 30.5-day file stayed. So did the 30-day file.

-mtime counts whole 24-hour periods and throws away the remainder, and +30 means strictly greater than 30 of them. A file aged 30 days and 12 hours counts as 30, and 30 is not greater than 30. A real 30-day policy is -mtime +29. Re-running with +29 returned all three, which is what I wanted in the first place.

Two more things that bit during the same session. Piping to xargs rm split app v2.log on the space — rm reported cannot remove 'ws/app' and cannot remove 'v2.log', and the file survived. And -delete is an action that returns true, so putting it before your filter is catastrophic: find ord -delete -name '*.log' removed a file called keep.txt and then the directory itself. Preview with -print, keep -delete last.

The multi-directory loop, the .gz variant, and the cron lines are in the find -mtime log retention walkthrough.

Catch a filling disk weeks before it becomes an outage

Tue, 06 Oct 2026 13:00:00 +0000

A disk almost never fails loudly. It fills a percent at a time until writes stop, and at 100% nothing reports "disk full" — the database throws write errors about its own files, apps crash-loop, and logging (a write itself) stops recording at the exact moment you need evidence. You end up debugging five fake problems instead of one real one.

The prevention costs one line on a daily cron:

df -h | awk 'NR>1 && $5+0 >= 80 {print $0}'

NR>1 skips the header. $5+0 is the trick: df's Use% column is a string like 82%, and awk's numeric coercion keeps the leading digits and drops the %, so the threshold comparison works without cut or sed. Pipe any output to mail -s and you get an email the day a filesystem crosses 80.

Why 80? At 80% a disk growing one percent a day leaves you weeks to investigate with du -ah | sort -rh | head -20. At 95% you're deleting files under pressure; at 100% every service on the box misbehaves at once.

This check is #1 in a set of 25 cron-ready bash scripts for sysadmins covering disk monitoring, backups, service watchdogs, SSL expiry, and port audits.

sed 's/port/listen_port/g' rewrote four lines of a four-line config

Thu, 01 Oct 2026 13:00:00 +0000

A four-line YAML file (port: 8080, support_email, transport: tcp, report_dir: /var/reports) and the rename most people type first: sed -i 's/port/listen_port/g' app.yml. GNU sed 4.9 exited 0 and printed nothing, because -i never does. Afterwards the file held suplisten_port_email, translisten_port and /var/relisten_ports. One line was meant to change. All four did.

sed matches characters, not words. The letters p-o-r-t sit inside support, transport and report, and g replaced every occurrence. GNU sed's \b word boundary fixes the match, and one pipe shows the damage before anything is written:

sed 's/\bport\b/listen_port/g' app.yml | diff app.yml -

Without -i the file on disk stays intact, and diff prints the lines that would change. It exits 1 when it finds differences, so the same check works inside a script. Only after the diff reads right does -i go back on.

Two caveats before trusting \b. Underscore counts as a word character, so db_port stays untouched. And \b is a GNU extension: BSD sed on macOS spells word boundaries [[:<:]] and [[:>:]], and its -i requires a suffix argument (sed -i '') where GNU's is optional.

The replacement side bites too. An unescaped & means the whole match, so s/CEO/Smith & Sons/ produces Smith CEO Sons.

The bulk-replace script, with a grep -rl file list, per-file diff and an --apply gate that fails toward reading, is on the sed find-and-replace snippet page.

ShellCheck rates SC2086 as info. It let rm delete two files I never named.

Tue, 29 Sep 2026 13:00:00 +0000

SC2086 sits at info, ShellCheck's lowest default severity, and fires on every unquoted $var, so it is usually the first rule anyone disables. I built the smallest test I could: a directory holding quarterly, report.txt and quarterly report.txt, and a four-line script under set -euo pipefail:

report="quarterly report.txt"
rm $report

It deleted quarterly. It deleted report.txt. It left the one file it was asked to remove, and exited 0. rm was handed two names and both existed, so strict mode had nothing to catch.

Bash expands the unquoted variable, splits the result on whitespace, globs any * or ? against the current directory, and only then builds the argument list. rm "$report" is one argument with no globbing, and it deletes the right file.

The same bug hides in other shapes. [ $name = "root" ] with an empty $name gives test two arguments: a complaint on stderr, a false condition, and the script keeps walking. for f in $files is not flagged by ShellCheck 0.11.0 at all, and still breaks on the first name with a space.

When you do want the split, such as a flag string read from a config, use an array or a one-line # shellcheck disable=SC2086 with the reason. A repo-wide .shellcheckrc disable covers every future rm as well. And shellcheck --include=SC2086 runs it alone as a CI gate.

Every run, pasted from ShellCheck 0.11.0, is in the SC2086 deep dive on word splitting and globbing.

I sent my report script SIGTERM mid-write. Yesterday's output survived intact.

Thu, 24 Sep 2026 13:00:00 +0000

A script that writes straight to the path a consumer reads can die halfway and leave a torn file that everything downstream loads as complete. No error appears anywhere, because the only process that failed is gone. I tested the fix the unfriendly way on bash 5.3.9: a CSV generator with a sleep 30 planted between rows, killed with SIGTERM one second in.

TMP_FILE=$(mktemp)
cleanup() { local code=$?; rm -f "$TMP_FILE"; exit "$code"; }
trap cleanup EXIT
generate_rows > "$TMP_FILE"
mv "$TMP_FILE" "$FINAL_PATH"

Exit 143, which is 128 plus signal 15. The temp file was gone, and out.csv still held the previous run's three complete lines. Stale is something monitoring can see. Torn is not.

EXIT fires on normal completion, an explicit exit 1, a set -e abort and after a handled signal, which is why this is five lines instead of twenty. And because mv on one filesystem is atomic, being half-written and being at the published path are never true at once. Even kill -9, which no trap catches, cannot publish a torn file.

The ordering is the part people move. I ran four handler variants against a script ending in exit 3, and only one exited 0: rm -f first, then local code=$?. It captured rm's success and laundered the failure. Capture first, clean second, exit last.

The complete script, subshell scoping rules and a line-by-line breakdown are on the trap cleanup page for temp files and atomic writes.

My ERR trap stayed silent until I added one letter to set -euo pipefail

Tue, 22 Sep 2026 13:00:00 +0000

A nine-line script, set -euo pipefail on line 2 and an ERR trap on line 3 whose only job is to print the failing command and its line number. A helper function then runs mkdir /proc/export, a directory that cannot be created. On bash 5.3.9 it printed mkdir's own complaint and exited 1. The trap said nothing: no command, no line, the two facts it was registered to report.

By default the ERR trap is not inherited by functions, command substitutions or subshells. Errexit killed the script from inside the function; the trap, registered at the top level, never saw the failure. One letter fixes it:

set -Eeuo pipefail
trap 'echo "✗ failed: ${BASH_COMMAND} (line ${LINENO})" >&2' ERR

-E (errtrace) makes the trap follow the script into functions. Same run, same mkdir error, now followed by ✗ failed: mkdir /proc/export (line 5).

Two more silent spots from the same session. local out=$(false) inside a function carried on to the next line and the script exited 0, because local succeeded and its status is the one bash checks. Declare on one line, assign on the next. And in an EXIT handler, capture $? on the first line: moved below an rm -f, it reports rm's zero, and a script ending in exit 3 exits 0.

Where errexit goes silent, nounset defaults and the assembled skeleton with both traps are in the strict-mode bash script template, checked against bash 5.3.

Who owns that port? One ss flag answers it without sudo

Wed, 16 Sep 2026 17:29:08 +0000

Who owns that port? One ss flag answers it without sudo

For months my dev server started on 3001 because something already held 3000, and I kept stepping around it. When I finally asked, ss -ltn 'sport = :3000' showed the port listening on every interface — with an empty Process column, because ss -p only names sockets owned by your own user. Without root, it looked like nobody owned it.

One flag closes most of that gap:

ss -ltnpe 'sport = :3000'

-e prints the socket's cgroup, and on a systemd machine the end of that path is the unit that opened it. Mine ended in docker.service. docker ps then named it: a docker-proxy publishing a container I had started in the spring and forgotten, reachable from the whole network.

Two things worth carrying away. The host only ever sees Docker's proxy, never the application behind it, so a host-side port list tells you what is exposed, not what is running. And "list open ports" is really three questions: what is listening, who owns it, and whether it is reachable. They need three different commands.

The full write-up covers all three — with and without root, without netstat, inside containers, and an audit script that diffs the listener list between runs: how to list open ports on Linux with the owning process.

Your Cron Job Works. That's Not the Same as Surviving Cron.

Wed, 08 Jul 2026 23:44:51 +0000

Level: Intermediate · Time: 7 min · Outcome: A mental model and three guards for unattended cron jobs

There's a gap between a script that works when you run it and a script that survives running unattended for a year. The first only has to succeed once, with you watching, on a good day. The second has to keep working through slow disks, hung sockets, recovering services, and reboots — with nobody watching, because the entire point of cron is that nobody is watching.

Almost every script I've seen take down a server was fine in a manual run and fell apart the first time conditions weren't ideal and there was no human there to notice. I've hit three versions of this personally: a sync that stacked copies until the box hit load 41, a backup that hung and didn't run for nine days without a peep, and a deploy that raced a database's boot and paged me at 11pm. Three commands, three failures, one underlying truth — "works on my machine" and "survives cron" are different engineering problems.

Cron jobs die quietly in exactly three ways, and there's a fourth problem underneath all of them that keeps the first three invisible.

The invisible problem first: cron eats your output

When cron runs a job, anything it prints to stdout or stderr goes nowhere unless you've configured otherwise. No log, no record, no trace. So a job can fail every night for a month and the only signal is the absence of whatever it was supposed to produce — and you notice that when something downstream breaks, not when the job fails.

Two things turn cron from silent to legible. MAILTO at the top of the crontab mails you any output a job produces. More reliably, redirect every job's output to a log file you control:

*/5 * * * * /usr/local/bin/safe-sync.sh >> /var/log/sync.log 2>&1

The 2>&1 is the part people forget — it folds stderr into the same stream as stdout, so your errors land in the log instead of evaporating. Everything below assumes you've done this. A locked, time-bounded, retried job that still logs nothing is a job whose failures you'll discover by accident.

Overlap: lock it to one instance

This is the failure that turns a transient slow patch into a self-inflicted outage. A sync scheduled every minute normally finishes in twenty seconds; one slow-disk afternoon it starts taking ninety, and cron launches a new copy every minute on top of the last, until a half-dozen copies are fighting over the same disk and the load climbs into the dozens.

The guard is flock: a kernel-held lock on an open file descriptor that lets exactly one copy run and makes the rest skip. Unlike a PID file, there's nothing to clean up — the kernel releases the lock automatically when the process exits, crashes, or is killed, so you never inherit a stale lock from a run that died badly. But a lock has a sharp edge: if the locked job hangs, it holds the lock forever and every future run skips. The job stops running entirely, silently. Which is the next problem.

Hang: bound the runtime

A hung command is worse than a failed one, because a failed command at least exits and frees its lock. A backup wedged on a held database lock, a curl against a dead socket, an ssh into a black hole — these never return. Under cron, "never returns" means the slot is jammed, the lock is held, and the job produces zero signal because it never gets far enough to log anything.

timeout is the outside bound the command can't set on itself:

# SIGTERM at 5 minutes; if it's ignored (process stuck in I/O), SIGKILL 20s later
timeout -k 20s 5m mysqldump --single-transaction mydb > /backup/mydb.sql.partial

It exits 124 when it had to step in and 137 when the command ignored the polite SIGTERM and had to be force-killed — codes worth branching on, because "slow" and "wedged" are different problems. The -k grace matters specifically for processes stuck in uninterruptible I/O, which can't act on SIGTERM at all. Timeout is what makes a lock safe: with both in place, a hang gets killed on a deadline, the lock always releases on time, and the failure becomes a loud 124 instead of a silent gap.

Transient failure: retry with backoff

Not every failure deserves to kill the run. A fresh database container that isn't accepting connections for six seconds, a 503 during a rolling restart, a 429 explicitly asking you to back off — these are transient. Treating the first one as fatal is how a normal six-second boot delay becomes a failed deploy at eleven at night.

The guard is a bounded retry with exponential backoff: try the command, and on failure wait a delay that doubles each round, with jitter so parallel callers don't retry in lockstep, capped so a genuinely dead dependency fails fast instead of looping forever.

# Wait for the port to be ready before doing the work that depends on it
retry 6 nc -z -w 2 db.internal 5432

The discipline that makes retries safe rather than dangerous: retry transient failures only. A 404 fails identically on every attempt, so retrying it just delays the real error and buries it under retry noise.

Composing all three — and knowing which job needs which

Locking, bounding, and retrying are independent guards, and the real value is stacking them: a single run that can't overlap, can't hang, and rides out a blip — and tells you, by mail or log, when it finally does give up. That composition is mechanical enough to generate, which is exactly what the Hardened Cron Wrapper Generator does: paste the command, toggle the guards, get a ShellCheck-clean wrapper and the crontab line.

But not every job needs all of it — wrapping a one-line date echo in flock and timeout is ceremony. Add a lock to anything that mutates shared state and can outrun its interval. Add a timeout to anything that touches the network or a database. Add retries to anything that depends on something which boots or recovers on its own clock. Log everything, always. The full decision table — guard by guard, when to add it and when to skip it — is in the guide.

Read the whole thing, including the "know when it broke" alerting layer and the decision matrix: https://bashsnippets.xyz/guides/bash-scripts-that-survive-cron

The three guards as standalone snippets: flock, timeout, retry with backoff. To write a new hardened script from a blank slate rather than wrap an existing command, the Bash Boilerplate Generator and the error-handling snippet cover the skeleton. The rest of the library is at https://bashsnippets.xyz

Originally published at bashsnippets.xyz

Three Lines of Bash Stand Between Your Cron Job and a Silent Outage

Sun, 05 Jul 2026 17:40:41 +0000

Three Lines of Bash Stand Between Your Cron Job and a Silent Outage

Level: Intermediate · Time: 6 min · Outcome: A hardened, ShellCheck-clean cron wrapper generated for you

Three separate incidents taught me the same lesson over about a year. A */1 rsync stacked six copies on a slow disk and drove a box to load 41. A mysqldump hung on a database lock and didn't run for nine days without a single alert. A deploy raced a database's boot and paged me at 11pm for a migration that wasn't broken. Different commands, different failures — but the same root shape every time: a script that was fine when I ran it by hand fell over the first time conditions weren't ideal and nobody was watching.

The fixes are individually small. A cron job that can't take down a server needs three guards: a lock so it can't overlap itself, a timeout so it can't hang forever, and a bounded retry so a transient blip doesn't kill the run. The problem isn't knowing that. It's that composing the three correctly by hand — getting the flock file descriptor right, the timeout SIGKILL grace right, the retry quoting right, and not mangling a pipeline in the process — is exactly the kind of fiddly that produces a wrapper that looks right and silently isn't.

So I built a generator that does the composition: the Hardened Cron Wrapper Generator.

What it actually does

You paste in the command your cron job runs, toggle the guards you want, set the schedule, and it emits two things: a complete wrapper script and the crontab line that calls it. The generated bash follows a strict standard — set -euo pipefail, CHECK/CROSS defined before use, comments that explain why each line is there — and it's ShellCheck-clean, which is the part that matters when you're pasting something into a root crontab.

The toggles map exactly to the three failure modes:

Lock (flock) stops overlap. You pick the behavior: Skip if busy (-n) for frequent jobs where a missed run is harmless because the next one catches up, or Wait then give up (-w) for jobs that must eventually run but can tolerate a short queue. The tool puts the lock file under /run/lock — a tmpfs cleared on reboot — not /tmp, where temp-cleaners can delete a lock mid-run and let a second copy through. That single path choice is a bug people ship constantly.

Bound the runtime (timeout) stops hangs. You set the max runtime and a SIGKILL grace period. The grace exists because timeout first sends a polite SIGTERM, and a process wedged in uninterruptible I/O — a dead NFS mount, a held database lock — physically can't act on SIGTERM. The grace is how long to wait before sending SIGKILL, which the kernel enforces unconditionally. The tool sets a sane default; you tune it for jobs you know touch slow storage.

Retry transient failures rides out the blip. You set max attempts and a base delay, and the backoff doubles each round up to a 30-second cap, with jitter so parallel callers don't retry in lockstep and re-overwhelm a recovering service.

The part that makes the output trustworthy

Two design decisions in the tool are worth calling out because they're the ones I'd get wrong by hand at 11pm.

First, the crontab line changes shape depending on what you enable. Lock and timeout are simple enough to inline directly on the crontab line — flock and timeout in front of the command, a one-liner. But retry and email-on-failure need a bash function and a mailx call, which can't live on a crontab line. So when you turn either of those on, the output switches from an inline crontab one-liner to a full wrapper script plus a crontab line that calls it. The tool makes that decision for you instead of generating something that won't run.

Second, it's explicit about pipelines. timeout and the retry loop bound the single process they launch — so a raw cmd1 | cmd2 would only be partly guarded. The tool surfaces this: for a pipeline or multiple commands, you wrap them in bash -c '...' yourself, and it tells you so, rather than silently generating a wrapper that bounds half your command.

# A pipeline must be wrapped so timeout/retry bound the whole thing, not just curl
bash -c 'curl -fsS https://api.example.com | jq .status'

It also handles the thing underneath all three guards that people forget: logging. Cron throws stdout and stderr away by default, so a job can fail every night for a month and the only signal is the absence of whatever it was supposed to produce. The tool's timestamped-logging toggle redirects output to a file you control, and the email-alert toggle sends a mailx failure notice after all retries are exhausted — so the job fails loudly instead of disappearing.

Why generate it instead of memorizing it

Because the failure mode of hand-writing these is that the wrapper looks correct and isn't. The flock that used /tmp. The timeout with no -k that left the wedged process alive. The retry loop that retried a 404 forever. Each of those is a one-character or one-decision mistake, and each one only shows up the night it matters. A generator that bakes the right defaults in — and explains them inline in the output so you can read what you're shipping — removes that whole class of error.

Build your wrapper here: https://bashsnippets.xyz/tools/cron-wrapper-generator

The three snippets it composes, if you want the full reasoning behind each guard, are flock, timeout, and retry with backoff. Build the schedule itself in the Cron Job Builder, and read the whole decision in Bash Scripts That Survive Cron. The rest of the library is at https://bashsnippets.xyz

Originally published at bashsnippets.xyz

I Spent 40 Minutes at 11pm Debugging a Deploy That Wasn't Broken

Thu, 02 Jul 2026 03:27:52 +0000

Level: Intermediate · Time: 6 min · Outcome: A deploy that waits for its dependencies instead of racing them

I once spent forty minutes at eleven at night debugging a deploy that wasn't broken. The release script ran the database migration, the migration threw connection refused, the script exited non-zero, the deploy rolled itself back, and I got paged.

So I did the things you do. I read the migration. I read the logs. I checked the database — it was up, it was healthy, it accepted my connection instantly. I re-ran the deploy and it worked. I chalked it up to gremlins and went to bed, which is the part I'm not proud of, because it happened again two days later. That time I watched the timing: the script brought up a fresh database container and started the migration about six seconds before Postgres finished initializing and began accepting connections. The migration was racing the database's boot. Most of the time it won. The times it lost, I lost forty minutes.

The script wasn't wrong about anything except one assumption: that a dependency is ready the instant you ask for it.

In production, dependencies are eventually ready

That's the mental model shift. Networks blip. A service you call returns a 503 for the two seconds it takes to finish a rolling restart. An API rate-limits you with a 429 it fully expects you to retry. A fresh container's database isn't accepting connections for its first few seconds. Treating the first failure as fatal turns every one of these normal, transient conditions into a paged engineer — and the script that handles them isn't smarter — it declines to give up on the first try.

But retrying naively is its own trap. Retry instantly and you hammer a recovering service into staying down. Retry forever and a genuinely dead dependency hangs your script indefinitely. Retry a 404 and you wait a minute to confirm what you already knew. Good retries are bounded, backed off, and selective.

A retry function you can reuse anywhere

#!/bin/bash
# Purpose: survive transient failures instead of dying on the first error
set -euo pipefail

CHECK="✓"
CROSS="✗"

# retry <max_attempts> <command> [args...]
retry() {
    local max_attempts="$1"; shift
    local attempt=1
    local delay=1            # base delay — doubles each round
    local max_delay=30       # cap so the backoff never runs away

    until "$@"; do
        if (( attempt >= max_attempts )); then
            echo "$CROSS '$*' failed after $attempt attempts" >&2
            return 1
        fi
        # 0–2s of jitter so parallel callers don't all retry on the same beat
        local pause=$(( delay + RANDOM % 3 ))
        echo "$CROSS attempt $attempt failed — retrying in ${pause}s" >&2
        sleep "$pause"
        attempt=$(( attempt + 1 ))
        delay=$(( delay * 2 ))
        (( delay > max_delay )) && delay=$max_delay
    done

    echo "$CHECK '$*' succeeded on attempt $attempt"
}

# The actual fix for my 11pm deploy: wait for Postgres to accept connections.
retry 6 nc -z -w 2 db.internal 5432
echo "$CHECK database reachable — running migration"

The whole engine is until "$@"; do ... done. until runs the command and executes the loop body only when it fails, exiting the instant it succeeds. Passing the command as "$@" (after shift-ing past the attempt count) means the function retries anything — a curl, an ssh, a port check, your own script — without caring what it is.

The backoff is the three lines at the bottom of the loop: sleep for the current delay plus a little jitter, then double the delay, capped at max_delay. That gives you 1s, 2s, 4s, 8s, 16s, 30s, 30s…

The jitter is not decoration

RANDOM % 3 looks trivial, and it's the line people delete to "clean up." Don't. Without jitter, a fleet of machines that all failed at the same instant — because the same service went down — will all retry at the same instant, and the same instant after that, producing a synchronized thundering herd that knocks the recovering service straight back over on every round. A few hundred milliseconds of randomness per client spreads the retries out so the service actually gets room to recover. At one machine it does nothing; at fifty it's the difference between recovery and a retry storm.

The mistake that makes retries dangerous

# Good: a transient failure that retrying can fix
retry 6 nc -z -w 2 db.internal 5432

# Bad: retrying a deterministic failure just delays the error 30 seconds
retry 6 curl -fsS https://api.example.com/v1/thing-that-returns-404

A retry loop is only as smart as what you point it at. A port check belongs in a loop because the answer changes — "no" until the database boots, then "yes." A request that returns 404 returns 404 on attempt one and attempt six; the loop just postpones the failure and buries the real status under retry noise. Retry transient failures — timeouts, connection-refused, 429, 5xx, DNS hiccups. Don't retry deterministic ones — a 404, a 401, a syntax error, a missing file. When you can, branch on the exit code or HTTP status and loop only on the codes worth looping on.

For plain curl, its built-in --retry 5 --retry-delay 2 does most of this and is simpler; reach for the function when the thing you're retrying isn't curl, or when you want one backoff policy across a database probe, an ssh call, and a download at once.

Back to 11pm

That deploy never paged me again once the migration waited for the port instead of assuming it. The database still took its six seconds to boot, the network still blipped occasionally — retrying didn't make the dependencies faster. It stopped a normal, transient slowness from being treated as a fatal error, which is most of what "production-ready" means for a script.

Full function with the wait-for-port pattern and the guidance on which failures to retry: https://bashsnippets.xyz/snippets/bash-retry-with-backoff

Retries are the third leg of an unattended job: flock stops overlap, timeout stops hangs, retry rides out the blip. The Hardened Cron Wrapper Generator wires all three into one wrapper, Bash Scripts That Survive Cron is the end-to-end version, and the rest of the library is at https://bashsnippets.xyz

Originally published at bashsnippets.xyz

# My Backup Hadn't Run in 9 Days and Nothing Told Me

Fri, 26 Jun 2026 03:41:25 +0000

My Backup Hadn't Run in 9 Days and Nothing Told Me

Level: Intermediate · Time: 5 min · Outcome: A cron command that can't hang forever

The backup ran fine every night for fourteen months, and then it didn't run for nine days, and nothing told me. No error in the log. No failed-job alert. No bounced cron mail. The nightly mysqldump had hung — the database had a long-held lock from a runaway analytics query, the dump opened its transaction and sat there waiting for it, forever.

Cron launched it at 2am, it never exited, and here's the cruel part: because I'd been smart enough to wrap it in a lock so two dumps couldn't run at once, every subsequent night's run saw the lock still held by the zombie from the 9th and skipped quietly. The clever lock turned a one-night hang into a nine-day outage. I found it when I went to restore a table and discovered my newest "backup" was a mysqldump process that had been running since the previous Tuesday.

That's nine days I'd have lost if anything had gone wrong with the live database. The ten minutes of feeling foolish when I traced it back was nothing next to that.

A hung command is worse than a failed one

This is the lesson worth internalizing, because it's counterintuitive. A failed command exits, frees its lock, and the next run tries again — the system self-heals. A hung command exits never. It holds resources, blocks its own future runs, and produces exactly zero signal because it never gets far enough to log anything. Failures are loud. Hangs are silent, and silence is what kills you in unattended automation.

You can't rely on a command to bound its own runtime, either. The whole problem is that it's wedged somewhere it can't time itself out of — blocked in the kernel waiting on a lock, or on a dead socket that will never send the FIN it's waiting for. So you bound it from the outside.

Bounding the runtime with timeout

#!/bin/bash
# Script: bounded-dump.sh
# Purpose: Stop a hung command from running forever and jamming the cron slot
set -euo pipefail

CHECK="✓"
CROSS="✗"

MAX_RUNTIME="5m"   # longer than the normal worst case, well under the interval
KILL_GRACE="20s"   # after SIGTERM, wait this long, then SIGKILL
DEST="/backup/mydb.sql"

# In an `if` so set -e doesn't abort before we read the exit code.
# Write to a .partial file so a timed-out run never leaves a corrupt "backup".
if timeout -k "$KILL_GRACE" "$MAX_RUNTIME" \
        mysqldump --single-transaction mydb > "$DEST.partial"; then
    mv "$DEST.partial" "$DEST"
    echo "$CHECK dump completed within $MAX_RUNTIME"
else
    code=$?
    rm -f "$DEST.partial"
    case "$code" in
        124) echo "$CROSS dump exceeded $MAX_RUNTIME — terminated (SIGTERM)" >&2 ;;
        137) echo "$CROSS dump ignored SIGTERM — force-killed (SIGKILL)" >&2 ;;
        *)   echo "$CROSS dump failed with exit code $code" >&2 ;;
    esac
    exit "$code"
fi

timeout -k "$KILL_GRACE" "$MAX_RUNTIME" mysqldump ... is the entire mechanism. At five minutes, timeout sends the dump a SIGTERM. A well-behaved program treats SIGTERM as "wrap up and exit." But the dump from my outage wasn't misbehaving — it was blocked in the kernel waiting on a lock, and a process in that state physically cannot act on SIGTERM. That's what -k 20s is for: twenty seconds after the polite signal, timeout sends SIGKILL, which the kernel enforces unconditionally. Nothing survives SIGKILL.

Read the exit code — it's the difference between a log and a mystery

124  # still running at the deadline — SIGTERM was sent
137  # 128 + 9 — ignored SIGTERM, had to be force-killed

Collapsing every outcome into "backup failed" throws away the one piece of information that tells you whether you have a slow database, a wedged one, or a broken dump command. A 124 says "this is taking too long — investigate the query." A 137 says "this is wedged in I/O — investigate the lock or the mount." They point at different problems. (If you ever blank on which code is which, the Bash Exit Code Lookup decodes 124 and 137 directly.)

The .partial trick matters more than it looks

If you redirect straight to the real backup file and the command times out mid-write, you've just replaced last night's good backup with a half-written, unrestorable file — and you won't know until the night you need it. Writing to a .partial path and mv-ing into place only on a clean exit means a failed or timed-out run leaves the previous good backup untouched. A mv on the same filesystem is atomic; the redirect is not.

For commands that talk to the network, layer the tool's own timeout underneath — curl --max-time, ssh -o ConnectTimeout, a net_read_timeout on the dump. Those fire first and fail cleanly. timeout is the outer hard stop for the night the inner one doesn't.

Back to the nine days

A timeout is what makes a lock safe. Locking a job to a single instance stops overlap, but a hang inside the locked job holds that lock forever — which is precisely how my nine-day gap happened. Bound the runtime and the lock always gets released, on a deadline, every time, and the failure becomes a loud 124 in the log instead of a silent gap you discover during a restore.

Full script with the exit-code branching and the FAQ on timing out a whole pipeline: https://bashsnippets.xyz/snippets/bash-timeout-command

The lock that made my hang invisible is flock, and the third guard is retry with backoff. The Hardened Cron Wrapper Generator composes all three, Bash Scripts That Survive Cron walks the whole decision, and the rest of the library is at https://bashsnippets.xyz

Originally published at bashsnippets.xyz

# A Cron Job Took Our Server to Load 41 by Attacking Itself

Tue, 23 Jun 2026 00:16:48 +0000

A */1 rsync took our staging box to a load average of 41 one afternoon, and it took me longer than I want to admit to work out why. The sync normally finished in about twenty seconds. That day the backup target's NFS mount went sluggish, the sync started taking ninety seconds, and cron — which does not know or care whether the last run is still going — launched a fresh copy every single minute on top of it.

Inside ten minutes there were a half-dozen rsyncs all reading the same tree off the same slow disk, each one making the disk slower, each new minute adding another. The box wasn't under attack. It was attacking itself, one polite copy at a time. The thing that stung was that nothing was broken — every individual rsync was correct, the disk eventually recovered on its own, and the only reason it became an outage is that cron has no concept of "the last one is still running."

That's the trap with scheduled jobs: a command that's perfectly fine when you run it by hand can take down a server the first time it runs longer than its interval with nobody watching.

The fix everyone reaches for first is the wrong one

The instinct is a PID file: write $$ to /var/run/job.pid on start, check whether that file exists on the next run, bail if it does. It almost works. Then one run gets kill -9'd, or the box reboots mid-job, and the PID file is left behind pointing at a process that died on Tuesday. Now every future run sees a "lock" owned by a PID that no longer exists, and the job never runs again — the opposite failure, just as silent.

There's also a race between the check and the write, and the times you most need the lock to be clean are exactly the times cleanup didn't happen, because the process died before it could clean up.

flock has none of that. The lock isn't a file you create and delete — it's a lock the kernel holds on an open file descriptor, and the kernel releases it automatically the instant that descriptor closes. The process exiting closes it. So does crashing. So does kill -9. There is no state to leave behind, which is the entire reason it survives the failure modes a PID file can't.

The single-instance pattern

#!/bin/bash
# Script: backup-with-lock.sh
# Purpose: Stop a cron job from overlapping itself when one run runs long
set -euo pipefail

CHECK="✓"
CROSS="✗"

# /run/lock is tmpfs, cleared cleanly on reboot. Never /tmp — temp-cleaners
# delete files there, and a deleted lock mid-run lets a second copy run.
LOCK_FILE="/run/lock/$(basename "$0").lock"

# The > opens (and creates) the lock file on fd 200 and holds it open for the
# whole script. The lock lives on this descriptor, not on the file existing.
exec 200>"$LOCK_FILE"

# -n = non-blocking: if a previous run still holds the lock, give up now
# instead of queueing another copy behind it.
if ! flock -n 200; then
    echo "$CROSS $(date '+%F %T') previous run still active — skipping" >&2
    exit 0
fi

echo "$CHECK $(date '+%F %T') lock acquired — starting"
rsync -a --delete /data/ /mnt/backup/data/
echo "$CHECK $(date '+%F %T') finished — kernel releases the lock on exit"

The two lines doing the work are exec 200>"$LOCK_FILE" and flock -n 200. The first opens the lock file on a descriptor that stays open for the life of the process. The second tries to grab the lock without waiting; if a sibling process already holds it, flock returns non-zero, we log it and exit 0 — a skipped run is normal, not an error, so we don't want it lighting up cron's mail.

Notice there is no cleanup. No trap to remove a PID file, no rm at the end. When this script exits for any reason, fd 200 closes and the lock is gone. That "for any reason" is the whole point.

You can lock a job without editing it at all

If the misbehaving job is already deployed and you don't want to touch it, wrap it from the crontab line:

# Skip the run if the last one is still going
*/1 * * * * /usr/bin/flock -n /run/lock/sync.lock /usr/local/bin/sync.sh

flock runs sync.sh only if it can grab the lock; if last minute's run is still holding it, this minute's run exits immediately and does nothing. It's the fastest retrofit for a job that's already on fire — no redeploy.

One thing worth burning into memory: -n skips, -w 30 waits up to thirty seconds then gives up, and a bare flock with neither blocks forever. On a fast cron schedule that bare form turns your "skipped" runs into a pile of stuck processes — the exact thing you were trying to prevent.

The part that actually mattered

The load-41 afternoon ended the moment I wrapped that rsync in flock -n. The slow NFS mount was still slow, but now exactly one sync ran at a time and the extras skipped harmlessly until the disk recovered. Locking didn't fix the slow disk — it stopped a transient slow disk from becoming a self-inflicted outage. That's the difference between a script that works when you run it and one that survives unattended.

A lock alone isn't the whole story, though. If the locked job itself hangs, it holds the lock forever and every future run skips — so the job silently stops running and you find out days later. That's why locking pairs with bounding runtime with timeout and retrying transient failures.

Full script, the -n vs -w decision, and the FAQ on where the lock file should live: https://bashsnippets.xyz/snippets/bash-flock-single-instance

If you're hardening a cron job, the next two guards are timeout and retry with backoff; the Hardened Cron Wrapper Generator stitches all three into one wrapper, and the full reasoning is in Bash Scripts That Survive Cron. The rest of the library is at https://bashsnippets.xyz

Originally published at bashsnippets.xyz

A Function Without local Overwrote My Variable and rm -rf Deleted the Wrong Directory

Mon, 22 Jun 2026 02:45:23 +0000

Bash variables are global by default, so a helper function clobbered the caller's path variable and rm -rf cleaned up the wrong directory. local, return-as-status, and getopts — all in one incident.

Tags: #bash #linux #scripting #devops

The deploy script had been running in production for four months without a problem. It built releases into a temp directory, ran some validation, and then cleaned up by removing whatever $target pointed at. $target was set near the top of the script to the current release directory — the one the running application was serving from. A helper function called prepare() also used a variable named target, because the person who wrote it (me, four months earlier) did not think about scope.

prepare() built the new release into a temp directory. On a good run, it set target to the temp path, did its work, and returned. The main script then did its cleanup at the end and removed $target — which, after calling prepare(), was the temp directory. That worked correctly for four months.

Then a deploy failed partway through prepare(). The temp directory was half-built. target was now pointing at the half-built temp path. The main script caught the failure, started its cleanup, and ran rm -rf "$target". It removed the half-built temp directory, which was correct. But then it kept going — there was a second cleanup step that also used $target and expected it to still be the release directory. By the time the script finished, it had removed the running application's release directory. The application restarted and found nothing to serve.

The users noticed before I did. The restart loop was filling logs, the application was returning 502, and I was sitting in the deploy output trying to figure out what had gone wrong in a failure path I had tested against a stub.

Variables in bash are global by default

This is the single most surprising thing about bash functions if you have written code in almost any other language. In Python, a variable assigned inside a function is local to that function unless you explicitly declare it global. In bash, it is the opposite. A variable assigned inside a function is visible — and writable — everywhere in the current shell unless you declare it local.

target="/srv/release/current"

prepare() {
  target=$(mktemp -d)    # No local — this overwrites the global $target
  echo "building in $target"
}

prepare
echo "target is now: $target"   # Prints the temp dir, not /srv/release/current

The function does exactly what it looks like — it sets target. The problem is that it sets target everywhere, not just inside itself. The caller's target is gone.

local confines the assignment to the function scope:

prepare() {
  local target              # confined to this function
  target=$(mktemp -d)
  echo "$target"            # hand the value out via stdout
}

build_dir=$(prepare)        # capture what the function echoed
echo "built in: $build_dir"
echo "release dir still: $target"   # unchanged

local target means: this variable exists only inside this function. When the function returns, the variable and its value vanish. The caller's target is never touched. The habit I now enforce on every bash function I write: local for every variable the function introduces, not just the ones I think might conflict. The conflict I do not predict is the one that deletes the wrong directory.

return is a status, not a value

After the incident, I audited every function in the deploy script. I found a second bug:

count_pending() {
  local n
  n=$(find "$QUEUE_DIR" -type f | wc -l)
  return "$n"      # WRONG if n > 255
}

count_pending
if [[ $? -gt 0 ]]; then
  echo "queue has items"
fi

return sets an exit status. Exit statuses are a single byte: 0 to 255. return 300 wraps to 44. For months the queue count had been above 255 on busy days, and the $? check was comparing against a wrapped value. The logic had been wrong for months and had accidentally worked because the wrapped values still triggered the gt 0 condition. But any script making real decisions based on the actual count — how many workers to spin up, whether to page someone — would have been working with garbage.

The correct pattern is to echo the value and capture it:

count_pending() {
  local n
  n=$(find "$QUEUE_DIR" -type f | wc -l)
  echo "$n"         # data goes to stdout
  return 0          # status: success
}

pending=$(count_pending)
echo "queue depth: $pending"

return answers "did this succeed." echo plus command substitution answers "what is the value." These are two different questions and bash gives you two different mechanisms for a reason. Mixing them is how a function that counts 300 items makes the caller think it counted 44.

Arguments and the $@ quoting rule

Inside a function, arguments arrive as positional parameters: $1, $2, all of them as "$@", the count as $#. Quoting "$@" is what keeps multi-word arguments intact:

process_hosts() {
  echo "processing $# hosts"
  for host in "$@"; do
    echo "  checking: $host"
  done
}

process_hosts "web-01" "db primary" "cache-02"

Without the quotes around "$@", db primary splits into two loop iterations and you are back to the word-splitting problem from a different angle. The quoted "$@" is the way bash passes an array of arguments through a function call with each element preserved.

This matters most when you are writing wrapper functions — functions that receive arguments and pass them to another command:

run_with_retry() {
  local retries="${1:?}"
  shift
  local attempt=0
  while ((attempt < retries)); do
    "$@" && return 0
    ((attempt++))
    echo "retry $attempt/$retries"
    sleep 2
  done
  return 1
}

run_with_retry 3 rsync -av "source dir/" remote:/dest/

"$@" after the shift is everything after the retry count — the command and all its arguments, each preserved as a separate item even if they contain spaces. Without the quotes, "source dir/" splits and rsync receives the wrong arguments.

getopts for anything with flags

For one or two fixed positional arguments, reading $1 and $2 directly is fine. The moment a function or script takes optional flags in any order, do not parse them by hand:

verbose=0
output_dir=""

while getopts "vd:" opt; do
  case "$opt" in
    v) verbose=1 ;;
    d) output_dir="$OPTARG" ;;
    *) echo "usage: $0 [-v] [-d dir]"; exit 1 ;;
  esac
done
shift $((OPTIND - 1))   # move past the flags to positional args

The colon after d marks it as requiring an argument, which arrives in $OPTARG. getopts handles flag bundling (-vd dir), missing arguments (-d with no path generates an error automatically), and unknown flags. Hand-rolled $1 parsing gets all of these wrong in subtle ways — it accepts -d at the end without a value, it does not handle -vd dir, and it requires the flags in a specific order.

The deploy script that caused the incident had hand-rolled argument parsing. Among other things, it silently accepted a --target flag with no value and proceeded with an empty string, which caused a different class of problem I had also not fully traced before the bigger incident made the whole thing visible.

What the deploy script looks like now

Every function declares local for every variable. Values that need to cross function boundaries go through echo and command substitution. Exit statuses communicate success or failure. getopts handles the flags. There is a trap on EXIT that cleans up the temp directory using a local variable that only the cleanup function can see:

cleanup() {
  local tmp_dir="${TEMP_BUILD_DIR:-}"   # local ref to the temp dir
  [[ -n "$tmp_dir" && -d "$tmp_dir" ]] && rm -rf "$tmp_dir"
}
trap cleanup EXIT

The $target variable in the main script is set once at the top and never touched by any function. Functions that need a temp directory create one, store it in a local variable, use it, and the cleanup trap handles removal. The variable naming conflict that caused the incident cannot happen because the pattern prevents it structurally.

The application has been running cleanly since then. The deploy script has hit the failure path twice since the fix — different failures, unrelated causes — and both times the cleanup ran correctly and left the running release untouched.

Full version with the local-scope fix, the echo-for-values pattern, the return-as-status trap, and a getopts template: https://bashsnippets.xyz/snippets/bash-functions-arguments

A function that fails should fail loudly — wrap the script in set -euo pipefail — and the bash boilerplate generator can scaffold all of this with the right traps and argument parsing wired in from the start. The rest is at https://bashsnippets.xyz

Originally published at bashsnippets.xyz/snippets/bash-functions-arguments

The Alert Never Fired Because the Loop Skipped the Last Line of the File

Fri, 19 Jun 2026 17:58:59 +0000

We kept a plaintext file of hostnames, one per line, and a monitoring script read the file and pinged each host every five minutes. When a host failed to respond, the script sent an email alert. The system had been running for months and it worked — we had caught three actual outages with it, which gave us real confidence in the setup.

app-07 was added to the list on a Thursday afternoon. The engineer who added it was using VS Code on a Mac, and VS Code by default does not add a trailing newline to a file when you append to it using certain editing workflows. The file had ended in a newline before the edit. After the edit, the last line — app-07 — had no trailing newline.

app-07 went down the following Sunday afternoon at 2:17pm. The monitoring script ran at 2:20, 2:25, 2:30, all the way through the evening. No alert ever fired. The on-call engineer found out at 8pm when a client emailed. The system had been down for almost six hours.

When I looked at the script, the bug was immediately obvious once I knew what to look for. But I had written that script, I had tested it, and I had been looking at the monitoring confirmation emails for months without ever noticing. The confirmation email listed the hosts it checked. app-07 was never in the list. I had been reading those emails without actually counting the hosts. I just scanned for the OK lines and moved on.

Why read drops the last line

read returns a success exit status when it reads a line and finds the newline that terminates it. When the file does not end in a newline, read still populates the variable with the final line's content, but it returns a non-zero (failure) exit status because it hit end-of-file before finding a terminator. A while read host loop checks the return status to decide whether to execute the loop body. On the final, newline-less line, read puts app-07 into host and then returns failure. The while loop sees failure and exits without running the body. The content is there. The variable is populated. The loop throws it away.

This behavior is documented in the POSIX spec for read. It is not a bash quirk. Any POSIX shell handles the missing-final-newline case this way. A plain while read line loop is incorrect for any file you do not personally control the formatting of, which in practice means nearly any file.

The fix is one extra clause:

#!/bin/bash

# Script: check-hosts.sh

# Purpose: ping every host in a file, including a newline-less final line

# Usage: ./check-hosts.sh hosts.txt

set -euo pipefail

CHECK="✓"
CROSS="✗"
HOST_FILE="${1:?Usage: check-hosts.sh <host-file>}"

while IFS= read -r host || [[ -n "$host" ]]; do
  [[ -z "$host" || "$host" == \#* ]] && continue
  if ping -c1 -W2 "$host" >/dev/null 2>&1; then
    echo "$CHECK up:   $host"
  else
    echo "$CROSS down: $host"
  fi
done < "$HOST_FILE"

|| [[ -n "$host" ]] says: if read returned failure but the variable is non-empty, run the loop body anyway. That is precisely the leftover-final-line case. read failed because it hit end-of-file, but it populated host with app-07 before returning. The || catches it. app-07 gets pinged.

What IFS= and -r actually do

You see while IFS= read -r line written in every correct read-loop example, and it is worth being specific about what each piece prevents because both have their own failure mode.

IFS= sets the field separator to empty for the duration of the read command. Without it, read strips leading and trailing whitespace from each line. A hostname like app-07 (with leading spaces, which some editors produce) becomes app-07, which might be correct. An indented config value, a Python-style YAML string, a log line that starts with spaces for alignment — all of these are silently modified. Setting IFS= tells read to take the line exactly as it appears.

-r prevents read from interpreting backslash sequences. Without -r, a line like C:\temp\logs has its backslashes consumed as escape characters and arrives as C:templogs. This matters less for hostname files and enormously for any script that processes Windows paths, config files that use backslash as a line-continuation character, or log files from mixed-OS environments. The -r flag is essentially free protection; there is no reason not to include it.

A bare read line without either flag silently mangles both whitespace and backslashes. The script works correctly on clean input and produces wrong output on input with edge cases. The wrong output does not produce an error. You find out when the data that mattered was the indented or backslash-containing kind.

The subshell trap that kills your counters

This is the one that is most likely to make you question your sanity:


# This looks correct. It is not.

fails=0
cat hosts.txt | while IFS= read -r host; do
  ping -c1 -W2 "$host" >/dev/null 2>&1 || ((fails++))
done
echo "Total failures: $fails"   # Always prints 0

The pipe creates a subshell for the right side. The while loop runs inside that subshell. fails increments correctly inside the subshell. When the subshell exits, the parent shell's fails is still 0, because the increment happened in a different process. The parent echo sees the original value.

This catches people because the loop body itself works — the pings happen, the increment logic is correct — but any state the loop was supposed to accumulate for later use is silently discarded. I spent forty minutes on a version of this problem before I remembered that pipes create subshells. It is the kind of thing that feels like a bash bug until you understand that it is behaving exactly as documented.

The fix is to redirect the file into the loop instead of piping into it:

fails=0
while IFS= read -r host || [[ -n "$host" ]]; do
  ping -c1 -W2 "$host" >/dev/null 2>&1 || ((fails++))
done < hosts.txt
echo "Total failures: $fails"   # Now correct

done < hosts.txt feeds the file to the loop's stdin without a pipe. The loop runs in the current shell. fails accumulates in the current shell. The echo sees the real count.

Why for loop is wrong for this


# Never do this — iterates words, not lines

for line in $(cat hosts.txt); do
  ping -c1 "$line"
done

$(cat hosts.txt) is command substitution. Bash captures the text output and word-splits it on IFS — spaces, tabs, newlines. For a file with one hostname per line and no spaces in the hostnames, this accidentally produces the right behavior. For any file with spaces — log lines, config values, paths with spaces, anything a non-developer might have generated — it splits lines into fragments and each fragment becomes a loop iteration.

There is no version of for line in $(cat file) that is correct for reading lines. The right tool is always while IFS= read -r line || [[ -n "$line" ]]; do ... done < file. The for loop is the right tool for iterating a known list that you control directly, not for reading file content.

The monitoring system, after the fix

After the app-07 incident we made three changes. The obvious one was fixing the read loop with the || [[ -n "$host" ]] guard. The second was adding a sanity check at the top of the script that counted the lines in the hosts file and compared it to the number of hosts the loop actually processed — a mismatch meant the file was malformed or something else was wrong. The third was adding a nightly email that included the count of hosts checked, not just the status of each one, so a future addition to the file that somehow got lost would show up as "expected 12 hosts, checked 11."

The confirmation email count was something I should have had from the start. If I had been looking at "checked 11/12 hosts" instead of a list of OK lines, I would have noticed app-07 missing on the first night. The monitoring was working. The observability of the monitoring was not.

Full version with CSV-field parsing, comment-skipping, and the subshell-safe redirect form: https://bashsnippets.xyz/snippets/bash-read-file-line-by-line

To iterate a list of files rather than a file's contents, reach for a for loop instead, and wrap anything that acts on what it reads in set -euo pipefail. More at https://bashsnippets.xyz

Originally published at bashsnippets.xyz/snippets/bash-read-file-line-by-line

A For Loop Skipped Every File With a Space and Called the Backup a Success

Thu, 18 Jun 2026 19:28:03 +0000

The nightly backup looped over for f in $(ls /data/exports) and copied each file to a backup volume. It exited clean every night. For three weeks, green exit codes, no errors, nothing in the logs to suggest anything was wrong. The backup script had been written by someone who left the company six months before I started, and it had never been tested against files with spaces in their names because the original export directory only ever had files like Q3.xlsx and report-final.xlsx.

Then someone generated Q3 final.xlsx.

It took three weeks because that file was generated once a quarter and the next time someone needed it was a quarter later. The person who needed it was the CFO. The CFO does not particularly enjoy being told that the backup system that was supposed to protect the quarterly export has been silently failing for an indeterminate period of time and we are not sure which other files it missed.

I know the specific backup had been running for three weeks because that was the date the file appeared in the exports directory. Every night since then, the loop had been splitting Q3 final.xlsx into two items — Q3 and final.xlsx — trying to copy two paths that did not exist, logging two harmless "no such file or directory" lines that nobody read, and moving on. Every file without a space in its name backed up fine. The script looked correct because most of the time it was.

Why $(ls) word-splits

When you write for f in $(ls /data/exports), bash runs ls, captures its text output, and splits it on IFS — the internal field separator, which defaults to spaces, tabs, and newlines. Filenames are just text in the output of ls. A file named Q3 final.xlsx is one filename, but ls outputs it as the string Q3 final.xlsx, and bash splits that string on the space into two separate items before the loop ever starts.

This is not a bug in ls. This is not a bug in bash. This is exactly what $( ) does to any command's output — it captures text and bash processes it as text. The problem is that filenames are not reliably text-safe; they can contain any character except null and the path separator. Spaces are common. Tabs are less common but legal. Newlines are technically legal. Treating command output as a filename list breaks the moment any of those show up.

The fix is to stop treating command output as a filename list and let bash build the list from the filesystem directly:

#!/bin/bash
# Script: backup-exports.sh
# Purpose: copy export files without losing ones with spaces in their names
# Usage: ./backup-exports.sh
set -euo pipefail

CHECK="✓"
CROSS="✗"
SRC_DIR="/data/exports"
DEST_DIR="/backup/exports"
shopt -s nullglob   # empty glob expands to nothing, not the literal pattern

for f in "$SRC_DIR"/*.xlsx; do
  if cp -- "$f" "$DEST_DIR/"; then
    echo "$CHECK backed up: $(basename "$f")"
  else
    echo "$CROSS failed: $(basename "$f")"
  fi
done

for f in "$SRC_DIR"/*.xlsx asks bash to expand the glob. Bash talks directly to the filesystem and gets back a properly-separated list of matching paths. No command output, no text splitting, no ambiguity about what the separator is. A file named Q3 final.xlsx stays one item because it was never turned into text and split back apart. The -- before "$f" tells cp to stop reading flags, so a filename that starts with a dash does not get interpreted as a cp option.

The second half: quoting on use

The glob fixes the loop header. Quoting fixes the point of use. These are separate.

for f in "$SRC_DIR"/*.xlsx; do
  cp $f "$DEST_DIR/"    # WRONG — $f re-splits here
  cp "$f" "$DEST_DIR/"  # RIGHT — the quotes prevent the split
done

Even with a correct glob, an unquoted $f in the cp command re-splits on spaces. Bash has already expanded the variable to Q3 final.xlsx, but when you use it unquoted, the shell processes word splitting again on that value and cp receives two arguments: Q3 and final.xlsx. The quotes around "$f" tell bash to pass the entire value as a single argument. The rule is: glob over parse to build the list, quote on use to keep each item intact.

I have seen this mistake repeated in scripts at three different companies. In each case the scripts had been running for months or years and the problem was invisible because the majority of files had no spaces. The ones that did — client names, quarterly reports, anything a non-technical person named — were being silently skipped or mishandled.

Ranges and counters: where the brace trap hides

n=5

# This does NOT produce a range — it produces the literal string {1..5}
for i in {1..$n}; do echo "attempt $i"; done

# This works — C-style, evaluates variables at runtime
for ((i = 1; i <= n; i++)); do echo "attempt $i of $n"; done

Brace expansion happens before variable expansion in bash's order of operations. By the time $n is replaced with its value, the brace expansion step has already passed and {1..$n} is just a string. This is the kind of thing that fails silently in a test environment where the count is small and hardcoded, and causes weird output in production where it is a variable.

The C-style loop is the correct form any time the bound is a variable. It evaluates at runtime and handles arithmetic naturally. If the count is genuinely fixed at write time, {1..10} works fine. If it is ever going to be a variable, use for (( )).

Arrays: the original bug wearing a different hat

servers=("web-01" "db primary" "cache-02")

# WRONG — word-splits "db primary" into two iterations
for s in ${servers[@]}; do
  ping -c1 "$s"
done

# RIGHT — quotes keep each element intact
for s in "${servers[@]}"; do
  ping -c1 "$s"
done

"${servers[@]}" with the quotes and [@] is the form that preserves each element as a single item regardless of what is in it. Without the quotes, bash word-splits the array expansion and db primary becomes two separate loop iterations — neither of which is a real hostname. This is exactly the same word-splitting mechanism as the $(ls) problem, just manifesting in arrays instead of command output.

Once you internalize that word-splitting happens wherever an unquoted variable or expansion appears, the rule becomes one rule instead of several: always quote expansions. The specific context changes; the mechanism does not.

The nullglob case

shopt -s nullglob
for f in /data/exports/*.xlsx; do
  echo "$f"
done

Without shopt -s nullglob, if no .xlsx files exist, the glob *.xlsx does not expand — it stays as the literal string *.xlsx. The loop runs once with f set to the literal string /data/exports/*.xlsx. Your script then tries to process a file with that exact path, which does not exist, and produces an error or silently does nothing depending on what you do with it.

Setting nullglob tells bash to expand a non-matching glob to nothing (an empty list), so the loop simply does not run. This is almost always the right behavior when you are iterating files that might not exist.

What happened to the Q3 export

We recovered it from the CFO's local machine, where she had downloaded it before the backup was supposed to preserve it. The fix to the script took four minutes. The conversation about why the backup system had been failing silently for three weeks took longer. The monitoring that we added afterwards — a nightly check that the backup directory has at least as many files as the source directory — took another twenty minutes.

The monitoring should have been there from the start. So should the glob. So should the set -euo pipefail that would have made the copy failures loud instead of silent. These are things you add before something breaks, and the only reason to know you need them is to have seen, or caused, or read about what happens when they are missing.

Full examples with the safe glob, counter, C-style loop, array form, and nullglob guard: https://bashsnippets.xyz/snippets/bash-for-loop-examples

For reading a file's lines one at a time, a for loop is the wrong tool — use while IFS= read -r — and wrap any loop that touches real files in set -euo pipefail. The rest is at https://bashsnippets.xyz

find . -delete Ran Before the Filter and Emptied the Whole Tree

Wed, 17 Jun 2026 23:57:57 +0000

I meant to delete the .cache files under a data directory. The server had been running for two months and the cache layer had grown to about 14GB. The application team told me it was safe to purge it — they'd rebuilt the cache logic and the old files were just dead weight. I typed find /data -delete -name "*.cache" because I was moving fast and I figured the order of arguments to find did not matter. It does. find evaluates its expression left to right, and -delete is not a filter — it is an action. It fired on every path find walked, starting at /data itself, and -name "*.cache" never got a chance to narrow anything. By the time I hit Ctrl-C the tree was roughly 80% gone.

That was not a test server.

The restore from backup took forty minutes. The application was down for those forty minutes. The postmortem was a forty-five minute conversation with people who did not particularly enjoy having it. I have run hundreds of find commands since then and I verify the expression order before every single one that has any destructive action attached to it — not because I've forgotten the rule, but because the cost of forgetting it once is not recoverable with an apology.

Why the order is the program

find does not have a flag parser that groups tests and actions separately. It walks a directory tree and evaluates its arguments as a logical expression, left to right, short-circuiting on false. When you write:

find /data -name "*.cache" -delete

it evaluates -name "*.cache" first on each path. If the name does not match, it short-circuits and -delete never runs on that path. When you write:

find /data -delete -name "*.cache"

it evaluates -delete first. -delete always succeeds — it removes the path and returns true, which means the expression continues to -name. The name check runs after the deletion, on a file that no longer exists, which is meaningless. The effect is that everything gets deleted and nothing is filtered.

This is not a bug. It is exactly how the man page says find works. It is just not how anyone instinctively reads a command the first few times they use it.

The rule is: tests filter, actions act, and actions must come after the tests that are supposed to narrow them. Write it in that order every time.

The quoting trap, right behind the ordering one

Here is the one that is even more subtle, because it causes the wrong behavior and produces no error at all:

# WRONG — the shell expands *.cache before find ever sees it
find /data -name *.cache

# RIGHT — quote the pattern so find does the matching
find /data -name "*.cache"

If there happens to be a single .cache file in your current working directory when you run this, the shell expands *.cache to that one filename and passes it to find -name as a literal string. find then searches for files named exactly that, everywhere under /data. It finds some, it finds none, but it is definitely not doing a wildcard search. If there are multiple .cache files in your current directory, find receives too many arguments for -name and errors out with something confusing about paths needing to precede the expression.

Either way you did not get what you intended, and if you added -delete, you deleted the wrong things silently.

Quoting the pattern is the fix. The quotes prevent the shell from expanding the glob so find receives the literal *.cache pattern and handles the wildcard itself, across the directory tree you pointed it at.

The age sign that everyone reverses at least once

find /var/log -name "*.log" -mtime +30   # older than 30 days
find /var/log -name "*.log" -mtime -1    # modified within the last day
find /var/log -name "*.log" -mtime 30    # exactly day 30 (almost never what you want)

+30 is older than thirty days. -1 is within the last day. A bare 30 means precisely thirty days ago, which is almost never the thing you're trying to match. The sign convention is the opposite of what feels natural — you want files "older than" a threshold and the intuitive symbol for "bigger number" is +, but a lot of people read +30 as "in the last thirty days" the first time they see it.

I have reversed this twice in production. Once I kept the wrong logs. Once I deleted logs I needed for an audit the following week. Neither was catastrophic but both were embarrassing, and both happened under the kind of mild time pressure where double-checking the man page feels slower than it actually is. The builder I built for this labels the output as "older than" or "within the last" in plain English next to the value, which removes the one decision in a find -delete job most likely to go wrong when you are already stressed.

The -exec batching difference nobody explains

# Runs the command once per file — slower, one PID per file
find /data -name "*.cache" -exec rm {} \;

# Batches files into one invocation — faster, one rm for many files
find /data -name "*.cache" -exec rm {} +

\; runs the command once per file. + collects as many paths as it can and passes them all to one invocation of the command, the same way xargs does. For something like rm, which accepts multiple arguments, the + form is faster and produces less process overhead. For something like a custom script that must process exactly one file at a time, \; is correct.

Most resources either do not mention this difference or mention it once in a reference table. The practical consequence is real — on a directory with ten thousand files, \; spawns ten thousand processes. The + form spawns a handful. On a cleanup job that runs in cron, the difference shows up in CPU load.

The preview step I skipped the day I broke things

The most reliable way to avoid the ordering and quoting mistakes is to build the command in two steps. First, run it with -print instead of -delete:

find /data -name "*.cache" -print

Read every line of that output. Confirm the list is what you intended. Then, and only then, swap -print for -delete. This adds maybe thirty seconds to the workflow. It would have saved me forty minutes and a postmortem. I skipped it because I was confident. Confidence is not a useful substitute for verification when the operation is irreversible.

The find command builder enforces this by showing a warning whenever you select -delete or -exec as the action, and offering to generate the -print version of the command first. It is the kind of nudge I would have appreciated having that day.

What the builder actually does

It assembles the expression in the legally correct order — tests first, action last — so you cannot accidentally replicate the mistake I made. You set the starting path, add tests in whatever order feels natural to you (name, type, age, size, exclude path), and pick an action. The output command always has the tests before the action, regardless of the order you clicked things in.

Every active flag gets a plain-English description inline. -mtime +30 reads as "modified more than 30 days ago." -name "*.cache" reads as "name matches the glob *.cache, quoted so find handles the wildcard." The -exec {} + form is the default when you pick -exec, with a note explaining why it is faster.

The whole point is to get from "I need to find and delete files matching these conditions" to a verified, copy-paste command without the thirty-second loop of man-page reading and second-guessing that I used to do and, on one bad morning, skipped.

What I upgraded while I was at it

The same discipline applies to the other tools. The rsync command builder now has presets for the three setups people build most often — local backup, push to remote, mirror — because those cover maybe ninety percent of rsync jobs and getting the flags right from scratch every time is where the dangerous ones like --delete get misapplied. The cron builder previews the next five run times after you build an expression, because a cron job I once deployed ran at 3am UTC instead of 3am local time and I did not notice until it fired on the wrong schedule for a week. You can also paste an existing crontab line and get the human-readable schedule back. The chmod builder now accepts an octal you paste in and sets the checkboxes — two-directional, because reading a file's permissions and understanding what they mean is just as common a task as setting them from scratch.

These are not features I planned in advance. They are all things I needed at 2am and did not have. That is the pattern most of this site runs on.

Build a find command with tests ordered before actions and every flag explained: https://bashsnippets.xyz/tools/find-command-builder

If you are scoping files before searching or transforming them, the whole pipeline is documented in Bash Text Processing: find, grep, sed, and awk. The rest of the free tools are at https://bashsnippets.xyz/tools

A for Loop Skipped 23 Files and Called It a Successful Backup

Sun, 14 Jun 2026 17:23:46 +0000

The backup ran every night at 2am and emailed me a green "847 files archived" summary. I'd built it, tested it against my own home directory where every file was named like report_2024.csv, watched it sail through, and shipped it. For weeks the summary said everything was fine.

Then a coworker asked me to restore a file. Q3 forecast.xlsx. It wasn't in the archive. Neither was Annual Review FINAL.pdf, or meeting notes (draft).md, or any of the other 23 files someone had named the way normal humans name files — with spaces in them. The backup had been quietly skipping the most important files on the share for a week, and the nightly email had been telling me the whole time that nothing was wrong.

Here's the line that did it:

for file in $(ls "$DIR"); do
  echo "Processing $file"
done

The problem is word splitting, and it's invisible until the day it isn't. An unquoted $(ls ...) splits its output on every space, so Q3 forecast.xlsx doesn't arrive as one filename — it arrives as two iterations, Q3 and forecast.xlsx. Neither one exists on disk. The loop tries both, finds nothing, shrugs, and moves on without a single error. The script "succeeds" because from its point of view it did exactly what it was told.

I've now got two habits burned in, and I don't write a loop without both. The first: glob the directory, never parse ls.

for file in "$DIR"/*; do
  [[ -e "$file" ]] || continue   # the glob yields a literal '*' on an empty dir
  echo "Processing $file"
done

"$DIR"/* hands you each real path as a single unit. No subshell, no string to re-split, no ls output to misread. The [[ -e "$file" ]] || continue guard covers the one quirk of globbing: when a directory is empty, * expands to the literal character *, and without the guard you'd try to process a file named *.

The second habit: quote every expansion, every time. "$file", never $file. The day a filename has a space in it, the unquoted version splits into two arguments and your command operates on paths that were never there.

Arrays follow the exact same rule, and the distinction that matters is [@] versus [*]:

servers=("web-01" "db-prod 02" "cache-03")

for host in "${servers[@]}"; do
  echo "Connecting to $host"   # three clean iterations, the space survives
done

"${servers[@]}" in double quotes gives you one word per element — db-prod 02 stays whole. "${servers[*]}" joins everything into a single string and is almost never what you want in a loop. Drop the quotes on either and you're back to the bug that ate my backup.

When you genuinely need a counter — walking app.log.1 through app.log.9, numbering batches, counting retries — use the C-style form. Do not reach for for i in {1..$n}; brace expansion runs before variable expansion, so $n never gets substituted and you loop once over the literal text {1..$n}. Ask me how I know.

for ((i = 1; i <= MAX_ROTATED; i++)); do
  log="$LOG_DIR/app.log.$i"
  [[ -f "$log" ]] || continue
  echo "Scanning $log"
done

And reading a file line by line — for line in $(cat file) is wrong in two directions at once. It splits on whitespace instead of newlines, so a line with spaces becomes several iterations and blank lines vanish, and it globs, so a line containing * expands to filenames in your current directory. The form that survives real files:

while IFS= read -r line; do
  echo "Line: $line"
done < "$INPUT_FILE"

IFS= stops bash from trimming leading and trailing whitespace. -r stops it from eating backslashes. One line per iteration, exactly as written.

The production script I keep on the site ties all of this together — globs the directory, quotes every use, counts successes and failures separately, and exits non-zero if anything failed. That last part is the one people skip, and it's the one that matters: a loop that processes files but always exits 0 is how you end up with a nightly email that says 847 when the real number is 824.

The cost of my mistake was a week of bad backups and the specific discomfort of a coworker finding the gap before my own tooling did. I'm not interested in repeating that, so the quoting is muscle memory now. Yours might as well be too.

Full script with all four loop forms — files, arrays, counters, and line-by-line reads — production-ready and ShellCheck-clean: https://bashsnippets.xyz/snippets/bash-for-loop-examples

If your loops are the thing crashing scripts, the Bash Error Handling snippet pairs with this one, and the rest of the library is at https://bashsnippets.xyz

The Pipeline Was Green for Three Weeks. It Had Been Shipping a Build That Never Compiled.

Fri, 12 Jun 2026 16:25:56 +0000

For three weeks a deployment pipeline reported every step green and shipped a build that had failed to compile on every single run. The build step ended in npm run build | tee build.log so the output could be archived. That pipe is the whole story: bash returns the exit status of the last command in a pipeline, which was tee, and tee always succeeds at copying text. The compiler's non-zero exit got thrown away the instant the pipe handed off. The error was sitting right there in build.log. GitHub Actions saw exit code 0, painted the step green, and deployed the broken artifact. Nobody read the log, because the checkmark said there was nothing to read.

That's the defining property of bash in CI, and it's why I treat pipeline scripts differently from anything I run in a terminal: a silent failure can present as success. On a server you watch a command fail in front of you. In a pipeline, a swallowed exit code produces a green checkmark over broken code, and the gap between "the logs show an error" and "the pipeline reports failure" is exactly where outages are born. I wrote the full guide because I've now been burned by every variation of this, and there's a consistent set of habits that close the gap.

There are four failure modes that are specific to CI and barely ever bite you at an interactive prompt. Exit codes swallowed by a pipe — the story above, any command | tee, command | grep, command | sort. Shell provisioning differences — ubuntu-latest gives you bash 5.x, macos-latest gives you bash 3.2 from 2007, and a script using associative arrays or ${var,,} passes on one runner and throws a syntax error on the other in the same workflow. Environment variable gaps — CI sets variables you don't control and omits ones you assume exist, and without set -u a missing $DEPLOY_TARGET becomes an empty string and does something quietly wrong. Interactive-shell assumptions — CI runs a non-interactive, non-login shell that never sources your .bashrc, so a command that works when you type it dies with command not found because the thing that defined it was never loaded.

The header that closes most of these is short, and every line earns its place:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

set -e exits the moment any command fails, so the step exits non-zero and the workflow actually registers a failure. set -u treats an unset variable as an error, so a typo'd $DPLOY_TARGET dies immediately instead of expanding to nothing and corrupting a path. set -o pipefail makes a pipeline return the first non-zero exit among its commands rather than only the last — that one flag is the direct fix for the | tee bug that ran green for three weeks.

Secrets deserve their own paragraph because CI hands you env: values and secrets: values identically — the shell can't tell them apart, the only difference is that Actions masks the secret's literal string in the log. The trap is that the moment you transform a secret (base64-decode it, slice it, interpolate it), the transformed value no longer matches the mask and prints in clear text. Validate required values up front with ${VAR:?} so a missing secret fails at startup with a clear message instead of on line 47 with a cryptic permission denied, and be very careful with set -x in any step that touches a secret.

The pipe-exit-code problem is worth one concrete tool beyond pipefail: PIPESTATUS is an array holding the exit code of every command in the last pipeline, read immediately after:

npm run build | tee build.log
build_rc="${PIPESTATUS[0]}"   # npm's code, not tee's
[[ "$build_rc" -eq 0 ]] || { echo "build failed: $build_rc" >&2; exit "$build_rc"; }

pipefail has one well-known false positive — grep returns 1 when it finds no matches, which is often fine, and under pipefail plus set -e that aborts the script. Absorb it deliberately with || true only where a non-match is genuinely acceptable, and nowhere else, because blanketing every command in || true just reinvents the silent-success problem you're trying to kill.

Docker has its own landmine: every entrypoint script must end with exec "$@". Without it, your script stays PID 1 and your app runs as a child, so when the orchestrator sends SIGTERM on docker stop or a rolling deploy, the signal hits the script, which doesn't forward it, and after the grace period the orchestrator escalates to SIGKILL — abrupt termination, dropped connections, lost in-flight work. exec "$@" replaces the shell with your app so it becomes PID 1 and receives signals directly. The guide pairs this with a wait_for dependency-check pattern and a trap, and the Bash trap & Signal Handler Builder generates the exact signal block an entrypoint needs.

Deploys get the same treatment: deploy into a fresh timestamped directory, flip a current symlink atomically with ln -sfn so traffic never sees a half-written release, keep the last several releases so rollback is just re-pointing the symlink, run a health check after the swap and fail the deploy if it doesn't pass, and stamp the git SHA into the release so "what's running right now" always has an answer. And when a step fails and the logs won't say why, set -x around just the suspect section shows you each command with its variables expanded — a doubled slash or an empty segment in the trace is usually your bug standing in plain sight.

The full guide is the field manual version of all of this — the four failure modes, the safe header, secret validation with a validate_env function, PIPESTATUS and pipefail, Docker entrypoints, the atomic-symlink deploy script with rollback and health check, debugging with set -x, and a production-ready checklist at the end: https://bashsnippets.xyz/guides/bash-scripting-for-ci-cd-pipelines

If your pipeline scripts are dying on the small stuff first — unquoted loops, bad argument parsing, missing traps — the snippet library that feeds into this guide is at https://bashsnippets.xyz

My Script Crashed and Left a Lock File Behind. Every Run After That Refused to Start.

Thu, 11 Jun 2026 16:55:08 +0000

A backup script of mine created a lock file on startup so two copies couldn't run at once — sensible. Then one night it hit an error partway through, set -e killed it on the spot, and it died without ever reaching the line that removes the lock. The lock file sat there. Every scheduled run for the next three days started up, saw the lock, printed "already running," and exited immediately. No backups ran. The cron job was firing perfectly on time and doing nothing, and the only symptom was an absence — backups that simply weren't there — until I went looking and found a stale lock from Tuesday.

The fix is a trap. A trap registers a cleanup handler that runs when the script exits for any reason — clean finish, set -e failure, Ctrl+C, kill. Put the lock removal in an EXIT trap and it runs no matter how the script dies. The lock would have been gone the instant that backup crashed, and the next run would have started fine.

So why did I write the script without one? Because I could never remember the syntax cold. Single quotes or double? Which signals? Does EXIT fire on exit 1 or only on a clean finish? How do I get the exit code inside the handler? Every time I needed a trap I ended up with three browser tabs open, reading the same Stack Overflow answers, second-guessing the quoting. Under pressure, in the middle of fixing something else, that friction is exactly when people skip the trap entirely — which is how I ended up with the stale lock in the first place.

So I built the thing I kept wishing existed: a Bash trap & Signal Handler Builder. You pick the signals you want to handle, check off the cleanup actions you need, and it writes a correct, ShellCheck-clean trap block you paste into your script. No tabs, no second-guessing the quoting.

It covers the signals that actually come up. EXIT — fires on every exit, the one that should carry your cleanup. ERR — fires when a command fails under set -e, the one that logs the exact failing line with $LINENO. INT — Ctrl+C. TERM — what kill, systemctl stop, and docker stop send. HUP — terminal closed or SSH dropped. PIPE — writing to a closed pipe. Each one has a one-line reminder of when it fires and what it's good for, because half the battle is just remembering that TERM is the one Docker sends.

The cleanup actions are the things people forget until a crash makes them care: remove temp files (with the TMPFILE=$(mktemp) declaration wired in up top), remove a lock file — the exact failure that bit me — stop background jobs the script started, log the exit reason with the code, and restore the terminal cursor. Tick the ones you need and they land in the handler.

The generated code isn't a toy snippet. It single-quotes the trap so the handler resolves when the signal fires instead of at definition time — the SC2064 gotcha most hand-written traps get wrong. It includes an idempotency guard so that when ERR and EXIT both fire on the same failure, your cleanup runs exactly once instead of twice. The ERR handler captures $LINENO so you find out which line actually blew up. I ran the generator's output through ShellCheck across a dozen different configurations while building it, and every one comes back clean — the whole point was that you can paste it and trust it, not paste it and then go debug the thing that was supposed to save you debugging.

There are two copy buttons, because there are two situations. "Copy trap block only" when you've already got a script and just want to drop the traps in. "Copy complete script header" when you're starting fresh and want the shebang, set -euo pipefail, the CHECK/CROSS vars, the resource declarations, and the handler all assembled in the right order. It only declares the variables the generated code actually uses, so you never paste in a CROSS you never reference and get a ShellCheck warning for your trouble.

The lock-file incident cost me three days of silently missing backups and ten minutes of feeling foolish when I found the cause. The trap that would have prevented it is four lines. The reason I didn't have those four lines was pure friction — I couldn't recall the syntax fast enough to be bothered in the moment. This tool removes the friction, which is the only thing that was ever standing between me and a correct script.

Build your trap block here — pick signals, pick cleanup actions, copy clean code: https://bashsnippets.xyz/tools/bash-trap-builder

If you want the full picture of where traps fit — strict mode, cleanup, the failure modes that make them necessary — the Bash Error Handling snippet is the companion read, and the rest of the tools are at https://bashsnippets.xyz/tools

I Packaged the Scripts I Copy to Every New Server Into a $9 Toolkit. Here's What's In It and Why.

Tue, 09 Jun 2026 15:43:01 +0000

Every time I provision a new server — whether it's a $5 DigitalOcean droplet for a side project or a client's production box — there's a set of scripts I copy to /opt/scripts before I do anything else.

Not after the app is deployed. Not after the first incident. Before I touch the application at all. Before I configure nginx. Before I set up the database. The monitoring layer goes in first because the first time you need it, you needed it yesterday.

Disk monitoring that fires before the outage. A backup pipeline with automatic retention. SSL certificate checks that run daily at 8am so I'm not finding out from a user's email. A service watchdog that restarts nginx or Postgres within 60 seconds of a crash instead of six hours later when someone notices the site is down.

These scripts took me about two years of production incidents to build. Not because they're complicated — they're not. Each one is 15-50 lines. Because each one was built in response to a specific failure where I didn't have the thing I needed and had to build it under pressure at a bad time of day.

I open-sourced the basic versions on BashSnippets.xyz. Those are free and they'll stay free. But the versions I actually run in production are different from the tutorial versions in ways that matter — and that gap is what I packaged into the toolkit.

What's Different Between the Free Snippets and the Toolkit

The free snippets on the site are single-purpose scripts that each solve one problem. They're complete. They work. If all you need is a disk space check, the free version does that.

The toolkit versions are built as a system.

There's a shared library — bashlib.sh — with 31 functions that every script sources. Logging, color output, email alerts, error handling, lock file management, threshold checks, dry-run support. Instead of each script reimplementing its own log() function and its own error handling and its own email logic, they all call bashlib.sh and get consistent behavior across the board.

That means when I change how logging works, it changes everywhere. When I add Slack webhook support to the alert function, every script that calls alert() gets Slack notifications without any changes to the script itself. The shared library is the infrastructure layer that turns six standalone scripts into a cohesive system.

The free snippets don't have this because a shared library adds a dependency — bashlib.sh has to exist at a known path, the scripts have to source it at startup, and if someone downloads one script without the library it breaks. That's fine for a toolkit you install as a package. It's bad for a tutorial page where someone wants to copy-paste one script and have it work immediately. Both approaches are correct for their context.

What's in the Box

6 production scripts:

Each one follows the same structure — set -euo pipefail, sourcing bashlib.sh, named variables for every threshold and path (no magic numbers buried in command pipelines), comments explaining not just what each line does but why it exists, and explicit non-zero exits on every failure path.

The scripts cover disk space monitoring, database backup with retention, SSL certificate expiry checking across multiple domains, service watchdog with automatic restart, log rotation and cleanup, and system health reporting.

bashlib.sh — the shared library (31 functions):

This is the part I'm most particular about. Functions for timestamped logging with severity levels. Color-coded terminal output that degrades gracefully when piped to a file (no escape codes in your log files). Email and webhook alerting. Lock file acquisition with stale-lock detection. Dry-run mode that every function respects. PID file management. Threshold comparison helpers. Configuration file loading.

Every function is documented with a usage comment. Every function handles its own error cases. The library passes ShellCheck with zero warnings at every severity level.

template.sh:

A starter template that sources bashlib.sh, sets up traps, parses arguments, and has placeholder sections for your own logic. When I need a new script on a server, I copy this template, fill in the business logic, and the error handling and logging are already done.

52-page field guide (PDF):

Not an API reference. Not a man page reformatted as a PDF. A field guide — structured as the things you need to know in the order you need to know them when you're setting up automation on a new server.

Covers the why behind every pattern in the scripts. Why set -euo pipefail and what each flag actually prevents. Why traps on EXIT instead of just INT. Why lock files need stale detection. Why backup retention has to be a separate step from the backup itself. Why SSL monitoring should be independent of your renewal tool.

Each section includes the real failure scenario that motivated the pattern, because "best practice" without the consequence attached is advice that gets skipped.

Why $9

I thought about this for a while. The scripts are worth more than $9 to anyone who's going to use them — preventing one 4am incident pays for the toolkit immediately. But I also know what it's like to be the person running a $5 VPS on a budget, and I wanted the price to be low enough that buying it doesn't require approval from anyone or a second thought.

$9. MIT license. Unlimited personal and commercial use. You can deploy these on client servers, modify them however you want, include them in your own automation. No subscription. No upsell. No "starter tier" with premium features behind another paywall.

The free snippets on the site are not going away. They're not a demo. They're complete, working scripts that I actively maintain. The toolkit is for people who want the production layer — the shared library, the integrated system, the field guide that ties it together — and want it in one download instead of building it themselves.

Who This Is For

If you're managing one or more Linux servers and you don't already have automated monitoring, backups, and alerting set up — this is the fastest path to having all three. Copy the scripts to the server, edit the config variables at the top of each file, add the cron entries from the guide, and you have a monitoring layer that didn't exist 10 minutes ago.

If you already have these things set up and you built them yourself, you probably don't need this. You might find something useful in bashlib.sh — the stale lock detection or the dry-run mode — but the scripts themselves won't tell you anything new.

If you're learning bash and want to see how production scripts are structured differently from tutorial scripts, the field guide is probably the most useful part. The "why" sections explain patterns that most bash tutorials skip because they're not relevant to a single-file script running on a laptop.

Every Script Passes ShellCheck at Zero Warnings

This one matters to me. ShellCheck is the static analysis tool for bash. It catches the bugs that work on happy-path input and break on edge cases — unquoted variables that split on whitespace, pipelines that swallow exit codes, deprecated syntax that newer bash versions handle differently.

Every script in the toolkit, including bashlib.sh, passes ShellCheck at the strictest severity level with zero warnings. Not "a few style notes we decided were fine." Zero. I treat ShellCheck warnings the same way I treat compiler warnings in C — they are bugs I haven't hit yet, and ignoring them is technical debt with a guaranteed due date.

If you run ShellCheck on these files and get output, something went wrong and I want to know about it.

The Link

→ bashsnippets.xyz/starter-kit

$9. Instant download. 6 production scripts + bashlib.sh shared library (31 functions) + template.sh + 52-page field guide. ShellCheck-clean. MIT license.

Already have scripts and need somewhere to run them? DigitalOcean droplets start at $4/month and you can get $200 in free credit to start:

→ Get $200 free credit — DigitalOcean

The free snippet library with 17+ scripts and 7 interactive tools is at:

→ bashsnippets.xyz

6 small things we shipped across the BashSnippets tools this week

Tue, 09 Jun 2026 03:47:42 +0000

Nobody announces small features. You ship them, they're in there, and the people who find them either notice or they don't. I want to start documenting these because some of them are the kind of thing that makes a tool actually worth using day-to-day instead of being something you visit once and close.

Six updates across six tools this week. None of them are headline features. All of them are fixes for specific annoyances that came up in my own usage, which is the only real source I trust for "does this actually help."

1. Bash Boilerplate Generator — Trap & Cleanup Handler

There's a now a toggle for trap handling. When on, the generated template includes:

cleanup() {
  # Remove temp files, release locks, undo partial changes
  rm -f "$TMPFILE"
}

trap cleanup EXIT INT TERM

Why this matters: trap cleanup EXIT means the cleanup() function runs no matter how the script exits. Normal completion. Ctrl+C. An uncaught error. A kill signal. The cleanup function runs. Every time.

Without this pattern, scripts that create temp files leave them behind when they're interrupted. Scripts that acquire lock files leave them locked. Scripts that make partial changes — creating a directory, writing part of a config — leave the system in an inconsistent state because the cleanup code at the end of the script never ran.

The trap-on-exit pattern is not optional for anything running unattended. It's the thing that separates "runs fine when nothing goes wrong" from "also recovers gracefully when something does." I've seen this pattern left out of boilerplate generators enough times that I wanted to make it explicit and easy to include.

The generated stub includes a TMPFILE=$(mktemp) line as a concrete example of something that needs cleanup. Replace it with whatever state your script actually manages.

→ bashsnippets.xyz/tools/bash-boilerplate-generator

2. Bash Exit Code Lookup — Export as `case` Statement

After looking up an exit code, there's a button that generates a ready-to-paste case $? in ... esac block with the explanation inline as a comment.

Before this, the lookup gave you the meaning of the exit code and you had to write the case handler yourself. That's not hard — the case syntax is simple — but it's friction. You looked up what 126 means ("command found but not executable — check permissions"), now you have to translate that into:

case $? in
  0)   echo "Success" ;;
  1)   echo "General error" ;;
  126) echo "Command found but not executable — check file permissions" ;;  # ← the one you just looked up
  127) echo "Command not found — check PATH or typo" ;;
  *)   echo "Unknown exit code: $?" ;;
esac

The export button generates that block for you, with the specific code you looked up highlighted in the right place and the explanation preserved as a comment. Paste it directly into your script. No rewriting the syntax from scratch.

The case template includes the four most common exit codes (0, 1, 126, 127) plus the wildcard catch-all. Delete the ones you don't need.

→ bashsnippets.xyz/tools/bash-exit-code-lookup

3. Cron Job Builder — Next 5 Run Times

This is the one I wanted for myself.

Enter any cron expression — say, 0 3 * * 1-5 — and it now immediately shows the next five times it will fire:

Next run times for: 0 3 * * 1-5

1.  Mon Jun 09 2026  03:00:00
2.  Tue Jun 10 2026  03:00:00
3.  Wed Jun 11 2026  03:00:00
4.  Thu Jun 12 2026  03:00:00
5.  Fri Jun 13 2026  03:00:00

The problem this solves: cron expressions are easy to get subtly wrong. 0 3 * * * fires at 3am. But 3am what — your server's local time or UTC? And is your server set to UTC? And does */6 mean every 6 hours starting at midnight, or starting at the first minute it's defined? And does 0 9 * * 1 fire on Monday or Sunday, depending on which cron implementation counts week starts?

Without something that shows you the actual fire times, you add the job, wait until the next day, and find out whether your assumptions were right or wrong. If they were wrong, you change it and wait another day. It can take three or four days to confirm a cron expression fires when you want it to, purely because of iteration time.

Showing the next five run times eliminates that cycle entirely. You know immediately whether your expression does what you think it does. Change it, verify it, add it to your crontab — all in under a minute.

→ bashsnippets.xyz/tools/cron-job-builder

4. Chmod Calculator — World-Writable Warning and Live Symbolic Mirror

Two updates here bundled together because they're related.

World-writable warning:

Enabling any world-write bit (the w in the "others" column) now shows a warning banner:

⚠️ World-writable permissions mean any user on the system can modify this file or directory. This is rarely correct. Verify this is intentional.

This is the chmod 777 trap. I've seen chmod 777 applied as a "quick fix" for permission errors more times than I can count. It works immediately, which is why people do it. The fact that it means "literally every user and process on this system can write to this file" gets lost in the urgency of making the error go away.

The warning doesn't prevent you from setting world-writable permissions. It just makes sure you've seen the words "any user on the system can modify this" before you click confirm. If you intended that, the warning is just noise. If you didn't, it might save you.

Live symbolic ↔ octal mirror:

Every permission you configure now shows both forms simultaneously:

Octal:    chmod 755
Symbolic: chmod u=rwx,go=rx

Both update in real time as you toggle permissions. This is useful for two reasons: you see the octal code for when you need to type it quickly in a terminal, and you see the symbolic form for when you need to express the same permission in a script where the explicit form is easier to read and maintain. Neither is "more correct" — they're the same thing expressed two ways, and having both removes the mental step of converting between them.

→ bashsnippets.xyz/tools/chmod-permissions-builder

5. PATH Debugger — Duplicate Entry Detector

Duplicate entries in $PATH accumulate over years. Every time a tool's installer adds itself to your PATH, every time you add an entry to .bashrc, every time a package manager prepends its bin directory to your environment — the list grows. On machines that have been around for a while, or on developer laptops where tools get installed and removed and reinstalled, you can end up with /usr/local/bin listed four times.

Duplicate entries don't usually cause obvious breakage. The correct binary still runs. It just runs with slightly more PATH resolution overhead on every command, and the duplicate entries make it harder to reason about which version of a tool will actually be picked when you have multiple installed. If /usr/local/bin appears before /usr/bin, the local install wins. When it appears four times, you've lost track of the actual resolution order.

The debugger now flags repeated entries and generates a one-liner to deduplicate and export a clean PATH:

# Generated dedup command:
export PATH=$(echo "$PATH" | tr ':' '\n' | awk '!seen[$0]++' | tr '\n' ':' | sed 's/:$//')

That pipeline: converts PATH from colon-separated to newline-separated, uses awk to keep only the first occurrence of each entry, converts back to colon-separated, and strips the trailing colon. The resulting PATH has the same resolution order as the original but with all duplicates removed.

Add that line to the bottom of your .bashrc and your PATH will deduplicate itself on every new shell session.

→ bashsnippets.xyz/tools/path-debugger

6. ShellCheck Error Decoder — Inline Script Scanner

ShellCheck is the best static analysis tool for bash scripts. If you're writing bash and you're not running your scripts through ShellCheck, you're missing real bugs. I'll say that plainly.

The problem is that ShellCheck error codes (SC2086, SC2046, SC1091, etc.) are meaningful if you know what they mean and opaque if you don't. When ShellCheck tells you SC2086: Double quote to prevent globbing and word splitting, that makes sense. When it just gives you SC2086 in isolation, you have to look it up.

The inline script scanner adds a paste box where you put your bash script directly. The tool maps recognized SC error codes to the lines in your script where they appear — not in the abstract documentation, but in your specific code. You see the line, the SC code, and the explanation together.

Important caveat I want to be direct about: this is not a replacement for running actual ShellCheck. The real ShellCheck tool parses bash properly, handles edge cases, and catches issues that a pattern-matcher won't. If you're writing scripts that matter, install ShellCheck and run it directly:

# Install
apt install shellcheck       # Debian/Ubuntu
brew install shellcheck      # macOS

# Run
shellcheck yourscript.sh

What the decoder in BashSnippets tools is useful for: understanding what a specific SC code means in the context of your own code rather than in a reference page. It's a learning tool and a quick lookup, not a substitute for the real thing.

→ bashsnippets.xyz/tools/shellcheck-error-decoder

All tools are free, no login, no account required. The full tools index is at:

→ bashsnippets.xyz/tools

→ bashsnippets.xyz

I was handed a server with no docs and no idea what it was listening on. One command fixed that.

Tue, 09 Jun 2026 00:02:18 +0000

A client handed me SSH access to a server they'd been running for two years.

No documentation. No handoff notes. No "here's what's running and why." Just a root password in a LastPass share and a Slack message that said "it hosts our web app, let us know if you need anything."

I didn't know what the web app was. I didn't know what was installed. I didn't know what ports were open to the internet, what services were running, or what this machine had been doing quietly for 730 days before I touched it.

This is not unusual. This is actually most of the "inherited server" situations I've been in. The person who set it up is gone, or was a contractor, or was the founder who also did DevOps because someone had to, and the documentation lived entirely inside one person's head and left with them.

The first thing I do on an unfamiliar server is not grep logs or check running services. It's figure out what the machine is actually reachable on from the outside. Not what anyone says it's supposed to be doing. What it is doing. What ports are in LISTEN state, what process owns each one, and whether that process is bound to localhost or to every network interface on the machine.

I ran netstat -an first, because that's what I knew at the time. I got back 200 lines of connection state. TIME_WAIT entries for connections that had already closed. ESTABLISHED entries for active sessions. CLOSE_WAIT entries from something that hadn't cleaned up properly. All of it technically useful for debugging specific connection issues. None of it useful for getting a fast security picture of what's actually listening for new connections.

Finding the ports in LISTEN state in netstat -an output feels like searching for a specific ingredient in a paragraph of text. It's all there, but you have to read every line.

Then someone showed me ss.

The Command

ss -tlnp

That's the whole thing.

-t — TCP only (use -u for UDP, or -tu for both)
-l — listening sockets only (filters out ESTABLISHED, TIME_WAIT, everything else)
-n — numeric output (don't resolve port numbers to service names, don't do reverse DNS)
-p — show the process name and PID holding each socket

Output on a typical web server:

State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port  Process
LISTEN  0       128     0.0.0.0:22           0.0.0.0:*          users:(("sshd",pid=1234,fd=3))
LISTEN  0       511     0.0.0.0:80           0.0.0.0:*          users:(("nginx",pid=5678,fd=6))
LISTEN  0       511     0.0.0.0:443          0.0.0.0:*          users:(("nginx",pid=5678,fd=7))
LISTEN  0       128     0.0.0.0:5432         0.0.0.0:*          users:(("postgres",pid=9012,fd=5))

Port 22 — SSH. Expected.
Port 80, 443 — nginx. Expected.
Port 5432 — Postgres. Bound to 0.0.0.0.

That last one.

0.0.0.0:5432 means Postgres was accepting connections from any IP address on the internet. Not just localhost. Not just the application server. Any IP. Port 5432, wide open, reachable from anywhere.

It had a strong password. It had been running that way for two years without an incident anyone knew about. But "strong password" is not a substitute for "not exposed to the internet." A service that should only accept connections from localhost or from your application tier has no business being bound to 0.0.0.0.

One command. Three seconds. Caught a misconfiguration that had been sitting there since the server was provisioned.

What the Output Is Actually Telling You

The column that matters is Local Address. This is where the socket is bound — who it will accept connections from.

0.0.0.0:5432    → Externally reachable on all IPv4 interfaces
127.0.0.1:5432  → Localhost only — not reachable from outside
:::5432         → All IPv6 interfaces (equivalent to 0.0.0.0 for IPv6)
::1:5432        → IPv6 localhost only

If a service should only be accessed from the same machine — a database, an internal cache, a monitoring agent — it should be bound to 127.0.0.1. If you see it bound to 0.0.0.0 and you don't have an explicit reason for that, that's a conversation to have with whoever configured the service.

For the Postgres case, the fix is one line in postgresql.conf:

listen_addresses = 'localhost'

Restart Postgres, confirm the bind address changed with ss -tlnp again, done. That configuration change is a 30-second fix. The two years it had been wrong before someone checked is the part that should concern you.

Why `ss` Instead of `netstat`

ss replaced netstat on most Linux distributions when iproute2 superseded net-tools around 2016. On a fresh Ubuntu or Debian install, netstat isn't installed by default — it's in the net-tools package which isn't pulled in automatically anymore. That's why you've probably SSH'd into a server, typed netstat, and gotten command not found.

ss reads kernel socket tables directly instead of going through /proc/net/tcp. It's faster on machines with thousands of connections, and it's maintained. net-tools has been in maintenance-only mode for years. For new work, use ss.

That said, netstat -tlnp does the same thing if you have it installed. The flags are identical. If you're on an older system or have net-tools available, either works.

The Process Name Caveat

Run ss -tlnp without sudo and you'll see the ports and bind addresses, but the Process column will be empty for sockets owned by other users. You'll see the port, you won't see what's holding it.

sudo ss -tlnp

With sudo, you see everything — the socket, the process name, the PID, the file descriptor number. That's the version to run when you're doing a security audit on a machine you have root on.

The `lsof` Alternative

lsof -i -P -n | grep LISTEN gives you the same information from a different angle — process name first, port second.

nginx    5678 root   6u  IPv4  12345  0t0  TCP *:80 (LISTEN)
nginx    5678 root   7u  IPv4  12346  0t0  TCP *:443 (LISTEN)
sshd     1234 root   3u  IPv4  12347  0t0  TCP *:22 (LISTEN)
postgres 9012 postgres 5u IPv4 12348  0t0  TCP *:5432 (LISTEN)

Useful when you know the service name and want to find its ports. Less useful when you want a port-first view of everything listening. I use ss for the initial audit and lsof when I'm tracking down a specific process.

What I Actually Do With Ports I Can't Identify

When I see a port in LISTEN state and the process name isn't immediately obvious — java, python3, something that isn't a clear service name — I do three things:

# 1. Find the full command that started the process
ps aux | grep <PID>

# 2. Check what package installed the binary
dpkg -S /path/to/binary    # Debian/Ubuntu
rpm -qf /path/to/binary    # RHEL/CentOS

# 3. Check what the process has open (files, connections, everything)
sudo lsof -p <PID>

Most of the time it's something benign — a monitoring agent, a language runtime for an app, a service the previous admin installed and forgot about. Occasionally it's something that shouldn't be there at all. You don't know until you check.

Building This Into Your Server Checklist

Every time I take over a server now, this is the third command I run. After uptime (how long has it been running) and df -h (is it about to run out of disk), it's sudo ss -tlnp (what is this machine telling the internet it's listening on).

Takes three seconds. Has caught real problems more than once. The Postgres 0.0.0.0 situation is the one I remember most clearly, but it's not the only one — I've found web apps listening on non-standard ports that were supposed to be internal, Redis instances bound to 0.0.0.0 on development machines that got promoted to production without anyone auditing the config, SSH running on a second non-standard port "for convenience" that had been left open after a migration.

None of those were catastrophic on their own. All of them were things the people running those servers didn't know were there.

Three seconds to find out. I don't know why I didn't run this on every server from the beginning.

Full script with UDP ports, both TCP and UDP in one pass, lsof cross-reference, and notes on what to do when you can't identify a process:

→ bashsnippets.xyz/snippets/list-open-ports-linux

→ bashsnippets.xyz

certbot had been auto-renewing my certs for two years. Until it wasn't.

Sun, 07 Jun 2026 05:04:36 +0000

certbot had been running quietly on my server for almost two years without a single issue.

Automatic renewals. Silent cron job. I never thought about it. Set it up once, tested it once, and mentally moved it to the "solved" column of my infrastructure checklist. That column felt good. certbot goes in there and stays there.

Then I checked my site on a Monday morning and got the red browser warning.

Not "connection is slow." Not a timeout. The full-page block. Chrome's red lock icon. "Your connection is not private." NET::ERR_CERT_DATE_INVALID in small gray text underneath, in case you wanted the clinical version of bad news.

I SSHd in immediately. The cert had expired three days earlier.

certbot had been running, technically. The cron job was firing. No errors in /var/log/syslog that I was watching. But the HTTP challenge was failing silently — something to do with a port conflict during renewal. A service I'd added a few months back was binding to port 80 for its own health check, and certbot couldn't complete the ACME challenge because port 80 wasn't free during the brief window it needed it. The renewal attempt failed. certbot logged it somewhere I wasn't watching. The cron job reported success because the cron job ran — it just happened to run a certbot process that quietly gave up.

No alert email. No log message in any file I had on my radar. Just a quiet failure, three renewal cycles in a row, and then an expired certificate.

I found out from a user who emailed me. Not from monitoring. Not from a script. From a stranger who was kind enough to say "hey, your SSL is broken" instead of just closing the tab.

The renewal fix took five minutes once I found the port conflict. What I didn't have — what I wish I'd had — was a script that told me the cert was about to expire before it happened. Something running every morning, checking the actual live certificate the way a browser would check it, and reporting "you have 18 days left — go fix this."

Turns out openssl can do this in one command. It's been able to do this for years. I just never looked.

The Core Command

echo | openssl s_client -connect yoursite.com:443 -servername yoursite.com \
  2>/dev/null | openssl x509 -noout -enddate

Run that and you get back something like:

notAfter=Aug 14 12:00:00 2026 GMT

That's the raw expiry date straight from the live certificate your server is presenting to the world. Not what certbot thinks. Not what your local files say. What a browser actually sees when it connects.

To turn that into days remaining:

EXPIRY=$(echo | openssl s_client -connect yoursite.com:443 -servername yoursite.com \
  2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)

DAYS=$(( ($(date -d "$EXPIRY" +%s) - $(date +%s)) / 86400 ))

echo "$DAYS days remaining on SSL cert"

That arithmetic converts two Unix timestamps (the expiry date and now) to seconds, subtracts them, and divides by 86400 to get days. It's ugly but it works.

Three Non-Obvious Things I Ran Into

1. The -servername flag is not optional on SNI hosts.

SNI — Server Name Indication — is how a single IP address serves multiple domains with different SSL certificates. Most shared hosting and most modern VPS setups use it. Without -servername yoursite.com, openssl doesn't know which certificate to request. It reads the server's default certificate instead of yours.

I spent twenty minutes getting clean results from this command before I realized I was checking the wrong cert. My server's default certificate was still valid. My production domain's certificate was the one expiring. I would have gotten a false "everything is fine" reading and missed the problem entirely. Always include -servername.

2. The echo | pipe is required for non-interactive use.

Without it, openssl s_client waits for stdin input after establishing the connection. In a cron job, that means the script hangs forever, waiting for input that never comes, holding the connection open until something kills it. The echo sends empty input immediately, which closes the connection after the certificate handshake completes. This is one of those things where the command works perfectly in your terminal and completely silently breaks in cron. The echo | is what makes it safe to automate.

3. The date -d syntax is Linux-only.

macOS uses date -j -f "%b %d %T %Y %Z" "$EXPIRY" +%s instead of date -d "$EXPIRY" +%s. If you're building a script that needs to run on both platforms, you'll need a conditional. The full version of the script at the link below handles this with an OS check. If you're only ever running this on Linux servers — which is probably most of you — you don't need to care about this.

The Full Script

#!/bin/bash

CHECK="✓"
CROSS="✗"

# --- Configuration ---
DOMAINS=(
  "yourdomain.com"
  "anotherdomain.com"
)
WARN_DAYS=30      # Alert if fewer than this many days remain
PORT=443

# --- Check each domain ---
for DOMAIN in "${DOMAINS[@]}"; do
  EXPIRY=$(echo | openssl s_client \
    -connect "${DOMAIN}:${PORT}" \
    -servername "$DOMAIN" \
    2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)

  if [[ -z "$EXPIRY" ]]; then
    echo "$CROSS Could not retrieve cert for $DOMAIN — is the server reachable?"
    continue
  fi

  EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s 2>/dev/null)
  NOW_EPOCH=$(date +%s)
  DAYS=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))

  if [[ "$DAYS" -lt 0 ]]; then
    echo "$CROSS $DOMAIN — CERT EXPIRED ${DAYS#-} days ago"
  elif [[ "$DAYS" -lt "$WARN_DAYS" ]]; then
    echo "$CROSS $DOMAIN — WARNING: $DAYS days remaining (expires $EXPIRY)"
  else
    echo "$CHECK $DOMAIN — OK: $DAYS days remaining"
  fi
done

Sample output with two domains:

✓ yourdomain.com — OK: 74 days remaining
✗ anotherdomain.com — WARNING: 11 days remaining (expires Sep 2 12:00:00 2026 GMT)

Why 30 Days as the Threshold

Let's Encrypt certificates are 90 days. certbot typically renews at 60 days remaining — 30 days before the default renewal window. If my monitoring fires at 30 days, I have a full renewal cycle's worth of time to figure out whatever certbot is failing on before anything breaks.

That's the math. 30 days is not conservative for its own sake — it's exactly one full renewal attempt window on a 90-day cert. If you're using a 1-year cert from a paid CA, you probably want 60 days. If you're on Let's Encrypt, 30 days gives you two full automated renewal attempts before you hit zero.

Adding an Email Alert

If you want to be notified instead of (or in addition to) just logging, add this to the warning block:

if [[ "$DAYS" -lt "$WARN_DAYS" ]]; then
  echo "SSL cert for $DOMAIN expires in $DAYS days" | \
    mail -s "CERT EXPIRY WARNING: $DOMAIN" *Emails are not allowed*
fi

This requires mailutils or sendmail to be configured on your server. If you don't have email set up, the cron log version is enough — as long as you actually read the log, which is its own challenge.

The Cron Setup

0 8 * * * /home/user/check-ssl-expiry.sh >> /var/log/ssl-check.log 2>&1

Every morning at 8am. Logs to /var/log/ssl-check.log. If something fires, the line is there when you check in the morning. If everything is green, the log file is a quiet record that your certs are healthy.

I run this across all my domains every day. 30-day warning threshold. On a Let's Encrypt setup that gives two full auto-renewal cycles before anything breaks. No more Monday morning surprises.

The cert that expired on me was a $0 certificate on a $5 server. The cost wasn't the problem. The embarrassment of a user telling me before my own monitoring did — that was the part I wasn't willing to repeat.

Full script with multi-domain array, configurable threshold, email alert variation, macOS compatibility, and cron setup instructions:

→ bashsnippets.xyz/snippets/check-ssl-certificate-expiry

→ bashsnippets.xyz

set -euo pipefail — the Line That Would Have Saved Me from Deleting the Wrong Directory

Wed, 03 Jun 2026 21:32:20 +0000

Here's a script that will ruin your day:

#!/bin/bash
cd /nonexistent/folder
rm -rf *
echo "Done"

The cd fails because the folder doesn't exist. Bash ignores the failure. rm -rf * runs in whatever directory you were already in. The script prints "Done." You have no idea anything went wrong until you notice your files are gone.

This isn't a contrived example. This is the actual failure mode of every bash script that doesn't have error handling. Bash's default behavior is to keep running after errors. Every command after a failure executes as if everything is fine. It's the most dangerous default in the entire language.

Two lines fix it.

The Template

#!/bin/bash
set -euo pipefail

trap 'echo "Error on line $LINENO — script stopped." >&2' ERR

# Your script starts here

Add this to the top of every script you write. Every single one. No exceptions.

What Each Flag Does

-e (errexit) — Stop the script immediately if any command exits with a non-zero status. This is the one that prevents the cd + rm disaster above. If cd fails, the script stops right there. rm never runs.

-u (nounset) — Treat any undefined variable as an error. Without this, $UNDEFINED_VAR silently becomes an empty string. Imagine this:

BACKUP_DIR=""  # Oops, forgot to set this
rm -rf "$BACKUP_DIR"/*

That expands to rm -rf /*. That's your entire filesystem. With -u, bash catches the empty variable and stops before the rm ever runs.

-o pipefail — Make a pipeline fail if ANY command in it fails, not just the last one. Without this:

cat nonexistent-file.txt | grep "pattern" | wc -l

cat fails, but grep gets empty input, and wc -l counts zero lines and exits successfully. The pipeline reports success even though the first command failed. With pipefail, the pipeline's exit status is the failure from cat, so -e catches it.

The `trap` Line

trap 'echo "Error on line $LINENO — script stopped." >&2' ERR

This tells bash: "When an error happens, before you exit, run this command." It prints which line number failed and writes to stderr (>&2). Without this, the script just stops silently and you have to guess which line caused it.

In a 200-line script, "it failed somewhere" is useless. "Error on line 47" is actionable.

The Real-World Difference

Without error handling:

#!/bin/bash
cd /tmp/build
make
make install
echo "Build complete"

If cd fails (the directory doesn't exist), make runs in the wrong directory. If make fails (compilation error), make install installs whatever was there from last time. The script prints "Build complete" regardless.

With error handling:

#!/bin/bash
set -euo pipefail
trap 'echo "Error on line $LINENO" >&2' ERR

cd /tmp/build
make
make install
echo "Build complete"

If cd fails, the script stops and prints "Error on line 5." If make fails, same thing. echo "Build complete" only runs if every single command before it succeeded.

The One Gotcha

set -e changes how you write conditional logic. This fails:

set -e
grep "pattern" file.txt  # exits 1 if pattern not found
echo "This never runs"

grep returns exit code 1 when it finds no matches. With -e, that's treated as an error and the script stops. But you didn't want it to stop — you just wanted to check if the pattern exists.

The fix is to use it in a conditional:

set -e
if grep -q "pattern" file.txt; then
  echo "Found it"
else
  echo "Not found"
fi

When a command is part of an if condition, -e doesn't trigger on its exit code. This is the standard pattern.

Or suppress the exit code explicitly:

grep "pattern" file.txt || true

The || true means "if grep fails, run true instead" — and true always succeeds, so -e doesn't trigger.

Where to Go from Here

Every script on BashSnippets.xyz uses set -euo pipefail and the CHECK="✓" / CROSS="✗" pattern. If you want a head start, the Bash Boilerplate Generator builds a complete script template with error handling, logging, and cleanup traps already configured. Pick your options, copy the output, and start writing from a safe foundation.

Full breakdown with more examples, the trap pattern, and common pitfalls:

bashsnippets.xyz/snippets/bash-error-handling.html

If you only take one thing from this: add set -euo pipefail to the top of every script. It takes 3 seconds and prevents the kind of failures that take hours to recover from.

grep -rn Changed How I Debug. I Use It Before I Open a Single File.

Mon, 01 Jun 2026 01:41:05 +0000

I used to open files to search for things.

A config had the wrong database host somewhere. I knew it was in one of the files in /etc/nginx/ but I didn't know which one. So I opened nginx.conf. Searched. Not there. Opened sites-available/default. Searched. Not there either. Opened three more files. Found it in the fourth one.

That took about 4 minutes. The grep command that does the same thing takes less than a second:

grep -rn "db.old-host.com" /etc/nginx/

Every file. Every subdirectory. Exact filename. Exact line number. Done.

I genuinely don't understand why it took me so long to make this my default workflow instead of manually opening files like it's 2004.

The Script

#!/bin/bash
# Search files for text — grep reference

KEYWORD="TODO"          # ← what to search for
SEARCH_DIR="$HOME"      # ← where to search
FILE_TYPE="*.txt"       # ← file types to include

echo "Searching for '$KEYWORD' in $SEARCH_DIR..."
echo "────────────────────────────────────────"

grep -rn "$KEYWORD" "$SEARCH_DIR" \
  --include="$FILE_TYPE" \
  --color=auto

COUNT=$(grep -rc "$KEYWORD" "$SEARCH_DIR" \
  --include="$FILE_TYPE" 2>/dev/null \
  | grep -v ":0$" | wc -l)

echo "────────────────────────────────────────"
echo "Found in $COUNT file(s)"

But honestly, you'll use the raw command more than the script. The script is a reference. The command is the weapon.

The Flags That Actually Matter

grep -r — recursive. Searches every file in the directory and all subdirectories. Without this, grep only searches the one file you hand it.

grep -n — line numbers. Instead of just showing the matching line, it shows filename:42: the matching line. Now you know exactly where to look.

grep -rn — both together. This is the one I type the most. grep -rn "ERROR" /var/log/ and I immediately see every error across every log file with the exact line number.

*`--include=".py"** — filter by file type. If you only want to search Python files, or only .conf files, or only .html` files, this prevents grep from wasting time on binary files and irrelevant formats.

--color=auto — highlights the matched keyword in the output. When you're scanning 50 results, the color makes the keyword jump out instantly.

-i — case-insensitive. grep -rni "error" /var/log/ catches "Error", "ERROR", "error", and any other capitalization.

-l — filenames only. When you don't need to see the matching lines, just which files contain the keyword. grep -rl "TODO" ./src/ gives you a clean list of files.

-c — count matches per file. grep -rc "TODO" ./src/ shows filename:3 meaning 3 matches in that file.

The Combinations I Use Every Week

Find every TODO in a project:

grep -rn "TODO" ./src/ --include="*.js"

Find which config file has a specific setting:

grep -rn "max_connections" /etc/

Find a function definition across a codebase:

grep -rn "def process_order" ./app/ --include="*.py"

Find IP addresses in log files:

grep -rn "192.168" /var/log/ --include="*.log"

Find files that contain a string but don't care about the matching lines:

grep -rl "database_url" ./config/

grep vs. Your Editor's Search

Your editor's search (Ctrl+Shift+F in VS Code, Cmd+Shift+F on Mac) does basically the same thing — but only for files that are open in your project.

grep works on any directory on any machine, including servers where you only have SSH access and no editor at all. When you're debugging a production issue over SSH at 11 PM, grep -rn is the fastest path from "I know the error is in here somewhere" to the exact line.

It's also faster on large directories because it doesn't need to index anything first. It just reads files sequentially and matches the pattern.

One Gotcha Worth Knowing

grep on binary files produces garbage output. If you're searching a directory that contains images, compiled files, or databases, add --binary-files=without-match to skip them:

grep -rn "search_term" /some/dir/ --binary-files=without-match

Or use --include to limit to specific text file types, which solves the same problem more precisely.

Full script, flag cheat sheet, and more combinations:

bashsnippets.xyz/snippets/search-files-for-text-grep.html

If you want to quickly look up the flags without reading a man page, I keep a reference table on the snippet page with every flag I've actually used in production.