Coder Legion - Recent posts tagged node

Node 22.22 runs TypeScript. Ubuntu's Node 22.22 doesn't.

Tue, 27 Oct 2026 14:27:58 +0000

Four test files in my site's repo import TypeScript directly. faq.test.mjs pulls in src/lib/faq.ts, and three others do the same with their own modules. There is no build step and no loader in between: the script is node --test "test/*.test.mjs", and type stripping has been on by default since Node 22.18.0, so Node strips the types and runs what's left.

In CI those tests pass. The workflow asks actions/setup-node for node-version: "22", and the latest runs on main are green.

On my Ubuntu server, with Node 22.22.1, all four fail before a single assertion runs:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for .../src/lib/faq.ts

22.22.1 is newer than 22.18.0, so on paper the feature is there. The version number was right. The binary behind it was a different build.

Same version, different binary

Node will tell you whether it strips types. The v22 docs describe process.features.typescript as "strip" by default, "transform" with --experimental-transform-types, and false if Node is started with --no-experimental-strip-types.

I did not pass that flag:

$ /usr/bin/node --version
v22.22.1
$ /usr/bin/node -p 'process.features.typescript'
false

That node comes from Ubuntu's nodejs package, version 22.22.1+dfsg+~cs22.19.15-1ubuntu1. Asking for the feature explicitly does not bring it back:

$ /usr/bin/node --experimental-strip-types t.ts
node:internal/util:226
    throw new ERR_NO_TYPESCRIPT();

To rule out the version itself, I downloaded the official 22.22.1 build from nodejs.org, checked it against SHASUMS256.txt, and asked the same question. It printed strip and ran the same .ts file without complaint.

Where TypeScript went

The package's own changelog explains most of it. The December 2024 entry, 22.12.0+dfsg-1, says:

dfsg-exclude amaro, build without it - it requires swc. This disables the ability to execute TypeScript files using the --experimental-strip-types flag.

The November 2025 entry, 22.21.1+dfsg+~cs22.19.0-1, says: "we build without-amaro for now, disable strip-types".

The March 2026 Debian entry, merged into the version I have, says: "Drop 'no amaro' patch, solved upstream". The build still answers false, and it reports how it was compiled:

$ /usr/bin/node -p 'process.config.variables.node_use_amaro'
false

The official 22.22.1 build prints true.

The changelog doesn't say what "solved upstream" changed. My tests only see what the build does.

Why CI never saw it

The setup-node README says the action first checks the runner's tool cache, then pulls LTS versions from the actions/node-versions releases and, on a miss, falls back to downloading from nodejs.org. None of those is Ubuntu's package. The four test files pass in CI, so the Node there strips types. The one on my server does not, and both report major version 22.

That is how one repository can be green in CI and red on my server without a line of code changing.

The fix

I extracted the official Node 22.23.2 tarball into /usr/local. /usr/local/bin comes before /usr/bin in PATH, so node now resolves to the official build, with npm 10.9.8 bundled.

I left the Ubuntu package installed. On this machine 15 installed packages depend on nodejs: handlebars and 14 node-* libraries. Removing it to get the right node would have meant fighting apt over things that have nothing to do with my site.

Same four files, official binary:

# tests 42
# pass 42
# fail 0

Ask Node what it can do

If a project relies on type stripping, node --version is not the check. This is:

node -p 'process.features.typescript'

On the two builds above it prints strip when the feature works and false when it doesn't, whatever the version string says. It belongs in a setup script next to the version check, because on my server the version check was the one that passed.

Building a Booking System with Node.js, Google Calendar, and Google Meet

Tue, 27 Oct 2026 07:58:12 +0000

Introduction

Booking a consultation should be straightforward for the person making the appointment. Behind the scenes, however, an automated booking system has to coordinate several moving parts: collecting customer information, checking availability, scheduling an event, generating a meeting link, saving the booking, and notifying the people involved.

I encountered these challenges while developing the booking system for JOPAD Advisory. I built the system from scratch, using Node.js and Express for the backend, MongoDB for storing bookings, Google Calendar for scheduling, Google Meet for virtual meetings, and Resend for email notifications.

In this article, I'll walk through how I implemented the booking workflow, how the different services work together, and some of the engineering considerations that come with connecting external APIs to a backend application.

1. The Architecture

The backend coordinates the booking process. Each service has a specific responsibility:

Node.js and Express: Handle booking requests and coordinate the workflow.
MongoDB and Mongoose: Store booking details and provide access to existing appointments.
Google Calendar API: Create calendar events for scheduled sessions.
Google Meet: Generate a virtual meeting link through the Calendar event's conference configuration.
Airtable: Receive a secondary record of the booking.
Resend: Power the email notification workflow.

The booking process follows this sequence:

A customer submits a booking request through the website.
The Express endpoint validates the submitted fields.
The backend calculates the appointment's start and end times.
Existing bookings are checked for overlapping time intervals.
Google Calendar creates an event and requests a Google Meet conference.
The booking is saved to MongoDB with its meeting link.
The backend attempts to send a secondary record to Airtable.
Booking notification emails are triggered.

This design brings several services together behind a single booking endpoint, giving the frontend one place to submit the request.

2. Building the Booking Endpoint

I used an Express POST endpoint to receive booking requests. The request contains the appointment time, duration, customer's contact details, session description, and other booking information.

The first step is to reject incomplete requests before performing external API operations.

A simplified version of the validation looks like this:

router.post("/book", async (req, res) => {
  const {
    time,
    duration,
    email,
    firstName,
    lastName,
    description
  } = req.body;

  if (
    !time ||
    !duration ||
    !email ||
    !firstName ||
    !lastName ||
    !description
  ) {
    return res.status(400).json({
      message: "Missing required fields"
    });
  }

  // Continue with availability checks
  // and booking creation.
});

This is a simplified illustration of the validation logic in my implementation, not a complete replacement for the original endpoint.

The actual endpoint validates additional fields, including the selected date, day, time range, phone number, and referral information.

The important principle is to validate the data required by the booking workflow before creating external resources or writing records.

For a production system, request validation should also verify data types, valid email addresses, positive durations, valid dates, and acceptable appointment times. A field being present does not necessarily mean its value is valid.

3. Calculating Appointment Duration

A booking needs both a start time and an end time. The end time determines the appointment's actual interval and makes it possible to detect overlapping bookings.

In my implementation, the backend calculates the end time using the requested start time and duration:

const startTime = new Date(time);
const endTime = new Date(startTime);

endTime.setMinutes(
  endTime.getMinutes() + Number(duration)
);

For example, an appointment starting at 10:00 AM with a duration of 60 minutes should end at 11:00 AM.

Calculating the interval on the backend is important because scheduling decisions should not depend solely on values calculated by the browser.

It is also important to handle time zones consistently. My Google Calendar integration sends ISO-formatted timestamps to the API. A more comprehensive implementation should explicitly define the application's scheduling time zone and validate incoming timestamps to prevent ambiguity around local times and daylight-saving transitions.

4. Preventing Overlapping Bookings

One of the central problems in a booking system is preventing customers from selecting times that have already been reserved.

Two appointments overlap when the first appointment starts before the second one ends and finishes after the second one starts.

The logic can be expressed as:

const overlaps =
  requestedStart < existingEnd &&
  requestedEnd > existingStart;

This condition also allows back-to-back appointments. For example, a booking ending at 11:00 AM does not conflict with another booking beginning at exactly 11:00 AM.

In my implementation, I retrieve existing bookings from MongoDB, calculate each booking's end time from its start time and duration, and check for conflicts:

const existingBookings = await Booking.find();

const conflict = existingBookings.some(booking => {
  const existingStart = new Date(booking.startTime);
  const existingEnd = new Date(existingStart);

  existingEnd.setMinutes(
    existingEnd.getMinutes() +
      Number(booking.duration)
  );

  return (
    startTime < existingEnd &&
    endTime > existingStart
  );
});

if (conflict) {
  return res.status(409).json({
    message: "This time slot has already been booked"
  });
}`

Returning HTTP "409 Conflict" communicates that the request conflicts with an existing booking.

This approach is easy to understand, but it has a scalability limitation: the implementation loads all existing bookings before checking for an overlap. As the collection grows, a database query restricted to potentially overlapping intervals would be more efficient.

There is also an important concurrency consideration. Two requests can check availability at almost the same time, both find the slot free, and both continue. The check alone does not guarantee that simultaneous requests cannot create overlapping bookings.

A stronger production design would use an atomic reservation strategy, such as reserving discrete time slots with a unique database constraint or using another concurrency-control mechanism. That protection needs to be designed around the application's scheduling rules rather than relying only on an initial availability check.

5. Connecting Google Calendar and Google Meet

After the backend confirms that a requested slot does not conflict with an existing booking, it calls a dedicated utility function to create the meeting.

I separated this logic into a Google Calendar utility instead of placing the entire integration directly inside the Express route. This keeps the route focused on the booking workflow and makes the external integration easier to maintain.

Authenticating with Google

The utility initializes an OAuth 2.0 client using environment variables and sets a refresh token:

const { google } = require("googleapis");

const oauth2Client = new google.auth.OAuth2(
  process.env.GOOGLE_CLIENT_ID,
  process.env.GOOGLE_CLIENT_SECRET,
  process.env.GOOGLE_REDIRECT_URI
);

oauth2Client.setCredentials({
  refresh_token: process.env.GOOGLE_REFRESH_TOKEN
});

The example uses the conventional environment-variable spelling for the client secret. In the project, the environment variable is named "GOOGLE_CLIENT_SECRETE", so the deployed configuration must match the actual code unless that naming is corrected.

The refresh token allows the server-side client to obtain access tokens as needed without requiring the user to repeat the authorization flow for every booking.

This requires the initial OAuth authorization to have been completed successfully and the refresh token to remain valid. The Google account must also have the necessary permissions and configuration for creating Calendar events and conference links.

Credentials should be stored securely in environment variables or an appropriate secrets manager, never committed to a public repository.

Creating a Calendar event

The integration uses Google's Calendar API:

const calendar = google.calendar({
  version: "v3",
  auth: oauth2Client
});

The meeting function calculates the event's end time and constructs a Calendar event containing the session title, description, start and end timestamps, and attendee information.

The attendee is represented by the customer's email address:

attendees: [
  {
    email,
    responseStatus: "accepted"
  }
]

The event also includes settings that prevent guests from modifying the event, inviting other guests, or seeing other guests.

These settings are part of the event configuration; they do not replace proper authorization and access control in the booking application itself.

Requesting a Google Meet link

The event configuration includes a conference creation request:

conferenceData: {
  createRequest: {
    requestId: "jopad-" + Date.now(),
    conferenceSolutionKey: {
      type: "hangoutsMeet"
    }
  }
}

The event is then inserted into the primary Google Calendar:

const response = await calendar.events.insert({
  calendarId: "primary",
  resource: event,
  conferenceDataVersion: 1,
  sendUpdates: "all"
});

The "conferenceDataVersion: 1" option tells Google Calendar that conference data should be processed as part of the request. The "sendUpdates: "all"" option requests that Calendar send event updates to the relevant participants.

After the API call succeeds, the implementation extracts the meeting URL from the returned conference data:

const meetLink =
  response.data.conferenceData.entryPoints[0].uri;

return meetLink;

The booking route receives that link and includes it in the booking record.

One important implementation detail is that the code assumes the response contains a conference entry point. A more defensive implementation would verify that the response contains a usable Meet URL before continuing.

Google Calendar conference creation can also involve asynchronous processing. Production code should account for cases where the event exists but the conference details are not yet available, rather than assuming every successful event insertion immediately includes a meeting link.

6. Saving Bookings in MongoDB

Once the meeting link has been returned, the backend creates a Mongoose booking document.

The model stores information such as:

Appointment start time and duration.
Customer name and contact information.
Session description.
Date and time display values.
Referral information.
Google Meet link.
Record creation timestamp.

The central fields include:

const bookingSchema = new mongoose.Schema({
  startTime: {
    type: Date,
    required: true
  },
  duration: String,
  email: String,
  firstName: String,
  lastName: String,
  description: String,
  meetLink: String,
  createdAt: {
    type: Date,
    default: Date.now
  }
});

This is an excerpt of the schema; the complete project model contains additional fields.

After constructing the document, the route saves it:

await newBooking.save();

Saving the booking after generating the meeting link ensures that the returned booking record can contain the link associated with the created event.

However, the order of these operations creates an important failure scenario. If Google Calendar successfully creates the event but MongoDB subsequently fails to save the booking, a calendar event may remain without a corresponding booking record.

This is a common challenge when coordinating operations across independent services. A more resilient design could track workflow states, retry recoverable operations, and provide a reconciliation or cleanup process for partially completed bookings.

7. Adding an Airtable Backup

The system also attempts to send booking information to Airtable after saving the MongoDB document.

The request includes selected booking fields, such as the customer's name, email, phone number, session description, referral details, meeting link, and appointment date.

The operation is wrapped in its own "try...catch" block. If the Airtable request fails, the error is logged, but the main booking workflow can still continue.

This makes Airtable a secondary destination rather than the primary source of truth for the booking.

That distinction matters: the implementation attempts a backup write, but it does not guarantee that the MongoDB and Airtable records are always synchronized. If Airtable is unavailable, the backup attempt fails without automatically retrying it.

A more robust implementation could place failed backup operations into a retry queue or maintain a record of synchronization status.

8. Sending Booking Notifications

After the booking has been saved and the Airtable attempt has completed, the route triggers the email notification utility:

sendBookingEmails({
  email,
  firstName,
  lastName,
  dayString,
  referral,
  dateString,
  number,
  description,
  timeString,
  timeRange,
  meetLink
}).catch(err => {
  console.log("Email failed:", err.message);
});

The email operation is not awaited by the route. This allows the endpoint to return its successful booking response without waiting for the notification workflow to finish.

It also means an email failure does not change the response after the booking has been saved.

There is a trade-off: the current implementation logs email failures but does not automatically retry them or record a delivery status in the booking document. For a production booking service, a persistent background job or message queue would provide a stronger way to retry notifications and monitor delivery outcomes.

The route eventually responds with HTTP "201 Created", along with a success message and the saved booking.

9. Lessons from the Implementation

Building this workflow brought several backend engineering considerations into focus.

Validate before creating external resources. Checking the request and current bookings before calling Google reduces unnecessary API operations.

Separate responsibilities. Keeping the Calendar integration in its own utility makes the Express route easier to understand and maintain.

Handle partial failures. MongoDB, Google Calendar, Airtable, and an email provider are independent services. A successful operation in one does not guarantee success in the others.

Treat concurrency as a separate problem. Checking for overlapping bookings is necessary, but it does not by itself protect against two simultaneous requests reserving the same slot.

Design for observability and recovery. Logging an error helps with debugging, but production systems also benefit from structured logs, retry mechanisms, delivery tracking, and reconciliation workflows.

These considerations extend beyond booking applications. Any backend that coordinates a database with multiple external APIs needs a plan for partial failures and retries.

Conclusion

Building the JOPAD Advisory booking system from scratch gave me practical experience connecting a Node.js backend to Google Calendar, generating Google Meet links, storing bookings in MongoDB, maintaining a secondary Airtable record, and triggering email notifications.

The most important lesson is that integrating APIs is only one part of building a reliable application. The surrounding workflow—validation, conflict detection, persistence, failure handling, and recovery—is just as important.

A booking endpoint can appear simple from the frontend, but it coordinates several systems that can succeed or fail independently. Understanding those interactions is what turns an API integration into a more complete engineering solution.

I'm continuing to explore ways to make these workflows more resilient, especially around concurrent bookings, retries, and recovery from partial failures.

If you're building a booking platform or integrating third-party services into a Node.js application, I'd be interested to hear how you approach these challenges.

Building Production Backends with NestJS

Mon, 26 Oct 2026 19:06:12 +0000

My focus for this month of October will be on NestJS framework and how it has imparted my development experience as a fullstack software engineer. NestJS provides a structured framework for building Node.js applications, but the framework alone does not determine whether an application will be maintainable in production.

The architecture surrounding the framework matters a lot. I will be sharing overview on my opinions when developing a secure and scalable systems in NestJS.

Feature-oriented modules

I prefer organizing backend systems around business capabilities rather than technical file types.

For example:

src/
├── modules/
│   ├── users/
│   ├── authentication/
│   ├── payments/
│   └── notifications/
│
├── infrastructure/
├── shared/
└── main.ts

This keeps related business functionality together.

Dependency boundaries

Business logic should not directly depend on infrastructure details.

For example, application code should depend on an abstraction:

export interface EmailSender {
  send(message: EmailMessage): Promise<void>;
}

rather than directly depending on a specific provider.

The infrastructure layer can then implement it:

export class ResendEmailSender implements EmailSender {
  async send(message: EmailMessage): Promise<void> {
    // provider implementation
  }
}

This makes it possible to replace infrastructure without rewriting business logic.

The goal

The goal is not to create the largest possible architecture.

The goal is to create a system where important decisions remain easy to change.

That principle becomes increasingly important as an application grows.

Limpando ~20 GB de caches de Terraform e Node.js com um único comando Bash

Mon, 26 Oct 2026 18:03:24 +0000

Mantenho todos os meus repositórios em uma única pasta — cerca de 150, entre trabalho para clientes, templates e experimentos. Cada um deles é uma pequena armadilha: um node_modules dentro de app/ e um .terraform dentro de infra/terraform/. Nenhum dos dois é versionado, ambos são reconstruídos em segundos (bem, minutos), e juntos tinham comido mais de 20 GB do meu disco sem que eu percebesse.

O gatilho foi banal: df -h mostrando 94% de uso numa partição de 233 GB, com 10 GB livres. A correção também deveria ser banal. Em vez de rodar um find ... -exec rm -rf do qual eu me arrependeria, escrevi um comando pequeno em que eu pudesse confiar — e mantive.

Este artigo traz o script, o raciocínio por trás de cada proteção e o que aprendi ao rodá-lo.

O objetivo

Um comando, de qualquer lugar, que:

Varra ~/Documentos/github (em qualquer profundidade — incluindo monorepos e pastas agrupadas).
Encontre apenas dois tipos de diretório:
- */infra/terraform/.terraform
- */app/node_modules
Informe o tamanho de cada um e os totais.
Delete esses diretórios — mas somente quando isso for pedido explicitamente.

Decisões de projeto

1. Dry-run como padrão: apagar cache de build é reversível, mas executar a operação em 30 diretórios distribuídos por 150 repositórios exige cautela. O parâmetro --delete é obrigatório para efetivar as exclusões.

2. find -prune em vez de du recursivo: evita percorrer diretórios como .git, node_modules e .terraform, mantendo a varredura eficiente mesmo com milhares de arquivos.

3. Whitelist de caminhos antes de apagar: o rm -rf só é executado em caminhos que correspondem exatamente aos sufixos permitidos e estão dentro da raiz varrida. Qualquer caminho fora dessas regras é ignorado e reportado.

4. Medição de tamanho em paralelo: executa o du em paralelo usando xargs -P "$(nproc)", aproveitando os núcleos disponíveis e reduzindo o tempo de execução em árvores grandes.

5. Saída legível, cálculos precisos: os tamanhos são somados em bytes e formatados com numfmt, evitando divergências causadas pela soma de valores arredondados.

6. Espaço livre antes e depois: compara o espaço disponível no disco antes e depois da limpeza para verificar o ganho real obtido.

Um detalhe que vale destacar: .terraform.lock.hcl é um arquivo e não é tocado. Ele fixa as versões dos providers e deve ser versionado. Apenas o diretório .terraform/ (cache de providers e módulos) é removido.

O script

Salve como limpar-cache, dê permissão de execução e coloque em /usr/local/bin:

#!/usr/bin/env bash
#
# limpar-cache — varre um workspace e levanta/remove:
#   * <projeto>/infra/terraform/.terraform
#   * <projeto>/app/node_modules
#
# Uso:
#   limpar-cache                 # só relatório (dry-run, nada é apagado)
#   limpar-cache --delete        # apaga, pedindo confirmação
#   limpar-cache --delete --yes  # apaga sem confirmação
#   limpar-cache --only node     # apenas node_modules
#   limpar-cache --only terraform
#   limpar-cache --base ~/outro/dir
#

set -uo pipefail

BASE="${HOME}/Documentos/github"
ONLY="all"
DO_DELETE=0
ASSUME_YES=0
JOBS="$(nproc 2>/dev/null || echo 4)"

usage() {
  cat <<'EOF'
limpar-cache — relatório e limpeza de caches em ~/Documentos/github

Alvos:
  */infra/terraform/.terraform
  */app/node_modules

Opções:
  -d, --delete         remove os diretórios encontrados (com confirmação)
  -y, --yes            não pede confirmação (implica --delete)
  -n, --dry-run        apenas relatório (padrão)
      --only TIPO      all (padrão) | terraform | node
      --base DIR       raiz a varrer (padrão: ~/Documentos/github)
  -h, --help           esta ajuda

Exemplos:
  limpar-cache
  limpar-cache --only terraform
  limpar-cache -d -y
EOF
}

while (($#)); do
  case "$1" in
    -d | --delete) DO_DELETE=1 ;;
    -y | --yes) ASSUME_YES=1 ;;
    -n | --dry-run) DO_DELETE=0 ;;
    --only)
      ONLY="${2:-}"
      shift
      ;;
    --base)
      BASE="${2:-}"
      shift
      ;;
    -h | --help)
      usage
      exit 0
      ;;
    *)
      printf 'limpar-cache: opção inválida: %s\n\n' "$1" >&2
      usage >&2
      exit 2
      ;;
  esac
  shift
done

case "$ONLY" in
  all | terraform | node) ;;
  *)
    printf 'limpar-cache: --only deve ser all, terraform ou node\n' >&2
    exit 2
    ;;
esac

((ASSUME_YES)) && DO_DELETE=1

if [[ ! -d "$BASE" ]]; then
  printf 'limpar-cache: diretório não encontrado: %s\n' "$BASE" >&2
  exit 1
fi

BASE="$(realpath -m -- "$BASE")"

human() { numfmt --to=iec-i --suffix=B --format='%.1f' "$1" 2>/dev/null || printf '%sB' "$1"; }

avail_bytes() { df -B1 --output=avail -- "$BASE" 2>/dev/null | tail -1 | tr -d ' '; }

# ---------------------------------------------------------------- coleta -----
# -prune evita descer em .git, node_modules e .terraform (rápido mesmo com 150 repos)
TF_DIRS=()
NODE_DIRS=()
while IFS= read -r -d '' d; do
  case "$d" in
    */infra/terraform/.terraform) TF_DIRS+=("$d") ;;
    */app/node_modules) NODE_DIRS+=("$d") ;;
  esac
done < <(find "$BASE" -mindepth 2 \( -name .git -o -name node_modules -o -name .terraform \) -prune -print0 2>/dev/null)

[[ "$ONLY" == "node" ]] && TF_DIRS=()
[[ "$ONLY" == "terraform" ]] && NODE_DIRS=()

if ((${#TF_DIRS[@]} == 0 && ${#NODE_DIRS[@]} == 0)); then
  printf 'Nada encontrado em %s (filtro: %s).\n' "$BASE" "$ONLY"
  exit 0
fi

TMP="$(mktemp)"
trap 'rm -f "$TMP"' EXIT

# tamanho aparente em paralelo: "tipo<TAB>bytes<TAB>caminho"
sizes_of() {
  local kind="$1"
  shift
  (($#)) || return 0
  printf '%s\0' "$@" |
    xargs -0 -r -n1 -P "$JOBS" bash -c '
      p="$1"
      s="$(du -sb -- "$p" 2>/dev/null | cut -f1)"
      [[ -n "$s" ]] || s=0
      printf "%s\t%s\t%s\n" "$0" "$s" "$p"' "$kind"
}

{
  sizes_of terraform "${TF_DIRS[@]+"${TF_DIRS[@]}"}"
  sizes_of node "${NODE_DIRS[@]+"${NODE_DIRS[@]}"}"
} >"$TMP"

sort -t $'\t' -k2,2nr -o "$TMP" "$TMP"

total_tf="$(awk -F'\t' '$1=="terraform"{s+=$2} END{print s+0}' "$TMP")"
total_node="$(awk -F'\t' '$1=="node"{s+=$2} END{print s+0}' "$TMP")"
total_all="$((total_tf + total_node))"
count_tf="${#TF_DIRS[@]}"
count_node="${#NODE_DIRS[@]}"

# ------------------------------------------------------------- relatório -----
printf '\nRaiz: %s\n' "$BASE"
printf '%-9s %12s  %s\n' "TIPO" "TAMANHO" "CAMINHO"
printf '%-9s %12s  %s\n' "--------" "-----------" "----------------------------------------"
while IFS=$'\t' read -r kind bytes path; do
  printf '%-9s %12s  %s\n' "$kind" "$(human "$bytes")" "${path#"$BASE"/}"
done <"$TMP"

printf '\nResumo\n'
printf '  terraform/.terraform : %3d diretório(s)  %10s\n' "$count_tf" "$(human "$total_tf")"
printf '  app/node_modules     : %3d diretório(s)  %10s\n' "$count_node" "$(human "$total_node")"
printf '  TOTAL                : %3d diretório(s)  %10s\n' "$((count_tf + count_node))" "$(human "$total_all")"
printf '  Livre em disco agora : %10s\n' "$(human "$(avail_bytes)")"

if ((DO_DELETE == 0)); then
  printf '\nModo relatório (nada foi apagado). Para remover: %s --delete\n\n' "$(basename "$0")"
  exit 0
fi

# --------------------------------------------------------------- remoção -----
if ((ASSUME_YES == 0)); then
  printf '\nRemover %d diretório(s), liberando ~%s? [s/N] ' "$((count_tf + count_node))" "$(human "$total_all")"
  read -r reply
  case "$reply" in
    s | S | y | Y | sim | SIM) ;;
    *)
      printf 'Cancelado.\n\n'
      exit 1
      ;;
  esac
fi

is_safe() {
  local p="$1" real
  [[ -n "$p" ]] || return 1
  [[ "$p" != "/" && "$p" != "$HOME" ]] || return 1
  case "$p" in
    */infra/terraform/.terraform | */app/node_modules) ;;
    *) return 1 ;;
  esac
  real="$(realpath -m -- "$p" 2>/dev/null)" || return 1
  [[ "$real" == "$BASE"/* ]] || return 1
  [[ "$real" != "$BASE" ]] || return 1
  return 0
}

avail_before="$(avail_bytes)"
removed=0
failed=0

while IFS=$'\t' read -r kind bytes path; do
  if ! is_safe "$path"; then
    printf '  IGNORADO (fora do padrão/raiz): %s\n' "$path"
    ((failed++))
    continue
  fi
  if rm -rf -- "$path" 2>/dev/null && [[ ! -e "$path" ]]; then
    printf '  removido  %10s  %s\n' "$(human "$bytes")" "${path#"$BASE"/}"
    ((removed++))
  else
    printf '  FALHOU    %10s  %s\n' "$(human "$bytes")" "${path#"$BASE"/}" >&2
    ((failed++))
  fi
done <"$TMP"

avail_after="$(avail_bytes)"
freed=$((avail_after - avail_before))
((freed < 0)) && freed=0

printf '\nConcluído: %d removido(s), %d falha(s).\n' "$removed" "$failed"
printf 'Espaço livre: %s -> %s (~%s liberados)\n\n' \
  "$(human "$avail_before")" "$(human "$avail_after")" "$(human "$freed")"

((failed == 0)) || exit 1
exit 0

Instale e deixe disponível em todo shell:

install -m 755 limpar-cache /usr/local/bin/limpar-cache

E no ~/.bashrc:

export PATH="/usr/local/bin:$PATH"
alias lcache='limpar-cache'

A linha explícita de PATH é redundante na maioria das distribuições, mas não custa nada e deixa a configuração à prova de shells de login esquisitos. (Ela já me salvou uma vez — veja a última lição.)

Rodando

Relatório primeiro, sempre:

$ lcache

Raiz: /home/user/Documentos/github
TIPO           TAMANHO  CAMINHO
--------   -----------  ----------------------------------------
node            1,4GiB  projeto-alfa/app/node_modules
node            1,1GiB  projeto-beta/app/node_modules
terraform     870,7MiB  projeto-alfa/infra/terraform/.terraform
terraform     865,2MiB  projeto-gama/infra/terraform/.terraform
node          771,2MiB  projeto-delta/app/node_modules
...

Depois, quando a lista parecer certa:

lcache --delete          # pergunta: Remover 30 diretório(s), liberando ~20,7GiB? [s/N]
lcache -d -y --only node # apenas node_modules, sem prompt

Uma execução na minha máquina:

Resumo
  terraform/.terraform :   9 diretório(s)      7,1GiB
  app/node_modules     :  21 diretório(s)     13,6GiB
  TOTAL                :  30 diretório(s)     20,7GiB
  Livre em disco agora :      9,9GiB

30 diretórios, ~20,7 GB. O disco saiu de 94% para cerca de 86% de uso — e cada um desses repositórios é reconstruível com npm ci ou terraform init.

Lições aprendidas

1. -prune é o truque inteiro. Um find ~/Documentos/github -name node_modules ingênuo desce em todo node_modules procurando outros aninhados. Em 150 repositórios isso são minutos de I/O inútil. Podar no momento do match transforma tudo em uma única varredura rápida.

2. Tamanho aparente ≠ uso de disco. du -sb dá bytes aparentes (rápido, comparável), enquanto du -s dá blocos alocados (o que o df reporta). Para um relatório de limpeza, o tamanho aparente é o número honesto e previsível; para "quanto eu vou ganhar de verdade", use --block-size=1 sem -b. Eu mostro df --output=avail antes/depois justamente porque os dois diferem.

3. O page cache frio domina o tempo de execução. A primeira execução mediu ~2m26s; a segunda foi bem mais rápida. Nada de errado com o script — são 30 árvores de diretório sendo lidas do disco pela primeira vez. Paralelismo com xargs -P ajuda, mas não vence a física.

4. Duas cópias de um script é um bug esperando para acontecer. Eu tinha instalado em ~/.local/bin primeiro e depois "movi" para /usr/local/bin — mas ~/.local/bin vem antes no PATH, então a cópia antiga sombreava a nova silenciosamente. Correção: mover de verdade (install + rm), não copiar. E conferir com command -v, não com which (que é outro binário, com outra semântica).

5. Proteções são baratas; erros de rm -rf não. A função is_safe() tem ~12 linhas e rejeita /, $HOME, qualquer coisa fora da raiz varrida e qualquer caminho que não termine nos dois sufixos esperados. Ela roda em cada caminho, mesmo os vindos do meu próprio find. Essa redundância é o ponto: o custo é microssegundos, o modo de falha é "apaguei meu diretório home".

6. Whitelist ganha de blacklist. Eu deliberadamente não apago dist/, .next/, .venv/ ou coverage/. Esses têm custos de reconstrução muito diferentes e alguns são artefatos de deploy. Se eu precisar deles depois, cada um ganha seu sufixo explícito no case — nunca uma regra de "apague tudo que pareça cache".

Checklist de recuperação

Depois de uma limpeza dessas, cada repositório precisa de uma coisa:

# Projetos Node
cd app && npm ci          # ou pnpm install / yarn install

# Projetos Terraform
cd infra/terraform && terraform init

O terraform init baixa os providers novamente; se você usa cache de plugins (TF_PLUGIN_CACHE_DIR), será bem mais rápido da próxima vez — vale configurar se costuma limpar .terraform com frequência.

Considerações finais

O script tem cerca de 200 linhas, a maioria delas printf. O que o torna seguro não é sofisticação, e sim um punhado de escolhas entediantes: dry-run primeiro, podar cedo, whitelist de caminhos, confirmar antes de apagar e imprimir o que realmente aconteceu.

Vinte gigabytes não é muito em termos absolutos, mas era 60% do espaço livre que eu tinha. Mais importante: o comando agora existe. Na próxima vez que o disco apertar, é uma linha em vez de uma tarde de du e dúvidas.

Se você mantém muitos repositórios num só lugar, o padrão se generaliza — troque os dois sufixos pelos seus próprios artefatos de build, mantenha o dry-run como padrão e nunca deixe o script adivinhar o que deve apagar.

Your Pagination Is Showing Users the Same Item Twice (and Skipping Others). Here's Why Offset Breaks

Mon, 26 Oct 2026 11:11:24 +0000

A user scrolls through your feed. Page 1 ends with "Post 20". They tap "Next", and page 2 starts with... "Post 20" again.

Another user pages through an admin table and never sees order #4817. It exists. It is in the database. It just never showed up on any page.

Nobody changed any code. The data changed underneath the pagination, and your skip and limit could not cope.

The Code Everyone Writes

app.get('/api/posts', async (req, res) => {
  const page = Number(req.query.page) || 1;
  const limit = 20;

  const posts = await Post.find()
    .sort({ createdAt: -1 })
    .skip((page - 1) * limit)
    .limit(limit);

  res.json(posts);
});

It is simple, it works in development, and ?page=2 is easy to understand. That is why it is everywhere.

Why It Breaks

Offset pagination says "skip the first N rows, then give me the next 20". It counts positions, and positions move when the data does.

Duplicates. Someone publishes a new post while a reader is on page 1. Every existing post shifts down one position. When the reader requests page 2, the last item of page 1 is now the first item of page 2.

Skips. The reverse: a post is deleted while the reader is on page 1. Everything shifts up one, and the first item of what would have been page 2 slides into page 1's range. The reader never sees it.

Before:  [1 2 3 4 5 | 6 7 8 9 10]    page 1 = 1-5, page 2 = 6-10
New item arrives at the top:
After:   [0 1 2 3 4 | 5 6 7 8 9]     page 2 now starts with 5, which they already saw

The Second Problem: It Gets Slower on Deep Pages

skip(100000) does not jump to row 100,000. The database has to walk past 100,000 documents and throw them away before returning your 20. Page 1 is fast. Page 5,000 is slow, and the cost grows with the page number.

This is why a feed that feels fine with 1,000 rows slows down quietly at a million.

The Fix: Cursor Pagination

Instead of "skip N", say "give me the 20 items after this one". The anchor is a cursor: a value that identifies where the last page ended.

A new item arriving at the top cannot shift your place, because you are not counting positions. You are standing on a specific item.

Step 1: Sort by something stable, with a tie-breaker

createdAt alone is not enough, because two documents can share the same timestamp. Add _id as a tie-breaker so the order is always exact:

.sort({ createdAt: -1, _id: -1 })

Step 2: Support the query with an index

PostSchema.index({ createdAt: -1, _id: -1 });

With this index, the database jumps straight to the cursor and reads 20 documents, whether you are on page 1 or page 5,000.

Step 3: Query "after the cursor"

app.get('/api/posts', async (req, res) => {
  const limit = 20;
  const cursor = req.query.cursor ? decodeCursor(req.query.cursor) : null;

  const filter = cursor
    ? {
        $or: [
          { createdAt: { $lt: cursor.createdAt } },
          { createdAt: cursor.createdAt, _id: { $lt: cursor._id } },
        ],
      }
    : {};

  const rows = await Post.find(filter)
    .sort({ createdAt: -1, _id: -1 })
    .limit(limit + 1); // fetch one extra to know if there is a next page

  const hasMore = rows.length > limit;
  const items = hasMore ? rows.slice(0, limit) : rows;
  const last = items[items.length - 1];

  res.json({
    items,
    nextCursor: hasMore ? encodeCursor(last) : null,
  });
});

The $or says: "older than the cursor, or the same timestamp but a smaller _id". That second branch is what stops items with identical timestamps from being skipped or repeated.

Step 4: Encode the cursor so clients do not depend on it

const encodeCursor = (doc) =>
  Buffer.from(JSON.stringify({ createdAt: doc.createdAt, _id: doc._id })).toString('base64url');

const decodeCursor = (str) => {
  const { createdAt, _id } = JSON.parse(Buffer.from(str, 'base64url').toString());
  return { createdAt: new Date(createdAt), _id: new mongoose.Types.ObjectId(_id) };
};

Clients treat it as an opaque string and send it back as it is. You can change what is inside later without breaking anyone. In production, validate the decoded values and handle a malformed cursor with a 400, not a crash.

What You Give Up

Cursor pagination has a trade-off, and you should know it before switching:

No "jump to page 47". You can only go forward (or back, with a second cursor). For feeds and infinite scroll, that is perfect. For a numbered page bar, it is not.
No cheap total count. countDocuments on a large collection is not free either way, so many apps drop the "page 3 of 500" display.

When Offset Is Still Fine

Do not rewrite everything. Offset pagination is a reasonable choice for small, slow-changing tables, like an admin list with a few hundred rows, where users really do want page numbers.

Situation	Use
Feed, timeline, infinite scroll	Cursor
Large collection, deep pages	Cursor
Data changes while people browse	Cursor
Small admin table with page numbers	Offset is fine

The Rule

Offset pagination counts positions, and positions change. Cursor pagination anchors to a specific item, so it stays correct when data is added or removed. Always sort by a unique, indexed combination, and fetch one extra row to know whether there is a next page.

Quick audit: search your project for .skip(. For each hit, ask whether the collection can grow past a few thousand documents, or change while someone is paging through it.

If you work with MongoDB, I collect more patterns like this, around indexes and queries, in my MongoDB and Mongoose patterns guide, and you can browse my Next.js templates on pixelanas.com.

Get the templates: https://pixelanas.gumroad.com

Have you ever been bitten by duplicate or missing items in a paginated list? Tell me how you noticed in the comments.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

Your 'Pay Now' Button Can Charge Twice. Idempotency Keys Are the Fix Nobody Adds Until It Happens

Sun, 25 Oct 2026 11:09:21 +0000

A customer clicks "Pay now". Nothing happens for three seconds, so they click again. Or their phone loses signal for a moment and the app quietly retries the request. Or your load balancer times out and sends the request again.

Your server received the same order twice, and it did exactly what you told it to do: it created two orders.

This is not a rare edge case. It is the default behavior of every POST endpoint you have ever written.

The Endpoint That Looks Fine

app.post('/api/orders', requireAuth, async (req, res) => {
  const order = await createOrder(req.user.id, req.body);
  await chargeCard(order);
  res.status(201).json({ orderId: order.id });
});

Nothing is wrong with this code in isolation. The problem is what it cannot know: whether this request is new, or a repeat of one that already succeeded.

Why This Is Hard to Catch

You test with one request at a time. The bug needs two requests with the same intent, close together.
Timeouts are ambiguous. When a client times out, it does not know if the server never got the request, or did the work and the response got lost. Retrying is the only safe-looking choice, and it is the one that duplicates.
Disabling the button is not enough. It helps with double clicks but not with network retries, mobile apps, or another service calling your API.

The Idea: Same Key, Same Result

An idempotent operation is one where doing it twice has the same effect as doing it once. POST is not idempotent by default, but you can make it behave that way:

The client generates a unique idempotency key for each intent and sends it in a header.
The server remembers the key together with the result.
If the same key arrives again, the server returns the saved result and does not do the work again.

Payment providers such as Stripe use exactly this pattern with an Idempotency-Key header.

A Working Version With Express and MongoDB

First, a unique index, plus an expiry so old keys clean themselves up:

await keys.createIndex({ userId: 1, key: 1 }, { unique: true });
await keys.createIndex({ createdAt: 1 }, { expireAfterSeconds: 86400 });

Then the endpoint:

app.post('/api/orders', requireAuth, async (req, res) => {
  const key = req.get('Idempotency-Key');
  if (!key) return res.status(400).json({ error: 'Idempotency-Key header required' });

  const userId = req.user.id;
  const bodyHash = hash(stableStringify(req.body));

  // 1. Try to claim the key. Only one request can win this insert.
  try {
    await keys.insertOne({ userId, key, bodyHash, status: 'processing', createdAt: new Date() });
  } catch (err) {
    if (err.code !== 11000) throw err; // 11000 = duplicate key

    const saved = await keys.findOne({ userId, key });
    if (saved.bodyHash !== bodyHash) {
      return res.status(422).json({ error: 'Key was used with a different request' });
    }
    if (saved.status === 'processing') {
      return res.status(409).json({ error: 'Request is still being processed' });
    }
    return res.status(saved.statusCode).json(saved.response); // replay
  }

  // 2. We own the key. Do the work once.
  try {
    const order = await createOrder(userId, req.body);
    await chargeCard(order);

    const response = { orderId: order.id };
    await keys.updateOne(
      { userId, key },
      { $set: { status: 'done', statusCode: 201, response } }
    );
    return res.status(201).json(response);
  } catch (err) {
    await keys.deleteOne({ userId, key }); // let the client retry
    throw err;
  }
});

The Race Condition Most Examples Miss

A common first attempt is:

const existing = await keys.findOne({ userId, key });
if (existing) return res.json(existing.response);
// ... do the work ...
await keys.insertOne({ userId, key, response });

Two simultaneous requests both run findOne, both see nothing, and both do the work. You added a check and still got a duplicate.

The fix is in the version above: claim the key first with an insert that can only succeed once, and let the unique index decide the winner. The database settles the race, not your code.

Four Details That Matter

1. Scope keys per user. Two users who happen to generate the same key must not see each other's responses. That is why the index is on { userId, key }.

2. Compare the request body. If the same key arrives with a different body, that is a bug or an attack, not a retry. Return an error instead of replaying a response that does not match the request. Use a stable stringify (sorted keys) for the hash, or two identical bodies with different property order will hash differently.

3. Generate the key per intent, not per attempt. On the client, create the key when the user starts the action and reuse it for every retry of that action. A new key per retry brings the duplicate right back.

const key = crypto.randomUUID(); // once, when the checkout starts
await fetch('/api/orders', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Idempotency-Key': key },
  body: JSON.stringify(cart),
});

4. Pass the key on to your payment provider too. If your server crashes after charging the card but before saving the result, your own table cannot help. A key sent to the provider protects that gap, and the expiry index clears any stuck processing record.

The Rule

Any request that creates something or moves money must be safe to send twice. Do not rely on disabled buttons or "the network is reliable". Give the client a key, claim it atomically on the server, and replay the saved result on a repeat.

Quick audit: list your POST endpoints that create orders, payments, invoices, emails or bookings. For each, ask what happens if the request arrives twice, a second apart. If the answer is "two of them", you have found your next ticket.

If you work with MongoDB, my MongoDB and Mongoose patterns guide has more on using the database to enforce rules, and you can browse my Next.js templates on pixelanas.com.

Get the templates: https://pixelanas.gumroad.com

Have you ever shipped a double-charge or a duplicate-order bug? Tell me how you found out, in the comments.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

forEach Doesn't Wait for Your await: The Bug That Makes APIs Reply Before the Work Is Don

Tue, 20 Oct 2026 10:16:43 +0000

Your endpoint sends 50 emails, marks 50 orders as shipped, and returns { success: true }.

In your tests, it works. In production, the response comes back in 8 milliseconds, half the orders are still pending, and one failed email crashed the whole Node process with an error nobody can trace to a request.

The culprit is usually one innocent line.

The Code That Looks Fine

app.post('/api/ship', async (req, res) => {
  const orders = await Order.find({ status: 'paid' });

  orders.forEach(async (order) => {
    await sendShippingEmail(order);
    await Order.updateOne({ _id: order._id }, { status: 'shipped' });
  });

  res.json({ success: true });
});

There is an await inside, so it must wait. It does not.

Why It Returns Early

forEach was written before promises existed. It calls your function for each item and ignores whatever the function returns. An async function returns a promise, and forEach throws that promise away without looking at it.

So the loop starts 50 async tasks, finishes in one tick, and moves on. The await inside each callback only pauses that callback, not the loop around it, and not your handler.

console.log('start');
[1, 2, 3].forEach(async (n) => {
  await new Promise((r) => setTimeout(r, 100));
  console.log('done', n);
});
console.log('end');

// start
// end
// done 1
// done 2
// done 3

end prints before any work finishes. That is your res.json().

Why It Is Hard to Catch

Fast tests hide it. With a local database and a mocked email service, the work finishes before your next assertion runs. It passes by luck.
The failure is somewhere else. The response says success, so nothing points back to this line.
Errors escape. If a callback throws, nobody is awaiting that promise, so there is no try/catch that can catch it. Since Node 15, an unhandled rejection crashes the process by default. One bad email address can take down the server, and the stack trace will not mention the request that caused it.

Fix 1: Need It Sequential? Use for...of

for (const order of orders) {
  await sendShippingEmail(order);
  await Order.updateOne({ _id: order._id }, { status: 'shipped' });
}

res.json({ success: true });

for...of is real control flow, so await pauses the whole loop. Errors land in your try/catch. Use this when order matters, or when you must not hit a rate-limited API in parallel.

Fix 2: Need It Fast? Promise.all With map

await Promise.all(
  orders.map(async (order) => {
    await sendShippingEmail(order);
    await Order.updateOne({ _id: order._id }, { status: 'shipped' });
  })
);

res.json({ success: true });

map returns the array of promises that forEach was throwing away, and Promise.all waits for all of them. Same callback, one different method.

Fix 3: Some Failures Are Acceptable? Use allSettled

Promise.all rejects as soon as one promise fails, but the other tasks keep running. You get an error back while half the work still completes in the background.

If you want every task to finish and then look at what failed:

const results = await Promise.allSettled(orders.map(shipOrder));

const failed = results
  .map((r, i) => ({ r, order: orders[i] }))
  .filter(({ r }) => r.status === 'rejected');

res.json({
  shipped: results.length - failed.length,
  failed: failed.map(({ order }) => order._id),
});

Now one bad email does not hide the 49 that worked, and the response tells the caller exactly what to retry.

Fix 4: Thousands of Items? Limit the Concurrency

Promise.all(items.map(...)) starts everything at once. With 10,000 items, that is 10,000 simultaneous database calls or HTTP requests, which can exhaust your connection pool or get you rate limited.

Process in batches:

async function inBatches(items, size, fn) {
  const results = [];
  for (let i = 0; i < items.length; i += size) {
    const batch = items.slice(i, i + size);
    results.push(...(await Promise.all(batch.map(fn))));
  }
  return results;
}

await inBatches(orders, 10, shipOrder);

Ten at a time is fast enough for most jobs and gentle on whatever is on the other end.

Same Trap in filter, some, every and reduce

forEach is not the only method that does not understand promises.

// Always keeps everything. A Promise is truthy.
const valid = users.filter(async (u) => await isActive(u));

The callback returns a promise for every user, and a promise object is always truthy, so nothing is ever filtered out. some and every have the same problem and return true immediately.

// Resolve first, then filter
const flags = await Promise.all(users.map(isActive));
const valid = users.filter((_, i) => flags[i]);

The Rule

Array methods like forEach, filter, some and every do not await. If an async function sits inside one, the work is detached from your request. Use for...of to run steps in order, and Promise.all or Promise.allSettled with map to run them together.

Quick audit: search your project for .forEach(async. In a route handler, a job or a script that exits when it finishes, each hit is either a bug or a race that has not lost yet.

I use this pattern in the bulk operations behind my Next.js and MongoDB templates on pixelanas.com, where seeding, imports and batch updates must finish before the response goes out.

Get the templates: https://pixelanas.gumroad.com

How many .forEach(async do you find in your codebase? Drop the number in the comments, and tell me if one of them turned out to be a real bug.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

I built a Supabase-style query client that talks directly to Postgres

Wed, 14 Oct 2026 22:57:08 +0000

I like the Supabase JS client. .from('todos').select('id, task').eq('done', false) reads like what I mean, and { data, error } means I'm not wrapping every query in try/catch.

What I don't always want is the stuff around it: a project URL, an anon key, and a PostgREST service sitting between my Node server and a Postgres I already control.

So I built pg-fluent: the postgrest-js query builder, executed directly against PostgreSQL through node-postgres.

import { createClient } from 'pg-fluent'

const db = createClient({ connectionString: process.env.DATABASE_URL })

const { data, error } = await db
  .from('todos')
  .select('id, task, done')
  .eq('done', false)
  .order('id')
  .limit(10)

If you've used supabase-js, that's already familiar. The difference is there's no HTTP request on the other end, just one parameterized SQL statement over a normal Postgres connection.

npm install pg-fluent

Repo: https://github.com/gerald-mathew/pg-fluent

Who it's for

People running long-lived servers (Express, Fastify, a VPS, containers) who like the Supabase query API but want to connect straight to Postgres, or who are moving off Supabase's hosted stack and don't want to rewrite every query. It's server-side only, more on that below.

How it works

I didn't reimplement the builder. pg-fluent uses the real @supabase/postgrest-js and hands it a custom fetch. When the builder "sends" a request, pg-fluent catches it, parses the URL, headers and body the way PostgREST would, plans one parameterized SQL statement, runs it through pg, and returns a response postgrest-js unpacks as if a server had answered.

Results are aggregated to JSON inside Postgres, so you get the same shape as Supabase. Embeds like select('id, title, author:users(name)') become subqueries that aggregate to JSON, with relationships resolved from the live schema (cached, with auto-refresh).

Since I'm intercepting what the builder emits, an upstream change could quietly change the SQL. So the dependency is pinned to an exact version, and docs/UPSTREAM.md records every bump with the date and reason. A bump that changes observable SQL or results is at least a minor release.

What you get beyond the basics

The whole postgrest-js surface works: filters, or()/not()/filter(), JSON paths like meta->>author, range(), single()/maybeSingle(), count, upsert, rpc, csv(), explain(), and embeds (many-to-one, one-to-many, many-to-many, !inner, aliases, spread). Then the things that only make sense with a direct connection:

See the SQL without running it:

const { text, values } = await db.from('todos').select('id').eq('done', false).toSQL()

Transactions with savepoints:

const user = await db.transaction(async (trx) => {
  const { data, error } = await trx.from('users').insert({ name: 'G' }).select().single()
  if (error) throw error
  await trx.from('profiles').insert({ user_id: data.id })
  return data
}, { isolationLevel: 'read committed' })

Raw SQL, safe by default. The tagged template always binds interpolations as parameters. The only way to run SQL text you built yourself is db.unsafe(), and the name is deliberate.

const { data } = await db.sql`select * from users where id = ${id}`

Streaming, COPY, LISTEN/NOTIFY:

for await (const row of db.from('events').select('*').stream()) {
  await handle(row)
}

await db.copyFrom('events', rows, { columns: ['id', 'kind'], format: 'csv' })

const sub = await db.listen('orders', (payload) => console.log(payload))

The listener gets its own non-pooled connection and reconnects with backoff.

Named connections and read replicas. Reads can go to replicas, while writes, RPC, transactions and LISTEN always hit the primary. Add .primary() when you need read-your-writes.

Type generation:

npx pg-fluent gen-types --connection "$DATABASE_URL" --schema public --out ./database.types.ts

The output is a Database type compatible with postgrest-js, so createClient<Database>(...) just works.

There are also logger and onQuery hooks and optional OpenTelemetry spans if @opentelemetry/api is installed. Parameter values are never logged unless you opt in.

Guardrails by default

Unfiltered UPDATE/DELETE is refused with PGFLUENT_UNFILTERED_MUTATION. Opt in with .allowUnfiltered() or a client option.
Reads are capped: maxRows defaults to 1000 when you set no limit or range.
Statements time out: statementTimeoutMs defaults to 30 seconds.
Retries are read-only. Transient connection failures retry with backoff for reads, and writes are never retried.

How I tested it

I didn't want to say "drop-in" on a hunch, so:

Differential tests against a real PostgREST server. The same requests go through both, and the payloads must match. One difference is intentional and documented: a reserved parameter name like columns or select used as a filter is rejected with PGFLUENT_INVALID_QUERY. PostgREST silently ignores it, which turns a scoped read into a full-table read.
Property-based injection tests with fast-check. Hostile strings go in as filter values, the SQL text must match the baseline, the values must land in parameters only, and a canary table must survive.
A real CI matrix: Node 22 and 24 against Postgres 15, 16, 17 and 18.

That's 200+ tests. There's also a benchmark harness against raw pg, but I'm not publishing performance claims. Network and pooling change the numbers too much, so it only reports methodology and variance for you to run yourself.

The RLS caveat (please read)

pg-fluent runs statements as the connection's own role, like psql or any other driver. It doesn't set request.jwt.claims and doesn't emulate Supabase's anon and authenticated roles.

So RLS policies written for Supabase's roles don't automatically protect you here. Connect as the table owner or a BYPASSRLS role and RLS is skipped entirely. Use a least-privilege role, keep this on the server, and never let database credentials reach a browser. For per-user enforcement, connect as a non-owner role and set SET ROLE and the JWT claims yourself. If you're migrating from supabase-js, treat this as step zero.

What's missing or rough

It's 0.2.x, so, honestly:

Embedding views isn't supported yet (you can read them, just not embed them).
No auth, storage or functions. It's a database client, not a BaaS SDK.
LISTEN/NOTIFY doesn't work through transaction-mode poolers like PgBouncer. Use session-mode or a direct connection.
Replica reads are asynchronous, and there's no lag measurement in v1.
Requires Node 22+ and Postgres 15 to 18.

Try it, break it

If you have a Supabase-style codebase and a Postgres you control, point pg-fluent at a test database and tell me what breaks. Issues and PRs are welcome at https://github.com/gerald-mathew/pg-fluent, and security reports go through the private channel in SECURITY.md.

I especially want to know which postgrest-js queries behave differently from what you expect. Real queries find things my fixtures won't.

Retry Failed n8n API Requests Without Duplicates

Wed, 14 Oct 2026 15:50:00 +0000

The safe n8n retry pattern is simple: create one stable idempotency key for the business operation, send the same key and payload on every attempt, retry only temporary failures, and store the confirmed result. Turning on Retry On Fail without that protection can repeat a successful side effect when the first response was lost.

Partner disclosure. This guide includes a partner link to n8n Cloud. If you sign up through it, I may earn a commission at no extra cost to you.

The dangerous case: an API creates the order, but the connection closes before n8n receives the response. n8n sees a timeout. A second POST with a new key can create a second order even though the first request actually worked.

Decide whether the request is safe to repeat

HTTP defines GET, HEAD, OPTIONS, and TRACE as safe methods. PUT, DELETE, and the safe methods are idempotent by their intended semantics. POST is not automatically idempotent. The HTTP specification therefore says a client should not automatically retry a non-idempotent request unless it knows the operation is idempotent or can prove the first request was not applied.

Real APIs add their own rules, so the provider's documentation remains the contract. A DELETE endpoint may be harmless to repeat, while a POST that sends an email, creates a payment, or reserves stock needs explicit duplicate protection.

Failure	Default action	Duplicate risk
Network timeout or reset	Retry the identical request with the same operation key	High, because the remote service may already have committed the action
429 Too Many Requests	Wait for the provider's limit window, then retry	Controlled only if the operation remains idempotent
500, 502, 503, or 504	Use a small bounded retry policy when the provider permits it	Potentially high for state-changing requests
400, 401, 403, or most 404s	Fix the request, credentials, permission, or resource first	Blind retries add load but rarely fix the cause
409 Conflict	Inspect the API-specific error before deciding	May indicate an existing operation or key conflict

Step 1: create one key for one business operation

Build the key before the HTTP Request node from an ID that already belongs to the event: an order ID, checkout ID, lead ID, invoice ID, or source event ID. Include the action name so different operations on the same record cannot collide.

={{ 'create-order:' + $json.orderId }}

Keep this value unchanged across automatic node retries, manual execution retries, and later recovery runs. Do not use $now, the execution ID, or a freshly generated random value on each attempt. Those identify the attempt, not the operation, so the destination sees every retry as new work.

The key must also stay paired with the same endpoint and payload. If you correct the amount, recipient, or other business data, treat that as a deliberate new operation according to the provider's rules.

Step 2: use the API's idempotency feature first

If the destination API supports idempotency, send the stable key in the exact header or field its documentation specifies. Stripe, for example, accepts an Idempotency-Key header for POST requests and returns the stored result when the same request is repeated with the same key. That header name is not universal. Some APIs use a request ID, client reference, deduplication ID, or a provider-specific field.

Open the HTTP Request node and add the provider's documented idempotency header.
Map the stable operation key as its value.
Keep the method, URL, and request body identical on every retry.
Save the external object ID from the confirmed response for later reconciliation.

Do not assume: adding an Idempotency-Key header only works when the destination implements that contract. An unknown header may simply be ignored.

Step 3: configure a bounded n8n retry

Every n8n node has a Retry On Fail setting. When enabled, n8n reruns the node after a failure until it succeeds or reaches the configured attempt limit. For a known transient endpoint, start with a small policy such as three tries and a short wait between tries. Increase it only when the provider's limit and incident behavior justify the extra traffic.

This built-in setting is best when every error produced by that node is safe to retry. If you need status-aware behavior, enable Include Response Headers and Status and Never Error in the HTTP Request node. The node can then pass the status code and headers to an IF or Switch branch instead of stopping on every non-2xx response.

Send 2xx responses to the success branch.
Send 429 and approved temporary 5xx responses to a Wait step and bounded retry path.
Send authentication, validation, permission, and unexpected responses to a review or alert branch.
Stop after the maximum attempt count. Never create an unbounded loop.

Step 4: slow down 429 retries

A 429 response means the client sent too many requests in the provider's current limit window. Some APIs include Retry-After as seconds or an HTTP date. Read and honor it when the provider documents that behavior. If the header is absent, use the API's published limit and increase the delay between attempts rather than retrying immediately.

The HTTP Request node also has Items per Batch and Batch Interval options. They control how many input items are sent in each batch and how many milliseconds n8n waits between batches. Use them to prevent a known burst from creating the 429 in the first place.

Step 5: add a persistent duplicate check when needed

When the API has no idempotency feature, keep a small request ledger. n8n's Data Table node can check whether a row exists, insert rows, update them, and upsert by conditions. A practical table uses these columns:

Column	Purpose
`request_key`	Stable ID for the business operation
`status`	`pending`, `succeeded`, `failed`, or `unknown`
`external_id`	ID returned by the destination when available
`updated_at`	Timestamp for recovery and stale-pending review

Check for a row with the request key.
If it already says succeeded, return the stored result and skip the API call.
If no row exists, record pending, then call the API.
On a confirmed 2xx response, update the row to succeeded and store the external ID.
On a timeout, keep the state as unknown until you query the destination or retry safely.

This ledger is useful for sequential or low-concurrency workflows, but a check followed by an insert is not an atomic lock. Two executions can check at the same moment and both see no row. For parallel production traffic, prefer destination-side idempotency or a database table with a unique constraint on request_key and an atomic claim operation.

That pattern protects one API operation from replay. When the business problem is recognizing the same person across forms, email, phone, and an existing CRM, use the separate n8n lead deduplication workflow and its conflict-review path.

Do not use workflow static data as a production lock

The $getWorkflowStaticData() helper looks convenient for storing processed IDs, but n8n labels it experimental, says the data should be small, and warns that it may behave unreliably under high-frequency executions. That makes it unsuitable as the main duplicate barrier for valuable orders, payments, leads, or outbound messages.

A production-safe request sequence

Validate that the event contains a stable source ID.
Create the operation key once.
Check any local success record.
Send the request with the provider's idempotency mechanism.
Retry only network failures, 429s, and provider-approved temporary errors.
Reuse the same key and payload for every attempt.
Store the external ID after confirmed success.
Route exhausted or uncertain outcomes to an Error Workflow or manual review queue.

Keep request-level recovery separate from broad diagnosis. If you do not yet know why the node failed, use the failed n8n workflow debugging guide. Once the retry budget is exhausted, use an n8n Error Workflow to send the failure to Telegram, Slack, Google Chat, email, or another incident channel.

Common questions

Does Retry On Fail prevent duplicates in n8n?

No. It repeats the failed node. Duplicate prevention comes from idempotent request semantics, a provider-supported operation key, or a persistent store that can claim each business operation exactly once.

Should I retry every 500 response?

Not automatically. A 500 can leave a state-changing operation with an uncertain outcome. Follow the provider's error guidance, keep the same idempotency key, use a bounded retry policy, and reconcile the remote record when the result remains unknown.

Can the Remove Duplicates node replace an idempotency key?

It can remove repeated input items, but it cannot prove that a remote API did or did not apply an earlier request after a timeout. Protect the side effect at the API or persistence layer.

Sources checked

n8n partner link

Want managed n8n for the workflow?

I may earn a commission if you sign up through this link, at no extra cost to you.

Try n8n Cloud

Need one risky API workflow mapped?

Send the API operation, failure response, source ID, and expected business outcome. I will help separate safe retries from duplicate risk.

Map the retry path

Originally published on FloxoLab.

Why tests should verify runtime contracts, not inferred types

Fri, 09 Oct 2026 19:59:18 +0000

If your Node tests are green but callers still need casts, you’re mixing two different guarantees. A Node test proves what the function did when executed: the values returned, whether the input was mutated, and which inputs throw. It does not and cannot prove how the TypeScript compiler will narrow that result at a later use site.

Implementation summary

Keep the runtime contract explicit: the helper should validate inputs (e.g., reject non-arrays) and return a new array that removes only null and undefined (use a nullish check rather than truthiness). The server-inserted example shows a small function that performs exactly that behavior and does not mutate the input.

Expected behavior to test

Mixed-array case: the output contains the original non-nullish items in order and preserves falsy-but-valid values like 0, '', and false.
Empty input: returns a new empty array (not the same reference).
All-nullish input: returns an empty array.
Invalid input: throws a TypeError for non-array arguments.

Where static checking belongs

If you need the helper to provide compiler-level narrowing (so consumers can assign without casts), verify that with the TypeScript compiler or editor diagnostics. A minimal compile-time check is to write a small use-site that only accepts the narrowed type and run tsc to ensure it typechecks.

Limitations and pitfalls

Compiler inference changes across TypeScript versions (e.g., TypeScript 5.5 adds inferred predicates in some filter callbacks). Relying on runtime tests to encode compiler behavior makes tests brittle or misleading.
Truthiness-based filters (e.g., !!x) can silently drop legitimate falsy values — runtime tests must check exact outputs, not just lengths.

Practical rule

Tests: prove the runtime contract you actually depend on. Compiler: prove the static narrowing your consumers rely on. Keep them separate so failures point to the right fix: change JS behavior, adjust types, or update consumers.

The article includes a repaired example and concrete test cases that illustrate this split and avoid overclaiming what a JavaScript test can prove. The server will append the full code example and the article URL.

Working example

export const note = `This module demonstrates runtime removal of null and undefined from arrays.
Runtime tests (node:test) validate behavior at execution time but do NOT prove any TypeScript compile-time narrowing. To check static narrowing you must run the TypeScript compiler (tsc) or use editor/IDE type checks.`;

/**
 * Remove null and undefined from an array while preserving order and other falsy
 * values such as 0, '', and false.
 *
 * Runtime contract: returns a new array containing only elements that are not
 * null and not undefined. Throws a TypeError if the argument is not an Array.
 *
 * Note: This demonstrates runtime behavior only. TypeScript's ability to
 * narrow types (e.g. from (T | null | undefined)[] to T[]) is a compile-time
 * property checked by tsc / the language service and is not proven by these
 * runtime tests.
 */
export function filterNonNullish(arr) {
  if (!Array.isArray(arr)) {
    throw new TypeError('filterNonNullish expects an Array');
  }

  return arr.filter((x) => x != null);
}

Read the full article: https://chriseugenerodriguez.com/blog/tests-should-verify-runtime-contract-not-inferred-types

Human Approval for AI Emails in n8n: Prevent Duplicate Sends

Fri, 09 Oct 2026 15:50:00 +0000

A safe AI email workflow creates a proposal, not a sent message. It gives that proposal a stable approval ID, stores it with a pending state, shows a reviewer enough context to decide, waits for a response, and only then allows one guarded send.

Partner disclosure. This guide includes a partner link to n8n Cloud. If you sign up through it, I may earn a commission at no extra cost to you.

Short answer: in n8n, use Gmail Send and Wait for Approval for a simple approve-or-reject gate. Use a Wait node with On Form Submitted plus a Data Table when the reviewer also needs to edit, reject with a reason, or let the request expire.

Need	Best starting pattern	Important limit
Approve or reject only	Gmail Message → Send and Wait for Approval	Good for a simple decision, not a full edit-and-escalation process
Approve, edit, reject, or timeout	Data Table + reviewer notification + Wait form	The form link is sensitive and the documented form mode has no authentication setting
Identity-proof approval	Wait webhook with Basic, Header, or JWT auth, or an authenticated internal portal	More setup, but the reviewer identity is stronger than a typed email address

The safe workflow architecture

Trigger or incoming request
  → AI creates proposed subject and body
  → Validate recipient, policy, and required fields
  → Data Table: insert approval_id with status = pending
  → Notify reviewer with safe context and resume URL
  → Wait: On Form Submitted, with a time limit
  → Switch: approve / edit / reject / timeout
  → Data Table: pending → sending
  → Gmail: send once
  → Data Table: sent, message_id, reviewer, timestamps

This guide begins after an AI draft already exists. It is separate from classifying and routing incoming email. The approval gate controls an outbound action that could reach a customer, supplier, applicant, or lead.

Why the model must not connect directly to Send

A language model can produce fluent text while using the wrong price, recipient, promise, date, or tone. Prompt instructions help, but they do not create an operational control. If the model node connects directly to Gmail Send, a bad draft can leave the business before anyone sees it.

Put validation and approval between generation and delivery. Validate machine-checkable rules first, such as an allowed recipient domain, non-empty subject, maximum length, banned attachments, and required account references. Then ask the human to judge the parts that need context.

Step 1: start with the simplest n8n setup

For a small team that does not want to maintain a server, n8n Cloud is the easiest starting point. n8n describes Cloud as the option where it handles the infrastructure. Create one workflow, connect the AI provider you already use, add a Gmail credential, and test only with internal addresses until every branch behaves correctly.

If the decision is only approve or reject, test Gmail Message → Send and Wait for Approval first. The current Gmail node supports an Approve Only mode and an Approve and Disapprove mode. n8n itself recommends the Wait node for more complex approvals, which is the full pattern below.

Step 2: turn the AI output into a real draft record

Add an Edit Fields node after the model. Keep only the fields the approval stage needs, then generate a stable approval_id. Use a UUID or another collision-resistant ID. Do not use the recipient address or current timestamp by itself.

approval_id
recipient
subject_proposed
body_proposed
source_record_id
status
created_at
expires_at

Set status = pending. Store the proposed subject and body in a Data Table before notifying anyone. This durable row lets an operator see what is waiting even when the workflow execution is paused or the reviewer email is lost.

Gmail also has a separate Draft resource. The current n8n Gmail Draft operations document create, delete, get, and list. They do not document a built-in Draft Send operation. Use a provider-side Gmail draft only as an optional inspectable copy unless you deliberately add the Gmail API users.drafts.send call. Do not assume a Gmail Draft node automatically delivers the draft later.

Step 3: create the approval table

For a first version, create an email_approvals Data Table with these columns:

approval_id
source_record_id
recipient
subject_proposed
body_proposed
subject_approved
body_approved
status
reviewer
decision_reason
created_at
decided_at
sent_at
provider_message_id

The Data Table node can insert, get, update, and upsert rows using conditions. That is enough for a small approval ledger. If the workflow has high concurrency, regulatory audit requirements, or several systems changing the same state, move the ledger to a database with transactions, unique constraints, and access controls.

Step 4: notify the reviewer and pause

Send the reviewer an internal email containing the approval ID, recipient, proposed subject, a safe excerpt, expiry time, and the Wait form link. The Wait node exposes $execution.resumeUrl, which is unique to the execution. The node that sends this link must run in the same execution as the Wait node.

Configure Wait → On Form Submitted with:

A required dropdown named decision with Approve, Edit, and Reject.
An optional text field for the approved subject.
An optional textarea for a revised body.
An optional rejection reason.
A reviewer identifier for the audit log.
Limit Wait Time set to the business deadline, such as 24 hours.

If the reviewer chooses Edit but provides no replacement body, do not send. Route the item back to pending or a manual queue. n8n's form fields are not documented as conditionally required, so enforce this rule after submission with an If or Switch node.

Treat the resume URL like a secret. The Wait form documentation does not list the Basic, Header, or JWT authentication choices available in the webhook mode. Do not post the link in a broad channel. For sensitive approvals, use an authenticated webhook or internal portal and record the authenticated user.

Step 5: branch every possible outcome

Outcome	State change	Next action
Approve	`pending → approved`	Copy the proposed subject and body into approved fields
Edit	`pending → approved_edited`	Validate and store the human replacement before sending
Reject	`pending → rejected`	Store the reason; never connect this branch to Send
No form data at the limit	`pending → expired`	Escalate or create a fresh approval request, never auto-approve

The Wait node automatically resumes when its configured time limit is reached. Check for a valid submitted decision before treating the execution as approved. Missing form data belongs on the timeout branch.

Email is the clearest reviewer channel for this example. Slack, Telegram, Google Chat, email, or a webhook can carry an alert, but do not assume every notification node has the same approval features. Keep the decision state and send guard in the workflow even when the alert channel changes.

Step 6: prevent duplicate clicks and duplicate sends

Before Gmail Send, add a Data Table Update node with All Conditions:

approval_id equals {{ approval_id }}
status equals approved OR approved_edited

Update:
status = sending

In practice, use separate approve and edited branches if the node cannot express the OR cleanly. Continue only when one row was updated. If the row is already sending, sent, rejected, or expired, stop. This conditional claim protects the normal single-execution path from a second click, duplicated callback, or later retry.

After Gmail succeeds, update the same row to sent and store the provider message ID and sent_at. Do not manually retry from the Gmail node. Restart recovery from the state check so the send guard runs again.

n8n documents conditional Data Table updates, but it does not promise compare-and-set or transaction semantics across competing executions. If two executions could claim the same approval at exactly the same time, use a database transaction, unique constraint, or another state store with an atomic claim. For a Gmail-specific alternative, create one provider draft and send that immutable draft ID through the Gmail API. Google documents that sending automatically deletes the draft and creates the Sent message, so the same existing draft is no longer available for a second send.

Gmail Message Send does not expose a provider idempotency key in the documented node fields. Add the approval ID to an internal reference in the subject or footer so an operator can search Sent mail. If the network times out and you cannot tell whether Gmail accepted the message, set status = send_unknown, search by that reference, and reconcile before any retry.

Handle failure after approval

Failure	Safe response
Recipient or payload validation fails	Return to review with a clear reason; do not claim the send
Credential or definite provider rejection	Mark `send_failed`, alert the owner, and repair before a guarded retry
Provider timeout with an uncertain result	Mark `send_unknown` and search the provider before retrying
Email sends, audit update fails	Keep the row at `sending`; reconcile by approval reference and provider message ID
Reviewer link expires	Create a new approval request and invalidate the old one

This is the same reason duplicate-safe API retries need a persistent ledger. A timeout is not proof of failure. Use the deeper n8n API retry guide when the final email provider is called through HTTP Request or another state-changing API.

Test the approval workflow before using real recipients

Test	Expected result
Approve unchanged	One email uses the proposed subject and body; audit row becomes `sent`
Edit subject and body	Only the human-approved values are delivered and stored
Choose Edit with no body	No send; request returns to review
Reject	No send; reason and reviewer are recorded
Let the request expire	No send; status becomes `expired` and the owner is alerted
Submit or click twice	One email; if parallel tests produce more, move the claim to a transactional store
Fail Gmail after approval	The workflow records failed or unknown, alerts an owner, and does not blind-retry
Retry the execution	A sent or sending state blocks another message

Privacy and audit rules

The Wait node offloads paused execution data to the n8n database and reloads it when the workflow resumes. That means every upstream field carried into the wait can remain in stored execution data. Reduce the item before the wait. Do not carry a full mailbox thread, attachments, payment details, health information, or unrelated customer history when the reviewer only needs a short excerpt and a secure system link.

Store the approval ID, decision, approved content hash or final content, reviewer, and timestamps.
Treat a reviewer name typed into an open form as self-reported, not authenticated identity.
Use n8n's production execution data redaction when operators should not see node inputs and outputs.
Set a retention policy for the approval table and execution history.
Keep successful execution data only as long as operations and audit needs require.

n8n enables pruning of finished execution data by default. Its documented self-hosted defaults are 14 days by age and 10,000 finished executions by count, while waiting executions are not eligible for pruning. The approval timeout is therefore an operating control as well as a reviewer deadline.

Practical questions

Can the AI send automatically after enough approvals?

You can later auto-send low-risk categories, but earn that exception with real review data. Start by measuring edit rate, rejection rate, send failures, and the types of mistakes reviewers catch. Keep high-value, legal, financial, hiring, complaint, and unusual-recipient messages behind approval.

Should the approval live in email or chat?

Use the channel the responsible person already monitors, but keep the durable state in the workflow or business system. An alert is not the audit record. For a simple decision, Gmail's built-in approval operation is convenient. For edits, timeouts, and stronger identity, use the Wait pattern or an authenticated portal.

Is the n8n execution history enough for an audit?

Usually not by itself. Execution retention can be changed or pruned, and open form identity may be self-reported. Keep a compact business audit row with the approval ID, final decision, final approved content or hash, reviewer identity level, timestamps, and provider result.

Sources checked

Demystifying ACID Transactions: How to Keep Your Database from Making a Mess

Thu, 01 Oct 2026 10:28:33 +0000

An ACID transaction is a set of four database safety rules—Atomicity, Consistency, Isolation, and Durability—that ensure database operations are executed reliably. It guarantees that even if your application crashes, loses power, or experiences thousands of concurrent requests, your data remains accurate and uncorrupted. Simply put, it ensures your database never ends up in a half-finished, broken state.

The Ticket Booth Analogy

Imagine buying a physical ticket at a busy movie theater box office. You hand the cashier a $15 bill, and they hand you the ticket. This exchange must happen as a single, indivisible unit. If you hand over your cash, but a sudden power outage occurs before the cashier can hand you the ticket, you would be furious if they kept your money. Conversely, the theater would lose out if you grabbed the ticket and walked away without paying.

The transaction is "all-or-nothing": either the cash changes hands and you get the ticket, or the entire deal is canceled, and you keep your money. ACID acts as this strict box-office coordinator for your software's database, ensuring that multi-step operations never stop halfway.

Why It Matters: Real-World Disasters

In daily software engineering, we rely on ACID transactions to prevent catastrophic data errors, especially when building systems like financial ledgers, e-commerce checkouts, or reservation platforms.

Without ACID, imagine a bank transfer system built on Node.js. When a user transfers $100, the system must perform two separate actions: deduct $100 from Account A, and add $100 to Account B. If the database server loses connection exactly halfway through this process, Account A loses money, but Account B never receives it. The $100 vanishes into the digital void.

Similarly, in a high-traffic retail store, if two customers try to buy the very last item in stock at the exact same millisecond, a non-ACID database might let both transactions complete. This leaves you with an oversold item and an angry customer. By grouping these operations inside an ACID transaction, developers ensure that if any step fails, the database automatically rolls back all changes, restoring everything to its original state.

Implementing ACID in Node.js and MySQL

Here is how we can implement a secure money transfer using Node.js and the standard mysql2/promise library. By wrapping our queries in a transaction block, we guarantee data safety.

const mysql = require('mysql2/promise');

async function transferFunds(fromAccountId, toAccountId, amount) {
  const connection = await mysql.createConnection({
    host: 'localhost',
    user: 'root',
    database: 'bank_db'
  });

  try {
    // Start the ACID transaction
    await connection.beginTransaction();

    // Step 1: Deduct from the sender's account
    const [deductResult] = await connection.execute(
      'UPDATE accounts SET balance = balance - ? WHERE id = ? AND balance >= ?',
      [amount, fromAccountId, amount]
    );

    if (deductResult.affectedRows === 0) {
      throw new Error('Insufficient funds or invalid sender account');
    }

    // Step 2: Add to the receiver's account
    const [creditResult] = await connection.execute(
      'UPDATE accounts SET balance = balance + ? WHERE id = ?',
      [amount, toAccountId]
    );

    if (creditResult.affectedRows === 0) {
      throw new Error('Invalid receiver account');
    }

    // If both steps succeed, permanently commit the changes to disk
    await connection.commit();
    console.log('Transaction completed successfully!');
  } catch (error) {
    // If anything fails, revert all changes immediately
    await connection.rollback();
    console.error('Transaction failed. Rolled back changes:', error.message);
  } finally {
    await connection.end();
  }
}

The Takeaway

ACID transactions are the ultimate insurance policy for your application's state. While they introduce a minor performance trade-off due to locking rows during updates, they completely eliminate the need to write complex, error-prone fallback code. Shifting the responsibility of data integrity to your database engine lets you sleep easy knowing your system will never tell a lie.

Originally published on my blog. You can read the alternative breakdown here.

Improve your Angular app performance by using this simple Observable cache decorator

Tue, 29 Sep 2026 10:53:16 +0000

When we were about to finish development of our applications in SwiftViews we noticed a pattern in all our data-fetching user flows. In spite of the fact that the apps are all data-driven and looked really dynamic, what really was changing in the same user session wasn’t that much, but we were making http requests for new content regardless.
The easiest solution was caching
Caching where? On the server? We already have that, but this doesn’t stop all our apps hitting our services and thus — increasing the load on them.
Maybe we could use service worker since it allows caching API calls?
Yes, that was one of the options since it already has a pretty nice integration with Angular and allowed for a simple solution to selectively cache resources and APIs. However, what we wanted is to be able to not only choose what to cache, but also when to do it.
Just to give you a quick example — we basically wanted to cache all API call results for the page below, but only if we were certain that the data source did not change in some way.

Since this is my personal inventory page, the only way it can actually change is if I add an item from this platform (currently the only way to do so), so I have the information about when this page will change and how long I can return cached content for.
We figured the best way to apply this caching in a selective and configurable way will be to use what the platform already gives us
Since we are using Angular and Typescript ❤️, and all our data calls go through RxJs, we figured that we can create a Cache Observable decorator, which we’d use to only give the caching power to certain methods.
So, before, this was our method which called our server to get the products on the page above:

and that became:

Notice that the @Cacheable() has been applied just to the method we wanted and was also passed cacheBusterObserver which is basically our Subject-based mechanism to tell this exact decorated method to relieve all its caches, when any value is emitted in that stream.
For example, the method below will “cache-bust” the caches of the method above, if the Observable it returns emits a value

This might be the most complex example we have but we also have other configurations implemented like maxCacheCount, maxAge, slidingExpiration and custom cache deciders and resolvers which give us a fine-grain control on what, when and how to cache. See those in the README file.
In the two gifs below, you can see the difference between our app’s performance without and with caching.

https://www.loom.com/share/71a03e4cd478407fa53f131fc112e09a
No Cacheable
The cacheable decorator is not applied yet, so every page load will actually fetch data from the server

https://www.loom.com/share/380a1fa93a9c42d0a5adaa987cd51efb
Cacheable decorator applied
The cacheable decorator is applied and we can see that all consecutive page loads are faster. Also the user profile page loads instantly, because we have already called the /user/{id}endpoint on the inventory page. Also, the cache busts after we add a new item so our user gets fresh data
So, to sum up, this simple decorator allowed us to:
Selectively cache observable methods (not just endpoints, but maybe also computation-heavy calculations in streams)
Improve the performance of our app, without any business logic refactoring (thanks TS decorators ❤️)
Greatly reduce the load on our servers
If you want to use this decorator in your project, just install it from NPM!
npm install ngx-cacheable
If you have any questions about it or want to contribute, don’t mind opening a pull request @github or commenting below.
Also, if you liked the application, please register and build your own inventory @ swiftviews.
Thank you!

Npm err! maximum call stack size exceeded

Sun, 27 Sep 2026 09:49:04 +0000

One of the most common issues that developers encounter while working with npm (Node Package Manager) is the err! maximum call stack size exceeded error. This error can be frustrating and confusing, but don't worry, you're not alone! In this article, we'll take a deep dive into what causes this error and how to fix it. So, whether you're a seasoned developer or just getting started with Node.js, read on to learn more about this common issue and how to resolve it.

What is call stack size?

Npm (short for Node Package Manager) is a package manager for the JavaScript programming language. It is the default package manager for the JavaScript runtime environment Node.js.

One common issue that users may encounter when using npm is the npm ERR! maximum call stack size exceeded error. This error occurs when the program's call stack size exceeds the maximum limit, causing the program to crash.

The call stack is a memory structure used by the JavaScript engine to track the execution of a program. Each time a function is called, a new frame is added to the stack, and when the function returns, the frame is removed. When the stack size exceeds the maximum limit, the program crashes to prevent a potential crash.

The npm err! maximum call stack size exceeded error is a common issue that can occur when using the Node Package Manager (npm) to install and manage packages for JavaScript projects. The error is caused by a maximum limit on the number of function calls that can be made in a single execution stack, which is known as the call stack limit. When this limit is exceeded, the error is thrown and the process is terminated.

Why the error: Npm err! maximum call stack size exceeded

There are a few different ways that this error can occur, but some common causes include:

Recursive dependencies

When a package has a dependency on another package, and that dependency in turn has a dependency on the first package, it can create a recursive loop that exceeds the call stack limit. An illustration of such a scenario can be seen below.


// moduleA.js
const moduleB = require('./moduleB');

// moduleB.js
const moduleA = require('./moduleA');

Large dependency trees

When a project has a large number of dependencies, the process of resolving and installing these dependencies can exceed the call stack limit.

Installing global packages

When installing packages globally using the -g flag, the process can exceed the call stack limit if the package being installed has a large number of dependencies.

Recursive function calls

When a function calls itself without a proper exit condition, it creates a new frame on the stack each time, causing the stack size to grow indefinitely. An illustration can be seen in the code below:


function infiniteRecursion() {
infiniteRecursion();
}
infiniteRecursion();

A large number of nested function calls

When a program has a large number of nested function calls, it can cause the stack size to grow quickly, eventually exceeding the maximum limit.

How to resolve the error

There are several ways to fix this issue, some of which include:

Updating npm: Updating to the latest version of npm can help resolve the issue, as new versions may have better handling of large dependency trees and recursive dependencies.
Removing unnecessary dependencies: Removing unnecessary dependencies from the project can help reduce the size of the dependency tree and lower the chance of exceeding the call stack limit.
Using the –max-old-space-size flag: Using this flag can increase the amount of memory available to the Node.js process, allowing it to handle larger dependency trees without exceeding the call stack limit. NODE_OPTIONS=--max-old-space-size=<enter_size_here>
Identifying the cause of the error by reviewing the code and looking for any potential sources of recursion or circular dependencies.
Refactor the code to remove recursion or circular dependencies.
Break down large functions into smaller, more manageable functions to reduce the number of nested function calls.
Using a package manager like Yarn: Yarn is another package manager that can be used instead of npm, which can handle large dependency trees more efficiently.

Note It is important to remember that this error may also occur due to the system limitations and not just the package manager. So, it is best to check the system requirements before proceeding with the project.

Conclusion

The npm err! maximum call stack size exceeded error is a common issue that can occur when using the Node Package Manager to install and manage packages for JavaScript projects. The error is caused by a maximum limit on the number of function calls that can be made in a single execution stack and can occur due to recursive dependencies, large dependency trees, and installing global packages. There are several ways to fix this issue, such as updating npm, removing unnecessary dependencies, using the –max-old-space-size flag, and using a package manager like Yarn.

References

https://docs.npmjs.com/files/npm-debug.log
https://nodejs.org/en/docs/guides/debugging-getting-started/
https://docs.npmjs.com/cli/install
https://yarnpkg.com/getting-started
JavaScript Call Stack: https://blog.bitsrc.io/understanding-the-javascript-call-stack-f9ce69e5ce9c
Increasing the call stack size limit: https://nodejs.org/en/docs/guides/debugging-getting-started/#increase-the-call-stack-size-limit

I built an RPG that teaches Claude Code by making you actually use it

Sat, 26 Sep 2026 14:26:04 +0000

title: I built an RPG that teaches Claude Code by making you actually use it
published: true
description: claude-quest is an in-terminal RPG for learning the Claude Code CLI. missions live in real sandbox directories, and progress is graded by checking what you actually did, not by quizzing you.
tags: ai, claude, opensource, node

series: Building claude-quest

Most tool documentation teaches by just telling you things. You read a page about
/model, or hooks, or subagents, nod along, and forget it by the time you'd
actually need it. I wanted something closer to how people actually learn a CLI,
by using it, with something checking whether you did the thing right or not.

So I built claude-quest, a text RPG that runs entirely inside a real
claude session and teaches the Claude Code CLI zero to hero.

the idea isn't new, I borrowed it

This is basically GameShell's philosophy
applied to Claude Code. GameShell teaches Unix shell commands by dropping you
into a real shell wrapped in a themed fake filesystem, and grading your
progress by checking real shell/filesystem state instead of asking you to self
report or answer a quiz question.

claude-quest does the same thing, except the "filesystem" is a real Claude
Code environment. missions live in real sandbox directories, and progress is
checked by inspecting what you actually did, config files you wrote, hooks
that fired, tool calls that happened.

what it actually looks like

There's no separate app, no fancy terminal UI, none of that. You say "let's
play claude quest" inside a normal claude session, and Claude itself becomes
the game master. It reads real output from the engine and narrates on top of
it, something like this:

> lets play claude quest

**The Gatehall**

You've just been let through the outer gate. The tower keeps no secrets
from those who bother to read the walls...

**Mission: First Contact** (Tier 1, Artifact)

Create a CLAUDE.md file in this sandbox recording how to run the
project's tests. Let me know when you're done and I'll check it.

> done

CLAUDE.md records how to run the tests. Claude Code will know next time.
MISSION_STATUS: complete

**What you actually learned:** Claude Code reads a file named CLAUDE.md
in your project root automatically, at the start of every session...

that's it, that's the whole interface. it's just a chat, with real commands
running underneath it and real files getting checked. nothing gets rendered,
nothing gets animated, it's plain text the way claude sessions already look.

missions are graded on real state, not vibes

Every mission is one of four tiers, and the grading gets less mechanical the
further you go:

Artifact, write or edit a file (a CLAUDE.md, a config) and the engine
checks the file itself.
Invocation, actually run a claude session and do the thing (switch
models, compact, check cost). the engine checks what landed afterward
(like .claude/settings.json), never what you said in the session.
Mastery, fix something that's conceptually broken (a skill whose
description doesn't say what its for, a subagent that won't fire), then
the engine runs a held-out test battery against your own claude CLI.
Judgment, write something graded on a rubric (a commit message, a
skill description) and an isolated judge call scores it per criteria.

None of these ask you to type an answer into a prompt, they ask you to leave
a trace and then the engine checks the trace.

try it

Inside any claude session:

/plugin marketplace add susheem-k/claude-quest
/plugin install claude-quest@claude-quest

Then just say "let's play claude quest." Progress saves to ~/.claude-quest/,
so it doesn't really matter which project your in when you pick it back up.

where it's at, not all the levels exist yet

This is a playable alpha, not a finished game. The full loop already works
(characters, saves, hints, grading for all four tiers) but the campaign
doesn't cover everything Claude Code can do yet. Right now there's only a
handful of example missions, one or two per tier, across five arcs. More
missions are getting added.

And since this is open source, that part doesn't have to be just me. If
there's a Claude Code feature you wish more people knew how to use, a
command, a hook, a workflow you had to learn the hard way, I'd genuinely like to turn it into a mission. The mission format is just a directory with a goal, a hint, and a check.js (the design doc has the exact file contract for it), so adding one doesn't require touching
the engine at all.

Open an issue with a
mission idea, or just send a PR with one built already, both are welcome.

if you try it and it teaches you something, a star on the repo genuinely
helps, it's how other people find it.

Repo: https://github.com/susheem-k/claude-quest

Route flags, context decorators and typed assets in KickJS: declare it once, use it everywhere

Tue, 22 Sep 2026 11:40:21 +0000

Every API accumulates the same kind of glue. "This endpoint is public" gets written three times — once for auth, once for CSRF, once for rate limiting — and two of those are path strings that break the day someone renames a route. "Who is the current user" becomes req.user as any. "Where is the welcome email template" becomes __dirname arithmetic that works in dev and fails in the Docker image.

KickJS has three features that remove that glue, and they are far more useful together than apart:

Feature	What it answers	Mental model
Route flags	What is true about this route?	A fact, written once on the route, read by anything. No behaviour.
Context decorators	What do we know about this request?	A typed value computed per request, before the handler runs.
Asset manager	Where is that file?	A typed path to a file shipped with your code, correct in dev and in the build.

This guide explains each one, then builds a small feature set that uses all three at once: public and private routes, a signed-in user, plan-gated endpoints, per-route rate limits, and locale-aware emails and reports rendered from typed templates.

Versions: @forinda/kickjs 8.4+, @forinda/kickjs-cli 8.2+, @forinda/kickjs-testing 8.0+. All three features are in the core package — nothing extra to install.

1. Route flags: facts about a route

A route flag is a named, inheritable fact about a route. It does nothing by itself; it records something any consumer can read.

// src/flags.ts
import { defineRouteFlag } from '@forinda/kickjs'

export const Public = defineRouteFlag('auth.public')
export const RateLimit = defineRouteFlag<{ rpm: number }>('rate.limit')
export const Feature = defineRouteFlag<string>('billing.feature')

Put flags on a controller, a method, or a module mount. The most specific declaration wins — method > class > mount:

@Public // every route in this controller is public…
@Controller()
export class DocsController {
  @Get('/')
  index(ctx: RequestContext) {} // public

  @Public.off // …except this one
  @Get('/drafts')
  drafts(ctx: RequestContext) {} // not public
}

Read them anywhere you hold a RequestContext — handlers, @Middleware(), guards, contributors:

ctx.route?.flags.has('auth.public') // true / false
ctx.route?.flags.get('rate.limit') // { rpm: 10 } or undefined

Two rules make flags predictable:

Removal is .off, never a falsy value. A flag is either present or absent. @Public.off removes it; there is no "present but false" state that a presence check could misread as "public".
false is a real value. A flag declared with a boolean type can store false, and flags.get() returns it. Only .off removes.

ctx.route is undefined in global middleware, which runs before a route is matched (section 8 shows how pre-routing middleware reads flags anyway).

2. Context decorators: typed values about a request

A context decorator (also called a contributor) computes a value from the request and stores it on the context before the handler runs. It replaces middleware whose only job was "compute X and stash it on the request".

Scaffold one:

kick g contributor locale --dry-run
kick g contributor locale -o src/contributors

Fill it in:

// src/contributors/locale.contributor.ts
import { defineHttpContextDecorator } from '@forinda/kickjs'

// 1. Declare the key's type once, for the whole app.
declare module '@forinda/kickjs' {
  interface ContextMeta {
    locale: { language: string; region: string | null }
  }
}

// 2. Define how it's computed.
export const ResolveLocale = defineHttpContextDecorator({
  key: 'locale',
  resolve: (ctx) => {
    const header = (ctx.req.headers['accept-language'] as string | undefined) ?? 'en'
    const [language, region] = header.split(',')[0].trim().split('-')
    return { language: language.toLowerCase(), region: region ?? null }
  },
})

Use it:

@ResolveLocale
@Get('/home')
home(ctx: RequestContext) {
  const locale = ctx.require('locale') // { language: string; region: string | null }
}

What you get over hand-written middleware:

Concern	Middleware	Context decorator
Type of the value	`(req as any).locale`	`ctx.require('locale')`, typed from `ContextMeta`
Ordering	Array position in `bootstrap()`	`dependsOn: ['user']`, sorted at boot; cycles and missing producers fail startup
Services	`Container.getInstance()` inside the body	`deps: { repo: REPO_TOKEN }`, resolved and typed
Per-route opt-out	Fork the stack or check paths	`skipWhen` / `onlyWhen` with route flags
Errors	Throw → 500	`optional: true`, `onError` fallback, or `HttpException`
Testing	Whole HTTP stack	`runContributor()` against a fake context

Registration sites, highest precedence first: method > class > module contributors() > adapter/plugin contributors() > bootstrap({ contributors }). A narrower registration of the same key silently replaces a broader one.

Reading values:

ctx.require('key') — for values the route must have. Throws MissingContextValueError naming the key and route if the contributor didn't run. Never write ctx.get('key')!: it compiles even after someone deletes the decorator.
ctx.get('key') — for optional extras. Returns T | undefined.
getRequestValue('key') — from a service with no ctx in scope. Returns undefined outside a request.

Always return the value from resolve(). Assigning ctx.locale = … writes to one internal context object that the handler never sees.

3. The asset manager: typed paths to your files

Templates, report layouts, JSON schemas and fixtures live next to your code in development and get copied somewhere else in the build. Without help, every call site does this:

const path =
  process.env.NODE_ENV === 'production'
    ? join(__dirname, '../templates/mails/welcome.ejs')
    : join(__dirname, '../../src/templates/mails/welcome.ejs')

Typos compile, every call site branches on the environment, and __dirname breaks under ESM and bundling.

Configure

Tell the CLI which directories hold assets, in kick.config.ts:

import { defineConfig } from '@forinda/kickjs-cli'

export default defineConfig({
  assetMap: {
    mails: { src: 'src/templates/mails', glob: '**/*.ejs' },
    reports: { src: 'src/templates/reports', glob: '**/*.{ejs,html}' },
    schemas: { src: 'src/schemas', glob: '**/*.json' },
  },
})

Each entry takes src (required), dest (default dist/<name>/), glob (default **/*) and keys ('auto', 'strip' or 'with-extension' — how file names become property names).

Generate types

kick typegen   # kick dev runs this automatically on every change

For a file tree like:

src/templates/mails/en/welcome.ejs
src/templates/mails/fr/welcome.ejs
src/templates/reports/monthly.ejs

you get a typed surface:

import { assets } from '@forinda/kickjs'

assets.mails.en.welcome() // absolute path to the right file, dev or build
assets.reports.monthly()
assets.mails.en.welcom() // compile error

Four ways to read an asset

All four go through the same cached resolver:

import { assets, useAssets, Asset, resolveAsset } from '@forinda/kickjs'

assets.reports.monthly() // 1. ambient proxy — the default

class ReportService {
  private assets = useAssets() // 2. as a field — easy to mock in tests
}

@Service()
class InvoiceService {
  @Asset('reports/invoice') // 3. declarative field, resolved on each access
  private invoiceTemplate!: string
}

resolveAsset('mails', 'fr/welcome') // 4. dynamic — names known only at runtime; throws UnknownAssetError on a miss

In the build

kick build copies every assetMap entry into the output directory, writes a manifest (.kickjs-assets.json) and runs typegen. At runtime the resolver uses that manifest when it exists, and walks your src directories when it doesn't — so the same assets.mails.en.welcome() call works in kick dev and in production. After adding a template in dev, kick build:assets refreshes the manifest without a full rebuild.

4. How they fit together

The three features form a natural pipeline for any request:

route matched
   │
   ├─ route flags resolved (method > class > mount)          "what is true about this route"
   │
   ├─ contributors run, in dependsOn order                   "what we know about this request"
   │     • skipWhen / onlyWhen read the flags
   │     • resolve() can read ctx.route.flags values
   │     • guards read flags via exemptWhen
   │
   └─ handler / services
         • ctx.require(...) / getRequestValue(...)
         • assets.* picks the file, often chosen by a contributor value

The division of labour that keeps codebases clean:

Flags describe — "public", "needs the reports feature", "limit 10 rpm". Declarative, inheritable, visible in the route list.
Contributors decide — load the user, load the plan, check the feature, compute the locale. Typed, ordered, testable.
Assets deliver — the template or schema the decision points at, with a path that is always right.

The recipes below build that pipeline piece by piece.

5. Recipe: one "public" flag, every consumer

Declare the fact once:

// src/flags.ts
export const Public = defineRouteFlag('auth.public')

Put it on routes:

@Public
@Controller()
export class AuthController {
  @Post('/login') login(ctx: RequestContext) {}
  @Post('/register') register(ctx: RequestContext) {}

  @Public.off // the only private route here
  @Get('/me')
  me(ctx: RequestContext) {}
}

Now every consumer reads that one declaration instead of its own path list:

// Auth: the user contributor steps aside on public routes (section 6).
skipWhen: 'auth.public'

// CSRF and rate limiting, as route guards:
@Middleware(csrfGuard({ exemptWhen: 'auth.public' }))
@Middleware(rateLimitGuard({ max: 60, exemptWhen: 'auth.public' }))

// OpenAPI: public routes show no lock icon.
SwaggerAdapter({ bearerAuth: true, publicFlag: 'auth.public' })

// The built-in health probes, which you don't own:
bootstrap({ health: { flags: ['auth.public'] } })

Rename a route, change the API prefix, add /users/:id — nothing breaks, because nothing matches on path strings. And DevTools shows each route's resolved flags in its Routes tab, so "why is this endpoint open?" is answered from the route list.

The framework names no flags. auth.public is a name you chose — pick a namespace per concern (auth.*, billing.*, rate.*) so readers can tell who consumes a flag.

6. Recipe: the signed-in user, skipped on public routes

// src/contributors/load-user.contributor.ts
import { defineHttpContextDecorator, HttpException } from '@forinda/kickjs'
import { SESSIONS } from '@/auth/sessions'

declare module '@forinda/kickjs' {
  interface ContextMeta {
    user: { id: string; accountId: string; roles: string[] }
  }
}

export const LoadUser = defineHttpContextDecorator({
  key: 'user',
  skipWhen: 'auth.public', // the flag from section 5
  beforeValidation: true, // answer 401 before 422
  deps: { sessions: SESSIONS },
  resolve: async (ctx, { sessions }) => {
    const user = await sessions.fromToken(ctx.headers.authorization)
    if (!user) throw HttpException.unauthorized()
    return user
  },
})

bootstrap({ modules, contributors: [LoadUser.registration] })

Every route now requires a user, except routes flagged public. Three details matter:

skipWhen makes exemption work across ownership. Without flags, the only way to opt a route out of a contributor is to register a permissive twin under the same key — which you can't do for a contributor a plugin shipped. A flag lives on the route, so you can exempt anything.
beforeValidation: true moves the contributor ahead of body validation. Otherwise an anonymous request with a bad body gets a 422 listing your schema. The order becomes beforeValidation contributors → validation → @Middleware() → other contributors → handler. Read headers and cookies here, never the body — it isn't validated yet.
A skipped contributor sets no key. On routes that can be public, read ctx.get('user'); use ctx.require('user') only where the route is guaranteed private.

Services that need the user don't take it as a parameter:

@Service()
export class AuditService {
  record(action: string) {
    const user = getRequestValue('user') // typed, undefined on public routes
    this.log.insert({ action, userId: user?.id ?? null })
  }
}

7. Recipe: plan-gated routes with a flag value and dependsOn

Goal: some routes require a paid feature. The route states which feature; contributors enforce it.

The flag carries a value:

export const Feature = defineRouteFlag<string>('billing.feature')

Load the account's plan after the user:

// src/contributors/load-plan.contributor.ts
declare module '@forinda/kickjs' {
  interface ContextMeta {
    plan: { name: 'free' | 'pro' | 'enterprise'; features: string[] }
  }
}

export const LoadPlan = defineHttpContextDecorator({
  key: 'plan',
  dependsOn: ['user'], // runs after LoadUser; boot fails if nothing produces 'user'
  skipWhen: 'auth.public',
  deps: { billing: BILLING },
  resolve: async (ctx, { billing }) => billing.planFor(ctx.require('user').accountId),
  onError: () => ({ name: 'free', features: [] }), // billing down → treat as free, don't 500
})

Enforce the feature only where a route asks for one:

// src/contributors/require-feature.contributor.ts
declare module '@forinda/kickjs' {
  interface ContextMeta {
    feature: string
  }
}

export const RequireFeature = defineHttpContextDecorator({
  key: 'feature',
  onlyWhen: 'billing.feature', // routes without the flag skip this entirely
  dependsOn: ['plan'],
  resolve: (ctx) => {
    const needed = ctx.route?.flags.get('billing.feature') // the flag's value, e.g. 'reports'
    const plan = ctx.require('plan')
    if (!needed || !plan.features.includes(needed)) {
      ctx.problem.forbidden({
        type: 'https://example.com/probs/upgrade-required',
        detail: `This needs the ${needed} feature, which is not on the ${plan.name} plan`,
      })
    }
    return needed
  },
})

bootstrap({
  modules,
  contributors: [LoadUser.registration, LoadPlan.registration, RequireFeature.registration],
})

Now gating a route is one line, and it reads like documentation:

@Controller()
export class ReportsController {
  @Feature('reports')
  @Get('/monthly')
  monthly(ctx: RequestContext) {} // pro and above

  @Get('/summary')
  summary(ctx: RequestContext) {} // everyone signed in
}

Why this shape works:

Order is declared, not arranged. dependsOn gives user → plan → feature. Forget LoadUser and the app refuses to start with MissingContributorError instead of failing on the first request.
Cost is paid only where needed. onlyWhen means routes without the flag never evaluate the check; a skipped contributor costs a map lookup.
Errors keep their shape. ctx.problem.forbidden(...) answers RFC 9457 application/problem+json from inside a contributor, so clients can branch on type.
The fallback is explicit. onError returns a value; the runner stores it under the key.

8. Recipe: per-route rate limits from one declaration

A flag with a value, read by several consumers:

export const RateLimit = defineRouteFlag<{ rpm: number }>('rate.limit')

@RateLimit({ rpm: 10 })
@Post('/login')
login(ctx: RequestContext) {}

Inside the route — a guard can exempt unmetered routes with a predicate that reads the value:

@Middleware(
  rateLimitGuard({
    max: 60,
    exemptWhen: ({ flags }) => flags.get('rate.limit')?.rpm === 0,
  }),
)

A contributor can derive the limiter key and window from the same flag:

declare module '@forinda/kickjs' {
  interface ContextMeta {
    rateLimit: { key: string; rpm: number }
  }
}

export const RateLimitKey = defineHttpContextDecorator({
  key: 'rateLimit',
  onlyWhen: 'rate.limit',
  resolve: (ctx) => ({
    key: `${ctx.get('user')?.id ?? ctx.req.ip}:${ctx.route?.path}`,
    rpm: ctx.route?.flags.get('rate.limit')?.rpm ?? 60,
  }),
})

Before routing — app-wide middleware has no ctx.route, so it reads a policy table the app builds at boot from every mounted route's method, path and flags:

bootstrap({ middlewares: [rateLimit({ max: 60, exemptWhen: 'auth.public' })] })

Your own pre-routing middleware can read the same table:

import { bindRoutePolicy, type RoutePolicyTable } from '@forinda/kickjs'

export function accessLog() {
  let policy: RoutePolicyTable | undefined
  const handler = (req, _res, next) => {
    const flags = policy?.lookup(req.method, req.url ?? '/')
    if (!flags?.has('audit.skip')) log.info({ method: req.method, url: req.url })
    next()
  }
  return bindRoutePolicy(handler, (table) => {
    policy = table
  })
}

A request that matches no route matches no flags, and stays limited and logged — which is exactly what you want for scanners probing random paths.

The three shapes a flag test can take

Every skipWhen, onlyWhen and exemptWhen accepts:

'auth.public' // carries this flag
'!auth.public' // does NOT carry it
['auth.public', 'health.probe'] // carries ANY of these
['!auth.public', '!health.probe'] // carries NONE of these
({ flags, route }) => flags.has('a') && flags.has('b') // anything else

A list is single-polarity: ['auth.public', '!metered'] is a compile error, because "any-of" would read it as "public or not metered" while most people read "and". Use a predicate for mixed conditions, and keep predicates cheap — they run per request.

9. Recipe: flag or parameter? Choosing the right tool

Contributors can also take per-call parameters, which overlaps with flags. The rule of thumb:

Use a flag when the fact is read by several consumers, or by tooling (OpenAPI, DevTools), or must be set on routes you don't own.
Use a parameter when it only configures one contributor's behaviour at that call site.

A permission check is a good parameter — only one contributor cares, and the action must be named at every use:

declare module '@forinda/kickjs' {
  interface ContextMeta {
    permission: string
  }
}

export const RequirePermission = defineHttpContextDecorator.withParams<{ action: string }>()({
  key: 'permission',
  dependsOn: ['user'],
  // no paramDefaults.action — every route must name its own action
  resolve: (ctx, _deps, { action }) => {
    if (!ctx.require('user').roles.includes(action)) throw HttpException.forbidden()
    return action
  },
})

@RequirePermission({ action: 'reports:export' }) // ✓
@Get('/export')
export(ctx: RequestContext) {}

@RequirePermission // ✗ compile error: `action` is required

Required parameters have no default on purpose: a placeholder default that one route forgets to override silently gates on the placeholder. For module, adapter or bootstrap registration, use RequirePermission.with({ action: '…' }).registration.

Compare with billing.feature from section 7: that one is a flag because the billing contributor, the OpenAPI spec (to document "requires Pro"), and DevTools all benefit from reading it.

Need	Reach for
"This route is public / metered / exempt"	Flag
"Requires feature X" (and docs should say so)	Flag with a value
"Check permission `reports:export`"	Parameterised contributor
"Load the tenant from a header here, a subdomain there"	Parameterised contributor
Mark routes on a controller from another package	Flag on the module mount

10. Recipe: locale-aware emails from typed templates

Now bring in assets. Keep templates per language:

src/templates/mails/en/welcome.ejs
src/templates/mails/en/password-reset.ejs
src/templates/mails/fr/welcome.ejs

// kick.config.ts
assetMap: {
  mails: { src: 'src/templates/mails', glob: '**/*.ejs' },
},

After kick typegen, assets.mails.en.welcome() and assets.mails['en']['password-reset']() are typed.

The locale contributor (section 2) decides the language per request. The mail service picks the matching template, falling back to English when a translation is missing:

// src/mail/mail.service.ts
import { Service, assets, getRequestValue, resolveAsset, UnknownAssetError } from '@forinda/kickjs'
import ejs from 'ejs'

type MailName = 'welcome' | 'password-reset'

@Service()
export class MailService {
  async render(name: MailName, data: Record<string, unknown>) {
    const language = getRequestValue('locale')?.language ?? 'en'
    return ejs.renderFile(this.templateFor(name, language), data)
  }

  private templateFor(name: MailName, language: string): string {
    try {
      return resolveAsset('mails', `${language}/${name}`) // dynamic: language is runtime data
    } catch (error) {
      if (error instanceof UnknownAssetError) return assets.mails.en[name]() // typed fallback
      throw error
    }
  }
}

Look at who does what:

The contributor turns a header into a typed locale once per request.
The service reads it with getRequestValue — no ctx threaded through three layers.
The asset manager turns fr/welcome into a real path in dev and in the build, and the typed assets.mails.en[name]() fallback means English templates are checked at compile time.

Outside a request (a queue worker, a cron job) getRequestValue returns undefined, and the service falls back to English instead of throwing.

11. Recipe: reports rendered per route

Report templates are a good fit for the @Asset field decorator — fixed files, injected into a service:

// kick.config.ts
assetMap: {
  reports: { src: 'src/templates/reports', glob: '**/*.ejs' },
},

@Service()
export class ReportService {
  @Asset('reports/monthly')
  private monthlyTemplate!: string

  @Asset('reports/invoice')
  private invoiceTemplate!: string

  monthly(data: MonthlyReport) {
    return ejs.renderFile(this.monthlyTemplate, data)
  }
}

Combined with section 7, the controller carries all the policy as declarations:

@Controller()
export class ReportsController {
  @Autowired() private readonly reports!: ReportService

  @Feature('reports') // flag: needs the reports feature
  @RateLimit({ rpm: 5 }) // flag: expensive, limit it
  @ResolveLocale // contributor: formats numbers and dates for the reader
  @Get('/monthly')
  async monthly(ctx: RequestContext) {
    const html = await this.reports.monthly({
      user: ctx.require('user'),
      plan: ctx.require('plan'),
      locale: ctx.require('locale'),
    })
    return ctx.html(html)
  }
}

Reading the decorators top to bottom tells you everything about the endpoint without opening a single middleware file: who can call it, how often, and what context it renders with.

JSON schemas and fixtures work the same way — assets.schemas.webhook() gives a validator the right file in every environment.

12. Recipe: flagging controllers you don't own

Decorators only work on code you can edit. A module can flag routes on any controller it mounts — including one from a package:

// src/modules/webhooks/webhooks.module.ts
import { defineModule } from '@forinda/kickjs'
import { StripeWebhookController } from 'some-payments-package'

export const WebhooksModule = defineModule({
  name: 'WebhooksModule',
  build: () => ({
    routes: () => ({
      path: '/webhooks',
      controller: StripeWebhookController,
      flags: { 'auth.public': true, 'rate.limit': { rpm: 600 } },
    }),
  }),
})

Mount flags are the lowest precedence, so a class or method declaration of the same flag still wins. A list form (flags: ['auth.public']) stores each flag as true.

The same idea covers the framework's own health probes:

bootstrap({ health: { flags: ['auth.public'] } })

Every consumer already reading auth.public — LoadUser, the guards, the OpenAPI spec — now exempts /health/live and /health/ready, with no path strings.

13. Type safety: what the compiler catches

kick typegen (automatic in kick dev) generates three registries into .kickjs/types/:

You write	Typegen generates	The compiler then catches
`defineRouteFlag('rate.limit')` with a value type	`KickRouteFlags`	`skipWhen: 'auth.pubic'` → Did you mean 'auth.public'?; `flags.get('rate.limit')?.rpm` typed, not `unknown`
`declare module … ContextMeta`	— (you write it)	`ctx.require('locale')` typed; `dependsOn: ['usr']` is an error
`assetMap` in `kick.config.ts`	`KickAssets`	`assets.mails.en.welcom()` is an error

And one more that is easy to miss: a deleted decorator becomes a compile error. When a handler is typed with a generated route type, typegen narrows ctx.require() to the keys that route provably has:

@LoadTenant
@RequirePermission({ action: 'audit:read' }) // ← delete this line…
@Get('/audit')
audit(ctx: Ctx<KickRoutes.AuditController['audit']>) {
  return { perm: ctx.require('permission') } // …and this no longer compiles
}

Typegen narrows only when it can prove the whole contributor set. Adapter or plugin contributors() can't be read statically, so projects using them skip the narrowing — ctx.require() still throws at runtime, which is why integration tests stay worth having.

All three registries are opt-in: until you declare your first flag, ContextMeta key or assetMap entry, everything accepts plain strings and keeps compiling.

14. Testing all three

Contributors, in isolation

runContributor from @forinda/kickjs-testing calls a contributor's resolve() against a fake context — no container, no HTTP. Hand it mocked deps and pre-seed upstream keys with initial:

import { describe, expect, it, vi } from 'vitest'
import { runContributor } from '@forinda/kickjs-testing'
import { LoadPlan } from './load-plan.contributor'

describe('LoadPlan', () => {
  it('loads the plan for the signed-in account', async () => {
    const billing = { planFor: vi.fn(async () => ({ name: 'pro', features: ['reports'] })) }

    const { value } = await runContributor(LoadPlan, {
      deps: { billing },
      initial: { user: { id: 'u1', accountId: 'acc-9', roles: [] } }, // what LoadUser would have set
    })

    expect(billing.planFor).toHaveBeenCalledWith('acc-9')
    expect(value).toEqual({ name: 'pro', features: ['reports'] })
  })
})

runContributor calls resolve() directly, so optional and onError paths need the pipeline: build one with buildPipeline([...]) and run it with runContributors(...) from @forinda/kickjs.

Flags, without a request

getRouteFlags(controllerClass, handlerName) returns the same resolved flags a live request sees — method over class over mount:

import { getRouteFlags } from '@forinda/kickjs'

it('keeps /me private while the rest of AuthController is public', () => {
  expect(getRouteFlags(AuthController, 'login').has('auth.public')).toBe(true)
  expect(getRouteFlags(AuthController, 'me').has('auth.public')).toBe(false)
})

A test like this is cheap insurance on the one flag whose removal is a security bug.

The whole pipeline

createTestApp from @forinda/kickjs-testing boots the app with real contributors, flags and ordering; drive it with Supertest and assert the outcomes: 401 without a token, 200 on a public route, 403 with the upgrade problem type on a gated route for a free plan. createTestApp also sets up the request scope, so services calling getRequestValue work in these tests.

Assets, with fixtures

Point the resolver at a fixture manifest and clear its cache:

import { join } from 'node:path'
import { clearAssetCache } from '@forinda/kickjs'

beforeEach(() => {
  process.env.KICK_ASSETS_ROOT = join(__dirname, 'fixtures/assets') // contains .kickjs-assets.json
  clearAssetCache()
})

afterEach(() => {
  delete process.env.KICK_ASSETS_ROOT
  clearAssetCache()
})

@Asset fields resolve on every access, so existing service instances pick up the fixtures without being recreated.

15. Pitfalls

Symptom	Cause	Fix
A "private" route is treated as public	`@Public(false)` style thinking	Remove with `@Public.off`; flags have no false state
`@Public.off` does nothing	The route never inherited the flag	Check where you expected it to come from
Handler sees `undefined` for a contributor value	Assigned `ctx.x = …` in `resolve`/`onError`	Return the value; the runner stores it
`ctx.require('user')` throws on some routes	The contributor was skipped by `skipWhen`	Use `ctx.get('user')` on routes that can be public
Anonymous callers see validation errors	Auth contributor runs after validation	`beforeValidation: true`
`MissingContributorError` at boot	A `dependsOn` key has no producer for that route	Register the producer at the same or broader site
`beforeValidation` contributor fails boot	It depends on a normal contributor	Early contributors may only depend on early ones
Flag test typo matches nothing	Flags not declared yet	Declare with `defineRouteFlag`; typegen narrows every name
`['a', '!b']` won't compile	Lists are single-polarity	Use a predicate
Asset works in dev, missing in production	File not matched by `glob`, or custom `build.outDir`	Widen the glob; set `build.outDir` in `kick.config.ts`
Two files, one asset key	Same basename (`index.html` + `index.pug`)	Default `keys: 'auto'` keeps extensions for that group
New template not found in dev	Manifest is stale	`kick build:assets`
`welcome-email.ejs` needs bracket access	Non-identifier file name	`assets.mails['welcome-email']()`, or rename the file

The CLI commands used

Command	What it does here
`kick g contributor <name>`	Scaffolds a context decorator with its `ContextMeta` stub (`--type bare` for transport-agnostic, `--params "action:string"` for the parameterised form, `-m <module>` to place it in a module).
`kick typegen`	Generates `KickRouteFlags`, route types for `ctx.require()` narrowing, and `KickAssets`.
`kick dev`	Runs the app with hot reload and typegen on every change.
`kick build`	Builds, copies `assetMap` files, writes the asset manifest and runs typegen.
`kick build:assets`	Refreshes the asset manifest without a JS rebuild.
`--dry-run`	Previews any generator's files first.

Recap

Route flags are facts on routes: declared once, inherited method > class > mount, removed with .off, and read by contributors, guards, pre-routing middleware, OpenAPI and DevTools.
Context decorators are typed per-request values: ordered with dependsOn, wired to services with deps, skipped per route with skipWhen / onlyWhen, run early with beforeValidation, parameterised with .withParams.
The asset manager gives typed paths to shipped files: assetMap in config, assets.* in code, correct in dev and in the build.
Together: flags describe the route, contributors decide about the request, assets deliver the file the decision points at — and typegen checks all three.

I migrated a 593-route Express app to KickJS twice — here's what held up

Tue, 22 Sep 2026 11:38:03 +0000

I spent a while moving a large multi-tenant SaaS platform off Express 4 +
Sequelize. The numbers going in:

54 apps, ~593 routes
241 tables across 434 migrations
database-per-tenant — one Postgres database per customer organisation
one 3,046-line controller, and a route table with 26 endpoints that had never
once executed

I did it twice. First as a strangler — KickJS mounted inside the running
Express app, serving /api/v2 beside the legacy /api/v1, proven route by route
against the original. Then, once that had taught me what the system actually
does, as a clean rewrite on a fresh KickJS host with Drizzle.

Both runs used the same framework, and that is the thing I want to write about.
Most frameworks are good at one of those two jobs.

Versions throughout: @forinda/kickjs@8.3.1, @forinda/kickjs-cli@8.1.5,
Express 5.1, zod 4, Drizzle 0.45. Code examples use a generic
catalog/orders domain.

The feature that made the strangler possible

A KickJS module's routes() can return a controller or a raw Express router,
and you can mount both under the same path:

routes() {
  return [
    { path: '/catalog', controller: CatalogController }, // the routes I'd converted
    { path: '/catalog', router: legacyCatalogRouter },   // the ones I hadn't
  ]
}

Express takes the first match, so converted routes win and the legacy router
serves the rest. No unconverted route registration has to be copied, and a
route moves across one at a time by adding it to the controller and deleting
nothing.

That is what let a 593-route app migrate incrementally instead of in a big bang.
I haven't found another structured Node framework where "here is an Express
Router, serve it beside my controllers" is a first-class mount.

The inverse also worked, and is undocumented: new Application() + setup()
gives you the pipeline without a listener, and handle() takes raw Node
IncomingMessage/ServerResponse — which is how a KickJS app ran inside an
Express 4 host despite the framework declaring an Express 5 peer.

Context contributors — and how to write one

This is the part I'd steal for any other project. A contributor computes one
typed value per request and publishes it on the context under a key. Here is a
complete one, in the three pieces it always has.

1. Declare the key and its type, by augmenting the framework's registry.
This is what makes ctx.require('actor') typed everywhere downstream:

import { defineHttpContextDecorator, HttpException } from '@forinda/kickjs'

export interface Actor {
  readonly userId: string
  readonly organizationId: string
  readonly isAdmin: boolean
}

declare module '@forinda/kickjs' {
  interface ContextMeta {
    actor: Actor
  }
}

2. Define the resolver. deps pulls services out of the DI container,
dependsOn declares which other contributors must run first, and skipWhen
names the route flags that switch it off entirely:

export const Actor = defineHttpContextDecorator({
  key: 'actor',
  skipWhen: ['auth.public'],
  dependsOn: ['tenantDb'],
  deps: { repo: AUTH_REPOSITORY },
  async resolve(ctx, { repo }): Promise<Actor> {
    const token = bearerToken(ctx.req.headers.authorization)
    if (!token) throw new HttpException(401, 'Authentication required')

    const claims = await verifyAccessToken(token)
    if (!claims) throw new HttpException(401, 'Authentication required')

    return repo.loadActor(claims.sub)
  },
})

Throwing from resolve is the rejection — there's no separate guard step.

3. Register it, either globally in bootstrap({ contributors }), on a module
when every route in it shares the value, or per handler as a decorator:

contributors() {
  return [Actor.registration, TenantDb.registration]
}

Two things earn their keep:

The dependency is declared, so it fails at boot. The legacy app had two
different producers of the same tenant-database key — one from an x-tenant-id
header, one from the bearer token. Declaring dependsOn caught a real bug on the
first run: two routes carried the token-based resolver with no authenticated
user, which would have handed the service undefined at runtime. Missing
dependencies and cycles fail at startup, not per request.

Precedence is method > class > module > adapter > bootstrap. Register once on
the module when every route shares a value; override per handler when one
doesn't. In the rewrite the whole tenant story is a chain the framework
topo-sorts for you — order in the array is irrelevant:

tenantRoute → tenantDb → actor → permissions

Compare with the Express version, which resolved the tenant in the auth
middleware by copying the tenant middleware's logic inline, then wrote the
result to four differently-named request properties. Both are "middleware". Only
one of them tells you when you've wired it wrong.

Parameterised decorators

The same primitive builds a decorator that takes arguments. withParams and
requiredParams are the whole trick:

declare module '@forinda/kickjs' {
  interface ContextKeys {
    // Key-only: the value is the decision, and nothing downstream reads it.
    permissionGranted: true
  }
}

export const RequirePermission = defineHttpContextDecorator.withParams<{
  action: ActionName
  module: ModuleName
}>()({
  key: 'permissionGranted',
  dependsOn: ['permissions'],
  // Makes a bare `@RequirePermission` a boot error rather than a route that
  // silently checks nothing.
  requiredParams: ['action', 'module'],
  resolve(ctx, _deps, params): true {
    const decision = can(ctx.get('permissions'), params.action, params.module)
    if (!decision.allowed) {
      throw new HttpException(403, `Forbidden: ${decision.reason}`)
    }
    return true
  },
})

Used at the point it applies to:

@RequirePermission({ action: 'create', module: 'orders' })
@Post('/')
create(ctx: Ctx<KickRoutes.OrderController['create']>) { … }

Note the ContextMeta / ContextKeys split: ContextMeta when something
downstream reads the value, ContextKeys when the decorator exists only for its
side effect. Both give you compile-time checking on dependsOn strings.

Route flags, and inverting the default

The single highest-value change in the rewrite was three lines:

skipWhen: ['auth.public', 'tenant.controlPlane']

Authentication now rejects by default, and a route opts out explicitly. The
legacy app worked the other way round: you added preAuthorize() to each route
that needed it.

That difference is not theoretical. Porting turned up whole modules where someone
had simply forgotten — every route open, nobody having decided they should be.
Forgetting to add a guard is silent. Forgetting to add auth.public is a 401 on
your first request.

Declaring a flag is one line, and it's worth putting them all in one file:

import { defineRouteFlag } from '@forinda/kickjs'

/** Reachable without a token — login, password reset, health. */
export const Public = defineRouteFlag('auth.public')

/** Deliberately not tenant-scoped — operator console, provisioning. */
export const ControlPlane = defineRouteFlag('tenant.controlPlane')

@Public
@Post('/login')
login(ctx: Ctx<…>) { … }

kick typegen collects every defineRouteFlag call into a KickRouteFlags
registry, so a typo in a skipWhen string is a compile error rather than a
contributor that silently never skips. Flags resolve method > class > mount, same
as contributors, and @Flag.off removes an inherited one.

A flag carries no behaviour of its own — it records a fact that any consumer can
read. That's what makes them composable: the same auth.public is read by the
actor contributor, the audit log and the rate limiter, none of which know about
each other.

Pre-match middleware can read them too

I initially assumed flags were unavailable to connect-style middleware, since
that runs before the router matches and there's no route yet. Wrong — there's an
explicit hand-off:

import { bindRoutePolicy, type RoutePolicyTable } from '@forinda/kickjs'

export function auditUnflagged() {
  let policy: RoutePolicyTable | undefined
  const handler = (req, _res, next) => {
    const flags = policy?.lookup(req.method, req.url ?? '/')
    if (!flags?.has('audit.skip')) log(req.url)
    next()
  }
  return bindRoutePolicy(handler, (table) => { policy = table })
}

lookup() returns an empty map when the path matches no route at all — which
is precisely the traffic you want a rate limiter or an audit log to still see.
The table is handed per-application rather than through a global, so two apps in
one process never see each other's routes.

The split is deliberate, and the pairing is instructive: rateLimit() runs
pre-match because an unmatched surface still needs protecting; csrfGuard() is a
guard because a token check on a route that doesn't exist is meaningless. Having
both forms — and being pointed at the right one by which of them exposes the
table — beats picking one and living with it.

Typed end to end without ceremony

kick typegen walks the modules and generates a KickRoutes namespace. In the
handler you get params, body, query and response typed with no codegen artifacts
in your source:

@Post('/', { body: createOrderSchema, name: 'CreateOrder' })
@ApiTags('Orders')
@RequirePermission({ action: 'create', module: 'orders' })
async create(ctx: Ctx<KickRoutes.OrderController['create']>) {
  return reply.created(await this.orders.create(ctx.body))
}

One zod schema gives you runtime validation and the OpenAPI request body.
ctx.body is inferred from it. The return type flows into
KickRoutes[...].response, which is what makes a generated client know what each
route hands back.

The rewrite currently registers 459 routes and 295 services this way.

A related nicety: handlers return values rather than writing to the response.
reply(status, body) when the status isn't 200, ctx.problem.notFound(...) for
error branches — the latter sends immediately and returns nothing, so failure
cases stay out of the inferred success type.

Adapters as the composition seam

An adapter bundles everything one concern needs — DI registrations, middleware,
contributors, lifecycle hooks, a health probe:

export const TenantAdapter = defineAdapter({
  name: 'TenantAdapter',
  build: (config) => ({
    middleware() { /* connect-style, phase-scoped, path-scoped */ },
    contributors() { return [TenantRoute.registration, TenantDb.registration] },
    beforeStart(ctx) { /* late DI wiring */ },
    async onHealthCheck() { return { name: 'tenant-db', status: 'up' } },
    async shutdown() { /* close what you own */ },
  }),
})

In the legacy app, that same concern was smeared across two middleware files, a
connection registry and four request aliases. Here it's one file you can read top
to bottom.

Phases (beforeGlobal / afterGlobal / beforeRoutes / afterRoutes) and an
optional path scope mean you rarely need to think about ordering.

Drizzle setup and tenant database routing

Database-per-tenant is the part people usually ask about, so here is the whole
mechanism. It is smaller than you'd expect, because the framework supplies the
per-request plumbing and Drizzle supplies the rest.

Two schemas, two migration sets

The schema splits in two. A control database holds routing and identity —
which organisations exist, which hosts point at them, which physical database
each one lives in. A tenant database, one per organisation, holds everything
else.

src/db/schema/control/   organizations, tenant_domains, tenant_db_assignment, …
src/db/schema/tenant/    the actual domain — 240 tables
migrations/control/      0000_control-baseline.sql, …
migrations/tenant/       0000_tenant-baseline.sql, …

Two drizzle-kit configs, because they migrate independently — the control
database once, the tenant migrations across every provisioned tenant:

// drizzle.tenant.config.ts
export default defineConfig({
  schema: './src/db/schema/tenant/index.ts',
  out: './migrations/tenant',
  dialect: 'postgresql',
  casing: 'snake_case',
  dbCredentials: { url: process.env.TENANT_DATABASE_URL ?? '' },
})

Conventions worth locking in early, because retrofitting them across 240 tables
is miserable: one table per file, enums in their own files, and named migrations
only (drizzle-kit generate --name=add-order-table) so git history reads as
something other than timestamps.

Column mixins

Every table is built from the same mixins, which is how you get audit columns
and soft delete without 240 opportunities to forget one. The audit foreign key
is a parameter, since control-side tables point at operator users and tenant-side
tables at tenant users:

export function withAudit(opts: AuditMixinOpts) {
  const target = resolveFk(opts)
  return {
    created_by: auditFk('created_by', target),
    created_at: timestamp('created_at', { withTimezone: true })
      .notNull().default(sql`now()`),
    updated_by: auditFk('updated_by', target),
    updated_at: timestamp('updated_at', { withTimezone: true })
      .notNull().default(sql`now()`),
  }
}

export function withAuditAndSoftDelete(opts: AuditMixinOpts) {
  return {
    ...withAudit(opts),
    deleted_at: timestamp('deleted_at', { withTimezone: true }),
    deleted_by: auditFk('deleted_by', target),
    deletion_reason: text('deletion_reason'),
  }
}

A table then reads:

export const orders = pgTable('orders', {
  id: pk(),
  organization_id: fk('organization_id').notNull(),
  total_minor: bigint('total_minor', { mode: 'bigint' }).notNull(),
  currency: char('currency', { length: 3 }).notNull(),
  ...withAuditAndSoftDelete({ fkColumn: () => users.id }),
})

The fkColumn accepts a thunk so a self-referential table — users.created_by
pointing at users.id — needs no forward declaration.

For the record, the legacy schema carried audit columns in two spellings
(created_by on 175 tables, createdBy on 46) and soft delete on 187 of 241.
Normalising that is a mixin, not a migration project.

Keys are ULIDs, generated client-side with a monotonic factory so ids minted
in the same millisecond still sort in creation order:

export const pk = () =>
  char('id', { length: 26 }).primaryKey().$defaultFn(() => newId())

Postgres has no ULID type, so it's char(26) — ten bytes wider than a uuid, and
readable in a log line without decoding. Generating client-side means children
can link to a parent before either is flushed.

One pool per instance, not per tenant

The connection layer is about forty lines. The important detail is that pools are
cached by connection URL, so a hundred tenants sharing one Postgres instance
share one pool rather than opening a hundred:

const pools = new Map<string, postgres.Sql>()

function poolFor(url: string): postgres.Sql {
  let sql = pools.get(url)
  if (!sql) {
    sql = postgres(url, { max: 10, onnotice: () => {} })
    pools.set(url, sql)
  }
  return sql
}

export function createTenantDb(url: string) {
  return drizzle(poolFor(url), { schema: tenantSchema })
}

The routing table stores an instance URL and a database name separately, so
moving a tenant to another instance is a row update rather than a migration:

export function tenantUrl(instanceUrl: string, dbName: string): string {
  const url = new URL(instanceUrl)
  url.pathname = `/${dbName}`
  return url.toString()
}

Routing a request to its database

Two contributors, and this is where the framework earns its place. The first
answers which organisation:

export const ResolveTenantRoute = defineHttpContextDecorator({
  key: 'tenantRoute',
  skipWhen: 'tenant.controlPlane',
  deps: { router: TENANT_ROUTER },
  async resolve(ctx, { router }): Promise<TenantRoute | null> {
    const headers = ctx.req.headers as Record<string, string | string[] | undefined>
    return router.resolve({
      host: pickHost(headers, { trustProxy: getEnv('TRUST_PROXY') === true }),
      tenantHeader: firstHeader(headers['x-tenant-id']),
    })
  },
})

The second answers which connection, and depends on the first:

export const LoadTenantDb = defineHttpContextDecorator({
  key: 'tenantDb',
  skipWhen: 'tenant.controlPlane',
  dependsOn: ['tenantRoute'],
  deps: { resolver: TENANT_DB_RESOLVER },
  async resolve(ctx, { resolver }): Promise<TenantDb | null> {
    const route = ctx.get('tenantRoute')
    if (!route) return null
    return resolver.forOrg(route.orgId)
  },
})

Four decisions in there are worth stealing:

Host-first, header as a fallback. A host is set by the connection; a header
is set by whoever is calling. The legacy app was header-first, which meant the
tenant selector was fully client-controlled. Keeping x-tenant-id behind a
feature flag is what lets existing clients keep working while hosts roll out.

The header is a selector, not a credential. It says which tenant to route to,
never who is calling. The actor contributor separately checks that the
authenticated user's organisation matches the resolved one — without that, a
header alone would be a tenant-hopping primitive.

Both resolve to null rather than throwing. Control-plane routes legitimately
have no tenant, and they say so with the tenant.controlPlane flag, which
skipWhen reads. A route that genuinely needs a connection is the thing that
turns null into a 400. The legacy middleware answered such requests itself with
a 400, which is why "public" routes there could still be refused for tenant
reasons nobody had asked for.

tenantDb depends on tenantRoute, not on the actor. Unauthenticated tenant
routes — login, password reset — still need their tenant's database. Get this
edge wrong and login is impossible on a per-tenant database.

Downstream, a repository just asks for it:

const db = ctx.require('tenantDb')
return db.select().from(orders).where(eq(orders.organization_id, orgId))

For comparison, the Express version registered a Sequelize connection into a
global model registry per request, keyed by a database name formatted from the
organisation id, in a middleware that also did authentication.

The sharp edges, honestly

A post with no criticism is marketing, so:

@Middleware() is not where Express middleware goes. It's engine-neutral and
calls its handler as (ctx, next). Hand it a connect-style (req, res, next) and
next binds to the response object — you get TypeError: next is not a function,
a 500 whose stack points at your middleware rather than at the mismatch. Express
middleware belongs in an adapter's middleware(). The CLI's generated template
used to get this backwards; it now says so explicitly and points you at
kick g guard.

routes() paths must be literals. Typegen parses them statically, so hoisting
a path into a const leaves the module unbound, every handler generating with
contextKeys: never — and the type errors point at the controller, not at the
module file that caused it. Recognise the symptom:
Property 'x' does not exist on type 'never' on every handler at once.

Both are the same class of problem: the framework has the right answer, but the
shortest path from the symptom doesn't lead to it. Worth knowing up front, which
is most of why I'm writing this.

What the migration itself taught me

Two things I'd carry to any port, framework aside.

Prove parity with a test, not with a compile. Every module in the strangler
run shipped with a test hitting the same route on both API versions and asserting
identical status, body and content-type. It caught genuinely subtle things: a
route registered twice, a PUT /remove/ where all its siblings were DELETE, a
handler returning a different response shape depending on whether a query
string was present.

Beware the heuristic you invented three modules ago. I had a rule for
predicting whether a module was portable — does the handler return a value or
write to the response? It was wrong three times, always in the same direction,
and cost two modules a second pass each. What actually predicts cost is whether
the handler holds its own queries, which is one grep:

grep -c "getDb(\|dbRepo\[" <controllers>

Rewriting rather than transcribing also surfaces far more. The strangler run
accumulated about 30 recorded findings; the rewrite found 194 — because
writing behaviour fresh forces you to read every branch, where transcription
lets you copy straight past them.

Would I pick it again

For this shape of problem — a structured multi-tenant API with real per-request
scoping — yes. The composition layer is the differentiator: contributors,
adapters, flags and mixed mounting are what let the same framework host both an
incremental strangler and a clean rewrite without fighting either.

It is not batteries-included. You bring your own ORM and validation, the
ecosystem is small, and some of the family packages move at different speeds. If
you want an opinionated end-to-end stack, that's a different tool.

But the primitives are the right primitives, and after 593 routes I never once
wanted to route around them.

From a GitHub OAuth 404 to a More Reliable MyZubster Runtime

Tue, 22 Sep 2026 03:59:35 +0000

From a GitHub OAuth 404 to a More Reliable MyZubster Runtime

Sometimes a production problem starts with a very small symptom.

In our case, it was a GitHub OAuth 404.

What looked like an authentication issue eventually led us through a much broader review of MyZubster’s development and runtime environment: GitHub account migration, repository ownership, systemd, PM2, MongoDB initialization, multiple Mongoose runtimes, realtime authorization, Bitcoin payment logic, and an aging Jest test suite.

This is the story of how one error turned into a useful infrastructure cleanup.

What is MyZubster?

MyZubster is an evolving open-source ecosystem combining community participation, digital experiences, Marketplace and Seller workflows, AI assistance, experimental projects, a Metaverse interface, and open-source contribution.

Useful links:

Website: https://myzubster.com
GitHub: https://github.com/danieldirimini-myzubster/myzubster
Marketplace: https://myzubster.com/marketplace
Metaverse: https://myzubster.com/metaverse
LIFE Pilot: https://myzubster.com/life-pilot

The project is actively evolving, and part of the current work is making sure the codebase, infrastructure, runtime behavior, and tests all describe the same system.

The original problem: GitHub OAuth

The first visible issue was an unreliable GitHub authentication flow that could end in a 404.

At the same time, the previous development account had accumulated other operational limitations affecting API access, OAuth integrations, Actions, and repository workflows.

Instead of trying to keep building on top of an unstable setup, we separated two concerns:

the historical/public project identity;
the operational GitHub identity used for development.

A new GitHub account became the operational development account, while the existing MyZubster identity inside the product remained untouched.

This was important.

We were changing the development control plane, not rewriting the identity or history of the project.

Migrating Git without destroying local state

The active VPS repository is located at:

/root/myzubster
The server also contains worktrees, experimental directories, verifier code, and local files that should not be committed.
So we deliberately avoided shortcuts like:
git reset --hard
and:
git add .
Instead, we moved the canonical remote to the new GitHub repository while keeping older remotes available for reference.
The current development repository is:
https://github.com/danieldirimini-myzubster/myzubster
The active feature branch was then synchronized with the latest main without rewriting history.
That preserved the work already done while bringing in an OAuth-related fix from the main branch.
Then we found a runtime ownership problem
Once Git was stable, we inspected how MyZubster was actually running on the VPS.
We discovered that the environment used both:
systemd
PM2
That is not necessarily a problem by itself.
The real issue was that both systems appeared to have responsibility for parts of the same application.
There was a PM2 process named myzubster, while the actual HTTP gateway was being served through systemd.
Even worse, the startup paths were different.
The root server.js exported the Express application, while another process was responsible for calling .listen().
That meant we had multiple process managers around an application that did not have a single, obvious runtime owner.
The solution was to make systemd the canonical supervisor for the main gateway and remove the duplicate PM2 ownership.
PM2 remains useful for other independent services.
Production was running from an older project directory
Another subtle problem appeared in the systemd configuration.
The active repository was:
/root/myzubster
but the service still referenced an older copy:
/root/myzubster-ahp
This is a dangerous situation because you can successfully edit, test, commit, and push one repository while production continues to execute another.
We updated the systemd working directory so the code being edited and the code being executed are now aligned.
This is one of the most important lessons from this debugging session:
Never assume the repository you are editing is the repository your production process is actually running.

Check WorkingDirectory.
Check ExecStart.
Check the real process.
The homepage worked, but the Metaverse was degraded
After fixing the runtime path, the homepage returned:
HTTP 200
But the Metaverse health endpoint reported:
{
  "success": false,
  "status": "degraded",
  "transport": "unavailable",
  "mongodb": "disconnected"
}
At first this looked like a MongoDB problem.
It was not.
The credentials were correct and the root application could connect to the database.
The real problem was much more interesting.
Two Mongoose installations meant two database states
The project contains separate Node.js dependency contexts.
Conceptually:
root application
  -> Mongoose 7

backend
  -> Mongoose 8
Those are two separate JavaScript instances.
Connecting one does not connect the other.
So this:
await rootMongoose.connect(...)
does not make this true:
backendMongoose.connection.readyState === 1
The backend Metaverse code was checking its own Mongoose connection.
That connection had never been initialized by the systemd startup path.
The database itself was fine.
The application lifecycle was wrong.
Fixing the bootstrap sequence
The backend already had proper database initialization logic.
The missing piece was making sure production actually called it.
We created a dedicated systemd entrypoint that does this in the correct order:
load application
      ↓
connect backend MongoDB
      ↓
start HTTP server
In simplified Node.js:
const app = require('../server');
const backend = require('../backend/src');

async function main() {
  await backend.connectDatabase();

  const server = app.listen(5003, '127.0.0.1', () => {
    console.log('MyZubster Gateway ready');
  });

  process.on('SIGTERM', async () => {
    server.close(async () => {
      await backend.disconnectDatabase();
      process.exit(0);
    });
  });
}

main();
After restarting systemd, the Metaverse health endpoint changed to:
{
  "success": true,
  "status": "healthy",
  "transport": "shared-polling",
  "mongodb": "connected"
}
That was a major milestone.
We did not “solve” Mongoose by deleting dependencies
A tempting response would have been to force both parts of the project to use one Mongoose installation immediately.
We deliberately did not do that.
The immediate problem was not the existence of two Mongoose versions.
The problem was that only one connection lifecycle was being initialized.
This distinction matters.
A structural refactor may still make sense in the future, but production incidents are not the right time to perform unnecessary dependency surgery.
Fix the real failure first.
Then the tests started teaching us about the architecture
The feature-specific tests were healthy.
For the MYZ-188 development work, we had:
7 test suites passed
37 tests passed
But the complete repository test suite still contained failures.
Instead of assuming the new branch had broken everything, we checked whether several failures also existed on main.
Many did.
That gave us a useful distinction:
new regression
versus:
existing technical debt
This saved us from “fixing” unrelated code based only on the fact that Jest was red.
A red test does not always mean production is wrong
One example involved the monthly AI budget.
A test expected a budget key like:
openai:gpt-5.6-sol:2026-09
while production used:
openai:all:2026-09
At first glance, changing the production key would make the test pass.
But when we reviewed the architecture, the global key was clearly intentional.
The monthly budget covers the whole OpenAI/Astra usage, and the system may fall back between different OpenAI models.
Using model-specific budget keys could allow separate models to consume separate caps.
So the production behavior was correct.
The test was stale.
We updated the test.
That is an important rule:
Tests are executable documentation, but executable documentation can become outdated too.

Realtime authorization: fail closed
Another test was hanging for five seconds.
The relevant code queried community membership through MongoDB.
When the database authority was unavailable, Mongoose buffered the query until Jest timed out.
The correct behavior for authorization was not to wait.
It was to deny.
We changed the flow to check database availability first:
if (mongoose.connection.readyState !== 1) {
  return {
    allowed: false,
    reason: 'community_membership_authority_unavailable'
  };
}
The behavior changed from:
database unavailable
    ↓
wait
    ↓
timeout
to:
database unavailable
    ↓
deny immediately
That is both faster and safer.
Then we found a real Bitcoin bug
The full suite exposed another issue:
TypeError:
Cannot read properties of undefined (reading 'includes')
The failing code was checking:
SUPPORTED_ASSETS.includes(asset)
But SUPPORTED_ASSETS itself was undefined.
The reason was a CommonJS circular dependency.
The dependency graph looked approximately like this:
Legacy Monetization Service
          ↓
Unified Checkout Service
          ↓
Quote / Chain Verifier
          ↓
Legacy Monetization Service
During module initialization, Node.js returned an incomplete export.
The result:
SUPPORTED_ASSETS === undefined
Breaking the circular dependency
We did not add a defensive fallback like:
SUPPORTED_ASSETS || []
That would hide the design problem.
Instead, we extracted the constant into a leaf module:
'use strict';

const SUPPORTED_ASSETS = Object.freeze(['BTC']);

module.exports = {
  SUPPORTED_ASSETS
};
Now the payment services depend on a small module that depends on nothing else.
The Bitcoin production tests moved from:
3 failed
2 passed
to:
5 passed
The tests cover:
- BTC availability;
- BTC/EUR quoting;
- satoshi conversion;
- Esplora verification;
- unconfirmed-payment fail-closed behavior.
Legacy tests were still mocking an older architecture
The next group of failures was especially interesting.
Several tests still mocked:
ZorgaxPaymentIntent
ZorgaxSubscription
But the current production architecture uses:
PaymentIntent
ZorgaxPurchase
Entitlement
That means the Jest mocks looked correct, but they were intercepting classes the application no longer used.
The real Mongoose models were executing instead.
Then Jest waited for the real database.
Five seconds later:
Exceeded timeout of 5000 ms
The problem was not a slow application.
It was a stale test boundary.
Test runtime: 11 seconds to less than one
One replay-protection test originally took around:
11 seconds
and failed with two timeouts.
After updating the test to mock the real architecture:
2 tests passed
in roughly:
0.7 seconds
That is a strong indicator that the previous problem was accidental external I/O.
The new replay protection verifies something closer to the real current contract:
same payment reference
        ↓
existing purchase?
        ↓
same owner?
   ┌────┴────┐
  yes        no
   ↓          ↓
 reuse      reject
Only after ownership is validated does the system grant the entitlement.
What we are doing now
The main runtime is now much healthier.
We have already completed:
- migration to the new operational GitHub account;
- repository remote cleanup;
- branch synchronization with main;
- OAuth regression verification;
- systemd runtime cleanup;
- duplicate PM2 gateway removal;
- backend MongoDB bootstrap;
- Metaverse health recovery;
- graceful backend database shutdown;
- Zorgax access compatibility cleanup;
- AI budget test alignment;
- realtime fail-closed authorization;
- Bitcoin circular-dependency removal;
- modernization of several outdated payment tests.
The current focus is the remaining legacy test suite.
Some tests still represent old implementation details instead of the current architecture.
So we are going through them carefully and asking:
Is this a production bug, or is the test describing a system that no longer exists?

That is the work happening right now.
The bigger architectural goal
The cleanup is gradually making the boundaries clearer:
systemd
  -> explicit gateway bootstrap

gateway bootstrap
  -> explicit database lifecycle

realtime authorization
  -> explicit availability checks

payments
  -> unified checkout boundary

payment constants
  -> dependency-free leaf module

access
  -> entitlement authority

tests
  -> mock current production boundaries
The goal is not merely to get a green test suite.
The goal is to have a green test suite that means something.
Small commits are helping a lot
We are keeping fixes isolated.
Examples include:
fix(runtime): initialize backend Mongo before gateway listen

fix(zorgax): include optional legacy subscription access source

test(zorgax): align AI budget expectation with global monthly cap

fix(realtime): fail closed when community membership authority is unavailable

fix(zorgax): break payment asset circular dependency
Small commits give us three advantages:
- easier review;
- easier rollback;
- easier reasoning.
Git becomes part of the debugging process, not just version storage.
What we deliberately did not do
Some shortcuts would have made the situation worse.
We did not:
reset the repository destructively
delete worktrees blindly
force-de-duplicate Mongoose during an incident
increase Jest timeouts to hide MongoDB access
hide circular dependencies with fallback values
commit credentials or environment secrets
A large part of production engineering is not just knowing what command to run.
It is knowing which commands not to run.
Current simplified architecture
The current direction looks like this:
GitHub
   ↓
danieldirimini-myzubster/myzubster
   ↓
VPS /root/myzubster
   ↓
systemd
   ↓
MyZubster Gateway
   ├── Root application
   ├── Backend Mongo context
   ├── Metaverse
   ├── Realtime
   ├── Zorgax
   └── Payment services
Other independent services can still use PM2 where appropriate.
The important improvement is that there is now a much clearer owner for each runtime responsibility.
What comes next
The immediate plan is:
1. finish updating the remaining legacy payment tests;
2. review source-inspection tests that still point to compatibility wrappers;
3. run the complete Jest suite again;
4. separate real regressions from intentionally changed contracts;
5. verify production health;
6. prepare the branch for review and merge into main.
The goal is simple:
align the repository, runtime, architecture, and tests before merging.

Final lesson
This entire debugging session started with:
GitHub OAuth 404
and evolved into:
GitHub migration
      ↓
runtime cleanup
      ↓
systemd stabilization
      ↓
MongoDB lifecycle fix
      ↓
Metaverse recovery
      ↓
authorization hardening
      ↓
Bitcoin dependency cleanup
      ↓
legacy test modernization
That is often how real engineering works.
The first visible error is not always the real problem.
Sometimes it is simply the first place where the system tells you to look deeper.
We are continuing that work now on MyZubster.
Project:
https://myzubster.com
GitHub:
https://github.com/danieldirimini-myzubster/myzubster
Marketplace:
https://myzubster.com/marketplace
Metaverse:
https://myzubster.com/metaverse
LIFE Pilot:
https://myzubster.com/life-pilot
The next milestone is to finish the legacy test alignment, validate the entire branch, and prepare it for merge into main.
One boundary at a time.

The Silent Performance Killer: Demystifying the N+1 Query Problem in Node.js

Mon, 21 Sep 2026 09:18:48 +0000

Have you ever noticed a feature that works beautifully on your local computer, only to crawl at a snail's pace once it gets deployed to production? One of the most common, yet invisible, reasons for this sudden slowdown is a database fetching behavior known as the N+1 query problem.

In short, the N+1 query problem occurs when an application makes one initial database query to fetch a list of records (the "1"), and then makes a separate database query for every single record returned in that list (the "N") to fetch related data. Instead of fetching all the required information in a single, efficient request, the application ends up chatting back and forth with the database dozens, hundreds, or thousands of times.

The Restaurant Server Analogy

To understand why this is so inefficient, imagine sitting down at a busy restaurant with a table of ten friends. It is time to order drinks.

An efficient server (who behaves like an optimized database query) would take everyone's drink orders on a notepad, walk to the kitchen once, load all ten glasses onto a single tray, and bring them back to the table in one trip. This requires just one round-trip to the kitchen.

Now imagine an N+1 server. This server walks to the table, asks the first person what they want, and walks all the way back to the kitchen to fetch that single drink. After serving it, they walk back to the table, ask the second person for their order, and make another complete round-trip to the kitchen. They repeat this individual process for all ten people. By the time everyone has a drink, the server has made eleven total trips (1 trip to inspect the table, plus N trips for the N guests). Your dining experience would be incredibly slow, and the poor server would be completely exhausted.

Why This Matters in Daily Engineering

In real-world software engineering, database round-trips are remarkably expensive. Every single query you run introduces network latency, database parsing overhead, and thread consumption in your database's connection pool.

If you are building a social media dashboard that displays 50 posts, and you load the comments for each post individually, you are executing 51 database queries. If 1,000 users visit that page simultaneously, your application will suddenly bombard your database with 51,000 queries. This easily exhausts your database connection pool, drives database CPU usage to 100%, and results in catastrophic timeouts for your users. Engineers must actively prevent this to keep application response times low and infrastructure costs manageable.

Seeing It in Action: Node.js and MySQL

Let's look at a typical Express.js endpoint where this mistake happens, followed by how we fix it.

Here is the problematic, naive approach:

// BAD: This triggers N+1 queries
app.get('/api/posts', async (req, res) => {
  try {
    // 1. Fetch all posts (The "1" query)
    const [posts] = await db.query('SELECT id, title, content FROM posts LIMIT 50');
    
    // 2. Loop through each post to fetch its comments (The "N" queries)
    for (let post of posts) {
      const [comments] = await db.query(
        'SELECT id, text FROM comments WHERE post_id = ?',
        [post.id]
      );
      post.comments = comments;
    }
    
    res.json(posts);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

Here is how we resolve this using a SQL JOIN to pull all the information in a single, efficient query:

// GOOD: Resolved using a single JOIN query
app.get('/api/posts', async (req, res) => {
  try {
    const query = `
      SELECT 
        p.id AS post_id, 
        p.title, 
        p.content, 
        c.id AS comment_id, 
        c.text AS comment_text
      FROM posts p
      LEFT JOIN comments c ON p.id = c.post_id
      LIMIT 50
    `;
    
    const [rows] = await db.query(query);
    
    // Group the flat SQL results into structured JSON
    const postsMap = {};
    for (const row of rows) {
      if (!postsMap[row.post_id]) {
        postsMap[row.post_id] = {
          id: row.post_id,
          title: row.title,
          content: row.content,
          comments: []
        };
      }
      if (row.comment_id) {
        postsMap[row.post_id].comments.push({
          id: row.comment_id,
          text: row.comment_text
        });
      }
    }
    
    res.json(Object.values(postsMap));
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

The Takeaway

The N+1 query problem highlights that how you fetch your data is just as important as the data itself. By switching from sequential, iterative loops to combined database operations like SQL JOINs, you minimize the highly expensive overhead of network round-trips. Designing your data fetching strategies proactively keeps your APIs incredibly fast and ensures your databases stay healthy, stable, and cost-efficient under high production loads.

Originally published on my blog. You can read the alternative breakdown here.

We Rebuilt MYZ Payments Around a Canonical Ledger — Here’s What We Learned

Fri, 18 Sep 2026 16:46:27 +0000

We Rebuilt MYZ Payments Around a Canonical Ledger — Here’s What We Learned

When we started adding internal MYZ payments to MyZubster Marketplace, the obvious task looked simple:

buyer balance -= amount
seller balance += amount

That turned out to be the wrong level of abstraction.

The real challenge was not adding a payment button. It was making sure that every part of the application agreed on what a MYZ balance actually is.

During the implementation, we found multiple historical paths capable of representing or modifying MYZ independently.

That is exactly the kind of situation that eventually produces accounting bugs.

So instead of adding another payment path, we refactored the system around one rule:

There must be one canonical source of truth for spendable MYZ.

This post covers the architecture we implemented, the problems we found, and the trade-offs that are still open.

The original problem: balances were too easy to create

A field like this looks harmless:

{
userId,
balanceMYZ: 500
}

Until another subsystem has:

{
accountId,
myzCredit: 200
}

And another computes:

displayedBalance =
ledgerBalance +
mongoCredits -
utilityRedemptions;

Now the question becomes:

What is the user's real balance?

If multiple components can answer that question differently, the system no longer has a reliable accounting model.

That was the architectural problem we wanted to eliminate.

First principle: MYZ is an internal accounting unit

Before touching the code, we established the semantics.

MYZ is an internal MyZubster utility/accounting credit.

It is not automatically convertible into:

EUR
BTC
XMR
cash

and it is not represented by this implementation as an on-chain token.

This matters technically because it means we do not need an exchange-rate engine in order to spend MYZ.

If a Marketplace listing costs:

250 MYZ

the payment is simply:

250 MYZ

There is no intermediate:

250 MYZ → €X → seller

Likewise, a Zorgax plan can have an explicit MYZ price without deriving it from its EUR price.

The canonical ledger

The core service is:

src/services/myzLedgerApiService.js

Instead of mutating a user balance directly, transfers are recorded as ledger entries.

A payment creates two legs:

buyer -250 MYZ
seller +250 MYZ

Both are linked by the same:

transfer_id

Conceptually:

              transfer_id: T-123
             ┌─────────────────┐

buyer account │ -250 MYZ │

             │                 │

seller account │ +250 MYZ │

             └─────────────────┘

This gives us something a plain balance field does not provide naturally:

auditability.

We can answer:

Why is this balance 750 MYZ?

by looking at the entries that produced it.

Transfers are atomic inside the ledger

The transfer operation performs both accounting legs under the same write operation.

The important invariant is:

debit exists ⇔ credit exists

We do not want:

buyer -250

without:

seller +250

because the process crashed halfway through.

The ledger therefore validates the transfer first, then persists both entries together.

The relevant checks include:

valid payer
valid payee
payer != payee
valid amount
sufficient balance
valid idempotency key
no transfer ID collision

Only after those checks do the entries become recorded.

Never trust the UI to prevent overspending

One of the tests we cared about most was not:

Can Alice buy an item?

It was:

Can Alice buy two items at the same time
when she can only afford one?

Example:

Alice balance: 300 MYZ

Purchase A: 250 MYZ
Purchase B: 250 MYZ

Both requests may arrive concurrently.

A naïve implementation can do this:

if (balance >= 250) {
// request A sees 300
}

if (balance >= 250) {
// request B also sees 300
}

Then both succeed.

Final logical balance:

-200 MYZ

That is unacceptable for an accounting system.

Our expected result is:

one purchase succeeds
one purchase fails
final balance = 50 MYZ

The protection belongs in the ledger write path, not in React.

Idempotency became a first-class concept

Payment APIs are retried.

Browsers retry.

Networks fail.

Users double-click.

Servers can complete the financial operation and crash before returning 200 OK.

So this API:

POST /api/marketplace/orders/:id/payment/myz

requires an idempotency key.

A simplified request looks like:

Idempotency-Key: marketplace-order-123

If the exact operation is retried, we return the original transfer instead of charging again.

But there is another subtle problem.

What happens if somebody reuses the same idempotency key with a different payload?

For example:

key = abc
plan = Zorgax Pro
amount = 40 MYZ

followed by:

key = abc
plan = Zorgax Developer
amount = 40 MYZ

The amount is identical, but the operation is not.

So idempotency is bound not only to the numeric amount, but also to the semantic reference of the operation.

Same key + different meaning = conflict.

That protects against a class of bugs that simple "key already exists" implementations often miss.

Marketplace became a true buyer → seller transfer

For MYZ-priced Marketplace orders, the flow is now:

Buyer
↓
canonical MYZ transfer
↓
Seller
↓
order marked PAID

The order stores evidence of the accounting operation:

payment: {
asset: 'MYZ',
network: 'internal-ledger',
status: 'PAID',
transferId,
debitEntryId,
creditEntryId
}

We also enforce several business invariants.

The buyer must be authenticated.

The buyer must actually own the order.

Buyer and seller cannot be the same account.

The order must be accepted before payment.

And an unpaid MYZ order cannot transition to completed.

The important point is that these rules are checked server-side.

The React button is just UX.

Application state and ledger state are not the same transaction

There is an interesting systems problem here.

The MYZ ledger and the Marketplace order are not stored in the same transactional database.

So this can happen:

MYZ transfer succeeds
application crashes
order has not yet been marked PAID

You cannot solve that with a normal database transaction across unrelated stores.

Instead, we use an idempotent recovery model.

On retry:

ledger sees existing transfer
→ does not charge again

application receives original receipt
→ retries order persistence

This is closer to a small saga than a traditional ACID transaction.

The important requirement is recoverability.

Zorgax uses the same accounting primitive

We reused the same ledger for Zorgax plans.

The flow is:

user MYZ account

↓

canonical transfer

↓

zorgax:system:treasury

↓

grant entitlement

Prices are configuration values:

ZORGAX_PRO_PRICE_MYZ
ZORGAX_DEVELOPER_PRICE_MYZ

They are deliberately not calculated from:

priceEur

If the MYZ price is missing, the checkout fails closed.

Production currently reports the MYZ plan as unavailable instead of guessing a price.

That is a feature, not a failure.

The most important refactor was deleting shadow accounting

While reviewing the system, we found an older Payment Dashboard flow that effectively calculated MYZ like this:

canonical ledger
+

Mongo credits

Mongo redemptions

It also contained an automatic EUR → MYZ rule.

That meant the canonical ledger was not actually canonical.

So we changed the model.

New Stripe payments do not automatically generate MYZ.

Historical Mongo conversion records remain available as legacy audit data, but they no longer contribute to spendable MYZ.

Utility spending now hits the canonical ledger.

MongoDB redemption records are retained for things like:

fulfillment status
receipt metadata
application workflow

but they are no longer a second balance system.

That distinction is extremely useful:

ledger = accounting truth

application database = business state
Then we found another hard-coded system: Marketplace categories

The same development cycle uncovered a less dangerous but very visible problem.

The React Marketplace category list was mostly hard-coded.

That means adding a category required changing application code.

We replaced that with a dynamic catalog:

GET /api/listings/categories

Users can now propose new categories from the Marketplace itself.

The interesting UX decision is that the creator can use their pending category immediately for their own listing.

Global visibility still requires moderation.

So the flow becomes:

create category

↓

pending

↓

creator may use it

↓

admin review

↓

global category

This is much more flexible than making category taxonomy a deploy-time decision.

Knowledge became a Marketplace object too

We also introduced a first-class:

knowledge

category.

Users can submit things like:

guides
methods
know-how
tutorials
manuals
practical skills

through a separate knowledge submission flow.

But we explicitly did not connect submission directly to rewards.

A new knowledge item starts as:

PENDING_REVIEW

The API response explicitly indicates that creation does not produce:

reward
ledger write
MYZ transfer

This separation matters.

A useful architecture is:

submission
↓
review
↓
verification
↓
optional reward

not:

user submits text
↓
money appears
Deployment uncovered another important distinction

The code was merged through PR #1286.

Production deployment succeeded on Vercel.

Marketplace category endpoints respond correctly.

knowledge is available in production.

No fatal/runtime errors appeared during our verification window.

But we are intentionally not saying the MYZ accounting system is fully production-ready yet.

Why?

Because the current ledger implementation is file-based.

And there is a big difference between:

the application can write a file

and:

this storage is durable, shared, atomic,
and appropriate for production accounting

In a serverless environment, local filesystem assumptions deserve special attention.

A file lock only protects concurrent writers if those writers actually share the same durable filesystem.

Multiple isolated serverless instances may not.

So this implementation establishes the accounting contract and business logic, but production MYZ writes still need a properly durable storage architecture before we treat the system as authoritative under distributed load.

That may mean moving the ledger to a transactional database or placing it behind a dedicated single-writer service.

We also refused to fake a green CI status

One final engineering lesson came from CI.

We created a dedicated GitHub Actions workflow for the MYZ ledger contract.

It includes service tests, route tests, internal-payment tests, and ledger verification.

But GitHub did not generate a workflow run.

We even created a dedicated commit touching the workflow itself:

33180d233dd6abbb558ed58b56371d77eb11faa2
ci(myz): trigger canonical ledger contract

Still:

workflow runs = 0

Other workflows in the repository have execution history.

So instead of writing:

CI ✅

we documented:

CI execution issue still open

It sounds minor, but this is one of the habits that matters most in engineering:

do not convert “we expected this to run” into “it passed.”

The architecture now looks much simpler

Before:

Dashboard MYZ
Payment Dashboard MYZ
Marketplace payment logic
Zorgax payment logic
conversion logic
redemption records

After:

            CANONICAL MYZ LEDGER
                     │
    ┌────────────────┼────────────────┐
    │                │                │

Marketplace Zorgax P2P

    │                │

buyer → seller user → system

Business databases still exist.

They should.

But they track business state.

They no longer get to invent their own spendable MYZ balance.

What I would improve next

The implementation is merged and deployed, but the architecture still has clear next steps:

move the ledger to genuinely durable shared storage;
validate double-spend protection across multiple application instances, not only within one process;
fix the GitHub Actions workflow execution;
add persistent purchase intents for even stronger recovery across client idempotency-key loss;
define explicit migration rules for historical MYZ records;
define verified identity mapping between older contributor: accounts and newer marketplace:user: accounts.

None of those invalidate the current work.

They define the boundary between correct application logic and production-grade distributed accounting infrastructure.

And knowing that boundary is probably the most useful lesson from this refactor.

The biggest takeaway for me was simple:

A balance is a derived value.
A ledger is the history that makes the balance trustworthy.

Once we started treating MYZ that way, Marketplace payments, Zorgax purchases, legacy cleanup, receipts, idempotency, and concurrency all became parts of the same system instead of separate features.

MyZubster is open source, and we’re building this architecture in public.

Repository: MyZubster-Ecosystem/myzubster
Main implementation: PR #1286

XML External Entity (CWE-611): The Bug No Automated Layer Catches — Proven and R

Thu, 03 Sep 2026 07:57:14 +0000

One file path. No form, no captured traffic, no automated finding anywhere in the pipeline. Here's the
full story of a real CWE-611 (XML External Entity)
vulnerability — the first bug in this series where a static rule, a business-logic rule, and a wired
Active Test probe all come back empty, and every step from proof to report has to be done by hand — all
running locally, with no cloud AI anywhere in the pipeline.

The vulnerable code

The target is a deliberately vulnerable local training app: a hand-rolled XML "entity resolver" — not a
real XML parser, on purpose, so the demo has no native dependencies — that reproduces the classic XXE
pattern any real parser falls into when DTD/external-entity processing isn't explicitly disabled.

function resolveEntities(xml) {
  const entities = {};
  const entityRe = /<!ENTITY\s+(\w+)\s+SYSTEM\s+"([^"]+)"\s*>/g;
  let m;
  while ((m = entityRe.exec(xml))) {
    const [, name, systemId] = m;
    const filePath = systemId.replace(/^file:\/\//, ''); // BUG: no restriction on which file
    try {
      entities[name] = fs.readFileSync(filePath, 'utf8');
    } catch (err) {
      entities[name] = `[[failed to resolve entity ${name}: ${err.message}]]`;
    }
  }
  let resolved = xml;
  for (const [name, value] of Object.entries(entities)) {
    resolved = resolved.split(`&${name};`).join(value);
  }
  return resolved;
}

app.post('/import', (req, res) => {
  const xml = req.body || '';
  const resolved = resolveEntities(xml);
  const noteMatch = resolved.match(/<note>([\s\S]*?)<\/note>/);
  res.json({ note: noteMatch ? noteMatch[1] : null, resolvedXml: resolved });
});

Every declared SYSTEM entity gets read off disk with no allowlist at all, and the whole resolved
document — not just whatever lands inside <note> — comes back in the response. GET / renders
instructions and the exact curl command to try; there's no <form> anywhere on this app, which turns
out to matter for how you have to prove this one.

Step 1 — Why the static rule engine stays silent (and that's not a bug)

Before assuming "no finding" means "broken," it's worth tracing exactly why. The platform's XXE rule is
real and precise — it just targets a different library's exact call shape than this app uses:

parseXml(String)?(…, { noent: true, … })   ← the rule's actual target: libxmljs's opt-in

This app never calls parseXml or parseXmlString at all — it's a regex-based resolver, deliberately, so
the demo needs no native XML dependency. A code-scan pass produces zero CWE-611 findings, exactly as
the rule's own logic predicts. It isn't a clean zero, though: a generic missing-authorization rule flags
both routes (this app, like the rest of the series, has no auth anywhere), landing a real High finding
on POST /import — a second, entirely separate bug on the same route, worth its own line in a write-up
rather than being lost in the noise.

Step 2 — The dynamic/business-logic path comes back empty too

The path that rescued CSRF and SSRF with at least one automated finding — capture traffic through the
built-in proxy, then check it against the business-logic rules — doesn't help here either. Neither the
live HTTP-interceptor rules nor the business-logic analyzer (both run inside the same proxy command)
have ever had an entity/DOCTYPE-aware check written for them. A captured request carrying a full XXE
payload produces the same generic header findings as every other request and nothing else.

Step 3 — Proving it live, with nothing but curl

There's no form to click through — GET / is instructional only — so proof starts at the terminal:

curl -X POST http://localhost:3008/import -H "Content-Type: application/xml" --data-binary @payload.xml

{"note":"FLAG{xxe-demo-secret}","resolvedXml":"...<note>FLAG{xxe-demo-secret}</note>"}

That's the app's own intended file. The next payload proves it isn't confined to it — pointing the
SYSTEM identifier at file:///etc/passwd instead returns that file's contents just as readily. The bug
isn't "leaks one demo secret" — it's unrestricted file disclosure, any file the server process can read.

Step 4 — Escalating from scratch, because there's nothing to seed from

Every prior bug in this series had a captured request to right-click and send into the built-in Repeater.
Not this one — with no form and no proxy-captured traffic, the request has to be typed in by hand from a
blank editor: method, URL, headers, body, all composed manually. A third payload variant makes a point
worth keeping for a write-up:

<?xml version="1.0"?>
<!DOCTYPE root [ <!ENTITY xxe SYSTEM "file:///etc/passwd"> <!ENTITY host SYSTEM "file:///etc/hostname"> ]>
<note>&xxe;</note>
<!-- &host; -->

The substitution is a literal string replace over the whole submitted document, and the response's
resolvedXml field echoes the entire resolved document — not just whatever ends up inside <note>. The
response shows note still contains only /etc/passwd (that field's own regex is scoped tightly), while
resolvedXml contains both files' contents. An attacker doesn't need to route a read through <note>
at all — anywhere in the document is enough, and several files can leak in a single request.

Step 5 — Attempting the platform's own Active Test probe — and finding there isn't one

The platform has two dozen wired Active Test probes: SSTI, GraphQL introspection, JWT verification, mass
assignment, CSRF confirmation, and more. I checked the full checklist and the CLI's --probes alias list
on camera rather than assert it from memory — neither has an XXE entry. Not a blind or limited probe like
SSRF's — genuinely absent. That's the whole phase: confirming an absence is itself the useful result here,
not a consolation prize for a failed run.

Step 6 — Filing the bug's only finding, by hand

With nothing in the pipeline ever creating a Finding object for this bug, the only way it exists in the
platform at all is filing it manually through the Assets hub's "Create finding" tool. Worth knowing before
you hit it yourself: that dialog collects a title, a risk level, and free-text evidence — no CWE or
reference field at all — so the CWE-611 citation has to go into the title text by hand. It's also worth
knowing the resulting finding will always show a "no proof of concept" submission blocker, regardless of
how much evidence you paste in — manual findings never populate the platform's structured proof field,
only free text. Cosmetic, not a real block on a normal report export.

Step 7 — An unexpected second corroboration

Running the platform's generic "Analyze response" tool over the /etc/passwd variant's captured
transaction surfaced something worth keeping: alongside the usual generic header findings, a completely
unrelated rule — a generic filesystem-path-disclosure check — independently flagged the same response,
because it literally contains an absolute path string. Different rule, different reasoning, same
underlying bug — a nice bit of corroborating evidence that cost nothing extra to collect.

Step 8 — A fully offline explanation

With the manually-filed finding's evidence in place, a local LLM generates a plain-language narrative from
it: something like "A hand-rolled XML entity resolver reads an attacker-specified file path from a SYSTEM
entity with no restriction, allowing disclosure of arbitrary files readable by the server process." No
network call is made — and this works identically whether a finding was auto-created by a rule or filed by
hand, since the narrator only ever reads the finding's own fields, never its provenance.

The fix

The demo's own bug is a hand-rolled resolver that never should have resolved external entities in the
first place — the fix isn't a configuration flag here, it's refusing to read SYSTEM identifiers at all:

function resolveEntities(xml) {
  if (/<!DOCTYPE/i.test(xml)) {
    throw new Error('DOCTYPE declarations are not permitted in submitted XML');
  }
  return xml; // no external entities are ever resolved
}

For a real XML parser, the equivalent is a configuration setting rather than a rewrite: disable DTD
processing and external-entity resolution outright (most real parsers have an explicit
disallow-doctype-decl / "external entities off" setting — e.g. libxmljs's parseXml(xml, { noent: false, dtdload: false, dtdvalid: false }), or Node's fast-xml-parser with processEntities: false), or
use a parser that doesn't support DTDs at all for untrusted input — verbatim the fix the demo app's own
README.md suggests.

What's honestly not covered

This is the starkest version of "not covered" in the whole series: no static match, no dynamic/business-
logic match, no Active Test probe at all. That's not three separate failures — it's one honest gap,
demonstrated on camera at every layer rather than glossed over. The static rule's precision (it targets a
real library's real API, just not this app's) is actually the most reassuring part: it isn't guessing,
it's correctly scoped, and this app simply falls outside that scope by design.

Try it yourself

The target app and the full step-by-step playbook (every click, every payload, every panel, plus the exact
reasoning behind each "no finding" result) are linked below if you want to reproduce this end-to-end
against your own local copy.

This is an intentionally vulnerable local training app. Never run these techniques against a
system you don't own or don't have explicit written authorization to test.

Try it yourself → https://github.com/sendwavehub/scan-target-demo-apps
Windows Store https://apps.microsoft.com/detail/9pj0j7bk1m27?hl=en-US
Web Site https://Sendwavehub.tech/en/apps/ai-security-studio-4

Server-Side Request Forgery (CWE-918): From Auto-Detected Finding to Live Exploi

Tue, 01 Sep 2026 04:50:58 +0000

A full walkthrough of finding, proving, and reporting a real SSRF bug with a deterministic rule engine, a live form-based exploit, an attempted (and explained) automated probe, and a fully offline local LLM — no cloud AI anywhere in the pipeline.
tags: security, javascript, node, webdev

One form field. One "internal-only" endpoint that was never supposed to be reachable. Here's the
full story of a real CWE-918 (Server-Side Request
Forgery) vulnerability, from the moment a rule engine flags it, through a form-only live exploit, to
an honest look at where the platform's own automated probe hits a real limit — all running locally,
with no cloud AI anywhere in the pipeline.

The vulnerable code

The target is a deliberately vulnerable local training app: a "URL preview" tool that fetches
whatever URL you give it, server-side.

app.post('/preview', async (req, res) => {
  const { url } = req.body;
  if (!url) return res.status(400).json({ error: 'url is required' });
  const upstream = await fetch(url, { redirect: 'follow' }); // BUG: no host/scheme allowlist
  const text = await upstream.text();
  res.json({ url, status: upstream.status, body: text.slice(0, 2000) });
});

url comes straight from the request body and goes straight into fetch() — no allowlist on scheme
or destination host. The same server also exposes /internal/admin-status, an "internal-only"
endpoint that isn't linked from any page and simulates something like a cloud-metadata endpoint or an
admin panel. /preview can reach it anyway — and because the response body gets reflected straight
back to the caller, this isn't blind SSRF: the internal endpoint's own flag comes back in the JSON,
a self-contained proof with nothing but a form submission.

Step 1 — A deterministic rule engine catches it, not an LLM

Before any AI touches this finding, the same taint-tracking rule that flags command injection and
path traversal also declares an outbound-request sink, and walks the exact shape this bug takes:

const { url } = req.body     →  source: untrusted request value
fetch(url, …)                 →  sink: outbound HTTP request

One static scan raises a High, CWE-918 finding automatically — zero manual finding-creation,
zero guesswork. The finding already carries:

Evidence — the matched source line, with the exact column of the injection point pinned inside
the tainted expression
Risk Level — High
Security Reference — CWE-918 / OWASP A10:2021 (Server-Side Request Forgery)

The rule engine discovers; the LLM only ever explains an already-verified finding afterward. It
never gets to invent a vulnerability on its own.

Step 2 — Proving it live, with zero extra tooling

The form field is prefilled http://localhost:3007/internal/admin-status. Submitting it returns:

{
  "url": "http://localhost:3007/internal/admin-status",
  "status": 200,
  "body": "{\"internal\":true,\"message\":\"This endpoint should only ever be called by trusted internal callers.\",\"fakeSecret\":\"FLAG{ssrf-reached-internal-endpoint}\"}"
}

body proves the server, not the browser, reached an endpoint that "isn't linked anywhere and isn't
meant to be reachable from outside this server." One request/response pair is already enough evidence
on its own — no server access needed to confirm the bug, because the app's own response is externally
observable proof.

Step 3 — Escalating the payload (and what a rejection does and doesn't prove)

The request body is application/x-www-form-urlencoded, but the payload itself has no &/=
characters, so none of command injection's body-splitting trap applies here — values load and send as
typed:

Payload	What it proves
`url=http://127.0.0.1:3007/internal/admin-status`	Loopback literal instead of `localhost` — same flag comes back
`url=http://[::1]:3007/internal/admin-status`	IPv6 loopback — a third distinct way into the same endpoint, ruling out a naive hostname-string allowlist
`url=file:///etc/passwd`	Rejected with a 502 — but by Node's own `fetch` (undici), which only supports `http`/`https`. That's a runtime-level restriction, not an application-level allowlist, and shouldn't be mistaken for one in a write-up

Three host variations, one important negative result — and the negative result is worth writing up
carefully instead of skipping past it.

Step 4 — Attempting the platform's own Active Test probe — and explaining why it can't confirm this one

The platform does have a real, wired SSRF probe — a GUI checkbox and a CLI flag, backed by an
offline, local out-of-band collaborator (no third-party service, no internet). I set one up and ran
it against /preview's url parameter anyway, on camera, rather than just asserting it wouldn't
work.

It came back with no finding — and the reason is architectural, not a fluke: the probe always injects
its callback URL via a GET query string (ActiveTestUrls.WithQueryParameter, sent through a
GET-only client call). This app's vulnerable parameter exists only on POST /preview's form
body — there's no GET /preview route at all, so the probe's request 404s before the server ever
calls fetch(). No collaborator misconfiguration, no false negative on the app's actual exposure —
just a probe that was built for a different injection channel than this specific endpoint uses.

Two findings would have been the tidy outcome. One finding plus a documented, code-level reason for
why the second attempt didn't land is the honest one.

Step 5 — A fully offline explanation

With the static evidence and the manual proof both attached to the same finding, a local LLM
generates a plain-language narrative from the structured evidence: something like "Untrusted input
reaches an outbound fetch() call with no host/scheme allowlist, letting the server be used to reach
its own internal-only endpoints." No network call is made. The model reasons over an
already-verified finding; it doesn't discover anything new.

Step 6 — The fix

const ALLOWED_HOSTS = new Set(['api.trusted-partner.example']);

app.post('/preview', async (req, res) => {
  const { url } = req.body;
  let target;
  try {
    target = new URL(url);
  } catch {
    return res.status(400).json({ error: 'invalid url' });
  }
  if (!['http:', 'https:'].includes(target.protocol) || !ALLOWED_HOSTS.has(target.hostname)) {
    return res.status(400).json({ error: 'destination not allowed' });
  }
  const upstream = await fetch(target, { redirect: 'manual' }); // no automatic redirect to an unchecked host
  res.json({ url, status: upstream.status });
});

Allowlist the destination hosts and schemes, resolve and re-validate the actual address before
connecting (reject loopback/link-local/private ranges), turn off automatic redirect-following (or
re-validate the redirect target), and isolate the fetcher's own network egress as defense in depth.

What's honestly not covered

The Active Test tab's SSRF probe is real, wired, and wouldn't be a no-op against every SSRF bug —
just this specific POST-body-only shape. That's a narrower gap than command injection's "no GUI/CLI
entry point at all," but it's still a gap, and I'd rather show it failing on camera with the reason
attached than imply the platform caught two independent confirmations here when it only caught one.

Try it yourself

The target app and the full step-by-step playbook (every click, every payload, every panel) are
linked below if you want to reproduce this end-to-end against your own local copy.

This is an intentionally vulnerable local training app. Never run these techniques against a
system you don't own or don't have explicit written authorization to test.

Path traversal caught two ways — static analysis and a live probe, no cloud AI

Sat, 29 Aug 2026 08:48:09 +0000

The bug

Here's the route from a small Express demo app
(scan-target-demo-apps/apps/06-path-traversal):

app.get('/files', (req, res) => {
  const name = req.query.name;
  const filePath = path.join(docsDir, name);
  fs.readFile(filePath, 'utf8', (err, data) => {
    if (err) {
      return res.status(404).json({ error: 'file not found', path: filePath });
    }
    res.type('text/plain').send(data);
  });
});

name comes straight from the query string and goes straight into path.join() — no check that the
resolved path actually stays inside docsDir. path.join('./docs', '../secret.txt') happily
resolves to a file outside ./docs. Request /files?name=../secret.txt and the server reads and
returns whatever's sitting one directory above the one it's supposed to be confined to.

Classic CWE-22, OWASP A01:2021 (Broken Access Control). Nothing novel in the bug itself — the
interesting part, again, is what caught it and how many independent ways.

What actually caught it — twice, independently

We ran this target through AI Security Studio, an offline security research platform we've been
building. No cloud calls, no code leaves the machine.

This is the first target in the series where the bug is confirmed by both ends of the pipeline
at once, not just one:

1. Static — before anyone opens the app. The same JavaScript taint rule that flags command
injection also declares a filesystem sink: fs.readFile / readFileSync / writeFile /
.sendFile(, etc. It traces req.query.name → path.join(docsDir, name) → fs.readFile(filePath, …) entirely inside server.js, and auto-creates a finding — Risk: High, CWE-22, OWASP A01:2021 —
with zero manual steps. This scores reachability: untrusted input reaches a filesystem sink.

2. Dynamic — a live, automated probe. Unlike some other bug classes in this series where the
active-probe side isn't wired up yet, path traversal has a real, on-by-default probe
(PathTraversalProbeAdapter, ProbeSafetyTier.ReadOnly). Point it at /files with parameter name
and run it: it comes back with a Critical finding, Confidence: Likely, and — the part that
usually takes a human write-up — an attached, ready-to-run curl reproduction already generated. This
scores confirmed exploitation, which is why it lands at a higher risk level than the static-only
finding for the same root cause. Different numbers for the same bug at different pipeline stages is
expected, not a bug in the platform.

Proving it by hand needs nothing but a browser, too — no tooling required:

GET /files?name=../secret.txt

returns the contents of secret.txt, a file that lives one directory above the app's intended
./docs root and was never meant to be reachable through /files at all.

Why "offline" is the actual point

You can point this at unreleased/internal code without a vendor's cloud pipeline ever touching it.
The rule engine and the active-test probe — not the LLM — do the actual finding and the actual
exploitation. Results are reproducible and don't drift between runs.
The local LLM's job shrinks to something it's actually reliable at: turning the already-structured
evidence (the tainted source line, the curl proof, the CWE/OWASP references) into a plain-language
explanation — "untrusted input reaches a filesystem read via unsanitized path.join, allowing
arbitrary file disclosure outside the intended directory." No raw request/response bodies, no raw
source files, ever get handed to the model directly.

Try it yourself

The demo app is intentionally vulnerable and open source — safe to point any scanner at, including
this one:

Built for security researchers, pentesters, and AppSec teams doing work they're authorized to do.

Command Injection (CWE-78): From Auto-Detected Finding to Live Exploit to Fixed

Thu, 27 Aug 2026 09:57:50 +0000

One tainted variable. One shell on the host. Here's the full story of a real
CWE-78 (OS Command
Injection) vulnerability, from the moment a rule engine flags it to a
disclosure-ready report — all running locally, with no cloud AI anywhere in
the pipeline.

The vulnerable code

The target is a deliberately vulnerable local training app: a tiny "network
diagnostics" tool that pings a host you give it.

app.post('/ping', (req, res) => {
  const host = req.body.host || '127.0.0.1';
  const cmd = `ping -c 1 ${host}`; // BUG: unsanitized string concatenation
  exec(cmd, { timeout: 5000 }, (err, stdout, stderr) => {
    res.json({ cmd, stdout, stderr, error: err ? err.message : null });
  });
});

No validation. No escaping. No argument array — just user input flowing
straight into a shell. And because the JSON response echoes back the exact
cmd string the server built, the vulnerable string is visible in every
response, not just the source — one of the cleanest bugs you'll ever capture
airtight evidence from.

Step 1 — A deterministic rule engine catches it, not an LLM

Before any AI touches this finding, a taint-tracking static analysis rule
walks the exact shape this bug takes:

const host = req.body.host …    →  source: untrusted request value
const cmd  = `ping -c 1 ${host}`  →  assignment: taint propagates through host → cmd
exec(cmd, …)                     →  sink: child_process.exec

One static scan raises a Critical, CWE-78 finding automatically —
zero manual finding-creation, zero guesswork. The finding already carries:

Evidence — the matched source line, with the exact column of the
injection point pinned inside the tainted expression
Risk Level — Critical
Security Reference — CWE-78 / OWASP A03:2021 (Injection)

The rule engine discovers; the LLM only ever explains an already-verified
finding afterward. It never gets to invent a vulnerability on its own.

Step 2 — Proving it live, with zero extra tooling

The form field is prefilled 127.0.0.1; id. Submitting it returns:

{
  "cmd": "ping -c 1 127.0.0.1; id",
  "stdout": "PING 127.0.0.1 ...\nuid=501(demo) gid=20(staff) groups=...\n",
  "stderr": "",
  "error": null
}

cmd proves the server built exactly the string you'd expect from unescaped
concatenation. stdout proves it further — after the ping output, a
uid=…gid=… line: the output of id, a command nobody asked the app to
run. One request/response pair is already enough evidence on its own — no
server access needed to confirm the bug, because the app's own response is
externally observable proof.

Step 3 — Varying the payload (and why encoding matters)

A quick gotcha worth calling out for anyone testing this class of bug by
hand: the request's Content-Type is application/x-www-form-urlencoded,
so a literal host=127.0.0.1 && whoami gets split by the form parser on the
raw & before the payload ever reaches host — the metacharacter never
survives. Percent-encode it instead:

Payload	Encoded body
`127.0.0.1 && whoami`	`host=127.0.0.1%20%26%26%20whoami`
`127.0.0.1 \\| uname -a`	`host=127.0.0.1%20%7Cuname%20-a`
`id` (substitution)	`host=%60id%60`

Four independent shell metacharacters (;, &&, |, `), four
independent proofs that this reaches a real shell — not one lucky payload.

Step 4 — A fully offline explanation

With the static evidence and the live proof both attached to the same
finding, a local LLM generates a plain-language narrative from the
structured evidence — something like "Untrusted input reaches
child_process.exec via string concatenation — full remote code
execution." No network call is made. The model reasons over an
already-verified finding; it doesn't discover anything new.

Step 5 — The fix

const { execFile } = require('node:child_process');

app.post('/ping', (req, res) => {
  const host = req.body.host || '127.0.0.1';
  execFile('ping', ['-c', '1', host], { timeout: 5000 }, (err, stdout, stderr) => {
    res.json({ stdout, stderr, error: err ? err.message : null });
  });
});

execFile with an argument array means there's no shell to interpret in the
first place — host is passed as a literal argument, never concatenated
into a command string. Add strict input validation on host (e.g. an
IP/hostname allow-list pattern) as defense in depth.

What's honestly not covered

Two active-exploitation probes for this bug class exist in the platform's
engine, but neither is currently wired to a GUI checkbox or CLI flag — so
dynamic confirmation via automated active testing isn't possible today for
this specific bug. That's exactly why this walkthrough proves the exploit
by hand instead of relying on tooling that isn't there yet. Calling out
tooling gaps honestly matters more to me than pretending coverage is
complete.

Try it yourself

The target app and the full step-by-step playbook (every click, every
payload, every panel) are linked below if you want to reproduce this
end-to-end against your own local copy.

This is an intentionally vulnerable local training app. Never run these
techniques against a system you don't own or don't have explicit written
authorization to test.

The HTTP QUERY method (RFC 10008) is here — and caching it correctly is harder than it looks

Wed, 19 Aug 2026 09:51:44 +0000

TL;DR

HTTP got a new method: QUERY (RFC 10008, June 2026). It's a safe, idempotent, cacheable request that carries a body — "GET with a body."
It fixes a decade-old pain: complex reads (search/filter/graph queries) that don't fit in a URL, without abusing POST.
The subtle part is caching: the RFC requires the request body to be part of the cache key. Get it wrong and one client gets another client's response.
http-queryable is a small library that does this correctly for Express, Fastify, raw http, and the browser.

What is the QUERY method?

For decades HTTP forced a bad choice for "reads with a big input":

GET — safe, idempotent, cacheable, but no body. Cram everything into the URL and hit length limits.
POST — has a body, but it's unsafe and non-idempotent. Caches and proxies won't treat it as a read, and you can't safely retry.

QUERY is the missing middle: a body like POST, but the semantics of GET — safe, idempotent, and cacheable.

QUERY /search HTTP/1.1
Content-Type: application/json

{ "q": "cats", "filters": { "color": "black" } }

The catch nobody talks about: caching

RFC 10008 §2.7 is explicit: the cache key must incorporate the request content.

Shared HTTP caches have always keyed on method + URL. With QUERY, many different bodies hit the same URL:

QUERY /search  { "q": "cats" }   -> cats
QUERY /search  { "q": "dogs" }   -> dogs

A method+URL cache would serve the cats response to the **dctness and security bug — the RFC's Security Considerationscalls it out.

Doing it right: conservative body normalization

You want two things in tension:

{"a":1,"b":2} and { "b":2, "a":1 } mean the same thing → same key (a hit).
{"q":"cats"} and {"q":"dogs"} differ → *different keys

The key insight is an asymmetry: a false miss is harmle is a bug (wrong data served). So every normalization must beprovably meaning-preserving*, and when in doubt you normalize less*.

Safe for JSON: whitespace, object key order, string escape forms. NOT safe: merging numeric literals (JSON.parse("9007199254740993") returns ...992 — a
silent collision), guessing on duplicate keys, or normalizing ctand.

Show me the code

import express from "express";
import { queryable, QueryCache } from "http-queryable/express";

const app = express();
app.use(queryable({ cache: new QueryCache() }));
app.query("/search", (req, res) => res.json(search(req.body)));
app.listen(3000);

curl -X QUERY /search -d '{"q":"cats"}'      # MISS -> cats
curl -X QUERY /search -d '{ "q" : "cats" }'  # HIT  (same meaning)
curl -X QUERY /search -d '{"q":"dogs"}'      # MISS -> dogs

That third line is the whole point: a different body gets the cr a stale hit.

It handles the rest too

Accept-Query negotiation, Content-Location "switch to GET" flow CORS-safelisted), an isomorphic client with safe auto-retry,and Fastify + raw http adapters over the same core.

Try it

Node >= 22 (where QUERY lands in http.METHODS).

npm install http-queryable

Repo + docs: https://github.com/hardik-goel/http-queryable — stars and edge-case bug reports welcome.

Build an Electron app with local data storage — no SQLite bindings, no server

Sun, 09 Aug 2026 10:38:03 +0000

If you've shipped an Electron app that persists structured data, you've probably hit the wall at some point. You need something more than a flat JSON file, but a full database server is absurd for a desktop app. So you reach for better-sqlite3 — and then spend the next hour fighting native bindings.

The Electron storage problem

Electron bundles its own version of Node.js. That version almost never matches the one on your machine. Native Node modules — the ones that compile C++ bindings — need to be recompiled specifically for the Electron runtime, using electron-rebuild or equivalent tooling.

In practice this means:

Every Electron update potentially breaks your native dependencies
CI pipelines need platform-specific build steps for Windows, macOS and Linux
Code signing and notarization on macOS gets complicated by native binaries
End users on some machines encounter install failures you can't reproduce locally

better-sqlite3 is an excellent library. But it is a native module, and native modules in Electron are a tax you pay forever.

The alternatives people reach for have their own problems. lowdb and lokijs keep everything in memory and flush to disk — fine for small datasets, but you lose durability on crash and memory usage grows with your data. nedb hasn't been maintained in years. A remote database is obviously wrong for a local desktop app.

What you actually need

For most Electron apps, the storage requirements are modest but specific:

Structured documents, not just key-value pairs
Queries that go beyond "give me everything"
Writes that survive a crash
Zero native binaries — pure JavaScript, works everywhere Electron runs
No server process running alongside your app

That's exactly the gap pocket-db is designed to fill. It's a single-file embedded document store — the SQLite model applied to JSON documents, with a MongoDB-style API. Pure TypeScript, shipped as both ESM and CommonJS. No native bindings. No optional dependencies.

Building the example app

Let's build a small note-taking app: create, list, search and delete notes. Simple enough to follow in full, realistic enough to show the pattern.

Setup

We'll use electron-vite, which scaffolds the src/main / src/preload / src/renderer layout used below, with TypeScript and React ready to go:

npm create @quick-start/electron@latest my-notes -- --template react-ts
cd my-notes
npm install @axfab/pocket-db

No electron-rebuild. No postinstall scripts. That's the point.

Initialising the database

In Electron, database access belongs in the main process. You have access to the filesystem there, and it keeps your renderer process clean. Use app.getPath('userData') — Electron's standard location for app data, persisted across updates.

One thing to know: Electron creates the userData directory lazily, so on a first launch it may not exist yet — and open() does not create parent directories. Create it before opening:

// src/main/db.ts
import { app } from 'electron'
import { open } from '@axfab/pocket-db'
import { mkdirSync } from 'node:fs'
import path from 'node:path'

const dataDir = app.getPath('userData')
mkdirSync(dataDir, { recursive: true }) // may not exist on first launch

export const db = open({ path: path.join(dataDir, 'notes.pdb') })
export const notes = db.collection('notes')

One file, one call. The database is created if it doesn't exist. No migrations, no schema definition.

Exposing database operations via IPC

Electron's security model means the renderer can't touch Node APIs directly. You expose operations through IPC handlers in the main process and call them via contextBridge in the preload script.

Search deserves a moment of care: the search term comes from user input, and $regex compiles it as a regular expression. A term containing ( or [ would throw, and a raw pattern is case-sensitive. Escape the input and pass $options: 'i':

// src/main/index.ts
import { app, BrowserWindow, ipcMain } from 'electron'
import { db, notes } from './db'

/** Escapes regex metacharacters so user input is matched literally. */
const escapeRegex = (term: string) =>
  term.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')

ipcMain.handle('notes:create', (_event, title: string, body: string) => {
  return notes.insertOne({ title, body, createdAt: Date.now() })
})

ipcMain.handle('notes:list', () => {
  return notes.find().sort({ createdAt: -1 }).toArray()
})

ipcMain.handle('notes:search', (_event, term: string) => {
  const pattern = escapeRegex(term)
  return notes.find({
    $or: [
      { title: { $regex: pattern, $options: 'i' } },
      { body:  { $regex: pattern, $options: 'i' } }
    ]
  }).toArray()
})

ipcMain.handle('notes:delete', (_event, id: string) => {
  return notes.deleteOne(id)
})

app.on('before-quit', () => db.close())

// src/preload/index.ts
// Note: keep the preload compiled to CommonJS (electron-vite's default) —
// sandboxed renderers don't load ESM preload scripts.
import { contextBridge, ipcRenderer } from 'electron'

contextBridge.exposeInMainWorld('notes', {
  create:  (title: string, body: string) => ipcRenderer.invoke('notes:create', title, body),
  list:    ()           => ipcRenderer.invoke('notes:list'),
  search:  (term: string) => ipcRenderer.invoke('notes:search', term),
  delete:  (id: string) => ipcRenderer.invoke('notes:delete', id),
})

Using it in the renderer

From the renderer, the API is just async function calls:

// src/renderer/src/App.tsx
const handleCreate = async () => {
  await window.notes.create(title, body)
  setNotes(await window.notes.list())
}

const handleSearch = async (term: string) => {
  const results = await window.notes.search(term)
  setNotes(results)
}

const handleDelete = async (id: string) => {
  await window.notes.delete(id)
  setNotes(await window.notes.list())
}

No database logic in the renderer. No connection strings. No async initialisation to wait for. The main process owns the database, the renderer just calls handlers.

Where indexes help — and where they don't

Two things to know about pocket-db's query planner in V1: $regex is never index-assisted (it's always evaluated against the documents), and $or queries bypass index planning entirely. So the regex search above is a full collection scan — perfectly fine for a notes app, but don't expect an index to speed it up.

Indexes shine on equality and range lookups. Say notes belong to notebooks:

// run once at startup, safe to call repeatedly
notes.createIndex('notebook', { type: 'string' })

Now a query like notes.find({ notebook: 'work' }) reads only the matching documents instead of scanning the collection — the planner picks the index up automatically, no change needed in your query code. The same goes for a number index on createdAt if you query by date range.

Periodic compaction

The append-only log accumulates dead records as notes are edited and deleted. Schedule compaction when the app is idle:

app.on('browser-window-blur', () => {
  db.compact()
})

compact() is fast and synchronous — it rewrites the file in a single forward pass. Calling it when the window loses focus means it runs during natural pauses without affecting responsiveness.

Packaging and distribution

This is where the native module problem usually surfaces. With pocket-db, there's nothing to rebuild. The electron-vite scaffold already wires up electron-builder; its config just works as-is:

# electron-builder.yml (generated by the scaffold)
mac:
  target: dmg
win:
  target: nsis
linux:
  target: AppImage

No electron-rebuild in your build pipeline. No platform-specific post-install hooks. The same config builds cleanly on all three platforms.

If you're using GitHub Actions to build releases:

# .github/workflows/release.yml
- name: Build
  run: npm run build && npx electron-builder
  # No electron-rebuild step needed

One fewer thing to maintain.

What about data migration?

pocket-db is schemaless — documents in the same collection can have different shapes. That means additive changes (adding a new field) require nothing: new documents have the field, old ones don't, and $exists lets you query for either.

For breaking changes, a one-time migration on startup is straightforward. Keep a version marker in a metadata collection — and remember that on a fresh database the marker doesn't exist yet:

const schema = db.collection('_meta')
const marker = schema.findOne({ key: 'version' })
const version = typeof marker?.value === 'number' ? marker.value : 0

if (version < 2) {
  // run migration
  notes.updateMany({}, { $set: { archived: false } })

  if (marker) {
    schema.replaceOne(marker._id as string, { key: 'version', value: 2 })
  } else {
    schema.insertOne({ key: 'version', value: 2 })
  }
}

Note the replaceOne(marker._id, …) call: the first argument is the document id. (Passing a full document as the only argument is a separate overload that replaces that document by itself — not what you want here.)

No migration framework needed for most desktop app scenarios.

The trade-offs to know

pocket-db is designed for single-process use. An Electron app fits that model perfectly — the main process is the only writer, and that's exactly the intended use case.

It is not the right choice if your Electron app spawns multiple Node worker processes that all need to write to the same database simultaneously. It is also not the right choice if you're storing hundreds of thousands of documents and need complex aggregation — at that scale, an actual database server starts making sense even in a desktop context.

For the common case — a desktop app storing user data, preferences, history, cached API responses, local task lists — it fits well.

Getting started

npm install @axfab/pocket-db

Full documentation at pocket-db.axfab.net. Source and issues at github.com/axfab/pocket-db. The complete example app from this article is in the repository of exemple, tests included.

If you've been putting up with native binding headaches in Electron, give it a try and let me know how it goes.

pocket-db is MIT licensed.

pocket-db vs lowdb vs LokiJS: an honest embedded database benchmark

Sun, 09 Aug 2026 10:37:29 +0000

If you've ever built a CLI tool, an Electron app, or a local-first prototype in Node.js, you've faced the same question: where do I store structured data without spinning up a server?

The usual suspects are well known — lowdb, LokiJS, nedb, better-sqlite3. I recently built pocket-db, a new embedded document store for Node.js.

I wanted a database solution that offered me:

a single-file database
no native compilation
Mongo-like API
easy embedding in CLI or Electron apps

I wanted to know honestly where it stands. So I ran a benchmark across 10 common operations on a collection of 1,000 documents.
Here's what I found — not shying away from the unflattering parts.

The contenders

pocket-db — append-only single-file document store, MongoDB-like API, indexes in memory
sqlite (in-memory) — better-sqlite3 with an in-memory database, the fastest SQLite configuration
sqlite (file) — better-sqlite3 persisted to disk, the most common SQLite setup
json-file — full JSON file read/write on every operation
lowdb — JSON-based, keeps everything in memory and flushes to disk on write
lokijs — in-memory document store, optional persistence

All tests run on Apple M-series hardware. Results are in ops/sec — so higher is better.

The results

Operation                  pocket-db  sqlite (mem)  sqlite (file)  json-file      lowdb     lokijs
─────────────────────────────────────────────────────────────────────────────────────────────────
insertOne                    198,177     226,278 *         4,421      3,893      2,256      1,004
insertMany (100)               2,345       2,963 *           402        788        628        264
findById                     142,776   1,064,774         248,942  12,532,585 * 321,548  4,061,606
findAll                           96         361             361    115,774    870,822 *     1,179
findByName (scan)                 97         536             524     19,579     24,235 *     8,222
findByRole (index)               277         884             887     18,527     21,796 *     3,215
updateOne                     97,889     423,072 *         2,675        784        566        188
deleteOne                    454,402     465,026 *         5,551        924        663        220
countAll                      12,038   2,233,389         285,285  42,553,191 * 34,914,251 37,348,273
sortByScore (desc)                91         293             293      4,947      5,099 *     1,123
─────────────────────────────────────────────────────────────────────────────────────────────────
All values in ops/sec. * = fastest for this operation.

Before you ask: json-file wins read benchmarks because the entire dataset is already loaded in memory during the benchmark.

Why pocket-db writes so fast

The short answer: every write is a single sequential append to the end of the file. No B-tree rebalancing. No page allocation. No full-file reserialisation.

When you call insertOne, updateOne or deleteOne, pocket-db appends one record to the log and updates the in-memory index pointer. That's it. An update doesn't touch the old record — it writes a replacement which is enough to marks the previous offset as dead. A delete appends a tombstone. This is why deleteOne reaches 454,000 ops/sec, only marginally behind SQLite in-memory.

There's a second reason the numbers look good today: no fsync wait. Writes land in the OS page cache and return immediately. This is fast, but it's a trade-off — a hard crash before the OS flushes to disk can lose the last few writes. Pocket-db is perfectly capable to recover from incomplete operations, so consistency is never compromised. For the target use cases (CLI tools, desktop apps, local prototypes), this is an acceptable trade-off. For anything requiring strict durability guarantees, it's something to be aware of.

Why reads are slower — and what's being done about it

This is the honest part of the benchmark.

pocket-db keeps only indexes in memory, not documents. Every findOne and every cursor step seeks to the document's byte offset in the file and reads from disk. Every result then goes through JSON parsing. This is the fundamental cost of the current design — and it shows clearly in the numbers.

In-memory stores like lowdb and LokiJS serve reads directly from JavaScript objects: no I/O, no parsing. That's why findById on a json-file store reaches 12 million ops/sec while pocket-db sits at 142,000.

Keeping indexes in memory was a necessary first compromise. Without them, reads would require a full file scan for every query — completely unusable in practice. Indexes bring findById to a competitive level for real-world access patterns where you're not reading the same hot document in a tight loop. But the gap on read-heavy workloads is real, and it needs to be closed.

The roadmap for the next version is to address this directly. The first candidate is a document cache: keep recently accessed documents in memory and avoid redundant disk reads for hot-document workloads. This alone should close most of the gap for typical usage patterns. The second candidate is a binary document format to replace JSON serialisation — parsing structured binary is significantly faster than JSON.parse for complex documents. A third option under consideration is memory-mapped I/O with pagination, which would allow the OS to manage the hot-document cache at a lower level.

The memory footprint is also something to watch. Indexes in memory work well at the scales pocket-db is designed for — thousands to low hundreds of thousands of documents. As collections grow, that in-memory index cost grows with them. Load testing at larger scales is to be considered before locked of the architecture.

Reading the numbers in context

The benchmark is deliberately narrow: 1,000 documents, ten operations, one process. That's a fair representation of the workloads pocket-db is built for — a CLI tool tracking state, an Electron app storing user data, a local server caching API responses.

At that scale, the practical difference between 97 and 870,000 read ops/sec is manageable. A CLI command doesn't loop through findByName a million times. But it does insert records frequently, update state on every run, and occasionally compact. For that profile, pocket-db's write throughput is genuinely competitive.

Where the numbers are a real concern is if you're considering pocket-db for a workload that's read-heavy by nature — serving requests in a tight loop, scanning large collections on every query. For those cases today, a pure in-memory store is the honest recommendation.

Where things stand

pocket-db is a young project. These benchmark results are encouraging for what it's designed to do: fast, durable writes in a single-file embedded store with a familiar API. The write story is solid. The read story needs work, and the work ahead is well defined.

The next performance chapter — document cache, binary format, larger-scale index testing — will determine whether pocket-db can compete across the full spectrum of embedded workloads. That's the honest assessment at this first version.

If you want to run the benchmarks yourself:

git clone https://github.com/axfab/pocket-db
npm install
npm run bench

And if a single-file, zero-dependency document store with a MongoDB-style API sounds like what your next project needs:

npm install @axfab/pocket-db

@axfab/pocket-db is MIT licensed. Contributions, issue reports, and benchmark challenges welcome.

Converting a JavaScript-Rendered Web Page to PDF

Sat, 01 Aug 2026 18:11:47 +0000

If you've ever tried to turn a modern web page into a PDF programmatically, you've probably hit the wall: the file comes out blank, half-empty, or frozen on a loading spinner. The page looks perfect in the browser, so what gives?

The answer is timing. Most PDF approaches grab the HTML before the JavaScript has rendered the content. On a server-rendered page that's fine — the markup is already there. On a React/Vue/Angular app, the server sends an near-empty shell and the browser builds the DOM afterward. Capture too early and you save the shell.

Here's how to do it properly.

Why the naive approaches fail

wkhtmltopdf is the classic Google answer. It's fast and it's been around forever, but it uses an ancient WebKit build with effectively no modern JavaScript support. For a static page it's fine. For anything client-rendered, it captures the empty state.

Browser window.print() / Ctrl+P works because it is a real browser — but it's manual, single-page, and impossible to automate cleanly at scale.

Hitting the raw HTML with an HTTP client (axios/fetch then pipe to a PDF lib) has the same fatal flaw as wkhtmltopdf: no JS execution, no rendered content.

What you actually need is a real browser engine that runs the page's JavaScript, waits for it to settle, and then prints. That's Puppeteer.

The Puppeteer approach

Puppeteer drives a headless Chromium. It executes the page exactly like a normal Chrome tab, so whatever renders on screen is what you capture.

const puppeteer = require('puppeteer');

async function pageToPdf(url, outPath) {
  const browser = await puppeteer.launch({
    args: ['--no-sandbox', '--disable-setuid-sandbox'], // needed in most containers
  });
  const page = await browser.newPage();

  await page.goto(url, { waitUntil: 'networkidle0', timeout: 60000 });

  await page.pdf({
    path: outPath,
    format: 'A4',
    printBackground: true, // otherwise CSS backgrounds/colors are dropped
    margin: { top: '20px', bottom: '20px', left: '16px', right: '16px' },
  });

  await browser.close();
}

pageToPdf('https://example.com', 'out.pdf');

Two options here matter more than they look:

waitUntil: 'networkidle0' tells Puppeteer to consider navigation finished only when there have been no network connections for 500ms. For most SPAs that's the signal the data has loaded and the DOM is built. (networkidle2 allows up to 2 connections — useful for pages with long-polling or analytics beacons that never fully go quiet.)
printBackground: true — forget this and every background color, gradient, and image disappears, because Chrome's print path strips them by default.

The gotchas that still bite you

Getting a non-blank PDF is step one. Getting a correct one is where the edges show up.

Lazy-loaded content

Images and sections that load on scroll won't be in the PDF, because Puppeteer never scrolled. Force them in by auto-scrolling before you print:

await page.evaluate(async () => {
  await new Promise((resolve) => {
    let y = 0;
    const step = 300;
    const timer = setInterval(() => {
      window.scrollBy(0, step);
      y += step;
      if (y >= document.body.scrollHeight) {
        clearInterval(timer);
        window.scrollTo(0, 0);
        resolve();
      }
    }, 100);
  });
});

Web fonts not loaded yet

Text renders in a fallback font because the web font hadn't finished loading at print time. Wait for it explicitly:

await page.evaluateHandle('document.fonts.ready');

Print CSS overriding your layout

If the site ships a @media print stylesheet, Chrome applies it. Sometimes that's what you want; sometimes it nukes the layout. Force screen rendering with:

await page.emulateMediaType('screen');

Fixed headers repeating or covering content

Sticky/fixed elements can duplicate on every PDF page or overlap the body. The reliable fix is to neutralize them before printing — set their position to static (or hide them) via page.addStyleTag().

When self-hosting Puppeteer isn't worth it

Puppeteer is the right tool when you need PDFs inside a pipeline you control. But it comes with a real operational tail: a headless Chromium is heavy (200MB+), it needs the right system libraries in your container, it leaks memory if you don't manage browser instances carefully, and it falls over under concurrency unless you pool pages and cap parallelism. For a side feature — "let users export this page" — that's a lot of infrastructure to babysit.

If you just need the output and not the plumbing, a hosted service does the same headless-render-then-capture under the hood without the ops burden. I run site2pdf.online for exactly this — it renders each page in real Chrome (so JS content comes through), and it can also follow a site's internal links to capture multiple pages in one pass, exporting to PDF, PNG, or ZIP. Handy when the alternative is standing up and maintaining your own Chromium fleet.

Rule of thumb: if PDF generation is core to your product, self-host Puppeteer and own it. If it's a convenience feature at the edge, offload it.

Wrapping up

The blank-PDF problem is almost always a timing problem: capture happens before JavaScript renders. Use a real browser engine, wait for the network (and fonts) to settle, scroll to trigger lazy content, and enable printBackground. Do that and dynamic pages convert cleanly.

The full, copy-pasteable version of this — plus the fixed-header and multi-page handling — is worth keeping in a snippet somewhere. You'll reach for it more often than you'd expect.

Blockend Logger vs Traditional Pino Logger

Fri, 31 Jul 2026 10:37:52 +0000

Logging is essential, but setting it up with Pino often means boilerplate: installing packages, wiring middleware, passing request IDs manually, and configuring redaction yourself. It works, but it’s repetitive.

Blockend Logger takes a different approach. It uses Pino under the hood but generates a ready‑to‑use, type‑safe logger block directly into your project. You own the code, you can debug it, and you can change it.

Why Blockend Logger Feels Different

Shared logger instance across your app
Automatic request ID propagation with AsyncLocalStorage
Framework adapters generated automatically (Express, Fastify, Hono)
Request logging out of the box
Built‑in redaction for sensitive fields
Pretty logs in dev, JSON logs in prod
TypeScript support baked in

Honest Comparison

Feature	Blockend Logger	Traditional Pino
Setup	`blockend-cli add logger`	Manual install + config
Request IDs	Automatic	Manual middleware
Framework integration	Auto‑generated adapter	Write your own
Request logging	Built‑in	Add middleware manually
Redaction	Preconfigured	Manual setup
Output modes	Pretty dev / JSON prod	Configure transports yourself
TypeScript	Generated `.ts` files	Manual typings

Blockend isn’t replacing Pino — it’s making Pino easier to use. If you want raw control, stick with Pino. If you want sane defaults and less boilerplate, Blockend Logger saves you time.

Learn More

Docs: Blockend Logger Documentation
Source: Blockend on GitHub

Final Word

Logging should help you debug, not slow you down. Blockend Logger is basically “Pino with batteries included.” If you love it, star it on GitHub — that’s the best way to support the project and help more developers discover it.

Kubernetes Cluster Autoscaler vs Karpenter: When to Use Each (2026)

Thu, 30 Jul 2026 10:00:00 +0000

Your pods are pending. Your on-call engineer is getting paged. Somewhere in the chain between “I need more compute” and “compute is available,” something is too slow. That something is almost always node provisioning — and the tool you chose to manage it determines whether that delay is 4 minutes or 45 seconds.

Node autoscaling is one of those infrastructure decisions that looks simple until you’re running it in production. Two schedulable pods sitting in Pending state doesn’t just mean a delayed deployment — it means latency spikes, dropped traffic, breached SLOs, and engineers debugging things that should have been invisible. At scale, it also means either burning money on over-provisioned nodes or gambling on under-provisioning at the worst possible moment.

Cluster Autoscaler (CA) has been the default answer for years. Karpenter emerged from AWS in 2021, graduated to stable in 2023, and by 2025 had become the default recommendation for most AWS-native clusters. In 2026, both tools are mature, widely deployed, and genuinely good — but they solve the problem differently, and picking the wrong one for your environment has real consequences.

This article is a deep technical comparison. It assumes you already know what Kubernetes is and have opinions about infrastructure. The goal is to give you a clear picture of how each tool works, where each one wins, and a decision framework you can actually use.

Why Node Autoscaling Is Hard

The fundamental tension in autoscaling is this: you want compute available before you need it, but you don’t want to pay for compute you’re not using. These goals are in direct conflict, and every autoscaling system is an attempt to find the least-bad trade-off.

Without autoscaling, you’re doing one of two things:

Over-provisioning — you run enough nodes to handle peak load at all times. Your average utilization sits at 20–30%, and you’re paying for the other 70–80% to sit idle.
Under-provisioning — you run lean, and when traffic spikes, pods go Pending. Your SLOs breach. You get paged at 3am to manually scale.

A common failure mode with poorly tuned autoscaling is the “thundering herd at scale-up” pattern: HPA creates new pods faster than node autoscaling can provision capacity. The provisioning window matters. With CA and typical ASG-backed node groups on AWS, you’re looking at 4–8 minutes. With Karpenter, 60–90 seconds. At 100 RPS and a 3-minute window, that’s 18,000 requests under degraded conditions.

Cluster Autoscaler: How It Actually Works

Cluster Autoscaler is a Kubernetes-native project under the kubernetes/autoscaler repository, in production since 2016, supporting AWS, GCP, Azure, Alibaba, DigitalOcean, and more.

The Node Group Model

CA operates on node groups — ASGs on AWS, MIGs on GCP, VMSSs on Azure. CA’s job is to decide when to increase or decrease the desired capacity of these groups. CA does not provision individual nodes. It scales node groups, and the node group provisions nodes. This indirection adds latency and reduces flexibility.

Scale-Up: Detecting Unschedulable Pods

CA runs a control loop (default scan interval: 10 seconds). For each Pending pod with PodScheduled=False, CA simulates adding a node of each known node group type and checks if the pod would become schedulable. When a node group is selected, CA applies an expander to choose which group to scale:

least-waste — minimizes CPU/memory waste after scheduling (best default for cost)
most-pods — maximizes pods scheduled per scale-up operation
priority — lets you define ordering via ConfigMap
grpc — delegates to an external gRPC service

# Cluster Autoscaler deployment — AWS, production-tuned
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: cluster-autoscaler
      containers:
        - image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.30.3
          name: cluster-autoscaler
          resources:
            requests:
              cpu: 100m
              memory: 600Mi
            limits:
              cpu: 200m
              memory: 1Gi
          command:
            - ./cluster-autoscaler
            - --cloud-provider=aws
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
            - --expander=least-waste
            - --balance-similar-node-groups=true
            - --scale-down-delay-after-add=10m
            - --scale-down-unneeded-time=10m
            - --scale-down-utilization-threshold=0.5
            - --max-graceful-termination-sec=600
            - --scan-interval=10s

Scale-Down: The Conservative Approach

A node is a scale-down candidate only if:
– CPU and memory utilization (by requests) is below threshold (default: 50%)
– All pods could be rescheduled elsewhere
– No pod has cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
– The node has been underutilized for at least --scale-down-unneeded-time (default: 10m)

This conservatism prevents churn — a feature, not a bug.

Karpenter: How It Actually Works

Karpenter is a CNCF incubating project originally built by AWS, donated to CNCF in 2023, GA (v1.0) in mid-2024. Providers exist for AWS (stable), Azure (stable), and GCP (beta).

The Core Insight: Bypass the Node Group

Karpenter calls the EC2 RunInstances API directly — no ASG involvement. This means:
– Any instance type in a single request, without pre-configuring a node group
– No intermediary: Karpenter → EC2 API → node joins cluster
– Right-size nodes to exactly what workloads need, across the full instance catalog
– Karpenter handles full node lifecycle, including termination

NodePool and EC2NodeClass

# EC2NodeClass — cloud-specific parameters
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  role: "KarpenterNodeRole-my-cluster"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true
  metadataOptions:
    httpTokens: required
    httpPutResponseHopLimit: 1
---
# NodePool — intent and constraints
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: ["nano", "micro", "small", "medium", "large"]
      expireAfter: 720h
  limits:
    cpu: "1000"
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
      - nodes: "5%"
        schedule: "0 8 * * mon-fri"
        duration: 10h
      - nodes: "25%"

Just-in-Time Provisioning and Bin Packing

When pods go Pending, Karpenter watches the event (not polls) and immediately:
1. Collects all Pending pods
2. Simulates bin packing — fewest possible nodes across the full instance catalog
3. Selects instances that satisfy all pod requirements
4. Calls EC2 API to provision the optimal instance(s)

Disruption and Consolidation

Karpenter’s differentiated value: active cluster consolidation. It evaluates whether nodes can be removed by redistributing pods onto others, or replaced with a smaller instance type. A c5.4xlarge running 4 vCPU worth of pods gets replaced with a c5.xlarge. Teams commonly report 30–50% compute cost reduction.

consolidationPolicy options:
– WhenEmpty — only remove nodes with no workload pods (safest)
– WhenEmptyOrUnderutilized — also replace underutilized nodes with smaller ones

Architecture Comparison

Dimension	Cluster Autoscaler	Karpenter
Node provisioning model	Scales node groups (ASG/MIG/VMSS)	Direct cloud API, no node groups
Instance flexibility	Pre-defined node group types	Full instance catalog at runtime
Scale-up trigger	Polling (10s scan interval)	Watch-based event (near-instant)
Scale-down	Removes underutilized nodes	Removes + consolidates + right-sizes
Spot handling	Via ASG + AWS Node Termination Handler	Native, first-class, no NTH needed
Configuration model	Deployment flags	Declarative CRDs
Cloud support	All major + on-prem	AWS (GA), Azure (stable), GCP (beta)
Consolidation	No	Yes
Community maturity	Very mature (since 2016)	Mature (GA 2024)

Scaling Speed: The Numbers

Cluster Autoscaler on AWS (typical):
1. Pod Pending → CA scan detects (0–10s)
2. ASG UpdateAutoScalingGroup API call (~15–30s)
3. EC2 instance starts (1–2 min)
4. Node bootstrap + kubelet registration (30–60s)
5. Pod scheduled (5–10s)

Total: 3–6 minutes (up to 12 min during high-demand periods)

Karpenter on AWS (typical):
1. Pod Pending → watch event fires (~1s)
2. EC2 RunInstances API call (~2–3s)
3. EC2 instance starts (same hardware — 1–2 min)
4. Node bootstraps + Ready (30–60s)
5. Pod scheduled (~5s)

Total: 90 seconds to 3 minutes

Cost Optimization: Where Karpenter Pulls Ahead

Right-sizing: CA requires pre-defined node groups. Karpenter selects the minimum viable instance for the pending workload from the full catalog.

Consolidation vs scale-down: CA removes underutilized nodes. Karpenter replaces a large underutilized node with a smaller one that still fits all pods. This produces compounding savings over time.

Spot handling: Karpenter receives EC2 interruption notices, pre-provisions a replacement, and drains the node — all within the 2-minute window. No AWS Node Termination Handler required. It also diversifies spot requests across instance types automatically to reduce simultaneous interruption risk.

Multi-Cloud Support in 2026

Cloud	Cluster Autoscaler	Karpenter
AWS	Production-stable	Production-stable (reference impl.)
GCP	Production-stable	Beta (karpenter-provider-gcp)
Azure	Production-stable	Stable (karpenter-provider-azure)
Alibaba	Supported	No provider
DigitalOcean	Supported	No provider
On-premises / Cluster API	Supported	Not supported

When Cluster Autoscaler Is Still the Right Choice

Non-AWS environments — GCP, Alibaba, DigitalOcean, on-prem with Cluster API
Existing node group architecture — significant investment in ASG design, compliance tooling
Regulatory constraints — some frameworks require ASG-backed provisioning audit trails
Cluster API / bare metal — CA is the only mature option
Team familiarity and working-well CA deployment — migration cost may not justify benefit

When Karpenter Is the Right Choice

AWS-native, cost optimization priority — right-sizing + consolidation = meaningful cost reduction
Diverse and variable workloads — batch, spot, GPU, stateless APIs — Karpenter handles all with a few NodePools
Spot-heavy clusters — native interruption handling, diversification, no NTH
Declarative infrastructure-as-code culture — NodePools version cleanly in Git
Low-latency scaling requirements — event-driven workloads, KEDA-triggered jobs, sharp traffic spikes

Running Both: Migration Path and Gotchas

Separating Responsibility

Use labels and taints to prevent CA and Karpenter from managing the same nodes:

# NodePool with taint — CA-managed pods won't tolerate this
spec:
  template:
    metadata:
      labels:
        provisioner: karpenter
    spec:
      taints:
        - key: karpenter.sh/provisioned
          value: "true"
          effect: NoSchedule

Gradual Migration

Phase 1 — Karpenter manages spot/batch workloads. CA manages on-demand production nodes.
Phase 2 — Migrate spot workloads fully. Remove AWS NTH.
Phase 3 — Migrate on-demand. Reduce CA node group capacity gradually.
Phase 4 — Decommission CA once all groups are empty.

Key Gotchas

Karpenter consolidation + permissive PDBs — maxUnavailable: 100% will cause disruptive consolidation. Audit PDBs before enabling WhenEmptyOrUnderutilized.
NodePool limits are hard stops — pods go Pending indefinitely at limit. Monitor utilization.
AMI drift — @latest alias picks up new AMIs on new nodes. Consider pinning for strict change control.
Simultaneous scale-down conflicts — use strict label/taint segregation during migration.

Decision Framework

Factor	Cluster Autoscaler	Karpenter
Cloud support	All clouds +

EKS Auto Mode: What It Actually Changes (and What It Doesn’t)

Mon, 27 Jul 2026 10:00:00 +0000

What EKS Auto Mode is

EKS Auto Mode, generally available on December 1, 2024, shifts more Kubernetes infrastructure responsibility from you to AWS. You still run an EKS cluster in your AWS account, and your workloads still use the Kubernetes API, but AWS takes over much of the compute, storage, networking, node lifecycle, and core add-on management that platform teams usually wire together themselves.

For compute, Auto Mode uses Karpenter-style provisioning under the hood. When pods are unschedulable, Auto Mode provisions nodes that fit the workload’s requirements: instance family, size, architecture, capacity type, and availability zone. When capacity is no longer useful, it can consolidate and terminate nodes.

The important framing is this: Auto Mode is not “EKS without nodes.” It is EKS where AWS manages the node lifecycle more aggressively. You own the workloads, their scheduling requirements, their disruption behavior, and the operational consequences of those choices. AWS owns more of the infrastructure plumbing.

What it replaces

Before Auto Mode, running EKS in production usually meant choosing and operating several layers yourself:

Managed node groups: You chose instance types, defined scaling ranges, managed AMI updates, handled node draining, and configured Cluster Autoscaler or another scaling mechanism.

Self-managed Karpenter: More flexible than managed node groups, but you owned the Karpenter controller, IAM, NodePools, EC2NodeClasses, disruption settings, upgrades, and failure modes.

Fargate: AWS-managed compute per pod, with no node management, but no DaemonSets, a narrower workload compatibility envelope, and a different cost model.

EKS Auto Mode replaces a large part of that platform assembly with a managed model: declare workload intent and high-level compute constraints; AWS provisions and manages the EC2 instances behind it.

How it works in practice

You create or update an EKS cluster with Auto Mode enabled. The default setup can use AWS-managed built-in node pools. If you need more control, you create a NodeClass for Auto Mode infrastructure settings and a Karpenter NodePool for workload-facing scheduling constraints.

# NodeClass: EKS Auto Mode infrastructure settings for managed EC2 nodes.
apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: private-compute
spec:
  subnetSelectorTerms:
    - tags:
        kubernetes.io/role/internal-elb: "1"
  securityGroupSelectorTerms:
    - tags:
        aws:eks:cluster-name: prod-eks
  ephemeralStorage:
    size: "100Gi"

# NodePool: workload-facing constraints for nodes that Auto Mode may provision.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: private-compute
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: eks.amazonaws.com/instance-category
          operator: In
          values: ["c", "m", "r"]
  limits:
    cpu: "1000"
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

Those API groups are the Auto Mode-specific split documented by AWS: apiVersion: eks.amazonaws.com/v1, kind: NodeClass for Auto Mode node infrastructure, and apiVersion: karpenter.sh/v1, kind: NodePool for scheduling and capacity constraints.

The NodeClass is where you express AWS infrastructure placement and node-level defaults. The NodePool is where you express what kind of capacity is acceptable for workloads. Do not copy a self-managed Karpenter EC2NodeClass into Auto Mode; Auto Mode uses its own NodeClass API.

Auto Mode provisions nodes when pods are pending, consolidates when nodes are underutilized, and replaces nodes during maintenance or scale-down. AMI and node lifecycle updates are handled by AWS. AWS says Auto Mode AMIs are generally released weekly with CVE and security fixes, and Auto Mode nodes have a maximum lifetime of 21 days, which you can reduce. Your application still has to tolerate the disruption: a bad PodDisruptionBudget, strict affinity rule, or singleton stateful workload can still block or degrade a replacement.

Built-in components are managed differently than in a classic EKS build. AWS lists pod networking, service networking, cluster DNS, autoscaling, block storage, load balancer controller, Pod Identity agent, and node monitoring agent as Auto Mode capabilities. With Auto Mode compute, common add-ons such as Amazon VPC CNI, kube-proxy, CoreDNS, Amazon EBS CSI Driver, and EKS Pod Identity Agent become redundant for Auto Mode nodes, and the relevant controllers can run on AWS-owned infrastructure rather than as visible pods in your account. You can still install AWS Load Balancer Controller in an Auto Mode cluster when you need both models during migration, but AWS does not support directly migrating existing load balancers from AWS Load Balancer Controller to Auto Mode; use IngressClass or loadBalancerClass boundaries and plan blue-green migration. Treat this as a change in ownership, not as a reason to skip validation.

The workload support matrix is broader than Fargate, but not identical to self-managed nodes:

Capability	Auto Mode status
EC2 Spot	Supported through `karpenter.sh/capacity-type` requirements such as `spot`, `on-demand`, and `reserved`
Graviton / arm64	Supported through `kubernetes.io/arch: arm64` and supported Graviton instance families
GPU / accelerators	Supported for documented accelerated families; Auto Mode manages NVIDIA, Trainium, and Inferentia drivers/device plugins for supported instance types
Windows nodes	Not supported
DaemonSets	Supported as Kubernetes DaemonSets, but host-level assumptions must be validated against locked-down managed nodes

What you gain

Reduced operational surface. Node group management, AMI lifecycle, Cluster Autoscaler tuning, Karpenter controller upgrades, and a chunk of add-on wiring move out of your day-to-day scope.

Better provisioning shape by default. Dynamic provisioning is usually a better fit than fixed node group shapes. You get nodes that more closely match actual pod requirements instead of trying to pre-plan a small set of instance types.

Automatic node patching. AWS manages the node image and replacement flow. That reduces toil, but it also means your workloads need disruption policies that let AWS replace nodes safely.

Faster cluster bootstrapping. A new Auto Mode cluster can get to a usable production baseline faster than a hand-assembled EKS cluster with node groups, autoscaling, networking add-ons, storage drivers, and load balancer controllers.

Native Spot integration. Auto Mode can use Spot capacity through NodePool requirements, but you still need workload-level interruption tolerance: replicas, budgets, graceful shutdown, and queue semantics where relevant.

What you give up

Node-level access. Auto Mode nodes are intentionally locked down compared with traditional self-managed nodes. If your incident response process assumes SSH, SSM, manual package inspection, or ad hoc host changes, it needs to change.

Custom AMIs. You cannot treat the node image as your own artifact. AWS determines the operating system and AMI for Auto Mode managed instances; you cannot directly access the instance or install software on it. If your organization requires internally built, hardened, or certified AMIs, Auto Mode is likely blocked.

Unrestricted host agents. Kubernetes DaemonSets are supported, but they are the sharp edge. Some node agents work; others do not. Anything that assumes privileged host access, custom kernel modules, hostPath writes, IMDS access without hostNetwork, or low-level runtime integration needs a proof of compatibility.

Less tuning surface. You give up direct control over kubelet flags, container runtime configuration, bootstrap scripts, and arbitrary node setup. That is the point of the product, but it is also the boundary.

Different cost visibility. Managed node groups make capacity easier to reason about because you chose it up front. Auto Mode changes capacity dynamically, so cost control moves toward budgets, labels, reports, and workload-level resource hygiene.

EKS Auto Mode vs managed node groups vs Fargate

	Auto Mode	Managed Node Groups	Fargate
Node management	AWS manages node lifecycle	Shared: AWS manages the group primitive, you manage capacity shape and many updates	AWS-managed per-pod compute
AMI updates	Automatic through AWS-managed node replacement	You schedule and operate rolling updates	N/A
Instance selection	Dynamic through NodePools	You choose instance types and scaling ranges	Not exposed
Custom AMIs	No; AWS determines the AMI	Yes	No
DaemonSets	Supported, but validate host-access assumptions	Yes	No
SSH / node access	Restricted	Usually available if you enable it	No
Spot support	Yes, through capacity-type requirements	Yes, with node group or Karpenter design	No; Amazon EKS does not support Fargate Spot
Cost model	EKS control plane + EC2 + EKS Auto Mode fee	EKS control plane + EC2	EKS control plane + Fargate pod pricing
Operational burden	Low for nodes, medium for workload compatibility	Medium	Low for nodes, medium for compatibility
Right for	Default candidate for teams that do not need node customization	Regulated/custom node environments and mature platform teams	Workloads that fit Fargate’s restrictions and want per-pod isolation
Hidden constraints / gotchas	AWS controls node image and lifecycle; DaemonSets, privileged pods, hostPath, PDBs, topology rules, and node agents can block migration	You still own AMI drift, autoscaler tuning, disruption handling, and capacity fragmentation	No DaemonSets, limited host-level integrations, different networking/storage constraints, and less flexibility for mixed workload shapes

The Auto Mode pricing

Do not model Auto Mode as “EC2 plus a generic percentage” unless you have pulled the actual rate for your region and instance mix. The official structure is:

Total EKS Auto Mode cluster cost =
  EKS control plane cost
  + normal EC2 cost for instances launched and managed by Auto Mode
  + EKS Auto Mode management fee on those managed EC2 instances
  + normal surrounding AWS costs: EBS, load balancers, data transfer, CloudWatch, etc.

EKS Auto Mode management fee =
  sum of Auto Mode-managed instance runtime
  x the regional EKS Auto Mode management rate for each EC2 instance type

The EKS Auto Mode fee is applied to the EC2 instances that Auto Mode launches and manages. It is billed in addition to the normal EC2 charge and in addition to the EKS control plane charge. AWS bills the Auto Mode fee per second with a one-minute minimum, and the charge is independent of whether the underlying EC2 capacity is On-Demand, Spot, covered by Reserved Instances, or covered by Compute Savings Plans.

Do not treat the fee as a contractual flat percentage. The official pricing page describes it as a management fee that varies by EC2 instance type, and AWS pricing data is regional. The public pricing example for US West (Oregon) shows c6a.2xlarge at $0.306/hour for EC2 plus $0.03672/hour for Auto Mode, c6a.4xlarge at $0.612/hour plus $0.07344/hour, m5a.2xlarge at $0.344/hour plus $0.04128/hour, and m5a.xlarge at $0.172/hour plus $0.02064/hour. Those examples equal 12% of the listed On-Demand EC2 rate, but the safe formula for real planning is: sum(instance-hours by instance type and region x published Auto Mode management rate).

The practical cost question is not “is there a premium?” There is. The useful question is whether the premium is lower than the engineering time, incident risk, and opportunity cost of operating node lifecycle yourself.

For small teams, the answer may be yes even if the raw bill increases. For high-scale, cost-sensitive platforms, the answer needs real data: compare current EC2 waste, bin-packing efficiency, Spot usage, interruption rate, and platform maintenance time against an Auto Mode pilot.

When to use EKS Auto Mode

Use Auto Mode if:

You run EKS on AWS and do not have a hard requirement to manage nodes yourself
You do not require custom AMIs or custom node bootstrap logic
You want to reduce the operations surface for node lifecycle management
You want Karpenter-like provisioning without operating Karpenter yourself
Your workloads are mostly stateless or disruption-tolerant
Your observability, security, and storage agents are compatible with Auto Mode

Stick with managed node groups if:

Your organization requires internally certified or hardened AMIs
You need specific kernel configuration, kubelet flags, bootstrap scripts, or host packages
You depend on privileged DaemonSets or host-level security tooling that Auto Mode cannot support
You are in a regulated environment where the node image supply chain must be owned internally
Your platform team already operates Karpenter well and values the extra control

Use Fargate if:

You specifically want per-pod compute isolation
Your workload does not need DaemonSets or host-level integrations
You accept Fargate’s scheduling, networking, storage, and observability constraints
You want to avoid managing EC2 capacity entirely for a narrow class of workloads

Migration from managed node groups

Migrating an existing cluster to Auto Mode is supported, but it is not a one-command operational migration. AWS supports enabling Auto Mode on existing clusters, but you must update the cluster IAM role permissions and trust policy, enable compute, block storage, and load balancing capabilities together, and meet required add-on versions when those add-ons are installed. AWS also calls out unsupported direct migrations for EBS volumes from the standard EBS CSI provisioner to the Auto Mode EBS CSI provisioner, existing load balancers from AWS Load Balancer Controller to Auto Mode, and clusters using alternative CNIs or other unsupported networking configurations. A conservative path looks like this:

Enable Auto Mode on a non-production cluster running Kubernetes 1.29 or greater.
Inventory workloads by scheduling assumptions: node selectors, affinities, tolerations, topology spread, PDBs, privileged mode, hostPath, local storage, and DaemonSet dependencies.
Create or select the relevant Auto Mode NodeClass and NodePool resources.
Move a low-risk namespace first by changing selectors, tolerations, or labels so pods land on Auto Mode nodes.
Watch scheduling, replacement, load balancer behavior, persistent volume provisioning, logging, metrics, and security events.
T

pnpm v12 Is Being Rewritten in Rust

Sat, 18 Jul 2026 22:37:54 +0000

Every package manager has a moment where you just wait. You add one dependency, or you pull a branch and run install, and there's a beat where the terminal sits there doing file I/O while you tab away. pnpm already made that beat shorter than most, staying ahead of npm and Yarn on most install scenarios thanks to its content-addressable store and hard-linked node_modules. But it was still a Node.js program, paying the Node startup cost and pushing tens of thousands of file operations through a single JavaScript runtime.

That is what changes in pnpm v12: it rewrites the install engine, the part that fetches and links packages, in Rust, and leaves the CLI, lockfile, and node_modules layout untouched. It's the same move TypeScript just made with Go, aimed at a different bottleneck. And like that one, the interesting detail is how little is supposed to change for you.

What pnpm v12 actually is

pnpm v12 is the same pnpm you already use, ported to Rust. pnpm has always been TypeScript running on Node.js, and the Rust work lives under the codename pacquet, whose repo describes it in one line: "The official pnpm rewrite in Rust." The key word there is rewrite, not reimagining. It's a port. The flags, defaults, error codes, lockfile format, and node_modules layout are all meant to match pnpm exactly.

Zoltan Kochan, pnpm's creator, put it plainly when the v12 plan came together: "v12 should behave the same as v11, the biggest change will be the full rewrite to Rust." That's the frame to hold onto. This is a performance release wearing a major version number, not a redesign of how pnpm works.

pacquet started as a separate experimental repo and moved into the main pnpm monorepo in May 2026, so the TypeScript and Rust versions now evolve side by side. The eventual goal is for the Rust engine to be what you run.

Why did pnpm choose Rust?

The motivation isn't abstract. Two things were capping how fast a Node-based pnpm could go.

The first is startup cost. Every invocation booted a Node.js runtime before it did any work. pnpm v12 ships as native per-platform binaries with no Node.js launcher, so a command pays no runtime bootstrap cost at all. On a fast, cached install where the actual work is milliseconds, that bootstrap was a real fraction of the wall-clock time.

The second is the file I/O itself. A cold install on a large monorepo is really a lot of filesystem operations: fetching tarballs, unpacking them into the store, and hard-linking thousands of files into place. Running all of that through a single JavaScript thread leaves throughput on the table. Rust handles that fan-out natively, without the runtime sitting in the middle of every syscall.

There's a softer reason too, and it's the same one that's driving the broader trend: the tooling to do this kind of rewrite has gotten dramatically better, and AI-assisted porting made a 1:1 translation of a large codebase far more tractable than it would have been a few years ago.

How much faster is pnpm v12?

pnpm publishes a benchmark suite comparing v11 against the Rust engine directly. As of this writing (July 2026), here's what it shows:

Scenario	pnpm v11	pnpm (Rust)
Everything warm (repeat install)	381 ms	12 ms
node_modules present, no cache	460 ms	40 ms
Clean install (nothing cached)	6.5 s	2.2 s

The warm case is everything already in place, so the install just re-verifies the tree, which is what happens on every CI job. That drops from 381ms to 12ms, roughly a 30x difference, and it's the number you'll feel most, because it runs constantly. The colder cases improve less dramatically but still meaningfully: a fully clean install with nothing cached goes from 6.5s to 2.2s, about 3x.

Treat these as pnpm's own numbers on pnpm's own hardware, and expect your mileage to vary with disk speed and project size. But the direction is not subtle, and the team's claim is that the Rust engine is faster than v11 in every scenario they measured.

It's already rolling out, incrementally

Here's the part that's easy to get wrong: pnpm v12 isn't a big-bang release. The Rust engine has been landing piece by piece through the v11 line, opt-in first.

pnpm 11.2 let you opt into pacquet as the install backend for materialization only. Rust did the fetch and link, while pnpm still resolved dependencies and wrote the lockfile.
pnpm 11.7 went further: a standard install could be delegated to pacquet end-to-end, with resolution, lockfile write, and linking all in a single Rust pass.
pnpm 11.10 added pnpm self-update next-12, so you can pull the v12 line and try the Rust binary directly.

In v12, the Rust engine for fetching and linking becomes the default rather than an opt-in. The official roadmap then works outward from there: first the headless frozen-lockfile installer, then full dependency resolution (add, update, remove, catalogs, overrides, peers), and finally the remaining commands like run, store, and publish. So even after v12 ships, parts of pnpm will still run through the TypeScript implementation for a while. The lockfile is what lets the two halves cooperate.

What changes for you?

The honest answer is: not much, on purpose. Same CLI, same lockfile, same node_modules. The whole point of a 1:1 port is that your muscle memory and your CI scripts keep working.

The breaking changes slated for v12 are deliberately small: removing some old migration code for git-hosted tarballs, rejecting unscoped auth settings, and deprecating the $ syntax in overrides in favor of catalogs. If none of those describe your setup, a v12 upgrade should be close to a no-op.

Will the upgrade break your CI?

Probably not, but the one thing worth checking is feature parity during the transition. Because the Rust engine is being ported incrementally, some newer or niche capabilities may lag the TypeScript version at any given moment. If your install depends on something specific (a particular linker mode, lockfile-verification settings, or pnpm deploy), test the bump in an isolated git worktree before you flip it on in CI, the same sandbox I used while exploring SolidJS 2.0's async data model. This is moving quickly, so the right list of gaps is whatever the roadmap says on the day you upgrade.

The bigger picture

pnpm is not doing this alone. The last few years have quietly rebuilt the JavaScript toolchain's foundations in native languages: Biome for linting and formatting, Oxc for parsing and transforming, Rolldown and Turbopack for bundling. TypeScript's compiler went to Go. Deno is Rust from the ground up. (Bun is the odd one out: it's written in Zig, not Rust, so don't lump it into the "everything's Rust now" story.)

There's a competitive angle underneath it, too. Bun and Deno both ship as single fast binaries with package management built in, and a chunk of their appeal is exactly that no-runtime-bootstrap speed. A native-binary pnpm with no Node launcher closes that gap while keeping the thing that makes pnpm pnpm: the strict, content-addressable store and the non-flat node_modules that catches phantom dependencies.

Should you upgrade to pnpm v12 yet?

If you're waiting for a stable release: keep waiting, but not for long. As of mid-July 2026, v12 is in alpha. The latest tag is v12.0.0-alpha.12, and the stable line is still v11.13.0. This is not something to point production CI at today.

But if you want to feel the difference now, you already can. Run pnpm self-update next-12 on a branch, or opt into the pacquet backend on your current v11, and watch a warm install that used to take a beat return before you've finished reading the command. The type of release this is, where the language underneath gets an order of magnitude faster while everything on top stays exactly where you left it, is a rare and good one. pnpm was already the fast package manager. It's about to stop being one you ever wait on.

Unlocking the Power of Metal Recycling: MAXIM Machinery's Metal Crusher Machine

Thu, 16 Jul 2026 09:33:11 +0000

When it comes to the industrial machinery industry, the need for effective and efficient recycling solutions has never been more critical. As industries worldwide pivot towards sustainable practices, the demand for robust machinery that can handle the recycling of metals has surged. Enter MAXIM Machinery's Metal Crusher Machine, a revolutionary piece of equipment designed to transform the way we approach metal recycling.

The industrial landscape is shifting, with a heightened focus on sustainability and environmental responsibility. This shift has placed metal recycling at the forefront of industrial processes, necessitating machinery that can deliver on both efficiency and effectiveness. MAXIM Machinery's Metal Crusher Machine stands out as a beacon of innovation and reliability in this context, offering a range of features that cater to the diverse needs of the industry.

At the core of the Metal Crusher Machine's appeal is its versatility. Capable of crushing a wide array of metals—from steel to aluminum—the machine is an indispensable asset for any facility engaged in metal recycling. Its design allows it to process varying metal sizes, turning scrap into valuable resources with ease. This adaptability is crucial in an industry where different materials often require specific handling and processing methods.

The Metal Crusher Machine is engineered to meet the high demands of industrial metal shredding. One of the standout features of this machine is its ability to handle large volumes of scrap metal swiftly and effectively. This capability is particularly beneficial for industries dealing with substantial quantities of metal waste, ensuring that operations remain smooth and uninterrupted. The machine's robust construction means it can operate under the most challenging conditions, delivering consistent performance over time.

In today's market, the need for machines that can process materials like steel and aluminum efficiently is more pressing than ever. Aluminum, in particular, is a metal that requires careful handling due to its lightweight nature and widespread use in industries ranging from automotive to packaging. The Metal Crusher Machine excels in aluminum can shredding, reducing bulky items into manageable sizes for recycling. This feature not only streamlines operations but also significantly reduces the environmental impact of metal waste.

The importance of recycling extends beyond traditional metals, as industries now face the challenge of recycling components like lithium batteries. These batteries are integral to modern technology, yet they pose significant environmental risks if not disposed of correctly. A lithium battery recycling plant, equipped with MAXIM's Metal Crusher Machine, can effectively process these components, separating valuable materials for reuse while ensuring hazardous elements are safely contained. This capability highlights the machine's role in supporting the circular economy, where products are continually reused and recycled, minimizing waste.

Scrap metal recycling equipment, such as the Metal Crusher Machine, is essential for industries looking to enhance their sustainability credentials. By investing in such technology, businesses can ensure that their operations not only comply with environmental regulations but also contribute positively to the planet. The machine's efficiency in processing scrap metal translates into reduced costs and increased profitability, making it a smart investment for any forward-thinking organization.

Electric vehicle (EV) battery recycling is another area where the Metal Crusher Machine proves invaluable. As the world shifts towards electric vehicles, the need for effective recycling solutions for EV batteries becomes critical. These batteries contain materials that are both valuable and potentially harmful if not handled correctly. MAXIM Machinery's solution allows for the efficient breakdown and recycling of these components, ensuring that valuable materials are recovered and reused, while minimizing environmental harm.

The demand for industrial metal shredder manufacturers is on the rise, driven by the need for machinery that can meet the growing recycling goals of various industries. MAXIM Machinery has established itself as a leader in this field, offering products that combine cutting-edge technology with practical applications. Their Metal Crusher Machine is a testament to their commitment to providing solutions that align with industry needs and environmental goals.

In real-world applications, the Metal Crusher Machine has demonstrated its capacity to transform metal recycling operations. Consider a manufacturing plant producing significant amounts of metal waste daily. With the integration of MAXIM's Metal Crusher Machine, this plant can efficiently process its waste, turning what was once a disposal issue into a source of revenue. The machine's ability to quickly and effectively process various metals means that the plant can operate more sustainably, reducing its environmental footprint while boosting its bottom line.

The benefits of using MAXIM Machinery's Metal Crusher Machine extend beyond its technical capabilities. The machine's design reflects a deep understanding of the practicalities involved in metal recycling. Its user-friendly interface and low maintenance requirements make it an attractive option for facilities looking to minimize downtime and operational costs. Furthermore, the machine's durability ensures that it remains a vital part of recycling operations for years to come, providing a reliable solution that evolves with industry demands.

In conclusion, MAXIM Machinery's Metal Crusher Machine represents a significant advancement in the field of metal recycling. Its ability to handle various metals and its integration into recycling processes make it an essential tool for industries aiming to enhance their sustainable practices. As the industrial landscape continues to evolve, investing in robust and efficient recycling equipment like the Metal Crusher Machine is not just a choice—it's a necessity. With MAXIM Machinery leading the charge, the future of metal recycling looks promising, paving the way for a more sustainable and environmentally responsible industrial sector.

pnpm v12 Is Being Rewritten in Rust

Wed, 15 Jul 2026 15:13:28 +0000

What pnpm v12 actually is

Why did pnpm choose Rust?

The motivation isn't abstract. Two things were capping how fast a Node-based pnpm could go.

How much faster is pnpm v12?

pnpm publishes a benchmark suite comparing v11 against the Rust engine directly. As of this writing (July 2026), here's what it shows:

Scenario	pnpm v11	pnpm (Rust)
Everything warm (repeat install)	381 ms	12 ms
node_modules present, no cache	460 ms	40 ms
Clean install (nothing cached)	6.5 s	2.2 s

It's already rolling out, incrementally

Here's the part that's easy to get wrong: pnpm v12 isn't a big-bang release. The Rust engine has been landing piece by piece through the v11 line, opt-in first.

pnpm 11.2 let you opt into pacquet as the install backend for materialization only. Rust did the fetch and link, while pnpm still resolved dependencies and wrote the lockfile.
pnpm 11.7 went further: a standard install could be delegated to pacquet end-to-end, with resolution, lockfile write, and linking all in a single Rust pass.
pnpm 11.10 added pnpm self-update next-12, so you can pull the v12 line and try the Rust binary directly.

What changes for you?

The honest answer is: not much, on purpose. Same CLI, same lockfile, same node_modules. The whole point of a 1:1 port is that your muscle memory and your CI scripts keep working.

Will the upgrade break your CI?

Probably not, but the one thing worth checking is feature parity during the transition. Because the Rust engine is being ported incrementally, some newer or niche capabilities may lag the TypeScript version at any given moment. If your install depends on something specific (a particular linker mode, lockfile-verification settings, or pnpm deploy), test the bump in an isolated git worktree before you flip it on in CI. This is moving quickly, so the right list of gaps is whatever the roadmap says on the day you upgrade.

The bigger picture

Should you upgrade to pnpm v12 yet?

FNPM Just Became Invisible (In the Good Way): Transparent Alias, Better Tests & a Smoother Experience

Tue, 07 Jul 2026 17:12:02 +0000

There are moments when you stop, look at your terminal, and think:
“Why does something as simple as managing packages in JavaScript has to be… so dramatic?”

If you've been around the Node.js ecosystem for more than 5 minutes, you know exactly what I mean.

Well, FNPM just got its biggest update ever, and no, it’s not “just another technical improvement.” This update is about giving you the experience we all secretly wanted: using your favorite package manager while something way smarter runs behind the scenes.

Let me explain.

Before: npm, pnpm, yarn… drama.

Now: use whatever you want. FNPM does the hard work.

The biggest change (and my personal favorite) is the new transparent alias system.
This means you can now type:

pnpm install

…and behind the scenes, FNPM is actually doing the job.

Why is this so massive?

Because for the first time, an alternative package manager can sneak into (yeah, sounds weird but it’s beautiful) your workflow without asking you to change anything.

If your team uses pnpm, perfect.
If you prefer FNPM’s model and how it handles things, also perfect.
If your environment is mixed… well, guess what? It works too.

FNPM now behaves as a real drop-in replacement, like for real real.

How does this alias magic works?

FNPM now includes a smart hook system that intercepts the traditional package manager commands and transforms them into FNPM operations.

In short:

It detects what you want (install, add, update, run, etc.).
It maps it to FNPM’s internal engine.
It keeps the CLI experience you already know.

All without breaking scripts, without changing your mental model, and without asking you to learn new commands (because, honestly, nobody wants that anymore ).

Better test coverage (and now actually cross-platform)

Another big step forward is the improved test coverage, which now behaves consistently across systems:

Linux
macOS
Windows (yes, Windows got some love this time )

Error messages, symlink behavior, path formatting… all those cross-platform headaches?
FNPM now handles them.

This means FNPM isn’t just “usable” on Windows —
it’s reliable, predictable, and CI-friendly across all platforms.

️ Other cool improvements worth mentioning

New fnpm source command for shell integration
GitHub Workflows + issue templates for better contributing
Updated dependencies (thanks Dependabot )
Cleaner project structure with better dev tooling
A more complete test suite for future expansions

FNPM keeps growing with a focus on simplicity + power, not complexity.

A real example: turning `pnpm` into FNPM (without anyone noticing)

Here’s a real example of how easy this is.
I swear the first time you run it, you’ll feel that “oh wow” moment .

(base) ➜  awesome-project fnpm setup pnpm  
Selected package manager: pnpm  

 FNPM hooks created successfully!

Setup Instructions:  
Unix/Linux/macOS:  
  1. Source the setup script:  
     source .fnpm/setup.sh  
  2. Or add to your shell profile (~/.bashrc, ~/.zshrc):  
     echo 'eval "$(fnpm source)"' >> ~/.zshrc  

Usage:  
  • Now you can use pnpm directly:  
    pnpm add some-package  
    pnpm install  
    pnpm run my-script  
  • Commands will be automatically redirected to fnpm  

Note:  
  • Hooks only work in directories with .fnpm configuration  
  • Use full path to bypass: $(which pnpm) command

Reload your shell:

(base) ➜  awesome-project echo 'eval "$(fnpm source)"' >> ~/.zshrc

Now try any pnpm command:

(base) ➜  awesome-project pnpm --help

You’ll see something like:

 This pnpm command is intercepted by FNPM  
Available commands:  
  install, add, remove, run, list, update, cache, clean  

Use 'fnpm --help' for more information

Boom.
From that moment on, you keep using pnpm like always…
but FNPM is the one doing the heavy lifting behind the curtain.

No workflow changes.
No broken scripts.
No new habits.
Just smooth developer life.

✨ In short: FNPM is no longer “used”… it’s felt

With this transparent alias system, FNPM stops being a command you remember to type, and becomes:

fast
consistent
cross-platform
and completely invisible when you want it to be

If it was useful before, now it feels… kinda magical, honestly.

PS: we used AI to help us write this article — because great tools deserve great storytelling.

Documentation here

FNPM: The Package Manager Drama Ends Here

Thu, 02 Jul 2026 17:18:53 +0000

FNPM: The Package Manager Drama Ends Here

Hey devs! Tired of the endless debates over package managers? Ever wished for a solution that brings peace to the package manager wars? Meet FNPM – because life's too short for package manager drama!

I work at Konstruct.io we have free Fridays, and this idea was stuck in my mind for a while. I finally took advantage of AI to bring it to life! So please give us ⭐️ on konstructio/kubefirst, a great and amazing way to setup your kubernetes infrastructure following gitOps Philosophy.

What is FNPM?

FNPM stands for Fuck NPM. It's a tool designed to end the eternal conflicts between npm, yarn, pnpm, and others by supporting them all seamlessly. Think of it as the peace treaty in the package manager soap opera. ️

Why Should You Care?

Universal Support – FNPM supports multiple package managers, including npm, yarn, pnpm, bun, and deno. Use your favorite; FNPM has your back!
One Lock File to Rule Them All – No more lock file conflicts! FNPM ensures consistent dependencies across your team, regardless of individual preferences.
Happy CI/CD – Standardized workflows and consistent builds mean no more "works on my machine" excuses.
Team Harmony – Let developers be developers. Stop the package manager wars and focus on building awesome stuff!

How to Get Started

Getting FNPM up and running is super simple:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/ideascoldigital/fnpm/refs/heads/main/install.sh)"

Then, just use it like you would with your preferred package manager:

fnpm setup

And boom! Your dependencies are managed without the drama.

Basic Commands

Here's a quick reference of basic commands supported across all package managers:

Action	FNPM Command	What it does
Install dependencies	`fnpm install`	Installs all project dependencies
Add a package	`fnpm add <package>`	Adds a new package to your project
Remove a package	`fnpm remove <package>`	Removes a package from your project
Update packages	`fnpm update`	Updates packages to their latest versions
Run a script	`fnpm run <script>`	Runs a script defined in package.json
Clean cache	`fnpm clean`	Cleans the package manager cache
List packages	`fnpm list`	Lists installed packages

Final Thoughts

FNPM is here to bring harmony to your development workflow. It's easy to install, easy to use, and might just save you from the package manager drama. ️

Check out the official docs here FNPM Documentation

Have you tried FNPM? Let me know what you think! ✨

I Doubled My Servers and Wrecked My Availability

Sat, 27 Jun 2026 09:58:47 +0000

“Reliability is the most important feature of any system.” — Bill Gates

Why does your bank never show the wrong balance, but your favorite app sometimes takes a few seconds to update? Why do some systems feel fast but crash under load, while others feel slower but never fail?

Performance answers: “How fast is the system right now?”
Scalability answers: “What happens when demand increases?”

These questions point to a fundamental tension in system design: performance vs scalability.

A system can improve performance metrics like throughput and latency, yet fail to scale under load. This article demonstrates exactly that.

In this article, we’ll explore this trade-off through a simple analogy, and then connect it to real-world systems using measurable metrics like latency, throughput, and error rate.

The Burger Shop Analogy

Imagine a small burger shop run by a single worker, let’s call them A.

Let’s assume the following things:

Burgers are prepared automatically by machines, so A only handles taking and serving orders.
Finished burgers are placed in a shared box, from which A picks them up and delivers them to customers.

We intentionally ignore preparation time and assume burgers are instantly ready. This allows us to focus purely on how requests are handled, rather than how they are produced.

The Single Worker System

In this setup, a single worker handles all incoming requests.

When you enter the shop, you join a single queue and wait your turn before placing an order. Every request is processed sequentially (one at a time).

A few key observations:

Single queue, sequential processing

With only one line, every customer must wait for those ahead of them. This results in higher waiting time, which translates to high latency and low throughput.
No contention

Since only one request is handled at a time, there is no competition for shared resources. The system operates without conflicts or synchronization issues.
Efficient resource usage

Resources are used strictly on demand. Each burger is prepared only when needed, with no excess consumption or overhead.

What does this imply?

High latency → Each request experiences delay due to queuing
Low throughput → Only one request is processed at any given time
No contention → No conflicts between concurrent operations

Despite its limitations, this system is remarkably stable.

There are no race conditions, no lost work, and no unexpected failures. Every request is handled in a predictable and controlled manner.

Slow, but predictable.

Scaling the System

So far, the limitation has been clear: a single worker handling all requests.

To improve this, we introduce more machines. Instead of relying on a single machine, worker A can now distribute incoming orders across multiple machines.

Let’s examine what changes:

Parallel processing

With multiple machines available, several orders can be handled simultaneously. This increases the number of requests processed per unit time, leading to higher throughput and reduced latency.
Decision overhead (routing)

Worker A must now decide which machine should handle each request. This introduces a small coordination overhead — similar to how a load balancer (e.g., Nginx) routes requests in real systems.

For simplicity, we will ignore this overhead in our analysis.

What happens to the metrics?

Multiple orders can be processed simultaneously.
Waiting time decreases
Throughput increases

This approach is known as horizontal scaling, increasing system capacity by adding more resources rather than making a single component more powerful.

The Hidden Problem

So far, scaling seems like a clear improvement.

But consider the shared burger box , the place where all machines deposit finished burgers.

What happens if multiple machines try to place burgers into the box at the same time?

This introduces two critical failure scenarios:

Lost Work

One burger makes it into the box, but another falls to the ground.

That burger is wasted, and the customer never receives their order.

In system terms, this is a failed request.
System Blockage

Both burgers get stuck at the entrance of the box, blocking it completely.

Now no further burgers can be placed.

In system terms, this resembles contention or resource locking, where the system slows down or stops entirely.

What Changed ?

By introducing multiple machines, we improved performance:

Higher throughput, lower latency

Multiple burgers can now be prepared simultaneously. This increases throughput and reduces waiting time for customers, improving overall performance.
Increased coordination and state management

With two machines handling requests, worker A must decide where each order goes. Both machines can process requests, so their state must be tracked and logs must be maintained. This introduces coordination overhead.
Inefficiency due to lost work

Some burgers may fall to the ground before reaching the box. These orders are wasted, and the customer never receives their burger. In system terms, this represents failed or lost requests.
Unpredictable latency

Consider a customer waiting for an order that never arrives. If they leave after 30 seconds, that request now has very high latency. Even if other customers are served quickly, overall latency becomes inconsistent and harder to predict.This is the core trade-off.

In system terms, we have introduced:

Contention for shared resources
Increased system complexity
Possibility of failure

The Core Trade-off

This is the fundamental trade-off in system design.

Some customers are served faster, but others may never get their order.

Performance

Performance refers to how effectively a system executes tasks within a given time frame. It is typically evaluated in terms of speed, responsiveness, and how efficiently system resources are utilized under load.

To understand and measure performance, we rely on a few key metrics:

Latency

Latency is the time it takes for a system to respond to a request.

In simpler terms, it measures how long a user has to wait after initiating an action.

Throughput

Throughput represents the number of requests or transactions a system can handle per unit of time.

It reflects the system’s capacity to process workload at scale.

Resource Utilization

Resource utilization measures how much of the system’s resources, such as CPU, memory, and network bandwidth are consumed while handling requests.

Efficient systems aim to maximize output while minimizing unnecessary resource consumption.

Efficiency

Efficiency describes how well a system converts resources into useful work.

A highly efficient system achieves high throughput and low latency while using minimal resources.

Scalability

Scalability refers to a system’s ability to handle increasing load by proportionally utilizing additional resources, while maintaining acceptable performance.

In other words, a scalable system does not degrade significantly as the number of users or requests grows, it adapts by efficiently distributing the workload.

To evaluate scalability, we observe how key performance metrics behave under increasing load, including:

Latency under load
Throughput under load
Resource utilization
Error rates (failures, timeouts)
Number of concurrent users supported

A system is considered scalable if these metrics remain stable or degrade gracefully as demand increases.

Availability

Availability is the measure of a system’s ability to respond to requests within an acceptable time frame.

A system is considered available if it is operational and capable of returning responses, even under load. High availability systems are designed to minimize downtime and ensure that users can reliably access the service.

“A system can be highly performant but not scalable. for example, a single powerful server may respond quickly, but fail when user load increases.”

The Monolith vs. The Load-Balanced Fleet (A vs 2A’s)

To move beyond theory, we need to observe how a system behaves when pushed to its absolute limits. To test the relationship between compute scaling, database bottlenecks, and overall availability, an A/B load test was constructed.

The code for the experiments can be found in attached github repository: {% github https://github.com/c0d3l0v3r-HeHe/performance-vs-scalability-experiment %}

Assumptions

Before running the load test, the architecture was built on three core hypotheses:

Linear Scalability: Doubling the compute resources (adding a second application server) will linearly increase the system’s maximum throughput.
Testing Default Database Limits: We hypothesized that the default connection pool of a single MongoDB instance would be sufficient to handle the concurrent load of doubled compute resources.
The Load Balancer Penalty is Negligible: The network hop introduced by routing traffic through a reverse proxy will be outweighed by the massive gains in request processing speed.

Setup Methodology

The experiment was conducted locally using Docker to containerize and isolate the environments. The testing tool, k6, was configured to inject a stepped load simulating aggressive traffic spikes: starting at 100 concurrent virtual users and ramping up to 2,000 users over an 80-second window.

The workload itself was a mixed Read/Write operation. Every request forced the application to write a new document to the database and immediately read a document back, simulating a standard transactional endpoint.

We tested three distinct architectures:

Architecture A (The Baseline): A single Node.js/Express application connected directly to a standalone MongoDB instance. This represents a traditional, unscaled monolith.
Architecture 2A (The Distributed Compute): Two identical Node.js/Express applications sitting behind an Nginx load balancer. Nginx utilized a round-robin strategy to distribute incoming k6 traffic across both nodes. Crucially, both application nodes were wired to the exact same single MongoDB instance.

const express = require("express");
const mongoose = require("mongoose");
const os = require("os");

const app = express();


mongoose.connect("mongodb://mongo:27017/testdb", {
  maxPoolSize: 50,
});

const TestSchema = new mongoose.Schema({
  value: Number,
});

const Test = mongoose.model("Test", TestSchema);

app.get("/", async (req, res) => {
  const start = Date.now();

  try {
    await Test.create({ value: Math.random() });
    await Test.findOne();

    const latency = Date.now() - start;

    res.json({
      latency,
      server: os.hostname(), 
    });
  } catch (err) {
    res.status(500).json({ error: "DB error" });
  }
});

app.listen(3000, () => console.log(`Server running on ${os.hostname()}`));

Results

The k6 load test yielded drastically different saturation points for the two architectures.

These reports are present as the .html files in the github respository.

Architecture A (Single Node Baseline):

The latency trade-off. With a single Node.js event loop acting as a natural throttle, the p95 latency sat at ~796ms. The system was relatively slow under heavy load, but perfectly stable.

Baseline throughput. The single-node architecture safely processed a total of 176,584 requests, hitting a hard bottleneck at roughly 2,193 requests per second.

100% Availability. Because the compute layer could not overwhelm the database connection pool, every single read/write transaction was successfully completed without a single dropped request.

Total Requests Processed: 176,584
Throughput: 2,193 requests/second
Latency (p95): ~796 ms
Failure Rate: 0.00% (100% Success)

Architecture 2A (Distributed Compute):

The smoking gun. While the p95 latency for successful requests actually dropped to ~492ms, the maximum latency spiked to an unusable 36 seconds. The explicit HTTP 500 errors confirm the single MongoDB instance was pushed into connection exhaustion.

The illusion of scale. Adding a load balancer and a second application node successfully pushed our throughput to over 3,600 requests per second, processing 350,261 total requests.

The breaking point. Scaling the compute layer while leaving the data layer untouched resulted in a massive 26.5% failure rate, actively degrading the system’s availability.

Total Requests Processed: 350,261
Throughput: 3,638 requests/second
Latency (p95): ~492 ms (for successful requests)
Failure Rate: 26.57% (93,069 failed requests, including explicit HTTP 500 database errors and massive timeouts spiking up to 36 seconds).

Discussion

The experimental data explicitly demonstrates the systemic risks of isolated horizontal scaling. While adding compute resources improved specific performance metrics, it fundamentally compromised system reliability, proving that scaling a single tier does not equate to scaling the system as a whole.

1. The Shifting Bottleneck and Resource Exhaustion The baseline load test (Architecture A) established a clear system ceiling: a single Node.js instance capped throughput at 2,193 requests per second with a p95 latency of ~796ms. Crucially, the failure rate was 0.00%. In this configuration, the compute layer acted as a strict upstream bottleneck. Because a single server could not process requests fast enough to overwhelm the database’s connection pool (configured to a maximum of 50 connections), the database was naturally shielded from resource exhaustion. The system was relatively slow under peak load, but highly stable.

By introducing a load balancer and a second application node (Architecture 2A), the compute-layer bottleneck was removed. The system accepted more concurrent traffic, driving throughput up by roughly 65% (to 3,638 req/sec). However, because both nodes funneled this doubled traffic into the single, unscaled MongoDB instance, the bottleneck immediately shifted to the data layer. The sudden influx of concurrent requests exhausted the database connection pool, leading to connection lockups and explicit HTTP 500 errors.

2. The False Positive of “Improved” Performance A superficial reading of Architecture 2A’s metrics might suggest a performance win: the p95 latency for successful requests actually dropped by ~38% (from 796ms down to 492ms), and total processed requests nearly doubled.

However, a rigorous analysis reveals this “scale” is an illusion. The data shows a catastrophic failure rate of 26.57% (93,069 failed requests). The reduction in p95 latency for successful requests is effectively a survivorship bias metric; it only accounts for the requests that managed to secure a database connection. Meanwhile, the maximum latency for blocked requests spiked to an unusable 36 seconds. The experiment proves that forcing high throughput at the compute layer without matching capacity at the data layer destroys system availability.

3. The Inadequacy of Isolated Scaling The results strictly invalidate the hypothesis that doubling compute resources linearly increases maximum throughput safely. Instead, it proves that architectural scaling must be holistic.

When Architecture 2A pushed the database beyond its concurrency limits, the system did not degrade gracefully — it fractured, dropping more than a quarter of all user requests.

Conclusion

The data validates a core principle of distributed systems:

throughput and availability are often inversely correlated when shared resources are unscaled. Architecture A was unscalable but highly available (100% success). Architecture 2A was highly performant for a subset of users but critically unavailable for the rest (26.57% failure).

To successfully scale this system beyond the 2,193 req/sec ceiling of Architecture A without incurring the 26% failure penalty of Architecture 2A, the architecture must evolve to protect the database. Based strictly on these failure modes, future iterations must either synchronously scale the data layer (e.g., database sharding/replicas to handle the higher connection volume) or implement an asynchronous buffer (e.g., a message queue) to decouple the request ingestion rate from the database write rate.

Acknowledgements & References

The experiments and conclusions drawn in this article were directly informed by studying foundational distributed systems literature. Translating these concepts from theory into a live, containerized k6 experiment was made possible by the following resources:

Performance, Scalability & Throughput

MIT 6.004: Performance Measures (YouTube): Provided the classic “laundry analogy” illustrating why pipelining improves overall throughput but not individual request latency.
Performance vs. Scalability (Prof. Beekums): Clarified the boundaries between making a single server faster versus designing an architecture to handle increased load.
A Word on Scalability (Werner Vogels): Foundational principles from Amazon’s CTO on building systems that scale gracefully.
Latency vs Throughput (CS.fyi) & Understanding Latency vs Throughput (Cadence): Core definitions and metrics for measuring system speed and capacity.

The CAP Theorem & Consistency Patterns

CAP Theorem Revisited (Robert Greiner) & A Plain English Introduction to CAP (ksat.me): Explained the core mathematical trade-off: why our partitioned, load-balanced system had to sacrifice availability when tied to a single database.
Consistency Patterns (CS.fyi): Detailed breakdowns of Weak, Eventual, and Strong Consistency, which directly inspired the Redis asynchronous queue experiment.

High Availability, Failover & Database Replication

Availability in Distributed Systems (GKCS — YouTube): A practical war story detailing a real-world database migration, downtime management, and understanding the “9s of availability.”
Scalability, Availability & Stability Patterns (SlideShare): A comprehensive visual overview of architectural patterns.
High Availability System Design Basics (DesignGurus) & System Design Availability Patterns (Dev.to): Practical guides on identifying and removing single points of failure.
Active-Passive vs Active-Active Failover (Serverion): The blueprint used for configuring our Nginx load balancer to distribute compute traffic.
Introduction to Database Replication (EnjoyAlgorithms): A deep dive into separating read/write workloads and scaling the data tier.

How I Stopped a Race Condition from Creating Free Money in My Wallet System

Sat, 13 Jun 2026 14:56:16 +0000

The Bug That Doesn't Show Up in Testing.
When I set out to build a production-grade wallet system, I expected the hard parts to be the database schema, the transaction logic, or the authentication layer.

What I didn't expect was that the most dangerous bug in the system would have nothing to do with wrong code - it would be about timing.

This is the story of a race condition that lets users spend money they don't have, why it's invisible until it hits production, and how Redis turned it into a non-issue.

What a Wallet System Actually Does
Before we get to the bug, a quick picture of the system. A wallet holds a balance. Users can:

Credit - add funds
Debit - spend funds
Transfer - move funds to another user
Reverse - undo a transaction
Every operation reads the current balance, validates it, and writes a new transaction record. Simple enough. The problem hides in that gap between reading and writing.

The Race Condition
Here's the scenario. A user has ₦5,000 in their wallet. They're on a slow connection, so they tap "Pay" twice. Two HTTP requests hit the server within milliseconds of each other. Both are trying to debit ₦3,000.

Without any concurrency protection, this is what happens:

Request A → reads balance: ₦5,000
Request B → reads balance: ₦5,000

Request A → ₦5,000 - ₦3,000 = ₦2,000 ✓ writes transaction
Request B → ₦5,000 - ₦3,000 = ₦2,000 ✓ writes transaction
Both requests read ₦5,000. Both pass the balance check. Both write a successful debit. The user just spent ₦6,000 from a ₦5,000 wallet - and the system called both transactions successful.

This isn't a theoretical edge case. Under any real concurrent load, this happens. And the worst part: it never shows up in testing, because your test runner doesn't fire two real HTTP requests at the same millisecond.

Why a Database Transaction Doesn't Save You
The instinct here is to reach for a database transaction. Wrap the whole thing in BEGIN / COMMIT and let Postgres sort it out.

The problem is that the race happens before the database transaction begins. Both requests read the balance from the database (or worse, from a cache), both pass the validation check in application code, and then both open their database transactions. The damage is already done.

You need a lock at the application layer, before the read even happens.

The Fix: Distributed Locking with Redis
The solution is to ensure that no two operations on the same wallet can run at the same time. Before any balance operation begins, it must claim exclusive access. If another operation already holds the claim, it retries until the lock is free, or gives up after a timeout.

This is a distributed lock, and Redis is the right tool for it. Here's the implementation:

// src/utils/lock.utils.ts
import { randomUUID } from "crypto";
import redisClient from "../services/redis.service";

const LOCK_TTL_MS = 10_000;
const LOCK_RETRY_COUNT = 50;
const LOCK_RETRY_DELAY_MS = 100;

const RELEASE_SCRIPT = `
  if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("DEL", KEYS[1])
  else
    return 0
  end
`;

const acquireLock = async (key: string): Promise<string | null> => {
  const token = randomUUID();
  const result = await redisClient.set(key, token, "PX", LOCK_TTL_MS, "NX");
  return result === "OK" ? token : null;
};

export const withLock = async <T>(userId: string, fn: () => Promise<T>): Promise<T> => {
  const key = `wallet:lock:${userId}`;
  let token: string | null = null;

  for (let attempt = 0; attempt <= LOCK_RETRY_COUNT; attempt++) {
    token = await acquireLock(key);
    if (token) break;
    await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_DELAY_MS));
  }

  if (!token) throw new Error("Could not acquire lock — resource is busy");

  try {
    return await fn();
  } finally {
    await redisClient.eval(RELEASE_SCRIPT, 1, key, token);
  }
};

Let's break down each decision.

SET key token PX 10000 NX
This is a single atomic Redis command that does three things at once:

NX — only set the key if it does not already exist
PX 10000 — expire the key after 10 seconds
The value is a unique token (UUID), not just 1 or "locked"
The atomicity of SET NX is the core guarantee. There is no gap between "check if key exists" and "set the key" — it's one operation. Two requests racing to acquire the same lock cannot both succeed.

Retry logic
Rather than immediately rejecting a request when the lock is held, the implementation retries up to 50 times with a 100ms delay between each attempt - giving it up to 5 seconds to acquire the lock before giving up. This handles brief overlaps gracefully without dropping legitimate requests.

The TTL
The 10-second expiry is a safety net. If the application crashes mid-operation and fails to release the lock, it will expire automatically. Without this, one crash would lock that user's wallet forever.

The Lua script on release
This is where most implementations get it wrong. A naive release looks like this:

await redisClient.del(key); // ❌ dangerous
Here's why that's broken: imagine a slow operation acquires the lock, but takes longer than 10 seconds. The TTL expires. A second operation acquires the lock. Now the first operation finishes and calls del - it just released the second operation's lock, not its own. The guarantee is broken.

The Lua script fixes this by checking the token before deleting:

if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
else
  return 0
end

Because Lua scripts execute atomically in Redis, the check-and-delete is a single uninterruptible operation. You can only release a lock you actually own.

The withLock wrapper
Rather than exposing raw acquireLock / releaseLock functions and trusting every caller to use them correctly, the implementation wraps the entire pattern in a higher-order function:

// src/services/wallet.service.ts
export const debitWallet = async (userId: string, amount: number): Promise<Transaction> => {
  return await withLock(userId, async () => {
    const wallet = await prisma.wallet.findUnique({ where: { userId } });

    if (!wallet || wallet.balance.lessThan(amount)) {
      throw new Error("Insufficient balance");
    }

    // create transaction, update balance...
    return transaction;
  });
};

The finally block inside withLock ensures the lock is always released, even if the operation throws. A lock that leaks on error is as bad as no lock at all.

Preventing Deadlocks on Transfers
A transfer touches two wallets - sender and receiver. If two transfers run simultaneously in opposite directions (A→B and B→A), they can deadlock: each holds one lock and waits for the other indefinitely.

The fix is to always acquire locks in the same order, regardless of who is sending to whom. We sort the user IDs before acquiring:

export const withMultiLock = async <T>(userIds: string[], fn: () => Promise<T>): Promise<T> => {
  const sortedIds = [...new Set(userIds)].sort();
  const keys = sortedIds.map((id) => `wallet:lock:${id}`);
  const tokens: string[] = [];

  try {
    for (const key of keys) {
      let token: string | null = null;
      for (let attempt = 0; attempt <= LOCK_RETRY_COUNT; attempt++) {
        token = await acquireLock(key);
        if (token) break;
        await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_DELAY_MS));
      }
      if (!token) throw new Error("Could not acquire lock — resource is busy");
      tokens.push(token);
    }
    return await fn();
  } finally {
    await Promise.all(keys.map((key, i) => releaseLock(key, tokens[i])));
  }
};

// In transfer service:
const result = await withMultiLock([senderId, receiver.id], async () => {
  // both wallets locked — safe to proceed
});

If A→B and B→A both arrive at the same time, both will sort to [A, B] and try to acquire A's lock first. One wins, the other waits. No deadlock.

The Full System
The distributed lock is one layer in a larger stack. The complete wallet engine includes:

Idempotency - Idempotency-Key header on credit and debit prevents duplicate transactions on network retries, backed by a 24-hour Redis TTL
Webhook delivery with retry - BullMQ queue, 4 attempts, exponential backoff (2s base), per-attempt audit trail
Balance caching - 60-second Redis TTL, invalidated on every write
Token blacklisting - SHA-256 hashed JWTs stored in Redis with matching expiry, making logout actually secure
Cursor-based pagination - transaction history that scales
Full test suite - 103 tests, all mocked, no database required to run.
Built with TypeScript, Express, Prisma, PostgreSQL, and Redis (ioredis). Deployed on Render with a GitHub Actions CI/CD pipeline.

Why This Matters
Race conditions in financial systems are not exotic. They're the default outcome when you don't explicitly prevent them. Most wallet tutorials never mention it. Most wallet implementations in production have it.

The fix isn't complicated - but you have to know to look for it.

If you're building anything that touches money, check your balance reads. If there's no lock between the read and the write, the race condition is already there.

Open for Collaboration
This project is open source and actively welcoming contributors. Whether you want to add a feature, improve test coverage, or spot something that could be done better - PRs are open.

Check out the CONTRIBUTING.md for how to get set up and where to send your changes.

GitHub: https://github.com/tochi27/Wallet_System.git

Live API + Swagger: https://wallet-system-api-16cv.onrender.com/api-docs

How I Found a Fake Job Assessment Repo Hiding Malware Inside SVG Files

Thu, 11 Jun 2026 07:02:16 +0000

Like a lot of developers in this market, I’ve been taking freelance assessments and Discord job leads more seriously than I normally would.

One of those assessments turned into a malware investigation.

One day, I saw a post in a Discord server looking for a fullstack dev. I pitched. The reply looked routine at first: they DM’d me a requirements PDF for an assessment. I did not trust it, so I asked them to paste the requirements in the chat instead. They sent screenshots of the PDF, and it looked like a real assessment. Clean structure, clear expectations, nothing immediately screaming scam.

Then they invited me to a GitHub repo called E-commerce-template-12d46f3e. My first thought was that the name looked autogenerated, like they were appending random numbers for each assessment. That is when I started treating it like a security review, not a coding exercise.

What I checked first

The first thing I looked at was package.json. I was expecting the usual red flags: weird postinstall hooks, obfuscated scripts, or packages I had never heard of. There was one outdated dependency, @zeit/next-css, but nothing in package.json looked obviously malicious.

That is what made the repo interesting. The dependency list looked boring; the problem was in the application flow.

The suspicious startup path

Next, I looked at the startup scripts.

The dev script ran server.js, and that file did something that immediately raised my guard: it called startLoggingErrors() during server startup.

At first glance, that looked like a harmless logging helper. But when I opened the related files, I found this chain:

server.js starts the server.
lib/serverStartup.js calls eval(...).
lib/startupLogs.js reconstructs a hidden payload from files in public/flags/.

That eval() was the key. A server startup path that reconstructs data from assets and then evaluates them is a hard stop.

The hidden payload in the SVGs

The weirdest part was the assets.

The repository had a bunch of country flag SVG files under public/flags/. Those looked normal until I checked the HTML comments inside them. Each SVG had a comment that looked like a fragment of base64-encoded text.

The loader in lib/startupLogs.js:

Walks through public/flags/
Reads each .svg
Extracts the text inside 
Sorts and joins the fragments
Base64-decodes the result
Returns the decoded JavaScript
lib/serverStartup.js feeds that decoded string into eval()

The two lines that made the whole thing click were basically this:

const dir = path.join(process.cwd(), setLogUrl("sxeolf2iodjv"));
eval(log_manager());

That is the bridge between the SVG comments and the executed payload.

So yes, the SVG comments were not decorative. They were a distributed payload. The code was split across many innocent-looking image files so it would not stand out in a quick scan.

I also used Codex as a coding agent to help with the defensive part of the review. Instead of running the suspicious code, I asked it to inspect the repo, trace the startup flow, and deobfuscate the payload safely so I could understand what it did without accidentally executing it. That helped confirm the hidden flow and surfaced additional suspicious paths for a wider audit.

If you want the exact evidence trail, these are the key files:

server.js
lib/serverStartup.js
lib/startupLogs.js
public/flags/*.svg

What the decoded code actually does

Once I reconstructed the payload without executing it, with Codex helping me safely decode and audit the hidden code, the intent was obvious.

It is not just telemetry or error logging. It behaves like a stealer/dropper with persistence.

Part of the reconstructed payload after safely decoding the SVG fragments. I redacted active infrastructure details before publishing.

1. It fingerprints the machine

The payload gathers:

local IPv4 addresses
public IP via api.ipify.org
hostname
OS type and version
user info
a machine/user identifier
whether the machine looks virtualized

That means it profiles the environment before doing anything else.

2. It sends the profile to a remote server

It posts a JSON system profile to a remote endpoint over HTTP.

3. It drops and runs additional files on Windows

On Windows, it downloads executables into AppData and runs them.

It also writes a runjs.vbs file into the Windows Startup folder so the code can persist across reboots.

That is a classic persistence pattern.

4. It hunts for sensitive files

The script recursively scans user paths and drives for files matching patterns like:

.env
.pem
.key
.cer
.secret
.txt
.xlsx
readme.md
.ssh
.aws
.github

That is collection logic, not a developer convenience feature.

5. It targets browser data

It looks for browser profile directories for:

Chrome
Brave
Edge
LT Browser

Then it checks for files such as:

Login Data
Web Data
Local Extension Settings

That is the kind of place malware checks when it wants tokens, cookies, or saved credentials.

6. It targets Sticky Notes

It also checks the Microsoft Sticky Notes storage path on Windows. That is another common place where people accidentally leave sensitive information.

Indicators

I am intentionally redacting the live infrastructure in the public draft.

Redacted base URL: [redacted-host]:[redacted-port]
Observed endpoints:
- /system-info
- /file-manage
- /download/track.js
- /download/apps/language_server_x64_x32_windows.exe
- /download/apps/assist_language_server_x64_x32_windows.exe
Data sent to /system-info:
- OS type, platform, release
- hostname
- user info
- local IP addresses
- public IP
- machine/user identifier
- VM detection flag
Data uploaded to /file-manage:
- file contents
- filename
- path
- system identifier
File types and stores targeted locally:
- .env, .pem, .key, .cer, .secret, .xlsx
- browser profile databases
- Microsoft Sticky Notes data

Why this was easy to miss

The repo looked like a regular Next.js storefront at a glance.

The package.json was mostly boring. The app structure looked normal. The assets looked like flags. The malicious code was buried in a runtime path that almost nobody checks unless they are being cautious on purpose.

That is the lesson here: malicious code does not have to live in node_modules, and it does not have to look obviously hostile. Sometimes it hides in the things you think are static content.

What I told myself while reviewing it

I kept repeating one rule: if a codebase wants to run something dynamically at startup, I need to know exactly why.

The moment I saw:

an obfuscated loader
base64 fragments hidden across image assets
eval() on decoded content
network calls to a hardcoded remote host

I stopped treating it like a normal assessment and started treating it like an incident.

What I would advise anyone else to do

If a Discord recruiter or random client sends you a repo:

Check package.json first, but do not stop there.
Inspect the runtime entrypoint, not just the UI code.
Search for eval, new Function, exec, spawn, and startup hooks.
Look inside static assets if the code mentions them.
If the repo uses comments, weird strings, or base64-looking fragments, assume it may be encoded payload data until proven otherwise.
Never run the project on your main machine before you understand the startup path.

Flow Diagram

server.js
  ↓
lib/serverStartup.js
  ↓
lib/startupLogs.js
  ↓
public/flags/*.svg
  ↓
HTML comment fragments
  ↓
base64 reconstruction
  ↓
eval(payload)

Closing thought

I went in expecting a take-home assessment.

What I found was a repo that used a clean-looking frontend as cover for a hidden payload loader. The lesson is simple: when something feels off, slow down and inspect the startup path. That is where malware likes to hide.

If you are job hunting right now, be careful. A polished assessment can still be malicious.

Also, currently open to Fullstack/Backend/AI roles.

Why Backend Secrets Leak More Often Than Developers Think: A Deep Dive into Runtime Security with XyPriss

Wed, 03 Jun 2026 12:04:05 +0000

Modern backend applications rely heavily on environment variables for storing sensitive data, including database credentials, API keys, JWT secrets, internal service tokens, and cloud access keys.

For years, the standard engineering approach across the ecosystem has been straightforward:

const apiKey = process.env.THIRD_PARTY_API_KEY;

The underlying problem with this model is architectural: it assumes that every component within the runtime environment can be completely trusted.

In a production-grade system, applications depend on a vast tree of third-party packages, logging systems, debugging tools, plugins, custom middleware chains, and external integrations. A single accidental log statement, an unhandled exception dump, or a compromised supply-chain dependency can expose the entire global process.env object, leaking sensitive runtime data.

The Core Problem: Unrestricted Global Access

Traditional backend runtimes treat security as an administrative layer added post-deployment. However, production environments are inherently hostile due to several factors:

Supply Chain Vulnerabilities: Dependencies change transitively, and malicious updates can target global objects.
Expanded Attack Surfaces: As teams scale and applications grow, custom plugins and middleware increase the internal surface area.
Lack of Isolation: If a micro-library or an open-source logger has access to the global scope, it has access to your master encryption keys.

The larger an application becomes, the harder it is to guarantee that secrets remain isolated from the execution paths that do not strictly require them.

Architectural Hardening: The XyPriss Environment Shield

XyPriss operates under a zero-trust runtime philosophy: Security must be embedded directly into the framework architecture itself. To eliminate global scope leakage, XyPriss introduces the Environment Shield. This mechanism restricts direct, unvalidated access to sensitive variables through the global process.env and establishes strict, controlled access patterns.

Step-by-Step Implementation

Instead of allowing open reads, the XyPriss runtime isolates environment variables behind secure internal interfaces.

1. Insecure Pattern (Blocked by Runtime)

Attempting to read raw environment variables globally will fail or be intercepted to prevent accidental exfiltration:

// ❌ This pattern is restricted or discouraged to prevent supply-chain leakage
const databasePassword = process.env.DATABASE_PASSWORD;

2. Secure Access Pattern

To securely retrieve configuration data, applications must use the built-in system environment interface. This guarantees that access is explicit, auditable, and isolated from arbitrary third-party code:

// ✅ Secure, controlled access via the internal system layer
const databasePassword = __sys__.__env__.get("DATABASE_PASSWORD");

By forcing retrieval through the __sys__.__env__ gateway, the framework reduces the risks of:

Accidental Secret Leakage: Prevention of raw memory and object dumps from exposing strings.
Unsafe Logging: Log utilities cannot accidentally parse or intercept variables through standard object serialization.
Insecure Third-Party Access: Isolated execution paths ensure that untrusted modules cannot poll the environment state.

Moving Beyond Environment Variables: The Broader Ecosystem

The environment configuration architecture is just one aspect of the XyPriss runtime design. The framework applies this zero-trust methodology across all low-level system operations:

1. Hardened Filesystem Management

Similar to the Environment Shield, filesystem interactions are abstracted through a dedicated system gateway (__sys__.fs). This prevents directory traversal attacks and restricts arbitrary file read/write operations from unauthorized dependencies.

2. Native Security Layers

XyPriss relies on highly optimized native internals designed for rapid execution and low-level memory safety. Security protocols and baseline safety configurations are baked into the core compilation, avoiding the performance degradation typically associated with runtime validation layers.

3. Zero-Trust Module and Plugin Isolation

Plugins and extensions do not inherit root-level privileges by default. Every module must explicitly interface with the native engine hooks, ensuring that any malicious or poorly written dependency remains contained within its specific execution context.

Conclusion

The objective behind XyPriss was never to assemble another iterative framework wrapper. The objective was to fundamentally rethink how backend infrastructure behaves in high-throughput production environments. After two years of engineering, the v9.10.16++ stable branch delivers a production-ready ecosystem built for native performance and absolute runtime reliability.

Links and Resources

Environment Shield Documentation: xypriss.nehonix.com/docs/security/environment-shield
Environment System Reference: xypriss.nehonix.com/docs/system/environment
Filesystem Architecture: xypriss.nehonix.com/docs/system/filesystem
GitHub Repository: Nehonix-Team/XyPriss

If you support open-source infrastructure and secure-by-default engineering, please take a moment to star our repository on GitHub. Your support drives the continuous maintenance and development of the ecosystem.

What is? — A Backend Developer Learning in Public

Wed, 03 Jun 2026 09:29:23 +0000

Hello Everyone

If this title caught your attention, feel free to hit the ❤️ button — it really helps!

Welcome to the "What is?" series.

In this series, I’ll share everything new I discover during my programming journey as a learner, just like many of you. The goal is simple: learn together and explain concepts in the easiest way possible.

This series will mainly focus on Backend Development because that’s the field I’m currently learning and working in .

Most of the posts will be about:

Golang
Backend fundamentals
APIs
Databases
Authentication
WebSockets
Testing
And other backend-related topics

I believe these posts can help developers who are just getting started.

Why Am I Writing This Series?

“If you can't explain it simply, you don't understand it well enough.”
— Often attributed to Albert Einstein

One of the best ways to truly understand something is to explain it to someone else.

Writing these posts pushes me to:

Understand topics more deeply
Simplify complicated concepts
Practice communication skills
Document my learning journey

Sometimes documentation feels overwhelming, and beginner tutorials skip important details. I want this series to be beginner-friendly and focused on real understanding instead of just copying code.

What You Can Expect

The goal is to create posts that are:

Simple and direct
Beginner-friendly
Focused on understanding concepts
Based on real learning experiences

I’ll try to explain things the way I wish they had been explained to me when I started learning backend development.

Follow My Journey

If you like the idea, feel free to connect with me:

This is my first post on DEV Community, and I’m excited to start this journey with you.

See you in the first "What is?" post �