Coder Legion - Recent posts tagged backend

Building an ATS Resume Scorer API with FastAPI, Celery, and Docker

Fri, 30 Oct 2026 07:57:26 +0000

Most resumes are filtered by ATS software before a human ever reads them. This project is a backend API that checks how well a resume matches a job description and suggests how to improve it.

What it does

PDF resume parsing: extracts text from uploaded resumes using pdfplumber
ATS keyword matching: compares the resume against a job description and calculates a match score
Smart suggestions: generates actionable tips to improve the resume for a specific role
JWT authentication: secured endpoints with role-based access for Job Seekers and Recruiters
Batch scoring: recruiters can score many resumes at once, handled in the background by Celery + Redis

Tech stack

Layer	Technology
Backend	Python, FastAPI
Auth	JWT (python-jose, passlib)
Database	MySQL 8.0, SQLAlchemy ORM, Alembic
Task queue	Celery + Redis
PDF parsing	pdfplumber
DevOps	Docker, Docker Compose, AWS EC2
CI/CD	GitHub Actions
Testing	Pytest

Design highlights

Background processing: batch scoring runs through a Celery queue, so the API stays responsive while many resumes are processed.
Role-based access: Job Seekers and Recruiters get separate permissions on the same API.
Reproducible setup: the whole stack (API, MySQL, Redis) starts with a single Docker Compose command.
Automated deployment: a GitHub Actions pipeline deploys automatically to AWS EC2.
Database migrations: schema changes are managed with Alembic.

Source code

https://github.com/priyadharshiny13/ats-resume-scorer

Feedback is welcome. If you were using a tool like this, what would matter more: a more accurate match score or more detailed improvement suggestions?

Building a Booking System with Node.js, Google Calendar, and Google Meet

Tue, 27 Oct 2026 07:58:12 +0000

Introduction

Booking a consultation should be straightforward for the person making the appointment. Behind the scenes, however, an automated booking system has to coordinate several moving parts: collecting customer information, checking availability, scheduling an event, generating a meeting link, saving the booking, and notifying the people involved.

I encountered these challenges while developing the booking system for JOPAD Advisory. I built the system from scratch, using Node.js and Express for the backend, MongoDB for storing bookings, Google Calendar for scheduling, Google Meet for virtual meetings, and Resend for email notifications.

In this article, I'll walk through how I implemented the booking workflow, how the different services work together, and some of the engineering considerations that come with connecting external APIs to a backend application.

1. The Architecture

The backend coordinates the booking process. Each service has a specific responsibility:

Node.js and Express: Handle booking requests and coordinate the workflow.
MongoDB and Mongoose: Store booking details and provide access to existing appointments.
Google Calendar API: Create calendar events for scheduled sessions.
Google Meet: Generate a virtual meeting link through the Calendar event's conference configuration.
Airtable: Receive a secondary record of the booking.
Resend: Power the email notification workflow.

The booking process follows this sequence:

A customer submits a booking request through the website.
The Express endpoint validates the submitted fields.
The backend calculates the appointment's start and end times.
Existing bookings are checked for overlapping time intervals.
Google Calendar creates an event and requests a Google Meet conference.
The booking is saved to MongoDB with its meeting link.
The backend attempts to send a secondary record to Airtable.
Booking notification emails are triggered.

This design brings several services together behind a single booking endpoint, giving the frontend one place to submit the request.

2. Building the Booking Endpoint

I used an Express POST endpoint to receive booking requests. The request contains the appointment time, duration, customer's contact details, session description, and other booking information.

The first step is to reject incomplete requests before performing external API operations.

A simplified version of the validation looks like this:

router.post("/book", async (req, res) => {
  const {
    time,
    duration,
    email,
    firstName,
    lastName,
    description
  } = req.body;

  if (
    !time ||
    !duration ||
    !email ||
    !firstName ||
    !lastName ||
    !description
  ) {
    return res.status(400).json({
      message: "Missing required fields"
    });
  }

  // Continue with availability checks
  // and booking creation.
});

This is a simplified illustration of the validation logic in my implementation, not a complete replacement for the original endpoint.

The actual endpoint validates additional fields, including the selected date, day, time range, phone number, and referral information.

The important principle is to validate the data required by the booking workflow before creating external resources or writing records.

For a production system, request validation should also verify data types, valid email addresses, positive durations, valid dates, and acceptable appointment times. A field being present does not necessarily mean its value is valid.

3. Calculating Appointment Duration

A booking needs both a start time and an end time. The end time determines the appointment's actual interval and makes it possible to detect overlapping bookings.

In my implementation, the backend calculates the end time using the requested start time and duration:

const startTime = new Date(time);
const endTime = new Date(startTime);

endTime.setMinutes(
  endTime.getMinutes() + Number(duration)
);

For example, an appointment starting at 10:00 AM with a duration of 60 minutes should end at 11:00 AM.

Calculating the interval on the backend is important because scheduling decisions should not depend solely on values calculated by the browser.

It is also important to handle time zones consistently. My Google Calendar integration sends ISO-formatted timestamps to the API. A more comprehensive implementation should explicitly define the application's scheduling time zone and validate incoming timestamps to prevent ambiguity around local times and daylight-saving transitions.

4. Preventing Overlapping Bookings

One of the central problems in a booking system is preventing customers from selecting times that have already been reserved.

Two appointments overlap when the first appointment starts before the second one ends and finishes after the second one starts.

The logic can be expressed as:

const overlaps =
  requestedStart < existingEnd &&
  requestedEnd > existingStart;

This condition also allows back-to-back appointments. For example, a booking ending at 11:00 AM does not conflict with another booking beginning at exactly 11:00 AM.

In my implementation, I retrieve existing bookings from MongoDB, calculate each booking's end time from its start time and duration, and check for conflicts:

const existingBookings = await Booking.find();

const conflict = existingBookings.some(booking => {
  const existingStart = new Date(booking.startTime);
  const existingEnd = new Date(existingStart);

  existingEnd.setMinutes(
    existingEnd.getMinutes() +
      Number(booking.duration)
  );

  return (
    startTime < existingEnd &&
    endTime > existingStart
  );
});

if (conflict) {
  return res.status(409).json({
    message: "This time slot has already been booked"
  });
}`

Returning HTTP "409 Conflict" communicates that the request conflicts with an existing booking.

This approach is easy to understand, but it has a scalability limitation: the implementation loads all existing bookings before checking for an overlap. As the collection grows, a database query restricted to potentially overlapping intervals would be more efficient.

There is also an important concurrency consideration. Two requests can check availability at almost the same time, both find the slot free, and both continue. The check alone does not guarantee that simultaneous requests cannot create overlapping bookings.

A stronger production design would use an atomic reservation strategy, such as reserving discrete time slots with a unique database constraint or using another concurrency-control mechanism. That protection needs to be designed around the application's scheduling rules rather than relying only on an initial availability check.

5. Connecting Google Calendar and Google Meet

After the backend confirms that a requested slot does not conflict with an existing booking, it calls a dedicated utility function to create the meeting.

I separated this logic into a Google Calendar utility instead of placing the entire integration directly inside the Express route. This keeps the route focused on the booking workflow and makes the external integration easier to maintain.

Authenticating with Google

The utility initializes an OAuth 2.0 client using environment variables and sets a refresh token:

const { google } = require("googleapis");

const oauth2Client = new google.auth.OAuth2(
  process.env.GOOGLE_CLIENT_ID,
  process.env.GOOGLE_CLIENT_SECRET,
  process.env.GOOGLE_REDIRECT_URI
);

oauth2Client.setCredentials({
  refresh_token: process.env.GOOGLE_REFRESH_TOKEN
});

The example uses the conventional environment-variable spelling for the client secret. In the project, the environment variable is named "GOOGLE_CLIENT_SECRETE", so the deployed configuration must match the actual code unless that naming is corrected.

The refresh token allows the server-side client to obtain access tokens as needed without requiring the user to repeat the authorization flow for every booking.

This requires the initial OAuth authorization to have been completed successfully and the refresh token to remain valid. The Google account must also have the necessary permissions and configuration for creating Calendar events and conference links.

Credentials should be stored securely in environment variables or an appropriate secrets manager, never committed to a public repository.

Creating a Calendar event

The integration uses Google's Calendar API:

const calendar = google.calendar({
  version: "v3",
  auth: oauth2Client
});

The meeting function calculates the event's end time and constructs a Calendar event containing the session title, description, start and end timestamps, and attendee information.

The attendee is represented by the customer's email address:

attendees: [
  {
    email,
    responseStatus: "accepted"
  }
]

The event also includes settings that prevent guests from modifying the event, inviting other guests, or seeing other guests.

These settings are part of the event configuration; they do not replace proper authorization and access control in the booking application itself.

Requesting a Google Meet link

The event configuration includes a conference creation request:

conferenceData: {
  createRequest: {
    requestId: "jopad-" + Date.now(),
    conferenceSolutionKey: {
      type: "hangoutsMeet"
    }
  }
}

The event is then inserted into the primary Google Calendar:

const response = await calendar.events.insert({
  calendarId: "primary",
  resource: event,
  conferenceDataVersion: 1,
  sendUpdates: "all"
});

The "conferenceDataVersion: 1" option tells Google Calendar that conference data should be processed as part of the request. The "sendUpdates: "all"" option requests that Calendar send event updates to the relevant participants.

After the API call succeeds, the implementation extracts the meeting URL from the returned conference data:

const meetLink =
  response.data.conferenceData.entryPoints[0].uri;

return meetLink;

The booking route receives that link and includes it in the booking record.

One important implementation detail is that the code assumes the response contains a conference entry point. A more defensive implementation would verify that the response contains a usable Meet URL before continuing.

Google Calendar conference creation can also involve asynchronous processing. Production code should account for cases where the event exists but the conference details are not yet available, rather than assuming every successful event insertion immediately includes a meeting link.

6. Saving Bookings in MongoDB

Once the meeting link has been returned, the backend creates a Mongoose booking document.

The model stores information such as:

Appointment start time and duration.
Customer name and contact information.
Session description.
Date and time display values.
Referral information.
Google Meet link.
Record creation timestamp.

The central fields include:

const bookingSchema = new mongoose.Schema({
  startTime: {
    type: Date,
    required: true
  },
  duration: String,
  email: String,
  firstName: String,
  lastName: String,
  description: String,
  meetLink: String,
  createdAt: {
    type: Date,
    default: Date.now
  }
});

This is an excerpt of the schema; the complete project model contains additional fields.

After constructing the document, the route saves it:

await newBooking.save();

Saving the booking after generating the meeting link ensures that the returned booking record can contain the link associated with the created event.

However, the order of these operations creates an important failure scenario. If Google Calendar successfully creates the event but MongoDB subsequently fails to save the booking, a calendar event may remain without a corresponding booking record.

This is a common challenge when coordinating operations across independent services. A more resilient design could track workflow states, retry recoverable operations, and provide a reconciliation or cleanup process for partially completed bookings.

7. Adding an Airtable Backup

The system also attempts to send booking information to Airtable after saving the MongoDB document.

The request includes selected booking fields, such as the customer's name, email, phone number, session description, referral details, meeting link, and appointment date.

The operation is wrapped in its own "try...catch" block. If the Airtable request fails, the error is logged, but the main booking workflow can still continue.

This makes Airtable a secondary destination rather than the primary source of truth for the booking.

That distinction matters: the implementation attempts a backup write, but it does not guarantee that the MongoDB and Airtable records are always synchronized. If Airtable is unavailable, the backup attempt fails without automatically retrying it.

A more robust implementation could place failed backup operations into a retry queue or maintain a record of synchronization status.

8. Sending Booking Notifications

After the booking has been saved and the Airtable attempt has completed, the route triggers the email notification utility:

sendBookingEmails({
  email,
  firstName,
  lastName,
  dayString,
  referral,
  dateString,
  number,
  description,
  timeString,
  timeRange,
  meetLink
}).catch(err => {
  console.log("Email failed:", err.message);
});

The email operation is not awaited by the route. This allows the endpoint to return its successful booking response without waiting for the notification workflow to finish.

It also means an email failure does not change the response after the booking has been saved.

There is a trade-off: the current implementation logs email failures but does not automatically retry them or record a delivery status in the booking document. For a production booking service, a persistent background job or message queue would provide a stronger way to retry notifications and monitor delivery outcomes.

The route eventually responds with HTTP "201 Created", along with a success message and the saved booking.

9. Lessons from the Implementation

Building this workflow brought several backend engineering considerations into focus.

Validate before creating external resources. Checking the request and current bookings before calling Google reduces unnecessary API operations.

Separate responsibilities. Keeping the Calendar integration in its own utility makes the Express route easier to understand and maintain.

Handle partial failures. MongoDB, Google Calendar, Airtable, and an email provider are independent services. A successful operation in one does not guarantee success in the others.

Treat concurrency as a separate problem. Checking for overlapping bookings is necessary, but it does not by itself protect against two simultaneous requests reserving the same slot.

Design for observability and recovery. Logging an error helps with debugging, but production systems also benefit from structured logs, retry mechanisms, delivery tracking, and reconciliation workflows.

These considerations extend beyond booking applications. Any backend that coordinates a database with multiple external APIs needs a plan for partial failures and retries.

Conclusion

Building the JOPAD Advisory booking system from scratch gave me practical experience connecting a Node.js backend to Google Calendar, generating Google Meet links, storing bookings in MongoDB, maintaining a secondary Airtable record, and triggering email notifications.

The most important lesson is that integrating APIs is only one part of building a reliable application. The surrounding workflow—validation, conflict detection, persistence, failure handling, and recovery—is just as important.

A booking endpoint can appear simple from the frontend, but it coordinates several systems that can succeed or fail independently. Understanding those interactions is what turns an API integration into a more complete engineering solution.

I'm continuing to explore ways to make these workflows more resilient, especially around concurrent bookings, retries, and recovery from partial failures.

If you're building a booking platform or integrating third-party services into a Node.js application, I'd be interested to hear how you approach these challenges.

is-kit Reached 100 Stars ⭐🎉 Here’s What Changed Since 50

Fri, 23 Oct 2026 12:24:38 +0000

Hoi hoi!

YAY! is-kit just reached 100 GitHub stars⭐ 🎉🎉🎉

When it reached 50 stars, I wrote about how we use it in a production TypeScript application serving more than 100,000 users. 👇

https://dev.to/nyaomaru/is-kit-reached-50-stars-heres-how-we-use-it-in-production-2i5b

I could write the same story again. But that would be boring. 😹

A lot has happened since then.

The library gained new APIs, some existing APIs became stricter, I started using it in more of my own OSS projects, and one real-world use case even exposed a type inference edge case in Effect.

So this time, instead of explaining the production adoption again, I want to look at:

What changed between 50 and 100 stars
Some of the most important APIs added since then
How I am actually using is-kit outside the original production application
A real bug report involving Effect.catchIf
What 100 stars means to me now

Let's dive in! 🔧

⭐ First, What Did 50 Stars Mean?

At 50 stars, the biggest thing I wanted to show was that is-kit was not only a toy project.

It was already being used in a real production application. 🚀

The application had repeated runtime checks like

typeof value === "string";
typeof value === "number";
value === null || value === undefined;

and several larger user-defined type guards for things like HTTP errors, literal unions, arrays, API values, and plain objects.

The interesting part was not that is-kit made those checks shorter.
The interesting part was that it gave them one reusable shape.

Small guards could be composed.

Application-owned helpers could sit between the library and feature code. And the result could still participate in normal TypeScript control flow.

That part has not changed.

But since 50 stars, the library itself has moved quite a bit. 👀

One thing I kept running into was this.

I already have a good type guard.
I just want to apply it to one property.

For example

declare const input: unknown;

Maybe I already know how to validate a string

import { isString } from "is-kit";

But now I want to express

"This object has a name property, and that property is a string."

That sounds simple, but there are actually a few different cases.

A required property is not the same as an optional property.
An array index is not the same as an object property.

And an optional property that becomes known to be defined should narrow differently from one that is still optional.

So is-kit added three property refinement helpers:

refineKey
refineDefinedKey
refineIndex

The idea is simple

Take a type guard you already trust and lift it onto a property or array element.

For example

import { isString, refineKey } from "is-kit";

const hasStringName = refineKey("name", isString);

Now the string guard is reusable at a deeper level instead of being rewritten inside another custom predicate.

This is a small API, but I like it because it follows the same design goal as the rest of is-kit.

Do not build one giant validator.

Build small checks that can be reused in normal TypeScript code.

🔍 Then `discriminatedUnion` Got Much More Serious

The biggest addition since the 50-star article is probably the exhaustive discriminated union API introduced in v1.15.0.

Consider a common TypeScript type

type Result = { ok: true; value: string } | { ok: false; error: string };

At runtime, validating this is not difficult.

You can check ok, then validate the corresponding fields.

But there is another question

Can TypeScript verify that the validator itself covers the whole union?

That is where discriminatedUnion comes in.

import { discriminatedUnion, isString, oneOfValues, typedStruct } from "is-kit";

type Result = { ok: true; value: string } | { ok: false; error: string };

const resultUnion = discriminatedUnion<Result>();

export const isResult = resultUnion("ok", [true, false], {
  true: typedStruct<Extract<Result, { ok: true }>>()({
    ok: oneOfValues(true),
    value: isString,
  }),
  false: typedStruct<Extract<Result, { ok: false }>>()({
    ok: oneOfValues(false),
    error: isString,
  }),
});

The runtime behavior is useful.

But the more interesting part is the compile-time contract.

The builder checks things like:

Missing discriminant branches
Extra branches
Incompatible branch guards
Incomplete discriminant tuples

So if the union changes, the guard cannot silently stay incomplete.

That matters because one of the easiest ways for a hand-written type guard to become dangerous is drift.

The TypeScript type evolves.
The runtime guard does not.
And the compiler cannot normally tell you that the two are no longer aligned.

discriminatedUnion is one attempt to make that class of mistake harder.

💪 The Library Also Became Stricter

Not every improvement between 50 and 100 stars was a new feature.

A lot of work was about making existing behavior more precise.

For example, recent releases included fixes around:

recordOf literal-key narrowing
native Error brand checks
array guards with overridden iterators
exact struct validation
symbol-keyed struct schemas
cross-realm struct schemas

There was also more package-level testing.

is-kit now checks published-package behavior from both ESM and CommonJS TypeScript consumers, and also has Deno / JSR smoke coverage.

This part is less exciting to show in a code snippet.

But for a small library, I think it is important.

Making sure they behave correctly in weird JavaScript and TypeScript environments is where things become real.

🤖 I Started Using `is-kit` in Other OSS Too

Since reaching 50 stars, I've also started using is-kit more naturally in other OSS projects I maintain.

For example, in my changelog tooling of changelog-bot, I use define to keep small predicates reusable and composable instead of turning every runtime check into an isolated helper.

A simplified example looks like this 👇

import { define, oneOfValues } from "is-kit";

const CHANGELOG_SECTIONS = ["Added", "Changed", "Fixed", "Removed"] as const;

const isChangelogSection = define<(typeof CHANGELOG_SECTIONS)[number]>(
  oneOfValues(CHANGELOG_SECTIONS),
);

The exact code is not the important part.

What changed for me is that I no longer think

"I should use my library here."

I just reach for the guard I need.

A library starts becoming useful to its maintainer when it stops feeling like a demo of itself.

🚀 Then an `is-kit` Guard Ran Into an Effect Type Bug

This was one of the most interesting things that happened recently.

An is-kit user was using a type guard with Effect's catchIf.

Conceptually, the refinement looked like this 👇

(error: unknown): error is SomeError => ...

That is completely normal for is-kit.

In fact, unknown is exactly where the library is designed to start.

But with Effect.catchIf, that refinement could affect generic inference in an unexpected way.

The error channel could become widened to unknown.

The reported case resulted in this Effect PR 👏 And merged 🚀

https://github.com/Effect-TS/effect/pull/9252

The relevant type originally looked roughly like this

refinement: Predicate.Refinement<E, EB>;

and the fix changed it to

refinement: Predicate.Refinement<NoInfer<E>, EB>;

The important idea is that the refinement's input type should not be allowed to redefine the error type already established by the Effect.

The PR also adds type tests for cases like

const result = Effect.catchIf(
  mixedEffect,
  (e: unknown): e is AiError => e instanceof AiError,
  () => Effect.succeed("ok"),
);

Without the fix, the refinement input could participate in inference and widen the error channel.

With NoInfer, the original error type remains the source of truth.

The same pattern was fixed for Channel and Stream too.

There was also a Bluesky post about the issue

https://bsky.app/profile/evadeva.bsky.social/post/3mxwiqfeo3s2j

I really liked this bug because it came from an interaction I would probably never have discovered by testing is-kit alone.

is-kit did not "break Effect". And Effect was not doing something absurd either.

It was a subtle TypeScript inference interaction between a generic API and a refinement whose input was intentionally broad.

That is exactly the kind of edge case that appears when libraries start meeting each other in real code.

👍 This Is the Kind of Feedback I Wanted

When is-kit reached 50 stars, I wrote

"Someone found this idea useful."

That was already enough to make me happy.

100 stars still does not mean the library is huge.

A star is still not a benchmark.
It is still not proof of correctness.
And it definitely does not mean every TypeScript project should install is-kit.

But something has changed.

People are not only starring the repository anymore.

They are

Using the library
Asking questions
Finding edge cases
Reporting bugs
Comparing it with other validation approaches
Using its guards with other TypeScript libraries
Making me reconsider APIs I thought were already finished

That is much more valuable than the number itself. 💖

💬 The Project Is Also Becoming Easier to Explain

Another thing changed between 50 and 100 stars.

I became much clearer about what is-kit is not.

It is not trying to replace every schema library.
It is not trying to turn every runtime value into a schema-first workflow.
And it is not trying to hide normal TypeScript control flow.

The core question is still very small

"Can this unknown value safely enter this branch?"

Sometimes a schema library is the right answer.

For example, if I need:

Rich validation errors
Transformations
Form validation
A contract shared across systems

I may want Zod, Valibot, ArkType, or something else.

But sometimes I already have normal TypeScript code and I only need a reusable narrowing function.

That is where I want is-kit to be useful.

if (isUser(value)) {
  value.id;
}

if (isHttpClientError(error)) {
  // Handle HTTP client errors
}

items.filter(isString);

result.catch(isHttpClientError);

const isMaybeNumber = nullish(isNumber);

Small runtime decisions.
Normal functions.
Normal TypeScript.

🐱 What 100 Stars Means to Me

100 stars is still small compared with the largest TypeScript OSS projects.

No problem.

What matters to me is that the project is becoming more real in ways that have nothing to do with the star counter.

Since the 50-star milestone:

New APIs were added
Existing APIs became stricter
Runtime edge cases were fixed
Package compatibility testing improved
I started using is-kit naturally in other OSS
External usage produced feedback I could not generate alone
An unknown-based refinement even helped expose an inference edge case in Effect

That last part is probably my favorite.

Because the goal was never to build a library that only looks nice in its own README!!!

The goal is to make reusable runtime narrowing work naturally inside real TypeScript code.

And now I am starting to see that happen outside my own repository too.

Thank you to everyone who starred the project, tried it, reported something, asked a question, opened an issue, or simply read the code. 🙏

There is still a lot I want to improve.

But 100 feels pretty nice. 😸⭐

If small composable type guards fit your TypeScript style, give it a try!

https://github.com/nyaomaru/is-kit

Thank you for reading! 🤲

Demystifying MongoDB Write Concern and Read Concern in Node.js

Mon, 19 Oct 2026 09:22:10 +0000

What Are Write Concern and Read Concern?

In MongoDB, write concern and read concern are configuration settings that specify the level of guarantee you require when saving or retrieving data across a distributed cluster. A write concern determines how many database nodes must successfully record an update—and whether that data must be written to disk—before MongoDB reports success to your application. A read concern determines how fresh and consistent the data returned by a query must be, protecting your application from reading temporary data that might later be reversed.

Together, these two settings let developers choose where their application sits on the classic spectrum between maximum execution speed and rock-solid data safety.

The Real-World Analogy: Certified Mail vs. Kitchen Whiteboards

Imagine you run a bustling catering kitchen with multiple head chefs.

When taking an order, you have two choices for your write concern. You could scribble the customer's allergy request on a loose scrap of paper and immediately yell "Order placed!" (a fast write, but if you drop the paper, the order is lost). Alternatively, you could walk the order ticket over to the main order board, watch two sous-chefs sign off on it, and wait for them to file it in the metal lockbox before confirming the order with the customer. The second route takes several seconds longer, but you are guaranteed that the kitchen will never forget the order even if one chef walks out.

Now consider the read concern. If a waiter needs to know the soup of the day, they can either glance at a whiteboard where someone started drafting ideas with an erasable marker, or they can refuse to serve anything until they see the official menu stamped and approved by the head kitchen manager. Glancing at the draft whiteboard is instantaneous, but you might serve an experimental soup that the manager ultimately rejects five minutes later.

Why It Matters in Daily Software Engineering

Modern web architectures rarely run on a single database server. Instead, they use replica sets—groups of connected servers where one "primary" node handles writes and multiple "secondary" nodes replicate that data for redundancy.

If you leave MongoDB on default settings during high-traffic moments, subtle race conditions emerge:

Rollback Catastrophes: If an API endpoint confirms an e-commerce order after writing only to the primary node, and that primary node suddenly experiences a hardware failure before syncing to secondary nodes, the cluster elects a new primary. Your customer's order is wiped out, yet your payment processor already charged their credit card.
Dirty Reads: A background worker updates a bank balance. A microservice queries the secondary replica before the primary finishes replicating. If that transaction aborts or fails over, the microservice has operated on data that never officially existed.

By tuning write and read concerns, software engineers intentionally shield critical operations (payments, account transfers, credential updates) from data loss while letting low-risk operations (logging, view counters, user analytics) run with blazing speed.

Implementation in Node.js

Here is how you apply write and read concerns when working with the official MongoDB Node.js driver:

import { MongoClient } from 'mongodb';

const client = new MongoClient('mongodb://localhost:27017,localhost:27018/?replicaSet=rs0');

async function processPayment(orderId, paymentDetails) {
  await client.connect();
  const db = client.db('store');

  // Configure high-durability options for financial transactions
  const ordersCollection = db.collection('orders', {
    writeConcern: {
      w: 'majority',      // Must be acknowledged by more than half of the replica set nodes
      j: true,            // Must be written to the on-disk journal before confirming
      wtimeout: 5000      // Abort if replicas do not respond within 5 seconds
    },
    readConcern: {
      level: 'majority'   // Only read data committed by a majority of nodes
    }
  });

  // Step 1: Safely write with acknowledgment
  const writeResult = await ordersCollection.updateOne(
    { _id: orderId },
    { $set: { status: 'PAID', payment: paymentDetails } }
  );

  // Step 2: Read guaranteed, durable data
  const confirmedOrder = await ordersCollection.findOne({ _id: orderId });

  console.log('Payment processed safely:', confirmedOrder._id);
  return confirmedOrder;
}

In this example, specifying w: 'majority' and j: true ensures that the write cannot be rolled back if the primary server abruptly crashes. Setting level: 'majority' on the read ensures your application reads only finalized records.

Key Takeaway

Database speed and data reliability exist on a continuous slider: write concern dictates how hard your cluster works to preserve your updates, while read concern dictates how strictly your cluster verifies the stability of what you fetch. Mastering both allows you to protect financial transactions and vital data without unnecessarily throttling high-volume, low-risk endpoints across your stack.

Enjoyed this breakdown? Support my open-source work on Buy Me a Coffee or explore react-hook-lab!

Originally published on my blog. You can read the alternative breakdown here.

A very Simple Rate Limiting App ( Node.JS )

Thu, 15 Oct 2026 13:40:19 +0000

enter code here

const express = require("express");

const app = express();

const requests = new Map();

const LIMIT = 5;
const WINDOW = 60 * 1000; // 1 minute

app.use((req, res, next) => {
  const ip = req.ip;
  const now = Date.now();

  let data = requests.get(ip);

  // First request or expired time window
  if (!data || now - data.startTime >= WINDOW) {
    data = {
      count: 1,
      startTime: now,
    };

    requests.set(ip, data);
    return next();
  }

  // Rate limit exceeded
  if (data.count >= LIMIT) {
    return res.status(429).json({
      message: "Too many requests. Try again later.",
    });
  }

  data.count++;
  next();
});

app.get("/", (req, res) => {
  res.json({ message: "Request successful!" });
});

app.listen(3000, () => {
  console.log("Server running on http://localhost:3000");
});

The Reconnect Bug Where Marking a User Online Erased the Data I Needed for Replay

Thu, 15 Oct 2026 07:00:41 +0000

While testing a reconnect system for a deliverer-facing WebSocket gateway: when someone reconnects after losing their connection, they should get a snapshot of their current job plus any events they missed while offline. Two pieces, both of which I was confident were correct on their own.

Marking presence:

async markOnline(userId: number) {
  await this.prisma.user.update({
    where: { id: userId },
    data: { isOnline: true, lastSeenAt: new Date() },
  });
}

Computing what was missed:

async replayDelivererEvents(delivererId: number) {
  const deliverer = await this.prisma.delivererProfile.findUnique({
    where: { id: delivererId },
    select: { user: { select: { lastSeenAt: true } }, activeJobId: true },
  });
  // ...fetch events since deliverer.user.lastSeenAt
}

And the connection handler that used both:

async handleConnection(client: Socket) {
  const deliverer = await this.getDelivererByUserId(payload.sub);

  await this.sessionService.markOnline(payload.sub);           // step 1
  const missedEvents = await this.replayService.replayForDeliverer(deliverer.id); // step 2

  client.emit('events.replay', missedEvents);
}

I tested it: disconnect, trigger a job assignment while offline, reconnect. events.replay came back empty, every time. The event genuinely happened. The query genuinely should have found it.

Why

Walk through the actual timeline. A deliverer disconnects — markOffline correctly stamps lastSeenAt with the disconnect time. Later, they reconnect. handleConnection fires, and the first thing it does is call markOnline, which immediately overwrites lastSeenAt to right now. Only after that does it call replayForDeliverer, which reads lastSeenAt to compute "what happened since they were last seen" except by then lastSeenAt no longer holds the disconnect time. It holds the current moment, because the line above just wrote over it.

getEventsSince({ since: lastSeenAt }) effectively became since: right now. The replay feature couldn't fail more quietly — no error, no exception, just a correctly-shaped empty array every single time.

What makes this bug worth writing about isn't the fix — it's that neither piece of code was wrong in isolation. Marking someone online should update lastSeenAt. Replay should read lastSeenAt. The bug only exists in the order they ran in, and that kind of bug doesn't show up by staring at either function — only by tracing the actual sequence of operations across the whole request.

The fix
Read before you overwrite:

async handleConnection(client: Socket) {
  const deliverer = await this.getDelivererByUserId(payload.sub);

  // Replay FIRST, using the lastSeenAt from before this connection
  const missedEvents = await this.replayService.replayForDeliverer(deliverer.id);

  // THEN mark online, which overwrites lastSeenAt for next time
  await this.sessionService.markOnline(payload.sub);

  client.emit('events.replay', missedEvents);
}

No schema change needed. Reordering two already-correct operations was the entire fix.

The part worth remembering

One field, two different jobs — "when did we last hear a heartbeat" and "what's the anchor point for replay" happened to share the same column, and that's exactly what made the collision possible. The one-line reorder fixes today's bug, but it leaves the deeper coupling in place: anything else that later wants to read lastSeenAt for a different purpose can walk into the same trap. The more robust version splits them. a dedicated lastDisconnectedAt, set only on disconnect, read only by replay so two unrelated concerns stop sharing state that one of them mutates as a side effect. I haven't built that yet. It's on the list specifically because I know why it would help, not just that it might.

Keeping a Legacy Cache Consistent During an Incremental Migration

Wed, 14 Oct 2026 04:04:39 +0000

During an incremental system migration, I ran into a subtle consistency problem: our new service could successfully update the database while the legacy service continued reading stale data from its own in-memory cache.

The obvious solutions such as distributed locks, cache invalidation, or adding another piece of infrastructure didn't fit our constraints.

The solution ended up being much simpler: the legacy system already had a lease mechanism that we could reuse.

This is the story of how I found it, how we used it, and how we handled the failure cases.

Architecture

Our system currently has three main components:

Main Service: The client-facing service that sends business requests.
Legacy Service: An older, high-performance engine that maintains an embedded in-memory cache. It loads a subset of database tables into its own memory for fast access.
New Service: A modern service that is gradually replacing the Legacy Service using the Strangler Fig pattern.

The New Service currently acts as a proxy between the Main Service and the Legacy Service.

If an endpoint has already been migrated, the New Service handles the request directly. Otherwise, it forwards the request to the Legacy Service.

The architecture looks roughly like this:

The Problem

The problem appeared when the New Service handled a migrated request.

The New Service writes updates directly to the database. However, the Legacy Service has its own in-memory copy of the data and has no way of knowing that the database changed.

That creates a dangerous sequence:

The Legacy Service caches status = pending.
A request is migrated to the New Service.
The New Service updates the database to status = completed.
A later request reaches the Legacy Service.
The Legacy Service reads status = pending from its local cache.

The database is correct, but the Legacy Service is operating on stale state.

We discovered these issues during QA testing. As more endpoints moved to the New Service, the problem would become increasingly common.

So we needed a way to guarantee that the Legacy Service could not continue operating on stale data while a migrated endpoint was modifying the same resource.

The Constraints

The textbook solutions weren't a good fit for this system.

We had several constraints:

The Legacy and Main Services should not be modified. The Legacy Service is largely a black box, so modifying its behavior is risky.
The Legacy Service could not temporarily operate on stale data. For these operations, we needed to prevent conflicting legacy operations rather than accept eventual consistency.
The Legacy Service had limited cache synchronization capabilities. It automatically reloaded its cache only periodically, or through a manual refresh operation.
We wanted to avoid introducing new infrastructure. Adding something like Redis purely for distributed locking would introduce another dependency to provision, operate, and maintain.

These constraints ruled out several conventional approaches.

The Naive Solution

My first thought was:

What if the New Service acquires a distributed lock, performs the database update, and then triggers a cache refresh on the Legacy Service?

That sounds reasonable, but there were two problems.

1. The Legacy Service wouldn't know about the lock

An external distributed lock only coordinates operations if the participating systems can use it.

The Legacy Service was a black box and wouldn't check our new lock before accessing its local cache.

We could add that behavior, but that violated one of our main constraints.

2. Refreshing the entire cache was too expensive

The Legacy Service's manual refresh operation reloaded a large amount of data.

Doing that after every migrated request would introduce significant latency and unnecessary work.

We needed something more targeted.

The Solution: Reusing an Existing Lease

While investigating the Legacy Service's existing APIs, I found something useful: it already had a built-in lease mechanism.

The Legacy Service exposed APIs to:

Acquire a lease for a specific resource.
Release the lease when the operation was complete.

More importantly, releasing the lease had an existing side effect: the Legacy Service refreshed the cached data associated with that resource.

The lease mechanism was already part of how the Legacy Service protected its own critical sections.

This changed the problem completely.

Instead of introducing a new distributed lock, we could use the same coordination mechanism the Legacy Service already understood.

The New Service could wrap a migrated operation like this:

1. Acquire lease for the resource
2. Update the database
3. Perform business logic
4. Release the lease
5. Legacy Service refreshes that resource's cached data

The important part is that the lease wasn't merely an arbitrary lock exposed by the Legacy Service. The Legacy Service's own operations already align with the same lease mechanism.

That gave us two properties we needed:

Conflicting Legacy Service operations could not proceed while the resource was leased.
Releasing the lease caused the Legacy Service to refresh the affected resource rather than performing a full cache reload.

So instead of adding Redis, Kafka, or another synchronization system, I reused a primitive that already existed in the system.

What If Releasing the Lease Fails?

The happy path was straightforward.

But distributed systems rarely fail only on the happy path.

Consider this sequence:

New Service
    |
    | acquire lease
    v
Legacy Service
    |
    | lease acquired
    |
    v
New Service
    |
    | update database
    |
    X---- network failure ----> release request never arrives

The database update succeeded, but the release request never reached the Legacy Service.

If leases could remain indefinitely, the resource could become permanently locked.

So I looked at how the Legacy Service handled abandoned leases.

It already had two mechanisms:

TTL (Time-To-Live): Every lease automatically expires after a configured duration.
Background reaper: A background process periodically detects expired leases and cleans them up.

Even better, lease cleanup also triggered the same cache refresh behavior.

That meant the failure path became:

Acquire lease
     |
     v
Update database
     |
     X  release request lost
     |
     v
Lease expires
     |
     v
Background reaper
     |
     v
Lease cleared + cache refreshed

We therefore didn't need to build a separate retry system just to recover from a lost release request.

The existing lease lifecycle already provided a recovery mechanism.

An Important Invariant

The key correctness property here is that the resource remains protected by the lease until the Legacy Service releases or expires it.

As long as the database update completes before the lease is released or expires, the Legacy Service cannot resume normal access to the resource using its old cached state during that protected window.

When the lease is eventually cleaned up, the cache refresh brings the Legacy Service back in sync with the database.

This is an important distinction: the TTL isn't what provides consistency by itself. The consistency comes from the lease preventing conflicting access; the TTL and reaper provide failure recovery when the normal release path doesn't happen.

The Tradeoff

The solution wasn't free.

The main cost was latency.

If a resource has many related records that need to be refreshed, releasing the lease and synchronizing the cache can take noticeable time and I verified this through tests.

So I didn't view this as a permanent architecture.

It's a migration bridge.

During the migration, migrated endpoints use the lease mechanism to safely coordinate with the Legacy Service.

As more functionality moves to the New Service, individual resources eventually become fully owned by the New Service.

At that point, the Legacy Service no longer needs to read or modify those resources.

That's our exit condition.

Once a resource is completely owned by the New Service, we can remove the lease and cache-synchronization logic for it.

This is important because migration code has a tendency to become permanent if nobody defines when it can be deleted. So I documented this as well.

What I Learned

1. Start with invariants, not technologies

My first instinct was to think about distributed locks and infrastructure such as Redis.

But the more useful question was:

What must never happen?

In this case, the important invariant was that the Legacy Service must not perform conflicting operations against a resource while the New Service is modifying it.

Once that was clear, the technology choice became much easier.

2. Existing systems often contain useful primitives

At first, I thought of the Legacy Service as something we couldn't modify and therefore mostly had to work around.

But "don't modify the legacy system" doesn't mean "don't understand the legacy system."

By digging into its existing APIs and lease lifecycle, I found a mechanism that already provided most of the coordination and recovery behavior we needed.

Sometimes the best solution isn't introducing another system. It's discovering what the current system can already do.

3. Temporary migration code needs an exit condition

A migration introduces a lot of code that ideally won't exist forever.

The lease wrapper is useful while ownership is split between the Legacy and New Services. But once the New Service completely owns a resource, keeping the wrapper would only add unnecessary latency and complexity.

Defining the exact conditions under which the lease logic can be removed makes it much less likely that temporary migration infrastructure becomes permanent technical debt.

Final Thoughts

This experience changed how I think about solving distributed systems problems.

It's easy to reach for familiar tools like Redis for locking, Kafka for asynchronous communication, an outbox for reliable delivery.

Sometimes those are exactly the right tools.

But before introducing another component, it's worth understanding the guarantees and primitives that already exist in the system.

In this case, the legacy system already had a resource-level lease, automatic expiration, a background reaper, and targeted cache synchronization.

The challenge wasn't building a new distributed system.

The best solution wasn't to build another system around the legacy one. It was to understand the system we already had well enough to make it work for us.

Calling a Service Inside prisma.$transaction Doesn't Make It Atomic — Here's the Bug That Taught Me

Mon, 12 Oct 2026 09:22:43 +0000

I was fixing a real bug in a delivery backend I'm building: a customer's order could get created in Postgres even when something downstream failed, leaving an order with no event history behind it. The fix looked simple. Wrap the whole thing in
prisma.$transaction:

const order = await this.prisma.$transaction(async (tx) => {
  const newOrder = await tx.order.create({ data: { ... } });

  await this.eventLogService.recordEvent({
    entityType: 'ORDER',
    entityId: newOrder.id,
    eventType: EVENTS.ORDER.CREATED,
    payload: { orderId: newOrder.id, status: OrderStatus.PENDING_VENDOR_CONFIRMATION },
  });

  return newOrder;
});

I ran it, broke the event-logging step on purpose to test the rollback, and the order still got committed. The transaction did nothing.

Why

$transaction(async (tx) => {...}) opens one database connection and starts a transaction on it. The tx object you get inside the callback is bound to that specific connection. Every query run through tx participates in the same transaction.

My EventLogService.recordEvent method didn't use tx. It used this.prisma — its own injected Prisma client, a completely different connection:

async recordEvent(params) {
  const event = await this.prisma.eventLog.create({ ... }); // not tx — its own connection
  ...
}

Nesting the call inside $transaction changes nothing here. Prisma has no way to know "we're inside a transaction right now, please route all queries through it." The only thing that determines whether a query participates in a transaction is which client object actually executed it. Since recordEvent ran on its own connection, its insert committed instantly and independently, with no awareness that anything else was "inside" a transaction around it.

The fix
Every method that needs to be transaction-capable has to accept an optional transaction client and fall back to the default when none is passed:

async recordEvent(params, tx?: Prisma.TransactionClient) {
  const client = tx ?? this.prisma;
  const event = await client.eventLog.create({ data: params });
  ...
}

Then the caller passes its own tx through explicitly:

await this.eventLogService.recordEvent(params, tx);

Now recordEvent's insert runs on the same connection, inside the same transaction, as everything else. If anything later in that transaction throws, Postgres rolls back the whole connection's uncommitted work — including this insert — because it genuinely was part of the same unit of work, not just nested inside it syntactically.

I retested the same failure scenario. This time, the order didn't get created either. Rollback worked as a single unit, order and event together.

The part worth remembering
Code structure and transaction membership are two different things. A function call being lexically inside a $transaction block tells you nothing about which database connection it actually used. If you're composing transactional logic across services in Prisma — or any ORM with a similar explicit-client model — every service in that chain needs to accept and thread through the transaction client deliberately. There's no implicit "ambient transaction" you can rely on.

The honest cost: this makes service signatures a bit noisier — an extra optional parameter on every method that might need to participate in a caller's transaction. I think that's a fair trade for not silently losing atomicity on exactly the kind of multi-step write where you'd assume you had it.

The Reliability Boundary: What AI Agents Need Before They Touch Production APIs

Sun, 11 Oct 2026 08:43:40 +0000

AI agents are moving from chat interfaces into real business workflows. They can search customer records, update accounts, trigger deployments, and initiate financial operations. That creates a practical engineering question: what should happen between an agent's decision and a production API call?

In a conventional backend, a request passes through authentication, authorization, validation, rate limits, transaction boundaries, observability, and error handling. Agents need those controls too. They can also misunderstand intent, follow malicious instructions embedded in retrieved content, retry an operation after a timeout, or select the wrong tool.

The answer is a reliability boundary: a layer that treats the agent's output as a proposal and makes the final decision through deterministic policy and backend services.

Separate reasoning from execution

An early architecture often looks like this:

User → AI agent → MCP tool → Production API

That path assumes the agent can decide when and how a business action should happen. For sensitive operations, use a more explicit path:

User request
    ↓
AI agent proposes an action
    ↓
MCP tool or tool gateway
    ↓
Policy, risk, and reliability boundary
    ↓
Validated command → Production API
    ↓
Audit events and reconciliation

The boundary checks identity, permissions, business rules, duplicate requests, state transitions, and the eventual outcome. This is familiar territory for payment and distributed systems: a syntactically valid request is only the start of the decision.

Why an API gateway is not enough

An API gateway can authenticate a caller, route traffic, enforce rate limits, and record access. It usually cannot determine whether an agent misunderstood the user's instruction, whether a retrieved document contained prompt injection, whether a selected tool fits the task, or whether the requested change is consistent with the current business state.

Those checks require application policy. The backend that owns the data or action must remain the final authority.

A production lesson from banking modernization

At UST, I worked on a Capital One modernization effort as SAS-based operational and business reporting was being moved toward Salesforce. The people using those reports were service agents, not AI agents. My part of the work included designing Spring Boot GET APIs for fraud/security, Section 78, and exception reporting, integrating the services with Snowflake, and exposing the endpoints to the Salesforce team.

The production question was broader than whether an endpoint returned the right JSON. A service agent had to see only the reports allowed by their role; the API had to apply that rule even when Salesforce was the caller. We worked with architects and product owners on the service design, used role-based access and business rules, and tested the contracts and behavior. JMeter and Gatling performance tests helped us check whether the services could meet SLA expectations under load. Docker, Kubernetes, and the DevOps workflow made deployment and operation part of the engineering responsibility, not an afterthought.

This was a read-oriented integration, not an AI agent making autonomous changes. Still, it taught a useful boundary lesson: moving a capability from one platform to another does not transfer the authority to decide who may use it. If an AI assistant were later allowed to retrieve one of these reports, its tool call would need to carry the human user's identity and scope. The backend would still enforce role access, validate the request, and record the outcome. An AI tool is another caller of a business capability; it is not a substitute for the controls around that capability.

A refund example

Imagine a customer tells an assistant, “Please refund my last order.” A basic agent may find the latest order and call a refund endpoint. A production workflow needs more answers:

Which customer is authenticated, and which order do they mean?
Is the order eligible under the current policy?
Has a refund already been requested or completed?
Is the amount within the caller's authority?
Does this amount or account require human approval?
If a tool call times out, how will the system discover its actual state?

A language model can interpret the request and prepare a proposed command. It should not make these policy decisions on its own.

Six controls for sensitive agent actions

1. Identity and narrow authorization

Do not give an agent a broad administrative token. Resolve the authenticated user's identity and grant the tool only the capability required for this workflow. Reading an order and issuing a refund are different permissions:

orders.read
refunds.preview
refunds.request
refunds.approve

Enforce authorization again in the backend. An MCP server is an integration layer; its presence does not remove the need for service-level authorization.

2. Structured commands and validation

Do not pass unrestricted prose to a state-changing API. Define a typed command with required fields and constraints:

public record RefundCommand(
    String customerId,
    String orderId,
    BigDecimal amount,
    String currency,
    String reason,
    String idempotencyKey
) {}

The agent may suggest initial arguments. The backend must check the authenticated identity, ownership, amount, currency, eligibility, and current state before accepting the command. Schema validation catches malformed inputs; business validation catches unsafe actions.

3. Idempotency and duplicate protection

Agents retry after timeouts, uncertain tool results, and orchestration failures. A retry must not create a second refund. Assign a stable idempotency key to the business operation, and persist the result associated with it. If the same operation is requested again, return the existing result.

First request: refund-order-7842 → REFUND_ACCEPTED
Retry:         refund-order-7842 → same recorded result

The key must remain stable across retries. Generating a fresh key on every attempt defeats duplicate protection.

4. Risk-based human approval

Approval should follow a policy, rather than being added after an incident. For example, reading an order may proceed automatically; a refund preview may be automatic after validation; a high-value refund or a change to bank details may require a person to review the exact payload.

The agent can prepare the action and explain it. Policy decides what is allowed. The human approves actions above the risk threshold. The backend validates and executes the final command.

5. Auditability

A production workflow should leave more than a chat transcript. Record the authenticated actor, workflow and tool identifiers, input arguments, policy decision, approval, correlation ID, idempotency key, backend response, and final business outcome. Exclude secrets and unnecessary personal data from logs.

{
  "eventType": "REFUND_REQUESTED",
  "workflowId": "wf-8271",
  "actor": "user-442",
  "resource": "order-7842",
  "policyDecision": "APPROVED",
  "idempotencyKey": "refund-order-7842",
  "correlationId": "corr-91a7"
}

These records support incident investigation, customer disputes, and reconciliation.

6. Reconciliation

A timeout does not prove failure. The backend may have accepted the action while the agent missed the response. Before retrying, look up the operation by its stable identifier and ask the system of record for its actual state.

What is the current state of refund-order-7842?

The final business outcome should come from the system of record, not the model's interpretation of a partial tool response.

Where MCP fits

Model Context Protocol (MCP) offers a standard interface for exposing tools and resources to AI applications. That can simplify integration, but it does not replace identity management, authorization, input validation, transaction handling, audit logging, approval, or reconciliation.

Expose controlled business capabilities, such as preview_refund(order_id, reason). Avoid tools that give an agent unrestricted database queries or administrative operations. Tool descriptions help an agent choose an action; backend policy determines whether the action may run.

Reference architecture

User request
   │
   ▼
AI agent ── proposes tool call ──► MCP tool server
                                      │
                                      ▼
                               Policy and risk engine
                                      │
                       ┌──────────────┴──────────────┐
                       ▼                             ▼
                Human approval                 Automatic path
                       └──────────────┬──────────────┘
                                      ▼
                          Validation + idempotency
                                      │
                                      ▼
                              Backend API
                                      │
                                      ▼
                          Audit + reconciliation

The key component is the policy and reliability layer between the model and the business operation. It keeps the model from becoming the final authority over sensitive actions.

What teams should measure

Response accuracy is only one measure of agent quality. Also track invalid tool calls, policy rejections, duplicate prevention, retry frequency, approval rates, time to reconcile uncertain operations, audit completeness, and recovery time after failures.

An agent that answers fluently but cannot explain or reconcile its actions is not ready for production.

Conclusion

The most useful enterprise agents will have bounded, observable, and recoverable autonomy. They can move quickly while the system verifies identity, permissions, business state, and outcomes. MCP helps connect agents to tools; a reliability boundary makes those connections safe enough for real work.

References

Half-Open Circuit Breaker Probes in Serverless Workers

Sun, 11 Oct 2026 02:44:43 +0000

Managing dependency recovery in serverless architectures requires coordinating multiple concurrent invocations to prevent a thundering-herd problem when a service heals. Quick solution: Coordinate the half-open lease through a single SQLite-backed Durable Object per dependency, keep the network probe outside the transaction, and defer competing callers.

async function acquireProbeLease(storage: DurableObjectStorage, now: number): Promise<boolean> {
  return await storage.transaction(async (txn) => {
    const state = await txn.get("breaker_state");
    if (state && state.status === "open" && state.cooldownUntil > now) {
      return false;
    }
    const lease = await txn.get("probe_lease");
    if (lease && lease.expiresAt > now) {
      return false;
    }
    await txn.put("probe_lease", { expiresAt: now + 5000 });
    return true;
  });
}

A circuit breaker protects downstream dependencies by failing fast while an upstream service is unhealthy. The recovery window introduces a new risk when independent request handlers attempt recovery simultaneously. This race condition happens whenever multiple concurrent invocations share a dependency and coordinate through shared state. A scheduled invocation runs at its configured time rather than automatically fanning out into multiple triggers, but a sudden influx of traffic can create the same competitive pressure.

Understanding the Circuit Breaker States

A reliable serverless circuit breaker relies on three distinct operational states:

Closed: normal traffic passes directly to a healthy dependency.
Open: calls fail fast or defer while the dependency remains unhealthy.
Half-open: after a cooldown period, the system admits a controlled probe instead of immediately restoring all traffic.

A successful probe closes the circuit breaker. A failed or timed-out probe reopens it and schedules another recovery attempt. A cooldown period represents a probe opportunity rather than a guarantee of full recovery.

Choosing a Coordination Boundary

Choosing the right coordination backend is essential for safe probe admission. Eventually consistent storage layers cannot provide the atomic guarantees required for single-probe coordination.

Option	When to Use It	Trade-off or Failure Mode	Recommendation
Workers KV	Global read-heavy configurations	Eventually consistent, lacks atomic read-modify-write	Do not use for transactional lease locks.
SQLite-backed Durable Object	Single-region state consistency and transactions	Bound to a single location, requires routing	Recommended for single coordination points.
External Relational Database	Existing enterprise database clusters	Adds network latency to serverless invocations	Avoid if serverless cold-start and latency matter.

A SQLite-backed Durable Object provides strong consistency and transactional guarantees for a single coordination point. Use a stable Durable Object identity for each protected dependency to manage lease admission in a short transaction.

Managing Recovery and Failure Handling

Leases must include an expiration timestamp so a crashed or timed-out claimant does not permanently block recovery. Callers that do not receive the lease should return a retryable or deferred outcome rather than spinning in a tight retry loop.

On a successful probe, close the breaker and clear the active lease. On a failure or timeout, reopen the breaker with an appropriate delay and jitter. For further reading on operational safety mechanisms, see Self-Expiring Kill Switches and Human-Readable Alerts.

Verification Checklist

Issue five simultaneous lease requests against the Durable Object and verify that exactly one caller receives permission to probe.
Confirm that expired leases allow a new claimant to acquire the probe slot after the timeout window elapses.
Verify that late probe results from an expired lease do not overwrite newer breaker states in storage.
Check that callers without a lease receive a deferred response rather than executing unauthorized network traffic.

Testing LLM Fallback Tiers Before Outages

Sat, 10 Oct 2026 09:03:31 +0000

When a primary language model provider rejects requests due to location or transient overloads, many teams rely on an in-platform fallback tier. Logging response bodies and separating deterministic blocks from transient failures solves the diagnosis layer, but it leaves a critical question unanswered. A fallback that never runs in production is merely a hope. If you do not test the backup path continuously, your first time seeing it fire will be during an outage, which is the worst possible time to discover a configuration bug.

Running a reliable secondary tier requires more than basic routing rules. You need a reliable warming pattern that does not inflate latency or cost, a strict response-shape contract so consumers never know which provider answered, and a scheduled drill cadence with a clear readiness metric. This guide covers how to implement these mechanisms in a serverless environment without breaking your budget.

Fractional Shadow Traffic With Diff Logging

To ensure a fallback model is ready, you need to send it realistic payloads. However, routing user traffic blindly to an unproven secondary provider risks increased latency and unexpected error rates. The solution is fractional shadow traffic.

In a serverless worker, you can mirror a small, sampled percentage of live requests to the fallback tier asynchronously. By using asynchronous execution primitives like waitUntil, the shadow request fires in the background without affecting the user-visible critical path. Both the primary response and the shadow response are logged, and their normalized shapes are compared.

export async function handleRequest(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
  const primaryResponse = await callPrimaryProvider(request, env);
  
  // Sample 2 percent of traffic for shadow warming
  if (Math.random() < 0.02) {
    ctx.waitUntil(runShadowFallback(request, primaryResponse, env));
  }
  
  return primaryResponse;
}

async function runShadowFallback(request: Request, primary: Response, env: Env): Promise<void> {
  try {
    const fallbackRaw = await callFallbackProvider(request, env);
    const diff = compareResponseShapes(primary, fallbackRaw);
    if (diff.hasMismatch) {
      await logShapeMismatch(env, diff);
    }
  } catch (error) {
    await logShadowError(env, error);
  }
}

This approach gives you continuous coverage. If the fallback provider alters its error format or response structure, your diff logs catch the mismatch on a quiet Tuesday rather than during a traffic spike.

Response-Shape Adapter Contracts

Different language model providers return unique JSON envelopes, usage metadata, and finish reasons. If your application code directly handles provider-specific payloads, switching to a fallback tier requires branching your business logic everywhere.

To eliminate this complexity, introduce a response-shape adapter. Neither provider's raw envelope should ever reach your core application. Instead, every provider output passes through a normalization layer that maps data to a single internal structure containing only the generated text, token counts, finish reason, and standardized error classes.

interface NormalizedResponse {
  text: string;
  inputTokens: number;
  outputTokens: number;
  finishReason: 'stop' | 'length' | 'error';
  errorClass?: string;
}

function adaptProviderResponse(raw: any, provider: string): NormalizedResponse {
  if (provider === 'primary') {
    return {
      text: raw.candidates[0].output,
      inputTokens: raw.usage.prompt_tokens,
      outputTokens: raw.usage.completion_tokens,
      finishReason: raw.candidates[0].finish_reason === 'MAX_TOKENS' ? 'length' : 'stop'
    };
  } else {
    return {
      text: raw.choices[0].message.content,
      inputTokens: raw.usage.input_tokens,
      outputTokens: raw.usage.output_tokens,
      finishReason: raw.choices[0].finish_reason
    };
  }
}

By forcing both primary and fallback responses through this adapter, failover becomes a simple routing decision. The rest of your application remains entirely agnostic of which provider fulfilled the prompt.

Scheduled Failover Drills and Readiness Metrics

Asynchronous shadow traffic proves that responses match, but it does not verify that your routing logic actually triggers when the primary provider fails. For that, you need scheduled failover drills.

A cron-triggered background worker can force a small batch of synthetic probe traffic through the fallback tier for a bounded window. The pass bar for these drills is binary. Every probe must succeed, and every response must conform to the normalized shape contract. If a single probe fails, the drill fails. For a related implementation, see Bounded Llm Fallback Chains.

To give the on-call engineer immediate confidence during an incident, combine your telemetry into a single readiness score. This score factors in three underlying data series:

Shadow success rate over the last twenty-four hours.
Shape-conformance rate from differential logging.
Days elapsed since the last successful green drill.

If the drills are skipped or if the shadow diff rate degrades, the readiness score drops automatically. Your incident runbook should require the on-call engineer to verify that this readiness score meets a minimum threshold before manually forcing a failover switch, removing guesswork from high-stress moments. For a related implementation, see Audit Macos System Data Before Deleting.

Bounding Cost and Complexity

Warming a fallback tier and running regular synthetic drills consumes billable API requests. Without cost controls, your reliability engineering efforts can inadvertently trigger a billing incident.

Keep shadow traffic tightly sampled, typically between one and five percent of total production volume. Cap the number of synthetic probes executed during weekly drills to the minimum necessary for statistical confidence. By bounding these operations and scheduling them during off-peak windows, you maintain a continuously verified backup system while keeping expenses entirely predictable.

⚡️ Caching Strategies for Web Apps: From Browser to Backend

Sat, 10 Oct 2026 03:33:24 +0000

Caching is one of the most powerful ways to improve application performance—reducing load times, server costs, and user frustration. Yet many developers treat caching as a “black box” or skip it due to fear of cache-related bugs.

In this blog, we will:
✅ Demystify caching: what it is and why it matters
✅ Explore frontend (browser) caching strategies
✅ Dive into backend and distributed caching with Go examples
✅ Cover Cache-Control headers, ETag, stale-while-revalidate
✅ Discuss common pitfalls and best practices

By the end, you’ll know how to confidently implement caching in your React frontend and Go backend to build scalable, fast, and efficient applications.

🚀 What is Caching?

Caching means storing data temporarily so future requests can be served faster without recomputing or refetching the same data repeatedly.

Think of it like preparing tea and storing it in a flask—you don’t need to reboil water each time you want to drink.

📈 Why Caching Matters

Speed: Return data faster to users.
Reduce Server Load: Avoid redundant computation or DB queries.
Improve Reliability: Serve cached responses when backend services are down.
Reduce Costs: Less compute = lower cloud bills.

🧩 Caching in the Frontend (Browser)

1️⃣ Browser Cache (HTTP Caching)

Browsers automatically cache assets (JS, CSS, images) and API responses based on HTTP headers.

Key Headers:

Cache-Control: Defines how long and under what conditions the browser can reuse the response.
ETag: Helps validate cached data; server sends a hash, and browser sends it back to check if the data changed.
Expires: Sets a hard expiration time for cache.
stale-while-revalidate: Allows serving stale content while revalidating in the background for a seamless UX.

Example:

Cache-Control: public, max-age=3600, stale-while-revalidate=600

✅ Use Case: Cache API GET responses for non-user-specific data (e.g., blog posts, product catalogs).

2️⃣ Client-Side Data Fetching Libraries

Libraries like React Query, SWR provide built-in caching for API calls.

Cache API responses in memory.
Background revalidation for fresh data.
Avoids unnecessary re-fetching on component re-mounts.

✅ Use for per-session data caching in React apps.

🗂️ Caching in the Backend

1️⃣ In-Memory Caching

Store frequently accessed data in memory for fast retrieval.

Examples: Golang’s sync.Map, groupcache.
Fastest but limited by server RAM and is not shared across instances.

✅ Use for small, frequently requested data (e.g., config, auth tokens).

2️⃣ Distributed Caching

Tools like Redis and Memcached provide a shared cache layer accessible across multiple servers.

Benefits:

Centralized cache for load-balanced apps.
Supports expiration policies.
Can handle large datasets.

🔹 Go Example: Using Redis for Caching

1. Install Redis client:

go get github.com/go-redis/redis/v8

2. Basic Usage:

rdb := redis.NewClient(&redis.Options{
    Addr: "localhost:6379",
})

ctx := context.Background()

// Set cache
rdb.Set(ctx, "product_123", jsonData, time.Hour)

// Get cache
val, err := rdb.Get(ctx, "product_123").Result()
if err == redis.Nil {
    // Cache miss, fetch from DB
} else {
    // Cache hit, use val
}

⚖️ Cache Invalidation: The Hard Part

“There are only two hard things in Computer Science: cache invalidation and naming things.” – Phil Karlton

Whenever the underlying data changes, you need to ensure the cache is updated to avoid stale data.

Strategies:

✅ Time-based expiration (TTL):
Set a cache expiry time so data auto-expires after a fixed duration, ensuring old data clears automatically.

✅ Manual invalidation:
Manually clear or update cache entries whenever the underlying data changes to keep data fresh.

✅ Write-through caching:
Write data to both the cache and the database at the same time, ensuring consistency during reads.

✅ Cache busting:
Change cache keys (e.g., v1:product_123 → v2:product_123) when your data structure changes to avoid serving stale or incompatible data.

✅ Best Practices for Caching

Cache only GET requests; avoid caching POST unless explicitly designed.
Set reasonable TTLs to avoid stale data.
Monitor cache hit/miss rates to tune cache policies.
Be cautious caching user-specific or sensitive data.
Use cache versioning (v1:product_123) to manage structure changes.
Document what is cached, where, and why.

⚠️ Common Pitfalls

🚫 Caching sensitive user data publicly.
🚫 Forgetting to invalidate cache on data updates.
🚫 Over-caching dynamic endpoints, leading to stale or incorrect UI.

🔮 Future of Caching: CDN & Edge Caching

Content Delivery Networks (CDNs) like Cloudflare, AWS CloudFront, Vercel Edge cache data closer to the user.

Benefits:
✅ Lower latency.
✅ Offload origin server traffic.
✅ Built-in stale-while-revalidate strategies.

Using edge caching in modern React + Go apps can drastically improve performance globally.

🎯 Final Thoughts

Caching isn’t just an optimization; it’s essential for building scalable, reliable, and fast applications.

Whether it’s browser caching, React Query in the frontend, or Redis in the backend, knowing when and how to cache makes you a stronger developer.

💬 Have you implemented caching in your apps? What caching challenges have you faced? Let’s discuss in the comments!

Nightly Analyst Digest from Slot Activity Logs

Sat, 10 Oct 2026 03:03:48 +0000

Running a high-frequency automation system that triggers dozens of times per day creates a difficult reporting dilemma. If every run sends an immediate notification, your inbox fills with noise. If you turn off notifications entirely, regressions go unnoticed until a user complains. The natural solution is a nightly digest, but building one reliably on serverless infrastructure introduces subtle failure modes. A naive implementation often results in missing reports when the underlying language model fails, misaligned days due to UTC offset handling, or raw log dumps that require too much manual effort to parse.

This article examines how to build a reliable scheduled digest email from activity log data. You will learn how to design a logging contract specifically for downstream aggregation, handle timezone boundaries without drifting, implement a graceful fallback when the analyst model is unavailable, and keep recommendations strictly advisory.

Sixty Emails versus One Digest

When a system executes tasks tens of times a day, individual alerts quickly lose their value. Developers stop reading them, and important failures hide inside a flood of routine success notices. A single daily email solves this fatigue by aggregating metrics, engagement statistics, and notable events into one readable summary. For a related implementation, see Email Account Safety Developers.

However, building this digest introduces three architectural challenges:

Data Availability: Slots that only execute tasks do not inherently preserve the context needed for a narrative summary.
Fragile Dependencies: If the nightly summary relies entirely on a third-party language model to generate text, an API outage will prevent the email from sending entirely.
Timezone Drift: Standardizing cron jobs on UTC while the human reader operates in a local time zone splits calendar days incorrectly, producing reports that mix evening events from the wrong day.

Solving these issues requires treating the nightly digest not as an afterthought, but as a core pipeline with its own data contract and error boundaries.

Logging for the Analyst

To produce a meaningful summary, individual execution slots must leave behind structured data designed for aggregation rather than raw debugging. Instead of unstructured console logs, every slot should append a compact JSON line to a date-keyed store with a multi-day time-to-live (TTL). For a related implementation, see Debugging Silent Skips In Poll Based.

Below is an example of a structured log entry recorded by a worker slot:

{
  "timestamp": "2023-10-25T08:30:00Z",
  "slot_id": "hourly-sync-04",
  "status": "success",
  "model": "gpt-4o-mini",
  "provider": "openai",
  "material_count": 12,
  "reply_count": 3
}

Writing these entries rides on existing key-value storage writes, avoiding the overhead of a dedicated database. To prevent timezone ambiguity, the storage key incorporates the audience local date directly rather than relying on raw UTC timestamps. For example, a key format of digest:2023-10-25 ensures all runs belonging to that calendar day in the target region are grouped together, regardless of UTC shift.

The Degraded Digest Pattern

Relying on a language model to synthesize raw activity logs into human-readable insights introduces a single point of failure. If the provider experiences downtime or rate limits at the cron trigger time, the entire digest fails to send, leaving the team blind.

To make the system resilient, the analyst call must be treated as best-effort. If the model call fails or times out, the service must catch the error, fall back to displaying the computed raw statistics block directly, and label the email accordingly.

Here is a conceptual TypeScript implementation of the execution handler with fallback logic:

async function generateDigestReport(rawLogs: LogEntry[]): Promise<DigestResult> {
  const stats = computeAggregates(rawLogs);
  
  try {
    const analysis = await callAnalystModel(stats);
    return {
      mode: "ai-enhanced",
      summary: analysis.text,
      recommendations: analysis.recommendations,
      stats
    };
  } catch (error) {
    return {
      mode: "degraded-raw-stats",
      summary: "Analyst model unavailable. Displaying raw computed metrics.",
      recommendations: [],
      stats
    };
  }
}

This pattern ensures that an upstream outage degrades the quality of the formatting rather than breaking delivery. The recipient still receives their morning metrics on schedule.

Questions, Not Actions

Another risk in automated reporting is allowing the analyst model to modify system behavior based on its own findings. If an LLM decides that a prompt or execution cadence should change and automatically applies that change, the reporting pipeline transforms into an unsupervised and potentially destructive deployment system.

To maintain safety, recommendations generated by the digest must be phrased strictly as questions for human review. For instance, rather than updating configuration files, the output should suggest adjustments like "Consider increasing the timeout limit for morning sync runs?" or "Cadence for slot B generated high error rates; review prompt constraints." An explicit human decision remains required to apply any configuration change.

Summary of Architectural Patterns

Building a dependable nightly analyst digest requires more than wiring a cron trigger to an AI model. By designing a dedicated logging contract, anchoring date keys to the audience timezone, implementing a non-LLM fallback for the summary step, and keeping recommendations advisory, you can achieve the best of both worlds. You get the readability of an AI-assisted narrative without sacrificing the reliability of traditional serverless infrastructure.

Debugging Silent Skips in Poll-Based Reply Bots

Fri, 09 Oct 2026 21:01:58 +0000

When a poll-based reply bot runs in a staging environment, it usually behaves predictably. It reads recent posts, checks for mentions, and generates answers. In production, however, a user report reveals that a valid reply sat unanswered for hours while every execution log reported complete success with zero errors. The telemetry shows nothing unusual because the system never encountered an unhandled exception. It simply decided that there was nothing to process. For a related implementation, see Viewmodel Partial Success Refresh Failure.

This silent failure mode typically stems from two intertwined design choices: bare continue statements that swallow item-level decisions and fixed lookback windows that slide past older content as new posts arrive. Understanding why these patterns cause silent failures helps make background workers transparent and robust.

The Silent Sieve of Bare Continue Statements

Most poll loops rely on a sequence of guards to decide whether an incoming post requires a response. These guards check if the bot already replied, whether the post originates from an authorized user, or if the content matches specific criteria. When a guard decides to skip an item, the standard implementation often uses a bare control flow statement.

for item in fetched_items:
    if already_replied(item):
        continue
    if not is_author_valid(item):
        continue
    process_reply(item)

This pattern creates an observability blind spot. Every skipped item vanishes into the loop structure. The function completes successfully, returns an empty result, and logs an ordinary run completion. To the monitoring system, a run that skipped every available item looks identical to a run where no items needed attention.

Replacing bare control flow with structured skip tracking changes the diagnostic profile of the worker. Instead of hiding the decision, each guard can record why an item was passed over.

skipped = []
for item in fetched_items:
    if already_replied(item):
        skipped.append({"id": item["id"], "reason": "already_replied"})
        continue
    if not is_author_valid(item):
        skipped.append({"id": item["id"], "reason": "invalid_author"})
        continue
    process_reply(item)

Returning this structured list alongside successful actions transforms log-diving into direct inspection. When an item remains unanswered, the output explicitly states whether the system saw it and why it chose to ignore it.

The Sliding Window Horizon Problem

Beyond guard logic, the query window itself often introduces silent omissions. To avoid scanning an entire history, bots typically request only the N most recent posts. If a bot reads the five most recent posts every ten minutes, it operates under the assumption that all relevant interactions happen within that horizon.

This assumption breaks down when post velocity increases. If a burst of activity pushes an unanswered mention beyond the fifth position, the sliding window moves past it. The post remains active on the platform, but the polling mechanism no longer includes it in the fetch payload. Manual testing often masks this issue because running a manual test posts fresh content to the timeline, pushing the neglected item even further out of bounds.

Fixing this requires sizing the lookback window against actual content velocity rather than relying on a static magic constant. If human replies typically arrive within a multi-hour window, the fetch limit must scale to cover that timeframe based on the average rate of incoming posts. Bounding the window remains necessary to control API usage, but the boundary should reflect operational reality instead of an arbitrary small number.

Testing the Negative

Because these bugs manifest as silence rather than crashes, standard test suites often fail to catch them. A test that only asserts on successful replies will pass even if the bot is silently skipping every valid input.

Comprehensive test coverage for a poll-based worker must assert on the negative state. A proper test verifies that when no work is required, the skip list explicitly populates with expected reasons rather than returning an ambiguous empty response. By asserting on both sent actions and structured skip records, engineers can prove that the worker is actively evaluating items rather than bypassing them.

Resolving Poll-Based Failures

Poll-based reply bots fail quietly when their control flow obscures item-level decisions and their fetch windows are too narrow for the surrounding content velocity. Replacing bare continue statements with structured skip records and tying lookback limits to real-world posting rates turns invisible failures into inspectable state, ensuring that background workers handle every interaction predictably.

Safe Public Demo Environments for Android Apps

Thu, 08 Oct 2026 23:37:04 +0000

Offering a public demo of an Android application presents a unique architectural challenge. A demo build is exposed to strangers and needs frequent, destructive data resets, yet it must never be able to target or corrupt the real customer database by accident. Relying on a simple runtime UI flag or environment toggle inside the production codebase creates an unacceptable risk of accidental data leakage. This article examines how to build a robust isolation boundary using separate build variants, dedicated demo backends, and fail-closed identity validation.

Why a Demo Toggle Fails as an Isolation Boundary

Many development teams initially attempt to support public demonstrations by adding a toggle within the primary application and backend. The logic often checks whether a demo mode flag is active before skipping certain write operations or routing requests to mock data. This approach fails because a single code path or misconfigured deployment can easily bypass the flag. If a build script accidentally packages the production credentials or connects the demo backend to the primary database, a public reset operation can wipe out real customer records.

True isolation requires separating the trust boundary at every layer of the architecture. The application package name, the backend service instance, and the database project reference must be distinct. If any component in the chain does not match the expected demo configuration, the service should refuse to start entirely rather than attempting to recover or fallback.

Separate Android Build Variants and App Identity

The first layer of defense begins in the Android client. Using build variants allows developers to maintain different application IDs, source sets, and configuration values within the same repository. By modifying the application ID for the demo variant, the operating system treats the demo build and the production build as entirely separate apps on the test device.

android {
    defaultConfig {
        applicationId "com.example.app"
    }
    buildTypes {
        release {
            minifyEnabled true
        }
    }
    flavorDimensions "mode"
    productFlavors {
        production {
            dimension "mode"
            applicationIdSuffix ""
        }
        demo {
            dimension "mode"
            applicationIdSuffix ".demo"
        }
    }
}

This configuration ensures that installation artifacts are distinct. More importantly, keeping the choice fixed in the build artifact prevents users or automated scripts from accidentally activating a demo mode inside a production binary.

Verify Backend and Database Identity

Client-side separation alone is insufficient if the backend can connect to any database via environment variables. A dedicated demo backend must require explicit configuration flags and perform strict startup validation. During initialization, the backend should compare the database project reference derived from its active connection string against an expected demo project reference.

If the strings do not match, the application must immediately throw an unhandled exception and shut down. This fail-closed mechanism ensures that a server misconfiguration becomes immediately visible during deployment rather than remaining a latent vulnerability.

Seed and Reset Guarded Disposable Data

Public demo environments require a mechanism to restore clean state after users alter or delete records. However, this reset mechanism must be guarded against accidental execution against live systems. Implementation evidence suggests a two-step validation model for data lifecycle management:

Initial seeding should only proceed if every demo table is entirely empty. If existing records are detected, the seed routine halts.
Subsequent resets require a persisted security sentinel string to match the configured sentinel, and the database identity must be re-verified immediately before any deletion takes place.

Furthermore, developers should strip all non-essential integrations from the demo service. Production push notification credentials, third-party payment hooks, and external spreadsheet integrations should be omitted so that a compromised or heavily tested demo environment cannot trigger external side effects.

What Process-Local Locks Do Not Protect

When implementing automated reset endpoints, developers frequently reach for concurrency primitives such as a language-specific mutex to prevent overlapping reset requests. While a process-local mutex successfully serializes concurrent coroutines or threads within a single backend instance, it offers no protection in a horizontally scaled architecture. For a related implementation, see Managing Concurrent Git Commits During Automated.

If the demo backend is deployed behind a load balancer with multiple active instances, each instance maintains its own memory space and its own mutex. Two distinct server processes could execute a database reset simultaneously. Protecting a multi-instance demo environment requires database-level locking, a dedicated single-instance worker queue, or another distributed coordination mechanism. For a related implementation, see Safe Multi Environment Database Orchestration.

Release Checklist for a Public Demo

Before launching a public demo environment, verify the following architectural controls:

The demo application variant uses a distinct application ID suffix.
The backend enforces explicit demo mode flags and fails startup if the database project reference does not match.
Initial data seeding refuses to run unless all target tables are empty.
Destructive reset operations require a valid persisted sentinel check and a pre-execution database identity re-validation.
Session secrets and sentinels meet minimum length requirements.
Non-essential production integrations and credentials are completely removed from the demo service deployment.

Self-Expiring Kill Switches and Human-Readable Alerts

Thu, 08 Oct 2026 22:16:58 +0000

When an unattended cron system encounters a total provider failure, engineers often reach for a manual kill switch. While effective at stopping runaway requests, the manual-only reset pattern introduces hidden operational costs. If an outage lasts twenty minutes, the system stays dark until an operator manually intervenes. When multiple transient interruptions occur within a single day, teams find themselves repeatedly waking up or context-switching to clear flags for zero new information. Furthermore, when these systems fail, they frequently dump raw JSON status codes into email inboxes. Over time, operators learn to ignore these notifications entirely, missing the genuine incidents hidden beneath the noise. For a related implementation, see Ai Agent Handoff Fallback Context.

Designing resilient background workflows requires solving two distinct problems. First, the system needs an automated mechanism to halt execution during widespread failures without trapping the infrastructure in a permanent locked state. Second, notifications must translate low-level errors into actionable prose so that operators can assess the situation at a glance. For a related implementation, see Fixing Malformed Json Errors In Ci.

The Problem with Manual-Only Resets

Traditional architectural patterns treat a kill switch as a binary toggle. An operator sets a flag, the background jobs stop, and someone must later run a command or update a database record to resume execution. In practice, this creates a mismatch between system recovery and human intervention. Providers frequently recover on their own within minutes. If the kill switch lacks an expiration mechanism, the recovery is bottlenecked by human availability. The operational burden shifts from fixing the infrastructure to administrative housekeeping.

Designing the Expiring Kill Flag

To balance safety with automation, the kill flag can be upgraded from a simple boolean to a timestamped window. When a total provider failure is detected, the system writes a kill key containing the exact engagement time. Subsequent execution cycles check this key before running. If the current time is within a bounded window, such as three hours, execution is suppressed.

Once the timestamp exceeds the expiration threshold, the system automatically clears the flag and resumes normal operations. This ensures that the infrastructure never remains silenced indefinitely due to an forgotten flag.

Security and failure modes require strict handling. If the kill key value is unreadable, malformed, or corrupted, the system must fail closed rather than assuming everything is fine. Expiry logic applies exclusively to well-formed timestamps.

{
  "kill_switch": {
    "status": "active",
    "engaged_at": "2023-10-25T14:00:00Z",
    "expires_at": "2023-10-25T17:00:00Z"
  }
}

When a cron worker evaluates this structure, it compares the current clock against the expiration boundary:

def is_kill_switch_active(flag_data):
    if not flag_data or "expires_at" not in flag_data:
        # Fail closed on corrupted or missing flags
        return True
    
    current_time = get_current_utc_time()
    if current_time < flag_data["expires_at"]:
        return True
        
    return False

Writing Alerts Humans Actually Read

Infrastructure notifications often fail because they optimize for machine parsing rather than human comprehension. Sending raw JSON dumps or raw status codes trains engineers to filter out alerts. A robust fallback reporting system replaces machine payloads with deterministic prose templates.

Instead of dispatching stack traces, the notification service maps known error shapes to human causes. For instance, a connection timeout combined with a specific gateway response gets translated into a clear sentence explaining that the upstream provider is experiencing degraded performance.

Furthermore, auto-clearing events do not require a dedicated notification. When the three-hour window expires and execution resumes, the transition is recorded as ordinary data in the next scheduled summary digest. This keeps communication channels quiet during self-healing incidents while preserving historical visibility.

Conclusion

Reliable background automation relies on bounding manual intervention and respecting human attention limits. By combining self-expiring timestamps with plain-language fallback reports, engineering teams can stop bleeding during provider outages without trapping their systems in permanent manual locks or drowning in unreadable alerts.

Publishing Threads Carousels from Worker-Hosted Images

Thu, 08 Oct 2026 18:41:04 +0000

When you want to publish multi-image carousels to Threads through its API, every item requires a publicly downloadable image URL. Traditional architectures solve this by uploading generated or processed assets to an object storage service like Amazon S3 or Cloudflare R2 before making API requests. If you are running your application entirely within a serverless worker, adding a separate storage bucket introduces extra infrastructure overhead, credentials management, and recurring billing for assets that an external platform downloads only once. This guide explores how to host carousel images directly inside a Cloudflare Worker using KV storage, bypass traditional object storage, and manage the multi-step publishing pipeline successfully. For a related implementation, see Running A Zero Cost Social Auto.

The Missing Image Problem in Serverless

Serverless workers excel at handling stateless API requests, executing webhook logic, and routing web traffic on the edge. However, they lack a persistent filesystem or a static file server. You cannot simply drop generated images into a public folder inside your worker project because the runtime environment evaluates code on demand rather than serving static assets from disk.

To bridge this gap, you need a mechanism to store binary data temporarily or semi-permanently within your existing serverless ecosystem and expose it via a public URL that Meta can reach. While external buckets are the default recommendation, small-scale or ephemeral workloads often do not justify the added configuration overhead.

KV as a One-Download CDN

Cloudflare KV allows you to store key-value pairs at the edge. By treating KV as a rudimentary object store, you can write binary image bytes directly into a key and serve them through a custom worker route such as /img/<asset-name>.

When storing images in KV, keep in mind the platform limits regarding value sizes. For typical compressed JPEGs or WebP thumbnails, individual images easily fit well within the free-tier value cap. When a request hits your public route, the worker fetches the binary payload from KV and responds with the appropriate MIME type and caching headers.

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);
    if (url.pathname.startsWith('/img/')) {
      const imageName = url.pathname.split('/')[2];
      const imageBuffer = await env.IMAGE_KV.get(imageName, 'arrayBuffer');
      
      if (!imageBuffer) {
        return new Response('Not found', { status: 404 });
      }
      
      return new Response(imageBuffer, {
        headers: {
          'Content-Type': 'image/jpeg',
          'Cache-Control': 'public, max-age=86400'
        }
      });
    }
    return new Response('Endpoint not found', { status: 404 });
  }
};

Before initiating any interactions with the Threads API, verify your route manually. Run a curl command to ensure the endpoint returns an HTTP 200 status, the correct content type header, and a non-zero byte count. If your server cannot download the image reliably, Meta servers will fail as well.

The Carousel Pipeline and Processing Wait

Publishing a carousel on Threads is not a single API call. It requires a strict sequence of container creation steps, server-side processing, and final publication. Skipping the required waiting period between container creation and publishing is the most frequent point of failure.

The publishing workflow consists of four distinct phases:

Item Containers: Create an individual media container for each image in your carousel, passing the public worker URL in the request parameters.
Processing Gap: Wait for Meta to download, validate, and process each image server-side. Publishing immediately after container creation results in processing errors because the remote servers have not finished ingesting the assets.
Carousel Parent: Create a parent container that references the child item container IDs.
Publishing: Trigger the final publish action on the parent container to push the carousel live to your profile.

Because status field names and readiness indicators can vary across API versions, implementing a reliable time-based buffer of roughly sixty seconds after container creation provides a pragmatic safety margin before executing the final publish request.

Managing Generative Visual Quality

When your workflow involves generating visuals programmatically before publishing them in a social media carousel, text rendering remains an ongoing challenge. Modern image generation models frequently attempt to render unintended words, letters, or gibberish characters into visuals unless explicitly instructed otherwise.

Even when prompt instructions explicitly state to avoid text entirely, stochastic models occasionally leak artifacts. Incorporating a human review gate into your publishing loop ensures that every generated asset is visually inspected before it enters a carousel container. Automated pipelines should handle byte storage, container creation, and API orchestration, while human verification safeguards final quality.

Summary of Worker-Hosted Carousel Publishing

By leveraging Cloudflare KV as a lightweight storage layer and enforcing a strict processing delay between container creation steps, you can publish Threads carousels entirely from within a serverless worker. This approach removes the need for external object storage buckets while maintaining full control over asset delivery and visual quality.

Preview Cross-System Data Changes Before Sync

Thu, 08 Oct 2026 12:04:12 +0000

A sync button can hide a destructive decision. When a spreadsheet and a backend database disagree because rows were added, removed, edited, duplicated, or marked for deletion, starting an automatic write without checking the shape of those differences risks corrupting data. Users need to know whether the selected source of truth is safe before any changes are applied.

To solve this, a preflight check must calculate a read-only diff first, require one explicit source of truth, block ambiguous keys, and verify the sync by recalculating the diff afterward. This guide explores how to build a reliable preflight check for cross-system data integration using aggregate difference counts and directional guards.

Why a Sync Button Needs a Preflight

Automatic synchronization assumes that both sides agree on identifiers and state. In practice, networks fail, users edit spreadsheets offline, and backend records are updated concurrently. If an application blindly pushes or pulls without an inspection phase, it can overwrite valid updates or propagate duplicate keys.

A preflight stage separates observation from mutation. By inspecting both sides without writing, the system computes an aggregate preview of the state gap. This gives operators visibility into what will change before they commit to an action.

Stable Keys and Row Signatures

Comparing data across systems requires reliable identification. Row positions in a spreadsheet are notoriously fragile because sorting or inserting a row shifts every subsequent index. Integrations must rely on stable business keys rather than physical row numbers.

In addition to stable keys, systems need canonical signatures to detect when shared rows have changed. A signature can be a cryptographic hash or a concatenated string of critical fields. If the key matches between Google Sheets and the database, but the signature differs, the record has been modified.

{
  "stable_key": "user_10293",
  "sheets_signature": "a1b2c3d4",
  "db_signature": "e5f6g7h8",
  "status": "changed"
}

Classifying Missing, Changed, Duplicate, and Deleted Rows

A comprehensive preview categorizes discrepancies into distinct buckets so users can assess the scope of a sync operation. The underlying service should evaluate several entity groups:

Keys that exist only in the spreadsheet.
Keys that exist only in the database.
Shared keys whose canonical signatures differ.
Duplicate keys appearing on either side.
Pending deletions flagged by business logic.

This implementation evidence relies on aggregate counts rather than a field-by-field diff. The preview informs the user how many records are affected, but it does not attempt to display every mutated attribute in a complex UI.

Choosing a Single Source of Truth

When systems diverge, automatic two-way merging often fails because conflict resolution requires domain context that automated scripts lack. If duplicate keys or changed common rows are present, a two-way mode becomes ambiguous and should be rejected.

Instead, the integration must force an explicit choice. The user or operator selects either the spreadsheet or the application database as the authoritative source of truth for that run. Once an authority is chosen, the sync operation flows in a single direction, eliminating merge conflicts.

Confirming Safely and Detecting Stale Previews

A safe run requires an active preview and refuses concurrent execution. If another sync is already running, or if the preview has not been generated, the system blocks the write request.

Process memory often holds these preview objects for simplicity. However, keeping previews in memory introduces a race condition if the underlying data changes between the time the preview is generated and the time the user clicks confirm. A production integration must expire previews after a short duration or bind them to source fingerprints, rejecting stale plans to prevent unintended overwrites.

Post-Sync Convergence Checks

Applying a sync direction is only half the workflow. To ensure the operation succeeded, the service must immediately recompute the differences after the write completes.

If the new preview shows zero discrepancies, the system reports full convergence. If differences remain due to validation errors or partial failures, the service reports a partial result rather than assuming success.

Conclusion

Cross-system synchronization requires discipline to prevent data loss. By inspecting changes in a read-only preflight, relying on stable keys, blocking ambiguous states, and verifying convergence after execution, applications can bridge external tools like spreadsheets and internal databases safely.

Measuring Scheduled Job Reliability

Thu, 08 Oct 2026 09:04:08 +0000

A scheduled workflow can start successfully, finish with no eligible work, or commit an article that is not yet live. Counting all three as successes turns a reliability percentage into a trigger counter. Long-running and manually triggered executions also distort the denominator unless they are labeled separately. The useful question is not simply whether the scheduler fired. It is whether the promised user-visible result became available, and how often that happened across scheduled opportunities.

Defining the Denominator and Success Event

To calculate an honest reliability rate, you must first define what constitutes an opportunity. If a cron trigger fires every hour, every one of those scheduled slots is a denominator entry. A slot where the system finds no work to process is still an opportunity. Omitting empty slots from the denominator quietly inflates your success percentage. For a related implementation, see Viewmodel Partial Success Refresh Failure.

Next, tie success to the final user-visible outcome rather than an intermediate API response. If a serverless function initiates a workflow that commits data to a database, success requires the record to be queryable at its canonical public endpoint. If the database write succeeds but a subsequent webhook notification fails, the core publishing action still succeeded. Separating core delivery from secondary notifications prevents false failures from dragging down your reliability metric. For a related implementation, see Designing A Serverless Ai Publishing Workflow.

Handling Manual Triggers and Dry Runs

Operational workflows often require manual intervention, testing, or dry runs. These paths must be isolated from automated metrics:

Automated scheduled runs populate the primary reliability denominator.
Manual live runs are tracked separately to measure operator-initiated actions without polluting automated cron statistics.
Dry runs are excluded entirely because they make no production attempt and produce no live outcome.

Failing to separate these paths leads to misleading reports where a heavy day of manual testing masks an underlying cron failure or vice versa.

Idempotent Ledger Schema and State Tracking

Using a versioned SQLite database like Cloudflare D1 allows you to record execution states reliably. Each scheduled slot receives a deterministic, idempotent run identifier. When the scheduler triggers, the application writes an initial RUNNING marker before starting any asynchronous work. If starting the workflow fails outright, that same row is finalized as a miss. A marker that remains open past a bounded timeout is also classified as a failed run.

Below is an example schema migration demonstrating how to store these execution states while keeping records compact and sanitized.

CREATE TABLE job_reliability_ledger (
    run_id TEXT PRIMARY KEY,
    scheduled_time TEXT NOT NULL,
    execution_type TEXT NOT NULL,
    status TEXT NOT NULL,
    blocker_category TEXT,
    created_at TEXT DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_scheduled_time ON job_reliability_ledger(scheduled_time);

When writing updates, use an upsert strategy or check the existing status to ensure replaying a terminal update does not create duplicate rows or alter a verified success timestamp.

Operational Reporting Considerations

When displaying reliability metrics to an engineering team, context matters as much as the raw percentage. Always state the data-collection start date next to every rate so partial history is not mistaken for lifetime performance. Calculate summaries using the audience local timezone rather than forcing UTC conversion on operations teams. Store only sanitized blocker categories, such as timeout or validation error, instead of raw provider replies or secrets.

Conclusion on Measuring Reliability

Accurate reliability measurement requires discipline around what you count as an attempt and what you verify as a success. By separating scheduled opportunities from manual tests, anchoring your denominator to every cron slot, and verifying real user-visible outcomes, you build an operational view that reflects actual system health rather than optimistic trigger counts.

📊 Can Kafka Handle Every Integration Use Case? Understanding the Architectural Trade-offs

Wed, 07 Oct 2026 13:39:23 +0000

Compared with traditional queue-based messaging platforms, Kafka is primarily designed for event streaming. It excels at distributing data from producers to one or many consumers at massive scale. However, unlike many traditional messaging platforms, Kafka intentionally minimizes broker-side routing and content-based filtering capabilities. Message selection is typically achieved through topic design, partitioning strategies, stream processing, or consumer-side filtering rather than broker-managed routing.

This raises an important architectural question:

❓ Can Kafka be used for all integration and messaging use cases?

The answer is yes, but organizations should understand and address several architectural and operational challenges before adopting Kafka as a universal messaging platform.

📈 1. Scaling Considerations

Kafka's distributed architecture allows horizontal scaling by adding additional brokers to support increasing workloads and client connections. However, consumer scalability is fundamentally tied to the number of partitions within a topic.

Key considerations:

✅ One partition can be consumed by only one consumer within the same consumer group.
✅ One consumer can consume multiple partitions.

⚠️ The maximum parallelism of a consumer group is limited by the partition count.

As a result, partitioning strategy becomes a critical design decision. Under-partitioning can limit future scalability, while over-partitioning may introduce unnecessary operational overhead, metadata overhead, replication traffic, and recovery times, making partition planning a long-term architectural decision.

Partition planning should be treated as a long-term architectural decision, with future scalability requirements carefully considered during initial design.

🛡️ 2. Reliability and Data Durability

Kafka provides flexible durability options through replication and acknowledgment settings.

Publishers can choose different acknowledgment modes, balancing performance against durability requirements.

This flexibility offers advantages, but it also introduces governance challenges:

⚠️ Different applications may adopt different reliability settings.
⚠️ Organizations need governance and standards to ensure mission-critical applications meet reliability requirements.

In addition, replication significantly reduces the risk of data loss, but durability guarantees ultimately depend on replication factors, acknowledgment settings, in-sync replica requirements, and storage configuration. Kafka's design prioritizes minimizing the probability of data loss while maintaining high throughput. Kafka achieves durable storage through persistent logs and replication. While Kafka leverages the operating system page cache to maximize throughput, durability ultimately depends on replication, acknowledgment settings, storage configuration, and broker availability. These mechanisms provide strong durability characteristics, although architects should carefully evaluate configurations based on business requirements.

Architects should therefore carefully assess durability requirements and select appropriate replication, acknowledgment, and storage configurations.

🔄 3. Consumer Group Rebalancing

Consumer groups provide scalability and high availability, but they come with an operational cost.

Whenever a consumer joins, leaves, fails, or recovers within a consumer group, Kafka may initiate a rebalance process.

During rebalancing:

Partition ownership is reassigned
Consumer-to-partition mappings are recalculated.
Message consumption may be temporarily paused.

For dynamic cloud-native applications that frequently scale in and out, excessive consumer group rebalancing can disrupt message processing and increase latency. By default, Kafka does not guarantee partition ownership stability during rebalances. To minimize disruption, consider using the Cooperative Sticky Assignor or Static Membership.

⚙️ 4. Group Coordinator Bottlenecks

Kafka's data plane is highly scalable and typically handles large volumes of producers and consumers efficiently.

However, consumer groups rely on a control-plane component known as the Group Coordinator.

The Group Coordinator is responsible for:

📌 Managing group membership
📌 Handling consumer rebalancing
📌 Processing heartbeats
📌 Managing offset commits and retrievals
📌 Enforcing group protocols and coordination logic

In very large deployments with many consumer groups and frequent rebalances, Group Coordinator load can become an important operational consideration. When many consumer groups rebalance simultaneously, Group Coordinator load may increase significantly, potentially leading to longer rebalance times and higher consumer recovery latency.

🔀 5. Message Distribution Models

Traditional message brokers and Kafka use fundamentally different message distribution approaches.

Message Bus Model

✅ Multiple consumers can subscribe to the same non-exclusive queue.
✅ The broker distributes messages among available consumers.
✅ Consumer scaling can be automated using tools such as Kubernetes Event-Driven Autoscaling (KEDA) based on queue depth.

Kafka Model

✅ Messages are distributed across partitions.
✅ Each partition is processed by only one consumer within a consumer group.
✅ Scaling is achieved through adding partitions and consumers.

However:

⚠️ Message distribution occurs at the partition level.
⚠️ New messages continue to be sent to assigned partitions regardless of individual consumer performance.

If one consumer becomes significantly slower than others, backlog accumulation can occur on specific partitions, resulting in uneven processing rates across the consumer group.

Kafka has introduced the Share Group model (KIP-932), which aims to provide queue-like work-distribution semantics similar to traditional non-exclusive queues. However, the feature is still evolving and has not yet reached the same maturity level as established enterprise messaging platforms.

🌐 6. Network Bandwidth Consumption

One of the key differences between Kafka and many traditional message buses is the absence of native broker-side routing and filtering.

In many messaging platforms, consumers receive only messages that match specific routing rules.

With Kafka:

📥 Consumers typically subscribe to topics.
📥 Kafka favors topic-based distribution. As consumer requirements diverge, additional topics, stream processing, or filtering may be needed, increasing operational overhead.
📥 When consumer requirements diverge significantly, organizations may need additional topics, stream processing, or consumer-side filtering, potentially increasing operational complexity and network traffic.

For environments with large event volumes and diverse consumer requirements, this approach may increase:

📡 Network bandwidth usage
🖥️ Consumer processing overhead

Increasing the number of Kafka topics is one way to address bandwidth and scalability challenges. However, this approach can significantly increase operational overhead and application complexity. Architects should carefully evaluate topic design and event segmentation strategies to minimize unnecessary data movement and avoid excessive topic proliferation.

📋7. Message Ordering vs Scalability

One of Kafka's key strengths is its ability to preserve message order, but this comes with an important architectural trade-off.

Kafka guarantees message ordering only within a partition.

To increase processing throughput and consumer parallelism, organizations often increase the number of partitions. However, as partitions increase, maintaining a single global order across all messages becomes more difficult.

📦 8. Topic and Metadata Management

In very large Kafka deployments, metadata size and frequent metadata updates can increase client startup time, subscription management overhead, and cluster coordination activity. While Kafka is designed to scale efficiently, excessive topic and partition growth can create additional operational and management challenges.

🎯 Final Thoughts

Kafka is an exceptional event-streaming platform and has become a popular foundation for modern event-driven architecture. However, "one platform for every use case" is not always the optimal architectural approach.

The key question is not whether Kafka can support a use case, but whether it is the most appropriate tool for that use case.

Successful architecture decisions require balancing:

📈 Scalability
🛡️ Reliability
⚙️ Operational Complexity
🔄 Consumer Distribution Requirements
🌐 Network Efficiency

High-scale Kafka environments require strong operational observability. Consumer lag, partition imbalance, replication health, broker utilization, and rebalancing behavior must be continuously monitored to ensure reliable service operation.

However, Kafka is intentionally designed with relatively simple architecture to deliver high-performance event streaming at scale. As a result, it does not provide the rich routing and message filtering capabilities commonly found in traditional message buses.

Understanding these trade-offs enables architects to select the right messaging paradigm for the right business problem rather than forcing every workload into a single technology pattern.

💡 Kafka can support most integration workloads, but its architectural model introduces trade-offs that architects must understand. The most successful implementations are those that align Kafka's strengths with business requirements, rather than assuming that a streaming platform is the optimal solution for every messaging scenario.

Fixing KV Propagation Delay in Media Publishing

Wed, 07 Oct 2026 06:02:44 +0000

When you build a serverless media pipeline that stores generated images in a key-value store and immediately sends those public URLs to a third-party publishing API, you will likely encounter a frustrating failure mode. Your local tests pass, your terminal curl requests return the correct image bytes with a 200 OK status, yet the publishing API responds with a cryptic media not found error. This happens because your own worker reads from the exact edge location that just wrote the key, while the platform fetcher hits a different edge node that has not received the replicated data yet. For a related implementation, see Viewmodel Partial Success Refresh Failure.

Green Locally, Dead Remotely

The most deceptive part of this issue is its misleading diagnostic feedback. When publishing fails, the natural instinct is to examine the container ordering, parameter payloads, or authentication tokens. Developers often curl the generated public URL from their own workstation or test runner, receive the correct MIME type and image bytes, and conclude that the serving layer works fine. This self-verification is a trap. Your edge succeeding proves only that one specific node has the data. It provides zero visibility into whether the global key-value fabric has synchronized across all regional datacenters.

Whose Edge Is It Anyway

Global key-value stores prioritize low-latency writes and eventual consistency. When your worker executes a put operation, the runtime confirms success once the local write succeeds. Replication to other regions happens asynchronously in the background. If a third-party platform attempts to download the asset immediately after receiving the URL, its request routes to an edge server near its own infrastructure. If that specific server has not received the update, the fetch fails. Because distributed platforms do not expose their internal replication state or routing tables, you cannot poll their edge directly to check availability. You must instead build a defensive timing model into your application code.

Two Waits, Two Handoffs

Solving this problem requires separating your publishing pipeline into distinct phases, each guarded by its own deliberate delay. Conflating these phases leads to brittle workarounds.

The Propagation Wait: After storing the asset in key-value storage, introduce a measured pause to allow global replication. This gives remote edge locations time to catch up before any external system attempts a download.
The Processing Wait: After the third-party platform initiates its import or container creation process, allow an additional window for the platform to process the file before you invoke the final publish command.

Hardcoding these delays into your core logic makes your test suite painfully slow. Instead, make your wait durations configurable by injecting milliseconds during testing while using production-appropriate values for live deployments. Keeping these constants clearly named in your source code prevents future maintainers from accidentally optimizing them away.

Implementing the Fix

To manage these handoffs safely, structure your asynchronous worker code to wait explicitly between storage and publishing steps. Here is an example configuration pattern in TypeScript that uses injected wait durations:

interface PublishConfig {
  propagationDelayMs: number;
  processingDelayMs: number;
  maxRetries: number;
}

async function storeAndPublishMedia(
  kv: KVNamespace,
  key: string,
  data: ArrayBuffer,
  config: PublishConfig
): Promise<string> {
  await kv.put(key, data);
  
  // Guard against global edge replication lag
  await new Promise((resolve) => setTimeout(resolve, config.propagationDelayMs));
  
  const mediaId = await initiatePlatformUpload(key);
  
  // Guard against remote processing lag
  await new Promise((resolve) => setTimeout(resolve, config.processingDelayMs));
  
  return await finalizePublish(mediaId);
}

Verifying the Publish Object

Never rely on your own curl command as proof of a successful publish operation. True success is defined solely by the response of the downstream publishing API. When the platform successfully fetches the asset from its local edge, processes it, and returns a valid media object with attached children or confirmation IDs, your pipeline has succeeded. If the call fails, adjust your propagation budget upward or inspect the platform error logs for download timeouts.

Final Thoughts

Eventual consistency models introduce hidden timing assumptions into modern serverless architectures. When your application hands off assets to external consumers, assuming instant global visibility will cause intermittent failures. By acknowledging edge propagation realities, separating your storage waits from your processing waits, and verifying platform objects rather than local URLs, you can make your media publishing pipelines resilient to distributed latency.

Building a High-Speed School Marketplace (Part 2): Multi-Role Security with Supabase RLS

Wed, 07 Oct 2026 04:15:37 +0000

This is Part 2 of our technical series on building a high-speed, lightweight school marketplace. In Part 1: \[Why We Ditched Frameworks for Vanilla JS & PWA,\]1 we discussed how we optimized the frontend for budget mobile devices. Today, we look at the backend architecture.

When planning the architecture for Aksara-Mart—our digital marketplace for vocational school canteens and cooperatives—we faced a fundamental infrastructure question:

Do we really need to build, maintain, and scale a traditional custom backend server (like Node.js/Express, Python, or Go) just to shuttle data between the browser and the database?
For a school environment, maintaining a traditional backend comes with real trade-offs:
1.Infrastructure & Hosting Overhead: A dedicated server needs 24/7 uptime monitoring, memory management, and scaling configuration for peak recess traffic.
2.Maintenance Burden: API routes, middleware controllers, and ORM abstractions require continuous patching and maintenance.
3.Unnecessary Latency: Every simple query (browsing the lunch menu, checking an order status) has to make an extra hop through an intermediate API server layer.
We wanted an architecture that could minimize backend server workload while keeping running costs close to zero.
That led us to Supabase. By allowing our frontend JavaScript to communicate directly with PostgreSQL via auto-generated APIs and WebSockets, we could offload the bulk of routine read queries directly to the client.
However, letting client-side JavaScript query a database directly sounds like a security nightmare—unless you have a bulletproof authorization layer built into the database itself.
That is where PostgreSQL Row Level Security (RLS) becomes the cornerstone of the entire system.

The Architecture: Direct Client Queries Without the Middleman
In a traditional web application, every user request passes through three layers:
```
[ Browser (Client) ] ──▶ [ Custom API Server (Express/Node) ] ──▶ [ Database ]
```
The API server spends most of its CPU cycles doing mundane tasks: authenticating tokens, verifying if a user is allowed to read a row, running the query, and formatting JSON.
With Supabase and PostgreSQL RLS, we simplified the architecture into a direct, high-performance two-tier model:
```
[ Browser (Vanilla JS Client) ] ──────────────▶ [ Supabase PostgreSQL + RLS ]
•Direct catalog SELECT                         - Kernel-level row authorization
•Direct order history lookup                   - Instant query evaluation (< 3ms)
•Real-time WebSocket subscriptions             - Atomic RPC stored procedures
```
The Advantages:
•Zero Backend Server Maintenance: No intermediate server processes to crash, restart, or run out of memory during a rush.
•Minimal Compute Overhead: PostgreSQL handles filtering and JSON serialization natively in C, which is vastly faster than parsing objects in Node.js.
•Direct Client Queries: Frontend JavaScript can fetch exactly the columns it needs, when it needs them, directly from the client.
The Core Problem: Securing Direct Database Access
If frontend JavaScript talks directly to the database using a public API key, what prevents a student from querying other users' private orders or inspecting canteen cost margins?
This is where Row Level Security (RLS) comes in.
Instead of writing authorization logic in backend middleware (if (req.user.id !== order.user_id) return res.status(403)), the authorization rules are compiled directly into PostgreSQL table definitions.
•Without RLS: A client query supabase.from('orders').select('*') returns every order in the database.
•With RLS: PostgreSQL automatically injects an internal filter into the query plan:
```
-- Automatically enforced by PostgreSQL on every query
WHERE email_customers = auth.jwt() ->> 'email'
```
Even if a user inspects the network tab and writes a custom script asking for all records, the database engine guarantees they only receive the rows they are explicitly permitted to see.
Managing Roles Across the School Ecosystem
A school marketplace serves several distinct groups with different permissions:
| Role | Access Scope | How Queries Are Handled |
| :--- | :--- | :--- |
| siswa (Students) | Browse active products, view own order history. | Direct client SELECT constrained by RLS. |
| guru (Teachers) | Same as students, with priority fulfillment flags. | Direct client SELECT constrained by RLS. |
| kasir (Cashiers) | View all active incoming orders, print receipts. | Direct client SELECT with staff privileges; no direct price/stock editing. |
| manajer (Managers) | Manage catalog items, update stock, view revenue. | Full CRUD permissions on product tables via RLS. |
| admin (System Admins) | System-wide configuration and role management. | Full administrative access. |
The admin_role Table
To keep role checks clean and reliable, we maintain a dedicated role table in PostgreSQL:
```
CREATE TABLE public.admin_role (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email TEXT UNIQUE NOT NULL,
role TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'active'
);
ALTER TABLE public.admin_role ENABLE ROW LEVEL SECURITY;
```
Practical Gotchas We Addressed:
4.Case-Insensitive Emails: Mobile keyboards often capitalize the first letter of an email (*Emails are not allowed*). Always use lower(trim(email)) in policies to prevent login lockouts.
5.Dual-Spelling Tolerance: In bilingual environments, staff may enter roles in English or Indonesian (manager vs manajer, cashier vs kasir). We check role aliases with IN ('manager', 'manajer', 'cashier', 'kasir') so minor spelling differences don't break authentication.
The Separation of Concerns: Direct Reads vs Server-Side Writes
Allowing frontend JavaScript to query the database directly works brilliantly for reads, but transactional writes require a different boundary.
A. Direct Client Reads (Safe via RLS)
Frontend JavaScript directly reads from the database for:
•The product catalog:
```
// Direct client query in skripmart.js
const { data: products } = await supabase
.from('product_list')
.select('id, name, price, stock, image_url')
.eq('status', 'active');
```
RLS ensures that wholesale costs (modal_price) and archived items are filtered out before data leaves PostgreSQL.
•Order tracking:
```
// Customers only receive their own orders
const { data: myOrders } = await supabase
.from('orders')
.select('*')
.order('created_at', { ascending: false });
```
B. Controlled Server-Side Writes (Via Database RPC)
While reading data directly from JavaScript is safe, creating orders and modifying stock should never happen via direct client INSERT or UPDATE:
```
// ❌ ANTI-PATTERN: Never allow direct client inserts for financial transactions
await supabase.from('orders').insert({
total_price: clientCalculatedTotal, // Client could tamper with this value!
items: cartItems
});
```
If we allowed direct client inserts on the orders table, a user could theoretically alter prices or bypass stock limits.
Instead, we follow a simple architectural rule:
Frontend JavaScript reads data directly using RLS.
Frontend JavaScript writes transactions via PostgreSQL Stored Procedures (RPC).
When a student checks out, the client invokes an RPC procedure (v2_create_order):
```
// ✅ SECURE: Transaction logic runs atomically inside PostgreSQL
const { data, error } = await supabase.rpc('v2_create_order', {
p_customer_name: userProfile.name,
p_delivery_type: deliveryOption,
p_items: cartPayload
});
```
Inside this procedure, the database recalculates prices from the official catalog, checks remaining stock, decrements inventory, and creates the order records in a single atomic transaction.
What We Gained from This Architecture
By offloading query processing directly to the client and database engine:
6.Zero Backend Maintenance: We do not maintain or pay for a custom Node.js/Express server layer.
7.Sub-3ms Authorization Overhead: RLS policy checks happen natively within PostgreSQL's query optimizer with virtually no latency penalty.
8.High Reliability During Traffic Spikes: When hundreds of students browse the menu simultaneously during recess, Supabase and Cloudflare serve the requests directly without choking an intermediate web server.
9.Complete Data Privacy: Every customer and staff member only receives data that belongs to their assigned role.

What's Next in Part 3?
With direct reads secured by RLS and client-side queries running smoothly, we still had to solve one critical concurrency problem:
During the 15-minute school break, dozens of students click "Checkout" at the exact same millisecond for the last few items in the canteen.
How do we prevent race conditions and overselling without a heavy application server?
In Part 3: Atomic Checkouts & Zero Overselling with PostgreSQL RPC, we will break down how we implemented pessimistic row locking (FOR UPDATE) inside PostgreSQL to handle high-concurrency checkout spikes in under 42 milliseconds!

Bounded LLM Fallback Chains

Wed, 07 Oct 2026 03:03:52 +0000

When a primary AI provider experiences an outage or a rate limit in the middle of the night, applications often suffer from unexpected downtime. The common reaction is to implement a multi-provider failover chain. However, standard fallback loops create a different kind of emergency: runaway billing events. Unbounded retries across multiple models can exhaust available budgets in minutes during a regional outage.

To prevent failover mechanisms from becoming cost incidents, systems need strict boundaries. A production-ready fallback tier must exhaust the primary provider completely before switching, enforce a strict per-run call cap, treat billing errors as permanent stop signs, and exclude fallbacks from routine dry runs.

The Cost of Unbounded Retries

Chaining multiple language model providers without constraints introduces several hidden failure modes. Without a defined architecture, an outage triggers cascading problems.

Unbounded failover burns capital rapidly. Each fallback call consumes tokens or credits. If a system cascades through four expensive models without a cap, a single failing background job can drain a monthly budget. Furthermore, applications often retry quota-exceeded and billing-restricted errors. These responses are permanent verdicts from the provider, not transient network glitches. Retrying them only multiplies the failure rate and increases error logs.

Another subtle issue involves testing and dry runs. If health checks or dry runs execute the entire fallback path, they consume the scarce fallback quota that production systems rely on during actual emergencies. Finally, silent provider drift occurs when a primary model fails permanently, but the system quietly runs on the expensive fallback tier for weeks without alerting the engineering team.

Designing the Provider Tier Contract

To solve these challenges, an LLM routing layer must enforce specific operational constraints.

First, the system must exhaust tier one completely before initiating tier two. Fallback models should never run in parallel or preemptively. Second, total fallback calls per execution run require a hard cap, such as a maximum of four attempts. This bounds the worst-case financial exposure regardless of how many models exist in the registry. For a related implementation, see Audit Macos System Data Before Deleting.

Third, error classification dictates the next step. Infrastructure issues like 503 service unavailable or gateway timeouts advance the chain. Conversely, quota exhaustion, invalid API keys, and paid-only model restrictions act as immediate stop signs. The runner halts execution and records the exact reason rather than proceeding to the next provider.

Fourth, dry runs must execute against the primary path only. Fallback bindings remain dormant during routine checks unless an explicit integration test flag is provided. Fifth, every successful response must record the provider and model name in the output metadata. This attribution makes silent fallback immediately visible in monitoring dashboards.

Implementing a Bounded Fallback Runner

Below is a TypeScript example demonstrating a bounded fallback executor with error classification and call limits.

type ProviderResult = {
  content: string;
  provider: string;
  model: string;
};

enum ErrorType {
  Transient,
  PermanentBilling,
}

interface Provider {
  name: string;
  model: string;
  execute(prompt: string): Promise<string>;
}

function classifyError(err: any): ErrorType {
  if (err.status === 429 || err.code === 'QUOTA_EXCEEDED') {
    return ErrorType.PermanentBilling;
  }
  return ErrorType.Transient;
}

async.executeWithFallback = async function(
  primary: Provider,
  fallbacks: Provider[],
  prompt: string,
  maxFallbackCalls: number = 2,

): Promise<ProviderResult> {
  try {
    const content = await primary.execute(prompt);
    return { content, provider: primary.name, model: primary.model };
  } catch (primaryError) {
    if (classifyError(primaryError) === ErrorType.PermanentBilling) {
      throw new Error('Primary failed with billing error; halting chain.');
    }

    let callsMade = 0;
    for (const fb of fallbacks) {
      if (callsMade >= maxFallbackCalls) {
        break;
      }
      callsMade++;

      try {
        const content = await fb.execute(prompt);
        return { content, provider: fb.name, model: fb.model };
      } catch (fbError) {
        if (classifyError(fbError) === ErrorType.PermanentBilling) {
          throw new Error('Fallback failed with billing error; halting chain.');
        }
      }
    }

    throw new Error('All providers exhausted without successful execution.');
  }
};

Verifying Fallback Behavior Safely

Proving that a fallback chain works without triggering a real production outage requires targeted unit and integration tests. Tests should simulate a complete primary failure and assert that the fallback tier engages exactly once. Additional test cases must verify that dry runs never invoke the fallback binding unless explicitly enabled, that hitting the per-run call cap throws an exception instead of executing unauthorized calls, and that billing-class errors halt the chain immediately with a documented stop reason.

Architectural Conclusion

Reliable AI infrastructure requires treating failover as a finite budget rather than an open-ended loop. By classifying provider errors, enforcing hard call limits, and maintaining strict attribution, engineering teams can build resilient multi-provider runtimes that withstand outages without risking financial exposure.

Backend Architecture Starts With Workflows, Not Frameworks

Tue, 06 Oct 2026 14:30:00 +0000

When teams begin designing a backend, technology discussions often arrive first.
Node.js or Python?
PostgreSQL or MongoDB?
Monolith or microservices?

Those choices matter, but the architecture usually becomes clearer if you start somewhere else:
the workflow.

Follow One Operation From Beginning To End

Suppose an e-commerce customer places an order.

The backend may need to:

Authenticate the customer
Check inventory
Validate pricing
Create an order
Contact the payment provider
Update inventory
Trigger fulfillment
Send confirmation

The architecture question is not simply:
Which framework should execute this?

A better question is:
Which steps must succeed together, which can happen later, and what happens when one fails?

That immediately exposes important design requirements.

Define Data Ownership

Modern applications frequently use multiple external platforms.

Customer information might appear in:

The main application database
A CRM
A payment platform
An analytics system
A support platform

Without explicit ownership rules, synchronization becomes unpredictable.
Backend design should identify which system owns each important data type and how changes move between systems.

Design Integration Failure Paths

Third-party APIs will eventually:

Time out
Reject requests
Apply rate limits
Return unexpected responses
Experience downtime

A production backend needs a strategy for those situations.
That may include retries, queues, idempotency, fallback behavior, error logging, or manual recovery.
The exact solution depends on the workflow, but ignoring the failure path is rarely a good solution.

Treat Authorization As Architecture

Authentication tells us who made the request.
Authorization decides what that person can do.
As applications grow, the second problem often becomes harder.

A SaaS application, for example, may have organization owners, administrators, managers, ordinary members, billing users, and support staff.

If permission rules are embedded randomly throughout controllers and endpoints, they become difficult to reason about.

Defining authorization boundaries early makes the rest of the system easier to maintain.

Choose Architecture Based On Operational Needs

Microservices are useful when independently managed services genuinely solve an organizational or operational problem.

They are not automatically an upgrade from a monolith.

A modular application with clearly separated responsibilities can be a strong choice for many products.

Architecture should match:

Product complexity
Team structure
Deployment requirements
Operational overhead
Expected change

Not architectural fashion.

Think About Production Before Production

Backend design should also answer:
How will we know when something is wrong?

Logs, error tracking, metrics, alerts, database visibility, and integration monitoring are part of the system, not extras to think about after launch.

The framework can change.

A poorly defined workflow is much harder to fix.

For teams currently mapping these decisions, backend development services provide a broader view of how APIs, databases, integrations, authentication, architecture, and maintenance fit together.

What do you define first when starting a new backend: domain boundaries, data model, API contracts, or infrastructure?

Reads That Survive the Kill Switch

Mon, 05 Oct 2026 09:02:25 +0000

When an unattended cron worker misbehaves, operators typically engage a kill switch to halt all execution. This stops harmful mutations, but it frequently blinds the operations team as well. While the switch is active, nobody can see whether requests are piling up, and the team discovers the backlog only after the system recovers. This occurs because writes and reads often share a single gate. The disabled flag blocks publishing and the demand check together, incorrectly conflating the instruction to stop acting with the need to observe system state.

A kill switch that stops a misbehaving cron worker also blinds it. While engaged, nobody can see whether customers are waiting, so the team discovers a backlog only after recovery, or never. Writes and reads share one gate, meaning the disabled flag blocks publishing and the demand check together. Silent slots hide demand because a disabled result with no context reads identically whether zero or fifty replies wait. Recovery starts blind when the flag clears, as the first healthy slot discovers the backlog by accident instead of by report. For a related implementation, see Schema First Gates Ai Publishing Pipelines.

The Peek Pattern

To keep visibility intact during an outage, systems need a separation of concerns between looking at the queue and modifying it. The peek pattern addresses this by placing read operations ahead of any mutation gates. The workflow executes three distinct phases:

Token validation: Load credentials first. If authentication fails, the run stops immediately.
The peek: Run the read-only demand check to count pending items without altering state.
Gate evaluation: Check the kill switch status. If active, skip writing while preserving the read data.

The read operation must never throw, write data, or trigger external notifications. Even if the underlying data store experiences intermittent pressure, the worst-case return value should be an empty count rather than a failed execution slot. For a related implementation, see Duplicate Push Notifications Fcm.

Demand on Every Result

A common monitoring flaw is reporting demand only when tasks succeed. When a kill switch is active, disabled slots often return a silent empty response with no context. Operators cannot tell whether zero tasks are waiting or fifty items are sitting in a blocked queue.

The solution is to attach the pending count to every possible execution outcome. Whether a slot is disabled, skipped, failed, or successfully published, the log entry or audit digest should carry the exact pending count. This guarantees that observability logs show continuous pressure trends even when the system is entirely fenced off from writing.

Answering Stays Gated

Visibility into demand must never weaken the security boundary of the kill switch. Observing the queue does not grant permission to process it. Publishing answers and executing outbound calls must remain safely behind every existing guard, including duplicate detection entries, budget caps, and audit requirements.

Consider this simplified TypeScript configuration pattern showing how a worker separates the read peek from the write action:

type RunResult = { status: string; pendingCount: number };

async function handleCronJob(isKillSwitchActive: boolean): Promise<RunResult> {
  const pendingCount = await fetchPendingDemandSafely();
  
  if (isKillSwitchActive) {
    return { status: "fenced", pendingCount };
  }

  await executeWrites();
  return { status: "published", pendingCount };
}

async function fetchPendingDemandSafely(): Promise<number> {
  try {
    return await getQueueDepth();
  } catch {
    return 0;
  }
}

In this model, the fetchPendingDemandSafely function wraps calls in a try-catch block to guarantee it never throws an error that could abort the inspection. The pending count is captured regardless of whether the kill switch is active.

Maintaining Visibility During Outages

Separating read operations from write mutations prevents the blind spots that typically accompany emergency interventions. By ensuring that demand checks execute independently of write gates, and by attaching pending counts to every outcome, engineering teams can size their recovery efforts accurately the moment restrictions are lifted.

Why I Chose a Simple Capability Token for Waitless — and When I Would Harden It

Mon, 05 Oct 2026 07:20:50 +0000

While working on Waitless, I needed customers to be able to track their position in a queue and leave the queue without creating an account.

That led to a small but interesting security design question:

How can I authorize an anonymous user without introducing a full authentication system?

My current solution is deliberately simple: each queue entry receives a random UUID token, and that token is used as a capability for the tracking link.

UUID token instead of a sequential ID

A sequential identifier could look like:

/track/123

The problem is that IDs such as 123, 124, and 125 are predictable and enumerable.

For an anonymous tracking system, I don't want a user to be able to guess another customer's resource simply by changing the ID.

Instead, an entry has a separate token:

model Entry {
  id      String @id @default(uuid())
  token   String @unique @default(uuid())

  name     String
  email    String?
  position Int
  status   EntryStatus @default(WAITING)

  queueId String
  // ...
}

The public URL therefore looks more like:

/track/550e8400-e29b-41d4-a716-446655440000

The important point is that the token is not intended to be an ordinary identifier.

It acts as a capability.

What is a capability URL?

A capability can be thought of as:

An unguessable secret that grants access to a specific resource.

In my case:

possess valid token
        ↓
access this queue entry

The capability is deliberately designed to be difficult to guess.

That is different from simply hiding a sequential ID and hoping nobody discovers it.

This distinction matters:

Security by obscurity
→ "Nobody knows the ID."

Capability-based design
→ "Access requires possession of a deliberately unguessable credential."

The second is a much more intentional security model.

Why I kept the token simple for now

Waitless is an anonymous queueing system.

A customer does not need an account, password, or traditional login just to:

see their position
receive queue updates
leave the queue

Introducing a complete authentication system would add considerable complexity to the product and to the user experience.

For the current use case, a capability link gives me a simple model:

Join queue
    ↓
Generate random token
    ↓
Give customer tracking link
    ↓
Customer presents token
    ↓
Server identifies the queue entry
    ↓
Allow limited actions

This is a conscious trade-off.

I'm choosing simplicity and frictionless anonymous access because the current resource is relatively limited in scope.

But the token is also a bearer credential

There is an important consequence.

If possession of the token grants access, then:

whoever possesses the token can use it.

That makes the token a bearer credential.

For example, the current leave() operation relies on the same tracking capability.

So if the token is leaked, copied, logged, or shared, another person could potentially use it.

That means the design is intentionally simple, but it is not the strongest possible security model.

Known limitation

The tracking token currently also functions as a bearer credential for leave(), with no rate-limiting or one-time consumption.

I consider this an accepted limitation of the current design rather than something I want to hide.

One way to harden the system would be to use the tracking link only as an initial capability.

For example:

/claim/<secret>
      ↓
validate capability
      ↓
create anonymous session
      ↓
HttpOnly + Secure cookie
      ↓
redirect to /track

After that, the token would no longer need to remain in the URL.

The browser would use the session cookie for future requests.

This would reduce some of the risks associated with repeatedly carrying a secret in a URL and would give me more control over session lifecycle, expiration and revocation.

But it introduces another trade-off.

The trade-off: security vs. convenience

The current capability link has an important usability property:

The link itself works across devices.

A customer can open it on their phone, copy it to another device, bookmark it, or reopen it later.

With a one-time capability-to-session exchange, that behavior becomes more complicated.

For example:

Phone
  ↓
claim token
  ↓
session created

Now the same one-time token may no longer be usable on another device.

Supporting multiple devices would therefore require additional identity/session handling, which increases complexity.

For a simple anonymous queue, I decided that this trade-off is not currently justified.

When would this design change?

The answer is: when the product starts having identity requirements.

Imagine Waitless later introduces a reservation system.

Now a customer might have:

Account
   ↓
Reservations
   ↓
Queue entries
   ↓
Payment history
   ↓
Cancellation rules
   ↓
Personal information

The security requirements become much stronger.

At that point, a random tracking capability may no longer be enough as the main authorization mechanism.

I would probably move toward:

User authentication
        ↓
Authenticated session
        ↓
Authorization
        ↓
Reservation / queue access

The capability token could still exist for specific use cases, but it would no longer be responsible for the entire identity model.

UUIDv4 vs. UUIDv7

This also made me think more carefully about what an identifier is supposed to do.

For database identifiers, UUIDv7 is attractive because it is time-ordered and therefore provides better index locality than completely random UUIDv4 values.

For a public capability, however, my primary requirement is different:

I care more about unpredictability than database ordering.

That's why I would keep the concepts separate:

Database ID
    ↓
UUIDv7 can be a good choice

Public capability token
    ↓
Cryptographically random token / UUIDv4

This is an example of a broader engineering principle:

The best identifier for the database is not necessarily the best credential for the client.

What about storing the token itself?

Another improvement would be to store only a hash of the capability:

raw token
    ↓
SHA-256
    ↓
tokenHash

Then the database contains the verifier rather than the original bearer secret.

The application can hash the presented token and compare the result.

This gives additional protection if the database contents are exposed.

Again, it adds some implementation complexity, so it is a hardening step rather than something I needed to introduce simply to make the current Waitless flow work.

My current decision

For the current version of Waitless:

Random capability token
        ↓
Tracking URL
        ↓
Anonymous access
        ↓
Limited actions

I deliberately chose this because it provides a very small and understandable authorization mechanism without forcing customers through registration and login.

The trade-off is that the token is a bearer credential, so the system needs to treat it as sensitive.

I have therefore documented the limitation rather than pretending the design is perfect.

My planned evolution is roughly:

Current
Random capability URL
        ↓
Simple anonymous access


Later
Rate limiting
+ expiry / revocation
+ stronger token handling


When identity becomes important
        ↓
Capability URL
        ↓
Session / authentication
        ↓
Fine-grained authorization

The interesting lesson for me was not simply “use UUIDs.”

It was learning to ask:

What is this value actually supposed to represent?

An identifier, a capability, and an authenticated identity are three different concepts.

Once that distinction is clear, the architecture becomes much easier to reason about.

When Gemini Rejects Cloudflare Workers by Location

Sun, 04 Oct 2026 15:45:20 +0000

When a scheduled worker calls the Gemini API around the clock, an unexpected failure can disrupt pipelines for hours. During an outage, error monitoring dashboards often show a mix of failures that appear to stem from a single overarching problem, yet they require opposite responses. Distinguishing transient overloads from static geo-blocking when using serverless egress IPs is essential to keeping automated workflows running without unnecessary provider migrations or wasted retry cycles. For a related implementation, see Normalize Cloudflare Workflows Trigger Payloads.

Everything Fails at Once

In a recent debugging session with the Morgans provider integration, scheduled workers executing on serverless edges began encountering widespread errors. The dashboard showed a wall of failed requests with status codes that looked like a standard capacity outage. However, closer inspection of the raw error payloads revealed that two entirely distinct failure modes were occurring simultaneously within the same incident window.

Understanding this behavior requires looking at how edge platforms route traffic. Serverless execution environments route requests through rotating egress pools. When an upstream provider updates its regional enforcement rules or experiences capacity shifts, certain egress IPs may trigger geographic restrictions while others continue to pass through normally. Without inspecting the specific error message returned by the provider, operations teams risk misdiagnosing the root cause entirely.

Two Errors, Opposite Cures

During the incident, two specific error signatures appeared in the logs. The first was a transient capacity issue, while the second was a deterministic regional rejection.

The first signature returned a standard HTTP 503 status code with a payload message indicating high demand. This error is temporary. It clears with time, and standard exponential backoff retries are the correct operational response. Forcing a migration away from the primary provider for a temporary demand spike introduces unnecessary complexity.

The second signature returned an HTTP 400 status code accompanied by a message stating that the user location is not supported for the API use. This error is deterministic per egress IP. It never clears with retries, and treating it as an overload wastes the entire incident window waiting for a recovery that will not happen on that IP address.

Treating a geo-block as an overload wastes valuable time, while treating an overload as a geo-block triggers an unnecessary provider migration. Correlating failures solely by clock time or status code makes two different problems look like one.

Routing Around Geography

When a geo-block is confirmed, waiting for it to resolve is not a viable architecture. The structural answer involves routing around the geographic restriction by promoting an in-platform fallback tier. For example, routing execution to a serverless model inference engine running on the same edge network avoids external region-gating rules entirely.

To implement this safely, the fallback tier must be exercised against live traffic regularly. A fallback that never runs is merely a hope rather than a reliable tier. Verifying response-shape handling across both the primary provider and the fallback ensures that switching over during an incident does not introduce secondary parsing failures.

Log Bodies, Not Codes

Status-only logging is the primary reason why teams misdiagnose edge routing failures. Relying solely on HTTP status codes obscures the difference between a demand spike and a regional policy rejection. Capturing a short snippet of the response body, such as the first 160 characters of an error payload, provides the exact context needed to automate routing decisions.

Consider a minimal JavaScript implementation running in an edge worker that inspects the provider response before deciding whether to retry or fail over:

async function callGeminiWithFallback(requestData) {
  try {
    const response = await fetch("https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(requestData)
    });

    if (!response.ok) {
      const errorBody = await response.text();
      const snippet = errorBody.substring(0, 160);

      if (response.status === 400 && snippet.includes("user location is not supported")) {
        return await executeInPlatformFallback(requestData);
      }

      if (response.status === 503) {
        throw new TransientDemandError("Model experiencing high demand: " + snippet);
      }

      throw new Error("Unhandled provider error: " + snippet);
    }

    return await response.json();
  } catch (error) {
    if (error instanceof TransientDemandError) {
      return executeWithBackoff(requestData);
    }
    throw error;
  }
}

This pattern ensures that geographic rejections trigger an immediate switch to the secondary tier, while transient capacity limits correctly trigger bounded retries.

Conclusion

Managing external model providers from serverless edge environments requires looking past simple status codes. By capturing error response bodies, distinguishing static geo-blocking from temporary capacity overloads, and maintaining a warm fallback tier, engineering teams can build resilient integrations that survive unexpected regional policy shifts without manual intervention. For a related implementation, see Managing Gemini Overload Intelligent Fallback Patterns.

Fixing Threads API Authorization Code Failures

Sun, 04 Oct 2026 09:03:46 +0000

Integrating the Threads API requires navigating a three-legged OAuth process that can stall at several distinct points. When an application fails to exchange an authorization code for a valid access token, debugging often becomes a frustrating exercise in trial and error. The API returns cryptic numeric codes and generic error messages that rarely point to the root cause of the failure.

The core problem stems from strict validation rules across the authorization code, the application credentials, the redirect URIs, and token families. Understanding why these failures occur helps engineering teams establish a reliable diagnostic process and complete successful token exchanges without burning single-use codes.

The Three-Legged Token Flow

The Threads API uses a standard OAuth three-legged flow to establish user-level access. First, the user visits an authorization URL hosted by the platform, which redirects back to the application with a short-lived authorization code. Second, the backend exchanges this authorization code for a short-lived user access token via a POST request. Third, this short-lived token is exchanged for a sixty-day long-lived token.

Each leg of this journey relies on precise inputs. The authorization code expires within minutes and can only be used once. If any parameter in the subsequent requests deviates from the expected format, the platform rejects the exchange.

The Four Failure Modes

When token exchanges fail, the error typically maps to one of four specific conditions:

Wrong token family (452 Session key invalid): Tokens minted in generic testing surfaces or alternative Meta dashboards belong to a different token family. The Threads endpoint rejects these tokens outright.
Wrong secret of two: Meta frequently issues multiple app ID and secret pairs for a single project. The Threads API requires the specific Threads credential pair, and because both secrets share the same length, visual inspection is unreliable.
Non-identical redirect URI (36008): The redirect URI supplied during the code exchange must match the initial authorization request byte-for-byte, including trailing slashes and parameter encodings.
Stale or consumed codes (36006): Authorization codes are strictly single-use. Retrying a failed request with an old code or keeping multiple browser tabs open will trigger this failure.

The Diagnostic Ladder

To isolate these issues without wasting tokens, follow a strict diagnostic order. Testing items sequentially prevents compounding errors.

First, verify the token family. If a 452 error occurs, check whether the credentials originated from the correct Meta app dashboard configured specifically for Threads rather than general Facebook Login products.

Second, test the secret pairing programmatically. Because visual checks fail on same-length secrets, send a lightweight client credentials grant or inspection call using the suspected secret to confirm authentication success before attempting the user code exchange.

Third, compare redirect URIs at the byte level. Print out the exact string sent in both the authorization redirect and the token exchange POST body. A missing trailing slash or an unencoded character will immediately cause a 36008 error.

Finally, treat authorization codes as perishable singletons. Open a fresh authorization URL, capture the code immediately, and execute the exchange request within seconds.

Code Example for Token Exchange

Executing the token exchange correctly requires sending the proper POST parameters with the right application credentials. Here is an example request using curl:

curl -X POST "https://graph.threads.net/oauth/access_token" \
  -F "client_id=YOUR_THREADS_CLIENT_ID" \
  -F "client_secret=YOUR_THREADS_CLIENT_SECRET" \
  -F "grant_type=authorization_code" \
  -F "redirect_uri=https://example.com/callback" \
  -F "code=AQD_SAMPLE_AUTHORIZATION_CODE"

If successful, the endpoint returns a JSON payload containing the short-lived access token and the user ID. Immediately pass this token to the long-lived token endpoint to secure a sixty-day session.

After the Token Lands

Once the sixty-day token is successfully retrieved, verify its permissions by calling a read-scoped endpoint before storing it. Store the final token securely in platform secret storage or a key-value store with strict rotation policies. Never commit secrets to version control or leave them exposed in local configuration files. For a related implementation, see Secrets Local Config Ci.

Conclusion

Navigating the Threads API authentication flow depends on precise adherence to credential pairings, token families, and byte-level URI matching. By following a structured diagnostic approach and treating authorization codes as perishable singletons, developers can eliminate invalid-code failures and build resilient social integrations. For a related implementation, see Site Wide Structured Data Social Graph.

Parsing JSON from Thinking-Model APIs

Sun, 04 Oct 2026 03:15:39 +0000

When you request structured data from a reasoning model using a JSON mime type, the response often arrives split across several content parts rather than sitting neatly in a single text block. This breaks standard assumptions in ingestion pipelines. Developers usually expect the model to return a single text payload starting with an opening brace. Reasoning models, however, routinely prepend prose preambles, attach internal thought signatures, and partition the final output across distinct blocks. For a related implementation, see Schema First Gates Ai Publishing Pipelines.

The core problem manifests in three distinct ways during API integration. First, if your code attempts to read only the first part of the response, it captures conversational filler like introductory remarks, causing JSON parsers to throw syntax errors. Second, the model's token output limit must accommodate both the internal reasoning trace and the final JSON payload. If you allocate a small budget, the model spends all its tokens on thinking, leading to abrupt truncation with a max tokens finish reason. Third, applying blind retries to these truncated payloads merely drains rate limits without resolving the underlying deterministic failure.

Anatomy of a Split Response

A typical API payload from a reasoning model arrives as an array of content parts. The first part may contain a conversational preamble. Subsequent parts can include encrypted or signed reasoning traces, followed eventually by the markdown-fenced JSON structure. Treating this stream as a monolithic document guarantees parser failure.

Consider a scenario where an agent requests a configuration object from a reasoning model with an output budget set to two hundred tokens. The model generates an extensive internal monologue to work through the schema requirements. It exhausts the token limit just as it begins writing the payload, returning a truncated string without a closing brace. Your application receives a finish reason indicating truncation, while the primary text parts contain only partial data and reasoning debris.

The Defensive Parsing Recipe

To handle multi-part responses reliably, your ingestion pipeline needs a defensive parsing strategy that combines concatenation, unfencing, strict parsing, and robust fallbacks. For a related implementation, see Multi Agent Review Pipeline.

interface ModelPart {
  text?: string;
}

interface ModelResponse {
  candidates?: Array<{
    content?: {
      parts?: ModelPart[];
    };
    finishReason?: string;
  }>;
}

function extractJsonPayload(response: ModelResponse): Record<string, unknown> {
  const parts = response.candidates?.[0]?.content?.parts ?? [];
  const rawText = parts.map(p => p.text ?? '').join('');
  
  const unfenced = rawText
    .replace(/^```json\s*/gm, '')
    .replace(/^```\s*$/gm, '');

  try {
    return JSON.parse(unfenced);
  } catch {
    const match = unfenced.match(/\{[\s\S]*\}/);
    if (match) {
      return JSON.parse(match[0]);
    }
    throw new Error('Failed to extract valid JSON from response parts');
  }
}

This function joins all available text parts into a single string before attempting any operations. It then strips markdown code fences and attempts a strict parse. If strict parsing fails due to surrounding prose, a lightweight regular expression isolates the outermost curly braces. This approach avoids heavy third-party parsing dependencies while successfully navigating messy model outputs.

Budgeting for Thought and Fallbacks

Sizing your output limits correctly prevents truncation before parsing even begins. Because reasoning models bill their internal monologue against the exact same token limit as the final answer, you must scale your output budgets upward. A budget of several thousand tokens is often necessary for complex schemas, separating the cost of thought from the cost of the data structure.

When a parse failure does occur despite proper budgeting, avoid infinite retry loops. Instead, implement a log-and-skip mechanism. Record the finish reason, capture a bounded preview of the uncleaned text for debugging, and advance your fallback model chain or queue. This diagnostic approach turns silent ingestion failures into actionable telemetry.

Conclusion

Reliable integration with reasoning models requires moving past the assumption of clean, single-blob JSON responses. By joining multi-part outputs, budgeting appropriately for internal reasoning traces, and implementing defensive string manipulation, you can stabilize your automated pipelines against model-specific quirks.

Building a Resilient Job Queue in Go: Postgres, Worker Pool and Graceful Shutdown

Thu, 01 Oct 2026 18:08:31 +0000

Building a Resilient Job Queue in Go: Postgres, Worker Pool and Graceful Shutdown

I am Pires Aurélio, Go Backend Engineer from Luanda, Angola. I build production systems in public.

For two years I built scrapers and simple REST APIs. I got tired of tutorials that only show Hello World. No jobs, no retries, no real production problems.

So I built a real one: A Production-Ready Background Job Queue in Go.

This is the production deep-dive companion to my DEV article. All links are in the first comment.

WHAT WE ARE BUILDING

A REST API where you POST a task, it goes to Postgres, and Go workers process it in background with retries and graceful shutdown. Like Sidekiq and Celery, but in pure Go.

Stack: Go + Gin + Postgres + Goroutines + Docker
Features: Clean Architecture, Postgres as queue, Worker Pool with 5 Goroutines, Graceful Shutdown, Exponential Backoff, Supervisor pattern.

ARCHITECTURE

Client -> POST /jobs -> API Gin -> Postgres jobs table -> Worker Pool -> Process -> Update Status

Why Postgres and not Redis? One less service and ACID guarantees. This proves you understand transactions and row-level locking.

THE PRODUCTION SCHEMA

Table jobs has: id UUID, payload JSONB, status pending processing done failed, attempts int, max_attempts int, last_error text, created_at and next_retry_at.

The most important part is the index on status and next_retry_at where status is pending. This makes the picker O log n with 1 million rows. See Image 1.

THE SAFE PICKER

The mistake in tutorials is SELECT then UPDATE. Two workers get the same job.

The solution is an atomic UPDATE where id comes from a sub-select with row-level locking and SKIP LOCKED. Only one worker gets a job. This is what separates toy from production. See Image 1.

WORKER POOL

Start with concurrency 5. For each worker start a goroutine. Loop select on ctx Done for shutdown, default picks a job and processes with retry.

If fails, increment attempts. If attempts >= max, mark failed. Else backoff = 2 power attempts seconds and set next_retry_at.

See Image 2.

SUPERVISOR - NEVER LOSE A JOB

If server crashes while processing, job stays processing forever.

On startup: Reset all jobs where status is processing and updated_at older than 5 minutes back to pending. This is what real companies do.

GRACEFUL SHUTDOWN - THE INTERVIEW QUESTION

Use signal NotifyContext for Interrupt and SIGTERM. Start workers with that context. Start HTTP server in goroutine. Block on ctx Done. On signal, create timeout context 10 seconds and call server Shutdown. Workers stop via ctx Done.

See Image 3. This proves you know context propagation.

BENCHMARK

Python + Celery + Redis: 120 jobs per sec, 180 MB RAM
Go + Postgres: 1850 jobs per sec, 18 MB RAM

15x faster, 10x less memory.

WHAT RECRUITERS SEE

You understand concurrency, you understand databases beyond SELECT, you think about failure, you can build infrastructure not just CRUD.

Full code with metrics is coming to my GitHub next week. Links in first comment.

Built from Luanda, Angola.

Building Smart Health API: A Production-Style REST API with CNN-Based Risk Prediction

Thu, 01 Oct 2026 15:25:23 +0000

By Priyadharshiny J — GitHub: priyadharshiny13

Overview

Smart Health API is a backend platform I built to explore how a real healthcare product might structure its API layer: role-based access for different user types, secure authentication, database migrations that don't break production, and a machine learning model wired into an async pipeline instead of blocking the request thread.

The result is a FastAPI service that lets Patients, Doctors, and Admins interact with the same system through different permission levels, and that returns CNN-based oral cancer risk scores without making the caller wait on model inference.

Stack

API layer: FastAPI
Database: MySQL, with SQLAlchemy ORM
Migrations: Alembic
Auth: JWT-based authentication with Role-Based Access Control (RBAC)
ML: TensorFlow CNN model for oral cancer risk prediction
Async processing: Celery + Redis
Infra: Docker, deployed toward AWS
CI/CD: GitHub Actions

Architecture Decisions

Why RBAC with three roles instead of a single user type?
Healthcare data access isn't uniform — a Patient should only see their own records, a Doctor needs read/write access to their assigned patients, and an Admin needs oversight without clinical permissions. I modeled this directly into the JWT claims and enforced it at the route-dependency level in FastAPI, rather than checking roles inside each endpoint's business logic. This keeps permission logic in one place and out of the individual handlers.

Why Alembic migrations instead of create_all()?
SQLAlchemy.create_all() is fine for a prototype, but it can't handle schema evolution — adding a column, renaming a table, backfilling data. Alembic gave the project versioned, reversible migrations, which is the same discipline a real production service needs the moment more than one person touches the schema.

Why Celery + Redis for the CNN inference?
Running a CNN prediction synchronously inside an HTTP request handler blocks the worker for however long inference takes, which doesn't scale under concurrent load. I moved prediction into a Celery task queue backed by Redis: the API accepts the image, enqueues the job, and returns a task ID immediately. The client polls (or would be pushed a webhook/notification in a fuller version) for the result. This separates the request/response cycle from the compute-heavy ML work.

A Real Debugging Story

Wiring up SQLAlchemy models across multiple modules (User, Patient, Doctor, Prediction) introduced circular import errors once relationships started referencing each other — Patient needed User, User needed Prediction, and Python's import system didn't like the cycle. The fix was restructuring model definitions to use string-based relationship references (relationship("Patient", back_populates=...)) instead of direct class imports, which lets SQLAlchemy resolve the relationship lazily at mapper-configuration time rather than at import time.

Small detail, but it's the kind of thing that only shows up once a schema has enough interrelated tables — and it's a good reminder that ORM relationship design needs the same intentionality as the schema itself.

What's Next

Add automated tests for the RBAC permission boundaries (currently manually verified)
Expose Celery task status through a WebSocket instead of polling
Add rate limiting on the prediction endpoint given its compute cost

Repo

Code and setup instructions: github.com/priyadharshiny13 — see the smart-health-api repository.

api-performance-profiler

Mon, 28 Sep 2026 04:48:58 +0000

As backend developers, we often fly blind when building APIs locally. You write a route, but you don't really know how fast it runs until you leave your editor, set up Postman,
or configure a heavy APM tool.

I wanted performance testing to feel native and frictionless, so I built an open-source, zero-dependency local profiling extension for Node.js (Express & NestJS). 

Here is what it brings directly to your editor:

🟢 Inline Latency Metrics: See real-time execution speeds (e.g., 🟢 84ms) rendered right beside your route definitions in the code.
🚀 1-Click Load Testing: A '▶ Load test' CodeLens sits directly above your route. Click it to replay the last recorded request 1,000 times under heavy load, simulating real

traffic instantly.

📊 Routes Dashboard: A dedicated VS Code sidebar showing all observed routes in your app, sorted from slowest to fastest.
🔐 100% Local & Secure: Everything is stored in local memory. No cloud accounts, no external dashboards, and no bloated tracking dependencies. 

It takes less than a minute to set up—just install the lightweight middleware and the VS Code extension, and your editor comes alive with performance data.

I’d love for you to try it out on your local Express or NestJS projects! Let me know your thoughts, feature requests, or feedback in the comments.

Horizontally Scaling PostgreSQL: Database Sharding with Citus

Sat, 26 Sep 2026 10:35:35 +0000

A practical guide to scaling your database infrastructure horizontally. Learn how to shard your data, optimize query routing, and build a distributed multi-tenant architecture using Postgres, Citus and Docker.

Prerequisites

What do you need to follow along?

A working Laptop/PC?
Working knowledge of SQL or PostgreSQL.
To have at least eaten something today (preferably swallow related).

Database sharding is a strategy used to solve scalability issues in applications that have a massive amount of data. It involves splitting data into smaller chunks called shards. Each shard is stored on a separate database server instance, effectively spreading the data and the workload across multiple machines.

Why Shard Data?

As an application grows, the number of users and the amount of data it stores increase over time. At a certain limit, a single database server will become a bottleneck. The application slows down and affects the user experience.

Instead of upgrading a single server with more CPU or RAM, which will eventually hit a hardware ceiling, sharding lets you add more servers to your cluster. This allows applications to handle nearly infinite growth in data volume and user traffic.

Understanding Shards and Shard Keys

Before we start sharding data, let's first understand what shards and shard keys are.

Shards

The partitioned data chunks are called logical shards. The machine that stores the logical shard is called a physical shard or database node. A physical shard can contain multiple logical shards.

Shard Key

A shard key is used to determine how to partition the dataset. A shard key is a specific column in your dataset that determines which rows of data group together to form a shard.

Sharding Methods

There are a few ways to shard data depending on the application requirements.

Key-based sharding (hash sharding): Hash sharding uses a mathematical formula called a hash function. The hash function takes your shard key and generates a hash value, this hash value determines the exact physical shard where the row will be stored.
Range-based sharding: Range-based sharding, or dynamic sharding, splits database rows based on a range of values. Then the database designer assigns a shard key to the respective range.
Directory-based sharding: Directory sharding uses a lookup table to match database information to the corresponding physical shard. A lookup table is like a table on a spreadsheet that links a database column to a shard key.
Geo sharding: Geo sharding splits and stores database information according to geographical location.

For this guide, we are focusing exclusively on Hash Sharding. This is the industry standard for multi-tenant applications like the e-commerce platform we will be building in this tutorial.

Choosing Your Shard Key

If you pick the wrong shard key, your database will still crash under load. When choosing a shard key, you must ensure the data is distributed evenly across your nodes:

Cardinality: This refers to the number of unique values a key can have. You want a key with high cardinality like a User ID, not low cardinality like a Gender or State column, so the data can be split into many small, even chunks.
Frequency: You must avoid keys where one specific value appears too often. For example, if you shard by a City column and 80% of your users live in Ikeja, the Ikeja node will be overloaded while the others sit idle.
Monotonic change: This is the rate of change of the shard key. A monotonically increasing or decreasing shard key results in unbalanced shards. For example, consider a feedback database that is split into three different physical shards as follows:
- Shard A stores feedback from customers who have made 0 to 10 purchases.
- Shard B stores feedback from customers who have made 11 to 20 purchases.
- Shard C stores feedback from customers who have made 21 or more purchases.
  As the business grows, customers will make 21 or more purchases. The application stores their feedback in Shard C. This results in an unbalanced shard because Shard C contains more feedback records than the other shards.

Is Sharding Actually For You?

While sharding solves massive scale requirements, it introduces significant technical and operational tradeoffs. You should not shard your database unless a single, heavily provisioned Postgres instance can no longer handle your traffic.

If you do decide to shard your database, here are some of the biggest challenges you will face:

Data hotspots: Even with a good hash function, uneven traffic can create hotspots where one server becomes overloaded by a small group of highly active users.
Performance regression: Queries that do not use the shard key require the system to scatter and gather across every server. Cross-shard joins are computationally expensive because the application must pull and merge data from multiple physical locations over the network.
Operational complexity: Managing a sharded environment increases the difficulty of routine tasks. Schema updates, backups, and point-in-time recovery must be coordinated across multiple independent instances.

How Can We Shard Our Postgres Database: Introducing Citus

Citus is an open-source extension that transforms standard PostgreSQL into a distributed database. Instead of being limited to a single server, Citus allows your database to scale horizontally across multiple machines.

How Citus Works

The Coordinator Node: When you ask your database a question, you talk to the coordinator. The coordinator directs your request to the right servers, gathers the answers, and combines them for your application.
Worker Nodes: Citus breaks down large tables into smaller, manageable chunks (shards) and spreads them across these worker servers. They do the heavy lifting of storing the data and running the queries.

Start Early, Scale Later

Migrating a traditional, monolithic PostgreSQL database to a distributed, sharded database later in its life is incredibly difficult. You have to change your schema, rewrite complex queries, and alter how your application connects to the database.

To avoid this nightmare, you can start using Citus on a single node from day one. You design your tables with distribution keys immediately, and your application code is written to query a distributed database from the start.

The Transition: When your data outgrows that single machine, you won't have to rewrite your application. You simply spin up new worker machines, and Citus moves the existing shards over to the new hardware in the background while the application keeps running.

Running Citus Locally

We will use Docker to run Citus locally. Why use Docker instead of adding the extension directly to your host machine's Postgres installation?

Simulating a distributed database means running multiple Postgres instances simultaneously. Doing this natively forces you to manually manage conflicting ports and multiple data directories. Docker Compose solves this by orchestrating the entire cluster's networking with a single command, giving you a clean state that can be torn down and rebuilt in seconds.

Single Node vs Multi-Node Setup

While it is possible to run a single node Citus instance using a simple docker run command:

docker run -d --name citus_standalone -p 5432:5432 -e POSTGRES_PASSWORD=postgres citusdata/citus:14.1.0

This guide will focus exclusively on setting up a Multi-Node Cluster. If you are interested in exploring the single node setup further, you can check out the official Citus documentation here.

For our distributed, multi-node cluster, we will use the following Docker Compose configuration:

services:
  master:
    container_name: "${PROJECT_NAME:-citus}_master"
    image: "citusdata/citus:14.1.0"
    ports: ["${COORDINATOR_EXTERNAL_PORT:-5433}:5432"]
    labels: ["com.citusdata.role=Master"]
    environment: &AUTH
      POSTGRES_USER: "${POSTGRES_USER:-postgres}"
      POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
      PGUSER: "${POSTGRES_USER:-postgres}"
      PGPASSWORD: "${POSTGRES_PASSWORD}"
      POSTGRES_HOST_AUTH_METHOD: "${POSTGRES_HOST_AUTH_METHOD:-trust}"
  worker:
    image: "citusdata/citus:14.1.0"
    labels: ["com.citusdata.role=Worker"]
    environment: *AUTH

Citus has a built-in initialization script in its official Docker image. When launched via Docker Compose with these specific labels like com.citusdata.role, the coordinator automatically detects and registers the workers for you.

We can build and start the cluster in the background with:

docker compose up -d

We can also scale up dynamically. If you want to spin up a second or third worker machine on the fly, you do not even need to edit the compose file. Just run:

docker compose up -d --scale worker=3

Where 3 is the total number of workers you want to have running.

Simply spinning up the worker containers is not enough. After adding new worker nodes, we must register them with the coordinator so Citus knows they are active and ready to hold data. To do this, we will first connect to the interactive Postgres terminal on the master node:

docker compose exec master psql -U postgres

Now we can register our workers:

SELECT citus_add_node('db-sharding-worker-1', 5432);
SELECT citus_add_node('db-sharding-worker-2', 5432);
SELECT citus_add_node('db-sharding-worker-3', 5432);

So, if you scale up again to say 5 workers, you will have to run this command 2 more times to register the 2 new workers.

Notice the db-sharding- prefix in the hostnames. This is necessary because Docker Compose prefixes your project folder name to the container's network alias.

A Real World Scenario

In order to showcase the functionality of this database sharding setup, let's try to work with a real world scenario.

Consider an e-commerce multi-tenant application. It serves multiple independent companies that all share the same database infrastructure. To scale this effectively, we need to use a hash-based distribution with a well-chosen shard key. Let's assume by design, every tenant has a shared company_id, we will use that as our shard key.

By using company_id as the shard key across all operational tables, Citus guarantees that all data belonging to a specific company (like its users, its products, and its orders) is physically co-located on the exact same worker node.

Now that our workers are registered, we can create our core tables directly in the master node. Citus will automatically handle the underlying sharding for us.

Please note that the following schema is highly simplified to focus on the core concepts of sharding.

Run the following SQL commands in your coordinator's postgres interactive terminal:

-- 1. Create Tenant Table
CREATE TABLE companies (
    company_id INT PRIMARY KEY,
    company_name TEXT NOT NULL
);

-- 2. Create Tenant-Scoped Orders Table
CREATE TABLE orders (
    company_id INT NOT NULL,
    order_id INT NOT NULL,
    user_id INT NOT NULL,
    status_id INT NOT NULL, -- References our global order status table
    order_amount NUMERIC(10,2) NOT NULL,
    PRIMARY KEY (company_id, order_id)
);

-- 3. Create Global Order Reference Table
CREATE TABLE order_statuses (
    status_id INT PRIMARY KEY,
    status_name TEXT NOT NULL
);

-- 4. Create Tenant-Scoped Users Table
CREATE TABLE users (
    company_id INT NOT NULL,
    user_id INT NOT NULL,
    username TEXT NOT NULL,
    PRIMARY KEY (company_id, user_id)
);

-- 5. Create Tenant-Scoped Products Table
CREATE TABLE products (
    company_id INT NOT NULL,
    product_id INT NOT NULL,
    product_name TEXT NOT NULL,
    price NUMERIC(10,2) NOT NULL,
    PRIMARY KEY (company_id, product_id)
);

Telling Citus How to Shard and Replicate

Now that we have created our core tables, we have to tell Citus how to shard and replicate them across our worker nodes.

Reference Tables

First, we will set order_statuses as a reference table. We do this using create_reference_table.

Why do we create a reference table? A reference table is designed for relatively small, slowly-changing data like user roles, country codes, or order statuses.

When you mark a table as a reference table, Citus physically replicates a full copy of that entire table to every single worker node. This bypasses network overhead completely during JOIN operations. Because the worker node already has the data locally, it does not need to fetch it from another node to complete a query.

Distributed Tables

Next, we need to distribute our large tenant-data tables. We do this using create_distributed_table.

This function is the core of Citus's sharding mechanism. It tells the coordinator to partition the table based on a specific column (our shard key, company_id), and spread those partitions across the available worker nodes.

Because we use company_id across all these tables, any query filtering by a specific company will be routed directly to a single node, making it incredibly fast.

-- Replicate order_statuses to ALL nodes
SELECT create_reference_table('order_statuses');

-- Shard data based on the Hash of company_id
SELECT create_distributed_table('companies', 'company_id');
SELECT create_distributed_table('users', 'company_id');
SELECT create_distributed_table('products', 'company_id');
SELECT create_distributed_table('orders', 'company_id');

Multi-Tenant Seed Data

Now we will populate our cluster with some dummy companies along with some global order status entries, orders, and tenant users.

-- Global Order Lookups
INSERT INTO order_statuses VALUES (1, 'Pending'), (2, 'Shipped'), (3, 'Delivered');

-- Company Tenants
INSERT INTO companies VALUES (1, 'EbaMart'), (2, 'FufuHub'), (3, 'GymGear');

-- Multi-Tenant Users scattered across companies
INSERT INTO users (company_id, user_id, username) VALUES
(1, 101, 'Samuel'),   -- EbaMart Customer
(2, 201, 'Folashade'),-- FufuHub Customer
(1, 102, 'Umoh'),     -- EbaMart Customer
(3, 301, 'Oladejo'),  -- GymGear Customer
(2, 202, 'BigSam');   -- FufuHub Customer

-- Orders
INSERT INTO orders (company_id, order_id, user_id, status_id, order_amount) VALUES
(1, 5001, 101, 1, 150.00),  -- EbaMart order: Pending
(1, 5002, 102, 3, 45.00),   -- EbaMart order: Delivered
(2, 6001, 201, 2, 99.00),   -- FufuHub order: Shipped
(2, 6002, 202, 1, 250.00);  -- FufuHub order: Pending

Verify Layout and Node Placement

We can now verify the layout and node placement to see where they are added to and display it by running:

SELECT
    u.company_id,
    u.user_id,
    u.username,
    s.shardid,
    s.nodename
FROM users u
JOIN citus_shards s
  ON s.shardid = get_shard_id_for_distribution_column('users', u.company_id)
ORDER BY u.company_id;

Output:

company_id	user_id	username	shardid	nodename
1	101	Samuel	102042	db-sharding-worker-2
1	102	Umoh	102042	db-sharding-worker-2
2	201	Folashade	102065	db-sharding-worker-1
2	202	BigSam	102065	db-sharding-worker-1
3	301	Oladejo	102056	db-sharding-worker-1

As you can see, all data for company_id = 1 sits neatly on db-sharding-worker-2, while company_id = 2 sits on db-sharding-worker-1. This guarantees that queries for a single tenant will only hit one node.

We can also inspect the query plan for joining distributed tables with reference tables:

EXPLAIN (VERBOSE, COSTS OFF)
SELECT o.order_id, o.order_amount, s.status_name
FROM orders o
JOIN order_statuses s ON o.status_id = s.status_id
WHERE o.company_id = 1;

Output:

                                      QUERY PLAN
---------------------------------------------------------------------
---------------------------------------------------------------------
---------------------------------------------------------------------
------------------------
 Custom Scan (Citus Adaptive)
   Output: remote_scan.order_id, remote_scan.order_amount, remote_scan.status_name
   Task Count: 1
   Tasks Shown: All
   ->  Task
         Query: SELECT o.order_id, o.order_amount, s.status_name FROM (public.orders_102106 o JOIN public.order_statuses_102008 s ON ((o.status_id OPERATOR(pg_catalog.=) s.status_id))) WHERE (o.company_id OPERATOR(pg_catalog.=) 1)
         Node: host=db-sharding-worker-2 port=5432 dbname=postgres
         ->  Hash Join
               Output: o.order_id, o.order_amount, s.status_name
               Hash Cond: (s.status_id = o.status_id)
               ->  Seq Scan on public.order_statuses_102008 s
                     Output: s.status_id, s.status_name
               ->  Hash
                     Output: o.order_id, o.order_amount, o.status_id
                     ->  Bitmap Heap Scan on public.orders_102106 o
                           Output: o.order_id, o.order_amount, o.status_id
                           Recheck Cond: (o.company_id = 1)
                           ->  Bitmap Index Scan on orders_pkey_102106
                                 Index Cond: (o.company_id = 1)
(19 rows)

The query plan shows a Custom Scan (Citus Adaptive) delegating the query to the specific worker node holding company_id = 1.

We can check the active worker nodes with:

SELECT * FROM master_get_active_worker_nodes();

Output:

node_name	node_port
db-sharding-worker-3	5432
db-sharding-worker-1	5432
db-sharding-worker-2	5432

Rebalancing the Cluster

Anytime you dynamically scale the cluster and register a new worker node, you must run the Citus Shard Rebalancer:

SELECT citus_rebalance_start();

When new nodes are added, the coordinator does not automatically move existing data onto them. We have to explicitly balance the load. The rebalancer handles this by dynamically moving a subset of your tenants' data over to the new worker servers in the background, without taking the database offline.

Production Setup

The Docker Compose setup is perfect for local testing and development. However, running a distributed database in production across multiple physical servers or virtual machines requires a slightly different approach.

We must first change how network addressing and orchestration work. In a local setup, Docker Compose handles networking automatically because everything shares a single machine loopback. On production VMs, each container must be bound to the public or private IP address of its physical host machine, and you must enforce strict firewall security.

1. Spin up your VMs

Spin up your virtual machines on your preferred cloud provider. Assuming we provision 4 VMs, our network layout will look something like this:

VM 1 (Coordinator Node): Private IP 10.0.0.10
VM 2 (Worker Node 1): Private IP 10.0.0.11
VM 3 (Worker Node 2): Private IP 10.0.0.12
VM 4 (Worker Node 3): Private IP 10.0.0.13

I used random private IPs, but you should substitute them with your real VMs' private IPs.

2. Install Docker

Next, you will need to install Docker and Docker Compose on every VM. The installation commands will depend on your specific Linux distribution, so refer to the official Docker documentation for your OS.

You will no longer use a single master compose file for the cluster. Instead, you deploy an isolated, optimized compose layout directly on each corresponding VM instance.

3. The Coordinator Instance (VM 1)

Create a docker-compose.yml file specifically tailored to map the Postgres engine port directly out to the host VM network interface:

services:
  master:
    container_name: "citus_coordinator"
    image: "citusdata/citus:14.1.0"
    ports:
      - "5432:5432"
    restart: always
    environment:
      POSTGRES_USER: "citus_admin"
      POSTGRES_PASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      PGUSER: "citus_admin"
      PGPASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      POSTGRES_HOST_AUTH_METHOD: "scram-sha-256"
    volumes:
      - coordinator_data:/var/lib/postgresql/data

volumes:
  coordinator_data:

4. The Worker Instances (VMs 2, 3, and 4)

Every worker VM gets its own standalone compose file. Notice we expose port 5432 on the workers too, because the coordinator node needs a direct network pipe to talk to them:

services:
  worker:
    container_name: "citus_worker"
    image: "citusdata/citus:14.1.0"
    ports:
      - "5432:5432"
    restart: always
    environment:
      POSTGRES_USER: "citus_admin"
      POSTGRES_PASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      PGUSER: "citus_admin"
      PGPASSWORD: "YOUR_SUPER_SECURE_PRODUCTION_PASSWORD"
      POSTGRES_HOST_AUTH_METHOD: "scram-sha-256"
    volumes:
      - worker_data:/var/lib/postgresql/data

volumes:
  worker_data:

5. Configure Network Firewall Security

Since you are running across independent cloud instances, your cloud platform's Security Groups or Firewalls (like AWS Security Groups or ufw on Ubuntu) must be strictly configured to prevent public database scanning.

On the Coordinator VM (VM 1): Allow inbound TCP traffic on port 5432 from Anywhere (0.0.0.0/0) only if your external application servers need public access. If your application servers are in the same VPC, restrict inbound port 5432 access exclusively to your app servers' IP subnet range.
On the Worker VMs (VMs 2, 3, 4): Allow inbound TCP traffic on port 5432 EXCLUSIVELY from the Coordinator's Private IP (10.0.0.10). Block all other inbound network requests to the workers. They should never be exposed to the internet.

Next, log in to each VM and run the containers:

docker compose up -d

Once all containers are healthy, access the interactive postgres interface on VM 1 (the coordinator node):

docker compose exec master psql -U citus_admin

SELECT citus_add_node('10.0.0.11', 5432); -- Register Worker VM 2
SELECT citus_add_node('10.0.0.12', 5432); -- Register Worker VM 3
SELECT citus_add_node('10.0.0.13', 5432); -- Register Worker VM 4

Verify everything works:

SELECT * FROM master_get_active_worker_nodes();

6. Configure Production Password Authentication

Because we updated POSTGRES_HOST_AUTH_METHOD to scram-sha-256 for security, your instances will require passwords to authenticate cross-node.

We must add a Postgres password passfile on the coordinator container:

docker compose exec master bash -c "echo '*:*:*:citus_admin:YOUR_SUPER_SECURE_PRODUCTION_PASSWORD' >> /var/lib/postgresql/.pgpass && chmod 600 /var/lib/postgresql/.pgpass"

Surely you will know to replace YOUR_SUPER_SECURE_PRODUCTION_PASSWORD with your actual password.

Here is a detailed documentation of Citus multi-node installation and setup for reference: Citus Multi-Node Setup.

Adding Citus to an Existing Project

To add Citus to an already existing Postgres database project, the process follows these general steps:

Install the Extension: Install the Citus extension on your Postgres server. You will also need to add citus to the shared_preload_libraries in your postgresql.conf file and restart the database.
Create the Extension: Run CREATE EXTENSION citus; in your database.
Select a Shard Key: Identify the column that will serve as the shard key for your distributed tables, such as a company or tenant ID.
Convert Tables: Use create_distributed_table() to convert your existing tables into distributed tables. Use create_reference_table() for lookup tables that do not need to be sharded.
Migrate Data: If you already have data in the tables, it might need to be exported and loaded back in, depending on the volume of data and the specific requirements of the migration.

Conclusion

Database sharding is an incredibly powerful architectural pattern, but it comes with immense operational tradeoffs. By using tools like Citus, we can bridge the gap between standard relational SQL databases and massively distributed systems.

Whether you are starting a new multi-tenant application or scaling out an existing infrastructure, adopting a distributed mindset and orchestrating your local environments with tools like Citus and Docker Compose will save you countless hours of architectural nightmares down the road.

You can find the full Docker Compose source code for this tutorial on GitHub: umohsamuel/database-sharding.

References

Stop Killing Your Database: The Power of Connection Pooling

Tue, 22 Sep 2026 08:51:06 +0000

Connection pooling is a software development technique used to maintain a cache of database connections that can be reused for future requests. Instead of opening and closing a brand-new connection to the database every time a user wants to read or write data, your application keeps a "pool" of active connections ready to go. This drastically reduces the overhead of establishing new database handshakes over and over again.

The Pizza Delivery Analogy

Imagine you run a busy pizza shop. Every time a delivery order comes in, instead of sending a driver who is already waiting at the shop, you post a job listing online, interview a candidate, hire them, buy them a car, have them deliver the pizza, and then fire them and sell the car immediately after.

It sounds completely insane, right? Yet, that is exactly what your server does when it creates a brand-new database connection for every single HTTP request, only to destroy it a millisecond later.

Connection pooling is like hiring a dedicated team of five delivery drivers who hang out at the shop. When an order is ready, one driver grabs it, delivers it, and immediately returns to the shop to wait for the next order.

Why Connection Pooling Matters Daily

In production environments, creating a database connection is an expensive operation that involves cryptographic handshakes, network latency, and authentication checks. If your Node.js application receives 100 concurrent API requests and tries to spin up 100 separate database connections simultaneously, your MySQL database will quickly run out of memory or reject connections altogether, throwing the dreaded "Too many connections" error.

By implementing a connection pool, you cap the maximum number of active database connections, protect your database from crashing under sudden traffic spikes, and shave off up to 90% of your API response latency because the connection is already warmed up and ready to run queries.

Implementation in Node.js & MySQL

Here is how simple it is to set up a connection pool in an Express.js backend using the popular mysql2 driver:

const express = require('express');
const mysql = require('mysql2/promise');

const app = express();

// Create a connection pool instead of a single connection
const pool = mysql.createPool({
  host: 'localhost',
  user: 'root',
  database: 'store_db',
  waitForConnections: true,
  connectionLimit: 10, // Max 10 active connections kept in the pool
  queueLimit: 0
});

app.get('/products', async (req, res) => {
  try {
    // The pool automatically rents out a connection, runs the query,
    // and returns the connection back to the pool when finished.
    const [rows] = await pool.query('SELECT * FROM products LIMIT 10');
    res.json(rows);
  } catch (error) {
    console.error('Database query error:', error);
    res.status(500).send('Database error');
  }
});

app.listen(3000, () => {
  console.log('Server running on port 3000');
});

The Takeaway

Optimizing your backend performance isn't always about writing faster algorithms; often, it's about managing your external resources wisely. By treating database connections as a reusable, limited public utility rather than disposable single-use items, you build highly resilient APIs capable of weathering massive traffic spikes without sweating.

Originally published on my blog. You can read the alternative breakdown here.

Why I Built EasyDown: Normalizing Public Media Across Social Platforms

Mon, 21 Sep 2026 11:12:52 +0000

I started building EasyDown after running into the same integration problem repeatedly: a user would paste a perfectly valid public social media URL, but every platform represented the media differently. A Reel was not shaped like a TikTok post, an Instagram carousel was not shaped like a Bilibili video, and even two URLs from the same platform could require different handling.

The first version looked simple. Detect the host, call a platform-specific parser, and return a video URL. That design stopped working as soon as I tested real posts. Some posts contained images rather than video. Some were mixed carousels. Some video formats separated audio and video. CDN links expired. Browser requests failed because of CORS or missing headers. A profile URL looked similar to a post URL but represented a completely different resource.

I decided that the useful product was not another collection of one-off download scripts. I needed a stable backend contract that could represent media from several public platforms without forcing every client to learn each upstream response format.

The problem I wanted the API to solve

From an application developer's perspective, the input should be one supported public post URL. The result should be a predictable object containing the platform, title, thumbnail, duration when available, and arrays for images, videos, and audio.

The arrays matter. Assuming that one post equals one MP4 is an easy mistake. An Instagram carousel may contain several photos and videos in a meaningful order. A TikTok photo post may have no video at all. A Bilibili response may expose multiple stream options. Returning a collection lets the product decide whether to show a picker, enqueue every item, or select a preferred rendition.

I also wanted failures to be explicit. Private posts, deleted media, expired Stories, login-only pages, profiles, live content, and region-restricted items should not be presented as successful parses. A client needs to distinguish an invalid input from a temporary upstream failure and from a valid post that has no downloadable media.

One normalized endpoint and platform-specific endpoints

EasyDown now has a common endpoint for applications that want one stable media model across supported networks. It also has platform-specific endpoints for integrations that need additional public fields from a particular service.

Instagram is a good example. The normalized media object is enough for a download workflow, but some applications also need the public shortcode, caption, author fields, image candidates, video renditions, or carousel items. I documented the current request and response contract in the Instagram API documentation I maintain.

The platform request is a normal authenticated POST from a backend:

import requests

response = requests.post(
    "https://api.easydown.org/api/v1/platforms/instagram/parse",
    headers={
        "Authorization": "Bearer ed_live_your_token",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://www.instagram.com/reel/DY2PcmxiBE2/"
    },
    timeout=30,
)

response.raise_for_status()
payload = response.json()
media = payload["data"]["media"]

for item in media.get("videos", []):
    print(item["url"], item.get("quality"), item.get("mimeType"))

The token stays on the server. I do not recommend placing a production token in browser JavaScript, a URL, analytics, or a public repository.

Platforms I support

The current API covers public post URLs from TikTok, Douyin, Toutiao, Instagram, Threads, Xiaohongshu or RedNote, Bilibili, Kuaishou or Kwai, Weibo, X or Twitter, and YouTube.

Support does not mean scraping every page on those sites. The contract is intentionally scoped to documented single-post URL formats and compatible share links. Profiles, search pages, private messages, private media, arbitrary bulk collection, and unavailable content are outside that scope.

Keeping the boundary narrow has made the API easier to reason about. Every new URL form has to pass through validation, platform detection, parsing, normalization, sensitive-field filtering, and media checks before the response is considered successful.

Lessons from building the media layer

The hardest production issue has been treating returned media URLs as temporary resources. Many social media CDNs use signatures or expiration parameters. Storing the URL as though it were a permanent asset creates jobs that fail hours or days later.

My current rule is to keep the original public post URL as the durable reference. A backend parses it when the application needs fresh media candidates, then downloads or proxies the selected file while it is valid. When a CDN URL expires, the application re-parses the post instead of retrying the expired URL indefinitely.

The second lesson is that browser playback and backend downloading are different operations. A URL that opens in one browser session can still fail in application code because of CORS, CORP, cookies, Referer requirements, or a short-lived signature. Media fetching belongs in the backend when the product needs predictable behavior.

The third lesson is to normalize conservatively. I keep stable cross-platform fields small and predictable, while versioning the extra platform data separately. That approach gives client applications a dependable base model without discarding useful public fields that only exist on one platform.

What I would test before shipping

For an Instagram integration, I would test a public Reel, a video post, a single-image post, a mixed carousel, a compatible share URL, and an unavailable post. I would also test an expired CDN URL and confirm that the application re-parses the original post.

Across every platform, I check that credentials remain server-side, errors do not leak tokens, media arrays can be empty, and the UI does not promise access to private or protected content. Those cases are less exciting than the first successful request, but they decide whether an integration survives real users.

Building EasyDown has mostly been an exercise in turning many unstable upstream shapes into one smaller contract that an application can trust. The parsing code still changes as platforms change. The goal of the API is to keep those changes behind a stable boundary so client code does not have to change with them.

Jev by TypeSafe AI: Hype, Tool or Threat?

Mon, 21 Sep 2026 09:42:28 +0000

A few days ago, TypeSafe AI introduced Jev, a new model designed to make fast, structured decisions that software can use directly.

I read the announcement and had two questions: Is Jev more of a hype? Is it really useful for me as a backend engineer? And does it present a new threat of replacing me as a software engineer? Jev also reminded me of the OpenAI REST API "structured outputs" endpoints. I wondered whether Jev would offer any advantages over OpenAI.

Here is what I found.

What Does Jev Actually Do

Jev (by TypeSafe AI) belongs to a class of narrowly specialized models that provide structured outputs rather than just prose. It accepts application state as input and returns structured decisions. The possible outputs and their types are defined in advance.

For example, an application could provide information about a customer and ask Jev to classify the customer's churn risk, determine which support department should handle a request, or score a potential business opportunity. The application receives predefined values that it can use in its business logic.

Very similar to OpenAI "structured outputs", I think.

TypeSafe AI claims that Jev provides several advantages:

Response times of 70–500 ms.
Input costs of $0.042 per million tokens.
Guaranteed adherence to predefined output types.
Calibrated probabilities and confidence scores accompanying its decisions.

The company also claims substantial improvements in speed and cost compared with existing LLMs performing similar tasks.

These are the company's published results. Jev is currently in early access, and I haven't tested it myself.

Jev vs OpenAI Structured Outputs

I have already worked with OpenAI's API to implement structured classification and decision-making workflows. OpenAI introduced Structured Outputs in August 2024. With strict: true, developers can define a JSON Schema and receive responses conforming to that schema.

We can already classify incoming data, select values from predefined enums, extract structured information, and use the results in application code. ([OpenAI][2]). Jev provides essentially the same capabilities for these use cases.

Jev's underlying generation mechanism (it generates its decision outputs in parallel and provides associated probabilities - confidence scores) is interesting, but there is little reason to judge Jev's advantage based on its implementation details.

The practical question for a backend engineer is whether Jev produces decisions of comparable accuracy, faster and at a lower cost, for the same workload. They claim it has an edge in this. But with OpenAI Structured Outputs we can select the seriously cheaper model still very capable of producing structured outputs with a large focused (cached) context.

Jev: Useful in Real Backend Systems?

At one innovation analytics startup, I worked on backend architecture and LLM-powered ingestion and scoring pipelines. The system needed to process incoming information, classify it using controlled vocabularies, and produce structured results that could be used by subsequent application components.

I implemented deterministic validation gates to check model outputs before allowing the code to proceed through the workflow. Looking at Jev's capabilities, I can see where it could fit into this architecture. For example, I could use it to classify incoming records or calculate particular scores.

I had great results with GPT-4o Mini (quite cheaper than GPT Luna that TypeSafe AI compares Jev with) with latencies well within 100ms and costs like $0.15 per 1,000 requests of ~2K prompt and ~700 bytes JSON response.

Separately, Jev's confidence scores may seem to be useful for workflows where an application needs to decide whether to accept a model's decision or send the case for further processing. But these scores can be equally misleading requiring additional deterministic validation gates, thus adding the complexity to a workflow.

My conclusion is Jev deserves some A/B testing under realistic production workloads vs OpenAI structured outputs before any decision. With millions of requests it may offer an advantage. As well its confidence scores may be used in some cases.

A Tool Or an Engineer's Replacement.

This was my original concern. A usual senior backend engineer's work involves designing application architecture, domain models and service boundaries, implementing APIs, working with databases, and building distributed systems, and often, implementing business workflows that involve classification, scoring, and other decisions based on incoming data.

Jev could automate some decision-making operations. As well as OpenAI structured outputs does. But hardly presents any threat to the entire engineering process. So the answer is "a tool", that has had alternatives for already ~2 years.

My Conclusion

After examining Jev and comparing it with OpenAI's existing Structured Outputs, I see it as a potentially useful addition to my backend engineering toolkit with advantages that have yet to be proven for production systems already using OpenAI structured outputs.

For now, I am interested in seeing how Jev performs on real business workloads and whether its claimed performance improvements justify replacing existing LLM calls in production systems.

HTTP Just Got a New Method After 16 Years—Meet QUERY.

Sun, 20 Sep 2026 09:21:23 +0000

The New HTTP QUERY Method: The First New HTTP Method in 16 Years

What if you could have the simplicity of GET with the flexibility of POST? That's exactly what the new HTTP QUERY method aims to deliver.

If you've built REST APIs, you've probably faced this situation before.

You need to fetch data from your server, so naturally, GET seems like the right choice. But then your filters become more complex. You need nested objects, multiple conditions, sorting, pagination, maybe even advanced search criteria. Suddenly, your URL starts looking like a puzzle.

GET /orders?status=paid&customer=Ravi&from=2026-01-01&to=2026-06-30&sort=-amount&page=2&include=products,shipping,payments

As the application grows, that URL only gets longer and harder to manage.

Most developers eventually switch to POST for search endpoints, even though they're only retrieving data.

For years, this has been one of the awkward compromises in REST API design.

That changes with the new QUERY HTTP method, standardized in RFC 10008 (June 2026).

Why Was QUERY Needed?

Let's understand the problem first.

Imagine you're building an e-commerce platform.

A user wants to search orders based on:

Customer name
Order status
Date range
Price range
Multiple product categories
Sort order
Pagination

With GET, everything has to fit inside the URL.

GET /orders?customer=Ravi&status=paid&category=laptop&category=gaming&page=3&sort=-price

This works...

...until it doesn't.

As the filters become more advanced, URLs become difficult to read, difficult to encode correctly, and can even exceed practical length limits imposed by browsers, servers, or proxies.

Another downside is that query parameters often appear in logs, browser history, and analytics systems, making them less suitable for sensitive search criteria.

Clearly, developers needed a better solution.

Why Not Just Use POST?

This is exactly what many APIs do today.

POST /orders/search
Content-Type: application/json

`{
"customer": "Ravi",
"status": "paid",
"categories": [

"Laptop",
"Gaming"

],
"price": {

"min": 500,
"max": 2000

},
"page": 3
}`

The request body solves many problems.

It's clean.

It's readable.

It supports complex JSON.

So what's wrong?

The issue isn't technical.

It's semantic.

POST traditionally represents an operation that may change server state.

Searching for data doesn't.

Using POST for read-only operations has always been a compromise because HTTP infrastructure can't safely assume the request is cacheable or automatically retryable.

Meet QUERY

QUERY combines the strengths of both GET and POST.

Instead of placing filters in the URL, you send them inside the request body.

QUERY /orders
Content-Type: application/json

`{
"customer": "Ravi",
"status": "paid",
"categories": [

"Laptop",
"Gaming"

],
"price": {

"min": 500,
"max": 2000

}
}`

Unlike POST, QUERY is explicitly defined as a safe and idempotent method.

That means everyone involved—the client, server, proxy, and cache—knows that executing the request will not modify server data.

This makes QUERY a much better fit for complex searches.

What Does "Idempotent" Actually Mean?

You'll often hear this word when talking about HTTP methods.

An operation is idempotent if performing it multiple times has the same effect as performing it once.

For example:

GET /users/42

Calling this request one time or ten times returns the same user.

Now compare that to:

POST /users

Calling it twice could create two different users.

That's why POST isn't considered idempotent.

QUERY behaves like GET in this regard.

If the network connection drops halfway through the response, a client can safely retry the QUERY request without worrying about accidentally changing server data.

Why This Matters

The QUERY method brings several advantages:

✅ Supports complex request bodies
✅ Safe for read-only operations
✅ Idempotent
✅ Easier for caching systems to understand
✅ Better HTTP semantics

Instead of inventing custom search endpoints like:

POST /users/search
POST /orders/filter
POST /products/query

Applications can eventually expose cleaner APIs using QUERY.

A Practical Example

Imagine you're building an analytics dashboard.

Today's API might look like this:

POST /analytics/report

Request body:

`{
"dateRange": {

"from": "2026-01-01",
"to": "2026-06-30"

},
"groupBy": "country",
"metrics": [

"sales",
"profit"

]
}`

The request doesn't create anything.

It simply retrieves data.

QUERY expresses that intent much more clearly.

QUERY /analytics/report

Same body.

Better semantics.

What About Spring Boot and Other Frameworks?

If you're a Java developer like me, you've probably written endpoints such as:

@PostMapping("/search")

even though they're only returning data.

At the time of writing, most frameworks—including Spring Boot, Express, FastAPI, ASP.NET Core, and many API gateways—don't yet provide first-class support for QUERY because the standard is still very new.

As ecosystem support grows over the next few years, it will be interesting to see how frameworks adopt it.

Should You Use QUERY Today?

Probably not in production.

Although the method has now been standardized, widespread support is still limited.

Many HTTP clients, frameworks, reverse proxies, API gateways, and developer tools have yet to implement it.

For now, the common approach remains:

Use GET for simple resource retrieval.
Use POST when you need complex search bodies and QUERY isn't available.
Keep an eye on framework support as the ecosystem evolves.

Final Thoughts

HTTP has remained remarkably stable over the years, and that's a good thing.

But there has always been a missing piece between GET and POST.

QUERY fills that gap by providing a standard way to perform complex, read-only requests without sacrificing HTTP semantics.

Will it replace GET?

No.

Will it replace POST?

Not entirely.

Instead, it gives developers a better tool for scenarios where neither GET nor POST feels quite right.

It may take a few years before QUERY becomes common in production APIs, but understanding it today will help you design better APIs tomorrow.

What do you think about the new QUERY method?

Would you use it once your favorite framework supports it? Let me know in the comments!

Thanks for reading! If you enjoyed this article, consider leaving a ❤️ and following me here on DEV Community for more content on Java, Spring Boot, backend engineering, and modern web technologies.

Why Your Node.js App Crashes Under Traffic: The Magic of Connection Pooling

Sun, 20 Sep 2026 08:56:50 +0000

Imagine launching your shiny new web application. During development, everything runs smoothly. But the moment you launch on social media and a few hundred users hit your API at the exact same time, your server grinds to a halt and starts throwing nasty database connection timeout errors. This common disaster is almost always caused by a lack of database connection pooling.

What is Connection Pooling?

Database connection pooling is a performance optimization technique where a set of active database connections is kept open and shared among multiple requests. Instead of creating and destroying a new database connection every single time an application needs to run a query, the application borrows an existing connection from a pre-allocated pool, executes the query, and immediately returns the connection back to the pool. This prevents the costly overhead of starting new communication channels from scratch.

The Rental Car Analogy

To understand this concept, think of a rental car agency at an airport. Imagine if every time a traveler landed at the airport, the rental agency had to buy a brand-new car from the factory, register it with the DMV, customize it for the driver, and then completely crush and destroy the car at the end of the traveler's weekend trip. That would be an absurdly slow and expensive process.

Instead, the agency maintains a "pool" of 50 pre-registered, fully fueled, ready-to-drive cars parked in a lot. When a traveler arrives, they borrow a car from the lot, use it for their trip, and then return it to the lot. The car is washed and immediately made available for the next traveler. This is connection pooling: instead of building and destroying connections on the fly, you manage a recycled fleet of them.

Why It Matters Daily in the Tech Industry

Opening a database connection is an incredibly resource-intensive operation. To establish a new connection, your backend application and database server must complete a network handshake, negotiate security protocols (like TLS/SSL), authenticate user credentials, and allocate dedicated RAM and CPU threads to handle that specific session.

If you build an application where every user request opens a brand-new connection, a sudden spike in web traffic will force your Node.js server to attempt to open hundreds of database connections simultaneously. Your database server (like MySQL) has a built-in safety limit of maximum allowable concurrent connections. Once that limit is hit, the database will aggressively reject all incoming connection requests, throwing "Too many connections" errors, causing API endpoints to return 500 errors, and crashing your platform. Connection pooling solves this by capping the total number of open database connections and keeping extra queries safely queued up in memory until a connection becomes available.

Connection Pooling in Action (Node.js & MySQL)

Here is how you implement connection pooling using Express and the standard mysql2/promise library in a Node.js environment:

const express = require('express');
const mysql = require('mysql2/promise');
const app = express();

// 1. Create a global connection pool with configuration limits
const pool = mysql.createPool({
  host: 'localhost',
  user: 'root',
  password: 'secure_password',
  database: 'ecommerce_db',
  waitForConnections: true,    // Queue queries if no connections are free
  connectionLimit: 10,         // Maximum number of open connections
  queueLimit: 0                // No limit on the number of queued queries
});

app.get('/api/products', async (req, res) => {
  try {
    // 2. pool.query automatically borrows an idle connection, 
    // executes the query, and returns it to the pool.
    const [products] = await pool.query('SELECT id, name, price FROM products LIMIT 20');
    res.json(products);
  } catch (error) {
    console.error('Database failure:', error);
    res.status(500).json({ error: 'Internal Server Error' });
  }
});

app.listen(3000, () => {
  console.log('Server is running on port 3000');
});

The Takeaway

Software scalability is not just about writing clean logic or optimizing loops; it is about protecting and sharing finite hardware resources under heavy load. By implementing connection pooling, you protect your database from crashing, reduce API response latency, and build backend systems capable of handling traffic spikes gracefully without costing a fortune in extra server infrastructure.

Originally published on my blog. You can read the alternative breakdown here.

Rate Limiting Isn't One Layer: What LogicVisor and Titan Actually Do Differently

Sat, 19 Sep 2026 09:19:55 +0000

"Add rate limiting" sounds like one task. It's actually three separate layers, each defending against a different failure mode, with different trade-offs and different amounts of trust you can place in them. I only understood the distinction properly once I had to pick a strategy for two systems solving different problems: LogicVisor (a public AI tool anyone can hit anonymously) and Titan (a payments platform where the cost of a bad actor is different from the cost of someone burning through free AI credits).

1. Client-side: UX, not security

Debouncing a search input, greying out a submit button after the first click, backing off exponentially after a 429. All of this makes an app feel considerate. None of it stops anyone. A malicious actor skips your JavaScript entirely and hits the endpoint directly with curl. Client-side limiting is worth doing (it saves you real traffic and prevents accidental double-submits), but it is not a security control. If it's the only thing standing between your API and abuse, you don't have rate limiting, you have a polite suggestion.

2. Server-side: the layer that actually protects you

This is where LogicVisor and Titan diverge, because they're not defending against the same thing.

LogicVisor: several checks before a single AI token gets spent

LogicVisor is public. Anyone gets 3 free code reviews with no signup, which means the abuse surface is wide open by design. The submission route runs a stack of checks before it ever calls Gemini or Groq, because every AI call costs real money:

// 1. Check if this exact code has already been reviewed by this model
const cachedReview = await getCachedAIReview(preferred_model.id + "-" + canonicalHash);
if (cachedReview) {
  return NextResponse.json({ success: true, data: cachedReview }, { status: 201 });
}

// 2. Enforce the actual rate limit (IP + session based for anon users)
const rateLimitResult = await enforceAIRateLimit(user.id, request);

// 3. Slow down premium users intelligently instead of hard-blocking them
const throttleDelay = await getThrottleDelay(user.id);
if (throttleDelay > 0) {
  await new Promise((resolve) => setTimeout(resolve, throttleDelay));
}

// 4. Under heavy load, degrade gracefully instead of rejecting outright
const shouldDegrade = await shouldGracefullyDegrade(user.id, "ai_request");
if (shouldDegrade) {
  // return a basic, non-AI response instead of a 429
}

A few things worth naming separately, because they get lumped together under "rate limiting" but aren't the same mechanism:

Abuse prevention: anonymous sessions are capped at 3 reviews, tracked by a UUID session (7-day expiry) plus a browser fingerprint (User-Agent + Accept-Language + Accept-Encoding). This is the classic server-side layer, IP and fingerprint based, closest to the "token bucket per identity" pattern.
Cost control via deduplication: before rate limiting even runs, the code is canonicalized (AST-based) and hashed. If someone (or the same person twice) submits identical logic, they get the cached review instead of triggering a new paid AI call. This isn't rate limiting in the strict sense, it's closer to database-level duplicate detection, but it does the same job of protecting a scarce resource.
Graceful degradation over hard rejection: instead of a flat 429 when usage spikes, authenticated users can get a stripped-down, non-AI response. It costs nothing and still gives the user something, rather than a wall.

None of this is a single algorithm out of a textbook. It's several cheap, layered checks, ordered so the most expensive resource (the AI call) is the last thing hit, not the first.

Titan: NestJS Throttler backed by Redis

Titan's rate limiting is intentionally boring by comparison, and that's the correct choice for what it is. It's @nestjs/throttler wired up as a global guard, with Redis swapped in as the storage backend instead of the default in-memory store:

ThrottlerModule.forRootAsync({
  imports: [ConfigModule],
  inject: [ConfigService],
  useFactory: (configService: ConfigService) => ({
    throttlers: [
      {
        ttl: 60000,
        limit: 10,
      },
    ],
    storage: new ThrottlerStorageRedisService(
      configService.get<string>('REDIS_URL'),
    ),
  }),
}),

// registered alongside the JWT guard as a global APP_GUARD
{
  provide: APP_GUARD,
  useClass: ThrottlerGuard,
},

10 requests per 60 seconds, globally, enforced before a request even reaches a controller. No per-route tuning yet, no per-user tiers, just a flat ceiling applied everywhere via APP_GUARD.

The reason Redis matters here isn't exotic: an in-memory counter only knows about requests hitting that one process. The moment you run more than one instance behind a load balancer, in-memory rate limiting stops being true rate limiting, each instance is independently under-counting. Swapping the storage to ThrottlerStorageRedisService gives every instance a shared, consistent view of who's made how many requests, which is the actual requirement once you're not running a single box.

LogicVisor's layered, identity-aware approach makes sense for a system where the main threat is "someone is farming free AI reviews." Titan's flat, Redis-backed throttle makes sense for a system where the main threat is "someone is hammering an endpoint," and the priority is consistency across instances over nuance per user.

3. Database-level: the layer I'm deliberately not using yet

Postgres and Supabase both support rate limiting closer to the data layer (unlogged tables with triggers, PL/pgSQL token bucket functions, TTL-based counters in Mongo). I don't use this in either project, and I don't think I should yet. It's the right call for protecting specific high-value operations (financial writes, heavy analytical queries) where you need the database itself to be the source of truth and can't tolerate a race between the app layer and the DB. LogicVisor's canonical-hash cache lookup is adjacent to this idea (checking the data layer before doing expensive work), but it's a caching pattern, not a rate limiter, and I'd rather keep that distinction honest than dress it up as something it isn't.

What actually mattered, in order

Client-side limiting is UX. Treat it as UX. It's not a line item in your security review.
Server-side is where the real decision lives, and the right shape of it depends on what you're protecting: identity-based abuse (LogicVisor) versus flat request volume across a distributed system (Titan). They are not the same problem and don't need the same solution.
The most expensive resource should be the last thing your code touches. LogicVisor checks a cache, then a rate limit, then a throttle delay, before it ever calls an LLM. Ordering is a rate limiting decision too, even when it's not framed as one.
Database-level limiting is a tool for a specific job, not a default. I'm not reaching for it until there's an operation that actually needs that guarantee.

Cover photo by HsinKai Tai on Unsplash

Streaming vs JSON: Trade-offs in AI-Powered Apps

Sat, 19 Sep 2026 08:33:43 +0000

This is a breakdown of a decision I made building LogicVisor, not a general "streaming vs JSON" guide. AI API responses aren't limited to text: some return images, audio, structured data. I only needed text, so streaming vs JSON became a real fork in the road. If I'd needed image responses too, this calculus changes entirely.

Two constraints were in tension from the start:

The AI evaluation needed to be stored in the database.
The same information needed to be available to the user immediately, at submission time.

Why you can't reliably stream JSON

JSON has to be parsed before it's usable, and you can't parse it in pieces as it arrives. It needs to be complete, start to finish, before JSON.parse() does anything with it. That forces a binary choice: stream, or don't. Streaming, in my case, meant raw text, because that's all the underlying APIs stream.

I started with streaming. Feedback was near-instant on the user end, and it felt great. Then I needed a more structured response, for both the user view and the database, so I switched the request over to ask for JSON instead of free text:

const groqParams: ChatCompletionCreateParamsNonStreaming = {
  messages: [{ role: "user", content: prompt.content }],
  model: groqModel,
  stream: false,
  response_format: {
    type: "json_object",
  },
};

const groqResponse = await groq.chat.completions.create(groqParams);
aiReviewText = groqResponse.choices[0].message.content ?? "";

stream: false plus response_format: json_object is the whole trick. The model still wraps its answer in a markdown code fence half the time regardless of what you ask for, so I still need to strip that off before parsing:

export function extractJSONFromMarkdown(markdownString: string) {
  let jsonString = markdownString.trim();

  if (jsonString.startsWith("```json")) {
    jsonString = jsonString.substring(7);
  }
  if (jsonString.endsWith("```")) {
    jsonString = jsonString.substring(0, jsonString.length - 3);
  }

  const parsedData: PromptResult = JSON.parse(jsonString.trim());
  return parsedData;
}

I got the data shape I wanted. What I lost was the user experience. Now the user was staring at a loading state for the full response time instead of watching text stream in.

What I actually learned

The streaming version was never faster in absolute terms. It just felt faster, because you're more patient when you can see progress happening. Streaming exploited that. JSON removed it.

Since I couldn't stream JSON (parsing requires a complete payload) and I couldn't get structured data out of raw streamed text without hacking something together after the fact, I stopped trying to force one approach to do both jobs. I fixed the UX problem directly instead: loading spinners, pulsers, rotating status text. Not real progress, but it reads as progress, and that's what the user actually responds to.

Two paths, two jobs

When I added anonymous, free-trial reviews, the storage requirements dropped. Anonymous users didn't need the same depth of data persisted as authenticated ones. So that path defaults to streaming, for the fastest perceived response, which is exactly what you want when you're trying to hook a new user during a free trial:

const groqStream = await groq.chat.completions.create({
  messages: [{ role: "user", content: prompt.content }],
  model: groqModel,
  stream: true,
});

for await (const chunk of groqStream) {
  const text = chunk.choices[0]?.delta?.content ?? "";
  fullText += text;

  controller.enqueue(
    encoder.encode(`data: ${JSON.stringify({ type: "content", content: text })}\n\n`),
  );
}

Same groq.chat.completions.create call as before, just stream: true instead of false, and no response_format. Each chunk gets pushed to the client over server-sent events as it arrives, and I accumulate fullText on the server side so I've still got the complete response to hand off for logging once the stream closes.

(The real implementation also retries on 503s with backoff and sends a "waking up" notice on cold starts. Left out here since it's not specific to the streaming vs JSON decision, just general resilience around a serverless AI provider.)

Authenticated users, who need full structured data persisted, get the non-streaming JSON path from the first section, feeding straight into a database write:

const result = extractJSONFromMarkdown(aiReviewText ?? "");

const review = await createAIReview({
  user_id: user.id,
  time_complexity: result?.time_complexity ?? "",
  overall_grade: result?.overall_grade ?? "",
  markdown_review: result?.markdown_review ?? "",
  submission_id: submission.id,
});

Two code paths, one architecture, each one earning its keep for a different constraint: speed for anonymous trial users, structure for authenticated ones.

The takeaway

Building AI-powered products (or honestly, any product) comes down to picking the trade-off that serves your actual goal, not the one that looks cleanest on paper. Streaming and JSON aren't competing solutions here, they're two tools solving two different problems in the same app.

Cover photo by Myles Bloomfield on Unsplash

How I Eliminated 19,494 Duplicate MongoDB Records (And What I'd Do Differently Today)

Sat, 19 Sep 2026 08:33:15 +0000

At a previous organization, I was handed ownership of a job platform serving 1,000+ users worldwide. Jobs on the platform could be scoped to a single city, a state, a whole country, or "available worldwide." Simple enough. At least, you'd think so.

When I joined, this was early in my career and I was a frontend dev. I was interested in backend work but hadn't done any real backend engineering yet — I wasn't battle-tested. Somewhere along the line, I got handed the whole platform. Frontend, backend, all of it. Bracing moment. I forced myself to slow down, actually understand the system end to end, and build a real plan before touching anything.

After talking with product management, I pulled up the backend task backlog. I learn best by doing, so I started at the bottom — smallest bugs first, working my way up. It went well for a while. Then I hit a ticket tied to core business logic.

The Wake-Up Call

Some endpoints were clocking API response times north of 7,000ms. I genuinely thought it was a typo. Then I remembered running into something similar on the frontend side before — so I patched the UX so users never felt the full weight of the delay. But the actual problem was still sitting there, untouched.

Before touching any code, I made a checklist. We were running MongoDB with NestJS:

What schemas was the endpoint touching?
What optimization had already been applied to those schemas?
How was the endpoint fetching data?
How many records did each related schema currently hold?

Checklist in hand, I went into the codebase.

My innocent dev brain was not prepared for the chaos I found.

What Was Actually Happening

Here's the setup: assume an empty jobs collection. A recruiter creates a job listing scoped to New York City. The backend takes the payload and creates one record with location set to New York City. Straightforward.

Now assume another empty collection, and this time the job is scoped to the United States instead of a single city. What do you think happens?

If you guessed "the same thing — one record, location set to United States" — I have bad news for you.

What the backend actually did: it looked at the payload, recognized "United States" as a country rather than a city, looked up every known city in the U.S. — and created a separate job record for each one.

Read that again.

Instead of one record, the platform generated one identical record per city, differing only in _id and location. And it got worse: a job marked "available worldwide" would spawn a record for every city on the planet.

This was a live product with real traffic. Querying that collection — even with indexes in place — was a nightmare.

For context on scale: a single job scoped to just the USA generated 19,495 records — one canonical listing plus 19,494 exact duplicates.


Database records (1 USA-wide job)	19,495
Exact duplicates among them	19,494
Query time on affected endpoints	~2.3s (worst cases exceeded 7s for worldwide-scoped jobs)

My first reaction was horrified confusion. After talking it over with management, I learned there was a reason for the original design — I won't get into it here, it's not the point of this piece. What mattered was that I had free rein to fix it if I could pull it off. Hold my Pokémon card.

The Constraints

Fixing the architecture wasn't the hard part conceptually — the hard part was fixing it under real constraints:

We still needed to resolve which cities a job was available in, for every existing feature that depended on that.
Whatever I built had to slot into the existing infrastructure without a rewrite of everything downstream.
I needed to show measurable, provable performance gains — not just "trust me, it's better."
None of this could disrupt daily active users. No maintenance windows, no downtime.

Designing the Fix

I started with the schema. The core question: how do I preserve full location fidelity while collapsing tens of thousands of records into one?

I thought back to basic compression concepts — take data, strip it to a minimal representation, and pair it with the instructions needed to reconstruct the original. That's essentially what I needed here: a compact way to represent "this job is available in X" without one document per X.

Two deliverables:

A new, efficient schema.
A migration path that collapsed the old duplicated records into the new schema.

For the new schema, I kept every existing field and replaced the flat location string with a structured locationObj:

interface LocationObj {
  country: string | string[];
  state: string | string[];
  city: string | string[];
}

The logic:

Single city → city is a string, state and country resolve to that city's parent state/country.
A specific group of cities → city becomes a string array.
An entire state → city becomes the sentinel value "all", state holds the actual state name.
An entire country → state and city both become "all".
Worldwide → country, state, and city all become "all".

So a job available across Berlin, Munich, and Hamburg looks like:

{
  title: "Marketing Manager",
  company: "TechCorp",
  locationObj: {
    country: "Germany",
    state: "all",
    city: ["Berlin", "Munich", "Hamburg"]
  }
}

And a job available anywhere in the world:

{
  title: "Software Developer",
  company: "TechCorp",
  locationObj: {
    country: "all",
    state: "all",
    city: "all"
  }
}

One document. Full location semantics preserved. No fan-out.

Migrating Without Downtime

The deduplication piece ran as a batch script against the old collection: group records that shared everything except location (same title, company, description, salary — the actual "identity" of a job), reconstruct the correct locationObj from the set of location values in that group, and write a single new document into the new collection.

Once the new collection was validated against the old one, I built new endpoints on top of the new schema, mirroring the contracts of the old ones, and prepared the frontend to point at them.

The cutover itself was a backfill with two catch-up passes, not a one-shot migration:

Run the migration script once against the live old collection.
Right before deploying the new endpoints, re-run it — this catches any jobs created while the first pass was running, since the old collection kept accepting writes the entire time.
Deploy the new endpoints and switch the frontend over.
Re-run the script once more right after — this catches anything created under the old schema during the deploy window itself, in case the rollout wasn't instantaneous.

I tested the full path — frontend, backend, and DB — on local and staging before any of this touched production, so by the time it shipped, the swap itself was just a normal deploy. From a user's perspective, they kept seeing exactly the jobs they were supposed to see, the whole way through.

Querying the New Schema

The query logic has to walk the hierarchy — country, then state, then city — because "all" at any level short-circuits the rest:

const query = {
  $or: [
    { "locationObj.country": "all" },
    { "locationObj.country": userCountry, "locationObj.state": "all" },
    { "locationObj.country": userCountry, "locationObj.state": userState, "locationObj.city": "all" },
    { "locationObj.country": userCountry, "locationObj.city": userCity },
    { "locationObj.country": userCountry, "locationObj.city": { $in: [userCity] } }
  ]
};

Combined with a compound index on locationObj.country, locationObj.state, and locationObj.city, this scales linearly with the number of actual jobs — not the number of cities they happen to be available in.

The Results

Metric	Before	After	Change
Database records (1 USA-wide job)	19,495	1	99.99% fewer
Query time	~2.3s	~0.12s	~95% faster
Storage footprint	847 MB	0.043 MB	99.99% smaller
Index size	156 MB	0.008 MB	99.99% smaller
Memory usage	~4.2 GB	~0.21 MB	99.99% less RAM
Exact duplicate documents	19,494	0	Eliminated

Management was, unsurprisingly, impressed.

How I'd Do It Now

Looking back at this with a few more years under my belt, the schema still holds up. What I'd change is the migration and verification process — at the time, "backfill, then re-run twice around the cutover" felt rigorous. It worked. But it also relied on timing more than I'd want to admit if I were reviewing this today.

The more rigid version of this — the one I'd actually reach for now — is the expand-contract pattern:

Expand: Deploy dual-writes behind a feature flag — every job creation or update writes to both the old and new schema at the same time.
Sync: Use MongoDB Change Streams to mirror any old-schema writes into the new schema in real time during the transition, instead of relying on timed reruns of a migration script. This closes the race-condition window entirely — there's no gap to miss, because nothing is waiting for a scheduled pass.
Contract: Once reads are fully on the new schema and stable, kill the old write path.

I'd also make the migration script itself idempotent and checkpointed — track the last processed _id so a crash mid-run means it resumes instead of reprocessing or silently skipping documents. Mine was a script I babysat through completion; a checkpointed version is one I could walk away from.

On verification — "compare the counts" was good enough to convince management, but it wasn't rigorous. I'd now do a real reconciliation pass: diff the reconstructed locationObj against the actual set of cities in the old schema, per job, and flag any mismatch automatically instead of trusting a spot check. And before flipping reads over, I'd run a dark-launch period — querying both schemas in production and logging discrepancies without serving the shadow result to users — so any gap shows up before it's user-facing, not after.

One more thing I'd change that isn't about the migration at all: I'd want this caught before it ever shipped. The root failure was that nothing stopped a single job-creation event from silently producing thousands of documents. A write-layer guardrail — "creating one job should never result in more than one document, alert if it does" — plus a five-line regression test asserting exactly that, would have caught this in review instead of in production months later.

Lessons Learned

Embedding vs. referencing isn't the real question — normalization is. The original design wasn't wrong because it embedded or referenced incorrectly; it was wrong because it materialized every possible resolution of a hierarchical value instead of storing the hierarchy itself. The general MongoDB advice of "embed for read-heavy, reference for write-heavy" didn't apply here — the fix was closer to a compression problem than a modeling problem.

Sentinel values are underrated. Using "all" as a stand-in for "every value at this level" let one field type (string | string[]) represent five distinct scopes without a single boolean flag or separate scope enum.

Get the actual scale of the problem before designing the fix. I didn't just need "make it faster" — I needed a number to prove it. Knowing it was 19,495 records for one job is what made the case to management, and it's what makes this story worth telling.

Fields, numbers, and code above are reconstructed from a project where I owned the full backend under an NDA — details are generalized where necessary, but the architecture, the failure mode, and the metrics are real.

Cover photo by Mika Baumeister on Unsplash

Supercharge Your Node.js App: Why You Need Database Connection Pooling

Sat, 19 Sep 2026 08:17:43 +0000

What is Connection Pooling?

Database connection pooling is a technique where a cache of database connections is kept open and shared among multiple queries, rather than opening and closing a brand-new connection for every single query. When your application needs to talk to the database, it borrows an existing connection from the pool, runs the query, and immediately returns it. This avoids the heavy performance penalty of creating and destroying connections constantly.

The Pizza Scooter Analogy

Imagine a busy local pizza shop. Every time a customer orders a pizza, if the shop had to buy a brand-new delivery scooter from the dealership, register it, insure it, deliver the pizza, and then scrap or sell the scooter immediately afterward, the business would go bankrupt in an hour.

Instead, the shop maintains a stable fleet of five delivery scooters parked in the back. When a delivery is ready, a driver grabs an available scooter from the fleet, completes the delivery, and parks it back in the lot for the next driver to use. Connection pooling is that stable fleet of scooters.

Why Connection Pooling Matters Daily

In backend development, establishing a connection to a database like MySQL is an incredibly heavy operation. It requires several steps:

A TCP network handshake.
SSL/TLS security negotiation.
User authentication and privilege verification.
Memory allocation on the database server.

If your Express.js server handles 1,000 requests per second and you open a new connection for each one, your database server's CPU will spike to 100% just managing handshakes. Eventually, the database will run out of resources and throw a dreaded "Too many connections" error, crashing your app. By using a connection pool, you bypass this startup overhead. Your queries run almost instantly because the connection is already active and authenticated.

Simple Code Example

Let's see the difference in a MERN stack application using Node.js, Express, and the mysql2 package.

The Bad Way (New Connection per Request)

const express = require('express');
const mysql = require('mysql2/promise');
const app = express();

const dbConfig = { host: 'localhost', user: 'root', database: 'store' };

app.get('/products', async (req, res) => {
  try {
    // Slow and resource-intensive: creating a connection on every request
    const connection = await mysql.createConnection(dbConfig);
    const [rows] = await connection.execute('SELECT * FROM products LIMIT 10');

    // Closing it manually
    await connection.end();
    res.json(rows);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.listen(3000);

The Good Way (Using a Connection Pool)

const express = require('express');
const mysql = require('mysql2/promise');
const app = express();

const dbConfig = { host: 'localhost', user: 'root', database: 'store' };

// Create a pool of up to 10 reusable connections
const pool = mysql.createPool({
  ...dbConfig,
  connectionLimit: 10,
  waitForConnections: true,
  queueLimit: 0
});

app.get('/products', async (req, res) => {
  try {
    // Automatically borrows an active connection and returns it when done
    const [rows] = await pool.execute('SELECT * FROM products LIMIT 10');
    res.json(rows);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.listen(3000);

Key Takeaway

Database connection pooling is not just a performance optimization; it is a fundamental survival mechanism for production databases. It turns chaotic database traffic into a predictable, queued stream, ensuring your server remains responsive and your database stays healthy under heavy load.

Originally published on my blog. You can read the alternative breakdown here.

Code Fingerprinting: Detecting Duplicate Submissions Without Losing Your Mind (or Your API Budget)

Sat, 19 Sep 2026 07:49:18 +0000

While building LogicVisor's review system, I was faced with a question I hadn't thought about before: what would happen if users submitted duplicate algorithm solutions?

I decided to test it myself. I submitted a solution to a simple palindrome challenge. The request went through, my solution was reviewed correctly. Normal happy stuff. Then I submitted the exact same solution a second time. And the normal happy stuff happened again. But I was far from happy.

That duplicate submission meant more tokens spent, more database memory consumed, more remote procedure calls, more AI calls, and more waiting time. For a problem that had already been solved.

I shuddered at the thought of a bad actor effectively DOS-ing my site or running up my AI bill just by hammering the same solution repeatedly. Rate limiting controls the number of requests per client — but it has nothing to do with the content of those requests. A user could stay well within their rate limit and still abuse the system by submitting the same code over and over.

This wasn't going to work. I needed a way to identify duplicate submissions and return a cached response instead of processing everything from scratch. Which led me to a more interesting question: how do you actually detect identical code?

The Problem With "Identical"

Programming by its very nature allows us to solve the same problem in multiple ways. But even a single solution can be written in an infinite number of ways.

User A could create a function that takes and returns a string, and name it strawberries. User B could write the exact same function with the exact same logic and name it bananas. Different names, same implementation, same result.

For LogicVisor's purposes, these should be counted as duplicates. But how do you get a computer to see past the surface differences and recognize that two pieces of code are structurally the same?

I started thinking about it from first principles. At the end of the day, programming languages get compiled down to binary and then to electrical pulses. If two solutions solve the same problem the same way, they're ultimately firing the same set of pulses at the machine level. Variable names be damned.

So the question became: can I strip a solution down to its most fundamental representation and compare that?

Enter the Abstract Syntax Tree

More research led me to the Abstract Syntax Tree — something my OS professor had mentioned in school that I'd filed away as "probably never useful in web dev." No knowledge is wasted.

An Abstract Syntax Tree (AST) is a hierarchical, tree-shaped data structure that represents the syntactic structure of source code. It strips away superficial syntax — semicolons, curly braces, whitespace, comments — and keeps only the structural and semantic skeleton of the code. The shape of what you wrote, not the clothing it's wearing.

This was it. But I hit a wall almost immediately: I can't store and index an entire AST reliably in PostgreSQL. It's a complex nested structure, not a scalar value you can slap an index on.

Then I remembered an article about handling duplicate file uploads in Express with Multer. The trick was generating a SHA-256 hash of the file's content. A hash is a fixed-length fingerprint — the same input always produces the same output, and different inputs produce different outputs. Perfect.

The plan crystallized:

Parse the submitted code into an AST
Normalize the AST (rename variables, strip comments, etc.)
Hash the normalized output with SHA-256
Store and index that hash
On every new submission, compute the hash and check if it already exists

If the hash is in the database, return the cached review. No AI call. No wasted tokens. No waiting.

Two Parsers for the Price of One

LogicVisor supports eight languages: JavaScript, TypeScript, Python, Java, C, C++, Go, and Rust. Getting an AST for all of them required two different tools, each with their own approach to the world.

Babel Parser (for JavaScript and TypeScript)

Babel is something most JavaScript developers know as a transpiler — the thing that converts modern JS to something older browsers can run. But under the hood, Babel has a full-featured parser that produces a detailed AST of your JavaScript or TypeScript code. The @babel/parser package exposes this directly.

I'd used Babel's output for years but never touched the parser API directly. It's surprisingly clean. You hand it source code and options, and it gives you back a structured AST you can traverse and manipulate.

What made Babel the right choice for JS/TS specifically:

Native TypeScript support via a plugin — no separate toolchain
JSX support out of the box
The @babel/traverse package lets you walk the AST and modify nodes in place, which is exactly how I rename identifiers
@babel/generator converts the modified AST back to compact source code

The JS/TS canonicalization path: parse with Babel → traverse the AST and rename all non-builtin identifiers to generic names (v0, v1, v2...) → optionally sort top-level declarations → regenerate compact source → hash it.

Tree-sitter (for everything else)

Tree-sitter is a different beast. It's a parser generator — a tool for building fast, incremental, error-tolerant parsers for programming languages. You might have encountered it without knowing it: if you've used Neovim with syntax highlighting that actually understands your code structure (my first encounter with the tool), that's Tree-sitter. Same if you've used editors where renaming a variable highlights every reference correctly. It's doing real parsing, not just regex matching.

What makes Tree-sitter interesting for this use case is that it has community-maintained grammars for a huge range of languages — C, C++, Go, Rust, Java, Python, and many more. Each grammar is a separate package that teaches Tree-sitter how to parse that language. You install the grammar, set it on a parser instance, and Tree-sitter gives you a concrete syntax tree you can walk.

I hadn't really understood what Tree-sitter was before this. I'd seen it referenced in Neovim config files and assumed it was some Vim-specific thing. It's not — it's a general purpose parsing library with bindings for multiple languages including Node.js via the node-tree-sitter package.

The Tree-sitter path: parse the source → walk the resulting syntax tree with a DFS traversal → collect all identifier nodes by their byte ranges → build a name mapping (first-occurrence order) → apply replacements from the end of the file backwards (so byte indices don't shift) → normalize whitespace → hash it.

The backwards-replacement trick is worth calling out. When you're doing string replacements by index, if you replace something early in the file, every subsequent index is now wrong. By sorting replacements by start position descending and working from the end, you avoid that entire class of bug.

The Implementation

Here's how it all fits together.

The Simple Canonicalization (Pre-AST)

Before I landed on the AST approach, I had a simpler version that just handled whitespace and comments:

export function canonicalizeCode(code: string): string {
  // Remove multi-line comments
  let cleanedCode = code.replace(/\/\*[\s\S]*?\*\//g, "");

  // Remove single-line comments
  cleanedCode = cleanedCode.replace(/\/\/.*$/gm, "");

  // Normalize all whitespace to a single space
  cleanedCode = cleanedCode.replace(/\s+/g, " ").trim();

  return cleanedCode;
}

This catches the obvious cases — same code with different formatting or comments. But it falls apart the moment someone renames a variable. The AST approach handles that.

The Router

The main entry point checks the language and dispatches to the right parser:

export async function canonicalizeCodeAST(
  code: string,
  language: LanguageKey,
  options?: CanonicalizeOptions
): Promise<string> {
  const opts = { ...defaultOptions, ...(options || {}) };

  if (language === "javascript" || language === "typescript") {
    return canonicalizeJS(code, opts);
  }

  // C/C++/Go/Rust/Java/Python go through Tree-sitter
  return canonicalizeWithTreeSitter(code, language, null);
}

The Babel Path (JS/TS)

function canonicalizeJS(
  code: string,
  opts: Required<CanonicalizeOptions>
): string {
  const plugins: any[] = ["jsx"];
  if (opts) plugins.push("typescript");

  const ast = babelParser.parse(code, {
    sourceType: "module",
    plugins,
    allowReturnOutsideFunction: true,
  });

  // Rename all non-builtin identifiers to generic names
  const nameMap = new Map<string, string>();
  let counter = 0;

  traverse(ast as any, {
    Identifier(path) {
      if (!opts.normalizeVariableNames) return;

      const name = path.node.name;
      if (isBuiltinJsName(name)) return;

      if (!nameMap.has(name)) {
        nameMap.set(name, `v${counter++}`);
      }

      path.node.name = nameMap.get(name)!;
    },
  });

  // Optionally sort top-level declarations for order-independent comparison
  if (
    opts.sortTopLevel &&
    (ast as any).program &&
    Array.isArray((ast as any).program.body)
  ) {
    (ast as any).program.body.sort((a: any, b: any) => {
      const aStr = generate(a).code;
      const bStr = generate(b).code;
      return aStr.localeCompare(bStr);
    });
  }

  // Regenerate compact source from the modified AST
  return generate(ast as any, {
    comments: !opts.removeComments,
    compact: true,
  }).code;
}

The Tree-sitter Path (Everything Else)

function canonicalizeWithTreeSitter(
  code: string,
  language: LanguageKey,
  opts: Required<CanonicalizeOptions> | null
): string {
  const LangCtor = (treeSitterMap as any)[language];

  if (!LangCtor) {
    throw new Error(`Unsupported language for tree-sitter: ${language}`);
  }

  const parser = new Parser();
  parser.setLanguage(LangCtor);

  const tree = parser.parse(code);
  const root = tree.rootNode;

  const identifierNodeTypes = identifierTypeMap[language] || ["identifier"];
  const idNodes: { start: number; end: number; text: string }[] = [];
  const commentNodes: { start: number; end: number }[] = [];

  // DFS traversal to collect identifier and comment nodes
  const stack = [root];
  while (stack.length) {
    const node = stack.pop()!;
    const type = node.type;

    if (type.includes("comment") && opts?.removeComments) {
      commentNodes.push({ start: node.startIndex, end: node.endIndex });
    }

    if (identifierNodeTypes.includes(type) || /identifier|name/i.test(type)) {
      const text = code.slice(node.startIndex, node.endIndex);
      idNodes.push({ start: node.startIndex, end: node.endIndex, text });
    }

    for (let i = 0; i < node.childCount; i++) {
      stack.push(node.child(i) as SyntaxNode);
    }
  }

  // Build name mapping in first-occurrence order
  const nameMap = new Map<string, string>();
  let counter = 0;

  for (const n of idNodes) {
    const orig = n.text;
    if (isLanguageBuiltinName(orig, language)) continue;
    if (!nameMap.has(orig)) {
      nameMap.set(orig, `v${counter++}`);
    }
  }

  // Build replacement list
  const replacements: { start: number; end: number; replacement: string }[] = [];

  if (opts?.removeComments) {
    for (const c of commentNodes) {
      replacements.push({ start: c.start, end: c.end, replacement: "" });
    }
  }

  if (opts?.normalizeVariableNames) {
    for (const id of idNodes) {
      const mapped = nameMap.get(id.text);
      if (mapped) {
        replacements.push({ start: id.start, end: id.end, replacement: mapped });
      }
    }
  }

  // Apply replacements from end to start so byte indices stay valid
  replacements.sort((a, b) => b.start - a.start);

  let out = code;
  for (const r of replacements) {
    out = out.slice(0, r.start) + r.replacement + out.slice(r.end);
  }

  return out.trim();
}

Hashing the Canonical Form

Once we have the normalized source, hashing is straightforward:

export async function createCanonicalHash(canonicalCode: string): Promise<string> {
  const encoder = new TextEncoder();
  const data = encoder.encode(canonicalCode);
  const hashBuffer = await crypto.subtle.digest("SHA-256", data);
  const hashArray = Array.from(new Uint8Array(hashBuffer));
  return hashArray.map((b) => b.toString(16).padStart(2, "0")).join("");
}

SHA-256 gives us a fixed 64-character hex string regardless of how long the original code was. That's what gets stored and indexed in PostgreSQL.

Putting It All Together in the Request Handler

let canonicalCode;
try {
  canonicalCode = await canonicalizeCodeAST(solution, preferred_language);
} catch (error) {
  console.error("Canonicalization error:", error);
  return NextResponse.json(
    {
      success: false,
      error: {
        code: "BAD_USER_INPUT",
        message: "Unable to parse your code. Please check for syntax errors and try again.",
      },
    },
    { status: 400 }
  );
}

const canonicalHash = await createCanonicalHash(
  typeof canonicalCode === "string" ? canonicalCode : ""
);

// Check cache before touching the AI provider
const cachedReview = await getCachedAIReview(
  preferred_model.id + "-" + canonicalHash
);

if (cachedReview) {
  return NextResponse.json(
    { success: true, data: cachedReview },
    { status: 201 }
  );
}

// No cache hit — proceed with AI review and store the result

The model ID is included in the cache key because the same code reviewed by Gemini and Groq will produce different outputs. You don't want to serve a Gemini review to someone who selected Groq.

Does It Actually Work?

Here's the test that proved it out. Five TypeScript implementations of the same palindrome algorithm — different variable names, different formatting, different comments, and one with operand order swapped:

// Solution 1 — clean, descriptive names
function isPalindrome(x: number): boolean {
  let originalNum = x;
  let reversedNum = 0;
  while (originalNum > 0) {
    let lastDigit = originalNum % 10;
    reversedNum = (reversedNum * 10) + lastDigit;
    originalNum = Math.floor(originalNum / 10);
  }
  return x === reversedNum;
}

// Solution 2 — different variable names
function isPalindrome(n: number): boolean {
  let temp = n;
  let result = 0;
  while (temp > 0) {
    let digit = temp % 10;
    result = (result * 10) + digit;
    temp = Math.floor(temp / 10);
  }
  return n === result;
}

// Solution 3 — compressed formatting + comments
function isPalindrome(x: number): boolean{
  // Store original
  let originalNum=x;let reversedNum=0;
  while(originalNum>0){
    let lastDigit=originalNum%10; /* get digit */
    reversedNum=(reversedNum*10)+lastDigit;
    originalNum=Math.floor(originalNum/10);
  }
  return x===reversedNum;
}

// Solution 4 — addition operands swapped
function isPalindrome(x: number): boolean {
  let originalNum = x;
  let reversedNum = 0;
  while (originalNum > 0) {
    let lastDigit = originalNum % 10;
    reversedNum = lastDigit + (reversedNum * 10); // addition order changed
    originalNum = Math.floor(originalNum / 10);
  }
  return reversedNum === x; // comparison order changed
}

// Solution 5 — completely different algorithm
function isPalindrome(x: number): boolean {
  const str = x.toString();
  return str === str.split('').reverse().join('');
}

And the test harness:

export async function testDuplicateDetection() {
  const solutions = [ /* ...the five solutions above... */ ];

  console.log("Code hash comparison:");

  solutions.forEach(async (code, index) => {
    const canonicalized = await canonicalizeCodeAST(code, "typescript");
    const hash = await createCanonicalHash(canonicalized);
    console.log(`\nSolution ${index + 1}:`);
    console.log(`Hash: ${hash.substring(0, 12)}...`);
    console.log(
      `Canonical: ${
        typeof canonicalized === "string" ? canonicalized.substring(0, 100) : ""
      }...`,
    );
  });

  // Canonicalize before hashing — not after
  const hashes = await Promise.all(
    solutions.map(async (code) => {
      const canonicalized = await canonicalizeCodeAST(code, "typescript");
      return createCanonicalHash(canonicalized);
    }),
  );
  const uniqueHashes = new Set(hashes);

  console.log(`\nOriginal solutions: ${solutions.length}`);
  console.log(`Unique hashes: ${uniqueHashes.size}`);
  console.log(`Duplicates detected: ${solutions.length - uniqueHashes.size}`);
}

The output:

Code hash comparison:

Solution 1:
Hash: f7142f71e35a...
Canonical: function v0(v1:number):boolean{let v2=v1;let v3=0;while(v2>0){let v4=v2%10;v3=v3*10+v4;v2=Math.v5(v2...

Solution 2:
Hash: f7142f71e35a...
Canonical: function v0(v1:number):boolean{let v2=v1;let v3=0;while(v2>0){let v4=v2%10;v3=v3*10+v4;v2=Math.v5(v2...

Solution 3:
Hash: f7142f71e35a...
Canonical: function v0(v1:number):boolean{let v2=v1;let v3=0;while(v2>0){let v4=v2%10;v3=v3*10+v4;v2=Math.v5(v2...

Solution 4:
Hash: da8268fe24dd...
Canonical: function v0(v1:number):boolean{let v2=v1;let v3=0;while(v2>0){let v4=v2%10;v3=v4+v3*10;v2=Math.v5(v2...

Solution 5:
Hash: e3d2696010ad...
Canonical: function v0(v1:number):boolean{const v2=v1.v3();return v2===v2.v4('').v5().v6('');}...

Original solutions: 5
Unique hashes: 3
Duplicates detected: 2

Solutions 1, 2, and 3 collapse to the same hash — variable names, whitespace, and comments are all stripped away during canonicalization, leaving identical ASTs. Solution 4 gets a distinct hash because operand order (lastDigit + reversedNum 10 vs reversedNum 10 + lastDigit) is a syntactic difference the AST preserves — commutativity is math, not structure, and the canonicalizer doesn't evaluate expressions. Solution 5 is correctly identified as a genuinely different algorithm.
Two duplicates detected. Working as intended.

The Tradeoffs Worth Knowing

This approach isn't magic. A few things it doesn't catch:

Semantic equivalence across algorithms. Two completely different implementations that produce the same result — bubble sort vs. merge sort, for example — will have different hashes. This system detects structural duplicates, not algorithmic equivalence. That's actually the right behavior for a code review platform.

Heuristic identifier detection in Tree-sitter. The identifier node types vary between language grammars, and I'm using pattern matching (/identifier|name/i) as a fallback. It works well in practice but it's not exhaustive. If a grammar names its identifier nodes something unusual, you might miss some.

Builtin lists are incomplete. The isBuiltinJsName and isLanguageBuiltinName functions cover the most common builtins but aren't exhaustive. If a user names their variable fetch or setTimeout, those won't be normalized. Not a critical flaw, but worth knowing.

What I Took Away From This

The part I didn't expect: how much I'd enjoy it. This was one of those problems that pulled me away from the normal web dev loop of "call an API, render some data" and into something that felt closer to computer science. Parsers, ASTs, tree traversal — things I'd learned in school that felt abstract at the time suddenly had a real home in a real production system.

The other thing I took away: sometimes the right caching strategy isn't about time-to-live or cache invalidation. Sometimes it's about recognizing that two requests are asking for the same thing, even when they look different on the surface. That realization changed how I thought about a whole class of optimization problems.

If you're building anything that takes code as input — a linter, a reviewer, a plagiarism detector, an interview platform — the AST + hash pattern is worth having in your toolkit.

LogicVisor is an AI-powered tool that reviews your algorithmic code, breaks down time and space complexity, and gives you the kind of feedback you'd want before a technical interview. You can try it at logicvisor.vercel.app.

Why I Move AI Model Calls to the Server — Security, Performance, and Everything In Between

Sat, 19 Sep 2026 07:45:04 +0000

When I was building Logicvisor — an AI-powered tool that reviews your algorithmic code, breaks down time and space complexity, and gives you the kind of feedback you'd want before a technical interview — I had to make a foundational architectural decision early on.

Where do the AI calls actually live?

It sounds simple. It really isn't. And I think it's a decision a lot of developers make too quickly, usually defaulting to whatever gets something running fastest. So I want to walk through how I thought about it, what the tradeoffs actually look like in practice, and why for Logicvisor — and honestly most production projects I work on — the answer was never really up for debate.

First, Let's Establish What We're Even Choosing Between

When your app needs to talk to an AI model — Gemini, Claude, GPT, whatever — that HTTP request to the model provider has to originate somewhere. You have two options:

Client-side: The browser makes the call directly to the AI provider's API.

Server-side: The browser calls your server, your server calls the AI provider, and the response comes back through your infrastructure.

That's the whole decision. But the consequences of each branch run deep.

The Case for Client-Side AI Calls

Let's be fair to the other side first, because client-side AI calls aren't just laziness — there are legitimate reasons to reach for them.

Zero backend overhead. If you're prototyping, building an MVP, or hacking something together for a weekend project, standing up a server just to proxy AI calls adds friction you might not need yet. The client calls the API, gets a response, done.

One less network hop. Client → AI provider is a straight line. Client → your server → AI provider is also a straight line, but a longer one. Every additional hop is a potential source of latency, and if your server is not geographically close to the AI provider, that gap compounds.

Fast iteration during development. Tweak a prompt, refresh the page, see the result. No redeployment cycle, no server restart. For the early exploratory phase of building with AI, this feedback loop is genuinely valuable.

Fine for purely client-facing tools. If you're building something that doesn't touch your own database, doesn't need user sessions, and doesn't have sensitive business logic — a personal productivity tool, a browser extension, an internal utility — client-side calls can be perfectly appropriate.

So that's the honest upside. Now here's where it falls apart.

Why Client-Side AI Calls Break Down in Production

The API Key Problem

This is the most obvious one, but it's worth being precise about why it's as bad as it is.

When you make an API call from the browser, your API key has to be in that request. There's no way around this — the provider needs to authenticate you. And since that request is made from the browser, the key is accessible to anyone who opens DevTools, intercepts traffic, or extracts it from your bundled JavaScript.

The consequence isn't just that someone can see your key. It's that they can use it. At your expense. Without your knowledge. AI API billing is usage-based, which means a single bad actor with your key can run up a bill that drains your account before your monitoring even fires an alert — if you have monitoring at all.

Key rotation helps, but it's reactive. The damage is usually already done.

Your Prompt Engineering Is Public

This one gets less attention but matters more than people realize.

The prompts you write are often where your actual product value lives. If you've spent time crafting a system prompt that makes your AI reviewer give structured, consistent, high-quality feedback on algorithmic code — that prompt is the product. Client-side calls expose it completely. A competitor can open DevTools, read your system prompt, and replicate your core feature in an afternoon.

On the server, your prompts never leave your infrastructure. The client sends input; the server decides what to do with it.

You Have No Control Over Abuse

On the client side, there's nothing stopping a user from writing a script that hammers your AI endpoint in a loop. Every one of those requests hits the AI provider and costs you tokens. You have no rate limiting, no request validation, no way to enforce quotas per user.

You're not just vulnerable to malicious actors either — a bug in your own frontend code that causes unintended re-fetching can silently burn through your API budget.

No Caching

AI API calls cost money per token. If multiple users ask your tool to review functionally identical code, why would you want to pay for that same inference three hundred times?

On the client side, you can't cache at the API level. Every identical request goes to the provider, incurs latency, and costs tokens. On the server, you can cache responses intelligently — hash the input, check your cache layer, return the cached result. You pay once.

Without server-side infrastructure, you have no centralized view of how your AI layer is actually being used. Which prompts are performing well? Which inputs are producing garbage responses? Which users are hitting rate limits? Where is your token spend going?

Client-side AI calls mean you're guessing at all of this. Logs, monitoring, and observability — the basic instrumentation of a production system — require a server in the loop.

The Case for Server-Side AI Calls

With that context established, here's what you actually get when the AI calls live on the server.

The API Key Never Touches the Client

Your key lives in an environment variable on the server. The client has zero knowledge of it, zero access to it, and zero ability to extract it. This is the minimum acceptable security posture for any application that will see real users.

You Control Rate Limiting

You decide how many requests a given user can make in a given window. You can enforce this per account, per IP, per session — whatever your threat model calls for. Abuse becomes something you manage rather than something that happens to you.

export async function enforceAIRateLimit(
userId: string,
request?: NextRequest
): Promise<RateLimitResult> {
const tierLimits = await getUserTierLimits(userId);

if (!tierLimits) {
throw new RateLimitError("Unable to determine user tier limits");
}

// Free/Pro users: per-minute limits
if (tierLimits.ai_requests_per_minute !== null) {
const minuteLimit = await checkRateLimit(userId, "ai_request", "minute");

if (!minuteLimit.isAllowed) {
  await logRateLimitViolation(userId, "ai_request", "minute", ...);
  throw new RateLimitError(``Rate limit exceeded. Resets at ${minuteLimit.resetTime.toISOString()}``);
}

return minuteLimit;
}

// Admin users: daily + monthly
const dailyLimit = await checkRateLimit(userId, "ai_request", "daily");
// ...monthly check follows same pattern
}

Response Caching Becomes Possible

Identical or near-identical inputs can return cached results, cutting both latency and cost. For a tool like Logicvisor where multiple users might submit similar sorting algorithm implementations, the savings on repeated inferences compound quickly.

// Normalize the code to a canonical form before hashing
// so that formatting differences don't result in cache misses
const canonicalCode = await canonicalizeCodeAST(solution, preferred_language);
const canonicalHash = await createCanonicalHash(
  typeof canonicalCode === "string" ? canonicalCode : ""
);

// Check cache before hitting the AI provider
const cachedReview = await getCachedAIReview(
  preferred_model.id + "-" + canonicalHash
);
if (cachedReview) {
  return NextResponse.json(
    { success: true, data: cachedReview },
    { status: 201 }
  );
}

You Can Inject Server-Side Context Into Prompts

This is where things get architecturally interesting. The client sends raw input — code, a question, a request. But your server knows things the client doesn't: who the user is, what their history looks like, what tier they're on, what language they've selected, what results they've already received. All of that context can be injected into the prompt before it ever leaves your infrastructure.

The client can't fake or manipulate that context because it never touches it.

Full Observability

Every request is logged. Every response is traceable. You can monitor token usage, flag anomalous behavior, track which prompts produce the best results, and debug production issues with actual data. This is what running software in production looks like.

await updateAPIUsageAnalytics("/api/internal", true, responseTime, aiTokensUsed, estimatedCost, user.id, {
  modelName: modelUsed,
  modelProvider,
  inputTokens,
  outputTokens,
  totalTokens: aiTokensUsed,
  actualCostUsd,
});

The BFF Argument — One Round Trip Instead of Many

There's a broader architectural win here that goes beyond just AI calls. Without a server in the middle, the client has to orchestrate everything itself: call the AI, wait for the response, then maybe hit your database, wait again, then update the UI. Each of those is a visible pause for the user.

With a Backend for Frontend (BFF) pattern, the client makes one request. The server handles the AI call, processes the response, queries the database if needed, applies any business logic, and returns a single resolved payload. The user feels one network round trip instead of a cascading waterfall of them.

Architecture Diagrams

Before getting into the implementation specifics, here's the architectural difference visualised.

Client-Side AI Calls — The Problem

Browser → AI Provider directly · API key exposed in transit

Server-Side AI Calls — The Solution

Browser → API Route → [Cache · Rate Limiter · AI Provider · DB] → Browser

How This Played Out in Logicvisor

Let me get concrete. Here's what moving the AI layer to the server actually looked like in practice.

Performance — Collapsing the Waterfall

Logicvisor uses Supabase on the backend. At various points in the app, I need to pull data from multiple tables, run the AI review, and return everything the page needs in one shot.

If this were all happening on the client, you'd be looking at: call Supabase for user context → wait → call the AI provider → wait → call Supabase again for historical reviews → wait → render. Each of those waits is visible to the user, and each one is an opportunity for something to fail mid-chain.

On the server, those calls happen in close proximity to each other and to the data. The AI call, the database queries, and any necessary transformations all resolve server-side, and the client gets one clean response. The user experiences a single loading state, not a series of UI flickers.

There's also the matter of compute. Parsing and processing a large AI response — stripping JSON fences, validating structure, transforming the output into the format the UI expects — is work that browsers are not well-suited for. Browsers are memory and CPU constrained by design, and they're competing with the DOM, with other tabs, with everything the user has open. A server doesn't have those constraints.

Security — API Obfuscation as a Feature

Moving calls to the server means the client's entire interaction is scoped to your own API. It calls your endpoint, gets a response, done. It has no visibility into what your server does with that request internally — which external APIs it calls, what keys those calls carry, or how the response was constructed.

This isn't security through obscurity; the obfuscation is a structural property of the architecture. OS-level network tools could theoretically expose some of this, but you've dramatically raised the bar for what an attacker needs to do to compromise your stack.

Your system prompts — the part of the product that actually encodes your domain knowledge and review methodology — never leave the server. That's not a small thing.

Authentication — Stateless and Seamless

A server in the loop makes proper auth architecture dramatically cleaner. I was able to set HttpOnly cookies, attach signed JWT tokens, and build a stateless authentication and authorization system that the client participates in without controlling.

Without a server, you end up storing tokens in localStorage or client-side state, which is a well-documented attack surface. The session becomes something the client manages, which means it's something an attacker can manipulate.

Caching — Paying for Inference Once

Token costs are real. For Logicvisor, where users might submit variations of common algorithm patterns — bubble sort, binary search, dynamic programming problems — I can cache AI responses keyed on a normalized hash of the input. A user submitting a well-known algorithm implementation gets a fast, cached response. The AI provider gets called once.

This also improves response times for cached queries significantly. The round trip to an AI provider is the most expensive part of the request by a wide margin. Eliminating it for repeat queries is the single highest-leverage performance optimization available to you.

Provider Abstraction — Switching Models Without Touching the Client

Here's something the client-side approach makes nearly impossible: swapping AI providers without your frontend caring at all.

Logicvisor supports both Gemini and Groq depending on the user's selected model. Gemini for deeper analysis with its thinking budget and Google Search grounding, Groq for speed. Two different SDKs, two different response shapes, two different token counting strategies, two different pricing models. The client knows none of this. It sends a request, it gets a review back.

That abstraction only works because the AI calls live on the server:

switch (modelProvider.toLowerCase()) {
  case "google":
    // Gemini — thinking budget + Google Search grounding
    const response = await ai.models.generateContent({
      model: preferred_model.id,
      config: {
        thinkingConfig: { thinkingBudget: prompt.estimatedTokens },
        tools: [{ googleSearch: {} }],
        seed: SEED,
        temperature: TEMPERATURE,
      },
      contents,
    });
    aiReviewText = response.text ?? "";
    // Extract actual token counts from usageMetadata
    inputTokens = response.usageMetadata?.promptTokenCount || 0;
    outputTokens = response.usageMetadata?.candidatesTokenCount || 0;
    break;

  case "groq":
    // Llama 3.3 70B — forces structured JSON response
    const groqResponse = await groq.chat.completions.create({
      messages: [{ role: "user", content: prompt.content }],
      model: groqModel,
      temperature: TEMPERATURE,
      seed: SEED,
      response_format: { type: "json_object" },
      stream: false,
    });
    aiReviewText = groqResponse.choices[0].message.content ?? "";
    inputTokens = groqResponse.usage?.prompt_tokens || 0;
    outputTokens = groqResponse.usage?.completion_tokens || 0;
    break;
}

If I wanted to add a third provider tomorrow — say, Claude for a specific model tier — that's a new case block on the server. The client contract doesn't change. No frontend deployment, no API key exposure, no breaking changes for users mid-session.

Try doing this cleanly when the calls are in the browser. You'd be shipping provider-specific SDK logic, API keys, and token counting math directly to the client — and every provider swap would mean a frontend change. The server is the only place this kind of abstraction is clean.

This also matters for cost tracking. Notice that each branch extracts token counts differently — Gemini from usageMetadata, Groq from usage. That per-provider normalization feeds into the analytics pipeline downstream, giving you a consistent view of cost across providers regardless of how each SDK reports it. That's only possible because all the provider-specific handling is in one place.

AI Response Processing — Cleaning Up the Model's Output

Here's the unglamorous part that doesn't get written about enough: AI models don't always return clean output.

Gemini, for example, sometimes wraps JSON responses in markdown fences even when you've explicitly told it not to. If your UI is trying to parse that response and render structured data, you need to clean it before it gets anywhere near the client.

On the server, I handle all of that: strip the fences, validate the JSON structure, handle the cases where the model returned something unexpected, and only send a clean, predictable payload to the client. If something goes wrong at this layer, I can log it, inspect it, and fix the prompt. The client just sees a well-structured response or a proper error.

export function extractJSONFromMarkdown(markdownString: string) {
  try {
    // Remove the ```json wrapper
    let jsonString = markdownString.trim();
    
    // Remove ```json from start
    if (jsonString.startsWith("```json")) {
      jsonString = jsonString.substring(7);
    }
    
    // Remove ``` from end
    if (jsonString.endsWith("```")) {
      jsonString = jsonString.substring(0, jsonString.length - 3);
    }
    
    // Parse the JSON
    const parsedData: PromptResult = JSON.parse(jsonString.trim());
    
    // Extract and decode the markdown_review field
    let markdownReview = parsedData.markdown_review;
    if (markdownReview) {
      // Decode JSON escape sequences
      markdownReview = markdownReview
        .replace(/\\n/g, "\n")   // Convert \n to actual newlines
        .replace(/\\t/g, "\t")   // Convert \t to actual tabs
        .replace(/\\r/g, "\r")   // Convert \r to carriage returns
        .replace(/\\"/g, '"')    // Convert \" to actual quotes
        .replace(/\\\\/g, "\\"); // Convert \\ to actual backslashes
        
      return { parsedData, markdownReview };
    } else {
      console.log("No markdown_review field found");
      return null;
    }
  } catch (error) {
    console.error("Error parsing JSON:", error);
    return null;
  }
}

If this processing happened on the client, every user's browser would be doing it — inconsistently, with no visibility, and with no way to fix edge cases without a frontend deployment.

So Was It Ever Really a Debate?

Honestly, not for Logicvisor.

Client-side AI calls are a valid tool in specific, narrow contexts — personal tools, internal utilities, quick prototypes where security is not a concern and scale is not a goal. The moment you have real users, real API costs, and real data flowing through your system, the calculus changes completely.

Security, performance, observability, and maintainability all point in the same direction. The extra infrastructure is real overhead. But it's the kind of overhead that pays for itself the first time an abuse attempt hits your rate limiter instead of your API bill.

For any production system where AI inference is part of the core product — keep it on the server. Build the client thin. Let the backend do the heavy lifting.

We Rebuilt MYZ Payments Around a Canonical Ledger — Here’s What We Learned

Fri, 18 Sep 2026 16:46:27 +0000

We Rebuilt MYZ Payments Around a Canonical Ledger — Here’s What We Learned

When we started adding internal MYZ payments to MyZubster Marketplace, the obvious task looked simple:

buyer balance -= amount
seller balance += amount

That turned out to be the wrong level of abstraction.

The real challenge was not adding a payment button. It was making sure that every part of the application agreed on what a MYZ balance actually is.

During the implementation, we found multiple historical paths capable of representing or modifying MYZ independently.

That is exactly the kind of situation that eventually produces accounting bugs.

So instead of adding another payment path, we refactored the system around one rule:

There must be one canonical source of truth for spendable MYZ.

This post covers the architecture we implemented, the problems we found, and the trade-offs that are still open.

The original problem: balances were too easy to create

A field like this looks harmless:

{
userId,
balanceMYZ: 500
}

Until another subsystem has:

{
accountId,
myzCredit: 200
}

And another computes:

displayedBalance =
ledgerBalance +
mongoCredits -
utilityRedemptions;

Now the question becomes:

What is the user's real balance?

If multiple components can answer that question differently, the system no longer has a reliable accounting model.

That was the architectural problem we wanted to eliminate.

First principle: MYZ is an internal accounting unit

Before touching the code, we established the semantics.

MYZ is an internal MyZubster utility/accounting credit.

It is not automatically convertible into:

EUR
BTC
XMR
cash

and it is not represented by this implementation as an on-chain token.

This matters technically because it means we do not need an exchange-rate engine in order to spend MYZ.

If a Marketplace listing costs:

250 MYZ

the payment is simply:

250 MYZ

There is no intermediate:

250 MYZ → €X → seller

Likewise, a Zorgax plan can have an explicit MYZ price without deriving it from its EUR price.

The canonical ledger

The core service is:

src/services/myzLedgerApiService.js

Instead of mutating a user balance directly, transfers are recorded as ledger entries.

A payment creates two legs:

buyer -250 MYZ
seller +250 MYZ

Both are linked by the same:

transfer_id

Conceptually:

              transfer_id: T-123
             ┌─────────────────┐

buyer account │ -250 MYZ │

             │                 │

seller account │ +250 MYZ │

             └─────────────────┘

This gives us something a plain balance field does not provide naturally:

auditability.

We can answer:

Why is this balance 750 MYZ?

by looking at the entries that produced it.

Transfers are atomic inside the ledger

The transfer operation performs both accounting legs under the same write operation.

The important invariant is:

debit exists ⇔ credit exists

We do not want:

buyer -250

without:

seller +250

because the process crashed halfway through.

The ledger therefore validates the transfer first, then persists both entries together.

The relevant checks include:

valid payer
valid payee
payer != payee
valid amount
sufficient balance
valid idempotency key
no transfer ID collision

Only after those checks do the entries become recorded.

Never trust the UI to prevent overspending

One of the tests we cared about most was not:

Can Alice buy an item?

It was:

Can Alice buy two items at the same time
when she can only afford one?

Example:

Alice balance: 300 MYZ

Purchase A: 250 MYZ
Purchase B: 250 MYZ

Both requests may arrive concurrently.

A naïve implementation can do this:

if (balance >= 250) {
// request A sees 300
}

if (balance >= 250) {
// request B also sees 300
}

Then both succeed.

Final logical balance:

-200 MYZ

That is unacceptable for an accounting system.

Our expected result is:

one purchase succeeds
one purchase fails
final balance = 50 MYZ

The protection belongs in the ledger write path, not in React.

Idempotency became a first-class concept

Payment APIs are retried.

Browsers retry.

Networks fail.

Users double-click.

Servers can complete the financial operation and crash before returning 200 OK.

So this API:

POST /api/marketplace/orders/:id/payment/myz

requires an idempotency key.

A simplified request looks like:

Idempotency-Key: marketplace-order-123

If the exact operation is retried, we return the original transfer instead of charging again.

But there is another subtle problem.

What happens if somebody reuses the same idempotency key with a different payload?

For example:

key = abc
plan = Zorgax Pro
amount = 40 MYZ

followed by:

key = abc
plan = Zorgax Developer
amount = 40 MYZ

The amount is identical, but the operation is not.

So idempotency is bound not only to the numeric amount, but also to the semantic reference of the operation.

Same key + different meaning = conflict.

That protects against a class of bugs that simple "key already exists" implementations often miss.

Marketplace became a true buyer → seller transfer

For MYZ-priced Marketplace orders, the flow is now:

Buyer
↓
canonical MYZ transfer
↓
Seller
↓
order marked PAID

The order stores evidence of the accounting operation:

payment: {
asset: 'MYZ',
network: 'internal-ledger',
status: 'PAID',
transferId,
debitEntryId,
creditEntryId
}

We also enforce several business invariants.

The buyer must be authenticated.

The buyer must actually own the order.

Buyer and seller cannot be the same account.

The order must be accepted before payment.

And an unpaid MYZ order cannot transition to completed.

The important point is that these rules are checked server-side.

The React button is just UX.

Application state and ledger state are not the same transaction

There is an interesting systems problem here.

The MYZ ledger and the Marketplace order are not stored in the same transactional database.

So this can happen:

MYZ transfer succeeds
application crashes
order has not yet been marked PAID

You cannot solve that with a normal database transaction across unrelated stores.

Instead, we use an idempotent recovery model.

On retry:

ledger sees existing transfer
→ does not charge again

application receives original receipt
→ retries order persistence

This is closer to a small saga than a traditional ACID transaction.

The important requirement is recoverability.

Zorgax uses the same accounting primitive

We reused the same ledger for Zorgax plans.

The flow is:

user MYZ account

↓

canonical transfer

↓

zorgax:system:treasury

↓

grant entitlement

Prices are configuration values:

ZORGAX_PRO_PRICE_MYZ
ZORGAX_DEVELOPER_PRICE_MYZ

They are deliberately not calculated from:

priceEur

If the MYZ price is missing, the checkout fails closed.

Production currently reports the MYZ plan as unavailable instead of guessing a price.

That is a feature, not a failure.

The most important refactor was deleting shadow accounting

While reviewing the system, we found an older Payment Dashboard flow that effectively calculated MYZ like this:

canonical ledger
+

Mongo credits

Mongo redemptions

It also contained an automatic EUR → MYZ rule.

That meant the canonical ledger was not actually canonical.

So we changed the model.

New Stripe payments do not automatically generate MYZ.

Historical Mongo conversion records remain available as legacy audit data, but they no longer contribute to spendable MYZ.

Utility spending now hits the canonical ledger.

MongoDB redemption records are retained for things like:

fulfillment status
receipt metadata
application workflow

but they are no longer a second balance system.

That distinction is extremely useful:

ledger = accounting truth

application database = business state
Then we found another hard-coded system: Marketplace categories

The same development cycle uncovered a less dangerous but very visible problem.

The React Marketplace category list was mostly hard-coded.

That means adding a category required changing application code.

We replaced that with a dynamic catalog:

GET /api/listings/categories

Users can now propose new categories from the Marketplace itself.

The interesting UX decision is that the creator can use their pending category immediately for their own listing.

Global visibility still requires moderation.

So the flow becomes:

create category

↓

pending

↓

creator may use it

↓

admin review

↓

global category

This is much more flexible than making category taxonomy a deploy-time decision.

Knowledge became a Marketplace object too

We also introduced a first-class:

knowledge

category.

Users can submit things like:

guides
methods
know-how
tutorials
manuals
practical skills

through a separate knowledge submission flow.

But we explicitly did not connect submission directly to rewards.

A new knowledge item starts as:

PENDING_REVIEW

The API response explicitly indicates that creation does not produce:

reward
ledger write
MYZ transfer

This separation matters.

A useful architecture is:

submission
↓
review
↓
verification
↓
optional reward

not:

user submits text
↓
money appears
Deployment uncovered another important distinction

The code was merged through PR #1286.

Production deployment succeeded on Vercel.

Marketplace category endpoints respond correctly.

knowledge is available in production.

No fatal/runtime errors appeared during our verification window.

But we are intentionally not saying the MYZ accounting system is fully production-ready yet.

Why?

Because the current ledger implementation is file-based.

And there is a big difference between:

the application can write a file

and:

this storage is durable, shared, atomic,
and appropriate for production accounting

In a serverless environment, local filesystem assumptions deserve special attention.

A file lock only protects concurrent writers if those writers actually share the same durable filesystem.

Multiple isolated serverless instances may not.

So this implementation establishes the accounting contract and business logic, but production MYZ writes still need a properly durable storage architecture before we treat the system as authoritative under distributed load.

That may mean moving the ledger to a transactional database or placing it behind a dedicated single-writer service.

We also refused to fake a green CI status

One final engineering lesson came from CI.

We created a dedicated GitHub Actions workflow for the MYZ ledger contract.

It includes service tests, route tests, internal-payment tests, and ledger verification.

But GitHub did not generate a workflow run.

We even created a dedicated commit touching the workflow itself:

33180d233dd6abbb558ed58b56371d77eb11faa2
ci(myz): trigger canonical ledger contract

Still:

workflow runs = 0

Other workflows in the repository have execution history.

So instead of writing:

CI ✅

we documented:

CI execution issue still open

It sounds minor, but this is one of the habits that matters most in engineering:

do not convert “we expected this to run” into “it passed.”

The architecture now looks much simpler

Before:

Dashboard MYZ
Payment Dashboard MYZ
Marketplace payment logic
Zorgax payment logic
conversion logic
redemption records

After:

            CANONICAL MYZ LEDGER
                     │
    ┌────────────────┼────────────────┐
    │                │                │

Marketplace Zorgax P2P

    │                │

buyer → seller user → system

Business databases still exist.

They should.

But they track business state.

They no longer get to invent their own spendable MYZ balance.

What I would improve next

The implementation is merged and deployed, but the architecture still has clear next steps:

move the ledger to genuinely durable shared storage;
validate double-spend protection across multiple application instances, not only within one process;
fix the GitHub Actions workflow execution;
add persistent purchase intents for even stronger recovery across client idempotency-key loss;
define explicit migration rules for historical MYZ records;
define verified identity mapping between older contributor: accounts and newer marketplace:user: accounts.

None of those invalidate the current work.

They define the boundary between correct application logic and production-grade distributed accounting infrastructure.

And knowing that boundary is probably the most useful lesson from this refactor.

The biggest takeaway for me was simple:

A balance is a derived value.
A ledger is the history that makes the balance trustworthy.

Once we started treating MYZ that way, Marketplace payments, Zorgax purchases, legacy cleanup, receipts, idempotency, and concurrency all became parts of the same system instead of separate features.

MyZubster is open source, and we’re building this architecture in public.

Repository: MyZubster-Ecosystem/myzubster
Main implementation: PR #1286

Stop Cascading Failures: Implementing the Circuit Breaker Pattern in Node.js

Fri, 18 Sep 2026 08:33:49 +0000

In modern web development, our systems rarely run in isolation. Your Node.js server likely connects to external APIs, databases, authentication providers, and third-party payment gateways. But what happens when one of those external systems slows down or crashes entirely? If your system continues to blindly send traffic to a broken dependency, you risk crashing your own backend. This is where The Circuit Breaker Pattern comes into play.

What is the Circuit Breaker Pattern?

The Circuit Breaker pattern is an architectural safety mechanism that monitors outbound requests to external dependencies and halts execution when failure rates cross a certain threshold. Instead of allowing your application to repeatedly attempt an operation that is almost guaranteed to fail, the circuit breaker trips. It quickly returns an error or a fallback response, saving system resources and preventing minor, isolated downtime from turning into a massive, system-wide outage.

A Relatable Real-Life Analogy

Imagine you are trying to call a local pizza shop to place an order, but their phone line is completely down due to a storm. If you sit there redialing their number every ten seconds, you are wasting your own time, draining your phone's battery, and growing increasingly frustrated.

A circuit breaker is like having a smart digital assistant. The first few times you try to call, the assistant lets the call go through. But after three failed attempts, the assistant intercepts your future attempts. Instead of dialing, it instantly tells you, "The shop is unreachable right now. I am not going to let you waste your time calling them. Let's try again in ten minutes, or order from another place instead." It protects your time and energy from being wasted on a guaranteed failure.

Why It Matters Daily in the Tech Industry

In a production environment, when an external API becomes sluggish or goes offline, incoming HTTP requests to your Express server will start to pile up. Each waiting request holds onto a system thread, a database connection, and a chunk of memory. Within seconds, your server runs out of available resources and crashes.

Engineers use circuit breakers to prevent these cascading failures. By failing fast, the circuit breaker immediately releases resources back to the server. Your application can then degrade gracefully—perhaps by serving cached data or displaying a polite "service temporarily offline" message—ensuring that the rest of your application remains fast and responsive for your users.

Code Example: Simulating a Circuit Breaker in Node.js

Here is a simple, plain Node.js implementation of a circuit breaker class protecting a route in an Express application:

const express = require('express');
const app = express();

class CircuitBreaker {
  constructor(requestFunction, failureThreshold = 3, cooldownPeriod = 10000) {
    this.requestFunction = requestFunction;
    this.failureThreshold = failureThreshold;
    this.cooldownPeriod = cooldownPeriod;
    this.state = 'CLOSED'; // States: CLOSED, OPEN, HALF-OPEN
    this.failureCount = 0;
    this.nextAttemptTime = Date.now();
  }

  async execute(...args) {
    if (this.state === 'OPEN') {
      if (Date.now() > this.nextAttemptTime) {
        this.state = 'HALF-OPEN';
        console.log('[Circuit Breaker] Transitioning to HALF-OPEN. Testing service...');
      } else {
        throw new Error('Circuit is OPEN. External service temporarily offline.');
      }
    }

    try {
      const result = await this.requestFunction(...args);
      this.reset();
      return result;
    } catch (error) {
      this.handleFailure();
      throw error;
    }
  }

  reset() {
    this.state = 'CLOSED';
    this.failureCount = 0;
    console.log('[Circuit Breaker] Success! Circuit reset to CLOSED.');
  }

  handleFailure() {
    this.failureCount++;
    console.warn(`[Circuit Breaker] Failure recorded. Count: ${this.failureCount}`);
    if (this.failureCount >= this.failureThreshold) {
      this.state = 'OPEN';
      this.nextAttemptTime = Date.now() + this.cooldownPeriod;
      console.error(`[Circuit Breaker] Threshold reached! Circuit is now OPEN for ${this.cooldownPeriod}ms.`);
    }
  }
}

// A mock external API function that always fails
const unstableExternalAPI = async () => {
  throw new Error('API Timeout');
};

const breaker = new CircuitBreaker(unstableExternalAPI);

app.get('/external-data', async (req, res) => {
  try {
    const data = await breaker.execute();
    res.json({ success: true, data });
  } catch (error) {
    res.status(503).json({
      success: false,
      message: error.message,
      fallbackData: { items: ['Cached Item 1', 'Cached Item 2'] } // Graceful fallback
    });
  }
});

app.listen(3000, () => console.log('Server running on port 3000'));

Key Takeaway

Designing resilient distributed systems means accepting that third-party dependencies will inevitably fail. Instead of hoping for 100% uptime from external APIs, developers must use the Circuit Breaker pattern to isolate those inevitable failures, shield their own resources, and keep their core application functional.

Originally published on my blog. You can read the alternative breakdown here.

Idempotency in APIs - Why Your Retry Logic Can Break Everything (And How to Fix It)

Sat, 12 Sep 2026 13:30:46 +0000

Modern applications rely heavily on APIs. But networks are unreliable - requests fail, time out, or get retried.

Now imagine this scenario:

User clicks “Pay Now”
Request sent to server
Payment processed successfully
But response is lost due to network issue
Client retries the request

💥 Payment gets processed again. User is charged twice.

This is one of the most dangerous and common problems in distributed systems.

The solution? Idempotency.

What is Idempotency?

An operation is idempotent if performing it multiple times produces the same result as performing it once.

In simple terms:

Same request → same effect → no duplicate side effects

Example:

Create Order (without idempotency):
Request sent twice → 2 orders created ❌

Create Order (with idempotency):
Request sent twice → only 1 order created ✅

Idempotency ensures system correctness even when retries happen.

Why Do Retries Happen?

Retries are extremely common due to:

Network timeouts
Server crashes
Client retries
Load balancers retrying requests
Mobile networks instability
API gateway retries
Reverse proxy retries

Retries are normal. Duplicate side effects are not.

Which HTTP Methods Are Idempotent?

According to HTTP specification:

Method	Idempotent	Explanation
GET	✅ Yes	Fetch data
PUT	✅ Yes	Replace resource
DELETE	✅ Yes	Delete resource
POST	❌ No	Create resource
PATCH	❌ No	Partial update

POST is NOT idempotent by default.

Example:

POST /orders

Calling twice → creates 2 orders.

This must be handled explicitly.

Real-World Example: Payment API Problem

Client sends:

POST /payments
{
  "user_id": "123",
  "amount": 100
}

If client retries due to timeout:

POST /payments

Without idempotency → duplicate payment ❌

With idempotency → single payment ✅

Solution: Idempotency Key

Client sends a unique key with request:

POST /payments
Idempotency-Key: abc123

Server behavior:

If key is new → process request
If key already exists → return previous result

This guarantees single execution.

How Idempotency Works Internally

Flow:

Client generates unique key
Sends request with key
Server checks database/cache for key
If key exists → return stored response
If not exists → process request
Store key and response
Return response

Go Implementation Example

Let’s implement idempotent payment API using Go.

Database Table

idempotency_keys table:

key (primary key)
response
created_at

Go Example (Basic Version)

type IdempotencyRecord struct {
    Key      string
    Response string
}

Handler Implementation

func PaymentHandler(w http.ResponseWriter, r *http.Request) {

    key := r.Header.Get("Idempotency-Key")

    if key == "" {
        http.Error(w, "Idempotency-Key required", http.StatusBadRequest)
        return
    }

    // Check if key exists
    record, exists := getIdempotencyRecord(key)

    if exists {
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(record.Response))
        return
    }

    // Process payment
    paymentID := processPayment()

    response := fmt.Sprintf(`{"payment_id": "%s"}`, paymentID)

    // Store key and response
    saveIdempotencyRecord(key, response)

    w.Header().Set("Content-Type", "application/json")
    w.Write([]byte(response))
}

Example Flow

First request:

POST /payments
Idempotency-Key: abc123

Response:
{
  "payment_id": "pay_001"
}

Retry request:

POST /payments
Idempotency-Key: abc123

Response returned from storage:

{
  "payment_id": "pay_001"
}

No duplicate payment created.

Production Implementation Using Redis (Recommended)

Redis is ideal because:

Extremely fast
Supports expiration (TTL)
Perfect for temporary idempotency storage

Redis Example in Go

func PaymentHandler(w http.ResponseWriter, r *http.Request) {

    key := r.Header.Get("Idempotency-Key")

    val, err := redisClient.Get(ctx, key).Result()

    if err == nil {
        w.Write([]byte(val))
        return
    }

    paymentID := processPayment()

    response := fmt.Sprintf(`{"payment_id": "%s"}`, paymentID)

    redisClient.Set(ctx, key, response, time.Hour*24)

    w.Write([]byte(response))
}

Important: Handle Race Conditions

Two requests may arrive simultaneously with same key.

Solution: Use atomic operations.

Redis example:

SET key value NX

NX = set only if not exists

Important Best Practice: Store Full Response

Do not just store key.

Store:

status code
response body
timestamp

Because retry must return exact same response.

Example Database Schema (PostgreSQL)

CREATE TABLE idempotency_keys (
    key TEXT PRIMARY KEY,
    response JSONB,
    status_code INT,
    created_at TIMESTAMP DEFAULT NOW()
);

Primary key ensures uniqueness.

Real-World Systems That Use Idempotency

Critical systems use idempotency extensively:

Payment APIs
Order creation APIs
Financial transactions
Booking systems
Wallet transfers
Stripe payments
Banking APIs

Without idempotency, financial systems would break.

Common Mistake Developers Make

❌ Using retry without idempotency

retry 3 times on failure

This multiplies side effects.

Example:

Retry logic: 3 times
Result: 3 payments

Dangerous bug.

Idempotency vs Duplicate Prevention Using Unique Constraints

Example:

UNIQUE(user_id, order_id)

This prevents duplicates at DB level.

But idempotency is better because:

Works at API level
Handles retries cleanly
Returns same response

Best approach: use BOTH.

When Should You Use Idempotency?

Use idempotency for operations that create side effects:

Payment creation
Order creation
Wallet transfer
Booking
Resource creation

Not required for:

GET requests
Read-only operations

How Client Should Generate Idempotency Key

Use UUID:

Example:

550e8400-e29b-41d4-a716-446655440000

Go example:

uuid.New().String()

Unique per operation.

Idempotency Key Expiration Strategy

Keys should expire after some time.

Common TTL:

24 hours
48 hours
7 days (depending on business need)

Redis makes expiration easy.

Example Architecture

Client → API Gateway → Service → Redis → Database

Redis handles idempotency keys
Database handles actual data

Fast and reliable.

Real Production Scenario: Mobile Network Retry

Mobile user taps "Place Order"

Network slow → timeout
Client retries automatically

Without idempotency → duplicate orders

With idempotency → safe retry

Important Difference: Safe Retry vs Unsafe Retry

Unsafe retry:

POST /orders

Safe retry:

POST /orders
Idempotency-Key: unique-key

Idempotency vs Exactly-Once Execution

Exactly-once execution is impossible in distributed systems.

Idempotency provides practical solution.

It ensures same final state.

Best Practices Checklist

Always:

Use idempotency keys for POST APIs
Store keys in Redis or database
Store full response
Use expiration
Handle race conditions
Use UUID keys
Combine with database constraints

Key Takeaway

Retries are normal.

Duplicate side effects are dangerous.

Idempotency ensures your APIs remain safe, reliable, and production-ready.

It is not optional for critical APIs - it is essential.

Final Thought

If your API handles payments, orders, bookings, or financial transactions - idempotency is mandatory.

Without it, retries can silently corrupt your system.

With it, your APIs become reliable, predictable, and safe.

Stop Recreating Database Connections: An Illustrated Guide to Connection Pooling

Fri, 11 Sep 2026 02:43:51 +0000

If you have ever built an application that talks to a database, you have used a database connection. A database is simply a structured storage system for digital information, and a connection is the secure, active pipeline established between your application's code and that database.

However, constantly opening and closing these connections for every single user request is incredibly slow and resource-heavy. To solve this, developers use a technique called connection pooling. Connection pooling is a performance optimization method where a group (or "pool") of database connections is kept open and idle, waiting to be reused by whoever needs them, rather than being destroyed and recreated on demand.

The Analogy: The Pizza Delivery Fleet

To understand why connection pooling is so vital, imagine you run a busy, high-volume pizza restaurant. Every time an order comes in, you need to deliver it to a customer.

Imagine if, for every single delivery, you had to hire a brand-new driver, run a background check, sign a legal employment contract, set up their vehicle registration, and purchase a GPS. Once that single delivery was complete, you immediately fired them, canceled their insurance, and sold the car. It sounds absurd, right? The administrative overhead of hiring and firing would take three times longer than the actual delivery, and your restaurant would go bankrupt in a week.

Instead, you hire a permanent, dedicated fleet of five delivery drivers. They sit in the break room, fully certified and ready. When a pizza is boxed, the first available driver grabs the box, drives to the house, delivers it, and returns to wait in the break room for the next order.

In this analogy, the drivers are the "connection pool," the deliveries are your database queries (requests for information), and the hiring process is the network handshake required to open a new database connection.

Why It Matters in Daily Software Engineering

In real-world software, opening a database connection is highly expensive in terms of computing power. The application must perform a security handshake, authenticate its username and password, allocate memory on the database server, and establish a network route. This process can easily take 50 to 100 milliseconds.

If your website experiences a surge in traffic—say, 1,000 users clicking a button at the exact same moment—your server would try to open 1,000 separate connections simultaneously. The database server would quickly run out of memory, slow down to a crawl, and eventually crash, displaying a "database connection error" to your users.

Connection pooling prevents this complete system collapse. It sets a strict limit on the maximum number of connections allowed (e.g., capping it at 20). If 1,000 users arrive, they politely share those 20 open connections. Because each connection is already established, queries execute in 1 to 2 milliseconds. Once a query is done, the connection is instantly recycled for the next user in line, protecting your database from crashing.

A Simple Code Example in Node.js

Below is a simple JavaScript example using a popular PostgreSQL database driver. It demonstrates how we initialize a pool and query the database without manually opening and closing connections.

const { Pool } = require('pg');

// 1. Create a pool with a maximum capacity of 10 connections
const pool = new Pool({
  user: 'db_user',
  host: 'database.server.com',
  database: 'mydb',
  password: 'securepassword',
  port: 5432,
  max: 10, // Maximum connections in the pool
});

async function fetchUserData(userId) {
  // 2. Grab an already-open connection from the pool
  const client = await pool.connect();
  
  try {
    // 3. Execute the database query
    const result = await client.query('SELECT * FROM users WHERE id = $1', [userId]);
    return result.rows[0];
  } catch (err) {
    console.error('Database query error:', err);
  } finally {
    // 4. Crucial: Release the connection back to the pool
    // This does NOT close the connection; it just makes it available for others!
    client.release();
  }
}

The Takeaway

Connection pooling turns a heavy, repetitive administrative chore into a fast, shared utility. By keeping a smart, limited buffer of active connections ready to work, software systems can handle tens of thousands of requests smoothly, ensuring high-speed page loads while protecting underlying servers from sudden traffic spikes.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Building a Real-Time Dashboard with FastAPI, WebSockets, and MySQL

Wed, 09 Sep 2026 08:42:41 +0000

I recently worked on a lightweight real-time dashboard built with FastAPI, WebSockets, and aiomysql.

The main goal was simple: retrieve live data from a MySQL database and push updates to the frontend in real time without relying on continuous polling.

This project provided a practical opportunity to explore asynchronous programming, WebSocket communication, and database integration using Python.

Project Overview

The application uses a FastAPI backend to communicate with a MySQL database and a WebSocket endpoint to deliver database updates to connected clients.

The overall architecture can be summarized as:

MySQL Database
      │
      ▼
   aiomysql
      │
      ▼
FastAPI Backend
      │
      ▼
  WebSocket
      │
      ▼
Frontend Dashboard

The WebSocket connection allows the server to push new data directly to the client, making the dashboard capable of displaying updates in real time.

What This Project Demonstrates

The project covers several important backend development concepts:

Setting up Python and the required dependencies
Building an asynchronous FastAPI backend
Connecting to MySQL using aiomysql
Implementing a WebSocket endpoint
Retrieving database records asynchronously
Sending live data to connected clients
Running the application with Uvicorn
Structuring a simple real-time backend application

Why Use WebSockets?

Traditional REST APIs generally follow a request/response model.

For example:

Client → Request → Server
Client ← Response ← Server

If the frontend needs updated information continuously, it may have to repeatedly send requests to the server.

This is commonly known as polling:

Client → Request
Server → Response

Client → Request
Server → Response

Client → Request
Server → Response

With WebSockets, the client establishes a persistent connection with the server:

Client ⇄ WebSocket Connection ⇄ Server

The server can then push new information to the client whenever it becomes available.

This makes WebSockets particularly useful for applications where data needs to be delivered quickly and continuously.

Potential Use Cases

This architecture can be useful for:

Real-time dashboards
Monitoring systems
Live administration panels
Student management systems
Record management applications
System status monitoring
Live data visualization
Notification systems

Documentation and Source Code

I created a complete guide covering the setup and implementation process, including:

Python installation
Required package installation
Project configuration
Server startup
Complete main.py implementation
FastAPI and WebSocket configuration

The GitHub repository also includes the documentation and source code for reference.

GitHub Repository

FastAPI WebSocket Real-Time Dashboard - GitHub Repository

The repository contains:

Complete Guide (PDF)
Source Code (main.py)

Key Technologies

Technology	Purpose
Python	Backend programming
FastAPI	Asynchronous web framework
WebSockets	Real-time client-server communication
aiomysql	Asynchronous MySQL connectivity
MySQL	Database
Uvicorn	ASGI server

Final Thoughts

This project was a useful exercise in combining asynchronous programming, database connectivity, and real-time communication into a practical backend application.

FastAPI and WebSockets provide a straightforward foundation for building applications where data needs to move between the server and client with minimal delay.

I look forward to extending this project with features such as authentication, multiple connected clients, automatic database change detection, improved error handling, and a more advanced frontend dashboard.

If you are working with FastAPI or real-time applications, I'd be interested to hear about your projects and approaches.

Also Published On

This article is also available on the following platforms:

LinkedIn: Read on LinkedIn
Medium: Read on Medium
AWS: Read on AWS
DEV Community: Read on DEV Community

Demystifying ACID Transactions: How Databases Prevent Chaos and Keep Your Data Safe

Tue, 08 Sep 2026 08:20:04 +0000

If you have ever used an online app to transfer money, buy a concert ticket, or update your profile, you have relied on a database. Behind the scenes, databases must process millions of instructions without making mistakes, even if the power suddenly cuts out or the server crashes. To guarantee this level of safety, databases rely on a concept called an ACID transaction, which is a set of four rules designed to keep your data perfectly accurate, clean, and trustworthy.

The Analogy: The Vending Machine

To understand how an ACID transaction works, imagine buying a bag of chips from a vending machine. This simple action relies on several rules to make sure nobody gets cheated:

Atomicity (All-or-Nothing): You enter your money and press the button. Either you get your chips and the machine keeps your money, or the machine jams, returns your cash, and keeps the chips. You will never experience a state where the machine keeps your money but does not give you the chips.
Consistency (Rule Following): The machine must always balance its books. If a bag of chips costs two dollars, the machine's internal tally must show exactly two dollars more in cash and one fewer bag of chips in inventory. It cannot end up with a random, unexplainable state.
Isolation (Privacy): If two people press buttons on the vending machine at the exact same moment, the machine processes one transaction first, finishes it completely, and then processes the second. The two orders do not get mixed up, and you do not end up getting each other's snacks.
Durability (Permanence): Once your chips drop into the retrieval slot, they are yours. If the power outlet is pulled from the wall a millisecond later, the chips do not magically fly back up into the machine. The physical transfer is permanent.

Why It Matters in Tech

Without ACID transactions, software development would be an absolute nightmare. Imagine writing code for a banking system without these guarantees. If a customer transfers fifty dollars from Account A to Account B, the system must perform two database updates: subtract fifty from Account A, and add fifty to Account B.

If the server crashes exactly halfway through this process, fifty dollars could simply vanish into thin air. Engineers use ACID transactions to bundle these two updates into a single, bulletproof package. If any part of the process fails, the database automatically rolls back to the beginning, behaving as if the transfer never started, protecting the user's money and the business's reputation.

Seeing It in Action

Here is a simple example in JavaScript using a hypothetical database library. It shows how we bundle multiple database operations into a single, safe transaction:

async function transferMoney(senderId, receiverId, amount) {
  // 1. Start the transaction block
  await database.query("BEGIN TRANSACTION");

  try {
    // 2. Subtract the amount from the sender
    await database.query(
      "UPDATE accounts SET balance = balance - $1 WHERE id = $2", 
      [amount, senderId]
    );

    // 3. Add the amount to the receiver
    await database.query(
      "UPDATE accounts SET balance = balance + $1 WHERE id = $2", 
      [amount, receiverId]
    );

    // 4. If both succeeded, lock in the changes permanently
    await database.query("COMMIT");
    console.log("Transfer completed successfully!");
  } catch (error) {
    // 5. If ANY error occurred, cancel and reverse all changes
    await database.query("ROLLBACK");
    console.error("Transfer failed. System rolled back to safety:", error);
  }
}

The Takeaway

ACID transactions are the unsung heroes of software reliability, transforming chaotic and unpredictable environments into predictable, safe spaces. By ensuring that database operations either succeed completely or fail harmlessly without a trace, they give developers the confidence to build robust systems that users can trust with their most sensitive financial, personal, and operational data.

Resources

GitHub Repository: react-hook-lab
react-hook-lab: npm package
Connect with me on LinkedIn: Saurav Pandey

Originally published on my blog. You can read the alternative breakdown here.

Coder Legion - Recent posts tagged backend

Building an ATS Resume Scorer API with FastAPI, Celery, and Docker

What it does

Tech stack

Design highlights

Source code

Building a Booking System with Node.js, Google Calendar, and Google Meet

Introduction

1. The Architecture

2. Building the Booking Endpoint

3. Calculating Appointment Duration

4. Preventing Overlapping Bookings

5. Connecting Google Calendar and Google Meet

6. Saving Bookings in MongoDB

7. Adding an Airtable Backup

8. Sending Booking Notifications

9. Lessons from the Implementation

is-kit Reached 100 Stars ⭐🎉 Here’s What Changed Since 50

⭐ First, What Did 50 Stars Mean?

🔑 Property Refinement Became a First-Class Use Case

🔍 Then discriminatedUnion Got Much More Serious

💪 The Library Also Became Stricter

🤖 I Started Using is-kit in Other OSS Too

🚀 Then an is-kit Guard Ran Into an Effect Type Bug

👍 This Is the Kind of Feedback I Wanted

💬 The Project Is Also Becoming Easier to Explain

🐱 What 100 Stars Means to Me

Demystifying MongoDB Write Concern and Read Concern in Node.js

What Are Write Concern and Read Concern?

The Real-World Analogy: Certified Mail vs. Kitchen Whiteboards

Why It Matters in Daily Software Engineering

Implementation in Node.js

Key Takeaway

A very Simple Rate Limiting App ( Node.JS )

The Reconnect Bug Where Marking a User Online Erased the Data I Needed for Replay

Keeping a Legacy Cache Consistent During an Incremental Migration

Architecture

The Problem

The Constraints

The Naive Solution

1. The Legacy Service wouldn't know about the lock

2. Refreshing the entire cache was too expensive

The Solution: Reusing an Existing Lease

What If Releasing the Lease Fails?

An Important Invariant

The Tradeoff

What I Learned

1. Start with invariants, not technologies

2. Existing systems often contain useful primitives

3. Temporary migration code needs an exit condition

Final Thoughts

Calling a Service Inside prisma.$transaction Doesn't Make It Atomic — Here's the Bug That Taught Me

The Reliability Boundary: What AI Agents Need Before They Touch Production APIs

Separate reasoning from execution

Why an API gateway is not enough

A production lesson from banking modernization

A refund example

Six controls for sensitive agent actions

1. Identity and narrow authorization

2. Structured commands and validation

3. Idempotency and duplicate protection

4. Risk-based human approval

5. Auditability

6. Reconciliation

Where MCP fits

Reference architecture

What teams should measure

Conclusion

References

Half-Open Circuit Breaker Probes in Serverless Workers

Understanding the Circuit Breaker States

Choosing a Coordination Boundary

Managing Recovery and Failure Handling

Verification Checklist

Testing LLM Fallback Tiers Before Outages

Fractional Shadow Traffic With Diff Logging

Response-Shape Adapter Contracts

Scheduled Failover Drills and Readiness Metrics

Bounding Cost and Complexity

⚡️ Caching Strategies for Web Apps: From Browser to Backend

🔍 Then `discriminatedUnion` Got Much More Serious

🤖 I Started Using `is-kit` in Other OSS Too

🚀 Then an `is-kit` Guard Ran Into an Effect Type Bug