Coder Legion - Recent posts tagged redis

Building the MyZubster Metaverse: From Prototype to Secure Realtime Infrastructure

Fri, 23 Oct 2026 05:22:14 +0000

Building the MyZubster Metaverse: From Prototype to Secure Realtime Infrastructure
The MyZubster Metaverse is evolving from a simple interactive experience into a distributed platform designed around secure identity, realtime communication, reliability, privacy, and verifiable collaboration.
During the latest development phase, we focused on the infrastructure required to operate the metaverse safely under real production conditions.
This article explains what we implemented, how the architecture works, what we verified, and what remains before the system can be considered production-ready.
The Current Architecture
The metaverse now follows a layered architecture:
Authenticated user

↓

MyZubster session

↓

Short-lived realtime token

↓

Authorized realtime channel

↓

Distributed gateway

↓

Redis coordination layer

↓

Presence, rate limits and observability
The objective is to prevent clients from connecting directly to unrestricted realtime resources.
Every connection must originate from an authenticated MyZubster session and receive only the permissions required for its specific room or operation.
Authenticated Realtime Sessions
We introduced a dedicated token exchange for realtime connections.
The server:

verifies the current MyZubster session;
checks whether the user may access the requested resource;
generates a short-lived realtime token;
restricts that token to authorized channels;
rejects anonymous or malformed requests.
This limits the damage caused by leaked tokens and prevents clients from subscribing to arbitrary rooms.
A client cannot simply invent a room identifier and connect to it. Authorization is evaluated server-side before realtime access is granted.
Channel Authorization
Realtime channels are treated as protected resources.
The authorization layer validates:
user identity;
session validity;
resource ownership;
room membership;
token expiration;
permitted operations.
This establishes a clear trust boundary between the browser and the realtime gateway.
The client requests access, but the server remains responsible for deciding whether that access is legitimate.
Distributed Presence with Redis
In-memory presence works only while a single application instance is running. It becomes unreliable when multiple instances or serverless functions are involved.
We therefore introduced Redis-backed presence coordination.
Redis is used to track:
active participants;
room membership;
temporary presence state;
expiration of abandoned sessions;
coordination between separate gateway instances.
Presence entries have a limited lifetime. If a client disappears without completing a normal disconnection, the corresponding state can expire automatically instead of remaining indefinitely.
Graceful Redis Failure Handling
Redis is now considered an explicit infrastructure dependency.
The system distinguishes between:
the application process being alive;
the realtime gateway being operational;
Redis being reachable;
the complete service being ready to accept traffic.
Failures are handled deliberately rather than silently ignored. This is important because returning a successful health response while the coordination layer is unavailable would provide misleading information to deployment systems and operators.
Distributed Rate Limiting
Local counters cannot reliably enforce limits across multiple runtime instances.
We implemented a distributed rate-limiting foundation backed by Redis so that requests are evaluated against shared state.
The rate-limiting system is designed to protect:
realtime token generation;
connection attempts;
room joins;
message publication;
presence updates;
abuse-sensitive API routes.
This provides a foundation for defending the metaverse against spam, accidental request loops and automated abuse.
Future work will refine quotas by route, identity, IP risk and operation type.
Privacy-Safe Observability
Reliability requires visibility, but observability must not expose private user information.
The realtime infrastructure now supports structured operational signals for:
authorization failures;
connection lifecycle events;
Redis availability;
rate-limit decisions;
degraded states;
health and readiness checks.
The goal is to make infrastructure problems diagnosable without logging raw credentials, session tokens or unnecessary personal information.
Health and Readiness Endpoints
A process being alive does not mean it is ready to serve realtime traffic.
We extended the health model to include Redis readiness. The service can now report a degraded state when Redis is unavailable instead of returning an unconditional success response.
Conceptually:
Process alive + Redis unavailable
```
       ↓
```
Service is not fully ready
```
       ↓
```
Health response reports degradation
This distinction is essential for:
deployment validation;
load balancers;
automated monitoring;
incident diagnosis;
rolling releases;
recovery procedures.
Shared Redis Client
A reusable Redis client layer was introduced to avoid creating duplicated connections across services and requests.
The shared client centralizes:
configuration;
connection reuse;
error handling;
readiness checks;
shutdown behavior;
instrumentation.
Centralizing this behavior reduces connection pressure and gives every Redis-dependent feature a consistent operational model.
Preview Infrastructure
The new realtime changes were deployed to an isolated Vercel Preview environment.
This allows us to validate infrastructure modifications before they affect the public production deployment.
The preview environment provides a safe place to test:
application builds;
serverless deployment behavior;
environment configuration;
readiness responses;
Redis connectivity;
authentication flows;
realtime regressions.
Production remains unchanged while the current pull request is still under review.
Testing and Continuous Integration
The implementation is covered by automated backend tests, including readiness behavior and Redis degradation scenarios.
The principal validation pipeline successfully completed:
backend tests;
frontend tests;
linting;
application builds;
Seller Free policy verification;
continuous evidence generation;
preview deployment.
This provides strong evidence that the new infrastructure integrates correctly with the existing application.
However, passing tests alone do not make the service production-ready. Runtime drills, dependency remediation and real multi-instance load testing are still required.
Current Development Status
The realtime readiness work is currently available in a draft pull request:
MyZubster realtime readiness PR #40
The implementation includes four focused commits:
feat(realtime): expose Redis readiness
test(realtime): cover Redis readiness
feat(realtime): gate health on Redis
test(realtime): cover Redis health degradation
Keeping the commits focused makes review and rollback easier.
Current Security Blocker
The security audit currently reports vulnerabilities in the transitive ip-address dependency for versions up to 10.7.3.
The available patched version is 10.7.4.
Before merging the realtime work, we need to:
identify which direct dependency introduces ip-address;
update the dependency chain safely;
regenerate the lockfile;
run the complete test suite;
repeat the security audit;
confirm that no high or critical vulnerability remains.
The pull request should remain in draft until this has been addressed and the runtime verification has been completed.
What Still Needs to Be Done
Dependency remediation
Upgrade the affected dependency chain and obtain a clean security audit.
Direct runtime verification
Verify the deployed preview response from:
/api/realtime/health
The response should explicitly show whether Redis is ready and whether the service is operating normally or in a degraded state.
Redis failure drill
Temporarily simulate Redis unavailability in an isolated environment and verify that:
readiness fails correctly;
the service does not claim to be healthy;
errors remain controlled;
recovery occurs after Redis reconnects.
Multi-instance load testing
Run concurrent tests against multiple gateway instances and verify:
shared rate limits;
consistent presence;
room synchronization;
connection cleanup;
acceptable latency;
stable Redis connection usage.
Restart and recovery testing
Restart application and Redis instances while clients are connected.
We need evidence that the platform can recover without leaving permanent ghost sessions or inconsistent presence records.
End-to-end acceptance testing
The complete flow must be tested:
Login
→ request realtime token
→ authorize channel
→ connect
→ join room
→ publish event
→ update presence
→ disconnect
→ clean up state
Unauthorized variations must also be tested.
Production rollout strategy
The production release should be gradual:
deploy to Preview;
complete runtime verification;
run load and failure drills;
review monitoring signals;
merge after approval;
deploy production;
observe error rates and Redis health;
retain a documented rollback procedure.
Longer-Term Metaverse Roadmap
Once reliability and security are validated, development can move toward the experience layer.
Planned areas include:
WebXR support;
immersive rooms;
interactive avatars;
spatial collaboration;
verified professional profiles;
Zorgax integration;
verified knowledge inside virtual environments;
community-managed spaces;
portable identity;
privacy-safe analytics;
independent interoperable nodes.
The long-term objective is not merely to build another virtual world.
MyZubster aims to combine identity, knowledge, collaboration and transparent open-source development inside an interoperable environment.
Lessons Learned
The most important lesson from this phase is that realtime development is not simply about sending messages quickly.
A reliable realtime platform must answer several harder questions:
Who is allowed to connect?
Which channels may they access?
How is state shared across instances?
What happens when Redis fails?
How is abuse limited globally?
Can operators distinguish “alive” from “ready”?
Can the system recover cleanly after a restart?
Can failures be investigated without exposing private data?
These concerns are now becoming first-class parts of the MyZubster architecture.
Conclusion
The MyZubster Metaverse has progressed beyond an isolated prototype.
We now have the foundation for:
authenticated realtime access;
server-controlled channel authorization;
Redis-backed presence;
distributed rate limiting;
privacy-aware observability;
dependency-aware health checks;
isolated preview deployments;
automated tests and evidence generation.
The next milestone is to prove that this architecture remains reliable under dependency failures, restarts, concurrency and realistic traffic.
Only after those checks—and after resolving the current security advisory—should the realtime infrastructure be promoted to production.
MyZubster remains an open-source project, and technical review, testing and constructive contributions are welcome.
Project Links
MyZubster repository
MyZubster Metaverse
Realtime readiness pull request
Repository GitHub: https://github.com/danieldirimini-myzubster/myzubster
Metaverso MyZubster: https://www.myzubster.com/metaverse
Pull request Realtime Readiness #40: https://github.com/danieldirimini-myzubster/myzubster/pull/40
Roadmap Linear MYZ-83: https://linear.app/myzubster/issue/MYZ-83/b6-realtime-observability-reliability-and-load-tests
Organizzazione GitHub MyZubster Ecosystem: https://github.com/MyZubster-Ecosystem
Profilo GitHub MyZubster: https://github.com/myzubster
Sito principale: https://www.myzubster.com
Knowledge Explorer: https://www.myzubster.com/knowledge
Profilo Zorgax: https://www.myzubster.com/zorgax-profile-onboarding

⚡️ Caching Strategies for Web Apps: From Browser to Backend

Sat, 10 Oct 2026 03:33:24 +0000

Caching is one of the most powerful ways to improve application performance—reducing load times, server costs, and user frustration. Yet many developers treat caching as a “black box” or skip it due to fear of cache-related bugs.

In this blog, we will:
✅ Demystify caching: what it is and why it matters
✅ Explore frontend (browser) caching strategies
✅ Dive into backend and distributed caching with Go examples
✅ Cover Cache-Control headers, ETag, stale-while-revalidate
✅ Discuss common pitfalls and best practices

By the end, you’ll know how to confidently implement caching in your React frontend and Go backend to build scalable, fast, and efficient applications.

🚀 What is Caching?

Caching means storing data temporarily so future requests can be served faster without recomputing or refetching the same data repeatedly.

Think of it like preparing tea and storing it in a flask—you don’t need to reboil water each time you want to drink.

📈 Why Caching Matters

Speed: Return data faster to users.
Reduce Server Load: Avoid redundant computation or DB queries.
Improve Reliability: Serve cached responses when backend services are down.
Reduce Costs: Less compute = lower cloud bills.

🧩 Caching in the Frontend (Browser)

1️⃣ Browser Cache (HTTP Caching)

Browsers automatically cache assets (JS, CSS, images) and API responses based on HTTP headers.

Key Headers:

Cache-Control: Defines how long and under what conditions the browser can reuse the response.
ETag: Helps validate cached data; server sends a hash, and browser sends it back to check if the data changed.
Expires: Sets a hard expiration time for cache.
stale-while-revalidate: Allows serving stale content while revalidating in the background for a seamless UX.

Example:

Cache-Control: public, max-age=3600, stale-while-revalidate=600

✅ Use Case: Cache API GET responses for non-user-specific data (e.g., blog posts, product catalogs).

2️⃣ Client-Side Data Fetching Libraries

Libraries like React Query, SWR provide built-in caching for API calls.

Cache API responses in memory.
Background revalidation for fresh data.
Avoids unnecessary re-fetching on component re-mounts.

✅ Use for per-session data caching in React apps.

🗂️ Caching in the Backend

1️⃣ In-Memory Caching

Store frequently accessed data in memory for fast retrieval.

Examples: Golang’s sync.Map, groupcache.
Fastest but limited by server RAM and is not shared across instances.

✅ Use for small, frequently requested data (e.g., config, auth tokens).

2️⃣ Distributed Caching

Tools like Redis and Memcached provide a shared cache layer accessible across multiple servers.

Benefits:

Centralized cache for load-balanced apps.
Supports expiration policies.
Can handle large datasets.

🔹 Go Example: Using Redis for Caching

1. Install Redis client:

go get github.com/go-redis/redis/v8

2. Basic Usage:

rdb := redis.NewClient(&redis.Options{
    Addr: "localhost:6379",
})

ctx := context.Background()

// Set cache
rdb.Set(ctx, "product_123", jsonData, time.Hour)

// Get cache
val, err := rdb.Get(ctx, "product_123").Result()
if err == redis.Nil {
    // Cache miss, fetch from DB
} else {
    // Cache hit, use val
}

⚖️ Cache Invalidation: The Hard Part

“There are only two hard things in Computer Science: cache invalidation and naming things.” – Phil Karlton

Whenever the underlying data changes, you need to ensure the cache is updated to avoid stale data.

Strategies:

✅ Time-based expiration (TTL):
Set a cache expiry time so data auto-expires after a fixed duration, ensuring old data clears automatically.

✅ Manual invalidation:
Manually clear or update cache entries whenever the underlying data changes to keep data fresh.

✅ Write-through caching:
Write data to both the cache and the database at the same time, ensuring consistency during reads.

✅ Cache busting:
Change cache keys (e.g., v1:product_123 → v2:product_123) when your data structure changes to avoid serving stale or incompatible data.

✅ Best Practices for Caching

Cache only GET requests; avoid caching POST unless explicitly designed.
Set reasonable TTLs to avoid stale data.
Monitor cache hit/miss rates to tune cache policies.
Be cautious caching user-specific or sensitive data.
Use cache versioning (v1:product_123) to manage structure changes.
Document what is cached, where, and why.

⚠️ Common Pitfalls

🚫 Caching sensitive user data publicly.
🚫 Forgetting to invalidate cache on data updates.
🚫 Over-caching dynamic endpoints, leading to stale or incorrect UI.

🔮 Future of Caching: CDN & Edge Caching

Content Delivery Networks (CDNs) like Cloudflare, AWS CloudFront, Vercel Edge cache data closer to the user.

Benefits:
✅ Lower latency.
✅ Offload origin server traffic.
✅ Built-in stale-while-revalidate strategies.

Using edge caching in modern React + Go apps can drastically improve performance globally.

🎯 Final Thoughts

Caching isn’t just an optimization; it’s essential for building scalable, reliable, and fast applications.

Whether it’s browser caching, React Query in the frontend, or Redis in the backend, knowing when and how to cache makes you a stronger developer.

💬 Have you implemented caching in your apps? What caching challenges have you faced? Let’s discuss in the comments!

My side project's free tiers were dying, so I counted what each service actually bills

Thu, 08 Oct 2026 22:42:45 +0000

Galaxy Movies is a movie discovery side project ~ Next.js Pages Router on Vercel, TMDB for data, Upstash Redis for caching & rate limiting, and OpenAI for a chat assistant & generated synopses. Everything runs on free tiers, which is the whole point of a side project.

A few weeks ago all three of those free tiers started groaning at once. Vercel function usage was climbing, Upstash was burning through its daily command quota well before midnight, and OpenAI spend had a suspicious correlation with traffic spikes that produced no actual readers.

The fixes all came down to the same realization: every "free" tier bills a different unit, and I was counting the wrong things. Here's what was actually billing me, and what I changed.

Act 1: The bots were running up my function bill

Vercel bills function duration, and my movie and TV detail pages were expensive: five parallel TMDB fetches (title, watch providers, credits, ratings, recommendations), plus Redis reads for the AI content slots, all in getServerSideProps.

The expensive traffic wasn't humans. SEO crawlers and AI scrapers sweep the entire TMDB ID space in bulk, and every one of those detail pages ran the full fetch chain. They don't index anything and send zero readers. Pure cost.

The fix was a thin render path just for them:

// Bots that don't index get a minimal render. They sweep the TMDB id
// space in bulk and don't need providers, credits, videos, or AI slots.
if (isBot(ua) && !isSearchEngine(ua)) {
  const titleRes = await fetch(`https://api.themoviedb.org/3/${media}/${id}?api_key=${apiKey}`)
  // on failure return notFound or errorProps; otherwise:
  context.res.setHeader('Cache-Control', 'private, no-store')
  return { props: { ...thinProps, media, title, topCast: [] } }
}

Two details matter here:

The isSearchEngine split. isBot matches a broad allowlist, but Googlebot, Bingbot, and DuckDuckGo still get the full render since they index. Social preview bots get the thin version. Whether a bot indexes is the only question that matters.
private, no-store is critical. If the stripped page entered the shared CDN cache, a cache collision could serve the bot version to a human, or worse, to Googlebot. Cheap responses must never contaminate the real cache.

I also found robots.txt was being rendered by a Next.js page, which meant every crawler's very first request was a function invocation. It moved to a static public/robots.txt, which doubles as an enforcement layer:

User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: MJ12bot
User-agent: Bytespider
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
# ...several more
Disallow: /

Yes, Disallow is an honor system, but the big commercial crawlers honor it, and they were most of the volume.

Finally, I lengthened the SSR CDN cache windows: s-maxage from 1 hour to 1 day, stale-while-revalidate from 1 day to 1 week. Repeat crawler hits on popular titles now terminate at the edge instead of running SSR again.

Act 2: Upstash bills every single Redis command

This one caught me off guard. Upstash's free tier bills per command, and a pipeline doesn't save you, because each member of a pipeline is its own billed command. My request paths fanned out into Redis calls that felt free because "Redis is fast."

The audit turned up some embarrassing finds. Cache reads on detail pages were ~97% misses, so I was paying a billed command for a null almost every time. Rate limiting cost up to three commands per check. A dataset updated once a day was being fetched again on every pageview.

The fixes, roughly in order of satisfaction:

Rate limiting became one Lua script. INCRBY plus a conditional EXPIRE plus a TTL recovery check run inside a single EVAL, billed once no matter how many redis.call()s it makes:

local count = redis.call("INCRBY", KEYS[1], ARGV[1])
if count <= tonumber(ARGV[1]) then
  redis.call("EXPIRE", KEYS[1], ARGV[2])
elseif count > tonumber(ARGV[3]) and redis.call("TTL", KEYS[1]) < 0 then
  redis.call("EXPIRE", KEYS[1], ARGV[2])
end
return count

The TTL check handles a real edge case: if a key somehow loses its expiry, a blocked client fixes it on the next attempt instead of being rate limited forever.

Check caches before paying the rate limit write. /api/prices was rate limiting requests that would have been cache hits, paying a write before doing a read. Reorder, and every cache hit is one command instead of two.

Memoize what changes daily. The streaming-changes diffs are rewritten once a day by cron but were being fetched per key on every pageview. A per instance LRU holding results for ten minutes turns repeat page loads into zero Redis commands:

const changesMemo = new LRUCache<string, Record<string, ProviderChanges>>({ max: 64, ttl: 10 * 60 * 1000 })

export async function getProviderChanges(providerKeys = [], { fresh = false } = {}) {
  const memoKey = [...providerKeys].sort().join(',')
  if (!fresh) {
    const memoized = changesMemo.get(memoKey)
    if (memoized) return memoized
  }
  // ...Redis reads, then changesMemo.set(memoKey, changes)
}

The fresh flag exists because the cron itself must never match subscribers against a diff it just replaced. Memoization that's correct most of the time is a bug with extra steps.

Batch the counters. Every HINCRBY is billed, so AI usage and analytics deltas accumulate in a Map and flush every 30 seconds (or 50 fields). Stats lag by up to half a minute and a dying instance loses its unflushed tail, a fine trade for telemetry.

Restructure data to match access patterns. Detail pages fetch both AI content slots in one MGET instead of two GETs. Generated lists moved from one key per slug into a single hash, where HGETALL is one command replacing an unbounded key fanout.

Net result: roughly half the daily command volume, with quota headroom for the first time in weeks.

Act 3: OpenAI spend, or "why is a scraper generating my content?"

The AI features on detail pages (generated synopses, similar title blurbs) trigger an OpenAI call on a cache miss. Guess who was causing cache misses at scale: the same scrapers from Act 1. Bots were literally generating my content.

So bots got read only access: they can consume AI content that's already cached but can never trigger generation (generate: false on the cache lookup). Human traffic is unaffected; scraper sweeps now cost zero tokens.

The other fixes were less dramatic but added up:

Failed cron seeds stopped retrying forever. A broken seed now records failed with an attempt counter and gives up after 3 tries, instead of burning a discover call on every daily run until the heat death of the universe.
Longer cache TTLs. Reviews and similar titles are static per title, so 365 day TTLs mean one generation per title, ever.
Prompt dieting. Similar title prompts dropped to 6 candidates with 200 character overviews; editorial prompts got explicit word budgets; structured JSON output removed the "please don't ramble" tax.
Shared chat cache. Identical chat requests (quick actions, common first questions; the bulk of traffic) now hit a Redis cache shared across instances, plus tighter limits: 10/hr per IP, 1000/day global.
Per task model control. All OpenAI calls route through one module with per task reasoning_effort: none for latency sensitive tasks, low only for the discover generator. Real token usage gets logged so I can see where spend actually goes.

The kill switch

The scariest scenario isn't gradual spend. It's a quota or billing problem silently converting every request into an error, or worse, into retry storms. Billing and auth failures (401/402, insufficient_quota) now trip a shared kill switch with a one hour cooldown:

export function isQuotaError(err: unknown): boolean {
  if (!(err instanceof OpenAI.APIError)) return false
  if (err.status === 401 || err.status === 402) return true
  const code = err.code || err.error?.code
  return code === 'insufficient_quota'
      || code === 'billing_hard_limit_reached'
      || code === 'billing_not_active'
}

While it's tripped: /api/chat returns 503 { code: 'ai_unavailable' } before spending budget, the chat widget checks /api/aiStatus on mount and hides itself entirely instead of showing an error bubble, the generators skip their calls, and the content cron skips the run without burning seed attempts, since an outage isn't a seed's fault. Crucially, ordinary 429 rate_limit_exceeded errors don't trip it; those are transient.

One deliberate choice throughout: degrade in the direction that saves money. When Redis errors on a check that protects spend, the code falls back to a per instance limiter and never fails open. An in memory limiter might be less accurate, but a Redis outage can't be used to run up my OpenAI bill.

The part where I broke things

Honest bookkeeping: the cost cuts shipped two bugs worth their own paragraph.

After moving the chat cache to Redis, identical requests started hitting on every quick action prompt, and the cache hit path returned plain JSON that the SSE client never parsed. Cache hits went from "the fast path" to "the broken path" overnight. Fix: return cache hits through the SSE stream like everything else.

And when bots stopped triggering AI generation, Lighthouse CI got caught in the crossfire. It audits the real detail pages, and its UA matched the bot list, so it was suddenly measuring the thin render. Chrome-Lighthouse went on the search engine side of the check.

Both bugs were the same shape: a filter added at the edge changed behavior for a consumer I forgot about. If your bot detection doesn't have a list of legitimate automated consumers, it will find one for you. In production. On a Sunday.

What I'd tell past me

Count the billed unit, not the abstraction. "Redis call," "function invocation," and "API request" are all wrong. It's commands, gigabyte seconds, and tokens, and the gap between what I thought I was spending and what I was billed lived entirely inside that distinction.
Segment bots by whether they index, not whether they're bots. That one boolean decided which requests deserved the expensive render, the Redis reads, and the cacheable output.
Cache at the cheapest layer that works. CDN edge > shared Redis > per instance LRU > regenerate. I had requests traversing the whole hierarchy when a static file would do (looking at you, robots.txt).
Free tier quotas are architecture constraints, not billing details. "A few Redis commands per request is fine" is a statement that only survives until you do the multiplication.
Instrument spend itself. The usage counters I added to survive the Upstash quota are now how I watch OpenAI costs. The fix for a quota crisis becomes your observability for the next one.

The site is galaxymovies.app if you want to see the thin renders in the wild. View source as a human, please; the bots are expensive enough. Happy to compare notes in the comments if you've fought the same free tier math!

Architecting a Zero-Loss Webhook Ingestion Engine with n8n and Redis (5k req/s DLQ)

Tue, 06 Oct 2026 08:26:44 +0000

Architecting a Zero-Loss Webhook Ingestion Engine with n8n and Redis (5k req/s DLQ)

Webhooks are inherently fragile. If an upstream payment gateway emits an invoice event during a scheduled database restart, or if your downstream CRM hits an unannounced rate limit (HTTP 429), standard automation flows drop the execution.

By default, native webhook nodes in low-code orchestration engines (like standard n8n workflows, Zapier, or Make) execute synchronously. If downstream services timeout or throw an uncaught 5xx error, the payload vanishes into log oblivion.

In this technical guide, we will design and deploy a Self-Healing Webhook Dead-Letter Queue (DLQ) & Payload Recovery Engine using n8n and Redis. This architecture isolates the ingestion layer from downstream consumers, enforces exponential backoff with full jitter, preserves metadata, and guarantees exactly-once processing using SHA-256 idempotency checks.

1. The Bottleneck: The Flaws of Synchronous Webhook Ingestion

Most teams route incoming webhooks directly to their target business logic:

[External Provider] ---> [Sync HTTP Endpoint (n8n)] ---> [Process / Parse] ---> [Update Postgres / HubSpot]

This simple pattern breaks catastrophically in production due to three distinct failure modes:

Unbuffered Burst Traffic: Black Friday or marketing flash sales can turn a typical 10 req/s stream into 3,000 req/s. Synchronous engines run out of Node.js event loop capacity or database connection pools, dropping connections with ECONNRESET.
The Cascading 429 Problem: When an external CRM starts returning 429 Too Many Requests, naive execution loops retry immediately or sequentially, worsening the rate limit and burning through workflow execution quotas.
Payload Loss on Uncaught Exceptions: If schema validation fails midway or an unhandled null parameter crashes an execution node, standard error handlers rarely isolate the raw payload for stateful replays without human intervention.

Commercial managed queues (like AWS SQS FIFO or Google Cloud Pub/Sub paired with Lambda) solve this, but they introduce vendor lock-in, complex IAM setup, and high maintenance costs when all you need is a deterministic, observable queue integrated into your n8n workflow ecosystem.

2. The Architecture: Decoupled Buffering & Replay Engine

To achieve true zero-loss architecture, we decouple ingestion from processing using Redis streams and sorted sets (ZSET).

                           +---------------------------------------+
                           |          Ingestion Layer              |
[Webhook Provider] ------> |  n8n Webhook Node (202 Accepted)      |
                           |  Compute SHA-256 Idempotency Key      |
                           |  Push to Redis Stream (Ingestion Bus) |
                           +---------------------------------------+
                                              |
                                              v
                           +---------------------------------------+
                           |          Worker / Consumer            |
                           |  Read from Stream (XREADGROUP)        |
                           |  Attempt Processing & Routing         |
                           +---------------------------------------+
                                      /               \
                            [Success]/                 \[Failure / 429 / 5xx]
                                    v                   v
                           +----------------+   +---------------------------------------+
                           |  Acknowledge   |   |        Retry Pipeline (ZSET)          |
                           |  (XACK)        |   |  Retry Count < Max?                   |
                           +----------------+   |  Calculate Exponential Backoff+Jitter |
                                                +---------------------------------------+
                                                        |                      |
                                                 [Under Limit]            [Max Retries]
                                                        |                      |
                                                        v                      v
                                                +----------------+   +-------------------+
                                                | Re-enqueue to  |   | Isolated DLQ      |
                                                | Stream at T+d  |   | Payload Sanitized |
                                                +----------------+   | Metadata Attached |
                                                                     +-------------------+
                                                                               |
                                                                               v
                                                                     +-------------------+
                                                                     | Auto/Manual Replay|
                                                                     | (Idempotent Sync) |
                                                                     +-------------------+

Pipeline Stages

Ingestion & De-duplication: The webhook accepts the payload, generates a SHA-256 hash of the deterministic payload body, and issues a Redis SETNX lock to prevent duplicate deliveries. The caller immediately receives 202 Accepted.
Buffer Stream: Payloads enter a Redis Stream (webhook:stream:raw).
Consumer & Error Catching: n8n worker loops consume batches. If an upstream call fails with a retryable code (408, 429, 500, 502, 503, 504), it moves to a scheduled retry set.
Exponential Backoff with Full Jitter: Instead of retrying on static intervals, the pipeline computes:
\[\text{Delay} = \min(\text{MaxDelay}, \text{Base} \times 2^{\text{attempt}}) \times \text{random}(0.5, 1.5)\]
Isolated DLQ: After $N$ failed retries (e.g., 5 attempts), the payload is sent to webhook:dlq:dead, sanitizing sensitive headers (e.g., Authorization) while preserving headers, execution error stacks, and original timestamps.

3. The Code & Logic

Below are the core n8n JavaScript Code node snippets that drive this engine.

A. Deterministic Idempotency Key & Header Sanitization

Add this to an n8n Code Node right after the incoming Webhook Node:

const crypto = require('crypto');

// 1. Extract payload and key headers
const payload = $json.body || $json;
const headers = $json.headers || {};

// 2. Remove non-deterministic variables (signatures, dynamic timestamps)
const sanitizedHeaders = { ...headers };
delete sanitizedHeaders['x-signature-timestamp'];
delete sanitizedHeaders['authorization'];

// 3. Create deterministic SHA-256 hash from stringified sorted payload
function getDeterministicHash(obj) {
  const sortedKeys = Object.keys(obj).sort();
  const normalizedString = JSON.stringify(obj, sortedKeys);
  return crypto.createHash('sha256').update(normalizedString).digest('hex');
}

const idempotencyKey = headers['idempotency-key'] || getDeterministicHash(payload);
const traceId = `trace_${Date.now()}_${crypto.randomBytes(4).toString('hex')}`;

return [{
  json: {
    idempotencyKey,
    traceId,
    originalPayload: payload,
    metadata: {
      receivedAt: new Date().toISOString(),
      originIp: headers['x-forwarded-for'] || 'internal',
      contentType: headers['content-type']
    }
  }
}];

B. Exponential Backoff with Jitter Computation

When a downstream service fails, route the payload to this calculation node before pushing to the Redis delay pool:

// Inputs from execution metadata
const currentAttempt = ($json.metadata && $json.metadata.retryCount) ? $json.metadata.retryCount + 1 : 1;
const MAX_RETRIES = 5;
const BASE_BACKOFF_MS = 2000;
const MAX_BACKOFF_MS = 300000; // 5 minutes

if (currentAttempt > MAX_RETRIES) {
  return [{
    json: {
      ...$json,
      status: 'DLQ_EXHAUSTED',
      finalFailureReason: $json.lastError || 'Max retries exceeded',
      movedToDlqAt: new Date().toISOString()
    }
  }];
}

// Exponential Backoff: Base * 2^(attempt - 1)
const exponentialDelay = Math.min(MAX_BACKOFF_MS, BASE_BACKOFF_MS * Math.pow(2, currentAttempt - 1));

// Apply Full Jitter: Uniform distribution between 0.5 and 1.5
const jitterFactor = 0.5 + Math.random();
const finalDelayMs = Math.floor(exponentialDelay * jitterFactor);
const nextExecutionTime = Date.now() + finalDelayMs;

return [{
  json: {
    ...$json,
    status: 'PENDING_RETRY',
    metadata: {
      ...$json.metadata,
      retryCount: currentAttempt,
      nextExecutionTime,
      delayAppliedMs: finalDelayMs
    }
  }
}];

C. Atomic Redis Deduplication & Lock

Use an n8n Redis node configured with the Raw Command action to ensure the worker process doesn't duplicate executions:

SET webhook:lock:{{ $json.idempotencyKey }} "1" NX EX 86400

If the command returns null, the payload has already been ingested within the last 24 hours. The n8n router can short-circuit the execution immediately with a 200 OK (Duplicate Ignored), eliminating downstream race conditions.

4. Deployment, Performance & Failure Recovery

Handling High Concurrency in n8n

By default, n8n writes execution histories to its main database. When operating at peak loads, you must tune execution modes to avoid bottlenecking SQLite/PostgreSQL:

Set Execution Data Pruning:
Set EXECUTIONS_DATA_SAVE_ON_SUCCESS=none in your n8n environment variables to prevent your disk from filling up with successful 200-ACK payloads.
Enable Queue Mode:
Deploy n8n in Queue Mode using Redis as the broker. Decouple your webhook listener instances (Webhook Processors) from your Worker instances (Consumers).

# Production n8n environment variables configuration snippet
EXECUTIONS_MODE=queue
QUEUE_BULL_REDIS_HOST=redis-master.internal
QUEUE_BULL_REDIS_PORT=6379
N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_MAX_AGE=72

The Automated Self-Healing Replay Pipeline

When downstream platforms recover from an outage, the Dead-Letter Queue contains hundreds of valid, pending payloads. Manually copying payloads out of log files is prone to human error.

The recovery engine handles this using a scheduled or trigger-based replay workflow:

Pulls batches from the Redis set webhook:dlq:dead via ZRANGEBYSCORE.
Verifies the target system's health via an HTTP OPTIONS/GET health-check endpoint.
Upon a valid 200 response from the target system, it resets retryCount to 0 and re-injects payloads directly into the primary consumer stream.
Idempotency guarantees (SETNX) ensure that partially completed side-effects are not duplicated.

5. Conclusion & Ready-to-use Workflow

Handling webhooks reliably requires moving away from fragile, synchronous direct-routing architectures. By introducing an in-memory Redis buffer with jittered exponential backoffs, metadata preservation, and automated dead-letter queueing, you can handle thousands of requests per second without losing business-critical data.

You can implement this system using the code snippets and architecture outlined in this article.

If you prefer a pre-built solution, you can get the production-ready package with test fixtures, container orchestration scripts, and end-to-end replay pipelines ready to import:

Instant Access on Whop: Download the Production Workflow
Direct Download on Gumroad: Download via Gumroad — use promo code EARLYBIRD for 20% off.

Why I Used BullMQ + Redis for Background Jobs in My NestJS Application — and When I Wouldn't

Sun, 04 Oct 2026 15:45:42 +0000

While working on Waitless, a real-time queue management platform, I came across a backend problem that looked simple at first:

A customer reaches the front of the queue, and the system needs to send notifications to other customers.

The first instinct could be to simply send the email inside the HTTP request.

But once I looked at the problem as a backend architecture problem rather than just an implementation task, I realized that sending the email and handling the user's request are two different workloads.

That led me to BullMQ + Redis.

The interesting part wasn't learning the API. The interesting part was understanding why this architecture made sense, what Redis was actually doing, and when I would choose something else instead.

The problem in Waitless

Waitless allows customers to join a digital queue and track their position without creating an account.

When one customer is called, another customer may need to receive a notification such as:

“It's your turn.”
“You're almost next.”

I already had the queue state stored in PostgreSQL. The question was:

Should the API send the email immediately, or should it create background work?

A synchronous approach could look like this:

HTTP request
     ↓
NestJS
     ↓
PostgreSQL
     ↓
Send email
     ↓
HTTP response

The problem is that sending an email depends on an external service.

If that service is slow or temporarily unavailable, my API request is affected.

I don't necessarily need the email to be completed before responding to the user.

What I actually need is:

“Remember that this email has to be sent, and process it separately.”

That is exactly the kind of problem a job queue is designed to solve.

Introducing BullMQ

BullMQ provides a job queue abstraction for Node.js applications.

Instead of doing the work immediately, the application creates a job:

Application
     ↓
   Queue
     ↓
    Job
     ↓
   Worker
     ↓
 Actual work

The application that creates the job is the producer.

The worker that processes the job is the consumer/worker.

This gives me a clean separation between:

Creating work

and:

Executing work

That separation becomes particularly useful when the work is slow, retryable, delayed, or needs to be processed independently from the HTTP request.

Where Redis comes in

One thing I initially had to clarify for myself was the relationship between BullMQ and Redis.

They are not the same thing.

A simplified mental model is:

NestJS
   ↓
BullMQ
   ↓
Redis

BullMQ is the queue system.

Redis is the backend that BullMQ uses to store and coordinate queue-related state.

In my NestJS application, I configure the connection like this:

BullModule.forRoot({
  connection: {
    host: 'localhost',
    port: 6379,
  },
});

Port 6379 is Redis's default port, so this configuration tells BullMQ which Redis instance to connect to.

The important distinction is:

BullMQ is not “running on port 6379.”
BullMQ is connecting to the Redis server running there.

Creating a notification job

In Waitless, I inject the notification queue into my service:

@InjectQueue('notifications')
private notifQueue: Queue

Now I can add jobs to that queue.

For example:

if (next.email) {
  await this.notifQueue.add('your-turn-now', {
    email: next.email,
    name: next.name,
    queueName: queue.name,
  });
}

This does not mean:

“Send the email now.”

It means:

“Create a job called your-turn-now and put it into the notifications queue.”

The payload contains the information the worker will need later.

Conceptually:

NestJS
   ↓
notifQueue.add(...)
   ↓
BullMQ
   ↓
Redis
   ↓
notifications queue
   ↓
your-turn-now job

The actual email is the responsibility of the worker.

Delayed jobs

One of the features I needed was delayed notification.

If another customer is second in line, I want to notify them shortly before their turn.

My implementation looks like this:

const secondInLine =
  await this.entryRepo.findSecondWaiting(queueId, next.id);

if (secondInLine?.email) {
  await this.notifQueue.add(
    'your-turn-soon',
    {
      email: secondInLine.email,
      name: secondInLine.name,
      position: 2,
      queueName: queue.name,
      trackingUrl: `http://localhost:3000/track/${secondInLine.token}`,
    },
    {
      delay: 30_000,
    },
  );
}

The important part is:

{ delay: 30_000 }

The job doesn't need to be processed immediately.

The architecture becomes:

Customer becomes second in line
            ↓
      Create BullMQ job
            ↓
       Delayed job
            ↓
          30 sec
            ↓
        Worker picks it up
            ↓
        Send notification

This is much more reliable than simply doing:

setTimeout(() => {
  sendEmail();
}, 30_000);

A timer belongs to the application process itself. A queue gives the work an explicit lifecycle outside the original request.

Why not just use PostgreSQL for the queue?

This was one of the more interesting questions I explored.

At first glance, PostgreSQL looks like a perfectly reasonable choice.

I could create a table such as:

jobs
-------------------
id
type
payload
status
created_at
processed_at

and continuously insert and delete jobs.

The problem is that the queue workload has different characteristics from my core business data.

My PostgreSQL database is responsible for things such as:

businesses
queues
customers
queue entries
relationships
transactions

Those are my core relational business data.

The queue workload is different:

job created
job becomes active
job succeeds
job fails
job retries
job is removed

This is a high-churn operational workload.

With PostgreSQL's MVCC model, repeatedly inserting and deleting rows creates dead tuples that later have to be cleaned up by VACUUM.

That doesn't mean PostgreSQL cannot implement queues. It absolutely can.

The real architectural question is:

Do I want the queue workload competing with my transactional business workload inside the same database?

For a small application, the answer might be yes.

For a higher-throughput system, separating those workloads can become valuable.

Why Redis made sense for my queue

Redis is designed around very fast operations on in-memory data structures.

That makes it a natural fit for workloads such as:

caching
counters
temporary state
queues
rate limiting
pub/sub
fast lookups

For my notification queue, the important characteristics were:

Asynchronous
High frequency
Short-lived job state
Low latency
Retries
Delayed execution
Worker concurrency

So instead of:

PostgreSQL
   ├── business data
   └── queue workload

I could separate them:

PostgreSQL
     ↓
Business state


Redis
     ↓
BullMQ
     ↓
Background jobs
     ↓
Workers

That separation was more important to me than simply saying:

“Redis is faster.”

What I learned about workload-driven architecture

This changed the way I think about databases.

The question shouldn't simply be:

Which database is better?

A better question is:

What workload am I dealing with?

For example:

Requirement	Possible choice
Relational transactional data	PostgreSQL
Very fast temporary state	Redis
Background jobs with retries	BullMQ + Redis
Document-oriented event data	MongoDB
Large event streams and replay	Kafka
Small simple background jobs	PostgreSQL or a scheduler
Managed cloud queue	SQS or similar service

These aren't absolute rules.

They are starting points for architectural reasoning.

BullMQ gives me more than a queue

Another thing I learned is that the real value of a job system is not just putting something into a list.

BullMQ provides concepts such as:

Jobs
Workers
Retries
Backoff
Delayed jobs
Concurrency
Rate limiting
Failed jobs
Job lifecycle

That matters because real systems fail.

For example:

Worker
   ↓
Send email
   ↓
Email provider unavailable
   ↓
Job fails
   ↓
Retry
   ↓
Worker tries again

The system can therefore treat background work as something with a lifecycle instead of a simple function call.

But there is an important failure model

A queue doesn't automatically mean:

“The job runs exactly once.”

A worker can crash.

A network call can time out.

A job can be retried.

The same operation may therefore be executed more than once.

That means job handlers should often be designed with idempotency in mind.

For example, a payment operation is much more dangerous to retry blindly than sending a harmless notification.

This was an important lesson for me:

Using a queue changes the failure model of the application.

You have to think about retries, duplicates, timeouts, and partial failures.

Could I use BullMQ without Redis?

Interestingly, yes.

Modern BullMQ also provides a PostgreSQL backend.

That means the conceptual architecture can also be:

NestJS
   ↓
BullMQ
   ↓
PostgreSQL

This can make sense when I want to keep the infrastructure simpler and the queue workload is moderate.

But there is a trade-off.

Using PostgreSQL for both workloads means:

PostgreSQL
 ├── transactional business data
 └── queue workload

So the queue and business operations share database resources.

That can be perfectly reasonable for a smaller application.

For a system with large queue throughput, isolating the queue workload may become more attractive.

This is why there isn't one universally correct answer.

When I would NOT use BullMQ + Redis

This was probably the most useful part of the exercise.

I don't want to add Redis and BullMQ just because they are popular technologies.

For something like:

const result = calculateTax(data);

there is no reason to create a queue.

If a task takes a few milliseconds and the user needs the result immediately, synchronous processing is simpler.

I also might avoid BullMQ for a very small scheduled task:

Every day at 08:00
Generate one report

A simple scheduler could be enough.

And if the application already uses PostgreSQL and has only a small number of background jobs, a PostgreSQL-backed solution may be simpler than adding another infrastructure component.

The lesson is:

Don't introduce distributed infrastructure without a workload that justifies it.

The decision I now make before choosing a queue

Instead of thinking:

“Should I use BullMQ?”

I ask:

1. Does this work need to happen asynchronously?

If yes, continue.

2. Does it need retries or failure recovery?

If yes, a real job system becomes more useful.

3. Does it need delayed execution?

For example:

Do this after 30 seconds.

4. Does it need multiple workers or concurrency?

For example:

Queue
 ├── Worker 1
 ├── Worker 2
 └── Worker 3

5. Does the workload justify another infrastructure component?

This is the trade-off question.

Only after answering those questions would I choose the technology.

My current mental model

The biggest thing I learned from implementing this feature is that architecture is less about memorizing technologies and more about matching workloads to the right tools.

For Waitless, my current architecture can be thought of as:

                    Waitless
                       │
          ┌────────────┼────────────┐
          ↓            ↓            ↓
     PostgreSQL      Redis       MongoDB
          │            │            │
   business state    jobs      event history
                    BullMQ
                       │
                    Workers
                       │
                   Email/API

PostgreSQL answers:

What is the current business state?

Redis + BullMQ answers:

What work needs to be processed asynchronously?

MongoDB answers:

What happened in the event history, and what can I derive from those events?

The important thing is not that I used three technologies.

The important thing is that each technology was chosen for a different workload.

Final takeaway

The best lesson I took from this wasn't actually “how to use BullMQ.”

It was this:

Before choosing a technology, understand the workload.

Ask:

What problem am I solving?
What are the access patterns?
What guarantees do I need?
How does the system fail?
What are the trade-offs?
What simpler alternative could work?

Once I started thinking this way, Redis, BullMQ, PostgreSQL, MongoDB and other technologies stopped looking like isolated tools.

They became different tools for different engineering problems.

And that is the mindset I want to continue developing as I work toward designing larger backend systems.

Redis Caching Strategies for High-Performance APIs

Sat, 12 Sep 2026 22:30:40 +0000

Redis Caching Strategies for High-Performance APIs

The Cache That Made My API Faster—and Introduced a New Problem.

When I first started thinking seriously about API performance, my instinct was simple:

If the database is slow, optimize the database.

Add indexes. Improve the query. Tune the connection pool.

And sometimes, that's exactly what you should do.

But eventually, you run into a different problem.

The database isn't necessarily slow.

You're just asking it the same question too many times.

Imagine an endpoint that returns product information.

GET /api/products/123

The product changes once every few hours, but the endpoint gets thousands of requests every minute.

Without caching, every request goes straight to PostgreSQL:

Request
   ↓
API
   ↓
PostgreSQL
   ↓
Response

The database is doing the same work over and over again.

That's where Redis comes in.

The Simple Idea Behind Caching

Instead of asking the database every time, we store frequently requested data temporarily.

Request
   ↓
Redis
   ↓
  HIT ─────→ Response
   │
  MISS
   ↓
Database
   ↓
Redis
   ↓
Response

The first request pays the database cost.

The next requests don't.

In a Node.js API, a simple cache-aside implementation looks like this:

const key = `product:${productId}`;

const cached = await redis.get(key);

if (cached) {
  return JSON.parse(cached);
}

const product = await prisma.product.findUnique({
  where: { id: productId },
});

if (!product) {
  throw new Error("Product not found");
}

await redis.set(
  key,
  JSON.stringify(product),
  { EX: 300 }
);

return product;

That's the basic pattern:

Check Redis → cache miss → query database → store result → return.

Simple.

But this is where caching stops being just a performance optimization.

Because now we have two copies of the data.

And they can disagree.

The Stale Data Problem

Suppose Redis contains:

product:123
price: ₦45,000

Then someone updates the product:

Database
price: ₦50,000

Redis still contains:

₦45,000

If the cache has a five-minute TTL, users could receive the old price for another five minutes.

This is the real challenge with caching.

Making data fast to access is easy. Keeping it correct is harder.

That's why TTL alone isn't enough.

TTL vs Invalidation

TTL answers:

"How long should this cache entry exist?"

Invalidation answers:

"When should I remove it because the underlying data changed?"

For example:

await prisma.product.update({
  where: { id: productId },
  data: updateData,
});

await redis.del(`product:${productId}`);

Now the next request becomes a cache miss and retrieves the latest value from the database.

For many APIs, cache-aside + TTL + delete-on-write is a very practical starting point.

You don't need a complicated caching architecture on day one.

Then There's the Cache Stampede

Here's where things get interesting.

Imagine a popular cache entry expires:

product:123

At the exact same time, 1,000 requests arrive.

They all check Redis:

MISS
MISS
MISS
MISS
...

And suddenly all 1,000 requests hit PostgreSQL.

The cache that was supposed to protect the database has just created a database traffic spike.

This is commonly called a cache stampede or thundering herd.

For extremely hot keys, you can prevent this by allowing only one request to rebuild the cache while the others wait.

Redis can help here through distributed locking.

But I wouldn't introduce locking everywhere.

If a query takes 10ms, the complexity may not be worth it.

If rebuilding the cache takes several seconds under heavy traffic, it's a different conversation.

Optimize for the workload you actually have.

What Should You Cache?

This is probably the most important question.

I wouldn't cache something simply because Redis is available.

Good candidates usually have three characteristics:

Frequently requested
+
Expensive to retrieve
+
Safe to be slightly stale

For example:

Product details
Public profiles
Search results
Configuration
Reference data
Expensive aggregations

I'd be much more careful with:

Account balances
Payment state
Authorization decisions
Inventory
Transaction state

Especially in financial systems.

If a wallet contains ₦100,000 and Redis says ₦100,000 while the database says ₦70,000 after a transaction, using that cached value for a financial decision is dangerous.

Fast wrong data is still wrong data.

For critical state, the database should remain authoritative.

Your Cache Key Is Part of Your Architecture

A cache key isn't just a random string.

This:

is a terrible cache key.

This is better:

user:123

And in a multi-tenant system:

tenant:456:user:123

For filtered results:

products:category:phones:page:1

Every parameter that changes the response needs to be represented appropriately in the key.

Otherwise, you can end up returning the wrong cached response to the right request.

That's a correctness bug—not a performance bug.

Redis Should Accelerate Your System, Not Become Your System

One of the most important design decisions is what happens when Redis fails.

For a cache-aside architecture, I'd rather have:

Redis available
    ↓
Use cache

Redis unavailable
    ↓
Use database

The API might become slower.

But it remains functional.

That's a much better failure mode than:

Redis unavailable
    ↓
Entire API unavailable

If your application completely depends on Redis just to retrieve data that already exists in PostgreSQL, you've accidentally turned your cache into a critical dependency.

Measure the Difference

Finally, don't add Redis and declare victory.

Measure it.

Before caching:

Average latency: 400ms
P95 latency:     800ms
DB queries/sec:  2,000

After caching:

Average latency: 70ms
P95 latency:     150ms
DB queries/sec:  500
Cache hit rate:  85%

Now you have evidence.

The goal isn't to make Redis busy.

The goal is to make the entire system do less unnecessary work.

The Bigger Lesson

Redis isn't what makes an API fast.

Avoiding unnecessary work makes an API fast.

Redis is simply one of the tools that helps you do that.

The moment you introduce a cache, you're making a trade:

More memory
+
More complexity
+
Potentially stale data

in exchange for:

Lower latency
+
Less database load
+
Higher throughput
+
Better scalability

The best caching strategy isn't the one that puts the most data into Redis.

It's the one that eliminates the most unnecessary work while keeping the system correct.

That's the difference between simply using Redis and actually designing with Redis.

Rate Limiting in the Next.js App Router: Field Notes on Middleware, Redis, and the Server Action That Looks Like Every Other POST

Fri, 21 Aug 2026 13:48:12 +0000

Headline: An in-memory rate limiter is wrong on serverless because every function instance keeps its own counter, so a limit of 10 requests per minute becomes 10 requests per minute per instance. The second trap is that every Next.js Server Action POSTs to the URL of the page that called it, so path-based limiting in middleware cannot tell one action from another.

A rate limiter is a counter with a deadline: allow N requests per identity per time window and reject the rest with HTTP 429. I have written that in ten lines before and been happy with it. Moving the same idea into a Next.js 16 App Router project on Vercel broke it in three places I did not expect — where the counter lives, who counts as an identity, and how you attach a limit to a Server Action that has no URL of its own.

Key takeaways

An in-memory Map rate limiter is per-instance on serverless. With ten warm function instances, a 10-requests-per-minute limit permits 100 requests per minute.
Next.js middleware is the cheapest place to shed abusive traffic because it runs before the route's own function boots, but it also sees RSC prefetch requests that carry the RSC: 1 header and that the user never intentionally made.
NextRequest.ip was removed in Next.js 15. On Vercel, read the client address with ipAddress(request) from @vercel/functions instead of trusting a raw x-forwarded-for header.
Every Next.js Server Action POSTs to the current page URL and carries a build-generated Next-Action header, so the only reliable place to limit a specific action is inside the action body.
Reject with status 429 and a Retry-After header, and decide fail-open versus fail-closed per route before your Redis has its first outage.

Why does an in-memory rate limiter break on serverless?

A module-scope Map is private to one function instance, and a serverless platform runs many instances at once. Each instance therefore enforces the full limit on its own, so the effective limit is your configured limit multiplied by the number of warm instances. That number is not something you control or can observe from inside the request.

This is the code I have shipped and regretted:

// Do not ship this to a serverless runtime.
const hits = new Map<string, { count: number; reset: number }>();

export function limit(key: string, max = 10, windowMs = 60_000) {
  const now = Date.now();
  const entry = hits.get(key);
  if (!entry || entry.reset < now) {
    hits.set(key, { count: 1, reset: now + windowMs });
    return { ok: true };
  }
  entry.count += 1;
  return { ok: entry.count <= max };
}

Vercel Fluid Compute makes this harder to notice rather than easier. Fluid Compute reuses a single function instance across concurrent requests instead of spawning one per request, so the Map survives far longer than it did under classic serverless. In local development and in a quiet preview deployment the limiter looks correct. It only comes apart under traffic spread across enough instances to matter, and nothing in the logs announces it.

The Map also never shrinks. On a long-lived instance every unique key you have ever seen stays resident until the instance is recycled, which is a slow memory leak wearing a rate limiter costume.

The fix is not a cleverer Map. The counter has to live in a store that every instance shares and that supports an atomic increment: Redis, or any datastore with a compare-and-set primitive.

Should the limit run in middleware or in the route handler?

Both, for different jobs. Middleware runs before Next.js resolves the route, so a request rejected there never boots the route's function and never touches your database. That makes middleware the correct place for a coarse, identity-agnostic abuse limit. The route handler knows the authenticated user, the parsed body, and the business meaning of the call, which makes it the correct place for a per-user quota.

// middleware.ts
import { NextResponse, type NextRequest } from 'next/server';
import { ipAddress } from '@vercel/functions';
import { ratelimit } from '@/lib/ratelimit';

export const config = {
  matcher: ['/api/:path*', '/login', '/signup'],
};

export async function middleware(request: NextRequest) {
  // RSC prefetches fire on link hover. The user did not ask for these.
  if (request.headers.get('rsc') === '1') return NextResponse.next();

  const ip = ipAddress(request) ?? '127.0.0.1';
  const { success, limit, remaining, reset } = await ratelimit.limit(`ip:${ip}`);

  if (success) return NextResponse.next();

  return NextResponse.json(
    { error: 'rate_limited', message: 'Too many requests.' },
    {
      status: 429,
      headers: {
        'Retry-After': String(Math.ceil((reset - Date.now()) / 1000)),
        'X-RateLimit-Limit': String(limit),
        'X-RateLimit-Remaining': String(remaining),
        'X-RateLimit-Reset': String(reset),
      },
    },
  );
}

The rsc header check is the line I added after watching my own quota drain while I did nothing but move a mouse. The Next.js router prefetches linked routes on hover and on viewport entry, and those requests are real HTTP requests that hit middleware with an RSC: 1 header. Counting them means a user who scrolls a navigation-heavy page is rate limited before they click anything.

Since Next.js 15.5 you can also opt middleware into the Node.js runtime with export const config = { runtime: 'nodejs' }, which lets you use a normal Redis client there. I still prefer an HTTP-based store in middleware, because middleware sits on the latency path of every matched request.

Placement	Sees	Cost of a rejection	Use it for
Middleware	URL, headers, cookies, IP	Lowest — route function never boots	IP-level abuse and brute-force shielding
Route handler	Everything, including session and body	Function has already started	Per-user quotas, per-endpoint cost control
Server Action body	Session, typed arguments	Function has already started	Form submissions and mutations

Which algorithm should I actually use?

Pick the cheapest algorithm whose failure mode you can live with.

Fixed window keeps one counter per window, so a check is a single INCR. Its flaw is the boundary: ten requests at 11:59:59 and ten more at 12:00:00 pass a "ten per minute" limit while delivering twenty requests in one second.
Sliding window log stores a timestamp per request and is exact, but its memory grows with your traffic, which is the wrong direction for a defence against traffic.
Sliding window counter weights the previous window by how much of it still overlaps the current one. Bounded memory, no boundary burst, and my default.
Token bucket gives each identity a capacity and a refill rate, so a quiet client may spend saved tokens at once. Right for APIs whose clients legitimately batch.

The @upstash/ratelimit package ships all four as fixedWindow, slidingWindow, tokenBucket, and cachedFixedWindow. If you write your own against Redis, the thing to get right is atomicity. INCR followed by a separate EXPIRE is two round trips, and if the process dies between them you have created a key with no expiry, which locks that identity out permanently. Do it in one script:

-- One round trip. The TTL is set only on the first hit of a window.
local current = redis.call('INCR', KEYS[1])
if current == 1 then
  redis.call('PEXPIRE', KEYS[1], ARGV[1])
end
return current

How do I identify the client when everything sits behind a proxy?

A rate limit is only as good as its key. NextRequest.ip and NextRequest.geo were removed in Next.js 15, so reading request.ip is now a type error rather than a subtle wrong answer. On Vercel the replacement is ipAddress(request) from @vercel/functions.

Do not key on a raw x-forwarded-for header unless you are certain a trusted proxy overwrites it. x-forwarded-for is an ordinary request header, so on any origin reachable directly, an attacker sets it to a new value per request and gets an unlimited number of fresh limit buckets. That is a complete bypass, not a partial failure.

IPv6 needs its own rule. A single residential subscriber is routinely assigned an entire /64 prefix, so keying on the full 128-bit address hands one attacker 2^64 distinct identities.

function identityKey(request: NextRequest, userId?: string) {
  // A stable account beats a network address whenever you have one.
  if (userId) return `user:${userId}`;

  const ip = ipAddress(request) ?? '0.0.0.0';
  if (ip.includes(':')) {
    // One subscriber can own a whole /64. Key the prefix, not the address.
    return `ip6:${ip.split(':').slice(0, 4).join(':')}`;
  }
  return `ip4:${ip}`;
}

How do I rate limit a Server Action when every action is a POST to the same URL?

A Server Action is a function marked with the 'use server' directive that the client invokes over the network. The invocation is an HTTP POST to the URL of the page the action was called from, with a Next-Action header containing a build-generated identifier for that specific action. There is no dedicated route path, which is exactly what breaks the obvious approach.

In middleware, a POST from a contact form on /contact and a POST from a delete-account button on /contact are the same URL and the same method. You can read request.headers.get('next-action') to at least distinguish action POSTs from ordinary document requests, but that identifier is a hash that changes when the build changes, so branch on its presence and never on its value.

The reliable place is inside the action:

'use server';

import { headers } from 'next/headers';
import { auth } from '@/lib/auth';
import { ratelimit } from '@/lib/ratelimit';

export async function sendMessage(_prev: State, formData: FormData): Promise<State> {
  const session = await auth();
  const key = session?.user.id ?? (await headers()).get('x-forwarded-for') ?? 'anon';

  const { success, reset } = await ratelimit.limit(`action:sendMessage:${key}`);
  if (!success) {
    const seconds = Math.ceil((reset - Date.now()) / 1000);
    // Return a value. A thrown error reaches production as an opaque digest.
    return { ok: false, error: `Too many messages. Try again in ${seconds}s.` };
  }

  // ...the real work
  return { ok: true };
}

Returning a value rather than throwing matters. An uncaught error inside a Server Action is redacted in production and surfaces to the client as a generic message with a digest, so the user is told something went wrong instead of being told to wait forty seconds. Rate limiting is a normal outcome, not an exception.

What should a 429 response actually contain?

HTTP 429 Too Many Requests is the correct status, and Retry-After is the header that makes a limiter usable by anyone other than a human staring at a browser. Its value is either a number of seconds or an HTTP date.

Alongside it, emit the limit state. The X-RateLimit-Limit / X-RateLimit-Remaining / X-RateLimit-Reset convention is not a standard but it is what most SDKs already parse. The IETF draft draft-ietf-httpapi-ratelimit-headers defines RateLimit and RateLimit-Policy as structured fields; adopt it only if your consumers understand it.

Two mistakes I have made and seen: returning 403 for a rate limit, which tells the client to stop forever rather than to retry, and returning 500, which pollutes your error rate with your own defences working correctly.

Then decide what happens when the store is unreachable, because it will be. Failing open on a login endpoint turns a Redis outage into an open brute-force window. Failing closed on a public read endpoint turns a Redis outage into a full outage of your site. I choose per route: fail closed on authentication and on anything that spends money, fail open on reads.

FAQ

Q: Can I rate limit in Next.js without Redis?
A: Only if your app runs as a single long-lived process, such as one container instance. On any serverless or autoscaled deployment the counter must live in a store shared across instances, because in-process state is multiplied by your instance count.

Q: Does Next.js middleware run on RSC prefetch requests?
A: Yes. Router prefetches are real HTTP requests that match your middleware matcher and carry the RSC: 1 header. Exclude them from user-facing quotas or they will consume a visitor's budget before the visitor clicks anything.

Q: How do I get the client IP in Next.js 15 and Next.js 16?
A: NextRequest.ip was removed in Next.js 15. On Vercel, call ipAddress(request) from @vercel/functions. Elsewhere, read the forwarded header your own trusted proxy sets and confirm that the proxy overwrites rather than appends it.

Q: Should a rate limiter fail open or fail closed when Redis is down?
A: Decide per route. Fail closed on login, signup, password reset, and payment endpoints, where failing open creates a security window. Fail open on public reads, where failing closed converts a dependency outage into a site outage.

Q: Is Vercel BotID a replacement for rate limiting?
A: No. Vercel BotID is bot detection, which answers whether a caller is automated. A rate limit answers how often any caller, human or not, may perform an expensive operation. They defend different things and compose well together.

Originally published on devya.dev. Also on eng-ahmed.com. Built by Devya Solutions.

Directus + Coolify: Should You Decouple Postgres & Redis?

Thu, 06 Aug 2026 21:39:55 +0000

This is Part 2 of the Directus + Coolify series. If you're new here, start with "Secure Your VPS Before Hackers Do" and the first Directus + Coolify post — the bundled, single-Compose-file setup — before following along with this one.

Introduction

In the first method, we coupled all of the services into one stack using a single Docker Compose file. The network between all services was created automatically, and we didn't have to start them up individually — which removes the risk of a race condition if service startup isn't handled properly.

If you're running a single app, that's genuinely the recommended way to set up Directus on a Coolify-managed VPS. Going in, I assumed there were several good reasons to split the services apart instead — more control over backups, monitoring, restarts, that kind of thing. So before recommending decoupling, I actually tested each of those assumptions on a live Coolify instance.

Most of them turned out to be wrong.

Myth 1: Restarting Directus Restarts the Whole Stack

I expected that restarting Directus inside the bundled Compose file would restart Redis and Postgres along with it. It doesn't. Coolify lets you restart each service in the stack independently — Directus, Database, and Cache each get their own Restart button, right there in the same view. No decoupling needed for this one.

Myth 2: You Need a Separate Database Resource for S3 Backups

Same story. Even with Postgres bundled inside the Directus Compose file, Coolify still gives it its own dedicated Backups option, S3 included. This isn't a separate-resource-only feature.

Myth 3: Scheduled Tasks Require Separate Services

Also not true. Coolify exposes a Scheduled Tasks tab per service, even inside a single bundled stack — complete with a Container name dropdown letting you target the cron job at just the database, or just Directus, without splitting anything apart.

What Actually Holds Up

Two things survived testing.

First: metrics. This one's confirmed directly in Coolify's own documentation — CPU and memory metrics collection is explicitly not available for Docker Compose–based deployments. If you want to see per-container resource usage through Coolify's built-in monitoring, the service needs to be created as its own standalone resource, not bundled inside a Compose file. This is a real, documented limitation of the bundled approach.

Second: sharing a database across multiple apps. This one isn't a Coolify feature at all — it's just how Docker networking works. A database defined inside one app's Compose file lives on that stack's own private network by default. A second, completely separate application can't reach it without deliberately bridging the two networks. If you've got a desktop app and a mobile app that both need to talk to the same Postgres instance, that database needs to exist as its own standalone resource from the start — it can't stay tucked inside one app's Compose file.

So this really comes down to two reasons to decouple, not five — one a genuine Coolify limitation, the other a structural fact about Docker networking. Let's set both of those up properly.

Adding the Resources

Adding the PostgreSQL Database

Dashboard → Add Project
+ Add Resource
Databases → PostgreSQL → Supabase PostgreSQL (with extensions)
Change the name to something human-friendly
Copy your username and password and save them somewhere — you'll need them shortly
Click Save
Click Start, and wait for the database to spin up (this can take a little while)

Once it's up, the status should read "Running (Healthy)."

💡 Enable metrics while you're here. In the sidebar, go to Servers → localhost → Metrics, and enable metrics. Back in your project, under Databases, click your Postgres database, then Metrics — you should now see live CPU/memory usage for it. This is the exact capability that isn't available on a bundled Compose deployment.

Adding Redis Cache

Dashboard → + Add Resource (from the project itself)
Databases → Redis
Rename it to something more convenient
Copy the Redis connection URL — you'll need it shortly
Save, then Start

Once both are running, SSH into your VPS and run:

docker ps

to confirm both containers are up.

The Docker Compose Config for Directus

This is what goes into the empty Compose file for the Directus resource:

services:
  directus:
    image: 'directus/directus:12.2.0'
    ports:
      - '8055:8055'
    volumes:
      - './uploads:/directus/uploads'
      - './extensions:/directus/extensions'
    healthcheck:
      test:
        - CMD-SHELL
        - 'wget --spider -q http://127.0.0.1:8055/server/ping || exit 1'
      interval: 10s
      timeout: 5s
      retries: 5
      start_interval: 5s
      start_period: 30s
    environment:
      SECRET: secretstring
      MARKETPLACE_TRUST: all
      DB_CLIENT: pg
      DB_HOST:
      DB_PORT: '5432'
      DB_DATABASE: postgres
      DB_USER: postgres
      DB_PASSWORD:
      CACHE_ENABLED: 'true'
      CACHE_STORE: redis
      CACHE_AUTO_PURGE: 'true'
      REDIS:
      ADMIN_EMAIL: *Emails are not allowed*
      ADMIN_PASSWORD: '1234567890'
      CORS_ENABLED: 'true'
      CORS_ORIGIN: 'true'
      CORS_CREDENTIALS: 'true'
      PUBLIC_URL:

Notice the healthcheck already uses 127.0.0.1 instead of localhost — that's the fix from Part 1. Carrying it forward here saves you from hitting the exact same "unhealthy" bug all over again.

Adding Directus

Dashboard → + Add Resource (from the project itself)
Applications → Docker Compose Empty
Paste in the Compose configuration above
Click Save
Network → Connect To Predefined Network → check the box
Services → Directus service → Settings
Add your Directus subdomain — remember to use https:// (e.g. https://directus.yourdomain.com)
Save

Connecting the Services Together

Variable	What it is
`SECRET`	A long, unguessable random string
`DB_HOST`	The name of your Postgres container — run `docker ps` on your VPS to find it
`DB_DATABASE`	The name of the database on your Postgres server (Coolify's default is usually `postgres`, but confirm it against your Postgres service)
`DB_USER`	The database username, from the Postgres service you created earlier
`DB_PASSWORD`	The password from that same Postgres service
`REDIS`	The Redis connection URL from the Redis service you created earlier
`PUBLIC_URL`	Your Directus subdomain

⚠️ Redis URL gotcha: the connection URL follows the format redis://username:password@host:port. Coolify's generated URL includes the username you set when creating the Redis resource — in my case, that username was also redis, so the URL looked like redis://redis:somelongvariable.... Directus doesn't need the username here, just the password, so strip that segment out: redis://:somelongvariable.... If you used a different username when creating your Redis resource, remove that value instead — not literally the word "redis."

Setting Up Environment Variables

Rather than hardcoding any of this directly into the Compose file, move it into Directus's environment variables:

Dashboard → Projects → Services / Directus
Environment Variables
Click + Add, enter the variable name in all caps, and its value
Save
Repeat for: SECRET, DB_HOST, DB_DATABASE, DB_USER, DB_PASSWORD, ADMIN_EMAIL, ADMIN_PASSWORD, and REDIS

Updating the Compose File

Now reference those variables instead of the raw values:

SECRET: '${SECRET}'
DB_HOST: '${DB_HOST}'

...and so on for each variable. Then:

Save
Restart

Launching Directus

Paste your Directus subdomain into the browser, and you should land on your running Directus instance. You can also click Links on the Directus service, then click the subdomain — it'll take you straight to your Directus Studio login.

Same as the last video: sign up for your free license, which arrives by email, and paste it into your Directus instance to unlock everything.

Any questions or hit a different result testing any of these yourself? Drop it in the comments — I'm genuinely curious whether this holds up across different Coolify versions and setups.

I Build a blog API with Redis - Here's every problem I Hit

Wed, 15 Jul 2026 00:51:31 +0000

I build a blog API with FastAPI + Redis + MySQL.
Three cache problems almost killed my app.
If you also face these problems!
Here's how I fixed each one.

1. Cache Penetration

What's Cache Penetration?

Cache Penetration refers to a scenario where the requested data exists neither in cache nor in the database,causing every request to hit the database directly,thereby increasing the database load.

How to solve?

1.Caching null values

Steps:

When a queried key exists in neither the cache nor the database,the result (such as null or an empty value) is cached with a short TTL.
This way,subsequent identical requests can be served directly from the cache,avoiding frequent database access.

Example:

...
redis = get_redis()
data = redis.get(cache_key)

# First, check the cache. If the cache entry exists, check whether it is __NULL__
if data is not None:
    if data == "__NULL__":
        return {"data": None, "source": "cache"}
    return {"data": data, "source": "cache"}

# Second, cache miss, query database
value = query_database(cache_key)

# Third, cache the result whether it exists in DB or not
if value is not None:
    redis.setex(cache_key, 60, value)          # cache real data
else:
    redis.setex(cache_key, 10, "__NULL__")     # cache null sentinel (short TTL)

return value

2.Cache breakdown

What's Cache breakdown?

Cache breakdownoccurs when a popular data entry expires in the cache while a massive number of concurrent requests are hitting that same data at the exact same time.All those requests penetrate straight through to the database,putting enormous pressure on it and potentially causing system performance issues.

How to solve?

Lock mechanism:When the cache expires,use a locking mechanism to ensure that only one thread can access the database and update the cache.The other threads wait util the cache is rebuilt before reading the data.

Example:

# Generate a lock key based on the given key
lock_key = f"mutex:{key}"

# Use SETNX operation to acquire the lock. If the key does not exist, set it with an expiration time of 300 seconds.
# This prevents concurrent access.
lock_acquired = redis_client.set(lock_key, "1", nx=True, ex=300)

if lock_acquired:
    try:
        # Successfully acquired the lock, query the database
        value = db_query_func(key)
        if value is not None:
            # Write the result to the cache with an expiration time of 3600 seconds
            redis_client.setex(key, 3600, value)
        return value
    finally:
        # Release the lock
        redis_client.delete(lock_key)
else:
    # Failed to acquire the lock, sleep for 0.1 seconds and then retry
    time.sleep(0.1)
    return get_data_with_mutex(key, redis_client, db_query_func)

3.Cache avalanche

What's cache avalanche?

Cache avalancerefers to a situation where a large amount of cached data expires at the same time or the cache service goes down. As a result, all requests are directly sent to the database, causing a sudden surge in the database's pressure and even leading to its downtime.

How to solve?

Randomized TTL

The core idea is to avoid a large number of keys expiring simultaneously.
When setting the expiration time for the cache, add a random value.

Example:

import random

expire_time = time + random.randint(0,300)
redis.set(key,value,ex=expire_time)

What about you?

Have you run into any of these cache problems in your projects?
Or do you use a different strategy? Let me know in the comments

If this was helpful, follow me for more backend deep dives.

Your Laravel Isn’t Slow, Your Architecture Is.

Fri, 26 Jun 2026 13:35:52 +0000

Your Laravel Isn’t Slow, Your Architecture Is: Mastering Scale with Advanced Octane & Reactive Caching

Introduction

If your Laravel API is struggling under heavy load, the instinct is often to dive deep into optimizing Eloquent queries or micro-tweaking individual lines of code. While performance profiling remains crucial, this approach often addresses symptoms, not the root cause. The real bottleneck in high-traffic Laravel deployments isn’t typically the framework itself; it’s the architectural paradigm. We need to shift our focus from optimizing individual operations to optimizing the flow and state of our application. This tutorial will guide you through two advanced strategies – leveraging Octane’s asynchronous capabilities and implementing a reactive caching strategy with Redis Streams – to unlock sustained low-latency throughput that transforms your Laravel application into a true high-performance powerhouse.

Code Layout & Architectural Walkthrough

Moving beyond basic Swoole or RoadRunner setups, we’re talking about a paradigm shift that integrates background processing directly into the request flow and introduces real-time cache invalidation.

1. Octane’s Asynchronous Power: Non-Blocking Operations

The key here is to release the HTTP response before non-critical operations complete, leveraging Octane’s persistent application state. Laravel’s dispatchAfterResponse() is the starting point, but we’ll integrate it with custom Octane bootloader hooks to ensure robust, non-blocking execution.

Conceptual Implementation:

Custom Octane Bootloader: Create a service provider that registers Octane worker lifecycle hooks. This allows you to perform setup and teardown tasks for each worker process, ensuring resources are ready.

// app/Providers/OctaneServiceProvider.php
namespace App\Providers;

use Illuminate\Support\Facades\Octane;
use Illuminate\Support\ServiceProvider;

class OctaneServiceProvider extends ServiceProvider
{
    public function boot()
    {
        Octane::onWorkerStart(function () {
            // Perform expensive, one-time worker setup here
            // e.g., warm up a complex service container dependency
        });

        Octane::onWorkerError(function (Throwable $e) {
            // Log or handle worker errors gracefully
        });

        // Consider Octane::onWorkerStop for cleanup if necessary
    }
}

Asynchronous Dispatch in Controllers/Services: In your critical API endpoints, identify operations that don’t strictly need to block the user’s response (e.g., audit logging, sending non-critical notifications, syncing data to a third-party service).

// app/Http/Controllers/OrderController.php
namespace App\Http\Controllers;

use App\Jobs\ProcessAuditLog;
use App\Jobs\SyncOrderToCRM;
use Illuminate\Http\Request;
use Illuminate\Routing\Controller;

class OrderController extends Controller
{
    public function store(Request $request)
    {
        // ... (Validate request, create order, etc.) ...
        $order = Order::create($request->all());

        // Dispatch non-critical jobs to the queue *after* response is sent
        // This leverages Octane's persistent connections without blocking the user.
        ProcessAuditLog::dispatch($order->id, 'Order Created')->afterResponse();
        SyncOrderToCRM::dispatch($order)->afterResponse();

        return response()->json(['message' => 'Order created successfully!', 'order_id' => $order->id], 201);
    }
}

By using ->afterResponse(), Laravel ensures these jobs are pushed to your configured queue (e.g., Redis, database) only after the HTTP response has been fully sent to the client. Octane’s persistent workers ensure that the queue workers are readily available and efficient, minimizing latency overhead for job dispatching.

2. Reactive Caching with Redis Streams for Real-time Invalidation

Simple TTL-based caching often leads to stale data or necessitates aggressive, short TTLs that negate performance benefits. A reactive caching strategy uses Redis Streams to broadcast data changes and trigger real-time invalidation across your data layer.

Conceptual Implementation:

Publisher (Model Event Listener): Whenever a critical data model is updated, deleted, or created, an event is published to a Redis Stream.

// app/Providers/AppServiceProvider.php (or a dedicated EventServiceProvider)
namespace App\Providers;

use App\Models\Product;
use App\Services\CacheInvalidationService;
use Illuminate\Support\ServiceProvider;

class AppServiceProvider extends ServiceProvider
{
    public function boot(CacheInvalidationService $cacheInvalidationService)
    {
        Product::created(fn (Product $product) => $cacheInvalidationService->publish('product_updated', $product->id));
        Product::updated(fn (Product $product) => $cacheInvalidationService->publish('product_updated', $product->id));
        Product::deleted(fn (Product $product) => $cacheInvalidationService->publish('product_deleted', $product->id));
    }
}

// app/Services/CacheInvalidationService.php
namespace App\Services;

use Illuminate\Support\Facades\Redis;

class CacheInvalidationService
{
    protected string $streamKey = 'cache_invalidation_stream';

    public function publish(string $event, int $entityId, array $data = []): void
    {
        Redis::xadd($this->streamKey, '*', [
            'event' => $event,
            'entity_id' => $entityId,
            'timestamp' => now()->timestamp,
            'data' => json_encode($data),
        ]);
    }
}

Consumer (Queue Worker/Octane Worker): A dedicated listener consumes this Redis Stream. Upon receiving an event, it knows exactly which cache keys (for complex objects or query results) need to be invalidated.

// app/Jobs/ProcessCacheInvalidationStream.php (a long-running Octane-aware consumer)
namespace App\Jobs;

use App\Services\CacheManagerService; // Manages complex cache keys
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Redis;
use Illuminate\Support\Facades\Log;

class ProcessCacheInvalidationStream implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function handle(CacheManagerService $cacheManagerService): void
    {
        $streamKey = 'cache_invalidation_stream';
        $consumerGroup = 'cache_invalidator_group';
        $consumerName = gethostname() . '-' . uniqid(); // Unique consumer name

        // Create consumer group if it doesn't exist
        try {
            Redis::xgroup('CREATE', $streamKey, $consumerGroup, '$', 'MKSTREAM');
        } catch (\Exception $e) {
            // Group likely already exists
        }

        // Continuously read from the stream
        while (true) {
            $messages = Redis::xreadgroup('GROUP', $consumerGroup, $consumerName, 'BLOCK', 10000, 'COUNT', 10, 'STREAMS', $streamKey, '>');

            if (!empty($messages[$streamKey])) {
                foreach ($messages[$streamKey] as $id => $message) {
                    $event = $message['event'];
                    $entityId = (int<span cl

How I Stopped a Race Condition from Creating Free Money in My Wallet System

Sat, 13 Jun 2026 14:56:16 +0000

The Bug That Doesn't Show Up in Testing.
When I set out to build a production-grade wallet system, I expected the hard parts to be the database schema, the transaction logic, or the authentication layer.

What I didn't expect was that the most dangerous bug in the system would have nothing to do with wrong code - it would be about timing.

This is the story of a race condition that lets users spend money they don't have, why it's invisible until it hits production, and how Redis turned it into a non-issue.

What a Wallet System Actually Does
Before we get to the bug, a quick picture of the system. A wallet holds a balance. Users can:

Credit - add funds
Debit - spend funds
Transfer - move funds to another user
Reverse - undo a transaction
Every operation reads the current balance, validates it, and writes a new transaction record. Simple enough. The problem hides in that gap between reading and writing.

The Race Condition
Here's the scenario. A user has ₦5,000 in their wallet. They're on a slow connection, so they tap "Pay" twice. Two HTTP requests hit the server within milliseconds of each other. Both are trying to debit ₦3,000.

Without any concurrency protection, this is what happens:

Request A → reads balance: ₦5,000
Request B → reads balance: ₦5,000

Request A → ₦5,000 - ₦3,000 = ₦2,000 ✓ writes transaction
Request B → ₦5,000 - ₦3,000 = ₦2,000 ✓ writes transaction
Both requests read ₦5,000. Both pass the balance check. Both write a successful debit. The user just spent ₦6,000 from a ₦5,000 wallet - and the system called both transactions successful.

This isn't a theoretical edge case. Under any real concurrent load, this happens. And the worst part: it never shows up in testing, because your test runner doesn't fire two real HTTP requests at the same millisecond.

Why a Database Transaction Doesn't Save You
The instinct here is to reach for a database transaction. Wrap the whole thing in BEGIN / COMMIT and let Postgres sort it out.

The problem is that the race happens before the database transaction begins. Both requests read the balance from the database (or worse, from a cache), both pass the validation check in application code, and then both open their database transactions. The damage is already done.

You need a lock at the application layer, before the read even happens.

The Fix: Distributed Locking with Redis
The solution is to ensure that no two operations on the same wallet can run at the same time. Before any balance operation begins, it must claim exclusive access. If another operation already holds the claim, it retries until the lock is free, or gives up after a timeout.

This is a distributed lock, and Redis is the right tool for it. Here's the implementation:

// src/utils/lock.utils.ts
import { randomUUID } from "crypto";
import redisClient from "../services/redis.service";

const LOCK_TTL_MS = 10_000;
const LOCK_RETRY_COUNT = 50;
const LOCK_RETRY_DELAY_MS = 100;

const RELEASE_SCRIPT = `
  if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("DEL", KEYS[1])
  else
    return 0
  end
`;

const acquireLock = async (key: string): Promise<string | null> => {
  const token = randomUUID();
  const result = await redisClient.set(key, token, "PX", LOCK_TTL_MS, "NX");
  return result === "OK" ? token : null;
};

export const withLock = async <T>(userId: string, fn: () => Promise<T>): Promise<T> => {
  const key = `wallet:lock:${userId}`;
  let token: string | null = null;

  for (let attempt = 0; attempt <= LOCK_RETRY_COUNT; attempt++) {
    token = await acquireLock(key);
    if (token) break;
    await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_DELAY_MS));
  }

  if (!token) throw new Error("Could not acquire lock — resource is busy");

  try {
    return await fn();
  } finally {
    await redisClient.eval(RELEASE_SCRIPT, 1, key, token);
  }
};

Let's break down each decision.

SET key token PX 10000 NX
This is a single atomic Redis command that does three things at once:

NX — only set the key if it does not already exist
PX 10000 — expire the key after 10 seconds
The value is a unique token (UUID), not just 1 or "locked"
The atomicity of SET NX is the core guarantee. There is no gap between "check if key exists" and "set the key" — it's one operation. Two requests racing to acquire the same lock cannot both succeed.

Retry logic
Rather than immediately rejecting a request when the lock is held, the implementation retries up to 50 times with a 100ms delay between each attempt - giving it up to 5 seconds to acquire the lock before giving up. This handles brief overlaps gracefully without dropping legitimate requests.

The TTL
The 10-second expiry is a safety net. If the application crashes mid-operation and fails to release the lock, it will expire automatically. Without this, one crash would lock that user's wallet forever.

The Lua script on release
This is where most implementations get it wrong. A naive release looks like this:

await redisClient.del(key); // ❌ dangerous
Here's why that's broken: imagine a slow operation acquires the lock, but takes longer than 10 seconds. The TTL expires. A second operation acquires the lock. Now the first operation finishes and calls del - it just released the second operation's lock, not its own. The guarantee is broken.

The Lua script fixes this by checking the token before deleting:

if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
else
  return 0
end

Because Lua scripts execute atomically in Redis, the check-and-delete is a single uninterruptible operation. You can only release a lock you actually own.

The withLock wrapper
Rather than exposing raw acquireLock / releaseLock functions and trusting every caller to use them correctly, the implementation wraps the entire pattern in a higher-order function:

// src/services/wallet.service.ts
export const debitWallet = async (userId: string, amount: number): Promise<Transaction> => {
  return await withLock(userId, async () => {
    const wallet = await prisma.wallet.findUnique({ where: { userId } });

    if (!wallet || wallet.balance.lessThan(amount)) {
      throw new Error("Insufficient balance");
    }

    // create transaction, update balance...
    return transaction;
  });
};

The finally block inside withLock ensures the lock is always released, even if the operation throws. A lock that leaks on error is as bad as no lock at all.

Preventing Deadlocks on Transfers
A transfer touches two wallets - sender and receiver. If two transfers run simultaneously in opposite directions (A→B and B→A), they can deadlock: each holds one lock and waits for the other indefinitely.

The fix is to always acquire locks in the same order, regardless of who is sending to whom. We sort the user IDs before acquiring:

export const withMultiLock = async <T>(userIds: string[], fn: () => Promise<T>): Promise<T> => {
  const sortedIds = [...new Set(userIds)].sort();
  const keys = sortedIds.map((id) => `wallet:lock:${id}`);
  const tokens: string[] = [];

  try {
    for (const key of keys) {
      let token: string | null = null;
      for (let attempt = 0; attempt <= LOCK_RETRY_COUNT; attempt++) {
        token = await acquireLock(key);
        if (token) break;
        await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_DELAY_MS));
      }
      if (!token) throw new Error("Could not acquire lock — resource is busy");
      tokens.push(token);
    }
    return await fn();
  } finally {
    await Promise.all(keys.map((key, i) => releaseLock(key, tokens[i])));
  }
};

// In transfer service:
const result = await withMultiLock([senderId, receiver.id], async () => {
  // both wallets locked — safe to proceed
});

If A→B and B→A both arrive at the same time, both will sort to [A, B] and try to acquire A's lock first. One wins, the other waits. No deadlock.

The Full System
The distributed lock is one layer in a larger stack. The complete wallet engine includes:

Idempotency - Idempotency-Key header on credit and debit prevents duplicate transactions on network retries, backed by a 24-hour Redis TTL
Webhook delivery with retry - BullMQ queue, 4 attempts, exponential backoff (2s base), per-attempt audit trail
Balance caching - 60-second Redis TTL, invalidated on every write
Token blacklisting - SHA-256 hashed JWTs stored in Redis with matching expiry, making logout actually secure
Cursor-based pagination - transaction history that scales
Full test suite - 103 tests, all mocked, no database required to run.
Built with TypeScript, Express, Prisma, PostgreSQL, and Redis (ioredis). Deployed on Render with a GitHub Actions CI/CD pipeline.

Why This Matters
Race conditions in financial systems are not exotic. They're the default outcome when you don't explicitly prevent them. Most wallet tutorials never mention it. Most wallet implementations in production have it.

The fix isn't complicated - but you have to know to look for it.

If you're building anything that touches money, check your balance reads. If there's no lock between the read and the write, the race condition is already there.

Open for Collaboration
This project is open source and actively welcoming contributors. Whether you want to add a feature, improve test coverage, or spot something that could be done better - PRs are open.

Check out the CONTRIBUTING.md for how to get set up and where to send your changes.

GitHub: https://github.com/tochi27/Wallet_System.git

Live API + Swagger: https://wallet-system-api-16cv.onrender.com/api-docs

Syncing the Real Market: Consuming Brapi and Feeding Redis with Spring Boot

Sun, 10 May 2026 11:03:34 +0000

Hello everyone!

Continuing the My Broker B3 series, today we'll talk about an essential component on the B3 side of our simulator: the B3 Market Sync API.

Before building the engine that decides whether an order gets executed or rejected, we need to ensure it has access to real market prices. That's exactly what b3-market-sync-api does: fetches real asset prices from brapi.dev and stores them in Redis, ready for real-time consumption.

️ What is the Market Sync?

This microservice has a single, well-defined responsibility: to be the market data producer of the ecosystem. It operates completely asynchronously and independently — no other service needs to call it directly.

The flow is simple and elegant:

[brapi.dev] ◀── Feign Client ── [MarketSyncScheduler]
                                         │
                              market:price:{TICKER}
                                         │
                                      [Redis]
                                         │
                              [B3 Matching Engine] (next post)

MVP Focus

The goal here is clear: have real prices available in Redis with low latency. The Matching Engine we'll build in the next post won't query a database or make HTTP calls to get prices — it goes straight to Redis. This ensures the matching decision happens in microseconds.

In this phase, I prioritized:

Scheduled sync with market hours guard
Redis cache with TTL to prevent stale data
External API rate limit protection
Critical bug fixes found during code review
REST API for querying cached prices
Swagger documentation

️ Tech Stack

Technology	Usage
Java 21 + Spring Boot 3.5.11	Service core
Spring Cloud OpenFeign	Declarative HTTP client for brapi.dev
Spring Data Redis	High-performance cache
Spring Scheduling	Periodic synchronization
Jackson JSR310	Java 8 Date/Time serialization support
SpringDoc OpenAPI	Swagger UI documentation

️ Implementation Pillars

1. Feign Client — Consuming Brapi

Spring Cloud OpenFeign lets you declare an HTTP client as a simple interface, with no boilerplate code:

@FeignClient(name = "brapiClient", url = "${app.brapi.url}")
public interface BrapiClient {

    @GetMapping("/quote/{ticker}")
    BrapiResponseDTO getQuote(
        @PathVariable String ticker,
        @RequestParam String token
    );
}

The URL and token are injected via application.yaml, keeping the code clean and environment-configurable.

2. Scheduler with Market Hours Guard

The heart of the service is a scheduled job that fires every 30 minutes. But before any API call, it checks whether the market is open:

@Scheduled(fixedRateString = "${app.sync.interval:1800000}")
public void sync() {
    if (!isMarketOpen()) {
        log.info("Sync aborted: market is closed (outside trading hours or weekend).");
        return;
    }
    // ... fetch and store prices
}

private boolean isMarketOpen() {
    ZonedDateTime now = ZonedDateTime.now(ZoneId.of("America/Sao_Paulo"));
    DayOfWeek day = now.getDayOfWeek();
    int hour = now.getHour();

    if (day == DayOfWeek.SATURDAY || day == DayOfWeek.SUNDAY) {
        return false;
    }

    // B3 trading hours: 10:00 to 18:00 (Sao Paulo time)
    return hour >= 10 && hour < 18;
}

Important detail: we use ZonedDateTime with America/Sao_Paulo explicitly. Inside a Docker container, the JVM may default to UTC — which would make the market hours guard behave incorrectly. We also pass -Duser.timezone=America/Sao_Paulo in the Dockerfile ENTRYPOINT to ensure the JVM honors the correct timezone:

ENTRYPOINT ["java", "-Duser.timezone=America/Sao_Paulo", "-jar", "app.jar"]

3. Redis Cache with Correct TTL

The MarketDataService stores each quote in Redis with a standardized key and 5-minute TTL:

private static final Duration CACHE_TTL = Duration.ofMinutes(5);

public void saveToCache(BrapiResultDTO quote) {
    String key = CACHE_KEY_PREFIX + quote.getTicker();
    // TTL of 5 minutes — if sync stops, stale prices expire quickly
    // preventing the Matching Engine from using outdated data
    redisTemplate.opsForValue().set(key, quote, CACHE_TTL);
    log.info("Ticker {} updated in cache: R$ {}", quote.getTicker(), quote.getRegularMarketPrice());
}

The TTL is fundamental: if the service goes down, prices expire in 5 minutes. The Matching Engine then fails to find the price in Redis and rejects orders — correct and predictable behavior, much better than using hours-old prices.

4. Epoch Timestamp Deserialization

A subtle bug I found: brapi.dev returns the regularMarketTime field as a Unix epoch (integer), but Jackson was trying to deserialize it as LocalDateTime (ISO string). This causes silent deserialization failure.

The solution was a custom deserializer:

public class EpochToLocalDateTimeDeserializer extends StdDeserializer<LocalDateTime> {

    private static final ZoneId SAO_PAULO = ZoneId.of("America/Sao_Paulo");

    @Override
    public LocalDateTime deserialize(JsonParser parser, DeserializationContext context)
            throws IOException {
        long epoch = parser.getLongValue();
        return LocalDateTime.ofInstant(Instant.ofEpochSecond(epoch), SAO_PAULO);
    }
}

And annotating it on the DTO:

@JsonDeserialize(using = EpochToLocalDateTimeDeserializer.class)
private LocalDateTime regularMarketTime;

5. Rate Limit Protection

Since we use the brapi.dev free plan, we added a delay between calls to avoid throttling:

for (String ticker : brapiProperties.getTickers()) {
    try {
        BrapiResponseDTO response = brapiClient.getQuote(ticker.trim(), brapiProperties.getToken());
        if (nonNull(response.getResults()) && !response.getResults().isEmpty()) {
            marketDataService.saveToCache(response.getResults().getFirst());
        }
        // Small delay between calls to avoid hitting rate limit on free plan
        Thread.sleep(200);
    } catch (Exception e) {
        log.error("Failed to sync ticker {}: {}", ticker, e.getMessage());
    }
}

The per-ticker try/catch is essential — if one asset fails, the loop continues for the rest.

6. REST API + Swagger

Although the Matching Engine consumes prices directly from Redis, we added REST endpoints to make debugging and observability easier:

Method	Endpoint	Description
GET	`/api/v1/quotes`	List prices for all configured tickers
GET	`/api/v1/quotes/{ticker}`	Get price for a specific ticker

Documented via Swagger UI at http://localhost:8096/swagger-ui.html.

Security — Token Out of Code

A critical issue I fixed: the brapi.dev token was hardcoded as a default value in application.yaml:

# ❌ Before — token exposed in public repository
token: ${BRAPI_TOKEN:mFk9YRLijAtxsP5y1XxxxX}

# ✅ After — no fallback, variable is required
token: ${BRAPI_TOKEN}

Never expose credentials as default values in public repositories. The application now fails on startup if the variable is not provided — correct behavior.

✅ Validating the Execution

With the application running locally:

✅ Tomcat started on port 8096
✅ Application initialized in under 3 seconds
✅ Scheduler fired and logged: Sync aborted: market is closed (outside trading hours or weekend)
✅ Swagger UI accessible at http://localhost:8096/swagger-ui.html

What's Next?

With b3-market-sync-api up and feeding Redis with real prices, we have the foundation the next service needs to work. In the next post we'll build the B3 Matching Engine — the engine that will consume these prices and decide whether the broker's orders are executed or rejected.

Got any questions about the Redis integration or the market hours guard? Drop them in the comments!

About the Series

⬅️ Previous Post: From Stream to Database: Processing Market Data with Spring Boot, Redis and Flyway

➡️ Next Post: The Heart of B3: Building the Matching Engine with RabbitMQ and Redis (coming soon)

Series Index: Series Roadmap

Links:

Redis Hacks Your Future Self Will Thank You For (No More Crashes)

Fri, 08 May 2026 20:52:18 +0000

Redis isn't just a fancy cache - it's a grenade. Pull the wrong pin (ahem, maxmemory), and BOOM
maxmemory & Eviction Policies: Stop Redis From Ghosting Your Data

The Problem: Redis doesn't magically clean up. Set maxmemory too high, and it'll eat your RAM like a Cookie Monster. Set it too low, and it'll evict data mid-transaction.

The Fix:

First:set the max memory limit that Redis can't exceed out of your server resources
Use the eviction strategy option that meets your needs. Example:
volatile-lru for mixed workloads (bye-bye, least-recently-used expired keys).
allkeys-lfu if you're a data hoarder (prioritize frequently used keys, even if they're immortal).

All options:

EvictionsPro Tip: Test evictions before your app trends on Hacker News.

Lua Scripts: Your Atomic Weapon Against Race Conditions

Why It Matters:
5 clients updating the same key? Chaos. Lua scripts execute atomically.
LuaLater, you can pull your data from Redis into whatever DB you use for consistency.
Retries, Timeouts & Processors (Yes, That's a Word Now)

Retries: 3 retries with exponential backoff. More? You're just DDoSing yourself.
Timeouts: connect_timeout: 5s, read_timeout: 2s. Your app isn't a hostage negotiator.
Processors: Redis is single-threaded. If your CPU cores are screaming, shard or migrate heavy ops to background workers.
Redis isn't 'set it and forget it' - it's 'set it, tweak it, and make your on-call rotations boring again'. Try one tip today, and thank me when you're sleeping through deploy windows.
Comment your worst Redis horror story - bonus points for plot twists involving FLUSHALL
Thanks for reading

Why Your "Fail-Fast" Strategy is Killing Your Distributed System (and How to Fix It)

Tue, 05 May 2026 19:11:27 +0000

It's 2 AM. PagerDuty fires. Redis master is down. Your application, trained to fail fast, dutifully fails — every single request, all at once. By the time Sentinel promotes a new master 12 seconds later, you've already generated 40,000 errors and three escalation calls. The system recovered on its own. Your application didn't let it.

This is the story of how "good engineering" can make a 12-second infrastructure event into a 12-minute outage — and how to design boundaries that prevent it.

tl;dr — During infrastructure failovers (Redis, Kafka, etcd), blind fail-fast amplifies instability. Bounded retry — centralized, time-boxed, invisible to business logic — absorbs the 10–15 second recovery window without leaking infrastructure noise to users. Resilience is not a library. It is a contract between layers.

The Core Question

When your session storage — Redis, Memcached, or any stateful dependency — goes temporarily unavailable, you face a fundamental architectural choice:

Should you fail fast? Or should you retry?

We all learned fail-fast as gospel. And it is — until it isn't. During transient infrastructure events like leader elections, blind fail-fast propagates instability instead of containing it. The response you choose determines whether the incident resolves itself in 12 seconds or snowballs into a 12-minute outage with three bridge calls.

What Actually Happens During Failover

To understand why fail-fast can backfire, look at the mechanics of a Redis Sentinel failover:

Phase	Duration	What Happens
Detection	~10–12s	Sentinel quorum detects master is down
Election	~1–2s	Sentinels agree on a new master
Promotion	~1s	Replica promoted, clients notified
Reconnection	~1–3s	Clients re-establish connections

Note: these phases overlap. Total failover typically completes in 12–15 seconds, not the sum of individual phases. Reconnection time also depends heavily on your client library — a Sentinel-aware client with topology refresh (e.g., Lettuce, go-redis with Sentinel support) reconnects in under a second, while a naive connection pool can take 30s+.

During this window, your application sees TCP dial timeouts and connection resets. Nothing is broken. No data is lost. The system is doing exactly what it was designed to do — electing a new leader. Your application just needs to not panic for 12 seconds.

If your application fails immediately on the first connection timeout during this window, four things happen in rapid succession:

1. Instability Amplification

A 3-second infrastructure blip becomes a user-visible outage. Every request during the failover window returns an error, even though the system would have recovered on its own.

2. Infrastructure Semantics Leak Upward

Your business layer now exposes raw infrastructure details — "Redis connection refused" — to clients that have no idea what Redis is or why it matters.

3. Uncontrolled Client Retries

Clients receiving errors start retrying independently. If you have 1,000 concurrent users and each retries 3 times, you just turned 1,000 QPS into 3,000 QPS — hitting an infrastructure layer that's already struggling to stabilize.

4. Retry Storms

This is the catastrophic outcome. Unbounded retries create cascading load amplification. CPU spikes prevent recovery. The system enters an instability feedback loop where the act of trying to recover keeps the system down. I've seen retry storms take down entire regions.

"Your timeout config was technically correct. Your system was functionally down. That's not a timeout problem — that's a design problem."

Here's the distinction that actually matters in production: the failure TYPE must determine your recovery strategy.

	Infrastructure-Level	Business-Level
Examples	Network jitter, leader election, connection reset, `READONLY` replica response	Validation error, permission denial, domain rule violation
Nature	Transient — will resolve on its own	Permanent — retrying won't help
Strategy	ABSORB — retry within bounds	FAIL FAST — return error immediately

Treating a leader election timeout the same as a schema validation error is an architectural mistake. One will resolve in seconds; the other will never succeed no matter how many times you retry.

The Failure Boundary Model

This is the architectural pattern that makes everything work:

The retry boundary sits in the infrastructure client wrapper — the thin layer between your business code and the dependency client. Not in HTTP middleware, not in individual service handlers, not in a sidecar. In the client wrapper itself.

Why does this matter? Because if retry logic exists at multiple layers, you get retry amplification. I've seen teams with retry in the HTTP handler, the service layer, AND the Redis client — producing 3 × 3 × 3 = 27 attempts per original request. That's not resilience. That's a DDoS against your own infrastructure.

Key principles:

Retry belongs at the infrastructure boundary — one place, one policy.
Business logic must remain fail-fast — semantic errors should never be retried.
By the time an error reaches the client, it has been vetted and classified. We are designing for predictability.

Bounded Retry: Implementation

If we're going to retry, we must do it with discipline. Four pillars:

1. Centralized

Retry logic lives in one place — the infrastructure client wrapper. Not in individual handlers, not in middleware, not in the business layer. One retry boundary per dependency, one policy, one set of metrics.

2. Time-Bounded

We define a retry budget — for example, 15 seconds. Why 15? Because it encapsulates the 10–12 second Sentinel detection window plus a margin for stabilization and reconnection. Time-based budgets are superior to pure attempt counts because they normalize across different failure modes — a retry that takes 5s per attempt behaves very differently from one that takes 100ms.

3. Attempt-Limited with Jitter

Maximum 2–3 retry attempts within the budget window, with exponential backoff and jitter. Without jitter, synchronized retries from multiple application instances create a thundering herd — everyone hits the new master at exactly the same moment.

4. Invisible to Business Logic

If the retry succeeds within the budget, the business layer never knew there was a problem. If it fails, the business layer receives a clean, classified error — not a raw TCP stack trace that means nothing to anyone above the infrastructure layer.

Here's what this looks like in practice:

// Bounded retry wrapper — lives in the infrastructure client layer
func withBoundedRetry(ctx context.Context, budget time.Duration, maxAttempts int, op func() error) error {
    deadline := time.Now().Add(budget)
    var lastErr error

    for attempt := 0; attempt < maxAttempts; attempt++ {
        if time.Now().After(deadline) {
            break
        }

        lastErr = op()
        if lastErr == nil {
            return nil // success — business layer never knew
        }

        if !isRetryable(lastErr) {
            return normalizeError(lastErr) // permanent failure — fail fast
        }

        // Exponential backoff with jitter
        backoff := time.Duration(1<<attempt) * 500 * time.Millisecond
        jitter := time.Duration(rand.Int63n(int64(backoff / 2)))
        select {
        case <-time.After(backoff + jitter):
        case <-ctx.Done():
            return ctx.Err()
        }
    }

    return normalizeError(lastErr) // budget exhausted — fail deterministically
}

┌─────────────────────────────────────────────┐
│          Retry Budget: 15 seconds           │
│                                             │
│  Attempt 1  →  timeout (5s)  →  backoff     │
│  Attempt 2  →  timeout (5s)  →  backoff     │
│  Attempt 3  →  success                      │
│                                             │
│  Total elapsed: ~11s                        │
│  Application impact: ZERO                   │
│                                             │
│  ─── OR ───                                 │
│                                             │
│  Budget exhausted → FAIL DETERMINISTICALLY  │
│  Clean, classified error to business layer  │
└─────────────────────────────────────────────┘

"Retry is not infinite. Retry is time-boxed. Once the budget is exhausted, we fail deterministically."

Error Normalization

This is where most teams get it wrong. They retry everything — or nothing. The retry decision must be driven by error classification:

Raw Error	Normalized To	Retryable?	Why
`TCP dial timeout`	`UNAVAILABLE`	Yes	Connection not established, may recover
`Connection reset`	`UNAVAILABLE`	Yes	Transient network disruption
`READONLY` (replica)	`UNAVAILABLE`	Yes	Sentinel failover in progress — replica not yet promoted
`Leader election in progress`	`UNAVAILABLE`	Yes	Raft/consensus transition
`OOM command not allowed`	`RESOURCE_EXHAUSTED`	No	Backpressure — retrying makes it worse
`WRONGTYPE`	`INVALID_ARGUMENT`	No	Schema error — will never succeed
`NOPERM` / `Permission denied`	`PERMISSION_DENIED`	No	Auth failure — will never succeed
`NOT_FOUND`	`NOT_FOUND`	No	Semantic absence — retry won't create the resource

The READONLY case deserves special attention. During Sentinel failover, a replica that hasn't been promoted yet responds with READONLY to write commands. If your retry layer treats this as a permanent error, your circuit breaker trips, clients get errors, and a 12-second failover becomes a 5-minute outage while someone manually resets the breaker. Classify READONLY as UNAVAILABLE — it will resolve when the new master is promoted.

The rule is simple: you cannot leak internal implementation details up the stack. Your retry layer must inspect and reclassify errors — not just map them 1:1. Error semantics must align across every layer.

The Relationship with Circuit Breakers

Bounded retry is the inner loop — it handles transient failures within a known recovery window. But what if the dependency is truly down, not just transitioning?

That's where circuit breakers serve as the outer loop:

Bounded retry absorbs transient events (leader election, network jitter) — seconds.
Circuit breaker protects against sustained outages (dependency truly dead) — minutes.

Without a circuit breaker, sustained failures chew through retry budgets on every request, wasting resources. Without bounded retry, every transient blip trips the circuit breaker unnecessarily. They are complementary, not redundant.

Observability: Instrument the Boundary

A production retry boundary must emit metrics. Without them, you're flying blind:

retry_attempt_total — how often retries fire (by dependency, by error type)
retry_budget_exhausted_total — how often the full budget is consumed without success
retry_success_on_attempt — which attempt number succeeds (histogram)
error_classification — distribution of retryable vs non-retryable errors

The key alert: if retry budget exhaustion rate exceeds ~5%, either your budget is too tight or your dependency is degraded beyond transient. This is the signal that distinguishes a leader election from a real outage — and it's the signal that should trigger your circuit breaker.

Beyond Redis: A Universal Pattern

If this looks Redis-specific, zoom out. The bounded retry pattern applies to any stateful dependency with leader election:

Redis Sentinel — master failover with quorum detection, 10–15s window
NATS JetStream — stream leader election in the Raft group, typically 2–5s with default election timeout
etcd / Consul — Raft leader election, ~1–2s with default settings, but watch streams may buffer longer
Kafka — partition leader election via controller, typically 5–15s depending on replica.lag.time.max.ms and ISR size
CockroachDB / TiKV — range leader election, similar Raft mechanics

The mechanics are the same everywhere: a detection window, a brief period of unavailability, and then recovery. Design your retry budget to absorb that window. Calibrate the budget to the specific system — 15s for Redis Sentinel, 5s for NATS, 20s for Kafka.

The Cross-Layer Contract

Resilience is not a library you import. It is a contract between layers:

Layer	Responsibility
Infrastructure	Absorbs transient instability via bounded retry
Business	Remains fail-fast for semantic integrity
Client	Retries only when signaled retryable

When failure is bounded and classified, the system becomes predictable. And predictability is the foundation of operational confidence.

Resilience Checklist

[ ] Retry Budget: Is my retry window matched to the dependency's failover time (e.g., 15s for Redis)?
[ ] Jitter: Do my retries have randomized sleep to avoid the "Thundering Herd"?
[ ] Error Classification: Does my code distinguish between READONLY (retryable) and PERMISSION_DENIED (not retryable)?
[ ] Centralization: Is my retry logic in the client wrapper, not leaked across handlers?
[ ] Observability: Do I have an alert if "Retry Budget Exhausted" exceeds 5%?

Key Takeaways

Fail fast — but not during transient infrastructure events. A leader election is not a business error. Don't treat it like one.
Retry must be bounded. Time-boxed, attempt-limited, with jitter. No open-ended retry loops.
Retry must be centralized. One retry boundary per dependency, at the infrastructure layer. Retry in multiple layers = retry amplification.
Failure semantics must be normalized. Retryable vs non-retryable must be explicit. Watch for READONLY — the most common Sentinel failover gotcha.
Resilience requires cross-layer alignment. Bounded retry (inner loop) + circuit breaker (outer loop) + observability = production-grade resilience.

Frequently Asked Questions

Should distributed systems always fail fast?

No. Fail fast for business-level errors (validation, permission, domain rules), but use bounded retry for transient infrastructure failures like leader election and temporary network instability.

What is a reasonable retry budget for Redis Sentinel failover?

In many production setups, 12-15 seconds is a practical starting point because it usually covers Sentinel detection, promotion, and client reconnection. Calibrate with your own failover timings and SLOs.

If the service already retries, should the client also retry?

Only when explicitly signaled retryable. Blind retries at both layers often create retry amplification and can trigger a retry storm.

How is bounded retry different from a circuit breaker?

Bounded retry handles short transient windows (inner loop). Circuit breaker handles sustained dependency failure and stops repeated expensive attempts (outer loop).

Why not use a Service Mesh (Istio) for retries?

While Mesh can retry, the application layer has better "semantic awareness." Only the app knows if a specific error is safe to retry based on idempotency.

When should I NOT use Bounded Retry?

For non-idempotent operations unless you have a robust request-ID tracking system. For business errors (400s), always fail fast.

Final Thought

Distributed systems are not about avoiding failure.
They are about designing boundaries.

If retry is everywhere, the system becomes unpredictable.
If retry is nowhere, transient instability leaks upward.

The goal is not infinite retry.
The goal is bounded retry.

That boundary is what keeps systems stable.

Resilience is not a library. It is a contract between layers.

Based on a talk I gave on failure boundary design in distributed systems.

Originally published at harrisonsec.com. Listen to the deep dive audio for a detailed walkthrough.

✨ A New Stack — Introducing Redis Nuxt Blog: A Sci-Fi Blog Boilerplate from a Galaxy Not So Far Away

Thu, 23 Apr 2026 16:49:31 +0000

“A long time ago, in a server rack far, far away, a lone developer sought a faster way to publish their thoughts across the galaxy…”

If you’re a developer roaming the cosmic void in search of the perfect blogging platform — one fast enough to outrun hyperspace lag, simple enough to configure from a single holocron file, and powered by the mighty Force of Redis — your quest ends here.

Welcome to Redis Nuxt Blog, a futuristic, high-performance boilerplate forged with Nuxt 3, Redis Stack, and a containerized fleet of Docker Star Destroyers.
Inspired by the legendary Salvatore “antirez” Sanfilippo, creator of Redis and intergalactic craftsman of absurdly fast data structures, this project aims to honor that sci-fi spirit — equal parts elegance, speed, and space-wizardry.

Grab your lightsabers, devs. We’re jumping to lightspeed. ⚡

Why Another Blog Boilerplate?

Because most blog starters are like broken droids — sluggish, uncustomizable, and likely to beep something rude at you when you change a font.

Redis Nuxt Blog is built for developers who want:

A fast, SSR-friendly, SEO-happy front-end
A database engine so quick it would embarrass a pod-racer
A deeply configurable UI without fiddling with 27 separate config files
A ready-to-launch Docker setup
A CLI so efficient it feels like talking to R2-D2 (but without the panic screams)
A theming system powered by Tailwind CSS, customizable from one file
Markdown-powered post content
Dynamic excerpts and content width control

✨ Core Features (Translated from Wookiee)

Nuxt.js 3 Frontend

Modern. Fast. SSR-powered.
Perfect for blogs, portals, and broadcasting your galactic manifestos.

⚡ Redis Stack as the Primary Database

Forget SQL. Forget slow disks.
We harness the Quantum Hyperdrive (a.k.a RedisJSON + Redis Sorted Sets) to store posts with blazing speed.

Salvatore Sanfilippo didn’t intend Redis to be a sci-fi artifact… but let’s be honest, it might as well have been forged from midichlorians.

One Config File to Rule Them All (Now With Full Tailwind Integration)

Every visual setting — fonts, weights, palette, spacing, even content width — comes from:

config/blog.config.ts

This single file now controls:

Typography per heading level
Full Tailwind color palette generation
Global content max-width
Header title & tagline
Layout constraints

Balance to the styling Force, at last.

✂️ Post Excerpts (New!)

Enable post previews on the homepage with:

postExcerpt.enabled
postExcerpt.maxLength

Perfect for teasing your readers like a Holonet trailer.

Navigate between posts like browsing encrypted Jedi archives.

Markdown Support

Posts are fully Markdown-powered — headings, lists, code blocks, even multi-line input via CLI.

Write your galactic knowledge in pure Markdown. No HTML acrobatics needed.

Dockerized Everything

Spin up Nuxt + Redis with one command:

docker compose up -d --build

Your dev environment becomes as consistent as Stormtrooper aim (but… good).

Content CLI (Upgraded!)

Create, list, delete posts with simple commands:

npm run new-post npm run post:list npm run post:delete

The CLI now supports:

Multi-line Markdown input
Automatically generated slugs
Default author fallback

You are now the administrator of your own Holocron Library.

️ Tech Stack: What Our Starship Is Made Of

Nuxt 3 — the elegant lightsaber
Redis Stack w/ RedisJSON — the hyperdrive core
Tailwind CSS — the robe of a modern Jedi developer
Google Fonts module — for typographic wizardry
Docker Compose — the Imperial Fleet
TypeScript + tsx runtime — because even smugglers need types

️ The Holocron of Configuration (Updated!)

Everything — fonts, colors, layout, excerpts, pagination — lives in:

config/blog.config.ts

New additions include:

headerTitle
headerTagline
Dynamic Tailwind theme generation
contentMaxWidth
Post excerpts config

Want your blog to have an imperial blue palette?
Done.

Want H1 in Englebert and H2 in Roboto Slab?
Absolutely.

Want your content width to feel like the inside of a Star Destroyer corridor?
Just set a new max-width.

API Endpoints (For the Droids Among Us)

Endpoint	Method	Description
`/api/posts`	GET	Retrieve posts. Supports `?page=N`
`/api/posts/:slug`	GET	Fetch single post + neighbors
`/api/tags`	GET	Retrieve all tags

Perfect for custom frontends, mobile apps, or your own droid companion.

Why Developers Love This

Because simplicity + power = developer bliss.

Because Redis is a piece of sci-fi technology disguised as a database.

Because Nuxt 3 is smooth as sliding through hyperspace.

Because Tailwind lets you customize everything without rewriting CSS.

Because Markdown posts feel like writing Jedi scrolls.

And because every post stored in Redis Sorted Sets is basically organized by the Force.

Author & Origin Story

Built by Luca Visciola — drummer, developer, and happy dad.

Inspired by Salvatore “antirez” Sanfilippo, whose creation Redis is widely believed to be running not on silicon… but on compressed fragments of a neutron star.

Final Transmission

If you're searching for a fast, flexible, sci-fi-friendly blog engine that handles posts faster than the Millennium Falcon outruns TIE Fighters, this boilerplate may be the one you’re looking for.

Clone it.
Customize it.
Seed it.
Deploy it.

GitHub Repository:
https://github.com/melasistema/redis-blog

And may the Force of Redis be with you. ❤️‍

Database Interaction: Connecting to Databases

Fri, 26 Jul 2024 10:22:07 +0000

In this article, we will address the key issue of database connections in software development. Large volumes of data are effectively stored and managed via databases, which form the foundation of many applications. Building scalable and reliable apps requires developers to have a solid understanding of database connections.
In software development, database connectivity is essential for maintaining data integrity, scalability, and system performance. Relational databases like MySQL and PostgreSQL offer structured storage and sophisticated querying, whereas NoSQL databases like MongoDB and Redis offer flexibility for unstructured data. Comprehending these distinctions is essential for developers to make well-informed decisions.

Selecting the Right Database

This tutorial offers information on different kinds of databases and things to think about.
Different Types of Databases:

Relational Databases: These databases arrange information into structured tables that have pre-established connections between them. PostgreSQL, SQL Server, and MySQL are a few examples.
NoSQL database: Large amounts of unstructured or semi-structured data are easily handled by NoSQL databases because they provide a more flexible data format. Redis, Cassandra, and MongoDB are a few examples.

Factors to Consider When Choosing a Database:

Data Structure and Requirements: NoSQL databases are better for unstructured or schema-less data, while relational databases are better for highly structured data that requires complex querying.
Scalability: Relational databases scale vertically by adding resources to a single server; NoSQL databases scale horizontally by sharing data across servers.
Performance: Relational databases perform better in ACID transactions, however NoSQL databases place a higher priority on low latency and high throughput.
Requirements for Consistency and Integrity: Strict consistency and transactional integrity are guaranteed for accurate data by relational databases. NoSQL databases may provide models for eventual consistency, putting high availability ahead of instant consistency.

from pymongo import MongoClient

client = MongoClient('mongodb://localhost:27017/')

db = client['my_database']

collection = db['my_collection']

collection.insert_one({'name': 'XYZ', 'age': 30, 'email': '*Emails are not allowed*'})

# Query documents

result = collection.find_one({'name': 'XYZ'})

print(result)

Tip: Prototype with different databases to assess performance, scalability, and suitability for your project before making a final decision.

Python's Common Database Connection Methods

There are numerous ways to connect to databases in Python, including:

1. Using built-in libraries:

Direct database connections without the need for additional dependencies are made possible by Python's built-in modules, such as sqlite3 for SQLite.

import sqlite3

conn = sqlite3.connect('example.db')

cursor = conn.cursor()

cursor.execute('SELECT * FROM table_name')

results = cursor.fetchall()

for x in results:

  print(x)

2. Third-party libraries:

External libraries provide additional functionality and optimizations for particular database systems, such as psycopg2 for PostgreSQL and mysql-connector-python for MySQL. These packages offer simple-to-use APIs for establishing database connections and running queries.

import mysql.connector

mydb= mysql.connector.connect(

    host='hostname',

    user='username',

    database='database_name'

)

mycursor = mydb.cursor()

mycursor.execute('SELECT * FROM table_name')

myresult = mycursor.fetchall()

for x in myresult:

  print(x)

3. ORM (Object-Relational Mapping) frameworks:

By abstracting database interactions, ORM frameworks such as SQLAlchemy save developers from writing raw SQL queries and enable them to work with Python objects. These frameworks offer strong capabilities for manipulating and managing databases.

from sqlalchemy import create_engine, MetaData, Table

engine = create_engine('postgresql://username:password@localhost/database_name')

metadata = MetaData()

table = Table('table_name', metadata, autoload=True, autoload_with=engine)

results = engine.execute(table.select()).fetchall()

Connecting to Relational Database

There are multiple processes involved in setting up connections to widely used relational databases such as SQLite, MySQL, and PostgreSQL. Here's a thorough explanation with examples to help clarify:

1. Setting up connections with each database:

A. SQLite:

Data is stored in a single file with the lightweight serverless database engine SQLite. With Python's built-in sqlite3 package, connecting to SQLite is simple.

import sqlite3

conn = sqlite3.connect('example.db')

B. MySQL:

The mysql-connector-python library can be used to create a connection for MySQL. Make sure pip install mysql-connector-python is installed on your computer.

import mysql.connector

conn = mysql.connector.connect(

    host='localhost',

    user='username',

    database='database_name'

)

C. PostgreSQL:

You can use the psycopg2 library to establish a connection to PostgreSQL. Install it (pip install psycopg2), please.

import psycopg2

conn = psycopg2.connect(

    host='localhost',

    user='username',

    password='password',

    database='database_name'

)

2. Handling connection errors and exceptions:

For your application to be robust, it is imperative that you handle exceptions and connection problems in a polite manner. Try-except blocks are a useful tool for handling and catching exceptions.

try:

    conn = psycopg2.connect(

        host='localhost',

        user='username',

        database='database_name'

    )

except psycopg2.Error as e:

    print(f"Error connecting to PostgreSQL: {e}")

3. Executing SQL queries and retrieving results:

Once connected, you may use cursor objects to run SQL queries and get the appropriate responses.

cursor = conn.cursor()

cursor.execute('SELECT * FROM table_name')

results = cursor.fetchall()

for row in results:

    print(row)

cursor.close()

Connecting to NoSQL Databases

In Python, connecting to well-known NoSQL databases such as MongoDB and Redis requires simple steps for database operations, error handling, and setup:

1. Setting up connections with each database:

A. MongoDB:

To create a connection for MongoDB, use the pymongo package.

from pymongo import MongoClient

client = MongoClient('mongodb://localhost:27017/')

B. Redis:

Redis can be accessed by using the redis library.

import redis

r = redis.Redis(host='localhost', port=6379, db=0)

2. Handling connection errors and exceptions:

Make sure to use try-except blocks to properly handle connection problems.

try:

    client = MongoClient('mongodb://localhost:27017/')

except Exception as e:

  print(f"Error connecting to MongoDB: {e}")

// Redis

try:

    r = redis.Redis(host='localhost', port=6379, db=0)

except Exception as e:

    print(f"Error connecting to Redis: {e}")

3. Executing database Operations:

Execute database operations like delete, update, and insert.

MongoDB:

db = client['my_database']

collection = db['my_collection']

collection.insert_one({'name': 'John', 'age': 30})

collection.update_one({'name': 'John'}, {'$set': {'age': 31}})

collection.delete_one({'name': 'John'})

Redis:

r.set('key', 'value')

value = r.get('key')

r.delete('key')

Error Handling

Error handling in database connections is crucial for program reliability and robustness, preventing crashes and enhancing user experience by addressing unforeseen database errors and connection issues.

Using try-except blocks: Try-except blocks can be used to catch and manage exceptions that are raised during database operations. This guarantees that your program can continue running even after encountering failures.

import psycopg2

try:

    conn = psycopg2.connect(

        host='localhost',

        user='username',

        database='database_name'

    )

except psycopg2.Error as e:

    print(f"Error connecting to PostgreSQL: {e}")

Handling specific error types: To improve error management precision, classify errors such as query problems or connection failures and provide customised error messages and actions.

import psycopg2

try:

    conn = psycopg2.connect(

        host='localhost',

        user='username',

        password='password',

        database='database_name'

    )

except psycopg2.OperationalError as e:

    print(f"Operational error connecting to PostgreSQL: {e}")

except psycopg2.DatabaseError as e:

    print(f"Database error connecting to PostgreSQL: {e}")

Implementing retry mechanisms: Retry techniques automatically retry database operations during temporary errors or network issues, enhancing application robustness, especially in unstable contexts.

import psycopg2

import time

max_retries = 3

retry_delay = 1  # seconds

for retry_count in range(max_retries):

    try:

        conn = psycopg2.connect(

            host='localhost',

            user='username',

            database='database_name'

        )

        break

    except psycopg2.Error as e:

        print(f"Error connecting to PostgreSQL: {e}")

        if retry_count < max_retries - 1:

            print(f"Retrying in {retry_delay} seconds...")

            time.sleep(retry_delay)

        else:

            print("Max retries exceeded, unable to connect.")

          break

Best Practices and Tips

By adhering to these best practices, Python programs may guarantee stable database connections:

To effectively manage database connections, make use of connection pooling.
In order to reduce overhead and boost efficiency, reuse connections.
To liberate resources, always properly close connections.
Handle connection credentials securely to avoid unwanted access.

from sqlalchemy import create_engine, Pool

# Create an engine with connection pooling

engine = create_engine('database://user:password@host:port/db_name', pool_size=5, max_overflow=10)

# Use the engine to connect to the database

connection = engine.connect()

# Perform database operations...

# Close the connection

connection.close()

FAQ
Q: Why is connection pooling important in database applications?
A: Connection pooling helps manage and reuse database connections, reducing the overhead of establishing new connections for each database operation. This improves performance and scalability, especially in applications with high traffic.

Conclusion

To sum up, this article highlights how important database connections are to software development. Comprehending the subtle differences between relational and NoSQL database types is essential for achieving the best possible project performance. Python provides flexible ways to provide error handling and make database connections. Through careful consideration of database selection and adherence to industry best practices, developers can construct resilient systems that effectively handle data and satisfy the requirements of contemporary software development.

Reference

For more reference:

PyMongo
PyMongo - MongoDB Python driver