Recent posts tagged reliability

Part 2 of 10: Reliable background jobs in Node.js, from zerohttps://blog.sandrodezerio.dev/. By Sandro Dezeriohttps://blog.sandrodezerio.dev/about/, senior full-stack engineer. The sign-up scenario is made up, the same one as in Part 1https://blog....

Timing tests look wonderfully objective. Start a stopwatch, call the component, stop the watch, and compare the result with a budget. The number is precise. The experiment may not be. That distinction matters in continuous integration, where many t...

Software often uses broad time windows for good reasons. A dashboard may need recent history, work that is actionable now, and the next scheduled item. Fetching all of that in one bounded query can be efficient and easy to reason about. The trouble ...

Idempotency is usually introduced with a simple promise: run the operation again and it will not create a duplicate. That promise is useful, but it can become dangerous when we apply it to an entire workflow instead of the specific durable step it p...

> Originally published at vibeagentmaking.comhttps://vibeagentmaking.com/blog/phoenix-payroll-2-6-billion-to-fix-57-million-to-replace/. A payroll is not a cost center. It is insurance against the one loss a state cannot pool, and Canada priced that...

A configured backup credential often remains unused when application logic only switches keys after a quota response, allowing a temporary service failure to skip an available recovery path. Quick solution: Configure your fallback wrapper to catch H...

Deterministic formatting normalization prevents expensive LLM generation retries when a model output fails structural validation due to minor whitespace variations. Quick solution: function normalizeQuickSolutionLabelcontent: string: string { re...

Crash reporting feels like a solved problem until your distribution platform says the obvious solution is not allowed. Quick solution: ~ts type ErrorRecord = { timestamp: string; level: "error" | "warning"; source: string; message: string; ...

> Originally published at vibeagentmaking.comhttps://vibeagentmaking.com/blog/southwest-skysolver-december-2022-threshold/. The same crew solver ran without incident at other airlines that week. That is what makes this a measurement rather than a we...

Background work is usually introduced to make a system more reliable. We persist an intent, let the request finish, and deliver the external side effect later. That protects the user-facing path from temporary provider failures and gives us room for ...

Some of the most misleading tests are not flaky, slow, or obviously incomplete. They are fast, deterministic, and green. They also pass for the wrong reason. That is especially dangerous in fault-tolerant code. A test may claim that a failed depend...

When an API returns an HTTP 422 response, client integrations often discard field-level details or dump entire response bodies into logs. Quick solution: async function readAndFormat422response: Response: Promise { const raw = await response.text...

A customer clicks Place order. Your API returns success. The order row is in the database. But the service that processes orders never sees an event. This is not a hypothetical edge case invented by Kafka. It is a failure window in a common Spring B...

When an unattended cron system encounters a total provider failure, engineers often reach for a manual kill switch. While effective at stopping runaway requests, the manual-only reset pattern introduces hidden operational costs. If an outage lasts tw...

A scheduled workflow can start successfully, finish with no eligible work, or commit an article that is not yet live. Counting all three as successes turns a reliability percentage into a trigger counter. Long-running and manually triggered execution...

When an application backend sleeps between requests or takes time to start, an Android app can mistake a temporary cold start for a permanent outage. The opposite failure is also common: a loading spinner retries forever and gives the user no clear w...

An event notification looks like a neat starting point for automation. A form is submitted, a record changes, or another system announces that a job is ready. The first design question is often which AI action should follow. A more useful question is...

A retry can duplicate a side effect even when the original command completed successfully. This is easy to miss in a method that does two ordinary things: save a record and send a message. Each operation is understandable on its own. The difficult p...

“Make it work offline” often turns into a storage task: serialize the last successful response, read it when the network fails, and move on. That is necessary, but it is not the hard part. The hard part is deciding what the stored response means ac...

Security integrations often fail in a surprisingly polite way. The library loads. Initialization succeeds. The function returns. Nothing crashes. Yet the protection we expected may not have participated at all. That distinction matters when an appli...