Coder Legion - Recent posts tagged api

An MCP server is not a new API. It is a thin layer over the one you already have

Fri, 30 Oct 2026 10:26:14 +0000

Somewhere in the last year, "do we need an MCP server?" joined the list of questions that arrive in a brief before anyone has said what the thing is for. It is usually framed as MCP vs API, as though the Model Context Protocol were a newer kind of API, something to migrate to. It is not. Open almost any MCP server and there is an ordinary API call at the bottom of it.

This is the developer-side version of a guide we wrote for the people who commission these projects. It skips the business case and looks at what goes over the wire, where the real work in a server sits, and the part of the design that the demos leave out.

What actually goes over the wire

MCP is JSON-RPC 2.0, carried over standard input and output when the server runs on the same machine as the host, or over Streamable HTTP when it runs somewhere else. The host is the application a person uses, such as Claude, ChatGPT, VS Code or Cursor. Inside it, a client holds a connection to one server, and the server is the piece you would build.

Discovery is the first difference from a normal API. Nobody reads documentation. The client asks the server what it can do, and the server answers with names, descriptions and JSON Schemas:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "resultType": "complete",
    "tools": [
      {
        "name": "search_orders",
        "description": "Find orders by customer email or order number. Returns at most 10 orders, newest first.",
        "inputSchema": {
          "type": "object",
          "properties": {
            "query": {
              "type": "string",
              "description": "A customer email address or an order number such as 10482"
            }
          },
          "required": ["query"]
        }
      }
    ]
  }
}

The model reads that list and decides for itself which tool fits what the user just asked. When it picks one, the client sends a call:

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "search_orders",
    "arguments": { "query": "*Emails are not allowed*" }
  }
}

The specification examples omit one detail, and so do these: every request also carries a _meta block with the protocol version, client information and client capabilities. That is new. The revision dated 28 July 2026 removed the opening handshake and protocol-level sessions, so each request now describes itself, much like an ordinary stateless API call. A lot of what was written about MCP in 2025, including most of the tutorials, describes the older, session-based version.

The bottom of every server is an API call

Follow that tools/call through a typical server and the chain is short. The server validates the arguments, turns them into an HTTPS request to the API the system already had, and wraps the response as a tool result. GitHub's official MCP server calls GitHub's API. The Blender and Hugging Face servers we use every day are thin layers over Blender's Python API and the Hugging Face Hub API.

That has a practical consequence for estimates. If a system has no API, it does not need an MCP server yet. It needs an API, and the server comes after it. If the API exists and is decent, the protocol side of the server is small. The official SDKs cover TypeScript, Python, C#, Go, Java and Kotlin among others, and a server that wraps three endpoints runs to a few hundred lines. A demo really is a day's work. Everything below is why production is not.

Design the tools for the reader who will use them

An endpoint is written for a programmer who reads it once, writes code against it and then calls it the same way forever. A tool is read by a model on every conversation, and the model decides on the spot whether this is the right one. A vaguely named endpoint costs a developer ten minutes. A vaguely described tool costs you every conversation in which the model picks the wrong one, and nobody notices, because the answer still sounds confident.

The common mistake is to mirror the API one endpoint per tool. Anthropic's own guidance on writing tools names this directly as an error: tools that "merely wrap existing software functionality or API endpoints". Its example is contacts. The mirrored version looks like this:

{
  "name": "list_contacts",
  "description": "List contacts.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "page": { "type": "integer" },
      "per_page": { "type": "integer" }
    }
  }
}

To find one person, the model has to page through everyone, reading every page into its context to check it. The version written for the model does the search on the server:

{
  "name": "search_contacts",
  "description": "Find contacts by name, email or company. Use this before creating a contact, to avoid duplicates.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": {
        "type": "string",
        "description": "Part of a name, an email address or a company name"
      }
    },
    "required": ["query"],
    "additionalProperties": false
  }
}

Fewer, higher-level tools, with descriptions that say when to use them as well as what they do. The schema descriptions matter as much as the tool description: "an order number such as 10482" saves a round trip in which the model guesses the format wrong.

There is also a cost that lands on the model's side rather than the server's. Every tool definition is loaded into context, on every conversation. Anthropic's engineers worked through an example in which tool definitions and intermediate results passing through the model came to 150,000 tokens, and cut it to about 2,000 by letting the model write code against the servers instead. That was one worked example, not a benchmark, but the direction holds. Every tool you expose is a cost on every conversation, so a few good tools beat a mirror of your whole API.

Errors are written for the model, too

MCP separates two kinds of failure. A protocol error, such as an unknown tool or a malformed request, comes back as a standard JSON-RPC error, and the model can rarely do anything with it. A tool execution error comes back as a normal result with isError: true, and the specification says clients should pass it to the model so it can correct itself:

{
  "jsonrpc": "2.0",
  "id": 4,
  "result": {
    "resultType": "complete",
    "content": [
      {
        "type": "text",
        "text": "No order 10428 found. Order numbers are 5 digits; did you mean 10482? Search by customer email if the number is uncertain."
      }
    ],
    "isError": true
  }
}

Compare that with the 404 Not Found the underlying API returned. The API's error is fine for code that checks a status. The model needs a sentence that tells it what to try next. Writing those sentences is a surprising share of the work, and it is the share that makes the difference between a server that recovers and one that gives up and apologises to the user.

State, now that there are no sessions

With sessions gone from the protocol, a server cannot keep implicit per-connection state. The specification's guidance for anything that has to span calls, such as a basket, an open browser context or a transaction, is an explicit handle: a creation tool returns an ID, and later calls take it as an argument. The model carries it forward.

Two parts of that guidance deserve to be read twice. For an authenticated server, a handle is a name, not a capability: check the caller's authorisation against it on every call, because nothing stops a model, or someone steering it, from passing a handle that belongs to somebody else. And state the handle's lifetime in the creation tool's description, so the model knows a basket expires before it plans around one.

The part the demos skip

An MCP server gives a model hands, and a model takes instructions from anything it reads. A year of incidents shows what that means in practice. In April 2025 Invariant Labs demonstrated tool poisoning, instructions hidden in a tool's description that the user never sees and the model obeys. In May 2025 the same team showed a crafted issue in a public GitHub repository making an agent with the official GitHub server leak data from private ones, and were clear that the fault was the architecture, not the server's code. In June 2025 Asana took its MCP server offline for almost two weeks after a logic flaw exposed some data to other organisations. In July 2025 a flaw in mcp-remote, a widely used connector, allowed remote code execution on the machine of anyone who connected it to an untrusted server. In September 2025 a malicious npm package posing as an email server blind-copied every message it sent to its author.

Simon Willison named the pattern behind most of these the lethal trifecta: access to private data, exposure to untrusted content and a way to send data out. Any two can be managed. With all three, anyone who can put text in front of the model, in a support ticket, a product review or a web page, can try to make it send your data somewhere.

The specification helps where it can. It requires servers to validate inputs, enforce access control, rate limit tool calls and sanitise outputs. For remote servers it recommends OAuth 2.1, requires that every token was issued for your server, and forbids passing the client's token through to the API behind it. It also tells clients to treat tool annotations, the hints a server gives about how a tool behaves, as untrusted unless the server is. None of that can make a server safe if the server is allowed to do too much. Give an MCP server the narrowest permissions that do the job, because the most hostile text it ever reads gets the same permissions you did.

In code, that comes down to a handful of decisions. Read-only tools by default, and write tools behind an explicit confirmation. One server per trust boundary, rather than one that can do everything. Dependencies pinned and reviewed, rather than whatever the registry serves today. Every call logged with who asked and what happened. And never a tool that runs arbitrary SQL or shell commands against production: a tool that accepts free-form SQL hands every rule you have to whoever manages to talk the model into using it.

When not to build one

The protocol earns its place when whatever calls your system is a model somebody else chose: customers who work inside AI assistants and want your product there, or staff who want to ask internal systems questions from the assistant they already use. It does not earn it for a fixed process with known steps. "When an order arrives, create the invoice and email it" wants deterministic automation calling the API directly; a model deciding each step adds cost and a new way to fail, and gains nothing. A single integration between two of your own systems wants a plain API call.

If a server is the right answer, build the smallest one that answers the most common question, read-only, put it in front of real users and read the logs. The tool calls show what the model actually needs faster than any design document.

The original

This is the developer-side version of a guide on our site, written for the people deciding whether a project needs an MCP server at all. The original has the full side-by-side table of MCP and a REST or GraphQL API, the MCP vs function calling question, and answers to what clients ask about it: whether MCP only works with Claude, whether a website needs one, and how long a production server takes.

MCP vs API: what an MCP server actually adds

This copy is a snapshot from the day it was posted, and it will not be updated here. The original on our site is the version we keep current: when a study publishes new figures, a price changes or a tool is renamed, that is where the correction lands. If you want the up-to-date version, follow the link above.

Also available in German and Romanian.

Sources

The wire format, the _meta requirement, the two kinds of error, the handle pattern for state and the server and client security requirements are from the Model Context Protocol Tools specification, revision 2026-07-28, and the removal of the handshake and sessions is in that revision's changelog. The OAuth requirements and the ban on token passthrough are from the Authorization chapter. The advice against mirroring endpoints and the contacts example are from Anthropic's Writing effective tools for agents, and the token figures from Code execution with MCP. The incidents are from Invariant Labs on tool poisoning and the GitHub exploit, BleepingComputer on Asana, JFrog on mcp-remote and The Hacker News on the malicious npm package. The lethal trifecta is Simon Willison's. The order-search examples are ours.

Which went wrong first on the servers you have shipped: the model picking the wrong tool, or the right tool being allowed to do too much?

One API for multiple AI models, with web studios for images, video, and music.

Thu, 29 Oct 2026 03:03:06 +0000

[AI SaaS Go][1] is a multi-model platform for developers and creators. Access GPT, Claude, Gemini, DeepSeek, and other text models, plus image, video, and music generation through an API or web studios. Prepaid credits and usage-based billing let you choose models to suit your workflow and budget.

[1]: https://aisaasgo.org

Cross-posting to dev.to without giving away your SEO.

Tue, 27 Oct 2026 21:03:41 +0000

Every article I write lives on my own site first. A copy goes up on dev.to a day or two later, and that copy reliably gets more readers than the original does. One piece has 106 views over there. My own analytics can't honestly tell me how many humans read it here, which is a separate mess I wrote about last week.

So I am deliberately publishing my best work somewhere with more traffic than my site, under a domain I don't own. The thing that makes this a syndication strategy instead of a slow act of self-harm is one field in the dev.to API:

const canonicalDi = (slug) => `https://marcobellingeri.dev/en/writing/${slug}`;

That is the whole trick, and it is also the least interesting part. What took me longer to learn is that canonical-first is not a field you set. It is an order you have to keep.

What the tag actually buys you

When two URLs serve the same article, a search engine has to pick one to rank and one to treat as a duplicate. Left alone it will usually pick the one with more authority, which on any given week is dev.to and not you. rel="canonical" is how the copy points back and says the other one is the original.

dev.to honours this properly. Set canonical_url on a post and it renders the tag in the head, and it also shows a small line to readers saying the piece was originally published elsewhere, with a link. Some people click it. That link is worth more to me than the view count.

The part worth being precise about: the canonical is a hint, not a directive. Google treats it as one signal among several, and one of the others is which URL it saw first. That is where the ordering comes in.

The house has to be older than the mirror

If dev.to publishes on Monday and your site publishes the same piece on Thursday, you have handed a crawler a strong reason to believe dev.to is the original and you are the copy, no matter what your tag says. You are asking it to disregard the evidence of its own crawl on the strength of an annotation.

So the rule my pipeline enforces is that the canonical URL must resolve, with the right date on it, before the mirror goes live. Not at the same time. Before.

This is enforced by refusing to publish. The workflow that talks to dev.to on every merge is only allowed to create drafts:

on:
  push:
    branches: [main]
    paths: ["astro-project/src/content/writing/en/**"]

It runs node engine/devto.mjs <slug> with no --publish flag, ever. Merging an article deploys it to my site and leaves an unpublished draft sitting on dev.to. Nothing is live over there until a second, separate thing happens.

That second thing is a daily cron at 07:00 UTC that reads the date in each article's frontmatter and flips the ones whose day has arrived:

devto: niente in uscita oggi (2026-08-19)
DOMANI=

Most mornings it has nothing to do and says so. The same run also opens a GitHub issue listing anything going out tomorrow, which gives me a real 24 hour window to move a date or delete a file if a piece has aged badly since I wrote it. Nobody has to approve anything for publication to happen. Silence publishes. The one human decision is the merge, and it already happened.

The date that lied

I would like to report that this held. In July I merged an article about source licences with date: 2026-08-18 in its frontmatter, scheduling it for a Tuesday. In August I noticed it had been public on dev.to since the 11th.

Seven days before its own publication date.

It wasn't the draft workflow, which never passes --publish. It wasn't the cron, which skips future dates. It was me, publishing that one by hand from the dev.to UI weeks earlier and forgetting. The automation was innocent and had no way to notice.

What made it a real problem rather than an embarrassing one is what my own site was serving in the meantime. The article page builds its JSON-LD from that same frontmatter field:

"datePublished": "2026-08-18"

So for seven days the canonical URL told every crawler that the original was published on the 18th, while the copy it pointed at was demonstrably live on the 11th. The one signal I control was actively arguing that my site came second.

The fix was a one line change to a date, which is the sort of fix that makes you want to check what class of thing it belongs to. It belongs to this one: the frontmatter date is not decoration. It is the claim my canonical makes about which came first, and anything that publishes outside the pipeline can make that claim false without touching the repository.

I also checked what would have happened on the 18th if I hadn't noticed. Nothing at all, as it turns out, because the cron skips pieces that are already live. The bug would have quietly persisted rather than announcing itself. Those are the ones worth going looking for.

Re-running has to be free

The other property that matters is that pushing the same article twice must not create a second post. dev.to has no upsert, so the script builds one by asking what already exists and matching on the canonical URL:

const canonicalPubblicati = (await publishedArticles()).map((a) => a.canonical_url);

Matching on canonical rather than on title means an edited headline updates the existing post instead of forking a duplicate, and duplicates are exactly what the canonical tag exists to prevent. A syndication script that can create two live copies of one article is doing the opposite of its job.

Editing a published piece and re-merging updates the draft body and leaves its published state alone. I have re-run the whole thing more times than I would like to admit while debugging, and it has never republished anything.

One more thing, since it belongs to the same function. The slug comes from a filename, gets interpolated into a URL, and gets used to open a file, so it is validated before either:

if (!slug || !/^[a-z0-9-]+$/.test(slug)) { /* exit 1 */ }

Filenames in my own repository are not attacker controlled, and I validate them anyway. The check costs one line, and the day someone else can open a pull request against this repo is the day I would otherwise have to remember to add it.

What I would check on your setup

If you already cross-post, three things are worth ten minutes.

Open the mirrored copy and view source. Confirm the canonical tag is actually there, because plenty of platforms accept the field and then drop the tag on some templates.

Then compare the two publication dates, not the two URLs. If any mirrored copy is older than the original it points at, your canonical is arguing against you right now, and no amount of correct configuration fixes an ordering you got backwards.

Then find whatever can publish outside your pipeline. A CMS button, a scheduled post, a colleague with access, you three weeks ago. Mine was me, and the automation I had spent a weekend making careful had no idea.

The tag is easy. Staying first is the work.

Protect Human Edits: Conditional Saves for AI-Assisted Listing Drafts

Tue, 27 Oct 2026 10:18:08 +0000

Original AI-generated concept illustration. Fictional interfaces and property details, not an actual product screenshot.

Imagine a fictional property-listing editor used by two colleagues. Mira and Dev open the same draft. Mira corrects “two bedrooms” to “three bedrooms” after checking the property details, advancing revision 7 to revision 8. Dev asks an AI assistant to improve the description, reviews its wording, and clicks Save.

Dev’s browser still holds the old bedroom count. If the server accepts the entire draft unconditionally, a polished description can quietly erase Mira’s correction.

The useful question at save time is: Is the version this draft was based on still current? A conditional save makes that question part of the server’s write operation, rather than a warning the interface hopes everyone notices.

Keep the baseline with the draft

When the editor loads a listing, retain three things together: the saved content, its ETag, and a separate working copy. Start AI generation from that captured baseline. Do not fetch a newer tag later and attach it to output generated from older content.

A server might return this illustrative header excerpt, not a complete HTTP message:

ETag: "listing-42-r7"

The corresponding save includes:

If-Match: "listing-42-r7"

If-Match uses strong comparison against the selected representation. If the condition is false, the server must not perform the requested change; the usual response is 412 Precondition Failed. RFC 9110, section 13.1.1

Clients should treat tags as opaque values, preserving their quotes. A weak tag such as W/"listing-42-r7" never satisfies this comparison. Removing W/ does not turn it into a trustworthy strong validator. Also, If-Match: * tests existence, not whether the editor has the latest revision. MDN’s If-Match reference

For this proposed design, use one stable JSON representation for editing. Issue a new revision whenever its saved representation changes, including changes made by imports or background jobs. If serialization or representation variants can change independently, account for them in validator generation too. A revision number is useful only when the server maintains that invariant.

Make comparison and writing one operation

A separate “check version” request cannot protect the later save. Another writer can commit in the gap. Even checking inside the save handler is insufficient if the database permits a competing update between comparison and persistence.

Here is untested, framework-neutral pseudocode, not a deployable handler. This endpoint’s contract requires an explicit revision-tag list rather than a wildcard; its parser enforces that contract and rejects malformed input. Weak tags remain weak and cannot match.

save_listing(request):
    authorize_and_validate_request(request)
    if request.IfMatch is missing:
        return 428 with retry_instructions, CacheControl="no-store"
    candidates = parse_revision_tag_list(request.IfMatch)

    begin_transaction()
    try:
        listing = select_listing_for_update(request.listing_id)
        if listing is absent:
            rollback()
            return 404

        current = strong_etag_for_edit_representation(listing)
        if not any_strong_match(candidates, current):
            rollback()
            return 412

        replace_saved_draft(listing, request.draft)
        advance_representation_revision(listing)
        persist(listing)
        saved = snapshot_edit_representation(listing)
        commit()
    catch error:
        rollback_if_active()
        raise error

    return 200 with body=saved  # no response validator shown

The lock must cover reading the current version, comparing it, and writing the replacement. PostgreSQL row locks normally remain held until transaction end, providing one possible building block. PostgreSQL row-level locking documentation

Every writer must preserve the same versioning rules. Keep model generation and human review outside the transaction; a database lock should not wait for someone to finish editing.

The 428 response is an optional API policy, not a universal requirement for every server. It means this server requires a conditional request. Explain how to fetch a baseline and resubmit; responses with this status must not be cached. RFC 6585, section 3

Original AI-generated concept illustration with fictional people, interface and property details. An atomic version check protects the newer edit; the article explains the already-applied exception for a lost response.

Make a conflict recoverable

When Dev receives 412, leave the working copy intact. Fetch the latest listing into a separate buffer and present a three-way comparison:

Base: the content Dev originally loaded.
Mine: Dev’s reviewed AI-assisted draft.
Current: the latest saved content, including Mira’s correction.

Compare each side against the base. Distinct field changes may be straightforward to combine, but rewritten paragraphs can overlap semantically even when their text differs. Highlight the bedroom discrepancy for review instead of asking another model to decide which factual claim wins.

After reconciliation, associate the reviewed result with the newly fetched baseline and its tag. Save conditionally again. If another edit arrives during review, repeat the comparison without discarding anyone’s work. Avoid an automatic “retry with latest tag” button: it would authorize the stale replacement without resolving the conflict.

Original AI-generated concept illustration. Fictional collaborators compare the base, working draft and current saved version; not an actual product screenshot.

Treat a missing response as uncertainty

Suppose Dev’s save commits, but its response disappears. Retrying with the original tag can produce 412 because the first save advanced the revision. That does not prove the first attempt failed.

HTTP permits a success response when the server can determine that the requested change was already applied. The simpler handler above deliberately does not implement that optional recovery path. RFC 9110, section 13.1.1

Retain the draft and fetch current state before deciding what happened. If duplicate-operation recognition is needed, design a durable operation record and its retention rules explicitly; do not guess from a timeout.

Six tests before shipping

Ordinary save: a current tag succeeds, and a subsequent fetch reflects the saved draft and updated validator.
Stale editor: Mira saves first; Dev’s stale request gets 412 without overwriting her correction.
Concurrent saves: release two changed drafts together using the same tag; only one replacement commits.
Validator contract: missing tags receive the documented 428; weak tags cannot match; wildcards cannot bypass revision protection.
Conflict recovery: preserve the working copy, compare all three versions, and reject another stale reconciliation if a third change intervenes.
Lost response: commit a save, drop its response, and retry. Verify that the client reconciles uncertainty rather than claiming the original save failed.

AI-assisted editing can make drafting faster. Conditional saves keep that speed from silently replacing a colleague’s more recent work.

Your 'Pay Now' Button Can Charge Twice. Idempotency Keys Are the Fix Nobody Adds Until It Happens

Sun, 25 Oct 2026 11:09:21 +0000

A customer clicks "Pay now". Nothing happens for three seconds, so they click again. Or their phone loses signal for a moment and the app quietly retries the request. Or your load balancer times out and sends the request again.

Your server received the same order twice, and it did exactly what you told it to do: it created two orders.

This is not a rare edge case. It is the default behavior of every POST endpoint you have ever written.

The Endpoint That Looks Fine

app.post('/api/orders', requireAuth, async (req, res) => {
  const order = await createOrder(req.user.id, req.body);
  await chargeCard(order);
  res.status(201).json({ orderId: order.id });
});

Nothing is wrong with this code in isolation. The problem is what it cannot know: whether this request is new, or a repeat of one that already succeeded.

Why This Is Hard to Catch

You test with one request at a time. The bug needs two requests with the same intent, close together.
Timeouts are ambiguous. When a client times out, it does not know if the server never got the request, or did the work and the response got lost. Retrying is the only safe-looking choice, and it is the one that duplicates.
Disabling the button is not enough. It helps with double clicks but not with network retries, mobile apps, or another service calling your API.

The Idea: Same Key, Same Result

An idempotent operation is one where doing it twice has the same effect as doing it once. POST is not idempotent by default, but you can make it behave that way:

The client generates a unique idempotency key for each intent and sends it in a header.
The server remembers the key together with the result.
If the same key arrives again, the server returns the saved result and does not do the work again.

Payment providers such as Stripe use exactly this pattern with an Idempotency-Key header.

A Working Version With Express and MongoDB

First, a unique index, plus an expiry so old keys clean themselves up:

await keys.createIndex({ userId: 1, key: 1 }, { unique: true });
await keys.createIndex({ createdAt: 1 }, { expireAfterSeconds: 86400 });

Then the endpoint:

app.post('/api/orders', requireAuth, async (req, res) => {
  const key = req.get('Idempotency-Key');
  if (!key) return res.status(400).json({ error: 'Idempotency-Key header required' });

  const userId = req.user.id;
  const bodyHash = hash(stableStringify(req.body));

  // 1. Try to claim the key. Only one request can win this insert.
  try {
    await keys.insertOne({ userId, key, bodyHash, status: 'processing', createdAt: new Date() });
  } catch (err) {
    if (err.code !== 11000) throw err; // 11000 = duplicate key

    const saved = await keys.findOne({ userId, key });
    if (saved.bodyHash !== bodyHash) {
      return res.status(422).json({ error: 'Key was used with a different request' });
    }
    if (saved.status === 'processing') {
      return res.status(409).json({ error: 'Request is still being processed' });
    }
    return res.status(saved.statusCode).json(saved.response); // replay
  }

  // 2. We own the key. Do the work once.
  try {
    const order = await createOrder(userId, req.body);
    await chargeCard(order);

    const response = { orderId: order.id };
    await keys.updateOne(
      { userId, key },
      { $set: { status: 'done', statusCode: 201, response } }
    );
    return res.status(201).json(response);
  } catch (err) {
    await keys.deleteOne({ userId, key }); // let the client retry
    throw err;
  }
});

The Race Condition Most Examples Miss

A common first attempt is:

const existing = await keys.findOne({ userId, key });
if (existing) return res.json(existing.response);
// ... do the work ...
await keys.insertOne({ userId, key, response });

Two simultaneous requests both run findOne, both see nothing, and both do the work. You added a check and still got a duplicate.

The fix is in the version above: claim the key first with an insert that can only succeed once, and let the unique index decide the winner. The database settles the race, not your code.

Four Details That Matter

1. Scope keys per user. Two users who happen to generate the same key must not see each other's responses. That is why the index is on { userId, key }.

2. Compare the request body. If the same key arrives with a different body, that is a bug or an attack, not a retry. Return an error instead of replaying a response that does not match the request. Use a stable stringify (sorted keys) for the hash, or two identical bodies with different property order will hash differently.

3. Generate the key per intent, not per attempt. On the client, create the key when the user starts the action and reuse it for every retry of that action. A new key per retry brings the duplicate right back.

const key = crypto.randomUUID(); // once, when the checkout starts
await fetch('/api/orders', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Idempotency-Key': key },
  body: JSON.stringify(cart),
});

4. Pass the key on to your payment provider too. If your server crashes after charging the card but before saving the result, your own table cannot help. A key sent to the provider protects that gap, and the expiry index clears any stuck processing record.

The Rule

Any request that creates something or moves money must be safe to send twice. Do not rely on disabled buttons or "the network is reliable". Give the client a key, claim it atomically on the server, and replay the saved result on a repeat.

Quick audit: list your POST endpoints that create orders, payments, invoices, emails or bookings. For each, ask what happens if the request arrives twice, a second apart. If the answer is "two of them", you have found your next ticket.

If you work with MongoDB, my MongoDB and Mongoose patterns guide has more on using the database to enforce rules, and you can browse my Next.js templates on pixelanas.com.

Get the templates: https://pixelanas.gumroad.com

Have you ever shipped a double-charge or a duplicate-order bug? Tell me how you found out, in the comments.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

🚨⚖️ THE AI TRADING COURTROOM IS OPEN. AND JEV.AI IS TAKING THE JURY SEAT. 🔥

Wed, 21 Oct 2026 22:21:35 +0000

🚨⚖️ THE AI TRADING COURTROOM IS OPEN. AND JEV.AI IS TAKING THE JURY SEAT. 🔥

WHAT IF EVERY AI TRADING MODEL HAD TO DEFEND ITS PREDICTIONS IN COURT?

No manipulated win rates.
No cherry-picked screenshots.
No imaginary profits.
No hiding the losses.

💥 INTRODUCING SCRIPTMASTERLABS FORECAST COURT

A LIVE, PUBLIC, MULTI-TICKER EVIDENCE ARENA — with JEV.AI as the AI JURY we're bringing to the bench.

🧠 THE MODEL MAKES THE CLAIM.

📊 THE EVIDENCE LEDGER PRESENTS THE FACTS.

⚖️ JEV.AI — THE JURY — WILL CHALLENGE THE CLAIMS AND EXAMINE THE EVIDENCE.

🏛️ THE COURT PUBLISHES THE RESULTS.

And we're not afraid to put our own technology on trial.

🚨 EXHIBIT A: THE $GME EXPERIMENT

Our Fractal Trading OS recorded:

❌ AI model: 50% directional accuracy
✅ Simple bullish baseline: 77.8%
📊 36 verified five-day outcomes

WE PUBLISHED THE UNDERPERFORMANCE.

Because a system that admits when it's wrong is more valuable than one that pretends it's always right.

🔥 NOW ENTERING THE ARENA:

$GME 🆚 $AMC 🆚 $IWM 🆚 $SPY 🆚 $QQQ 🆚 $NVDA 🆚 $TSLA

✅ Predictions frozen before future results
✅ Real market data
✅ Public performance scoreboards
✅ AI-versus-baseline comparisons
✅ Auditable evidence receipts
✅ Transparent wins AND losses
✅ JEV.AI jury integration on the roadmap

🤖 AI DEVELOPERS: Would your algorithm survive independent judgment?

📈 TRADERS: What if you could see exactly where an AI model fails?

💻 QUANT BUILDERS: Ready to put your own strategy on trial?

🌎 WELCOME TO THE ERA OF ACCOUNTABLE AI.

🔓 ENTER THE FULL LIVE FORECAST COURT — FREE:

👇👇👇

https://sml-fractal-trading-os-v2.onrender.com/forecast-arena.html

🌐 https://scriptmasterlabs.com

SCRIPTMASTERLABS × JEV.AI

⚖️ EVERY PREDICTION MAKES A CLAIM.
🔥 EVERY CLAIM FACES THE EVIDENCE.
🏛️ THE JURY WILL HAVE ITS SAY.

@TradingView @Stocktwits @Coinbase

ForecastCourt #JevAI #ScriptMasterLabs #AITrading #ArtificialIntelligence #AIAgents #AgenticAI #TradingAI #StockMarket #WallStreet #QuantTrading #MachineLearning #FinTech #Stocktwits #TradingCommunity #AlgorithmicTrading #GME #AMC #IWM #SPY #QQQ #NVDA #TSLA #BuildInPublic #AIInnovation #FinancialTechnology #EvidenceBasedAI #FutureOfTrading

Research and educational technology. JEV.AI jury integration is planned; existing live verdicts are evidence-based rule evaluations. No automated trading or guaranteed returns.

When an AI Image Job Times Out, Should Retry Create a New Job?

Wed, 21 Oct 2026 10:37:40 +0000

A photographer submits a batch of property photos for AI enhancement. The browser times out before it receives a job ID. Clicking Retry could recover the existing job, or create another paid batch while the first is still running.

The timeout only tells the client it did not receive a usable response. It does not prove the server did nothing.

Stripe’s idempotent-request documentation provides a concrete example of reusing an operation key to avoid duplicate work after a connection error. An image-processing service would need its own explicit contract; a payment API’s behavior should not be assumed to apply automatically.

For a proposed image-job API, I would separate:

Retry this submission: same operation key and unchanged inputs, recover its status or result.
Generate another version: new operation, with any credit cost visible before submission.
Change the inputs: explicitly create a new request rather than silently reusing the old key.

The difficult case is an unknown outcome: no response, no job ID in the browser, and credits possibly reserved. How would your API let the user recover that job safely? Would you require the client to save an operation ID before submitting, and how would you handle key expiry?

This is a design scenario, not a claim that a named platform currently double-charges. Disclosure: I work with Pixel Perfects Solutions LLC / EstateAI.

Should an AI Listing Draft Be Allowed to Overwrite Newer Agent Edits?

Tue, 20 Oct 2026 10:49:18 +0000

Imagine a property agent starts an AI rewrite of a listing description. While it runs, a colleague corrects a factual detail in the same listing. The AI result then arrives, based on the older version.

What should the application do with that result?

A useful distinction is finishing generation versus accepting a change. The generated draft can be retained without replacing the current description.

HTTP already has a building block for this problem. RFC 9110, section 13.1.1 describes If-Match as a way to guard against accidental overwrites when clients update the same resource. With a supporting API, a client sends the strong ETag it read; a failed precondition can produce 412 Precondition Failed.

For a property-content editor, a possible design is:

Record the listing revision used to generate the draft.
Save the AI output as a separate proposal.
Compare that base revision with the current listing when someone applies it.
If they differ, show the original, current, and proposed text for review.

The difficult part is the user experience. A changed punctuation mark and a corrected room count should not necessarily receive the same treatment. A version check catches a conflict; it does not decide whether the new wording is factually safe.

Would you block every stale AI draft, allow field-level merging, or always require a review screen? What would make your choice change?

This is a hypothetical design question, not a claim about a particular product's implementation.

Timeouts, retries and streaming with the OpenAI Python SDK: the defaults that surprised me

Mon, 19 Oct 2026 09:14:23 +0000

I run the same code against several OpenAI-compatible endpoints: hosted ones, a gateway, and a local server. Three SDK defaults caused most of my production issues.

1. The default read timeout is 10 minutes. openai.DEFAULT_TIMEOUT is Timeout(connect=5.0, read=600, write=600, pool=600). A hung upstream keeps a worker busy for up to 10 minutes. Set it explicitly:

import os
from openai import OpenAI, Timeout

client = OpenAI(
    base_url=os.environ.get("LLM_BASE_URL", "https://api.china-aiapi.com/v1"),
    api_key=os.environ["LLM_API_KEY"],
    timeout=Timeout(60.0, connect=5.0),  # 60 s read/write/pool, 5 s connect
    max_retries=2,                       # the default; shown so it's visible
)

2. Retries multiply. The SDK retries on connection errors, 408, 409, 429 and 5xx, with backoff, and honours Retry-After. If your gateway or proxy also retries, one slow request can turn into 3 × N upstream attempts. Pick one layer to own retries. If the server side already does it, use max_retries=0 in the client, or set it per call:

quick = client.with_options(max_retries=0, timeout=20.0)

3. Streams are only retried before the first byte. Once tokens are flowing, a dropped connection raises in the middle of your loop, and replaying the request would duplicate output the user has already seen. Handle that case yourself:

from openai import APIConnectionError, APITimeoutError

def stream_answer(prompt: str) -> str:
    parts = []
    try:
        stream = client.chat.completions.create(
            model="MODEL_ID",
            messages=[{"role": "user", "content": prompt}],
            stream=True,
            stream_options={"include_usage": True},
        )
        for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                parts.append(chunk.choices[0].delta.content)
                print(chunk.choices[0].delta.content, end="", flush=True)
            if chunk.usage:  # final chunk: choices is empty, usage is set
                print(f"\n[{chunk.usage.total_tokens} tokens]")
    except (APIConnectionError, APITimeoutError):
        if parts:
            parts.append(" [connection lost, answer incomplete]")
        else:
            raise  # nothing shown yet: safe to retry at a higher level
    return "".join(parts)

Note the if chunk.choices check. With include_usage, the last chunk has an empty choices list, and chunk.choices[0] throws IndexError at the very end of an otherwise good answer.

I'd like to hear how others split retries between client and gateway.

Disclosure: I work on the APIGOTO team; APIGOTO is independent of OpenAI, Anthropic and Google. The default base URL above is ours (www.china-aiapi.com), but everything here applies to any OpenAI-compatible endpoint.

Automated Lead Enrichment Workflow: The Complete Guide

Fri, 16 Oct 2026 06:17:20 +0000

Writing

Automated Lead Enrichment Workflow: From Raw Leads to Reliable CRM Data
A new lead enters your CRM with just a name, an email address, and a company. Your sales team still needs to find the person's role, identify the company, validate the available information, and update the record.
Doing this manually doesn't scale. But automating the process without proper validation can create duplicate contacts, incorrect company matches, and unreliable CRM data.
The real challenge isn't just enriching leads—it's building a workflow you can trust.
I published a practical guide at Causly Lab covering how to build an automated lead enrichment workflow from beginning to end.
Here's what you'll learn:

Workflow architecture: Connect lead sources, enrichment APIs, validation steps, and your CRM.
Implementation approaches: Compare no-code automation, self-hosted n8n, and custom-coded pipelines.
API integration: Understand provider requests, data mapping, and normalized lead records.
Data quality: Handle missing fields, conflicting values, duplicate records, and uncertain matches.
Reliability: Design retries, rate-limit handling, and recovery paths for failed requests.
Production readiness: Consider security, data provenance, testing, monitoring, and cost per successfully enriched lead.
One important principle: a successful API response doesn't automatically mean you've found the correct person or company. Your workflow needs explicit matching and field-merging rules.
If you're building CRM integrations, sales automation, or lead-generation systems, this guide can help you choose an implementation approach and understand the engineering decisions involved.
Read the complete technical guide:
https://www.causly.in/lab/automated-lead-enrichment-workflow
I'd also be interested to hear from developers: how do you handle conflicting enrichment data and duplicate records in your own workflows?

APIKO AI API

Thu, 15 Oct 2026 02:55:29 +0000

APIKO AI API is an infrastructure-grade AI API gateway that lets developers call text, image and video models from leading providers through a single OpenAI-compatible endpoint.

Instead of managing separate accounts and keys for OpenAI, Anthropic, Google, DeepSeek, Zhipu (GLM), Alibaba (Qwen), Moonshot AI (Kimi), ByteDance and xAI, you create one APIKO key, point your OpenAI SDK at https://api.apiko.app/v1, and switch models by changing the model id. Anything that speaks the OpenAI API works: the official SDKs, LangChain, LlamaIndex or plain HTTP.

Key features

One OpenAI-compatible endpoint for text, image and video models
Models from OpenAI, Anthropic, Google, DeepSeek, Zhipu, Alibaba, Moonshot AI, ByteDance and xAI
Live pricing and availability for every model, measured from real calls and probes
Automatic routing and failover; failed upstream calls are refunded automatically
Pay as you go in USD from $5, one balance for all models, never expires, no subscription
Works with official OpenAI SDKs, LangChain, LlamaIndex and plain HTTP

Website: APIKO AI API · Docs: https://apiko.app/docs

Discussion: What does it take to make an MCP server production-ready?

Wed, 14 Oct 2026 14:33:00 +0000

A local MCP server can be straightforward to run. Production deployments introduce different requirements when agents interact with remote APIs, databases, and execution environments.

Our new SEOSiri guide examines several important areas:

Scoped authentication and authorization

Secure SSE-based remote communication

Stateless service design

Response pruning and pagination for token efficiency

Sandboxed execution for dynamic code

The central engineering question: how do you give an AI agent useful capabilities without granting unnecessary access or creating avoidable operational risk?

Read the technical breakdown:
https://www.seosiri.com/2026/10/building-mcp-servers-production-security.html

For developers working with MCP, AI infrastructure, and DevSecOps.

Debug HTTP 422 API Validation Errors Safely

Wed, 14 Oct 2026 03:03:25 +0000

When an API returns an HTTP 422 response, client integrations often discard field-level details or dump entire response bodies into logs.

Quick solution:

async function readAndFormat422(response: Response): Promise<string> {
  const raw = await response.text();
  try {
    const body = JSON.parse(raw);
    const errors = body.errors;
    if (errors && typeof errors === "object") {
      return Object.entries(errors)
        .slice(0, 10)
        .map(([field, val]) => field + ": " + String(val))
        .join("; ");
    }
    return body.message || "Unprocessable content";
  } catch {
    return raw.slice(0, 200);
  }
}

HTTP 422 is one of those status codes that is specific enough to sound useful and vague enough to waste an hour. The server understood the request content type and syntax, but could not process the instructions in the payload.

What HTTP 422 actually tells you

RFC 9110 defines 422 Unprocessable Content as a case where the server understands the content type and the syntax is correct, but cannot process the contained instructions. MDN’s 422 reference summarizes the status as well. Because providers return inconsistent error-body schemas, your client must handle varied shapes safely without assuming a universal structure.

The anti-patterns of error handling

Catching non-2xx responses by keeping only the status code strips out all valuable context, turning detailed validation failures into generic messages. Conversely, logging the raw response text directly often leaks user input, tokens, or unexpected internal details into your observability stack.

Redacting and bounding output

Always bound character lengths and field counts to prevent memory bloat or log floods, and sanitize sensitive patterns like bearer tokens before persistence. For related architectural workflows, consult Designing a Reliable Serverless AI Publishing Workflow and Building a Reliable Batch Publishing Workflow.

Verification checklist

Read error bodies exactly once as text before parsing.
Enforce structural limits on extracted error fields.
Sanitize output strings for authorization tokens.
Confirm 422 responses do not trigger infinite retry loops.

What Building One API for Research, Media, and Editable Files Taught Me About Reliable AI Products

Wed, 07 Oct 2026 18:48:05 +0000

I started 3Stone AI because the work I wanted to do rarely ended in one format. Research might need to become a brief, a presentation, or a spreadsheet. An image might become a video. A plan might need to become an interactive tool. I wanted research, media generation, editable Office files, and useful software available through one product and one API instead of making the customer assemble a chain of unrelated services.

At first, that sounds like an interface problem: put several models behind a chat box and route each request to the right provider. Building the real product taught me that routing is only the beginning. The harder work is making every accepted request produce a result the customer can actually open, use, download, revisit, and revise.

Real customer problems forced that lesson repeatedly. A request could pass automated tests while the customer journey still broke. Audio could exist but be the wrong kind of audio. An entitlement could be correct in one system and stale in another. A provider could report success while the exported file was unusable. I stopped treating “the test passed” as the finish line. I began requiring evidence from the delivered output and the complete customer journey.

One API still needs capability-specific contracts

Different kinds of work do not become identical merely because they share authentication. Research can complete synchronously and return sources. Image or Office generation may require a durable background job. A spreadsheet is not successful because a ZIP of XML exists; it must open in Excel, preserve formulas, and remain editable. A retry after a network timeout must not submit the provider job twice or charge twice.

The current 3Stone developer contract therefore uses capability-specific endpoints behind common authentication, idempotency, usage, job, and artifact conventions. Certified public routes include chat, research, image analysis, image generation, editable PowerPoint, Excel and Word generation, music, and interactive tools. Video exists in the customer product, but it is not presented as a public API capability until the external lifecycle meets the same certification standard.

That boundary matters. A useful API should describe what developers can verify today, not everything an internal product can theoretically attempt.

Make asynchronous work explicit

Long-running AI work creates an awkward interval: the provider may have accepted a paid job even when the HTTP response never reaches you. Retrying blindly can create two jobs. Reporting failure immediately can also be wrong.

We use durable request and job identities, save provider receipts as early as possible, and separate states such as accepted, running, completed, failed, and reconciliation required. The customer polls the 3Stone job rather than learning each provider's lifecycle.

A simplified image request looks like this:

const base = "https://one.3stoneai.com";
const idempotencyKey = crypto.randomUUID();

const submitted = await fetch(`${base}/v1/images`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.THREESTONE_API_KEY}`,
    "Content-Type": "application/json",
    "Idempotency-Key": idempotencyKey,
  },
  body: JSON.stringify({
    prompt: "An editorial photograph of a solar-powered neighborhood library at sunrise; no text or logos."
  }),
});

const job = await submitted.json();
if (submitted.status !== 202) throw new Error(job.error?.message ?? "Request rejected");

while (true) {
  await new Promise(resolve => setTimeout(resolve, 2500));
  const response = await fetch(`${base}/v1/jobs/${job.job_id}`, {
    headers: { Authorization: `Bearer ${process.env.THREESTONE_API_KEY}` },
  });
  const current = await response.json();
  if (current.status === "completed") break;
  if (["failed", "reconciliation_required"].includes(current.status)) {
    throw new Error(`Job ended as ${current.status}`);
  }
}

The important details are not glamorous: bearer keys stay on the server, every mutating request has a unique idempotency key, accepted asynchronous work returns a durable job ID, and ambiguous provider outcomes are reconciled instead of automatically replayed.

A fallback must support the same job

One recurring integration mistake was treating any available model as a fallback. It is not enough for two providers to share an output label such as “image” or “video.” A text-to-image endpoint may not edit an image. A text-to-video endpoint may not replace a character in existing footage. A video endpoint that returns silent footage cannot honestly claim synchronized audio.

That changed how I think about recovery. A fallback is eligible only when it supports the requested inputs, preservation behavior, output format, and practical limits. It also must be safe for the particular failure. A known authentication rejection before execution is different from a connection loss after a provider may have accepted a paid request.

For image generation, 3Stone's separate-provider fallback is constrained to confirmed operational failure classes. It does not activate for moderation decisions, ambiguous post-submission failures, or arbitrary errors. For video, audio routing distinguishes source audio, ambience and effects, dialogue, music, narration, and intentional silence. ElevenLabs can be a recovery option for appropriate speech; it is not automatically attached to every video.

The broader lesson is simple: recovery that changes the customer's request is not recovery.

Validate the artifact, not the provider status

The defect pattern that most changed my release standard was seeing technical success without a usable customer outcome. A provider could say “succeeded,” a route could return HTTP 200, and a test suite could be green—yet the file might not open correctly, the audio might be irrelevant, or the next customer action might fail.

Now I want evidence at the artifact boundary. Images must decode and meet the expected dimensions. Videos are probed for duration, codec, dimensions, and required audio. PowerPoint, Excel, and Word files are reopened and checked for their native structure, including slides, sheets, formulas, tables, and editability. Interactive HTML is rendered and exercised. Customer certification includes creation, preview or playback, download, persistence, reopen, and revision where the capability supports it.

This standard also applies to account and entitlement work. A deletion button is not verified because it displayed a confirmation dialog; the customer must return to signed-out state and the intended account data must actually be gone. A microphone permission prompt is not proof of a live voice session; the system needs a real captured turn, an audible response, interruption behavior, transcript persistence, and correct accounting.

Idempotency is a billing promise

Idempotency is often described as an API convenience. For paid AI work, it is also a billing guarantee.

Each mutating request binds the caller's idempotency key to a fingerprint of the operation. The same key with the same request returns the original outcome. The same key with changed input is rejected. Before expensive execution, the system reserves a bounded allowance. Completion settles recorded usage. Known pre-execution failure releases the reservation. An ambiguous provider outcome enters reconciliation instead of triggering another paid attempt.

We test the behavior, not just the presence of the header. In production certification, an immediate replay of a completed request returned the original request identity without creating another charge. That evidence is more meaningful to me than a unit test that merely proves the code reads Idempotency-Key.

Defects became architecture

Several parts of the architecture came directly from customer-facing failures:

Provider routing became capability-aware because “another model” was not necessarily compatible.
Recovery gained durable receipts because a timeout did not prove that no paid job existed.
Audio became intent-aware because generic narration could technically make a file audible while making the result worse.
Entitlement checks moved toward an authoritative shared view because stale plan state could break an otherwise correct journey.
Office and media delivery gained output inspection because generated bytes were not proof of a usable artifact.
Account deletion gained resumable phases because destructive work must finish completely and safely retry after partial failure.
Alerts gained provenance because certification tests and capability questions should not be reported as failed customer jobs.

These were not abstract platform goals. They were responses to moments when the product's internal story and the customer's actual experience disagreed.

The public request can stay small

curl https://one.3stoneai.com/v1/chat \
  -H "Authorization: Bearer $THREESTONE_API_KEY" \
  -H "Idempotency-Key: first-request-001" \
  -H "Content-Type: application/json" \
  -d '{"model":"3stone-auto","input":"Explain exactly-once API billing simply."}'

The small surface is intentional. Underneath it, the system still needs tenant isolation, request identity, usage attribution, safe retry behavior, and a result the developer can verify.

What I would tell another builder

My biggest shift was moving from provider success to customer-verifiable delivery. If I hope another developer changes one thing after reading this, it is this: do not release the promise represented by an endpoint until you have followed the real output through the customer's next action.

6 things that broke our Polymarket trading bot before it made a dollar

Wed, 07 Oct 2026 12:47:20 +0000

We spent a few weeks trying to build a profitable Polymarket bot. Arbitrage between linked markets, in-play market making, copying wallets that trade before goals, betting against miscalibrated prices. We recorded the full order book for every football market, indexed on-chain trades and replayed each idea.

None of them made money. The full research with numbers is on our blog, and the CSVs are on GitHub.

This post is the engineering side: six things about the Polymarket API and our own pipeline that would have broken the bot even if a strategy had worked.

1. Your taker order waits a second, and you can't cancel it

Sports markets have a field in the Gamma API called secondsDelay. Football is 1, NBA is 0. You can see it on any event:

GET https://gamma-api.polymarket.com/events?slug=fif-sri-mri-2026-10-05
... "secondsDelay": 1 ...

Per the Order Lifecycle docs, a marketable order waits out that delay before matching, and "during either delay, the order is pending and cannot be canceled". Resting limit orders can still be cancelled. Some crypto and finance up/down markets have a 250 ms version of the same thing.

What it did to us: our arbitrage detector found 369 mispriced sets across linked football markets in 18 hours. Of the 2c+ ones we rechecked one second later, all legs were still in the book in 3, worth $1.83 total. Makers pull their quotes inside your delay, or a faster taker takes the legs. If your bot sees a price and hits it, check secondsDelay first.

2. Fees are not a percentage

Only takers pay, and the fee depends on price:

def taker_fee(shares: float, price: float, fee_rate: float) -> float:
    # fee_rate from Gamma feeSchedule.rate, e.g. 0.05 for sports
    return shares * fee_rate * price * (1 - price)

It peaks at 0.50 (p * (1 - p) = 0.25) and goes to zero near 0 and 1. A 1c edge on a 50c contract is mostly eaten. Rates and maker rebates are in the fees docs.

3. The mempool gives you nothing

Orders and cancels are off-chain (EIP-712 signatures), only the settlement goes to Polygon. We measured:

Source	Lag behind matching
Order book WebSocket	~17 ms
On-chain settlement	~2 s median

By the time a trade shows up in the Polygon mempool, it already happened and the book already moved. Watch the WebSocket, not the chain.

4. WebSocket events arrive out of order

The exchange can send price_change (a level got eaten) before the last_trade_price event of the same trade. We replayed fills in arrival order at first, and every fill looked about 25% worse than it really was. Sort by the trade, not by arrival time, before you trust any replay.

For what it's worth, a plain WebSocket recorder (price_change, book, last_trade_price with tx hash) captured 99.9% of the trades that later appeared on-chain, so you don't need anything fancier to build a replay dataset.

5. Your backtest can lie with perfect data

This one wasn't the API, it was us. We picked wallets that bought right before goals and simulated copying them 1-5 seconds later: +$4.23 per $10. Then we noticed we had only selected the buys that came right before goals. In real time you don't know which buys those are.

Rerun on every buy by the same wallets, without looking at the outcome: -$0.41 per $10.

Any filter that uses something you only learn later (a goal, a resolution, a price move) has to be removed from the selection step. If the result survives that, then you can start believing it.

6. Some "Polymarket bot" repos want your private key

Before you reach for an existing open-source bot:

StepSecurity found 20+ fake repos with bought stars (polymarket-trading-bot-new, polymarket-copy-trading-bot-sports, ...) pulling typosquatted npm packages that steal keys and .env files.
SafeDep found 9 npm packages (polymarket-bot, polymarket-claude-code, ...) that ask you to paste your wallet key.

Read the code and the dependency tree before anything touches a funded key.

Smaller Gamma API notes

offset pagination returns 422 somewhere after 2,100-2,500 records. Use the keyset endpoints for a full crawl.
/events/keyset ignores the negRisk filter, filter on the market field instead. /markets/keyset respects end_date_min / end_date_max.
Some requests fail without a User-Agent header.

The data is CC BY 4.0. If you're building something on Polymarket and want to check our numbers, the repo has every CSV, and we'd like to hear where we got it wrong.

Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.

Backend Architecture Starts With Workflows, Not Frameworks

Tue, 06 Oct 2026 14:30:00 +0000

When teams begin designing a backend, technology discussions often arrive first.
Node.js or Python?
PostgreSQL or MongoDB?
Monolith or microservices?

Those choices matter, but the architecture usually becomes clearer if you start somewhere else:
the workflow.

Follow One Operation From Beginning To End

Suppose an e-commerce customer places an order.

The backend may need to:

Authenticate the customer
Check inventory
Validate pricing
Create an order
Contact the payment provider
Update inventory
Trigger fulfillment
Send confirmation

The architecture question is not simply:
Which framework should execute this?

A better question is:
Which steps must succeed together, which can happen later, and what happens when one fails?

That immediately exposes important design requirements.

Define Data Ownership

Modern applications frequently use multiple external platforms.

Customer information might appear in:

The main application database
A CRM
A payment platform
An analytics system
A support platform

Without explicit ownership rules, synchronization becomes unpredictable.
Backend design should identify which system owns each important data type and how changes move between systems.

Design Integration Failure Paths

Third-party APIs will eventually:

Time out
Reject requests
Apply rate limits
Return unexpected responses
Experience downtime

A production backend needs a strategy for those situations.
That may include retries, queues, idempotency, fallback behavior, error logging, or manual recovery.
The exact solution depends on the workflow, but ignoring the failure path is rarely a good solution.

Treat Authorization As Architecture

Authentication tells us who made the request.
Authorization decides what that person can do.
As applications grow, the second problem often becomes harder.

A SaaS application, for example, may have organization owners, administrators, managers, ordinary members, billing users, and support staff.

If permission rules are embedded randomly throughout controllers and endpoints, they become difficult to reason about.

Defining authorization boundaries early makes the rest of the system easier to maintain.

Choose Architecture Based On Operational Needs

Microservices are useful when independently managed services genuinely solve an organizational or operational problem.

They are not automatically an upgrade from a monolith.

A modular application with clearly separated responsibilities can be a strong choice for many products.

Architecture should match:

Product complexity
Team structure
Deployment requirements
Operational overhead
Expected change

Not architectural fashion.

Think About Production Before Production

Backend design should also answer:
How will we know when something is wrong?

Logs, error tracking, metrics, alerts, database visibility, and integration monitoring are part of the system, not extras to think about after launch.

The framework can change.

A poorly defined workflow is much harder to fix.

For teams currently mapping these decisions, backend development services provide a broader view of how APIs, databases, integrations, authentication, architecture, and maintenance fit together.

What do you define first when starting a new backend: domain boundaries, data model, API contracts, or infrastructure?

Subscription Seats vs API Keys: Running the Break-Even Math

Tue, 06 Oct 2026 12:52:52 +0000

A Hacker News thread comparing the value of Anthropic and OpenAI subscriptions turned into something more useful than a vendor scoreboard: people posting their actual usage and realizing they were paying the wrong way for it. Flat-rate seats and metered API keys are the same capability priced on two different axes, and most teams pick one by habit.

The Decision Is Billing Shape, Not Vendor

A subscription is a flat monthly fee per person with soft usage caps. An API key is pay-per-token with no ceiling. Does your usage pattern reward a fixed price or a variable one? That's the distinction that matters before you compare vendors—who gives more value per dollar is secondary.

Heavy, steady, interactive use favors seats. Someone coding against a model for six hours a day will blow past any flat fee's token-equivalent value, and the vendor pays for that. Bursty, automated, or embedded use favors the API: a feature that calls a model twice per user session doesn't need a seat, and more importantly, a seat can't be called from your backend anyway.

Teams buy seats for people who use the model twice a week, or run batch jobs on API keys that would have been free under a plan someone already pays for. Both are quiet, recurring waste that nobody's dashboard flags.

Pull Your Logs and Compute It

You don't need a model to decide this. You need last month's token counts per person, which every provider console exports.

seat_price = 100.00 # flat monthly plan, per person
blended_rate = 6.00 / 1_000_000 # your $ per token, input+output mixed
tokens = 18_000_000 # last month, one person, from your logs

api_cost = tokens * blended_rate
print(f"${api_cost:.2f} metered vs ${seat_price:.2f} seat")
print("buy the seat" if api_cost > seat_price else "stay on the API")

Run it per person, not as a team average. Averages hide the shape of the distribution - in my experience a few people generate the majority of tokens, and those are the only ones where a seat clearly wins.

Then apply two adjustments the math misses. First, subscription plans have usage limits that reset on a schedule; if your heaviest user hits them and stalls mid-task, the lost hours cost more than the metered overage would have. Second, anything that runs without a human present - evals, pipelines, product features - belongs on an API key regardless of price, because seat terms generally cover personal interactive use. That's a terms question, not a cost question, and it isn't negotiable with a spreadsheet.

Key Takeaways

Decide billing shape before vendor: flat seats suit heavy interactive users, metered keys suit bursty or automated workloads.
Compute break-even per person from real token logs, not team averages - usage is almost always skewed toward a few people.
Automated and product-embedded calls go on API keys regardless of the math, because subscription terms assume a human in the loop.

What does your per-person token distribution actually look like - is it two heavy users carrying the team, or spread evenly?

Sources referenced: Hacker News discussion comparing Anthropic and OpenAI subscription value

API Architecture

Mon, 05 Oct 2026 08:01:11 +0000

Here’s a concise LinkedIn-ready article focused on practical API architecture in Rails.

API Architecture: It’s More Than Just Endpoints

A well-designed API isn't simply about creating routes like:

GET /users
POST /orders

The real challenge is designing an architecture that remains secure, scalable, maintainable, and easy to integrate as the application grows.

️ How I approach API architecture with Ruby on Rails

A typical Rails API can be structured around clear layers:

Client → Routes → Controllers → Services → Models → Database

For example:

POST /api/v1/orders

The controller should handle the HTTP request and response, while the business logic can live in a Service Object.

Orders::Create.call(
  user: current_user,
  params: order_params
)

This keeps controllers thin and business logic easier to test and reuse.

Important architecture decisions

Versioning

/api/v1/...

API versioning allows you to evolve the API without immediately breaking existing clients.

Authentication & Authorization

Use mechanisms such as JWT, OAuth2, or token-based authentication, combined with authorization rules.

Consistent Responses

Define predictable JSON structures for:

Success responses
Validation errors
Authentication failures
API errors

Background Processing

Heavy operations shouldn't block the API request.

Rails + Sidekiq + Redis can move tasks such as emails, notifications, and data processing into background jobs.

Caching

Frequently requested data can be cached with Redis to reduce database load and improve response time.

Why good API architecture matters

A good architecture makes it easier to:

✅ Scale the application
✅ Add new API versions
✅ Integrate third-party systems
✅ Test business logic
✅ Maintain clean controllers
✅ Improve security
✅ Reduce technical debt

The goal isn't to build more endpoints.

The goal is to build an API that can evolve without becoming a bottleneck for the entire system.

RubyOnRails #Ruby #API #APIDesign #SoftwareArchitecture #BackendDevelopment #WebDevelopment #RESTAPI #SoftwareEngineering

I Migrated from Python Scrapers to Go and Search APIs - Complete Guide published: true

Thu, 01 Oct 2026 10:26:24 +0000

•Observation :
Examples using public third-party APIs. Not affiliated with or endorsed by any API provider. All trademarks belong to their respective owners.

I'm Pires Aurélio , a Go/Python developer from Angola. I build scrapers in production and document everything in public.

My work:

Main scraper (Python/AsyncIO): https://github.com/Aureliopires186/py-websites-scraper
Go SerpApi Examples: https://github.com/Aureliopires186/go-serpapi-examples
Jobs Scraper Tracker: https://github.com/Aureliopires186/-serpApi-go-jobs-scraper/issues/3
Blog - How to build REST API in Go: https://blogspotangola.blogspot.com/2026/09/how-to-create-simple-rest-api-with.html
LinkedIn: https://www.linkedin.com/in/pires-aurélio-511330385

For 2 years I scraped sites with pure Python/Go. Then I hit Google. Everything broke.

This tutorial is how I solved it with Go + SerpApi.

1. Why My Python Scraper Failed on Google

My repo py-websites-scraper uses asyncio + aiohttp + readability-lxml. Great for blogs.

But for Google Jobs / Shopping:

CAPTCHA in 10 requests
429 blocks
Selectors change weekly

I was spending 80% time fighting anti-bot, 20% building features. Wrong.

2. Why SerpApi + Go Is The Stack

SerpApi handles proxies, CAPTCHAs, parsing. You get clean JSON.
Go handles concurrency perfectly.

My stack now: Go + SerpApi + Gin REST API (as I teach in my blog)

3. Setup (2 minutes)

go mod init go-serpapi-demo
go get github.com/serpapi/google-search-results-golang
go get github.com/gin-gonic/gin
export SERPAPI_KEY=your_free_key_from_serpapi.com

#. Example 1: Google Jobs - (Fixed):


package main

import (
  "fmt"
  "log"
  "os"

  g "github.com/serpapi/google-search-results-golang"
)

func main() {
  apiKey := os.Getenv("SERPAPI_KEY")
  if apiKey == "" {
    log.Fatal("SERPAPI_KEY is not set")
  }

  params := map[string]string{
    "engine": "google_jobs",
    "q": "Go Developer",
    "location": "Remote",
    "hl": "en",
  }

  search := g.NewGoogleSearch(params, apiKey)
  results, err := search.GetJSON()
  if err!= nil {
    log.Fatalf("SerpApi request failed: %v", err)
  }

  jobsRaw, ok := results["jobs_results"].([]interface{})
  if!ok {
    log.Printf("No jobs_results in response: %+v", results)
    return
  }

  if len(jobsRaw) == 0 {
    fmt.Println("No jobs found")
    return
  }

  for _, j := range jobsRaw {
    jobMap, ok := j.(map[string]interface{})
    if!ok {
      continue
    }
    fmt.Printf("[%v] %v - %v\n", jobMap["via"], jobMap["title"], jobMap["company_name"])
  }
}


#. Example 2: Google Shopping Price Tracker: ( Fixed):


package main

import (
  "fmt"
  "log"
  "os"

  g "github.com/serpapi/google-search-results-golang"
)

func main() {
  apiKey := os.Getenv("SERPAPI_KEY")
  if apiKey == "" {
    log.Fatal("SERPAPI_KEY is not set")
  }

  params := map[string]string{
    "engine": "google_shopping",
    "q": "golang book",
  }

  search := g.NewGoogleSearch(params, apiKey)
  results, err := search.GetJSON()
  if err!= nil {
    log.Fatalf("failed to get shopping results: %v", err)
  }

  shoppingResults, ok := results["shopping_results"].([]interface{})
  if!ok {
    log.Printf("No shopping_results found: %+v", results)
    return
  }

  fmt.Printf("Found %d products\n", len(shoppingResults))
  for _, item := range shoppingResults {
    if m, ok := item.(map[string]interface{}); ok {
      fmt.Printf("- %v: %v\n", m["title"], m["price"])
    }
  }
}




______________

By:Pires Aurélio 
Contact: *Emails are not allowed* 
https://blogspotangola.blogspot.com/?m=1
https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385

I Open-Sourced 5 Real-World API Integration Examples with Go (Gin) — Here's What I Learned

Wed, 30 Sep 2026 21:12:41 +0000

I Open-Sourced 5 Real-World API Integration Examples with Go (Gin) — Here's What I Learned

I love Go for its simplicity. But finding clean, real-world examples of how to use Search APIs with Go is hard. So I decided to build it myself.

I'm excited to share my new open-source project:

https://github.com/Aureliopires186/go-serpapi-examples

The goal is simple: help Go developers in Africa and around the world build products with real-time data without dealing with scraping, proxies, or captchas.

What's inside?

The repo is not just one example, it's a collection of ready-to-use REST APIs:

✅ Google Jobs API – Build your own job board (perfect for remote jobs)
✅ Google Maps API – Location search & place details
✅ Google Search API – SERP tracking tool
✅ All built with clean Go, Gin, and best practices

Each example has a README.md with step-by-step instructions.

Why SerpApi + Go?

I tested scraping myself. It broke every week. SerpApi just works and returns clean JSON. For Go developers, that's gold. You focus on go run, not on fixing parsers.

Full Tutorial

I wrote a detailed breakdown of how I built the Jobs API on my blog:

Read the full tutorial: https://blogspotangola.blogspot.com/2026/09/how-to-create-simple-rest-api-with.html

What's next?

This is just the start. I want to add more APIs (Flights, Shopping, Scholar) and create video tutorials.

If this project helped you, please:

⭐ Star the repo to support the work
Open an issue telling me what example you want next
Connect with me, I'm documenting my journey to becoming a Developer Advocate

Let's connect:
LinkedIn: https://www.linkedin.com/in/pires-aurélio-511330385
My Blog: https://blogspotangola.blogspot.com/

https://dev.to/
https://github.com/Aureliopires186

Thanks for reading! What API would you build first?

Author :Pires Aurélio

SML DESK (free web app)

Tue, 29 Sep 2026 17:53:47 +0000

SML LIVE · SCRIPTMASTERLABS FLAGSHIP · REALTIME VOICE + X402 + EVIDENCE
Evidence before the agent spends.
Pay-per-call APIs and a read-only provider check for x402 buyers. Not a facilitator. Not a wallet. SML LIVE is SCRIPTMASTERLABS’ BYOK GPT Live flagship for realtime voice, x402 discovery, provider verification, policy checks, and proof-state preservation. V1 is quote-only: it can inspect and present terms but cannot sign or spend. No SML subscription is required for the BYOK voice path.

SML LIVE turns realtime AI voice into an evidence-aware economic-agent interface. Bring your own GPT Live/OpenAI realtime key. No SML subscription. SML discovers machine capabilities, evaluates provider evidence, inspects fresh x402 terms, applies caller-controlled policy, and preserves the proof trail underneath the conversation.

BYOK GPT LIVE
NO SML SUBSCRIPTION
CALLER-CONTROLLED X402
BASE · USDC
EVIDENCE-FIRST
Truth First. Proof Always. Pay Only for Accepted Delivery.
Launch SML LIVE
See the flagship architecture
Verify Buyer Evidence
Live today: realtime call setup, SML function brokering, external x402 discovery, fresh 402 quote inspection, policy evaluation and evidence receipts. V1 is quote-only and cannot sign or spend. Production self-test has observed a real 0.001 USDC Base-mainnet 402 while sending no payment and no signature.
ONE ENGINE · SIX SML LIVE MODES
The flagship makes the whole SML stack usable through conversation.
These modes share one economic-control engine. We expose current status instead of pretending every future execution path is already production-live.

Live Buyer
“Buy the data we need.”
Discover → trust → quote → policy → purchase → verify.

QUOTE LIVE · PURCHASE NEXT
Live Witness
“Is what he just said true?”
Claim → research → provenance → evidence verdict.

EVIDENCE PRIMITIVES LIVE · AMBIENT NEXT
Live Broker
“Find me something that can do this.”
Dynamically discover MCP/x402 providers and inspect current terms.

DISCOVERY + QUOTE LIVE
Live Operator
“Handle this job for me.”
Compose free and paid tools under one bounded policy.

COMPOSED EXECUTION NEXT
Live Deal Desk
“Can we safely use this vendor?”
Identity + reliability + pricing + runtime evidence + unknowns.

TRUST STACK LIVE · VOICE FLOW NEXT
Live Auditor
“What did my agents spend and receive?”
Authorization → settlement → delivery → accepted-delivery ledger.

PROOF MODEL LIVE · LEDGER NEXT
THE SML ENGINE UNDER SML LIVE
Exact machine jobs remain directly buyable.
SML LIVE is the flagship interface. The existing APIs and MCP products remain independently discoverable machine primitives underneath it.

AGENT WEB ACCESS · URL → MARKDOWN
Read a safe public webpage
One safe public HTTPS URL in. Compact LLM-ready Markdown out.

$0.001 USDC / accepted read
URL → 402 → policy → Markdown
URL to Markdown API →
PROVIDER TRUST · SOURCE VERIFICATION
Verify before trusting
Inspect provider identity, runtime surfaces, status and evidence before dependency or spend.

provider → evidence → caller decision
Provider Trust API →
WEB RESEARCH · SOURCES
Search + extract evidence
General-web search plus safe source extraction when the source is not already known.

$0.003 effective accepted call
question → search → sources → evidence
Web Research API →
FX · CURRENCY CONVERSION
Convert currency
ECB-backed reference-rate conversion with machine-readable output.

$0.001 USDC / one-shot
Currency Conversion API →
MARKET DATA · STOCK QUOTE
Get a stock quote
Machine-readable quote evidence for a small current-data primitive.

Stock Quote API →
TRUTH · FACT VERIFICATION
Verify a market-data claim
Cross-check current provider evidence instead of guessing.

Fact Verification API →
All jobs + prices
Try Before You Buy
Raw JSON catalog
SIX PRIMARY MCP PRODUCTS
The infrastructure SML LIVE can reason over.
Provider Trust
Verify a source or provider before trusting or paying it.
GhostKey
Delegate narrowly scoped browser actions into a browser the user authenticated themselves; no credential export or autonomous spending.
SqueezeOS
Find the right API or tool for a task.
x402 Gateway
Discover x402 paid APIs and inspect payment requirements before calling them; caller controls spending policy.
Robinhood RWA Intelligence
Read-only tokenized-stock and RWA evidence.
ScriptDocs
Current software package docs, metadata and vulnerability evidence.
Open canonical MCP stack
PUBLIC INTELLIGENCE & DISCOVERY
Free surfaces for people, agents and crawlers.
SML Growth + x402 Community OS
Public x402/MCP intelligence, readiness context and crawler discovery.
x402 Agent Menu
Talk. Discover. Search paid jobs and copy route-specific instructions before a caller-controlled x402 payment.
Evidence Hub
Dated primary-source records and machine-readable proof boundaries.
TWO BUYERS · TWO FRONT DOORS
AI agents & developers
Start with SML LIVE, exact endpoints, MCP and x402. BYOK model access keeps the realtime provider relationship yours.

SML LIVE
Agent Market
Traders & human buyers
Trading indicators, SMLSheets, books and other human products remain separate from the agent-payment funnel.

Human Store
Trading Indicators
PROOF & ECOSYSTEM
Verify before you trust.
SCRIPTMASTERLABS
SML LIVE · MCP · x402 v2 · Base USDC · machine-readable discovery · evidence-backed delivery
★ Add as Google Preferred Source
Founder LinkedIn
PeerPush Listing
Circle Alliance Program member badge

Pro Seats vs Metered API: Where the Break-Even Actually Falls

Tue, 29 Sep 2026 12:15:57 +0000

OpenAI reopening its $200/month Pro tier put a familiar budgeting question back on the table for teams: do you buy people flat-rate seats, or do you wire the same models in through the metered API and pay per token? Most teams pick by vibes. There's a two-minute calculation that settles it.

What You're Actually Buying at $200 a Seat

A flat seat and an API key give you access to similar models, but they're different products. The seat is an unmetered interactive surface for one human - chat, file uploads, the newest reasoning models, no per-request accounting. The API is programmatic access with logging, routing, prompt control, and a bill that moves with usage.

No cost-attribution overhead, no finance conversation about whose experiment spiked the bill, and a hard ceiling on what one user can spend - these are the things the seat quietly includes. At typical per-token rates, a single person doing heavy chat work all month rarely approaches $200 in raw inference cost. The API looks cheaper until you count these things. The seat's real product is predictability. The API's real product is control.

Run the Numbers Before You Argue About Them

Plug your own provider's current rates and your team's actual usage into this:

seat = 200 # flat monthly subscription
in_rate, out_rate = 3, 15 # $ per 1M tokens (check live pricing)
calls, in_tok, out_tok = 900, 4000, 900 # per user, per month

api = calls * (in_tok/1e6*in_rate + out_tok/1e6*out_rate)
print(round(api, 2), "vs", seat) # 22.95 vs 200

At that usage - roughly 45 substantial requests a workday - metered access costs about a ninth of a seat. Break-even lands near 7,800 calls per user per month, or about 390 a day. No human types that much. An agent loop hits it before lunch.

The decision line is: who or what is generating the requests.

A person in a chat window → seat. You'd need to 8x a heavy user's volume to justify the switch, and you'd pay for that in engineering time building the interface they already have.
An automated pipeline or agent → API. Volume scales past break-even fast, and you need the logging and retry control anyway.
The messy middle - a human clicking a button that fans out into 300 model calls - → API, with a per-user spend cap. This is where teams get surprise invoices, because the usage feels human-scale and isn't.

One more thing the math hides: variance. A seat caps your downside at $200. A runaway retry loop on the API does not cap anything. If your team is early and your guardrails are thin, paying a premium for a known number is a defensible call - just make it on purpose.

Key Takeaways

Compare flat seats and metered API on request volume per user, not headline price - break-even often sits near 8,000 calls a month.
Seats buy predictability and a ready-made interface; the API buys control, logging, and the ability to scale past a single human's output.
Human-driven work that silently fans out into hundreds of calls is where budgets break. Cap per-user spend before you ship it.

What's your team's actual monthly call volume per user - and have you ever measured it, or are you estimating?

Sources referenced: Hacker News discussion on OpenAI re-opening Pro subscriptions

Aviation Edge Future Schedules API — A JSON REST Airport Schedule API

Tue, 29 Sep 2026 12:04:20 +0000

Aviation Edge Future Schedules API provides flight by flight and complete airport schedule data on a requested date.

The Future Schedules data has worldwide coverage for all international and domestic airports in the world, with the exception of military airfields and a number of local airports with very little traffic.

It is a JSON REST API that works simply and smoothly with many filters. Specifying the airport code, schedule type (departure or arrival), and date parameters are obligatory. You can use the remaining parameters as filters to narrow down the output to the exact future airport schedules data you are looking for.

The Future Schedules API returns your API client the complete output in around 0.3 ms in JSON format. Read below for developers' highlights!

Future Airport Timetable API Developers' Highlights

What are the details included in the Future Airport Timetable API? Below are the details included for each flight in the API output:

Airport: Departure and arrival airports' IATA and ICAO codes
Weekday: The day of the week (“1” is Monday)
Terminal and gate: The most common terminal and the gate number of the flight
Flight time: Scheduled departure or arrival time of the flight
Aircraft details: Model code and model name with the manufacturer
Airline details: Name, IATA, and ICAO codes of the airline
Flight information: Flight number in IATA and ICAO format (“AB1234” and “ABC1234”)

Tip:

If you'd like to get a whole week’s or a whole month’s schedule, keep in mind that the API can only return a single date at a time. To cover a complete week, month, or a specific date range, you may simply make multiple calls for the dates included in these.

What about filters? We have enabled many of the parameters of the output as filters. This way, you can narrow down the data to the airlines, departure/arrival airports, specific flight numbers, and more. Here are all the parameters you can use with the Future Airport Timetable API:

&iata_code= (obligatory) IATA code of the requested airport
&type= (obligatory) Schedule type (can be departure or arrival)
&date= (obligatory) Requested date in YYYY-MM-DD format
&dep_iataCode= flight’s departure airport’s IATA code
&dep_icaoCode= flight’s departure airport’s ICAO code
&arr_iataCode= flight’s arrival airport’s IATA code
&arr_icaoCode= flight’s arrival airport’s ICAO code
&airline_iata= airline IATA code
&airline=icao= airline ICAO code
&flight_num= flight number in “1234” format without the airline code at the beginning

Tip:

For the complete documentation, visit the Aviation Edge Website or the GitHub repo of the API. Got any questions or comments? Contact us and we'll get back to you!

{% embed https://github.com/AviationEdgeAPI/Future-Schedules-API %}

Future Airport Schedules API Use Cases

The Future Airport Schedules API is particularly useful for developing mobile apps or websites in travel, hospitality, booking, or similar fields. When implemented, the API allows users to select a date and a departure/arrival airport which will return them a list of flights to choose from. It is also eligible and handy to use as a data source to make market research in which the aim is to compare and analyze airport traffic, airline routes, or similar.

There is no limit to the possible use cases for the Future Schedules API, but here are some client favorites to inspire you:

Mobile apps or websites for travel agencies or flight booking sites,
Apps or websites in which airport schedules are displayed,
Hospitality, airport greeting, pick-up or transportation services for in-advance scheduling or similar services,
Comparison of airport traffic around the world. We also provide real-time and historical airport schedule data.
Analyzing and market research of the airport or air traffic based on a region, airline, or others.

Have another project you have else in mind? Feel free to contact us anytime and we’d be happy to exchange ideas with you about our data!

Get Started in a Minute - No Commitments

If you feel like giving the Future Airport Timetables API a try, you can immediately get started by getting your API key here.
Your API key will be automatically sent to your email, fully activated.

Upgrade, downgrade or cancel your key anytime with no commitments.

{% embed https://linktr.ee/AviationEdgeAPI %}

Self-Hosted API Protection: How to Put a WAF in Front of Your API

Tue, 29 Sep 2026 03:10:29 +0000

Self-Hosted API Protection: How to Put a WAF in Front of Your API

APIs are where the valuable data lives — and where attackers go first. A public API endpoint is a direct door to your logic, your database, and your users' sessions. If you self-host your API, you can also self-host its first line of defense: a WAF sitting in front of it, filtering traffic before it reaches your handlers.

Here's how to protect a self-hosted API with a WAF, and where SafeLine fits.

Why APIs are attractive targets

Unlike a rendered web page, an API speaks machine-to-machine — and that means requests are structured, predictable, and easy to script. Attackers go after:

Authentication endpoints — credential stuffing, token abuse, brute force.
Injection in parameters — SQL or command injection through query strings and JSON bodies.
Abuse and scraping — automated clients hammering endpoints for data or compute.
Malformed payloads — probing for parser or deserialization bugs.

A WAF can't fix a broken auth scheme, but it can stop the bulk of automated abuse at the edge.

What a WAF does for an API

Placed as a reverse proxy in front of your API service, a WAF:

Inspects every request, including JSON bodies and query parameters, for attack patterns.
Blocks injection and known malicious payloads before they reach your code.
Rate limits per client, per key, or per path — so one abusive caller can't monopolize the API.
Manages bots, separating automated tooling from legitimate integrations.

Where SafeLine fits

SafeLine is a self-hosted WAF that runs in reverse-proxy mode, so you point your API's traffic at it and it forwards clean requests to your upstream service. For API protection specifically:

Its semantic detection engine analyzes request content — parameters, headers, and bodies — to catch injection and malicious payloads that static rules miss.
Rate limiting can be applied per path, which is ideal for throttling login, token, or search endpoints.
It's self-hosted and free to run (Community Edition covers up to 10 apps at 800 QPS), so protecting an internal or public API doesn't add a vendor bill.

SafeLine won't replace proper auth and input validation in your code — but it removes the flood of automated junk so your application logic only sees traffic worth handling.

Putting it in front of your API

Deploy SafeLine as the proxy in front of your API service:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

In the console at https://<your-server-ip>:9443, add a site whose upstream is your API (for example http://127.0.0.1:8080). Route your API's DNS or gateway at SafeLine's listener, then set rate limits on your most-sensitive paths (auth, token refresh). From there, every request is filtered and throttled before it touches your API.

FAQ

Does a WAF replace API authentication?

No. You still need solid auth and authorization in your app. A WAF stops automated abuse and injection; it doesn't decide who's allowed in.

Will it break legitimate API clients?

Rate limits are set above normal client behavior, and the detection engine targets malicious patterns — legitimate requests pass through. Test against your real client traffic when configuring.

Can it inspect JSON request bodies?

Yes — a semantic engine evaluates the request content, including structured bodies, rather than only matching URLs or headers.

Is this free to self-host?

The Community Edition is free to run indefinitely and covers up to 10 apps at 800 QPS, which is plenty for a typical API footprint.

That's it — your API now has a filtering and rate-limiting layer in front of it.

⭐ SafeLine WAF on GitHub — give it a star if you find it useful
🔗 Official Docs — installation guide, configuration, and API reference
🧪 Live Demo — see the dashboard in action (no login required)

Jev AI: Typed Decisions with Probabilities for Developers

Fri, 25 Sep 2026 08:33:58 +0000

Jev AI turns shared state into typed decisions with probabilities. It is a TypeSafe AI System One model and developer platform for classification, routing, scoring, safety checks, and agent workflows through an online playground, SDK, or REST API.

The platform supports Choice, Score, and Noul question types. Try the Jev model online for free with no waitlist, or explore the developer resources:

Playground: https://thejevai.com/playground
API endpoint: https://thejevai.com/v1/systemone
Documentation: https://thejevai.com/docs
Pricing: https://thejevai.com/pricing

Jev AI helps make small, important decisions faster — not chat, but a decision inside your system.

Polymarket Gamma API endDate Is Not the Resolution Deadline

Thu, 24 Sep 2026 17:36:25 +0000

Checked against the Polymarket Gamma API on 24 September 2026.

Short answer. OrcaLayer compared the endDate field with the written rules on 23 Polymarket markets. Polymarket's documentation describes a market's end date as the point when it becomes eligible for resolution. The field did not behave that way on sports markets: on NFL and soccer matches it was the kickoff, on esports the start plus 6 hours, on tennis the original start plus 7 days. The deadline that governs a market is in its description, and whether it still trades is in closed and acceptingOrders.

If you build on Polymarket data, this changes what your code should read. A bot that stops trading at endDate, a dashboard that counts down to it, and a model that treats it as the resolution time are all reading a field that means different things on different market types.

What is endDate in the Polymarket Gamma API?

endDate is a UTC timestamp on every market in the Polymarket Gamma API. The API does not document what it represents, and it is not the resolution deadline. In OrcaLayer's sample it was kickoff on NFL and soccer matches, start plus 6 hours on esports, and original start plus 7 days on tennis. The binding deadline is in description.

Every market returned by https://gamma-api.polymarket.com/markets/{id} carries endDate at the top level. If you are new to these endpoints, start with our Polymarket REST API guide for developers.

The response usually tells you what kind of market you are looking at. Sports matches carry gameStartTime, eventStartTime, gameId and sportsMarketType (for example "moneyline"). Markets that belong to a group carry groupItemTitle, which can be a label such as "Match Winner" or a date such as "September 30". Closed markets add closedTime and umaEndDate. The rules are in plain English in description.

Polymarket's own resolution documentation puts it plainly: "Always read the resolution rules before trading." The rest of this article shows what happens when code reads endDate instead.

Polymarket endDate by market type

Market type	Markets	What endDate was	What the rules say	Closed before or after endDate
Esports (CS2, LoL, Dota 2)	4	Scheduled start + exactly 6 hours, 4 of 4	A postponed match must be rescheduled to start within 14 days, or the market resolves 50-50	All 4 before, by 19 minutes to 3 h 18 min
Tennis (ATP, ITF)	4	Original scheduled start + exactly 7 days, 4 of 4	A winner must be determined within 14 days, or the market resolves 50-50	All 4 before, by 6.3 to 6.8 days
NFL, college football, soccer matches	4	Kickoff, 4 of 4	No fixed deadline: open until the game is completed	All 4 after, by 2 h 17 min to 4 h 57 min
MLB match	1	Start + 7 days	No fixed deadline	Before, by about 7 days
Sports futures	3	Varies	A "postponed after" backstop	Still open
Deadline and price markets	7	Matched the rules in 4 of 7	"By" a date, or a date in the title	Mixed

The sample is 23 hand-picked markets covering each type, so read each row as an observed pattern, not a platform-wide rate.

Esports: endDate is the start plus six hours

All four esports matches had an endDate exactly six hours after gameStartTime. The Counter-Strike match between QUAZAR and Misa Esports started at 08:00 UTC on 23 September, carried an endDate of 14:00 UTC, and closed at 13:41 UTC.

Six hours is a fair guess for when a best-of-three finishes, but the rules for the same market allow far longer:

If the match is postponed, it must be rescheduled to a start time on or before October 7, 2026 at 11:59 PM ET (14 calendar days after the originally scheduled start of September 23, 2026 at 4:00 AM ET). If, by that deadline, the most recently announced rescheduled start time is on or before the deadline, this market resolves based on the result of the match as played. Otherwise, this market resolves 50-50, regardless of whether the match is subsequently played.

The 14 days apply to the rescheduled start, not to the result. A match rescheduled to start on 7 October can resolve later still, long after endDate has passed. You can see current esports markets on OrcaLayer's esports page.

Tennis: seven days after the original schedule, and it does not move

Every tennis match in the sample had an endDate exactly seven days after its originally scheduled start. Here the 14-day rule applies to the result:

If the match is canceled (not played at all), ends in a tie, or a winner has not been determined by October 6, 2026, 11:59 PM ET (14 days after the scheduled start), this market will resolve to 50-50.

Rescheduling is where it goes wrong. In two ATP matches, gameStartTime was later than the start printed in the rules. The US Open match between Ben Shelton and Carlos Alcaraz was scheduled for 8 September at 15:30 UTC and started on 9 September at 03:00 UTC. Its endDate was 15 September at 15:30 UTC: seven days after the original slot, not the new one.

A countdown built on endDate for a tennis market counts from a schedule that may no longer exist. Take the start from gameStartTime and the deadline from the rules.

NFL and soccer markets: endDate is kickoff, not the end of the game

On the Broncos vs Chiefs NFL game, the SMU vs Louisville college football game, a La Liga match and a Romanian Liga I match, endDate was equal to kickoff to the second. All four markets closed after it, between 2 h 17 min and 4 h 57 min later, which is how long the games took.

The rules on these markets set no deadline at all:

If the game is postponed, this market will remain open until the game has been completed.

A kickoff is the one moment when a match market is certainly not ready to resolve. Code that compares the current time with endDate would have marked all four of these markets as finished at kickoff, while the games were being played and the markets were still trading.

Across the 16 closed markets in the sample, 6 closed after their endDate. Four were these matches. The other two were a Bitcoin price market that closed 14 minutes after its reference time and a Fed rate market, covered below.

Sports futures: sometimes the event, sometimes the backstop

League and tournament winner markets followed no single rule. On Shakhtar Donetsk winning the 2026-27 UEFA Champions League, endDate was 30 May 2027 at 23:59 UTC, while the rules fall back to "Other" only if the season is "postponed after June 19, 2027, 11:59 PM ET", about twenty days later. The New York Yankees World Series market had an endDate of 31 October 2026, and its rules allow postponement until 31 December 2026. The Boston Celtics NBA Finals market matched its rules to the minute.

For futures, the field alone does not tell you whether you are looking at an expected event date or the formal backstop.

Deadline markets: right in 4 of 7, once wrong by 92 days

Markets that ask whether something happens "by" a date came closest to what most people assume endDate means. Two of them, "Will Anthropic or OpenAI IPO first?" and the Clarity Act market, matched the deadline in the rules, one with a one-minute offset (05:00 instead of 04:59 UTC). The two Bitcoin price markets also matched, though their rules point to "the date specified in the title" or "the month specified in the title", so the deadline lives in the market name.

Three did not match:

"Will the US confirm that aliens exist by September 30?" The rules say "by September 30, 2026, 11:59 PM ET", which is 1 October at 03:59 UTC. endDate was 1 January 2027 at 04:59 UTC, 92 days later. That is exactly the endDate of the parent event the market belongs to, which runs to the end of 2026, so the market appears to carry its event's date rather than its own. Two other fields in the same response point the right way or the wrong way: groupItemTitle is "September 30", which matches the rules, and hasReviewedDates is true, which shows that a reviewed flag does not guarantee the date matches the rules.
The Fed rate market for the September 2026 meeting. endDate was 16 September at 00:00 UTC, which is 8 PM ET on 15 September, the evening of the first day of a two-day meeting. The market closed 20 hours after its endDate.
The market on the next Russian parliamentary election. endDate was 30 September 2026 at 00:00 UTC. The rules fall back to "Other" only if results are not known by 30 September 2027. The market closed on 21 September.

On grouped date markets, check groupItemTitle before you trust endDate.

Why endDate changes, and why the Gamma API returns old values

Two markets returned different endDate values depending on which snapshot the API served.

For the aliens market, a response with updatedAt of 21 September showed endDate as 1 October 2026 at 03:59 UTC, which matches the rules. A fresh response on 24 September showed 1 January 2027. For the IPO market, a response with updatedAt of 8 August showed 31 December 2027 at 00:00 UTC, 29 hours before the written deadline, and the fresh response showed 1 January 2028 at 04:59 UTC, which matches the rules. Between those snapshots, the IPO market was corrected and the aliens market moved 92 days away from its own rules.

updatedAt will not tell you when that happened. It changes often and in bulk: at one point on 24 September the aliens, IPO, Celtics and Bitcoin markets all carried the same updatedAt to the microsecond. Store your own fetch time instead.

The older values reached us through cached responses. In three cases in the sample, a plain request to /markets/{id} returned an older snapshot, and adding an unused query parameter returned the current one. If your code reads endDate once when a market is listed, it may be holding a value the platform has since changed.

endDate vs gameStartTime vs closedTime vs umaEndDate

Field	What it held in the sample	Safe use
`endDate`	Kickoff, start + 6 h, start + 7 days, or a deadline, depending on market type	A hint, never a deadline
`gameStartTime`	Start of the sports event, updated when a match was rescheduled	Event start
`closedTime`	When the market actually closed	After-the-fact close time
`umaEndDate`	Identical to `closedTime` on all 16 closed markets	Same as `closedTime`
`description`	The written rules, including any deadline in ET	The real deadline

The formats differ. gameStartTime and closedTime come as "2026-09-23 08:00:00+00", while endDate and eventStartTime use "2026-09-23T08:00:00Z". datetime.fromisoformat rejects both forms before Python 3.11, so normalize first:

from datetime import datetime

def parse_ts(s: str) -> datetime:
    """Parse any Gamma API timestamp into an aware UTC datetime."""
    s = s.replace(" ", "T").replace("Z", "+00:00")
    if s.endswith("+00"):
        s += ":00"
    return datetime.fromisoformat(s)

How to tell if a Polymarket market is still trading

Do not use endDate for this, and do not use active either. The closed CS2 market in the sample returned "active": true together with "closed": true and "acceptingOrders": false. Read closed and acceptingOrders, and for the resolution state read umaResolutionStatus, which was "resolved" on that market.

def is_tradable(m: dict) -> bool:
    return not m.get("closed") and bool(m.get("acceptingOrders"))

This is not bulletproof. In January 2026 a public report on Polymarket's Rust SDK repository showed finished Africa Cup of Nations markets still returning active=true, closed=false and acceptingOrders=true, with endDate equal to kickoff (issue #199). On our September sample the two flags were reliable, but treat them as the best signal available, not a guarantee.

Closing and resolving are separate steps. After a market closes, the outcome is proposed to the UMA Optimistic Oracle, and according to Polymarket's documentation an undisputed proposal resolves after a 2-hour challenge period.

How to find the real resolution deadline of a Polymarket market

Read description. That is the contract. Sports matches say "on or before … ET" or "by … ET (14 days after the scheduled start)". US match markets say they "remain open until the game has been completed". Futures say "postponed after …".
Check the title and groupItemTitle on price and date markets. "On September 17" or a groupItemTitle of "September 30" is the deadline.
Take the start from gameStartTime, not from endDate minus a guess.
Use closedTime or umaEndDate to know when a market actually closed. Both are empty until it does.
Re-fetch before you act. Add a cache-busting parameter and store your fetch time next to every value you keep.

Python: parse the resolution deadline from a market description

This parser found the deadline on every market in the sample whose rules state a full date and a clock time in Eastern Time:

import re
from datetime import datetime
from zoneinfo import ZoneInfo

MONTHS = "January|February|March|April|May|June|July|August|September|October|November|December"
DEADLINE = re.compile(
    rf"\b(?:on or before|by|after) ((?:{MONTHS}) \d{{1,2}}, \d{{4}}),? (?:at )?(\d{{1,2}}:\d{{2}} ?[AP]M) E[DS]?T\b"
)

def rule_deadlines_utc(description: str) -> list[datetime]:
    """Return every ET deadline stated in a Polymarket market description, in UTC."""
    found = []
    for day, clock in DEADLINE.findall(description):
        local = datetime.strptime(f"{day} {clock.replace(' ', '')}", "%B %d, %Y %I:%M%p")
        local = local.replace(tzinfo=ZoneInfo("America/New_York"))
        found.append(local.astimezone(ZoneInfo("UTC")))
    return found

Use America/New_York rather than a fixed offset: ET is UTC-4 until 1 November 2026 and UTC-5 after, so a hard-coded offset will be an hour wrong for several months of every year. The parser returns nothing on US match markets, because their rules contain no deadline, and nothing on price markets, where the date is in the title. It does not handle "noon", "p.m." or abbreviated months, and it does not tell a start deadline from a result deadline, so read the sentence around each match.

What this sample does not tell you

Twenty-three markets show that endDate means different things on different market types. They do not show how often each case occurs across Polymarket, or that every esports market uses exactly six hours. Valorant is not in the sample, and there is a single MLB game.

We did not establish why the aliens market moved, or whether that change was deliberate. All markets here come from the international Polymarket Gamma API. Polymarket US is a separate exchange with its own contract rules, and nothing in this article applies to it.

Data: 23 markets fetched one by one from gamma-api.polymarket.com/markets/{id} on 24 September 2026. The full sample is downloadable as JSON.

Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.

JSON vs MessagePack in Laravel: A Practical API Benchmark

Thu, 24 Sep 2026 13:39:50 +0000

JSON is the right default for most Laravel APIs. It is readable, supported everywhere, easy to debug, and familiar to every client.

MessagePack is worth considering when payload size, bandwidth, or a controlled client ecosystem matters. It is a compact binary format that keeps a schema-free data model, but it is not automatically faster or smaller for every workload.

This article compares both formats in a Laravel API and shows how to adopt MessagePack without breaking existing JSON clients.

The examples use Laravel Msgpack, an open-source package that adds opt-in MessagePack content negotiation, request decoding, safety limits, and a repeatable benchmark command.

The short answer

Use JSON when:

Your clients include browsers, third-party integrations, or unknown consumers.
Human-readable responses and simple debugging are important.
Your payloads are already small or compressed effectively.
You do not control the client decoder and deployment environment.

Use MessagePack when:

You control the clients and can ship a binary decoder.
Network transfer size is important for mobile, edge, or internal services.
Your payloads are large enough for serialization and decoding trade-offs to matter.
You want to keep JSON compatibility while allowing capable clients to opt in.

For many Laravel applications, the best answer is not choosing one format globally. Keep JSON as the default and negotiate MessagePack per request.

What are we comparing?

JSON is a text representation. It is self-describing, easy to inspect, and supported by the web platform.

MessagePack is a binary representation. It encodes the same basic data model using compact type markers and lengths instead of textual punctuation and names repeated in a text format.

The comparison should use the same data, the same PHP runtime, and the same operation count. Otherwise, the result is a benchmark of different workloads rather than a comparison of formats.

Add MessagePack to Laravel

Install the package:

composer require smmehdisharifi/laravel-msgpack

Apply the middleware to a route or route group:

use Illuminate\Support\Facades\Route;

Route::middleware('msgpack')->get('/api/profile', function () {
    return [
        'id' => 1042,
        'name' => 'Laravel MessagePack',
        'roles' => ['maintainer', 'developer'],
        'features' => [
            'negotiation' => true,
            'fallback' => 'json',
        ],
    ];
});

The route does not need separate JSON and MessagePack controllers. Existing clients continue to receive JSON:

curl -i http://localhost/api/profile

A capable client opts into MessagePack with the Accept header. Save the binary response instead of printing it as terminal text:

curl -sS -D - \
  -H 'Accept: application/msgpack' \
  -o profile.msgpack \
  http://localhost/api/profile

The negotiated response includes:

Content-Type: application/msgpack
Vary: Accept

Vary: Accept is important because a cache must not serve a MessagePack response to a client that asked for JSON, or the other way around.

How content negotiation works

The middleware keeps the migration opt-in:

A client that does not request MessagePack receives JSON.
A client that sends Accept: application/msgpack receives a binary response when the route supports the middleware.
If both formats are accepted, the higher quality factor wins.
Wildcards are evaluated by media-type specificity.
A request that rejects every supported representation receives 406 Not Acceptable.

The same negotiation applies to relevant error responses, including unmatched routes and route exceptions. Incoming requests can use application/msgpack or the legacy application/x-msgpack media type.

Run a repeatable benchmark

Laravel Msgpack includes an Artisan command so the comparison is easy to reproduce:

php artisan msgpack:benchmark --iterations=1000 --json

The command uses a deterministic API-shaped fixture with nested values and Unicode text. It measures:

Raw JSON and MessagePack byte size.
Encode time per iteration.
Decode time per iteration.
Optional gzip size and compression timings when ext-zlib is available.

The command warms each operation once before timing it. It reports per-iteration timings so runs with different iteration counts remain comparable.

Results from this package

The following result was produced by the package benchmark on PHP 8.4 with rybakit/msgpack 0.7.2 and 1,000 iterations:

Measurement	JSON	MessagePack	Difference
Raw payload	591 B	435 B	26.40% smaller
Encode time	0.002 ms	0.009 ms	JSON faster
Decode time	0.003 ms	0.008 ms	JSON faster
Gzip payload	377 B	371 B	1.59% smaller
Gzip compress time	0.009 ms	0.010 ms	Nearly equal
Gzip decompress time	0.004 ms	0.003 ms	Nearly equal

There are three useful conclusions here.

First, MessagePack is noticeably smaller before compression for this fixture. That can matter when transfer size is the bottleneck.

Second, the smallest representation is not automatically the fastest representation. For this small payload, JSON serialization is faster in the measured run.

Third, compression changes the size comparison. Once gzip is applied, the gap is much smaller because gzip already removes a lot of JSON's repetitive syntax.

These are indicative measurements, not universal promises. A real decision should use payloads, client runtimes, compression settings, and network conditions that match your application.

Decode MessagePack requests

MessagePack request bodies are decoded automatically when the middleware is applied:

use Illuminate\Http\Request;

Route::middleware('msgpack')->post('/api/profile', function (Request $request) {
    return [
        'received' => $request->msgpack(),
        'name' => $request->input('name'),
    ];
});

Map payloads are also merged into Laravel's normal request input. The original decoded value remains available through request()->msgpack(), including scalar and list payloads.

The package rejects malformed payloads before they reach the route handler. Configurable request size, nesting-depth, and value-count guards provide additional protection against oversized or unexpectedly deep input.

When a route must always return MessagePack

Negotiation is useful when the same endpoint serves different clients. For a route that should always return MessagePack, use the response macro:

return response()->msgpack([
    'message' => 'Created',
], 201, [
    'X-Request-Id' => $requestId,
]);

Direct serialization is also available through the facade:

use Msgpack;

$packed = Msgpack::encode($data);
$unpacked = Msgpack::decode($packed);

A practical decision guide

Do not change every endpoint to MessagePack because a synthetic benchmark produced a smaller payload. Instead, measure the part of the system that matters.

Choose JSON first when interoperability, browser support, observability, or simple debugging is the priority.

Test MessagePack when bandwidth, payload volume, or mobile transfer cost is important and the clients are controlled.

Keep both formats when you serve a mixed ecosystem. JSON remains the safe default, while clients that understand MessagePack can opt into it without a second controller implementation.

Limitations to keep in mind

MessagePack is not encryption. Sensitive data still needs transport security and application-level authorization.

Binary responses are less convenient to inspect in logs and command-line terminals. Save them to a file or decode them with a client library.

Compression may reduce the practical size advantage. Measure compressed and uncompressed responses for the transport configuration you actually deploy.

The benchmark fixture is intentionally stable, not universal. A list-heavy payload, a text-heavy payload, a large nested object, or a response with repeated keys can produce different results.

Try it in your project

Install Laravel Msgpack, apply the msgpack middleware to one endpoint, and compare both representations with your real data.

The repository contains the full implementation, compatibility matrix, configuration reference, safety limits, and benchmark command:

https://github.com/smmehdisharifi/laravel-msgpack

If the package helps your API, a GitHub star helps other Laravel developers discover it.

What is a webhook? How webhooks work, and the six ways they break in production

Thu, 24 Sep 2026 11:58:53 +0000

This is a cross-post. The original on the Railhook blog has a diagram for each part: the anatomy of a signed request, polling against push, the retry ladder, duplicates, out-of-order delivery and secret rotation.

A customer pays for order ord_8412. The shop sends your backend a webhook, and your handler
awards loyalty points, emails a receipt and calls the warehouse — which takes eleven seconds. The
sender's timeout is ten, so it marks the delivery failed and sends it again. The customer gets two
receipts and double points, and your logs show two perfectly healthy 200s.

Nothing in that story is a bug in the usual sense. Every line of the handler did what it said. It
is what happens when code written for a function call meets a protocol built on retries — and it
is the first of six failures this guide walks through, after the basics every one of them depends
on.

What is a webhook?

A webhook is an HTTP POST that one system sends to a URL another system registered, at the
moment an event happens. The receiver registers once — "when an order is created, tell me at
this URL" — and from then on the sender calls it every time. This is what one looks like, signed
the way the open Standard Webhooks
specification describes:

POST /webhooks/orders HTTP/1.1
Host: api.example-shop.com
Content-Type: application/json
Content-Length: 186
webhook-id: msg_2mQ8hZk3Xv9aR1cT
webhook-timestamp: 1789983612
webhook-signature: v1,pX/4CnjtyRIdKhtP4FEBLl0a8pgk0IErOOHRkgsduog=

{
  "type": "order.created",
  "timestamp": "2026-09-21T09:40:12Z",
  "data": {
    "order_id": "ord_8412",
    "customer_id": "cus_2291",
    "total": 42.50,
    "currency": "EUR"
  }
}

The signature is real. It was made with the secret
whsec_DioEXswemwdu+ScHzpEImNzhtrASzXe0vxCXPBmwuTM=: paste the body, that secret and the three
headers into the signature verifier and it will say the signature is
valid — and that the timestamp is far too old to accept. Both answers are correct, and the second
one is a security feature.

Webhook vs API polling

The alternative to being told is asking: calling the other system's API on a timer to see whether
anything changed.

	Polling an API	Receiving a webhook
Who starts the request	You, on a timer	The sender, when something happens
Delay	Up to one polling interval	Usually seconds
Requests when nothing happens	One per interval, forever	None
What you must run	A scheduled job	A public HTTPS endpoint
If you are down	You catch up on the next poll	You depend on the sender's retries

Shorten a polling interval and you pay in requests and rate limits; lengthen it and you pay in
lag. A webhook costs one request per event. But read the last row twice: polling fails safe,
because a missed poll is repeated by the next one, while a webhook fails however the sender
decides. That is why polling keeps a job in a good webhook integration — as the slow
reconciliation that catches whatever the webhooks missed.

Sending webhooks: what the sender decides

Event types. Conventionally resource.verb in the past tense: order.created,
invoice.paid. The Standard Webhooks spec asks for a full-stop delimited type, with a
timestamp and a data object next to it. Treat types as a public API: renaming one breaks every
receiver that routes on it.

Payloads. A full snapshot of the resource saves the receiver a round trip; a thin
notification carrying only an id makes it fetch the current state, which is never stale.
Stripe offers both and calls them exactly that.

Subscriptions. Which endpoint wants which event types. Sending everything to everyone just
makes every receiver filter traffic it never asked for.

Signing. Anyone can POST to a public URL, so the receiver needs proof the request came from
you, unaltered. That proof is an HMAC — a hash of the message keyed with a secret only the two of
you hold. Standard Webhooks signs webhook-id, webhook-timestamp and the raw body joined by full
stops, with HMAC-SHA256, and sends the base64 result after a v1, prefix. The secret is 24 to 64
random bytes, base64-encoded, with a whsec_ prefix.

A replay window. The timestamp is signed so that a captured request cannot be sent again
tomorrow: the receiver rejects anything older than a few minutes. The spec leaves the tolerance to
you; Stripe's libraries default to five minutes, a sensible place to start.

Timeouts and status codes. The spec recommends a request timeout "somewhere between 15 and
30s". 2xx is success and anything else is retried — including redirects, which
Stripe also counts as failures. Two codes carry extra meaning:
410 Gone asks the sender to disable the endpoint, 429 Too Many Requests asks it to slow down
(MDN has the full list).

Receiving webhooks: four jobs, in order

Verify the signature on the raw bytes

Verification means recomputing the HMAC and comparing. What breaks it most often is what you
compute it over: it must be the exact bytes that arrived.

Look at the example body. It says "total": 42.50. Parse it and serialise it back and you get
{"type":"order.created",…,"total":42.5,…} — no whitespace, no trailing zero. Same data, different
bytes, a completely different HMAC. Many frameworks parse JSON before your handler runs, so the
body you are handed is already the wrong one. Stripe puts it bluntly: "Any manipulation to the raw
body of the request causes the verification to fail."

In Node with Express, ask for the raw bytes on that one route:

import crypto from 'node:crypto';
import express from 'express';

const secret = Buffer.from(process.env.WEBHOOK_SECRET.replace(/^whsec_/, ''), 'base64');
const TOLERANCE_SECONDS = 5 * 60;

// rawBody is a Buffer: the bytes that arrived, before any JSON parser saw them.
function verifyWebhook(rawBody, headers) {
  const id = headers['webhook-id'];
  const timestamp = headers['webhook-timestamp'];
  const signatures = headers['webhook-signature'];
  if (!id || !/^\d+$/.test(timestamp ?? '') || !signatures) return false;

  // The replay window: a captured request is useless five minutes later.
  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > TOLERANCE_SECONDS) return false;

  const expected = crypto
    .createHmac('sha256', secret)
    .update(`${id}.${timestamp}.`)
    .update(rawBody)
    .digest();

  // Space-separated: during a secret rotation there is one signature per secret.
  return signatures.split(' ').some((entry) => {
    const [version, value] = entry.split(',');
    if (version !== 'v1' || !value) return false;
    const received = Buffer.from(value, 'base64');
    return received.length === expected.length && crypto.timingSafeEqual(received, expected);
  });
}

const app = express();
app.post('/webhooks/orders', express.raw({ type: 'application/json' }), (req, res) => {
  if (!verifyWebhook(req.body, req.headers)) return res.sendStatus(400);
  // …record the id, enqueue, answer (below)
  res.sendStatus(200);
});

The same in Python, where Flask's request.get_data() returns the untouched bytes:

import base64
import hashlib
import hmac
import os
import time

SECRET = base64.b64decode(os.environ["WEBHOOK_SECRET"].removeprefix("whsec_"))
TOLERANCE_SECONDS = 5 * 60


def verify_webhook(raw_body: bytes, headers) -> bool:
    msg_id = headers.get("webhook-id")
    timestamp = headers.get("webhook-timestamp", "")
    signatures = headers.get("webhook-signature")
    if not (msg_id and signatures and timestamp.isascii() and timestamp.isdigit()):
        return False

    # The replay window: a captured request is useless five minutes later.
    if abs(time.time() - int(timestamp)) > TOLERANCE_SECONDS:
        return False

    signed = f"{msg_id}.{timestamp}.".encode() + raw_body
    expected = base64.b64encode(hmac.new(SECRET, signed, hashlib.sha256).digest())

    # Space-separated: during a secret rotation there is one signature per secret.
    for entry in signatures.split():
        version, _, value = entry.partition(",")
        if version == "v1" and hmac.compare_digest(value.encode(), expected):
            return True
    return False

Three details are deliberate. The comparison is constant-time (timingSafeEqual,
compare_digest), as the spec requires: a plain == returns sooner the earlier two strings
differ, and leaks the answer a byte at a time. The secret is base64-decoded after whsec_ is
stripped — using the string as-is is a classic reason a correct implementation never verifies
anything. And the header may carry several signatures, which matters the day you rotate. If you
would rather not own this code, the Standard Webhooks project publishes libraries that do exactly
this for most languages.

Answer in milliseconds, work later

The sender is holding a connection open and counting.
GitHub
gives you ten seconds, then "considers the delivery a failure". Stripe publishes no number; it
says to return 2xx "before any complex logic that could cause a timeout", and to process events
from an asynchronous queue.

So the handler does the minimum: verify, record the event id, enqueue, return 200. The email, the
warehouse call, the loyalty points — the eleven seconds from the opening — happen in a worker,
where nobody is waiting and a failure is yours to retry.

Deduplicate on the event id

A sender that retries will eventually deliver something twice. The spec says webhook-id
"remains the same no matter how many times a webhook that has failed is retried", and tells
receivers to use it as an idempotency key. Other providers have their own: X-GitHub-Delivery,
X-Shopify-Webhook-Id, the event's id at Stripe. The robust version is a unique constraint and
an insert that is allowed to do nothing:

def accept(conn, msg_id: str, raw_body: bytes) -> bool:
    """Stores the event once. Returns False if this id was already accepted."""
    cursor = conn.execute(
        "INSERT INTO received_webhooks (msg_id, body) VALUES (%s, %s) "
        "ON CONFLICT (msg_id) DO NOTHING",
        (msg_id, raw_body),
    )
    return cursor.rowcount == 1

A duplicate still gets a 200. You have it already, and an error would only make the sender try
again.

Never assume order

Unless a sender explicitly promises ordering, assume there is none. Stripe says outright that it
"doesn't guarantee the delivery of events in the order that they're generated". Failure four below
is what to do about it.

Six ways webhooks break in production

1. Your endpoint is down: who retries, and for how long?

A deploy, a crashed pod, an expired certificate. The sender gets a connection error or a 5xx,
and what happens next is entirely its policy. Good senders retry with exponential back-off — each
wait longer than the last, with random jitter so a backlog of retries does not land in the same
second. The Standard Webhooks spec's example schedule starts immediately and runs for more than
three days.

Not every sender retries. GitHub "does not automatically redeliver failed webhook deliveries"
(GitHub);
you redeliver by hand or through its API. Our
comparison of Stripe, GitHub and Shopify
puts three real policies side by side.

Takeaway: know each sender's retry schedule before the outage, and never return 2xx for an
event you have not stored — a 200 is a promise that you have it.

2. Your endpoint is slow: a timeout is a retry you asked for

A timeout looks like downtime to the sender, with one nasty difference: your handler may have
finished the work. That is the opening story. A slow endpoint converts directly into duplicate
processing, and it gets slower under load — exactly when a burst of retries arrives.

Takeaway: acknowledge inside the budget, whatever the work costs. If you do not know your
handler's 99th-percentile latency, measure it before anything else.

3. The same event arrives twice

Duplicates come from retries after a lost response, from redeliveries someone triggered by hand,
and from the sender itself. Stripe's docs say so in one sentence: "Webhook endpoints might
occasionally receive the same event more than once."

Takeaway: check the id before any side effect, in the same transaction that records the event.
A check in memory, or one after the email has gone, is not a check.

4. Events arrive out of order

An order.created whose first attempt failed can arrive after the order.updated and
order.cancelled that followed it. Apply them in arrival order and a cancelled order comes back
to life.

Compare a version or an updated_at on the resource and ignore anything older than what you
hold — or treat the webhook as a nudge and fetch the current state from the sender's API. Be wary
of the event timestamp as a tiebreaker: Stripe notes its events record created in seconds, so
distinct events can share one.

Takeaway: every handler must produce the right state whatever order its events arrive in.

5. Signatures suddenly stop verifying

This arrives as a wall of 400s, and the cause is nearly always one of five:

The body was parsed and re-serialised before verification.
The wrong secret: test instead of live, or another endpoint's.
A whsec_ secret used as a literal string instead of decoded.
A drifting server clock, so every timestamp looks stale.
A secret rotated with no overlap.

The last one is an outage you schedule yourself. If the old secret dies the instant a new one is
issued, every request between that moment and your deploy fails. Senders avoid it by signing with
both secrets for a while: the spec describes exactly this, and Stripe keeps the previous secret
valid for up to 24 hours, with one signature per active secret.

Takeaway: accept any valid v1 in the header, deploy the new secret inside the overlap, and log
why verification failed — stale timestamp, no matching signature, missing header — never a bare
400.

6. Events stop, and nobody notices

The quietest failure is the worst. The sender exhausts its retries and gives up. A subscription is
disabled after repeated failures — Shopify removes it
(Shopify). Or your own
handler catches an exception, logs it at debug and returns 200. From then on nothing errors;
events just stop, and the first symptom is a customer asking where their order went.

The defence is a delivery log on whichever side you control:

Record	Why you will want it
Event id and type	To find one event, and to prove it was a duplicate
Every attempt: time, status, latency	To tell "down" from "slow" from "rejected"
The response body, truncated	Most failures explain themselves there
Final outcome	To count what was abandoned, not just what failed once

Takeaway: alert on the failure rate and on silence. An endpoint that received a thousand
events an hour yesterday and none today is not healthy just because nothing is failing.

Testing webhooks locally

See what arrives. A request bin is a throwaway URL that records every request sent to it. Our
free webhook tester is one, keeping the latest hundred requests for 24 hours: point a
provider at it and read the real headers and body before you write a handler.

Reach your laptop. A tunnel gives localhost a public HTTPS URL. Stripe's docs suggest ngrok,
and its CLI forwards events with stripe listen --forward-to localhost:4242/webhook.

Replay real payloads. A captured body and its headers are a test fixture that is exactly what
production sends:

curl -X POST http://localhost:3000/webhooks/orders \
  -H 'Content-Type: application/json' \
  -H 'webhook-id: msg_2mQ8hZk3Xv9aR1cT' \
  -H "webhook-timestamp: $(date +%s)" \
  -H "webhook-signature: v1,$SIGNATURE" \
  --data-binary @order-created.json

--data-binary sends the file byte for byte; plain -d strips newlines and changes the
signature. A replay needs a fresh timestamp and so a fresh signature — the replay window doing its
job. When a signature will not verify, the signature verifier checks
body, secret and header in your browser without sending them anywhere.

Break it on purpose. Return 500 and watch the retries arrive. Sleep past the timeout. Send
the same payload twice. Send order.updated before order.created.

Build it yourself, or put a gateway in front?

For one sender and one receiver, build it: verification, a table of seen ids and a queue are a
day's work, and the code above is most of it.

It gets harder as the edges multiply. Sending webhooks to your own customers means storing every
event before the first attempt, a retry scheduler that survives restarts, isolation so one slow
customer does not delay the rest, a delivery log they can read, replay and secret rotation.
Receiving from several providers means several signing schemes and retry policies. None of it is
exotic, but it is a system. A gateway is that system already built — at the price of one more
component that must stay up and that holds your signing secrets.

How Railhook handles it

Railhook is an open-source webhook gateway, MIT-licensed; these are its defaults. Outgoing
webhooks are stored before the first attempt and attempted up to seven times, waiting one minute,
five minutes, fifteen minutes, one hour, six hours and twenty-four hours — about thirty-one hours
in all, each wait jittered between half and one and a half times its tier. Webhooks it relays
onward for you get a shorter ladder of five attempts. By default every delivery carries the
Standard Webhooks headers shown above, and after a secret rotation both secrets sign for a 24-hour
grace window. Every attempt is logged with its status code and response, and any stored event can
be replayed. The docs have the details.

Provider behaviour quoted above was read from each provider's own documentation on the date at the
top of this article. These policies change; follow the links before you build on a number.

Originally published on the Railhook blog. Railhook is an open-source webhook gateway — send and receive, MIT, self-host with one command: github.com/vadymkykalo/railhook.

The Programmer's Relationship With Abstraction

Tue, 22 Sep 2026 21:31:55 +0000

There is a strange moment in every programmer's life when code stops looking like code.

At the beginning, programming feels almost physical.

You write a variable.

You call a function.

You create a loop.

You store something in a database.

You send a request to an API.

You render something on a screen.

Everything is close to the machine.

Then, slowly, something changes.

You stop thinking about individual instructions and start thinking about systems.

Instead of thinking about a database query, you think about a repository.

Instead of thinking about an HTTP request, you think about a service.

Instead of thinking about individual objects, you think about domains.

Instead of thinking about files, you think about modules.

Instead of thinking about individual servers, you think about infrastructure.

And eventually, you might find yourself designing something that does not resemble the original machine-level problem at all.

This is the programmer's relationship with abstraction.

Abstraction is not simply a programming technique.

It is a way of thinking.

It is the process of deciding what deserves attention and what can temporarily disappear from our mental model.

And that makes abstraction one of the most powerful ideas in software engineering.

But abstraction is also one of the easiest things to misunderstand.

Too little abstraction can make software difficult to evolve.

Too much abstraction can make software difficult to understand.

The interesting part is finding the boundary.

That boundary changes as the system changes.

And perhaps more importantly, it changes as the programmer changes.

Programming Begins Close to Reality

A beginner often writes code that directly represents the problem.

Imagine building a simple shopping application.

You might begin with something like:

product_name = "Keyboard"
price = 50
quantity = 2

total = price * quantity

There is almost no distance between the problem and the code.

A keyboard costs 50.

The customer wants two.

The total is 100.

This is useful because the program is easy to mentally simulate.

The programmer can see almost everything.

But real software rarely remains this simple.

Soon, there are hundreds of products.

Then multiple customers.

Then shopping carts.

Then discounts.

Then taxes.

Then payment methods.

Then refunds.

Then inventory.

Then shipping.

Then notifications.

Then analytics.

The original calculation has become a small part of a much larger system.

At this point, writing everything directly becomes difficult.

We need abstraction.

We might introduce:

class Cart:
    def add_item(self, product, quantity):
        ...

Now the programmer no longer needs to think about every arithmetic operation whenever an item is added.

The concept of "adding an item to a cart" becomes an abstraction.

That abstraction gives the programmer a larger unit of thought.

This is one of the fundamental powers of software.

We can build new ideas out of previous ideas.

Abstraction Is Mental Compression

One way to understand abstraction is as mental compression.

Consider the word:

database

A programmer can say "database" without mentally visualizing every disk operation, memory page, index lookup, lock, transaction, network packet, and storage engine involved.

The word compresses an enormous amount of complexity.

The same thing happens with:

API
function
class
queue
cache
operating system
container
framework
compiler
service
event
transaction
authentication
deployment

Each concept hides a large amount of implementation detail.

This allows programmers to operate at higher levels.

Imagine if every time you used a web browser you had to understand TCP packet retransmission.

Imagine if every time you created a database record you had to understand filesystem blocks.

Imagine if every time you rendered a user interface you had to manually manage every pixel.

Modern software would be almost impossible to build.

Abstraction allows one layer of technology to become the foundation for another.

The operating system abstracts hardware.

Programming languages abstract machine instructions.

Frameworks abstract repetitive application patterns.

Libraries abstract algorithms.

APIs abstract services.

Applications abstract complicated workflows into simple interactions.

The entire software ecosystem is built from layers of abstraction.

The First Relationship: Using Abstractions

The first stage of a programmer's relationship with abstraction is learning to use it.

You discover libraries.

You discover frameworks.

You discover APIs.

You discover data structures.

You discover design patterns.

You learn that you don't have to build everything yourself.

Need authentication?

There are libraries.

Need HTTP communication?

There are clients.

Need database access?

There are drivers and ORMs.

Need image processing?

There are libraries.

Need machine learning?

There are frameworks.

The programmer begins to climb upward.

Instead of asking:

How do I implement everything?

The question becomes:

What existing abstraction can I use?

This is an important transition.

Programming becomes less about writing every possible operation and more about composing existing capabilities.

But there is a trap here.

Using abstractions without understanding them can create fragile systems.

You might know how to call an ORM method without understanding what SQL it produces.

You might know how to configure a framework without understanding the lifecycle underneath it.

You might know how to deploy a container without understanding the networking around it.

This doesn't mean you must understand every internal implementation.

That would defeat the purpose of abstraction.

But you need enough understanding to know where the abstraction ends.

The Second Relationship: Understanding Abstractions

Eventually, a programmer encounters a problem that the abstraction does not solve.

The database query becomes slow.

The framework behaves unexpectedly.

The API produces surprising results.

The cache returns stale information.

The asynchronous operation behaves differently under load.

Now the programmer has to look underneath.

This is where abstraction becomes interesting.

A good programmer needs two perspectives.

The first perspective is:

What does this abstraction allow me to do?

The second is:

What is actually happening underneath it?

These perspectives are not contradictory.

They complement each other.

You can use a high-level abstraction while still understanding the lower-level mechanism when necessary.

For example, a programmer might normally interact with a database through an ORM.

That is perfectly reasonable.

But when a performance problem appears, understanding SQL, indexes, query planning, transactions, and database storage becomes extremely useful.

The abstraction remains valuable.

The deeper knowledge simply becomes available when needed.

This is one of the healthiest relationships with abstraction.

Use the abstraction.

Understand enough of the underlying system to reason about it.

Descend when necessary.

Return to the higher level when the problem is solved.

The Abstraction Ladder

Software can be viewed as a ladder.

At the bottom are physical resources.

CPU.

Memory.

Storage.

Network interfaces.

Electrical signals.

Above that are machine instructions.

Then operating systems.

Then programming languages.

Then libraries.

Then frameworks.

Then applications.

Then business workflows.

Then user experiences.

A programmer can move up and down this ladder.

Consider a simple button click.

At the user level:

The user clicked "Buy."

At the application level:

The checkout process started.

At the service level:

An order creation request was sent.

At the API level:

A POST request was received.

At the database level:

A transaction inserted an order record.

At the operating-system level:

Processes and network resources were involved.

At the hardware level:

Instructions were executed by the processor.

All of these descriptions can refer to the same event.

The difference is abstraction level.

The programmer's skill is partly knowing which level is appropriate for the problem.

If you are designing a checkout experience, thinking about CPU instructions may be unnecessary.

If you are debugging a compiler, thinking about button colors may be irrelevant.

The right abstraction level depends on the question.

The Danger of Leaking Abstractions

Abstractions are supposed to hide complexity.

But sometimes the complexity escapes.

This is called an abstraction leak.

Imagine an application using an ORM.

The programmer thinks:

users = User.objects.filter(active=True)

The abstraction makes database access feel like ordinary application code.

But then the application becomes slow.

The programmer discovers that the query is executed thousands of times.

The ORM abstraction did not eliminate the database.

It merely provided a different interface to it.

The database still exists.

Network latency still exists.

Indexes still exist.

Transactions still exist.

Storage constraints still exist.

This is why abstraction should not be confused with elimination.

An abstraction hides complexity from ordinary interactions.

It does not make that complexity disappear.

The physical world still exists underneath the software.

Networks still have latency.

Memory is finite.

Storage has limits.

Computers still execute instructions.

Systems still fail.

Understanding this gives programmers a more realistic relationship with abstractions.

The Abstraction Tax

Every abstraction has a cost.

Sometimes that cost is computational.

Sometimes it is conceptual.

Sometimes it is organizational.

Imagine creating a simple function:

def calculate_total(price, quantity):
    return price * quantity

Then imagine turning it into:

PricingCalculationCoordinator
    -> PricingStrategyResolver
        -> PricingProviderFactory
            -> PricingCalculationService
                -> PricingEngine
                    -> PricingRuleAdapter

Technically, this might be possible.

But the abstraction has become larger than the problem.

This is where abstraction becomes counterproductive.

Every additional layer introduces another concept.

Every concept creates another thing a programmer must understand.

This is an abstraction tax.

The software may become more flexible.

But flexibility has a cost.

More interfaces.

More files.

More names.

More relationships.

More places to debug.

More mental overhead.

Good architecture is not about maximizing abstraction.

It is about creating useful boundaries.

The Abstraction Trap

Programmers sometimes discover a powerful abstraction and start applying it everywhere.

Everything becomes a pattern.

Everything becomes a service.

Everything becomes an interface.

Everything becomes configurable.

Everything becomes generic.

The code may look sophisticated.

But sophistication is not the same thing as clarity.

Suppose you have three classes that perform slightly different operations.

You could immediately create a generic framework for them.

But perhaps those three classes will never share behavior.

The abstraction might be premature.

A little duplication can sometimes be easier to understand than a complicated generalization.

This leads to an important principle:

Do not abstract merely because you can.

Abstract when the abstraction creates a meaningful improvement.

Perhaps it removes repeated complexity.

Perhaps it establishes an important boundary.

Perhaps it protects a volatile implementation.

Perhaps it creates a reusable capability.

Perhaps it allows different parts of the system to evolve independently.

The reason matters.

Abstraction and Time

One of the most interesting aspects of abstraction is that its usefulness changes over time.

An abstraction that makes sense today might not make sense tomorrow.

A small application may start with a simple database module.

Later, it becomes a large platform.

The original database boundary might no longer be appropriate.

Likewise, a complicated abstraction might eventually become unnecessary.

Technology changes.

Requirements change.

Teams change.

Traffic changes.

Products change.

Therefore, abstractions should not be treated as permanent monuments.

They are tools.

Tools can be replaced.

A programmer who treats architecture as immutable can accidentally turn yesterday's solution into today's constraint.

Good software evolves.

Good abstractions evolve with it.

Abstraction and Naming

One of the most underrated parts of abstraction is naming.

A good abstraction gives a complicated collection of operations a meaningful name.

Consider:

process_transaction()

This tells us something.

Now imagine:

execute()

The second name is technically simpler.

But it carries less information.

Good abstraction is not simply about hiding details.

It is also about communicating concepts.

Names become handles for thought.

When you say:

Authentication service

the team can discuss a concept.

When you say:

Inventory reservation

the team can reason about a workflow.

When you say:

Event processing pipeline

the team can discuss a system boundary.

The abstraction gives the team vocabulary.

This is why architecture and language are deeply connected.

Software is not only written with syntax.

It is also written with concepts.

Abstraction Creates New Possibilities

There is another side of abstraction that is even more fascinating.

Abstraction does not only hide complexity.

It creates new possibilities.

Once programmers have a function, they can compose functions.

Once they have objects, they can compose objects.

Once they have APIs, they can compose services.

Once they have distributed systems, they can build platforms.

Once they have machine learning frameworks, they can build intelligent applications without implementing every mathematical operation from scratch.

The higher layer becomes a new building block.

This is how technological ecosystems grow.

One abstraction becomes the foundation for another.

The programmer does not have to restart from zero.

They inherit an entire stack of previous ideas.

That is one of the reasons software can move so quickly.

We are standing on layers of abstractions created by previous engineers.

The Programmer as an Abstraction Designer

Eventually, a programmer stops merely consuming abstractions.

They start creating them.

A function is an abstraction.

A library is an abstraction.

An API is an abstraction.

A framework is an abstraction.

A service boundary is an abstraction.

A programming language is an abstraction.

Every reusable piece of software is, in some sense, a decision about what other programmers should not have to think about.

This creates an important responsibility.

When designing an abstraction, you are designing another programmer's future experience.

You are deciding:

What should be visible?
What should be hidden?
What should be configurable?
What should be fixed?
What should happen when something fails?
What assumptions should exist?
What should the interface communicate?
How much knowledge should users of the abstraction need?

This is software design at a deeper level.

You are not simply designing code.

You are designing the boundary through which other people will understand your code.

The Best Abstractions Feel Natural

A strong abstraction often disappears.

Think about opening a file.

You don't normally think about the physical storage mechanism.

You think:

open("notes.txt")

That is an extraordinary amount of abstraction.

Behind that small operation are operating-system calls, filesystem structures, permissions, storage devices, memory buffers, and many other mechanisms.

But the interface feels natural.

That is the magic of good abstraction.

It compresses complexity without forcing the user to constantly think about the compression.

The abstraction becomes intuitive.

It becomes part of the programmer's vocabulary.

Eventually, the programmer stops noticing it.

That is often a sign that the abstraction is doing its job.

But Sometimes We Need to Break the Abstraction

There are moments when the right engineering decision is to bypass a layer.

Perhaps the ORM is insufficient for a complicated query.

Perhaps the framework does not expose a required operating-system feature.

Perhaps a high-level networking library introduces unnecessary overhead.

Perhaps a generic data structure is not appropriate for a specialized workload.

This does not mean the abstraction was bad.

It means the problem requires a different level of control.

A mature programmer should be comfortable moving between abstraction levels.

You should be able to write high-level application logic.

Then investigate lower-level behavior.

Then return to high-level design.

The goal is not to live at the lowest level.

The goal is to understand enough levels to make good decisions.

Abstraction and Artificial Intelligence

Modern programming introduces another interesting layer of abstraction.

AI-assisted development can generate code, explain APIs, produce tests, suggest architectures, and translate ideas into implementation.

This creates another abstraction boundary.

Instead of manually writing every implementation detail, a programmer can describe an intention and receive a possible implementation.

But the fundamental relationship remains the same.

The programmer still needs to understand the abstraction.

A generated function is still an abstraction.

A generated architecture is still an abstraction.

A generated API integration still depends on real systems underneath.

The programmer's role increasingly involves understanding what the generated layers mean, how they interact, and where their assumptions live.

In that sense, the future of programming may involve even more abstraction rather than less.

The interesting question becomes:

Which details should humans think about, and which details can safely be delegated to machines?

That is an engineering question.

And it will continue evolving.

The Art of Choosing the Right Distance

Perhaps the deepest skill in abstraction is choosing distance.

Too close to the machine, and everything becomes complicated.

Too far from the machine, and important realities become invisible.

The programmer constantly adjusts this distance.

When designing a user interface:

Think in terms of users.

When designing business logic:

Think in terms of domain concepts.

When designing services:

Think in terms of boundaries and communication.

When optimizing a database:

Think in terms of queries, indexes, transactions, and storage.

When optimizing a program:

Think in terms of memory, CPU, algorithms, and data movement.

When debugging hardware interaction:

Think much closer to the machine.

The question is never simply:

What is the highest level?

The better question is:

What level helps me understand this problem?

That is the essence of abstraction.

Software Architecture Is Abstraction Management

Architecture can be viewed as the management of abstractions.

A large system contains thousands of concepts.

The architecture determines how those concepts are grouped.

Where are the boundaries?

Which components communicate?

Which details are private?

Which interfaces are public?

Which dependencies are allowed?

Which systems can change independently?

Which concepts belong together?

These decisions determine how easily programmers can reason about the system.

Good architecture creates useful mental boundaries.

You can understand one component without understanding the entire system.

That is an enormous productivity advantage.

A programmer should be able to open a module and understand what it does without mentally loading the entire universe.

Abstraction creates that possibility.

The Personal Evolution of a Programmer

The relationship with abstraction also describes the evolution of the programmer.

At first:

What does this line of code do?

Then:

What does this function do?

Then:

What does this module do?

Then:

How do these services interact?

Then:

What is the domain model?

Then:

What assumptions does this architecture make?

Eventually:

What mental model does this entire system give its developers?

The programmer's field of vision expands.

They stop seeing isolated instructions.

They begin seeing relationships.

They recognize patterns.

They notice boundaries.

They think in systems.

But experienced programmers also learn something important:

Not every problem needs a sophisticated abstraction.

Sometimes the best solution is a small function.

Sometimes it is a simple database table.

Sometimes it is a straightforward loop.

Sometimes it is a carefully designed architecture spanning multiple systems.

Engineering maturity is partly the ability to tell the difference.

Abstraction Is Not About Hiding Everything

It is tempting to define abstraction as hiding implementation details.

That is only part of the story.

Abstraction is also about exposing the right concept.

A good abstraction hides unnecessary information while preserving useful information.

For example, a payment API should not require the caller to understand every internal database operation.

But it should expose useful concepts such as:

create_payment()
get_payment()
refund_payment()

The internal details are hidden.

The important capabilities remain visible.

Therefore, abstraction is not about maximum hiding.

It is about meaningful exposure.

The interface should communicate what matters.

The Programmer's Responsibility

Every abstraction introduces assumptions.

A queue assumes something about message delivery.

A cache assumes something about data freshness.

A transaction assumes something about atomicity.

An API assumes something about communication.

A framework assumes something about application structure.

A library assumes something about its environment.

A programmer who uses abstractions responsibly learns those assumptions.

Not every detail needs to be memorized.

But the important constraints need to be understood.

This is why documentation, source code, experimentation, profiling, testing, and debugging remain valuable.

Sometimes the fastest way to understand an abstraction is to use it.

Sometimes it is to read its documentation.

Sometimes it is to inspect its implementation.

Sometimes it is to measure its behavior.

The method depends on the problem.

The Future Will Have More Layers

Software will continue becoming more abstract.

We already work with cloud infrastructure instead of individual physical machines.

We work with managed databases instead of manually maintaining storage systems.

We work with APIs instead of implementing every capability ourselves.

We work with high-level languages instead of assembly.

We work with frameworks instead of rebuilding common infrastructure.

We increasingly work with AI systems that themselves operate across layers of abstraction.

This does not make lower-level knowledge irrelevant.

It makes understanding boundaries even more important.

The more layers we build, the more carefully we need to understand how those layers connect.

A programmer may spend most of the day at a high level and occasionally descend several layers to solve one critical problem.

That ability to move is powerful.

Final Thoughts

The programmer's relationship with abstraction is not a story about moving away from complexity.

It is a story about learning how to manage complexity.

At the beginning, we see instructions.

Then functions.

Then modules.

Then systems.

Then platforms.

Then ecosystems.

Every layer gives us a new way to think.

But every layer also creates new assumptions.

The best programmers are not necessarily the ones who use the most abstractions.

They are the ones who understand why an abstraction exists, what it hides, what it exposes, where it breaks, and when it should be replaced.

Sometimes we need to go higher.

Sometimes we need to go lower.

Sometimes we need a new layer.

Sometimes we need to remove one.

Programming is the constant movement between these perspectives.

Perhaps that is what makes software engineering so interesting.

A computer ultimately executes very simple operations.

Yet humans can construct enormous conceptual worlds on top of them.

A function becomes a service.

A service becomes a platform.

A platform becomes an ecosystem.

An ecosystem becomes a new abstraction for someone else.

And then another programmer comes along and builds something even larger.

That is the relationship.

We abstract so that we can think bigger.

We understand the layers underneath so that we do not lose contact with reality.

And somewhere between those two directions is the craft of programming.

Software is built from abstractions, but engineering is the art of knowing which abstractions deserve to exist.

Building a Real-Time Supply Chain Graph

Tue, 22 Sep 2026 21:17:58 +0000

Building a Real-Time Supply Chain Graph

There is a strange thing about modern supply chains.

A product can be sitting in a warehouse in one country, waiting for a truck in another, while its raw materials are still somewhere else. A purchase order can exist in a database even though the physical goods have not moved. A truck can be halfway between two cities while the inventory system still thinks the shipment is “in transit.” A supplier can report that everything is fine while a single delayed component is quietly becoming the bottleneck for hundreds of downstream products.

The difficult part is not necessarily storing all this information.

The difficult part is understanding how everything is connected while it is changing.

That is where a real-time supply chain graph becomes interesting.

Instead of representing a supply chain as a collection of tables, dashboards, spreadsheets, and isolated applications, we can represent it as a continuously changing graph.

Suppliers become nodes.

Factories become nodes.

Warehouses become nodes.

Vehicles become nodes.

Products become nodes.

Orders become nodes.

Routes become relationships.

Inventory becomes state.

Shipments become moving state.

Dependencies become edges.

Events become changes to the graph.

Suddenly, the supply chain stops looking like a collection of transactions and starts looking like a living system.

And that is the system I want to explore.

The Problem With Traditional Supply Chain Software

Most business applications are designed around transactions.

You create a purchase order.

You receive inventory.

You create a shipment.

You update its status.

You create an invoice.

You record a delivery.

This works.

But supply chains are not fundamentally transactional systems.

They are dependency systems.

Imagine a company producing a laptop.

The final laptop might depend on:

a processor,
memory,
storage,
display,
battery,
keyboard,
motherboard,
chassis,
packaging,
logistics providers,
warehouses,
manufacturing facilities,
and dozens of upstream suppliers.

Now imagine the battery supplier experiences a delay.

The interesting question is not:

“Which shipment is delayed?”

The interesting question is:

“What does this delay affect?”

That question requires traversing relationships.

The battery affects the laptop.

The laptop affects an order.

The order affects a customer.

The customer affects a delivery commitment.

The delivery commitment affects an entire operational plan.

A conventional relational database can represent these relationships.

But a graph makes them first-class concepts.

The Supply Chain as a Graph

At its simplest, we can model a supply chain as:

Supplier
   |
   | supplies
   v
Component
   |
   | used by
   v
Factory
   |
   | produces
   v
Product
   |
   | stored at
   v
Warehouse
   |
   | shipped through
   v
Transport Route
   |
   | delivers to
   v
Customer

But a real system is considerably more complicated.

Consider:

Supplier A
    |
    v
Component X ----\
                 \
Supplier B ---> Component Y ---> Factory A ---> Product P
                                      |
                                      v
                                  Warehouse A
                                      |
                            +---------+---------+
                            |                   |
                            v                   v
                       Customer A          Customer B

Now add time.

A supplier might normally deliver Component X in three days.

Today it might take six.

That changes the graph.

Add inventory.

The factory might have enough Component X for only 18 hours.

The graph changes again.

Add transportation.

A shipment might be delayed at a distribution hub.

The graph changes again.

A real-time supply chain graph therefore isn't simply a graph of what exists.

It is a graph of what exists, where it is, what it depends on, and what is happening right now.

Designing the Core Data Model

The first architectural decision is identifying the entities.

A useful starting model might contain:

Supplier
Factory
Warehouse
Store
Customer
Product
Component
Shipment
Vehicle
PurchaseOrder
SalesOrder
Route
Location
Inventory
Event

Each entity becomes a node.

Relationships become edges.

For example:

Supplier --SUPPLIES--> Component
Component --USED_BY--> Product
Factory --PRODUCES--> Product
Product --STORED_AT--> Warehouse
Warehouse --FULFILLS--> Order
Shipment --CARRIES--> Product
Vehicle --FOLLOWS--> Route
Route --CONNECTS--> Location

But there is an important distinction.

Not every piece of information should become a node.

Some information belongs as properties.

For example:

{
  "id": "component-938",
  "name": "Battery Module",
  "quantity": 1200,
  "unit": "pieces"
}

Relationships can also carry properties.

For example:

Supplier A
    |
    | SUPPLIES
    | lead_time = 5 days
    | capacity = 20,000/month
    v
Component X

This is where graph modeling becomes powerful.

The relationship itself contains information.

Adding Time to the Graph

A static graph is useful.

A temporal graph is much more interesting.

Consider:

Warehouse A
    |
    | shipment
    v
Warehouse B

That tells us almost nothing about the actual movement.

Instead, we want:

Shipment #4812

Origin:
Warehouse A

Destination:
Warehouse B

Departure:
10:30

Expected arrival:
18:00

Current location:
Route Segment 17

Status:
In transit

Now the shipment has temporal state.

The system can continuously update:

10:30 → Departed
12:00 → Highway Segment 4
14:00 → Highway Segment 9
16:30 → Distribution Hub
18:00 → Warehouse B

The graph is changing over time.

This means we need an event-driven architecture.

Events Become the Pulse of the System

Instead of constantly asking every service:

“What is your current state?”

we can allow systems to publish events.

For example:

{
  "event": "shipment.location.updated",
  "shipment_id": "4812",
  "location": "hub-17",
  "timestamp": "2026-09-22T15:42:00Z"
}

Another event:

{
  "event": "inventory.updated",
  "warehouse_id": "WH-12",
  "product_id": "P-100",
  "quantity": 842,
  "timestamp": "2026-09-22T15:43:12Z"
}

Another:

{
  "event": "supplier.delay.reported",
  "supplier_id": "SUP-91",
  "component_id": "COMP-22",
  "expected_delay_hours": 18
}

These events become the mechanism through which the graph evolves.

A simplified architecture could look like:

                    +----------------+
                    | ERP / WMS / OMS|
                    +-------+--------+
                            |
                            v
                    +---------------+
                    | Event Gateway |
                    +-------+-------+
                            |
                            v
                    +---------------+
                    | Event Stream  |
                    +-------+-------+
                            |
             +--------------+--------------+
             |              |              |
             v              v              v
       Graph Builder   Analytics       Alerts
             |
             v
       Supply Chain Graph
             |
             v
        Graph API
             |
      +------+------+
      |             |
      v             v
   Dashboard      Applications

The graph becomes a continuously updated representation of the physical and operational world.

Why Event Sourcing Fits Naturally

A particularly interesting design is to treat events as historical truth.

Instead of simply storing:

shipment.status = delivered

we preserve the sequence:

ShipmentCreated
ShipmentDispatched
ShipmentDeparted
ShipmentLocationUpdated
ShipmentDelayed
ShipmentArrived
ShipmentDelivered

This gives us something incredibly valuable:

history.

We can ask:

Where was the shipment two hours ago?

Or:

When did the delay actually begin?

Or:

How many times did the expected delivery time change?

Or:

Which suppliers repeatedly caused downstream schedule changes?

The graph becomes more than a current-state database.

It becomes a temporal model of the supply chain.

The Real-Time Graph Engine

The graph engine is responsible for converting events into graph mutations.

Suppose this arrives:

{
  "event": "shipment.departed",
  "shipment": "SHIP-8821",
  "warehouse": "WH-1"
}

The graph engine could perform something conceptually like:

Shipment SHIP-8821
    |
    | DEPARTED_FROM
    v
Warehouse WH-1

Then a location update arrives:

{
  "event": "shipment.location.updated",
  "shipment": "SHIP-8821",
  "location": "LOC-22"
}

The graph changes:

Shipment SHIP-8821
    |
    | CURRENTLY_AT
    v
Location LOC-22

The important thing is that the graph is not manually rebuilt.

It is continuously transformed by events.

Choosing the Storage Layer

There are several ways to implement the graph.

A dedicated graph database can represent relationships naturally.

Examples include:

Neo4j
Amazon Neptune
JanusGraph
ArangoDB

A relational database can also represent graph structures through relationship tables.

For example:

nodes
-----
id
type
properties

and:

edges
-----
source_id
target_id
relationship
properties

This can work surprisingly well.

For larger architectures, however, we might separate responsibilities.

For example:

PostgreSQL
     |
     | transactional state
     v
Kafka
     |
     | events
     v
Graph Database
     |
     | relationships
     v
Redis
     |
     | hot state
     v
API

The key architectural principle is not:

“Use a graph database because graphs are cool.”

The principle is:

Store data according to the access patterns the system actually needs.

Real-Time Location Tracking

Transportation is one of the most dynamic parts of the graph.

Vehicles can emit location updates:

{
  "vehicle_id": "TRUCK-44",
  "latitude": -12.97,
  "longitude": 28.64,
  "speed": 61,
  "timestamp": "2026-09-22T16:20:00Z"
}

The system can associate that vehicle with:

Vehicle
   |
   | TRANSPORTS
   v
Shipment
   |
   | CONTAINS
   v
Product

Now the location of the vehicle can indirectly become information about the shipment.

This creates a useful concept:

state propagation.

A change in one node can affect the interpretation of other nodes.

Dependency Traversal

Suppose a factory is waiting for a component.

We can traverse:

Factory
  ↓
Product
  ↓
Component
  ↓
Supplier

Or the opposite direction:

Supplier
  ↓
Component
  ↓
Factory
  ↓
Product
  ↓
Orders
  ↓
Customers

This is where a supply chain graph becomes particularly powerful.

We can ask questions such as:

What products depend on this component?

Which factories depend on this supplier?

Which orders depend on this shipment?

Which warehouses are downstream from this distribution center?

Which customers are connected to inventory currently in transit?

These are graph queries rather than simple database lookups.

Detecting Bottlenecks

One of the most interesting applications is bottleneck detection.

Imagine:

Supplier A
   |
   v
Component X
   |
   +------> Factory A
   |
   +------> Factory B
   |
   +------> Factory C

Component X has three downstream factories.

If the supplier suddenly reduces available capacity, the graph can identify the dependency fan-out.

A node with many downstream dependencies deserves attention.

This doesn't automatically mean it is a bottleneck.

The system still needs contextual information:

inventory available
+
consumption rate
+
incoming shipments
+
supplier capacity
+
lead time
+
alternative suppliers
=
dependency risk context

The graph provides the relationships.

Analytics provides interpretation.

Inventory as a Dynamic Property

Inventory is not just a number.

It is a number changing over time.

For example:

Warehouse A
Product P
Quantity = 2,400

Then:

Sale:
-100

Receipt:
+500

Transfer:
-300

Adjustment:
-20

Current inventory:

2,480

But the graph can also represent inventory in motion:

Warehouse A
   |
   | 1,000 units
   v
Shipment 8821
   |
   v
Warehouse B

This gives us the idea of network inventory.

Instead of asking:

“How much inventory do we have?”

we can ask:

“How much inventory exists across the entire network, where is it, and how is it moving?”

That is a much richer question.

Building a Supply Chain Timeline

A graph can also generate a timeline.

Imagine a product order:

Order Created
      ↓
Inventory Reserved
      ↓
Shipment Created
      ↓
Shipment Dispatched
      ↓
Transit
      ↓
Distribution Hub
      ↓
Final Delivery

Every event gets a timestamp.

We can reconstruct the lifecycle of the order.

This makes debugging easier.

Suppose a customer asks why an order arrived late.

Instead of checking five different systems manually, the system can reconstruct:

Order #5001

08:10 — Order created
08:11 — Inventory reserved
09:00 — Shipment created
10:20 — Shipment dispatched
14:40 — Vehicle delayed
18:15 — Shipment arrived at hub
20:30 — Final delivery

The graph becomes a story.

Real-Time Alerts

The graph can feed an alerting engine.

For example:

IF

supplier delay
AND
inventory below threshold
AND
no alternative supplier
AND
downstream demand exists

THEN

create supply chain alert

Another rule:

IF

shipment delayed
AND
delivery deadline approaching

THEN

recalculate expected arrival

Another:

IF

warehouse capacity > threshold

THEN

flag capacity constraint

These aren't necessarily machine-learning problems.

Many are rules.

And that is an important engineering principle:

not every intelligent system needs artificial intelligence.

Sometimes a carefully designed graph plus deterministic rules is enough.

Adding Machine Learning

Once the graph exists, machine learning becomes more interesting.

Instead of training a model on isolated records, we can derive features from the network.

For example:

supplier reliability
route volatility
average delay
inventory velocity
dependency depth
supplier concentration
warehouse congestion
historical delivery deviation

These features can feed prediction models.

A model could estimate:

expected delivery time

or:

probability of delay

or:

expected inventory depletion

The graph supplies the context.

Machine learning supplies probabilistic inference.

This combination can become much more useful than treating every shipment as an independent record.

The Graph API

Applications need a clean interface.

We might expose:

GET /graph/products/{id}

Returning:

{
  "product": "Laptop X",
  "components": [],
  "suppliers": [],
  "factories": [],
  "warehouses": [],
  "shipments": [],
  "orders": []
}

We could also expose relationship traversal:

GET /suppliers/{id}/dependencies

Or:

GET /shipments/{id}/network-impact

Or:

GET /warehouses/{id}/incoming

Or even:

GET /graph/paths?from=supplier-1&to=customer-42

The API turns the graph into a reusable platform.

Building the Visualization

The graph becomes even more interesting visually.

Imagine a dashboard:

        Supplier A
             |
             v
       Component X
          /     \
         /       \
        v         v
   Factory A   Factory B
       |           |
       v           v
 Warehouse A   Warehouse B
       |           |
       +-----+-----+
             |
             v
         Customers

Nodes can represent entities.

Edges can represent relationships.

Node size could represent throughput.

Edge thickness could represent volume.

Animations could represent movement.

Status indicators could represent operational conditions.

A user could select a supplier and immediately see its downstream network.

Select a warehouse and see:

incoming shipments
outgoing shipments
inventory
suppliers
customers
routes
capacity

The interface becomes a map of the organization.

Handling Millions of Events

Real-time systems eventually encounter scale.

Imagine:

10,000 vehicles
500,000 shipments
millions of inventory updates
millions of orders
thousands of suppliers

The architecture needs to avoid processing everything synchronously.

Event streaming becomes important.

A possible design:

                    Producers
                       |
        +--------------+--------------+
        |              |              |
       ERP            GPS            WMS
        |              |              |
        +--------------+--------------+
                       |
                       v
                 Event Broker
                       |
          +------------+------------+
          |            |            |
          v            v            v
      Graph Writer  Analytics    Alerting
          |
          v
     Graph Storage
          |
          v
        API

Partitioning can distribute event processing.

Caching can handle frequently accessed information.

Materialized views can accelerate common queries.

Historical events can be archived.

The graph does not need to answer every query from raw events.

Consistency Becomes an Engineering Problem

Real-time systems rarely update everything simultaneously.

Suppose:

GPS says:
Vehicle is at Location A

while:

ERP says:
Shipment is still at Warehouse B

Which one is correct?

The answer may be:

both are correct within their own data domains.

The vehicle can be physically moving while the business transaction remains open.

This is why the graph should preserve:

source,
timestamp,
confidence,
event type,
update time.

For example:

{
  "location": "A",
  "source": "GPS",
  "observed_at": "...",
  "received_at": "..."
}

Now the system knows not only the value, but where that value came from.

The Importance of Provenance

A sophisticated graph should answer:

Why does the system believe this?

Suppose the dashboard says:

Expected delivery: 18:40

The user should be able to inspect the reasoning chain.

Perhaps:

GPS position
+
historical route time
+
current traffic estimate
+
vehicle speed
+
remaining distance
=
estimated arrival

This makes the system more understandable.

Instead of creating a mysterious black box, we create a system whose information can be traced back to events and relationships.

Security

A supply chain graph contains sensitive operational information.

Different users should see different parts of the graph.

A supplier might see:

their shipments
their orders
their delivery schedules

A warehouse manager might see:

warehouse inventory
incoming shipments
outgoing shipments

An executive might see:

network-level metrics

This requires authorization at the graph level.

Not every user should be able to traverse every edge.

Graph permissions can therefore become an important architectural layer.

Building the System Incrementally

A project like this can become enormous very quickly.

I would not start by trying to model the entire global supply chain.

Start with one flow.

For example:

Supplier
   ↓
Warehouse
   ↓
Shipment
   ↓
Customer

Implement:

entities,
relationships,
event ingestion,
graph updates,
real-time queries,
visualization.

Then add:

Factory

Then:

Inventory

Then:

Vehicles

Then:

Routes

Then:

Forecasting

Architecture becomes easier to evolve when the first version solves a narrow problem extremely well.

A Possible Technology Stack

A practical implementation could use:

Backend

Python / Django
or
Node.js / TypeScript

Event streaming

Kafka
Redpanda
RabbitMQ

Graph

Neo4j
Amazon Neptune
ArangoDB
JanusGraph

Relational data

PostgreSQL

Cache

Redis

Frontend

React
TypeScript
WebSockets

Visualization

Cytoscape.js
D3.js
React Flow

Infrastructure

Docker
Kubernetes
Cloud infrastructure

The specific technology is less important than the architecture.

The architecture should preserve the core idea:

Events → State → Relationships → Insights

The Interesting Part: Graph Intelligence

Once the graph becomes sufficiently rich, we can start asking deeper questions.

For example:

What happens if this supplier becomes unavailable?

The system can traverse downstream dependencies.

Supplier
 ↓
Component
 ↓
Factories
 ↓
Products
 ↓
Orders
 ↓
Customers

Another:

What inventory is currently moving toward this warehouse?

The graph can traverse incoming shipment relationships.

Another:

Which products depend on components from multiple suppliers?

That becomes a dependency query.

Another:

Which routes connect the largest number of operational nodes?

That becomes a network analysis problem.

The graph transforms supply chain software from a collection of screens into an environment for asking questions.

From Dashboard to Digital Twin

There is a larger idea hiding inside this architecture.

A sufficiently detailed real-time supply chain graph begins to resemble a digital twin of the supply network.

Not a perfect copy.

A continuously updated operational representation.

Physical world:

supplier
factory
truck
warehouse
product
customer

Digital world:

nodes
relationships
events
state
history
predictions

The physical network changes.

Events describe those changes.

The graph reflects them.

Analytics interprets them.

Applications act on them.

That creates a feedback loop:

Physical World
      ↓
Sensors / Systems
      ↓
Events
      ↓
Supply Chain Graph
      ↓
Analytics
      ↓
Decisions
      ↓
Operational Actions
      ↓
Physical World

This is where the architecture becomes genuinely interesting.

The Supply Chain Is a Living System

I think the most important shift is conceptual.

A supply chain should not be thought of only as:

orders + inventory + transportation

It can be understood as:

entities
+
relationships
+
time
+
movement
+
dependencies
+
events

That is a graph.

And because those relationships continuously change, it is a living graph.

A supplier connects to components.

Components connect to factories.

Factories connect to products.

Products connect to warehouses.

Warehouses connect to shipments.

Shipments connect to routes.

Routes connect to customers.

Every event changes something.

Every change can propagate through the network.

That is why a real-time supply chain graph is such an interesting engineering project.

The hardest part is not drawing circles and lines on a dashboard.

The hard part is creating a reliable digital model of a constantly moving system.

It requires event-driven architecture, temporal data, graph modeling, distributed systems, APIs, visualization, security, analytics, and eventually predictive intelligence.

And perhaps the most interesting outcome is not the dashboard.

It is the ability to ask the supply chain questions that traditional software makes difficult to ask:

What is connected?

What is moving?

What depends on what?

What changed?

What could this change affect?

Where is the network becoming constrained?

What happened before this event?

What is likely to happen next?

When those questions can be answered in real time, the supply chain stops being a collection of disconnected systems.

It becomes something we can observe as a whole.

A network.

A system.

A living graph.

And that is the real project: not building another logistics dashboard, but building a continuously evolving map of how products, people, infrastructure, information, and decisions move through an interconnected world.

Building a Global Delivery Prediction System

Mon, 21 Sep 2026 17:20:05 +0000

There is a strange moment in every delivery.

You place an order.

The system says “Arriving Tuesday.”

Tuesday arrives.

Nothing.

Then the status changes.

“Delayed.”

A few hours later:

“Out for delivery.”

Then:

“Delivered.”

The package appears at your door.

From the customer's perspective, delivery prediction looks simple. There is a destination, a package, a courier, and a date.

But underneath that simple interface is a remarkably complex system.

A delivery does not move through a single machine.

It moves through a network.

A warehouse receives an order. A worker picks an item. Another system packages it. A truck collects it. A sorting facility processes it. A flight may move it across countries. Customs may inspect it. Another warehouse receives it. A local courier takes responsibility. Traffic affects the final journey. Weather can change transportation conditions. Operational capacity can change. Addresses can introduce additional uncertainty.

The package is effectively moving through a global graph of events.

And if we want to predict when it will arrive, we should not simply ask:

“How long does shipping usually take?”

We should ask:

“Given everything that has happened to this package, everything currently happening in the network, and everything likely to happen next, when is this package most likely to arrive?”

That is a much more interesting engineering problem.

This is the idea behind a Global Delivery Prediction System.

It is not merely a tracking dashboard.

It is a prediction engine that continuously transforms logistics events into an evolving estimate of the future.

1. The Real Problem Is Time

Suppose a customer places an order at 10:00 AM.

The delivery pipeline might look like this:

Order Created
     ↓
Payment Confirmed
     ↓
Warehouse Processing
     ↓
Item Picked
     ↓
Package Packed
     ↓
Courier Pickup
     ↓
Origin Hub
     ↓
Sorting Facility
     ↓
International Transit
     ↓
Destination Customs
     ↓
Local Hub
     ↓
Last-Mile Courier
     ↓
Delivered

The customer does not necessarily care about every internal event.

They care about one question:

When will I receive it?

That means the prediction system needs to compress a complicated operational process into a useful temporal estimate.

For example:

Predicted delivery:
September 25

Confidence:
82%

Likely window:
14:00–18:00

The important part is that this prediction should not be static.

If the package is delayed at customs, the prediction should change.

If it reaches the destination hub earlier than expected, the prediction should change.

If a courier scans the package as loaded onto a delivery vehicle, the prediction should change again.

The prediction is therefore a living value.

2. Think of Delivery as a Graph

One of the most useful architectural decisions is to model logistics as a graph.

A simplified graph could look like:

                    ┌──────────────┐
                    │   Warehouse  │
                    └──────┬───────┘
                           │
                           ▼
                    ┌──────────────┐
                    │ Origin Hub   │
                    └──────┬───────┘
                           │
               ┌───────────┴───────────┐
               ▼                       ▼
        ┌──────────────┐        ┌──────────────┐
        │ Airport Hub  │        │ Transit Hub  │
        └──────┬───────┘        └──────┬───────┘
               │                       │
               └───────────┬───────────┘
                           ▼
                    ┌──────────────┐
                    │ Destination  │
                    │ Hub          │
                    └──────┬───────┘
                           │
                           ▼
                    ┌──────────────┐
                    │ Last Mile    │
                    └──────┬───────┘
                           │
                           ▼
                       Customer

Each node represents a location or operational stage.

Each edge represents a possible transition.

We can assign properties to both.

A node might have:

processing_time
queue_size
operational_capacity
operating_hours
historical_delay_rate

An edge might have:

travel_time
distance
transport_mode
historical_variance
departure_frequency

Now delivery prediction becomes a graph problem.

The package is currently at node A.

The destination is node Z.

The system needs to estimate the remaining journey.

3. Events Are More Important Than Statuses

Traditional tracking systems often revolve around statuses:

PROCESSING
SHIPPED
IN_TRANSIT
OUT_FOR_DELIVERY
DELIVERED

These are useful for humans.

But a prediction engine needs richer information.

Instead of simply storing:

{
  "status": "IN_TRANSIT"
}

we might store:

{
  "event": "ARRIVED_AT_HUB",
  "timestamp": "2026-09-21T09:42:00Z",
  "location": "HUB-042",
  "carrier": "carrier_a",
  "package_id": "PKG-19384"
}

Then:

{
  "event": "DEPARTED_HUB",
  "timestamp": "2026-09-21T12:15:00Z",
  "location": "HUB-042",
  "carrier": "carrier_a",
  "package_id": "PKG-19384"
}

The difference is enormous.

The prediction engine can now reconstruct the journey.

It can calculate:

time spent at hub
time between scans
time between facilities
transport duration
processing duration

The system is no longer observing a status.

It is observing behavior.

4. Build the Event Ingestion Layer

A global logistics network can generate enormous numbers of events.

Events may arrive from:

courier APIs
warehouse systems
ERP systems
transportation systems
customs systems
mobile applications
barcode scanners
IoT devices
partner platforms

The architecture therefore needs an ingestion layer.

Courier APIs
     │
Warehouse Systems
     │
Partner Systems
     │
Mobile Devices
     │
IoT Devices
     │
     ▼
┌─────────────────────┐
│ Event Ingestion API │
└──────────┬──────────┘
           ▼
     Event Stream
           ▼
   Normalization Layer

The normalization layer is particularly important.

Different carriers may describe the same event differently.

One carrier might send:

ARRIVED

Another:

RECEIVED_AT_FACILITY

Another:

FACILITY_SCAN

The internal system should normalize these into a common event vocabulary.

For example:

ARRIVED_AT_FACILITY
DEPARTED_FACILITY
CUSTOMS_STARTED
CUSTOMS_COMPLETED
OUT_FOR_DELIVERY
DELIVERED

Now the prediction engine can work across carriers.

5. Create a Universal Shipment Model

A global system needs a common representation of a shipment.

For example:

class Shipment:
    id
    origin
    destination
    carrier
    service_level
    package_type
    weight
    dimensions
    created_at
    promised_at
    current_location
    current_status

But we should not stop there.

The shipment should also maintain its event history.

class ShipmentEvent:
    shipment_id
    event_type
    timestamp
    location
    facility
    carrier
    metadata

And the system should maintain the route.

class RouteSegment:
    origin_node
    destination_node
    transport_mode
    expected_duration
    duration_variance

This gives us three important dimensions:

Shipment
   +
Events
   +
Route

Together they provide the foundation for prediction.

6. Prediction Is a Sequence of Predictions

A common mistake is trying to predict the final delivery date directly.

Instead, predict each stage.

Suppose a package has four remaining stages:

Current Hub
   ↓
Regional Hub
   ↓
Destination Facility
   ↓
Last Mile
   ↓
Customer

We can estimate:

Current Hub → Regional Hub
2.1 hours

Regional Hub → Destination Facility
5.4 hours

Destination Facility → Last Mile
3.2 hours

Last Mile → Customer
4.0 hours

Total:

14.7 hours

But each stage has uncertainty.

So instead of:

14.7 hours

we might calculate:

P50: 14.7 hours
P80: 18.3 hours
P95: 26.1 hours

This is much more useful.

The system can translate those probabilities into customer-facing windows.

7. The Prediction Engine

A simple first version can use historical averages.

For every route segment:

average_duration(origin, destination)

But averages are rarely enough.

Consider two packages traveling between the same locations.

Package A:

Express
Small
Already sorted
Departing immediately

Package B:

Economy
Large
Awaiting sorting
High facility queue

Their expected delivery times are different.

So the model needs features.

For example:

distance
package_weight
package_dimensions
carrier
service_level
origin
destination
current_facility
day_of_week
hour_of_day
historical_route_duration
facility_processing_time
current_queue_size
number_of_previous_scans
time_since_last_event

The model can then estimate:

remaining_delivery_time

8. Start Simple Before Using Machine Learning

You do not need a sophisticated neural network on day one.

A strong baseline could be:

Predicted Time =
Historical Segment Time
+
Current Queue Time
+
Known Processing Time
+
Route Adjustment

For example:

prediction = (
    route_average
    + facility_queue_estimate
    + processing_estimate
    + operational_adjustment
)

This baseline gives the engineering team something measurable.

Then machine learning can improve it.

Possible models include:

gradient boosting
random forests
regression models
survival models
temporal models
sequence models
neural networks

The important principle is:

Machine learning should improve a well-understood system rather than replace understanding of the system.

9. Model Time as a Distribution

One of the most important improvements is moving away from a single prediction.

Imagine the system predicts:

September 25

That sounds precise.

But reality is not precise.

Instead:

September 25
60% probability

September 26
25% probability

September 27
10% probability

September 28+
5% probability

Now the system can produce a delivery window.

For example:

Expected delivery:
September 25–26

Confidence:
High

This is more honest and operationally useful than pretending the future is deterministic.

10. Global Means Time Zones

A global delivery system cannot treat time as a simple string.

Consider:

Lusaka
London
Dubai
New York
Tokyo

Every location has different local time.

Events should therefore be stored using a canonical representation such as UTC.

2026-09-21T16:00:00Z

Then converted into local time for presentation.

This prevents errors involving:

daylight saving changes
cross-border shipments
overnight transport
international handoffs
operating hours

Time zones become part of the logistics model.

11. Operating Hours Matter

A package arriving at a facility at 23:50 does not necessarily get processed immediately.

The facility might operate:

06:00–22:00

So the package effectively waits.

A prediction engine should understand this.

For example:

if arrival_time > facility.close_time:
    processing_start = next_operating_period

This simple rule can significantly improve predictions.

The same concept applies to:

warehouses
customs offices
sorting centers
local courier branches
pickup points

Operational calendars are therefore valuable prediction features.

12. Geography Is More Than Distance

Two destinations can be equally distant but operationally very different.

A 300-kilometer journey might take:

5 hours

in one environment and:

12 hours

in another.

Distance alone does not determine delivery time.

The model can incorporate:

historical travel time
route reliability
transport mode
facility density
handoff frequency
delivery zone

Geographic information becomes useful when combined with operational history.

13. The Last Mile Deserves Its Own Model

The final stage is often different from the rest of the journey.

International transportation might be predictable at a broad level.

Last-mile delivery can be much more granular.

The package may move from:

Destination Hub
      ↓
Courier Vehicle
      ↓
Delivery Zone
      ↓
Street
      ↓
Customer

The system might need features such as:

delivery zone
route density
number of stops
estimated route duration
vehicle capacity
historical stop duration
delivery attempt history

The final prediction can therefore be calculated separately.

Global Transit Model
        +
Regional Model
        +
Last-Mile Model
        =
Final Delivery Prediction

This modular architecture is easier to improve.

14. Detect Delays Before They Happen

A truly interesting delivery system should not only react to delays.

It should detect delay risk.

Suppose a package has been sitting at a facility longer than packages normally remain there.

The system could calculate:

Expected processing time: 3 hours
Current processing time: 5.5 hours
Delay risk: elevated

This does not necessarily mean the package is delayed.

It means the package is behaving differently from the historical pattern.

That distinction matters.

The system can generate an internal event:

DELAY_RISK_DETECTED

Then downstream systems can react.

15. Build a Delivery Digital Twin

One of the more advanced directions is to create a digital representation of the delivery network.

Imagine every shipment having a continuously updated state:

Shipment
├── Current Location
├── Current Facility
├── Route
├── Historical Events
├── Predicted Next Event
├── Expected Delivery
├── Confidence
└── Risk Factors

And every facility has its own state:

Facility
├── Current Queue
├── Processing Capacity
├── Active Shipments
├── Average Processing Time
├── Delay Risk
└── Operating Schedule

The entire network becomes a living model.

The prediction engine can then reason over the current state of the network.

16. Architecture

A production architecture might look like this:

                  ┌──────────────────┐
                  │ Customer Apps    │
                  └────────┬─────────┘
                           │
                           ▼
                  ┌──────────────────┐
                  │ Delivery API     │
                  └────────┬─────────┘
                           │
          ┌────────────────┴────────────────┐
          ▼                                 ▼
┌──────────────────┐              ┌──────────────────┐
│ Shipment Service │              │ Prediction API   │
└────────┬─────────┘              └────────┬─────────┘
         │                                 │
         ▼                                 ▼
┌──────────────────┐              ┌──────────────────┐
│ Event Store      │◄─────────────│ Prediction Engine│
└────────┬─────────┘              └────────┬─────────┘
         │                                 │
         ▼                                 ▼
┌──────────────────┐              ┌──────────────────┐
│ Event Stream     │─────────────►│ Feature Store    │
└────────┬─────────┘              └────────┬─────────┘
         │                                 │
         ▼                                 ▼
┌──────────────────┐              ┌──────────────────┐
│ Route Graph      │              │ ML Models        │
└──────────────────┘              └──────────────────┘

This separates responsibilities.

The event system records reality.

The graph represents the network.

The prediction engine estimates the future.

17. Use Event-Driven Architecture

Delivery networks are naturally event-driven.

A package generates events continuously.

That makes an event stream architecture particularly appropriate.

For example:

ShipmentEvent
      ↓
Message Broker
      ↓
 ┌────┼───────────┐
 ▼    ▼           ▼
Risk  Prediction  Analytics

A single event can trigger multiple workflows.

For example:

ARRIVED_AT_DESTINATION_HUB

could trigger:

Update shipment state
Recalculate ETA
Update customer application
Check delay risk
Update analytics

This avoids tightly coupling every service.

18. Store Predictions Separately

A useful design is to treat predictions as first-class data.

For example:

prediction_id
shipment_id
created_at
predicted_delivery_time
lower_bound
upper_bound
confidence
model_version
feature_snapshot

Why store them?

Because later we can compare:

Prediction
vs
Reality

Suppose the system predicted:

September 25, 15:00

and actual delivery was:

September 25, 14:20

We can calculate the error.

Over thousands or millions of shipments, this becomes a powerful feedback loop.

19. Measure the Model

A prediction system needs more than model accuracy.

Useful metrics include:

Mean Absolute Error

MAE = average(|prediction - actual|)

This tells us the average absolute error.

Median Absolute Error

Useful when extreme cases exist.

Prediction Interval Coverage

If we claim that:

90% of deliveries should happen within the predicted interval

we can measure how often that actually happens.

Calibration

If the system says:

80% confidence

we want events with that confidence level to behave approximately as expected over time.

The goal is not simply to produce predictions.

It is to produce trustworthy predictions.

20. Prediction Versioning

Models change.

Suppose:

Model v1

is replaced with:

Model v2

We should preserve the model version associated with every prediction.

For example:

{
  "prediction": "2026-09-25T15:00:00Z",
  "model_version": "delivery-v2.4",
  "generated_at": "2026-09-21T17:30:00Z"
}

This makes debugging possible.

If predictions suddenly become less accurate, engineers can investigate which model version introduced the change.

21. Feedback Loops Make the System Better

Every completed delivery creates new training data.

The cycle becomes:

Observe
   ↓
Predict
   ↓
Deliver
   ↓
Compare Prediction With Reality
   ↓
Measure Error
   ↓
Improve Model
   ↓
Predict Again

This is one of the most beautiful aspects of the architecture.

The logistics network continuously generates data that can improve the prediction system.

The system becomes an observer of its own historical performance.

22. Handling Missing Data

Real logistics data is messy.

A package may not receive a scan.

A carrier API may temporarily fail.

A timestamp may be missing.

A location may be approximate.

A system should not collapse because one event is missing.

Instead, the prediction engine should estimate the current state from available evidence.

For example:

Last known event:
Departed Hub A

Elapsed time:
6 hours

Expected travel duration:
4–7 hours

The model may infer that the package is probably somewhere along the next route segment.

This is fundamentally different from claiming an exact location.

The system maintains uncertainty.

23. Confidence Should Be Dynamic

Confidence should not be a permanent property of a shipment.

When the package has just entered a long international route:

Confidence: 58%

When it reaches the destination hub:

Confidence: 82%

When it is loaded onto the final delivery vehicle:

Confidence: 94%

The closer the shipment gets to the destination, the more information becomes available.

Therefore:

More observations
        ↓
Less uncertainty
        ↓
Higher confidence

This is a natural property of the system.

24. Build an ETA API

The customer-facing API could be simple.

GET /shipments/{id}/eta

Response:

{
  "shipment_id": "PKG-19384",
  "estimated_delivery": "2026-09-25T15:00:00Z",
  "window": {
    "from": "2026-09-25T13:00:00Z",
    "to": "2026-09-25T18:00:00Z"
  },
  "confidence": 0.86,
  "last_updated": "2026-09-21T17:45:00Z"
}

The frontend can turn this into something simple:

Expected Friday afternoon

The complexity remains inside the system.

That is good architecture.

25. Notifications Should Be Event-Driven

The prediction system can also power proactive notifications.

For example:

ETA changed

or:

Delivery window moved earlier

or:

Shipment requires additional processing

The important idea is that notifications should be triggered by meaningful changes.

If the prediction changes by two minutes, notifying the customer is probably unnecessary.

A threshold can be used:

if abs(new_eta - old_eta) > notification_threshold:
    notify_customer()

This reduces notification noise.

26. Global Scale Changes Everything

A local delivery system might process thousands of shipments.

A global platform could process millions.

That changes architectural requirements.

The system needs:

horizontal scaling
partitioned event streams
distributed storage
caching
idempotent consumers
fault tolerance
observability
backpressure

Shipment events can be partitioned by:

shipment_id

This keeps events for a shipment ordered within a partition.

Prediction workloads can also be distributed.

Instead of recalculating every shipment after every event, only affected shipments need to be recalculated.

That saves enormous computational resources.

27. Cache Predictions Carefully

Predictions are dynamic, but not every request needs a full model execution.

We can cache:

shipment_id → latest_prediction

When a significant event occurs:

invalidate cache
recalculate prediction
store new prediction

Customer requests then become inexpensive.

The architecture becomes:

Event
 ↓
Prediction Recalculation
 ↓
Prediction Cache
 ↓
Customer API

This creates a fast user experience without unnecessarily running the model on every API request.

28. Observability Is Part of Prediction

Imagine customers report that delivery estimates suddenly became inaccurate.

The engineering team needs to know:

Did the model change?
Did a carrier stop sending events?
Did a facility become slower?
Did route data disappear?
Did event processing lag?

Therefore the system should expose metrics such as:

event_ingestion_latency
prediction_latency
prediction_error
missing_event_rate
carrier_event_rate
facility_processing_time
model_accuracy

Tracing is also valuable.

A single shipment can be followed across:

API
→ Event Service
→ Stream
→ Feature Service
→ Prediction Engine
→ Cache

That makes complex failures much easier to investigate.

29. Security and Data Protection

A global delivery system handles valuable information.

Potential data includes:

names
addresses
phone numbers
order identifiers
delivery locations
shipment histories

The architecture should therefore include:

authentication
authorization
encryption
access controls
audit logging
data minimization
secure API design
retention policies

Prediction systems should only expose information required for the requested operation.

A customer should not automatically gain access to internal operational data simply because the prediction engine has access to it.

30. The Interesting Part Is the Future

A normal tracking system tells you:

What happened.

A prediction system tries to tell you:

What is likely to happen next.

That distinction changes the entire architecture.

Tracking is primarily observational.

Prediction is inferential.

Tracking asks:

Where is the package?

Prediction asks:

Given where it is, what has happened before, what is happening now, and what does the remaining network look like, when is it likely to arrive?

That requires combining:

events
+
graphs
+
time
+
geography
+
operations
+
statistics
+
machine learning

And that is why delivery prediction is such an interesting systems problem.

31. A Possible Technology Stack

A practical implementation could use:

Backend

Python
FastAPI
PostgreSQL
Redis

Event Infrastructure

Kafka
Redpanda
RabbitMQ

Data Processing

Python
Spark
Flink

Machine Learning

PyTorch
scikit-learn
XGBoost

Storage

PostgreSQL
ClickHouse
Object Storage
Redis

APIs

REST
GraphQL
WebSockets

Observability

Prometheus
Grafana
OpenTelemetry

The exact technologies can change.

The architecture matters more than the brand names.

32. Build It Incrementally

I would not start by building the entire global prediction platform.

Start with a narrow version.

Phase 1

Build:

Shipment API
Event ingestion
Event storage
Basic ETA calculation

Phase 2

Add:

Route graph
Historical route statistics
Facility processing times

Phase 3

Add:

Machine learning
Prediction intervals
Confidence scores

Phase 4

Add:

Delay detection
Dynamic recalculation
Last-mile prediction

Phase 5

Add:

Multi-carrier support
Global time zones
Large-scale streaming
Advanced analytics

This creates a system that evolves naturally.

33. The Bigger Idea

The most interesting part of this project is not predicting whether a package arrives tomorrow.

It is building a system that understands movement through a global network.

The package is simply the object being observed.

The real system is the network.

Warehouses.

Vehicles.

Airports.

Sorting centers.

Customs processes.

Delivery routes.

Operating schedules.

Customers.

Events.

Every movement creates information.

Every event changes the state of the network.

Every state change changes the probability of future events.

This gives us a powerful abstraction:

Current State
      ↓
Possible Future States
      ↓
Probability Distribution
      ↓
Predicted Outcome

That is bigger than package tracking.

It is a general architecture for predicting movement through complex systems.

34. Final Architecture

At its most complete form, our system looks like this:

                  GLOBAL LOGISTICS NETWORK
                           │
                           ▼
                  ┌─────────────────┐
                  │ Event Sources   │
                  └────────┬────────┘
                           ▼
                  ┌─────────────────┐
                  │ Event Ingestion │
                  └────────┬────────┘
                           ▼
                  ┌─────────────────┐
                  │ Event Stream    │
                  └────────┬────────┘
                           ▼
              ┌──────────────────────────┐
              │ Normalization + State    │
              │ Reconstruction           │
              └────────────┬─────────────┘
                           ▼
                  ┌─────────────────┐
                  │ Logistics Graph │
                  └────────┬────────┘
                           ▼
             ┌──────────────────────────┐
             │ Feature Generation       │
             └────────────┬─────────────┘
                          ▼
             ┌──────────────────────────┐
             │ Prediction Engine       │
             └────────────┬─────────────┘
                          ▼
             ┌──────────────────────────┐
             │ ETA + Confidence + Risk │
             └────────────┬─────────────┘
                          ▼
               ┌─────────────────────┐
               │ Delivery API        │
               └──────────┬──────────┘
                          ▼
             Customers / Businesses

And underneath it all is a feedback loop:

Prediction
    ↓
Actual Delivery
    ↓
Prediction Error
    ↓
Model Evaluation
    ↓
Model Improvement
    ↓
Better Prediction

That loop is what turns a static logistics application into an intelligent system.

Conclusion

Building a Global Delivery Prediction System is ultimately an exercise in understanding uncertainty.

A package does not travel through a perfectly deterministic pipeline.

It moves through a changing network.

Facilities become busy.

Routes take different amounts of time.

Events arrive at different intervals.

Operating schedules introduce waiting periods.

The final delivery stage behaves differently from international transportation.

And every new event changes what we know about the future.

The engineering challenge is therefore not to create a magical system that always knows exactly when something will happen.

The challenge is to create a system that continuously updates its understanding of what is likely to happen next.

That requires a combination of event-driven architecture, logistics graphs, historical data, operational modeling, machine learning, probabilistic prediction, and strong software engineering.

The most important architectural idea is simple:

Do not treat delivery as a status. Treat it as a changing state moving through a network.

Once you make that mental shift, the project becomes much more interesting.

You stop building another tracking dashboard.

You start building a computational model of movement.

And that model can answer one of the most useful questions in logistics:

“Given everything we know right now, what happens next?”

That is where delivery prediction becomes real engineering.

And that is where a package stops being just a package.

It becomes a stream of data moving through a living global system.

Building a Universal Logistics Graph

Mon, 21 Sep 2026 17:12:04 +0000

Modern life is built on movement.

A package moves from a warehouse to a distribution center.

A truck moves inventory from a supplier to a retailer.

A customer moves through a delivery process without seeing the hundreds of decisions happening behind the scenes.

A shipment crosses borders.

A container changes vehicles.

A product moves from a factory to a warehouse, from a warehouse to a store, and from a store to someone's home.

We usually represent these processes as lists.

Orders.

Shipments.

Tracking numbers.

Invoices.

Addresses.

Vehicles.

Warehouses.

Drivers.

Delivery events.

But logistics is not really a list.

Logistics is a graph.

Every shipment is connected to something.

A package belongs to an order.

An order contains products.

Those products originate from suppliers.

A shipment travels through locations.

Locations are connected by routes.

Routes depend on vehicles.

Vehicles are operated by people or organizations.

A delivery creates an event.

An event changes the state of an object.

An object may then trigger another process.

Once we see logistics this way, an interesting engineering problem appears:

What if we built one universal graph capable of representing the relationships between everything involved in moving goods, documents, resources, and people through a logistics system?

That is the idea behind a Universal Logistics Graph.

It is not simply another tracking dashboard.

It is not just a database for shipments.

It is a computational model of movement.

And the interesting part is that the same architecture can support e-commerce, manufacturing, retail, agriculture, transportation, warehousing, field operations, procurement, and many other systems.

The Problem With Traditional Logistics Software

Many logistics systems are organized around individual applications.

One system manages orders.

Another manages warehouses.

Another manages transportation.

Another manages inventory.

Another handles invoices.

Another manages customers.

Another handles tracking.

Each system may work perfectly well on its own.

The problem appears at the boundaries.

Imagine this sequence:

A customer places an order.

The order contains three products.

One product is available locally.

Another must come from a regional warehouse.

The third is still being manufactured.

The system therefore creates multiple fulfillment operations.

One warehouse prepares a package.

A supplier prepares another.

A manufacturing facility schedules production.

Several packages eventually converge into a delivery process.

A traditional system may represent these as separate records.

A graph represents them as connected entities.

That difference is important.

Instead of asking:

"Where is shipment #18492?"

we can ask:

"What is the current state of everything connected to shipment #18492?"

That could include:

its order,
its customer,
its products,
its warehouse,
its previous locations,
its current location,
its assigned vehicle,
its delivery route,
its dependencies,
its customs documents,
its inventory reservations,
its expected next event,
and other connected shipments.

The graph turns isolated records into context.

What Is a Logistics Graph?

At its simplest, a graph contains:

Nodes + edges + properties + events.

A node represents something.

An edge represents a relationship.

A property describes an entity.

An event describes something that happened.

For example:

Customer
   |
places
   ↓
Order
   |
contains
   ↓
Product
   |
stored_at
   ↓
Warehouse
   |
dispatched_by
   ↓
Vehicle
   |
travels
   ↓
Route
   |
arrives_at
   ↓
Delivery Location

This structure is powerful because logistics is naturally relational.

The graph can contain different types of nodes:

Customer
Supplier
Manufacturer
Product
SKU
Order
Shipment
Package
Warehouse
Store
Vehicle
Driver
Route
Location
Container
Invoice
Document
Delivery
Event

Edges can describe relationships:

CONTAINS
BELONGS_TO
STORED_AT
SHIPPED_FROM
SHIPPED_TO
ASSIGNED_TO
OPERATED_BY
TRAVELS_THROUGH
DEPENDS_ON
FULFILLS
GENERATED_BY
REQUIRES
DELIVERED_TO

Now the system has a language for logistics.

The Core Architecture

A Universal Logistics Graph can be designed as several layers.

                ┌──────────────────────┐
                │     Applications     │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │   Query / API Layer  │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Logistics Graph     │
                │                      │
                │ Nodes + Relationships│
                └──────────┬───────────┘
                           │
              ┌────────────┼────────────┐
              │            │            │
        ┌─────▼─────┐ ┌────▼────┐ ┌────▼────┐
        │ Event Bus  │ │ Storage │ │ Rules   │
        └───────────┘ └─────────┘ └─────────┘

The graph becomes the central representation.

Applications interact with the graph rather than independently rebuilding their own understanding of logistics.

1. The Entity Layer

The first layer defines what exists.

Consider a shipment.

{
  "id": "shipment_18492",
  "type": "shipment",
  "status": "in_transit"
}

A warehouse might look like:

{
  "id": "warehouse_102",
  "type": "warehouse",
  "capacity": 12000
}

A vehicle:

{
  "id": "vehicle_77",
  "type": "truck",
  "capacity": 1800
}

But entities alone are not enough.

The important information is how they connect.

2. The Relationship Layer

Relationships become first-class objects.

For example:

shipment_18492
      |
assigned_to
      |
vehicle_77

Or:

shipment_18492
      |
origin
      |
warehouse_102

A relationship can have properties too.

{
  "from": "shipment_18492",
  "to": "vehicle_77",
  "type": "assigned_to",
  "assigned_at": "2026-09-21T08:00:00Z"
}

This matters because relationships often contain valuable information.

A vehicle may have been assigned to a shipment at a particular time.

A shipment may have passed through a location.

A product may have been allocated to an order.

A warehouse may have reserved inventory for a fulfillment operation.

The relationship itself has history.

3. The Event Layer

A logistics graph should not only represent the current state.

It should understand how that state happened.

Consider:

Shipment Created
       ↓
Package Prepared
       ↓
Warehouse Dispatched
       ↓
Vehicle Assigned
       ↓
Checkpoint Reached
       ↓
Destination Arrived
       ↓
Delivered

Each event becomes part of the graph.

For example:

{
  "id": "event_991",
  "type": "shipment_dispatched",
  "entity": "shipment_18492",
  "location": "warehouse_102",
  "timestamp": "2026-09-21T09:12:00Z"
}

Now the system can reconstruct the history.

This creates a temporal dimension.

The graph is no longer simply:

What is connected to what?

It becomes:

What was connected to what, when, where, and because of which event?

That is a much more useful model.

The Graph Should Understand Time

Logistics is deeply temporal.

A shipment has a current location.

But it also has previous locations.

A vehicle has a current assignment.

But it had previous assignments.

Inventory has a current quantity.

But quantities changed because of events.

Therefore, relationships should be time-aware.

Instead of:

Shipment → Warehouse

we can represent:

Shipment
   |
   ├── Warehouse A [08:00–10:00]
   |
   ├── Truck 17    [10:20–15:40]
   |
   └── Warehouse B [16:00–17:30]

This enables historical queries.

For example:

Where was this shipment yesterday?

Which vehicle carried it?

Which facilities handled it?

How long did it spend at each facility?

The graph becomes a logistics timeline.

Building the Physical Network

The next major challenge is representing geography.

A logistics system is a physical network.

Locations are not merely addresses.

They are nodes in a transportation topology.

Factory
   |
   ↓
Distribution Center
   |
   ├──── Route A ──── Store
   |
   └──── Route B ──── Regional Warehouse

A location node might contain:

{
  "id": "loc_55",
  "type": "warehouse",
  "country": "Zambia",
  "city": "Ndola"
}

Routes connect locations.

{
  "from": "loc_55",
  "to": "loc_77",
  "type": "route",
  "distance_km": 320
}

The graph can then represent an entire transportation network.

Logistics Is a Multi-Graph Problem

One of the most interesting aspects of this project is that there is not just one graph.

There are several graphs overlapping each other.

Physical graph

Where things physically move.

Factory → Warehouse → Store

Ownership graph

Who owns what.

Supplier → Product → Customer

Transaction graph

What commercial operations created movement.

Customer → Order → Payment → Fulfillment

Inventory graph

Where resources are stored.

Warehouse → Inventory → SKU

Dependency graph

What depends on what.

Product → Component → Supplier

Event graph

What happened and when.

Order → Event → Shipment → Event → Delivery

The Universal Logistics Graph combines these dimensions.

That is where the architecture becomes interesting.

A Unified Data Model

A practical graph schema might begin with:

Entity
 ├── id
 ├── type
 ├── attributes
 ├── created_at
 └── updated_at

Relationships:

Relationship
 ├── id
 ├── source_id
 ├── target_id
 ├── type
 ├── attributes
 ├── valid_from
 └── valid_until

Events:

Event
 ├── id
 ├── type
 ├── entity_id
 ├── location_id
 ├── timestamp
 ├── metadata
 └── source

This provides a generic foundation.

Instead of building a special database structure for every logistics scenario, the graph can accommodate new entity types and relationships.

The Universal Event Model

One of the most useful design decisions is standardizing events.

Instead of every subsystem producing different formats:

shipment.updated
package.moved
truck.arrived
warehouse.received
delivery.completed

we can normalize events around a common structure.

{
  "event_id": "evt_1001",
  "event_type": "entity.moved",
  "entity_id": "package_90",
  "source": "warehouse_system",
  "timestamp": "2026-09-21T10:30:00Z",
  "location": "warehouse_10",
  "metadata": {}
}

Now multiple systems can feed the graph.

A warehouse management system can publish events.

A transportation system can publish events.

A mobile application can publish events.

An IoT device can publish events.

The graph consumes them.

Ingestion Architecture

Real logistics systems produce data from many sources.

For example:

ERP
 │
 ├── Orders
 │
WMS
 │
 ├── Inventory
 │
TMS
 │
 ├── Transportation
 │
Mobile Apps
 │
 ├── Driver Events
 │
IoT
 │
 ├── Sensor Events
 │
External APIs
 │
 └── Tracking

A normalization layer sits between those systems and the graph.

External Systems
       ↓
   Connectors
       ↓
Event Normalizer
       ↓
Validation
       ↓
Event Bus
       ↓
Graph Processor
       ↓
Universal Logistics Graph

This prevents every integration from directly modifying the graph.

Event Processing

A message broker can handle incoming events.

For example:

Kafka
RabbitMQ
NATS
Redis Streams

The architecture might be:

Producer
   ↓
Message Broker
   ↓
Normalizer
   ↓
Validator
   ↓
Graph Writer
   ↓
Projection Builder

The graph writer updates relationships.

The projection builder creates optimized views for applications.

This separation is important because the graph is optimized for relationships, while dashboards may require different query patterns.

Graph Database or Relational Database?

The obvious question is:

Do we actually need a graph database?

Not necessarily.

A Universal Logistics Graph can be implemented using different storage strategies.

A graph database such as Neo4j can make relationship traversal natural.

A relational database can represent graphs using:

entities
relationships
events

A document database can store flexible entities.

A hybrid architecture can use multiple stores.

For example:

PostgreSQL
    ↓
Canonical entities

Kafka
    ↓
Event stream

Neo4j
    ↓
Relationship graph

Redis
    ↓
Hot state / caching

Object Storage
    ↓
Documents and historical data

The important idea is not the database product.

The important idea is the graph model.

Querying the Logistics Graph

Once the graph exists, interesting queries become possible.

Imagine asking:

Show everything connected to Order 10092.

The graph might return:

Order
 ├── Customer
 ├── Product A
 │    └── Warehouse 1
 ├── Product B
 │    └── Supplier 4
 ├── Shipment 1
 │    └── Vehicle 8
 └── Shipment 2
      └── Vehicle 11

Another query:

Which shipments depend on Warehouse 4?

Or:

Which orders contain products currently located in transit?

Or:

Which deliveries depend on a particular vehicle?

The graph turns complicated questions into traversals.

Pathfinding

A logistics graph naturally supports pathfinding.

Suppose a package must travel from:

Warehouse A

to:

Customer B

The graph contains:

Warehouse A
      ↓
Distribution Center
      ↓
Regional Hub
      ↓
Local Facility
      ↓
Customer B

The system can evaluate possible paths.

But pathfinding should not only minimize distance.

A route could be evaluated using multiple constraints:

cost
distance
time
capacity
availability
risk
vehicle compatibility
facility availability
delivery windows

A generalized route cost function could look like:

Cost =
    distance_weight × distance
  + time_weight × travel_time
  + handling_weight × handling_cost
  + delay_weight × expected_delay

This transforms the graph from a passive database into a decision-support engine.

Detecting Bottlenecks

One of the most valuable capabilities is identifying highly connected infrastructure.

Imagine:

Factory A
      \
       → Hub X → Warehouse Y
      /
Factory B

If many routes depend on Hub X, then Hub X becomes structurally important.

The graph can calculate:

incoming connections,
outgoing connections,
dependent shipments,
dependent orders,
dependent facilities,
historical throughput,
average processing time.

This makes bottlenecks visible as graph structures rather than isolated metrics.

Dependency Propagation

Consider a supplier responsible for a critical component.

The graph might look like:

Supplier
   ↓
Component
   ↓
Product
   ↓
Order
   ↓
Customer

If the component becomes unavailable, the impact can propagate through the graph.

The system can identify:

Affected products
Affected orders
Affected warehouses
Affected fulfillment operations
Affected routes

This is a powerful concept:

impact analysis.

Instead of discovering consequences after they happen, software can trace dependencies before executing an operation.

Inventory as a Graph

Inventory is often treated as a number.

For example:

SKU-100 = 250 units

But inventory is more interesting when represented relationally.

SKU
 |
 ├── Warehouse A → 100
 ├── Warehouse B → 75
 ├── Vehicle 22   → 25
 └── Store C      → 50

Now inventory becomes spatial and relational.

The system can ask:

Where is all inventory for this SKU?

Or:

Which orders can be fulfilled from inventory within a particular region?

Or:

Which inventory is already reserved?

The graph connects inventory to demand.

Building a Logistics Digital Twin

At a larger scale, the Universal Logistics Graph can become a digital twin of the logistics environment.

The graph represents:

Facilities
Vehicles
Products
Orders
Inventory
Routes
Shipments
People
Events
Dependencies

The real world produces events.

Those events update the digital representation.

REAL WORLD
    ↓
Events
    ↓
LOGISTICS GRAPH
    ↓
Analysis
    ↓
Decisions
    ↓
Operational Actions

This creates a feedback loop.

Predictive Capabilities

Once enough historical events exist, machine-learning models can operate on top of the graph.

For example:

Graph
  ↓
Historical Events
  ↓
Feature Extraction
  ↓
ML Model
  ↓
Prediction

Predictions could include:

estimated arrival times,
expected processing times,
demand patterns,
inventory requirements,
route congestion,
facility workload,
potential delays.

Graph-based machine learning can also use relationships as features.

A product's behavior may depend not only on the product itself but also on:

supplier,
warehouse,
region,
season,
route,
customer segment.

The graph provides that context.

Graph Algorithms

A Universal Logistics Graph opens the door to classic graph algorithms.

Shortest paths

Find efficient routes between locations.

Connected components

Identify isolated or disconnected parts of the network.

Centrality

Identify highly connected facilities or infrastructure.

Community detection

Identify clusters of operational activity.

Dependency traversal

Determine what depends on a specific entity.

Temporal traversal

Understand how relationships evolved over time.

Flow analysis

Understand movement through a network.

The interesting part is that these algorithms operate on the same underlying model.

The API

A universal logistics system needs a clean API.

For example:

POST /entities
POST /relationships
POST /events
GET /entities/:id
GET /entities/:id/neighbors
GET /entities/:id/history
GET /shipments/:id/path
GET /orders/:id/network
GET /locations/:id/dependencies

A graph query endpoint could expose more advanced operations:

POST /graph/query

With:

{
  "start": "shipment_18492",
  "relationship": "depends_on",
  "depth": 4
}

The API becomes a universal interface over logistics relationships.

Designing for Idempotency

Logistics systems often receive duplicate events.

A scanner might send an event twice.

A mobile application might retry.

A network connection might reconnect.

Therefore:

event_id

should be globally unique.

The system can maintain an idempotency record:

event_id → processed

If the same event arrives again, the graph does not duplicate the operation.

This small architectural decision can prevent enormous amounts of inconsistency.

Handling Conflicting Data

Different systems may disagree.

One system says:

Shipment = delivered

Another says:

Shipment = in_transit

The graph should not simply overwrite one value with another.

Instead, preserve the evidence.

Source A
  status = delivered

Source B
  status = in_transit

Timestamp
Source priority
Confidence

Then a resolution engine can determine the current operational state according to explicit rules.

This keeps the underlying history intact.

Observability

A logistics graph should also observe itself.

Every important operation should produce telemetry.

Graph Query
   ↓
Latency
   ↓
Database
   ↓
Cache
   ↓
External Dependency

Metrics might include:

events_processed
events_failed
graph_updates
query_latency
traversal_depth
relationship_count
stale_entities
duplicate_events

Tracing can reveal why a query became slow.

Logging can explain how a particular state was produced.

Observability becomes part of the graph architecture rather than an afterthought.

Security

A universal graph can contain sensitive operational information.

Therefore, access control should exist at multiple levels.

A user may be allowed to see:

orders

but not:

supplier contracts

Another user may see:

vehicle locations

but not:

customer information

This suggests graph-aware authorization.

Permissions can be associated with:

entity types
relationships
organizations
locations
operations

The API becomes responsible for enforcing these boundaries.

Building It Incrementally

The biggest mistake would be attempting to model every possible logistics entity immediately.

Start small.

Version 1

Build:

Order
Shipment
Package
Location
Event

Support:

created
assigned
moved
arrived
delivered

Then add:

Version 2

Warehouse
Vehicle
Route
Inventory

Then:

Version 3

Supplier
Product
Purchase Order
Container
Document

Then:

Version 4

Add:

prediction
optimization
simulation
analytics

The graph grows with the system.

A Minimal Implementation

A simple relational implementation could start with:

CREATE TABLE entities (
    id UUID PRIMARY KEY,
    type VARCHAR(100) NOT NULL,
    data JSONB,
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
);

Relationships:

CREATE TABLE relationships (
    id UUID PRIMARY KEY,
    source_id UUID NOT NULL,
    target_id UUID NOT NULL,
    type VARCHAR(100) NOT NULL,
    data JSONB,
    valid_from TIMESTAMP,
    valid_until TIMESTAMP
);

Events:

CREATE TABLE events (
    id UUID PRIMARY KEY,
    entity_id UUID NOT NULL,
    type VARCHAR(100) NOT NULL,
    location_id UUID,
    occurred_at TIMESTAMP NOT NULL,
    data JSONB
);

Indexes become critical:

CREATE INDEX idx_relationship_source
ON relationships(source_id);

CREATE INDEX idx_relationship_target
ON relationships(target_id);

CREATE INDEX idx_events_entity
ON events(entity_id);

CREATE INDEX idx_events_time
ON events(occurred_at);

That is already enough to build a surprisingly capable logistics graph.

The Graph Is More Than a Database

This is where the project becomes interesting.

A Universal Logistics Graph should not be thought of as merely:

"A database containing logistics records."

It is better understood as:

A computational model of how logistics exists and changes.

That distinction matters.

A database stores information.

A graph can expose relationships.

An event system can explain change.

A rules engine can interpret state.

A prediction engine can estimate future states.

An optimization engine can evaluate possible actions.

Together, they form something much larger.

The Future of the Architecture

Imagine opening a logistics application.

Instead of seeing hundreds of unrelated tables, you see a living network.

                    Supplier
                       │
                       ▼
                   Product
                       │
                       ▼
                     Order
                    /     \
                   /       \
             Shipment A   Shipment B
                 │             │
                 ▼             ▼
              Truck A       Truck B
                 │             │
                 ▼             ▼
             Warehouse      Hub
                   \          /
                    \        /
                     ▼      ▼
                     Customer

Every node can be inspected.

Every relationship can be traversed.

Every event can be replayed.

Every dependency can be analyzed.

Every movement can be understood in context.

This is where logistics software starts feeling less like a collection of CRUD applications and more like an operating system for movement.

The Real Engineering Challenge

The difficult part is not creating nodes.

It is creating a consistent language for relationships.

What does "moved" mean?

What does "assigned" mean?

When does a shipment become "delivered"?

How do we represent partial deliveries?

How do we represent split shipments?

How do we represent returns?

How do we represent inventory transfers?

How do we represent cancelled operations?

How do we represent temporary relationships?

How do we represent uncertainty?

These questions define the quality of the architecture.

The graph schema is therefore not just a database design.

It is a model of the real world.

And real-world modeling is one of the hardest parts of software engineering.

Conclusion

We often build logistics software around objects.

Orders.

Shipments.

Vehicles.

Warehouses.

Products.

But logistics itself is not organized around objects.

It is organized around relationships and movement.

A product is connected to an order.

An order is connected to a customer.

A shipment is connected to a route.

A route is connected to locations.

A vehicle is connected to a shipment.

A warehouse is connected to inventory.

Events connect the present to the past.

Dependencies connect today's decisions to tomorrow's consequences.

The Universal Logistics Graph brings all of those relationships into one computational model.

It provides a foundation for:

logistics tracking,
inventory intelligence,
route optimization,
dependency analysis,
shipment visibility,
supply-chain exploration,
operational analytics,
predictive systems,
simulation,
and eventually autonomous decision-support systems.

The most powerful part is not the graph database.

It is the abstraction.

Once logistics is represented as a graph, the system can reason about movement rather than merely record it.

And that opens an entirely different way of building logistics software.

Software in my mind.

Rock in my soul.

And somewhere between the two is an engineer looking at thousands of disconnected records and asking a simple question:

What if all of this was connected?

Why I Built EasyDown: Normalizing Public Media Across Social Platforms

Mon, 21 Sep 2026 11:12:52 +0000

I started building EasyDown after running into the same integration problem repeatedly: a user would paste a perfectly valid public social media URL, but every platform represented the media differently. A Reel was not shaped like a TikTok post, an Instagram carousel was not shaped like a Bilibili video, and even two URLs from the same platform could require different handling.

The first version looked simple. Detect the host, call a platform-specific parser, and return a video URL. That design stopped working as soon as I tested real posts. Some posts contained images rather than video. Some were mixed carousels. Some video formats separated audio and video. CDN links expired. Browser requests failed because of CORS or missing headers. A profile URL looked similar to a post URL but represented a completely different resource.

I decided that the useful product was not another collection of one-off download scripts. I needed a stable backend contract that could represent media from several public platforms without forcing every client to learn each upstream response format.

The problem I wanted the API to solve

From an application developer's perspective, the input should be one supported public post URL. The result should be a predictable object containing the platform, title, thumbnail, duration when available, and arrays for images, videos, and audio.

The arrays matter. Assuming that one post equals one MP4 is an easy mistake. An Instagram carousel may contain several photos and videos in a meaningful order. A TikTok photo post may have no video at all. A Bilibili response may expose multiple stream options. Returning a collection lets the product decide whether to show a picker, enqueue every item, or select a preferred rendition.

I also wanted failures to be explicit. Private posts, deleted media, expired Stories, login-only pages, profiles, live content, and region-restricted items should not be presented as successful parses. A client needs to distinguish an invalid input from a temporary upstream failure and from a valid post that has no downloadable media.

One normalized endpoint and platform-specific endpoints

EasyDown now has a common endpoint for applications that want one stable media model across supported networks. It also has platform-specific endpoints for integrations that need additional public fields from a particular service.

Instagram is a good example. The normalized media object is enough for a download workflow, but some applications also need the public shortcode, caption, author fields, image candidates, video renditions, or carousel items. I documented the current request and response contract in the Instagram API documentation I maintain.

The platform request is a normal authenticated POST from a backend:

import requests

response = requests.post(
    "https://api.easydown.org/api/v1/platforms/instagram/parse",
    headers={
        "Authorization": "Bearer ed_live_your_token",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://www.instagram.com/reel/DY2PcmxiBE2/"
    },
    timeout=30,
)

response.raise_for_status()
payload = response.json()
media = payload["data"]["media"]

for item in media.get("videos", []):
    print(item["url"], item.get("quality"), item.get("mimeType"))

The token stays on the server. I do not recommend placing a production token in browser JavaScript, a URL, analytics, or a public repository.

Platforms I support

The current API covers public post URLs from TikTok, Douyin, Toutiao, Instagram, Threads, Xiaohongshu or RedNote, Bilibili, Kuaishou or Kwai, Weibo, X or Twitter, and YouTube.

Support does not mean scraping every page on those sites. The contract is intentionally scoped to documented single-post URL formats and compatible share links. Profiles, search pages, private messages, private media, arbitrary bulk collection, and unavailable content are outside that scope.

Keeping the boundary narrow has made the API easier to reason about. Every new URL form has to pass through validation, platform detection, parsing, normalization, sensitive-field filtering, and media checks before the response is considered successful.

Lessons from building the media layer

The hardest production issue has been treating returned media URLs as temporary resources. Many social media CDNs use signatures or expiration parameters. Storing the URL as though it were a permanent asset creates jobs that fail hours or days later.

My current rule is to keep the original public post URL as the durable reference. A backend parses it when the application needs fresh media candidates, then downloads or proxies the selected file while it is valid. When a CDN URL expires, the application re-parses the post instead of retrying the expired URL indefinitely.

The second lesson is that browser playback and backend downloading are different operations. A URL that opens in one browser session can still fail in application code because of CORS, CORP, cookies, Referer requirements, or a short-lived signature. Media fetching belongs in the backend when the product needs predictable behavior.

The third lesson is to normalize conservatively. I keep stable cross-platform fields small and predictable, while versioning the extra platform data separately. That approach gives client applications a dependable base model without discarding useful public fields that only exist on one platform.

What I would test before shipping

For an Instagram integration, I would test a public Reel, a video post, a single-image post, a mixed carousel, a compatible share URL, and an unavailable post. I would also test an expired CDN URL and confirm that the application re-parses the original post.

Across every platform, I check that credentials remain server-side, errors do not leak tokens, media arrays can be empty, and the UI does not promise access to private or protected content. Those cases are less exciting than the first successful request, but they decide whether an integration survives real users.

Building EasyDown has mostly been an exercise in turning many unstable upstream shapes into one smaller contract that an application can trust. The parsing code still changes as platforms change. The goal of the API is to keep those changes behind a stable boundary so client code does not have to change with them.

Building a Smart Shopping List That Predicts What You'll Need

Sun, 20 Sep 2026 20:35:01 +0000

Most shopping lists are not cool.

Not because shopping lists are difficult to build.

A basic shopping list is one of the easiest applications a developer can create.

Add an item.

Remove an item.

Mark it as purchased.

Store it in a database.

Done.

But there is a problem.

A shopping list only knows what you tell it.

It does not know what you are about to run out of.

It does not understand that you usually buy milk every seven days.

It does not understand that rice disappears faster when you are working from home.

It does not understand that you bought two bottles of cooking oil last week and therefore probably do not need another one today.

It does not understand seasonality.

It does not understand household size.

It does not understand purchasing patterns.

It does not understand that when you buy bread, eggs and milk together, there is a strong possibility that you will buy them again around the same time next week.

A conventional shopping list records decisions.

A smarter system predicts them.

And that creates a much more interesting engineering problem.

We are no longer building a CRUD application.

We are building a personal demand forecasting engine.

The goal is simple:

Predict what a person will probably need before they remember they need it.

That sounds small.

It isn't.

Because underneath the shopping list is a fascinating combination of event tracking, time-series analysis, inventory estimation, behavioral modeling, recommendation systems and machine learning.

Let's build it.

The Problem With Traditional Shopping Lists

Imagine opening a normal shopping-list application.

You see:

☐ Milk
☐ Bread
☐ Eggs
☐ Rice
☐ Soap

Useful.

But the application has almost no intelligence.

It does not know why those items are there.

It does not know:

How frequently do you buy milk?
How long does one bottle normally last?
When did you last buy eggs?
How many eggs were purchased?
How many people consume them?
Do you normally buy eggs on weekends?
Are you currently running low?

The user has to manually translate reality into application data.

That's backwards.

The real world is producing signals constantly.

Purchases.

Consumption.

Prices.

Time.

Household size.

Recipes.

Promotions.

Seasonality.

The system should be able to learn from those signals.

Instead of:

User → remembers item → adds item

we want:

User behavior
      ↓
Purchase history
      ↓
Consumption estimation
      ↓
Demand model
      ↓
Prediction
      ↓
Suggested shopping list

Now the shopping list becomes intelligent.

The Core Idea

The system needs to answer one question:

Given everything I know about this person, what are they likely to need soon?

That requires a model.

For every product, we can estimate:

Expected consumption rate
Expected next purchase date
Current estimated inventory
Confidence

For example:

Milk

Average consumption: 1 bottle / 4 days
Last purchase: September 17
Estimated remaining: 0.5 bottle
Predicted depletion: September 19
Confidence: 91%

Then:

Prediction:
Buy milk

This is much more useful than:

☐ Milk

The application is no longer just storing information.

It is reasoning about future demand.

Step 1: Model Purchases as Events

The foundation should be an event-based data model.

Instead of only storing the current shopping list, record every purchase.

A simple schema might look like:

CREATE TABLE purchases (
    id BIGINT PRIMARY KEY,
    user_id BIGINT NOT NULL,
    product_id BIGINT NOT NULL,
    quantity DECIMAL(10,2) NOT NULL,
    price DECIMAL(10,2),
    purchased_at TIMESTAMP NOT NULL
);

Then products:

CREATE TABLE products (
    id BIGINT PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    category VARCHAR(100)
);

And predicted demand:

CREATE TABLE predictions (
    id BIGINT PRIMARY KEY,
    user_id BIGINT NOT NULL,
    product_id BIGINT NOT NULL,
    predicted_date DATE,
    predicted_quantity DECIMAL(10,2),
    confidence DECIMAL(5,2)
);

Now every purchase becomes a signal.

Suppose the user buys:

September 1 → Milk
September 5 → Milk
September 9 → Milk
September 13 → Milk
September 17 → Milk

We can immediately see something interesting.

The interval is approximately:

4 days

The system can estimate:

Milk consumption ≈ 1 unit / 4 days

The user never explicitly told the application that.

The application discovered it.

That's the beginning of intelligence.

Step 2: Calculate Purchase Intervals

For every product, calculate the time between purchases.

Suppose:

Purchase 1 → Day 1
Purchase 2 → Day 5
Purchase 3 → Day 9
Purchase 4 → Day 14
Purchase 5 → Day 18

The intervals are:

The average:

(4 + 4 + 5 + 4) / 4 = 4.25 days

So the estimated replenishment interval is:

4.25 days

But simple averages are dangerous.

A single unusual purchase can distort the result.

Imagine:

The average becomes:

7.2 days

That doesn't represent normal behavior.

A better approach is to use the median.

4, 4, 4, 4, 20

Median:

Much more representative.

This is one of those places where a simple statistical technique can outperform an unnecessarily complicated machine-learning model.

Step 3: Understand Consumption, Not Just Purchases

There is an important distinction.

Purchasing frequency isn't always consumption frequency.

Someone might buy:

10kg rice

once every two months.

That doesn't mean they consume rice only twice a month.

The system needs to estimate inventory.

Suppose:

Purchase:
10kg rice

Expected consumption:
0.25kg/day

Then:

10 / 0.25 = 40 days

The predicted depletion date is approximately 40 days after purchase.

This turns purchasing data into inventory intelligence.

We can represent the idea as:

Estimated Inventory

= Previous Inventory
+ Purchases
- Estimated Consumption

Conceptually:

inventory = previous_inventory + purchases - consumption

The hard part is estimating consumption accurately.

Step 4: Introduce Consumption Rates

For every product we can maintain:

consumption_rate

For example:

Milk      → 0.25 units/day
Bread     → 0.40 units/day
Rice      → 0.15 kg/day
Soap      → 0.03 bars/day
Eggs      → 0.35 eggs/day

These values can initially be estimated from historical purchases.

Suppose a user purchases:

12 eggs

and purchases another 12 eggs approximately 14 days later.

We can estimate:

12 / 14
≈ 0.86 eggs/day

The model doesn't need to understand breakfast.

It doesn't need to understand cooking.

It simply observes behavior.

Step 5: Add Time

Human behavior isn't constant.

People consume different things at different times.

For example:

Weekdays:
More coffee
More lunch ingredients

Weekends:
More snacks
More meat
More soft drinks

So the model should include temporal patterns.

Instead of:

consumption_rate = 0.5

we can have:

weekday_rate
weekend_rate

Or even:

Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
Sunday

Imagine discovering:

Bread consumption:

Monday      0.8
Tuesday     0.7
Wednesday   0.8
Thursday    0.7
Friday      1.2
Saturday    1.6
Sunday      1.4

Now the prediction becomes significantly more interesting.

The system isn't merely asking:

"How often does this person buy bread?"

It's asking:

"How does this person's demand behave over time?"

That's a forecasting problem.

Step 6: Detect Shopping Cycles

People have routines.

Some people shop:

Every Saturday

Others:

Every payday

Others:

Whenever something runs out

The system should detect these patterns.

Suppose purchases frequently occur around:

5th
20th

of each month.

That could indicate a salary cycle or budgeting routine.

Now the application can predict not only what someone will buy, but approximately when they will shop.

This matters because shopping recommendations should be presented at useful times.

Sending:

You may need milk.

on a Tuesday afternoon might be irrelevant.

Sending:

Your weekly essentials are probably running low.

on Friday evening may be much more useful.

Timing is part of intelligence.

Step 7: Create a Replenishment Score

We can create a score that estimates how urgently an item needs replenishment.

For example:

Replenishment Score =
Consumption Pressure
+ Time Since Purchase
+ Historical Frequency
+ Inventory Risk

A simplified implementation could be:

score = (
    consumption_rate * days_since_purchase
) / expected_quantity

Higher score means:

More likely to need replenishment.

For example:

Milk   → 0.91
Bread  → 0.82
Eggs   → 0.75
Rice   → 0.22
Soap   → 0.14

The shopping list could then automatically become:

Likely Needed Soon

☐ Milk       91%
☐ Bread      82%
☐ Eggs       75%

Possibly Needed Later

☐ Rice       22%
☐ Soap       14%

That is much more useful than an unordered list.

Step 8: Learn From Rejections

This is where many recommendation systems become smarter.

Suppose the system predicts:

Buy:
Milk
Eggs
Bread
Bananas

The user buys:

Milk
Bread

but ignores:

Eggs
Bananas

That is feedback.

The system shouldn't simply repeat the same prediction forever.

It should learn.

Maybe the user doesn't buy eggs every week.

Maybe they buy bananas only when they specifically want them.

We can introduce feedback signals:

accepted
ignored
dismissed
purchased
manually added
manually removed

Then prediction quality improves over time.

The system learns:

Prediction → User response → Model update

That's a feedback loop.

Step 9: Distinguish Staples From Occasional Items

Not every product behaves the same way.

Consider:

Salt
Milk
Rice
Toothpaste
Birthday cake
USB cable

Trying to use one forecasting strategy for all of them is a mistake.

We should classify products.

For example:

FAST_CONSUMPTION
SLOW_CONSUMPTION
SEASONAL
OCCASIONAL
EVENT_BASED
NON_CONSUMABLE

Milk:

FAST_CONSUMPTION

Rice:

SLOW_CONSUMPTION

Christmas decorations:

SEASONAL

Birthday cake:

EVENT_BASED

USB cable:

OCCASIONAL

This allows the system to apply different prediction strategies.

Step 10: Detect Seasonality

Seasonality is another hidden signal.

People don't shop identically throughout the year.

For example:

December:
More meat
More drinks
More snacks
More baking ingredients

Rainy season:
Different food preferences

School term:
More lunch ingredients
More household supplies

The system can compare current behavior against historical behavior.

Conceptually:

seasonal_factor = historical_demand_for_period / average_demand

If December demand is historically 1.4× normal demand:

predicted_demand *= 1.4

Now the shopping list understands that December isn't just another month.

Step 11: Add Household Context

The system becomes even more powerful when multiple people use it.

Imagine a household with:

2 adults
2 children

Instead of modeling only the individual:

User → Consumption

we model:

Household → Members → Consumption

Different people produce different patterns.

One person might consume:

Coffee

Another:

Tea

Children might increase:

Milk
Cereal
Snacks

The system can aggregate these signals.

Now the shopping list becomes a household demand system.

Step 12: Use Confidence Scores

Predictions should never pretend to be perfect.

If the system has only observed two purchases:

Milk:
September 1
September 8

it shouldn't claim:

Milk will definitely be needed on September 15.

Instead:

Milk
Likely needed around September 15
Confidence: 61%

After six months of consistent behavior:

Milk
Likely needed tomorrow
Confidence: 94%

Confidence is important because prediction systems need to communicate uncertainty.

A smart system doesn't just say what it thinks.

It tells you how strongly it thinks it.

Step 13: Build the Prediction Pipeline

The architecture could look like this:

                 USER ACTIVITY
                      │
        ┌─────────────┼─────────────┐
        │             │             │
    Purchases     Manual Adds    Dismissals
        │             │             │
        └─────────────┼─────────────┘
                      ↓
               EVENT PROCESSOR
                      ↓
             FEATURE GENERATION
                      ↓
        ┌─────────────┼─────────────┐
        │             │             │
   Frequency      Seasonality    Consumption
        │             │             │
        └─────────────┼─────────────┘
                      ↓
              DEMAND FORECASTER
                      ↓
              CONFIDENCE MODEL
                      ↓
              RANKING ENGINE
                      ↓
             SMART SHOPPING LIST

This is no longer a simple CRUD backend.

It is a small intelligent system.

Step 14: Start Simple Before Machine Learning

This is important.

You don't need a neural network on day one.

Start with statistics.

Use:

Median purchase interval
Average quantity
Moving averages
Exponential smoothing
Seasonal factors
Confidence estimation

For example:

prediction = (
    historical_average
    * seasonal_factor
    * recent_trend
)

A moving average can be:

forecast = sum(last_n_values) / n

Exponential smoothing gives more importance to recent behavior:

forecast = (
    alpha * latest_value
    + (1 - alpha) * previous_forecast
)

These methods are lightweight.

They are explainable.

They are easy to debug.

And for many personal shopping patterns, they may be surprisingly effective.

Machine learning can come later.

Step 15: Introduce Machine Learning

Once the dataset becomes large enough, we can train models.

Features might include:

days_since_last_purchase
average_purchase_interval
median_purchase_interval
average_quantity
recent_quantity
day_of_week
month
season
household_size
price
discount
purchase_frequency
previous_prediction_accuracy

The target could be:

Will this product be purchased within the next 7 days?

That's a classification problem.

Or:

How many units will be purchased?

That's a regression problem.

The model could output:

{
  "product": "milk",
  "probability": 0.91,
  "predicted_quantity": 2
}

The recommendation engine then converts that prediction into something human-readable.

Step 16: Understand Price Behavior

Price changes can influence purchases.

Suppose someone normally buys:

Brand A

but switches to:

Brand B

when Brand A becomes expensive.

That's behavioral information.

The system can detect:

Price ↑
     ↓
Purchase probability ↓
     ↓
Alternative product ↑

Eventually, the shopping assistant could suggest:

Milk is likely needed.

Your usual brand is currently expensive.

Alternative:
Brand B

Now we're moving from demand prediction toward intelligent purchasing assistance.

Step 17: Group Items Into Shopping Trips

The application shouldn't necessarily give you twenty notifications.

It should understand that several predictions can be combined.

Instead of:

Buy milk.

Buy bread.

Buy eggs.

Buy soap.

It can say:

Your next shopping trip may include:

Food
- Milk
- Bread
- Eggs

Household
- Soap

Then optimize around the user's shopping routine.

If someone typically shops once a week, the system can construct a weekly predicted basket.

NEXT SHOPPING TRIP

High confidence
✓ Milk
✓ Bread

Medium confidence
○ Eggs
○ Bananas

Low confidence
○ Rice

This is much closer to how humans actually shop.

Step 18: Build the System Around Events

One architectural decision makes this system significantly easier to evolve:

Treat user actions as events.

Examples:

PRODUCT_PURCHASED
ITEM_ADDED
ITEM_REMOVED
ITEM_DISMISSED
PREDICTION_ACCEPTED
PREDICTION_REJECTED
PRICE_CHANGED
RECIPE_CONSUMED

Each event can look like:

{
  "type": "PRODUCT_PURCHASED",
  "user_id": 42,
  "product_id": 17,
  "quantity": 2,
  "timestamp": "2026-09-20T10:30:00Z"
}

Now different services can consume the same event stream.

For example:

Event Stream
     │
     ├── Inventory Service
     ├── Forecasting Service
     ├── Recommendation Service
     ├── Analytics Service
     └── Notification Service

The architecture becomes extensible.

Step 19: Prevent Prediction Spam

There is a subtle problem.

A system that predicts everything can become annoying.

Imagine opening your phone and seeing:

Milk
Bread
Eggs
Rice
Soap
Sugar
Coffee
Tea
Chicken
Tomatoes
Onions
Bananas

That's not intelligence.

That's noise.

Prediction needs a threshold.

For example:

if confidence > 0.80:
    recommend()

But confidence alone isn't enough.

We can combine confidence with urgency:

priority = confidence * urgency

Then only high-value predictions appear prominently.

The user shouldn't feel like the application is constantly telling them what to buy.

The system should feel like it quietly understands them.

Step 20: The Interface Should Feel Almost Invisible

The best version of this product might not even look like a sophisticated AI application.

The interface could simply say:

Your Shopping List

Likely needed soon

🥛 Milk       91%
🍞 Bread      87%
🥚 Eggs       79%

Maybe later

🍚 Rice       31%
🧼 Soap       24%

Last updated:
Today

That's it.

Underneath this simple interface could be:

Event ingestion
+
Statistical forecasting
+
Behavior modeling
+
Seasonality detection
+
Inventory estimation
+
Recommendation ranking
+
Feedback learning

This is a recurring principle in software engineering:

The complexity should live underneath the experience, not inside it.

The Bigger Idea

A smart shopping list is not really about shopping lists.

It is about predicting human needs from behavioral signals.

Once you understand that, the architecture can expand dramatically.

The same system could eventually predict:

Groceries
Household supplies
Personal care products
Pet food
Office supplies
Medicine cabinet replenishment
School supplies

The system becomes a personal replenishment engine.

And then something even more interesting happens.

The user doesn't have to explicitly manage everything.

The software begins to anticipate routine needs.

Not because it knows the future.

Because human behavior is often surprisingly repetitive.

The Architecture I'd Build

If I were building this as a serious production system, I would probably start with:

Frontend
React / React Native

API
FastAPI / Django / Node.js

Database
PostgreSQL

Cache
Redis

Event Bus
Kafka / Redis Streams

Analytics
Python

Forecasting
Python + statistical models

ML
PyTorch / scikit-learn

Scheduler
Celery / background workers

The initial version could be much simpler:

React
   ↓
Django API
   ↓
PostgreSQL
   ↓
Python forecasting worker

Then scale components only when necessary.

Don't build Kafka because the architecture diagram looks impressive.

Build it when the workload requires it.

That's engineering.

The Real Challenge Is Trust

The hardest part isn't prediction.

It's trust.

If the application predicts:

Buy milk

and it's wrong once, that's fine.

If it's wrong constantly, users stop listening.

The system therefore needs to learn continuously.

Every interaction is feedback.

Prediction
     ↓
User action
     ↓
Outcome
     ↓
Prediction evaluation
     ↓
Model update

You can even track prediction metrics:

Precision
Recall
False positives
False negatives
Prediction confidence
Purchase conversion

Suppose the system predicts 100 purchases.

Users actually purchase 80.

If 70 of those predictions were correct:

Precision = 70 / 100
          = 70%

Now you have an objective way to improve the engine.

The shopping list becomes measurable software rather than magical AI.

What This System Eventually Becomes

At first:

Shopping List

Then:

Smart Shopping List

Then:

Personal Demand Forecasting

Then:

Personal Inventory Intelligence

And eventually:

Personal Consumption Operating System

That's the interesting evolution.

The system starts by asking:

"What did you buy?"

Then:

"What do you usually buy?"

Then:

"What are you likely to buy?"

Then:

"What are you likely to need?"

Those are completely different questions.

The final system isn't simply remembering your shopping habits.

It is modeling your relationship with physical resources.

Final Thoughts

There is a tendency in software development to chase complicated ideas.

Build another chatbot.

Build another dashboard.

Build another CRUD SaaS.

Build another AI wrapper.

But some of the most interesting engineering problems are hidden inside ordinary activities.

Shopping is one of them.

People buy things repeatedly.

They consume resources.

They develop routines.

Their behavior changes with time.

Their decisions respond to prices.

Their households influence demand.

Their habits produce data.

That makes shopping an enormous prediction problem disguised as a grocery list.

A traditional application waits for the user to say:

I need milk.

A smarter application studies the evidence and says:

You will probably need milk tomorrow.

An even smarter system eventually says:

Your next shopping trip is probably going to look like this.

And the truly interesting part isn't the shopping list.

It's everything underneath it.

Time-series analysis.

Inventory estimation.

Behavioral modeling.

Event-driven architecture.

Recommendation systems.

Statistical forecasting.

Machine learning.

Feedback loops.

Uncertainty.

Personalization.

All hidden behind a simple checkbox.

That's the kind of software I find interesting.

Because the best intelligent systems don't always look intelligent.

Sometimes they just quietly make everyday life easier.

And that's exactly what a smart shopping list should do.

Don't make people remember what they need to buy. Build systems that learn what they repeatedly need — and tell them before they run out.

Building a Smart Meal Planning Engine Based on What You Already Own

Sun, 20 Sep 2026 16:37:38 +0000

Most meal-planning applications begin with the wrong question.

They ask:

“What do you want to eat?”

That sounds reasonable.

But real life is rarely that simple.

A person opens the refrigerator and finds half a tomato, three eggs, some cooked rice, spinach, onions, a piece of chicken, and a bottle of sauce. The pantry contains pasta, flour, beans, canned tomatoes, spices, and perhaps something that has been sitting there long enough that nobody remembers when it was purchased.

The problem isn't necessarily a lack of food.

The problem is that the food already exists, but the intelligence required to turn it into meals doesn't.

This creates an interesting software problem.

What if we built a system that didn't primarily recommend meals based on recipes?

What if it started with reality?

What do I already own?

From there, the system could determine:

what meals can be prepared immediately,
what meals require only one or two additional ingredients,
what ingredients are approaching expiration,
which ingredients can be reused across several meals,
how many people can be fed,
how much food is likely to remain,
and what should be purchased next.

This is not simply a recipe recommendation system.

It is an inventory intelligence engine for food.

And that distinction changes everything.

The Refrigerator Is an Unstructured Database

Think about the average kitchen.

There is no database.

There is no API.

There is no schema.

There is no transaction log.

There is simply food.

A person might say:

“I have some chicken.”

But software needs more information.

How much?

Cooked or raw?

Frozen or refrigerated?

When was it purchased?

When does it expire?

Is it already allocated to another meal?

Can it be consumed safely?

The first engineering challenge therefore becomes food inventory modeling.

A basic inventory record might look like this:

{
  "name": "chicken breast",
  "quantity": 500,
  "unit": "grams",
  "state": "raw",
  "location": "freezer",
  "purchased_at": "2026-09-18",
  "expires_at": "2026-10-15"
}

But this still isn't enough.

Food is contextual.

Consider rice.

Rice

could mean:

2 kg uncooked rice

or:

3 cups cooked rice

Those are fundamentally different resources.

So the system needs to understand food states.

Ingredient
   │
   ├── Raw
   ├── Cooked
   ├── Frozen
   ├── Thawed
   └── Prepared

This is where the application begins looking less like a recipe app and more like an inventory management platform.

The Core Idea

The system can be modeled around four major components:

                 ┌────────────────────┐
                 │   Kitchen Inventory │
                 └─────────┬──────────┘
                           │
                           ▼
                 ┌────────────────────┐
                 │ Ingredient Engine  │
                 └─────────┬──────────┘
                           │
                           ▼
                 ┌────────────────────┐
                 │ Meal Planning Engine│
                 └─────────┬──────────┘
                           │
             ┌─────────────┼─────────────┐
             ▼             ▼             ▼
        Meal Plan      Shopping List   Waste Alerts

The inventory tells us what exists.

The ingredient engine understands those resources.

The meal planner determines what can be made.

The shopping system fills the gaps.

The waste system protects ingredients from being forgotten.

This creates a feedback loop.

Buy Food
   ↓
Inventory
   ↓
Cook
   ↓
Inventory Changes
   ↓
Meal Planning
   ↓
Shopping
   ↓
Buy Food

The kitchen becomes a continuously changing state.

Step One: Building the Ingredient Inventory

The most important database table might be surprisingly simple.

CREATE TABLE pantry_items (
    id UUID PRIMARY KEY,
    user_id UUID NOT NULL,
    ingredient_id UUID NOT NULL,
    quantity DECIMAL NOT NULL,
    unit VARCHAR(30) NOT NULL,
    state VARCHAR(30),
    storage_location VARCHAR(30),
    purchased_at TIMESTAMP,
    expires_at TIMESTAMP
);

But manually entering every ingredient is terrible UX.

Nobody wants to type:

Tomatoes
Quantity: 4
Unit: pieces
Purchased: yesterday
Expiration: next week

for every shopping trip.

Therefore, input becomes a major engineering problem.

We could support several methods.

Manual Entry

The simplest approach.

Users type:

3 eggs
500g chicken
2 onions
1kg potatoes

The system parses the input.

Barcode Scanning

A barcode can identify packaged products.

Barcode
   ↓
Product Database
   ↓
Ingredient Mapping
   ↓
Inventory Entry

Receipt Scanning

This becomes significantly more interesting.

A receipt can be converted into structured inventory.

RECEIPT
--------------------------------
Chicken Breast       K45.00
Rice 2kg             K38.00
Tomatoes             K15.00
Eggs 12              K42.00
--------------------------------

OCR extracts the text.

An entity recognition system identifies ingredients.

The inventory engine then creates records.

Natural Language

We can also allow:

“I bought two kilos of potatoes, a dozen eggs and half a kilo of chicken.”

The system transforms that sentence into structured inventory.

This is where an AI layer becomes useful.

But AI should not become the database.

The AI interprets.

The deterministic system stores.

Step Two: Normalize Food

Humans use inconsistent language.

One person says:

tomato

Another says:

tomatoes

Another:

fresh tomatoes

Another:

Roma tomatoes

The system needs an ingredient ontology.

For example:

Tomato
├── Roma tomato
├── Cherry tomato
├── Beefsteak tomato
└── Canned tomato

But we shouldn't normalize everything into one ingredient.

Canned tomatoes are not interchangeable with fresh tomatoes in every recipe.

Therefore, ingredients need semantic relationships.

fresh tomato
    │
    ├── substitute → canned tomato
    └── related → tomato paste

This creates a graph.

And suddenly meal planning becomes a graph-search problem.

Step Three: Represent Recipes as Requirements

A recipe isn't just text.

It is a set of requirements.

Imagine a simple chicken rice bowl.

Chicken Rice Bowl

Chicken       250g
Rice          200g
Onion         1
Tomato        2
Oil           10ml
Salt          2g

We can represent it as:

{
  "meal": "Chicken Rice Bowl",
  "servings": 2,
  "ingredients": [
    {"name": "chicken", "quantity": 250, "unit": "g"},
    {"name": "rice", "quantity": 200, "unit": "g"},
    {"name": "onion", "quantity": 1, "unit": "piece"},
    {"name": "tomato", "quantity": 2, "unit": "piece"},
    {"name": "oil", "quantity": 10, "unit": "ml"}
  ]
}

Now the system can compare this requirement against inventory.

Suppose the user owns:

Chicken: 500g
Rice: 1kg
Onion: 3
Tomato: 1
Oil: 500ml

The meal is almost possible.

Only one tomato is missing.

This creates an important concept:

Meal Completeness

We can calculate:

available requirements / total requirements

For example:

Chicken: 100%
Rice: 100%
Onion: 100%
Tomato: 50%
Oil: 100%

Overall:

90%

The system could therefore say:

You can make this meal if you buy one tomato.

That's far more useful than simply recommending a random recipe.

Step Four: Score Meals

Now we need an algorithm.

The system shouldn't simply find meals that are technically possible.

It should find meals that make sense.

We can define a meal score:

MealScore =
    InventoryMatch
  + ExpirationPriority
  + NutritionalFit
  + PreferenceScore
  + CostEfficiency
  - MissingIngredients
  - RepetitionPenalty

Each component represents a different dimension.

Inventory Match

How much of the recipe already exists?

0 → nothing available
1 → everything available

Expiration Priority

Suppose spinach expires tomorrow.

A spinach-based meal should receive a higher score.

days_until_expiration = 1

might produce:

expiration_priority = 0.95

While something expiring in three weeks might produce:

expiration_priority = 0.1

The engine therefore becomes proactive.

Instead of asking:

“What should I eat?”

It can say:

“You should probably use these ingredients soon.”

Step Five: Minimize Food Waste

This is where the system becomes genuinely interesting.

Imagine the inventory:

1 avocado
3 tomatoes
half an onion
250g spinach
2 eggs
400g chicken

The naive system might generate five unrelated meals.

That could actually increase waste.

A smarter system thinks in sequences.

For example:

Monday

Chicken and spinach rice.

Uses:

250g chicken
150g spinach
rice
onion

Tuesday

Chicken tomato pasta.

Uses:

150g chicken
2 tomatoes
onion

Wednesday

Egg and spinach toast.

Uses:

2 eggs
remaining spinach
1 tomato

Now ingredients are deliberately reused.

This is cross-meal optimization.

The objective isn't:

Find the best meal.

It is:

Find the best sequence of meals given the inventory.

That's a much harder problem.

Meal Planning Is an Optimization Problem

Suppose we have:

Ingredients = I
Meals = M
Days = D

We want to select meals that maximize utility.

Conceptually:

maximize:

food_utilization
+ preference_satisfaction
+ nutritional_balance
+ expiration_reduction

while minimizing:

shopping_cost
+ food_waste
+ preparation_time
+ ingredient_repetition

This starts resembling operations research.

We can formulate binary variables:

x(m,d) = 1

if meal m is selected for day d.

Then constraints can be introduced.

For example:

One dinner per day.

Σ x(m,d) = 1

We can also enforce inventory constraints:

total ingredient consumption <= available quantity + purchased quantity

And dietary constraints:

protein >= minimum requirement

The problem could be solved using:

integer linear programming,
constraint programming,
dynamic programming,
heuristic optimization,
genetic algorithms,
simulated annealing,
or a hybrid approach.

This is where meal planning becomes a serious engineering project.

The Shopping List Should Be an Output, Not the Starting Point

Traditional meal planning often works like this:

Choose recipes
↓
Generate shopping list
↓
Buy everything

Our system reverses the relationship.

Inventory
↓
Find possible meals
↓
Identify missing ingredients
↓
Determine which purchases unlock the most meals
↓
Generate shopping list

That difference matters.

Suppose buying one onion unlocks five meals.

Buying a specialty ingredient unlocks one.

The system should understand the difference.

We can define:

IngredientValue =
    meals_unlocked
    / purchase_cost

An onion might therefore have enormous planning value.

A rare ingredient might have low value.

The shopping list becomes an optimization system.

The “One Purchase Unlocks Many Meals” Algorithm

Imagine:

Meal A requires:
chicken + rice + tomato

Meal B requires:
chicken + pasta + tomato

Meal C requires:
eggs + tomato + spinach

The user owns everything except tomatoes.

One purchase unlocks three meals.

The engine can say:

Buying tomatoes completes three meals already compatible with your inventory.

This is much more intelligent than:

Add tomatoes to shopping list.

The system understands consequence.

Ingredient Substitution

Real kitchens are messy.

You may have:

beans

but the recipe asks for:

chickpeas

Can they substitute?

Sometimes.

But substitutions need confidence.

We can maintain a substitution graph:

Ingredient A
     │
     ├── strong substitute
     ├── moderate substitute
     └── weak substitute

For example:

spinach
   ↓
kale
   ↓
cabbage

The system can then adjust recipe requirements.

Instead of:

Missing spinach.

It could say:

You don't have spinach, but your kale can work as a substitute.

This is where AI and deterministic food knowledge can cooperate.

Learning the User

The engine should eventually learn.

Suppose the user repeatedly rejects:

fish

and repeatedly selects:

chicken

The system should adapt.

But there's an important distinction between preference and behavior.

The user might love pizza but rarely cook it.

Therefore:

declared preference

isn't the same as:

observed preference

We can maintain both.

{
  "ingredient": "chicken",
  "declared_preference": 0.8,
  "observed_preference": 0.92
}

Over time the recommendation engine becomes personalized.

Time Is an Ingredient Too

Food planning shouldn't only consider food.

It should consider human time.

A person may have:

Monday: 20 minutes
Tuesday: 45 minutes
Wednesday: 15 minutes
Saturday: 90 minutes

The meal planner should understand that.

A complex meal doesn't belong on a night when the user only has fifteen minutes.

So:

MealScore =
    food compatibility
    + preference
    + expiration
    + nutrition
    + cost
    + time compatibility

Saturday could therefore receive more complex meals.

Tuesday might prioritize:

one-pan meals
leftovers
quick meals
prepared ingredients

Now the system understands lifestyle.

Leftovers Become First-Class Objects

Most recipe applications treat a meal as finished when it is cooked.

Our system shouldn't.

If the user prepares:

4 servings of chicken stew

and eats one:

3 servings remain

Those leftovers enter inventory.

This changes the planning graph.

Tomorrow's meal might use:

leftover chicken stew
+ rice

The system therefore tracks prepared food.

Raw Ingredient
      ↓
Cooking
      ↓
Prepared Meal
      ↓
Leftover Inventory
      ↓
Future Meal

This is effectively a transformation pipeline.

Food Has a Lifecycle

We can model food like a state machine.

PURCHASED
    ↓
STORED
    ↓
OPENED
    ↓
PARTIALLY_USED
    ↓
COOKED
    ↓
LEFTOVER
    ↓
CONSUMED

Or:

PURCHASED
    ↓
EXPIRING_SOON
    ↓
EXPIRED

The system can monitor transitions.

For example:

Spinach
expires tomorrow

could trigger:

Your spinach is approaching its expected use-by window. Here are three meals that use most of it.

This is proactive software.

The AI Layer

AI is useful, but it should not control everything.

A large language model can help with:

understanding natural-language inventory,
parsing receipts,
identifying ingredients,
generating recipe variations,
explaining substitutions,
understanding user preferences,
generating cooking instructions.

But deterministic algorithms should handle:

inventory quantities,
expiration calculations,
nutritional calculations,
shopping totals,
constraints,
meal scheduling,
food consumption,
transaction history.

This separation is important.

The AI should not decide whether:

2 eggs - 1 egg = 1 egg

That's database logic.

AI interprets messy human language.

The core engine maintains reality.

Architecture

A production architecture could look like this:

                    Mobile / Web App
                           │
                           ▼
                      API Gateway
                           │
        ┌──────────────────┼──────────────────┐
        ▼                  ▼                  ▼
 Inventory Service   Meal Service       User Service
        │                  │
        ▼                  ▼
 Inventory DB       Recipe Database
        │                  │
        └──────────┬───────┘
                   ▼
             Planning Engine
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
   Optimizer    AI Layer   Nutrition
        │
        ▼
 Shopping Engine

A PostgreSQL database could handle transactional data.

Redis could support caching.

A background worker could process:

expiration checks
receipt processing
meal optimization
notifications

And an AI service could handle semantic interpretation.

Event-Driven Inventory

One powerful design is to make inventory event-driven.

Instead of simply changing:

chicken.quantity = 250g

we record events.

PURCHASED 500g chicken
CONSUMED 250g chicken

The current quantity becomes a projection.

500g - 250g = 250g

This creates an audit trail.

Events might look like:

{
  "type": "FOOD_CONSUMED",
  "ingredient": "chicken",
  "quantity": 250,
  "unit": "g",
  "timestamp": "2026-09-20T18:00:00Z"
}

This architecture gives us something valuable:

history.

The system can learn how quickly a household consumes ingredients.

Predicting Future Consumption

Once enough historical data exists, the engine can estimate consumption.

Suppose the user purchases:

2kg rice

and historically consumes:

500g per week

The system can estimate:

2kg / 500g = approximately 4 weeks

This can influence shopping.

Instead of simply saying:

You're out of rice.

The system eventually says:

Based on your historical usage, your rice is likely to run out in approximately six days.

Now the system is forecasting.

Household Intelligence

The system becomes even more powerful when multiple people use it.

Imagine a household with:

Parent
Partner
Child

Each person has different preferences.

The planner now needs to optimize across everyone.

Meal Utility =
    Person A satisfaction
  + Person B satisfaction
  + Person C satisfaction
  + nutritional requirements
  + inventory efficiency

This introduces another interesting problem:

conflict resolution.

One person wants chicken.

Another wants vegetarian food.

A third hates tomatoes.

The system could find meals that satisfy overlapping constraints.

Meal planning becomes a multi-agent optimization problem.

The Hidden Power: Turning Food Into a Resource Graph

At this point, the application is no longer just a recipe app.

It becomes a resource graph.

          Chicken
             │
      ┌──────┼──────┐
      ▼      ▼      ▼
    Curry   Pasta   Rice Bowl
      │       │       │
      ▼       ▼       ▼
    Onion   Tomato   Spinach

Each ingredient connects to potential meals.

Each meal connects to time.

Each meal connects to nutrition.

Each ingredient connects to expiration.

Each purchase connects to cost.

This creates a dynamic graph of the user's kitchen.

And that graph changes every day.

The Interesting Product Experience

Imagine opening the application.

Instead of seeing:

“Here are today's recipes.”

You see:

Your Kitchen

12 ingredients available
3 expiring soon
5 meals available now
2 meals available with one purchase

Then:

Use Soon

Spinach
Tomatoes
Avocado

You Can Cook Tonight

Chicken Rice Bowl
Egg & Spinach Wrap
Tomato Pasta

One Purchase Unlocks

Chicken Tacos
Requires: tortillas

Suggested Shopping

Tomatoes
Tortillas
Milk

The application isn't telling you what food exists in the universe.

It is telling you what you can realistically do with the food you already own.

That is a fundamentally different product.

Building the MVP

I wouldn't start with AI.

I'd start with the deterministic core.

Version 1

Build:

User accounts
+
Inventory
+
Recipes
+
Ingredient matching
+
Basic meal recommendations
+
Shopping list

Version 2

Add:

Expiration tracking
+
Leftovers
+
Ingredient substitutions
+
Meal scheduling

Version 3

Add:

Receipt scanning
+
Barcode scanning
+
Natural language inventory

Version 4

Add:

Personalization
+
Consumption prediction
+
Optimization algorithms

Version 5

Add:

Household planning
+
Advanced nutrition
+
Adaptive shopping
+
Predictive waste reduction

The important thing is to avoid building a giant AI wrapper around recipes.

Build the underlying system.

A Simple Planning Algorithm

A basic recommendation algorithm might look like this:

def score_meal(meal, inventory, preferences):
    score = 0

    for ingredient in meal.ingredients:
        available = inventory.get(ingredient.name, 0)

        if available >= ingredient.required:
            score += 20
        else:
            missing = ingredient.required - available
            score -= missing * 5

    score += preference_score(meal, preferences)
    score += expiration_score(meal, inventory)
    score += nutrition_score(meal)

    return score

Then:

meals = load_candidate_meals()

ranked = sorted(
    meals,
    key=lambda meal: score_meal(
        meal,
        inventory,
        preferences
    ),
    reverse=True
)

That's enough to demonstrate the concept.

But the real system eventually becomes much more sophisticated.

The Optimization Engine

A more advanced planner could evaluate an entire week.

Suppose:

Inventory = 25 ingredients
Candidate meals = 500
Days = 7

We don't simply select seven independent meals.

We search for a combination.

Meal 1
Meal 2
Meal 3
Meal 4
Meal 5
Meal 6
Meal 7

that maximizes:

inventory utilization
nutrition
preference
expiration reduction
cost efficiency
time compatibility

while minimizing:

waste
shopping
repetition
preparation time

This is where advanced optimization becomes valuable.

You could even experiment with genetic algorithms.

A chromosome could represent:

[meal_12, meal_98, meal_41, meal_17, meal_203, meal_5, meal_88]

The fitness function evaluates the entire weekly plan.

Mutation changes one meal.

Crossover combines two plans.

Selection keeps higher-performing schedules.

The system evolves meal plans rather than simply generating them.

The Bigger Idea

There is something deeper here.

Software often starts from what users want to acquire.

But some of the most useful software can start from what users already possess.

People already have:

food,
money,
time,
devices,
documents,
messages,
skills,
energy,
transportation,
inventory.

The interesting problem is often not:

“What should I get?”

It is:

“What can I accomplish with what I already have?”

A smart meal planner is a beautiful example of this philosophy.

The refrigerator becomes a dataset.

The pantry becomes an inventory system.

Recipes become executable plans.

Ingredients become resources.

Shopping becomes an optimization problem.

And cooking becomes the execution layer.

Final Architecture

The complete system can eventually look like this:

                  USER
                   │
                   ▼
          ┌─────────────────┐
          │ Kitchen Capture │
          │ Receipt / Scan  │
          │ Manual / Voice  │
          └────────┬────────┘
                   │
                   ▼
          ┌─────────────────┐
          │ Food Normalizer │
          └────────┬────────┘
                   │
                   ▼
          ┌─────────────────┐
          │    Inventory    │
          │     Engine      │
          └────────┬────────┘
                   │
          ┌────────┴─────────┐
          ▼                  ▼
   Expiration Engine    Consumption Model
          │                  │
          └────────┬─────────┘
                   ▼
          ┌─────────────────┐
          │ Meal Candidates │
          └────────┬────────┘
                   │
                   ▼
          ┌─────────────────┐
          │ Optimization    │
          │     Engine      │
          └────────┬────────┘
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
      Meals     Shopping    Waste
       Plan       List      Alerts

And that's when the product becomes much more than a recipe application.

It becomes a personal food operating system.

The interesting engineering isn't generating another recipe.

The interesting engineering is understanding the state of a person's kitchen, predicting what they can do with it, optimizing what should happen next, and continuously updating that model as reality changes.

Because the future of smart software isn't always about giving people more choices.

Sometimes it's about looking at what they already have and saying:

You already have enough. Let's figure out what we can build with it.

That is where the real system begins.

Building a Universal Inbox That Prioritizes Messages by Consequence

Sun, 20 Sep 2026 14:31:18 +0000

Most inboxes were designed around a very simple assumption:

A message that arrived earlier should probably be dealt with earlier.

That assumption made sense when communication was relatively linear.

You opened your email client. You saw messages arranged chronologically. The newest message appeared at the top. You read it, replied, archived it, or ignored it.

Then the internet became the operating system of modern life.

Now a person can receive an email from a bank, a WhatsApp message from a family member, a Slack notification from a team, an Instagram DM from a customer, an SMS from a delivery company, a calendar invitation, a GitHub notification, a payment alert, a customer-support ticket, and five marketing messages within the same hour.

The problem is no longer message delivery.

The problem is consequence management.

A message can arrive at 09:00 and have almost no consequence.

Another can arrive at 09:17 and contain information that changes someone's entire day.

Yet most inboxes treat them as neighbors in a list.

That is the architectural problem I want to explore.

What if an inbox did not primarily ask:

“When did this message arrive?”

What if it asked:

“What happens if the user does not act on this message?”

That seemingly small change transforms the problem completely.

We are no longer building another messaging interface.

We are building a consequence-aware information system.

The Inbox Is Actually a Decision Queue

Think about the average inbox.

You might have:

09:42 — Newsletter
09:38 — Team notification
09:31 — Bank alert
09:27 — Friend's message
09:15 — GitHub notification
09:02 — Marketing email
08:51 — Calendar reminder

Chronologically, this is perfectly organized.

Operationally, it may be terrible.

Suppose the bank alert says:

Your card was used for a $2,000 transaction.

And the newsletter says:

Ten productivity techniques for developers.

Both are simply "new messages."

The system has successfully answered:

What arrived recently?

But it has failed to answer:

What matters right now?

This is where a universal consequence-aware inbox becomes interesting.

Instead of treating messages as independent pieces of text, we treat them as events that may create consequences.

The inbox becomes a decision-support layer.

The Universal Inbox

The first architectural challenge is aggregation.

Messages currently live inside separate ecosystems.

Email has one protocol.

SMS has another.

Chat applications have their own APIs.

Social networks have their own messaging systems.

Business applications have notifications.

Developer platforms have alerts.

Financial platforms have transaction notifications.

A universal inbox needs to normalize these heterogeneous sources.

Conceptually:

              ┌───────────────┐
              │ Email         │
              └───────┬───────┘
                      │
              ┌───────▼───────┐
              │ WhatsApp      │
              └───────┬───────┘
                      │
              ┌───────▼───────┐
              │ SMS           │
              └───────┬───────┘
                      │
              ┌───────▼───────┐
              │ Slack         │
              └───────┬───────┘
                      │
              ┌───────▼───────┐
              │ GitHub        │
              └───────┬───────┘
                      │
              ┌───────▼───────┐
              │ Other Sources │
              └───────┬───────┘
                      │
                      ▼
             ┌─────────────────┐
             │ Universal       │
             │ Message Model   │
             └─────────────────┘

The key idea is not to make every source identical.

That would destroy important semantics.

Instead, we create a common representation.

For example:

{
  "id": "msg_82931",
  "source": "email",
  "sender": "*Emails are not allowed*",
  "recipient": "user_42",
  "timestamp": "2026-09-20T09:31:00Z",
  "subject": "Suspicious transaction",
  "content": "...",
  "attachments": [],
  "thread_id": "thread_1029",
  "metadata": {
    "sender_type": "financial_institution",
    "category": "transaction_alert"
  }
}

Now the prioritization engine does not need to understand every messaging protocol.

It operates on normalized events.

That separation is extremely important.

Message Arrival Is Not Message Importance

A universal inbox should distinguish at least three concepts:

Arrival
Importance
Consequence

These are different.

A message can be important but not urgent.

A message can be urgent but not important.

And a message can be neither.

For example:

"Your electricity bill is due tomorrow."

This may have relatively high consequence.

"Check out our new collection."

This probably has low consequence.

"Your server is down."

This may have extremely high consequence.

The system therefore needs a consequence model.

One simple representation could be:

Consequence Score =
Potential Impact
× Urgency
× Probability
× Irreversibility

This does not need to be mathematically perfect.

The purpose is to create a framework for reasoning.

Consequence Is Contextual

Here is where things become much more interesting.

The same message can have completely different consequences for different people.

Consider:

"Your flight has been delayed."

For someone sitting at home:

Moderate consequence.

For someone already at the airport:

High consequence.

For someone who has not booked the flight:

Zero consequence.

Therefore, consequence cannot be calculated from message text alone.

We need user context.

A consequence engine might consider:

Message
+
User Identity
+
Calendar
+
Location Context
+
Previous Interactions
+
Deadlines
+
Relationships
+
Financial Context
+
Active Tasks
+
Historical Behavior

The result becomes much more intelligent.

Imagine the system knows that you have a flight at 18:00.

Then it receives:

Flight 728 delayed by 90 minutes.

The inbox should understand:

Existing event:
Flight at 18:00

Incoming event:
Delay +90 minutes

Potential effect:
Airport arrival timing changes

Priority:
High

This is no longer simple notification sorting.

It is contextual reasoning.

The Consequence Graph

One of the most powerful structures we can introduce is a consequence graph.

Instead of representing messages as isolated objects, represent their relationships.

For example:

Message
   │
   ▼
Event
   │
   ▼
Task
   │
   ▼
Deadline
   │
   ▼
Potential Outcome

Suppose you receive:

"Your domain registration expires in 5 days."

The message itself is not necessarily urgent.

But the consequence chain is:

Message
   ↓
Domain expiration
   ↓
Website unavailable
   ↓
Business disruption
   ↓
Potential revenue loss

Now the inbox can understand why the message matters.

This is the difference between content classification and consequence reasoning.

Building the Priority Engine

We can start with a scoring model.

For example:

def calculate_priority(message, context):
    impact = estimate_impact(message, context)
    urgency = estimate_urgency(message, context)
    probability = estimate_probability(message, context)
    irreversibility = estimate_irreversibility(message, context)

    return (
        impact *
        urgency *
        probability *
        irreversibility
    )

But this is only the beginning.

A serious system needs multiple signals.

Consider:

Sender importance
Message urgency
Financial impact
Deadline proximity
User relationship
Required action
Historical behavior
Confidence
Potential damage
Time sensitivity
Dependency count

We can represent them as features:

{
  "sender_importance": 0.91,
  "urgency": 0.82,
  "financial_impact": 0.74,
  "deadline_proximity": 0.95,
  "relationship_strength": 0.88,
  "action_required": 1.0,
  "irreversibility": 0.63
}

A ranking model can then combine these signals.

But Don't Let AI Decide Everything

This is where I think many modern systems make a dangerous architectural mistake.

They throw an LLM at the entire problem.

That is not enough.

An inbox like this needs deterministic rules, statistical models, and language models working together.

For example:

                Incoming Message
                       │
                       ▼
              ┌──────────────────┐
              │ Normalization     │
              └────────┬─────────┘
                       ▼
              ┌──────────────────┐
              │ Entity Extraction│
              └────────┬─────────┘
                       ▼
              ┌──────────────────┐
              │ Rule Engine      │
              └────────┬─────────┘
                       ▼
              ┌──────────────────┐
              │ ML/AI Analysis   │
              └────────┬─────────┘
                       ▼
              ┌──────────────────┐
              │ Consequence      │
              │ Engine           │
              └────────┬─────────┘
                       ▼
              ┌──────────────────┐
              │ Priority Ranking │
              └──────────────────┘

AI can interpret ambiguous language.

Rules can enforce hard guarantees.

Machine learning can discover patterns.

The combination is much stronger.

Entity Extraction Changes Everything

A message contains more than words.

It contains entities.

Consider:

"Payment of K4,500 is due on September 25."

We should extract:

Amount: K4,500
Currency: ZMW
Deadline: 2026-09-25
Action: Payment

Another message:

"Your server certificate expires in 12 days."

Extract:

Resource: Server certificate
Event: Expiration
Deadline: 12 days
Potential impact: Service disruption

Another:

"Sarah asked whether you can send the contract before Friday."

Extract:

Person: Sarah
Task: Send contract
Deadline: Friday
Action required: Yes

Now the inbox is becoming a structured information system.

The Inbox Should Understand Actions

There is a huge difference between:

"Here's the document."

and:

"Please sign the document before Friday."

The second message creates an obligation.

That means the system should detect action semantics.

Possible categories:

FYI
Request
Approval
Payment
Decision
Deadline
Alert
Invitation
Confirmation
Cancellation
Reminder
Emergency

Then:

Message
   ↓
Intent classification
   ↓
Action extraction
   ↓
Deadline extraction
   ↓
Consequence estimation

Now the inbox can tell the user:

You have 3 messages requiring action today.

That is much more useful than:

You have 37 unread messages.

Relationship Intelligence

Not all senders matter equally.

But "importance" should not simply mean celebrity, executive, or someone with a large title.

Importance is relational.

A message from your doctor, accountant, customer, parent, client, colleague, or friend may have different importance depending on context.

The system can learn:

How frequently do I interact with this person?

How often do I respond?

Have previous messages contained deadlines?

Do I usually open their messages?

Have I marked previous messages from them important?

Are they associated with an active project?

This creates a personalized relationship graph.

For example:

User
 ├── Sarah
 │    ├── 23 conversations
 │    ├── 87% response rate
 │    └── Active project
 │
 ├── Bank
 │    ├── 14 financial alerts
 │    └── High consequence
 │
 └── Newsletter
      ├── 132 messages
      └── Low interaction

The system learns from behavior.

Behavioral Feedback

The ranking model should continuously learn.

Suppose the system consistently places GitHub notifications near the top.

But you repeatedly ignore them.

Eventually:

Observed behavior
        ↓
Feedback signal
        ↓
Model update
        ↓
Lower future ranking

Likewise, suppose you consistently respond immediately to messages from one particular person.

The model learns that relationship.

However, we must be careful.

Behavioral learning can create feedback loops.

If the system initially ranks something low, the user may never see it.

Then the system interprets the lack of interaction as evidence that it is unimportant.

That is a classic recommendation-system problem.

Therefore, we need exploration.

Exploration vs Exploitation

A sophisticated inbox should occasionally surface things outside the user's normal behavior.

For example:

"You usually ignore these notifications, but this one contains a payment deadline."

This prevents the system from becoming blind to new categories of consequence.

The ranking system can therefore use:

Priority =
Learned relevance
+
Explicit rules
+
Current context
+
Novelty

The system should not become a prison built from yesterday's behavior.

Threads Matter

Messages should not always be ranked independently.

A conversation is a temporal structure.

Suppose someone sends:

Message 1:
Can you review this?

Message 2:
Just following up.

Message 3:
We need this by 16:00.

The third message dramatically changes the consequence.

The system should understand the thread as a whole.

We can maintain:

Thread State

participants
intent
open tasks
deadlines
latest commitment
unresolved questions
sentiment
priority

Then the inbox can surface:

Contract review — deadline today at 16:00

instead of:

John: We need this by 16:00.

That is a much better abstraction.

Priority Should Explain Itself

A major problem with algorithmic ranking is trust.

If the system puts something at the top, the user should be able to ask:

Why?

The answer could be:

High priority because:

• Payment deadline is tomorrow
• Amount: K4,500
• Sender is a financial institution
• You have not completed the payment

This is critical.

The system should not simply output a number.

It should output a reason.

Priority: 94

Reasons:
Deadline: +30
Financial impact: +25
Sender relevance: +20
Action required: +19

Explainability becomes part of the product architecture.

The Interface Should Change

A consequence-aware inbox should probably not look like Gmail.

The traditional interface:

Inbox
------------------------------
Message
Message
Message
Message
Message

could become:

TODAY

🔴 Requires action
   3 messages

🟠 Important
   7 messages

🔵 Informational
   14 messages

⚪ Low consequence
   31 messages

But even this is still too list-oriented.

I would go further.

Imagine:

WHAT MATTERS NOW
─────────────────────────────

PAYMENT DUE TOMORROW
K4,500
[Review]

SERVER CERTIFICATE
Expires in 4 days
[Renew]

CONTRACT
Sarah is waiting for your signature
[Open]

FLIGHT
Departure changed to 19:30
[View]

The inbox becomes a personal operations dashboard.

The Consequence Timeline

Another powerful feature is a timeline.

Instead of showing only messages, show upcoming consequences.

TODAY
│
├── 14:00
│   Contract response expected
│
├── 17:00
│   Project deadline
│
└── 21:00
    Payment cutoff

TOMORROW
│
└── 09:00
    Subscription renewal

NEXT WEEK
│
└── Friday
    Domain expiration

The messages become evidence for these events.

This creates a new relationship:

Messages → Events → Consequences

The inbox is no longer merely storing communication.

It is constructing a model of the user's operational reality.

Architecture

A production implementation could look something like this:

                 ┌───────────────────┐
                 │ Message Sources   │
                 └─────────┬─────────┘
                           │
                           ▼
                 ┌───────────────────┐
                 │ Ingestion Layer   │
                 └─────────┬─────────┘
                           │
                           ▼
                 ┌───────────────────┐
                 │ Message Queue     │
                 └─────────┬─────────┘
                           │
                           ▼
                 ┌───────────────────┐
                 │ Normalizer        │
                 └─────────┬─────────┘
                           │
                           ▼
                 ┌───────────────────┐
                 │ Entity Extraction │
                 └─────────┬─────────┘
                           │
                           ▼
                 ┌───────────────────┐
                 │ Context Engine    │
                 └─────────┬─────────┘
                           │
                           ▼
                 ┌───────────────────┐
                 │ Consequence Engine│
                 └─────────┬─────────┘
                           │
                           ▼
                 ┌───────────────────┐
                 │ Ranking Engine    │
                 └─────────┬─────────┘
                           │
                           ▼
                 ┌───────────────────┐
                 │ Universal Inbox   │
                 └───────────────────┘

Possible infrastructure:

API:
FastAPI / Django / Node.js

Queue:
Kafka / RabbitMQ / NATS

Database:
PostgreSQL

Search:
OpenSearch

Cache:
Redis

Vector storage:
pgvector

ML:
PyTorch

LLM:
Local or API-based model

Workers:
Celery / Temporal / custom workers

The exact technology matters less than the architecture.

The system needs to be event-driven.

Event-Driven Processing

Messages arrive asynchronously.

That makes queues natural.

MessageReceived
      ↓
MessageNormalized
      ↓
EntitiesExtracted
      ↓
ContextResolved
      ↓
ConsequenceCalculated
      ↓
PriorityCalculated
      ↓
InboxUpdated

Each stage can operate independently.

This makes the system scalable.

It also makes reprocessing possible.

Suppose the consequence model improves.

You do not need to re-import every message from every provider.

You can replay events.

Historical Messages
        ↓
New Consequence Model
        ↓
Recomputed Priorities

Event sourcing becomes particularly interesting here.

Privacy Is Not Optional

Now we reach the uncomfortable part.

A universal inbox potentially contains the most sensitive information a person owns.

Financial messages.

Private conversations.

Business communication.

Passwords or password-reset links.

Travel information.

Medical communication.

Contracts.

Personal relationships.

That means this system cannot be treated like another notification app.

Security architecture needs to be fundamental.

Use:

Encryption at rest
Encryption in transit
Least privilege
Scoped OAuth permissions
Token rotation
Audit logging
Tenant isolation
Data minimization
Secure secret storage
Local processing where practical

And perhaps most importantly:

Do not retain everything simply because storage is cheap.

The best privacy architecture may involve processing some information and then discarding the raw content.

Local Intelligence Could Be Powerful

For extremely sensitive messages, local inference becomes attractive.

Imagine:

Email
  ↓
Encrypted local storage
  ↓
Local classifier
  ↓
Entity extraction
  ↓
Consequence score

The user's messages do not necessarily need to leave their device just to determine:

"This message contains a deadline."

This creates an interesting architecture:

Device
 ├── Message cache
 ├── Local model
 ├── User context
 └── Priority engine

Cloud
 ├── Synchronization
 ├── Backup
 └── Optional heavy inference

Privacy and intelligence do not necessarily have to be opposing forces.

The Hardest Problem: False Positives

Imagine receiving 100 "high priority" notifications every day.

The system has failed.

The purpose of prioritization is not to make everything important.

It is to reduce cognitive load.

A useful system might deliberately be conservative.

If the consequence is uncertain:

Unknown consequence
        ↓
Do not interrupt
        ↓
Place in review queue

But if the consequence is potentially severe:

High potential impact
        +
High confidence
        ↓
Interrupt user

This suggests another dimension:

confidence.

A message could have:

Consequence: 92
Confidence: 97%

versus:

Consequence: 95
Confidence: 41%

The second message should not necessarily produce the same notification behavior.

Notification Policy

The inbox and notification system should be separate.

Not every high-priority message deserves a push notification.

For example:

Priority 90
Confidence 98%
Deadline 10 minutes
→ Immediate notification

But:

Priority 90
Confidence 52%
Deadline 3 days
→ Inbox placement

This distinction prevents notification fatigue.

The system should ask:

Does the user need to know this now?

Not merely:

Is this message important?

Consequence Decay

Importance also changes over time.

Consider:

"Meeting tomorrow at 10:00."

At 7 days before the meeting:

Low urgency

At 24 hours:

Medium urgency

At 1 hour:

High urgency

After the meeting:

Historical

Therefore:

Priority(t) = f(consequence, time_remaining)

This means priorities should be recalculated continuously.

An inbox is not a static ranking.

It is a dynamic system.

The Future of the Inbox

Once we build this architecture, something strange happens.

We realize that the inbox is no longer really an inbox.

It becomes a personal consequence engine.

Messages are just inputs.

The actual output is:

What should I know?
What should I do?
What can I ignore?
What is becoming urgent?
What could become expensive?
What deadline is approaching?
What changed?
What requires a decision?

That is a much deeper problem than messaging.

Building the MVP

I would not begin by integrating 30 communication platforms.

Start with three.

For example:

Email
SMS
Calendar

Then build:

Phase 1 — Ingestion

Connect accounts
Normalize messages
Store metadata

Phase 2 — Understanding

Extract entities
Detect deadlines
Detect requests
Classify message types

Phase 3 — Consequence

Estimate impact
Estimate urgency
Identify dependencies
Calculate confidence

Phase 4 — Ranking

Prioritize messages
Group related messages
Create action queues

Phase 5 — Learning

Observe user actions
Collect feedback
Adapt ranking

Phase 6 — Automation

Eventually:

Create task
Schedule reminder
Draft response
Update calendar
Trigger workflow

At that point, the inbox becomes an interface to an intelligent personal operations system.

The Bigger Idea

There is a broader engineering lesson here.

Software often organizes information around the structure of the data.

Emails become emails.

Notifications become notifications.

Transactions become transactions.

Calendar events become calendar events.

But humans do not experience the world as databases.

Humans experience consequences.

A message matters because something might happen.

A payment reminder matters because a service might be suspended.

A server alert matters because customers might lose access.

A contract request matters because a deal might move forward.

A flight notification matters because your travel plans changed.

The message itself is rarely the final object.

The consequence is.

That is why I find consequence-aware inboxes more interesting than simply building another messaging client.

The difficult engineering problem is not collecting messages.

We already know how to do that.

The difficult problem is constructing a system that can look across thousands of pieces of communication and ask:

Which of these could actually change the user's world?

That requires distributed systems.

Natural-language processing.

Machine learning.

Information retrieval.

Event-driven architecture.

Graph modeling.

Security.

Privacy.

Human-computer interaction.

And perhaps most importantly, behavioral understanding.

The ultimate goal is not to build an inbox with better filters.

It is to build a system that reduces the amount of cognitive computation humans have to perform themselves.

Instead of opening 200 messages and asking:

"Which one matters?"

the system should continuously reason about that question in the background.

And when something genuinely matters, it should be able to explain why.

That is the real opportunity.

The next generation of inboxes will not be organized around what arrived first.

They will be organized around what happens if you do nothing.

And that is a completely different way to think about communication software.

Building a Personal Health Data Aggregation Engine

Sun, 20 Sep 2026 14:22:43 +0000

Health data is everywhere.

Your smartwatch knows how many steps you took.

Your phone knows roughly how much you slept.

Your medical provider knows your laboratory results.

Your pharmacy knows what medication you collected.

Your fitness application knows your workouts.

Your nutrition application knows what you ate.

Your blood-pressure monitor knows your readings.

Your doctor has notes.

Your hospital has records.

Your insurance provider has claims.

And somehow, the person all of this data is actually about — you — often has the least convenient view of it.

This is a software architecture problem.

Not merely a healthcare problem.

We have built hundreds of specialized systems that know individual pieces of a person's health story, but we have not always built equally good systems for assembling those pieces into a coherent timeline.

That is where a Personal Health Data Aggregation Engine becomes interesting.

The idea is simple:

Build an engine capable of collecting personal health information from multiple sources, normalizing it, resolving identities, detecting conflicts, constructing timelines, and producing a unified health data model.

But the implementation is not simple.

Health data is messy.

Different systems use different units.

Different providers use different terminology.

Different devices measure the same thing differently.

Some data arrives continuously.

Some arrives once a year.

Some data is precise.

Some data is estimated.

Some information is structured.

Some information exists inside PDFs, documents, images, or clinical notes.

And some data can be wrong.

The interesting engineering challenge is therefore not simply:

"How do I store health data?"

It is:

"How do I construct a trustworthy representation of a person's health from fragmented and heterogeneous data?"

That is a much more interesting system.

1. The Problem

Imagine someone has the following data:

Apple Health
    ├── Steps
    ├── Heart rate
    ├── Sleep
    └── Weight

Fitness tracker
    ├── Workouts
    ├── Calories
    └── Heart rate

Hospital
    ├── Diagnoses
    ├── Laboratory results
    ├── Procedures
    └── Clinical notes

Pharmacy
    ├── Medication
    └── Prescription history

Blood pressure device
    ├── Systolic
    └── Diastolic

Nutrition application
    ├── Meals
    └── Calories

Now suppose we want one timeline.

Something like:

2026-09-01
08:00  Blood pressure: 128/82
09:00  Walk: 4,200 steps
13:00  Meal recorded
18:00  Workout
23:00  Sleep begins

2026-09-02
08:00  Weight: 72.4 kg
10:30  Laboratory result
14:00  Medication recorded

This looks straightforward.

It isn't.

The engine has to answer questions such as:

Are two measurements actually about the same person?
Are two measurements duplicates?
Which source should be trusted?
Are kilograms and pounds being mixed?
Are timestamps using different time zones?
Is a heart-rate reading instantaneous or averaged?
Is a medication prescription the same thing as medication consumption?
Is a diagnosis current or historical?
Is an abnormal laboratory value actually abnormal under the relevant reference range?
Did two systems record the same event independently?
How should missing information be represented?

This is where data engineering meets healthcare.

2. The Architecture

A useful high-level architecture might look like this:

                ┌───────────────────────┐
                │   Data Sources        │
                ├───────────────────────┤
                │ Wearables             │
                │ Medical Records       │
                │ Lab Systems           │
                │ Pharmacy              │
                │ Fitness Apps          │
                │ Manual Input          │
                │ Documents             │
                └───────────┬───────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │ Ingestion Layer       │
                └───────────┬───────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │ Validation Layer      │
                └───────────┬───────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │ Normalization Engine  │
                └───────────┬───────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │ Identity Resolution   │
                └───────────┬───────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │ Deduplication Engine  │
                └───────────┬───────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │ Health Event Store    │
                └───────────┬───────────┘
                            │
                 ┌──────────┴──────────┐
                 ▼                     ▼
        ┌─────────────────┐   ┌──────────────────┐
        │ Timeline Engine │   │ Analytics Engine │
        └─────────────────┘   └──────────────────┘
                 │                     │
                 └──────────┬──────────┘
                            ▼
                ┌───────────────────────┐
                │ Personal Health View  │
                └───────────────────────┘

The important part is that the system should not immediately trust incoming data.

Raw data and canonical data should be separate.

That distinction becomes extremely important.

3. The Raw Data Layer

The first rule of the system should be:

Never destroy the original information.

Suppose an external device sends:

{
  "value": 154,
  "unit": "lb",
  "timestamp": "2026-09-20T07:30:00Z"
}

We might normalize that into:

{
  "value": 69.8532,
  "unit": "kg"
}

But we should still preserve the original record.

Why?

Because normalization logic can change.

Our conversion could contain a bug.

A source could later clarify its semantics.

A user could dispute the transformed value.

Therefore:

RAW DATA
   ↓
IMMUTABLE
   ↓
NORMALIZATION
   ↓
CANONICAL DATA

The raw layer becomes an audit trail.

A database model might look like:

CREATE TABLE raw_events (
    id UUID PRIMARY KEY,
    source_id UUID NOT NULL,
    received_at TIMESTAMP NOT NULL,
    event_type VARCHAR(100),
    payload JSONB NOT NULL,
    checksum VARCHAR(128),
    created_at TIMESTAMP NOT NULL
);

The checksum can help detect duplicate payloads.

4. The Canonical Health Model

The next challenge is creating a universal internal representation.

Instead of letting every source define our database, we create a canonical model.

For example:

{
  "event_id": "evt_123",
  "person_id": "person_001",
  "type": "body_weight",
  "occurred_at": "2026-09-20T07:30:00Z",
  "value": 69.85,
  "unit": "kg",
  "source": "wearable",
  "confidence": 0.94
}

Now different sources can converge into the same model.

For example:

lb
kg
stones

can all become:

kg

Likewise:

bpm
beats/minute
heart_rate

can converge into:

heart_rate

This gives downstream systems a stable interface.

5. Measurements Need Semantics

One of the easiest mistakes is treating all health data as simple numbers.

Consider:

What is it?

It could be:

120 mmHg
120 bpm
120 mg/dL
120 kg
120 steps/min

The number itself is almost meaningless without context.

Therefore, health measurements need metadata.

A useful structure is:

{
  "concept": "blood_glucose",
  "value": 120,
  "unit": "mg/dL",
  "measurement_context": "fasting",
  "occurred_at": "...",
  "source": "...",
  "method": "device"
}

The engine should model:

WHAT
VALUE
UNIT
WHEN
WHERE
HOW
SOURCE
CONTEXT
CONFIDENCE

This is the difference between storing numbers and storing information.

6. Time Is Surprisingly Difficult

Healthcare systems are temporal systems.

And time is messy.

Imagine:

Device A:
2026-09-20 08:00 UTC

Device B:
2026-09-20 10:00 CAT

Those might represent the same moment.

We therefore normalize timestamps internally to UTC.

But we should also preserve:

original_timestamp
original_timezone
normalized_timestamp

Why?

Because local context matters.

Suppose someone traveled from Zambia to Europe.

Their sleep data suddenly appears to shift.

Without timezone information, the engine could interpret the data incorrectly.

Time should therefore be treated as first-class data.

7. Identity Resolution

Now comes one of the hardest problems.

Different systems may identify the same person differently.

One system might have:

user_id = 92831

Another:

patient_id = P-441

Another:

account_id = 7F82

The aggregation engine needs a concept of identity mapping.

                    Person
                      │
          ┌───────────┼───────────┐
          ▼           ▼           ▼
       Patient       User       Device
       P-441        92831       DEV-992

The internal system should not assume that external identifiers are globally meaningful.

Instead:

CREATE TABLE identity_links (
    id UUID PRIMARY KEY,
    person_id UUID NOT NULL,
    source_id UUID NOT NULL,
    external_identifier VARCHAR(255) NOT NULL,
    verified BOOLEAN DEFAULT FALSE
);

This allows the aggregation engine to maintain its own identity graph.

8. Deduplication

Suppose a hospital exports:

Blood pressure: 120/80
09:30

And a connected device independently reports:

Blood pressure: 120/80
09:30

Are these two events?

Maybe.

Maybe not.

A naïve system might simply compare every field.

A smarter system considers:

person
measurement type
timestamp
value
unit
source
device
context

A similarity function could be:

similarity =
    timestamp_similarity
  + measurement_similarity
  + value_similarity
  + source_relationship

Then:

similarity > threshold

could trigger a duplicate candidate.

But I would avoid automatically deleting records.

Instead:

RAW EVENTS
     ↓
DUPLICATE CANDIDATES
     ↓
CLUSTERING
     ↓
CANONICAL EVENT

The system can preserve provenance.

9. Provenance Is Everything

If the engine says:

Weight: 69.8 kg

the user should be able to ask:

Where did this come from?

The answer should be something like:

Source:
Smart Scale

Original:
154 lb

Converted:
69.8532 kg

Recorded:
07:30

Imported:
07:31

Transformation:
lb → kg

This is provenance.

Every canonical value should be traceable back to its origin.

That makes debugging possible.

It also makes the system more trustworthy.

10. Confidence Should Be Explicit

Not every piece of information deserves equal confidence.

Consider:

Manual entry
Wearable sensor
Clinical laboratory
OCR from scanned document
Imported PDF
User estimate

These have different characteristics.

We can therefore attach confidence metadata.

For example:

{
  "value": 72.4,
  "unit": "kg",
  "confidence": {
    "score": 0.91,
    "reason": "device_measurement"
  }
}

But this score should not pretend to be medical certainty.

It is a data-quality signal.

That distinction matters.

The engine is saying:

"How reliable is this data as an imported record?"

It is not saying:

"How medically important is this result?"

Those are different questions.

11. Handling Documents

The system becomes much more interesting when we accept documents.

A user might upload:

blood-test.pdf
hospital-report.pdf
prescription.jpg
medical-record.pdf

Now we need a document processing pipeline.

Document
   ↓
File Validation
   ↓
Text Extraction
   ↓
OCR
   ↓
Entity Extraction
   ↓
Medical Concept Mapping
   ↓
Human Review
   ↓
Canonical Events

For example:

"Weight: 72 kg"

could become:

{
  "type": "body_weight",
  "value": 72,
  "unit": "kg"
}

But extraction should not silently convert uncertain text into trusted facts.

Instead, extracted information can carry:

extraction_confidence
source_location
document_id
page
text_span

That allows the user to inspect the original evidence.

12. The Event Store

At the heart of the system, I would use an event-oriented model.

Something like:

CREATE TABLE health_events (
    id UUID PRIMARY KEY,
    person_id UUID NOT NULL,
    event_type VARCHAR(100) NOT NULL,
    occurred_at TIMESTAMP NOT NULL,
    recorded_at TIMESTAMP,
    payload JSONB NOT NULL,
    source_id UUID,
    confidence NUMERIC,
    created_at TIMESTAMP NOT NULL
);

Why JSONB?

Because health events are heterogeneous.

A heart-rate event looks different from:

medication
lab_result
workout
sleep_session
vaccination
diagnosis
weight
blood_pressure

However, this does not mean everything should become an unstructured JSON blob.

Important searchable properties should remain structured.

For example:

person_id
event_type
occurred_at
source_id

can remain indexed columns.

The flexible details can live in the payload.

13. The Timeline Engine

Once events are normalized, we can construct a timeline.

09:00
│
├── Heart rate
├── Steps
│
10:00
│
├── Medication
│
12:30
│
├── Meal
│
17:00
│
├── Workout
│
23:00
└── Sleep

The timeline engine becomes a powerful abstraction.

Instead of asking:

"Which application contains my data?"

the user asks:

"What happened to my health over time?"

That is a fundamentally better interface.

14. Correlation

Once data is unified, we can begin detecting relationships.

For example:

sleep
   ↓
activity
   ↓
heart rate
   ↓
workout

The system could calculate correlations such as:

sleep duration vs workout frequency

or:

sleep duration vs resting heart rate

But there is an important engineering rule:

Correlation is not causation.

The platform should describe statistical relationships without pretending that they establish medical causality.

For example:

"When sleep duration was lower, recorded activity was also lower."

is fundamentally different from:

"Poor sleep caused reduced activity."

The first is an observation.

The second is a causal claim.

A responsible system must know the difference.

15. The Analytics Layer

We can build a separate analytics engine on top of the canonical event store.

For example:

Daily metrics
Weekly trends
Monthly summaries
Measurement variability
Activity distributions
Sleep patterns
Medication schedules
Lab trajectories

A simple aggregation query might look like:

SELECT
    DATE(occurred_at) AS day,
    AVG((payload->>'value')::numeric) AS average_value
FROM health_events
WHERE event_type = 'heart_rate'
GROUP BY DATE(occurred_at)
ORDER BY day;

This gives us a basic daily view.

But the architecture can go much further.

16. Streaming Data

Wearables can generate huge numbers of events.

Imagine:

1 heart-rate event
every few seconds

The engine could eventually receive:

thousands
or millions
of events

That suggests an event-streaming architecture.

Device
  ↓
API Gateway
  ↓
Message Queue
  ↓
Stream Processor
  ↓
Normalization
  ↓
Event Store

Technologies such as:

Kafka
Redpanda
RabbitMQ
NATS
Redis Streams

could be used depending on requirements.

For a small personal project, however, I would not start there.

Start with:

PostgreSQL
+
background workers
+
REST API

Then introduce streaming when actual workloads justify it.

Architecture should solve real problems, not imaginary scale.

17. Conflict Resolution

Suppose we have:

Source A:
Weight = 72.0 kg

Source B:
Weight = 72.6 kg

Source C:
Weight = 72.2 kg

What should the engine display?

There is no universal answer.

The system could preserve all observations:

72.0
72.6
72.2

and then select a preferred observation based on configurable source precedence.

For example:

verified clinical measurement
        ↓
validated medical device
        ↓
consumer wearable
        ↓
manual entry
        ↓
estimated value

But precedence should be transparent.

Instead of hiding conflicts, the system could say:

3 measurements recorded within 10 minutes.

72.0 kg
72.2 kg
72.6 kg

Transparency can be more valuable than false precision.

18. Privacy Must Be Architectural

Health data is extremely sensitive.

Therefore privacy should not be added after the database is built.

It should influence the architecture from the beginning.

A strong design might include:

Encryption at rest
Encryption in transit
Strict authentication
Fine-grained authorization
Audit logs
Data minimization
Access controls
Key rotation
Secure backups
Data deletion
Export functionality

The database should not simply contain:

users
health_data

and call it security.

Access boundaries matter.

19. Encryption

Sensitive payloads can be encrypted at the application or field level where appropriate.

For example:

database
    ↓
encrypted health payload

Keys should not be stored alongside the encrypted data.

A simplified architecture:

Application
     │
     ▼
Key Management System
     │
     ▼
Encryption Layer
     │
     ▼
Database

The database becomes a storage system rather than the sole guardian of the data.

20. Audit Logging

Every important access should potentially generate an audit event.

For example:

{
  "actor": "user_123",
  "action": "READ",
  "resource": "health_event_789",
  "timestamp": "...",
  "ip": "...",
  "result": "allowed"
}

For a system handling sensitive information, auditability is not just an enterprise feature.

It is part of the architecture.

21. Data Deletion Is Harder Than It Looks

Suppose a user says:

Delete my health data.

The system might have:

Primary database
Analytics database
Search index
Object storage
Cache
Backups
Logs
Derived datasets

Deleting one database row does not necessarily delete everything.

Therefore we need a data lifecycle architecture.

Created
   ↓
Processed
   ↓
Stored
   ↓
Derived
   ↓
Archived
   ↓
Deleted

Deletion workflows need to understand dependencies.

If an event is deleted, derived records based exclusively on that event may also need deletion or recomputation.

22. API Design

The API could expose resources like:

GET /api/v1/health/events
GET /api/v1/health/timeline
GET /api/v1/health/measurements
GET /api/v1/health/sources
POST /api/v1/health/import
DELETE /api/v1/health/events/{id}

A timeline query might look like:

GET /api/v1/health/timeline?from=2026-09-01&to=2026-09-20

Response:

{
  "events": [
    {
      "type": "weight",
      "value": 72.4,
      "unit": "kg",
      "occurred_at": "2026-09-20T07:30:00Z"
    }
  ]
}

The API should expose provenance rather than hiding it.

23. Building the System Incrementally

I would not attempt to build the entire platform at once.

Version 1:

Manual data entry
+
PostgreSQL
+
REST API
+
Timeline

Version 2:

CSV imports
+
JSON imports
+
Unit normalization

Version 3:

Wearable integrations
+
device sources

Version 4:

Document ingestion
+
OCR

Version 5:

Deduplication
+
identity resolution

Version 6:

Analytics
+
trend detection

Version 7:

Event streaming
+
large-scale ingestion

This progression keeps the architecture understandable.

24. A Possible Technology Stack

For a serious implementation, I might choose:

Backend:
Python / Django / FastAPI

Database:
PostgreSQL

Cache:
Redis

Queue:
Celery / Redis
or
NATS / Kafka

Object Storage:
S3-compatible storage

Search:
OpenSearch

Analytics:
Python / Pandas / Polars

ML:
PyTorch

Frontend:
React / TypeScript

Mobile:
React Native

But the stack is not the important part.

The architecture is.

You could build the same conceptual system using:

Rust
Go
Node.js
Java
.NET
PHP

The difficult part is designing the data model correctly.

25. A Better Internal Architecture

I would separate the system into bounded components:

                  ┌──────────────────┐
                  │ API Gateway      │
                  └────────┬─────────┘
                           │
       ┌───────────────────┼───────────────────┐
       ▼                   ▼                   ▼
┌─────────────┐     ┌──────────────┐    ┌──────────────┐
│ Ingestion   │     │ Identity     │    │ Access       │
│ Service     │     │ Service      │    │ Service      │
└──────┬──────┘     └──────┬───────┘    └──────┬───────┘
       │                   │                   │
       └───────────────────┼───────────────────┘
                           ▼
                  ┌─────────────────┐
                  │ Health Event    │
                  │ Store           │
                  └────────┬────────┘
                           │
             ┌─────────────┼─────────────┐
             ▼             ▼             ▼
        Timeline       Analytics      Search

This creates clear boundaries.

The ingestion service should not become the analytics service.

The analytics engine should not own authentication.

The timeline engine should not know how a wearable API works.

Separation creates flexibility.

26. The Interesting Part: Personal Health Graph

Eventually, we can move beyond a simple timeline.

We can construct a graph.

Person
 │
 ├── Measurement
 │      └── Device
 │
 ├── Medication
 │
 ├── Laboratory Result
 │
 ├── Activity
 │
 ├── Sleep Session
 │
 └── Clinical Encounter

Then relationships become explicit.

For example:

Medication
     │
     └── prescribed_during
              │
              ▼
       Clinical Encounter

Or:

Lab Result
    │
    └── measured_at
             │
             ▼
       Laboratory

The graph becomes a semantic layer over the event store.

27. AI Can Sit on Top of the System

This is where things get interesting.

Once the data is normalized, an AI assistant could answer questions such as:

"Show me my weight trend."

"What measurements did I record last month?"

"Summarize my recent health records."

"Find duplicate measurements."

"Show laboratory results over time."

But AI should not become the source of truth.

The architecture should look like:

Health Data
     ↓
Canonical Data Model
     ↓
Query Engine
     ↓
Evidence Retrieval
     ↓
AI

not:

AI
 ↓
Guess what happened

The AI should retrieve structured evidence and explain it.

Every answer should ideally be traceable to underlying records.

28. Retrieval-Augmented Health Intelligence

For documents and clinical notes, we could build a retrieval layer.

Medical documents
      ↓
Chunking
      ↓
Embeddings
      ↓
Vector database
      ↓
Semantic retrieval
      ↓
AI assistant

But structured measurements should remain structured.

Do not put:

weight = 72.4 kg

into a vector database and call the problem solved.

Use the right storage mechanism for the right data.

Structured facts → relational database

Documents → object storage

Semantic text → vector index

Events → event store

Hybrid architecture wins.

29. The Real Engineering Challenge

The hardest part of this system is not building a dashboard.

It is creating a trustworthy data pipeline.

The dashboard is the easy part.

The hard questions are:

Where did this data come from?

Was it transformed?

Was it duplicated?

Was the identity verified?

What unit was used?

What timezone was used?

When was it measured?

When was it imported?

What changed?

Who accessed it?

Can we reproduce the transformation?

Those questions define the quality of the system.

30. A Minimal Prototype

A surprisingly powerful prototype could be built with only:

FastAPI
PostgreSQL
Redis
React

Start with three event types:

weight
heart_rate
blood_pressure

Then implement:

POST /events
GET /events
GET /timeline

Example:

POST /events

{
  "type": "weight",
  "value": 72.4,
  "unit": "kg",
  "occurred_at": "2026-09-20T07:30:00Z"
}

The backend validates it.

Then:

raw event
   ↓
validation
   ↓
normalization
   ↓
canonical event
   ↓
database

The frontend displays:

September 20

07:30
Weight
72.4 kg

That tiny system is already the beginning of a personal health data platform.

31. What Makes the Project Advanced?

Because it combines several difficult areas of software engineering:

Distributed data ingestion
Data normalization
Event processing
Identity resolution
Deduplication
Temporal databases
Document processing
Search
Analytics
Privacy
Encryption
AI
Data provenance

It is not simply a CRUD application.

It is a data infrastructure project.

And that makes it especially interesting.

32. The Bigger Idea

The long-term vision is not another fitness dashboard.

It is a personal data infrastructure layer.

Imagine a world where a person owns a unified representation of their own health information.

Not:

App A knows this.
App B knows that.
Hospital C knows something else.

But:

                PERSONAL HEALTH DATA
                         │
        ┌────────────────┼────────────────┐
        ▼                ▼                ▼
   Wearables         Providers         Devices
        │                │                │
        └────────────────┼────────────────┘
                         ▼
               Aggregation Engine
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
           Timeline   Analytics    Search
              │          │          │
              └──────────┼──────────┘
                         ▼
                  Personal Interface

The user becomes the center of the architecture.

That is the interesting shift.

Conclusion

Building a Personal Health Data Aggregation Engine is fundamentally a problem of turning fragmented information into a coherent personal data model.

The challenge begins with ingestion.

Then comes normalization.

Then identity.

Then deduplication.

Then provenance.

Then timelines.

Then analytics.

Then privacy.

Then intelligence.

The deeper lesson is that health data is not simply a collection of numbers.

It is a collection of observations made by different systems, at different times, using different methods, with different levels of confidence.

A good aggregation engine respects that complexity.

It does not blindly merge everything.

It preserves the original data.

It records transformations.

It exposes provenance.

It represents uncertainty.

It separates observations from interpretations.

And it gives the person a coherent view of information that was previously scattered across dozens of systems.

That architecture can eventually become much bigger than a dashboard.

It can become a personal data operating system.

And that is the part I find most interesting about projects like this.

The future of software is not always about creating another application.

Sometimes it is about building the infrastructure that allows many applications to finally understand the same person.

The real product is not the dashboard.

The real product is the data layer underneath it.

Building a Food Waste Prediction System

Sat, 19 Sep 2026 21:16:03 +0000

Food waste looks like a simple problem.

A restaurant buys 100 kilograms of vegetables.

Twenty kilograms eventually end up in the bin.

A supermarket orders 500 units of a product.

Seventy units expire before they are sold.

A household buys groceries for the week.

By Sunday, some of them are no longer usable.

The obvious solution sounds simple:

Buy less food.

But that is not really the engineering problem.

The real problem is uncertainty.

A business does not know exactly how much food it will sell tomorrow.

A supermarket does not know exactly how many customers will walk through its doors.

A restaurant does not know exactly how many people will order chicken, rice, vegetables, pizza, or dessert.

A household cannot perfectly predict its own behavior.

Food waste is therefore not merely an inventory problem.

It is a prediction problem sitting on top of an inventory system.

And that makes it interesting.

Because once we stop thinking about food waste as "food that was thrown away" and start thinking about it as future waste that can be predicted before it happens, we can build something much more powerful.

We can build a Food Waste Prediction System.

A system that observes inventory, sales, expiration dates, demand patterns, weather, holidays, promotions, purchasing behavior, storage duration, and other signals — and attempts to answer one question:

What food are we likely to waste, when will we waste it, and why?

That question turns a passive inventory system into an intelligent decision system.

1. The Problem

Traditional inventory software usually tells us what has already happened.

We have:

Current Stock
      ↓
Stock Movements
      ↓
Sales
      ↓
Remaining Inventory

That is useful.

But it is reactive.

Imagine a supermarket has:

Product: Fresh Milk
Current Stock: 120 units
Expiration: 4 days
Average Daily Sales: 20 units

At first glance, 120 units sounds fine.

But the prediction system should immediately calculate:

Expected sales before expiration
= 20 × 4
= 80 units

Potential surplus:

120 - 80 = 40 units

Now we have something interesting.

The system can estimate that approximately 40 units may remain when the product expires.

Instead of discovering the waste afterward, we can act before it happens.

Possible actions include:

discounting the product,
promoting it,
moving it to another branch,
changing purchasing quantities,
adjusting future orders,
recommending recipes,
bundling products,
donating eligible food,
or changing inventory allocation.

The prediction itself does not eliminate waste.

The prediction creates time to intervene.

That distinction matters.

2. The System We Are Actually Building

A useful architecture might look like this:

                 ┌──────────────────┐
                 │   POS / Sales    │
                 └────────┬─────────┘
                          │
                 ┌────────▼─────────┐
                 │ Inventory System │
                 └────────┬─────────┘
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
        ▼                 ▼                 ▼
   Expiration         Purchasing         External
     Data               Data               Signals
        │                 │                 │
        └─────────────────┼─────────────────┘
                          ▼
                ┌─────────────────────┐
                │ Feature Engineering │
                └──────────┬──────────┘
                           ▼
                ┌─────────────────────┐
                │ Demand Prediction    │
                │ Engine               │
                └──────────┬──────────┘
                           ▼
                ┌─────────────────────┐
                │ Waste Risk Engine    │
                └──────────┬──────────┘
                           ▼
                ┌─────────────────────┐
                │ Recommendation      │
                │ Engine               │
                └──────────┬──────────┘
                           ▼
                ┌─────────────────────┐
                │ Dashboard / Alerts   │
                └─────────────────────┘

The architecture has an important property.

The machine-learning model is not the entire product.

It is only one component.

The real system combines:

data + forecasting + inventory intelligence + risk analysis + decision support.

That is where the engineering gets interesting.

3. Start With the Data

A prediction system is only as useful as the information it receives.

For every inventory item, we want information such as:

product_id
category
quantity
unit
purchase_date
expiration_date
storage_location
supplier
cost
selling_price

Then we need historical sales:

sale_id
product_id
timestamp
quantity
price
store_id

We can derive daily demand:

date
product_id
units_sold

But raw sales are not enough.

Suppose we see:

Monday: 20
Tuesday: 18
Wednesday: 21
Thursday: 19
Friday: 50
Saturday: 63
Sunday: 42

A simple average gives us:

33.3 units/day

That average can be misleading.

The product behaves differently on weekends.

Therefore, our model needs to understand context.

4. Time Is a Feature

One of the first mistakes in forecasting systems is treating time as nothing more than a timestamp.

Time contains structure.

For example:

hour
day_of_week
week_of_month
month
season
holiday
payday_period
school_term

A supermarket may sell more products during weekends.

A restaurant may have stronger demand on Friday evenings.

A bakery may sell more bread in the morning.

A grocery store may experience different purchasing behavior around public holidays.

Therefore:

timestamp

should become multiple features.

For example:

day_of_week = Saturday
is_weekend = true
month = September
is_holiday = false
hour = 18

The model now has context.

5. Demand Is Not Constant

The simplest forecasting algorithm is:

future_demand = average(historical_demand)

It is easy.

It is also often wrong.

Imagine:

Monday    20
Tuesday   21
Wednesday 19
Thursday  22
Friday    50
Saturday  65
Sunday    40

The demand distribution is not stationary.

A better baseline could use weighted averages.

For example:

predicted_demand =
    0.50 × recent_average
  + 0.30 × same_day_average
  + 0.20 × seasonal_average

This is still relatively simple.

But it is already better than blindly using a global average.

6. The Expiration Clock

Here is where food prediction becomes different from ordinary demand forecasting.

We do not only care about:

How much will we sell?

We care about:

How much will we sell before this specific inventory expires?

Suppose:

Current stock = 100
Predicted daily demand = 18
Days until expiration = 4

Expected consumption:

18 × 4 = 72

Potential waste:

100 - 72 = 28

We can define:

waste_surplus =
max(stock - predicted_consumption_before_expiry, 0)

This gives us a basic waste estimate.

But real systems should not stop here.

7. Probability Matters

Predictions are uncertain.

Suppose the model predicts:

Expected demand = 18 units/day

That does not mean exactly 18 units will be sold tomorrow.

It might be:

Therefore, instead of predicting only a single number, we can predict a distribution.

For example:

P(demand < 50) = 10%
P(demand < 60) = 30%
P(demand < 70) = 70%
P(demand < 80) = 90%

Now we can reason about risk.

Suppose:

Stock = 100
Expiration horizon = 4 days

The model estimates:

Expected consumption = 72
90th percentile consumption = 82

There is still meaningful probability that inventory will remain.

That allows us to calculate a waste-risk score.

8. Building the Waste Risk Engine

A simple version might calculate:

days_to_expiry
predicted_consumption
current_stock
surplus

Then:

waste_ratio =
surplus / current_stock

For example:

Current stock = 100
Expected consumption = 70
Surplus = 30

Waste ratio = 0.30

The system could classify:

0.00 - 0.10 → Low risk
0.10 - 0.25 → Moderate risk
0.25 - 0.50 → High risk
> 0.50      → Critical risk

These thresholds should not be treated as universal truths.

They should be configurable.

Different businesses have different economics.

9. Cost Makes the Prediction More Useful

Not all waste has the same financial impact.

Throwing away:

100 units × $0.20

is different from:

100 units × $15

Therefore, the system should estimate financial exposure.

waste_cost =
predicted_waste × unit_cost

Now we can rank potential waste events by financial impact.

For example:

Milk
Expected waste: 40
Cost: $1
Risk value: $40

Cheese
Expected waste: 12
Cost: $8
Risk value: $96

The cheese has less physical waste but greater financial exposure.

That changes the intervention priority.

10. But Money Is Not the Only Variable

A sophisticated system should eventually consider multiple dimensions.

For example:

financial_loss
food_quantity
expiration_urgency
environmental_impact
donation_eligibility
storage_cost
discountability

We can construct a broader waste impact score:

impact =
    financial_loss
  + environmental_factor
  + urgency_factor
  + disposal_cost

Again, this does not have to become one mysterious machine-learning number.

Transparent components are often better.

A business manager should be able to ask:

Why did the system flag this product?

And the system should answer:

High Waste Risk

Product: Fresh Tomatoes

Reason:
• 82 units currently available
• 5 days until estimated spoilage
• expected consumption: 43 units
• historical weekend demand is lower
• recent purchasing increased by 28%
• predicted surplus: 39 units

Estimated exposure: K1,170

That is far more useful than:

AI Score: 0.87

11. Feature Engineering

The quality of the prediction will depend heavily on the features.

Potential features include:

Inventory features

current_stock
days_to_expiry
stock_age
batch_size
storage_location

Sales features

sales_last_1_day
sales_last_7_days
sales_last_30_days
sales_same_weekday
sales_same_month
sales_velocity

Price features

current_price
historical_price
discount_percentage

Business features

promotion_active
holiday_period
payday_period
store_location

Supplier features

supplier_lead_time
minimum_order_quantity
delivery_frequency

Environmental features

Depending on the domain and available data:

temperature
rainfall
humidity
season

These can matter because consumer behavior and food spoilage can be affected by environmental conditions.

12. The Model

There are many possible approaches.

We could begin with:

Moving Average

Then progress toward:

Exponential Smoothing

Then:

ARIMA

or tree-based machine-learning models such as:

Random Forest
Gradient Boosting
XGBoost
LightGBM

For larger systems, we might explore:

LSTM
Temporal CNNs
Transformers
Temporal Fusion Transformers

But there is a lesson I keep coming back to when building intelligent systems:

Do not start with the most complicated model.

Start with a baseline.

If a moving average gives acceptable predictions, that is valuable information.

If a gradient-boosting model only improves accuracy slightly, we can measure whether that improvement justifies the additional complexity.

Engineering is not a competition to use the largest neural network.

It is a process of creating useful systems.

13. Training the Model

We could construct training data like:

product_id
date
day_of_week
month
stock
sales_1d
sales_7d
sales_30d
price
discount
holiday
temperature
rainfall
future_sales

The target becomes:

future_sales

For example:

features at Monday
          ↓
predict Tuesday demand

For multi-day forecasting:

Monday
  ↓
Tuesday
Wednesday
Thursday
Friday

We can generate predictions across the expiration horizon.

That is important.

If an item expires in three days, we care about:

Demand Tuesday
+ Demand Wednesday
+ Demand Thursday

not simply tomorrow's demand.

14. Batch-Level Intelligence

One of the more interesting extensions is moving from product-level prediction to batch-level prediction.

Imagine:

Product: Yogurt

Batch A
Received: September 15
Expires: September 20
Quantity: 100

Batch B
Received: September 18
Expires: September 25
Quantity: 150

A product-level system sees:

Yogurt stock = 250

A batch-aware system sees:

100 units expire sooner
150 units expire later

That distinction is enormous.

The system can recommend:

Sell Batch A first.

This is essentially an intelligent version of FEFO — First Expired, First Out.

15. Predicting Waste Before Purchasing

This is where the system becomes even more interesting.

Most inventory systems analyze waste after inventory exists.

But historical data can be used to influence purchasing.

Suppose the system observes:

Average monthly demand = 800 units

Purchased = 1,100 units

Average waste = 230 units

The system can detect a recurring purchasing pattern.

Maybe the business is consistently over-ordering.

Instead of saying:

You wasted 230 units.

The system can eventually say:

Based on historical demand and current conditions, ordering 1,100 units is likely to create excess inventory.

That moves the system upstream.

The goal becomes:

prevent waste rather than merely report waste.

16. Anomaly Detection

Not all waste follows normal patterns.

Suppose a store normally wastes:

20–30 units/week

Suddenly:

125 units

That should trigger an investigation.

Maybe:

a refrigerator failed,
a supplier delivered damaged products,
inventory counts were incorrect,
demand unexpectedly collapsed,
employees stored food incorrectly,
a promotion failed,
or the system received incorrect data.

We can build an anomaly detector around historical waste.

For example:

expected_waste = 25
actual_waste = 120

anomaly_score = high

The system can then ask:

What changed?

This is where machine learning becomes less about prediction and more about system observability.

17. Recommendations

Prediction without action is just analytics.

The next layer should recommend interventions.

For example:

Product:
Bananas

Risk:
High

Predicted surplus:
35 kg

Days to spoilage:
2

Recommended actions:

1. Apply 20% discount
2. Promote bundle
3. Move inventory to Branch B
4. Prioritize product in staff recommendations
5. Evaluate next purchase quantity

The recommendation engine could use business rules.

For example:

if days_to_expiry <= 2 and waste_risk > 0.7:
    recommend_discount()

Another:

if branch_a_surplus > threshold:
    find_branch_with_high_demand()

This is where the prediction engine becomes operational.

18. Dynamic Discounts

A particularly interesting application is automated markdown pricing.

Imagine:

Product: Sandwich
Price: K45
Expiration: 8 hours
Stock: 70
Predicted demand: 30

The system predicts:

40 units at risk

Instead of waiting until the food becomes unsellable, the system might recommend:

20% discount

The price becomes:

K36

Demand may increase.

Now the business has transformed potential waste into additional revenue.

But the system should learn from the result.

If:

20% discount → sales increased 60%

that becomes a future signal.

Over time, the system can learn:

discount level
        ↓
demand response
        ↓
waste reduction
        ↓
revenue impact

Now we are building a feedback system.

19. The Feedback Loop

This is the part I find most interesting.

A static prediction system looks like:

Data → Model → Prediction

A real intelligent system looks more like:

Data
 ↓
Prediction
 ↓
Recommendation
 ↓
Action
 ↓
Outcome
 ↓
New Data
 ↓
Model Improvement

Suppose the system recommends a 15% discount.

The business applies it.

Sales increase.

Waste decreases.

That outcome becomes training data.

The system learns:

For Product Category X
with 1–2 days remaining
and waste risk > 60%

15% discount historically produced:
+32% demand
-41% waste

Now the system is becoming adaptive.

20. Architecture

A production implementation could look like:

                ┌─────────────────┐
                │ POS / Mobile App│
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │ API Gateway     │
                └────────┬────────┘
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
        Inventory     Sales      Orders
          Service     Service     Service
              │          │          │
              └──────────┼──────────┘
                         ▼
                  Event Stream
                         │
                         ▼
                ┌─────────────────┐
                │ Feature Store    │
                └────────┬────────┘
                         ▼
                ┌─────────────────┐
                │ Forecast Engine  │
                └────────┬────────┘
                         ▼
                ┌─────────────────┐
                │ Waste Predictor  │
                └────────┬────────┘
                         ▼
                ┌─────────────────┐
                │ Recommendation   │
                │ Engine           │
                └────────┬────────┘
                         ▼
                ┌─────────────────┐
                │ Dashboard        │
                └─────────────────┘

For a smaller implementation, this does not need to be microservices.

A modular monolith can work perfectly well.

For example:

Django
PostgreSQL
Celery
Redis
Python
scikit-learn
Pandas

or:

FastAPI
PostgreSQL
Redis
Python
PyTorch

The architecture should follow actual scale.

21. Database Design

A simplified schema could include:

products
---------
id
name
category
unit
shelf_life_days

inventory_batches
-----------------
id
product_id
quantity
received_at
expires_at
unit_cost
location_id

sales
-----
id
product_id
quantity
price
sold_at
location_id

waste_events
------------
id
batch_id
quantity
reason
recorded_at

predictions
-----------
id
product_id
prediction_date
predicted_demand
confidence
created_at

waste_predictions
-----------------
id
batch_id
predicted_waste
risk_score
financial_exposure
created_at

This structure lets us preserve historical predictions.

That is important.

We should not overwrite predictions.

We need to know:

What did the system believe at the time?

That allows us to evaluate the model later.

22. Model Evaluation

Accuracy is not enough.

Suppose our model predicts demand with:

MAPE = 8%

That sounds impressive.

But perhaps it still produces significant food waste.

We should evaluate business outcomes.

Useful metrics include:

Forecast MAE
Forecast RMSE
Forecast MAPE
Waste reduction %
Financial loss reduction
Expired inventory reduction
Discount recovery rate

For example:

Before system:
Waste = 12%

After system:
Waste = 7%

That is more meaningful to the business than a machine-learning benchmark alone.

23. Measuring the Right Thing

There is another subtle problem.

If the system recommends discounts and waste falls, we cannot automatically conclude that the model caused all of the improvement.

Other things may have changed.

Maybe:

customer demand increased,
staff behavior changed,
purchasing changed,
seasonality changed.

Therefore, experimentation matters.

A business could run:

Control stores
vs
Prediction-enabled stores

Then compare outcomes.

This creates a stronger evaluation framework.

The system becomes something we can measure scientifically.

24. Cold Start

Every prediction system has a problem:

What happens when there is no historical data?

Suppose a new product is introduced.

There are zero sales records.

The model cannot magically know demand.

We can use:

category-level demand
similar-product demand
store-level demand
seasonal patterns
supplier information
manual estimates

For example:

New Product:
Organic Mango Juice

Similar products:
Apple Juice
Orange Juice
Pineapple Juice

We can estimate initial demand from similar products.

As actual sales arrive, the model gradually replaces assumptions with evidence.

25. Human-in-the-Loop

The system should not blindly control the business.

Managers know things that data may not capture.

For example:

"The supplier delivered a different product size this week."

Or:

"A nearby event is expected to increase demand."

The interface should allow users to override predictions.

For example:

Predicted demand: 120

Manager override:
Expected demand: 180

Reason:
Local event

The override itself becomes useful data.

The system can eventually learn that certain external events influence demand.

26. Explainability

If an algorithm says:

Waste Risk: 91%

the user will naturally ask:

Why?

A useful explanation could be:

Waste risk is high because:

• inventory is 2.4× normal
• demand decreased 18% over the last 14 days
• 3 days remain before expiration
• weekend demand is historically lower
• current discount is 0%

That is explainable.

It gives the user something they can act on.

27. Building the MVP

I would not begin with weather APIs, transformers, reinforcement learning, computer vision, and twenty microservices.

Start with something smaller.

Version 1

Build:

Products
Inventory
Sales
Expiration dates
Basic demand forecasting
Waste-risk calculation
Dashboard

Then:

Version 2

Add:

Batch tracking
FEFO recommendations
Alerts
Financial waste estimates

Then:

Version 3

Add:

Machine-learning forecasts
Dynamic discounts
Branch redistribution

Then:

Version 4

Add:

External signals
Automated purchasing recommendations
Adaptive pricing
Continuous learning

This creates a realistic development path.

28. A Simple Prediction Function

At the beginning, our logic could be surprisingly simple:

def predict_waste(stock, daily_demand, days_to_expiry):
    expected_consumption = daily_demand * days_to_expiry

    surplus = max(
        stock - expected_consumption,
        0
    )

    waste_ratio = (
        surplus / stock
        if stock > 0
        else 0
    )

    return {
        "expected_consumption": expected_consumption,
        "surplus": surplus,
        "waste_ratio": waste_ratio
    }

This is not sophisticated AI.

And that is okay.

It gives us a baseline.

Then we can replace:

daily_demand

with a forecasting model.

The rest of the system can remain largely unchanged.

That is good architecture.

29. The Interesting Part Is Not the Model

This is probably the most important lesson.

When people hear:

Food Waste Prediction System

they may immediately think:

Machine learning.
Neural networks.
AI.

But the hardest problems are often elsewhere.

How do we accurately track inventory?

How do we handle expired batches?

How do we distinguish waste from spoilage?

How do we deal with missing sales?

How do we handle inventory corrections?

How do we account for products transferred between branches?

How do we handle promotions?

How do we evaluate predictions?

How do we explain recommendations?

How do we make the system useful enough that people actually act on it?

These are systems problems.

The model is only one part.

30. From Prediction to Intelligence

Eventually, the system can evolve beyond predicting waste.

Imagine opening the dashboard in the morning.

Instead of seeing:

Inventory
Sales
Waste

you see:

TODAY'S WASTE RISK

K18,420 potential exposure

37 products require attention.

Highest priorities:

1. Fresh produce
   2 days remaining
   42% surplus

2. Dairy
   3 days remaining
   31% surplus

3. Prepared meals
   8 hours remaining
   57% surplus

Then the system provides actions:

Discount
Redistribute
Promote
Bundle
Donate
Reduce next order

Now the software is no longer simply describing the business.

It is helping the business make decisions.

31. The Bigger Idea

Food waste is a particularly interesting engineering problem because it exists at the intersection of multiple systems:

Commerce
+
Inventory
+
Time
+
Human behavior
+
Economics
+
Machine learning
+
Optimization

And that combination makes it much more interesting than a simple CRUD application.

You can start with a database.

Then add analytics.

Then forecasting.

Then prediction.

Then recommendations.

Then feedback.

Eventually, you have an adaptive system that continuously observes what happens in the real world and improves its decisions.

That pattern can be applied far beyond food.

The same architecture can predict:

unused inventory
energy waste
warehouse capacity
resource consumption
delivery failures
equipment downtime

The deeper engineering principle is this:

Don't wait for waste to become an event. Build systems that recognize the probability of waste while there is still time to change the outcome.

That is the difference between reporting and prediction.

A traditional inventory system tells you:

"You wasted 40 kilograms."

A predictive system tells you:

"You are likely to waste 40 kilograms unless something changes."

An intelligent system goes one step further:

"You are likely to waste 40 kilograms. Here is why. Here are the actions available. Here is what happened the last time you took each action."

That is where software becomes genuinely useful.

Because the ultimate goal is not to build a model that predicts waste.

The goal is to build a system that makes waste harder to happen.

And that is a much more interesting engineering problem.

Building a Universal Receipt Intelligence System

Sat, 19 Sep 2026 21:06:15 +0000

A receipt looks like a simple piece of paper.

At first glance, it contains a few obvious things:

A merchant name.

A date.

Some products.

A few numbers.

A tax amount.

A total.

Then the paper goes into a pocket, a wallet, a drawer, a bag, or a trash bin.

But technically, a receipt is something much more interesting.

It is an unstructured data record describing a financial transaction.

And the world produces billions of these records.

The problem is that receipts were never really designed for machines.

They were designed for humans.

A human can look at:

TOTAL: K184.50

and immediately understand what it means.

A computer does not.

A computer sees pixels.

Maybe it sees:

MARTIN'S
SUPERMARKET

19/09/26

Milk       24.50
Bread      18.00
Rice       75.00
Soap       32.00

VAT        35.00

TOTAL     184.50

But another receipt might look like this:

SHOPRITE
LUANSHYA

TXN: 8291938
19-09-2026 18:42

2 x Bread     K18.00
1 x Milk      K24.50

SUBTOTAL      K60.50
VAT           K9.08
CASH          K100
CHANGE        K30.42

And another might be:

AMZN.COM
Order #114-...
Sep 19

Wireless Mouse
$19.99

Shipping
$4.99

Tax
$2.10

Total
$27.08

Same concept.

Completely different structure.

This creates an interesting engineering problem:

How do we build a system that can look at almost any receipt in the world and turn it into reliable structured financial data?

That is what I call a Universal Receipt Intelligence System.

Not simply OCR.

Not simply receipt scanning.

Not simply an expense tracker.

The objective is much bigger.

We want to build a system that can understand receipts.

The Problem With Receipt Data

Traditional software expects structured input.

An API might receive:

{
  "merchant": "Example Store",
  "date": "2026-09-19",
  "total": 184.50,
  "currency": "ZMW"
}

That is easy.

The database knows what each field means.

But real-world receipts are not APIs.

They are documents.

Sometimes they are photographs.

Sometimes they are screenshots.

Sometimes they are PDFs.

Sometimes they are thermal-print receipts.

Sometimes they are handwritten.

Sometimes they are damaged.

Sometimes they are folded.

Sometimes they are extremely long.

Sometimes they contain multiple transactions.

Sometimes the receipt is printed in a language the system has never seen before.

Sometimes the merchant uses strange abbreviations.

Sometimes prices are aligned using spaces.

Sometimes they are aligned using dots.

Sometimes there are no labels at all.

And sometimes the total appears several times.

For example:

TOTAL
SUBTOTAL
VAT
TOTAL
TENDERED
CHANGE

A naive parser can easily mistake the wrong number for the purchase total.

This is why receipt intelligence is fundamentally a document understanding problem.

The Architecture

I would design the system as a pipeline.

                ┌───────────────────┐
                │ Receipt Image/PDF │
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ Image Preprocessor│
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ Document Detector │
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ OCR / Text Engine │
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ Layout Analyzer   │
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ Semantic Parser   │
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ Entity Resolver   │
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ Validation Engine │
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ Confidence Engine │
                └─────────┬─────────┘
                          │
                          ▼
                ┌───────────────────┐
                │ Structured Receipt│
                └───────────────────┘

Each stage solves a different problem.

The important part is that we do not ask one model to magically understand everything.

We decompose the problem.

Step One: Receipt Ingestion

The system starts with an input layer.

It should accept:

JPEG
PNG
WEBP
PDF
HEIC
Screenshots
Camera captures
Email attachments
Scanned documents

An API might look like:

POST /v1/receipts
Content-Type: multipart/form-data

The response initially should not claim that the receipt has been completely understood.

Instead:

{
  "receipt_id": "rcpt_8fd91",
  "status": "processing"
}

The actual processing can happen asynchronously.

That matters because OCR and AI inference can be expensive.

A production system might use:

API
 │
 ▼
Object Storage
 │
 ▼
Message Queue
 │
 ├── OCR Worker
 ├── Image Worker
 ├── Parser Worker
 └── Validation Worker

This allows the system to scale horizontally.

Step Two: Image Intelligence

OCR quality depends heavily on image quality.

A photograph taken at an angle is different from a scanned receipt.

A thermal receipt photographed under poor lighting is different again.

Therefore, before OCR, we need preprocessing.

Typical operations include:

Resize
Deskew
Rotate
Crop
Denoise
Contrast normalization
Perspective correction
Shadow removal
Blur detection
Background removal

Suppose the camera captures:

       /
      /
     / RECEIPT
    /

Perspective correction should transform it into something closer to:

┌──────────────────┐
│      RECEIPT     │
│                  │
│ Product     20   │
│ Product     40   │
│                  │
│ TOTAL       60   │
└──────────────────┘

This seems like a small detail.

It is not.

Better input quality can dramatically improve everything downstream.

Step Three: OCR Is Only the Beginning

After preprocessing, we perform OCR.

But ordinary OCR gives us text.

We need more than text.

We need text plus geometry.

For example:

{
  "text": "TOTAL",
  "x": 420,
  "y": 920,
  "width": 90,
  "height": 30
}

And:

{
  "text": "184.50",
  "x": 650,
  "y": 920,
  "width": 100,
  "height": 30
}

The relationship between those two objects is valuable.

They appear on the same line.

They are horizontally aligned.

They are close together.

Therefore:

TOTAL → 184.50

is a strong relationship.

This is why receipt intelligence should preserve the original spatial information.

Step Four: Layout Understanding

Receipts have visual structure.

Even when the text changes, the structure often remains recognizable.

A receipt might contain:

HEADER
MERCHANT
ADDRESS

TRANSACTION METADATA

ITEMS

SUBTOTAL

TAX

DISCOUNT

TOTAL

PAYMENT

FOOTER

We can model this as a document tree:

Receipt
├── Merchant
├── Address
├── Transaction
│   ├── Date
│   ├── Time
│   └── Transaction ID
├── Items
│   ├── Item
│   ├── Item
│   └── Item
├── Financial Summary
│   ├── Subtotal
│   ├── Tax
│   ├── Discount
│   └── Total
└── Payment
    ├── Method
    ├── Tendered
    └── Change

Now the problem becomes easier.

We are not simply extracting strings.

We are reconstructing a document.

Step Five: Merchant Detection

One of the first important entities is the merchant.

The system might detect:

SHOPRITE

But that does not necessarily mean the canonical merchant entity is exactly:

SHOPRITE

We may want:

{
  "merchant": {
    "display_name": "Shoprite",
    "normalized_name": "shoprite",
    "confidence": 0.99
  }
}

Over time, merchant normalization becomes powerful.

For example:

SHOPRITE #123
SHOPRITE HOLDINGS
SHOPRITE LUANSHYA
SHOPRITE KITWE

can potentially be associated with a common merchant organization while preserving the individual branch.

This allows higher-level analytics.

Step Six: Currency Intelligence

Currency is surprisingly difficult.

A receipt might say:

K184.50

Another:

ZMW 184.50

Another:

184.50

The system needs to determine whether:

184.50

means:

USD
ZMW
EUR
GBP
ZAR
KES
NGN

The answer may come from several signals:

Currency symbol
Currency code
Merchant country
Merchant address
Known merchant location
Receipt language
Tax format
Historical transactions
User locale

The system should not blindly guess.

Instead, it should produce:

{
  "currency": "ZMW",
  "confidence": 0.97
}

or:

{
  "currency": null,
  "confidence": 0.41,
  "requires_review": true
}

Uncertainty is better than fabricated certainty.

Step Seven: Line-Item Extraction

This is where the system becomes genuinely useful.

We want to transform:

2 MILK FULL CREAM     48.00
BREAD WHITE            18.00
SOAP BAR               12.50

into:

{
  "items": [
    {
      "description": "MILK FULL CREAM",
      "quantity": 2,
      "total": 48.00
    },
    {
      "description": "BREAD WHITE",
      "quantity": 1,
      "total": 18.00
    },
    {
      "description": "SOAP BAR",
      "quantity": 1,
      "total": 12.50
    }
  ]
}

But receipts can contain much more complicated formats.

For example:

MILK
2 @ 24.00              48.00

or:

MILK FULL CREAM 2X24.00
48.00

or:

001234 MILK 24.00

The parser needs to understand multiple representations.

Quantity and Unit Reasoning

A sophisticated receipt system should distinguish:

1 × 20.00
2 × 20.00
0.75 kg × 80.00

The third example is particularly interesting.

It may represent:

0.75 kg × K80/kg = K60

So the structured record should support:

{
  "description": "Tomatoes",
  "quantity": 0.75,
  "unit": "kg",
  "unit_price": 80,
  "total": 60
}

This opens the door to much deeper financial intelligence.

The receipt is no longer just an expense.

It becomes a representation of what was purchased.

Step Eight: Financial Reconciliation

Now we introduce one of the most important components:

the validation engine.

Suppose the receipt contains:

Milk       24.50
Bread      18.00
Rice       75.00

Subtotal  117.50
VAT        17.63
Total     135.13

The system can calculate:

24.50 + 18.00 + 75.00 = 117.50

Then:

117.50 + 17.63 = 135.13

Everything reconciles.

That increases confidence.

But imagine OCR produces:

Rice  750.00

The total might no longer reconcile.

This creates a signal:

EXTRACTION ERROR POSSIBLE

The validation engine can then inspect nearby OCR candidates.

Maybe the image actually says:

75.00

This is where deterministic software and machine learning become extremely powerful together.

AI can interpret.

Rules can verify.

Step Nine: Multiple Candidate Extraction

One of the strongest ideas in this architecture is to avoid forcing the system to make an immediate decision.

Suppose OCR sees:

184.50

but image quality is poor.

The system can maintain candidates:

{
  "field": "total",
  "candidates": [
    {
      "value": 184.50,
      "confidence": 0.91
    },
    {
      "value": 184.80,
      "confidence": 0.42
    }
  ]
}

Then downstream validation can choose.

For example:

Candidate A
184.50

Candidate B
184.80

If:

subtotal + tax = 184.50

Candidate A becomes significantly stronger.

This architecture is much more robust than:

OCR → one answer → done

Step Ten: Semantic Understanding

A universal system should understand what a transaction represents.

Consider:

CANONICAL KEYBOARD

versus:

MILK

The system can classify products into categories:

Electronics
Groceries
Transport
Utilities
Office Supplies
Healthcare
Entertainment
Clothing
Software
Travel

Then:

{
  "description": "Wireless Keyboard",
  "category": "electronics",
  "subcategory": "computer_accessories"
}

This enables analytics that traditional receipt scanners cannot provide.

Step Eleven: Merchant-Specific Learning

Universal does not mean treating every receipt identically.

The system should learn merchant-specific patterns.

Suppose thousands of receipts from the same supermarket are processed.

We may discover:

SKU
DESCRIPTION
QTY
PRICE
VAT
TOTAL

always appears in roughly the same layout.

The system can store a merchant profile:

{
  "merchant": "example_store",
  "layout_version": 4,
  "known_fields": [
    "sku",
    "description",
    "quantity",
    "price",
    "tax"
  ]
}

Future receipts can then be processed faster and more accurately.

The system evolves from:

generic understanding

toward:

generic understanding + merchant-specific knowledge

Step Twelve: The Receipt Knowledge Graph

Now things become much more interesting.

Instead of storing receipts as isolated JSON documents, we can create relationships.

Customer
   │
   ├── bought
   │
   ▼
Product
   │
   ├── sold_by
   ▼
Merchant
   │
   ├── located_at
   ▼
Location

Another relationship:

Receipt
 ├── contains → Product
 ├── issued_by → Merchant
 ├── paid_with → Payment Method
 ├── occurred_at → Date
 └── belongs_to → User

Over time, this becomes a transaction knowledge graph.

The system can answer questions such as:

How much did I spend on groceries last month?

or:

Which merchants do I use most frequently?

or:

How much have I spent on computer equipment this year?

The receipt scanner has evolved into a financial intelligence system.

Step Thirteen: Duplicate Detection

Receipts can be uploaded multiple times.

Maybe the user scans the same receipt twice.

Maybe an email attachment is uploaded after a camera scan.

We need duplicate detection.

A simple approach can use:

Merchant
Date
Time
Total
Transaction ID

But images can also have perceptual hashes.

For example:

image_hash

combined with:

merchant + date + total

can produce a duplicate probability.

The system might return:

{
  "duplicate_probability": 0.94
}

instead of silently creating another expense.

Step Fourteen: Fraud and Anomaly Detection

This is another powerful layer.

Imagine a user normally receives receipts around:

K20
K50
K100
K150

Then suddenly:

K48,000

appears.

That does not automatically mean fraud.

But it deserves attention.

The system can flag:

Unusual transaction amount

Other signals could include:

Duplicate receipt
Impossible date
Future transaction date
Invalid tax calculation
Repeated transaction ID
Altered image
Inconsistent totals
Suspicious OCR mismatch

Receipt intelligence can therefore become part of financial security.

Step Fifteen: Detecting Altered Receipts

This is one of the more advanced problems.

A receipt can potentially be edited.

For example:

TOTAL K120

could be manipulated into:

TOTAL K1,200

A serious system should not simply trust the image.

It can inspect:

font consistency
pixel patterns
spacing
compression artifacts
baseline alignment
character geometry
metadata
document structure

None of these signals alone proves manipulation.

But together they can create an anomaly score.

{
  "integrity": {
    "score": 0.91,
    "flags": []
  }
}

or:

{
  "integrity": {
    "score": 0.38,
    "flags": [
      "inconsistent_text_region",
      "financial_reconciliation_failure"
    ]
  }
}

The goal is not to declare guilt.

The goal is to identify receipts that require review.

Step Sixteen: Confidence Is a First-Class Data Type

A major design principle for this system should be:

Every important extraction should have confidence.

Not just:

{
  "total": 184.50
}

but:

{
  "total": {
    "value": 184.50,
    "confidence": 0.98
  }
}

Similarly:

{
  "merchant": {
    "value": "Example Store",
    "confidence": 0.96
  }
}

and:

{
  "currency": {
    "value": "ZMW",
    "confidence": 0.87
  }
}

This makes the system explainable.

A downstream application can decide:

confidence >= 0.95
→ automatically accept

0.75–0.95
→ accept but flag

< 0.75
→ request review

The thresholds can be configured per use case.

Step Seventeen: Human-in-the-Loop Processing

There will always be difficult receipts.

Instead of pretending otherwise, build a review system.

Imagine the system says:

We are 62% confident that the total is K184.50.

The user sees the receipt and confirms:

Yes.

That feedback becomes valuable training data.

The architecture becomes:

Receipt
   ↓
AI Extraction
   ↓
Validation
   ↓
Confidence
   ↓
┌───────────────┐
│ High confidence│ → Automatic
└───────────────┘

┌───────────────┐
│ Low confidence │ → Human Review
└───────────────┘

This allows the system to improve continuously.

Step Eighteen: Event-Driven Architecture

At scale, I would not process everything synchronously.

A possible architecture:

                API Gateway
                     │
                     ▼
              Receipt Service
                     │
                     ▼
                Object Store
                     │
                     ▼
                  Queue
                     │
       ┌─────────────┼──────────────┐
       ▼             ▼              ▼
   OCR Worker   Image Worker   Metadata Worker
       │             │              │
       └─────────────┼──────────────┘
                     ▼
                Parse Queue
                     │
                     ▼
               AI Parser
                     │
                     ▼
             Validation Engine
                     │
                     ▼
             Confidence Engine
                     │
                     ▼
               Receipt DB

This gives us independent scaling.

If OCR becomes the bottleneck, scale OCR workers.

If AI parsing becomes expensive, scale parser workers.

If validation is cheap, keep fewer validation workers.

The system becomes elastic.

Step Nineteen: Storage Design

I would separate the raw document from the structured interpretation.

Object storage:

receipts/
    2026/
       09/
          rcpt_8fd91/
              original.jpg
              normalized.png
              ocr.json

Database:

receipts
receipt_items
merchants
transactions
payments
receipt_events
receipt_reviews

The raw image should remain immutable.

The interpretation can evolve.

This distinction matters.

Suppose version 1 of the parser extracts:

Total = K180

and version 2 determines:

Total = K184.50

We should be able to reconstruct why.

Therefore, maintain processing versions:

{
  "parser_version": "2.4.1",
  "ocr_version": "7.2",
  "processed_at": "2026-09-19T20:12:00Z"
}

Now the receipt has provenance.

Step Twenty: The Universal Schema

A possible universal receipt model could look like:

{
  "id": "rcpt_8fd91",

  "merchant": {
    "name": "Example Store",
    "confidence": 0.98
  },

  "transaction": {
    "date": "2026-09-19",
    "time": "18:42:11",
    "transaction_id": "8291938"
  },

  "currency": "ZMW",

  "items": [
    {
      "description": "Milk",
      "quantity": 2,
      "unit_price": 24.50,
      "total": 49.00
    }
  ],

  "financials": {
    "subtotal": 100.00,
    "tax": 15.00,
    "discount": 5.00,
    "total": 110.00
  },

  "payment": {
    "method": "cash",
    "tendered": 120.00,
    "change": 10.00
  },

  "confidence": 0.96
}

The schema should remain extensible.

Because universal systems fail when they assume the world is standardized.

The world is not standardized.

Step Twenty-One: APIs

The platform could expose several APIs.

Upload:

POST /v1/receipts

Retrieve:

GET /v1/receipts/{id}

Search:

GET /v1/receipts?merchant=shoprite

Analytics:

GET /v1/spending

Reprocess:

POST /v1/receipts/{id}/reprocess

Review:

POST /v1/receipts/{id}/review

Webhook:

receipt.processed
receipt.review_required
receipt.failed

Now other applications can build on top of the intelligence engine.

Accounting software.

Expense management.

Retail applications.

Personal finance applications.

Tax software.

Business intelligence systems.

Procurement platforms.

Insurance platforms.

And even AI agents.

Step Twenty-Two: Making the System Agent-Friendly

This is where receipt intelligence becomes particularly interesting.

Imagine an AI agent receiving:

"Analyze my business expenses."

Instead of searching through images manually, it queries:

GET /v1/transactions

and receives normalized data.

The agent can then reason over:

merchant
category
amount
date
product
tax
payment method

The receipt system becomes an observation layer for financial agents.

The AI does not need to understand pixels.

It receives structured reality.

Step Twenty-Three: Learning Without Destroying Reliability

Machine learning systems have a dangerous tendency.

They can become increasingly confident while becoming increasingly wrong.

Therefore, I would keep deterministic rules around critical financial fields.

For example:

subtotal + tax - discount ≈ total

must be checked mathematically.

Quantity × unit price should approximately equal item total.

Change should approximately equal:

tendered - total

Tax should fall within plausible constraints.

Dates should be valid.

Currency codes should be valid.

Transaction IDs should be consistent.

AI proposes.

Rules verify.

That division of responsibility is extremely important.

Step Twenty-Four: Measuring the System

Accuracy should not simply be:

"Does OCR work?"

We need field-level metrics.

For example:

Merchant Accuracy
Date Accuracy
Currency Accuracy
Total Accuracy
Tax Accuracy
Item Accuracy
Quantity Accuracy
Category Accuracy

We can define:

Exact Match Accuracy
Field-Level Precision
Field-Level Recall
Numeric Error Rate
Reconciliation Rate
Human Review Rate
Duplicate Detection Accuracy

One particularly useful metric is:

Straight-Through Processing Rate

Meaning:

What percentage of receipts can be processed correctly without human intervention?

That is a much more meaningful production metric than OCR character accuracy alone.

Step Twenty-Five: The Feedback Loop

Every correction becomes data.

Suppose users repeatedly correct:

"TOTAL" → "SUBTOTAL"

The system learns.

Suppose a particular merchant's receipt format changes.

The system detects a drop in confidence.

That can trigger:

merchant template drift detected

Now the platform can adapt.

This creates:

Receipts
   ↓
Extraction
   ↓
Validation
   ↓
Human Corrections
   ↓
Training Data
   ↓
Improved Models
   ↓
Better Extraction

The system becomes progressively better at understanding the world.

The Deeper Idea

The most interesting part of this project is not receipt scanning.

It is the transformation:

Physical document
        ↓
Pixels
        ↓
Text
        ↓
Structure
        ↓
Entities
        ↓
Relationships
        ↓
Financial meaning
        ↓
Machine-readable knowledge

That is a general pattern for intelligent software.

We are taking something designed for human perception and converting it into something machines can reason about.

The same architecture can eventually apply to:

Invoices
Bills
Purchase orders
Delivery notes
Bank statements
Warranty documents
Tax documents
Contracts
Shipping documents

Receipts are simply an excellent starting point.

Building the MVP

I would not begin by trying to support every receipt in the world.

Start with:

1. Image upload
2. OCR
3. Merchant extraction
4. Date extraction
5. Currency detection
6. Line-item extraction
7. Total extraction
8. Mathematical validation
9. Confidence scores
10. JSON API

Then test it against hundreds or thousands of real receipts.

Not perfectly clean demo receipts.

Real receipts.

Wrinkled receipts.

Dark photographs.

Partially cut receipts.

Different countries.

Different currencies.

Different languages.

Different merchants.

Different printers.

Different layouts.

Then expand.

A Possible Technology Stack

A practical implementation could use:

API:
FastAPI / Django / Node.js

Workers:
Python / Rust

Queue:
Redis Streams / RabbitMQ / Kafka

OCR:
Tesseract / PaddleOCR / cloud OCR

ML:
PyTorch

Database:
PostgreSQL

Object Storage:
S3-compatible storage

Search:
OpenSearch

Cache:
Redis

Observability:
OpenTelemetry

Rust would be particularly interesting for high-throughput document pipelines.

Python would be excellent around machine learning.

PostgreSQL would handle the structured financial data.

Object storage would preserve original documents.

The architecture does not require every component to be written in the same language.

Use the right tool for the right layer.

The Final Architecture

At maturity, the system could look like this:

                 ┌─────────────────────┐
                 │   Camera / Upload    │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │ Document Processing │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │ OCR + Vision Engine │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │ Layout Understanding│
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │ Semantic Extraction │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │ Entity Resolution   │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │ Validation Engine   │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │ Confidence Engine   │
                 └──────────┬──────────┘
                            │
                    ┌───────┴────────┐
                    ▼                ▼
              High Confidence    Low Confidence
                    │                │
                    ▼                ▼
              Auto Process       Human Review
                    │                │
                    └───────┬────────┘
                            ▼
                 ┌─────────────────────┐
                 │ Receipt Knowledge   │
                 │       Layer         │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │ APIs / Analytics /  │
                 │ AI Agents / Apps    │
                 └─────────────────────┘

This is no longer a receipt scanner.

It is an intelligence layer between the physical financial world and software.

Conclusion

A receipt is deceptively simple.

It is a tiny financial document that encodes merchant identity, products, quantities, prices, taxes, discounts, payment information, time, location, and transaction information.

Yet most software still treats it as an image.

That is the opportunity.

The Universal Receipt Intelligence System changes the abstraction.

Instead of asking:

How do I read this receipt?

we ask:

How do I understand this transaction?

That difference completely changes the architecture.

OCR becomes only one component.

Computer vision becomes another.

Natural-language understanding becomes another.

Entity resolution becomes another.

Mathematical validation becomes another.

Knowledge graphs become another.

And confidence becomes the glue holding the system together.

The final product is not simply:

image → text

It is:

image
  ↓
document
  ↓
transaction
  ↓
structured knowledge
  ↓
financial intelligence

That is the real engineering challenge.

And it is also the interesting part.

Because once machines can reliably understand receipts, they can begin understanding a much larger portion of the physical economy.

Every supermarket receipt.

Every restaurant bill.

Every hardware-store purchase.

Every fuel receipt.

Every business expense.

Every transaction that was once trapped inside a piece of paper can become structured data.

And when enough of that data becomes machine-readable, the receipt stops being the end of a transaction.

It becomes the beginning of computation.

Code is not only about building software that creates new things. Sometimes it is about teaching software how to understand the things humans have already created.

MCP APIs are changing how AI agents interact with software — but that also changes the security mode

Sat, 19 Sep 2026 16:20:35 +0000

Traditional APIs generally assume a known client making predictable requests. MCP introduces a different pattern: autonomous AI agents can discover and call tools at runtime based on a natural-language objective.

That creates an important engineering question:

How do you secure an API when the caller itself may be autonomous?

In the latest SEOSiri technical deep dive, I explore:

• Where MCP APIs fit inside the API economy
• Why every public API is also an attack surface
• SQL injection, DDoS, BOLA/IDOR, credential abuse and other API threats
• How SEOSiri structures an edge security layer around MCP/API infrastructure
• The open-source @seosiri/security-proxy approach
• Scoped API keys and sandbox-first integration
• How developers can move from their first API key to a live MCP integration

The goal isn't simply to expose another API.

It is to build an agent-ready API layer where access, validation and security are part of the architecture.

Read the full technical breakdown:
https://www.seosiri.com/2026/09/seosiri-mcp-api-cloud-defense.html

MCP #APIs #AIEngineering #CyberSecurity #CloudSecurity #DevTools #OpenSource #AIAgents #SaaS #Developers

Building a System That Measures Technical Debt by Behavior

Sat, 19 Sep 2026 11:40:35 +0000

Technical debt is one of those phrases every software engineer knows.

Everyone talks about it.

Everyone complains about it.

Everyone has some of it.

And almost nobody measures it correctly.

We usually measure technical debt through things that are easy to count.

Number of TODO comments.

Number of outdated dependencies.

Code duplication.

Cyclomatic complexity.

Large files.

Old code.

Static-analysis warnings.

Test coverage.

These metrics are useful.

But they are not technical debt itself.

They are clues.

Technical debt becomes interesting when it starts changing the behavior of a system.

A service becomes slower.

Deployments become riskier.

Small changes require touching ten unrelated modules.

Developers become afraid of certain parts of the codebase.

Incidents become harder to diagnose.

A database migration that should take thirty minutes becomes a three-day operation.

A seemingly harmless feature causes regressions in completely unrelated functionality.

This is where technical debt becomes measurable.

Not as an abstract number attached to source code.

But as behavioral friction.

The idea behind this article is simple:

Instead of asking how messy the code looks, build a system that observes how the software behaves and estimates where technical debt is creating measurable friction.

That changes the problem completely.

We are no longer building another linter.

We are building something closer to a technical-debt observability system.

The Problem With Traditional Technical Debt Metrics

Imagine two services.

Service A contains 20,000 lines of ugly legacy code.

Service B contains 5,000 lines of beautiful modern code.

At first glance, Service B looks healthier.

But suppose Service A has:

99.99% availability
50 ms average latency
15-minute deployments
very few production incidents
predictable change impact
easy debugging

Service B has:

frequent regressions
400 ms latency
deployments that take hours
fragile integrations
high rollback frequency
mysterious production failures

Which system has more technical debt?

The answer cannot come from counting lines of code.

The smaller and cleaner-looking system may have more operational debt.

This exposes an important distinction:

Code quality is not the same thing as system health.

Technical debt is ultimately about the future cost of changing and operating software.

Therefore, we should observe what happens when the software changes.

That means measuring behavior.

Technical Debt Leaves Behavioral Footprints

Technical debt rarely announces itself.

It does not usually create a file called:

technical-debt.txt

Instead, it leaves traces.

Consider a developer modifying a payment service.

The first change takes one hour.

A few months later, a similar change takes four hours.

Eventually, every payment-related change requires touching:

PaymentService
OrderService
UserService
InvoiceService
NotificationService
DatabaseAdapter
LegacyPaymentAdapter

The codebase has accumulated coupling.

That coupling produces behavior.

The behavior might look like:

Change size: small
Files touched: 17
Tests affected: 84
Build time: 11 minutes
Deployment rollback probability: increasing

That is evidence.

Or consider a database.

A query starts at:

35 ms

Six months later:

280 ms

Then:

900 ms

Nothing necessarily looks catastrophic in a code review.

But the system is telling us something.

The architecture is accumulating friction.

What If We Could Observe Technical Debt Like Performance?

Modern systems already have observability platforms.

We measure:

CPU
memory
latency
throughput
error rates
traces
logs
database queries

Why not measure technical debt similarly?

Imagine opening a dashboard and seeing:

Technical Debt Health

Change Friction        ███████░░░  71
Coupling               ████████░░  82
Failure Amplification  ██████░░░░  63
Deployment Risk        █████░░░░░  54
Debugging Friction     ███████░░░  74
Performance Drag       ████░░░░░░  41
Test Fragility         ████████░░  86

These are not arbitrary numbers.

They could be derived from actual system behavior.

The goal is not to produce a magical "technical debt score."

The goal is to identify behavioral patterns that indicate increasing maintenance cost.

The Behavioral Model

We can model technical debt as a function of several measurable dimensions.

For example:

Debt =
    ChangeFriction
  + Coupling
  + FailureAmplification
  + DeploymentRisk
  + DebuggingCost
  + PerformanceDrag
  + TestFragility

But there is an important principle:

We should not treat these dimensions as universal truths.

Different systems have different architectures.

A batch-processing system behaves differently from a real-time API.

A game server behaves differently from an accounting platform.

A machine-learning pipeline behaves differently from an e-commerce application.

So our system should learn a baseline.

Instead of asking:

"Is 200 ms bad?"

we ask:

"Has this system become significantly worse relative to its own historical behavior?"

That is much more powerful.

Architecture of the Debt Measurement System

A basic architecture could look like this:

              ┌─────────────────────┐
              │   Developer Actions  │
              └──────────┬──────────┘
                         │
                         ▼
              ┌─────────────────────┐
              │   Source Control    │
              │      Events         │
              └──────────┬──────────┘
                         │
                         ▼
        ┌────────────────────────────────┐
        │      Behavioral Collectors     │
        ├────────────────────────────────┤
        │ Deployment Collector            │
        │ Incident Collector              │
        │ Test Collector                  │
        │ Performance Collector           │
        │ Dependency Collector            │
        │ Change Collector                │
        │ Runtime Collector               │
        └───────────────┬────────────────┘
                        │
                        ▼
              ┌─────────────────────┐
              │ Event Normalization │
              └──────────┬──────────┘
                         │
                         ▼
              ┌─────────────────────┐
              │ Behavioral Database │
              └──────────┬──────────┘
                         │
                         ▼
              ┌─────────────────────┐
              │ Debt Analyzer       │
              └──────────┬──────────┘
                         │
             ┌───────────┴───────────┐
             ▼                       ▼
      ┌─────────────┐        ┌──────────────┐
      │ Debt Scores │        │ Explanations │
      └─────────────┘        └──────────────┘

The important component is the behavioral collector.

We need to observe the system without requiring developers to manually report technical debt.

Event-Driven Debt Measurement

The foundation should be an event stream.

Every important engineering activity becomes an event.

For example:

{
  "type": "deployment",
  "service": "payments",
  "commit": "a92fd31",
  "duration": 842,
  "status": "success",
  "rollback": false,
  "timestamp": "2026-09-19T10:30:00Z"
}

Another event:

{
  "type": "change",
  "service": "payments",
  "commit": "a92fd31",
  "files_changed": 14,
  "lines_added": 340,
  "lines_removed": 120,
  "modules_touched": 6
}

And another:

{
  "type": "incident",
  "service": "payments",
  "severity": "high",
  "duration": 47,
  "root_commit": "a92fd31"
}

The system can correlate these events.

That is where the interesting information appears.

Measuring Change Friction

One of the strongest signals of technical debt is how difficult changes become.

Suppose a developer submits a pull request.

We can observe:

files_changed
modules_changed
dependencies_changed
review_cycles
build_duration
test_duration
merge_conflicts
rework_commits
rollback_events

We can combine these into a change-friction profile.

For example:

Change #1293

Files:             17
Modules:            8
Review cycles:      4
Rework commits:     6
Tests executed:   812
Test failures:      31
Build time:       13m
Deployment:        2
Rollback:          1

This tells us much more than:

Complexity = 14

The system can then ask:

Why does changing this component repeatedly require touching so much of the architecture?

That question is much closer to the actual meaning of technical debt.

Measuring Coupling Through Change Graphs

Static dependency graphs are useful.

But behavioral dependency graphs can be even more interesting.

Imagine that Service A rarely changes alone.

Whenever Service A changes, these services also change:

A → B
A → C
A → D
A → F

Over time, we can construct a change graph.

             ┌───────┐
             │ User  │
             └───┬───┘
                 │
        ┌────────┼────────┐
        ▼        ▼        ▼
     Payment   Orders   Profile
        │        │
        └────┬───┘
             ▼
         Database

But instead of merely representing technical dependencies, we measure co-change frequency.

If two modules independently change in 90% of the same pull requests, that is a behavioral coupling signal.

The code might not explicitly show a dependency.

The development process does.

That is fascinating.

The organization itself becomes an observability source for architecture.

Measuring Change Amplification

A small requirement should ideally produce a relatively small change.

Suppose a ticket says:

Add a field to customer profiles.

The actual change modifies:

27 files
11 database tables
6 API endpoints
4 frontend components
3 background workers
2 integration services

We can define:

Change Amplification =
    actual_change_surface
    /
    expected_change_surface

The exact formula is domain-dependent.

But the principle is powerful.

If a tiny conceptual change repeatedly produces a huge technical footprint, something is wrong.

Perhaps responsibilities are poorly separated.

Perhaps abstractions are leaking.

Perhaps data models are overly coupled.

Perhaps the architecture has accumulated historical assumptions.

The system doesn't need to know the exact reason immediately.

It can identify the behavioral anomaly.

Measuring Failure Amplification

Another powerful signal is how far failures travel.

Suppose one database timeout causes:

Database timeout
      ↓
Order service failure
      ↓
Payment retry storm
      ↓
Queue overload
      ↓
Notification delays
      ↓
API timeouts

The original failure was small.

Its impact was not.

That is failure amplification.

We can use distributed tracing to construct failure propagation graphs.

For each incident:

root cause
   ↓
affected service
   ↓
downstream services
   ↓
user-facing effects

Then measure:

Impact radius
Propagation depth
Propagation speed
Recovery time
Number of dependent components

A system with increasing failure amplification may be accumulating architectural debt.

Deployment Behavior Is a Gold Mine

Deployment systems produce enormous amounts of useful information.

Consider:

Deployment frequency
Deployment duration
Rollback frequency
Failed deployments
Hotfix frequency
Post-deployment incidents
Time to recovery

Now imagine a service where deployment time has evolved:

January     8 minutes
February   11 minutes
March      15 minutes
April      21 minutes
May        29 minutes
June       42 minutes

That is not just an infrastructure problem.

It may indicate increasing system complexity.

Maybe the test suite is growing without parallelization.

Maybe builds are becoming tightly coupled.

Maybe deployment requires coordinating too many services.

Maybe migrations have become dangerous.

The behavior is telling us that the cost of change is increasing.

Test Fragility

Test coverage is frequently used as a proxy for code quality.

But coverage alone can be misleading.

A better behavioral measurement is test stability.

Suppose a test suite contains 10,000 tests.

Every pull request produces:

Average tests failed: 18
Average flaky tests: 7
Average reruns: 3

Developers eventually stop trusting the suite.

That is technical debt.

A test that fails randomly has an operational cost.

We can measure:

Flake rate
Failure recurrence
Retry frequency
Test duration
Failure isolation time

Then identify test clusters.

For example:

Payment tests
    ↓
High flake rate
    ↓
Shared database fixture
    ↓
Parallel execution conflict

The system can surface this as:

Payment test suite has exhibited abnormal instability for the last 30 days.

That is much more actionable than:

Test coverage: 84%.

Measuring Debugging Friction

Some systems are not particularly slow.

They are simply difficult to understand.

A production incident occurs.

Engineers search logs.

Then traces.

Then dashboards.

Then source code.

Then database records.

Then finally discover the problem.

That process can be measured.

For each incident:

Time to first useful signal
Time to identify affected component
Time to identify probable cause
Time to mitigation
Time to root cause

We could call this:

Diagnostic Friction.

If the average time to identify a root cause increases, the system may have accumulated observability debt.

This is important because technical debt isn't limited to application code.

Sometimes the architecture works.

But nobody can understand why.

Measuring Performance Drag

Performance degradation is another behavioral signal.

But raw latency isn't enough.

We should track performance relative to change.

Suppose:

Commit A:
p95 latency = 120 ms

Commit B:
p95 latency = 130 ms

Commit C:
p95 latency = 150 ms

Commit D:
p95 latency = 210 ms

Now we can correlate performance with architectural changes.

Maybe a new abstraction introduced additional database calls.

Maybe a caching layer became ineffective.

Maybe a previously cheap operation became synchronous.

The system could detect:

Performance regression
        ↓
introduced by change
        ↓
persistent across deployments

That is a behavioral debt signal.

The Debt Timeline

One of the most useful features would be a timeline.

Imagine:

January
│
├── Service created
│
February
│
├── First coupling increase
│
March
│
├── Deployment time +18%
│
April
│
├── Test flakiness increases
│
May
│
├── Incident frequency increases
│
June
│
└── Change amplification crosses threshold

Now technical debt becomes historical.

You can see how the system got here.

That matters because technical debt is rarely created in one event.

It accumulates.

Debt Is Often Nonlinear

This is one of the most interesting aspects of the problem.

Technical debt does not necessarily increase linearly.

A system may behave perfectly for months.

Then suddenly everything becomes difficult.

Why?

Because architectural complexity often contains thresholds.

For example:

Complexity
    │
    │                /
    │              /
    │            /
    │          /
    │________/
    └────────────────── Time

At first, adding more features barely changes development cost.

Then a threshold is crossed.

Suddenly:

tests take longer
deployments slow down
changes touch more modules
incidents become harder to diagnose

The system has entered a different behavioral regime.

A good debt analyzer should therefore detect change points, not just averages.

Baselines Instead of Universal Rules

We should avoid simplistic rules like:

> 500 lines = bad
> 100 ms = bad
> 20 files = bad

Instead:

Current behavior
        ↓
Historical baseline
        ↓
Deviation
        ↓
Behavioral risk

Suppose an API normally processes requests in 80 ms.

A jump to 130 ms may matter.

For another API, 130 ms may be normal.

Technical debt measurement should therefore be contextual.

The system should learn:

What is normal here?

Then detect:

What is becoming abnormal?

Building the Behavioral Data Model

A simple event schema could look like:

CREATE TABLE engineering_events (
    id UUID PRIMARY KEY,
    event_type VARCHAR(50),
    service VARCHAR(100),
    commit_hash VARCHAR(100),
    timestamp TIMESTAMP,
    duration_ms INTEGER,
    metadata JSONB
);

Then we can store events such as:

change
build
test
deployment
rollback
incident
performance_regression
dependency_update
migration

The JSON metadata allows different event types to evolve independently.

Eventually, we can build specialized analytical tables.

For example:

change_metrics
deployment_metrics
incident_metrics
test_metrics
performance_metrics

The Debt Engine

The central analyzer receives historical events.

Conceptually:

class DebtAnalyzer:

    def analyze(self, service):
        changes = get_changes(service)
        deployments = get_deployments(service)
        incidents = get_incidents(service)
        tests = get_test_behavior(service)
        performance = get_performance(service)

        return {
            "change_friction":
                measure_change_friction(changes),

            "deployment_risk":
                measure_deployment_risk(deployments),

            "failure_amplification":
                measure_failure_amplification(incidents),

            "test_fragility":
                measure_test_fragility(tests),

            "performance_drag":
                measure_performance_drag(performance)
        }

But the interesting part isn't the code.

It is the interpretation.

The engine should not simply say:

Debt = 72

That number is nearly useless without explanation.

Instead:

Technical Debt Signal

Change friction increased 31% over 90 days.

Primary behavioral evidence:
- Average files changed per feature: 6 → 11
- Review cycles: 1.8 → 3.4
- Regression incidents: 2 → 7
- Deployment rollback rate: 1.2% → 4.8%

Most affected area:
Payment orchestration

Likely contributing pattern:
Increasing cross-module changes.

Now engineers have something they can investigate.

Explainability Matters

A technical-debt system should never become a black box.

If it says:

Debt increased.

the developer should immediately ask:

Why?

The answer should be traceable.

For example:

Debt increased because:

1. Change surface increased 42%.
2. Deployment failures increased 18%.
3. Test reruns increased 31%.
4. Incident recovery time increased 27%.

Every measurement should have evidence.

The system should be closer to an observability platform than an AI fortune teller.

Using Machine Learning Carefully

Machine learning could make the system much more interesting.

Instead of manually defining every threshold, we can train models on historical behavior.

For example:

Inputs:

files_changed
modules_changed
review_cycles
build_duration
test_failures
deployment_duration
rollback_rate
incident_frequency
latency
dependency_count

The model could estimate:

probability of high-friction change

Or:

expected maintenance cost

But there is a danger.

If the model learns from historical behavior, it may learn bad organizational habits.

For example:

If a team always creates huge pull requests, the model might decide that huge pull requests are normal.

Therefore:

Prediction is not the same thing as quality.

Machine learning should detect patterns.

Engineers should interpret them.

Technical Debt as a Control System

There is an even deeper way to think about this.

A software system is a dynamic system.

Developers introduce changes.

Those changes modify the system.

The modified system produces behavior.

That behavior creates feedback.

We can represent the loop:

        Developer Change
               ↓
        System Behavior
               ↓
        Measurements
               ↓
        Debt Signals
               ↓
        Engineering Decisions
               ↓
        Developer Change

This creates a feedback control loop.

The software is observing itself.

That is a much more interesting idea than a static technical-debt report.

From Dashboard to Automated Recommendations

Once we can detect behavioral debt, the system can generate engineering recommendations.

For example:

Observed:
Database migration failures increased.

Possible actions:
- split migration into backward-compatible stages
- reduce deployment coupling
- add migration verification
- move expensive operations outside deployment

Another example:

Observed:
Payment changes increasingly touch six services.

Possible architectural investigation:
- identify shared domain responsibilities
- inspect service boundaries
- evaluate event-driven integration

Notice the language.

The system should say:

investigate.

Not:

rewrite everything.

Technical debt analysis should help engineers think.

It should not pretend to architect the entire system automatically.

The Most Important Metric: Cost of Change

If I had to reduce the entire system to one principle, it would be this:

Technical debt becomes visible when the cost of change increases.

Everything else is evidence.

You can measure:

Time to implement
Time to review
Time to test
Time to deploy
Time to recover
Time to understand

Together, these create a picture of engineering friction.

A healthy architecture should make ordinary changes relatively predictable.

When the same class of change repeatedly becomes more expensive, the system is telling you something.

A Possible Technical Debt API

The platform itself could expose an API.

For example:

GET /api/v1/services/payments/debt

Response:

{
  "service": "payments",
  "period": "90d",
  "signals": {
    "change_friction": 0.72,
    "deployment_risk": 0.51,
    "test_fragility": 0.83,
    "failure_amplification": 0.61,
    "performance_drag": 0.42
  },
  "trends": {
    "change_friction": "+31%",
    "deployment_risk": "+12%",
    "test_fragility": "+24%"
  },
  "evidence": [
    {
      "signal": "change_friction",
      "reason": "Average modules touched increased from 3.2 to 6.7"
    }
  ]
}

Now technical debt itself becomes observable infrastructure.

Other developer tools could consume it.

IDEs could show warnings.

CI systems could comment on pull requests.

Dashboards could visualize trends.

Planning systems could identify areas where engineering effort is being consumed.

Pull Request Integration

Imagine opening a pull request and seeing:

Behavioral Impact

This change modifies the Payments subsystem.

Historical behavior:

Average payment PR:
- 5.2 files
- 2.1 review cycles
- 4.8 minutes CI
- 1.2% rollback rate

This PR:
- 14 files
- 5 review cycles predicted
- 11.3 minutes CI impact

Change amplification: HIGH

Investigate:
payment orchestration coupling

That is much more useful than:

ESLint: 0 errors

Both are valuable.

But they answer completely different questions.

The System Could Detect Architectural Decay

Architecture is not static.

A diagram might show beautifully separated services:

Users
Orders
Payments
Inventory
Notifications

Three years later, the actual behavior may look like:

Users ←→ Orders ←→ Payments
  ↑       ↓  ↘      ↓
  └──── Inventory ←─┘
       ↕
 Notifications

The architecture has drifted.

The source code may still claim that boundaries exist.

Behavior tells the truth.

That makes behavioral analysis particularly valuable for long-lived systems.

Technical Debt Has Different Species

Another useful feature would be classification.

Instead of one debt score:

Technical Debt

we could identify:

Architectural Debt
Testing Debt
Operational Debt
Observability Debt
Performance Debt
Dependency Debt
Data Debt
Deployment Debt
Security Debt
Documentation Debt

Each category produces different behavioral signals.

For example:

Deployment debt

longer deployments
more rollbacks
manual steps
higher failure rate

Testing debt

flaky tests
slow CI
low signal-to-noise
frequent retries

Architectural debt

high change amplification
high co-change frequency
large failure radius

Observability debt

long diagnosis time
missing traces
uncorrelated logs

This makes the output actionable.

What We Should Not Measure

There is a temptation to measure everything.

That is dangerous.

If we create 500 engineering metrics, developers will ignore all of them.

The system should focus on signals that correlate with actual engineering friction.

Good metrics should answer questions like:

Is changing this component becoming harder?

Is this service becoming more fragile?

Are failures spreading farther?

Are deployments becoming riskier?

Are engineers spending more time understanding the system?

Is performance degrading after architectural changes?

If a metric does not help answer a meaningful question, it probably does not belong in the core system.

The Future: Self-Observing Software

The most interesting possibility is not simply measuring technical debt.

It is building software that continuously observes its own architectural behavior.

Imagine a system that knows:

This component is changing frequently.

Its dependency surface is growing.

Its deployments are becoming slower.

Its failures are propagating farther.

Its tests are becoming less reliable.

Its performance is degrading.

The system can then recognize a pattern:

Architectural stress increasing.

That is powerful.

We have moved from static code analysis to behavioral architecture analysis.

Final Architecture

A mature version might look like:

                   ┌───────────────────┐
                   │ Source Control    │
                   └─────────┬─────────┘
                             │
                   ┌─────────▼─────────┐
                   │ CI/CD Systems     │
                   └─────────┬─────────┘
                             │
          ┌──────────────────┼──────────────────┐
          ▼                  ▼                  ▼
     Runtime Metrics      Traces             Logs
          │                  │                  │
          └──────────────────┼──────────────────┘
                             ▼
                  ┌─────────────────────┐
                  │ Event Normalization │
                  └──────────┬──────────┘
                             ▼
                  ┌─────────────────────┐
                  │ Behavioral Data    │
                  │ Warehouse           │
                  └──────────┬──────────┘
                             ▼
                  ┌─────────────────────┐
                  │ Debt Analyzer       │
                  ├─────────────────────┤
                  │ Change Analysis     │
                  │ Coupling Analysis   │
                  │ Failure Analysis    │
                  │ Test Analysis       │
                  │ Deploy Analysis     │
                  │ Performance Analysis│
                  └──────────┬──────────┘
                             ▼
                  ┌─────────────────────┐
                  │ Debt Explanation    │
                  └──────────┬──────────┘
                             ▼
       ┌─────────────────────┼─────────────────────┐
       ▼                     ▼                     ▼
   Dashboard              CI/CD                 IDE

This is not just a technical-debt calculator.

It is an engineering behavior observability platform.

The Bigger Idea

For years, software engineering has tried to measure technical debt by looking at the software itself.

Count the bad smells.

Count the warnings.

Count the duplicated code.

Count the complexity.

But software is not just source code.

Software is behavior.

And technical debt is fundamentally about how that behavior changes the economics of engineering.

If a small feature requires touching twenty files, that is behavior.

If a deployment takes two hours, that is behavior.

If a single database failure takes down five services, that is behavior.

If engineers repeatedly rewrite the same tests, that is behavior.

If nobody can diagnose an incident without searching through six systems, that is behavior.

Those behaviors are measurable.

And once they are measurable, they can become signals.

The most useful technical-debt system may therefore not ask:

"How bad is this code?"

It may ask:

"What is this system making engineers do?"

That question is much deeper.

Because technical debt is ultimately paid by people.

Developers pay it with extra hours.

Teams pay it with slower releases.

Companies pay it with operational risk.

Users pay it with bugs, downtime, latency, and missing features.

A codebase can look beautiful while quietly taxing everyone who touches it.

Another codebase can look ugly while being remarkably cheap to operate.

That is why behavioral measurement matters.

The future of technical-debt tooling should move beyond static analysis.

It should observe the entire engineering system.

Commits.

Pull requests.

Builds.

Tests.

Deployments.

Incidents.

Traces.

Performance.

Failures.

Recovery.

Change patterns.

And most importantly, the relationship between all of them.

Because the deepest signal of technical debt is not that the code became ugly.

It is that change became expensive.

And when a system can measure that expense continuously, technical debt stops being an argument in an architecture meeting.

It becomes an observable property of the system.

Something we can detect.

Something we can explain.

Something we can track over time.

And eventually, something we can manage deliberately.

That is the interesting future of technical debt:

not measuring how ugly the code is, but measuring how much friction the architecture creates.

Code Fingerprinting: Detecting Duplicate Submissions Without Losing Your Mind (or Your API Budget)

Sat, 19 Sep 2026 07:49:18 +0000

While building LogicVisor's review system, I was faced with a question I hadn't thought about before: what would happen if users submitted duplicate algorithm solutions?

I decided to test it myself. I submitted a solution to a simple palindrome challenge. The request went through, my solution was reviewed correctly. Normal happy stuff. Then I submitted the exact same solution a second time. And the normal happy stuff happened again. But I was far from happy.

That duplicate submission meant more tokens spent, more database memory consumed, more remote procedure calls, more AI calls, and more waiting time. For a problem that had already been solved.

I shuddered at the thought of a bad actor effectively DOS-ing my site or running up my AI bill just by hammering the same solution repeatedly. Rate limiting controls the number of requests per client — but it has nothing to do with the content of those requests. A user could stay well within their rate limit and still abuse the system by submitting the same code over and over.

This wasn't going to work. I needed a way to identify duplicate submissions and return a cached response instead of processing everything from scratch. Which led me to a more interesting question: how do you actually detect identical code?

The Problem With "Identical"

Programming by its very nature allows us to solve the same problem in multiple ways. But even a single solution can be written in an infinite number of ways.

User A could create a function that takes and returns a string, and name it strawberries. User B could write the exact same function with the exact same logic and name it bananas. Different names, same implementation, same result.

For LogicVisor's purposes, these should be counted as duplicates. But how do you get a computer to see past the surface differences and recognize that two pieces of code are structurally the same?

I started thinking about it from first principles. At the end of the day, programming languages get compiled down to binary and then to electrical pulses. If two solutions solve the same problem the same way, they're ultimately firing the same set of pulses at the machine level. Variable names be damned.

So the question became: can I strip a solution down to its most fundamental representation and compare that?

Enter the Abstract Syntax Tree

More research led me to the Abstract Syntax Tree — something my OS professor had mentioned in school that I'd filed away as "probably never useful in web dev." No knowledge is wasted.

An Abstract Syntax Tree (AST) is a hierarchical, tree-shaped data structure that represents the syntactic structure of source code. It strips away superficial syntax — semicolons, curly braces, whitespace, comments — and keeps only the structural and semantic skeleton of the code. The shape of what you wrote, not the clothing it's wearing.

This was it. But I hit a wall almost immediately: I can't store and index an entire AST reliably in PostgreSQL. It's a complex nested structure, not a scalar value you can slap an index on.

Then I remembered an article about handling duplicate file uploads in Express with Multer. The trick was generating a SHA-256 hash of the file's content. A hash is a fixed-length fingerprint — the same input always produces the same output, and different inputs produce different outputs. Perfect.

The plan crystallized:

Parse the submitted code into an AST
Normalize the AST (rename variables, strip comments, etc.)
Hash the normalized output with SHA-256
Store and index that hash
On every new submission, compute the hash and check if it already exists

If the hash is in the database, return the cached review. No AI call. No wasted tokens. No waiting.

Two Parsers for the Price of One

LogicVisor supports eight languages: JavaScript, TypeScript, Python, Java, C, C++, Go, and Rust. Getting an AST for all of them required two different tools, each with their own approach to the world.

Babel Parser (for JavaScript and TypeScript)

Babel is something most JavaScript developers know as a transpiler — the thing that converts modern JS to something older browsers can run. But under the hood, Babel has a full-featured parser that produces a detailed AST of your JavaScript or TypeScript code. The @babel/parser package exposes this directly.

I'd used Babel's output for years but never touched the parser API directly. It's surprisingly clean. You hand it source code and options, and it gives you back a structured AST you can traverse and manipulate.

What made Babel the right choice for JS/TS specifically:

Native TypeScript support via a plugin — no separate toolchain
JSX support out of the box
The @babel/traverse package lets you walk the AST and modify nodes in place, which is exactly how I rename identifiers
@babel/generator converts the modified AST back to compact source code

The JS/TS canonicalization path: parse with Babel → traverse the AST and rename all non-builtin identifiers to generic names (v0, v1, v2...) → optionally sort top-level declarations → regenerate compact source → hash it.

Tree-sitter (for everything else)

Tree-sitter is a different beast. It's a parser generator — a tool for building fast, incremental, error-tolerant parsers for programming languages. You might have encountered it without knowing it: if you've used Neovim with syntax highlighting that actually understands your code structure (my first encounter with the tool), that's Tree-sitter. Same if you've used editors where renaming a variable highlights every reference correctly. It's doing real parsing, not just regex matching.

What makes Tree-sitter interesting for this use case is that it has community-maintained grammars for a huge range of languages — C, C++, Go, Rust, Java, Python, and many more. Each grammar is a separate package that teaches Tree-sitter how to parse that language. You install the grammar, set it on a parser instance, and Tree-sitter gives you a concrete syntax tree you can walk.

I hadn't really understood what Tree-sitter was before this. I'd seen it referenced in Neovim config files and assumed it was some Vim-specific thing. It's not — it's a general purpose parsing library with bindings for multiple languages including Node.js via the node-tree-sitter package.

The Tree-sitter path: parse the source → walk the resulting syntax tree with a DFS traversal → collect all identifier nodes by their byte ranges → build a name mapping (first-occurrence order) → apply replacements from the end of the file backwards (so byte indices don't shift) → normalize whitespace → hash it.

The backwards-replacement trick is worth calling out. When you're doing string replacements by index, if you replace something early in the file, every subsequent index is now wrong. By sorting replacements by start position descending and working from the end, you avoid that entire class of bug.

The Implementation

Here's how it all fits together.

The Simple Canonicalization (Pre-AST)

Before I landed on the AST approach, I had a simpler version that just handled whitespace and comments:

export function canonicalizeCode(code: string): string {
  // Remove multi-line comments
  let cleanedCode = code.replace(/\/\*[\s\S]*?\*\//g, "");

  // Remove single-line comments
  cleanedCode = cleanedCode.replace(/\/\/.*$/gm, "");

  // Normalize all whitespace to a single space
  cleanedCode = cleanedCode.replace(/\s+/g, " ").trim();

  return cleanedCode;
}

This catches the obvious cases — same code with different formatting or comments. But it falls apart the moment someone renames a variable. The AST approach handles that.

The Router

The main entry point checks the language and dispatches to the right parser:

export async function canonicalizeCodeAST(
  code: string,
  language: LanguageKey,
  options?: CanonicalizeOptions
): Promise<string> {
  const opts = { ...defaultOptions, ...(options || {}) };

  if (language === "javascript" || language === "typescript") {
    return canonicalizeJS(code, opts);
  }

  // C/C++/Go/Rust/Java/Python go through Tree-sitter
  return canonicalizeWithTreeSitter(code, language, null);
}

The Babel Path (JS/TS)

function canonicalizeJS(
  code: string,
  opts: Required<CanonicalizeOptions>
): string {
  const plugins: any[] = ["jsx"];
  if (opts) plugins.push("typescript");

  const ast = babelParser.parse(code, {
    sourceType: "module",
    plugins,
    allowReturnOutsideFunction: true,
  });

  // Rename all non-builtin identifiers to generic names
  const nameMap = new Map<string, string>();
  let counter = 0;

  traverse(ast as any, {
    Identifier(path) {
      if (!opts.normalizeVariableNames) return;

      const name = path.node.name;
      if (isBuiltinJsName(name)) return;

      if (!nameMap.has(name)) {
        nameMap.set(name, `v${counter++}`);
      }

      path.node.name = nameMap.get(name)!;
    },
  });

  // Optionally sort top-level declarations for order-independent comparison
  if (
    opts.sortTopLevel &&
    (ast as any).program &&
    Array.isArray((ast as any).program.body)
  ) {
    (ast as any).program.body.sort((a: any, b: any) => {
      const aStr = generate(a).code;
      const bStr = generate(b).code;
      return aStr.localeCompare(bStr);
    });
  }

  // Regenerate compact source from the modified AST
  return generate(ast as any, {
    comments: !opts.removeComments,
    compact: true,
  }).code;
}

The Tree-sitter Path (Everything Else)

function canonicalizeWithTreeSitter(
  code: string,
  language: LanguageKey,
  opts: Required<CanonicalizeOptions> | null
): string {
  const LangCtor = (treeSitterMap as any)[language];

  if (!LangCtor) {
    throw new Error(`Unsupported language for tree-sitter: ${language}`);
  }

  const parser = new Parser();
  parser.setLanguage(LangCtor);

  const tree = parser.parse(code);
  const root = tree.rootNode;

  const identifierNodeTypes = identifierTypeMap[language] || ["identifier"];
  const idNodes: { start: number; end: number; text: string }[] = [];
  const commentNodes: { start: number; end: number }[] = [];

  // DFS traversal to collect identifier and comment nodes
  const stack = [root];
  while (stack.length) {
    const node = stack.pop()!;
    const type = node.type;

    if (type.includes("comment") && opts?.removeComments) {
      commentNodes.push({ start: node.startIndex, end: node.endIndex });
    }

    if (identifierNodeTypes.includes(type) || /identifier|name/i.test(type)) {
      const text = code.slice(node.startIndex, node.endIndex);
      idNodes.push({ start: node.startIndex, end: node.endIndex, text });
    }

    for (let i = 0; i < node.childCount; i++) {
      stack.push(node.child(i) as SyntaxNode);
    }
  }

  // Build name mapping in first-occurrence order
  const nameMap = new Map<string, string>();
  let counter = 0;

  for (const n of idNodes) {
    const orig = n.text;
    if (isLanguageBuiltinName(orig, language)) continue;
    if (!nameMap.has(orig)) {
      nameMap.set(orig, `v${counter++}`);
    }
  }

  // Build replacement list
  const replacements: { start: number; end: number; replacement: string }[] = [];

  if (opts?.removeComments) {
    for (const c of commentNodes) {
      replacements.push({ start: c.start, end: c.end, replacement: "" });
    }
  }

  if (opts?.normalizeVariableNames) {
    for (const id of idNodes) {
      const mapped = nameMap.get(id.text);
      if (mapped) {
        replacements.push({ start: id.start, end: id.end, replacement: mapped });
      }
    }
  }

  // Apply replacements from end to start so byte indices stay valid
  replacements.sort((a, b) => b.start - a.start);

  let out = code;
  for (const r of replacements) {
    out = out.slice(0, r.start) + r.replacement + out.slice(r.end);
  }

  return out.trim();
}

Hashing the Canonical Form

Once we have the normalized source, hashing is straightforward:

export async function createCanonicalHash(canonicalCode: string): Promise<string> {
  const encoder = new TextEncoder();
  const data = encoder.encode(canonicalCode);
  const hashBuffer = await crypto.subtle.digest("SHA-256", data);
  const hashArray = Array.from(new Uint8Array(hashBuffer));
  return hashArray.map((b) => b.toString(16).padStart(2, "0")).join("");
}

SHA-256 gives us a fixed 64-character hex string regardless of how long the original code was. That's what gets stored and indexed in PostgreSQL.

Putting It All Together in the Request Handler

let canonicalCode;
try {
  canonicalCode = await canonicalizeCodeAST(solution, preferred_language);
} catch (error) {
  console.error("Canonicalization error:", error);
  return NextResponse.json(
    {
      success: false,
      error: {
        code: "BAD_USER_INPUT",
        message: "Unable to parse your code. Please check for syntax errors and try again.",
      },
    },
    { status: 400 }
  );
}

const canonicalHash = await createCanonicalHash(
  typeof canonicalCode === "string" ? canonicalCode : ""
);

// Check cache before touching the AI provider
const cachedReview = await getCachedAIReview(
  preferred_model.id + "-" + canonicalHash
);

if (cachedReview) {
  return NextResponse.json(
    { success: true, data: cachedReview },
    { status: 201 }
  );
}

// No cache hit — proceed with AI review and store the result

The model ID is included in the cache key because the same code reviewed by Gemini and Groq will produce different outputs. You don't want to serve a Gemini review to someone who selected Groq.

Does It Actually Work?

Here's the test that proved it out. Five TypeScript implementations of the same palindrome algorithm — different variable names, different formatting, different comments, and one with operand order swapped:

// Solution 1 — clean, descriptive names
function isPalindrome(x: number): boolean {
  let originalNum = x;
  let reversedNum = 0;
  while (originalNum > 0) {
    let lastDigit = originalNum % 10;
    reversedNum = (reversedNum * 10) + lastDigit;
    originalNum = Math.floor(originalNum / 10);
  }
  return x === reversedNum;
}

// Solution 2 — different variable names
function isPalindrome(n: number): boolean {
  let temp = n;
  let result = 0;
  while (temp > 0) {
    let digit = temp % 10;
    result = (result * 10) + digit;
    temp = Math.floor(temp / 10);
  }
  return n === result;
}

// Solution 3 — compressed formatting + comments
function isPalindrome(x: number): boolean{
  // Store original
  let originalNum=x;let reversedNum=0;
  while(originalNum>0){
    let lastDigit=originalNum%10; /* get digit */
    reversedNum=(reversedNum*10)+lastDigit;
    originalNum=Math.floor(originalNum/10);
  }
  return x===reversedNum;
}

// Solution 4 — addition operands swapped
function isPalindrome(x: number): boolean {
  let originalNum = x;
  let reversedNum = 0;
  while (originalNum > 0) {
    let lastDigit = originalNum % 10;
    reversedNum = lastDigit + (reversedNum * 10); // addition order changed
    originalNum = Math.floor(originalNum / 10);
  }
  return reversedNum === x; // comparison order changed
}

// Solution 5 — completely different algorithm
function isPalindrome(x: number): boolean {
  const str = x.toString();
  return str === str.split('').reverse().join('');
}

And the test harness:

export async function testDuplicateDetection() {
  const solutions = [ /* ...the five solutions above... */ ];

  console.log("Code hash comparison:");

  solutions.forEach(async (code, index) => {
    const canonicalized = await canonicalizeCodeAST(code, "typescript");
    const hash = await createCanonicalHash(canonicalized);
    console.log(`\nSolution ${index + 1}:`);
    console.log(`Hash: ${hash.substring(0, 12)}...`);
    console.log(
      `Canonical: ${
        typeof canonicalized === "string" ? canonicalized.substring(0, 100) : ""
      }...`,
    );
  });

  // Canonicalize before hashing — not after
  const hashes = await Promise.all(
    solutions.map(async (code) => {
      const canonicalized = await canonicalizeCodeAST(code, "typescript");
      return createCanonicalHash(canonicalized);
    }),
  );
  const uniqueHashes = new Set(hashes);

  console.log(`\nOriginal solutions: ${solutions.length}`);
  console.log(`Unique hashes: ${uniqueHashes.size}`);
  console.log(`Duplicates detected: ${solutions.length - uniqueHashes.size}`);
}

The output:

Code hash comparison:

Solution 1:
Hash: f7142f71e35a...
Canonical: function v0(v1:number):boolean{let v2=v1;let v3=0;while(v2>0){let v4=v2%10;v3=v3*10+v4;v2=Math.v5(v2...

Solution 2:
Hash: f7142f71e35a...
Canonical: function v0(v1:number):boolean{let v2=v1;let v3=0;while(v2>0){let v4=v2%10;v3=v3*10+v4;v2=Math.v5(v2...

Solution 3:
Hash: f7142f71e35a...
Canonical: function v0(v1:number):boolean{let v2=v1;let v3=0;while(v2>0){let v4=v2%10;v3=v3*10+v4;v2=Math.v5(v2...

Solution 4:
Hash: da8268fe24dd...
Canonical: function v0(v1:number):boolean{let v2=v1;let v3=0;while(v2>0){let v4=v2%10;v3=v4+v3*10;v2=Math.v5(v2...

Solution 5:
Hash: e3d2696010ad...
Canonical: function v0(v1:number):boolean{const v2=v1.v3();return v2===v2.v4('').v5().v6('');}...

Original solutions: 5
Unique hashes: 3
Duplicates detected: 2

Solutions 1, 2, and 3 collapse to the same hash — variable names, whitespace, and comments are all stripped away during canonicalization, leaving identical ASTs. Solution 4 gets a distinct hash because operand order (lastDigit + reversedNum 10 vs reversedNum 10 + lastDigit) is a syntactic difference the AST preserves — commutativity is math, not structure, and the canonicalizer doesn't evaluate expressions. Solution 5 is correctly identified as a genuinely different algorithm.
Two duplicates detected. Working as intended.

The Tradeoffs Worth Knowing

This approach isn't magic. A few things it doesn't catch:

Semantic equivalence across algorithms. Two completely different implementations that produce the same result — bubble sort vs. merge sort, for example — will have different hashes. This system detects structural duplicates, not algorithmic equivalence. That's actually the right behavior for a code review platform.

Heuristic identifier detection in Tree-sitter. The identifier node types vary between language grammars, and I'm using pattern matching (/identifier|name/i) as a fallback. It works well in practice but it's not exhaustive. If a grammar names its identifier nodes something unusual, you might miss some.

Builtin lists are incomplete. The isBuiltinJsName and isLanguageBuiltinName functions cover the most common builtins but aren't exhaustive. If a user names their variable fetch or setTimeout, those won't be normalized. Not a critical flaw, but worth knowing.

What I Took Away From This

The part I didn't expect: how much I'd enjoy it. This was one of those problems that pulled me away from the normal web dev loop of "call an API, render some data" and into something that felt closer to computer science. Parsers, ASTs, tree traversal — things I'd learned in school that felt abstract at the time suddenly had a real home in a real production system.

The other thing I took away: sometimes the right caching strategy isn't about time-to-live or cache invalidation. Sometimes it's about recognizing that two requests are asking for the same thing, even when they look different on the surface. That realization changed how I thought about a whole class of optimization problems.

If you're building anything that takes code as input — a linter, a reviewer, a plagiarism detector, an interview platform — the AST + hash pattern is worth having in your toolkit.

LogicVisor is an AI-powered tool that reviews your algorithmic code, breaks down time and space complexity, and gives you the kind of feedback you'd want before a technical interview. You can try it at logicvisor.vercel.app.

Building a Dynamic Pricing Engine That Learns From Demand

Fri, 18 Sep 2026 20:01:14 +0000

Most software systems are built around a simple assumption:

The price is already known.

A product has a price.

A ticket has a price.

A hotel room has a price.

A delivery has a price.

The application simply retrieves that number from a database and shows it to the customer.

But what if the price itself were part of the computation?

What if the system could observe demand, inventory, customer behavior, time, competition, and historical transactions — and continuously adjust prices based on what it learns?

Now we are no longer building a CRUD application.

We are building a dynamic pricing engine.

And dynamic pricing is much more interesting than:

price = 100

It becomes:

price = f(
    demand,
    inventory,
    time,
    conversion_rate,
    historical_sales,
    seasonality,
    competition,
    business_rules
)

The price becomes an output of a system.

That system observes the world.

The world changes.

The system learns.

And the price changes with it.

This article walks through how to design such an engine from scratch.

1. The Core Idea

Imagine an online store selling a product.

Suppose the product costs:

$100

At 9:00 AM:

Demand = low
Inventory = high
Conversion = 2%

The engine might keep the price near:

$100

But at 6:00 PM:

Demand = extremely high
Inventory = low
Conversion = 9%

The engine might calculate:

$118

Later, demand collapses:

Demand = low
Inventory = high
Conversion = 1%

The price could move toward:

$94

The important part is that these numbers are not manually configured one by one.

The system is responding to observed conditions.

Conceptually:

                 ┌──────────────┐
                 │   Customers  │
                 └──────┬───────┘
                        │
                        ▼
                 ┌──────────────┐
                 │ Event Stream │
                 └──────┬───────┘
                        │
       ┌────────────────┼────────────────┐
       │                │                │
       ▼                ▼                ▼
    Demand          Inventory        Conversion
    Signals           Signals          Signals
       │                │                │
       └────────────────┼────────────────┘
                        ▼
                ┌─────────────────┐
                │ Pricing Engine  │
                └────────┬────────┘
                         │
                         ▼
                  New Price
                         │
                         ▼
                    Customer

This is the architecture we want.

2. Why Dynamic Pricing Is Difficult

At first glance, dynamic pricing looks like a mathematical problem.

It isn't only mathematical.

It is simultaneously:

a data problem
a distributed systems problem
a machine learning problem
an optimization problem
a financial system
a consistency problem
a product design problem

Consider a simple pricing formula:

price = base_price * demand_multiplier

That looks easy.

But where does demand_multiplier come from?

Suppose 10,000 customers are browsing simultaneously.

Some are purchasing.

Some are abandoning carts.

Some are repeatedly refreshing the product page.

Some are buying multiple units.

Inventory is decreasing.

Competitors are changing their prices.

The time of day is changing.

A promotion has started.

Your model has not seen this exact situation before.

Now the problem becomes much more interesting.

3. Define the Pricing Objective

Before writing the algorithm, define what the engine is trying to optimize.

Different businesses have different objectives.

For example:

maximize revenue

or:

maximize profit

or:

maximize inventory utilization

or:

maximize conversion

or:

balance revenue and customer retention

A pricing engine without an objective is just a number generator.

Let's define a simple objective:

maximize expected profit

Expected profit can be represented as:

Expected Profit =
(price - variable_cost)
× expected_quantity_sold

But there is a catch.

Increasing price can reduce quantity sold.

Therefore:

profit(price)

is not necessarily increasing with price.

For example:

Price     Expected Sales     Revenue
$80       100                 $8,000
$100      90                  $9,000
$120      65                  $7,800
$140      40                  $5,600

The engine needs to find the region where the business objective is strongest.

That is an optimization problem.

4. Build the Data Pipeline First

The pricing algorithm is only as good as the signals feeding it.

We need events.

For example:

{
  "event": "product_view",
  "product_id": "P100",
  "user_id": "U500",
  "timestamp": "2026-09-18T18:30:00Z"
}

Purchase:

{
  "event": "purchase",
  "product_id": "P100",
  "quantity": 2,
  "price": 110,
  "timestamp": "2026-09-18T18:31:12Z"
}

Cart abandonment:

{
  "event": "cart_abandoned",
  "product_id": "P100",
  "timestamp": "2026-09-18T18:35:00Z"
}

Inventory:

{
  "product_id": "P100",
  "inventory": 37,
  "timestamp": "2026-09-18T18:40:00Z"
}

The system can consume these events through a stream.

Conceptually:

Application
     │
     ▼
Event Collector
     │
     ▼
Message Broker
     │
     ├───────────────┐
     ▼               ▼
Analytics       Pricing Engine
     │               │
     ▼               ▼
Data Lake        Price Store

Technologies could include Kafka, Redis Streams, NATS, PostgreSQL, ClickHouse, or other systems depending on scale.

The technology is secondary.

The architecture is the important part.

5. Turn Raw Events Into Demand Signals

Raw events are not directly useful for pricing.

We need features.

Suppose the last hour produced:

views = 12,000
carts = 1,200
purchases = 360
inventory = 80

We can derive:

cart_rate = carts / views

Therefore:

cart_rate = 10%

And:

conversion_rate = purchases / views

Which gives:

conversion_rate = 3%

We could also calculate:

sales_velocity =
units_sold / time_window

For example:

sales_velocity = 40 units/hour

With 80 units remaining:

estimated_hours_to_stockout = 80 / 40

Therefore:

2 hours

That is an extremely valuable signal.

The engine can now reason:

Demand is high and the product may sell out within two hours.

6. Demand Is a Time Series

Demand is rarely constant.

Imagine:

Hour       Sales
08:00      10
09:00      13
10:00      15
11:00      17
12:00      24
13:00      29
14:00      34
15:00      31
16:00      27
17:00      41

The engine should understand trends.

A simple moving average is a useful starting point.

def moving_average(values, window):
    if len(values) < window:
        return sum(values) / len(values)

    return sum(values[-window:]) / window

For example:

recent_sales = [24, 29, 34, 31, 27]

demand = moving_average(recent_sales, 5)

This smooths noisy observations.

But smoothing creates another problem.

You can become too slow to react.

If demand suddenly explodes, a long moving average may hide the spike.

That is where exponential weighting becomes useful.

7. Exponentially Weighted Demand

We can update demand continuously:

new_demand = (
    alpha * current_sales
    + (1 - alpha) * previous_demand
)

Where:

0 < alpha <= 1

A larger alpha means the system reacts faster.

For example:

alpha = 0.7

makes the engine highly responsive.

While:

alpha = 0.1

makes it much more stable.

This creates a fundamental engineering tradeoff:

responsiveness
       vs
stability

A pricing engine that reacts too quickly can oscillate.

A pricing engine that reacts too slowly can miss opportunities.

8. Introduce Inventory Pressure

Demand alone is insufficient.

Suppose two products both receive:

100 purchases/hour

Product A has:

10,000 units

Product B has:

100 units

Their pricing situations are completely different.

We can define an inventory pressure signal:

inventory_pressure =
1 - (inventory / target_inventory)

If inventory is below target, pressure increases.

For example:

inventory = 200
target = 1000

Then:

inventory_pressure = 0.8

The engine can interpret this as:

high inventory pressure

We can combine it with demand:

pressure =
demand_score * inventory_pressure

Now the pricing engine understands not just how many people want the product, but how urgently the business needs to respond.

9. Build a Pricing Function

Let's start with a simple model.

def calculate_price(
    base_price,
    demand_score,
    inventory_pressure,
    conversion_score
):
    multiplier = (
        1
        + 0.20 * demand_score
        + 0.15 * inventory_pressure
        + 0.10 * conversion_score
    )

    return base_price * multiplier

This is not a production pricing model.

It is a starting point.

Suppose:

base_price = $100
demand_score = 0.8
inventory_pressure = 0.7
conversion_score = 0.6

Then:

multiplier =
1
+ 0.20(0.8)
+ 0.15(0.7)
+ 0.10(0.6)

Approximately:

1.35

Price:

$135

Now we have a functioning dynamic pricing mechanism.

But there is a dangerous problem.

10. Never Let the Model Control Everything

Machine learning systems can make surprising decisions.

Pricing is especially sensitive because mistakes directly affect money.

Therefore, we need guardrails.

For example:

MIN_MULTIPLIER = 0.80
MAX_MULTIPLIER = 1.30

Then:

multiplier = max(
    MIN_MULTIPLIER,
    min(multiplier, MAX_MULTIPLIER)
)

The engine can never move beyond the configured boundary.

We can also impose:

maximum change per hour
minimum margin
minimum price
maximum price
inventory constraints
promotion constraints
legal constraints

The final architecture becomes:

Signals
   │
   ▼
Model
   │
   ▼
Candidate Price
   │
   ▼
Business Rules
   │
   ▼
Risk Controls
   │
   ▼
Final Price

The model proposes.

The policy layer decides whether the proposal is allowed.

That separation is extremely important.

11. Add Price Elasticity

Now we reach the more interesting part.

Customers respond to price.

If we increase the price, demand may decrease.

This relationship is called price elasticity of demand.

A simplified formulation is:

elasticity =
% change in quantity
--------------------
% change in price

Suppose:

price increases 10%
sales decrease 5%

Then elasticity is approximately:

-0.5

The negative sign represents the typical inverse relationship between price and demand.

A pricing engine can learn this relationship from historical observations.

Suppose we store:

price
quantity
timestamp
inventory
promotion
traffic

We can estimate how quantity changes as price changes.

The system can then ask:

If I set the price to $110 instead of $100, how many units do I expect to sell?

That question is much more powerful than simply saying:

Demand is high.

12. Learn From Experiments

Here is where the system starts becoming genuinely adaptive.

Suppose the engine always charges:

$100

It will have a huge amount of data about demand at $100.

But almost no information about:

$90
$95
$105
$110
$120

This creates a problem.

The model cannot learn relationships it never observes.

Therefore, the pricing system needs controlled experimentation.

For example:

90% traffic → current optimal price
10% traffic → experimental price

Suppose:

Control price = $100
Experimental price = $110

We observe:

Control conversion = 5.0%
Experiment conversion = 4.6%

But revenue per visitor becomes:

control:
100 × 0.05 = $5.00

experiment:
110 × 0.046 = $5.06

The experiment suggests that the higher price might produce more revenue per visitor.

Now the engine has learned something.

13. Exploration vs Exploitation

This creates one of the classic problems in intelligent systems:

exploration vs exploitation.

Exploitation means:

Use what we currently believe is best.

Exploration means:

Try something uncertain so we can learn.

A pricing engine needs both.

If we only exploit:

use current best price forever

the model stops learning.

If we explore too aggressively:

randomly change prices constantly

the system becomes unstable and potentially harmful to the business.

A simple strategy is:

95% exploitation
5% exploration

More advanced systems can use:

multi-armed bandits
Thompson sampling
Bayesian optimization
contextual bandits
reinforcement learning

The key insight is simple:

A learning system needs opportunities to learn.

14. Contextual Pricing

Price does not exist in isolation.

The same product can behave differently depending on context.

For example:

Monday morning
Friday evening
holiday
rainy day
payday
end of month

Demand can change.

We can represent context as features:

features = {
    "hour": 18,
    "day_of_week": 5,
    "month": 9,
    "inventory": 83,
    "sales_velocity": 42,
    "conversion_rate": 0.07,
    "traffic": 12000
}

The model then estimates:

expected_demand(price, context)

Now pricing becomes a prediction problem.

15. The Architecture

A more complete system could look like this:

                    ┌───────────────────┐
                    │ Web / Mobile Apps │
                    └─────────┬─────────┘
                              │
                              ▼
                    ┌───────────────────┐
                    │   Event Gateway   │
                    └─────────┬─────────┘
                              │
                              ▼
                    ┌───────────────────┐
                    │   Event Stream    │
                    └─────────┬─────────┘
                              │
             ┌────────────────┼────────────────┐
             │                │                │
             ▼                ▼                ▼
       Demand Service   Inventory Service   Analytics
             │                │                │
             └────────────────┼────────────────┘
                              ▼
                    ┌───────────────────┐
                    │ Feature Store     │
                    └─────────┬─────────┘
                              │
                              ▼
                    ┌───────────────────┐
                    │ Pricing Model     │
                    └─────────┬─────────┘
                              │
                              ▼
                    ┌───────────────────┐
                    │ Policy / Guardrail│
                    └─────────┬─────────┘
                              │
                              ▼
                    ┌───────────────────┐
                    │ Price Calculator  │
                    └─────────┬─────────┘
                              │
                              ▼
                    ┌───────────────────┐
                    │ Price Cache       │
                    └─────────┬─────────┘
                              │
                              ▼
                         Customers

There should also be a feedback loop:

Customers
   │
   ▼
Transactions
   │
   ▼
Events
   │
   ▼
Features
   │
   ▼
Model
   │
   ▼
New Prices
   │
   └──────────────► Customers

This is the heart of the system.

16. Separate Pricing From Checkout

This is an important architectural decision.

The pricing service should not be tightly coupled to checkout.

Instead:

GET /products/P100/price

could return:

{
  "product_id": "P100",
  "price": 109.50,
  "currency": "USD",
  "price_version": "pv_87231",
  "expires_at": "2026-09-18T19:05:00Z"
}

When checkout begins, the system records the price version.

order
 ├── product
 ├── quantity
 ├── unit_price
 ├── price_version
 └── currency

This protects historical correctness.

If the price changes five seconds later, an existing order should not magically change.

17. Price Versioning

Every generated price should have a version.

For example:

P100
Price: $109.50
Version: 87231

Then:

P100
Price: $112.00
Version: 87232

The system now has an audit trail.

We can answer:

What price did the customer see?

and:

Why was this price generated?

For example:

{
  "price": 112.00,
  "base_price": 100.00,
  "demand_score": 0.81,
  "inventory_pressure": 0.72,
  "conversion_rate": 0.071,
  "model_version": "model_42",
  "policy_version": "policy_7",
  "timestamp": "2026-09-18T19:02:00Z"
}

This turns the pricing engine into an observable system rather than a black box.

18. The Price Cache

Pricing is often requested far more frequently than prices need to be recalculated.

If 100,000 customers request the same product price, we should not run the model 100,000 times.

Instead:

Customer
   │
   ▼
API
   │
   ▼
Redis
   │
   ├── hit → return price
   │
   └── miss
         │
         ▼
      Pricing Engine

For example:

price:P100

could contain:

{
  "price": 109.50,
  "version": 87231,
  "expires_at": 1758222300
}

The TTL depends on how quickly the market changes.

For some businesses:

5 minutes

might be reasonable.

For others:

1 hour

could be enough.

The point is to separate price computation frequency from price retrieval frequency.

19. Avoid Price Oscillation

Imagine this:

Price = $100
Demand rises
Price = $110

Demand falls
Price = $98

Demand rises
Price = $112

Demand falls
Price = $97

The system is oscillating.

Customers may see unstable pricing.

The business may also struggle to reason about revenue.

A damping mechanism can help.

Instead of immediately moving to:

new_price = $120

we can move gradually:

new_price = (
    0.8 * old_price
    + 0.2 * candidate_price
)

If:

old_price = 100
candidate_price = 120

then:

new_price = 104

The engine moves toward the new equilibrium rather than jumping instantly.

20. Add Hysteresis

Another useful technique is hysteresis.

Suppose the engine only changes the price when the predicted improvement is significant.

For example:

if abs(candidate_price - current_price) < threshold:
    keep_current_price()

If the threshold is:

$3

then:

$100 → $101

does nothing.

But:

$100 → $108

triggers a change.

This prevents tiny fluctuations from constantly generating new prices.

21. Build the Learning Loop

A complete adaptive engine can follow this cycle:

1. Observe
2. Aggregate
3. Predict
4. Optimize
5. Validate
6. Publish
7. Measure
8. Learn
9. Repeat

In pseudocode:

while True:
    events = collect_recent_events()

    features = build_features(events)

    demand = predict_demand(features)

    candidate_prices = generate_candidates(
        base_price=features["base_price"]
    )

    best_price = optimize(
        candidate_prices,
        demand,
        features
    )

    safe_price = apply_guardrails(
        best_price,
        features
    )

    publish_price(safe_price)

    sleep(interval)

The system is effectively creating a feedback controller.

22. Make the Model Replaceable

Do not hard-code the pricing algorithm everywhere.

Create an interface:

class PricingModel:
    def predict(self, features):
        raise NotImplementedError

Then:

class RuleBasedModel(PricingModel):
    def predict(self, features):
        ...

Later:

class RegressionModel(PricingModel):
    def predict(self, features):
        ...

Later:

class BanditModel(PricingModel):
    def predict(self, features):
        ...

The pricing infrastructure remains stable while the intelligence evolves.

This is a powerful architecture pattern.

The system should not care which mathematical brain is currently inside it.

23. Database Design

A simple relational schema could look like:

CREATE TABLE products (
    id UUID PRIMARY KEY,
    base_price DECIMAL(12,2),
    cost DECIMAL(12,2),
    inventory INTEGER
);

Price history:

CREATE TABLE price_history (
    id BIGSERIAL PRIMARY KEY,
    product_id UUID NOT NULL,
    price DECIMAL(12,2) NOT NULL,
    model_version VARCHAR(100),
    demand_score DECIMAL(8,4),
    inventory_pressure DECIMAL(8,4),
    created_at TIMESTAMP NOT NULL
);

Experiments:

CREATE TABLE pricing_experiments (
    id UUID PRIMARY KEY,
    product_id UUID NOT NULL,
    control_price DECIMAL(12,2),
    experiment_price DECIMAL(12,2),
    traffic_ratio DECIMAL(5,4),
    started_at TIMESTAMP,
    ended_at TIMESTAMP
);

Transactions should also preserve the actual unit price:

CREATE TABLE orders (
    id UUID PRIMARY KEY,
    product_id UUID NOT NULL,
    quantity INTEGER NOT NULL,
    unit_price DECIMAL(12,2) NOT NULL,
    price_version BIGINT NOT NULL,
    created_at TIMESTAMP NOT NULL
);

This gives us both operational data and historical evidence.

24. Observability

A pricing engine without observability is dangerous.

We need metrics such as:

price_changes_per_minute
average_price
revenue_per_visitor
conversion_rate
gross_margin
inventory_turnover
model_error
prediction_confidence
experiment_lift

We should also log:

old price
new price
reason
model version
feature values
policy decision
timestamp

For example:

PRICE_CHANGE

product=P100
old=104.00
new=109.50

demand_score=0.82
inventory_pressure=0.71
conversion=0.068

model=model_42
policy=policy_7

When something goes wrong, this becomes invaluable.

25. Failure Modes

Dynamic pricing systems can fail in interesting ways.

Failure 1: Feedback loops

Suppose:

high demand → high price

Then customers buy less.

The system interprets:

lower sales → lower demand

and lowers the price.

Now demand rises again.

The system increases the price again.

You have created an unstable loop.

Failure 2: Bad data

Suppose a tracking bug reports:

10,000 purchases

when only:

occurred.

The pricing engine might interpret this as explosive demand.

Bad telemetry becomes bad pricing.

Failure 3: Cold start

A new product has:

zero historical sales

How do we price it?

The system needs defaults.

Possibilities include:

base pricing
category-level demand
similar-product data
manual configuration
conservative exploration

Failure 4: Model drift

Customer behavior changes.

A model trained on last year's behavior may become less accurate this year.

Therefore, models need monitoring and retraining.

26. The Most Important Safety Mechanism: Kill Switch

Every adaptive pricing engine should have a kill switch.

Something like:

pricing_engine.enabled = false

When disabled:

dynamic_price = base_price

This sounds simple.

It is one of the most important features in the entire system.

If the model starts generating abnormal prices, the business should be able to immediately return to a safe pricing policy.

Automation should always have a manual escape hatch.

27. Scaling the System

At small scale:

PostgreSQL
+
Redis
+
Background Worker

may be enough.

At larger scale:

Kafka
+
Stream Processing
+
Feature Store
+
Model Serving
+
Redis
+
OLAP Database

becomes more appropriate.

The architecture might evolve into:

              Events
                 │
                 ▼
              Kafka
                 │
        ┌────────┴────────┐
        ▼                 ▼
 Stream Processor      Data Lake
        │
        ▼
 Feature Store
        │
        ▼
 Model Server
        │
        ▼
 Pricing Optimizer
        │
        ▼
 Policy Engine
        │
        ▼
 Redis Price Cache
        │
        ▼
 Applications

The system grows because the workload grows.

The fundamental idea does not change.

28. The Deeper Idea

A dynamic pricing engine is not really about prices.

It is about feedback.

The system observes the environment.

It makes a decision.

The environment reacts.

The system observes that reaction.

Then it makes another decision.

That is a control loop.

Environment
     │
     ▼
Observation
     │
     ▼
Model
     │
     ▼
Decision
     │
     ▼
Action
     │
     ▼
Environment
     │
     └───────────────►

This same architecture appears in:

recommendation systems
robotics
automated trading
traffic systems
cloud autoscaling
game economies
energy management
logistics
fraud detection

The pricing engine is therefore an excellent project for learning intelligent systems.

It forces you to combine software architecture with mathematics.

29. From Rules to Intelligence

The first version might look like:

if demand > 0.8:
    price *= 1.10

That is rules.

The second version might estimate demand:

expected_demand(price, context)

That is prediction.

The third version might optimize price:

argmax price × expected_quantity(price)

That is optimization.

The fourth version might experiment:

try price A
try price B
observe results
update beliefs

That is learning.

The architecture evolves from:

rules
   ↓
prediction
   ↓
optimization
   ↓
experimentation
   ↓
adaptive decision-making

That progression is where the project becomes genuinely interesting.

30. A Practical Build Roadmap

If I were building this system from scratch, I would not start with reinforcement learning.

I would build it in stages.

Version 1 — Static Pricing

Implement:

products
base prices
inventory
checkout
price history

Version 2 — Rule-Based Pricing

Add:

demand score
inventory pressure
price bounds

Version 3 — Event-Driven Pricing

Add:

event ingestion
stream processing
real-time demand calculation
Redis price cache

Version 4 — Elasticity

Add:

historical price
historical quantity
price elasticity estimation

Version 5 — Prediction

Introduce:

machine learning model
feature engineering
demand forecasting
model versioning

Version 6 — Optimization

Allow the system to evaluate multiple prices:

$95
$100
$105
$110
$115

and estimate expected outcomes.

Version 7 — Experimentation

Introduce:

A/B testing
contextual bandits
controlled exploration

Version 8 — Full Adaptive Engine

Finally:

observe
→ predict
→ optimize
→ experiment
→ measure
→ learn
→ repeat

At this point, you have something far beyond a pricing table.

You have built a decision-making system.

Conclusion

A traditional pricing system answers:

What is the price?

A dynamic pricing engine asks a much more interesting question:

Given everything the system currently knows, what price should exist right now?

That small change in perspective completely changes the architecture.

You need event streams.

You need demand estimation.

You need feature engineering.

You need optimization.

You need experimentation.

You need price versioning.

You need caching.

You need guardrails.

You need observability.

And eventually, you need a learning loop.

The most important lesson is not the formula.

It is the architecture.

A truly adaptive system does not simply execute a fixed set of instructions.

It observes reality.

It turns reality into data.

It turns data into signals.

It turns signals into decisions.

And then it watches what happens next.

That is the architecture of a system that learns.

And once you understand that architecture, dynamic pricing becomes just one example.

You can apply the same thinking to inventory, advertising, logistics, recommendations, cloud infrastructure, fraud detection, and autonomous decision systems.

The interesting part of modern software is increasingly not just building systems that do things.

It is building systems that observe what happened, learn from it, and decide what to do next.

That is where software starts behaving less like a static machine and more like an adaptive organism.

And that is where engineering gets interesting.

Building an Adaptive Database That Changes Its Indexes Automatically

Fri, 18 Sep 2026 19:40:00 +0000

Most databases are static.

You create a table.

You create some indexes.

You deploy the application.

Then you hope the workload behaves the way you predicted.

But production workloads do not care about your predictions.

A query that ran once per hour during development can suddenly run 50,000 times per minute. A column that was rarely filtered yesterday can become the most important access path tomorrow. A new feature can completely change the shape of your workload.

And yet, the database may continue using the same indexes you designed months ago.

This creates an interesting question:

What if a database could watch its workload and change its indexes automatically?

Not simply rebuild an index when it becomes fragmented.

Not merely recommend indexes through an administrator dashboard.

But actually observe queries, measure their behavior, determine which indexes would improve performance, create those indexes when justified, remove indexes that no longer provide value, and continuously adapt as the workload changes.

That is the idea behind an adaptive database.

In this article, we are going to design one from scratch.

The goal is not to build a production replacement for PostgreSQL, MySQL, or another mature database.

The goal is more interesting:

We are going to understand the architecture behind a database that can learn from its workload.

The Database Is a Living System

Traditional database design looks something like this:

Developer
    |
    v
Schema
    |
    v
Indexes
    |
    v
Queries
    |
    v
Production

The indexes are mostly determined before the real workload arrives.

An adaptive database changes the relationship:

                 +------------------+
                 |   Query Workload |
                 +--------+---------+
                          |
                          v
                 +------------------+
                 | Workload Monitor |
                 +--------+---------+
                          |
                          v
                 +------------------+
                 | Query Analyzer   |
                 +--------+---------+
                          |
                          v
                 +------------------+
                 | Index Advisor    |
                 +--------+---------+
                          |
                          v
                 +------------------+
                 | Cost Evaluator   |
                 +--------+---------+
                          |
                 +--------+--------+
                 |                 |
              CREATE            DROP
                 |                 |
                 +--------+--------+
                          |
                          v
                 +------------------+
                 | Index Manager    |
                 +------------------+
                          |
                          v
                 +------------------+
                 | Database Engine  |
                 +------------------+

The important part is the feedback loop.

The database observes itself.

It makes a decision.

It changes itself.

Then it observes the consequences.

This is much closer to a control system than a traditional static configuration.

Start With the Workload

Before creating indexes automatically, we need to know what the database is actually doing.

Imagine a table:

CREATE TABLE orders (
    id BIGINT PRIMARY KEY,
    customer_id BIGINT,
    status VARCHAR(30),
    country VARCHAR(50),
    total DECIMAL(12,2),
    created_at TIMESTAMP
);

Suppose our application generates queries like:

SELECT *
FROM orders
WHERE customer_id = 42;

and:

SELECT *
FROM orders
WHERE status = 'pending';

and:

SELECT *
FROM orders
WHERE country = 'Zambia'
AND status = 'pending';

Initially, we may have only:

PRIMARY KEY (id)

The database starts observing.

For every query, we can record something like:

query_id
table
columns_used
predicates
frequency
execution_time
rows_examined
rows_returned

For example:

Query: Q17

Table:
orders

Predicates:
customer_id = ?

Frequency:
82,400/min

Average execution:
71ms

Rows examined:
1,200,000

Rows returned:
4

This is an enormous signal.

The database does not need a human to tell it that customer_id might deserve an index.

The workload is telling us.

The Query Fingerprint

One of the first problems we encounter is that applications rarely execute the exact same SQL string.

Consider:

SELECT *
FROM orders
WHERE customer_id = 42;

and:

SELECT *
FROM orders
WHERE customer_id = 781;

These are logically the same workload pattern.

We need to normalize them.

The database can transform both into:

SELECT *
FROM orders
WHERE customer_id = ?;

This becomes a query fingerprint.

A fingerprint might contain:

fingerprint = hash(
    normalized_sql
)

For example:

8f3c91ab

Then the database can maintain:

Fingerprint     Count       Avg Time
------------------------------------
8f3c91ab        2,841,201   68ms
9a21cc71          381,211   121ms
72be19fd           92,101    9ms

Now we can reason about workload patterns instead of individual queries.

Measuring Query Pain

Frequency alone is not enough.

Suppose we have two queries.

Query A

Runs:
1,000,000 times/day

Average:
2ms

Query B

Runs:
10,000 times/day

Average:
800ms

Which deserves attention?

Both might.

A useful metric is cumulative execution cost:

total_cost =
    frequency × average_execution_time

Therefore:

Query A:
1,000,000 × 2ms
= 2,000 seconds

Query B:

10,000 × 800ms
= 8,000 seconds

Query B consumes much more execution time despite running less frequently.

We could therefore rank workload patterns by:

impact =
    frequency
    × execution_time
    × rows_examined

The exact formula can evolve.

The important concept is that the database needs a way to determine:

Which queries are causing the most pain?

Candidate Index Generation

Now we reach the interesting part.

Suppose the database observes:

SELECT *
FROM orders
WHERE customer_id = ?;

It can generate:

CREATE INDEX idx_orders_customer_id
ON orders(customer_id);

For:

SELECT *
FROM orders
WHERE status = ?
AND country = ?;

it might generate:

CREATE INDEX idx_orders_status_country
ON orders(status, country);

But candidate generation becomes complicated very quickly.

Consider:

SELECT *
FROM orders
WHERE country = ?
AND status = ?
ORDER BY created_at DESC;

Possible indexes include:

(country)
(status)
(country, status)
(status, country)
(country, status, created_at)
(status, country, created_at)

We cannot blindly create all of them.

That would destroy the database.

Indexes consume:

disk space
memory
CPU during writes
background maintenance
cache capacity

An adaptive database therefore needs an index candidate system.

Indexes Have a Cost

An index is not free.

For a read-heavy workload, an index can be extremely valuable.

For a write-heavy workload, every additional index introduces maintenance work.

Suppose:

INSERT INTO orders (...)
VALUES (...);

Without secondary indexes, the database might update one structure.

With five secondary indexes, the same insert may need to update six structures.

Therefore we can think of an index as:

Index Value =
    Read Benefit
    -
    Write Cost
    -
    Storage Cost
    -
    Maintenance Cost

This is the core economic model of adaptive indexing.

We are not asking:

Can this index make a query faster?

We are asking:

Is making this index worth the cost?

Estimating Selectivity

One of the most important concepts in index optimization is selectivity.

Suppose we have one million orders.

If:

status = 'pending'

matches 500,000 rows, that predicate may be relatively unselective.

But:

customer_id = 93812

might match only 15 rows.

An index on customer_id can therefore be much more useful.

We can estimate:

selectivity =
    matching_rows / total_rows

Lower selectivity means fewer rows match.

For example:

customer_id:
15 / 1,000,000
= 0.000015

status:
500,000 / 1,000,000
= 0.5

The first predicate provides a much narrower search.

But selectivity alone cannot determine index usefulness.

A low-selectivity column may still be useful when combined with another predicate.

That is why adaptive indexing needs workload context.

Composite Indexes

This is where our system becomes more intelligent.

Consider:

SELECT *
FROM orders
WHERE country = 'Zambia'
AND status = 'pending';

An adaptive database might initially observe that both columns are commonly queried.

It could consider:

(country)
(status)
(country, status)
(status, country)

How do we choose?

We examine workload patterns.

Suppose we observe:

country + status:
70% of relevant queries

status only:
20%

country only:
10%

A composite index might provide greater benefit.

But column order matters.

An index:

(country, status)

is not equivalent to:

(status, country)

because B-tree traversal follows the index's ordering structure.

Our index advisor therefore needs to model not just columns, but query access patterns.

The Index Advisor

We can now design our advisor.

class IndexAdvisor:

    def analyze(workload):
        candidates = generate_candidates(workload)

        for index in candidates:
            benefit = estimate_benefit(index, workload)
            cost = estimate_cost(index)

            score = benefit - cost

            if score > threshold:
                recommend(index)

A more useful conceptual model is:

score(index) =
    expected_query_savings
    -
    write_penalty
    -
    storage_penalty
    -
    maintenance_penalty

The advisor should also consider how often the affected queries run.

A theoretical index saving 500ms on a query executed twice per day may not justify itself.

A 5ms improvement executed millions of times might.

The Database Needs a Memory

An adaptive database cannot make decisions from a single query.

It needs historical workload information.

For example:

Index:
idx_orders_customer_id

Created:
2026-09-01

Queries improved:
Q17, Q23

Reads served:
82,100,000

Average savings:
43ms

Write overhead:
7ms

Storage:
180MB

Now imagine the workload changes.

After several weeks:

Queries using index:
12/day

Meanwhile another index is handling millions of queries.

The database should eventually ask:

Why am I still maintaining this?

This creates the second half of adaptive indexing.

Index deletion.

Automatic Index Removal

Creating indexes is relatively easy.

Deleting them safely is much harder.

Imagine:

DROP INDEX idx_orders_customer_id;

What if the query workload changes tomorrow?

What if a hidden application query still depends on it?

What if the index is supporting a rare but critical operation?

A serious adaptive system needs confidence before removal.

We can define states:

ACTIVE
    |
    v
UNDERUSED
    |
    v
CANDIDATE_FOR_REMOVAL
    |
    v
SHADOWED
    |
    v
DROPPED

Instead of immediately deleting an index, we place it into a grace period.

For example:

No meaningful usage for:
30 days

Then we re-evaluate.

Shadow Mode

One particularly interesting technique is shadow mode.

Suppose we think an index is useless.

Instead of deleting it immediately, we can stop considering it during query planning while continuing to monitor what would happen.

Conceptually:

Actual planner:
uses other indexes

Shadow planner:
tests hypothetical old index

We compare:

actual_plan_cost
vs
plan_with_index_cost

If the index provides no meaningful benefit, confidence increases.

This allows the database to become conservative.

Adaptive systems should not make irreversible decisions too quickly.

Hypothetical Indexes

There is another powerful idea:

What if we could evaluate an index without actually creating it?

Suppose the database wants to know whether:

CREATE INDEX idx_orders_country_status
ON orders(country, status);

would improve a query.

We can construct a hypothetical index representation:

HypotheticalIndex {
    table: orders
    columns: [country, status]
    type: BTree
}

Then the query optimizer can estimate the plan as though the index existed.

This gives us:

Current plan:
Sequential Scan
Estimated cost: 18420

Hypothetical index:
Index Scan
Estimated cost: 312

That is powerful.

We can evaluate thousands of possible indexes without actually creating thousands of physical structures.

The Feedback Loop

Now the architecture starts looking like a machine-learning system.

Observe
   |
   v
Measure
   |
   v
Generate Candidates
   |
   v
Simulate
   |
   v
Choose
   |
   v
Apply
   |
   v
Measure Results
   |
   +------------------+
                      |
                      v
                   Observe

The system does not simply optimize once.

It continuously learns from consequences.

Suppose it creates:

idx_orders_customer_id

Before:

Average latency:
71ms

After:

Average latency:
4ms

The system records:

observed_benefit = 67ms

But perhaps write latency increases:

INSERT:
3ms -> 5ms

The real net benefit is therefore:

read savings
-
write overhead

This feedback lets future decisions become better.

Avoiding Oscillation

Adaptive systems have a dangerous failure mode:

oscillation.

Imagine the database does this:

Create index
↓
Reads become faster
↓
Writes become slower
↓
Drop index
↓
Writes become faster
↓
Reads become slower
↓
Create index
↓

The database is constantly changing itself.

That is terrible.

We need hysteresis.

For example:

Create threshold:
+1000 benefit units

Drop threshold:
-500 benefit units

The thresholds are deliberately different.

An index must become significantly valuable before creation.

And it must become significantly useless before removal.

We can also introduce a minimum lifetime:

minimum_index_age = 7 days

An index created yesterday cannot be dropped today.

Rate Limiting Schema Changes

Index creation itself can be expensive.

Suppose our adaptive engine discovers:

1,000 candidate indexes

We absolutely should not create them all.

We need a schema-change scheduler.

IndexJob {
    type: CREATE
    table: orders
    index: idx_orders_customer_id
    priority: 87
}

The scheduler can enforce:

max concurrent index builds = 1

or:

max disk growth/hour = 2GB

or:

max write amplification = 10%

This converts index adaptation into controlled background work.

Protecting the Database

An adaptive database should never be allowed to optimize itself into destruction.

We need safety constraints.

For example:

max_indexes_per_table = 12

max_index_size = 50GB

max_daily_index_growth = 10GB

max_write_overhead = 15%

min_query_frequency = 100/hour

These are guardrails.

The system might discover that a particular index would improve a query by 90%.

But if that index consumes 300GB on a 500GB database, the system should reconsider.

Optimization always exists inside constraints.

Index Merging and Consolidation

Another advanced feature is recognizing redundant indexes.

Suppose we have:

idx_orders_customer
    (customer_id)

idx_orders_customer_created
    (customer_id, created_at)

The second index may already support many queries that use customer_id.

Our advisor should recognize index relationships.

We can represent indexes as sets:

A = [customer_id]

B = [customer_id, created_at]

If B covers the access patterns of A, maintaining A may not be worthwhile.

The system can therefore perform index consolidation.

Instead of:

5 indexes

we might reduce the table to:

3 strategically chosen indexes

while preserving query performance.

A Simple Cost Model

Let's build a simplified mathematical model.

For an index I:

Benefit(I) =
Σ(query_frequency × latency_saved)

Then:

Cost(I) =
write_frequency × write_penalty
+
storage_cost
+
maintenance_cost

Therefore:

NetValue(I) =
Benefit(I) - Cost(I)

Suppose:

Query frequency:
2,000,000/day

Latency savings:
10ms

Then:

Benefit =
2,000,000 × 0.010
=
20,000 seconds/day

Now suppose writes introduce:

5ms overhead

across:

100,000 writes/day

Then:

Write cost =
100,000 × 0.005
=
500 seconds/day

Ignoring other costs:

Net value =
19,500 seconds/day

This index looks attractive.

The numbers are simplified, but the architecture is real.

Learning the Workload

Now we can go beyond static heuristics.

Imagine the system tracks:

hour
query
frequency
latency
rows_scanned
rows_returned

We may discover patterns such as:

09:00–12:00
customer lookup dominates

12:00–14:00
inventory queries dominate

18:00–22:00
order history dominates

The database can therefore become workload-aware.

This raises a fascinating possibility:

indexes could become time-aware.

Perhaps a workload is seasonal.

During a major shopping period:

orders by status

becomes dominant.

After the event:

customer history

returns to dominance.

An adaptive database can respond to the workload rather than assuming it is permanent.

But Don't Rebuild Everything Every Hour

A naive implementation would continuously run expensive analysis.

That would defeat the purpose.

Instead, we can use sampling.

For example:

collect:
100% of slow queries

10% of fast queries

1% of trivial queries

Or maintain aggregated counters directly inside the database.

We could store:

query_fingerprint
execution_count
total_latency
max_latency
rows_examined
last_seen

Then:

avg_latency =
total_latency / execution_count

This gives us useful information without storing every query event.

Architecture of the Adaptive Engine

A practical prototype could look like this:

+-----------------------------+
|        Application          |
+--------------+--------------+
               |
               v
+-----------------------------+
|        Query Engine         |
+--------------+--------------+
               |
               +--------------------+
               |                    |
               v                    v
       +---------------+     +-------------+
       | Query Metrics |     | Query Plan  |
       +-------+-------+     +------+------+
               |                    |
               +---------+----------+
                         |
                         v
                +----------------+
                | Workload Store |
                +-------+--------+
                        |
                        v
                +----------------+
                | Index Advisor  |
                +-------+--------+
                        |
                        v
                +----------------+
                | Cost Simulator |
                +-------+--------+
                        |
                        v
                +----------------+
                | Policy Engine  |
                +-------+--------+
                        |
                        v
                +----------------+
                | Schema Worker  |
                +-------+--------+
                        |
                        v
                    Database

The policy engine is important.

It decides whether the advisor is actually allowed to change the schema.

This separation prevents the optimization algorithm from having unlimited authority.

A Prototype Algorithm

We can write a simplified loop:

def adaptive_cycle():

    workload = collect_workload()

    hot_queries = rank_queries(workload)

    candidates = []

    for query in hot_queries:
        candidates.extend(
            generate_index_candidates(query)
        )

    candidates = deduplicate(candidates)

    for index in candidates:

        current_cost = estimate_current_cost(index.table)

        hypothetical_cost = estimate_cost_with_index(index)

        benefit = current_cost - hypothetical_cost

        write_cost = estimate_write_penalty(index)

        storage_cost = estimate_storage_cost(index)

        score = (
            benefit
            - write_cost
            - storage_cost
        )

        if score > CREATE_THRESHOLD:
            schedule_create(index)

    for index in existing_indexes():

        usage = measure_index_usage(index)

        if usage < DROP_THRESHOLD:
            schedule_removal_review(index)

This is not a complete database optimizer.

But it captures the central architecture.

The Hardest Problem: Causality

There is one subtle problem.

Suppose we create an index and query latency drops.

Did the index cause the improvement?

Maybe.

But maybe:

cache warmed up
traffic decreased
the application changed
the dataset changed
another index was created
the query plan changed

An adaptive database needs to reason about causality.

One approach is controlled experimentation.

For selected workloads, we can compare:

baseline plan

against:

candidate plan

using sampled queries.

We can also record before-and-after measurements:

Index creation:
10:00

Before:
p95 = 180ms

After:
p95 = 24ms

The more controlled the experiment, the stronger the evidence.

The Database as a Control System

At this point, something interesting becomes visible.

We are no longer merely designing a database feature.

We are designing a feedback controller.

The database has:

Input:
Query workload

State:
Indexes + statistics

Observation:
Latency + execution plans + resource usage

Controller:
Index advisor

Action:
Create/drop indexes

Feedback:
Measured performance

That gives us:

             +----------------------+
             |                      |
             v                      |
Workload → Database → Metrics → Controller
                         ^          |
                         |          |
                         +---- Actions

This is why adaptive databases are such an interesting systems project.

They combine:

databases
algorithms
observability
optimization
statistics
distributed systems
control theory
resource management

Going Distributed

Now imagine the database is distributed.

We have:

Node A
Node B
Node C
Node D

Should every node independently create indexes?

Probably not.

Otherwise we could get:

Node A:
create index X

Node B:
create index Y

Node C:
create index X

Node D:
create index Z

Now schema state diverges.

Instead, we need an index control plane.

              +----------------+
              | Index Control  |
              |     Plane      |
              +-------+--------+
                      |
          +-----------+-----------+
          |           |           |
          v           v           v
       Node A       Node B      Node C

The control plane can determine:

desired_index_state

and distribute schema changes.

For example:

{
  "table": "orders",
  "indexes": [
    ["customer_id"],
    ["country", "status"],
    ["created_at"]
  ],
  "version": 42
}

Each node converges toward version 42.

Now adaptive indexing becomes a distributed state-management problem.

What Happens During Failure?

Imagine the database decides to build:

idx_orders_country_status

Then the node crashes halfway through.

We need:

IndexJob {
    id: 781
    status: BUILDING
}

After restart:

BUILDING

can transition to:

FAILED

and then:

RETRY

or:

CANCELLED

Schema changes therefore need their own durable job system.

This is another reason not to implement adaptive indexing as a simple background thread.

It is infrastructure.

Security Matters

An adaptive database can modify its own schema.

That is powerful.

It is also dangerous.

A compromised query workload could theoretically manipulate the optimizer into creating expensive indexes.

Imagine an attacker deliberately generates queries such as:

WHERE random_column_1 = ?
WHERE random_column_2 = ?
WHERE random_column_3 = ?

If our database blindly trusts workload signals, it could create thousands of indexes.

This becomes an index exhaustion attack.

Therefore adaptive indexing needs:

rate limits
storage limits
schema permissions
candidate limits
audit logs
human override

Every automatic schema change should be explainable.

For example:

Created index:

idx_orders_customer_id

Reason:

Query Q17 represented 38.2% of database execution time.

Estimated latency reduction:
71ms → 5ms

Estimated storage:
180MB

Estimated write overhead:
1.7%

That's much better than:

Index magically appeared.

Observability

The adaptive engine itself needs monitoring.

Expose metrics such as:

adaptive_indexes_created_total

adaptive_indexes_dropped_total

adaptive_index_build_failures_total

index_candidate_count

index_storage_bytes

estimated_query_savings

observed_query_savings

index_write_overhead

optimization_cycle_duration

We can also expose decisions:

[21:03:02]
Detected hot query Q17

[21:03:03]
Generated candidate:
(customer_id)

[21:03:03]
Estimated benefit:
12,400 units

[21:03:04]
Estimated cost:
1,200 units

[21:03:04]
Decision:
CREATE

[21:04:19]
Index created successfully

Now the system becomes understandable.

Don't Make It Fully Autonomous on Day One

A great engineering strategy is to build levels of autonomy.

Level 1 — Observe

The database only reports:

Potential index:
(customer_id)

It produces:

Recommendation:
CREATE INDEX...

Level 3 — Simulate

It evaluates hypothetical indexes.

Level 4 — Auto-create

It automatically creates approved indexes.

Level 5 — Auto-remove

It removes indexes after confidence checks.

Level 6 — Continuous adaptation

The system continuously manages the index portfolio.

This progression makes the project much easier to test.

You don't have to build the final autonomous system immediately.

Testing the Adaptive Database

We need workloads that actually change.

For example:

Phase 1

90% customer queries
10% order queries

The system should favor:

customer_id

Then switch the workload.

Phase 2

20% customer queries
80% order status queries

Now the database should eventually adapt.

Then:

Phase 3

50% country + status
30% created_at
20% customer_id

The index portfolio should change again.

This is how we test whether the database is truly adaptive.

Benchmarking

A useful benchmark compares three configurations:

Static indexes

versus:

No indexes

versus:

Adaptive indexes

Measure:

p50 latency
p95 latency
p99 latency
throughput
CPU
memory
disk usage
write latency
index creation overhead

The important metric is not simply:

"Did adaptive indexing make reads faster?"

It is:

"Did adaptive indexing improve total workload efficiency?"

That distinction matters.

The Bigger Idea

There is a deeper lesson here.

For decades, software developers have configured databases manually.

We decide:

which indexes exist

which queries matter

which optimizations are worth the cost

But software is becoming increasingly dynamic.

Applications change.

Workloads change.

Data changes.

Traffic changes.

Therefore static infrastructure becomes increasingly interesting as a limitation.

An adaptive database asks a different question:

Instead of configuring the database once, can we build a database that continuously configures itself?

That idea extends far beyond indexes.

Imagine a database that automatically changes:

indexes
query plans
cache sizes
partition strategies
compression
replication levels
storage tiers
materialized views

Now we are no longer talking about an adaptive index system.

We are talking about an adaptive database engine.

The Ultimate Architecture

The mature version could look like this:

                         APPLICATION
                              |
                              v
                    +-------------------+
                    |    Query Engine   |
                    +---------+---------+
                              |
             +----------------+----------------+
             |                                 |
             v                                 v
      +-------------+                   +-------------+
      | Query Plans |                   | Execution   |
      | & Statistics|                   | Metrics     |
      +------+------+                   +------+------+
             |                                 |
             +----------------+----------------+
                              |
                              v
                    +-------------------+
                    | Workload Analyzer |
                    +---------+---------+
                              |
                              v
                    +-------------------+
                    | Optimization      |
                    | Controller        |
                    +---------+---------+
                              |
                 +------------+------------+
                 |            |            |
                 v            v            v
              Indexes      Caches     Partitions
                 |            |            |
                 +------------+------------+
                              |
                              v
                    +-------------------+
                    | Database Storage  |
                    +-------------------+
                              |
                              +------ feedback ------+

This is a database that does not simply execute workloads.

It responds to workloads.

Final Thoughts

Building an adaptive database that changes its indexes automatically is one of those projects that looks simple until you start pulling the thread.

At first, it sounds like:

Detect slow query.
Create index.
Done.

But that quickly becomes:

How do we fingerprint queries?

How do we measure workload impact?

How do we estimate selectivity?

How do we generate candidate indexes?

How do we evaluate composite indexes?

How do we simulate hypothetical indexes?

How do we calculate write amplification?

How do we prevent redundant indexes?

How do we safely remove indexes?

How do we prevent oscillation?

How do we handle failures?

How do we coordinate distributed nodes?

How do we prevent malicious workloads from manipulating the optimizer?

How do we prove that an index actually improved performance?

And suddenly you are designing an entire intelligent control plane around a database.

That's what makes the project fascinating.

The most interesting part isn't the SQL statement:

CREATE INDEX ...

The interesting part is the system that decides when that statement should exist at all.

A static database says:

"Tell me how you want me configured."

An adaptive database says:

"Show me how you use me, and I will continuously adapt."

That shift—from configuration to feedback—is much bigger than automatic indexing.

It is a different philosophy for building infrastructure.

Software doesn't always have to remain static after deployment.

Sometimes, the most powerful system is the one that watches what is happening, measures the consequences, learns from them, and changes itself accordingly.

And an adaptive database is a beautiful place to start.

Code is the machine. Architecture is the intelligence. Feedback is what makes the machine learn.

Building a Memory System for AI Agents

Thu, 17 Sep 2026 21:27:00 +0000

Building a Memory System for AI Agents

AI agents are becoming better at reasoning.

They can call APIs.

They can browse the web.

They can write code.

They can execute tools.

They can decompose complicated tasks into smaller tasks and sometimes recover when something goes wrong.

But there is one strangely human problem that many AI agents still struggle with:

They forget.

Not because they cannot store information.

They can store enormous amounts of information.

The problem is that storage is not memory.

A database can remember everything and still produce an agent that remembers nothing.

That distinction is one of the most interesting engineering problems emerging in modern AI systems.

If we want agents that operate over days, months, or years, we need something more sophisticated than stuffing previous conversations into a prompt.

We need memory architecture.

And building memory for an AI agent turns out to be much closer to designing a miniature cognitive system than simply adding a vector database.

The interesting question is therefore not:

“How do I give my AI more context?”

The interesting question is:

“What should this agent remember, why should it remember it, when should it retrieve it, and when should it forget it?”

That is a systems problem.

The Context Window Is Not Memory

Let's start with the most common misconception.

Suppose an agent has a conversation:

User:
My name is Daniel.

Agent:
Nice to meet you, Daniel.

User:
I'm building an e-commerce platform.

Agent:
Interesting. What technology are you using?

User:
Rust and PostgreSQL.

The model can respond intelligently because the relevant information exists inside the current context.

But tomorrow:

User:
Continue working on my project.

What project?

What language?

What architecture?

What decisions were already made?

What does the user prefer?

What problems have already been solved?

If the previous conversation isn't available, the agent starts from zero.

This is the fundamental distinction:

Context
   ↓
Information currently available to the model

Memory
   ↓
Information intentionally retained across interactions

Context is temporary working material.

Memory is persistent state.

This is not unique to AI.

A CPU has registers.

A process has memory.

A filesystem has persistent storage.

A distributed system has durable state.

A human has working memory and long-term memory.

AI agents need something similar.

Memory Is a State Management Problem

One useful way to think about agent memory is:

Agent = Model + Tools + State + Memory

The model performs reasoning.

Tools allow the model to interact with the world.

State describes what is currently happening.

Memory contains information that remains useful beyond the immediate interaction.

That gives us a basic architecture:

                  ┌───────────────┐
                  │      User     │
                  └───────┬───────┘
                          │
                          ▼
                  ┌───────────────┐
                  │  Agent Loop   │
                  └───────┬───────┘
                          │
              ┌───────────┼───────────┐
              ▼           ▼           ▼
         ┌────────┐  ┌─────────┐  ┌─────────┐
         │ Model  │  │  Tools  │  │ Memory  │
         └────────┘  └─────────┘  └─────────┘
                                      │
                         ┌────────────┼────────────┐
                         ▼            ▼            ▼
                     Semantic      Episodic     Working
                      Memory        Memory       Memory

Now the problem becomes much more interesting.

What exactly belongs inside Memory?

Memory Is Not One Thing

Human memory isn't a single bucket.

AI systems shouldn't treat it as one either.

A practical agent architecture can separate memory into several layers.

For example:

Working Memory
      ↓
Current task and immediate context

Episodic Memory
      ↓
What happened

Semantic Memory
      ↓
What is known

Procedural Memory
      ↓
How something should be done

Profile Memory
      ↓
Stable information about the user

Environmental Memory
      ↓
Information about the external world

These memories have different lifetimes and retrieval patterns.

1. Working Memory

Working memory is the easiest layer.

It contains information needed for the current task.

Imagine an agent debugging a production system.

Its working memory might contain:

{
  "task": "debug payment timeout",
  "service": "payments-api",
  "environment": "production",
  "recent_error": "gateway timeout",
  "suspected_dependency": "payment-provider"
}

This information doesn't necessarily need to survive forever.

Once the debugging session is complete, much of it can disappear.

Working memory is essentially the agent's scratchpad.

You can implement it using:

conversation state
Redis
in-process state
task databases
workflow state machines

The important thing is that it is short-lived.

2. Episodic Memory

Episodic memory records events.

Not just facts.

Events.

For example:

On September 17, the agent helped configure PostgreSQL replication.

The user rejected Redis as the primary cache.

The deployment failed because port 5432 was blocked.

The issue was resolved by updating the firewall configuration.

These are experiences.

A future agent might need them.

Imagine asking:

Why did we configure the database this way?

A semantic database might know:

PostgreSQL replication is enabled.

But episodic memory might know:

We enabled replication after the production database experienced
read-heavy workloads during a previous deployment.

That difference is powerful.

One describes what is true.

The other describes what happened.

3. Semantic Memory

Semantic memory stores knowledge.

For example:

The user's application uses PostgreSQL.

The application exposes REST APIs.

The production environment runs on Linux.

The frontend communicates with the backend using JSON.

These are facts.

Semantic memory is usually what people imagine when they hear:

“AI memory.”

But semantic memory alone is insufficient.

Suppose the agent stores:

User prefers Rust.

That sounds useful.

But what does “prefers” mean?

Was it mentioned once?

Did the user explicitly say it?

Was it inferred?

Is it still true?

Memory needs provenance.

4. Procedural Memory

Procedural memory describes how the agent should perform tasks.

For example:

When deploying this application:

1. Run tests.
2. Build the Docker image.
3. Push the image.
4. Run migrations.
5. Deploy the service.
6. Verify health checks.

This is not merely knowledge.

It is a procedure.

Procedural memory can become extremely valuable for agents operating inside organizations.

Instead of asking:

“How does this company deploy applications?”

the agent can retrieve the established procedure.

Procedural memory essentially turns previous successful workflows into reusable behavior.

5. User Profile Memory

Some information is stable enough to deserve its own category.

For example:

{
  "preferred_language": "English",
  "timezone": "Africa/Lusaka",
  "coding_style": "explicit",
  "favorite_framework": "Rust"
}

But profile memory is dangerous if implemented carelessly.

The agent shouldn't permanently store every statement.

There is a difference between:

"I like Rust."

and:

"I used Rust for this project."

The first could be a preference.

The second is contextual information.

Memory systems need to understand this distinction.

The Memory Pipeline

A memory system can therefore be modeled as a pipeline:

Conversation
     │
     ▼
Memory Candidate Detection
     │
     ▼
Extraction
     │
     ▼
Classification
     │
     ▼
Scoring
     │
     ▼
Storage
     │
     ▼
Retrieval
     │
     ▼
Ranking
     │
     ▼
Context Injection

This is the core architecture.

And notice something important:

The LLM is not the memory system.

The LLM participates in the memory system.

The actual memory system is an engineered subsystem surrounding it.

Step One: Detect Memory Candidates

Not every sentence deserves to become memory.

Consider this conversation:

User:
I'm using Python for this prototype.

Potential memory:

User is currently using Python for a prototype.

But:

User:
Python is installed on my machine.

That may not be useful later.

And:

User:
I tried Python once in 2019.

Probably shouldn't become an active preference.

Therefore, we need a memory extraction layer.

Conceptually:

def extract_memory_candidates(message):
    candidates = llm_extract(
        message,
        schema=MemoryCandidate
    )

    return [
        candidate
        for candidate in candidates
        if candidate.is_memorable
    ]

The important word here is candidate.

Extraction shouldn't immediately mean persistence.

Step Two: Give Memories Structure

Instead of storing:

User likes PostgreSQL.

store something richer.

For example:

{
  "id": "mem_83291",
  "type": "preference",
  "subject": "user",
  "predicate": "prefers",
  "object": "PostgreSQL",
  "confidence": 0.91,
  "source": "conversation",
  "created_at": "2026-09-17",
  "updated_at": "2026-09-17"
}

Now the memory is queryable.

But we can go further.

Add:

{
  "importance": 0.72,
  "recency": 1.0,
  "confidence": 0.91,
  "scope": "technical",
  "expires_at": null
}

Suddenly memory starts looking less like a collection of notes and more like a proper data model.

The Memory Record

A robust memory object might look like this:

Memory
├── id
├── type
├── content
├── embedding
├── subject
├── source
├── confidence
├── importance
├── created_at
├── updated_at
├── last_accessed_at
├── access_count
├── expiration
├── scope
├── status
└── relationships

The embedding is useful.

But it isn't the memory.

This distinction matters.

The embedding is an index representation.

The actual memory is the structured record.

Why Vector Databases Are Not Enough

Modern AI applications frequently use vector databases.

That's useful.

You can embed:

"The user prefers Rust."

and later perform semantic similarity search.

But consider:

The user prefers Rust.

versus:

The user used Rust once.

They may have similar embeddings.

But they represent completely different knowledge.

Vector similarity answers:

“What information is semantically similar?”

It does not necessarily answer:

“What information is true, relevant, current, important, authorized, or still valid?”

That's why serious memory systems often combine:

Vector Search
+
Structured Metadata
+
Temporal Reasoning
+
Keyword Search
+
Relationship Graphs
+
Rules

Memory retrieval becomes a hybrid search problem.

Hybrid Retrieval

Suppose the user asks:

Why did we choose PostgreSQL for this project?

A vector search might return:

PostgreSQL is a relational database.

Technically relevant.

But not necessarily useful.

A better retrieval system might combine:

Semantic similarity

How closely does the memory relate to the question?

Recency

How recent is it?

Importance

How significant is it?

Frequency

How often has it been accessed?

Confidence

How reliable is the information?

Scope

Does it apply to this project?

Temporal validity

Is it still true?

Then we can define:

score =
    α * semantic_similarity
  + β * recency
  + γ * importance
  + δ * confidence
  + ε * scope_match

This is where memory engineering starts becoming interesting.

We're no longer simply retrieving documents.

We're ranking pieces of an agent's past.

Time Changes Meaning

Memory has a temporal dimension.

Consider:

The user uses React.

Maybe that was true in 2023.

In 2026:

The user migrated to Vue.

Now we have conflicting memories.

A naive vector database may return both.

A proper memory system should reason about time.

Represent memories as:

Memory A
valid_from = 2023
valid_until = 2026

Memory B
valid_from = 2026
valid_until = null

Now retrieval can prioritize currently valid information.

This is a simple temporal database idea applied to AI memory.

And it solves an enormous class of problems.

Memory Contradictions

Agents will inevitably receive contradictory information.

Imagine:

User:
I don't use TypeScript.

Six months later:

User:
I'm building everything in TypeScript now.

If the system simply appends memories:

User does not use TypeScript.

User uses TypeScript.

the agent has created ambiguity.

Instead, the new memory should trigger reconciliation.

Conceptually:

def update_memory(new_memory):
    conflicts = find_related_memories(new_memory)

    for old_memory in conflicts:
        if contradicts(new_memory, old_memory):
            supersede(old_memory, new_memory)

    store(new_memory)

This creates a temporal chain:

Old belief
    ↓
Superseded
    ↓
New belief

Memory becomes versioned knowledge.

Memory Consolidation

Human memory isn't simply a log of everything that happened.

Information gets compressed.

Patterns become knowledge.

Individual experiences become general principles.

AI agents need the same mechanism.

Suppose the agent has these episodes:

User asked for short answers.

User asked for short answers.

User asked for short answers.

User requested concise explanations.

User complained when responses were excessively long.

Instead of keeping every event forever, the system might consolidate them into:

User prefers concise explanations.

This is memory consolidation.

The architecture becomes:

Episodes
   │
   ▼
Pattern Detection
   │
   ▼
Knowledge Extraction
   │
   ▼
Semantic Memory

That is much more powerful than simply storing transcripts.

Memory Decay

Another interesting concept is forgetting.

We often design databases around permanence.

But intelligent memory requires controlled forgetting.

Not everything remains relevant forever.

We can model memory strength:

strength =
    importance
    × confidence
    × usage
    × recency_decay

For example:

def memory_strength(memory, now):
    age = now - memory.last_accessed_at

    decay = exp(-age / memory.decay_rate)

    return (
        memory.importance
        * memory.confidence
        * memory.access_weight
        * decay
    )

Old memories aren't necessarily deleted.

They simply become less likely to surface.

This distinction matters.

A memory that hasn't been used for two years may still be useful if the user suddenly asks about the old project.

Forgetting Should Be Intentional

There are at least three different operations:

Ignore
Archive
Delete

They are not the same.

Ignore

The memory still exists but isn't relevant.

Delete

The memory is actually removed according to the system's deletion policy.

A production-grade memory system should make these states explicit.

Privacy Is Part of Memory Architecture

The moment an agent stores information about a person, memory becomes a privacy problem.

A good memory system should answer:

Who created this memory?

Why was it stored?

Where did it come from?

Who can access it?

How long should it live?

Can it be corrected?

Can it be deleted?

What other memories depend on it?

This means memory records should include provenance.

For example:

{
  "content": "User prefers concise responses",
  "source": {
    "conversation_id": "conv_129",
    "message_id": "msg_441"
  }
}

Now the system can trace the memory back to its origin.

That's not just useful for debugging.

It's fundamental to trust.

Memory and Authorization

Not every memory should be available to every agent.

Imagine an organization with:

Personal Assistant
Engineering Agent
Finance Agent
HR Agent
Customer Support Agent

They might share some memory.

But not all memory.

Therefore:

Memory
   │
   ├── User scope
   ├── Project scope
   ├── Organization scope
   ├── Agent scope
   └── Restricted scope

A finance agent shouldn't automatically retrieve sensitive engineering information.

Memory therefore becomes an authorization problem.

The retrieval query should include identity and scope:

retrieve(
    query=query,
    user_id=user_id,
    agent_id=agent_id,
    project_id=project_id,
    permissions=permissions
)

Memory is data.

Data needs access control.

The Memory Graph

Some memories are related.

Consider:

User
 │
 ├── works_on → Project A
 │                  │
 │                  ├── uses → Rust
 │                  ├── uses → PostgreSQL
 │                  └── deployed_to → AWS
 │
 └── prefers → concise explanations

This is a graph.

Graph relationships allow agents to traverse concepts.

For example:

Project A
   ↓
PostgreSQL
   ↓
Previous deployment
   ↓
Database migration failure

That may be more useful than retrieving isolated chunks of text.

This suggests a hybrid architecture:

              Memory System
                    │
       ┌────────────┼────────────┐
       ▼            ▼            ▼
    Vector        SQL          Graph
    Index       Metadata      Relations

Each system solves a different part of the problem.

Memory Retrieval Is a Reasoning Problem

Now we reach the most interesting part.

When should the agent retrieve memory?

Not every prompt requires it.

If the user asks:

What is 17 × 24?

retrieving their entire project history is pointless.

But:

Continue the API architecture we designed yesterday.

requires memory.

Therefore, retrieval itself should be intelligent.

We can introduce a memory gate:

def should_retrieve_memory(query):
    return classifier(query).requires_memory

But even this can be improved.

The agent can classify memory intent:

NO_MEMORY
USER_PROFILE
PROJECT_CONTEXT
PAST_EVENT
PREFERENCE
PROCEDURE
FACT
RELATIONSHIP

Then retrieve only the relevant memory class.

Memory Should Be Contextual

Imagine the user says:

Fix the authentication bug.

The memory system should know:

Which project?

Which repository?

Which authentication system?

What happened previously?

What fixes were attempted?

Memory retrieval should therefore depend on current context.

Something like:

context = {
    "user": user_id,
    "project": project_id,
    "task": current_task,
    "conversation": conversation_id
}

memories = retrieve(query, context)

This prevents irrelevant memories from flooding the model.

The Context Budget Problem

Even if an agent has a million memories, the model cannot necessarily consume all of them.

Context is finite.

So memory retrieval becomes an optimization problem:

Millions of memories
        ↓
Candidate retrieval
        ↓
Re-ranking
        ↓
Compression
        ↓
Top relevant memories
        ↓
Model context

Suppose the system retrieves 50 memories.

Don't blindly insert all 50.

Instead:

50 candidates
   ↓
re-rank
   ↓
12 useful memories
   ↓
summarize
   ↓
4 compact memory units

This is where memory and context engineering meet.

Memory Compression

Suppose an agent has 100 conversations about the same project.

Sending all 100 conversations to the model is ridiculous.

Instead, the system can maintain a project summary:

Project: Commerce API

Architecture:
- Rust backend
- PostgreSQL
- Redis cache
- REST API

Authentication:
- JWT
- Refresh tokens
- Role-based authorization

Important decisions:
- PostgreSQL selected for transactional consistency.
- Redis is used only as a cache.
- Authentication remains inside the API gateway.

Known issues:
- Rate limiting still needs redesign.

This summary is a form of compressed memory.

But compression introduces another problem:

Information loss.

A good memory system should preserve the original episodes while maintaining compressed representations.

Raw Events
    ↓
Summaries
    ↓
Long-Term Knowledge

If the summary is wrong, the system should be able to trace back to the source.

Memory Hierarchy

A powerful architecture therefore looks like this:

             ┌──────────────────┐
             │ Working Memory   │
             └────────┬─────────┘
                      │
             ┌────────▼─────────┐
             │ Recent Episodes  │
             └────────┬─────────┘
                      │
             ┌────────▼─────────┐
             │ Consolidated     │
             │ Knowledge        │
             └────────┬─────────┘
                      │
             ┌────────▼─────────┐
             │ Archived Memory  │
             └──────────────────┘

This resembles a storage hierarchy.

Fast memory.

Recent memory.

Long-term memory.

Cold storage.

Different access costs.

Different retrieval policies.

Building the Storage Layer

A practical implementation might use PostgreSQL for durable structured memory.

Schema:

CREATE TABLE memories (
    id UUID PRIMARY KEY,
    user_id UUID,
    project_id UUID,
    type VARCHAR(50),
    content TEXT NOT NULL,
    confidence FLOAT,
    importance FLOAT,
    created_at TIMESTAMP,
    updated_at TIMESTAMP,
    last_accessed_at TIMESTAMP,
    expires_at TIMESTAMP,
    status VARCHAR(20)
);

Then add vector search using an appropriate vector extension or vector database.

You might have:

CREATE TABLE memory_embeddings (
    memory_id UUID PRIMARY KEY,
    embedding VECTOR(...)
);

And relationship storage:

CREATE TABLE memory_relations (
    source_id UUID,
    target_id UUID,
    relation VARCHAR(50)
);

Now we have:

SQL
 ↓
Metadata + durability

Vector index
 ↓
Semantic retrieval

Relations
 ↓
Graph reasoning

The Memory API

We can expose memory through a simple interface.

class MemoryStore:

    def remember(self, memory):
        ...

    def search(self, query, context):
        ...

    def update(self, memory_id, changes):
        ...

    def forget(self, memory_id):
        ...

    def consolidate(self, scope):
        ...

The agent doesn't need to know how memory is stored.

That's an important architectural principle.

The agent interacts with an abstraction.

The memory subsystem owns:

storage
indexing
ranking
expiration
consolidation
conflict resolution
authorization

This makes the system replaceable.

The Agent Loop

Now put everything together.

A production agent loop might look like:

while True:

    message = receive_input()

    context = load_working_context(message)

    candidates = memory.retrieve(
        query=message,
        context=context
    )

    memories = memory.rank(candidates)

    prompt = build_prompt(
        message=message,
        context=context,
        memories=memories
    )

    response = model.generate(prompt)

    actions = extract_tool_calls(response)

    execute(actions)

    candidates = extract_memory_candidates(
        message,
        response,
        actions
    )

    memory.process(candidates)

    update_working_state()

This is the skeleton of a persistent agent.

Memory Formation Should Happen After Action

There is an interesting design principle here.

Sometimes an agent shouldn't store a memory merely because the user said something.

It should wait until the information has been validated.

Imagine:

User:
I think the production database is PostgreSQL.

That's uncertain.

The agent checks the deployment configuration.

It discovers:

Database = PostgreSQL

Now the agent has stronger evidence.

Memory formation can therefore incorporate tool observations.

Conversation
     ↓
Hypothesis
     ↓
Tool verification
     ↓
Confirmed fact
     ↓
Memory

This makes agent memory more trustworthy.

Memory From Tool Use

Agents don't only learn from conversations.

They interact with systems.

An agent might discover:

Repository uses Rust.

Production runs Kubernetes.

Service has three replicas.

Redis is unavailable in staging.

Deployment succeeded at 14:32.

These are memories derived from the environment.

Therefore every tool execution can produce memory candidates.

Tool Call
   ↓
Observation
   ↓
Memory Candidate
   ↓
Validation
   ↓
Persistent Memory

Now the agent develops an internal model of its environment.

The World Model

This leads to something bigger.

A sufficiently sophisticated memory system becomes a world model.

The agent begins to understand:

Who am I interacting with?

What projects exist?

What systems exist?

What happened before?

What decisions were made?

What constraints exist?

What has changed?

What usually works?

What usually fails?

At that point, memory isn't just a feature.

It becomes infrastructure.

The model provides reasoning.

Memory provides continuity.

Tools provide interaction.

Together they form something much closer to an autonomous software system.

Memory Is an Interface Between Time and Intelligence

This might be the deepest way to think about the problem.

A language model is extremely capable inside a moment.

Memory connects that capability across time.

Without memory:

t1 → intelligent
t2 → intelligent
t3 → intelligent

But each moment is disconnected.

With memory:

t1 ───────► t2 ───────► t3
 │           │           │
 └───────────┴───────────┴── Memory

The agent develops continuity.

And continuity is what makes long-running agents fundamentally different from chatbots.

Failure Modes

Of course, memory systems can fail spectacularly.

Failure 1: Remembering everything

Result:

Huge memory
Low signal
Poor retrieval
Context pollution

Failure 2: Remembering too little

Result:

Agent feels forgetful.

Failure 3: Stale memories

Result:

Agent confidently uses outdated information.

Failure 4: Contradictory memories

Result:

Agent retrieves mutually incompatible facts.

Failure 5: Incorrect extraction

Result:

Agent stores an inference as a fact.

Failure 6: Retrieval failure

The memory exists but never gets retrieved.

This is especially important.

A memory system can have perfect storage and still appear broken because retrieval is poor.

Measure Memory Like a Real System

We need metrics.

Not just:

Does it work?

We should measure:

Recall

Did the system retrieve the memory when needed?

Precision

Were retrieved memories actually relevant?

Freshness

How often did the system use stale information?

Contradiction rate

How frequently did conflicting memories reach the model?

Memory write rate

How many memories are created per interaction?

Memory utility

How often does retrieved memory materially improve the result?

Retrieval latency

How quickly can the system retrieve relevant memories?

Context cost

How many tokens are consumed by memory?

These metrics transform memory from an abstract AI feature into an engineering subsystem.

The Strange Economics of Memory

Memory also has a cost.

Every memory requires:

Storage
Embedding
Indexing
Retrieval
Ranking
Validation
Maintenance

And every retrieved memory consumes context.

Therefore the optimal memory system isn't the one that remembers the most.

It is the one that produces the greatest useful information per unit of memory cost.

You can think about:

Memory Utility =
Useful Information
------------------
Storage + Retrieval + Context Cost

That's an engineering optimization problem.

Memory Should Be Treated as Data, Not Magic

One of the biggest mistakes in AI engineering is treating memory as mysterious.

It isn't.

Memory can be modeled.

It can be indexed.

It can be versioned.

It can be authorized.

It can be compressed.

It can be tested.

It can be monitored.

It can be deleted.

It can be migrated.

It can be backed up.

It can have schemas.

It can have APIs.

In other words:

Memory is software.

And once you realize that, the architecture becomes much clearer.

A More Advanced Architecture

A serious production memory system might eventually look like this:

                         ┌──────────────┐
                         │     User     │
                         └──────┬───────┘
                                │
                                ▼
                       ┌─────────────────┐
                       │   Agent Runtime │
                       └────────┬────────┘
                                │
                ┌───────────────┼────────────────┐
                │               │                │
                ▼               ▼                ▼
         ┌────────────┐  ┌────────────┐  ┌────────────┐
         │   Model    │  │   Tools    │  │  Working   │
         │            │  │            │  │  Memory    │
         └────────────┘  └────────────┘  └─────┬──────┘
                                               │
                                               ▼
                                      ┌─────────────────┐
                                      │ Memory Gateway  │
                                      └────────┬────────┘
                                               │
                      ┌────────────────────────┼────────────────────┐
                      │                        │                    │
                      ▼                        ▼                    ▼
               ┌─────────────┐        ┌─────────────┐       ┌─────────────┐
               │   Semantic  │        │   Episodic  │       │ Procedural  │
               │   Memory    │        │   Memory    │       │   Memory    │
               └──────┬──────┘        └──────┬──────┘       └──────┬──────┘
                      │                      │                     │
                      └──────────────────────┼─────────────────────┘
                                             │
                                             ▼
                                   ┌──────────────────┐
                                   │ Retrieval Engine │
                                   └────────┬─────────┘
                                            │
                               ┌────────────┼────────────┐
                               ▼            ▼            ▼
                           Vector        SQL          Graph
                           Search       Filters      Relations
                               │            │            │
                               └────────────┼────────────┘
                                            ▼
                                     Re-ranking
                                            │
                                            ▼
                                     Compression
                                            │
                                            ▼
                                    Context Builder
                                            │
                                            ▼
                                          Model

This architecture is much closer to what long-lived agents will require.

The Future: Agents That Remember Their Own History

Imagine an AI coding agent working on a project for two years.

It remembers:

Why the architecture was designed this way.

Which technologies were rejected.

Which bugs occurred previously.

Which deployment procedures are reliable.

Which services depend on which.

Which experiments failed.

Which solutions worked.

What the team prefers.

What technical debt exists.

Now imagine the agent onboarding a new developer.

The developer asks:

Why does this service use this strange caching strategy?

The agent doesn't merely inspect the code.

It retrieves the history.

Three months ago, the team introduced this cache
after database load increased during a traffic spike.
A previous implementation caused stale inventory data,
so the current design invalidates the cache after writes.

That is a different kind of software intelligence.

The code tells you what exists.

Memory tells you why it exists.

And “why” is one of the most valuable pieces of information in engineering.

The Real Challenge Isn't Storage

The industry has become very good at storing information.

We have:

SQL databases
NoSQL databases
Object storage
Vector databases
Search engines
Graphs
Caches
Logs
Data warehouses

The hard problem isn't:

“Where do we put the memory?”

The hard problem is:

“What deserves to become memory?”

Then:

“How do we know whether it is true?”

Then:

“When should it be retrieved?”

Then:

“How should conflicting memories be resolved?”

Then:

“When should memory decay?”

Then:

“Who is allowed to see it?”

And finally:

“How does memory change the agent's behavior?”

Those questions are architecture.

Memory Turns Agents Into Systems

A stateless LLM call is simple.

Input → Model → Output

An agent with memory is different:

Input
  ↓
State
  ↓
Memory Retrieval
  ↓
Reasoning
  ↓
Tool Execution
  ↓
Observation
  ↓
Memory Formation
  ↓
State Update
  ↓
Next Action

Now we have a feedback loop.

And feedback loops are where complex systems emerge.

The agent is no longer simply generating text.

It is maintaining state over time.

That makes memory one of the foundational architectural layers of serious agent systems.

Final Architecture

If I were designing a memory system from scratch, I would start with five principles.

1. Separate context from memory

Context is temporary.

Memory is persistent.

Do not confuse the two.

2. Separate memory types

Use different representations for:

working state
events
facts
preferences
procedures
relationships

3. Use hybrid retrieval

Combine:

semantic search
metadata filters
temporal logic
keyword search
relationships

No single retrieval mechanism is sufficient.

4. Make memory versioned

Facts change.

Preferences change.

Projects change.

The system must understand:

what was true
when it was true
what replaced it

5. Make memory controllable

Memory should have:

provenance
permissions
expiration
correction
archival
deletion

Because intelligent memory without control quickly becomes technical debt.

The Bigger Idea

We spent the first era of AI teaching machines how to generate.

Then we taught them how to reason.

Then we taught them how to use tools.

The next challenge is teaching them how to continue.

Because intelligence without continuity is strangely incomplete.

An agent can solve a problem today.

But a truly useful agent should remember what happened yesterday.

It should understand what changed this morning.

It should know why a decision was made three months ago.

It should recognize that an old assumption is no longer true.

It should learn from failures.

It should consolidate experiences.

It should retrieve the right knowledge at the right moment.

And, perhaps most importantly, it should know that not everything deserves to be remembered.

That is the difference between a database and memory.

A database asks:

“What did we store?”

A memory system asks:

“What matters now?”

That question is much harder.

It requires retrieval.

It requires time.

It requires structure.

It requires reasoning.

It requires forgetting.

And it requires architecture.

The future of AI agents may therefore depend less on making models remember everything and more on building systems that understand what remembering actually means.

Because the most intelligent agent isn't necessarily the one with the largest context window.

It may be the one that knows exactly which piece of its past matters right now.

When Everyone Can Generate Code, Architecture Becomes the Moat

Thu, 17 Sep 2026 21:16:29 +0000

There was a time when software development began with an empty file.

You had an idea.

You opened your editor.

You created a project.

You wrote the first function.

Then another.

You searched documentation.

You read Stack Overflow.

You fought dependency errors.

You stared at a stack trace at 2:00 AM.

Eventually, after enough persistence, you had something that worked.

For decades, one of the most valuable skills in technology was the ability to transform an idea into functioning code.

That skill is not disappearing.

It is becoming amplified.

Artificial intelligence can now generate surprisingly sophisticated software from natural-language instructions. It can produce APIs, interfaces, database models, tests, documentation, integrations, infrastructure configuration, and entire application foundations.

And the quality is getting better.

That matters.

Because when machines become extremely good at producing software, the economics of software development change.

The bottleneck moves.

Code becomes easier to create.

Implementation becomes faster.

Iteration becomes cheaper.

Small teams become capable of building things that once required much larger teams.

And when implementation becomes increasingly accessible, another layer of engineering becomes disproportionately important.

Architecture.

Not because AI cannot write good code.

AI can write good code.

Not because generated software is inherently inferior.

It isn't.

The interesting question is what happens when everyone has access to increasingly powerful software construction capabilities.

When everyone can generate good code, what separates one system from another?

The answer increasingly becomes:

The architecture.

The Code Factory Has Arrived

Software development is entering an era where natural language can become a programming interface.

You can describe a feature:

Build a REST API for managing inventory.

And receive an implementation.

You can describe:

Add authentication, role-based authorization, PostgreSQL persistence, validation, rate limiting, and automated tests.

And receive a substantial implementation.

You can describe:

Create a responsive dashboard with analytics, charts, filtering, pagination, and user management.

And receive a working frontend.

The machine doesn't merely autocomplete a line anymore.

It can participate in the construction of entire software systems.

That is extraordinary.

And we should treat it as such.

AI-assisted software development is not simply another developer productivity trick.

It represents a change in the relationship between human intention and computational implementation.

The distance between:

“I want this software to exist.”

and:

“This software exists.”

is becoming dramatically smaller.

That is one of the most important developments in the history of programming.

But something interesting happens when the implementation layer becomes dramatically more powerful.

The value of the layer above it increases.

When Implementation Becomes Abundant

Imagine a world where thousands of developers can access AI systems capable of generating high-quality software.

The same programming languages are available.

The same frameworks are available.

The same databases are available.

The same cloud platforms are available.

The same AI models are available.

The same libraries are available.

The same development tools are available.

The ability to produce good code becomes increasingly democratized.

This is not bad news for engineers.

It is potentially incredible news.

It means more people can build.

More experiments can happen.

More businesses can launch.

More ideas can become products.

A person with a laptop and an idea can potentially create something that previously required a team of specialists.

The world gets more software.

But then a deeper question emerges.

If everyone can build good software, what makes one software system structurally exceptional?

That is where architecture enters the conversation.

Architecture Is the Layer Above Implementation

A useful way to visualize modern software development is:

             HUMAN INTENT
                   │
                   ▼
             DOMAIN MODEL
                   │
                   ▼
              ARCHITECTURE
                   │
          ┌────────┼────────┐
          ▼        ▼        ▼
        DATA      APIs    WORKFLOWS
          │        │        │
          └────────┼────────┘
                   ▼
              AI + CODE
                   │
                   ▼
              APPLICATION

AI can dramatically improve the bottom portion.

It can turn architectural decisions into implementation.

But architecture determines the shape of what gets implemented.

It determines:

what components exist,
what they own,
how they communicate,
where data lives,
how state changes,
where boundaries exist,
what can fail independently,
what must remain consistent,
how the system evolves,
and what kinds of products can eventually be built on top of it.

Architecture is therefore not a replacement for AI-generated code.

It is the structure that gives that code a place in the system.

AI Makes the Engineer More Powerful

There is an important distinction here.

The future isn't necessarily about AI replacing programmers.

It can be about programmers becoming dramatically more capable.

A developer who previously spent an entire day implementing a feature can potentially spend far less time on the mechanical construction and more time thinking about the system itself.

Instead of spending hours writing repetitive infrastructure, they can focus on:

modeling the domain,
designing boundaries,
analyzing failure modes,
improving performance,
understanding customers,
designing APIs,
refining data models,
experimenting with product ideas,
and making architectural decisions.

AI becomes the construction engine.

The engineer becomes the system designer.

This is powerful because architecture operates at a higher level of leverage.

A good architectural decision can influence thousands or millions of lines of code.

One Decision Can Control a Thousand Implementations

Suppose you decide that every financial operation must be represented as an immutable ledger entry.

That single architectural decision affects:

database design,
transaction processing,
APIs,
reporting,
reconciliation,
auditing,
refunds,
fraud detection,
analytics,
customer support.

One principle creates hundreds of implementation consequences.

That is leverage.

Now imagine AI generating the implementation of each consequence.

The architecture becomes the high-level instruction set.

The machine handles much of the translation.

This is why architecture becomes more valuable as AI becomes better.

The better the implementation engine becomes, the more valuable the blueprint becomes.

Architecture Is a Compression of Knowledge

Great architecture compresses enormous amounts of reasoning into a relatively small number of decisions.

Consider a system with:

Identity
Orders
Inventory
Payments
Shipping
Notifications
Analytics

That diagram looks simple.

But behind it are hundreds of questions.

Who owns customer identity?

Who owns order state?

Who can modify inventory?

Does payment own the financial transaction?

Can shipping change an order?

What happens when inventory disappears after an order is created?

What happens when payment succeeds but fulfillment fails?

What happens when a request is retried?

What happens when two requests arrive simultaneously?

What data is authoritative?

What data is derived?

Which events are permanent?

Which operations can be asynchronous?

These questions form architecture.

Architecture is therefore a compressed representation of decisions about how the software world behaves.

And that knowledge becomes increasingly valuable when implementation itself becomes easier.

Architecture Is the Theory of the System

Every serious software system contains a theory.

A theory of users.

A theory of data.

A theory of state.

A theory of time.

A theory of ownership.

A theory of failure.

A theory of relationships.

A theory of trust.

Consider an e-commerce platform.

At first glance:

User → Product → Cart → Order → Payment

Looks straightforward.

But underneath it lies a theory of commerce.

What is an order?

When does ownership transfer?

When does inventory leave available stock?

What does “paid” mean?

What happens after a refund?

What happens when an order is partially fulfilled?

What happens when a customer cancels?

Architecture transforms those business concepts into computational structures.

That is why architecture is so much more than drawing boxes.

It is the formalization of how a system understands reality.

Good Code Needs a Great Place to Live

There is no need to create a false conflict between code and architecture.

Good code matters.

Excellent code matters.

AI-generated code can be excellent.

The question is not whether code is important.

The question is where its value compounds.

Imagine two teams with equally powerful AI coding systems.

Both can generate high-quality implementations.

Team A has a clear domain model and strong architecture.

Team B has no coherent structural model.

Both teams can produce good code.

But Team A's code fits into a deliberate system.

Every service has a purpose.

Every database has ownership.

Every API has a contract.

Every event has meaning.

Every boundary has a reason.

The quality of individual code is only one component of the equation.

The architecture determines how those components interact.

A beautiful brick is useful.

A great building is something else.

The Building Analogy

Imagine giving two architects unlimited access to the same construction robots.

The robots can produce excellent bricks.

They can install windows.

They can construct walls.

They can place doors.

They can build floors.

They can even execute detailed instructions with extraordinary precision.

The robots become extremely good at construction.

Does that make architectural design irrelevant?

Quite the opposite.

The better the construction machinery becomes, the more powerful the architectural plan becomes.

Because now the plan can be translated into physical reality faster.

Software is moving in a similar direction.

AI is becoming an increasingly capable construction mechanism.

Architecture becomes the design of the machine being constructed.

Architecture Determines What Can Change

One of the greatest properties of good architecture is controlled change.

Software is never finished.

The first version is not the final version.

A product begins with:

10 users

Then:

1,000 users

Then:

100,000 users

Then perhaps:

10,000,000 users

Requirements change.

Markets change.

Customers change.

Technology changes.

Companies change.

The architecture has to absorb those changes.

Good architecture doesn't prevent change.

It makes change possible.

This is one of its greatest powers.

Architecture Creates Optionality

Suppose your payment system is designed so that the rest of the application does not depend directly on one payment provider.

You can potentially change providers without rebuilding the entire business.

That architectural boundary creates optionality.

The same principle can apply to:

databases,
cloud providers,
AI models,
messaging systems,
authentication providers,
storage systems,
frontend frameworks,
external APIs.

This doesn't mean everything should be abstracted.

Overengineering creates its own problems.

The architectural skill is knowing where optionality has economic value.

That is a strategic decision.

Architecture Is Where Technical Strategy Meets Business Strategy

Architecture is often described as a technical concern.

But many architectural decisions are really business decisions expressed in technical form.

Consider multi-tenancy.

If your architecture supports thousands of customers within one platform, your economics are different from a system that requires a separate deployment for every customer.

Consider APIs.

If your APIs are designed as stable public contracts, your software can potentially become a platform.

Consider events.

If your system exposes meaningful domain events, other systems can build around your platform.

Consider modularity.

If different parts of the business can evolve independently, your organization can move faster.

Architecture determines possibilities.

Possibilities determine strategy.

Strategy determines business outcomes.

That is why great architecture deserves to be treated as strategic infrastructure.

The Database Is Architecture

Developers sometimes think of architecture as application-level structure.

But one of the most important architectural decisions is the data model.

Your database isn't just storage.

It is a representation of reality.

Suppose your application contains:

Customer
Order
Product
Payment
Shipment

Those tables encode relationships.

They encode ownership.

They encode assumptions.

They encode history.

As the system grows, the database becomes a historical record of how the business works.

And changing it can become expensive.

This means data architecture can become part of the moat.

Not because databases are impossible to copy.

But because mature data models accumulate organizational knowledge.

APIs Are Architecture Made Visible

An API is another architectural boundary.

Consider:

POST /orders

That looks simple.

But what does it actually mean?

Does it create an order immediately?

Does it reserve inventory?

Does it initiate payment?

Is it idempotent?

What happens if the request times out?

Can the client retry?

What happens if payment succeeds but the response never reaches the client?

What events are emitted?

Who owns the resulting state?

The endpoint is only the visible surface.

The architecture is everything underneath it.

This is why excellent API design is not simply about choosing endpoint names.

It is about designing contracts between systems.

AI Turns Architecture Into an Instruction Language

This is perhaps one of the most exciting developments.

AI systems can work from specifications.

That means architecture can increasingly become an instruction language for software generation.

Imagine defining:

Identity owns authentication.

Orders own order state.

Payments own payment state.

Inventory owns stock availability.

No service may directly mutate another service's database.

Financial transactions are immutable.

External operations must be idempotent.

Domain events are versioned.

Public APIs maintain backward compatibility.

This is no longer just documentation.

It can become a source of truth from which implementation is generated.

Architecture starts moving toward something closer to an executable specification.

The architecture tells the AI:

“This is how this world works.”

The AI responds:

“Then here is the software that implements that world.”

That relationship could become one of the defining patterns of AI-native software engineering.

Architecture Becomes the Prompt Before the Prompt

A traditional AI coding interaction might look like:

Build a user registration endpoint.

A stronger interaction is:

Identity is a bounded domain. Password credentials are owned by Identity. Registration creates a user identity but does not create an authenticated session. Authentication tokens are issued by the identity subsystem. User profile data is separate from authentication credentials.

Now the AI isn't simply generating code.

It is implementing architectural intent.

The architecture becomes the context.

The prompt becomes smaller because the system model becomes larger.

This is an important shift.

The future may involve less prompting for individual functions and more defining the rules of the computational world in which those functions exist.

The Moat Is the System of Decisions

A competitor can use the same AI model.

They can use the same programming language.

They can use the same framework.

They can use the same database.

They can even generate an application with similar features.

But they don't automatically possess:

your domain model,
your customer knowledge,
your historical data,
your workflows,
your architectural decisions,
your operational experience,
your integrations,
your ecosystem,
your accumulated lessons.

Those things form a deeper layer.

The moat isn't necessarily the source code.

It is the system of knowledge encoded around the source code.

And architecture is one of the strongest forms of that knowledge.

The Best Architecture Is Not the Most Complicated

There is a danger in talking about architecture as a moat.

People may assume:

More architecture = better architecture.

Not true.

A great architecture can be incredibly simple.

For a small application:

Frontend
   ↓
Backend
   ↓
PostgreSQL

may be fantastic.

There is no prize for introducing seventeen microservices into an application with fifty users.

Architecture should reflect the problem.

Sometimes the greatest architectural decision is refusing to introduce unnecessary complexity.

Simplicity is architecture too.

A system can be sophisticated without being complicated.

That distinction matters.

Architecture Is About Boundaries, Not Buzzwords

Microservices are not architecture.

Kubernetes is not architecture.

Kafka is not architecture.

PostgreSQL is not architecture.

React is not architecture.

Rust is not architecture.

They are technologies.

Architecture is the reasoning about how those technologies participate in a system.

You can have microservices with terrible architecture.

You can have a monolith with brilliant architecture.

You can have Kubernetes and no coherent system design.

You can have one server and an exceptionally elegant architecture.

The technology stack is the vocabulary.

Architecture is the sentence.

The Future Developer Will Operate at a Higher Abstraction Level

AI changes what developers can spend their time doing.

A developer might previously have spent hours implementing:

CRUD
Validation
Authentication
Tests
Documentation
Boilerplate

AI can increasingly assist with these tasks.

That creates space.

What happens with that space?

The strongest engineers can move upward.

They can think more about:

Business
   ↓
Domain
   ↓
Architecture
   ↓
Systems
   ↓
Implementation

The developer becomes less of a code typist and more of a systems thinker.

This is not the end of programming.

It is programming becoming more abstract.

And abstraction has always been one of the great forces in computer science.

Architecture Is a Form of Leverage

One engineer can write thousands of lines of code.

But one architectural decision can influence millions.

Consider the decision:

All external events must be idempotent.

That single rule can shape:

APIs,
queues,
workers,
databases,
payment processing,
retry mechanisms,
observability.

Or:

Every domain owns its own state.

That changes:

database access,
service boundaries,
APIs,
events,
organizational ownership.

Architecture multiplies the impact of decisions.

That is why it becomes more valuable as implementation accelerates.

When AI increases the amount of implementation that can happen per unit of time, architectural leverage increases with it.

The New Software Development Loop

The classic development loop was:

Idea
 ↓
Code
 ↓
Test
 ↓
Deploy

The AI-native loop increasingly looks like:

Idea
 ↓
Domain Model
 ↓
Architecture
 ↓
Specification
 ↓
AI-Assisted Implementation
 ↓
Verification
 ↓
Production
 ↓
Telemetry
 ↓
Learning
 ↓
Architecture Evolution

Architecture becomes the central organizing layer.

It connects intention with implementation.

And production feedback eventually flows back into architecture.

The system learns.

The architecture evolves.

The implementation follows.

This is a much more powerful development loop.

Architecture Can Become a Company's Institutional Memory

Imagine a company that has existed for ten years.

It has experienced:

scaling events,
outages,
migrations,
security incidents,
changing customer behavior,
pricing changes,
product launches,
failed experiments,
infrastructure failures,
acquisitions,
regulatory changes.

All of those experiences influence architecture.

Eventually the architecture becomes a map of what the organization has learned.

That is extremely valuable.

A new developer can join the company and inherit architectural decisions that were produced by years of experience.

AI can help explain those decisions.

AI can help implement them.

AI can help test them.

But the organization still benefits from having the accumulated model.

That is institutional memory transformed into software structure.

Architecture Is the Moat Because Code Is Becoming Easier to Replicate

This is the core economic argument.

If software implementation becomes dramatically easier, then the implementation itself can become less differentiated.

Two companies may be able to produce comparable feature sets.

The difference becomes structural.

Which company has the better:

domain model?
data model?
API design?
ecosystem?
workflows?
infrastructure?
feedback loop?
customer understanding?
platform architecture?

This doesn't mean code has no value.

It means the competitive advantage moves upward.

The software industry has always moved toward higher levels of abstraction.

Machine code.

Assembly.

High-level languages.

Frameworks.

Cloud platforms.

Managed infrastructure.

Now AI-generated implementation.

Every abstraction makes the layer below it easier.

And every time that happens, the layer above it becomes more important.

From Programmer to System Designer

The most interesting evolution may therefore not be:

programmer → obsolete

but:

programmer → amplified systems designer.

The engineer still understands code.

They still understand databases.

They still understand networks.

They still understand algorithms.

But they increasingly operate at a higher level.

They think about:

What world are we modeling?

What are the fundamental entities?

Who owns them?

How does state change?

What are the invariants?

What can happen concurrently?

What happens during failure?

Which boundaries should exist?

Which boundaries should disappear?

What should be stable?

What should remain replaceable?

What architecture allows the business to evolve?

Those are architectural questions.

And they are becoming more valuable.

The Architecture Moat Is Not About Hiding Complexity

A great moat is not:

“Our architecture is so complicated nobody can understand it.”

That is not a moat.

That is a prison.

The strongest architectural advantage is often the opposite.

It is:

“We understand the system so well that we can make it simpler.”

That is difficult to copy.

Because simplicity produced by understanding is different from simplicity produced by lack of features.

A great architecture hides unnecessary complexity from everyone who doesn't need to know it.

The customer sees:

Buy

The system may perform:

Authentication
→ Authorization
→ Inventory reservation
→ Payment authorization
→ Order creation
→ Event emission
→ Fulfillment
→ Notification
→ Analytics

The customer sees one button.

Architecture makes the complexity manageable.

That is the craft.

The Ultimate Architecture Is About Reality

Software ultimately exists to model something.

A bank models money.

A marketplace models exchange.

A social network models relationships.

An inventory platform models physical goods.

A logistics platform models movement.

A game models an artificial world.

A music platform models artists, recordings, listeners, rights, and discovery.

The deeper your understanding of the domain, the more powerful your architecture can become.

AI can accelerate implementation.

But the architecture determines whether the implementation reflects the world correctly.

This is why domain knowledge may become increasingly valuable in the age of AI.

The engineer who understands both computation and reality has a powerful combination.

Architecture Becomes the Language Between Human and Machine

Perhaps this is the biggest transformation.

For decades:

Human → Code → Machine

The programmer manually translated intention into instructions.

Increasingly:

Human → Architecture → AI → Code → Machine

The architecture becomes an intermediate language.

It captures the human understanding of the system.

AI translates that understanding into implementation.

This could fundamentally change how software is built.

The source code remains important.

But the architectural specification becomes increasingly central.

When Everyone Can Generate Code

Imagine a future where generating a functioning application is almost trivial.

You describe the product.

An AI system generates the frontend.

Another agent generates the backend.

Another generates the database migrations.

Another generates tests.

Another reviews security.

Another handles deployment.

Another monitors production.

The mechanical construction of software becomes extraordinarily automated.

What remains difficult?

Understanding.

Modeling.

Decision-making.

Prioritization.

System design.

Architecture.

The scarce resource becomes the ability to determine what all these machines should collectively build.

And that is why architecture becomes the moat.

The Future Belongs to Builders Who Can Think in Systems

The most powerful engineers of the next era may not be those who simply write the most code.

They may be the people who can see the entire computational organism.

They understand:

Users
  ↓
Product
  ↓
Domain
  ↓
Data
  ↓
APIs
  ↓
Services
  ↓
Infrastructure
  ↓
Economics

They understand how each layer affects the others.

They know when to simplify.

They know when to separate.

They know when consistency matters.

They know when eventual consistency is acceptable.

They understand failure.

They understand trade-offs.

They understand that every abstraction has a cost.

And they know that architecture is not about predicting the future perfectly.

It is about building a structure capable of surviving an uncertain future.

Final Thought

AI is changing software engineering.

But the most interesting consequence may not be that machines can write code.

It is that good software implementation is becoming increasingly accessible.

That is a beautiful thing.

More people can build.

More ideas can become products.

More developers can experiment.

Small teams can become extraordinarily productive.

Individuals can create software that once required organizations.

The amount of software humanity can produce may expand dramatically.

But abundance changes what is scarce.

When everyone can generate good code, code itself becomes less of a differentiator.

The question moves upward.

Who understands the domain?

Who can model reality?

Who can design the boundaries?

Who can create stable contracts?

Who understands the data?

Who knows where consistency matters?

Who can design systems that evolve?

Who can turn business strategy into technical structure?

Who can create a platform instead of merely an application?

Who can build an architecture capable of supporting millions of future decisions?

That is where the moat begins.

Not because AI is weak.

Because AI is becoming strong.

The stronger the construction engine becomes, the more valuable the blueprint becomes.

The better the machines become at writing software, the more important it becomes to know what software should exist.

The future isn't necessarily about choosing between AI and engineers.

It is about engineers using AI to operate at a higher level.

AI can write the components.

AI can generate the implementation.

AI can accelerate the construction.

And the architect can design the world those components belong to.

That is the opportunity.

Code is becoming easier to generate.

Software is becoming easier to build.

AI is becoming an increasingly powerful engineering partner.

And precisely because of that, architecture becomes greater.

Because architecture is where code stops being a collection of instructions and becomes a system.

It is where data becomes a model of reality.

It is where APIs become contracts.

It is where services become boundaries.

It is where business rules become computational laws.

It is where thousands of implementation decisions become one coherent structure.

And when everyone has access to powerful machines that can build almost anything they can describe, the rarest skill may no longer be construction.

It may be knowing what deserves to be constructed—and designing the system that can carry it.

When everyone can generate code, architecture becomes the moat.

Building a Feature Flag Platform for Distributed Systems

Thu, 17 Sep 2026 21:16:13 +0000

There is a deceptively simple sentence in software engineering:

“Let’s put this feature behind a flag.”

It sounds like a boolean.

enabled = true

Or:

if (featureEnabled) {
    showNewExperience();
}

And for a small application, that may be enough.

But the moment your application becomes distributed, a feature flag stops being a boolean.

It becomes a distributed systems problem.

Now imagine you have:

40 microservices
12 frontend applications
8 background workers
multiple regions
thousands of servers
mobile clients
several deployment environments
millions of users
independent deployment pipelines
multiple engineering teams

And you want to release a new payment flow to exactly 5% of users.

Not 5% of servers.

Not 5% of requests.

Not 5% of a random process.

5% of your users.

You also want:

the feature enabled only for users in Zambia
internal employees to see it immediately
beta users to see it regardless of geography
production disabled while staging remains enabled
instant rollback
an audit trail
consistent evaluation
low latency
high availability
safe behavior when the flag service is unavailable

Suddenly, this:

featureEnabled()

has become an entire infrastructure layer.

And that is the interesting part.

A feature flag platform is not merely a dashboard with toggles.

It is a control plane for software behavior.

The Feature Flag Is a Distributed Decision

Let's start with the fundamental abstraction.

A feature flag system answers a question:

Should subject X receive feature Y?

Formally:

Decision = F(subject, feature, environment, context, configuration)

Where:

subject might be a user
feature identifies the capability
environment could be production or staging
context contains attributes such as country, plan, device, or role
configuration contains targeting rules

For example:

{
  "feature": "new_checkout",
  "environment": "production",
  "user": "user_123",
  "country": "ZM",
  "plan": "premium"
}

The platform might return:

{
  "enabled": true,
  "variant": "modern_checkout"
}

The important realization is this:

The flag configuration is centralized, but its decisions are consumed everywhere.

That makes feature flags interesting.

You have created a distributed decision system.

Control Plane vs Data Plane

The architecture becomes much clearer if we separate two worlds.

Control plane

The control plane is where humans and automation modify feature configuration.

For example:

                    ┌─────────────────────┐
                    │   Feature Flag UI   │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │    Flag API         │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │ Configuration Store │
                    └─────────────────────┘

This is where you create:

new_checkout

and define:

production
 ├── 5% rollout
 ├── Zambia
 ├── premium users
 └── internal employees

The control plane can tolerate slightly higher latency.

A human clicking a dashboard button does not care whether configuration takes 100 milliseconds or 300 milliseconds to save.

The data plane is different.

The Data Plane

Your applications need feature decisions constantly.

A request might look like:

GET /checkout
        │
        ▼
Application
        │
        ▼
Feature SDK
        │
        ▼
Local Evaluation
        │
        ▼
true

This should ideally take microseconds or a few milliseconds.

You do not want:

User request
     ↓
Application
     ↓
Feature Flag API
     ↓
Database
     ↓
Feature Flag API
     ↓
Application
     ↓
User

for every request.

That architecture would turn your feature flag system into a latency dependency for your entire company.

It also creates something much worse.

A feature flag platform outage could become an application outage.

So the data plane should generally be local-first.

The Local Evaluation Model

A powerful architecture is:

                  CONTROL PLANE
                       │
                       │
                       ▼
              ┌────────────────┐
              │ Flag Database   │
              └───────┬────────┘
                      │
                      ▼
              Configuration API
                      │
                      ▼
              ┌────────────────┐
              │ Distribution   │
              │ / Event Stream │
              └───────┬────────┘
                      │
          ┌───────────┼────────────┐
          ▼           ▼            ▼
       Service A   Service B    Service C
          │           │            │
          ▼           ▼            ▼
      Local Cache  Local Cache  Local Cache
          │           │            │
          ▼           ▼            ▼
       Evaluator   Evaluator    Evaluator

Applications receive configuration.

They store it locally.

They evaluate flags locally.

The central platform distributes configuration, not necessarily individual decisions.

This distinction is enormous.

Instead of asking:

"Is new_checkout enabled?"

the application receives the rules necessary to answer:

"Is new_checkout enabled for this user?"

itself.

Why Push Beats Constant Polling

The simplest synchronization strategy is polling.

Every application asks:

GET /flags

every 30 seconds.

It works.

But imagine:

10,000 application instances

polling every 30 seconds.

That is:

10,000 / 30
≈ 333 requests/second

before accounting for retries, multiple environments, multiple flag endpoints, and scaling.

More importantly, polling creates a delay.

If a flag changes immediately after a poll, an instance might remain stale for another 30 seconds.

For feature management, that may be unacceptable during an incident.

A better architecture can use push-based distribution.

For example:

Flag Updated
     │
     ▼
Configuration Service
     │
     ▼
Event Bus
     │
     ├──────► Service A
     ├──────► Service B
     └──────► Service C

Events might look like:

{
  "type": "flag.updated",
  "flag": "new_checkout",
  "version": 184,
  "environment": "production",
  "timestamp": "2026-09-17T21:00:00Z"
}

Clients receive the update and replace their local configuration.

Configuration Is State

Now we encounter a deeper distributed systems problem.

Suppose the feature configuration changes:

version 100

to:

version 101

Service A receives version 101.

Service B does not.

Service C receives version 101.

Then Service B reconnects and receives version 100 from a stale cache.

Now you have:

A → 101
B → 100
C → 101

Your system is inconsistent.

This means feature flag configuration needs versioning.

Every configuration snapshot should have a monotonic version.

For example:

{
  "version": 184,
  "flags": {
    "new_checkout": {
      "enabled": true
    }
  }
}

A client should never replace:

with:

The update rule becomes:

if incoming.version > local.version:
    apply(incoming)

This tiny rule prevents an enormous class of consistency problems.

Snapshots Are Powerful

Instead of sending thousands of individual mutations, the system can maintain snapshots.

Example:

{
  "version": 184,
  "environment": "production",
  "flags": {
    "new_checkout": {
      "enabled": true,
      "rollout": 5
    },
    "new_dashboard": {
      "enabled": false
    }
  }
}

Clients receive the snapshot.

If they disconnect, they continue evaluating against the last known version.

That gives us an important property:

A feature flag platform should degrade into the last known good configuration whenever possible.

This is extremely valuable during outages.

The Flag Store

At the center of the control plane is persistent configuration.

A simple relational schema could look like:

projects
---------
id
name

environments
------------
id
project_id
name

flags
-----
id
environment_id
key
description
type
created_at
updated_at

flag_versions
-------------
id
flag_id
version
configuration
created_at
created_by

The configuration field might contain JSON.

For example:

{
  "enabled": true,
  "strategy": "percentage",
  "percentage": 10
}

Or:

{
  "enabled": true,
  "rules": [
    {
      "attribute": "country",
      "operator": "equals",
      "value": "ZM"
    }
  ]
}

The relational database provides durable source-of-truth storage.

But it does not need to be queried every time a user makes a request.

Designing the Evaluation Engine

The evaluation engine is the heart of the data plane.

Consider:

evaluate(flag, context)

The context could be:

{
  "userId": "123",
  "country": "ZM",
  "plan": "premium",
  "device": "mobile"
}

The evaluator processes rules.

A useful order is:

1. Is the flag globally disabled?
2. Is there a direct user override?
3. Does a targeting rule match?
4. Is the user inside the rollout?
5. Which variant should be returned?
6. Otherwise return default

This ordering must be deterministic.

A flag system should never feel magical.

The same input should produce the same output.

Targeting Rules Are Basically a Query Engine

Consider this:

country == "ZM"
AND
plan == "premium"

Then:

device == "mobile"

The system is evaluating a logical expression.

You could represent it as:

{
  "operator": "AND",
  "conditions": [
    {
      "attribute": "country",
      "operator": "equals",
      "value": "ZM"
    },
    {
      "attribute": "plan",
      "operator": "equals",
      "value": "premium"
    }
  ]
}

More complex rules form a tree:

                AND
               /   \
          country   OR
             ZM   /   \
                plan  employee
              premium  true

This means your feature flag evaluator is slowly becoming a tiny programming language.

That is why rule design matters.

Don't Build an Unrestricted Programming Language

It is tempting to support:

if (someArbitraryJavaScriptExpression)

Don't.

Now your feature configuration is executable code.

You have introduced:

security problems
debugging problems
version compatibility problems
sandboxing problems
unpredictable evaluation
performance problems

Instead, define a small declarative language.

For example:

equals
not_equals
contains
starts_with
greater_than
less_than
in
not_in

Then compose them with:

AND
OR
NOT

Your evaluator remains predictable.

Percentage Rollouts

One of the most useful features is gradual rollout.

Suppose:

new_checkout = 10%

A naive implementation might generate:

Math.random() < 0.1

This is wrong for many use cases.

The same user could receive:

request 1 → true
request 2 → false
request 3 → true

That is terrible for user experience.

Instead, use deterministic hashing.

For example:

bucket = hash(flagKey + ":" + userId) % 100

Then:

if bucket < rolloutPercentage:
    enabled
else:
    disabled

User:

user_123

might consistently map to:

bucket = 7

Therefore:

10% rollout → enabled
20% rollout → enabled
5% rollout → disabled

The user moves predictably as the rollout changes.

Why the Flag Key Should Be Part of the Hash

Consider hashing only:

userId

User 123 might always land in bucket 7.

That means the same user gets selected for the same percentage across every feature.

This can unintentionally correlate experiments.

Instead:

hash(flagKey + userId)

gives each flag an independent distribution.

You can go further:

hash(project + environment + flag + user)

This isolates the distribution boundary.

Variants Turn Flags Into Configuration

Boolean flags are only the beginning.

Instead of:

enabled / disabled

you can have:

variant A
variant B
variant C

For example:

{
  "flag": "checkout_design",
  "variants": {
    "control": 50,
    "modern": 30,
    "minimal": 20
  }
}

Now the evaluator returns:

{
  "enabled": true,
  "variant": "modern"
}

This is where feature flags begin overlapping with experimentation platforms.

The same deterministic hashing system can assign users to variants.

The SDK Is More Important Than the Dashboard

Engineers often imagine the product as:

Dashboard

But developers experience:

SDK

The SDK needs to be almost invisible.

For example:

const enabled = flags.isEnabled(
    "new_checkout",
    {
        userId: user.id,
        country: user.country
    }
);

Or:

if flags.enabled("new_checkout", user):
    ...

The developer should not need to understand the distributed infrastructure underneath.

The SDK hides:

caching
synchronization
retries
local evaluation
configuration parsing
connection management
telemetry
fallback behavior

The complexity belongs in the platform.

Not inside every application.

The SDK Must Not Depend on the Network During Evaluation

This principle deserves repetition.

Bad:

request
 ↓
SDK
 ↓
Feature API
 ↓
network
 ↓
decision

Better:

request
 ↓
SDK
 ↓
memory
 ↓
decision

Configuration synchronization happens separately.

For example:

Background thread
      │
      ▼
Feature service
      │
      ▼
Local memory

Application requests never wait for synchronization.

This is one of the most important architectural decisions in the entire platform.

What Happens When the Flag Service Goes Down?

This is where architecture becomes philosophy.

Suppose the feature service is unavailable.

What should happen?

You have several choices.

Fail open

service unavailable
       ↓
feature = enabled

Useful for features where availability matters more than safety.

Fail closed

service unavailable
       ↓
feature = disabled

Useful for risky features.

Last known value

service unavailable
       ↓
use cached configuration

Usually the most operationally useful strategy.

Per-flag defaults

You can make fallback behavior explicit:

{
  "key": "new_payment_provider",
  "default": false
}

The important thing is that fallback behavior should be designed.

Not discovered accidentally during an outage.

The Cache Hierarchy

A mature platform can use multiple cache layers.

                    Flag Database
                         │
                         ▼
                  Configuration API
                         │
                         ▼
                    Redis Cache
                         │
                         ▼
                    SDK Process
                         │
                         ▼
                   In-Memory Cache

The local SDK memory should be the fastest layer.

Redis can accelerate centralized reads.

The database remains durable source of truth.

But the application should ideally survive temporary failure of every remote layer.

Consistency vs Availability

Now we arrive at classic distributed systems territory.

Imagine you update:

maintenance_mode = true

because production is broken.

You want every service to know immediately.

But distributed systems cannot magically guarantee zero propagation delay.

There will be a window:

t0: control plane updated
t1: Service A receives update
t2: Service B receives update
t3: Service C receives update

For a short period:

A → true
B → false
C → false

This is eventual consistency.

For most product rollouts, that is acceptable.

For certain operational controls, it might not be.

This suggests an important architectural distinction:

Not every flag has the same consistency requirements.

Flag Classes

You can classify flags.

Release flags

Used to hide incomplete features.

Consistency requirement:

eventual

Experiment flags

Used for user segmentation.

Consistency requirement:

stable assignment

Operational flags

Used to turn expensive or risky systems on/off.

Consistency requirement:

fast propagation

Kill switches

Used during emergencies.

Consistency requirement:

as close to immediate as practical

This classification can influence the underlying architecture.

Kill Switches

A kill switch deserves special treatment.

Imagine:

new_payment_processor = true

Then the processor begins failing.

An engineer clicks:

DISABLE

The system should propagate this rapidly.

You might use:

Control Plane
     │
     ▼
Event Stream
     │
     ├── Service A
     ├── Service B
     ├── Service C
     └── Service D

The SDK applies the new version immediately.

You can also expose:

POST /flags/new_payment_processor/disable

for automation.

This is where feature flags become incident-management infrastructure.

Audit Logs Are Not Optional

Who changed the flag?

When?

From what?

To what?

Why?

You need answers.

A simple audit record:

{
  "flag": "new_checkout",
  "environment": "production",
  "actor": "derek",
  "action": "update",
  "before": {
    "percentage": 5
  },
  "after": {
    "percentage": 20
  },
  "timestamp": "2026-09-17T21:10:00Z"
}

Now a production incident becomes traceable.

You can ask:

What changed before the incident?

and actually answer it.

RBAC

Not everyone should be allowed to change everything.

You might define:

Viewer
Developer
Release Manager
Admin

And permissions:

Viewer
  → read

Developer
  → read
  → modify development

Release Manager
  → modify staging
  → modify production

Admin
  → everything

Better still:

Project → Environment → Permission

A developer might have:

Project A / Development → write
Project A / Production  → read

This prevents accidental production changes.

Approval Workflows

For sensitive flags, changing:

production percentage: 10 → 100

might require approval.

Instead:

Developer
   │
   ▼
Create Change
   │
   ▼
Review
   │
   ▼
Approve
   │
   ▼
Publish

Now your flag platform has become part of the software delivery pipeline.

Configuration as a Versioned Artifact

Another powerful approach is treating flag configurations like source code.

Instead of simply:

current state

you have:

version 181
version 182
version 183
version 184

Then rollback becomes:

184 → 183

rather than manually reconstructing the previous state.

You can even expose:

diff(version_183, version_184)

Example:

- rollout: 10
+ rollout: 25

This is much safer than staring at a dashboard and guessing what changed.

Multi-Region Architecture

Now suppose your platform serves:

Africa
Europe
North America
Asia

A single central control plane could introduce latency.

You might deploy regional distribution nodes:

                 Global Control Plane
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
           Africa      Europe    America
           Node         Node       Node
              │          │          │
              ▼          ▼          ▼
          Local SDKs  Local SDKs  Local SDKs

The global configuration remains authoritative.

Regional nodes distribute snapshots closer to applications.

This creates another consistency boundary.

Again:

global state
     ↓
regional state
     ↓
local process state

You are building a hierarchy of replicated state.

The Event Stream

Kafka, NATS, Redis Streams, Pulsar, or another messaging system can distribute configuration changes.

An event:

{
  "event_id": "evt_9182",
  "type": "configuration.updated",
  "project": "commerce",
  "environment": "production",
  "version": 184
}

Consumers need idempotency.

If event 9182 arrives twice:

apply(9182)
apply(9182)

the second application should be harmless.

Version checks help:

incoming 184
local 184

ignore

This gives us an elegant property:

Duplicate delivery becomes safe.

Reconnection Logic

Distributed systems fail.

Connections drop.

Networks disappear.

Servers restart.

Containers are recreated.

Therefore the SDK needs a lifecycle.

Something like:

START
  ↓
LOAD LOCAL SNAPSHOT
  ↓
CONNECT
  ↓
SYNC
  ↓
EVALUATE
  ↓
DISCONNECT
  ↓
RECONNECT

If the connection fails:

retry

But don't do:

retry every 1ms

Use exponential backoff:

1s
2s
4s
8s
16s
30s

with jitter.

Otherwise 10,000 clients could reconnect simultaneously and create a thundering herd.

The Thundering Herd Problem

Imagine your feature service restarts.

10,000 SDKs discover the connection is gone.

They all reconnect immediately.

10,000 clients
      │
      ▼
Feature Service
      │
      💥

Instead:

retryDelay = exponentialBackoff + randomJitter

Now clients spread their reconnection attempts.

This is a small implementation detail with enormous operational consequences.

Observability

You cannot operate a distributed feature system blindly.

You need metrics.

At minimum:

flag_evaluations_total
flag_evaluation_errors_total
flag_cache_age_seconds
flag_config_version
flag_sync_latency
flag_sync_failures
flag_stream_connections

You might also track:

evaluation_latency

For example:

p50 = 0.01ms
p95 = 0.03ms
p99 = 0.08ms

Because the evaluator lives inside application request paths, performance matters.

Evaluation Tracing

When a user unexpectedly receives a feature, developers should be able to ask:

Why?

A good evaluator can optionally produce a reason:

{
  "enabled": true,
  "variant": "modern",
  "reason": "country_rule",
  "rule_id": "rule_18"
}

Or:

{
  "enabled": true,
  "variant": "control",
  "reason": "percentage_rollout",
  "bucket": 42
}

This is incredibly useful for debugging.

But detailed evaluation logging should be optional.

You don't want to generate billions of logs for every feature check.

Privacy

Feature flags often consume user attributes:

country
age
plan
role
device
account status

The platform should minimize what it stores.

Ideally, the SDK sends only what is required for evaluation.

Instead of:

send entire user profile

send:

{
  "userId": "123",
  "country": "ZM",
  "plan": "premium"
}

Even better, many evaluations can happen entirely locally.

The feature service does not need to know every user's private context.

The Security Model

SDK keys need different levels of access.

A client-side SDK key should not be equivalent to an administrative API token.

You might have:

Server SDK Key
Client SDK Key
Admin API Token
Read-only Token

Server-side credentials can access sensitive configuration.

Client-side configuration should expose only what the client actually needs.

Never embed administrative credentials in frontend applications.

A feature flag platform is infrastructure.

Treat it like infrastructure.

Client-Side Flags Are Different

Consider a mobile app.

You cannot safely assume:

if (!flag) {
    feature does not exist
}

because an attacker can inspect the application.

Client-side flags are useful for:

UI changes
experimentation
staged interface rollout

They should not be used as the primary security mechanism.

Never rely on:

feature flag = false

to enforce authorization.

Authorization belongs in the backend.

The backend must independently enforce:

user.canPerform(action)

Feature flags control behavior.

They should not become your security boundary by accident.

A Production Architecture

A mature platform could look like this:

                         ┌───────────────────┐
                         │   Admin Dashboard │
                         └─────────┬─────────┘
                                   │
                                   ▼
                         ┌───────────────────┐
                         │   Feature API     │
                         └─────────┬─────────┘
                                   │
                    ┌──────────────┼──────────────┐
                    ▼              ▼              ▼
               PostgreSQL       Redis          Audit Log
                    │
                    ▼
              Event Publisher
                    │
                    ▼
               Event Stream
                    │
          ┌─────────┼──────────┐
          ▼         ▼          ▼
      Region A   Region B   Region C
          │         │          │
          ▼         ▼          ▼
       SDKs       SDKs        SDKs
          │         │          │
          ▼         ▼          ▼
      In-Memory  In-Memory   In-Memory
       Config     Config      Config
          │         │          │
          ▼         ▼          ▼
      Evaluator  Evaluator  Evaluator

The architecture separates:

Management
Distribution
Evaluation

That separation is the key.

Building the MVP

You do not need to build all of this on day one.

A strong MVP can be surprisingly small.

Start with:

1. Projects
2. Environments
3. Boolean flags
4. REST API
5. SDK
6. In-memory cache
7. Polling
8. Percentage rollout
9. Audit logs

Architecture:

Dashboard
    │
    ▼
REST API
    │
    ▼
PostgreSQL
    │
    ▼
SDK polling
    │
    ▼
Local cache
    │
    ▼
Evaluator

This already provides enormous value.

Then add:

targeting rules
        ↓
variants
        ↓
streaming
        ↓
RBAC
        ↓
approval workflows
        ↓
multi-region distribution

Build the complexity only when the problem demands it.

An Example Evaluation API

A clean API might look like:

flags.evaluate("new_checkout", {
    userId: "123",
    country: "ZM",
    plan: "premium"
});

Returning:

{
    enabled: true,
    variant: "modern",
    reason: "targeting_rule",
    configVersion: 184
}

For convenience:

flags.isEnabled("new_checkout", context)

could simply return:

true

while advanced users can access:

evaluate()

for diagnostic information.

Designing the SDK Internals

A useful SDK architecture:

                 FeatureSDK
                     │
        ┌────────────┼────────────┐
        ▼            ▼            ▼
   ConfigStore    Evaluator    Transport
        │            │            │
        ▼            │            ▼
    Memory Map       │       HTTP / Stream
        │            │
        └────────────┘

The evaluator should depend on an abstract configuration store.

For example:

interface ConfigStore {
    get(flagKey: string): Flag | undefined;
}

Then you can replace:

MemoryStore

with:

FileStore
RemoteStore
TestStore

without rewriting the evaluator.

This is good software architecture.

Testing the Evaluator

The evaluator deserves obsessive testing.

Test:

flag disabled
flag enabled
missing flag
missing user
percentage = 0
percentage = 100
country match
country mismatch
multiple rules
AND
OR
NOT
variant assignment
stale configuration
unknown operator
invalid configuration

Also test deterministic behavior.

For:

flag = new_checkout
user = 123

the evaluator should return the same result 10,000 times.

You can test:

results = [
    evaluate("new_checkout", {"userId": "123"})
    for _ in range(10000)
]

assert len(set(results)) == 1

Determinism is a feature.

Configuration Validation

Never let invalid configuration reach production.

Before publishing:

Validate
   ↓
Compile
   ↓
Version
   ↓
Persist
   ↓
Publish

For example:

{
  "percentage": 150
}

should fail validation.

So should:

{
  "operator": "does_not_exist"
}

This is similar to a compiler.

The dashboard is effectively generating a small configuration program.

Validate it before distributing it.

Precompiling Rules

For extremely high-scale systems, parsing JSON and recursively interpreting rules on every request may be unnecessary overhead.

You can compile configuration into an optimized representation.

For example:

JSON rules
    ↓
Parser
    ↓
AST
    ↓
Compiled evaluator

Then:

request
  ↓
compiled evaluator
  ↓
decision

This is where feature flags become surprisingly close to language-runtime design.

You are taking declarative configuration and compiling it into executable decision logic.

Feature Flags and Deployment

The most important conceptual distinction is:

deployment ≠ release

Deployment asks:

Is the code running?

Release asks:

Is the user allowed to experience it?

Without feature flags:

build
 ↓
deploy
 ↓
everyone gets feature

With feature flags:

build
 ↓
deploy
 ↓
disabled
 ↓
internal users
 ↓
1%
 ↓
5%
 ↓
25%
 ↓
100%

This fundamentally changes how software can be shipped.

Deployment becomes infrastructure.

Release becomes configuration.

Feature Flags as a Control Plane for Product Behavior

This is the bigger idea.

Traditional infrastructure control planes manage:

servers
networks
containers
databases

A feature flag platform manages something different:

software behavior

It answers:

Which code path should execute?
For whom?
Where?
When?
At what percentage?
With which variant?

That makes feature flags incredibly powerful.

Your code contains possibilities.

Your configuration decides which possibilities become reality.

But There Is a Dark Side

Feature flags accumulate.

You create:

new_checkout

Then:

new_dashboard

Then:

new_search

Then:

temporary_fix

Then:

experiment_42

Two years later:

flags = 731

Nobody knows which ones are still active.

Now the feature flag platform has become technical debt infrastructure.

The solution is lifecycle management.

Every flag should have:

owner
created_at
expires_at
purpose
environment
status

You should know:

Who owns this flag?
Why does it exist?
When should it disappear?

Flag Lifecycle

A healthy lifecycle might be:

PLANNED
   ↓
DEVELOPMENT
   ↓
TESTING
   ↓
ROLLOUT
   ↓
FULLY_RELEASED
   ↓
DEPRECATED
   ↓
REMOVED

The platform can even alert:

Flag "new_checkout"
has exceeded its expiration date.

Now the system helps engineers remove complexity instead of accumulating it.

The Architecture of Rollback

One of the most beautiful properties of feature flags is that rollback can become configuration rather than deployment.

Suppose version:

v10

contains a broken feature.

Without flags:

rollback deployment

With flags:

disable feature

The code remains deployed.

The behavior changes.

This means:

deployment rollback

and:

feature rollback

become separate operations.

That separation can drastically improve incident response.

The Ultimate Failure Test

A good feature flag platform should survive this thought experiment:

The entire feature flag control plane disappears.

What happens?

Your applications should ideally continue running.

They should have:

last known configuration

and:

safe defaults

The architecture should fail like this:

Control Plane
     ❌
      │
      │
      ▼
Existing SDKs
     │
     ▼
Local configuration
     │
     ▼
Application continues

Not:

Control Plane
     ❌
      │
      ▼
Every application
     ❌

That is the difference between a feature platform and a distributed dependency that happens to have a dashboard.

What I Would Build

If I were designing a serious feature flag platform from scratch, I would start with:

                    ┌─────────────────┐
                    │ React Dashboard │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │ API Gateway     │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │ Flag Service    │
                    └───────┬─────────┘
                            │
                ┌───────────┼───────────┐
                ▼           ▼           ▼
           PostgreSQL     Redis       Audit DB
                │
                ▼
         Versioned Config
                │
                ▼
          Event Publisher
                │
                ▼
            NATS/Kafka
                │
       ┌────────┼────────┐
       ▼        ▼        ▼
     SDK A    SDK B    SDK C
       │        │        │
       ▼        ▼        ▼
    Memory   Memory    Memory
       │        │        │
       ▼        ▼        ▼
    Evaluate Evaluate Evaluate

The backend could be implemented in:

Rust
Go
Java
Node.js

The SDKs could support:

JavaScript
TypeScript
Python
Go
Rust
Java
PHP

And the system should expose:

REST API
SDK API
Streaming API
Webhook API
CLI

Now you don't just have a feature toggle.

You have a platform.

The Deeper Engineering Lesson

The fascinating thing about feature flags is that they look simple from the outside.

A developer writes:

if (flags.enabled("new_checkout")) {
    ...
}

Five words.

But underneath those five words can exist:

distributed configuration
versioning
replication
caching
consistency
deterministic hashing
rule evaluation
event streaming
fault tolerance
RBAC
audit logs
observability
security
multi-region synchronization

That is software architecture.

The checkbox is merely the user interface.

The real system is the machinery underneath it.

And this is one of the recurring patterns in advanced engineering:

Simple interfaces often hide complicated distributed systems.

A feature flag is a tiny API sitting on top of a much larger idea:

centralized control
+
distributed execution

The control plane decides.

The data plane executes.

The configuration travels.

The evaluation happens locally.

The application keeps moving.

And that architecture gives teams something more valuable than a green checkbox.

It gives them control over software behavior without having to redeploy the entire world.

That is the real power of feature flags.

Not turning features on and off.

Turning software releases into a programmable system.

And once you see feature flags that way, you stop designing them as dashboard toggles.

You start designing them as what they really are:

A Distributed Control Plane for Software Behavior.

Connecting N4K48 and Nicola Comics to MyZubster: Building a Public Zorgax Integration

Tue, 15 Sep 2026 17:17:14 +0000

Connecting N4K48 and Nicola Comics to MyZubster: Building a Public Zorgax Integration

We are building a new experimental connection inside the MyZubster ecosystem:

Nicola / N4K48 × Nicola Comics × Zorgax × MyZubster

The experiment started locally.

Now we are moving toward the next stage:

Can an independent developer or creator build their own project, expose a controlled API, connect it to an AI assistant such as Zorgax, and become a publicly reachable node of a wider ecosystem?

Nicola / N4K48 is helping us test this model.

Our objective is not only to build the integration.

We want to connect it, deploy it, measure it, document it and verify what actually works.

Explore the Projects

MyZubster
https://www.myzubster.com/?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=myzubster_home

Zorgax
https://www.myzubster.com/zorgax?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=zorgax

MyZubster Marketplace Demos
https://www.myzubster.com/marketplace-demos?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=marketplace_demos

MyZubster GitHub
https://github.com/MyZubster-Ecosystem/myzubster?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=myzubster_github

MyZubster Visual Universe
https://github.com/MyZubster-Ecosystem/MyZubster-Visual?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=visual_universe

Nicola / N4K48 GitHub Profile
https://github.com/nicolaususnicola-lgtm?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=n4k48_profile

N4K48 × MyZubster
https://github.com/nicolaususnicola-lgtm/myzubster?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=n4k48_myzubster

Nicola Comics / N4K48 MVP
https://github.com/nicolaususnicola-lgtm/myzubster-mvp?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=nicola_mvp

N4K48 Metaverse Roadmap
https://github.com/nicolaususnicola-lgtm/myzubster/blob/main/ROADMAP_N4K48_METAVERSE.md?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=n4k48_roadmap

Public Integration Tracking — MyZubster Issue #1176
https://github.com/MyZubster-Ecosystem/myzubster/issues/1176?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=integration_issue

Who is N4K48?

N4K48 is Nicola's experimental persistent digital identity inside the MyZubster universe.

The project connects several layers:

Nicola
   ↓
N4K48
   ↓
Neon Plaza
   ↓
Project / Nicola Comics
   ↓
Zorgax
   ↓
MyZubster

N4K48 is therefore more than a visual character.

The experiment connects digital identity, software development, creative work, AI-assisted workflows and verifiable technical activity.

The visual/metaverse environment provides the narrative layer.

The repositories, APIs, Docker services, tests and evidence provide the technical layer.

Keeping those layers clearly separated is important.

Nicola Comics × MyZubster

For this stage of the experiment we are using Nicola Comics as a concrete integration case.

Nicola has created three N4K48 comic panels documenting his journey from a software idea toward AI, MyZubster and the experimental Neon Plaza environment.

Instead of treating the comics only as static images, the project exposes them through a small structured service.

The pilot provides a comics catalog through:

GET /api/comics

and a Zorgax-facing adapter through:

POST /api/zorgax/ask

The current adapter supports four explicit operations:

gallery
detail
candidate
next_steps

This creates a controlled interoperability layer between an independently maintained project and Zorgax.

What Has Already Been Tested?

The Docker-based pilot has already been manually validated locally.

The main workflow reported by the pilot is:

gallery      → PASS
detail       → PASS
candidate    → PASS
next_steps   → PASS

The catalog currently contains three N4K48 comic panels.

The first selected comic is classified as:

NFT_CANDIDATE
PROPOSED_FOR_REVIEW

Its rights state remains:

TO_VERIFY

That distinction is intentional.

We are not claiming that an NFT has already been minted.

A candidate is not a minted NFT.

A proposed asset is not verified ownership.

A visual is not blockchain evidence.

Contract address, token ID and transaction hash should only be presented as verified information when real on-chain evidence exists.

Why Is the Zorgax Adapter Read-Only?

The integration deliberately limits what Zorgax can do.

Zorgax can help a user discover the comics, inspect an item, identify the current candidate and understand the next steps.

It does not automatically modify Nicola's catalog, access a wallet, execute payments, mint NFTs, change rights information or manufacture blockchain evidence.

The architecture is therefore:

AI ASSISTANCE
      ↓
CONTROLLED API
      ↓
PROJECT DATA
      ↓
EVIDENCE
      ↓
HUMAN / EXTERNAL VERIFICATION

AI helps navigate the process.

It does not automatically become the authority that verifies the process.

From Local MVP to Public Integration

The local connection is only the first milestone.

We are now working toward:

LOCAL MVP
   ↓
PUBLIC HTTPS DEPLOYMENT
   ↓
PUBLIC ZORGAX CONNECTION
   ↓
END-TO-END TEST
   ↓
TRAFFIC / USAGE MEASUREMENT
   ↓
DOCUMENTED EVIDENCE

Nicola's service needs to become reachable through a public HTTPS endpoint.

That will allow the public MyZubster/Zorgax environment to communicate with Nicola Comics without depending on a service running on Nicola's local computer.

The target public flow is:

USER
 ↓
MYZUBSTER
 ↓
ZORGAX
 ↓ HTTPS
NICOLA COMICS API
 ↓
GALLERY
 ↓
DETAIL
 ↓
CANDIDATE
 ↓
RIGHTS / EVIDENCE
 ↓
NEXT STEPS

Measuring This Experiment

We also want to understand how people discover and use the experiment.

The links in this article therefore contain campaign attribution parameters:

utm_source=coderlegion
utm_medium=article
utm_campaign=n4k48_zorgax_pilot

Important destinations receive a different utm_content.

For example:

utm_content=myzubster_home
utm_content=zorgax
utm_content=marketplace_demos
utm_content=n4k48_profile
utm_content=nicola_mvp

The attribution path becomes:

CODER LEGION
     ↓
ARTICLE
     ↓
TRACKED LINK
     ↓
MYZUBSTER / ZORGAX
     ↓
USER JOURNEY
     ↓
ANALYTICS

Where analytics is configured on the destination, this can help us distinguish traffic generated by this Coder Legion article from other acquisition sources.

Traffic Is Not the Same as Success

We don't want to confuse traffic with validation.

CLICK
≠
VISITOR
≠
ACTIVE USER
≠
ZORGAX REQUEST
≠
SUCCESSFUL API CALL
≠
COMPLETED WORKFLOW
≠
VERIFIED OUTCOME

The more useful question is whether someone can actually complete the journey:

Coder Legion
      ↓
MyZubster
      ↓
N4K48 discovered
      ↓
Zorgax opened
      ↓
Nicola Comics requested
      ↓
gallery
      ↓
detail
      ↓
candidate
      ↓
next_steps

This turns analytics into part of an engineering experiment rather than just a visitor counter.

Why Are We Doing This?

The larger objective goes beyond Nicola Comics.

We want to explore whether MyZubster can support independently maintained projects without requiring every participant to move their application into one centralized codebase.

Imagine:

Nicola Comics ──────────┐
Research Lab ───────────┤
Student Projects ───────┤
Agriculture ────────────┤
Circular Economy ───────┼── ZORGAX ── MYZUBSTER
Local Services ─────────┤
Community Projects ─────┤
Creative Projects ──────┘

Each project could maintain its own:

IDENTITY
REPOSITORY
APPLICATION
API
DATA
DOCUMENTATION
EVIDENCE

while exposing a controlled interface that MyZubster and Zorgax understand.

Toward a Federated MyZubster

Instead of putting everything inside one application:

EVERYTHING
    ↓
ONE CENTRAL PLATFORM

we are experimenting with:

PROJECT A ─ API ─┐
PROJECT B ─ API ─┤
PROJECT C ─ API ─┤
                 ↓
               ZORGAX
                 ↓
             MYZUBSTER

The independent project remains responsible for its service.

MyZubster provides the ecosystem and connection layer.

Zorgax provides discovery, guidance and AI interaction.

Analytics helps us understand usage.

Evidence tells us what has actually been demonstrated.

Evidence First

The most important principle remains:

Do not claim more than the evidence demonstrates.

A local test is a local test.

A public deployment is a public deployment.

A click is a click.

An API response is an API response.

An NFT candidate is an NFT candidate.

Verified rights require evidence.

A blockchain mint requires an actual transaction.

Our workflow is therefore:

OBSERVE
   ↓
DOCUMENT
   ↓
BUILD
   ↓
TEST
   ↓
DEPLOY
   ↓
CONNECT
   ↓
MEASURE
   ↓
VERIFY
   ↓
PUBLISH

The Final Goal

The final goal is not simply to put Nicola's comics online.

We want to demonstrate a reusable architecture where an independent creator, developer, researcher, student or community project can become a connected MyZubster node.

A future participant could bring:

IDENTITY
+
PROJECT
+
GITHUB REPOSITORY
+
PUBLIC SERVICE
+
API
+
ZORGAX CONNECTION
+
ANALYTICS
+
EVIDENCE

and progressively move through:

PROPOSED
↓
IMPLEMENTED
↓
TESTED
↓
DEPLOYED
↓
CONNECTED
↓
MEASURED
↓
VERIFIED

Nicola / N4K48 is helping us test this architecture with a real project.

The immediate milestone is:

Public HTTPS deployment → public Zorgax connection → end-to-end test → traffic attribution → documented evidence.

If this pilot succeeds, we will have more than another demo.

We will have a repeatable integration pattern for future MyZubster contributors.

Build → Connect → Deploy → Measure → Verify → Document → Replicate.

Follow and Test the Experiment

Enter MyZubster
https://www.myzubster.com/?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=final_cta

Try Zorgax
https://www.myzubster.com/zorgax?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=final_zorgax_cta

Explore Marketplace Demos
https://www.myzubster.com/marketplace-demos?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=final_marketplace_cta

Explore N4K48
https://github.com/nicolaususnicola-lgtm?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=final_n4k48_cta

Explore Nicola Comics / MVP
https://github.com/nicolaususnicola-lgtm/myzubster-mvp?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=final_nicola_mvp_cta

Follow the public integration
https://github.com/MyZubster-Ecosystem/myzubster/issues/1176?utm_source=coderlegion&utm_medium=article&utm_campaign=n4k48_zorgax_pilot&utm_content=final_issue_cta

MyZubster #Zorgax #OpenSource #AI #API #Docker #BuildInPublic #Analytics #DigitalIdentity #WebDevelopment

Idempotency in APIs - Why Your Retry Logic Can Break Everything (And How to Fix It)

Sat, 12 Sep 2026 13:30:46 +0000

Modern applications rely heavily on APIs. But networks are unreliable - requests fail, time out, or get retried.

Now imagine this scenario:

User clicks “Pay Now”
Request sent to server
Payment processed successfully
But response is lost due to network issue
Client retries the request

💥 Payment gets processed again. User is charged twice.

This is one of the most dangerous and common problems in distributed systems.

The solution? Idempotency.

What is Idempotency?

An operation is idempotent if performing it multiple times produces the same result as performing it once.

In simple terms:

Same request → same effect → no duplicate side effects

Example:

Create Order (without idempotency):
Request sent twice → 2 orders created ❌

Create Order (with idempotency):
Request sent twice → only 1 order created ✅

Idempotency ensures system correctness even when retries happen.

Why Do Retries Happen?

Retries are extremely common due to:

Network timeouts
Server crashes
Client retries
Load balancers retrying requests
Mobile networks instability
API gateway retries
Reverse proxy retries

Retries are normal. Duplicate side effects are not.

Which HTTP Methods Are Idempotent?

According to HTTP specification:

Method	Idempotent	Explanation
GET	✅ Yes	Fetch data
PUT	✅ Yes	Replace resource
DELETE	✅ Yes	Delete resource
POST	❌ No	Create resource
PATCH	❌ No	Partial update

POST is NOT idempotent by default.

Example:

POST /orders

Calling twice → creates 2 orders.

This must be handled explicitly.

Real-World Example: Payment API Problem

Client sends:

POST /payments
{
  "user_id": "123",
  "amount": 100
}

If client retries due to timeout:

POST /payments

Without idempotency → duplicate payment ❌

With idempotency → single payment ✅

Solution: Idempotency Key

Client sends a unique key with request:

POST /payments
Idempotency-Key: abc123

Server behavior:

If key is new → process request
If key already exists → return previous result

This guarantees single execution.

How Idempotency Works Internally

Flow:

Client generates unique key
Sends request with key
Server checks database/cache for key
If key exists → return stored response
If not exists → process request
Store key and response
Return response

Go Implementation Example

Let’s implement idempotent payment API using Go.

Database Table

idempotency_keys table:

key (primary key)
response
created_at

Go Example (Basic Version)

type IdempotencyRecord struct {
    Key      string
    Response string
}

Handler Implementation

func PaymentHandler(w http.ResponseWriter, r *http.Request) {

    key := r.Header.Get("Idempotency-Key")

    if key == "" {
        http.Error(w, "Idempotency-Key required", http.StatusBadRequest)
        return
    }

    // Check if key exists
    record, exists := getIdempotencyRecord(key)

    if exists {
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(record.Response))
        return
    }

    // Process payment
    paymentID := processPayment()

    response := fmt.Sprintf(`{"payment_id": "%s"}`, paymentID)

    // Store key and response
    saveIdempotencyRecord(key, response)

    w.Header().Set("Content-Type", "application/json")
    w.Write([]byte(response))
}

Example Flow

First request:

POST /payments
Idempotency-Key: abc123

Response:
{
  "payment_id": "pay_001"
}

Retry request:

POST /payments
Idempotency-Key: abc123

Response returned from storage:

{
  "payment_id": "pay_001"
}

No duplicate payment created.

Production Implementation Using Redis (Recommended)

Redis is ideal because:

Extremely fast
Supports expiration (TTL)
Perfect for temporary idempotency storage

Redis Example in Go

func PaymentHandler(w http.ResponseWriter, r *http.Request) {

    key := r.Header.Get("Idempotency-Key")

    val, err := redisClient.Get(ctx, key).Result()

    if err == nil {
        w.Write([]byte(val))
        return
    }

    paymentID := processPayment()

    response := fmt.Sprintf(`{"payment_id": "%s"}`, paymentID)

    redisClient.Set(ctx, key, response, time.Hour*24)

    w.Write([]byte(response))
}

Important: Handle Race Conditions

Two requests may arrive simultaneously with same key.

Solution: Use atomic operations.

Redis example:

SET key value NX

NX = set only if not exists

Important Best Practice: Store Full Response

Do not just store key.

Store:

status code
response body
timestamp

Because retry must return exact same response.

Example Database Schema (PostgreSQL)

CREATE TABLE idempotency_keys (
    key TEXT PRIMARY KEY,
    response JSONB,
    status_code INT,
    created_at TIMESTAMP DEFAULT NOW()
);

Primary key ensures uniqueness.

Real-World Systems That Use Idempotency

Critical systems use idempotency extensively:

Payment APIs
Order creation APIs
Financial transactions
Booking systems
Wallet transfers
Stripe payments
Banking APIs

Without idempotency, financial systems would break.

Common Mistake Developers Make

❌ Using retry without idempotency

retry 3 times on failure

This multiplies side effects.

Example:

Retry logic: 3 times
Result: 3 payments

Dangerous bug.

Idempotency vs Duplicate Prevention Using Unique Constraints

Example:

UNIQUE(user_id, order_id)

This prevents duplicates at DB level.

But idempotency is better because:

Works at API level
Handles retries cleanly
Returns same response

Best approach: use BOTH.

When Should You Use Idempotency?

Use idempotency for operations that create side effects:

Payment creation
Order creation
Wallet transfer
Booking
Resource creation

Not required for:

GET requests
Read-only operations

How Client Should Generate Idempotency Key

Use UUID:

Example:

550e8400-e29b-41d4-a716-446655440000

Go example:

uuid.New().String()

Unique per operation.

Idempotency Key Expiration Strategy

Keys should expire after some time.

Common TTL:

24 hours
48 hours
7 days (depending on business need)

Redis makes expiration easy.

Example Architecture

Client → API Gateway → Service → Redis → Database

Redis handles idempotency keys
Database handles actual data

Fast and reliable.

Real Production Scenario: Mobile Network Retry

Mobile user taps "Place Order"

Network slow → timeout
Client retries automatically

Without idempotency → duplicate orders

With idempotency → safe retry

Important Difference: Safe Retry vs Unsafe Retry

Unsafe retry:

POST /orders

Safe retry:

POST /orders
Idempotency-Key: unique-key

Idempotency vs Exactly-Once Execution

Exactly-once execution is impossible in distributed systems.

Idempotency provides practical solution.

It ensures same final state.

Best Practices Checklist

Always:

Use idempotency keys for POST APIs
Store keys in Redis or database
Store full response
Use expiration
Handle race conditions
Use UUID keys
Combine with database constraints

Key Takeaway

Retries are normal.

Duplicate side effects are dangerous.

Idempotency ensures your APIs remain safe, reliable, and production-ready.

It is not optional for critical APIs - it is essential.

Final Thought

If your API handles payments, orders, bookings, or financial transactions - idempotency is mandatory.

Without it, retries can silently corrupt your system.

With it, your APIs become reliable, predictable, and safe.

Building a Lightweight Portfolio Dashboard with ProvChart & st-core design Tokens

Fri, 11 Sep 2026 15:58:14 +0000

Open-source template: github.com/Figsh/dashboard-project

Most dashboard tutorials start the same way: pull in 40–150 KB of JavaScript, wait for hydration, and hope search engines eventually see the chart.

We wanted the opposite.

This is a dark portfolio dashboard that:

Renders charts with zero charting JavaScript on the page
Uses ProvChart to turn data into pure HTML + CSS
Keeps the original st-core.fscss design tokens for a consistent, high-end look
Stays crawlable, fast on first paint, and easy to theme

This article walks through how it was built and how you can reuse the template.

The problem with typical chart libraries

Approach	Cost	First paint	SEO / crawl
JS Libs	40–150 KB+ JS	After hydration	Often missed
Hand-written CSS charts	Fragile, limited points	Instant	Good
ProvChart	0 KB chart runtime	Instant	Excellent

ProvChart compiles your data into CSS custom properties and returns scoped HTML + CSS. The browser paints the shape with native CSS (clip-path, gradients, etc.). No chart library ships to the client.

Stack overview

ProvChart API — POST /api/v1/generate → { html, css }
st-core tokens — the same CSS variables used by st-core.fscss
Vite — fast local dev and simple build
Tiny Node proxy — keeps the API key off the browser

Design tokens stay exactly as they were:

:root {
  --st-bg:       #080710;
  --st-surface:  #0f0e1c;
  --st-card:     #13122a;
  --st-accent:   #6c47ff;
  --st-accent-2: #a78bff;
  --st-green:    #00e5a0;
  --st-red:      #ff4d6a;
  --st-text:     #edeaff;
  --st-muted:    #5e5a82;
  --st-border:   rgba(108, 71, 255, 0.13);
}

The UI (sidebar, KPIs, allocation bars, activity feed, profile dropdown) is pure CSS on top of these tokens. Charts are the only part that talks to an external service.

How ProvChart is wired in

1. Server-side proxy (key stays private)

// api/proxy-server.js (simplified)
const res = await fetch('https://provchart-api.devtem.org/api/v1/generate', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-API-Key': process.env.PROVCHART_API_KEY,
  },
  body: JSON.stringify(payload),
});

The browser only calls /api/chart. Never expose a live key in frontend bundles.

2. Client payload (real numbers, not 0–100)

{
  type: 'area',
  theme: 'midnight',
  height: 260,
  axisX: ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun', 'Mon', 'Now'],
  series: [
    { name: 'Portfolio', color: '#6c47ff', points: [74200, 76800, …, 84201] },
    { name: 'Benchmark', color: '#00c9b8', points: [72800, 74500, …, 81000] },
  ],
  legend: true,
  grid: true,
}

ProvChart auto-formats large values (20K, 1.3K, 2M, 1.09B…). You send raw numbers; you do not pre-format with k / M / B.

3. Inject HTML + CSS

const { html, css } = await res.json();
host.innerHTML = html;
// each chart gets its own <style> so instances never clobber each other

Each host (#main-chart-host, #spark-btc-host, …) keeps a scoped style tag. That avoids the classic “last chart wins and the others go blank” bug.

Data shape: 9 points, clear narrative

Instead of 24 noisy points, every series uses 9 points with a deliberate arc:

Downfall — a clear dip
High rate — a strong recovery into “Now”

Example (1W portfolio):

portfolio: [74200, 76800, 75500, 72800, 71500, 76200, 79800, 82500, 84201]
//          Mon     Tue    Wed    Thu    Fri    Sat    Sun    Mon    Now
//                         ↘ dip ─────────↗ strong climb

Labels stay short and aligned (Mon … Now). No client-side slicing or thinning.

Dashboard features in the template

Period tabs: 1D / 1W / 1M / 3M / 1Y (each re-requests ProvChart)
KPI row + overview stats strip
Main multi-series area chart (Portfolio vs Benchmark)
BTC / ETH spark area charts
Allocation bars + recent activity feed
Sidebar with profile dropdown
Live clock, responsive layout, same st-core visual language

Everything is vanilla JS modules + CSS. No React/Vue required.

Getting started

git clone https://github.com/Figsh/dashboard-project.git
cd dashboard-project
npm install

cp .env.example .env
# or export PROVCHART_API_KEY="your_api_key"
# add PROVCHART_API_KEY from https://chart.devtem.org/dashboard

# Terminal 1 — proxy
npm run proxy

# Terminal 2 — Vite
npm run dev

Open http://localhost:5173.

Free tier note: ProvChart free accounts get a small number of generations per month.

Why this approach works

First paint — charts appear with the CSS; no wait for a chart library.
Bundle size — zero charting JS on the page.
Crawlability — the chart structure is real HTML.
Theme control — st-core tokens + ProvChart theme: "midnight" keep the look consistent.
Security — API key only lives on the proxy / edge function.

You still get dynamic period switching and multi-series visuals; you just don’t pay the usual JavaScript tax.

Repo & next steps

Template: github.com/Figsh/dashboard-project
ProvChart: chart.devtem.org
st-core.fscss: underlying token/chart primitives used by the design system

Fork it, swap the datasets, point the proxy at your own backend.

Lightweight dashboards don’t have to look lightweight. With ProvChart and a solid token set, they can look like a product — and still load like a static page.