Coder Legion - Recent posts tagged software-engineering

Day 2: An Overview of Definition-Driven Architecture

Sat, 31 Oct 2026 01:00:08 +0000

The previous post, Why ERP Needs a Definition-Driven Framework, ended by promising this one would be about which decisions a framework should make on its users' behalf.

The difference between a framework and a library is usually explained as "you call a library, a framework calls you." That describes the mechanism, not the price. The real price is that once control is inverted, the framework's defaults become your defaults. It decides what data looks like, how a request flows, and where the extension points are. You are free to disagree, and the cost of disagreeing is working around it, which is usually more expensive than not using it at all.

So the question to ask about a framework is not what it can do. The more things it can do, the fewer decisions it has made for you, and the same decision has to be made again for every single form.

This post answers the four questions a definition-driven framework has to settle first. Every other design choice is a consequence of these four answers:

Which layer holds the single source of truth
Which layers are derived from it
What is left to the application
Where this whole approach does not hold

None of the four can be left blank. If the framework does not answer them, the people using it will answer them individually, and their answers will not agree. This module puts the truth in an entity, that one puts it in the table, a third puts it in a form configuration file on the front end. A good part of the three long-standing problems from the previous post grows out of exactly that.

1. Which layer holds the single source of truth

First, a distinction that is easy to blur: a Code-First stack is not short of definitions. The API contract is a definition, the migration script is a definition, so is the database's own metadata. The difference is not whether definitions exist. It is which way the arrow points. Those definitions are dumped out as products. Editing them is meaningless, because the next generation run overwrites them.

So the question is not whether to have definition files. It is where the arrow should point, and there are only three candidates:

Candidate	Paradigm	The catch
Code	Code-First	A type system is wide enough to hold any semantics you can think of. The price is that the only reader of the truth is the compiler, and changing the truth means going through compilation, packaging and deployment
The database	Database-First	Data is the final fact, but the type system is too narrow
Definitions	Definition-Driven	Has to satisfy two conditions at once

Where the database is narrow: it knows that total_amount is a decimal(18,4), but not whether that number is an amount, a quantity or an exchange rate. It does not know whether the value should be rounded, how many digits to show, whether it is required, whether it can be edited, or whether it belongs in the query list. Those semantics have nowhere to live, so they end up somewhere else, and the truth has scattered again.

For definitions to be the truth, two conditions, neither optional

Wide enough to carry semantics beyond structure
They are data, not code, which was the whole subject of the previous post

In this framework, what a single field can declare falls into six categories:

Category	What is declared	Can the database hold it?
Structure	Field name, data type, length, default value	✅
Behavior	Required, read only, visible	Partly
Presentation	Caption, control type, display format, width	❌
Semantics	What kind of number this is, which field carries the currency, which field carries the unit of measure	❌
Relationships	The related form, the field mapping, the display field	❌
Security	Data scope roles, sensitive data classification	❌

Code can hold all six, but only as a stack of attributes, and once they are stacked someone still has to parse them back into data before anything can use them.

The width of that declaration is deliberate. It has to cover the UI, the database and validation in one pass.

A single source of truth is not a claim, it is a dependency direction

Any framework can write "this framework treats X as the single source of truth" in its documentation. To verify it, the dependency direction is enough: the layer holding the truth should depend on nobody, and everybody should depend on it.

The definition layer package has an extremely short reference list. It does not know a database exists, or HTTP, or a UI. In the other direction, data access, business logic, the API contract, the client, and every front end all reference it.

That direction buys two things:

The definition layer can be processed by tools on its own. A definition editor, a generator, a batch checking script: each needs to reference only this one package, without dragging in the entire database and API stack. The claim in the previous post that a thousand definitions can be swept by one script rests on this.
AI benefits too. To make it understand a system, feeding it the definitions is more efficient than having it scan the whole code base, and the conclusions are more reliable. The difference is not file size. It is whether relationships are declared or inferred. Ask which forms an order touches, and reading the definitions means reading relationships that were already declared; reading the code means chasing types, chasing queries, chasing configuration, and then inferring intent, and inference gets things wrong.

The tokens saved are a side effect. As for the business judgments that definitions do not cover, the code still has to be read.

2. Which layers are derived from definitions

The previous post used plain language: the UI, the database, CRUD SQL, validation rules. The precise list comes in two kinds. Most of it is generation:

What is generated	From
The single record form layout	The form structure (master block plus detail block)
The list layout	The list fields the form structure declares
The lookup window layout	The queryable fields the form structure declares
The physical table structure	Every table in the form structure
INSERT / UPDATE / DELETE	The form structure's field set
SELECT, including cross table joins	The form structure's fields and declared relationships
The structure of the data container that crosses layers	The form structure's field set

The other kind is evaluation. Here the framework does not build anything; it takes the rules and expressions declared in the definition and evaluates them. Validation rules are evaluated at save and delete. Computed fields are recalculated the moment the user changes a field they depend on.

When a requirement converges into a new shared behavior, a matching mechanism grows. The query criteria screen for a report, for example, is essentially a set of fields plus editors, and a layout definition can describe it.

Grouped by who consumes it:

Form structure definition
│
├─ Generates
│   ├─ Three layouts: single record / list / lookup window  ->  screens on every front end
│   ├─ Physical table structure  ->  the database
│   ├─ Two kinds of SQL: write / query, including cross table joins  ->  data access
│   └─ The cross layer data container structure  ->  transport
│
└─ Evaluates
    ├─ Validation rules  ->  at save and delete
    └─ Computed fields  ->  the moment a dependent field changes

Generating code, or generating objects

Growing something out of a definition at build time and at run time produces completely different results:

A build time generator produces code. When the definition changes, you regenerate, recompile and redeploy. Not one step of the delivery path from the previous post has been removed, and everything saved earlier is handed back at the final gate.
Run time derivation produces objects in memory. Swap the definition, and the next read is the new one. This is exactly where "no compilation, packaging or deployment between changing a definition and having it take effect" holds.

Two things are paid for it:

Compile time type checking is gone. A misspelled field name is not discovered until run time, and how that debt gets repaid is the subject of Data and Definitions Split Apart, So Forms Need No Types of Their Own.
Cache invalidation goes from an optimization to a problem that must be solved. Definitions will be cached for performance, so the framework needs a notification mechanism that carries a definition changed on one node out to every other node.

3. What is left to the application

Where the framework should stop has two rules, one positive and one negative.

The positive one: any behavior that is standardized and repetitive should converge into a framework mechanism.

This is not merely about writing fewer lines:

There is one way to do each thing, and a fix goes in one place, which saves maintenance
New joiners, consultants and whoever inherits the system learn it once, and understanding one form means understanding all of them, which saves training and is the most commonly underestimated of the two

The symptoms of losing control listed in the previous post are all invoices issued by the absence of a general mechanism, and they are reissued per person, per year.

The negative rule is a single test: would every ERP in the world do this the same way? What is the same belongs to the framework. What differs belongs to the application.

By that line the framework stays out of three areas:

Business judgment: how a status transitions, when shipping must be blocked. Every company differs.
SQL for reports and batch jobs: multi table joins, aggregation, dynamic criteria, performance tuning. Forcing these into definitions only makes them more complicated.
The application shell: the login screen, connection settings, navigation menu, windows and theme.

Staying out is not the same as ignoring. Having handed those three over, the framework picks up two obligations it must get right:

A registry that maps a form identifier to the class you wrote
A few extension points at unambiguous positions, so your logic can hook into the save and delete flows

How well those two are done decides directly whether the application is pleasant or awkward to write, and a later chapter takes them apart.

Handing work over does not reduce complexity either. What definitions can absorb is repetition, not the inherent difficulty of the business. The hard parts have not disappeared. They have been concentrated in one place and given to whoever knows why they are hard.

The third area is clearest in the case study. The Northwind front end has one UI project shared by every front end, and only four hand written screens: the main window, login, connection settings, and the layer that hosts forms. All four are shell. Not one of the eight business forms needs screen code.

You can see where the line is drawn at a glance: the hand written part is precisely the part definitions did not take over.

4. Architectural patterns: three sources, one piece from each

With the first three questions answered, the shape of the framework is fixed. Before the fourth, one description that gets applied to it deserves handling: a hybrid of N-Tier, Clean Architecture and MVVM. "Hybrid" turns into a decorative word very easily, so here is what was taken and what was not.

Source	What was taken	What was not
N-Tier	The layer boundaries and the cross layer data container. One request travels front end, API, business object, repository, data access infrastructure, database, and each layer knows only its neighbor. The most completely taken of the three	Nothing
Clean Architecture	The inward dependency direction from the first section, verifiable by opening the project reference lists	Its requirement on the domain. The pure approach gives every business concept its own strongly typed domain entity, and modeling a thousand tables one at a time is the scale cost from the previous post itself
MVVM	All three roles are present. The model is the cross layer data container, the view is rendered from the layout definition, and the view model is a framework provided `DataObject`	Nothing; the only difference is who writes the view model

That DataObject carries the full job: wrapping the data container, exposing field reads and writes for screen binding, raising notifications for value changes and unsaved state, managing the start and cancellation of detail row editing, and loading, saving and deleting against the back end. The difference is that it is not hand written per form. Its shape comes from the definition. The price is that the screen binds not to a property like order.CustomerId but to a field name.

The cross layer data container is not a custom type either, and a later post is devoted to what that choice costs.

5. Systems this should not be applied to

Three questions before you commit. If any one of them does not survive, seriously consider not applying this.

Q1: Are your screens standardized forms?
Definitions can drive layout only because the layout patterns converged, and what converged here is a block of master fields plus a detail block. Dashboards, charts, drag and drop scheduling and maps are not in the pattern. The framework's answer is not "we support those too." It is that this road does not go through definitions, and you write it yourself.

Q2: Is your request a single document?
The cross layer data container is self describing, and the price is that every response carries the field structure with it. For one order that is worth it. For an event stream at tens of thousands per second it is not.

Q3: How long does your system have to live?
The cost is concentrated up front and the return does not show for three to five years. A system that ships, closes and gets rewritten wholesale three years later pays the cost and never collects.

Not many scenarios satisfy all three at once, and internal enterprise management systems happen to hit every one. Finance, purchasing, inventory, sales, HR, customer relations: the screens are almost all standardized forms, the requests are almost all documents, and once it goes live it lives a long time.

These boundaries are not "the framework has not got there yet." They are places where getting there would make things worse. A framework that can do anything has made no decisions for you, and the decisions are exactly where a framework's value is.

The Northwind case study: how much one definition does

The order in the case study has a single form structure definition, and it states:

One master with one detail
Three cross table relationships on the master, to customer, employee and shipper, each a lookup window that brings one record back
Each detail row picks a product
Amounts are not typed by anyone. They are calculated from quantity, unit price and discount
Three validation rules have to pass before saving

Once that one definition is written, everything in the table from the second section is in place. The tables are created from it, the list and lookup window layouts grow directly out of it, the write and query SQL is assembled by the framework, the data carries its structure with it across layers, and the validation rules are checked at the moment of saving. Not one of those requires additional code.

The same definition is read by four front ends: desktop, browser, iOS and Android, all sharing one UI project.

The value of a framework is not how much it can do. It is whether it can tell which decisions belong to it: the thing that is identical a thousand times over is its, and the thing that differs every time is yours. Draw that line crooked and both sides carry the weight. Draw it too conservatively and you redo the same work a thousand times. Draw it too greedily and you spend more effort working around it.

The next post, The Northwind Case Study: Eight Forms and a Custom Business Object, brings the case study on stage.

📚 Part of the thirty-post series ERP Architect's Notes: Designing a Definition-Driven Framework

📘 Original article on HackMD, in Traditional Chinese:
👉 https://hackmd.io/@jeff377/day-02-framework-decisions

📢 You are welcome to republish this, please credit the source.
📬 Follow for more engineering notes:
Facebook ｜ HackMD ｜ GitHub ｜ NuGet

How to Handle Duplicate Files Without Losing the Record That Matters

Fri, 30 Oct 2026 07:14:35 +0000

Every office has the same quiet problem. A client sends a contract on WhatsApp, someone downloads it, and the same file arrives again by email the next morning. A colleague then saves a "final" copy to their desktop. Within a week, one document has become five, and nobody is sure which one is real.

Cleaning this up sounds simple. Delete the extras and move on. But if you delete the wrong copy, you may lose a signed version, a date stamp, or the proof of who sent what. This guide explains how to deal with duplicate files safely, and how good Document Management Software in India can make the whole job easier.

Why Duplicate Files Pile Up So Quickly

Duplicates rarely come from carelessness. They come from the way work actually happens.

Documents reach a team from many directions, such as chat groups, inboxes, scanners and shared drives. Each channel creates its own copy. People also save files "just in case," rename them to suit themselves, and forward attachments instead of sharing links. Nobody means to create clutter, yet the clutter grows anyway.

The cost is real. Staff waste time hunting for the right file, and they sometimes act on an outdated one. In a law, accounting or compliance setting, working from the wrong version can lead to a missed clause, a wrong figure or an awkward conversation with a client.

Not All Duplicates Are the Same

Before you delete anything, understand what kind of duplicate you are looking at. They fall into three broad groups.

Exact copies

These are identical files. The content, size and date are the same, and only the location or name differs. They are the safest to clean up, but even here you should confirm that the file is really identical, not just similarly named.

Near duplicates and versions

This group causes most of the trouble. Two files look alike, but one has an extra paragraph, a corrected amount or a new signature page. Neither is a true copy of the other. Deleting the older one may erase the history of how the document changed, which matters a great deal in contracts, board resolutions and tax records.

The same document from different channels

A client may send one agreement through WhatsApp and again by email. The content may match, but the two arrivals tell you different things. One shows when the client first sent it, and the other shows when it reached the formal inbox. That origin and timing can become important evidence later.

How to Spot the Record That Matters

The "record that matters" is not always the newest file, and it is not always the biggest one. It is the version you could rely on if somebody asked you to prove it tomorrow.

Ask yourself these questions:

Is this the signed, stamped or approved version?
Does it have a clear date, sender and source?
Is it the last version that went to the client, the auditor or the regulator?
Can I show how it differs from earlier drafts?

If you can answer yes to most of these, you have likely found your master record. Everything else is either a duplicate to retire or a version to preserve as history.

A Simple Way to Clean Up Duplicate Files

You do not need a huge project to start. Work through these steps in order, and you will avoid most of the common mistakes.

Pick one place to begin. Choose a single client folder or a single month. Trying to fix the whole archive at once usually ends in frustration.
Group files that look related. Put together anything with the same document type, party and date range.
Compare, don't assume. Open the files side by side. Check dates, amounts, names and signatures, not just filenames.
Mark the master record. Decide which file is the reliable one, and note why you chose it.
Keep the history, retire the noise. Exact copies can be removed. Real versions should be kept with their dates and authors.
Record what you did. A short note on who cleaned up what, and when, saves arguments later.
Set a rule for next time. Decide where new files go, so the same mess does not return.

Doing this by hand works for a small folder. For a larger archive, however, it becomes slow, tiring and easy to get wrong.

Why Deleting Is the Riskiest Move

Deleting feels tidy, but it is the one step you cannot take back. Neither a recycle bin nor a backup is a safe plan if you did not know what you were removing in the first place.

There are three common ways deletion goes wrong:

You remove the only signed copy and keep an unsigned draft.
You lose the provenance, meaning the proof of who sent the file and when.
You break a legal or audit requirement, because some records must be kept for a set period, or kept untouched if they are under review.

A safer habit is to retire duplicates, not destroy them. Move them out of daily view, keep a record of the action, and delete only when your retention rules clearly allow it.

What to Look For in Document Management Software in India

Good Document Management Software in India does more than store files. It helps you stay in control when documents arrive from many places and must be produced again years later. When you compare tools, look for these features.

Version history with restore. Every revision should be kept with its author and timestamp, so you can go back to any earlier state.
Visual comparison. You should be able to see exactly what changed between two versions, without reading both line by line.
An access and audit log. Every view, edit, share and deletion should be recorded, ideally in a way that cannot be altered.
Check-in and check-out. This stops two people from quietly overwriting each other's work.
Provenance tracking. The tool should remember which channel a file came from, who sent it and when.
Retention rules and legal hold. You should be able to set how long records stay, and freeze anything under scrutiny.
Per-file permissions. Not everyone needs to see everything, and clients should only see what you share with them.
Data control. Since many Indian firms care about where their data lives, check whether the tool supports your region, your policy and India's data protection rules.

Whether you run a law practice, a CA firm or a compliance team, these basics matter more than a long list of flashy extras.

How IntelloDocs Approaches the Problem

IntelloDocs is an AI-native document vault built by Imbibe Technologies in Karnal. It takes a slightly different route from older systems. Instead of asking people to sort files after the fact, it deals with the mess at the point of arrival.

Files can be captured from WhatsApp, email attachments, office scanners and direct uploads. As each file comes in, the system keeps its origin, sender and timestamp attached to it. That means you are not only storing a document but also keeping the story of how it reached you.

The AI then reads the content of the document instead of relying on the filename. It picks out the kind of record, the parties, dates and amounts, and suggests where the file should go. A person confirms it with one tap, so nothing is filed silently.

Several other features help when duplicates and versions pile up:

Versioning and visual diffs show how a document changed over time.
Check-in and check-out prevent accidental overwrites.
An immutable access log records every open, edit, share and deletion.
Retention rules and legal hold protect records that must not be touched.
Cited answers let you ask a question in plain language and see the exact sentence the answer came from.

IntelloDocs is currently in Phase 1 early access, so it is worth speaking with the team about your own setup and archive before you decide.

Habits That Keep Duplicates From Coming Back

Even the best software works better with a few simple team habits.

Share links, not attachments, whenever you can.
Name files clearly, or let the system name them for you.
Agree on one "home" for each client or matter, so nobody wonders where a file belongs.
Review your archive on a schedule, not only when it becomes a crisis.
Treat provenance as part of the record. A file without a source is a weaker file.

These small changes cost very little, but they save hours over a year.

Frequently Asked Questions

1. Is it safe to delete duplicate files?

It is safe only when you are sure the files are exact copies and no retention or legal rule applies. Near duplicates and different versions should be kept, because they show how a document changed. The safest approach is to retire duplicates first and delete them later, with a clear record of what was removed.

2. How does Document Management Software in India help with duplicate files?

Document Management Software in India gives you version history, audit logs, permissions and retention rules in one place. Instead of guessing which copy is right, you can see who changed what and when, restore earlier versions, and keep every action on record. Tools that capture the source and timestamp of each file make it even easier to find the record that matters.

3. What is the difference between a duplicate file and a document version?

A duplicate is an identical copy of a file. A version is a changed copy, such as a draft with edits or a final with a signature added. Duplicates can usually be removed, but versions should be kept and labelled, since they form the history of the document.

Final Thoughts

Duplicate files are not just a storage problem. They are a trust problem, because a team that cannot tell which file is real cannot fully rely on its own records. The answer is neither to hoard everything nor to delete in a hurry. Instead, identify the record that matters, keep its history and source, and retire the rest with care.

If your team handles contracts, tax papers or client records, a well-built document management system will protect you far better than a folder full of renamed copies. Take a look at IntelloDocs at intellodocs to see how it handles capture, versioning and cited search in one place.

The Ultimate Guide to Online Education Platforms with Analytics for Coaching Institutes

Fri, 30 Oct 2026 05:33:49 +0000

The way in which coaching institutes teach manage students. Measure academic progress is changing fast. Today’s learners expect flexibility, timely feedback and access to learning resources that go beyond the classroom. At the time coaching institutes have to manage batches, attendance, performance and show meaningful learning outcomes.

This can become a challenge to handle all these responsibilities manually especially when an institute has courses, hundreds of students and growing faculty team. This is where online education platforms with analytics can really make a difference.

These platforms combine learning management and performance tracking which enables coaching institutes to make data driven decisions boost student engagement and streamline operations. With so many solutions available how does an institute make the right choice?

Knowing the features, benefits and selection criteria can help you identify the best online education platform for coaching institutes with analytics and build a more effective data-driven learning environment.

What is an online education platform with analytics?

An online education platform is a learning environment where teachers are able to deliver courses, share study materials, assess learners and interact with students. If the platform has analytic capabilities it can also display useful data on student participation, academic performance and learning progress.

Coaching institutes can move beyond attendance registers, spreadsheets or periodic test results to a centralized platform to get insights on how students are interacting with learning content and where they need additional support.

For instance an institution could observe that multiple students are struggling with the subject or that attendance declines at a particular point in a course. Learning analytics can assist educators in detecting patterns and exploring the reasons for them.

The aim is not just to gather data. Its purpose is to transform information into actionable decisions that enhance teaching and learning.

Why Learning Analytics Should Be Adopted By Coaching Institutes?

All coaching institutes want their students to do well stay engaged and finish their courses successfully. Without reliable information it is difficult to find the reasons for the inconsistent performance.

Analytics helps fill this gap shedding light on student activity and academic progress. When teachers are able to identify knowledge gap they can step in before small challenges become large learning barriers.

More Insight into Student Performance

Test scores are a measure of academic progress but they don’t always tell the story behind why a student is struggling. A student might understand the ideas. Make mistakes under pressure of time. Another may perpetually miss questions from a given topic because the underlying fundamentals are murky.

Assessment analytics in a platform help the educators to look at the performance across the subjects, topics, tests and batches. This helps to identify problems that recur and change the teaching approach accordingly.

Enhance Student Engagement

You can also use attendance, assignment completion, participation and learning activity as measures of engagement. When all of these indicators are in one place faculty can identify students who may be falling behind.

If participation drops suddenly for instance a teacher can figure out if a student is having trouble or needs more support.

Analytics can’t account for every change in behavior but it can help educators to recognize when a conversation or intervention might be appropriate.

Smarter College Choices

Coaching institutes decide batch allocation, revision schedules, faculty support, extra practice sessions regularly. Without information these decisions can be highly assumption-based or isolated observations.

Analytics provide a structured basis for assessing what’s working and where there’s scope for improvement. Over time institutes may utilize this insight to improve their processes and provide a more consistent learning experience.

Important Features to Consider in an Online Education Platform

The choice of an education platform is not simply a matter of comparing the cost of subscriptions or verifying that online classes are supported. The right solution should align with the institute’s teaching model, operational needs and long-term growth plans.

Complete Learning Management

A good platform should allow you to organize courses upload study materials, schedule classes and manage learning resources. Students should be able to access content without having to wade through complex systems.

Course and batch management is very important for coaching institutes offering programs or preparation for competitive examinations. A good learning environment keeps students on track. Reduces repetitive administrative work for faculty members.

Analytics of Performance and Assessment

Assessment tools should consist of tests, quizzes, assignments and performance tracking. Ideally the educator should be able to see the results of a student as well as trends across batches and subjects.

Useful analytics can include test scores, topic-wise accuracy, assessment completion and changes in performance over time. Specific capabilities vary by platform and institutes should assess whether the reports suit their academic needs.

The best online education platform for coaching institutes with analytics should make these insights easy to understand and act upon without teachers with complex dashboards.

Student Support and Communication

Regular communication keeps students informed of classes, assignments, test schedules and academic expectations. Institutes should seek features that allow sharing of updates and timely provision of guidance.

These capabilities may include notifications, announcements, discussion spaces or other communication tools depending on the platform. The conclusion is that the right info is readily available to students and teachers without a lot of hassles.

Data Security and Access Control

Education platforms will store student information, assessment records and sometimes payment or contact details. This information should therefore be a consideration in platform selection and protected.

Institutes should consider access rights, authentication, data protection standards, backup arrangements and the provider’s privacy position. They should also have internal procedures for determining who has access to student records and reports.

A feature-rich platform is only valuable if used with the security and privacy practices.

The Role of Analytics in Improving Teaching and Learning Outcomes

The true meaning of analytics is when teachers use data to enhance the learning process.

Suppose there is a coaching institute which conducts a mathematics test The overall batch average seems satisfactory but if we look at topic wise results many students are finding algebra difficult. The faculty team can instead go back. Revisit the concept provide additional practice and do a follow-up assessment instead of just moving on to the next chapter.

Analytics may also reveal that students who always do the practice assignments tend to perform on the mock tests. Faculty can use this observation to motivate practice and identify students who may require additional support.

Analytics can also help institutes understand how successful their teaching methods are. When students keep having problems with a topic over batches it might be worthwhile to examine the teaching methodology learning resources or the test format.

These examples show how data can lead to academic decision making. Analytics should support teachers’ professional judgments, not replace them. The context of the student of interactions in the classroom and of individual learning needs are critical to understanding performance.

Best Online Education Platforms in India for Coaching Institutes – How to Choose?

The Indian coaching industry caters to a set of learners from school students preparing for board examinations to aspirants preparing for competitive entrance and government recruitment examinations. Thus the need of one institute may be quite different from those of another institute.

While judging the best online education platforms in India for coaching institutes know what problems your institute needs to solve first.

An institute focused on classes would put emphasis on video delivery, scheduling and interactive learning tools. A test-prep center might focus more on question banks, practice tests, automated scoring and detailed performance reports. If you’re an organization with multiple branches you may require centralized administration, role-based access and consolidated reporting.

Scalability is another factor. A good platform for a batch should ideally be able to handle more students, courses and faculty without creating unnecessary operational complexity.

Ease of use is also important. A sophisticated system will be of value if it is difficult for teachers to manage or for students to navigate. Institutions should consider the student and faculty experience before committing term.

Pricing should be taken in context well. Check subscription costs, implementation requirements, support arrangements and additional charges for some functionalities. Of going for the cheapest option look for a platform that provides the best mix of features ease of use and long-term benefits.

Finally consider the level of customer support and the provider’s ability to help your team onboard to the system. A smooth implementation process can make a difference to the platform’s overall effectiveness.

Mistakes to Avoid When Selecting an Education Platform

A mistake is to pick a platform based on the number of features it has. More features don’t automatically mean results. Institutes should concentrate on the capabilities that're directly responsive, to their teaching and administrative needs.

Another mistake is to ignore how useful analytics can be. A dashboard can show a lot of information. The institute needs to be able to understand the data and use it to make good decisions. Before you pick a platform look at sample reports. Check if they answer the questions that your faculty members usually ask.

Institutes also must not use technology in place of teaching. Even though an education platform can help learning by being organized easy to access and giving insights the main thing that helps students grow is teaching, regular practice, proper feedback and real interaction between teachers and students.

Finally don't forget about how to put the system into place and train people. Teachers need time to learn the system set up their ways of working and use the features in their teaching. Letting teachers be part of choosing the platform can help them use it better and make sure the platform fits what happens in classrooms.

The Future of Coaching Centers: Data-Driven Learning

As digital education keeps changing analytics is expected to become more important in how coaching institutes understand what students need and provide help.

Newer systems could give detailed learning insights, personalized suggestions and automatic reports. Artificial intelligence could also help teachers see patterns in a lot of test data. The quality of the insights would depend on the quality of the data and how well the tools are used.

Institutes should also have a way of doing things. Students’ data needs to be treated, automated suggestions should be checked and teachers should stay involved in important academic choices.

The goal is not to make education fully based on data. It is to mix the data with human understanding so that every student gets better guidance and more chances to improve.

Transforming Learning Data into Improved Student Outcomes

For any coaching institute that wants to improve its efficiency, student involvement and make academic choices picking an online education platform with analytics is an important choice. The best solution includes learning materials, tests, tracking progress and helpful information to support both teachers and students.

Think about what your institute wants before making any decision. Compare the features ways to report how easy it is to grow, security, how easy it is to use and support. Important make sure the platform helps your teachers take information and use it to make teaching better.

ItLearn helps coaching institutes move to an online education setup to manage teaching and learning. If your institute is ready to make academic work easier improve the learning experience and find better ways to track student progress contact ItLearn now. Find out how the right education technology can help your institute build an organized, interesting and ready, for the future learning experience.

ERP Architect's Notes: Designing a Definition-Driven Framework · Series Index

Fri, 30 Oct 2026 01:23:04 +0000

The complexity of an ERP is not in how hard any single feature is. It is in how many times the same fact gets restated, multiplied by how many tables, multiplied again by how many customizations.

A thirty-post series written for iThome Ironman 2026, a thirty-day technical writing challenge in Taiwan, published from 2026-08-17 to 2026-09-15. The series is complete.

This series is not about building one form with a framework. It is about designing a framework that can drive hundreds of forms: the single source of truth lives in definitions rather than code, and the database structure, SQL, screens for multiple front ends, validation and computation, multiple languages, multi-tenancy and customization are all derived from the same layer of definitions. The Northwind case study verifies it, but the case study is evidence, not the subject.

It is about mechanisms and trade-offs, not a usage tutorial, and the cost of every design is written as plainly as its benefits.

Chapter One · Why Definition-Driven

What the problem is, where the paradigms diverge, and the case study's first appearance. This chapter sets side by side how many places one field touches under each paradigm.

Day 1 · Why ERP Needs a Definition-Driven Framework
Day 2 · An Overview of Definition-Driven Architecture
Day 3 · The Northwind Case Study: Eight Forms and a Custom Business Object

Chapter Two · Definition Layer

What kinds of definition files there are, why their lifecycles differ, and how one structural description grows into the database structure, CRUD SQL, screens for multiple front ends and business rules.

Day 4 · The Purpose and Classification of Definition Files
Day 5 · How FormSchema Drives SQL
Day 6 · TableSchema and Database Structure Upgrades
Day 7 · FormLayout and Screen Generation for Multiple Front Ends
Day 8 · Declarative Business Rules and Computed Fields

Chapter Three · Data Access and Caching

Without writing DTOs, how data crosses layers; the only two things multiple databases have in common; and how a cache knows its source has changed.

Day 9 · Data and Definitions Split Apart, So Forms Need No Types of Their Own
Day 10 · Multi-Database Support: CRUD and Structure Upgrades
Day 11 · Connections, Transactions and Query Conditions in the Data Access Layer
Day 12 · Cross-Node Invalidation for Definition and Data Caches

Chapter Four · Business Logic and API

Who takes over the part definitions cannot express, where the extension points open, and what a call passes through on its way from the caller to the business object.

Day 13 · Session Management and Lifecycle
Day 14 · Extension Points on a Business Object and When to Override Them
Day 15 · API Contract Layering and the Type Registry
Day 16 · JSON-RPC 2.0: Protocol Choice and the Request Pipeline
Day 17 · The Connector and the Convergence of Client Responsibilities
Day 18 · The Error Contract and User Visible Messages

Chapter Five · Multi-Tenancy and Customization

How one codebase lets every company look different without branching and without recompiling or redeploying. Where the customization work currently stops is stated plainly too.

Day 19 · The Scope of the Customization Layer and Overlay Granularity
Day 20 · Multilingual Customization and Resource Structure
Day 21 · Business Logic Customization and the Four Plugin Stages

Chapter Six · Transport, Security and Audit

What data looks like when it is sent, which protections it passes through, who is allowed to call, and what can be looked up afterward.

Day 22 · The Division of Labor Between XML, JSON and MessagePack
Day 23 · The API Payload Security Pipeline: Order, Protection Levels and Keys
Day 24 · The Three Axes of Permissions: Action, Row and Column
Day 25 · The Audit Trail and the Before and After Values of a Change

Chapter Seven · Data Semantics

Amounts, quantities, dates and times of day: how the meaning of these values is kept from the definition all the way to the database, without any layer quietly changing it along the way.

Day 26 · Numeric Semantics, Multiple Currencies and Units of Measure
Day 27 · The Semantic Boundaries Between Date, DateTime and Time
Day 28 · Time Zone Conversion of DateTime

Chapter Eight · Case Review and The Big Picture

The Northwind application examined item by item: which part definitions, the framework and application code each own. Then a single big picture, and the systems this design does not suit.

Day 29 · Which Part Definitions, the Framework and Application Code Each Own
Day 30 · Three Depths of Development and the Big Picture of the Framework

About the framework behind this series

The mechanisms discussed in the series are all built into an open source framework, bee-library. The case study is a separate repository, bee-northwind-avalonia, which pulls the framework in as a package and holds only the application's own definition files and code.

Update: Bee.NET (bee-library), the framework used when this series was written, has since been taken over by Polhem. The mechanisms carry over, the framework has been substantially refactored, and it is open source under the MIT license. The bee-library and bee-northwind-avalonia repositories are archived; type names and code in the posts follow bee-library as it was at the time of writing.

The framework's only role in this series is to show that these ideas are not just talk on paper. You can skip it entirely, take away only the idea of letting definitions be the source, and build that idea in the languages and tools you know.

📘 Original index on HackMD, in Traditional Chinese:
👉 https://hackmd.io/@jeff377/ironman-2026

📢 You are welcome to republish this, please credit the source.
📬 Follow for more engineering notes:
Facebook ｜ HackMD ｜ GitHub ｜ NuGet

Day 1: Why ERP Needs a Definition-Driven Framework

Fri, 30 Oct 2026 01:23:02 +0000

This is the first post in a thirty-part series, ERP Architect's Notes: Designing a Definition-Driven Framework.

What follows is not about building one form with some framework. It is about designing a framework that can drive hundreds of forms: which decisions it should make on your behalf, which ones it should leave to you, and why each mechanism ended up shaped the way it is. Every claim is checked against a working application built on the Northwind sample database, but the case study is evidence, not the subject.

This post covers:

The three layers of ERP development cost: a single change, scale, and delivering customization
The three long-standing problems those layers converge on, and why tooling does not remove them
The point where the paradigms diverge: is the single source of truth code, or definitions?
A controlled experiment: how many places does one field touch under each paradigm?

Who I am

I have spent more than twenty years as a software architect in packaged ERP and HRM products. Packaged software has a very long product cycle, ten or twenty years is normal, and over that span the system is not merely maintained. It has to grow a new face every time the front-end generation turns over. I have been through three of them: desktop, web, and mobile apps. Each time the question was the same. Do we rebuild all several hundred screens?

On a product line like that, an architectural decision does not get rewritten by someone else three years later. You watch it grow for ten years and then keep maintaining it. I stayed in the same role the whole time. I never moved into managing projects or teams and left the code behind, so I did not miss the consequences of a single one of those decisions.

About the framework, and about this series

Some of the ideas I accumulated over those years were built inside real projects. More of them were only ever thought through, because writing all of it alone was never worth the cost. That cost is different now. I used Claude Code to build the ones I never got to, and the result is an open source framework, bee-library. The Northwind case study in later posts runs on it.

The case study itself lives in a separate repository, bee-northwind-avalonia. The framework repository holds the mechanisms; the case study repository holds only the application's own definition files and code, with the framework pulled in as a package. If you want to know what applying this framework actually requires, which definition files you prepare and what code you still have to write, the case study repository shows it more clearly than the framework does.

The framework is still moving. Things that deserve to become mechanisms will keep being folded in, interfaces will change with them, and breaking changes will happen. If you want to reference the packages directly today, that has to be part of your calculation. You can also clone the whole thing and shape it into what you need.

This series is not a manual for it. The framework has exactly one job here: to show that these ideas are not just talk. What the thirty posts discuss is the mechanisms that already exist, what problem each one ran into, how it was solved, what it cost, and why that option rather than another. Implementation details are not walked through line by line. You are entirely free not to use it, and to take only the idea that definitions should be the source, and build it in whatever language and tools you already know.

The whole series rests on one sentence:

The complexity of an ERP is not in how hard any single feature is. It is in how many times the same fact gets restated, multiplied by how many tables, multiplied again by how many customizations.

1. The cost of a single change: how many places one field lands in

The requirement: orders need one more field, Required Date. No calculation, no dependent behavior, no effect on existing logic. The user says this is "just one field, it should be easy," and the user is right.

In a typical Code-First stack, ORM plus web API plus front end, that field lands in thirteen places:

#	Location	What changes
1	Entity class	Add a property
2	Type mapping configuration	Precision, nullability, column name mapping
3	Migration	Generate it, review it, commit it, run it at deployment
4	Query projections	The `Select` for list queries, the column list for reports
5	DTOs	Once for the request, once for the response
6	Mapper configuration	Entity to DTO
7	Validation	Required, range, ordering relative to the order date
8	API contract	The OpenAPI schema, republished to consumers
9	Front-end types	A TypeScript interface or a C# model
10	Front-end form	Control, label, position in the layout
11	Front-end list	Column, width, sorting
12	Localization resources	One per language
13	Test data	Fixtures, seeds, fake data generators

To be fair, several of those can be generated. Migrations come from a tool, OpenAPI is produced by scanning assemblies, front-end types can be generated from OpenAPI. The number you actually hand write is closer to five or six.

But the number you hand write is not the point:

The single fact that "Required Date is a date, it is required, and it shows after the order date" has been restated thirteen times in this stack.

Every restatement is a place where the copies can disagree:

Forgot the DTO, and the value reaches the database but never the front end
Forgot the validation, and users submit it empty
Forgot the list column, and users ask why they cannot see it on the query screen

None of these are hard problems. They are counting problems, and each inconsistency has to be found by a test or by a user report.

A team boundary runs straight through that table

Items 1 through 8 are back end. Items 9 through 13 are front end. If those are two teams, the line cuts through the middle of the table, and every item that crosses it changes character. It stops being editing and becomes coordination.

A request to add one field then grows into this:

A meeting to agree the contract first, then separate scheduling on each side
Sequencing: until the back end ships there is nothing for the front end to consume, so starting early means hand rolling a mock
Release windows: whichever side goes first has to cope with the other not having caught up
And during integration, the discovery that one side read "required" as a front end block and the other read it as a back end block

Not one of those activities is writing the field. What they consume is not effort, it is waiting. The user counts the time from asking to shipping; the engineer counts the time actually spent working. So one side asks why changing a single field takes this long, and the other side does not feel it was slacking. Both are right, because the cost is real, it just does not land on the part where anyone is typing.

2. The cost of scale: multiply that table by a thousand

A full packaged ERP covers finance, purchasing, inventory, manufacturing, sales and HR. Business tables run into the thousands.

The founding premise of an ORM is that objects map to tables, so:

thousands of business tables
  -> thousands of entity classes
  -> plus DTOs, usually more than one: List / Detail / Create / Update
  -> plus mapper configuration
  -> plus repositories

The overwhelming majority of those classes resemble each other. The entity for Supplier and the entity for Shipper differ in field names and field count; structurally they came from the same mold.

This is not an argument that ORMs are bad. An ORM works well in the setting it was designed for. The problem is quantity. When the same pattern has to be copied a few thousand times, the copying itself becomes the dominant cost.

Relationships then multiply that quantity again:

ERP tables are densely connected by nature. An order points at a customer, a salesperson and a shipper; a shipment points at an order, a warehouse and a lot number.
Every one of those relationships needs someone to decide how it is expressed, how it is loaded, and who maintains it. Each decision is reasonable on its own. Spread a few thousand tables out at once and nobody can hold the whole picture.

Take the thirteen slots from the first section and multiply them by a few thousand, and multiply the team boundary along with them. Cross team coordination is not paid once per table, it is paid once per request, and the number of requests is itself proportional to the number of tables.

This is not inconvenience. It is loss of control. Anyone who has maintained a large system recognizes the symptoms:

Nobody dares touch shared classes, because nobody is sure who depends on them
New joiners take a long time before they will go near core modules
The same concept has several different spellings across modules, because different people grew them at different times

3. The cost of customization: how do you deliver it once it is done

The first two layers are about how much work a change takes. The third one is more fundamental, and there is no way around it, because:

An ERP that gets installed and simply used essentially does not exist.

Every rollout comes with customization. This client numbers documents differently, that one calculates cost differently, this department wants two extra fields, that report needs a different grouping. The only variable is the path the customization travels down.

Under Code-First, all thirteen of those places are code, and code has exactly one delivery path:

edit -> compile -> test -> package -> deploy

The same path hurts in three different ways depending on the deployment shape:

Deployment shape	Where customization hurts
A single organization's internal system	The simplest case: schedule one maintenance window. But changing a decimal scale from 2 to 4 still means a full build, test and deploy
Packaged software sold to many customers, each with their own deployment	Only two options. Put it in the trunk behind a switch, and the switches accumulate until nobody dares delete any of them. Or branch, and every branch evolves on its own while every fix in the trunk has to be carried across one at a time
Multi-tenant SaaS on shared assemblies	Tenant A changes one formula, and that means coordinating a maintenance window across all tenants and regression testing every tenant's functionality. If any one tenant breaks, the whole batch rolls back

Three shapes, one conclusion:

The size of the requirement and the cost of delivering it have come completely apart.

Whether the requirement is changing a decimal scale from 2 to 4 or reworking an entire module, it travels the same delivery path and carries the same deployment risk. The result is that small requirements have to be scheduled into the release rhythm too, and the engineering team is forced to treat every change as a significant event.

The root of that decoupling is that customization was expressed as code. The deployment shape only decides how much it hurts. It does not decide whether it hurts.

This layer is what definition-driven design is actually for:

Definitions are data, not code. Changing data requires neither recompilation nor redeployment of assemblies.

4. Three problems that never got solved

The costs from the first three sections converge into these:

Long-standing problem	Symptom
Specifications scattered across three layers	The same field is defined once in the UI, once in the DTO and once in a database migration, and they drift apart easily
Business logic scattered	Different modules are maintained by different engineers, styles diverge, and the same work gets done more than once
Customization that resists standardization	Custom logic cannot flow back into the standard product, and accumulates as technical debt nobody can govern

For the past twenty years the usual answer has been more tooling: code generators, project templates, smarter mappers, more automatic migrations. Those tools genuinely made each restatement cheaper.

What they did not change is the structure in which one fact gets restated thirteen times. They only made restating it faster.

For the third layer, customization cost, tooling helps even less. Generated code is still code, and still has to be compiled and deployed. The reason the third problem has been open for twenty years is right here.

5. Where the paradigms diverge

The problem is not the tooling. It is the paradigm. There is only one question to answer:

Is the single source of truth in this system the code, or the definitions?

Code-First: the entity class is the source. The database is derived from it through migrations, the API contract is derived from it through OpenAPI, and front-end types are derived from that. Definition files are a by-product.
Definition-Driven: a structural description is the source. The UI, the database, the CRUD SQL and the validation rules are all derived from it. Code appears only where a real business judgment has to be made.

The difference is not that a file extension changed from .cs to .xml. It is the nature of the source:

	Code is the source	Definitions are the source
How a thousand tables are represented	A thousand classes	A thousand definition records
What you can bring to bear on them	Compilers, IDEs, refactoring tools	All of that, plus batch scripts, generators, programmatic checks, and AI
The delivery path for a change	Compile, package, deploy	Swap in a different definition
How customization is represented	Branches or switches scattered through source	A layer of data stacked on top of the standard definition
How the front end learns which fields exist	A second, hand-written set of types kept in sync by contract and by people	Fetching the same definition at run time
What a new front-end generation costs	Rebuilding every screen	Writing one new rendering layer
Who can change it	Developers	Developers, consultants, tools, and even users

Two of those rows deserve more than a line:

A thousand classes and a thousand definition records are the same count, but managing them is not the same order of magnitude. A thousand structurally identical XML files can be swept by one script that checks whether every amount field has the right decimal scale. That takes minutes. Doing the equivalent check across a thousand classes is considerably harder.
Generational cost is the one I feel most strongly. Front-end technology turns over roughly every ten years. If screens are written by hand one at a time, a generation change means rebuilding all of them. If screens grow out of definitions, a generation change means writing one new rendering layer. The more forms you have, the wider the gap.

That is also the reason layout deserves to be its own layer of definition. Structure is stable and follows the business; layout is volatile and follows the front-end generation. Once they are separated, what gets rewritten at generation change is the rendering layer, not the thousands of structural definitions.

6. A controlled experiment: how many places for the same field

Back to adding Required Date to orders. I counted it in this series' case study application, and the answer is three definition edits and zero lines of code.

Table structure:

<DbField FieldName="required_date" Caption="Required Date" DbType="Date" />

Form structure:

<FormField FieldName="required_date" Caption="Required Date" DbType="Date" />

Layout:

<LayoutField FieldName="required_date" Caption="Required Date" />

What follows automatically after that: the database column, the form control, the list column, the SQL for insert, update and delete, and transport across layers.

No entity class to keep in sync, because the SQL is assembled from the definition at run time
No DTO or mapper to keep in sync, because the data crossing layers carries its own structural description

The count moves by one in either direction depending on the situation, but the count is not the point. The point is that all three places are definition data.

What happened to the team boundary

The front end no longer declares the field a second time. It asks the back end for that definition at run time and renders accordingly:

front end starts -> asks the back end for the definition -> grows controls, list columns and validation from it

The contract does not have to be negotiated, because the contract is that definition. There is no second copy that could disagree with it.
There is nothing to wait for. The back end changes the definition, and the front end gets the new one the next time it asks.
The coordination, scheduling, sequencing and release windows from the first section mostly just disappear.

What is left is half of the first step: what this field is called in business terms, where it goes, and whether it is required. Somebody was always going to decide that. It is a business judgment, not communication overhead.

The boundary has not vanished, though. It moved.

The front end still needs a rendering layer that knows how to read definitions, and somebody has to build it. The difference is that it gets built once, and after that every form and every field is free.
The team boundary drops from "every field" down to "the rendering layer," and the frequency of conversation changes from "every request" to "whenever framework capability changes."
Anything that genuinely cannot be declared as a definition still has to be discussed. A rule about when a document may not be shipped still needs someone to decide who blocks it, at which layer, and what the message says. No architecture escapes that. What definition-driven design can eliminate is structural restatement, not semantic negotiation.

After the change lands

Two practical problems remain, and the framework has to absorb both:

The physical database does not have that column yet. The framework compares the definition against the actual structure and adds the column, so nobody hand writes a migration. A later post takes this apart.
The system is running, so how does it know the definition changed? The node that edited the definition has to be able to notify the others so their cached copies are invalidated. Cross-Node Invalidation for Definition and Data Caches covers that.

The point is one sentence: between changing a definition and having it take effect there is no compilation, no packaging and no assembly deployment.

7. A map of the thirty posts

Chapter	Theme
One (posts 1 to 3)	Why Definition-Driven
Two (posts 4 to 8)	Definition Layer
Three (posts 9 to 12)	Data Access and Caching
Four (posts 13 to 18)	Business Logic and API
Five (posts 19 to 21)	Multi-Tenancy and Customization
Six (posts 22 to 25)	Transport, Security and Audit
Seven (posts 26 to 28)	Data Semantics
Eight (posts 29 to 30)	Case Review and The Big Picture

Every post is written roughly the same way. First what the problem requires you to weigh, then how the framework is designed and why it was designed that way, and where a case study can verify it, a closing section with the actual Northwind definitions.

Two things to be clear about up front:

The subject is long term cost, not development speed. In the short term Code-First wins almost every time: mature tooling, a complete ecosystem, a fast start. The cost of definition-driven design is concentrated at the beginning, and the return does not show until year three or year five. If your system ships, closes, and gets rewritten wholesale three years later, this series is of limited value to you. It is written for people who know they will be maintaining something for a long time.
Trade-offs and costs will be written as plainly as benefits. The value of a framework is not in what it claims to do. It is in knowing precisely what it does not do, and why. Highly dynamic UIs, dashboards, reports and anything that is not form-shaped will only hurt more if you force them into this model.

A thread to pick up later

The next post, An Overview of Definition-Driven Architecture, is about which decisions a framework should make on its users' behalf. The one after that, The Northwind Case Study: Eight Forms and a Custom Business Object, brings the case study on stage.

The case study builds a complete application out of Northwind: eight forms, master-detail, three cross-table relationships, validation, all of it. Exactly one of those forms needs code written for it. What is inside that one, and which places we deliberately chose not to express as definitions, gets reconciled item by item in Which Part Definitions, the Framework and Application Code Each Own.

The enemy of ERP is not complexity. It is restatement. And the reason tooling cannot eliminate restatement is that restatement is a product of the paradigm. As long as code is the source, the same fact is condemned to be said many times over.

📚 Part of the thirty-post series ERP Architect's Notes: Designing a Definition-Driven Framework

📘 Original article on HackMD, in Traditional Chinese:
👉 https://hackmd.io/@jeff377/day-01-why-definition-driven

📢 You are welcome to republish this, please credit the source.
📬 Follow for more engineering notes:
Facebook ｜ HackMD ｜ GitHub ｜ NuGet

Keeping a no-signup publishing tool useful

Wed, 28 Oct 2026 12:32:15 +0000

A generated report is ready. It looks fine in your browser. Now someone else needs to read it, and sharing it has become a second task: create a repository, pick a host, set up a deployment, or send a file with instructions.

I work on BrewPage, a free tool for publishing HTML, Markdown, small static sites and files through a shareable HTTPS link. Publishing doesn't require an account. You can use the website or an API, and there are integrations for AI agents.

That makes the first interaction pleasantly short. Running the service brings a longer question: how do you keep that convenience useful when an open publishing tool also attracts spam?

The tension shows up in ordinary product decisions. Someone sharing a small demo shouldn't have to become a hosting administrator. The person opening it still needs to be able to understand what they're looking at and report a problem. Both people matter, even though only one of them pressed Publish.

I don't think a signup form settles that question. For this product, publishing without an account is part of the point. Adding a mandatory account would change the experience for every person who only wants to share one report.

Take ownership. A link to read a page and a link to manage it serve different purposes. BrewPage returns an owner token and a management link for updating or deleting a publication. Those stay with the publisher. The reader gets the reading link. An optional account helps manage publications across devices, but it isn't a prerequisite for sharing one.

That distinction deserves attention in the interface and the instructions. Consider an agent finishing a report: its answer should contain the link a colleague can open. The credentials that allow changes belong with the publisher. A successful upload alone doesn't tell you whether the handoff was clear.

Visibility needs equally plain language. Public pages are listed in the gallery. An unlisted link can still be opened by anyone who has it; password protection is a separate choice. These are different ways to share something, and the wording should help people choose deliberately.

There is also a less exciting detail: expiry. BrewPage is temporary hosting. Publications expire, including those managed through an account. That suits a report being reviewed or a demo being discussed, but the author should know the lifetime before sending the link. They should keep their source files too.

For spam and harmful content, the public-facing part is straightforward: keep strengthening protection for users and publications, and make reporting a problem easy to find. BrewPage has a report-abuse form. That gives readers somewhere to go when they encounter something wrong. It doesn't justify promising that every link will be harmless.

These details are easy to lose behind the appeal of instant publishing. They are also what I would look at when evaluating another tool like this: who can change the result, who can read it, how long it lasts, and where a reader can ask for help.

Should a Failed Upload Clear the Form?

Wed, 28 Oct 2026 10:23:21 +0000

Cover: original AI-generated concept illustration with a fictional interface, not a working product screenshot.

Imagine filling in a project name, description and photo captions, then having the upload fail. Should the whole form reset, or should only the failed step need attention?

My starting point would be to keep completed fields, identify which file failed and explain whether the project itself was saved. “Upload failed” should not leave someone guessing whether they need to recreate everything.

I would also distinguish retrying the same file from choosing a replacement. If the original request's outcome is unknown, the app should check its status before offering a retry that might create a duplicate.

Keeping work has boundaries: a shared device may make persistent local drafts inappropriate, and sensitive content should not be retained indefinitely just for convenience.

Where would you draw the line between preserving a form in the current session and saving a recoverable draft? What should the interface tell users before they leave?

What Does an Empty Dashboard Mean?

Tue, 27 Oct 2026 10:47:59 +0000

Original AI-generated concept illustration, not a working product screenshot.

An empty dashboard can mean three different things: nobody has added a record, filters exclude the existing records, or the data request failed. Showing “No results” for all three leaves the user guessing.

My default would be one short explanation and one relevant next step: create a record, clear the filters, or retry the request. I would avoid a creation prompt when the app has not confirmed whether records already exist.

There is a trade-off, though. Explaining every possible cause can make a simple screen noisy, and some users may not have permission to see why data is unavailable.

What is the minimum context you would show in an empty state, and when would you offer more than one action?

Protect Human Edits: Conditional Saves for AI-Assisted Listing Drafts

Tue, 27 Oct 2026 10:18:08 +0000

Original AI-generated concept illustration. Fictional interfaces and property details, not an actual product screenshot.

Imagine a fictional property-listing editor used by two colleagues. Mira and Dev open the same draft. Mira corrects “two bedrooms” to “three bedrooms” after checking the property details, advancing revision 7 to revision 8. Dev asks an AI assistant to improve the description, reviews its wording, and clicks Save.

Dev’s browser still holds the old bedroom count. If the server accepts the entire draft unconditionally, a polished description can quietly erase Mira’s correction.

The useful question at save time is: Is the version this draft was based on still current? A conditional save makes that question part of the server’s write operation, rather than a warning the interface hopes everyone notices.

Keep the baseline with the draft

When the editor loads a listing, retain three things together: the saved content, its ETag, and a separate working copy. Start AI generation from that captured baseline. Do not fetch a newer tag later and attach it to output generated from older content.

A server might return this illustrative header excerpt, not a complete HTTP message:

ETag: "listing-42-r7"

The corresponding save includes:

If-Match: "listing-42-r7"

If-Match uses strong comparison against the selected representation. If the condition is false, the server must not perform the requested change; the usual response is 412 Precondition Failed. RFC 9110, section 13.1.1

Clients should treat tags as opaque values, preserving their quotes. A weak tag such as W/"listing-42-r7" never satisfies this comparison. Removing W/ does not turn it into a trustworthy strong validator. Also, If-Match: * tests existence, not whether the editor has the latest revision. MDN’s If-Match reference

For this proposed design, use one stable JSON representation for editing. Issue a new revision whenever its saved representation changes, including changes made by imports or background jobs. If serialization or representation variants can change independently, account for them in validator generation too. A revision number is useful only when the server maintains that invariant.

Make comparison and writing one operation

A separate “check version” request cannot protect the later save. Another writer can commit in the gap. Even checking inside the save handler is insufficient if the database permits a competing update between comparison and persistence.

Here is untested, framework-neutral pseudocode, not a deployable handler. This endpoint’s contract requires an explicit revision-tag list rather than a wildcard; its parser enforces that contract and rejects malformed input. Weak tags remain weak and cannot match.

save_listing(request):
    authorize_and_validate_request(request)
    if request.IfMatch is missing:
        return 428 with retry_instructions, CacheControl="no-store"
    candidates = parse_revision_tag_list(request.IfMatch)

    begin_transaction()
    try:
        listing = select_listing_for_update(request.listing_id)
        if listing is absent:
            rollback()
            return 404

        current = strong_etag_for_edit_representation(listing)
        if not any_strong_match(candidates, current):
            rollback()
            return 412

        replace_saved_draft(listing, request.draft)
        advance_representation_revision(listing)
        persist(listing)
        saved = snapshot_edit_representation(listing)
        commit()
    catch error:
        rollback_if_active()
        raise error

    return 200 with body=saved  # no response validator shown

The lock must cover reading the current version, comparing it, and writing the replacement. PostgreSQL row locks normally remain held until transaction end, providing one possible building block. PostgreSQL row-level locking documentation

Every writer must preserve the same versioning rules. Keep model generation and human review outside the transaction; a database lock should not wait for someone to finish editing.

The 428 response is an optional API policy, not a universal requirement for every server. It means this server requires a conditional request. Explain how to fetch a baseline and resubmit; responses with this status must not be cached. RFC 6585, section 3

Original AI-generated concept illustration with fictional people, interface and property details. An atomic version check protects the newer edit; the article explains the already-applied exception for a lost response.

Make a conflict recoverable

When Dev receives 412, leave the working copy intact. Fetch the latest listing into a separate buffer and present a three-way comparison:

Base: the content Dev originally loaded.
Mine: Dev’s reviewed AI-assisted draft.
Current: the latest saved content, including Mira’s correction.

Compare each side against the base. Distinct field changes may be straightforward to combine, but rewritten paragraphs can overlap semantically even when their text differs. Highlight the bedroom discrepancy for review instead of asking another model to decide which factual claim wins.

After reconciliation, associate the reviewed result with the newly fetched baseline and its tag. Save conditionally again. If another edit arrives during review, repeat the comparison without discarding anyone’s work. Avoid an automatic “retry with latest tag” button: it would authorize the stale replacement without resolving the conflict.

Original AI-generated concept illustration. Fictional collaborators compare the base, working draft and current saved version; not an actual product screenshot.

Treat a missing response as uncertainty

Suppose Dev’s save commits, but its response disappears. Retrying with the original tag can produce 412 because the first save advanced the revision. That does not prove the first attempt failed.

HTTP permits a success response when the server can determine that the requested change was already applied. The simpler handler above deliberately does not implement that optional recovery path. RFC 9110, section 13.1.1

Retain the draft and fetch current state before deciding what happened. If duplicate-operation recognition is needed, design a durable operation record and its retention rules explicitly; do not guess from a timeout.

Six tests before shipping

Ordinary save: a current tag succeeds, and a subsequent fetch reflects the saved draft and updated validator.
Stale editor: Mira saves first; Dev’s stale request gets 412 without overwriting her correction.
Concurrent saves: release two changed drafts together using the same tag; only one replacement commits.
Validator contract: missing tags receive the documented 428; weak tags cannot match; wildcards cannot bypass revision protection.
Conflict recovery: preserve the working copy, compare all three versions, and reject another stale reconciliation if a third change intervenes.
Lost response: commit a save, drop its response, and retry. Verify that the client reconciles uncertainty rather than claiming the original save failed.

AI-assisted editing can make drafting faster. Conditional saves keep that speed from silently replacing a colleague’s more recent work.

When Should Delete Offer Undo?

Mon, 26 Oct 2026 10:29:03 +0000

Undo is useful only when it genuinely restores what the user lost. A confirmation dialog is useful only when it explains the consequence—not just “Are you sure?”

Consider a hypothetical media app where someone deletes a set of property photos. My starting point would be:

Reversible removal: move files to trash, show a clear Undo action, and keep a discoverable recovery route after the toast disappears.
Permanent or bulk deletion: show the affected names or count, what cannot be restored, and whether shared access will be affected before proceeding.
Partial failure: report which files changed instead of claiming the entire operation succeeded.

The tricky case is shared assets: restoring a file may not reverse notifications or other effects that already happened.

W3C’s error-prevention guidance provides a useful reference for certain important submissions and changes to user-controllable stored data: reversal, checking, or confirmation. It does not mean adding a confirmation dialog to every routine edit.

Where do you draw the line between Undo, confirmation, and both? Would your answer change for one recoverable file versus a folder shared with a team?

Cover: original AI-generated conceptual illustration. The interfaces and recovery period shown are fictional, not a product demonstration.

BPMN as Code: From a Text-Based Model to a Working Microservices Sagas

Mon, 26 Oct 2026 08:43:45 +0000

Contents

Issues With Conventional BPMN Tools
Discovering Mermaid-BPMN
Making It Practical
A Real-World Example: Modeling a Payment Saga
From BPMN Source to an Executable Saga
Conclusions

I've used Documentation as Code for years when designing backends and entire applications. Most of my diagrams are written in PlantUML, with their source stored alongside other design documents. It suits me much better than drawing everything by hand.

Then I had to model a microservices Saga for Camunda. The process involved parallel checks, deadlines, human decisions and compensation. I wanted to write BPMN the same way I wrote UML, but couldn't find a toolchain I was comfortable using. Eventually I found Mermaid-BPMN and built a workbench around it.

Issues With Conventional BPMN Tools

I tried the obvious options. I find arranging shapes in draw.io or Camunda Modeler slow, particularly when a process keeps changing. I'd rather edit source and immediately see what the diagram looks like.

BPMN XML is source too, technically. But it is cumbersome to write from scratch and not much better to edit by hand. I also looked at other BPMN DSLs. Their diagrams didn't look right to me.

I wanted readable source and conventional BPMN diagrams, including horizontal pools and lanes. I also wanted a live editor on my own host and a CLI for generating SVGs from local files. Although my implementation used Camunda, I didn't want the modeling tools tied to one process engine.

And the diagrams had to be readable by people besides developers: architects, product owners, business analysts and anyone reviewing the process.

Discovering Mermaid-BPMN

After I inspected a few other tools, I came across the great derari/mermaid-bpmn, an external Mermaid plugin with its own BPMN DSL, rich and comprehensive. It supports vertical and horizontal BPMN directions, activities, pools and lanes, gateways, events and annotations. There was also a basic live-editor example.

I liked the language and the diagrams it generated. The DSL, parser, renderer and original editor are all the original author's work. I started with that project and added what I needed for day-to-day use. The result is Mermaid BPMN Workbench.

Making It Practical

First, I packaged the editor for deployment with Docker. It builds to static files served by Nginx, so I can run it on my own host without keeping the development server running. The image is available on Docker Hub.

I then worked on the editor itself. With a large process diagram, I need to zoom in, move around and quickly return to the full view. So I added pan, zoom and fit controls. I added source-pane width presets and font-size controls, too. The source and these settings are saved in the browser and restored after a reload. There is also a clear-source button and an option to open the full SVG in a separate tab.

One detail took a little care. Pan and zoom change the SVG displayed in the editor. Exporting that modified SVG could give you the part you were looking at rather than the complete diagram. I keep a clean snapshot for downloads and the Open SVG action.

I also needed SVGs for documentation, without opening the browser editor each time. I built a Mermaid-BPMN CLI for that. It generates an svg with this one-liner.

npx mermaid-bpmn-cli --browser /path/to/chrome payment-process.mmd

It writes payment-process.svg next to the source. Currently it requires Node.js 22.12 or later and an installed Chrome or Chromium executable; you have to pass the browser path explicitly. I have used the CLI in my project's documentation workflow since that. It simplifies the workflow a lot. Also can be used in automated CI workflows for Documentation-as-Code.

A Real-World Example: Modeling a Payment Saga

The process that prompted all this was a payment Saga (here is the link to the anonymized source).

It starts by debiting the payer, then runs anti-fraud and compliance/restriction checks in parallel. The results can lead to approval, denial, manual review or suspicious-payment handling.

The checks have a 30-second deadline. If they don't finish in time, the process refunds the payer. Manual review has a 20-minute deadline, with the same default outcome if nobody makes a decision. We chose these defaults because processing an erroneous payment would cost considerably more than rejecting a valid one. I put that reasoning in annotations next to the relevant paths.

The Diagram

![The Diagram](./images/payment-saga.svg)

The transfer to the payee is marked as an irreversible pivot point. Earlier failures are handled by compensation: an automatic refund and a rollback of payment status. Suspicious payments also trigger a security notification in parallel with the refund, with the paths joining before the final outcome.

This is the kind of process I wanted to keep in a text file: branches, timers, compensation, and the business reasons for some of the decisions. I can change the source, review the diff and regenerate the diagram.

From BPMN Source to an Executable Saga

I also took the Saga through implementation. The microservices were deployed, the Saga coordinated them, and I tested the cases in the implemented process.

For the engine, I needed BPMN XML. I had an AI agent translate the DSL, reviewed the result myself, manually validated it and used Camunda Modeler for static debugging before deployment. I was glad to have the Modeler for that part, even though I didn't want to use it for writing the original model.

This translation step isn't a Workbench feature. The workbench gives me the source editor and SVG generation; translation, engine validation, deployment and testing are steps in my project. I checked that the executable process preserved the deadlines, branches and compensation behavior in the model.

Conclusions

I believe, this tooling is useful for any modeler - architects, developers, business analyst and product owners - all who has to model and align the business processes with the stakeholders conveniently and quickly.

You can try my hosted editor. The Workbench repository includes deployment instructions, and the CLI is on npm.

Also, it were not be possible without the Mermaid-BPMN implementation project by Derari.

Too Many AI Tools, Too Much Rework: Fixing the Filmmaking Process for Agencies

Sun, 25 Oct 2026 13:27:10 +0000

AI video tools are getting better every month. A small creative team can now make product shots, short ads, character scenes and social media videos without arranging a full shoot every time.

That sounds like good news for advertising agencies. And it is.

But there is another side to it.

One tool creates the image. Another turns it into a video. A third handles the voice. Someone downloads the files and opens an editing app. Then the client asks for a different version, and the team has to go back through half of the process again.

The work was supposed to become easier. Somehow, the team now has more tabs open than before.

This is a problem many agencies face when they start using AI for video production. The issue is not always the quality of the AI output. Often, it is the way all the tools fit together.

The real challenge is not generating more clips. It is turning those clips into finished videos without doing the same work again and again.

How Did AI Filmmaking Become So Complicated?

At first, an agency may only need one AI video generator. The team tests a few prompts, creates some clips and shares them with the client.

Soon, the requests become bigger.

The client wants a product video with a person in it. The person needs to look the same in every scene. The product label must remain clear. The video needs a voiceover, subtitles and background music. The marketing team also wants vertical and horizontal versions.

One tool may not handle all these tasks well, so the agency adds another one.

Then another.

Before long, the team is using different websites for images, video generation, voice, editing, resizing and other small jobs. Each tool may work well on its own, but the full process becomes difficult to manage.

Files move from one place to another. Prompts get copied. References are uploaded again. Team members keep asking which version is the latest.

This is where AI filmmaking software needs to be judged differently. A good tool should not only create an interesting clip. It should also help the team move from the first idea to the finished video.

The Hidden Cost of Using Too Many Tools

The cost of separate tools is not limited to monthly subscriptions.

Think about the time people spend moving between them.

An editor downloads a clip, changes its format, uploads it somewhere else and waits for the next result. A designer searches for the correct product image. A creative lead tries to find the prompt used for the earlier version. Then someone notices that the new clip does not match the old one.

Each task may take only a few minutes.

But when the same steps happen across dozens of videos, those minutes add up.

There is also the cost of mistakes. A team might use an older logo, the wrong product image or an outdated script. A file may get saved in the wrong folder. A client may approve one version while the editor accidentally works on another.

None of these problems looks serious on its own. Together, they can slow down an entire campaign.

This is why agencies should look at the full production process before adding another AI tool. If a new tool solves one problem but creates three more steps, it may not be helping as much as expected.

Start With One Clear Production Workflow

The first fix is not necessarily buying new software. It is deciding how work should move through the agency.

A simple video workflow might look like this:

Brief → Script → Storyboard → References → Video generation → Editing → Sound → Review → Export

The exact steps will depend on the project. A short product ad may need less planning than a film with several characters and scenes.

Still, having a shared process gives everyone a clear starting point.

The person writing the script knows what information is needed. The designer knows which product images to prepare. The video creator knows which scenes need to be generated. The editor knows where to find the approved clips.

It also becomes easier to spot delays.

If a project keeps getting stuck because the team does not have approved references, that is a planning problem. If the team spends hours making new formats, that may be a workflow problem. If the client keeps changing the main idea after production has started, the approval process needs attention.

AI can help with production, but it cannot fix every problem caused by an unclear brief.

A clear workflow gives the technology something useful to support.

Keep Brand Assets and References in One Place

One common problem with AI-generated videos is consistency.

A character looks right in the first scene but different in the next one. A product changes shape. A logo appears in the wrong place. The colours slowly move away from the brand's usual style.

This becomes a bigger issue when the agency produces several videos for the same client.

The solution starts with better references.

Before generating a video, the team should know which product images, logos, colours, characters and visual examples are approved. Those references should be easy to find and reuse.

It also helps to keep a short record of the creative decisions. Which character reference was used? Which product image was approved? What visual style did the client choose? Which version of the script is final?

Without this information, team members may repeat work that has already been done.

Some AI creative platforms now provide shared asset and brand features for this reason. Lemonpeel, for example, lists an Asset Reference System, Brand Kit, Team Controls and Character Consistency among its platform features. These are intended to help creative teams keep references and brand details available across their work.

The important point is simple: do not make every new video start from zero when the same brand assets are already available.

Stop Treating Every Video as a Separate Project

Imagine an agency making short product videos for an online clothing brand.

The first video takes time because the team is still working out the process. They prepare the product images, decide on the visual style, create the scenes and finish the edit.

The next video is for another product from the same brand.

If the team starts from a blank page again, it will repeat much of the same work.

Instead, the agency can keep a reusable workflow. The product changes, the script changes and some of the scenes change, but the basic production steps remain the same.

This is where reusable workflows can make a difference.

Lemonpeel's Flows library includes ready-made processes for tasks such as product campaigns, UGC-style videos, storyboards, multi-shot scenes and product-focused content. Teams can open a workflow and run it with their own inputs rather than building every process from scratch.

A reusable workflow does not mean every video should look identical. It simply gives the team a starting point.

The creative idea can still be new. The production process does not need to be new every time.

Use Storyboards Before Generating Lots of Clips

Another source of rework is creating videos before the team has agreed on what the video should contain.

A creative person generates a few clips. The client sees them and says the story is wrong. The team changes the idea and starts generating again.

This is expensive in terms of time, even when each clip is quick to create.

A storyboard can help prevent this.

For a simple product ad, the storyboard might contain four shots:

The product appears in a clear opening shot.
A person picks it up or uses it.
The video shows a closer view of an important feature.
The final shot shows the product with the campaign message.

The client can review the direction before the team spends time generating all the shots.

It is much easier to change a rough plan than to rebuild a finished video.

AI tools can also help create storyboard images and turn those plans into scenes. Lemonpeel's Flows library includes a Prompt to Storyboard workflow, a Storyboard Creator and a Storyboard to Film workflow.

These features do not remove the need for creative decisions. They give the team a way to make those decisions earlier.

That is often where the biggest time savings begin.

Make Multi-Shot Videos Easier to Manage

A single AI-generated clip can look impressive. A finished video, however, often needs several shots that feel like they belong together.

Think of a short coffee ad. The first shot shows coffee beans falling into a grinder. The second shows the machine working. The third shows coffee pouring into a cup. The final shot shows the finished drink.

If each shot is created separately without a plan, the lighting, product details or visual style may change. The editor then has to find ways to hide those differences.

This is why shot planning and consistency matter.

A multi-shot workflow can help the team build several connected shots around one idea. It also makes the intended order of the scenes clearer.

Lemonpeel lists workflows such as Multi Shot Builder, Multi Shot Vertical and Multi Shot with Characters & Location. These are designed for creating short videos with several shots.

Agencies should test this kind of workflow with an actual client brief. Check whether the shots work together, whether the product stays consistent and how much editing is still needed.

The goal is not to remove editing. It is to reduce avoidable fixes before the editor even starts.

Plan for Different Formats From the Beginning

A video may look great on a desktop screen but fail when cropped for a phone.

The product may move outside the frame. The text may become too small. A person who was clearly visible in the wide version may be cut off in the vertical version.

Then the agency has to make changes and export the video again.

This is a common source of rework because teams sometimes treat aspect ratios as a final step.

A better approach is to decide the required formats during planning. If a campaign needs 9:16 for Reels and 16:9 for YouTube, the team should know that before generating every shot.

Some scenes may need different framing. Text placement may need to change. In certain cases, separate versions may be better than a simple crop.

Lemonpeel lists Reframe & Resize among its platform features, alongside workflows that allow different aspect ratios to be selected during creation.

Still, the agency should check the actual results. Resizing a video does not automatically make every version suitable for every platform.

The useful habit is to plan for the final places where the video will appear, instead of trying to fix every format at the end.

Choose Tools Based on the Work Your Agency Does

Not every agency needs the same AI setup.

An agency making simple social media ads may need quick product videos, captions and resizing. A team making story-led films may need character references, scene planning and more control over shots. An agency serving many brands may care more about shared assets, team access and repeatable workflows.

So there is no need to choose a tool just because it has the longest feature list.

Start with the tasks your team repeats most often.

Where do people lose time? Which steps cause the most revisions? Which jobs require opening several different tools? What do clients ask for again and again?

Then look for software that helps with those exact problems.

AI filmmaking software for agencies should be judged by how well it fits real agency work, not just by how good a sample video looks.

It is also worth checking how the pricing works. A tool that looks affordable may become costly if every revision or generation uses more credits than expected. Test the full process before making a long-term decision.

Give Your Team a Simple Way to Review and Approve Work

A messy review process can waste as much time as a messy production process.

The creative team may finish a video, but feedback comes from several people in different places. One person comments on the script. Another asks for a new opening. Someone else sends a message about the product colour.

The editor then has to work out which feedback is final.

A better process gives each project a clear review stage. The team should know who can approve the script, who checks brand details and who gives the final approval.

It also helps to collect feedback in one place and keep track of the version being reviewed.

This is not an AI feature. It is a team habit.

But it makes AI production much easier to manage because faster generation can lead to more versions. Without a clear review process, the agency may end up creating more work rather than finishing work sooner.

Measure the Time Saved, Not Just the Number of Videos Made

It is easy to feel productive when the team generates 30 clips in an afternoon.

But how many of those clips became finished videos? How many needed major changes? How much time did the editor spend fixing them? How many versions did the client actually use?

These are better questions to ask.

An agency can track a few simple things: time from brief to approval, number of revision rounds, number of tools used for one project and the amount of manual work needed for each format.

The team should also check the quality of the final output. A faster workflow is not useful if the brand looks inconsistent or the message is unclear.

Over time, these numbers can show which parts of the process are working and which parts need attention.

They can also help an agency decide whether it needs another tool or simply a better way to use the tools it already has.

What a Better AI Filmmaking Process Looks Like

A better process does not have to be complicated.

The agency starts with a clear brief. It prepares the right references. It plans the main shots. It uses a repeatable workflow where possible. The team generates the footage, checks consistency, edits the video and prepares the required formats.

Feedback is collected in a clear way. Approved assets and useful workflows are kept for the next project.

The next video still needs creative thinking, but the team does not have to repeat every small task.

This is the real value of AI filmmaking software for agencies.

It is not about making every step automatic. It is about letting technology handle useful parts of production while people focus on the story, the brand and the quality of the final work.

AI tools will keep changing. New models will appear, and some current tools may become less useful over time. A clear process helps the agency adapt without rebuilding its entire way of working every few months.

Final Thoughts: Fewer Disconnected Tools, Less Repeated Work

Using more AI tools does not always mean an agency will work faster.

If each tool adds another upload, another download and another place to manage files, the team may end up with a more complicated process.

The better goal is to connect the work.

Start with a clear workflow. Keep brand references organised. Plan shots before generating them. Reuse processes where it makes sense. Think about aspect ratios early. Make client reviews easier to manage. Measure the time needed to deliver the final video.

Then choose software that supports those habits.

An agency does not need to use every new AI feature. It needs a process that helps its people deliver good work without repeating the same tasks on every project.

Want to Bring More of Your Video Workflow Together?

Lemonpeel is building an AI creative workspace for agencies, studios and brands. Its platform brings multiple AI models and creative tasks together, including scripts, hooks, shot lists, image creation, video generation, sound, editing, subtitles and resizing. It also offers Marketing Studio, Film Studio, reusable Flows, asset references, Brand Kit and team features.

If your team is tired of switching between different tools just to finish one campaign, take a look at Lemonpeel and see whether its workflows fit the way your agency works.

Explore Lemonpeel: https://www.lemonpeel.ai

The aim is not to add one more tool to your list. It is to find a better way to turn creative ideas into finished videos, with less repeated work along the way.

When Is a Progress Bar Misleading?

Sun, 25 Oct 2026 10:26:33 +0000

A progress bar should tell users what has been measured, not just reassure them that something is happening.

Consider a hypothetical photo-processing tool: upload progress is measurable in bytes, but the next step might wait in a queue or take a different amount of time for each image. A bar that reaches 99% and then stalls can turn a useful signal into false precision.

My proposed default would be:

Use a percentage for work with a meaningful, known total.
Use explicit stages such as Queued → Processing → Ready when completion time is uncertain.
Show completed-file counts for batches, while making clear that files may take different amounts of time.
Keep failure and cancellation states separate from progress.

There is an accessibility detail too: status updates should be available to assistive technology without moving keyboard focus. W3C's guidance on status messages also warns against making updates too chatty; announcing every tick is not necessarily helpful.

Where would you draw the line between a useful estimate and false precision? Would you show stage labels, an estimated range, or a percentage when processing time varies?

Cover: original AI-generated concept illustration of fictional processing interfaces, not a product screenshot or performance result.

When Should a Client Preview Link Expire?

Sat, 24 Oct 2026 10:39:25 +0000

Cover: original AI-generated concept illustration, not a screenshot of a working product. The durations are discussion examples, not recommendations.

A photographer shares an unpublished property-photo preview on Friday. The agent opens it on Monday, after the link has expired. A longer lifetime would reduce friction, but the link could also be forwarded.

For a small client-review tool, which approach would you choose?

A short-lived direct link, with a clear way to request a new one.
A longer review window with an explicit expiry notice.
An authenticated review page that issues short-lived asset links after checking access.

AWS documents that S3 presigned URLs act as bearer tokens and can expire earlier when their underlying credentials expire: official S3 guidance. That distinction matters when the UI promises a review window. Expiry also cannot recall a file someone has already downloaded.

What should the recipient see after expiry, and how would you make recovery easy without silently extending access? I would favour a clear expiry message and an explicit renewal path, but I am interested in the trade-offs for clients who do not want another account.

This is a hypothetical design question, not a claim about any product’s current implementation.

Should a Resized Image Export Need a Fresh Visual Review?

Fri, 23 Oct 2026 13:03:59 +0000

Cover: an original AI-generated illustration of a hypothetical review/export workflow, not an actual product result.

A reviewer approves revision R-2 of an AI-edited property photo. The export step then creates a smaller JPEG for delivery.

Where should the review boundary sit?

Allow predetermined resizing/compression after approval, with a separate export record.
Require a final visual check of every delivered file.
Use both, with a stricter check when the transform changes the crop, label or visible content.

The distinction I would test is whether the derivative merely changes encoding or changes what the client sees. Even a routine export can make a disclosure label unreadable.

A file digest can identify the exact exported bytes, but it does not establish visual correctness. MDN explains the narrow role of a digest here: https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest

What rule has worked in your image or document pipeline, and which edge case made you change it?

Context: I work on EstateAI content and SEO. This is a hypothetical workflow question, not a claim about its current implementation.

Keep the Source: Provenance Records for AI Property Photos

Fri, 23 Oct 2026 13:01:05 +0000

Cover: an original AI-generated editorial concept showing fictional room-photo revisions. It is not an EstateAI result or a real property's before-and-after comparison.

A property photographer delivers a staged image. A week later, someone asks a simple question: which original photograph produced this version?

The answer should be easier to find than a message thread called “final images.” A useful application can connect the source file, each revision, the review decision and the exact export that left the system. This record matters even when the team never uses a formal content-provenance standard.

The following is a proposed design for a fictional property-photo application. It is not a claim about how EstateAI or another provider currently stores its files.

Start with four separate questions

Before choosing a database schema, separate the questions people actually ask:

Identity: Which file is this?
History: What inputs and operations produced it?
Review: Who approved this particular revision, and for what use?
Permission: Is the requested processing or publication covered by the recorded permission?

One “verified” boolean cannot answer all four. A checksum can help identify bytes, but it does not establish copyright ownership. A reviewer can approve the furniture placement while the team still lacks permission to publish the photograph in a public portfolio. A readable edit log can also contain an incorrect description.

Keeping these distinctions visible makes the product easier to reason about and gives support teams something specific to investigate.

Preserve the source as a separate asset

Store the original upload as its own asset with a stable internal ID. Create new records for derivatives instead of replacing that source whenever a preview is regenerated.

A minimal source record could contain:

An opaque source ID and the relevant workspace ID
An internal storage reference
A digest of the received file bytes
Media type, dimensions and ingestion time
A reference to the applicable processing-permission record

Treat the storage URL as a delivery mechanism rather than the asset's identity. URLs and access tokens can change; the relationship between a source and its derivatives should remain understandable.

For a browser-side preflight, the Web Crypto API supports SHA-256 through SubtleCrypto.digest(). The MDN documentation also notes that this method reads the entire input into memory rather than supporting streaming. Set sensible upload limits, and calculate the authoritative digest again on the server over the bytes actually stored.

A digest comparison has a narrow purpose. Re-encoding an image can change its file bytes even when it looks similar, so record the new export separately. Do not describe a matching hash as proof that the photographed scene is accurate.

Make every revision explain its parent

A revision should point to the source or earlier revision it used. Record the operation and the tool information you can actually observe. If a provider does not expose its model version, leave that field unknown; a guessed value creates false confidence.

For example, a revision record might say:

Revision R-2 belongs to source S-41
The requested operation was virtual staging
The input was the original source S-41
The output was stored as a new asset
The provider and version were recorded only where available
The revision was created at a particular time

A multi-input edit needs a list of inputs. A single parent field is insufficient when a workflow combines an original photograph, a mask and another supplied asset.

The initial implementation does not need to become a complicated event-sourcing system. The important properties are stable relationships, controlled updates and an audit trail for changes to the history itself.

Original AI-generated diagram of the illustrative application records used in this article. It is not a product screenshot, a signed Content Credential or a guarantee of image accuracy.

Bind approval to a revision, not the whole job

Suppose a reviewer approves R-2. A later request changes the sofa and produces R-3. Copying an approved flag from the job onto R-3 would silently approve an image nobody reviewed.

Instead, make the review record refer to the exact revision. Include the outcome, time, reviewer identifier and review scope. A new revision starts with a fresh review state. The approved revision can remain available in the history without making it the current delivery candidate.

Likewise, an export should point back to its approved revision and record its own digest. If delivery uses a resized JPEG, keep that relationship explicit rather than pretending the JPEG is byte-identical to the reviewed source format.

Keep the review label honest. “Approved for delivery” describes a workflow decision. It does not certify that the room exists exactly as shown, that every material detail is unchanged or that every destination permits that presentation.

Understand where Content Credentials fit

C2PA's explainer describes Content Credentials as cryptographically bound provenance information. They can carry assertions about a file's origin, edits and AI use. C2PA also makes an important distinction: provenance does not, by itself, establish that depicted content is true or factual.

An ordinary database row is not a C2PA Content Credential. Calling an internal audit log “C2PA verified” would be inaccurate unless the actual implementation and verification support that statement.

If a workflow handles credentials, preserve and inspect them using appropriate tooling, and record the validation result separately from human review. Absence of a credential should remain “no credential found” rather than becoming “camera original.” The IPTC guidance on trust and credibility is useful background on embedding structured media information and working with C2PA.

Keep public disclosure smaller than the internal record

An internal support record may need a reviewer ID, permission reference and provider request ID. A public caption usually needs far less. Avoid putting client names, private storage locations, access tokens or detailed prompts into downloadable metadata by default.

Design separate internal and public representations. The public version can explain an editing category and the relevant disclosure without exposing the operational record behind it. Maintain a retention policy for originals and revisions; preserving useful history is not a reason to retain every file forever.

Test the relationships before adding a badge

A focused test set can catch common mistakes:

Create two revisions from the same source. Confirm that each can retrieve the correct parent.
Approve R-2, then create R-3. Confirm that R-3 is unreviewed.
Export a resized copy. Confirm that it receives its own file identity and points to R-2.
Import an image without credentials. Confirm that the UI does not call it authentic or unedited.
Try to open another workspace's source through a known asset ID. Confirm that authorization is enforced.
Generate a public record. Confirm that private IDs, tokens and internal notes are excluded.

The result is a more useful answer to “where did this image come from?” The system can show which source was used, how the current revision relates to it and what was actually reviewed. That is a practical foundation for accountable image workflows.

From Lab to Code #1: Exploring Classical Computer Vision with Python and OpenCV

Thu, 22 Oct 2026 12:53:42 +0000

Most conversations about computer vision today revolve around deep learning, neural networks, and increasingly sophisticated AI models.

But not every image processing problem requires a trained model.

Sometimes, a Gaussian filter is enough to reduce noise. A threshold can separate objects from their background. A Fourier transform can reveal interference that is difficult to understand in the spatial domain.

These techniques are relatively simple, mathematically interpretable, and still useful in practical applications.

Recently, while exploring image and signal processing concepts in my computer engineering studies, I decided to start a small programming series called From Lab to Code.

The idea is simple: take engineering concepts I encounter, build independent implementations, visualize the results, and share what I learn.

For the first project, I built a small interactive image processing toolkit using Python, OpenCV, NumPy, and Streamlit.

GitHub repository: https://github.com/Ertugrulmutlu/from-lab-to-code

Project 01 — Image Processing: https://github.com/Ertugrulmutlu/from-lab-to-code/tree/main/01-image-processing

What Are We Building?

The application explores four classical computer vision techniques:

Image denoising using Gaussian and median filters.
Object counting using illumination correction, thresholding, and connected components.
Periodic noise removal using the Fast Fourier Transform (FFT).
Color clustering using K-means.

The application provides interactive controls, generated example images, and visual comparisons.

No pretrained models, API calls, or external datasets are required.

All bundled demonstration images are generated programmatically.

1. Image Denoising: Gaussian vs. Median Filtering

Digital images are often affected by noise.

Noise can originate from camera sensors, poor lighting conditions, signal transmission, or image acquisition hardware.

Two common types are:

Gaussian noise: Random variations in pixel intensity, often modeled using a normal distribution.
Salt-and-pepper noise: Individual pixels unexpectedly taking very dark or very bright values.

Different noise distributions require different filtering strategies.

Gaussian Filtering

A Gaussian filter performs a weighted average of neighboring pixels.

Pixels closer to the center receive higher weights, following a Gaussian distribution.

Mathematically, the two-dimensional Gaussian kernel can be written as:

G(x, y) = (1 / 2πσ²) × exp(-(x² + y²) / 2σ²)

The parameter σ controls the amount of smoothing.

A larger σ produces stronger smoothing but can also remove fine image details.

In OpenCV, Gaussian filtering is straightforward:

import cv2

gaussian = cv2.GaussianBlur(
    image, (5, 5), 0
)

Median Filtering

Unlike Gaussian filtering, a median filter replaces the center pixel with the median value of its neighborhood.

Consider these pixel values:

[10, 11, 12, 13, 255]

Their average is 60.2, but their median is 12.

If the value 255 represents an isolated noise spike, the median is much closer to the surrounding pixel values.

This makes median filtering particularly useful for salt-and-pepper noise.

median = cv2.medianBlur(image, 5)

Comparing the Results

The application applies both filters to the same noisy image.

It also calculates the Mean Squared Error (MSE) against the original image when a reference is available.

MSE = (1/N) × Σ(original - processed)²

A lower MSE indicates a smaller average squared difference from the reference image.

However, MSE alone does not fully describe perceptual image quality.

Key takeaway: Gaussian filtering is useful for smoothing distributed noise, while median filtering is often more effective against isolated intensity spikes.

Neither method is universally superior.

2. Object Counting: From Pixels to Objects

Suppose we want to count small objects placed on a table.

At first, this sounds like a simple thresholding problem.

If the objects are darker than the background, we could classify pixels based on their intensity.

But what happens when the background illumination is uneven?

One side of the image might be noticeably darker than the other, making a single global threshold unreliable.

Step 1: Estimate the Background

We can approximate slowly changing illumination using a large Gaussian blur.

background = cv2.GaussianBlur(
    gray, (0, 0), sigmaX=30
)

Subtracting the original image from the estimated background helps emphasize darker foreground objects.

corrected = cv2.subtract(
    background, gray
)

This reduces the influence of gradual illumination changes.

Step 2: Apply Otsu Thresholding

Otsu's method automatically selects a threshold based on the image's intensity distribution.

Instead of manually choosing a number such as 100 or 150, it searches for a threshold that separates the intensity classes according to their statistical variance.

_, mask = cv2.threshold(
    corrected,
    0,
    255,
    cv2.THRESH_BINARY + cv2.THRESH_OTSU
)

The output is a binary image containing foreground and background pixels.

Step 3: Identify Connected Components

After thresholding, neighboring foreground pixels can be grouped into connected regions.

OpenCV provides a useful function for this:

count, labels, stats, centroids = (
    cv2.connectedComponentsWithStats(
        mask, connectivity=8
    )
)

The function returns component labels and information such as bounding boxes and pixel areas.

By excluding the background and removing regions below a minimum area, we can estimate the number of objects.

The project also applies a small morphological opening operation before counting to remove isolated foreground noise.

Where Can This Be Useful?

This approach can be applied to controlled visual inspection tasks involving separate objects, such as small manufactured components.

However, it comes with important assumptions.

Objects should be sufficiently separated, and the contrast between objects and background should be reasonably clear.

Touching objects may merge into a single connected component.

Key takeaway: Reliable object counting often requires preprocessing, not just thresholding.

3. Fourier Transform: Removing Periodic Noise

This was one of the most interesting parts of the project.

Images are normally represented in the spatial domain, where each pixel corresponds to a position.

But some image artifacts are easier to identify in the frequency domain.

For example, imagine an image affected by regularly repeating stripes.

Those stripes represent periodic variations in pixel intensity.

Moving to the Frequency Domain

The two-dimensional Discrete Fourier Transform decomposes an image into spatial frequency components.

Each component represents a particular frequency and orientation.

Using NumPy:

import numpy as np

spectrum = np.fft.fft2(gray)

shifted = np.fft.fftshift(spectrum)

magnitude = np.log1p(
    np.abs(shifted)
)

Here:

fft2() computes the two-dimensional Fourier transform.
fftshift() moves the zero-frequency component to the center.
abs() obtains the magnitude.
log1p() makes the magnitude spectrum easier to visualize.

Periodic interference may appear as distinct peaks in the Fourier spectrum.

Removing Specific Frequencies

If we know which frequencies correspond to interference, we can suppress them using a notch filter.

A notch filter removes selected frequency regions while preserving much of the remaining spectrum.

For real-valued images, the Fourier spectrum has conjugate symmetry.

This means a sinusoidal interference component generally produces a symmetric pair of frequency peaks.

Both should be handled to preserve the symmetry of the filtered spectrum.

The basic operation is:

filtered_spectrum = shifted * mask

Here, mask contains zeros at the frequencies we want to suppress and ones elsewhere.

We then reconstruct the image using the inverse Fourier transform.

restored = np.fft.ifft2(
    np.fft.ifftshift(filtered_spectrum)
).real

An Important Limitation

In this project, the periodic noise is generated with known frequencies.

Therefore, the notch filter already knows where the unwanted frequency components are located.

This is intentional: it lets us isolate and study the effect of frequency-domain filtering.

Automatically identifying unknown interference frequencies in arbitrary photographs is a separate problem and is not implemented here.

A notch filter can also remove useful image information if the rejected frequencies overlap with actual image content.

Key takeaway: The Fourier transform provides another perspective on image processing, allowing us to manipulate structures according to their spatial frequencies rather than their pixel positions.

4. K-Means: Unsupervised Color Clustering

The final technique explores unsupervised learning.

Unlike classification models, K-means does not require predefined class labels.

Instead, it groups observations based on similarity.

For an RGB image, each pixel can be represented as a three-dimensional vector:

Pixel = [Red, Green, Blue]

K-means attempts to divide these vectors into K clusters by minimizing the squared distances between pixels and their assigned cluster centers.

The objective function is:

J = Σ ||xᵢ - μ(clusterᵢ)||²

where xᵢ is a pixel vector and μ is the corresponding cluster center.

Implementing K-Means with OpenCV

First, reshape the image into a collection of RGB vectors:

pixels = image.reshape(
    -1, 3
).astype(np.float32)

Then run K-means:

criteria = (
    cv2.TERM_CRITERIA_EPS +
    cv2.TERM_CRITERIA_MAX_ITER,
    25,
    0.2
)

_, labels, centers = cv2.kmeans(
    pixels,
    4,
    None,
    criteria,
    5,
    cv2.KMEANS_PP_CENTERS
)

We can reconstruct an image by replacing each pixel with the color of its assigned cluster center.

This produces a simplified image containing a small number of representative colors.

Color Clustering Is Not Semantic Segmentation

This distinction is important.

K-means only sees RGB values in this implementation.

It does not understand objects, scenes, or semantic categories.

Two unrelated objects with similar colors may belong to the same cluster.

Likewise, one object with multiple colors may be split across several clusters.

Still, color clustering is useful for exploring image structure, reducing color complexity, and understanding unsupervised learning.

Running the Project Locally

The project is available on GitHub:

https://github.com/Ertugrulmutlu/from-lab-to-code/tree/main/01-image-processing

You will need Python 3.10 or newer.

Clone the repository:

git clone https://github.com/Ertugrulmutlu/from-lab-to-code.git
cd from-lab-to-code/01-image-processing

Create a virtual environment:

python -m venv .venv

Activate it on Windows PowerShell:

.\.venv\Scripts\Activate.ps1

Install the dependencies:

python -m pip install -r requirements.txt

Start the application:

python -m streamlit run app.py

Streamlit will open an interactive application in your browser.

You can switch between the four techniques, adjust processing parameters, and compare the results.

Each mode includes a programmatically generated example image. You can also upload an image of your own.

To regenerate the demonstration gallery:

python demo.py

To run the included tests:

python -m unittest discover -s tests -v

The processing algorithms are implemented in processing.py, while synthetic.py generates the example images and app.py contains the Streamlit interface.

What I Learned

Building this project reinforced several important ideas.

First, understanding the characteristics of a problem matters as much as selecting an algorithm.

Gaussian filtering and median filtering may look similar at first, but they behave differently depending on the noise distribution.

Second, preprocessing can determine whether a simple computer vision algorithm succeeds or fails.

In the object counting example, background illumination correction can be just as important as the thresholding operation itself.

Third, the frequency domain provides a different way to reason about images.

Some patterns that are difficult to isolate spatially become much easier to analyze after a Fourier transform.

Finally, unsupervised clustering demonstrates that useful structure can be extracted from images without labeled training data, even though grouping colors is not the same as understanding a scene.

Why I Started From Lab to Code

I wanted a lightweight way to turn engineering concepts into practical software projects.

Not every project needs to become a startup, a large framework, or a long-term product.

Sometimes, a small, well-explained implementation is enough to explore an idea and demonstrate what you have learned.

That is the philosophy behind From Lab to Code.

Each entry will focus on an engineering topic, build a small independent implementation, and document the underlying concepts and results.

The projects are independently implemented, with original demonstration materials rather than redistributed university coursework.

This first project focuses on classical image processing. Future entries may explore other engineering topics as I encounter them.

If you are learning computer vision or signal processing, I hope the code and explanations help you experiment with these concepts yourself.

Explore the series: https://github.com/Ertugrulmutlu/from-lab-to-code

Explore the image processing project: https://github.com/Ertugrulmutlu/from-lab-to-code/tree/main/01-image-processing

Thanks for reading!

C$2.6 Billion to Fix, C$57 Million to Replace: Canada Was Told in 2019, and Phoenix Is Still Running

Thu, 22 Oct 2026 09:34:19 +0000

Originally published at vibeagentmaking.com.

A payroll is not a cost center. It is insurance against the one loss a state cannot pool, and Canada priced that risk at zero.

Somewhere in the Government of Canada, a public servant opened a pay stub and the number was wrong. Then it happened again, to a nurse at Veterans Affairs, to a border officer, to a scientist at Environment Canada, to tens of thousands of people who had done nothing but show up to work. Between 2016 and the early 2020s, Canada eventually paid damages to 324,346 current and former employees for the experience of being paid wrong: about C$400.5 million in a single fiscal year, roughly C$1,200 a head. That payment is usually filed under "settlement." It is worth reading it as something more precise. It has the exact shape of an insurance claim, paid out on a risk that the system's original business case had priced at zero.

Phoenix, the federal pay system that went live in February 2016, is one of the most documented public technology failures in the world. It is usually told as a story about project governance, or vendor management, or the culture of a public service that shipped a system it knew was not ready. Those stories are true. But underneath them is an economics story that the Auditor General's own 2026 tables and the Parliamentary Budget Officer's 2019 numbers tell cleanly, and it is a story every technical leader who has ever signed off on a migration should read. A payroll system is not a cost center. It is insurance against the one loss a state cannot pool. Phoenix is what happens when you buy that insurance as if it were a savings plan.

The payment that was a claim, not a settlement

Look first at the structure of what Canada paid its workers, because the structure is the argument. Two damages agreements, one in June 2019 with several unions and one in October 2020 with the Public Service Alliance of Canada, compensated employees not for provable financial loss but for the ordinary experience of being paid through Phoenix. The schedule reads like an actuarial table: C$1,000 for anyone who worked at least one day in fiscal 2016 to 2017, then C$500 for each of the next three fiscal years, with separate processes layered on top for people who could demonstrate real money lost or severe personal hardship.

That is not how a lawsuit pays out. A lawsuit pays the harmed party in proportion to the harm. This paid a flat per-person indemnity on a defined event, which is what an insurance policy does. And it went to 324,346 people, which is close to the entire federal public service of the period. When a payout is that near-universal, the state has quietly conceded something large: the exposure was not to some unlucky subset, it was to everyone on the payroll at once.

That concession is the whole insurance problem in one sentence. Insurance works by pooling risks that are independent. Ten thousand houses can be insured against fire because they do not all burn on the same night. But when a single pay system is the common cause, every employee's risk becomes the same risk, perfectly correlated, and there is nothing left to pool. The loss cannot be diversified away inside the workforce, because the workforce shares one point of failure. Phoenix did not just fail some people. It made everyone's paycheck depend on the same broken thing, which is the one shape of risk that insurance cannot absorb.

What a payroll actually insures

So what is the payroll insuring against? Not fraud, and not arithmetic. It is insuring the willingness of hundreds of thousands of people to keep coming to work while trusting that the money will be right. That trust is an asset, and it is the one asset a government cannot reinsure, borrow, or buy back on short notice. Most state risks can be pooled or transferred. This one cannot, because the moment the same system pays everyone wrong, the trust breaks for everyone at the same time, in the same direction, for the same reason.

The 2009 business case did not have a line for that. In July 2009 the cabinet approved C$310 million for the Transformation of Pay Administration, a two-part plan to modernize and consolidate federal pay, and the promise attached to it was about C$70 million a year in savings once an automated, self-serve system replaced hundreds of compensation advisors. Read that promise carefully. It is a savings case. It prices the upside of automation, and it is silent on the cost of the one scenario that would matter most, the scenario where the automated system pays a third of a million people incorrectly for years. That risk was not mispriced. It was assumed away, which is worse, because a number you leave off the page cannot be argued down. It simply is not there when the decision is made.

A catastrophe that keeps paying out

Phoenix went live in February 2016. By 2018 the Auditor General estimated it would cost up to C$2.2 billion to fix by 2023. The claims kept arriving: the C$400.5 million in damages in the year ending March 2021, and a reported cumulative total around C$560 million. And behind the claims sat the backlog, which is the number that decides whether any of this ever ends. As at 30 September 2025, the Auditor General reported that the pay center's backlog "stood at over 233,000 pay transactions, impacting over 133,000 employees." Of those transactions, 155,217 were older than a single year. The department's own reporting through 2025 already conceded it would miss the target, set back in early 2023, to clear that year-old cohort by March 2026.

This is the shape of a catastrophe, and it is worth being precise about why. A normal operational error looks like a bell curve: small mistakes scattered around a mean, most of them minor, canceling out in the aggregate. Phoenix is not that. It is a fat tail, where one correlated failure produces a loss that does not cancel and does not settle, it compounds. Each uncleared case is not a static debt. It generates more errors downstream, more manual interventions, more overpayments to claw back and underpayments to make good. And each year the government keeps running the failed system, it pays another year of what is, in every meaningful sense, a premium: the ongoing cost of carrying a policy that has already paid out and keeps paying.

The 2019 number that should stop you

Here is the fact that should stop any reader who has ever argued for keeping a system alive one more year. In May 2019, the Parliamentary Budget Officer, Parliament's own independent costing office, reported that stabilizing Phoenix would cost about C$2.6 billion and take another four years, while building a replacement would cost, in its words, "a comparatively modest $57 million," a figure covering procurement, testing, and training over six years, with annual operating costs later. Its conclusion was blunt: the cost of building a new pay system "should pale in comparison to the cost of stabilizing the failed Phoenix system."

Sit with the gap. In 2019, in public, the government's own budget watchdog said that exit was on the order of forty times cheaper than continuing to fix what it had. The replacement is now scheduled to finish somewhere between 2028 and 2031. That is roughly a decade of paying to run a system after being told, officially, that replacing it was the cheaper choice.

The reason is not stupidity, and pretending it is misses the lesson. The exit cost is a visible, budgeted, attributable line that someone has to stand up and defend in a given fiscal year. The premium, the ongoing cost of running the broken system, is diffuse, already appropriated, and paid largely by people who are not in the room when the decision gets made. One number has a name attached to it and a meeting on the calendar. The other is spread across a workforce and a decade. Institutions keep paying an invisible premium over a visible exit cost, not because they cannot do the arithmetic, but because only one of the two numbers ever has to be said out loud.

The root cause, now a line item in the rescue

The most quotable sentence in the entire 2026 record is the Auditor General's, from her statement to reporters: "It is concerning to me that, a decade later, there has been little progress made to simplify these rules." Phoenix was never mainly a software failure. The pay rules it had to implement, accumulated across dozens of collective agreements over decades, were extraordinarily complex, and the plan had assumed those rules would be simplified before or alongside the new system. They were not. A decade on, they still have not been.

So look at what the replacement does about it. Rather than simplify the rules, the plan is to buy cloud extensions, estimated at at least C$4 million a year, to carry the unsimplified rules into Dayforce, the new system. The rescue budget now contains a permanent annual line whose only purpose is to preserve the exact complexity that caused the disaster.

Any engineer who has led a migration knows this move in their bones. It is the decision to port the legacy logic instead of retiring it, to wrap the old mess in an adapter because rewriting it is politically or technically expensive, and to tell yourself the cleanup will happen later. Later does not come. The complexity that broke the old system moves into the new one, now with a recurring invoice attached, and the new system inherits its predecessor's root cause as a fixed operating cost. Canada has put a dollar figure on "we will simplify it later," and the figure is at least four million a year, forever.

The number that is not yet counted, and the bet

There is one more line the honest version of this ledger has to carry, and the Auditor General wrote it herself. Public Services and Procurement Canada's estimate for Dayforce is "more than $4.2 billion," and in the same breath the report adds: "This estimate did not include important costs needed for all departments and agencies to transition." So C$4.2 billion is not the cost of the rescue. It is the cost of the rescue before the departments are moved, and the auditor says so on the page. A ledger built honestly has a line that reads "not yet counted," and here that line is load-bearing. It is also worth remembering the trajectory: the replacement that Parliament's costing office sketched at tens of millions in 2019 is now a program estimated in the billions, on broader scope and different accounting, but pointing the same way. Delay shows up in the exit price.

And then there is the acceleration. The end date for Phoenix moved from 2034, to March 2031, and then, in January 2026 after the audit period had closed, Public Services and Procurement Canada shortened the schedule by about three more years, largely to avoid the cost of running two pay systems in parallel. The Auditor General named the trade in a single sentence: "Although this decision is intended to reduce the complexity and cost of running 2 systems in parallel, it means less time to clear the backlog and prepare departments for the transition."

That is a clean, legible bet, and it points two ways at once. What accelerating buys is fewer years of paying to operate Phoenix and Dayforce at the same time, and each parallel year is a real, recurring, avoidable cost. That is cost arithmetic, and by cost arithmetic the acceleration is defensible. What it bets is that the 233,000-odd transactions, 155,217 of them over a year old, can be cleared faster than they have ever been cleared, by a department that already said it would miss the March 2026 target for exactly that older cohort. That is risk arithmetic, and by risk arithmetic the acceleration raises the probability of the one failure the whole project exists to avoid. The Auditor General states the danger directly: the backlog "creates a risk of transferring existing pay errors into the Dayforce system, which could undermine its effectiveness from the start." Every uncleared case is a wrong number that the replacement inherits as an opening balance. The failed system's output becomes the successful system's starting data. A government that had priced the 2016 risk correctly might notice it is making the same trade again: a certain saving now against an unpriced, correlated loss later.

What a technical leader can take from this

The temptation is to read Phoenix as a story about government, and to conclude that large public IT is uniquely doomed. That is the comfortable reading and the useless one. The transferable lesson is about pricing a specific risk that every migration and every automation carries, and there are three moves that follow directly from the ledger above.

First, put the correlated-failure line in the business case. A savings case that omits "what does this cost if it is wrong for everyone at once" is not a business case, it is a brochure. The worst case is not the average error multiplied by the number of users. It is a single correlated failure across all of them at the same time, and that number belongs on the page next to the projected savings, where it can be argued about before the decision, not discovered after.

Second, never migrate the errors. The backlog is the tell in the Phoenix story, and it is the tell in most failed cutovers. A migration that carries uncleared wrong state into the new system does not start clean. It starts with the old system's liabilities as its ground truth. Clear the bad data, or quarantine it explicitly, before it becomes the opening balance the new system trusts.

Third, fix the root cause or price it honestly and forever. If the complexity that broke the old system is ported into the new one, you have not solved the problem. You have bought an annuity on it, and the C$4 million a year for cloud extensions is what that annuity costs when "we will simplify it later" turns into never.

Underneath all three is the thing a payroll actually protects, which is trust, and trust is the one asset that insurance cannot pool, because when it breaks it breaks for everyone on the payroll at once, in the same direction, for the same reason. Price that risk at zero in the business case and you will not avoid paying for it. You will pay for it later, per person, under someone else's signature. Canada paid about C$1,200 a head, to 324,346 people, and a decade after its own budget office said the exit was cheaper than the fix, it is still paying the premium.

The old system's wrong numbers become the new system's opening balance

The Auditor General's warning about Phoenix is a provenance problem stated in payroll terms: uncleared errors get inherited by the replacement as ground truth, because nothing in the handover records which figures were already known to be wrong. Chain of Consciousness is that record for agent work, a verifiable log of what was seen, decided and asserted, so "do not migrate the errors" becomes a query you can actually run instead of a resolution you make.

pip install chain-of-consciousness · npm install chain-of-consciousness

Hosted Chain of Consciousness · Verify a record

Sources

Office of the Auditor General of Canada, 2026 report on modernizing the federal pay system, and the Auditor General's accompanying opening statement (canada.ca), for the "more than $4.2 billion" Dayforce estimate and its exclusion of department transition costs, the 30 September 2025 backlog of over 233,000 transactions affecting over 133,000 employees, the "little progress made to simplify these rules" statement, the risk of transferring pay errors into Dayforce, and the parallel-running trade-off.
Parliamentary Budget Officer, May 2019 report on the cost of Phoenix, for the roughly C$2.6 billion stabilization estimate over four years versus the approximately C$57 million to build a replacement, and the finding that replacing would be cheaper than stabilizing (as reported by CBC News, Global News, The Globe and Mail, and Canada's National Observer).
Government of Canada Public Accounts and reporting by The Globe and Mail, for the C$400.5 million in Phoenix damages paid to 324,346 employees in the year ending 31 March 2021 and the reported cumulative damages of about C$560 million; the June 2019 and October 2020 damages agreements for the C$1,000-plus-C$500 schedule.
Contemporaneous audit reporting (The Globe and Mail on the C$4.2 billion estimate; CBC on the Dayforce transition risks; Global News on backlog progress) and the July 2009 cabinet approval of C$310 million for the Transformation of Pay Administration with its roughly C$70 million annual savings projection.

Figures note: the 2026 Auditor General quotations were read from the primary report; the 2009 business-case figures, the 2018 C$2.2-billion estimate, the 2019 Parliamentary Budget Officer figures, and the damages totals are drawn from public reporting of the underlying documents and are consistent across independent outlets. The C$4.2 billion Dayforce estimate is preliminary and, by the auditor's own note, excludes department transition costs; it is a different system and a different accounting from Phoenix's own multi-year cost, and the two should not be added together. The backlog is reported in three distinct denominators, pay transactions, cases older than one year, and employees affected, which are not interchangeable.

When an AI Image Job Times Out, Should Retry Create a New Job?

Wed, 21 Oct 2026 10:37:40 +0000

A photographer submits a batch of property photos for AI enhancement. The browser times out before it receives a job ID. Clicking Retry could recover the existing job, or create another paid batch while the first is still running.

The timeout only tells the client it did not receive a usable response. It does not prove the server did nothing.

Stripe’s idempotent-request documentation provides a concrete example of reusing an operation key to avoid duplicate work after a connection error. An image-processing service would need its own explicit contract; a payment API’s behavior should not be assumed to apply automatically.

For a proposed image-job API, I would separate:

Retry this submission: same operation key and unchanged inputs, recover its status or result.
Generate another version: new operation, with any credit cost visible before submission.
Change the inputs: explicitly create a new request rather than silently reusing the old key.

The difficult case is an unknown outcome: no response, no job ID in the browser, and credits possibly reserved. How would your API let the user recover that job safely? Would you require the client to save an operation ID before submitting, and how would you handle key expiry?

This is a design scenario, not a claim that a named platform currently double-charges. Disclosure: I work with Pixel Perfects Solutions LLC / EstateAI.

Building What Users Ask For vs. Solving What They Actually Need

Tue, 20 Oct 2026 14:07:41 +0000

Every developer has faced this situation.

A user asks for a new feature. You understand the requirement, write the code, test it, and ship it. The feature works exactly as requested.

But after the release, something unexpected happens.

Nobody uses it.

At first, you might think users need more time to understand the feature. Maybe the UI needs improvement. Maybe you need to add a few more options.

But sometimes, the real problem is much simpler.

You built what the user asked for, not what the user actually needed.

The Difference Between a Request and a Problem

Imagine a user tells you:

"We need a search button on this page."

You add a search button, connect the API, handle the results, and ship the feature.

A few days later, the user complains that finding information is still difficult.

After investigating, you discover that the actual problem wasn't the absence of a search button. The page contains hundreds of records, and users need better filters, sorting, and a way to find relevant results quickly.

The search button was their proposed solution, not necessarily the right solution.

Users usually understand their problems better than they understand the technical solutions to those problems.

And that's completely normal. Understanding the solution is part of the engineering work.

Why Developers Make This Mistake

There are a few common reasons.

1. We Want to Start Coding Immediately

A new feature sounds exciting. We open the IDE, create components, write API endpoints, and start implementing.

Asking more questions can feel like slowing down the work.

But starting quickly doesn't always mean finishing quickly. If we misunderstand the requirement, we may spend days building something that needs to be redesigned.

2. We Treat Every Request as a Complete Requirement

"Add an export button."

"Send an email notification."

"Create another dashboard."

These requests describe features, but they don't always explain the problem, the intended user, or the expected outcome.

Before implementing, we need to understand why the feature matters.

3. We Focus on Whether the Feature Works

Developers naturally test whether the code behaves correctly.

Does the API return the right response? Does the button work? Is the data saved correctly?

Those checks matter, but there's another question:

Does this feature actually make the user's job easier?

A feature can pass every technical test and still fail to solve the real problem.

Ask Better Questions Before Writing Code

You don't need a long meeting for every small feature. Sometimes, a few clear questions are enough.

What problem are we trying to solve?
Who experiences this problem?
How do they handle it today?
What would a successful outcome look like?
Is there a simpler way to achieve the same result?

These questions don't replace implementation. They help ensure that implementation is worth doing.

For example, instead of immediately building a complex reporting dashboard, you might discover that users only need a downloadable CSV with three specific columns.

Less code. Less maintenance. A better outcome.

Learn to Challenge the Proposed Solution

This doesn't mean developers should reject every request or assume they know better than the user.

Users have valuable context that developers may not have. The goal is to combine that context with technical understanding.

If someone requests an unnecessary feature, explain the trade-offs. If their proposed solution is appropriate, build it. If a simpler alternative solves the same problem, discuss it.

The key is to challenge the solution respectfully, not the person asking for it.

Instead of saying, "We don't need this feature," try asking:

"Would it solve your problem if we did it this way instead?"

That small change can lead to much better conversations.

Build, Measure, and Learn

Sometimes, you won't know the best solution until users try it.

That's where a smaller first version helps.

Build the simplest useful solution, share it with the intended users, observe how they interact with it, and gather feedback.

Don't assume that a feature is successful just because it has been deployed.

Look for evidence: Are people using it? Does it reduce manual work? Does it save time? Has the original problem become less frequent?

Real usage often reveals things that requirements documents cannot.

AI Makes This Even More Important

AI tools can generate components, APIs, forms, dashboards, and entire application flows faster than ever.

But faster implementation doesn't guarantee better problem-solving.

If you give an AI tool a poorly understood requirement, it may produce a polished implementation of the wrong idea.

The better approach is to use AI to explore the problem before committing to a solution. Ask it to identify assumptions, suggest simpler alternatives, compare trade-offs, and consider edge cases.

AI can help you implement faster, but you still need to decide what deserves to be implemented.

The Real Skill Is Knowing What to Build

As developers gain experience, their work gradually changes.

They still write code, debug issues, and review pull requests. But they also learn to question assumptions, understand user workflows, simplify requirements, and recognize when a feature adds unnecessary complexity.

The goal isn't to build the biggest solution or deliver the most features.

It's to solve the right problem with a reliable, maintainable solution.

The next time someone asks you to build a feature, don't stop at asking yourself, "How should I implement this?"

Ask one more question:

"If I build this exactly as requested, will it actually solve the problem?"

That question might save you more time than writing the code faster ever could.

Real Estate AI Workflows in 2026: Comparing Review, Revisions, and Human Handoffs

Tue, 20 Oct 2026 10:51:56 +0000

Cover illustration: AI-generated concept showing an empty room, virtual staging and review. It is not an actual product result.

A property photographer sends an edited room image to an agent. The agent requests a furniture change. Another team member downloads the earlier version and adds it to the listing.

The image generator might have worked perfectly. The delivery workflow still failed.

When comparing real estate AI and editing platforms, it helps to follow one asset from upload through revision and approval. This reveals who controls each step, where context can get lost, and what the surrounding software needs to track.

Disclosure: I work in Content & SEO at Pixel Perfects Solutions, the company behind EstateAI. This comparison uses official product pages checked on 20 October 2026. The capabilities below are vendor-described; hands-on image-quality and performance testing has not been performed.

A. Three different workflow choices

1. EstateAI with Pixel Perfects: a property-centered hybrid workflow

EstateAI's product overview describes keeping original photos, generated versions, review, and delivery connected to a property. Its stated sequence is to create the property, analyze its photos, choose services, review versions, and deliver approved media.

The professional-editing page describes a route to Pixel Perfects editors for tasks such as complex masking, detailed object removal, retouching, and custom instructions.

The useful evaluation question is whether the brief survives that handoff. In a trial, choose a difficult image, request a manual correction, and check whether the source, requested changes, and returned version remain easy to trace. The public description is a starting point; a real job should establish the operational details.

2. Apply Design: user-controlled staging revisions

Apply Design describes virtual staging with furniture removal, design reuse across images of the same space, and a drag-and-drop editor for revisions.

That suggests a different evaluation emphasis: how much control does the person preparing the listing need over the staged scene?

Use two angles of one room in a trial. Make a specific change, such as replacing a sofa, then inspect both views. Check placement, scale, sightlines, and whether the revision process remains manageable. Multi-angle consistency is a vendor-described capability here, not a measured result from this comparison.

3. BoxBrownie: managed editing and quality assurance

BoxBrownie's workflow describes uploading images, professional editing, a quality-assurance step, and downloading the result. Its site presents a pay-as-you-go service model.

For a managed service, evaluate the brief and correction loop. Can you describe the required change precisely? What happens when the result needs a rework? Who confirms that it is ready for the client?

A provider's internal QA and the agent's final approval answer different questions. Build room for both in the delivery process.

Editorial decision guide: compare revision control and human-review needs before cost and service terms. This is a decision aid, not a performance ranking.

B. Compare the same job, with the same acceptance criteria

These options place control in different parts of the workflow. A useful trial should expose those differences without pretending that a feature list proves image quality.

Prepare a small, authorized test set with:

A vacant room shown from two angles.
A furnished room with a clearly defined removal request.
A difficult area such as a reflection or partly obscured fixture.
A precise brief listing what may change and what must remain faithful to the photographed property.

Run only comparable services against the same brief. Do not score an enhancement service as though it were a full staging service.

Record five things for each result:

Source traceability: Can a reviewer find the original photo and the instructions that produced this version?
Revision control: Can the team tell which version was requested, returned, approved, and delivered?
Correction effort: How many actions and how much active staff time did the requested correction take?
Property fidelity: Did the agreed review identify changes to permanent features, proportions, or views?
Delivery clarity: Can the recipient identify the approved set and distinguish concepts or staged images from original photos?

Record observed results and unresolved questions separately. A vendor promise, a successful test, and a missing answer should not share one score.

C. A small data model can protect the workflow

For developers building the surrounding system, the following is a proposed model, not a description of any provider's internal database:

Source asset: property ID, asset ID, original file reference, and upload time.
Editing job: source asset IDs, requested operation, brief revision, provider, and status.
Output version: job ID, output file reference, creation time, and parent version if revised.
Review decision: exact output version, reviewer, decision, and notes.
Delivery record: exact approved version IDs, recipient or destination, and delivery time.

Keep a completed job separate from an approved output. A job can finish successfully while its result still requires correction.

A manual handoff should carry the source and brief with it. If a new output arrives after approval, create a new review decision rather than silently treating the earlier approval as transferable.

For batch work, track each asset separately. One failed room image should be visible even when the rest of the property is ready.

D. The acceptance test that catches a costly mix-up

Before adopting a workflow, rehearse this sequence:

Generate or commission version A.
Request a correction and receive version B.
Approve B.
Attempt to deliver A from an older browser tab or saved selection.

The system should make the mismatch visible and require an intentional decision. Also check what happens when the client requests another revision after delivery.

This test measures a concrete operational risk. It does not depend on which provider produced the image.

E. Choose around the work your team needs to own

A property-centered hybrid workflow deserves a closer look when preserving context across automated and manual steps is important. An editable staging workflow deserves a closer look when the operator needs direct scene control. A managed editing service deserves a closer look when the team wants to submit a brief and review the returned work.

The final choice needs evidence from your own authorized images, service scope, review process, and actual costs. Public documentation alone cannot establish a universal winner.

For background on deciding which work to automate, see AI vs Human Real Estate Photo Editing. For organizing the source set before editing, see the EstateAI Photo Library guide.

Sources checked: EstateAI, EstateAI professional editing, Apply Design, and BoxBrownie, 20 October 2026.

Should an AI Listing Draft Be Allowed to Overwrite Newer Agent Edits?

Tue, 20 Oct 2026 10:49:18 +0000

Imagine a property agent starts an AI rewrite of a listing description. While it runs, a colleague corrects a factual detail in the same listing. The AI result then arrives, based on the older version.

What should the application do with that result?

A useful distinction is finishing generation versus accepting a change. The generated draft can be retained without replacing the current description.

HTTP already has a building block for this problem. RFC 9110, section 13.1.1 describes If-Match as a way to guard against accidental overwrites when clients update the same resource. With a supporting API, a client sends the strong ETag it read; a failed precondition can produce 412 Precondition Failed.

For a property-content editor, a possible design is:

Record the listing revision used to generate the draft.
Save the AI output as a separate proposal.
Compare that base revision with the current listing when someone applies it.
If they differ, show the original, current, and proposed text for review.

The difficult part is the user experience. A changed punctuation mark and a corrected room count should not necessarily receive the same treatment. A version check catches a conflict; it does not decide whether the new wording is factually safe.

Would you block every stale AI draft, allow field-level merging, or always require a review screen? What would make your choice change?

This is a hypothetical design question, not a claim about a particular product's implementation.

Why Your Product Design Is Costing You Users and You Do Not Know It

Tue, 20 Oct 2026 09:08:55 +0000

There is a version of your product that works perfectly. The features are built. The flows are connected. Every screen renders correctly on every device. You have tested it yourself, and it makes sense to you.

And users are still leaving.

Not because your product does not work. Because it does not communicate clearly enough for someone who is encountering it for the first time, under real conditions, with divided attention and no patience for confusion. The product works. The design is failing them, quietly, in ways that do not show up as errors and do not trigger support tickets. They just leave.

This is the most expensive kind of design problem because it is invisible until you look for it specifically. It does not announce itself as a design failure. It announces itself as a conversion rate that is lower than it should be, a retention number that does not reflect the quality of the product, a user acquisition cost that keeps climbing because you are filling a leaking bucket rather than fixing the leak.

This post is about what those failures look like in practice, why they are so easy to miss, and what it takes to fix them before they cost you another month of users who could have stayed.

Design problems do not announce themselves as design problems. They announce themselves as low conversion, poor retention, and users who churn after a single session without leaving a reason.

WHY GOOD PRODUCTS HAVE BAD DESIGN

The most common reason is that the people who built the product know it too well to see it clearly.

When you have been working on a product for months, every screen makes intuitive sense. You know what the button does. You know where to find the settings. You know that the empty state means you need to create something first. That knowledge is so deeply embedded that it is impossible to unknow it when you test the product yourself.

Your users do not have that knowledge. They arrive at your product with no context, a low tolerance for confusion, and a competing option one tap away. Every moment of uncertainty is a moment where they are deciding whether to push through or leave. Most of them leave without telling you why.

The second reason is that design quality is usually evaluated visually rather than functionally. Does it look good? Does it feel modern? Is the colour scheme consistent? These are valid questions but they are the wrong primary measure. The right primary measure is: does a user who has never seen this product before know what to do within the first ten seconds? Everything else is secondary to that.

THE DESIGN FAILURES THAT COST YOU THE MOST

No clear primary action

Every screen in your product should have one thing it is primarily asking the user to do. One button, one link, one action that is visually dominant and immediately obvious. When a screen has three buttons of equal size and equal visual weight, the user does not see three choices. They see a decision they have to make before they can proceed, and many of them will not make it.

A common version of this failure: an onboarding screen that asks the user to connect an integration, invite a team member, or complete their profile, presented as three equally prominent options. The user came to use the product, not to manage their account setup. The right design leads them to the value first and surfaces the setup options progressively. The wrong design stalls them at the threshold and calls it onboarding.

Forms that ask for too much too early

Every field you add to a form is a question you are asking the user to answer before they have received any value from your product. Early in a user journey, before trust is established, before the user has experienced what your product does for them, that question has a cost. The longer the form, the higher the cost, and the more users abandon before completing it.

A registration form that asks for name, email, password, company name, company size, role, and phone number is not collecting useful data. It is filtering out users who are not willing to invest that much before seeing anything. Most of them are not unwilling to give you that information eventually. They are unwilling to give it before they have a reason to trust you with it.

The fix is not to remove fields permanently. It is to defer them. Ask for the minimum required to create an account. Let the user experience the product. Ask for the rest when you have given them a reason to provide it.

Error messages that describe what went wrong without explaining what to do

Error messages are the moment a product most needs to communicate clearly and most often fails to. A user who hits an error is already frustrated. An error message that says "Invalid input" or "Something went wrong" gives them nothing to work with. They do not know what was invalid, what went wrong, or what action would resolve it. Many of them close the tab.

A form that rejects a password and says "Password does not meet requirements" without showing those requirements anywhere on the screen is a form that will lose a measurable percentage of its signups at that exact moment. A checkout that fails and says "Payment declined. Please try again" without suggesting an alternative payment method or explaining why the payment was declined is a checkout that is losing revenue at its most critical point.

Every error message in your product should answer three questions: what happened, why it happened, and what the user should do next. If it only answers the first one, it is an error message that creates friction rather than resolving it.

An error message that describes what went wrong without explaining what to do next is not helpful. It is a dead end dressed up as communication.

Products built by developers often have navigation structures that mirror the underlying data model or the feature development sequence. The sections are named for what they are technically rather than what they mean to the user. The hierarchy reflects the engineering architecture rather than the user journey.

A user looking for their billing information should not have to choose between Account, Settings, and Profile before finding it. A user trying to invite a team member should not have to navigate through three levels of menu to reach a function they will use regularly. Navigation that makes technical sense and user sense are different things, and conflating them is one of the most common sources of silent user frustration.

The test for navigation is simple: give the product to someone who has never used it and ask them to complete a specific task without guidance. Where they hesitate, where they click the wrong thing, where they express uncertainty, is where the navigation is failing them.

Empty states that leave users stranded

Every new user encounters your product in its empty state: no data, no history, no content, no evidence of what the product does or why they should keep going. This is the moment most products lose the users who were genuinely interested but needed one more signal to commit.

An empty state that shows a blank screen with a small grey message that says "No items yet" is an empty state that offers nothing. An empty state that shows what the product looks like when it is working, explains the first action the user should take, and makes that action immediately available is an empty state that converts curiosity into commitment.

The empty state is not an afterthought. It is the first real experience a new user has of your product in use. Designing it with the same care as the populated states is not extra work. It is the work that determines whether a new user becomes a retained user.

Mobile experiences that were designed on a desktop

A product designed on a large monitor by someone using a mouse will have interaction patterns optimised for that context. Hover states that reveal information. Small click targets that a cursor navigates precisely. Navigation placed at the top of the screen where a thumb cannot comfortably reach. Dense information layouts that make sense on a 27-inch display and become overwhelming on a 6-inch one.

When that product is used on a phone by someone in a real-world context, one-handed, distracted, in variable lighting, those decisions become friction. The user does not diagnose the problem as a design failure. They just find the product harder to use than they expected and stop using it.

If more than thirty percent of your users are on mobile and your product was not specifically designed for that experience, you have a gap between your actual user behaviour and your design assumptions that is costing you retention every day.

WHY THESE PROBLEMS ARE SO HARD TO SEE FROM INSIDE

The people closest to the product are the worst positioned to see these failures. Not because they lack skill, but because they lack distance. Every confusion that would stop a new user is invisible to someone who knows the product well enough to navigate around it unconsciously.

The most reliable way to see these problems is to watch a real user use the product for the first time without helping them. Not a user test with instructions and prompts. A genuine first-time session where you observe without intervening. What you see in thirty minutes of watching someone encounter your product fresh will surface more design problems than months of internal review.

The second most reliable method is your own data. Screens with high drop-off rates are screens with design problems. Forms with high abandonment rates have specific fields where users are stopping. Flows with low completion rates have a specific point where most users are leaving. The data does not tell you what the problem is. It tells you exactly where to look for it.

WHAT FIXING THESE PROBLEMS ACTUALLY REQUIRES

Most of the failures described in this post are not expensive to fix once they are identified. Clarifying a primary action is a design change, not a rebuild. Simplifying a registration form is a product decision, not a development project. Improving an error message is an afternoon of work. Redesigning an empty state is a focused sprint.

What they require is the willingness to look at the product through the eyes of someone who does not know it, to take the friction they experience seriously rather than explaining it away, and to treat the user experience as a product metric rather than an aesthetic concern.

The products that retain users are not always the ones with the most features or the best technology. They are the ones that make the first ten minutes feel effortless, because someone thought carefully about what a new user needs at each moment and designed for that rather than for the team that built it.

CLOSING THOUGHT

Your product is probably better than your metrics suggest. The gap between the product you built and the results you are seeing is often a design gap rather than a feature gap. Before adding the next feature, it is worth asking whether the features you already have are communicating clearly enough for the users who are encountering them for the first time. The answer, in most products, is that they are not. And the cost of that gap compounds with every user you acquire into it.

If your product is losing users at a specific point in the flow and you are not sure why, that is exactly the kind of problem I work on. Tell me about what you are seeing and I can tell you whether design is where the answer lives.

AI Building Its Own Successor: Are Developers Ready?

Tue, 20 Oct 2026 08:17:53 +0000

AI Is Starting to Build Its Own Successor. Are Developers Ready?

In May 2025, Google DeepMind introduced something that raised an interesting question about the future of artificial intelligence.

Its system, AlphaEvolve, could generate algorithms, test alternatives, and identify improvements to computing infrastructure. Some of those improvements were applied to systems used in AI development itself.

AI was no longer simply helping programmers write software. It was contributing to the engineering processes behind future AI systems.

This is an important development in AI-assisted AI development, but it needs context. Improving an algorithm or optimizing model-training infrastructure is not the same as an AI independently designing, training, and deploying a more intelligent successor.

That level of unrestricted recursive self-improvement has not been demonstrated.

Still, the direction of research deserves attention.

When machines begin contributing to algorithm discovery and scientific experimentation, developers face a different engineering challenge: evaluating whether the proposed improvements actually work.

Understanding that challenge means examining what current systems have achieved, where their limitations remain, and how software engineering responsibilities may change.

When AI Becomes Part of the Research Process

AI coding assistants have become familiar tools for generating functions, explaining code, and suggesting corrections.

Research systems are beginning to take a different approach.

Instead of producing one implementation for a developer to inspect, they can generate multiple candidate solutions, test them against defined objectives, and use the results to guide additional experiments.

Consider a hypothetical engineering team trying to optimize a scheduling algorithm.

A conventional approach might involve profiling performance, identifying bottlenecks, modifying the algorithm, and running benchmarks.

An AI-assisted experimentation system could explore several implementations automatically.

Some candidates might improve execution time. Others might reduce memory usage. Many would fail correctness tests or offer no meaningful improvement.

The interesting part is how successful candidates are identified.

A language model can propose code, but proposing code does not prove that it works. Automated testing and evaluation provide the feedback needed to distinguish promising changes from ineffective ones.

Earlier research already demonstrated elements of this approach. In 2023, Google DeepMind reported that AlphaDev discovered improved sorting algorithms, with selected results incorporated into the LLVM C++ library.

The broader implication is that some engineering tasks can be treated as search problems: generate alternatives, measure results, and retain improvements.

That creates new opportunities for automation, especially when objectives are measurable.

AlphaEvolve: An Early Glimpse of AI Improving AI Infrastructure

AlphaEvolve offers one of the clearest examples of AI contributing to the development of technology used by other AI systems.

Introduced by Google DeepMind in May 2025, the system combines Gemini language models, automated evaluation, and evolutionary search.

Its purpose is to discover and optimize algorithms across mathematical and computing problems.

How AlphaEvolve Works

The process begins with a problem that can be expressed in code and evaluated programmatically.

Gemini models propose possible solutions. Automated evaluators then execute and score those candidates according to criteria such as correctness or performance.

Promising implementations are retained and used to inform future proposals.

Over repeated iterations, the system explores a broader range of algorithmic possibilities than a single model response would provide.

The basic workflow looks like this:

Problem Definition → Candidate Algorithms → Automated Evaluation → Selection → Further Optimization

Unlike ordinary code completion, this process includes a feedback mechanism that helps determine which changes are worth keeping.

What Has AlphaEvolve Achieved?

Google reported several concrete applications.

One involved data center scheduling. According to DeepMind, an AlphaEvolve-discovered heuristic helped recover an average of 0.7% of Google's worldwide computing resources.

Another involved optimizing a matrix multiplication kernel used in Gemini's architecture. Google reported a 23% speedup for that kernel, contributing to approximately a 1% reduction in model training time.

These are specific, Google-reported results, not performance improvements that automatically apply to other systems.

Source: Google DeepMind — AlphaEvolve: A Gemini-Powered Coding Agent

Why does this matter?

Training and operating advanced AI models depend on enormous amounts of computation. Even relatively small efficiency gains can have practical value when an operation runs repeatedly at scale.

But AlphaEvolve's success also reveals an important limitation.

The system is particularly effective when candidate solutions can be evaluated objectively. It does not demonstrate that AI can independently solve every research problem or guarantee that generated algorithms are suitable for production.

Its results depend on the quality of the evaluation process and subsequent engineering validation.

The Difference Between AI-Assisted Progress and Recursive Self-Improvement

The phrase AI improving AI sounds straightforward, but it describes several different levels of capability.

Understanding those differences helps prevent genuine research achievements from becoming exaggerated predictions.

AI-Assisted Optimization

An AI model proposes improvements to an algorithm, software component, or computing process.

The proposed changes are evaluated using predefined tests or performance objectives.

Human engineers still define the problem and decide how validated improvements should be implemented.

AI-Assisted Research

Research systems help generate hypotheses, analyze literature, compare possible explanations, and suggest experimental directions.

They may reduce the effort required to explore a large research space, but scientific findings still depend on suitable evidence.

Autonomous Experimentation

An automated system may coordinate several stages of experimentation, including generating candidates, running tests, analyzing feedback, and selecting the next experiment.

The degree of autonomy depends on its permissions, capabilities, and operating environment.

Recursive Self-Improvement

Recursive self-improvement describes a more ambitious possibility: an AI system repeatedly improves its own capabilities, with those improvements enabling further advances.

Improving one algorithm used in AI training does not establish that an entire AI system can autonomously improve its general intelligence.

AI Co-Scientists and the New Research Workflow

Algorithm optimization is not the only area where AI has begun contributing to research.

In February 2025, Google Research introduced an AI co-scientist designed to assist scientists with generating hypotheses and developing research proposals.

The system uses multiple specialized agents built around Gemini models.

These agents perform activities such as hypothesis generation, reflection, ranking, and refinement.

Given a research objective, the system can explore different explanations and propose directions for further investigation.

Google reported applications involving biomedical research, including drug repurposing and antimicrobial resistance.

Source: Google Research — Accelerating Scientific Breakthroughs With an AI Co-Scientist

One especially interesting example involved bacterial gene-transfer mechanisms.

Researchers reported that the system independently proposed a mechanism consistent with findings scientists had already established experimentally but had not yet publicly disclosed.

That is a meaningful research demonstration. It does not mean the AI independently performed the experiments or established a new scientific fact without human involvement.

A plausible hypothesis and a verified discovery are different things.

The former suggests something worth investigating. The latter requires sufficient evidence, appropriate experiments, and reproducible findings.

AI-assisted scientific discovery may help researchers explore ideas more efficiently, but it cannot eliminate the need for scientific judgment.

The Problem Nobody Can Skip: Who Verifies the Improvements?

Suppose an AI system discovers a scheduling algorithm that performs 15% faster than an existing implementation.

Would that make it a better algorithm?

Not necessarily.

The answer depends on what was measured, which inputs were tested, and whether the optimization introduces other problems.

A scheduling algorithm might perform well under average workloads but become unreliable when requests arrive simultaneously.

A generated implementation could also consume more memory, introduce security vulnerabilities, or fail under unusual conditions.

When the Benchmark Measures the Wrong Thing

Automated systems optimize against the feedback they receive.

If an evaluator measures only execution speed, it may reward a solution that sacrifices correctness.

If tests cover only common inputs, the system may produce changes that fail on edge cases.

This creates a fundamental challenge for autonomous AI research: an optimization is meaningful only when the evaluation process measures what actually matters.

The problem becomes harder when objectives are ambiguous.

Scientific novelty, code maintainability, production reliability, and security cannot always be reduced to a single numerical score.

Why Reproducibility Matters

An improvement discovered in one environment may not transfer to another.

Hardware configuration, compiler versions, datasets, runtime dependencies, and execution settings can all influence results.

Developers therefore need reproducible experiments.

A credible AI-assisted optimization workflow should preserve:

The original implementation and proposed changes.

Test datasets and evaluation criteria.
Environment and dependency versions.
Performance results and failure cases.
Human review and approval records.

Security deserves particular attention.

An AI agent executing generated code should operate within clearly defined permissions and controlled environments.

Automated experimentation should not grant unrestricted access to production systems.

Even when automated tests confirm functional behavior, engineers still need to examine broader operational consequences.

The ability to generate more solutions increases the importance of deciding which evidence is sufficient to trust them.

The Developer's Role Is Changing, Not Disappearing

AI coding systems are improving, but their effect on actual software development is more complicated than faster code generation suggests.

In July 2025, research organization METR published a randomized study examining how AI tools affected experienced open-source developers working on familiar repositories.

The researchers found that participants took 19% longer to complete selected tasks when using the AI tools available during the study.

Source: METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

That result does not establish that AI tools always reduce productivity. It reflects particular developers, tasks, tools, and experimental conditions.

In February 2026, METR reported that newer tools showed signs of improving productivity, while participant-selection effects prevented reliable estimates from its updated experiment.

Source: METR — Developer Productivity Experiment Update

These findings illustrate why productivity claims require careful measurement.

Generating code quickly is not identical to delivering working, maintainable software quickly.

Developers still need to understand requirements, investigate unexpected behavior, design systems, and verify that changes do not create new problems.

As AI-assisted AI development advances, these responsibilities may become more important.

Engineers may spend less time manually exploring every implementation and more time defining objectives, reviewing experiments, and evaluating results.

There is genuine uncertainty about how employment and team structures will change.

What is clearer is that trustworthy AI-assisted engineering requires expertise beyond operating a coding assistant.

What Comes Next for AI-Assisted Research?

The next stage of AI-assisted research will probably involve deeper integration between language models, development tools, evaluators, and scientific workflows.

Some developments are relatively easy to imagine because their foundations already exist.

Automated systems may become better at generating alternative implementations, executing controlled experiments, and identifying optimization opportunities.

Specialized research agents may help organize literature, compare hypotheses, and prepare experimental designs.

AI coding systems may also become more effective at longer engineering tasks.

However, longer task execution does not automatically mean reliable autonomy.

Research from METR examines the duration of software tasks that AI agents can complete at specified success rates.

Its work highlights progress in agent capabilities while emphasizing that results are tied to particular benchmarks and evaluation conditions.

Source: METR — Measuring AI Ability to Complete Long Software Tasks

More ambitious possibilities remain uncertain.

A system might eventually coordinate complex research projects with limited human intervention. It might also encounter difficulties involving incomplete evidence, poorly defined objectives, or errors accumulated across many steps.

Predictions about unrestricted recursive self-improvement go considerably further.

Today's algorithm-discovery systems provide evidence that AI can contribute to parts of the development process. They do not establish a timeline for autonomous systems capable of independently designing and validating more intelligent successors.

The practical question is how much of the engineering and research workflow can be automated reliably.

Conclusion: Progress Is Real. Verification Still Matters.

AI is beginning to contribute to the algorithms, computing systems, and research processes that influence future AI development.

AlphaEvolve demonstrates how automated evaluation can help discover useful algorithmic improvements. AI co-scientist research shows how models can assist with generating and refining scientific hypotheses.

These achievements are significant, but their limitations are equally important.

A proposed improvement still needs verification. A promising scientific hypothesis still requires evidence. An AI-generated implementation must still meet real engineering requirements.

For developers, the emerging challenge is not simply learning to generate more code with AI.

It is learning how to design better evaluations, investigate generated solutions, and build systems where automated experimentation produces trustworthy results.

As AI becomes more capable of proposing its own improvements, could verifying those improvements become one of the most valuable skills in software engineering?

Frequently Asked Questions

1. Can AI really improve itself?

AI can optimize selected algorithms and infrastructure used in AI development. However, these achievements do not demonstrate unrestricted autonomous recursive self-improvement.

2. What is AlphaEvolve?

AlphaEvolve is a Google DeepMind system that combines language models, automated evaluation, and evolutionary search to discover and optimize algorithms.

3. Can AI independently conduct scientific research?

AI can assist with hypothesis generation, literature analysis, and some automated experiments. Scientific conclusions still require appropriate validation, and reliable general-purpose autonomous research remains an open challenge.

4. Will AI replace software developers?

Future employment effects remain uncertain. AI may automate some implementation tasks, while architecture, testing, security, verification, and production reliability remain essential engineering responsibilities.

Your TypeScript Is Lying to You, and These 5 Places Hide Real Bugs

Mon, 19 Oct 2026 12:30:35 +0000

TypeScript's most dangerous keyword is two letters long.

as does not convert anything. It does not check anything. It tells the compiler "trust me", and the compiler does. If you are wrong, the error moves from your editor to your users.

Here are five places I keep finding it in real codebases, each one compiling cleanly and each one a crash waiting for the right input.

1. Casting an API Response

const res = await fetch('/api/users/1');
const user = (await res.json()) as User;

console.log(user.profile.avatarUrl);

res.json() returns Promise<any>. Casting any to User is the compiler equivalent of a handshake agreement. If the backend renames a field, makes profile optional, or returns an error object with a 200 status, this line still compiles and the crash happens three functions later with a message that has nothing to do with the cause.

What to write instead: validate the data at the boundary.

import { z } from 'zod';

const UserSchema = z.object({
  id: z.string(),
  email: z.string().email(),
  profile: z.object({ avatarUrl: z.string().url().nullable() }),
});

type User = z.infer<typeof UserSchema>;

const user = UserSchema.parse(await res.json());

Now the type is derived from a check that actually ran. If the shape is wrong, you get a clear error at the exact line where bad data entered your app.

2. The Non-Null Assertion on Environment Variables

const client = new MongoClient(process.env.DATABASE_URL!);

process.env.X is typed string | undefined, which is honest. The ! deletes the undefined from the type and nothing else. When the variable is missing in a new environment, you get a confusing failure from deep inside a library instead of "DATABASE_URL is missing".

function requireEnv(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`Missing environment variable: ${name}`);
  return value;
}

const client = new MongoClient(requireEnv('DATABASE_URL'));

The app now fails at startup, with the variable name in the message. That is the cheapest place to find out.

3. The Non-Null Assertion on `find()`

const user = users.find((u) => u.id === id)!;
sendEmail(user.email);

find returns undefined when nothing matches, and "nothing matches" is exactly the case your tests forgot. The ! makes the compiler stop warning you about the one scenario worth warning about.

const user = users.find((u) => u.id === id);
if (!user) throw new Error(`User ${id} not found`);

sendEmail(user.email);

After the check, TypeScript narrows user to a real object on its own. No assertion required.

4. Casting Object.keys

type Settings = { theme: string; lang: string };

function logSettings(s: Settings) {
  (Object.keys(s) as (keyof Settings)[]).forEach((k) => {
    console.log(s[k].toUpperCase());
  });
}

const fromApi = { theme: 'dark', lang: 'en', retries: 3 };
logSettings(fromApi);

This compiles. TypeScript types are structural, so an object with extra properties is still a valid Settings when it arrives through a variable. Object.keys returns plain string[] for exactly this reason, and the cast to keyof Settings overrides that protection. At runtime retries is a number, and toUpperCase is not a function.

const SETTING_KEYS = ['theme', 'lang'] as const;

function logSettings(s: Settings) {
  SETTING_KEYS.forEach((k) => console.log(s[k].toUpperCase()));
}

Iterate over a list of keys you control, not over whatever the object happens to contain.

5. Casting the Caught Error

try {
  await saveProfile(data);
} catch (e) {
  toast((e as Error).message);
}

In JavaScript you can throw anything: a string, a number, undefined, a plain object. Libraries do it. With strict on, TypeScript types e as unknown to keep you honest, and as Error is the line that undoes it. If something throws a string, .message is undefined and your toast says "undefined".

catch (e) {
  const message = e instanceof Error ? e.message : 'Something went wrong';
  toast(message);
}

A Bonus Lie That Lives in Your Config

const items: Item[] = [];
const first = items[0];
first.name; // typed as Item, crashes at runtime

Array index access is typed as Item, not Item | undefined, because the compiler cannot know the length. One line in tsconfig.json fixes it:

{
  "compilerOptions": {
    "strict": true,
    "noUncheckedIndexedAccess": true
  }
}

This flag is not part of strict, so most projects never turn it on. It adds a few extra checks, and every one of them is a real case where the array could have been empty.

The Rule

A cast is a promise you make to the compiler that it cannot verify. Data from outside your program needs a runtime check, not a type assertion.

An easy audit: search your project for as and for !. and !;. For each hit, ask what happens when the assumption is false. Casts on your own literals are usually fine. Casts on network responses, environment variables, search results and caught errors are where the bugs live.

I keep these checks in the shared lib/ folder of the TypeScript-first templates on pixelanas.com, so a missing environment variable or a changed API shape fails loudly on the first request instead of silently in production.

Get the templates: https://pixelanas.gumroad.com

How many as casts does your project have, and how many of them are on data you do not control? Drop your count in the comments.

Anas, full-stack Next.js developer building SaaS products and premium templates. X: @ASheikh69751

AI Hasn’t Shifted The Bottleneck From Coding To Code Review

Mon, 19 Oct 2026 12:09:32 +0000

Why we’ve stopped noticing the real improvement opportunity

Since AI arrived, many people have mentioned that the bottleneck has shifted from coding to code review. This isn’t quite right, as coding wasn’t the bottleneck in the first place, and it’s not code review now. The reason we think either of these things is constraining the flow of value is that mossy hill we all stare past when we look at the mountains.

Here’s a simple test. For the application or service you work on, how many changes have passed code review but haven’t yet been deployed and enabled for users? If the answer is none or one, accept my apologies: in your specific case, I’m wrong. But I rarely get that answer. It’s usually more than one and that tells you the bottleneck is elsewhere.

Number of changes per deployment batch (Source: Octopus Deploy)

We’re conducting original research in this area right now, and half of all teams have between 2 and 10 changes sitting in a batch, and a quarter have 11–50. Overall, more than 90% of teams ship in batches rather than one change at a time.

This number reveals an industry-wide visibility gap. People believe Claude Code, Cursor, and GitHub Copilot have shifted the bottleneck from coding to code review, but that ignores everything that happens after the review and that’s not a personal failing, it’s an industry-wide misperception.

We’ve grown so used to working in batches that the practice looks like it belongs. It’s overgrown with moss, indistinguishable from the surrounding hills. When you search for ways to speed up software delivery, you won’t see it, because it doesn’t look like a problem. It looks just as things have always been.

What happens when AI floods a batch

Writing code is a small part of a longer value stream that starts with an opportunity and ends when a user gets the value they need. The impact across this whole value stream is lumpy. AI will help more in some areas than others, and the end-to-end benefit depends on you noticing the areas where work accumulates.

GitLab’s 2026 AI Accountability Report found 85% of respondents agree AI has shifted the bottleneck from writing code to reviewing it. Yet we can see from deployment batches that 92% of these people are likely wrong, because if there’s accumulation after the code review, it means code review isn’t the bottleneck. It also means speeding up the reviews will make the real bottleneck worse.

This isn’t to say increased coding speed doesn’t put pressure on code review. Faros AI’s research across 10,000 developers found that teams with high AI adoption merge 98% more pull requests, but review time for those changes grows 91%, and average pull request size grows 154%. Cursor’s own study, run with a University of Chicago economist, found companies merge 39% more pull requests once its coding agent becomes the default.

However, approving changes faster is a strategy that only works if the change then flows smoothly to production. In the majority of cases, it simply moves into the queue of changes awaiting some other handling, like testing and deployment. If you up the rate and size of changes passing through the review stage, pressure is simply transferred to the real bottleneck.

Code review looks like a constraint only because it has a visible queue, while the downstream queues are hidden by their general acceptance across the industry. The job of your pipeline is to get changes to production, where they can be used, not to gather them in a “pending deployment” queue.

And while all those unreleased changes are building up, risk accumulates with it.

Batches are signposts

Ask the batch-size question, and you’ll find what’s really constraining your value stream: a manual verification step, a cumbersome change approval or release train process, or no easy way to deploy changes. Not coding. Not code review.

It’s likely you were working in batches before you started your AI initiative. The introduction of AI will increase the size of your batches and that can cause problems. Increasing code review throughput doesn’t solve the problem, it simply moves changes to the bottleneck faster.

Using the true constraint to set the pace of your whole value stream will help you invest in solving the right problem. If your retrospectives keep failing to produce noticeable improvements, you’re likely missing the batch problem. It’s why some AI initiatives pay off while others flop.

The studies miss it, too

The research into the impact of AI is useful, but like many studies on this topic, it stops at the point code gets merged. It looks at pull requests opened, pull requests merged, hours spent reviewing. None of it asks how long changes wait after the review, or how many are bundled together before anyone sees them in production. Without that number, you can’t find the real constraint.

Having invested in AI to speed up coding, you’ll be tempted to fix code review next, or give up on code reviews altogether. If you’re shipping in batches, neither will make much difference to how quickly you remove risk or deliver value to the people using the software.

The reason your organization resists fixing the batch problem is the real problem you need to solve.

Database Migrations: Can Your Previous Release Still Run?

Mon, 19 Oct 2026 07:45:55 +0000

A deployment can roll back cleanly while the database stays incompatible with the old application. Before shipping a schema change, check whether the previous release can still read and write the data that the new release creates.

Consider a fictional account service. Version 1 reads a column called display_name. Version 2 expects public_name. Renaming the column and deploying the new code together looks tidy, until an old worker, a rolling-deployment instance, or an emergency rollback still asks for display_name.

The migration succeeded. The service can still fail.

Separate compatibility from cleanup

Danilo Sato’s explanation of parallel change describes three stages: expand, migrate, and contract. The useful idea is to introduce the new interface while the old one remains available, move consumers, then remove the old interface.

For a database rollout, turn that idea into explicit release gates:

Expand: introduce the new storage shape without removing what the currently deployed application needs.
Migrate: move reads and writes deliberately, checking compatibility during the transition.
Contract: remove the old shape only after its consumers and the intended rollback window are accounted for.

These stages are a planning model, not a guarantee of zero downtime. The exact migration still depends on the database, data volume, application behavior, and operational limits.

Work through the awkward middle

In the account-service example, adding public_name is only the beginning. Before copying values, decide which field is authoritative at each stage.

A possible transition is to deploy compatibility code that continues serving old readers while maintaining both values. If both columns live in the same row, update them in the same database transaction where practical. That still leaves an important question: can any old application instance or background job write only the original column?

Do not switch reads to the new column while those writers can silently make it stale. Either finish upgrading writers first or design a deliberate synchronization mechanism. A fallback read helps with missing values; it does not detect an outdated non-empty value.

Backfills need the same care. A naive copy can overwrite a newer value if it races with a live update. Define an update condition, concurrency strategy, and restart behavior before running the job. Record progress so an interrupted backfill can resume without blindly replaying the entire table.

Test the rollback with new data

A useful rehearsal includes more than deploying version 2 successfully:

Start with representative data, including empty and unusual values.
Apply the expansion and deploy the compatibility release.
Run the backfill alongside representative writes.
Create and edit records using the new release.
Restore the application version you actually intend to roll back to.
Verify that it can read and update those new records correctly.

Include background workers, scheduled jobs, exports, and direct database consumers in that exercise. A healthy web endpoint does not prove that every writer is compatible.

Use synthetic or appropriately protected test data. This checklist proposes evidence to collect; it does not claim that a production migration has been tested.

A compatible change can still block traffic

In PostgreSQL, ALTER TABLE generally acquires an ACCESS EXCLUSIVE lock unless the documentation specifies otherwise. Compatibility at the application level does not eliminate lock waits.

PostgreSQL also supports separating certain constraint additions from validation with NOT VALID and VALIDATE CONSTRAINT. The permitted constraint types and locking details matter. Check the documentation for your deployed version rather than applying the technique to every schema operation. See the PostgreSQL ALTER TABLE reference.

Before execution, define acceptable lock wait, query latency, replication lag, and backfill load. Decide who can stop the rollout and what the safe next step is if a threshold is crossed.

Make cleanup its own decision

Keep a short release note that answers four questions:

Which application versions can run against this schema?
Which data changes can the previous version understand?
What evidence allows the old column or interface to be removed?
After removal, does recovery mean application rollback, a forward fix, or data restoration?

Give the cleanup an owner. Leaving parallel paths indefinitely creates its own maintenance cost, but removing them too early destroys the compatibility you built.

The practical takeaway: rehearse rollback against data written by the new release before treating it as a recovery plan.

What is the most easily missed database consumer in your deployment process?

Waqas Ahmad | Software Engineer | Content Creator

Build a Search Box That Won’t Show Stale Results

Sun, 18 Oct 2026 10:08:35 +0000

A search box can display the wrong answer even when every request succeeds.

Imagine someone types cat, pauses long enough to trigger a request, then adds an s. The request for cats finishes first. A moment later, the older cat request finishes and replaces the results. The input says one thing; the page answers something else.

A reliable implementation needs a clear rule: only work associated with the latest input may change the search UI. That rule must cover results, errors, and loading indicators.

Here, “stale” means belonging to an older input. Data freshness inside your backend or cache is a separate concern.

A. Give each mechanism one job

Three pieces work together:

Debouncing delays starting a request while the user keeps typing.
Cancellation stops an obsolete fetch when possible.
A version guard decides whether completed work still owns the UI.

Debouncing alone leaves previously started requests running. Cancellation helps with those requests, but a version guard makes the UI rule explicit, including for future asynchronous work that does not support cancellation.

The subtle detail is when to advance the version: immediately when input changes. If you wait until the next request starts, an old response can still render during the debounce interval.

B. Start with a small page and an explicit response shape

<label for="query">Search articles</label>
<input id="query" type="search" autocomplete="off">
<p id="status" role="status">Type to search.</p>
<ul id="results" aria-busy="false"></ul>
<script src="search.js" defer></script>

The example expects a same-origin endpoint, /api/search?q=..., that returns JSON like this:

{"items": [{"title": "Understanding browser requests"}]}

That endpoint is an application dependency; the browser code does not implement it. An empty search response should be {"items": []}. This example treats a missing or malformed body as an error.

C. Invalidate first, then schedule

Put this first part in search.js:

const input = document.querySelector("#query");
const status = document.querySelector("#status");
const results = document.querySelector("#results");

let version = 0;
let timer;
let activeController;

function scheduleSearch() {
  const mine = ++version;
  clearTimeout(timer);
  activeController?.abort();
  activeController = undefined;

  const query = input.value.trim();
  results.replaceChildren();
  results.setAttribute("aria-busy", "false");

  if (!query) {
    status.textContent = "Type to search.";
    return;
  }

  status.textContent = "Waiting for you to finish typing…";
  timer = setTimeout(() => search(query, mine), 250);
}

Every edit clears the timer, invalidates older work, and aborts the active request. Clearing the field follows the same path, so a pending request cannot refill an intentionally empty results list.

The example clears visible results immediately. If your product keeps previous results visible while loading, label which query they belong to so users do not mistake them for the current answer.

The 250 ms delay is an illustrative product choice. Adjust it for your interface. Timers also do not promise exact execution times; busy execution and browser throttling can delay them. See MDN’s timer behavior notes.

D. Guard every asynchronous outcome

Add the request function and event listener below the first block:

async function search(query, mine) {
  if (mine !== version) return;

  const controller = new AbortController();
  activeController = controller;
  status.textContent = "Searching…";
  results.setAttribute("aria-busy", "true");

  try {
    const params = new URLSearchParams({ q: query });
    const response = await fetch(`/api/search?${params}`, {
      signal: controller.signal
    });

    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    const data = await response.json();
    if (mine !== version || controller.signal.aborted) return;

    if (!Array.isArray(data?.items) ||
        !data.items.every(item => typeof item?.title === "string")) {
      throw new Error("Unexpected search response");
    }

    const rows = data.items.map(item => {
      const li = document.createElement("li");
      li.textContent = item.title;
      return li;
    });
    results.replaceChildren(...rows);
    status.textContent = rows.length
      ? `${rows.length} result${rows.length === 1 ? "" : "s"}.`
      : "No results found.";
  } catch (error) {
    if (mine !== version || controller.signal.aborted) return;
    status.textContent = "Search failed. Edit the query to try again.";
    console.error("Search failed:", error);
  } finally {
    if (mine === version) results.setAttribute("aria-busy", "false");
    if (activeController === controller) activeController = undefined;
  }
}

input.addEventListener("input", scheduleSearch);

Each request gets its own controller. An aborted signal cannot be reset and reused for a later request. Passing that signal to fetch connects the request to cancellation. See MDN’s AbortSignal documentation.

URLSearchParams handles query-string encoding, including characters such as & and +. Pass the original query value rather than pre-encoding it. See MDN’s URLSearchParams reference.

There are two asynchronous boundaries here: receiving the response and reading its JSON body. Aborting can also interrupt body consumption. The guard after response.json() checks ownership before any results are rendered. MDN explains the fetch and response-body lifecycle.

Notice that the catch block performs the same ownership check. An obsolete request's failure should not replace a newer success with an error message. Cancellation caused by another edit is expected behavior and stays quiet.

Cleanup needs ownership checks too. An older finally block must not turn off a newer request's loading state or discard its controller. The version protects the loading state; the identity comparison protects the controller reference.

HTTP errors deserve separate handling. A 404 or 500 response does not automatically reject the fetch promise. This example checks response.ok, which is true for statuses 200 through 299, and throws for unsuccessful responses. See MDN’s Response.ok reference.

The response shape is validated before rendering, and titles are assigned through textContent. That keeps API-provided titles in the text path instead of interpreting them as HTML. See MDN’s textContent reference.

E. Test the order of events deliberately

Fast local responses can hide this bug. Use controlled promises and fake timers so tests choose exactly when each request completes.

Cover these scenarios:

Rapid typing: several edits within the debounce window produce one request for the latest value.
The debounce gap: request A starts, the input changes, and A resolves before request B starts. A must not render.
Reversed completion: B resolves, then A resolves. B's results remain visible.
Cleared input: an old response arrives after the field is emptied. The empty state remains.
A stale failure: A rejects while B is loading. B keeps its loading state and controller.
Delayed body consumption: A's response headers arrive, but its JSON body completes after a newer search. A still cannot render.
Current failures: HTTP errors, network errors, malformed JSON, and an unexpected response shape produce an error state.
Zero matches: an empty items array produces the no-results state rather than an error.

For the ownership tests, deliberately let the fake fetch resolve even after its signal is aborted. That exercises the guard independently from cancellation. It does not represent normal aborted-fetch behavior.

A deterministic harness for this example passed 12 checks covering those cases, encoding, and text assignment. It used fake DOM objects and fetch promises; it did not verify browser rendering, accessibility announcements, or a real server.

F. Carry the rule into production

If the component can be removed, add cleanup that advances the version, clears the timer, aborts active work, and removes its listeners.

Also check input methods used by your audience. An IME-aware version can defer requests during composition while still invalidating obsolete work immediately. If application code sets the input value directly, call the scheduling function explicitly: programmatic value changes do not trigger the normal input event. See MDN’s input event reference.

The durable rule is simple: give each search a version, and let only the current version publish an outcome. Keep that rule intact when adding caching, retries, pagination, or other asynchronous steps.

What evidence should an AI coding assistant provide before saying a bug is fixed?

Sun, 18 Oct 2026 10:06:10 +0000

An AI coding assistant can produce a plausible patch and a confident explanation in the same response. What evidence should accompany that explanation before a team treats the issue as resolved?

A useful baseline might include:

A reproduction that fails before the change and passes afterward
The exact tests or commands run, with their outcomes
An explanation connecting the change to the reported failure
A clear statement of anything that could not be checked

The difficult part is deciding how much evidence is enough. A typo in a static label and a change to payment processing deserve different levels of verification. Passing tests also leave an important question: do those tests cover the behavior the user actually reported?

For a timing bug, a happy-path test seems especially weak. A stronger check would deliberately reverse the order of events, delay a response, or force a failure at the point where the bug appears.

Would you require a regression test for every AI-generated bug fix, or allow a documented manual check for smaller changes? And should assistants distinguish between “patch prepared,” “automated checks passed,” and “verified in the target environment” instead of compressing all three into “fixed”?

Curious where you would set the minimum bar, especially when the assistant cannot access the production environment.

OAuth Is Not “Login.” I Found That Out the Hard Way in Production

Fri, 16 Oct 2026 17:09:26 +0000

Most people, including a lot of engineers, think OAuth means “the button that says Sign in with Google.” That’s not wrong exactly, but it’s like saying a car is “the thing with a steering wheel.” Technically true, misses basically everything that matters.

I want to actually walk through what OAuth is, why it exists, what it costs you, and the specific way it bit me in production, because the bug taught me more about OAuth than any tutorial did.

If I use a term that sounds like engineer-speak, I’ll flag it like this:

🚨 Jaggy jaggy alert: here’s the jargon, here’s what it actually means in plain English.

Let’s go.

What OAuth actually is
OAuth is not a login system. OAuth is a permission system. It answers one question: “can this app do this specific thing on my behalf, without me handing it my password?”

That’s it. That’s the whole idea.

Before OAuth, if you wanted some third party app to, say, post to your Twitter or read your Gmail contacts, you had exactly one option: give that app your actual Twitter or Gmail password. The app would log in as you, directly, using your real credentials. Which means that random app now has full access to your account forever, until you go change your password, and even then you’re trusting a third party company stored your password responsibly in the first place.

OAuth replaces that with a system where you never hand your password to the third party app at all. Instead, you briefly go to Google (or whoever), log in there, and Google hands the app a token, a kind of limited, revocable permission slip, instead of your actual credentials.

🚨 Jaggy jaggy alert: token. Think of it as a hotel key card instead of your house key. It opens specific doors, for a specific amount of time, and the hotel can deactivate it instantly without you having to change your actual house locks.

Could the internet work without OAuth
Technically, yes. Plenty of early 2000s apps worked exactly how I described above, you just gave them your password. It’s not that the internet collapses without OAuth.

What actually happens without it is worse in a slower, quieter way. Every app you connect to anything ends up holding a copy of your real password. A breach at any one of those apps means your actual account password leaks, not just some limited token. You can’t selectively revoke access (“let this app see my calendar but not send emails as me”) because the app either has your full password or it has nothing. And you can’t tell your bank, your email provider, or your social accounts “trust these five apps, not that one,” because from their point of view every connected app looks identical: someone logged in with your password.

So yes, the world could work without OAuth. It would just be a world where a breach anywhere is a breach everywhere, and “revoke access” isn’t really a button that exists.

A quick word on OAuth 1.0, the version before this one
Before the OAuth everyone actually uses today, there was OAuth 1.0, back around 2007. Same basic idea, token instead of password, but the mechanics were painful. Every single request had to be cryptographically signed by the app using a shared secret, a process involving HMAC-SHA1 and exact timestamps, before the server would trust it.

🚨 Jaggy jaggy alert: cryptographic. Just means “based on math that’s specifically designed to be really, really hard to fake or reverse without the right secret ingredient.” When something is described as cryptographically secure, breaking it isn’t impossible, it’s just so expensive and slow that nobody sane bothers trying.

🚨 Jaggy jaggy alert: signature (signing). Not a handwritten signature. A short code generated from the actual contents of a message plus a secret value, proving two things at once: this message wasn’t altered in transit, and it genuinely came from whoever holds that secret. Change one character of the message and the signature stops matching, which is exactly how tampering gets caught.

🚨 Jaggy jaggy alert: secret. A private value, basically a password, known only to the two parties who are supposed to trust each other. Whoever holds the secret can prove they are who they claim to be. Whoever doesn’t, can’t fake it convincingly. The whole system falls apart the moment that “secret” stops actually being secret.

🚨 Jaggy jaggy alert: HMAC-SHA1 signing. A way of mathematically stamping each request so the server can verify nothing was tampered with in transit, instead of just trusting an encrypted connection to do that job. You don’t need the math, just know it meant extra, fiddly cryptography homework on every single request.

In practice this meant developers spent real time fighting signature mismatches and clock sync issues instead of building their actual app. It worked, but it was miserable, and it still assumed every app had somewhere safe to keep a permanent secret. OAuth 2.0 showed up in 2012 and basically said, let’s just lean on HTTPS to protect the data in transit like everyone already does for everything else, and drop the per-request signing entirely.

🚨 Jaggy jaggy alert: HTTPS. The padlock in your browser’s address bar. It encrypts everything traveling between your browser and the server, so anyone snooping on the connection, a nosy coffee shop WiFi for instance, sees scrambled nonsense instead of your actual data. OAuth 2.0 leans on this instead of signing every request itself, which is most of why it’s so much simpler than OAuth 1.0.

Simpler for developers, which is exactly why OAuth 2.0 is the one that actually won and is what the rest of this article has been describing the whole time.

But that “every app has somewhere safe to keep a secret” assumption didn’t go away, it just got quietly inherited by OAuth 2.0. And a few years later, that assumption stopped being true for a huge chunk of apps. More on that in a minute.

How it actually works, the short version
Here’s the flow in plain English, using “Sign in with Google” as the example:

You click the button. The app sends you over to Google, not the other way around, you never type your Google password into the third party app’s own form. At Google, you log in (or you’re already logged in) and Google shows you a screen saying “This app wants to see your email and profile picture, allow it?” You say yes. Google then redirects you back to the app with a short-lived code. The app takes that code and, behind the scenes, trades it with Google for an actual token. From then on, the app uses that token to ask Google for the specific things you approved, nothing more.

🚨 Jaggy jaggy alert: scope. A scope is the specific permission being requested, like “read your email address” or “post on your behalf.” Apps are supposed to ask for the narrowest scope that does the job, though in practice a lot of apps ask for way more than they need, because broader scopes are easier to code against and nobody’s really checking.

Download the Medium app
🚨 Jaggy jaggy alert: redirect URI. This is the exact web address Google is allowed to send you back to after login. It’s locked in advance when the app registers with Google, specifically so an attacker can’t trick Google into sending your freshly issued code to some address they control instead.

That whole flow above quietly assumes one more thing: that the app trading the code for a token is a proper backend server, one that can hold onto a permanent, secret password of its own (a “client secret”) to prove to Google “yes, this request really is from the real app.” Which brings us to a problem nobody really had to think about back in 2012.

The problem nobody saw coming, apps that can’t keep a secret
Fast forward a few years and suddenly half the apps in the world are single-page apps running entirely in the browser, or mobile apps sitting on someone’s phone. A SPA’s entire codebase gets shipped straight to the user, anyone can pop open dev tools and read it line by line. A mobile app’s installer can be decompiled by anyone with a weekend and mild curiosity. There is nowhere to actually hide a secret in either case. Whatever you bake in, someone can pull back out, which is a uniquely bad punchline for something literally called a “client secret.”

🚨 Jaggy jaggy alert: client secret. A password-like value, known only to the real backend and Google, used to prove a token request is genuinely coming from the real app and not an imposter. Fine when there’s a server to keep it locked up. Useless the moment the “app” is just JavaScript running on a stranger’s laptop.

The fix is called PKCE, Proof Key for Code Exchange, and it’s pronounced “pixie,” which might be the single most delightful name in all of security engineering.

🚨 Jaggy jaggy alert: PKCE. Instead of a long-lived secret baked into the app, the app invents a random, one-time value right before login starts, the code verifier, and sends over a hashed version of it, the code challenge, when kicking off the flow. At the very end, when trading the code for a token, it has to produce that original, un-hashed value again. If someone intercepts the authorization code mid-flow, it’s useless to them, they don’t have the random value that generated it, and they can’t fake having it either.

🚨 Jaggy jaggy alert: hashing. A one-way math function that turns any input into a fixed-length scrambled output, and crucially, you can’t run it backwards to get the original input out again. Same input always produces the same output, which is exactly what makes it useful here, the app can prove “I know the original value” without ever saying the original value out loud until the very last step.

No permanent secret to leak, because there was never a permanent secret to begin with, just a fresh, disposable value that self-destructs after one use. Genuinely a cleaner idea than the thing it replaced, and these days it’s recommended for basically every app, server backed or not.

The trade-off nobody puts on the slide
Here’s what the “just add OAuth” tutorials gloss over: you’re trading simplicity for control, and that trade costs real engineering effort.

With a plain username and password system, you own the entire flow. One table, one hashing function, done. With OAuth, you now depend on a third party’s uptime, their API changing underneath you, tokens that expire and need refreshing, and an entire second system, your own backend, that has to track who’s logged in even though the actual login happened somewhere else entirely.

🚨 Jaggy jaggy alert: refresh token. Access tokens are deliberately short-lived, often an hour or less, so if one leaks, the damage window is small. A refresh token is a longer-lived, more sensitive token your backend holds onto quietly, so it can go get a fresh access token without making the user log in again every hour. Lose control of a refresh token and you’ve basically lost control of the session, so these need to be stored far more carefully than access tokens.

That’s the actual trade-off. You get: no passwords for you to store or leak, users can revoke access with one click from Google’s side, scoped and limited permissions, and single sign-on convenience. You pay for it with: more moving parts, a dependency on an external provider, token lifecycle management, and a genuinely larger surface area for things to quietly go wrong.

How I actually use this in production
On one platform I worked on, we had Google OAuth as a social login option sitting alongside our own email and password system, with our own JWT issued after either path succeeded.

🚨 Jaggy jaggy alert: JWT. Short for JSON Web Token. This is different from an OAuth access token, even though people use the words interchangeably all the time. A JWT is a signed, self-contained token your own backend creates, that your own backend can verify without a database lookup, basically a sealed, tamper-proof note saying “this is user 482, and I, the server, vouch for that.” OAuth tokens come from Google. JWTs, in this setup, come from us.

That distinction, two different kinds of token doing two different jobs, is exactly where things went wrong for me once.

What actually goes wrong: my real production bug
We had a production authentication bug where users were randomly getting logged out, or logged in as the wrong state, with no clear pattern. Intermittent, hard to reproduce locally, the worst kind of bug.

The root cause, once I actually traced it through production logs, was almost embarrassingly simple: the OAuth token coming back from the social login flow and our own JWT were both being stored under the same cookie key. Whichever one got written last silently overwrote the other. So depending on timing, a user’s session cookie might end up holding Google’s OAuth token instead of our own JWT, or vice versa, and the backend would try to verify it as the wrong type of token entirely, fail, and drop the session.

Nobody designed it that way on purpose. It happened because OAuth tokens and JWTs look similar enough, both are just strings you shove in a cookie, that it’s easy to treat them as interchangeable “auth stuff” instead of two distinct tokens with two distinct lifecycles and two distinct purposes. The fix was straightforward once I actually understood the distinction: give each token its own, clearly named cookie key, so they could never collide again.

The real lesson wasn’t the one line fix. It was that OAuth tokens and your own session tokens are not the same thing wearing a different hat, and treating them as the same thing is exactly the kind of assumption that only breaks in production, under real timing and real concurrent requests, never in your local dev environment where everything happens conveniently one at a time.

Other pitfalls worth knowing before you hit them
A few more ways this goes sideways, things I’ve seen or would actively watch for:

Storing tokens in localStorage instead of an httpOnly cookie, which means any cross-site scripting vulnerability on your site can just read the token straight out and walk away with the user's session.

🚨 Jaggy jaggy alert: httpOnly cookie. A cookie flagged so that JavaScript running in the browser can’t read it at all, only the browser itself can send it along with requests. It’s a deliberate wall against exactly the kind of theft I just described.

Mismatched redirect URIs between what’s registered and what the app actually sends, which just breaks login outright and is a surprisingly common first deploy bug when you move from a local dev URL to a real production domain and forget to register the new one.

Asking for way more scopes than you need, which makes users nervous at the consent screen and gives you a bigger blast radius if your token ever does leak.

Treating “the user has an OAuth token” as “the user is logged in” without actually verifying who they are. OAuth by itself is about authorization, what you’re allowed to do, not identity. Actual login on top of OAuth is a separate layer called OpenID Connect, and conflating the two is a genuinely common source of security bugs.

So, do you actually need it
If you’re building something small and internal, OAuth is probably overkill, a simple username and password system with properly hashed passwords is less to maintain and less to get wrong. OAuth earns its complexity the moment you want users to log in with an identity they already have elsewhere, or you want to act on a third party service on their behalf without ever touching their password. At that point, yes, deal with the token juggling, because the alternative, a world where “give the app your password” is normal, is worse in ways that only show up after something’s already gone wrong.

That’s the real shape of it. Not a login button. A permission system, with real trade-offs, that will absolutely find the one gap in your assumptions if you let two different tokens share a cookie key and call it a day.

What habits daily make you a really great software engineer?

Fri, 16 Oct 2026 17:09:05 +0000

Well, what do you think what are some top notch habits that makes you in top zone of engineering

Is Linux an Operating System or Just a Kernel?

Wed, 14 Oct 2026 08:34:17 +0000

When I first started learning Linux, I constantly heard two different statements:

"Linux is an operating system."
"Linux is just a kernel."

At first, I thought one of them had to be wrong. After learning how Linux actually works, I realized both statements make sense depending on the context.

Let's understand why.

What Is an Operating System?

An operating system (OS) is the software that acts as a bridge between users, applications, and the computer's hardware.

Without an operating system, applications wouldn't know how to communicate with the CPU, memory, keyboard, display, storage devices, or network hardware.

An operating system provides services such as:

Process management
Memory management
File management
Device management
Security and permissions
Networking

These services allow applications to run without worrying about the underlying hardware.

What Is a Kernel?

The kernel is the core component of an operating system.

Think of it as the manager that is always running in the background. Every program that needs hardware access must go through the kernel.

The kernel is responsible for:

Scheduling CPU time
Allocating memory
Managing devices using drivers
Handling filesystems
Managing processes
Enforcing permissions
Providing system calls

Applications cannot directly control hardware. Instead, they request services from the kernel.

Application
      │
System Calls
      │
Linux Kernel
      │
Hardware

Then What Is Linux?

Technically, Linux refers to the Linux kernel created by Linus Torvalds in 1991.

The kernel alone cannot provide a complete user experience.

For example, the kernel cannot:

Let you type commands
Display directories with ls
Copy files using cp
Edit files
Browse the web

It simply provides the core functionality needed for these programs to work.

So Where Do Commands Come From?

Commands like:

ls
cp
mv
mkdir
grep
cat

are not part of the Linux kernel.

Most of these commands come from the GNU Project, which provides many of the utilities we use every day.

The shell (such as Bash) is also a separate program that lets us interact with the kernel.

When you type:

ls

the shell starts the ls program, which eventually makes system calls to the Linux kernel. The kernel retrieves the directory information from the filesystem and returns it to the program, which displays it on your screen.

What Is a Linux Distribution?

A Linux distribution combines everything needed to create a usable operating system.

A typical distribution includes:

Linux kernel
GNU utilities
Shell (Bash, Zsh, etc.)
Package manager
System libraries
Desktop environment (optional)
Applications

Examples include:

Ubuntu
Debian
Fedora
Arch Linux
Linux Mint

Each distribution uses the Linux kernel but packages it with different software and tools.

Why Do Some People Say GNU/Linux?

Richard Stallman and the GNU Project argue that most of the user-space tools come from GNU, while Linux is only the kernel.

Because of this, some people prefer calling the complete operating system GNU/Linux instead of simply Linux.

Others use the shorter name "Linux" because it is more common.

Both terms are widely used today, depending on the context.

How Everything Fits Together

+-----------------------------+
| Applications                |
+-----------------------------+
| Shell (Bash)                |
+-----------------------------+
| GNU Utilities               |
+-----------------------------+
| System Libraries            |
+-----------------------------+
| Linux Kernel                |
+-----------------------------+
| Hardware                    |
+-----------------------------+

What I Learned

Before learning Linux, I assumed Linux itself contained every command I used.

Now I understand that the Linux kernel is only the core of the operating system. The commands, shell, libraries, and applications are separate software that work together with the kernel to create the complete Linux experience.

This also explains why different Linux distributions look and behave differently while sharing the same Linux kernel.

Conclusion

Linux is technically a kernel, not a complete operating system by itself. However, in everyday conversation, people often use the word Linux to refer to the complete operating system provided by Linux distributions.

Understanding this distinction helped me better understand how Linux works internally and why different distributions can share the same kernel while offering different user experiences.

Understanding Before Implementation: My Philosophy of Engineering in the AI Era

Tue, 13 Oct 2026 08:02:50 +0000

Today, an AI assistant can generate a working API endpoint, write a complex SQL query, or configure a multi-stage CI/CD pipeline in seconds. When code generation is nearly frictionless, it is easy to mistake generating an implementation for understanding a system.

AI coding assistants excel at boilerplate, syntax, API orchestration, and fast prototyping. But syntax was never the hardest part of software engineering. The harder part has always been framing the real problem, understanding how components interact, identifying constraints, evaluating trade-offs, and knowing how to verify that a system actually does what it is supposed to do.

Now that AI can execute code generation on command, conceptual understanding is not less important—it is the very foundation that allows us to command AI effectively.

The Difference Between Framework Proficiency and System Understanding

When picking up a new technology, there are two distinct ways to approach learning:

Learning to reproduce an implementation: Memorizing specific APIs, following step-by-step tutorials, or asking AI to generate code for a framework without understanding its underlying mechanics.
Learning to build a mental model: Understanding what problem the technology was created to solve, why its abstractions exist, where it fits into a larger architecture, and what trade-offs it introduces.

Frameworks and SDKs teach us how to interact with specific abstractions. Conceptual understanding helps us decide which abstractions we need, whether they are appropriate, and when a different approach is necessary.

This does not mean hands-on experimentation is secondary. Theory without practice can lead to rigid, over-engineered designs that fail in production. Rather, conceptual reasoning and hands-on implementation should reinforce each other in a continuous loop:

+-------------------------------------------------------------+
|                                                             |
|   1. Build a Mental Model ---> 2. Define Architecture      |
|              ^                             |                |
|              |                             v                |
|   4. Refine Architecture <--- 3. Implement & Test           |
|                               (via AI, SDKs, Experiments)   |
|                                                             |
+-------------------------------------------------------------+

When you understand the underlying mechanics, you learn a framework with clear intent rather than copying code blindly.

From DevOps to Agentic FinOps: A Personal Journey

I experienced the power of this philosophy firsthand when I transitioned into the world of AI agents.

My professional background was firmly rooted in DevOps—managing cloud infrastructure, automated pipelines, and operational reliability. I had minimal formal experience with AI or agentic frameworks when an opportunity arose to design an agentic solution for a complex DevOps problem: cloud financial operations (FinOps).

Faced with an unfamiliar domain, I had two choices:

Approach A: Spend weeks memorizing popular agentic framework APIs, copying example agent scripts, and trying to force-fit my problem into pre-built templates.
Approach B: Step back and build a mental model of agentic systems first. What makes an agentic system different from a script or a deterministic workflow? What components are required—reasoning, memory, tool access, state management? How should data and decisions flow?

I chose Approach B.

Before diving deep into specific agent frameworks or SDKs, I focused on understanding the system boundaries and architectural capabilities required for an Agentic FinOps solution:

What exact operational problem were we solving in cloud cost governance?
Which tasks required autonomous decision-making versus deterministic automation?
How would an agent reason about cloud resource data, evaluate policies, and execute actions?
What state and context would need to persist across executions?
Where were the potential security, cost, and reliability failure modes?

+-----------------------------------------------------------------+
|                   Agentic FinOps Conceptual Model              |
+-----------------------------------------------------------------+
|                                                                 |
|  [ Data Sources ]       [ Reasoning Engine ]    [ Action Layer ]|
|  - Cloud Metrics  --->   - Policy Rules    ---> - Remediation   |
|  - Usage Data           - Context/Memory        - Alerts        |
|                                                                 |
+-----------------------------------------------------------------+

By first mapping out the system's core requirements, I could evaluate frameworks objectively. Instead of letting a framework dictate what my system should look like, I used frameworks and SDKs—along with AI assistants—to implement specific, well-defined components of the architecture I had designed.

I began building and testing core components individually to validate feasibility. Did I have every answer upfront? No. Was every part of the system immediately production-ready? No. The architecture continues to evolve as I learn more. But starting with conceptual clarity allowed me to make meaningful progress in an unfamiliar domain without getting lost in implementation details.

Engineering in the AI Era: Taking the Lead

AI tools have transformed how quickly we can move from an idea to a working prototype. But generated code is not inherently secure, performant, cost-effective, or architecturally sound.

To "take the lead to command AI" does not mean typing every line of code manually. It means taking responsibility for:

Problem Definition: Articulating clear, context-rich goals and constraints.
System Architecture: Deciding how components decouple, communicate, and fail.
Critique and Evaluation: Interrogating AI-generated code for security flaws, edge cases, and hidden performance costs.
Validation: Designing small experiments to verify that a proposed solution works under real conditions.

If you don't understand what you are asking AI to build, you cannot effectively evaluate what it returns.

A Transferable Framework for Learning New Technologies

Whether you are exploring agentic AI, Kubernetes, distributed databases, or service meshes, this iterative approach can guide your learning:

Define the Problem: Understand what pain point the technology addresses before choosing tools.
Build a Mental Model: Map out the core concepts, data flows, and state management mechanisms.
Identify Component Responsibilities: Determine what each part of the system does and why it exists.
Evaluate Trade-offs: Analyze cost, latency, security, and operational complexity across different options.
Accelerate with AI and Frameworks: Use AI tools, documentation, and SDKs to rapidly write and prototype code.
Test Core Assumptions: Run small, targeted experiments to validate critical capabilities.
Inspect and Verify: Audit generated code—do not assume "working code" is correct code.
Refine Iteratively: Adjust your architecture as practical findings reveal gaps in your mental model.

Why I Write Technical Content This Way

This philosophy directly shapes how I write technical articles. My goal is not to publish syntax cheat sheets or step-by-step code tutorials that become outdated with the next SDK release.

Instead, I aim to help readers understand why systems are built the way they are—their underlying components, decision points, and trade-offs. Once you grasp the foundational architecture, applying any specific SDK, API, or AI tool becomes straightforward.

Final Thoughts

The ability to generate code quickly is more accessible than ever. But true engineering value lies in judgment, critical thinking, and system architecture.

Before diving straight into implementation or asking AI to generate a solution, take time to understand the problem, the system, and the trade-offs. Use AI and modern frameworks to build faster, but keep your hands on the architectural wheel.

The Death of SaaS - What Happens When People Can Make Software Fit Their Lives?

Tue, 13 Oct 2026 06:35:19 +0000

In February 2026, investors wiped approximately $830 billion from the market value of software and services companies over six trading sessions, frightened by the possibility that AI could erode businesses built around selling access to software. By October, US software stocks had recovered much of that ground, and Wall Street appeared considerably less alarmed. [1][2]

Outside the financial markets, however, customers had begun doing something that stock prices could scarcely capture. They were making their own software.

Some wanted to replace expensive subscriptions. Others were tired of squeezing their work into applications designed around somebody else’s habits.

With AI coding tools and existing open-source components, they could begin assembling applications around what they needed, however peculiar the requirements. An application serving five colleagues might be quite sufficient.

It didn’t need a market.

That is the change beneath the argument about The Death of SaaS: people are gaining the ability to decide how their software should work, rather than accepting those decisions as part of the price of admission.

In October, an ambitious attempt to recreate Adobe’s professional applications brought that change in expectations into public view.

1. Adobe and the Cost of Learning Someone Else’s Software

The developer behind ArtCraft began publishing seven open-source applications in Rust, with substantial AI assistance. Their targets were familiar: Photoshop, Illustrator, Premiere Pro, Lightroom, Acrobat, After Effects, and InDesign. The repositories were public, and other developers could inspect the code and contribute.

Under the Reddit announcement, somebody asked for another program.

“Now do AutoCAD.”

The developer responded with a link to CADCraft, an additional project already underway. [3]

It’s easy to understand the excitement. Adobe’s software has become part of the working environment of designers and photographers, much as Autodesk’s products have shaped established CAD workflows.

People spend years learning these systems. Training courses teach their interfaces, clients request particular file formats, and businesses accumulate work that becomes difficult to move elsewhere.

The subscription is often the most visible expense. The time already invested in learning the software is harder to calculate.

ArtCraft’s early applications, however, were a long way from matching the products they imitated. In an October 10 review, Steve Paris found that PhotoCraft couldn’t open his HEIC photographs and saved edited work only as JPG. FilmCraft struggled with basic trimming and produced unusable audio. [4]

These are understandable limitations for an early experimental project attempting software that large engineering teams have spent decades refining.

A hobbyist can afford to experiment with an unfinished editor; a professional has a client waiting for tomorrow’s revisions. Try explaining that the editable layers are gone, but hey, the toolbar looks just like Photoshop.

The distinction matters in practical work, where an application has to preserve what people create. A designer leaving Adobe may still need to exchange files with clients who haven’t left.

In a July 2025 Reddit discussion, a designer described moving their small company away from Adobe after nearly twenty years of professional use. The responses showed how much previous work and established production habits could complicate a transition, even when people were unhappy with the subscription. [5]

ArtCraft won’t make those dependencies disappear overnight. The developer’s description of the project as a clean-room recreation also remains a claim about the development approach rather than an independently established legal finding.

But there is something peculiar about the assumption that an alternative must reproduce the original product in its entirety.

A photographer may need a reliable RAW-processing workflow without the surrounding application ecosystem. A small business might require a handful of image operations performed consistently. Someone producing illustrations may care deeply about brush behavior while having no use for dozens of other features.

An application developed for a vast customer base necessarily contains decisions made for people whose work bears little resemblance to yours.

A faithful clone would inherit many of those choices, including the ones that were inconvenient in the first place.

We have spent so many years adjusting ourselves to the shape of software that changing the tool itself sounds almost unreasonable, although people who work with physical tools would recognize the impulse immediately.

2. The Hammer and the Way We Work

A carpenter can replace a worn hammer handle or choose a different weight for a particular job. Years of practice still matter, and the tool’s design has a considerable influence on the work. But modifying equipment to suit a particular task has always been part of the craft.

Computer applications trained us into another relationship.

We learned their terminology and remembered where their designers had placed the controls. A business might change its procedures because the software handled one sequence more conveniently than another.

Employees became proficient at navigating features they seldom needed while finding elaborate workarounds for the few that were missing.

Once everybody knew the procedure, the inconvenience became routine. Nobody wanted to disturb a system that had taken so long to learn.

In July 2026, a construction-business operator described this frustration in a Reddit discussion. They had tried Buildertrend, a commercial construction-management platform, and found it bloated and cumbersome for the way their company operated. Rather than return to a costly subscription, they had spent about a month intermittently developing their own management software.

The application was still taking shape. What they wanted from it was unmistakable:

“It works how our business runs instead of us trying to adapt to the way they think we should do things.” [6]

The parent Reddit post was removed, although the relevant comment remains visible.

That is the experience I find more revealing than an ambitious claim about replacing an entire software company. The construction operator already knew how their business ran. Their problem was finding software that respected those practices.

Other contributors in the discussion described similarly narrow approaches to construction administration, including scheduling through SMS so subcontractors could confirm appointments without adopting another application.

Consider what disappears from the usual software-selection process. The business no longer needs to evaluate hundreds of capabilities it never requested. It can start from the tasks employees actually perform and add functions as those tasks require them.

The resulting program could be unsuitable for almost every other construction company. Its owner has little reason to care.

The ambition here is surprisingly modest: to have the right tool for the work already being done.

And a great deal of the technical foundation for such tools has been available for years.

3. We Already Had the Building Materials

Open-source software had been expanding access to advanced technical capabilities long before the present AI boom. People often discuss these projects as inexpensive substitutes for commercial applications, overlooking the substantial engineering they contain and the professional work already produced with them.

The alternatives cover familiar software categories:

Microsoft 365 → LibreOffice / ONLYOFFICE. Document editing, spreadsheets, and presentations.
Autodesk Maya → Blender. 3D modeling, animation, rendering, and visual production.
AutoCAD / Fusion → FreeCAD. Parametric CAD and mechanical design.
Commercial antivirus suites → Windows Security / Bitdefender Free Free baseline malware protection.
TechSmith Snagit → ShareX/ OBS Studio. Screen capture, desktop recording, and workflow automation.
Wondershare UniConverter → HandBrake/ FFmpeg. Video conversion, compression, and media processing.
Adobe Premiere Pro / After Effects → DaVinci Resolve / Kdenlive Video editing and production tools.
Adobe Lightroom → darktable/ RawTherapee. RAW photography and image development.
Adobe Photoshop → GIMP/ Krita/ PhotoCraft Image editing and digital painting; PhotoCraft remains experimental.
Adobe Illustrator → Inkscape/ VectorCraft. Vector illustration and scalable graphics; VectorCraft remains experimental.

Some products in this list are free proprietary software rather than open source, and the applications vary in scope. They are alternatives for particular kinds of work, with different features and degrees of compatibility.

But consider Blender

The animated feature Flow, directed by Gints Zilbalodis, was produced using Blender and EEVEE.

The film won the Academy Award for Best Animated Feature in 2025. Zilbalodis, who had previously worked with Maya, described adopting Blender because its rendering workflow gave him the control and speed he wanted for his production. [7]

This is hardly an example of professionals settling for inadequate software because they couldn’t afford the real thing. The director chose a tool that suited the way he wanted to make a film.

FFmpeg provides a different illustration. Its media-processing capabilities can sit underneath a small application that handles a production team’s particular conversion requirements. The person constructing that application doesn’t need to develop a video codec. Much of the difficult engineering already exists.

AI coding assistants can help people create the surrounding application: a small interface, a sequence of processing steps, perhaps an integration with another tool used in the same workflow.

Open-source projects such as Screenshot-to-Code push the starting point even closer to the user by generating interface code from visual examples. A screenshot is only part of an application, but it can give someone an initial structure to modify. [8]

There is a long history behind this kind of customization. People built intricate applications in Excel, and platforms such as Airtable enabled teams to organize databases around their own activities well before AI coding assistants arrived.

What is expanding now is the range of changes people can attempt through natural-language interaction with code. They can start with components created by skilled developers and shape their own application around those foundations.

The benefit becomes especially clear when the intended audience is very small.

4. Software for One

In June 2026, a Reddit user asked which paid applications people had replaced by building their own with AI.

The replies brought that possibility down to the scale of everyday work. People had grown tired of accommodating software that never quite fitted their needs, and some had begun changing the tools themselves.

One reply described taking a screenshot of a Monday.com board and asking Bolt to build something similar. Within hours, the user reported having a working application they named Tuesday.

They subsequently tailored it to their team’s habits, including synchronization with a shared Google Calendar, and claimed savings of approximately $800 per month in Monday.com licenses. Those figures come from the user’s account rather than an independent audit. [9]

The name Tuesday has a kind of accidental charm. The user wasn’t trying to launch a rival to Monday.com. Their team needed a particular way to manage work, and the person building the application could decide which functions deserved attention.

Elsewhere in the same discussion, a user described making a budgeting application that relied on manually entered transactions. Another participant explained why manual entry could be desirable: recording expenses deliberately was part of how they kept track of their spending. An automatic bank import, usually treated as an obvious improvement, would remove part of the experience they valued. [9]

This is the sort of disagreement commercial software designers must routinely reconcile. A feature can make the application better for one customer while making it less suitable for another.

Software developed for a single person can accept peculiar requirements without taking a vote.

A further Reddit contributor described creating their own document-management system after a product they had used for fifteen years was discontinued following an acquisition. The purchaser was attempting to move customers onto a more expensive subscription service.

The user reported building an alternative that could work with a collection approaching 100,000 documents, despite having little recent programming experience. This remains a self-reported account, but the motive is hard to miss: they wanted to retain control over work that another company’s acquisition had disrupted. [9]

Similar cases have appeared outside developer communities. In January, TechCrunch reported on people making micro-apps for themselves and small groups.

Rebecca Yu created Where2Eat to help her friends choose restaurants. Other applications were built for household tasks and temporary needs that would never have justified a conventional commercial product. [10]

A conventional software company must consider market size. The owner of a personal application can ask whether the software serves the people who will actually use it.

That difference changes what counts as a successful product.

If a useful tool lasts six months and is then replaced because its owner’s requirements have changed, it may have fulfilled its purpose. If it contains only four features, those four can receive attention that a large general-purpose application would have distributed across far more capabilities.

More importantly, the user can participate in deciding how those features work.

Research into generative and malleable interfaces has begun exploring this relationship more formally. Yining Cao, Peiling Jiang, and Haijun Xia examined interfaces organized around task-driven data models, allowing users to modify generated applications through natural language and direct interaction. Their CHI 2025 work explores how applications can evolve alongside the requirements of the people using them. [11]

An application can therefore become something less fixed than the product delivered at the moment of purchase. Its structure can reflect the work as the work changes.

This experience can seem unfamiliar because professional software has been distributed through standardized applications for so long.

Historically, however, the movement of technical knowledge toward a wider population is hardly unprecedented.

5. Gutenberg, the Encyclopédie, and the Open Web

Around the middle of the fifteenth century, Johannes Gutenberg helped transform book production in Europe.

Printing with movable metal type was already known in Korea, as demonstrated by the surviving Jikji of 1377, but Gutenberg’s European printing system made the repeated production of books more efficient within its own historical environment.

The Gutenberg Bible, printed around 1454–1455, remains one of the defining artifacts of that change. [12][13]

Books still required literacy, and access was far from universal. Yet the economics of making and circulating written material had changed. Knowledge that depended heavily on laborious manuscript copying could now travel through a growing printing industry.

Three centuries later, Denis Diderot and Jean le Rond d’Alembert undertook another revealing project.

Published between 1751 and 1772, the Encyclopédie attempted to gather knowledge across the sciences and the mechanical arts. Its enormous collection of illustrations documented machinery and working processes in remarkable detail.

The University of Chicago’s ARTFL project records seventeen volumes of text and eleven volumes of plates in the original publication. [14]

The decision to document practical trades was particularly significant.

At the time, much technical knowledge belonged to the working traditions of individual craftspeople and workshops. Some manufacturing practices were guarded closely. Diderot wanted to put these processes into words and pictures so that people outside the original workshops could study them.

John R. Pannabecker’s historical research describes how the Encyclopédie helped shift the mechanical arts from knowledge embedded largely in working communities toward knowledge represented systematically in published materials. [15]

An engraving couldn’t make its reader a master craftsperson. There are forms of judgment that cannot be transferred completely through a diagram. But someone who had never entered a particular workshop could begin to understand machinery and production methods previously difficult to examine.

That is a striking antecedent to the open-source idea: useful technical knowledge becomes available for study beyond the people and institutions that originally developed it.

Then came the World Wide Web.

Tim Berners-Lee developed the Web at CERN beginning in 1989 to help researchers exchange information. On April 30, 1993, CERN released core Web software into the public domain. That decision made the basic technology available for others to use and improve without paying CERN for permission. [16]

The Web grew on the foundation of an internet that already existed, and its expansion depended on browsers and technical contributions from many organizations. CERN’s own history emphasizes how openly available software and accessible browsers helped the Web spread beyond its original scientific setting.

One detail is particularly relevant today: Berners-Lee’s original WorldWideWeb browser was also an editor. The early vision included the ability to create and change pages, even though most people eventually came to know the Web primarily as something they browsed. [16]

For ordinary users, the arrival of accessible publishing tools changed who could put information online. A person no longer needed to own a newspaper or a broadcasting network to publish something others could read.

There is a thread running through these episodes, though each belongs to a different technological world.

Printing altered the reproduction of written material. The Encyclopédie tried to bring practical technical knowledge beyond the workshops where it had accumulated. The open Web expanded opportunities to publish and exchange information across institutional boundaries.

The consequences extended well beyond the industries that created the original technologies.

AI-assisted software development is opening another part of that history. A person who understands a particular job can increasingly participate in making the software used to perform it.

The technical foundations are often already available through open-source projects. AI helps bridge some of the distance between understanding a problem and constructing a working tool.

The possibilities remain uneven. Some applications are easy to modify, while others embody extraordinary engineering complexity. A sophisticated CAD system cannot be conjured into professional reliability merely by reproducing its interface.

Yet people are gaining practical ways to influence software’s design without first becoming specialists in every underlying technology.

That changes the relationship between a person and the tools they depend on.

Final Thoughts — The Choice We Didn’t Have

The traditional software business was built around producing applications that large numbers of people could use. Standardization brought real benefits, but it also required customers to accept decisions made for a much wider population.

If a function was missing, users searched for another product or waited. If a workflow didn’t fit, they learned a workaround. Years of practice made those compromises familiar, while subscriptions kept access to the software tied to decisions made elsewhere.

Now people are beginning to make different choices. A construction operator is developing software around the way his company already works. A small team has built Tuesday because Monday.com carried more than it needed. Someone with years of documents has attempted to replace a system whose owner wanted to move them onto another subscription.

Meanwhile, open-source projects continue to make advanced technology available for reuse, and AI coding tools give more people a way to assemble that technology around their own requirements.

Services that genuinely help people will continue to find customers. A user can choose to pay for infrastructure or collaboration without surrendering every decision about the application that sits above them.

The part of SaaS losing its hold is the assumption that useful software must arrive as a standardized package whose structure the customer has little power to change.

That brings me back to the ArtCraft announcement.

“Now do AutoCAD.”

The request was for another familiar professional application. After reading the accounts of people making software for their own work, though, I wonder what that person actually needed AutoCAD to do.

Perhaps the answer wouldn’t require another enormous application. Perhaps it would require something smaller, built around the particular design problem sitting on their desk.

For a long time, that distinction hardly mattered. Most users had to choose among the applications somebody else was willing to build.

They are beginning to have a say in the tools themselves.

References

[1] Reuters, “Selloff wipes out nearly $1 trillion from software and services stocks as investors debate AI’s existential threat,” Reuters, February 4, 2026.

[2] Reuters, “US software stocks scale fresh 2026 highs as AI disruption worries fade,” Reuters, October 6, 2026.

[3] ArtCraft developer, “I’ve created Open Source 100% Rust clean-room replacements for 7 of Adobe’s apps,” Reddit, r/SaaS, October 2026.

[4] Steve Paris, “One dev is recreating 7 Adobe apps in Rust for free — so I tested it out,” TechRadar, October 10, 2026.

[5] r/graphic_design contributor, “Why I’m (we’re) leaving Adobe,” Reddit, r/graphic_design, July 2025.

[6] r/ClaudeAI commenter, “Construction-management discussion, including Buildertrend replacement,” Reddit, r/ClaudeAI, July 12, 2026. Parent post removed; relevant comment remains accessible.

[7] Francesco Siddi, “Making Flow — Interview with director Gints Zilbalodis,” Blender, January 22, 2025; updated to reflect subsequent awards.

[8] Abi Raja and contributors, “screenshot-to-code,” GitHub repository, accessed October 2026.

[9] r/ClaudeAI contributors, “What paid apps have you ditched by vibe coding a replacement?,” Reddit, r/ClaudeAI, June 2026 and subsequent comments.

[10] Dominic-Madori Davis, “The rise of ‘micro’ apps: non-developers are writing apps instead of buying them,” TechCrunch, January 16, 2026.

[11] Yining Cao, Peiling Jiang, and Haijun Xia, “Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model,” Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Article 686, 2025.

[12] UNESCO, “Baegun hwasang chorok buljo jikji simche yojeol (vol. II),” Memory of the World Register, 2001.

[13] Library of Congress, “The Gutenberg Bible at the Library of Congress: Introduction,” Library of Congress Research Guide, undated.

[14] University of Chicago, “The ARTFL Encyclopédie,” ARTFL Encyclopédie Project, institutional digital edition.

[15] John R. Pannabecker, “Diderot, the Mechanical Arts, and the Encyclopédie: In Search of the Heritage of Technology Education,” Journal of Technology Education, 6(1), Fall 1994.

[16] CERN, “A Short History of the Web,” CERN institutional history, undated.

GraphQL Solved Facebook's Problem. Does It Solve Yours?

Mon, 12 Oct 2026 21:00:21 +0000

GraphQL divides teams, and I don't claim the last word on it. This post is my research into my own experience with GraphQL and into decisions I'm still making. I'm not calling anyone out or dismissing what teams have built with it. I'm asking for caution before an organization commits to a tool its product and customers may not need.

Facebook's 2015 engineering post introducing GraphQL explained where the idea came from: "We don't think of data in terms of resource URLs, secondary keys, or join tables; we think about it in terms of a graph of objects." The line most people quote from that post is that "the shape of the returned data is determined entirely by the client's query." That's true, but it's a consequence of the unusually complex and deep graph, not the reason GraphQL exists. No server team can design an endpoint for every path through a graph that size, across every app version reading it, so the client has to write the query.

GraphQL solved Facebook's problem, and the burden of proof sits with everyone importing it to show that it solves theirs. I don't think most teams can meet it.

Most of what teams adopt GraphQL for is available without it. What's left is a client composing its own traversal of a graph the server can't predict. GraphQL's own best practices steer private APIs away from even that, by registering every query ahead of time. Client teams can still write their own queries before release, but planning across teams, or a backend for their screens, usually gives them the same control. That stops being true only when a product's screens take so many paths through its graph, and change them so often, that keeping an endpoint for each costs more than securing a query language. Facebook's did.

Most of GraphQL's Benefits Aren't GraphQL's

The case for adopting GraphQL usually arrives as a bundle: typed contracts, payloads shaped to each screen, flexible filtering, declarative authorization, one endpoint over many services, and frontends that don't wait on backends. For each, the test is whether the client must compose the operation. Most don't.

Typed Contracts and Codegen Come From Contract-First APIs

A GraphQL schema gives clients types and generated code, as an OpenAPI document does for an HTTP API and a Protocol Buffers definition does for gRPC. What differs is what the contract commits the service to.

In contract-first development, the team designs and publishes each operation on purpose, so the contract states what the service supports. A GraphQL schema publishes types and entry points instead. The operations that matter, meaning which fields combine at what depth, get written later by whoever calls it. Unless every query is registered in advance, the schema commits the service to every operation its types and limits allow.

Filtering and Sorting Are the Server's Choice, Not a Query Language

Hot Chocolate, a GraphQL server for .NET and the best-designed one I've used, adds filtering, sorting, and paging to a list with an attribute or two. But each attribute is the server choosing which lists callers may filter, on which fields, and how far they may page.

An endpoint makes the same choices about its query parameters. OData, an OASIS standard for REST APIs, standardizes those parameters as $filter, $orderby, $top, and $skip, and ASP.NET Core lets each endpoint allow only the ones it chooses.

Field-Level Authorization Is a Cost of Client-Written Operations

The policy engine behind GraphQL authorization isn't GraphQL's, and the GraphQL Foundation tells production codebases to "delegate authorization logic to the business logic layer." Hot Chocolate's [Authorize] attribute, for one, runs the same roles and policies ASP.NET Core applies to an endpoint. What GraphQL changes is where the check has to sit.

An endpoint is a designed operation, so its code knows what the caller is doing and can apply the rule that fits. In GraphQL, a read can reach a field through any path the schema's types allow. Only root mutations are designed like endpoints.

Say a support agent may see a customer's email while working that customer's ticket, but not in a sales report the agent can also open. In GraphQL, the same Customer.email field sits under both. A check on the field alone can confirm the agent has an open ticket with that customer, but it can't tell whether the request came through the ticket or the report, so the email shows in both.

A ticket-specific customer type would fix that, as a ticket DTO would in REST. Either fix works, but an endpoint exposes only the paths its designer wrote, while a schema must be right for every path its types allow.

One Endpoint Over Many Services Fights the Teams Behind It

Apollo Federation and Hot Chocolate's Fusion put one GraphQL endpoint in front of several services, often for teams with predictable queries. Each service publishes a subgraph, and a gateway composes them into one schema and splits each query into requests to the services that own the fields.

The stitching a client would have done still happens, in a query planner where the fetches are harder to see. Apollo's query-plan documentation says the router runs subgraph fetches in sequence "whenever one subgraph's response depends on data that first must be returned by another subgraph."

A product built around bounded contexts gives each team authority over its own model. A federated schema asks for the opposite. Every team contributes types to one shared graph, extends other teams' entities, and passes a composition check that another team's conflicting change can fail. The bounded contexts are still there, but the schema pretends they aren't.

A backend-for-frontend looks like the same facade but points the other way. Sam Newman describes it as a backend dedicated to one user experience and owned by the team building that interface. It calls each domain's published contract like any client, so each domain keeps its own schedule and authority. Its fan-out sits in code its team owns and can tune, and it shapes each response to its screens, so clients don't over-fetch.

Members of my team once proposed a single query endpoint over our services, and I rejected the prototype we built. It added too much latency for our data-heavy, highly responsive web app, and it fought our caching. Each domain, and often each kind of record, had its own TTL. One response mixing those lifetimes defeated a request-level cache, and caching per source meant streaming every domain's invalidations into a gateway no domain team owned. What worked was each layer caching the data it owned.

Latency isn't the lasting case against federation, since a backend-for-frontend's fan-out pays the same network hops. Ownership is. One product doesn't prove a rule, and a product without our volatile data, varied cache lifetimes, and team per bounded context might find a federated graph harmless.

Fragment Colocation Needs GraphQL but Solves a Team Problem

Relay, Meta's GraphQL client for React, is built around colocated fragments. Each component declares the data it needs, and the compiler nests those fragments into the queries a screen sends.

GraphQL's defenders build on this, arguing that even when every query is registered in advance, the unpredictability hasn't gone away. It has moved from runtime to development time, because the server team can't predict what the frontend will need next month.

Colocation buys frontend convenience, and it pays for it by giving up intentional contracts. The need it serves, a frontend team that can't wait on a backend team, is a planning failure before it's a technical one. Most of a screen's data needs show in its design before it's built, and leads who set priorities across teams can schedule its endpoint in the same release.

So this case for the technology rests on one premise, that the teams can't plan together, and that premise is a needle holding up a foundation.

Even good planning can't always keep pace, as when a screen changes mid-build or an experiment varies it. Then the frontend team needs to own its operations, and a backend-for-frontend can give it that as well as colocation can.

Say a listing card on web, iOS, and Android gains its seller's rating. With colocation, each client edits a fragment and its build registers a new document. With a backend-for-frontend, each client team adds the field to its own backend's response in the same release. Neither waits on a domain team unless the rating doesn't exist yet, and then both do.

Each fix has a price. A backend-for-frontend needs a backend per platform, each with its own pipeline and on-call. Colocation avoids that and saves an endpoint edit per backend. But saved time isn't a benefit until it's weighed against what it leaves behind. Colocation leaves an operation no server code was written for, even when a frontend team runs GraphQL as its own backend-for-frontend. That operation carries the authorization problem above. Only registered paths run, but no server team designed them. A designed endpoint settles that check in code its team owns.

Adopting GraphQL because it removes the wait between teams, and then collecting reasons it fits, invites confirmation bias. Choosing it on purpose means proving its security, performance, stability, and testability against the problem at hand. Slowing down to plan together is how a product speeds up.

What's Left Is a Graph Nobody Can Predict

GraphQL started in 2012, when Facebook began rebuilding its iOS and Android apps. Native apps needed "an API data version of News Feed," and where a REST service "may require multiple round-trips," "GraphQL naturally follows relationships between objects."

Every item in a feed links to its author, its comments, the people who commented, and the groups and pages it came from. A reader can follow any of those links to the next view. That data also changes constantly. A designed feed endpoint would have to return every relationship any screen on any client version might open next, or come in a variant per screen and version. Facebook meets the burden of proof, and I agree with its conclusion.

REST Follows One Link at a Time, GraphQL Follows Many

The round trips Facebook described are what REST does by design. Roy Fielding's dissertation defined REST around hypermedia, where each response carries links to the client's next steps and the client follows one per request. Fielding said the REST interface is "optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction." A feed is one of those other forms.

Screen shape	What fits	Why
One view at a time, with links to the next	Hypermedia REST, as Fielding defined it	One request per link
Many items per view, each linked to many relationships, in a large graph that changes constantly	GraphQL	One request states the whole traversal
Known views taking a few planned paths through data that changes slowly enough to cache	Designed endpoints	The server ships each view's shape and caches what it can

Most APIs called REST today sit in the third row. They're RPC, designed operations over HTTP.

Each relationship a list item links into needs its own designed operation. For a first-party product, two questions test, screen by screen, whether its operations outrun hand-written endpoints:

Does each item in a list summarize itself and open one detail view, rather than link into many of its relationships?
Does the data behind those views change slowly enough for a designed endpoint to serve and cache it?

If both answers are yes for most screens, hand-written endpoints can keep pace. A marketplace's listing and detail screens pass both, since each card opens one listing whose text and photos are read far more often than they change.

Volatile data hurts a designed endpoint only when items also link into many relationships, because everything returned just in case must then be fetched fresh.

GitHub, Agents, and Graph Databases Face Unpredictable Queries

GitHub's repositories, issues, and pull requests form a graph that third-party callers traverse in ways GitHub can't predict, so no screen test applies. Over a graph database, GraphQL is a thin layer, and Neo4j's GraphQL Library compiles each operation into "a single Cypher query which is executed against the database."

AI agents compose queries as they go, which should make them GraphQL's best fit, yet Anthropic and Apollo both point them toward designed tools. Anthropic's agent-tool guidance recommends "a few thoughtful tools targeting specific high-impact workflows." Apollo's Model Context Protocol server, which exposes GraphQL as agent tools, offers an open execute tool but steers developers toward governing what agents reach "by adopting predefined persisted queries." Predefining an agent's queries gives up the reason to expose a graph to it, and those operations could just as well be RPC endpoints.

Open Queries Can Be Secured Without a Registry

These cases need open queries, and a server that accepts them accepts them from anyone, because the endpoint shows in the browser's network tab and the schema is often one introspection query away. A query like users(first: 100) { friends(first: 100) { friends(first: 100) { name } } } asks for a million records. Aliases let one request call a login mutation a hundred times, and a per-request rate limiter counts that as one attempt, as a PortSwigger lab shows.

The GraphQL Foundation's security guidance recommends limiting query depth, applying "a separate smaller limit to how deeply lists can be nested," capping operations per batch, and restricting aliases. Cost analysis prices the rest. It weights each field and charges for what a query could return rather than for the request that carried it, so a hundred aliased logins cost a hundred mutations.

GitHub caps a call at 500,000 nodes and meters each user at 5,000 points an hour, scored by the connections a query could traverse. The price is a cost model the server tunes field by field as the schema grows.

GraphQL's Own Best Practices Steer Private APIs Away From the Point

Trusted Documents Are for First-Party APIs Only

For private APIs, GraphQL's own guidance avoids tuning a cost model. It refuses any query the server hasn't seen. With persisted queries, the client's operations are hashed and registered at build time, and in production the client sends only the hash. Relay, Hot Chocolate's OnlyAllowPersistedDocuments option, Apollo's safelisting, and the GraphQL Foundation's "trusted documents" describe the same arrangement.

Benjie Gillam, a GraphQL Working Group contributor, calls an allowlist "very much a best practice" for the "vast majority of GraphQL users." The GraphQL Foundation adds that "trusted documents can't be used for public APIs." So for a private API, the recommended practice gives up open queries and leaves client teams composing operations at build time.

Serving Partners Brings Back the Cost Model

In my experience, what starts private often has to interoperate later, with a partner integration, another department, or a customer who wants their data. Some of those consumers expect HTTP conventions, such as per-resource URLs and HTTP caching. For every GraphQL endpoint I built, I ended up needing a more RESTful one beside it.

A designed contract can often open to outsiders behind auth and rate limits, because the server chose each operation and parameter, bounding its cost. The locked-down GraphQL API can't serve them as built. Registering operations for partners means the server team writes each one anyway. A gateway that translates for partners only moves those endpoints into infrastructure no domain team owns. Opening the API, or adding an open surface beside it, brings back the cost model the registry avoided.

Persisted Queries Carry the Costs of Both Models

Under persisted queries, the server stores and can refuse each document. It doesn't choose which fields combine in a request. The client gives up runtime freedom, and the server gives up design.

Both teams run the registry. Its clearest advantage is that it ties each field to the builds that read it, which makes deprecation safe, especially with long-lived mobile apps. A backend-for-frontend matches that only by keeping each release's response version until its last app build retires.

Registration in CI keeps attackers' queries out, but not a teammate's costly one. No server author designed that document's field combination, so catching its cost means reviewing or scoring each document at registration.

A Monorepo Makes the Registry Cheaper

Gillam's write-up says the allowlist practice "has been used within Facebook since before GraphQL was open sourced." So Facebook did know its queries before they ran, but only because its client build generated the list rather than a team writing it.

Facebook's code also lives largely in one repository, which Meta's post on its Sapling source control system puts at "tens of millions of files." Meta hasn't said its persisted queries depend on that, but I read them as benefiting from it. When client, compiler, and registry change in one commit, the hash is a detail of one build rather than a contract between two teams.

Where client and server ship from separate repositories, the registry becomes that contract. Without a graph like Facebook's, keeping a query language only to lock it down is forcing a square block into a round hole.

Most Teams Don't Need GraphQL

Most teams adopting GraphQL don't have a graph nobody can predict. Their screens follow known paths, and designed endpoints give them what they wanted from GraphQL. For those teams, the server should own each operation, and the teams should plan together instead of building around each other.

GraphQL earns its place where nobody can predict the queries. That covers public graph APIs like GitHub's, agents whose tasks can't be listed in advance, graph databases, and graphs like Facebook's, whose screens take too many changing paths to keep an endpoint for each. Persisted queries fit only that last case. Anywhere else, a team that locks GraphQL down has already listed its queries and keeps a query language only to compose them at build time.

None of this says you can't use GraphQL, or that trying it was a mistake. It says to prove you have Facebook's problem before you take on Facebook's solution.

Jev AI Explained: A Single-Pass "Decision Model" for the Tasks LLMs Are Overkill For

Mon, 12 Oct 2026 16:18:29 +0000

Look at where LLM calls actually go in a typical production system and a pattern shows up fast. A big share of them are not conversations or content generation. They are decisions: which queue does this ticket belong to, how risky is this action, is this document relevant. Each one is a pick-from-a-list problem, and each one is being solved with an expensive, slow, text-generating model.

Jev, released in September 2026 by TypeSafe AI, targets exactly that. It is a decision model, not a language model. It cannot write text, but it can answer structured questions quickly and attach a calibrated confidence score to every answer.

Why the Architecture Matters

An autoregressive LLM produces output token by token, so even a one-word label requires a chain of sequential forward passes. That chain is the main source of latency and cost.

Jev is non-autoregressive. It reads the input once and emits a typed result in a single forward pass. That removes a whole class of problems at the source: no malformed JSON, no hallucinated fields, no retry logic for broken output. In practice, responses land in roughly 70 to 500 milliseconds, versus multiple seconds for a generative model on the same job.

The constraint is obvious: Jev only evaluates structured decisions. No free text, no multi-step reasoning, no arithmetic.

The Three Primitives

Every request is composed from three question types:

Choice selects one label from a fixed set, for example routing a ticket to billing, technical support, or sales.

Score rates something on an ordered scale, for example severity from low to critical.

Noul answers a yes or no question with a probability between 0 and 1, for example whether an action is destructive.

Because each result carries a confidence value, your application can set thresholds: act automatically above one level, escalate to a human or a full LLM below it.

Cost: Read the Fine Print

TypeSafe's headline claim is 40 to 400 times cheaper than standard LLMs. That holds against a frontier reasoning model. Compared with the small, cheap models most teams already use for classification, the realistic saving is closer to 5 times. At high request volume that still adds up, but it is a different story than the marketing number suggests.

Where It Fits in a Real System

The strongest use cases replace an LLM call that was only ever a multiple-choice question.

The first is routing in front of expensive steps. In a multi-step agent pipeline, a fast classifier can decide whether a request needs a full agent run, a simple lookup, or no model call at all. This is the same principle behind well-designed AI agent orchestration, where cheap, deterministic decisions gate the costly ones.

The second is safety gating. Before an agent executes a delete, a schema change, or a write to production data, a Noul check can block it or require approval, with a probability you can threshold.

The third is retrieval filtering. In a RAG pipeline, scoring each retrieved chunk for relevance before it reaches the main model cuts both noise and token spend, which is a practical optimization for any generative AI and RAG system running at scale.

Support triage and lead scoring follow the same shape: several independent scores from one request, combined by your own business logic instead of a long prompt.

Known Limitations

Jev has documented blind spots you should design around:

It is unreliable at math and numeric comparison, so keep calculations in application code.

It struggles with counting items in a list. Query per item instead.

It treats dates as plain text and cannot reason about ordering in time.

It only sees what is explicitly in the input. Implied context is missed.

Like any model reading untrusted text, it is exposed to prompt injection, so input still needs sanitizing.

The practical conclusion is to treat it as a fast classifier inside your own code, not as a reasoning engine.

A Category That Is Filling Up Quickly

Within about three weeks of launch, alternatives appeared. Cloudflare shipped two open-weight models, Clef and Clef-flash, with a compatible API and image input that Jev lacks. Clef-flash is reported at around 39 milliseconds on Cloudflare's edge network. Laya, an open-source option, runs locally without a GPU, with claimed latencies as low as 9 milliseconds. OpenAI has previewed its own decision-oriented API, though it was not broadly available at the time of writing.

None of these figures are independently verified, so benchmark on your own traffic before choosing a provider.

Takeaway

If a meaningful part of your LLM spend goes to routing, scoring, or yes or no gates, decision models are worth testing. They will not replace your main model, but they can take over the decisions that never needed one. Teams that want help deciding where this kind of component belongs and shipping it to production can look at InterCode's custom AI development work.

Raising the bar on software delivery

Mon, 12 Oct 2026 13:24:05 +0000

In this episode of Continuous Delivery Office Hours, Tony, Bob, and Steve propose a raising of standards for software delivery.
While most development teams can agree on some basic capabilities, like using version control rather than a network share to store code, as soon as you go beyond the simplest things, the level of agreement drops dramatically.

But we have decades of high-quality research to guide us toward something better, and we should take inspiration from other industries that continually raise the bar for "what good looks like".
Watch now to delve into:

The evolution of delivery standards: While high-performing teams worked out new standards as they increased their throughput through Lean and Agile, those who haven't raised the bar are now struggling to improve throughput with AI.
Feedback is the litmus test: Just as Ron Westrum captured the essence of organization in terms of information flow, we can test our software delivery by paying attention to feedback loops.
The 23 things: We think there are about 23 interconnected capabilities and practices that set the minimum standard for professional software delivery (but we don't list them all in this episode!
Shopping, watertight houses, and the Omaha basement: We draw inspiration from buildings to highlight the necessity of interconnected practices.

Post-Mortem Analyses of Publicly Known AI Security Incidents: Key Architectural Insights

Mon, 12 Oct 2026 10:09:49 +0000

As artificial intelligence systems transition from isolated proof-of-concept models to deeply integrated production architectures, their attack surface has undergone a fundamental shift. Conventional software security paradigms—which focus primarily on memory safety, injection vulnerabilities, and role-based access control—are inadequate for systems based on probabilistic outputs, unstructured inputs, and vast parameter spaces.

The investigation of publicly disclosed AI security incidents provides crucial empirical data for hardening modern machine learning pipelines. The following section analyzes recurring classes of vulnerabilities, their systemic causes, and architectural safeguards from a technical perspective.

Taxonomy of Vulnerabilities in AI Production Systems

A. Prompt Injection & Indirect Control Flow Hijacking

Mechanism: Untrusted external data (e.g., crawled web pages, emails, or user-uploaded files) contain instructions designed to bypass system prompts or the control mechanisms of autonomous agents.
Incident profile: LLM-based agents execute external API calls or apply Retrieval-Augmented Generation (RAG) to malicious datasets. When the model processes untrusted tokens, it interprets them as executable instructions rather than mere data.
Cause: Failure to maintain a strict separation between the instruction context (system/developer prompts) and the data context (user inputs and retrieved documents).

B. Training Data Poisoning & Supply Chain Compromise

Mechanism: Adversarial manipulation of datasets or pre-trained base models prior to fine-tuning or deployment.
Incident profile: Attackers inject subtle backdoors or trigger patterns into open-source datasets. This causes the models to behave securely on standard evaluation benchmarks while failing catastrophically under specific trigger conditions.
Cause: Inadequate provenance tracking, lack of cryptographic verification of training artifacts, and absence of automated anomaly detection for training corpora.

C. Modeling of Infiltration, Extraction, and Inversion

Mechanism: Systematic querying of proprietary models to reconstruct proprietary weights, extract training data, or infer hyperparameters.
Incident profile: Compromise of commercial APIs via mass side-channel queries that map decision boundaries or expose personally identifiable information (PII) embedded during training.
Cause: Excessively permissive rate limiting, absence of output perturbation or differential privacy mechanisms, and unlimited granularity of API responses.

Structural Root Causes

The analysis of these incidents reveals three systemic patterns of failure in technical organizations:

The "software vs. statistics" category error: treating ML models as deterministic components rather than as non-deterministic statistical systems. Since models execute semantic logic rather than syntactic rules, conventional input validation (such as sanitization using regular expressions) falls short when dealing with semantic payloads.
Unrestricted operational authority for agents: Granting LLM agents extensive, unrestricted capabilities to use tools (e.g., shell access, unvalidated database write permissions, or unrestricted HTTP clients) without intermediary human validation or limiting access rights to the necessary minimum (principle of least privilege).
Blind spots in the supply chain: reliance on unverified model checkpoints and third-party datasets downloaded from public platforms—without robust artifact scanning and behavior-based regression testing.

Engineering Playbook: Architectural Safeguards

To develop a resilient AI infrastructure, teams must implement a multi-layered defense strategy (defense-in-depth) across the entire lifecycle:

Dual-model protection mechanisms & input sanitization: Implement intermediate classification layers (smaller, highly specialized security models) to evaluate incoming data before it reaches the primary processing mechanisms. Treat all data chunks retrieved via RAG as untrusted input.
Isolation of structural context: Leverage new API primitives and token-level structural markers that enforce strict separation between system logic and dynamic user data (e.g., explicit delimiters for chat templates and role isolation).
Execution of tools based on the principle of least privilege: Agentic tool environments run in a sandbox. API calls are embedded in deterministic middleware that validates parameters against strict JSON schemas and business logic constraints prior to execution.
Artifact Provenance & SBOMs for AI: Implement Software Bill of Materials (SBOM) standards for datasets and model weights. Use cryptographic hashing (SHA-256) and secure artifact registries to track every stage of the MLOps pipeline.
Continuous Red Teaming & Behavioral Fuzzing: Integrate automated adversarial testing (e.g., automated prompt injection fuzzers) into CI/CD deployment pipelines to detect regressions in the model's security boundaries prior to production deployment.

Conclusion

Securing AI systems is an ongoing process; it requires a synthesis of traditional cybersecurity disciplines and novel, ML-specific threat models. As AI engineers and technical architects, our task is to develop systems that anticipate adversarial manipulation by default and shift our approach from reactive patching to proactive, demonstrable resilience.

What architectural patterns has your team implemented to minimize risks from prompt injection and supply chain attacks in production? Let’s discuss this in the comments.

Why LeetCode Interviews Measure the Wrong Thing

Thu, 08 Oct 2026 20:56:22 +0000

Coding tests can effectively filter out candidates who can't write working code. That's a legitimate purpose. The problem emerges when organizations test for timed algorithm-puzzle skills in roles that require entirely different capabilities: system design, debugging distributed systems, architectural decision-making, or cross-functional collaboration.

A senior engineer who can architect scalable systems and debug production failures shouldn't be rejected because they can't solve a dynamic programming puzzle in thirty minutes.

The interview tests for skills the job doesn't require while ignoring skills that define success in the role.

Timed Puzzles Test a Skill Most Roles Rarely Use

Some roles genuinely require strong algorithm skills. If you're building database engines, compilers, or performance-critical systems, testing for algorithmic thinking makes sense. The problem is applying this filter by default regardless of what the job actually requires.

A problem a competent engineer can solve cold with basic data structures, like reversing a linked list or finding duplicates with a hash map, checks the coding fluency every role needs. The mismatch starts with problems that are hard to finish in thirty minutes unless you recognize a named technique the role doesn't use, such as dynamic programming or a specific graph algorithm. The argument here is about those technique-dependent problems. Difficulty labels on practice sites don't mark that line, since many medium problems are plain hash-map work.

Most software engineering roles demand system design thinking to architect scalable solutions, production debugging to troubleshoot complex failures, cross-functional collaboration to align technical decisions with business needs, and trade-off evaluation to balance competing constraints. Algorithm optimization rarely appears in the day-to-day work, yet coding rounds built on technique-dependent problems test almost exclusively for it. Larger interview loops add design and behavioral rounds, but the coding round often comes first as a screen. When that screen uses technique-dependent problems, a candidate who fails it never reaches the rounds that sample the job.

Even when performance optimization matters in typical production systems, it's usually about database query optimization, caching strategies, network request patterns, and system architecture rather than the algorithmic complexity of isolated functions. For roles building web applications or distributed services, the engineer who can identify an N+1 query problem delivers more immediate value than the engineer who can implement graph traversal algorithms from memory. Different roles, different requirements.

A candidate might struggle to find the optimal solution to a graph problem in thirty minutes while being excellent at diagnosing why a distributed system exhibits inconsistent behavior under load. They might fumble dynamic programming problems while being the engineer who catches architectural flaws in design reviews. When the role requires the latter skills, the interview measures their weakest capability and ignores their strongest. Public data doesn't show how often that happens across a hiring pool, so the case here rests on what the test contains rather than on counted rejections.

Drilled Patterns Pass for Reasoning

Pattern matching also masquerades as algorithmic thinking. Practice sites group their problems by technique, such as two pointers, sliding window, and dynamic programming. A candidate who has drilled enough of them can solve a hard problem by recognizing which template it fits. Recognizing patterns is a useful skill, and engineers diagnose production issues partly that way. But recognizing a problem from a drilled catalog measures hours of preparation.

Under time pressure, recall produces the same answer on the whiteboard as reasoning from first principles. An interviewer can try to tell them apart by changing a constraint and asking why. That probe works only as well as the interviewer runs it, so the signal comes from the interviewer, not the format.

One Live Round Adds Noise

The live format adds noise of its own. In a randomized controlled trial published at ESEC/FSE 2020, Mahnaz Behroozi, Chris Parnin, and colleagues gave 48 computer science students a whiteboard problem, half with an interviewer watching and half alone in a private room. Being watched cut performance by more than half. That penalty comes from observation, not from the puzzle, so it applies to any live exercise.

Aline Lerner found a second source of noise in 2016 data from interviewing.io, a practice-interview platform. Only about a quarter of candidates who interviewed more than once performed consistently, and many who earned a top score in one interview earned a failing score in another. That instability belongs to any single interview. Neither finding favors one format's content over another's. Both argue against letting one live round decide, and a lone timed puzzle used as a gate is exactly that. It measures preparation and composure on that day alongside a skill the role rarely uses.

The Screen's Misses Fall Unevenly

The usual defense is that this filter accepts false negatives to avoid false positives, because a bad hire is expensive and a large company can afford to lose good candidates. The trade can hold for the company, but it rests on an assumption. For a role that doesn't use the puzzle skill, the screen catches bad hires only as far as puzzle performance happens to track the skills that predict success in that role.

Its false negatives also aren't spread evenly. A filter that rewards drilled patterns favors whoever had time to drill, and the talent pool narrows in that direction. Defenders read those drilling hours as a sign of conscientiousness, but hours available to drill also track free time, so the signal mixes diligence with circumstance.

Most Roles Run on Debugging, Design, and Review

Most software engineering involves systems thinking applied to building products. That means understanding how components interact, how failures propagate, how changes impact users, and how technical decisions affect business outcomes. It means reading production metrics to diagnose issues, reviewing code to catch bugs before deployment, mentoring junior engineers to improve team capability, and collaborating with product teams to ensure what gets built actually solves user problems.

Consider what a senior application engineer actually does. They investigate why a service's 99th percentile latency spiked from 200ms to 2 seconds overnight and discover a deployment introduced inefficient database queries. They review a pull request, identify a race condition in concurrent access to shared state, and suggest refactoring with proper locking. They pair with a junior developer to debug why a feature works locally but fails in staging due to differences in environment configuration. They join a design meeting and explain why the proposed real-time notification feature requires WebSocket infrastructure and database schema changes that will take three weeks, not three days.

None of these activities resemble solving algorithm puzzles under time pressure.

Work-Like Formats Keep What Puzzles Get Right

Puzzles persist for reasons that hold up. They're cheap to run at scale, every candidate gets a comparable problem, and the scoring is easy to defend. That consistency comes from giving every candidate the same problem and scoring it against written criteria. Neither step depends on puzzle content, and a work-like exercise can do both.

Prediction Is Tied, So Content Decides

Puzzles' strongest defenders also treat them as a proxy for general reasoning ability. Public data tying algorithm-round scores to later job performance is scarce, so the case for or against puzzles has to lean on general selection research. Paul Sackett and colleagues re-analyzed personnel-selection studies in the Journal of Applied Psychology in 2022. They estimated validity at .42 for structured interviews, .40 for job knowledge tests, .33 for work samples, and .31 for cognitive ability. Those are averages across occupations, and work samples don't clearly beat cognitive ability on their own.

So the numbers don't crown a content winner. When prediction is roughly tied and no outcome data exists for a particular test, selection practice falls back on content validity. The US Uniform Guidelines on Employee Selection Procedures (29 CFR 1607.5) describe it as showing that a test's content is "representative of important aspects of performance on the job."

Puzzles fall short of that standard whichever way you classify them. Treated as a cognitive-ability test, a puzzle is a proxy that ranks below job knowledge. Treated as a job-knowledge test, it samples algorithm technique the role rarely uses. Either way, a structured exercise built from the role's own work meets the content standard more directly. That standard is about defensibility, not a proven gain in hires. But when prediction is tied, matching the job's content is the reason left to prefer one format, so it breaks the tie.

Two problems from the first section, drilled recall and uneven misses, also favor the work-like format. Preparation bends the puzzle proxy, because puzzles draw from a public catalog of thousands of problems with published solutions, so a candidate who has seen the problem is measured partly on recall. And the screen's false negatives skew toward people without time to drill.

A work-like format costs more. Scenarios have to be written per role, calibrated, and rotated, and interviewers have to be trained and recalibrated on the rubric. Most of that cost recurs per role. The longer formats also take more interviewer time per candidate, so a short debugging or review exercise suits the screen and the longer ones suit later rounds. A puzzle screen is cheaper to run, but its cost lands elsewhere, in each candidate it turns away for content the role doesn't use. No public outcome data shows which cost is larger.

Match the Format to the Work

Some companies have made that switch, and the community-maintained Hiring Without Whiteboards list on GitHub catalogs hundreds of them. The list shows the switch is workable at scale, not that those companies hire better, so the case for it rests on the research above. The format should follow the work the role does most:

When the role mostly...	Test it with	What it reveals
Builds performance-critical internals like database engines, compilers, or low-latency libraries	Algorithm and data-structure problems, ideally drawn from the domain	Whether the candidate can reason about complexity where it decides the outcome
Extends and maintains a product codebase	Pair programming on an existing codebase	Collaboration, code quality standards, and how they work through ambiguous requirements
Delivers features end to end	A small take-home project with tests and documentation	Whether they can build a working feature without artificial time pressure
Runs services in production	Diagnosing a seeded bug from logs and metrics	How they form and test hypotheses about a failure
Shapes how services fit together	A system design discussion on a realistic problem	Their grasp of distributed systems, scalability patterns, and trade-off evaluation
Reviews others' work and mentors	Critiquing a real pull request	Whether they can find bugs, suggest improvements, and give feedback constructively

Some employers hire into a general pool and place engineers on teams later, so there is no single role to sample, and puzzles appeal because they are role-neutral. But a generalist pool still shares a core of work: reading unfamiliar code, debugging a failure, and reviewing a pull request. The debugging and review rows above sample that core for any team, and they meet the content standard where a puzzle doesn't.

Structure Makes Any Format Fairer

These approaches aren't perfect. Take-home projects favor candidates with more free time, the same skew a drilled screen has, which is a reason to keep them small. Pair programming can feel stressful, and system design discussions are harder to standardize.

The observation penalty from the Behroozi study also hits every live row in the table, so give candidates private time to work before they discuss their approach wherever the format allows.

Private work brings its own problem, because AI assistants make unobserved work hard to attribute. Any private or take-home portion needs a live follow-up where the candidate walks through their work and extends it. That follow-up is still observed, but the candidate explains a solution they already reached instead of searching for one under a watching eye. The Behroozi study's private condition used the same shape, with participants solving alone and then explaining their work in a retrospective think-aloud.

Each format earns its place only when it's structured. That means the same scenario for every candidate, and a rubric written beforehand that lists what a strong answer finds, such as the seeded bug's root cause or the race condition in the pull request. Scenarios drawn from the company's own systems leak too, through interview-review sites. But they have no published solution set and can be rotated, so they resist drilling better than a public catalog does. Built that way, structure supplies the consistency and the job's own work supplies the content, rather than memorized algorithm patterns.

Learn Algorithms for the Work, Not the Interview

Build your algorithmic foundation because it matters beyond the interview. Understanding data structures helps you choose the right tool for the job, and knowing complexity analysis helps you write efficient code. Study algorithms to become a better engineer, not just to pass interviews.

But recognize that grinding LeetCode is interview preparation, not professional development. The hours spent memorizing dynamic programming patterns could be spent building systems, contributing to open source, or learning distributed systems concepts. Those investments develop professional capability while LeetCode grinding mostly develops interview performance.

Strong engineers architect scalable systems, debug production failures, and collaborate across teams to ship features. Whether they can solve algorithm puzzles in thirty minutes is an indirect, noisy signal of any of that.

The problem isn't the candidates' capability. It's an interview process that measures the wrong thing for the role.

Until hiring changes, engineers are stuck preparing for interviews that don't reflect the job, while companies reject candidates on content that says little about the role.

🧩 Engineering a Machine-Validatable Contributor Evidence Workflow

Thu, 08 Oct 2026 13:50:59 +0000

🧩 Engineering a Machine-Validatable Contributor Evidence Workflow

I just merged a new infrastructure milestone into MyZubster:

PR #1572 — standardized contributor interoperability checkpoints

The problem I wanted to solve was not just “how do we document a successful contributor test?”

The real problem was:

How do we make technical evidence portable across contributors, technologies and verification paths without losing provenance or inflating claims?

Until now, MyZubster already had reproducible contributor checkpoints, but the structure was mostly expressed through Markdown, PR discussions and manually reviewed evidence.

That works at small scale.

It does not scale well when multiple contributors are producing different kinds of evidence.

So I introduced a common machine-readable format:

myzubster.interoperability-checkpoint.v1

Each checkpoint now describes:

contributor identity
repository / PR
immutable commit reference
technical bridge type
exact scope
test environment
reproduction procedure
observed output
evidence state
limitations
canonical provenance

Why the limitations field matters

One of the most important design decisions was making limitations mandatory.

A technical checkpoint should not silently become a broader claim.

For example:

A successful broker-to-agent test does not automatically mean:

full decentralization
direct P2P networking
security certification

A research Knowledge Card marked SUPPORTED should not become “scientifically validated.”

A signed webhook regression should not become “payment settlement verified.”

The schema is intentionally designed to preserve those boundaries.

Evidence state is constrained by observation

The validator enforces an important rule:

TESTED

requires:

observation.result = PASS

That sounds obvious, but encoding it into the schema means it is no longer just a documentation convention.

It becomes something CI can reject.

AJV + CI integration

I added an AJV-based validator and connected it to the existing Continuous Evidence Gate.

So the pipeline now looks like this:

Contributor work
        ↓
Scoped reproducible test
        ↓
Machine-readable checkpoint
        ↓
JSON Schema validation
        ↓
Continuous Evidence Gate
        ↓
Human review
        ↓
Canonical merge

The important distinction:

automation validates evidence structure; it does not replace engineering judgment.

Testing the format across different contribution types

I did not want the first schema version to be designed around one specific contributor.

So the initial dataset already includes multiple technical paths.

N4K48 / Nicola

Runtime interoperability:

MyZubster VPS
→ authenticated HTTPS broker
→ contributor-controlled agent
→ local catalog
→ result returned to Bridge

wasim-builds

Security regression:

unconfigured auth → rejected
missing credential → rejected
wrong credential → rejected
correct credential → accepted

khongten124 / Open Period Care

Semantic evidence bridge:

public research evidence
→ structured ingestion
→ contributor-scoped retrieval
→ Knowledge Cards returned
→ original SUPPORTED state preserved

Aming9303

Signed webhook regression:

HMAC signing
→ stable delivery identity
→ unsigned request rejection
→ replay protection
→ stale-event rejection

These are very different systems.

That is exactly why using the same evidence structure is useful.

Current maturity

I would describe the workflow as:

repeatable
→ auditable
→ multi-contributor
→ machine-validatable

The next layer will be:

automated ingestion
→ Contributor Passport linkage
→ Knowledge Graph integration

while preserving the original evidence state and provenance.

Merge result

PR #1572 passed:

CI – Test & Lint
Security Audit
Continuous Evidence Gate
policy checks

Canonical merge:

2be06b7e34f22744ecf81a9daef12f7ba11864be

What I like about this milestone is that it is not visually impressive.

There is no shiny UI involved.

But it improves something much more fundamental:

the trust model between contributors and the platform.

Instead of:

“This was tested.”

the system can increasingly answer:

Who tested it?
Against which commit?
In which environment?
Using which procedure?
What was observed?
What does the evidence not prove?

That is the direction I want to keep pushing.

Evidence should be inspectable data, not just prose.

CoderLegion #SoftwareEngineering #JSONSchema #AJV #CI #DevOps #SystemDesign #OpenSource #Reproducibility #Testing #DeveloperTools

What Building One API for Research, Media, and Editable Files Taught Me About Reliable AI Products

Wed, 07 Oct 2026 18:48:05 +0000

I started 3Stone AI because the work I wanted to do rarely ended in one format. Research might need to become a brief, a presentation, or a spreadsheet. An image might become a video. A plan might need to become an interactive tool. I wanted research, media generation, editable Office files, and useful software available through one product and one API instead of making the customer assemble a chain of unrelated services.

At first, that sounds like an interface problem: put several models behind a chat box and route each request to the right provider. Building the real product taught me that routing is only the beginning. The harder work is making every accepted request produce a result the customer can actually open, use, download, revisit, and revise.

Real customer problems forced that lesson repeatedly. A request could pass automated tests while the customer journey still broke. Audio could exist but be the wrong kind of audio. An entitlement could be correct in one system and stale in another. A provider could report success while the exported file was unusable. I stopped treating “the test passed” as the finish line. I began requiring evidence from the delivered output and the complete customer journey.

One API still needs capability-specific contracts

Different kinds of work do not become identical merely because they share authentication. Research can complete synchronously and return sources. Image or Office generation may require a durable background job. A spreadsheet is not successful because a ZIP of XML exists; it must open in Excel, preserve formulas, and remain editable. A retry after a network timeout must not submit the provider job twice or charge twice.

The current 3Stone developer contract therefore uses capability-specific endpoints behind common authentication, idempotency, usage, job, and artifact conventions. Certified public routes include chat, research, image analysis, image generation, editable PowerPoint, Excel and Word generation, music, and interactive tools. Video exists in the customer product, but it is not presented as a public API capability until the external lifecycle meets the same certification standard.

That boundary matters. A useful API should describe what developers can verify today, not everything an internal product can theoretically attempt.

Make asynchronous work explicit

Long-running AI work creates an awkward interval: the provider may have accepted a paid job even when the HTTP response never reaches you. Retrying blindly can create two jobs. Reporting failure immediately can also be wrong.

We use durable request and job identities, save provider receipts as early as possible, and separate states such as accepted, running, completed, failed, and reconciliation required. The customer polls the 3Stone job rather than learning each provider's lifecycle.

A simplified image request looks like this:

const base = "https://one.3stoneai.com";
const idempotencyKey = crypto.randomUUID();

const submitted = await fetch(`${base}/v1/images`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.THREESTONE_API_KEY}`,
    "Content-Type": "application/json",
    "Idempotency-Key": idempotencyKey,
  },
  body: JSON.stringify({
    prompt: "An editorial photograph of a solar-powered neighborhood library at sunrise; no text or logos."
  }),
});

const job = await submitted.json();
if (submitted.status !== 202) throw new Error(job.error?.message ?? "Request rejected");

while (true) {
  await new Promise(resolve => setTimeout(resolve, 2500));
  const response = await fetch(`${base}/v1/jobs/${job.job_id}`, {
    headers: { Authorization: `Bearer ${process.env.THREESTONE_API_KEY}` },
  });
  const current = await response.json();
  if (current.status === "completed") break;
  if (["failed", "reconciliation_required"].includes(current.status)) {
    throw new Error(`Job ended as ${current.status}`);
  }
}

The important details are not glamorous: bearer keys stay on the server, every mutating request has a unique idempotency key, accepted asynchronous work returns a durable job ID, and ambiguous provider outcomes are reconciled instead of automatically replayed.

A fallback must support the same job

One recurring integration mistake was treating any available model as a fallback. It is not enough for two providers to share an output label such as “image” or “video.” A text-to-image endpoint may not edit an image. A text-to-video endpoint may not replace a character in existing footage. A video endpoint that returns silent footage cannot honestly claim synchronized audio.

That changed how I think about recovery. A fallback is eligible only when it supports the requested inputs, preservation behavior, output format, and practical limits. It also must be safe for the particular failure. A known authentication rejection before execution is different from a connection loss after a provider may have accepted a paid request.

For image generation, 3Stone's separate-provider fallback is constrained to confirmed operational failure classes. It does not activate for moderation decisions, ambiguous post-submission failures, or arbitrary errors. For video, audio routing distinguishes source audio, ambience and effects, dialogue, music, narration, and intentional silence. ElevenLabs can be a recovery option for appropriate speech; it is not automatically attached to every video.

The broader lesson is simple: recovery that changes the customer's request is not recovery.

Validate the artifact, not the provider status

The defect pattern that most changed my release standard was seeing technical success without a usable customer outcome. A provider could say “succeeded,” a route could return HTTP 200, and a test suite could be green—yet the file might not open correctly, the audio might be irrelevant, or the next customer action might fail.

Now I want evidence at the artifact boundary. Images must decode and meet the expected dimensions. Videos are probed for duration, codec, dimensions, and required audio. PowerPoint, Excel, and Word files are reopened and checked for their native structure, including slides, sheets, formulas, tables, and editability. Interactive HTML is rendered and exercised. Customer certification includes creation, preview or playback, download, persistence, reopen, and revision where the capability supports it.

This standard also applies to account and entitlement work. A deletion button is not verified because it displayed a confirmation dialog; the customer must return to signed-out state and the intended account data must actually be gone. A microphone permission prompt is not proof of a live voice session; the system needs a real captured turn, an audible response, interruption behavior, transcript persistence, and correct accounting.

Idempotency is a billing promise

Idempotency is often described as an API convenience. For paid AI work, it is also a billing guarantee.

Each mutating request binds the caller's idempotency key to a fingerprint of the operation. The same key with the same request returns the original outcome. The same key with changed input is rejected. Before expensive execution, the system reserves a bounded allowance. Completion settles recorded usage. Known pre-execution failure releases the reservation. An ambiguous provider outcome enters reconciliation instead of triggering another paid attempt.

We test the behavior, not just the presence of the header. In production certification, an immediate replay of a completed request returned the original request identity without creating another charge. That evidence is more meaningful to me than a unit test that merely proves the code reads Idempotency-Key.

Defects became architecture

Several parts of the architecture came directly from customer-facing failures:

Provider routing became capability-aware because “another model” was not necessarily compatible.
Recovery gained durable receipts because a timeout did not prove that no paid job existed.
Audio became intent-aware because generic narration could technically make a file audible while making the result worse.
Entitlement checks moved toward an authoritative shared view because stale plan state could break an otherwise correct journey.
Office and media delivery gained output inspection because generated bytes were not proof of a usable artifact.
Account deletion gained resumable phases because destructive work must finish completely and safely retry after partial failure.
Alerts gained provenance because certification tests and capability questions should not be reported as failed customer jobs.

These were not abstract platform goals. They were responses to moments when the product's internal story and the customer's actual experience disagreed.

The public request can stay small

curl https://one.3stoneai.com/v1/chat \
  -H "Authorization: Bearer $THREESTONE_API_KEY" \
  -H "Idempotency-Key: first-request-001" \
  -H "Content-Type: application/json" \
  -d '{"model":"3stone-auto","input":"Explain exactly-once API billing simply."}'

The small surface is intentional. Underneath it, the system still needs tenant isolation, request identity, usage attribution, safe retry behavior, and a result the developer can verify.

What I would tell another builder

My biggest shift was moving from provider success to customer-verifiable delivery. If I hope another developer changes one thing after reading this, it is this: do not release the promise represented by an endpoint until you have followed the real output through the customer's next action.

Planning for Plan Continuation Bias: How to Keep Building the Right Thing

Wed, 07 Oct 2026 17:08:45 +0000

Many of the software projects that go wrong go wrong through misalignment. They solve a need nobody had, accept tradeoffs nobody weighed, or carry risks nobody examined. PMI's 2014 survey of more than 2,000 practitioners put poor requirements management behind almost half of projects that missed their goals. Alignment isn't a phase that ends when the work starts. Every plan, discovery, commit, and test is a chance for the work to drift away from what it was meant to accomplish, and a chance to pull it back.

Pulling it back requires something most advice skips over. You have to stop. For years, I couldn't. When a better idea surfaced mid-implementation, my first instinct wasn't to evaluate it but to finish what I was doing. By the time I finished, the switching costs had mounted and the moment for cheap redirection had passed. It was rarely pride, just a pre-rational impulse to proceed, as though stopping mid-stride carried some cost that continuing didn't.

What changed it for me was learning to think about work in terms of business value and to treat my own direction as an assumption to test rather than a plan to complete. I stopped being a code cowboy. The impulse didn't disappear, though, and it isn't mine alone. It has a formal name, and it explains why realignment so often fails to happen.

Continuing Wins Because It Takes Less Thought

Pilots call it get-there-itis. Researchers call it plan continuation bias. Judith Orasanu and colleagues at NASA's Ames Research Center went through the tactical decision errors in the 37 accidents covered by the NTSB's 1994 review of flightcrew-involved major accidents. Of the 51 tactical decision errors in that corpus, 38 were decisions to continue the original plan despite cues suggesting a different course of action.

The ratio isn't a pure measure of bias. The sample counts only flights that ended in accidents, so nobody knows how often crews continued in similar conditions and landed safely. Crews also choose to continue far more often than to divert, so continuing has more chances to go wrong. What the cases do show is crews who had the cues calling for a change and carried on anyway.

Orasanu's team locates much of the cause in effort rather than stubbornness. Revising your understanding of a situation and working out a new course of action costs more cognitive work than continuing with a plan whose details are already settled. People tend to take the cheap operation when it's available. Workload, time pressure, and ambiguous signals all shrink the budget for the expensive operation at exactly the moment it's needed. The original plan doesn't win the argument. It wins by default, because it never has to have one.

The stakes and the minute-by-minute time pressure don't transfer to software, but the default does. Revising a plan still takes more thought than continuing it, and the budget for that thought still shrinks under deadlines and under commitments other people have built on. There is evidence for this at the scale of whole projects. Mark Keil's 1995 case study of a runaway software project found a failing course of action kept alive by project, psychological, social, and organizational factors together. Nobody has measured the single-task version. It rests on the same mechanism.

What software lacks is the warning. A pilot pressing into weather has minutes and no undo. A developer pressing on with the wrong approach has weeks, git can undo any of it, and the work ends the way good work does, reviewed and merged. The pilot who got away with it at least knows it was close. The developer rarely learns there was anything to get away with. When the cost does arrive, it arrives months later as slow delivery or a rewrite, blamed on tech debt rather than on the choice. So the software version of the bias seldom corrects itself.

Our own advice makes it worse. "Finish what you started" is plan continuation bias repackaged as a character virtue. It assumes that what you started should be finished, which is exactly the question the bias keeps you from asking. That isn't the Kanban rule of limiting work in progress, which stops new work from starting and says nothing about whether the current work is right. "Think before you act" assumes the thinking was sound, and once you've thought and decided, the decision has inertia. The problem was never acting without thinking. It's acting without reconsidering.

The Impulse Shows Up at Every Stage

Each stage of software work has its own version of the impulse, and each one lets the work drift a little further from the need before anyone looks up. Planning and discovery are the worst offenders, because the decisions made there shape everything built afterward, and their misalignment stays invisible the longest.

Planning: The First Workable Design Becomes the Plan

A plan is supposed to come from weighing options against the need, the tradeoffs, and the risk. Under the impulse, the plan is the first design that seemed workable, and the weighing never happens. Once that design is written into tickets, estimates, and a roadmap, it has momentum before a line of code exists. A team that sketches a synchronous integration in a kickoff meeting, estimates it, and schedules it rarely circles back to ask whether its behavior under failure fits the business's tolerance for delay. Questioning it now means reopening commitments other people rely on, so the plan proceeds unchallenged and its misalignment ships inside every task built from it.

Discovery: New Information Gets Filed as "Later"

Discovery is when the work learns something the plan didn't know: a requirement nobody mentioned, a constraint in a system you integrate with, users who behave differently than assumed. That is the moment realignment exists for, and it's the moment the impulse is strongest, because acting on the discovery means stopping work already in motion. So the discovery becomes a follow-up ticket, a tech-debt item, or a note for the retrospective. It's acknowledged without being acted on, and the work continues against a picture of the need that is now known to be wrong. Deferring is right when a discovery doesn't change the direction, and it's the bias when the discovery would have.

Coding: "Let Me Just Get This Working First"

This is the version I see most in newer developers, and they make it constantly. Finishing first and then reevaluating sounds like sequencing rather than avoidance. The plan is to weigh the other approach once this one works. Then you spend another hour building out the current approach. You write tests around it. Other code starts depending on it. A colleague reviews it and builds understanding of how it works. At that point, "considering the other approach" means throwing away not just your work but the organizational investment in reviewing, understanding, and integrating what you've built.

The impulse to finish manufactures the very sunk costs that now appear to justify not switching. That is what makes the bias self-sealing. Ordinary sunk-cost reasoning is a mistake about money you already spent. This one does the spending itself, then cites the receipt.

Barry Staw's 1976 study Knee-Deep in the Big Muddy found that people who feel personally responsible for an initial decision commit more resources to it when it starts failing, not fewer. Among participants who received the same news of failure, the ones who had made the original decision themselves invested more, so the escalation tracked authorship of the decision rather than evidence about it. In software, this looks like the developer who hits a warning sign, such as a test that only passes with a workaround. They spend two more days making the approach work rather than an hour checking whether a different one would be simpler.

Sometimes getting it working first is the right test, because some approaches can't be judged until something runs. A spike has a time box and an agreement to throw it away, so it tests the approach. "Let me just get this working" has neither, so it builds on the approach.

Testing: Proving Instead of Testing

Proving asks "can I make this work?" and the answer is almost always yes given enough effort. Testing asks "should I be making this work?" and that's the question alignment depends on. The impulse turns testing into proving.

Tests written after the approach is settled check that the code does what its author intended, and rarely whether what the author intended is what the need required. A green suite then reads as confirmation of the direction, when all it confirmed was the implementation. Tests that come from the need before the approach is chosen, such as a check against the business's tolerance for delay, are the ones built to fail a direction. Without them, the current assumption gets the full weight of implementation and verification, while the alternative gets a hypothetical conversation, maybe, later, if there's time.

A Team's Momentum Feels Like Validation

The group version is worse, and it doesn't work the way most people assume.

The familiar picture of groupthink is dissent being actively suppressed. That happens, but I'd argue a subtler and more passive version also exists, and it's harder to see. Nobody explicitly validated the direction. Nobody consciously suppressed alternatives. Everyone just assumed that someone else must have validated it, and the group's momentum itself became evidence that the direction is correct.

This shares something with the bystander effect, where responsibility diffuses until nobody acts, but it runs a step further. Diffused responsibility explains why nobody checked. Momentum read as confirmation explains why nobody felt the need to.

Questioning the direction doesn't just feel unproductive. It feels like you're slowing the team down, which in most team cultures marks you as the obstacle rather than the one asking the right question. From the inside, speed is indistinguishable from conviction.

Orasanu's team didn't isolate momentum, but their data does show social pressure feeding the bias. Alongside ambiguous conditions, they named organizational and socially-induced goal conflicts as the second context that contributes to plan-continuation errors. So the group is more than a second instance of the bias. It's one of the conditions that feed it.

Process Decides Which Direction Is Free

Most methods for keeping work aligned assume that when discovery arrives, someone will notice and act. None of them do the registering for you.

Fixed Intervals Put Correction at the Far End

Any cadence that commits a group to a fixed goal or set of work for a fixed window places the routine moment for changing direction at the end of that window. In a two-week sprint, the review is the official place to say the plan was wrong, and by the time it arrives two weeks of sunk cost have accumulated. The correction point sits exactly where the bias is strongest.

Scrum is not blind to this. The Scrum Guide fixes the Sprint Goal rather than the scope, and the Daily Scrum adapts the plan toward that goal every day. A Product Owner can cancel a sprint, and the sprint boundary itself is a replanning point. But daily adaptation serves the goal rather than questioning it, and cancellation is a lever someone has to decide to pull, which is the very decision the bias suppresses. The standup, as many teams run it, has each person report progress against work already committed, which quietly rewards continuation every morning. It is part of what the framework trades for predictability.

Some Processes Make Stopping the Default

Basecamp's Shape Up is organized around three moves that make a discovery cheaper to act on. A circuit breaker means unfinished work expires by default instead of receiving an extension, so stopping requires no argument from anyone. An appetite fixes the time and varies the scope, which makes dropping part of a plan an ordinary decision rather than an admission of failure. Shaping before committing means less has been invested at the moment the "this isn't right" signal arrives, giving the signal a chance to compete with momentum.

What matters about a process is which direction it makes free, and a sprint and a circuit breaker pull in opposite directions. The Scrum Guide returns unfinished work to the backlog, but nothing stops the next planning session from pulling it straight back in, and the work already done argues for exactly that. Under a sprint, stopping has to be argued for. Under a circuit breaker, expiry is the default, so continuing has to be argued for. Shape Up's six-week window is longer than a sprint, so its advantage comes from the default, not from stopping sooner.

Circuit breakers trip only at defined boundaries, so they miss the smaller discoveries between them, like the "this approach isn't quite right" signal on a Tuesday afternoon that the impulse to finish overrides.

What Helps When Awareness Isn't Enough

Knowing about the bias doesn't prevent it, because the impulse operates faster than reflection. What worked for me wasn't trying harder to notice. It was changing what I thought the work was. Once each task was a bet on delivering some business value, my approach to it became an assumption about how to deliver that value, and assumptions get tested. A plan you treat as a commitment pulls you toward finishing it. A plan you treat as an assumption pulls you toward checking it. Reframing changes what stopping feels like, but the impulse stays, so the moments to stop still have to come from structure.

Checking a Doubt Costs Less Than Building on It

For most people, stopping feels like failure or waste. You were making progress and now you're not. But once a doubt has surfaced, evaluating it is usually cheaper than building more on top of it, and seeing it that way changes the emotional calculus even when the impulse is still there.

Low Switching Costs Keep the Signal Audible

The impulse draws power from switching costs that accumulate with every hour of continued execution. Cheap experiments before commitment, small commits, well-defined interfaces, and feature flags all keep the cost of being wrong low for as long as possible.

A Fixed Rhythm Beats Waiting for Permission

The Pomodoro technique was designed for productivity, but it accidentally created the kind of permission structure this bias requires. Every 25 minutes, you stop. That forced pause is a moment where "am I still working on the right thing?" can be asked without the social or psychological cost that usually prevents reassessment. The technique puts nothing about direction into the break, though, so the pause needs a switch criterion, written in advance, to check against. Events can trigger the same check, like a test that needs a workaround or an estimate that overruns.

The natural objection is that these pauses cost working hours that would otherwise ship something. A check against a criterion written in advance is usually quick, though, and a doubt too big to check quickly deserves the time. Four hours of uninterrupted coding on a misunderstood problem doesn't produce the right solution, and the context-switching argument can't tell that case apart from the one where the four hours were well spent. Breaking focus on work you understood throws away context you had built. Breaking focus on work you hadn't examined throws away nothing you would want to keep.

The Stop Test

The pause only helps if something happens inside it. The person asking whether to switch is usually the one who made the original decision, and Staw's finding predicts they'll lean toward their own choice. Itamar Simonson and Barry Staw later tested ways to counter that, and one that worked was setting, in advance, a minimum target that triggers a change if it isn't met. So before starting, write down what result, cost, or discovery would make you switch, and which assumptions the plan depends on, so an unexpected discovery has something to contradict.

That puts the criterion before the sunk cost exists and gives every pause something to check against. It also guards against the opposite mistake, switching whenever a new idea merely feels better. The stop test is then three questions for the pause, answered against what you wrote down, ideally by someone who didn't make the original call:

Am I proving or testing? Can this be made to work, or should it be?
Is switching getting more expensive while I keep going? Compare what you would throw away by switching now with what you would have thrown away an hour ago. If that pile keeps growing while the doubt sits unresolved, you are building the sunk cost that will later argue against switching.
If I hadn't already started, would I choose this approach today? This is Peter Drucker's question from The Effective Executive, "If we did not already do this, would we go into it now?", scaled down from a program to a single task. A no here is the entire signal. Everything after it is negotiation with sunk cost.

Alignment Depends on the Ability to Stop

You can't realign after discovery if you can't stop long enough to receive the discovery. You can't weigh a tradeoff you never paused to see, or catch a risk while you're busy proving the approach works. Alignment isn't something you achieve at kickoff. You recover it at every plan, discovery, commit, and test, and each recovery starts with a stop.

So the work isn't only to try harder at noticing. The work is to arrange things so that stopping doesn't wait on noticing:

Rhythms that pause whether or not anything looks wrong, and check against criteria set before the work began
Boundaries that expire on their own
Commitments small enough that changing your mind costs less than defending it

Build those, and the moment you would otherwise have missed arrives on a schedule that isn't yours to override.

Analyzing TP-Link Tapo Protocol Using Wireshark

Wed, 07 Oct 2026 05:48:40 +0000

Introduction

Wireshark dissector for a proprietary protocol found on TP-Link Tapo cameras, dissecting device discovery packets on ports UDP/20002 and UDP/20004, as well as control and media stream packets on port TCP/8800 and keep-alive packets on port TCP/8001. Tested on TP-Link Tapo C100 IP camera.

While trying to understand how my personal accounts and personal data is being secured, I've done some research on TP-Link's Tapo Android application as well (it includes analysis of HTTP request signature and termID building processes), which can be found by following this link to a Github repo.

Filters

The following filters are programmed for this Tapo protocol dissector:

Port	Packet Type	Field Name	Description
UDP/20002	Device discovery packet	tapo.header	Full header, saved as raw bytes
UDP/20002	Device discovery packet	tapo.version	Packet version
UDP/20002	Device discovery packet	tapo.type	Packet type
UDP/20002	Device discovery packet	tapo.opcode	Operation code for current message
UDP/20002	Device discovery packet	tapo.payload_length	Payload length
UDP/20002	Device discovery packet	tapo.flags	Flags for current message
UDP/20002	Device discovery packet	tapo.align	Extra value for struct alignment
UDP/20002	Device discovery packet	tapo.serial_number	Serial number
UDP/20002	Device discovery packet	tapo.checksum	Cheksum
UDP/20002	Device discovery packet	tapo.data	JSON payload
UDP/20002	Device discovery packet	tapo.binary	Discovery packet + binary data
TCP/8800	Contol/media packet	tapo.device_stream_boundary	Device stream boundary
TCP/8800	Contol/media packet	tapo.content_type	Content Type
TCP/8800	Contol/media packet	tapo.content_length	Content Length
TCP/8800	Contol/media packet	tapo.session_id	Session ID
TCP/8800	Contol/media packet	tapo.encrypted	Payload encryption flag
TCP/8001	Keep-alive packet	tapo.data	JSON payload

Device Discovery Packets

Ports UDP/20002 and UDP/20004 are used with configuration and cryptography-related information being sent.

Header Structure

A writeup on command injection vulnerability within tdpServer daemon that listens on port UDP/200002, as well as a writeup on remote code execution (RCE) vulnerability found in the TP-Link AC1750 Smart Wi-Fi router, describe that a device discovery packet has 16 bytes long header with it's fields defined as follows:

The fields are identified as follows:

Byte 1: Packet version.
Byte 2: Packet type.
Bytes 3-4: Packet opcode.
Bytes 5-6: Packet payload length.
Byte 7: Packet flags.
Byte 8: Struct alignment.
Bytes 9-12: Serial Number.
Bytes 13-16: Checksum.

Below is the header's representation in Wireshark:

More information about header structure of the tapo device discovery packet can be found on this Stack Overflow discussion thread.

Payloads

Observed payloads include:

Device configuration messages, some data encrypted with AES key:

{
  "error_code": 0,
  "result": {
    "device_id": "REDACTED",
    "device_name": "Tapo C100",
    "device_type": "SMART.IPCAMERA",
    "device_model": "C100",
    "ip": "REDACTED",
    "mac": "REDACTED",
    "hardware_version": "4.0",
    "firmware_version": "1.1.21 Build 220511 Rel.42882n(4555)",
    "factory_default": false,
    "mgt_encrypt_schm": {
      "is_support_https": true
    },
    "encrypt_info": {
      "sym_schm": "AES",
      "key": "REDACTED",
      "data": "REDACTED"
    }
  }
}

Public RSA key:

{
  "params": {
    "rsa_key": "-----BEGIN PUBLIC KEY-----\\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvzPNrLCSw3Rqf+AKwla2\\nntENZ+P4UUwGDzwvSc5HyJIYkY5+aQ8tpdQAO6NfN1JlVAUm94gxH5cw9s+xGgok\\nusbhFCHvMFp2g0CxvFg1P225+fq8Hmqqm9JN/ZkL2qwR4to6yvn256IqUxzmwKi8\\nFiX60VmAkmeJDOy8XGcN8ch385T9/sX1KyKZNxzX4z5xJEOlwcr5ZdhUeOmB6dbF\\njFF8GliOKSZdE01GVgeZPqJA21/8EQa1yJy8v31o+NNRd3QJaWjIzFqjGs1sl+Zj\\niPggKWcR/LqGt8Dd031velv2gVgmyBIjqi779cxns8IA5zYsvJbkxm0pFr3CU3GZ\\nYQIDAQAB\\n-----END PUBLIC
KEY-----\\n"
  }
}

Control and Media Packets

A proprietary protocol that manages JSON control messages and encrypted media (audio + video) streams runs on port TCP/8800.

Header Structure

Header is formed from the first 28 bytes of a control and media packet. Fields are text based rather than byte-based:

Field Name	Description
----device-stream-boundary--	Every control/media packet starts with this text as a start boundary
Content-Type:	`application/json` for JSON payloads and `video/mp2t` for video stream
Content-Length:	Length of a single message
X-Session-Id:	ID of a current session
X-If-Encrypt:	Payload encryption flag; 0 - unencrypted, 1 - encrypted

Payloads

Payloads of this protocol include JSON-formatted stream information and encrypted video stream with a type of mp2t.

Video stream request:

{
  "params": {
    "preview": {
      "audio": [
        "default"
      ],
      "audio_config": {
        "encode_type": "G711alaw",
        "sample_rate": "8"
      },
      "channels": [
        0
      ],
      "deviceId": "REDACTED",
      "resolutions": [
        "HD"
      ],
      "track_id": "REDACTED"
    },
    "method": "get"
  },
  "seq": 1,
  "type": "request"
}

Response on success:

{
  "type": "response",
  "seq": 1,
  "params": {
    "error_code": 0,
    "session_id": "1"
  }
}

Channel lens mask info:

{
  "type": "notification",
  "params": {
    "event_type": "channel_lens_mask_info",
    "channels": 0,
    "enabled": "off"
  }
}

Channel resolution info:

{
  "type": "notification",
  "params": {
    "event_type": "channel_resolution_info",
    "channels": 0,
    "resolution": "1280*720"
  }
}

We Promise Transparency. Not the "Liquid Glass" Kind.

Tue, 06 Oct 2026 11:10:46 +0000

Let's get the joke out of the way first, because I can already see someone's fingers hovering over the comment box: no, when we talk about "transparency" here, we are not launching a translucent UI skin with rounded corners and a marketing deck about "depth" and "light." Nobody's redesigning your icons to look like they're floating in a glass of water. Apple can keep that one. Transparency, as we mean it, isn't about making things look like glass. It's about making sure there's nothing opaque left to hide behind in the first place.

Transparency Is the Rarest Feature in Modern Software

Here's the uncomfortable observation that started all of this: almost nothing you run today will actually tell you what it's doing. You call a function, and underneath it, a dozen layers you never asked for quietly make decisions on your behalf — a runtime schedules something, an allocator decides where your data actually lives, a framework "helpfully" retries a failed call you didn't know could fail, a standard library wraps three other standard libraries you've never heard of. Most of the time that's a perfectly reasonable trade. Most software genuinely benefits from not making every engineer re-derive memory management from first principles on a Tuesday.

But somewhere along the way, "we abstracted this for your convenience" quietly turned into "we abstracted this so thoroughly that nobody on the team, including us, can tell you with certainty what your program actually does when it touches the kernel." That's not convenience anymore. That's a black box with a changelog. And an entire generation of developers has been trained to treat "it works, don't ask why" as an acceptable final answer, because asking why usually means descending through four more layers of someone else's unexamined assumptions.

Developer transparency, the version we actually care about, is the opposite stance, stated plainly: no hidden magic, no invisible runtime quietly squatting on your CPU cycles like a tenant you never agreed to lease to, and no stack of "convenience" layers standing between your code and the hardware that's supposed to be running it. You should be able to point at any behaviour your program exhibits and trace it to a decision — yours, specifically — not to an abstraction that made the call for you somewhere you'll never see.

This is where Gloria comes in, because it's the sharpest, most literal expression of that idea we've built so far.

What Transparency Actually Looks Like in Gloria

When you write something in Gloria, you know exactly what instruction goes to the CPU, exactly what memory gets mapped, and exactly which register is holding your data at any given moment. You are not down on one knee, hat in hand, begging a heavyweight framework or a sprawling C standard library to please do the thing for you. You own the execution, start to finish, with nobody else's fingerprints on it.

From a Linux perspective, Gloria treats the kernel as its one and only direct interface. No linking against glibc. No standard C startup routine quietly initializing half a universe of state before your main function has even had the decency to run. Gloria generates self-contained, raw ELF64 binaries for AMD64, and that's genuinely all that's in there.

When your code needs to open a socket, allocate memory, or spin up a bit of lightweight concurrency, it doesn't go politely knocking on some third-party wrapper function's door and waiting to be let in. It loads the syscall number straight into RAX, lays out the arguments exactly per the System V AMD64 ABI, and hits the kernel directly with a syscall instruction. No chaperone. No translation layer quietly editorializing your request on the way through. The resulting binaries stay under 9KB, because there is precisely zero runtime bloat riding along for the trip. Put that next to the multi-megabyte "hello world" some modern frameworks ship by default, and you start to get an honest sense of exactly how much weight we've actually cut.

Why This Needed to Exist at All

Here's the pitch, cynicism included at no extra charge: modern systems engineering has gone soft. Not incompetent — soft. Comfortable. Padded on every side with abstractions that exist specifically to protect you from having to understand what your own program is doing. Most software doesn't need to care which register holds a value. This project exists for the software that does, and for the engineers who'd rather know than be protected from knowing.

We wanted something that gives you the raw control and speed of C, the type expressiveness and memory-safety thinking of Rust, and the dead-simple concurrency model of Go — minus the overhead each of those languages still quietly carries around. This is built for the specific kind of developer who needs to wring every last drop of performance out of a box, who's writing ultra-fast microservices, custom network infrastructure, or logic that has to survive on genuinely minimal resources, where every kilobyte and every CPU cycle is a line item somebody's actually tracking, not an abstraction nobody bothers to measure.

The Roadmap, Stated Without the Usual Hedging

Our self-hosting plan is straightforward and, I'll admit, a little uncompromising. Right now, the ForgeZero compiler, fz, is written in Go. That's a deliberate choice, not a compromise we're quietly hoping nobody notices — Go let us bootstrap the language fast, iterate on syntax without a chicken-and-egg problem slowing every decision down, and shake the remaining edge cases out of the lexer, parser, and semantic checker while the ground was still allowed to move under our feet.

But Go is a bridge, not a destination. The actual milestone the entire Gloria ecosystem is being built toward is full self-hosting. We're building out the language constructs, inline assembly support, and memory primitives specifically so that Gloria can eventually compile its own compiler, from scratch, with nothing borrowed. The day fz is rewritten in pure Gloria is the day we cut the last external dependency loose entirely — and the day this architecture stops being "an interesting idea with training wheels" and starts being something you can point at and say, flatly, that it's ready for serious production.

That's the whole philosophy, really, scaled up from one register to an entire compiler: nothing in this stack should exist that we can't fully explain, and nothing should run that isn't accountable to someone who actually understands it.

Watch It Happen Instead of Reading About It Later

If any of this is the kind of problem that genuinely interests you — not the polished retrospective six months from now, but the actual process, bugs included — we post it all in close to real time on our Telegram channel, @forgezeroteam. Build updates, broken things, fixed things, syntax decisions argued out loud before they're final. It's faster and considerably more honest than waiting for the cleaned-up version to show up here.

No liquid glass. No rounded corners pretending to be depth. Just a binary you can actually read, a register you can actually name, and a compiler that, one day soon, won't owe its existence to anything but itself.

The Goldfish Brain: How AI fakes a memory it doesn't have, and why the illusion is so expensive

Mon, 05 Oct 2026 19:59:56 +0000

EP:5 The Curiosity Crunch

You've been chatting with an AI for an hour. You brainstormed a product name, told it an embarrassing story from college, and cracked a joke about your landlord. Twenty minutes later, it calls back to that joke at exactly the right moment.

It feels personal. It feels like someone is listening, keeping notes, and getting to know you.

Nobody is. There's no one in there who remembers you.

Every time you press Send, the AI starts from zero. It has no idea who you are, what you said a minute ago, or that a conversation is happening at all. It's the most convincing magic trick in modern tech, and by the end of this episode you'll know how it works.

The Crunch in one sentence: an AI doesn't remember your conversation. Your chat app re-reads the whole thing to it, from the top, every single time.

1. The 50 First Dates problem

Models like ChatGPT, Claude and Gemini are stateless. A model is a giant file of numbers (its "weights") that was frozen when training ended. When you send a message, the model reads your text, produces a reply, and stops. Nothing about your conversation is written back into it. It never learns from the chat in real time, and it doesn't keep a file with your name on it.

If you've seen 50 First Dates, you know the setup. Lucy wakes up every morning with her memory wiped. Henry gets around this by showing her a video that catches her up on everything she's missed.

Your chat app is Henry, and it plays that video every time you speak.

When you type a new message, the app doesn't send only that message. It grabs:

the hidden instructions the company gave the model ("you are a helpful assistant…"),
everything you've said so far,
everything the AI has said so far,
your new message,

and glues them into one long document. That whole document goes to the model. The model reads it in a blink, writes the next reply, and forgets everything again.

Here's roughly what your chat app sends on your third message:

[system]    You are a helpful assistant.
[user]      Help me name my coffee startup.
[assistant] How about "Bean There"?
[user]      Lol, my landlord says "Bean There, Done That" is his catchphrase.
[assistant] Ha! Maybe that's a sign. Want a few variations?
[user]      Can you make it funnier?        <-- the only part that's actually "new"

The model sees all six lines as if for the first time. When it replies to "Can you make it funnier?", it can mention the landlord's catchphrase because the catchphrase is sitting right there in the text, not because it remembered anything.

Try it yourself. Use any AI's API (or a playground) and send only "Can you make it funnier?" with no history. You'll get a confused answer. The model is the same. The only thing missing is the transcript.

2. The conveyor belt (a.k.a. the context window)

That transcript can't be infinite. The maximum amount of text a model can read at once is called its context window.

Context windows are measured in tokens, which are chunks of text, roughly three-quarters of a word each. Early chatbots could handle a few thousand tokens, about a few pages. Modern models handle hundreds of thousands of tokens, and a few reach a million or more, which is about the length of several novels.

That sounds like plenty, but think of the context window as a conveyor belt with a fixed length. Every message you send and every reply you receive goes onto the belt. When the belt is full, something has to fall off the end. Depending on the app, one of three things happens:

What the app does	What you experience
Truncates: drops the oldest messages	The AI forgets the rule you gave it at the very start
Summarizes: compresses old messages into a short recap	The AI remembers the gist but loses exact wording and details
Refuses: tells you the conversation is too long	"Please start a new chat"

This is why, deep into a long chat, an AI suddenly breaks a rule you set in message one ("always answer in bullet points"). It didn't get lazy or stop listening. That message fell off the belt. If it isn't in the transcript, then as far as the model is concerned, it never existed.

3. "But ChatGPT does remember my name"

Fair objection. Some products now have a memory feature. They really do seem to know your job, your dog's name and your coding style across completely separate chats.

It's the same trick with better stage management.

The app saves short notes about you in an ordinary database, something like "User is a backend engineer. Has a dog named Biscuit. Prefers concise answers." The next time you start a chat, the app quietly pastes those notes onto the top of the transcript before sending it to the model.

The model still remembers nothing. It's just handed a cheat sheet, and the cheat sheet lives in the app, not in the AI.

That's the real distinction worth knowing:

The model is stateless. Same input in, same kind of output out.
The app around the model holds all the memory: chat history, saved notes, retrieved documents.

When people say "AI memory," they almost always mean the second thing.

4. The detective's corkboard (self-attention)

Now the puzzle. If the model has to re-read the entire transcript on every message, how does it reply so quickly, and how does it know that "it" in "make it funnier" refers to the landlord joke and not the startup name?

The answer is the breakthrough behind modern AI, self-attention, introduced in the 2017 paper "Attention Is All You Need."

Picture a detective in front of a corkboard covered in photos. Using red string, the detective connects pieces of evidence that relate to each other: this suspect to that location, that receipt to this timestamp.

Attention does the same thing with words. For every token in the transcript, the model asks, "Which other tokens matter most for understanding this one?" and draws a weighted red string to each of them. A thick string means "very relevant," and a hair-thin string means "mostly ignore."

For "Can you make it funnier?", the thickest string from "it" runs to the joke you told earlier. That's how the model works out what "it" means.

Here's the part that surprises people. Models don't read left to right the way you do. When they take in your transcript, they process all the words in parallel, which is exactly what GPUs are built for. That's a big reason the first part of the response (reading your whole history) is so fast.

(One honest footnote: when the model writes its reply, it does go one token at a time, left to right. That's why you see the answer stream onto the screen word by word. Reading is parallel and writing is sequential.)

5. The staggering cost of a PDF

Those red strings come at a price.

If every token connects to every other token, then a transcript with N tokens needs about N × N connections. That's quadratic scaling.

Transcript length	Red strings to compute
1,000 tokens	~1 million
2,000 tokens	~4 million (2× the text, 4× the work)
10,000 tokens	~100 million (10× the text, 100× the work)
100,000 tokens	~10 billion

Doubling the document quadruples the work. This is why dropping a 100-page PDF into a chat is such a heavy lift, and why long-context models are expensive to run.

Engineers are fighting back. Real systems don't pay the full bill every time. Here's what they use:

KV caching. The model saves the intermediate math for the part of the transcript it already processed. When you send message #31, it only needs to work on what's new, not redo messages #1 to #30. (Providers now even sell "prompt caching" as a discount feature.)
FlashAttention and similar kernels. Smarter GPU code that does the same math using far less memory and time.
Sparse and sliding-window attention. Some models don't draw a string between every pair of tokens, only the ones that probably matter.

So the honest picture is more nuanced than "AI recomputes everything from scratch." Conceptually it re-reads everything, and in practice clever engineering skips the repeated work. But the underlying cost of long context is real, and it's why providers charge more for longer inputs and why "just give it a million tokens" isn't free.

6. So what? Practical takeaways

Understanding the trick makes you better at using the tool:

Long chats get worse, not better. As the belt fills with old, irrelevant text, quality drops and rules get lost. Start a new chat for a new topic.
Repeat what matters. If a rule is critical, restate it in your latest message. Don't trust that it survived 60 messages.
Front-load a summary. Before a long session, paste in a short "here's who I am and what we're doing" block. You're doing manually what memory features do automatically.
Don't treat "forgot" as "ignored you." Usually the instruction was truncated or diluted, not defied.
Be careful what you paste. Everything in the chat is re-sent to the model on every turn. That includes anything private you pasted in at the start.

The Crunch (key takeaways)

No memory in the model. The AI is frozen and stateless. It lives in a permanent present tense.
The transcript trick. Your chat app re-sends the entire conversation, plus hidden instructions, with every message.
The conveyor belt. The context window is a fixed-size box. When it fills, old messages are dropped or summarized.
"Memory" features are sticky notes. The app stores notes about you and pastes them into the prompt. The AI never learns you.
Attention connects the dots. Red strings between tokens let the model understand "it" and "that joke," and they're why the cost grows quadratically with length.
The illusion is engineered. KV caching and smarter attention make it affordable, but it's still a lot of computation.

The next time an AI nails a detail you mentioned an hour ago, don't be flattered. It didn't remember you.

It just read the whole script again, faster than you can blink.

Until next time, keep crunching!!!

Enjoyed this? Forward it to the friend who says "my AI knows me." And tell us what to crunch next: reply with the topic you're most curious about.

Entitlements Are a Domain, Not a Feature Flag

Mon, 05 Oct 2026 18:58:45 +0000

It usually starts with one feature. The product team wants the new reporting dashboard available only on the enterprise plan, and there's no plan model in the code yet. The flag service is already wired in and supports targeting, and a segment called enterprise-customers takes ten minutes to set up. A year later there are a dozen such segments. Every grant now goes through engineering, the code understands plans only as segment names, billing and access drift apart, and change logs are not sufficient to audit the proper state of customer product access. Most people involved feel the strain, but there's no clear way back.

I think the way back starts with seeing that this was never a flag decision. Release toggles, experiments, and kill switches are decisions about how code behaves, made by the people who build and run it. Deciding which customers are owed a feature is a commercial decision, and it belongs to a business domain with its own owner, policies, and audits. Beyond the server-side and operational issues, a solution also has to account for client-side apps, which can quickly accelerate the pain and complicate a solution.

Feature Flags Decide How Code Behaves

Martin Fowler's 2010 entry on feature toggles presents them as a way to keep unfinished work deployed but switched off on the main branch when a feature takes longer than a release cycle. Unleash, an open-source feature flag platform, still describes a release flag as a way to "manage the release of new or incomplete features," with an expected lifetime of 40 days.

Experiments and kill switches use the same switch to decide which version should ship or whether code should run right now.

When the team that builds and runs the product decides how the code should behave, it's a feature flag. That holds even when a product manager runs the experiment, because no customer is owed the variant they landed in. By the same test, a beta promised to named design partners is owed for as long as the promise runs, so it's an entitlement. When sales, finance, or a contract decides who gets a feature, it's an entitlement that happens to use a flag. A flag service can still carry that answer, but then it runs one department's decisions through a tool built for another's.

The Standard References Treat Entitlement as a Flag Type

Pete Hodgson's catalog of toggle types, published on Fowler's site, sets permissioning toggles beside release, experiment, and ops toggles. Permissioning toggles change "the features or product experience that certain users receive." Hodgson notes these "may be very-long lived compared to other categories of Feature Toggles - at the scale of multiple years."

The vendors built the category in. Unleash ships a permanent permission flag type to "control feature access based on user roles or entitlements." LaunchDarkly's entitlements guide treats entitlement flags as permanent ones, which "are part of the everyday operation of the application." A 2020 post on LaunchDarkly's blog by Dawn Parzych goes further: "Instead of creating a custom build for a customer requesting a specific feature, wrap a flag around the feature and release it to that particular customer."

A 2020 interview study by Meinicke, Wong, Vasilescu, and Kästner shows teams drifting there without deciding to. Practitioners "usually do not distinguish between different kinds of flags," and "a feature flag that initially guards a new feature may become a configuration option that should be only available for 'premium' users."

What a Flag Service Can't Do When It Holds the Entitlement

Each of a flag service's design choices is likely sensible for a release decision and wrong for a commercial one.

No Single Fallback Is Right for Every Customer

Flag clients are built to never break the application. OpenFeature, a vendor-neutral standard for flag evaluation APIs, requires in its specification that client methods "MUST NOT throw exceptions" and that evaluation calls "must always return the default value in the event of abnormal execution." For a release flag, that's right, because falling back to off means the old behavior everyone has been running.

For an entitlement there's no safe default. LaunchDarkly's entitlements guide warns that if the application can't connect, "all of your end users will receive a single fallback variation." Default to off, and every enterprise customer loses what they bought. Default to on, and every free customer gets it.

An entitlement service can fail too. A client that starts with nothing cached falls back to one default, no matter which system holds the grant. The difference is who decides how long a cached grant still counts. LaunchDarkly's offline-mode documentation says a client SDK that can't reach the vendor uses "the latest stored flag variation values." Unless every app build adds its own expiry check, a cancelled customer's device keeps any paid feature that works offline for as long as it stays offline.

An entitlement system makes that an explicit window. RevenueCat, which manages in-app subscriptions, fixes one for every app, so an entitlement active when a device went offline "will remain active for up to three days." A team that owns its entitlements lets the plan owner set it.

A Change Log Isn't an Audit Trail

When finance, support, or legal asks why a tenant has a feature, they want the agreement that granted it, when it started, and when it ends. A flag service records that someone added a tenant ID to a segment on a given day. Giving sales a role in the flag tool changes who edits, not what's recorded.

Approvals, a comment, and a removal date can sit on that change, but the contract, trial, or sales exception behind it lives in a ticket or a Slack thread, if anywhere. Even a required contract ID is free text the flag service can't validate or query. Asking which tenants had a feature last quarter, and under which agreements, means replaying segment edits. A segment's history is enough for a rollout but not for a decision with money attached.

Billing and Access Drift Apart

Once the flag service decides access, billing knows what customers bought and the flag service knows what they can use, and usually nothing reconciles the two. A downgrade in billing doesn't remove the tenant from the segment, and a sales exception granted in the segment never reaches billing. Schematic is an entitlement platform vendor with an interest in the point. It describes the common path: "many teams use their feature flag tool for entitlements early on, and then realize the two concerns need to be separated as their pricing gets more complex."

LaunchDarkly's guide offers segments that stay in sync with an external system, so billing could feed one, but only if every grant, sales exceptions included, goes into billing first. A separate domain still needs that discipline, but it can enforce it. Billing sends plan changes as events, and the entitlement store refuses any grant with no agreement behind it. Exceptions and trials start in the entitlement store and flow out to billing. A hand-edited segment can't refuse anything.

Entitlement Is a Domain With Its Own Owner

Product access encodes the company's pricing, contracts, and obligations to each customer. It has an owner outside engineering, usually product and sales, policies about who may grant exceptions, and audit requirements because what a customer can use is part of what they're billed for.

It also has its own model of plans that bundle features, contracts that override them with their own dates, expiring trials, and seat limits, none of which a segment expresses. That's the profile of a bounded context, not a flag type. Modeling it as its own domain, with grants that are dated and never overwritten, closes each gap above. A service answers "may this tenant use this feature?" from data it owns. Billing can host that domain if its catalog maps plans to features and also records exceptions and trials with their dates.

Concern	Entitlement held in a flag service	Entitlement held in its own domain
Who grants access	Whoever edits segments, checked against nothing	Product and sales grant it, under their own exception policy
What a grant records	Who changed a segment, and when	The agreement behind it, with start and end dates
What a renewal touches	Removal dates edited per segment, linked to no contract	The contract's dates, once, and every plan feature follows
How billing stays in step	Two systems hold one fact, and neither wins	Billing sends plan changes in, and exceptions flow out to it
What happens in an outage	On clients, the last stored value with no expiry, or one coded fallback	A fallback and offline window the plan owner sets

One Contract Can Justify a Temporary Flag

The hardest case is the one Parzych's advice describes. A single customer's contract requires a feature nobody else has asked for. For now, everything about it is in doubt, including its specification, its stability, whether it meets the customer's needs, and whether it will ever sell. The contract item itself isn't negotiable.

Who is owed the feature is settled. Whether the feature survives is the open question, and it's partly about the code and partly about the market.

Engineering's usual reason to keep unproven code behind a flag is the ability to turn it off. For this tenant, turning it off breaches a contract, so the off switch is a commercial decision and not an operational one. The fallback problem returns in its starkest form, since no single default can be on for the one tenant who holds the contract and off for everyone else. Where an entitlement domain already exists, the grant goes there and the flag only isolates the code. Where none exists, the flag carries both decisions.

Every Exit Removes the Flag

Neither the off switch nor the fallback problem goes away, but both can be accepted on purpose. Making an unproven feature that one contract requires into a plan feature would be premature, and the flag keeps the code isolated until the open question is answered. I think that makes this case a legitimate use of a flag, declared as intentional technical debt and owned by product and sales rather than engineering. The off switch and the outage fallback are now their risks to carry.

That ownership means only product or sales may change the grant, and its exit trigger is recorded with the contract. If engineering needs an incident kill switch, it's a separate flag whose use product signed off on in advance.

What separates it from Hodgson's multi-year permissioning toggles is that it has a decision point, due no later than the first renewal, and every way out of that decision removes the flag:

Another customer wants it. The grant moves into the entitlement model as a plan feature, and the flag goes away.
It becomes standard. The code path becomes permanent and the flag is deleted.
The contract ends or the feature fails. The code is removed with the flag.

Parzych's advice is right on the first day. It becomes a problem only when nobody records the exits and the flag turns into a permanent grant with no owner.

On Client Apps, Vendor Coupling Compounds the Entitlement Problem

Mobile, desktop, and browser apps need the same answer as servers, sometimes offline, and flags usually reach them through the vendor's SDK.

Clients Still Need Flags, but Never Decide Entitlements

Client apps arguably need flags more than servers do, since a client fix waits on app store review and on users who don't update. But the client is never the authority on an entitlement. It can hold a grant the server issued for as long as the plan's rules trust it, but the server enforces, and the client only decides what to show.

The Vendor's SDK Ties Every Build to the Vendor

The vendor's SDK, and its representation of the answer, become part of every installed build. Swapping vendors waits on app releases, and until old builds disappear, you can't put your own layer between the client and the vendor. The layer serving the answer isn't yours either, so you can't merge sources, set expiry and invalidation for each kind of data, or push your own change events. I think direct vendor SDK access from client apps is a trap, though many teams accept it for the streaming updates and built-in experiment analysis that come with it.

Entitlements in the flag service make that coupling worse. Every build then depends on the vendor for what customers are owed. Exposure gets worse too. LaunchDarkly's secure mode exists because a malicious end user "could use a context or user key to identify what flag values another end user receives," and it covers only its JavaScript-based SDKs. Its other SDKs have no such protection. If tenant keys are readable slugs like acme, a modified client can probe flags and learn which paid features a named customer holds, even when the server enforces every grant.

A Projection Fixes the Source but Not the Layer

For a team already deep in the hack, a first step back is fixing where entitlements come from while the flag service keeps serving them. I've used this shape before, with a read store that only receives changes from the audited business store and is never written to directly. The flag service becomes a projection of an entitlement store that sales and product own.

 ┌──────────────────────────────────┐
 │ Entitlement store                │ <── sales and product write
 │ audited source of truth          │
 └────────────────┬─────────────────┘
                  │ events (only writer)
                  v
 ┌──────────────────────────────────┐
 │ Flag service                     │
 │ release flags + entitlement      │
 │ segments, rebuilt from source    │
 └────────────────┬─────────────────┘
                  │ vendor SDK
                  v
               Client

That makes LaunchDarkly's synced segment, described earlier, a rule instead of a hope, and two conditions keep the projection honest. Only the sync integration may write to entitlement segments, enforced in the vendor's permissions rather than by agreement, and the whole projection can be rebuilt by replaying the source. It works, but the sync and the rebuild can each break. It also leaves the device getting grants through the vendor's SDK, with every problem above.

Entitlements Can Ride on an Endpoint You Already Own

The cheaper way out takes grants off the vendor's path entirely, and it may not need a new endpoint. The app already asks the server who the user is and which tenant they belong to, and that session or account lookup can return the tenant's grants with the answer:

{
  "user": "u_1842",
  "tenant": "acme",
  "entitlements": {
    "generatedAt": "2026-10-05T14:00:00Z",
    "refreshAfterSeconds": 3600,
    "staleSeconds": 259200,
    "grants": ["advanced-reporting"]
  }
}

One number says how often to refresh, and the other says how long a cached answer still counts when a refresh fails. Contract grants change rarely, so hourly is plenty, and an in-app purchase forces a refresh. The plan owner decides grants stay trusted for three days offline. That window is a promise, so data a user creates under a cached grant has to sync even if the grant ended while the device was offline. What happens to that data next is the plan owner's call too. generatedAt lets the client tell how old its copy is, and nothing in the exchange touches the flag vendor.

A Facade Decouples Flags From the SDK, as a Separate Choice

Release flags still come through the vendor's SDK, and moving them can wait. The usual shape is a thin endpoint you own in front of the flag service. It's the kind of server component Sam Newman describes in his backends-for-frontends pattern. The vendor's contract ends at the server. If one endpoint serves both grants and flags, the response should keep them apart, because one flat map of booleans can't tell a release toggle from a contract grant.

The SDK also records who saw each variation, for experiments. Behind a facade, the app takes over that job. Statsig's experimentation guidance explains that counting users who never reached a feature dilutes an experiment's result. For a change that lives entirely in the interface, only the client knows whether the user reached it.

A facade also gives up the vendor's real-time streaming of flag changes, so a client kill switch waits for the next refresh. A change too urgent for that usually belongs on the server. A client-only one, like disabling a crashing native screen, is a reason to keep release flags on the vendor's SDK.

A Flag Can Stand In When the Gap Is Written Down

None of this means a startup with one paid tier needs an entitlement service on day one. But I think holding product access in a flag is still a hack, and the only good reason for it is a timeline or budget that would otherwise block delivery. Even then it's a cost, because paying it off means refactoring the code and migrating the grants. Someone should decide it on purpose and write down three things: that the flag is standing in for an entitlement domain, who owns that gap, and what will trigger paying it off. A good trigger is the first grant no plan explains, such as a sales exception, after which segments hold facts billing doesn't.

Unstated, the gap compounds. Each segment and sales exception adds to a later migration. Each client build that reads entitlements from the vendor's SDK adds a contract you can't recall. And the flag service keeps gaining authority because nobody recorded that it was never supposed to have any.

Hacking feature flags can work because each wrong answer is usually recoverable on its own. A tenant who gets a feature a day late complains and gets a credit. No credit undoes exposing what other customers bought, though. A different system layer that decides who can sign in, move money, or read another tenant's data has no such margin, and the same shortcut is never an option. A hack can hold together for a while, but the leaks will show and the costs will add up.

The Goldfish Brain: How AI fakes a memory it doesn't have, and why the illusion is so expensive

Mon, 05 Oct 2026 15:37:37 +0000

EP:5 The Curiosity Crunch

It feels personal. It feels like someone is listening, keeping notes, and getting to know you.

Nobody is. There's no one in there who remembers you.

The Crunch in one sentence: an AI doesn't remember your conversation. Your chat app re-reads the whole thing to it, from the top, every single time.

1. The 50 First Dates problem

If you've seen 50 First Dates, you know the setup. Lucy wakes up every morning with her memory wiped. Henry gets around this by showing her a video that catches her up on everything she's missed.

Your chat app is Henry, and it plays that video every time you speak.

When you type a new message, the app doesn't send only that message. It grabs:

the hidden instructions the company gave the model ("you are a helpful assistant…"),
everything you've said so far,
everything the AI has said so far,
your new message,

and glues them into one long document. That whole document goes to the model. The model reads it in a blink, writes the next reply, and forgets everything again.

Here's roughly what your chat app sends on your third message:

[system]    You are a helpful assistant.
[user]      Help me name my coffee startup.
[assistant] How about "Bean There"?
[user]      Lol, my landlord says "Bean There, Done That" is his catchphrase.
[assistant] Ha! Maybe that's a sign. Want a few variations?
[user]      Can you make it funnier?        <-- the only part that's actually "new"

Try it yourself. Use any AI's API (or a playground) and send only "Can you make it funnier?" with no history. You'll get a confused answer. The model is the same. The only thing missing is the transcript.

2. The conveyor belt (a.k.a. the context window)

That transcript can't be infinite. The maximum amount of text a model can read at once is called its context window.

What the app does	What you experience
Truncates: drops the oldest messages	The AI forgets the rule you gave it at the very start
Summarizes: compresses old messages into a short recap	The AI remembers the gist but loses exact wording and details
Refuses: tells you the conversation is too long	"Please start a new chat"

3. "But ChatGPT does remember my name"

Fair objection. Some products now have a memory feature. They really do seem to know your job, your dog's name and your coding style across completely separate chats.

It's the same trick with better stage management.

The model still remembers nothing. It's just handed a cheat sheet, and the cheat sheet lives in the app, not in the AI.

That's the real distinction worth knowing:

The model is stateless. Same input in, same kind of output out.
The app around the model holds all the memory: chat history, saved notes, retrieved documents.

When people say "AI memory," they almost always mean the second thing.

4. The detective's corkboard (self-attention)

The answer is the breakthrough behind modern AI, self-attention, introduced in the 2017 paper "Attention Is All You Need."

For "Can you make it funnier?", the thickest string from "it" runs to the joke you told earlier. That's how the model works out what "it" means.

5. The staggering cost of a PDF

Those red strings come at a price.

If every token connects to every other token, then a transcript with N tokens needs about N × N connections. That's quadratic scaling.

Transcript length	Red strings to compute
1,000 tokens	~1 million
2,000 tokens	~4 million (2× the text, 4× the work)
10,000 tokens	~100 million (10× the text, 100× the work)
100,000 tokens	~10 billion

Doubling the document quadruples the work. This is why dropping a 100-page PDF into a chat is such a heavy lift, and why long-context models are expensive to run.

Engineers are fighting back. Real systems don't pay the full bill every time. Here's what they use:

KV caching. The model saves the intermediate math for the part of the transcript it already processed. When you send message #31, it only needs to work on what's new, not redo messages #1 to #30. (Providers now even sell "prompt caching" as a discount feature.)
FlashAttention and similar kernels. Smarter GPU code that does the same math using far less memory and time.
Sparse and sliding-window attention. Some models don't draw a string between every pair of tokens, only the ones that probably matter.

6. So what? Practical takeaways

Understanding the trick makes you better at using the tool:

Long chats get worse, not better. As the belt fills with old, irrelevant text, quality drops and rules get lost. Start a new chat for a new topic.
Repeat what matters. If a rule is critical, restate it in your latest message. Don't trust that it survived 60 messages.
Front-load a summary. Before a long session, paste in a short "here's who I am and what we're doing" block. You're doing manually what memory features do automatically.
Don't treat "forgot" as "ignored you." Usually the instruction was truncated or diluted, not defied.
Be careful what you paste. Everything in the chat is re-sent to the model on every turn. That includes anything private you pasted in at the start.

The Crunch (key takeaways)

No memory in the model. The AI is frozen and stateless. It lives in a permanent present tense.
The transcript trick. Your chat app re-sends the entire conversation, plus hidden instructions, with every message.
The conveyor belt. The context window is a fixed-size box. When it fills, old messages are dropped or summarized.
"Memory" features are sticky notes. The app stores notes about you and pastes them into the prompt. The AI never learns you.
Attention connects the dots. Red strings between tokens let the model understand "it" and "that joke," and they're why the cost grows quadratically with length.
The illusion is engineered. KV caching and smarter attention make it affordable, but it's still a lot of computation.

The next time an AI nails a detail you mentioned an hour ago, don't be flattered. It didn't remember you.

It just read the whole script again, faster than you can blink.

Until next time, keep crunching!!!

Enjoyed this? Forward it to the friend who says "my AI knows me." And tell us what to crunch next: reply with the topic you're most curious about.

Hi, I'm the Guy Who Got Angry at Garbage Collectors and Built an Entire Toolchain About It

Mon, 05 Oct 2026 15:29:39 +0000

Hey, everyone. Alex Voste here — the person behind ForgeZero and its resident experimental language, Gloria. I've written plenty about the engineering internals on this blog already, but I realized I never actually introduced the why, or told you where to go if you want to watch this thing get built in real time instead of reading about it after the fact. Let's fix both.

The Origin Story, Condensed

I wanted total control. Over the code, over the hardware, over every byte between my logic and the CPU actually executing it. What I kept running into instead was an industry quietly content with heavy runtimes, abstraction stacked on abstraction, and garbage collectors that show up uninvited in the middle of your hot path like a relative who "was just in the neighbourhood." None of that is evil. Most of it is genuinely useful for most people. It is also, for the specific thing I wanted to build, a tax I was no longer willing to pay.

So I built ForgeZero — an orchestration toolchain for assembly, C, C++, Objective-C, and Gloria, designed to get projects built without Makefiles, without a pile of configuration nobody fully understands, and without the "trust me" energy most build systems ask of you. It's fast, it's memory-conscious, and critically, it never asks you to take anything on faith — every cache decision, every toolchain choice, every flag is something you can actually see, not something an abstraction layer quietly decided on your behalf. GitHub's right here: github.com/forgezero-cli.

Then Gloria Happened, Because One Obsession Is Never Enough

Once ForgeZero existed, I wanted a language to go with it — something low-level and genuinely powerful, but without the syntax feeling like a punishment for wanting control. You've probably already seen a few of the pieces I'm proudest of: chained file operations with >>, error propagation with |? that doesn't force you into a pyramid of nested checks, and repeat loops that read the way you'd actually describe the loop out loud instead of the way a 1970s compiler wanted you to describe it.

There is no garbage collector in Gloria. There are no comfortable abstractions sitting between you and the machine. Gloria compiles straight to raw x86_64 binary and talks to the operating system directly through syscalls, because every layer between your intent and the instruction stream is a layer that can lie to you about what's actually happening. I'd rather know exactly what's happening, even when what's happening is occasionally my own fault.

Yes, I Use AI. No, It Doesn't Get to Make the Decisions.

I don't build this alone — I keep a neural net around as a constant sparring partner. It's genuinely useful for kicking around new syntax ideas, helping refactor, stress-testing a concept before I commit to it, and yes, occasionally helping me phrase a post like this one without sounding like I've been awake for thirty hours (I frequently have been).

What it doesn't get is unsupervised trust. If you've read my other posts here, you already know my relationship with AI-generated code is closer to "overconfident intern I review like a hostile code auditor" than "pair programmer I defer to." It's a tool that accelerates thinking. It is not the thing doing the thinking, and it never gets anywhere near the parts of this project where a mistake means a corrupted binary instead of an awkward sentence.

Where Gloria Actually Stands Right Now

We've got solid, working filesystem interaction — reading and writing data confidently, not as a toy demo. Right now the active front line is dynamic arrays (vectors), which isn't a feature for its own sake — it's the foundation everything downstream actually needs, the language and the compiler both.

Where This Is Going

Networking is next — full socket support, which cracks the door open to high-performance servers, network utilities, and game engines written in Gloria, with nothing hiding between your code and the actual packets going over the wire.

And the actual north star, the thing I'm building all of this toward: self-hosting. The day ForgeZero's compiler for Gloria is itself written in Gloria is the day this entire project proves its own premise. Until a language can bootstrap itself, it's still borrowing credibility from whatever built it. I want Gloria standing entirely on its own.

Come Build It With Us

If any of this sounds like your kind of problem — if you genuinely enjoy getting into the hardware, obsessing over a single instruction more than is strictly healthy, and want to watch something real, low-level, and unapologetically system-oriented get built in public, not just read the polished retrospective six months later — this is your invitation.

We post live updates, real bugs, actual wins, and everything happening behind the curtain on our Telegram channel: t.me/forgezeroteam

No garbage collectors were harmed in the making of this project. Several were, however, deeply and personally offended.

Why Good Agent Evaluation Wasn’t Enough for Production

Mon, 05 Oct 2026 10:28:24 +0000

"They used it once and refused to touch it again."

When that feedback landed two weeks in, I could barely believe it. An agent validated against a benchmark built on real-world data had still failed to solve the foundation’s workflow problem.

Only in hindsight did I realize: I had evaluated the agent's processing capability, but I never evaluated whether volunteers could actually use it to get their job done.

1. Every House Has a Kitchen—Why Build a Community Canteen?

A friend working at a non-profit foundation once shared a story with me:

A rural village had poured resources into numerous senior welfare initiatives over the years, yet life satisfaction barely moved. What ultimately made a difference was building a communal canteen.

When the village head first heard the project proposal, he was deeply skeptical: Every single elderly household has its own kitchen. Why on earth do we need a village canteen?

Field visits later validated two straightforward reasons. First, cooking had simply become too physically taxing for the seniors—especially firing up a stove in the sweltering heat, which was pure misery. Second, beyond serving hot meals, the canteen at the village entrance created a regular venue for group interaction. It gave them a place to run into people and chat, directly addressing their need for social connection.

This counterintuitive, genuine need was surfaced by foundation staff and volunteers through round after round of authentic home-visit conversations with local seniors.

Their organization runs diverse programs nationwide. These projects rely on cohorts of short-term volunteers who pay regular visits to seniors across different age groups, speaking varied regional dialects. Those conversational touchpoints hold the richest, yet most frequently overlooked, context regarding the seniors' current living conditions and real needs.

While my friend had the foresight to archive these interview audio recordings, data collection still leaned entirely on volunteers manually filling out questionnaires.

That survey workflow was time-consuming and labor-intensive, making standardization difficult. More crucially, longitudinal insights—such as tracking changes in a senior's condition over time—rarely materialized.

Meanwhile, a lean foundation team couldn't realistically audit hours of raw audio by hand.

They needed a way to truly listen to these seniors at scale.

2. From an Audio File to an Auditable Interview Record

I realized the foundation didn’t need raw audio files; they needed reliable, verifiable interview records. That breakdown came down to three concrete requirements.

First: Accurate content transcription. Across these interviews, seniors speaking regional Chinese dialects was the baseline reality. This required a rigorous vendor evaluation for speech-to-text (STT).

I requested a small batch of authentic interview datasets. These were dialect-heavy recordings—each running over 30 minutes, spanning dozens of back-and-forth turns, with standard Mandarin and local dialects constantly interspersed. I benchmarked several speech recognition vendors that supported dialects against these files.

Evaluation was constrained to spot-checks by team members fluent in the respective dialects, paired with holistic context sanity checks. Through this filtering, I locked down our technical vendor and measured average transcription costs on real-world data.

Second: Accurate speaker identification. Listening closely to the tapes revealed messy, real-world conversational dynamics, typically involving multi-party conversations.

Volunteers often conducted visits in small teams. Meanwhile, the senior being interviewed was frequently surrounded by family members or neighbors chiming in. For instance, if an extraction pipeline relied purely on raw transcripts to log the senior's health updates, it could easily misattribute statements made by the spouse.

However, STT vendors supporting broad dialect coverage rarely offered out-of-the-box speaker identification. In our processing pipeline, we therefore had to run speaker identification on the raw audio of each segmented utterance, injecting speaker identity as a critical context layer.

Third: Grounded rationale extraction. The survey questions had to be answered strictly based on the interview dialogue, and many items had direct evidentiary clues scattered across the transcript.

My primary concern here was hallucination in the underlying LLM—specifically, downstream evaluative conclusions generated without factual backing.

I addressed this with a two-layer design: 1) Structurally, the workflow retained a human-in-the-loop review step for volunteers to verify answers. 2) Technically, in addition to outputting the structured form fields, the agent was forced to simultaneously extract the verbatim dialogue snippets serving as evidence.

This second design unlocked a standardized benchmark to score relevance between the model’s outputs and the source transcript, measuring precision and recall on evidence retrieval. While this doesn't offer an absolute algorithmic guarantee against hallucinations, it effectively benchmarked different model providers. It also enabled runtime quality checks on citation integrity in production, providing a reliable proxy for dialogue session health.

Anchored on these three core principles, I set out to hand-label the small sample of real interview data to stand up our benchmark.

Dialect testing was limited to spot-checking rather than turn-by-turn ground-truth matching, so the benchmark baseline assumed clean, pre-segmented utterances. I built a lightweight internal web tool to load this context, play back audio per utterance, and let me label speaker identities.

For the post-interview survey extraction, I created ground-truth annotations of verbatim evidence and corresponding answers across the sample cases. The evidence retrieval was cleanly testable with deterministic metrics, while answer accuracy was evaluated using an LLM-as-a-judge approach as a directional reference.

With this benchmark running, I vibe-coded an end-to-end web prototype and iterated through multiple testing loops. This involved refactoring the data processing pipeline, redesigning the web interface flow, and selecting key vendors—particularly balancing latency, accuracy, and operational cost.

Ultimately, the prototype reached a very usable level on the benchmark. The dataset was small, but I had reasonable confidence in the pipeline's output quality.

3. The Benchmark Looked Great. Why Did Volunteers Refuse to Use It?

The benchmark numbers gave me strong confidence in this build. I recorded a demo video for my friend, who was also impressed with the output. Within two days, he assigned a remote intern and the volunteer operations lead to pilot the tool in an upcoming regional field visit.

For the next few weeks, I repeatedly checked our dashboard, watching trial volunteers sign up and trigger runs. The counters showed tasks completing normally, and I convinced myself that the tool had solved the foundation's interview processing problem.

Two weeks later, my friend came back with the field report: the end-user experience was terrible. After testing it once, the frontline volunteers refused to touch it again.

I have to admit, when that feedback first came in, I found it hard to believe. With an evaluation backed by real-world audio showing strong results, the actual production experience should not have been so poor.

Where did it break?

I first synced with the intern, who insisted the product vision was completely spot-on and genuinely meaningful. We brainstormed a backlog of incremental feature optimizations. Then, pulled away by other priorities over the next few months, I let the project stall.

Later, my friend insisted on pushing this tool through to full deployment for ongoing volunteer operations. I joined one of their internal working sessions. That was when I realized the intern was working remotely; like me, he had limited visibility into the day-to-day work of frontline volunteers.

In that meeting, I finally spoke with the volunteer coordinator who had reported the poor experience. Listening to her describe the workflow, I caught the core friction point causing the backlash: for volunteers, the operational overhead had multiplied, while the perceived efficiency gain over their legacy process was negligible.

I asked her to walk me through their baseline workflow. They already had a standardized SOP for regional field teams: volunteers simply completed their onsite interview, uploaded the audio file, and manually filled out the survey form.

Volunteers executed this entire legacy flow on their phones via WeChat, simply tapping a Feishu web form link provided by headquarters.

My "solution," by contrast, required volunteers to open a third-party speech SaaS on desktop, register an account, pre-import questionnaire prompt templates, upload audio files turn-by-turn, and export the transcribed extraction as a CSV file.

It was an embarrassingly obvious failure mode. In retrospect, I struggled to understand how I missed such glaring workflow friction when the initial feedback came in.

It boils down to one reality: I wasn’t on the ground.

4. An Agent Only Works When It Disappears into the Existing Workflow

That feedback was useful because the source of the problem was clear.

A straightforward idea surfaced: How could I wire this benchmarked pipeline directly into the volunteers' existing workflow? Ideally, the integration would be completely invisible to them, dropping adoption friction to near zero.

I focused on Feishu, their existing workspace tool. Volunteers were already relying on its native file upload and survey forms for automated data collection. Digging into the platform capabilities, I found that Feishu supported custom workflows combining audio ingestion and LLM execution. I could rebuild the pipeline natively on top of their existing automation infrastructure.

I stood up the working prototype over a single weekend. It mirrored the exact audio submission entry point volunteers were already accustomed to, handling dialect transcription, speaker parsing, and grounded Q&A extraction entirely downstream.

The turnaround was immediate and stark.

The intern quickly grasped why our earlier disconnect from the operational context had derailed us. After reviewing the prototype architecture, he took over and built out the complete internal production implementation. He also spearheaded rollout validation across pilot regions, mapping the geographic distribution of elderly dialects to define the hard operating boundaries of the system. In regions speaking dialects beyond our coverage limits, the legacy manual workflow was kept intact.

Only then was the project truly in production use.

Whenever I look back at this case, I stay paranoid about one question: What are the actual measurement boundaries of the benchmark in front of me? And what is it failing to measure?

This experience didn't invalidate our benchmark. What it made painfully clear was that the benchmark answered a very narrow question: Given an audio file, can the agent generate an accurate, auditable interview record? What it completely failed to answer was: Can a volunteer, as an end user, complete their end-to-end task inside their existing workflow with less operational friction?

I now view the workflow itself as part of the evaluation, rather than just scoring raw model outputs. How input enters the system, how outputs are reviewed and submitted, and whether human users continue using the tool all help determine whether an agent delivers sustained real-world value.

The API Gateway Shouldn't Know Your Business

Fri, 02 Oct 2026 13:53:46 +0000

I appreciate what an API gateway brings to most systems, at the right time and in the right place. It gives every service a shared first layer. TLS, rate limiting, and access logging happen once at the edge, and bad tokens get turned away before any service spends work on them. The services behind it still check what reaches them, but they start from cleaner traffic. The gateway also lets those services move and split behind a stable address without breaking their clients.

But a gateway's configuration tends to collect things nobody designed it to hold. It might be a check on a customer's plan, a response field renamed back for an older client version, or a call to a second service to decide whether the first one should be reached.

That drift, away from the gateway's purpose and from the code that owns the rule, isn't always carelessness. The gateway is the cheapest place to apply a business domain rule to every external request, so rules flow toward it. The decade-old advice that business logic doesn't belong there hasn't removed that pull. It says where business logic shouldn't go, and little about the forces that send it there or how to keep a rule out when the need is urgent.

A business domain rule belongs in the gateway only if the team that owns the gateway could change it correctly without asking a domain team. Keeping the rest out takes an answer to each force, and an emergency rule that does go in goes in as debt, with a domain owner and a scheduled removal.

The Advice Has Been Clear Since 2014

In March 2014, James Lewis and Martin Fowler described microservices as favoring "smart endpoints and dumb pipes," against Enterprise Service Bus products that "often include sophisticated facilities for ... applying business rules." Thoughtworks put "overambitious API gateways" on Hold in its Technology Radar from November 2015 through May 2018. The April 2016 entry calls the pattern "a worrying re-emergence of this disease," the disease being business smarts pushed into middleware. It says "any domain smarts such as data transformation or rule processing should live in applications or services where they can be controlled by product teams."

Microsoft's Gateway Offloading pattern agrees, saying "Never offload business logic to the gateway."

Why Rules Drift Into the Gateway

Four forces pull a rule toward the gateway, and each makes sense to whoever makes the change that day.

Every External Request Passes Through It

Premium customers can request to export more than 10,000 rows, and everyone else can't. In the services, that rule is a change to the export service, and possibly to each service that exposes a large download. In the gateway, it's one policy that rejects the request before any service sees it. The person making the change may not see it as a pricing rule, because it can look like an urgent fix to keep large exports from degrading the system.

It Ships on a Different Schedule

A gateway policy usually ships as configuration, and a platform team often owns it, so adding the export limit there doesn't wait on the export team's backlog. What the gateway offers is a different queue, not necessarily a faster one. When the export team is busy and the deadline is close, the rule goes wherever it can ship this week.

After that, every change to the export limit waits in the platform team's queue, even when the export team could ship it sooner. The Gateway Offloading pattern says offloading doesn't fit when centralizing concerns "creates a change-management bottleneck" behind a slower gateway release cycle. The schedule that let the rule in once is the schedule it's stuck with.

Vendors Compete on Programmability

A rule can only move into the gateway if the gateway can express it, and gateway products have made sure it can. Azure API Management runs policy expressions written in C#, and its send-request policy calls another service mid-request. Kong's Pre-Function plugin "lets you dynamically run Lua code." Thoughtworks' later radar entries said vendors kept adding such features to differentiate their products.

Suppose the export team is booked for the quarter, so leadership overrides the queue and hands the export limit to the platform team, which runs Azure API Management. The platform team writes a policy that takes four steps on each export request:

Read the customer ID from the caller's validated token.
Ask the entitlements service for that customer's plan with send-request, caching the answer for a few minutes.
Read the requested row count from the rows query parameter, treating a missing value as zero so clients that omit it keep working.
Return a 403 if the plan isn't premium and the count is over 10,000.

Every step uses a standard gateway feature, and the result is business logic. Once the plan lookup exists, the next rule that needs a customer's plan costs one more condition.

Security Wants One Place to Check

Authorization can be the first business rule to land in the gateway, pushed by a security group that wants one place to audit rather than trusting each dev team to get it right. The gateway already checks tokens, so it looks like that place. The drift starts when the gateway moves from checking that a token is valid to deciding what the caller may do. It usually begins with a role check on a route, which grows into decisions about which records the caller may touch.

A Gateway Rule Loses Its Owner, Its Coverage, and Its Tests

Its Meaning and Its Enforcement Get Different Owners

Step 4 of the platform team's export policy encodes two business facts, that a plan called premium exists and that it allows larger exports than every other plan. The entitlements team decides what plans exist and the export team decides what they allow, but the platform team owns the policy. The team that can change the rule's meaning can't see where it's enforced, and the team that enforces it can't judge whether it's still right.

Suppose sales introduces an enterprise plan. The entitlements service starts returning enterprise, and the gateway rule still compares against premium, so enterprise customers, who pay the most, get rejected on large exports. No test fails, because no test in either service ever ran the rule. Hard-coding premium would be a bug in the service too. Here, though, someone must first find the rule, and then the platform team must change a rule whose meaning they didn't write.

The same split makes the rule hard to remove. The platform team can see the rule but can't tell whether any client depends on it, and the domain team could answer that but may not know the rule exists. So rules can accumulate, each one cheaper to leave than to trace, and the gateway drifts toward the middleware Lewis and Fowler described.

The Rule Covers Only What the Gateway Can See

The first gap is the routes that never cross the gateway. Follow one free-plan customer who wants 50,000 rows. Calling the public API, they get the 403. Then they click Export in the product's web app, whose backend calls the export service over the internal network, and the export runs. They set up a nightly scheduled export. The scheduler, acting with its own service identity, publishes an ExportRequested message that the export service consumes, and that export runs too. The same customer asks the same question three ways and gets two different answers.

The Gateway Offloading pattern recommends that backends accept requests only through the gateway, which stops outside clients from bypassing it but doesn't help here. The web app's backend and the scheduler are part of the product. The queued message never becomes an HTTP request. Even a scheduler call routed through the gateway would carry the scheduler's identity, with no customer for the gateway to look up. So the export service needs its own check, and the enterprise-plan change now has to reach two copies of the rule.

The second gap is on the gateway's own route, where it can check only what a request declares, while an export's size is known only after the query runs. The service treats rows as an optional cap, so a request for ?from=2020-01-01&to=2026-01-01 can leave it out. Step 3 treats a missing count as 0, so the request passes and gets every row the date range matches. Requiring rows would close the hole, but that changes the export API's contract, which is the export team's decision.

The Rule Leaves the Domain's Tests

The export team's test suite exercises the export service, not the policy. Unless gateway config runs in the export team's pipeline, the rule has no test that runs when the export service or the plan catalog changes. API Management's documentation adds that policy expressions have "only limited verification" when defined and run at request time, where any exception becomes a runtime error.

Keep the Gateway to Rules Its Own Team Can Change

One question tests whether a rule is about the request or about the business: could the team that owns the gateway change this rule correctly without asking a domain team?

Kind of rule	Example	Where it runs	Who decides its meaning
About the request	Token expiry, request size, per-caller rate limit	Gateway	Gateway team
An adapter for a backend or client that can't change	Renaming fields for a vendor system or an old client version	Gateway, until retired	A named team, with a test against the backend
Composition for one client	One response for a mobile screen	Backend for frontend	That client's team
A stopgap during an incident	Blocking large free-plan exports until a fix ships	Gateway, until the fix ships	Domain team, with a removal date
A business rule	Export limits by plan	Owning service	Domain team

Rules about the request pass because their correct values don't change when the business changes its mind. A rule that reads a business concept, such as a plan, account status, or price, fails, however little code it takes.

Two better-built versions of the export rule still fail. In the first, the export team keeps its own gateway config. The split owners and the platform team's queue go away, but the coverage gaps remain.

In the second, the policy asks entitlements for the customer's limit as a number, so plan names leave the gateway. But which routes count as exports is still the export team's decision. A new bulk download endpoint would slip past the gateway rule, while a check in the export code covers it on release. Tagging export routes in the export team's spec closes that gap, but the gateway copy still misses the internal paths.

Authentication Passes, Authorization Doesn't

Tokens sit on both sides of the ownership test. Rejecting a token that is malformed, badly signed, expired, or tied to a revoked session passes, because what makes a token valid doesn't change when the business does.

Any decision about what the caller may do fails the test, even at the route level. A scope per route looks like a property of the token, but which scope POST /exports requires is the export team's call. If they split exports:write into a standard and a bulk scope, the gateway enforces the old contract until the platform team catches up. The exception is a scope check generated from the service's API spec. It removes the lag and can stay as an early reject, because the export team still decides its meaning and the service runs its own check.

The Gateway Offloading pattern lists authorization among the concerns a gateway can centralize, so this line is narrower than Microsoft's. The gateway sees /accounts/4417/exports and a user ID, but only the service that loads account 4417 knows who belongs to it. Broken object level authorization tops the 2023 OWASP API Security Top 10, which describes the check as one "usually implemented at the code level."

Capacity Limits Pass, Plan Quotas Don't

Rate limits sit on both sides too. A per-caller limit that protects capacity, such as 100 requests a second per key, passes. Its value tracks capacity, not what a customer bought, and it only has to protect the edge. A quota sold as part of a plan, such as 10,000 API calls a month on the basic tier, fails. The export limit splits the same way. A size cap that protects the system passes, and the extra rows premium customers may request belong in the export service.

Amazon API Gateway's usage plans attach a quota to each API key, which makes the gateway a second copy of each customer's plan to update on every upgrade and cancellation. The exception is an API that is itself the product, with its plans sold in the gateway. There the gateway's owner owns the plan, and the quota passes.

Adapters and Composition Are Narrow Exceptions

A transformation for a backend or client that can't change, such as a vendor system, an old client, or a service partway through a migration, can live in the gateway. It needs an owner, a test against the backend, and a recorded change that will retire it. Its meaning belongs to the backend's owner, so it fails the ownership test. Like a stopgap, it may still run in the gateway, but only on those conditions. Composition for one client belongs in a backend for frontend, Sam Newman's pattern of a service owned by the team that builds the client. That service owns presentation, not the domain rules behind it.

Answer Each Force

Under a deadline, the easiest place for a rule wins, so each force needs an answer that makes the owning service as easy a choice.

Every Request Reaches the Service

Put the check in the export service from the start, and keep plan details out of it. The service asks the entitlements service for this customer's export limit, as a number, and refuses any export over it. That adds a call to each service that serves large downloads, but a shared client library with caching keeps it to a few lines. When sales adds an enterprise plan, only the entitlements service's limits change.

Borrow the Gateway's Schedule Only as Debt

For a one-time change it can make alone, the platform team's queue is short, and sometimes that's what a missed deadline or an incident needs. Suppose a Friday release breaks the export service's limit check, and free customers start pulling millions of rows through the public API. The platform team can ship the four-step policy within the hour, changed to refuse any export without a declared row count, long before the export team can safely deploy a fix. Ship it, but ship it as debt:

The export team owns the fix and the required rows parameter the policy imposed, booked into a release the day the policy ships. Once the incident ends, nothing pushes anyone to replace a policy that seems to work.
The policy carries its owner, ticket, and removal date, so anyone reading the gateway can tell a stopgap from a rule that belongs there.
The fix isn't done until the policy is deleted and the service's own test covers the routes the stopgap never saw.

Make Code in the Gateway Require an Owner

A business rule needs only the platform team's approval to get into the gateway, and that review tends to ask whether the policy works, not whose rule it is. A policy needs cross-team review if it calls another service, runs custom code, or compares a claim to a business value. A pipeline can detect the first two. The review comes from the domain team that owns the concept, or from architecture review when no team clearly owns it.

Some policies can't wait for review or can't pass it. If the owning team can't review before an emergency ships, the review joins the stopgap's debt. If the policy fails the ownership test, it merges only as a dated stopgap or an owned adapter.

Split Authorization by Who Owns the Facts

Security can get consistent, auditable authorization without the gateway. Role-based decisions, such as what the support-agent role may do or which tenant a user belongs to, rest on facts that rarely change mid-request. They can live in one central policy, distributed to every service and evaluated locally on every route, with each domain team owning the entries for its own capabilities. Contextual decisions, such as whether a user may export account 4417 given its status or a legal hold, rest on domain facts that can change from one request to the next. Centralizing those repeats the gateway's split one layer over, so they stay in the domain service, tested by the team that owns the facts.

Where security doesn't own the rules, it can still own how every service enforces them. Shared middleware can reject any request a service doesn't explicitly allow, so a forgotten check refuses access instead of granting it. Every service can also log its authorization decisions to one place security audits.

Either way, the check runs in the service. A gateway-only check trusts the web app and scheduler routes because they run inside the network. NIST's zero trust architecture, SP 800-207, rejects that, holding that there's no "implicit trust granted to assets or user accounts based solely on their physical or network location."

Find the Business Rules Already in Your Gateway

Open your gateway's policies, plugins, mapping templates, and route rules:

Search for business concepts. Claim names and domain terms find most of them. Put each through the ownership test.
Move each rule that fails into the service that owns it, with a check and a test that cover every path to the capability, including jobs, consumers, internal tools, and other gateways.
Give any stopgap that has to stay an owner and a removal date, and add a pipeline check that rejects a policy missing either or past its date. A stopgap with neither is business logic that stayed.