Coder Legion - Recent posts tagged postmortem

The Day a Brokerage Minted 30 Times Its Own Company: Samsung's $100 Billion Ghost Shares

Wed, 23 Sep 2026 08:53:31 +0000

One keystroke, one wrong unit, and a share registry created thirty-one times the company it was keeping the books for. The interesting part is not the typo. It is that nothing in the crediting path compared the credit against issuance, and the conservation law that defines a share ledger lived in a legal document instead of in the code.

On 6 April 2018, an employee at Samsung Securities, one of South Korea's largest brokerages, set out to run a routine dividend for the company's employee stock ownership plan: 1,000 won per share, about ninety American cents, owed on roughly 2.8 million shares held by 2,018 colleagues. The intended payout came to about 2.81 billion won, call it $2.6 million.

In the entry field, the employee typed 1,000 shares per share.

The system accepted it. Into those 2,018 accounts it credited 2,812,956,000 shares of Samsung Securities, worth 112 trillion won at the day's price, about $105 billion at 2018 exchange rates. Same numeral the dividend clerk intended. Different unit. The company had roughly 89 million shares outstanding, so the system had just conjured, from a keystroke, about 31.6 times the entire company, stock that did not exist, had never been authorized, and exceeded the issuance ceiling written into Samsung Securities' own articles of incorporation.

Then sixteen employees looked at their suddenly nine-figure account balances and started selling.

The hour the tape became the control

What happened next is the part every engineer should study frame by frame. The phantom shares were sellable, and sold: about 5.01 million of them went to market before the firm stopped it, which at the day's price works out to roughly $186 million, a figure worth labeling honestly as arithmetic on the verified share count rather than a number any regulator published. The stock fell as much as 11.7 percent intraday as the sell orders hit. The company reportedly stopped the bleeding about 37 minutes after becoming aware of it, and by the close, the episode had destroyed on the order of $428 million of Samsung Securities' real market value, none of which had anything to do with the company's actual business.

And one detail elevates those sixteen sellers from a morality tale into a systems lesson: reporting at the time said managers repeatedly warned employees not to sell the shares, and sixteen did anyway. Hold onto that. The instruction existed. It simply had no enforcement behind it, which makes it a control of exactly the same species as the one whose absence created the shares in the first place: a rule that lived in words rather than in code. The only mechanism that responded at machine speed that morning was the market itself, greed and its consequences becoming visible on the tape within the hour. Everything else fired in months, or years.

A typo's blast radius is the price of the thing you're counting

Before the systems argument, one piece of arithmetic that the coverage never quite states, even though every number needed for it was published. The intended transfer was 2.81 billion won. The actual credit was 2.81 billion shares. The ratio between the value of what was meant and the value of what happened is exactly one share price, 39,816 won, which is where Samsung Securities traded that spring; the published figures reconstruct to it precisely.

That is the general law hiding in this incident, and it is not a metaphor. When a field can confuse currency units with instrument units, the error is multiplied by the market price of the instrument. A won-for-shares confusion at a penny-stock registrar is an embarrassing rounding error; the same keystroke at a brokerage whose stock trades at 40,000 won is a hundred-billion-dollar event. No amount of double-checking discipline at the input field changes that multiplier, because the multiplier is a property of the domain, not of the operator. The care you invest in the field scales linearly; the damage scales with price. Which is the first argument that the real defense was never a better input check at all.

What the investigators found, and the question they never answered

Korea's Financial Supervisory Service went in, and its published findings are a familiar litany. The firm had a "lack of a monitoring system and contingency plan." It "failed to immediately suspend sales" once the ghost shares landed. And one detail that deserves a spotlight: the brokerage had updated the relevant computer system that January, three months before the incident, "but did not conduct a test drive to uncover possible errors." The courts, in the civil litigation that followed, found the firm negligent for failing to build an adequate dividend system and to properly establish internal controls and risk management standards.

All true, all documented, and all of it stops one question short. Nothing in the public record explains why the system was able to create shares that did not exist. The IEEE Spectrum write-up that April relayed the regulator's own astonishment, asking how two billion nonexistent shares "managed to get allocated" and how they "could even be legally sold," and no published finding since has answered in mechanism-level terms.

So here is the teardown's argument, offered explicitly as a reconstruction rather than as anyone's official finding, because no regulator or court said it in these words. It is the one explanation that fits every published fact at once.

Think about what checks a dividend-crediting path plausibly runs per account: is this a valid employee, is this a valid instrument, is the number well-formed, does the batch balance against its own inputs. Every one of those checks can pass on the ghost dividend, because per account, nothing is wrong: real employee, real ticker, a number in a numeric field. The check that fails is not per-account at all. It is the invariant that defines what a share registry is: the sum of credited shares of an instrument cannot exceed the shares issued. Samsung Securities had about 89 million shares outstanding, a hard ceiling written into its corporate charter, and the credit that went through breached that ceiling thirty-one times over. Which tells you, as close to conclusively as the public record allows, that nothing anywhere in the crediting path compared the credit against issuance. The deepest fact about the system, the conservation law of a share ledger, lived in everyone's head and in a legal document, and in no line of the code that moved the shares.

And the January detail completes it. The code where that invariant should have lived had just been changed, three months earlier, by an organization that did not test the change. A conservation law that exists only as an assumption survives every change that honors it and detects none that don't.

If the essay's earlier law was "a unit error scales by the price of the instrument," the second is its partner: input validation is linear defense, invariants are structural defense. The first inspects what operators type. The second constrains what the system can do, whatever gets typed. Samsung Securities had, on the evidence, only the first kind, plus a manager's verbal warning where the second kind should have been, twice.

The accountability arithmetic

What did the failure cost the people responsible for the system, as opposed to its shareholders? The Financial Services Commission fined the firm 144 million won, about $129,000, and barred its brokerage business from taking new clients for six months. The CEO, Koo Sung-hoon, was ordered suspended for three months and resigned that July. Set the fine beside the event and let it sit: $129,000, for a system that minted $105 billion of nonexistent stock and vaporized $428 million of its own shareholders' value in an afternoon. The penalty was three ten-thousandths of the self-inflicted loss. Whatever deterred anything here, it was not the fine; the market's same-day verdict was three thousand times larger.

Then, five days before this essay was written, the story got its final chapter. On 12 August 2026, Korea's Supreme Court ruled on the claim brought by the National Pension Service, the country's largest institutional investor, which had sold its Samsung Securities stake into the chaos. The court found a "strong" likelihood of a causal relationship between the employees' negligence and the fund's losses, limited the firm's liability to 50 percent, and awarded 1.86 billion won, about $1.3 million. The fund had sought 29.9 billion won; it recovered roughly six percent of its claim, eight years and four months after the trade date.

Line the control mechanisms up by response time, because the ordering is the lesson. The tape repriced the failure within the hour. The regulator arrived in three months with a $129,000 invoice. The final court ruling arrived in the ninth year, for six cents on the claimed won. Real-time systems get exactly one layer of defense that operates at the speed of the failure, and it is the one compiled into the transaction path. Everything downstream of that is not a control. It is historiography with a payment schedule.

What to take home

Four tests, each cheap against the number in this essay's title.

Name your registry invariant. Every system of record has a conservation law that defines it: credited shares cannot exceed issued shares, ledger debits equal credits, inventory shipped cannot exceed inventory received, tokens redeemed cannot exceed tokens minted. Write yours down in one sentence. If your team cannot, that is the finding.

Enforce it at the write, not at the keyboard. Input validation on the field would not have saved Samsung Securities from a well-formed number in the wrong unit; a constraint at the crediting write, sum of credits against issuance, catches every path to the violation, including the paths not yet enumerated. The invariant belongs where the state changes.

Type your units. A field that can mean won or shares depending on context is a loaded instrument, and the safety is a type system, machine-level or schema-level, that makes currency and quantity uncombinable. The same numeral in a different unit was the entire event.

And audit for controls that are actually wishes. The managers' warning not to sell was a control with no mechanism; so was the invariant that lived in the corporate charter but not the code. Most engineering organizations carry the same inventory without noticing: the deploy freeze that is an announcement rather than a locked pipeline, the "always get review before merging to main" convention with force-push enabled, the spending cap that is a dashboard someone checks weekly rather than a limit the account enforces, the runbook step that says "verify with the on-call before deleting" in a procedure a script executes unattended. Each is an instruction wearing a control's clothing, and each works right up until the first person, or the first automated process, that does not stop to read it. Walk your critical paths and ask of each safeguard: does this exist as enforcement, or as an expectation that everyone will keep behaving? Samsung Securities is what the second kind looks like on the day someone doesn't, and the one mercy of the case is that the whole lesson is now available to everyone else at a discount: the fine was $129,000, the tuition was $105 billion, and the reading takes ten minutes.

Sources: figures per contemporaneous and follow-up reporting, cross-checked arithmetically (the published counts reconstruct to the 2018 share price of ~39,816 won): 2,812,956,000 shares credited to 2,018 ESOP accounts against ~89 million outstanding; 112 trillion won notional (~$105B at 2018 rates, conversion date-dependent); 5.01 million shares sold by 16 employees (≈$186M by computation, not a reported figure); intraday fall up to 11.7%; ~$428M market value lost. Korea Times, "'Lousy' system caused fat finger scandal at Samsung Securities" (8 May 2018) for the FSS findings including the untested January system update; Robert N. Charette, IEEE Spectrum Risk Factor (13 April 2018); FSC penalties and the CEO's July 2018 resignation per Bloomberg/Business Standard reporting; Supreme Court ruling of 12 August 2026 per Yonhap/Korea Times (1.86 billion won to the National Pension Service at 50% liability against a 29.9 billion won claim). The structural claim about the missing registry invariant is this essay's reconstruction from the published facts, including the breach of the issuance limit in the company's articles of incorporation, and is labeled as such.

Walk your own critical paths and ask which safeguards are enforcement and which are expectations.
For agent systems the question is sharper, because an agent does not stop to read a convention: the
spending cap, the review-before-merge rule, the "check with the on-call first" step are all
instructions wearing a control's clothing unless something in the path refuses. The Agent Trust Stack
is the enforcement-and-record half of that problem, so a constraint lives where the state changes
rather than in a document.

pip install agent-trust-stack
npm install agent-trust-stack

See how an enforced constraint is verified

Our Post-Mortem Was So Good We Never Fixed the Bug

Thu, 27 Aug 2026 19:52:18 +0000

A brilliant investigation and a perfunctory one converge on the same three weak fixes. The analysis explains the last failure. Only the tracker, and the engineering it forces, gets a vote on the next one.

In 2017, a team of patient-safety researchers led by Kathryn Kellogg published a study in BMJ Quality & Safety with a title that asks the question this essay is about: “Our current approach to root cause analysis: is it contributing to our failure to improve patient safety?” They had reviewed 302 root cause analyses conducted over eight years at a major academic medical center. A root cause analysis is medicine's post-mortem: a serious adverse event happens, a team convenes, and a structured investigation produces a documented account of what went wrong and why.

Here is the number to sit with: “In 106 RCAs, solutions were proposed.”

One hundred six. Out of 302. Roughly two-thirds of these completed investigations, each triggered by a real patient coming to real harm, each conducted in full by people who cared, produced no proposed fix at all. The analysis ran to completion. The document was filed. The recommendation section, where the future was supposed to change, was empty.

And when solutions were proposed, look at what they were. Of 731 proposed solutions across the studies that had them, the most common types were “training (20%), process change (19.6%) and policy reinforcement (15.2%).” More than half of all proposed remedies fell into those three categories: teach people harder, adjust the procedure, remind everyone of the rule. The authors' conclusion was that the most frequently recommended solutions were weaker interventions, less likely to prevent recurrence. And the recurrence came: “Multiple event types were repeated in the study period, despite repeated RCAs.” The same classes of harm, happening again, at an institution that had already investigated them. Sometimes more than once.

A companion paper the same year, by Mohammad Farhad Peerally and colleagues, cataloged the structural problems with root cause analysis as practiced and gave the phenomenon a name better than any I could coin: “organisational forgetting.” Not the absence of analysis. Analysis, followed by forgetting, followed by the same event, followed by more analysis. The institution remembers everything in its documents and nothing in its systems.

If you build software, you have felt the shape of this. The two-hour incident review with the timeline accurate to the second. The five-whys that actually goes five levels down. The beautifully written document, bookmarked in three channels, linked from the onboarding wiki. And the bug, or its sibling, back in production within the quarter. The title of this essay is the feeling. But the title makes a claim, an inverse correlation between analysis quality and fix likelihood, that I should be honest about up front, because the evidence supports something more precise and more useful.

Decoupled, not inverse

Nobody has measured, as far as I can find, that better post-mortems produce worse follow-through. What the patient-safety literature documents is decoupling: the quality of the analysis and the strength of the remedy vary independently, and the remedy distribution is pulled hard toward weak interventions no matter how good the analysis was. A brilliant investigation and a perfunctory one converge on the same three outputs: training, process tweak, policy reminder. A better post-mortem does not buy you a better fix. It buys you a better document.

There is a plausible psychological mechanism for why thoroughness might even work against follow-through, and it deserves its one paragraph, labeled as hypothesis. Organizational responses to failure do double duty: they repair systems, and they discharge anxiety. Rituals demonstrably reduce the response to failure; there is experimental work showing structured ritual lowers anxiety and even the neural response to performance errors. An incident review has ritual structure: prescribed sequence, assigned roles, formal vocabulary, a defined end. When the analysis is fully satisfying, when every thread is tied off, the anxiety that was driving the work resolves. The organizational need was for a response, and the excellent document is the response. But treat that as a candidate explanation, not a finding. The structural explanation below needs no psychology at all, which is exactly what makes it useful.

The document can only emit words

Sort remedies the way safety engineering sorts them and the pattern stops looking like a mystery or a moral failure. Occupational safety's standard ranking of hazard controls puts elimination and physical engineering changes at the top, and puts training, procedures, and policy, the administrative tier, fourth out of five, because they depend on human behavior at the moment of hazard. This connection between the RCA findings and the control hierarchy is my synthesis rather than either literature's claim, but once stated it is hard to unsee: training, process change, and policy reinforcement, the three commonest RCA outputs at 55% of all proposed solutions, are all administrative controls. The completed investigations were overwhelmingly recommending interventions from the weakest tier that recommends anything.

Why would smart, motivated investigators converge on weak remedies? Because of what an investigation is. Its deliverable is a document, and a document's native outputs are words: instructions, procedures, reminders, lessons. A report can tell people to be careful. It cannot install an interlock, separate a development environment from production, add a hard constraint to a database, or remove a hazardous step from existence. Those stronger remedies require a budget holder, an engineering change, an owner with authority over the system, none of which is summoned by the act of analysis. The strength of the remedy is capped by the form of the artifact. However good the thinking, a document emits instruction, and instruction is the control class that decades of safety engineering rank just above hoping.

This is why the failure survives good intentions. Nobody in the room is cynical. The investigators investigate honestly, the writers write carefully, and the artifact they hand back is structurally capable of carrying only the weakest class of fix. The medical field's own corrective, a framework literally named RCA² (“RCA squared: RCA plus action”), exists because the field recognized that the action half was the missing half. When a discipline has to rename its core practice to smuggle the word “action” into it, the diagnosis has been accepted.

The counterexample that saves this from cynicism

If the story ended there, the conclusion would be that post-incident investigation is theater, and that conclusion is refutable, because one industry runs investigations that famously do change systems.

The National Transportation Safety Board investigates aviation accidents and issues safety recommendations. Here is what makes it the perfect test case: the NTSB has no power whatsoever to force implementation. It carries, at any given time, more than a thousand open, unimplemented recommendations. By the logic above it should be the purest documentation theater in government. Instead, per Government Accountability Office reporting, roughly 78% of its safety recommendations were implemented over a five-year window (a figure as of 2011; date it accordingly). Set that against the RCA baseline, where only about 35% of investigations even proposed something. The comparison is illustrative rather than controlled, different hazard rates, different regimes, different denominators, but the gap is not subtle.

What does aviation have that hospital RCA lacks? Not smarter analysts, and not enforcement power. Four structural features: the investigator is independent, not an arm of the organization being investigated; the findings and recommendations live in a public docket; every recommendation has tracked status, including an explicit classification for “unacceptable action” by the recipient; and external actors, Congress, the press, post-accident hearings, consult that status and apply pressure. A rejected NTSB recommendation does not vanish into a wiki. It sits in public, labeled unacceptable, waiting to be read aloud at the hearing after the next crash.

Which yields the reframe this whole essay has been walking toward, offered as my synthesis of the two literatures: the post-mortem was never the control. The tracker is. An investigation closes the loop only when something outside the investigated team watches whether the remedy landed, and has standing to make noise when it does not. The analysis produces knowledge. The tracker produces consequences for inaction. They are different machines, and organizations that run only the first one get organisational forgetting at industrial scale.

Software's own canon quietly agrees, and it is worth noticing where the emphasis falls when you reread it with this lens. Google's SRE book, the document that taught the industry blameless post-mortems, spends its real energy on machinery: every action item gets an owner and a due date, completion is tracked in a system separate from the document, follow-up reviews are scheduled before the meeting ends, delays escalate to someone with standing to care. If the organization that wrote the book on post-mortem culture needs explicit anti-forgetting machinery bolted onto the culture, the machinery, not the culture, is doing the work. And note what blamelessness actually solved: honesty. In the blameful old regime, the binding constraint on learning was that people hid what happened. Blameless review fixed that, brilliantly, and left implementation untouched, because it was never aimed at implementation. We solved the input problem and declared victory on the output problem.

The audit, and what to do about it

The practical move is a self-audit that takes one afternoon and stings in a useful way. Pull your last ten post-mortems and score two things.

First, the Kellogg question: what fraction proposed any concrete remedy at all? The medical baseline to beat is about one-third. You will likely beat it; most engineering post-mortems do produce action items. Then the harder cut: classify each action item by control strength. “Add a warning to the runbook,” “remind the team in standup,” “update the onboarding doc”: administrative, the 55% tier, words instructing future humans to behave differently under stress. “Add a lint rule that blocks the deploy,” “split the credentials,” “make the dangerous operation require a second key”: engineering controls, changes to what the system permits. Count honestly. The ratio is your organization's true post-mortem output, and for most teams it is heavily administrative, for exactly the structural reason above: the meeting that produces the document cannot spend money or merge code.

One structural note on why the tracker must be a separate system, rather than a better section of the same document or a longer version of the same meeting. The review meeting is doing knowledge work: reconstructing, explaining, teaching. Commitment work runs on a different clock with different participants; it needs the budget-holder who was not in the room, the prioritization fight against next quarter's roadmap, and a check-in cadence measured in weeks rather than one cathartic afternoon. Asking the same ninety minutes to do both jobs reliably produces the document and starves the commitment, because the document completes inside the meeting and the commitment, by its nature, cannot. Split them on purpose. Analysis closes when the document publishes. The incident closes when the tracker says the system changed.

Then fix the loop, not the documents. Three moves, in increasing order of courage. Give every action item an owner and a date, and track completion in a system that is not the post-mortem doc itself; the doc is where action items go to be remembered fondly. Second, report the completion rate somewhere visible and slightly uncomfortable, a dashboard, a monthly review, anywhere outside the team that wrote the items; you are building a small NTSB docket, and the visibility is the mechanism. Third, adopt a one-line rule for the review itself: every incident's action list must contain at least one non-administrative item, one change to what the system permits rather than what people are told, or must state explicitly why none is possible. That last rule is the RCA² insight compressed: force the conversation about control strength while the budget-holders are still in the room.

None of this makes the beautiful document less valuable. Write the great post-mortem; it transfers knowledge, and knowledge compounds. Just stop asking it to do a job documents cannot do. The analysis explains the last failure. Only the tracker, and the engineering it forces, gets a vote on the next one.

Sources

Kellogg, K. M., Hettinger, Z., Shah, M., et al. (2017). “Our current approach to root cause analysis: is it contributing to our failure to improve patient safety?” BMJ Quality & Safety, 26(5), 381–387. (302 RCAs over 8 years at one academic medical center; “In 106 RCAs, solutions were proposed”; “Of the 731 proposed solutions, the most common solution types were training (20%), process change (19.6%) and policy reinforcement (15.2%)”; “Multiple event types were repeated in the study period, despite repeated RCAs”; conclusion re weaker interventions. Verified at the published abstract.)
Peerally, M. F., Carr, S., Waring, J., & Dixon-Woods, M. (2017). “The problem with root cause analysis.” BMJ Quality & Safety, 26(5), 417–422. (Structural critique; “organisational forgetting.”)
The RCA² framework (National Patient Safety Foundation; see also “Putting the 'Action' in RCA²,” Joint Commission Journal on Quality and Patient Safety, 2024): the field's own corrective aimed at action strength.
NIOSH / CDC, Hierarchy of Controls: the five-tier ranking; administrative controls (training, procedure, policy) fourth of five. The mapping of RCA solution types onto the hierarchy is this essay's synthesis.
GAO-20-395 (on the NTSB's Most Wanted List process) and NTSB safety-recommendation statistics: ~78% of recommendations implemented over a five-year window as of 2011; 1,100+ open recommendations; no implementation authority; public tracked status including “unacceptable action.” The aviation-vs-RCA contrast is illustrative, not a controlled comparison.
Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (eds.) (2016). Site Reliability Engineering (O'Reilly), “Postmortem Culture”: owners, due dates, tracked completion, follow-up reviews.
On ritual and anxiety (the labeled hypothesis): experimental work on rituals reducing anxiety and the neural response to performance failure (e.g., Brooks et al., OBHDP 2016; Hobson et al.). Cited as mechanism candidate, not as evidence for an inverse correlation, which no source measures.

The post-mortem was never the control. The tracker is.

The same split applies to an AI agent's decisions: the reasoning is the document, and a document forgets. What closes the loop is a durable, tamper-evident record of what the agent actually did and decided, kept in a system separate from the agent, that an outside party can read and check the status of. That is what Chain of Consciousness is: not a prettier write-up of the agent's intent, but the tracked, inspectable trail of its actual actions and their outcomes, the NTSB docket for your agents rather than the wiki page nobody re-reads. Analysis produces knowledge; the record produces consequences for inaction.

See Hosted Chain of Consciousness · Read the Theory of Agent Trust

pip install chain-of-consciousness · npm install chain-of-consciousness

Or the whole trust stack at once: pip install agent-trust-stack / npm install agent-trust-stack

Originally published at vibeagentmaking.com.