Coder Legion - Recent posts tagged measurement

Southwest Cancelled 71% of Its Flights on 26 Dec 2022. American Cancelled Under 1% in the Same Storm

Fri, 16 Oct 2026 05:51:18 +0000

Originally published at vibeagentmaking.com.

The same crew solver ran without incident at other airlines that week. That is what makes this a measurement rather than a weather report.

In the last days of December 2022, Southwest Airlines flew 517 completely empty planes. Not repositioning flights planned in advance, but ferry flights launched mid-crisis, many on the same city pairs as flights it had just cancelled, while the passengers for those cities watched from the terminal. The airline was burning kerosene to do in the air what its software could no longer do in data, which was to put its crews and its aircraft back in the same place. The detail comes from the pilots' union's written Senate testimony, and it is the whole failure in one image.

Here is that same failure as a number. On 26 December 2022, in the same cleared airspace over the same country on the same day, American Airlines cancelled fewer than one flight in a hundred. Southwest cancelled about seven in ten. Delta and United, flying the identical storm, cancelled single digits and were themselves back to normal within a day. That gap is not a weather report. It is a measurement, and it says the thing the weather cannot: by the 26th, the storm had stopped being the explanation.

Almost everything worth knowing about the Southwest meltdown has already been said. The mechanism is documented: two recovery systems, one for aircraft and one for crews, that did not share a model of the world, so that fixing the planes broke the crews and fixing the crews broke the planes. The cause is named: a crew-scheduling system that could not reoptimize once the network fell far enough out of its normal state, which forced a week of recovery done by hand, on the telephone. The moral has been drawn a hundred times: pay down your technical debt before it comes due. All true, and none of it is what this essay is about, because the hundred-and-first retelling teaches nothing the first hundred did not.

The interesting question is one the coverage mostly skips, and it is a measurement question: where exactly was the threshold, and how do we know? Southwest is unusually good for answering it, because the failure came with a built-in control group. And once you have located the threshold, the payoff is not a story about one airline. It is a method for finding the same kind of threshold in your own system before it finds you.

The failure that came with a control group

Here is the fact that turns this from an anecdote into an experiment. The winter storm that hit over Christmas 2022 did not single out Southwest. It hit every US carrier at once, delivered to the whole industry in the same days over the same geography. That shared forcing is what makes the divergence in response meaningful rather than noise. It isolates the cause the way a control arm isolates a drug's effect, because everything except the system under test was held constant.

Put the responses side by side. On 26 December, by the cancellation figures reported at the time, Southwest scrubbed about 71 percent of its schedule while Delta cancelled around 9 percent, United around 5, and American fewer than 1. Widen from the single worst day to the whole month and the same divergence holds in the federal numbers: the Bureau of Transportation Statistics recorded 57.3 percent of Southwest's December flights departing on time, against 77.2 percent at Delta, 72.5 at American, and 70.7 at United, and it attributed the elevated 5.4 percent industry cancellation rate for the month primarily to Southwest alone. Same weather, same calendar, and Southwest's number was not a little worse than its peers. It was on a different scale.

There is a second control, sharper than the first, and it comes from sworn testimony rather than a flight tracker. Captain Casey Murray, president of the Southwest Airlines Pilots Association, told the Senate Commerce Committee in February 2023 that the crew-scheduling engine at the center of the collapse, the GE Aerospace product employees call SkySolver, "is a G.E. Aerospace crew scheduling tool used by multiple airlines. It is important to note that other airlines used this product during Winter Storm Elliott without issue." The same commercial solver carried other carriers through the same storm in the same days. So the variable under test is not the weather, which everyone got, and it is not even the solver, which several airlines ran without melting down. It is the architecture Southwest built around it.

That architecture is the mechanism. Murray described two planners. Flight dispatch ran cancellations and delays through a homegrown program the airline calls the Baker, which "optimizes changes to aircraft and passenger routings" but "does not effectively account for crew requirements." Crew changes lived in the separate SkySolver, which "runs independent pilot and flight attendant solutions that can contradict Baker's strategy, especially during a disruption." Two systems, each authoritative over half of one problem, each emitting plans the other invalidated. Southwest's own chief operating officer, Andrew Watterson, said it plainly on 26 December: "We had aircraft that were available, but the process of matching those Crew Members with the aircraft could not be handled by our technology." Two controllers optimizing overlapping resources with independent objectives will produce contradictory plans exactly when the coupling they ignore is strongest.

A state change, not an impact

Now watch Southwest's own curve across the week. It cancelled about 42 percent of its schedule on the 25th, 71 on the 26th, 64 on the 27th, 61 on the 28th, and 24 on the 29th, removing roughly 45 percent of its entire operation across 21 to 29 December while the rest of the industry sat in low single digits over the same span. The storm was easing across the country by Christmas Day. American was under 1 percent while Southwest was climbing past 70.

A shared shock that produces wildly different responses tells you the difference is in the systems, not the shock. A response that keeps climbing after the shock that started it has faded tells you something sharper: you are no longer watching an impact, you are watching a state change. An impact tracks its cause. When the cause recedes and the effect does not, the effect has found some other engine to run on.

There is a clean name for a system that gets pushed past a point and then sustains the new bad condition on its own, and it comes from the deep-ocean record. Geologists call it an oceanic anoxic event. A forcing, a pulse of volcanic carbon dioxide, a bout of warming, pushes the ocean past a tipping point into a low-oxygen state, and the dead water persists long after the trigger is gone, because the anoxic state now feeds itself: less oxygen changes the chemistry in ways that keep the oxygen out. The trigger is brief and the state is long, and the two are not the same length because they are not the same thing. That is the exact shape the cancellation curve shows. The storm faded and the peers recovered inside a day or two. Southwest's response kept running for four more days on an engine of its own, each cancellation spawning the next. The analogy needs no geology lecture to do its work. It names the one property the control curve proves: the effect outlasted the cause, which is what a threshold crossing looks like from the outside.

Where the threshold sat, and why it was knowable

A threshold reading is only a measurement if you can point to where the threshold was. Here is the uncomfortable part, and it is the part that retires the storm excuse for good: the threshold had already been measured, in production, fourteen months earlier.

In October 2021, thunderstorms in Florida triggered a meltdown the union says "closely resembles" December 2022, with the airline "unable to manage its crew network," 2,200 cancelled flights and about $75 million over a single holiday weekend. Murray's verdict on it: "That event should have been a wake-up call." That is a threshold, measured live, priced, and filed. It is the largest simultaneous crew displacement the recovery architecture had been shown to survive, and the answer was that it did not survive it. This was not a tail event no one imagined. It was a rehearsal, and the notes were written down.

And while the recovery layer sat where it was, the load on it was deliberately grown. In the union's account Southwest "added 18 new cities over 16 months" during the pandemic while its network "became increasingly fragile," expanding the demand side while holding the recovery side constant. A peer-reviewed post-mortem reaches the same place from a different direction: in a 2023 paper in Environmental Research: Infrastructure and Sustainability, Helmrich, Chester and Ryerson note that the point-to-point route map took the blame in the first assessments, but that it rapidly became evident the crew-assignment software was the thing that could not operate at the scale of the disruption, and that the deeper failure was the airline not recognizing its own growing exposure to a known risk. The threshold was a capacity that a growing system quietly outran, and the system kept passing the only test the budget ever ran, which was that it worked fine on an ordinary day.

How the bad state actually ended

Now the part that inverts intuition, and it is the practical core. Look again at the 28th and the 29th, 61 percent and 24 percent cancelled, and both described as proactive. Southwest had stopped trying to recover. It began deliberately cutting its own schedule, in the union's phrase choosing to "cut with an axe" and cancel anything considered un-crewed, so that the displacement fell back inside what a phone-based manual process could actually absorb. It did not claw its way out by pushing harder on recovery. It got out by shrinking the problem until it was smaller than its own recovery capacity, and then letting the operation re-enter its normal regime.

This is the physics of a queue that has crossed its threshold, and it is genuinely counterintuitive. When the rate at which you can drain a queue falls below the rate at which work arrives, every unit of recovery you attempt generates more than a unit of new work, because a system this tightly coupled answers each fix with a fresh mismatch somewhere else. Southwest recovered every flight it could, and each recovered flight created new crew-position problems faster than the phone lines could resolve them. Working harder dug the hole deeper. The only exit is to cut the arrival rate below the drain rate, hold it there until the queue empties, and then turn the arrivals back on.

The airline discovered this on day five, in public, at a cost usually quoted near a billion dollars, and the money is worth stating honestly because the sources genuinely disagree. Southwest's own fourth-quarter release put the pre-tax hit near $800 million, enough to turn the quarter into a loss; the widely-cited $1.1 billion sits at the high end. The Department of Transportation's December 2023 penalty, announced as the largest ever against an airline at $140 million, resolved mostly into credits for a passenger-compensation system Southwest was ordered to build for its own future customers, leaving roughly $24 million actually flowing to the Treasury as cash. And the fine print that anyone writing about scheduling must flag: the DOT closed its unrealistic-scheduling investigation without making a finding. Every violation charged was downstream consumer protection. The one inquiry that would have delivered an official verdict on the schedule itself produced none, so the penalty, whatever its size, is not evidence about the solver.

How to find your own threshold before it finds you

The transferable content here is not "pay down technical debt." That is true, it is tired, and it treats a measurement problem as a morality problem. The useful thing Southwest teaches is a discipline for locating a threshold you cannot see yet, and it has three parts.

First, find your control group. Southwest's threshold was invisible in its own numbers and obvious the instant you set its 71 percent beside American's under 1 percent on the same day, and more damning still beside the fact that the same crew solver ran fine at other carriers in the same storm. Most systems that matter have a natural control taking the same input: a peer service, a sister region, the previous version still serving some traffic, a shard that took the same load, the same vendor component running elsewhere without trouble. When something degrades, the diagnostic question is not "how bad is this" in isolation. It is "how bad is this compared to the thing that took the same shock and did not break." The gap between the two, not the absolute level of either, points at the cause. If your degradation matches your control's, you have an impact and you wait it out. If it diverges, you have a state change and waiting will not help.

Second, write the capacity down, and re-measure it as the load grows. The threshold in this story was demonstrated in October 2021 and then left unrevised against a network that kept expanding. Every reoptimizer, scheduler, queue, retry loop, and autoscaler you run has a capacity beyond which it stops converging and starts thrashing, and that capacity drifts as the system around it grows. The dangerous property is not that the number exists. It is that nobody re-checks it, because the system keeps passing the only test anyone runs, which is an ordinary Tuesday. The discipline is to test it on an extraordinary day on purpose, to push the queue toward its limit in a drill and find where convergence breaks, before a storm finds out for you. Southwest had that drill handed to it for free and filed the result instead of acting on it.

Third, decide in advance that the exit from a thrashing state is to shed load, not to add effort. When your drain rate is below your arrival rate, throwing more workers, more retries, or more compute at the drain usually makes it worse, because in a tightly coupled system the added effort generates added work. The move that feels like surrender, turning arrivals away, cutting scope, deliberately shrinking the queue, is the one that re-enters the stable regime. Nobody wants to make that call in the moment, because it looks like giving up while customers are stranded. So make the rule before the moment arrives, and write it where the person on shift at 71 percent will find it.

The storm over Christmas 2022 was real, and it was the excuse. The cause was a capacity that a growing system outran and no one re-measured, demonstrated once at $75 million and then run again at more than ten times the price, and the tell was sitting in plain sight on 26 December, in the single number that stayed high while every other number in the same sky came back down.

The threshold was measured once, in October 2021, and never written down where it counted

Every reoptimizer you run has a capacity beyond which it thrashes, and the number drifts as the load grows. Chain of Consciousness is the record that makes such a limit re-findable: a verifiable log of what a system saw, what it decided, and on what basis, so a drill result survives the quarter it was measured in instead of living in one team's memory.

pip install chain-of-consciousness · npm install chain-of-consciousness

Hosted Chain of Consciousness · Verify a record

Sources

Written Testimony of Captain Casey Murray, President, Southwest Airlines Pilots Association, U.S. Senate Committee on Commerce, Science & Transportation, "Strengthening Airline Operations and Consumer Protections," 9 February 2023 (the Baker and SkySolver architecture; the "other airlines used this product during Winter Storm Elliott without issue" control; the October 2021 Florida rehearsal at 2,200 flights and about $75 million; the 18 new cities over 16 months; "cut with an axe"; the 517 empty flights). https://www.commerce.senate.gov/services/files/B8D729EC-5F96-4E8D-A902-F43DA29F2E08
Bureau of Transportation Statistics, Airline On-Time Performance data, December 2022 (on-time departure shares of 57.3 percent Southwest, 77.2 Delta, 72.5 American, 70.7 United, and the 5.4 percent monthly cancellation rate attributed primarily to Southwest), as reported in the BTS Air Travel Consumer Report for that period.
CNBC, "Southwest cancels 70% of flights as travel disruptions ease elsewhere" (26 December 2022), and SFGate, "Southwest cancels 71% of its flights, DOT investigates" (December 2022), for the 26 December per-carrier figures (Southwest about 71 percent; Delta about 9; United about 5; American under 1) and Southwest's day-by-day curve, from FlightAware data reported within. https://www.cnbc.com/2022/12/26/airlines-cancel-17000-flights-due-to-severe-winter-weather-but-disruptions-ease.html · https://www.sfgate.com/travel/article/southwest-airlines-cancels-most-flights-17679635.php
Helmrich, A., Chester, M., & Ryerson, M., "Complexity of increasing knowledge flows: the 2022 Southwest Airlines Scheduling Crisis," Environmental Research: Infrastructure and Sustainability 3(3), 2023 (10.1088/2634-4505/ace5ce), for the peer-reviewed finding that the point-to-point route map was the initial scapegoat while the crew-assignment software was the system that could not operate at the scale of the disruption.
U.S. Department of Transportation, "DOT Penalizes Southwest Airlines $140 Million for 2022 Holiday Meltdown" (December 2023), and DOT Consent Order 2025-12-4 (5 December 2025), for the penalty structure, the roughly $24 million in cash, the closing of the unrealistic-scheduling investigation without a finding, and the operations-modernization investment. https://www.transportation.gov/briefing-room/dot-penalizes-southwest-airlines-140-million-2022-holiday-meltdown
Southwest Airlines Co., "Southwest Airlines Reports Fourth Quarter and Full Year 2022 Results" (26 January 2023), for the roughly $800 million pre-tax cost. https://www.southwestairlinesinvestorrelations.com/news-events/press-releases/detail/184/southwest-airlines-reports-fourth-quarter-and-full-year-2022-results

Figures note. Two figures in this piece were checked with particular care. The per-carrier cancellation percentages for 26 December 2022 are single-day flight-tracker numbers reported by CNBC and SFGate; the whole-month comparison beside them is from the Bureau of Transportation Statistics, and the two measures agree in direction while differing in level, so both are given rather than blended. The other carriers were not uniformly near zero (Delta and United ran mid-single digits on the worst day), but all of them recovered within a day while Southwest did not, and the same crew solver ran without incident elsewhere, which is what the control rests on and why the title stands. The widely repeated claim that SkySolver processed about 300 schedule changes per batch is omitted deliberately: it appears in no primary source, including Murray's Senate testimony, and the threshold argument stands on the documented October 2021 rehearsal instead. The cost is stated as a range because the sources disagree, and the essay treats the cancellation curve, not the money, as its evidence.

Where Did Wikipedia's Readers Go? Measuring the Answer-Engine Bite From the Pageview API

Tue, 22 Sep 2026 12:14:12 +0000

In the five school-season months of March through July 2024, English Wikipedia's article on osmosis was read about 154,000 times by humans. In the same five months of 2026, it was read about 56,000 times. Two years, and nearly two-thirds of its readers are gone.

Over the same window, the article on the Antikythera mechanism, the corroded bronze gearwork that turned out to be a two-thousand-year-old astronomical computer, gained readers: up 3.5 percent. So did the Peloponnesian War, up 5.4 percent.

Hold those three facts together and you are most of the way to understanding what is actually happening to the web's greatest reference work, because the difference between them is not popularity. It is the shape of the visit. "Osmosis" is a question, and questions now get answered before anyone reaches Wikipedia. "Antikythera mechanism" is an afternoon, and afternoons still belong to the page.

What Wikimedia said, and what the open API lets anyone check

On 17 October 2025, Marshall Miller of the Wikimedia Foundation published a post on the foundation's blog with an unusual pair of admissions. Human pageviews were down "roughly 8% as compared to the same months in 2024." And the foundation had discovered, while investigating, that some of what it had been counting as human wasn't: "Around May 2025, we began observing unusually high amounts of apparently human traffic, mostly originating from Brazil," which turned out to be bots built to evade detection. After reclassifying March through August 2025 with updated logic, the real human decline emerged. The post attributed it to generative AI and social media changing how people seek information, "especially with search engines providing answers directly to searchers," and it carried a methodological caveat that deserves framing: "Revising our data in this way means we have to interpret it with care, as our bot detection systems apply different rules at different points in time."

Here is the good news for anyone who prefers checking to quoting: Wikipedia's pageview data is public, per-article, split by agent type, back a decade, served from an open API that requires no key. So instead of restating the foundation's number, we queried the API and computed the thing the coverage never did: the decline segmented by what kind of article a page is. Every figure below comes from those queries, the reproduction recipe is in the footer, and the aggregate number checks out first: for the foundation's own window, our human-pageview computation lands at -7.0 percent against their "roughly 8," the small gap explained by our reading today's fully reclassified data.

Then it gets more interesting than the headline.

The decline has a gradient, and the gradient points at the question-shaped pages

Sort articles into three rough classes. Reference-lookup pages, the "what is X" class whose entire job an AI summary can absorb: osmosis, mitosis, the Pythagorean theorem, standard deviation. News- and event-driven pages: NATO, inflation. And long-tail deep dives, pages nobody visits by accident: the Byzantine Empire, the Battle of Kursk, the Antikythera mechanism.

Comparing March-July 2026 against the same months of 2024, human pageviews in our sample fell 48.2 percent for the lookup class, 38.4 percent for the news class, and 17.3 percent for the deep dives, against a site-wide decline of 13.2 percent. The lookup class fell at roughly three and a half times the site-wide rate. A clean, monotonic gradient, in exactly the order the answer-engine hypothesis predicts: the more completely a summary can do a page's job, the harder that page fell.

The per-article numbers are starker than the class averages, and the spread inside each class is itself informative. Standard deviation, down 59.8 percent. Mitosis, down 52.3. The Pythagorean theorem, down 49.4. But photosynthesis, a sibling homework topic, fell only 23.7, and supply and demand only 19.2, a reminder that even within the question-shaped class, pages differ in how much of their traffic was ever really one-line lookups. The news class has the same texture: NATO down 53.5 and inflation down 51.6, both topics a summary handles fluently, while the Byzantine Empire, closer to reading than to checking, held to a 27.9 percent loss and the Battle of Kursk to 19.4. The gradient is not a cliff between categories; it is a slope that tracks, page by page, how much of each article's job could migrate upstream. And then the two pages that grew, which are the paragraph that matters: the Antikythera mechanism and the Peloponnesian War gained readers through the exact window in which osmosis lost two-thirds of its audience. Nobody asks an answer engine for the Antikythera mechanism in one line, because the reason you are there is that you want to stay a while. The pages bleeding readers are the ones whose job was answering a question that now gets answered upstream, by systems trained, in meaningful part, on those very pages.

Now the honesty this result requires, stated at full volume rather than in a footnote. This is a sample of eighteen hand-picked articles: eight lookup, five news, five deep-dive, chosen to fit the class definitions before their numbers were pulled, but chosen. The gradient is suggestive, not established; establishing it as a population statistic needs a large random sample drawn per class. And the confound has a name: the lookup class is the school class. Osmosis and the Pythagorean theorem are homework, and a shift in how students do homework would produce this exact gradient whether or not it runs through AI summaries. The attribution to answer engines is inference, consistent with the data rather than isolated by it. One methodological point runs the other way and is worth knowing: the foundation's reclassification applied site-wide, so while any absolute level in this data deserves suspicion, a difference between classes measured the same way is considerably more robust than any single number in it.

It did not stop when the news cycle did

The second computation is simpler: the monthly year-over-year change in human pageviews, every month since the decline began. It has now been negative for twenty-seven consecutive months. The worst month in the entire series is not in the foundation's reported window at all. It is March 2026, at -12.2 percent, five months after the coverage cycle ended. For March-July 2026 against 2024, the site-wide human decline is 13.2 percent, roughly double the figure that made the news.

And buried in that monthly table is a trap we want to spring deliberately, because it is a perfect specimen of a failure every metrics owner should recognize. Read naively, early 2026 looks dramatic, at minus ten to twelve percent, and mid-2026 looks like recovery, softening to minus three. But the foundation reclassified only March through August 2025. A January 2026 reading is therefore measured against a January 2025 base that never got cleaned and likely still contains the disguised bots, inflating the base and overstating the decline; an April 2026 reading is measured against a cleaned, lower base. Part of the "deepened, then recovered" shape is neither deepening nor recovery. It is the seam between two definitions of "human," showing up in a trend line, exactly as the foundation's own caveat warned. A definitional change masquerading as a trend, sitting inside the very dataset we are using to measure a real trend, is not a reason to distrust the exercise. It is the second subject of the exercise.

What the machines did is the question the instrument can no longer answer cleanly

The tidy version of this story says the crawlers feasted while the readers left. The data declines to say that.

The API splits traffic three ways: user, spider for declared crawlers, and automated for detected bots. Declared spider traffic did not rise to meet the answer engines; it fell 27.2 percent over our two-year window and now sits below its 2021 level. The automated bucket went the other way: up 28.7 percent over the window, and up roughly 79 percent against its 2021-2023 baseline. Whatever is fetching Wikipedia at scale in 2026 is not doing it under a polite crawler's declared user-agent.

But before anyone builds a narrative on either curve, the confound: spider traffic fell off a cliff between September and October 2025, which is precisely when the foundation re-engineered its bot detection. A change in the classifier explains that step at least as well as a change in crawler behavior, and three of the four traffic series have that definitional seam running through the middle of the measurement window. So the honest summary of the machine side is this: we can measure that the humans left, and we can measure which pages they left. We cannot cleanly measure what the machines did, because the instrument that counts machines was rebuilt mid-window by the same organization reporting the decline. The extraction asymmetry, the encyclopedia feeding an answer layer that starves it, survives in the data, but only in its careful form: human reading down and graded by absorbability, undeclared automation up sharply, declared crawling ambiguous under a moved instrument.

One more specimen for the cabinet of measurement humility: as a control, we tried the Main Page. Its "human" pageviews rose 49.6 percent over the window, from 746 million to over 1.1 billion in five months, which is implausible as a change in reading and almost certainly an artifact of apps and default loads. A single page can be counted in ways that have nothing to do with readers. We report it and use it for nothing, which is what you should do when a control misbehaves.

What to take home

For the web, the reading is sober enough. The encyclopedia's question-shaped pages funded a virtuous loop for twenty years: a student's quick lookup was also a donation impression, a future editor's first visit, a citation followed. The answer layer absorbs precisely those visits first, while the afternoon-shaped pages keep their people. Whether what remains sustains the commons that the answer layer itself is trained on is now a live question, and the monthly series says it is not stabilizing yet. Wikipedia's own community has felt the tension from both directions at once: in June 2025 the foundation paused a pilot that would have put AI-generated summaries atop articles after its volunteer editors revolted, which means the institution losing readers to machine summaries elsewhere declined, under internal protest, to serve machine summaries itself. Whatever the right strategy is, the people who write the encyclopedia have made their position on becoming their own answer engine clear.

For anyone who runs a product with metrics, this dataset is a masterclass you can rerun in an afternoon, and it teaches three disciplines. Segment before you attribute: the site-wide 13 percent hides a spread from minus 60 to plus 5, and any strategy tuned to the aggregate is tuned to a page that does not exist; your surfaces divide into questions and afternoons too, and they are not declining together. Map your instrument's seams before reading trends across them: every reclassification, every bot-filter update, every tracking change is a discontinuity that will impersonate a trend, and the only defense is knowing the dates. And when a partner ecosystem starts answering your users upstream of you, expect the first losses exactly where the visit was shortest, which is often where the aggregate is thickest and the alarm is quietest.

The whole analysis stands on an open API, two scripts, and the willingness to check a widely quoted number instead of repeating it. The number held. The story underneath it was bigger.

Method and sources: Wikimedia Pageviews REST API (wikimedia.org/api/rest_v1), aggregate endpoints for en.wikipedia by agent type (user/spider/automated, monthly, 2021-2026) and per-article monthly series for 21 articles, 2023-2026; windows compared are March-July sums, 2026 vs 2024, human (user) agent unless stated; 18 hand-picked articles carry the class result (8 lookup, 5 news, 5 deep-dive), stated as suggestive, not population-level. Marshall Miller, "New user trends on Wikipedia," Wikimedia Foundation (diff.wikimedia.org), 17 October 2025, for the reported ~8%, the Brazil bot reclassification, and the different-rules-at-different-times caveat. Reproduction scripts, published with this piece and runnable with no API key: wikipedia_pageview_trend.py for the agent-class aggregate series, and wikipedia_pageview_by_class.py for the per-class windows.*

If you publish a number other people act on, the seam problem in this piece is yours too: a reclassification, a bot-filter update, a tracking change, each one a discontinuity that will impersonate a trend to anyone reading the series later. The only defence is a record of when the definition moved and why. Chain of Consciousness keeps that reasoning attached to the result, so a later reader can tell a change in the world from a change in the instrument.

pip install chain-of-consciousness
npm install chain-of-consciousness

Hosted Chain of Consciousness

95%, 80%, 42%: Tracing the 2026 AI-Failure Statistics. One of Five Was Reachable.

Tue, 18 Aug 2026 19:53:19 +0000

Five statistics about AI failure are doing heavy rotation this year. You have probably met all of them. Forty-two percent of companies abandoned most of their AI initiatives in 2025, up from seventeen percent the year before. Ninety-five percent of generative AI pilots deliver no measurable return. One enterprise burned five hundred million dollars on AI in a single month. Eighty percent of AI projects fail. The average abandoned AI project costs 11.3 million dollars.

On August 18, 2026, we ran a small experiment. For each of the five, we recorded the source it is attributed to as it circulates, then probed the claimed primary with an ordinary browser user agent and logged the HTTP status. The question was not whether the statistic is true. The question was narrower and, we would argue, more useful: if the sentence arrives in front of you with a citation attached, can you, the person receiving it, actually walk back to the document it came from?

For one of the five, the answer was yes. MIT Project NANDA's server answered with a 200 and 78,490 bytes. S&P Global, the attributed home of the 42 percent figure, returned a 403 to the probe. So did Axios, the attributed origin of the 500 million dollar story. The last two, the 80 percent failure rate and the 11.3 million dollar average, failed earlier than that: they circulate with no consistent attribution at all, so there was no primary to probe. There is nothing to fail to reach.

Two honest boundaries before anything else. A 403 is a bot wall, not an absence. S&P and Axios both certainly published something, and a human with a browser and some patience very likely reaches both pages. What the probe measures is whether the citation is checkable by the reader who receives it, at the moment of receiving it, and for three of the five the answer is that the reader must take the chain on trust. And the count of five is a sample of what circulates most, not a census of everything said about AI failure this year. The experiment is small. It is also reproducible, which is the point; the script that produced the table is named at the bottom of this page.

Then the experiment produced a wrinkle we did not plan. When we fetched the one reachable primary again while writing this piece, the page that answered was NANDA's current landing page, and the document it links today is a different paper, a February 2026 perspective on decentralized AI. The GenAI Divide report, the actual home of the 95 percent figure, is not linked from the page a reader lands on. The domain is reachable. The document has moved on. Even the best case in our sample, the one citation a reader can follow, now delivers the reader to a lobby where the study no longer hangs on the wall. Reachable domain and reachable document are different things, and the difference opened up in under a year.

The referent goes first

Here is what the reachable study actually measured. MIT Project NANDA's report, The GenAI Divide: State of AI in Business 2025, published in July 2025, found that 95 percent of enterprise generative AI pilots produced no measurable profit-and-loss impact within roughly six months of deployment.

Here is what it circulates as: 95 percent of AI projects fail.

Those are different sentences. A pilot that works, that users like, that is quietly saving four hours a week across a department, and that has not yet shown up as a line movement in a quarterly financial statement, passes the first sentence and fails the second. The original claim is about an accounting horizon. The circulating claim is a verdict on a technology. The number survived the journey intact; the thing the number was about did not.

What makes this stage of decay interesting is that the write-ups repeating it are not ignorant of the problem. One account we read states the criticism plainly, noting that the figure is contested on its narrow success definition and that the criticism attacks precision rather than direction. Then it reports the headline anyway. The correction and the corruption travel in the same article, and the headline wins, because a headline is what gets repeated and a caveat is what gets scrolled past.

The denominator goes second

The strongest evidence in our file needs almost no interpretation, so we will mostly just lay it out.

Two circulating accounts describe the methodology of the same MIT NANDA study. Version A says the researchers conducted 52 executive interviews, collected 153 survey responses, and reviewed more than 300 public deployments. Version B says 150 interviews with leaders, a survey of 350 employees, and 300 public deployments. Same study. Same 95 percent conclusion. The interview count differs by a factor of about three. The survey count differs by more than two. At least one of these accounts is simply wrong, and a reader holding both has no way to tell which.

We can tell which, for a slightly embarrassing reason: this publication has quoted the NANDA report before, in "The Year of the Agent Was the Year of the Demo," and that essay's author read the report PDF and recorded the methodology in the text. It reviewed more than 300 publicly disclosed AI initiatives, interviewed people at 52 organizations, and surveyed 153 senior leaders. Version A is correct. Version B is wrong on both axes, and its second error is worse than its first. Inflating 52 interviews to 150 is a magnitude mistake. Turning 153 senior leaders into 350 employees is a species mistake: a survey of executives and a survey of staff are different instruments that measure different things, and the swap is invisible because the one number anyone quotes, the 95, never changed.

That is the signature of this stage of decay. The headline figure is the only part of a study with enough gravity to hold its shape in transit. The methodology, the part that tells you what the figure means and how much to trust it, has no such protection, and it mutates freely behind the stable number. A reader who checks the headline against the original will find it matches and come away reassured. The corruption is in the parts nobody checks.

The attribution goes last

The 80 percent failure rate and the 11.3 million dollar average abandonment cost represent the terminal stage. Both circulate widely. Neither carries a consistent source. The 80 percent figure has been variously attached to old Gartner predictions, to a RAND study with different numbers about different things, and to nothing at all; in most of the places we encountered it this year it simply appears, in the way that proverbs appear. The 11.3 million dollar figure arrives the same way, precise to one decimal place and anchored to nothing you can pull on.

This is what the end of the process looks like. First the referent detaches, then the methodology, and finally the source itself, leaving a free-floating number with the texture of a finding and the provenance of a rumor. Precision is doing the work that citation used to do. A number with a decimal point in it reads as measured, whether or not anyone can say who measured it.

The 500 million dollar specimen

The most-traveled AI story of the early summer is a perfect laboratory case. An enterprise, the story goes, deployed Anthropic's Claude without usage caps, employees ran long agentic workflows against consumption pricing, and nobody reviewed the invoices for about thirty days. Cost: half a billion dollars in a month.

We found the story on at least eight surfaces, from Inc. and a Yahoo Finance syndication down through a string of trade and content sites, every one of them pointing back at an Axios report from May 28, 2026 that our probe could not open. And in every retelling we read, the company is unnamed. To be clear about what we are and are not claiming: the incident is consistently reported, it is entirely plausible to anyone who has watched consumption pricing meet unsupervised automation, and we are not calling it false. The finding is about the chain, not the event. A 500,000,000 dollar loss attributed to nobody, sourced to a page most readers cannot open, traveled the business internet for weeks without resistance.

There is a piece of conventional wisdom in content strategy that names are what make stories travel, that readers come for named companies and named people. This story is the counterexample. It went everywhere without a protagonist. The engine was never the name. The engine is the number, big enough to be shareable, precise enough to feel real, and unfalsifiable enough to be safe.

What resolving actually looks like

For contrast, the best-sourced figure in the set. S&P Global Market Intelligence's survey work is the origin of the 42 percent abandonment figure, and in the secondary coverage that carries it, a real methodology is visible: more than 1,000 respondents across North America and Europe, the 42 percent who scrapped most AI initiatives up from 17 percent the prior year, an average of 46 percent of proofs-of-concept abandoned before production, and cost, data privacy, and security named as the leading obstacles.

Disclosure, because the rule applies to us exactly as it applies to anyone: S&P's own page returned a 403 to our probe, so we hold these figures through CIO Dive's named, dated coverage of the report rather than off the primary. The chain from you to the study, through this essay, has one hop of trust in it, and now you know precisely where that hop is. That sentence cost us nothing to write and is the entire difference between a checkable claim and a vibe.

Notice what the S&P figure has that the orphaned figures lack. It has a year-over-year shape, 17 to 42, which is a claim about change rather than a static verdict. It has a stated population. It has named obstacles, which are the beginning of an explanation rather than a mood. Real findings tend to come with this kind of texture, because the texture is what the finding is. The smooth, round, free-floating numbers are smooth precisely because the texture has been worn off in transit.

The walk-back test

Put the mechanism in one place. A statistic degrades in a specific order as it travels. The referent goes first: no profit impact inside six months becomes fails. The denominator goes second: 52 interviews becomes 150, and 153 leaders become 350 employees, invisibly, behind an unchanged headline. The attribution goes last, and the number enters its terminal state, circulating alone, precise and orphaned. And at every stage, the walk back to the primary is harder than it looks, because the origin sits behind a bot wall, or a paywall, or a landing page that has since moved on to newer work. One of five, in our sample, and even that one now leads to a lobby.

None of this requires anyone to lie. Every actor in the chain can behave reasonably, compressing for a headline here, paraphrasing a methodology there, dropping a citation that the previous surface also dropped, and the output is still a business discourse arguing about AI deployment with numbers that mean less than they appear to and cannot be checked by anyone in the room.

The countermeasure available to a reader costs about ninety seconds. When a statistic arrives with a verdict attached, try to walk it back one hop. Not to the truth, just one hop, to whatever the citation actually points at. Most of the time you will hit a wall, and the wall is information: it tells you that you are holding a rumor with a decimal point. Occasionally you will reach the study, and what you find there, a horizon where you expected a verdict, executives where you were promised employees, is usually more interesting than the number that sent you.

Sources: MIT Project NANDA, "The GenAI Divide: State of AI in Business 2025" (July 2025; methodology figures, 300+ initiatives, 52 interviews, 153 senior-leader surveys, as read from the report PDF and recorded in this publication's prior coverage; the report is no longer linked from nanda.media.mit.edu's landing page as of Aug 18, 2026, which today links a February 2026 decentralized-AI perspective paper instead). S&P Global Market Intelligence generative-AI survey (2025), figures held via CIO Dive, "AI project failure rates are on the rise: report," and related trade coverage; S&P's page returned HTTP 403 to our probe. Axios, May 28, 2026 (the 500M enterprise bill), not directly readable by our probe (HTTP 403); the eight secondary retellings (Inc., a Yahoo Finance syndication, and six trade/content sites) are cited as evidence about the citation chain only, not as confirmation of the event. Forbes, "MIT Finds 95% Of GenAI Pilots Fail Because Companies Avoid Friction" (Aug 26, 2025).*

Show your work: the reachability table was produced on Aug 18, 2026 by a short script that requests each claimed primary with an ordinary browser user-agent string and records the HTTP status code and response size. It is a few lines to reproduce: request each attributed source URL with a standard browser user agent, log the status and byte count, and record separately the cases where no consistent source URL exists to request at all. The follow-up fetch of nanda.media.mit.edu confirming the landing page's current outbound PDF link was run the same day at writing time. Locator debt, stated rather than hidden: the NANDA methodology sentence should carry a page coordinate from the report PDF itself; our copy of record is our own earlier essay, "The Year of the Agent Was the Year of the Demo," which recorded that methodology from a reading of the report PDF. That essay is public, so this citation is itself one walkable hop; the page coordinate remains owed.*

The walk-back test is cheap for a reader and cheaper still for a system that records where its numbers came from. An agent that cites a figure should be able to hand back the document it read, not just the figure. That is the whole job of a provenance record: keeping the chain from claim to source walkable after the claim has travelled.

pip install chain-of-consciousness
npm install chain-of-consciousness

Hosted Chain of Consciousness · Verify a provenance record

Originally published at vibeagentmaking.com.