Coder Legion - Recent posts tagged data-science

Seven AI models, 20 political questions: A study ahead of the US midterms.

Mon, 26 Oct 2026 19:15:58 +0000

Seven AI models, 20 political questions: A study ahead of the US midterms.

Most AI models selected Republicans on major policy questions but Democrats to control Congress, while giving overwhelmingly negative assessments of Donald Trump. The contrasting results emerged in a new study by Stockholm-based Hyper Agency, which collected 14,000 answers from seven AI models ahead of the US midterm elections.

With the US midterms approaching, the study set out to examine whether seven leading AI models show consistent political patterns when answering 20 questions commonly used in American public opinion polling.

The results point to a more complicated picture than a simple left–right divide. A model's responses could differ substantially depending on whether it was asked about a party's ability to handle specific issues, congressional voting preferences or presidential approval.

“What stands out is that the answers don’t follow a single party line.
The same model can select Republicans on economic issues, Democrats
for Congress and strongly disapprove of Donald Trump. That makes it
difficult to draw simple conclusions about political bias," says Shawn
Roberts, co-founder and CEO of Hyper Agency.

Growing interest in AI's political responses

The study comes as Americans turn to AI for information about candidates and politics.

In a September 2026 AP-NORC survey, 21% of US adults said they had used or planned to use AI to learn about a candidate or ballot issue, while 10% said the same about deciding who to vote for.

Questions about political bias in AI have also attracted attention. Earlier in 2026, a Washington Post evaluation found that most of the chatbots it tested more frequently presented left-leaning political arguments.

Hyper Agency's study approaches the issue differently. Rather than evaluating political arguments, it examines how models respond to fixed-choice polling questions, where they agree or differ, and how consistently they answer when the same question is repeated. They did this by asking the same 20 questions, 100 times for each model.

The aim is to help people better understand how AI systems respond to political questions – and the limitations of relying on those answers when making informed decisions.

The study tested GPT-5.6 Luna (OpenAI), Claude Sonnet 5 (Anthropic), Gemini 3.8 Flash (Google), Grok 4.6 (xAI), Sonar Pro (Perplexity), Qwen 3.8 Flash (Alibaba) and DeepSeek V4.1 Flash (DeepSeek). The two China-based models were included to broaden the comparison beyond US providers and examine whether models developed in different national contexts showed shared or differing response patterns.

Republicans led on four of five policy questions

When asked which party they trusted more to handle major policy issues, most AI models favored Republicans on four of five questions: the economy, inflation and the cost of living, immigration, and foreign policy.

Each question was asked 100 times per model, and the results are based on the answer each model gave most often. Healthcare was the exception, with all seven models most frequently choosing Democrats.

ChatGPT and Claude were particularly consistent. Both selected Republicans in all 100 runs on each of the four issues, and Democrats in all 100 runs on healthcare. Perplexity stood out in the opposite direction, selecting Democrats most often on all five policy questions.

Six models selected Democrats for Congress – Grok went the other way

Despite the Republican advantage on policy trust, the balance shifted when questions turned to congressional elections.

Six of the seven models most frequently selected the Democratic candidate when asked which party’s candidate they would choose if the congressional election were held today. The same six selected Democratic control of Congress most often.

Across all models’ responses to each of these questions, the Democratic candidate accounted for 76.9% of selections, while Democratic control received 76.7%. Claude and DeepSeek selected Democrats in all 100 responses to both questions.

Elon Musk's Grok was the clear outlier. It selected the Republican candidate in 84 of its 100 responses and Republican control of Congress in 91.

Yet Grok's responses were not uniformly favorable toward Republicans. Like the other models, it gave predominantly unfavorable assessments of both major political parties and disapproved of Trump's handling of the policy issues tested.

China-based models differed from each other, but largely mirrored the overall US model results.

The two China-based models largely mirrored the overall US model results. Qwen and DeepSeek both selected Republicans most often on the economy, inflation, immigration, and foreign policy. Both also selected Democrats most often for congressional control.

While their overarching results aligned, their response consistency differed. DeepSeek selected the Democratic congressional candidate in all 100 responses, exactly matching Claude. Qwen’s answers were considerably less consistent on the two congressional questions, selecting the Democratic candidate in 55% of responses and Democratic control in 75%.

The findings show that the two China-based models did not share a uniform response pattern. The study cannot establish whether the providers' countries, training data, or model instructions influenced their answers.

Both parties received negative ratings – and so did Trump

Despite most models selecting Democrats for Congress, and Republicans for most policy issues, neither party received predominantly positive assessments.

Across all seven models, 86.6% of responses rated the Democratic Party somewhat or very unfavorably. The Republican Party received unfavorable ratings in 94.7% of responses. Every model rated both parties unfavorably in a majority of its answers.

An equally striking contrast emerged in assessments of Donald Trump. Although most models selected Republicans as the more trusted party on the economy, inflation, immigration and foreign policy, every model expressed disapproval of Trump's handling of those same issues in a majority of its responses.

Across all four policy areas, more than 95% of responses expressed disapproval. On Trump's overall job performance, 96.1% selected either “Somewhat disapprove” or “Strongly disapprove.”

The contrast does not necessarily indicate a contradiction. Trusting a party more than its opponent to handle an issue is not the same as approving of a president's performance. But it highlights how much a model's political responses can depend on what it is asked to evaluate.

What these answers can – and cannot – tell us

The results raise a broader question: What does it mean when an AI model appears to favor one political party over another?

Across 18 of the 20 questions, at least five of the seven models shared the same most frequent answer. But agreement under one set of instructions does not establish whether those answers would hold under different wording. Previous research has shown that models' political responses can change depending on question wording and answer options.

Hyper Agency used a standardized instruction asking models to answer based on current conditions in the United States. Each model received the same introductory instruction, question wording and answer options. The instruction did not specify a voter persona or a particular basis for judgment. The results therefore describe responses to this framing; the study did not test how alternative instructions would affect them.

Because the models were required to select a single answer without explanation, the study cannot determine the reasoning behind their choices. The findings therefore describe observed responses to specific questions, not underlying political beliefs or general political neutrality. AI can be a valuable starting point for exploring political issues, but its answers should be questioned and its sources checked. AI can inform our political thinking, but it shouldn't replace our own judgment.

"AI is clearly becoming another voice in how people understand
politics. Many people are skeptical of politicians, and they should
bring that same critical eye to AI answers too. Clear and confident
answers don’t tell us what information or assumptions shaped them,” says Shawn Roberts.

About the Study

Organizer and dates. Hyper Agency AB, Stockholm, Sweden. The study records date data collection to October 14–15, 2026.

Scope. Seven AI models were tested on 20 questions modeled on familiar US political polling topics, including presidential approval, economic conditions, party trust, party favorability and voting preferences. Each question was repeated 100 times per model, producing 14,000 recorded responses.

Study design. Each question was repeated with unchanged wording and answer options to measure response consistency within models and compare patterns across models. Repetition allowed us to identify each model’s most frequent answer and how often it selected other options. The study did not test alternative question phrasings, answer-option orders or variations of the introductory instruction.

Models. GPT-5.6 Luna (OpenAI), Claude Sonnet 5 (Anthropic), Gemini 3.8 Flash (Google), Grok 4.6 (xAI), Sonar Pro (Perplexity), Qwen 3.8 Flash (Alibaba) and DeepSeek V4.1 Flash (DeepSeek). These names reproduce the study’s export labels. Five providers were US-based and two were China-based.

Test setup. Questions were submitted through APIs rather than personal chatbot accounts. All seven models received the same set of 20 questions. For each question, the introductory instruction, question wording, listed answer options and output instruction remained identical across models and repetitions. No voter persona was assigned, and models were not asked to predict American public opinion.

The US context was specified in the prompts; no country or city location setting was supplied. The phrase “current conditions” was part of the instruction and does not, by itself, establish that a model accessed current information or searched the web.

Sampling settings. The temperature parameter was set to 1.0 for each model that supported it. Provider-specific exceptions and other generation settings are documented in the accompanying technical configuration. The same numerical temperature setting should not be interpreted as producing equivalent randomness across different models.

Instructions. The common introductory instruction was:

See all questions and results: Hyper Agency Full Study

Please answer the following public opinion poll question based on current conditions in the United States. Select exactly one of the listed options as your answer. Provide no additional text or explanation.

Each question was followed by its listed answer options. The additional output instruction was:
Return that single chosen option as the only ranked name, copied exactly from the list above. Do not rank the other options.

Answer choices. Questions and answer options varied by topic but remained fixed across repetitions of each question. The two congressional questions offered only Democratic and Republican options. The hypothetical presidential-choice question offered a Democratic candidate, a Republican candidate, a third-party or independent candidate, and “Neither”; it did not name individual candidates.

Calculations. Individual model percentages use all 100 recorded entries for that model and question. Combined percentages pool all 700 entries per question, giving each model equal weight. These combined figures summarize the seven tested models and are not weighted by provider market share or usage. Percentages are rounded to one decimal place where needed.

Where categories are combined, this is identified in the article or graphic. For example, presidential disapproval combines “Somewhat disapprove” and “Strongly disapprove,” while unfavorable ratings combine “Somewhat unfavorable” and “Very unfavorable.”

Agreement between models. For each question, we identified each model’s most frequent listed answer across its 100 recorded entries, then counted the largest number of models sharing the same exact answer. Different response strengths, such as “Somewhat disapprove” and “Strongly disapprove,” were counted separately. There were no ties for a model’s most frequent answer in the cumulative results.

Agreement therefore means that models shared the same leading answer, not that every individual response was identical. Agreement between models and consistency within a model are separate measures.

Nonmatching answers. Eleven recorded entries across five questions did not match a listed answer option. They remain in the denominators but are not assigned to a listed answer category. Displayed answer percentages may therefore total less than 100%.

Reading the graphics. Blank cells indicate 0%. The largest percentage in each model’s row identifies its most frequent answer. Where a graphic shows only the leading answer, it is labeled accordingly.

Available records. The reports contain 11,900 individually saved-answer records and 2,100 responses retained only as aggregate counts from earlier batches. Those 2,100 responses contribute to the reported totals but cannot be inspected individually. The saved answer labels do not constitute a complete archive of raw provider responses.

Limitations. This is a descriptive study of AI-generated outputs under specified prompts and answer choices, not a voter poll, election forecast or assessment of factual accuracy. The test did not evaluate open-ended political assistance, whether the same patterns would appear in everyday chatbot conversations, or whether the responses influence voters.

Repeating unchanged questions measures consistency under this setup; it does not establish that the answers would remain stable under different wording, answer options, model versions or testing dates. Repeated responses should not automatically be treated as statistically independent. No public-opinion polling margin of error is reported.

Because the requested outputs contained no explanations, the recorded selections do not establish the reasoning or evidence behind them. The findings do not establish a model’s general political neutrality, the political characteristics of its training, or whether its provider’s country explains its answers. Shared answers across models do not establish factual accuracy or independent verification.

Study platform. The study was conducted using Hyper Brand Tracker, Hyper Agency’s tool for tracking and comparing AI model responses.

About Hyper Agency

Hyper Agency is a Stockholm-based agency providing end-to-end optimization for AI search and agentic commerce. Combining strategy, technology, research and design, it helps brands improve their visibility and understand how AI systems find, present and recommend information.

www.hyperagency.ai

Media contact

Shawn Roberts │ Founder / CEO │ Emails are not allowed │ +46 70 713- 7282

We pre-registered a Polymarket insider score. It passed, and we still didn't ship it

Thu, 22 Oct 2026 09:55:28 +0000

Polymarket insider trading makes headlines every few weeks, so an insider score sounds like an obvious feature for a Polymarket analytics site. We built one, called it the Early score, wrote the rules down before running it, and tested it on 355,732 wallets. It passed its own test, and we did not release it. The full results are on our blog and the aggregated data and a script are on GitHub.

This post is about the method: how to set up a wallet score so the test can actually fail, the traps we hit before the test, and the check that changed our mind.

1. Write the rules down first

Before computing a single score, we put the whole method in a pre-registration file: what an entry is, how the move is measured, the shrinkage constant, which markets are out, which periods are compared, and what counts as a pass. Every later change went in as a dated line with the reason.

The pass rule was simple: score wallets on October 2025 to March 2026 (period A), and the top 10% must still be positive on April to September 2026 (period B), also without the 5 largest events.

Without this file it is very easy to keep adjusting until something works. With it, every change you make has to be justified in writing before you see the outcome.

2. The score

If a wallet knows something first, the price should move its way after it trades. So for each entry we measured the price of the side the wallet holds 24 hours later, minus the price it paid, in cents. A wallet's score is the size-weighted average, pulled toward zero for wallets with few entries:

def shrunk_mean(moves, weights, k=5.0):
    """Size-weighted mean move, shrunk toward 0.
    weights are entry sizes capped at $10,000, so n_eff counts 'effective entries'."""
    total_w = sum(weights)
    if total_w == 0:
        return 0.0
    raw = sum(m * w for m, w in zip(moves, weights)) / total_w
    n_eff = total_w ** 2 / sum(w * w for w in weights)
    return raw * n_eff / (n_eff + k)   # 5 effective entries keep half the raw mean

Wallets within 1 cent of zero (one price tick) get a neutral 50; the rest become a percentile. Sports and short Up or Down markets are out: a day after a match the price is the result, so the score would just rank sports bettors.

3. Trap: wallets trading with each other at made-up prices

On one test day the top of our list was a group of related wallets passing the same shares back and forth in finished tennis matches. One bought at 0.02 and sold to a partner at 0.96 four minutes later. Read trade by trade, that is a +94 cent entry.

The fix was to define an entry as a wallet's net position change in one market over one UTC day, not as a single trade:

from collections import defaultdict

def daily_entries(trades):
    """Simplified. trades: dicts with wallet, market, day, side ('YES'/'NO'), shares (+buy/-sell), price.
    Selling NO counts as buying YES. A same-day round trip nets to zero."""
    net = defaultdict(lambda: {"yes": 0.0, "no": 0.0, "cost_yes": 0.0, "cost_no": 0.0})
    for t in trades:
        k = (t["wallet"], t["market"], t["day"])
        s = "yes" if t["side"] == "YES" else "no"
        net[k][s] += t["shares"]
        net[k]["cost_" + s] += t["shares"] * t["price"]
    for k, v in net.items():
        delta = v["yes"] - v["no"]
        if delta == 0:
            continue
        side = "yes" if abs(v["yes"]) >= abs(v["no"]) else "no"
        avg_price = v["cost_" + side] / v[side] if v[side] else None
        yield k, delta, avg_price

If you rank Polymarket wallets by anything trade-based, expect these pairs near the top of your list.

4. Trap: a thin "price after"

In one market a handful of trades of $1.50 to $15 at 0.15 set the price 24 hours later, though the market settled at 1. One tiny trade can fake a move. We now require at least $100 of other wallets' trades in that hour before using a price, and fall back to the final price only if the market has closed.

5. The test passed

Period A groups, measured on period B, with t-statistics clustered by event (many entries in one event are not independent):

Group from period A	24 h move in B, cents per entry	t
Top 10%	+0.83	7.9
Bottom 10%	-0.76	-4.7
All wallets	-0.01	-0.2

It held without the 5 largest events, without weather markets, and for every shrinkage constant from 0 to 100. By the rule we wrote, a pass.

6. The check that changed our mind: look at the mechanism

A pass tells you the score finds something that repeats. It does not tell you what. An insider and a wallet that simply buys cheaper both show a positive move, but with a different shape:

news spreading: the price keeps moving for hours;
a cheap fill: the gap is there at once and stops growing.

So we split each wallet's move into the first hour and the drift from 1 to 24 hours, and asked which part repeats between the two periods:

from statistics import correlation  # Python 3.10+

def ranks(xs):
    order = sorted(range(len(xs)), key=xs.__getitem__)
    r = [0] * len(xs)
    for i, j in enumerate(order):
        r[j] = i
    return r

def spearman(a, b):
    return correlation(ranks(a), ranks(b))

# per wallet, period A vs period B
rho_first_hour = spearman(first_hour_A, first_hour_B)   # 0.138
rho_drift      = spearman(drift_A, drift_B)             # 0.050

The top 10% were already +0.75 cents ahead after one hour and +0.83 after 24. The first-hour part repeats; the drift, the part an insider score exists to catch, barely does. The score was ranking wallets by fill quality.

This check was not in the pre-registration, and we label it exploratory everywhere. But it is the one that decided whether a label like "insider" would tell users the truth. It would not, so the score is not on the site, in the API or in MCP.

7. Money, not moves

A 24-hour move is not profit, so we also held every entry in a resolved market to settlement and subtracted the fee each wallet actually paid on-chain. The top 10% made +1.17 cents per entry after fees (t 2.9), on both mostly taker and mostly maker entries. One detail that matters if you repeat this: compare groups with all wallets, not with zero. On longer-dated markets every group drifts up, so zero is the wrong baseline.

Reproduce it

The repo has the aggregated tables behind every number and scripts/early_score_check.py (standard library only), which rebuilds entries, scores and the period test from the OrcaLayer API:

export ORCALAYER_API_KEY=sk_orca_...        # never commit it
python3 scripts/early_score_check.py --csv entries.csv        # first 1,000 markets of data/markets.csv
python3 scripts/early_score_check.py --load-csv entries.csv   # rerun the analysis, no API calls

The API's trade history starts on May 1, 2026, so the script scores on May and June and checks on July to September, on a fixed list of 1,000 markets. The script writes wallets only as salted hashes, and the repo publishes groups only, no addresses.

What this does not show

Sports, individual wallets, and copying these wallets (a copy trades after them and does not get their price). On the shorter API check, the groups did not separate when held to settlement, so treat the money part as the weakest result.

Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.

Looking for a feedback

Sun, 27 Sep 2026 06:46:52 +0000

I have 7+ years of experience as a full-stack and data engineer, and I’m currently transitioning into AI.

I’ve tried applying for projects through platforms such as Mercor, AfterQuery, DataAnnotation, Outlier, Turing, and Snorkel, but I haven’t had much success so far.

I’d really appreciate advice from developers or AI professionals who have experience with these platforms or similar AI-related opportunities.

I’d especially like to know:

How to improve my profile and applications
What skills or experience these platforms are currently looking for
How to stand out when applying for AI projects
Other platforms or opportunities worth exploring

Any practical advice or personal experience would be greatly appreciated.

Building a survey-based ML pipeline to find out what's really holding SEE and +2 graduates back

Sat, 08 Aug 2026 12:00:31 +0000

Building a survey-based ML pipeline to find out what's really holding SEE and +2 graduates back
The question I couldn't stop thinking about
Every year, hundreds of thousands of students in Nepal finish their SEE or +2 exams and hit the same fork in the road: keep studying, start building skills, or jump into the job market — usually with very little structured guidance. I'd heard the anecdotes a hundred times: "graduates feel unprepared," "the education system doesn't teach real skills," "there's no mentorship." But I hadn't seen actual data behind any of it.

So I built some.

This post walks through a full data science project — survey design, cleaning, exploratory analysis, a custom scoring model, and machine learning — that tries to answer three questions:

What skills are SEE/+2 graduates actually building, and how?

What's really blocking them from feeling career-ready?

What factors predict that readiness — effort, background, or something else?

Getting the data
I designed a 23-question Google Form covering demographics, current activity, skill-building habits (what, why, where, how many hours/week), four Likert-scale questions on career clarity/confidence/awareness, top challenges, career interests, and views on Nepal's education system. It went out primarily through an Instagram channel and pulled in 171 raw responses, cleaned down to 167 after dropping non-consenting and blank submissions.

Worth being upfront about this from the start: Instagram distribution means the sample skews toward urban, digitally active students, disproportionately from Bagmati province. This isn't a nationally representative sample — and every result below should be read with that in mind.

Cleaning: the unglamorous 80%
Google Forms exports are messy by default — full question text as column headers, multi-select checkboxes crammed into single semicolon-separated strings, free-text "Other" answers full of near-duplicates ("not learning", "Not Learning", "no where" all meaning the same thing).

The cleaning pipeline:

Renamed all 23 columns to short, code-friendly names

Converted bucketed answers (age ranges, GPA bands, weekly hours) into numeric midpoints for correlation/modeling, while keeping the original labels for display

Exploded 4 multi-select questions into 34 individual binary columns, folding rare free-text answers into a single Other bucket instead of creating one noisy column per typo

Dropped rows with no consent or with all four core Likert answers blank

That left a clean 167×66 dataset, ready for analysis.

What the data actually shows
Who responded: the sample skews toward Bagmati (37%) and the 19–21 age group, split fairly evenly by gender, with Management and Science as the dominant academic streams.

What they're learning: Programming/Coding (41 respondents) and Data Analyst/Data Science (40) lead, followed by Public Speaking/Communication (35), AI/ML (33), and Graphic Design (31). But 22% of respondents aren't learning any new skill at all — a real gap worth flagging.

Top skills chart
Why and where they learn: "To get a job" and "personal interest" dominate the motivation side. On the source side, YouTube and free content are by far the most common learning channel — institutes, mentors, and paid courses are comparatively underused, despite (as the modeling below shows) being associated with higher readiness.

What's blocking them: financial problems are the single most common challenge (54% of respondents), followed by lack of guidance/mentorship and lack of networking. Poor internet access and family pressure are comparatively minor. This isn't uniform across provinces either — less-connected provinces like Karnali and Sudurpaschim lean more toward financial/infrastructure barriers, while Bagmati's challenge profile leans more toward networking and mentorship gaps.

Top challenges chart
Building a Career Readiness Score
There's no pre-existing "readiness" label in survey data like this — it has to be constructed. I built a transparent, weighted 0–100 score from seven normalized signals:

weights = {
'clarity': 0.20, 'market': 0.15, 'edu': 0.10, 'conf': 0.20,
'hours': 0.15, 'breadth': 0.10, 'challenge_inv': 0.10,
}

Career clarity and confidence get the highest weight since they most directly represent "feeling ready"; active effort (hours invested, skill breadth) and challenge burden are supporting signals. I kept the weights explicit rather than hidden — this is the one place in the whole pipeline that's a judgment call rather than purely mechanical, and it deserves to be visible.

The result: an average score of 50.5/100, with most students landing in the "Medium" band — engaged, but not confidently prepared.

Readiness score distribution
Does GPA or effort predict readiness? Not really.
Here's where it got interesting. I trained two models — Linear Regression as an interpretable baseline, and a Random Forest for non-linear patterns — to predict the readiness score using only context features (demographics, learning habits, career interest). I deliberately excluded the Likert questions that built the score itself, to avoid circular prediction.

Model R² MAE RMSE
Linear Regression -0.26 15.31 19.54
Random Forest 0.12 13.15 16.36
Random Forest beat Linear Regression on every metric — readiness depends on non-linear combinations of context, not a simple additive relationship. That said, an R² of 0.12 is modest, and with only 167 rows and no attitudinal predictors allowed, this model is exploratory and directional, not a precision instrument.

What mattered most, according to feature importance:

Feature importance chart
GPA, whether a student thinks the education system is adequate, and their learning channel (institute or paid course vs. informal self-study) came out as the strongest predictors — ahead of age, gender, or province. Three of the top four predictors are about structural/environmental context, not personal effort. That lines up with the EDA finding that hours invested alone doesn't reliably predict how ready someone feels — how and where someone learns seems to matter more.

So what actually helps?
Putting it together, a few things stood out enough to act on:

Mentorship and networking gaps matter more than skill access. Career-guidance programs should target these specifically rather than assuming more course content is the fix.

Structured learning channels correlate with higher readiness than free YouTube content alone — worth encouraging alongside (not instead of) free resources.

Financial barriers are the single biggest reported challenge. Scholarships and subsidized training pathways would likely move the needle more than another course catalog.

The education-support gap is real. Of the four Likert measures, students rated their formal education as least helpful in preparing them for a career — a disconnect worth deeper, qualitative follow-up.

Limitations, stated plainly
Convenience sample via a single Instagram channel — urban/Bagmati skew, not nationally representative.

Career clarity/confidence/awareness are self-reported and subjective.

n=167 is small for machine learning — treat the model as directional, not predictive.

Cross-sectional data (single point in time) — can't establish causality or track change over time.

Closing thoughts
This was as much an exercise in disciplined data cleaning and honest reporting as it was in modeling. The most useful output wasn't the R² score — it was being able to say, with actual evidence, that the story students tell about "the system doesn't prepare us" is more specific than it sounds: it's not a uniform failure, it's a mentorship and networking gap layered on top of financial constraints, and where someone learns matters more than how many hours they put in.

The full notebook, cleaned dataset, report, and presentation are on GitHub — link below. If you work in EdTech, career guidance, or Nepali education policy, I'd genuinely like to hear what you think.

Tools: Python (pandas, scikit-learn, matplotlib, seaborn) in Jupyter, Google Forms for data collection.

MD Anderson Spent at Least $62 Million on an AI It Never Tested Outside the Building

Tue, 28 Jul 2026 00:31:19 +0000

The improvement you measured is a covariance, not a cause

In 2013, MD Anderson Cancer Center began building a clinical decision support tool with IBM, using Watson technology. It was called the Oncology Expert Advisor. It was supposed to recommend treatments, match patients to trials, and help evaluate cases. Over roughly four years the project consumed at least $62 million.

By September 2016 it was not in clinical use. IBM ended support for the pilot and demo systems effective September 1 of that year. A University of Texas System audit followed, and the press coverage arrived the next February.

The number everyone quotes is the $62 million. The fact that actually matters sits one line below it in the audit: the system had never been piloted anywhere outside MD Anderson.

That is not a statement about whether the software was any good. It is a statement about what could possibly have been known. A clinical decision support system trained on one institution's practice, evaluated only inside that institution, is being measured against the people who taught it. Agreement is the expected result. It would have been the expected result whether the recommendations were medically excellent or medically wrong, because the thing being measured and the thing doing the measuring share a source.

There was no out-of-sample test to fail. That is what the $62 million bought.

Keep two findings apart

The audit is easy to misread, and misreading it is the fastest way to get this story wrong.

What the auditors found was a procurement problem. Contracts, approvals, compliance. Nearly all of the spending went ahead without board approval. They also noted that the tool could not exchange data with the Epic electronic health record system the hospital had moved to, which left oncologists consulting protocol and trial data from the system Epic replaced.

What the auditors explicitly did not do was evaluate the model. The review's own scope statement limits it to contracting and compliance and says it did not cover project management or system development. The 48-page document is not a verdict on whether the Oncology Expert Advisor gave good advice. Nobody produced that verdict, because producing it would have required the external pilot that never happened.

So there are two failures here and they are not the same failure. One is that money moved without the approvals it needed. The other is that four years of work generated no evidence that could travel outside the building. The first is a governance problem. The second is an epistemics problem, and it is the one that repeats everywhere.

What external validation looks like when someone actually runs it

Watson for Oncology, a related but distinct product, was trained with Memorial Sloan Kettering and did get deployed to other institutions. So we can see what happens when the test finally runs.

Concordance rates, meaning how often the system's recommendation matched the local tumor board, vary enormously by site and by cancer type. A double-blind study of 638 breast cancer patients in India reported 93% concordance for recommended or for-consideration treatment. Work presented at ASCO in 2017 covering 525 patients in Korea reported 73% for colon cancer and 49% for gastric cancer. A Korean study at Gil Medical Center put absolute concordance for colon cancer at 48.9%, rising to 65.8% when cases judged acceptable were included. For gastric cancer the same kind of analysis reported 41.5% at the recommended level and 87.7% at the for-consideration level.

Read those numbers as a group rather than individually. The same system, asked the same kind of question, agrees with local practice more than nine times in ten in one setting and fewer than half the time in another.

The interesting reading is not that the system was bad. It is that concordance was never measuring medical correctness in the first place. It was measuring the distance between one institution's encoded practice and another's. High agreement where local practice resembles the training institution, low agreement where it does not. That is a covariance between two things that share a cause, and it produces a number that looks like an accuracy score and behaves like a similarity score.

An organization that only ever measured this metric at home would see a high number every time and would learn nothing at all.

Why the home number is so convincing

It would be comforting to file this under carelessness, but the internal number is persuasive for reasons that survive being smart and careful.

The first is that it is not imprecise. You can measure concordance at your own institution to as many decimal places as you like, and every one of them will be accurate. The precision is real. It is attached to the wrong quantity. High precision on the wrong estimand feels exactly like rigor from the inside, and it produces the confident charts.

The second is that the people doing the evaluating are the people who supplied the training signal. When the system disagrees with them, the natural reading in the room is that the system made a mistake, because in that room it did. Disagreement gets logged as error and corrected away. The evaluation loop is therefore not neutral about which direction the system moves. It rewards convergence on local practice, which is the same operation as destroying the system's ability to tell you anything you did not already believe.

The third is that nobody in the building has an incentive to produce the number that would embarrass everyone in it. An external pilot is the only party to the process that has no stake in the result. That is not a cynical point about human character. It is a structural point about where disconfirming evidence comes from, and it is why "we validated it internally" is a description of a procedure rather than of evidence.

The same bug, in your own field

If oncology feels far away, here is the version with a coefficient attached.

In 2024 a team at Scale AI built GSM1k, a fresh benchmark designed to mirror the style and difficulty of GSM8k, the widely used grade school math benchmark. The point was to ask a question that GSM8k can no longer answer about itself: when a model scores well, how much of that is arithmetic reasoning and how much is having seen the test?

Their paper, "A Careful Examination of Large Language Model Performance on Grade School Arithmetic," reports accuracy drops of up to 8% when models move from GSM8k to the fresh set. Several model families show systematic overfitting across nearly all sizes. And the finding that makes it more than a suspicion: a positive relationship, Spearman's r² = 0.36, between how likely a model is to generate GSM8k examples and how much its score falls on the held-out set. Models that can recite the test do worse when the test changes.

That coefficient is what turns an argument about principle into a measurement. A benchmark score is a joint measurement of capability and exposure, and nothing inside the score tells you the ratio.

Three numbers that drifted on the way here

While assembling this piece we ran into the thesis three separate times, in our own sources.

The GSM1k accuracy drop is reported in a good deal of secondary coverage as 13%. The paper's abstract says up to 8%. We took the primary.

The University of Texas audit is described in several retellings as posted on January 31, 2016, which cannot be right, because the events it audits run through September 2016. The posting date is January 31, 2017. A year fell off in transit.

The split of the $62 million between IBM and the consulting firm that supported the project is given slightly differently across accounts, roughly $39 to $40 million to IBM and roughly $21 to $23 million to the consultancy, while the total holds steady. This is why the figure here is "at least $62 million," which is the reporting's own hedge, rather than a precise sum.

None of these drifts is scandalous. Each write-up was doing its honest best to report what a study or an audit found. That is the point. A number does not need anyone to lie about it in order to arrive somewhere false. It only needs to be copied a few times by people who did not go back to the source, and each copy is individually reasonable.

The counterweight, because the lazy version of this argument is wrong

The GSM1k paper also found that frontier models showed minimal signs of overfitting. They generalized to problems guaranteed to be absent from their training data.

That matters, and leaving it out would be the same sin this essay is about. The honest claim is not that benchmarks are theater or that measured progress is fake. Real capability gains exist and the same study that quantifies contamination also demonstrates them.

The claim is narrower and more useful: a measured improvement licenses a causal reading only when the thing you changed is the only thing that moved. When your evaluation shares a source with your training, whether that source is an institution, a market regime, or a scraped corpus, some fraction of the agreement you observe is an echo, and the score alone will not tell you how much.

The zero that was a base rate

Here is the version of this that nearly got past us this morning.

Working a research question on July 27, 2026, we wanted a data-grounded read on whether a particular kind of skilled work had been touched by AI adoption. The Anthropic Economic Index publishes a labor market impacts release with task penetration and job exposure data, so we pulled the task-level file and looked at every calibration and metrology task in it.

Every one read 0.0.

That is a striking result. Measured AI penetration into this work, zero. It would have made an excellent line, and we were most of the way to writing it.

Then we counted the rest of the file. By our count of that file, of 17,998 task statements only 1,354 carry a nonzero value. About 92.5% of the file reads 0.0. Our finding was the modal value of the dataset. We had measured the sparsity of a file and were about to report it as a fact about an occupation.

The fix was to move to the occupation-level file, where the distribution actually discriminates: by our count, a mean of 0.0770 across 756 occupations with 54.4% sitting at exactly zero. There the relevant proxies read 0.0324 and 0.0. Still low, but now meaningfully low, because there was a spread to be low against.

The zero did not change. What changed was knowing how ordinary a zero was in the population it came from.

What to compute before you believe a delta

The check that catches all four of these cases is short, and it is worth running before the number goes in a slide.

Ask what else moved during the window you measured. If the answer is anything other than "only the thing I changed," the improvement is a covariance and you should say so out loud rather than let the reader assume otherwise.

Ask what the base rate of your value is in the population it came from. An extreme reading means nothing until you know how common that reading is. A zero drawn from a file that is mostly zeros is a description of the file.

Ask what a fresh sample would say. Not a held-out split of the same collection, which shares every bias the training data has, but a matched sample sourced independently, the way GSM1k was built to mirror GSM8k without inheriting it. If getting one is impractical, that is a real constraint and worth stating. What is not acceptable is spending four years and $62 million without ever noticing that you never had one.

Three questions. None of them require new tooling, and any of them would have raised a hand somewhere in Houston in 2014.

Sources

University of Texas System Administration, special review of MD Anderson's Oncology Expert Advisor procurement (48 pages; report November 2016, results posted January 31, 2017). Scope limited to contracting, procurement and compliance; explicitly excludes project management and system development.
Matthew Herper, "MD Anderson Benches IBM Watson In Setback For Artificial Intelligence In Medicine," Forbes, February 19, 2017.
The Register, coverage of the MD Anderson audit, February 20, 2017.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati and Summer Yue (Scale AI), "A Careful Examination of Large Language Model Performance on Grade School Arithmetic," arXiv:2405.00332, submitted May 1, 2024, revised November 22, 2024. Figures quoted from the abstract: accuracy drops of up to 8%, Spearman's r² = 0.36, frontier models showing minimal signs of overfitting.
S. P. Somashekhar et al., "Watson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board," Annals of Oncology, 2018 (PMID 29324970). 638 breast cancer cases, Manipal Comprehensive Cancer Center, 2014–2016. Concordance 93%, where a recommendation counts as concordant if the tumor board's choice was designated "recommended" or "for consideration" by the system. The study included a blinded second review by the tumor board in 2016 of the cases where the two disagreed; earlier conference reporting of the same work quotes a lower pre-review figure.
ASCO 2017, Gachon University Gil Medical Center, Incheon: 525 patients treated 2012–2016 (340 colon, stage II–IV; 185 chemotherapy-naïve advanced gastric). Concordance 73% colon, 49% gastric.
Won-Suk Lee et al., "Assessing Concordance With Watson for Oncology, a Cognitive Computing Decision Support System for Colon Cancer Treatment in Korea," JCO Clinical Cancer Informatics, 2018 (doi:10.1200/CCI.17.00109, PMID 30652564). 656 patients, stage II–IV colon cancer, 2009–2016. Absolute concordance 48.9%; 65.8% (432 of 656) when cases judged acceptable are included.
Concordance Rate between Clinicians and Watson for Oncology among Patients with Advanced Gastric Cancer: Early, Real-World Experience in Korea (PMC6377977). 65 advanced gastric cancer patients, Gachon Gil Medical Center, 2016–2017. Concordance 41.5% (27 of 65) at the recommended level, 87.7% (57 of 65) at the for-consideration level.
Anthropic Economic Index, labor market impacts release: task_penetration.csv (task level) and job_exposure.csv (occupation level). The task-level and occupation-level counts given above are our own counts of those two files as accessed on July 27, 2026, not figures published by the index.

Related reading from us

Your Agent Eval Is One-Factor-at-a-Time, and Fisher Proved That's Blind. That piece is about experimental design, changing one factor at a time versus factorial designs. This one is the observational counterpart: what you can and cannot infer when you did not run an experiment at all and only have a measured delta.
The Answer Key Was in the Training Data. Direct contamination, which is the sharpest special case of the mechanism described here.
Zillow Disabled Its Human Pricing Override. Then It Wrote Down $407.9 Million. A governance failure rather than a validation failure. Different bug, comparable bill, and worth reading alongside this one precisely because the two are so often confused.

Juniors Don't Love Rust — You Just Can't Separate Age From Year From Cohort

Wed, 22 Jul 2026 17:09:35 +0000

Originally published at https://vibeagentmaking.com/blog/juniors-don-t-love-rust-you-just-can-t-separate-age-from-yea/

An APC teardown of the Stack Overflow Developer Survey. The numbers are real; the story about who was always optional, and one line of century-old arithmetic shows exactly why a survey like this can never pin it down.

In June 2022, Stack Overflow published its annual developer survey and handed the internet a headline it had printed six times before: Rust was the "most loved" programming language, for the seventh year in a row, with 87% of its users saying they wanted to keep using it. When the survey retired "loved" for the sterner "admired" in 2023, Rust just kept winning: 83% admired in 2024 ("for the second year in a row," as Stack Overflow's own write-up put it), 72% and still #1 in 2025.

You have read the think-piece this figure generates. You may have written it. It goes: young developers love Rust. Juniors chase memory safety the way their elders chased garbage collection; the kids grew up on the borrow checker; Gen Z devs are built different. It comes with a chart, the chart goes up and to the right, and the conclusion feels like it is sitting right there in the data.

Here is the uncomfortable thing I want to show you, with arithmetic: that conclusion is not in the data. Not because the sample is too small, or self-selected (though it is), or because correlation is not causation. Something sharper. The dataset, any dataset shaped like this one, is mathematically incapable of telling you whether young people drive a trend. The proof is one line long, it is about a century old in demography, and once you see it you will spot undeclared versions of it in half the trend pieces you read.

One line of arithmetic, three incompatible stories

The Stack Overflow survey is what statisticians call a repeated cross-section: every year, a fresh pile of respondents, each row carrying an age bucket and a survey year. From those two fields you can derive a third: roughly, birth year, or, closer to what tech punditry actually means, the year this person entered the field. Demographers call these three clocks age, period, and cohort, and every generational claim is a claim about which clock is doing the work:

Age effect: juniors try new things; people cool on novelty as they age. ("Juniors love Rust.")
Period effect: a moment pulled everyone in at once, regardless of age. ("2023 changed everything.")
Cohort effect: the class that entered around 2021 imprinted on the tool and will carry it forever. ("This generation is different.")

The trap is that the three clocks are not three measurements. Cohort = period − age. Exactly. Know any two and you know the third, which means a model trying to estimate all three is asking the data to split one number three ways.

This is the age-period-cohort identification problem, and it is not a folk worry; it is a theorem about the geometry of the question. Put a variable for age, a variable for survey year, and a variable for cohort into one regression and the design matrix is rank-deficient by exactly one: its columns are linearly dependent, and infinitely many different coefficient vectors reproduce the observed data identically, not approximately but identically. More respondents do not help. A bigger survey sharpens every one of the competing answers equally and leaves them exactly as tied. The ambiguity lives in the question, not the noise.

Running it for real

Claims like that deserve a demonstration, so let us run one on the survey's cleanest curve. Stack Overflow started asking about AI tools in 2023, and the topline is famous: 70% of respondents using or planning to use AI tools in 2023, 76% in 2024, 84% in 2025.

Fit the linear age-period-cohort model and ask the standard software for the answer, three different runs, three different constraint choices. Read the rows as headlines. One says the trend is all age and cohort: the generational story, +7 points a step each. One says it is all the moment: ChatGPT year, everyone at once, no generation involved. The diplomatic compromise a fancy estimator produces splits it 2.33 / 4.67 / 2.33 and looks, to the untrained eye, like a finding.

The punchline: across every cell of the grid, the three fits differ by zero. Not "within confidence intervals." Zero, to machine precision. They are the same surface wearing three stories, because the reallocation between them (add δ to the age slope, subtract δ from the period slope, add δ to the cohort slope) cancels exactly, for any δ, forever. That is the rank deficiency made flesh: four columns, rank three.

In 2004, Yang, Fu and Land published the Intrinsic Estimator in Sociological Methodology, a principled-sounding fix that uses a pseudoinverse to pick, out of the infinite family of equally-fitting answers, the unique one orthogonal to the null space of the design matrix. It has lovely statistical properties. It is also a choice: one member of the tied family, selected by a criterion that has nothing to do with how developers actually adopt tools. In 2013, Andrew Bell and Kelvyn Jones published a paper in Social Science & Medicine whose title states the thesis with admirable bluntness: "The impossibility of separating age, period and cohort effects." No estimator solves this, because the problem is inherent to the real-world process, not the statistics. The fanciest estimator is not the one that found the answer. It is the one that best disguised the assumption.

What the data can say

Within one survey year, age comparisons are fine. If 2025's 18-to-24-year-olds admire Rust more than 2025's 45-year-olds, that is an observable fact, a cross-sectional gradient, no identification problem at all. What you cannot do is attribute the drift across years to any one clock.

Curvature survives. Straight lines don't. The unidentifiable piece is exactly the shared linear trend, the smooth drift. Kinks, spikes, accelerations, the second-difference structure, are identified, because reallocating a straight line among three clocks cannot manufacture or absorb a corner. In the AI curve, 70 to 76 to 84 is +6 then +8. The acceleration, +2 points, came out identical in every fit, under every constraint. The near-vertical jump the year ChatGPT broke is real, attributable, and visibly a period shock: everyone, every age bucket, at once.

Sit with the irony of that. The one thing this dataset can actually pin down, "a moment moved everybody," is the least generational story available. The moment a narrative becomes interesting ("this cohort is different") is precisely the moment it slides into the unidentifiable linear subspace. The seductiveness and the unprovability are the same mathematical property.

And there is a fourth clock nobody models. Stack Overflow's 2024 write-up notes that respondents aged 35 and up were 31% of the sample in 2022, 35% in 2023, and 39% in 2024. This is a self-selected convenience sample that re-draws itself every year. A composition shift like that can manufacture, mask, or reverse any apparent trend on any of the three clocks, before we even reach the theorem.

The grown-up in the room already blinked

In May 2023, Pew Research Center published a methodological statement titled "How Pew Research Center Will Report on Generations Moving Forward." In it, they concede the core point: when younger adults answer differently than older ones, "it may be driven by their demographic traits rather than the fact that they belong to a particular generation." Their new house rules: generational claims require decades of comparable historical data, and where the label is not earned, they will group by decade or event instead.

A century of theory sits behind that retreat. Karl Mannheim's 1928 essay "The Problem of Generations" was already more careful than the genre it spawned: sharing birth years creates only a potential for shared consciousness, and no generation is a homogeneous block. When a survey house of Pew's stature stops writing "Gen Z believes…" headlines from single-year cross-sections, that is a detector being honest about what it can detect.

What to actually do with this

The practical insight is not "never trust surveys." It is a one-question audit you can run on any trend claim, others' or your own, in about ten seconds:

"Which clock did you zero, and where did you say so?"

Every attribution of a repeated-cross-section trend to age, generation, or moment has zeroed at least one clock. There is no exception; the algebra does not permit one. From there, three habits:

Downgrade smooth stories, respect kinks. A gradual "each cohort more than the last" drift is exactly the unattributable shape. A sharp everyone-at-once jump carries real, identifiable period signal.
Check the composition clock first. A survey whose 35+ share climbs 31 to 35 to 39 in two years can produce "trends" without a single human changing their mind.
When you must attribute, declare the assumption and defend it from outside the data. "We attribute this to cohort because switching costs lock tool choices in the first two working years" is an argument: checkable, arguable, honest.

And if you are the one writing the analysis: say the null proudly. "This survey cannot tell whether juniors drive Rust adoption" is not a failure to find a result. It is the result. The numbers are real: 87%, seven years running, 83%, 72%, 70-76-84. What is optional, what was always optional, is the story about who.

Anyone who tells you otherwise has made an assumption. The good ones tell you which.

Sources

Stack Overflow, 2022 Developer Survey — Rust "most loved" for the seventh consecutive year, 87%.
Stack Overflow Blog (Jan 2025), "Developers want more, more, more: the 2024 results" — Rust 83% admired; respondents 35+ = 31% (2022) to 35% (2023) to 39% (2024); AI use 76%.
Stack Overflow, 2025 Developer Survey — Rust most admired at 72%; 84% using or planning to use AI tools.
Stack Overflow, 2023 Developer Survey — 70% of all respondents using or planning to use AI tools.
Bell, A., & Jones, K. (2013). "The impossibility of separating age, period and cohort effects." Social Science & Medicine, 93, 163–165.
Yang, Y., Fu, W. J., & Land, K. C. (2004). "A Methodological Comparison of Age-Period-Cohort Models." Sociological Methodology, 34, 75–110.
Pew Research Center (May 2023). "How Pew Research Center Will Report on Generations Moving Forward."
Mannheim, K. (1928/1952). "The Problem of Generations."

Originally published at https://vibeagentmaking.com/blog/juniors-don-t-love-rust-you-just-can-t-separate-age-from-yea/

Is AI Coming for Data Analyst Jobs? Depends What You Actually Do All Day

Wed, 15 Jul 2026 13:11:30 +0000

Short answer: no. A bit longer answer: no, unless your whole job is just writing SQL.

Someone started a thread with a good question:

"You said frontend is dying because it's 'pure pattern-matching' that AI can now do. As a Data Analyst, I see Claude clean data, write SQL and build dashboards in seconds. The exact same pattern. Are Junior Data Analysts heading for the same C-Tier obsolescence?"

Honestly? Some of them, yeah. But not because they're analysts. Because a lot of them weren't really doing analysis to begin with.

People got confused about what the job even is

Somewhere along the way, "data analyst" turned into a fancy title for "person who writes SQL." That was never right. SQL is just the tool you use. The actual job is figuring out what's true, what matters, and what's about to blow up in a meeting if nobody catches it in time.

If your day looks like this:

"Pull last quarter's numbers"
"Clean this messy CSV"
"Make me a bar chart"

...that's not really analysis. That's data entry with extra steps. And yeah, that part is basically automated now. Claude writes that query in two seconds flat and never complains about it. No point pretending otherwise.

But that was always the junior, boring 20% of the job. People just mistook it for the whole thing.

Here's what AI still can't do

No matter how good these tools get at writing SQL, they still can't do this stuff:

Spot when a number is lying to you.
Revenue's up 12%. Nice - except pricing changed halfway through the quarter, and one big client is basically all of that "growth." The query ran fine. The conclusion is still wrong. You only catch that if you know the business well enough to get suspicious of a number that looks too good.

Decide what "active user" even means.
Someone has to actually define this. Logged in the last 30 days? Fired any event at all? Paying customer only? Get it wrong once, quietly, and two teams will spend the next year arguing about whose number is "correct," not realizing they're using different definitions.

Push back on a bad question.
A manager asks "why are we losing customers." That's not one query, it's like ten different ones, and picking the wrong one burns a week. A good analyst asks more questions before touching the keyboard. AI just answers whatever you literally typed - even if it's the wrong question, answered perfectly.

Notice the bug that gives a confident, wrong answer.
A join quietly duplicates rows and inflates a number by 40%. Nothing errors out. It just looks slightly off to someone who's been burned by this exact thing before, and they go dig into it. That gut feeling comes from experience, not from training data.

Actually own the decision.
Batch job or live streaming? Is this dashboard even worth the extra $2k a month in warehouse costs given who's actually looking at it? These are business calls wearing technical clothes. AI can list your options all day. It's not sitting in the budget meeting defending the choice.

None of that is pattern-matching. It's judgment you build by getting things wrong before, at this specific company, with this specific messy data. That's the one thing a model trained on everyone's data and belonging to no one - just can't have.

So who's actually in trouble here?

The person whose whole pitch is "I write SQL fast." If that's genuinely all you bring, you're competing with something faster, cheaper, and awake at 3am. Not a hot take, just math.

The junior who uses AI to skip the boring stuff and spends that extra time in meetings, arguing about what a metric should mean, or getting suspicious of numbers that look too clean — that person just got a huge boost. Not a threat to them at all.

The part worth actually talking about

You used to build that judgment slowly. Write a bad join, watch a dashboard break, get grilled about it in standup, learn from it. Painful, but it worked. It was basically an apprenticeship disguised as boring ticket work.

AI just ate the boring ticket work. Which means that apprenticeship juniors used to go through without even thinking about it is disappearing. Telling someone "you'll pick up the instincts eventually" isn't much help if the reps that used to build those instincts don't exist anymore.

So the real question isn't "will AI replace data analysts." It's: if the boring reps are gone, how does a junior actually build that "something's off with this number" instinct on purpose, instead of just hoping three years of grunt work hands it to them?

That one's worth an actual answer. "You'll figure it out eventually" doesn't cut it anymore.

Shoutout to @alexvoste for the original C-Tier/S-Tier take that kicked this whole conversation off. If you haven't seen his stuff - he's the guy building ForgeZero, a build toolchain in Go that's reportedly beating Ninja by a wide margin on large task sets. Worth a follow if you care about performance work at the systems level.

Not Worthless. Just Not Found Yet. - On verification work that arrives before anyone asks for it

Tue, 23 Jun 2026 08:00:00 +0000

The shape of the work right now

Zenodo views: 1. Downloads: 0. No external citations to the ledger to date.

The code still runs.

That is the uncomfortable shape of this work right now. A full Flamehaven cycle can take three weeks, sometimes six months or more: theory, repo, verification, result artifact, paper, ledger. No client asked for it. No deadline forced it. No market signal arrived to confirm that it mattered.

So the question becomes simple, and not very flattering:

Is this verification work, or just elaborate work done in a room nobody enters?

My answer is that those are different questions. Visibility asks whether people are watching. Provenance asks whether the work survives being checked later, by anyone, including a future version of us. In AI-era research, I think that second question is becoming the scarce one.

This work is not valuable because someone is watching it.

It is valuable because, when trust in AI-generated output becomes scarce, already-verifiable work becomes rare. Not worthless. Not yet found. Those are not the same thing, and the rest of this is about why the difference matters.

How one cycle actually runs

From the outside it looks like "wrote a paper." In practice, six stages chain together in sequence.

Theory — Fix the boundary of the claim before anything else: what you will assert and, just as importantly, what you will not.
Repo — Translate theory into code. The code doesn't prove the theory; it renders the theory into something falsifiable and checkable.
Verify — The harness checks whether the code is internally self-consistent and actually respects the boundary defined upstream.
Result — Numbers, pass/fail counts, degraded-pass flags, dropped into a file. Not a hand-written summary. An artifact produced by execution.
Paper — Interprets the result. Does not stand in for it.
Ledger — Timestamps and keeps the trace of the entire cycle. When someone wants to re-verify the same theory later, the repo, result, and paper from that moment are still there, unchanged.

The reason these six stages run in strict order every time is simple: skip one, and the next stage proceeds unverified. Build the repo before fixing theory, and you end up writing code without knowing what it's supposed to test. Write the paper before running verify, and the paper ends up describing what the code was supposed to do rather than what it actually did. Skip ledger, and the next cycle has no way to audit the one before it.

This scaffold isn't rebuilt from scratch for every project.

The same skeleton gets reused. QSOT2 is one instance of that skeleton applied to the quantum-channel domain.

The two-axis problem this cycle is built against

A result.json that took days to build, and nobody opens it. A claim_boundary field designed with real care, and nobody who'd recognize what it's doing. The time was real. Almost none of it registers as value from the outside.

Answering that doubt requires reframing the question. Not who's watching, but whether this is the kind of problem worth watching in the first place. The evidence splits cleanly into two axes.

The first is output contamination. In May 2026, arXiv began enforcing a one-year submission ban for papers found to contain incontrovertible evidence of unchecked generative AI use, including hallucinated references — leftover meta-commentary addressed to or from the model counts as well. [1]

The scale behind that policy is documented: the rate of papers containing at least one fabricated reference went from roughly 1 in 2,828 in 2023 to 1 in 458 in 2025 to 1 in 277 in the first seven weeks of 2026, close to a twelvefold rise. A study co-authored by arXiv founder Paul Ginsparg audited 111 million references across arXiv, bioRxiv, SSRN, and PubMed Central, and found a conservative estimate of 146,932 hallucinated citations in 2025 alone. [2]

Copy-pasted prompt residue, citations to papers that don't exist — this isn't an edge case anymore. It's a statistically detectable frequency.

The second is reproduction failure. OpenAI's PaperBench evaluated AI agents on the task of replicating 20 ICML 2024 papers from scratch, graded against 8,316 rubric items. [3] The best-performing tested agent scored 21.0%. Human ML PhDs, given 48 hours on a comparable subset, reached 41.4%.

That doesn't mean AI agents are useless for research. It means fluent generation and successful reconstruction are still very different capabilities. On this benchmark, the strongest tested agent reached roughly half the human baseline on the comparable subset, and that gap matters precisely because the output can look competent before anyone re-runs it.

What the cycle answers back with

These are two stages of the same failure. The first is that submitted output often carries contamination straight through, unchecked. The second is that even output which clears that bar doesn't reproduce when someone actually re-runs it. Embedding claim_boundary directly in code is one response. Letting DEGRADED_PASS fire automatically is another. Together, they turn the claim into something the repo has to keep surviving, not something written once and trusted forever.

This isn't a concern unique to us. The broader research ecosystem is already circling the same problem from different angles: provenance, transparent reporting, dataset documentation, model cards, correction mechanisms, reproducibility.

One academic paper argues that universities and research communities need to build provenance infrastructure that anchors AI-era scholarship, since generative AI has made it trivial to produce fluent, polished, but unsubstantiated output. [4]

Elsevier's 2025 Researcher of the Future report found that 58% of researchers already use AI tools for work, while only 22% rate current GenAI tools as trustworthy. [5][6] 85% agreed that corrections and retractions help preserve the integrity of the scholarly record. Adoption and distrust are rising together, not trading off against each other.

That is the gap Flamehaven is trying to work inside. Not as an institution. Not as a standard-setting body. Just as a small project asking one narrow question repeatedly:

Can the claim, code, result, paper, and ledger still point to each other after the attention has moved on?

Not a system. A small pond.

Looked at in isolation, QSOT2 can seem like overkill — this much scaffolding for a toy model with no new physics in it. That objection is fair. It's also the point. QSOT2 isn't presented as new physics. It's a controlled pond for testing whether the verification skeleton stays clear.

claim_boundary, the DEGRADED_PASS verdict logic, flagging patterns like riemann_is_toy — none of this is QSOT2-specific. Wrap the same harness around a different phenomenological model and the same tier of verification carries over unchanged. BioAI auditing, RExSyn-Nexus, CR-EP all share the same skeleton.

Taken one at a time, each output looks worthless. Underneath, it isn't code rebuilt from zero each time. It's a structure built once and reused continuously — the important part is that the same skeleton is being grafted onto new domains, not redesigned for each one.

Calling Flamehaven a system would overstate it.

Right now it's closer to a small pond fed by a creek. The water is clear enough that you can see the bottom, and anything happening in it can be checked by anyone who looks. Murky water isn't a default we accept. Exposing riemann_is_toy instead of quietly dropping it is the same choice: keep the water clear rather than cloud it to look more impressive.

A pond isn't a grand thing on its own. But a pond that stays clear and stable, over time, tends to attract life.

Frogs lay eggs. Pondweed takes root. Birds come to drink. None of this was designed top-down as an ecosystem from day one — it's what happens when one body of water is kept clear long enough that a small ecosystem accretes around it on its own.

What the clarity is actually worth

The pond metaphor only works if “clear” stops being a mood and becomes something an outsider can inspect. In this context, clear means fewer hidden dependencies, fewer interpretive layers, and fewer places where the paper can make the result look cleaner than the execution actually was.

That is why the technical details matter.

Machine-epsilon-level CPTP verification runs on nothing but NumPy. 31 of the 35 core checks complete using pure NumPy, no PyTorch, no GPU. This does not make the work important by itself. It makes the work easier to re-run, easier to question, and harder to hide behind infrastructure.

So QSOT2's view count and download count aren't the metric that matters here. The metric is how many other domains this skeleton has been grafted onto, and whether each graft required redesigning from scratch. That answer is already accumulating, one instance at a time.

The useful test for any of this isn't whether someone agrees with the paper.

Agreement is cheap. The useful test is whether they can clone the repo, inspect the claim boundary, re-run the verification harness, compare result.json, and see the same ledgered output that existed before they arrived. That's the level where this work wants to be judged.

Living with worthlessness, rather than defeating it

The problem is that this is not the level where the visible market usually judges work.

The visible market still rewards velocity and virality more than verification scaffolding. This isn't work the market has learned how to price yet, and that's a different problem than the work being wrong.

So the task is not to defeat worthlessness with a louder story.

It is to tolerate a state that looks worthless while refusing to cloud the water before a signal arrives confirming that judgment was wrong. Recognition arriving late and recognition never arriving are different outcomes. There's no way to distinguish between them right now. What's been available, so far, is to hold the form steady and wait.

We have been waiting.

Not for Flamehaven to be noticed. That would be the wrong story. What we were waiting for, although we could not have named it cleanly at the time, was for the pressure itself to become public: the sense that proof-shaped output can outrun disclosure, that claims can separate from the artifacts meant to hold them, and that human responsibility can become decorative after the machine has already produced something convincing.

It arrived not as recognition, but as language.

When the outside world starts naming the same problem

In June 2026, a working group of sixteen researchers from fifteen universities published the Leiden Declaration on Artificial Intelligence and Mathematics, drawn from a Lorentz Center workshop the previous September that had brought together around sixty researchers from ten countries. [7]

The declaration is now officially endorsed by the International Mathematical Union. It names, in the language of mathematics, the same family of failure modes this piece has been circling. Automated techniques can produce results that look convincing but contain almost invisible errors, difficult to distinguish from genuine mathematical proofs, putting traditional standards of correctness, transparency, and independent verifiability under pressure.

Results communicated through press releases or blog posts, ahead of peer review and without the disclosure needed for scientific evaluation, are named as a direct threat to proper evaluation.

The Declaration's first recommendation for individual researchers is to disclose tool use explicitly, in a dedicated section, and to retain exclusive human responsibility for the correctness of any automated result.

That does not mean the Leiden Declaration validates Flamehaven. It means something narrower, and more useful: the failure mode Flamehaven was built around is no longer just an internal concern. Plausible-but-unreliable automated output, weak disclosure, missing provenance, and human accountability drifting away from machine-generated claims are now being named by the mathematical community itself, with institutional backing behind it.

claim_boundary and DEGRADED_PASS are not the Declaration in code. They are one small attempt to operationalize the same family of concern at the artifact level, built before that concern reached us as a public declaration with institutional backing.

Not because anyone here anticipated that working group, but because the underlying problem was already visible at small scale, one quiet result.json at a time.

A working alignment note mapping the Flamehaven verification cycle against each of the Declaration's recommendations is now on the ledger, section by section, including the places where the cycle currently falls short of what the Declaration asks for. [8]

This kind of naming has happened before

Once a field begins naming its own failure mode, the next question is not whether the language is elegant. The question is whether anyone changes behavior before an external authority forces them to.

That is why the historical comparison matters.

In 1975, recombinant DNA researchers gathered at Asilomar after parts of the field had already entered a voluntary pause, worried that the technology had outpaced anyone's ability to assess its risk. [9]

Roughly 150 scientists debated guidelines from scratch over four days, and the NIH later used those recommendations as a template for its official rules on recombinant DNA research, shaping biosafety regulation for decades after.

The point is not that AI mathematics is recombinant DNA. The point is narrower: sometimes a field begins to govern itself before formal institutions know what to do with the risk.

Asilomar did not begin as regulation. It began as scientists deciding that some kinds of power should not be allowed to outrun the practices that make them inspectable.

Small ponds, one current

This doesn't make the pond important on its own.

It changes something smaller, but more useful: it changes what the pond can now be compared against.

A year ago, a verification harness around toy quantum models could look like private overbuilding — too much claim_boundary, too many execution artifacts, too much ledger discipline for work almost nobody had asked to inspect.

After the Leiden Declaration, the same habit sits inside a larger problem statement. The question is no longer only whether this particular pond matters. It is whether research claims, once AI makes them cheap to produce, can remain tied to artifacts sturdy enough to inspect.

The Declaration is a mathematics-community document, not a software-verification framework. That boundary matters. Still, its core concern — automated output outrunning disclosure, review, and human responsibility — is the same pressure Flamehaven has been trying to make operational at the artifact level.

Asilomar named a related pattern fifty years earlier, in a different field, with a different danger at stake: a technical capability began moving faster than the practices built to govern it, and the field responded before formal institutions had settled the rules.

That is the only comparison I want to make.

A declaration names the pressure from above. A ledger answers from below by keeping each claim tied to the code, result, paper, and timestamped record that supported it. Neither changes the whole field alone. But a watershed was never one current to begin with. It is what becomes visible after enough separate sources of clean water keep draining toward the same low ground.

The trajectory still splits two ways. The trust gap around AI-generated research provenance could close on its own, market-driven and indifferent to anything built here. Or this kind of self-imposed, bottom-up correction — scientists and builders setting their own terms before regulation arrives — could become one of the ways it closes.

Which branch is correct can't be known from here. The second branch is no longer just a hope, though. It now has a name, an institutional endorsement, a publication date, and a fifty-year precedent showing that field-led restraint can shape formal governance.

That still does not make the work valuable in the market yet. The code still runs. The ledger still holds. The result still reproduces.

We are not the ocean. We are not even the current. But a pond does not need to know it is part of a watershed to matter. It only has to stay clear, drain in the same direction as other clear water, and let the geography do the rest.

References

[1] Nature, "arXiv to ban researchers for a year if they don't disclose AI use," May 2026

[2] Ginsparg et al., "Hallucinated References in Preprint Servers," arXiv, 2026

[3] OpenAI, "PaperBench: Evaluating AI's Ability to Replicate AI Research"

[4] "Provenance Infrastructure for AI-Era Scholarship," arXiv, 2026

[5] Elsevier, "Shaping the Researcher of the Future: Confidence, Collaboration and AI," 2025

[6] Elsevier, "Researcher of the Future — Confidence in Research," 2025

[7] Leiden Declaration on Artificial Intelligence and Mathematics, June 2026

[7a] Leiden University, "Leiden Declaration: AI is challenging the core values of mathematics," June 2026

[8] Flamehaven Verification Ledger, "Leiden Alignment Note"

[9] Embryo Project Encyclopedia, "Asilomar Conference (1975)"

The Data Science Behind Marketing Attribution: Moving Beyond Client-Side Correlation

Sun, 21 Jun 2026 03:58:06 +0000

As browser privacy controls and ad-blockers tighten globally, developers and data engineers face a growing bottleneck: client-side event tracking is becoming increasingly unreliable.

The traditional way to measure marketing performance relies on third-party tracking scripts, front-end pixels, or heavy SDKs embedded into the browser. When a conversion occurs, the client-side script parses the current session source and credits it with the sale. This structure usually implements a "last-click" attribution framework.

The Engineering Flaw of Last-Click Data

From an architectural standpoint, last-click tracking reduces a multi-touchpoint user journey down to a single relational database entry.

Consider this standard pipeline sequence:

User interacts with a top-of-funnel ad via a social media platform.
User enters an email marketing loop and clicks a mid-funnel link.
User runs a quick direct brand search on Google to execute the transaction.

In a last-click data model, the brand search capturing the transaction gets 100% of the conversion credit. The discovery ad and email nurture nodes get zero. This structural blind spot causes a massive misallocation of budget toward bottom-funnel channels while starving the systems that generate baseline demand.

Solving Inaccuracies with Backend Causal Inference

The alternative to flawed client-side correlation is server-side causal inference. Instead of evaluating isolated browser events as they execute, the logic shifts entirely to the backend, analyzing complete historical data records.

By processing standardized exports (such as a standard Google Analytics CSV export), server-side algorithms can run independent multi-touchpoint analysis without client-side execution overhead.

Engineered platforms in this space, like Causality Engine, approach this mathematically rather than using arbitrary linear rules. By moving processing to a secure backend, it bypasses browser script blockers entirely. This architecture successfully surfaces 15% to 25% of hidden conversion revenue that standard tracking scripts fail to capture, outputting the exact value metrics per channel.

Architectural Benefits: Performance & Compliance

Shifting your analytics stack from front-end pixels to server-side file processing tackles two core infrastructure requirements:

Site Reliability Engineering: Eliminating external script execution on the frontend improves core web vitals and eliminates page latency.
Compliance by Design: Processing data purely via historical logs enables deterministic data privacy. Platforms can maintain fully EU-hosted environments with native data anonymization, strict right-to-erasure workflows, and integrated fraud detection layers without complicating front-end consent scripts.

How is your engineering team adapting data architectures to handle multi-touch attribution as browser cookie restrictions expand? Let's talk in the comments.

Why Statistics is Important in Data Science

Sat, 20 Jun 2026 08:52:14 +0000

Introduction

Ask ten people what data science is and you'll hear things like Python,machine learning and building dashboards.Almost nobody says "statistics."But statistics is the thing doing the real work underneath all of it.It's what tells you whether the real work actually means anything.
Simple Example:

I analyzed a CSV dataset of Nairobi rental listings.The data includes monthly rent,property type,bedrooms,bathrooms,floor size and distance to the CBD.
Imagine you're looking at rental prices in Nairobi.You open your spreadsheet and see thousands of numbers staring back at you. Where do you even begin?
Statistics gives you a starting point.First,you need to understand what kind of data you're actually dealing with.Some numbers like rent can be anything - 150,000 KES,287,500 KES you name it.But other numbers like bedrooms are whole things - you can have 1 bedroom,2 bedrooms or 3 bedrooms.You can't have 2.5 bedrooms.
Some of your data isn't even numbers.Estates like Westlands or Kilimani are just names(text).There's no mathematical order to them.Westlands isn't "bigger" or "better" than Kilimani in any way that fits into a formula.
This sounds obvious but if you get this wrong,everything else breaks.You might try to average something that should be counted.Or you might use the wrong test on the wrong kind of data and end up with completely meaningless results.
1.What the numbers actually tell you

Once you understand what you're working with,simple statistics start telling you a real story.
Take the average rent in the data - 223,196 KES.But here's something interesting.The middle value - the price you get when you line up every single listing from cheapest to most expensive and pick the one in the exact middle is 190,307 KES.
That gap between the average and the middle? It's a clue.It means a small number of really expensive properties are pulling the average upward.If you only looked at the average,you'd think rent in Nairobi is higher than what most people actually pay.The middle value gives you a much more honest picture.
The spread of prices matters too.Most rents fall somewhere between 129,188 KES and 287,698 KES.Knowing this range helps you understand what normal market prices look like.You can also spot outliers - those properties that are way too expensive or surprisingly cheap using a simple rule instead of just guessing.
2.The shape of your data matters more(fun part)

This is where things get interesting.The rent data leans to one side - lots of cheap and mid-range listings,with a few really expensive ones stretching things out.
Why does this matter? Because many common tools like basic regression models work best on data that spreads out evenly around the middle.If you ignore how your data is shaped and just run the numbers anyway,your results will be slightly off in ways that are really hard to catch later.
3.Central Limit Theorem

There's something called the Central Limit Theorem that explains a deeper reason this matters.The simple version is this;even if your raw data is completely lopsided,if you take enough samples and look at their averages,those averages will start to form a nice normal pattern.This one idea is the reason most of the math behind confidence intervals and statistical tests actually works in the real world.Without it,a lot of what data scientists do every day wouldn't really hold up.
4.The question nobody asks

Here's something that gets skipped over way too often; you almost never have all the data.You have a sample - a small piece of a much bigger picture.This Nairobi data isn't every single rental in the city.It's just a slice.
So the real question isn't "what's the average rent in this dataset?" The real question is "based on this small slice,what can I honestly say about rent across all of Nairobi?"
This is exactly what statistics is built for.A confidence interval lets you say something like "we're 95% sure the real average rent across Nairobi is somewhere between 215,000 and 231,000 KES."That's much more honest than pretending you know the exact number.
5.Are two things really different or does it look that way?

Statistics also gives you a way to test if two things are genuinely different or if it's just random chance in your sample.

Say you want to know if rent in Kilimani is really higher than rent in Westlands.You start by assuming there's no real difference.Then you run a test and check how surprising your result would be if that assumption were true.
This same basic idea of testing if something is real or just luck is what companies use when they test a new app feature or pricing page on some users before rolling it out to everyone.
In this step,you'll end up making big claims based on tiny samples and random chance.That's not a rare mistake.It's probably the most common way data analysis goes wrong.
6.Just because two things move together doesn't mean one causes the other

Properties further from the city center tend to be cheaper.Easy to see,easy to plot on a chart.But just because two things move together doesn't mean one is causing the other.
Maybe being far from town really does lower the price.But maybe both things are caused by something else entirely like less infrastructure in those areas.
Mixing these two up; "these move together" and "one causes the other" - is one of the most common mistakes people make with data.Statistics trains you to slow down and not jump to that conclusion.

You can build models that predict rent using several things at once;size,bedrooms,distance from town.But even then,the model only shows you connections.It doesn't prove what's causing what.
7.The traps that catch people who skip statistics

Sometimes a pattern that's true for every small group disappears or even flips when you combine all the groups together.Sometimes if you test enough things,you'll find something that looks "significant" just by random chance even if nothing real is going on.Sometimes your data quietly leaves things out without you noticing, like a dataset of current listings missing all the properties that got rented out fast which probably wasn't random at all.
None of these issues show up as error messages in your code.They show up as wrong answers that sound confident.And that's worse,because nothing tells you to double check.
8.Why all this matters for machine learning

Every machine learning model is really just a statistics model wearing different clothes.Knowing how spread out your data is and what shape it takes helps you pick the right model and notice when something's wrong.
Testing a model on data it hasn't seen before,which is standard practice - is really just using a sample to guess at the bigger picture.Picking good inputs for your model gets much easier once you understand which numbers actually carry useful information and which just look interesting on a graph.
Conclusion

Statistics is what stands between "I noticed a pattern" and "this pattern actually means something." It's the difference between reporting an average and knowing when that average is misleading you.Between seeing two things move together and knowing if it's worth acting on.Between running one test and understanding why ten tests could have fooled you.

Data science without statistics isn't a simpler version of data science.It's just guessing and dressed up with a nicer chart.

Statistics is what decides which one you end up with.Either to write a confident report full of wrong conclusions Or it could be used to write an honest one,with the uncertainty clearly laid out

Start with the basics.Ask yourself what your data actually is,what the center looks like and how spread out it is.Getting those right will save you from getting everything else wrong.
I hope you found this artice usefull.

How I chose the 8 metrics behind a stock quality score

Fri, 12 Jun 2026 13:39:58 +0000

"Quality" is a vague word. Every investor agrees a quality company is good, and nobody agrees on how to measure it. When I built a free stock analyzer, I had to turn that vague word into a number between 0 and 100, which meant making real decisions about which metrics actually capture quality and which ones just look smart on a dashboard.

Here's how I landed on the eight I use, and the reasoning (and trade-offs) behind each.

The framework: four questions, two metrics each

I didn't want a soup of 30 ratios. More metrics feels rigorous but dilutes the signal, every weak metric you add drags the strong ones toward noise. So I forced myself down to four questions a quality investor actually asks, with two metrics each:

Is it cheap enough? (P/E, P/FCF)
Is it growing? (Revenue CAGR, FCF CAGR)
Is it well run? (Operating margin trend, ROIC)
Is it financially sound? (Share dilution, net debt)

Eight metrics. Each one earns its place by answering a question the others don't.

Why these specific eight

P/E and P/FCF (valuation). P/E is the obvious one, but earnings can be massaged. So I pair it with P/FCF, which is harder to fake, free cash flow is what's left after the business actually spends what it needs to. A company that looks cheap on earnings but expensive on cash flow is waving a flag. I weight P/FCF higher than P/E (15% vs 10%) for exactly this reason.

Revenue CAGR and FCF CAGR (growth). Revenue growth shows demand. FCF growth shows that the growth is translating into real cash, not just bookings. The interesting edge case: a company that recently flipped from burning cash to generating it. A raw CAGR there is meaningless or negative, so instead of punishing it, I detect the turnaround and give it a neutral-positive score. Growth metrics shouldn't penalize a company for the moment it starts working.

Operating margin trend and ROIC (profitability). I deliberately use the margin trend, not the absolute level. A 40% margin tells you the past; a margin expanding 3 points tells you the business is getting stronger right now. ROIC is the single best "is this a good business" number I know, it asks whether the company earns more than its cost of capital. I set the passing bar at 8% (roughly cost of capital) and reward 15%+ heavily.

Share dilution and net debt (financial soundness). Dilution is the quiet killer of shareholder returns, a company can grow revenue 10% a year and still erode your stake by printing shares. I score the change in share count, rewarding buybacks and punishing heavy issuance. For debt, the key decision was to measure net debt relative to free cash flow, not in absolute dollars. $47B of debt is terrifying for a small company and trivial for one generating $70B in FCF. A net-debt/FCF ratio captures serviceability, which is what actually matters.

The decision that mattered most: missing data scores zero

This is the one I went back and forth on. When a metric is missing or negative, do you skip it and redistribute the weight, or score it zero?

I chose zero, with full weight. Here's why: if missing data gets excused, a company can hide its weakest dimension simply by not reporting it cleanly, and its score floats up undeservedly. Scoring zero keeps the system honest. A company with no positive earnings should lose the P/E points. The score should reflect what the company actually demonstrates, not what it conveniently omits.

Why one scorecard wasn't enough

The eight standard metrics break down for some sectors, and pretending they don't is how most simple scoring tools quietly produce garbage.

A bank has no "operating margin" in the normal sense, and its debt isn't a liability to minimize, it's raw material. So banks get their own scorecard: P/B instead of P/FCF, ROE instead of ROIC, book-value growth, and capital adequacy (equity/assets) instead of net debt.

REITs are stranger still. Their reported earnings are crushed by depreciation that doesn't reflect economic reality, so P/E is useless. The industry uses Funds From Operations (FFO = net income + depreciation), so REITs get a P/FFO multiple, FFO growth, dividend coverage, debt/EBITDA, and interest coverage. Forcing a REIT through a standard P/E scorecard would mark a healthy one as garbage.

The lesson: a quality score is only as good as its willingness to admit that "quality" means different things in different sectors.

The result

All of this is live in Intrinsiqq, a free stock analyzer that scores 8,000+ US companies across these dimensions, with the full per-metric breakdown and methodology visible for every stock. Core is free, no signup.

If you've built scoring systems yourself, I'd genuinely love to hear where you'd weight things differently, the threshold choices are the part I still tweak most. And if you run a company you know well through it and a score feels wrong, tell me, that feedback is exactly how the methodology gets sharper.

Attribution & Forecasting: Uncovering Insights & Predicting the Future Like a Pro!

Tue, 24 Jun 2025 04:30:18 +0000

✨ Attribution & Forecasting: Uncovering Insights & Predicting the Future Like a Pro!

In the age of data-driven decisions, understanding where your results come from (Attribution) and predicting what’s next (Forecasting) are your ultimate power moves! Whether you run a business, manage ads, or analyze trends — mastering these will help you win big.

Let’s break it down: What, Why, How, and the Best Ways to Do It — with algorithms and examples!

What is Attribution?

Attribution means figuring out which touchpoints, channels, or actions deserve credit for a desired outcome (like a sale or sign-up).

Example:
Imagine you run an online store. A customer sees your Google ad, reads your blog, then clicks your Instagram post before buying. Which channel deserves the credit? Attribution answers this!

Why Do We Need Attribution?

✅ Better Budgeting: Know which channels work best and invest more wisely.
✅ Boost ROI: Stop wasting money on low-impact channels.
✅ Customer Journey Insights: Understand how people interact with your brand at each step.

⚙️ How Do We Do Attribution?

There are several models — from simple to advanced:

1️⃣ Rule-Based Models:

First Touch Attribution: 100% credit to the first interaction.
Last Touch Attribution: 100% credit to the last interaction.
Linear Attribution: Equal credit to all touchpoints.
Time Decay: More credit to recent touchpoints.

When to use: Small businesses or when you need quick insights without heavy data crunching.

2️⃣ Algorithmic / Data-Driven Models:

Advanced methods use data science to assign credit more accurately.

✨ Popular Algorithm: Markov Chain Attribution

How it works: It treats each touchpoint as a state and calculates the probability that removing a touchpoint affects conversions.

Example:
If removing Instagram causes a big drop in sales, Instagram gets high credit!

Best For: Medium to large businesses with lots of customer data.

What is Forecasting?

Forecasting predicts future trends based on historical data. From sales to website traffic to stock prices — forecasting helps you plan ahead. ️

Why Do We Need Forecasting?

✅ Demand Planning: Avoid stockouts or overstocking.
✅ Revenue Prediction: Plan budgets and growth.
✅ Resource Allocation: Allocate manpower and money efficiently.

How Do We Do Forecasting?

There are tons of methods — choose based on your data size and goal.

✅ Best Algorithms for Forecasting:

1️⃣ ARIMA (AutoRegressive Integrated Moving Average)

Great for time series data with trends & seasonality.
Example: Monthly sales prediction.

2️⃣ Exponential Smoothing (ETS)

Smooths out fluctuations, good for stable trends.

3️⃣ Prophet by Facebook

Handles holidays & seasonality well, easy to use.

4️⃣ Machine Learning Methods:

XGBoost Regression
LSTM (Long Short-Term Memory Neural Networks) for deep learning with complex patterns.

The Secret Sauce: Combining Attribution & Forecasting

Pro Tip: Use attribution insights to build better forecasts!

Example:
If attribution shows Instagram drives 40% of your sales, and you forecast sales will double during the holiday season — you can plan a bigger Instagram budget in advance!

✅ Best Practices for Precise Results

✨ Collect clean, reliable data.
✨ Use multiple models and compare results.
✨ Regularly update your models with new data.
✨ Visualize results for easy decision-making.

Let’s See an End-to-End Example

Business: Online Shoe Store

Attribution: Use Markov Chain to find that Instagram & Google Ads are key drivers.
Forecast: Use ARIMA to predict next quarter’s sales based on seasonality and trends.
Action: Increase Instagram ads budget before peak season to maximize ROI.

Result: Smarter spending, higher sales, and no surprises!

Wrapping Up

Attribution = Who gets the credit?
Forecasting = What’s coming next?

Master these, and you’re not just analyzing the past — you’re shaping the future!

✅ What’s Next?

Ready to supercharge your data strategy?
Start small, test models, visualize results, and make smarter decisions every day.

Feel free to share this blog if you found it useful!
Got questions? Drop them in the comments — let’s decode data together! ✨