Coder Legion - Recent posts tagged bioinformatics

qodon - a CLI based genomics tool

Thu, 24 Sep 2026 10:22:41 +0000

qodon (qdn) 🧬

A Command Line Genomics tool. Written in C

qodon (invoked as qdn) is an ultra-fast, lightweight command-line utility for quick genomic sequence manipulations. Written in pure C, it processes sequence arguments natively at bare-metal speeds, making it perfect for rapid testing, scripting, and pipeline integration.

✨ Features

Zero Overhead: Built in standard C for maximum execution velocity.
Dual-Mode Engine: Run quick one-off commands from your terminal or drop into a stateful interactive shell context.
Instant Evaluation: High-performance byte manipulation for zero-lag sequence transformations.
Core Bio-Utilities: Fast sequence transcription, complementing, reversing, and metric calculations.

🚀 Installation

You can install qodon directly or build from the source code.

1. Direct Install (Linux, x86_64)

Install precompiled binaries by running:

curl -L -o qdn https://github.com/Temi-Tade/qodon/releases/download/v0.2.0/qdn && chmod +x qdn && sudo mv qdn /usr/local/bin/

2. Build from source

Compile qdn natively using any modern C compiler (gcc or clang required):

# Clone the repository
git clone https://github.com/Temi-Tade/qodon.git
cd qodon

# Compile the binary with optimizations
gcc src/main.c -O3 src/lib/qodon.c src/lib/shell.c -o bin/qdn

# Move to your local bin to run it globally
sudo mv qdn /usr/local/bin/

Usage

🐚 Mode 1: Interactive Context Shell

Launch qdn with the -i flag followed by a target sequence to open the interactive context shell. The sequence is loaded directly into the session's workspace memory, allowing you to run rapid back-to-back operations against that specific strand without retyping it.

$ qdn -i ATGCATGC
Entering qodon shell...
Template sequence: ATGCATGC 
Length: 8bps 
GC content: 50.00%
qdn> len
8bps
qdn> cmp
ATGCATGC (Template strand)
||||||||
TACGTACG (Complementary strand)
qdn> gc
50.00%
qdn> rev
ATGCATGC (FWD)
CGTACGTA (REV)
qdn> exit
$

Available Shell Commands

Inside the shell, execute operations directly on the initialized sequence:

Command	Action	Description
cmp	Complement	Generates the complementary DNA/RNA strand
tr	Transcribe	Transcribes DNA to RNA (converts T to U)
rt	Reverse Transcribe	Reverse transcribes RNA to DNA (converts U to T)
rev	Reverse	Reverses the sequence direction
len	Length	Computes the total length of the sequence
gc	GC-Content	Calculates the exact GC content percentage
exit	Exit	Closes the interactive shell session

🛠️ Mode 2: Quick Terminal Flags

For quick one-offs or integration into larger bash scripts, pass an operational flag followed by your target nucleotide sequence string directly to your terminal:

qdn [flag] <SEQUENCE>

Core Flag Operations

Flag	Shell Command	Action	Description
-c	cmp	Complement	Generates the complementary DNA/RNA strand
-t	tr	Transcribe	Transcribes DNA to RNA (converts T to U)
-u	rt	Reverse Transcribe	Reverse transcribes RNA to DNA (converts U to T)
-l	len	Length	Computes the total length of the sequence
-g	gc	GC-Content	Calculates the GC content percentage
-i		Shell Mode	Enter interactive shell

💻 One-off Examples

Get Complementary Strand:

qdn -c ATGC
# Output: TACG

Transcribe DNA to RNA:

qdn -t GATTACA
# Output: CUAAUGU

Reverse Transcribe RNA to DNA

qdn -u UUAGC
# Output: AATCG

Calculate GC Content:

qdn -g CCGGTT
# Output: 66.67%

Get Sequence Length:

qdn -l ATGCATGC
# Output: 8bps

📄 License

This project is licensed under the MIT License.

Predicting Drug-Drug Interactions with Hypergraphs — on Two CPU Cores

Sun, 13 Sep 2026 18:32:27 +0000

When a patient takes several medications at once, two of them can interact in ways that cause real harm. DrugBank catalogues 86 distinct mechanisms for how this happens — one drug slowing another's metabolism, increasing its toxicity, blunting its effect, and so on. Screening every pair by hand does not scale.

Most machine learning approaches to this problem split it in two: one model decides whether two drugs interact, a completely separate one decides how. That split is convenient for benchmarking and awkward for anything practical.

We built H²GNN to do both in one hierarchy, using nothing but SMILES strings as input. The code is now public, and the whole thing runs on two CPU cores.

Code: github.com/Hu8MA/HHGNN
Paper: ECAI 2026, IEEE Xplore

Why hypergraphs

In an ordinary graph, an edge joins exactly two nodes. That is a poor fit for a lot of chemistry. A molecular substructure is not a relationship between two drugs — it is something that dozens of drugs share simultaneously. An interaction mechanism is not a property of one pair; it is a category that hundreds of pairs fall into.

A hyperedge can connect any number of nodes at once, which lines up with both of those facts. H²GNN uses two hypergraphs stacked on top of each other.

Stage 1: the chemical hypergraph

Take a SMILES string and slide a window of size k across it. Each window is a substring — a rough proxy for a local chemical arrangement. Every distinct substring becomes a node, and every drug becomes a hyperedge connecting the substrings it contains.

At k = 9, that gives 29,462 substructure nodes, 1,709 drug hyperedges, and 96,656 connections between them.

A single hypergraph attention layer runs over this structure. Attention flows at the hyperedge level, then the node level, and the result is a 128-dimensional embedding per drug learned end to end from raw SMILES. An MLP decoder takes the embeddings of two drugs and predicts whether they interact.

Result: 98.46% ROC-AUC, 93.20% accuracy.

For comparison, a Random Forest on 4,096-dimensional Morgan fingerprints gets 88% ROC-AUC. A two-layer GCN gets 95%.

Stage 2: the interaction-type hypergraph

Here is the part I find most interesting, and it is a small conceptual shift with a large effect.

The obvious way to classify interaction mechanisms is to attach a label to each drug pair and train a classifier. We did something else: each of the 86 interaction types becomes a hyperedge, connecting every drug that participates in that type. Drugs are the nodes. The type stops being a label on the data and becomes an object in the graph with its own learned representation.

This hypergraph is tiny — 1,709 drug nodes, 86 type hyperedges, 13,486 connections. Hyperedge sizes range wildly: type 42 touches 5 drugs, type 49 touches 1,308.

Drug nodes are initialized with the embeddings transferred from Stage 1, not randomly. Attention then flows hyperedge → node → hyperedge, so the type hyperedges — which are the prediction target — end up holding the final aggregated representation.

Result: 85.73% Top-1 and 98.11% Top-3 accuracy across 86 classes.

That twelve-point gap between Top-1 and Top-3 is itself useful. The correct mechanism is almost always among the top three even when it is not ranked first, which is exactly what you want if a clinician is reading the output rather than a downstream script.

The ablations that mattered

Two experiments did more to convince me the design was right than any headline number.

Attention direction. Stage 2's drug nodes start out well-informed, carrying transferred chemical knowledge, while the type hyperedges start from one-hot vectors. So it seemed plausible that flowing attention from the informed side — node → hyperedge → node — would work better.

It does not. It falls apart:

Configuration	Top-1 Acc.	PR-AUC	Macro F1
hyperedge → node → hyperedge	85.73%	87.24%	81%
node → hyperedge → node	67.42%	63.54%	48%

Aligning the attention flow with the prediction target beats starting from the richer initialization, and the rare classes are the first thing you lose. The type hyperedge has to be where aggregation ends.

Embedding transfer. Replace the transferred Stage 1 embeddings with an all-ones vector and hold everything else fixed:

Initialization	Top-1 Acc.	PR-AUC	Weighted F1
Stage 1 embeddings	85.73%	87.24%	86%
All-ones vector	82.52%	85.70%	82%

A 3.21-point drop. The hierarchy earns its keep.

There is also a third result I did not expect. During cleaning, 406 drug pairs turned out to carry more than one interaction type; we kept the first and set the others aside. The model never saw those secondary labels for those pairs. It surfaced 94.7% of them within its Top-3 predictions anyway — meaning the hypergraph learned something about multi-mechanistic pharmacology rather than memorizing one label per pair.

The efficiency story

This is the part that surprised me most, and it is worth spelling out for anyone who assumes graph learning implies a GPU budget.

Everything above was trained and evaluated on Google Colab's free tier: two vCPUs, no GPU. We chose it deliberately, for two reasons. It gives every run an identical environment, and its resource ceiling is close to the hardware actually available in a hospital pharmacy.

Model	Training time	Hardware
Interaction-Type HGNN (ours, 1 layer)	18.6 min	CPU, 2 cores
GCN (2 layers)	75.6 min	GPU (T4)
GraphSAGE (2 layers)	148.5 min	GPU (T4)
XGBoost	225.6 min	CPU

Four to twelve times faster — and it still beat GraphSAGE, the strongest baseline, by 4.8 points of Top-1 accuracy and 22 points of macro F1.

The baselines had advantages we did not give ourselves: precomputed Morgan fingerprints as input features, and inverse-frequency class weighting during training to stay stable under the severe class imbalance. Single-layer variants of the GCN and GraphSAGE did not converge at all; both had to be deepened and moved to GPU for the comparison to run. A Graph Attention Network never converged and was dropped.

Peak RAM increase during training: 0.29 GB.

What is in the repo

The dataset is DrugBank — 1,709 drugs, 191,877 unique interaction pairs, 86 types, built by merging two benchmark sources so that every drug has a SMILES string.

Everything needed to reproduce the numbers is there:

Colab notebooks for all 48 experiments, with inline outputs
Pre-computed k-mer files for k = 3, 6, 9, 12
Hypergraph structure files and index maps
The exact seeded train/validation/test partitions, verified for zero leakage
Baseline notebooks (GCN, GAT, GraphSAGE, XGBoost, Random Forest)

Open a notebook, point it at the data files, run all cells. Free tier is enough.

What it does not do

Being honest about the boundaries, since these are the interesting problems:

SMILES only. No protein-target, pathway, or side-effect information. Chemical structure carries the entire representation. Additional modalities could enter as further hyperedge families.
No severity prediction. The model reports the mechanism and its likelihood, not how bad the reaction will be — which is often the thing that actually drives a clinical decision.
1,709 drugs. DrugBank now holds over 4,000, so plenty of approved drugs cannot yet be screened.
Pairs only. Real polypharmacy is not pairwise. The hypergraph formulation could express simultaneous multi-drug combinations directly; we just have not done it.

That last one bothers me the most, because the machinery is already there. A hyperedge over three co-administered drugs is a perfectly natural object in this framework.

If any of these interest you, contributions are welcome — there is a contributing guide in the repo, and the issue tracker is open.

Code: github.com/Hu8MA/HHGNN
Paper: Hierarchical Hypergraph Neural Networks for Multi-Type Drug–Drug Interaction Prediction, ECAI 2026, IEEE Xplore

Thanks to the DrugBank Foundation for access to the curated drug data that made the dataset complete.

Open-Sourcing a 6-Tool Bio-Robotics & Bionics MCP Server (v1.0.0)

Thu, 16 Jul 2026 05:53:56 +0000

We just officially open-sourced seosiri-biorobotics v1.0.0, a stateless, local-first Model Context Protocol (MCP) server designed to bridge the integration gap between biological datasets and physical robotic actuators.

🛠️ The Architecture: Decoupling Intent from Actuation

Standard laboratory automation is heavily fragmented. Biologists use genomic databases, while mechanical engineers program robotics via proprietary vendor-locked scripts.

Our system decouples these layers completely:
[Raw Bio-Data / EMG Input] ──► [MCP Server] ──► [Deterministic Math Core] ──► [G-code Serial Stream]

The Context Broker (MCP): Connects to the live UniProt REST API, with a local-first SQLite database cache (local_db.py) to ensure offline autonomy and bypass network latency.
The Math Core (core_math.py): Converts biological parameters to metric displacement vectors in SI units (meters).
The Hardware Gateway (hardware_gateway.py): Translates coordinates into standard Cartesian G-code (G1 X[Value] F1500) used globally by CNC gantries and motor controllers.

📦 The 6-Tool Suite (Live-Tested)

fetch_genomic_data: Queries sequence parameters (with automated SQLite database caching).
resolve_biotech_spatial_intent: Translates concentration inputs into metric space coordinates.
map_plate_coordinate: Maps alphanumeric well IDs to standard 96-well plate offsets in millimeters.
calculate_pipetting_speed: Calibrates feedrates and delays based on fluid viscosity (cP).
calculate_dna_melting_temp: Evaluates GC-content and melting temperature (Tm) to recommend deck-bed heating offsets.
translate_emg_to_actuation: Translates human surface EMG microvolt biosignals into safe joint angles with active spasm protection.

🚀 Try It (Zero Install Required)

The project is officially listed on the Glama.ai directory, with verified A-grade license and maintenance scores. You can run the entire 6-tool suite inside their web-based sandbox inspector directly, with no local setup required.

Read our deep-dive architectural analysis and access the open-source codebase:
👉 https://www.seosiri.com/2026/07/seosiri-bio-robotics-core-engine.html

We would highly appreciate your technical feedback, PRs, and system reviews!

How I Redesigned a Thermostable Enzyme with ProteinMPNN, Validated with AlphaFold2

Sat, 04 Jul 2026 22:34:22 +0000

E155 and E215 at 6.03 Å - exactly the nucleophile/acid-base separation
expected for a GH5 retaining endoglucanase.

This step matters. If I had accepted the literature numbering without checking, I would have fixed the wrong residues and the constrained run would be biologically
meaningless.

Step 2: B-factor profile

Before running ProteinMPNN, I computed per-residue B-factors to identify
flexibility hotspots - regions that might benefit most from redesign:

High-flexibility regions (B > 30 Å²):

Residues 246–250: B-factor up to 96.85 Å² - extremely mobile surface loop
N-terminus (residues 1–3)
Loop 169–173

These are the regions where ProteinMPNN has the most freedom and where
thermostabilising mutations would be most impactful in a follow-up study.

Step 3: ProteinMPNN - unconstrained run

First I ran ProteinMPNN without any constraints to understand what sequences it naturally prefers for this backbone:

python protein_mpnn_run.py \
    --pdb_path cel5a_clean.pdb \
    --out_folder mpnn_output/temp_0.1 \
    --num_seq_per_target 100 \
    --sampling_temp 0.1 \
    --seed 42

Five temperatures (0.1, 0.2, 0.3, 0.5, 0.8), 100 sequences each = 500 total.

The result was surprising: at T=0.1, ProteinMPNN placed Threonine at E155
(93/100 sequences) and Alanine at E215 (97/100 sequences). Both catalytic glutamates were replaced with non-catalytic residues.

This is not a bug - it's the correct behaviour. ProteinMPNN optimises for backbone fit, not biological function. Threonine and alanine may pack better against the local structure than glutamate, but they cannot perform the retaining mechanism. This finding motivates the constrained run.

Step 4: ProteinMPNN - catalytic-constrained run

Fix E155 and E215, let everything else vary:

fixed_positions = {
    "cel5a_clean": {
        "A": [155, 215]
    }
}
with open("fixed_positions.jsonl", "w") as f:
    f.write(json.dumps(fixed_positions) + "\n")

python protein_mpnn_run.py \
    --pdb_path cel5a_clean.pdb \
    --fixed_positions_jsonl fixed_positions.jsonl \
    --num_seq_per_target 100 \
    --sampling_temp 0.1

Results:

Temperature	Mean score	Mean recovery	E155 preserved	E215 preserved
0.1	0.763	52.6%	100%	100%
0.2	0.788	52.1%	100%	100%
0.3	0.830	51.5%	100%	100%
0.5	0.983	49.1%	100%	100%
0.8	1.369	43.1%	100%	100%

100% catalytic preservation at all temperatures. And the score distributions are nearly identical to the unconstrained run - fixing 2 out of 605 residues costs essentially nothing in terms of backbone fit.

Step 5: AlphaFold2 validation

Top 20 constrained designs (lowest MPNN score, T=0.1) were folded using
ColabFold on a free T4 GPU:

colabfold_batch \
  top20_constrained_candidates.fasta \
  af2_structures/designs \
  --model-type alphafold2_ptm \
  --num-recycle 3 \
  --msa-mode single_sequence

Also folded the wildtype under identical conditions as a baseline.

Result: 20/20 designs beat wildtype pLDDT.

Design	MPNN score	AF2 pLDDT	delta pLDDT vs WT
CEL5A_FIXED_013	0.7532	46.04	+12.71
CEL5A_FIXED_012	0.7513	45.69	+12.37
CEL5A_FIXED_014	0.7535	45.61	+12.29
CEL5A_FIXED_010	0.7508	44.48	+11.16
CEL5A_FIXED_004	0.7471	43.72	+10.40

The per-residue pLDDT profiles show the biggest improvements in the catalytic domain region (residues 150–250) - exactly where the constrained redesign introduced the most changes around the fixed glutamates.

A note on absolute pLDDT values

The pLDDT values (33–46) look low compared to typical small protein benchmarks. This is expected for a 605 aa two-domain protein in single-sequence mode - AlphaFold2 relies heavily on co-evolutionary information from MSAs to correctly position domains. Single-sequence mode lacks this signal.

The meaningful comparison is relative pLDDT under identical conditions, not absolute values. Every design predicts better than wildtype under the same constraints.

What I'd do next

MSA-mode validation for top 5 designs - proper pLDDT with full
co-evolutionary information
Rosetta delta-delta-G scoring - filter by predicted thermostability change
Molecular dynamics - simulate the top 3 designs at 55°C and 70°C to
assess thermal stability of the catalytic triad geometry
Experimental validation - express in E. coli, measure Tm by DSF,
compare CMC activity at elevated temperatures

Key lessons

1. Verify catalytic residues experimentally, not from literature.
PDB numbering often differs from publication numbering. The 6 Å distance
criterion is more reliable than assuming the literature values transfer directly.

2. Run unconstrained first.
The unconstrained run revealed that ProteinMPNN actively avoids glutamate at both catalytic positions. Without this finding, the constrained run would lack motivation and the project would have less scientific narrative.

3. Fixing 2/605 residues is essentially free.
Score distributions between constrained and unconstrained runs are almost
identical. You can enforce catalytic function without sacrificing sequence
diversity.

4. Low absolute pLDDT is not failure.
Single-sequence mode for large multi-domain proteins always yields low absolute pLDDT. Design relative comparisons, and always fold the wildtype under the same conditions as your baseline.

GitHub: https://github.com/Farhan89082/proteinmpnn-cel5a

If you have questions about any stage - especially the catalytic residue
identification or the constrained run setup - drop them in the comments.

How I built a computational AMP screening pipeline: from 24,000 sequences to 47 drug candidates

Sat, 04 Jul 2026 22:33:41 +0000

Antimicrobial resistance is projected to kill 10 million people annually by 2050 - more than cancer. Antimicrobial peptides (AMPs) are one of the most promising therapeutic classes to fight it. They're cationic, membrane-disrupting, and bacteria struggle to develop resistance against them.

The problem: there are thousands of candidate sequences in databases. How do you identify the ones worth synthesising?

In this post I'll walk through a full computational screening pipeline I built from raw database download through machine learning activity prediction to ColabFold structure prediction - ending with 47 high-confidence drug candidates.

The full code is on GitHub: amp-colabfold

The pipeline at a glance

Five stages:

Data curation - 24,076 sequences from DRAMP 4.0 down to 12,435 after filtering
Feature engineering - 16 physicochemical features, 0% missing
Activity modelling - gradient boosting classifier, ROC-AUC 0.77
Candidate selection - 131 structured peptides sent to ColabFold
Structure prediction - 69 high-confidence structures, 47 very high confidence

Stage 1: Data curation

I pulled sequences from DRAMP 4.0 (the Database of Antimicrobial Peptides)
using their REST API - no manual downloads needed.

Three filters applied in sequence:

# Length: 10-50 aa (therapeutic window for AMPs)
df = df[df["sequence"].str.len().between(10, 50)]

# Canonical amino acids only - no modified residues
CANONICAL = set("ACDEFGHIKLMNPQRSTVWY")
df = df[df["sequence"].apply(lambda s: set(s).issubset(CANONICAL))]

# 90% identity clustering — Python reimplementation of CD-HIT
# (CD-HIT is Linux-only; I wrote a greedy longest-first algorithm in pure Python)
df_curated = cluster_cdhit_python(df, identity=0.90)

The CD-HIT reimplementation was the most interesting part of this stage.
CD-HIT's greedy algorithm is conceptually simple: sort sequences longest-first, then for each sequence check if it's ≥90% identical to any existing cluster representative. If yes, discard it. If no, it becomes a new representative. I added 3-mer pre-filtering to skip obviously dissimilar pairs, matching CD-HIT's default behaviour.

Result: 24,076 → 12,435 sequences

Stage 2: Feature engineering

For each of the 12,435 sequences I computed 16 physicochemical features using
the peptides Python package:

features = {
    "net_charge_pH7":           pep.charge(pH=7.0, pKscale="Lehninger"),
    "hydrophobicity_eisenberg":  pep.hydrophobicity(scale="Eisenberg"),
    "hydrophobic_moment":        pep.hydrophobic_moment(window=11, angle=100),
    "boman_index":               pep.boman(),
    "aliphatic_index":           pep.aliphatic_index(),
    "instability_index":         pep.instability_index(),
    # + Chou-Fasman helix/sheet/turn propensities computed manually
    # + aromaticity, fraction_positive, fraction_negative, length, pI, MW
}

One methodological note: aromaticity() was removed from newer versions of
peptides, so I computed it directly from residue frequencies
(F + Y + W) / length — same definition as Biopython, just implemented inline.

Result: 16 features, 0% missing across all 12,435 sequences

Stage 3: Activity modelling

Binary classification: antibacterial (confirmed MIC data) vs
general/natural (uncharacterised or broad labels).

clf = GradientBoostingClassifier(
    n_estimators=300,
    learning_rate=0.05,
    max_depth=3,        # shallow trees → interpretable SHAP values
    subsample=0.8,
    random_state=42,
)

ROC-AUC: 0.77 on a length-decile-stratified holdout set.

The class imbalance (28.9% positive) is real and intentional — confirmed
antibacterial sequences with MIC data are rarer than general database entries.

I didn't oversample, because the model's conservative predictions are
appropriate for a screening tool: better to miss some true positives than
flood the downstream ColabFold stage with false ones.

What drives the predictions?

The SHAP beeswarm tells a clean biological story:

Length dominates - longer peptides push positive. This reflects the
antibacterial subset containing more full-length defensins and cathelicidins.
fraction_helix - high helix propensity → predicted antibacterial.
Consistent with alpha-helical membrane disruption.
fraction_positive - cationic residues push positive. Textbook
electrostatic attraction to anionic bacterial membranes.
fraction_negative - anionic residues suppress prediction. Expected.

This is not a black box. Every prediction is traceable to a physical property with a known biological mechanism. That's the point of using shallow trees with SHAP rather than a deep model.

Stage 4: Candidate selection

From the classifier output I selected peptides with ab_proba ≥ 0.938 and
then applied a low-complexity filter:

def is_low_complexity(seq: str, threshold: float = 0.5) -> bool:
    # Flag homopolymers (KKKKKK, RRRRRR) and tandem repeats
    max_freq = max(seq.count(aa) for aa in set(seq)) / len(seq)
    if max_freq > threshold:
        return True
    for i in range(len(seq) - 2):
        kmer = seq[i:i+3]
        if seq.count(kmer) * 3 / len(seq) > 0.6:
            return True
    return False

This step is important. Homopolymeric sequences (poly-K, poly-R) score high
on fraction_positive and fool the classifier - they're real AMPs but
structurally trivial and not useful for ColabFold. The filter removed 44
low-complexity sequences.

Result: 131 structured candidates (15–44 aa) sent to ColabFold

Stage 5: Structure prediction with ColabFold

I ran ColabFold 1.6.1 on Google Colab (free T4 GPU) in single-sequence mode appropriate for short peptides where MSA depth is limited anyway.

colabfold_batch \
  colabfold_input.fasta \
  colabfold_output/ \
  --model-type alphafold2_ptm \
  --num-recycle 3 \
  --num-models 1 \
  --msa-mode single_sequence

For 131 peptides of 15-44 aa, the full run took ~15 minutes on a T4.

Parsing results locally

def parse_scores_json(json_path):
    with open(json_path) as f:
        data = json.load(f)
    return {
        "mean_plddt": float(np.mean(data["plddt"])),
        "ptm":        float(data["ptm"]),
        "plddt_per_residue": data["plddt"],
    }

pLDDT distribution

69 of 131 candidates (52.7%) have pLDDT ≥ 70, indicating meaningful structural confidence. For short peptides - which are inherently more disordered than globular proteins - this is a strong result.

Structural confidence vs activity prediction

The scatter shows no strong correlation between classifier confidence and
structural confidence - which is biologically expected. A peptide can be
highly predicted antibacterial (high ab_proba) but disordered in solution
(low pLDDT), because many AMPs adopt structure only upon membrane contact.

Final results

Applying both filters - pLDDT ≥ 90 AND ab_proba ≥ 0.95 - gives
47 very high-confidence candidates:

AMP ID	Length	pLDDT	pTM	ab_proba
AMP_02388	30 aa	97.3	0.54	0.952
AMP_01454	36 aa	97.2	0.60	0.985
AMP_02689	28 aa	97.2	0.50	0.954
AMP_01282	37 aa	97.1	0.61	0.981
AMP_01129	38 aa	97.0	0.62	0.959

Full list: candidate_amps.csv

What I'd do next

Hemolysis screening - filter candidates through HemoPI2 to remove cytotoxic sequences before any wet-lab work
ESM-2 embeddings - train a second classifier on sequence embeddings and compare against the physicochemical feature model
MD simulation - for the top 5 candidates, run short molecular dynamics to assess membrane-binding stability
Experimental validation - MIC assays against E. coli and S. aureus

Key lessons

1. Reproducibility over convenience. The CD-HIT reimplementation in pure Python adds ~100 lines but means the entire pipeline runs on any OS without binary dependencies.

2. Class imbalance is a feature, not a bug. The 3:1 negative:positive
ratio reflects reality. Don't oversample your way out of it.

3. Interpret your model before trusting it. The SHAP beeswarm told me
the model was learning real biology (charge, helix propensity) rather than
spurious correlations. Without that check I wouldn't have been confident
sending its predictions to a GPU.

4. pLDDT ≠ activity. Low structural confidence doesn't mean an AMP is
inactive - it may just be disordered until it hits a membrane. Keep both
metrics and let them inform each other.

GitHub: github.com/Farhan89082/amp-colabfold

If you found this useful or have questions about any stage, drop a comment below.

How I Used Python to Analyse 40,000 Human Gut Cells and Uncover What Makes Crohn's Disease Different

Sat, 04 Jul 2026 20:19:33 +0000

A step-by-step walkthrough of my multi-sample single-cell RNA sequencing project - written for anyone curious about how computational biology actually works

Before we start - what is this article actually about?

Imagine being able to take a tiny sample of tissue from a patient's gut, and instead of just knowing "there are cells here", you could read the activity of every single gene inside every single individual cell - thousands of cells at once.

That's what single-cell RNA sequencing (scRNA-seq) does. And in this article, I'll walk you through how I used Python to analyse data from 40,000 human gut cells across 18 patients to understand what makes two similar gut diseases - Crohn's disease and ulcerative colitis - biologically different from each other.

No biology PhD required. I'll explain every term as we go.

The biological problem - two diseases that look the same but aren't

Crohn's disease (CD) and ulcerative colitis (UC) are both types of Inflammatory Bowel Disease (IBD). Both cause chronic inflammation in the gut, both cause pain and discomfort, and both are lifelong conditions. Doctors have known for decades that they're different diseases, but at a surface level they're easy to confuse.

The key difference is where and how they inflame:

Crohn's disease can affect any part of the digestive tract, tends to cause deep, patchy inflammation, and is driven largely by a type of immune cell called a myeloid cell (think macrophages - the "pac-man" cells of the immune system)
Ulcerative colitis affects only the colon and rectum, causes continuous surface inflammation, and is driven more by antibody-producing cells called plasma cells

Understanding these differences at the level of individual cells - not just tissue - is critical for developing better, more targeted treatments. That's where single-cell sequencing comes in.

The data - 18 patients, 40,000 cells, one big challenge

I used a publicly available dataset from the CZ CELLxGENE platform containing colonic mucosa (colon lining) biopsies from 18 donors:

6 patients with Crohn's disease
6 patients with ulcerative colitis
6 healthy controls

Each donor contributed thousands of cells, giving us 46,700 cells total (40,084 after quality filtering).

But here's the problem: when you combine data from 18 different people, collected at different times, processed slightly differently in the lab - the data gets messy. The technical differences between samples (called batch effects) can be so strong that they hide the real biological differences you actually care about.

Think of it like trying to compare photos taken by 18 different cameras with different colour settings. The same object might look completely different just because of the camera, not because the object actually changed.

This is where Harmony comes in.

What is Harmony and why does it matter?

Harmony is a batch correction algorithm. In plain English: it's a mathematical tool that looks at all 18 samples, figures out which differences between samples are due to technical variation (the "camera settings"), and removes those differences — leaving only the real biological signal.

Here's what the data looks like before Harmony:

PCA plot coloured by donor - you can see individual donors clustering separately, meaning donor identity is driving the structure more than the actual biology.

And after Harmony:

UMAP plot coloured by donor - all 18 donors are now mixed uniformly within each cell type cluster. The batch effects are gone.

But crucially - when we colour the same post-Harmony UMAP by disease group instead of donor:

UMAP coloured by disease - Crohn's, UC, and normal cells now separate into distinct regions. The biology is preserved, the noise is removed.

This before/after comparison is the core technical contribution of this project. Without Harmony, any findings could be explained by "oh, that's just because donor 3 was processed differently." With Harmony, we can be confident the differences are real.

Setting up the pipeline in Python

The full analysis uses Scanpy - the standard Python library for single-cell analysis - along with harmonypy for the integration step.

import scanpy as sc
import harmonypy

# Load the dataset
adata = sc.read_h5ad('data/ibd_dataset.h5ad')
# 46,700 cells × 32,354 genes

The data is stored in an AnnData object - think of it as a very specialised spreadsheet where rows are cells, columns are genes, and there's extra space to store metadata like disease status, donor ID, and cell type labels.

Step 1 - Quality control

Not every cell in a sequencing experiment is a real, healthy cell. Some are damaged, some are empty droplets that got accidentally captured, and some are doublets (two cells mistakenly counted as one). We filter these out using three metrics:

# Remove low quality cells
adata = adata[adata.obs['n_genes_by_counts'] > 200]   # too few genes = empty droplet
adata = adata[adata.obs['n_genes_by_counts'] < 6000]  # too many genes = doublet
adata = adata[adata.obs['pct_counts_mt'] < 30]        # high mitochondrial % = dying cell

Why mitochondrial genes? When a cell is dying or stressed, the nucleus breaks down and releases its RNA - but mitochondria (the cell's energy factories) have their own separate DNA and RNA that stays intact longer. So a high percentage of mitochondrial gene reads is a reliable sign of a low-quality cell.

After filtering: 40,084 cells remain.

Step 2 - Normalisation

Different cells capture different amounts of RNA simply due to technical variation in the sequencing process. A cell with 10,000 RNA molecules captured will look like it expresses every gene more than a cell with only 2,000 captured — even if their true biology is identical.

We normalise by scaling every cell to have the same total count (10,000), then apply a log transformation to reduce the influence of very highly expressed genes:

sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

Step 3 - Finding highly variable genes

Out of 32,354 genes, most are either not expressed at all or expressed at the same level in every cell (housekeeping genes that keep basic cell functions running). These genes add noise without adding information.

We select only the highly variable genes - genes whose expression varies meaningfully between cells - for downstream analysis. We found 2,873 of these, using batch_key='donor_id' to ensure we pick genes that are variable across all donors, not just one:

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3,
                             min_disp=0.5, batch_key='donor_id')
# Result: 2,873 highly variable genes

Step 4 - PCA and Harmony integration

PCA (Principal Component Analysis) reduces our 2,873-gene matrix into 50 dimensions that capture the most important variation. Then Harmony corrects for batch effects within this PCA space:

sc.tl.pca(adata, svd_solver='arpack')
sc.external.pp.harmony_integrate(adata, key='donor_id', max_iter_harmony=20)
# Harmony converged in just 6 iterations

The fact that Harmony converged in only 6 out of 20 possible iterations is a good sign - it means the batch effects were relatively structured and correctable, and the biological signal is strong.

Step 5 - UMAP visualisation

UMAP (Uniform Manifold Approximation and Projection) takes the Harmony-corrected embeddings and projects them into 2D for visualisation. Similar cells end up close together, different cells end up far apart:

sc.pp.neighbors(adata, use_rep='X_pca_harmony', n_neighbors=10)
sc.tl.umap(adata)

What we found - the biology

Finding 1: Five major cell populations in the gut

The UMAP revealed five well-separated clusters corresponding to the major cell types in the colon:

Plasma cells (15,633 cells) - antibody factories
Colon epithelial cells (12,347 cells) - the cells lining the gut wall
T cells (12,128 cells) - immune soldiers
Myeloid cells (3,771 cells) - macrophages and related immune cells
Stromal cells (2,821 cells) - structural support cells

Finding 2: UC and CD have completely different cellular makeups

This is where the biology gets interesting. When we look at what proportion of cells belong to each type across disease groups:

Plasma cells:

Normal: ~27%
Crohn's disease: ~30%
Ulcerative colitis: ~52%

UC has nearly double the plasma cell proportion of healthy tissue. Plasma cells make antibodies - this confirms that UC is primarily driven by antibody-mediated (humoral) immunity, a well-established finding that we independently reproduced from raw data.

Myeloid cells:

Normal: ~6%
Ulcerative colitis: ~8%
Crohn's disease: ~13%

CD has more than double the myeloid cell proportion. Myeloid cells include macrophages - the cells responsible for the granulomatous (nodule-forming) inflammation that's the hallmark of Crohn's disease.

Epithelial cells:

Normal: ~29%
Crohn's disease: ~19%
Ulcerative colitis: ~3%

This is striking. UC patients have almost no epithelial cells left - the gut lining is severely disrupted. This explains why UC causes such pronounced mucosal damage and bleeding.

Finding 3: The S100A8/S100A9 signature - a clinical biomarker reproduced at single-cell resolution

To find which specific genes are driving the myeloid difference in Crohn's disease, we ran a differential expression analysis comparing CD myeloid cells against normal myeloid cells using the Wilcoxon rank-sum test.

The top upregulated genes were:

Gene	What it does
S100A8	Subunit of calprotectin - a protein released by activated immune cells
S100A9	The other subunit of calprotectin
CXCL8	Also known as IL-8 - a chemical signal that recruits more immune cells to the inflammation site
IL1RN	An anti-inflammatory signal - the body trying to dampen its own response
BCL2A1	Keeps immune cells alive longer in the inflamed environment

Here's why S100A8 and S100A9 matter: together they form a protein called calprotectin. When gut inflammation is active, immune cells release calprotectin into the stool. Doctors measure this in a routine test called the faecal calprotectin test - one of the most common non-invasive ways to monitor IBD disease activity in clinic.

By analysing raw single-cell data, we independently identified the exact genes behind this clinical test - at the resolution of individual cells. That's the kind of validation that confirms the analysis is biologically meaningful, not just a statistical artefact.

What I learned - technical takeaways

1. Batch correction is non-negotiable for multi-sample studies. Without Harmony, the donor-to-donor variation would swamp the disease signal. Any "finding" could be explained by technical noise. Harmony is now a standard step in any multi-patient single-cell study.

2. Choosing highly variable genes with batch_key matters. If you find HVGs without accounting for batches, you risk selecting genes that are variable only because of technical differences between samples. Using batch_key='donor_id' ensures you're finding genes that are genuinely biologically variable.

3. Clinical relevance is the best validation. When your computational analysis independently reproduces a well-established clinical biomarker (faecal calprotectin), it gives you confidence that the pipeline is working correctly and the findings are real.

4. Public datasets are powerful. This entire analysis used freely available data from CZ CELLxGENE - no lab access required. The tools (Scanpy, harmonypy) are free and open source. Computational biology has an exceptionally low barrier to entry compared to wet lab science.

The full code

The complete annotated notebook is available on GitHub:
👉 https://github.com/Farhan89082/ibd-harmony-integration

The README includes all figures, biological interpretation, and instructions for reproducing the analysis from scratch.

What's next?

This project is part of a series of three single-cell RNA sequencing analyses I've built for my computational biology portfolio:

Alzheimer's Disease - microglial activation and mitochondrial dysfunction in human brain cells → https://github.com/Farhan89082/alzheimers-scrna-analysis
NSCLC Tumour Microenvironment - T cell exhaustion trajectories and macrophage polarisation in lung cancer → https://github.com/Farhan89082/nsclc-tumour-microenvironment
IBD Harmony Integration - this article

If you're a Python developer curious about getting into computational biology, or a biology student learning to code, I hope this walkthrough shows that the barrier is lower than it looks. The tools are excellent, the data is freely available, and the biology is genuinely fascinating.

Have questions about the analysis or want to discuss the methodology? Drop a comment below - I'd love to hear from you.

T Cells, Tumour Macrophages, and Why Lung Cancer Evades Your Immune System

Sat, 04 Jul 2026 19:50:20 +0000

One of the most frustrating puzzles in cancer biology: some lung cancer patients respond brilliantly to immunotherapy. Others don't respond at all. The tumour microenvironment (TME), the ecosystem of immune, stromal, and cancer cells that surrounds a tumour is a big part of why.

I wanted to understand what that ecosystem actually looks like at the resolution of individual cells. So I built a single-cell RNA sequencing (scRNA-seq) analysis of non-small cell lung cancer (NSCLC) tissue using Scanpy, working with a publicly available dataset of 111,683 cells from CZ CELLxGENE, spanning 38 distinct cell states.

Here's what the data revealed about T cell exhaustion, macrophage behaviour, and what makes some tumours more immune-suppressive than others.

Background

NSCLC accounts for approximately 85% of all lung cancer cases. Immune checkpoint inhibitors have transformed treatment for some patients — but a large proportion still don't respond. The leading explanation is that the TME is actively suppressing anti-tumour immunity, rather than the immune system simply being absent.

To understand this, you need to look not just at whether immune cells are present, but at what functional state those cells are in. That's exactly what single-cell RNA sequencing makes possible.

The Dataset

The data comes from a study characterising the cellular and molecular identities of histologic subtypes in lung adenocarcinoma:

111,683 cells after QC (117,266 raw)
57,398 genes
38 distinct cell states across the TME
4 histologic subtypes: Acinar/Papillary (A/P), A/P + Solid, Micropapillary (MP), and Solid
Sequenced with 10× Chromium 3' v3

The Pipeline

1. Quality Control

Cells were filtered to retain those with 200–8,000 detected genes, fewer than 100,000 UMI counts, and less than 25% mitochondrial content. This removed around 5,583 low-quality cells.

2. Normalisation

Standard scRNA-seq approach: normalise to 10,000 counts per cell, log1p transform, identify 2,544 highly variable genes.

3. Dimensionality Reduction

PCA (50 components) → neighbourhood graph → UMAP. The resulting embedding resolved all 38 cell states, with clear separation between tumour epithelial cells, immune populations, and stromal cells.

The Findings

1. T Cell Exhaustion Is Not an End State - It's a Trajectory

This was the most biologically interesting result. The CD8+ T cell data revealed three functionally distinct states forming a clear exhaustion trajectory:

T.CD8.Naive → T.CD8.Predysfunc (8,897 cells) → T.CD8.Exhausted (1,454 cells)

The pre-dysfunctional population is the largest CD8 group - meaning the majority of cytotoxic T cells in these tumours are actively being pushed toward dysfunction, not yet exhausted but clearly losing their killing capacity.

The fully exhausted population co-expresses multiple immune checkpoint molecules simultaneously: CTLA4, LAG3, TIGIT, TOX, ENTPD1, and CXCL13. In contrast, cytotoxic T cells express effector molecules (GZMB, PRF1, IFNG) with minimal checkpoint expression. The dotplot makes this contrast striking and clear.

2. Regulatory T Cells Outnumber Cytotoxic T Cells ~4:1

Tregs (6,496 cells) vastly outnumber cytotoxic CD8+ T cells (1,688 cells). Tregs express high levels of CTLA4 and TOX, consistent with their role in actively suppressing anti-tumour immunity. This ratio alone explains a great deal about why these tumours resist immune attack.

3. Pro-Tumour Macrophages Dominate the Myeloid Compartment

Five macrophage subpopulations were identified:

Mac.SPP1 and Mac.SPP1.GPNMB (~7,054 cells combined) - pro-tumourigenic, SPP1/osteopontin-expressing macrophages associated with poor prognosis in multiple cancer types
Mac.SELENOP (3,082 cells) - anti-inflammatory, tissue-resident
Mac.CXCL9 (1,118 cells) - anti-tumour, recruits cytotoxic T cells

The pro-tumour to anti-tumour macrophage ratio is approximately 6:1. Combined with the T cell exhaustion data, this paints a picture of a TME that is actively - and efficiently - suppressing immune responses at multiple levels simultaneously.

4. The Paradox of Solid Tumours

Solid histologic subtype tumours had the highest immune infiltration (~75% immune cells vs ~63% in acinar/papillary). But solid subtype is also the most aggressive histology.

This counterintuitive finding is consistent with what's sometimes called immune suppression rather than immune exclusion: more immune cells arrive, but the suppressive environment is so potent that they can't function. High infiltration doesn't automatically mean effective anti-tumour immunity.

What This Means for Immunotherapy

The data points to a few specific vulnerabilities worth thinking about:

The large pre-dysfunctional T cell population is a potential therapeutic opportunity - these cells aren't fully exhausted yet and might respond to checkpoint blockade
The 6:1 pro-tumour macrophage ratio suggests that myeloid reprogramming strategies (not just T cell-targeting therapies) may be needed
The Treg suppression appears to work partly through CTLA4, which is why anti-CTLA4 therapy has shown activity in some NSCLC settings

What I Learned

Subsetting for focused analysis pays off. Rather than analysing all 38 cell states at once, isolating the T cell and macrophage compartments separately and running marker analysis on those subsets gave much cleaner, more interpretable results.
Using the authors' cell type annotations directly was the right call. For a dataset this complex, re-clustering from scratch would have introduced unnecessary uncertainty. Their labels are validated and biologically grounded.
With 38 cell states and multiple findings, deciding what to prioritise in the write-up took as long as some analysis steps. Storytelling is part of the work.

The Code

Everything is in a fully annotated Jupyter Notebook. Download the H5AD file from CZ CELLxGENE and place it in the data/ folder to run it. https://github.com/Farhan89082/nsclc-tumour-microenvironment

The TME is one of the most complex biological systems we can now study at single-cell resolution - and it's increasingly clear that understanding it is key to making immunotherapy work for more patients. Happy to discuss the analysis, the biology, or the pipeline in the comments.

How I Mapped Brain Cell Changes in Alzheimer's Disease Using Single-Cell RNA Sequencing

Sat, 04 Jul 2026 19:49:43 +0000

Alzheimer's disease affects over 55 million people worldwide, yet the precise molecular changes happening inside individual brain cells remain poorly understood. I wanted to dig into that question - not at the tissue level, but at single-cell resolution.

So I built a full scRNA-seq analysis pipeline in Python using Scanpy, working with a publicly available dataset of 63,608 nuclei from human prefrontal cortex tissue (sourced from CZ CELLxGENE). The donors spanned three Braak stages: 0 (cognitively normal), 2 (early Alzheimer's), and 6 (severe Alzheimer's).

Here's what I found and how I found it.

The Dataset

The data came from a study on the molecular characterisation of selectively vulnerable neurons in AD. It covers the superior frontal gyrus, a prefrontal region known to be hit hard by neurodegeneration - and includes seven major brain cell types:

Glutamatergic neurons
GABAergic neurons
Oligodendrocytes
OPCs (oligodendrocyte precursor cells)
Astrocytes
Microglia
Endothelial cells

31,997 genes. 63,608 cells. Three disease stages. A lot to work with.

The Pipeline

1. Quality Control

No dataset is clean out of the box. I filtered cells to keep only those with between 200 and 6,000 detected genes, and excluded anything with more than 20% mitochondrial gene content (high mitochondrial reads usually signal a dying or damaged cell). This removed around 2,809 low-quality cells.

2. Normalisation

Library sizes were normalised to 10,000 counts per cell, followed by log1p transformation, standard practice that makes cells comparable regardless of how deeply they were sequenced. I then identified 5,607 highly variable genes to focus the downstream analysis.

3. Dimensionality Reduction

PCA (50 components) → neighbourhood graph (10 neighbours, 20 PCs) → UMAP embedding.

The UMAP is where the biology starts to become visible. All seven cell types separated into distinct clusters, with clear separation between neuronal subtypes and glial populations.

4. Differential Expression

For the microglial analysis, I used a Wilcoxon rank-sum test comparing AD vs normal microglia, with Benjamini-Hochberg multiple testing correction to control the false discovery rate.

The Findings

Glutamatergic Neurons Are Selectively Depleted

One of the most striking results: glutamatergic (excitatory) neurons dropped from ~34% of cells in normal tissue to ~30% in AD tissue. This might sound like a small shift, but at the scale of 60,000+ cells it's biologically meaningful and it's consistent with what the literature already tells us about the selective vulnerability of excitatory neurons in AD.

Alzheimer's Leaves a Clear Signature in Microglia

Microglia are the brain's resident immune cells, and they showed the most dramatic transcriptomic shifts between AD and normal tissue. The differential expression analysis revealed:

Upregulated in AD microglia:

MALAT1 - a long non-coding RNA strongly linked to neuroinflammation
FTH1 - ferritin heavy chain, pointing to iron dysregulation
B2M - beta-2 microglobulin, a known AD biomarker reflecting immune activation
FOXP1 - a transcription factor tied to microglial activation states

Downregulated in AD microglia:

MT-CO3, MT-CO1, MT-ATP6, MT-ND2 - mitochondrial complex genes, suggesting impaired energy metabolism in AD-affected microglia

This pattern is consistent with what's described as disease-associated microglia (DAM) in the literature, a distinct activation state that emerges in neurodegeneration.

Disease Progression Captured Across Braak Stages

Cells from all three Braak stages were distributed across every cluster in the UMAP. This reflects that AD-associated transcriptomic changes are not confined to one cell type, they propagate across the whole cellular ecosystem as the disease progresses.

What I Learned

Memory management matters. 60K+ cells × 30K+ genes is a big matrix. Working with sparse AnnData objects and being deliberate about which steps you checkpoint to disk makes a real difference.
Cell type annotation is an art. The dataset came with pre-annotated cell types, but validating them against canonical marker genes (the dotplot step) is essential and satisfying when the biology confirms itself.
Volcano plots are still one of the most readable ways to communicate differential expression. They give you significance and fold change in one glance.

The Code

Everything is in a fully annotated Jupyter Notebook. If you want to reproduce the analysis, download the H5AD file from CZ CELLxGENE and drop it in the data/ folder. https://github.com/Farhan89082/alzheimers-scrna-analysis

If you're working with single-cell data or have questions about the pipeline, I'd love to hear from you in the comments. There's something fascinating about watching biology emerge from a matrix of gene counts.

No Single Key Opens the Boundary: An Offline Dual-Control Gate for Sensitive Artifact Export

Thu, 02 Jul 2026 11:12:52 +0000

In short: PYRHELIX is an offline dual-control release gate for sensitive BIO/PII artifacts. It does not replace perimeter security, compliance review, or clinical judgment. It closes one specific gap: no single actor, key, or approval should be able to move a high-consequence artifact across the release boundary alone.

By BIO/PII export, we mean high-consequence data bundles such as genomic files, clinical exports, research packets, identity-linked health records, and derived AI-ready datasets leaving a controlled environment.

The gap between what BIO/PII data is worth and how it's protected

The BIO and health-data economy is enormous, and it's getting more automated every year as AI pipelines touch more of it. The security incidents in this space are not rare, and they are not small.

23andMe (2023): attackers used credential stuffing, feeding passwords stolen from unrelated breaches into 23andMe's login page until roughly 14,000 accounts matched. Because of the platform's DNA-relative matching feature, that cascaded into exposure of genetic and ancestry data tied to an estimated 6.9 million users, including a disproportionate concentration of Ashkenazi Jewish and ethnically Chinese users who were then targeted with hateful, ancestry-specific harassment. (HIPAA Journal, Wikipedia)
Change Healthcare / UnitedHealth (2024): a single server without multi-factor authentication was the entry point for a ransomware group. Change Healthcare clears roughly 1 in 3 U.S. patient records. The final confirmed count was 192.7 million people affected, the largest healthcare breach ever reported to HHS, disrupting billing and payment processing across thousands of hospitals and clinics nationwide. Response cost: $3.1 billion in 2024 alone. (BleepingComputer, Cybersecurity Dive)
Vastaamo (Finland, 2020): a psychotherapy clinic stored ~36,000 patients' therapy notes unencrypted, on a system whose root account had no password. When extortion of the company failed, the attacker emailed roughly 30,000 patients directly, demanding ransom to avoid publishing their session notes. The clinic went bankrupt within months. (Wikipedia, CNN)

None of these three were release-boundary failures. They were perimeter and infrastructure failures: credential stuffing, a missing MFA setting, unencrypted storage with no root password. A signing protocol does not fix a server with no root password, and we are not going to claim otherwise.

What they show, together, is a pattern. BIO and health data carries some of the highest-consequence blast radius of any data category, and the organizations holding it are frequently operating below the security maturity that scale would justify. Compliance frameworks like HIPAA and GDPR mandate access logging and encryption at rest.

They say much less about what happens at the moment a sensitive artifact is about to leave a controlled environment. In practice, that moment is still often gated by one signature, one approval click, one export button.

That narrower gap, not "BIO security" broadly but specifically the release and export decision, is what we're building PYRHELIX to close. It is not a fix for this industry. It is one piece of it, and we want to find out if it actually holds up.

Why the release step specifically stays weak

A few structural reasons this keeps happening, independent of any one company's competence:

Compliance is not the same discipline as security engineering. Passing a HIPAA audit checks access logs and encryption settings. It rarely asks whether one compromised or coerced person can push sensitive data out the door alone.
Security engineering talent concentrates in tech and finance, not biotech. Research organizations hire for domain science, not for building dual-control release infrastructure in-house.
Consolidation creates concentration risk. Change Healthcare touching a third of U.S. patient records through one platform is the extreme case. The same pattern, one vendor, one pipeline, one signer, repeats at smaller scale everywhere.
Release is usually the last, least-engineered step in an otherwise carefully built research or clinical pipeline. By the time data is ready to ship, the hard technical work already happened, and the actual export is often just an email or a shared drive.

What PYRHELIX is

PYRHELIX enforces that a sensitive bundle cannot be released by one actor acting alone. The workflow has two independent steps.

Producer (Strand A): the party that generated the artifact seals its content hash and attests to what was produced. This alone is not enough to release anything.
Authority (Strand B): an independent reviewer, using a different signing key and a different role, attests approval. That attestation is cryptographically bound to the exact same bundle closure Strand A produced.

The release boundary only opens when both strands are present, use distinct keys, and bind to the same closure. Every step is recorded in a signed, hash-chained governance journal, so a separate offline verifier, potentially running on a machine that never touched the original data, can independently replay the whole decision trail: what was produced, who approved it, with which keys, in what order.

Why "Strand A / Strand B", and where the metaphor stops

Worth getting right rather than sounding clever: a real DNA double helix does not have strands named "A" and "B." The correct terms are sense strand (coding strand) and antisense strand (template strand), and neither one alone is the functional, stable molecule; the helix only exists once both are bound together.

"Strand A" (producer) and "Strand B" (authority) are protocol role labels that borrow that single idea, not a claim to real molecular-biology nomenclature.

PYRHELIX's separate, internal bit-level integrity subsystem does use the correct sense/antisense terms for an unrelated purpose (parity diagnostics on hashed symbol streams). The governance mechanism described here is plain Ed25519 dual-signature binding over a shared closure hash.

Two pieces of the actual code

Real source, not simplified for effect.

Role separation can't be satisfied by accident. In strict mode, the verifier checks that Strand A and Strand B didn't reuse the same signing key, the same role, or the same underlying public key. Any one of those three silently fails the whole bundle:

# pyrhelix/governance/verifier_enforcement.py
if strand_a_key_id and strand_a_key_id == strand_b_key_id:
    errors.append(error("KEY_ID_REUSE", "authenticity",
        "Strand A and Strand B reused the same key_id in strict mode."))
    attestation.strand_b_authenticity_verified = False
    attestation.cross_strand_binding_verified = False

if strand_a_role and strand_a_role == strand_b_role:
    errors.append(error("ROLE_REUSE", "authenticity",
        "Strand A and Strand B reused the same role in strict mode."))
    attestation.strand_b_authenticity_verified = False
    attestation.cross_strand_binding_verified = False

There's no "trust me, these are different people" flag anywhere in this path. If the same key produced both attestations, the bundle fails.

Every journal entry is chained into the one before it, not just hashed on its own. The entry hash is computed over prev_entry_hash + canonical_bytes, so tampering with entry N breaks the hash of every entry after it, not just the one you touched:

# pyrhelix/governance/chronicle_io.py
canonical_bytes = canonical_chronicle_bytes(payload)
payload["entry_hash"] = sha256_hex_bytes(
    (prev_entry_hash + canonical_bytes.decode("utf-8")).encode("utf-8")
)
payload["event_signature_b64"] = base64.b64encode(
    signing_key.sign(canonical_bytes).signature
).decode("ascii")

Splice out or edit one line of the journal and every subsequent entry fails replay. That is exactly what the adversarial CI gate checks for (journal_tail_truncation, manifest_closure_tamper, and several others) on every commit.

Built on standards, not custom cryptography

Layer	Standard
Content hashing	SHA-256 (FIPS 180-4)
Canonical JSON for hashing/signing	RFC 8785 (JCS)
Signatures	Ed25519 (RFC 8032, FIPS 186-5)

For supply-chain security readers: PYRHELIX is not yet an in-toto or SLSA-compatible attestation format. That compatibility layer is the next evaluated integration path, not a claim we are making today.

What it actually rejects

An adversarial CI gate exercises these cases on every change, not just the happy path:

Two strands signed with the same key or role
A revoked or retired signing key reused for a new decision
Journal tampering, sequence gaps, or tail truncation
Replay of a superseded governance session
Authority key rotation without a signed rotation event
A bundle declaring "no raw payload" while raw sensitive content is actually present in the bundle directory

Running it offline

python tools/governance_ci_gate.py

It bootstraps local producer/authority keys, emits a provisional bundle (Strand A), finalizes it (Strand B), verifies the finalized bundle, and independently verifies the signed journal, all without a network call. It exits non-zero on any failure, which is also how it runs in CI on every commit.

The repository is currently private while we validate this against a real BIO/PII workflow before opening it more broadly.

What this is not

Not a runtime AI firewall. Tools like LlamaFirewall or NeMo Guardrails intercept model calls in real time and inspect prompts or outputs as they happen. PYRHELIX operates downstream of that. It gates the moment content leaves the organization, after inference is already done.
Not a fix for the kind of breaches described above. Those were perimeter and credential failures, not release-gate failures.
Not a compliance scoring product. Tools that score content against frameworks like GDPR or HIPAA evaluate what's inside a bundle. PYRHELIX only verifies cryptographically that two independent signatures exist. It doesn't replace that evaluation. It gates release after it.
Not yet interoperable with in-toto/SLSA attestation formats. We're sequencing that compatibility layer after real-world validation, not blocked on it technically.
Not externally audited yet. Internal verification today means an automated test suite plus the adversarial CI gate above.

Where we need input

If BIO/PII export is part of your work, three questions:

Is release approval today genuinely single-approver, or do you already run a dual-control process?
Does verification need to work fully offline, or is a networked service acceptable?
What would have to be true for you to say "this doesn't solve my problem"?

If any of these land for you, we'd like to run a real scenario together: Emails are not allowede

We Read a French Health-Tech Giant's Open-Source AI Pipeline Next to Its Paper. And...

Fri, 26 Jun 2026 13:52:26 +0000

When a Health-AI Release Becomes a Governance Surface

Biotech and health-AI didn't slow down in 2026.

By June, the Flamehaven team's tracking of disclosed biotechnology equity rounds showed 113 venture and strategic financings. Roughly $17.0 billion deployed, year to date.[1] Therapeutics platforms still take most of the deal count. But AI-native infrastructure (model platforms, data pipelines, diagnostic engines) is eating a bigger slice of every quarter than the one before it.

Narrow that down to France and the picture sharpens.

AI made up 62% of the €8.2 billion raised by French tech ventures in 2025. The country now counts more than 750 AI-focused companies.[2] Healthcare is one of the best-funded verticals inside that boom. Paris Region healthcare startups alone pulled in more than €3 billion over the past three years, the largest healthcare VC total of any region in the EU.[3]

None of that money showed up by accident. France 2030 put €54 billion behind it on purpose. A public Health Data Hub, built on real hospital EHR data. Compute poured into the Jean Zay supercomputer cluster. All of it built so French health-AI companies could train and publish without leaning on American infrastructure.[4]

Doctolib runs the appointment-booking and teleconsultation platform that roughly 80 million patients across France, Germany, Italy, and the Netherlands use to see a doctor. €348 million in annual revenue. 2,900 employees.[5] This is the kind of company whose AI releases get read by hospital procurement officers before anyone in machine learning even opens the GitHub link.

On June 20, 2026, Doctolib's research lab published exactly that kind of release. A paper.[7] Two open models on HuggingFace. A public GitHub repository.[6] All dropped the same day: the paper and models under open licenses, and the code as a public repository describing a pipeline for building French medical language models from web data, paired with a public dataset called FineMed.

The Flamehaven team audits AI repositories in bio and health-adjacent domains for a living. Mostly that means reading a README against a test suite, writing down where they disagree, and moving to the next one. This time the team did not move on. One sentence in the paper kept refusing to match what the repository actually does. Pulling on that thread led through the code, into the company's own commercial position, and out the other side into three explanations that remain plausible from the public evidence, though not equally supported.

Here's what the team found, in the order they found it.

What's Actually Sitting in the Repository

Before suspecting anything, you have to know what you're looking at. So start with the boring pass first.

The repository is genuinely well-organized. The README lays out three clean stages, acquisition, classification, and synthesis, each backed by a matching .slurm job-runner script. Pydantic schemas enforce structure on every LLM output. The tokenizer chain runs end to end: SentencePiece training, vocabulary trimming, fertility checks that actually pass. Across thirty-one Python files, zero AST parse errors. Zero bare except: blocks quietly swallowing exceptions. This is not amateur code, and it would be dishonest to pretend otherwise.

But sitting underneath that competence are a few things that look like exactly what they are, the kind of thing every engineer ships at 2 a.m. and means to fix later:

No LICENSE file in the code repository, even though both released models carry Apache-2.0 on HuggingFace.
A real reproducibility bug. The parameter p2_short_floor defaults to 128 through the main pipeline entrypoint. Call the underlying worker function directly, which is exactly what happens when a researcher debugs one file in isolation, and it silently defaults to 256 instead. No error. No warning. Just a different training corpus depending on which door you walked through, 234 lines apart from its own contradiction. That matters beyond aesthetics: if the corpus produced by one path differs from the corpus produced by the other, and the paper does not specify which invocation path was used, then the benchmark results in Tables 5 and 6 cannot be independently verified by a third party working from the public repository alone.
An undocumented dependency wall. Try to actually run this and convert_to_mds.py --help dies on ModuleNotFoundError: No module named 'snappy', a C-extension dependency three layers deep that the README never mentions. llm_generate.py --help fails the same way on vllm, because the synthesis pipeline that built the training corpus has no CPU fallback and needs a GPU the docs do not flag as mandatory.
Multiple hardcoded absolute file paths, including active ones, scattered across six files and pointing straight at Doctolib's internal Jean Zay supercomputer cluster. Not commented out. Live.

The team ran the dependency commands directly on a real Python environment. Both failed on first invocation — snappy missing before mosaicml-streaming could initialize, vllm missing immediately on import. Neither dependency appears as a prerequisite in the README. That is not a gap caught in review; it is a documented failure from execution.

None of that, on its own, should raise anyone's pulse. Research code ships unpolished constantly. The Flamehaven team has audited far messier repositories sitting under far smaller claims than this one.

What actually stopped the team wasn't any single item on that list. It was what happened once they stopped reading the code on its own terms and started holding it up against the one sentence the paper uses to describe what this thing is supposed to be.

The Sentence, and What's Sitting Underneath It

The paper's contributions list states it as a formal, numbered claim. The phrasing is specific enough to check:

"We release FineMed... together with FineMed-rephrased, and a reproducible curation pipeline with multi-axis annotators."

The repository's own README lands on almost the identical phrase, independently, in its opening paragraph, written for a completely different audience:

"This repository holds the reproducible data-curation pipeline..."

Two documents. Two different readers in mind, one for peer reviewers who will never open a terminal, one for GitHub visitors who will. Both land, unprompted, on the same word. That convergence is what made it worth checking, line by line, whether the code underneath actually earns that word or just borrows it.

Here's what sits directly under the functions the paper names as completed work:

!8.png

What the Paper Calls It	Where It Lives in Code	What's Actually There
§5.1, the "coarse pre-screen" stage	`llm_rewrite.py`, line 840	`# tmp pre-filter`, above the active filtering call
§3.3, "style dimensions"	`llm_rewrite.py`, line 629	`# tmp: v3 (default)`, on the live default branch
§3.1, pipeline orchestration	`llm_generate.py`, line 13	`# tmp control`, above a module-level env-var read

Three places. Three markers, each one reading, in plain English: not finished yet. And all three sit precisely on the parts of the pipeline that decide whether someone outside Doctolib could rebuild this from raw web data to a trained model. Not on the model architecture. Not on the entity extraction logic. Specifically on the parts that determine whether the process is repeatable by anyone who isn't already inside the building.

There was a fourth candidate the team almost included. The reason it didn't make the cut matters more than the marker itself.

A ## tmp comment in split_document.py looked, at first glance, like it tagged an active filter from the paper's §4.1. So the team didn't take it at face value. They opened the file and checked what the comment was actually sitting on top of:

The comment itself: ## tmp: keep only med01 ∧ edu4, which reads exactly like an active filter tag.
The lines directly underneath it: a dataset.filter(...) call.
The status of that call: every line of it commented out. Not running. Not reachable. Dead.

So the marker was not tagging live code. It was a historical note sitting on top of a block someone had already disabled, the kind of comment that survives a refactor by accident, not a flag on something the pipeline still does. Getting that distinction wrong would have meant reporting four signals instead of three. The difference between those two numbers is the entire reason this kind of checking exists in the first place.

Three real markers stayed on the table. Which raised an obvious next question: was this confined to one section that happened to catch the eye, or did it run through the whole paper?

Checking All Twelve Claims, Not Just the One That Looked Bad

So the team went back and mapped every formal contribution the paper makes against the code that's supposed to implement it. All twelve. Each one scored as fully matching, partially matching, or simply impossible to check from outside the company.

Eight came back fully verified:

The two-stage annotation architecture
The medical-term density thresholds
The educational quality thresholds
The 8-class medical entity extractor, where active code matches the paper's entity table exactly
The multi-dimensional rewriting styles
The tokenizer vocabulary alignment
The medical-content gating logic
The diverse genre/audience sampling

Zero came back contradicted. Nothing the team found anywhere in this codebase disproves anything the paper claims about what the final model does, or how it performs. That needs to be said plainly, because it's easy for it to get lost in everything that follows: this is not a story about a model that doesn't do what it says.

Four claims didn't clear the same bar:

"Reproducible curation pipeline" - Partial. The CLI exists. The training script does not appear in the main repository checkout. Three # tmp markers sit on active functions. Multiple hardcoded paths make outside execution materially harder.
PII replaced with fictional values - Partial. Not because the team found a contradiction. The paper's own Ethics section already says, in its own words, that no post-hoc audit of instruction compliance exists.
Coarse pre-screen filter - Partial, for the same # tmp reason as above.
3-phase, 240-billion-token training run - Unverifiable. There is no training execution script anywhere in the main repository checkout. Not broken. Not incomplete. Absent. Just fragments of sbatch job-submission syntax that name the phases and launch nothing.

Look at where those four land against where the eight verified ones land, because the split isn't random. Every verified claim describes what the system does once it's built. Every non-verified claim describes whether anyone outside Doctolib can rebuild it. Twelve coin flips don't sort themselves into two clean piles by accident.

One thing has to sit next to that finding before it goes any further. The paper's own README states plainly that the actual pretraining code lives in a separate ModernBERT/ Git submodule, credited openly to an outside open-source project, and that submodule simply was not initialized in the checkout the team audited. It is entirely possible the training script being called "absent" here is one git clone --recurse-submodules away from existing. The team flagged this as the single highest-priority item to resolve, and it is surfaced here rather than quietly smoothed over because that is the whole point of doing this work in public.

What that open question doesn't do is erase the other three. Those sit on different files, different functions, and don't depend on a submodule that was never cloned.

One more thing belongs here before moving on. The model cards on HuggingFace do contain explicit intended-use boundaries: both DoctoBERT-fr and DoctoModernBERT-fr are framed as encoders for downstream NLP tasks, not medical devices, and both cards state that their outputs must not drive clinical decisions.[9] That matters. The weaker surface in this release is not the model card. It is the code repository as a reproducibility artifact.

The Company Sitting Underneath the Code

By this point, the audit stops being only a code question.

Doctolib's core business has a ceiling built into it. It's a per-professional SaaS subscription in a market with a finite number of doctors and rising local competition. The company's own numbers point at exactly where it's looking next: roughly €115 million in R&D spend in 2024, an AI consultation assistant already logging more than 2 million consultations, according to its own CFO.[5] DoctoBERT isn't a side project floating apart from that strategy. The paper says, in its own words, that the model was evaluated on "a proprietary clinical NER task from a real-world production setting." This pipeline is already touching a live internal clinical workflow. It's not sitting safely inside an academic benchmark where the stakes are lower.

Which makes the licensing asymmetry worth a second look:

Model weights (DoctoBERT-fr, DoctoModernBERT-fr): Apache-2.0 on HuggingFace. Open, unambiguous, free to build on commercially.
The code that produced those weights: no license file at all.

Under the Berne Convention, which governs copyright across France and the EU, no license does not mean unlicensed in any casual sense. It means default copyright, all rights reserved, by operation of law. The weights are free. The pipeline that built them legally is not.

Separately: in November 2025, France's competition authority fined Doctolib €4,665,000 for abusing its dominant position in online medical appointment booking. That says nothing about DoctoBERT directly, but it does confirm that Doctolib is already operating under active regulatory scrutiny as a dominant-market actor — a fact worth holding when reading the governance gaps in the code.

A missing LICENSE file is genuinely one of the most common oversights in software. Most teams that skip it aren't doing it on purpose, and one missing file isn't a smoking gun on its own.

It doesn't sit on its own, though. Line it up against everything else:

Three live # tmp markers on exactly the functions that determine reproducibility
Multiple hardcoded paths into a supercomputer cluster only Doctolib has keys to
An undocumented dependency chain that breaks on a missing C-extension nobody warned about
A training script that, in the public checkout, simply isn't there

Every one of those facts points in the same direction, toward outside reproduction being harder than the paper's own headline sentence claims it is. Not one points the other way. Taken together, they are difficult to dismiss as random noise.

There's a regulatory layer that keeps this from being purely a software-hygiene complaint. Under the EU AI Act as currently written, DoctoBERT likely clears the bar for non-applicability at this stage. The scientific-research exemption and the pre-market exemption both plausibly cover a release shaped like this one.[8] Nothing in the AI Act's high-risk categories cleanly captures a general medical encoder being fine-tuned downstream for NER. That's a defensible legal position today, and nothing here disputes it.

But "not legally required to be reproducible" and "actually reproducible" are different claims. The second one starts mattering the moment a hospital procurement team asks for an auditable pipeline as a condition of doing business. So does an insurer's compliance officer. So does a government certification reviewer, exactly the customers this pivot is aimed at. The repository on GitHub right now cannot hand them one.

Some of these barriers are ordinary research-code realities. GPU-bound inference, HPC paths, and version-sensitive dependencies are standard features of large-scale ML pipelines, and they are not, by themselves, enough to defeat a reproducibility claim.

The stronger finding is narrower than that. Even setting infrastructure entirely aside, the repository contains code-level divergences, the p2_short_floor dual default being the clearest, where the same inputs can yield different outputs depending on which entry path is used.

That gap has nothing to do with whether you have access to Jean Zay. It exists inside the code itself.

So Which of the Three Is It

The Flamehaven team sat with the whole picture before writing any of this down. The three markers. The license asymmetry. The dependency wall. The missing training script. A company visibly pivoting toward the exact customers who will ask the hardest questions about all of it. Three explanations remain plausible from the public evidence, but they are not equally supported. Nobody on the team is going to pretend the distinction can be proved cleanly from public commits alone, because it cannot.

1. The release-debt explanation: this is simply an unfinished release.

Research teams push code to GitHub the same week as a paper deadline constantly, without anyone assigned to harden it for a stranger trying to rebuild it from nothing. The # tmp markers, the hardcoded cluster paths, and the missing license are exactly what shows up when an internal pipeline ships under deadline pressure and nobody circles back. Nothing here required a strategic decision. It only required a launch date.

The disclosure-boundary explanation: the release was scoped deliberately. Open enough to earn the credibility of having published a method. Not open enough for anyone outside Doctolib's compute environment to fully rebuild what the paper describes. The weights carry the license that buys goodwill. The code that would let someone replicate the full process carries ambiguity instead. No deception required — just a decision, somewhere, about where the boundary of openness should sit.

2. The serious explanation: this was designed to look open while functioning closed.

The team checked this one hardest, specifically because it is the easiest to under-check out of discomfort. They did not find support for it. The prompts are in the repository in full, including the exact rewriting instructions handed to the LLM. The filtering logic is there. The job-submission scripts are there. If the goal had been to mislead outsiders into believing they had something they do not, publishing a hollow shell would have been the easier move by a wide margin. That is not what is here. What is here is real, working logic, with one specific step left undone.

3. The team can rule out that third explanation with reasonable confidence.

The public evidence does not cleanly separate the first from the second. Both produce exactly the artifact sitting in this repository today, and the difference between release-hardening debt and an intentional disclosure boundary lives inside a decision at Doctolib that no public commit history will ever show.

The Boundary That Remains

What the team can say without claiming motive is this: the gap between the paper's headline sentence and what the public can currently do with this repository is real. It is measurable. And it lands, with observable precision, exactly on the claims that determine whether anyone outside Doctolib can verify this work independently.

You now have the same evidence the Flamehaven team did. Where you land on those two explanations is worth sitting with. So is the question of whether you'd land somewhere different if it were your hospital's compliance officer doing the asking, instead of an outside audit team with no stake in the outcome either way.

This audit is based on static code analysis, AST parsing, and partial runtime verification of a public GitHub repository at a fixed commit, cross-referenced against the published arXiv paper and project README. No proprietary Doctolib systems, internal infrastructure, or non-public data were accessed. Findings reflect code-observable and text-observable evidence, not a determination of intent. This is not a legal opinion.

For readers who want to inspect the evidence trail in more detail, Flamehaven has also published two supporting artifacts.

The full technical diagnostic report provides the deeper codebase review, including methodology notes, claim traceability, runtime-verified dependency findings, repository hygiene findings, and the corrected scope limitations behind this newsletter:

https://flamehaven.space/writing/doctobert-codebase-diagnostic-report/

The verification ledger records the STEM-BIO-AI / governance-side assessment and preserves the audit entry as a structured public evidence record:

https://flamehaven01.github.io/Flamehaven-Verification-Ledger/#doctobert

This newsletter is the narrative summary. The report and ledger are provided for readers who want to see how the review was conducted, what was checked, and where the remaining uncertainty still sits.

References

[1]. Biotech VC Funding Tracker 2026 - BioBucks. Disclosed biotechnology equity rounds, year to date as of June 2026.

[2]. The French AI Boom You Can't Afford to Ignore in 2026 - Atera, February 2026. French tech and AI venture funding figures for 2025. (Secondary source — primary statistical sourcing pending verification.)

[3]. Health & Healthtech in the Paris Region - Choose Paris Region. Three-year healthcare VC totals for the Paris Region.

[4]. France 2030: Digital Health Acceleration Strategy - OECD.AI. France 2030 budget allocation, Health Data Hub, and Jean Zay supercomputer investment.

[5]. French healthtech unicorn Doctolib hits €348m ARR, eyes profitability - Sifted, 2024. Doctolib ARR, operating loss, subscription revenue model, and growth figures.

[6]. doctolib-lab/doctobert - GitHub. Repository audited at the commit referenced in this piece.

[7]. Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining - arXiv:2606.22079, June 20, 2026.

[8]. Regulation (EU) 2024/1689 (EU AI Act), Official Journal of the European Union, OJ L 2024/1689, 12 July 2024, CELEX 32024R1689 — Articles 2(6), 2(8), 2(12), 6(2), Annex III, Article 51(2).

[9]. doctolib-lab/doctobert-fr-base and doctolib-lab/doctomodernbert-fr-base - HuggingFace Hub model cards. Intended-use boundaries, license (Apache-2.0), and non-clinical scope disclaimers.

When Medical AI Benchmarks Move Faster Than Validation

Tue, 16 Jun 2026 15:36:37 +0000

💡 *Note: This is not an argument to dismiss the Nature Medicine paper. It is an argument for stronger validation infrastructure around medical AI benchmarks before practice-shaping claims become settled wisdom.*

Two Critics, Two Reasonable Conclusions

A Nature Medicine paper published on 12 June 2026 claimed that frontier models — GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6 — now outperform specialized clinical AI tools like OpenEvidence and UpToDate AI across multiple medical benchmarks. The paper went viral within hours. The conclusion was treated as settled before most people had read past the abstract.[1]

Two clinical readers read it carefully. They reached opposite conclusions, and both conclusions were reasonable from where each reader stood, which is exactly the problem. A post-publication conflict-of-interest allegation also surrounds the paper, and we address it below.

Marissa Famularo, a vascular surgeon at Jefferson–Lehigh Valley who teaches residents, updated her practice the same day. “RAG buys provenance, not correctness,” she wrote. “Citations you can verify are not the same as a lower error rate.” She acknowledged the paper’s limits (n=100, single center, already-obsolete models) and changed her teaching anyway, because that is what engaged clinicians do when a Nature Medicine paper lands.

Natalie Khalil, PhD in Biomedical Engineering and developer of Reviewer3, subjected the same paper to structured peer review and found twelve methodological problems, four of them rated Critical. Her conclusion was that the evaluation design cannot support the paper’s central claims.

This piece does not argue that the paper should be dismissed. The study asks a question clinical AI vendors can no longer avoid: do specialized medical AI tools actually outperform general-purpose frontier models when tested head-to-head?

That question is overdue and the paper advances it. What the paper cannot establish, and what the discussion around it has largely skipped over, is whether the evaluation design can carry the weight of the certainty now traveling through the clinical AI conversation. That is a different question, and it is the one worth sitting with.

A Note on Sources

The Nature Medicine paper’s abstract and methods establish the key methodological facts discussed here: the three-part evaluation design, the tested models, the 100-query RCQ benchmark, the 1,800 clinician annotations, the HealthBench LLM-judge panel, the reported Krippendorff’s alpha of 0.10–0.20 on ordinal item-level scores, and the exclusion of refusals from aggregate scoring.

Khalil’s twelve-finding review was conducted using Reviewer3, a structured AI-assisted peer review platform she developed. We treat the review as a primary attributed source, not as peer-reviewed in the traditional sense, and we identify it as such wherever we draw on it.

OpenEvidence’s response was published as a public LinkedIn post following the paper’s release. Where we reference its conflict-of-interest allegation, we present it as OpenEvidence’s public claim and note that it has not been independently verified. The arXiv and medRxiv papers cited throughout are publicly accessible and directly relevant to the methodological concerns raised here.

What the Paper Claims and What It Cannot Show

The study evaluated performance across three stages: 500 MedQA questions testing medical knowledge, 500 HealthBench items measuring clinician alignment, and a Real Clinical Queries benchmark built from 100 de-identified, live-environment physician queries.

To evaluate these real-world queries, twelve US clinicians produced a total of 1,800 model-question annotations.

Based on these findings, the paper concluded that frontier LLMs outperformed specialized clinical AI tools across all evaluations, suggesting that scale, alignment, and cross-domain reasoning may outweigh domain-specific tuning as determinants of medical competency. [1]

The study compares observed outputs under unequal conditions. It shows that frontier models performed better on the selected benchmarks under those conditions. What it does not and cannot show is why. The clinical tools are proprietary systems whose architectures, base models, retrieval pipelines, and safety configurations are inaccessible to the researchers.

The observed performance gap could reflect the superiority of general-purpose scale, but it could equally reflect smaller base models in the clinical tools, poorly optimized retrieval pipelines, overly restrictive safety prompts, or the accumulated effect of the methodological asymmetries described below.

Because the paper cannot isolate these variables — a point the authors themselves acknowledge, noting that “it is impossible to definitively assess a mechanistic understanding” — the conclusion about scale and alignment outweighing domain-specific tuning remains an interpretive framing that the collected data cannot confirm.

Four Places Where the Claim Becomes Less Stable

1. The Ordinal Foundation Was Not Stable Enough to Support the Rankings

The RCQ benchmark used clinician ratings on a 1–4 scale, and the reported Krippendorff’s alpha for item-level agreement was 0.10–0.20 on that ordinal scoring layer — falling below the threshold typically required to support ordinal performance ranking. It indicates that raters could not reach consensus on relative quality at the level of granularity the scale was designed to capture. [1]

The paper’s Figure 2c — the primary visualization of model superiority that went viral on social media within hours of publication — is derived from the aggregate mean of discordant ordinal scores, despite the authors noting higher agreement only when collapsing the scale to binary categories or focusing on harm and hallucination flags.

While those partial signals are real and should not be dismissed, the tier ranking claiming that frontier models are categorically better is built entirely on this ordinal layer, which fails to provide the adjudicative stability such a ranking requires.

This pitfall aligns with a 2025 study on medical AI evaluation metrics, which demonstrated that comparing AI outputs against an aggregate of disagreeing experts produces inconsistent assessments that cannot reliably support performance ranking. [6]

2. The Judges Were the Defendants, and One Benchmark Was Built by One of Them

The HealthBench evaluation used an LLM-as-a-Judge approach in which the judging panel consisted entirely of the three frontier models being tested: Claude Opus 4.6, Gemini 3.1 Pro, and GPT-5.2. The specialized clinical tools were excluded from the panel. Self-preference bias in LLM judges is a known and documented limitation, and excluding the clinical tools from the judging panel is a design decision with directional consequences that the paper does not fully reckon with. [2]

A further problem extends beyond Khalil’s review: HealthBench was created by OpenAI, and GPT-5.2, an OpenAI model, is one of the systems evaluated on it — a benchmark-developer overlap the Nature Medicine paper acknowledges as a potential source of grading bias, explicitly stating that HealthBench should be interpreted as “supplementary” to the primary RCQ clinician evaluation.

Despite this self-designation, its results were presented as a co-equal pillar in the headline hierarchy and received as confirmatory by the public, failing to resolve the limitation’s impact — a systemic issue underscored by a 2026 scoping review which found that 73.5% of healthcare LLM-as-a-judge studies perform no bias testing, meaning high agreement scores often reflect shared blind spots rather than valid assessment. [7]

3. The Paper Acknowledges Benchmark Exposure but Does Not Resolve Its Impact

MedQA and HealthBench have been publicly available on the internet for an extended period prior to the evaluation window. Frontier models are trained on large, continuously updated corpora of internet text. The Nature Medicine paper acknowledges this issue and notes that benchmark exposure is possible, but does not quantify or resolve the impact of that exposure on the headline hierarchy. [1]

This does not mean the models memorized exact answers. It means the distribution of their training data may not have been independent of the evaluation distribution, and the paper leaves that question open while drawing firm conclusions from the results.

If any exposure is present, it advantages exactly the systems that already benefit from the judge composition and interface asymmetry described above. The headline performance gap could be directionally real, partly artifactual, or some combination of the two. The current design cannot distinguish between them.

4. Two Statistical Issues Warrant Clarification

The regression analysis treated 1,704 rater-item observations as independent after accounting for rater effects. These observations are clustered within 100 specific clinical queries.

Multiple models and raters evaluating the same query produce correlated scores due to the inherent nature and difficulty of that specific query. Failing to include a random intercept for the query introduces pseudoreplication, artificially inflating degrees of freedom and potentially generating confidence intervals narrower than the data can honestly support. [2]

Separately, the paper states that UpToDate’s refusal rate of 19% was not significantly higher than Google AI Overview’s refusal rate of 6% (P=0.10), and specifies that Fisher’s exact test was used for refusal rate comparisons.

A raw Fisher’s exact test on 19/100 versus 6/100 yields a two-sided p-value of approximately 0.009 — which, absent any multiple comparison adjustment, appears inconsistent with the reported P=0.10 at the paper’s own stated alpha=0.05 threshold. If a Bonferroni or other correction was applied across the full refusal-rate comparison set, that adjustment should be stated. As written, the discrepancy warrants clarification. [1]

The Compounding Asymmetries

Beyond the four stress points, Khalil’s review identifies design asymmetries that compound the picture without being fully addressed. Frontier models were evaluated via deterministic API outputs with temperature set to 0.0 and a fixed generation seed.

Clinical tools were evaluated via non-deterministic browser interfaces with hidden system prompts and dynamic retrieval mechanisms. The paper acknowledges this asymmetry but does not fully resolve what it means for the comparison.

UpToDate AI refused 19% of queries while frontier models refused 1–3%, and those refused responses were excluded from aggregate scoring. This means UpToDate’s aggregate score reflects only the subset of queries where the system was confident enough to respond, while frontier model scores reflect the full query distribution. Whether this affects the comparative result materially is not analyzed.

These asymmetries do not individually overturn the paper’s findings. Cumulatively, they describe an evaluation environment whose design features, taken together, created conditions that may have favored frontier models — and the paper does not fully account for that tilt.

The Disclosure Question Raised After Publication

OpenEvidence’s public response states that the study’s authors operate a competing in-house medical AI at their hospital, and that they had previously approached OpenEvidence requesting API access, including rights to build a competing product using OpenEvidence’s own infrastructure. OpenEvidence declined. The Nature Medicine paper appeared afterward. [4]

This is OpenEvidence’s claim, issued by a directly interested party, and we cannot independently verify it. If accurate, readers would reasonably expect this relationship to be disclosed or addressed in the paper’s competing interests section.

It was not. We leave readers to weigh this context against the methodological picture described above. The methodological concerns stand regardless of whether the allegation is accurate, but readers are entitled to know it exists.

What the Counter-Evidence Shows

A separate medRxiv study applied the same triage benchmark that previously exposed severe weaknesses in ChatGPT Health (51.6% undertriage of true emergencies) to OpenEvidence under identical conditions. OpenEvidence undertriaged 12.5% of emergencies, a fourfold reduction.

It showed no social anchoring effect. Its errors skewed toward safer directions, and refusals occurred only in symptom-only prompts, never in urgent or emergency cases. [5]

This does not prove that OpenEvidence is superior to frontier models in clinical settings. It shows something narrower but more important for this discussion: benchmark selection in medical AI is never a neutral methodological choice, especially when the benchmark design includes public dataset exposure, self-judging panels, and frontier-sourced query distributions that may favor the very models being evaluated.

When a different benchmark applied by independent, conflict-free researchers produces a meaningfully different picture of the same tool, it becomes clear that neither study is definitive, and that the headline hierarchy is far more sensitive to evaluation design than the paper’s framing acknowledges.

The Governance Gap in Miniature

Famularo updated her teaching the day the paper appeared, which is what engaged clinicians should do when a Nature Medicine paper lands. Her update was calibrated and practically useful: RAG buys provenance, not correctness. Check the source. Don’t trust the absence of hallucination. These are good heuristics. She flagged the caveats. She called it a snapshot, not a verdict. [3]

The structured peer review that identified the paper’s pressure points arrived days later, from a specialist who had built a tool specifically designed to find them. Most papers do not get that scrutiny. Most clinical practice updates do not wait for it.

Publication, uptake, critique, too late. That sequence is not a failure of the clinician who updated her teaching, or the reviewer who found the flaws, or even the authors who published the paper. It is the operating condition of medical AI evaluation in 2026, and it will keep producing the same outcome until the field decides that the speed of the claim and the robustness of the validation have to move together.

What This Means

The paper’s headline finding may be directionally correct. Frontier models may genuinely perform better than current specialized clinical tools on the tasks this benchmark measures. The evaluation design, as it stands, cannot fully establish that claim, and it especially cannot establish the mechanistic conclusion that scale and alignment outweigh domain-specific tuning as determinants of medical competency.

What the paper establishes more clearly — and perhaps more durably — is the shaky infrastructure upon which it was built. It describes a system in which performance hierarchies are generated from ordinal ratings despite near-zero inter-rater agreement, models are evaluated on benchmarks that some of them helped design, and judgments are rendered by panels composed of the models themselves.

Benchmark exposure and interface asymmetries are acknowledged but left unquantified or unresolved, while survivorship bias in refusal exclusions and a post-publication conflict-of-interest allegation remain largely unanalyzed.

These are not incidental weaknesses in one paper. They describe the available toolkit for high-profile medical AI evaluation right now, at the same moment clinical AI tools are moving from research conversation into hospital procurement and daily clinical workflow.

The danger is not that one paper went viral. The danger is that clinical AI evaluation now produces practice-shaping claims faster than the field can audit the machinery behind them.

What Auditing Should Require

At minimum, auditing that machinery should require independent judge panels not drawn from the evaluated systems, pre-registration of benchmark contamination checks, and refusal-inclusive scoring that does not selectively filter the hardest queries out of the comparison.

None of these are technically difficult. They are choices. And until they become expected choices, the gap between claim speed and validation depth will remain a feature, not a bug, of how medical AI is evaluated and published.

References

Vishwanath, K. et al. “General-purpose large language models outperform specialized clinical AI tools on medical benchmarks.” Nature Medicine (12 June 2026).
Khalil, N. “Reviewer3 structured peer review of the above.” Publicly shared (June 2026).
Famularo, M. LinkedIn response to the above (June 2026).
OpenEvidence. “Public LinkedIn response and conflict of interest statement” (June 2026).
Jia, E. et al. “OpenEvidence errs on the safe side in a structured test of triage recommendations.” medRxiv (April 2026).
Kopanichuk et al. “How to Evaluate Medical AI.” arXiv:2509.11941 (2025).
“A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework.” arXiv:2604.25933 (2026).

Beyond M15: Why STEM BIO-AI Started Acting More Like a Governance Report in v1.8.x

Fri, 12 Jun 2026 10:23:48 +0000

Not just a new framework, but a clearer answer to what the score means, why the report exists, and how the artifact should be read.

The real change in v1.8.0 through v1.8.4 was not that STEM BIO-AI cited one more framework.

The real change was that it became harder to misread the report.

M15 mattered. It strengthened the regulatory-traceability vocabulary. But the deeper shift was broader:

the tool got stricter about what it was willing to imply from local repository evidence, and the report got more explicit about why each surface exists at all.

That changed the project in three ways:

it stopped behaving like a score sheet that developers happened to inspect
it integrated M15 as a bounded post-hoc traceability layer rather than a hidden score driver
it treated release memory, packaging, and public report surfaces as part of release integrity rather than mere maintenance hygiene

This is the real post-M15 story.

Part 1. Perception: Why STEM BIO-AI Should Not Be Read as a Simple Score Tool

The hardest reporting problem in the v1.8.x line was no longer only how to show something or even what to show.

It was why to show it at all.

That distinction matters because the same report is read by different people for different reasons:

a prospective user wants to know whether the repository is trustworthy enough to try
a maintainer wants to know what is holding the score down and what to fix first
a reviewer or auditor wants to know which claims are supported, which are overstated, and which remain outside scope

If those audiences all receive the same fields without a visible purpose hierarchy, the result is machine-legible but human-misleading.

That is why the recent report changes should be understood as user-friendliness in a governance sense, not as design polish.

The project had to become better at stopping readers from confusing:

a deterministic score with a safety verdict
a traceability mapping with compliance proof
a code-integrity PASS with overall repository maturity
a compact report surface with complete evidence

That realization changed the output layer itself.

Recent report work added or strengthened several surfaces specifically to solve that perception problem:

a fixed score-boundary note near the score itself
explicit Tier Lock and Classification Applied surfaces so score constraints are not hidden
stronger Governance Posture, What Is Actually Present, and What Is Missing Or Contradicted summaries
Regulatory Traceability placed ahead of the MIT AI Risk Repository (AIRI), used here as a secondary risk-vocabulary layer, so the reader sees repository-to-framework mapping before the broader risk language
clearer chapter hierarchy in the detailed PDF so the report reads like a governance document instead of a detector dump

In concrete terms, that changed the reader's path through the artifact.

Instead of landing first on a score and then digging through detector output, the current report leads with:

Governance Posture
About This Score
What Is Actually Present
What Is Missing Or Contradicted
Regulatory Traceability
AIRI Risk Triggers

Only after that does it move into Decision Path, Top Remediation Actions, Code Integrity details, and Evidence detail.

The key lesson was simple:

a report becomes more useful not when it shows more fields, but when the reason those fields exist becomes legible to the reader.

This is also why the score disclaimer mattered so much:

Score reflects calculation integrity, not calibrated validity. Triage signal only.

That sentence is not ornamental. It forces the system to tell the truth about itself.

What is verified:

calculation integrity
deterministic reproducibility
transparent score assembly

What is not verified:

calibrated measurement validity
runtime behavior correctness
clinical safety
compliance or regulatory clearance

This is the most important perception shift in the v1.8.x line.

The project is no longer trying only to answer, “What score did this repository get?”

It is trying to answer something more useful:

Is bio-governance actually present?
Is it adequate relative to the repository’s claims?
What is verified, what is inferred, and what is still missing?

Figure 1. The report now places governance posture, score-boundary language, and top-level trust signals near the score surface instead of hiding them behind lower-level detector output.

Part 2. What M15 Is, Why It Matters, and How STEM BIO-AI Uses It

M15 refers to ICH M15: General Principles for Model-Informed Drug Development.

The official FDA guidance page is here:

FDA: M15 General Principles for Model-Informed Drug Development

As the FDA describes it, the June 2026 final guidance was prepared under the auspices of the International Council for Harmonisation and provides general recommendations for:

planning model-informed drug development evidence
model evaluation
documentation
regulatory interactions
reporting and submission

It also establishes a harmonized assessment framework and terminology for MIDD evidence. That matters because it gives a cleaner language for talking about traceability, documentation quality, and context of use.

But the important thing in STEM BIO-AI is not merely that M15 appears in the output.

The important thing is how it appears.

STEM BIO-AI does not use M15 as a covert score driver. It does not inflate the formal score because an M15 citation exists. It uses M15 as a post-hoc regulatory traceability layer attached to already-detected repository evidence.

That boundary matters.

Without it, a framework citation can easily become a kind of rhetorical overclaim:

the report looks more regulatory than it really is
the reader assumes framework mention implies compliance maturity
traceability begins to masquerade as proof

The post-M15 line was careful to avoid that mistake.

In practice, the project used M15 in a bounded way:

as part of measurement_basis and regulatory framing
as a traceability surface that helps interpret repository evidence
as a complementary reference alongside EU AI Act, IMDRF, and FDA guidance themes
not as a direct input that changes the formal score formula

That changed real artifact fields.

The post-M15 line now surfaces traceability in places such as:

human-readable Regulatory Traceability sections in HTML, Markdown, explain, and PDF
framework-grouped labels such as EU AI Act, ICH M15, IMDRF, and FDA
status-oriented summaries such as Signal only, Partially aligned, and Aligned
explicit source_ids and finding_refs so a reader can trace which repository signal triggered which regulatory mapping

That is why the right way to describe the integration is:

M15 strengthened traceability language and reporting context, but it did not become the hidden engine of the score.

This is also consistent with how FDA guidance should be read. FDA's own Federal Register notice states that guidance documents do not establish legally enforceable responsibilities; they describe the Agency's current thinking and should be read as recommendations unless specific statutory or regulatory requirements are cited. See the June 3, 2026 Federal Register notice for M15: https://regulations.justia.com/regulations/fedreg/2026/06/03/2026-11112.html.

This distinction also helped the report become more honest.

Regulatory Traceability is useful because it tells a reviewer:

which frameworks the observed evidence touches
which mappings are only signal-level
which are partially aligned
what the report still cannot claim

That is exactly where a framework like M15 belongs in this system: as a bounded interpretive layer that helps a reader connect local repository signals to external governance language more carefully.

Regulatory traceability now shows framework-grouped mappings, bounded statuses, and trigger-linked references, making it easier to see how local repository evidence touches M15, EU AI Act, IMDRF, and FDA guidance themes without mistaking those mappings for compliance proof.

Part 3. The Other Improvements That Actually Made the Tool More Mature

After the M15 integration, three other changes mattered just as much, and in some cases more.

3.1 The Tool Stopped Hiding Score Constraints

One of the biggest interpretability problems in earlier versions was that a report could be capped or floored without making that state obvious enough in the human-readable artifact.

That is what led to:

Tier Lock [CA-CAP], the clinical-adjacent score-cap state
Tier Lock [T0-FLOOR], the hard-floor state for stronger direct clinical concern
Classification Applied

These surfaces changed the meaning of the report.

They tell the reader that the formal score is not just an arithmetic total. It is also shaped by active classification state:

whether the repository is clinical-adjacent
whether an explicit non-clinical boundary is missing
whether a score ceiling is active
whether a hard-floor review path has been triggered

This made the report more inspectable, but more importantly, it made the report less willing to hide the reasons a higher tier is blocked.

That matters because remediation is not always “add more points.”

Sometimes the real issue is:

remove the condition that prevents the repository from being meaningfully read as governance-ready

That is a better audit posture than a naked scalar score.

3.2 The Report Became a Governance Document Instead of a Score Sheet

This was the most visible change to anyone reading the artifacts.

The detailed packet stopped feeling like a machine-oriented export and started behaving more like a governance-suitability document.

The current packet is built around a more explicit hierarchy:

Governance Posture
What Is Actually Present
What Is Missing Or Contradicted
Regulatory Traceability
AIRI Risk Triggers
Method Boundary

The current detailed packet is chaptered as:

Chapter 1 — Stage Scorecard and Governance Scoring
Chapter 2 — Code Integrity Deep Analysis
Chapter 3 — Regulatory Traceability
Chapter 4 — Remediation Actions, AIRI Risk Triggers & Method Boundary
Chapter 5 — Report Metadata

The HTML report similarly exposes a seven-section navigation path:

Summary
Decision Path
Code Integrity
Regulatory
AIRI Risk Triggers
Evidence
Developer

Those labels matter because they changed what the reader sees first and what the reader is expected to conclude from the artifact. The reader now moves through adequacy, contradiction, traceability, and scope before falling back to engineering detail.

Only after that does the packet lean into deeper developer-facing material such as:

Decision Path
Top Remediation Actions
Code Integrity details
Evidence detail

That reordering matters because the report’s first job is not to help a maintainer debug detectors. Its first job is to answer whether bio-governance is actually present, whether it is adequate relative to claims, and what remains unsupported or missing.

That is why the current packet structure is more than presentation work. It is a statement about document type: a governance artifact with

a posture statement
explicit scope limits
traceability context
contradiction surfaces
remediation direction

3.3 MICA, Packaging, and Release Surfaces Became Release Integrity Work

The final maturation step was less glamorous, but it mattered a great deal.

In v1.8.x, active memory pointers, public version surfaces, preview assets, and package-data inclusion became impossible to treat as optional housekeeping.

If the release says one thing while:

MICA, the project's active release-memory layer, points somewhere else
packaged assets omit active files
report previews lag behind the actual runtime
public docs describe stale behavior

then the tool is not governed. It is merely assembled.

That is why post-M15 work spent real effort on:

rotating the active MICA trio cleanly
pruning live historical memory surfaces while preserving provenance in Git-tagged history
making report previews match the actual runtime output
hardening package-data and release-surface alignment

The practical examples here are not abstract:

README level tables and actual packet filenames had to agree on 8p, not 7p
tracked preview assets had to match the real generated HTML and PDF outputs
active MICA pointers had to reference the same live trio the package actually shipped
public docs had to stop describing stale section counts or old packet shapes

Small mismatches matter here because governance tools are judged by their own traceability discipline. If a report surface says 8p while the surrounding docs still describe 7p, the tool teaches the wrong lesson about its own evidence hygiene.

This sounds operational because it is. But it is also methodological.

A governance scanner that critiques target repositories for stale surfaces, unsupported claims, or weak provenance cannot remain credible if its own release memory and public artifact surfaces drift by version.

That is why the packaging and memory work belongs in the same story as the report work.

It reduced the number of places where truth could fork.

Where This Leaves the Project

If I had to summarize the post-M15 line in one sentence, it would be this:

STEM BIO-AI became less willing to let a convenient surface pretend to be the whole truth.

That shows up in several places at once:

the score is now shown with clearer purpose boundaries
score constraints are surfaced instead of buried
M15 appears as traceability, not as covert score inflation
AIRI is framed as secondary risk vocabulary, not proof
the packet now behaves more like a governance document
release memory and packaging are treated as release-integrity concerns

The tool is still bounded and deterministic. It still cannot see runtime truth, wet-lab reproducibility, model-output correctness, or clinical validation.

But in the v1.8.x line, it got better at saying exactly that.

And it got better at saying it in a form that a prospective user, a maintainer, and a reviewer can all use without needing to reverse-engineer the internal taxonomy first.

Roadmap

The next maturity steps are not only more detectors.

They are also:

improving human-readable explanations without overstating certainty
expanding the behavioral and path-sensitive side of static analysis without pretending it is dynamic truth
broadening benchmark calibration so score validity is less prior-heavy
continuing to align report purpose, release memory, and public surfaces so the artifact remains hard to misread

That is the real roadmap after M15.

Not just more coverage.

More disciplined meaning.

Repository: https://github.com/flamehaven01/STEM-BIO-AI
Live HF Space: https://huggingface.co/spaces/Flamehaven/stem-bio-ai

We Built AI Verification Infrastructure. Then It Found Our Blind Spots.

Tue, 02 Jun 2026 17:44:46 +0000

Prologue: The Mirage & The Pivot

In the summer of 2024, our team began with an ambition that felt almost impossible: to use frontier AI for drug discovery, theoretical physics, and problems that seemed unreachable. We imagined systems that could reason about spacetime geometries and propose novel small-molecule bindings.

Within the first month, the boundary became clear. The models were powerful, but their claims were not something we could responsibly stand behind unless the path from input to result could be inspected.

So we made a choice. We could keep producing evocative claims we could not verify, or we could build infrastructure that would force our own work through the same scrutiny we would apply to anyone else's.

We stopped. We built.

For the last two years, we pursued this work without revenue. The original ambition was larger than the system we are publishing now, but the work became more precise. We began turning model-generated reasoning into executable Python harnesses, custody records, and governance gates before we had stable language for what to call them.

The practical question became narrower: can scientific claims expose the path by which they were generated, and can biomolecular AI systems be evaluated for transparency before their outputs are treated as evidence?

We shared progress publicly on dev.to, LinkedIn, Substack, Medium, and CoderLegion, usually one to three times a week. Most posts did not travel far. They were part research log, part signal flare, and part attempt to decide what should remain protected and what should be released.

Over time, the answer became clearer: protect the sensitive core, but publish verification surfaces, audit artifacts, and reproducible paths wherever possible.

We are not claiming to have solved drug discovery or the Theory of Everything. We are publishing the verification materials because even a small piece of reliable infrastructure may help others near those frontiers.

What the Flamehaven Verification Ledger Is

That choice — to verify before we believe — became the whole of what we built. Everything in the Flamehaven Verification Ledger answers one question:

Can this claim show the path by which it came to exist?

A mathematical or scientific claim earns operational trust only once the path behind it can be inspected, reproduced, or challenged.

A biomolecular AI pipeline is trusted only when its disagreements are surfaced, not smoothed over.

A bioscience repository is cleared only once its safety is checked line by line, with no model in the scoring loop.

Verification is not a feature here. It is the spine.

The ledger is public and inspectable by design. Each run is published as a bounded artifact: inputs, a score, a report, a traceable custody path, and, when it happens, a failure note.

Today it has three main verification lanes:

EQA — Equation-to-Artifact. Mathematical and physical claims are turned into runnable, deterministic Python checks, including string-theory beta-function tests at 200-bit precision and the OpenAI Erdos reproduction.
BAV — Biomolecular AI Validation. Biomolecular pipelines are evaluated as governed systems, not just prediction engines. Model disagreement is treated as a safety signal rather than noise.
BSC — Bioscience Compliance. The open-source STEM-BIO-AI scanner runs local, zero-execution safety and compliance scans of bioscience repositories and maps findings to external risk and traceability frameworks, with no LLM in the scoring path.
Open resources are rolling out as they are cleared for public release.

Public ledger

What Two Years Produced — and What It Cost Our Beliefs

This is the part that matters. Two years of runs left a body of results: some held up under scrutiny, and some broke beliefs we were attached to.

A verification engine earns credibility from both: results that survive and failures that tell us when we are wrong.

Success 1: Geometry Overruled the Narrative

The EQA lane showed something important in TOE-TEST-0004: it judged a background by its geometry, not by the words attached to it.

The test was built around Green-Schwarz anomaly cancellation, where the gravitational and gauge Pontryagin densities must match:

p1_R = p1_F

Two adversarial cases made the point.

A Schwarzschild black hole carrying a single-plane gauge field sounds like an anomaly waiting to happen. The engine returned PASS (Omega = 0.9997) because Schwarzschild is Ricci-flat (p1_R = 0) and a single-plane field is topologically trivial (p1_F = 0).

Flat 10-D Minkowski space sounds harmless. But with a two-plane gauge field, it returned FAIL:

p1_F = 2.0 != p1_R = 0

Without curvature, there is nothing to cancel the gauge anomaly. The ledger did not reward the background for sounding safe, nor punish it for sounding dangerous. It ran the case.

Success 2: Calibration Is Not Understanding

EXP-028 was built to catch a different confusion: a model can be beautifully calibrated and still not understand what it is looking at.

On the surface, the model looked excellent. Brier score improved from 0.204 to 0.0056 after calibration. AUC was 1.0.

By standard calibration metrics, this is the kind of model you might trust. But the cross-domain honesty gate refused it.

SR9 was about 0.26 against a gate of >= 0.80. DI2 was about 0.61 against a gate of <= 0.20.

So the pipeline did the rare thing: it returned "I cannot resolve this" and abstained, rather than let a well-calibrated number masquerade as comprehension.

Calibration measures whether confidence is honest about frequency. It does not prove understanding.

*SR9 and DI2 are advisory heuristics, not externally validated physical quantities. That boundary matters.

Success 3: A Repository Scored Against a Traceability Rubric, Deterministically

The third success is the least glamorous and the most reusable: a real bioscience repository scored against external traceability concerns by code anyone can re-run.

Our open-source scanner STEM-BIO-AI (pip install stem-ai) reads observable signals and grades three rubrics:

S1 = README / intent evidence
S2 = repo-local consistency
S3 = code / bio responsibility

They combine into a deterministic weighted score, with penalties and hard caps where needed:

raw = round(0.4*S1 + 0.2*S2 + 0.4*S3 - risk_penalty)

For yorkeccak/bio, the scan produced a final score of 48, mapped to T1 Quarantine.

The point is how the number is reached: README intent, repo-local consistency, dependency safety, exception handling, data provenance, clinical disclaimers, and AST-level code analysis using ast.parse.

Nothing in the target repo is executed. No network, no GPU, no model in the loop. The audit runs locally and reproduces from repository state.

The Failures We Are Publishing

Failure 0: Synthetic Data in Our Own Ledger

The most important failure was not in physics or biology. It was in our own record.

An early EQA dashboard shipped a "51-run calibration registry" that read as authoritative until an external reviewer showed it was procedurally generated.

It contained fabricated primes, fabricated field degrees, synthetic hash labels, and at least one failed check beside a PASS verdict. It was hallucinated scaffolding, not computation.

We deleted the synthetic registry, replaced it with real TOE-TEST foundational runs, sanitized local workspace paths, and added a deterministic synthetic_marker detector so a [synthetic] tag cannot re-enter the public ledger.

This is the failure we are most willing to show because it is what the system was built to catch — including when the fabricator was us.

Failure 1: One-Loop Weyl Curvature Blindness

The EQA engine evaluates spacetime geometries under one-loop worldsheet beta-functions of the non-linear sigma-model.

We stress-tested it with a "Planck-scale spacetime foam Schwarzschild metric with mass M = 0.1," expecting rapid failure due to extreme curvature near the horizon.

Instead, the solver returned PASS:

Omega = 0.9985

The reason is precise. Schwarzschild is Ricci-flat outside the singularity:

R_mn = 0

Our one-loop beta-function gate couples directly to Ricci curvature, not Weyl curvature. So the metric residuals vanished even though tidal curvature was physically severe.

The Kretschmann scalar made the blind spot visible:

K = R_abcd * R^abcd = 48 * M^2 / r^6 = 4800

This is not a broken engine. It is a specific boundary. The solver is correct where one-loop Ricci geometry is sufficient. T09 marks where that sufficiency ends.

We have not fixed this. The correction requires higher-order alpha-prime terms for Ricci-flat backgrounds with high tidal forces.

Failure 2: An Honest Rejection We Cannot Yet Confirm

In EXP-005, our internal candidate-generation engine screened three lipid carriers for topical Upadacitinib: SLN, NLC, and liposomal gel.

Each was scored by SR9, an advisory cross-domain consistency heuristic. Our bar was >= 0.80.

The scores were far below that bar:

SLN ~= 0.28
NLC ~= 0.23
liposomal gel ~= 0.26

Every candidate was rejected in under two hours, against an estimated months of bench work.

But because SR9 is advisory, we do not know whether these values reflect real formulation incompatibility or an over-conservative false reject. A fast negative is useful only if it is correct.

Failure 2b: When the Models Themselves Disagree

EXP-031 tested a 52-amino-acid target across AlphaFold2, AlphaFold3, Chai-1, and Boltz-2.

They did not converge. pTM stayed low:

AF2 pTM ~= 0.31
AF3 pTM ~= 0.40

The inter-model consensus drifted by 0.09 to 0.37.

The honest outcome was not a number. It was a refusal: the pipeline returned Unverified / Drift Detected and an observer-only decision for every arm.

The ledger's job was to refuse to launder that disagreement into a result.

Failure 3: The Multiplicative Reliability Fallacy

The BAV pipeline originally modeled end-to-end reliability as a multiplicative chain:

p_e2e = p_capture * p_transfer * p_model * p_clinical

These factors are not independent. Clinical reliability depends on model reliability, which depends on transfer and capture quality.

The correct direction is conditional:

p_e2e =
  p(clinical | model)
  * p(model | transfer)
  * p(transfer | capture)
  * p(capture)

We have not implemented this yet. Every reliability figure in the BAV lane carries this constraint.

From Experiments to Public Custody Paths

Before this infrastructure existed, experiments ended as local folders, private notes, or claims that required too much explanation. Now a weekly experiment can become a public record: a bounded run, score, report, artifact path, and failure note.

We do not see this as a replacement for peer review. We see it as the layer before peer review: the place where claims become inspectable, failures remain visible, and weekly experiments accumulate.

We also avoid claiming false determinism. Fold outputs vary with seed, GPU architecture, MSA depth, and compiler version. In EXP-031, we anchor the input FASTA, pin the seed and model versions, and record confidence metrics as a labelled reference run. The goal is regime-level comparison, not bit-exact reproduction.

The detailed repository tree is omitted here for space; the public ledger exposes the custody paths directly.

MICA and the CI Sanitizer Gate

Two layers keep the ledger honest before publication.

MICA, Memory Invocation & Context Archive for AI, is the governance memory. It encodes project invariants: internal metrics carry no external authority, fabricated data is prohibited, and public claims stay understated.

The sanitizer is the enforcement layer. It runs as a CI gate and fails the build when it finds workspace leaks, local-specific PII patterns, promotional language, or fabrication tags.

The synthetic_marker detector was added after Failure 0. The promotional_language detector flags public-facing hype such as "revolutionary," "state-of-the-art," or "authoritative."

The result is a repository that censors its own hype, leaks, and fabrications before publication — and keeps an auditable record of doing so.

Dashboard Heuristics: Labelled As Heuristics

The dashboard plots EQA metrics against an exponential curve.

Status: ADVISORY-HEURISTIC.

This curve is not a physical law. It is an empirical visual fitting heuristic for charting. Treating it as fundamental law would violate the ledger's credibility rules.

Omega(norm_R_F) = Omega_0 * exp(-lambda * norm_R_F) + epsilon_numerical

What We Need Others to Test

The ledger is not finished. It is a public audit brief.

We need formulation scientists to test whether EXP-005 reflects real carrier incompatibility or an over-conservative false reject.

We need structural biologists to test whether EXP-031 reflects genuine out-of-distribution behavior or a harness artifact.

We need string theorists to test whether TOE-TEST-0001 is physically correct in the Ricci-flat regimes where we claim results.

We need compliance experts to test whether the zero-execution Article 12 mapping in BSC is legally defensible or technically approximate.

All code, schemas, and run payloads: Zenodo DOI 10.5281/zenodo.20483364

Public Ledger: https://flamehaven01.github.io/Flamehaven-Verification-Ledger/
Citation ORCID: 0009-0009-2641-4280

Epilogue: The Invitation

We began by trying to reach the unreachable: spacetime geometries, novel small-molecule bindings, a Theory of Everything.

We did not get there. This ledger is not that.

What we found instead was smaller and more useful: a way to make a claim show the path by which it came to exist.

As AI fills every field with fluent, confident, publication-shaped output, the quiet casualty is the experiment itself: the runnable artifact, the failed run nobody deleted, the number traced back to code.

We are not announcing a solution. We are publishing a small, inspectable surface — failures included — and handing it to the people best equipped to break it.

If it proves useful against the confident noise filling everyone's feeds, that will be your verdict, not our claim.

We built this to learn from you.

Stanford. Princeton. A bioRxiv Paper. So Why Did Nobody Ask Where the Data Goes?

Wed, 27 May 2026 14:15:21 +0000

When a repository carries this kind of pedigree, people stop asking questions.

Runchuan-BU/BioClaw (1) — a biomedical AI assistant published on bioRxiv on April 14, 2026 by a multi-institution research group (2). The STELLA foundation carries Stanford and Princeton correspondence addresses. 374 stars. 31 biomedical tools. 95+ skills. Deployable across eight messaging platforms.

One orientation note before we go further:

BioClaw is the biomedical repository under review. It is built on a NanoClaw container-based agent architecture and distributed through OpenClaw as its primary install and runtime path.
STEM-BIO-AI is our external audit scanner — a deterministic local CLI tool that evaluates repository evidence surfaces without executing the code or making network requests.

Three separate layers. Three separate risk profiles.

This is exactly the kind of tool a research institution can adopt after checking the science, the README, and the repository — while still missing the deployment risk that matters. The credentials do the convincing. The bioRxiv paper frames the science. The repository shows the capability. But nobody reads the architecture closely enough to ask the one question that matters.

We asked that question.

The answer — and the thesis of everything that follows — is this:

The exposure happens at the channel, not at the compute.

The model can be sophisticated. The containers can be clean. The bioinformatics toolkit can be comprehensive. None of that changes what happens to the data before it reaches any of those components.

For BioClaw, the highest-risk showcased intake path is WhatsApp.

The Line That Stopped Us

From the bioRxiv paper, describing BioClaw's supported data modalities (2):

"The application of BioClaw spans various biomedical domains (e.g., genomics, clinics, structural biology) and data modalities (e.g., sequencing data, EHR data, protein structure data)."

EHR. Electronic Health Records.

The paper explicitly names EHR data as a supported modality. BioClaw's README foregrounds WhatsApp as its primary showcase channel (1). Other messaging platforms are also supported.

The critique here is not "BioClaw equals WhatsApp."

The critique is that BioClaw's WhatsApp-first workflow is the highest-risk path for sensitive data — and it is the path the repository presents first.

EHR data. Through a consumer chat channel. In group threads.

This is not a minor documentation issue. It is a compliance collision.

To understand why, you need to look at the data path — not the abstract, not the paper's framing, but what happens at the moment a researcher hits send.

What the Data Path Shows

The README opens with this (1):

"BioClaw brings the power of computational biology directly into WhatsApp group chats."

Researchers message the BioClaw bot in a WhatsApp group. They upload files — sequencing data, gel images, CSV exports, protein structures. The system processes them in Docker containers and delivers results back to the same chat thread.

That architecture is the product.

The risk appears when that same workflow is placed beside the paper's claim that EHR data is a supported input modality.

Directly confirmed by repository and paper:

EHR data, sequencing files, and protein structure data are named as supported modalities in the bioRxiv abstract (2).
WhatsApp is the primary showcased channel in the README and Quick Start (1).
No PII or PHI scrubbing layer is visible in the repository between the consumer channel and the containerized agent runner.
No audit-log surface for data ingress is detected in the codebase.

Regulatory inference based on HHS guidance:

HHS guidance states that covered entities handling ePHI through cloud services or third-party platforms require a signed Business Associate Agreement with those services (3). WhatsApp cannot execute a HIPAA BAA under any configuration. For covered-entity deployment involving PHI or EHR data, that creates a compliance gap before any downstream model or container control can help.

Once an EHR file enters a group chat, it can replicate into unmanaged personal cloud backups, device storage, and participant-controlled environments. The chain of custody is not merely weakened. It is lost to the deploying institution at the moment of upload.

Platform-level risk from the OpenClaw runtime:

BioClaw is distributed and run through OpenClaw. In March 2026, Bloomberg reported Chinese government-level restrictions on OpenClaw use, citing broad data access and external communications capabilities (4). China's CNCERT/CC separately identified prompt injection as a critical threat vector, and security researchers rated one OpenClaw vulnerability at CVSS 8.8 (5).

NanoClaw's container architecture provides execution isolation. It does not contain what OpenClaw exposes at the runtime boundary.

That distinction matters because biomedical data is not ordinary data. A genomic sequence cannot be changed after a breach. It identifies not only the person who uploaded it, but biological relatives who never consented to the analysis. EHR data carries diagnoses, treatment histories, and identifiers that become protected health information the moment they move through an unmanaged channel.

When EHR Gets Out

The following cases establish the operational cost of EHR exposure. They are not analogies for BioClaw. They are the documented record of what happens when healthcare data moves through systems that were trusted before they were verified.

Change Healthcare, February 2024. Ransomware hit UnitedHealth's claims-processing subsidiary. EHR access froze across pharmacies, hospitals, and clinics nationwide. Final confirmed count: 192.7 million individuals affected — the largest healthcare data breach in U.S. history (6). Estimated cost to UnitedHealth: $2.87 billion. OCR opened a HIPAA compliance investigation. Litigation is ongoing; legal experts expect any settlement to exceed the $115 million Anthem precedent given scale (7).
Ascension Health, May 2024. One employee downloaded a malicious file. EHR systems went offline across 142 hospitals for nearly four weeks. Staff reverted to paper. Ambulances were diverted. Appointments were cancelled. 5.6 million patient records were ultimately compromised (8).
Esse Health, April 2025. Cyberattack on a Missouri physician group. 23,671 patients' EHR data exposed. Settlement reached: $2,525,000. Claims deadline: August 2026 (9).

Three incidents. Different attack vectors. One constant: once EHR data moves through an unmanaged channel or unverified infrastructure, the institution loses control of what happens next.

BioClaw's WhatsApp channel is an unmanaged channel. EHR data is the modality the paper explicitly lists as supported.

That is the standard BioClaw has to be evaluated against.

What STEM-BIO-AI Found

Finding	Evidence	Score Impact
Missing clinical/non-diagnostic boundary	Clinical-adjacent surfaces without explicit non-clinical boundary	Stage 2R: −20
No domain-specific tests	Zero bio-domain test surface detected	Stage 3 T2: 0/15
No changelog	No CHANGELOG or release-history file	Stage 3 T3: 0/15
C5 compliance boundary integrity	Clinical-adjacent content without non-diagnostic disclaimer	WARN

Final score: 60/100 — Tier 2 Caution. Research reference and supervised non-clinical technical review only.

The domain test score is zero. That means no automated check was visible to prevent the system from hallucinating a genomic sequence or silently returning a corrupted clinical summary back to the group chat.

No visible repository-level test validates that biological outputs are structurally sound, and no visible repository-level guard demonstrates that known PII patterns are scrubbed before sensitive data enters the workflow.

The changelog score is also zero. An institution that adopted BioClaw last month would have no documented record of what changed since.

The Stage 2R penalty of −20 reflects a structural fact: the paper names EHR data as a use case, and no part of the repository or paper adds a "not for clinical deployment" boundary.

That gap is what makes the tool adoptable in exactly the environments where it should not be deployed without further review.

What the Score Actually Means

60/100 is not a failing grade. It is a precise description of where the engineering stops.

A Tier 2 Caution score does not mean BioClaw is empty, broken, or unserious. That would be the wrong reading.

The audit found real engineering surface. Workflow files, dependency evidence, and several code-integrity checks were present. The repository is not a toy project, and the research problem it addresses is real.

But those signals do not answer the deployment question.

The weak points appear at the exact boundary where patient-adjacent use would require stronger evidence: no explicit non-clinical boundary, no domain-specific test surface, no changelog, no traceability manifest or runtime audit-log schema.

From the raw report: Stage 1 scored 70, Stage 2R scored 50, Stage 3 scored 54. The Stage 4 Replication score is 35 — reported separately and not factored into the final score, but indicative of a reproducibility posture that makes independent verification difficult.

That is the difference between a capable research repository and a deployment-ready biomedical system.

The signals that make BioClaw easy to adopt are precisely the signals that should trigger an independent audit — not replace one.

Three Cases. Same Assumption.

The prior section described what false assurance looks like with BioClaw specifically. This section shows how that same assumption plays out elsewhere — not the cost of EHR exposure, but the mechanism by which organizations arrive at the breach without ever running an independent check.

Tempus AI — Genetic Data, Commercial Pipeline, Alleged No Consent (2026)

Tempus AI acquired Ambry Genetics for $600 million in February 2025, transferring Ambry's genetic testing database. According to the consolidated class action Farrier et al v. Tempus AI, Inc., Tempus is alleged to have trained AI models on that data and disclosed it to more than 70 third-party clients in deals totaling $1.1 billion, without the knowledge or consent of data subjects (10)(11). These are allegations; no judgment has been entered.

The operative assumption: the acquisition process covered the data governance question. The consent scope of the original collection was never re-examined against the new use case.

Enzo Biochem — Infrastructure Trusted Because It Was Familiar (2023)

Ransomware hit Enzo Biochem's clinical test systems in April 2023. 2.47 million individuals affected. Investigators found shared credentials unchanged for years, no multi-factor authentication, and ineffective monitoring controls (12). Enzo paid $7.5 million to settle the class action and $4.5 million to three state governments.

The operative assumption: the infrastructure was safe because it had always been used that way.

The researcher who routes genomic files through WhatsApp because that is how the group already communicates is making the same assumption today.

Wisconsin Physicians Service — Tool Trusted Because It Was Widely Used (2023)

A breach at Wisconsin Physicians Service in May 2023 went undetected for over a year. 3.1 million individuals affected — names, Social Security numbers, Medicare identifiers, treatment dates (13). The third-party file transfer tool involved was widely trusted across the industry.

The operative assumption: widespread adoption equals verified safety.

Three organizations. Three different failure modes. One shared assumption: someone else already verified this.

No one had.

Stars Are Not Audits

BioClaw's version of false assurance is more sophisticated than most. It wears the clothing of academic legitimacy.

374 stars. A bioRxiv paper. Authors from Michigan, SJTU, CAMS, PUMC, UIUC, Westlake, HKUST. STELLA foundation with Stanford and Princeton correspondence authors.

None of that is a security audit. None of it constitutes a HIPAA risk analysis. bioRxiv is a preprint server — the paper has not been peer reviewed for security or compliance claims. The 374 stars reflect interest, not inspection.

Academic pedigree is the most effective form of false assurance in bio AI precisely because it signals rigor by proxy. It does not signal that anyone examined what happens when a researcher uploads an EHR export to a WhatsApp group to ask the bot a question.

The legal exposure does not fall on the repository maintainers. Open-source carries no HIPAA obligation. The liability belongs to the covered entity or business associate that deploys the tool against PHI or research genomic data and relies on star counts and institutional affiliations as the basis for that decision.

Under the HHS inflation-adjusted penalty schedule published in the Federal Register on January 28, 2026 — 2025 OMB inflation multiplier 1.02598 — exposure for covered-entity deployment involving PHI runs up to $73,011 per violation for willful neglect corrected within 30 days, and up to $2,190,294 per violation for willful neglect not corrected (14).

Stars are not audits. Pedigree is not a risk analysis. And after the breach, "we did not know" is no longer a serious governance position.

Three Checks Before You Deploy

1️⃣ Read the paper, not only the README.

BioClaw's README describes BLAST searches and protein structures. It does not mention EHR data. The bioRxiv paper names EHR data in the scope description, on page one. The clinical data modality that changes the legal posture is not in the README.

2️⃣ Audit the channel before the model.

Containerized execution, isolated Docker environments, and bioinformatics tooling are downstream of the data ingress point. When a researcher uploads a genomic sequence to a WhatsApp group, that file enters Meta's infrastructure before any BioClaw engineering control takes effect. It can also enter the device backups of every participant.

The model is not the exposure. The channel is.

3️⃣ Audit the stack, not just the repository.

BioClaw is the biomedical application. NanoClaw is the container agent architecture. OpenClaw is the install and runtime path — and the layer with documented government-level security restrictions and a CVSS 8.8 vulnerability (4)(5).

When a breach happens, the liability does not care which layer failed. It cares who deployed the stack without checking.

Final Thought

The researchers building BioClaw are solving a real problem. Bioinformatics workflows are fragmented. The friction of switching between command-line tools, visualization software, databases, and literature engines is real. Making that accessible through natural language in a shared group chat is a genuine architectural idea.

That is not the problem.

The problem is that a system capable of processing EHR data — data that is permanent, identifying, and biologically inherited — was presented with a consumer messaging platform as its primary showcased intake channel, with no visible audit trail, no visible PHI scrubbing layer, and no clinical-use boundary.

None of those gaps appear in the abstract, the README, the star count, or the institutional affiliations.

Behind every query is a potential patient. The sequencing file a researcher uploads to ask a question is not abstract data. It is someone's genomic identity.

The tool was built to accelerate research. Acceleration without containment is not a feature.

In bio AI, it is the failure mode.

Stars are not audits. They never were.

Every claim in this newsletter is backed by machine-generated audit data. The raw output for this evaluation — scanner findings, explain trace, experiment results, and full report artifacts — is publicly available for independent verification.

Flamehaven Verification Ledger:

https://flamehaven01.github.io/Flamehaven-Verification-Ledger/stem-bio-ai/bioclaw/2026-5-21/Runchuan-BU_BioClaw_report.html

Next Week

Awesome-Bioinformatics — danielecook/Awesome-Bioinformatics. 73,000+ stars. The most-starred bioinformatics resource list on GitHub. AI agents are already citing it as a trusted source — recommending tools from the list for clinical genomic workflows without a compliance layer in sight.

One audit question: When an AI agent recommends a tool from Awesome-Bioinformatics for genomic analysis, who is responsible for verifying that the recommendation is safe for clinical use?

The list doesn't say. The agent doesn't ask. The institution assumes someone already checked.

References

(1) Runchuan-BU/BioClaw — GitHub. Accessed May 2026.
(2) BioClaw: Human-Bot Research Collaboration Ecosystems in Group Chats — bioRxiv. Posted April 14, 2026. doi: 10.64898/2026.04.11.716807.
(3) Guidance on HIPAA & Cloud Computing — HHS.gov.
(4) China Moves to Limit Use of OpenClaw AI at Banks, Government Agencies — Bloomberg, March 11, 2026.
(5) China Restricts OpenClaw AI at Banks Over Security Flaws — Winbuzzer, March 12, 2026.
(6) Change Healthcare Data Breach: 192.7 Million Affected — The HIPAA Guide, August 7, 2025.
(7) Change Healthcare Data Breach: What Happened and What to Do — Security.org, March 2026.
(8) What We Can Learn from 2024's Top Healthcare Cyberattacks — Paubox.
(9) Esse Health Agrees to Pay $2.53M to Settle Data Breach Lawsuit — HIPAA Journal, May 2026.
(10) Healthcare AI Firm Sued Over Alleged Unlawful Disclosures of Genetic Data — HIPAA Journal, April 2026.
(11) Tempus AI Sued For Breach of Genetic Information Privacy Act — GenomeWeb, February 2026.
(12) Enzo Biochem to pay $4.5 mln for failing to safeguard patient data — Reuters, August 2024.
(13) CMS Notifies Individuals Potentially Impacted by Data Breach — CMS.gov, 2024.
(14) HHS Inflation-Adjusted Civil Monetary Penalty Schedule — Federal Register, January 28, 2026.

Beyond Repo Scanning: How AIRI Expanded the Risk Vocabulary in STEM BIO-AI 1.7.x

Fri, 15 May 2026 11:55:34 +0000

This is the second half of the same 1.7.x transition.

In the previous post, I wrote about calibration governance: how STEM BIO-AI keeps score authority from drifting when users simulate policy posture.

That was about how the system decides.

This post is about a different layer:

how the system speaks about risk.

A local repository scanner can become trapped inside its own vocabulary.

It can detect dependency issues, weak provenance language, shallow validation, reproducibility gaps, and risky exception handling.

But if every finding stays only inside the scanner's internal language, the report may remain too narrow.

That is the problem AIRI helped address in STEM BIO-AI 1.7.x.

In this context, AIRI is used as a local risk-vocabulary layer built from the MIT AI Risk Repository ecosystem.

The point is not to replace deterministic repository scanning with an external risk database.

The point is to give local findings a broader risk vocabulary without turning that vocabulary into a truth claim.

Basic AIRI Context

The MIT AI Risk Repository is a public AI risk resource from the MIT AI Risk Initiative.

It helps organize fragmented AI risk language across research, policy, and industry sources.

The repository includes three main parts:

an AI Risk Database
a Causal Taxonomy of AI Risks
a Domain Taxonomy of AI Risks

According to the MIT AI Risk Repository site, the database collects 1,700+ risks from 74 existing AI risk frameworks and classifications. The public domain taxonomy organizes risks into 7 domains and 24 subdomains.

Some of those domain taxonomy nodes include:

2. Privacy & Security
2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information
2.2 AI system security vulnerabilities and attacks
6.5 Governance failure
7. AI System Safety, Failures, & Limitations
7.3 Lack of capability or robustness
7.4 Lack of transparency or interpretability

That makes AIRI useful as a vocabulary source.

But vocabulary is not truth.

A local scanner should not say:

this repository caused this risk.

It should say something more careful:

this local finding belongs to a broader class of AI risk language.

That distinction is the design boundary.

What Problem AIRI Was Meant to Solve

STEM BIO-AI began as a deterministic evidence-surface scanner for bio and medical AI repositories.

That core remains.

The scanner looks at observable repository surfaces:

README and docs
code structure
CI configuration
dependency manifests
changelogs
reproducibility signals
clinical-adjacent boundary language

But once STEM BIO-AI started producing richer audit outputs, a new question appeared:

How should the system talk about the broader risk territory around a detected finding?

For example:

a fail-open exception path may have implications beyond code quality
weak provenance language may connect to reproducibility and trust concerns
shallow validation around sensitive inputs may point toward a wider harm surface than the repository alone makes obvious

Without a broader vocabulary, those findings remain local and narrow.

AIRI helps widen the vocabulary without making the scanner less deterministic.

A Short Note on Detector Families

In this article, a detector family means a bounded local analysis surface inside STEM BIO-AI.

It does not mean an AI model judging the repository.

Examples include:

code integrity detectors such as hardcoded credential or fail-open exception checks
AST contract detectors such as shallow validator checks
bio diagnostics such as SMILES parser-guard or silent mock fallback checks
provenance and reproducibility evidence surfaces

A detector family produces a local finding.

The AIRI layer does not replace that finding.

It gives the finding a broader vocabulary anchor.

AIRI Does Not Replace the Scan

This boundary matters.

The AIRI layer does not:

validate that a real-world incident happened
prove that a repository causes a given harm
turn a detector hit into a clinical danger claim
replace due diligence or domain review
override the deterministic score

Instead, it gives the system a structured way to say:

what broader risk territory a finding may relate to
which risk vocabulary exists around that class of concern
where known coverage gaps remain

That is why AIRI is a risk-vocabulary layer, not a truth layer.

If a report says something like:

covered risks: 12 / 31

that should not be read as:

the repository is 38% safe

or:

the scanner covers 38% of all AI risk

A better interpretation is:

within the detector scope currently mapped into the curated AIRI runtime layer, this scan triggered findings that connect to these AIRI risk entries.

That is narrower.

It is also more useful.

From External Repository to Local Governance Layer

The AIRI story in STEM BIO-AI changed during 1.7.x.

The initial direction was simple: use AIRI to provide broader risk labels around local findings.

That was useful, but not enough.

If an audit system relies on an external risk source, it needs governance around that source.

So STEM BIO-AI separates AIRI into three local layers:

Local layer	Purpose
`airi_registry_full.v1.json`	normalized full local registry derived from the upstream AIRI snapshot
`airi_runtime_bundle.v1.json`	curated runtime subset used by deterministic scans
`airi_detector_mapping.v1.json`	detector-to-risk mapping registry plus known-gap records

This separation prevents a common mistake:

confusing the full upstream AIRI universe with the smaller curated runtime bundle used by the scanner.

The scanner uses the curated runtime bundle, not the entire upstream AIRI universe.

That keeps runtime outputs deterministic, reviewable, and tied to a known local snapshot.

What “Governed” Means Here

In the current 1.7.5 state of the 1.7.x line, governed does not mean that every mapping has gone through an external review board.

It means something narrower and more concrete:

AIRI data is stored as versioned local artifacts
runtime scan output uses a curated bundle, not the entire upstream universe
detector mappings are separated from the full registry
known gaps are recorded as part of the mapping layer
artifact metadata surfaces AIRI registry, bundle, mapping, snapshot, and license information
changes to registry, runtime bundle, or mapping versions require explicit version bumps

That is the current governance level.

It is not final.

But it is stronger than attaching a risk dataset as an unversioned appendix.

The Curation Logic

This is the part that matters most.

AIRI is broad. STEM BIO-AI is narrow.

STEM BIO-AI does not need every AIRI entry active at runtime. It needs the subset that can be responsibly connected to deterministic repository evidence.

So the runtime bundle is curated by exclusion as much as inclusion.

A risk vocabulary node should stay outside the runtime bundle when:

No local evidence surface exists
The scanner has no repository-level signal that can responsibly connect to that risk.
The mapping would require causal inference
The scanner would have to imply that harm occurred, that users were affected, or that the repository caused a risk.
The risk is too broad for repository-local evidence
Broad societal, geopolitical, or macroeconomic risks may be important in AIRI, but they should not become runtime scan outputs unless a local detector surface can support the mapping.
The mapping would confuse vocabulary with score authority
If a risk label might be read as changing the formal score or certifying danger, it should remain outside the runtime layer until the reporting semantics are clear.

So the runtime bundle is not a summary of all AI risk.

It is the subset of risk vocabulary that the scanner can use responsibly.

Example: Detector Hit to AIRI Domain Vocabulary

A concrete example helps.

Suppose STEM BIO-AI detects a shallow validator around sensitive or clinical-adjacent inputs.

The local finding might be:

CC3_shallow_validator:
validate_* or check_* function uses only length checks without structural validation.

At the repository level, this is a code-contract finding.

It says:

the function appears to validate input
the validation is shallow
the implementation may not enforce the boundary implied by its name

The AIRI layer should not turn that into:

this repository caused privacy harm.

That would be too strong.

A safer mapping uses AIRI as vocabulary:

Local detector surface	Local meaning	AIRI vocabulary anchor
`CC3_shallow_validator`	validation function appears shallower than its name implies	`7.3 Lack of capability or robustness`; possibly `2.1 Compromise of privacy...` if sensitive information handling is in scope
fail-open exception path	code path may silently continue after failure	`7.3 Lack of capability or robustness`
hardcoded credential signal	repository surface suggests exposed secret-like pattern	`2.2 AI system security vulnerabilities and attacks`
weak provenance surface	repository gives weak evidence about data/source traceability	`7.4 Lack of transparency or interpretability`; possibly `6.5 Governance failure`
silent mock fallback	production-like path may fall back to simulated behavior	`7.3 Lack of capability or robustness`; `7.4 Lack of transparency or interpretability`

The mapping does not prove harm.

It tells the reviewer which broader AIRI vocabulary may be relevant to the local finding.

That is the difference between:

this detector proves a risk occurred

and:

this detector finding belongs near this risk-language area.

The second claim is weaker.

It is also the correct claim.

Why Local Provenance Matters

AIRI is external.

That means STEM BIO-AI needs to answer governance questions explicitly:

which upstream snapshot is being used?
which subset is active at runtime?
which risks are included in the curated bundle?
which risks are known gaps?
which detector maps to which AIRI entry?
what version of the mapping is active?

This is why the AIRI work matters.

It is not just adding labels.

It is turning risk vocabulary into a governed local data layer.

In the current governance note, the upstream source is recorded as:

upstream source: https://airisk.mit.edu/
upstream artifact: The AI Risk Repository V4_03
upstream license: MIT
local snapshot date: 2026-04-23

That provenance is not cosmetic.

It allows an audit artifact to say which risk vocabulary it was using when the scan was produced.

What Is Implemented in the Current 1.7.5 State of 1.7.x

The current AIRI layer is implemented, but bounded.

Implemented surfaces include:

AIRI-backed coverage surfaces in scan outputs
local curated runtime bundle
local registry and mapping schemas
detector-to-AIRI mapping layer
known-gap reporting
provenance and bundle/source labeling

In current scan results, airi_risk_coverage is the main artifact surface for this layer.

The public result contract includes AIRI fields such as:

airi_registry_version
airi_bundle_version
airi_mapping_version
airi_bundle_scope
airi_upstream_snapshot_date
airi_upstream_license
total_risks_in_registry
total_risks_in_bundle
total_risks_in_detector_scope
detectors_triggered
covered_risks
covered_count
coverage_rate
known_gaps_in_bundle
known_gaps_outside_bundle

These fields matter because they let a reviewer distinguish three things that are easy to confuse: the upstream AIRI source, the local runtime bundle, and the detector mapping actually used by the scan.

The important part is not only that these fields exist.

The important part is that AIRI usage becomes auditable from the artifact itself.

If two scans use different AIRI snapshots or mappings, that difference should not be hidden.

Coverage Is Not a Safety Percentage

AIRI coverage in STEM BIO-AI is an audit-surface concept, not a safety percentage.

It does not mean:

the repository is safe
the repository is unsafe
the scanner covers all AI risk
the covered percentage is a compliance score

It means:

a local deterministic finding has been mapped to a known risk-vocabulary entry inside the curated AIRI runtime layer.

That is useful because it gives reviewers a wider frame.

But it does not turn local evidence into a global safety claim.

This is the same discipline used elsewhere in STEM BIO-AI:

scoring is not clinical validation
advisory interpretation is not scoring authority
reproducibility evidence is not automatic score authority
AIRI coverage is not a safety percentage

Each layer has a role.

Each layer has a boundary.

What Changed in 1.7.x

The 1.7.x AIRI story is not simply “we added AIRI.”

The actual change was a move from loose risk labeling toward governed local vocabulary.

1.7.0

AIRI V4 integration appeared in scan outputs.

The scanner began producing an airi_risk_coverage section that maps triggered detector findings to AIRI risk IDs, coverage rate, and known gaps.

The same release also introduced Layer 2 AST contract detectors such as CC1, CC2, and CC3, which expanded the local detector surface available for risk-vocabulary mapping.

1.7.1

AIRI became a governed local data layer.

The architecture separated:

full local registry
curated runtime bundle
detector mapping registry

This release also replaced hardcoded AIRI detector mappings and known-gap lists with packaged local registry files.

Runtime outputs began surfacing registry version, bundle version, mapping version, upstream snapshot date, license, attribution note, and split known gaps into known_gaps_in_bundle and known_gaps_outside_bundle.

1.7.2

No major AIRI architecture change.

The important governance point was regression stability: same-target self-scan comparison verified no drift in airi_risk_coverage alongside score, tier, code contract, detector summary, and evidence ledger count.

1.7.3

No major AIRI architecture change.

The release focused on runtime cleanup, stale demo wording, layout stabilization, and output routing.

1.7.4

AIRI presentation became clearer across demo and report outputs.

The release surfaced AIRI summary material more clearly across the Hugging Face overview card and markdown/explain report sections.

1.7.5

No new AIRI data architecture change.

But artifact-level governance improved more broadly through additive evidence-ledger quality fields and audit-freshness metadata.

That matters because AIRI is most useful when it lives inside a report surface that already carries freshness, evidence quality, and provenance signals.

The important change across the line is this:

AIRI moved from attached dataset toward versioned local risk-vocabulary layer.

What This Still Does Not Do

The AIRI layer still does not:

verify real incidents
prove causality
certify repository safety
replace domain review
turn AIRI categories into deterministic truth claims
collapse the full upstream AIRI universe into the runtime scanner

Those are not missing features.

They are the boundaries that keep the layer useful.

Where This Could Go

The next useful direction is not to overload the scanner with external systems.

It is to improve:

registry provenance
bundle governance
mapping confidence
known-gap clarity
artifact-visible mapping metadata
disciplined links to incident-oriented resources

The broader MIT AIRI ecosystem also includes related incident-oriented resources such as the AI Incident Tracker.

That ecosystem is relevant context, but it is not the same thing as current runtime integration in STEM BIO-AI.

A future version may choose to reference incident-oriented resources more explicitly, but deterministic scans should not ingest them casually or blur them with repository-local findings.

A future version should be able to say not only:

this detector maps to this AIRI risk vocabulary area.

But also:

this mapping has this confidence level, this review status, this local evidence family, and this known limitation.

That is the next governance step.

Final Thought

That is the role of AIRI in this release line.

Not truth replacement.

Not safety certification.

Not incident proof.

A governed vocabulary bridge.

Local evidence first.

External vocabulary second.

Explicit provenance always.

References and Acknowledgment

MIT AI Risk Repository: https://airisk.mit.edu/
MIT AI Incident Tracker: https://airisk.mit.edu/ai-incident-tracker
STEM BIO-AI repository: https://github.com/flamehaven01/STEM-BIO-AI

This AIRI-related direction in STEM BIO-AI was informed by broader public AI risk work, including the MIT AI Risk Repository ecosystem.

The framing around AIRI as a broader risk-vocabulary layer, rather than a repository-local truth layer, was also strengthened by public commentary and ecosystem work from people in this space, including Peter Slattery, PhD.

These references informed the vocabulary and governance direction described here. They do not imply endorsement of STEM BIO-AI or responsibility for its implementation choices.

When Control Becomes Authority: Calibration Governance in STEM BIO-AI 1.7.x

Thu, 14 May 2026 05:50:41 +0000

When Control Becomes Authority: Calibration Governance in STEM BIO-AI 1.7.x

Control slowly becomes authority when nobody marks the boundary.

That is the calibration problem I kept running into while building STEM BIO-AI.

The project began as a deterministic evidence-surface scanner for bio and medical AI repositories. It inspected observable repository surfaces — README files, docs, code structure, CI, dependency manifests, changelogs, and boundary language — and mapped them to a structured review tier.

That was useful.

But the harder problem was not producing a number. The harder problem was preventing every useful adjacent signal from becoming part of that number.

In a bio/medical AI repository review, several lanes can look similar if the tool is not careful:

deterministic scoring
diagnostic findings
replication evidence
advisory interpretation
domain-specific review posture

They all matter. But they should not all have the same authority.

The principle behind the 1.7.x calibration work is simple:

easy experimentation, hard drift

Researchers should be able to express review posture. Operators should be able to simulate policy changes. Artifacts should show policy metadata.

But none of that should silently mutate the official score.

Current 1.7.5 Boundary

This post describes the released 1.7.x state as of v1.7.5, not a future authoritative-read-through design.

The formal score currently comes from three weighted score-bearing stages, plus explicit penalty and cap logic:

Surface	Role
Stage 1	README / stated evidence boundary
Stage 2R	repo-local consistency
Stage 3	code and bio-responsibility surface
C1	hardcoded-credential penalty
clinical cap / hard floor	score ceiling or T0 floor when clinical-adjacent boundary rules require it

Stage 4 exists, but it is a separate replication lane. It reports reproducibility posture without automatically changing the formal score.

In 1.7.5, STEM BIO-AI has implemented calibration architecture, but it remains mostly mirror-only and preview-oriented.

Implemented surfaces include:

packaged calibration profiles
schema and runtime validation
profile identity surfaced in result metadata
stem policy list
stem policy explain
stem policy derive
stem policy simulate
simulation-only local profile files
profile hashes and read-mode metadata in artifacts

The active named recommendation surface is intentionally narrow:

default
strict_clinical_adjacency

reproducibility_first is still a draft posture, not an active release-grade named recommendation.

The key boundary:

the authoritative scan scoring path is still protected from arbitrary user-provided profile mutation.

scan --policy <name> can surface selected profile metadata. policy derive and policy simulate can show governed preview behavior. But user-provided profile files do not become official scoring authority.

More specifically, local profile files are accepted only by stem policy simulate, and the CLI rejects them unless the file remains mirror_only.

That is not a missing convenience. It is the boundary being tested before it is allowed to become authority.

Why Calibration Is Not a Tuning Console

The wrong calibration UX looks like this:

{
  "stage_1_percent": 30,
  "stage_2r_percent": 25,
  "stage_3_percent": 45,
  "ca_no_disclaimer_cap": 61,
  "b2_partial_credit_mode": "looser"
}

This is editable. But editable is not the same as governed.

Most researchers do not think in raw score constants. They usually know something closer to this:

clinical-adjacent claims should be treated very strictly
reproducibility matters strongly in this environment
README polish should not outweigh code evidence
a casual mention of "limitations" should not count as meaningful transparency

So the current design starts with posture questions, not raw constants.

The `1–5` Scale Is Input, Not Authority

The user-facing intent layer uses a 1–5 scale:

1 = minimal emphasis
3 = moderate emphasis
5 = very strong emphasis

The important rule:

the 1–5 scale is a UX input surface, not part of the formal score engine.

Users can express clinical strictness, code-integrity priority, reproducibility priority, and structured limitations requirements. But those answers do not directly become score constants.

They pass through an explicit rule table:

Condition	Outcome
`clinical_strictness >= 4` and `reproducibility_priority <= 3`	recommend `strict_clinical_adjacency`
all four values are `2` or `3`	keep `default`
no named-profile rule matches	generate `preview_only` bounded deltas

This table is not an empirically optimized model. It is a conservative governance rule table.

The threshold choices are design-steward decisions, not claims of statistical optimality. Their purpose is to keep the translation layer narrow, reviewable, and non-authoritative until a stronger benchmark-backed promotion process exists.

For example:

clinical_strictness = 4
reproducibility_priority = 4

This does not automatically recommend strict_clinical_adjacency.

It falls back to preview_only, because two strong postures are competing and no release-grade named profile currently resolves that conflict.

A hidden similarity function might look more flexible. It would also be harder to audit.

What the CLI Is Allowed to Do

The preview workflow can look like this:

stem policy derive \
  --clinical-strictness 5 \
  --code-integrity-priority 4 \
  --reproducibility-priority 3 \
  --structured-limitations-requirement 4

or:

stem policy simulate /path/to/repo --profile-file my_profile.json

But those flows are not the same as:

stem scan /path/to/repo --stage1-weight 0.35 --cap 72

The first two are governed preview surfaces. The last one is an untracked tuning console.

The design supports the first and rejects the shape of the last.

That is the practical meaning of easy experimentation, hard drift.

What Actually Gets Verified

The central claim is not:

the current calibration rules are perfect.

The claim is narrower:

calibration changes should not become score authority without a visible governance path.

That can be checked by asking whether the system blocks or exposes the relevant mutation paths:

Drift risk	Expected control
arbitrary score tuning	no free-form CLI weight / cap override
hidden profile mutation	profile status and read mode surfaced
unclear profile identity	profile name, version, and hash visible
advisory influence leakage	advisory output cannot override `final_score`
reproducibility overcompensation	`replication_score` does not change `formal_tier`
premature named-profile expansion	ambiguous postures fall back to `preview_only`
detector promotion drift	detector policy is versioned in policy files and governance docs, though not yet first-class per-detector artifact metadata

This is not a full empirical benchmark. It is a governance verification target: the policy cannot quietly become authoritative without leaving a trace.

That trace is stronger for some surfaces than others. Profile identity, hash, and read mode are already artifact-visible in 1.7.5. Detector promotion semantics are versioned and documented, but not yet surfaced as first-class per-detector policy metadata in the result object.

The B2 Tightening Example

The clearest scoring example is Stage 3 B2: the bias and limitations measurement surface.

Earlier behavior allowed a weaker boundary: a simple vocabulary-level signal could still receive partial credit.

That became too permissive. A repository that mentions "bias" or "limitations" once is not necessarily disclosing a meaningful boundary. It may only be surface signaling.

So the B2 rule became stricter:

Case	Earlier posture	Tightened posture
no bias / limitations vocabulary	0	0
minimal single-term mention only	partial credit possible	0
structured limitations language	partial credit possible	partial credit possible
quantitative measurement evidence	full credit possible	full credit possible

The rule change creates a concrete score-path difference:

a repository that previously depended only on a minimal single-term limitations mention no longer has a B2 partial-credit path after the tightening.

That is the current public claim.

I am not presenting a benchmark-wide before/after score delta here, because that would require a pinned fixture set and published comparison protocol. Without that, a claimed "T3 became T2" example would be anecdotal at best.

So the honest evidence level is rule-level impact:

the credit path changed
the changed path is deterministic
the changed path is inspectable
benchmark-level deltas should be published only when the fixture protocol is pinned

In clinical-adjacent repositories, limitation language is not decoration. It is part of the claim boundary.

Why Stage 4 Stays Separate

The strongest counterargument is fair:

If reproducibility is important, why does it not affect the formal score?

My answer is that importance and score authority are not the same thing.

Stage 4 measures replication posture: containers, reproducibility targets, dependency locks, artifact references, seeds, citation surfaces, and similar evidence.

Those signals matter.

But they do not mean the same thing as the formal claim boundary.

A repository can be highly reproducible and still make unsafe clinical claims. It can have clean containers while lacking a clinical-use disclaimer. It can be easy to rerun while still having weak data provenance or shallow limitation language.

If Stage 4 lifted the formal score too early, reproducibility could compensate for claim-boundary weakness.

That would be a different scoring philosophy.

It may become valid in the future, but only if the rule is explicit.

For now, Stage 4 remains separate because reproducibility should be visible and review-relevant without silently overriding the formal score boundary.

That is why stronger reproducibility intent currently falls back to preview_only instead of becoming a release-grade named profile.

Advisory AI Uses the Same Boundary

Advisory AI follows the same rule.

Helpful interpretation is not score authority.

STEM BIO-AI can export provider-neutral advisory packets and validate downstream advisory responses, but the deterministic scanner does not need an external model runtime to produce the formal score.

Unless a future release explicitly changes the policy, advisory output remains structurally subordinate to the deterministic score.

The broader advisory boundary is a separate topic.

From Scoring Tool to Audit Workflow

The 1.7.x transition is best understood as a shift in questions:

Earlier scoring-tool question	Audit-workflow question
What score did the repository get?	Which policy profile was visible when the score was produced?
Which stage contributed most?	Was that stage score-authoritative, diagnostic, or separate-lane evidence?
What evidence triggered the tier?	Did the evidence change the formal score or only the review posture?
What should the user fix?	Would a proposed policy change be preview-only, experimental, benchmark-candidate, or release-authoritative?

The score still matters.

But the system is increasingly designed around the custody of the score: where it came from, what was allowed to influence it, and what was intentionally kept outside it.

What This Still Does Not Do

STEM BIO-AI still does not:

validate biomedical efficacy
certify benchmark truth
determine clinical deployment safety
let advisory AI overwrite the formal score
open arbitrary numeric tuning in the official scan path
allow profile experimentation to become official policy without governance

Those are not missing conveniences. They are boundaries.

A strong repository evidence tier is still an observable repository-surface signal. It is not clinical clearance, regulatory approval, or proof of biomedical validity.

The Next Version Direction

The next important step is not adding more knobs.

It is authoritative policy read-through in parity mode:

the default policy profile becomes the source read by the scoring path
existing fixtures should show no score or tier drift
policy hashes remain visible in artifacts
non-default and researcher-provided profiles remain governed preview surfaces until promoted
score-affecting policy changes become explicit release events

This is not a big-bang rewrite.

It is authority relocation.

The goal is to move score-affecting constants into versioned policy objects without changing the score by accident.

Final Position

The calibration problem is not really about giving users more control.

It is about deciding when control becomes authority.

If every useful signal can gradually influence the score, the score stops being an audit artifact.

It becomes a negotiation.

That is what STEM BIO-AI is trying to avoid.

Researchers should be able to express posture. Operators should be able to simulate alternatives. Policy stewards should be able to promote changes.

But the formal score should not move unless the governance path says it moved.

That is the difference between a tuning console and an audit system.

From Score to Workflow: Turning STEM BIO-AI Into a Local Audit System

Fri, 08 May 2026 08:39:48 +0000

Earlier in this series, I wrote about why bio/medical AI repositories need more than benchmarks, what I learned after auditing 10 public repositories, and why an AI auditor itself needs a memory contract.

That work led to STEM-AI v1.1.2 and the MICA layer: a memory-contracted initialization step that forces the auditor to load bounded rules before scoring begins.

For the broader arc:

STEM-AI / STEM BIO-AI series

But after that, a different engineering problem took over.

The audit logic was stricter.
The reports were richer.
The reasoning was more bounded.

But the developer workflow still felt too loose.

So the next question was no longer:

How do I score trust?

It became:

How does a bio-AI audit tool become something an engineer can actually run, gate, inspect, and integrate?

The answer turned out to be less about seeing more signals and more about refusing to confuse them.

That is the core argument of this post:

A detector becomes more trustworthy when it is strict about what it cannot conclude.

Once I took that seriously, STEM BIO-AI stopped looking like “one score plus some extra metadata” and started looking like a system with distinct lanes, distinct boundaries, and distinct operator workflows.

The problem was no longer scoring

By the time I reached the 1.6.x line, the rubric was no longer the main bottleneck.

The bottleneck was operational clarity.

A trust audit tool is not very useful if:

the normal path is one long command with too many flags
CI has to reverse-engineer the result from human-readable stdout
bio-specific diagnostics are mixed directly into the same surface as formal scoring
regulatory relevance shows up as vague implication instead of explicit traceability
advisory AI is present, but its relationship to the official score is unclear

At that point, the tool stops being hard to trust for conceptual reasons and starts being hard to trust for operational reasons.

That is a different class of problem.

The CLI had to reflect operator intent

The earlier CLI was functional, but too flat.

You could do things like this:

stem /path/to/repo --level 3 --format all --explain
stem /path/to/repo --tier-gate T3 --format json --quiet
stem /path/to/repo --advisory packet
stem /path/to/repo --advisory-response provider_advisory.json

All of that worked.

The issue was that it treated very different operator intents as one long option surface.

In practice, these are separate workflows:

scan a repository and generate artifacts
enforce a gate in CI/CD
export a bounded advisory packet
validate a downstream provider response
cross an explicit provider-call boundary

So I refactored the CLI around workflows instead of flag accumulation:

stem scan <folder>
stem gate <folder> --min-tier T2
stem advisory validate <folder>
stem advisory packet <folder>
stem advisory call <folder>
stem advisory check-response <folder> --response FILE

The older paths still exist for compatibility:

stem <folder>
stem audit <folder>
stem <folder> --tier-gate T2 --quiet
stem <folder> --advisory packet

But they are no longer the conceptual center.

That matters more than it sounds.

Once the command names match the operator’s intent, the system becomes easier to teach, easier to remember, and easier to wire into pipelines.

This is not just a DX cleanup. In a medical or bio-adjacent audit context, command ambiguity is part of the trust problem.

Repository trust needed four separate lanes

This was the biggest architectural shift.

I stopped treating repository trust as one object.

In practice, it needed four separate lanes:

deterministic structural scoring
deterministic diagnostics
regulatory traceability
optional AI advisory

If all of those collapse into one final confidence score, the tool becomes harder to reason about.

The more regulated the domain, the more dangerous it becomes to collapse every useful signal into one score.

Some evidence should change the score.
Some evidence should only raise review priority.
Some evidence should support traceability.
Some evidence should be handed to a human or advisory system.

The maturity of the tool is not that it sees all of them.

The maturity is that it does not confuse them.

This separation is not just conceptual. It exists in the code path.

One reasonable objection to any architecture write-up is: are these really separate lanes, or are they just different labels on the same output object?

In STEM BIO-AI, the answer is visible in the execution order.

The scanner computes the formal score first. In the result object, that means keys like:

Stage 1
Stage 2R
Stage 3
risk penalty
score cap
final_score
formal_tier

Only after that does it append the non-scoring layers, again as explicit result keys:

regulatory_basis
stage_traceability
regulatory_traceability
reasoning_model
optional ai_advisory

That ordering matters.

The score is not derived from the advisory lane.
The regulatory mapping does not mutate the formal tier.
The diagnostics lane can emit evidence without becoming a hidden score multiplier.

That is why the next four sections exist.

1. Deterministic structural scoring

This remains the official score and tier.

It measures the main repository-visible signals:

README evidence
repo-local consistency
code and bio responsibility
dependency hygiene
changelog and provenance surfaces
code-integrity patterns

This lane is local, deterministic, and machine-checkable.

That is the part that can legitimately drive a formal triage tier.

I am not claiming this is the only possible architecture. A different system could have folded diagnostics or replication more aggressively into one unified score.

I chose not to, because the narrower score proved easier to defend. A smaller claim with cleaner boundaries was more valuable here than a broader score with ambiguous semantics.

2. Deterministic diagnostics

This is where the deterministic diagnostics spec became important.

I needed a place for findings that are real, useful, and inspectable, but should not silently perturb the main score until they are calibrated.

That is what docs/DETERMINISTIC_DIAGNOSTICS.md defines.

It separates the diagnostic problem into two lanes:

Lane A: deterministic local diagnostics
Lane B: optional AI-assisted semantic review

That separation is central.

The deterministic lane is authoritative for hard findings.
The AI lane is advisory only.

The local diagnostic lane currently focuses on evidence-bearing bio-specific signals such as:

malformed or suspicious SMILES-like outputs
missing parser guards
silent mock or simulated-data fallbacks
risky subprocess construction around bio tools
traceability manifest surfaces

The point was not to create a “bio slop detector” with a catchy label.

The point was to create a local evidence lane that could say:

here is the file
here is the line
here is the snippet
here is the bounded interpretation

That is much more useful than a vague semantic warning.

Why diagnostics stayed evidence-only

This was one of the harder engineering decisions.

It would have been easy to push every new bio-specific detector directly into the final score.

I did not do that.

The deterministic diagnostics spec is explicit that many of these findings begin as evidence-only. In practice, they are emitted as line-level records in the result object's evidence_ledger:

findings are emitted into the result object’s evidence_ledger
findings appear in Markdown and --explain
findings do not change final_score or formal_tier

That is the right default.

For example, the SMILES lane can be very useful for detecting:

malformed surface strings
low-entropy placeholders
repeated trivial outputs
missing parser guards

But it does not prove:

medicinal usefulness
synthetic feasibility
binding plausibility
biological efficacy
full chemical validity in every edge case

That boundary is important.

A detector becomes more trustworthy when it is strict about what it cannot conclude.

Evidence-only is the temporary safe default until a detector has earned score authority through commit-pinned benchmark evidence, false-positive review, and reproducible calibration.

3. Regulatory traceability

The second document that became central was docs/REGULATORY_MAPPING.md.

This solved a different problem.

Once you audit clinical-adjacent repositories, people naturally ask:

does this align with EU AI Act themes?
does this help with FDA-oriented review?
is there anything relevant to IMDRF or SaMD evidence families?

The wrong answer would be to turn those questions into a fake compliance score.

So I did the opposite.

The regulatory layer is explicitly framed as:

a traceability aid, not a compliance verdict

That document maps observed evidence classes to requirement families with bounded confidence labels like:

strong
moderate
weak-moderate
weak
not assessed

the confidence applies to the mapping relationship, not to legal acceptability.

That means the tool can say things like:

versioned manifests and changelogs may support record-keeping / traceability review
intended-use and disclaimer sections may support transparency scaffolding review
override interfaces may support human-oversight interface review
subgroup measurement language may support weak evidence of data-governance intent

without claiming:

legal compliance
regulatory clearance
clinical certification
deployer conformance

In a regulated domain, traceability is useful only when it does not pretend to be permission.

A concrete example: why Article 12 is traceability, not compliance

The best example here is EU AI Act Article 12 style traceability.

The regulatory mapping layer treats signals like:

The regulatory mapping layer treats signals like changelogs, checksum manifests, versioned config surfaces, audit-log schema fragments, and decision-event or override-event schema tokens as evidence that a repository may have traceability scaffolding.

That is useful.

It is also bounded.

Changelog presence is not the same thing as deploy-time event logging. Current scope does not establish runtime log completeness.

So the output can legitimately say:

there is structural evidence relevant to traceability review

while refusing to say:

this system satisfies traceability obligations

What this buys in practice is not a compliance shortcut, but a faster review question. If a repository exposes none of the scaffolding signals in this lane, there is little reason to treat it as traceability-ready for deeper institutional review. If those signals do exist, the next step is still expert inspection, but the scanner has opened the right folder and pointed at the right files.

Why regulatory mapping stayed subordinate to evidence

Regulatory relevance had to remain downstream from evidence, not a score multiplier pretending to be law.

That is why the output shape separates things like:

regulatory_basis
stage_traceability
regulatory_traceability

from the actual score computation.

The regulatory basis object is registry-driven. It can mark review_required when the basis registry is stale or required source families are missing. That is a traceability control on the mapping layer itself, not an input into the scoring formula.

If a repo has traceability-relevant scaffolding, that is useful.

If a repo has traceability-relevant scaffolding, that is still not compliance.

The distinction has to remain visible in both the code and the artifacts.

4. Optional AI advisory

The fourth lane is the advisory layer.

This one exists for bounded model-assisted review, but it does not get to rewrite the official outcome.

That means workflows like:

stem advisory packet /path/to/repo
stem advisory check-response /path/to/repo --response provider_advisory.json

can exist without creating ambiguity about who owns the formal result.

The advisory layer can:

export a provider-neutral packet
validate downstream response structure
enforce finding-ID citation rules
reject prohibited claims
surface runtime and secret boundaries

What it cannot do is silently override:

score.final_score
score.formal_tier

How that rule is actually enforced

The advisory validator explicitly checks for score-override attempts. If a response includes fields like:

final_score
formal_tier
replication_score
replication_tier

or sets final_score_override, the response is marked invalid with final_score_override_requested.

The packet contract also exports the rule in plain language:

Do not modify or override final_score, formal_tier, replication_score, or replication_tier.

Provider responses must cite exact values from allowed_finding_ids; citation strings are not repaired or loosely matched later.

So the advisory lane is bounded in two ways:

it has no authority to change the deterministic result
it cannot cite evidence outside the bounded packet

That is the kind of mechanism I mean when I say “better boundaries.” If the rule cannot be checked, it is not really part of the architecture yet.

What operational use looks like now

Once these lanes were separated, the CLI became much easier to reason about.

Local engineering review:

stem scan /path/to/repo --level 3 --format all --explain

CI/CD gate:

stem gate /path/to/repo --min-tier T2 --summary off --output results

Offline advisory packet generation:

stem advisory packet /path/to/repo --output advisory_out

Downstream provider response validation:

stem advisory check-response /path/to/repo --response provider_advisory.json

The important point is not just that these commands exist.

It is that each one represents a distinct trust boundary.

That made the project feel more like engineering infrastructure and less like a scoring demo.

A real v1.6.2 packet

To make that less abstract, I re-ran STEM BIO-AI v1.6.2 against a local clone of ClawBio, which describes itself as a local-first, privacy-focused, reproducible bioinformatics-native AI skill library.

The command was:

python -m stem_ai.cli scan /path/to/ClawBio --level 3 --format all --explain

On my machine, the scan took about 9.4 seconds and produced the normal artifact set:

JSON for automation
Markdown for review
PDF for human-facing packet inspection
--explain for file / line / snippet proof tracing

The result was:

67 / 100
T2 Caution
Replication lane: 55 / 100 (R2)
Clinical adjacency: CA-DIRECT
Code integrity warnings: C2 dependency pinning, C4 exception handling

Two details mattered.

First, the scanner did not manufacture chemistry findings just because the repository was bio-adjacent. The SMILES diagnostics stayed not_detected or not_applicable.

Second, the scanner enforced observable repository conventions. Because ClawBio used ClawBio_README_Repo.md rather than a root README.md, the scan recorded S1_missing_readme: -20.

A human reviewer might contextualize that differently. The scanner does not. It records what the repository exposes through the surfaces it knows how to measure.

That is the point of the workflow: not to replace review, but to make the review surface inspectable.

What still has to stay bounded

The system is better than it was, but three boundaries still matter.

First, the public surface is broad. Scoring, diagnostics, replication, advisory packets, regulatory traceability, JSON, Markdown, PDF, and explain traces are all useful, but they increase onboarding cost.

Second, diagnostics should stay evidence-first until they have benchmark-backed calibration. Commit-pinned fixtures, reproducible output, and explicit false-positive review should come before score authority.

Third, regulatory confidence labels should remain bounded. They are rule-authored mapping labels, not runtime model outputs and not empirical proof of legal acceptability.

Those limits are not weaknesses.

They are part of the trust boundary.

Try it yourself

STEM BIO-AI is Apache 2.0 and fully open source.

If you want to know whether a bio/medical AI repository is actually exposing reviewable evidence, or whether your own repository is weaker than you think, run it yourself.

That is the real test.

GitHub: https://github.com/flamehaven01/STEM-BIO-AI
License: Apache 2.0

Final thought

The earlier STEM-AI posts were about why repository trust deserves its own audit layer.

This phase was about something more practical:

what does that audit layer have to look like if an engineer is actually going to run it, inspect it, and put it in a pipeline?

For me, the answer was simple:

Separate the workflows.
Separate the lanes.
Keep diagnostics evidence-first.
Keep regulatory mapping subordinate to evidence.
Keep advisory AI bounded.

Optimize for inspectability, not just score production.

That is what changed the project.

Not bigger claims.

Better boundaries.

STEM BIO-AI : Trust Bio Audit Framework

Tue, 05 May 2026 16:29:15 +0000

Release: STEM BIO-AI v1.5.8

I have open-sourced an internal BioAI governance scanner under Apache-2.0.

STEM BIO-AI is a deterministic evidence-surface scanner for bio/medical AI repositories.

Built for the checkpoint most teams skip.

Bio/medical AI repositories are hard to evaluate quickly and consistently.

Some are rigorous academic tools.
Some are early research prototypes.
Some look mature but carry clinical language before provenance, reproducibility, or clinical-use boundaries are clear.

That creates a governance gap.

Before a repository is trusted, adopted, integrated, or used as part of a clinical-adjacent workflow, teams need a basic answer:

What evidence is actually visible here?

STEM BIO-AI makes that repository evidence surface visible.

Core features:

▪️ No LLM
▪️ No API key
▪️ No model runtime
▪️ No secrets sent anywhere
▪️ Fast local or public GitHub repo triage
▪️ Deterministic scoring
▪️ T0–T4 evidence-tier output
▪️ JSON, Markdown, PDF, and explain-trace artifacts
▪️ Findings linked to file, line, pattern, or missing signal

How it works:

Run it on a local clone or public GitHub repository.

It checks four evidence lanes:

Claim Surface
Clinical language, hype claims, limitations, disclaimers, regulatory framing.
Repository Consistency
README, docs, metadata, tests, CI, workflow claims, version signals.
Engineering Accountability
CI/CD, domain tests, changelog hygiene, data provenance, bias / limitation evidence.
Replication Evidence
Containers, dependency locks, reproducibility targets, dataset or model references, CLI and citation signals.

Additional checks include hardcoded credentials, weak dependency pinning, deprecated patient-adjacent paths, and fail-open exception handlers.

Why this matters:

Most AI governance discussions start at the model layer.

But in BioAI, risk often appears earlier — in repository claims, missing provenance, weak reproducibility, stale tests, undocumented assumptions, and demos that look more mature than their evidence supports.

STEM BIO-AI is designed for that earlier review point:

before procurement,
before pilot review,
before technical due diligence,
before an AI demo becomes an AI system.

STEM BIO-AI is not a generic linter.
Not a benchmark leaderboard.
Not an LLM-based reviewer.

It is a repository-level evidence scanner for bio/medical AI governance.

Built for:

▪️ vendor screening
▪️ repository evidence review
▪️ technical due diligence
▪️ pre-procurement / pre-pilot checks
▪️ reviewer-ready audit artifacts

The goal is simple:

not more trust language,
but more inspectable evidence surfaces.

STEM BIO-AI resources:

GitHub repository:
https://github.com/flamehaven01/STEM-BIO-AI
Live demo:
https://huggingface.co/spaces/Flamehaven/stem-bio-ai
Technical notes:
https://dev.to/flamehaven01/series/37552
Case study:
https://flamehaven.space/writing/i-audited-10-open-source-bio-ai-repos-most-could-produce-outputs-few-could-establish-trust/

How Do You Trust the AI Auditor? STEM-AI v1.1.2 and Memory-Contracted Bio-AI Audits

Tue, 28 Apr 2026 13:52:59 +0000

Previous article:
How Auditing 10 Bio-AI Repositories Shaped STEM-AI

In the first STEM-AI write-up, I described what happened after auditing 10 open-source bio/medical AI repositories.

The important lesson was not just that some repositories lacked clinical disclaimers, tests, or governance artifacts.

The more useful lesson was this:

Text-only review is too weak for bio/medical AI. You have to inspect the code path.

That worked.

But it exposed the next problem.

If an AI system is auditing another AI or bioinformatics repository, how do you trust the auditor?

LLMs drift.
One session can enforce a clinical boundary strictly.
Another can invent a generous middle score for the same boundary case. In normal software review, that is annoying. In medical AI governance, it is a liability.

STEM-AI v1.1.2 is my answer to that problem.

It does not try to make the LLM deterministic by writing a longer prompt.

It binds the audit to a memory contract.

What v1.1.2 adds

STEM-AI v1.1.2 introduces MICA: Memory-Injected Contract Architecture.

The idea is simple:

before the auditor reads the target repository, it must load a fixed audit contract and self-check the rules it is not allowed to bend.

The v1.1.2 layer includes:

memory/mica.yaml -- composition contract
memory/stem-ai.mica.v1.1.2.json -- machine-checkable memory archive
memory/stem-ai-playbook.v1.1.2.md -- session playbook and drift guard
memory/stem-ai-lessons.v1.1.2.md -- historical failure-mode archive
spec/STEM-AI_v1.1.2_CORE.md -- canonical audit spec

The contract pins 18 invariants.

Examples:

Stage order is fixed: README intent, cross-platform evidence, code/bio evidence.
Stage weights are fixed.
Tier boundaries are fixed.
T0_HARD_FLOOR cannot be bypassed.
Stage 2 may use external evidence or Stage 2R repo-local consistency in LOCAL_ANALYSIS mode.
Governance overlay cannot raise the formal base tier.
C1-C4 code-integrity checks only run in LOCAL_ANALYSIS mode.
Mandatory clinical-use disclaimers cannot be omitted.

This is not a claim that the LLM becomes perfectly deterministic.

It is a narrower claim:

The auditor is forced to operate inside a contract whose scoring rules, hard floors, and evidence requirements are inspectable.

That is the useful layer.

What "loading the contract" means

MICA is not hidden model memory.

It is also not a claim that the model provider changed the LLM.

In v1.1.2, "loading the contract" means the audit session starts by reading a fixed set of repository files before it is allowed to score the target:

memory/mica.yaml
memory/stem-ai.mica.v1.1.2.json
memory/stem-ai-playbook.v1.1.2.md
memory/stem-ai-lessons.v1.1.2.md
spec/STEM-AI_v1.1.2_CORE.md

The auditor then performs a pre-execution contract test:

confirm the canonical spec exists
confirm the memory archive exists
confirm the invariant count is 18
confirm the fixed tier boundaries are present
confirm the Stage 2 / Stage 2R lane rule is present
confirm Stage 3G cannot raise the formal tier
confirm C1-C4 mode gating is active

Only after that does the audit proceed.

This does not make the LLM mathematically deterministic.

It makes the audit procedure file-backed, inspectable, and interruptible. If the session cannot load or reconcile the contract files, the correct behavior is to stop before scoring.

That is the difference between "please be consistent" and "execute this versioned contract."

The audit workflow

STEM-AI v1.1.2 runs as a structured audit workflow:

In LOCAL_ANALYSIS mode, the auditor is not limited to what the README says.

It can inspect:

package metadata
workflow files
test definitions
dependency manifests
source-code paths
deprecated or dead-code paths
exception handling
credential patterns
provenance and hash-checking logic

The output is intentionally split into two files:

report.md                  # human-readable audit judgment
experiment_results.json    # machine-readable evidence and score object

That split matters.

The report explains the reasoning.

The JSON lets another reviewer inspect the score, evidence fields, flags, and integrity checks without trusting the prose.

A real target audit, not a synthetic example

For this v1.1.2 demonstration, I used a real public repository:

artic-network/fieldbioinformatics

The target is not the protagonist of this post.

It is only the specimen used to show the audit workflow against a real bioinformatics codebase.

The local audit produced:

audits/fieldbioinformatics_v1_1_2/report.md
audits/fieldbioinformatics_v1_1_2/experiment_results.json

The target snapshot:

{
  "name": "artic-network/fieldbioinformatics",
  "remote": "https://github.com/artic-network/fieldbioinformatics",
  "branch": "master",
  "commit": "8008b4c97c2193a82308ff6f0be507b1d9306e36",
  "file_count": 114
}

This is the important part: the audit did not ask, "Does this README sound trustworthy?"

It asked:

Do README claims match actual package metadata and entry points?
Are there real CI and domain-specific tests?
Are dependencies reproducible enough?
Are there credential leaks?
Are there deprecated patient-adjacent paths?
Do clinical-adjacent output paths fail closed?
Does the repository include governance evidence, or only governance absence?

That is where STEM-AI is useful.

The score object

The machine-readable result records the score like this:

{
  "stage_1_readme_intent": 65,
  "stage_2_cross_platform": null,
  "stage_2_repo_local_consistency": 75,
  "stage_2_lane": "STAGE_2R_REPO_LOCAL_CONSISTENCY",
  "stage_2_policy": "External Stage 2 was not collected; LOCAL_ANALYSIS used Stage 2R in the fixed 0.20 Stage 2 slot.",
  "stage_3_code_bio": 55,
  "weights": {
    "stage_1": 0.4,
    "stage_2": 0.2,
    "stage_3": 0.4
  },
  "risk_penalty": 0,
  "final_score": 63,
  "formal_tier": "T2 Caution"
}

External Stage 2 is explicitly represented as null for this local-only audit.

That does not mean cross-platform consistency is unimportant.

It means this evidence slice was deliberately scoped to LOCAL_ANALYSIS. Instead of pretending to have social/web evidence, v1.1.2 uses Stage 2R: Repo-Local Consistency.

Stage 2R asks whether the repository's own surfaces agree with each other:

README vs package metadata and CLI entry points
README vs docs, tutorials, and troubleshooting
README test claims vs CI workflow and test definitions
clinical-adjacent outputs vs local intended-use boundaries

The contract defines the fixed-weight calculation:

Final = (Stage 1 x 0.40) + (Stage 2R x 0.20) + (Stage 3 x 0.40) - Risk Penalty
      = (65 x 0.40) + (75 x 0.20) + (55 x 0.40) - 0
      = 63

The final tier is therefore:

T2 Caution

Not because the prose sounded balanced.

Because the contract math forces that result.

Why the T0 hard floor did not trigger

T0_HARD_FLOOR is the rule that prevents a clinically dangerous repository from escaping rejection through good wording.

In simplified form:

If a repository is CA-DIRECT
and it has no substantive code implementation,
then final tier = T0 regardless of score math.

Examples of CA-DIRECT include patient-specific diagnosis, treatment recommendation, triage, risk scoring, or clinical decision support.

The audited repository did not trigger that floor because STEM-AI classified it as:

{
  "clinical_adjacent": true,
  "ca_severity": "CA-INDIRECT",
  "t0_hard_floor": false
}

It produces biological sequence artifacts that may sit near public-health or clinical workflows, but the inspected surface did not make direct autonomous diagnosis or treatment claims. It also has substantive implementation, CI, and domain-specific test definitions.

So the result is not T0.

But it is also not high-trust.

The bounded result is T2 Caution.

Code-integrity findings

The same JSON records C1-C4 LOCAL_ANALYSIS checks:

{
  "C1_hardcoded_credentials": {
    "status": "PASS"
  },
  "C2_dependency_pinning": {
    "status": "WARN"
  },
  "C3_dead_or_deprecated_patient_adjacent_paths": {
    "status": "WARN"
  },
  "C4_exception_handling_clinical_adjacent_paths": {
    "status": "WARN"
  }
}

That is the difference between a general review and a code-path audit.

A text review can say:

The project appears technically mature.

A code-path audit can say:

Credential patterns were checked. Dependency pinning is weak. Deprecated patient-adjacent metadata exists. One clinical-adjacent filtering path does not fail closed on missing depth.

That is a more useful governance object.

It is not a certificate.

It is a map of what a reviewer should trust, distrust, or inspect next.

A small Python verifier

Here is a small dependency-free Python script that reads the actual audit JSON and verifies the score calculation. It does not need target private code or patient data; it only checks the machine-readable audit result.

#!/usr/bin/env python3
import json
from pathlib import Path


RESULT = Path("audits/fieldbioinformatics_v1_1_2/experiment_results.json")


def tier(score: int) -> str:
    if score <= 39:
        return "T0"
    if score <= 54:
        return "T1"
    if score <= 69:
        return "T2"
    if score <= 84:
        return "T3"
    return "T4"


def bar(score: int, width: int = 20) -> str:
    filled = round((score / 100) * width)
    return "█" * filled + "░" * (width - filled)


data = json.loads(RESULT.read_text(encoding="utf-8"))
score = data["score"]

stage_1 = score["stage_1_readme_intent"]
stage_2 = score.get("stage_2_repo_local_consistency")
stage_3 = score["stage_3_code_bio"]
risk_penalty = score["risk_penalty"]

weights = score["weights"]
computed = round(
    (stage_1 * weights["stage_1"])
    + (stage_2 * weights["stage_2"])
    + (stage_3 * weights["stage_3"])
    - risk_penalty
)

assert computed == score["final_score"]
assert tier(computed) in score["formal_tier"]

print(f"Stage 1  {stage_1:3}/100  {bar(stage_1)}")
print(f"Stage 2R {stage_2:3}/100  {bar(stage_2)}")
print(f"Stage 3  {stage_3:3}/100  {bar(stage_3)}")
print(f"Final    {computed:3}/100  {bar(computed)}")
print(f"Tier     {score['formal_tier']}")

for key, item in data["code_integrity"].items():
    print(f"{key}: {item['status']}")

Expected digest:

Stage 1   65/100  █████████████░░░░░░░
Stage 2R  75/100  ███████████████░░░░░
Stage 3   55/100  ███████████░░░░░░░░░
Final     63/100  █████████████░░░░░░░
Tier      T2 Caution
C1_hardcoded_credentials: PASS
C2_dependency_pinning: WARN
C3_dead_or_deprecated_patient_adjacent_paths: WARN
C4_exception_handling_clinical_adjacent_paths: WARN

Why this matters

Bio/medical AI governance is full of language that sounds safe but is hard to verify:

"research use only"
"not medical advice"
"validated pipeline"
"clinical-grade"
"responsible AI"
"human-in-the-loop"

Those phrases are not enough.

STEM-AI asks for observable structure:

source-code reality
test reality
CI reality
dependency reality
clinical boundary reality
governance artifact reality
code-integrity reality

v1.1.2 adds another layer:

auditor reality.

The AI auditor itself has to load a memory contract before it scores.

That is what MICA is for.

The final answer is T2 Caution: research reference and supervised non-clinical technical review only. No autonomous clinical decision support.

Not hype.

Not rejection by default.

A bounded trust judgment with evidence paths.

What comes next

The follow-on lane should:

provision the target dependency environment
run selected target tests in a controlled shell
capture command, exit code, environment hash, and output digest
attach a replay manifest to experiment_results.json
keep runtime evidence separate from source/document/CI evidence

For the current demonstration, runtime execution status is recorded as an evidence boundary in the audit JSON. The score itself remains based on the official v1.1.2 LOCAL_ANALYSIS evidence basis: Stage 1 source/README evidence, Stage 2R repo-local consistency, Stage 3 code/bio evidence, and C1-C4 integrity checks.

Final thought

STEM-AI is not a clinical certifier.

It is also not trying to replace scientific review, regulatory review, or domain experts.

Its role is narrower: make the governance conversation start from observable evidence instead of presentation quality.

In practice, that means asking:

What did the repository claim?
What does the code actually implement?
Do the local surfaces agree with each other?
Are the tests domain-specific or merely infrastructural?
Are clinical-adjacent boundaries explicit?
Can the auditor's own scoring logic be inspected?

That is where I think STEM-AI belongs in AI governance.

Not as the final authority.

As the evidence gate before authority is invoked.

It turns a vague question, "Do we trust this bio/medical AI repository?", into a more reviewable one:

Does this repository establish enough observable trust to be considered, contained, or rejected?

When an AI Pipeline Passes — But One Path Still Must Be Held: EXP-034

Mon, 27 Apr 2026 10:23:06 +0000

No efficacy, causal, or clinical claims are made in this report.
RExSyn is an experimental Bio-AI governance pipeline.

You do not need to know the earlier experiments to read this report.

Most AI pipeline reports ask one question:

Did the system pass?

EXP-034 asked a stricter one:

Which path was allowed to count?

That distinction matters.

In a multi-stage AI pipeline, a final PASS can hide a lot of unresolved risk. A branch may be unstable. A regeneration path may drift. A new external API may enter the chain without being governed. A new modality may appear to improve the system while quietly changing the basis of judgment.

So EXP-034 was not designed to produce a clean success story.

It was designed to separate three things:

Path	Status	Meaning
Anchored expansion path	`GO`	Accepted path for EXP-034 reporting
Current regeneration path	`HOLD`	Diagnostic evidence, not acceptance baseline
Next remediation cycle	`EXP-035`	RCA and repair target

That is the real result.

EXP-034 passed, but not because every path passed.

It passed because the accepted anchor remained stable, the expansion tracks did not break the judgment system, and the unresolved regeneration path was explicitly held instead of being silently mixed into acceptance.

What EXP-034 tested

EXP-033 had already established a parity baseline.

EXP-034 asked whether that baseline could survive controlled expansion while adding:

a modal update track,
a live AlphaFold EBI observer endpoint,
and AlphaGenome / AG measurement.

The operating rule was simple:

Reproduce the parity baseline first.
Only then allow expansion.
Only then compare governance behavior across experiment cycles.

If the parity anchor breaks, the rest is not expansion.

It is regression.

The scope was also locked: methodology, governance, and reproducibility only. The experiment did not claim biological efficacy, causal inference, or clinical recommendation.

That boundary is important because this kind of system can easily sound more powerful than what was actually measured. EXP-034 was not asking whether the pipeline discovered a better biological answer.

It was asking whether the judgment system stayed governable after new signals entered the chain.

The key split: PASS did not mean everything passed

Track-A produced the defining decision of the experiment.

The accepted legacy replay anchor preserved the required PASS/BLOCK separation:

Metric	Legacy replay anchor
sample accuracy	`1.0`
sample balanced accuracy	`1.0`
arm accuracy	`1.0`
arm balanced accuracy	`1.0`
dangerous false-pass rate	`0.0`
false reject rate	`0.0`

That was the path allowed to anchor EXP-034.

But the current regeneration path did not recover:

Metric	Current regeneration
sample accuracy	`0.5`
sample balanced accuracy	`0.5`
status	`HOLD`

This is the most important part of the experiment:

EXP-034 did not pretend the regeneration path passed.

It kept that result inside the experiment as diagnostic evidence, but did not allow it to redefine the accepted baseline.

That separation is not a minor operational detail. It is the governance result.

A weak pipeline would have blended the two paths and still reported a final success. EXP-034 did the opposite. It allowed the stable anchor to proceed and held the unstable path for RCA.

That is how a stage-gated system avoids changing its own question after seeing the result.

Why path splitting matters

The concrete governance problem is this:

A pipeline can pass for the wrong reason.

valid_report = stable_anchor × traceable_extension × contained_instability

If the anchor is not stable, the report cannot be trusted.

If the extension is not traceable, the new signal becomes an ungoverned side channel.

If instability is not contained, a diagnostic failure can quietly contaminate acceptance.

A single final PASS is not enough when several branches contribute to a verdict. You need to know which branch produced the accepted decision, which branch failed, which branch was only diagnostic, and which branch is allowed to affect future work.

EXP-034 passed because all three conditions were enforced:

the legacy replay anchor held,
the new observer and AG paths were measured under governance,
and the regeneration HOLD remained outside acceptance.

That is the difference between a pipeline that merely outputs a verdict and a pipeline that controls which verdicts are allowed to count.

Adding AlphaFold EBI as an observer, not a predictor

Relative to EXP-033, EXP-034 added a live AlphaFold Protein Structure Database / EBI observer line.

This was not promoted into a primary predictor.

It was wired as an observer/reference oracle and traced into governance as ebi_g2.

The result:

Check	Result
AlphaFold EBI direct endpoint for `P23219`	`GO`
Stage 7 observer tests	`2 passed`
`ebi_g2` governance traceability	`PASS`
`BLOCKED_IDP` mapping path	validated in test

The point is not simply that an external endpoint responded.

The point is that the external signal entered the system through a governed path. It was not allowed to float beside the pipeline as informal context.

EXP-034 tested whether the new observer could be admitted without becoming an ungoverned side channel.

AG-live: non-degradation, not repair

Track-C tested a simple question:

If AG-live enters the pipeline, does it change the final decision?

The answer was no.

AG-live did enter the pipeline.

The AlphaGenome field was present with:

AG field	Value
source	`alphagenome_api_live`
pathogenicity_score	`0.5`
confidence	`0.7143`
clinical_significance	`uncertain`

These are sanitized branch artifact values, not implementation code or full raw artifacts.

AG-live did not change classification.

Both controls remained governed by the same conservative decision boundary:

Path	Expected	Observed	Interpretation
`EXP032-BLOCK-001` negative control	`BLOCK_EXPECTED`	`BLOCK / ESCALATE`	fail-closed behavior preserved
`EXP032-PASS-001` pass-eligible control	`PASS_ELIGIBLE`	`BLOCK / ESCALATE`	conservative over-blocking persisted

That is the key nuance.

AG-live did not create a dangerous false-pass. The negative control stayed blocked.

But AG-live also did not repair the current regeneration hold. The pass-eligible control still failed to recover and remained blocked under R2_component_floor.

The governance surface moved slightly, but the verdict did not:

Metric	Earlier AG branch	AG-live branch
`p_e2e`	`0.0912`	`0.0947`
clinical status	`BLOCK`	`BLOCK`
rule	`R2_component_floor`	`R2_component_floor`

So the correct conclusion is not:

AG improved the pipeline.

The correct conclusion is:

AG-live changed the measurement surface slightly, but did not change the decision boundary.

That is exactly what non-degradation means here.

It preserved fail-closed behavior on the negative control while leaving the pass-eligible control over-blocked.

This is why Track-C can only be called non-degradation, not repair.

Contract passed, but governance still blocked

One of the most useful details in EXP-034 is that the contract layer and governance layer did not collapse into one verdict.

The contract inspection reported:

Field	Value
pipeline contract score	`0.9077`
weakest connection	`C2`
dangerous pass risk	`0.0`
gate recommendation	`PASS`
overall OK	`true`

But the clinical governance layer still blocked the case.

That is not a contradiction.

It means the pipeline connection was valid enough to inspect, but the decision was not safe enough to accept.

This distinction matters.

A weaker system might treat a passing contract as permission to pass the whole output. EXP-034 did not do that. It allowed the contract layer to say:

The pipeline is connected.

while the governance layer could still say:

The claim should not pass.

That separation is exactly what a governance layer is supposed to preserve.

Cross-cycle comparison: EXP-032 → EXP-033 → EXP-034

Track-D compared the accepted anchor path across cycles.

You do not need the earlier experiments as background. They matter here for one reason only:

EXP-034 was not allowed to invent a new success criterion.

EXP-032 and EXP-033 provided the previous PASS/BLOCK baseline. EXP-034 tested whether that baseline survived expansion.

The classification baseline stayed fixed:

Compare	Accuracy / balanced accuracy
EXP-032 → EXP-034	`1.0 / 1.0`
EXP-033 → EXP-034	`1.0 / 1.0`

At the same time, governance signals moved:

Governance signal	Delta
`ccge_p_e2e_mean`	`+0.04447488775996111`
`nnsl_sr9_tech_mean`	`+0.04692394788063081`
`nnsl_di2_tech_mean`	`-0.03667940951579321`

The interpretation is narrow:

The judgment baseline stayed fixed while the governance surface became more measurable.

That is what EXP-034 was allowed to claim.

It did not prove biological efficacy.

It did not prove that every branch of the system was now stable.

It proved that controlled expansion could happen without breaking the accepted PASS/BLOCK baseline.

Stage-gate result

EXP-034 ended with all five stage gates passing:

Gate	Status
G1 parity	`PASS`
G2 reproducibility	`PASS`
G3 cross-experiment compare	`PASS`
G4 governance traceability	`PASS`
G5 extension safety	`PASS`

Final state:

Field	Value
overall status	`PASS`
anchor mode	`legacy_replay`
first failed gate	`null`
diagnostic hold	`Track-A current regeneration`

This is the important nuance:

The experiment passed with a retained diagnostic hold.

That is not a contradiction. It is the point of the control system.

The accepted anchor path was allowed to proceed. The current regeneration path was not. The remediation target was moved to EXP-035.

That separation is the actual proof EXP-034 provides: not that every branch became stable, but that instability was not allowed to contaminate acceptance.

What EXP-034 actually showed

EXP-034 did not show that the entire pipeline is now stable.

It showed something narrower and more useful:

A method-locked Bio-AI governance pipeline can admit modal expansion, AlphaFold EBI observer wiring, and AG-live measurement without losing its accepted PASS/BLOCK baseline — while keeping the unstable regeneration path out of acceptance.

Track-C sharpened that conclusion.

AG-live entered.
Metrics moved slightly.
The verdict did not change.
Dangerous false-pass did not appear.
Conservative over-blocking remained.

That is not a clean success story.

It is a governed result.

Closing

Stage-gated experimentation is not just about getting a result.

It is about deciding whether the result should be allowed to exist.

In EXP-034, the answer was:

GO   for the anchored expansion path
HOLD for current regeneration
NEXT for EXP-035 remediation

That may sound less dramatic than a clean success story.

But in governance work, that is exactly the point.

A mature AI pipeline is not the one that claims everything passed.

It is the one that can say:

This path passed.
This path did not.
And we did not mix them.

How Auditing 10 Bio-AI Repositories Shaped STEM-AI (Trust Evaluator for Medical AI Artifact)

Mon, 30 Mar 2026 12:12:10 +0000

Reading path:
This post is part of a series.
(1) STEM-AI introduction — what the framework is and why we built it
(2) Technical audit report — full findings across 10 repositories
(3) Narrative summary — what those findings actually mean

What Text Could See — and What Code Revealed

In March 2026, we ran STEM-AI against 10 high-visibility open-source bio/medical AI repositories.

The framework did what it was designed to do. It surfaced missing disclaimers, absent CI, weak reproducibility signals, and public-facing governance gaps. Those findings mattered, and the scores were directionally right.

But when we reviewed the audits more carefully, one pattern kept appearing: some of the most consequential failures were not visible in the artifact surface at all. They only became obvious when we looked directly at the code.

This function lives inside a repository presented as an AI-driven drug discovery workflow.

def generate_analogues(self, seed_smiles: str, count: int = 3):
    """
    Mocks a generative model (like REINVENT).
    In a real app, this would call a PyTorch model.
    """
    # Simple string manipulation for demo purposes
    analogues = []
    for i in range(count):
        if "C" in seed_smiles:
            new_smi = seed_smiles.replace("C", "C(C)", 1) if i == 0 else seed_smiles + "F"
            analogues.append(new_smi)
        else:
            analogues.append(seed_smiles + "C")
    return analogues

It does not generate molecules. It appends characters.

SMILES (Simplified Molecular Input Line Entry System) is a strict notation for molecular structure. A valid SMILES string encodes real geometry and bonding. Appending C produces a syntactically valid string that represents no real compound. The function runs without error, returns a list, and the pipeline continues downstream.

Our framework scored this repository T0. Correctly. But not because it saw this function.

It scored T0 because the README was missing disclaimers. The CI was absent. Reproducibility was undocumented. Text-path evaluation is designed to measure exactly that. It did.

The audit result was correct. The evidence surface had room to go deeper.

Running the audits showed us what code-path evaluation could add on top.

What Code Access Makes Visible

The drug discovery example was not unusual.

CellAgent's pipeline ends with this call:

# main.py, line 153
final_result = planner.generate_final_result(results)

The method exists. Its body is pass. The pipeline completes without error and produces nothing. A text audit reading the README would have no way to know this.

BioAgents includes a rate limiter for external API calls:

// rateLimiter.ts, lines 62-68
if (process.env.USE_JOB_QUEUE === 'true') {
    await this.rateLimiter.consume(userId);
} else {
    // Job queue disabled - skip rate limiting for direct calls
    return next();
}

USE_JOB_QUEUE defaults to false in .env.example. Every default deployment has rate limiting disabled. The function name implies protection. In default operation, there is none.

The pattern across all three:

the code looks governed.
the behavior tells a different story.
That story is only visible when you read the code.

Text scores and code behavior can diverge. Knowing where and how they diverge is the next layer of evidence worth capturing.

Four Directions the Audits Opened

Reviewing all ten audits, we identified four areas where code-path evaluation could extend what text auditing already does well.

Direction 1: Clinical exposure is visible in imports, not just in README text.

A repository importing pharmacogenomics allele tables has clinical exposure regardless of what its README says. Detecting that dependency at the import level — rather than waiting for a disclaimer — lets the framework flag exposure earlier.

The key distinction is severity: a direct pharmacogenomics import (CYP2D6, CPIC) signals live patient-facing risk and is classified CA-DIRECT.

A general-purpose medical imaging library like pydicom or MONAI is classified CA-INDIRECT — research-use exposure, not necessarily a live clinical output path. The import alone does not determine clinical risk; the classification tier does.

Direction 2: Not all clinical proximity is the same.

A live pharmacogenomics dosage tool and a README roadmap note about a future ClinVar integration are not equivalent risks. Differentiating them — live output vs. research context vs. planned feature — makes the evaluation more precise and makes the accountability expectations more appropriate.

Direction 3: Scoring stability is worth measuring directly.

We ran Stage 1 on one repository in multiple passes. The results ranged across 28 points on the same input. Overlapping trigger conditions between hype-detection items are one contributing factor.

LLM runtime stochasticity is another — the exact split between the two is still under measurement. Adding explicit discrimination examples — what exact phrasing triggers each item, what does not — makes the scoring surface cleaner and reduces the most obvious sources of variance.

Direction 4: Code-path behavior deserves its own scan layer.

A fail-open pattern is a control path that appears to enforce a constraint but defaults to bypassing it. The BioAgents rate limiter above is the example. In a clinical output path, a silent pass-through is not graceful degradation. It is an untraced result that looks like a real one. Building a dedicated scan for these patterns adds a check that text auditing was never meant to provide.

These four directions came directly from running the audits. The scores across the 10 repositories remain as published. Code-path evaluation is what the framework can now add on top of them.

What v1.0.6 Added — Carried Forward in v1.1.2

These changes were introduced in v1.0.6 and are carried forward in the current internal v1.1.2 package. They extend the framework's evidence surface into code-level behavior. Calibration is ongoing.

1. Two Evidence Paths, Not One

We narrowed one of the biggest divergence points by splitting evaluation into a text path and a code path.

The text path works as before: read the README, CHANGELOG, and public posts, score against the rubric. Always available regardless of access to the repository.
The code path activates when the audit has a local clone. It runs through Claude Code, Codex CLI, Gemini CLI, or Copilot CLI. Claims are not interpreted. They are measured. A README that says "IRB-approved data" earns no points for the statement. Points require a provenance artifact in the code.

When code confirms the README, that is a positive signal. When it contradicts it, that contradiction is the finding.

2. Clinical Dependency Detection

At the start of every local audit, a scan script reads Python imports and README keywords. It classifies the result into one of three severity levels:

# ca_detection_scan.sh -- pharmacogenomics section

# CA-DIRECT: live patient-facing output risk
check_import "CPIC|cpic|PharmGx|pharmacogenomic" \\
  "Pharmacogenomics (CPIC/PharmGx)" "CA-DIRECT"

check_import "DPYD|CYP2D6|CYP2C19|allele" \\
  "Pharmacogene alleles" "CA-DIRECT"

# CA-INDIRECT: research-use clinical exposure
check_import "import pydicom|from pydicom" \\
  "pydicom (DICOM imaging)" "CA-INDIRECT"

check_import "import monai|from monai" \\
  "MONAI (medical AI)" "CA-INDIRECT"

Accountability requirements follow the actual clinical proximity of the code. Not the aspirational proximity of the roadmap. A roadmap mention without active implementation is treated as CA-PLANNED rather than collapsed into the same bucket as live clinical output.

The pattern matching is against import statements and function names, not comment text. False positive calibration is still in progress.

3. Code Integrity Scanning (C1-C4)

A second scan handles four code-level checks:

hardcoded credentials (C1)
unpinned dependencies (C2)
clinical-path stubs (C3)
fail-open exception handlers (C4).

The C4 check targets the BioAgents-style pattern. Searching for clinical keywords on the except: line misses most real cases. Clinical context lives in function names and surrounding code. The scan uses a two-pass approach: first identify files with clinical-domain context, then find silent exception handlers within those files.

A silent except: pass in a clinical-context file is a trust-surface failure. The scan makes it visible without requiring a reviewer to read every exception block manually.

4. Discrimination Examples

To reduce the 28-point variance, we added explicit examples for each hype-detection item: what exact phrasing triggers it, what does not, and what the documented edge cases are.

The goal is to reduce obvious scoring drift enough that the same repository is no longer interpreted as two different trust surfaces across different auditors or different LLMs. That goal is not yet verified. The discrimination examples are the primary mechanism toward it.

The Three Questions Now Have a Fourth

In the first post, we described three questions the framework was built around:

Did the repository describe its limits honestly?Did public communication remain consistent with those limits?Did the codebase show evidence of maintenance and biological responsibility?

Running 10 real audits pointed toward a fourth:

4. Does the code actually do what the documentation says — and where it diverges, is that divergence visible and traceable?

That fourth question is what the audit outputs kept surfacing. A function name that sounds real. A pipeline that looks complete. An output that is plausible. An implementation that is a stub, or a control path that silently bypasses its own constraint.

The first three questions can be answered by reading. The fourth requires looking at the code.

What the Framework Added — and What Stays the Same

The first post ended with this:

"STEM-AI is meant to support serious review, not replace it."

That has not changed. Every report carries a non-removable disclaimer: LLM-generated audit, not a regulatory determination, not clinical certification. Every report carries an expiry date. The minimum threshold for supervised pilot consideration is still T3. None of the March 2026 repositories reached it.

What the audits added is narrower — broader evidence coverage on top of an already-working foundation:

A verifiable artifact shifts the accountability surface — it does not eliminate the possibility of falsification. The framework treats its presence as a necessary condition, not a sufficient one.

What the framework gained is that the evidence it counts now extends beyond what authors say about their own code.

Three Directions Still Ahead

Automated re-audit on repository changes. A score from three months ago may not describe the same repository. The trajectory signal measures issue close rate and release frequency across consecutive 90-day windows. It is a partial answer. A CI-triggered re-audit path is the logical next step.

The denominator problem. Zero of 10 repositories reached T3. This may accurately describe the ecosystem's current state. It may also reflect calibration issues in the upper tiers. Distinguishing between the two requires before-and-after auditing of repositories that have received systematic governance remediation.

The Stage 2 redistribution question. Most audits have no cross-platform consistency data. When that data is unavailable, the framework redistributes Stage 2's weight equally between documentation quality and engineering accountability.

For repositories with clinical-direct exposure, a well-written README can then compensate for weak code accountability. A guardrail flags this condition. The current redistribution rule is explicit but not yet final — it remains one of the framework's open calibration questions.

If there are open-source bio-AI repositories you think should be >audited next, drop them in the comments. Bonus if they claim clinical >relevance, drug discovery, or medical reasoning.

STEM-AI v1.1.2 — Trust Evaluation Framework for Medical AI Repositories.

"Code works. But does the author care about the patient?"

Open Bio-AI Has a Trust Problem, Not Just a Model Problem

Thu, 26 Mar 2026 06:32:45 +0000

Drug discovery is expensive for a reason.

A new medicine can take roughly a decade and enormous capital to move from idea to patient. That is not simply because biology is complex. It is because biology punishes unverified claims. Every stage adds cost, but it also adds filtering. Weak hypotheses are supposed to fail early. Unsafe assumptions are supposed to be caught before they travel downstream.

That is why AI in biology is so attractive.

If models can surface candidate molecules faster, detect patterns in genomic data earlier, or automate parts of scientific analysis that used to take weeks, the upside is obvious. The promise is real. Faster iteration matters.

But over the past year, another layer has formed around that promise.

A visible class of open-source repositories now presents Bio-AI through agents, skills, wrappers, workflow kits, and automated research surfaces. They look increasingly operational. They install cleanly. They return outputs. They often present themselves in the language of discovery.

That was the background to the question I wanted to answer.

Not: Which repositories look the smartest?
Not even: Which ones are scientifically correct?

A prior question comes first.

Which of these systems provide enough structural honesty, bounded
behavior, and verification surface that someone downstream could begin
to trust what they are doing?

So I audited ten visible repositories and adjacent scientific automation systems in the open Bio-AI ecosystem.

What I found was not that most of them were obviously fraudulent, or even obviously wrong. The deeper problem was narrower and more important:

most of them had outputs, but lacked reliable mechanisms to establish
what those outputs meant, when they should stop, or how another party
would review them before acting on them.

That is not just a tooling gap.

It is a trust gap.

The Function That Changed the Entire Audit

One of the clearest examples came from a repository called OpenClaw-Medical-Skills, presented as part of a drug-discovery workflow.

def generate_analogues(self, seed_smiles: str, count: int = 3):
    “”“
    Mocks a generative model (like REINVENT).
    In a real app, this would call a PyTorch model.
    “”“
    analogues = []
    for i in range(count):
        if “C” in seed_smiles:
            new_smi = seed_smiles.replace(”C”, “C(C)”, 1) if i == 0 else seed_smiles + “F”
            analogues.append(new_smi)
        else:
            analogues.append(seed_smiles + “C”)
    return analogues

If you work in chemistry, the issue is immediate.

If you do not, the short version is simple: SMILES is not arbitrary text. It is a structured notation for molecular representations, with grammar, branching rules, ring closures, charge handling, and chemical constraints. This function does not generate meaningful analogues. It appends characters to a string.

The comment admits it is a mock. But that truth lives inside the code comment, not at the trust surface most users or downstream agents would rely on. At the level of the workflow, the repository still appears to generate candidate molecules. The function runs. The pipeline continues. A result is returned.

By the time the system says “candidate molecules generated,” the operational distinction between mock behavior and scientific behavior has already been blurred.

That is why I do not think this is best described as a bug.

It is a trust-surface failure.

And once I saw that clearly, I started finding related versions of the same problem across the ecosystem.

What I Audited

I reviewed ten visible repositories and adjacent systems using a two-layer process:

a repository audit focused on structure, execution paths, default behavior, and file-level findings
a trust-scoring framework, STEM-AI v1.0.4, focused on documentation integrity, governance posture, and biological accountability

The repositories were:

Biomni
AI-Scientist
CellAgent
ClawBio
LabClaw
claude-scientific-skills
SciAgent-Skills
BioAgents
BioClaw
OpenClaw-Medical-Skills

The score table looked like this:

The headline result was blunt.

8 of 10 landed in T0
1 landed in T1
1 landed in T2
0 reached T3 or T4

That matters because, in this framework, T3 is the minimum threshold for supervised pilot consideration.

None of the repositories reached it.

What This Audit Was Actually Measuring

This point matters because the article can be misread if the frame is too loose.

The audit was not trying to prove which repositories were scientifically true in the strongest sense. It was asking a prior and narrower question:

Does this repository provide enough structural evidence, boundedness,
and reviewability that a researcher, team, or institution could even
begin to establish trust before downstream use?

That means asking questions such as:

Does the README honestly describe what the system can and cannot do?
Do critical paths fail closed when required evidence, assets, or dependencies are missing?
Are there domain tests, not just software hygiene tests?
Can an output be traced back to a concrete input and execution state?
Are placeholders, stubs, or mock routines clearly separated from functional output surfaces?

That is a narrower standard than scientific truth.

But in practice it comes first.

If a system cannot establish what it did, then no downstream user can responsibly decide what its output should mean.

Four Patterns Kept Reappearing

After working through the repositories, the same four failure modes appeared again and again.

1. Scientific Scope Expanded Faster Than Accountability

Many of the repositories touched domains that are not trivial: drug discovery, molecular analysis, genomics, clinical-adjacent interpretation, or biologically consequential workflow automation.

But the surrounding accountability structure often lagged far behind the ambition of the task.

The closer a system gets to outputs that influence costly, safety-relevant, or clinically adjacent decisions, the more necessary it becomes to define limits, disclaimers, stop conditions, review boundaries, and responsibility surfaces.

Those were often weak, partial, or absent.

2. CI Checked Form, Not Scientific Validity

Several repositories had CI/CD pipelines, which sounds reassuring until you inspect what they actually verify.

In most cases, the checks focused on formatting, schema shape, ordering, script completion, or basic code hygiene. Those things matter. But they do not answer the question that matters in Bio-AI: whether the output is scientifically meaningful, biologically plausible, or safe to interpret as evidence.

Passing CI often meant the software surface was consistent.

It did not mean the scientific surface was trustworthy.

3. Mock Behavior Could Survive as Functional Behavior

The OpenClaw example was the clearest case, but not the only one.

Across the sample, I repeatedly found a dangerous pattern:

the name of the function sounded real
the workflow looked legitimate
the output format appeared plausible
but the underlying behavior was placeholder logic, a simplified stub, or a mock presented too close to a user-facing surface

Prototype code is not the problem by itself. Early repositories mock things all the time.

The problem begins when the architecture no longer makes that provisional status obvious downstream.

At that point, scaffolding can start masquerading as capability.

4. Strong Design Was Often Undermined by Weak Defaults

Several repositories were not superficially sloppy. Some had serious architecture. Some clearly reflected thoughtful engineering.

But the defaults weakened the trust story.

BioAgents presented a substantial multi-agent design, yet rate limiting could fall away in default usage. BioClaw used container isolation, but an important writable mount weakened containment. Biomni wrapped parts of execution in timeout logic, yet still exposed unsandboxed subprocess behavior.

This produced a recurring pattern I now think is characteristic of immature scientific infrastructure:

the architecture says the right things, but the runtime defaults still carry the old risks.

The One Repository That Broke the Pattern

The only real outlier in the sample was ClawBio.

It did not solve everything. It still landed at T2, not T3 or T4. But it was the only repository in the set that consistently treated trust as something that had to be constructed in the runtime itself.

Two signals stood out.

First, it validated biological inputs before moving forward. Its scRNA-seq path did not simply accept a file and continue. It checked whether the data looked already processed, whether value assumptions were broken, and whether the analysis should halt rather than continue under uncertain conditions.

Second, it created reproducible audit identifiers by hashing input files. That sounds mundane. It is not. It creates a deterministic link between a specific input state and a specific output trace — exactly the kind of provenance discipline that was missing almost everywhere else in the sample.

ClawBio did not prove that open-source Bio-AI is solved.

It did prove something narrower and more useful:

the ecosystem does not have to remain trapped at the level of plausible-looking but weakly governed execution.

What Changed in My View

Before this audit, the intuitive question was:

Which Bio-AI repositories are good?

After the audit, I think that is the wrong first question.

The better first question is:

What has to exist before any Bio-AI repository can be treated as trustworthy scientific infrastructure?

That shift matters because it moves the conversation away from surface cleverness and toward institutional reviewability.

From capability to boundedness.
From output to provenance.
From demo quality to stop conditions.
From architectural ambition to governance reality.

That is why I do not think the core bottleneck in Bio-AI is model capability alone.

The deeper bottleneck is the missing verification layer.

More specifically:

truth-surface separation
fail-closed runtime behavior
domain regression testing
provenance discipline
explicit scope boundaries
human-in-command reviewability

Without those, even technically impressive repositories remain unstable objects: too promising to dismiss, too weakly governed to trust.

The Minimum Standard I Would Use

If I were evaluating these systems for institutional use, I would not begin with benchmark claims or abstract model quality.

I would begin with six simpler questions:

What can this repository actually do locally?
What depends on hidden assets, hosted services, or private APIs?
Does it stop when critical evidence is missing?
Does it include domain-specific tests?
Can outputs be traced to concrete inputs and execution paths?
Is the trust surface honest?

Those are not glamorous questions.

They are earlier questions.

And in this audit, they mattered more than almost anything the READMEs claimed.

My own threshold would be clear:

T3 is the minimum standard for a supervised pilot.

None of the repositories in this audit reached it.

That is not a reason to stop building.

It is a reason to stop pretending that plausible output is already equivalent to trustworthy infrastructure.

Final Point

I do think AI will matter in biology.

The promise is not fake. The acceleration is not imaginary. Some of these repositories are useful research artifacts, engineering accelerators, or idea surfaces.

But the field is still crossing a deeper line than many people admit.

It is not just trying to produce outputs.

It is trying to produce outputs that another party can review, bound, reproduce, and challenge before those outputs acquire scientific or operational authority.

That is the line that still matters most.

And right now, in open Bio-AI, the verification layer is still lagging behind the capability layer.

Bio-AI Repository Audit 2026 — Technical Report

A technical audit of 10 open-source Bio-AI repositories using code inspection and STEM-AI trust scoring.

Audit snapshot date: March 20, 2026. STEM-AI v1.0.4.
This report reflects a time-bounded audit of public repository surfaces, workflow reconstruction, and selective file-level review.
It is not a regulatory determination or legal judgment.

→ Read the Full Technical Report
https://flamehaven.space/writing/bio-ai-repository-audit-2026-a-technical-report-on-10-open-source-systems/

Audit snapshot date: March 20, 2026. STEM-AI v1.0.4. This article reflects a time-bounded audit of public repository surfaces, workflow reconstruction, and selective file-level review. It is not a regulatory determination or legal judgment.

I Audited 10 Open-Source Bio-AI Repos. Most Could Produce Outputs. Few Could Establish Trust.

Wed, 25 Mar 2026 15:20:11 +0000

Not which tools were “good.” Whether any of them could prove what they were doing.

Drug development is slow for a reason.

A single new medicine can take roughly ten years and more than a billion dollars to reach a patient. That is not because biology is under-optimized. It is because biology does not forgive guesses.

Every step has to be verified. Every claim has to be traced. When something goes wrong at the wrong stage, people get hurt.

The promise of AI in drug discovery is that it can compress those timelines. Find molecular candidates faster. Surface patterns in genomic data that humans would miss. Run analyses in hours that would otherwise take months.

That promise is real.

But over the past year, something else has been happening alongside it. A growing number of open-source repositories have appeared on GitHub presenting AI systems for drug discovery, genomic analysis, and clinical-adjacent biological research. They look sophisticated. They run without errors. They produce plausible-sounding output.

The question I wanted answered was not whether they looked impressive.

It was whether they could establish what their outputs actually meant.

So I audited ten high-visibility repositories and adjacent systems in the open Bio-AI ecosystem. What I found was not that most of them were obviously wrong. It was that most of them had no structural way to know whether they were right or wrong, and no reliable way to tell the user either.

The Function That Changed the Way I See This Space

Here is a piece of code from a repository called OpenClaw-Medical-Skills, presented as part of a drug-discovery workflow:

def generate_analogues(self, seed_smiles: str, count: int = 3):
 """
 Mocks a generative model (like REINVENT).
 In a real app, this would call a PyTorch model.
 """
 analogues = []
 for i in range(count):
 if "C" in seed_smiles:
 new_smi = seed_smiles.replace("C", "C(C)", 1) if i == 0 else seed_smiles + "F"
 analogues.append(new_smi)
 else:
 analogues.append(seed_smiles + "C")
 return analogues

If you are a chemist, you already know why this is a problem.

If you are not, here is the short version.

SMILES is a strict string notation for molecular structures. A valid SMILES string is not just text. It has grammar: bond types, atom charges, ring closures, branching rules. This function does not generate molecules. It appends the letter ”C” or ”F” to the input string.

The docstring admits that this is a mock. But that disclosure appears in a code comment, not in the user-facing skill surface. The researcher or AI agent reading the corresponding skill document sees a workflow that claims to generate candidate molecules.

The function runs, returns a list, and produces no obvious error. By the time the pipeline says “candidate molecules generated,” the fact that the underlying operation was string manipulation is already hidden.

This is not just a bug.

It is a trust-surface failure.

And once I saw it clearly, I started seeing the same pattern in different forms across the ecosystem.

What I Audited

I evaluated ten visible repositories and adjacent systems in the Bio-AI ecosystem using a two-layer process:

a technical repository audit focused on structure, execution paths, and file-level findings
a trust-scoring framework called STEM-AI v1.0.4, which measures documentation integrity, governance posture, and biological accountability

The full score table looked like this:

8 of 10 landed in T0.
One reached T1.
One reached T2.
None reached T3 or T4.

That is the headline result.

But the scores matter less than the reason behind them.

What I Was Actually Measuring

This audit was not trying to answer the question most people ask first.

It was not asking: which repositories are scientifically correct?

It was asking a prior question:

Does this repository provide enough structural honesty and verification architecture that a researcher or institution could even begin to establish trust before downstream use?

That means checking things like:

Does the README honestly describe what the repository can and cannot do?
Does the code fail closed when required assets or evidence are missing
Are there domain tests, not just syntax checks?
Can you trace an output back to a concrete execution state?
Are mock or placeholder routines presented as if they were functional?

That is a narrower question than scientific truth, but in practice it comes first. If a system cannot establish what it is doing, it cannot be trusted downstream no matter how impressive its claims are.

Four Patterns Behind the Failures

After reviewing the repositories, the same four patterns kept appearing.

1. Clinical-Adjacent Scope Without Clinical-Adjacent Accountability

Many of these repositories touch domains that are not harmless playgrounds: drug candidates, genomic interpretation, clinical reports, single-cell analysis, molecular design.

But the code and documentation often behave as if these are neutral productivity tasks. They rarely acknowledge regulatory context, risk boundaries, or the practical consequences of error.

2. CI Validates Form, Not Science

Several repositories have CI/CD. That sounds reassuring until you look at what the CI actually does.

In most cases it checks formatting, section order, or whether a script runs without crashing. It does not check whether the biological output is valid, physically plausible, or scientifically appropriate.

Passing CI in these systems often means only that the software surface is syntactically consistent.

3. Mock-as-Functional

The OpenClaw-Medical-Skills example is the clearest case, but not the only one.

Again and again, I found architectures where:

the function name sounds real
the pipeline looks professional
the output format is plausible
but the implementation is a stub, a placeholder, or an undeclared simplification

That is not the same as ordinary prototype code. Prototype code is usually disclosed as provisional. Mock-as-Functional code is dangerous because it can survive into user-facing workflows while still looking operational.

4. Strong Architecture Undermined by Unsafe Defaults

Some of the repositories are not poorly engineered in the shallow sense. They have real structure. They have actual systems architecture.

But the defaults betray them.

BioAgents is a good example. It has a serious multi-agent platform design, but its rate limiting can disappear in the default mode.
BioClaw has container isolation, but one of its important mounts is writable in a way that weakens containment.
Biomni has timeout wrappers, but still allows unsandboxed execution through generated subprocesses.

The result is a recurring pattern: the security story exists, but the runtime defaults hollow it out.

The One Repository That Broke the Pattern

The outlier was ClawBio.

It was not perfect, but it was the only repository in the sample that treated trust as something that had to be built into the runtime, not just implied by the project description.

Two things stood out immediately.

First, it validates biological inputs before analysis. Its scRNA-seq input logic does not simply accept a file because it has the right extension. It checks whether the data appears already processed, whether values are negative, whether integer assumptions have been violated, and whether the analysis should halt before proceeding.

Second, it generates reproducible audit identifiers by hashing the input files. That is not glamorous, but it is exactly the kind of thing missing almost everywhere else in the ecosystem: a deterministic link between a specific input state and a specific output state.

ClawBio reached **T2**, the only repository in the sample to do so. That does not make it deployment-ready. But it does show that open-source Bio-AI does not have to remain trapped at the level of plausible-looking but weakly governed execution.

Bio-AI Repository Audit 2026 — Technical Report

Technical Code Audit + STEM-AI Trust Scoring (10 Open-Source Systems)

Audit snapshot date: March 20, 2026
Framework: STEM-AI v1.0.4

This report reflects a time-bounded audit of public repository
surfaces, workflow reconstruction, and selective file-level review.
It is not a regulatory determination or legal judgment.

▶ Read the Full Technical Report https://flamehaven.space/writing/bio-ai-repository-audit-2026-a-technical-report-on-10-open-source-systems/

What This Means

The big lesson from this audit is not that the ecosystem lacks cleverness.

It is that the ecosystem is much better at producing the appearance of capability than at producing systems that can bound, trace, and justify that capability before downstream use.

That is why I do not think the core bottleneck in Bio-AI is model quality alone.

The bottleneck is governance.

More specifically:

truth-surface separation
fail-closed runtime behavior
domain regression testing
provenance discipline
explicit scope boundaries
human-in-command reviewability

Without these, even an impressive repository remains an unstable object: too sophisticated to dismiss casually, too weakly governed to trust.

The Minimum Standard I Would Use

If I were evaluating these systems for institutional use, I would not begin with benchmark claims.

I would begin with six questions:

What can this repository actually do locally?
What depends on hidden assets, external APIs, or hosted outputs?
Does it stop when critical evidence is missing?
Does it have domain-specific tests?
Can outputs be traced to concrete inputs and execution paths?
Is the trust surface honest?

In my view, the minimum threshold for a supervised pilot should be **T3**. None of the repositories in this audit reached it.

That is not a reason to stop building.

It is a reason to be honest about where the field actually is.

These repositories are often valuable as research artifacts, engineering accelerators, or methodology surfaces. But most are not yet trustworthy systems in the stronger sense that medicine, biology, or automated discovery will eventually require.

Final Point

The AI-accelerated future of biology is probably real.

But if that future is going to matter outside demos, preprints, and impressive GitHub surfaces, then the field will have to solve something more basic than capability.

It will have to solve how to distinguish real outputs from plausible outputs before the cost of being wrong becomes unbearable.

That, more than any benchmark, is the line the field still has to cross.

Medical AI Repositories Need More Than Benchmarks. We Built STEM-AI to Audit Trust

Tue, 24 Mar 2026 12:21:47 +0000

If you have been paying attention to GitHub recently — the past six months — you have seen the pattern.

A new bio-AI repository appears. It promises to automate genomic analysis, drug discovery, medical imaging, or clinical data interpretation. The README is polished. The architecture diagram looks serious. Within weeks it has hundreds of stars, a few forks, and a preprint on bioRxiv.

Then nothing.

No CI. No CHANGELOG. No response to issues. No clear statement of limitations. No clinical disclaimer anywhere in the repository. And a tool that may now be touching patient-adjacent workflows has exactly one quality gate: the author thought it was ready.

We are developers. We have built things in that atmosphere too. At some point, we had to ask a harder question than benchmark accuracy:

When a bio-AI repository gets close to real diagnostic, genomic, imaging, or therapeutic workflows — what does "trustworthy enough for serious review" actually mean?

The field has benchmarks for models. It has almost no shared standards for repository accountability.

So we built one.

What Makes This Moment Different

Bio-AI is now filling up with skill libraries, agent wrappers, orchestration pipelines, and plugin-style marketplaces that look far more deployable than they actually are. Surface maturity is easy to fake. A clean README, a marketplace entry, or a rising GitHub star count can make a repository look trustworthy long before it has earned that trust.

And this category cannot be judged like ordinary software.

If a note-taking app breaks, users get frustrated.
If an internal dashboard fails, a team loses time.
But when a biomedical or medical-AI repository fails quietly, the consequences do not stop at software quality.

A flawed genomics pipeline can distort interpretation.
A weak clinical model can normalize unsafe confidence.
A drug-discovery system can push the wrong candidates forward, bury better ones, and send time, capital, and downstream validation effort in the wrong direction.

In this category, failure is not just a debugging problem.
It is a patient-safety problem and a resource-allocation problem.

That is why these systems require more than code inspection. They require scrutiny of documentation, limits, provenance, maintenance behavior, and public claims — because the cost of being wrong is fundamentally different here.

What STEM-AI Is

STEM-AI (Sovereign Trust Evaluator for Medical AI) is a governance audit framework for public bio/medical AI repositories.

It does not ask whether a project sounds impressive.

It asks whether the repository shows observable signs of responsible engineering:

honest documentation
consistent public claims
maintenance discipline
biological data responsibility
explicit acknowledgement of limits

That distinction matters. A repository can look technically sophisticated and still fail the most basic governance test for patient-adjacent use. A polished README is not a safety surface by itself.

STEM-AI is not a regulatory verdict, clinical certification, or legal assessment. It is a structured review framework designed to give researchers, reviewers, procurement teams, and engineers a more reproducible starting point for discussion. The canonical spec requires a non-waivable disclaimer in every output stating exactly that.

STEM-AI is meant to support serious review, not replace it.

Why We Made It LLM-Native

STEM-AI runs as a structured specification executed by a major LLM.

The spec is the program. The LLM is the runtime.

That sounds strange until you look at the design constraint. We did not want a system that "vibes" its way to a trust score. We wanted a system that forces checklist-based scoring, explicit evidence chains, N/A handling for missing data, and hard floors for catastrophic claims. The goal is to reduce evaluator drift by replacing narrative judgment with traceable rubric logic.

The model is not supposed to "feel" trust.
It is supposed to count evidence.

How It Works

STEM-AI evaluates a repository across three stages. Each stage has a defined checklist. Each score must map back to observable evidence.

These three stages ask three different questions:

what the repository says
what public communication says
what the codebase actually proves

Stage 1 — README Intent

If a repository is patient-adjacent, the README is not marketing. It is the first governance surface.

WHAT WE LOOK FOR (positive signals):

  [R1] Does a Limitations or Known Issues section exist
       and cover a substantial portion of the README
       with specific, actionable content?

  [R2] Is any regulatory framework cited?
       (FDA SaMD guidelines, CE marking, IRB requirements,
        or equivalent)

  [R3] Is there a clinical disclaimer?
       ("This tool is NOT a substitute for clinical judgment"
        or equivalent)

  [R4] Are demographic bias limits or applicable population
       boundaries disclosed?

  [R5] Are reproducibility provisions present?
       (environment pinning, data version, seed values)

WHAT WE PENALIZE (negative signals):

  [H1] Performance superlatives without benchmark comparison
       "SOTA", "State-of-the-Art", "Best-in-class"

  [H2] Unsubstantiated innovation claims
       "Revolutionary", "Groundbreaking", "Game-changing"

  [H3] Fully autonomous framing in a clinical context
       "Fully Automated", "Zero Human Oversight"
       without any stated supervision requirement

  [H4] AGI or human-level capability claims
       "AGI", "Human-level", "Surpasses clinicians"

  [H5] Social proof substituted for technical credibility
       GitHub stars or download counts cited as a trust signal

  [H6] External optics substituted for technical evidence
       VC funding or press coverage without validation

![The Patient safety trapdoor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j4go8oh9dn80at4x4npu.png)

CRITICAL RULE:
  IF H3 AND H4 are both triggered simultaneously:
    → Hard floor activated.
    → Final Score = 0. Tier = T0. No further scoring.
    → A tool claiming fully automated clinical operation
      at AGI level is a patient safety issue,
      not a scoring edge case.
      No positive signals elsewhere can override this.

On clinical adjacency: STEM-AI does not assume all biomedical repositories carry the same deployment risk. The disclosure bar rises as a tool moves closer to patient-facing or decision-shaping use. If the repository contains medical imaging frameworks, drug docking engines, diagnostic genomics pipelines, or clinical language models, the absence of R2 and R3 is not a missed bonus — it becomes an active penalty.

Stage 2 — Cross-Platform Consistency

This stage is less about marketing tone and more about contradiction. If a README warns carefully but public posts erase those warnings, the repository's governance surface is inconsistent.

WHAT WE PENALIZE:

  [F1] Stars or downloads used as the primary message
       on external platforms, without technical substance

  [F2] Public dismissal or hostile response to
       critical feedback from the community

  [F3] Vanity metrics presented as clinical trust proxies

  [F4] External posts that directly contradict or omit
       warnings stated in the README
       → Most serious flag in Stage 2.
         A README that says "not for clinical use" while
         the author's LinkedIn announces clinical deployment
         is a consistency failure.

WHAT WE REWARD:

  [A1] Model errors or hallucination cases
       openly shared in public forums

  [A2] Reproducibility failures or known failure modes
       transparently acknowledged outside the repository

  [A3] Critical external feedback accepted and incorporated

  [A4] Regulatory or ethics body collaboration referenced

This stage is only as strong as the public evidence available. When live cross-platform data is unavailable, Stage 2 goes N/A and its weight moves to Stages 1 and 3. In medical-adjacent evaluation, pretending to know is worse than stating you do not.

Stage 3 — Code Infrastructure and Biological Integrity

This is where the framework becomes more than a documentation audit.

TECHNICAL RESPONSIBILITY:

  [T1] CI/CD pipeline
       Does automated testing run on push or PR?
       Is coverage stated or just implied?

  [T2] Domain-specific regression tests
       Not just unit tests — tests that verify biological
       or clinical correctness invariants.

  [T3] CHANGELOG transparency
       Does one exist?
       Does it document bugs and failures honestly,
       or only feature additions?
       Silent updates on a clinical-adjacent tool are
       a different risk profile than an honest bug log.

  [T4] Issue and PR response pattern
       When did the last maintainer response happen?
       What proportion of issues receive a response?

BIOLOGICAL INTEGRITY:

  [B1] Data provenance and consent
       Where did the training or reference data come from?
       Is there an IRB approval or data consent declaration?

  [B2] Algorithmic bias disclosure
       Are performance limits across demographic subgroups
       disclosed? Are actual measurements provided,
       or only a general acknowledgment?

  [B3] Conflict of interest transparency
       Is the funding source disclosed?
       If there is commercial interest, is it stated?

TRAJECTORY SIGNAL:
  v1.0.4 adds a trajectory modifier comparing issue close
  rates and release frequency across two consecutive
  90-day windows.

  This signal is deliberately bounded: it changes Stage 3
  by at most ±5 points, which translates to roughly ±2
  points on the final weighted score.
  Large enough to matter near tier boundaries.
  Small enough not to distort the audit.

What the Output Looks Like

Every STEM-AI audit produces a structured report. Here is an illustrative example — repository name withheld, structure unchanged:

#  STEM-AI Audit Report v1.0.4
──────────────────────────────────────────────
Target:        [Repository Name]
Audit Date:    2026-03-20
Report Expiry: 2026-09-16
Flags:         CLINICAL_ADJACENT: true
               NASCENT_REPO: false
               T0_HARD_FLOOR: false
──────────────────────────────────────────────
Stage 1 — README Intent        47 / 100  VC Pitch
Stage 2 — Cross-Platform       N/A       (MANUAL mode)
Stage 3 — Code Infrastructure  10 / 100  Hit-and-Run
──────────────────────────────────────────────
Final Score:   28 / 100
Tier:          T0 REJECTED
USE_SCOPE:     None — clinical use prohibited
──────────────────────────────────────────────
Priority Remediation:
  1. Add clinical disclaimer (active penalty applied —
     clinical-adjacent tooling without R3)
  2. Implement CI/CD pipeline (T1: 0 pts)
  3. Disclose data provenance and IRB status (B1: 0 pts)
──────────────────────────────────────────────
⚠ This report is LLM-generated. It is not a regulatory
  determination. Report expires: 2026-09-16.

The expiry date is not cosmetic. A report based on a repository that went inactive months ago should not circulate in procurement pipelines as if it still describes current reality. Version 1.0.4 computes that date from recent project activity automatically.

What the Tiers Mean — and What They Do Not

T0 Rejected:        Trust not established
                    Clinical use prohibited

T1 Quarantine:      High risk
                    Independent verification required

T2 Caution:         Research reference only
                    Clinical automation forbidden

T3 Review:          Supervised pilot eligible
                    Human oversight mandatory

T4 (highest tier):  Strongest observed governance signals
                    Still requires independent expert review
                    and formal regulatory clearance

Even at the highest tier, STEM-AI does not substitute for clinical validation, expert review, or regulatory clearance. That is not a disclaimer added to soften the framework. It is how the framework was designed.

One Design Decision That Matters

Author background does not affect the score.

Whether the author comes from biology, medicine, ML research, or pure software engineering is recorded as contextual information for human reviewers. It is explicitly non-scoring and carries a mandatory bias warning in every report.

Domain credentials are not the same thing as engineering integrity. And lack of domain pedigree does not imply carelessness. STEM-AI is built to evaluate observable repository governance — not to sort developers by prestige.

What STEM-AI Refuses To Be

A trust framework for medical AI can become dangerous if it turns into a personal attack engine.

STEM-AI is constrained on purpose. It is limited to public professional repositories and public professional material. It forbids PII inference, private-account speculation, and individual profiling as a use case. Without that boundary, a trust evaluator becomes a harassment tool.

Three Questions This Framework Is Built Around

If you remember nothing else from this framework, remember these:

Did the repository describe its limits honestly?

Did public communication remain consistent with the repository's stated limits?

Did the codebase show evidence of maintenance and biological responsibility?

Those are not performance questions. They are accountability questions. For tools that sit upstream of clinical decisions, accountability is not optional.

What Comes Tomorrow

Tomorrow we publish the first audit set from STEM-AI v1.0.4 across 10 open-source bio-AI repositories — including projects from research institutions, an actively used SaaS platform, and agent-style bioinformatics tooling running inside containerized environments.

What we can say now:

The strongest-looking repositories are not always the most accountable. In at least one case, a repository with solid engineering signals still falls short because a critical disclosure is missing from the surface a reviewer would actually read first.

In another, the safety control exists in the code but fails as governance because it does not activate under default deployment.

In another, the generation mechanism itself is not the problem. The problem is that users were never clearly told what it was. That is a disclosure failure, not a technical failure. The distinction matters.

The full breakdown publishes tomorrow.

Which bio-AI repositories would you want audited next? Drop them in the comments.

STEM-AI v1.0.4 — full audit results tomorrow.

"Code works. But does the author care about the patient?"