Coder Legion - Recent posts tagged smoe

The Mountain Exhales — Riding the 30B Wind Before the 477B Avalanche

Tue, 21 Jul 2026 09:00:00 +0000

This is the seventh entry in a curious builder's diary. Over six chapters, we carved a door through Silicon Valley's artificial memory wall, coaxing a 235-billion-parameter mountain to speak, remember, run, and finally converse fluently in the OpenAI dialect. As we push past an average of 1.84 tokens per second and touch peaks of 2.01 on a consumer 48 GB MacBook, the engine is humming. But before we pack our gear for the next impossible summit, we are claiming a builder's privilege. After months of writing lock-free C++ to move heavy stone, it is time to take a breath of thin, fresh air.

1. The 477B Avalanche

S-MoE was built to prove a point: that the memory wall is a software convention, not a law of physics. We took Qwen3-235B — a model that standard runtimes insist requires cloud clusters — and shattered it into pieces that stream perfectly off a MacBook's SSD via Apple Silicon's Unified Memory Architecture.

Because we broke the diary dependency and invited standard frontends (like Open WebUI) into the architecture in Article 06, we are now ready to tackle an even more colossal peak: the GLM MoE 477B parameter model.

The climb will be grueling. A 477-billion-parameter model will push consumer hardware to an edge no one thought possible. It will test every byte of our zero-allocation loops, our Direct I/O DMA transfers, and our patience. But we have the map now.

However, before we shoulder that massive weight, we realized something crucial. S-MoE shouldn't just be an engine for the 235B. It should be an engine for any fine-grained MoE.

2. The Self-Describing Vault (SARC)

Until now, the engine was heavily hardcoded for Qwen3's specific math. If we were going to run different models — the daily driver 30B, the massive 235B, or the upcoming 477B — we couldn't keep recompiling C++ to change the RoPE theta or the routing Top-K.

We needed the engine to be model-agnostic.

The solution was elegantly simple: the Smoe ARChitecture (SARC) block. During the offline shatter process, the Python script now reads the model's Hugging Face config.json and extracts the mathematical DNA — RoPE theta, MoE Top-K, GQA head geometry, and normalization flags. It serializes these into a tiny, 128-byte binary block and stamps it directly into the .smoe vault's header.

Now, when you boot S-MoE, there are no config files or CLI flags to set. The engine simply reads the 128 bytes, understands the topology of the mind it was just handed, and configures its own internal Scout and Metal shaders accordingly.

We suddenly had a "fleet." You can have a directory full of different vaults, and simply switching the model in your Chat UI dropdown seamlessly swaps the models at runtime.

3. The 30B Wind

What happens when you take an engine hardened by the brutal math of a 235B model, and plug a 30B model into it?

We shattered the Qwen3-30B-A3B checkpoint to find out. We wanted to experience pure, frictionless velocity. We wanted to ride the wind.

But when we first spun it up, the numbers were confusingly low. We were seeing about ~5 to 7 tokens per second. Fast, but not "frictionless velocity" fast. We expected the 30B to scream on a 48 GB machine.

The culprit? A single hardcoded flag from our testing days: --ring 1024.

4. The Magic of Auto-Sizing

The Streamer's ring buffer is the heart of S-MoE's speed. It's a pre-allocated LRU cache in RAM. When a token needs an expert, the engine checks the ring first. If it's there (a cache hit), we skip the SSD entirely.

By hardcoding --ring 1024, we were forcing the ring to only hold 1,024 experts (about ~2.5 GB of cache for the 30B).

But S-MoE has a brilliant auto-sizing mechanism (--ring 0). When allowed to auto-size, the engine dynamically checks the OS for available RAM, subtracts a safe overhead, and gives the rest to the ring. On a 48 GB MacBook, the auto-tuner allocates roughly ~15 GB of ring capacity.

The entire 30B vault is only ~14 GB.

The moment we removed --ring 1024 and let the engine auto-size, the ring simply swallowed the entire model. After the first few turns, the SSD reads dropped to virtually zero. The engine ascended from a streaming pipeline into an pure, in-memory execution loop.

The numbers instantly skyrocketed.

5. The True Benchmarks

With the auto-tuner unleashed and the engine running isolated, the true benchmarks emerged.

The Daily Driver: Qwen3-30B

Cold Time-To-First-Token: ~5.85 seconds
Decode Speed: 13.71 tokens / second
The Feeling: Instantaneous. Pure velocity. The entire mind rests comfortably in RAM, producing outputs faster than you can read them.

The Frontier: Qwen3-235B

Cold Time-To-First-Token: ~26.16 seconds
Decode Speed: 1.84 tokens / second (peaking over 2.01 on warm turns)
The Feeling: Cinematic deliberation. The SSD streams experts at ~150+ GB/s, and the mountain answers thoughtfully.

6. The Plateau Before the Climb

We built S-MoE to prove that the 235B frontier could live on a desk. But in making the architecture truly agnostic, we accidentally built the fastest possible way to run the 30B models, too. The same lock-free memory gymnastics that make the heavy models possible make the lighter models absolutely fly.

Because sometimes, before you carry the next mountain, you just need to feel how fast you can sprint across the plateau.

The engine is ready. The architecture is model-agnostic. The UI is clean.

Next stop: 477 Billion parameters.

S-MoE is open source under the MIT License. The code, the philosophy, and the mistakes are all public: github.com/melasistema/s-moe.
Built by Luca Visciola and an AI agent who hopefully knows C++ and low level programming, because I do not 😅.

S-MoE Started to Speak OpenAI — Point Any Chat App at Your MacBook and Watch a 235B Model Think

Mon, 20 Jul 2026 09:00:00 +0000

There is a diary behind this — five entries of a self-taught web developer and an AI agent cutting a door in the memory wall so a 235-billion-parameter frontier model could run off the SSD of a 48 GB MacBook. You don't need any of it to read this one. This entry isn't about a bug we killed or a metric we moved. It's about the day the engine stopped being a private experiment I ran in a terminal and started speaking the one language every AI app on earth already understands.

1. The Moment

For five articles, talking to S-MoE meant talking to a terminal. A raw prompt loop. Something only its author could love. The mountain could speak, remember, predict itself, and run — but you had to be me, at my desk, in a shell, to hear it.

Then, on an ordinary evening, this happened:

.venv/bin/python serve.py --port 8000

And a 235B frontier model streaming its experts off a consumer NVMe became — as far as any other program can tell — OpenAI.

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")

resp = client.chat.completions.create(
    model="s-moe",
    messages=[{"role": "user", "content": "Who are you?"}],
)

That is not a screenshot of the real OpenAI. That's my laptop. The openai SDK cannot tell the difference, and — this is the whole point — neither can Open WebUI, LibreChat, your editor's AI plugin, or the thousand other apps built against that API. They ask for OpenAI. They get 235 billion parameters running three inches from the CPU, on hardware you already own.

2. The Numbers, Plainly

Same MacBook. Same 48 GB. Same 117 GB expert vault cold on the SSD. Nothing about the engine changed for this entry — this is a story about a doorway, not a speedup. But the doorway has numbers too.

Thing	Reality
Server	`serve.py` — one file, Python stdlib `http.server` + the tokenizer
New dependencies added to the engine	zero
API surface	OpenAI Chat Completions — streaming (SSE) & non-streaming, `/v1/models`, `/health`
Works with	Open WebUI, LibreChat, editor plugins, the `openai` SDK, plain `curl`
Built-in chat console	Yes — open `http://127.0.0.1:8000/` in a browser, no install
Decode speed	~1.43 tokens/s — a bit over one word per second
First token, cold 45-token prompt	~29 s, then it streams
Intelligence	The real Qwen3-235B. Undegraded. Only the speed is consumer-grade.

The zero in that table is the one I'm proudest of. The engine's founding vow was no dependencies — POSIX syscalls, Metal, and nothing else. The server keeps the vow. It is stdlib HTTP and the same tokenizer the console already loaded. An entire ecosystem of front-ends, unlocked, without adding a single line anyone else has to trust.

3. The Thinnest Possible Translator

I want to be honest about how little there is here, because the smallness is the design.

The server owns exactly three things: HTTP, tokenization, and the chat template. That's it. It applies Qwen3's chat template to your messages, turns them into token IDs, and hands those IDs to the persistent engine subprocess over the same private protocol the terminal console has always spoken. The engine never learns what HTTP is. It never sees a word of English. It sees numbers, streams experts, and sends numbers back.

Everything OpenAI clients expect is a thin mapping on top of that:

Streaming arrives as server-sent events, tokens appearing the instant they're generated.
temperature, top_p, top_k, repetition_penalty map straight onto the engine's per-request sampling — temperature: 0 selects the pure greedy path and is bit-identical every time.
finish_reason is honest: "stop" when the model chose to end, "length" when it hit your max_tokens mid-thought. Clients that auto-continue on "length" just work.
usage carries real token counts — prompt, completion, total — so anything measuring throughput divides real numbers by wall-clock, not estimates.

There's one deliberate constraint worth naming: the engine holds one conversation context and prefix-matches every request against it. So requests are serialized — one lock, one at a time — and there is no batching, because the hardware has no parallelism to batch into. The upside is that a normal chat client, which resends the growing message list every turn, hits the warm KV cache and only pays to prefill the new turn. Continued conversations stay fast. It is the honest shape of the machine, exposed honestly.

4. A Console That Ships You the Feelings

Third-party apps are the point, but I wanted something that needed zero setup to prove the plumbing — and, selfishly, something built for what I actually do all evening: watch this thing think and try to understand what it's doing.

So the server serves its own chat console. Open http://127.0.0.1:8000/ and it's just there — one self-contained HTML file, no build step, no npm install, served same-origin so there isn't even a CORS dance. It streams tokens as they land. But it's not really a chat UI. It's an instrument panel wearing a chat UI's clothes, and it exists because this whole project has only ever obeyed one law: measure, don't believe.

Every reply carries a live monitor. Before the first token, it doesn't sit in dead silence — it names the exact phase and size the moment they're known: "prefilling 128 tok prompt · 12.4s · cold start." On a machine where an answer takes tens of seconds, that one line is the entire difference between "it's thinking" and "it's hung." Then time-to-first-token, live token count, and decode tok/s, updating as the words stream, frozen on completion with the exact counts from usage.

Two meters sit in the header, because for an experimental researcher the feeling of fullness is half the story:

A context meter — 1,240 / 4,096 — tracking the conversation against the engine's fixed KV window, filling and turning amber as you talk, warning once at the wall where the window starts to slide and the model begins, gently, to forget the beginning.
A system-RAM meter — installed memory, current use, and the free headroom S-MoE's expert-streaming ring can grow into: 14.0 / 48.0 GB · 34.0 free. It's colored by macOS memory pressure, not raw percentage — because a 48 GB box legitimately runs hot under a 235B stream, and a naive bar would scream "full!" all day and cry wolf. Its tooltip breaks out wired, compressed, and the engine's own resident set.

None of this is product polish. It's the opposite. It's the dashboard of someone who has been lied to by too many systems and refuses to fly blind. It ships you the feelings — cold prefill, warm turn, headroom, the window about to slide — directly, as numbers, in real time.

5. Grab the Popcorn

Here's where I'd normally embed a chart. Instead, watch the video. It's three turns of real conversation, unedited, no narration, no speed-up. I could describe it, but this diary has a rule about shipping the raw thing instead of the story about the thing.

A word of preparation, because the experience is genuinely unusual and I don't want you to misread it.

This is not fast. I won't insult you by pretending it is. A word arrives roughly every second. But watch what streaming does to that second: it turns the wait into suspense. The tokens come one at a time, and because you can see them arrive, the pause reads as deliberation, not lag. You are literally watching a 235-billion-parameter mind assemble a sentence, left to right, on a laptop that also has your browser and your IDE open and your music playing.

And here is the thing I keep failing to convince people of until they see it: it is not slow. "Slow" is the wrong word, and I've been reaching for a better one for months. It's cinematic. It's a frontier model answering at reading speed, on a busy consumer MacBook, off an SSD, with the whole 235B of it undegraded behind every word. You don't tap your foot. You lean in. You grab the popcorn. The wait isn't a bug you tolerate — it's the show.

I have watched this thing generate paragraphs while I made coffee, and come back to a genuinely frontier-quality answer waiting for me, and felt something I can only describe as the opposite of API anxiety. No meter running. No tokens billed. No sentence leaving the room. Just my laptop, thinking, at its own pace, about my question, for me.

6. And Now, Every App You Already Use

The console proves the plumbing in ten seconds. The real unlock is that you never have to use it.

Because the API is genuinely OpenAI-shaped, the entire open-source chat ecosystem works unmodified. Point Open WebUI at http://127.0.0.1:8000/v1 with any non-empty API key, pick the s-moe model, and you have a polished, multi-conversation, markdown-rendering front-end talking to a 235B model on your desk. LibreChat, the same. Your editor's AI plugin, the same base URL, the same result — a frontier model doing code review inside your IDE, sourced entirely from your own SSD.

None of these apps were told about S-MoE. None of them were patched. They asked for OpenAI and my MacBook answered. That is the entire thesis of this project, finally wearing a suit: the memory wall around frontier AI was always part physics and part software convention — and the convention just fell to a single file of Python stdlib.

The speed profile sets honest expectations, and I'll state them plainly: this is a machine for thoughtful, asynchronous Q&A — a frontier-quality answer worth waiting a beat for — not rapid-fire agentic loops that fire a thousand calls a minute. Use streaming. Ask real questions. Let it think. It's the difference between a chatbot and a correspondent.

7. What This Entry Is Actually About

The five entries before this were about deletion — wiped caches, a fired prophet, write-only memory, scattered reads, tiny GPU round-trips. Subtraction, every time. This one is the first that's about addition, and it's the smallest addition in the whole diary: a translator that speaks a language everyone already knows, built out of nothing the engine didn't already have.

But it's the entry that makes all the others matter to someone other than me. A frontier model on a consumer laptop is a curiosity if the only person who can reach it is the person who built it. The moment it speaks OpenAI, it belongs to anyone with the same laptop and the same question — running in their editor, their chat app, their scripts, their evening. No cluster. No API key. No sentence leaving the room.

The authorship stays the experiment inside the experiment: the AI agent wrote the server, the streaming layer, the sampling protocol, and the instrument-panel console; I brought the questions — what does an app on the other side of this actually expect? where does the wait feel like a hang instead of a thought? can a researcher feel the fullness of the machine at a glance? — and a decade of web-developer instinct for the exact shape of an HTTP contract.

The mountain vibrated, then it talked, then it remembered, then it predicted itself, then it ran.

Now it speaks a language the whole world already knows — and it's waiting at localhost:8000 for you to say hello.

Grab the popcorn.

Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. He taught this project to listen to a surface for news of the interior — and an API, it turns out, is just another surface the whole world already knows how to read.

The Mountain Runs — I Don't Know If We're Chasing Something or Being Chased, but the Speed Won't...

Thu, 16 Jul 2026 07:31:21 +0000

The Mountain Runs — I Don't Know If We're Chasing Something or Being Chased, but the Speed Won't Stop Climbing

This is the fifth entry in a curious builder's diary. In the first, a self-taught web developer borrowed a satellite engineer's intuition and asked whether frontier AI could run on the MacBook on his desk. In the second, a 235-billion-parameter model said "Hello, world!" through a door we cut in the memory wall. In the third, it learned to converse without making you wait minutes between turns. In the fourth, we fired our own founding metaphor and the model got faster and smarter. That article ended at the wall of physics: a saturated SSD, nothing left to delete. This one is about what happened next, which is that the speed kept climbing anyway — and for one terrifying evening, it looked like it had collapsed by 85% instead.

1. The Moment

A microbenchmark, late at night. Same MacBook. Same GPU. Same weights, same mathematics, same number of bytes read. Two kernels that compute the identical matrix-vector product:

K/V projection (512 × 4096, bf16)  
  
old kernel:   12.3 GB/s new kernel:  285.3 GB/s        (23×)

Twenty-three times faster. Nothing was added — no new hardware, no new math, no quantisation trick. We changed the shape of the question the GPU asks its own memory, and the same silicon that had been trudging along at 12 GB/s broke into a sprint.

Last article ended with me declaring, confidently, that we had hit the wall of physics — that the SSD was saturated and there was no more bloated software between us and the flash memory. That was true, and it remains true. What I had not understood is that the SSD was only one of the walls, and behind it the GPU had been quietly doing the memory-system equivalent of taking a taxi one block at a time.

2. The Numbers, Plainly

Same hardware, same 117 GB vault cold on the SSD, same canonical prompt discipline as every previous entry. Only the engine changed.

Thing	Last article	Now
Generation speed	1.13 tokens/s	1.43 tokens/s
…since first light	0.48 tokens/s	3× total
One generated token costs	~885 ms	~700 ms
First token, cold 45-token prompt	~32 s	~29.4 s
GPU dense projections	12–136 GB/s	207–285 GB/s
GPU syncs during prefill	2 per token	2 per layer
Output token IDs	—	bit-identical, before and after
New hardware required	none	none

And one number that isn't in the table, because it deserves its own section: for a few hours in the middle of this campaign, the official benchmark read 0.20 tokens/second — six times slower than where we started the evening. That story is below, and it is the most web-developer-shaped lesson of the entire diary.

3. The GPU Was Running N+1 Queries

Every web developer knows the N+1 query bug. You ask the database for a list of users, then loop over them asking for each one's profile — a thousand tiny round-trips where one JOIN would do. The database isn't slow. Your access pattern is rude.

It turns out a GPU's memory system has exactly the same personality.

The old kernels assigned one thread per output row of the weight matrix. Sounds reasonable — it's the obvious loop, the one you'd write first. But 32 neighbouring GPU threads execute in lockstep, and under that layout their simultaneous memory requests land at addresses two kilobytes apart. The memory system, which serves wide consecutive bursts beautifully, has to satisfy 32 scattered pleas one crumb at a time. A thousand tiny round-trips. N+1 queries against your own RAM.

The new kernels invert it: one simdgroup — 32 threads — owns one output row together, and its lanes read consecutive 16-byte chunks. Same bytes, same total work, but now every request is one coalesced 512-byte burst. The JOIN.

The quantised expert FFN kernels — the ones that dequantise 2/4-bit codes in register space, the heart of the whole engine — jumped 7×, to 158 GB/s effective.
The dense Q/K/V/O projections and the LM head jumped to 207–285 GB/s.
The comedy prize goes to the K/V projection: its matrix is only 512 rows tall, and the old dispatch geometry handed the entire job to 2 threadgroups — on a GPU built to run thousands. We had booked a stadium and seated two people. That's the 23×.

The pattern of this diary holds for a fifth consecutive entry: the exotic layer was innocent. The dequantisation math was right, the Metal code was correct, every byte arrived. What was broken was the shape of the request — the most database-shaped bug imaginable, living in the middle of a GPU shader.

4. The Prompt Stopped Playing Ping-Pong

The second front was the prefill — the silence before the first word, which article three cut from 93 seconds to 32.

Reading the prompt, the engine was still submitting GPU work per token: every one of the 45 prompt tokens made its own little QKV round-trip and its own little o_proj round-trip, each ending with the CPU standing at the door waiting for the GPU to ring the bell. Roughly 8,460 command buffers for one prompt. Now each layer submits all tokens' QKV projections in one command buffer, and all their output projections in another — 188 submissions total, two per layer. The GPU finally gets asked the whole question at once.

And one more theft from the I/O timeline: during routing, the moment the first token votes for an expert, that expert fires at the SSD immediately — while the remaining tokens are still deciding. The reads were always going to happen; now they start milliseconds earlier, hidden under compute. After last article's lesson about speculation, I want to be precise about why this one is legal: replaying yesterday's routing as a guess measured 37% slower and stays banned. This is different — it is certain demand, pulled forward. Never bet. Just start walking earlier.

5. The Evening the Benchmark Lied

Here is the danger in the title.

We land the new coalesced kernel — the one the microbenchmark just crowned at 285 GB/s — into the real engine, run the canonical benchmark, and:

decode: 0.20 tokens/s

Not a typo. One-sixth of the previous speed. The kind of number that, at 11 PM, makes you believe you've corrupted something fundamental — a memory fence, an alignment, the laws of thermodynamics.

Old me would have started reverting commits. Instead, the instrument panel — still the only law of this project — got the first word. And the instrument said something astonishing: the dense math bucket had gotten faster, exactly as the microbench promised. The regression was entirely inside io-spin: the SSD, which reliably delivers 6.3 GB/s, was crawling at 4.4.

The kernel was innocent. The machine was tired. Running benchmark after benchmark back-to-back, each launch mapping a 16 GB Scout and churning gigabytes of Direct I/O, had driven macOS's memory pressure into the ground — and the OS quietly throttles you when you do that. The benchmark wasn't measuring my code. It was measuring my impatience.

One quiet run later: 1.43 tokens/second. The rule is now written into the project's law books: one launch per comparison, on a rested machine — the systems-programming equivalent of "never trust a load test you ran against staging while the nightly backup was running." My decade of being lied to by test environments finally paid rent in C++.

And in the spirit of publishing everything: the numbers in this article were measured on a machine whose memory was half-eaten by my own IDE, because a web developer's laptop is never actually quiet. The reference conditions may be slightly kinder. I'd rather publish the conservative number than wait for a silence that never comes.

6. The Wall, Revisited

So where does a generated token's ~700 ms actually go now?

Where	ms	What it is
SSD spin	~490	demand misses at the NVMe's saturated bandwidth — pure physics
Dense math	~155	weight bandwidth through the new coalesced kernels
GPU wait	~46	expert kernels finishing on late-arriving misses
LM head	~8	one vocabulary-sized matvec

Seventy percent of every token is now the raw, honest cost of flash memory delivering bytes at its rated speed. Last article's cliffhanger hasn't moved, it has sharpened: you cannot make a saturated bus faster, but you can send half the bytes. The 2-bit vault remains the next mountain — it halves the SSD bill, doubles the ring, and finally arms the speculative machinery that has been sitting in the code, patient and default-off, for two articles now. The remaining sliver of dense time has its own successor waiting: the last per-layer CPU round-trip, where the residual and norms still come home between command buffers.

7. What This Entry Is Actually About

The title is a genuine confession. When the speed keeps climbing after you've declared the wall of physics, you start to wonder which direction you're running. Are we chasing some optimal engine that's always one commit ahead? Or fleeing something — the 0.20 t/s ghost, the next benchmark that lies?

Four campaigns in, I finally looked over my shoulder. This is what I found.

And the answer is: neither. Nothing is chasing us. What's behind us is just everything we deleted — the wiped caches, the fired prophet, the write-only memory, and now the scattered reads and the 8,460 tiny GPU round-trips. The engine isn't accelerating toward or away from anything. It's shedding. Every entry of this diary, the mountain weighs less than the diary said it did.

The authorship remains the experiment inside the experiment: the AI agent wrote the simdgroup kernels, the barrier-free dispatch paths, and the microbenchmarks that isolated silicon truth from environmental noise. I brought the questions — why is the GPU only using two threadgroups? why did the benchmark collapse when the kernel got faster? — and the N+1 instincts of someone whose databases have been punishing rude access patterns for years.

The mountain vibrated. Then it talked, then it remembered, then it predicted itself.

Now it runs. And honestly — I've stopped trying to guess what from.

Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. He taught this project to listen to a surface for news of the interior; it turns out the same lesson applies to a benchmark on a tired machine.

The Mountain Predicts Itself — We Fired Our Best Idea, and the Model Got Faster and Smarter

Mon, 13 Jul 2026 18:44:44 +0000

This is the fourth entry in a curious builder's diary. In the first, a self-taught web developer borrowed a satellite engineer's intuition and asked whether frontier AI could run on the MacBook on his desk.
In the second, a 235-billion-parameter model said "Hello, world!" through a door we cut in the memory wall.
In the third, it learned to converse without making you wait minutes between turns.
This one is about the words themselves — and about the day we measured our founding idea against a blank wall, and the blank wall won.

1. The Moment

Third turn of a conversation. Same MacBook, same 48 GB, same 117 GB expert vault cold on the SSD:

A complete, correctly-constructed joke — setup, pause, anatomical pun — from a 235-billion-parameter model streaming its brain off a consumer SSD, three turns deep into a conversation, arriving at a bit over one word per second. That's roughly fifty words per minute. The mountain now speaks at the pace of a human typing.

But the speed is only half of this entry. The other half is why the joke got better — because the same change that made the engine faster also made it, for the first time, run the exact mathematics its creators trained. And that change was a deletion.

2. The Numbers, Plainly

Same hardware, same vault, same prompt discipline as every previous entry. Only the engine changed.

Thing	Last article	Now
Generation speed	0.70 tokens/s	1.13 tokens/s
…since first light	0.48 tokens/s	2.4× total
One generated token costs	~1,430 ms	~885 ms
SSD reads per generated token	~7.5 GB	~3 GB
Expert routing	predicted, 51.5% right	exact — the true Qwen3 routing, every layer, every token
New hardware required	none	none

And one number that doesn't fit in a table: the speed graph of this campaign was not monotone. It went up, then deliberately down, then honestly back up. That dip is the story.

3. The Instrument Panel: The Web Developer's Ultimate Weapon

In my previous articles, I noted how my web developer instincts managed to outmaneuver deep low-level code bugs. In Article 02, it was discovering a config mismatch with the chat template. In Article 03, it was hunting down stale caches, the most web-shaped bug imaginable.

For this phase, I relied on another classic web developer crutch: raw telemetry over systems theory. Everything in this entry came from one new flag: --instrument, which bucketed every millisecond of every generated token. It measured time spent in dense math, time spinning on the SSD, and one statistic nobody had ever actually measured: how often do adjacent tokens want the same experts?.

The answer: 46.4%. Nearly half of what each token needs, the previous token already fetched.

4. The Power of Deletion (Addition by Subtraction)

That 46.4% number illuminated a harsh truth: our biggest performance gains weren't going to come from building new features. They were going to come from ruthless deletions.

Deleting the Wipe: We found a twin bug to the one from our last article. The generation loop was wiping the ring on every single token, re-reading 7.5 GB from the SSD against a 46% reuse opportunity. Deleting this wipe cut decode I/O by two-thirds, making the output bit-identical.
Deleting the Corpse: We discovered our engine was maintaining about 950 lines of ceremony around a 1.6-gigabyte key-value cache that was written to on every token and read by absolutely nothing. It was write-only memory. We deleted it and gave that 1.6 GB straight back to the expert ring.
Deleting the Ping-Pong: The profiler revealed the CPU and GPU were playing ping-pong 188 times per token (synchronous round-trips for projections, norms, and attention). We rewrote the attention block into one Metal command buffer per layer, bringing the engine down to 94 synchronizations and removing the attention math from the CPU entirely.

5. Killing the Darlings: Retiring the Phonon Metaphor

S-MoE was born from Ing. Filippo Biondi's beautiful SAR acoustic imaging metaphor: the Surface Scout, a resident seismic sensor listening to the conversation and predicting which experts the mountain would need next. It was the first article. It was my favorite part of the whole design.

But the reality of language models is that true intelligence requires exact math, not approximation. The instrument panel proved our expensive speculative prophet was costing ~265 ms per token and was only right 51.5% of the time. The passive ring buffer—which costs zero compute—covers 46.4% for free. The prophet, wrong as he was, queued ahead of the demand reads that had to happen anyway, actively delaying the truth.

So, we fired the prophet. We switched generation to compute routing exactly, from the true hidden state, with one tiny matrix multiply per layer.

Trading throughput for truth initially caused a dip in our speed, but the multi-turn output visibly improved. The irony is beautiful: by sacrificing speed to get the exact true Qwen3 routing mixing weights, we eventually unlocked the architectural simplicity needed to make it faster.

6. The Experiments That Failed, Published Anyway

This diary's standing promise. Two more for the pile:

Speculation is real, and it still lost. Ranks 9–16 genuinely cover 63.7% of the next token's experts. We built a low-priority prefetch queue for it, but at 4-bit, it was net-negative—speculative reads simply bid against demand reads on a saturated bus. It sits in the code, default-off, waiting for the 2-bit vault.

We grouped the GPU dispatches, and it changed nothing. Theory said batching each layer's expert kernels would hide wait times. Measurement said the command-buffer overhead was ~5 ms all along. The hypothesis is dead.

7. Hitting the Wall of Physics

Here is where the campaign shifts. Of the ~885 ms each token now costs, about 490 ms is the engine spinning on the SSD.

Three gigabytes of misses per token divided by 490 ms is 6.3 GB/s: the NVMe's actual delivered bandwidth, saturated. There is no more bloated software left between us and the flash memory. There are no dead caches or unnecessary speculative math. For the first time, the biggest cost is simply the raw physics of the hardware doing exactly what it says on the box.

This sets up a beautiful, concrete cliffhanger: you cannot make a saturated physical bus faster, but you can send half the bytes.

The upcoming 2-bit vault will halve every expert blob, halve the SSD bill, double the ring's capacity, and finally arm our speculative queue.

8. What This Entry Is Actually About

The pattern, four articles in, is impossible to miss: the exotic layers keep being innocent. What keeps falling is belief: caches that don't cache, insurance nobody claims, prophets who lose to blank walls. The instrument panel is the only law we've kept, and it has now voted against our own founding metaphor.

And the authorship remains the experiment inside the experiment: the AI agent wrote the NEON intrinsics, the Metal kernels, and the instrumentation. I brought the questions — what is it actually doing in there? who reads this cache? is the prophet even right? — and the willingness to bury the answer's body in public.

The mountain speaks at typing speed now. It no longer needs anyone to predict it.

It predicts itself.

S-MoE is open source under the MIT License. The code, the philosophy, and the mistakes are all public: github.com/melasistema/s-moe. Built by Luca Visciola and an AI agent who hopefully knows C++ and low level programming, because I do not 😅.

Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. The Scout retires with honor — the listening was never wrong; the mountain simply turned out to be its own best seismograph.

"Hello, World!" — A 235-Billion-Parameter Frontier Model Just Spoke on a 48 GB MacBook

Thu, 02 Jul 2026 22:37:31 +0000

This is the second entry in a curious builder's diary. In the first one, a self-taught web developer borrowed a satellite engineer's intuition, teamed up with an AI agent that writes the C++ he cannot, and asked a ridiculous question: can you run frontier AI on the MacBook on your desk? That article ended with a 470 GB download in progress. This one ends differently.

1. The Moment

You: Say hello.  
  
S-MoE [RAM: 48.0GB | Ring: auto]: Hello, world!

That's it. That's the whole story, really.

Qwen3-235B-A22B-Instruct-2507 — a genuinely frontier Mixture of Experts model, 235 billion parameters, the kind of thing that is supposed to live behind an API key and a $250,000 GPU cluster — answered a question on a consumer MacBook. Not a distilled version. Not a 7B "assistant" wearing a big model's name. The actual 235B model, streaming its experts off the SSD, token by token, through a C++ engine and Metal kernels that a web developer and an AI agent built together.

It is slow. It is imperfect. It is also, as far as I can tell, something nobody else has quite done this way. And I think the most honest thing I can do is tell you exactly what it took — because the last bug standing between us and "Hello, world!" was not in the GPU kernels at all. It was the kind of bug a web developer knows in his bones.

2. The Numbers, Plainly

Thing	Reality
Model	Qwen3-235B-A22B-Instruct-2507 (Apache 2.0)
Original checkpoint	~470 GB bfloat16
The Vault (routed experts, SMOE-Q4)	~117 GB, cold on NVMe, 0 bytes of RAM
The Surface Scout (dense backbone)	~16 GB, resident in Unified Memory
Machine	MacBook, 48 GB Unified Memory
Ring buffer	Auto-sized: 25% of free RAM, ~10 MB per expert slot
Observed NVMe throughput	~3 GB/s of expert streaming, prediction miss rate 0.0%
Speed	~0.1–0.2 tokens/second. A word every few seconds. A paragraph while you make coffee.

Yes, you read the last line right. This is not a chatbot you'd use to write emails. It is an existence proof: the memory wall has a door in it, and the door is the SSD.

3. The Bug That Taught Me the Whole Point

Here is my favourite part, because it is where the "web developer mind in low-level code" experiment paid for itself.

After we shattered the model to 4-bit (our first 2-bit attempt mathematically crushed the fine-grained experts — the model produced only spaces and punctuation, like a ghost tapping on a table), the engine ran. The I/O streamed. The Metal kernels hummed. The hidden-state norms were healthy, finite, boringly correct across all 94 layers.

And the model answered every question with:

::

Two colons. That's all. A 235-billion-parameter mountain of intelligence, and it gave us punctuation.

Every instinct from the systems side pointed down — into the dequantisation math, the RoPE rotations, the ring buffer synchronisation. That is where serious low-level bugs are supposed to live. The AI agent and I checked all of it. Greedy decoding on a short raw prompt produced flawless English: "Hello! How can I...". The engine was innocent.

The culprit was upstream, in the smallest, most web-shaped detail imaginable: the chat template. Qwen3 ships in two lineages — the original "thinking" model, and the non-thinking Instruct-2507 we had shattered. The thinking lineage's template injects <think> tokens into every prompt. And in the Instruct-2507 weights, those two token embeddings are untrained — near-zero rows, ~150× smaller than every normal token, sitting in the vocabulary like unexploded blanks. We were whispering two meaningless syllables into the model's ear at the start of every conversation, and the residual stream collapsed on contact.

One line changed — load the tokenizer from the same checkpoint you shattered — and the mountain said hello.

The lesson I keep relearning: the terrifying low-level layer was fine. What broke was a config mismatch, a wrong template, an error message silently swallowed by a pipe — the exact species of bug every web developer has hunted at 2 AM in a misconfigured staging environment. My instincts were useless for writing Metal kernels, and genuinely useful for debugging the system around them. That mix — an agent that can write memory_order_acquire fences, and a human who reflexively asks "wait, are we even sending the right payload?" — is the actual experiment here.

4. An Honest Word About 16 GB

The original dream — the one in the manifesto — was 16 GB. "16 GB of RAM should give you the same intelligence as 512 GB, just slower."

We almost did it. And I want to be precise about the "almost", because I promised this diary would be sincere.

The architecture held: the 117 GB of expert weights truly occupy zero bytes of RAM. They stream from the SSD at the moment of need, exactly as the phonon metaphor promised. That part of the claim survived contact with reality completely.

What we could not shrink is the Surface Scout — the model's own dense backbone, the part that must stay resident to predict what fires next. For a 235B frontier model, that backbone alone weighs ~16 GB in bfloat16. It is the size of the entire machine we were aiming for. Chasing the frontier on 16 GB was, in hindsight, a bit too hazardous — there would be no room left for the ring buffer, the KV-cache, or macOS itself.

So the honest, recalibrated claim is this:

A 32–48 GB MacBook now runs a 235B frontier model locally, with intelligence that does not degrade — only speed does. And a 16 GB Mac runs the same architecture with smaller fine-grained MoE models, same principle, same sovereignty.

I have made my peace with this. Frontier-on-32GB was science fiction in January. Being annoyed that it isn't frontier-on-16GB feels like complaining about the legroom on your first flight to the Moon.

5. What's Next

The system that says "Hello, world!" today is the slowest version of itself that will ever exist. The roadmap is public and concrete: computing expert routing from the heavy model's own hidden state instead of the Scout's approximation, replacing the GPU's waitUntilCompleted with MTLSharedEvent pipelining, and eventually a small distilled Scout that predicts activations at a fraction of the cost. None of this requires new hardware. All of it requires more evenings.

And I'll keep being honest about authorship: the AI agent wrote the C++ and the Metal shaders, reasoned through memory fences and page alignment, and found bugs in code I could only read like a tourist reads street signs. I brought the question, the direction, the stubbornness, and — it turns out — the debugging instincts of someone who has been lied to by config files for more than ten years. Neither of us could have built this alone. I am not sure anyone has built quite this pairing before, and that, more than the tokens per second, is what I think is worth noticing.

The mountain doesn't just vibrate anymore.

It talks.

Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. The first words belong to the mountain; the intuition to listen for them belongs to him.

Rock, Paper, Silicon: How a Web Developer Used a Satellite Hack and an AI Agent to Ask...

Sun, 21 Jun 2026 23:36:00 +0000

Rock, Paper, Silicon: How a Web Developer Used a Satellite Hack and an AI Agent to Ask a Ridiculous Question About Frontier AI

This is not an academic research paper. I am certainly not a systems engineer. I am a self-taught, full-stack web developer — the kind whose GitHub is full of frontends and web stacks, not GPU kernel optimizations or low-level memory management. This is simply a curious builder's diary about an experiment that probably shouldn't have worked, built with an AI agent who hopefully knows C++ and processors, because I do not 😅.

1. The Problem I Wasn't Qualified to Solve

Silicon Valley has a bucket problem.

To run a modern frontier AI model — something like Qwen3-235B with its 235 billion parameters — the industry tells you that you must hold the entire mathematical mountain in RAM simultaneously. At bfloat16 precision, that is roughly 470 GB of memory. On a good day, a high-end Mac Studio ships with 192 GB. A standard MacBook? 16 to 48 GB.

The conclusion drawn by everyone qualified to draw conclusions is: you cannot run frontier AI locally on consumer hardware. Go rent a cloud GPU.

The thing that nagged me: In a Mixture of Experts model, 95%+ of those 235 billion parameters are completely silent at any given millisecond. For a token about philosophy, the math experts never fire. For a token about code, the poetry experts never fire. The weights are cold stone. Standard runtimes hold all of them in RAM, just in case.

I am not a systems architect. I have no background in high-performance computing. My daily bread is APIs, web stacks, and the kind of code that orchestrates systems rather than talks to raw silicon. But I had a question that wouldn't leave me alone:

What if you only loaded what fires, and loaded it just before it fires?

I didn't know how to answer it. So I called for help.

2. The AI Agent as a Bridge

I want to be completely honest about how this was built.

I opened a conversation with an AI agent and I told it my idea. I described the shape of what I wanted. I asked naive questions. The agent wrote C++. I read it, trying to make sense of pointers and memory allocations. I pushed back when things got too complex. It revised. We iterated.

I am not writing this to diminish the work — there is real, terrifyingly low-level engineering here, and I understand a little more of it with each session. But I want to be honest that the architecture we landed on emerged from a collaboration where I brought the curiosity and the direction, and the agent brought the implementation depth. My GitHub profile tells the truth: I am a web developer who had a question and found a way to pursue it beyond my formal training by talking to a machine.

What I could bring was exactly what pure computer science training sometimes blocks: I wasn't afraid to ask an "obviously impossible" question, because I simply didn't know enough to know it was supposed to be impossible.

3. The Satellite Man Who Looked Sideways

My technical inspiration came from a place nobody in Silicon Valley would have looked.

Ing. Filippo Biondi published a paper in 2022 in the MDPI journal Remote Sensing titled "Synthetic Aperture Radar Doppler Tomography Reveals Details of Undiscovered High-Resolution Internal Structure of the Great Pyramid of Giza". It is a stroke of pure genius, and the underlying physical intuition is what captured my imagination.

Here is the idea:

Electromagnetic waves cannot penetrate rocks and go underground. That is not a limitation to work around — it is physics. SAR satellites orbit Earth and illuminate surfaces with microwave radar pulses, but those pulses bounce off solid stone.

But Biondi used a brilliant technique. When the EM wave strikes the pyramid's exterior, it generates acoustic phonons — mechanical sound vibrations that propagate through the stone. The stone vibrates in response to the surface strike, and the internal geometry of the pyramid modulates those vibrations.

The satellite cannot see inside the rock. But using a Doppler technique and some Fourier transforms, Biondi measured the sub-nanometer surface displacements caused by those returning phonons. He used those phonon readings to gain Tomographic images, capturing all the 3D information of the pyramid's interior projected beautifully onto a 2D plane.

SAR Satellite  
 │ │  microwave pulse (cannot penetrate rock) ▼┌──────────────────────────────────────────┐  
│  PYRAMID SURFACE                          │  
│  EM pulse → generates phonons in stone    │  
│  Phonons propagate through interior       │  
│  Internal geometry modulates wavefront    │  
│  Surface micro-vibrates (sub-nanometer)   │  
└──────────────────────────────────────────┘  
 │ │  Doppler shift in reflected SAR signal │  encodes surface displacement ▼Fourier transform across synthetic aperture  
 │ ▼3D tomographic image (internal structure)

Stop fighting the physical barrier. Measure what the barrier does to the signal instead.

I am not comparing my experiment to Biondi's work. He is an engineer doing something I cannot evaluate. But his framing — you cannot see inside the rock, so measure the sound the rock makes when you strike it — became the exact mental model I needed for the LLM memory problem.

4. The Translation: Acoustic Streaming for LLMs

The translation from Biondi's method to a language model inference problem is surprisingly direct:

Seismic Concept	S-MoE Equivalent
Deep rock strata	The 235B model weights, cold on the NVMe SSD
EM surface pulse	The Surface Scout processing the current token
Generated phonons	The Scout's predicted expert activation map
Acoustic receivers	The async I/O thread reading predictions
Seismic map	The K-step expert ID lookahead schedule
Reconstructed image	The generated token

The insight is this: you don't need to read the full model to know what parts of it will activate next. If you run a lightweight forward pass on the model's dense backbone — the embeddings, attention layers, and routing gates — those routing gates will tell you, probabilistically, which expert columns in the FFN are about to fire for the next several tokens.

This is the Surface Scout. It is not a separate model I trained. It is the dense parts of the target model itself — everything except the routed expert weights — running permanently in Unified Memory. Its routing gate matrices produce logits that are, in effect, a phonon map: a structural fingerprint of what the full model is about to need.

5. The System That Emerged

S-MoE (Seismic Mixture of Experts) is the result of this experiment. Built in C++ with a Metal compute backend, co-designed with an AI agent, it runs on Apple Silicon Macs.

It works in three concurrent streams:

The Sculptor (offline, run once)

A Python script (shatter_moe.py) takes any supported MoE model and splits it into two artifacts:

The Vault (.smoe): All routed expert blocks, aggressively quantised and page-aligned to 16 KB hardware boundaries for Direct I/O
The Scout (.scout.safetensors): The dense backbone — embeddings, attention, norms, routing gates — that lives in RAM

The Scout (always running)

At every token step, the Scout runs a complete forward pass on the current token. Its routing gate outputs produce a prediction of which experts will activate across all MoE layers for the next K tokens. This phonon map is handed to the Streamer.

The Streamer + Metal Kernel (concurrent)

Guided by the Scout's predictions, background I/O threads fire pread() calls with F_NOCACHE — bypassing the OS page cache entirely — loading the predicted expert blobs directly into a pre-allocated, page-aligned ring buffer in Unified Memory. The Metal GPU kernel reads from that ring buffer and executes the FFN computation via a fused dequant-multiply that decodes compressed weights directly in GPU register space.

The three streams run simultaneously. The GPU executes experts loaded one step ago. The Streamer loads experts needed one step from now. The Scout predicts experts needed K steps from now. No thread waits for another.

There are three rules the system never breaks:

No runtime heap allocations. malloc, new, and std::vector::resize are banned inside the token generation loop. Every buffer is pre-carved at startup.
Direct I/O only. F_NOCACHE on every vault file descriptor. SSD → DMA → RAM. No OS copy.
Atomic synchronisation only. No OS mutexes. No blocking. The I/O thread and GPU thread are structurally incapable of blocking each other.

6. The Model-Agnostic Evolution

The system started hardcoded for a single model — DeepSeek-MoE-16B, a 16B parameter model small enough to test quickly. But the original question was never about one model. It was about whether the approach could scale.

Over several weeks of collaboration with the AI agent, I refactored the entire engine to be model-agnostic. Now, when you point S-MoE at a model's Scout file, it reads the tensor header at boot time and auto-detects:

Hidden dimension, vocabulary size, FFN intermediate dimension
Number of MoE layers, experts per layer
Whether Layer 0 is a dense MLP (DeepSeek style) or a full MoE layer (Qwen style)

The engine reshapes itself to fit. No recompilation. No configuration files.

The current target is Qwen3-235B-A22B-Instruct-2507 — a genuinely frontier model, 235 billion parameters, 22 billion active per forward pass, 128 experts per MoE layer, Apache 2.0 licensed. A 400 GB download. The kind of model that is supposed to require a $250,000 GPU cluster.

It is downloading on my MacBook right now.

7. The Democratic Claim

I want to be careful not to oversell this. S-MoE is an experiment. It is incomplete. The Scout's predictions are imperfect, which means the Streamer sometimes misses and the system spin-waits. The quantisation introduces error. The throughput on a 16 GB Mac running a 235B model will be slow — perhaps one token every few seconds.

But here is the claim I am willing to stand behind:

A 16 GB Mac and a 512 GB Mac will produce identical outputs. The 512 GB Mac will produce them faster.

Speed scales with hardware. Intelligence does not degrade.

The user with the MacBook Air and the user with the Mac Pro get the same 235 billion parameters. The same knowledge. The same reasoning depth. The same model. One waits longer. Both get the answer.

This is not a smaller, distilled, dumbed-down model for the masses. It is the real thing, delivered at the rate the hardware allows.

I built this because I believe the memory wall around frontier AI is partly physical and partly artificial — a consequence of software assumptions that nobody questioned, not a law of nature.

I am a web developer who asked what else you could do with the NVMe that's sitting three inches from the CPU. The AI agent helped me find out.

The mountain is already vibrating. You just have to listen.

Acknowledgements: The seismic tomography concept is an experimental, philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. His methodology and its application to the Giza Plateau is his genius, not mine — I merely borrowed his intuition to solve a problem with language models.