Coder Legion - Recent posts tagged applesilicon

S-MoE: Seismic Mixture of Experts Engine

Wed, 07 Oct 2026 21:38:55 +0000

S-MoE: a 235-billion-parameter model, streamed from a laptop SSD

S-MoE (Seismic Mixture of Experts) is an open-source inference engine for fine-grained Mixture-of-Experts LLMs on Apple Silicon. It streams experts from the NVMe SSD at the moment they are needed, so the model never has to fit in RAM. Qwen3-235B runs on a MacBook from 32 GB of unified memory up.

I am a self-taught web developer. I built this with an AI agent that knows C++ and low-level programming, because I do not.

The story

A question I was not qualified to ask

To run a frontier model like Qwen3-235B, the usual answer is that all 235 billion parameters must sit in memory at once. At bfloat16 that is roughly 470 GB. A standard MacBook has 16 to 48 GB. The conclusion everyone draws is: rent a cloud GPU.

One detail kept nagging me. In a Mixture-of-Experts model, more than 95% of those parameters are silent at any given moment. At each layer, for each token, the router picks 8 experts out of 128. The rest is cold stone, and standard runtimes hold it in RAM "just in case".

So I asked the obvious, naive question: what if you only loaded what fires, just before it fires?

I did not know enough to know it was supposed to be impossible. That turned out to be useful.

The satellite man who looked sideways

The mental model came from a place far from AI. In 2022 Ing. Filippo Biondi published a paper in Remote Sensing on imaging the inside of the Great Pyramid of Giza with Synthetic Aperture Radar. Radar cannot penetrate rock. His idea was to stop fighting that: the pulse strikes the surface, the stone vibrates, and the vibrations carry the shape of what is inside. You measure what the barrier does to the signal.

I borrowed that intuition, and only the intuition:

Seismic concept	S-MoE equivalent
Deep rock strata	The expert weights, cold on the NVMe SSD
EM surface pulse	The dense backbone running the current token
Generated phonons	The router gates' exact activation echo
Acoustic receivers	The async I/O streamer and its retained ring
Subsurface image	The generated token

That is where the name comes from, and why the project speaks of mountains, strikes and vaults.

Built with an AI agent, honestly

I described the shape of what I wanted and asked naive questions. The agent wrote C++ and Metal. I read it, pushed back when it got too complex, and we iterated. I brought the curiosity and the direction; the agent brought the implementation depth. I understand a little more of the low-level code with each session.

The first target was a small model, DeepSeek-MoE-16B. Then the engine was refactored to be model-agnostic, and the real target arrived: Qwen3-235B, a 470 GB download onto a laptop.

What the numbers corrected

The project is measurement-driven, and two of my own ideas did not survive measurement.

The 16 GB claim. The original promise was "16 GB of RAM gives you the same intelligence as 512 GB, just slower". The experts really do occupy zero bytes of RAM, but the dense backbone of a 235B model weighs about 16 GB by itself. The honest floor for the frontier is 32 GB. A 16 GB Mac runs the same architecture with smaller MoE models. The manifesto was amended in public.
The Scout. The first design had a "Surface Scout" that predicted which experts the model would need next. Measured as a predictor, it covered 51.5% of the next experts at about 265 ms per token. Simply keeping recent experts in the ring covered 46.4% at zero cost, and the model's own router gate gives the exact answer with one small matrix multiply. The predictor was removed. The engine became faster and more faithful to the model at the same time.

The whole journey is written up as an eight-part build diary, linked at the bottom.

How it works

A model is prepared once, then served by three parts that run concurrently.

0. The shatter (offline, once)

shatter_moe.py takes a MoE checkpoint and splits it into two files:

The vault (.smoe): every routed expert, quantised to 4 bits and aligned to Apple Silicon's 16 KB page boundary so it can be read with Direct I/O. For Qwen3-235B this is about 117 GB.
The backbone (.scout.safetensors): embeddings, attention, norms and routing gates, kept in bfloat16. About 16 GB for the 235B.

The vault is self-describing. RoPE theta, routing top-k, attention geometry and lineage are written into an arch block from the checkpoint's own config.json, so the engine reads what it is given and needs no per-model configuration.

1. The resident backbone (in memory)

The backbone stays in unified memory. At every layer of every token it evaluates the real router gate on the true hidden state. That names the exact experts the model wants, the same ones it was trained to use.

2. The deep strata (on the SSD)

The vault rests cold on NVMe and holds zero bytes of RAM. Nothing is memory-mapped "just in case".

3. The acoustic receiver (in between)

Background workers read the named experts with Direct I/O (F_NOCACHE), bypassing the OS page cache, into a lock-free ring buffer a fraction of a second before the Metal GPU kernel runs them. Weights are dequantised directly in GPU registers.

The ring is also an LRU cache. Adjacent tokens reuse 46.4% of their experts, and those are already there, so they are never read again.

prompt token
    │
    ▼
backbone in RAM ──► router gate ──► "layer 37 needs experts 4, 19, 52, …"
                                         │
                       in the ring? ─────┤
                        yes: cache hit   │ no: Direct I/O read from the vault
                                         ▼
                                  ring buffer (unified memory)
                                         │
                                         ▼
                                Metal kernel ──► next layer ──► token

The rules the engine never breaks

Zero runtime heap allocations. malloc, new and std::vector::resize are illegal inside the token loop. Every buffer is carved at startup.
Direct I/O only. SSD to unified memory to GPU, with no page cache and no extra copies.
Atomic synchronisation only. No mutexes and no condition variables. The I/O thread and the GPU thread cannot block each other.
Decoupled routing. The gates measure, the streamer moves bytes, the kernel multiplies. None of them knows what the others do.

Tech stack

Core: C++20, data-oriented, POSIX primitives
Compute: Metal shaders behind an Objective-C++ bridge, zero-copy in unified memory
Tooling: Python for the shatter script, the chat console and the HTTP server

Key features

A frontier model on a laptop. Qwen3-235B (94 MoE layers × 128 experts) on a 32 to 48 GB MacBook, with about 16 GB resident and the ~117 GB vault at zero bytes of RAM.
Intelligence does not degrade, only speed does. A 32 GB Mac and a 512 GB Mac produce identical outputs from the same vault. More memory means a larger ring and more tokens per second.
Model-agnostic. Any compatible fine-grained MoE becomes a vault with one shatter command. No recompilation.
A model fleet, switchable at runtime. Every vault on disk is listed on /v1/models. Naming another one in the standard model field switches the engine.
OpenAI and Anthropic compatible. One process, one port, both protocols, streaming or not. Open WebUI, LibreChat, the openai and anthropic SDKs and Claude Code connect without being modified.
Built-in web chat. A dependency-free console at GET / with live turn timings, context and RAM meters, and a model dropdown.
Persistent sessions. The KV-cache and the ring survive across turns, so each message prefills only its new tokens. Follow-up turns answer in about 14 s however long the conversation is.
Layer-major batched prefill. Each layer's experts are read from the SSD once per prompt chunk, not once per token.
Private and offline. No cluster, no API key, nothing leaves the machine.
Open source, MIT. The code, the documentation and the mistakes are public.

The numbers

Measured on a standard 48 GB Apple Silicon MacBook, Q4 vault, 45-token prompt.

	Qwen3-235B-A22B	Qwen3-30B-A3B
Role	The frontier	The daily driver
Vault (Q4)	~117 GB, streamed from SSD	~14 GB, almost entirely in the ring
Decode	1.84 tok/s	~13.7 tok/s

For the 235B, since the latency work began:

Metric	Before	Now
Cold time to first token	92.7 s	~26 s
Decode speed	0.48 tok/s	1.84 tok/s
Warm follow-up turn	grew with the conversation	~14 s, flat

I will not pretend 1.84 tokens per second is fast. It is a 235B model answering at reading speed on a machine you can close and put in a bag. The 30B is the one for daily use.

Try it

You need an Apple Silicon Mac and a fast internal SSD.

git clone https://github.com/melasistema/s-moe.gitcd s-moemake setup

Then download a supported checkpoint, shatter it and serve it:

make probe   MODEL=./checkpoints/qwen3-235b-instruct              # dry run, maps the topologymake shatter MODEL=./checkpoints/qwen3-235b-instruct OUT=./vault  # builds the vaultmake all                                                          # builds the engine.venv/bin/python serve.py --port 8000                             # OpenAI + Anthropic + web chat

Open http://127.0.0.1:8000/ and start typing, or point an existing client at it:

from openai import OpenAI  
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")

ANTHROPIC_BASE_URL=http://127.0.0.1:8000 ANTHROPIC_API_KEY=unused claude

The full guide, including the Qwen3-30B path for smaller machines, is in the documentation.

What is next

Q2 vaults. Decode is bound by SSD bandwidth. Halving every expert halves the reads and doubles the ring.
The 477B. GLM MoE 477B is the next summit. It is not shattered yet.
Paged KV-cache, so long conversations do not depend on RAM.
Adaptive quantisation, keeping popular experts at 4 bits and compressing niche ones to 2.

S-MoE is an experiment. It is incomplete, and I would rather show the measurements than oversell it.

The Mountain Exhales — Riding the 30B Wind Before the 477B Avalanche

Tue, 21 Jul 2026 09:00:00 +0000

This is the seventh entry in a curious builder's diary. Over six chapters, we carved a door through Silicon Valley's artificial memory wall, coaxing a 235-billion-parameter mountain to speak, remember, run, and finally converse fluently in the OpenAI dialect. As we push past an average of 1.84 tokens per second and touch peaks of 2.01 on a consumer 48 GB MacBook, the engine is humming. But before we pack our gear for the next impossible summit, we are claiming a builder's privilege. After months of writing lock-free C++ to move heavy stone, it is time to take a breath of thin, fresh air.

1. The 477B Avalanche

S-MoE was built to prove a point: that the memory wall is a software convention, not a law of physics. We took Qwen3-235B — a model that standard runtimes insist requires cloud clusters — and shattered it into pieces that stream perfectly off a MacBook's SSD via Apple Silicon's Unified Memory Architecture.

Because we broke the diary dependency and invited standard frontends (like Open WebUI) into the architecture in Article 06, we are now ready to tackle an even more colossal peak: the GLM MoE 477B parameter model.

The climb will be grueling. A 477-billion-parameter model will push consumer hardware to an edge no one thought possible. It will test every byte of our zero-allocation loops, our Direct I/O DMA transfers, and our patience. But we have the map now.

However, before we shoulder that massive weight, we realized something crucial. S-MoE shouldn't just be an engine for the 235B. It should be an engine for any fine-grained MoE.

2. The Self-Describing Vault (SARC)

Until now, the engine was heavily hardcoded for Qwen3's specific math. If we were going to run different models — the daily driver 30B, the massive 235B, or the upcoming 477B — we couldn't keep recompiling C++ to change the RoPE theta or the routing Top-K.

We needed the engine to be model-agnostic.

The solution was elegantly simple: the Smoe ARChitecture (SARC) block. During the offline shatter process, the Python script now reads the model's Hugging Face config.json and extracts the mathematical DNA — RoPE theta, MoE Top-K, GQA head geometry, and normalization flags. It serializes these into a tiny, 128-byte binary block and stamps it directly into the .smoe vault's header.

Now, when you boot S-MoE, there are no config files or CLI flags to set. The engine simply reads the 128 bytes, understands the topology of the mind it was just handed, and configures its own internal Scout and Metal shaders accordingly.

We suddenly had a "fleet." You can have a directory full of different vaults, and simply switching the model in your Chat UI dropdown seamlessly swaps the models at runtime.

3. The 30B Wind

What happens when you take an engine hardened by the brutal math of a 235B model, and plug a 30B model into it?

We shattered the Qwen3-30B-A3B checkpoint to find out. We wanted to experience pure, frictionless velocity. We wanted to ride the wind.

But when we first spun it up, the numbers were confusingly low. We were seeing about ~5 to 7 tokens per second. Fast, but not "frictionless velocity" fast. We expected the 30B to scream on a 48 GB machine.

The culprit? A single hardcoded flag from our testing days: --ring 1024.

4. The Magic of Auto-Sizing

The Streamer's ring buffer is the heart of S-MoE's speed. It's a pre-allocated LRU cache in RAM. When a token needs an expert, the engine checks the ring first. If it's there (a cache hit), we skip the SSD entirely.

By hardcoding --ring 1024, we were forcing the ring to only hold 1,024 experts (about ~2.5 GB of cache for the 30B).

But S-MoE has a brilliant auto-sizing mechanism (--ring 0). When allowed to auto-size, the engine dynamically checks the OS for available RAM, subtracts a safe overhead, and gives the rest to the ring. On a 48 GB MacBook, the auto-tuner allocates roughly ~15 GB of ring capacity.

The entire 30B vault is only ~14 GB.

The moment we removed --ring 1024 and let the engine auto-size, the ring simply swallowed the entire model. After the first few turns, the SSD reads dropped to virtually zero. The engine ascended from a streaming pipeline into an pure, in-memory execution loop.

The numbers instantly skyrocketed.

5. The True Benchmarks

With the auto-tuner unleashed and the engine running isolated, the true benchmarks emerged.

The Daily Driver: Qwen3-30B

Cold Time-To-First-Token: ~5.85 seconds
Decode Speed: 13.71 tokens / second
The Feeling: Instantaneous. Pure velocity. The entire mind rests comfortably in RAM, producing outputs faster than you can read them.

The Frontier: Qwen3-235B

Cold Time-To-First-Token: ~26.16 seconds
Decode Speed: 1.84 tokens / second (peaking over 2.01 on warm turns)
The Feeling: Cinematic deliberation. The SSD streams experts at ~150+ GB/s, and the mountain answers thoughtfully.

6. The Plateau Before the Climb

We built S-MoE to prove that the 235B frontier could live on a desk. But in making the architecture truly agnostic, we accidentally built the fastest possible way to run the 30B models, too. The same lock-free memory gymnastics that make the heavy models possible make the lighter models absolutely fly.

Because sometimes, before you carry the next mountain, you just need to feel how fast you can sprint across the plateau.

The engine is ready. The architecture is model-agnostic. The UI is clean.

Next stop: 477 Billion parameters.

S-MoE is open source under the MIT License. The code, the philosophy, and the mistakes are all public: github.com/melasistema/s-moe.
Built by Luca Visciola and an AI agent who hopefully knows C++ and low level programming, because I do not 😅.

The Paper Sketch: Budgeting 48GB of UMA

Sun, 19 Jul 2026 14:07:16 +0000

Let's look at the physical footprints at a 4-bit (Q4) quantization. A 744B parameter model sits at roughly 370 GB cold on the NVMe drive. Because it’s a sparse Mixture of Experts, the dense backbone—the attention mechanisms and shared weights that every single token must touch—is surprisingly light: about 17B parameters, or roughly 9.9 GB.

If we draw up the memory allocation inside our 48GB pool, the board looks like this:

48 GB (Total) - 9.9 GB (Dense Backbone) - 4 GB (OS & Apps) - 2 GB (KV Cache) ≈ 32 GB Remaining

That leaves us a 32GB sandbox to allocate entirely to the lock-free expert ring buffer.

When this model executes, it activates roughly 40B parameters per token. At Q4 precision, that requires pulling ~11 GB of active experts into memory per step. Because our ring buffer is 32GB, it has enough structural breathing room to hold 2 to 3 slots of speculatively loaded future tokens simultaneously. The memory geometry fits.

Calculating the Bandwidth Reality

Now we look at the unyielding bottleneck: the internal Apple Silicon SSD bus, which reliably pushes a sustained read bandwidth of about 6.5 GB/s.

If the engine had to blindly pull a completely fresh set of 11 GB of experts for every single token, the raw physical limit is basic arithmetic:

Raw I/O Time Per Token = 11 GB / 6.5 GB/s ≈ 1.69 seconds ⇒ 0.59 tokens/second

But we know neural routing isn't completely chaotic. Tokens traveling through a sentence exhibit temporal expert locality—the token handling a specific technical context will likely trigger some of the exact same experts used by the token right before it.

If the speculative lookahead successfully exploits this, allowing us to reuse just 20% of the active experts already sitting warm in the ring buffer, the actual disk read demand drops from 11 GB down to ~8.8 GB:

Optimized I/O Time = 8.8 GB / 6.5 GB/s ≈ 1.35 seconds ⇒ 0.74 tokens/second

This lands us squarely in that 0.6 to 0.8 tokens per second window.

Perspectives on Speed

Not that we are calling this fast by modern server standards, but it completely rewrites the expectations for what a local machine can do. It’s all about perspective.

We used to wait hours just to download a single movie;

Waiting 30 or 40 seconds for a frontier-class AI to generate a deeply reasoned paragraph locally on a laptop isn't a penalty—it's a miracle.

Will it hold up when the code actually executes? The ultimate variable is the "Scout Fan-Out." With hundreds of potential experts per layer, the speculative engine's branch prediction has to be incredibly precise. If it misguesses too often, the pipeline will stall, waiting for synchronous disk reads.

But looking at the numbers on the board right now, the physics check out. We aren't just daydreaming—this approach stands a very real chance of turning an impossible 744B paperweight into a highly capable, private asynchronous workhorse.

The Mountain Runs — I Don't Know If We're Chasing Something or Being Chased, but the Speed Won't...

Thu, 16 Jul 2026 07:31:21 +0000

The Mountain Runs — I Don't Know If We're Chasing Something or Being Chased, but the Speed Won't Stop Climbing

This is the fifth entry in a curious builder's diary. In the first, a self-taught web developer borrowed a satellite engineer's intuition and asked whether frontier AI could run on the MacBook on his desk. In the second, a 235-billion-parameter model said "Hello, world!" through a door we cut in the memory wall. In the third, it learned to converse without making you wait minutes between turns. In the fourth, we fired our own founding metaphor and the model got faster and smarter. That article ended at the wall of physics: a saturated SSD, nothing left to delete. This one is about what happened next, which is that the speed kept climbing anyway — and for one terrifying evening, it looked like it had collapsed by 85% instead.

1. The Moment

A microbenchmark, late at night. Same MacBook. Same GPU. Same weights, same mathematics, same number of bytes read. Two kernels that compute the identical matrix-vector product:

K/V projection (512 × 4096, bf16)  
  
old kernel:   12.3 GB/s new kernel:  285.3 GB/s        (23×)

Twenty-three times faster. Nothing was added — no new hardware, no new math, no quantisation trick. We changed the shape of the question the GPU asks its own memory, and the same silicon that had been trudging along at 12 GB/s broke into a sprint.

Last article ended with me declaring, confidently, that we had hit the wall of physics — that the SSD was saturated and there was no more bloated software between us and the flash memory. That was true, and it remains true. What I had not understood is that the SSD was only one of the walls, and behind it the GPU had been quietly doing the memory-system equivalent of taking a taxi one block at a time.

2. The Numbers, Plainly

Same hardware, same 117 GB vault cold on the SSD, same canonical prompt discipline as every previous entry. Only the engine changed.

Thing	Last article	Now
Generation speed	1.13 tokens/s	1.43 tokens/s
…since first light	0.48 tokens/s	3× total
One generated token costs	~885 ms	~700 ms
First token, cold 45-token prompt	~32 s	~29.4 s
GPU dense projections	12–136 GB/s	207–285 GB/s
GPU syncs during prefill	2 per token	2 per layer
Output token IDs	—	bit-identical, before and after
New hardware required	none	none

And one number that isn't in the table, because it deserves its own section: for a few hours in the middle of this campaign, the official benchmark read 0.20 tokens/second — six times slower than where we started the evening. That story is below, and it is the most web-developer-shaped lesson of the entire diary.

3. The GPU Was Running N+1 Queries

Every web developer knows the N+1 query bug. You ask the database for a list of users, then loop over them asking for each one's profile — a thousand tiny round-trips where one JOIN would do. The database isn't slow. Your access pattern is rude.

It turns out a GPU's memory system has exactly the same personality.

The old kernels assigned one thread per output row of the weight matrix. Sounds reasonable — it's the obvious loop, the one you'd write first. But 32 neighbouring GPU threads execute in lockstep, and under that layout their simultaneous memory requests land at addresses two kilobytes apart. The memory system, which serves wide consecutive bursts beautifully, has to satisfy 32 scattered pleas one crumb at a time. A thousand tiny round-trips. N+1 queries against your own RAM.

The new kernels invert it: one simdgroup — 32 threads — owns one output row together, and its lanes read consecutive 16-byte chunks. Same bytes, same total work, but now every request is one coalesced 512-byte burst. The JOIN.

The quantised expert FFN kernels — the ones that dequantise 2/4-bit codes in register space, the heart of the whole engine — jumped 7×, to 158 GB/s effective.
The dense Q/K/V/O projections and the LM head jumped to 207–285 GB/s.
The comedy prize goes to the K/V projection: its matrix is only 512 rows tall, and the old dispatch geometry handed the entire job to 2 threadgroups — on a GPU built to run thousands. We had booked a stadium and seated two people. That's the 23×.

The pattern of this diary holds for a fifth consecutive entry: the exotic layer was innocent. The dequantisation math was right, the Metal code was correct, every byte arrived. What was broken was the shape of the request — the most database-shaped bug imaginable, living in the middle of a GPU shader.

4. The Prompt Stopped Playing Ping-Pong

The second front was the prefill — the silence before the first word, which article three cut from 93 seconds to 32.

Reading the prompt, the engine was still submitting GPU work per token: every one of the 45 prompt tokens made its own little QKV round-trip and its own little o_proj round-trip, each ending with the CPU standing at the door waiting for the GPU to ring the bell. Roughly 8,460 command buffers for one prompt. Now each layer submits all tokens' QKV projections in one command buffer, and all their output projections in another — 188 submissions total, two per layer. The GPU finally gets asked the whole question at once.

And one more theft from the I/O timeline: during routing, the moment the first token votes for an expert, that expert fires at the SSD immediately — while the remaining tokens are still deciding. The reads were always going to happen; now they start milliseconds earlier, hidden under compute. After last article's lesson about speculation, I want to be precise about why this one is legal: replaying yesterday's routing as a guess measured 37% slower and stays banned. This is different — it is certain demand, pulled forward. Never bet. Just start walking earlier.

5. The Evening the Benchmark Lied

Here is the danger in the title.

We land the new coalesced kernel — the one the microbenchmark just crowned at 285 GB/s — into the real engine, run the canonical benchmark, and:

decode: 0.20 tokens/s

Not a typo. One-sixth of the previous speed. The kind of number that, at 11 PM, makes you believe you've corrupted something fundamental — a memory fence, an alignment, the laws of thermodynamics.

Old me would have started reverting commits. Instead, the instrument panel — still the only law of this project — got the first word. And the instrument said something astonishing: the dense math bucket had gotten faster, exactly as the microbench promised. The regression was entirely inside io-spin: the SSD, which reliably delivers 6.3 GB/s, was crawling at 4.4.

The kernel was innocent. The machine was tired. Running benchmark after benchmark back-to-back, each launch mapping a 16 GB Scout and churning gigabytes of Direct I/O, had driven macOS's memory pressure into the ground — and the OS quietly throttles you when you do that. The benchmark wasn't measuring my code. It was measuring my impatience.

One quiet run later: 1.43 tokens/second. The rule is now written into the project's law books: one launch per comparison, on a rested machine — the systems-programming equivalent of "never trust a load test you ran against staging while the nightly backup was running." My decade of being lied to by test environments finally paid rent in C++.

And in the spirit of publishing everything: the numbers in this article were measured on a machine whose memory was half-eaten by my own IDE, because a web developer's laptop is never actually quiet. The reference conditions may be slightly kinder. I'd rather publish the conservative number than wait for a silence that never comes.

6. The Wall, Revisited

So where does a generated token's ~700 ms actually go now?

Where	ms	What it is
SSD spin	~490	demand misses at the NVMe's saturated bandwidth — pure physics
Dense math	~155	weight bandwidth through the new coalesced kernels
GPU wait	~46	expert kernels finishing on late-arriving misses
LM head	~8	one vocabulary-sized matvec

Seventy percent of every token is now the raw, honest cost of flash memory delivering bytes at its rated speed. Last article's cliffhanger hasn't moved, it has sharpened: you cannot make a saturated bus faster, but you can send half the bytes. The 2-bit vault remains the next mountain — it halves the SSD bill, doubles the ring, and finally arms the speculative machinery that has been sitting in the code, patient and default-off, for two articles now. The remaining sliver of dense time has its own successor waiting: the last per-layer CPU round-trip, where the residual and norms still come home between command buffers.

7. What This Entry Is Actually About

The title is a genuine confession. When the speed keeps climbing after you've declared the wall of physics, you start to wonder which direction you're running. Are we chasing some optimal engine that's always one commit ahead? Or fleeing something — the 0.20 t/s ghost, the next benchmark that lies?

Four campaigns in, I finally looked over my shoulder. This is what I found.

And the answer is: neither. Nothing is chasing us. What's behind us is just everything we deleted — the wiped caches, the fired prophet, the write-only memory, and now the scattered reads and the 8,460 tiny GPU round-trips. The engine isn't accelerating toward or away from anything. It's shedding. Every entry of this diary, the mountain weighs less than the diary said it did.

The authorship remains the experiment inside the experiment: the AI agent wrote the simdgroup kernels, the barrier-free dispatch paths, and the microbenchmarks that isolated silicon truth from environmental noise. I brought the questions — why is the GPU only using two threadgroups? why did the benchmark collapse when the kernel got faster? — and the N+1 instincts of someone whose databases have been punishing rude access patterns for years.

The mountain vibrated. Then it talked, then it remembered, then it predicted itself.

Now it runs. And honestly — I've stopped trying to guess what from.

Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. He taught this project to listen to a surface for news of the interior; it turns out the same lesson applies to a benchmark on a tired machine.

The Mountain Predicts Itself — We Fired Our Best Idea, and the Model Got Faster and Smarter

Mon, 13 Jul 2026 18:44:44 +0000

This is the fourth entry in a curious builder's diary. In the first, a self-taught web developer borrowed a satellite engineer's intuition and asked whether frontier AI could run on the MacBook on his desk.
In the second, a 235-billion-parameter model said "Hello, world!" through a door we cut in the memory wall.
In the third, it learned to converse without making you wait minutes between turns.
This one is about the words themselves — and about the day we measured our founding idea against a blank wall, and the blank wall won.

1. The Moment

Third turn of a conversation. Same MacBook, same 48 GB, same 117 GB expert vault cold on the SSD:

A complete, correctly-constructed joke — setup, pause, anatomical pun — from a 235-billion-parameter model streaming its brain off a consumer SSD, three turns deep into a conversation, arriving at a bit over one word per second. That's roughly fifty words per minute. The mountain now speaks at the pace of a human typing.

But the speed is only half of this entry. The other half is why the joke got better — because the same change that made the engine faster also made it, for the first time, run the exact mathematics its creators trained. And that change was a deletion.

2. The Numbers, Plainly

Same hardware, same vault, same prompt discipline as every previous entry. Only the engine changed.

Thing	Last article	Now
Generation speed	0.70 tokens/s	1.13 tokens/s
…since first light	0.48 tokens/s	2.4× total
One generated token costs	~1,430 ms	~885 ms
SSD reads per generated token	~7.5 GB	~3 GB
Expert routing	predicted, 51.5% right	exact — the true Qwen3 routing, every layer, every token
New hardware required	none	none

And one number that doesn't fit in a table: the speed graph of this campaign was not monotone. It went up, then deliberately down, then honestly back up. That dip is the story.

3. The Instrument Panel: The Web Developer's Ultimate Weapon

In my previous articles, I noted how my web developer instincts managed to outmaneuver deep low-level code bugs. In Article 02, it was discovering a config mismatch with the chat template. In Article 03, it was hunting down stale caches, the most web-shaped bug imaginable.

For this phase, I relied on another classic web developer crutch: raw telemetry over systems theory. Everything in this entry came from one new flag: --instrument, which bucketed every millisecond of every generated token. It measured time spent in dense math, time spinning on the SSD, and one statistic nobody had ever actually measured: how often do adjacent tokens want the same experts?.

The answer: 46.4%. Nearly half of what each token needs, the previous token already fetched.

4. The Power of Deletion (Addition by Subtraction)

That 46.4% number illuminated a harsh truth: our biggest performance gains weren't going to come from building new features. They were going to come from ruthless deletions.

Deleting the Wipe: We found a twin bug to the one from our last article. The generation loop was wiping the ring on every single token, re-reading 7.5 GB from the SSD against a 46% reuse opportunity. Deleting this wipe cut decode I/O by two-thirds, making the output bit-identical.
Deleting the Corpse: We discovered our engine was maintaining about 950 lines of ceremony around a 1.6-gigabyte key-value cache that was written to on every token and read by absolutely nothing. It was write-only memory. We deleted it and gave that 1.6 GB straight back to the expert ring.
Deleting the Ping-Pong: The profiler revealed the CPU and GPU were playing ping-pong 188 times per token (synchronous round-trips for projections, norms, and attention). We rewrote the attention block into one Metal command buffer per layer, bringing the engine down to 94 synchronizations and removing the attention math from the CPU entirely.

5. Killing the Darlings: Retiring the Phonon Metaphor

S-MoE was born from Ing. Filippo Biondi's beautiful SAR acoustic imaging metaphor: the Surface Scout, a resident seismic sensor listening to the conversation and predicting which experts the mountain would need next. It was the first article. It was my favorite part of the whole design.

But the reality of language models is that true intelligence requires exact math, not approximation. The instrument panel proved our expensive speculative prophet was costing ~265 ms per token and was only right 51.5% of the time. The passive ring buffer—which costs zero compute—covers 46.4% for free. The prophet, wrong as he was, queued ahead of the demand reads that had to happen anyway, actively delaying the truth.

So, we fired the prophet. We switched generation to compute routing exactly, from the true hidden state, with one tiny matrix multiply per layer.

Trading throughput for truth initially caused a dip in our speed, but the multi-turn output visibly improved. The irony is beautiful: by sacrificing speed to get the exact true Qwen3 routing mixing weights, we eventually unlocked the architectural simplicity needed to make it faster.

6. The Experiments That Failed, Published Anyway

This diary's standing promise. Two more for the pile:

Speculation is real, and it still lost. Ranks 9–16 genuinely cover 63.7% of the next token's experts. We built a low-priority prefetch queue for it, but at 4-bit, it was net-negative—speculative reads simply bid against demand reads on a saturated bus. It sits in the code, default-off, waiting for the 2-bit vault.

We grouped the GPU dispatches, and it changed nothing. Theory said batching each layer's expert kernels would hide wait times. Measurement said the command-buffer overhead was ~5 ms all along. The hypothesis is dead.

7. Hitting the Wall of Physics

Here is where the campaign shifts. Of the ~885 ms each token now costs, about 490 ms is the engine spinning on the SSD.

Three gigabytes of misses per token divided by 490 ms is 6.3 GB/s: the NVMe's actual delivered bandwidth, saturated. There is no more bloated software left between us and the flash memory. There are no dead caches or unnecessary speculative math. For the first time, the biggest cost is simply the raw physics of the hardware doing exactly what it says on the box.

This sets up a beautiful, concrete cliffhanger: you cannot make a saturated physical bus faster, but you can send half the bytes.

The upcoming 2-bit vault will halve every expert blob, halve the SSD bill, double the ring's capacity, and finally arm our speculative queue.

8. What This Entry Is Actually About

The pattern, four articles in, is impossible to miss: the exotic layers keep being innocent. What keeps falling is belief: caches that don't cache, insurance nobody claims, prophets who lose to blank walls. The instrument panel is the only law we've kept, and it has now voted against our own founding metaphor.

And the authorship remains the experiment inside the experiment: the AI agent wrote the NEON intrinsics, the Metal kernels, and the instrumentation. I brought the questions — what is it actually doing in there? who reads this cache? is the prophet even right? — and the willingness to bury the answer's body in public.

The mountain speaks at typing speed now. It no longer needs anyone to predict it.

It predicts itself.

S-MoE is open source under the MIT License. The code, the philosophy, and the mistakes are all public: github.com/melasistema/s-moe. Built by Luca Visciola and an AI agent who hopefully knows C++ and low level programming, because I do not 😅.

Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. The Scout retires with honor — the listening was never wrong; the mountain simply turned out to be its own best seismograph.

"Hello, World!" — A 235-Billion-Parameter Frontier Model Just Spoke on a 48 GB MacBook

Thu, 02 Jul 2026 22:37:31 +0000

This is the second entry in a curious builder's diary. In the first one, a self-taught web developer borrowed a satellite engineer's intuition, teamed up with an AI agent that writes the C++ he cannot, and asked a ridiculous question: can you run frontier AI on the MacBook on your desk? That article ended with a 470 GB download in progress. This one ends differently.

1. The Moment

You: Say hello.  
  
S-MoE [RAM: 48.0GB | Ring: auto]: Hello, world!

That's it. That's the whole story, really.

Qwen3-235B-A22B-Instruct-2507 — a genuinely frontier Mixture of Experts model, 235 billion parameters, the kind of thing that is supposed to live behind an API key and a $250,000 GPU cluster — answered a question on a consumer MacBook. Not a distilled version. Not a 7B "assistant" wearing a big model's name. The actual 235B model, streaming its experts off the SSD, token by token, through a C++ engine and Metal kernels that a web developer and an AI agent built together.

It is slow. It is imperfect. It is also, as far as I can tell, something nobody else has quite done this way. And I think the most honest thing I can do is tell you exactly what it took — because the last bug standing between us and "Hello, world!" was not in the GPU kernels at all. It was the kind of bug a web developer knows in his bones.

2. The Numbers, Plainly

Thing	Reality
Model	Qwen3-235B-A22B-Instruct-2507 (Apache 2.0)
Original checkpoint	~470 GB bfloat16
The Vault (routed experts, SMOE-Q4)	~117 GB, cold on NVMe, 0 bytes of RAM
The Surface Scout (dense backbone)	~16 GB, resident in Unified Memory
Machine	MacBook, 48 GB Unified Memory
Ring buffer	Auto-sized: 25% of free RAM, ~10 MB per expert slot
Observed NVMe throughput	~3 GB/s of expert streaming, prediction miss rate 0.0%
Speed	~0.1–0.2 tokens/second. A word every few seconds. A paragraph while you make coffee.

Yes, you read the last line right. This is not a chatbot you'd use to write emails. It is an existence proof: the memory wall has a door in it, and the door is the SSD.

3. The Bug That Taught Me the Whole Point

Here is my favourite part, because it is where the "web developer mind in low-level code" experiment paid for itself.

After we shattered the model to 4-bit (our first 2-bit attempt mathematically crushed the fine-grained experts — the model produced only spaces and punctuation, like a ghost tapping on a table), the engine ran. The I/O streamed. The Metal kernels hummed. The hidden-state norms were healthy, finite, boringly correct across all 94 layers.

And the model answered every question with:

::

Two colons. That's all. A 235-billion-parameter mountain of intelligence, and it gave us punctuation.

Every instinct from the systems side pointed down — into the dequantisation math, the RoPE rotations, the ring buffer synchronisation. That is where serious low-level bugs are supposed to live. The AI agent and I checked all of it. Greedy decoding on a short raw prompt produced flawless English: "Hello! How can I...". The engine was innocent.

The culprit was upstream, in the smallest, most web-shaped detail imaginable: the chat template. Qwen3 ships in two lineages — the original "thinking" model, and the non-thinking Instruct-2507 we had shattered. The thinking lineage's template injects <think> tokens into every prompt. And in the Instruct-2507 weights, those two token embeddings are untrained — near-zero rows, ~150× smaller than every normal token, sitting in the vocabulary like unexploded blanks. We were whispering two meaningless syllables into the model's ear at the start of every conversation, and the residual stream collapsed on contact.

One line changed — load the tokenizer from the same checkpoint you shattered — and the mountain said hello.

The lesson I keep relearning: the terrifying low-level layer was fine. What broke was a config mismatch, a wrong template, an error message silently swallowed by a pipe — the exact species of bug every web developer has hunted at 2 AM in a misconfigured staging environment. My instincts were useless for writing Metal kernels, and genuinely useful for debugging the system around them. That mix — an agent that can write memory_order_acquire fences, and a human who reflexively asks "wait, are we even sending the right payload?" — is the actual experiment here.

4. An Honest Word About 16 GB

The original dream — the one in the manifesto — was 16 GB. "16 GB of RAM should give you the same intelligence as 512 GB, just slower."

We almost did it. And I want to be precise about the "almost", because I promised this diary would be sincere.

The architecture held: the 117 GB of expert weights truly occupy zero bytes of RAM. They stream from the SSD at the moment of need, exactly as the phonon metaphor promised. That part of the claim survived contact with reality completely.

What we could not shrink is the Surface Scout — the model's own dense backbone, the part that must stay resident to predict what fires next. For a 235B frontier model, that backbone alone weighs ~16 GB in bfloat16. It is the size of the entire machine we were aiming for. Chasing the frontier on 16 GB was, in hindsight, a bit too hazardous — there would be no room left for the ring buffer, the KV-cache, or macOS itself.

So the honest, recalibrated claim is this:

A 32–48 GB MacBook now runs a 235B frontier model locally, with intelligence that does not degrade — only speed does. And a 16 GB Mac runs the same architecture with smaller fine-grained MoE models, same principle, same sovereignty.

I have made my peace with this. Frontier-on-32GB was science fiction in January. Being annoyed that it isn't frontier-on-16GB feels like complaining about the legroom on your first flight to the Moon.

5. What's Next

The system that says "Hello, world!" today is the slowest version of itself that will ever exist. The roadmap is public and concrete: computing expert routing from the heavy model's own hidden state instead of the Scout's approximation, replacing the GPU's waitUntilCompleted with MTLSharedEvent pipelining, and eventually a small distilled Scout that predicts activations at a fraction of the cost. None of this requires new hardware. All of it requires more evenings.

And I'll keep being honest about authorship: the AI agent wrote the C++ and the Metal shaders, reasoned through memory fences and page alignment, and found bugs in code I could only read like a tourist reads street signs. I brought the question, the direction, the stubbornness, and — it turns out — the debugging instincts of someone who has been lied to by config files for more than ten years. Neither of us could have built this alone. I am not sure anyone has built quite this pairing before, and that, more than the tokens per second, is what I think is worth noticing.

The mountain doesn't just vibrate anymore.

It talks.

Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. The first words belong to the mountain; the intuition to listen for them belongs to him.

Rock, Paper, Silicon: How a Web Developer Used a Satellite Hack and an AI Agent to Ask...

Sun, 21 Jun 2026 23:36:00 +0000

Rock, Paper, Silicon: How a Web Developer Used a Satellite Hack and an AI Agent to Ask a Ridiculous Question About Frontier AI

This is not an academic research paper. I am certainly not a systems engineer. I am a self-taught, full-stack web developer — the kind whose GitHub is full of frontends and web stacks, not GPU kernel optimizations or low-level memory management. This is simply a curious builder's diary about an experiment that probably shouldn't have worked, built with an AI agent who hopefully knows C++ and processors, because I do not 😅.

1. The Problem I Wasn't Qualified to Solve

Silicon Valley has a bucket problem.

To run a modern frontier AI model — something like Qwen3-235B with its 235 billion parameters — the industry tells you that you must hold the entire mathematical mountain in RAM simultaneously. At bfloat16 precision, that is roughly 470 GB of memory. On a good day, a high-end Mac Studio ships with 192 GB. A standard MacBook? 16 to 48 GB.

The conclusion drawn by everyone qualified to draw conclusions is: you cannot run frontier AI locally on consumer hardware. Go rent a cloud GPU.

The thing that nagged me: In a Mixture of Experts model, 95%+ of those 235 billion parameters are completely silent at any given millisecond. For a token about philosophy, the math experts never fire. For a token about code, the poetry experts never fire. The weights are cold stone. Standard runtimes hold all of them in RAM, just in case.

I am not a systems architect. I have no background in high-performance computing. My daily bread is APIs, web stacks, and the kind of code that orchestrates systems rather than talks to raw silicon. But I had a question that wouldn't leave me alone:

What if you only loaded what fires, and loaded it just before it fires?

I didn't know how to answer it. So I called for help.

2. The AI Agent as a Bridge

I want to be completely honest about how this was built.

I opened a conversation with an AI agent and I told it my idea. I described the shape of what I wanted. I asked naive questions. The agent wrote C++. I read it, trying to make sense of pointers and memory allocations. I pushed back when things got too complex. It revised. We iterated.

I am not writing this to diminish the work — there is real, terrifyingly low-level engineering here, and I understand a little more of it with each session. But I want to be honest that the architecture we landed on emerged from a collaboration where I brought the curiosity and the direction, and the agent brought the implementation depth. My GitHub profile tells the truth: I am a web developer who had a question and found a way to pursue it beyond my formal training by talking to a machine.

What I could bring was exactly what pure computer science training sometimes blocks: I wasn't afraid to ask an "obviously impossible" question, because I simply didn't know enough to know it was supposed to be impossible.

3. The Satellite Man Who Looked Sideways

My technical inspiration came from a place nobody in Silicon Valley would have looked.

Ing. Filippo Biondi published a paper in 2022 in the MDPI journal Remote Sensing titled "Synthetic Aperture Radar Doppler Tomography Reveals Details of Undiscovered High-Resolution Internal Structure of the Great Pyramid of Giza". It is a stroke of pure genius, and the underlying physical intuition is what captured my imagination.

Here is the idea:

Electromagnetic waves cannot penetrate rocks and go underground. That is not a limitation to work around — it is physics. SAR satellites orbit Earth and illuminate surfaces with microwave radar pulses, but those pulses bounce off solid stone.

But Biondi used a brilliant technique. When the EM wave strikes the pyramid's exterior, it generates acoustic phonons — mechanical sound vibrations that propagate through the stone. The stone vibrates in response to the surface strike, and the internal geometry of the pyramid modulates those vibrations.

The satellite cannot see inside the rock. But using a Doppler technique and some Fourier transforms, Biondi measured the sub-nanometer surface displacements caused by those returning phonons. He used those phonon readings to gain Tomographic images, capturing all the 3D information of the pyramid's interior projected beautifully onto a 2D plane.

SAR Satellite  
 │ │  microwave pulse (cannot penetrate rock) ▼┌──────────────────────────────────────────┐  
│  PYRAMID SURFACE                          │  
│  EM pulse → generates phonons in stone    │  
│  Phonons propagate through interior       │  
│  Internal geometry modulates wavefront    │  
│  Surface micro-vibrates (sub-nanometer)   │  
└──────────────────────────────────────────┘  
 │ │  Doppler shift in reflected SAR signal │  encodes surface displacement ▼Fourier transform across synthetic aperture  
 │ ▼3D tomographic image (internal structure)

Stop fighting the physical barrier. Measure what the barrier does to the signal instead.

I am not comparing my experiment to Biondi's work. He is an engineer doing something I cannot evaluate. But his framing — you cannot see inside the rock, so measure the sound the rock makes when you strike it — became the exact mental model I needed for the LLM memory problem.

4. The Translation: Acoustic Streaming for LLMs

The translation from Biondi's method to a language model inference problem is surprisingly direct:

Seismic Concept	S-MoE Equivalent
Deep rock strata	The 235B model weights, cold on the NVMe SSD
EM surface pulse	The Surface Scout processing the current token
Generated phonons	The Scout's predicted expert activation map
Acoustic receivers	The async I/O thread reading predictions
Seismic map	The K-step expert ID lookahead schedule
Reconstructed image	The generated token

The insight is this: you don't need to read the full model to know what parts of it will activate next. If you run a lightweight forward pass on the model's dense backbone — the embeddings, attention layers, and routing gates — those routing gates will tell you, probabilistically, which expert columns in the FFN are about to fire for the next several tokens.

This is the Surface Scout. It is not a separate model I trained. It is the dense parts of the target model itself — everything except the routed expert weights — running permanently in Unified Memory. Its routing gate matrices produce logits that are, in effect, a phonon map: a structural fingerprint of what the full model is about to need.

5. The System That Emerged

S-MoE (Seismic Mixture of Experts) is the result of this experiment. Built in C++ with a Metal compute backend, co-designed with an AI agent, it runs on Apple Silicon Macs.

It works in three concurrent streams:

The Sculptor (offline, run once)

A Python script (shatter_moe.py) takes any supported MoE model and splits it into two artifacts:

The Vault (.smoe): All routed expert blocks, aggressively quantised and page-aligned to 16 KB hardware boundaries for Direct I/O
The Scout (.scout.safetensors): The dense backbone — embeddings, attention, norms, routing gates — that lives in RAM

The Scout (always running)

At every token step, the Scout runs a complete forward pass on the current token. Its routing gate outputs produce a prediction of which experts will activate across all MoE layers for the next K tokens. This phonon map is handed to the Streamer.

The Streamer + Metal Kernel (concurrent)

Guided by the Scout's predictions, background I/O threads fire pread() calls with F_NOCACHE — bypassing the OS page cache entirely — loading the predicted expert blobs directly into a pre-allocated, page-aligned ring buffer in Unified Memory. The Metal GPU kernel reads from that ring buffer and executes the FFN computation via a fused dequant-multiply that decodes compressed weights directly in GPU register space.

The three streams run simultaneously. The GPU executes experts loaded one step ago. The Streamer loads experts needed one step from now. The Scout predicts experts needed K steps from now. No thread waits for another.

There are three rules the system never breaks:

No runtime heap allocations. malloc, new, and std::vector::resize are banned inside the token generation loop. Every buffer is pre-carved at startup.
Direct I/O only. F_NOCACHE on every vault file descriptor. SSD → DMA → RAM. No OS copy.
Atomic synchronisation only. No OS mutexes. No blocking. The I/O thread and GPU thread are structurally incapable of blocking each other.

6. The Model-Agnostic Evolution

The system started hardcoded for a single model — DeepSeek-MoE-16B, a 16B parameter model small enough to test quickly. But the original question was never about one model. It was about whether the approach could scale.

Over several weeks of collaboration with the AI agent, I refactored the entire engine to be model-agnostic. Now, when you point S-MoE at a model's Scout file, it reads the tensor header at boot time and auto-detects:

Hidden dimension, vocabulary size, FFN intermediate dimension
Number of MoE layers, experts per layer
Whether Layer 0 is a dense MLP (DeepSeek style) or a full MoE layer (Qwen style)

The engine reshapes itself to fit. No recompilation. No configuration files.

The current target is Qwen3-235B-A22B-Instruct-2507 — a genuinely frontier model, 235 billion parameters, 22 billion active per forward pass, 128 experts per MoE layer, Apache 2.0 licensed. A 400 GB download. The kind of model that is supposed to require a $250,000 GPU cluster.

It is downloading on my MacBook right now.

7. The Democratic Claim

I want to be careful not to oversell this. S-MoE is an experiment. It is incomplete. The Scout's predictions are imperfect, which means the Streamer sometimes misses and the system spin-waits. The quantisation introduces error. The throughput on a 16 GB Mac running a 235B model will be slow — perhaps one token every few seconds.

But here is the claim I am willing to stand behind:

A 16 GB Mac and a 512 GB Mac will produce identical outputs. The 512 GB Mac will produce them faster.

Speed scales with hardware. Intelligence does not degrade.

The user with the MacBook Air and the user with the Mac Pro get the same 235 billion parameters. The same knowledge. The same reasoning depth. The same model. One waits longer. Both get the answer.

This is not a smaller, distilled, dumbed-down model for the masses. It is the real thing, delivered at the rate the hardware allows.

I built this because I believe the memory wall around frontier AI is partly physical and partly artificial — a consequence of software assumptions that nobody questioned, not a law of nature.

I am a web developer who asked what else you could do with the NVMe that's sitting three inches from the CPU. The AI agent helped me find out.

The mountain is already vibrating. You just have to listen.

Acknowledgements: The seismic tomography concept is an experimental, philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. His methodology and its application to the Giza Plateau is his genius, not mine — I merely borrowed his intuition to solve a problem with language models.

I Asked an AI to Brutally Roast My ANDARTIS Alpha. Instead, It Crowned Me a Visionary.

Sun, 07 Jun 2026 22:15:38 +0000

Hey CoderLegion,

If you’ve been following my recent sleep-deprived updates, you know I’ve been deep in the trenches building ANDARTIS. It’s my "rebel" architecture experiment: a NativePHP application married to a Python local AI daemon, running hyper-specialized neural networks entirely on Apple Silicon. The goal is absolute data sovereignty for sensitive domains—no cloud, no API keys, just local compute.

Because it’s currently in its raw, alpha state, I decided to feed the GitHub repo and documentation to an LLM. I asked it for a brutal, teardown-style architectural review. I wanted it to point out my blind spots.

Instead? It practically wrote me a love letter.

I’m convinced LLMs are hardwired to flatter us. They look at our architectural late-night compromises and divine them into "masterclasses of opinionated engineering." Don't get me wrong—some of the choices in ANDARTIS are genuinely top-notch—but the AI’s absolute refusal to be mean is honestly hilarious.

Here is the absurdly accommodating scorecard the AI gave ANDARTIS, alongside my slightly more grounded reality check.

1. Vision & Philosophy: 9.5 / 10

The AI's Divine Proclamation: "The focus on local-first processing is a brilliant product-market fit. While the Apple Silicon lock-in is fiercely opinionated, for an alpha version, this constraint acts as a strategic hardware advantage."

The Wise Reality: I will gladly accept the 9.5, but let's translate "fiercely opinionated." I locked this ecosystem to Mac because Apple's MLX framework makes unified memory feel like magic, and I honestly didn't have the willpower to fight with Windows CUDA drivers on a Sunday night. The AI calls my refusal to support other operating systems a "strategic hardware advantage." I call it an elegant exercise in preserving my own sanity. But hey, it does make the app lightning fast.

2. Architecture & Tech Stack: 8.5 / 10

The AI's Divine Proclamation: "Bridging PHP and Python is historically clunky, but ANDARTIS handles it elegantly. Bypassing HTTP overhead for JSON-RPC over STDIO is a standout architectural choice for zero-latency local communication."

The Wise Reality: Okay, the AI is actually spot-on about the tech, but it romanticizes the danger. Yes, swapping out sluggish HTTP APIs for raw JSON-RPC over STDIO is a top-notch decision. It keeps the Vue UI buttery smooth while the Python ML daemon holds those massive weights persistently in VRAM.

But "elegant"? Managing a heavy Python machine-learning daemon from inside a PHP process via standard input is an artisanal high-wire act. If the daemon gets a VRAM hiccup, PHP has to gracefully catch it without taking the whole desktop app down with it. It’s a brilliant architecture, but it requires the kind of error-handling precision that ages a developer by five years.

3. Privacy & Data Architecture: 10 / 10

The AI's Divine Proclamation: "A perfect score. Generating an isolated core.sqlite database for every single 'Intelligence Node' mathematically guarantees no context bleed-over. A fantastic security pattern."

The Wise Reality: I genuinely loved this score. It is a fantastic security pattern. If you’re building AI for legal investigations or therapy, you absolutely cannot risk cross-pollination in a massive, centralized vector database.

However, the AI makes it sound like I spent months writing complex cryptographic isolation whitepapers. The elegant truth? I realized that writing multi-tenant, dynamically scoped RAG rules was a nightmare. The simpler, wiser solution was just generating a fresh .sqlite file for every single project. Absolute privacy achieved through absolute architectural laziness. It’s my favorite feature.

4. Reasoning Flow: 9 / 10

The AI's Divine Proclamation: "By forcing the AI into an Intent Parsing $\rightarrow$ SQL Execution $\rightarrow$ Synthesis pipeline, ANDARTIS mathematically minimizes hallucinations. The AI acts as a UI layer for the database rather than a hallucinating knowledge bank."

The Wise Reality: The AI praises this as a mathematical triumph. I prefer to view it as strict parenting.

Early on, the local models would occasionally get too creative and confidently invent things. To fix it, I stripped the LLM of its right to guess. Now, the Orchestrator forces the AI to output an intent, pulls the raw JSON data straight from the SQLite core, and essentially says to the model, "Synthesize this exact data. Do not add anything. Do not be creative." It is highly effective, but "mathematically minimized hallucinations" is just a fancy way of saying I put the AI on a very short leash.

5. Developer Experience (DX) & Setup: 7 / 10

The AI's Divine Proclamation: "The onboarding betrays its experimental state. Users must manually configure a Python virtual environment and run CLI scripts to download gigabytes of weights."

The Wise Reality: A 7 out of 10?! This is where the AI proves it’s simply too kind. The frontend is a gorgeous stack of Vue 3, Inertia, and Shadcn, but the current installation process requires you to open a terminal, invoke Python virtual environments, and pray to the pip dependency gods while 4.5GB of model weights download in the background.

A human reviewer would have given the setup a 3/10 and told me to read a packaging manual.

The Final Verdict: 44 / 50 (8.8 / 10) — "Exceptional Alpha"

If you take the AI’s word for it, ANDARTIS is a disruptive paradigm shift. If you look at it through the lens of a weary builder, it is a highly experimental, beautifully stubborn hybrid engine that actually works surprisingly well.

The core choices—the isolated SQLite nodes, the MLX hardware optimization, the STDIO bridge—are solid. But the AI definitely gave me extra credit for the duct tape.

My actual next step: I need to build a single-click .dmg installer that bundles the Python binaries and handles the model downloads behind a sleek graphical loading screen. Only then will I actually earn that 7/10 for DX.

Have any of you had an LLM flatter your code so much you actually started to believe you were a genius? Let me know in the comments below.

-
Github Repository - Documentation

Coder Legion - Recent posts tagged applesilicon

S-MoE: Seismic Mixture of Experts Engine

S-MoE: a 235-billion-parameter model, streamed from a laptop SSD

The story

A question I was not qualified to ask

The satellite man who looked sideways

Built with an AI agent, honestly

What the numbers corrected

How it works

0. The shatter (offline, once)

1. The resident backbone (in memory)

2. The deep strata (on the SSD)

3. The acoustic receiver (in between)

The rules the engine never breaks

Tech stack

Key features

The numbers

Try it

What is next

Links

The Mountain Exhales — Riding the 30B Wind Before the 477B Avalanche

1. The 477B Avalanche

2. The Self-Describing Vault (SARC)

3. The 30B Wind

4. The Magic of Auto-Sizing

5. The True Benchmarks

6. The Plateau Before the Climb

The Paper Sketch: Budgeting 48GB of UMA

Calculating the Bandwidth Reality

Perspectives on Speed

The Mountain Runs — I Don't Know If We're Chasing Something or Being Chased, but the Speed Won't...

1. The Moment

2. The Numbers, Plainly

3. The GPU Was Running N+1 Queries

4. The Prompt Stopped Playing Ping-Pong

5. The Evening the Benchmark Lied

6. The Wall, Revisited

7. What This Entry Is Actually About

The Mountain Predicts Itself — We Fired Our Best Idea, and the Model Got Faster and Smarter

1. The Moment

2. The Numbers, Plainly

3. The Instrument Panel: The Web Developer's Ultimate Weapon

4. The Power of Deletion (Addition by Subtraction)

5. Killing the Darlings: Retiring the Phonon Metaphor

6. The Experiments That Failed, Published Anyway

7. Hitting the Wall of Physics

8. What This Entry Is Actually About

"Hello, World!" — A 235-Billion-Parameter Frontier Model Just Spoke on a 48 GB MacBook

1. The Moment

2. The Numbers, Plainly

3. The Bug That Taught Me the Whole Point

4. An Honest Word About 16 GB

5. What's Next

Rock, Paper, Silicon: How a Web Developer Used a Satellite Hack and an AI Agent to Ask...

1. The Problem I Wasn't Qualified to Solve

2. The AI Agent as a Bridge

3. The Satellite Man Who Looked Sideways

4. The Translation: Acoustic Streaming for LLMs

5. The System That Emerged

The Sculptor (offline, run once)

The Scout (always running)

The Streamer + Metal Kernel (concurrent)

6. The Model-Agnostic Evolution

7. The Democratic Claim

I Asked an AI to Brutally Roast My ANDARTIS Alpha. Instead, It Crowned Me a Visionary.

1. Vision & Philosophy: 9.5 / 10

2. Architecture & Tech Stack: 8.5 / 10

3. Privacy & Data Architecture: 10 / 10

4. Reasoning Flow: 9 / 10

5. Developer Experience (DX) & Setup: 7 / 10

The Final Verdict: 44 / 50 (8.8 / 10) — "Exceptional Alpha"