S-MoE Started to Speak OpenAI — Point Any Chat App at Your MacBook and Watch a 235B Model Think

Mon, 20 Jul 2026 09:00:00 +0000

There is a diary behind this — five entries of a self-taught web developer and an AI agent cutting a door in the memory wall so a 235-billion-parameter frontier model could run off the SSD of a 48 GB MacBook. You don't need any of it to read this one. This entry isn't about a bug we killed or a metric we moved. It's about the day the engine stopped being a private experiment I ran in a terminal and started speaking the one language every AI app on earth already understands.

1. The Moment

For five articles, talking to S-MoE meant talking to a terminal. A raw prompt loop. Something only its author could love. The mountain could speak, remember, predict itself, and run — but you had to be me, at my desk, in a shell, to hear it.

Then, on an ordinary evening, this happened:

.venv/bin/python serve.py --port 8000

And a 235B frontier model streaming its experts off a consumer NVMe became — as far as any other program can tell — OpenAI.

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")

resp = client.chat.completions.create(
    model="s-moe",
    messages=[{"role": "user", "content": "Who are you?"}],
)

That is not a screenshot of the real OpenAI. That's my laptop. The openai SDK cannot tell the difference, and — this is the whole point — neither can Open WebUI, LibreChat, your editor's AI plugin, or the thousand other apps built against that API. They ask for OpenAI. They get 235 billion parameters running three inches from the CPU, on hardware you already own.

2. The Numbers, Plainly

Same MacBook. Same 48 GB. Same 117 GB expert vault cold on the SSD. Nothing about the engine changed for this entry — this is a story about a doorway, not a speedup. But the doorway has numbers too.

Thing	Reality
Server	`serve.py` — one file, Python stdlib `http.server` + the tokenizer
New dependencies added to the engine	zero
API surface	OpenAI Chat Completions — streaming (SSE) & non-streaming, `/v1/models`, `/health`
Works with	Open WebUI, LibreChat, editor plugins, the `openai` SDK, plain `curl`
Built-in chat console	Yes — open `http://127.0.0.1:8000/` in a browser, no install
Decode speed	~1.43 tokens/s — a bit over one word per second
First token, cold 45-token prompt	~29 s, then it streams
Intelligence	The real Qwen3-235B. Undegraded. Only the speed is consumer-grade.

The zero in that table is the one I'm proudest of. The engine's founding vow was no dependencies — POSIX syscalls, Metal, and nothing else. The server keeps the vow. It is stdlib HTTP and the same tokenizer the console already loaded. An entire ecosystem of front-ends, unlocked, without adding a single line anyone else has to trust.

3. The Thinnest Possible Translator

I want to be honest about how little there is here, because the smallness is the design.

The server owns exactly three things: HTTP, tokenization, and the chat template. That's it. It applies Qwen3's chat template to your messages, turns them into token IDs, and hands those IDs to the persistent engine subprocess over the same private protocol the terminal console has always spoken. The engine never learns what HTTP is. It never sees a word of English. It sees numbers, streams experts, and sends numbers back.

Everything OpenAI clients expect is a thin mapping on top of that:

Streaming arrives as server-sent events, tokens appearing the instant they're generated.
temperature, top_p, top_k, repetition_penalty map straight onto the engine's per-request sampling — temperature: 0 selects the pure greedy path and is bit-identical every time.
finish_reason is honest: "stop" when the model chose to end, "length" when it hit your max_tokens mid-thought. Clients that auto-continue on "length" just work.
usage carries real token counts — prompt, completion, total — so anything measuring throughput divides real numbers by wall-clock, not estimates.

There's one deliberate constraint worth naming: the engine holds one conversation context and prefix-matches every request against it. So requests are serialized — one lock, one at a time — and there is no batching, because the hardware has no parallelism to batch into. The upside is that a normal chat client, which resends the growing message list every turn, hits the warm KV cache and only pays to prefill the new turn. Continued conversations stay fast. It is the honest shape of the machine, exposed honestly.

4. A Console That Ships You the Feelings

Third-party apps are the point, but I wanted something that needed zero setup to prove the plumbing — and, selfishly, something built for what I actually do all evening: watch this thing think and try to understand what it's doing.

So the server serves its own chat console. Open http://127.0.0.1:8000/ and it's just there — one self-contained HTML file, no build step, no npm install, served same-origin so there isn't even a CORS dance. It streams tokens as they land. But it's not really a chat UI. It's an instrument panel wearing a chat UI's clothes, and it exists because this whole project has only ever obeyed one law: measure, don't believe.

Every reply carries a live monitor. Before the first token, it doesn't sit in dead silence — it names the exact phase and size the moment they're known: "prefilling 128 tok prompt · 12.4s · cold start." On a machine where an answer takes tens of seconds, that one line is the entire difference between "it's thinking" and "it's hung." Then time-to-first-token, live token count, and decode tok/s, updating as the words stream, frozen on completion with the exact counts from usage.

Two meters sit in the header, because for an experimental researcher the feeling of fullness is half the story:

A context meter — 1,240 / 4,096 — tracking the conversation against the engine's fixed KV window, filling and turning amber as you talk, warning once at the wall where the window starts to slide and the model begins, gently, to forget the beginning.
A system-RAM meter — installed memory, current use, and the free headroom S-MoE's expert-streaming ring can grow into: 14.0 / 48.0 GB · 34.0 free. It's colored by macOS memory pressure, not raw percentage — because a 48 GB box legitimately runs hot under a 235B stream, and a naive bar would scream "full!" all day and cry wolf. Its tooltip breaks out wired, compressed, and the engine's own resident set.

None of this is product polish. It's the opposite. It's the dashboard of someone who has been lied to by too many systems and refuses to fly blind. It ships you the feelings — cold prefill, warm turn, headroom, the window about to slide — directly, as numbers, in real time.

5. Grab the Popcorn

Here's where I'd normally embed a chart. Instead, watch the video. It's three turns of real conversation, unedited, no narration, no speed-up. I could describe it, but this diary has a rule about shipping the raw thing instead of the story about the thing.

A word of preparation, because the experience is genuinely unusual and I don't want you to misread it.

This is not fast. I won't insult you by pretending it is. A word arrives roughly every second. But watch what streaming does to that second: it turns the wait into suspense. The tokens come one at a time, and because you can see them arrive, the pause reads as deliberation, not lag. You are literally watching a 235-billion-parameter mind assemble a sentence, left to right, on a laptop that also has your browser and your IDE open and your music playing.

And here is the thing I keep failing to convince people of until they see it: it is not slow. "Slow" is the wrong word, and I've been reaching for a better one for months. It's cinematic. It's a frontier model answering at reading speed, on a busy consumer MacBook, off an SSD, with the whole 235B of it undegraded behind every word. You don't tap your foot. You lean in. You grab the popcorn. The wait isn't a bug you tolerate — it's the show.

I have watched this thing generate paragraphs while I made coffee, and come back to a genuinely frontier-quality answer waiting for me, and felt something I can only describe as the opposite of API anxiety. No meter running. No tokens billed. No sentence leaving the room. Just my laptop, thinking, at its own pace, about my question, for me.

6. And Now, Every App You Already Use

The console proves the plumbing in ten seconds. The real unlock is that you never have to use it.

Because the API is genuinely OpenAI-shaped, the entire open-source chat ecosystem works unmodified. Point Open WebUI at http://127.0.0.1:8000/v1 with any non-empty API key, pick the s-moe model, and you have a polished, multi-conversation, markdown-rendering front-end talking to a 235B model on your desk. LibreChat, the same. Your editor's AI plugin, the same base URL, the same result — a frontier model doing code review inside your IDE, sourced entirely from your own SSD.

None of these apps were told about S-MoE. None of them were patched. They asked for OpenAI and my MacBook answered. That is the entire thesis of this project, finally wearing a suit: the memory wall around frontier AI was always part physics and part software convention — and the convention just fell to a single file of Python stdlib.

The speed profile sets honest expectations, and I'll state them plainly: this is a machine for thoughtful, asynchronous Q&A — a frontier-quality answer worth waiting a beat for — not rapid-fire agentic loops that fire a thousand calls a minute. Use streaming. Ask real questions. Let it think. It's the difference between a chatbot and a correspondent.

7. What This Entry Is Actually About

The five entries before this were about deletion — wiped caches, a fired prophet, write-only memory, scattered reads, tiny GPU round-trips. Subtraction, every time. This one is the first that's about addition, and it's the smallest addition in the whole diary: a translator that speaks a language everyone already knows, built out of nothing the engine didn't already have.

But it's the entry that makes all the others matter to someone other than me. A frontier model on a consumer laptop is a curiosity if the only person who can reach it is the person who built it. The moment it speaks OpenAI, it belongs to anyone with the same laptop and the same question — running in their editor, their chat app, their scripts, their evening. No cluster. No API key. No sentence leaving the room.

The authorship stays the experiment inside the experiment: the AI agent wrote the server, the streaming layer, the sampling protocol, and the instrument-panel console; I brought the questions — what does an app on the other side of this actually expect? where does the wait feel like a hang instead of a thought? can a researcher feel the fullness of the machine at a glance? — and a decade of web-developer instinct for the exact shape of an HTTP contract.

The mountain vibrated, then it talked, then it remembered, then it predicted itself, then it ran.

Now it speaks a language the whole world already knows — and it's waiting at localhost:8000 for you to say hello.

Grab the popcorn.

S-MoE is open source under the MIT License. The code, the philosophy, and the mistakes are all public: github.com/melasistema/s-moe.
Built by Luca Visciola and an AI agent who hopefully knows C++ and low level programming, because I do not 😅.

Acknowledgements: The seismic tomography concept remains a philosophical translation of the published work of Ing. Filippo Biondi on SAR-based subsurface imaging. He taught this project to listen to a surface for news of the interior — and an API, it turns out, is just another surface the whole world already knows how to read.

Coder Legion - Recent posts tagged localfrontierai