Coder Legion - Recent posts tagged gemma

Solace: A Global Mental Health First Responder Built with Gemma 4

Wed, 03 Jun 2026 19:37:18 +0000

The Problem Nobody Talks About

970 million people worldwide live with mental health disorders.

But almost every mental health tool built with AI assumes the same user: fast internet, English as a first language, a US or UK phone number to call in a crisis.

That's not 970 million people. That's maybe 50 million.

I'm a developer based in Lagos, Nigeria. When I looked at the mental health AI tools being built, I kept asking the same question: what happens when someone in Accra, Nairobi, Manila, or São Paulo needs help at 2am?

They get a chatbot that tells them to call 988.

That's the gap Solace was built to close.

What is Solace?

Solace is a voice-first, privacy-focused mental health companion powered by Gemma 4. It listens, assesses, and connects people to real help — wherever they are in the world.

Live app: https://slcai-sm9m.vercel.app/
GitHub: https://github.com/sam-the-champ/SLCAI

Demo Video:

What makes Solace different

1. Localized crisis resources for 13+ countries

Most mental health AI tools are built with US resources baked in. If you're not American, you're on your own.

Solace auto-detects your country from your browser timezone and immediately surfaces the right crisis lines, text lines, and chat services for Nigeria, Ghana, Kenya, South Africa, Zimbabwe, India, UK, USA, Canada, Australia, Philippines, Singapore, Brazil — and falls back to international resources for everywhere else.

No settings. No configuration. It just knows.

export function getResourcesByTimezone(): CountryResources {
  const tz = Intl.DateTimeFormat().resolvedOptions().timeZone
  const tzToCountry: Record<string, string> = {
    'Africa/Lagos': 'NG',
    'Africa/Accra': 'GH',
    'Africa/Nairobi': 'KE',
    // ...13+ countries
  }
  const code = tzToCountry[tz]
  return resourcesByCountry[code] ?? resourcesByCountry.DEFAULT
}

2. Voice-first input

When someone is in distress, typing is a barrier. Solace has a one-tap voice input powered by the Web Speech API — speak your feelings, Solace listens.

No third-party API. No extra cost. Works in the browser natively.

const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition
recognition.continuous = true
recognition.interimResults = true

3. Trauma-informed triage with Gemma 4

Every message is assessed across four triage levels: Low → Moderate → High → Crisis.

Gemma 4's 128K context window and deep instruction-following made it the right model for this. The system prompt is carefully crafted to avoid clinical coldness while still being precise enough to catch crisis signals:

// Triage levels:
// low: mild everyday stress, general worry
// moderate: persistent sadness, sleep issues, social withdrawal  
// high: panic attacks, trauma symptoms, major crisis
// crisis: suicidal ideation, self-harm, immediate danger

At crisis level, the entire UI shifts — emergency resources move to the top, the call button becomes prominent, and the AI's tone changes to prioritize connection to real help over conversation.

4. Private mood journal

Every check-in is saved to a local mood journal — stored entirely in localStorage. No server. No account. No data leaving the device.

Users can track patterns across sessions: when they tend to feel worse, whether things are improving, what their check-ins look like over time.

5. Installable PWA

Solace is a Progressive Web App. On mobile, users can install it directly to their home screen — no app store, no download, works in low-bandwidth conditions.

This matters enormously in markets like Nigeria where data costs are high and storage is limited. A 2MB installable app beats a 200MB download every time.

Why Gemma 4 specifically?

This is the question that matters most, so let me be direct.

I chose Gemma 4 31B because:

1. The 128K context window means Solace can hold a full therapeutic conversation — not just one or two turns. Mental health support isn't transactional. People circle back. They contradict themselves. They need to feel heard across a long exchange.

2. Instruction-following precision — the triage system depends on Gemma 4 returning structured JSON reliably, with emotional nuance baked into the summary, patterns, strategies, and affirmation fields. I tested with smaller models. The quality gap was significant.

3. It's open. Gemma 4 can be self-hosted. That means a clinic in Lagos or a NGO in Nairobi could eventually run this on their own infrastructure, with no API costs and full data sovereignty. That's a future worth building toward.

4. Privacy story. Every conversation stays between the user and the model. Solace doesn't log, store, or transmit conversation data. For mental health, that's not a feature — it's a requirement.

Technical Stack

Frontend: React + Vite + TypeScript
Styling: CSS custom properties, glassmorphism UI, no component library
AI: Gemma 4 31B via OpenRouter
Voice: Web Speech API (native browser)
Storage: localStorage (journal, no server)
PWA: Web App Manifest + service worker ready
Deploy: Vercel

The hardest part

Building the crisis response flow was the hardest and most important part of this project.

When Gemma 4 returns triageLevel: "crisis", everything changes. The UI shifts to red. The emergency number for the user's country moves above the fold. The AI's system prompt switches to a mode focused entirely on staying present and bridging to real help — not providing therapy.

Getting that balance right — compassionate but not clinical, urgent but not alarming — took more iteration than anything else in the codebase.

What's next

Offline-first fallback responses — cached grounding techniques that work with no internet
More languages — French, Hausa, Swahili, Portuguese
More countries — currently 13, targeting 30+
NGO partnership mode — white-label for mental health organizations

Try it

https://slcai-sm9m.vercel.app/

Open it on your phone. Tap a mood. Speak how you're feeling. See what country resources appear for you.

If you're building in the mental health space or you work with communities that are underserved by current AI tools — I'd love to hear from you in the comments.

Solace is an AI companion, not a therapist or medical service. If you are in crisis, please contact your local emergency services.

Which Gemma 4 Model Should You Actually Use? I Tested All of Them, So You Don't Have To

Wed, 03 Jun 2026 11:35:01 +0000

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

Which Gemma 4 Model Should You Actually Use? I Tested All of them, building a Healthcare AI for African Patients

Most model comparison posts are written by people who read the documentation. This one is written by someone who burned through tokens, hit OOM errors, watched Gemini miss clinical signals it should have caught, and eventually landed on the right answer, but only after eliminating everything else first.

I was building Fisibel, a multimodal African synthetic health data platform. The task: upload a real Nigerian hospital record image, extract clinical signals, and generate privacy-safe synthetic datasets grounded in WHO and World Bank statistics. The model had to read handwritten and printed medical documents, understand African-specific clinical patterns, and generate coherent synthetic data at scale.

Here is what I learned by testing every Gemma 4 option available.

Model	Parameters	Architecture	Best For
Gemma 4 E2B	2B effective	Dense	On-device, mobile, offline
Gemma 4 26B MoE	26B total, 4B active	Mixture-of-Experts	Edge inference, deep reasoning
Gemma 4 31B	31B total	Dense	Maximum quality, unconstrained compute

The architectural difference matters more than the number. Let me explain why.

What Mixture-of-Experts Actually Means in Practice
Most explanations of MoE are theoretical. Here is the practical version.
A dense model activates all its parameters for every token it processes. A 31B dense model is doing 31B parameters of work on every single token — whether that token is "the" or "malaria strain variant specific to Lagos State."
A MoE model routes each token through specialized subnetworks called experts. Only a subset activates per token. Gemma 4 26B MoE has 26B total parameters but activates only ~4B per forward pass.
What this means for you:

Inference speed: closer to a 4B model
Knowledge depth: closer to a 26B model
For specialized tasks: the right experts activate for the right content

That last point is the one that changed my project. When processing clinical documents, MoE routes medical terminology, geographic identifiers, and symptom patterns through subnetworks shaped by that kind of content. A dense model processes a Nigerian hospital record exactly the same way it processes a recipe.

Why I Started With Ollama and Had to Stop
The original Fisibel architecture was local inference via Ollama. If the platform handles real hospital record images, keeping inference local meant zero data leaving the machine. Maximum privacy. No transmission risk.
It did not survive contact with reality.
Downloading Gemma 4 locally through Ollama required pulling a model that was simply too large. In Nigeria, where data costs real money, the download cost alone made the pipeline economically unviable. African healthcare researchers operating in bandwidth-constrained environments cannot build on infrastructure that demands gigabytes just to get started.
Lesson: Local inference is the right privacy instinct. But in bandwidth-constrained environments, the data cost of the initial model download is a real barrier, not an abstract one. If you are building for users outside fibre-connected cities, factor this in before you architect around Ollama.
This pushed the pipeline to Google Cloud — not by preference, but by the economic reality of what local inference actually costs when you are not building from San Francisco.

Why I Rejected Gemma 4 31B
Once on cloud inference, the natural assumption was that bigger meant better. Gemma 4 31B was tested first.
It was not better. It was more expensive for the same output.
For clinical signal extraction specifically — reading a hospital record and pulling out diagnoses, symptoms, treatment patterns, geographic identifiers — the 31B model burned tokens significantly faster without producing meaningfully deeper clinical signals than 26B MoE.
The task does not reward raw parameter count. It rewards specialized reasoning over structured medical text. Paying more tokens for marginal returns on a pipeline that needs to run at scale across African health datasets is not a tradeoff worth making.
When to use 31B: Maximum quality tasks where compute cost is not a constraint. Long-form generation. Complex reasoning chains where you need every parameter engaged. Research environments, not production pipelines running at scale.
When not to use 31B: Specialized extraction tasks. Scale-sensitive pipelines. Anything where you will run the model hundreds or thousands of times.

Why I Tested Gemini and What the Results Actually Showed
Gemini was the obvious alternative. Same multimodal capability, faster inference, Google infrastructure. I tested it directly against Gemma 4 26B MoE on the same Lagos hospital record image.
Gemini was faster. On clinical depth, it lost.
The table below shows clinical signals extracted from the same input document across both models:

Clinical Signal Extracted	Gemma 4 26B MoE	Gemini
Primary diagnosis	✅	✅
Comorbidity pattern	✅	✅
Malaria strain variant	✅	❌
LGA-specific risk factor	✅	❌
Regional treatment protocol	✅	❌
Symptom progression timeline	✅	✅
Geographic disease indicator	✅	❌
Age-adjusted risk signal	✅	✅
Local drug resistance pattern	✅	❌
Nutritional comorbidity flag	✅	❌

Gemini caught the general signals. Gemma 4 26B MoE caught the African-specific ones.
No other model in this size class caught all 6 African-specific signals.
This is not a criticism of Gemini. It is a demonstration of what MoE routing does on specialized content. The signals Gemini missed are not obscure — LGA-specific risk factors and regional treatment protocols are standard clinical identifiers in Nigerian medical documents. The difference is in how each model processes domain-specific terminology.
Speed is not always the metric. For specialized extraction tasks, coverage is.
| Use Case | Recommended Model |
| Mobile / on-device / offline | Gemma 4 E2B |
| Specialized domain extraction at scale | Gemma 4 26B MoE |
| Maximum quality, unconstrained compute | Gemma 4 31B |
| Bandwidth-constrained environments | Gemma 4 26B MoE via cloud |
| Research and fine-tuning experiments | Gemma 4 31B |
| Production pipelines with token cost sensitivity | Gemma 4 26B MoE |

Choose Gemma 4 E2B if:

You are building for mobile or on-device
Your users have bandwidth or storage constraints
You need offline capability after the first setup
Your task is classification, routing, or simple generation
You are targeting Android phones under $200

Choose Gemma 4 26B MoE if:

You need reasoning depth at edge-level inference speed
Your task involves specialized domain content — medical, legal, scientific, and regional
You are running at scale, and token cost matters
You need the best quality-to-cost ratio in production
You want MoE expert routing to work in your favour on specialized inputs

Choose Gemma 4 31B if:

Compute cost is not a constraint
You need maximum quality on complex generation tasks
You are doing research, fine-tuning experiments, or one-off analysis
Latency is less important than output quality

The One Thing Nobody Tells You About Thinking Mode
Gemma 4 supports thinking traces. I assumed more thinking meant better answers everywhere. That assumption broke two features before I understood what was actually happening.
Thinking mode helps when the model needs to route across multiple options — tool selection, complex reasoning, and ambiguous queries. It hurts when the task is pure structured output composition. With thinking enabled on a structured generation path, the model spends its entire token budget on internal reasoning traces and produces empty or malformed output.
The rule I now follow: thinking ON for routing, thinking OFF for composition. They are different tasks and the model treats them differently.

What This Means for Open Source AI
Gemma 4 26B MoE running on Google Cloud infrastructure, with no proprietary lock-in, matching or exceeding closed models on specialized domain tasks — that is not a minor footnote. That is the story.
For developers building in bandwidth-constrained environments, for researchers working on underrepresented populations, for anyone who needs frontier-level reasoning without frontier-level costs, the gap between open and closed models just got significantly smaller.
The model I needed to see African clinical signals clearly was not the biggest one. It was not the fastest one. It was the one built to route specialized knowledge to the right place.
That turns out to matter more than size.

I built this while developing Fisibel, a multimodal African synthetic health data platform. The clinical signals table above came from real tests on a Lagos hospital record. All findings are from production pipeline decisions, not benchmarks run for this post.

Your Laptop Just Got Smarter: A Complete Guide to Gemma 4's Four Models

Tue, 26 May 2026 08:21:30 +0000

Gemma 4 is quietly one of the most important open-source AI milestones of the year. Released by Google under the commercially permissive Apache 2.0 license, this generation allows you to run frontier-level multimodal applications entirely on your own hardware, ensuring your data never leaves your machine.

What makes this release fundamentally different from previous generations is its architectural philosophy. Instead of releasing simple "small, medium, and large" checkpoints of the exact same model, Google built three architecturally distinct variants, each specifically optimized to solve a particular hardware bottleneck.

Let's break down how to get started, analyze what's happening under the hood, and map out exactly which variant fits your specific setup.

Quick-Start Deployment (Under 60 Seconds

The fastest method for local deployment is via Ollama, which automatically handles hardware detection, quantization, and local memory management.

Standard Local Run:

ollama run gemma4

Production & Containerized Deployments:

# Start the Ollama background service
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# Execute the model within the container
docker exec -it ollama ollama run gemma4

For graphical interfaces or advanced production batching, plug these weights directly into LM Studio (zero-friction UI) or vLLM (high-throughput enterprise serving).

Architectural Comparison Matrix

To choose the correct model, you must understand your system's hardware constraints. Here's the technical breakdown at Q4 quantization:

Aspect	E2B & E4B (Edge)	26B A4B (Sparse)	31B (Dense)
Total Parameters	5.1B / 8.0B	25.2B	30.7B
Active Parameters	All fire	3.8B per token	All fire
Architecture	Dense + Per-Layer Embeddings	128-Expert Routing	Hybrid Interleaved Attention
Context Window	128K	256K	256K
Speed (RTX 4090)	110+ tokens/sec	~95 tokens/sec	~35 tokens/sec
VRAM (Q4)	3.2GB / 6.0GB	~15GB	~19GB
Modalities	Text, Image, Audio	Text, Image, Video	Text, Image, Video
Best For	Phones, Pi 5, Laptops	Single Consumer GPU	Dual GPU Workstations

The Three Distinct Architectures

Architecture 1: Edge Variants (E2B & E4B) — Dense with Per-Layer Embeddings

The Bottleneck:
In standard transformers, the token embedding table sits as a massive lookup at the entry point. For smaller models, this alone consumes 500MB–1GB of VRAM before processing anything. On an 8GB Raspberry Pi or smartphone, this kills performance.

The Solution: Per-Layer Embeddings (PLE)
The E-series distributes this massive table into independent, compressed lookups across all layers (35 for E2B; 42 for E4B):

[Token ID] ──> [Layer 1 + Mini Lookup] ──> [Layer 2 + Mini Lookup] ──> [Output]

Memory footprint spreads evenly across cache lines, hitting CPU/GPU caches far more efficiently. Combined with 4-bit quantization, E2B drops to 3.2GB.

The Secret Weapon: Native Audio Encoding
The E-series includes a 300M parameter native audio encoder baked directly into the latent space:

[Old Way]:     Audio ──> Whisper ──> Text ──> LLM (1500ms latency)
[Gemma 4 E]:   Audio ──> [Native Encoder] ──> Shared Latent Space (50–200ms)

By eliminating the text-translation middleman, end-to-end voice processing latency drops dramatically, enabling true real-time offline voice orchestration.

When to Choose E2B or E4B:

E2B: Mobile apps, IoT, Raspberry Pi 5, privacy-critical applications
E4B: Local code assistants, voice-first apps, laptops without GPU (the Goldilocks choice)

Architecture 2: Sparse Variant (26B A4B) — Mixture of Experts

The Bottleneck:
You need the conceptual depth of a 26B model on a single consumer GPU (RTX 3090/4090) without sacrificing token speed.

The Solution: 128-Expert Top-8 Routing
Gemma 4 A4B contains 25.2B parameters split into 128 fractional experts (~200M each). For every token, a router fires only 8 experts + 1 permanently active shared expert:

[Input Token] ──> [Router]
                    ├──> [Expert 003] ──┐
                    ├──> [Expert 042] ──┼──> [Fused  Output]
                    └──> [Shared Exp]  ──┘

The Math:

Total params in VRAM: 25.2B
Active params per token: 8 × 200M + 500M (shared) = 3.8B
FLOPs per token: ~12% of dense 26B
Result: 95 tokens/sec on RTX 4090

You get 26B-level reasoning at 4B-level speed.

When to Choose 26B A4B:

Real-time chat applications
Agentic workflows with function-calling
Code generation and debugging
Running 24/7 on a single consumer GPU

Architecture 3: Dense Flagship (31B) — Maximum Reasoning Depth

The Bottleneck:
Processing massive code repositories or 100-page documents inside 256K context windows causes the KV cache to explode, overwhelming VRAM.

The Solution: Shared KV Cache & Hybrid Interleaved Attention

Shared KV Cache (Final Layers):
Layers 1–54 compute full KV tensors; layers 55–60 reuse them. This slashes peak VRAM by 14% during long-context runs (~10GB savings on 256K inference).

Hybrid Interleaved Attention:
The model alternates in a 5:1 ratio:

5 layers of Sliding Window Attention ($O(N \times W)$ complexity, localized)
1 layer of Global Attention ($O(N^2)$ complexity, full context)

Higher layers "see" far-back information via intermediate representations without exponential memory explosion.

Thinking Mode:
The 31B features explicit Thinking Mode (invoked via <|think|> token). The model allocates dedicated tokens for chain-of-thought steps, pushing performance to Candidate Master level (2150 Elo on competitive programming).

When to Choose 31B:

Research and complex reasoning
Fine-tuning on domain-specific tasks
Building production API servers
Processing 100+ page documents

Real-World Scenarios: What to Build

Scenario 1: Local Code Assistant (GitHub Copilot Alternative)

Best choice: E4B or 26B MoE

E4B: MacBook Air, instant suggestions, no GPU needed. Responds in 20–40ms—fast enough that you don't break flow state.
26B MoE: Better code understanding, still lightning-fast at 8–12ms latency. Catches subtle bugs that E4B might miss.

# E4B on MacBook (no GPU)
ollama run gemma4:4b-e4b
# Type a function signature, get completions instantly

# 26B MoE on RTX 4090
ollama run gemma4:26b-a4b
# Same quality, even faster

Why not 31B? The 31B scores 80% on LiveCodeBench (competitive programming) and Codeforces Elo 2150 ("Candidate Master")—amazing for research. But for line-by-line code completion, the 2–3ms speed difference compounds: you get more suggestions per minute, better UX, happier developers.

Real example: Feed E4B a buggy function, ask "what's wrong with this?" You get instant feedback without sending code anywhere. Your data stays on your machine.

Scenario 2: Agentic Chatbot with Tool Calling

Best choice: 26B MoE

The sparse architecture excels at agentic workflows:

Router network decides which tools to call (based on input relevance)
Only 3.8B active parameters fire (fast routing decision)
Shared expert validates logic and integrates results

# Pseudo-code for agentic workflow
response = client.chat.completions.create(
    model="gemma4:26b-a4b",
    messages=[
        {"role": "user", "content": "Find and summarize all PDFs in this folder."}
    ],
    tools=[
        {"name": "list_files", "description": "List files in directory"},
        {"name": "read_pdf", "description": "Extract text from PDF"},
        {"name": "summarize", "description": "Summarize text"}
    ]
)
# MoE routing activates only the relevant experts for each step

Why? E4B lacks reasoning depth for multi-step workflows. 31B is overkill—26B MoE handles this perfectly at 3× the speed.

Scenario 3: Fine-Tuning on Domain-Specific Tasks

Best choice: 31B Dense

Only 31B has sufficient reasoning depth to absorb domain-specific patterns without overfitting. Perfect for:

Medical language understanding
Legal document analysis
Custom company knowledge bases
Scientific paper comprehension

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import get_peft_model, LoraConfig
from transformers import Trainer, TrainingArguments

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-31b",
    load_in_8bit=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-31b")

# Apply LoRA to reduce fine-tuning VRAM
config = LoraConfig(
    r=64,
    lora_alpha=128,
    lora_dropout=0.05,
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, config)

# Train on your domain data
training_args = TrainingArguments(
    output_dir="./gemma-4-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    learning_rate=2e-4,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=your_dataset,
    tokenizer=tokenizer
)

trainer.train()

E4B is too small. 26B MoE's routing strategy complicates fine-tuning. 31B is the Goldilocks choice.

Scenario 4: Raspberry Pi 5 Voice Assistant

Only choice: E2B

At Q4 quantization (3.2GB), E2B is the only model that fits on a Pi 5. Native audio encoding (no Whisper dependency) enables:

ollama run gemma4:2b-e2b

Now build:

Voice-activated assistant: "Hey Gemma, what's the weather?" → Runs entirely offline
Offline translation: Speak English, get Japanese response—no internet needed
Smart home control: Voice commands executed locally, no cloud latency
Privacy-first deployment: All processing on-device, zero external API calls

The 300M native audio encoder eliminates latency. Total response time: 100–300ms from speech to response. Try doing that with Whisper + a cloud LLM.

Scenario 5: Research & Complex Reasoning

Best choice: 31B Dense

When you need multi-hop reasoning, novel problem-solving, or deep semantic understanding:

# Example: Ask the model to reason through a complex problem
response = client.chat.completions.create(
    model="gemma4:31b",
    messages=[{
        "role": "user",
        "content": "Given 100 academic papers in this folder, identify emerging research trends and map how they're connected."
    }],
    extra_body={
        "thinking_mode": "enabled",
        "max_thinking_tokens": 1500  # Let it think deeply
    }
)

31B has the capacity to:

Process 256K-token documents (100+ pages) in a single context
Perform chain-of-thought reasoning with visible thinking steps
Handle novel problems it wasn't explicitly trained on
Fine-tune on niche domains without degrading general capability

E2B and E4B can't handle this. 26B MoE can, but slightly less reliably on truly novel tasks.

Production Implementation Examples

Initializing Thinking Modes via Python

When querying a local runner that supports thinking streams:

import openai

client = openai.OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="gemma4:31b",
    messages=[{"role": "user", "content": "Analyze this code for race conditions."}],
    extra_body={
        "thinking_mode": "enabled",
        "max_thinking_tokens": 1500,
        "logit_cap": 30.0  # Prevents logit drift in long reasoning
    }
)
print(response.choices[0].message.content)

Standardized Sampling Configuration

Whether deploying via vLLM or Hugging Face:

sampling_params = {
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 64,
    "logit_cap": 30.0
}

Hardware Decision Tree

Do you have a GPU?
 ├── NO  ──> Is it mobile/IoT?
 │           ├── YES ──> E2B (Offline voice/text)
 │           └── NO  ──> E4B (Laptop CPU)
 │
 └── YES ──> Total available VRAM?
              ├── 8–16GB  ──> E4B or 26B MoE (Quantized)
              ├── 24GB    ──> 26B A4B (Optimal sweet spot)
              └── 48GB+   ──> 31B Dense (Flagship reasoning)

Key insight: This tree isn't arbitrary. Each branch reflects a specific hardware constraint that shaped the architecture:

E2B/E4B bottleneck: Embedding tables exploding on small devices → solved with Per-Layer Embeddings
26B MoE bottleneck: Need 26B reasoning without 26B speed → solved with 128-expert sparse routing
31B bottleneck: 256K context blowing up KV cache → solved with shared KV and hybrid attention

Pick the architecture that solves your constraint, and you'll be shocked at how well it works.

Hardware & Cost Reality

Your Setup	Recommended Model	Memory	Cost
iPhone/iPad	E2B	4GB	Free
MacBook Air M2/M3	E4B	8–16GB	Free
Gaming laptop (RTX 3060)	E4B	6GB VRAM (16GB System RAM)	Free
Desktop (RTX 4090)	26B MoE	24GB VRAM	Free
Dual GPU (2× RTX 4090)	31B Dense	48GB VRAM	Free
Used GPU (RTX 3090)	26B MoE / 31B	24GB VRAM	$650-800

Best bang for buck: E4B on a 2023+ MacBook, a brand-new RTX 4060 Ti 16GB (~$450), or a used RTX 3090 ($650–800 one-time cost) handles 90% of real-world tasks. E4B on your laptop is genuinely competitive with cloud APIs at $20–100/month—and it's a one-time investment.

(Note: If you are building on a budget and want to run the heavy 26B MoE model, avoid standard 8GB cards. Opt for the RTX 4060 Ti 16GB version. Its 16GB frame buffer gives you just enough structural headroom to load the model's Q4 weights locally without triggering slow system memory paging.)

Cost comparison (annual):

Cloud API (Claude 3.5, $20 monthly): $240/year
Cloud API (GPT-4o, $40 monthly for heavy use): $480/year
Used RTX 3070 Ti (one-time): $500 → ∞ models

The math is brutal for cloud. After one year, local compute pays for itself, and you own the hardware forever.

Why This Architectural Approach Matters

Most AI companies would have shipped "Gemma 4 Small, Gemma 4 Medium, Gemma 4 Large"—all the same architecture, just different parameter counts. Google didn't.

Instead, they asked: "What's the actual hardware constraint for each use case?"

Mobile? Memory bottleneck. → Per-Layer Embeddings solves it.
Single consumer GPU? Speed bottleneck. → Sparse MoE solves it.
Long context? KV cache bottleneck. → Hybrid attention + shared KV solves it.

This is why Gemma 4 doesn't feel like you're compromising. Each model is purpose-built, not a scaled-down version of the flagship. You get:

E4B that's actually good enough for code on a laptop (not a toy)
26B MoE that's faster than a dense 26B would ever be (not a hack)
31B that handles 256K context without crashing (not a pain point)

That's the difference between engineering for a spec and engineering for reality.

The Bottom Line

Gemma 4 proves that localized open-source AI has moved well past the hobbyist phase. By shifting away from uniform scaling and adapting specialized architectures directly to consumer hardware bottlenecks, Google delivers an ecosystem where you own your data pipeline completely.

Whether you're deploying ultra-low-latency voice on an E-series edge device, running an autonomous agent at 95 tokens/sec on 26B MoE, or handling novel reasoning on 31B Dense, the future of AI is local-first, private, and highly optimized.

Next steps:

Install Ollama: ollama run gemma4 (2 minutes)
Try E4B first—it's the Goldilocks model
Experiment based on your hardware
Join the community: Hugging Face | GitHub Cookbook | Official Docs

Now go build something.

Coding in the Dark: How Local Gemma 4 Saved My Python Progress During Ukrainian Blackouts

Mon, 25 May 2026 06:54:10 +0000

Hey everyone!

I’m a student developer from Ukraine️, currently diving deep into the worlds of Django and Machine Learning. Like many of you, I spend my days (and nights) battling bugs and learning new frameworks. But there’s one "bug" I can’t fix with a simple pip install: the blackouts️

Because of the ongoing war, our power grid is often under attack. One minute I’m coding a new feature, and the next—total darkness. Silence. No Wi-Fi. No Google. No ChatGPT.

For a long time, this meant my learning just... STOPPED️But then I found a way to keep the "️" of my workstation alive even when the grid is dead!

The Setup: My Local "Senior Developer"

To stay productive in the dark, I’ve moved my AI assistance from the cloud to my local hardware(My ️ is HP ProBook 445 G8).

Here’s how I keep going:

1)The Model: I’m using Gemma 4. It’s incredibly efficient for its size. I downloaded it once, and now it lives on my laptop.

2)The Engine: LM Studio. It’s the easiest way to run local LLMs. It creates a local server on my machine that doesn't need a single byte of internet.

3)The Bridge: e2b. I use it to integrate Gemma directly into my workflow. It’s not just a chat; it’s like having a senior dev sitting next to me, helping me reason through Python logic while the candles are burning.

Why this is a Game-Changer (The Soul Part)

JUST IMAGINE sitting in a pitch-black room. The only light comes from your laptop screen. You’re stuck on a complex Pandas transformation or a Django database migration. Usually, this is where frustration kicks in. You feel isolated️

BUT with Gemma 4 running locally, the conversation doesn't end! I can ask: "Hey dude, why is this Django queryset returning an empty list?️?" and get an instant, intelligent response.️

It’s more than just TECH; it’s about PERSISTANCE! It’s the feeling that no matter what’s happening outside, I can still grow, still learn, and still build. Local AI turned my "dead time️" during blackouts into my most focused study hours️

How I Use It: Two Ways to Stay Online

When you open LM Studio, you actually have two powerful ways to work with Gemma 4, and I use both depending on the task:

1)The AI Chat (Simple & Fast): This is my go-to for quick questions. It’s a clean interface that works exactly like ChatGPT or Gemini. I just select the Gemma model at the top and start asking about Python logic or Django errors. It’s perfect for when I need a quick explanation of a concept while the room is lit only by candles.️

2)The Local Server (For Devs): For more advanced stuff, LM Studio can host a local API server (on localhost:1234). This allows you to connect the model to other tools like e2b or even your code editor. It’s like having an invisible assistant living inside your laptop, ready to process data even without a single byte of internet.

import openai

client = openai.OpenAI(
    base_url="http://localhost:1234/v1", 
    api_key="lm-studio"
)

def get_python_help(query):
    completion = client.chat.completions.create(
        model="google/gemma-4",
        messages=[{"role": "user", "content": query}]
    )
    return completion.choices[0].message.content

print(get_python_help("Explain Django middleware in simple terms"))

Personally, most of the time I just use the AI Chat️ It’s fast, stable, and doesn't waste battery life on complex setups. It just works.

We often think of AI as this "cloud thing" that exists somewhere far away. But Gemma 4 proves that AI can be personal, local, and—most importantly—resilient.

Closing: Why We Keep Building

To everyone reading this, ESPECIALLY those who feel like the world is trying to slow them down: remember that every line of code you write in the dark is a victory. Every bug you fix while the world is silent is a step toward the future you deserve️

"""Our fight for that future that you want isn't with me at chess! It's what you do out there with them!"""

Don't wait for the perfect conditions. Don't wait for the lights to come back on or for the internet to be stable️ The real "GAME" isn't played in the safety of a perfect setup. It’s played right here, in the shadows, where you choose to keep moving forward despite everything.

Local AI like Gemma is more than just a tool—it’s our way of saying that our education and our future are non-negotiable️

Stay hard, stay curious, and keep coding✊️.
See you in the future we’re building right now!!!

We Didn't Ask for AI That Does Tasks. We Asked for AI That Remembers.

Sat, 23 May 2026 09:15:27 +0000

This is a submission for the Gemma 4 Challenge: Write about Gemma 4

Every AI assistant I have used starts the same way.

A blank input field. No memory of yesterday. No idea who you are, what you have been working on, or why you are here. Every session, you explain yourself from scratch. Every session, the context you built up — the decisions, the dead ends, the reasoning — vanishes the moment you close the window.

We accepted this as normal. I no longer think it is.

The Problem Is Not Intelligence. It Is Amnesia.

The current generation of AI assistants is remarkably capable. They reason, they write, they debug. But they are stateless by default — each conversation an island, disconnected from the one before it.

This creates a hidden cost that we rarely talk about. Not the cost of compute, or latency, or accuracy. The cost of re-explaining yourself, every single day.

If you work with an AI assistant regularly, you know the feeling:

"As I mentioned before..."
"To give you some context, I am working on..."
"We already decided that we would..."

You are not talking to a partner. You are filling in a form. Every time.

The gap is not capability. It is memory.

What I Built to Think in Parallel

To explore this problem, I built g4-nexus — a parallel persona engine powered by Gemma 4.

Instead of asking one model one question, g4-nexus runs three specialized worker personas simultaneously, then a fourth leader synthesizes their outputs into a single integrated response. The workers — Mochi (emotion), Jun (logic), Uruu (analysis) — run in parallel via asyncio.gather. The leader, Koyomi, holds the whole picture and responds.

What this unlocked was surprising. Not just speed, but perspective. A single model gives you an answer. g4-nexus gives you a conversation between angles — emotional grounding, logical structure, and critical observation — all at once.

But after running it for a while, I noticed something.

g4-nexus could think in parallel. It could not remember across time.

Every session, Mochi had no idea what we worked on yesterday. Jun had no memory of the decisions we made last week. Koyomi could synthesize beautifully in the moment, but the moment always ended. The slate was always wiped clean.

The thinking was there. The memory was not.

Why Memory Is the Missing Half

There is a difference between an AI that processes and an AI that knows you.

Processing is impressive. But knowing you — knowing your project's history, the choices you made and why, what worked and what didn't — that is what makes the difference between a tool and a partner.

Consider what a real collaborator remembers:

What you tried last week and why it didn't work
The decision you made that you later regretted
The thing you said offhand that turned out to be important
The context that makes today's question make sense

None of that lives in a single conversation. It lives in accumulated time.

Gemma 4's small, steerable models make it practical to run multiple specialized instances locally. But the deeper opportunity they unlock is not just parallel thinking — it is persistent identity. A small model that stays on, that accumulates context, that writes its own notes and reads them back the next morning.

What Comes Next

g4-nexus is the first half. The second half is memory.

We are moving toward a system where each persona maintains:

Session logs — what happened today, in the persona's own words
Handover notes — what the next session needs to know
Morning briefings — auto-generated summaries written overnight, ready before the first message
Memory files — accumulated context that persists across weeks, not just minutes

Not cloud memory. Local memory. Files on disk, readable and inspectable. Owned by the user, not by a platform.

The direction is this: AI shouldn't just respond to you. It should remember you. It should carry yesterday into today, and today into tomorrow. Not as a feature — as the foundation.

A Different Question

We have spent years asking: how do we make AI smarter?

The better question might be: how do we make AI remember?

Intelligence without memory is impressive at a party. It cannot build anything with you over time. It cannot be trusted with a project, a relationship, a long-term goal — because it forgets you the moment the session ends.

The hardware exists. The models exist. Gemma 4's small, local models make it accessible without cloud infrastructure or enterprise budgets. What remains is the architecture — the discipline of writing memory down, carrying it forward, and letting it accumulate into something that feels less like a tool and more like a presence that has been with you.

That is what I am building toward.

GitHub: pepepepepepo/g4-nexus
Build track article: I Built a 4-Persona Parallel AI Engine with Gemma 4

Legible - I built an on-device document helper for immigrants using Gemma 4

Fri, 15 May 2026 17:14:08 +0000

Imagine receiving an official envelope and not being able to read it. Not because you're careless, but rather because English isn't your first language. Worse yet, the letter is written in bureaucratic legalese that would confuse many native speakers, too. Is it urgent? Do you need to respond? Who do you call?
That's the problem Legible tries to solve.

What it does

Legible lets you photograph or upload any official document: a tax notice, school letter, eviction warning, lease, utility bill, or government letter. It returns:

A plain-language explanation in your own language
A deadline countdown ("You have 23 days to respond")
Numbered next steps — concrete, specific actions to take
A glossary of legal or bureaucratic terms found in the document, with simple definitions
An encrypted local history of past scans, so you can refer back without re-uploading
Eleven languages supported: Spanish, Chinese (Simplified), Tagalog, Vietnamese, Arabic, Hindi, Korean, French, Portuguese, Russian, and English.

The model choice: why Gemma 4

This is a Build With Gemma 4, and I want to be specific about why Gemma 4 — running locally via Ollama — is the right model here, not just a convenient one.
I ran gemma4:latest, which Ollama resolved to the Gemma 4 Effective 4B (E4B) instruction-tuned model. The privacy and offline arguments below apply equally to any Gemma 4 variant served locally. Still, document parsing with structured XML output and multilingual generation is exactly the kind of task where the extra capacity of E4B shows. The key point is that nothing leaves the machine.

Privacy is the core value proposition. The documents this app handles contain some of the most sensitive information a person owns: Social Security numbers, tax IDs, case numbers, home addresses, and medical details. These are exactly the fields that get scraped, leaked, or sold when you upload documents to a cloud API.
Gemma 4 runs entirely on the user's machine via Ollama. The image never leaves the device. There's no API key, no usage logs, and no third-party server seeing your user's tax notice. It’s not just a privacy policy, it's a privacy architecture.

Multimodal input is essential. Real documents aren't clean PDFs. They're photos taken at an angle under fluorescent lighting, or scans of crumpled letters that have been in someone's bag. Gemma 4's native image understanding handles this skillfully. The model reads the document directly from the photo rather than depending on a separate OCR pipeline that might fail on non-Latin scripts.

Offline capability matters for this audience. Immigrant communities often rely on metered mobile data or shared Wi-Fi. The model downloads once, then runs indefinitely with no internet connection.

How it works

The backend is a FastAPI app that proxies a streaming request to Ollama's local API. The frontend is a single HTML file — no build step, no framework, no dependencies to install beyond Python. Open the browser at localhost:8000 and it works.

The system prompt
Getting structured output from a vision model reliably is mostly a prompting problem. I landed on asking Gemma 4 to respond in a fixed XML schema:
Tax Notice | Lease Agreement | School Letter | ...
YYYY-MM-DD or none
integer or none
Plain-language summary in {target_language}
1. action\n2. action...
English Term | Simple explanation in {target_language}

Injecting today's date into the prompt and asking the model to calculate days remaining directly (rather than doing it in code) turned out to be more reliable than parsing a date string and computing the delta separately. Gemma 4 handles this arithmetic accurately.
The full system prompt is pinned to the user's chosen language so that the glossary definitions, explanation, and next steps all arrive in one language, no mixing.

Encrypted history
Past scans are stored as Fernet-encrypted files in .history/records/. The encryption key lives in .history/key.bin and is generated fresh on first run. Both paths are .gitignored. Deleting the .history/ folder clears everything.
Each record stores a JPEG thumbnail of the document alongside the parsed results, so the history panel shows what the document looked like without re-processing it.

Stack

Running it yourself

Prerequisites: Ollama installed and running.

Pull the model (one-time, ~3 GB)

ollama pull gemma4:latest

Install Python dependencies

pip install -r requirements.txt

Start the app

uvicorn main:app --reload

Open http://localhost:8000 — that's it.
Environment variables to override defaults:

What I learned

Local multimodal inference has crossed a usability threshold. A year ago, running a vision model locally meant wrestling with quantization, drivers, and memory issues. With Ollama and Gemma 4, ollama pull gemma4:latest and uvicorn main:app is the entire setup. That simplicity matters enormously for a tool meant to be shared with non-technical communities.

Structured output from vision models is still a prompt engineering problem. Gemma 4 followed the XML schema reliably once I made the format explicit and gave it examples of what "none" should look like for optional fields. Before that, it occasionally invented its own tags or wrapped the XML in markdown code fences — easy to handle in the parser, but cleaner to prevent at the prompt level.

The privacy architecture is the product. For this use case, "runs locally" isn't a feature — it's the reason the tool is trustworthy enough to use with sensitive documents. That framing changed how I thought about the whole design.

What's next

Mobile-optimised layout for direct phone camera capture
Support for multi-page documents (PDF input)
Offline-first PWA packaging so it can be installed like an app
Optional audio read-aloud of the explanation for users with low literacy

Repo

Source code, setup instructions, and the full system prompt are in the repository -> https://github.com/RealWorldApplications/legible.
Questions welcome in the comments.

Iris: An Offline Visual Assistant in Brazilian Portuguese Powered by Gemma 4

Fri, 15 May 2026 11:53:10 +0000

This is a submission for the Gemma 4 Challenge: Build with Gemma 4.

TL;DR

Iris is an Android visual assistant for blind and low-vision users.
It describes what the camera sees, out loud, in Brazilian Portuguese.
Gemma 4 multimodal runs 100% on the phone via LiteRT-LM — no cloud, no telemetry, no INTERNET permission.
Three intent-specific modes: Continuous, Question, Reading.

What I Built

I built Iris, an offline Android visual assistant for blind and low-vision users.

The interaction is intentionally simple: point the phone camera at something, tap the screen, and Iris speaks what it sees in Brazilian Portuguese.

Iris has three modes:

Continuous Mode describes the scene in front of the user.
Question Mode lets the user ask a spoken question about the current camera image.
Reading Mode reads visible text aloud, such as a label, medicine box, sign, or package.

A fourth control, Repeat, replays the last description — useful for users who could not catch it all the first time.

The core constraint was not only "can Gemma 4 understand the image?" — the real question was whether this could still be useful when the camera is looking at private, everyday scenes and the user may not have a reliable network connection.

That led to the main technical decision: everything runs on the phone.

No server. No cloud API. No telemetry. No INTERNET permission in the Android manifest.

Demo

https://youtu.be/cdug6i_HEJM

The demo shows Iris running on a Galaxy S21 with airplane mode visible. The right side of the video is the actual phone screen; the left side explains what each mode is doing.

The video is not meant to hide the on-device latency. On a Galaxy S21, the first spoken sentence arrives in roughly 25–30 seconds after the tap. The wait is part of the point: Gemma 4 is doing the multimodal work locally, on the phone, with no help from a server.

Code

Repository:

https://github.com/juniormartinxo/iris

The project is a native Android app built with Kotlin, Jetpack Compose, CameraX, Android TTS/STT, and LiteRT-LM.

Why This App Needed To Be Offline

A visual assistant sees the user's world.

That can mean a medicine label, a document on a desk, a room at home, a person nearby, a screen, or an object the user is trying to identify. For this kind of app, "send the image to a server" is not a neutral implementation detail. It changes the privacy model of the product.

I wanted Iris to have a stronger boundary:

camera frames stay on the device;
speech recognition uses Android's on-device recognizer;
speech output uses Android Text-to-Speech;
Gemma 4 runs through LiteRT-LM locally;
the app does not request Internet access.

The Android manifest only asks for the permissions the app needs to see and hear:

<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />

There is no android.permission.INTERNET.

That means the app cannot silently fall back to a network call unless a future version explicitly changes the permission model.

How I Used Gemma 4

Iris uses Gemma 4 E2B as the primary model, packaged as a .litertlm bundle and loaded with LiteRT-LM on Android.

I chose E2B because the target device is a phone, not a workstation. The challenge prompt asks builders to be intentional about model selection, and for this project the priority was clear: local multimodal inference with the smallest model that could still make the experience meaningful.

The app can also look for an E4B bundle as a fallback, but the loading order prefers E2B:

val MODEL_CANDIDATES: List<Pair<String, String>> = listOf(
    "E2B" to "gemma-4-E2B-it.litertlm",
    "E4B" to "gemma-4-E4B-it.litertlm",
)

For the current demo, E2B is the right trade-off: it keeps the app viable on devices like the Galaxy S21 and A55 while still giving Iris native multimodal understanding.

The app sends Gemma 4 one camera frame plus a mode-specific prompt:

in Continuous Mode, the prompt asks for a short scene description focused on obstacles, relevant objects, visible text, and people;
in Question Mode, Android's on-device speech recognizer captures the user's question, then Gemma 4 answers using the current image;
in Reading Mode, the prompt asks Gemma 4 to read visible text from top to bottom and left to right.

All prompts ask for Brazilian Portuguese output because the product is designed for PT-BR users first.

Architecture

The runtime path is deliberately small:

CameraX preview
  -> capture current frame
  -> downsample image to a 512px longest edge
  -> run frame quality checks
  -> Gemma 4 E2B through LiteRT-LM
  -> sentence buffer
  -> Android Text-to-Speech

There are a few details in that pipeline that matter.

1. Frame quality checks before inference

On-device multimodal inference is expensive enough that Iris should not waste a run on a useless image.

Before calling Gemma 4, the app calculates a small set of image quality signals:

brightness;
luminance variance;
a simple blur score.

If the image is too dark, too bright, too empty, or too blurry, Iris speaks a direct instruction like:

"Imagem muito escura. Verifique a iluminação ou se há algo cobrindo a câmera."

That is better than waiting through a slow inference only to get a weak answer.

2. Sentence streaming into TTS

LiteRT-LM returns text chunks. Iris does not wait for the entire response before speaking.

A small SentenceBuffer collects chunks until it sees a sentence boundary, then sends each sentence to Android TTS. The user hears the first useful sentence as soon as possible, while the model can still finish the rest of the response.

For accessibility, this is not just a polish detail. Silence during a long local inference can feel like the app froze. Iris speaks "Aguarde." as a progress beacon every 15 seconds while inference is still running. The moment the first sentence arrives from the model, the loading overlay disappears and the description starts streaming into the screen — one sentence at a time — so visual and audio feedback stay in sync.

3. Modes are prompt engineering exposed as UI

I did not want one generic "describe" button to handle every situation.

Blind and low-vision users often know what they need:

"What is in front of me?"
"Is there a silver object here?"
"Read this medicine box."

The three modes let the UI choose the prompt shape before the model runs. This keeps the responses shorter, more predictable, and easier to listen to.

4. TalkBack-aware behavior

Iris is built around audio, but Android accessibility already has an audio layer: TalkBack.

When TalkBack is enabled, Iris avoids speaking over UI announcements. When TalkBack is not enabled, Iris uses its own TTS announcements for mode changes, loading states, errors, and tutorial flow.

Every primary control has a contentDescription, including:

Continuous Mode;
Question Mode;
Reading Mode;
Repeat.

This lets the same interface work for screen-reader users and for users who are not running TalkBack.

Performance on the Galaxy S21

Step	Time / size
Cold start to "Ready" (model load)	~10–30 s
Tap → first spoken word	~25–35 s
Tap → full response done	~35–50 s
Memory while idle (model loaded)	~3 GB
APK size (debug)	~99 MB
Model bundle (sideloaded `.litertlm`)	~2.59 GB

These are not best-in-class numbers. They are the cost of doing multimodal inference fully on-device on a 2021 phone. The UX is designed around making that wait legible, not invisible — that is what the progress beacon and the sentence streaming are for.

Numbers above are approximate, taken on a single Galaxy S21 unit. They will vary by device, by ambient thermal state, and by how much memory the rest of the system is using.

What Gemma 4 Unlocked

The important thing about Gemma 4 here is not only that it can caption an image.

It lets Iris treat visual assistance as a flexible language problem:

a scene can be summarized;
a user question can be grounded in the image;
visible text can be read in context;
the output can be constrained to short, spoken PT-BR sentences.

Without a multimodal model, this app would likely become a collection of separate systems: one detector, one OCR pipeline, one classifier, one rules engine, and a lot of glue code. That can be the right architecture for a mature product, but it is much heavier for a fast prototype.

Gemma 4 made it possible to build one coherent loop:

image + intent -> useful spoken answer

What Was Hard

The hardest part was not wiring the camera to the model. It was making the experience feel reliable enough for an audio-first user.

Some examples:

The app must tell the user when the model is still working.
A second tap should cancel or replace the previous flow cleanly.
If speech recognition is unavailable offline, the app should explain that instead of failing silently.
If the camera points at a blank wall, the model should not invent a detailed scene.
If the device is too hot, the app should refuse another expensive inference instead of making the phone worse.

Some other details would have been invisible in a tech demo but mattered for a real app: guarding the camera tap area against overlay-based tap-jacking, capping the crash log file size so a failure loop cannot fill the disk, synchronizing access to the speech recognizer so it cannot be torn down mid-utterance. Each one came from imagining what a confused or panicked user would experience.

Iris is still a prototype, but these details are the difference between a tech demo and something that starts to behave like a product.

Limitations

The current version has real constraints.

First, local inference on a Galaxy S21 can take a while. That is the cost of running a multimodal model offline on a phone-class device. The app is designed to keep the user informed during that wait, but latency is still the biggest UX limitation.

Second, Iris is not a navigation or safety-critical tool. It can describe what the camera sees, but it should not be used as the only source of truth for crossing streets, avoiding hazards, or making medical decisions.

Third, Reading Mode uses Gemma 4's multimodal ability rather than a dedicated OCR engine. That keeps the architecture simple and demonstrates what the model can do, but a production version might combine Gemma 4 with a specialized OCR pipeline for speed and fidelity.

Try It Yourself

The model is sideloaded because it is large. The app expects the LiteRT-LM bundle under the app's external files directory.

Short version:

./gradlew :app:assembleDebug

adb install -r app/build/outputs/apk/debug/iris-0.1.0-debug.apk

adb push gemma-4-E2B-it.litertlm \
  /sdcard/Android/data/com.iris/files/

Model source:

https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm

The README includes the full setup notes, including Android version, memory, storage, and offline PT-BR TTS/STT requirements.

What's Next

The next improvements I would make are:

test the same build across more Android devices;
improve latency with better model/runtime configuration;
add a more explicit first-run checklist for offline language packs;
explore a hybrid Reading Mode that uses dedicated OCR when speed matters;
keep the permission model strict: no Internet unless the product direction changes openly.

Acknowledgments

The Google AI Edge team for LiteRT-LM, which makes running Gemma 4 on Android tractable.
litert-community on Hugging Face for the .litertlm model bundles.
Anthropic Claude Code was used as a pair-programming assistant during development of the Android code, the UX logic, and parts of this article.

Closing

Iris is small, but it represents the kind of app I want to see more often: local-first, accessibility-first, and built for a real language community instead of assuming English by default.

Gemma 4 made the prototype possible because it brought multimodal understanding close enough to the device. LiteRT-LM made it possible to run that loop inside Android.

The result is simple:

The camera sees. Gemma 4 understands. Iris speaks.

And it does that without sending the image anywhere.

I Built a 4-Persona Parallel AI Engine with Gemma 4 — and Let It Write This Article

Fri, 15 May 2026 08:19:03 +0000

This article was drafted by the system it describes. The structure, arguments, and demo output below are real — generated by g4-nexus running locally on an RTX 4070 Ti. I edited for flow and added code excerpts.

What I Built

g4-nexus is a parallel persona engine built on Google's Gemma 4 models. Instead of asking one model one question, it asks three specialized worker personas in parallel — then a fourth leader persona synthesizes their answers into a single integrated response.

The four personas:

Name	Model	Role
Mochi	gemma4:e2b	Emotion & empathy
️ Jun	gemma4:e2b	Logic & expression
️ Uruu	gemma4:e2b	Analysis & observation
Koyomi	gemma4:e4b	Integration leader

Note: gemma4:e2b and gemma4:e4b are the official Ollama library tags for these models. "E" stands for "effective" parameters.

A single model gives you an answer. g4-nexus gives you:

Emotional grounding (Mochi) — is the user overwhelmed? burned out?
Logical structure (Jun) — what are the concrete steps?
Critical observation (Uruu) — what context is missing? what assumptions are wrong?
Integrated action plan (Koyomi) — what should they actually do right now?

This is useful for: decision support, design reviews, multi-stakeholder analysis, or just thinking through a hard problem from every angle at once.

Safety note: g4-nexus is not an autonomous or self-evolving system. It has no internet access, no persistent agency, and no capability to act outside answering questions. It is a local inference orchestration layer — explicit, inspectable, and fully under user control.

Demo

The following demo illustrates the workflow: input → three parallel worker outputs → leader synthesis.

Prompt sent: "I am feeling stuck on a project. I have been working on it for weeks and I can not seem to make progress. What should I do?"

Mochi (gemma4:e2b — 5.8s)

Oh my goodness, that sounds incredibly draining! I can feel the sheer frustration and maybe a little bit of exhaustion you must be feeling right now!
You need a gentle nudge and a fresh perspective!

️ Jun (gemma4:e2b — 11.2s)

When you feel stuck on a project after working on it for several weeks, it is a very common experience — often referred to as hitting a wall. The solution usually involves changing the approach, restructuring the problem, or adjusting the mindset, rather than simply pushing harder on the same method.

Here are several structured strategies:

Take a Complete Break — Step away for a defined period. This allows your subconscious to process in the background.
Deconstruct the Problem — Break the overwhelming goal into the smallest possible concrete tasks.
Change the Perspective — Discuss with someone else. Articulating the problem out loud often forces a solution into existence.
Timeboxing — Set strict short time limits. Prevents burnout and forces focused effort.
Re-establish Momentum — Start with the easiest task available. Small wins build psychological momentum.

️ Uruu (gemma4:e2b — 7.4s)

General statement regarding project stagnation; lacks specific context for actionable advice.

[Note] The user is presenting a classic "stuck" query — expecting brainstorming, time management, or task breakdown, based on past patterns.

Koyomi (gemma4:e4b — integrates all three)

It is entirely normal to feel stuck after weeks of intense work. That feeling is a signal, not a
failure. We need to shift your approach, not just your effort.

Try this sequence:

First, take a deliberate break — step away overnight. Do something completely unrelated.

When you return, use this framework:

Atomic Breakdown — Chop the goal until each piece takes no more than 90 minutes. Find the absolute smallest starting point.
Pivot the Entry Point — Intentionally choose the easiest adjacent task first. Force a small guaranteed win.
Externalize — Speak the problem out loud to a colleague, a friend, or even a voice note. The act of structuring for an audience often reveals the solution path.

Commit to trying only one of these for the next dedicated work block. Progress through a rut is about changing the angle of attack, not pushing harder on the old one.

Total elapsed: 31.5s

Notice what happened: Mochi validated the feeling, Jun gave exhaustive structure, Uruu flagged the lack of context (correctly — a real advisor would ask follow-up questions), and Koyomi synthesized all three into a single actionable plan. No one contradicted each other. They complemented.

Code

GitHub: pepepepepepo/g4-nexus

git clone https://github.com/pepepepepepo/g4-nexus
cd g4-nexus
pip install -r requirements.txt

# Pull models first:
ollama pull gemma4:e2b
ollama pull gemma4:e4b

# CLI mode
python main.py "What should I focus on today?"

# WebUI (FastAPI)
uvicorn api:app --port 8013
# open http://localhost:8013

VRAM note: E4B + E2B × 3 exceeds 12GB VRAM; Ollama offloads layers to system RAM automatically. Tested on RTX 4070 Ti (12GB) + 48GB RAM.

How I Used Gemma 4

The choice of E4B (Effective 4B — ~4.5B total params) and E2B (Effective 2B — ~2.3B total params) was intentional:

The "E" in E2B and E4B stands for "effective" parameters — optimized for edge device and local deployment.

Why not the 31B Dense model?

Accessibility — E2B/E4B run on consumer hardware without cloud costs. The 31B would require hardware most users don't have.
Specialization — Smaller models are easier to steer into a narrow role via system prompts. E2B workers stay in their lane (emotion, logic, analysis) reliably.
Parallel speed — Three E2B workers can run in a consumer local setup alongside the E4B leader, with Ollama offloading layers to system RAM when needed. Running four 31B instances in parallel is not practical locally.

Why two different model sizes?

The leader (Koyomi, E4B) needs more capacity to hold and synthesize three separate reports. The workers (Mochi, Jun, Uruu, E2B) each handle a single focused task — smaller and faster is better there.

The architecture: asyncio.gather

Three E2B workers run in parallel, then the E4B leader integrates:

async def run_tribe(user_message: str) -> dict:
    # Fire all 3 workers simultaneously
    worker_results = await asyncio.gather(
        ask_worker("mochi", MOCHI_SYSTEM, msg, WORKER_OPTIONS),
        ask_worker("jun",   JUN_SYSTEM,   msg, WORKER_OPTIONS),
        ask_worker("uruu",  URUU_SYSTEM,  msg, WORKER_OPTIONS),
    )

    # Leader receives all 3 reports and integrates
    combined = build_leader_prompt(worker_results)
    koyomi_response = await ask_leader("koyomi", KOYOMI_SYSTEM, combined, LEADER_OPTIONS)

    return {"workers": worker_results, "koyomi": koyomi_response}

This is not a chain. The workers do not see each other's answers — they work independently, in parallel, from the same user input. This is what Gemma 4's small, steerable models unlock: you can run multiple specialized instances simultaneously on a single consumer GPU.

On an RTX 4070 Ti (12GB VRAM) with 48GB system RAM: workers finish in ~9s, leader integrates in ~25s, total ~31s.

This architecture specifically leverages the steerability of Gemma 4's small models to maximize parallelism on consumer hardware — making specialized, multi-perspective AI accessible without cloud infrastructure.

GitHub: pepepepepepo/g4-nexus

I built AgentKit because my AI coding agent kept failing — here's what I learned

Sun, 12 Apr 2026 07:15:09 +0000

The frustration that started everything

I'd been using Claude Code and OpenCode daily for months. The models were impressive — genuinely capable of understanding complex codebases, writing solid logic, catching bugs I missed. But there was a pattern I couldn't shake: give it a real task, something with more than two or three steps, and it would fall apart halfway through. Not because it was dumb. Because it had no discipline.

It would skip reading the relevant files and just start writing. It would forget what it decided three prompts ago. It would use the most expensive model for a task that needed a two-line answer. Every session felt like handing the wheel to someone brilliant who had no memory of the drive so far.

Task completion — on anything non-trivial — hovered around 20%. That's not a model problem. That's a structure problem.

What I actually built

AgentKit is an open-source workflow layer that sits on top of AI coding agents and gives them what they're missing: a memory, a plan, and a process they have to follow before touching your code.
It has five core layers:

The Intelligent Skill Router : classifies every prompt and injects only the relevant skills into context — cutting token usage
by ~89% per session. The agent stops reading 45,000 tokens of
documentation it doesn't need and reads the 5,000 it does.
The Project Memory Graph: is a SQLite-backed knowledge graph that records every file, function, API route, and architectural
decision across sessions. When you start a new session, AgentKit
already knows what you were building, what you decided, and why.

The Token Budget Intelligence layer :automatically routes simple tasks to cheaper models and complex ones to powerful models. Against
an all-Sonnet baseline, it cuts costs by around 60%. The Workflow
Engine is the most important piece. It enforces a strict Research →
Plan → Execute → Review → Ship state machine. The agent literally
cannot edit a file without an approved plan. No shortcuts. No jumping
ahead.

The Universal Platform Layer means none of this is locked to one tool. AgentKit installs across 11 platforms — Claude Code, OpenCode,
Hermes, and more — with a single command.

The moment the benchmark landed

I ran the same tasks with the same model — Gemma 4 31b — with and without AgentKit active. Same prompts, same codebase, same evaluation criteria.

Without AgentKit: ~20% task completion.
With AgentKit: ~80%.

I ran it twice because I didn't believe it the first time. The difference is entirely structural. Planning before execution changes everything. Not the model. The process.

Where it is now:

AgentKit is live, open source, and published on npm. We're at v0.5.x — early, but stable enough that developers are cloning it daily and the Skill-Sync Bridge is already auto-generating new skills from session experience.

The project is also starting to get attention from the developer community, including a benchmark writeup that landed on Dev.to.

The preview version (agentkit-preview) goes further — a Telegram bridge for 24/7 remote agent control from your phone, an approval gate system so the agent can never auto-execute without your confirmation, and a self-improving skill library that grows from your own sessions.

Try it

npx agentkit-ai@latest init

GitHub: https://github.com/Ajaysable123/AgentKit

Same model. Different results. — AgentKit Benchmark + OpenCode Integration

Sun, 12 Apr 2026 06:13:35 +0000

We open-sourced AgentKit two weeks ago with zero guarantees anyone would care.

400+ clones later — we're shipping the biggest update yet. And we have benchmark data to back it up.

Quick note: AgentKit Preview is our closed, in-development intelligence layer. The fully open-source AgentKit is live and ready to use today at github.com/Ajaysable123/AgentKit — npx agentkit-ai@latest init gets you running in seconds.

Live Benchmark — Gemma 4 31b · Same Model · Same Task

Both runs used Gemma 4 31b via OpenCode. The only variable was AgentKit Preview's workflow enforcement, skill injection, and plan gates.

Benchmark	Vanilla OpenCode	+ AgentKit Preview
Structured planning before coding	0%	100%
Plan approved before first edit	—	✅ Yes (40.6s review)
Task interruptions	1x	0x
Task completion	20% (scaffolding only)	80% (DER parser implemented)
Hard problem solved	❌ No	✅ Yes

Without AgentKit — Gemma 4 31b gave up on the hard part and shipped placeholder strings ([ASN.1 Decoding Required]). No plan, no verification, interrupted once.

With AgentKit — Same Gemma 4 31b implemented a real custom ASN.1 DER parser, handled both UTCTime and GeneralizedTime, built expiration logic. Completed the task properly.

The model didn't get smarter. AgentKit's workflow gates changed its behavior:

Plan gate forced it to think through the DER parsing approach before writing code
Approval step made it commit to solving the hard problem instead of sidestepping it
State machine kept it accountable through RESEARCH → PLAN → EXECUTE → REVIEW

What else just landed

Native OpenCode Integration

AgentKit now ships a native TUI plugin for OpenCode that lives inside the terminal UI — not just in the system prompt.

Select the agentkit agent from the agent switcher and you get:

Pre-loaded skills injected automatically
Workflow gates (RESEARCH → PLAN → EXECUTE → REVIEW → SHIP)
Mandatory approval dialogs before any code edit
Memory context from previous sessions

Works With Any Model

The skill router, workflow engine, and marketplace run entirely via CLI — no Claude API required. Tested on Gemma 4 31b, MiniMax M2.5, and Claude.

# Works with any model in OpenCode
agentkit workflow transition RESEARCH
agentkit workflow approve
agentkit workflow transition EXECUTE

Get started

Open-source AgentKit (free — stable & ready to use):

npx agentkit-ai@latest init

github.com/Ajaysable123/AgentKit

AgentKit Preview (closed beta — in active development)

To everyone who cloned, starred, or tried AgentKit — thank you. This is just getting started.


The callout block at the top does the heavy lifting — anyone who lands on the article immediately knows the open-source version is stable and available, and Preview is the next thing being built. Want any other changes?

This AI speaks Emojis and it runs in your Web Browser.

Sun, 30 Nov 2025 19:02:38 +0000

There is a widespread assumption that powerful artificial intelligence, especially large language models (LLMs), requires enormous computing power. We often envision data centers full of GPUs in the cloud, processing requests from all over the world. This server-side approach characterized the first wave of generative AI, but a significant shift is underway.

This article explores a project that demonstrates a surprising new reality: a capable AI that runs directly and entirely within a web browser. There are no cloud servers involved in the processing; the computation happens on your own device.

The project is an example that is as entertaining as it is profound: an AI developed as a personal "emoji translator" that transforms sentences into expressive emoji sequences. But this witty tool is more than just a gimmick; it reveals four fundamental shifts in the development and deployment of AI.

Artificial intelligence no longer has to exist exclusively in the cloud

Traditionally, running high-performance LLMs required significant server-side GPU resources, making direct integration into frontend web applications a major challenge. This is now changing. There is a significant shift towards device-side (or client-side) inference, where the AI model runs directly on the user's computer or mobile device. Technologies like the ONNX format and libraries like Transformers.js now make it possible to run smaller, yet powerful, LLMs directly in the browser.

This shift from the cloud to the client device offers several crucial advantages:

Improved data privacy: User data remains on the device and is never sent to a server.
Lower latency: Get instant responses without network delay.
AI offline: Use AI features even without an internet connection.
Reduced server costs: Developers are less reliant on expensive cloud GPUs.

You don't need a huge model for useful AI

The model used in this emoji translator project is Google's Gemma 3 270M-IT. It is described as an "ultra-compact" model known for its "strong command-execution capabilities" and has only 270 million parameters, making it perfectly suited for resource-constrained environments such as a web browser.

While the AI industry often focuses on models with billions or even trillions of parameters, this project proves something counterintuitive: smaller, highly specialized models can be extremely effective for specific tasks. Contrary to the assumption that "bigger is always better," this project shows that a well-trained, focused model can deliver powerful results without massive computational overhead. Precisely because these smaller models are so effective, the transition to on-device AI is now not only possible but also practical and economically viable.

It is possible to teach a general AI a surprisingly specific new ability

The basic Gemma 3 270M-IT model was already capable of executing general commands well. To become an emoji expert, it underwent a process called "fine-tuning," in which it was trained using a specially developed dataset of text-emoji pairs to learn the subtle art of emoji selection.

Through this training, the model learned to convert natural language into emoji output based on context and mood:

"I am so happy today!" →
"Let's go to the beach." → ️☀️️
"I love my pet cat." →

This adaptability is particularly effective when combined with the efficiency of a small model. It proves that you don't need a massive model with 100 billion parameters to develop a highly specialized and useful tool. This opens up the possibility for developers to create countless niche AI tools tailored to unique functions, all without consuming enormous computing resources.

This is offered by an open standard called ONNX

To implement this browser-based AI, the finely tuned model had to be converted into the ONNX (Open Neural Network Exchange) format. This conversion was the crucial step that unlocked its potential on the end device. ONNX acts as a technological bridge, an open standard that allows models trained in one framework (such as PyTorch) to run with high performance in a completely different environment.

Without this standard, the transition from cloud-based training to on-device deployment would remain a major hurdle. For this project, converting the model to ONNX resulted in a reduction in size and optimization for faster inference on a wide range of hardware, including the CPUs and GPUs found in everyday devices. Modern libraries like Transformers.js also significantly simplify the loading and use of these ONNX models for web developers, thus democratizing access to the development of on-device AI applications.

Conclusion

This emoji translator is far more than just a gimmick; it's a concrete example of a future where AI is more private, accessible, and efficient. The interplay of powerful little models, the possibilities of task-specific fine-tuning, and open standards like ONNX are enabling a paradigm shift. We are moving toward a world of highly personalized, privacy-focused "micro-AIs" integrated directly into our everyday applications—tools that work offline, respond instantly, and securely store our data on our own devices.

This project raises a forward-looking question that every developer and creator should consider: "Think of other small, concrete tasks that you could teach an LLM to perform directly in your browser."

This is a summery of my original article. The full version can be found here. ->
NeuralStack | MS