Coder Legion - Recent posts tagged local-ai

What Is Context Size in Ollama? QWEN 3.8 Context Window Explained into Ollama

Sun, 18 Oct 2026 11:13:04 +0000

Confused about context size in Ollama? Wondering what an AI context window is, why it matters, and how to configure it for Qwen 3.8 27B?

In this beginner-friendly tutorial, you'll learn how context size works in local AI models, how to increase the context length in Ollama, and how to test different context sizes with real examples.

I'll walk you through the basics of AI context windows, tokens, num_ctx, memory usage, and performance so you can run your local AI models more effectively.

Whether you're just getting started with Ollama or experimenting with local LLMs, this video will help you understand context size and configure it with confidence.

🚀 What You'll Learn

✅ What is context size in Ollama?
✅ AI context window explained in simple terms
✅ Qwen 3.8 native context length vs. Ollama's configured context
✅ How to increase context size using Ollama
✅ How to test different context sizes with real examples
✅ How context size affects RAM, VRAM, memory usage, and performance
✅ How to check whether your configured context size is being used
✅ Tips for choosing the right context size for local AI models

Zaylo Local AI

Wed, 07 Oct 2026 15:02:47 +0000

Zaylo: Turn Your Existing Hardware Into a Private AI Server

Running AI locally usually means dealing with model compatibility, hardware limitations, inference runtimes, and configuration.

Zaylo is built to simplify that process.

Zaylo turns an existing computer into a private AI server by detecting the available hardware, helping select suitable models, managing local inference, and exposing an OpenAI-compatible API that applications can use.

Why Zaylo?

AI infrastructure doesn't always need to live in the cloud.

Many developers already have capable CPUs, GPUs, RAM, and storage sitting on their desks. Zaylo is designed to make that hardware usable as local AI infrastructure without requiring a complicated setup.

The goal is simple:

Your hardware → Zaylo → Local AI infrastructure

What Zaylo Does

Zaylo handles several parts of the local AI workflow:

Hardware detection and capability analysis
Model compatibility and recommendation
Local model management
AI inference through local runtimes
OpenAI-compatible API access
RAG workflows
Vision model support
Automatic process recovery
Local and private AI execution

This allows developers to connect their own applications to locally running models instead of sending every request to a cloud AI provider.

Zaylo V2

Zaylo V2 expands the platform beyond basic local model serving.

RAG Support

Applications can use locally running models with retrieval-augmented generation workflows, making it possible to build knowledge-based AI systems while keeping the underlying data local.

Vision Models

Zaylo V2 adds support for vision-capable models, allowing applications to work with images alongside traditional text-based AI workloads.

Automatic Process Recovery

Local AI processes can fail for many reasons, especially when hardware resources are limited.

Zaylo includes automatic process recovery to make local inference more reliable instead of requiring users to manually restart the service.

Built for Developers

Zaylo exposes an OpenAI-compatible API, making it easier to integrate local models into existing applications.

Developers can build:

AI assistants
RAG applications
Private document processing systems
Vision applications
Local development environments
Internal AI tools
AI-powered developer tools

without having to redesign their applications around a completely different API.

The Bigger Idea

Cloud AI is convenient, but local AI provides something different: control.

With Zaylo, the objective is to make existing hardware a practical part of the AI infrastructure stack.

Instead of asking:

"Which cloud service should I use?"

developers can also ask:

"What can my own hardware run?"

Zaylo is being developed by LoftyTech Labs with a focus on making local AI infrastructure easier to deploy, manage, and integrate.

The project is actively maintained and evolving toward supporting more models, runtimes, hardware configurations, and AI workloads.

Zaylo — your hardware, your models, your AI infrastructure.

Qwen2.5 7B vs Qwen3 4B & 8B for Writing Correction: 60 Local Ollama Responses on Windows

Mon, 28 Sep 2026 04:48:41 +0000

I expected Qwen2.5 7B to retain a noticeable advantage over the smaller Qwen3 4B model for writing correction.

In this experiment, it didn't.

Across the same 20 paired writing cases, Qwen2.5 7B and Qwen3 4B produced exactly the same complete-case outcome: both succeeded on the same 18 cases and failed on the same 2.

The difference became much more noticeable in local execution time: in the tested cold-start setup, Qwen3 4B averaged 23.99 seconds, compared with 54.37 seconds for Qwen2.5 7B.

That raised another practical question:

Can a smaller model from a newer generation reproduce the practical writing-correction performance of a larger model from the previous generation?

This benchmark is a follow-up to my earlier Qwen3 4B vs 8B vs 14B writing-correction experiment.

In that first study, Qwen3 8B and 14B both achieved 19/20 complete cases, while the mean cold-start time for Qwen3 8B was roughly half that of Qwen3 14B.

This second experiment approaches the same model-selection question from another direction: instead of comparing sizes within Qwen3, it compares Qwen2.5 7B with Qwen3 4B and Qwen3 8B under the same paired writing-coach evaluation framework.

Benchmark setup

The benchmark uses 20 paired writing cases, including 16 sentences with expected errors and 4 already-correct controls.

Each case was evaluated with the same frozen reference and the same evaluation framework across three locally hosted Qwen models:

qwen2.5:7b
qwen3:4b-q4_K_M
qwen3:8b-q4_K_M

This produced 60 local responses in total.

The models were executed through Ollama on Windows under the same benchmark conditions. The objective was not to identify a universal “best model,” but to compare several dimensions separately:

complete correction of the expected errors;
expected-correction coverage;
localization of the error regions;
preservation of the original facts;
compliance with the requested output structure;
explanation-language and consistency checks;
local cold-start execution time.

For the primary metric discussed here, a case was considered complete only when all expected correction requirements for that case were satisfied without an incorrect replacement, false error, or configured fact loss.

Results

Complete-case correction

The primary correction results were close:

Qwen3 4B: 18/20 complete cases — 90%
Qwen2.5 7B: 18/20 complete cases — 90%
Qwen3 8B: 19/20 complete cases — 95%

On the primary metric alone, Qwen3 8B completed one additional case in this run.

The more distinctive result, however, was the paired comparison between Qwen2.5 7B and Qwen3 4B.

They did not merely finish with the same 18/20 score. They succeeded on the same 18 cases and failed on the same 2 cases.

Across all 20 paired cases:

both succeeded: 18/20
both failed: 2/20
different complete-case outcomes: 0/20

So, for this specific 20-case writing benchmark, moving from Qwen2.5 7B to Qwen3 4B preserved the same primary correction outcome case by case.

This should not be interpreted as evidence that the two models are universally equivalent. It means that no complete-case difference was observed between them within this controlled test set.

Local execution time

The difference became much larger when local cold-start execution time was considered.

In the tested configuration, the mean total cold-start times were:

Qwen3 4B: 23.99 s
Qwen2.5 7B: 54.37 s
Qwen3 8B: 60.24 s

Qwen3 4B therefore required less than half the mean cold-start time of Qwen2.5 7B in this run.

Expressed as a ratio, Qwen3 4B was approximately 2.27× faster than Qwen2.5 7B on this specific cold-start measurement, corresponding to about a 55.9% reduction in mean cold-start time.

This makes the paired correction result more practically interesting: Qwen3 4B preserved the same complete-case outcome as Qwen2.5 7B across all 20 cases while responding substantially faster in the tested local setup.

Qwen2.5 7B and Qwen3 8B, by contrast, were much closer in execution time. Qwen2.5 7B averaged 54.37 s versus 60.24 s for Qwen3 8B, making the 7B model about 9.7% faster on mean cold-start time in this run.

These timing results should be interpreted within the tested configuration. They describe cold-start execution time on this local Windows/Ollama setup, not a universal model-speed ranking.

Error localization and explanation checks

Complete-case correction does not capture every difference between the models.

Across the 21 expected error regions:

Qwen3 4B: 20/21 regions localized — 97.6% F1
Qwen2.5 7B: 20/21 regions localized — 97.6% F1
Qwen3 8B: 21/21 regions localized — 100% F1

Neither Qwen3 4B nor Qwen2.5 7B produced a false-positive error region in this run, while Qwen3 8B localized all 21 expected regions with no false positive.

Expected-correction coverage followed a similar pattern:

Qwen3 4B: 19/21 expected corrections — 90.5%
Qwen2.5 7B: 19/21 expected corrections — 90.5%
Qwen3 8B: 20/21 expected corrections — 95.2%

So Qwen2.5 7B and Qwen3 4B again matched on both correction coverage and error-localization F1, while Qwen3 8B retained a small advantage on these measures.

The explanation checks showed a more nuanced difference.

For explanation-language compliance:

Qwen2.5 7B: 40/40 evaluated fields passed
Qwen3 8B: 41/41 evaluated fields passed
Qwen3 4B: 37 passes and 3 warnings

All three models nevertheless produced parseable JSON, usable core output and full-contract-compliant output in all 20 cases.

The configured fact-preservation check also passed all 20 cases for every model.

Among the 16 error-containing cases, the action–explanation consistency check recorded 16 consistent results for each model, with no contradictory, ambiguous or missing entries.

These automatic checks should not be interpreted as a complete measure of pedagogical quality. They capture selected properties such as language compliance, output discipline and consistency, while deeper explanation quality would require additional linguistic or human evaluation.

What this means for local writing assistants

The main takeaway is not that one model is universally better than the others.

In this benchmark, Qwen2.5 7B and Qwen3 4B produced the same complete-case outcome on all 20 paired cases, while Qwen3 4B required substantially less cold-start time in the tested local setup.

Qwen3 8B, however, completed one additional case and reached 100% error-localization F1.

This creates a practical trade-off rather than a single winner.

For this specific workload:

Qwen3 4B offered the strongest efficiency result: the same paired complete-case outcome as Qwen2.5 7B, with much lower measured cold-start time.
Qwen2.5 7B remained a credible previous-generation baseline: it matched Qwen3 4B on complete cases, expected-correction coverage and error-localization F1, while showing cleaner explanation-language compliance in this run.
Qwen3 8B provided the strongest correction coverage: one additional complete case, one additional expected correction and full error localization.

This suggests that choosing a local writing model should be treated as a multi-criteria decision rather than as a simple parameter-count comparison.

For an interactive writing assistant, the relevant trade-off may involve correction reliability, explanation behavior, latency, model size and the available hardware.

In this run, Qwen3 8B provided a small measurable correction advantage, but that advantage came with substantially higher cold-start time than Qwen3 4B.

Limitations

This benchmark is exploratory and should be interpreted within its experimental scope.

Several limitations are important:

20 paired cases: the dataset is intentionally focused and is not large enough to support a universal ranking of the models.
One language scenario: the benchmark evaluates English writing correction with explanations requested in French. Results may differ for other languages or writing tasks.
One model family: the comparison is limited to Qwen2.5 7B, Qwen3 4B and Qwen3 8B and should not be generalized to unrelated local model families.
One local machine: execution time depends on hardware, background workload, model format, quantization and Ollama configuration.
Cold-start timing: the reported measurements describe this specific cold-start workflow rather than warm-start interaction latency.
Reference-based evaluation: outputs were compared against the same frozen, versioned correction reference and acceptance rules across all three models.
Automatic explanation checks: language compliance, output discipline and consistency were evaluated automatically, but these checks do not constitute a complete linguistic or pedagogical assessment.
No direct resource measurement: RAM, VRAM, CPU load and energy consumption were not measured in this run.

The last point is particularly important for interpreting the efficiency result.

Qwen3 4B has a substantially smaller nominal parameter count than Qwen2.5 7B, but parameter count alone should not be treated as a direct measurement of hardware-resource usage.

The benchmark therefore supports a measured latency comparison, not a claim that Qwen3 4B used a specific percentage less RAM, CPU or energy.

What I plan to test next

This second benchmark leaves several useful directions open for further evaluation.

The next iterations could extend the framework in four directions:

measure peak RAM and VRAM usage alongside correction quality;
compare cold-start and warm-start latency under the same local conditions;
extend the benchmark to a larger and more diverse writing test set;
compare additional local model families under the same evaluation protocol.

Direct resource measurements would be particularly useful.

In this run, Qwen3 4B showed a strong latency advantage over Qwen2.5 7B while preserving the same paired complete-case outcome, but memory and compute usage were not measured directly.

Adding RAM, VRAM and processor-load measurements would make it possible to evaluate the trade-off more completely across:

correction reliability;
explanation behavior;
latency;
memory footprint;
compute load;
and model size.

I am particularly interested in whether the efficiency pattern observed here remains stable when the task becomes more linguistically demanding and when resource usage is measured directly.

Conclusion

This experiment did not identify a universal winner, but it revealed a clear practical trade-off.

For this specific 20-case writing benchmark, Qwen3 4B and Qwen2.5 7B produced the same complete-case outcome on every paired case:

both completed 18/20 cases;
both applied 19/21 expected corrections;
both reached 97.6% error-localization F1.

The difference was much larger in local execution time.

Qwen3 4B averaged 23.99 seconds of total cold-start time, compared with 54.37 seconds for Qwen2.5 7B.

In this tested configuration, that corresponds to approximately 2.27× faster cold-start execution for Qwen3 4B while preserving the same primary paired correction outcome.

Qwen3 8B remained slightly stronger on correction coverage, completing 19/20 cases and reaching 100% error-localization F1, but with a mean cold-start time of 60.24 seconds.

The more useful question, therefore, may not be:

Which model has the most parameters?

but rather:

Which model provides the best balance between correction reliability, explanation behavior, responsiveness and available local hardware for the intended task?

For local interactive applications, model size alone is not enough to answer that question.

Earlier benchmark

Before this cross-generation comparison, I tested Qwen3 4B, 8B and 14B on the same 20 writing cases.

Read the earlier Qwen3 benchmark →

Disclosure: AI assistance was used to refine the wording and organization of this article. The benchmark results and measurements reported here come from the documented experimental runs and were reviewed for accuracy before publication.

From Electronics to Local AI — Sami M.’s Developer Journey

Thu, 24 Sep 2026 11:11:33 +0000

From Electronics to Local AI — Sami M.’s Developer Journey

Developer Stories — CoderLegion

From childhood curiosity about computer hardware to robotics, artificial intelligence, multi-agent systems, and today's local LLMs, Sami M.'s journey has always been driven by one thing: understanding how intelligent systems work.

Sami M. is an Electronics Engineer and PhD in Artificial Intelligence (2014). Today, his work and experiments focus heavily on AI, local AI, LLMs, and Ollama, with a particular interest in understanding what these technologies can actually do in practical, real-world situations.

You can find Sami on CoderLegion.

His recent CoderLegion work includes an experiment comparing Qwen3 4B, 8B and 14B for writing correction using 60 local Ollama responses on Windows.

But the story behind that experiment started much earlier.

It Started With an Intel 286

Sami's interest in electronics began when he was very young.

He remembers being fascinated by the Intel 286 and by the idea that a processor and electronic components could somehow make his favourite game, Gorillas under MS-DOS, come to life.

He didn't just want to use the computer.

He wanted to understand what was happening inside it.

That curiosity naturally pushed him toward electronics and engineering.

His move toward artificial intelligence came later through robotics.

One of his early projects involved autonomous trajectory following with a Pioneer 3-DX mobile robot. From there, his interests expanded into more ambitious work involving collaborative multi-agent systems and task-allocation mechanisms based on auction algorithms.

The combination of electronics and AI still feels like a natural continuation of that original curiosity.

From Robotics to Artificial Intelligence

What attracted Sami to AI was ultimately a question about intelligence itself.

He became interested in biological intelligence and biological neural networks, and then in how some of those ideas could inspire artificial systems.

His early work included robotics, computer vision, and mobile-robot path planning.

Classical graph-search techniques such as Dijkstra's algorithm and heuristic search methods such as A* were part of that journey.

Over time, his interest moved toward systems that could do more than execute a fixed algorithm.

He became increasingly interested in systems that could:

Perceive a situation
Reason about it
Make decisions
Interact with other intelligent systems

That path eventually led to his PhD in Artificial Intelligence, completed in 2014.

How AI Has Changed

The AI landscape Sami entered during his academic journey is very different from the one developers work with today.

One of the biggest changes he notices is the move from highly task-specific systems toward models capable of representing much richer relationships between language, context, and meaning.

He is careful about the word "understanding."

He doesn't claim that an LLM understands meaning exactly as a human does.

But the semantic behaviour these models can demonstrate is still remarkable to him.

Earlier AI systems often required intelligence to be explicitly designed through rules, features, decision logic, or algorithms tailored to a particular problem.

Today, much richer behaviour can emerge from representations learned in high-dimensional spaces.

And one development particularly interests him:

local AI.

Why Local AI?

For Sami, one of the strongest attractions of local AI is control over information.

When AI runs locally, sensitive or private information can remain on his own machine.

That matters when working with technical documents, private information, or other material that he may not want to send to an external service.

It also gives him more freedom to design the architecture around his own requirements.

RAG is one example.

Sami has experimented with large collections of training videos. Instead of manually watching hundreds of hours of material, he can:

Transcribe the content
Divide it into useful segments
Generate embeddings
Retrieve relevant passages
Interact with that knowledge through a local model

This turns a huge collection of information into something he can query directly.

For Sami, local AI isn't simply about running a model without an internet connection.

It's about privacy, control, architecture, and practical usefulness.

Experimenting With Qwen and Ollama

Sami's recent CoderLegion article, Qwen3 4B vs 8B vs 14B for Writing Correction: 60 Local Ollama Responses on Windows, came from a very practical need.

He writes technical reports and emails, sometimes in very different communication styles.

Constantly switching between correction, improvement, explanation, and tone adjustment was taking time.

So he wanted something that could:

Correct genuine mistakes
Suggest improvements when useful
Work with text from different Windows applications
Allow him to keep his original wording when preferred
Explain corrections in a language he understands well

Instead of simply assuming that the largest model would perform best, Sami built a reusable benchmark framework to test different local models under the same conditions.

The results reinforced an important lesson:

Model size alone isn't enough.

In his tests, a larger model did not necessarily improve the main correction result, while the difference in execution time could be significant.

Benchmarks vs. Real-World Testing

Sami sees published benchmarks as useful.

They can help create a shortlist.

But when it comes to deciding whether a model is actually useful, he prefers testing it against his own real-world requirements.

His writing-correction experiment also connects to something personal.

English isn't his first language, and he doesn't have much time to study English separately from his engineering work.

So he thought:

Why not learn while already working?

He already needed to write reports and emails in English.

By using AI to correct mistakes, suggest improvements, and explain corrections, he could learn while completing work he already needed to do.

He doesn't claim this is a formally validated language-learning method.

But personally, he has found it useful.

For Sami, the important point is simple:

A benchmark can tell you something about a model. Your own problem tells you whether that model is useful.

Human First

Sami's approach to AI can be summed up simply:

Human first.

He uses AI extensively, but he doesn't want AI making important decisions for him.

Instead, he uses it to:

Challenge an idea
Show him another perspective
Point out something he may have missed
Save time on repetitive work
Explore possibilities

Sometimes he will have several exchanges with an AI system before deciding whether an idea is worth keeping.

But the final judgment remains his.

"For me, the best use of AI is not to replace human expertise, but to make that expertise more effective."

That philosophy is particularly relevant as AI becomes increasingly integrated into everyday development work.

The Challenge of Multi-Agent Robotics

One of the most challenging technical areas Sami has worked on is heterogeneous multi-agent robotics.

Imagine a humanoid robot and a tracked mobile robot working together on the same task.

They may have completely different:

Ways of moving
Sensors
Architectures
Physical capabilities

The challenge isn't simply getting them to communicate.

The harder questions are:

Who should do what?

When should they do it?

How can the system take advantage of the fact that the agents are different?

Sami worked with dynamic task-allocation approaches, including auction-based mechanisms where tasks can be allocated according to the capabilities and current state of different agents.

It is a problem of making different intelligent systems cooperate even though they don't necessarily experience or interact with the world in the same way.

Choosing AI Without Following the Hype

The AI ecosystem changes incredibly quickly.

New models and tools appear constantly.

Sami tries not to choose a model simply because it is fashionable or because everyone is talking about it.

He starts with the problem.

He asks:

What exactly am I trying to achieve?
What level of quality do I need?
What latency is acceptable?
How much memory do I have?
Do I need privacy?
Does it need to work offline?
Do I need RAG?
Do I need structured output?

Once those constraints are clear, he tests.

A smaller model with good context and a well-defined task can sometimes be more useful than a massive model used with noisy input or without a clear understanding of the problem.

But Sami doesn't treat that as a universal rule.

He prefers to measure.

"For me, specifications come before trends."

What Comes Next for Local LLMs?

Sami doesn't think the future of local LLMs is simply about making models bigger.

He believes efficiency will be just as important:

Better architectures
Improved quantization
Better memory management
Greater efficiency with fewer parameters

One example from his own experiments illustrates this.

For his particular test set, Qwen3 4B matched Qwen2.5 7B on the primary complete-case writing-correction metric he was measuring.

That doesn't mean the two models have the same overall reasoning capabilities.

It means that for that particular workload, the larger parameter count didn't improve the main result he was measuring.

That is precisely why he prefers experimentation over assumptions.

He expects increasingly capable local models to continue running on ordinary consumer hardware, and that is one of the developments he finds most exciting.

Electronics Still Shapes How He Thinks About AI

Despite spending much of his time working with software and AI, Sami hasn't forgotten the physical layer underneath it all.

His electronics background influences the way he evaluates AI systems.

He naturally thinks about:

GPU architecture
Memory
Data movement
Computational cost
Latency
Energy consumption
Responsiveness
Efficiency

A model isn't just an abstract collection of parameters.

It has to fit into memory.

Data has to move through the system.

The hardware has limits.

Energy and latency matter.

That physical view of computing is one reason local AI interests him so much.

His Advice to Developers Exploring Local AI

Sami's advice is straightforward:

Start with the problem, not with the model.

His suggested approach is:

Define one clear use case.
Decide what a good result actually looks like.
Choose a reasonably small model.
Run it locally using something accessible such as Ollama.
Create representative test cases.
Measure the results.
Only then consider moving to a larger model.

And if the application involves RAG or external knowledge, spend time cleaning and structuring the data.

Good context can make a significant difference.

For Sami, a well-defined problem, a suitable model, and good-quality input can often matter more than simply choosing the largest model available.

Finding a Technical Community

Sami joined CoderLegion in September 2026 and has already become active in the AI, Dev Tools, and Python communities.

What he appreciates most is the technical relevance of the discussions.

He has received questions about the methodology behind his experiments and discussions about the technical choices he made.

As an engineer, he says he is much more comfortable discussing the substance of a project than promoting himself.

That makes a technically focused community a natural fit.

As CoderLegion grows, he hopes it maintains that technical spirit:

People sharing what they tested, what worked, what didn't work, and why.

For Sami, that kind of discussion is more useful than visibility for its own sake.

Looking Ahead

One of Sami's current research interests is radio-spectrum management.

He is exploring how computational methods, optimization, and AI can support problems such as:

Spectrum sharing
Interference analysis
More efficient use of limited spectral resources

But even here, his philosophy remains the same.

He doesn't want to add AI simply because AI is available.

He prefers to first understand the engineering problem and then identify where AI can provide measurable value.

That philosophy connects his past and present work.

From the Intel 286 that first sparked his curiosity, to robotics, artificial intelligence, multi-agent systems, and today's local LLMs, the technologies have changed dramatically.

The curiosity hasn't.

About Sami M.

Sami M. is an Electronics Engineer and holds a PhD in Artificial Intelligence (2014). His interests and work span electronics, robotics, artificial intelligence, multi-agent systems, local AI, LLMs, and practical experimentation with AI tools such as Ollama.

Follow Sami on CoderLegion.

CoderLegion Developer Stories

Real developers. Real journeys. Real conversations.

Developer Stories is a CoderLegion series featuring developers, engineers, researchers, and builders sharing the experiences, challenges, ideas, and lessons behind their work.

If you'd like to share your own developer journey, join CoderLegion.

Qwen3 4B vs 8B vs 14B for Writing Correction: 60 Local Ollama Responses on Windows

Tue, 08 Sep 2026 10:25:29 +0000

I expected the larger Qwen3 model to show a clear advantage for writing correction. In this experiment, it didn't.

Across 20 paired writing cases, Qwen3 8B and 14B both achieved 19/20 complete-case corrections, while Qwen3 4B reached 18/20.

The difference became much more noticeable in local execution time: in the tested cold-start setup, the mean cold-start time for Qwen3 8B was roughly half that of Qwen3 14B.

That raised a practical question:

For a local writing assistant, how much model size do we actually need?

To explore this more systematically, I built a reproducible benchmark comparing Qwen3 4B, 8B and 14B under the same local conditions with Ollama on Windows.

Benchmark setup

The benchmark uses 20 paired writing cases, including 16 sentences with expected errors and 4 already-correct controls.

Each case was evaluated with the same reference and the same evaluation framework across three locally hosted Qwen3 models:

qwen3:4b-q4_K_M
qwen3:8b-q4_K_M
qwen3:14b-q4_K_M

This produced 60 local responses in total.

The models were executed through Ollama on Windows under the same test conditions. The objective was not to identify a universal “best model,” but to compare several dimensions separately:

complete correction of the expected errors;
localization of the error regions;
preservation of the original meaning;
compliance with the requested output structure;
quality of the explanations;
local execution time.

For the primary metric discussed here, a case was considered complete only when the expected correction requirements for that case were satisfied.

Results

Complete-case correction

The primary correction results were close:

Qwen3 4B: 18/20 complete cases — 90%
Qwen3 8B: 19/20 complete cases — 95%
Qwen3 14B: 19/20 complete cases — 95%

On this metric alone, increasing the model size from 8B to 14B did not improve the number of complete cases in this test.

Local execution time

The difference was much larger in execution time.

In the tested cold-start configuration, the mean cold-start times were approximately:

Qwen3 4B: 26.6 s
Qwen3 8B: 60.0 s
Qwen3 14B: 119.4 s

So, while 8B and 14B achieved the same complete-case score, Qwen3 8B took about half as long as Qwen3 14B on average in this cold-start setup.

The 4B model was substantially faster again, while still completing 18 of the 20 cases.

This does not mean that 8B is universally better than 14B. It means that, for this specific writing-correction workload and hardware configuration, the additional model size did not translate into a better primary correction score, while the execution-time cost increased considerably.

Error localization and explanation quality

Complete-case correction and error localization measure different aspects of model behavior.

Across the 21 expected error regions:

Qwen3 4B missed one expected region, with an error-localization F1 of 97.6%.
Qwen3 8B localized all 21 expected regions, reaching 100% F1.
Qwen3 14B missed one expected region and reported one false-positive region, resulting in 95.2% F1.

This distinction matters for a writing coach: producing an acceptable corrected sentence does not necessarily mean that the model identified the underlying error with the same precision.

Explanation quality was therefore evaluated separately from correction success.

In this run, all 41 evaluated explanation fields produced by Qwen3 8B passed the language check. Qwen3 4B showed more explanation-language and terminology issues.

These automatic checks should not be interpreted as a complete measure of pedagogical quality. They cover selected language patterns and contradictions, while deeper explanation quality remains a separate area for linguistic review.

What this means for local writing assistants

The main takeaway is not that one model is universally better than another.

In this benchmark, the three Qwen3 models produced relatively similar complete-case correction results, but their local execution times differed much more substantially.

For this specific workload:

4B offered the lowest execution time, with a small reduction in correction coverage.
8B matched 14B on complete-case correction and achieved the strongest error-localization result in this run.
14B did not improve the primary correction score over 8B, while requiring substantially more execution time in the tested configuration.

This suggests that model selection for a local writing assistant should be treated as a multi-criteria decision, rather than assuming that a larger model will automatically provide the best practical experience.

Depending on the use case and available hardware, the relevant trade-off may involve:

correction reliability;
explanation quality;
latency;
memory and compute requirements;
and the expected frequency of use.

For a writing coach in particular, responsiveness also matters. A model that is slightly larger or theoretically more capable may not necessarily provide the best user experience if each interaction becomes significantly slower.

Limitations

This benchmark is exploratory and should be interpreted within its experimental scope.

Several limitations are important:

20 paired cases: the dataset is intentionally focused and is not large enough to support a universal ranking of the models.
One language scenario: the benchmark evaluates English writing correction with explanations requested in French. Results may differ for other languages or writing tasks.
One model family: only Qwen3 4B, 8B and 14B were compared. The results should not be generalized to other local model families.
One local machine: execution time depends on hardware, model quantization, available memory and Ollama configuration.
Cold-start timing: Ollama was configured with keep_alive=0, meaning the model was reloaded for each response. The reported timings therefore represent this specific cold-start workflow rather than warm-start interaction latency.
Reference-based evaluation: outputs were compared against a frozen, versioned correction reference prepared for this study. This improves consistency across models, but it does not capture every possible acceptable linguistic formulation.
Explanation-quality checks: automatic checks cover selected language patterns and contradictions. They should not be treated as a complete assessment of pedagogical or linguistic quality.

These limitations are also why the benchmark is presented as a controlled comparison rather than a claim about the absolute capabilities of each model.

What I plan to test next

This first benchmark leaves several useful directions open for further evaluation.

The next iterations could extend the framework in four directions:

compare additional local model families under the same evaluation protocol;
examine explanation quality with deeper linguistic and human review;
compare cold-start and warm-start performance;
evaluate the same workflow on more constrained hardware.

A larger and more diverse test set would also help determine whether the patterns observed here remain stable beyond these 20 cases.

I am particularly interested in whether the 8B/14B trade-off observed in this run persists when the task becomes more linguistically demanding.

Conclusion

This experiment did not produce a universal winner, but it did highlight an important practical point: model size alone is not a sufficient criterion for choosing a local writing model.

For this specific workload, Qwen3 8B matched 14B on complete-case correction while its mean cold-start time was roughly half that of 14B in the tested configuration.

The more useful question, therefore, may not be “Which model is the largest?” but rather:

Which model provides the best balance between correction quality, explanation quality, responsiveness, and available hardware for the intended use case?

If you use local LLMs for writing assistance or other interactive tasks, I would be interested to know which trade-off matters most in your own setup: quality, latency, memory requirements, or something else?

Follow-up benchmark

I later extended this evaluation by comparing Qwen2.5 7B with Qwen3 4B and Qwen3 8B under the same 20-case writing-coach framework.

In that follow-up, Qwen3 4B matched Qwen2.5 7B on every paired complete-case outcome while averaging 23.99 s of cold-start time versus 54.37 s for Qwen2.5 7B.

Read the follow-up benchmark →

Disclosure: AI assistance was used to refine the wording and organization of this article. The benchmark results and measurements reported here come from the documented experimental runs and were reviewed for accuracy before publication.

The Paper Sketch: Budgeting 48GB of UMA

Sun, 19 Jul 2026 14:07:16 +0000

Let's look at the physical footprints at a 4-bit (Q4) quantization. A 744B parameter model sits at roughly 370 GB cold on the NVMe drive. Because it’s a sparse Mixture of Experts, the dense backbone—the attention mechanisms and shared weights that every single token must touch—is surprisingly light: about 17B parameters, or roughly 9.9 GB.

If we draw up the memory allocation inside our 48GB pool, the board looks like this:

48 GB (Total) - 9.9 GB (Dense Backbone) - 4 GB (OS & Apps) - 2 GB (KV Cache) ≈ 32 GB Remaining

That leaves us a 32GB sandbox to allocate entirely to the lock-free expert ring buffer.

When this model executes, it activates roughly 40B parameters per token. At Q4 precision, that requires pulling ~11 GB of active experts into memory per step. Because our ring buffer is 32GB, it has enough structural breathing room to hold 2 to 3 slots of speculatively loaded future tokens simultaneously. The memory geometry fits.

Calculating the Bandwidth Reality

Now we look at the unyielding bottleneck: the internal Apple Silicon SSD bus, which reliably pushes a sustained read bandwidth of about 6.5 GB/s.

If the engine had to blindly pull a completely fresh set of 11 GB of experts for every single token, the raw physical limit is basic arithmetic:

Raw I/O Time Per Token = 11 GB / 6.5 GB/s ≈ 1.69 seconds ⇒ 0.59 tokens/second

But we know neural routing isn't completely chaotic. Tokens traveling through a sentence exhibit temporal expert locality—the token handling a specific technical context will likely trigger some of the exact same experts used by the token right before it.

If the speculative lookahead successfully exploits this, allowing us to reuse just 20% of the active experts already sitting warm in the ring buffer, the actual disk read demand drops from 11 GB down to ~8.8 GB:

Optimized I/O Time = 8.8 GB / 6.5 GB/s ≈ 1.35 seconds ⇒ 0.74 tokens/second

This lands us squarely in that 0.6 to 0.8 tokens per second window.

Perspectives on Speed

Not that we are calling this fast by modern server standards, but it completely rewrites the expectations for what a local machine can do. It’s all about perspective.

We used to wait hours just to download a single movie;

Waiting 30 or 40 seconds for a frontier-class AI to generate a deeply reasoned paragraph locally on a laptop isn't a penalty—it's a miracle.

Will it hold up when the code actually executes? The ultimate variable is the "Scout Fan-Out." With hundreds of potential experts per layer, the speculative engine's branch prediction has to be incredibly precise. If it misguesses too often, the pipeline will stall, waiting for synchronous disk reads.

But looking at the numbers on the board right now, the physics check out. We aren't just daydreaming—this approach stands a very real chance of turning an impossible 744B paperweight into a highly capable, private asynchronous workhorse.

What I Learned Building a Local RAG Agent

Mon, 01 Jun 2026 20:45:43 +0000

A Quick Intro

I recently built a local RAG agent that reads a bunch of documents stored as markdown files and lets you ask questions about them in plain English. It goes through all the documents and figures out an answer based on what's actually in them.

The Architecture

Here's a high-level look at the 5-stage pipeline it uses:

Your Markdown Files
       ↓
 [1. INGESTION]    — Reads files, splits into small pieces ("chunks")
       ↓
 [2. EMBEDDINGS]   — Converts each chunk into a list of numbers (a "vector")
                     that captures its *meaning*
       ↓
 [3. STORAGE]      — Saves those vectors to a local database (ChromaDB)
       ↓
 [4. RETRIEVAL]    — When you ask a question, finds the most relevant chunks
       ↓
 [5. ORCHESTRATION] — Feeds those chunks to a local AI model
                      which writes a full answer

Key Concepts I Picked Up

1. Ingestion

This is the step where your files get read from disk and broken into small, meaningful pieces called chunks, so the AI can make sense of them.

It has 3 parts:

Part 1: The Watcher

This watches a folder and triggers an event whenever something changes. A library listens to the OS, and when it detects a file change at the specified path, it fires off an event like one of these:

FileEvent(path="notes/docker.md", event_type="created")
FileEvent(path="notes/docker.md", event_type="modified")
FileEvent(path="notes/docker.md", event_type="deleted")

Part 2: The Parser

This is the reader. It takes raw file content and turns it into clean, usable text.

It does a few useful things:

Strips all markdown symbols (**bold** → bold, # Heading → Heading) so the AI gets plain text
Pulls out any YAML front matter (the --- metadata block at the top of some files)
Splits the document into sections by heading level

Part 3: The Chunker

This takes each section and cuts it into chunks of roughly 512 words, with a 64-word overlap between consecutive chunks.

Why overlap? Imagine a sentence that falls right at the boundary between two chunks, without overlap, you'd lose that context. With overlap, both chunks carry a little bit of their neighbour's content, so nothing important gets cut off.

Ingestion in one line

A folder watcher detects file changes → the parser reads and cleans the text → the chunker slices it into overlapping bite-sized pieces, each tagged for change detection.

2. Embedding

Goal: Take the chunks from the Ingestion phase and attach a vector (a list of numbers) to each one, so it can be stored and searched by meaning.

Think of it like a translator, it takes human-readable text and converts it into a mathematical form that the database can actually compare.

It has 2 parts:

Part 1: The Embedder

This is the worker at the translation desk. Its one job: take a list of text strings, send them to a locally running embedding model, and get back a list of vectors.

Part 2: The Smart Manager

This sits above the Embedder and decides which chunks actually need embedding, because embedding is slow and costs compute.

This is what makes repeat runs fast, if you add one new file to a folder with 200 already-indexed files, only the new file's chunks get processed.

Embedding flow

List[Chunk]  (from Phase 1)
     ↓
ChunkEmbedder.embed_chunks()
     ├── check (content_hash, source_path) against known pairs
     ├── filter to only NEW chunks
     ├── call Embedder  ← in batches
     └── pair each chunk with its vector
     ↓
List[EmbeddedChunk]  (Chunk + vector, ready for storage)

Embedding in one line

The smart manager skips already-seen chunks and only sends the new ones to the embedder, which calls the local model to produce float vectors, outputting EmbeddedChunk objects ready for storage.

3. Storage

Goal: Save the EmbeddedChunk objects (text + vectors) to a database so they stick around between sessions and can be searched later.

A special kind of database called a vector database is used for this.

Here's what a stored record looks like:

| Field         | Example                          |
|---------------|----------------------------------|
| id            | "docker-guide:3"                 |
| vector        | [0.21, -0.83, 0.44, ...]         |
| metadata      | {content, source, hash, index…}  |

What's in the metadata?

Every stored chunk carries this info alongside its vector:

{
  "content":      "Docker is a platform for running containers...",
  "content_hash": "a3f9c2...",
  "source_path":  "/notes/docker.md",
  "chunk_index":  3,
  "heading":      "What is Docker?",
  "word_count":   87
}

Storage in one line

The vector database saves each chunk as a (vector + metadata) record on disk, supports fast similarity lookups, and on startup returns a list of already-indexed hashes so nothing gets re-embedded unnecessarily.

4. Retrieval

Goal: When you ask a question, find the most relevant chunks from the database, before handing anything to the AI.

How semantic search works

It's a simple two-step process:

Step 1: Embed the question

Your question is just text. To compare it against stored vectors, it first needs to be turned into a vector too:

"What is Docker?"  →  embed(["What is Docker?"])  →  [0.18, -0.71, ...]

Step 2: Find the closest matches

That query vector is compared against all stored vectors using cosine similarity to find the top-K closest ones:

"These 5 chunks are most similar to your question:"
  → chunk from docker-guide.md    (score: 0.91)
  → chunk from docker-guide.md    (score: 0.87)
  → chunk from rest-api-design.md (score: 0.62)
  ...

Each result comes back with a relevance score:

RetrievedChunk:
    chunk: Chunk    # full text, source path, metadata
    score: float    # 0.0 → 1.0 — higher = more relevant

Retrieval in one line

The retriever embeds the user's question into a vector, asks the database for the top-K closest stored chunks by cosine similarity, and returns them scored and sorted, ready for the AI.

5. Orchestration

Goal: Take the relevant chunks from Retrieval and use the LLM to write a real, human-readable answer.

This is the brain of the whole system, where raw retrieved text becomes an intelligent response.

The top-K chunks from the Retrieval phase are passed as context to the language model, which uses them to construct an answer.

The big benefit here is that it reduces hallucination. By giving the AI a focused set of chunks as context, it's nudged to base its answer on your documents rather than making things up from its general training. That said, tuning the top_k value matters, too few chunks and the answer is thin, too many and the model can get confused. The sweet spot is somewhere in the middle.

Orchestration in one line

The query engine wraps retrieval + LLM together and processes your question into a grounded, context-aware answer.

Wrapping Up

Building this agent taught me more than I expected, not just about RAG, but about how these systems actually think.

Each of the 5 stages has a very specific job, and they're all fairly simple on their own. The magic happens when you chain them together. A file changes → it gets chunked → embedded → stored → retrieved → answered.

A few things I'd highlight if you're building something similar:

Overlap in chunking matters more than you think. Without it, you lose context at boundaries and the answers suffer.
Deduplication at the embedding stage is a must. Re-embedding everything on every run is slow and wasteful. Track hashes.
Tune your top_k. Too small and the AI doesn't have enough to work with. Too large and it overthinks. Test it with real questions.
Local models are surprisingly capable. You don't always need a cloud API to get useful answers from your own documents.

If you're curious about RAG or local AI, this kind of project is a great starting point, it's small enough to understand fully, but complex enough to teach you the real fundamentals.

Everyone’s Building AI Agents. Here’s the One I Built for Myself

Fri, 29 May 2026 05:12:53 +0000

These days, everyone seems to be building AI agents.

So I figured I should probably build one too.

But instead of another generic demo, I wanted to solve a small problem I actually had.

Over time, I had collected a bunch of blog posts and technical notes in a folder. I wanted a quick way to understand them without opening every file one by one.

Questions like:

What topics do I write about most?
Do some posts contradict each other?
Can I create a learning path from my own notes?

Uploading everything to ChatGPT did not feel right.

My content. My machine.

So I built a small local RAG agent.

How it works

It reads markdown files, understands their meaning, and lets me query across all of them in plain English.

Under the hood, it looks something like this:

Markdown Files
      ↓
Chunking & Parsing
      ↓
nomic-embed-text (via Ollama)
      ↓
ChromaDB (local vector storage)
      ↓
Relevant Context Retrieval
      ↓
Mistral (via Ollama)
      ↓
Answer

Everything runs locally on my laptop.

No cloud. No external APIs. Just Python and a few focused tools.

Where AI Actually Helped

The architecture was clear before I started coding. The “vibe coding” part was mostly using AI to speed up repetitive work like tests, boilerplate, and wiring pieces together.

Idea to working CLI in about a week.

I’ve open sourced the project if you want to take a look:

Repo: Knowledge Onboarding Agent

What’s next

Next, I might add PDF support and a lightweight web UI.

Curious to know, have you built something similar recently? If yes, what problem did you solve?

Also curious, if you were building this, would you keep it fully local or use cloud APIs? What would you do differently?

From Manifesto to Metal: The Architecture of Local Sovereignty

Sun, 24 May 2026 01:15:41 +0000

How ANDARTIS moved from an abstract philosophical rebellion into a hardened, zero-latency reality on macOS.

In my previous writing, I spoke of a dream: The Sovereign Cortex. I painted a vision of a world where independent thinkers—the rogue researchers, the deep-dive clinicians, the authors, and the edge-case investigators—could divorce themselves from the rented cognition of the cloud and reclaim their intellectual privacy.

But dreams are light, airy things. Hardware is heavy.

When you transition from writing manifestos about private intelligence to actually compiling a local-first macOS application using NativePHP, philosophy crashes headfirst into the brutal constraints of the metal. If you have been following the development of ANDARTIS, you know that the early stages were a crucible of experimentation. We had grand ideas, but we also faced real-world bottlenecks: cold-start loading delays, memory choking, and the sheer chaos of orchestrating neural models on consumer laptops.

A rebel doesn’t just need a weapon; they need a beautifully engineered mechanism that doesn’t explode in their hands.

Today, the high-flying concepts have matured. The documentation is live, and the blueprint is set. Here is how we anchored the philosophy of private intelligence into the physical architecture of ANDARTIS.

The Humility of Constraints

Building a local-first AI application sounds deeply romantic until you actually try to boot a model.

In our early iterations, we ran into the harsh realities of local compute. If a user wanted to ask a quick question, they had to wait for the model weights to be read from disk and shoved into memory. It felt like trying to start a vintage diesel engine in the dead of winter. Worse, if the application tried to run an embedding process while simultaneously executing a conversational query, the Apple Neural Engine (ANE) would choke, memory would saturate, and the application would stall.

We had to stop treating local AI like a cloud server wrapped in a desktop shell. We had to respect the machine.

The breakthrough didn't come from chasing a bigger model or a heavier framework. It came from a humble realization: true speed comes from a symbiotic architecture that separates state from execution.

   [ User Interface (Inertia/Vue) ]
                  ▲
                  ▼
   [ Laravel & NativePHP Core ]  ◄───►  [ Isolated SQLite DB ]
                  ▲
                  │  (Zero-Latency JSON-RPC over STDIO)
                  ▼
   [ Persistent Python Daemon ]  ◄───►  [ Apple Neural Engine / VRAM ]

The Unlikely Symbiosis: NativePHP Meets MLX

To build a sovereign tool, we chose an unorthodox, deeply rebellious stack: Laravel, NativePHP, and Apple's MLX engine.

To the enterprise architect, marrying a PHP backend with a Python machine learning engine inside a desktop app looks strange. But to the artisan programmer, it is pure elegance. We don't spin up heavy local web servers or open vulnerable network ports on your Mac. Instead, the NativePHP core communicates with a background Python daemon using the humblest, oldest, and most reliable medium available: a zero-latency JSON-RPC pipeline over standard I/O (STDIO).

This hybrid architecture splits the labor perfectly, turning your Mac into a highly optimized, dual-engine machine.

1. The Persistent Daemon (Ending the Cold Start)

We eliminated the "diesel engine" delay by turning the Python side into a persistent daemon. When ANDARTIS boots, the daemon initializes once and cradles the neural weights gently in your Mac’s unified memory (VRAM). Because the weights never leave memory, the cold-start delay is obliterated. When you prompt the system, the execution is instantaneous.

2. PHP as the Sequential Guardrail

While Python handles the raw mathematics of neural execution, Laravel handles the sanity of the system. PHP acts as a wise conductor. It schedules execution queues so that heavy tasks—like model fine-tuning or massive data ingestions—run sequentially. It acts as a protective guardrail, ensuring that the Apple Neural Engine is never pushed to a memory-choking panic, keeping your laptop cool and responsive.

The Librarian: Quietly Resonating at the Edge

One of the greatest optimizations from our early stages is how ANDARTIS handles data ingestion. In the past, passing thousands of pages into a Large Language Model to figure out what they were about was a recipe for a thermal meltdown.

Now, we deploy The Librarian.

The Librarian is an ultra-fast, high-fidelity ingestion pipeline that operates entirely in the background. It walks your local directories, calculates cryptographic hashes to detect modified or new files, and indexes them without breaking a sweat.

But it doesn't wake up the heavy conversational model to understand your data. Instead, it uses a highly specialized global micro-model to perform what we call Semantic Resonance.

By matching your raw documents against forged structural prototypes, this tiny, hyper-efficient model extracts clean JSON metadata (like dates, names, or custom keys you define via Lenses) in milliseconds. It quietly writes this structured data into an isolated core.sqlite file. The main conversational brain stays completely asleep, conserving your battery and your memory, until the exact moment you ask a question.

Grounded Synthesis: The Speed of SQL, the Grace of AI

When you finally query ANDARTIS, the magic of this optimized architecture comes alive.

If you ask your local machine, "What was my total expenditure in March?" across 5,000 invoices, a typical local LLM will fail. The context window will overflow, or the system will grind to a halt trying to read every word of those 5,000 files.

ANDARTIS doesn't send the files to the AI. It sends a database query.

The Intent Blade parses your natural language query using a lightweight local model to determine what you are looking for.
It generates a clean SQL statement and hands it back to the PHP core.
The PHP core queries the isolated SQLite Node Core in sub-milliseconds, filtering out the exact rows and numbers for March.
Only that highly distilled, absolute truth is handed to the Senior Analyst (our local, quantized SLM) for a final synthesis pass.

The result? A fluid, hallucination-free answer delivered at the speed of a relational database, wrapped in the conversational grace of a modern language model.

Sovereignty on the Metal

It is easy to write a manifesto about taking power back from big tech. It is far harder to construct a cross-runtime, memory-guarded, zero-latency ingestion pipeline that functions flawlessly while you are completely offline in a cabin in the woods.

ANDARTIS is no longer just a conceptual chase for the Holy Grail of private intelligence. It is a working framework designed exclusively for macOS. It proves that we do not need to trade our privacy for utility, nor do we need a data center to achieve corporate-grade intelligence.

The architecture is set. The guardrails are built. The loom is yours to own.

Forge the wisdom. Keep it local. Never look back.

Docs: https://docs.andartis.it

FREYA Family — Sovereign Household AI

Sat, 25 Apr 2026 22:43:33 +0000

swdrffghrtj

FREYA Personal — Sovereign AI Assistant

Sat, 25 Apr 2026 18:00:00 +0000

Shipped my first product today after months of nights and weekends. FREYA is a personal AI assistant that runs entirely on the user's own machine — no cloud, no API calls out, no telemetry, no account required.
Technical:

Code-locked, cryptographically validated build
Local SQLite memory database (freya_memory.db) stays on the user's disk
Zero external calls at runtime — no heartbeat ping, no license server call-home
One-click install for non-technical users
Tested on hardware from mini PCs up to a 40GbE cluster

Pricing:

$49 one-time, Personal edition (one user, one machine)
Family / Small Business / Enterprise editions coming

Why I built it:
Big Tech can't sell a $49 one-time local AI without cannibalizing their $20/month subscription revenue. That gap is where small shops win. Built this for the people who want ChatGPT-style help without Big Tech reading every word they type.
Built in Idaho by Teton Data Systems LLC. Happy to answer technical questions about the architecture.

Optimize each location page with unique, locally-focused content, and consistent business details.

Sat, 09 Aug 2025 10:41:11 +0000

Location Page SEO in 2025: A Developer’s Guide to Local Visibility

In 2025, local search is evolving fast—“near me” results, Google’s Local Pack, and map-based searches drive urgency and foot traffic. A well-crafted location page isn’t just helpful—it’s mission-critical for any multi-location or neighborhood-targeted site.

What Makes a High-Impact Location Page?

Why This Works

Rank in Local Packs & Maps: Google uses local pages to fuel its Map Pack and local snippets, yielding high visibility and click-through rates.
Drive Local Conversions: Local intent searches often translate into offline visits—optimizing location pages improves your chance of conversion.
Scalable & Sustainable: With templates plus region-specific customizations, optimizing multiple locations becomes efficient without compromising SEO value.

Also Read, GEO and AEO redefine SEO for 2025 by structuring content to be cited—rather than just ranked—by AI!

Developer-Friendly Optimization Blueprint

Define your geo keywords and produce unique content per location.
Apply structured markup and embed maps.
Set localized metadata and clean, meaningful URLs.
Add local visual trust elements like reviews and testimonials.
Create FAQ sections tailored to each area.
Link your location pages from a central “Locate Us” hub with clear naming.
Test mobile performance and interaction—optimize load metrics and UX flows.

In Conclusion

Well-optimized location pages bridge your business with users actively looking nearby. They're your digital storefront—relevant, trust-building, and action-ready. Delve deeper into local SEO or need help creating scalable location templates for your brand? I’d be glad to assist.

Infinite Compute Glitch - Why Local AI Matters?

Thu, 27 Mar 2025 23:15:19 +0000

The Infinite Compute Glitch: Why Local AI Matters

In our headlong rush toward AI-powered everything, we've created an interesting paradox. Companies and individuals alike have begun to operate as if compute resources are infinite—just fire off another API call to OpenAI, Claude, or Gemini. But this mindset is creating a significant blind spot in how we approach technology, with consequences that extend far beyond our screens.

I wasn't just creating another tool when I was building Starlight, a desktop app that indexes and enables semantic search across your files using local models. I was making a statement about how we should approach AI in our daily lives.

Starlight lets you chat with your data and find information across your files without ever sending that data to external services. Everything happens on your machine, using your compute resources. This blog explores why apps like Starlight matter and why we should not ignore local AI.

The Infinite Compute Glitch

I've started calling our current approach to AI "the infinite compute glitch"—a collective delusion that we can just keep scaling up models and data centers without consequences.

Let's look at the hard numbers:

A single ChatGPT query consumes approximately 10-15x more energy than a Google search RW Digital
Training GPT-4 is estimated to have used over 62 million kilowatt-hours of electricity
AI data centers can use between 3-5 million gallons of water per day for cooling
The carbon footprint of training a single large language model can equal that of five cars over their entire lifetimes

A 2023 study from the University of Massachusetts found that training a single transformer-based AI model can emit as much carbon as five cars over their entire lifetimes. Meanwhile, Microsoft had to limit Azure AI services in some regions due to capacity constraints, showing that even the cloud giants are hitting physical limits.

Why Local First Matters

Your Data Stays Home

The most obvious benefit is privacy. When your files never leave your computer, you maintain complete control over your information. No terms of service changes, no worrying about how your data might be used to train future models.

Consider this: in 2023 alone, major AI companies updated their terms of service multiple times, often expanding their rights to use user data. Samsung even had to ban employees from using generative AI tools after sensitive code was uploaded to ChatGPT and became part of its training data. With local AI, these scenarios become impossible.

At ASML, we are not allowed to use AI tools because the company carries a lot of sensitive information that is also of national interest. I understand their concern, but when other less sensitive industries utilize AI to make their employees more productive, we feel stuck in the past era. This can easily be solved by empowering the employees with local AI, either on-device or on-prem.

Distributed Computing Is Resilient Computing

We're creating a more distributed system by leveraging the computing power already sitting on our desks and in our laps. Instead of centralizing all AI compute in massive data centers owned by a handful of companies, we're spreading the load across millions of devices.

The average modern laptop now has more computing power than what was used to train early versions of BERT, one of the breakthrough language models from 2018. A typical gaming PC with a decent GPU can run models with billions of parameters. We're sitting on a vast, untapped ocean of compute power.

Speed Without the Wait

For many tasks, local processing is actually faster. No upload time, no API latency, no waiting for your request to be queued behind thousands of others. The results appear as quickly as your computer can process them.

In our internal testing, Starlight's local search returned results from a 1GB document collection in under 200ms, while the equivalent cloud API call took over 2 seconds when accounting for network latency and server processing time. That's a 10x difference in speed for common queries.

Right-Sized Solutions

Here's something that might surprise you: more than 80% of the AI tasks the average person needs day-to-day can be handled by smaller, local models. We don't need GPT-4 to summarize a document or help draft an email. By matching the task to the appropriate model size, we save enormous resources.

Take document summarization: a 1.5 billion parameter model running locally can produce summaries nearly indistinguishable from those generated by models 100x larger for most business documents. The difference? The smaller model uses a fraction of the energy and requires no data transmission.

What is Propelling Local AI Forward

Here's what's making local AI happen:

1. Hardware: Increased Processing Power

It's pretty amazing how much more powerful our devices have become! Laptops, smartphones, and even smaller embedded systems now pack a serious punch with their processors and memory. This means they're not just for basic tasks anymore; they've got the muscle to handle sophisticated AI models directly. Think about it: your phone today has more computing power than entire computers from not too long ago. This increase in processing power is a huge factor in enabling local AI. With the recent release of ARM based processors on PCs, local AI has become even more plausible. The key advancements have been make CPU and GPU share an unified memory, allowing you to fit even larger models on the GPU and super efficient CPU/GPU. This is only going to improve going forward so we need to keep pushing AI on the edge to make use of this untapped compute power.

2. Model Architecture and Data

There is no doubt that the LLM models have become ever more intelligent. Every day a new, SOTA model is released. But even more important is that the models are getting smaller and more efficient. While companies like OpenAI are pushing toward ever more larger models which I believe is the wrong direction to move in, companies like Google, Mistral and Meta continue to surprise us with small models that are as powerful as the largest models from last year. For example Gemma3 27b, an open-source 27 billion parameter model from Google is as good as their own closed-source Gemini-1.5-pro which is believed to be over 300 billion parameters. We can see from the below plot that both Mistral Small 3.1 and Gemma-3 are better than GPT-4o-mini which must be a very large model at this point. Even though these small models are good, they can only work well on high end computers. The user has to settle for even smaller versions of these models if they have a medium spec computer. But this is a start none the less.

Several new pieces of innovations have made this possible. Introduction of KV caching, Flash Attention, Rotary Position Embedding (RoPE) are just some of the innovations that have truly reduced the gap between the performance of a small local model and a large cloud model. Also the quality of data that is used to train these models has improved over the years. Instead of training on just raw data from the internet which can be very noisy, a lot of clever preprocessing is being done to allow smaller models to learn just as well.

3. Model Serving: Open-Source Models and Tools

Lastly, the final piece is how are these local AI models distributed. Here, the open source community has been quite active and projects like Ollama, LlamaCPP, Candle, vLLM are making it easier to allow consumer grade hardware to run these open source models efficiently. These projects are the interface between the model weights and the the actual hardware. Making this interaction efficient is the key to making the best models run on the least amount of compute.

But just serving these models isn’t enough. Tools like ChatGPT provide several more user experiences like Canvas, Web Search, Voice to Text etc, which make people use their services. Even if we made best models, available locally, people will still gravitate towards, OpenAI, Claude etc for the user experience. That’s where products like Starlight come in. The new age software has to be as much local first as possible and allow users to get the same experience that they get on ChatGPT or other similar platforms using their own local AI models. At this point, it is more about breaking the habit. Let’s say I want to summarize a research paper. The default habit that has developed over the last three years for people is to take the paper and upload it to ChatGPT and ask summarization. But this is a very easy task for local models. So how can we push software to break this habit and exploit use cases where local AI can outperform cloud AI. This is going to be a big challenge which we at Starlight strive to solve by building features that make you gravitate towards local AI rather than cloud models for most of your tasks.

Innovating Within Constraints

At Starlight, we're embracing constraints rather than ignoring them. We're finding clever ways to make smaller models more effective through better indexing, retrieval, and context management.

For example, we've developed a hybrid approach that uses local embedding models to index documents and then employs efficient retrieval algorithms to find the most relevant content. This approach reduces the context window needed for generation tasks, allowing even small local models to produce high-quality results on par with much larger cloud models. Moreover, we are implementing methods like recursive summarization and graph traversal to compress large amounts of information into the constrained context of local models. Also, it's not just about context; even if the local models have a long context, usually, the hardware cannot process this context as it might not be able to fit on the memory or the computation takes just too long.

This approach forces us to be more thoughtful about how we use AI. Rather than throwing compute at every problem, we ask: What's the most efficient way to solve this? How can we deliver value while minimizing resource use?

A More Balanced Future: The Hybrid Approach

I'm not suggesting we abandon cloud AI entirely—there are absolutely use cases where larger models are necessary. But by being intentional about when and how we use these resources, we can create a more balanced, sustainable approach to AI.

Imagine a world where:

Your personal documents, emails, and notes are processed entirely on your devices, maintaining your privacy
Basic creative tasks like drafting emails or summarizing articles happen locally, with instant response times
Only specialized tasks that truly require massive models—like complex code generation or scientific research—call out to cloud services
Organizations maintain their own small, specialized models trained on their data, running on local infrastructure

This isn't science fiction—all the technology exists today. What's missing is the mindset shift.

Taking Action

So what can you do to be part of this shift?

Try local-first AI tools like Starlight for your personal and professional needs
Be mindful of your API usage when you do use cloud services
Support open-source AI projects that are making models more accessible and efficient

The future isn't about unlimited compute—it's about smart compute. It's about knowing when a local model is sufficient and when you truly need something more powerful.

So next time you're about to send that API call, ask yourself: Could this happen locally instead? Your privacy, your wallet, and our planet might thank you for it.