Coder Legion - Recent posts tagged ollama

What Is Context Size in Ollama? QWEN 3.8 Context Window Explained into Ollama

Sun, 18 Oct 2026 11:13:04 +0000

Confused about context size in Ollama? Wondering what an AI context window is, why it matters, and how to configure it for Qwen 3.8 27B?

In this beginner-friendly tutorial, you'll learn how context size works in local AI models, how to increase the context length in Ollama, and how to test different context sizes with real examples.

I'll walk you through the basics of AI context windows, tokens, num_ctx, memory usage, and performance so you can run your local AI models more effectively.

Whether you're just getting started with Ollama or experimenting with local LLMs, this video will help you understand context size and configure it with confidence.

🚀 What You'll Learn

✅ What is context size in Ollama?
✅ AI context window explained in simple terms
✅ Qwen 3.8 native context length vs. Ollama's configured context
✅ How to increase context size using Ollama
✅ How to test different context sizes with real examples
✅ How context size affects RAM, VRAM, memory usage, and performance
✅ How to check whether your configured context size is being used
✅ Tips for choosing the right context size for local AI models

What Is Context Size in Ollama? QWEN 3.8 Context Window Explained for Beginners

Sun, 18 Oct 2026 09:34:56 +0000

Confused about context size in Ollama? Wondering what an AI context window is, why it matters, and how to configure it for Qwen 3.8 27B?

In this beginner-friendly tutorial, you'll learn how context size works in local AI models, how to increase the context length in Ollama, and how to test different context sizes with real examples.

I'll walk you through the basics of AI context windows, tokens, num_ctx, memory usage, and performance so you can run your local AI models more effectively.

Whether you're just getting started with Ollama or experimenting with local LLMs, this video will help you understand context size and configure it with confidence.

🚀 What You'll Learn

# Debugging ZORGAX: We Captured the Tool-Calling Loop Between Open WebUI and Ollama

Sat, 10 Oct 2026 03:09:08 +0000

Debugging ZORGAX: We Captured the Tool-Calling Loop Between Open WebUI and Ollama

From 28 repeated function calls to a successful end-to-end test — and a new hypothesis about context-window limits.

Introduction

In my previous article, Building ZORGAX: Turning a Local LLM into a DevOps AI Agent, I described our effort to transform a local language model into a controlled infrastructure assistant for the MyZubster ecosystem.

ZORGAX runs on Ollama, uses a custom model based on Qwen 2.5 3B, and connects to Open WebUI inside a Docker-based environment.

Our objective is not to give AI unrestricted control over a server. We want an assistant capable of observing infrastructure, explaining problems, and eventually performing carefully authorized operations.

But we encountered a problem that made one thing clear:

An AI agent that knows how to call a tool must also know when to stop.

When ZORGAX called list_automations, the tool correctly returned an empty list. Yet the agent repeatedly called the same function instead of giving a final answer.

In earlier tests, we observed 28 repeated calls, followed by another run with 16 calls.

This article documents how we investigated that behavior, what we verified, and why the root cause is still an open research question.

1. The original problem

We asked ZORGAX to check active automations.

The tool returned:

{
  "automations": [],
  "total": 0
}

The expected answer was straightforward:

There are currently no active automations.

Instead, ZORGAX kept invoking list_automations.

For a read-only function, unnecessary repetition wastes time and computing resources.

For a future DevOps agent with permission to restart services, deploy software, or modify infrastructure, uncontrolled repetition could be much more serious.

We therefore decided to investigate the entire tool-calling workflow before enabling additional operational capabilities.

2. Testing ZORGAX directly through Ollama

Our first experiment bypassed Open WebUI.

We constructed a synthetic tool-calling conversation using Ollama's /api/chat endpoint.

The sequence contained a user request, an assistant function call, and a matching tool response containing the empty automation list.

ZORGAX completed the interaction successfully.

It generated a final answer and did not request another tool call.

This showed that the model could handle an empty tool response correctly.

However, the direct experiment used a simplified context. It was not identical to the original Open WebUI interaction, so it could not establish which component was responsible for the loop.

3. A suggestion from the developer community

After publishing the first article on Coder Legion, a commenter, lisarconnect, suggested capturing the actual request sent to Ollama immediately after the first tool response.

The key question was:

Does the next request preserve the assistant's tool call and the corresponding tool result?

This gave us a concrete investigation strategy.

Instead of guessing whether the problem came from Qwen, Ollama, or Open WebUI, we could inspect the integration boundary directly.

We also wanted to distinguish a successful empty result from a tool execution error.

Those two situations should not trigger the same behavior.

4. Inspecting Open WebUI's tool-call handling

We examined the installed Open WebUI source and identified the code responsible for continuing conversations after tool execution.

We then tested its actual convert_output_to_messages() function using a synthetic fixture.

The conversion preserved the expected structure:

[
  {
    "role": "assistant",
    "content": "",
    "tool_calls": [
      {
        "id": "zorgax-test-001",
        "type": "function",
        "function": {
          "name": "list_automations",
          "arguments": "{}"
        }
      }
    ]
  },
  {
    "role": "tool",
    "tool_call_id": "zorgax-test-001",
    "content": "{\"automations\":[],\"total\":0}"
  }
]

The assistant tool-call identifier matched the tool-result identifier.

The empty result was preserved.

This was an important finding, but it covered only the isolated message-conversion function.

We still needed evidence from the actual HTTP requests reaching Ollama.

5. Building a separate diagnostic environment

To avoid affecting the main MyZubster installation, we deployed a separate Open WebUI instance.

We also configured:

CHAT_RESPONSE_MAX_TOOL_CALL_ITERATIONS: "3"

In the Open WebUI build we inspected, the default tool-call iteration limit was 256.

Reducing that limit helped prevent another long-running diagnostic loop.

It was a safety measure, not a fix for the underlying behavior.

We kept operational tools disabled and used a harmless simulated automation function.

class Tools:
    def list_automations(self) -> dict:
        """Return a synthetic empty automation list."""
        return {
            "automations": [],
            "total": 0
        }

This function performs no infrastructure operation. It always returns the same predictable result.

6. Capturing the real Open WebUI → Ollama requests

We created a temporary HTTP proxy between the diagnostic Open WebUI instance and Ollama.

The proxy forwarded requests while recording structural information such as:

Model name
Available tool names
Message roles
Tool-call identifiers
Presence of the simulated empty result

We intentionally avoided logging complete private prompts and credentials.

One earlier captured request advertised approximately 35 built-in tools.

That was very different from our successful direct Ollama test, which exposed only list_automations.

We also reproduced the repeated-call behavior in the isolated Open WebUI environment, where it stopped after reaching the configured three-iteration limit.

At this stage, the number of tools became an interesting variable to investigate.

But correlation is not causation.

We had not yet demonstrated that having many available tools caused the repeated calls.

7. The successful end-to-end experiment

We created a diagnostic model configuration called ZORGAX Diagnostic and enabled only one harmless tool.

Then we asked:

Controlla quali automazioni sono attive usando list_automations. Se non ce ne sono, dimmelo.

This time, ZORGAX called the tool once and responded correctly:

Non ci sono automazioni attive.

The proxy captured the complete follow-up structure.

The first request exposed:

Tools: ['list_automations']
Messages: ['user']

The next request contained:

Tools: ['list_automations']
Messages: ['user', 'assistant', 'tool']
Empty automation result: True

This was the evidence we had been looking for.

Open WebUI successfully transmitted the assistant tool call and the empty tool result to Ollama, and ZORGAX produced a final answer.

We later repeated the successful single-tool experiment.

The proxy again confirmed that the tool result reached Ollama and that ZORGAX completed the task.

This does not solve the original loop, but it substantially narrows our investigation.

8. A second hypothesis: context-window truncation

Another Coder Legion commenter, Mike Dabydeen, suggested examining the model's available context window.

His hypothesis was that a conversation containing a system prompt, extensive tool definitions, and chat history might exceed the available context.

If older messages are truncated, the model might receive the latest tool result without sufficient context to understand the original request.

That could potentially explain repeated tool calls.

We investigated the Ollama configuration.

Our environment uses:

Ollama version: 0.32.14
Model: zorgax:latest

The ZORGAX Modelfile does not explicitly specify num_ctx.

The Ollama service also does not explicitly configure OLLAMA_CONTEXT_LENGTH.

After running a diagnostic inference, we queried the loaded model:

NAME             PROCESSOR    CONTEXT
zorgax:latest    100% CPU     4096

We confirmed that the successful single-tool test operated with a 4096-token context window.

We also updated our diagnostic proxy to record explicitly supplied values of num_ctx and num_predict.

The logging worked correctly.

For real requests generated by our diagnostic Open WebUI configuration, num_ctx was not explicitly present in the captured request options.

That is different from saying the model had no context limit. Ollama still allocated a context window, as confirmed by ollama ps.

What remains unverified

We have not yet captured evidence that Ollama actually truncated messages during the problematic 35-tool interaction.

We have also not completed a controlled comparison of the same tool workload at 4096 versus 8192 tokens.

Context truncation is therefore a plausible hypothesis, not our final diagnosis.

The next experiment must keep the model, system instructions, tool schemas, and user request as consistent as possible while changing one variable at a time.

9. What these experiments taught us

The investigation revealed three separate questions that matter for reliable AI agents.

First: Did the tool execute correctly?

An empty list can be a successful result. It must not automatically be treated as an error.

Second: Did the result reach the language model?

The conversation should preserve the relationship between the assistant's tool call and the tool's response.

Third: Did the model use the result correctly?

Even a correctly transmitted response does not guarantee that a model will stop calling tools.

All three stages need independent testing.

For infrastructure agents, safeguards must also exist outside the model:

Restricted tool permissions
Bounded execution and retries
Duplicate-call detection
Idempotent operations where applicable
Human authorization for consequential actions
Auditable execution results

A prompt instruction saying “do not repeat the tool call” is not a substitute for these controls.

10. Opening the investigation to contributors

We have published our diagnostic findings in the public MyZubster repository.

GitHub investigation issue:

https://github.com/MyZubster-Ecosystem/myzubster/issues/1577

Technical reproduction guide:

https://github.com/MyZubster-Ecosystem/myzubster/blob/main/docs/zorgax/TOOL-CALLING-INVESTIGATION-2026-10-09.md

Developers working with Ollama, Open WebUI, Qwen, Python, or local AI agents are invited to reproduce the experiments.

Useful contributions include controlled context-window comparisons, tool-schema experiments, sanitized HTTP request analysis, and regression tests for empty results and repeated calls.

Contributors can use their own systems. No access to the MyZubster production VPS or private infrastructure is required.

Conclusion

We started with an AI agent repeatedly invoking a function that had already returned a valid result.

We now have a working diagnostic environment, a trace of the actual tool-result continuation, and two successful end-to-end single-tool experiments.

We have also confirmed a 4096-token context allocation and identified a new hypothesis worth testing.

The root cause of the original loop remains open.

That is an important distinction.

We are not claiming that we have fixed a bug simply because one controlled configuration works.

We are building an evidence-based process to understand why the integration behaves differently under different conditions.

For ZORGAX, the objective remains unchanged: create a useful local DevOps assistant that operates within clear security boundaries and earns trust through reproducible tests.

The next milestone is a controlled 4096-versus-8192 context experiment with safe tool fixtures.

Thanks to lisarconnect and Mike Dabydeen for their technical suggestions and to everyone following the MyZubster development process.

The investigation continues in public.

Project: MyZubster Ecosystem
Agent: ZORGAX
Stack: Ollama, Qwen 2.5 3B, Open WebUI, Docker, Python, Ubuntu
Status: Experimental; root cause under investigation

Build in public. Test with evidence. Keep humans in control.

AI Privacy Is a Data-Flow Problem, Not a Model-Selection Problem

Wed, 30 Sep 2026 06:54:25 +0000

AI Privacy Is a Data-Flow Problem, Not a Model-Selection Problem
How a conversation with a skeptical programmer led us to redesign Zorgax around trust boundaries, fail-closed processing, local AI, controlled egress, and auditable decisions
A conversation with another programmer recently forced me to reconsider what we mean when we call an AI system “private.”
It wasn't a discussion about model benchmarks, context windows, or whether one LLM was better than another.
I was speaking with Nicola's father, a programmer who works in a company where the use of AI has to be considered carefully. Lawyers are involved in deciding what can be authorized because employees may handle personal information, internal company information, or other data that shouldn't casually be sent to an external AI provider.
His concern could be reduced to a very practical question:
What does the AI actually take, and where does that information go?

That question led us to inspect the AI architecture we're building around Zorgax in MyZubster.
And we found something worth writing about.
“We Use Ollama” Doesn't Prove Anything
Inside our AI routing logic, we had a provider identified as ollama.
That sounds straightforward:
OpenAI → external
Ollama → local

Except that's not what the complete data flow guaranteed.
When we traced the fallback path instead of trusting the provider label, we found that the non-OpenAI path could eventually reach a public AI gateway.
The architecture effectively allowed this:
User prompt

 │
 ▼

Web search

 │
 ▼

Model router

 │
 ├──── OpenAI
 │
 └──── "ollama"
            │
            ▼
      Public AI gateway

We also had a genuinely local Ollama integration elsewhere in the system.
So the problem wasn't Ollama.
The problem was architectural:
We were using a provider name as evidence of a privacy property.

But privacy doesn't care what a variable is called.
It cares where the bytes go.
Start With the Trust Boundary
That changed the question we wanted Zorgax to answer.
Instead of starting with:
Which model should process this request?

we now start with:
Is this data permitted to cross an external trust boundary?

Only after answering that should model selection happen.
Consider a prompt containing:
Customer: Emails are not allowed
Internal project: Aurora
API token: ...

The first decision shouldn't be whether GPT or another model would produce the best response.
The first decision is whether that payload is allowed to leave the machine at all.
That led us to introduce an explicit privacy policy layer.
Four Data Classes
We currently distinguish four classes:
PUBLIC
INTERNAL
CONFIDENTIAL
PII

Those feed into processing modes:
EXTERNAL_ALLOWED
LOCAL_ONLY
DENY

One design choice here matters more than it initially appears:
missing classification fails closed.
If Zorgax doesn't know what the data is, it doesn't assume that it is public.
It defaults toward:
INTERNAL

↓

LOCAL_ONLY

For an enterprise-oriented AI system, I think this is an important inversion.
Many systems effectively behave like this:
not known to be sensitive

probably safe to send

We wanted:
not known to be public

do not send externally

PUBLIC Is Not Permission
Classification and authorization are different concepts.
This distinction became one of the most important parts of the implementation.
Even if information is classified as:
PUBLIC

Zorgax still doesn't automatically get permission to send it to an external AI/search provider.
External processing requires another explicit condition:
externalProcessingAllowed === true

So the effective decision looks more like this:
PUBLIC
+
trusted external authorization
=
external processing may proceed

while:
PUBLIC
+
no external authorization
=
external processing denied

And:
INTERNAL / CONFIDENTIAL / PII
=
LOCAL_ONLY

Why separate the two?
Because:
“What kind of information is this?” and “Are we authorized to send it there?” are not the same question.

This also means we don't want a browser to simply submit something equivalent to:
{
"classification": "PUBLIC",
"externalProcessingAllowed": true
}

and grant itself permission.
We added tests around the public routes specifically to prevent client-controlled classification or authorization from silently becoming trusted server-side policy.
Similarly, having access to a paid feature is not treated as privacy authorization.
Entitlement and data governance are separate concerns.
Sensitive-Data Detection Gets a Veto
There is another problem with classifications: they can be wrong.
Imagine receiving:
classification = PUBLIC

alongside:
Please analyze Emails are not allowed

Or an API token.
Or private-key material.
The system shouldn't say:
“The caller said PUBLIC, so we're good.”

The policy layer therefore performs sensitive-data detection independently.
The current implementation recognizes patterns including:
email addresses
IPv4 addresses
Italian fiscal codes
IBANs
payment-card candidates
Bearer tokens
API keys
secret/password/token assignments
private-key material

If those detectors identify sensitive content, they can override a PUBLIC declaration.
In other words:
declared: PUBLIC
detected: PII

effective result: PII

External authorization doesn't override that either.
This creates a useful security invariant:
A declaration cannot downgrade detected sensitive information into externally processable data.

“Local” Now Has a Technical Definition
We also implemented a dedicated local AI path.
Zorgax can use a local Ollama endpoint, with the default targeting:
http://127.0.0.1:11434

But there's an important restriction.
For LOCAL_ONLY, we don't accept an arbitrary Ollama URL and assume it is local.
The current implementation restricts the endpoint to explicit loopback hosts:
127.0.0.1
localhost
::1

A server elsewhere on the private network may still be trustworthy in a particular deployment.
But it isn't the same trust boundary.
If we support that later, it should probably become an explicit policy concept rather than quietly redefining LOCAL_ONLY.
This is one of the broader lessons from the work:
Security properties become useful when their definitions are narrow enough to test.

Search Is Egress Too
AI privacy discussions often focus entirely on the LLM endpoint.
But consider this pipeline:
Sensitive prompt

  │
  ▼

Search provider

  │
  ▼

Local LLM

The final inference may be completely local.
The sensitive information may already have left the system.
Zorgax can interact with external search providers, so web research has to pass through the same privacy thinking.
We therefore treat external destinations such as:
OpenAI
public AI gateways
web-search providers

as egress boundaries.
That required changing the ordering of operations.
The dangerous mental model is:
prompt
↓
search
↓
choose model
↓
privacy somehow happens here

The architecture we want is:
prompt
↓
classification
↓
policy
↓
allowed destinations
↓
minimization
↓
search/model execution

Privacy has to happen before the first external request, not just before LLM inference.
Minimize Even Authorized Data
Once external processing is authorized, we still don't want to send more than necessary.
So we introduced an external-processing minimization step.
It can:
normalize input
redact recognized sensitive patterns
limit payload size
minimize history

before a permitted external call.
There is an important boundary here too.
Minimization is defense in depth.
It is not used to turn forbidden data into permitted data.
We don't want:
PII
↓
redact some pieces
↓
declare it safe
↓
external provider

Instead, the privacy decision is made against the original data.
Only an already-permitted external operation gets minimized before transmission.
Don't Forget Conversation History
An assistant request isn't just the latest message.
Suppose the conversation is:
User:
My customer's email is Emails are not allowed

User:
Can you summarize what I just told you?

The second message contains no PII by itself.
But sending the conversation history externally would.
So Zorgax evaluates the relevant history when determining the privacy mode.
Sensitive context in previous messages can therefore force the request into local processing.
This seems obvious once you see it.
It's also very easy to miss when privacy logic is implemented only around a message field.
The boundary needs to inspect what will actually be transmitted.
Audit the Decision, Not the Secret
Enterprise systems also need observability.
We introduced a privacy audit layer that can record properties such as:
classification
processing mode
destination
decision
reason
content digest
timestamp

But the audit record deliberately avoids storing the raw prompt or conversation history.
Otherwise you risk building this:
Privacy system:
"Don't send this sensitive prompt externally."

Audit system:
"Great, I'll make another copy of the entire sensitive prompt."

The local audit file is also created with restrictive filesystem permissions.
This is not yet a complete enterprise audit platform.
We still have work to do around centralized audit lifecycle, configurable retention, administrative access and provider allowlists.
But the principle is already useful:
Record enough to understand the privacy decision without unnecessarily reproducing the sensitive payload.

The Resulting Architecture
The new flow is conceptually closer to this:

                  USER
                    │
                    ▼
            ┌──────────────┐
            │ CLASSIFY DATA│
            └──────┬───────┘
                   │
                   ▼
            ┌──────────────┐
            │ POLICY ENGINE│
            └──────┬───────┘
                   │
      ┌────────────┼────────────┐
      │            │            │
      ▼            ▼            ▼
   PUBLIC       LOCAL_ONLY      DENY
      │            │             │
      ▼            │             X
Authorized?        │
  │      │          │
 NO     YES         │
  X      │          │
         ▼          ▼
    MINIMIZATION  LOOPBACK
         │         OLLAMA
         ▼
   EXTERNAL EGRESS
         │
  ┌──────┼────────┐
  ▼      ▼        ▼

OpenAI Search Public

                Gateway

          │
          ▼
    PRIVACY AUDIT

The most important box isn't Ollama.
It isn't OpenAI either.
It's the policy decision before either one.
Testing the Network Boundary
We didn't want to stop at unit tests for classification.
A test like this is useful:
PII → LOCAL_ONLY

But it doesn't demonstrate that another code path won't accidentally perform an external fetch.
So some of our runtime tests work from the opposite direction.
For a PII request, the mocked network layer rejects any request that isn't going to the expected loopback Ollama endpoint.
If a future regression tries to send that data to an external AI or search destination, the test fails.
We also test cases including:
PII in conversation history → local processing

remote OLLAMA_URL → rejected for LOCAL_ONLY

PUBLIC without explicit external authorization → external denied

PUBLIC containing detected secrets → local

client-controlled privacy authorization → not trusted

external payload → minimized

privacy audit → no raw prompt/history

Before integration, the combined Evidence Graph and privacy test gate completed:
19 test suites passed
97 tests passed
0 failed

Tests don't establish legal compliance.
They do give us evidence about specific technical invariants.
That's the level at which I think privacy engineering becomes much more concrete.
Deployment Had One More Lesson for Us
After testing and merging the architecture, we restarted the MyZubster backend.
It didn't come back.
The privacy code wasn't responsible.
A systemd drop-in expected a startup wrapper that was missing from the working tree.
We found the historical wrapper elsewhere on the server and compared it with the version originally committed to Git.
Their SHA-256 hashes matched exactly.
We restored it.
The application then exposed another pre-existing problem: two Italian strings in an authentication route contained ASCII apostrophes inside single-quoted JavaScript strings.
Something as innocent as:
l'automazione

was enough to stop Node from parsing the route.
After correcting those strings and verifying the syntax, we started the service again.
This time:
active (running)

Connected to MongoDB

MyZubster Gateway listening on 127.0.0.1:5003

Backend MongoDB initialized

It was an unrelated deployment issue, but it reinforced the exact principle behind the privacy redesign.
A repository saying something is correct isn't the final evidence.
A configuration saying something is local isn't the final evidence.
A provider being named ollama isn't the final evidence.
The runtime matters.
What This Does — and Doesn't — Mean
I want to be careful about one thing.
We're not calling this “GDPR compliant AI.”
Technical privacy controls are only part of the picture.
Real organizational compliance can involve legal basis, contracts, retention, governance, access control, processor relationships, organizational procedures and the details of the deployment itself.
What we've implemented is narrower and easier to verify:
Technical controls intended to prevent sensitive or unauthorized AI data from silently crossing an external trust boundary.

There is more to build.
But that's a property we can reason about.
And, importantly, test.
Back to the Conversation That Started It
The interesting thing about my conversation with Nicola's father is that he wasn't really asking whether AI was intelligent enough.
He was asking whether a company could understand and control what happened to its information.
That leads to a different set of questions for AI developers:
What information enters the AI system?

How is it classified?

Which information can leave the organization?

Which providers can receive it?

Who authorizes external processing?

Can a client grant itself that authorization?

What happens when sensitive data is detected?

Does conversation history count?

What exactly does "local" mean?

What is recorded in the audit trail?

Which test fails if the boundary is accidentally removed?

And is the expected architecture actually running?

Those questions aren't as exciting as announcing a new model integration.
But if AI is going to move deeper into companies, I suspect they're going to become some of the most important questions we ask.
What Changed for Us
At the beginning, we thought about privacy partly in terms of providers:
OpenAI = external
Ollama = local

Now we think about it in terms of data paths and trust boundaries.
That's a much stronger model.
Because the important question isn't:
Which AI are you using?

It's:
What path is this particular piece of information allowed to travel?

That's what we're now building into Zorgax.
And the rule we're taking forward is simple:
Don't trust the label.
Trace the data flow.
Enforce the boundary.
Test the egress.
Verify the runtime.

Qwen2.5 7B vs Qwen3 4B & 8B for Writing Correction: 60 Local Ollama Responses on Windows

Mon, 28 Sep 2026 04:48:41 +0000

I expected Qwen2.5 7B to retain a noticeable advantage over the smaller Qwen3 4B model for writing correction.

In this experiment, it didn't.

Across the same 20 paired writing cases, Qwen2.5 7B and Qwen3 4B produced exactly the same complete-case outcome: both succeeded on the same 18 cases and failed on the same 2.

The difference became much more noticeable in local execution time: in the tested cold-start setup, Qwen3 4B averaged 23.99 seconds, compared with 54.37 seconds for Qwen2.5 7B.

That raised another practical question:

Can a smaller model from a newer generation reproduce the practical writing-correction performance of a larger model from the previous generation?

This benchmark is a follow-up to my earlier Qwen3 4B vs 8B vs 14B writing-correction experiment.

In that first study, Qwen3 8B and 14B both achieved 19/20 complete cases, while the mean cold-start time for Qwen3 8B was roughly half that of Qwen3 14B.

This second experiment approaches the same model-selection question from another direction: instead of comparing sizes within Qwen3, it compares Qwen2.5 7B with Qwen3 4B and Qwen3 8B under the same paired writing-coach evaluation framework.

Benchmark setup

The benchmark uses 20 paired writing cases, including 16 sentences with expected errors and 4 already-correct controls.

Each case was evaluated with the same frozen reference and the same evaluation framework across three locally hosted Qwen models:

qwen2.5:7b
qwen3:4b-q4_K_M
qwen3:8b-q4_K_M

This produced 60 local responses in total.

The models were executed through Ollama on Windows under the same benchmark conditions. The objective was not to identify a universal “best model,” but to compare several dimensions separately:

complete correction of the expected errors;
expected-correction coverage;
localization of the error regions;
preservation of the original facts;
compliance with the requested output structure;
explanation-language and consistency checks;
local cold-start execution time.

For the primary metric discussed here, a case was considered complete only when all expected correction requirements for that case were satisfied without an incorrect replacement, false error, or configured fact loss.

Results

Complete-case correction

The primary correction results were close:

Qwen3 4B: 18/20 complete cases — 90%
Qwen2.5 7B: 18/20 complete cases — 90%
Qwen3 8B: 19/20 complete cases — 95%

On the primary metric alone, Qwen3 8B completed one additional case in this run.

The more distinctive result, however, was the paired comparison between Qwen2.5 7B and Qwen3 4B.

They did not merely finish with the same 18/20 score. They succeeded on the same 18 cases and failed on the same 2 cases.

Across all 20 paired cases:

both succeeded: 18/20
both failed: 2/20
different complete-case outcomes: 0/20

So, for this specific 20-case writing benchmark, moving from Qwen2.5 7B to Qwen3 4B preserved the same primary correction outcome case by case.

This should not be interpreted as evidence that the two models are universally equivalent. It means that no complete-case difference was observed between them within this controlled test set.

Local execution time

The difference became much larger when local cold-start execution time was considered.

In the tested configuration, the mean total cold-start times were:

Qwen3 4B: 23.99 s
Qwen2.5 7B: 54.37 s
Qwen3 8B: 60.24 s

Qwen3 4B therefore required less than half the mean cold-start time of Qwen2.5 7B in this run.

Expressed as a ratio, Qwen3 4B was approximately 2.27× faster than Qwen2.5 7B on this specific cold-start measurement, corresponding to about a 55.9% reduction in mean cold-start time.

This makes the paired correction result more practically interesting: Qwen3 4B preserved the same complete-case outcome as Qwen2.5 7B across all 20 cases while responding substantially faster in the tested local setup.

Qwen2.5 7B and Qwen3 8B, by contrast, were much closer in execution time. Qwen2.5 7B averaged 54.37 s versus 60.24 s for Qwen3 8B, making the 7B model about 9.7% faster on mean cold-start time in this run.

These timing results should be interpreted within the tested configuration. They describe cold-start execution time on this local Windows/Ollama setup, not a universal model-speed ranking.

Error localization and explanation checks

Complete-case correction does not capture every difference between the models.

Across the 21 expected error regions:

Qwen3 4B: 20/21 regions localized — 97.6% F1
Qwen2.5 7B: 20/21 regions localized — 97.6% F1
Qwen3 8B: 21/21 regions localized — 100% F1

Neither Qwen3 4B nor Qwen2.5 7B produced a false-positive error region in this run, while Qwen3 8B localized all 21 expected regions with no false positive.

Expected-correction coverage followed a similar pattern:

Qwen3 4B: 19/21 expected corrections — 90.5%
Qwen2.5 7B: 19/21 expected corrections — 90.5%
Qwen3 8B: 20/21 expected corrections — 95.2%

So Qwen2.5 7B and Qwen3 4B again matched on both correction coverage and error-localization F1, while Qwen3 8B retained a small advantage on these measures.

The explanation checks showed a more nuanced difference.

For explanation-language compliance:

Qwen2.5 7B: 40/40 evaluated fields passed
Qwen3 8B: 41/41 evaluated fields passed
Qwen3 4B: 37 passes and 3 warnings

All three models nevertheless produced parseable JSON, usable core output and full-contract-compliant output in all 20 cases.

The configured fact-preservation check also passed all 20 cases for every model.

Among the 16 error-containing cases, the action–explanation consistency check recorded 16 consistent results for each model, with no contradictory, ambiguous or missing entries.

These automatic checks should not be interpreted as a complete measure of pedagogical quality. They capture selected properties such as language compliance, output discipline and consistency, while deeper explanation quality would require additional linguistic or human evaluation.

What this means for local writing assistants

The main takeaway is not that one model is universally better than the others.

In this benchmark, Qwen2.5 7B and Qwen3 4B produced the same complete-case outcome on all 20 paired cases, while Qwen3 4B required substantially less cold-start time in the tested local setup.

Qwen3 8B, however, completed one additional case and reached 100% error-localization F1.

This creates a practical trade-off rather than a single winner.

For this specific workload:

Qwen3 4B offered the strongest efficiency result: the same paired complete-case outcome as Qwen2.5 7B, with much lower measured cold-start time.
Qwen2.5 7B remained a credible previous-generation baseline: it matched Qwen3 4B on complete cases, expected-correction coverage and error-localization F1, while showing cleaner explanation-language compliance in this run.
Qwen3 8B provided the strongest correction coverage: one additional complete case, one additional expected correction and full error localization.

This suggests that choosing a local writing model should be treated as a multi-criteria decision rather than as a simple parameter-count comparison.

For an interactive writing assistant, the relevant trade-off may involve correction reliability, explanation behavior, latency, model size and the available hardware.

In this run, Qwen3 8B provided a small measurable correction advantage, but that advantage came with substantially higher cold-start time than Qwen3 4B.

Limitations

This benchmark is exploratory and should be interpreted within its experimental scope.

Several limitations are important:

20 paired cases: the dataset is intentionally focused and is not large enough to support a universal ranking of the models.
One language scenario: the benchmark evaluates English writing correction with explanations requested in French. Results may differ for other languages or writing tasks.
One model family: the comparison is limited to Qwen2.5 7B, Qwen3 4B and Qwen3 8B and should not be generalized to unrelated local model families.
One local machine: execution time depends on hardware, background workload, model format, quantization and Ollama configuration.
Cold-start timing: the reported measurements describe this specific cold-start workflow rather than warm-start interaction latency.
Reference-based evaluation: outputs were compared against the same frozen, versioned correction reference and acceptance rules across all three models.
Automatic explanation checks: language compliance, output discipline and consistency were evaluated automatically, but these checks do not constitute a complete linguistic or pedagogical assessment.
No direct resource measurement: RAM, VRAM, CPU load and energy consumption were not measured in this run.

The last point is particularly important for interpreting the efficiency result.

Qwen3 4B has a substantially smaller nominal parameter count than Qwen2.5 7B, but parameter count alone should not be treated as a direct measurement of hardware-resource usage.

The benchmark therefore supports a measured latency comparison, not a claim that Qwen3 4B used a specific percentage less RAM, CPU or energy.

What I plan to test next

This second benchmark leaves several useful directions open for further evaluation.

The next iterations could extend the framework in four directions:

measure peak RAM and VRAM usage alongside correction quality;
compare cold-start and warm-start latency under the same local conditions;
extend the benchmark to a larger and more diverse writing test set;
compare additional local model families under the same evaluation protocol.

Direct resource measurements would be particularly useful.

In this run, Qwen3 4B showed a strong latency advantage over Qwen2.5 7B while preserving the same paired complete-case outcome, but memory and compute usage were not measured directly.

Adding RAM, VRAM and processor-load measurements would make it possible to evaluate the trade-off more completely across:

correction reliability;
explanation behavior;
latency;
memory footprint;
compute load;
and model size.

I am particularly interested in whether the efficiency pattern observed here remains stable when the task becomes more linguistically demanding and when resource usage is measured directly.

Conclusion

This experiment did not identify a universal winner, but it revealed a clear practical trade-off.

For this specific 20-case writing benchmark, Qwen3 4B and Qwen2.5 7B produced the same complete-case outcome on every paired case:

both completed 18/20 cases;
both applied 19/21 expected corrections;
both reached 97.6% error-localization F1.

The difference was much larger in local execution time.

Qwen3 4B averaged 23.99 seconds of total cold-start time, compared with 54.37 seconds for Qwen2.5 7B.

In this tested configuration, that corresponds to approximately 2.27× faster cold-start execution for Qwen3 4B while preserving the same primary paired correction outcome.

Qwen3 8B remained slightly stronger on correction coverage, completing 19/20 cases and reaching 100% error-localization F1, but with a mean cold-start time of 60.24 seconds.

The more useful question, therefore, may not be:

Which model has the most parameters?

but rather:

Which model provides the best balance between correction reliability, explanation behavior, responsiveness and available local hardware for the intended task?

For local interactive applications, model size alone is not enough to answer that question.

Earlier benchmark

Before this cross-generation comparison, I tested Qwen3 4B, 8B and 14B on the same 20 writing cases.

Read the earlier Qwen3 benchmark →

Disclosure: AI assistance was used to refine the wording and organization of this article. The benchmark results and measurements reported here come from the documented experimental runs and were reviewed for accuracy before publication.

Qwen3 4B vs 8B vs 14B for Writing Correction: 60 Local Ollama Responses on Windows

Tue, 08 Sep 2026 10:25:29 +0000

I expected the larger Qwen3 model to show a clear advantage for writing correction. In this experiment, it didn't.

Across 20 paired writing cases, Qwen3 8B and 14B both achieved 19/20 complete-case corrections, while Qwen3 4B reached 18/20.

The difference became much more noticeable in local execution time: in the tested cold-start setup, the mean cold-start time for Qwen3 8B was roughly half that of Qwen3 14B.

That raised a practical question:

For a local writing assistant, how much model size do we actually need?

To explore this more systematically, I built a reproducible benchmark comparing Qwen3 4B, 8B and 14B under the same local conditions with Ollama on Windows.

Benchmark setup

The benchmark uses 20 paired writing cases, including 16 sentences with expected errors and 4 already-correct controls.

Each case was evaluated with the same reference and the same evaluation framework across three locally hosted Qwen3 models:

qwen3:4b-q4_K_M
qwen3:8b-q4_K_M
qwen3:14b-q4_K_M

This produced 60 local responses in total.

The models were executed through Ollama on Windows under the same test conditions. The objective was not to identify a universal “best model,” but to compare several dimensions separately:

complete correction of the expected errors;
localization of the error regions;
preservation of the original meaning;
compliance with the requested output structure;
quality of the explanations;
local execution time.

For the primary metric discussed here, a case was considered complete only when the expected correction requirements for that case were satisfied.

Results

Complete-case correction

The primary correction results were close:

Qwen3 4B: 18/20 complete cases — 90%
Qwen3 8B: 19/20 complete cases — 95%
Qwen3 14B: 19/20 complete cases — 95%

On this metric alone, increasing the model size from 8B to 14B did not improve the number of complete cases in this test.

Local execution time

The difference was much larger in execution time.

In the tested cold-start configuration, the mean cold-start times were approximately:

Qwen3 4B: 26.6 s
Qwen3 8B: 60.0 s
Qwen3 14B: 119.4 s

So, while 8B and 14B achieved the same complete-case score, Qwen3 8B took about half as long as Qwen3 14B on average in this cold-start setup.

The 4B model was substantially faster again, while still completing 18 of the 20 cases.

This does not mean that 8B is universally better than 14B. It means that, for this specific writing-correction workload and hardware configuration, the additional model size did not translate into a better primary correction score, while the execution-time cost increased considerably.

Error localization and explanation quality

Complete-case correction and error localization measure different aspects of model behavior.

Across the 21 expected error regions:

Qwen3 4B missed one expected region, with an error-localization F1 of 97.6%.
Qwen3 8B localized all 21 expected regions, reaching 100% F1.
Qwen3 14B missed one expected region and reported one false-positive region, resulting in 95.2% F1.

This distinction matters for a writing coach: producing an acceptable corrected sentence does not necessarily mean that the model identified the underlying error with the same precision.

Explanation quality was therefore evaluated separately from correction success.

In this run, all 41 evaluated explanation fields produced by Qwen3 8B passed the language check. Qwen3 4B showed more explanation-language and terminology issues.

These automatic checks should not be interpreted as a complete measure of pedagogical quality. They cover selected language patterns and contradictions, while deeper explanation quality remains a separate area for linguistic review.

What this means for local writing assistants

The main takeaway is not that one model is universally better than another.

In this benchmark, the three Qwen3 models produced relatively similar complete-case correction results, but their local execution times differed much more substantially.

For this specific workload:

4B offered the lowest execution time, with a small reduction in correction coverage.
8B matched 14B on complete-case correction and achieved the strongest error-localization result in this run.
14B did not improve the primary correction score over 8B, while requiring substantially more execution time in the tested configuration.

This suggests that model selection for a local writing assistant should be treated as a multi-criteria decision, rather than assuming that a larger model will automatically provide the best practical experience.

Depending on the use case and available hardware, the relevant trade-off may involve:

correction reliability;
explanation quality;
latency;
memory and compute requirements;
and the expected frequency of use.

For a writing coach in particular, responsiveness also matters. A model that is slightly larger or theoretically more capable may not necessarily provide the best user experience if each interaction becomes significantly slower.

Limitations

This benchmark is exploratory and should be interpreted within its experimental scope.

Several limitations are important:

20 paired cases: the dataset is intentionally focused and is not large enough to support a universal ranking of the models.
One language scenario: the benchmark evaluates English writing correction with explanations requested in French. Results may differ for other languages or writing tasks.
One model family: only Qwen3 4B, 8B and 14B were compared. The results should not be generalized to other local model families.
One local machine: execution time depends on hardware, model quantization, available memory and Ollama configuration.
Cold-start timing: Ollama was configured with keep_alive=0, meaning the model was reloaded for each response. The reported timings therefore represent this specific cold-start workflow rather than warm-start interaction latency.
Reference-based evaluation: outputs were compared against a frozen, versioned correction reference prepared for this study. This improves consistency across models, but it does not capture every possible acceptable linguistic formulation.
Explanation-quality checks: automatic checks cover selected language patterns and contradictions. They should not be treated as a complete assessment of pedagogical or linguistic quality.

These limitations are also why the benchmark is presented as a controlled comparison rather than a claim about the absolute capabilities of each model.

What I plan to test next

This first benchmark leaves several useful directions open for further evaluation.

The next iterations could extend the framework in four directions:

compare additional local model families under the same evaluation protocol;
examine explanation quality with deeper linguistic and human review;
compare cold-start and warm-start performance;
evaluate the same workflow on more constrained hardware.

A larger and more diverse test set would also help determine whether the patterns observed here remain stable beyond these 20 cases.

I am particularly interested in whether the 8B/14B trade-off observed in this run persists when the task becomes more linguistically demanding.

Conclusion

This experiment did not produce a universal winner, but it did highlight an important practical point: model size alone is not a sufficient criterion for choosing a local writing model.

For this specific workload, Qwen3 8B matched 14B on complete-case correction while its mean cold-start time was roughly half that of 14B in the tested configuration.

The more useful question, therefore, may not be “Which model is the largest?” but rather:

Which model provides the best balance between correction quality, explanation quality, responsiveness, and available hardware for the intended use case?

If you use local LLMs for writing assistance or other interactive tasks, I would be interested to know which trade-off matters most in your own setup: quality, latency, memory requirements, or something else?

Follow-up benchmark

I later extended this evaluation by comparing Qwen2.5 7B with Qwen3 4B and Qwen3 8B under the same 20-case writing-coach framework.

In that follow-up, Qwen3 4B matched Qwen2.5 7B on every paired complete-case outcome while averaging 23.99 s of cold-start time versus 54.37 s for Qwen2.5 7B.

Read the follow-up benchmark →

Disclosure: AI assistance was used to refine the wording and organization of this article. The benchmark results and measurements reported here come from the documented experimental runs and were reviewed for accuracy before publication.

How I Connected a Dockerized Python API to a Local Mistral Model

Wed, 02 Sep 2026 17:04:05 +0000

How I Connected a Dockerized Python API to a Local Mistral Model

I’m currently building the AI foundation for MyZubster, and one of my first goals was to create a local development environment that could run without relying on paid external AI APIs.

The stack uses:

Python and Flask for the application API
Docker Compose for orchestration
Ollama for local model inference
Mistral as the language model
Qdrant for vector storage
Open WebUI for manual testing

This post explains how the services are connected and highlights a few issues I encountered along the way.

The architecture

The planned AI workflow is:

User request
    ↓
MyZubster Flask API
    ↓
Semantic search in Qdrant
    ↓
Relevant MyZubster data
    ↓
Mistral through Ollama
    ↓
Generated response

Qdrant will provide the retrieval part of the RAG pipeline, while Mistral will generate an answer using the retrieved context.

Why Ollama runs on the host

Ollama was already installed and running directly on Windows at:

http://localhost:11434

The mistral:latest model was also already downloaded.

I decided not to add another Ollama container because that would duplicate model storage, increase memory usage and conflict with the port already occupied by the host process.

Docker Desktop provides the special hostname:

host.docker.internal

Containers can use it to connect to services running on the host machine.

Inside the MyZubster API container, the Ollama address therefore becomes:

http://host.docker.internal:11434

Verifying the connection

Before changing the application, I tested the connection using a temporary curl container:

docker run --rm curlimages/curl:8.12.1 `
  http://host.docker.internal:11434/api/tags

Ollama returned the installed model:

{
  "models": [
    {
      "name": "mistral:latest",
      "parameter_size": "7.2B",
      "quantization_level": "Q4_K_M"
    }
  ]
}

This confirmed that Docker could communicate with Ollama on the Windows host.

Docker Compose stack

Here is the Compose configuration used for the local environment:

services:
  api:
    build:
      context: .
    init: true
    ports:
      - "5000:5000"
    environment:
      MYZUBSTER_HOST: "0.0.0.0"
      MYZUBSTER_PORT: "5000"
      MYZUBSTER_OBSERVATIONS_FILE: "/data/observations.json"
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
      OLLAMA_MODEL: "mistral:latest"
      QDRANT_URL: "http://qdrant:6333"
      QDRANT_COLLECTION: "myzubster"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - observations-data:/data
    depends_on:
      qdrant:
        condition: service_started
    restart: unless-stopped

  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "127.0.0.1:6333:6333"
      - "127.0.0.1:6334:6334"
    volumes:
      - qdrant-data:/qdrant/storage
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
      WEBUI_AUTH: "true"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui-data:/app/backend/data
    restart: unless-stopped

volumes:
  observations-data:
  qdrant-data:
  open-webui-data:

networks:
  default:
    name: myzubster-ai-network

The API communicates with Qdrant using its Compose service name:

http://qdrant:6333

It communicates with Ollama through the host gateway:

http://host.docker.internal:11434

Starting the environment

I first validate the Compose file:

docker compose config

Then I pull the external images and start the stack:

docker compose pull
docker compose down
docker compose up -d --build

The service status can be checked with:

docker compose ps

Useful logs are available through:

docker compose logs --tail 100 api
docker compose logs --tail 100 qdrant
docker compose logs --tail 100 open-webui

Testing Ollama from the application container

A successful request from Windows does not necessarily prove that the application container can reach Ollama.

I tested the full connection from inside the API container:

docker compose exec api python -c "import json,os,urllib.request; url=os.environ['OLLAMA_BASE_URL']+'/api/generate'; data=json.dumps({'model':os.environ['OLLAMA_MODEL'],'prompt':'Reply only with: MyZubster connected','stream':False}).encode(); req=urllib.request.Request(url,data=data,headers={'Content-Type':'application/json'}); result=json.loads(urllib.request.urlopen(req,timeout=120).read()); print(result['response'].strip())"

The container received a generated response, confirming that the networking and environment variables were configured correctly.

A PowerShell trap

One mistake I made was pasting YAML directly into PowerShell:

services:
  api:
    build:
      context: .

PowerShell interpreted each YAML key as a command and returned errors such as:

The term 'services:' is not recognized as the name of a cmdlet

A Compose configuration must be saved as docker-compose.yml; it cannot be executed directly.

To open the file in Visual Studio Code:

code docker-compose.yml

Always validate it afterward:

docker compose config

Available services

After startup, the local tools are available at:

MyZubster API: http://localhost:5000
Open WebUI: http://localhost:3000
Qdrant dashboard: http://localhost:6333/dashboard
Ollama API: http://localhost:11434

Qdrant and Open WebUI are bound to 127.0.0.1, so their ports are not exposed to other devices on the network.

What comes next

The containers provide the infrastructure, but the actual RAG pipeline still needs to be implemented.

The next development steps are:

Install an embedding model in Ollama.
Create a Qdrant collection with the correct vector dimensions.
Generate embeddings for MyZubster content.
Store vectors together with useful metadata.
Embed incoming user questions.
Retrieve the most relevant records from Qdrant.
Add the retrieved context to the Mistral prompt.
Expose the workflow through an API endpoint.
Add integration tests and response validation.

One possible endpoint will be:

POST /api/ai/ask

Example request:

{
  "question": "Which products use sustainable packaging?"
}

The application will search its own indexed data and ask Mistral to generate an answer grounded in those results.

Final thoughts

This setup creates a practical local AI development environment while keeping data and model inference on the developer’s machine.

The most important lesson was to verify each connection independently:

Host to Ollama
Docker to Ollama
API to Qdrant
Browser to Open WebUI

Once those connections work, debugging the application layer becomes much easier.

The MyZubster AI integration is still evolving, but the infrastructure is now ready for the next phase: building and testing the RAG workflow.

What local model and vector database are you using for your AI projects?

Docker #ArtificialIntelligence #Python #Ollama #Mistral #Qdrant #RAG

How to Use Qwen 3 with VS Code (FREE) | The Best QWEN 3 Setup with Ollama?

Mon, 03 Aug 2026 07:13:42 +0000

In this step-by-step tutorial, I'll show you how to Use Qwen 3 with VS Code using Ollama and turn your computer into a FREE local AI coding assistant for free.

Whether you're a beginner or an experienced developer, this guide will help you install everything from scratch, configure VS Code, fix common setup issues, and generate your first piece of code with QWEN 3.

🔥 In This Video:
✅ What is Qwen 3?
✅ Why use Qwen 3 for coding?
✅ Install Ollama
✅ Download and run Qwen 3 locally
✅ Connect Qwen 3 with VS Code
✅ Generate your first code
✅ Common setup mistakes and fixes

🎯 This is Part 1 of the series.

In Part 2, we'll use the same setup to build a complete React GitHub Profile Finder application using Qwen 3 step by step.

👇 Resources

• Ollama
https://ollama.com

• Qwen 3 Model
https://ollama.com/library/qwen3

💻 Tech Used
• VS Code
• Qwen 3
• Ollama

Join our growing community of learners and developers. Don't forget to like, comment, and subscribe to stay updated with the latest tutorials, tips, and projects.

Connect with us:

Subscribe: https://youtube.com/@TheCodeStreet
Join our Facebook Community: https://facebook.com/TheCodeStreet
Visit our website: https://thecodestreet.dev

Get vidIQ to grow your channel faster! 🚀
https://vidiq.com/TheCodeStreet

Together, let's code the future at CodeStreet! 💡

Image Generation with Ollama is back with Japanese, Korean and Chinese Languages Support!

Thu, 23 Apr 2026 05:55:01 +0000

An international collaborative work

Introduction

At the start of the year, I developed an image generation application using Bob to build a custom interface. The backend integrates the ‘x/flux2-klein’ and ‘x/z-image-turbo’ models alongside Ollama. Recently, the application received significant enhancements thanks to contributions from my colleague, Akira Onishi-san, in Japan.

What’s truly fantastic is the way my colleague transformed the app; by leveraging Bob, he made the architecture modular and much more secure. He also went the extra mile by refining the bash scripts, making the automatic startup and shutdown processes more robust than ever.

So let’s dig into the application’s enhancements.

Implementation

The architecture of the application remains the same, but the main enhanced modules are provided hereafter (the change log is accessible on GitHub)

app.js: The latest update to app.js introduces significant hardening and performance tracking. On the security front, we’ve eliminated vulnerabilities by replacing innerHTML with safe DOM manipulation and implementing filename sanitization to prevent directory traversal. Additionally, we’ve integrated memory usage tracking within the history logs and refined the IME composition handling to ensure a seamless experience for Japanese input.

// Application state
let generationHistory = [];

// DOM elements
const promptInput = document.getElementById('prompt');
const modelSelect = document.getElementById('model');
const generateBtn = document.getElementById('generateBtn');
const resultContainer = document.getElementById('result');
const statusDiv = document.getElementById('status');
const historyList = document.getElementById('historyList');
const btnText = generateBtn.querySelector('.btn-text');
const loader = generateBtn.querySelector('.loader');

// Check Ollama connection on load
checkOllamaConnection();

// Event listeners
generateBtn.addEventListener('click', generateImage);

// Check if Ollama is running
async function checkOllamaConnection() {
    statusDiv.textContent = 'Checking connection...';
    statusDiv.className = 'status checking';
    
    try {
        const response = await fetch('/api/health');
        const data = await response.json();
        
        if (data.status === 'ok') {
            statusDiv.textContent = '✓ Connected to Ollama';
            statusDiv.className = 'status connected';
        } else {
            statusDiv.textContent = '✗ Ollama not connected';
            statusDiv.className = 'status disconnected';
        }
    } catch (error) {
        statusDiv.textContent = '✗ Cannot connect to server';
        statusDiv.className = 'status disconnected';
    }
}

// Generate image function
async function generateImage() {
    const prompt = promptInput.value.trim();
    const model = modelSelect.value;
    
    if (!prompt) {
        alert('Please enter a prompt');
        return;
    }
    
    // Disable button and show loading state
    generateBtn.disabled = true;
    btnText.textContent = 'Generating...';
    loader.style.display = 'block';
    
    // Clear previous result
    resultContainer.innerHTML = '';
    const placeholder = document.createElement('div');
    placeholder.className = 'placeholder';
    const p = document.createElement('p');
    p.textContent = 'Generating your image...';
    placeholder.appendChild(p);
    resultContainer.appendChild(placeholder);
    
    try {
        const response = await fetch('/api/generate', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify({ prompt, model })
        });
        
        const data = await response.json();
        
        if (data.success) {
            displayResult(data, prompt, model);
            addToHistory(prompt, model, data.result, data.memoryUsage);
        } else {
            displayError(data.error || 'Failed to generate image');
        }
    } catch (error) {
        displayError('Network error: ' + error.message);
    } finally {
        // Re-enable button
        generateBtn.disabled = false;
        btnText.textContent = 'Generate Image';
        loader.style.display = 'none';
    }
}

// Display the result
function displayResult(data, prompt, model) {
    const resultContent = document.createElement('div');
    resultContent.className = 'result-content';
    
    // Check if result contains image data (base64)
    if (data.result && (data.result.startsWith('data:image') || data.result.includes('base64'))) {
        const img = document.createElement('img');
        img.src = data.result;
        img.alt = 'Generated image';
        img.className = 'result-image';
        resultContent.appendChild(img);
        
        // Add download button
        const downloadBtn = document.createElement('button');
        downloadBtn.className = 'download-btn';
        downloadBtn.textContent = '⬇️ Download Image';
        downloadBtn.onclick = () => downloadImage(data.result, prompt);
        resultContent.appendChild(downloadBtn);
    } else {
        // Display as text if not an image
        const textDiv = document.createElement('div');
        textDiv.className = 'result-text';
        textDiv.textContent = data.result;
        resultContent.appendChild(textDiv);
    }
    
    // Add info section
    const infoDiv = document.createElement('div');
    infoDiv.className = 'result-info';
    
    const modelLabel = document.createElement('strong');
    modelLabel.textContent = 'Model:';
    infoDiv.appendChild(modelLabel);
    infoDiv.appendChild(document.createTextNode(' ' + model));
    infoDiv.appendChild(document.createElement('br'));
    
    const promptLabel = document.createElement('strong');
    promptLabel.textContent = 'Prompt:';
    infoDiv.appendChild(promptLabel);
    infoDiv.appendChild(document.createTextNode(' ' + prompt));
    
    resultContent.appendChild(infoDiv);
    
    resultContainer.innerHTML = '';
    resultContainer.appendChild(resultContent);
}

// Download image with custom filename
function downloadImage(imageDataUrl, defaultPrompt) {
    // Generate a default filename from the prompt (sanitized)
    const sanitizedPrompt = defaultPrompt
        .toLowerCase()
        .replace(/[^a-z0-9\s]/g, '')
        .replace(/\s+/g, '-')
        .substring(0, 50);
    
    const timestamp = new Date().toISOString().split('T')[0];
    const defaultFilename = `${sanitizedPrompt}-${timestamp}.png`;
    
    // Prompt user for filename
    const filename = prompt('Enter filename for the image:', defaultFilename);
    
    // If user cancels, don't download
    if (!filename) return;
    
    // Remove all path separators to prevent directory traversal
    const sanitizedFilename = filename.replace(/[^a-zA-Z0-9._-]/g, '_');
    
    // Ensure filename has .png extension
    const finalFilename = sanitizedFilename.endsWith('.png') ? sanitizedFilename : `${sanitizedFilename}.png`;
    
    // Create a temporary link element and trigger download
    const link = document.createElement('a');
    link.href = imageDataUrl;
    link.download = finalFilename;
    document.body.appendChild(link);
    link.click();
    document.body.removeChild(link);
}

// Display error message
function displayError(message) {
    const errorDiv = document.createElement('div');
    errorDiv.className = 'error-message';
    
    const strong = document.createElement('strong');
    strong.textContent = 'Error:';
    
    const messageText = document.createTextNode(' ' + message);
    
    const br1 = document.createElement('br');
    const br2 = document.createElement('br');
    
    const small = document.createElement('small');
    small.textContent = 'Make sure Ollama is running and the models are installed.';
    
    errorDiv.appendChild(strong);
    errorDiv.appendChild(messageText);
    errorDiv.appendChild(br1);
    errorDiv.appendChild(br2);
    errorDiv.appendChild(small);
    
    resultContainer.innerHTML = '';
    resultContainer.appendChild(errorDiv);
}

// Add to history
function addToHistory(prompt, model, result, memoryUsage) {
    const historyItem = {
        prompt,
        model,
        result,
        timestamp: new Date().toLocaleString(),
        memoryUsage: memoryUsage || null
    };
    
    generationHistory.unshift(historyItem);
    
    // Keep only last 10 items
    if (generationHistory.length > 10) {
        generationHistory.pop();
    }
    
    updateHistoryDisplay();
}

// Update history display
function updateHistoryDisplay() {
    historyList.innerHTML = '';
    
    if (generationHistory.length === 0) {
        const emptyMessage = document.createElement('p');
        emptyMessage.style.color = '#9ca3af';
        emptyMessage.style.textAlign = 'center';
        emptyMessage.textContent = 'No generation history yet';
        historyList.appendChild(emptyMessage);
        return;
    }
    
    generationHistory.forEach((item, index) => {
        const historyItemDiv = document.createElement('div');
        historyItemDiv.className = 'history-item';
        
        const promptDiv = document.createElement('div');
        promptDiv.className = 'history-item-prompt';
        promptDiv.textContent = item.prompt;
        historyItemDiv.appendChild(promptDiv);
        
        const modelDiv = document.createElement('div');
        modelDiv.className = 'history-item-model';
        modelDiv.textContent = item.model;
        historyItemDiv.appendChild(modelDiv);
        
        const timeDiv = document.createElement('div');
        timeDiv.className = 'history-item-time';
        timeDiv.textContent = item.timestamp;
        historyItemDiv.appendChild(timeDiv);
        
        if (item.memoryUsage) {
            const memoryDiv = document.createElement('div');
            memoryDiv.className = 'history-item-memory';
            memoryDiv.textContent = `Memory: RSS +${item.memoryUsage.rssDiff}MB, Heap +${item.memoryUsage.heapUsedDiff}MB`;
            historyItemDiv.appendChild(memoryDiv);
        }
        
        historyItemDiv.addEventListener('click', () => {
            promptInput.value = item.prompt;
            modelSelect.value = item.model;
            displayResult({ result: item.result }, item.prompt, item.model);
        });
        
        historyList.appendChild(historyItemDiv);
    });
}

// Allow Enter key to submit (with Shift+Enter for new line)
// Ignore Enter key during Japanese IME composition
promptInput.addEventListener('keydown', (e) => {
    if (e.key === 'Enter' && !e.shiftKey && !e.isComposing) {
        e.preventDefault();
        generateImage();
    }
});

// Made with Bob

server.js: The server.js file has been completely refactored into a modular, well-documented architecture using helper functions. To improve observability, we integrated comprehensive memory tracking (monitoring RSS, heap, and external usage), alongside centralized configuration constants for easier maintenance. Finally, the service is now more resilient and easier to troubleshoot thanks to enhanced error handling, robust response parsing, and the addition of conditional debug logging.

const express = require('express');
const cors = require('cors');
const axios = require('axios');
const { exec } = require('child_process');
const util = require('util');
const path = require('path');

const execPromise = util.promisify(exec);

const app = express();
const PORT = process.env.PORT || 3000;
const OLLAMA_API_URL = 'http://localhost:11434';
const MAX_CONTENT_SIZE_BYTES = 50 * 1024 * 1024; // 50MB
const IMAGE_GENERATION_TIMEOUT_MS = 180000; // 3 minutes timeout for image generation
const OLLAMA_REQUEST_CONFIG = {
    timeout: IMAGE_GENERATION_TIMEOUT_MS,
    maxContentLength: MAX_CONTENT_SIZE_BYTES,
    maxBodyLength: MAX_CONTENT_SIZE_BYTES
};

app.use(cors());
app.use(express.json());
app.use(express.static('public'));

// ============================================================================
// Helper Functions
// ============================================================================

/**
 * Validate request body for image generation
 */
function validateRequest(body) {
    const { prompt, model } = body;
    if (!prompt || !model) {
        return { valid: false, error: 'Prompt and model are required' };
    }
    return { valid: true, prompt, model };
}

/**
 * Measure current memory usage
 */
function measureMemory() {
    return process.memoryUsage();
}

/**
 * Calculate and log memory usage difference
 */
function logMemoryUsage(memBefore, memAfter) {
    const memDiff = {
        rss: ((memAfter.rss - memBefore.rss) / 1024 / 1024).toFixed(2),
        heapTotal: ((memAfter.heapTotal - memBefore.heapTotal) / 1024 / 1024).toFixed(2),
        heapUsed: ((memAfter.heapUsed - memBefore.heapUsed) / 1024 / 1024).toFixed(2),
        external: ((memAfter.external - memBefore.external) / 1024 / 1024).toFixed(2)
    };
    
    debugLog('=== Memory Usage Before Generation ===');
    debugLog(`RSS: ${(memBefore.rss / 1024 / 1024).toFixed(2)} MB`);
    debugLog(`Heap Total: ${(memBefore.heapTotal / 1024 / 1024).toFixed(2)} MB`);
    debugLog(`Heap Used: ${(memBefore.heapUsed / 1024 / 1024).toFixed(2)} MB`);
    debugLog(`External: ${(memBefore.external / 1024 / 1024).toFixed(2)} MB`);
    
    debugLog('=== Memory Usage After Generation ===');
    debugLog(`RSS: ${(memAfter.rss / 1024 / 1024).toFixed(2)} MB (Δ ${memDiff.rss} MB)`);
    debugLog(`Heap Total: ${(memAfter.heapTotal / 1024 / 1024).toFixed(2)} MB (Δ ${memDiff.heapTotal} MB)`);
    debugLog(`Heap Used: ${(memAfter.heapUsed / 1024 / 1024).toFixed(2)} MB (Δ ${memDiff.heapUsed} MB)`);
    debugLog(`External: ${(memAfter.external / 1024 / 1024).toFixed(2)} MB (Δ ${memDiff.external} MB)`);
    
    return memDiff;
}

/**
 * Track memory usage for a generation operation
 */
function trackMemoryUsage(beforeMem) {
    const afterMem = measureMemory();
    return logMemoryUsage(beforeMem, afterMem);
}

/**
 * Conditional debug logging
 */
function debugLog(...args) {
    if (process.env.NODE_ENV === 'development' || process.env.DEBUG === 'true') {
        console.log(...args);
    }
}

/**
 * Extract base64 image data from various response formats
 */
function extractImageData(data) {
    if (data.image) return `data:image/png;base64,${data.image}`;
    if (data.images?.[0]) return `data:image/png;base64,${data.images[0]}`;
    if (Buffer.isBuffer(data)) return `data:image/png;base64,${data.toString('base64')}`;
    return null;
}

/**
 * Parse NDJSON string and return last line as JSON
 */
function parseNDJSON(data) {
    const lines = data.trim().split('\n');
    return JSON.parse(lines[lines.length - 1]);
}

/**
 * Handle string response (NDJSON format)
 */
function parseStringResponse(data, log) {
    log('Response is a string, length:', data.length);
    
    try {
        const parsed = parseNDJSON(data);
        log('Parsed response keys:', Object.keys(parsed));
        
        const imageData = extractImageData(parsed);
        if (imageData) {
            log('✓ Found image in parsed response');
            return { imageData, ollamaResponse: '' };
        }
        
        if (parsed.response) {
            log('Found response field, length:', parsed.response.length);
            return { imageData: null, ollamaResponse: parsed.response };
        }
    } catch (e) {
        log('Failed to parse as JSON:', e.message);
        return { imageData: null, ollamaResponse: data };
    }
    
    return { imageData: null, ollamaResponse: '' };
}

/**
 * Handle object/Buffer response
 */
function parseObjectResponse(data, log) {
    const imageData = extractImageData(data);
    if (imageData) {
        log('✓ Found image in response');
        return { imageData, ollamaResponse: '' };
    }
    
    if (data.response) {
        log('Response length:', data.response.length);
        return { imageData: null, ollamaResponse: data.response };
    }
    
    return { imageData: null, ollamaResponse: '' };
}

/**
 * Parse Ollama API response and extract image/text data
 *
 * CRITICAL: Ollama returns newline-delimited JSON (NDJSON) for image generation
 * Format: Each line is a separate JSON object
 * Example:
 * {"model":"x/flux2-klein:4b","created_at":"...","response":"","done":false}
 * {"model":"x/flux2-klein:4b","created_at":"...","response":"","done":false}
 * {"model":"x/flux2-klein:4b","created_at":"...","done":true,"image":"base64data..."}
 *
 * The LAST line contains the complete image in the 'image' field (singular, not 'images')
 */
function parseOllamaResponse(data) {
    debugLog('Ollama API response received');
    debugLog('Response data type:', typeof data);
    
    if (typeof data === 'object' && !Buffer.isBuffer(data)) {
        debugLog('Response data keys:', Object.keys(data));
    }
    
    if (typeof data === 'string') {
        return parseStringResponse(data, debugLog);
    }
    
    if (typeof data === 'object' || Buffer.isBuffer(data)) {
        return parseObjectResponse(data, debugLog);
    }
    
    debugLog('✗ No image or response data found');
    return { imageData: null, ollamaResponse: '' };
}

/**
 * Build API response object
 */
function buildResponse({ imageData, ollamaResponse, model, memDiff, responseData }) {
    return {
        success: true,
        result: imageData || ollamaResponse || 'No image generated',
        model,
        hasImage: !!imageData,
        memoryUsage: {
            rssDiff: memDiff.rss,
            heapUsedDiff: memDiff.heapUsed
        },
        debug: {
            responseType: typeof responseData,
            isBuffer: Buffer.isBuffer(responseData),
            responseLength: ollamaResponse.length,
            hasImages: !!(responseData.images)
        }
    };
}

/**
 * Generate image using Ollama API
 */
async function generateImageWithOllama(model, prompt) {
    return axios.post(
        `${OLLAMA_API_URL}/api/generate`,
        { model, prompt, stream: false },
        OLLAMA_REQUEST_CONFIG
    );
}

/**
 * Handle error responses
 */
function handleError(error, res) {
    console.error('Error generating image:', error.message);
    if (error.response?.data) {
        console.error('Error details:', error.response.data);
    }
    if (error.stack) {
        console.error('Stack trace:', error.stack);
    }
    res.status(500).json({
        error: 'Failed to generate image',
        details: error.message,
        response: error.response?.data || ''
    });
}

// ============================================================================
// API Endpoints
// ============================================================================

/**
 * Endpoint to generate image using Ollama HTTP API
 * This endpoint handles image generation requests from the frontend
 */
app.post('/api/generate', async (req, res) => {
    const { valid, error, prompt, model } = validateRequest(req.body);
    if (!valid) {
        return res.status(400).json({ error });
    }

    try {
        const memBefore = measureMemory();
        debugLog(`Generating image with model: ${model}`);

        const response = await generateImageWithOllama(model, prompt);
        const { imageData, ollamaResponse } = parseOllamaResponse(response.data);
        const memDiff = trackMemoryUsage(memBefore);
        
        res.json(buildResponse({
            imageData,
            ollamaResponse,
            model,
            memDiff,
            responseData: response.data
        }));

    } catch (error) {
        handleError(error, res);
    }
});

// Endpoint to check available mod

I Ported the Ollama Desktop App to Linux Just in Time for Ubuntu 26.04 LTS

Sat, 11 Apr 2026 22:50:53 +0000

Ubuntu 26.04 LTS drops this week. I've been running the RC for months. And I think it's going to be the release that finally pulls a meaningful chunk of people off Windows — especially AI-curious developers who are tired of the telemetry, the forced updates, and the general feeling that Microsoft is treating them like a product.

So I've been on a mission: take the cool AI tooling that still assumes Windows or macOS, and make it work natively on Linux.

This week's project: the official Ollama desktop app.

The Problem

Ollama is incredible. Local LLMs, dead simple, fast. But their desktop app — the polished GTK-style chat UI with model management, file attachments, settings, chat history — officially ships for Windows and macOS only.

Every single Go file in the app/ directory of ollama/ollama starts with:

//go:build windows || darwin

Linux users are left running ollama serve and pointing a browser at localhost:11434. It works, but it's not the experience.

The Discovery

Before writing a single line of code, I did what any good engineer does: I read the source.

The Ollama desktop app is built on webview — a lightweight C++ library that wraps the platform's native browser engine. On Windows it uses WebView2 (Edge). On macOS it uses WebKit. And on Linux?

// From webview.h, line ~400:
#if defined(WEBVIEW_GTK)
// Full GTK3 + WebKit2GTK implementation

The Linux support was already there. Fully implemented. Sitting in the header file, waiting. The Go build tags were the only thing blocking it.

The Port

Six new files. A patch. About 350 lines of Go.

1. Unlock the build tags

~40 files needed one line changed:

// before
//go:build windows || darwin

// after  
//go:build windows || darwin || linux

2. Wire up the CGo flags

app/webview/webview.go needed Linux-specific build flags to find GTK and WebKit:

#cgo linux CXXFLAGS: -DWEBVIEW_GTK -std=c++14
#cgo linux LDFLAGS: -ldl
#cgo linux pkg-config: gtk+-3.0 webkit2gtk-4.1

Note: Ubuntu 26.04 ships webkit2gtk-4.1 — not 4.0. If you're targeting older distros, check your version.

3. Platform files

The app architecture expects platform-specific implementations of a handful of interfaces. I wrote six _linux.go files:

app_linux.go — GTK window management via CGo, lockfile-based single-instance enforcement, signal cleanup:

func handleExistingInstance(startHidden bool) {
    f, err := os.OpenFile(lockPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
    if err != nil {
        slog.Info("another ollama-app instance detected, exiting")
        os.Exit(0)
    }
    f.Close()
    go func() {
        c := make(chan os.Signal, 1)
        signal.Notify(c, syscall.SIGINT, syscall.SIGTERM, syscall.SIGHUP)
        <-c
        removeLock()
    }()
}

server_linux.go — The app normally spawns the Ollama daemon itself. On Linux, systemd handles that. So this file is just a health-checker that pings localhost:11434 every 30 seconds and logs if it's unreachable. Never spawns a process.

updater_linux.go — All no-ops. systemd and apt manage updates on Linux.

filepicker_linux.go — File/folder dialogs via zenity:

func selectDirectory(title string) (string, error) {
    out, err := exec.Command("zenity", "--file-selection", "--directory", "--title="+title).Output()
    if err != nil {
        return "", err
    }
    return strings.TrimSpace(string(out)), nil
}

dialog/dlgs_linux.go — zenity implementations of the app's dialog interface (yes/no, info, error, file pickers).

webview_linux.go — The Webview struct without the Windows menu.h CGo baggage, plus all the JS bindings: ready, close, drag, doubleClick, zoomIn, zoomOut, selectFiles, selectWorkingDirectory.

4. The one frontend bug

The React sidebar had:

{isWindows && (
  <Link href="/settings">Settings</Link>
)}

On Linux, navigator.platform returns "Linux x86_64" — so isWindows is false and Settings just... doesn't appear. One-line fix:

const isMac = navigator.platform.toLowerCase().includes("mac");

{!isMac && (
  <Link href="/settings">Settings</Link>
)}

The Result

git clone https://github.com/johnohhh1/ollama-webchat-ubuntu.git
cd ollama-webchat-ubuntu
bash build.sh
sudo cp build/ollama-app /usr/local/bin/ollama-app
ollama-app

A native GTK3 window. WebKit2GTK rendering the full React SPA. Chat history in SQLite. File attachments. Model management. Settings. Dark mode. Wayland-native — no XWayland needed.

The NVIDIA gotcha

Running an RTX 5070 (Blackwell) with the proprietary driver? You'll see a wall of EGL warnings on startup:

libEGL warning: egl: failed to create dri2 screen
libEGL warning: pci id for fd 17: 10de:2f04, driver (null)

Ignore them. There's no open-source Mesa driver for Blackwell yet — WebKit probes Mesa EGL, fails, and falls back to Cairo software rendering for the compositor. For a chat UI it's completely imperceptible. Page layout and text were always CPU-rendered anyway.

Why Ubuntu 26.04 changes things

26.04 LTS is the first Ubuntu release with X11 removed entirely — Wayland only. GNOME 50. The desktop finally feels like a first-class target, not an afterthought.

At the same time, the local AI tooling ecosystem has exploded. Ollama, LM Studio, Open WebUI, llama.cpp — these tools are genuinely good, and they mostly work on Linux. The desktop app experience is just lagging behind.

That's the gap I'm trying to close. If you're a developer making the jump from Windows to Ubuntu 26.04 this week, you shouldn't have to give up your tools.

Repo

github.com/johnohhh1/ollama-webchat-ubuntu

PRs welcome — especially if you test on Fedora, Arch, or Ubuntu 24.04. And if you're building other AI tools that are stuck behind Windows/macOS build tags, the pattern here is reusable.

The C library usually already supports Linux. The build tags are almost always the only thing in the way.

My Experience Running Ollama Local Models

Mon, 24 Nov 2025 16:56:10 +0000

Over the past few weeks, I’ve been experimenting with Ollama to run local models on my machine.
Here’s what I discovered:

⚡ Performance & Speed

Lightweight models like Gemma 3B and Phi-3 Mini run surprisingly well even on constrained hardware.
Caching and modular setup helped me keep inference times low.

️ Workflow & Setup

Ollama’s model management makes it easy to swap and curate a lean library.
I trimmed unused models and focused on a 3-model setup under 2GB each for rapid prototyping.

Use Cases

Built a Streamlit chatbot powered by Gemma 3B:1 — smooth local inference with a polished UI.
Compared models side by side for reasoning depth vs. speed, which gave me practical insights into trade-offs.

Takeaways

Local deployment isn’t just about independence from cloud — it forces you to think strategically about efficiency.
Branding, UI polish, and repo structure matter just as much as raw performance when sharing projects.

I’d love to hear how others are balancing model variety vs. hardware constraints.
What’s your favourite local llm model ?

Build AI Workflows Visually, Fully Local & Private with Agentic Signal

Wed, 20 Aug 2025 17:19:22 +0000

I just launched Agentic Signal — a visual AI workflow builder that integrates tightly with Ollama. Everything runs locally and privately, so there’s no cloud dependency.

Key Features

Visual drag-and-drop workflow creation
Full streaming support with local LLMs (Ollama)
Tool calling and external service integration (email, calendar, cloud storage, browser automation)
Conversation memory and structured outputs (JSON schema validation)
Model management right from the UI

Tech Stack

React, TypeScript, Vite, Deno, GraphQL Yoga, React Flow, Material-UI (MUI), Playwright, Docusaurus, JSONata, Ajv, Zod, Bun, Chart.js, SCSS, OAuth 2.0, Google APIs

How To Run Ollama In Android (Without Root)

Thu, 07 Aug 2025 17:25:15 +0000

Yes, you can run Ollama directly on your Android device without needing root access, thanks to the Termux environment and its package manager. This turns your smartphone into a portable powerhouse for running local large language models (LLMs).

Read Full Post

Prerequisites

Before you start, you'll need two things:

A modern Android device. Performance will heavily depend on your phone's RAM and processor. A device with at least 8 GB of RAM is recommended for a smoother experience.
The Termux application. It's crucial to install it from F-Droid, as the version on the Google Play Store is outdated and no longer maintained.

Installation and Setup Guide

Follow these simple steps to get Ollama up and running.

1. Install and Update Termux

First, download and install Termux from the F-Droid app store. Once installed, open the app and update its core packages to ensure everything is current.

pkg update && pkg upgrade

You will be prompted several times during the update; it's generally safe to answer with the default option, which is often 'Y' or 'N'.

2. Install Ollama

With Termux up to date, installing Ollama is as simple as running a single command. The Ollama package is now available in the official Termux repositories.

pkg install ollama

This command will download and install the Ollama server and command-line interface (CLI) on your device.

3. Run the Ollama Server

Ollama operates as a background server that manages the models. You need to start this server before you can run and chat with an LLM. It's best practice to run this in its own dedicated Termux session.

Open Termux and start the server:

ollama serve

You'll see some log output, indicating the server is running. You need to keep this session open. You can use the termux-wake-lock command before this to prevent your phone from killing the process when it sleeps.

4. Download and Run a Model

Now, open a new Termux session by swiping from the left edge of the screen and tapping "NEW SESSION". In this new terminal, you can interact with the Ollama CLI.

To download and start chatting with a model (for example, Mistral), use the run command:

ollama run mistral

The first time you run this command, it will download the specified model, which might take some time and storage space. Subsequent runs will be much faster. Once downloaded, you can start chatting with the model directly in your terminal!

Important Tips

Model Size Matters: Mobile devices have limited resources. For better performance, start with smaller models like Phi-3 Mini, TinyLlama, or Gemma:2b. Running larger models like Llama 3 8B might be slow or crash if your phone doesn't have enough RAM.
Storage Space: LLMs are large files, often several gigabytes. Ensure you have enough free storage on your device before downloading models.
Keep it Running: Android's battery optimization can be aggressive. Use the termux-wake-lock command in the server session to prevent Termux from being shut down.

Using offline AI models for free in your Phyton scripts with Ollama

Wed, 05 Mar 2025 06:45:38 +0000

First published on Substack: https://andresalvareziglesias.substack.com/p/using-offline-ai-models-for-free

The Ollama project allows us to donwload and use AI models for offline usage with our computer resources. This allows us to experiment with the AI in our Python projects without any cost, an testing a lot of models to find the ideal choice for our project. It’s awesome.

Installation of Ollama

The installation of Ollama in a Linux device (for MacOS and Windows, check Ollama Github page) is very, very easy. Just write this command in a terminal:

curl -fsSL https://ollama.com/install.sh | sh

After a long wait, Ollama will be fully installed and configured.

Download a model

Once installed, we can download any model to our computer for offline usage. In the Ollama library page we can read the full list of available models.

For example, to download gemma2 with 2 billions of parameters, the command will be:

ollama pull gemma2:2b

The model will be downloaded to /usr/share/ollama/.ollama/models local folder, if you are curious (as I am).

Use the downloaded model in Python

Now, we can use the downloaded Gemma model as any other cloud model, like this:

from ollama import Client, ResponseError

try:
    client = Client(
        host='http://localhost:11434',
        headers={}
    )

    response = client.chat(
        model='gemma2:2b',
        messages=[{
            'role': 'user',
            'content': 'Describe why Ollama is useful',
        }]
    )

    print(response['message']['content'])

except ResponseError as e:
    print('Error:', e.error)

The program will output our requested answer. Wonderful!

A real example: check this article about Ollama… with Ollama

We can program a very simple article checker with Ollama and Python, like this

from ollama import Client, ResponseError

try:
    client = Client(
        host='http://localhost:11434',
        headers={}
    )

    prompt  = "I am an spanish writer that is learning how to "
    prompt += "write in english. Please, review if this article "
    prompt += "is well written. Thank you!\n\n"

    with open('article.md') as f:
        prompt += f.read()

    response = client.chat(
        model='gemma2:2b',
        messages=[{
            'role': 'user',
            'content': prompt,
        }]
    )

    print(response['message']['content'])

except ResponseError as e:
    print('Error:', e.error)

When executed, Gemma will give us a detailer analysis of this article, with advices for improvement.

Awesome! The possibilities are limitless!

A lot to learn, a lot of fun

Ollama allows us to test different models with our most precious data, without any privacy concern. Ollama allows to to save costs in the initial stages of development of an AI powered application.

And you? What kind of projects will you develop witn the help of Ollama?

Happy coding!

About the list

Among the Python and Docker posts, I will also write about other related topics, like:

Software architecture
Programming environments
Linux operating system
Etc.

If you found some interesting technology, programming language or whatever, please, let me know! I’m always open to learning something new!

About the author

I’m Andrés, a full-stack software developer based in Palma, on a personal journey to improve my coding skills. I’m also a self-published fantasy writer with four published novels to my name. Feel free to ask me anything!

Coder Legion - Recent posts tagged ollama

What Is Context Size in Ollama? QWEN 3.8 Context Window Explained into Ollama

What Is Context Size in Ollama? QWEN 3.8 Context Window Explained for Beginners

# Debugging ZORGAX: We Captured the Tool-Calling Loop Between Open WebUI and Ollama

Debugging ZORGAX: We Captured the Tool-Calling Loop Between Open WebUI and Ollama

Introduction

1. The original problem

2. Testing ZORGAX directly through Ollama

3. A suggestion from the developer community

4. Inspecting Open WebUI's tool-call handling

5. Building a separate diagnostic environment

6. Capturing the real Open WebUI → Ollama requests

7. The successful end-to-end experiment

8. A second hypothesis: context-window truncation

What remains unverified

9. What these experiments taught us

10. Opening the investigation to contributors

Conclusion

AI Privacy Is a Data-Flow Problem, Not a Model-Selection Problem

Qwen2.5 7B vs Qwen3 4B & 8B for Writing Correction: 60 Local Ollama Responses on Windows

Benchmark setup

Results

Complete-case correction

Local execution time

Error localization and explanation checks

What this means for local writing assistants

Limitations

What I plan to test next

Conclusion

Earlier benchmark

Qwen3 4B vs 8B vs 14B for Writing Correction: 60 Local Ollama Responses on Windows

Benchmark setup

Results

Complete-case correction

Local execution time

Error localization and explanation quality

What this means for local writing assistants

Limitations

What I plan to test next

Conclusion

Follow-up benchmark

How I Connected a Dockerized Python API to a Local Mistral Model

How I Connected a Dockerized Python API to a Local Mistral Model

The architecture

Why Ollama runs on the host

Verifying the connection

Docker Compose stack

Starting the environment

Testing Ollama from the application container

A PowerShell trap

Available services

What comes next

Final thoughts

Docker #ArtificialIntelligence #Python #Ollama #Mistral #Qdrant #RAG

How to Use Qwen 3 with VS Code (FREE) | The Best QWEN 3 Setup with Ollama?

Image Generation with Ollama is back with Japanese, Korean and Chinese Languages Support!

Introduction

Implementation

I Ported the Ollama Desktop App to Linux Just in Time for Ubuntu 26.04 LTS

The Problem

The Discovery

The Port

1. Unlock the build tags

2. Wire up the CGo flags

3. Platform files

4. The one frontend bug

The Result

The NVIDIA gotcha

Why Ubuntu 26.04 changes things

Repo

My Experience Running Ollama Local Models

Build AI Workflows Visually, Fully Local & Private with Agentic Signal

Key Features

Tech Stack

Links

How To Run Ollama In Android (Without Root)

Read Full Post

Prerequisites

Installation and Setup Guide

1. Install and Update Termux