How to Install Ollama: 12 Steps, 60 Min [2026]

Every prompt sent to ChatGPT, Claude, or Gemini leaves your machine, crosses the internet, and lands on somebody else’s server with a per-token invoice attached. Ollama flips that arrangement. Install a small background service, pull an open-weight model such as Llama 3.1, Mistral, Gemma, or DeepSeek Coder, and everything after that point runs on the hardware sitting in front of you: no API key, no usage meter, no network round trip once the model is on disk. The approach has caught on fast: Ollama’s user base grew from 3.5 million in May 2025 to 5.0 million by May 2026, according to Presenc AI, and by July 2026 the project had closed a $65 million Series B while reporting 8.9 million monthly active users, per Open Source For You.

This tutorial walks through installing Ollama on macOS, Linux, and Windows, verifying the install, pulling and running your first local model, wiring it into a REST API and a Python script, building a custom model with a Modelfile, containerizing the whole thing with Docker and GPU acceleration, and finishing with a complete, working retrieval-augmented generation (RAG) project that answers questions from your own documents without touching the internet. Budget around 60 minutes for a first pass through all 12 steps. If you skip the Docker and RAG sections, closer to 20 minutes gets you a working local chatbot.

This is for developers who want a coding assistant that still works on a plane, teams that can’t send proprietary source code to a third-party API, anyone tired of hitting rate limits mid-project, and the simply curious who want to know what a 2026-era open-weight model can do on hardware they already own. If you need frontier-level reasoning on the hardest research problems, cloud models still hold a real edge — this guide will show you exactly where that line currently sits.

·

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

What Is Ollama and Why Run AI Models Locally in 2026

Ollama is an open-source local AI runtime: one binary that handles downloading quantized model weights, serving them over a local HTTP API, and exposing a CLI you can chat through directly. Instead of juggling raw GGUF files and hand-rolled inference scripts, you run one install command, then ollama pull and ollama run for anything in its library. That library isn’t small anymore — the registry had grown to roughly 1,200 models by May 2026, up from about 950 in November 2025, according to Presenc AI, and the individual model pages on Ollama’s search index now list dozens of tags per family, filterable by category: newest, popular, thinking, vision, tools, and embeddings, depending on what you’re trying to build.

The case for running models locally instead of through an API comes down to four things: privacy (your prompts and documents never leave your machine), cost (no per-token billing once the download is done), latency and availability (no rate limits, no outage windows, works offline), and control (you can fine-tune behavior with a Modelfile instead of a system prompt you’re hoping the provider doesn’t silently override). None of that means local models have caught up to the frontier. An April 2026 model ranking put Gemini 3.1 Pro at the top of SWE-bench Verified at 77.8%, with GLM-5 leading the open-source pack at 1455.6.77.8% on the same benchmark and the top open-model Chatbot Arena Elo among open models at 1455.6. Meanwhile a mid-2026 search-enabled leaderboard tracking 744,152 votes across 31 models had claude-opus-4-6-search in first place at an Elo of 1252±5, priced at $5 per million input tokens and $25 per million output tokens. That price tag, multiplied across a team running thousands of requests a day, is exactly the bill a local model eliminates.

The open-weight side has been closing the gap fast. Llama 4 Scout now ships the largest context window of any open-weight model tracked in that same April 2026 ranking, at 10 million tokens, and Qwen3.5-397B-A17B alongside Mistral Large 3 were flagged as the strongest open-weight generalist options available. Realistically, none of those three are something you run on a laptop — they’re server-class models that need multiple data-center GPUs. This tutorial focuses on the sizes that actually fit on a desktop or a decent laptop: Llama 3.1 8B, Mistral 7B, Gemma 2 and Gemma 4, Qwen 2, DeepSeek Coder, and LLaVA for vision tasks. For a deeper look at how those smaller models stack up against each other, see our comparisons of Phi-4 Mini vs Gemma 3 vs Llama 3.2 and DeepSeek V4 vs GLM-5.2 vs Qwen.

Prerequisites: Hardware, OS, and Software Versions You’ll Need

Ollama itself is free and runs on modest hardware, but what you can comfortably run depends on RAM, disk space, and whether you have a supported GPU. Check the table below before you start pulling multi-gigabyte model files.

Requirement	Minimum	Recommended
Operating System	macOS 12, Windows 10, or a modern Linux distro	macOS 14+, Windows 11, Ubuntu 22.04+
RAM	8GB (runs small models like Gemma 2 2B or Qwen 2 0.5B only)	16-32GB for comfortable 7B-9B model use
Disk space	10GB free	50GB+ free if you plan to keep several model families installed
GPU (optional)	None — CPU inference works, just slower	Nvidia RTX with compute capability 5.0+, driver 550+; or Apple Silicon
Python (for API/RAG steps)	Python 3.9+	Python 3.11+
Docker (optional, Step 11)	Not required for basic use	Latest Docker Desktop or Docker Engine

On the GPU side, Ollama’s official hardware documentation is specific: Nvidia GPUs need “compute capability 5.0+ and driver version 550 and newer,” with cards in the 5.0-6.2 range needing driver 570 or newer. AMD GPUs need the ROCm v7 driver on Linux, or a ROCm v7/HIP7-capable driver stack on Windows. Apple Silicon Macs get GPU acceleration automatically through Metal — there’s no separate driver to install. Intel GPUs are supported through Vulkan on Linux. None of this is mandatory; Ollama runs fine on CPU-only hardware, it’s just noticeably slower per token, which matters more once you get to the RAG project in Step 12.

How much RAM does a given model actually need

As a rule of thumb, budget system RAM (or VRAM, if running on GPU) roughly equal to a model’s quantized download size, plus 2-3GB of headroom for the OS, the context buffer, and Ollama’s own runtime. A 4.9GB download like Llama 3.1 8B comfortably fits in 8GB of RAM with the model loaded alone, but 16GB gives you room to keep a browser and an IDE open at the same time without swapping to disk. We’ll cover exact download sizes for each model family later in this guide.

Step 1-3: Install Ollama on macOS, Linux, and Windows

Ollama ships native installers for all three major desktop platforms. Follow whichever section matches your machine — the CLI behaves identically once installed, regardless of which OS you’re on.

Step 1: Install on macOS

Download the macOS build from ollama.com/download, mount the resulting .dmg file, and drag the Ollama application into your system-wide Applications folder. On first launch, Ollama will ask permission to create a command-line symlink in /usr/local/bin — accept it, since every command in this tutorial assumes ollama is available in your terminal’s PATH.

Step 2: Install on Linux

The fastest path on any modern Linux distribution is the official install script:

curl -fsSL https://ollama.com/install.sh | sh

That one-liner is also exactly what Ollama’s own quickstart documentation tells you to run, and it’s stayed the documented path across releases — LocalAIMaster’s write-up of v0.32.1 in April 2026 still points to the same curl install.sh command, just as its notes on v0.30.0 a few weeks earlier walked through the equivalent curl-plus-chmod sequence for installing the raw Linux binary by hand. It’s worth pausing on that pattern for a second, though: a Hacker News commenter discussing Linux installers put it bluntly — “you’re essentially encouraged to just curl https://ollama.ai/install.sh | sh.” Piping a remote script straight into a shell as root is convenient, but it also means you’re trusting that URL completely at the moment you run it. If that bothers you, download the script first, read it, then execute it:

curl -fsSL https://ollama.com/install.sh -o install.sh
less install.sh
sh install.sh

If you’d rather install manually without the script, grab the platform-specific archive directly. For x86_64 systems:

curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr

ARM64 systems use the equivalent ollama-linux-arm64.tar.zst archive, and machines with an AMD GPU that want ROCm acceleration use ollama-linux-amd64-rocm.tar.zst instead. After a manual install, start the server once with ollama serve, or set it up as a proper systemd service so it survives reboots:

sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
sudo usermod -a -G ollama $(whoami)
sudo systemctl daemon-reload
sudo systemctl enable ollama
sudo systemctl start ollama

Step 3: Install on Windows

Download OllamaSetup.exe from ollama.com/download and run it — there’s no separate configuration screen, it installs and starts a background tray app automatically. If you need it on a non-default drive or path (common on machines where the system drive is a smaller SSD), pass the directory as an argument:

OllamaSetup.exe /DIR="d:somelocation"

A standalone ollama-windows-amd64.zip is also available if you’d rather skip the installer entirely, along with a ROCm-specific package for AMD GPU owners. Once installed, open PowerShell or Command Prompt to confirm the CLI is on your PATH before moving to Step 4.

Step 4: Verify Your Installation and Explore the CLI

Before pulling any models, confirm Ollama is actually installed and the background service is responding. Check the version first:

ollama -v

Example output:

ollama version is 0.x.x

Ollama version numbers move fast enough that quoting a specific one here would be stale within weeks — the project’s latest release was v0.5.12 back on February 26, 2025, the v0.32.x line stabilized with v0.32.3 on July 23, 2026, and by August 27, 2026 the GitHub repository had logged 248 total releases with v0.33.2 as the current build, per the official Ollama GitHub releases page and AI-Radar. Always trust what ollama -v reports on your own machine over any number printed in a tutorial. Next, list installed models (this will be empty on a fresh install):

ollama list

By default the Ollama server binds to 127.0.0.1 on port 11434 — local machine only, nothing exposed to your network. You can confirm the server is alive with a plain curl request:

curl http://localhost:11434

A healthy install returns a plain-text Ollama is running. If you get a connection-refused error instead, the background service isn’t running — on macOS and Windows, check that the Ollama tray app is actually open; on Linux, run ollama serve in a terminal (or sudo systemctl status ollama if you set it up as a service) to see the startup logs directly.

Step 5-6: Pull and Run Your First Local Model

With the install confirmed, pull a general-purpose model. Llama 3.1’s 8B tag is a solid first choice: capable, well-documented, and a 4.9GB download that fits comfortably on 16GB-RAM hardware.

ollama pull llama3.1

Example output:

pulling manifest
pulling 8934d96d3f08... 77.8% ▕████████████████▏ 4.9 GB
pulling 8c17c2ebb0ea... 77.8% ▕████████████████▏ 7.0 KB
pulling 7c23fb36d801... 77.8% ▕████████████████▏ 4.8 KB
verifying sha256 digest
writing manifest
success

Downloads are resumable — if your connection drops mid-pull, re-running the same command picks up where it left off rather than starting over. If you’d rather try a smaller or differently-tuned model first, a OneUptime engineering guide to local LLM development uses exactly this pattern to demonstrate pulling “ollama pull mistral” as its first example — Mistral’s 7B tag is a 4.4GB download and a common alternative default. Once a model finishes downloading, confirm it’s registered locally:

ollama list

NAME              ID              SIZE      MODIFIED
llama3.1:latest   a80c4f17acd5    4.9 GB    2 minutes ago

Now run it interactively:

ollama run llama3.1

This drops you into a chat prompt (>>>) where you can type directly. Ask it something and confirm you get a coherent response back — that’s your local model working end to end, with nothing leaving your machine.

Step 7: Chat With Models and Adjust Runtime Parameters

The interactive CLI supports a handful of slash commands worth knowing beyond just typing prompts. /bye exits the session. /show info prints the currently loaded model’s parameters and template. /set parameter num_ctx 8192 changes the context window for the rest of that session without needing a Modelfile. /save my-session and /load my-session let you checkpoint and resume a conversation.

Runtime behavior is controlled through parameters you can set per-session or bake into a custom model (Step 10). The two you’ll adjust most often are temperature, which controls how deterministic versus creative responses are (lower is more focused, higher is more varied), and num_ctx, the context window size in tokens — Ollama’s default context length is 4096 tokens unless you or the model’s Modelfile override it. For a quick one-off change inside a running session:

/set parameter temperature 0.3
/set parameter num_ctx 8192

Lower temperature values are worth using by default for coding and factual Q&A tasks, where you want the same input to produce a consistent, focused answer rather than a creative one.

Step 8: Call Ollama Through Its REST API

Every model you run through Ollama is also reachable over HTTP on port 11434, which is what makes it useful as a backend for real applications rather than just a chat toy. The core generation endpoint is /api/generate. Ollama’s own API documentation shows the pattern with this exact example:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Why is the sky blue?"
}'

By default this streams back a sequence of JSON objects, one per generated token chunk. For a single combined response instead of a stream, add "stream": false to the request body:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

Example (abbreviated) response:

{
  "model": "llama3.1",
  "response": "The sky appears blue because of a phenomenon called Rayleigh scattering...",
  "done": true,
  "total_duration": 4218421900,
  "eval_count": 132
}

There’s an equivalent /api/chat endpoint that takes a messages array instead of a single prompt string, structured the same way OpenAI-compatible chat APIs work — useful if you’re porting an existing chatbot integration over to a local backend instead of rewriting it from scratch.

Step 9: Use Ollama From Python With the Official Library

For anything beyond quick curl tests, the official ollama-python library is the cleaner path, and it’s far from the only ecosystem integration built on top of Ollama — Drupal’s community-maintained ai_provider_ollama module, for instance, was running on 439 sites as of August 2, 2026, with 393 of those already on the 1.2.x branch, according to Drupal.org’s own usage statistics. Install the Python library with pip:

pip install ollama

A basic chat call looks like this:

from ollama import chat

response = chat(model='llama3.1', messages=[
    {'role': 'user', 'content': 'Explain what a context window is in two sentences.'}
])
print(response['message']['content'])

For longer generations, streaming the response token-by-token keeps your application responsive instead of blocking until the whole answer is ready:

from ollama import chat

stream = chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'Write a short poem about local compute.'}],
    stream=True,
)
for chunk in stream:
    print(chunk['message']['content'], end='', flush=True)

This same library is what we’ll use to build the RAG project in Step 12, so keep it installed.

Step 10: Create a Custom Model With a Modelfile

A Modelfile lets you bake a system prompt, default parameters, and a base model together into one named model you can call with ollama run like anything else in the library — no need to repeat the same system prompt in every API call. Create a file named Modelfile with no extension:

FROM llama3.1

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

SYSTEM You are a senior code reviewer. Point out bugs, security issues, and unclear naming. Be direct and specific — cite line numbers when possible, and never pad your answer with unnecessary praise.

Build it into a named model with ollama create:

ollama create code-reviewer -f ./Modelfile

Then run it exactly like a library model:

ollama run code-reviewer

Every prompt in that session now inherits the lower temperature, the larger context window, and the system persona automatically, and the same code-reviewer name is callable from the REST API and Python library exactly like any pulled model.

Step 11: Run Ollama in Docker With GPU Acceleration

Running Ollama inside a container is useful for keeping it isolated from your host system, deploying it to a server, or running multiple versions side by side. The official ollama/ollama image handles CPU-only inference with a single command:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

If you have an Nvidia GPU with the NVIDIA Container Toolkit installed, add --gpus=all to pass it through to the container:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

The -v ollama:/root/.ollama volume mount is what keeps your downloaded models persistent across container restarts — skip it, and every docker restart throws away everything you pulled. Once the container is running, pull a model inside it the same way you would locally:

docker exec -it ollama ollama pull llama3.1

Because the container exposes the same port 11434 as a native install, every curl and Python example from Steps 8 and 9 works identically against a containerized Ollama — the only thing that changed is where the process is running.

Step 12: Build a Complete Local RAG Project From Scratch

This is the payoff step: a small, complete retrieval-augmented generation (RAG) project that answers questions from your own text files, fully offline. The idea is simple — split your documents into chunks, turn each chunk into a vector with an embedding model, store those vectors, then at query time find the chunks most similar to the question and hand them to a chat model as context.

First, pull a dedicated embedding model. nomic-embed-text is a 274MB download built specifically for this purpose, rather than repurposing a chat model:

ollama pull nomic-embed-text

Next, install the one extra Python dependency this project needs for similarity math:

pip install numpy

Create a folder called docs and drop a few .txt files into it — anything from personal notes to internal documentation works. Then save the following as local_rag.py in the parent directory:

import os
import numpy as np
from ollama import chat, embed

DOCS_DIR = "docs"
EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "llama3.1"
CHUNK_SIZE = 500

def load_chunks(docs_dir):
    chunks = []
    for filename in os.listdir(docs_dir):
        if not filename.endswith(".txt"):
            continue
        path = os.path.join(docs_dir, filename)
        with open(path, "r", encoding="utf-8") as f:
            text = f.read()
        for i in range(0, len(text), CHUNK_SIZE):
            chunk = text[i:i + CHUNK_SIZE].strip()
            if chunk:
                chunks.append({"source": filename, "text": chunk})
    return chunks

def embed_chunks(chunks):
    for chunk in chunks:
        result = embed(model=EMBED_MODEL, input=chunk["text"])
        chunk["vector"] = np.array(result["embeddings"][0])
    return chunks

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def retrieve(query, chunks, top_k=3):
    query_vector = np.array(embed(model=EMBED_MODEL, input=query)["embeddings"][0])
    scored = [
        (cosine_similarity(query_vector, c["vector"]), c)
        for c in chunks
    ]
    scored.sort(key=lambda x: x[0], reverse=True)
    return [c for _, c in scored[:top_k]]

def answer(query, chunks):
    top_chunks = retrieve(query, chunks)
    context = "nn".join(f"[{c['source']}]: {c['text']}" for c in top_chunks)
    prompt = f"""Answer the question using only the context below.
If the context doesn't contain the answer, say so directly.

Context:
{context}

Question: {query}"""
    response = chat(model=CHAT_MODEL, messages=[{"role": "user", "content": prompt}])
    return response["message"]["content"]

if __name__ == "__main__":
    print("Loading and embedding documents...")
    all_chunks = embed_chunks(load_chunks(DOCS_DIR))
    print(f"Indexed {len(all_chunks)} chunks from {DOCS_DIR}/")

    while True:
        query = input("nAsk a question (or 'quit'): ")
        if query.lower() == "quit":
            break
        print("n" + answer(query, all_chunks))

Run it with:

python local_rag.py

Example output, assuming your docs folder contains a project README and a couple of meeting-notes files:

Loading and embedding documents...
Indexed 47 chunks from docs/

Ask a question (or 'quit'): What database does this project use?

Based on the context provided, the project uses PostgreSQL 16 as its primary
database, with a connection pool managed through the config file described
in README.txt. The context does not mention any secondary databases.

Everything here — the embedding calls, the chat call, the documents themselves — stays on your machine. Swap CHAT_MODEL for a code-focused model, point DOCS_DIR at a folder of Markdown files instead of plain text, or replace the linear cosine-similarity scan with a proper vector database once your document set outgrows a few hundred chunks; the retrieval logic in retrieve() is the only piece that would need to change.

Choosing the Right Model for Your Hardware

Ollama’s library spans dozens of families, but for local, single-machine use, the parameter count and quantized download size matter more than which lab trained it. The table below uses the exact tag sizes currently listed for each model on Ollama’s own library pages.

Model	Tag	Download Size	Context Window	Best For
Llama 3.1	8b	4.9GB	128K	General-purpose default; strong all-rounder
Llama 3.1	70b	43GB	128K	Workstation/server-grade quality, needs 48GB+ RAM
Mistral	7b	4.4GB	32K	Fast, efficient general chat
Gemma 2	9b	5.4GB	8K	Balanced everyday use
Gemma 2	2b	1.6GB	8K	Low-RAM machines, edge devices
Gemma 4	12b	7.6GB	256K	Long-document analysis
Qwen 2	7b	4.4GB	—	Strong multilingual support
DeepSeek Coder	6.7b	3.8GB	16K	Code generation and review
LLaVA	7b	4.7GB	32K	Vision + text, image understanding
nomic-embed-text	latest	274MB	2K	Embeddings for RAG (Step 12)

Notice DeepSeek Coder’s footprint on Ollama’s own listing: it’s trained on two trillion code and natural-language tokens and has racked up 4.5 million downloads, yet the 6.7B tag is still a 3.8GB file that runs comfortably on an 8GB-RAM laptop. For image understanding, LLaVA’s 13b and 34b tags step up to 8GB and 20GB respectively, in exchange for noticeably better visual reasoning. If you’re deciding between local hardware options for running the larger tags, our Nvidia DGX Spark vs Mac Studio comparison covers unified-memory machines built specifically for this workload, and our open-source LLM roundup tracks how these smaller local models compare against the current open-weight leaders on standard benchmarks.

Ollama vs LM Studio vs llama.cpp: Picking the Right Local AI Tool

Ollama isn’t the only way to run open-weight models locally, and it isn’t reinventing the inference engine from scratch either — like most tools in this space, it builds on the same GGUF quantization format popularized by llama.cpp, an MIT-licensed project that has passed 122.6k GitHub stars and 21.3k forks. Where these tools differ is the layer on top of that engine.

Tool	Interface	License / Price	Best For
Ollama	CLI + REST API, minimal GUI	Free, MIT-licensed	Developers who want scriptable local models and a clean API
LM Studio	Full desktop GUI, plus a CLI (lms) and JS/Python SDKs	Free to download	Non-technical users who want a point-and-click chat interface
llama.cpp	CLI, OpenAI-compatible server, built-in web UI	Free, MIT-licensed	Maximum control over quantization and hardware backends

LM Studio is built on llama.cpp and Apple’s MLX under the hood, and by 2026 has expanded well past a simple chat window — it now offers local document editing, real-time voice transcription processed on-device, and agentic task automation, alongside optional cloud-model access with zero data retention for models it doesn’t run locally. If your priority is a polished GUI you don’t have to script against, that’s the stronger starting point. llama.cpp sits one layer lower: it’s the engine both Ollama and LM Studio lean on, and going straight to it gives you the widest hardware backend support of the three, including AVX-optimized CPU paths, CUDA, ROCm, Metal, SYCL, and Vulkan, plus quantization ranging from 1.5-bit to 8-bit. The tradeoff is that you’re working with more moving parts and less hand-holding than either Ollama or LM Studio provides. For most developers building an application rather than just chatting, Ollama’s balance of a simple CLI, a stable REST API, and first-class Python/JavaScript libraries is the more comfortable middle ground — which is why the rest of this tutorial is built around it.

Common Pitfalls, Troubleshooting, and Performance Tuning

Most problems people run into with a self-hosted LLM setup fall into a small, repeatable set of categories. Working through the pitfalls and the troubleshooting table below before assuming something is fundamentally broken will save real time.

Common pitfalls to avoid

• Pulling a model too large for your RAM. A 70B model at 43GB will technically download onto an 8GB-RAM laptop — it just won’t run without swapping to disk, which makes it feel frozen rather than slow.
• Forgetting the Docker volume mount. Skip -v ollama:/root/.ollama in Step 11 and every container restart silently deletes your pulled models, forcing a re-download.
• Assuming every model shares one context window. Gemma 2 defaults to an 8K window while Gemma 4 offers up to 256K — check each model’s actual context length rather than assuming it matches the last one you used.
• Running curl-piped installers on production systems without review. Fine for a personal laptop; on a shared or production server, download and inspect the script first, as covered in Step 2.
• Confusing embedding models with chat models. nomic-embed-text only produces vectors — trying to ollama run it for a conversation will not behave like a chat model.
• Leaving the default 4096-token context window in place for long documents. If your RAG context or long conversation is getting truncated, raise num_ctx explicitly rather than assuming the model will adapt automatically.

Troubleshooting common Ollama errors

Problem	Likely Cause	Fix
“connection refused” on localhost:11434	Background service isn’t running	Open the tray app (macOS/Windows) or run ollama serve manually (Linux) to see logs
ollama pull hangs or fails partway	Unstable connection or firewall blocking the download	Re-run the same pull command — downloads resume rather than restart
Model runs but responses are painfully slow	Running entirely on CPU with no GPU acceleration	Confirm GPU driver meets the minimums in the Prerequisites table, or use a smaller model tag
“model not found” when calling the API	Model name typo, or model was never pulled	Run ollama list to confirm the exact tag name before calling it
Out-of-memory crash mid-generation	Model plus context window exceeds available RAM/VRAM	Lower num_ctx, switch to a smaller quantized tag, or add RAM
Docker container loses all models after restart	No persistent volume was mounted	Re-run with -v ollama:/root/.ollama as shown in Step 11
GPU installed but Ollama still uses CPU	Driver version below Ollama’s minimum	Update to Nvidia driver 550+ (570+ for older compute-capability 5.0-6.2 cards) or the required ROCm v7 build
Custom Modelfile model ignores your SYSTEM prompt	Old cached build under the same name	Delete and rebuild with ollama rm then ollama create again
REST API call returns nothing until generation fully completes	Default streaming behavior wasn’t disabled or handled	Either read the streamed JSON lines as they arrive, or set "stream": false

Advanced tips: quantization, context windows, and performance

A few environment variables and parameters make a real difference once the basics are working. keep_alive controls how long a model stays loaded in memory after your last request — the official FAQ documents the default as 5 minutes, with options to pass a duration string like "10m", a number of seconds, -1 to keep a model loaded indefinitely, or 0 to unload it immediately after each response. For a machine running a single model all day, setting keep_alive: -1 avoids the reload delay every time it’s been idle for a few minutes.

Context length can be set three different ways: the environment variable OLLAMA_CONTEXT_LENGTH=8192 ollama serve, the interactive /set parameter num_ctx 4096 command, or as an API request option. OLLAMA_NUM_PARALLEL controls how many requests a single loaded model processes simultaneously (the default is 1), and OLLAMA_MAX_LOADED_MODELS controls how many different models can sit in memory at once — by default, three times the number of GPUs, or three for CPU-only inference. If you’re switching between a chat model and an embedding model constantly, as in the RAG project in Step 12, raising OLLAMA_MAX_LOADED_MODELS avoids repeatedly evicting one to load the other. Finally, OLLAMA_MODELS lets you point model storage at a different drive entirely, useful if your system disk is a small NVMe drive and you’re keeping several multi-gigabyte model families on hand — just make sure the target directory has the correct read/write permissions for the account Ollama runs as.

Updating Ollama, Removing Models, and Managing Disk Space

Multi-gigabyte model files add up quickly once you’ve pulled a handful of families to compare. Before you run low on disk space, it’s worth knowing how to clean up, update, and — if you ever need to — fully remove Ollama.

To delete a single model you no longer need, use ollama rm with the exact tag name from ollama list:

ollama rm llava:34b

That immediately frees the disk space the model was using, without touching anything else you’ve pulled. On macOS and Windows, Ollama’s tray app checks for updates automatically and prompts you to install them; there’s no separate update command to run yourself. On Linux, updating is just re-running the same install script or manual archive download from Step 2 — it overwrites the existing binary in place and picks up any new model-library features without disturbing models you’ve already pulled.

If you need to remove Ollama entirely, the process differs by platform. On macOS, quit the app from the tray icon and drag Ollama out of Applications to the Trash, then delete ~/.ollama to reclaim the space used by downloaded models. On Windows, uninstall through Settings → Apps → Installed apps like any other program. On Linux, the official uninstall sequence removes the service, the binary, and the dedicated user account it created during setup:

sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm -r $(which ollama | tr 'bin' 'lib')
sudo rm $(which ollama)
sudo userdel ollama
sudo groupdel ollama
sudo rm -r /usr/share/ollama

That last step, removing /usr/share/ollama, is the one that actually deletes your downloaded model weights — skip it if you’re planning to reinstall later and want to avoid re-downloading everything from scratch.

Frequently Asked Questions

Is Ollama free to use?
Yes. Ollama itself is free and open-source, and every model in its library is a free download. The only cost is your own electricity and hardware — there’s no subscription or per-token charge for local inference.

Do I need a GPU to run Ollama?
No. Ollama runs on CPU-only hardware without any extra configuration. A GPU speeds up generation substantially, but it’s an optimization, not a requirement, especially for smaller model tags in the 2B-9B range.

What’s the difference between Ollama, LM Studio, and llama.cpp?
llama.cpp is the underlying inference engine most of this ecosystem builds on. Ollama wraps it in a simple CLI and REST API aimed at developers. LM Studio wraps a similar stack in a full desktop GUI aimed at people who’d rather click than script.

Can I use Ollama models without an internet connection?
Yes, once a model is downloaded. You only need internet access to pull new models; every chat, API call, and the RAG project in Step 12 run entirely offline afterward.

How much RAM do I actually need to run a local LLM comfortably?
16GB is a comfortable baseline for 7B-9B models like Llama 3.1 8B or Mistral 7B. 8GB works for the smallest tags (Gemma 2 2B, Qwen 2 0.5B). Anything in the 70B range realistically needs 48GB or more.

Can Ollama run multiple models at the same time?
Yes, up to the limit set by OLLAMA_MAX_LOADED_MODELS, which defaults to three times your GPU count (or three, on CPU-only systems). This is exactly what the RAG project in Step 12 relies on, running a chat model and an embedding model side by side.

Is a local model as good as ChatGPT or Claude?
Not on the hardest reasoning benchmarks — cloud frontier models still lead there as of 2026. For everyday coding help, writing, summarization, and question-answering over your own documents, a well-chosen local model in the 7B-9B range is close enough for most day-to-day work, at zero marginal cost per request.

Does Ollama send any of my data back to a server?
Model downloads and updates require network access. Once a model is running, prompts, responses, and any documents you feed it through a project like the one in Step 12 stay entirely local unless you explicitly call an external API from your own code.