Run Ollama Locally: Self-Host an LLM in 13 Steps [2026]

Running a large language model on hardware you own, with no API key, no per-token invoice, and no terms-of-service surprise, has gone from a hobbyist curiosity to a mainstream developer habit. Ollama is the tool most people land on first, and as of October 2026 its GitHub releases page lists v0.40.1, shipped October 8, 2026, as the current build. This tutorial walks through the full setup on Windows, macOS, and Linux, covers GPU acceleration for NVIDIA, AMD, and Apple Silicon, shows how to run Ollama inside Docker, hooks it up to a Python app and a browser chat UI, and ends with a complete working project: a local chatbot script with persistent memory that never leaves your machine.

Everything below is built from Ollama’s own GitHub releases, official documentation, and model library pages, plus the pricing and decision-model documentation Ollama publishes. Where a detail (which plan name maps to which usage-credit tier, for instance) is not clearly published, this guide says so directly rather than guessing, because a wrong assumption about memory requirements is the fastest way to crash a machine mid-download.

·

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

What Is Ollama, and Why Self-Hosting an LLM Still Makes Sense in 2026

Ollama is a local runtime for large language models. It wraps the lower-level work of loading quantized model weights, managing a GPU or CPU inference backend, and exposing a clean REST API, so a developer can run ollama run llama3.1 and get a working chatbot without touching CUDA drivers or writing a single line of C++. The project’s core client and server have been open source since launch, and that has not changed with the latest release: the local software stays free, while Ollama layers a separate paid hosted option, Ollama Turbo, on top for people who want to run models too large for their own GPU.

The case for self-hosting in late 2026 is less about saving a few cents per request and more about control. A local model never sends your prompts to a third party, keeps working when your internet connection does not, and does not change its pricing or deprecate a model string on someone else’s schedule. The tradeoff is upfront: you need the hardware, you manage the updates, and you are responsible for picking a model that actually fits in the RAM or VRAM you have. For teams already running local GGUF-quantized models or experimenting with LangChain-style agent frameworks, Ollama is usually the fastest path from zero to a running local endpoint.

What makes this a good moment to set it up is the model selection available right now. The Ollama library currently lists Qwen3 in both dense and mixture-of-experts variants (including the flagship Qwen3-235B-A22B and the newer Qwen3.6:27B agentic-coding build), OpenAI’s open-weight gpt-oss:20b and gpt-oss:120b, Meta’s Llama 3.1 family up to 405B, DeepSeek-V3’s 671B mixture-of-experts model, NVIDIA’s Nemotron 3.5 Lightning (30 billion parameters, 3B active, built for agents that stay running), Google’s Gemma 3 and Gemma 4 (which became up to 90% faster in Ollama 0.31 on Apple Silicon with coding agents, as measured with the Aider polyglot benchmark), Mistral’s 0.3, Large 2, and Medium 3.5 releases, and smaller specialist models like GLM-OCR. That range covers everything from a 7B model that runs on a laptop to a 671B model that needs serious server hardware.

Prerequisites: Hardware, OS, and Software Versions You Need

Before installing anything, confirm your setup matches the following. This list is deliberately specific, because vague prerequisites are how people end up debugging a GPU driver issue three steps into a tutorial that assumed it already worked.

• Operating system: Windows 10 or 11 (64-bit), macOS 12 Monterey or later (Apple Silicon recommended for GPU acceleration via Metal), or a 64-bit Linux distribution (Ubuntu 22.04+, Debian 12+, Fedora 39+, or equivalent).
• Ollama version: v0.40.8 or later. Check Ollama’s GitHub releases page before installing, since point releases ship frequently.
• RAM: 8 GB minimum for small models (7B class), 16 GB for a comfortable experience with 13–14B models, 32 GB or more if you plan to run 32–34B models, and 64 GB or more for 70B-class models.
• Disk space: At least 20 GB free for your first model or two; gpt-oss:120b alone is roughly a 65 GB download, and model files accumulate quickly if you experiment.
• GPU (optional but recommended): An NVIDIA GPU with CUDA support, an AMD GPU with ROCm support, or Apple Silicon (M-series) for Metal acceleration. CPU-only inference works but is noticeably slower, especially above 13B parameters.
• Docker Desktop (optional): Latest stable release, only needed if you plan to run Ollama in a container rather than installing it natively.
• Python 3.10 or later: Needed for the API integration and the complete working project later in this guide.
• A terminal and basic command-line comfort: Every step below runs from a shell, whether that is PowerShell, Terminal, or a Linux shell of your choice.

The table below gives a practical memory budget per model size class at the default 4-bit quantization most Ollama models ship with. These are loading estimates, not performance guarantees; longer context windows and concurrent requests both add overhead on top of these numbers.

Model size	Approximate Q4 model size on disk	Practical RAM (CPU)	Practical VRAM (GPU)
7B (e.g. Qwen3:7b, Mistral 0.3)	~4–5 GB	8 GB	6–8 GB
13–14B (e.g. Qwen3:14b)	~8–10 GB	16 GB	12–16 GB
32–34B	~19–22 GB	32 GB	24 GB
70B (e.g. Llama 3.1:70b)	~40–45 GB	64 GB	48 GB

Step 1: Confirm Your Hardware Can Run the Model You Want

Match your available RAM or VRAM against the table above before pulling anything larger than 7B. On Windows or Linux with an NVIDIA GPU, run nvidia-smi in a terminal to confirm the driver is installed and see how much VRAM is free. On macOS, Activity Monitor’s Memory tab shows available RAM, and Apple Silicon Macs share memory between CPU and GPU, so your total system RAM is effectively your usable budget for GPU-accelerated inference.

# NVIDIA GPU check (Windows / Linux)
nvidia-smi

# Apple Silicon / macOS: check total memory
sysctl hw.memsize

# Linux: check total and available RAM
free -h

If nvidia-smi returns a command-not-found error, either the NVIDIA driver is missing or you do not have an NVIDIA GPU; either way, plan on CPU inference or an AMD ROCm path instead, and pick a smaller model class until GPU acceleration is confirmed working in Step 8.

Step 2: Install Ollama on Windows, macOS, or Linux

Ollama ships a native installer for all three major platforms, available from ollama.com/download. The commands differ slightly by OS, but the result is the same: an ollama binary on your PATH and a background service ready to serve models.

# macOS and Linux: one-line install script
curl -fsSL https://ollama.com/install.sh | sh

# Windows: download and run the installer from
# https://ollama.com/download/windows
# then confirm from PowerShell:
ollama --version

On Linux, the install script sets up a systemd service automatically on most distributions, so Ollama starts on boot without extra configuration. On macOS, the app installs to Applications and runs as a menu-bar process. On Windows, the installer registers Ollama as a background service that starts automatically after installation completes. In every case, the install finishes by confirming the service is listening on its default port, which you will use in the next step.

Step 3: Verify the Install and Check Your Version

Before pulling any models, confirm the installation actually succeeded and that you are running a current build. This takes ten seconds and saves a debugging session later if an old cached binary is quietly shadowing the new install.

ollama --version
# Expect something like: ollama version is 0.40.8 (or later)

ollama list
# Should return an empty table on a fresh install, not an error

If ollama --version returns a “command not found” error, the install did not add the binary to your shell’s PATH. On macOS and Linux, restart your terminal session first; if that does not fix it, check that /usr/local/bin (or wherever the installer placed the binary) is actually in your PATH variable. On Windows, a restart of PowerShell or a full reboot after installation resolves this in most cases.

Step 4: Pull and Run Your First Model

With Ollama installed, pulling a model downloads its weights from Ollama’s library and caches them locally. The first run of any model combines the pull and the chat session into one command, which is the fastest way to confirm everything works end to end.

# Pull a small model explicitly (good first test on modest hardware)
ollama pull llama3.1:8b

# Or pull and immediately start an interactive chat session
ollama run llama3.1:8b

Expect the first pull to take anywhere from thirty seconds to several minutes depending on your connection and the model size; an 8B model is roughly 4–5 GB at default quantization. Once the download finishes, ollama run drops you into an interactive prompt. Type a question, press Enter, and you should see tokens stream back in your terminal within a few seconds on most modern hardware. Type /bye to exit the session.

Step 5: Call the Ollama REST API Directly

The interactive chat is useful for a sanity check, but most real use cases talk to Ollama over its REST API, which listens on port 11434 by default at http://localhost:11434. Two endpoints cover almost everything: /api/generate for single-prompt completion, and /api/chat for multi-turn conversations with role-based messages.

# Single-turn generation, non-streaming
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain what a REST API is in two sentences.",
  "stream": false
}'

# Multi-turn chat with message history
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [
    {"role": "user", "content": "What is Ollama?"}
  ],
  "stream": false
}'

By default, stream is true, which returns a sequence of newline-delimited JSON objects as the model generates each token. Setting "stream": false, as shown above, waits for the full response and returns one JSON object, which is simpler to parse for a quick test or a batch job that does not need token-by-token output.

Step 6: Build a Custom Model With a Modelfile

A Modelfile lets you layer a custom system prompt and generation parameters on top of a base model, then save the result as a new named model you can run like any other. This is the mechanism behind every “specialized assistant” built on Ollama, and the syntax is deliberately minimal: three directives cover most use cases.

# Save as a file named "Modelfile" (no extension)
FROM llama3.1:8b

PARAMETER temperature 0.4
PARAMETER num_ctx 8192

SYSTEM """
You are a terse coding assistant. Answer with working code first,
a one-sentence explanation second. Avoid filler language.
"""

Create and run the custom model with two commands:

ollama create terse-coder -f Modelfile
ollama run terse-coder

FROM sets the base model or, for a fully custom model, a path to a local GGUF file, and GGUF support got broader when Ollama 0.30 launched in June 2026 with compatibility for LFM, Prism, and Unsloth models. PARAMETER sets runtime options such as temperature (randomness), num_ctx (context window size in tokens), and num_predict (maximum output length). SYSTEM defines the persistent instruction sent with every request, which is where most of the actual customization happens.

Step 7: Run Ollama Inside Docker With GPU Acceleration

A containerized setup is useful when you want Ollama isolated from your host system, or when deploying to a server that already standardizes on Docker. The official image is ollama/ollama on Docker Hub.

# CPU-only container
docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

# With NVIDIA GPU acceleration (requires NVIDIA Container Toolkit)
docker run -d \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

# Pull and run a model inside the running container
docker exec -it ollama ollama run llama3.1:8b

The -v ollama:/root/.ollama flag creates a named Docker volume so downloaded models persist across container restarts; without it, every model re-downloads whenever the container is recreated. The --gpus=all flag only works once the NVIDIA Container Toolkit is installed and Docker is configured to use it, which is the single most common source of “it works natively but not in Docker” confusion.

Step 8: Configure GPU Acceleration (NVIDIA CUDA, AMD ROCm, Apple Metal)

Ollama auto-detects available GPU acceleration on install and at startup, so in most cases there is nothing to configure manually beyond having the right driver installed. The specifics differ by platform:

• NVIDIA (Windows/Linux): Install the latest NVIDIA driver for your card first; Ollama bundles its own CUDA runtime, so a separate CUDA toolkit install is not required for native use. Confirm detection by running a model and watching GPU memory climb in nvidia-smi during inference.
• AMD (Linux primarily): Requires ROCm-compatible drivers installed at the OS level. Supported card generations vary by ROCm release, so check AMD’s ROCm compatibility list for your specific GPU before assuming acceleration will work out of the box.
• Apple Silicon (M-series Macs): Metal acceleration is used automatically; there is no separate driver to install. Every Apple Silicon Mac running a current macOS version gets GPU acceleration for free, which is part of why M-series Macs are popular for local LLM work despite lacking a discrete GPU in the traditional sense.

To confirm which backend Ollama is actually using, check the service logs right after starting a model. On Linux, journalctl -u ollama -f shows real-time logs; on macOS, Console.app or the terminal output from a manually launched ollama serve session shows the same information.

Step 9: Expose Ollama to Your Network and Change the Default Port

By default, Ollama binds to 127.0.0.1, meaning only the local machine can reach it. To access it from another device on your network, or to run it on a non-default port because something else already uses 11434, set the OLLAMA_HOST environment variable before starting the service.

# Linux/macOS: bind to all network interfaces on the default port
export OLLAMA_HOST=0.0.0.0:11434
ollama serve

# Run on a different port instead, e.g. if 11434 is already taken
export OLLAMA_HOST=127.0.0.1:11435
ollama serve

On Windows, set the same variable through System Properties > Environment Variables, or temporarily in PowerShell with $env:OLLAMA_HOST = "0.0.0.0:11434" before restarting the Ollama service. If you bind to 0.0.0.0, remember that anyone on your network segment can now reach the API with no authentication layer in front of it by default, so this is a setting for a trusted home or lab network, not a public-facing server without additional protection in front of it.

Step 10: Connect Ollama to a Python Application

Most real projects do not call the raw REST API directly; they go through a thin client library. Ollama’s official Python package wraps the same endpoints used in Step 5 behind a cleaner interface.

# Install the client library
pip install ollama

import ollama

response = ollama.chat(
    model="llama3.1:8b",
    messages=[
        {"role": "user", "content": "Summarize what Ollama does in one sentence."}
    ],
)

print(response["message"]["content"])

Run this and you should see a one-sentence summary print to your terminal within a few seconds, assuming the model is already pulled locally. If you get a connection error instead, confirm the Ollama service is actually running with ollama list in a separate terminal before debugging the Python side.

Step 11: Add a Browser Chat UI With Open WebUI

The terminal and curl commands are fine for testing, but a browser-based chat interface is what most people actually want day to day. Open WebUI is the most widely used front end for Ollama, and it runs as its own Docker container that talks to your existing Ollama instance over the network.

docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Once the container is running, open http://localhost:3000 in a browser, create a local account on first launch (this account only exists on your machine), and Open WebUI should auto-detect the Ollama instance running on your host. From there you get a ChatGPT-style interface, conversation history, and the ability to switch between every model you have pulled, all without sending a single token to an external server.

Step 12: Manage, Update, and Remove Installed Models

Models accumulate fast once you start experimenting, and a 70B model you pulled once to test can quietly eat 40+ GB of disk space indefinitely if you never clean up. A handful of commands cover the full lifecycle.

# List every model currently downloaded, with size
ollama list

# Pull an updated version of a model you already have
ollama pull llama3.1:8b

# Remove a model you no longer need
ollama rm llama3.1:8b

# Show detailed metadata for a specific model
ollama show llama3.1:8b

Re-running ollama pull on a model you already have checks for an updated version and only downloads the changed layers, similar to how Docker image pulls work, so it is far faster than the original download in most cases. Run ollama list periodically to audit disk usage, especially on a machine with limited storage.

Step 13: Monitor Resource Usage and Keep Ollama Updated

Once Ollama is part of your regular workflow, two habits prevent most long-term headaches: watching resource usage while a model runs, and keeping the Ollama binary itself current.

# See which models are currently loaded in memory and for how long
ollama ps

# macOS/Linux: reinstall to get the latest version
curl -fsSL https://ollama.com/install.sh | sh

# Windows: re-run the installer from ollama.com/download

ollama ps is the equivalent of docker ps for loaded models, showing which ones are currently resident in memory and the processor split between CPU and GPU. Idle models unload automatically after a default timeout to free up resources, which is worth knowing if you notice the first request after a break feels slower than the rest, since that is the model reloading rather than a real performance regression.

Choosing a Model for Your Hardware: Qwen3, Llama, DeepSeek, gpt-oss, and More

With setup complete, the real decision is which model to actually run day to day. The Ollama library’s current lineup spans a wide range of sizes and specialties, and picking the right one matters more for day-to-day usability than almost any other setting in this guide.

Model	Size / architecture	Good fit for
Llama 3.1	8B / 70B / 405B dense	General-purpose chat on modest hardware (8B) up to server-grade reasoning (405B)
Qwen3 / Qwen3.6	Dense and MoE, up to 235B-A22B flagship; Qwen3.6:27B for agentic coding	Multilingual tasks (100+ languages) and coding-agent workflows
gpt-oss	20B (~14 GB) / 120B (~65 GB), 128K context	OpenAI’s open-weight models for long-context, locally hosted reasoning
DeepSeek-V3	671B total, 37B active (MoE)	High-end reasoning on serious multi-GPU server hardware only
Gemma 3	Runs on a single GPU	Lightweight deployments where simplicity matters more than raw scale
Mistral (0.3 / Large 2 / Medium 3.5)	7B / 128K-context flagship / 128B	A European open-weight alternative across small and mid-size tiers

If you are brand new to local inference, start with an 8B-class model like llama3.1:8b or qwen3:8b. Both run comfortably on a mid-range laptop and give you a real feel for local-model latency and quality before investing in anything larger. Teams already comparing hosted models such as DeepSeek, Claude Haiku, and Gemini Flash often use a local Ollama model as a free baseline before deciding whether a paid API is worth the upgrade for a specific task.

Common Pitfalls When Self-Hosting an LLM With Ollama

Most first-week problems with a local LLM setup trace back to a short list of repeatable mistakes. These are the ones most likely to trip up a new Ollama install specifically.

• Picking a model too large for your RAM or VRAM. A 70B model will technically “run” on 16 GB of RAM through aggressive CPU offloading, but at a speed measured in seconds per token rather than tokens per second. Match the model to the hardware table above before judging Ollama’s performance.
• Forgetting that stream defaults to true. Code written against the API that expects one clean JSON object back will break on the default streaming response. Set "stream": false explicitly for non-streaming use cases, as shown in Step 5.
• Assuming Docker automatically sees your GPU. Without the NVIDIA Container Toolkit installed and configured, the --gpus=all flag silently falls back to CPU-only inference instead of throwing a clear error.
• Not checking disk space before a large pull. A 65 GB download for gpt-oss:120b that fails halfway through due to a full disk wastes significant bandwidth and time; check free space with df -h (Linux/macOS) or File Explorer’s drive properties (Windows) first.
• Running multiple large models simultaneously without planning for it. Each loaded model consumes its own memory allocation; ollama ps shows what is currently resident, and loading a second 14B model while a 32B model is already active can exhaust even 64 GB of RAM.
• Binding to 0.0.0.0 on an untrusted network. Exposing the API beyond localhost with no authentication in front of it means anyone on that network segment can send requests and consume your GPU, not just you.
• Treating a Modelfile’s SYSTEM prompt as a hard constraint. Smaller open-weight models follow system instructions less reliably than flagship hosted models; test actual behavior rather than assuming a Modelfile guarantees strict compliance.

Troubleshooting Guide: 9 Common Ollama Errors and Fixes

The following issues cover the errors most people hit in the first hour of running Ollama, pulled directly from Ollama’s documentation and common setup patterns.

Error / symptom	Likely cause	Fix
Port 11434 already in use	Another Ollama process, or a different app, is already bound to the default port	Check with lsof -i :11434 (macOS/Linux) or the Windows equivalent, stop the duplicate process, or set OLLAMA_HOST to a different port
“ollama: command not found”	Binary not on PATH, or terminal session predates the install	Restart your terminal; on Linux/macOS confirm the install directory is in PATH; on Windows, reboot after install
CUDA / GPU not detected	Missing or outdated NVIDIA driver, or unsupported GPU	Run nvidia-smi to confirm the driver works at all before blaming Ollama; update the driver if the command fails
Model pull times out or stalls	Slow or unstable connection, proxy/firewall interference, or low disk space	Retry the pull, check free disk space, and verify no corporate proxy is blocking the download
Out-of-memory / crash during inference	Model size exceeds available RAM/VRAM for the current context length	Switch to a smaller model, reduce num_ctx in a Modelfile, or close other memory-heavy applications
Docker container has no GPU access	NVIDIA Container Toolkit not installed, or Docker not configured to use it	Install the toolkit, restart the Docker daemon, and confirm with nvidia-smi run inside the container
WSL2 GPU passthrough fails on Windows	Outdated Windows build, WSL kernel, or NVIDIA WSL-compatible driver	Update Windows, run wsl --update, install the NVIDIA driver with WSL support, then wsl --shutdown and restart
“model not found” when calling the API	Model name typo, or the model was never pulled	Run ollama list to confirm the exact model tag, then ollama pull it if missing
API connection refused from another app	Ollama service not running, or bound only to localhost while the client is remote	Confirm the service is active with ollama ps, and set OLLAMA_HOST=0.0.0.0:11434 if remote access is intended

Advanced Tips: Quantization, Context Length, and Production Use

Once the basic setup works, a few adjustments make a real difference for anyone running Ollama as more than a weekend experiment.

Understand that quantization level trades accuracy for footprint. Most Ollama library tags default to 4-bit quantization, which is the sweet spot for most consumer hardware. If a model’s output quality feels noticeably worse than you expected from published benchmarks, check whether a less-aggressively-quantized tag (often labeled with a suffix like q8_0 or fp16) exists in the library and whether your hardware has room for it.

Right-size num_ctx for your actual use case. A larger context window consumes meaningfully more memory even before you fill it with content. If you are building a simple Q&A bot rather than a document-analysis tool, a smaller context window (2048–4096 tokens) frees up memory that a 128K window would otherwise reserve.

Consider Ollama Turbo for models that outgrow your hardware. Ollama’s hosted Turbo option runs larger models on managed infrastructure rather than your local GPU, intended for workloads that exceed what you can realistically self-host. Ollama’s pricing page lists a Team plan starting at $500 per month with unlimited users and $1,000 of usage credits included each month, shared across the team, and according to the v0.40.8 release notes, ollama launch can use usage credits without upgrading to a paid plan. Check ollama.com/turbo directly for the rest of the plan details, since that structure can change independently of the open source client covered in this tutorial.

Pin exact model tags in any script or deployment you rely on. Using a bare tag like llama3.1 resolves to a default variant that the library maintainers can change over time. Pinning to llama3.1:8b specifically, as used throughout this guide, keeps behavior predictable across machine reinstalls.

Benchmark on your own prompts, not published leaderboard numbers. Quantized local models behave differently from the full-precision versions used in most public benchmarks. Run a handful of representative prompts from your actual use case before committing to a specific model size for a real project.

Complete Working Project: A Local AI Chat Script With Memory

The script below ties together everything from this tutorial: the Python client from Step 10, multi-turn chat history, and basic error handling for a model that is not yet loaded. Save it as local_chat.py and run it from a terminal for a fully local, persistent-memory chatbot with zero external API calls.

import sys
import ollama

MODEL = "llama3.1:8b"

def main():
    print(f"Local chat with {MODEL}. Type 'exit' to quit.\n")
    history = [
        {"role": "system", "content": "You are a helpful, concise assistant."}
    ]

    while True:
        try:
            user_input = input("You: ").strip()
        except (EOFError, KeyboardInterrupt):
            print("\nExiting.")
            break

        if user_input.lower() in ("exit", "quit"):
            break
        if not user_input:
            continue

        history.append({"role": "user", "content": user_input})

        try:
            response = ollama.chat(model=MODEL, messages=history)
        except Exception as exc:
            print(f"Error talking to Ollama: {exc}")
            print("Check that the Ollama service is running (ollama ps) "
                  f"and that '{MODEL}' has been pulled (ollama pull {MODEL}).")
            history.pop()
            continue

        reply = response["message"]["content"]
        print(f"Assistant: {reply}\n")
        history.append({"role": "assistant", "content": reply})

if __name__ == "__main__":
    sys.exit(main())

Run it with python local_chat.py. The script keeps the full conversation history in memory for the duration of the session, so follow-up questions correctly reference earlier turns, the same way a hosted chatbot would. Swap MODEL for any tag you have pulled, including a custom Modelfile-based model from Step 6, to change the assistant’s behavior without touching the rest of the script.

Ollama vs Cloud AI APIs: When Self-Hosting Actually Saves Money

Self-hosting is not automatically cheaper than a hosted API, and pretending otherwise leads to bad decisions. The honest comparison depends heavily on usage volume, hardware you already own, and how much your time is worth.

Factor	Self-hosted via Ollama	Cloud API (e.g. hosted model providers)
Upfront cost	Zero software cost; hardware cost if you need to buy a GPU	Zero upfront; pay per token from the first request
Marginal cost per request	Effectively zero once hardware is owned (electricity aside)	Scales linearly with usage, can spike unpredictably
Data privacy	Prompts never leave your machine	Prompts sent to a third-party provider’s servers
Model quality ceiling	Capped by the largest open-weight model your hardware can run	Access to proprietary flagship models regardless of local hardware
Maintenance burden	You manage updates, drivers, and uptime	Provider manages all infrastructure

A reasonable rule of thumb: if you are already running hardware capable of a 14B-class model for other reasons (a gaming GPU, a workstation, an M-series Mac), self-hosting with Ollama is close to free and worth defaulting to for anything that does not need flagship-tier reasoning. If your workload genuinely needs a 400B+ parameter model’s reasoning quality and you do not own server-grade hardware, a hosted API, or Ollama Turbo’s managed infrastructure, will almost always be more practical than buying a multi-GPU rig to self-host it.

Frequently Asked Questions

Is Ollama free to use?

Yes. Ollama’s core client and server are open source and free to run locally on your own hardware. Ollama Turbo and its Pro, Max, and Team plans are a separate, paid, hosted layer for running models beyond what your local machine can handle; they do not affect the free local software covered throughout this guide.

What is the default Ollama API port?

Port 11434. The service listens at http://localhost:11434 by default, with /api/generate and /api/chat as the two primary endpoints, as shown in Step 5.

How much RAM do I need to run a 7B model with Ollama?

Around 8 GB of RAM for comfortable CPU inference, or 6–8 GB of VRAM if running on a GPU, based on the default 4-bit quantization most 7B models use in the Ollama library. Larger models scale up from there, as detailed in the prerequisites table above.

Can Ollama run on a GPU, or is it CPU-only?

Ollama supports GPU acceleration on NVIDIA GPUs via CUDA, AMD GPUs via ROCm, and Apple Silicon Macs via Metal, and auto-detects available acceleration at startup. CPU-only inference also works and is the automatic fallback when no supported GPU is detected, just at noticeably lower speed for larger models.

What models can I run with Ollama?

The Ollama library currently includes Llama 3.1 (8B to 405B), the Qwen3 and Qwen3.6 family, OpenAI’s open-weight gpt-oss:20b and gpt-oss:120b, DeepSeek-V3, NVIDIA’s Nemotron 3.5 Lightning, Gemma 3 and Gemma 4, and the Mistral 0.3, Large 2, and Medium 3.5 releases, among many others. Browse the full, constantly updated list at ollama.com/library.

How do I run Ollama in Docker with GPU support?

Install the NVIDIA Container Toolkit on the host first, then run the official ollama/ollama image with the --gpus=all flag, as shown in Step 7. Without the toolkit installed, the container silently falls back to CPU-only inference instead of throwing an obvious error, which is the most common point of confusion with this setup.

Why is my Ollama model running slowly?

The most common causes are a model size that exceeds comfortable VRAM for your GPU (forcing partial CPU offloading), running entirely on CPU for a model larger than 13B, or a context window set far larger than the conversation actually needs. Check ollama ps to see the current CPU/GPU memory split, and consider a smaller or more aggressively quantized model if performance does not meet your needs.

Does Ollama send my prompts to the cloud?

No, not when running models locally through the standard installation covered in this tutorial. Prompts and responses stay entirely on your machine. The exception is if you explicitly opt into Ollama Turbo, which routes requests to Ollama’s hosted infrastructure for models too large to run locally; that is a separate, optional feature and not the default behavior.