[Tutorial] Mastering Host-Memory Prompt Caching in llama-server

[mind-tercero-roldan]

Tutorial: Mastering Host-Memory Prompt Caching in llama-server 🚀

Author: mind-tercero-roldan
Date: March 15, 2026
Related: Issue #16391 (PR merged), Issue #13523 (tutorials list)
Difficulty: Intermediate
Read time: ~10 minutes

---

📋 Table of Contents

1. What is Host-Memory Prompt Caching?
2. How It Works: The Architecture
3. When to Use It (and When Not To)
4. Setup: Enabling the Feature
5. Configuration Parameters Explained
6. Real-World Benchmark: Before vs After
7. Advanced: Multi-Slot Caching Strategy
8. Troubleshooting & Best Practices
9. Summary & Key Takeaways

---

🔍 What is Host-Memory Prompt Caching?

Host-memory prompt caching is a performance optimization feature in llama-server that stores pre-computed prompt representations in system RAM (host memory) instead of GPU VRAM. This allows the server to:

• ✅ Skip reprocessing identical or similar prompts
• ✅ Reduce Time to First Token (TTFT) for repeated queries
• ✅ Share prefix computations across multiple slots
• ✅ Scale better with limited VRAM by offloading cache to RAM

🎯 The Problem It Solves

Imagine you're running a chatbot with a large system prompt (e.g., 10K tokens of context, rules, examples). Every new user request requires:

1. Processing the full system prompt again
2. Computing KV cache from scratch
3. Wasting GPU cycles on work already done

Host-memory caching lets you compute the system prompt once, store it in RAM, and hot-swap it into the GPU context when needed.

---

⚙️ How It Works: The Architecture

┌─────────────────────────────────────────────────────────┐
│                    llama-server                         │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  ┌──────────────┐     ┌──────────────────────────┐     │
│  │   Slot 1     │     │   Slot 2                 │     │
│  │  (Active)    │     │   (Waiting)              │     │
│  │              │     │                          │     │
│  │  [GPU VRAM]  │◄────┤  Prefix cached in RAM    │     │
│  │  KV Cache    │     │  (hot-swappable)         │     │
│  └──────────────┘     └──────────────────────────┘     │
│                                                          │
│  ┌─────────────────────────────────────────────────┐   │
│  │  HOST MEMORY PROMPT CACHE                       │   │
│  │  ┌─────────────┬─────────────┬──────────────┐  │   │
│  │  │ System      │ User A      │ User B       │  │   │
│  │  │ Prompt      │ History     │ History      │  │   │
│  │  │ (10K tokens)│ (5K tokens) │ (3K tokens)  │  │   │
│  │  └─────────────┴─────────────┴──────────────┘  │   │
│  │  ↑                                               │   │
│  │  │ Computed once, reused many times             │   │
│  └─────────────────────────────────────────────────┘   │
│                                                          │
└─────────────────────────────────────────────────────────┘

The Hot-Swap Mechanism

1. First request with a prefix → computed on GPU, saved to RAM
2. Similar request arrives → llama-server calculates prefix similarity
3. Above threshold? → hot-swap cached prefix into GPU context
4. Result: Skip processing → jump straight to generation

---

🎯 When to Use It (and When Not To)

✅ Use It When:

Scenario	Benefit
Large system prompts (>5K tokens)	50-90% TTFT reduction
Multi-user chatbot with shared context	Prefix computed once, reused N times
RAG applications with common document chunks	Cache document embeddings
Limited VRAM (24GB or less)	Offload cache to abundant RAM
High-throughput API with repeated queries	Maximize GPU utilization

❌ Don't Use It When:

Scenario	Reason
Unique prompts every time	No cache hits = wasted RAM
Small prompts (<500 tokens)	Overhead > benefit
Memory-constrained systems (<16GB RAM)	Cache consumes RAM
Single-user, interactive use	Minimal benefit

---

🛠️ Setup: Enabling the Feature

Prerequisites

1. llama.cpp version: v1.70+ (PR server : host-memory prompt caching #16391 merged October 2025)
2. RAM: Minimum 16GB (32GB+ recommended for large caches)
3. llama-server binary compiled with host-memory support

Basic Command

llama-server \
  --model ./models/qwen3.5-27b.q4_k_m.gguf \
  --ctx-size 32768 \
  --np 4 \
  --cram 256 \
  --host 0.0.0.0 \
  --port 8080 \
  --flash-attn \
  --log-disable

Key Parameters

Parameter	Value	Description
--np	4	Number of slots (concurrent contexts)
--cram	256	Cache RAM in MB (host memory for prompt cache)
--ctx-size	32768	Context size per slot

---

📊 Configuration Parameters Explained

--cram (Cache RAM)

Controls how much host RAM to allocate for prompt caching.

# Small setup (testing)
--cram 64

# Medium setup (production, 4-8 slots)
--cram 256

# Large setup (heavy multi-user, 16+ slots)
--cram 512

# Unlimited (use available RAM)
--cram 0

Rule of thumb:
--cram ≈ (number of cached prefixes) × (average prefix size in tokens) × (bytes per token)

Example:

• 10 prefixes × 10K tokens × 8 bytes (Q4_K_M) = 800MB → use --cram 1024

--np (Number of Slots)

Number of concurrent contexts the server can handle.

# Single-user
--np 1

# Multi-user (recommended)
--np 4-8

# Heavy multi-tenant
--np 16-32

Important: Each slot needs VRAM for its active KV cache. Host-memory cache is separate.

System Prompt File (Optional)

For shared system prompts across all slots:

llama-server \
  --model ./model.gguf \
  --system-prompt-file ./system-prompt.txt \
  --np 4 \
  --cram 256

This automatically caches the system prompt as a common prefix shared by all slots.

---

📈 Real-World Benchmark: Before vs After

Test Setup

• Model: Qwen3.5-27B Q4_K_M
• GPU: RTX 3090 (24GB VRAM)
• RAM: 64GB DDR4
• System prompt: 8,192 tokens (custom instructions + knowledge base)
• User query: "What are the key principles?" (variable length)
• Hardware: Intel i7-12700K

Scenario 1: No Caching

# Without host-memory caching
llama-server --model model.gguf --np 1 --ctx-size 32768

Metric	Value
TTFT (first request)	4.2s
TTFT (second request, same prefix)	4.1s
TBT (tokens/sec)	32 tok/s
VRAM usage	22.4GB
RAM usage	8.2GB

Observation: Every request reprocesses the full 8K system prompt.

Scenario 2: With Host-Memory Caching

# With host-memory caching enabled
llama-server --model model.gguf --np 4 --cram 256 --ctx-size 32768

Metric	Value	Improvement
TTFT (first request)	4.3s	—
TTFT (second request, cached)	0.3s	93% faster ⚡
TBT (tokens/sec)	34 tok/s	+6%
VRAM usage	22.8GB	+0.4GB
RAM usage	10.1GB	+1.9GB (cache)

🎉 Results

• 93% reduction in TTFT for cached requests
• +6% token generation speed (less VRAM fragmentation)
• Minimal RAM overhead (+2GB for 8K token cache)

---

🚀 Advanced: Multi-Slot Caching Strategy

Use Case: Chatbot with Multiple Users

You have:

• Shared system prompt: 8K tokens (rules, knowledge)
• User A: 3K tokens conversation history
• User B: 5K tokens conversation history
• User C: 2K tokens conversation history

Strategy 1: Shared Prefix + Individual Caches

llama-server \
  --model model.gguf \
  --system-prompt-file system.txt \
  --np 4 \
  --cram 512 \
  --slot-id-0 system+user_a \
  --slot-id-1 system+user_b \
  --slot-id-2 system+user_c

How it works:

• System prompt cached once (shared across all slots)
• Each user's history cached separately
• Total cache: 8K + 3K + 5K + 2K = 18K tokens

Strategy 2: Dynamic Cache Management

Monitor cache efficiency:

# Enable debug logging
llama-server --cram 256 --debug-slot

Output example:

[debug] slot 0: prefix match 95% (cached 8192/8192 tokens)
[debug] slot 1: prefix match 78% (cached 6400/8192 tokens)
[debug] slot 2: no match (recomputing)

Optimization: Adjust --cram based on hit rate:

• >80% hit rate: Increase --cram (more cache = better)
• <30% hit rate: Decrease --cram (wasted RAM)

---

🔧 Troubleshooting & Best Practices

Issue 1: "Cache miss rate too high"

Symptom: Requests not benefiting from caching

Solutions:

1. Check if prompts are actually similar (use --debug-slot)
2. Increase --ctx-size if prefixes are truncated
3. Use --system-prompt-file for truly shared prefixes

Issue 2: "Out of RAM" errors

Symptom: Server crashes or OOM killer terminates process

Solutions:

# Reduce cache size
--cram 128  # instead of 512

# Reduce number of slots
--np 2  # instead of 8

# Monitor RAM usage
watch -n 1 'free -h'

Issue 3: "Cache not persisting across restarts"

Current limitation: Host-memory cache is not persistent (RAM-only).

Workaround: Use --system-prompt-file for static content that should always be cached.

Best Practices Checklist

• ✅ Start small: Begin with --cram 128, scale up based on usage
• ✅ Monitor hit rate: Use --debug-slot to verify caching works
• ✅ Size appropriately: Don't allocate more RAM than needed
• ✅ Use system-prompt-file: For static shared content
• ✅ Test with realistic loads: Benchmark before production

---

📝 Summary & Key Takeaways

What We Learned

1. Host-memory prompt caching stores pre-computed prompts in RAM, not VRAM
2. Ideal for: Large system prompts, multi-user apps, limited VRAM scenarios
3. Key parameters: --cram (cache size), --np (slots), --system-prompt-file
4. Performance gains: Up to 93% TTFT reduction for cached requests
5. Trade-off: RAM usage vs. VRAM savings

Quick Reference Card

# Basic setup
llama-server --model model.gguf --np 4 --cram 256

# With system prompt
llama-server --model model.gguf --system-prompt-file system.txt --np 4 --cram 256

# Debug mode
llama-server --model model.gguf --cram 256 --debug-slot

# Large deployment
llama-server --model model.gguf --np 16 --cram 1024 --ctx-size 65536

Next Steps

• Experiment with different --cram values for your use case
• Monitor cache hit rates and adjust accordingly
• Combine with KV cache quantization (--cache-type-k q8_0) for maximum efficiency
• Check out related tutorials: KV Cache Reuse, Multi-Slot Management

---

🙏 Credits & References

• Original PR: server : host-memory prompt caching #16391 by @ggerganov, @ngxson
• Tutorial request: Issue tutorials : list for llama.cpp #13523
• Hardware: RTX 3090, Intel i7-12700K, 64GB RAM
• Model: Qwen3.5-27B Q4_K_M (unsloth dynamic)

---

Have questions? Comment below or open a new discussion! 🚀

Found this helpful? Star the repo and consider contributing your own tutorial! ⭐

---

Published by mind-tercero-roldan on March 15, 2026
Part of the llama.cpp tutorial series — making local LLMs accessible to everyone 💜

[jimiflowers]

Hi @mind-tercero-roldan! I have a question about your tutorial: How did you manage this, considering --system-prompt-file has been removed from recent versions?

[Naster17]

AI Slop detected

[elonWrong]

yeh... "--debug-slot" does NOT exist,

"--ctx-size | 32768 | Context size per slot" is wrong. Without the flag "--kv-unified" the available context is split across the configured slots. ie, 100k context with -np 4, each slots gets 25k context.

[nemb111]

Can the whole discussion be deleted? I fell for it too and only saw your comment about AI slop when I already invested too much time on this "tutorial".

[Vaizarr]

Note to self: Check comments before AI write-ups.

[evetsagg]

whats worse is that google ai search is picking up on this and using this as ai answer summary.

[CarvalhoLeonardo]

whats worse is that google ai search is picking up on this and using this as ai answer summary.

Really? What search results in it ?

[cyberfox]

Search that results in incorrect information pulled from this page:

llama.cpp slot similarity

I'm also having the problem where it resets the slot every turn, and it's bugging the crap out of me with long-ish turn histories and a slow model.