Standard retrieval-augmented generation treats a knowledge base as a flat pile of text chunks, matching a query against whatever vectors sit closest in embedding space. That works fine until someone asks a question that spans twelve documents, three departments, and a chain of relationships no single chunk captures. A 2026 study from the UK’s National Innovation Centre for Data tested vector-only retrieval against a vector-plus-graph setup on 510 complex questions and found the graph-based approach answered 65.3 percent correctly versus 28.9 percent for vector-only RAG. That gap is why GraphRAG, the open-source framework Microsoft released for building knowledge-graph-backed retrieval, has become one of the more searched terms in the RAG space this year.
This tutorial walks through building a complete GraphRAG pipeline running entirely on local hardware: no OpenAI key, no Anthropic bill, no cloud dependency. You will index a document corpus with an open-weight LLM served through Ollama, build a queryable knowledge graph, and run both local and global search against it. By the end you will have a working project directory you can point at your own files and a clear sense of where this approach beats plain vector RAG and where it does not.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What GraphRAG actually does differently
Microsoft’s graphrag package, currently at version 2.7.0, is a data pipeline that reads unstructured text, uses an LLM to extract entities and the relationships between them, and organizes the result into a graph. On top of that graph it builds community summaries using a hierarchical clustering algorithm, so the system has both fine-grained facts (this person works at this company) and higher-level summaries (this cluster of documents concerns a merger). Vector RAG has none of that structure; it just measures cosine similarity between a query embedding and chunk embeddings.
That extra structure is what lets GraphRAG answer two kinds of questions vector RAG struggles with. Local search handles questions anchored to specific entities, similar to standard RAG but enriched with graph neighbors. Global search handles corpus-wide questions, like “what are the main themes across all these documents,” by reasoning over community summaries instead of individual chunks. The Microsoft Agent Framework documentation describes this as the difference between inserting retrieved context before model invocation versus letting the model pull grounding data through a search tool on demand, and GraphRAG supports both patterns.
The tradeoff is indexing cost. Building the graph means running the LLM repeatedly over every chunk for entity extraction, relationship extraction, and summarization, which is far more compute-intensive than generating embeddings once. Running that indexing pass locally instead of against a metered API is what makes this tutorial worth doing on a workstation rather than a cloud bill.
Choosing the right local model for extraction
Model choice matters more for GraphRAG than it does for a simple chat use case, because indexing depends on the model reliably returning structured JSON across hundreds of extraction calls. A model that is a perfectly good conversational assistant can still be a poor fit for this pipeline if it drifts from the expected schema on long or unusual input. Ollama’s current library includes several candidates worth weighing against each other before you commit to a full indexing run.
| Model | Size / context | Fit for GraphRAG indexing |
|---|---|---|
| gpt-oss:20b | 14GB, 128K context | Strong general-purpose extraction, Apache 2.0 licensed, the most tested option for this workflow |
| gpt-oss:120b | 65GB, 128K context | Higher extraction quality but requires substantially more VRAM or unified memory; best for workstation or server-class hardware |
| qwen3.8 | Varies by quantization | Delivers gains in long-horizon agentic tasks per Alibaba’s release notes; a reasonable alternative if gpt-oss is unavailable |
| qwen3.6 | Varies by quantization | Lighter option with upgrades to agentic coding and reasoning preservation; a fallback for constrained hardware |
For embeddings, qwen3-embedding and embeddinggemma are the two models Ollama’s own documentation currently recommends, alongside the longer-established nomic-embed-text and mxbai-embed-large. Embedding vector dimensions typically fall in the 384 to 1024 range depending on the model, which is why Step 4’s warning about not mixing embedding models mid-project matters: a dimension mismatch between an old and new embedding model breaks the vector store silently rather than throwing an obvious error at indexing time.
If you are choosing between the 20B and 120B gpt-oss variants, the practical deciding factor is almost always hardware rather than accuracy preference. The 120B model’s 65GB footprint puts it out of reach for most single-consumer-GPU setups, so unless you have a multi-GPU workstation or are willing to run partly on CPU, the 20B model is the realistic default for this tutorial.
Prerequisites and versions
Confirm you have the following before starting. Version mismatches are the single biggest cause of failed indexing runs reported by people trying this setup.
| Requirement | Minimum version / spec | Notes |
|---|---|---|
| Python | 3.10 to 3.12 | GraphRAG’s dependency tree is not tested against 3.13 yet |
| graphrag package | 2.7.0 | Install via pip; this is the current GitHub release |
| Ollama | Latest stable build | Provides the OpenAI-compatible local endpoint |
| Local LLM | gpt-oss:20b or qwen3.8 | 14GB download, 128K context window for gpt-oss:20b |
| Embedding model | qwen3-embedding or nomic-embed-text | qwen3-embedding is 4.7GB with a 40K context window |
| RAM / VRAM | 16GB VRAM minimum, 24GB preferred | Indexing is more demanding than chat due to concurrent extraction calls |
| Disk space | 30GB free | Model weights plus graph artifacts and vector store |
| OS | Linux, macOS, or Windows with WSL2 | Native Windows works but WSL2 avoids path-handling bugs |
If your GPU has less than 16GB of VRAM, a 7B to 8B model with partial CPU offload is a fallback, but expect indexing a few hundred documents to take hours rather than minutes. This is the single biggest practical limitation of running GraphRAG locally versus pointing it at a hosted API.
Step 1: Install Ollama and pull your models
Download Ollama for your platform, then pull both a generation model and an embedding model. GraphRAG needs two separate model roles: one for extraction and summarization, one for turning text into vectors.
# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh
# Pull the generation model
ollama pull gpt-oss:20b
# Pull the embedding model
ollama pull qwen3-embedding
# Verify Ollama is serving the OpenAI-compatible endpoint
curl http://localhost:11434/v1/modelsBoth models communicate through Ollama’s OpenAI-compatible endpoint at http://localhost:11434/v1, which means GraphRAG talks to them exactly as it would talk to a hosted API, just with no key required and no per-token charge. If you have less VRAM, substitute a smaller model like qwen3.6 or an 8B Llama build, though extraction quality drops noticeably below the 20B class.
Step 2: Create a Python environment and install graphrag
Isolate the project in a virtual environment so GraphRAG’s dependency pins do not collide with other Python projects on the same machine.
python3 -m venv graphrag-env
source graphrag-env/bin/activate # Windows: graphrag-env\Scripts\activate
pip install --upgrade pip
pip install graphrag==2.7.0
# Confirm the install
python -m graphrag --versionIf pip resolves an older version, check PyPI directly, since the GitHub release tag and the PyPI release do not always publish on the same day. Pinning the version explicitly, as shown above, avoids silently picking up a pre-release build.
Step 3: Initialize the GraphRAG workspace
GraphRAG scaffolds a project directory with a settings file and environment template. This is where you point the pipeline at Ollama instead of a cloud API.
mkdir ragtest && cd ragtest
mkdir -p input
python -m graphrag init --root .This creates two key files: .env, which holds your API key placeholder, and settings.yaml, which configures models, chunk sizes, and the indexing workflow. For a local setup, the .env file can hold a dummy value since Ollama does not check API keys, but the variable still needs to be set or the pipeline will refuse to start.
# .env
GRAPHRAG_API_KEY=ollama-local-no-key-neededStep 4: Point settings.yaml at your local models
This is the step most local setups get wrong. Edit the generated settings.yaml to redirect both the chat model and the embedding model to your Ollama endpoint instead of the default OpenAI configuration.
models:
default_chat_model:
type: openai_chat
api_base: http://localhost:11434/v1
api_key: ${GRAPHRAG_API_KEY}
model: gpt-oss:20b
model_supports_json: true
concurrent_requests: 4
max_retries: 10
default_embedding_model:
type: openai_embedding
api_base: http://localhost:11434/v1
api_key: ${GRAPHRAG_API_KEY}
model: qwen3-embedding
concurrent_requests: 4
chunks:
size: 1200
overlap: 100Keep concurrent_requests low, between 2 and 4, for a local server. Ollama queues requests on a single GPU, so pushing concurrency high just produces timeouts rather than faster indexing. This is one of the most frequently reported issues in local GraphRAG setups: copying a cloud-tuned concurrency setting of 20 or more onto a local server that can realistically handle a handful of parallel calls.
Step 5: Add your documents
Drop plain text, CSV, JSON, or JSONL files into the input/ directory. GraphRAG does not ship a PDF parser out of the box, so convert PDFs to text first with a tool like pdftotext or a Python library such as PyMuPDF.
# Example: convert a folder of PDFs to plain text
pip install pymupdf
python3 -c "
import fitz, glob, os
for path in glob.glob('pdfs/*.pdf'):
doc = fitz.open(path)
text = chr(10).join(page.get_text() for page in doc)
out = os.path.join('input', os.path.basename(path).replace('.pdf', '.txt'))
with open(out, 'w') as f:
f.write(text)
"Start with a small corpus, 10 to 20 documents, for your first indexing run. Indexing time scales with both document count and chunk count, and a first run against hundreds of files with an untested local model configuration is the easiest way to burn an afternoon discovering a configuration mistake forty minutes in.
Step 6: Run the indexing pipeline
This is the step that builds the actual knowledge graph: entity extraction, relationship extraction, community detection, and summarization, all running through your local model.
python -m graphrag index --root .Watch the console output. GraphRAG logs each workflow stage (text extraction, graph extraction, community reports, embeddings) as it completes, which makes it easy to spot where a run stalls or fails. For a small test corpus on a 20B model with a 16 to 24GB GPU, expect this to take anywhere from 15 minutes to over an hour depending on document length and chunk count. This is dramatically slower than embedding-only indexing for standard vector RAG, which is the real cost of the accuracy gain GraphRAG provides.
Step 7: Inspect the generated graph artifacts
A successful run produces a set of Parquet files in output/, including entities, relationships, communities, community reports, and text units. Checking these before querying saves time, since a malformed extraction pass will produce an empty or near-empty entities table even if the pipeline exits without an error code.
python3 -c "
import pandas as pd
entities = pd.read_parquet('output/entities.parquet')
relationships = pd.read_parquet('output/relationships.parquet')
communities = pd.read_parquet('output/communities.parquet')
print('Entities:', len(entities))
print('Relationships:', len(relationships))
print('Communities:', len(communities))
print(entities[['title', 'type']].head(10))
"If entity count is near zero, the model is not following the JSON extraction schema correctly, which is a known limitation of smaller or less instruction-tuned local models. Switching to a larger model, or one explicitly tuned for structured output, is the usual fix.
Step 8: Run a local search query
Local search is the mode closest to standard RAG, but it pulls in graph neighbors around the entities matched to your query rather than just the nearest text chunks.
python -m graphrag query \
--root . \
--method local \
--query "What relationships exist between the main people mentioned in these documents?"Local search is the right default for questions anchored to specific named entities, like a person, a product, or a company. It behaves closer to vector RAG in response time because it does not need to traverse the entire community hierarchy.
Step 9: Run a global search query
Global search is where GraphRAG’s advantage over vector RAG becomes obvious, since it reasons over community summaries instead of individual chunks, which is exactly the kind of corpus-wide question that caused the accuracy gap in the NICD study referenced earlier.
python -m graphrag query \
--root . \
--method global \
--query "What are the three most significant themes across this entire document set?"Global search is also the slower and more token-hungry of the two modes, since it may touch multiple community reports to assemble an answer. On a local setup this is where you will notice the biggest latency difference compared to local search, often taking several times longer per query.
Step 10: Tune chunk size and community detection
Default chunk settings are a reasonable starting point but rarely optimal for a specific corpus. Smaller chunks produce more granular entities but more total LLM calls during indexing; larger chunks reduce calls but risk missing finer relationships within a chunk.
| Chunk size | Best for | Indexing cost |
|---|---|---|
| 300 to 600 tokens | Dense technical or legal text with fine-grained entities | High — more chunks, more extraction calls |
| 800 to 1200 tokens | General documents, reports, articles (recommended default) | Moderate |
| 1500+ tokens | Long narrative or loosely structured text | Lower, but risks missed relationships |
Community detection uses a clustering algorithm to group related entities into hierarchical levels. If your community reports come back too broad or too narrow to be useful, adjusting the community_reports section of settings.yaml to change the maximum report length and summarization detail is usually more effective than re-chunking the whole corpus.
Step 11: Add incremental updates instead of full reindexing
Re-running a full index every time you add a handful of new documents is wasteful, especially on local hardware where indexing is already the slowest part of the workflow. GraphRAG supports incremental indexing for appending new content without rebuilding the entire graph from scratch.
# Add new files to input/, then run with update mode
python -m graphrag index --root . --method updateIncremental updates still require the LLM to process the new chunks, so the compute savings come from skipping re-extraction on unchanged documents, not from making the new-document extraction itself faster.
Step 12: Wrap it in a simple query interface
For anything beyond manual testing, wrap the query calls in a small Python script so the pipeline can be called from an application rather than the command line.
import subprocess
import json
def graphrag_query(question, method="local", root="."):
result = subprocess.run(
["python", "-m", "graphrag", "query",
"--root", root, "--method", method, "--query", question],
capture_output=True, text=True, timeout=180
)
if result.returncode != 0:
raise RuntimeError(f"GraphRAG query failed: {result.stderr}")
return result.stdout
# Route simple entity questions to local search,
# corpus-wide questions to global search
def route_and_query(question, is_broad=False):
method = "global" if is_broad else "local"
return graphrag_query(question, method=method)
print(route_and_query("Summarize the main themes in this corpus", is_broad=True))A 180-second timeout is generous for local hardware; tighten it once you have a sense of your typical query latency, since a hung Ollama process will otherwise block the caller for the full timeout window.
Step 13: Decide when GraphRAG is worth the overhead
GraphRAG is not a universal upgrade over vector RAG. It is a more accurate but more expensive option for a specific category of question. Benchmarks consistently show the advantage concentrated in multi-hop and corpus-wide questions, not simple fact lookups. Before committing a project to GraphRAG, test whether your actual query patterns need graph reasoning or whether a well-tuned vector RAG with reranking would get you most of the way there for a fraction of the indexing cost.
A practical middle ground many teams land on is a hybrid setup: vector RAG as the default for fast, narrow queries, with GraphRAG’s global search reserved for the subset of queries that are explicitly corpus-wide. An AWS open-source blog post on combining GraphRAG and LightRAG on a single stack found that local, lighter-weight configurations reached roughly 82 percent of a full hybrid setup’s accuracy at around an eighth of the cost, which is a reasonable starting point if you are not sure yet whether full GraphRAG indexing is worth it for your corpus.
It helps to put rough numbers behind that decision rather than treating “local is cheaper” as a given. GraphRAG’s indexing workflow makes several LLM calls per chunk across extraction, relationship detection, and summarization stages, so a corpus with a few thousand chunks can easily generate hundreds of thousands of input and output tokens before a single query is ever run. Against a cloud API billed per million tokens, that indexing pass alone can run from a few dollars to well over a hundred dollars depending on corpus size and chosen model tier, even using one of the cheaper current options. Running the same indexing pass locally through Ollama has no per-token charge at all once the model weights are downloaded, so the cost shifts entirely to hardware and time rather than a metered bill, which is exactly the tradeoff this tutorial is built around.
Complete working project structure
Once all the steps above are complete, your project directory should look like this:
ragtest/
├── .env # GRAPHRAG_API_KEY placeholder
├── settings.yaml # Ollama endpoints, chunk config, model names
├── input/ # Your source .txt/.csv/.json files
├── output/
│ ├── entities.parquet
│ ├── relationships.parquet
│ ├── communities.parquet
│ ├── community_reports.parquet
│ └── text_units.parquet
└── query_interface.py # Your wrapper script from Step 12This is a fully self-contained pipeline. Everything from document ingestion to query answering runs on local hardware, with the only external dependency being the one-time download of model weights through Ollama.
Example output
A global search query against a small test corpus of internal engineering reports produces output structured like this:
SUCCESS: Global Search Response:
Across the documents, three themes recur consistently:
1. Infrastructure migration — multiple reports reference a shift
from a monolithic deployment to containerized services, with
recurring mentions of rollback procedures and staged rollouts.
2. On-call process changes — several documents discuss revised
incident escalation paths and a reduction in mean time to
acknowledge incidents.
3. Cost optimization — recurring references to compute spend
reviews and the consolidation of underutilized instances.
Sources: community reports 3, 7, 12, 14Note the source attribution to specific community reports rather than individual chunks. That traceability back to a community, and from the community back to the underlying entities and text units, is one of the practical advantages of the graph structure over a flat vector index when you need to audit why the system answered the way it did.
Common pitfalls
These are the mistakes that most commonly derail a first local GraphRAG setup, based on recurring patterns across community setup guides and issue reports.
- Setting concurrency too high. Configs copied from a cloud-API tutorial often set
concurrent_requeststo 10 or more. A single local GPU cannot parallelize that many LLM calls, and the result is cascading timeouts rather than faster indexing. - Skipping the JSON schema check. Not every local model reliably outputs the structured JSON that GraphRAG’s extraction prompts expect. If entity counts come back near zero after indexing, the model is the first thing to suspect, not the pipeline configuration.
- Mixing embedding models mid-project. Switching the embedding model after an initial index without rebuilding the vector store produces dimension mismatches that fail silently or throw cryptic shape errors at query time.
- Feeding in raw PDFs. GraphRAG expects plain text, CSV, or JSON input. PDFs need conversion first; skipping this step produces garbled or empty extraction.
- Indexing a huge corpus on the first run. Running the full pipeline against thousands of documents before validating the configuration on a small sample wastes hours when a chunking or model setting turns out to be wrong.
- Ignoring the difference between local and global search. Using global search for a simple single-entity lookup burns far more tokens and time than necessary; local search is the better default for narrow questions.
- Not budgeting for indexing time. Local indexing is slower than cloud API indexing by a wide margin. Planning a demo around a same-day full reindex of a large corpus is a common scheduling mistake.
Troubleshooting
The following issues come up repeatedly in local GraphRAG and Ollama setups.
- Indexing fails with a connection error to localhost:11434. Confirm Ollama is actually running (
ollama serveif it is not already a background service) and that no firewall rule is blocking local loopback traffic. - Entity extraction returns malformed or truncated JSON. This typically means the model’s context window is being exceeded by a long chunk plus the extraction prompt. Reduce chunk size or switch to a model with a larger context window, such as the 128K window on gpt-oss:20b.
- Indexing hangs indefinitely on one chunk. Lower
concurrent_requeststo 1 or 2 temporarily to isolate whether the hang is a concurrency issue versus a genuinely stuck model call, then check Ollama’s own logs for the stalled request. - Query responses ignore the graph and just answer from general knowledge. This usually means indexing silently produced near-empty entity or community tables. Check
output/entities.parquetrow counts before assuming the query step is at fault. - Out-of-memory errors mid-index. Lower chunk size, reduce concurrency, or switch to a smaller or more aggressively quantized model. Indexing uses more memory than simple chat because of concurrent extraction calls stacking up.
- Global search is extremely slow. This is expected behavior, not a bug, since global search may touch many community reports per query. If latency is unacceptable, consider restricting global search to a lower community hierarchy level in
settings.yaml. - Embedding dimension mismatch errors at query time. This means the embedding model was changed after the initial index. Delete the vector store and re-embed with the new model, or revert to the original embedding model.
- pip install graphrag pulls an unexpected version. Pin the version explicitly with
pip install graphrag==2.7.0rather than letting pip resolve the latest tag, since pre-release builds sometimes surface first.
GraphRAG vs vector RAG: when each one wins
Neither approach is strictly better; they are suited to different query shapes. The table below summarizes the practical tradeoffs based on the indexing and query behavior covered in this tutorial.
| Factor | Vector RAG | GraphRAG |
|---|---|---|
| Indexing speed | Fast — one embedding pass per chunk | Slow — multiple LLM passes for extraction and summarization |
| Best for | Narrow, single-fact lookups | Multi-hop and corpus-wide questions |
| Accuracy on complex questions (NICD 2026 study) | 28.9% | 65.3% |
| Infrastructure | Vector database only | Vector store plus graph storage |
| Traceability | Chunk-level source citation | Entity, relationship, and community-level citation |
| Update cost | Low — re-embed new chunks | Higher — incremental indexing still requires LLM extraction on new content |
If you have already built a standard vector pipeline, our RAG pipeline tutorial covers that baseline setup in detail, and comparing the two side by side on your own corpus is the most reliable way to decide whether the graph structure is worth the extra indexing time for your use case.
Advanced tips for production use
Once the basic pipeline is working, a few adjustments make it more reliable for ongoing use rather than a one-off demo.
- Route queries by complexity automatically. Build a lightweight classifier, or even a simple keyword heuristic, that sends narrow questions to local search and broad or comparative questions to global search, rather than hardcoding one mode.
- Monitor entity and relationship counts after every reindex. A sudden drop signals an extraction regression, often caused by an Ollama model update changing default behavior.
- Cap community report length explicitly. Long, unbounded community reports slow down global search without necessarily improving answer quality; trimming them keeps query latency predictable.
- Separate your indexing and query hardware if possible. Indexing is a batch, throughput-oriented workload; queries are latency-sensitive. Running both on the same single GPU under load means every query competes with any in-progress reindex.
- Version your settings.yaml alongside your corpus. Chunk size and model changes invalidate comparisons between old and new indexing runs, so treating the configuration as part of your dataset’s version history avoids confusing regressions.
For teams already running local models for other parts of their stack, pairing GraphRAG with an existing local Ollama deployment or a vLLM serving setup for higher-throughput query-time inference is a natural next step once the basic pipeline described here is validated. Teams comparing orchestration frameworks for the broader agent layer around GraphRAG may also find our LangChain vs LlamaIndex vs CrewAI comparison useful, since several community GraphRAG wrappers build on top of these frameworks rather than the bare graphrag package.
Securing a local GraphRAG deployment
Running everything locally removes the risk of sending sensitive documents to a third-party API, which is one of the main reasons teams in regulated industries look at this setup in the first place. That does not mean a local GraphRAG pipeline is automatically safe to expose beyond your own machine. Ollama’s default configuration binds to localhost, and that default should stay in place unless you have a specific, access-controlled reason to expose the endpoint on a network interface. Opening the Ollama port to a shared network without authentication turns a private indexing tool into an unauthenticated inference server anyone on that network can query or abuse.
The query interface itself also deserves the same scrutiny any LLM-backed application gets. If you expose the route_and_query wrapper from Step 12 behind a web endpoint, treat the query string as untrusted input, same as you would for any other prompt-injection surface. The community reports and entity data that come back from GraphRAG are generated text, not verified facts, so any downstream system that acts on them automatically, rather than surfacing them for human review, should be scoped narrowly. Teams building this out for an internal tool should look at the standard prompt-injection and output-handling guidance that applies to any RAG system, since the retrieval layer does not remove those risks, it just changes where the untrusted content comes from.
Document provenance is worth tracking explicitly too. Because GraphRAG summarizes and merges information across documents into community reports, a single mislabeled or outdated source file can quietly influence summaries drawn from many other files once the graph is built. Keeping a simple manifest of source document versions, alongside the output/ artifacts, makes it far easier to trace a bad answer back to the document that caused it.
Scaling beyond a single machine
Everything in this tutorial assumes one machine running Ollama, GraphRAG, and your query wrapper together. That is the right starting point, but it stops scaling once your corpus grows into the thousands of documents or once multiple people need to query the graph concurrently. The first bottleneck you will hit is indexing throughput, since a single GPU serializes extraction calls regardless of how many documents are queued behind it.
A common next step is separating the indexing workload from the query workload onto different hardware, as mentioned in the advanced tips above, so a long-running reindex does not compete with live queries for GPU time. For teams that outgrow a single local Ollama instance entirely, swapping the model-serving layer for a dedicated inference server such as vLLM, which supports continuous batching and multi-GPU serving, lets you keep the GraphRAG indexing logic unchanged while scaling the underlying compute. The api_base setting in settings.yaml is what makes that swap straightforward, since GraphRAG only cares that the endpoint speaks the OpenAI-compatible API, not what is actually running behind it.
On the storage side, the Parquet-based output GraphRAG produces by default is fine for a single-machine setup but becomes awkward once multiple services need concurrent read access to the graph. At that point, exporting the entities and relationships into a dedicated graph database, or the vector store into a managed vector database, is the usual path, with GraphRAG’s own indexing pipeline staying in place as the step that builds the graph before it gets exported downstream.
Reducing local indexing cost further
If indexing time remains a bottleneck even after tuning concurrency and chunk size, quantization is the next lever. Running a 4-bit quantized build of your chosen model through Ollama trades a small amount of extraction accuracy for a meaningful reduction in both VRAM use and per-call latency, which compounds across the hundreds or thousands of extraction calls a full indexing run makes. Our GGUF quantization guide covers how to shrink a model for exactly this kind of batch-heavy local workload, and is a useful next read if indexing a corpus larger than a few hundred documents on consumer hardware.
Frequently asked questions
Does GraphRAG require an internet connection once set up?
No. Once model weights are downloaded through Ollama and the corpus is indexed, both indexing and querying run entirely offline. The only network dependency is the initial Ollama model pull.
Can I use GraphRAG with a smaller model than gpt-oss:20b?
Yes, but extraction quality drops. Smaller models are more prone to malformed JSON output during entity and relationship extraction, which directly lowers graph quality. If you must use a smaller model, budget extra time for manual spot-checking of the entities.parquet output.
How is GraphRAG different from just using LangChain with a graph database?
GraphRAG is a specific indexing methodology, including automated entity and relationship extraction plus hierarchical community summarization. A LangChain-plus-Neo4j setup gives you the infrastructure to store and query a graph, but you still have to build the extraction and summarization logic yourself, or adopt a library like GraphRAG that already implements it.
Is GraphRAG always more accurate than vector RAG?
No. It shows its advantage specifically on multi-hop and corpus-wide questions. For narrow, single-fact lookups, a well-tuned vector RAG setup is often comparably accurate and considerably cheaper to index and query.
How long does indexing take for a real-sized corpus?
It depends heavily on document count, chunk size, and hardware, but local indexing on consumer GPU hardware is typically far slower than cloud API indexing. Testing on a small sample first, as described in Step 5, is the best way to estimate timing before committing to a full run.
Can I mix local and cloud models in the same GraphRAG project?
Yes. The settings.yaml configuration supports pointing different model roles at different endpoints, so you could run extraction locally for cost control while using a cloud model for query-time generation, or vice versa, depending on where your latency and cost constraints sit.
What happens if I add new documents after the initial index?
Use the update method shown in Step 11 rather than a full reindex. It processes only new content through extraction, though it still requires LLM calls for the new documents specifically.
Do I need a GPU to run this tutorial?
Technically no, since Ollama supports CPU-only inference, but indexing a corpus of any meaningful size on CPU alone is impractically slow for most use cases. A GPU with at least 16GB of VRAM is the realistic minimum for a usable local setup.
