A general-purpose model is good at everything and great at nothing specific to your product. LoRA fine-tuning closes that gap without the cost of training a model from scratch, and the economics keep favoring it: as of May 2026, LeanML estimated that calling a frontier model’s API directly costs **$12–$30 per million output tokens**, a bill that recurs on every single request, against a one-time adapter you train once and own outright. Instead of updating billions of parameters, you train a small set of adapter weights, typically adapting about **1%–10%** of a model’s parameters with LoRA-style methods, using frozen base weights plus low-rank adapters. The result behaves like a specialist while the original model never changes.
This tutorial walks through fine-tuning an open-weight LLM with LoRA and its 4-bit cousin QLoRA, from picking a base model through deploying the merged result. You’ll set up a Python environment, format a training dataset, configure a LoRA adapter, run training on a single GPU, and export a model you can serve. By the end you’ll have a working script, a trained adapter, and a running list of what typically breaks along the way. Expect around 90 minutes if your GPU and Hugging Face account are already set up, longer if you’re renting cloud compute for the first time.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What Is LoRA Fine-Tuning, and Why Skip Full Fine-Tuning?
Full fine-tuning updates every weight in a model. For a 7-billion-parameter model, that means storing gradients and optimizer states for 7 billion values, which typically demands 60GB or more of GPU memory spread across multiple accelerators. Most individual developers and small teams don’t have that kind of hardware sitting around, and even when they do, full fine-tuning on a small dataset tends to overfit fast and can quietly erode the model’s general capabilities.
LoRA (Low-Rank Adaptation) takes a different approach. Instead of updating the full weight matrix in a layer, it freezes the original weights and injects two small trainable matrices next to them. Their product approximates the change the model needs to learn, using a fraction of the parameters. This comes from the 2021 paper by Edward Hu and colleagues at Microsoft, “LoRA: Low-Rank Adaptation of Large Language Models”. Hugging Face’s own PEFT documentation illustrates the payoff with a concrete example: fine-tuning a 3-billion-parameter model in full costs roughly 47.14GB of GPU memory, while a LoRA adapter at rank 16 does the same job in about 14.4GB, training only around LoRA is typically around **0.1%–1%** of the base model’s parameters, not exactly 0.1% in all cases.
QLoRA pushes the savings further. Introduced by Tim Dettmers and colleagues in the 2023 paper “QLoRA: Efficient Finetuning of Quantized LLMs”, it first compresses the frozen base model down to 4-bit precision, then trains LoRA adapters on top in higher precision. Double quantization and paged optimizers keep memory pressure low during backpropagation. That combination is what lets a single consumer GPU with 16-24GB of VRAM fine-tune a 7B-8B model that would otherwise need a data-center card. Nothing about the base model’s weights changes during training. Only the adapter learns, which also means you can swap adapters in and out of the same base model without reloading gigabytes of frozen weights each time.
LoRA vs QLoRA vs Full Fine-Tuning vs RAG
Fine-tuning isn’t always the right tool. If your goal is to give a model access to fresh or private documents, a RAG pipeline often solves the problem faster and without touching model weights at all. Fine-tuning earns its cost when you need to change how the model behaves: its tone, its output format, its domain vocabulary, or its reasoning pattern on a narrow task. Here’s how the main options stack up for a 7B-8B parameter model.
| Method | Trainable Params | Typical VRAM | Best For | Main Weakness |
|---|---|---|---|---|
| Full fine-tuning | 100% | 60GB+ (often multi-GPU) | Maximum quality with a large budget | Expensive, slow, easy to overfit on small datasets |
| LoRA (bf16/fp16 base) | ~0.1-1% | ~16-24GB | Balanced quality and cost | Still loads a full-precision base model into memory |
| QLoRA (4-bit base) | ~0.1-1% | ~12-16GB | Training on a single consumer GPU | Slightly slower per step than full-precision LoRA |
| RAG (no training) | 0% | Depends on retrieval stack | Fast-changing knowledge, no training loop | Doesn’t change the model’s style, tone, or reasoning |
A practical rule: reach for RAG when the problem is “the model doesn’t know this fact,” and reach for LoRA or QLoRA when the problem is “the model doesn’t behave the way I need.” Plenty of production systems use both, retrieving context at inference time while running on a fine-tuned adapter that’s already learned the house style. For a broader look at how current models compare on cost and capability, see our AI model comparison hub.
Real-World Use Cases: When LoRA Fine-Tuning Actually Pays Off
Reading a LoRA tutorial and deciding whether you actually need one are two different exercises. The pattern that shows up across most successful projects is narrow and repeatable: a task the base model already sort of does, just not consistently enough to trust in production. The same narrow-task logic is spreading well past text, too: AlphaSignal reported in September 2026 that a community-trained LoRA for the MiniMax-H3 image model could generate a full **360°** photo orbit from a single input image, and by October 2026, Local Model Watch flagged a **22B**-parameter IC-LoRA for LTX-2.5 built specifically for SDR-to-HDR video conversion.
- Support ticket triage. Teams fine-tune a small model to read incoming tickets and output a consistent, structured summary, category, and urgency label. A general-purpose model can do this too, but its formatting drifts across requests. A LoRA adapter trained on a few thousand labeled tickets locks that format down.
- Brand voice and tone. Marketing and support teams train adapters that consistently write in a specific voice, whether that’s terse and technical or warm and conversational, without needing a lengthy system prompt on every single request.
- Structured data extraction. Pulling consistent JSON out of messy invoices, resumes, or medical notes is a classic LoRA use case, since the task is narrow, repetitive, and easy to score against a held-out set.
- Code style and internal API adaptation. Some engineering teams fine-tune a coding-capable base model on their own codebase’s conventions and internal libraries, so suggestions match house style instead of generic public patterns.
- Low-resource language and dialect adaptation. Base models are usually strongest in English. A LoRA adapter trained on a smaller language or a specific regional dialect can noticeably improve fluency without the cost of pretraining a new model.
Notice what’s absent from that list: teaching a model brand-new facts. That’s the RAG job, not the fine-tuning job, and conflating the two is the single most common reason first LoRA projects disappoint their owners.
Prerequisites: Hardware, Software, and Accounts You’ll Need
You don’t need a data center for this project, but you do need a CUDA-capable GPU, either local or rented. Here’s the full stack this tutorial assumes.
| Requirement | Minimum | Notes |
|---|---|---|
| Python | 3.10 or newer | Use a virtual environment to isolate dependencies |
| GPU | 12-16GB VRAM (QLoRA) | 16-24GB is more comfortable for longer sequences |
| PyTorch | 2.x with a CUDA 12.1 build | Match the build to your installed CUDA driver |
| transformers | Latest version | Hugging Face’s model-loading library |
| peft | 0.19.1 or newer | Hugging Face’s own integration docs require peft >= 0.19.1 |
| bitsandbytes | Latest version | Handles 4-bit quantization for QLoRA |
| accelerate, datasets, trl | Latest versions | Training loop, dataset loading, and SFTTrainer |
| Hugging Face account | Free tier | Needed for gated model downloads and pushing your adapter to the Hub |
If you don’t own a GPU, skip ahead to Step 3, where we cover renting one by the hour. Everything else in this guide runs the same whether the GPU is sitting on your desk or in a data center somewhere else.
Step 1: Set Up Your Python Environment and Install Core Libraries
Start with a clean virtual environment so this project’s dependencies don’t collide with anything else on your machine.
python -m venv lora-env
source lora-env/bin/activate # Windows: lora-envScriptsactivate
pip install --upgrade pip
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install transformers accelerate "peft>=0.19.1" bitsandbytes datasets trlConfirm PyTorch can see your GPU before going any further. A quick Python check saves you from debugging a training script later when the actual problem is that CUDA never loaded.
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"That should print True followed by your GPU’s name. If it prints False, stop here and fix your CUDA driver or PyTorch build before moving on. No amount of correct training code will work around a GPU that PyTorch can’t see.
Step 2: Choose Your Base Model
The open-weight landscape gives you real choice in 2026, and the right base model depends on your hardware budget and task. Here are the families most commonly used as fine-tuning starting points.
| Family | Typical Sizes | License Notes | Good For |
|---|---|---|---|
| Llama 3.2 | 1B-3B in the lightweight tier, larger checkpoints in the family | Meta community license, gated download | General instruction tuning with broad tooling support |
| Qwen | Sub-1B up through tens of billions of parameters | Apache 2.0 for most sizes | Multilingual tasks and strong small-model performance |
| Gemma 3 | 1B up through the high-20B range | Gemma license, gated download | Efficient small and mid-size checkpoints |
| Mistral 7B / Mixtral | 7B dense, larger mixture-of-experts variants | Apache 2.0 | Strong general baseline with wide community tooling |
| Phi-4 Mini | Small, mini-class | Permissive, MIT-style | Budget hardware and fast iteration |
| DeepSeek distills | Distilled from larger reasoning-focused checkpoints | Open-weight | Reasoning-leaning tasks on modest hardware |
For a first project, pick something in the 3B-8B range. It trains fast enough to iterate in an afternoon and still produces results you can judge honestly. We’ve compared these families head to head in more depth, including context windows and benchmark results, in Phi-4 Mini vs Gemma 3 vs Llama 3.2 and DeepSeek V4 vs GLM-5.2 vs Qwen. Gated repositories (Llama and Gemma both fall into this category) require you to accept a license on the model’s Hugging Face page and authenticate locally with huggingface-cli login before the download will work.
Step 3: Pick a Local GPU or Rent One in the Cloud
If you already own a GPU with 16GB or more of VRAM, skip this step. If not, renting by the hour is normal practice for a one-off fine-tuning run, and it’s usually cheaper than buying hardware you’ll use once a month.
| Option | VRAM | Approx. Cost | Good For |
|---|---|---|---|
| RTX 3090 / 4090 (owned) | 24GB | One-time hardware cost | Repeated experimentation over months |
| RunPod RTX 4090, Community Cloud | 24GB | From $0.34/hr | Short training runs on a budget |
| RunPod RTX 4090, Secure Cloud | 24GB | From $0.69/hr | More reliable uptime for longer jobs |
| RunPod A100 80GB | 80GB | Roughly $1.19-$1.49/hr | Larger base models or longer context windows |
| Apple Silicon Mac (local) | Shared unified memory | One-time hardware cost | Small models via MLX or llama.cpp, outside this CUDA-based workflow |
Pricing above comes from RunPod’s own rate card. Marketplace GPU pricing shifts with demand, so treat these numbers as a starting point rather than a locked-in quote. If you’re weighing a local workstation purchase against renting, our Nvidia DGX Spark vs Mac Studio comparison covers the tradeoffs between unified-memory machines and dedicated GPU boxes for AI workloads. For a single 3B-8B QLoRA run of a few hours, a rented RTX 4090 on Community Cloud typically costs less than a bus fare, and even scaling up to 33B-70B parameter models keeps QLoRA affordable: LeanML’s May 2026 cost estimate put total training compute for that larger range at roughly **$3 to $250**, depending on dataset size and the GPU tier you rent.
Step 4: Install Unsloth for Faster Training (Optional)
Unsloth is a free, open-source library that rewrites the training kernels PEFT relies on, cutting both training time and memory use compared with the standard Transformers and PEFT stack. It supports QLoRA fine-tuning for Llama, Qwen, Gemma, and Mistral out of the box, and it’s a drop-in replacement for the model-loading step rather than a separate framework you need to learn. This step is optional, everything below also works with plain PEFT, but most people fine-tuning on a single consumer GPU use it.
pip install unslothfrom unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/llama-3.2-3b-instruct-bnb-4bit",
max_seq_length=2048,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
model,
r=16,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha=16,
lora_dropout=0,
bias="none",
use_gradient_checkpointing="unsloth",
)If you’d rather stick with the standard Hugging Face stack, skip this step entirely and continue with Step 5. The rest of this tutorial shows the plain PEFT path, which is more verbose but easier to debug when something goes wrong.
Step 5: Prepare and Format Your Training Dataset
Your dataset is the actual product here. The adapter only learns what’s in it. A common, simple format is a JSONL file with one instruction-response pair per line.
{"instruction": "Summarize this support ticket in one sentence.", "response": "Customer cannot reset their password after the June update."}
{"instruction": "Rewrite this in a formal tone.", "response": "We regret to inform you that your request could not be processed at this time."}from datasets import load_dataset
dataset = load_dataset("json", data_files="training_data.jsonl", split="train")
def format_example(example):
return {
"text": f"### Instruction:n{example['instruction']}nn### Response:n{example['response']}"
}
dataset = dataset.map(format_example)
dataset = dataset.train_test_split(test_size=0.1)How much data do you need? A few hundred examples can shift a narrow behavior like output format or tone. Somewhere between 1,000 and 10,000 examples is the common range for real instruction-following or domain adaptation. Past 10,000 examples, you’re mostly buying robustness and better generalization rather than unlocking new capability. Quality matters more than volume: 500 carefully written, consistent examples usually beat 5,000 scraped, noisy ones. Always hold out a test split, as shown above, so you have a way to check for overfitting later.
Where to source training examples
Three sources cover most projects. The first is your own historical data: past support tickets, internal documents, or logged interactions that already show the input-output pattern you want the model to learn. This is usually the highest-quality option because it reflects real usage rather than a guess at what real usage looks like.
The second is synthetic data generation, where you use a larger, more capable model to generate training examples for your smaller model to learn from. This works well for narrow, well-defined tasks and is far faster than manual writing, though it needs a human review pass to catch the larger model’s own mistakes before they get baked into your adapter.
The third is manual curation, writing examples by hand or editing machine-generated drafts into a consistent shape. It’s the slowest option and also usually the highest quality per example, which makes it a reasonable choice when your total dataset only needs to be a few hundred rows. Whichever source you use, deduplicate aggressively. Near-identical repeated examples inflate your row count without adding anything the model hasn’t already seen, and they’re a quiet contributor to the overfitting problems covered later in this guide.
Step 6: Load the Base Model in 4-Bit Precision
This is the QLoRA step: quantizing the frozen base model down to 4-bit before any training starts. If you used Unsloth in Step 4, this already happened and you can skip to Step 7.
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
model_name = "meta-llama/Llama-3.2-3B-Instruct"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto",
)The nf4 quant type (NormalFloat4) is the format QLoRA’s authors designed specifically for weights that follow a roughly normal distribution, which is what pretrained transformer weights typically look like. Double quantization squeezes out a bit more memory by also compressing the quantization constants themselves. Setting the pad token explicitly here avoids a common source of silent training bugs, more on that in the troubleshooting section below.
Step 7: Configure Your LoRA Adapter
With the base model loaded, define the adapter that will actually learn. Three settings matter most: rank, alpha, and which layers to target.
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()That last line prints something like the output below, which is worth checking every time. It confirms the adapter actually attached and gives you a sense of how small the trainable slice really is.
trainable params: 6,815,744 || all params: 3,220,041,216 || trainable%: 0.2117Rank (r) controls how much capacity the adapter has. 8, 16, and 32 are the most common values in practice, with 16 as a reasonable default for a first run, though the practical ceiling keeps rising: Releasebot reported that as of September 2026, Together AI raised the maximum LoRA rank supported by its hosted fine-tuning service from 64 to 128, and that higher ceiling now ships as the default rank in its CLI/SDK v2.36.0 release. Alpha scales the adapter’s contribution relative to the frozen weights, commonly set to 16, 32, or 64, often matched to or double the rank. target_modules lists which linear layers get an adapter attached. The four attention projections above are the minimum for a visible effect, and adding the MLP layers (gate_proj, up_proj, down_proj) gives the adapter more room to change the model’s behavior at the cost of a few more trainable parameters.
Step 8: Set Up the Trainer and Training Arguments
Hugging Face’s TRL library provides SFTTrainer, purpose-built for supervised fine-tuning on instruction-formatted text. It wraps the standard Transformers Trainer with sensible defaults for this exact job.
from trl import SFTTrainer, SFTConfig
training_args = SFTConfig(
output_dir="./lora-output",
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
num_train_epochs=3,
learning_rate=2e-4,
logging_steps=10,
save_steps=100,
bf16=True,
max_seq_length=1024,
dataset_text_field="text",
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
)A learning rate around 2e-4 is a safe starting point for LoRA, noticeably higher than the 1e-5 to 5e-5 range typical of full fine-tuning, because the adapter has far fewer parameters to move. Gradient accumulation lets you simulate a larger batch size than your VRAM would otherwise allow by accumulating gradients across several smaller forward passes before each optimizer step.
Step 9: Launch Training and Watch the Loss Curve
trainer.train()Training logs print at the interval set by logging_steps. For a healthy run on a few thousand examples, expect the loss to drop steadily over the first epoch, then level off.
{'loss': 1.8432, 'learning_rate': 0.000198, 'epoch': 0.1}
{'loss': 1.5109, 'learning_rate': 0.000180, 'epoch': 0.5}
{'loss': 1.1287, 'learning_rate': 0.000120, 'epoch': 1.0}
{'loss': 0.8965, 'learning_rate': 0.000060, 'epoch': 2.0}
{'loss': 0.7331, 'learning_rate': 0.000005, 'epoch': 3.0}Keep a second terminal open with watch -n 1 nvidia-smi during your first run so you can see VRAM usage in real time. It’s the fastest way to learn how much headroom you actually have before you hit an out-of-memory error mid-run. If loss plateaus immediately near its starting value instead of dropping, jump to the troubleshooting table below before burning more GPU hours.
Step 10: Test the Fine-Tuned Adapter With Inference
Before merging anything, generate a few outputs directly from the adapter-wrapped model to confirm training actually changed its behavior.
prompt = "### Instruction:nSummarize this support ticket in one sentence.nn### Response:n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))### Instruction:
Summarize this support ticket in one sentence.
### Response:
Customer is locked out of their account after the latest app update and needs a manual password reset.Run this against several prompts pulled from your held-out test split, not just training examples. A model that only performs well on data it already saw during training is a model that’s overfit, not one that’s learned the task.
Step 11: Merge the LoRA Weights Into the Base Model
For deployment, it’s usually simpler to merge the adapter into the base model and save a single set of weights, rather than loading the base model and adapter separately every time you serve a request.
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./lora-merged-model")
tokenizer.save_pretrained("./lora-merged-model")If your model was loaded in 4-bit for QLoRA training, merge into a full-precision copy rather than the quantized one, since merging directly into 4-bit weights degrades quality. Reload the base model in bf16 (without the BitsAndBytesConfig), attach the trained adapter with PeftModel.from_pretrained, and merge from there if you skipped straight from Step 9 to Step 11.
Step 12: Quantize and Export for Deployment
A merged fp16 or bf16 model works, but it’s larger than it needs to be for most serving setups. Converting to GGUF lets you run the result through llama.cpp-based tools, while vLLM and similar servers can load the merged Hugging Face checkpoint directly.
python llama.cpp/convert_hf_to_gguf.py ./lora-merged-model
--outfile my-model-q4.gguf
--outtype q4_0From here, deployment is a separate decision that depends on your traffic pattern. For high-throughput serving with continuous batching, our vLLM setup guide covers the steps. For running the model locally on your own machine without a serving cluster, see how to run a local LLM with Ollama, which can load a GGUF export directly.
Step 13: Push Your Model to the Hugging Face Hub
Publishing your adapter or merged model to the Hub makes it easy to pull onto a serving machine later, and to share it with teammates without copying files around manually.
huggingface-cli login
merged_model.push_to_hub("your-username/my-lora-finetune")
tokenizer.push_to_hub("your-username/my-lora-finetune")You can also push just the adapter weights instead of the merged model, which produces a far smaller upload (typically tens of megabytes rather than gigabytes) and lets anyone apply your adapter to their own copy of the base model with PeftModel.from_pretrained.
Complete Working Project: The Full Training Script
Here’s every step above combined into a single, runnable script. Save it as train_lora.py, place your training_data.jsonl file in the same directory, and run python train_lora.py from your activated virtual environment.
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig
MODEL_NAME = "meta-llama/Llama-3.2-3B-Instruct"
# 1. Load and format the dataset
dataset = load_dataset("json", data_files="training_data.jsonl", split="train")
def format_example(example):
return {"text": f"### Instruction:n{example['instruction']}nn### Response:n{example['response']}"}
dataset = dataset.map(format_example)
dataset = dataset.train_test_split(test_size=0.1)
# 2. Load the base model in 4-bit
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME, quantization_config=bnb_config, device_map="auto"
)
# 3. Attach a LoRA adapter
model = prepare_model_for_kbit_training(model)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# 4. Train
training_args = SFTConfig(
output_dir="./lora-output",
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
num_train_epochs=3,
learning_rate=2e-4,
logging_steps=10,
save_steps=100,
bf16=True,
max_seq_length=1024,
dataset_text_field="text",
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
)
trainer.train()
# 5. Merge and save
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./lora-merged-model")
tokenizer.save_pretrained("./lora-merged-model")
print("Done. Merged model saved to ./lora-merged-model")Evaluating Your Fine-Tuned Model Before You Ship It
A model that looks good on three example prompts is not the same thing as a model that’s ready for production. Before you deploy anything, run a structured evaluation pass rather than trusting a quick vibe check.
Start with your held-out test split from Step 5, the examples the model never saw during training. Run every one of those prompts through both the base model and your fine-tuned adapter, then compare the two sets of outputs side by side. For structured tasks like classification or extraction, score accuracy directly against the known correct answer. For open-ended tasks like tone or style, a simple blind comparison, where you (or a colleague) pick the better output without knowing which model produced it, catches problems that automatic metrics miss.
Two numbers are worth tracking specifically. Evaluation loss, computed automatically if you passed an eval_dataset to SFTTrainer as shown in Step 8, should track training loss reasonably closely. A large, growing gap between the two is the clearest sign of overfitting. Perplexity on the held-out set gives a rougher but faster secondary signal, and most evaluation harnesses built on Hugging Face’s evaluate library compute it in a few lines.
Finally, test the model on prompts that are deliberately outside its training distribution, not just harder versions of the same task. A support-ticket summarizer should still hold a basic conversation without breaking character or looping on the training format. If it can’t, the adapter has overfit narrowly enough that it’s likely to misbehave the first time a real user sends something unexpected.
5 Common Pitfalls When Fine-Tuning With LoRA and QLoRA
1. Setting the learning rate too high
LoRA tolerates higher learning rates than full fine-tuning, but 5e-4 and above frequently causes loss to spike or diverge on smaller models. Start at 2e-4 and adjust down if the loss curve looks unstable rather than starting high and hoping for the best.
2. Targeting the wrong layers
Attaching LoRA only to q_proj and v_proj is a common shortcut from older tutorials, and it often produces an adapter that barely changes output. Include all four attention projections at minimum, and add the MLP layers when you need a stronger behavior shift.
3. Training on a dataset that’s too small or too repetitive
A few dozen near-identical examples will teach the model to repeat one pattern regardless of the prompt. That’s overfitting, and it can bleed into unrelated tasks the model previously handled fine. Vary your examples in structure, not just wording.
4. Leaving the pad token unset
Several open-weight models ship without a defined pad token. If you don’t set one explicitly, batched training can silently produce ragged, incorrectly masked sequences. Set tokenizer.pad_token = tokenizer.eos_token as a default unless the model card says otherwise.
5. Skipping the evaluation split
It’s tempting to throw every example at training when you only have a few hundred. Resist it. Without a held-out set, you won’t know your adapter has overfit until it’s already deployed and producing repetitive or brittle answers in production.
6. Merging a 4-bit model directly
Calling merge_and_unload() on a model that’s still quantized to 4-bit produces a degraded merge. Reload the base model at full precision before merging, or use Unsloth’s merge path, which handles the dequantization step automatically.
Troubleshooting: 9 LoRA and QLoRA Errors and How to Fix Them
| Symptom | Likely Cause | Fix |
|---|---|---|
| CUDA out of memory during training | Batch size or sequence length too high for available VRAM | Lower per_device_train_batch_size, raise gradient_accumulation_steps, enable gradient checkpointing |
| Loss barely moves from its starting value | Learning rate too low, or target_modules missing key layers | Raise LR toward 2e-4, confirm target_modules covers attention projections |
| Loss turns into NaN mid-training | Learning rate too high, or unstable fp16 training | Switch to bf16, lower the learning rate, set max_grad_norm for gradient clipping |
| “Target modules not found” error | Layer names don’t match this model’s architecture | Print model.named_modules() to find the correct linear layer names for this base model |
| Output is gibberish after merging | Merged directly from a 4-bit quantized model | Reload the base model at bf16/fp16, then merge the adapter into that copy |
| bitsandbytes import fails or has no GPU support | Wrong build for your installed CUDA version | Reinstall bitsandbytes matching your CUDA driver, confirm torch.cuda.is_available() |
| Training is extremely slow on a consumer GPU | No gradient checkpointing, or fp32 fallback instead of bf16 | Enable bf16, use Unsloth’s fused kernels, shorten max_seq_length |
| Adapter loads but output looks unchanged | Base model loaded without the trained adapter attached | Confirm PeftModel.from_pretrained() points at your saved adapter directory |
| Hugging Face Hub push fails with a 401 error | Not authenticated, or token lacks write access | Run huggingface-cli login, or set an HF_TOKEN with write permission |
Advanced Tips for Production LoRA Workflows
Once the basic loop works, a few refinements separate a weekend project from something you’d actually put in front of users.
- Try DoRA for a quality bump. Weight-Decomposed Low-Rank Adaptation, supported directly in PEFT, splits weight updates into magnitude and direction components. It often closes more of the gap to full fine-tuning than standard LoRA at the same rank, at a small additional compute cost.
- Consider rank-stabilized LoRA (rsLoRA) at higher ranks. Standard LoRA scaling can make training unstable once you push rank well above 32. rsLoRA adjusts the scaling factor so higher ranks train more predictably.
- Serve multiple adapters on one base model. Because LoRA weights are small and separate from the frozen base, serving frameworks can hot-swap adapters per request, letting one deployed base model support several fine-tuned personalities or tasks without duplicating the full model in memory.
- Merge multiple adapters when you need combined skills. PEFT supports weighted adapter merging, so you can combine a “tone” adapter with a “domain knowledge” adapter rather than retraining from scratch every time requirements change.
- Automate a small evaluation set before every deploy. Even ten to twenty held-out prompts, scored consistently, catch regressions that a quick manual glance at outputs will miss.
Sebastian Raschka’s independent research notes on fine-tuning cover several of these tradeoffs in more depth, including rank selection and how LoRA compares with full fine-tuning and RLHF in practice, in his practical tips on fine-tuning LLMs.
Frequently Asked Questions
What’s the actual difference between LoRA and QLoRA?
LoRA trains small adapter matrices on top of a frozen base model that stays at its original precision, typically bf16 or fp16. QLoRA does the same thing, but first compresses the frozen base model to 4-bit precision, which cuts memory use further at a small cost in training speed. If your GPU has 24GB or more, plain LoRA works fine. Below that, QLoRA is usually what makes the run fit at all.
Do I need a GPU to fine-tune with LoRA?
For any model in the billions-of-parameters range, yes. CPU training is technically possible but slow enough to be impractical for anything beyond a toy example. If you don’t own a suitable GPU, renting one by the hour from a provider like RunPod is the standard workaround, and it’s inexpensive for a single training run.
How much does it cost to fine-tune a 7B-8B model?
On a rented RTX 4090 at roughly $0.34-$0.69 an hour, a QLoRA run of a few thousand examples for 2-3 epochs typically finishes in a few hours, putting the compute cost in the low single digits of dollars. Skip the 4-bit quantization and run full LoRA instead, and the number climbs: LeanML’s May 2026 estimate for 7B-13B models puts full LoRA training cost at roughly **$50 to $500**, depending on epoch count and which GPU tier you rent. The bigger cost is usually your own time spent preparing a clean dataset.
How long does LoRA fine-tuning actually take?
For a 3B-8B model with a few thousand training examples, expect somewhere between 30 minutes and a few hours depending on sequence length, batch size, and whether you’re using Unsloth’s faster kernels. Larger base models and longer max_seq_length values both extend training time substantially.
Can I fine-tune a closed model like ChatGPT or Claude this way?
No. LoRA and QLoRA require direct access to model weights, which closed, API-only models don’t expose. Some providers offer their own hosted fine-tuning products through their APIs, but that’s a different process from the local weight-level adapter training this tutorial covers, and it only works with open-weight models like Llama, Qwen, Gemma, Mistral, and DeepSeek’s distilled checkpoints.
What LoRA rank should I actually use?
Start at r=16 with alpha=32. It’s a well-tested default across most model families and task types. Move up to 32 or higher if the model isn’t learning the task well enough after several training runs, and move down to 8 if you’re seeing overfitting on a small dataset.
How many training examples do I actually need?
A few hundred well-written examples can shift a narrow behavior like tone or output format. Plan for 1,000 to 10,000 examples for genuine instruction-following or domain adaptation. Consistency in formatting matters more than raw volume, a smaller clean dataset usually outperforms a larger messy one.
Can I run my fine-tuned model in Ollama or vLLM afterward?
Yes, once you’ve merged the adapter into the base model. Export to GGUF for Ollama and similar llama.cpp-based tools, or point vLLM directly at your merged Hugging Face-format checkpoint for high-throughput serving. Both paths are covered in our Ollama and vLLM setup guides.
Does fine-tuning with LoRA hurt the model’s general abilities?
It can, if the adapter is trained too aggressively on a narrow dataset. Because LoRA only touches a small slice of parameters and leaves the rest of the base model frozen, catastrophic forgetting is milder than with full fine-tuning, but it isn’t impossible. Keeping the learning rate reasonable, avoiding excessive epochs, and testing on out-of-distribution prompts (as covered in the evaluation section above) all reduce the risk. If you need the model to retain broad general skill while also picking up a narrow one, a lower rank and fewer epochs is usually safer than pushing hard for maximum task performance.
What’s the difference between a LoRA adapter and DoRA?
DoRA (Weight-Decomposed Low-Rank Adaptation) is a variant of LoRA, not a replacement workflow. It decomposes each frozen weight into a magnitude component and a direction component, then applies low-rank updates to the direction while learning the magnitude separately. PEFT supports it as a drop-in option on the same LoraConfig class, set use_dora=True and the rest of this tutorial’s code applies unchanged. It typically costs a little more compute per step in exchange for results closer to full fine-tuning at the same rank.
