Open-weight models close most of the gap with closed frontier models on narrow, well-defined tasks, but out of the box they don’t know your product, your support tone, or the exact output format your app expects. Full fine-tuning used to be the only fix, and it meant updating every one of a model’s billions of weights on a rack of GPUs most teams don’t have access to. LoRA fine tuning changed that math, and the evidence keeps accumulating: Open-Finance-Lab’s FinLoRA project, published May 2025, benchmarked multiple LoRA methods across several base LLMs on numerous financial datasets and found the adapted models closing much of the gap with far more expensive full fine-tuning runs, and the research pipeline hasn’t slowed down since, with newer revisions like EulerLoRA v2, submitted to arXiv in August 2026, and LoRA-DA, which reached a v3 revision on arXiv in June 2026 after its v2 landed that April. Low-Rank Adaptation freezes the base model and trains a small set of adapter matrices instead, often less than 1% of the parameters in the original model, and a March 2026 arXiv medical-domain study pushed that even further, fine-tuning Flan-T5-Large with LoRA while updating just 0.6% of its parameters. A 7B or 8B model typically needs around **14–16 GB of VRAM** for full-precision inference; QLoRA reduces *training* memory enough for fine-tuning on a single consumer GPU, but not because 1% of parameters by itself makes the full model fit in VRAM.
This tutorial builds one project start to finish: fine-tuning an open-weight instruction model into a support-ticket triage assistant that reads a raw customer message and returns structured JSON with a category, a priority level, and a one-line summary. Generic instruction-tuned models handle that kind of rigid output inconsistently, which makes it a realistic test of what LoRA fine tuning is actually good for. By the end you’ll have a working adapter, a merged model, a quantized export, and a local server you can query with curl.
Budget around 90 minutes to work through the 12 steps below, plus separate unattended time for the training job itself, which depends on your dataset size and hardware. Every tool used here is open source: Hugging Face Transformers, PEFT, TRL, and bitsandbytes.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What LoRA and QLoRA Actually Change Inside the Model
A pretrained model’s weights live in large matrices, one per attention and feed-forward layer. Full fine-tuning updates every value in every one of those matrices, which is why it needs so much GPU memory: the optimizer has to track gradients and states for billions of parameters at once. Every one of those tracked values also needs to be written back to disk in a checkpoint, which is part of why full fine-tuning checkpoints for an 8B model can run well past 30GB apiece. LoRA fine tuning takes a different approach entirely.
How LoRA Adds Small Adapter Matrices
Instead of touching the original weight matrix, LoRA freezes it completely and adds two small matrices next to it, commonly labeled A and B, that multiply together to produce a low-rank update. Only A and B get trained, and only they need gradient and optimizer memory, which is the entire source of LoRA’s memory savings over full fine-tuning.
“LoRA is an improved finetuning method where instead of finetuning all the weights that constitute the weight matrix of the pre-trained large language model, two smaller matrices that approximate this larger matrix are fine-tuned.”
Databricks Engineering Blog
The size of that update is controlled by a single number: the rank, written as r. A small r keeps the adapter tiny and fast to train but limits how much the model’s behavior can shift. A larger r gives the adapter more room to learn, at the cost of more trainable parameters and a higher risk of overfitting on a small dataset. Technical writer Iuliia Solomenikova summarizes the mechanism this way: “The core insight is deceptively simple: instead of updating all the weights in the model, you freeze the original weights and train small adapter matrices that sit alongside them,” as she wrote in a breakdown of LoRA in production. Because the adapter is so much smaller than the base model, saving a checkpoint after every epoch costs megabytes instead of gigabytes, which makes it cheap to keep several versions around and compare them later.
How QLoRA Adds 4-Bit Quantization on Top
QLoRA adds one more layer on top of that idea. Instead of loading the frozen base model at its native 16-bit precision, QLoRA loads it in 4-bit precision using a quantization format called NF4, then trains the LoRA adapter on top of the quantized weights while keeping the adapter’s own math in a higher-precision dtype like bfloat16. The base model’s memory footprint drops by roughly 4x compared with 16-bit loading, which is what makes fine-tuning an 8B or even 13B model realistic on a single 24GB consumer GPU instead of a data-center card. The original LoRA paper came out of Microsoft Research in 2021, and the QLoRA paper followed in 2023 from researchers at the University of Washington. Both remain the reference implementations behind the Hugging Face PEFT library used throughout this guide, and the rest of this tutorial leans on QLoRA specifically so the whole project fits on a single consumer GPU.
Prerequisites: Hardware, Software Versions, and Accounts You’ll Need
Get the environment right before touching training code. Fine-tuning jobs fail more often from a mismatched CUDA build or an undersized GPU than from anything in the actual script. It’s also worth pinning to a current release: Hugging Face shipped PEFT 0.18.0 in June 2026 with support for several newer adapter methods, and the library moves fast enough that version drift alone can break a script that worked fine a few months earlier.
For hardware, plan around a single GPU with at least 24GB of VRAM if you want to run QLoRA fine-tuning on a 7B-8B model locally. Without that, cloud GPU rental (covered later in this guide, with real pricing) costs less than most people assume for a one-off training run, and GPU-free options are starting to open up too: KTransformers v0.7.0 added LoRA fine-tuning support on compatible AVX512 x86 CPUs in August 2026, though it remains a niche path compared with the GPU workflow this guide walks through. Set aside at least 50GB of free disk space too, since the base model, checkpoints, and any merged or quantized exports all add up fast.
| Component | Minimum requirement | Notes |
|---|---|---|
| Python | 3.10 or newer | 3.11 recommended for library compatibility |
| PyTorch | Latest stable release, CUDA build | Match the CUDA build to your driver version |
| transformers | Latest release | Hugging Face’s model loading library |
| peft | 0.18.0 or newer | Required for current LoRA/QLoRA integration, on the 0.19.x line as of spring 2026 |
| bitsandbytes | Latest release | Handles 4-bit and 8-bit quantization |
| accelerate | Latest release | Manages device placement and mixed precision |
| trl | Latest release | Supplies SFTTrainer for supervised fine-tuning |
| GPU driver | CUDA 12.x compatible | Verify with nvidia-smi |
You’ll also need a Hugging Face account with an access token if you’re pulling a gated model like Llama 3.1, and optionally a Weights & Biases account if you want training curves logged somewhere other than your own terminal. If you’ve never pulled and run an open model before, our guide on how to run an LLM locally covers the basics of model downloads and inference before you add training on top.
Step 1: Set Up Your Python Environment
Isolate this project in its own virtual environment so package versions don’t collide with anything else on your machine.
python3 -m venv lora-ft
source lora-ft/bin/activate
pip install --upgrade pip
# Install PyTorch with CUDA support - check pytorch.org for the
# exact command that matches your installed CUDA version
pip install torch --index-url https://download.pytorch.org/whl/cu121
# Install the fine-tuning stack
pip install "peft>=0.18.0" transformers accelerate bitsandbytes trl datasets huggingface_hub
# Authenticate with Hugging Face (required for gated models like Llama 3.1)
huggingface-cli loginConfirm PyTorch can see your GPU before going further:
python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"That command should print True followed by your GPU’s name. If it prints False, stop here and fix your PyTorch/CUDA install before writing any training code, since every later step assumes a working GPU.
Step 2: Choose the Right Base Model for Fine-Tuning
The base model you start from sets a ceiling on what fine-tuning can achieve. A weak base model won’t turn into a strong one just because you trained a LoRA adapter on top of it, so it pays to start with the strongest model your hardware can actually run. Two things matter more than raw benchmark scores when picking a base model for a LoRA fine tuning project: how well the model already follows instructions in its base instruct-tuned form, and whether its license permits your intended use, since some open-weight releases restrict commercial deployment above a certain user count.
| Model family | Common sizes | Approx. VRAM for QLoRA fine-tuning | Good fit for |
|---|---|---|---|
| Llama 3.1 | 8B, 70B, 405B | 8B: ~16-24GB | General-purpose instruction following |
| Qwen2.5 | 0.5B up to 72B (7B/8B widely used) | 7B/8B: ~16-24GB | Multilingual and coding-leaning tasks |
| Mistral / Mixtral | 7B dense, 8x22B mixture-of-experts | 7B: ~16-24GB | Lightweight general-purpose tasks |
| Gemma 2 | Up to 27B | 9B class: ~20GB | Strong reasoning at a small footprint |
| DeepSeek (distilled) | 7B/8B up to 32B+ | 7-8B: ~16-24GB | Reasoning-heavy tasks |
For this tutorial’s ticket-triage project, Meta’s Llama-3.1-8B-Instruct is the practical choice: it fits on a single 24GB GPU with QLoRA, follows structured-output instructions reasonably well out of the box, and is well documented across the ecosystem. The gains from adapting it aren’t just theoretical, either: Open-Finance-Lab’s FinLoRA benchmark, published May 2025, found a LoRA adapter lifted Llama-3.1-8B’s accuracy by 37.69 points over the base model, part of a broader evaluation that tested multiple LoRA methods across several base LLMs on numerous financial datasets. If you’d rather skip local hardware entirely and fine-tune on a managed platform, AI Prompts Hub listed Llama 4 8B LoRA training at $0.40 per 1M tokens and Llama 4 70B LoRA training at $1.20 per 1M tokens on Together AI in its June 2026 pricing rundown, a figure UsagePricing corroborated that same month, and Together AI’s own rate card priced LoRA supervised fine-tuning at a flat $0.48 per 1M tokens for models up to 16B parameters as of August 2026, per HatchWorks, worth weighing against the rented-GPU costs covered later in this guide. Together AI also expanded what its platform can handle on longer documents: by October 2026 its Qwen3.8-27B LoRA SFT context window had grown from 32,768 to 131,072 tokens, and its LoRA DPO context window from 16,384 to 65,536 tokens, giving longer-context fine-tunes a managed option that doesn’t require 24GB-plus local hardware. If you’re deciding between families rather than following along with this exact project, our roundup of the best open-source LLMs and our head-to-head comparison of Llama, Qwen, and Mistral both go deeper into which model wins on which benchmark.
Step 3: Load the Model in 4-Bit With bitsandbytes
QLoRA’s memory savings come from a BitsAndBytesConfig object that tells Transformers to load the base model’s weights in 4-bit NF4 format instead of the usual 16-bit floats, while keeping compute in bfloat16 for numerical stability.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
model_id = "meta-llama/Llama-3.1-8B-Instruct"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=bnb_config,
device_map="auto",
)An 8B model stores roughly 8 billion parameters, so at 2 bytes per parameter in 16-bit precision, just loading the weights costs around 16GB of VRAM before you add optimizer states or activations. Dropping to 4-bit NF4 cuts that raw weight footprint to somewhere around 5-6GB, which is the difference between a training job that fits on a single consumer GPU and one that doesn’t fit at all.
Step 4: Prepare and Format Your Training Dataset
Fine-tuning quality depends more on dataset consistency than dataset size. Most narrow, single-task fine-tunes converge on a few hundred to a few thousand well-curated examples, and adding more low-quality examples on top tends to hurt more than it helps. For the ticket-triage project, each training example needs to match the exact structure you want the model to produce at inference time, formatted as chat messages so it lines up with Llama 3.1’s instruction-tuned chat template.
Save your dataset as JSONL, one example per line:
{"messages": [{"role": "system", "content": "You are a support ticket triage assistant. Respond with JSON only."}, {"role": "user", "content": "My payment failed twice and I was charged both times, please help fast."}, {"role": "assistant", "content": "{"category": "billing", "priority": "high", "summary": "Customer charged twice after failed payment attempts."}"}]}
{"messages": [{"role": "system", "content": "You are a support ticket triage assistant. Respond with JSON only."}, {"role": "user", "content": "The dashboard has looked a little outdated lately, any refresh planned?"}, {"role": "assistant", "content": "{"category": "feedback", "priority": "low", "summary": "Customer suggests refreshing dashboard UI."}"}]}Load it with the datasets library and split off a held-out evaluation set. Skipping this split is one of the most common mistakes in this whole process, covered in more detail further down.
from datasets import load_dataset
dataset = load_dataset("json", data_files="ticket_triage.jsonl", split="train")
dataset = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = dataset["train"]
eval_dataset = dataset["test"]
print(f"Train examples: {len(train_dataset)}, Eval examples: {len(eval_dataset)}")Step 5: Configure the LoRA Adapter
This is the step that defines what gets trained. LoraConfig sets the rank, the scaling factor alpha, and which layers receive an adapter. In practice, rank commonly lands on 8, 16, or 32, with 64 historically treated as the ceiling for larger or more demanding fine-tunes, though that ceiling is already moving on managed platforms: Together AI raised its maximum supported LoRA rank from 64 to 128 in October 2026, giving teams training bigger adapters more headroom than the typical local default. Alpha is typically set equal to r or to twice r, and target modules usually cover all seven of the attention and MLP projection layers rather than just the attention block, since skipping the MLP projections leaves a large share of the model’s capacity untouched.
from peft import LoraConfig, get_peft_model
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()On Llama-3.1-8B, wrapping those seven projection layers with r=16 adds somewhere around 42 million trainable parameters against a base model of roughly 8.03 billion. The printed output looks something like trainable params: 41,943,040 || all params: 8,030,261,248 || trainable%: 0.5224, though your exact numbers shift slightly with the target_modules list and rank you choose. Either way, you’re training well under 1% of the model.
Step 6: Set Training Arguments and Hyperparameters
TRL’s SFTConfig wraps the standard Transformers training arguments with a few additions for supervised fine-tuning. The learning rate matters more here than in most other settings: LoRA adapters tolerate, and generally need, a higher learning rate than full fine-tuning, commonly somewhere in the 1e-4 to 2e-4 range, occasionally pushed as high as 3e-4 for small datasets.
from trl import SFTConfig
training_args = SFTConfig(
output_dir="./llama31-8b-ticket-triage-lora",
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
num_train_epochs=3,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
logging_steps=10,
eval_strategy="epoch",
save_strategy="epoch",
bf16=True,
optim="paged_adamw_8bit",
max_seq_length=2048,
report_to="none",
)The batch size of 2 with 8 gradient accumulation steps gives an effective batch size of 16 while keeping actual VRAM use low, a common trick when training on a single GPU. paged_adamw_8bit keeps optimizer states compressed, which matters because even with a frozen base model, the optimizer still has to track state for every trainable adapter parameter.
Step 7: Launch Fine-Tuning With SFTTrainer
With the model, dataset, and training arguments defined, SFTTrainer ties everything together and handles the training loop, including applying the LoRA config to the model if you haven’t already.
from trl import SFTTrainer
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
peft_config=lora_config,
)
trainer.train()
trainer.save_model("./llama31-8b-ticket-triage-lora/final")A healthy run’s console output logs steadily decreasing loss values roughly every 10 steps, in a pattern like {'loss': 1.842, 'epoch': 0.3} early on, dropping toward {'loss': 0.61, 'epoch': 2.7} by the final epoch. The exact numbers depend entirely on your dataset and task, so treat the shape of the curve, steadily down, then flattening, as the signal to watch rather than any specific loss value.
Step 8: Monitor Training Loss and Catch Overfitting Early
Because eval_strategy="epoch" is set, the trainer scores your held-out eval set after every epoch alongside the training loss. Watch both numbers, not just one. Training loss alone will almost always keep dropping the longer you train, which tells you the model is memorizing, not necessarily learning something that generalizes.
- Training loss and eval loss dropping together: the adapter is learning the task normally.
- Training loss keeps falling while eval loss flattens or rises: classic overfitting, usually fixed by fewer epochs, a lower rank, or more dropout.
- Both losses stay flat near their starting value: the learning rate is likely too low, or the target modules aren’t covering enough of the model.
- Loss spikes or turns into NaN: usually a learning rate that’s too high, or a precision mismatch between the base model and the adapter.
If you set report_to="wandb" instead of "none" in Step 6, you get these curves plotted automatically instead of having to read raw log lines, which is worth the two-minute setup once your datasets grow past a few hundred examples.
Step 9: Evaluate the Fine-Tuned Model
Loss curves tell you whether training converged, not whether the model is actually good at the task. For general capability checks, the field leans on standard benchmarks like MMLU, IFEval, MT-Bench, Arena-Hard, and HellaSwag, typically run through the open-source lm-evaluation-harness project, and newer LoRA variants get put through the same style of testing: an August 2026 arXiv paper on AQLoRA reported results across BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA, the same commonsense-reasoning suite commonly used to validate general-purpose fine-tunes. The same style of head-to-head testing shows up in the March 2026 arXiv medical-domain study mentioned earlier: on Flan-T5-Large, its LoRA adapter reached 43.52 ROUGE-1 against 40.67 for the fully fine-tuned model, evidence that adapter-based tuning doesn’t just save memory, it can match or beat the full-parameter baseline outright. Domain-specific LoRA evaluations tell a similar story: in press coverage from August 11, 2026, Fastino and NVIDIA reported that a LoRA adapter on their Nemotron-3.5-Lightning finance base, Fastino-Nemotron-3.5, pushed FinQA accuracy from 15.86% to 59.23%, a 43.37-point jump, and lifted BizFinBench macro accuracy from 49.65% to 57.46%. NVIDIA’s own BioNeMo release notes show the same technique spreading well beyond finance and chat models: the August 2026 release added LoRA finetuning support for the ESM2 protein language model, underscoring how far LoRA-based adaptation now reaches into scientific and life-sciences work alongside text. The managed-platform side of the Nemotron lineup kept pace too: Baseten added managed LoRA tuning for NVIDIA’s Nemotron 3.5 Super VL model in October 2026, making that kind of domain adaptation available without owning the GPU fleet Fastino and NVIDIA used for their own benchmark. For a narrow task like ticket triage, a task-specific check is more useful: run the eval set through the model and measure whether the output is valid JSON, whether the category field matches, and whether the priority field matches.
from peft import PeftModel
test_ticket = "App crashes every time I try to export a PDF, this is blocking my whole team."
prompt = [
{"role": "system", "content": "You are a support ticket triage assistant. Respond with JSON only."},
{"role": "user", "content": test_ticket},
]
inputs = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt").to(model.device)
output = model.generate(inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(output[0][inputs.shape[1]:], skip_special_tokens=True))Run that same prompt through the base model before fine-tuning and compare the two outputs side by side. A typical base-model response for the PDF export example above reads something like “It looks like you’re experiencing a crash when exporting a PDF. This could be caused by several factors, including…”, helpful in tone but useless as structured input for a routing system. The fine-tuned adapter, by contrast, should return only {"category": "bug", "priority": "high", "summary": "PDF export crashes app, blocking whole team."} with no surrounding text. That gap, explanatory prose versus clean structured output, is the entire point of this fine-tune, and it’s worth checking on several held-out tickets rather than just one before you move on to merging.
Step 10: Merge LoRA Weights Into the Base Model
You can deploy a LoRA adapter two ways: load the base model and the adapter separately at inference time, or merge the adapter’s weights directly into the base model to produce a single standalone checkpoint. Merging is simpler for single-task deployment. Keeping adapters separate matters more if you plan to swap between several task-specific adapters on the same base model, covered in the advanced tips section below.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
model_id = "meta-llama/Llama-3.1-8B-Instruct"
base_model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
merged_model = PeftModel.from_pretrained(
base_model, "./llama31-8b-ticket-triage-lora/final"
).merge_and_unload()
merged_model.save_pretrained("./llama31-8b-ticket-triage-merged")
tokenizer.save_pretrained("./llama31-8b-ticket-triage-merged")Note that this reload uses the full 16-bit base model rather than the 4-bit quantized version from Step 3. Merging into a quantized model directly isn’t reliable, so pull the base model at full precision for this one step, even though everything upstream used 4-bit loading to save memory during training.
Step 11: Quantize and Export for Deployment
A merged 8B model still takes around 16GB on disk at 16-bit precision, more than you need for most serving setups. Converting it to GGUF format and quantizing it lets you run it through llama.cpp-based tools at a fraction of the size, with a small, usually acceptable, quality tradeoff. Our llama.cpp setup tutorial covers the conversion tooling in more depth if you haven’t used it before.
# From inside a cloned llama.cpp repository
python convert_hf_to_gguf.py ../llama31-8b-ticket-triage-merged
--outfile ticket-triage-f16.gguf
--outtype f16
./llama-quantize ticket-triage-f16.gguf ticket-triage-q4_k_m.gguf Q4_K_MQ4_K_M is a reasonable default: it cuts file size roughly in half again compared with an 8-bit quantization while staying close to full-precision quality for most tasks. If your triage assistant’s output quality noticeably drops after quantization, step up to Q5_K_M or Q6_K rather than troubleshooting the fine-tune itself, since the quality loss is coming from compression, not from the adapter.
Step 12: Serve Your Fine-Tuned Model in Production
For low-traffic or local use, Ollama can load your quantized GGUF file directly through a Modelfile. For higher-throughput production serving, vLLM’s OpenAI-compatible server is the more common choice, and it can load LoRA adapters directly without a separate merge step. Our comparisons of Ollama, LM Studio, and Jan and vLLM against Ollama break down the tradeoffs between the two approaches in more detail.
# Modelfile
FROM ./ticket-triage-q4_k_m.gguf
PARAMETER temperature 0.1
SYSTEM "You are a support ticket triage assistant. Respond with JSON only."ollama create ticket-triage -f ./Modelfile
ollama run ticket-triage "Refund never showed up after 10 business days, third time asking."If you’d rather skip merging and quantization entirely and serve the adapter directly, vLLM supports that path too, which is often faster to iterate on during development since you’re not re-exporting a GGUF file after every training run.
vllm serve meta-llama/Llama-3.1-8B-Instruct
--enable-lora
--lora-modules ticket-triage=./llama31-8b-ticket-triage-lora/finalLoRA vs QLoRA vs Full Fine-Tuning: Which One Should You Use?
Each method sits at a different point on the tradeoff between memory footprint and how much control you have over training precision. None of them is universally correct, the right pick depends on your hardware and how far the target task sits from the base model’s existing behavior.
| Method | Trainable parameters | Approx. VRAM (8B model) | Best for |
|---|---|---|---|
| Full fine-tuning | 100% (all ~8B) | 60GB+, often multi-GPU | Large domain shifts, teams with a big compute budget |
| LoRA (16-bit base) | Under 1% (tens of millions) | ~24-32GB | Most production adapters, faster iteration |
| QLoRA (4-bit base) | Under 1% (tens of millions) | ~16-24GB | Single consumer or prosumer GPU setups |
In practice, QLoRA is the default starting point for most teams fine-tuning a 7B-13B model on their own hardware, precisely because it removes the GPU budget as the limiting factor. Plain LoRA at 16-bit is worth the extra VRAM when you already have a large GPU sitting idle and want to shave some training time off, since dequantizing 4-bit weights on the fly adds a small amount of per-step overhead. The gap shows up starkly in real hardware requirements: in a September 2025 guide, Amir Teymoori clocked full fine-tuning at 8x A100 80GB GPUs running for 48 hours, versus a single A100 40GB finishing the equivalent LoRA job in about 4 hours. On the quality side, a September 2026 arXiv study found that once several LoRA variants are properly tuned, they tend to land within 1-2% of each other’s peak performance, suggesting the choice of variant matters less than getting the hyperparameters right. The parameter math tells the same story outside the language-model world: an August 2026 benchmark from AY Robots found a rank-32 LoRA adapter on the OpenVLA robotics model trained just 97.6 million parameters against 7,188.1 million for a full fine-tune, cutting peak memory to 59.7GB at batch size 16 in the process. Enterprise pricing comparisons echo the same split: Tencent Cloud Developers put 7B LoRA training at $50-$300 per run against $5,000-$15,000 for a full fine-tune in a June 2026 breakdown of Chinese enterprise workloads. Full fine-tuning still has a place for deep domain shifts, think adapting a general model into a specialized legal or medical assistant from a large proprietary corpus, but for most narrower tasks like the ticket-triage project in this guide, it’s paying for capability you won’t use.
Hardware and Cost Planning: What Fine-Tuning Actually Costs
If you don’t have a 24GB-plus GPU sitting around, renting one for a single training run is often cheaper than buying one. Independent estimates land in a similar range: GridStackHub pegged a 7B LoRA fine-tune on a single rented H100 at 4-8 hours and $7-$14 in a May 2026 analysis, AdaptiveRecall put QLoRA fine-tuning of a 7-8B model at $1-$5 per run on cloud GPUs versus $12-$72 for a 70B model in a July 2026 breakdown, while YuSMP Group’s May 2026 benchmark put a full production-grade LoRA program spanning 7B to 70B models at $30,000-$180,000 end-to-end once data preparation, engineering time, and infrastructure are counted in. Here’s what GPU rental actually costs on RunPod’s Secure Cloud tier, current as of mid-2026:
| GPU | VRAM | RunPod Secure Cloud, per hour | Good fit for |
|---|---|---|---|
| RTX 4090 | 24GB | $0.69 | 7B-8B QLoRA fine-tuning |
| A100 PCIe | 80GB | $1.39 | 13B-30B LoRA/QLoRA, larger batch sizes |
| A100 SXM | 80GB | $1.49 | Same as above, faster interconnect for multi-GPU jobs |
| H100 PCIe | 80GB | $2.89 | Faster training when turnaround time matters |
| H100 SXM | 80GB | $2.99 | Multi-GPU jobs on larger base models |
Those figures come from RunPod’s published pricing page for its Secure Cloud tier. Community Cloud listings on the same platform, and spot pricing on marketplaces like Vast.ai, regularly undercut these rates, though availability and reliability vary more at those lower prices. Other vendors’ numbers land in the same neighborhood: YuSMP Group’s May 2026 breakdown put raw GPU compute alone at $200-$1,500 for a 7B-13B LoRA job and $1,500-$6,000 once you scale up to a 70B model, Stratagem Systems priced full QLoRA training runs in July 2026 at $3-$10 over 2-4 hours on a rented A100 for a 7B model and $15-$30 for a 70B model, and AITot’s 2026 cost calculator listed 7B LoRA fine-tunes at $1-$50 taking 15 minutes to 2 hours, versus $50-$500 and 1-8 hours for a 70B model, in a May 2026 update. As a rough planning number, a QLoRA run over a few thousand examples for 3 epochs on a single RTX 4090 typically finishes within a few hours, which puts total compute cost for a single training run at well under $10 on rented hardware, a figure io.net’s August 2026 budget guide echoed for community-cloud GPU compute, before you count the time spent on data preparation. For teams that would rather skip infrastructure management entirely, Baseten’s managed LoRA tuning service for NVIDIA’s Nemotron 3.5 Super VL supports context windows up to 262,144 tokens as of October 2026, a turnkey alternative to the rented-GPU math above for long-document fine-tuning jobs.
5 Common Pitfalls When Fine-Tuning With LoRA/QLoRA
- Only targeting attention layers. Setting target_modules to just q_proj and v_proj is a common shortcut from older tutorials, but it leaves the MLP projections (gate_proj, up_proj, down_proj) untouched. Most tasks benefit from adapting all seven projection layers, not just the attention block.
- Setting rank too high for a small dataset. A rank of 64 on a dataset of a few hundred examples gives the adapter enough capacity to memorize your training set instead of generalizing from it. Start at r=16 and raise it only if evaluation results show the adapter is underfitting.
- Training on raw text instead of the model’s chat template. Instruction-tuned models like Llama-3.1-Instruct expect a specific chat format with role tags. Formatting your dataset as plain text instead of applying the tokenizer’s chat template breaks instruction-following after the adapter is merged.
- Reusing full fine-tuning learning rates. Learning rates that work for full fine-tuning, often in the 1e-5 to 5e-5 range, are too low for LoRA and will leave the adapter badly undertrained. LoRA generally wants something closer to 1e-4 to 2e-4.
- Skipping a held-out evaluation set. Watching only training loss hides overfitting completely, since training loss keeps improving even as the model’s real-world performance degrades. Always reserve 5-10% of your data for evaluation before you start training, not after.
Troubleshooting: 8 Fine-Tuning Errors and How to Fix Them
| Error or symptom | Likely cause | Fix |
|---|---|---|
| CUDA out of memory | Batch size or sequence length too large for available VRAM | Lower per_device_train_batch_size, raise gradient_accumulation_steps, enable gradient checkpointing |
| Loss becomes NaN after a few steps | Learning rate too high, or a precision mismatch | Lower the learning rate, confirm bnb_4bit_compute_dtype and bf16 settings match, add gradient clipping |
| Model output looks identical to the base model | Adapter not actually loaded at inference, or merge step skipped | Confirm you’re loading the PeftModel or merged checkpoint, not the raw base model |
| ImportError or CUDA mismatch from bitsandbytes | bitsandbytes build doesn’t match installed CUDA version | Reinstall bitsandbytes matching your CUDA toolkit version |
| Tokenizer padding errors during training | No pad token set for a causal LM tokenizer | Set tokenizer.pad_token = tokenizer.eos_token before training |
| Training is far slower than expected | Gradient checkpointing overhead, or the model isn’t actually on GPU | Check device_map placement and nvidia-smi utilization, then balance checkpointing against batch size |
| Eval loss rises while train loss keeps falling | Overfitting on a small dataset | Reduce epochs, lower the rank, increase lora_dropout |
| Gibberish output after GGUF quantization | Quantization level too aggressive for the model or task | Move up from Q2/Q3 to Q4_K_M or higher |
Advanced Tips for Production Fine-Tuning
Once the basic workflow above is working, a few extensions are worth knowing about as your fine-tuning needs grow past a single adapter, and the training stack itself keeps getting faster underneath you: LoRAFusion, accepted to EuroSys 2026 in October 2025, demonstrated up to a 1.96x end-to-end speedup by fusing operations across concurrent LoRA training jobs, Exxact reported in June 2026 that simply scaling a LoRA training job from one to four RTX PRO 4500 GPUs delivered a 3.86x speedup, and a July 2026 ACL 2026 paper on SLoRA reported up to a 12% accuracy gain and a 29% reduction in catastrophic forgetting on the TRACE benchmark compared with vanilla LoRA baselines.
- DoRA (Weight-Decomposed Low-Rank Adaptation). A newer variant supported directly in PEFT that decomposes weights into magnitude and direction components before applying the low-rank update, often producing results closer to full fine-tuning at the same rank. Enable it by setting use_dora=True in LoraConfig.
- Rank-stabilized LoRA (rsLoRA). Changes the alpha scaling formula to remain stable at higher ranks, useful if you’re pushing rank up past 32 and seeing training instability. Enable with use_rslora=True.
- Multi-adapter serving. Instead of merging and deploying one model per task, keep several task-specific adapters and swap them at inference time on top of the same frozen base model. vLLM’s –enable-lora flag, shown in Step 12, supports loading multiple named adapters and routing requests to the right one, which saves significant disk space and memory compared with hosting several fully merged models.
- Faster training with Unsloth. The Unsloth library reimplements key parts of the LoRA training loop with custom kernels, commonly cutting both training time and memory use compared with the stock Transformers plus PEFT stack, with minimal code changes required to adopt it.
- Continued fine-tuning from a merged checkpoint. If you need to add a second round of training later, you can train a fresh LoRA adapter on top of your already-merged model rather than starting from the original base weights, which lets you layer improvements without retraining from scratch.
- Expanding a small dataset with model-assisted labeling. If you only have a few dozen real examples, use a strong general-purpose model to draft additional candidate examples in the same format, then have a human review and correct each one before adding it to the training set. This scales a small seed dataset into a few hundred examples faster than writing every one by hand, as long as a person still checks the labels rather than training on unreviewed synthetic output.
Putting the Complete Project Together
It’s easy to lose the thread across 12 steps of setup, training, and deployment code, so here’s the full ticket-triage project laid out as a single checklist. Each item maps back to the step where you built it, and every file it references comes from a command earlier in this guide.
- A Python virtual environment with transformers, peft, bitsandbytes, accelerate, and trl installed (Step 1).
- Llama-3.1-8B-Instruct loaded in 4-bit through BitsAndBytesConfig (Step 3).
- A ticket_triage.jsonl dataset formatted as chat messages, split into train and eval sets (Step 4).
- A LoraConfig targeting all seven projection layers at r=16 (Step 5).
- An SFTConfig with a 2e-4 learning rate and a cosine schedule, saved to ./llama31-8b-ticket-triage-lora (Step 6).
- A completed SFTTrainer run, monitored for the training-loss-down/eval-loss-flat pattern that signals healthy convergence (Steps 7-8).
- A side-by-side base-model-versus-adapter comparison confirming the fine-tune actually improved JSON consistency (Step 9).
- A merged standalone checkpoint at ./llama31-8b-ticket-triage-merged (Step 10).
- A quantized ticket-triage-q4_k_m.gguf export, roughly a quarter the size of the merged 16-bit checkpoint (Step 11).
- A running Ollama or vLLM endpoint you can send real ticket text to and get structured JSON back (Step 12).
Swap the dataset and the system prompt in Step 4 for a different task, and the rest of this pipeline carries over largely unchanged. The same 12 steps work for a coding-style assistant, a document classifier, or a domain-specific chat tone, since none of the LoRA configuration or training logic in between is specific to ticket triage.
Frequently Asked Questions
How much data do I need to fine-tune with LoRA?
There’s no fixed number, but most narrow, single-task fine-tunes see solid results from a few hundred to a few thousand well-curated examples. Consistency in formatting and labeling matters more than raw volume.
Is QLoRA slower than regular LoRA?
Per training step, yes, slightly, because the 4-bit base weights need to be dequantized on the fly during the forward pass. In exchange, QLoRA uses substantially less VRAM, which usually matters more than the small per-step slowdown for anyone training on a single GPU.
Can I fine-tune on a laptop GPU?
If your laptop GPU has at least 16GB of VRAM, QLoRA fine-tuning of a smaller model (in the 3B-7B range) with a short sequence length and small batch size is realistic. Below that, cloud GPU rental is the more practical route.
Do I need to merge the adapter before deploying?
No. Merging simplifies deployment into a single standalone checkpoint, but serving frameworks like vLLM can load a base model and a LoRA adapter separately, which is often preferable if you’re running multiple task-specific adapters.
How is LoRA fine-tuning different from RAG, and when should I use each?
RAG retrieves relevant documents at query time and feeds them into the prompt, which suits tasks where the underlying facts change often. LoRA fine-tuning changes the model’s behavior and output style directly, which suits tasks like the ticket-triage example here, where you need a consistent format or tone rather than access to fresh external facts. The two combine well: a fine-tuned model that’s better at following your output format, pulling from documents retrieved through RAG.
Can I run multiple LoRA adapters on one base model at the same time?
Yes. This is one of LoRA’s practical advantages over full fine-tuning: because adapters are small, serving frameworks like vLLM can hold several of them in memory alongside a single copy of the base model and route each request to the right adapter.
What happens if I train for too many epochs?
Training loss keeps dropping, but eval loss flattens and then rises, a clear sign of overfitting. The model starts memorizing training examples rather than learning the general pattern behind them. Watching both loss curves, as covered in Step 8, is how you catch this before it wastes an entire training run.
Is full fine-tuning ever still worth it?
Occasionally, for large domain shifts backed by a big proprietary dataset and a serious compute budget. For most production use cases, including narrow task adapters, tone changes, and structured-output tasks like this tutorial’s example, LoRA or QLoRA gets you most of the benefit at a fraction of the hardware cost.
