Fine-tuning stopped being a research-lab luxury somewhere around 2024, and by mid-2026 it’s a normal Tuesday-afternoon task for any engineer with a decent GPU and a few hundred good examples. Sivaro’s July 2026 cost analysis puts a number on that shift: a standard 7B LoRA fine-tune that ran about $120 in 2024 now costs roughly $45. Tools like Unsloth have cut VRAM requirements enough that a 7B or 8B model now trains on a single consumer card instead of a rented cluster. This guide walks through the full process of fine-tuning an open-source LLM with LoRA and QLoRA, from checking your CUDA setup to exporting a merged GGUF file you can run in Ollama. By the end, you’ll have trained a small support-chatbot model on your own dataset, tested it, and packaged it for deployment.
Budget about 90 minutes if you’re following along on a rented cloud GPU, longer if you’re downloading large model checkpoints on a slow connection. You don’t need a machine learning background to finish this. You do need to be comfortable in a terminal, willing to read a training log, and patient enough to wait out one full training run before declaring victory. Everything here uses tools that are free to download. The only real expense, if you don’t already own a capable GPU, is a few dollars of rented compute time, covered in detail later in this guide.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
Why Fine-Tune an LLM Instead of Relying on Prompting or RAG Alone
Prompting and retrieval-augmented generation solve real problems, but neither one changes how a model actually behaves. A long system prompt can steer tone for a while, then the model drifts back toward its default voice halfway through a long conversation. RAG pipelines are excellent at injecting fresh facts a model was never trained on, but they don’t teach a model a consistent output format, a specific vocabulary, or a house style.
Fine-tuning attacks a different problem: repetition. If you need thousands of support tickets answered in the same tone, or a coding assistant that always returns a specific JSON schema, or a model that follows your company’s writing guidelines without a 2,000-word system prompt eating your context window on every request, fine-tuning gets you there more reliably than prompting ever will. It works by updating the model’s own weights, at least a small, targeted slice of them, rather than hoping instructions in the prompt hold for the whole conversation.
LoRA is an improved finetuning method where instead of finetuning all the weights that constitute the weight matrix of the pre-trained large language model, two smaller matrices that approximate this larger matrix are fine-tuned.
Databricks Engineering Blog, Efficient Fine-Tuning With LoRA Guide
That’s the mechanical trick that makes this practical on ordinary hardware instead of a data center.
Consider a real scenario: a support team fielding a few thousand tickets a week wants every reply to follow a specific structure (acknowledge, explain, resolve) in a specific tone. Prompting a general-purpose model with those rules works for the first few replies in a session, then quality drifts as the conversation grows. A model fine-tuned on a thousand examples of that exact structure holds the pattern indefinitely, because the pattern is now baked into its weights rather than living in a prompt it might partially ignore. The same logic applies to code generation tied to an internal API, structured data extraction into a fixed schema, or any task where consistency matters more than raw creativity. None of this replaces RAG for factual grounding, though. If your use case needs a model that knows this week’s inventory levels or yesterday’s support tickets, pair a fine-tuned model with a retrieval layer rather than trying to bake constantly-changing facts into its weights. Our guide to building a RAG pipeline covers that half of the stack in detail. The two techniques solve different problems and often show up together in the same production system.
Prerequisites: Hardware, Software, and Accounts You Need
You don’t need a data center to follow this guide, but you do need to check a few boxes first. Here’s the full list before you open a terminal.
Hardware: an NVIDIA GPU with at least 8GB of VRAM handles QLoRA fine-tuning on smaller 7B-class models comfortably. If you’re on Apple Silicon or don’t own a discrete GPU, skip ahead to the cloud GPU pricing section later in this guide and rent one by the hour instead. If you’d rather buy dedicated local AI hardware than rent by the hour, Nvidia’s DGX Spark and RTX Spark both target this exact workload, with up to 128GB of unified memory, enough to fine-tune models far larger than a typical consumer GPU could handle.
Software: Python 3.10 or 3.11, a recent NVIDIA driver, and CUDA 12.1 or newer. Install PyTorch through the official instructions at pytorch.org rather than a generic pip install, since the CUDA build has to match your driver version.
Accounts: a free Hugging Face account to download gated model weights and, optionally, push your finished model back to the Hub. If you’re renting compute instead of using local hardware, you’ll also want an account with a GPU cloud provider.
| Model Size | Min VRAM (QLoRA 4-bit) | Min VRAM (Standard LoRA 16-bit) | Example Consumer GPU |
|---|---|---|---|
| 3B | ~4-6GB | ~8GB | RTX 3060 12GB |
| 7B-8B | ~6-8GB | ~16GB | RTX 4060 Ti 16GB |
| 13B | ~10-12GB | ~28GB | RTX 4090 24GB |
| 27B-34B | ~18-20GB | ~60GB+ | RTX 5090 32GB or A100 40GB |
| 70B | ~40-48GB | 140GB+ | A100 80GB or multi-GPU |
Treat those VRAM figures as planning estimates, not guarantees. Actual use shifts with sequence length, batch size, and which framework is doing the loading. QLoRA’s savings come from loading the frozen base model in 4-bit precision with double quantization, a technique that cuts memory use by roughly 75% memory savings relative to full 16-bit precision while keeping accuracy close to the unquantized original.
Time and cost: budget 90 minutes end to end, including install time. If you’re renting a GPU, expect somewhere between $1 and $6 in compute costs for the walkthrough in this guide, based on published hourly rates covered later in this piece. That figure is just raw GPU time, not a full production program: a May 2026 benchmark from YuSMP Group pegs end-to-end LoRA fine-tuning programs across the 7B-70B range at $30,000 to $180,000 once data preparation, engineering time, and evaluation are included, with GPU compute itself accounting for only $200 to $6,000 of that total.
Step 1: Verify Your GPU, CUDA, and Driver Setup
Before installing anything else, confirm your GPU is visible and CUDA is actually working. Open a terminal and run:
nvidia-smi
python3 -c "import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"If nvidia-smi doesn’t return a GPU, stop here and fix your driver installation before continuing. The PyTorch check should print a version number, then True, then your GPU’s name. If torch.cuda.is_available() returns False despite nvidia-smi working, you almost certainly installed the CPU-only build of PyTorch. Reinstall it using the CUDA-specific command from pytorch.org that matches the CUDA version listed in the top-right corner of the nvidia-smi output.
Step 2: Create an Isolated Python Environment
Fine-tuning pulls in a specific, sometimes fragile combination of torch, CUDA bindings, and quantization libraries. Installing them into your system Python invites version conflicts with every other project on your machine. Create a dedicated virtual environment instead.
python3 -m venv finetune-env
source finetune-env/bin/activate
python -m pip install --upgrade pipKeep this environment active for the rest of the tutorial. If you close your terminal and come back later, just run source finetune-env/bin/activate again before picking up where you left off.
Step 3: Install Unsloth, Transformers, PEFT, TRL, and Bitsandbytes
With your environment active, install the core stack. Unsloth wraps Hugging Face’s Transformers, PEFT, and TRL libraries with custom CUDA kernels that speed up training and cut memory use, so it needs those libraries installed alongside it.
pip install unsloth
pip install --upgrade transformers peft trl bitsandbytes accelerate datasetsUnsloth’s documentation confirms this is the standard local install path, and its maintainers report the optimized backend needs roughly 70% less VRAM than a standard LoRA pipeline while running about twice as fast. Skip pinning exact version numbers in a requirements file until after you’ve confirmed everything works. This stack moves fast enough that pinning too early tends to cause more problems than it prevents.
To perform QLoRA, all that is needed is the following: 1. Load the model to GPU memory in 4-bit (bitsandbytes enables this process).
Databricks Engineering Blog, Efficient Fine-Tuning With LoRA Guide
That single step, handled by bitsandbytes, is what lets a 24GB consumer GPU handle a model that would otherwise need 40GB or more in full precision.
Step 4: Choose a Base Model to Fine-Tune
| Model | Parameters | License | Best For |
|---|---|---|---|
| Llama 3.1 / 3.2 | 1B-8B (dense) | Meta’s custom Llama license | General-purpose, largest community ecosystem |
| Qwen2.5 | 0.5B-7B+ | Apache 2.0 (most sizes) | Multilingual, coding, long context |
| Gemma 2 | 2B / 9B / 27B | Google’s Gemma license | Efficient inference, smaller footprints |
| Phi-3.5 | 3.8B (mini) | MIT | Small footprint, reasoning-focused tasks |
| Mistral 7B | 7B | Apache 2.0 | Fast, strong general baseline |
Llama 3.1 and 3.2 remain a safe default thanks to the largest ecosystem of guides, quantized versions, and community LoRA adapters to learn from. Qwen2.5 punches above its weight on coding and multilingual tasks and ships under a permissive license for most sizes, and the fine-tuning community keeps pushing that family further: a September 2026 fine-tune called ThinkingCap-Qwen3.8-27B cut reasoning-token usage by 37.2% across twelve benchmarks without giving up accuracy, evidence that a well-targeted adapter can trim inference cost as much as it changes behavior. Gemma 2 and Phi-3.5 both target efficient inference on smaller hardware, which matters if you plan to deploy the finished model on a laptop rather than a server.
For this walkthrough, we’ll fine-tune an 8B-class instruct model, a size that fits QLoRA training on a single 16GB to 24GB GPU while still producing genuinely useful output. If you’re on tighter hardware, drop to a 3B model from the same family. The workflow below doesn’t change, only the model_name string in Step 6.
Small Models vs Large Models: Which Should You Fine-Tune First?
Start smaller than you think you need. A 3B or 7B model trains faster, iterates faster, and makes it much easier to tell whether a bad output came from your dataset or your hyperparameters. Once your pipeline produces a model that behaves the way you want on a small base, scaling the same LoRA config up to a 13B or larger model is mostly a matter of more VRAM and more training time, not a redesign.
For a wider comparison of the open-weight models available right now, including context window and licensing differences, see our breakdown of Phi-4 Mini vs Gemma 3 vs Llama 3.2. If your use case leans toward coding or multilingual data, our DeepSeek V4 vs GLM-5.2 vs Qwen comparison covers pricing and benchmark differences between those three families, though that lineup has already moved on: Z.ai released a fine-tunable GLM-5.3 on August 14, 2026, reusing GLM-5.2’s 743-billion-parameter base after additional post-training, and the coding gains show up directly in the benchmarks, with GLM-5.3 lifting its Terminal-Bench 3.0 score from 4.6 to 28.3 by October 2026. Together AI added official GLM-5.3 fine-tuning support on October 1, 2026, so you no longer need to self-host that base model to adapt it.
Step 5: Prepare and Format Your Training Dataset
Your dataset matters more than every hyperparameter in this guide combined. A clean set of 500 examples beats a noisy set of 50,000 nearly every time. Two formats dominate instruction fine-tuning right now: Alpaca-style, which separates instruction, input, and output into distinct fields, and ChatML-style, which structures data as role-tagged messages (system, user, assistant) closer to how chat models are actually served.
Here’s a small Alpaca-style example for a customer-support fine-tune, saved as JSONL (one JSON object per line):
{"instruction": "Explain why the customer's shipment is delayed and offer a resolution.", "input": "Order #48213 has been stuck in transit for 6 days.", "output": "I'm sorry about the delay on order #48213. Your package has been sitting at a regional hub since Tuesday. I've flagged it for priority handling and issued a $10 credit to your account for the inconvenience. You should see movement within 24 hours."}
{"instruction": "Summarize this bug report in one sentence for the engineering queue.", "input": "App crashes when tapping the export button on iOS 18 with more than 200 rows selected.", "output": "iOS export crashes on selections over 200 rows."}Once your examples are saved, load and format them with Hugging Face’s datasets library:
from datasets import load_dataset
dataset = load_dataset("json", data_files="support_tickets.jsonl", split="train")
def format_prompt(example):
example["text"] = (
f"### Instruction:n{example['instruction']}nn"
f"### Input:n{example['input']}nn"
f"### Response:n{example['output']}"
)
return example
dataset = dataset.map(format_prompt)Aim for at least a few hundred examples for a narrow style or format change, and a few thousand if you’re teaching the model a genuinely new task. Keep a held-out slice of 10-15% of your data separate from training so you can measure whether the model generalizes or just memorizes.
Step 6: Load the Model With 4-Bit QLoRA Quantization
from unsloth import FastLanguageModel
max_seq_length = 2048
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/llama-3.1-8b-instruct-bnb-4bit",
max_seq_length=max_seq_length,
dtype=None,
load_in_4bit=True,
)The load_in_4bit=True flag is doing the heavy lifting here. It loads the frozen base model using 4-bit NormalFloat quantization with double quantization on top, the technique behind QLoRA. Setting dtype=None lets Unsloth auto-detect the best compute dtype for your GPU: bfloat16 on newer Ampere and Hopper cards, float16 on older ones.
If this step throws an out-of-memory error immediately, before training even starts, your GPU doesn’t have enough VRAM for this model size even quantized. Drop to a smaller base model rather than fighting the batch size settings in the next step.
Step 7: Configure the LoRA Adapter
model = FastLanguageModel.get_peft_model(
model,
r=16,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha=16,
lora_dropout=0,
bias="none",
use_gradient_checkpointing="unsloth",
random_state=3407,
)Rank (r) controls how many trainable parameters the LoRA adapter gets. Sixteen is a reasonable starting point for style and format adaptation, thirty-two or sixty-four if you’re teaching more complex behavior. Set lora_alpha equal to the rank or at least twice the rank, and start with a learning rate around 2e-4 for most LoRA and QLoRA runs. target_modules lists which weight matrices inside each transformer layer get an adapter. The seven modules above, covering both attention and the MLP block, is the configuration most guides converge on for instruction fine-tuning.
Setting use_gradient_checkpointing=”unsloth” instead of the default True trades a small amount of speed for meaningfully lower memory use on long sequences, useful if your examples run past a few hundred tokens each.
Step 8: Set Your Training Hyperparameters
A handful of settings decide whether this run produces a usable model or an overfit one:
- Learning rate: start at 2e-4 for LoRA and QLoRA. Drop to 1e-4 if loss spikes or becomes unstable.
- Epochs: one full pass over your dataset is often enough. Two or three epochs on a small dataset commonly overfits rather than improves the model.
- Batch size and gradient accumulation: these two settings behave as roughly equivalent trade-offs, so a batch size of 2 with 4 accumulation steps trains similarly to a batch size of 8, just slower and with less peak memory use.
- Optimizer: adamw_8bit keeps optimizer state in 8-bit, saving additional VRAM with negligible quality loss.
These aren’t arbitrary suggestions. They come directly from the fine-tuning guidance Unsloth and Hugging Face publish for their own tooling, and they hold up across most 7B to 13B instruction fine-tunes.
Step 9: Launch the Fine-Tuning Run With SFTTrainer
from trl import SFTTrainer, SFTConfig
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=dataset,
dataset_text_field="text",
max_seq_length=max_seq_length,
args=SFTConfig(
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
num_train_epochs=1,
learning_rate=2e-4,
logging_steps=10,
output_dir="outputs",
optim="adamw_8bit",
),
)
trainer_stats = trainer.train()Once training starts, TRL’s SFTTrainer prints a running log. Here’s what a representative pattern looks like, with loss trending down as the model adapts to your data:
{'loss': 1.8342, 'learning_rate': 0.0002, 'epoch': 0.08}
{'loss': 1.2107, 'learning_rate': 0.00018, 'epoch': 0.32}
{'loss': 0.9053, 'learning_rate': 0.00012, 'epoch': 0.64}
{'loss': 0.7481, 'learning_rate': 0.00004, 'epoch': 0.96}Watch the loss column, not the clock. A healthy run shows loss dropping steadily for the first 60-70% of training, then leveling off. If loss is still falling sharply at the very last step, you likely stopped too early. If it flattened out long before training finished, additional epochs probably won’t help much.
Step 10: Monitor Training Loss and Watch for Overfitting
Training loss alone doesn’t tell the whole story. It only measures how well the model fits data it has already seen. Pass an eval_dataset into SFTConfig alongside your training set, and TRL reports eval loss at whatever interval you set. When training loss keeps dropping but eval loss starts climbing, that’s the model memorizing your training examples instead of learning the underlying pattern. That’s the classic sign of overfitting, and it’s exactly why one epoch is the recommended starting point rather than three or four.
If you don’t have enough data to justify a separate eval set, watch the model’s actual outputs instead. Generate a handful of test responses every few hundred steps. If they start repeating training examples verbatim rather than producing new but on-style answers, pull back the epoch count or add more varied data.
Step 11: Test Your Fine-Tuned Model With Inference
FastLanguageModel.for_inference(model)
prompt = (
"### Instruction:nExplain why the customer's shipment is delayed.nn"
"### Input:nOrder #55190, 4 days late.nn### Response:n"
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))Compare the output against what the same prompt produces on the un-tuned base model. The difference should be obvious within a few sentences: tighter formatting, consistent tone, or whatever specific behavior your dataset targeted. If the fine-tuned model’s answer looks identical to the base model’s, something didn’t take. Double-check that FastLanguageModel.for_inference(model) actually ran and that you’re loading the adapter checkpoint, not the original base weights.
Step 12: Merge LoRA Adapters and Export to GGUF
# Merge LoRA weights into the base model
python -c "
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained('outputs')
model.save_pretrained_merged('merged-model', tokenizer, save_method='merged_16bit')
"
# Convert to GGUF with llama.cpp
python llama.cpp/convert_hf_to_gguf.py merged-model --outfile support-bot.gguf --outtype q4_k_mA LoRA adapter by itself is tiny, often under 200MB, because it only stores the low-rank update matrices, not the full model. To run it outside a Python training script, merge those adapter weights back into the base model, then convert the result to GGUF, the format llama.cpp and Ollama both expect. The outtype flag controls the final quantization level. q4_k_m gives a good balance of size and quality for most use cases, while q8_0 keeps more precision at roughly double the file size.
Once you have a .gguf file, deploying it locally through Ollama takes one more step: write a short Modelfile pointing at the GGUF path and run ollama create. Our Ollama setup guide walks through that half of the process if you haven’t installed Ollama yet.
Step 13: Evaluate, Benchmark, and Deploy Your Fine-Tuned Model
Before shipping anything, run a structured comparison rather than trusting your gut on a handful of prompts. Pull 20-30 examples from your held-out eval slice, run them through both the base model and your fine-tune, and score the outputs against a rubric relevant to your task: format compliance, tone match, factual accuracy, whatever matters for your use case. For classification-style tasks, a simple accuracy comparison works. For open-ended generation, either grade outputs manually or use a stronger model as a judge.
If the results hold up, push the merged model to the Hugging Face Hub with model.push_to_hub() so teammates can pull it without re-running training, or keep the GGUF file local and deploy it through Ollama, vLLM, or your existing inference server. Tag the release with the base model, dataset version, and hyperparameters you used. Six months from now, when you want to reproduce or improve this run, that metadata saves far more time than it costs to write today.
Putting It Together: The Complete Project Structure
Every code snippet above belongs to one continuous project. Laid out on disk, the support-chatbot fine-tune from this guide looks like the structure below once you’ve worked through all 13 steps.
finetune-support-bot/
├── finetune-env/ # virtual environment from Step 2
├── support_tickets.jsonl # Alpaca-style dataset from Step 5
├── train.py # loads model, config, and SFTTrainer from Steps 6-9
├── test_inference.py # inference script from Step 11
├── outputs/ # LoRA checkpoints saved during training
│ ├── checkpoint-100/
│ └── checkpoint-200/
├── merged-model/ # full merged weights from Step 12
└── support-bot.gguf # final quantized file for Ollama or llama.cpptrain.py is just Steps 6 through 9 concatenated into one file: the model-loading call, the LoRA config, the hyperparameters, and the trainer.train() call, in that order. Running python train.py from your activated virtual environment kicks off the whole fine-tuning job end to end. Keep test_inference.py as a separate script so you can reload any saved checkpoint and compare it against the base model without retraining anything. That separation matters once you’re iterating: you’ll rerun test_inference.py far more often than train.py, and keeping them apart means a bad test run never risks touching your training code.
Version each piece independently. A dataset revision, a hyperparameter change, and a new base model release are three different reasons to retrain, and tracking them separately (in a spreadsheet, a README, or a proper experiment tracker) makes it far easier to answer the question every fine-tuning project eventually faces: which run actually produced the model currently in production, and can you reproduce it.
Cloud GPU Pricing for Fine-Tuning in 2026
| GPU | Community Cloud | Secure Cloud |
|---|---|---|
| RTX 4090 24GB | $0.34/hr | $0.59/hr |
| A100 80GB PCIe | $1.19/hr | $1.39/hr |
| H100 80GB PCIe | $1.99/hr | $2.89/hr |
If you don’t own hardware that fits the VRAM requirements from earlier in this guide, renting by the hour is the practical option, and it’s cheaper than most people expect for a single fine-tuning run. RunPod’s published rates put an RTX 4090 at $0.34 an hour on its Community Cloud tier or $0.59 an hour on Secure Cloud, with an A100 80GB running $1.19 to $1.39 an hour and an H100 80GB at $1.99 to $2.89 an hour depending on tier. A 3-hour QLoRA run on a 7B model, well within the 2-4 hour range typical for this workload, costs roughly $1 to $6 depending on which GPU and tier you pick, a range echoed by IO.net’s June 2026 budget guide, which puts a 7B LoRA fine-tune at under $10 and a larger 70B QLoRA run at $15 to $30. Other providers price by the training run rather than the GPU-hour: Spheron’s July 2026 cost study cites one provider charging a flat $100 an hour per training run, with standard-tier inference afterward billed at $4 per million input tokens and $16 per million output tokens.
Peer-to-peer marketplaces like Vast.ai often undercut those rates further since you’re renting spare capacity directly from other GPU owners, though prices there shift by the hour based on live demand rather than a fixed rate card. Community Cloud tiers generally cost less than Secure Cloud because you’re sharing infrastructure with other renters, a trade-off that’s usually fine for a training job that takes a few hours and doesn’t need guaranteed uptime. AWS, which hiked its own EC2 GPU pricing twice in 2026, remains a common choice for teams already inside that ecosystem, though dedicated GPU rental platforms usually beat it on raw price per hour for a short job like this one.
Common Pitfalls When Fine-Tuning LLMs
- Training on too little data variety. Ten examples of the exact same sentence structure teaches the model to repeat that structure, not to generalize. Vary phrasing, length, and edge cases even within a narrow task.
- Running too many epochs on a small dataset. As covered in Step 10, one epoch is frequently enough. Watching eval loss, not just training loss, catches this before you waste a training run.
- Mismatched prompt templates between training and inference. If you trained on an Alpaca-style ### Instruction: format but query the model with a plain question at inference time, output quality drops noticeably. The format has to match on both ends.
- Setting the learning rate too high for the LoRA rank you chose. Higher ranks pack in more trainable parameters, which can tolerate a higher learning rate. Lower ranks need a gentler one, or loss becomes unstable within the first few hundred steps.
- Skipping the merge step before deployment. A raw LoRA adapter won’t run in Ollama, llama.cpp, or most inference servers on its own. It has to be merged into the base model first, then converted to GGUF or whatever format your serving stack expects.
- Fine-tuning for new factual knowledge instead of behavior. As covered earlier, fine-tuning is a poor tool for teaching a model facts that change often. Use RAG for that and save fine-tuning for tone, format, and task-specific behavior.
- Deleting the training environment before saving the adapter. This sounds obvious until it happens at 2 a.m. after a four-hour training run. Save checkpoints to persistent storage, not just the ephemeral disk on a rented GPU pod.
Troubleshooting Guide: Errors and Fixes
Fine-tuning fails in predictable ways. Here are the errors you’re most likely to hit, and the fastest fix for each one.
| Error or Symptom | Likely Cause | Fix |
|---|---|---|
| CUDA out of memory | Batch size or sequence length too high for available VRAM | Lower per_device_train_batch_size, raise gradient_accumulation_steps, reduce max_seq_length, or drop LoRA rank |
| Loss stuck at the same value | Learning rate too low, or wrong layers targeted | Raise learning rate toward 2e-4, confirm target_modules includes attention and MLP layers |
| Loss becomes NaN | Learning rate too high or fp16 instability | Lower the learning rate, switch to bf16 if your GPU supports it, enable gradient clipping |
| Model outputs gibberish after training | Tokenizer or chat-template mismatch between training and inference | Use the same prompt template and tokenizer settings at both stages |
| Great training loss, poor real-world answers | Overfitting on a small or repetitive dataset | Add variety to the dataset, drop to one epoch, expand dataset size |
| ImportError installing bitsandbytes | CUDA toolkit mismatch or outdated pip | Reinstall inside a fresh venv, upgrade pip, confirm the CUDA version matches the wheel |
| Model forgets general knowledge after fine-tuning | Catastrophic forgetting from a high rank, high alpha, or too many epochs | Lower LoRA rank, mix in general-purpose instructions, cut epochs to one |
| GGUF export fails in llama.cpp | Adapter wasn’t merged first, or llama.cpp is outdated | Run save_pretrained_merged() before conversion, update llama.cpp to the latest commit |
| Training is much slower than expected | Gradient checkpointing off or Unsloth’s optimized kernels not active | Set use_gradient_checkpointing=”unsloth”, confirm the model loaded through FastLanguageModel |
| Hugging Face download fails with a 401 error | Gated model requires accepting the license first | Accept the model’s license on huggingface.co, then run huggingface-cli login |
Advanced Tips: Multi-GPU Training, DPO, and Cost Control
Once basic SFT fine-tuning works reliably, a few techniques push results further, and the frontier keeps moving: OpenAI’s September 3, 2026 release of GPT-6 Astra leaned heavily on computer-use post-training rather than SFT alone, a reminder that the same preference-tuning ideas covered below are now standard practice even at the largest labs.
Direct Preference Optimization (DPO): The Step After SFT
Supervised fine-tuning teaches a model to imitate your examples. Direct Preference Optimization goes a step further by training on pairs of responses, a preferred one and a rejected one, so the model learns which of two plausible outputs you’d rather see. TRL’s DPOTrainer follows nearly the same setup as SFTTrainer, swapping in a preference dataset instead of instruction pairs. Most teams run SFT first to establish baseline behavior, then DPO on top to sharpen it.
Multi-GPU and Distributed Training
A single LoRA config scales to multiple GPUs through Hugging Face’s Accelerate library with minimal code changes, mainly launching your training script with accelerate launch instead of python directly. For models too large to fit on one card even in 4-bit, DeepSpeed ZeRO or fully sharded data parallel (FSDP) splits both the model and optimizer state across GPUs. Neither is necessary for the 7B-8B walkthrough in this guide, but both become relevant once you’re fine-tuning 30B-plus models.
Cost control tip: on rented cloud GPUs, spot or interruptible instances typically run 40-60% cheaper than on-demand pricing. Since a QLoRA fine-tune only takes a few hours and checkpoints save automatically at whatever interval you set, an interrupted spot instance costs you a resume, not a lost run. Set logging_steps and save_steps close together, every 10-20 steps for short runs, so an interruption never costs more than a few minutes of progress.
Choosing Rank Based on Task Complexity, Not Habit
Most tutorials default to rank 16 and move on, but the right value depends on what you’re teaching the model. A narrow style or formatting change, like the support-ticket example in this guide, rarely needs more than rank 8 or 16. Teaching a model a new skill, such as a specialized reasoning pattern or an unfamiliar output schema, benefits from rank 32 or 64, since a higher rank gives the adapter more room to represent complex changes. Going higher than that mostly adds training time and overfitting risk without a proportional quality gain. If you’re unsure where to start, train the same dataset at rank 16 and rank 32, compare the eval loss and a handful of manual outputs, and let the smaller rank win any tie. A smaller adapter trains faster, merges faster, and is easier to reason about when something goes wrong.
For more on the open-weight models and training hardware shaping this space, browse our full AI & Machine Learning coverage.
Frequently Asked Questions
How much does it cost to fine-tune an LLM in 2026?
With QLoRA and a personal GPU with at least 8GB of VRAM, fine-tuning a 7B model on a few thousand examples costs $0 beyond electricity. Renting cloud compute instead, RunPod prices an RTX 4090 at $0.34 an hour on its Community Cloud and an A100 80GB at $1.19 an hour, so a typical 2 to 4 hour QLoRA run on a 7B model lands between roughly $0.70 and $4.80; Spendark’s June 2026 analysis puts the same 7B LoRA run at a broader $40 to $200 per job once overhead is included, with a full (non-LoRA) 7B fine-tune on 8x A100 GPUs running $327 to $1,638. The gap widens further against closed-model APIs: AITOT’s May 2026 pricing guide found a 5-million-token fine-tune plus 100 million tokens a month of inference costs $248 on Fireworks-hosted Llama 4 8B versus $6,495 on OpenAI’s GPT-4o, and a smaller GPT-5 mini fine-tune runs $60 to train plus $108 a month in inference, or roughly $1,356 across a full year of use.
Do I need my own GPU to follow this tutorial?
No. Every step in this guide works on a rented cloud GPU. If you don’t own an NVIDIA card with enough VRAM, spin up a pod on RunPod or a similar provider, connect over SSH, and run the same commands.
What’s the actual difference between LoRA and QLoRA?
LoRA freezes the base model’s weights and trains two small low-rank matrices that approximate the changes you want, updating a fraction of a percent of the original parameters. QLoRA does the same thing but first loads the frozen base model in 4-bit precision, cutting VRAM use by roughly 75% through 4-bit NormalFloat and double quantization, so you can fine-tune bigger models on smaller GPUs. The gap shows up directly in the bill: Sivaro’s July 2026 guide puts a full fine-tune of an 8B model on an H100 at roughly $2,400 for a 40-hour run, versus about $480 for an 8B LoRA job that finishes in around 8 hours on the same hardware.
How many training examples do I actually need?
For style, tone, or narrow-domain adaptation, a few hundred to a few thousand well-written examples usually beat a sloppy 50,000-row dataset. Quality and consistency matter more than raw volume once you’re past a few hundred rows.
Can I fine-tune ChatGPT, Claude, or Gemini this way?
No. This tutorial applies to open-weight models you download and run yourself, like Llama, Qwen, Gemma, Phi, or Mistral. OpenAI, Anthropic, and Google offer their own hosted fine-tuning options for their closed models, but those use different tooling and pricing than the local LoRA and QLoRA workflow covered here, and that landscape keeps shifting faster than most teams can track: OpenAI shipped a fine-tuning-friendly successor, GPT-5.6-terra, on July 23, 2026, then began winding down its older fine-tuning platform entirely in October 2026, locking out new users just as it rolled out GPT-6 Sol and GPT-6 Luna on October 2, 2026. Google added its own moving target in the meantime, releasing Gemini 3.8 Flash on September 2, 2026 as a new hosted fine-tuning option. It’s a reminder that hosted fine-tuning options can be discontinued or replaced in ways a locally-trained open-weight model never will be.
Should I fine-tune or build a RAG pipeline instead?
They solve different problems. Fine-tuning changes how a model writes, formats, and behaves. RAG feeds a model fresh facts at query time without retraining anything. Many production systems in 2026 use both together: a fine-tuned model for tone and structure, paired with RAG for up-to-date knowledge.
How long does the fine-tuning step actually take?
On a single A100, a QLoRA run on a 7B model over a small dataset typically finishes in 2 to 4 hours. On a consumer RTX 4090 expect it to run longer, though Unsloth’s optimized kernels close much of that gap.
Can I run my fine-tuned model on a laptop or phone afterward?
Yes, once you merge the LoRA adapter into the base model and convert it to GGUF format. A quantized 7B or 8B GGUF model runs comfortably on a modern laptop through tools like Ollama or llama.cpp, and smaller fine-tunes can even run on high-end phones.
