Coder Legion - Recent posts tagged fine-tuning

Count your agent's narrow calls before you move any of them to a small model

Wed, 28 Oct 2026 13:46:09 +0000

A team decides its agent costs too much. Someone reads that small models are the future of agent work, picks a 3B model, and points the whole agent at it. Ticket classification holds up. The open-ended replies fall apart within a day, and the team rolls everything back.

The reason is structural. The team swapped the whole agent, while the savings live in individual calls.

Some calls repeat with little variation. Others need broad reasoning. Until you know which is which, you cannot know what a small model can take.

Many agent calls are narrow and repetitive

NVIDIA Research and Georgia Tech made this case in a 2025 position paper, Small Language Models are the Future of Agentic AI (arXiv 2506.02153). Many agent calls are narrow, specialized tasks. For those calls, the paper argues, small models are sufficient, better suited and more economical. It counts most models under about 10B parameters as small.

The cost gap is large. The paper states that serving a 7B model is 10 to 30 times cheaper in latency, energy and FLOPs than a 70B to 175B model. In a multi-step chain, that saving applies at every step.

Large models keep a role. The authors argue for heterogeneous agents. Small models take calls by default. A large model is called when general reasoning or open-domain dialogue matters.

The published percentages are estimates, not your numbers

The authors studied three open-source agents. They estimate that about 60% of MetaGPT's LLM queries could move to specialized small models. For Open Operator the estimate is about 40%. For Cradle it is about 70%.

Those numbers come from a position paper. They are not benchmark results. They also vary widely across three agents, and that spread is the useful part. Your agent's share depends on what your agent does.

So treat them as a reason to measure. Log a week of calls and count how many are narrow tasks that repeat with little variation.

Log every call that is not talking to a person

The paper's conversion recipe starts with logging. Capture the prompt, the output, the tool-call contents and the latency of every call that is not a direct exchange with a person.

The exchange with the user is usually the open-ended part. The calls behind it are where the repetition lives: classify the ticket, pull the order ID, pick the tool.

Record these as OpenTelemetry spans. Wrap the agent, each tool and the overall chain, so every call shows up under its parent. You need that structure later, when a bad answer has to be traced back to one step.

Cluster the calls into jobs

Once the logs fill up, group the calls into recurring tasks. Typical clusters are intent recognition, data extraction and document summarization.

Count each cluster. Its size tells you how much traffic a small model could take over.
Read a sample from each one. A cluster that looks narrow can still hide long or unusual inputs.
Strip sensitive data first. Remove PII, PHI and other sensitive fields before any call becomes a training example.

The paper's rule of thumb for fine-tuning is 10K to 100K examples per task. Check how close each cluster gets to that range before you plan a fine-tune for it.

Pick and train one small model per job

For each cluster, pick candidate models. Weigh capability, license and deployment footprint. Then test them on your own eval set, because a public benchmark measures general skill and your agent step is narrow.

Several current small models document tool calling on their model cards. They include Qwen3.5 (0.8B to 9B), Gemma 4 E2B and E4B, Ministral 3, Phi-4-mini and SmolLM3-3B. In that set, Qwen3.5-0.8B is the smallest with documented tool calling.

Fine-tune with LoRA or QLoRA on a single GPU. Another option is distillation: train the small model to mimic the large model's outputs on the task data. Either way, build each dataset to look like production traffic, with the same length, noise and label distributions.

Score every swap against the model it replaces

Before any traffic moves, run the small model and the model it replaces through the same evals. Use a metric that fits the job:

Router. Routing accuracy on a held-out set.
Extractor. Exact-match or structured-field accuracy.
Reasoning step. Faithfulness against the supplied context.

Then run end-to-end scenarios across the whole chain. Per-agent evals miss handoff bugs. One agent emits a malformed handoff token, the next silently drops the request, and the workflow returns a wrong but plausible answer. A full-chain test is what catches it.

Keep a fallback and retrain one piece at a time

Route the easy calls to the small model, and keep a frontier model as a fallback behind the same endpoint. Small models are less reliable on adversarial or out-of-distribution inputs, so low-confidence cases go to the stronger model.

The last step of the recipe is iteration. Retrain on fresh data as usage shifts. When one agent regresses, retrain or revert that one component and leave the others alone. Each change stays small and easy to roll back.

Pitfalls that keep showing up

Swapping the whole agent at once. The open-ended calls fail and take the narrow wins down with them.
Quoting someone else's percentage. The 40% and the 70% came from different agents. Yours needs its own count.
Training on raw logs. Logs carry PII and PHI. Strip them before a call becomes a training example.
Picking by leaderboard. The paper lists generalist benchmarks as one of the barriers to adoption. Your own eval set decides.
Skipping end-to-end tests. Each agent can pass its own suite while a handoff between them drops requests.

The model lineup, a LoRA sketch and the tracing setup are in this guide to small language models for agentic AI.

Count the calls first. A small model earns its place only on a job you have measured, scored and can roll back.

Fine-Tuning vs Prompt Engineering vs RAG: When to Use Each (And the Fourth Option Nobody Mentions)

Sun, 09 Aug 2026 14:33:21 +0000

You have an AI model that hallucinates with your company's data. You open a support ticket and ask the chatbot about your returns policy. The chatbot, powered by GPT-4 or Gemini 2.5, responds with a fabricated policy that sounds perfectly plausible but has nothing to do with your company's reality. Your boss stares at you. Your customer complains. You open Google and search "how to connect LLM to my data".

The top 10 results offer three contradictory answers: "use RAG," "do fine-tuning," "improve your prompt." All three are right. All three are wrong. Because the correct answer isn't any of the three in the abstract — it's the one that fits your specific use case. And there's a fourth option that almost nobody mentions, and which, in my experience, is the right call in more cases than the industry admits.

This article is the decision tree I wish I'd had when I started building the AI systems behind this blog. I've distilled it after implementing all four techniques in real production: Prompt Engineering across the entire Autopilot series, RAG in the Ops Copilot with Algolia, pure Tool Calling in the Obsolescence Radar, and experimental fine-tuning in industrial classification pipelines. It closes the trilogy that began with RAG: 7 Anti-Patterns and continued with MCP Protocol.

The Question Nobody Asks

Before choosing a technique, ask yourself this: Does the knowledge your LLM needs change, or is it static?

If the answer is "it changes frequently" (product documentation, inventory, prices, regulations), you need a technique that accesses data in real time without retraining the model. If the answer is "it's static or changes very slowly" (brand tone, formatting rules, domain nomenclature), you can consider techniques that incorporate that knowledge into the model.

This distinction is the first node of the decision tree. It seems obvious written this way. Yet most teams I've seen jump straight to whatever technique is trending (RAG in 2024, fine-tuning in 2023, prompt engineering always) without asking this fundamental question.

Option 1: Prompt Engineering — 80% of Cases

The uncomfortable truth the AI tooling industry doesn't want you to know: for 80% of use cases, a well-designed prompt is sufficient. You don't need RAG. You don't need fine-tuning. You need a system prompt that clearly defines the role, context, constraints, and expected output format.

When it's sufficient:

The required knowledge fits in the model's context window (Gemini 2.5 handles up to 1 million tokens; Claude up to 200K).
The task is generic but needs structure (drafting emails, summarizing documents, classifying text, generating code).
You don't need updated proprietary data — the model's general knowledge is enough.

Advanced techniques that make the difference:

Structured system prompts: Define the role ("You are a senior supply chain engineer"), constraints ("Always respond in technical English"), and output format ("Return a JSON with fields: analysis, recommendation, confidence").
Few-shot prompting: Include 3-5 correct input-output examples in the prompt. In the Autopilot series, CrewAI agents use few-shot to maintain style consistency across generated articles.
Chain-of-thought (CoT): Instruct the model to "think step by step" before giving the final answer. Dramatically improves accuracy in reasoning, calculation, and multi-step analysis tasks.
Prompt chaining: Break complex tasks into sequential subtasks, each with its own optimized prompt. This is exactly what CrewAI does with its agent architecture: each agent has a specialized prompt for its role.

Cost: Virtually zero (only API token cost). A well-designed prompt can take hours of iteration, but operational cost is minimal.

Fatal limitation: The context window has a limit. If you need the model to "know" about 10,000 documents from your knowledge base, you can't inject them all into the prompt. This is where RAG enters.

Option 2: RAG — Updatable Proprietary Knowledge

RAG (Retrieval-Augmented Generation) is the right answer when you need the LLM to respond about your proprietary knowledge and that knowledge updates frequently.

When it's necessary:

Product documentation, technical manuals, internal knowledge bases that update weekly or monthly.
The user can ask unpredictable questions about a broad document corpus (you don't know in advance which fragment the LLM will need).
You need citability: the answer must include the sources it draws from (critical for compliance, as we documented in the EU AI Act, Article 13 on transparency).

When NOT to use it: When data is structured (SQL tables, APIs with defined schemas) or when you need numerical precision. As I extensively documented in Anti-Pattern 7 of the RAG article, RAG over structured data generates narrative hallucinations where you need exact figures.

Correct architecture (summarized):

Semantic chunking (not fixed-length — Anti-Pattern 1)
Evaluated embeddings with your domain benchmark (Anti-Pattern 2)
Reranking between retriever and LLM (Anti-Pattern 3)
Generous context (top-10/15, not top-3 — Anti-Pattern 4)
Evaluation with RAGAS/DeepEval before production (Anti-Pattern 6)

Real cost: Moderate. The vector store (Pinecone, Weaviate, Algolia) has a monthly cost (€0-100 depending on volume), plus embedding cost (low) and generation cost (API tokens). In the Ops Copilot, the total RAG cost with Algolia was under €3/month for the blog's ~70 posts.

Real case at Datalaria: The Ops Engineering Copilot (Autopilot Part 8) uses RAG with Algolia Agent Studio to answer questions about blog content. Posts are indexed as semantic records (one record per section), and the copilot retrieves relevant fragments before generating the response. Works well for semantic search over free text.

Option 3: Fine-Tuning — The Scalpel, Not the Hammer

Fine-tuning is the most powerful technique and the most misused. It involves partially retraining a base model (Gemini, Llama, Mistral) with your own data so the model internalizes specific knowledge, style, or behavior.

When it's essential:

You need the model to adopt a very specific tone or style consistently (a brand with strict voice & tone, a domain with very particular technical jargon).
The task is highly specialized and generalist models don't solve it well even with advanced prompting (industrial defect classification, proprietary nomenclature entity extraction, specialized medical diagnosis).
You need to reduce latency and cost in production: a fine-tuned smaller model (7B-13B parameters) can match the quality of a large model (70B+) on your specific task, at a fraction of the cost and latency.

When NOT to use it (the most widespread myth):

Don't use fine-tuning to "teach the model data." Fine-tuning is not a database. If you need the model to know your product catalog, use RAG. Fine-tuning "burns in" behavioral patterns, not updatable facts.
Don't use fine-tuning if your knowledge changes frequently. Each update requires retraining, which can cost hours and hundreds of euros. RAG is instant: update the document and the retriever finds it immediately.

Modern tools:

LoRA (Low-Rank Adaptation): The standard technique. Instead of retraining the model's billions of parameters, LoRA trains only low-rank matrices "attached" to the model's layers. Reduces training cost by 90%+ and stores the fine-tuned model as a few MB of "adapters."
QLoRA: LoRA applied to a 4-bit quantized model. Enables fine-tuning 70B-parameter models on a single consumer GPU (24GB VRAM). Democratized fine-tuning for startups and teams without GPU clusters.
Vertex AI Tuning / OpenAI Fine-Tuning API: Managed services where you upload your training dataset (instruction-response pairs) and the platform runs the fine-tuning without you managing GPU infrastructure.

Real cost: Variable. Fine-tuning with LoRA on a 7B-parameter model with 10,000 examples costs €5-20 on cloud (Google Cloud, AWS). A 70B model can cost €50-200 per training session. Plus the cost of preparing the dataset (hours of human work). As we analyzed in The Hidden Economics of AI, fine-tuning's hidden cost isn't compute — it's training dataset curation.

Option 4: Tool Calling / MCP — The One Nobody Mentions

This is the option I discovered by elimination after RAG failed spectacularly on the Obsolescence Radar. Tool Calling means the LLM doesn't try to "know" the answer; instead, it knows who to ask — that is, which tool to execute to get the information with deterministic precision.

When it's the right option:

Data is structured (SQL databases, REST APIs, spreadsheets with schemas).
You need absolute numerical precision (financial calculations, inventory metrics, sensor data).
The operation requires actions, not just answers (create a ticket, send an email, run a query, call an external API).
You want to standardize connections between the LLM and tools to avoid vendor lock-in — exactly the problem solved by MCP (Model Context Protocol).

Architecture: The LLM (Gemini 2.5, Claude) acts as a semantic orchestrator: it understands the user's natural language intent, decides which tool(s) to execute, constructs the parameters, executes the tool(s), and interprets results for the user. Tools are deterministic Python functions (decorated with @tool in CrewAI) that execute precision operations: SQL queries to Supabase, supplier API calls, linear programming calculations with PuLP.

Cost: The lowest of all four options. You only pay for LLM tokens (typically few, since the prompt is short) and tool execution (SQL queries, API calls). In the Obsolescence Radar, the cost per complete execution (analyze a component, traverse the BOM graph, calculate financial impact, generate executive report) was under €0.02 per query.

Real case at Datalaria: The Agentic Obsolescence Radar uses Tool Calling exclusively. The LLM (Gemini 2.5 via CrewAI) understands the obsolescence alert in natural language, but all data operations — SQL query to the component catalog, BOM graph traversal, P&L calculation, PDF generation — are executed by deterministic Python tools. Result: executive reports in 4 seconds with 0% numerical hallucination.

The Decision Matrix

Criterion	Prompt Engineering	RAG	Fine-Tuning	Tool Calling
Initial cost	⭐ Minimal	⭐⭐ Low-medium	⭐⭐⭐ High	⭐⭐ Low
Operational cost	⭐ Low	⭐⭐ Medium	⭐ Low (small model)	⭐ Minimal
Precision (free text)	⭐⭐ Medium	⭐⭐⭐ High	⭐⭐⭐ Very high	⭐ N/A
Precision (structured data)	⭐ Low	⭐ Low	⭐ Low	⭐⭐⭐ Exact
Data freshness	⭐⭐⭐ Instant	⭐⭐⭐ Instant	⭐ Requires retraining	⭐⭐⭐ Real-time
Implementation effort	⭐ Hours	⭐⭐ Days-weeks	⭐⭐⭐ Weeks-months	⭐⭐ Days
Maintenance	⭐ Minimal	⭐⭐ Medium	⭐⭐⭐ High (data drift)	⭐⭐ Medium
Traceability (EU AI Act)	⭐ Difficult	⭐⭐⭐ High (citable sources)	⭐ Opaque (black box)	⭐⭐⭐ Total (deterministic)
Ideal use case	Generic tasks with clear instructions	Updatable proprietary text knowledge	Specific style/tone, ultra-specialized tasks	Structured data, numerical precision, actions

The 3-Question Framework

If the matrix seems dense, I've distilled a 3-question framework that resolves 90% of decisions:

Question 1: Does the data the LLM needs fit in the prompt?

Yes → Prompt Engineering. Inject context directly. Simpler, cheaper, faster.
No → Next question.

Question 2: Is the data free text or structured?

Free text (documentation, manuals, posts) → RAG. Semantic retrieval is superior for searching unstructured text.
Structured (SQL, APIs, tables, calculations) → Tool Calling. Deterministic tools that execute exact queries.
Both → Hybrid architecture (RAG for textual context + Tool Calling for structured data, as we proposed in the RAG article).

Question 3: Do you need a behavior or style the base model can't reproduce even with the best prompt?

Yes (unique brand tone, ultra-specific domain jargon, task that generalist models consistently fail) → Fine-Tuning on a base model.
No → Go back to Prompt Engineering and refine your prompt before considering more complex techniques.

What I Learned Implementing All Four

The most valuable lesson from operating these four techniques in production fits in a single sentence: always start with the simplest technique that could work.

The temptation is to jump straight to RAG or fine-tuning because they're more "sophisticated." But sophistication doesn't correlate with effectiveness. In the Autopilot, the majority of output quality comes from Prompt Engineering — carefully designed system prompts, few-shot examples, and Chain-of-thought. RAG added marginal value in the Ops Copilot for blog search. Fine-tuning wasn't necessary in any case. And Tool Calling was the transformative technique in the Obsolescence Radar, where RAG had failed.

The evaluation order should always be:

Prompt Engineering (hours, ~€0)
Tool Calling if data is structured (days, ~€0)
RAG if you need access to proprietary text (days-weeks, ~€3-50/month)
Fine-Tuning only if the previous three consistently fail (weeks, €50-500+)

And in the era of the EU AI Act, there's a fifth consideration that isn't technical but legal: traceability. Article 10 of the regulation requires that training data for high-risk systems be "relevant, representative, and to the extent possible, free of errors and complete." This applies directly to fine-tuning: if you fine-tune a model with biased or incorrect data, and that model makes decisions in a regulated domain, you're exposed to sanctions. RAG and Tool Calling, being transparent in their sources, offer traceability that fine-tuning cannot match.

As we wrote in The Hidden Economics of AI, the 10x Rule applies: if a more complex technique doesn't give you a result 10 times better than the previous one, it probably doesn't justify its added cost and complexity. Start simple. Measure. Scale only when the data demands it.

Sources of Interest:

Fine-Tune SLMs in Colab for Free : A 4-Bit Approach with Meta Llama 3.2

Sun, 18 May 2025 05:29:07 +0000

Fine-tuning large language models (LLMs) sounds complex — until you meet Unsloth. Whether you’re a complete beginner or an experienced ML tinkerer, this guide walks you through the simplest and most efficient way to fine-tune LLaMA models on free GPUs using Google Colab. Best of all? No fancy hardware or deep ML theory required.

This article breaks down every keyword, library, and function, defining each term precisely but in the simplest language possible.

In this article, you’ll learn how to:

Install and configure Unsloth in Colab
Load models in quantized (4-bit) mode to save memory
Understand core concepts (parameters, weights, biases, quantization, etc.)
Apply PEFT and LoRA adapters to fine-tune only a small part of the model
Prepare Q\&A data for training with Hugging Face Datasets and chat templates
Use SFTTrainer for supervised fine-tuning
Switch to inference mode for faster generation
Save and reload your fine-tuned model

Getting Comfortable With some Core Concepts

Disclaimer: I promise this will be the friendliest GenAI glossary — your cheat sheet, wittier than autocorrect and way less judgmental!

Language Model — Word-Predictor to put it simply! — like a smart autocomplete that predicts the next word based on what came before, much like your phone suggests “you” after “how are”. It learns these patterns by “reading” massive amounts of text, so it knows common word sequences and can fill in blanks. Behind the scenes, it models the probabilities of word sequences, assigning higher scores to more natural continuations.

Attention — Imagine you’re reading a sentence and want to know which earlier words matter most to understand each new word — that’s attention. Instead of reading a sentence strictly left-to-right, attention lets every word weigh how much it should consider all the others, like skimming a page and highlighting only key phrases. This selective focus makes predictions more accurate and efficient, ignoring irrelevant details.

Parameter — A number inside a model that can change during learning (like a dial that the model tweaks)

Weight — Mostly synonymous with parameters controlling how strongly one part of input affects the output

Data vs Parameters vs Weights — “Data” refers to the information used to train a model, while “parameters” are the values the model learns from that data, and “weights” are a specific type of parameter representing the strength of connections between model variables. To put it more simply, Data is the input, parameters are what the model adjusts to make predictions, and weights are a subset of those parameters.

Bias — A small extra number added so the model can shift outputs up or down, like a baseline adjustment

Transformer — _A Transformer is a special model built around attention, letting it “look” at every word in a sentence in parallel rather than one by one. It’s like a study group where everyone reads the entire essay at once and then discusses which sentences are most important to the main idea. Introduced in 2017 in Google’s “Attention Is All You Need” paper, Transformers are the backbone of today’s LLMs and power everything from translation tools to chatbots._

Quantization — Reducing precision of weights (e.g. 16-bit → 4-bit) to slash memory use, with minimal accuracy loss.

PEFT — (Parameter-Efficient Fine-Tuning) — updating only tiny adapter layers instead of the whole model.

LoRA — (Low-Rank Adaptation) — A smart shortcut for teaching a huge AI model new tricks by only tweaking a tiny part of it instead of retraining the entire thing. You “freeze” most of the model’s parameters and insert two small, trainable matrices into each layer. During fine-tuning, only these add-ons learn, drastically cutting down on time and compute cost.

LoRA “r” — The adapter’s rank (size). Higher r gives more capacity but uses more memory.

LoRA (alpha) — A scaling factor for adapter updates — like a “volume knob” for learning strength.

Dropout — Randomly turning off some adapter connections during training to prevent overfitting (can be set to 0).

Gradient Checkpointing — Recomputes parts of the model during backpropagation to halve peak VRAM usage, at a slight speed cost.

4-bit Mode — Quantized mode storing weights in 4 bits, cutting memory by \~4× compared to 16/32-bit.

Inference Mode — After training, use a special mode optimized for fast text generation (2× speed).

Overfitting — When a model “memorizes” a tiny dataset and fails on new inputs — always test on unseen data.

Checkpoint — A saved snapshot of model weights you can reload later.

Token — A token is a small chunk of text (\~4–5 characters) — a word, part of a word, punctuation mark, or symbol — that serves as the basic unit a model processes.

Tokenizer — The tokenizer is the program that “cuts” raw text into those tokens and then converts each token into a unique number (ID) the model can work with (e.g., “unhappiness” → “un”, “happi”, “ness” → “un” = 137, “happi” = 428, etc).

SLMs vs. LLMs

SLMs (Small Language Models) have fewer parameters and focus on specific tasks or domains — think of them as pocket calculators solving one type of math problem quickly and efficiently.
LLMs (Large Language Models) are like supercomputers trained on vast, diverse data; they can tackle many tasks — writing essays, summarizing articles, or coding — because they’ve “read” almost the entire internet.
SLMs require less compute power and are ideal for on-device or specialized applications, whereas LLMs need massive cloud resources but offer broader versatility.
In practice, you might use an SLM for customer-service chat on your phone, but call an LLM when you need deep research help or creative story generation.

1. Getting Started: Colab Setup

Why Google Colab & Tesla T4?

Cost: Free GPU access, that's all!
Performance: Tesla T4 handles mid-size LLMs effectively when paired with quantization and PEFT
Accessibility: No local GPU required — ideal for beginners

Installing Unsloth

# Stable release from PyPI:
pip install unsloth

# OR

# Install the Nightly (latest GitHub) for cutting-edge features:
pip uninstall unsloth -y
pip install --upgrade --no-cache-dir --no-deps \
  git+https://github.com/unslothai/unsloth.git@nightly \
  git+https://github.com/unslothai/unsloth-zoo.git

pip install unsloth: grabs the vetted, stable version
uninstall & install: fetches the newest commits from GitHub (may include experimental updates)

2. Loading a Model Efficiently

We load the Llama 3.2 1B model using Unsloth in a memory-efficient 4-bit quantization mode, using roughly one-quarter the memory of full precision so it runs faster and fits on small GPUs. It also sets how long each input can be (up to 2048 tokens).

from unsloth import FastLanguageModel
import torch

# Configuration
max_seq_length = 2048      # How many tokens each input can have
dtype = None               # None for auto detection. Float16 for Tesla T4, V100; Bfloat16 for Ampere+
load_in_4bit = True        # Use 4-bit quantization to reduce memory usage
model_name = "unsloth/Llama-3.2-1B-Instruct"

# Load both model and tokenizer
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=model_name,
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

FastLanguageModel.from_pretrained: downloads and prepares the model + tokenizer
max_seq_length: sets the maximum context length

3. Introducing PEFT & LoRA

Instead of updating all model weights (which can be billions), PEFT adds small adapter layers you train. LoRA is one such method.

model = FastLanguageModel.get_peft_model(
    model,
    r=16,          # Adapter rank (size). Suggested: 8, 16, 32, 64, 128
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "up_proj", "down_proj"
    ],
    lora_alpha=16,                  # Scales adapter updates
    lora_dropout=0,                 # No dropout
    bias="none",                    # Skip bias updates
    use_gradient_checkpointing="unsloth",  # Unsloth-optimized checkpointing
    random_state=3407,              # For reproducibility
    use_rslora=False,               # Optional advanced LoRA variant
    loftq_config=None,              # Optional LoftQ config
)

r controls the size of the LoRA layers; higher uses more memory.
lora_alpha is like a “volume knob” for learning strength.
use_gradient_checkpointing="unsloth" trades compute for lower VRAM.

4. Preparing Your Dataset for Training

When preparing datasets for fine-tuning models like LLaMA 3.1 and Phi-4, format multi-turn conversations per each model’s expected structure.

LLaMA 3.1: Chat Template Format

LLaMA 3.1 wraps each message with special tokens:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Hello!<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Hi there! How can I assist you today?<|eot_id|>

<|begin_of_text|> marks the start
<|start_header_id|>/<|end_header_id|> mark roles
<|eot_id|> ends each message

Use Unsloth’s standardize_sharegpt to convert existing data into this format.

Phi-4 uses ChatML JSON:

{
  "messages": [
    {"role": "system",    "content": "You are a helpful assistant."},
    {"role": "user",      "content": "Can you explain machine learning?"},
    {"role": "assistant", "content": "Certainly! Machine learning is..."}
  ]
}

Converting Between Formats

Identify current format (CSV, ShareGPT, ChatML).
Convert to a ShareGPT-like structure (from/value).
Standardize to role/content with standardize_sharegpt.
Apply chat template via get_chat_template and apply_chat_template.

Below, set USE_CSV = True or False to choose your data source.

Custom Dataset (CSV)

We used a fictional 30-question dataset from “Eastern Caverns” to illustrate fine-tuning on domain-specific Q\&A.

{% gist https://gist.github.com/kingrishabdugar/8f10fca3368c4e57db7266ef6a765007.js %}

Configuration & Imports

# Configuration & Imports
USE_CSV     = True                 # False → load a ShareGPT dataset instead
CSV_PATH    = "your_data.csv"      # CSV must have 'question' & 'answer' columns
SHAREGPT_DS = "mlabonne/FineTome-100k"  # HF ShareGPT-style dataset

import pandas as pd
from datasets import Dataset, load_dataset
from unsloth.chat_templates import get_chat_template, standardize_sharegpt

Loading & Wrapping Data

if USE_CSV:
    df = pd.read_csv(CSV_PATH)
    ds = Dataset.from_pandas(df)

    def to_sharegpt_format(ex):
        return {
            "conversations": [
                {"from": "system", "value": "You are a Victorian-era assistant…"},
                {"from": "human",  "value": ex["question"]},
                {"from": "gpt",    "value": ex["answer"]},
            ]
        }
    ds = ds.map(to_sharegpt_format, remove_columns=df.columns.tolist())
else:
    ds = load_dataset(SHAREGPT_DS, split="train")

Standardizing & Applying the Chat Template

ds = standardize_sharegpt(ds)

CHAT_TEMPLATE = "llama-3.1"
tokenizer = get_chat_template(tokenizer, chat_template=CHAT_TEMPLATE)

def format_prompts(examples):
    texts = [
        tokenizer.apply_chat_template(
            convo,
            tokenize=False,
            add_generation_prompt=False
        )
        for convo in examples["conversations"]
    ]
    return {"text": texts}

nds = ds.map(format_prompts, batched=True)

print("===== BEFORE =====")
print(nds[0]["conversations"])
print("===== AFTER =====")
print(nds[0]["text"])

5. Supervised Fine Tuning with SFTTrainer

Too many things here — lets explore one by one,

️ SFTTrainer

Purpose: Trainer for Supervised Fine-Tuning (SFT) of LLMs.
Why: Streamlines fine-tuning with built-in utilities.

Key Components

model, tokenizer, train_dataset — The model, tokenizer, and dataset (nds).
dataset_text_field="text" — Uses the "text" field for input.
DataCollatorForSeq2Seq — Pads and batches seq2seq data.
TrainingArguments — Hyperparameters (batch size, learning rate, epochs, etc.).
per_device_train_batch_size=2 — Examples per device.
gradient_accumulation_steps=4 — Simulate larger batches.
warmup_steps=5 — Smooth learning rate start.
max_steps=100 — Total training steps.
optim="adamw_8bit" — 8-bit AdamW optimizer for memory savings.
output_dir="outputs" — Where checkpoints go.

from trl import SFTTrainer
from transformers import TrainingArguments, DataCollatorForSeq2Seq
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=nds,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer),
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        num_train_epochs=1,
        max_steps=100,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.01,
        output_dir="outputs",
        report_to="none",
    ),
)

6. Kicking-off the training

Use Unsloth’s train_on_responses_only to compute loss only on the assistant’s output:

from unsloth.chat_templates import train_on_responses_only

trainer = train_on_responses_only(
    trainer,
    instruction_part="<|start_header_id|>user<|end_header_id|>\n\n",
    response_part="<|start_header_id|>assistant<|end_header_id|>\n\n",
)

stats = trainer.train()

You’ll see the loss drop steadily—proof your 4-bit Llama 3.2 is learning efficiently.

7. Inference & Saving Your Model

Fast Inference Mode

model = FastLanguageModel.for_inference(model)

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "<Your Question Here>"}],
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=256)
print(tokenizer.batch_decode(outputs)[0])

Save & Reload

# Save
model.save_pretrained("/content/drive/MyDrive/my_llama3_model_eastern_caverns")
tokenizer.save_pretrained("/content/drive/MyDrive/my_llama3_model_eastern_caverns")

# Reload in 4-bit mode
from transformers import AutoTokenizer
from unsloth import FastLanguageModel

tokenizer = AutoTokenizer.from_pretrained("/content/drive/MyDrive/my_llama3_model_eastern_caverns")
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="/content/drive/MyDrive/my_llama3_model_eastern_caverns",
    load_in_4bit=True,
    max_seq_length=2048,
)

# Quick test
inputs = tokenizer("What backup options are available for the CavernDB cluster?", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Conclusion

Fine-tuning a large language model is all about balancing precision, speed, and the right data.

Precision vs. Quantization
- Full precision (FP32) → \~4.3 billion weight values
- 4-bit quantization → 16 levels (tiny rounding error for big memory savings)
Why 4-Bit Helps
- 7 B params in FP16 needs \~28 GB RAM; in 4-bit \~7 GB
- Avoids out-of-memory on free GPUs (Colab, Kaggle)
Unsloth’s Speed Boost
- Up to 2× faster fine-tuning, \~70% VRAM reduction
- Memory-efficient kernels without accuracy loss
Picking the Right Dataset
- Too small → overfitting; too big/wild → underfitting
- Aim for focused, high-quality examples in proper order

That’s your bird’s-eye view: squeezing precision, saving resources, turbo-charging training with Unsloth, and choosing data wisely. Feedback and questions are always welcome!

One last thing — if you’ve made it this far, I’ve dropped the Colab notebook link in the comments below. Feel free to dive in and give it a spin !