Fine-tuning LLMs

🎓 Complete all tutorials to earn your Free LLMs & Transformers Certificate
Shareable on LinkedIn • Verified by AITutorials.site • No signup fee

🔧 Why Fine-tune? When RAG and ICL Aren't Enough

In the previous tutorial, you learned that RAG and In-Context Learning solve 95% of LLM application needs. But what about the other 5%? When do you actually need to fine-tune a model?

Scenarios Where Fine-tuning Wins

Example: Customer support chatbot 
trained on 5,000 past conversations 
learns your company's empathetic, 
professional tone automatically.

Example: Medical diagnosis assistant 
fine-tuned on 10,000 case studies 
learns clinical reasoning patterns 
that few-shot examples can't teach.

RAG: 500ms retrieval + 1000 tokens
Fine-tuned: 0ms retrieval + 200 tokens
Result: 3x faster, 5x cheaper!

Example: Code generation model 
trained on 50,000 examples learns 
to always produce valid, properly 
formatted code.

Example: SQL query generation
Fine-tuned 7B model: 95% accuracy
GPT-4 few-shot: 87% accuracy
Cost: 50x cheaper!

Healthcare: Train Llama 2 on internal 
patient data, deploy on-premise.
Zero external API calls.

The Decision Framework

START: "I need an LLM to do X"
    ↓
Question 1: Can few-shot examples (2-10) show the model how to do it?
    ✅ YES → Use In-Context Learning (ICL)
    ❌ NO → Continue
    ↓
Question 2: Is it primarily about accessing external knowledge/documents?
    ✅ YES → Use RAG
    ❌ NO → Continue
    ↓
Question 3: Do I have 1,000+ high-quality training examples?
    ❌ NO → Go back to ICL/RAG, you don't have enough data
    ✅ YES → Continue
    ↓
Question 4: Is it about style, behavior, or specialized reasoning?
    ✅ YES → Consider fine-tuning
    ❌ NO → Probably can solve with RAG + ICL
    ↓
Question 5: Have I truly exhausted prompt engineering?
    ❌ NO → Go back and try harder with prompts
    ✅ YES → Fine-tune!
    ↓
Question 6: Which fine-tuning approach?
    • Need 99% accuracy, have A100 GPUs? → Full fine-tuning
    • Consumer GPU, good balance? → LoRA
    • Minimal resources, 7-13B models? → QLoRA ✅ (most common)

Fine-tuning vs RAG: Side-by-Side Example

Scenario: Build a customer support bot for a SaaS company.

When NOT to Fine-tune

💾 Full Fine-tuning: Training All Parameters

Full fine-tuning (also called supervised fine-tuning or SFT) means updating every single parameter in the model through backpropagation. If you're fine-tuning a 7B parameter model, you're training all 7 billion weights.

When to Use Full Fine-tuning

Hardware Requirements by Model Size

The Full Fine-tuning Process

Here's a complete, production-ready fine-tuning pipeline using HuggingFace Transformers:

Step 1: Prepare Your Training Data

// train.jsonl (instruction-following format)
{"instruction": "Summarize this article", "input": "Article text here...", "output": "Summary text here..."}
{"instruction": "Translate to French", "input": "Hello world", "output": "Bonjour le monde"}
{"instruction": "Answer the question", "input": "What is Python?", "output": "Python is a programming language..."}

// Recommendation: 1,000+ examples minimum, 10,000+ for best results

Step 2: Complete Training Script

# install: pip install transformers datasets accelerate torch
import torch
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    TrainingArguments,
    Trainer,
    DataCollatorForLanguageModeling
)

# 1. Load your model and tokenizer
model_name = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,  # Use FP16 for memory efficiency
    device_map="auto"  # Automatically distribute across GPUs
)

# Add padding token if missing
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
    model.config.pad_token_id = model.config.eos_token_id

# 2. Load and prepare dataset
dataset = load_dataset("json", data_files={
    "train": "train.jsonl",
    "validation": "validation.jsonl"
})

def format_instruction(example):
    """Convert instruction-input-output to a prompt"""
    instruction = example["instruction"]
    input_text = example.get("input", "")
    output = example["output"]
    
    if input_text:
        prompt = f"### Instruction:\n{instruction}\n\n### Input:\n{input_text}\n\n### Response:\n{output}"
    else:
        prompt = f"### Instruction:\n{instruction}\n\n### Response:\n{output}"
    
    return {"text": prompt}

# Format all examples
dataset = dataset.map(format_instruction)

# 3. Tokenize dataset
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=512,  # Adjust based on your task
        padding="max_length"
    )

tokenized_dataset = dataset.map(
    tokenize_function,
    batched=True,
    remove_columns=dataset["train"].column_names
)

# 4. Set up training arguments
training_args = TrainingArguments(
    output_dir="./llama-2-7b-finetuned",
    num_train_epochs=3,  # 3-5 epochs typical
    per_device_train_batch_size=4,  # Adjust based on VRAM
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,  # Effective batch size = 4 * 4 = 16
    learning_rate=2e-5,  # Low LR to prevent catastrophic forgetting
    weight_decay=0.01,
    warmup_steps=100,  # Gradual learning rate warmup
    lr_scheduler_type="cosine",  # Cosine decay schedule
    logging_steps=10,
    evaluation_strategy="steps",
    eval_steps=100,
    save_steps=500,
    save_total_limit=3,  # Keep only 3 best checkpoints
    load_best_model_at_end=True,
    fp16=True,  # Mixed precision training (faster, less memory)
    report_to="tensorboard",  # Log to TensorBoard
    push_to_hub=False
)

# 5. Create data collator
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False  # We're doing causal LM, not masked LM
)

# 6. Initialize Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
    data_collator=data_collator
)

# 7. Train!
print("🚀 Starting training...")
trainer.train()

# 8. Save the final model
print("💾 Saving model...")
trainer.save_model("./llama-2-7b-final")
tokenizer.save_pretrained("./llama-2-7b-final")

print("✅ Training complete!")

Step 3: Monitor Training

# Watch training progress in real-time
tensorboard --logdir ./llama-2-7b-finetuned/runs

# Monitor GPU usage
watch -n 1 nvidia-smi

# Expected training time for 7B model on A100:
# - 5,000 examples: ~8-12 hours
# - 10,000 examples: ~16-24 hours
# - 50,000 examples: ~80-120 hours

Step 4: Evaluate Your Model

# Load fine-tuned model
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("./llama-2-7b-final")
model = AutoModelForCausalLM.from_pretrained(
    "./llama-2-7b-final",
    torch_dtype=torch.float16,
    device_map="auto"
)

# Test it!
def generate_response(instruction, input_text=""):
    if input_text:
        prompt = f"### Instruction:\n{instruction}\n\n### Input:\n{input_text}\n\n### Response:\n"
    else:
        prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
    
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        top_p=0.9,
        do_sample=True
    )
    
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response.split("### Response:\n")[1].strip()

# Try it
print(generate_response(
    "Summarize this article",
    "Python is a high-level programming language known for its simplicity and readability..."
))

Key Hyperparameters Explained

The Catastrophic Forgetting Problem

Before Fine-tuning:
Q: "What is Python?"
A: "Python is a high-level programming language created by Guido van Rossum..."
(General knowledge intact)

After Aggressive Fine-tuning (LR too high, too many epochs):
Q: "What is Python?"
A: "### Instruction:\nAnswer this question\n\n### Response:\n..."
(Model only knows training format, forgot general knowledge!)

Solution:
✅ Low learning rate (2e-5)
✅ Few epochs (3-5)
✅ Validation monitoring
✅ Early stopping if val loss increases

Cost Analysis: Full Fine-tuning

⚡ LoRA (Low-Rank Adaptation): The Game Changer

LoRA (Low-Rank Adaptation) is a breakthrough technique published by Microsoft researchers in 2021 that revolutionized LLM fine-tuning. Instead of updating billions of parameters, LoRA trains tiny adapter modules (< 1% of model size) that achieve 95-99% of full fine-tuning performance at a fraction of the cost.

The Core Idea: Low-Rank Decomposition

LoRA is based on a key insight: fine-tuning updates to model weights are low-rank. This means most of the "learning" happens in a low-dimensional subspace, not across all dimensions.

Standard Fine-tuning:
--------------------
Original Weight Matrix W: 4096 × 4096 = 16,777,216 parameters
During training: Update ALL 16M parameters
Memory needed: W + gradients + optimizer states = 4x model size

LoRA Fine-tuning:
-----------------
Original Weight Matrix W: 4096 × 4096 (frozen, not trained)
LoRA Adapter:
  - Matrix A: 4096 × 8 = 32,768 parameters
  - Matrix B: 8 × 4096 = 32,768 parameters
  - Total: 65,536 parameters (0.4% of original!)

During training: Update only A and B
Memory needed: Only adapters' gradients + optimizer states
Result: 100x less memory!

How LoRA Works: The Math

In a standard transformer layer, attention computation looks like:

Standard Forward Pass:
----------------------
output = W × input

Where W is a huge weight matrix (e.g., 4096 × 4096)

LoRA Forward Pass:
------------------
output = W × input + (B × A) × input
         ↑           ↑
         Frozen      Trained adapters
         
Where:
- W: Original pretrained weights (frozen)
- A: Low-rank matrix (e.g., 4096 × r, r=8)
- B: Low-rank matrix (e.g., r × 4096)
- r: Rank (typically 8, 16, 32, or 64)

Key insight: (B × A) is a 4096 × 4096 matrix, but we only 
            train r × (4096 + 4096) parameters instead of 4096²!

LoRA Configuration Parameters

Complete LoRA Fine-tuning Example

# Install: pip install peft transformers datasets accelerate
import torch
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    TrainingArguments,
    Trainer
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# 1. Load base model (in FP16 for memory efficiency)
model_name = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# 2. Configure LoRA
lora_config = LoraConfig(
    r=16,  # Rank: 16 is a good balance
    lora_alpha=32,  # Alpha: 2 × r
    target_modules=[
        "q_proj",   # Query projection in attention
        "k_proj",   # Key projection
        "v_proj",   # Value projection
        "o_proj",   # Output projection
        "gate_proj",  # For Llama's FFN
        "up_proj",
        "down_proj"
    ],
    lora_dropout=0.05,  # 5% dropout for regularization
    bias="none",  # Don't train bias terms
    task_type="CAUSAL_LM"  # Causal language modeling
)

# 3. Apply LoRA to model
model = get_peft_model(model, lora_config)

# Print trainable parameters
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.062%

# 4. Load and prepare dataset (same as full fine-tuning)
dataset = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})

def format_instruction(example):
    return {"text": f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"}

dataset = dataset.map(format_instruction)

def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True, max_length=512, padding="max_length")

tokenized_dataset = dataset.map(tokenize_function, batched=True)

# 5. Training arguments (optimized for LoRA)
training_args = TrainingArguments(
    output_dir="./llama-2-7b-lora",
    num_train_epochs=3,
    per_device_train_batch_size=8,  # Can use larger batch size!
    per_device_eval_batch_size=8,
    gradient_accumulation_steps=2,
    learning_rate=3e-4,  # LoRA can use higher LR than full fine-tuning
    weight_decay=0.01,
    warmup_steps=100,
    logging_steps=10,
    evaluation_strategy="steps",
    eval_steps=100,
    save_steps=500,
    fp16=True,
    report_to="tensorboard"
)

# 6. Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"]
)

print("🚀 Starting LoRA training...")
trainer.train()

# 7. Save LoRA adapters (only ~10-50MB!)
model.save_pretrained("./llama-2-7b-lora-final")
tokenizer.save_pretrained("./llama-2-7b-lora-final")

print("✅ LoRA training complete!")
print("💡 Saved adapter weights are only ~20MB instead of ~14GB!")

Loading and Using LoRA Models

# Load the base model + LoRA adapters
from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LoRA adapters on top
model = PeftModel.from_pretrained(base_model, "./llama-2-7b-lora-final")
tokenizer = AutoTokenizer.from_pretrained("./llama-2-7b-lora-final")

# Use it!
def generate(instruction):
    prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
    return tokenizer.decode(outputs[0], skip_special_tokens=True).split("### Response:\n")[1]

print(generate("Explain quantum computing in simple terms"))

# Merge LoRA weights into base model (optional, for deployment)
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./llama-2-7b-merged")
# Now you have a standalone model without needing the base + adapter separately!

LoRA Hardware Requirements

LoRA vs Full Fine-tuning: Performance Comparison

Research shows LoRA achieves 95-99% of full fine-tuning performance across diverse tasks:

LoRA Best Practices

🔋 QLoRA (Quantized LoRA): The Ultimate Memory Saver

QLoRA (Quantized Low-Rank Adaptation) takes LoRA one step further by combining it with 4-bit quantization. The result? You can fine-tune a 13B model on a single RTX 3090 (24GB) or a 70B model on a single A100 (80GB)—something previously impossible without multiple high-end GPUs.

How QLoRA Works: Quantization + LoRA

QLoRA combines three key innovations:

Memory Breakdown: QLoRA vs LoRA vs Full

Fine-tuning a 7B model (in FP16):

Full Fine-tuning:
├── Model weights: 14GB
├── Gradients: 14GB
├── Optimizer states (Adam): 28GB
└── Activations: 8-12GB
    TOTAL: ~64-68GB (needs A100 80GB)

LoRA (r=16):
├── Model weights: 14GB (frozen, no gradients)
├── LoRA adapters: 0.02GB
├── LoRA gradients: 0.02GB
├── Optimizer states: 0.04GB
└── Activations: 8-12GB
    TOTAL: ~22-26GB (fits RTX 3090 24GB, barely)

QLoRA (r=16, 4-bit):
├── Model weights (4-bit): 3.5GB (no gradients!)
├── LoRA adapters (FP16): 0.02GB
├── LoRA gradients: 0.02GB
├── Optimizer states: 0.04GB
└── Activations: 4-6GB
    TOTAL: ~7.5-9.5GB (fits RTX 3060 12GB!)

Complete QLoRA Implementation

# Install: pip install bitsandbytes peft transformers accelerate
import torch
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    BitsAndBytesConfig,
    TrainingArguments,
    Trainer
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# 1. Configure 4-bit quantization with NF4
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,  # Use 4-bit quantization
    bnb_4bit_quant_type="nf4",  # NormalFloat4 (optimal for neural nets)
    bnb_4bit_compute_dtype=torch.float16,  # Compute in FP16 for speed
    bnb_4bit_use_double_quant=True  # Double quantization for extra savings
)

# 2. Load model in 4-bit
model_name = "meta-llama/Llama-2-13b-hf"  # 13B model!
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",  # Automatically distribute across available GPUs
    trust_remote_code=True
)

# 3. Prepare model for k-bit training
model = prepare_model_for_kbit_training(model)

# 4. Configure LoRA
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# 5. Apply LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 13,016,035,328 || trainable%: 0.322%

# 6. Load dataset (same as before)
dataset = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})

def format_instruction(example):
    return {"text": f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"}

dataset = dataset.map(format_instruction)

def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True, max_length=512, padding="max_length")

tokenized_dataset = dataset.map(tokenize_function, batched=True)

# 7. Training arguments (optimized for QLoRA)
training_args = TrainingArguments(
    output_dir="./llama-2-13b-qlora",
    num_train_epochs=3,
    per_device_train_batch_size=4,  # 4-bit allows larger batches
    gradient_accumulation_steps=4,
    learning_rate=2e-4,  # Slightly higher than full fine-tuning
    warmup_steps=100,
    logging_steps=10,
    save_steps=500,
    fp16=True,  # Mixed precision
    optim="paged_adamw_8bit",  # 8-bit Adam with paging (crucial for QLoRA!)
    lr_scheduler_type="cosine",
    evaluation_strategy="steps",
    eval_steps=100,
    report_to="tensorboard"
)

# 8. Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"]
)

print("🚀 Starting QLoRA training of 13B model on consumer GPU...")
trainer.train()

# 9. Save
model.save_pretrained("./llama-2-13b-qlora-final")
tokenizer.save_pretrained("./llama-2-13b-qlora-final")
print("✅ QLoRA training complete! Saved adapters (~40MB).")

QLoRA Hardware Requirements: The Real Game-Changer

QLoRA Performance: How Much Quality Do You Lose?

QLoRA achieves 96-98% of full fine-tuning performance across tasks:

📊 Comparison

📋 Best Practices

• Start with prompting/RAG: Try those first. They're faster
• Use QLoRA: Best cost-benefit for most use cases
• Low learning rates: 1e-4 to 5e-5 typical
• Evaluate regularly: On validation set during training
• Don't overfit: 3-5 epochs usually optimal
• Use instruction data: Format data as instructions for best results
• Monitor perplexity: Track validation loss

📋 Summary

What You've Learned:

• Full fine-tuning updates all parameters (expensive)
• LoRA trains small adapter modules (10x cheaper)
• QLoRA combines quantization + LoRA (most practical)
• QLoRA: ~1M trainable params, 6-8GB VRAM for 7B models
• Start with RAG/prompting. Fine-tune only when needed

What's Next?

In our next tutorial, Inference Optimization, we'll learn how to deploy models efficiently.