Transfer Learning & Fine-tuning

🎓 Complete all tutorials to earn your Free Deep Learning Certificate
Shareable on LinkedIn • Verified by AITutorials.site • No signup fee

🚀 Why Transfer Learning? The Practical Superpower

Imagine training a neural network from scratch: you'd need millions of labeled examples, weeks of GPU time, and thousands of dollars in cloud costs. Transfer learning changes everything—leverage models pre-trained on massive datasets and achieve state-of-the-art results with just hundreds of examples in hours.

The Economics: Training from Scratch vs Transfer Learning

Real-World Success Stories

What Gets Transferred?

Pre-trained models have learned hierarchical features that generalize across tasks:

When Does Transfer Learning Work?

📚 Pre-trained Model Zoo: Your Starting Point

Computer Vision Models

Most vision models are pre-trained on ImageNet (14 million images, 1,000 categories). They've learned to recognize objects, textures, scenes—ready to adapt to your task.

Natural Language Processing Models

NLP models are pre-trained on billions of words from books, Wikipedia, web crawls. They understand grammar, semantics, world knowledge—ready for your text task.

Quick Selection Guide

Where to Find Pre-trained Models

🎯 Two Transfer Learning Strategies

There are two main approaches to transfer learning, each with different tradeoffs. Choose based on your data size and similarity to the pre-training domain.

Decision Tree: Which Strategy?

Strategy 1: Feature Extraction (Frozen Backbone)

\n Use the pre-trained model as a fixed feature extractor. The convolutional base extracts features, your new classifier learns to map them to your classes.\n

# ============ FEATURE EXTRACTION: COMPLETE EXAMPLE ============\nimport tensorflow as tf\nfrom tensorflow.keras.applications import ResNet50\nfrom tensorflow.keras import layers, models\nimport numpy as np\n\n# Step 1: Load pre-trained ResNet50 (without top classification layer)\nbase_model = ResNet50(\n    weights='imagenet',        # Use ImageNet weights\n    include_top=False,         # Remove final dense layers\n    input_shape=(224, 224, 3)  # Standard ImageNet size\n)\n\n# Step 2: FREEZE the base model (critical!)\nbase_model.trainable = False\n\nprint(f\"Base model has {len(base_model.layers)} layers\")\nprint(f\"Trainable: {base_model.trainable}\")  # Should be False\n\n# Step 3: Build new model with custom classifier\nmodel = models.Sequential([\n    # Pre-trained feature extractor\n    base_model,\n    \n    # Global pooling: (7, 7, 2048) → (2048,)\n    layers.GlobalAveragePooling2D(),\n    \n    # Optional: add dense layer for more capacity\n    layers.Dense(256, activation='relu'),\n    layers.BatchNormalization(),\n    layers.Dropout(0.5),  # Prevent overfitting\n    \n    # Final classification layer (YOUR classes)\n    layers.Dense(10, activation='softmax')  # 10 classes example\n])\n\n# Step 4: Compile with standard settings\nmodel.compile(\n    optimizer='adam',  # Can use higher LR since base is frozen\n    loss='categorical_crossentropy',\n    metrics=['accuracy']\n)\n\nprint(model.summary())\n\n# Count parameters\ntrainable_params = sum([tf.size(w).numpy() for w in model.trainable_weights])\nprint(f\"\\nTrainable parameters: {trainable_params:,}\")  # Only ~0.5M (classifier only)\n\n# Step 5: Train (FAST! Only training classifier)\nhistory = model.fit(\n    train_dataset,\n    epochs=10,\n    validation_data=val_dataset,\n    callbacks=[\n        tf.keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)\n    ]\n)\n\n# Step 6: Evaluate\ntest_loss, test_acc = model.evaluate(test_dataset)\nprint(f\"Test accuracy: {test_acc:.4f}\")

Strategy 2: Fine-tuning (Unfrozen Backbone)

\n Unfreeze some or all pre-trained layers and train with a very low learning rate. This allows the model to adapt pre-trained features to your specific domain.\n

# ============ FINE-TUNING: COMPLETE EXAMPLE ============\nimport tensorflow as tf\nfrom tensorflow.keras.applications import ResNet50\nfrom tensorflow.keras import layers, models\n\n# Step 1: Start with feature extraction model (from previous section)\nbase_model = ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))\nbase_model.trainable = False\n\nmodel = models.Sequential([\n    base_model,\n    layers.GlobalAveragePooling2D(),\n    layers.Dense(256, activation='relu'),\n    layers.Dropout(0.5),\n    layers.Dense(10, activation='softmax')\n])\n\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n\n# Train classifier first (feature extraction phase)\nprint(\"Phase 1: Training classifier only...\")\nhistory1 = model.fit(train_dataset, epochs=10, validation_data=val_dataset)\n\n# Step 2: UNFREEZE base model for fine-tuning\nbase_model.trainable = True\n\nprint(f\"\\nBase model has {len(base_model.layers)} layers\")\n\n# Option A: Unfreeze ALL layers\n# (Use when you have 10,000+ examples)\nfor layer in base_model.layers:\n    layer.trainable = True\n\n# Option B: Unfreeze only TOP layers (RECOMMENDED)\n# (Use when you have 1,000-10,000 examples)\nfor layer in base_model.layers[:-30]:  # Freeze first 145 layers, unfreeze last 30\n    layer.trainable = False\nfor layer in base_model.layers[-30:]:\n    layer.trainable = True\n\nprint(f\"Trainable layers: {sum([layer.trainable for layer in base_model.layers])}\")\n\n# Step 3: Recompile with VERY LOW learning rate (critical!)\nmodel.compile(\n    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-5),  # 100x lower!\n    loss='categorical_crossentropy',\n    metrics=['accuracy']\n)\n\n# Step 4: Fine-tune with careful monitoring\nprint(\"Phase 2: Fine-tuning top layers...\")\nhistory2 = model.fit(\n    train_dataset,\n    epochs=20,\n    validation_data=val_dataset,\n    callbacks=[\n        # Stop if validation loss increases (overfitting)\n        tf.keras.callbacks.EarlyStopping(\n            monitor='val_loss',\n            patience=5,\n            restore_best_weights=True\n        ),\n        # Reduce LR if plateau\n        tf.keras.callbacks.ReduceLROnPlateau(\n            monitor='val_loss',\n            factor=0.5,\n            patience=3,\n            min_lr=1e-7\n        )\n    ]\n)\n\n# Step 5: Evaluate\ntest_loss, test_acc = model.evaluate(test_dataset)\nprint(f\"\\nFinal test accuracy: {test_acc:.4f}\")

Progressive Fine-tuning Strategy

\n For best results, fine-tune in stages: classifier → top layers → more layers. This prevents destroying pre-trained weights.\n

Fine-tuning Results & Expectations

🤖 NLP Transfer Learning with Transformers

NLP transfer learning has revolutionized text tasks. Pre-trained models like BERT understand language deeply from billions of words. You fine-tune them on your specific task with just hundreds of examples.

Complete BERT Fine-tuning Pipeline

# ============ COMPLETE NLP TRANSFER LEARNING ============
from transformers import (
    AutoTokenizer, AutoModelForSequenceClassification,
    Trainer, TrainingArguments, DataCollatorWithPadding
)
from datasets import load_dataset, Dataset
import numpy as np
from sklearn.metrics import accuracy_score, f1_score
import torch

# Step 1: Load pre-trained model and tokenizer
model_name = "bert-base-uncased"  # 110M parameters, trained on Wikipedia+BookCorpus
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=2  # Binary classification (positive/negative)
)

print(f"Model has {model.num_parameters():,} parameters")
print(f"Fine-tuning only classification head? {False}")  # We'll fine-tune ALL layers

# Step 2: Prepare your dataset
# Example: Sentiment analysis on IMDB reviews
train_texts = [
    "This movie was fantastic! Loved every minute.",
    "Terrible film, complete waste of time.",
    "Absolutely brilliant performances all around.",
    # ... your texts ...
]
train_labels = [1, 0, 1, ...]  # 1 = positive, 0 = negative

val_texts = ["Pretty good movie, would recommend.", ...]
val_labels = [1, ...]

# Step 3: Tokenization (CRITICAL step in NLP)
def tokenize_function(examples):
    return tokenizer(
        examples['text'],
        padding='max_length',    # Pad to max length
        truncation=True,         # Truncate if too long
        max_length=128,          # BERT max is 512, but 128 is often enough
        return_tensors='pt'      # Return PyTorch tensors
    )

# Convert to Hugging Face Dataset format
train_dataset = Dataset.from_dict({'text': train_texts, 'label': train_labels})
val_dataset = Dataset.from_dict({'text': val_texts, 'label': val_labels})

# Apply tokenization
train_dataset = train_dataset.map(tokenize_function, batched=True)
val_dataset = val_dataset.map(tokenize_function, batched=True)

# Step 4: Define training configuration
training_args = TrainingArguments(
    output_dir='./results',
    
    # Training hyperparameters
    num_train_epochs=3,                    # Usually 2-4 epochs for fine-tuning
    per_device_train_batch_size=16,        # Adjust based on GPU memory
    per_device_eval_batch_size=32,
    learning_rate=2e-5,                    # CRITICAL: Very low LR (typical: 2e-5 to 5e-5)
    weight_decay=0.01,                     # L2 regularization
    
    # Evaluation & logging
    evaluation_strategy="epoch",           # Evaluate after each epoch
    save_strategy="epoch",
    logging_steps=100,
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    
    # Optimization
    warmup_steps=500,                      # Gradual warmup of learning rate
    fp16=torch.cuda.is_available(),        # Mixed precision training (faster)
    
    # Other
    report_to="none",                      # Disable W&B/TensorBoard
    seed=42
)

# Step 5: Define evaluation metrics
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = np.argmax(predictions, axis=1)
    
    return {
        'accuracy': accuracy_score(labels, predictions),
        'f1': f1_score(labels, predictions, average='weighted')
    }

# Step 6: Create Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    compute_metrics=compute_metrics,
    data_collator=DataCollatorWithPadding(tokenizer=tokenizer)
)

# Step 7: Train!
print("\n========== Fine-tuning BERT ==========\n")
trainer.train()

# Step 8: Evaluate
print("\n========== Evaluation ==========\n")
eval_results = trainer.evaluate()
print(f"Test Accuracy: {eval_results['eval_accuracy']:.4f}")
print(f"Test F1: {eval_results['eval_f1']:.4f}")

# Step 9: Save model
model.save_pretrained('./fine_tuned_bert')
tokenizer.save_pretrained('./fine_tuned_bert')

# Step 10: Inference on new text
def predict(text):
    inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
    outputs = model(**inputs)
    probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
    prediction = torch.argmax(probs, dim=-1).item()
    confidence = probs[0][prediction].item()
    
    return "positive" if prediction == 1 else "negative", confidence

text = "This movie exceeded all my expectations!"
label, conf = predict(text)
print(f"\nText: {text}")
print(f"Prediction: {label} (confidence: {conf:.2%})")

Different NLP Tasks with Transfer Learning

💡 Best Practices & Common Pitfalls

Golden Rules for Transfer Learning Success

Learning Rate Selection Guide

Common Mistakes & Solutions

Debugging Checklist

Performance Optimization Tips

When NOT to Use Transfer Learning

📋 Summary & Key Takeaways

Core Concepts Review

Decision Tree: Your Transfer Learning Strategy

Mental Model: The Transfer Learning Ladder

Quick Reference: Vision vs NLP Transfer Learning

Practice Projects

What's Next?

In the final Deep Learning tutorial, Generative Models & GANs, we'll shift from classification to creation—learning how to generate new images, text, and data with deep learning!