Attention & Transformers

🎓 Complete all tutorials to earn your Free Deep Learning Certificate
Shareable on LinkedIn • Verified by AITutorials.site • No signup fee

🚀 The Transformer Revolution

In June 2017, a team of Google researchers published a paper titled "Attention is All You Need" that would fundamentally transform artificial intelligence. The Transformer architecture they introduced replaced RNNs and LSTMs entirely, becoming the foundation for virtually every state-of-the-art AI system today: GPT-4, Claude, Gemini, BERT, ChatGPT, and countless others.

Why RNNs Failed to Scale

Before Transformers, RNNs and LSTMs dominated sequential modeling. But they had fundamental limitations:

The Impact: From Research to Reality

What Makes Transformers Special?

👀 The Attention Mechanism: The Core Innovation

Intuitive Example: Translation with Attention

The Mathematics: Scaled Dot-Product Attention

Attention uses three learned linear projections called Query, Key, and Value. This is the famous QKV attention.

Concrete Example with Numbers

Why Scaling by √d_k?

Self-Attention: Every Position Attends to Every Position

In Transformers, Q, K, and V all come from the same input sequence. This is called self-attention (vs cross-attention in encoder-decoder models).

Implementation: Attention from Scratch

import numpy as np
import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Compute scaled dot-product attention.
    
    Args:
        Q: Query matrix (batch, seq_len, d_k)
        K: Key matrix (batch, seq_len, d_k)
        V: Value matrix (batch, seq_len, d_v)
        mask: Optional mask (batch, seq_len, seq_len)
    
    Returns:
        Output: Attention output (batch, seq_len, d_v)
        Weights: Attention weights (batch, seq_len, seq_len)
    """
    # Get dimension for scaling
    d_k = Q.size(-1)
    
    # Step 1: Compute attention scores (QK^T)
    # Shape: (batch, seq_len, d_k) @ (batch, d_k, seq_len) → (batch, seq_len, seq_len)
    scores = torch.matmul(Q, K.transpose(-2, -1))
    
    # Step 2: Scale by sqrt(d_k)
    scores = scores / np.sqrt(d_k)
    
    # Step 3: Apply mask if provided (e.g., for causal attention in GPT)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float('-inf'))
    
    # Step 4: Apply softmax to get attention weights
    # Each row sums to 1.0
    weights = F.softmax(scores, dim=-1)
    
    # Step 5: Weighted sum of values
    # Shape: (batch, seq_len, seq_len) @ (batch, seq_len, d_v) → (batch, seq_len, d_v)
    output = torch.matmul(weights, V)
    
    return output, weights

# ============ EXAMPLE USAGE ============
batch_size = 2
seq_len = 6  # "The cat sat on the mat"
d_model = 512  # Model dimension (typical for Transformer)

# Simulate input embeddings
embeddings = torch.randn(batch_size, seq_len, d_model)

# Create Q, K, V with learned projection matrices
W_Q = torch.randn(d_model, d_model)
W_K = torch.randn(d_model, d_model)
W_V = torch.randn(d_model, d_model)

Q = torch.matmul(embeddings, W_Q)
K = torch.matmul(embeddings, W_K)
V = torch.matmul(embeddings, W_V)

# Compute attention
output, weights = scaled_dot_product_attention(Q, K, V)

print(f"Output shape: {output.shape}")        # (2, 6, 512)
print(f"Attention weights shape: {weights.shape}")  # (2, 6, 6)

# Inspect attention pattern for first batch, position 2 ("sat")
print("\nAttention weights for 'sat' (position 2):")
print(weights[0, 2, :])  # Should show high weight for position 1 ("cat")

🏗️ Complete Transformer Architecture

The full Transformer combines multiple components into a powerful architecture. Let's build it piece by piece, understanding each component's role.

1. Multi-Head Attention: Multiple Perspectives

Instead of one attention mechanism, Transformers use multiple attention heads in parallel. Each head learns different relationships!

Example: 8 Heads with d_model = 512

import torch
import torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model=512, num_heads=8):
        super().__init__()
        assert d_model % num_heads == 0, "d_model must be divisible by num_heads"
        
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads  # 512 / 8 = 64
        
        # Linear projections for Q, K, V (all heads at once)
        self.W_Q = nn.Linear(d_model, d_model)
        self.W_K = nn.Linear(d_model, d_model)
        self.W_V = nn.Linear(d_model, d_model)
        
        # Output projection
        self.W_O = nn.Linear(d_model, d_model)
        
    def forward(self, Q, K, V, mask=None):
        batch_size = Q.size(0)
        
        # 1. Linear projections
        # Shape: (batch, seq_len, d_model)
        Q = self.W_Q(Q)
        K = self.W_K(K)
        V = self.W_V(V)
        
        # 2. Split into multiple heads
        # Shape: (batch, seq_len, d_model) → (batch, seq_len, num_heads, d_k)
        Q = Q.view(batch_size, -1, self.num_heads, self.d_k)
        K = K.view(batch_size, -1, self.num_heads, self.d_k)
        V = V.view(batch_size, -1, self.num_heads, self.d_k)
        
        # Transpose for attention: (batch, num_heads, seq_len, d_k)
        Q = Q.transpose(1, 2)
        K = K.transpose(1, 2)
        V = V.transpose(1, 2)
        
        # 3. Apply scaled dot-product attention for each head
        scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(self.d_k)
        
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))
        
        weights = torch.softmax(scores, dim=-1)
        attention_output = torch.matmul(weights, V)
        
        # 4. Concatenate heads
        # (batch, num_heads, seq_len, d_k) → (batch, seq_len, num_heads, d_k)
        attention_output = attention_output.transpose(1, 2).contiguous()
        
        # (batch, seq_len, num_heads, d_k) → (batch, seq_len, d_model)
        attention_output = attention_output.view(batch_size, -1, self.d_model)
        
        # 5. Final linear projection
        output = self.W_O(attention_output)
        
        return output, weights

2. Positional Encoding: Where Am I?

Problem: Attention has no notion of order! "Cat chased dog" = "Dog chased cat" in pure attention.
Solution: Add positional information to embeddings.

import numpy as np
import torch

def get_positional_encoding(max_seq_len, d_model):
    """
    Generate sinusoidal positional encodings.
    
    Args:
        max_seq_len: Maximum sequence length
        d_model: Model dimension (512 typical)
    
    Returns:
        Positional encoding matrix (max_seq_len, d_model)
    """
    position = np.arange(max_seq_len)[:, np.newaxis]  # (max_seq_len, 1)
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
    
    pe = np.zeros((max_seq_len, d_model))
    pe[:, 0::2] = np.sin(position * div_term)  # Even dimensions
    pe[:, 1::2] = np.cos(position * div_term)  # Odd dimensions
    
    return torch.FloatTensor(pe)

# Visualize positional encoding patterns
pe = get_positional_encoding(100, 512)
print(f"Positional encoding shape: {pe.shape}")  # (100, 512)

# Position 0 and position 1 have different patterns
print(f"Position 0: {pe[0, :8]}")  # First 8 dims
print(f"Position 1: {pe[1, :8]}")  # Different values!

3. Feed-Forward Network: Non-Linear Transformation

After attention, each position passes through a 2-layer feed-forward network (same network applied to each position independently).

class FeedForward(nn.Module):
    def __init__(self, d_model=512, d_ff=2048, dropout=0.1):
        super().__init__()
        self.linear1 = nn.Linear(d_model, d_ff)
        self.linear2 = nn.Linear(d_ff, d_model)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, x):
        # x: (batch, seq_len, d_model)
        x = self.linear1(x)              # (batch, seq_len, d_ff)
        x = torch.relu(x)                # ReLU activation
        x = self.dropout(x)
        x = self.linear2(x)              # (batch, seq_len, d_model)
        return x

4. Layer Normalization: Stabilize Training

Normalize activations to have mean=0, variance=1. Critical for training deep Transformers (GPT-3 has 96 layers!).

class LayerNorm(nn.Module):
    def __init__(self, d_model, eps=1e-6):
        super().__init__()
        self.gamma = nn.Parameter(torch.ones(d_model))  # Scale
        self.beta = nn.Parameter(torch.zeros(d_model))  # Shift
        self.eps = eps
        
    def forward(self, x):
        mean = x.mean(-1, keepdim=True)
        std = x.std(-1, keepdim=True)
        return self.gamma * (x - mean) / (std + self.eps) + self.beta

5. Residual Connections: Skip Connections

Add input to output of each sub-layer. Enables gradient flow in very deep networks.

Putting It All Together: Transformer Block

class TransformerBlock(nn.Module):
    """
    Complete Transformer encoder block with:
    - Multi-head self-attention
    - Feed-forward network
    - Layer normalization
    - Residual connections
    """
    def __init__(self, d_model=512, num_heads=8, d_ff=2048, dropout=0.1):
        super().__init__()
        self.attention = MultiHeadAttention(d_model, num_heads)
        self.feed_forward = FeedForward(d_model, d_ff, dropout)
        self.norm1 = LayerNorm(d_model)
        self.norm2 = LayerNorm(d_model)
        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)
        
    def forward(self, x, mask=None):
        # 1. Multi-head self-attention with residual + norm
        attention_output, _ = self.attention(x, x, x, mask)
        x = self.norm1(x + self.dropout1(attention_output))
        
        # 2. Feed-forward with residual + norm
        ff_output = self.feed_forward(x)
        x = self.norm2(x + self.dropout2(ff_output))
        
        return x

# Stack multiple blocks for full Transformer
class Transformer(nn.Module):
    def __init__(self, vocab_size, d_model=512, num_heads=8, num_layers=6, 
                 d_ff=2048, max_seq_len=512, dropout=0.1):
        super().__init__()
        
        # Token + positional embeddings
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.pos_encoding = get_positional_encoding(max_seq_len, d_model)
        self.dropout = nn.Dropout(dropout)
        
        # Stack of Transformer blocks
        self.blocks = nn.ModuleList([
            TransformerBlock(d_model, num_heads, d_ff, dropout)
            for _ in range(num_layers)
        ])
        
        # Output projection
        self.norm = LayerNorm(d_model)
        self.output = nn.Linear(d_model, vocab_size)
        
    def forward(self, x, mask=None):
        # x: (batch, seq_len) - token IDs
        seq_len = x.size(1)
        
        # Embedding + positional encoding
        x = self.embedding(x)  # (batch, seq_len, d_model)
        x = x + self.pos_encoding[:seq_len, :].to(x.device)
        x = self.dropout(x)
        
        # Pass through Transformer blocks
        for block in self.blocks:
            x = block(x, mask)
        
        # Output projection
        x = self.norm(x)
        logits = self.output(x)  # (batch, seq_len, vocab_size)
        
        return logits

The Complete Architecture Visualized

🤖 Transformer Variants: BERT, GPT, T5

The original Transformer had both encoder and decoder. Modern models specialize: encoder-only for understanding, decoder-only for generation, or encoder-decoder for translation tasks.

1. Encoder-Only: BERT (Bidirectional Encoder Representations)

BERT (Google, 2018) uses only the Transformer encoder. Processes text bidirectionally—sees both past and future context simultaneously.

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer
import torch

# ============ LOAD PRE-TRAINED BERT ============
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=2  # Binary classification (positive/negative)
)

# ============ PREPARE DATA ============
texts = [
    "This movie was fantastic! I loved every minute.",
    "Terrible film, waste of time and money.",
    "Pretty good, would recommend to friends."
]
labels = [1, 0, 1]  # 1 = positive, 0 = negative

# Tokenize
encodings = tokenizer(
    texts,
    padding=True,
    truncation=True,
    max_length=128,
    return_tensors='pt'
)

# ============ FINE-TUNE ON YOUR DATA ============
# (In practice, use Trainer API with train/eval datasets)
outputs = model(**encodings, labels=torch.tensor(labels))
loss = outputs.loss
logits = outputs.logits

print(f"Loss: {loss.item()}")
print(f"Predictions: {torch.argmax(logits, dim=1)}")

# ============ INFERENCE ============
new_text = "Amazing experience, highly recommend!"
inputs = tokenizer(new_text, return_tensors='pt')
with torch.no_grad():
    outputs = model(**inputs)
    prediction = torch.argmax(outputs.logits, dim=1)
    
sentiment = "Positive" if prediction == 1 else "Negative"
print(f"Sentiment: {sentiment}")

2. Decoder-Only: GPT (Generative Pre-trained Transformer)

GPT (OpenAI, 2018-2023) uses only the Transformer decoder. Processes text left-to-right—generates one token at a time, conditioning on previous tokens.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# ============ LOAD PRE-TRAINED GPT ============
model_name = "gpt2"  # or "gpt2-medium", "gpt2-large", "gpt2-xl"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Set pad token (GPT-2 doesn't have one by default)
tokenizer.pad_token = tokenizer.eos_token

# ============ TEXT GENERATION ============
prompt = "Once upon a time in a distant galaxy,"

# Encode prompt
input_ids = tokenizer.encode(prompt, return_tensors='pt')

# Generate with different strategies
outputs = model.generate(
    input_ids,
    max_length=100,              # Maximum length
    num_return_sequences=3,      # Generate 3 variations
    temperature=0.8,             # Randomness (0.0 = deterministic, 1.0+ = creative)
    top_k=50,                    # Sample from top 50 tokens
    top_p=0.95,                  # Nucleus sampling (top 95% probability mass)
    do_sample=True,              # Enable sampling
    pad_token_id=tokenizer.eos_token_id
)

# Decode and print
for i, output in enumerate(outputs):
    text = tokenizer.decode(output, skip_special_tokens=True)
    print(f"\n=== Generation {i+1} ===")
    print(text)

# ============ FEW-SHOT PROMPTING ============
few_shot_prompt = """Translate English to French:
English: Hello, how are you?
French: Bonjour, comment allez-vous?

English: I love learning AI.
French: J'aime apprendre l'IA.

English: The weather is beautiful today.
French:"""

input_ids = tokenizer.encode(few_shot_prompt, return_tensors='pt')
output = model.generate(input_ids, max_length=150, temperature=0.3)
print(tokenizer.decode(output[0], skip_special_tokens=True))

# Attention mask for "The cat sat"
      The   cat   sat
The   ✅    ❌    ❌     (can only see "The")
cat   ✅    ✅    ❌     (can see "The cat")
sat   ✅    ✅    ✅     (can see all previous)

# This enables autoregressive generation!

3. Encoder-Decoder: T5 (Text-to-Text Transfer Transformer)

T5 (Google, 2019) uses the full Transformer architecture (encoder + decoder). Frames every task as text-to-text!

from transformers import T5Tokenizer, T5ForConditionalGeneration

# ============ LOAD T5 ============
model_name = "t5-small"  # or "t5-base", "t5-large", "t5-3b", "t5-11b"
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)

# ============ SUMMARIZATION ============
article = """The Transformer architecture has revolutionized natural language processing. 
Introduced in 2017, it relies entirely on attention mechanisms, eschewing recurrence. 
This allows for much more parallelization and has led to models like BERT and GPT-3."""

input_text = "summarize: " + article
input_ids = tokenizer.encode(input_text, return_tensors='pt', max_length=512, truncation=True)

# Generate summary
summary_ids = model.generate(
    input_ids,
    max_length=50,
    num_beams=4,           # Beam search for better quality
    early_stopping=True
)
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print(f"Summary: {summary}")

# ============ TRANSLATION ============
input_text = "translate English to French: How are you today?"
input_ids = tokenizer.encode(input_text, return_tensors='pt')

translation_ids = model.generate(input_ids, max_length=50)
translation = tokenizer.decode(translation_ids[0], skip_special_tokens=True)
print(f"Translation: {translation}")

# ============ QUESTION ANSWERING ============
context = "Paris is the capital and largest city of France. It has a population of 2.2 million."
question = "What is the capital of France?"
input_text = f"question: {question} context: {context}"

input_ids = tokenizer.encode(input_text, return_tensors='pt', max_length=512)
answer_ids = model.generate(input_ids, max_length=20)
answer = tokenizer.decode(answer_ids[0], skip_special_tokens=True)
print(f"Answer: {answer}")

Comparison: BERT vs GPT vs T5

📋 Summary & Key Takeaways

Congratulations! You've mastered the Transformer architecture—the foundation of modern AI. Let's consolidate everything you've learned.

Core Concepts Mastered

Architecture Components

Transformer Variants Quick Reference

Why Transformers Dominate: The Full Picture

Mental Models & Decision Trees

Common Pitfalls & Best Practices

Practice Projects

What You've Mastered

The Transformer Impact

What's Next?

You've conquered the architecture that powers modern AI! In the next tutorial, Transfer Learning & Fine-tuning, you'll learn how to:

• 🎯 Leverage pre-trained models for your tasks
• 🔧 Fine-tune BERT, GPT, and T5 on custom data
• ⚡ Use adapters and LoRA for efficient fine-tuning
• 📊 Evaluate and deploy your models
• 💡 Build production ML systems with transfer learning