In-Context Learning & RAG

🎓 Complete all tutorials to earn your Free LLMs & Transformers Certificate
Shareable on LinkedIn • Verified by AITutorials.site • No signup fee

🎯 The Problem: LLM Knowledge Limitations

Imagine you've deployed a customer support chatbot using GPT-4. A customer asks: "What's your return policy for electronics purchased during Black Friday sales?" The LLM confidently generates an answer... that's completely wrong because it doesn't know your company's specific policies.

LLMs face three fundamental knowledge limitations:

Traditional Solutions (And Why They Don't Scale)

Problem: Want LLM to know 10,000 product documentation pages
Approach: Fine-tune GPT-3.5 on all documents
Cost: $5,000 - $20,000
Time: 2-3 weeks training
Maintenance: Every doc update requires retraining ($$$)
Result: Expensive, slow, not maintainable

Problem: Answer questions about 100-page policy manual
Approach: Copy entire manual into prompt
Token count: ~50,000 tokens
Cost per query: $0.50 - $2.00 (at GPT-4 rates)
Context limit: Most models max at 8K-128K tokens
Result: Expensive, hits limits, slow

The Core Insight

LLMs have a remarkable emergent ability: they can learn new tasks from examples provided in the prompt without any parameter updates. This is called in-context learning (ICL).

Traditional ML:
❌ Collect data → Train model → Deploy → Use
   (weeks of training required for each task)

In-Context Learning:
✅ Show examples in prompt → Model adapts → Answer
   (instant, no training needed)

RAG (Retrieval Augmented Generation):
✅ Find relevant docs → Add to prompt → Model answers from docs
   (grounded, verifiable, maintainable)

🧠 In-Context Learning (ICL): Learning Without Training

In-context learning is the phenomenon where LLMs adapt to new tasks by observing examples in the prompt—without any gradient updates or parameter changes. It's one of the most remarkable emergent abilities of large language models.

How In-Context Learning Works

The model observes the pattern in your examples and extrapolates to new instances:

Prompt: "Translate English to French.

English: Hello
French: Bonjour

English: Goodbye
French: Au revoir

English: Thank you
French: Merci

English: Good morning
French: ?"

Output: "Bonjour" (or "Bon matin")

The model:
1. Recognizes this is a translation task
2. Infers the pattern from 3 examples
3. Applies the pattern to the new input
4. Generates the translation

ALL WITHOUT TRAINING!

Zero-Shot vs Few-Shot vs Many-Shot

Real Example: Sentiment Classification

Classify the sentiment of this review as positive or negative:
"The product broke after 2 days. Waste of money."

Output: "Negative"
Accuracy: ~70% (model must guess from instructions alone)

# Few-shot prompt
prompt = """Classify sentiment as positive or negative:

Review: "Amazing quality! Best purchase ever."
Sentiment: Positive

Review: "Terrible customer service, never buying again."
Sentiment: Negative

Review: "Product works great, very happy with it."
Sentiment: Positive

Review: "The product broke after 2 days. Waste of money."
Sentiment:"""

# Output: "Negative"
# Accuracy: ~92% (learned pattern from examples)

Why Does This Work?

During pre-training on trillions of tokens, LLMs see countless examples of:

• Question → Answer patterns
• Example → Example → Example → New case patterns
• Format specifications followed by formatted outputs
• Task descriptions followed by task execution

The model learns a meta-pattern: "When I see examples of a pattern, apply that pattern to the next input." This is learning to learn!

In-Context Learning Best Practices

In-Context Learning Code Example

from openai import OpenAI

client = OpenAI()

def few_shot_classifier(text, examples, labels):
    """
    Perform few-shot classification using in-context learning
    
    Args:
        text: Text to classify
        examples: List of example texts
        labels: List of corresponding labels
    """
    # Build few-shot prompt
    prompt = "Classify the following texts:\n\n"
    
    for example, label in zip(examples, labels):
        prompt += f"Text: {example}\nLabel: {label}\n\n"
    
    # Add the new text to classify
    prompt += f"Text: {text}\nLabel:"
    
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,  # Deterministic for classification
        max_tokens=10
    )
    
    return response.choices[0].message.content.strip()

# Usage: Email spam classification
examples = [
    "Congratulations! You've won $1 million! Click here now!",
    "Hi John, can we schedule a meeting for tomorrow at 2pm?",
    "URGENT: Your account will be suspended unless you verify now!!!",
    "Thanks for your email. I'll review the document and get back to you.",
]

labels = ["Spam", "Not Spam", "Spam", "Not Spam"]

new_email = "Limited time offer! Buy now and get 90% off! Act fast!"

result = few_shot_classifier(new_email, examples, labels)
print(f"Classification: {result}")  # Output: "Spam"

# The model learned the spam detection pattern from just 4 examples!

🔍 Retrieval Augmented Generation (RAG)

While In-Context Learning gives LLMs examples to learn from, Retrieval Augmented Generation (RAG) gives them knowledge to draw from. RAG combines the power of information retrieval with LLM generation, enabling models to answer questions accurately using external, up-to-date information without needing retraining.

The Problem RAG Solves

Imagine you're building a customer support chatbot for an e-commerce company. The chatbot needs to answer questions like:

• "What's your return policy for opened electronics?"
• "Do you ship to Alaska?"
• "How do I track my order #12345?"

Problem: GPT-4 doesn't know your company's specific policies (they're not in its training data), and even if you fine-tune it, you'd need to retrain every time policies change.

RAG Solution: Store your company policies in a searchable knowledge base. When a user asks a question:

1. Search the knowledge base for relevant documents
2. Add those documents as context to the LLM prompt
3. LLM generates an answer grounded in those documents

The RAG Pipeline: How It Works

RAG has two main phases: Indexing (setup, done once) and Retrieval + Generation (happens every query).

Phase 1: Indexing (One-Time Setup)

Documents:
- returns_policy.pdf
- shipping_info.docx
- faq_database.txt
- product_catalog.csv

Original: 10-page return policy
Chunked:
- Chunk 1: "Returns within 30 days..."
- Chunk 2: "Refund processing takes..."
- Chunk 3: "Damaged items should..."

Chunk: "Returns within 30 days"
Embedding: [0.21, -0.54, 0.12, ..., 0.08]
           (1536 dimensions)

Vector Database (Pinecone/Weaviate):
ID: chunk_001
Vector: [0.21, -0.54, ...]
Text: "Returns within 30 days..."
Metadata: {source: "returns_policy.pdf"}

Phase 2: Retrieval + Generation (Every Query)

When a user asks a question, the RAG system springs into action:

🧑 User Question: "What's the return policy for opened electronics?"

Step 1: EMBED THE QUESTION
Question embedding: [0.19, -0.51, 0.11, ..., 0.07]  (1536 dims)

Step 2: SEARCH VECTOR DATABASE (Semantic Search)
Query the vector DB to find chunks with similar embeddings:

Results (sorted by similarity):
📄 Chunk 42 (similarity: 0.94) - "Returns_Policy.pdf"
   "Opened electronics can be returned within 14 days if in original 
    packaging. A 15% restocking fee applies. Items must include all 
    accessories and manuals..."

📄 Chunk 107 (similarity: 0.87) - "Electronics_FAQ.pdf"
   "For electronics returns: Unopened items qualify for full refund. 
    Opened items may incur restocking fees. Check original packaging 
    requirements..."

📄 Chunk 23 (similarity: 0.81) - "Returns_Policy.pdf"
   "Standard return window is 30 days. Special categories like electronics, 
    software, and personalized items have different rules..."

Step 3: ASSEMBLE PROMPT (Context + Question)
System: You are a helpful customer support assistant. Answer based ONLY 
        on the provided context. If you can't answer from the context, 
        say "I don't have that information."

Context:
[Document 1 - Returns_Policy.pdf]: 
Opened electronics can be returned within 14 days if in original packaging...

[Document 2 - Electronics_FAQ.pdf]: 
For electronics returns: Unopened items qualify for full refund...

[Document 3 - Returns_Policy.pdf]:
Standard return window is 30 days. Special categories like electronics...

User Question: What's the return policy for opened electronics?

Step 4: GENERATE ANSWER
🤖 LLM (GPT-4): "For opened electronics, you can return them within 14 days 
   if they're in the original packaging with all accessories and manuals. 
   A 15% restocking fee will apply. This differs from our standard 30-day 
   policy for most items."

   Sources: Returns_Policy.pdf, Electronics_FAQ.pdf

Step 5: RETURN TO USER
User sees: Answer + source citations for verification!

Vector Databases: The RAG Backend

Vector databases are specialized databases optimized for storing and searching high-dimensional vectors (embeddings). Here are the main options:

Embedding Models: Converting Text to Vectors

Embeddings are the magic that makes semantic search work. Here are your options:

Chunking Strategies: Breaking Documents Smartly

How you chunk documents dramatically affects RAG performance. Too small = lost context. Too large = irrelevant information.

chunk_size = 500  # characters
overlap = 50      # overlap between chunks

# Simple but effective
chunks = split_text(text, 
                   chunk_size=500, 
                   overlap=50)

# Split by paragraphs, sections
chunks = text.split('\n\n')

# Or use semantic similarity
# to group related sentences
chunks = semantic_split(text)

# LangChain's approach
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    separators=['\n\n', '\n', '. ', ' ']
)
chunks = splitter.split_text(text)

Retrieval Strategies: Beyond Basic Search

1. Semantic Search (Standard RAG)

# Convert query to embedding
query_embedding = embedding_model.embed("return policy")

# Find similar document embeddings (cosine similarity)
results = vector_db.search(query_embedding, top_k=5)

# Returns: Top 5 most semantically similar chunks

2. Hybrid Search (Semantic + Keyword)

# Combine embedding search with keyword (BM25) search
semantic_results = vector_db.search(query_embedding, top_k=10)
keyword_results = bm25_search(query_text, top_k=10)

# Merge and rerank results using Reciprocal Rank Fusion (RRF)
final_results = rrf_merge(semantic_results, keyword_results, top_k=5)

# Best of both worlds: catches exact terms + semantic meaning

3. Reranking (Two-Stage Retrieval)

# Stage 1: Fast retrieval (get 20-50 candidates)
candidates = vector_db.search(query_embedding, top_k=20)

# Stage 2: Precise reranking with cross-encoder
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
scores = reranker.predict([(query, doc.text) for doc in candidates])

# Get top 5 after reranking
final_docs = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:5]

# Result: Much better relevance than retrieval alone!

💻 Building RAG Systems with LangChain

Simple RAG: End-to-End Example

Let's build a complete RAG system that answers questions about your company's documentation:

# Install: pip install langchain openai chromadb tiktoken
import os
from langchain.document_loaders import DirectoryLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA

# Set API key
os.environ['OPENAI_API_KEY'] = 'your-api-key-here'

# Step 1: Load documents
print("📄 Loading documents...")
loader = DirectoryLoader('company_docs/', glob="**/*.txt", loader_cls=TextLoader)
documents = loader.load()
print(f"Loaded {len(documents)} documents")

# Step 2: Chunk documents
print("✂️ Chunking documents...")
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = text_splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")

# Step 3: Create embeddings & vector store
print("🧮 Creating embeddings...")
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")
vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"  # Save to disk
)
print("✅ Vector store created!")

# Step 4: Create RAG chain
print("🔗 Setting up RAG chain...")
llm = ChatOpenAI(model="gpt-4", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",  # Pass all retrieved docs to LLM
    retriever=vector_store.as_retriever(search_kwargs={"k": 3}),  # Top 3 docs
    return_source_documents=True  # Include sources in response
)

# Step 5: Query the system
print("\n💬 RAG system ready! Asking questions...\n")

questions = [
    "What's the return policy for electronics?",
    "Do you ship internationally?",
    "How do I contact customer support?"
]

for question in questions:
    print(f"❓ Q: {question}")
    result = qa_chain({"query": question})
    print(f"🤖 A: {result['result']}")
    print(f"📚 Sources: {[doc.metadata['source'] for doc in result['source_documents']]}")
    print("-" * 80 + "\n")

❓ Q: What's the return policy for electronics?
🤖 A: Electronics can be returned within 14 days if in original packaging. 
      A 15% restocking fee applies. Items must include all accessories.
📚 Sources: ['company_docs/returns_policy.txt', 'company_docs/electronics_faq.txt']

Production-Ready RAG with Pinecone

For production apps, use a managed vector database like Pinecone:

# Install: pip install langchain openai pinecone-client
import os
import pinecone
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Pinecone
from langchain.chat_models import ChatOpenAI
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory

# Initialize Pinecone
pinecone.init(
    api_key="your-pinecone-api-key",
    environment="us-west1-gcp"  # Your environment
)

# Create index (one-time setup)
index_name = "company-knowledge-base"
if index_name not in pinecone.list_indexes():
    pinecone.create_index(
        name=index_name,
        dimension=1536,  # OpenAI embedding dimension
        metric="cosine"
    )

# Set up embeddings and vector store
embeddings = OpenAIEmbeddings()
vector_store = Pinecone.from_documents(
    documents=chunks,
    embedding=embeddings,
    index_name=index_name
)

# Create conversational RAG (remembers chat history!)
memory = ConversationBufferMemory(
    memory_key="chat_history",
    return_messages=True,
    output_key="answer"
)

llm = ChatOpenAI(model="gpt-4", temperature=0)
conversational_rag = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=vector_store.as_retriever(search_kwargs={"k": 4}),
    memory=memory,
    return_source_documents=True,
    verbose=True
)

# Multi-turn conversation
print("💬 Conversational RAG (type 'quit' to exit)\n")

while True:
    question = input("You: ")
    if question.lower() == 'quit':
        break
    
    result = conversational_rag({"question": question})
    print(f"Bot: {result['answer']}")
    print(f"Sources: {[doc.metadata['source'] for doc in result['source_documents']]}\n")

Advanced RAG Techniques

Take your RAG system to the next level with these advanced techniques:

Original: "Can I return opened stuff?"
Rewritten: "What is the return policy 
            for opened products?"

Result: Better retrieval!

Query: "Return policy?"
HyDE: Generate fake answer
"Returns allowed within 30 days..."
Embed fake answer → search
Result: Finds actual policy!

Original: "Return policy?"
Variations:
- "How do I return items?"
- "What's the refund process?"
- "Can I get my money back?"
Retrieve for all → merge results

Search: Small precise chunks
Return: Full parent document
Benefit: Precision + Context

Query: "Electronics returns in 2023"
LLM extracts:
- Search: "returns"
- Filter: category=electronics, 
          year=2023
Result: Filtered search!

1. Initial retrieval
2. LLM identifies gaps
3. Follow-up retrieval
4. Final answer
Result: More thorough answers

RAG Evaluation & Optimization

How do you know if your RAG system is working well? Track these metrics:

📊 RAG vs Fine-tuning vs In-Context Learning

When should you use each technique? Here's a comprehensive comparison:

Decision Framework: Which Technique to Use?

Real-World Example: Customer Support Bot

# Combining RAG + ICL for a customer support bot

def answer_customer_question(question):
    # 1. RAG: Retrieve relevant company policies
    relevant_docs = vector_store.similarity_search(question, k=3)
    
    # 2. ICL: Few-shot examples for formatting
    few_shot_examples = """
    Example 1:
    Customer: How long does shipping take?
    Agent: Based on our shipping policy, standard shipping takes 5-7 business days. 
           You can track your order using the tracking number sent to your email.
    
    Example 2:
    Customer: Can I return worn shoes?
    Agent: According to our returns policy, shoes must be unworn with tags attached 
           to qualify for a return within 30 days.
    """
    
    # 3. Assemble prompt with RAG context + ICL examples
    prompt = f"""
    You are a helpful customer support agent.
    
    Company Policies:
    {relevant_docs}
    
    {few_shot_examples}
    
    Now answer this question in the same style:
    Customer: {question}
    Agent:
    """
    
    # 4. Generate answer
    response = llm.generate(prompt)
    return response

# Result: Accurate (RAG) + Properly formatted (ICL) answers!

📋 Summary & Key Takeaways

🎯 What You've Mastered:

📊 Quick Reference: ICL vs RAG

🔥 Most Important Concepts

1. Emergent Ability: ICL emerges in large models (100B+ params), not explicitly trained
2. RAG = Retrieval + Generation: Don't memorize everything, look it up when needed
3. Vector Search: Semantic similarity in embedding space enables intelligent retrieval
4. Chunking Matters: Too small = lost context, too large = noisy retrieval
5. Default to RAG: For 95% of knowledge-based applications, RAG is the right choice

🧪 Self-Check Quiz

🚀 What's Next?

You've mastered two of the most powerful techniques for leveraging LLMs without retraining. In our next tutorial, Fine-tuning LLMs, we'll explore when and how to actually retrain models for specialized tasks, learning about techniques like full fine-tuning, LoRA, QLoRA, and more.