Inference Optimization

🎓 Complete all tutorials to earn your Free LLMs & Transformers Certificate
Shareable on LinkedIn • Verified by AITutorials.site • No signup fee

⚡ Why Optimize Inference? The Production Reality

You've trained your LLM, fine-tuned it, achieved great results. Now comes the hard part: serving it to millions of users in production. This is where inference optimization becomes critical.

The Real Cost of Unoptimized Inference

Scenario: You're running a chatbot that serves 1 million queries per day.

What Makes Inference Slow?

7B model (FP32): 28GB weights
A100 memory bandwidth: 2 TB/s
Time to load: 28GB ÷ 2TB/s = 14ms

For each token: 14ms load + 1ms compute
Result: Memory-bound, not compute-bound!

Generate 100 tokens:
Token 1: Full forward pass
Token 2: Full forward pass
Token 3: Full forward pass
...
Total: 100 × forward pass time

Single query: 10% GPU utilization
A100 80GB: $1.50/hour
Wasted: ~$1.35/hour doing nothing!

Tokenization: 5-10ms
Model loading (first token): 200ms
Generation: 50ms/token × 50 tokens = 2.5s
Decoding: 5-10ms
Total: ~2.8s for 50 tokens

The Optimization Stack: What You'll Learn

Layer 1: Model Compression
├── Quantization (INT8, INT4, NF4)
├── Pruning (remove unnecessary weights)
└── Distillation (train smaller model)
    Result: 4-8x smaller, 4-8x faster

Layer 2: Serving Optimization
├── Batching (continuous, dynamic)
├── KV Caching (reuse computations)
├── PagedAttention (memory efficiency)
└── Speculative Decoding (parallel generation)
    Result: 5-20x higher throughput

Layer 3: Infrastructure
├── vLLM / TensorRT-LLM (optimized engines)
├── Ray Serve (distributed serving)
├── Model parallelism (split across GPUs)
└── Monitoring & autoscaling
    Result: Production-ready, reliable, scalable

Combined Impact: 50-100x cost reduction! 🚀

🔴 Quantization: The #1 Optimization Technique

Quantization reduces the precision of model weights from 32-bit or 16-bit floating-point numbers to 8-bit or even 4-bit integers. This is the single most impactful optimization you can apply—easy to implement and delivers massive speedups with minimal quality loss.

Precision Levels Explained

Quantization Methods

Popular Quantization Formats

GPTQ (GPU-Optimized)

Best for: GPU inference with maximum speed

# Quantize model to GPTQ INT4 (one-time process)\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nimport torch\n\n# 1. Configure quantization\nquantize_config = BaseQuantizeConfig(\n    bits=4,  # 4-bit quantization\n    group_size=128,  # Quantize in groups of 128 weights\n    desc_act=False  # Activation order (False = faster)\n)\n\n# 2. Load model\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    \"meta-llama/Llama-2-7b-hf\",\n    quantize_config=quantize_config\n)\n\n# 3. Quantize (requires ~2K calibration examples)\nimport datasets\ncalibration_data = datasets.load_dataset(\"c4\", split=\"train[:2000]\")\nmodel.quantize(calibration_data)\n\n# 4. Save quantized model\nmodel.save_quantized(\"./llama-2-7b-gptq-int4\")\n\n# Result: 28GB → 4GB model, 6-8x faster inference!\nprint(\"✅ GPTQ quantization complete!\")

GGUF (CPU-Optimized)

Best for: Running models on CPU (laptops, servers without GPU)

# Download pre-quantized GGUF model from HuggingFace\nwget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf\n\n# Or convert your own model using llama.cpp\ngit clone https://github.com/ggerganov/llama.cpp\ncd llama.cpp\nmake\n\n# Convert HuggingFace model to GGUF\npython convert.py /path/to/llama-2-7b --outfile llama-2-7b-f16.gguf\n\n# Quantize to 4-bit\n./quantize llama-2-7b-f16.gguf llama-2-7b-Q4_K_M.gguf Q4_K_M\n\n# Run inference on CPU!\n./main -m llama-2-7b-Q4_K_M.gguf -p \"Hello, world!\" -n 100 -t 8\n\n# Result: Run 7B model on MacBook Pro M2! 🚀

AWQ (Activation-Aware Quantization)

Best for: Highest quality 4-bit quantization

# AWQ quantization (state-of-the-art quality)\nfrom awq import AutoAWQForCausalLM\nfrom transformers import AutoTokenizer\n\nmodel_path = \"meta-llama/Llama-2-7b-hf\"\nquant_path = \"llama-2-7b-awq-int4\"\n\n# Quantize\nmodel = AutoAWQForCausalLM.from_pretrained(model_path)\ntokenizer = AutoTokenizer.from_pretrained(model_path)\n\nmodel.quantize(tokenizer, quant_config={\"zero_point\": True, \"q_group_size\": 128})\nmodel.save_quantized(quant_path)\n\nprint(\"✅ AWQ quantization complete! Better quality than GPTQ.\")

BitsAndBytes: Easiest Quantization

For quick experimentation, BitsAndBytes provides zero-effort quantization:

# Install: pip install bitsandbytes accelerate\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\nimport torch\n\n# INT8 quantization (one line!)\nmodel_int8 = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama/Llama-2-7b-hf\",\n    load_in_8bit=True,  # ✅ Quantize to INT8\n    device_map=\"auto\"\n)\n\n# 4-bit quantization (even faster)\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",  # NormalFloat4 (optimal for neural nets)\n    bnb_4bit_compute_dtype=torch.float16,\n    bnb_4bit_use_double_quant=True\n)\n\nmodel_4bit = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama/Llama-2-7b-hf\",\n    quantization_config=bnb_config,\n    device_map=\"auto\"\n)\n\n# Use it!\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-hf\")\ninputs = tokenizer(\"Hello, how are you?\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model_4bit.generate(**inputs, max_new_tokens=50)\nprint(tokenizer.decode(outputs[0], skip_special_tokens=True))\n\n# Result: 7B model runs on 16GB GPU instead of 40GB! 🎉

Quantization Quality Comparison

How much quality do you lose? Real benchmarks on Llama 2 7B:

📚 Distillation

Train a smaller "student" model to mimic a larger "teacher" model. Student is faster but nearly as accurate.

How It Works

1. Train on task with large model (teacher)
2. Use teacher outputs as soft targets for student
3. Student learns to mimic teacher
4. Deploy student (faster, smaller)

Results

• 7B student mimics 13B teacher (10x faster)
• 95-98% of teacher performance
• Much faster inference
• Works great with quantization

# Knowledge distillation (simplified)
# Teacher model: large, accurate
# Student model: small, fast
# Goal: train student to match teacher

for batch in dataloader:
    # Get teacher predictions (soft targets)
    with torch.no_grad():
        teacher_logits = teacher(batch)
    
    # Get student predictions
    student_logits = student(batch)
    
    # KL divergence loss (match probabilities)
    loss = KL_divergence(
        student_logits / temperature,
        teacher_logits / temperature
    )
    
    loss.backward()
    optimizer.step()

⚙️ Inference Frameworks

vLLM Example

# Install
pip install vllm

# Serve model
python -m vllm.entrypoints.openai_compatible_server \
  --model meta-llama/Llama-2-7b-hf \
  --quantization gptq

# Now use like OpenAI API!

llama.cpp Example

# Download GGUF model
wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf

# Run locally
./main -m llama-2-7b.Q4_K_M.gguf -p "Hello" -n 128

📊 Optimization Checklist

1. Quantize: Use INT8 or INT4. 4x speed gain
2. Use vLLM: Batching + caching. 2x-10x speedup
3. Batch requests: Process multiple queries together
4. Cache KV values: Don't recompute for same context
5. Distributed inference: Split model across multiple GPUs
6. Monitor: Track latency, throughput, cost

📋 Summary

What You've Learned:

• Quantization: Reduce precision (32→8/4 bit). 4-8x speed for <2% quality loss
• Distillation: Train small model to mimic large. 10x speedup with 95% quality
• vLLM: Fast batched inference with KV caching
• llama.cpp: CPU inference, run on any machine
• Optimization is critical for production deployment

What's Next?

In our final tutorial, Building LLM Applications, we'll put it all together into production systems.