Probability Foundations & Rules

🎓 Complete all 6 tutorials to earn your Free Statistics for AI Certificate
Shareable on LinkedIn • Verified by AITutorials.site • No signup required

🎯 Why Probability Matters in AI

Every AI system deals with uncertainty. Will a user click this ad? Is this email spam? What's the chance it will rain tomorrow? Probability is the mathematical language we use to quantify uncertainty and make optimal decisions when we can't be 100% certain.

Modern AI algorithms like Naive Bayes classifiers, Hidden Markov Models, and Bayesian Neural Networks are built entirely on probability theory. Even deep learning uses probabilistic concepts like dropout (random neuron deactivation) and cross-entropy loss (measuring probability distributions).

🎲 What is Probability?

Probability measures the likelihood of an event occurring, expressed as a number between 0 and 1 (or 0% to 100%). A probability of 0 means impossible, 1 means certain, and 0.5 means equally likely to happen or not happen.

Basic Probability Formula

For equally likely outcomes:

# P(Event) = Number of favorable outcomes / Total number of possible outcomes

# Example: Rolling a die
# What's the probability of rolling a 4?
favorable_outcomes = 1  # Only one way to roll a 4
total_outcomes = 6      # Die has 6 sides
probability = favorable_outcomes / total_outcomes
print(f"P(rolling a 4) = {probability:.4f}")  # 0.1667 or 16.67%

# What's the probability of rolling an even number?
favorable_outcomes = 3  # Can roll 2, 4, or 6
total_outcomes = 6
probability = favorable_outcomes / total_outcomes
print(f"P(even number) = {probability:.4f}")  # 0.5000 or 50%

Key Probability Rules

• Rule 1: Probabilities are between 0 and 1 - 0 ≤ P(A) ≤ 1
• Rule 2: Sum of all probabilities = 1 - All possible outcomes sum to 100%
• Rule 3: Complement Rule - P(not A) = 1 - P(A)

import random
import numpy as np

# Simulate 10,000 coin flips to verify probability converges to 0.5
num_flips = 10000
heads_count = 0

for _ in range(num_flips):
    flip = random.choice(['Heads', 'Tails'])
    if flip == 'Heads':
        heads_count += 1

# Law of Large Numbers: As trials increase, empirical probability → theoretical probability
probability_heads = heads_count / num_flips
print(f"After {num_flips} flips:")
print(f"Heads appeared {heads_count} times")
print(f"P(Heads) = {probability_heads:.4f}")  # Should be close to 0.5000
print(f"Theoretical probability = 0.5000")

# Complement rule example
probability_tails = 1 - probability_heads
print(f"P(Tails) = {probability_tails:.4f}")

🔗 Conditional Probability

Conditional probability asks: "What's the probability of A happening, given that B has already happened?" This is written as P(A|B), read as "probability of A given B."

The Formula

P(A|B) = P(A and B) / P(B)

This is crucial in AI for classification, recommendation systems, and decision making under uncertainty.

# Email spam classification example
# Given an email contains the word "lottery", what's the probability it's spam?

# Historical data (counts from 10,000 emails)
total_emails = 10000
spam_emails = 2000
contains_lottery = 500
spam_and_lottery = 450

# P(Spam) - prior probability
p_spam = spam_emails / total_emails
print(f"P(Spam) = {p_spam:.4f}")  # 0.2000 or 20%

# P(Lottery) - probability email contains "lottery"
p_lottery = contains_lottery / total_emails
print(f"P(Lottery) = {p_lottery:.4f}")  # 0.0500 or 5%

# P(Lottery and Spam) - joint probability
p_lottery_and_spam = spam_and_lottery / total_emails
print(f"P(Lottery and Spam) = {p_lottery_and_spam:.4f}")  # 0.0450

# P(Spam | Lottery) - conditional probability using formula
p_spam_given_lottery = p_lottery_and_spam / p_lottery
print(f"\nP(Spam | Lottery) = {p_spam_given_lottery:.4f}")  # 0.9000 or 90%

# Interpretation: If email contains "lottery", there's 90% chance it's spam!

Real-World Application: Medical Diagnosis

# Medical test accuracy scenario
# Disease affects 1% of population
# Test has 95% sensitivity (detects disease when present)
# Test has 90% specificity (negative when disease absent)

# Given a positive test, what's the probability of actually having the disease?

p_disease = 0.01              # P(D) - 1% prevalence
p_no_disease = 0.99           # P(not D)
sensitivity = 0.95            # P(Positive Test | Disease)
specificity = 0.90            # P(Negative Test | No Disease)
p_false_positive = 1 - specificity  # 0.10

# P(Positive Test) using law of total probability
p_positive = (sensitivity * p_disease) + (p_false_positive * p_no_disease)
print(f"P(Positive Test) = {p_positive:.4f}")  # 0.1085

# P(Disease | Positive Test) - conditional probability
p_disease_given_positive = (sensitivity * p_disease) / p_positive
print(f"P(Disease | Positive Test) = {p_disease_given_positive:.4f}")  # 0.0876

print("\nInterpretation: Only 8.76% chance of having disease despite positive test!")
print("This happens because the disease is rare (1% prevalence)")

🎯 Independence

Two events A and B are independent if knowing one occurred doesn't change the probability of the other. Mathematically: P(A|B) = P(A) or equivalently P(A and B) = P(A) × P(B).

Examples of Independence

• Independent: Flipping a coin twice - first flip doesn't affect second
• Independent: Rolling two dice - one die doesn't know what the other rolled
• Not Independent: Drawing cards without replacement - first card affects what's left
• Not Independent: Weather tomorrow and weather today - correlated

# Testing independence: Coin flips
import random

def flip_coin():
    return random.choice(['H', 'T'])

# Flip two coins 10,000 times
trials = 10000
both_heads = 0
first_heads = 0
second_heads = 0

for _ in range(trials):
    flip1 = flip_coin()
    flip2 = flip_coin()
    
    if flip1 == 'H':
        first_heads += 1
    if flip2 == 'H':
        second_heads += 1
    if flip1 == 'H' and flip2 == 'H':
        both_heads += 1

# Calculate probabilities
p_first = first_heads / trials
p_second = second_heads / trials
p_both = both_heads / trials
p_independent = p_first * p_second  # If independent: P(A and B) = P(A) × P(B)

print(f"P(First = Heads) = {p_first:.4f}")     # ~0.5000
print(f"P(Second = Heads) = {p_second:.4f}")   # ~0.5000
print(f"P(Both Heads) = {p_both:.4f}")         # ~0.2500
print(f"P(First) × P(Second) = {p_independent:.4f}")  # ~0.2500
print(f"\nAre they independent? {abs(p_both - p_independent) < 0.01}")

Independence in Machine Learning

# Naive Bayes assumes feature independence (hence "naive")
# Example: Email spam classification with multiple words

# P(Spam | "lottery" and "click" and "winner")
# Naive Bayes assumption: words are independent given spam/ham

p_spam = 0.20
p_lottery_given_spam = 0.70
p_click_given_spam = 0.60
p_winner_given_spam = 0.65

p_lottery_given_ham = 0.05
p_click_given_ham = 0.20
p_winner_given_ham = 0.10

# Using independence assumption: P(A and B | C) = P(A|C) × P(B|C)
# Calculate likelihood for spam class
likelihood_spam = (p_lottery_given_spam * 
                   p_click_given_spam * 
                   p_winner_given_spam * 
                   p_spam)

# Calculate likelihood for ham (not spam) class
p_ham = 1 - p_spam
likelihood_ham = (p_lottery_given_ham * 
                  p_click_given_ham * 
                  p_winner_given_ham * 
                  p_ham)

# Normalize to get probabilities
p_spam_given_words = likelihood_spam / (likelihood_spam + likelihood_ham)
p_ham_given_words = likelihood_ham / (likelihood_spam + likelihood_ham)

print(f"P(Spam | words) = {p_spam_given_words:.4f}")  # ~0.9812
print(f"P(Ham | words) = {p_ham_given_words:.4f}")    # ~0.0188
print(f"Classification: {'SPAM' if p_spam_given_words > 0.5 else 'HAM'}")

🧮 Bayes' Theorem

Bayes' Theorem is one of the most important formulas in AI and statistics. It lets us flip conditional probabilities: if we know P(B|A), we can calculate P(A|B). This is the foundation of Bayesian inference and many ML algorithms.

The Formula

P(A|B) = [P(B|A) × P(A)] / P(B)

Components:

• P(A|B) - Posterior: What we want to know (probability of A given evidence B)
• P(B|A) - Likelihood: Probability of observing B if A is true
• P(A) - Prior: Our initial belief about A before seeing evidence
• P(B) - Evidence: Probability of observing B (often calculated using law of total probability)

# Classic example: Is it raining given that someone has an umbrella?

# Prior knowledge
p_rain = 0.20              # P(Rain) - 20% chance of rain today
p_no_rain = 0.80           # P(No Rain)

# Likelihood - how people behave
p_umbrella_given_rain = 0.90      # 90% carry umbrella if raining
p_umbrella_given_no_rain = 0.20   # 20% carry umbrella if not raining

# Calculate P(Umbrella) - evidence (law of total probability)
p_umbrella = (p_umbrella_given_rain * p_rain + 
              p_umbrella_given_no_rain * p_no_rain)
print(f"P(Umbrella) = {p_umbrella:.4f}")  # 0.3400

# Apply Bayes' Theorem: P(Rain | Umbrella)
p_rain_given_umbrella = (p_umbrella_given_rain * p_rain) / p_umbrella
print(f"\nP(Rain | Umbrella) = {p_rain_given_umbrella:.4f}")  # 0.5294

print("\nInterpretation:")
print(f"Prior belief: {p_rain:.1%} chance of rain")
print(f"After seeing umbrella: {p_rain_given_umbrella:.1%} chance of rain")
print("Our belief increased from 20% to 53% based on evidence!")

ML Application: Bayesian Spam Filter

# Build a simple Bayesian spam classifier
import numpy as np

class NaiveBayesSpamFilter:
    def __init__(self):
        self.p_spam = 0.20  # 20% of emails are spam
        self.p_ham = 0.80
        
        # Word probabilities (trained from data)
        self.word_probs_spam = {
            'free': 0.70, 'winner': 0.65, 'click': 0.60,
            'meeting': 0.05, 'report': 0.10, 'hello': 0.30
        }
        
        self.word_probs_ham = {
            'free': 0.05, 'winner': 0.02, 'click': 0.15,
            'meeting': 0.40, 'report': 0.35, 'hello': 0.60
        }
    
    def classify(self, words):
        """Classify email using Bayes' Theorem and independence assumption"""
        
        # Calculate likelihood for spam
        likelihood_spam = self.p_spam
        for word in words:
            if word in self.word_probs_spam:
                likelihood_spam *= self.word_probs_spam[word]
        
        # Calculate likelihood for ham
        likelihood_ham = self.p_ham
        for word in words:
            if word in self.word_probs_ham:
                likelihood_ham *= self.word_probs_ham[word]
        
        # Normalize to get posterior probabilities
        total = likelihood_spam + likelihood_ham
        p_spam_given_words = likelihood_spam / total
        p_ham_given_words = likelihood_ham / total
        
        return {
            'classification': 'SPAM' if p_spam_given_words > 0.5 else 'HAM',
            'spam_probability': p_spam_given_words,
            'ham_probability': p_ham_given_words
        }

# Test the classifier
classifier = NaiveBayesSpamFilter()

# Test email 1: Spam-like words
email1 = ['free', 'winner', 'click']
result1 = classifier.classify(email1)
print("Email 1: 'free winner click'")
print(f"Classification: {result1['classification']}")
print(f"P(Spam) = {result1['spam_probability']:.4f}")
print()

# Test email 2: Professional words
email2 = ['meeting', 'report', 'hello']
result2 = classifier.classify(email2)
print("Email 2: 'meeting report hello'")
print(f"Classification: {result2['classification']}")
print(f"P(Spam) = {result2['spam_probability']:.4f}")

➕✖️ Addition & Multiplication Rules

Addition Rule (OR)

For probability of A OR B happening:

• If mutually exclusive: P(A or B) = P(A) + P(B)
• If not mutually exclusive: P(A or B) = P(A) + P(B) - P(A and B)

# Addition rule example: Drawing cards

# Mutually exclusive events (can't both happen)
# P(Drawing Ace OR King) - card can't be both
p_ace = 4/52
p_king = 4/52
p_ace_or_king = p_ace + p_king  # Mutually exclusive: just add
print(f"P(Ace or King) = {p_ace_or_king:.4f}")  # 0.1538

# Non-mutually exclusive events (can overlap)
# P(Drawing Heart OR Face card) - card can be both (e.g., King of Hearts)
p_heart = 13/52
p_face = 12/52  # Jack, Queen, King in 4 suits
p_heart_and_face = 3/52  # Jack, Queen, King of Hearts

# Must subtract overlap to avoid double-counting
p_heart_or_face = p_heart + p_face - p_heart_and_face
print(f"P(Heart or Face) = {p_heart_or_face:.4f}")  # 0.4231

# Verify by counting directly
hearts = 13
face_cards = 12
overlap = 3  # J♥, Q♥, K♥
unique_cards = hearts + face_cards - overlap  # 22 unique cards
print(f"Direct counting: {unique_cards}/52 = {unique_cards/52:.4f}")

Multiplication Rule (AND)

For probability of A AND B both happening:

• If independent: P(A and B) = P(A) × P(B)
• If not independent: P(A and B) = P(A) × P(B|A)

# Multiplication rule example: User conversion funnel

# E-commerce website conversion probabilities
p_visit = 1.00           # User visits site (given)
p_view_product = 0.60    # P(Views product | Visits)
p_add_cart = 0.40        # P(Adds to cart | Views product)
p_checkout = 0.70        # P(Checks out | Adds to cart)
p_complete = 0.90        # P(Completes purchase | Checks out)

# Probability of full conversion: Visit → View → Cart → Checkout → Purchase
# These are dependent events (each depends on previous step)
p_conversion = (p_visit * p_view_product * p_add_cart * 
                p_checkout * p_complete)

print(f"P(Full Conversion) = {p_conversion:.4f}")  # 0.1512 or 15.12%

# This means out of 1000 visitors, expect 151 purchases
visitors = 1000
expected_purchases = visitors * p_conversion
print(f"\nOut of {visitors} visitors:")
print(f"Expected purchases: {expected_purchases:.0f}")

# Calculate where users drop off
print(f"\nFunnel breakdown:")
print(f"Visitors: {visitors}")
print(f"View product: {visitors * p_view_product:.0f}")
print(f"Add to cart: {visitors * p_view_product * p_add_cart:.0f}")
print(f"Checkout: {visitors * p_view_product * p_add_cart * p_checkout:.0f}")
print(f"Purchase: {expected_purchases:.0f}")

🌍 Real-World ML Applications

1. Recommendation Systems

# Netflix-style recommendation using conditional probability

# User watch history probabilities
p_likes_action = 0.60
p_likes_comedy = 0.45
p_likes_both = 0.30

# P(Likes comedy | Likes action) - for recommendation
p_comedy_given_action = p_likes_both / p_likes_action
print(f"P(Likes Comedy | Likes Action) = {p_comedy_given_action:.4f}")

# If user watched 5 action movies, recommend comedy with this probability
recommendation_confidence = p_comedy_given_action * 100
print(f"Recommend comedy with {recommendation_confidence:.1f}% confidence")

2. A/B Test Decision Making

# A/B test: Which button color gets more clicks?

# Version A (blue button)
visitors_a = 1000
clicks_a = 120
p_click_a = clicks_a / visitors_a

# Version B (green button)  
visitors_b = 1000
clicks_b = 145
p_click_b = clicks_b / visitors_b

print(f"Version A (Blue): {p_click_a:.1%} click rate")
print(f"Version B (Green): {p_click_b:.1%} click rate")
print(f"Improvement: {(p_click_b - p_click_a)/p_click_a * 100:.1f}%")

# Simple significance check (detailed hypothesis testing in Tutorial 5)
difference = abs(p_click_b - p_click_a)
print(f"\nDifference: {difference:.3f} or {difference*100:.1f} percentage points")
if difference > 0.02:  # Rule of thumb: >2pp difference is meaningful
    print("Result: Statistically meaningful! Choose green button.")

3. Fraud Detection

# Credit card fraud detection using probability

# Prior probability based on historical data
p_fraud = 0.001  # 0.1% of transactions are fraudulent

# Risk factors (likelihoods)
p_high_amount_given_fraud = 0.80      # Fraudsters often try large amounts
p_high_amount_given_legit = 0.10      # Legit users occasionally spend big

p_foreign_given_fraud = 0.70          # Fraud often involves foreign transactions
p_foreign_given_legit = 0.15          # Legit users sometimes travel

# Transaction to evaluate: High amount + Foreign location
# Assuming independence of features given fraud/legit

def fraud_probability(high_amount, foreign):
    """Calculate P(Fraud | Features) using Bayes' Theorem"""
    
    # Likelihood of fraud given features
    if high_amount and foreign:
        likelihood_fraud = (p_fraud * 
                           p_high_amount_given_fraud * 
                           p_foreign_given_fraud)
    else:
        likelihood_fraud = p_fraud
    
    # Likelihood of legitimate given features
    p_legit = 1 - p_fraud
    if high_amount and foreign:
        likelihood_legit = (p_legit * 
                           p_high_amount_given_legit * 
                           p_foreign_given_legit)
    else:
        likelihood_legit = p_legit
    
    # Posterior probability
    total = likelihood_fraud + likelihood_legit
    return likelihood_fraud / total

# Test different scenarios
scenarios = [
    (False, False, "Normal transaction"),
    (True, False, "High amount only"),
    (False, True, "Foreign only"),
    (True, True, "High amount + Foreign")
]

print("Fraud Detection Results:")
print("-" * 60)
for high_amt, foreign, description in scenarios:
    prob = fraud_probability(high_amt, foreign)
    risk_level = "HIGH" if prob > 0.01 else "MEDIUM" if prob > 0.001 else "LOW"
    print(f"{description:30} P(Fraud)={prob:.4f} [{risk_level}]")

💻 Practice Exercises

Exercise 1: Customer Behavior Analysis

Scenario: An online store has the following data:

• 60% of visitors are mobile users, 40% desktop
• Mobile users make purchases 8% of the time
• Desktop users make purchases 15% of the time

Questions:

1. What's the overall purchase rate P(Purchase)?
2. If someone made a purchase, what's the probability they're on mobile P(Mobile|Purchase)?

Exercise 2: Medical Screening

Scenario: A disease affects 2% of the population. A test has:

• 98% sensitivity (detects disease when present)
• 95% specificity (negative when disease absent)

Question: If someone tests positive, what's the probability they actually have the disease?

Exercise 3: Recommendation System

Scenario: Build a simple movie recommender:

• P(User watches Sci-Fi) = 0.35
• P(User watches Action) = 0.50
• P(User watches both) = 0.20

Questions:

1. Are Sci-Fi and Action preferences independent?
2. If a user likes Sci-Fi, what's P(likes Action | likes Sci-Fi)?
3. What's P(likes at least one genre)?

Exercise 4: Build a Simple Classifier

Challenge: Implement a Naive Bayes classifier to predict if a student will pass based on:

• Study hours (>5 hours or ≤5 hours)
• Attendance (>80% or ≤80%)

Use the training data to calculate probabilities, then classify new students.

📝 Summary

In this tutorial, you've mastered the probability foundations essential for AI and machine learning:

🎯 Test Your Knowledge