The Story So Far

The Problem We're Solving

You trained a model. It looks great on your data.

But will it work on NEW data?

On Training Data	On New Data	Status
95% accuracy	50% accuracy	Problem!
85% accuracy	83% accuracy	Good!

Today's Questions

1. Why do some models fail on new data?
2. How do we detect this problem?
3. How do we choose between different models?

A Simple Example

Task: Predict house prices from size

You have 10 houses with prices. You fit a model.

Question: Which model is best?

Model 1: Too Simple (Underfitting)

Model: Average price for all houses = ₹50 lakhs

Problem: The model ignores the pattern completely!

Model 2: Too Complex (Overfitting)

Model: Memorizes every house exactly

Perfect on training data!

But for a NEW house of 1500 sq ft? → Crazy prediction!

Model 3: Just Right

Model: Price = ₹20 lakhs + ₹30 per sq ft

Not perfect on training, but captures the pattern!

For a NEW house of 1500 sq ft? → ₹65 lakhs (reasonable!)

The Key Insight

Visual: Polynomial Fitting Example

What Controls Complexity?

More complexity = More risk of overfitting!

Underfitting: Like a Student Who Skipped Class

Symptoms:

• Bad on training data
• Bad on test data
• Model is too simple to capture the pattern

Solutions:

• Use more features
• Use a more complex model
• Train longer

Overfitting: Like a Student Who Memorized Without Understanding

Symptoms:

• Great on training data
• Bad on test data
• Model memorized the training examples

Solutions:

• Get more training data
• Use a simpler model
• Use regularization

Analogy: Studying for an Exam

Think of it like preparing for a test:

• Underfitting: Didn't study → Failed homework, failed exam
• Overfitting: Memorized answers → Perfect homework, failed exam
• Good fit: Understood concepts → Good on both!

Another Analogy: Fitting Clothes

A model should "fit" the data well enough to be useful, but not so tight it can't generalize!

More Everyday Analogies

Good models generalize to new situations!

Signs You're Overfitting

Watch out for these warning signs:

Signs You're Underfitting

Real-World Overfitting: COVID X-Ray Detection

What happened: Model achieved 90%+ accuracy but learned to read hospital IDs in corner of X-rays, not lung features! COVID patients came from specific hospitals.

How Much Data is Enough?

Rough guidelines:

Visual: The Complexity Tradeoff

The Bias-Variance Tradeoff

Total Error = Bias² + Variance + Noise

Model	Bias	Variance	Result
Too simple	High	Low	Underfitting
Too complex	Low	High	Overfitting
Just right	Low	Low	Good!

The Sweet Spot

Goal: Find the model complexity that minimizes TOTAL error

Why We Need Multiple Sets

Scenario: You train a model and evaluate it on the same data.

model.fit(X, y)           # Train on all data
accuracy = model.score(X, y)  # Test on same data
print(f"Accuracy: {accuracy:.0%}")  # 99% !!!

Problem: Of course it's high - the model saw these examples!

The Solution: Hold Out Test Data

from sklearn.model_selection import train_test_split

# Split: 80% train, 20% test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train on training set only
model.fit(X_train, y_train)

# Evaluate on unseen test set
test_accuracy = model.score(X_test, y_test)  # Honest evaluation!

Train vs Test Error

train_accuracy = model.score(X_train, y_train)  # 95%
test_accuracy = model.score(X_test, y_test)     # 85%

The Gap Tells You Everything

gap = train_accuracy - test_accuracy

Gap	Diagnosis
< 5%	Good generalization
5-15%	Some overfitting
> 15%	Severe overfitting

What About Model Selection?

Problem: You want to try different models and pick the best.

If you use the test set to choose... you're peeking!

# Don't do this!
for model in [model1, model2, model3]:
    model.fit(X_train, y_train)
    score = model.score(X_test, y_test)  # Using test to choose!

Three-Way Split

Solution: Add a validation set for model selection.

The Three Sets

In Code

from sklearn.model_selection import train_test_split

# First split: separate test set
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Second split: train and validation
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42
)

# Now: 60% train, 20% val, 20% test

Using the Three Sets

# 1. Try different models on validation set
for model in models:
    model.fit(X_train, y_train)
    val_score = model.score(X_val, y_val)
    print(f"{model}: {val_score:.2%}")

# 2. Pick the best model
best_model = ...  # Based on validation scores

# 3. Final evaluation on test set (ONCE!)
final_score = best_model.score(X_test, y_test)
print(f"Final test score: {final_score:.2%}")

Why Do We Need Cross-Validation?

Problem: A single train/test split can be lucky or unlucky!

train_test_split(X, y, random_state=1)  # Score: 87%
train_test_split(X, y, random_state=2)  # Score: 92%
train_test_split(X, y, random_state=3)  # Score: 84%

Which is the true score? We don't know!

The Problem with Single Splits

We need a more reliable way to estimate performance!

K-Fold Cross-Validation: The Solution

Key insight: Every data point gets to be in the test set exactly once!

K-Fold: A Concrete Example

You have 100 samples, K=5 folds:

Final score:

Why K-Fold Works

Example: Score = 89% ± 1.5% tells us much more than just "89%"

K-Fold in sklearn

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()

# 5-fold cross-validation
scores = cross_val_score(model, X, y, cv=5)

print(f"Scores per fold: {scores}")
# [0.87, 0.89, 0.91, 0.88, 0.90]

print(f"Mean: {scores.mean():.3f} ± {scores.std():.3f}")
# Mean: 0.890 ± 0.015

Choosing K: Trade-offs

But Wait... What About Hyperparameters?

New problem: We want to tune hyperparameters (like regularization C)

# Which C is best?
for C in [0.01, 0.1, 1, 10, 100]:
    model = LogisticRegression(C=C)
    score = cross_val_score(model, X, y, cv=5).mean()
    print(f"C={C}: {score}")

Danger: We used the test folds to CHOOSE the best C!

Now our "test" score is biased — we've leaked information!

The Data Leakage Problem

What went wrong:

1. We evaluated C=0.01 on folds → got a score
2. We evaluated C=0.1 on folds → got a score
3. We picked the C with best score
4. We reported that score as "test accuracy"

But that score was used to MAKE A DECISION!

It's like a student seeing the test before the exam.

Data Leakage: A Hiring Analogy

Imagine you're hiring:

In ML:

• Test set = the "real interview"
• Using test set to choose model = seeing answers first
• Your "accuracy" is now meaningless

Nested Cross-Validation: The Fix

How Nested CV Works

Process for each outer fold:

1. Hold out test fold (don't touch it!)
2. Run inner CV on remaining data to find best hyperparameters
3. Train final model with best hyperparameters
4. Evaluate on test fold → one honest score

Average all outer fold scores → reliable estimate!

Nested CV in sklearn

from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.linear_model import LogisticRegression

# Inner loop: find best hyperparameters using 3-fold CV
param_grid = {'C': [0.01, 0.1, 1, 10, 100]}
inner_cv = GridSearchCV(LogisticRegression(), param_grid, cv=3)

# Outer loop: honest evaluation using 5-fold CV
scores = cross_val_score(inner_cv, X, y, cv=5)

print(f"Nested CV score: {scores.mean():.3f} ± {scores.std():.3f}")
# This score is honest — no data leakage!

Summary: When to Use What

The Model Selection Workflow

Step-by-Step: Model Selection

Common Mistakes to Avoid

Common Hyperparameters

Hyperparameters = Settings YOU choose before training

Simple Hyperparameter Tuning

# Try different max_depth values
for depth in [2, 3, 5, 10, None]:
    model = DecisionTreeClassifier(max_depth=depth)
    scores = cross_val_score(model, X, y, cv=5)
    print(f"depth={depth}: {scores.mean():.3f}")

depth=2:    0.78  ← Underfitting
depth=3:    0.85
depth=5:    0.88  ← Sweet spot
depth=10:   0.85
depth=None: 0.75  ← Overfitting

What to Report

When presenting your model, always report:

Red Flags to Watch For

Summary: The Key Ideas

1. Overfitting = memorizing, not learning

   • High training accuracy, low test accuracy

2. Underfitting = too simple to learn

   • Low training AND test accuracy

3. Train/Validation/Test split is essential

   • Never use test set for model selection!

4. Cross-validation gives reliable estimates

   • Use cross_val_score in sklearn

Regularization: Preventing Overfitting

What is Regularization?

Regularization = Add a penalty for complexity

λ is a hyperparameter you tune!

Regularization Intuition

Think of it like a budget constraint:

Analogy: Writing an essay

• No limit → rambling, covers every detail
• Word limit → focused, captures key points

Types of Regularization

from sklearn.linear_model import Ridge, Lasso

model = Ridge(alpha=1.0)   # L2: all features kept, smaller weights
model = Lasso(alpha=0.1)   # L1: some features removed entirely

When to Use Regularization?

Learning Curves: Diagnosing Problems

Reading Learning Curves

from sklearn.model_selection import learning_curve
train_sizes, train_scores, val_scores = learning_curve(
    model, X, y, cv=5, train_sizes=[0.2, 0.4, 0.6, 0.8, 1.0]
)

Grid Search: Automated Tuning

Instead of manually trying hyperparameters:

from sklearn.model_selection import GridSearchCV

param_grid = {
    'max_depth': [3, 5, 10, None],
    'min_samples_split': [2, 5, 10]
}

grid_search = GridSearchCV(
    DecisionTreeClassifier(),
    param_grid,
    cv=5,
    scoring='accuracy'
)
grid_search.fit(X_train, y_train)

print(f"Best params: {grid_search.best_params_}")
print(f"Best score: {grid_search.best_score_:.3f}")

What We Skipped (Advanced Topics)

These are important but more advanced:

You'll learn these in advanced ML courses!

Code Summary

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Cross-validation comparison
from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X_train, y_train, cv=5)
print(f"CV Score: {scores.mean():.3f} ± {scores.std():.3f}")

# Final evaluation (only once!)
final_score = model.score(X_test, y_test)

PyTorch: Data Splitting

from torch.utils.data import random_split, DataLoader

# Assume dataset has 1000 samples
dataset = MyDataset(...)

# Split: 60% train, 20% val, 20% test
train_size = int(0.6 * len(dataset))
val_size = int(0.2 * len(dataset))
test_size = len(dataset) - train_size - val_size

train_set, val_set, test_set = random_split(
    dataset, [train_size, val_size, test_size]
)

PyTorch: DataLoaders

# Create DataLoaders for batching
train_loader = DataLoader(train_set, batch_size=32, shuffle=True)
val_loader = DataLoader(val_set, batch_size=32, shuffle=False)
test_loader = DataLoader(test_set, batch_size=32, shuffle=False)

# Training loop
for epoch in range(num_epochs):
    for X_batch, y_batch in train_loader:
        # Train on batch
        ...

    # Validate after each epoch
    for X_batch, y_batch in val_loader:
        # Compute validation loss
        ...

PyTorch: K-Fold Cross-Validation

from sklearn.model_selection import KFold

kfold = KFold(n_splits=5, shuffle=True)

for fold, (train_idx, val_idx) in enumerate(kfold.split(dataset)):
    train_subset = Subset(dataset, train_idx)
    val_subset = Subset(dataset, val_idx)

    train_loader = DataLoader(train_subset, batch_size=32)
    val_loader = DataLoader(val_subset, batch_size=32)

    # Train and evaluate for this fold
    score = train_and_evaluate(model, train_loader, val_loader)
    print(f"Fold {fold}: {score:.3f}")