The Story So Far

A Child vs. A Computer

A 3-year-old child can:

• Recognize their parent in any photo
• Spot a dog across the street
• Know if a picture is upside down

A computer sees:

• Just numbers (pixels)
• No concept of "dog" or "parent"
• Millions of numbers

Why Computer Vision Matters

Today's Agenda

1. Images as Data - How computers "see"
2. CNNs - Neural networks for images
3. Object Detection - Finding objects and their locations
4. YOLO - Real-time detection

What Is an Image to a Computer?

An image is just a grid of numbers!

Each number = brightness of one pixel

Human View vs Computer View

Same image, different perspectives:

Grayscale Images

Each pixel = one number (0-255)

import numpy as np
from PIL import Image

img = Image.open('digit.png').convert('L')  # Grayscale
pixels = np.array(img)
print(pixels.shape)  # (28, 28)
print(pixels[14, 14])  # Value at center pixel

Color Images (RGB)

Each pixel = 3 numbers (Red, Green, Blue)

pixels = np.array(img)
print(pixels.shape)  # (224, 224, 3)
#                      Height × Width × RGB

MNIST: The "Hello World" of Vision

Handwritten digits: 28×28 grayscale

• Each image is a single digit (0-9)
• Just 784 numbers per image
• Task: Classify which digit it is

This is what your phone keyboard uses for handwriting!

MNIST: What the Computer Sees

Same image, two views:

Each pixel is a number from 0 (black) to 255 (white).

The Challenge

Same dog, different positions:

To a human: Same dog!
To a computer: Completely different!

MNIST: Real Examples

What the computer actually sees:

# Loading real MNIST data
from torchvision import datasets
mnist = datasets.MNIST('./data', download=True)
image, label = mnist[0]
print(image.size)  # (28, 28)
print(label)       # 5 (it's a digit 5!)

Why is Vision Hard?

A good vision system must handle ALL of these!

Why Not Fully Connected Networks?

Problem 1: Too many parameters

Problem 2: No spatial understanding

• FC treats every pixel independently
• CNN looks at neighboring pixels together

The Key Idea: Look at Small Regions

Instead of looking at entire image at once...

Look at small patches and find patterns!

A small filter (like a 3x3 edge detector) slides across the entire image, detecting patterns everywhere.

Same edge detector works EVERYWHERE in the image!

Filters (Kernels)

A filter is a small grid of numbers that detects a pattern:

The network LEARNS which filters are useful!

Convolution: Step by Step

How a 3×3 filter slides across an image:

Image (5×5):          Filter (3×3):        Output:
[1 2 3 4 5]           [1 0 -1]
[1 2 3 4 5]     *     [1 0 -1]      =     [? ? ?]
[1 2 3 4 5]           [1 0 -1]            [? ? ?]
[1 2 3 4 5]                               [? ? ?]
[1 2 3 4 5]

1. Place filter at top-left
2. Multiply element-wise, sum up → one output number
3. Slide right, repeat
4. Slide down, repeat

Convolution Example

Detecting vertical edges:

Input patch:     Filter:         Calculation:
[10 10 100]     [1  0 -1]       10×1 + 10×0 + 100×(-1) +
[10 10 100]  ×  [1  0 -1]   =   10×1 + 10×0 + 100×(-1) +
[10 10 100]     [1  0 -1]       10×1 + 10×0 + 100×(-1)

                            =   30 - 300 = -270 (strong edge!)

High output = strong edge detected at this location!

Why Does This Filter Detect Edges?

Look at what the filter does:

If left = right: Sum ≈ 0 (no edge)
If left ≠ right: Sum is large (EDGE!)

Example:

• Uniform region [100, 100, 100]: +100 - 100 = 0
• Edge region [10, 10, 200]: +10 - 200 = -190 (strong!)

Vertical vs Horizontal Filters

Same image, different filters → different edges detected!

Edge Detection in Action

Combining filters reveals all edges in an image!

Multiple Filters, Multiple Features

Each filter detects something different:

• Filter 1: Vertical edges
• Filter 2: Horizontal edges
• Filter 3: Diagonal edges
• Filter 4: Corners
• ... and so on

The network learns what filters are useful for the task!

Channels: From Grayscale to Color

Each channel gets its own filter weights, then sum them up!

CNN: The Big Picture

Convolutional Neural Network:

Image → [Conv → ReLU → Pool] → [Conv → ReLU → Pool] → FC → Output
         ↓                       ↓                      ↓
      Detect               Detect complex          Classify
      edges                patterns (ears, eyes)

Pooling: Shrinking the Image

Max Pooling: Take the maximum value from each region

Pooling: The Intuition

Think of it like summarizing:

Why max pooling specifically?

• If ANY pixel detected a feature strongly → keep it!
• Exact location doesn't matter, presence does
• Cat moved 2 pixels left? Same features detected!

Stride and Padding

Two important concepts:

Stride 1: Filter moves 1 pixel → larger output
Stride 2: Filter moves 2 pixels → smaller output (half size)

Receptive Field

Each neuron in later layers "sees" a larger part of the image:

Deep layers see the whole image! This is how CNNs understand global context.

A Simple CNN in PyTorch

import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        # Convolutional layers
        self.conv1 = nn.Conv2d(1, 32, kernel_size=3)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3)
        self.pool = nn.MaxPool2d(2)

        # Classification layer
        self.fc = nn.Linear(64 * 5 * 5, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 64 * 5 * 5)  # Flatten
        return self.fc(x)

Why CNNs Work So Well

The Feature Hierarchy

Why CNNs work: Each layer builds on the previous one!

• Layer 1: Edges, lines (simple)
• Layer 2: Corners, textures (combinations)
• Layer 3: Parts like eyes, ears (meaningful)
• Layer 4+: Full objects and concepts (abstract)

What Does the CNN "See"?

Imagine you're the network looking at a cat photo:

The network builds up understanding layer by layer!

Famous CNN Moment: ImageNet 2012

Before CNNs (hand-crafted features): 25.8% error
AlexNet (deep CNN):                  16.4% error  ← HUGE jump!
Human performance:                   ~5% error

This started the deep learning revolution!

ImageNet: The Game Changer

Why it mattered:

• Large enough to train deep networks
• Diverse enough to test generalization
• Competition drove rapid progress!

The ImageNet Journey

2015: ResNet beat human-level performance!

Famous CNN Architectures

Trend: Deeper networks with clever tricks to train them!

Data Augmentation

Problem: Not enough training images?

Solution: Create variations of existing images!

Data Augmentation in PyTorch

from torchvision import transforms

train_transform = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(10),
    transforms.ColorJitter(brightness=0.2),
    transforms.RandomResizedCrop(224),
    transforms.ToTensor(),
])

# Apply during training
train_dataset = ImageFolder('train/', transform=train_transform)

Result: 1000 images become effectively 10,000+!

Transfer Learning

The big insight: CNNs trained on ImageNet learn general features!

Transfer Learning in Practice

from torchvision import models

# Load pre-trained ResNet
model = models.resnet18(pretrained=True)

# Freeze early layers (don't train them)
for param in model.parameters():
    param.requires_grad = False

# Replace final layer for our task (e.g., 10 classes)
model.fc = nn.Linear(512, 10)

# Now train only the new layer!

Result: Train on 1000 images instead of millions!

When to Use Transfer Learning?

Transfer Learning: The School Analogy

Instead of teaching a child from scratch:

ImageNet pre-training gives you:

• 10+ years of "visual education"
• Knowledge of edges, textures, shapes
• Just add your specific knowledge on top!

This is why models fine-tuned on 100 images beat models trained from scratch on 1000!

Let's Build Up Gradually

We'll go step by step:

Each step adds complexity. Let's master each one!

Step 1: Classification (Review)

What the neural network outputs:

Image → CNN → FC → Softmax → [0.9, 0.05, 0.05]
                              cat   dog   bird

Loss: Cross-entropy (how wrong is the class prediction?)

Step 2: Localization - Add 4 Numbers!

New idea: Predict class AND location!

Network outputs TWO things:

Same CNN backbone, two output heads!

What Do the 4 Numbers Mean?

Bounding box = 4 numbers:

Two common formats:

Localization: Two Losses!

We need to train BOTH outputs:

balances the two losses (usually 1-10).

Box Loss: How Far Off?

Mean Squared Error on coordinates:

Total box loss:

Localization in PyTorch

class LocalizationNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn = ...  # CNN backbone
        self.fc_class = nn.Linear(512, 3)   # 3 classes
        self.fc_box = nn.Linear(512, 4)     # 4 box coords

    def forward(self, x):
        features = self.cnn(x)
        class_out = self.fc_class(features)  # [batch, 3]
        box_out = self.fc_box(features)      # [batch, 4]
        return class_out, box_out

# Two losses!
class_loss = F.cross_entropy(class_pred, class_true)
box_loss = F.mse_loss(box_pred, box_true)
total_loss = class_loss + 5 * box_loss

Measuring Box Quality: IoU

How do we know if a predicted box is good?

IoU = Intersection over Union

IoU is used everywhere: loss functions, evaluation, NMS

The Challenge: Multiple Objects!

Localization works great for ONE object.

But what if there are 3 cats and 2 dogs?

Image with 5 objects → Neural Network → ???

Neural networks want FIXED output size!

Naive Approach: Sliding Window

Idea: Slide a window, classify each patch.

Problems:

Result: ~50 seconds per image! Way too slow.

We Need a Smarter Approach!

What we want:

• Fixed output size (neural nets like this!)
• Find ALL objects in ONE forward pass
• Fast enough for real-time (30+ FPS)

The insight:

Instead of sliding a window...
Divide the image into a grid!

YOLO's Big Idea

Divide image into S × S grid (e.g., 7×7)

Rule: If object's center is in a cell, that cell predicts it!

Why Grid? The Clever Insight

Think of it like assigning responsibility:

Each cell answers TWO questions:

1. "Is there an object centered in me?"
2. "If yes, what class and where exactly?"

What Does Each Cell Predict?

For EACH cell, predict:

Per cell: numbers

Example: 20 classes → each cell outputs 25 numbers

Understanding Confidence Score

Confidence = "Is there an object here, and how good is my box?"

High confidence = "I'm sure there's an object AND my box is accurate"

Multiple Boxes Per Cell

What if two objects have centers in the same cell?

Solution: Each cell predicts B boxes (usually B=2)

Example: B=2 boxes, C=20 classes → 30 numbers per cell

The Full YOLO Output

For an S × S grid with B boxes and C classes:

Output shape: S × S × (B × 5 + C)

Example: S=7, B=2, C=20

ONE forward pass → 1470 predictions!

YOLO Predicts EVERYTHING at Once

The full YOLO pipeline:

Output tensor: 7×7 grid × 30 numbers per cell = 1470 predictions

Key insight: Detection as a single regression problem!
One forward pass → all boxes + classes + confidences

YOLO Loss Function (3 Parts!)

Total loss = Box loss + Confidence loss + Class loss

Box Loss in Detail

Only for cells that HAVE an object:

Why and ?

• Small boxes: 2 pixel error is big deal!
• Large boxes: 2 pixel error doesn't matter much
• Square root makes errors more equal across sizes

Confidence Loss

Two cases:

— don't penalize empty cells too much!

Class Loss

Only for cells WITH an object:

Simple squared error on class probabilities.

If cell has a dog:

• Want: P(dog) = 1, P(cat) = 0, P(car) = 0
• Penalize deviation from this

Putting It All Together

YOLO Training Summary:

For each training image:
1. Divide into 7×7 grid
2. For each object, assign to cell containing its center
3. Forward pass → get 7×7×30 predictions
4. Compute all three losses
5. Backpropagate and update weights

At test time:

1. Forward pass (one!)
2. Get 7×7×30 = 1470 numbers
3. Filter low-confidence boxes
4. Non-max suppression to remove duplicates

Why YOLO is Fast

YOLO processes 45 frames per second!

Traditional: Image → Find regions → Classify each → Slow
YOLO:        Image → CNN → All detections at once → Fast!

Anchor Boxes (Brief Intuition)

Problem: Different objects have different shapes!

Solution: Pre-define "anchor boxes" of different shapes

• YOLO uses anchor boxes to predict adjustments
• Easier to predict "adjust anchor by 10%" than "raw box from scratch"

Non-Maximum Suppression (NMS)

Problem: Multiple cells may detect the same object!

Before NMS:        After NMS:
[Dog: 0.9]         [Dog: 0.9]  ← Keep highest
[Dog: 0.8]         (removed)
[Dog: 0.7]         (removed)

Algorithm:

1. Keep the detection with highest confidence
2. Remove all overlapping boxes (IoU > 0.5)
3. Repeat for remaining detections

Result: One clean box per object!

NMS Intuition: Voting with Cleanup

Think of it like election results:

Why do duplicates happen?

• Object spans multiple grid cells
• Each cell predicts its own box
• All point to roughly the same location

NMS = "Only the best detection survives!"

Using YOLO in Practice

from ultralytics import YOLO

# Load pre-trained model
model = YOLO('yolov8n.pt')

# Detect objects in an image
results = model('photo.jpg')

# Print detections
for result in results:
    for box in result.boxes:
        x1, y1, x2, y2 = box.xyxy[0].tolist()
        confidence = box.conf[0].item()
        class_id = int(box.cls[0].item())
        print(f"Found object at ({x1:.0f}, {y1:.0f}) to ({x2:.0f}, {y2:.0f})")

YOLO Model Sizes

"n" = nano (smallest), "x" = extra-large

YOLO Versions: A Quick History

Use YOLOv8 — it's the most modern and easy to use!

One-Stage vs Two-Stage Detectors

Speed vs Accuracy Trade-off

Why does this matter?

You choose based on your constraints!

IoU Calculation: Step by Step

Ground Truth box: (30, 30) to (100, 100)
Predicted box: (50, 50) to (120, 120)

IoU = 0.34 → Not a good match (need > 0.5)

Real-World Detection

Detection Challenges

Modern detectors (YOLO v8) handle most of these well!

COCO Dataset: The Benchmark

Common Objects in Context (COCO):

If your model works well on COCO, it probably works in the real world!

From Detection to Segmentation

Segmentation = Detection's precise cousin

• Self-driving needs to know exact road boundaries
• Medical imaging needs exact tumor boundaries

Training Your Own Detector

Steps to train YOLO on your data:

# 1. Collect and label images (use tools like Roboflow)
# 2. Export in YOLO format

# 3. Train
from ultralytics import YOLO
model = YOLO('yolov8n.pt')  # Start from pre-trained
model.train(data='my_data.yaml', epochs=50)

# 4. Use it!
results = model('new_image.jpg')

Transfer learning: Start from pre-trained weights, fine-tune on your data!

Summary: The Vision Pipeline

Key Takeaways

1. Images are grids of numbers (pixels)

2. CNNs use filters to detect patterns

   • Same filter works everywhere (weight sharing)
   • Build hierarchy: edges → shapes → objects

3. Object Detection = Classification + Location

   • Predict 4 coordinates for each box
   • IoU measures overlap quality

4. YOLO enables real-time detection

   • One forward pass for entire image
   • Fast enough for self-driving cars!

What We Skipped (Advanced Topics)

You'll learn these in advanced CV courses!