The Story So Far

Today: Build a mini language model from scratch!

The Shocking Truth About ChatGPT

ChatGPT, Claude, Gemini, LLaMA...

All these AI systems do ONE thing:

That's it. That's the whole trick.

You Already Use This!

All next-word prediction!

Today's Goal

We'll build a model that generates Indian names!

Same principle as ChatGPT, just smaller!

What is Next-Token Prediction?

The ONE question every language model answers:

"Given the text so far, what comes next?"

Why Prediction = Understanding

To predict well, you must understand!

Our Toy Problem: Name Generation

Goal: Generate Indian names character-by-character

Training data: A list of names

aarav
priya
nipun
kavya
rohan
zara
aditya

The Prediction Task

Given previous characters, predict the next one.

This is ALL we need to learn!

Our Vocabulary

First, define what characters we can use:

Vocabulary size = 27 (26 letters + 1 special)

Creating Training Examples

From name "aarav", create (context → target) pairs:

Wait, Context is Just 1 Character?

Problem: Looking at only the last character isn't enough!

Solution: Use a window of k previous characters!

Context Window of Size 3

Each example has 3 chars of context:

Sliding window creates training data!

Building the Dataset in Python

# Our training names
names = ["aarav", "priya", "nipun", "kavya", "rohan"]

# Create character to index mapping
chars = ['.'] + list('abcdefghijklmnopqrstuvwxyz')
char_to_idx = {c: i for i, c in enumerate(chars)}

# Context window size
context_size = 3

# Build training examples
X, Y = [], []  # X = contexts, Y = targets
for name in names:
    name = '.' * context_size + name + '.'
    for i in range(len(name) - context_size):
        context = name[i:i+context_size]
        target = name[i+context_size]
        X.append([char_to_idx[c] for c in context])
        Y.append(char_to_idx[target])

What Does Our Dataset Look Like?

For name "aarav" with context_size=3:

X = input, Y = what we want to predict

The Problem: Neural Nets Need Numbers

Neural networks work with continuous numbers, not discrete tokens!

Better idea: Learn a vector for each character!

Character Embeddings

Give each character a learned vector:

These vectors are LEARNED during training!

The Embedding Layer

import torch
import torch.nn as nn

vocab_size = 27      # 26 letters + '.'
embed_dim = 10       # Each char → 10 numbers

# Create embedding layer
embedding = nn.Embedding(vocab_size, embed_dim)

# Look up a character
char_idx = 1  # 'a'
vector = embedding(torch.tensor([char_idx]))
# vector is now a 10-dimensional vector!

Why Embeddings Are Powerful

The embedding layer is a learnable lookup table:

During training:

• Similar characters get similar vectors
• The network learns useful representations

Embeddings Learn Similarity!

After training, similar characters cluster together:

The network discovers:

• Vowels group together
• Common ending letters cluster
• Rare letters are pushed to the edges

Embedding the Context

Our context is 3 characters. Embed each one!

context = [1, 1, 18]  # "aar"

# Embed each character
emb1 = embedding(1)   # 10-dim vector for 'a'
emb2 = embedding(1)   # 10-dim vector for 'a'
emb3 = embedding(18)  # 10-dim vector for 'r'

# Concatenate them
context_emb = torch.cat([emb1, emb2, emb3])
# context_emb is now 30-dimensional!

Model Architecture

The full pipeline:

Simple MLP!

The Model in PyTorch

class NameGenerator(nn.Module):
    def __init__(self, vocab_size=27, embed_dim=10,
                 hidden_dim=100, context_size=3):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.fc1 = nn.Linear(context_size * embed_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, vocab_size)

    def forward(self, x):
        # x: (batch, context_size) - character indices
        emb = self.embedding(x)       # (batch, context, embed_dim)
        emb = emb.view(emb.size(0), -1)  # (batch, context*embed_dim)
        h = torch.tanh(self.fc1(emb))    # (batch, hidden)
        logits = self.fc2(h)              # (batch, vocab_size)
        return logits

Understanding the Forward Pass

The Output: Logits

The model outputs 27 numbers (one per character):

logits = model(context)  # Shape: (27,)
# logits = [2.1, 0.5, -1.2, 0.8, ...]

These are NOT probabilities yet!

Softmax: Logits → Probabilities

Convert logits to probabilities:

probs = torch.softmax(logits, dim=-1)
# probs = [0.45, 0.12, 0.02, 0.15, ...]
# Now they sum to 1!

The Loss Function: Cross-Entropy

How wrong was our prediction?

Lower probability for correct answer → higher loss!

Why Cross-Entropy Works

Cross-entropy punishes confident wrong predictions:

The model learns to be confident only when right!

Cross-Entropy: Worked Example

Context "aar" → True next char: 'a' (index 1)

Model predicts:

logits = [0.5, 2.1, 0.3, ...]  # 'a' has score 2.1
probs  = softmax(logits) = [0.12, 0.58, 0.08, ...]

Loss calculation:

The Training Loop

model = NameGenerator()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
criterion = nn.CrossEntropyLoss()

for epoch in range(1000):
    # Forward pass
    logits = model(X)  # X is our context tensor

    # Compute loss
    loss = criterion(logits, Y)  # Y is target tensor

    # Backward pass
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if epoch % 100 == 0:
        print(f"Epoch {epoch}: Loss = {loss.item():.4f}")

What Gets Learned?

All weights are learned from data!

Training Progress

Epoch 0:    Loss = 3.29  (random guessing)
Epoch 100:  Loss = 2.45  (learning patterns)
Epoch 500:  Loss = 1.82  (getting better)
Epoch 1000: Loss = 1.54  (good predictions!)

Loss decreases as model learns patterns in names!

Generating a Name

Algorithm:

1. Start with context "..."
2. Predict probabilities
3. Sample one character
4. Update context, repeat
5. Stop at end token "."

Generation Code

def generate_name(model, context_size=3):
    context = [0] * context_size  # Start with "..."
    name = ""

    while True:
        # Get probabilities
        logits = model(torch.tensor([context]))
        probs = torch.softmax(logits, dim=-1)

        # Sample next character
        next_idx = torch.multinomial(probs, 1).item()

        if next_idx == 0:  # End token
            break

        name += chars[next_idx]
        context = context[1:] + [next_idx]  # Slide window

    return name

Generated Names

After training on Indian names:

>>> generate_name(model)
'arya'

>>> generate_name(model)
'priti'

>>> generate_name(model)
'kavish'

>>> generate_name(model)
'neha'

The model learned patterns of Indian names!

Wait... What Did We Just Build?

Let's appreciate this moment:

You gave it names. It learned the RULES of names!

Why Sampling, Not Argmax?

If we always pick the most likely character:

generate()  →  "a"
generate()  →  "a"
generate()  →  "a"  (same every time!)

Sampling gives variety:

generate()  →  "arya"
generate()  →  "priti"
generate()  →  "neel"  (different each time!)

The Temperature Knob

Temperature controls how "peaked" the distribution is:

Temperature Examples

How T changes the distribution:

Low T = predictable, High T = surprising

Temperature in Code

def generate_with_temp(model, temperature=1.0):
    context = [0] * context_size
    name = ""

    while True:
        logits = model(torch.tensor([context]))

        # Apply temperature!
        logits = logits / temperature

        probs = torch.softmax(logits, dim=-1)
        next_idx = torch.multinomial(probs, 1).item()

        if next_idx == 0:
            break
        name += chars[next_idx]
        context = context[1:] + [next_idx]

    return name

Temperature Demo

Same model, different temperatures:

Low T = safe, High T = creative

Temperature: A Real-World Analogy

Imagine the model is choosing where to eat:

When to use what?

What We Built vs ChatGPT

Same principle. Different scale!

Key Differences in Real LLMs

Tokenization: Not Chars, Not Words

Real LLMs use subword tokens:

Tokenization Examples

Common words = 1 token, rare words = multiple tokens

The "Strawberry" Problem

"How many r's in strawberry?"

The model sees: ["str", "aw", "berry"]

It doesn't see individual letters!

Try It Yourself: OpenAI Tokenizer

https://platform.openai.com/tokenizer

Type any text and see how it gets tokenized!

What is Attention?

Problem: Long-range dependencies

"The cat sat on the mat. It was comfortable."

What does "It" refer to?

Attention: Let each position "look at" all other positions!

Attention: The Intuition

Our MLP can only look at fixed 3 characters.

What if we need to look at something 100 characters ago?

Key insight: The model LEARNS which positions to attend to!

"When predicting after 'It', pay attention to 'cat' not 'mat'"

The Transformer (2017)

"Attention Is All You Need"

This enabled GPT, BERT, and all modern LLMs!

From Prediction to Assistant

Pre-training alone gives a text completer, not an assistant.

Lecture 08 covers this journey!

Summary: The Recipe

1. Define vocabulary (chars or tokens)
2. Build training data (context → target pairs)
3. Create embeddings (tokens → vectors)
4. Build model (embed → hidden → output)
5. Train with cross-entropy (predict correctly)
6. Generate by sampling (predict → sample → repeat)
7. Control with temperature (creativity knob)

Key Takeaways

What's Next?

Lecture 08: From Language Model to Assistant