The whole email gets one label: spam or not spam.

For this task, choose among positive, negative, and neutral.

Here, the labels belong to individual words.

The output has a label for each word, rather than one label for the sentence.

The network must produce another sequence of words.

The output is shorter text that preserves the main information.

How could a network produce text of any length?

A name in our data: aabid

During training, the name tells us what came next.

During generation, the model chooses a character and continues from there.

Run the snippets in order. We use the same 27-token vocabulary throughout.

import torch
from torch import nn
from torch.nn import functional as F
vocab = ["-"] + list("abcdefghijklmnopqrstuvwxyz")
stoi = {c: j for j, c in enumerate(vocab)}

stoi maps a character to an integer: "-" → 0, "a" → 1, "b" → 2.

These snippets construct a fresh model. The numerical worksheets show a saved trained model; creating a layer does not load those trained values.

For aabid, the window a a b is the input. The next character i is the training target.

We have read a a b. How likely is i next?

Read $\mid$ as “given”: the probability of “i” next, given “a a b” already read.

ctx = torch.tensor([[stoi[c] for c in "aab"]])
target = torch.tensor([stoi["i"]])
print(ctx.shape, target.shape)

ctx has shape [1, 3]: one example, three known character IDs.

target has shape [1]: the next character ID for that example.

Integer tensors select embedding rows. The extra outer brackets keep the batch dimension, even for one example.

w = 3
padded = [stoi["-"]] * w + [stoi[c] for c in "aabid-"]
X = torch.tensor([padded[j:j+w] for j in range(6)])
y = torch.tensor(padded[w:])

X has shape [6, 3]. Each row holds one context. y has shape [6]. Each entry is that row's observed target.

The first three boundary IDs pad the opening. The final boundary is the target that teaches stopping.

In Python, j:j+w includes position j and stops before j+w.

one_hot = F.one_hot(ctx, num_classes=27)
print(one_hot.shape)  # [1, 3, 27]

Each of the three character IDs becomes a row with one 1 and twenty-six 0s.

This representation contains no learned parameters. The ID simply chooses where the 1 goes.

Could a learned representation make a useful distinction available to the next-character predictor?

One-hot IDs give each character a separate identity. We will let training choose a short row of numbers for each character.

A network can learn from one-hot inputs too. The limitation belongs to the raw one-hot geometry, not to every model that uses it.

The vector for a is a row of two coordinates. The plot shows the same two numbers as a point.

This row comes from our saved character model. Values here are rounded. Next we will ask how training chose them.

Here a and i are closer to each other than to b.

Our toy training constrains coordinate 1 to be positive for vowels and negative for consonants. This picture is not evidence that an unconstrained model discovered that rule.

The quotation marks identify a character value. They are not part of the name.

In another example, c could have the value "b" or "i".

IDs start at 0 in our code. 1 identifies a's table row, regardless of where a occurs in a name.

Three of 27 rows are shown. Each embedding contains two numbers. Character and ID just label its row.

The two blue coordinates form the embedding. The purple 1 selects them and is not part of the vector. This is selection, not multiplication.

The two saved coordinates, rounded to two decimal places.

Capital E with subscript tok is the whole table. Lowercase e with subscript "a" is one retrieved row. Reading a row leaves the table unchanged.

Read right to left: find the character's ID, look up that row in the table, and call the returned vector its embedding.

[ ] selects the row. = says the expressions on both sides name the same vector.

Start with initial coordinates. Make a prediction, measure its error, and update the table along with the predictor.

The objective is better next-character prediction. We do not directly tell every pair of characters how far apart to sit.

embedding = nn.Embedding(27, 2)
char_id = torch.tensor([1])  # id("a") = 1
e_a = embedding(char_id)
print(e_a.shape)  # [1, 2]

embedding.weight is the table $\pIn{E_{\text{tok}}}$, with 27 rows of two numbers.

The character "a" has ID 1. embedding(char_id) selects that row.

e_a holds one embedding with two numbers. The [1, 2] here describes its shape, not its coordinate values.

This code creates a fresh table. Its coordinates differ from the saved trained table shown in the worksheet.

ctx means context. Here the characters "a", "a", "b" have IDs 1, 1, 2.

ctx = torch.tensor([[1, 1, 2]])  # one example: "a", "a", "b"
e = embedding(ctx)
print(e.shape)  # [1, 3, 2]

ctx has shape [1, 3]: 1 example, 3 character IDs. The outer brackets keep the example dimension.

e has shape [1, 3, 2]: 1 example, 3 positions, 2 embedding coordinates per position.

We reuse the same embedding table. The two occurrences of ID 1 select the same row we just retrieved for "a".

An embedding is a numerical vector used to represent an object so that a model can work with it.

A representation is useful when it preserves information the task needs. Its coordinates usually have no simple English names.

King and queen can occur in similar surroundings. Across many sentences, these contexts supply a learning signal.

Word2vec trains word vectors by predicting nearby words. Similar contexts can encourage related representations.

These two sentences illustrate the idea; they are not enough to train useful word vectors.

The observed neighbour provides the target. Prediction errors update both the selected embedding row and the predictor.

This direction is called skip-gram. Its context can include words on either side, unlike left-to-right name generation.

Each point represents one word vector: man, woman, king, queen.

A hand-drawn two-dimensional illustration, not measured Word2vec vectors. Real vectors have many coordinates, usually without named meanings.

The move from man to king resembles the move from woman to queen.

In a learned space, some relationships can appear as similar vector offsets. The diagram makes that pattern exact just to show the idea.

This is the familiar king − man + woman ≈ queen example, read as a move through the vector space.

This is an analogy to test after training, not Word2vec's training rule. It is not guaranteed for every model or word pair.

man, woman, king, queen = torch.tensor(
    [[1., 1.], [3., 1.], [1., 3.], [3., 3.]])
candidate = king - man + woman
print(candidate)  # tensor([3., 3.])

The move is [0, 2]. Starting at [3, 1] reaches [3, 3]: the toy queen vector.

These are the diagram's invented coordinates. We have not trained Word2vec here; its learned vectors only approximately support some such relationships.

Ordinary Word2vec looks up the same stored bank vector in both sentences.

Its training contexts influence that row, but the lookup does not read the current sentence. Part 2 will make a representation depend on the context of this occurrence.

A search system might compare a query about borrowing money with a passage about loan rates.

It needs an encoder trained for that comparison. The query and document vectors must live in a compatible space.

doc_embedding = nn.Embedding(4, 2)
doc_ids = torch.tensor([[1, 2, 3]])  # bank lends money
doc_rows = doc_embedding(doc_ids)
e_doc = doc_rows.mean(dim=1)

One passage with three word vectors, [1, 3, 2], becomes one document vector, [1, 2].

An average is a simple baseline. It gives “dog bites man” and “man bites dog” the same vector when they use the same lookup rows.

This fresh, untrained table only shows the shapes. Learned document encoders can preserve more context and order; this mean is not a Doc2vec implementation.

The image vector represents this image. A new image can receive a vector by running the same encoder.

This is the encoder's role, not a measured model output for this scene. A vocabulary lookup alone cannot encode arbitrary new images.

pixels = torch.tensor([[[[0., 1.], [1., 0.]]]])
image_encoder = nn.Conv2d(1, 4, kernel_size=2)
e_image = image_encoder(pixels).flatten(1)
print(e_image.shape)  # [1, 4]

The input is a separate 2 × 2 grayscale toy, shaped [batch, channels, height, width]. Four learned filters produce four numbers.

These fresh filters are untrained. They do not encode the photograph above. The vision lessons explain how filters or attention learn useful image features.

A useful similarity depends on the training objective. There is no single geometry that captures every possible task equally well.

An encoder can represent a window of electricity use, movement, or temperature readings.

The window vector can feed a classifier or a forecaster. For forecasting, encode only observations available at prediction time.

For recognizing repeating behaviour, A and B share a useful pattern despite their different levels. The isolated spike is different.

These are synthetic signals. For a task about absolute consumption, their levels also matter. Training determines which distinctions the representation keeps.

signal = torch.tensor([[[0., 1., 0., -1., 0., 1.]]])
time_encoder = nn.Conv1d(1, 3, kernel_size=3)
e_time = time_encoder(signal).mean(dim=-1)
print(e_time.shape)  # [1, 3]

signal is [1, 1, 6]: one example, one channel, six readings. Three filters give [1, 3, 4]; averaging over time gives [1, 3].

This fresh encoder only demonstrates the interface. Its numbers become useful after training for a suitable objective.

We cannot compare unrelated embeddings just because both contain, say, 128 numbers.

Image–text retrieval needs training that aligns the two spaces. We will return to that idea in the CLIP lesson.

Our input is still a a b, and the observed next character is i. We learn the character table together with the MLP.

The lookup gives three rows of two numbers. Next, we place them side by side to form the predictor's input.

a0 = e.flatten(start_dim=1)
print(a0.shape)  # [1, 6]

e has three rows of two coordinates. a0 places those six numbers side by side, in position order.

start_dim=1 preserves dimension 0, the batch dimension. For 6 examples, the same operation maps [6, 3, 2] to [6, 6].

Each hidden unit can combine information from all six input coordinates.

After adding the bias, apply the same rule to every hidden unit:

Negative results become zero. Positive results stay unchanged.

before_relu = torch.tensor([-2., 0., 3.])
after_relu = torch.relu(before_relu)
print(after_relu)  # tensor([0., 0., 3.])

These three numbers illustrate the rule. Our model applies it to all 32 hidden results.

hidden = nn.Linear(6, 32)
a1 = torch.relu(hidden(a0))
print(a1.shape)  # [1, 32]

nn.Linear(6, 32) creates the learned weights and bias. a1 contains 32 hidden activations for each example.

hidden(a0) multiplies and adds the bias. torch.relu then acts on each result.

W1 = hidden.weight.T
b1 = hidden.bias
check = torch.relu(a0 @ W1 + b1)
torch.testing.assert_close(a1, check)

Our equation uses $\pParam{W_1}$ with shape [6, 32]. PyTorch stores hidden.weight with shape [32, 6].

.T swaps rows and columns. The bias has 32 entries and is added to every example. Both expressions calculate the same hidden row.

These scores become probabilities in the next step.

output = nn.Linear(32, 27)
z = output(a1)
print(z.shape)  # [1, 27]

z contains one logit for each vocabulary token. output.weight.T is $\pParam{W_2}$; output.bias is $\pParam{b_2}$.

A column index selects a possible next character. For example, z[0, stoi["i"]] is the score for “i”.

model_seq = nn.Sequential(
    embedding, nn.Flatten(1), hidden, nn.ReLU(), output
)
z_seq = model_seq(ctx)

nn.Sequential runs the same layers in order, with the same parameters.

z_seq: 27 logits per example. Softmax comes next.

class NameMLP(nn.Module):
    def __init__(self, embedding, hidden, output):
        super().__init__()
        self.embedding = embedding
        self.hidden = hidden
        self.output = output
    def forward(self, ctx):
        a0 = self.embedding(ctx).flatten(1)
        a1 = torch.relu(self.hidden(a0))
        return self.output(a1)

model = NameMLP(embedding, hidden, output)
z = model(ctx)  # [1, 27], same logits as model_seq(ctx)

__init__: store layers. forward: compute logits on each call.

For $\pIn{a_0}$, $\pAct{a_1}$, and $\pScore{z}$: rows count examples; columns count numbers per example.

Our whole window a a b is one example, so each array has one row.

With B examples, the shapes become [B, 6], [B, 32], and [B, 27]. The feature and vocabulary widths stay the same.

Weight-matrix axes count input and output features—not examples.

A bias row's leading 1 does not count examples. Its offsets are shared.

PyTorch stores biases as [32] and [27]. Broadcasting adds them to every row.

Take the first four context windows from our training array X. Each window becomes one row.

a0_batch = embedding(X[:4]).flatten(1)
a1_batch = torch.relu(hidden(a0_batch))
z_batch = output(a1_batch)
print(a0_batch.shape, a1_batch.shape, z_batch.shape)
# [4, 6], [4, 32], [4, 27]

4 rows of 6 input features. 4 rows of 32 hidden activations. 4 rows of 27 vocabulary scores.

Each row is processed independently using the same weights and biases. We changed the number of examples, not the network.

Two entries from the forward pass for a a b:

The output layer repeats this calculation for every vocabulary token, giving 27 logits. Softmax will turn them into probabilities.

Displayed values are rounded. Both sums use the full-precision products.

$\sum_{r\in\mathcal V}$ adds over the vocabulary $\mathcal V$. The index $r$ visits each token.

p = z.softmax(dim=-1)
print(p.shape, p.sum(dim=-1))  # [1, 27], one total

dim=-1 selects the vocabulary columns. Every example's probability row sums to one.

The observed next character is i. Its probability is . Next, we use that probability to measure the prediction's loss.

loss = F.cross_entropy(z, target)
print(loss.item())

z: logits shaped [1, 27]. target: the observed token ID, shaped [1].

loss is one scalar. .item() reads it as a Python number for display.

Pass logits, not p. This function combines a numerically stable log-softmax with the target's negative log-probability.

$\pProb{p}(\text{target})$ is the probability of the observed next character. Loss measures the penalty for that prediction.

$\pIn{E_{\text{tok}}}$ stores embedding rows. $\pParam{W_1,W_2}$ are weight matrices. $\pParam{b_1,b_2}$ are bias rows.

Our model is a NameMLP, which inherits parameters() from nn.Module.

for name, param in model.named_parameters():
    print(name, list(param.shape))

Assigning layers to self registers them. parameters() yields their tensors without names.

Continue with model, the NameMLP object we inspected.

params = list(model.parameters())  # the five tensors above
optimizer = torch.optim.SGD(params, lr=0.1)
model.train()

The optimizer keeps references to these same five tensors. lr=0.1 sets the learning rate.

model.train() selects training mode. optimizer.step() will change the parameters after backpropagation.

optimizer.zero_grad()
z = model(X)
loss = F.cross_entropy(z, y)
loss.backward()
optimizer.step()

zero_grad() clears old gradients. The next two lines calculate fresh predictions and their mean loss.

backward() computes gradients through the whole graph. step() uses them to change the stored parameters.

This is one illustrative update. Training the name model repeats updates over many batches from the training names.

print(model.embedding.weight.grad.shape)  # [27, 2]
print(model.hidden.weight.grad.shape)     # [32, 6]

A parameter's .grad has the same shape as that parameter. It describes how changing each entry would affect the loss.

The optimizer updates the tables and weights. The next forward pass recomputes e and a0, a1, z, and p.

Each new input uses the newest three tokens. The output name keeps all the letters chosen so far.

Keep the input − s a and the trained model fixed. Change only the temperature setting.

The highest-scoring character stays n. Greedy still chooses it. Sampling uses the changed probabilities.

Dividing by a number below 1 amplifies the differences between logits. Dividing by a number above 1 reduces those differences.

The model's weights stay fixed. We change how its scores become sampling probabilities.

@torch.no_grad()
def sample_next(ctx, temperature=1.0):
    z = model(ctx)
    p = (z / temperature).softmax(dim=-1)
    return torch.multinomial(p, num_samples=1)

multinomial samples one token ID from each probability row. For our single name, its result has shape [1, 1].

Divide z by a positive temperature before softmax. no_grad() skips the gradient graph. For greedy choice, use argmax rather than dividing by zero.

model.eval()
ctx = torch.full((1, 3), stoi["-"], dtype=torch.long)
name = []
temperature = 1.0

ctx is the opening window - - -. name will collect the chosen letters.

eval() selects evaluation behaviour. It does not disable gradients; the decorator on sample_next does that. Our small model has no dropout or batch normalization, so its forward calculation is unchanged.

for _ in range(18):
    next_id = sample_next(ctx, temperature)
    if next_id.item() == stoi["-"]: break
    name.append(vocab[next_id.item()])
    ctx = torch.cat([ctx[:, 1:], next_id], dim=1)

ctx = torch.tensor([[stoi["-"], stoi["s"], stoi["a"]]])
with torch.no_grad(): p = model(ctx).softmax(dim=-1)
greedy_id = p.argmax(dim=-1, keepdim=True)
sampled_id = torch.multinomial(p, num_samples=1)

argmax takes the largest entry. multinomial draws from all 27 entries. Both return one ID, shape [1, 1].

The full name aabid is in the data. We can prepare every input and target before running the model.

A wrong prediction for a a b does not change the next training input a b i. The i comes from the data.

batch_z = model(X)
batch_loss = F.cross_entropy(batch_z, y)

X holds the six windows we just saw: [6, 3] means 6 examples, 3 token IDs each. The output [6, 27] has 27 scores for each example.

y is [6]: one observed target ID per example. Cross-entropy pairs each score row with its target and averages the six losses.

Computing a loss alone does not update the model. That still requires backward() and optimizer.step().

No next letter is supplied. Continue from aab by sampling from the frozen model.

Within one name, each step waits for the previous choice. We can still generate several names in a batch.

text = "deep learning is amazing"
char_tokens = list(text)
word_tokens = text.split()

These Python operations choose the units first. Character tokens include spaces; split() produces four word tokens here.

Next, a vocabulary maps tokens to IDs, and torch.tensor stores those IDs. A trained subword tokenizer replaces this simple splitting step when we want subword tokens.

None of these whole words is in our small, fixed word vocabulary.

All four use the same <UNK> ID. The model cannot recover which spelling appeared from that ID alone.

The same words, split into letters. All their letters are in a–z.

The spellings remain distinct. We need an entry for each letter, not each whole word.

This works because every letter is covered. An unsupported character would still need handling.

The same words, using our demo's fixed merge rules and base letters a–z.

unbelievable uses three pieces. nanobotany stays as letters because none of our rules merges them.

These illustrative merges preserve each spelling. Subwords still need a vocabulary that covers the input.

word_vocab = ["<unk>", "The", "cat", "sat", "on", "the"]
word_stoi = {t: j for j, t in enumerate(word_vocab)}
word_ctx = torch.tensor([[2, 3, 4]])  # cat sat on
word_embedding = nn.Embedding(len(word_vocab), 2)
word_e = word_embedding(word_ctx)

word_e is still [1, 3, 2]: one example, three tokens, two coordinates.

Keep the latest three words of “The cat sat on” to predict what follows. This is a separate word vocabulary and a fresh table, which must be trained on word sequences.

word_hidden = nn.Linear(6, 32)
word_output = nn.Linear(32, len(word_vocab))
word_a1 = torch.relu(word_hidden(word_e.flatten(1)))
word_z = word_output(word_a1)

word_z has shape [1, 6], matching this six-word vocabulary.

The operations stay the same. The learned parameters belong to this new word model; they are not the trained character model's weights.

Shapes show one example. Hidden width stays 32, vocabulary size stays 27.

Larger w changes W₁. Larger d changes $E_{\mathrm{tok}}$ and W₁. Token positions share the embedding table.

new_w, new_d = 5, 4
size_model = nn.Sequential(
    nn.Embedding(27, new_d), nn.Flatten(1),
    nn.Linear(new_w * new_d, 32), nn.ReLU(), nn.Linear(32, 27))
print(sum(p.numel() for p in size_model.parameters()))  # 1671

5 embeddings × 4 numbers = 20 inputs per example. The table is [27, 4]. W₁ is [20, 32].

PyTorch stores the first linear weight as [32, 20], the transpose of our W₁. Each linear layer includes a bias.

This example creates a new, untrained model. The earlier trained model still uses w = 3 and d = 2.

w = 100 token slots; d = 256 numbers per embedding.
Hidden width $d_h$ = 1,024 units.

That is about 26 million weights in W₁ alone. Its 1,024 biases and the other parameters are extra.

The model still expects exactly 100 input slots.

e = embedding(ctx)
a0 = e.flatten(1)
a1 = torch.relu(hidden(a0))
z = output(a1)
p = z.softmax(dim=-1)

Lookup and concatenate. Calculate the hidden activations. Produce vocabulary scores. Normalize them into probabilities.

During training, send z and an observed target to cross-entropy. During generation, sample from p and form the next context.

Download the code in lecture order

Next: Part 2, Self-attention