Character-Level Language Model
Learn next-token prediction by building a model that generates names
LLM From Scratch
Build a Large Language Model from the ground up.
Seven notebooks. Pure PyTorch. No magic.
7
Parts
600K
Parameters
100%
From Scratch
The Series
01Shakespeare Text Generation
Scale up to literature with temperature sampling and creative text generation
BPE Tokenization
Implement Byte-Pair Encoding to understand subword tokenization
Self-Attention & Transformers
Build the transformer architecture: attention, multi-head, positional encoding
Instruction Tuning
Transform your base model into an instruction-following assistant with SFT
DPO Alignment
Implement Direct Preference Optimization for human preference alignment
Python Code Generation
Train a model to generate working Python code with automated testing
Get Started
git clone https://github.com/nipunbatra/llm-from-scratch.git cd llm-from-scratch pip install torch matplotlib jupyter