The video loads from YouTube only after you press Play.
Let's build the GPT Tokenizer
A 2.2-hour lecture by Andrej Karpathy about tokenization, byte pair encoding and GPT, on the AI shelf. It was posted to r/LocalLLaMA as “Karpathy: Let's build the GPT Tokenizer [new video]”.
Andrej Karpathy, Lecture, 2 h 14 min, 2024, on the AI shelf
A code-oriented tutorial building a GPT tokenizer and explaining language-model tokenization.







