Let's build GPT: from scratch, in code, spelled out.

We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people...

Beginner - 116 min - AI & Data - Andrej Karpathy

The full course: Learn Deep Learning free

What you'll need

• GPU-ready laptop: Train small models locally.
• Laptop stand: Screen at eye level for long study sessions.
• Noise-cancelling headphones: Focus through long video lessons.

Books

1. Deep Learning with Python by Francois Chollet - View on Amazon
2. Deep Learning by Ian Goodfellow - View on Amazon

Next lessons

• Transformers, the tech behind LLMs | Deep Learning Chapter 5 (27 min, 3Blue1Brown, Intermediate)
• But what is a neural network? | Deep learning chapter 1 (19 min, 3Blue1Brown, Beginner)
• Gradient descent, how neural networks learn | Deep Learning Chapter 2 (21 min, 3Blue1Brown, Intermediate)
• Let's reproduce GPT-2 (124M) (241 min, Andrej Karpathy, Intermediate)
• The spelled-out intro to neural networks and backpropagation: building micrograd (146 min, Andrej Karpathy, Beginner)
• Attention in transformers, step-by-step | Deep Learning Chapter 6 (26 min, 3Blue1Brown, Beginner)
• How might LLMs store facts | Deep Learning Chapter 7 (23 min, 3Blue1Brown, Intermediate)
• Let's build the GPT Tokenizer (134 min, Andrej Karpathy, Intermediate)