Attention in transformers, step-by-step | Deep Learning Chapter 6

Demystifying attention, the key mechanism inside transformers and LLMs. An equally valuable form of support is to simply share the videos. Demystifying self-attention, multiple heads, and cross-attention. Here are a few other relevant resources Build a GPT from scratch, by Andrej Karpathy If you...

Beginner - 26 min - AI & Data - 3Blue1Brown

The full course: Learn Deep Learning free

What you'll need

• GPU-ready laptop: Train small models locally.
• Laptop stand: Screen at eye level for long study sessions.
• Noise-cancelling headphones: Focus through long video lessons.

Books

1. Deep Learning with Python by Francois Chollet - View on Amazon
2. Deep Learning by Ian Goodfellow - View on Amazon

Next lessons

• Let's build GPT: from scratch, in code, spelled out. (116 min, Andrej Karpathy, Beginner)
• Transformers, the tech behind LLMs | Deep Learning Chapter 5 (27 min, 3Blue1Brown, Intermediate)
• But what is a neural network? | Deep learning chapter 1 (19 min, 3Blue1Brown, Beginner)
• Gradient descent, how neural networks learn | Deep Learning Chapter 2 (21 min, 3Blue1Brown, Intermediate)
• Let's reproduce GPT-2 (124M) (241 min, Andrej Karpathy, Intermediate)
• The spelled-out intro to neural networks and backpropagation: building micrograd (146 min, Andrej Karpathy, Beginner)
• How might LLMs store facts | Deep Learning Chapter 7 (23 min, 3Blue1Brown, Intermediate)
• Let's build the GPT Tokenizer (134 min, Andrej Karpathy, Intermediate)