Overview
Across 14 modules and a capstone you build one project: tinyformer-lab, your own configurable transformer, in which every architectural choice is a switch you can flip and measure. It is set at Aurora Station, a fictional research station on Mars whose slow link to Earth means it must run its own small models on its own CPUs. Everything runs in Google Colab on the free CPU runtime.
What you will learn
- 111 written parts, yours for good
- 43h 12m of reading, measured not estimated
- Written for the advanced level
- Every future revision included
- UPI, cards and netbanking
Prerequisites
- A Google account, to run code in Google Colab. No GPU and no installation for most lessons.
- Python and NumPy, basic calculus (the chain rule) and linear algebra (matrix products, transposes).
- Having trained at least one neural network before. If you have not, do the AI Engineering course's Module 4, Deep Learning Basics first: it covers backpropagation, optimisers, schedules and initialisation.
- Helpful but not required: the AI Engineering course's Module 5 (tokenisation and why recurrent models struggle) and Module 6 (the transformer in overview, model families, fine-tuning and inference). This course goes underneath all of them.
- A free Groq or Gemini key only for the few lessons that compare the lab's model with a large hosted one.
Curriculum
16 sections · 111 parts · 43h 12m2h 24m free to read · 40h 48m with the course
01Course guide5 parts · 66 min
- Transformers: From Attention to Trained, Served and Interpreted ModelsPreview4 min
- How This Course WorksPreview6 min
- Setup: Colab, Your Model Provider, and Running LocallyPreview15 min
- The Project: tinyformer-lab at Aurora StationPreview11 min
- Shared Lab DataPreview30 min
02Module 1: The Road to Attention6 parts · 78 min
- Module 1: The Road to AttentionPreview1 min
- Lesson 1: Six prerequisites, each checked by running codePreview16 min
- Lesson 2: Why recurrence needed replacing, measuredPreview18 min
- Lesson 3: Attention before transformers, rebuilt on one taskPreview20 min
- Lab: Start tinyformer-lab with one honest racePreview16 min
- Incident: The causal flag that masked nothingPreview7 min
03Module 2: Self-Attention Mechanics8 parts · 133 min
- Module 2: Self-Attention Mechanics1 min
- Lesson 1: Scaled dot-product attention, derived and built by hand18 min
- Lesson 2: The scaling factor, derived, measured and removed20 min
- Lesson 3: Multi-head attention, split, joined and switched off21 min
- Lesson 4: Masking, built, tested and broken on purpose19 min
- Lesson 5: Head variants and what attention really costs23 min
- Lab: Package attention and test every claim20 min
- Incident: A scale factor from the wrong width11 min
04Module 3: The Transformer Block7 parts · 151 min
- Module 3: The Transformer Block1 min
- Lesson 1: The config object and the feed-forward network27 min
- Lesson 2: The residual stream, where every layer adds its part26 min
- Lesson 3: Normalisation, and where the norm sits27 min
- Lesson 4: Embeddings, the output head and the shape of a model27 min
- Lab: One config, one block, every switch27 min
- Incident: Fine in bfloat16, NaN in float1616 min
05Module 4: Positional Information7 parts · 133 min
- Module 4: Positional Information1 min
- Lesson 1: Why attention needs positions, and the absolute fix26 min
- Lesson 2: Positions inside attention: relative, ALiBi and rotary24 min
- Lesson 3: Stretching a trained model to longer inputs25 min
- Lesson 4: Position on a grid, a graph, or nowhere20 min
- Lab: Every position method behind one switch26 min
- Incident: A rope setting the loader never read11 min
06Module 5: Building a Transformer from Scratch8 parts · 163 min
- Module 5: Building a Transformer from Scratch1 min
- Lesson 1: The whole model, traced shape by shape29 min
- Lesson 2: Two attention paths, starting weights and reference checks25 min
- Lesson 3: Numerics, what 16 bits keep and what they lose23 min
- Lesson 4: Where a training step spends its time and memory26 min
- Lesson 5: Testing a model, from cheapest check to costliest25 min
- Lab: A tested model that runs in 16 bits25 min
- Incident: Weights that never moved, and no error9 min
07Module 6: Training Dynamics7 parts · 150 min
- Module 6: Training Dynamics1 min
- Lesson 1: The training loop, and what the loss is telling you25 min
- Lesson 2: Optimisers, schedules and the step-22 spike28 min
- Lesson 3: Stable runs, and the settings that matter26 min
- Lesson 4: Watching a run, regularising it, resuming it exactly29 min
- Lab: One trainer, one recipe, exact resume25 min
- Incident: A 104B model that was really 58B16 min
08Module 7: Architecture Families and Design Choices7 parts · 139 min
- Module 7: Architecture Families and Design Choices1 min
- Lesson 1: Three shapes of transformer, one block24 min
- Lesson 2: Landmark designs, rebuilt and counted exactly18 min
- Lesson 3: The modern default stack, measured one switch at a time24 min
- Lesson 4: Mixture of experts, more weights for the same work27 min
- Lab: Presets, wirings and an expert switch, tested29 min
- Incident: A balancing loss that stopped balancing16 min
09Module 8: Pretraining7 parts · 174 min
- Module 8: Pretraining1 min
- Lesson 1: The data pipeline, and comparing objectives fairly29 min
- Lesson 2: Eight ways to learn from raw text32 min
- Lesson 3: Cleaning the corpus before it trains the model32 min
- Lesson 4: Running a pretraining job, start to finish30 min
- Lab: Corpus, tokeniser, packing and objectives, tested35 min
- Incident: A decontamination filter that skipped long documents15 min
10Module 9: Scale and Distributed Training7 parts · 156 min
- Module 9: Scale and Distributed Training1 min
- Lesson 1: Scaling laws, and what a small grid can say28 min
- Lesson 2: Spending a compute budget, then splitting the batch27 min
- Lesson 3: Sharding, tensor and pipeline parallelism25 min
- Lesson 4: Long inputs, experts, and what training costs30 min
- Lab: Hana's answer, costed and split, tested34 min
- Incident: Faulty chips that trained models quietly wrong11 min
11Module 10: Efficiency and Long Context7 parts · 163 min
- Module 10: Efficiency and Long Context1 min
- Lesson 1: What is expensive, and sparse attention27 min
- Lesson 2: Linear attention, low rank, and what they cannot learn28 min
- Lesson 3: Attention that respects memory, and the KV cache31 min
- Lesson 4: Long context, what it really buys you25 min
- Lab: Efficient attention, KV cache and long-context probes, tested35 min
- Incident: A sliding window that opened the future16 min
12Module 11: Inference and Serving10 parts · 293 min
- Module 11: Inference and Serving2 min
- Lesson 1: Prefill, decode and the decoding loop27 min
- Lesson 2: Choosing a decoder, and fencing in what it writes30 min
- Lesson 3: Speculative decoding and continuous batching, measured28 min
- Lesson 4: Paged cache memory and who runs next29 min
- Lesson 5: Quantisation: fewer bits per weight, measured29 min
- Lesson 6: Pruning and distillation: fewer weights, smaller models26 min
- Lesson 7: Serving in practice: quality, engines, adapters, cost, honest numbers32 min
- Lab: Serve base locally, one cache, tested71 min
- Incident: A guessable hash that shared another prompt's cache19 min
13Module 12: Adaptation and Alignment8 parts · 245 min
- Module 12: Adaptation and Alignment1 min
- Lesson 1: The adaptation job, its data and its evaluation28 min
- Lesson 2: Full fine-tuning and what it forgets26 min
- Lesson 3: Training a small adapter instead of the whole model27 min
- Lesson 4: Preference optimisation: teaching which card is better27 min
- Lesson 5: Reasoning steps, verifiers and the ship decision31 min
- Lab: Package the card model and Maya's check87 min
- Incident: Fine-tuned models that never learned to stop18 min
14Module 13: Beyond Text7 parts · 195 min
- Module 13: Beyond Text1 min
- Lesson 1: A vision transformer that reads camera test cards21 min
- Lesson 2: Boxes, masked patches and CLIP on test cards21 min
- Lesson 3: Connecting pixels, frames and sound to a language model29 min
- Lesson 4: Generating test cards, and transformers on tables25 min
- Lab: One image encoder for cards, captions and CLIP86 min
- Incident: The same pictures, resized two ways12 min
15Module 14: Analysis, Interpretability, and Frontiers7 parts · 218 min
- Module 14: Analysis, Interpretability, and Frontiers1 min
- Lesson 1: Instruments for looking inside a model22 min
- Lesson 2: Finding circuits and features inside a model31 min
- Lesson 3: What a model does, and what we can prove28 min
- Lesson 4: State-space models, hybrids, and what attention keeps24 min
- Lab: Instruments, circuits and a state-space layer100 min
- Incident: Accurate in every test, reading the wrong thing12 min
16Module 15: Capstone Project3 parts · 135 min
- Module 15: Capstone Project1 min
- Lab: One claim, measured, costed and written up122 min
- Incident: Two groups of one algorithm, called different12 min
Part 1 · freeTransformers: From Attention to Trained, Served and Interpreted Models · 4 min
Most explanations of transformers stop at a diagram of attention. This course goes all the way through: you derive the mechanics, build the model, train it, scale it, serve it cheaply, adapt it, and then take it apart to see what it learnt.
Across 14 modules and a capstone you build one project: tinyformer-lab, your own configurable transformer, in which every architectural choice is a switch you can flip and measure. It is set at Aurora Station, a fictional research station on Mars whose slow link to Earth means it must run its own small models on its own CPUs. Everything runs in Google Colab on the free CPU runtime.
Who this course is for
| You are... | Start at |
|---|---|
| Comfortable with Python and NumPy, have trained one neural network, new to transformers | Module 1 |
| Have done the AI Engineering course up to its Module 6 | Module 1, skimming the sections each module README marks as covered there |
| Already build transformers at work | Any module: each README's What you need first says whether it is safe to jump in |
Reviews
No reviews yet. The first comes from a reader who finishes the course.
to review this course once you have finished it.
Next to this one
Retrieval-Augmented Generation: The Complete Course95 parts · 38h 33m · ₹399Large Language Models: The Complete Course103 parts · 34h 48m · ₹399Forward Deployed Engineer: The Complete Course85 parts · 18h 15m · ₹399AI Engineering: The Complete Course105 parts · 23h 44m · ₹399Agentic AI Engineer: The Complete Course97 parts · 33h 10m · ₹399
Large Language Models80 parts · 27h 0m · Free
Model Context Protocol83 parts · 14h 44m · ₹199
RAG82 parts · 13h 36m · ₹399
Python79 parts · 7h 28m · Free