CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 13: Beyond Text · part 100 of 111
Part 100 · Module 13: Beyond Text

Lab: One image encoder for cards, captions and CLIP

86 min read·13 Oct 2026
Module 13 · Lab · About 70 minutes · Needs: lessons 1 to 4 of this module, and the Module 11 lab

What you are building

TINYFORMER-LAB AFTER MODULE 13 vision.py: cards, SPLIT = 1600, patchify VisionEncoder 65 slots out ViT reads slot 0 CLIP reads slot 0 Captioner reads slots 1 to 64 v0.11.0 package unchanged; Tubelet and sound patches added test_m13_vision 8 tests m13_vision run classify, CLIP, caption git tag v0.13.0
One encoder (blue) feeds three users: the classifier and CLIP read the class token, the captioner the patch features. Eight tests (green) pin it; earlier files (grey) stay.

Sol 447 at Aurora Station. Isla Duarte's camera test cards went through four lessons, which grew three image encoders: Lesson 1's (4 by 4 patches, 107,651 weights), Lesson 2's (8 by 8, 112,768) and Lesson 3's (nn.TransformerEncoder, 107,328, no class token). Lessons 2 and 3 promised the lab would reconcile them.

Today they become tinyformer/vision.py: the cards and one split, patches, one VisionEncoder that the classifier, CLIP and the captioner all read, and Lesson 3's video and sound sketches. Eight tests, one experiment, tag v0.13.0.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside