Setup: Colab, Your Model Provider, and Running Locally

This takes about ten minutes. You do it once.

Most lessons in this course need no provider and no key. They train and take apart small models with PyTorch, and use only the torch setup cell in section 6. A few lessons compare the lab's own model with a large hosted one, mainly Module 12, Lessons 1 and 5, where you ask a hosted model to fill in Maya's log cards before you fine-tune your own. Those lessons start with the provider setup cell in section 4, and say so at the top. If you only want the mechanism lessons, skip to section 6.

1. Choose a provider

The course works with three ways of reaching a language model. You only need one. You switch by changing one word in the setup cell.

If you are using Colab, choose Groq or Gemini. They need a free key and nothing else.

If you want Ollama, work on your own computer in an editor such as VS Code. Ollama runs the model on your machine, so no key is needed and nothing you type leaves your computer. Section 5 shows the local setup.

Free plans limit how many requests you can send per minute and per day. If you hit a limit you get an error with the number 429. Wait a minute and try again, or switch provider.

Model names change. The names below were checked against each provider's documentation in October 2026. If a call fails with "model not found", open the provider's model list (links below) and put a current name into the setup cell. Nothing else in the lesson needs to change.

2. Get a key (Groq or Gemini)

Groq

1. Go to console.groq.com and sign in.
2. Open API Keys and create a key.
3. Copy it. You will not be shown it again.

Model list: console.groq.com/docs/models

Google Gemini

1. Go to aistudio.google.com and sign in.
2. Choose Get API key and create a key.
3. Copy it.

Model list: ai.google.dev/gemini-api/docs/models

3. Store the key safely in Colab

Never paste a key into a code cell. Notebooks get shared, and a key in a shared notebook is a key someone else can spend.

1. Open colab.research.google.com and create a new notebook.
2. In the left sidebar, click the key icon (Secrets).
3. Add a secret. The name must be exactly GROQ_API_KEY (for Groq) or GOOGLE_API_KEY (for Gemini). The value is your key.
4. Turn on Notebook access for it.

The secret is remembered across notebooks. For each new notebook you only switch access on.

4. The setup cell

Every lesson that calls a model begins with these two cells. Paste them, set PROVIDER, and run.

The course uses LangChain, a widely used Python library for building applications on language models. One of the things it does is give every provider the same interface, so the rest of your code does not care which one you chose. Later modules also use LangGraph, its companion library for building multi-step workflows and agents.

%pip install -q -U langchain langgraph langchain-groq langchain-google-genai langchain-ollama

import os
from langchain.chat_models import init_chat_model

PROVIDER = "groq"   # "groq" or "gemini" in Colab; "ollama" on your own computer
MODELS = {"groq": "groq:openai/gpt-oss-20b", "gemini": "google_genai:gemini-3.5-flash-lite", "ollama": "ollama:llama3.2:3b"}
KEY = {"groq": "GROQ_API_KEY", "gemini": "GOOGLE_API_KEY", "ollama": None}[PROVIDER]
if KEY and KEY not in os.environ:
    from google.colab import userdata   # keys live in Colab's Secrets panel
    os.environ[KEY] = userdata.get(KEY)
llm = init_chat_model(MODELS[PROVIDER], temperature=0, max_retries=6)

print(llm.invoke("Reply with the single word: ready").text)

If it prints ready (or something close), you are set up.

Code walkthrough

• %pip install ... installs the libraries once per notebook. langchain and langgraph are the frameworks; the three langchain-... packages are the connectors for each provider.
• PROVIDER = "groq" is the one word you change to switch provider.
• MODELS names the model for each provider as provider:model-name. This is the only place model names appear, so when a name changes you edit one line.
• KEY is the name of the secret the provider needs. Ollama needs none.
• The if block runs only in Colab when the key is not already set: it copies your key from the Secrets panel into an environment variable, which is where LangChain looks for it.
• init_chat_model(...) creates the model object, llm. temperature=0 asks for the most consistent output. max_retries=6 makes the client wait and try again when the provider says "too many requests" (a 429); Groq's free plan allows 8,000 tokens a minute, which a batch of log cards can pass. It does not retry a reply the provider rejected as malformed: see "When the provider rejects a reply" below.
• llm.invoke(...) sends one message and returns a reply; .text is the reply as a string.

Why one cell works for three providers

init_chat_model reads the part before the colon (groq, google_genai, ollama) and builds the right connector. All connectors offer the same methods: .invoke() to get a reply, .stream() to get it piece by piece, and so on. Your code talks to llm and never mentions the provider again.

This is also how real teams avoid being locked to one provider: the provider is a setting, not something written all through the code.

When the provider rejects a reply

Sometimes a provider refuses the model's own reply with an error instead of returning it. In the course's test runs on 12 October 2026 (in the sister RAG course), Groq did this a few times with openai/gpt-oss-20b: an answer that should have been structured came back empty, or the model called a tool it was never given. max_retries does not help, because the request itself was fine. The course handles it the same way everywhere: try once more, and if that also fails, say so plainly instead of crashing.

def ask(model, messages):
    """Returns (reply, None), or (None, reason) after two refusals."""
    for attempt in range(2):                     # one retry: the next reply is usually fine
        try:
            return model.invoke(messages), None
        except Exception as err:
            last = err
    return None, f"provider_error: {type(last).__name__}"

Code walkthrough

• ask sends the messages and returns the reply with no reason, or no reply with a reason.
• On an error it tries once more; after a second error it gives up and names the error, for example provider_error: BadRequestError.

Lessons that ask a hosted model to fill in Maya's log cards use this shape: a log that gets no answer is scored as an empty card with the reason provider_error, and the loop moves on to the next log, so one bad reply never stops a notebook.

Prices and limits (checked 9 October 2026)

This is the course's one table of model prices and limits. Lessons that count cost say "prices from course-guide/02" and use these numbers; when a provider changes them, this table changes and the lessons follow. Prices are US dollars per 1,000,000 tokens on each provider's paid plan.

Sources: Groq, "Supported models" (https://console.groq.com/docs/models) and "Rate limits" (https://console.groq.com/docs/rate-limits); Google, "Gemini Developer API pricing" (https://ai.google.dev/gemini-api/docs/pricing). Free plans change often and your own limits are on each provider's console page; these are the most likely numbers in this course to go out of date.

5. Running on your own computer (required for Ollama)

You can run every lesson locally with any provider. For Ollama it is the way to do it.

Install

1. Python 3.10 or newer from python.org. Check with python --version (on some systems python3 --version).
2. A code editor. VS Code with the Python and Jupyter extensions is a good choice. Cursor, PyCharm and others work the same way.
3. Ollama, if you are using it, from ollama.com/download. After installing, download the course's default model once, in a terminal:

ollama pull llama3.2:3b

That is a download of about 2 GB. The model runs acceptably on a recent laptop with 8 GB of memory or more. A small local model makes more mistakes than the cloud models. That is useful for this course: several "break it" steps are easier to see on a small model.

Create a project folder

In a terminal:

mkdir transformers-course
cd transformers-course
python -m venv .venv

Switch the virtual environment on:

# macOS and Linux
source .venv/bin/activate

# Windows (PowerShell)
.venv\Scripts\Activate.ps1

Then install the libraries:

pip install -U langchain langgraph langchain-groq langchain-google-genai langchain-ollama pydantic python-dotenv ipykernel

Keys on your own computer

Ollama needs no key. For Groq or Gemini, create a file named .env in the project folder:

GROQ_API_KEY=paste-your-key-here
GOOGLE_API_KEY=paste-your-key-here

and add these two lines at the very top of the setup cell:

from dotenv import load_dotenv
load_dotenv()

Never share the .env file or commit it to Git.

Run the lessons

Open the folder in your editor and create a new notebook file, for example module-02-lesson-1.ipynb. Select the .venv Python when asked. A notebook in VS Code works like Colab: paste each code block into a cell and run it.

Set PROVIDER = "ollama" in the setup cell. You can skip the %pip install cell, because you installed the libraries already.

On a Mac: FAISS and PyTorch together

On macOS, loading FAISS and PyTorch in the same Python process can crash with an OpenMP error, because each brings its own copy of the OpenMP library. If you see it, set two environment variables in the terminal before you start Python, Jupyter or your editor:

export KMP_DUPLICATE_LIB_OK=TRUE
export OMP_NUM_THREADS=1

The first tells OpenMP to accept the second copy; it is a workaround, not a fix. The second keeps OpenMP to one thread, which is slower on large indexes but fine for the small ones in this course. This course does not use FAISS; you only need this if you load it next to PyTorch yourself. You do not need this in Colab.

From notebook cells to a project

In a job you would not keep code in notebook cells. The course grows one repository, tinyformer-lab, around a single package called tinyformer/. Module 1's lab starts it: the shared data blocks become tinyformer/data.py and a tests/ folder holds the checks as pytest tests. Modules 2 to 5 add attention, the block, positions and the finished model; every later module adds its own file inside the same package, and Module 15's capstone collects everything. Labs that must run in a temporary folder for testing create tinyformer/ there with the same layout and say so. In Colab, a lab saves each file with %%writefile tinyformer/<name>.py; Colab's disk is wiped when the runtime stops, so push the folder to your own GitHub repository at the end of each lab. The project page (course-guide/03) lists which module adds which file.

6. The torch setup cell (most lessons)

Every lesson that trains or inspects a model, and does not call a hosted one, starts with this cell instead of the one in section 4.

import torch
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"   # every lesson runs on "cpu"; a GPU is optional
torch.manual_seed(11)                                     # the course seed (see course-guide/06)
print("torch", torch.__version__, "| device", DEVICE, "| CPU threads", torch.get_num_threads())

Code walkthrough

• import torch loads PyTorch. Colab has it already, so there is nothing to install.
• DEVICE is "cuda" if a GPU is attached and "cpu" otherwise. Lessons move tensors with .to(DEVICE) only where a lesson says a GPU is useful; everything also runs on the CPU.
• torch.manual_seed(11) sets the course seed, so random weights and batches match the course's.
• The print shows your versions and how many CPU threads PyTorch uses. The course was tested with torch 2.14.1; on the author's laptop the cell printed torch 2.14.1 | device cpu | CPU threads 5.

Which PyTorch version

Colab ships its own build of PyTorch, and Google updates it on its own schedule. The course's code was run with torch 2.14.1, the newest release on pypi.org on 9 October 2026, and uses only long-standing parts of PyTorch. If a lesson's numbers differ from the course's in more than the last digit, or a function is missing, install the tested version and restart:

%pip install -q torch==2.14.1

Then choose Runtime > Restart session and run the lesson again. This is a large download (the Linux wheel alone is 555 MB), so do it only if you need to.

CPU first, GPU optional

Every lesson runs on Colab's free CPU runtime. Most lesson training runs are sized to finish in a few minutes there; the shared data page gives the timings measured on the author's laptop. Some Module 12 runs take longer (a LoRA run at the 448-character card window takes 3 to 4 minutes), and the lesson says so before the cell.

Timings in lessons are from the author's Mac with one CPU thread (OMP_NUM_THREADS=1, the course standard). Colab, or your own machine with more threads, will differ, sometimes by a factor of two. A few lessons say "optional GPU": a GPU makes the measurement more interesting (for example, attention kernels in Module 10) but is not needed. To attach one in Colab, choose Runtime > Change runtime type and pick a GPU. Free GPU time is limited and not guaranteed.

The reference models from Hugging Face

The old version of this course shipped a pretrained model. This one does not: lessons that need the lab's own model train it from the shared data in under a minute (course-guide/06, block 19). Lessons that need a real, published model to inspect load one of two small open models from Hugging Face. Both were checked on 9 October 2026; neither needs a key or an account.

The parameter counts and shapes above were measured by loading each model with transformers 5.19.0. A lesson loads the reference model like this:

from transformers import AutoModelForCausalLM, AutoTokenizer

REF = "openai-community/gpt2"          # the course's reference model: GPT-2 small, MIT licence
tok = AutoTokenizer.from_pretrained(REF)
ref = AutoModelForCausalLM.from_pretrained(REF, attn_implementation="eager").eval()
ids = tok("The oxygen recycler on the station", return_tensors="pt")
with torch.no_grad():
    out = ref(**ids, output_attentions=True)
print(f"{sum(p.numel() for p in ref.parameters()):,} parameters,", len(out.attentions), "layers,",
      tuple(out.attentions[0].shape), "attention shape")
print(repr(tok.decode(out.logits[0, -1].argmax())))

Code walkthrough

• AutoTokenizer and AutoModelForCausalLM read the model's name, download its files once (Colab keeps them until the runtime stops) and build the right classes.
• attn_implementation="eager" asks for the plain attention code. By default transformers 5 uses a fused kernel that does not return attention weights, and output_attentions=True then gives an empty result. Any lesson that reads attention patterns needs this setting.
• .eval() switches off training-only behaviour such as dropout.
• tok(...) turns the sentence into token ids; ref(**ids, output_attentions=True) runs the model and keeps every layer's attention weights.
• Output on the author's laptop: 124,439,808 parameters, 12 layers, (1, 12, 7, 7) attention shape: one sentence, 12 heads, 7 tokens attending to 7 tokens. The most likely next token was "'s".

In Colab you may see a warning that no HF_TOKEN secret is set. These models are public; the warning can be ignored.

Extra libraries, by module (checked 9 October 2026)

The torch setup cell covers most lessons. Some lessons need one more library. Those lessons install it in their own cell, just before first use, with the version pinned so that your run matches the course's. Every version below was the newest on pypi.org on 9 October 2026.

A lesson writes the install as, for example:

%pip install -q transformers==5.19.0

The course does not use trl. Module 12 describes its DPOTrainer for direct preference optimisation but builds the loss by hand and does not run the library, so there is nothing to install for it.

If pip reports a version conflict in Colab, keep the pinned library and let pip choose its dependencies. On your own computer, add the libraries you need to the pip install line in section 5, plus torch==2.14.1.

Things that need more than one process

Module 9 runs two processes that talk to each other, the smallest real case of distributed training. A notebook cell cannot start them directly, so the lesson saves a script with %%writefile and launches it with !torchrun --nproc_per_node=2 script.py, using PyTorch's CPU backend (gloo). On the author's Mac this worked only when bound to the loopback network interface (GLOO_SOCKET_IFNAME=lo0, --master_addr=127.0.0.1); on Linux and in Colab the interface is called lo. Module 9 gives the exact command it was tested with.

7. When something goes wrong