A set of labs, walked in order, that run real TransformerLens models behind a web page. Type a prompt, pick a model, and see how it is tokenized, where every head attends, and when the model knows its answer.
What does the model actually read?
Nine tokenizers, no model loaded: GPT-2, GPT-NeoX/Pythia, BLOOM, Qwen 2.5, Llama 2, GPT-4's cl100k, GPT-4o's o200k, BERT and XLM-R.
Where does each token look?
Per-layer, per-head attention heatmaps from a real HookedTransformer forward pass, rendered live as you type.
How does a model copy what it has seen before?
Repeated random tokens, per-head induction scores, and the loss drop in the second half of the sequence.
When does the model know the answer?
Decode the residual stream after the embedding and after every attention and MLP sub-layer, and watch the prediction form. One forward pass; every view is instant.
Which components actually matter?
Zero- or mean-ablate heads and measure the change in loss and logits.
Learning mechanistic interpretability usually means a loop of opening a Colab notebook, waiting for a model to load, editing a cell and re-running it. attnlab removes the loop. It was inspired by ARENA 3.0 chapter 1.2 (Intro to Mech Interp) and pins the same TransformerLens version, so its numbers reproduce the notebooks exactly.
Each lab answers one question and hands its text and model to the next, so the labs read as one course rather than a pile of widgets.
Small models, chosen so that every forward pass finishes in well under a second on a CPU. Adding a model is a data change, not a code change.
A React and TypeScript front end draws everything on canvas. Behind it, a FastAPI server runs TransformerLens HookedTransformer models from a memory-budgeted model cache and refuses work rather than running out of memory.
Attention patterns travel in a compact binary format (triangle-packed, square-root companded, 8-bit) instead of raw float32. For gpt2-small at 512 tokens, raw attention would be 151 MB; nobody's browser needs that.
attnlab stands on open-source work and published research. Thank you to everyone behind it.
Follow four tokens through a single transformer block from the residual stream's point of view. Every matrix is shown with real numbers, computed live, so you can trace one token's row from embedding to next-token probabilities.
Watch GPT-2 build its prediction layer by layer. A runnable walkthrough that decodes the residual stream after every block, shows why ln_final matters, and plots top-k and rank heatmaps.