One concrete configuration, followed from start to finish: a 1.27B-parameter model with Llama 2 7B's layer shape, trained on a single 8-GPU node split 2-way tensor × 2-way pipeline × 2-way data parallel, then post-trained and deployed.
Explanations of distributed training usually stay abstract. This one fixes a single setup and computes everything for it: 5 layers with Llama 2 7B's exact layer shape (hidden size 4,096, 32 heads, a SwiGLU MLP 11,008 wide, a 32,000-token vocabulary), 1.27 billion parameters, 2,048-token sequences and a global batch of 32,768 tokens per step.
The model is shallow but every layer is full size, so the per-layer shapes, bytes, FLOPs and communication are what you would see in a real 7B-class model. One colour scheme marks each kind of parallelism throughout.
Anyone who understands a single transformer and wants to know what actually happens when it is trained across many GPUs: which tensors are split, who talks to whom and how often, and where the memory goes. Numbers are rounded estimates for building intuition, and the reading list names the papers behind each technique.
To see the single block that all of this splits up, start with the neuron-by-neuron explainer.
Training a transformer on eight GPUs stands on open-source work and published research. Thank you to everyone behind it.
A set of labs, walked in order, that run real TransformerLens models behind a web page. Type a prompt, pick a model, and see how it is tokenized, where every head attends, and when the model knows its answer.
A tiny real transformer (d = 8, 2 heads, an 8 → 32 → 8 MLP, 4 blocks) runs on the sentence you type. Every circle holds the number it actually computed, and hovering one shows the weighted lines that feed it.
Follow four tokens through a single transformer block from the residual stream's point of view. Every matrix is shown with real numbers, computed live, so you can trace one token's row from embedding to next-token probabilities.