TechnicalPractitioner4 min readVideo
Anatomy of a distributed training step
Data load, forward pass, backward pass, NCCL AllReduce, optimizer: what really happens during a step and why high GPU utilization can hide little useful compute.
For: engineers and SREs · architectsPrerequisites: Have read chapter 1 of the course.
Picture a group of thirty-two hikers who must all meet at the top of each pass before setting off together again. The group moves at the pace of the slowest. One hiker with a sore ankle and the other thirty-one spend their time sitting at the top, waiting. From a distance they are all “out hiking”. In reality most of them are not walking.
Distributed training works the same way. It moves forward in steps: at each step, every GPU processes a batch of data, then they all pool their results before moving on to the next batch. This chapter takes one step apart and shows why a dashboard can display very busy GPUs that compute very little.
The five phases of a step
Section titled “The five phases of a step”A step chains five phases. First, data is loaded and sent to the GPU. Then comes the forward pass: data flows through the model, layer by layer, up to a prediction. The backward pass measures the error of that prediction (the loss) and goes back through the model in reverse to compute the gradients, the correction to apply to each parameter. Next is the AllReduce, the exchange in which all GPUs average their gradients, organized by NCCL, NVIDIA’s communication library. The optimizer step, finally, applies the averaged correction to every parameter.
The table below details each phase and what slows it down. A few terms: workers are side processes that prepare the data; tokenizing means cutting text into tokens (pieces of words); pinned memory is a region of CPU memory set aside so that the transfer to the GPU can happen directly (DMA, direct memory access) over the PCIe bus; AdamW is the most common optimization algorithm.
| Phase | What happens | If it is slow |
|---|---|---|
| 1. Data load and host to device | workers read from Lustre, tokenize, load into pinned CPU memory, then DMA transfer to the GPU over PCIe (roughly 16 to 64 GB/s depending on generation); must overlap with the previous batch | the workload is I/O bound |
| 2. Forward pass | data goes through the model layer by layer; each layer is mostly a matrix multiplication, tiled across Streaming Multiprocessors; activations stay in HBM for the backward pass | compute or memory saturated |
| 3. Backward pass | loss computation, then backpropagation layer by layer; each GPU computes gradients on its own data subset | same |
| 4. AllReduce (NCCL) | a ring AllReduce averages gradients across all GPUs over InfiniBand; its share of step time varies widely with model size, network and overlap with compute | network or straggler |
| 5. Optimizer step | AdamW updates all parameters in parallel; the cycle restarts on the next batch | rarely the bottleneck |
In a ring AllReduce, the GPUs are arranged in a ring and each passes a share of the gradients to its neighbour, round after round, until all of them hold the full average.
One step across four nodes and 32 GPUs
Section titled “One step across four nodes and 32 GPUs”Here is the step over time, GPU by GPU:
0 ms ~8 ms ~16 ms ~28 ms ~42 ms |-----------|--------------|-------------------|-----------------------| GPU 0 [data load ][ forward pass ][ backward pass ][ NCCL AllReduce ][opt] GPU 1 [data load ][ forward pass ][ backward pass ][ NCCL AllReduce ][opt] GPU 2 [data load ][ forward pass ][ backward pass ........ ][ AllReduce ][opt] <- straggler GPU 3 [data load ][ forward pass ][ backward pass ][ waiting... ][opt] ... ^ synchronization barrierThe AllReduce acts like the top of the pass: it is a synchronization barrier. Every GPU waits there for the slowest one to finish. Here GPU 2 takes longer than the others to complete its backward pass; it is called a straggler. A single slow GPU is enough to hold back all 32.
Why does 87% utilization not mean 87% compute?
Section titled “Why does 87% utilization not mean 87% compute?”Take an illustrative example, with values constructed for the explanation (not a measurement). The dashboard shows 87% GPU utilization. Yet the time actually spent computing is only about 19% of the step, while 43% of the step goes into the NCCL AllReduce.
The gap comes from what the indicator measures. GPU utilization tells you whether the GPU has something running, not whether it is computing usefully. A GPU waiting for a collective to finish counts as busy, just as our hikers sitting at the top count as “out hiking”. A straggler can thus cut a large share of global efficiency without triggering a single alert. The simulator below lets you put figures on scenarios.
Revised on 2 October 2026: the AllReduce share of step time is described as variable instead of an unsourced range, the 87% example is presented as constructed for the explanation rather than taken from profiling and straggler efficiency loss points to the simulator instead of an unsourced percentage.