TechnicalPractitioner3 min read (+1 technical)
How do GPUs train a model together?
Eight GPUs, five phases, a ring-shaped pooling of results and a barrier: slow one GPU down and watch the others wait while reported utilization stays high.
For: engineers and SREs · architectsPrerequisites: Basic notions of training models on GPUs.
An illustrative scenario: a dashboard shows 90% utilization on the eight GPUs of a server. Everyone is reassured. Yet training is crawling and the compute bill keeps rising. What the dashboard does not show is that seven cards out of eight spend much of their time waiting for the eighth.
To see why, picture eight teachers sharing a stack of exam papers. Each one grades their own pile (for a GPU, that is computing the gradients, the corrections to apply to the model). Then they all meet to align their marks and update the shared grading scheme: that is the AllReduce, the average of every GPU’s gradients. The meeting cannot start until the last teacher is done. That is the barrier. If one of them is slower, the other seven wait with their pens down, yet the monitoring tool still counts them as busy.
This method is called data parallelism: each GPU handles its share of the batch, then they all pool their results. The animation plays one training step on 8 GPUs, phase by phase. The phases are detailed in the chapter Anatomy of a distributed training step.
Parameters
The 8 GPUs and the ring AllReduce
Step timeline, GPU by GPU
What the step costs
Illustrative durations: loading 6 ms, forward 10 ms, backward 16 ms, AllReduce 10 ms, optimizer 2 ms; the slowdown applies to the straggler's forward and backward passes. Simplification: phases are sequential; in practice PyTorch DDP overlaps part of the AllReduce with the backward pass (gradient buckets), which shrinks the wait without removing it.
Try this
Section titled “Try this”- Watch a step without a straggler. During the pooling, the GPUs are arranged in a ring (the ring AllReduce): each one sends an eighth of its gradients to its neighbor, fourteen times in a row. The first seven exchanges add up contributions (reduce-scatter: each GPU’s small cells fill up). The next seven hand back the already averaged chunks (all-gather). Use “Next step” to follow the exchanges one at a time.
- Slow GPU 2 down to 1.5×. The other seven finish their backward pass, the phase where they compute their corrections, then wait, hatched, at the barrier. Step time goes from 44 to 57 ms and efficiency drops to about 77%. Yet reported GPU utilization goes up: waiting counts as busy.
- Compare the two gauges. At 2×, reported utilization exceeds 90% while useful compute falls below 25%. That is the gap in the illustrative example of the reference chapter: 87% reported for about 19% of real compute.
- Turn on prefetching, which prepares the next step’s data during compute. Loading overlaps with compute and almost vanishes from the timeline: the step gets shorter while useful compute improves. With a 2× straggler, the gain is still 5 ms, whereas the straggler costs 26. You do not fix GPUs waiting for each other by reading data faster.
What the animation simplifies
Section titled “What the animation simplifies”Durations are orders of magnitude chosen for readability, with phases running strictly one after another. In a real training run, part of the pooling starts before compute is over, which hides some of the waiting. The principle does not change: the barrier aligns everyone on the slowest.
For technical teams: what really happens
In practice, PyTorch DDP launches the AllReduce in gradient buckets during the backward pass, hiding part of the communication, while NCCL splits each send into finer chunks. A ring AllReduce moves about 2 × (N - 1) / N times the gradient size per GPU.
What to measure for real
Section titled “What to measure for real”The right indicator is not GPU utilization but the duration of each step, measured for each GPU (each rank). An alert on the gap between the slowest GPU and the median catches a straggler far better than an average does. To know whether a card is really computing, you need counters finer than utilization. To know where a step’s time goes, you break it down into loading, compute and communication, on a few steps only. Finally, if the network is to blame, its counters and the communication library’s logs will tell you.
For technical teams: metrics and tools
| Question | What to measure | With which tool |
|---|---|---|
| Is the step slowing down? | step time, per rank and median | training loop instrumentation (timestamp per step and per rank), exported to a time series database (VictoriaMetrics, Prometheus or a SaaS equivalent) |
| Is there a straggler? | gap between the slowest rank and the median | same metric, alert on the gap rather than the average |
| Is the GPU really computing? | SM activity, occupancy, tensor core usage | DCGM profiling metrics: DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, through dcgm-exporter |
| Where does step time go? | breakdown into loading, compute, communication | PyTorch Profiler or Nsight Systems on a few steps, not continuously |
| Is the network to blame? | collective latency, retransmissions and port waits on InfiniBand | NCCL logs (NCCL_DEBUG=INFO), InfiniBand port counters |
DCGM_FI_DEV_GPU_UTIL, the equivalent of the utilization column in nvidia-smi, only measures the share of time a kernel is running: useful to spot an idle GPU, misleading for judging efficiency.
The full monitoring platform is described in The HPC AI observability stack and the straggler effect simulator puts a cost on it at cluster scale.