Skip to content

TechnicalPractitioner3 min read (+1 technical)

How do GPUs train a model together?

Eight GPUs, five phases, a ring-shaped pooling of results and a barrier: slow one GPU down and watch the others wait while reported utilization stays high.

For: engineers and SREs · architectsPrerequisites: Basic notions of training models on GPUs.

Reading mode

An illustrative scenario: a dashboard shows 90% utilization on the eight GPUs of a server. Everyone is reassured. Yet training is crawling and the compute bill keeps rising. What the dashboard does not show is that seven cards out of eight spend much of their time waiting for the eighth.

To see why, picture eight teachers sharing a stack of exam papers. Each one grades their own pile (for a GPU, that is computing the gradients, the corrections to apply to the model). Then they all meet to align their marks and update the shared grading scheme: that is the AllReduce, the average of every GPU’s gradients. The meeting cannot start until the last teacher is done. That is the barrier. If one of them is slower, the other seven wait with their pens down, yet the monitoring tool still counts them as busy.

This method is called data parallelism: each GPU handles its share of the batch, then they all pool their results. The animation plays one training step on 8 GPUs, phase by phase. The phases are detailed in the chapter Anatomy of a distributed training step.

Parameters

loadingforwardbackwardwaitingAllReduceoptimizer

The 8 GPUs and the ring AllReduce

Step timeline, GPU by GPU

What the step costs

Step duration
ideal without straggler
Scaling efficiency
ideal duration / actual duration
Waiting at the barrier
total over the 8 GPUs, per step
AllReduce share
of step time
Reported GPU utilization
GPU "busy", including while waiting and communicating (DCGM_FI_DEV_GPU_UTIL, nvidia-smi)
Useful compute
nominal compute × active SM share, divided by step time (close to DCGM_FI_PROF_SM_ACTIVE)

Illustrative durations: loading 6 ms, forward 10 ms, backward 16 ms, AllReduce 10 ms, optimizer 2 ms; the slowdown applies to the straggler's forward and backward passes. Simplification: phases are sequential; in practice PyTorch DDP overlaps part of the AllReduce with the backward pass (gradient buckets), which shrinks the wait without removing it.

  1. Watch a step without a straggler. During the pooling, the GPUs are arranged in a ring (the ring AllReduce): each one sends an eighth of its gradients to its neighbor, fourteen times in a row. The first seven exchanges add up contributions (reduce-scatter: each GPU’s small cells fill up). The next seven hand back the already averaged chunks (all-gather). Use “Next step” to follow the exchanges one at a time.
  2. Slow GPU 2 down to 1.5×. The other seven finish their backward pass, the phase where they compute their corrections, then wait, hatched, at the barrier. Step time goes from 44 to 57 ms and efficiency drops to about 77%. Yet reported GPU utilization goes up: waiting counts as busy.
  3. Compare the two gauges. At 2×, reported utilization exceeds 90% while useful compute falls below 25%. That is the gap in the illustrative example of the reference chapter: 87% reported for about 19% of real compute.
  4. Turn on prefetching, which prepares the next step’s data during compute. Loading overlaps with compute and almost vanishes from the timeline: the step gets shorter while useful compute improves. With a 2× straggler, the gain is still 5 ms, whereas the straggler costs 26. You do not fix GPUs waiting for each other by reading data faster.

Durations are orders of magnitude chosen for readability, with phases running strictly one after another. In a real training run, part of the pooling starts before compute is over, which hides some of the waiting. The principle does not change: the barrier aligns everyone on the slowest.

For technical teams: what really happens

In practice, PyTorch DDP launches the AllReduce in gradient buckets during the backward pass, hiding part of the communication, while NCCL splits each send into finer chunks. A ring AllReduce moves about 2 × (N - 1) / N times the gradient size per GPU.

The right indicator is not GPU utilization but the duration of each step, measured for each GPU (each rank). An alert on the gap between the slowest GPU and the median catches a straggler far better than an average does. To know whether a card is really computing, you need counters finer than utilization. To know where a step’s time goes, you break it down into loading, compute and communication, on a few steps only. Finally, if the network is to blame, its counters and the communication library’s logs will tell you.

For technical teams: metrics and tools
QuestionWhat to measureWith which tool
Is the step slowing down?step time, per rank and mediantraining loop instrumentation (timestamp per step and per rank), exported to a time series database (VictoriaMetrics, Prometheus or a SaaS equivalent)
Is there a straggler?gap between the slowest rank and the mediansame metric, alert on the gap rather than the average
Is the GPU really computing?SM activity, occupancy, tensor core usageDCGM profiling metrics: DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, through dcgm-exporter
Where does step time go?breakdown into loading, compute, communicationPyTorch Profiler or Nsight Systems on a few steps, not continuously
Is the network to blame?collective latency, retransmissions and port waits on InfiniBandNCCL logs (NCCL_DEBUG=INFO), InfiniBand port counters

DCGM_FI_DEV_GPU_UTIL, the equivalent of the utilization column in nvidia-smi, only measures the share of time a kernel is running: useful to spot an idle GPU, misleading for judging efficiency.

The full monitoring platform is described in The HPC AI observability stack and the straggler effect simulator puts a cost on it at cluster scale.