TechnicalPractitioner3 min read
Why does one slow GPU slow down the whole training run?
A synchronous training step lasts as long as its slowest GPU: measure the lost efficiency, the wasted GPU-hours and the gap between reported utilization and useful compute.
For: engineers and SREs · architectsPrerequisites: Basic notions of distributed training on GPUs. The diagram "GPUs working together" is enough to get started.
An illustrative scenario: a team rents 64 GPUs for a month to train a model. One of the cards runs a little hot and works at 70% of its speed. Nobody notices: the dashboard shows every card busy. By the end of the month, training is behind schedule and part of the bill has paid for waiting.
Think of a group hike. At every break, everyone waits for the last walker before setting off again. If one hiker has a sore knee, the whole group moves at their pace, even if everyone else is in great shape. Distributed training works the same way. Each GPU computes its share of the work, then they all meet at a rendezvous point, the barrier, to pool their results (an operation called the AllReduce). The step can only finish once the slowest has finished. A single degraded GPU, called a straggler, sets the pace for everyone else. And while they wait for it, the GPUs are reported as “busy”.
The simulator helps settle one question: how much does this delay cost you, at your scale? It expresses the loss in GPU-hours, the billing unit for compute (one GPU for one hour). Prices change fast and vary by contract: this guide gives none; use your provider’s dated price list. Enter your GPU-hour price, or your internal cost, to see the loss in euros.
Parameters
Result
- Loading
- Compute
- Waiting at barrier
- AllReduce
Values for the GPUs shown
| GPU | Speed | Compute | Wait |
|---|
Simplified data-parallel model: each GPU computes its batch, all of them wait for the slowest at the barrier, then AllReduce takes the same time for everyone. The ideal assumes every GPU at nominal speed with no variability. The optimizer step and compute/communication overlap are ignored. Variability is a single reproducible draw (fixed seed), not an average: expected efficiency can differ by a few points. Orders of magnitude.
Things to try
Section titled “Things to try”- Start from the default: a single GPU at 70% out of 64. Efficiency drops to about 76%, that is 367 GPU-hours lost per day out of 1,536 available and 11,001 over 30 days. Switch to 1,024 GPUs: efficiency is the same, but the GPU-hours lost are multiplied by sixteen (5,867 per day, 176,008 over 30 days). The cost of a straggler grows with cluster size.
- Set 0 slowed-down GPUs and 5% variability: no card is faulty, each one simply varies a little from step to step. Compare 8 GPUs and 1,024 GPUs. Displayed efficiency goes from about 99% to about 90%, because the more GPUs there are, the further the slowest drifts from the mean. Even a healthy fleet loses efficiency as it grows, simply because there is always one GPU running a little late.
- Compare reported utilization with useful compute. With a straggler, reported utilization stays close to 100% while useful compute drops. No alert based on GPU utilization will ever fire.
- Raise the pooling step (the AllReduce) to 200 ms, as if the network were busier. Efficiency versus ideal even improves slightly, because the straggler weighs less in a longer step. But useful compute drops further: the network becomes the topic and optimization is no longer about the GPUs.
For technical teams: the simulator’s draw
For try 2, the simulator makes a single, reproducible draw (fixed seed) and that draw is a lucky one at 8 GPUs. Averaged over many draws, expected efficiency is closer to 95% at 8 GPUs and 89% at 1,024 GPUs. The gap narrows, the trend holds.
The rule to remember
Section titled “The rule to remember”The duration of a synchronous step is the maximum of the compute times, not their mean. As with the hike, the bigger the group, the more likely someone lags behind: the phenomenon is structural and gets worse with scale. I therefore recommend tracking step time and its breakdown for each GPU (each rank), with an alert on the gap between the slowest rank and the median, rather than the GPU utilization rate.
Going further: anatomy of a training step, the technical dimension, the cost of observability for the business and the bridges between dimensions.
Revised on 2 October 2026: prices removed; the loss shows in GPU-hours and in euros only with your price.