TechnicalPractitioner4 min read
HPC AI cluster architecture
The five layers of an AI training cluster: GPU nodes, interconnect, parallel storage, scheduler, software stack. For each, what to observe and what to protect.
For: engineers and SREs · architectsPrerequisites: Basic HPC notions: node, scheduler, interconnect.
Illustrative scenario. A research team reports that its training run, usually steady, has been advancing at half speed since yesterday. The cause could be a GPU running hot, a network cable dropping packets, storage saturated by another job, a badly tuned queue or a library update. To search quickly, you need to know which layers the cluster is made of and what each one can tell you about itself.
Physical topology
Section titled “Physical topology”A training cluster is like an office building. On each floor (a node), colleagues (the GPUs) talk through a very short corridor; between floors they take the lift (the network), which is slower. Everyone fetches their files from the basement archive (storage) while reception (the scheduler) decides who sits in which office.
flowchart TB
SCHED["Slurm / PBS scheduler"]
subgraph N1["GPU node 1"]
direction LR
G0["GPU 0<br/>80 GB HBM"] --- G1["GPU 1"] --- G2["GPU 2"] --- G3["GPU 3"]
end
subgraph N2["GPU node 2"]
direction LR
G4["GPU 4"] --- G5["GPU 5"] --- G6["GPU 6"] --- G7["GPU 7"]
end
IB["InfiniBand / RoCE<br/>latency in the order of a microsecond"]
FS["Parallel storage<br/>Lustre / IBM Storage Scale / BeeGFS"]
SCHED --> N1
SCHED --> N2
N1 <--> IB
N2 <--> IB
IB <--> FS
Inside a node, GPUs talk over NVLink, NVIDIA’s direct GPU-to-GPU link. Between nodes, they go through InfiniBand, a network built for computing, or RoCE, Ethernet capable of direct memory-to-memory transfers. Parallel storage feeds everyone; the scheduler decides what runs where.
The five layers
Section titled “The five layers”A few terms come up in the table. HBM (High Bandwidth Memory) is the very fast memory mounted on the GPU. SMs (Streaming Multiprocessors) are the GPU’s compute blocks: their occupancy tells you whether the GPU is really working. ECC errors are errors caught by the memory’s error-correcting code; throttling is the clock slowdown a GPU imposes on itself when it runs hot or exceeds its power budget. A fat tree is a tree-shaped network whose branches widen towards the top, so that any node can talk to any other at full speed. NCCL is NVIDIA’s library that organizes exchanges between GPUs, including AllReduce, the operation that averages the gradients of every GPU at each step. Prolog and epilog scripts are run by the scheduler at the start and end of each job.
| Layer | What it contains | What to observe | What to protect |
|---|---|---|---|
| L1. GPU nodes | 4 to 8 GPUs per node linked by NVLink (600 GB/s on A100, 900 GB/s on H100, 1.8 TB/s on Blackwell), 80 GB of HBM per GPU on A100 and H100, more on later generations (141 GB on H200), 2 orchestration CPUs, several hundred GB of RAM, local NVMe for staging | dcgm-exporter: SM occupancy, HBM bandwidth, ECC errors, throttling, energy | firmware, GPU memory between jobs |
| L2. Interconnect | InfiniBand HDR (200 Gb/s), NDR (400 Gb/s) or XDR (800 Gb/s, Blackwell generation), or RoCE; non-blocking fat tree topology; gradients flow over it during the NCCL AllReduce | ibstat, perfquery, port counters, retransmits | the RDMA blind spot |
| L3. Parallel storage | Lustre or IBM Storage Scale (formerly Spectrum Scale, or GPFS), files striped across dozens of servers, aggregate throughput from a few tens of GB/s to several TB/s depending on system size; GPUs must never wait for data | Lustre throughput, metadata latency | integrity of training data |
| L4. Scheduler | Slurm or PBS: queues, GPU allocation, job isolation; prolog and epilog scripts to inject instrumentation | slurm-exporter: jobs, queues, wait times | isolation between jobs and between tenants |
| L5. ML software stack | CUDA or ROCm driver, PyTorch or JAX, NCCL for collective communications, Docker or Apptainer (formerly Singularity) containers | PyTorch profiler, NCCL logs | software supply chain |
On the last row: CUDA (NVIDIA) and ROCm (AMD) are the software layers that drive the GPUs; PyTorch and JAX are the libraries models are written in; Apptainer is a container format common in HPC. The “RDMA blind spot” in layer L2 is covered in chapter 4.
Four families of signals
Section titled “Four families of signals”Back to the slow training run. The useful signals fall into four families. On the GPU side, you look at SM utilization, HBM bandwidth, ECC errors and clock throttling. On the network side, InfiniBand port counters, retransmission rate and per-link throughput. On the storage side, Lustre throughput and metadata latency (the time to open a file or list a directory). On the model side, finally, the loss (the error the model tries to reduce), gradient norm, step time and AllReduce latency.
In my view, the fourth family is the one most often forgotten, even though it is the one that tells you whether training is making progress. In our scenario, it is also the one that would have shown when the slowdown began. It has to be tied to the job and the account, like the other three.
Revised on 2 October 2026: Blackwell generation added (NVLink 5, InfiniBand XDR), IBM Storage Scale instead of Spectrum Scale, storage throughputs aligned with the introductory course, interconnect latency brought back to the order of a microsecond.