TechnicalPractitioner2 min read
How is an AI compute cluster built?
Slurm scheduler, nodes of 8 H100 GPUs, NVLink, InfiniBand NDR, parallel storage: click each element to see what it does, what to watch there and what threatens it, then play a training step with or without a lagging GPU.
For: engineers and SREs · architectsPrerequisites: Basic notions of GPUs and distributed training.
An illustrative scenario: a research team books GPUs to train a model. Training takes far longer than planned, yet the dashboard shows the cards running almost flat out. Where is the time going? And, as security will ask sooner or later, who can get at the resulting model?
A training cluster is like a manufacturing workshop. The workstations are the nodes, servers that each hold several compute cards (GPUs). Workstations talk through very fast corridors: the interconnect, NVLink inside a node and InfiniBand between nodes. Raw materials and finished goods sit in a shared warehouse, the parallel storage, where everyone reads and writes at once. A shop foreman hands out the work: the scheduler, Slurm here, which assigns machines to each task (a job). Finally, a layer of software runs the training itself.
The diagram shows a scaled-down version of that workshop: two nodes of 8 NVIDIA H100 SXM GPUs. Each element opens a card that says what it does, what to watch there and with which tool, then what threatens it. Observe mode places the collection tools and their signals. Threats mode lays the six attack paths described in the reference course on the parts they target.
Click any element of the diagram, or walk it with the keyboard (Tab, then Enter or Space): its card opens below the diagram.
Swipe the diagram sideways to see all of it.
Where the collectors sit
The six threat vectors
- Data load
- Forward pass
- Backward pass
- NCCL AllReduce
- Optimizer
- Waiting at the barrier
Illustrative example: durations built from the order of magnitude in chapter 2 (a step of about 42 ms); the straggler takes twice as long for its backward pass. For more depth: the animated training step and the straggler effect simulator.
Try this
Section titled “Try this”- Open a GPU, then the node that holds it. The GPU publishes its counters through a small program, dcgm-exporter. On their own, those counters do not say who the card is working for. The
hpc_joblabel, a tag Slurm writes at the start and end of each job (the prolog and epilog scripts), is what ties them to a job, then to a user and an account. - Play a training step without a laggard, then with one. At every step, the GPUs must pool their results (the AllReduce) and wait for each other at a barrier to do so. With one slow GPU, a straggler, every other GPU stops at that barrier: step time aligns on the slowest, while every GPU looks busy.
- Switch to Threats mode and open attack path 3. Direct machine-to-machine transfers (RDMA traffic) bypass the operating system: neither firewalls nor eBPF probes see them. The only signals available are InfiniBand counters, the same ones operations uses.
- Open the model weights, meaning the output of the training. All six attack paths converge on them and the tools that protect them are those of Observe mode.
Go further
Section titled “Go further”- HPC AI: observability and security reference architecture, the course this diagram is drawn from
- HPC AI cluster architecture: the five layers
- Anatomy of a distributed training step: the five phases and the straggler effect
- The HPC AI observability stack: collection, storage, visualization, eBPF
- HPC AI threat surface: the six vectors and how to detect them
- The animated training step: the ring AllReduce and the barrier in detail
- The straggler effect simulator: efficiency lost from 8 to 1024 GPUs