Skip to content

TechnicalPractitioner2 min read

How is an AI compute cluster built?

Slurm scheduler, nodes of 8 H100 GPUs, NVLink, InfiniBand NDR, parallel storage: click each element to see what it does, what to watch there and what threatens it, then play a training step with or without a lagging GPU.

For: engineers and SREs · architectsPrerequisites: Basic notions of GPUs and distributed training.

Reading mode

An illustrative scenario: a research team books GPUs to train a model. Training takes far longer than planned, yet the dashboard shows the cards running almost flat out. Where is the time going? And, as security will ask sooner or later, who can get at the resulting model?

A training cluster is like a manufacturing workshop. The workstations are the nodes, servers that each hold several compute cards (GPUs). Workstations talk through very fast corridors: the interconnect, NVLink inside a node and InfiniBand between nodes. Raw materials and finished goods sit in a shared warehouse, the parallel storage, where everyone reads and writes at once. A shop foreman hands out the work: the scheduler, Slurm here, which assigns machines to each task (a job). Finally, a layer of software runs the training itself.

The diagram shows a scaled-down version of that workshop: two nodes of 8 NVIDIA H100 SXM GPUs. Each element opens a card that says what it does, what to watch there and with which tool, then what threatens it. Observe mode places the collection tools and their signals. Threats mode lays the six attack paths described in the reference course on the parts they target.

Click any element of the diagram, or walk it with the keyboard (Tab, then Enter or Space): its card opens below the diagram.

Swipe the diagram sideways to see all of it.

HPC AI cluster: Slurm scheduler, two nodes of 8 H100 GPUs linked by NVLink, InfiniBand NDR interconnect, parallel storage, model weights and observability planeObservability planecollect · store · GrafanaL4 · Slurm schedulerslurm-exporterGPU node 1 · 8 × H100 SXMdcgm-exporterFalco · TetragonL5 · CUDA · PyTorch · NCCLL5 · PyTorch profiler · NCCL logsGPU 0GPU 1GPU 2GPU 3GPU 4GPU 5GPU 6GPU 7NVLink · 900 GB/sGPU node 2 · 8 × H100 SXMdcgm-exporterFalco · TetragonL5 · CUDA · PyTorch · NCCLL5 · PyTorch profiler · NCCL logsGPU 8GPU 9GPU 10GPU 11GPU 12GPU 13GPU 14GPU 15NVLink · 900 GB/sL2 · InfiniBand NDR · 400 Gb/s · fat tree · RDMAibstat · perfqueryL3 · Parallel storageLustre / IBM Storage ScaleLustre metricsModel weightsasset to protecttargeted by all 6 vectorseBPF · Falco165234

Where the collectors sit

The six threat vectors

  1. Data load
  2. Forward pass
  3. Backward pass
  4. NCCL AllReduce
  5. Optimizer
  6. Waiting at the barrier

Illustrative example: durations built from the order of magnitude in chapter 2 (a step of about 42 ms); the straggler takes twice as long for its backward pass. For more depth: the animated training step and the straggler effect simulator.

  1. Open a GPU, then the node that holds it. The GPU publishes its counters through a small program, dcgm-exporter. On their own, those counters do not say who the card is working for. The hpc_job label, a tag Slurm writes at the start and end of each job (the prolog and epilog scripts), is what ties them to a job, then to a user and an account.
  2. Play a training step without a laggard, then with one. At every step, the GPUs must pool their results (the AllReduce) and wait for each other at a barrier to do so. With one slow GPU, a straggler, every other GPU stops at that barrier: step time aligns on the slowest, while every GPU looks busy.
  3. Switch to Threats mode and open attack path 3. Direct machine-to-machine transfers (RDMA traffic) bypass the operating system: neither firewalls nor eBPF probes see them. The only signals available are InfiniBand counters, the same ones operations uses.
  4. Open the model weights, meaning the output of the training. All six attack paths converge on them and the tools that protect them are those of Observe mode.