Explorable diagram: an HPC AI cluster

Reading mode

EngineerDecision-maker

An AI training cluster stacks five layers: GPU nodes, an interconnect, parallel storage, a scheduler and a software stack. This diagram shows a version reduced to two nodes of 8 NVIDIA H100 SXM GPUs. Each element opens a card: what it is, what to observe there and with which tool, what threatens it. Observe mode places the collectors and their signals; Threats mode lays the six attack vectors from the reference course on the parts they target.

ObserveThreats

Click any element of the diagram, or walk it with the keyboard (Tab, then Enter or Space): its card opens below the diagram.

Swipe the diagram sideways to see all of it.

Where the collectors sit

• slurm-exporter → Slurm scheduler
• dcgm-exporter (hpc_job label) → NVIDIA H100 SXM
• node-exporter, Falco, Tetragon → GPU node
• PyTorch profiler, NCCL logs → ML software stack
• ibstat, perfquery → InfiniBand NDR
• Lustre metrics → Lustre / IBM Storage Scale
• eBPF on checkpoints → Model weights

The six threat vectors

• 1Supply chain (PyPI)
• 2Data poisoning
• 3RDMA blind spot
• 4Gradient injection
• 5GPU memory leakage
• 6Container escape

Play one training step One straggler GPU (GPU 2)

1. Data load
2. Forward pass
3. Backward pass
4. NCCL AllReduce
5. Optimizer
6. Waiting at the barrier

Illustrative example: durations built from the order of magnitude in chapter 2 (a step of about 42 ms); the straggler takes twice as long for its backward pass. For more depth: the animated training step and the straggler effect simulator.

Try this

1. Open a GPU, then the node that holds it. The GPU exposes its counters through dcgm-exporter; the hpc_job label, written by Slurm prolog and epilog scripts, is what ties those counters to a job, then to a user and an account.
2. Play a step without a straggler, then with one. With a straggler GPU, every other GPU stops at the barrier before the AllReduce: step time aligns on the slowest, while every GPU looks busy.
3. Switch to Threats mode and open vector 3. RDMA traffic bypasses the kernel: neither firewalls nor eBPF see it. The only signals are InfiniBand counters, the same ones operations uses.
4. Open the model weights. All six vectors converge on them; the collectors that protect them are those of Observe mode.

Go further

• HPC AI: observability and security reference architecture, the course this diagram is drawn from
• HPC AI cluster architecture: the five layers
• Anatomy of a distributed training step: the five phases and the straggler effect
• The HPC AI observability stack: collection, storage, visualization, eBPF
• HPC AI threat surface: the six vectors and how to detect them
• The animated training step: the ring AllReduce and the barrier in detail
• The straggler effect simulator: efficiency lost from 8 to 1024 GPUs

BridgesWhat this page changes beyond the technical team.

BusinessValue, cost, risk

A GPU cluster is expensive to buy and is often run by its utilization rate. The diagram shows where real efficiency is measured (step time, its breakdown, per-rank time) and why a single slow GPU makes all the others wait.

OrganizationWho decides, who owns

Each layer has its owner: scheduler, network, storage, software stack, training team. Tying every GPU metric to a job, a user and an account is what lets them talk with the same numbers.

PeopleTeams, on-call, skills

Threats mode lays the six vectors the security team watches on the same diagram where Observe mode shows the operations team's collectors: the same stack serves both, a reason to make them work together.

Business linesProduct, finance, risk, customer

For a research team, the diagram answers a simple question: where does my training time go, and who can read my model's weights.

Why bridges matter →