Skip to content

TechnicalPractitioner4 min readVideo

HPC AI: observability and security reference architecture

How an AI training cluster is built, what happens during one training step, the sovereign observability stack and the six attack vectors specific to HPC AI.

For: engineers and SREs · architectsPrerequisites: Basic HPC notions: node, scheduler, interconnect.

Reading mode

Illustrative scenario. A committee has to decide whether to buy a new batch of GPUs. The dashboard on screen looks reassuring: the current GPUs run at 87% utilization, so they must be saturated. Looking closer, you find they only compute usefully about 19% of the time. Most of the rest goes into exchanging data with the other GPUs and waiting for them. Buying more hardware would not fix that.

This course explains how such a gap arises, how to measure it and how to protect a training cluster shared by several teams.

HPC (High Performance Computing) brings together hundreds to thousands of machines to solve problems too large for a single computer. Since the early 2020s, training large AI models has become a major workload and the dominant one on clusters dedicated to AI. This module gives a reference architecture to observe and secure them with a self-hosted open source stack, meaning one installed and run on your own infrastructure.

Read the video text

87% reported GPU utilization.

The dashboard is green. Yet the model is training slowly. Where does the time go?

A CPU has a few powerful cores: it chains varied tasks, one after another. A GPU has thousands of small cores: they all run the same operation at the same time. And a neural network is mostly matrix multiplications: billions of identical calculations. The model lives in the GPU’s memory: 80 GB, read at 3.35 TB/s on an H100.

A 70-billion-parameter model weighs about 140 GB, for its weights alone. It does not fit in a single GPU. And training it? About 1.7 million GPU hours for Llama 2 70B (Meta, 2023).

So hundreds of GPUs must work together: that is a cluster. Inside a server, 8 GPUs linked by NVLink: the internal highway, 900 GB/s. Between servers, InfiniBand: a very fast network, 400 Gb/s per port. Around them: shared storage feeding the GPUs, and Slurm handing out the work.

At each step, every GPU computes on its share of the data… … then they all pool their results before moving on. But if a single GPU is slower… … all the others wait for it. And while waiting, they count as “busy”. That is where GPU hours disappear.

Observability runs the investigation. GPU metrics, tied to the job, show which GPU slows every step. Each GPU’s traces show in which phase the time is lost. The job context says whom to talk to and what to check: data, network or hardware. Which job, which GPU, which phase: the answer fits on one screen.

But nobody can investigate every job by hand. The answer: make observability a service of the platform. Golden path: the job label is injected at launch, without asking the researcher anything. Self-service: every researcher sees their own job, without opening a ticket. Service objectives: queue wait, successful jobs, real GPU efficiency. Chargeback: the truly useful GPU hours, team by team. Default guardrails: security is provided by the platform.

A shared cluster also attracts threats specific to HPC AI. Some escape classic tools, like RDMA traffic, invisible to a firewall. They are detected with the same observability stack.

Key takeaways:

  1. A “busy” GPU is not necessarily a computing GPU.
  2. Observe the job, the GPU and the phase, not just the machine.
  3. The platform makes this observability automatic, for everyone.

Keywords:

  • Matrix: a table of numbers; a model’s weights are made of them
  • HBM: memory stacked right against the GPU processor
  • Parameters: the numbers the model learns; 2 bytes each in 16-bit
  • NVLink: direct link between the GPUs of one server
  • InfiniBand: very low latency network between servers
  • Slurm: scheduler: it assigns GPUs to jobs
  • Forward and backward: computing the prediction, then the corrections to make
  • AllReduce: pooling the results of every GPU
  • Trace: the detailed path of an operation, step by step
  • Platform engineering: a team provides other teams with self-service internal tools
  • SLO: a numeric service quality objective
  • RDMA: direct access to another server’s memory, bypassing its system

Explore the cluster in the interactive diagram: www.meantimetolearn.com.

Music: “Radar Focus”, Blue Saga (Epidemic Sound).

A few reference points give the scale. The first is measured in petaflop/s-days: one petaflop/s-day is a million billion operations per second sustained for a whole day.

Reference pointOrder of magnitude
GPT-3 training compute (2020)about 3,640 petaflop/s-days, i.e. hundreds of GPU-years
Memory to load Llama 3 70B weights in 16-bitabout 140 GB (70 billion x 2 bytes)
H100 SXM5 HBM memory bandwidth3.35 TB/s
Typical laptop16 GB RAM, 8 GB GPU memory

A model’s weights are its parameters, the billions of numbers it adjusts during training. In 16-bit, each parameter takes 2 bytes: the weights alone of a 70-billion-parameter model therefore weigh 140 GB and do not fit in the memory of an 80 GB GPU. They would fit on a 141 GB or 192 GB GPU, or in 8-bit. But training needs several times more memory than the weights: you also have to keep the gradients (the corrections computed at each step), the optimizer states (the algorithm that applies those corrections) and the activations (the intermediate results of each layer of the model). The work must therefore be spread across several GPUs and several nodes, which then have to cooperate at every step.

A CPU, the classic processor, is like a small team of versatile head chefs: each one can do anything, make decisions, fetch an ingredient from anywhere. A GPU is like a brigade of thousands of kitchen hands all peeling the same thing at the same time. To multiply huge tables of numbers, which is most of what an AI model does, the brigade wins by far.

CPUGPU
Coresfrom 8 to 192 powerful cores depending on the model (192 on an AMD EPYC 9965)thousands of simple CUDA cores (16,896 on an H100 SXM, for example)
Clock2 to 5 GHz1 to 2 GHz
Strengthbranching, arbitrary memory access, orchestration, I/Othe same simple operation on thousands of data points at once
Role in AIorchestrate, load data, controlthe matrix multiplications at the heart of models

Core counts are examples: the maximums rise with every generation. CUDA cores are the compute units of NVIDIA GPUs; CUDA is also the name of the software platform used to program them.

  1. Cluster architecture: the five layers and what each one exposes.
  2. Anatomy of a training step: data load, forward and backward passes, AllReduce, optimizer and the straggler effect, which explains the gap between 87% and 19%.
  3. The observability stack: collection, storage, visualization and eBPF to observe without changing code.
  4. The threat surface: six vectors specific to HPC AI and how to detect them.

Revised on 2 October 2026: place of large models in clusters dated correctly, memory needed for a 70-billion-parameter model clarified, core counts presented as examples and updated, 87% and 19% example presented as illustrative.