Skip to content

TechnicalPractitioner8 min read (+1 technical)

HPC observability foundations

HPC observability foundations: what each cluster component exposes, per-job metrics, tying GPU metrics to Slurm jobs and a dashboard a director can read.

For: engineers and SREs · architectsPrerequisites: Linux command line, basics of Prometheus or Grafana.

Reading mode

Tuesday, 02:42. The training run just died.

Section titled “Tuesday, 02:42. The training run just died.”

Illustrative scenario, built for the course.

An AI training run had been going for four hours on eight A100 GPUs (NVIDIA graphics processors built for data centres). It has just stopped. Four hours on eight GPUs means 32 GPU-hours lost, to multiply by your provider’s hourly price or your internal cost. Prices change fast and vary by contract: this guide gives none, so use your provider’s dated price list.

Here are the last lines of the job’s log:

02:42:14 INFO job 184729 step 14 done
02:42:15 INFO rank 0 allreduce init
02:42:16 ERROR ECC error GPU 3
02:42:17 WARN retrying step 15
02:42:18 WARN retrying step 15
02:42:19 INFO job 184729 terminated, exit 137

Step 14 finished. Process number 0 (rank 0) started the AllReduce, the exchange that pools the results of every GPU, then the memory of GPU 3 reported an ECC error: its error-correcting code detected corrupted data. Two retries later, the job stops.

Could you tell, in five minutes, which GPU failed, on which node, for which user? Whether the hardware was already degrading in the days before? Which account just absorbed those 32 GPU-hours?

Without a correlation prepared in advance, answering means cross-checking job logs, GPU metrics and the accounting of Slurm, the scheduler that hands out machines, all by hand. The data exists, but it sits in separate tools that nobody has wired together. This course teaches you to wire them.

Three trends make this kind of observability necessary. The first is the scale of AI training: a run uses tens to thousands of GPUs for days or weeks (an order of magnitude). A silent GPU degradation then loses part of that compute.

The second is regulatory pressure. NIS2 applies to the entities in scope. Directive (EU) 2023/1791 on energy efficiency, through its Article 12, requires data centres with at least 500 kW of installed IT power to report their energy performance every year. Compute allocations also call for usage reports. In every case, auditors ask who did what, when.

The third is accountability towards several tenants. When several teams or customers share the cluster, showback, chargeback and fairshare all depend on knowing which account each joule belongs to.

What HPC changes for metrics, logs and traces

Section titled “What HPC changes for metrics, logs and traces”

The three basic signals stay the same: metrics (numeric measurements over time), logs (event records) and traces (the path of one operation through the system). HPC adds three constraints.

Cardinality, meaning the number of distinct measurement series, is higher, since you track every GPU and every rank. Retention gets longer, because researchers routinely compare against runs from years ago. Finally, every observation has to be tied to a job, a user and an account. Without that link, it is useless for chargeback or audit.

The first module walks through the five components of the cluster and what each one exposes. On the servers, node_exporter publishes system metrics and reveals NUMA placement (how processors and memory are grouped inside a machine, which affects performance). On the GPUs, the DCGM exporter (Data Center GPU Manager, NVIDIA’s GPU monitoring tool) identifies each card by a UUID label, its unique identifier. The network provides its InfiniBand or RoCE port counters (RoCE is Ethernet capable of direct memory-to-memory transfers). Parallel storage comes in three common forms: Lustre, IBM Storage Scale (formerly Spectrum Scale) and BeeGFS. Slurm, finally, exposes its jobs, accounts and partitions (the groups of machines it hands work out to).

Lab L1: write a first Python Redfish exporter (Redfish is the standard hardware management interface for servers, giving access to temperatures and power supplies), have it scraped by VictoriaMetrics (or Prometheus), display it in Grafana and trigger a thermal alert through fault injection.

The second module deals with tying GPU metrics to jobs, the heart of the 02:42 scenario. Slurm can run two scripts around each job: the prolog at start, the epilog at the end. The prolog writes, for each allocated GPU, the job ID into a directory read by dcgm-exporter. That GPU’s metrics then carry an hpc_job label. The job is then linked to its user and account with sacct (Slurm’s accounting command) or a Slurm exporter, for one business answer: power consumed per account.

Lab L2: run dcgm-exporter with this mapping side by side with a Slurm exporter, write a PromQL join on the job ID that attributes GPU power to accounts, then an energy-budget alert.

Lab L3: detect contention inside a GPU split with MIG (Multi-Instance GPU, which shares one card between several users). A MIG GPU instance can hold several compute instances, which share that instance’s memory and bandwidth. They can therefore get in each other’s way. That contention stays invisible to standard utilization metrics.

For technical teams: the mapping setting

The directory read by dcgm-exporter is set with the --hpc-job-mapping-dir option or the DCGM_HPC_JOB_MAPPING_DIR environment variable. Metrics for the GPUs concerned then carry the hpc_job label.

A published lab: the energy bill of each job

Section titled “A published lab: the energy bill of each job”

This mapping already runs in a lab published on the forge, slurm-otel-energy, which goes all the way to the bill. Think of a building’s electricity bill that you want to split by flat, then by room. The lab does the same for a GPU cluster. For every finished job it tells you what code ran, on which GPUs, for how long and how much energy each phase cost: data loading, compute, evaluation, checkpoints.

Everything travels through OpenTelemetry. The bill is therefore not a home-made report: it is a trace and a set of metrics that any compatible tool can read. The lab runs on a laptop with simulated GPUs and on real NVIDIA GPUs with the same code.

The bundled scenario runs three jobs for three users. One of them is a notebook that holds a GPU without ever using it. It is a reminder of a little-known truth: a reserved GPU uses energy even when nothing runs on it. The repository also says what the lab cannot measure yet. Very short phases stay blurry, the share of the server charged to each job is a split rule rather than a measurement. Finally, a GPU shared by several jobs is divided equally.

A first, simpler version is still published: hpctrace. It runs on a single Slurm node and turns every job into an OpenTelemetry trace, with energy per job and per phase. It works without a GPU or with a real NVIDIA card. It was the basis for a semantic conventions proposal for HPC, that is, shared attribute names (hpc.job.id, for example) so that every tool describes a job the same way.

For technical teams: running the lab and what it shows

You need Docker with Compose v2, an x86_64 machine and about 4 GB of free memory. No GPU is required: the simulator speaks exactly like dcgm-exporter.

Terminal window
git clone https://forge.erythix.tech/labstraining/labs/slurm-otel-energy.git
cd slurm-otel-energy
docker compose up -d --build
docker compose exec slurmctld /opt/lab/jobs/submit-scenario.sh

The prolog writes the job id for each GPU, in the format dcgm-exporter already understands. The TaskProlog hands the job a TRACEPARENT computed from its id, so every node finds the same trace without talking to the others. The application opens one span per phase with the standard OpenTelemetry SDK. After the job, an accounting service reads Slurm, takes the energy from the GPUs’ hardware counters, checks it against the integrated power samples and splits it by phase. On the reference run published in the repository, with simulated GPUs, the two calculations agree within a few hundredths of a percent.

The third module asks for a real deliverable. The setting: a GPU cluster of several hundred nodes shared by 6 accounts. The deliverable: one dashboard that passes the five-second test in front of a director (within five seconds, they must see whether all is well) plus three alerts that only page when intervention is truly needed.

Capstone: design the dashboard (six panels max), build it in Grafana and vmalert (VictoriaMetrics’ alerting engine), validate it against three injected faults, then defend it in five minutes.

Every lab will include a starter file, a reference solution, automated validators and fault-injection scripts.

By the end of the course, you will be able to:

  1. deploy a standard stack from scratch: VictoriaMetrics (or Prometheus), Grafana, vmalert, Alertmanager;
  2. read DCGM and Slurm metrics fluently, knowing the five GPU and six Slurm metrics that matter most;
  3. write a small Python Prometheus exporter that respects naming conventions and label discipline;
  4. tie metrics from different collectors to a job, then to a Slurm account, to attribute consumption;
  5. design a dashboard that passes the five-second test in front of a non-technical stakeholder;
  6. write an alert rule that does not flap (switch on and off in a loop), with a runbook annotation and a documented threshold rationale.

The course is aimed at SREs and platform engineers new to HPC, HPC sysadmins modernizing their monitoring and DevOps engineers onboarding AI workloads. It also serves research-computing staff under audit pressure as well as sovereign-cloud teams preparing AI infrastructure.

It does not cover machine learning research, Slurm administration or Kubernetes-centric platforms.

You need the Linux command line and basic Prometheus or Grafana. If you are a complete beginner, start with Understanding HPC.

Two follow-ups are planned. The first covers instrumentation, fabric and storage. You write a production-grade Slurm exporter in Go, with a lab target of sustaining a simulated load of 5,000 jobs per second. It also covers InfiniBand fabric observability, Lustre in depth, capacity and queue-wait forecasting plus full showback.

The second covers AI training, the sovereign stack and AIOps (using machine learning for operations). You run rank-wise anomaly detection on a simulated 64-rank training run with an autoencoder (a neural network that learns what normal behaviour looks like), to catch an injected straggler: a rank slower than the others, whose penalty the lab sets (for example 30% of throughput, a simulation parameter, not a measurement). Then come collective communication health, an air-gapped stack and failure prediction.


Revised on 2 October 2026: prices removed, scenario cost expressed in GPU-hours, GPU-to-job attribution described with dcgm-exporter job mapping, MIG contention clarified (compute instances of the same GPU instance), reference to Directive (EU) 2023/1791, IBM Storage Scale, actual state of the labs.