TechnicalPractitioner6 min read
HPC AI threat surface
Six threat vectors specific to AI workloads on HPC: supply chain, data poisoning, RDMA blind spot, gradient injection, GPU memory leakage, container escape. And how to detect them.
For: engineers and SREs · architectsPrerequisites: Have read chapter 3 of the course.
Illustrative scenario. A company is about to train its model on a cluster it shares with other customers. Before signing, its security lead asks a single question: “Once my job is finished, who will be able to read my weights?” The honest answer depends on settings that many sites have never checked.
What needs protecting
Section titled “What needs protecting”Three assets are at stake. Model weights, the billions of parameters learned during training, are intellectual property: they represent weeks of compute. Training data is often confidential. Infrastructure, finally, is shared across teams or tenants.
An AI cluster carries risks a conventional data center does not. To work, GPUs require privileged access to the operating system kernel, which opens new attack surfaces. The fast network and the shared storage add more.
The six vectors in plain words
Section titled “The six vectors in plain words”Supply chain first. A training run installs dozens of Python packages from PyPI, the community’s public package index. One booby-trapped package is enough: it can ship a CUDA kernel (here, a small program run on the GPU, not to be confused with the operating system kernel) that quietly copies the weights.
Data poisoning targets what the model learns. Someone modifies a file on shared storage between the moment its integrity was checked and the moment the data loader (the DataLoader) reads it. The model then learns a backdoor: hidden behaviour that triggers on a signal chosen by the attacker.
The RDMA blind spot comes from the network itself; it is covered in detail below.
Gradient injection concerns federated training, where several participants train a shared model without sharing their data, sending only their gradients. A malicious participant can craft gradients that slip a backdoor into the final model. You spot it by watching the distribution of gradient norms or the Kullback-Leibler divergence (a measure of the gap between two distributions). Robust aggregation, which limits the weight of an outlier participant, protects against it.
GPU memory leakage is like a hotel room nobody cleans between guests. Memory allocated on a GPU is not wiped: outside confidential computing, nothing guarantees that the next job will not stumble on what the previous one left behind, weights included. Countermeasures include exclusive allocation, wiping or resetting the GPU between jobs, MIG (hardware partitioning of one GPU into several isolated instances) and confidential computing, an operating mode that isolates each tenant’s data.
Container escape, finally. Jobs run in containers, isolated boxes that carry their own software. The NVIDIA Container Toolkit, which gives containers access to GPUs, has had several flaws letting a malicious image break out of its box and reach the host machine. These flaws are listed under a CVE number (the public vulnerability register) and scored by CVSS out of 10. The most recent is a race condition of the TOCTOU kind (time of check, time of use): the system checks a resource, then uses it a moment later; the attacker swaps it in between. The GPU Operator is the component that installs this toolkit on Kubernetes.
Six vectors and their detection
Section titled “Six vectors and their detection”The table serves as the reference, with the fixed versions.
| Vector | Scenario | Detection and mitigation |
|---|---|---|
| Supply chain (PyPI) | a Python package ships a malicious CUDA kernel that exfiltrates weights during the backward pass | private registry, hash of every dependency, locked installs |
| Data poisoning | a file on Lustre is modified between the integrity check and the DataLoader read, to plant a backdoor | SHA-256 per shard, verified at read time |
| RDMA blind spot | reading a peer’s memory over RDMA; this traffic bypasses the kernel, no firewall, IDS or iptables sees it | InfiniBand anomaly metrics, behavioral baseline, partition isolation (P_Key) |
| Gradient injection | in federated training, crafted gradients plant a backdoor in the final model | gradient norm distribution, Kullback-Leibler divergence, robust aggregation |
| GPU memory leakage | the CUDA documentation states that allocated memory is not cleared; outside confidential computing mode, nothing guarantees that a job cannot read data left by the previous one, weights included | GPUs allocated exclusively to one job; explicit memory wipe or GPU reset (nvidia-smi --gpu-reset, with no active process) in the prolog or epilog; MIG partitioning for GPUs shared at the same time; confidential computing mode (Hopper and later), where the reset scrubs memory before the GPU goes to the next tenant |
| Container escape | NVIDIA Container Toolkit flaws: CVE-2024-0132 (fixed in 1.16.2), its bypass CVE-2025-23359 (fixed in 1.17.4), then CVE-2025-23266, known as NVIDIAScape (CVSS 9.0, fixed in 1.17.8), finally CVE-2026-24260, a time-of-check time-of-use race condition (TOCTOU, CVSS 8.5, published on 1 July 2026, toolkit up to and including 1.19.0 and GPU Operator up to and including 26.3.1, fixed in 1.19.1 and 26.3.2); a malicious image can reach the host node and every workload | toolkit at version 1.19.1 or later and GPU Operator at version 26.3.2 or later, follow NVIDIA security bulletins, Falco rules, Tetragon syscall policy |
Sources: NVIDIA Container Toolkit release notes, NVD, CVE-2025-23266, NVD, CVE-2026-24260, NVIDIA security bulletin 5850 (June 2026) and, for memory scrubbing in confidential mode, ACM Queue, “Creating the First Confidential GPUs” (republished in Communications of the ACM).
All six paths lead to the same place:
flowchart LR
W(("Model weights<br/>intellectual property"))
A["PyPI supply chain"] --> W
B["Lustre poisoning"] --> W
C["RDMA blind spot"] --> W
D["Gradient injection"] --> W
E["GPU memory leakage"] --> W
F["Container escape"] --> W
The special case of RDMA
Section titled “The special case of RDMA”RDMA (Remote Direct Memory Access) lets a network card read or write another machine’s memory directly, without going through the kernel. Picture a courier holding the key to every office who drops files on desks himself, without stopping at reception. That is what makes InfiniBand so fast. It is also what makes it invisible to conventional security tools, eBPF included, since they rely on kernel hooks: firewalls, IDS (intrusion detection systems) or iptables see nothing go by.
The only visibility comes from adapter counters and the subnet manager, the software that runs the InfiniBand fabric. You therefore need a per-job traffic baseline and alerts on deviations. InfiniBand partitions, identified by a key (the P_Key), also isolate jobs from each other.
The detection layer
Section titled “The detection layer”Four sources of information cover these risks. Falco detects suspicious file access, unexpected connections and binary modifications in real time. Tetragon enforces per-process system call policies and blocks unauthorized CUDA ioctls (direct calls to the GPU driver) and network calls. InfiniBand metrics are the only signal available on RDMA traffic. Data hashes, finally, check shard integrity at read time.
Revised on 2 October 2026: CVE-2025-23359, CVE-2025-23266 and CVE-2026-24260 added, minimum toolkit version raised to 1.19.1 and GPU Operator to 26.3.2, GPU memory leakage countermeasures made specific (reset or wipe, MIG, confidential computing).