TechnicalExpert5 min read
Beyond the LLM: GPUs and model quality
Two modules to go beyond the LLM: shared GPU infrastructure, then model quality over time (classic ML, statistical drift, retraining). Assumes the LLM observability course has been completed.
For: engineers and SREs · architectsPrerequisites: Having completed 'LLM observability: the labs'; experience industrializing AI models in production.
This path is the last step of the progression. It goes beyond the LLM, first towards GPU infrastructure, then towards model quality over time. It is aimed at ML engineers, platform engineers, SREs and MLOps teams (the people who put models into production and keep them there) who already deal with industrializing models.
It assumes the LLM observability: the labs course (GenAI instrumentation, RAG, drift, cost, alerts) and does not repeat it. For the concepts, see the guide.
Its own scope is classic ML (prediction or classification models on tabular data, as opposed to generative AI), statistical drift, the model registry (MLflow) and retraining. The guide, which focuses on generative AI, leaves them out.
The path has two modules, both in preparation. Module 1 covers GPU and AI infrastructure monitoring (DCGM, NVIDIA exporters, Slurm, Kubernetes); HPC observability foundations and the HPC AI reference architecture already cover part of it. Module 2 covers advanced drift detection and model quality (PSI, KS, Wasserstein, Evidently, MLflow, retraining).
Module 1: GPU and AI infrastructure monitoring
Section titled “Module 1: GPU and AI infrastructure monitoring”On GPU infrastructure, the utilization rate says little. It is a bit like judging a motorway by the number of cars on it without looking at the queues at the toll booths. A GPU showing 80% can hide saturated memory bandwidth, underused tensor cores (the units specialized in matrix computation) or a network bottleneck between machines.
An illustrative scenario: a training run shows 85% utilization while memory bandwidth sits at 99%. The bottleneck is memory and the utilization figure does not show it.
The module first teaches you to map the layers of an AI infrastructure: the GPU, its memory, the interconnect between GPUs, the scheduler that distributes jobs, the application. It then shows how to deploy the DCGM exporter (NVIDIA’s tool that exposes GPU metrics) and collect the advanced metrics. These include SM occupancy (the streaming multiprocessors, the GPU’s compute blocks), tensor core utilization, memory bandwidth, thermal throttling (the GPU slows down when it runs too hot), NVLink and NCCL (the fast link between GPUs and the library that makes them talk) and power use.
The module also covers instrumenting Slurm jobs (the scheduler common in computing centers) and Kubernetes AI pods with the right labels, then building a dashboard an SRE who is not an AI specialist can use. It teaches you to recognize problem patterns: saturated bandwidth, slow NCCL, insufficient memory during training. It ends with defining the SLOs of an AI infrastructure: availability, memory overrun rate, wait time, throughput.
Two labs are planned (in preparation). The first sets up a DCGM stack with a Prometheus-compatible metrics backend and detects underused tensor cores on a synthetic job. The second monitors Slurm or Kubernetes, with per-job metrics, failure rate and cost allocation per team.
Module 2: advanced drift detection and model quality
Section titled “Module 2: advanced drift detection and model quality”A model that predicts payment default risk was trained on the customers of two years ago. Since then, the customer base has changed. The model keeps answering, only now on a population it has never seen. That is drift, like a road map that ages while the roads change.
Spotting drift is not enough. You still have to qualify it statistically and tell a real change from a false positive, for example a seasonal spike that will come back every year. Then you choose between alerting, retraining or letting it pass. Finally you keep a record of those decisions.
The module starts by formally distinguishing drift types and placing them in the site’s three families (data, concept, pipeline), whose reference is module 3 of the LLM course. Covariate shift (the inputs change), prior shift (the proportion of cases changes) and virtual drift belong to data. Concept drift, where the relationship between inputs and outcome changes, belongs to concept. Then come their patterns: sudden, gradual, incremental, recurring.
It then puts into practice the statistical tests that compare two distributions, on tabular features and on embeddings: PSI (population stability index), Kolmogorov-Smirnov, Chi², Wasserstein and MMD. You learn the strengths and limits of each. The module also shows how to configure Evidently or an equivalent in production.
It covers retraining strategy, triggered by drift, by a drop in performance or by the calendar. The new model (the challenger) is compared with the one in place (the champion) before replacing it, with a safe rollback. The module covers keeping a model registry with MLflow or an equivalent. It also deals with production lineage: knowing which model, trained on which data, runs where.
Finally, it sets up an AI quality committee and its rituals. It also prepares the technical documentation the AI Act requires for high-risk systems. Since Regulation (EU) 2026/1744, these obligations apply from 2 December 2027 for Annex III systems and from 2 August 2028 for Annex I systems.
Three labs are planned (in preparation): detection on tabular features with handling of seasonal false alarms, detection on text embeddings built into a RAG pipeline, then the full chain from an Evidently alert to a controlled deployment through a webhook, an MLflow job and a validation step.
For Monday morning
Section titled “For Monday morning”While the modules are being prepared, look at your GPU dashboards. If they show only the utilization rate, add memory bandwidth and SM occupancy: that is where the bottleneck in the scenario above was hiding. On the model side, write down for each one the date and population of its last training. It is the first step towards a registry.
Revised on 2 October 2026: AI Act timeline updated after Regulation (EU) 2026/1744, modules 2 and 3 and their labs flagged as in preparation, GPU example presented as an illustrative scenario, mention of a non-existent security path removed.
Revised on 4 October 2026: path renamed “Beyond the LLM: GPUs and model quality”, the LLM course becomes a prerequisite rather than a module, modules renumbered, drift terms placed in the site’s taxonomy (data, concept, pipeline) with a link to module 3 of the LLM course, own scope clarified (classic ML, MLflow, retraining), page moved to MDX with the progression box.
Revised on 25 October 2026: page rewritten for a wider audience (images, terms explained, objectives written as prose), with no change of substance.