TechnicalPractitioner5 min read
Module 3: the three drifts
A model in production degrades in three ways: the questions change, reality changes or a component changes. For each one, the causes, examples, reproducible indicators and alert thresholds.
For: engineers and SREsPrerequisites: Have followed lab 2 of the course on the RAG pipeline.
Back to the HR assistant from the overview (an illustrative scenario). At launch, it answers leave questions well. Six months later, quality has dropped and the servers noticed nothing. Digging in, the team finds three possible explanations. Employees now ask a lot about internal mobility, a topic missing from the document base. The collective agreement changed in the spring, while the base still quotes the old version. A developer changed a search setting during a code cleanup.
These are the three drifts this module covers. All of them are silent and invisible from infrastructure monitoring.
Data, concept, pipeline
Section titled “Data, concept, pipeline”Data drift is a shop whose customers start asking for products it does not sell. Users ask questions different from the ones the system was designed for: an HR assistant built for leave receives questions about internal mobility.
Concept drift is a road map that did not keep up with the roadworks. The outside world has changed without the system knowing: the collective agreement evolves and the assistant still quotes the old version.
Pipeline drift is a recipe where someone changed a quantity without telling anyone. A component was modified without testing: top_k, the number of document chunks retrieved, goes from 5 to 1 during a code cleanup and the context becomes insufficient.
That last one does not only come from your own team. When you call an LLM through an API, the provider can change the model served behind the same name or alias without your code moving: that is pipeline drift too. To see it, record in each span the requested model (gen_ai.request.model) and the model the provider actually returned (gen_ai.response.model), along with the prompt template version and the RAG chunking settings.
From the cause to the signal that reveals it, then to the matching response:
flowchart LR C1["Data<br/>questions change"] --> S1["Intent distribution,<br/>out-of-domain volume"] --> A1["Extend the document base<br/>and the reference set"] C2["Concept<br/>reality changes"] --> S2["Faithfulness against ground truth,<br/>negative feedback"] --> A2["Update the sources<br/>and reindex"] C3["Pipeline<br/>a component changes"] --> S3["Span attributes,<br/>latency per step"] --> A3["Roll back the change<br/>and test it"]
In my view, pipeline drift is the easiest to diagnose, provided the pipeline configuration has been instrumented in the spans: its cause is found in the trace itself, without having to analyze user questions.
Mapping of terms
Section titled “Mapping of terms”This module is the site’s reference for classifying drift: the guide’s glossary and the other pieces align with it. You will come across other terms elsewhere. Here is where they fit in the three families:
| Term encountered | Canonical family |
|---|---|
| input drift, covariate shift, embedding drift (detection) | data |
| prior shift (the label distribution changes) | data |
| virtual drift (the input distribution changes without changing the input-to-output relationship) | data |
| conceptual drift, concept drift | concept |
| silent change of the provider or the model, of the prompt template, of RAG chunking | pipeline |
| output drift | not a family: it is the symptom revealed by evaluations, whose cause is data, concept or pipeline drift |
Reproducible indicators
Section titled “Reproducible indicators”For each family there are measurements you can recompute the same way every week. First, a few terms from the table. Embeddings are the vectors of numbers that sum up the meaning of each question. PSI (population stability index) and the Kolmogorov-Smirnov (KS) test compare two distributions: today’s questions and those of a reference period. Ground truth is the right answer, known and checked, against which the model’s answer is compared.
| Drift | Indicator | Method | Trigger |
|---|---|---|---|
| Data | intent distribution | PSI or Kolmogorov-Smirnov on embeddings | PSI or KS statistic above a threshold calibrated on a reference period |
| Data | out-of-domain volume | intent classifier or similarity with the base | more than 15% of traffic out of domain |
| Concept | faithfulness against ground truth | LLM judge on a sample | drop of more than 10 points |
| Concept | negative feedback | explicit thumbs-down or abandonment | more than 5% over 24 h |
| Pipeline | distribution of span attributes | histogram of top_k, of the model name | appearance of a new value |
| Pipeline | latency per step | 7-day sliding histogram | unexplained variation of more than 30% |
These thresholds are illustrative starting points, not drawn from a study: recalibrate them on your own traffic after a few weeks. The 0.6 quality score is the one used by every lab in the kit (lab 3 detection, lab 4 alert and SLO).
Green infrastructure, degraded quality
Section titled “Green infrastructure, degraded quality”Drift shows up as decorrelation: all technical indicators are normal, only the quality score drops. A dashboard that puts latency, error rate and quality score side by side makes this gap visible at a glance.
Start simple
Section titled “Start simple”A moving average of the quality score and a threshold are enough to start. Statistical tests (PSI, KS, Wasserstein) come later, when volume justifies the effort. Advanced statistical detection is covered in the path Beyond the LLM: GPUs and model quality.
Before lab 3
Section titled “Before lab 3”Data, concept and pipeline drift have neither the same cause nor the same response. Pipeline drift can be read in span attributes, provided the configuration was put there: check now that your spans carry the requested model, the returned model, the prompt version and top_k. In every case, the symptom to watch is the same: green infrastructure and a dropping quality score. It is measured against the retrieved context or, for concept drift, against ground truth, since faithfulness to the context can stay high when the base itself is outdated.
Revised on 2 October 2026: unsourced claim about the frequency of pipeline drift removed, thresholds presented as illustrative and aligned on 0.6, module code replaced by a link to the advanced AI observability path.
Revised on 4 October 2026: module designated as the site’s reference for the drift taxonomy, mapping table of terms added (output drift presented as a symptom), silent change of the provider or model classified as pipeline drift.