Skip to content

TechnicalPractitioner4 min read

Lab 3: detect drift automatically

Establish a baseline, cause data, concept and then pipeline drift in turn, spot it on the dashboards, compare Grafana and Phoenix and write a response plan.

For: engineers and SREsPrerequisites: Have completed lab 2 and read module 3 of the course.

Duration: 1 h 30. Prerequisites: lab 2 completed, module 3.

In this lab you play out the three situations from module 3 on your own machine. The lab 2 RAG pipeline first runs normally, to set a baseline. Then the script asks it off-topic questions, makes it quote information that has become wrong and quietly changes one of its settings. Each time, the goal is to see the drift appear in the data, find its cause and decide what to do.

The script runs four phases in sequence. It publishes under its own service, lab3-drift, reusing the lab 2 pipeline. The detection rule is simple: a phase is drifting when its average falls below 0.6, the threshold shared by all labs.

PhaseWhat happensTracked scoreExpected order of magnitude
Baseline5 in-domain questions, 3 repetitionsfaithfulness to contextabove the threshold, around 0.75
Data drift5 out-of-domain questions (cooking, weather, sport), 2 repetitionsfaithfulness to contextwell below the threshold, around 0.20
Concept drift2 questions whose answer has changed in reality, 3 repetitionscomparison with ground truthclose to 0 as long as the knowledge base is not updated
Pipeline drifttop_k silently drops from 5 to 1faithfulness to contextbelow the threshold, around 0.45

These values are indicative: they vary with the model and from one run to the next. The kit’s faithfulness is a ratio of words shared between the answer and the retrieved context, not a measure of meaning. It is enough to show a break, not to grade an answer finely.

Terminal window
cd app && python lab3_drift_detection.py
# a single phase, several passes:
python lab3_drift_detection.py --phase data --repeat 2

On the Drift Detection dashboard, first look for the score drop when moving from the baseline to data drift. Then find the drift events emitted (mttl_drift_events_total, one per drifting phase), then the table of drifting customers (drift-tenant, average score below 0.6 over 15 min).

For pipeline drift, open a trace: the cause, mttl.rag.top_k = 1, is visible in the attributes of the retrieval span. That is the payoff of putting the configuration in the spans.

The concept phase compares the answers with a ground truth, the known right answer (VERITE_TERRAIN in the script). The fictional offer now costs 59 EUR and responds within 2 hours, while the knowledge base still says 49 EUR and 4 hours. Faithfulness to context stays high, since the model faithfully repeats an outdated knowledge base. Only the comparison with ground truth reveals the drift.

  1. Edit DOCS in lab2_rag_pipeline.py (59 EUR, 2 hours).
  2. Run python lab3_drift_detection.py --phase concept again: nothing changes, because the scripts never reindex on their own.
  3. Run python lab2_rag_pipeline.py --reindex, then the concept phase again: the score goes back up.

Question for discussion: how do you industrialize this detection? A few leads: versioning the knowledge base, rebuilding embeddings, running non-regression tests on a reference set.

Traces also go to Phoenix: the Collector sends them through the otlp/phoenix exporter (port 4317 inside the Docker network, published on 4319 on the host). Open http://localhost:6006 and compare the two approaches.

# otel/otel-collector-config.yaml (excerpt)
exporters:
otlp/phoenix:
endpoint: phoenix:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [resource/defaults, transform/genai_compat, batch]
exporters: [otlp/tempo, otlp/phoenix]

Grafana is strong on metrics, alerts and on-call: it is the tool to detect and alert. Phoenix is built to explore conversations and evaluate them: it is the tool to understand and qualify. I recommend keeping both. The alert comes from Grafana, then the investigation of the conversations involved happens in Phoenix.

Write a short plan for each type of drift. For data drift: whom to notify and how quickly. For concept drift: the knowledge base update process and its frequency. For pipeline drift: non-regression tests and quality gates in continuous integration, which block a change until the tests pass.

NotionWhat the lab shows
Statistical detectionmoving average and threshold: enough to get started
Next stepKS, PSI, Wasserstein
Grafana and Phoenixoperational alerting on one side, exploration and evaluation on the other
Triageevery alert has an owner and a procedure

Revised on 2 October 2026: box on the kit status, expected scores presented as indicative, the kit’s faithfulness measure explained, customer label added to the drift metric, reindexing separated into the --reindex option, configuration of trace export to Phoenix.