TechnicalPractitioner5 min read (+1 technical)
LLM observability: the labs
Instrument an LLM application, measure the quality of its answers, keep its cost in check and defend it in production, with a self-hosted stack that is open source but for one component. 5 modules, 4 labs, about 13 h 20.
For: engineers and SREs · architectsPrerequisites: Have already called an LLM API, know a classic observability tool and be able to read a Python script.
One Monday, HR receives an irritated message: the internal assistant told an employee about a leave entitlement that has not applied since the spring. The tech team opens its dashboards. Everything is green: servers are fine, answers arrive in under two seconds, no errors. The scenario is made up, yet it sums up the problem this course deals with.
Classic tools watch infrastructure: CPU, memory, response time, technical errors. Nothing they measure touches the meaning of what the model says. A sales assistant inventing the price of a discontinued product moves no chart at all. Infrastructure stays green while the model drifts.
An LLM (large language model) is the engine that generates the text, such as Mistral or Claude. RAG (retrieval-augmented generation) means handing it excerpts from your own documents before each answer: that is what lets an HR assistant quote your collective agreement instead of generalities. These labs cover the LLM and RAG. Agents and MCP servers, which give the model tools to call, are covered in the article Observing an LLM system and in the GenAI observability method. The basics (signals, maturity, evaluators) come from the guide Understanding generative AI observability, to read first.
Who is the course for?
Section titled “Who is the course for?”It is aimed at platform and data teams who are comfortable with a monitoring tool without necessarily being LLM specialists. Each module explains its terms the first time it uses them. The labs are hands-on exercises on your own machine: you run commands, read traces and trigger alerts.
In practice, you should have already called an LLM API (Mistral, OpenAI, Claude) or a local model through Ollama, know at least one classic observability tool (Prometheus, Grafana, Elastic) and be able to read and modify a Python script. Knowing the basics of OpenTelemetry helps without being required.
What you will be able to do afterwards
Section titled “What you will be able to do afterwards”First, you will be able to explain why an LLM cannot be monitored like a classic application. Then you will instrument one, meaning you add to the code what produces the measurements, using the OpenTelemetry GenAI conventions, the standard vocabulary for model calls. With those measurements you will build quality, cost and performance dashboards.
Next comes the part specific to LLMs: spotting data, concept and pipeline drift with reproducible indicators, then setting up continuous evaluation of the answers (a judge model, RAG-specific metrics, user feedback). You will finish with SLOs (quantified service objectives) and alerts suited to generative workloads, all on a self-hosted stack where every component can be replaced.
Program
Section titled “Program”| Step | Content | Indicative duration |
|---|---|---|
| Module 1 | why an LLM cannot be observed like a classic application | 1 h 30 |
| Module 2 | OpenTelemetry GenAI conventions | 1 h 45 |
| Lab 1 | instrument an LLM application | 1 h 45 |
| Lab 2 | an end-to-end observed RAG pipeline | 1 h 30 |
| Module 3 | the three drifts | 1 h 30 |
| Module 4 | continuous evaluation in production | 1 h 45 |
| Lab 3 | detect drift automatically | 1 h 30 |
| Lab 4 | cost, performance, SLOs and alerts | 1 h 35 |
| Module 5 | industrialize: 30-, 60- and 90-day plan, AI Act | 30 min |
The stack
Section titled “The stack”The labs rely on a self-hosted stack: every component runs on your machine, so you see every link in the chain. The table gives the versions pinned in the kit.
| Component | Kit version | License | Role |
|---|---|---|---|
| OpenTelemetry (Python SDK and Collector contrib) | SDK 1.28.2, Collector 0.108.0 | Apache 2.0 | instrumentation, GenAI conventions, routing |
| VictoriaMetrics | v1.110.0 | Apache 2.0 | metrics storage |
| Tempo | 2.6.0 | AGPL 3.0 | trace storage |
| Grafana | 11.2.0 | AGPL 3.0 | dashboards, exploration, alerts |
| Arize Phoenix | 5.9.1 | Elastic License 2.0 (source-available) | conversation exploration, evaluation |
| Qdrant | v1.12.0 | Apache 2.0 | RAG vector database |
| Ollama | 0.3.12 | MIT | local LLM (Mistral 7B) and embedding model |
Every component is open source except Phoenix: its Elastic License 2.0 makes the code available to read without being an open source license. The whole stack runs on a workstation with Docker (16 GB of memory recommended). Each component has equivalents, open source or commercial. Module 1 gives an overview and chapter 6 of the guide maps the full tools landscape and their licenses.
Lab kit status
Section titled “Lab kit status”The kit (folder labs/llm-observability/, Apache 2.0 license) is not published yet. It will be on the Labs & Trainings forge. Until then, the lab lessons read as exercise statements.
On 2 October 2026 the kit was fixed for a series of defects that prevented it from producing what the lessons describe. The most visible change for you: prices are gone. Prices change fast and vary by contract, so this guide gives none. The lab budget and SLOs are expressed in tokens. The euro cost is computed only with a dated price list you supply, your provider’s.
For technical teams: the list of fixes from 2 October 2026
- unpinned Python dependencies, while the recent Qdrant client no longer has the
search()method used by labs 2 to 4: dependencies pinned and code moved toquery_points(); - metric names missing the
_secondsand_totalsuffixes added by the exporter: empty latency and cost panels, queries fixed; - Grafana data sources without a
uid: the five alerts and the trace-to-metric link pointed to a non-existent source; - a Collector cost calculation that never applied and a filter defined outside any pipeline: removed, cost is computed in the application;
- Phoenix at the center of the diagram but with no export: a trace pipeline to Phoenix was added;
- in-house extensions under
gen_ai.*and the deprecatedgen_ai.systemattribute: moved tomttl.*andgen_ai.provider.name; - promised metrics that were missing (hallucinations, tenant on the drift score, per-feature breakdown): added;
- an impossible concept drift scenario (reindexing at every run) and alerts that lab 4 could not trigger: explicit reindexing and a configurable simulator;
- an illustrative price grid in lab 4: removed.
Revised on 2 October 2026: total duration corrected (13 hours), prerequisites, stack versions and licenses added, kit status and list of fixes added, AI Act described in terms of monitoring and logs; prices removed.
Revised on 4 October 2026: title “LLM observability: the labs”, scope sentence (LLM and RAG in labs; agents and MCP referred to the article and the method; prerequisite: the guide), progression box added, page moved to MDX.
Revised on 5 October 2026: lab 4 duration aligned with its steps (1 h 35), bringing the total to about 13 h 20.