Skip to content

TechnicalPractitioner5 min read (+1 technical)

LLM observability: the labs

Instrument an LLM application, measure the quality of its answers, keep its cost in check and defend it in production, with a self-hosted stack that is open source but for one component. 5 modules, 4 labs, about 13 h 20.

For: engineers and SREs · architectsPrerequisites: Have already called an LLM API, know a classic observability tool and be able to read a Python script.

Reading mode

One Monday, HR receives an irritated message: the internal assistant told an employee about a leave entitlement that has not applied since the spring. The tech team opens its dashboards. Everything is green: servers are fine, answers arrive in under two seconds, no errors. The scenario is made up, yet it sums up the problem this course deals with.

Classic tools watch infrastructure: CPU, memory, response time, technical errors. Nothing they measure touches the meaning of what the model says. A sales assistant inventing the price of a discontinued product moves no chart at all. Infrastructure stays green while the model drifts.

An LLM (large language model) is the engine that generates the text, such as Mistral or Claude. RAG (retrieval-augmented generation) means handing it excerpts from your own documents before each answer: that is what lets an HR assistant quote your collective agreement instead of generalities. These labs cover the LLM and RAG. Agents and MCP servers, which give the model tools to call, are covered in the article Observing an LLM system and in the GenAI observability method. The basics (signals, maturity, evaluators) come from the guide Understanding generative AI observability, to read first.

It is aimed at platform and data teams who are comfortable with a monitoring tool without necessarily being LLM specialists. Each module explains its terms the first time it uses them. The labs are hands-on exercises on your own machine: you run commands, read traces and trigger alerts.

In practice, you should have already called an LLM API (Mistral, OpenAI, Claude) or a local model through Ollama, know at least one classic observability tool (Prometheus, Grafana, Elastic) and be able to read and modify a Python script. Knowing the basics of OpenTelemetry helps without being required.

First, you will be able to explain why an LLM cannot be monitored like a classic application. Then you will instrument one, meaning you add to the code what produces the measurements, using the OpenTelemetry GenAI conventions, the standard vocabulary for model calls. With those measurements you will build quality, cost and performance dashboards.

Next comes the part specific to LLMs: spotting data, concept and pipeline drift with reproducible indicators, then setting up continuous evaluation of the answers (a judge model, RAG-specific metrics, user feedback). You will finish with SLOs (quantified service objectives) and alerts suited to generative workloads, all on a self-hosted stack where every component can be replaced.

StepContentIndicative duration
Module 1why an LLM cannot be observed like a classic application1 h 30
Module 2OpenTelemetry GenAI conventions1 h 45
Lab 1instrument an LLM application1 h 45
Lab 2an end-to-end observed RAG pipeline1 h 30
Module 3the three drifts1 h 30
Module 4continuous evaluation in production1 h 45
Lab 3detect drift automatically1 h 30
Lab 4cost, performance, SLOs and alerts1 h 35
Module 5industrialize: 30-, 60- and 90-day plan, AI Act30 min

The labs rely on a self-hosted stack: every component runs on your machine, so you see every link in the chain. The table gives the versions pinned in the kit.

ComponentKit versionLicenseRole
OpenTelemetry (Python SDK and Collector contrib)SDK 1.28.2, Collector 0.108.0Apache 2.0instrumentation, GenAI conventions, routing
VictoriaMetricsv1.110.0Apache 2.0metrics storage
Tempo2.6.0AGPL 3.0trace storage
Grafana11.2.0AGPL 3.0dashboards, exploration, alerts
Arize Phoenix5.9.1Elastic License 2.0 (source-available)conversation exploration, evaluation
Qdrantv1.12.0Apache 2.0RAG vector database
Ollama0.3.12MITlocal LLM (Mistral 7B) and embedding model

Every component is open source except Phoenix: its Elastic License 2.0 makes the code available to read without being an open source license. The whole stack runs on a workstation with Docker (16 GB of memory recommended). Each component has equivalents, open source or commercial. Module 1 gives an overview and chapter 6 of the guide maps the full tools landscape and their licenses.

The kit (folder labs/llm-observability/, Apache 2.0 license) is not published yet. It will be on the Labs & Trainings forge. Until then, the lab lessons read as exercise statements.

On 2 October 2026 the kit was fixed for a series of defects that prevented it from producing what the lessons describe. The most visible change for you: prices are gone. Prices change fast and vary by contract, so this guide gives none. The lab budget and SLOs are expressed in tokens. The euro cost is computed only with a dated price list you supply, your provider’s.

For technical teams: the list of fixes from 2 October 2026
  • unpinned Python dependencies, while the recent Qdrant client no longer has the search() method used by labs 2 to 4: dependencies pinned and code moved to query_points();
  • metric names missing the _seconds and _total suffixes added by the exporter: empty latency and cost panels, queries fixed;
  • Grafana data sources without a uid: the five alerts and the trace-to-metric link pointed to a non-existent source;
  • a Collector cost calculation that never applied and a filter defined outside any pipeline: removed, cost is computed in the application;
  • Phoenix at the center of the diagram but with no export: a trace pipeline to Phoenix was added;
  • in-house extensions under gen_ai.* and the deprecated gen_ai.system attribute: moved to mttl.* and gen_ai.provider.name;
  • promised metrics that were missing (hallucinations, tenant on the drift score, per-feature breakdown): added;
  • an impossible concept drift scenario (reindexing at every run) and alerts that lab 4 could not trigger: explicit reindexing and a configurable simulator;
  • an illustrative price grid in lab 4: removed.

Revised on 2 October 2026: total duration corrected (13 hours), prerequisites, stack versions and licenses added, kit status and list of fixes added, AI Act described in terms of monitoring and logs; prices removed.

Revised on 4 October 2026: title “LLM observability: the labs”, scope sentence (LLM and RAG in labs; agents and MCP referred to the article and the method; prerequisite: the guide), progression box added, page moved to MDX.

Revised on 5 October 2026: lab 4 duration aligned with its steps (1 h 35), bringing the total to about 13 h 20.