TechnicalPractitioner5 min read
Lab 1: instrument an LLM application
Wire OpenTelemetry into a small application that calls a local LLM: initialize the SDK, set the GenAI attributes, create the standard metrics, then check that data reaches VictoriaMetrics and Tempo.
For: engineers and SREsPrerequisites: Have read modules 1 and 2 of the course, know how to use Docker and read Python.
Duration: 1 h 45. Prerequisites: modules 1 and 2, Docker, Python 3.10 or later, 16 GB of memory recommended.
An application that calls an LLM without instrumentation is silent: there is no way to know how many tokens it consumes, how long each call takes or which customer it is working for. The script in this lab asks questions to a local model and it is already instrumented. You will read that code, run it, change it, then check that each call leaves a trace you can open in Grafana and feeds metrics broken down by model, customer and feature. To finish, you will cause a failure to see what it looks like.
Prepare the environment
Section titled “Prepare the environment”The first command starts the kit’s stack. The next two download the local models: Mistral 7B answers the questions, while nomic-embed-text is an embedding model that turns a text into a vector of numbers (it will be used in lab 2).
cd labs/llm-observabilitydocker compose up -d
# Local models (5 to 10 minutes on first launch)docker exec -it mttl-ollama ollama pull mistral:7bdocker exec -it mttl-ollama ollama pull nomic-embed-text
cd app && pip install -r requirements.txt| Service | Address | Nature |
|---|---|---|
| Grafana | http://localhost:3000 | interface (admin / admin) |
| VictoriaMetrics | http://localhost:8428/vmui | query interface |
| Tempo | http://localhost:3200 | HTTP API, no interface: traces are explored from Grafana |
| Phoenix | http://localhost:6006 | interface |
Image versions in the kit: Collector contrib 0.108.0, VictoriaMetrics v1.110.0, Tempo 2.6.0, Grafana 11.2.0, Phoenix version-5.9.1, Qdrant v1.12.0, Ollama 0.3.12. They are pinned in docker-compose.yml; Python dependencies are pinned in app/requirements.txt.
Step 1: read the code (15 min)
Section titled “Step 1: read the code (15 min)”Open app/lab1_llm_app.py. There are four things to find.
The init_telemetry() function sets up the plumbing: the trace and metric providers (the OpenTelemetry SDK objects that build spans and measurements) and the OTLP exporters, which send them to the Collector. Next comes the manual span opened around the model call, then the gen_ai.* attributes set on that span. Finally, the create_genai_metrics() function creates the instruments, meaning the counters and histograms: tokens and duration (standard), cost and errors (mttl.* extensions). Lab 1 emits no cost: the euro cost is computed only in lab 4, from a price list you supply.
The core of the instrumentation fits in a few lines:
labels = { GEN_AI_PROVIDER_NAME: Provider.OLLAMA, # gen_ai.provider.name GEN_AI_OPERATION_NAME: Operation.CHAT, GEN_AI_REQUEST_MODEL: MODEL, MTTL_TENANT_ID: tenant, # in-house extension: mttl.tenant.id MTTL_FEATURE: feature, # in-house extension: mttl.feature}with tracer.start_as_current_span(f"chat {MODEL}", attributes={**labels, ...}) as span: try: response = get_ollama().chat(model=MODEL, messages=[...], options={...}) input_tokens = response.get("prompt_eval_count", 0) output_tokens = response.get("eval_count", 0)
span.set_attribute(GEN_AI_USAGE_INPUT_TOKENS, input_tokens) span.set_attribute(GEN_AI_USAGE_OUTPUT_TOKENS, output_tokens) metrics["duration"].record(time.time() - start, labels) metrics["tokens"].record(input_tokens, {**labels, GEN_AI_TOKEN_TYPE: "input"}) metrics["tokens"].record(output_tokens, {**labels, GEN_AI_TOKEN_TYPE: "output"}) except Exception as e: err = {**labels, ERROR_TYPE: type(e).__name__} span.record_exception(e) span.set_status(trace.Status(trace.StatusCode.ERROR, str(e))) metrics["duration"].record(time.time() - start, err) metrics["errors"].add(1, err) raiseThe constants (GEN_AI_PROVIDER_NAME, MTTL_TENANT_ID…) are defined in app/shared/conventions.py. When the convention evolves, only one file changes. Standard names stay in gen_ai.*, the kit extensions go in mttl.*.
One detail will save you a long hunt for an empty panel: in VictoriaMetrics, the Collector exporter adds unit and type suffixes to the names. The duration becomes gen_ai_client_operation_duration_seconds_bucket, the error counter mttl_client_errors_total and the mttl.tenant.id attribute becomes the mttl_tenant_id label. These are the names that dashboards and alerts query.
Step 2: first run (20 min)
Section titled “Step 2: first run (20 min)”python lab1_llm_app.pyIn Grafana, open Explore with the Tempo source, look for the lab1-llm-app service and open a trace. Check that the provider, model and usage attributes are present. Then open the LLM Overview dashboard: tokens and latency should appear.
Step 3: guided changes (25 min)
Section titled “Step 3: guided changes (25 min)”- Add the
gen_ai.request.top_pattribute to the span. - Vary the temperature from 0.0 to 1.0 and watch the spread of latency.
- Add a span event for the prompt:
span.add_event("prompt", {...}). This capture is acceptable in this local lab. In production it stays off by default, is enabled only by an explicit choice (opt-in) and comes with masking of sensitive data (see module 2 and §2 of the article Observing an LLM system).
Step 4: several customers and features (25 min)
Section titled “Step 4: several customers and features (25 min)”The script sends five requests spread across two customers (acme-corp, client-test) and three features (support, search, summary). Your turn: add to the PROMPTS list a third customer (globex-energy) and a fourth feature (translation), run again, then check on the LLM Overview dashboard that metrics break down by customer and by feature (panels “Jetons par client” and “Jetons par fonctionnalité”, that is tokens per customer and per feature; model and tenant variables at the top of the dashboard).
Step 5: deliberate failure (20 min)
Section titled “Step 5: deliberate failure (20 min)”You stop the model to see how a real failure shows up in the data.
docker stop mttl-ollamapython lab1_llm_app.pyWatch the spans in error in Tempo (with the error.type attribute), the increment of the mttl_client_errors_total counter and the absence of consumed tokens. Then run docker start mttl-ollama again.
What the lab showed
Section titled “What the lab showed”| Notion | What the lab shows |
|---|---|
| Traces | in this lab, an LLM call produces one span; child spans per step arrive in lab 2 |
| Metrics | always labeled by model, customer and feature |
| Errors | span.set_status(StatusCode.ERROR) plus a dedicated error counter |
| Cost | tracked here in tokens; in euros only in lab 4, with your versioned and dated price list |
Pitfalls to avoid
Section titled “Pitfalls to avoid”Do not put full prompts in attributes: they are bulky and may contain personal data. Be wary of a free-form customer identifier if you have thousands of them: every value opens new series and cardinality explodes.
Finally, remember auto-instrumentation, a library that produces spans for you with common clients. The official OpenTelemetry project package for the OpenAI client is opentelemetry-instrumentation-openai-v2; opentelemetry-instrumentation-openai, without suffix, is the one from Traceloop’s OpenLLMetry project.
Revised on 2 October 2026: box on the kit status, gen_ai.system replaced by gen_ai.provider.name, kit extensions moved from gen_ai.* to mttl.*, metric names with the suffixes added by the exporter, Tempo presented as an API, exercise aligned with the code (two customers, three features), official opentelemetry-instrumentation-openai-v2 package specified; no more zero cost emitted for the local model, since the euro cost is computed only with a supplied price list.