TechnicalPractitioner6 min read
Module 1: why an LLM cannot be observed like a classic application
An LLM answers the same question differently, gets things wrong without raising an error and costs money on every request. This module sets out the three questions to ask it (quality, cost, performance), where to instrument it and how to choose a stack.
For: engineers and SREs · architectsPrerequisites: Have already called an LLM API and know a classic observability tool.
Take a customer support assistant built on an LLM (an illustrative scenario). Its APM dashboard, the tool that tracks an application’s health (response time, errors, throughput), has shown perfect latency and zero errors for a week. Meanwhile, some customers are getting wrong answers. Nobody notices, because nothing the APM measures looks at the content of the answers.
This module explains why and sets the frame for the labs that follow.
Two opposite behaviors
Section titled “Two opposite behaviors”A classic application works like a calculator: it is deterministic, the same input always gives the same output. An LLM is more like a colleague you ask the same question twice: it is statistical, its wording changes from one time to the next and sometimes its content does too.
Errors do not show up the same way. A classic application that fails returns a visible error: an HTTP 4xx or 5xx code, an exception in the logs. An LLM is politely wrong, with a 200 OK: the request succeeded, only the answer is wrong. For a classic application, infrastructure metrics are often enough. For an LLM application, they can stay green while quality degrades.
The quality guarantee changes too. You secure a classic application with unit tests, written once. An LLM needs continuous evaluation in production, because its answers vary and the world around it keeps moving.
Cost, finally, follows a different logic. A classic application mostly costs its infrastructure (CPU, memory, disk). An LLM costs per request, depending on the size of the context, meaning the amount of text sent to it and produced by it. That amount is counted in tokens, pieces of words that most providers use as their billing unit.
Three questions to ask together
Section titled “Three questions to ask together”To know whether an LLM system is healthy, I recommend asking it three questions and reading the answers together: quality, cost and performance. These are not signals in the technical sense. To answer them, you draw on the five signals defined in chapter 1 of the guide: logs, metrics, traces, evaluations and user feedback.
| Axis | Question | Typical metrics | Signals used |
|---|---|---|---|
| Quality | is the answer relevant, faithful to the context, correct? | faithfulness, context recall, hallucination rate, user feedback | evaluations, user feedback; traces for diagnosis |
| Cost | how much per request, per customer, per feature, per model? | input and output tokens, euros per request, monthly cost, cache rate | metrics (tokens), traces (usage of each request) |
| Performance | does the user get the answer on time and without error? | time to first token (TTFT), p95 and p99 latency, error rate, throughput | metrics, traces (latency per step), logs (errors) |
A few terms from the table, which later modules come back to. Faithfulness measures whether the answer relies on the documents given to the model. Context recall measures whether those documents held the information needed. A hallucination is an invented claim stated with confidence. Time to first token (TTFT) is the delay before the first word appears on screen: it is what the user feels. p95 latency is the time within which 95% of requests complete.
Looking at one axis alone is misleading. A drop in cost can come from a truncated context, which degrades quality. A rise in latency can come from a better model, which improves it.
Where to instrument?
Section titled “Where to instrument?”An LLM call is like an assembly line with five stations: the client asking the question, preparation (the prompt, meaning the text sent to the model, plus the document search for RAG), the model itself, post-processing (parsing the answer, guardrails) and the final response, evaluated and logged. Each station produces its own measurements: spans (a span is one timed step of a trace, like one line on a parcel’s tracking slip), metrics and logs, each with its own attributes.
Client Pre-processing Model Post-processing Responseapp, API, --> prompt, --> LLM --> parsing, --> evaluation,agent RAG retrieval inference guardrails log
tenant_id rag_stage model guardrail_action quality_scorefeature top_k tokens parser_status hallucinationuser_id docs_retrieved latencyObserving a call means covering these five stations. If only the model call is instrumented, the other four stay in the dark, even though it is preparation (document search in particular) that decides what the model has in front of it.
Choosing a stack
Section titled “Choosing a stack”The labs only need to place their stack among three families of tools, plus an instrumentation layer common to all of them. The first two families are products you buy or install as a whole; the third is an assembly of components; the instrumentation layer stores nothing: it is the libraries that produce the data inside your code.
| Family | Examples |
|---|---|
| General-purpose SaaS observability platforms | Datadog LLM Observability, New Relic AI Monitoring, Grafana Cloud |
| LLM-specialized platforms, SaaS or self-hosted | Langfuse, Arize Phoenix or Arize AX, LangSmith, Helicone |
| Assembled self-hosted stack | OpenTelemetry Collector, VictoriaMetrics, Prometheus or Mimir, Tempo or Jaeger, Grafana, Loki |
| Instrumentation layer (libraries, not storage products) | OpenTelemetry SDK and instrumentations with the GenAI conventions, OpenLLMetry (Traceloop), OpenInference (Arize) |
The full tools landscape, the selection criteria (data sovereignty, cost, integration) and the licenses are covered in chapter 6 of the guide. The versions and licenses of the kit components are in the overview. Keep in mind that Phoenix is under Elastic License 2.0: its code is available, though the license is not open source.
These labs use a self-hosted stack, complemented by Phoenix, for a teaching reason: you see every link in the chain. The choice remains reversible. OpenTelemetry, the open standard for collecting traces, metrics and logs, makes instrumentation portable: the same spans and metrics can be sent to any of the families above.
The target architecture
Section titled “The target architecture”flowchart TB APP["LLM application<br/>Python, OTel SDK,<br/>GenAI conventions"] -->|"OTLP"| COL["OTel Collector<br/>reception, enrichment,<br/>routing"] COL -->|"metrics"| VM["VictoriaMetrics"] COL -->|"traces"| TP["Tempo"] COL -->|"traces"| PX["Phoenix<br/>exploration, evaluation"] VM --> GR["Grafana<br/>dashboards, alerts"] TP --> GR
The application is instrumented only once. It sends everything to the Collector over OTLP, the OpenTelemetry protocol. The Collector then acts like a postal sorting center: metrics go to VictoriaMetrics, the same traces go to Tempo and to Phoenix. Grafana is the reading point for metrics, traces and alerts. Phoenix has its own interface to explore and evaluate conversations.
What to take into the next module
Section titled “What to take into the next module”An LLM fails silently, with errors of meaning rather than technical errors. Before module 2, try this on your own application: which dashboard would tell you today that an answer is wrong? If the answer is “none”, that is your starting point. Instrument all five stations of the chain, not only the model call, then read quality, cost and performance together. With OpenTelemetry at the center, the choice of storage tools remains reversible.
Revised on 2 October 2026: tool families are compared on the same criteria (getting started, cost, data location, operations), instrumentation libraries are separated from storage products, licenses are specified (Phoenix under Elastic License 2.0, source-available) and Phoenix now receives traces from the Collector.
Revised on 4 October 2026: the three axes are presented as questions and mapped onto the guide’s five signals; the “Choosing a stack” section is reduced to what the labs need and points to chapter 6 of the guide for the tools landscape and licenses.