Skip to content

TechnicalPractitioner6 min read

Module 1: why an LLM cannot be observed like a classic application

An LLM answers the same question differently, gets things wrong without raising an error and costs money on every request. This module sets out the three questions to ask it (quality, cost, performance), where to instrument it and how to choose a stack.

For: engineers and SREs · architectsPrerequisites: Have already called an LLM API and know a classic observability tool.

Take a customer support assistant built on an LLM (an illustrative scenario). Its APM dashboard, the tool that tracks an application’s health (response time, errors, throughput), has shown perfect latency and zero errors for a week. Meanwhile, some customers are getting wrong answers. Nobody notices, because nothing the APM measures looks at the content of the answers.

This module explains why and sets the frame for the labs that follow.

A classic application works like a calculator: it is deterministic, the same input always gives the same output. An LLM is more like a colleague you ask the same question twice: it is statistical, its wording changes from one time to the next and sometimes its content does too.

Errors do not show up the same way. A classic application that fails returns a visible error: an HTTP 4xx or 5xx code, an exception in the logs. An LLM is politely wrong, with a 200 OK: the request succeeded, only the answer is wrong. For a classic application, infrastructure metrics are often enough. For an LLM application, they can stay green while quality degrades.

The quality guarantee changes too. You secure a classic application with unit tests, written once. An LLM needs continuous evaluation in production, because its answers vary and the world around it keeps moving.

Cost, finally, follows a different logic. A classic application mostly costs its infrastructure (CPU, memory, disk). An LLM costs per request, depending on the size of the context, meaning the amount of text sent to it and produced by it. That amount is counted in tokens, pieces of words that most providers use as their billing unit.

To know whether an LLM system is healthy, I recommend asking it three questions and reading the answers together: quality, cost and performance. These are not signals in the technical sense. To answer them, you draw on the five signals defined in chapter 1 of the guide: logs, metrics, traces, evaluations and user feedback.

AxisQuestionTypical metricsSignals used
Qualityis the answer relevant, faithful to the context, correct?faithfulness, context recall, hallucination rate, user feedbackevaluations, user feedback; traces for diagnosis
Costhow much per request, per customer, per feature, per model?input and output tokens, euros per request, monthly cost, cache ratemetrics (tokens), traces (usage of each request)
Performancedoes the user get the answer on time and without error?time to first token (TTFT), p95 and p99 latency, error rate, throughputmetrics, traces (latency per step), logs (errors)

A few terms from the table, which later modules come back to. Faithfulness measures whether the answer relies on the documents given to the model. Context recall measures whether those documents held the information needed. A hallucination is an invented claim stated with confidence. Time to first token (TTFT) is the delay before the first word appears on screen: it is what the user feels. p95 latency is the time within which 95% of requests complete.

Looking at one axis alone is misleading. A drop in cost can come from a truncated context, which degrades quality. A rise in latency can come from a better model, which improves it.

An LLM call is like an assembly line with five stations: the client asking the question, preparation (the prompt, meaning the text sent to the model, plus the document search for RAG), the model itself, post-processing (parsing the answer, guardrails) and the final response, evaluated and logged. Each station produces its own measurements: spans (a span is one timed step of a trace, like one line on a parcel’s tracking slip), metrics and logs, each with its own attributes.

Client Pre-processing Model Post-processing Response
app, API, --> prompt, --> LLM --> parsing, --> evaluation,
agent RAG retrieval inference guardrails log
tenant_id rag_stage model guardrail_action quality_score
feature top_k tokens parser_status hallucination
user_id docs_retrieved latency

Observing a call means covering these five stations. If only the model call is instrumented, the other four stay in the dark, even though it is preparation (document search in particular) that decides what the model has in front of it.

The labs only need to place their stack among three families of tools, plus an instrumentation layer common to all of them. The first two families are products you buy or install as a whole; the third is an assembly of components; the instrumentation layer stores nothing: it is the libraries that produce the data inside your code.

FamilyExamples
General-purpose SaaS observability platformsDatadog LLM Observability, New Relic AI Monitoring, Grafana Cloud
LLM-specialized platforms, SaaS or self-hostedLangfuse, Arize Phoenix or Arize AX, LangSmith, Helicone
Assembled self-hosted stackOpenTelemetry Collector, VictoriaMetrics, Prometheus or Mimir, Tempo or Jaeger, Grafana, Loki
Instrumentation layer (libraries, not storage products)OpenTelemetry SDK and instrumentations with the GenAI conventions, OpenLLMetry (Traceloop), OpenInference (Arize)

The full tools landscape, the selection criteria (data sovereignty, cost, integration) and the licenses are covered in chapter 6 of the guide. The versions and licenses of the kit components are in the overview. Keep in mind that Phoenix is under Elastic License 2.0: its code is available, though the license is not open source.

These labs use a self-hosted stack, complemented by Phoenix, for a teaching reason: you see every link in the chain. The choice remains reversible. OpenTelemetry, the open standard for collecting traces, metrics and logs, makes instrumentation portable: the same spans and metrics can be sent to any of the families above.

flowchart TB
  APP["LLM application<br/>Python, OTel SDK,<br/>GenAI conventions"] -->|"OTLP"| COL["OTel Collector<br/>reception, enrichment,<br/>routing"]
  COL -->|"metrics"| VM["VictoriaMetrics"]
  COL -->|"traces"| TP["Tempo"]
  COL -->|"traces"| PX["Phoenix<br/>exploration, evaluation"]
  VM --> GR["Grafana<br/>dashboards, alerts"]
  TP --> GR

The application is instrumented only once. It sends everything to the Collector over OTLP, the OpenTelemetry protocol. The Collector then acts like a postal sorting center: metrics go to VictoriaMetrics, the same traces go to Tempo and to Phoenix. Grafana is the reading point for metrics, traces and alerts. Phoenix has its own interface to explore and evaluate conversations.

An LLM fails silently, with errors of meaning rather than technical errors. Before module 2, try this on your own application: which dashboard would tell you today that an answer is wrong? If the answer is “none”, that is your starting point. Instrument all five stations of the chain, not only the model call, then read quality, cost and performance together. With OpenTelemetry at the center, the choice of storage tools remains reversible.

Revised on 2 October 2026: tool families are compared on the same criteria (getting started, cost, data location, operations), instrumentation libraries are separated from storage products, licenses are specified (Phoenix under Elastic License 2.0, source-available) and Phoenix now receives traces from the Collector.

Revised on 4 October 2026: the three axes are presented as questions and mapped onto the guide’s five signals; the “Choosing a stack” section is reduced to what the labs need and points to chapter 6 of the guide for the tools landscape and licenses.