TechnicalPractitioner7 min read
Understanding generative AI observability
A technical guide to generative AI observability (LLM, RAG, agents, MCP) for SRE, platform and ML engineering teams: what to measure, in what order, with which tools and how to keep the platform alive.
For: engineers and SREs · architectsPrerequisites: Know classical observability (logs, metrics, traces). Having seen an application that calls an LLM helps but is not required.
Take an illustrative scenario. A health insurer launches an assistant that answers members’ questions from the policy terms. The dashboards are green: the service responds in under two seconds with no errors. Three weeks later, customer service starts forwarding complaints. The assistant claims a treatment is covered when it has not been since the last policy update.
Nothing in the monitoring had moved. That is where this guide starts: an AI system can be up, fast and wrong. Classical observability sees the first two adjectives. This guide explains how to see the third.
Who is this guide for?
Section titled “Who is this guide for?”The scope is generative AI. That covers LLMs (large language models, the models that produce text), RAG pipelines (the model answers from documents it is handed), agents (the model chains actions in a loop) and MCP tools (the standard protocol through which a model calls tools). Classic ML and computer vision are out of scope: they are covered in Beyond the LLM: GPUs and model quality.
The intended readers are SRE, platform and ML engineering teams who run, or are about to run, AI features in production. I assume you know logs, metrics and traces. I do not assume you are an AI specialist: every domain term is explained the first time it shows up.
By the end, you should know what to measure, in what order, with which tools and how to keep the platform running over time. Two constraints come back in every chapter: budget and personal data.
The guide in fifteen points
Section titled “The guide in fifteen points”- Up, fast and wrong. Classical observability covers system health (the horizontal axis). AI observability adds output quality (the vertical axis): is the answer correct, faithful to its sources and safe?
- Five properties of AI systems break classical assumptions: the same question does not always get the same answer, the exchanges are natural language, quality only shows at runtime, behavior drifts over time and cost varies from one request to the next (a single agent loop can cost as much as thousands of requests).
- Five signals, one key. Evaluations and user feedback join logs, metrics and traces. The trace ID links every step, every score and every piece of feedback into one queryable object.
- Every layer of the AI stack emits its own signals, from the application down to the model, through document retrieval and tools. All of them connect through the trace ID.
- There are four overlapping system types: LLM-only, RAG, agents, MCP tools. Real applications combine them, so the platform has to handle all four.
- A trace tree tells the story of a request. Evaluations live in the same trace as the request they score; user feedback attaches to it later, through the trace ID.
- The maturity ladder has six levels. Place yourself honestly: without an explicit observability strategy, an AI feature stays at Level 0. In my view, Level 3 (online evaluation) is a realistic six-month objective.
- Implementation follows seven ordered steps: define the questions, instrument, collect, store, evaluate, visualize and alert, close the loop. You instrument to answer questions you have already written down. Instrumenting first and hoping the questions will show up later does not work.
- The OpenTelemetry Collector is the integration point. This relay between the application and storage redacts, enriches, samples, batches and exports the data. Order matters: redaction runs before batching.
- I propose a self-hostable reference architecture where each layer can be swapped independently: OpenInference, Collector, VictoriaMetrics, Phoenix, Tempo, VictoriaLogs, Ragas, Grafana. Choosing OpenInference for instrumentation is a trade-off, not an unconditional recommendation (see chapter 6). Each layer can be compared with its alternatives, such as Prometheus, Mimir, Langfuse, Loki or managed services.
- There are three types of evaluators with very different costs: rules, classifiers and the LLM judge (a model that grades another model’s answers). For example, cheap checks on 100 percent of traces, small judges on 10 percent, large judges on 1 percent plus flagged traces. Always evaluate every error and every piece of negative feedback.
- Online and offline evaluation are both required. The first catches drift in production. The second acts as the gate before each upgrade.
- To run the platform: tail sampling by default, bounded metrics dimensions, high cardinality kept on traces, redaction in the collector, role-based access control (RBAC) and an audit log on trace queries, an egress allowlist for judges.
- Version everything that influences the answer, replay old traces against new versions, use shadow mode, A/B tests and canaries. Block the build as soon as a score regresses.
- Avoid the anti-patterns and close the loop. Without a weekly review that turns failed traces into regression tests, the platform is an expensive read-only archive. Start from your current level (see the appendix).
Contents
Section titled “Contents”- Foundations: the two axes, the five signals, the vocabulary.
- Scope and taxonomy of AI systems: LLM-only, RAG, agents, MCP and the shape of their traces.
- The maturity roadmap: from Level 0 (blind operation) to Level 5 (closed loop).
- Implementation in seven steps: from the questions to the closed loop, with the OpenTelemetry GenAI attributes.
- Evaluator design: catalog, LLM-as-judge patterns, calibration, aggregation, sampling.
- The tools landscape: self-hosting or managed service, licenses, reference stacks.
- Operating the platform: sampling, cost, cardinality, personal data, security, ownership.
- Versioning, replay and experimentation: changing the system without breaking it.
- Anti-patterns and common pitfalls: twelve traps and their fixes.
- Appendix: glossary, references, next steps by level.
Topics this guide is the reference for
Section titled “Topics this guide is the reference for”Several pieces on this site deal with generative AI observability. To keep diverging versions from circulating, each topic has a single home. The other pieces summarize it in one sentence and link here.
| Topic | Reference lesson |
|---|---|
| The five signals (logs, metrics, traces, evaluations, user feedback) | 1. Foundations |
| The four system types (LLM-only, RAG, agents, MCP) | 2. Scope and taxonomy |
| Maturity levels 0 to 5 | 3. The maturity roadmap |
| Evaluator design, LLM-as-judge, judge sampling, online and offline evaluation | 5. Evaluator design (and chapter 8 for replay and CI regression gating) |
| Tools landscape and component licenses | 6. The tools landscape |
| Cost of observability | 7. Operating the platform, section 7.2 |
| Anti-patterns | 9. Anti-patterns and common pitfalls |
Other topics live elsewhere and the guide links to them. The drift taxonomy (data, concept, pipeline) is in module 3 of the labs. The OpenTelemetry GenAI conventions (versions, renaming history, content capture, MCP) are in the production article. Quality SLOs, the cost of non-observability, the AI Act and the RACI are in the GenAI method.
What next
Section titled “What next”To practice, LLM observability: the labs puts chapters 4 and 5 to work on a self-hosted stack (instrumentation, RAG, drift, cost).
To go to production, the article Observing an LLM system gives the current OpenTelemetry GenAI conventions and the deployment playbooks (Collector, sampling, RAG, MCP, privacy). It continues with VictoriaMetrics as an LLM observability backend.
To go deeper into method, the GenAI method covers quality SLOs, quantified impact, compliance and organization.
To step beyond the LLM, Beyond the LLM: GPUs and model quality covers GPU infrastructure, classic ML and statistical drift.
Revised on 4 October 2026: title “Understanding generative AI observability”, explicit scope (LLM, RAG, agents, MCP), progression box, table of the topics this guide is the reference for, links to the labs, the production article and the method.
Revised on 25 October 2026: overview rewritten around an illustrative scenario, domain terms explained where they first appear.