TechnicalPractitioner6 min read
3. The maturity roadmap
Six levels of AI observability maturity, from blind operation to the closed loop, with the telltale signs, the gains and the requirements of each.
For: engineers and SREs · architects · team managers · business and productPrerequisites: Have read chapter 1 of the guide.
On the day the complaints arrived, the insurer’s team (still in our illustrative scenario) tried to find the faulty answer. It could not: nothing was recorded beyond HTTP status codes. It wanted to know how long the problem had been going on: there was no quality measurement. It wanted to know what the assistant cost: that meant waiting for the provider’s invoice.
That team was at Level 0. The roadmap below describes how to get out of it, one step at a time, a bit like learning to drive: first you learn to start the car, then to read the gauges, then to anticipate.
AI observability capability progresses through six levels, from 0 to 5. Each level builds on the previous one. An AI feature shipped without an explicit observability strategy stays at Level 0; a simple log of calls moves it to Level 1. In my view, Level 3 is the realistic objective for a six-month program. Level 5 is rare: it marks an organization that treats observability as the engine of continuous improvement rather than as a passive monitoring layer.
This ladder is the reference for every piece on this site. The maturity grid of the GenAI method and the 30-, 60- and 90-day milestones of the labs are read against these levels.

Figure 5. The maturity ladder, from Level 0 (blind operation) to Level 5 (closed loop). Level 3 is, in my view, a realistic six-month objective.
3.1 Level 0: blind operation
Section titled “3.1 Level 0: blind operation”The system runs in production with no signal beyond HTTP-level metrics. Cost shows up on the end-of-month invoice and quality through user complaints. This is the default state of an LLM feature shipped without an explicit observability strategy.
Indicators you are at Level 0
Section titled “Indicators you are at Level 0”You are probably here if you cannot say how many tokens were consumed yesterday, by feature or by tenant (a customer or entity sharing the platform). Or if you cannot retrieve the prompt and response behind a specific complaint. Other signs: you discover hallucinations only through user reports, you have no baseline to compare against when changing the model and your only available remediation is to roll back.
3.2 Level 1: basic signals
Section titled “3.2 Level 1: basic signals”A structured log per request records the prompt, the response, the model identity, the token counts and the latency. Cost is computed and aggregated. Errors are tracked as metrics.
What you gain
Section titled “What you gain”You can attribute cost by feature, user, tenant or environment. You can set latency budgets per model. After an incident, you have a searchable history of past requests.
What you still lack
Section titled “What you still lack”Multi-step execution stays opaque: a RAG pipeline or an agent looks like a single LLM call. You have no quality signal beyond user feedback, no drift detection and no way to gate a change on its quality.
Many teams stop here and plateau, because logs alone produce no improvement. The move to Level 2 is the one that unlocks everything else.
3.3 Level 2: structured tracing
Section titled “3.3 Level 2: structured tracing”The team adopts the OpenTelemetry GenAI semantic conventions. OpenTelemetry is the open standard for collecting telemetry; its semantic conventions fix attribute names, so that everyone names “model” or “input tokens” the same way. Each request emits a trace tree with one span per logical step. Spans can be queried by their attributes, including model, user, feature and tenant.
What you gain
Section titled “What you gain”RAG and agent flows become visible as trace trees. Latency breaks down step by step. Attributes are the same from one provider to the next, which avoids vendor lock-in at the data layer. And all of it fits into the OpenTelemetry infrastructure you may already run.
Required components
Section titled “Required components”First, OTel-compatible instrumentation: the native OTel SDKs (the official libraries you add to your code), OpenLLMetry, or OpenInference, which exports over OTLP, OpenTelemetry’s transport protocol, but with its own attributes (see chapter 6). Next, an OpenTelemetry Collector, the relay that receives, transforms and routes the data. Finally, a trace backend, the database that stores and serves traces, such as Tempo, Phoenix, SigNoz or Jaeger.
3.4 Level 3: online evaluation
Section titled “3.4 Level 3: online evaluation”Each production trace triggers automated evaluations. Their results attach as attributes to the originating trace and are exposed as metrics for aggregation. Quality becomes a measurement rather than a guess.
Common evaluators at this level
Section titled “Common evaluators at this level”Five evaluators come up often at this stage. Faithfulness checks that the answer rests on the provided context. Answer relevancy checks that it addresses the question. Format compliance checks that the output matches the expected schema. Toxicity detects harmful content. PII leakage spots sensitive personal data in the output.
What you gain
Section titled “What you gain”You get a continuous quality signal that no longer depends on user feedback. A quality drop can raise an alert before users notice it. You also have a baseline for comparing model and prompt changes. In our scenario, the insurer’s assistant would have seen its faithfulness fall on the day the policy was updated.
3.5 Level 4: drift and feedback integration
Section titled “3.5 Level 4: drift and feedback integration”Level 4 adds three capabilities to a Level 3 platform.
The first is embedding drift monitoring. Recall that an embedding turns a text into numbers that reflect its meaning. If the distribution of those numbers shifts over time, the questions or answers are changing in nature. It is measured with KL divergence, Wasserstein distance or, more simply, population stability indices (PSI): three ways of putting a figure on the gap between two distributions.
The second is RAG quality tracking: retrieval precision and recall are measured in batch against a maintained gold set.
The third is user feedback integration: thumbs up or down, edits to the answer, retries, conversions, all tied to trace IDs.
What you gain
Section titled “What you gain”You get early warning when the input distribution shifts: new use cases, attempts to get around the model’s safeguards (jailbreaks), language changes. You get a reliable signal of real-world usefulness, not just rubric-based quality. You have data to tune retrievers, rerankers and prompt templates. Above all, you can compare automated scores with user satisfaction. To me that is the most useful signal for tuning your evaluators.
3.6 Level 5: closed loop
Section titled “3.6 Level 5: closed loop”At Level 5, evaluations and feedback feed back into the system itself. Observability becomes an engine of improvement rather than a passive monitor.
In practice, failed traces become regression test cases in continuous integration (the automated pipeline that tests every change before it ships). High-quality traces become examples placed in the prompt (few-shot) or fine-tuning data (a light retraining of the model on your data). Drift triggers automated re-indexing or model re-selection. Feedback trains preference models or task-specific scorers. And every model or prompt change has to pass the regression set.

Figure 6. The closed loop. Production traces feed evaluation and feedback, which feed curation, which feeds prompt and model iteration, which deploys back to production.
To know what to do right now given your level, the appendix suggests a next step and a timeframe for each one. The next chapter details how to climb to Level 3.
Next: 4. Implementation in seven steps.
Revised on 4 October 2026: 0 to 5 ladder presented as the site’s reference, with links to the method and the labs.
Revised on 25 October 2026: opening on the illustrative scenario, terms explained (OTLP, backend, PSI, few-shot, fine-tuning), lists written as prose, link to the appendix for next steps.