Skip to content

TechnicalExpert10 min read

Part I. The framework

Why a generative AI application fails without making a sound, the four-dimension framework behind the method, the five signals and their storage, the OpenTelemetry GenAI baseline and the content capture rule.

For: engineers and SREs · architectsPrerequisites: Basic notions of metrics, traces and OpenTelemetry.

Take an illustrative scenario. A company launches an assistant that answers employees’ questions about leave and health cover. One Monday, the team swaps the document base for a reorganized version. Nothing breaks: every request gets an answer in under two seconds, the error rate stays at zero and the dashboards are green. Three weeks later, HR notices that employees are phoning them more often than before. The assistant still answers, only it answers wrong: it quotes the old company agreement or makes up a rule that does not exist.

Classical observability was built for failures that make noise, like a fire alarm: an error page (the HTTP 500 code), a timeout, a crashed program. Systems built around an LLM (a large language model, the engine that generates the text) fail differently. Their three most common failures set off no alarm at all.

Quality degrades with no error raised. The model hallucinates (it states something false with full confidence), gives half an answer or drifts slowly over the weeks.

Cost drifts with no alert. The prompt, the text sent to the model, grows with every release. An agent, a program that chains calls to the model and to tools on its own, gets stuck in a loop and burns tokens, the unit providers bill text in.

Security is compromised at the level of meaning rather than the network. A poisoned tool (tool poisoning) hides instructions in its own description; a prompt injection hijacks the model with malicious text. No firewall sees an attack go by.

The silent failure model fits in one sentence. For any signal you do not measure, the right question is not “is the service down?” but “how long does the failure pile up before anyone sees it?”. In the scenario, the answer is three weeks. In my view, a GenAI observability setup should be judged on how well it makes that silence visible. The rest of the method follows from this idea; Part V puts a figure on what the delay costs.

To decide what to measure, the method files every signal along four dimensions. Think of them as the columns of a form you fill in for each measurement: where the signal comes from, what it is for, which questions it must answer before you keep it, at what maturity stage it starts to pay off.

flowchart TB
  subgraph D1["Dimension 1: the 4 layers (where the signal occurs)"]
    L1["Infrastructure: GPU, network, vector store, MCP server"]
    L2["Model / inference: latency, tokens, saturation"]
    L3["Application: RAG, prompt, orchestration, guardrails"]
    L4["Agent / task: trajectory, tools, end-to-end success"]
    L1 --> L2 --> L3 --> L4
  end
  subgraph D2["Dimension 2: the 4 axes (why you monitor)"]
    A1["Reliability"]
    A2["Availability"]
    A3["Cost"]
    A4["Compliance"]
  end
  subgraph D3["Dimension 3: the 5 questions (per signal)"]
    Q["What -> Why -> How -> Threshold -> Impact if absent"]
  end
  subgraph D4["Dimension 4: maturity (the guide's levels 0 to 5)"]
    M["0. Blind -> 1. Basic signals -> 2. Tracing -> 3. Online evaluation -> 4. Drift and feedback -> 5. Closed loop"]
  end
  D1 --> D2 --> D3 --> D4

Every signal belongs to a layer, from infrastructure (graphics cards or GPUs, the network, the vector store that finds documents by meaning, the MCP server that exposes tools to the model) up to the task the agent has to complete.

The layers behave like the floors of a building where a leak on the second floor ends up staining the ceiling on the fourth. That is the cascade rule: a failure low down surfaces as a symptom higher up. In the example below, the provider caps the number of allowed calls (a rate limit) at 10:47. The code retries, latency rises, then the task success rate collapses. To go back down from symptom to cause you need a thread running through every layer: the trace_id, the unique identifier that follows the request from end to end. Without it, you see symptoms with no cause.

flowchart LR
  C2["Layer 2: provider rate-limit at 10:47"] --> C3["Layer 3: retries, prompt latency"] --> C4["Layer 4: task success rate dropping"]
  C4 -.->|"downward diagnosis via trace_id"| C2

Elsewhere, the site describes the stack of a generative AI application in five layers, in AI in cross-section: data, infrastructure, model, orchestration, application. The method’s four layers do not line up exactly with them. The word “application” in particular means different things on each side. The table gives the mapping.

Method layerAI in cross-section layer(s)
1. Infrastructure: GPU, network, vector store, MCP serverinfrastructure (GPUs, network, servers); the indexed content of the vector store belongs to data; the tools exposed by an MCP server are called from orchestration
2. Model / inference: latency, tokens, saturationmodel, at its boundary with infrastructure: TTFT, tokens per second and saturation are measured at the inference server, which the diagram places in infrastructure
3. Application: RAG, prompt, orchestration, guardrailsorchestration (prompts, document retrieval); output guardrails also touch the application layer (policy violations, personal data in outputs)
4. Agent / task: trajectory, tools, end-to-end successorchestration for the trajectory and tool calls; application for the success the user perceives
no dedicated layerdata: source freshness and coverage, input distribution, which the method handles in the RAG grid of Part III

A few acronyms from the table: TTFT (time to first token) is the delay before the first word of the answer appears. RAG (retrieval-augmented generation) means fetching relevant documents before asking the model to answer. Guardrails are the filters that block a forbidden input or output.

The axes say why you monitor: reliability (is the answer right?), availability (does the service respond?), cost and compliance with internal rules and regulation. The grids of Part III read each component along these four axes.

Before adding a signal, you give it a short job interview with five questions. I recommend never instrumenting anything without an answer to “why” and “impact if absent”: a signal that informs no decision is just noise on the bill.

QuestionWhat it establishes
WhatThe precise signal: metric, trace, score, event
WhyThe decision it informs. A signal with no decision is noise
HowThe source and the instrument: OTel attribute, exporter, evaluator
ThresholdThe SLI and the SLO, hence the alert trigger
Impact if absentThe failure that accumulates without this signal and its severity

The SLI (service level indicator) is the measurement you track, for example the share of answers judged correct. The SLO (service level objective) is the target you commit to on that measurement, for example 95%. Part IV is devoted to them.

Maturity works like belts in a martial art: you do not jump from white to black. The method used to have its own five-level scale (Logging, Tracing, Evaluation, Monitoring, Closed loop). The site keeps only one, that of the guide, chapter 3, with six levels from 0 to 5. The table gives the correspondence.

Guide levelWhat characterizes itFormer method level
0. Blind operationHTTP metrics only; cost discovered on the bill, quality through complaintsnone: the method’s scale started at logging
1. Basic signalsstructured per-request logs (prompt, response, model, tokens, latency), aggregated cost, errors as metrics1. Logging
2. Structured tracingOpenTelemetry GenAI conventions, one span per logical step, queryable attributes2. Tracing
3. Online evaluationautomated evaluations attached to traces and exposed as metrics, alertable quality regressions3. Evaluation
4. Drift and feedback integrationembedding drift, batch RAG quality, user feedback correlated with traces4. Monitoring (approximate match: the method did not detail this level)
5. Closed loopfailing traces become regression cases, changes gated on the regression set5. Closed loop

The quality SLOs of Part IV assume at least level 3. Without online evaluation, meaning automated scoring of a share of production answers, there is simply no sample to measure.

In my view, many organizations stop at level 2 and believe they observe their system. They see what happened, not whether it was any good. The HR assistant in the scenario was at level 2. The gain comes at level 3; mastery comes at level 5.

Signals are the five ways a system tells you what it is doing. They are those of the guide, chapter 1: logs (one line per event), metrics (numbers sampled at regular intervals), traces (the path of one request step by step), evaluations (a score given to an answer) and user feedback (the thumbs up or down). The trace ID ties them together. The method adds the typical storage of its reference configuration.

SignalTypical storage
LogsVictoriaLogs, Loki
MetricsVictoriaMetrics, Prometheus
TracesOTLP backend (Tempo, Jaeger)
EvaluationsLangfuse, Arize Phoenix
User feedbackattached to the trace by its ID, next to the evaluation scores

The first version of the method also listed “content events” (the prompt, the tool call, its result). They are not a separate signal: they belong to logs and traces, under the content capture rule described below.

OpenTelemetry is the open standard that defines how an application describes what it does, with the same names whichever tool stores the data. Its GenAI semantic conventions act as a shared dictionary for model calls: the same attribute means the same thing for every provider. I recommend adopting them as the instrumentation baseline, bearing in mind that they are still in Development status and may change.

Their versions, their history (including the move to a dedicated repository since 1.42), their attributes and content capture are covered only once on the site, in the article Observing an LLM system, §2. The method relies mostly on gen_ai.request.model and gen_ai.response.model, gen_ai.provider.name, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (the basis of cost), gen_ai.response.finish_reasons (why the model stopped, a first-order reliability signal) and gen_ai.operation.name. Its recording rules (Part II) start from the metrics of versions 1.40 and 1.41 (gen_ai.client.operation.duration, gen_ai.client.token.usage): the revision note at the bottom of the page says what changes after that.

That leaves the question of content: should you record what the user wrote and what the model answered? It helps you understand a bad answer, though it also means storing data that is sometimes personal. The specification settles the default: instrumentations capture neither instructions nor inputs nor outputs. Capturing them is an explicit choice (an opt-in). It describes three patterns.

The first is to record nothing, which is the default. The second is to record the content on spans (gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages), reserved for cases where volume stays manageable and storage complies with regulation, for example in pre-production. The third is to store the content elsewhere and put only a reference to it on the span; this is the pattern the specification recommends in production. In a regulated environment the method retains the third, with sensitive data masked at the collector. In every case, content never goes into a metric label (see the guide, 9.6).

Take your application and look for the longest failure you might be missing today: a quality drop, a cost drift or a hijack through injection. Place it on one of the four layers, then run it through the five questions. If “impact if absent” worries you, that is the first signal to instrument.


Revised on 2 October 2026: since version 1.42 (June 2026), the GenAI conventions live in the semantic-conventions-genai repository. At that date, its main branch, whose changes are not yet published in a release, lists gen_ai.client.operation.duration as recommended rather than required. It also replaces gen_ai.client.token.usage with gen_ai.client.inference.usage.* counters (input, output, cache and reasoning tokens) and gen_ai.client.inference.operation.* histograms. Check which version your instrumentation emits before writing your rules.

Revised on 4 October 2026: signals aligned on the guide’s five (content events belong to logs and traces), correspondence table to the five layers of AI in cross-section, maturity scale replaced by the guide’s levels 0 to 5, OpenTelemetry conventions recap reduced to a summary with a pointer to the article, content capture rule aligned with the specification.

Revised on 25 October 2026: page rewritten for a wider audience (opening scenario, terms explained), with no change of substance.