TechnicalPractitioner10 min read (+1 technical)
4. Implementation in seven steps
Seven ordered steps to move an AI system from Level 0 to a working Level 3 platform: questions, instrumentation, collection, storage, evaluation, visualization and the closed loop.
For: engineers and SREs · architectsPrerequisites: Have read chapters 1 to 3 of the guide; basic OpenTelemetry notions.
Illustrative scenario. Six months after launch, the insurer’s leadership asks a simple question: what does the assistant cost for each regional branch and are its answers equally good everywhere? The team had instrumented generously. Hundreds of attributes per span, dashboards full of charts. But nobody had thought of recording the regional branch on each request. The question goes unanswered.
This chapter describes seven ordered steps to move a system from Level 0 to a working Level 3 platform. The order matters. Skipping steps gives you a stack that emits data nobody queries.
Step 1: Define the questions first
Section titled “Step 1: Define the questions first”Before any instrumentation, write down the questions the platform must answer. It is like preparing a survey: you draft the questionnaire before going into the field, otherwise you come back with answers to questions nobody asked. The instrumentation strategy follows from the questions, not the reverse.
The typical failure is the one in the scenario: you instrument everything, then realize that the attributes needed to answer the real questions were never captured. They are often high-cardinality attributes, meaning attributes that take many different values (a customer, branch or session identifier).
I sort the questions into three families. Here are examples for each.
Operational questions
Section titled “Operational questions”- What is the p95 and p99 latency of each pipeline stage, by tenant? (The p95 is the duration under which 95 percent of requests complete: it shows what the worst-served users experience.)
- Which feature consumed the most tokens last week and how does that compare to the prior week?
- How often does the agent enter a loop of more than ten steps and which tools are involved?
- What is the error rate of each MCP server and is any single server dominating overall latency?
Quality questions
Section titled “Quality questions”- What is the faithfulness score of answers grounded in document set Z?
- What is the trend of toxicity flags over the past 30 days?
- Which prompt versions have produced regressions in the gold set?
- Is there a drift in input language distribution that correlates with quality drops?
Forensic questions
Section titled “Forensic questions”- Show me the full prompt and response for the request that user X submitted at time T (which assumes content capture has been explicitly enabled, with controlled access: see step 2).
- Show me all traces where the model output contained a credit card number pattern.
- Show me all traces that called the
billing.invoice_fetchtool with arguments containing tenant ID Y.
Set yourself an internal target for answering forensic questions, for example under one minute. No regulation imposes this figure; it is illustrative. But during an incident or in front of an auditor, forensic speed becomes a feature in its own right.
Step 2: Instrument
Section titled “Step 2: Instrument”Instrumenting means placing sensors in the code so it reports what it does. I see three options, in order of preference.
The first is native OpenTelemetry GenAI instrumentation, when it exists. The SDKs are still maturing but adopting them is, in my view, the right long-term bet.
The second uses auto-instrumentation libraries such as OpenInference (Arize) or OpenLLMetry (Traceloop). They wrap common client libraries (OpenAI SDK, Anthropic SDK, LangChain, LlamaIndex) and emit spans automatically, without you touching every call.
The third is creating spans by hand around business logic specific to your application, the kind no known framework covers.
For each LLM call, capture at least the model name, input and output tokens, latency and error status. Prompt and completion follow the content capture rule explained below. Other useful attributes: temperature, top-p, system prompt hash, tool definitions, request ID, tenant ID, prompt version, retriever version. A hash is a code computed from a text: two identical texts give the same hash, which lets you spot a change without storing the text.
OpenTelemetry GenAI semantic conventions: the core
Section titled “OpenTelemetry GenAI semantic conventions: the core”The GenAI conventions fix attribute names so that every tool speaks the same language: the requested model is gen_ai.request.model everywhere, input tokens are gen_ai.usage.input_tokens. The core table and version details are in the collapsible section below.
These conventions are still in development status. They have been maintained in a dedicated repository since June 2026 and several attributes have already been renamed (for example gen_ai.system, the name used in the first version of this guide, now gen_ai.provider.name). The full attribute list, versions and renaming history are kept up to date in the production article, section 2: that is the site’s reference on this point.
For technical teams: the core attribute table
| Attribute | Type | Purpose |
|---|---|---|
gen_ai.operation.name | string | Operation type (chat, embeddings, retrieval, execute_tool, invoke_agent, etc.) |
gen_ai.provider.name | string | Provider identity (openai, anthropic, mistral_ai, etc.) |
gen_ai.request.model | string | Requested model name |
gen_ai.response.model | string | Model actually used (may differ from request) |
gen_ai.usage.input_tokens | int | Input token count |
gen_ai.usage.output_tokens | int | Output token count |
gen_ai.response.finish_reasons | string[] | Why generation ended (stop, length, content_filter, tool_call, error) |
gen_ai.input.messages, gen_ai.output.messages | structured, opt-in | Prompt and completion content (see the capture rule below) |
Capturing prompt and response content
Section titled “Capturing prompt and response content”Prompts and responses often contain personal data: the member’s name, policy number, sometimes health details. Hence a single rule across the site: content capture is off by default and only enabled explicitly (opt-in).
This is also what the conventions say. gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions have the “Opt-In” requirement level and instrumentations should not capture them by default. For production, the specification recommends storing content outside traces with its own access controls, keeping only references on the spans. Recording it in span attributes is better suited to pre-production, or to cases where trace storage complies with privacy rules. When content does flow through traces, it is masked in the Collector before storage (step 3). Enablement flags, events and Collector rules are detailed in the production article (specification).
Your custom attributes should carry a prefix specific to your organization, so they never collide with future conventions. Example: acme.tenant_id, acme.prompt_version, acme.retriever.index_name.
For technical teams: example of a manual span for a RAG retrieval
By default this pseudocode records the length and hash of the query, never its text. The text is only added, as an event, when content capture has been enabled.
# Python pseudocode using opentelemetry.trace and hashlib# CAPTURE_CONTENT: configuration flag, False by defaultdef search(rewritten_query, tenant_id): with tracer.start_as_current_span("retriever.search") as span: span.set_attribute("acme.retriever.store", "qdrant") span.set_attribute("acme.retriever.top_k", 8) # By default: query length and digest, never the text span.set_attribute("acme.retriever.query_length", len(rewritten_query)) span.set_attribute("acme.retriever.query_sha256", hashlib.sha256(rewritten_query.encode()).hexdigest()) if CAPTURE_CONTENT: # explicit opt-in, off by default # text as an event, outside indexed attributes, masked in the Collector span.add_event("acme.retriever.query", {"text": rewritten_query}) span.set_attribute("acme.tenant_id", tenant_id) span.set_attribute("acme.retriever.index_name", "kb-prod-v3") chunks = vector_store.search(rewritten_query, k=8) span.set_attribute("acme.retriever.hits", len(chunks)) if chunks: span.set_attribute("acme.retriever.top_score", chunks[0].score) span.add_event("chunks", {"ids": [c.id for c in chunks]}) return chunksStep 3: Collect
Section titled “Step 3: Collect”Run an OpenTelemetry Collector between the application and the backend (the database that stores the data). The Collector works like a postal sorting center: it receives every shipment, sorts it, removes what must not leave and sends the rest to the right destinations.
It handles concerns that do not belong in the application: batching, retries on failure, back-pressure (slowing down when storage is saturated), attribute enrichment, filtering, sampling, redaction (masking sensitive data) and fan-out to several backends at once.

Figure 7. OpenTelemetry Collector pipeline. Receivers ingest data, processors transform it, exporters route to backends. Three pipelines run in parallel for traces, metrics and logs.
Critical processors for AI traces
Section titled “Critical processors for AI traces”Six processors do most of the work. One protects the Collector from running out of memory, one redacts, two enrich the spans, one samples and the last batches before sending. Their exact role is in the collapsible section.
For technical teams: the six processors and their parameters
memory_limiter: protects the collector from OOM under traffic spikes.redaction: attribute scrubbing before storage. It deletes attributes missing from an allow list of keys (allowed_keys), then masks or hashes values matching blocked regular expressions (blocked_values). It does not perform named entity recognition (NER): NER, like LLM-based scrubbing, requires an external component or a custom processor (processor documentation).attributes: inject environment, region, tenant and deployment version uniformly.tail_sampling: policy-based decision after the trace completes.batch: bundle spans for transmission efficiency. Goes last in the processor chain.resource: inject service identity and version into every span.
Step 4: Store
Section titled “Step 4: Store”Each signal type has its own storage, the way a library shelves books, journals and archives separately while keeping one shared catalog. Here the shared catalog is the trace ID: a coherent stack picks one backend per signal and makes sure they share that identifier, so you can move from one to another.
| Signal | Self-hostable software | Managed services (SaaS) |
|---|---|---|
| Metrics | Grafana Mimir, Prometheus, Thanos, VictoriaMetrics | Chronosphere, Datadog, Grafana Cloud, New Relic |
| Traces | Grafana Tempo, Jaeger, Phoenix (ELv2, source-available), SigNoz | Arize AX, Datadog APM, Grafana Cloud, Honeycomb |
| Logs | Elasticsearch, Grafana Loki, OpenObserve, VictoriaLogs | Datadog, Elastic Cloud, Grafana Cloud, Splunk |
| Evaluations | Attached to traces, aggregated as metrics | Same: this is convention, not product |
| Feedback | Event store keyed by trace ID | Same: custom-built |
| Cold archive | S3-compatible object store | S3, Azure Blob, GCS |
Options are listed alphabetically; chapter 6 details licenses and selection criteria. Three constraints drive storage choices: retention policy, query speed on large prompt payloads and data residency requirements. When a residency requirement applies (internal policy, sectoral regulator or contract), it often determines the stack.
Step 5: Evaluate
Section titled “Step 5: Evaluate”Online evaluation can be triggered in two ways.
In synchronous (inline) mode, each request triggers an evaluation before the response goes out. That adds latency. I keep this mode for cases where the result must decide the response, for example a toxicity filter or a refusal classifier: if the content is judged dangerous, the response is not sent.
In asynchronous mode, a background job picks traces and evaluates them off the request path. The user does not wait. This is the default mode.

Figure 8. Online evaluation runs against live production traces. Offline evaluation runs against fixed datasets in CI or scheduled batches. Both are necessary: online for drift detection, offline for regression gating.
Chapter 5 covers evaluator design in depth: LLM-as-judge prompt patterns, calibration and score aggregation.
Step 6: Visualize and alert
Section titled “Step 6: Visualize and alert”Build dashboards that answer the Step 1 questions, not dashboards that show everything available. Group them by audience and concern: finance does not look at the same charts as the on-call team.
Dashboard families
Section titled “Dashboard families”- Cost and usage: tokens per feature, per tenant, per model, with anomaly bands.
- Latency and reliability: p50, p95, p99 per pipeline stage, error rate by failure mode.
- Quality and safety: evaluator scores over time, with regression alerts on quality drops.
- Drift and distribution: input length, language, topic distribution and embedding drift.
- Agent behavior: iteration histograms, tool selection frequency, loop indicators.
Alert conditions
Section titled “Alert conditions”A dashboard is for diagnosis; an alert wakes someone up. The conditions below are examples. An SLO (service level objective) is a numeric service target, such as “95 percent of answers in under two seconds”.
- Quality regression beyond a threshold (for example, faithfulness drops more than two standard deviations).
- Cost spike beyond a baseline multiple (for example, tokens per minute exceeds 3x the trailing seven-day median).
- Latency degradation (p95 exceeds SLO for more than five minutes).
- Detection of PII or credentials in outputs (any non-zero count over a one-minute window).
- Agent loop indicator (more than ten iterations in any trace, consistent with the Step 1 question, or any iteration count above a per-feature ceiling).
Step 7: Close the loop
Section titled “Step 7: Close the loop”Without a regular review, data piles up and nothing improves. I recommend a short weekly meeting with four topics.
You go through failed traces and low-score traces, then add the most telling ones to the regression set. You also look at high-score traces to pull examples to include in prompts (few-shot). You check that every model or prompt change passes the regression set: no merge if an evaluator score regresses. Finally, you review user feedback trends and turn them into actions on retrieval, prompts or model choice.
This loop is what distinguishes an observability platform from a logging system. Without it, you have an expensive read-only archive.
For Monday morning, Step 1 is enough. Get platform, product and a business representative in a room for an hour. Write your first questions together.
Next: 5. Evaluator design.
Revised on 2 October 2026: GenAI conventions moved to the semantic-conventions-genai repository (v1.42.0), content attribute history and finish_reasons values corrected, actual behavior of the redaction processor, position of the batch processor and exporter-side batching, neutral storage table columns.
Revised on 4 October 2026: conventions table reduced to the core, renaming history replaced by a link to the production article, content capture rule aligned with the specification (off by default, explicit opt-in, separate storage recommended in production).
Revised on 25 October 2026: opening on an illustrative scenario, terms explained (cardinality, p95, hash, redaction, SLO), attribute table, code example and processor list moved into collapsible sections.