Skip to content

TechnicalPractitioner10 min read (+1 technical)

4. Implementation in seven steps

Seven ordered steps to move an AI system from Level 0 to a working Level 3 platform: questions, instrumentation, collection, storage, evaluation, visualization and the closed loop.

For: engineers and SREs · architectsPrerequisites: Have read chapters 1 to 3 of the guide; basic OpenTelemetry notions.

Illustrative scenario. Six months after launch, the insurer’s leadership asks a simple question: what does the assistant cost for each regional branch and are its answers equally good everywhere? The team had instrumented generously. Hundreds of attributes per span, dashboards full of charts. But nobody had thought of recording the regional branch on each request. The question goes unanswered.

This chapter describes seven ordered steps to move a system from Level 0 to a working Level 3 platform. The order matters. Skipping steps gives you a stack that emits data nobody queries.

Before any instrumentation, write down the questions the platform must answer. It is like preparing a survey: you draft the questionnaire before going into the field, otherwise you come back with answers to questions nobody asked. The instrumentation strategy follows from the questions, not the reverse.

The typical failure is the one in the scenario: you instrument everything, then realize that the attributes needed to answer the real questions were never captured. They are often high-cardinality attributes, meaning attributes that take many different values (a customer, branch or session identifier).

I sort the questions into three families. Here are examples for each.

  • What is the p95 and p99 latency of each pipeline stage, by tenant? (The p95 is the duration under which 95 percent of requests complete: it shows what the worst-served users experience.)
  • Which feature consumed the most tokens last week and how does that compare to the prior week?
  • How often does the agent enter a loop of more than ten steps and which tools are involved?
  • What is the error rate of each MCP server and is any single server dominating overall latency?
  • What is the faithfulness score of answers grounded in document set Z?
  • What is the trend of toxicity flags over the past 30 days?
  • Which prompt versions have produced regressions in the gold set?
  • Is there a drift in input language distribution that correlates with quality drops?
  • Show me the full prompt and response for the request that user X submitted at time T (which assumes content capture has been explicitly enabled, with controlled access: see step 2).
  • Show me all traces where the model output contained a credit card number pattern.
  • Show me all traces that called the billing.invoice_fetch tool with arguments containing tenant ID Y.

Set yourself an internal target for answering forensic questions, for example under one minute. No regulation imposes this figure; it is illustrative. But during an incident or in front of an auditor, forensic speed becomes a feature in its own right.

Instrumenting means placing sensors in the code so it reports what it does. I see three options, in order of preference.

The first is native OpenTelemetry GenAI instrumentation, when it exists. The SDKs are still maturing but adopting them is, in my view, the right long-term bet.

The second uses auto-instrumentation libraries such as OpenInference (Arize) or OpenLLMetry (Traceloop). They wrap common client libraries (OpenAI SDK, Anthropic SDK, LangChain, LlamaIndex) and emit spans automatically, without you touching every call.

The third is creating spans by hand around business logic specific to your application, the kind no known framework covers.

For each LLM call, capture at least the model name, input and output tokens, latency and error status. Prompt and completion follow the content capture rule explained below. Other useful attributes: temperature, top-p, system prompt hash, tool definitions, request ID, tenant ID, prompt version, retriever version. A hash is a code computed from a text: two identical texts give the same hash, which lets you spot a change without storing the text.

OpenTelemetry GenAI semantic conventions: the core

Section titled “OpenTelemetry GenAI semantic conventions: the core”

The GenAI conventions fix attribute names so that every tool speaks the same language: the requested model is gen_ai.request.model everywhere, input tokens are gen_ai.usage.input_tokens. The core table and version details are in the collapsible section below.

These conventions are still in development status. They have been maintained in a dedicated repository since June 2026 and several attributes have already been renamed (for example gen_ai.system, the name used in the first version of this guide, now gen_ai.provider.name). The full attribute list, versions and renaming history are kept up to date in the production article, section 2: that is the site’s reference on this point.

For technical teams: the core attribute table
AttributeTypePurpose
gen_ai.operation.namestringOperation type (chat, embeddings, retrieval, execute_tool, invoke_agent, etc.)
gen_ai.provider.namestringProvider identity (openai, anthropic, mistral_ai, etc.)
gen_ai.request.modelstringRequested model name
gen_ai.response.modelstringModel actually used (may differ from request)
gen_ai.usage.input_tokensintInput token count
gen_ai.usage.output_tokensintOutput token count
gen_ai.response.finish_reasonsstring[]Why generation ended (stop, length, content_filter, tool_call, error)
gen_ai.input.messages, gen_ai.output.messagesstructured, opt-inPrompt and completion content (see the capture rule below)

Prompts and responses often contain personal data: the member’s name, policy number, sometimes health details. Hence a single rule across the site: content capture is off by default and only enabled explicitly (opt-in).

This is also what the conventions say. gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions have the “Opt-In” requirement level and instrumentations should not capture them by default. For production, the specification recommends storing content outside traces with its own access controls, keeping only references on the spans. Recording it in span attributes is better suited to pre-production, or to cases where trace storage complies with privacy rules. When content does flow through traces, it is masked in the Collector before storage (step 3). Enablement flags, events and Collector rules are detailed in the production article (specification).

Your custom attributes should carry a prefix specific to your organization, so they never collide with future conventions. Example: acme.tenant_id, acme.prompt_version, acme.retriever.index_name.

For technical teams: example of a manual span for a RAG retrieval

By default this pseudocode records the length and hash of the query, never its text. The text is only added, as an event, when content capture has been enabled.

# Python pseudocode using opentelemetry.trace and hashlib
# CAPTURE_CONTENT: configuration flag, False by default
def search(rewritten_query, tenant_id):
with tracer.start_as_current_span("retriever.search") as span:
span.set_attribute("acme.retriever.store", "qdrant")
span.set_attribute("acme.retriever.top_k", 8)
# By default: query length and digest, never the text
span.set_attribute("acme.retriever.query_length", len(rewritten_query))
span.set_attribute("acme.retriever.query_sha256",
hashlib.sha256(rewritten_query.encode()).hexdigest())
if CAPTURE_CONTENT: # explicit opt-in, off by default
# text as an event, outside indexed attributes, masked in the Collector
span.add_event("acme.retriever.query", {"text": rewritten_query})
span.set_attribute("acme.tenant_id", tenant_id)
span.set_attribute("acme.retriever.index_name", "kb-prod-v3")
chunks = vector_store.search(rewritten_query, k=8)
span.set_attribute("acme.retriever.hits", len(chunks))
if chunks:
span.set_attribute("acme.retriever.top_score", chunks[0].score)
span.add_event("chunks", {"ids": [c.id for c in chunks]})
return chunks

Run an OpenTelemetry Collector between the application and the backend (the database that stores the data). The Collector works like a postal sorting center: it receives every shipment, sorts it, removes what must not leave and sends the rest to the right destinations.

It handles concerns that do not belong in the application: batching, retries on failure, back-pressure (slowing down when storage is saturated), attribute enrichment, filtering, sampling, redaction (masking sensitive data) and fan-out to several backends at once.

OpenTelemetry Collector pipeline: receivers, processors, exporters, three parallel pipelines for traces, metrics and logs

Figure 7. OpenTelemetry Collector pipeline. Receivers ingest data, processors transform it, exporters route to backends. Three pipelines run in parallel for traces, metrics and logs.

Six processors do most of the work. One protects the Collector from running out of memory, one redacts, two enrich the spans, one samples and the last batches before sending. Their exact role is in the collapsible section.

For technical teams: the six processors and their parameters
  • memory_limiter: protects the collector from OOM under traffic spikes.
  • redaction: attribute scrubbing before storage. It deletes attributes missing from an allow list of keys (allowed_keys), then masks or hashes values matching blocked regular expressions (blocked_values). It does not perform named entity recognition (NER): NER, like LLM-based scrubbing, requires an external component or a custom processor (processor documentation).
  • attributes: inject environment, region, tenant and deployment version uniformly.
  • tail_sampling: policy-based decision after the trace completes.
  • batch: bundle spans for transmission efficiency. Goes last in the processor chain.
  • resource: inject service identity and version into every span.

Each signal type has its own storage, the way a library shelves books, journals and archives separately while keeping one shared catalog. Here the shared catalog is the trace ID: a coherent stack picks one backend per signal and makes sure they share that identifier, so you can move from one to another.

SignalSelf-hostable softwareManaged services (SaaS)
MetricsGrafana Mimir, Prometheus, Thanos, VictoriaMetricsChronosphere, Datadog, Grafana Cloud, New Relic
TracesGrafana Tempo, Jaeger, Phoenix (ELv2, source-available), SigNozArize AX, Datadog APM, Grafana Cloud, Honeycomb
LogsElasticsearch, Grafana Loki, OpenObserve, VictoriaLogsDatadog, Elastic Cloud, Grafana Cloud, Splunk
EvaluationsAttached to traces, aggregated as metricsSame: this is convention, not product
FeedbackEvent store keyed by trace IDSame: custom-built
Cold archiveS3-compatible object storeS3, Azure Blob, GCS

Options are listed alphabetically; chapter 6 details licenses and selection criteria. Three constraints drive storage choices: retention policy, query speed on large prompt payloads and data residency requirements. When a residency requirement applies (internal policy, sectoral regulator or contract), it often determines the stack.

Online evaluation can be triggered in two ways.

In synchronous (inline) mode, each request triggers an evaluation before the response goes out. That adds latency. I keep this mode for cases where the result must decide the response, for example a toxicity filter or a refusal classifier: if the content is judged dangerous, the response is not sent.

In asynchronous mode, a background job picks traces and evaluates them off the request path. The user does not wait. This is the default mode.

Online evaluation on live traces versus offline evaluation on fixed datasets

Figure 8. Online evaluation runs against live production traces. Offline evaluation runs against fixed datasets in CI or scheduled batches. Both are necessary: online for drift detection, offline for regression gating.

Chapter 5 covers evaluator design in depth: LLM-as-judge prompt patterns, calibration and score aggregation.

Build dashboards that answer the Step 1 questions, not dashboards that show everything available. Group them by audience and concern: finance does not look at the same charts as the on-call team.

  • Cost and usage: tokens per feature, per tenant, per model, with anomaly bands.
  • Latency and reliability: p50, p95, p99 per pipeline stage, error rate by failure mode.
  • Quality and safety: evaluator scores over time, with regression alerts on quality drops.
  • Drift and distribution: input length, language, topic distribution and embedding drift.
  • Agent behavior: iteration histograms, tool selection frequency, loop indicators.

A dashboard is for diagnosis; an alert wakes someone up. The conditions below are examples. An SLO (service level objective) is a numeric service target, such as “95 percent of answers in under two seconds”.

  • Quality regression beyond a threshold (for example, faithfulness drops more than two standard deviations).
  • Cost spike beyond a baseline multiple (for example, tokens per minute exceeds 3x the trailing seven-day median).
  • Latency degradation (p95 exceeds SLO for more than five minutes).
  • Detection of PII or credentials in outputs (any non-zero count over a one-minute window).
  • Agent loop indicator (more than ten iterations in any trace, consistent with the Step 1 question, or any iteration count above a per-feature ceiling).

Without a regular review, data piles up and nothing improves. I recommend a short weekly meeting with four topics.

You go through failed traces and low-score traces, then add the most telling ones to the regression set. You also look at high-score traces to pull examples to include in prompts (few-shot). You check that every model or prompt change passes the regression set: no merge if an evaluator score regresses. Finally, you review user feedback trends and turn them into actions on retrieval, prompts or model choice.

This loop is what distinguishes an observability platform from a logging system. Without it, you have an expensive read-only archive.

For Monday morning, Step 1 is enough. Get platform, product and a business representative in a room for an hour. Write your first questions together.

Next: 5. Evaluator design.

Revised on 2 October 2026: GenAI conventions moved to the semantic-conventions-genai repository (v1.42.0), content attribute history and finish_reasons values corrected, actual behavior of the redaction processor, position of the batch processor and exporter-side batching, neutral storage table columns.

Revised on 4 October 2026: conventions table reduced to the core, renaming history replaced by a link to the production article, content capture rule aligned with the specification (off by default, explicit opt-in, separate storage recommended in production).

Revised on 25 October 2026: opening on an illustrative scenario, terms explained (cardinality, p95, hash, redaction, SLO), attribute table, code example and processor list moved into collapsible sections.