TechnicalPractitioner3 min read (+1 technical)
What happens during a single request?
Three OpenTelemetry traces to walk step by step: an AI assistant working normally, the same assistant failing silently and a slowed-down online payment.
For: engineers and SREsPrerequisites: Basic notions of distributed traces.
An illustrative scenario: one Monday, the support team’s AI assistant answers faster than usual and costs less. The dashboard shows no errors. On Tuesday, the first customers complain about answers that miss the point. The dashboard said all was well. It did not say what was going on inside each request.
That is what a trace is for. Think of parcel tracking: leaving the warehouse, passing through the sorting center, loaded onto the truck, delivered, with the time of each stop and any incident along the way. A trace does the same for a request. It splits the request into nested steps called spans, each with its duration, its status (succeeded or failed) and attributes, meaning attached details (the configuration version, the number of documents consulted, the number of words the model processed). A dashboard says a request is slow or failing. The trace says where and why.
The three traces below are hand-built. Two involve an assistant that searches a document base before answering (a RAG agent, for retrieval-augmented generation): one working normally, the other failing silently. The third follows a checkout on an online shop, to show that the reading method is the same with or without AI.
For technical teams: conventions followed and exceptions
The traces follow the OpenTelemetry semantic conventions as of 2 October 2026: GenAI conventions from the semantic-conventions-genai repository, still at Development status, plus MCP conventions for protocol version 2025-06-18. MCP version 2026-07-28 removes sessions; these traces keep 2025-06-18, the version used in the convention’s example. One deliberate exception: the evaluate spans are an application choice outside the conventions; the scores themselves are carried by the gen_ai.evaluation.result event the conventions define.
Red outline: span in error. Amber outline: evaluation below its threshold. "key" marker in the details: attribute worth reading.
What this trace tells us
Hand-built example traces in a simplified format (times in milliseconds relative to the start of the trace). Attributes follow the OpenTelemetry semantic conventions; the generative AI, MCP and evaluation conventions are still in development and may change. Attributes prefixed with app. are application-specific. The "evaluate" spans are an application choice outside the conventions: the conventions define no evaluation span, they carry scores in the gen_ai.evaluation.result event, to be attached preferably to the span of the evaluated operation, otherwise through gen_ai.response.id; here it is placed in the application span.
Try this
Section titled “Try this”- Read the baseline, the trace of normal operation. Find the two
chatcalls to the model: they account for most of the duration. Compare their input token count (gen_ai.usage.input_tokens, the word fragments sent to the model) with that of the embeddings call, the step that turns the question into a vector to search the base. - Find the silent failure. Switch to the “silent failure” trace. No error, HTTP 200, a faster request. Only the faithfulness evaluation, which scores whether the answer sticks to the documents, is flagged “low score”. Open it: its
gen_ai.evaluation.resultevent carries a score of 0.34. Then clickquery kb_support_fr, theninvoke_agent. Theapp.retrieval.top_kattribute, the number of passages retrieved from the base, is 1 and theapp.config.loadedevent names the faulty configuration version. The assistant was reading a single passage before answering. - Compare tokens. The “LLM tokens” tile drops from about 9,000 to 2,600. On a cost dashboard, the incident looks like a successful optimization. That is the Monday of the scenario.
- Step outside AI. On the online shop trace, use the keyboard (up and down arrows) to reach the failed step. A 3 s timeout, then a retry (
http.request.resend_count), explain the 4.3 s the customer waited.
What to remember
Section titled “What to remember”Time reads top to bottom; the cause sits in the attributes. The waterfall shows where time goes, while attributes and events say why. A trace without business details, such as the configuration version, the number of passages retrieved or the key that prevents paying twice, cannot support a conclusion.
An “OK” status does not guarantee a good answer. For a system that generates text, I recommend attaching the quality evaluation to the trace itself. Without it, the silent failure stays invisible until the complaints arrive.
Finally, watch out for look-alike names. Two attributes can have almost the same name yet mean different things: that is the case here, hence the choice of an application-specific attribute.
For technical teams: evaluation and homonyms
The conventions provide the gen_ai.evaluation.result event, attached to the evaluated operation or, failing that, to its answer through gen_ai.response.id. Here, each event sits in an application evaluation span, which makes the evaluation time visible in the waterfall.
gen_ai.request.top_k refers to model sampling, not to the number of retrieved passages: that is why the trace uses an application attribute, app.retrieval.top_k. The idempotency key is the business attribute that stops a retried payment from being charged twice.
Go further: GenAI conventions in the LLM observability training, the three drifts of a RAG system in the dedicated module, quality SLOs in the GenAI method and the platform that carries these traces in the clickable architecture.
Revised on 2 October 2026: evaluation scores move to the conventions’ gen_ai.evaluation.result event, evaluation spans are flagged as an application choice, while each trace states its build date and the convention versions it follows.