TechnicalPractitioner7 min read
1. Foundations
What AI observability is, why classical observability falls short, the five signals, the core vocabulary and what the discipline is not.
For: engineers and SREs · architects · business and productPrerequisites: A rough idea of what logs, metrics and traces are.
Back to the illustrative scenario from the overview. The insurer’s assistant answers in under two seconds, the error rate is zero and the provider’s API returns 200 OK on every call. Yet some of the answers about coverage are wrong. No light comes on, because no light measures whether an answer is right.
This chapter lays the groundwork for seeing that kind of problem: what AI observability means, why the usual tools miss it, which signals to collect and which words to use.
1.1 What AI observability is
Section titled “1.1 What AI observability is”AI observability means measuring, recording and interpreting the behavior of AI systems in production. It takes classical observability and adds signals specific to systems that do not always give the same answer and whose raw material is text.
Those signals have names you will meet everywhere. The prompt is the text sent to the model, instructions included. The completion is the text the model sends back. An embedding is a text translated into a list of numbers, so that two texts with similar meaning give similar lists: that is what makes it possible to search documents “by meaning”. Add to these the documents retrieved to answer (the retrieval context), tool calls and quality evaluations.
The core question changes. You no longer only ask whether the system is up and responding on time. The system can be up, fast and wrong. AI observability therefore adds a second axis: is the answer correct, faithful to its sources and safe?

Figure 1. The two axes of AI observability. Classical observability covers the horizontal axis. AI observability adds the vertical axis.
If your monitoring can tell you the system is up but not that the answers are right, you are in the bottom-right quadrant. That is exactly the failure AI observability exists to prevent. It is also where our assistant sits.
1.2 Why classical observability is insufficient
Section titled “1.2 Why classical observability is insufficient”Classical observability rests on a few unspoken assumptions: a program gives the same result for the same input, its exchanges have a stable shape, code that works today will work tomorrow. Five properties of AI systems break those assumptions.
The first is non-determinism. Ask the same question twice and you may get two different answers. You cannot assume a problem will reproduce identically.
The second is the nature of the exchanges. Requests and responses are natural language, not structured data with a fixed schema. A JSON field can be validated; a sentence has to be read. The shape of the payload is the answer itself.
The third is that quality only shows at runtime. No compiler guarantees an answer will be correct. A system that gave 100 perfect answers can invent a fact on the 101st without anyone touching the code. That kind of invented answer is called a hallucination.
The fourth is drift. Providers update their models’ weights, the parameters learned during training. Embeddings shift as documents change. Users’ questions evolve. The same code can produce different answers from one week to the next.
The fifth is variable cost. Model providers bill per token, the unit text is split into, often a piece of a word. The token count depends on the length of questions and answers, with no predictable ceiling. A single agent loop can burn the cost of thousands of normal requests.
A 200 OK from an LLM endpoint says nothing about whether the answer was correct. Nor does it say whether the model hallucinated, leaked personal data or produced biased content. Yet those are the questions that matter for an AI feature in production.
1.3 The five signals of AI observability
Section titled “1.3 The five signals of AI observability”Traditional observability has three signals. Logs tell what happened, metrics give figures aggregated over time, traces follow a request step by step, like parcel tracking. AI observability adds two. Evaluations grade the quality of an answer. User feedback is, for example, a thumbs up or down. These two connect what the system did to whether it was actually useful.

Figure 2. The five signals. Classical observability supplies the first three. Evaluations and feedback are the AI-specific additions and are the defining capability of the discipline.
All five signals are tied together by a single identifier, the trace ID. A trace is made of spans: each span is one timed step, such as a call to the model or a document search. Every span, every score, every piece of feedback belongs to one trace. That is what makes the platform queryable end to end. Without this common key, the signals stay in silos nobody can connect.
For our assistant, it means a member who clicks “thumbs down” points to one specific trace. In it you find the question asked, the documents retrieved, the answer and the score the evaluator gave it.
1.4 Vocabulary
Section titled “1.4 Vocabulary”A handful of terms recur throughout this guide and across all the tools. Their meaning does not change from vendor to vendor and I recommend using them precisely.
| Term | Definition |
|---|---|
| Trace | The full path of a request, made of spans linked by their parent IDs. |
| Span | A single timed operation: one LLM call, one retrieval, one tool invocation. |
| Attribute | A key-value pair attached to a span, used for filtering and aggregation. |
| Online evaluation | Scoring that runs on live production traces, usually asynchronously. |
| Offline evaluation | Scoring that runs on a fixed dataset, typically in CI or as a scheduled batch. |
| LLM-as-judge | An evaluation method where a model grades another model’s answer against a rubric. |
| Drift | A change over time in the inputs, in the reality the system describes or in the pipeline itself, which can degrade answers. Three families: data, concept, pipeline (see the glossary). |
| Grounding | An answer being supported by a verifiable source rather than by what the model retained from training. |
| Hallucination | An answer asserting facts that neither training data, nor the provided context, nor any verifiable source supports. |
| Faithfulness | An answer being grounded in the provided context, scored as a continuous quality. |
| Gold set | A set of questions whose ideal answer is known, used for offline evaluation. |
| Regression set | A set of questions that failed in the past, used to gate future changes. |
1.5 What AI observability is not
Section titled “1.5 What AI observability is not”Three confusions come up often. Better to clear them early.
AI observability is not MLOps. MLOps covers model training, packaging and deployment. AI observability covers what happens after deployment, once real requests arrive.
Nor is it expensive logging. Recording prompts in logs is a starting point. Without evaluation, drift detection or a link to user feedback, the data piles up and nothing improves.
Finally, it is not a product you buy. It is a stack you assemble: instrumentation, collection, storage, evaluation and visualization. As far as I know, no single tool covers all five functions well across the four system types described in the next chapter.
Keep the insurer’s assistant in mind: chapter 2 shows what to capture depending on whether you have a plain model call, document retrieval, an agent or a tool.
Next: 2. Scope and taxonomy of AI systems.
Revised on 4 October 2026: the five signals presented as the site’s reference model, with how the labs’ axes, the method’s content events and the article’s planes map onto them.
Revised on 25 October 2026: opening on an illustrative scenario, basic terms (prompt, completion, embedding, token, span) explained, properties and confusions written as prose.