Skip to content

TechnicalPractitioner5 min read

Appendix: glossary, references, next steps

Glossary of AI observability terms, reference material and the recommended next step for each starting maturity level.

For: engineers and SREs · architectsPrerequisites: Know the maturity levels from chapter 3.

This appendix is a memo. The glossary gathers the terms explained throughout the guide, the references point to primary sources and the last section says what to do next given your current level.

Attribute: a key-value pair attached to a span, used for filtering and aggregation. Example: gen_ai.request.model = "provider-model".

Calibration: measuring how well an LLM judge’s scores agree with human judgment, on a set prepared for the purpose. It is a prerequisite before trusting any LLM-as-judge metric.

Cardinality: the number of distinct value combinations a metric’s labels take. High cardinality on metrics inflates storage; it belongs on traces.

Drift: a change over time that can degrade an AI system’s answers. There are three families. Data drift, when the inputs change: input drift or covariate shift, embedding drift, prior shift, virtual drift. Concept drift, when the reality the system describes changes. Pipeline drift, when a component changes: model or provider silently changed, prompt template, RAG chunking. Output drift is not a fourth family: it is the symptom revealed by evaluations, whose cause belongs to one of the three. The taxonomy and its correspondence table are in module 3 of the labs, which is the reference.

Embedding: a text translated into a list of numbers, such that two texts with similar meaning give similar lists. It is what makes search by meaning possible.

Faithfulness: is the answer grounded in the provided context? A faithful answer may still be wrong if the context is wrong. Faithfulness is therefore distinct from factual correctness.

Gold set: a set of questions whose ideal answer is known, used for offline evaluation.

Grounding: an answer being supported by a verifiable source rather than by what the model retained from training (its parametric knowledge).

Hallucination: an answer asserting facts that neither training data, nor the provided context, nor any verifiable source supports.

Head sampling: a decision taken at trace start, by hash or probability. Stateless and cheap, but it cannot select on outcome.

LLM (large language model): a model that produces text from an input text, the prompt.

LLM-as-judge: an evaluation method where a model grades another model’s answer against a rubric. A common approach for semantic quality, to be calibrated against human judgment.

MCP (Model Context Protocol): a protocol that standardizes how a language model calls tools, whatever the provider.

OpenInference: semantic conventions and instrumentation libraries originally from Arize, designed for AI tracing. They export over OTLP, but with their own attributes (openinference.span.kind, llm.*) rather than the OpenTelemetry gen_ai.* attributes: you either convert attributes (Collector transform processor) or pick a single convention.

OpenTelemetry GenAI: semantic conventions extending OpenTelemetry to language model operations. Still in development status, maintained in a dedicated repository since June 2026.

RAG (retrieval-augmented generation): before answering, the system looks for useful passages in a document base and hands them to the model along with the question.

Redaction: masking or removing sensitive data before storage, using regular expressions, named entity recognition or an LLM.

Regression set: a set of questions that failed in the past, used to gate future changes. It grows as new failures are triaged.

Span: a single timed operation, such as one LLM call, one retrieval, one tool invocation. Spans nest to form trace trees.

Tail sampling: a decision to keep a trace or not, taken at trace end by policy. Complete traces have to be held in memory while the decision is made, but you can select on outcome (error, slowness).

Token: the unit a language model splits text into, often a piece of a word. Providers bill per token, for input and output.

Trace: the full path of a request, made of spans linked by parent IDs. The basic unit of AI observability.

Section titled “A.3 Recommended next steps by starting level”

The timeframes below are orders of magnitude, to be adjusted to your context. Find your level using the signs in chapter 3, then read only the section that applies to you.

Target Level 1 in two weeks. Add structured logging around every LLM call. Capture prompt, response, model, token counts, latency, error and a request ID. Aggregate token usage into a basic cost dashboard. Do not start with OpenTelemetry or with evaluation: ship the logs first.

Target Level 2 in one to two months. Adopt the OpenTelemetry GenAI conventions and migrate your home-grown logging to span attributes. Deploy an OpenTelemetry Collector and a trace backend. If you want an AI-specific UI without extra integration, a self-hosted tool such as Phoenix or Langfuse is a good starting point; if you already run Tempo or Jaeger, start with it. Check that you can answer the questions from Step 1 of Chapter 4.

Target Level 3 in two to three months. Add online evaluation on the most critical path. Start with three evaluators: format compliance, toxicity and one domain-specific scorer. Wire the scores to alerts. Set up a weekly review of failed and low-score traces.

Target Level 4 in three months. Add embedding drift monitoring. Feed user feedback into the trace store. Build and maintain a gold set for RAG evaluation. Start offline evaluation in CI to gate model and prompt changes.

Target Level 5 within a quarter. Formalize the regression set, the few-shot example set and the upgrade gate. Automate re-indexing on drift. Make the observability platform the source of truth for model and prompt iteration.

Back to the overview.

Revised on 2 October 2026: GenAI conventions link moved to the semantic-conventions-genai repository, Phoenix and DeepEval links updated, Langfuse and Traceloop acquisitions, OpenInference’s own attributes clarified, first version of the guide cited.

Revised on 4 October 2026: drift definition aligned on the site’s three families (data, concept, pipeline), with a link to module 3 of the labs.

Revised on 25 October 2026: glossary extended with the basic terms explained in the guide (LLM, RAG, MCP, embedding, token, cardinality, redaction), definitions reworded, introduction to the next steps.