Skip to content

TechnicalPractitioner8 min read

2. Scope and taxonomy of AI systems

The four AI system types (LLM-only, RAG, agents, MCP tools), what to capture for each, their characteristic failures and the shape of their traces.

For: engineers and SREs · architectsPrerequisites: Have read chapter 1 of the guide.

In the insurer’s illustrative scenario, the assistant started simple: one question, one model call, one answer. Then the team gave it access to the policy terms so it could answer from the texts. Later, it was allowed to look up the member’s file through a tool. At each step the system changed nature. So did the ways it could fail.

AI observability covers four overlapping system types. Each needs its own instrumentation and has its typical failures. The same observability stack has to handle all four, because real applications mix them freely. A production agent usually performs document retrieval and calls MCP tools: a single request then exercises three of the four types.

This taxonomy is the reference for every piece on this site about generative AI observability. The labs, the production article and the GenAI method build on it. Classic ML and computer vision are out of scope: see Beyond the LLM: GPUs and model quality.

The four system types: LLM-only, RAG, agentic, MCP and tools, with their typical failure modes

Figure 3. The four system types. Failure modes are illustrative, not exhaustive.

A single call to a language model, with no retrieval and no tools. Text goes in, text comes out. It is the simplest case and often the first one shipped to production.

  • The prompt, with the system prompt (the fixed instructions supplied by the application) and the user prompt (the question asked) in separate fields.
  • The response, with finish_reason, the reason the model stopped: stop (normal end), length (length limit reached), content_filter (response filtered), tool_call (the model asks for a tool), error, in the OpenTelemetry conventions vocabulary.
  • Token usage, input and output. Distinguish cached tokens where the provider exposes them.
  • End-to-end latency and, if the response is streamed, the time to first token: that is the delay the user actually feels.
  • Model identity, including the version actually served, which often differs from the requested name.
  • Sampling parameters, which control how much randomness goes into the answer: temperature, top-p, top-k, seed if set.

The model can invent entities, dates or quotations. It can ignore the expected format: the application wanted JSON and got prose. A user can slip instructions into a question that pull the model away from its own instructions; that is prompt injection. Cost can run away when response length is not capped. Finally, the provider can go down or swap the model without warning.

Retrieval-augmented generation (RAG) works like a student allowed to bring notes into the exam. Before answering, the system looks for useful passages in a document base, then hands them to the model along with the question. The search can work by meaning in a vector database (which stores embeddings), by keywords with an algorithm such as BM25, or by combining the two.

Observability then also has to capture the retrieval step and the link between the passages retrieved and the answer produced. Retrieval opens a second source of failure: the model can be perfect and still answer wrongly because it was handed the wrong documents. That is what happened to the insurer’s assistant, which was reading an old version of the policy.

  • The retrieval query, which is often a rewrite of the user’s question.
  • The retrieved chunks, meaning the pieces of documents, with their identifiers, source documents and similarity scores.
  • The number of chunks kept (the top-k, the k best ones) and any filters applied: date, source, tenant. A tenant is a customer or entity sharing the same platform as the others.
  • The reranking step, where a second model puts the chunks back in order of relevance: model identity, input chunks, output order.
  • The chunk identifiers present in the final prompt, so each chunk can be tied to the answer that used it.

Four questions sum up RAG quality. Context precision measures the share of retrieved chunks that are relevant. Context recall measures the share of relevant chunks that were actually retrieved. Faithfulness checks that the answer rests on the retrieved context, whether or not that context is correct. Answer relevancy checks that it addresses the question asked.

With low recall, the relevant document was never retrieved and the model invents a plausible answer. With an irrelevant top-k, the similarity scores were high but the chunks did not really answer the question. Contradiction happens when the answer contradicts the retrieved context, which is a faithfulness failure. Stale chunks come from an index that was not updated after a change. Duplicate chunks, finally, repeat the same content and bias the answer.

An agent works like an intern handed a task: it thinks about what to do, acts, looks at the result, then starts again until it judges the job done. That is the plan, act, observe loop. Observability has to capture the structure of the loop and the reasoning at each turn.

Agents are the hardest system type to observe, because some of their failures have no natural limit: they can go round in circles, blow up costs or stop too early.

  • The planner’s reasoning at each step, when the model exposes it. For reasoning models, the chain of thought (the intermediate reasoning the model writes out) is itself a span attribute.
  • Tool calls with their name, arguments and the response received.
  • The iteration count and the reason the loop ended: success, budget exhausted, maximum iterations reached, error.
  • Per-step cost and latency, with running totals over the whole trace.
  • Repeated tool calls with identical arguments, which usually signal a loop.

The most visible is the infinite or near-infinite loop, stopped only by the iteration cap or the budget. The agent can also pick the wrong tool when several tools have overlapping descriptions. It can pass malformed arguments: wrong types, missing required fields, invented values for a closed list. It can stop too early and answer without finishing the task. And cost explodes when the context grows with every turn of the loop.

The Model Context Protocol (MCP) plays the role for tools that a standard plug plays for electrical appliances. It standardizes how a language model calls tools, whatever the provider. An MCP server exposes one or more tools (look up a file, fetch an invoice), which agents can discover and call.

When MCP servers are part of the system, observability has to capture the protocol level, not just the function call. An MCP server can become a bottleneck shared by several agents and applications.

  • MCP server identity and version.
  • Tool name, arguments and the response received.
  • Latency and error rate per tool and per server.
  • The authentication scope of each call, meaning the token or identity passed along and the permissions it grants.
  • Discovery events: the moment the agent first asked for the list of available tools.
  • The choice between servers when several MCP servers are connected.

The tool can be unavailable because the MCP server is down or slow. The schema can drift between server and client when the server publishes a new tool signature. Scope escalation happens when an agent uses a tool that requires more permissions than the user has. A slow tool can block the whole agent loop: a single tool with a 30-second timeout can weigh more than the rest of the trace. Finally, with tool poisoning, the MCP server returns crafted output designed to manipulate the agent.

The same span library produces very different trace trees depending on the system type. Figure 4 shows a realistic RAG-agent trace combining all four types: a root span, an agent loop, two LLM calls, two retrieval steps, an MCP tool call and three evaluation spans attached after generation.

Trace tree of a RAG-agent request: root span, agent loop, LLM calls, retrievals, MCP tool call and evaluation spans

Figure 4. Trace tree of a single RAG-agent request. The same trace contains the four system types: LLM calls, retrieval, agent loop and MCP tool. Evaluations attach as child spans of the trace.

Three things can be read off this shape. Evaluations live in the same trace as the request they score: they are not separate logs. Feedback that arrives later, such as a user’s thumbs-down, is attached after the fact but found through the trace ID. Finally, the width of the bars gives the latency of each step. The widest bar is the dominant cost and I suggest optimizing it first.

Next: 3. The maturity roadmap.

Revised on 4 October 2026: four-type taxonomy presented as the site’s reference, scope (classic ML and vision excluded) restated.

Revised on 25 October 2026: opening on the illustrative scenario, each type introduced with an image, technical terms explained (top-k, reranking, tenant, BM25, chain of thought), failures written as prose.