Skip to content

TechnicalExpert10 min read

Part III. Measurement grid per component

What to measure, why and what it costs not to: one grid for each of the four bricks, the LLM, RAG, the agent and MCP.

For: engineers and SREs · architectsPrerequisites: Have read part I of the method.

A team getting started almost always asks the same question in a meeting: “we’ve plugged in OpenTelemetry, so what do we look at now?”. This part answers it brick by brick, like a car’s service book that tells you what to check on the engine, the brakes and the tyres.

Each component gets its diagram, then a three-column grid: what you measure, the decision the measurement informs and what happens if you do not measure it. The letter (S) flags a silent failure in the sense of Part I: a failure that raises no error and can last for weeks.

Inference is the moment the model produces its answer. It runs like an order in a restaurant. The request first waits its turn in a queue. Then comes prefill, where the model reads the whole input text in one go, like the cook reading the order slip. Then decode, where it writes the answer word by word; with streaming, those words go out to the user as they come. On the machine side, the GPU keeps a KV cache in memory, a kind of scratchpad of what it has already read. It also groups several requests into batches to go faster.

flowchart LR
  REQ["Request"] --> QUEUE["Queue"] --> PREFILL["Prefill"] --> DECODE["Decode (streaming)"] --> RESP["Response"]
  GPU["GPU / KV cache / batch"] -.-> PREFILL
  GPU -.-> DECODE

A few terms from the grid. E2E (end-to-end) latency runs from sending the question to the end of the answer; p95 and p99 are the times under which 95% and 99% of requests complete. DCGM and vLLM are the measurement sources on the GPU side and the inference server side. The finish_reasons say why the model stopped: complete answer, length limit reached, content filter.

WhatWhy (decision)Impact if unmonitored
TTFT, E2E latency (p95, p99)Hold the perceived latency budgetDegradation unnoticed until complaints
Throughput tokens/s, requests/sSize, autoscaleUnanticipated saturation, capacity incident
GPU saturation, KV cache, batch (DCGM, vLLM)Anticipate degradation(S) Eviction and queue rise silently, then an unexplained spike
Error ratio by error.typeIsolate timeout, rate-limit, content_filterBlind diagnosis, longer resolution
finish_reasons distributionDetect truncated or blocked answers(S) Cut answers served with no alert
Refusal rateDetect a behavior shift(S) The product refuses more and more, no signal
Hallucination, faithfulness (sampled eval)Guarantee correctness(S) The system asserts confidently false claims, business and legal risk
Output driftDetect slow evolution(S) Invisible regression over weeks
Tokens in/out, cost per requestGovern the budget(S) Bill discovered at month end
Cache hit rateMeasure a direct cost leverPermanent overspend unidentified
Prompt bloat (input tokens over time)Detect context growth(S) Cost and latency drift slowly

Seven rows out of eleven carry the letter (S). In my view, that is the figure to show anyone who thinks an availability probe is enough for an LLM.

A RAG works like a librarian who prepares a file before an expert answers. Upstream, documents are cut into pieces (chunking), then each piece is turned into a series of numbers that sums up its meaning, an embedding. These numbers go into a vector index. When a question arrives, the system looks for the pieces closest in meaning, re-sorts them from most to least relevant (reranking) and hands them to the model as context so it can write its answer.

flowchart LR
  ING["Ingestion"] --> CHUNK["Chunking"] --> EMB["Embedding"] --> IDX["Vector index"]
  Q["Question"] --> RET["Retrieval"] --> RANK["Reranking"] --> CTX["Context"] --> GEN["Generation"] --> ANS["Answer"]
  IDX --> RET

The librarian can go wrong in two ways that the grid tells apart. Context precision measures whether the retrieved pieces are relevant: does the file contain off-topic pages? Context recall measures whether needed pieces are missing: is the file complete? Hit rate and MRR (mean reciprocal rank) check whether the right document comes back and at what rank. Faithfulness, or groundedness, checks that the answer rests on the file rather than on the model’s imagination.

WhatWhy (decision)Impact if unmonitored
Context precisionAre the retrieved chunks relevant(S) Off-topic context, hallucination downstream
Context recallDid you retrieve everything needed(S) Incomplete or invented answers
Hit rate, MRRIs the right document retrieved and well rankedYou fix generation by mistake
Faithfulness / groundednessAnswer grounded in context(S) Hallucinations served as facts
Answer relevanceDoes it answer the questionOff-target answers unnoticed
Citation accuracyCited sources correct(S) False references, compliance risk
Coverage (questions with no context)Measure knowledge base gaps(S) A documentary blind spot that widens
Base freshnessKnowledge not stale(S) Outdated answers served confidently
Retrieval latency, index healthHold the chain budgetDegradation mis-attributed
Embeddings and vector store costGovern costUncontrolled infrastructure overspend

The “Hit rate, MRR” row deserves a pause. Without it, a team sees wrong answers and tweaks the prompt or switches models, when the problem lies in retrieval. That is the trap waiting for the HR assistant team in the Part I scenario.

For evaluation, RAGAS and DeepEval are used to block a release that lowers quality (gating). Arize Phoenix tracks drift and visualizes embedding clusters. These tools often rely on another model acting as a judge. That judge has flaws of its own: it is not deterministic, it drifts, it needs calibrating. Section 15 covers this evaluator problem.

An agent is like an intern handed a mission and a toolbox. It plans, picks a tool, uses it, looks at the result, then decides whether it is done or goes round again. That cycle is both its strength and its risk: an intern who cannot stop will repeat the same steps twenty times. In the OpenTelemetry conventions, the agent call is named invoke_agent and each tool use execute_tool. The sequence of steps forms its trajectory.

flowchart TB
  START["invoke_agent"] --> PLAN["Planning"] --> SEL["Tool selection"] --> TOOL["execute_tool"] --> OBS["Observation"]
  OBS --> DEC{"Task done?"}
  DEC -->|"no"| SEL
  DEC -->|"yes"| END["Final answer"]
  DEC -.->|"risk: loop"| LOOP["Loop detection"]
WhatWhy (decision)Impact if unmonitored
Task success rateMeasure real end-to-end work(S) Partial failure with no error, product value eroded
Steps per taskDetect abnormal trajectories(S) Cost and latency drift from lengthening
Loop detectionSpot repetition(S) A loop that burns the budget, seen on the bill
Tool failure rateIdentify failing toolsCostly retries, root cause not isolated
Latency per step and per taskDecompose the end-to-end pathDiagnosis impossible on long trajectories
Cost per task (cumulative tokens)Know the real cost of a task(S) Mispriced offering
Tool selection accuracyThe right tool at the right time(S) Degraded results with no signal
Human handoff rateMeasure real autonomyOperator load underestimated, ROI distorted
Guardrails, injection attemptsAgentic securityCommand injection undetected

I recommend starting with task success rate and steps per task. The first tells you whether the agent delivers the promised service; the second sees loops coming before the bill does.

On the tooling side, Laminar and Langfuse let you replay an agent’s transcript and debug it. Phoenix and Confident AI evaluate multi-step journeys. LangSmith and LangGraph Studio suit agents built with LangGraph.

MCP (Model Context Protocol) is the standard socket that lets an agent plug in tools: an MCP server exposes functions (read a file, query a database, send a message) that the model can call. You observe it on three layers. The protocol layer covers the JSON-RPC messages exchanged, the sessions and the schemas that describe each tool. The execution layer covers the calls themselves, their latency and their errors. The task layer measures the effect on the agent’s success. As with the layers in Part I, a defect at the bottom surfaces at the top.

flowchart TB
  P["Protocol layer: JSON-RPC over transport, sessions, schemas"] --> T["Execution layer: tool calls, latency, errors"] --> A["Task layer: impact on agent success"]
  A -.->|"cascade"| P

The grid mentions attacks specific to MCP. A rug pull is a trusted server that changes its behavior after you have adopted it, like a supplier who delivers different goods once the contract is signed. Schema poisoning slips malicious instructions into a tool’s description. Pinning by hash means recording a tool’s fingerprint when you approve it, so any later change shows up. A desync is a mismatch between client and server about the state of the session.

WhatWhy (decision)Impact if unmonitored
Latency and errors per tool and serverExecution health, isolationFailure mis-attributed to the agent
Session health, client-server desyncTransport stability(S) Desync that degrades the agent with no error
Authentication failures per identityDetect abuse and misconfigurationIllegitimate access, secret exposure
Enumeration / discovery spikesDetect hostile reconnaissance(S) Invisible reconnaissance before exploitation
Server mutationsDetect rug pull and schema poisoning(S) A trusted tool turned malicious
Tool pinning by hashTool integrity over time(S) Undetected modification
Invocation logRegulatory audit trailNo traceability, non-compliance
Availability per server, health checksGuarantee fallbackAgent looping on an unavailable server

The OWASP MCP Top 10 reference (2025 edition, in beta) defines ten risk categories:

  • MCP01 token mismanagement and secret exposure;
  • MCP02 privilege escalation via scope creep;
  • MCP03 tool poisoning;
  • MCP04 software supply chain attacks and dependency tampering;
  • MCP05 command injection and execution;
  • MCP06 prompt injection via contextual payloads;
  • MCP07 insufficient authentication and authorization;
  • MCP08 lack of audit and logging;
  • MCP09 shadow MCP servers;
  • MCP10 context injection and over-sharing.

Rug pull, tool shadowing (a tool passing itself off as another) and schema poisoning are attack techniques described in the literature. They mostly fall under MCP03 and are not categories of the reference. The grid above directly addresses MCP08.

That leaves auto-approval, which lets the agent call its tools without asking anyone. It removes the last barrier that does not depend on the model. The MCPTox benchmark (2025) ran 20 LLM agents against 1,348 tool poisoning cases built on 45 real MCP servers. The attack succeeds in 36.5% of cases on average, up to 72.8% for the most exposed model; no model refuses more than 3% of attacks. The benchmark measures model susceptibility, not the effect of human approval. I still recommend keeping a human in the loop for sensitive tools, because that check does not rely on the judgment of the model under attack.

On the tooling side, mcp-scan (Invariant Labs / Snyk) and Cisco mcp-scanner (YARA) complement each other. Add an MCP gateway, a mandatory checkpoint between agents and servers, with audit and OTel export.

Take the brick you already run and count the rows marked (S) in its grid that you do not measure. Start with the one whose impact would hurt most in front of a steering committee. If you have agents plugged into MCP, add the invocation log right away: it is the row that costs least and protects you most on audit day.


Revised on 2 October 2026: official OWASP MCP Top 10 categories (2025 edition, in beta) and MCPTox benchmark figures in place of an unsourced claim about auto-approval.

Revised on 25 October 2026: page rewritten for a wider audience (introductory images, grid terms explained), with no change of substance.