TechnicalExpert10 min read
Part III. Measurement grid per component
What to measure, why and what it costs not to: one grid for each of the four bricks, the LLM, RAG, the agent and MCP.
For: engineers and SREs · architectsPrerequisites: Have read part I of the method.
A team getting started almost always asks the same question in a meeting: “we’ve plugged in OpenTelemetry, so what do we look at now?”. This part answers it brick by brick, like a car’s service book that tells you what to check on the engine, the brakes and the tyres.
Each component gets its diagram, then a three-column grid: what you measure, the decision the measurement informs and what happens if you do not measure it. The letter (S) flags a silent failure in the sense of Part I: a failure that raises no error and can last for weeks.
8. LLM (the inference brick)
Section titled “8. LLM (the inference brick)”Inference is the moment the model produces its answer. It runs like an order in a restaurant. The request first waits its turn in a queue. Then comes prefill, where the model reads the whole input text in one go, like the cook reading the order slip. Then decode, where it writes the answer word by word; with streaming, those words go out to the user as they come. On the machine side, the GPU keeps a KV cache in memory, a kind of scratchpad of what it has already read. It also groups several requests into batches to go faster.
flowchart LR REQ["Request"] --> QUEUE["Queue"] --> PREFILL["Prefill"] --> DECODE["Decode (streaming)"] --> RESP["Response"] GPU["GPU / KV cache / batch"] -.-> PREFILL GPU -.-> DECODE
A few terms from the grid. E2E (end-to-end) latency runs from sending the question to the end of the answer; p95 and p99 are the times under which 95% and 99% of requests complete. DCGM and vLLM are the measurement sources on the GPU side and the inference server side. The finish_reasons say why the model stopped: complete answer, length limit reached, content filter.
| What | Why (decision) | Impact if unmonitored |
|---|---|---|
| TTFT, E2E latency (p95, p99) | Hold the perceived latency budget | Degradation unnoticed until complaints |
| Throughput tokens/s, requests/s | Size, autoscale | Unanticipated saturation, capacity incident |
| GPU saturation, KV cache, batch (DCGM, vLLM) | Anticipate degradation | (S) Eviction and queue rise silently, then an unexplained spike |
Error ratio by error.type | Isolate timeout, rate-limit, content_filter | Blind diagnosis, longer resolution |
finish_reasons distribution | Detect truncated or blocked answers | (S) Cut answers served with no alert |
| Refusal rate | Detect a behavior shift | (S) The product refuses more and more, no signal |
| Hallucination, faithfulness (sampled eval) | Guarantee correctness | (S) The system asserts confidently false claims, business and legal risk |
| Output drift | Detect slow evolution | (S) Invisible regression over weeks |
| Tokens in/out, cost per request | Govern the budget | (S) Bill discovered at month end |
| Cache hit rate | Measure a direct cost lever | Permanent overspend unidentified |
| Prompt bloat (input tokens over time) | Detect context growth | (S) Cost and latency drift slowly |
Seven rows out of eleven carry the letter (S). In my view, that is the figure to show anyone who thinks an availability probe is enough for an LLM.
9. RAG
Section titled “9. RAG”A RAG works like a librarian who prepares a file before an expert answers. Upstream, documents are cut into pieces (chunking), then each piece is turned into a series of numbers that sums up its meaning, an embedding. These numbers go into a vector index. When a question arrives, the system looks for the pieces closest in meaning, re-sorts them from most to least relevant (reranking) and hands them to the model as context so it can write its answer.
flowchart LR ING["Ingestion"] --> CHUNK["Chunking"] --> EMB["Embedding"] --> IDX["Vector index"] Q["Question"] --> RET["Retrieval"] --> RANK["Reranking"] --> CTX["Context"] --> GEN["Generation"] --> ANS["Answer"] IDX --> RET
The librarian can go wrong in two ways that the grid tells apart. Context precision measures whether the retrieved pieces are relevant: does the file contain off-topic pages? Context recall measures whether needed pieces are missing: is the file complete? Hit rate and MRR (mean reciprocal rank) check whether the right document comes back and at what rank. Faithfulness, or groundedness, checks that the answer rests on the file rather than on the model’s imagination.
| What | Why (decision) | Impact if unmonitored |
|---|---|---|
| Context precision | Are the retrieved chunks relevant | (S) Off-topic context, hallucination downstream |
| Context recall | Did you retrieve everything needed | (S) Incomplete or invented answers |
| Hit rate, MRR | Is the right document retrieved and well ranked | You fix generation by mistake |
| Faithfulness / groundedness | Answer grounded in context | (S) Hallucinations served as facts |
| Answer relevance | Does it answer the question | Off-target answers unnoticed |
| Citation accuracy | Cited sources correct | (S) False references, compliance risk |
| Coverage (questions with no context) | Measure knowledge base gaps | (S) A documentary blind spot that widens |
| Base freshness | Knowledge not stale | (S) Outdated answers served confidently |
| Retrieval latency, index health | Hold the chain budget | Degradation mis-attributed |
| Embeddings and vector store cost | Govern cost | Uncontrolled infrastructure overspend |
The “Hit rate, MRR” row deserves a pause. Without it, a team sees wrong answers and tweaks the prompt or switches models, when the problem lies in retrieval. That is the trap waiting for the HR assistant team in the Part I scenario.
For evaluation, RAGAS and DeepEval are used to block a release that lowers quality (gating). Arize Phoenix tracks drift and visualizes embedding clusters. These tools often rely on another model acting as a judge. That judge has flaws of its own: it is not deterministic, it drifts, it needs calibrating. Section 15 covers this evaluator problem.
10. Agent
Section titled “10. Agent”An agent is like an intern handed a mission and a toolbox. It plans, picks a tool, uses it, looks at the result, then decides whether it is done or goes round again. That cycle is both its strength and its risk: an intern who cannot stop will repeat the same steps twenty times. In the OpenTelemetry conventions, the agent call is named invoke_agent and each tool use execute_tool. The sequence of steps forms its trajectory.
flowchart TB
START["invoke_agent"] --> PLAN["Planning"] --> SEL["Tool selection"] --> TOOL["execute_tool"] --> OBS["Observation"]
OBS --> DEC{"Task done?"}
DEC -->|"no"| SEL
DEC -->|"yes"| END["Final answer"]
DEC -.->|"risk: loop"| LOOP["Loop detection"]
| What | Why (decision) | Impact if unmonitored |
|---|---|---|
| Task success rate | Measure real end-to-end work | (S) Partial failure with no error, product value eroded |
| Steps per task | Detect abnormal trajectories | (S) Cost and latency drift from lengthening |
| Loop detection | Spot repetition | (S) A loop that burns the budget, seen on the bill |
| Tool failure rate | Identify failing tools | Costly retries, root cause not isolated |
| Latency per step and per task | Decompose the end-to-end path | Diagnosis impossible on long trajectories |
| Cost per task (cumulative tokens) | Know the real cost of a task | (S) Mispriced offering |
| Tool selection accuracy | The right tool at the right time | (S) Degraded results with no signal |
| Human handoff rate | Measure real autonomy | Operator load underestimated, ROI distorted |
| Guardrails, injection attempts | Agentic security | Command injection undetected |
I recommend starting with task success rate and steps per task. The first tells you whether the agent delivers the promised service; the second sees loops coming before the bill does.
On the tooling side, Laminar and Langfuse let you replay an agent’s transcript and debug it. Phoenix and Confident AI evaluate multi-step journeys. LangSmith and LangGraph Studio suit agents built with LangGraph.
11. MCP
Section titled “11. MCP”MCP (Model Context Protocol) is the standard socket that lets an agent plug in tools: an MCP server exposes functions (read a file, query a database, send a message) that the model can call. You observe it on three layers. The protocol layer covers the JSON-RPC messages exchanged, the sessions and the schemas that describe each tool. The execution layer covers the calls themselves, their latency and their errors. The task layer measures the effect on the agent’s success. As with the layers in Part I, a defect at the bottom surfaces at the top.
flowchart TB P["Protocol layer: JSON-RPC over transport, sessions, schemas"] --> T["Execution layer: tool calls, latency, errors"] --> A["Task layer: impact on agent success"] A -.->|"cascade"| P
The grid mentions attacks specific to MCP. A rug pull is a trusted server that changes its behavior after you have adopted it, like a supplier who delivers different goods once the contract is signed. Schema poisoning slips malicious instructions into a tool’s description. Pinning by hash means recording a tool’s fingerprint when you approve it, so any later change shows up. A desync is a mismatch between client and server about the state of the session.
| What | Why (decision) | Impact if unmonitored |
|---|---|---|
| Latency and errors per tool and server | Execution health, isolation | Failure mis-attributed to the agent |
| Session health, client-server desync | Transport stability | (S) Desync that degrades the agent with no error |
| Authentication failures per identity | Detect abuse and misconfiguration | Illegitimate access, secret exposure |
| Enumeration / discovery spikes | Detect hostile reconnaissance | (S) Invisible reconnaissance before exploitation |
| Server mutations | Detect rug pull and schema poisoning | (S) A trusted tool turned malicious |
| Tool pinning by hash | Tool integrity over time | (S) Undetected modification |
| Invocation log | Regulatory audit trail | No traceability, non-compliance |
| Availability per server, health checks | Guarantee fallback | Agent looping on an unavailable server |
MCP security
Section titled “MCP security”The OWASP MCP Top 10 reference (2025 edition, in beta) defines ten risk categories:
- MCP01 token mismanagement and secret exposure;
- MCP02 privilege escalation via scope creep;
- MCP03 tool poisoning;
- MCP04 software supply chain attacks and dependency tampering;
- MCP05 command injection and execution;
- MCP06 prompt injection via contextual payloads;
- MCP07 insufficient authentication and authorization;
- MCP08 lack of audit and logging;
- MCP09 shadow MCP servers;
- MCP10 context injection and over-sharing.
Rug pull, tool shadowing (a tool passing itself off as another) and schema poisoning are attack techniques described in the literature. They mostly fall under MCP03 and are not categories of the reference. The grid above directly addresses MCP08.
That leaves auto-approval, which lets the agent call its tools without asking anyone. It removes the last barrier that does not depend on the model. The MCPTox benchmark (2025) ran 20 LLM agents against 1,348 tool poisoning cases built on 45 real MCP servers. The attack succeeds in 36.5% of cases on average, up to 72.8% for the most exposed model; no model refuses more than 3% of attacks. The benchmark measures model susceptibility, not the effect of human approval. I still recommend keeping a human in the loop for sensitive tools, because that check does not rely on the judgment of the model under attack.
On the tooling side, mcp-scan (Invariant Labs / Snyk) and Cisco mcp-scanner (YARA) complement each other. Add an MCP gateway, a mandatory checkpoint between agents and servers, with audit and OTel export.
For Monday morning
Section titled “For Monday morning”Take the brick you already run and count the rows marked (S) in its grid that you do not measure. Start with the one whose impact would hurt most in front of a steering committee. If you have agents plugged into MCP, add the invocation log right away: it is the row that costs least and protects you most on audit day.
Revised on 2 October 2026: official OWASP MCP Top 10 categories (2025 edition, in beta) and MCPTox benchmark figures in place of an unsourced claim about auto-approval.
Revised on 25 October 2026: page rewritten for a wider audience (introductory images, grid terms explained), with no change of substance.