TechnicalExpert3 min read
Part VIII. Anti-patterns
The costly pitfalls specific to the method and how to fix them; the common pitfalls are in chapter 9 of the guide.
For: engineers and SREs · architectsPrerequisites: Have read parts I to IV of the method.
An anti-pattern is a solution that looks sensible and then turns against you, like a shortcut that makes the trip longer. The catalog of common AI observability pitfalls is in chapter 9 of the guide. This part keeps only the six pitfalls the guide does not cover, those that follow directly from the method. Their numbers are used as references elsewhere on the site.
The six pitfalls specific to the method
Section titled “The six pitfalls specific to the method”1. Alerting on raw latency only
Section titled “1. Alerting on raw latency only”This is the team whose dashboards stay green while the assistant gets things wrong, as in the Part I scenario. An alert on latency and errors misses every silent failure by design. The fix is to alert on quality, cost and drift as well.
2. Evaluations disconnected from production
Section titled “2. Evaluations disconnected from production”A hand-built test set is like an exam where you know the questions in advance: you pass it, then reality asks different ones. It gives a false sense of safety. I recommend sampling from production, representatively and stratified.
3. A quality SLO with no confidence interval
Section titled “3. A quality SLO with no confidence interval”Announcing 96% good answers based on thirty scored answers means displaying a precision you do not have. The fix is the one in Part IV: compare the lower bound of the interval to the target and size the sample.
4. A single provider with no fallback
Section titled “4. A single provider with no fallback”At the first rate limit, when the provider caps the number of calls, you suffer the outage with no way out. The fix is a fallback to another provider, tested in advance, plus graceful degradation: the service does less but keeps answering.
5. MCP tool auto-approval
Section titled “5. MCP tool auto-approval”Letting an agent call its tools without any check opens a wide attack surface, as the MCPTox figures quoted in Part III show. I recommend human-in-the-loop validation on sensitive tools and least privilege: each tool can reach only what it needs.
6. Mixing content and metadata retention
Section titled “6. Mixing content and metadata retention”Keeping prompts as long as trace metadata is expensive and creates a compliance problem. The fix comes down to two rules: distinct retention policies and separate access boundaries (Part VI).
Pitfalls covered in the guide
Section titled “Pitfalls covered in the guide”Four pitfalls the method often mentions are described in detail in the guide.
Putting prompts in metric labels is covered in the guide, 9.6 (high-cardinality attributes on metrics). Labels stay low-cardinality: model, operation, error type.
Capturing all content by default is covered in the guide, 9.3 (storing prompts and responses without redaction). The site rule is: no capture by default, explicit activation (opt-in), masking at the collector. In production, content is stored outside traces with a reference on the span, as the OpenTelemetry GenAI semantic conventions recommend. Details are in the article Observing an LLM system, §2 and in Part I.
An unversioned judge is covered in the guide, 9.10 (evaluating without calibrating); watching the judge, or meta-observability, is described in Part IV.
Dashboards with no associated decision are covered in the guide, 9.5 (dashboards without alerts).
For Monday morning
Section titled “For Monday morning”Reread your current alerts. If none of them covers quality, cost or drift, you are in pitfall 1 and it is the first one to fix. Then check whether your agents approve their own calls to sensitive tools.
Revised on 4 October 2026: the pitfalls already described in chapter 9 of the guide (former 1, 4, 5 and 9) are replaced with pointers, the others are renumbered 1 to 6; the content capture rule is aligned with the OpenTelemetry specification.
Revised on 25 October 2026: page rewritten for a wider audience (each pitfall illustrated and written as prose), with no change of substance or numbering.