Skip to content
MTTL, Mean Time To Learn

TechnicalExpert

Part VIII. Anti-patterns

For: engineers and SREs · architectsPrerequisites: Have read parts I to IV of the method.

The catalog of common pitfalls is in chapter 9 of the guide. This part keeps only the pitfalls the guide does not cover.

  1. Alerting on raw latency only. Misses all silent failures. Fix: alert on quality, cost and drift, not only on latency and errors.
  2. Evals disconnected from production. An artificial test dataset gives a false sense of safety. Fix: sample from production, representatively and stratified.
  3. Quality SLO with no confidence interval. False precision on a small sample. Fix: compare the lower bound of the interval to target, size the sample (Part IV).
  4. A single provider with no fallback. Unavailability suffered at the first rate-limit. Fix: tested multi-provider fallback, graceful degradation.
  5. MCP tool auto-approval. A major exploitation surface. Fix: human-in-the-loop validation on sensitive tools, least privilege.
  6. Mixing content and metadata retention. Non-compliance and overspend. Fix: distinct retention policies, separate access boundaries (Part VI).

Pitfalls covered in the guide

Section titled “Pitfalls covered in the guide”
  • Prompts in metric labels: see the guide, 9.6 (high-cardinality attributes on metrics). Labels stay low-cardinality (model, operation, error type).
  • Capturing all content by default: see the guide, 9.3 (storing prompts and responses without redaction). Site rule: no capture by default, explicit activation (opt-in), masking at the collector and, in production, content stored outside telemetry with a reference on the span, as the OpenTelemetry GenAI semantic conventions recommend; details in the article Observing an LLM system, §2 and in Part I.
  • Unversioned judge: see the guide, 9.10 (evaluating without calibrating); judge meta-observability is in Part IV.
  • Dashboards with no associated decision: see the guide, 9.5 (dashboards without alerts).

Revised on 4 October 2026: the pitfalls already described in chapter 9 of the guide (former 1, 4, 5 and 9) are replaced with pointers, the others are renumbered 1 to 6; the content capture rule is aligned with the OpenTelemetry specification.