TechnicalPractitioner4 min read
9. Anti-patterns and common pitfalls
Twelve common pitfalls, described as typical scenarios, that produce data nobody uses, dashboards nobody reads or platforms nobody trusts, with their fixes.
For: engineers and SREs · architects · team managersPrerequisites: Have read chapters 1 to 4 of the guide.
An anti-pattern is a solution that looks reasonable at the time and turns against you later. The twelve below are described as typical scenarios: none is the account of a real case, but each one leads to the same place, data nobody uses, dashboards nobody reads or a platform nobody trusts. Avoiding them is cheaper than fixing them.
For each pitfall, the first paragraph describes what you see, the second the fix.
9.1 Instrumenting before defining questions
Section titled “9.1 Instrumenting before defining questions”The trace tree is exhaustive, with hundreds of attributes per span. Yet nobody can answer “what is the faithfulness score by tenant?”, because the useful attributes were never indexed.
The fix is Step 1 of Chapter 4: write the questions first, instrument second.
9.2 Treating logs as observability
Section titled “9.2 Treating logs as observability”The team piles up large volumes of structured logs in Loki or Splunk, with no spans and no evaluations. Cost is high and getting value out of those logs stays manual work.
Move to Level 2, structured tracing. Logs are a fallback channel, not the primary signal.
9.3 Storing prompts and responses without redaction
Section titled “9.3 Storing prompts and responses without redaction”A production trace contains an API key a user pasted into the chat. That trace now sits in a queryable backend that many people can access.
Redact in the collector, before any exporter. Never trust the application to scrub the data.
9.4 LLM-as-judge on every trace
Section titled “9.4 LLM-as-judge on every trace”The evaluation token cost exceeds the production LLM cost. Worse, the evaluation data is so noisy that nobody acts on it.
Tier the evaluators. For example, cheap rule-based scorers on 100 percent of traces; medium LLM judges on 10 percent; expensive judges on 1 percent plus flagged traces. Use self-consistency only where it matters (see chapter 5).
9.5 Dashboards without alerts
Section titled “9.5 Dashboards without alerts”The team has built beautiful Grafana boards that nobody looks at. Regressions are discovered by users.
Every panel that represents a real SLO, a numeric service target, should have its alert. Dashboards are for diagnosis, alerts are for operations.
9.6 High-cardinality attributes on metrics
Section titled “9.6 High-cardinality attributes on metrics”The metrics backend runs short of memory, queries slow down and cost suddenly explodes when a new tenant onboards.
The fix is attribute hygiene. Keys that take many different values belong on traces. Metrics dimensions stay bounded (see section 7.3).
9.7 No versioning of prompts and configs
Section titled “9.7 No versioning of prompts and configs”A quality regression is detected, but nobody can tell which change caused it: several things shipped that week.
Version every artifact, record the version on every span and correlate deployments with version changes (see chapter 8).
9.8 Treating drift as a problem to be fixed
Section titled “9.8 Treating drift as a problem to be fixed”A drift alarm fires. The team rebuilds the index or swaps the model. Drift comes back the next week.
Drift is a signal, not a defect: it tells you the world has changed. The right response is to investigate and the right action is often to do nothing, because only some drift comes with a loss of quality.
9.9 Buying a tool before defining the stack
Section titled “9.9 Buying a tool before defining the stack”The team subscribes to a commercial LLM observability product and integrates it. Then it finds out the product cannot be deployed in the required jurisdiction or cannot handle the cardinality.
The choice between self-hosting and a managed service is a strategic decision, made against sovereignty, cost and integration constraints (see chapter 6). Make it before evaluating products.
9.10 Evaluating without calibrating
Section titled “9.10 Evaluating without calibrating”In this illustrative scenario, faithfulness scores look stable around 0.85. But spot checks show the judge often contradicts human judgment. Nobody trusts the metric any more.
Build a calibration set whose right answers were established by humans. Measure agreement with a chance-corrected statistic such as Cohen’s kappa. Rework the judge prompt until it reaches the threshold set in advance, then recalibrate whenever the judge model changes.
9.11 Skipping the closed loop
Section titled “9.11 Skipping the closed loop”Evaluations run, scores are stored and nothing changes. The platform produces data but no improvement.
Set up the weekly review and keep it going. Failed traces become regression set entries. Without this ritual, observability shrinks to expensive read-only storage.
9.12 Conflating MLOps and AI observability
Section titled “9.12 Conflating MLOps and AI observability”An MLOps team owns the AI observability stack. It tries to instrument the LLM inference layer the way it instruments model training. The concepts do not match.
AI observability is a production observability concern. It is owned by the platform team, working with ML and security. MLOps owns the model lifecycle, not the request lifecycle.
If you recognize your team in several of these pitfalls, start with 9.1 and 9.11: they decide whether the platform is any use at all.
Next: Appendix.
Revised on 25 October 2026: notion of anti-pattern introduced, symptoms and fixes written as prose without bold labels, links to the relevant chapters.