OrganizationExpert5 min read
Part IX. Implementation
Where to start and in what order? The seven-phase approach, the reference lab and per-axis checklists.
For: engineers and SREs · architects · team managersPrerequisites: Have read parts I to VIII of the method.
26. Step-by-step approach
Section titled “26. Step-by-step approach”A team that has read the previous eight parts often ends up with a list of work items too long for one quarter. The method sorts them into seven phases, a bit like renovating a house: you draw the plan before running the cables, you run the cables before fitting the switches. The last phase loops back to the third, because continuous improvement keeps the cycle turning.
flowchart LR P0["Phase 0<br/>Mapping"] --> P1["Phase 1<br/>Instrumentation"] --> P2["Phase 2<br/>SLI, SLO, alerting"] --> P3["Phase 3<br/>Evaluation"] --> P4["Phase 4<br/>FinOps"] --> P5["Phase 5<br/>Compliance"] --> P6["Phase 6<br/>Continuous improvement"] P6 -.->|"closed loop"| P3
Phase 0 is mapping. You list the components, the data flows, the sensitivity boundaries (where personal or confidential data passes), the regulatory obligations and the people who will use observability.
Phase 1 is instrumentation. You put OTel GenAI everywhere, with no content capture by default and a collector wired to the existing backends (see Part II).
Phase 2 sets up service, quality and cost SLIs and SLOs. It brings dashboards linked by trace and alerts on quality and cost drift.
Phase 3 is evaluation: building the golden set, calibrating the judge, gating in continuous integration, sampling in production (see Part IV).
Phase 4 is FinOps, managing cost. You attribute cost to a team or a feature, track it per request, session and task, set budgets and drift alerts, then pull the saving levers.
Phase 5 is compliance: masking, separating content from metadata, distinct retention, immutable audit, escalating MCP events to the SOC (the security operations center), evidence matrix (see Part VI).
Phase 6 is continuous improvement. Production feeds the test datasets, observed regressions become tests, thresholds are revised.
In my view, the temptation is to jump from phase 1 to phase 4 because the bill hurts. I recommend going through phase 3 anyway: without evaluation, a cost saving can hide a drop in quality.
27. The reference lab
Section titled “27. The reference lab”For a method to become a skill, you need to be able to trigger the failures and practise catching them, like a flight simulator. The target is an executable environment that reproduces each silent failure and shows how to detect it. This full lab is in preparation: it is not published yet. The configurations and rules in this method have not yet been run there.
Two labs already exist on the public labstraining forge.
The first, labs/genaiotel, is a RAG application instrumented with the OTel GenAI conventions. The model and embeddings are simulated, so no API key is needed. A Collector computes the cost of each span (with OTTL) and derives per-model metrics (spanmetrics). VictoriaMetrics and vmalert carry the cost rules and the alerts on latency, cost, errors, input tokens, weak retrieved context and evaluated quality. Jaeger and Grafana are there for exploration. An evaluation pass sends results to Phoenix. It covers part of scenarios 2 and 3 below. It contains no agent, no MCP and no Wilson bound sidecar.
The second, labs/aiobsagent, is a read-only alert investigation agent written in Go. It fences third-party content against prompt injection and keeps a chained journal, where each entry is linked to the previous one so that none can be deleted unnoticed. It illustrates the injection guardrails and least privilege of the agent and MCP grids in Part III, but reproduces none of the four scenarios below.
The full lab, in preparation, will have the following structure.
labstraining/observability-genai/ docker-compose.yml # OTel collector, VictoriaMetrics, VictoriaLogs, Grafana, Langfuse otel-collector.yaml # reference config (part II) rules/ # MetricsQL recording rules and alerts dashboards/ # provisioned Grafana dashboards agent-demo/ # OTel GenAI instrumented agent (Python) scenarios/ # reproducible failure scenarios evals/ # golden set, RAGAS or DeepEval harnessIt plans one teaching scenario per silent failure.
- Agent loop. The agent goes round in circles and cost per task explodes. The learner enables loop detection and the cost alert, then replays the scenario.
- Prompt bloat. Input tokens grow gradually; only the dedicated recording rule makes it visible.
- Hallucination from degraded RAG context. Retrieval is degraded, faithfulness drops and the sampled evaluation catches it while latency stays green.
- MCP server mutation. A tool changes definition between two calls; mutation detection and tool pinning catch it.
28. Checklists per axis
Section titled “28. Checklists per axis”These lists work as a review grid, for example at the end of a quarter. They follow the four axes of Part I and add organization.
Reliability
Section titled “Reliability”- Production sampling evaluation active
- Golden set built, judge agreement tracked, judge versioned
- Quality SLO with confidence interval
- Regression tests in continuous integration
- Drift detection configured
-
finish_reasonsdistribution monitored
Availability
Section titled “Availability”- Service SLOs per layer
- Health checks for inference, vector store, MCP servers
- Tested multi-provider fallback
- Graceful degradation on tool unavailability
- Tokens and cost traced with an attribution dimension
- Budgets and alerts at 80 percent
- Prompt bloat and agent loop monitoring
- Cache hit rate measured and levers active
Compliance
Section titled “Compliance”- Content not captured by default, opt-in capture controlled
- Masking and redaction at the collector
- Content and metadata separation, distinct retention
- Immutable audit trail on invocations and access
- MCP security events escalated to the SOC
- AI Act, NIS2, GDPR evidence matrix documented
- Self-hosted backends if sovereignty required
Organization
Section titled “Organization”- RACI established over the eight capabilities
- Quality on-call defined, mitigation playbooks written
For Monday morning
Section titled “For Monday morning”Do phase 0 in half a day: a whiteboard, the components, the flows, the sensitive data. Then tick the checklists above honestly. In my view, the empty boxes on the reliability axis are the ones that matter most, because they belong to the silent failures.
Revised on 2 October 2026: the full lab of the method is presented as in preparation, with a precise pointer to the two existing public labs.
Revised on 25 October 2026: page rewritten for a wider audience (phases written as prose, terms explained), with no change of substance.