OrganizationExpert3 min read
Part VII. Organization: who owns observability
Who does what in GenAI observability? A RACI over eight capabilities and an on-call rotation dedicated to answer quality.
For: architects · team managersPrerequisites: Know the capabilities described in parts I to VI of the method.
An illustrative scenario. On a Friday at 11 pm, the quality SLO alert fires: the assistant’s faithfulness bound drops below target. The platform on-call engineer looks at the dashboards. Everything is green; the servers respond. He does not know whether to roll back to the old prompt, or who is allowed to make that call. The data scientists are not on call. The decision waits until Monday.
In my view, this is where many implementations fail. GenAI observability straddles five functions: platform and SRE, ML and data science, security, product, compliance. Without a clear owner nobody watches the right signals, or everyone assumes someone else does.
24. RACI
Section titled “24. RACI”A RACI is a table that says, for each activity, who does what. The four letters stand for Responsible (the one who does the work), Accountable (the one who answers for it and decides), Consulted (the one you ask before deciding) and Informed (the one you tell afterwards). You can read it like the roles in a football team: who takes the penalties, who is captain, who you check with on the bench.
The table covers eight capabilities. The SOC is the security operations center; the DPO is the data protection officer. FinOps means managing cost.
| Capability | Platform / SRE | ML / Data Science | Security / SOC | Product | Compliance / DPO |
|---|---|---|---|---|---|
| OTel instrumentation | A, R | C | I | I | I |
| Service SLOs | A, R | C | I | C | I |
| Quality SLOs | C | A, R | I | C | I |
| Evals and golden set | C | A, R | I | C | C |
| GenAI FinOps | A, R | C | I | C | I |
| MCP security | C | I | A, R | I | C |
| Audit and compliance | C | C | C | I | A, R |
| Quality on-call | R | A, R | I | C | I |
Look at the last row. The platform executes (R), yet the ML team is both responsible and accountable (A, R). In the Friday night scenario, they are the ones who should have been reachable.
25. Quality on-call
Section titled “25. Quality on-call”This is new compared with classical SRE: you need on-call coverage for quality regressions, not only for availability. One big difference separates the two: you do not restart a hallucination. Rebooting a server does not make a model any more accurate.
The response plan, called a playbook, is therefore specific. The possible moves range from lightest to most drastic. You can roll back to the previous prompt or model version. You can tighten the guardrails, meaning the filters on inputs and outputs. You can switch to a fallback model. As a last resort, you disable the degraded feature: I recommend preferring a switched-off service to a wrong answer served with confidence.
These playbooks must exist before the incident, with a named owner in the RACI. At 11 pm on a Friday, nobody should have to wonder who decides.
For Monday morning
Section titled “For Monday morning”Get the five functions in a room for an hour and fill in the RACI with names, not just teams. Then write the first quality on-call playbook, even a short one: which alert, who gets called, which fallback move and who authorizes it.
Revised on 25 October 2026: page rewritten for a wider audience (opening scenario, RACI explained, playbook written as prose), with no change of substance.