OrganizationPractitioner4 min read
Module 5: industrialize
Moving from the labs to production: a 30-, 60- and 90-day plan, who watches what in the organization and what the AI Act requires from the observability of an AI system.
For: architects · team managers · finance, risk and compliancePrerequisites: Have followed the previous modules and labs of the course.
The labs are over. The next day, in a steering meeting, someone asks: “Fine, so where do we start, who owns it and are we compliant with the AI Act?” (an illustrative scenario). This module answers those three questions, in that order.
The 30-, 60- and 90-day plan
Section titled “The 30-, 60- and 90-day plan”You build LLM observability like a house: foundations, then walls, then roof. Maturity grows in three stages, in this order. There is no point automating reindexing or a model change before you have a reliable dashboard.
| Deadline | Theme | Deliverables |
|---|---|---|
| 30 days | instrument | OpenTelemetry SDK on the main chain, GenAI conventions applied, LLM Overview dashboard, breakdown by customer and feature |
| 60 days | evaluate | faithfulness measured continuously, automatic drift detection, judge on 1% of traffic, self-hosted Phoenix |
| 90 days | govern | SLOs formalized and signed, one procedure per alert, monthly quality and cost review, logs and documentation required by the AI Act in place if the system is in scope |
The guide offers a maturity scale with six levels, from 0 to 5. Here is where this plan puts you on that scale:
| Deadline | Guide levels targeted |
|---|---|
| 30 days | levels 1 and 2: basic signals (tokens, cost, latency, errors), then structured tracing with the GenAI conventions and attributes per customer and feature |
| 60 days | level 3 (online evaluation: faithfulness measured continuously, judge on a sample) and a first element of level 4 (drift detection) |
| 90 days | consolidation of levels 3 and 4: SLOs, procedures and the monthly review belong to governance, which the guide’s scale does not measure as such; level 5 (closed loop) is beyond this plan |
These deadlines hold for a narrow scope, one main chain like the one in the labs. For a full program, the guide considers level 3 realistic in six months.
Who watches what?
Section titled “Who watches what?”Each role has its indicators and its rhythm. On-call, the team that steps in during incidents, follows latency, error and token budget alerts continuously. The product team looks at the quality score, user feedback and drift every week. Finance tracks cost per customer and per feature every month. An AI quality committee also meets monthly to review trends, incidents and change decisions (prompt, retrieval, model version).
The RACI matrix, the table that states for each task who does it, who is accountable, who is consulted and who is informed, is detailed in part VII of the method along with the organization of quality on-call. Organization models in general are covered in the organization section.
What the AI Act requires
Section titled “What the AI Act requires”The AI Act, the European regulation on artificial intelligence, does not place the same obligations on everyone. They depend first on your role: the provider places the system on the market, the deployer uses it. They depend next on the system’s risk level. Record-keeping, technical documentation and post-market monitoring cover high-risk systems; their deployer keeps automatically generated logs for at least six months. Article 50, on transparency, covers any system that interacts with people.
The timeline comes from Regulation (EU) 2026/1744: Article 50 has applied since 2 August 2026, obligations for high-risk systems of Annex III apply from 2 December 2027 and those of Annex I from 2 August 2028.
An OpenTelemetry stack provides the technical material: logs, measurements, history. It does not do everything. Qualifying the system and your role, documenting it, then maintaining the setup over time remain to be done. The detailed table of requirements, article by article, with the role concerned and what the stack covers, is in part VI of the method.
What the course covered and what you do on Monday
Section titled “What the course covered and what you do on Monday”The top-tier indicators change: drift, faithfulness, hallucination and time to first token come before CPU, p95 latency and error rate. GenAI conventions make instrumentation portable, provided you absorb their changes in the Collector. A self-hosted stack (in the kit: OpenTelemetry, VictoriaMetrics, Tempo, Grafana and Phoenix) covers these needs; other combinations, open source or commercial, do so too and OpenTelemetry instrumentation keeps the choice reversible.
On Monday, I recommend starting small: pick a single chain (the one that matters most to your users) and apply the first step of the plan to it, the 30-day one. Maturity is built in this order: instrument, evaluate, then govern.
Going further
Section titled “Going further”- Observing an LLM system: the production stack, MCP, privacy.
- GenAI observability method: quality SLOs, quantified impact, compliance, organization.
- Beyond the LLM: GPUs and model quality: GPU infrastructure and statistical drift detection.
- GenAI conventions: the
semantic-conventions-genairepository (since version 1.42 of the conventions). - Phoenix, RAGAS, Evidently and VictoriaMetrics documentation.
Revised on 2 October 2026: AI Act section corrected (provider and deployer roles, Article 26 and log retention, Article 50 applicable to any conversational system, timeline from Regulation (EU) 2026/1744), link to the new GenAI conventions repository.
Revised on 4 October 2026: milestones of the 30-, 60- and 90-day plan mapped onto the guide’s maturity levels, “retraining” replaced by the actual changes of an LLM called through an API (prompt, retrieval, model version); “Who watches what” and the AI Act section summarized with pointers to parts VII and VI of the method, which now holds the table of requirements by role.