Skip to content

OrganizationPractitioner4 min read

Module 5: industrialize

Moving from the labs to production: a 30-, 60- and 90-day plan, who watches what in the organization and what the AI Act requires from the observability of an AI system.

For: architects · team managers · finance, risk and compliancePrerequisites: Have followed the previous modules and labs of the course.

The labs are over. The next day, in a steering meeting, someone asks: “Fine, so where do we start, who owns it and are we compliant with the AI Act?” (an illustrative scenario). This module answers those three questions, in that order.

You build LLM observability like a house: foundations, then walls, then roof. Maturity grows in three stages, in this order. There is no point automating reindexing or a model change before you have a reliable dashboard.

DeadlineThemeDeliverables
30 daysinstrumentOpenTelemetry SDK on the main chain, GenAI conventions applied, LLM Overview dashboard, breakdown by customer and feature
60 daysevaluatefaithfulness measured continuously, automatic drift detection, judge on 1% of traffic, self-hosted Phoenix
90 daysgovernSLOs formalized and signed, one procedure per alert, monthly quality and cost review, logs and documentation required by the AI Act in place if the system is in scope

The guide offers a maturity scale with six levels, from 0 to 5. Here is where this plan puts you on that scale:

DeadlineGuide levels targeted
30 dayslevels 1 and 2: basic signals (tokens, cost, latency, errors), then structured tracing with the GenAI conventions and attributes per customer and feature
60 dayslevel 3 (online evaluation: faithfulness measured continuously, judge on a sample) and a first element of level 4 (drift detection)
90 daysconsolidation of levels 3 and 4: SLOs, procedures and the monthly review belong to governance, which the guide’s scale does not measure as such; level 5 (closed loop) is beyond this plan

These deadlines hold for a narrow scope, one main chain like the one in the labs. For a full program, the guide considers level 3 realistic in six months.

Each role has its indicators and its rhythm. On-call, the team that steps in during incidents, follows latency, error and token budget alerts continuously. The product team looks at the quality score, user feedback and drift every week. Finance tracks cost per customer and per feature every month. An AI quality committee also meets monthly to review trends, incidents and change decisions (prompt, retrieval, model version).

The RACI matrix, the table that states for each task who does it, who is accountable, who is consulted and who is informed, is detailed in part VII of the method along with the organization of quality on-call. Organization models in general are covered in the organization section.

The AI Act, the European regulation on artificial intelligence, does not place the same obligations on everyone. They depend first on your role: the provider places the system on the market, the deployer uses it. They depend next on the system’s risk level. Record-keeping, technical documentation and post-market monitoring cover high-risk systems; their deployer keeps automatically generated logs for at least six months. Article 50, on transparency, covers any system that interacts with people.

The timeline comes from Regulation (EU) 2026/1744: Article 50 has applied since 2 August 2026, obligations for high-risk systems of Annex III apply from 2 December 2027 and those of Annex I from 2 August 2028.

An OpenTelemetry stack provides the technical material: logs, measurements, history. It does not do everything. Qualifying the system and your role, documenting it, then maintaining the setup over time remain to be done. The detailed table of requirements, article by article, with the role concerned and what the stack covers, is in part VI of the method.

What the course covered and what you do on Monday

Section titled “What the course covered and what you do on Monday”

The top-tier indicators change: drift, faithfulness, hallucination and time to first token come before CPU, p95 latency and error rate. GenAI conventions make instrumentation portable, provided you absorb their changes in the Collector. A self-hosted stack (in the kit: OpenTelemetry, VictoriaMetrics, Tempo, Grafana and Phoenix) covers these needs; other combinations, open source or commercial, do so too and OpenTelemetry instrumentation keeps the choice reversible.

On Monday, I recommend starting small: pick a single chain (the one that matters most to your users) and apply the first step of the plan to it, the 30-day one. Maturity is built in this order: instrument, evaluate, then govern.

Revised on 2 October 2026: AI Act section corrected (provider and deployer roles, Article 26 and log retention, Article 50 applicable to any conversational system, timeline from Regulation (EU) 2026/1744), link to the new GenAI conventions repository.

Revised on 4 October 2026: milestones of the 30-, 60- and 90-day plan mapped onto the guide’s maturity levels, “retraining” replaced by the actual changes of an LLM called through an API (prompt, retrieval, model version); “Who watches what” and the AI Act section summarized with pointers to parts VII and VI of the method, which now holds the table of requirements by role.