Quiz: GenAI observability method

Reading mode

EngineerDecision-maker

Ten questions to check what you retain from the course. Each answer is corrected immediately, with an explanation and a link to the relevant lesson. Nothing is sent: your best score stays in this browser.

1. Question 1 of 10According to the silent failure model, what is the right question to ask about a signal you do not measure?One answer

   • Is the service going down?
   • How long does the failure accumulate before we see it?
   • How many metrics can we export to the backend?
   • Which model provider is the fastest?

   Check

2. Question 2 of 10A trusted MCP server changes the definition of one of its tools between two calls (rug pull). Which signal from the MCP grid makes this change visible?One answer

   • Latency and errors per tool and per server
   • Cache hit rate
   • Server mutation detection and tool pinning by hash

   Check

3. Question 3 of 10Which of these practices match how the method defines and monitors a quality SLO?Several answers possible

   • Compare the lower bound of the 95 percent confidence interval (Wilson bound) to the target, not the point value
   • Over sample the at risk segments, such as sensitive requests and new flows
   • Measure quality on 100 percent of traffic, exactly like availability
   • Use a long window (24 h to 7 d) with a moderate burn rate to catch slow drift

   Check

4. Question 4 of 10The agreement rate between your LLM judge and the golden set drops sharply. What should you conclude?One answer

   • Nothing in particular, as long as the quality SLI stays above target
   • That the golden set must be expanded at every measurement window
   • That the production model must have changed version
   • That the quality measures produced by this judge are no longer reliable and it must be recalibrated

   Check

5. Question 5 of 10Bridge: businessTo defend an observability budget in front of finance, how does the method estimate the value of a signal?One answer

   • The failure's accumulation rate multiplied by the gap between the detection delay without the signal and with it
   • The number of dashboards produced multiplied by their number of panels
   • The license cost of the observability tool divided by the request volume

   Check

6. Question 6 of 10Bridge: businessIn the worked example of part V (50,000 calls per day), what does an achievable 30 percent cache hit rate left unused represent?One answer

   • About 9,600 calls per month
   • About 1,800 million input tokens per month
   • About 450,000 avoidable calls per month, that is 30 percent of the base cost
   • Nothing, since no latency or error alert fires

   Check

7. Question 7 of 10Bridge: organizationIn the method's RACI, who is accountable (A) for quality SLOs?One answer

   • Platform / SRE
   • ML / Data Science
   • Product
   • Compliance / DPO

   Check

8. Question 8 of 10Bridge: peopleYour team is setting up quality on-call. Which actions belong to the mitigation playbook described by the method?Several answers possible

   • Roll back to the previous prompt or model version
   • Restart the service, as for an availability outage
   • Disable the degraded feature rather than serve false quality
   • Write the fallback procedures after the first incident, based on its lessons learned

   Check

9. Question 9 of 10Bridge: business linesYour compliance team asks how you demonstrate the security of your supply chain under NIS2. Which signal does the evidence matrix tie to this obligation?One answer

   • Human handoff rate and validation points
   • Quality SLOs and drift detection
   • Redaction at the collector and opt-in content
   • Tool pinning, mutation detection and authentication failures

   Check

10. Question 10 of 10Where and how should the content of exchanges (prompts, responses) be handled in telemetry, to avoid two costly anti-patterns?Several answers possible

    • Put it in a metric label so dashboards can be filtered by prompt
    • Keep it out of metric labels, which stay limited to low cardinality dimensions (model, operation, error type), and, if its capture is enabled (opt-in), place it outside indexed span attributes (event or dedicated log), masked
    • Do not capture it by default, redact it at the collector and, if it must be kept, store it externally encrypted with a reference on the span

    Check

Start over

Going further

• Course overview
• The economic exposure calculator
• Glossary
• Bridges between the planes

---

Revised on 4 October 2026: explanations of the questions on dashboards and content updated (pointers to chapter 9 of the guide, content capture rule aligned with the OpenTelemetry specification).

BridgesWhat this page changes beyond the technical team.

BusinessValue, cost, risk

You can put a figure on a blind spot (accumulation rate times detection delay) and justify observability in euros in front of finance.

OrganizationWho decides, who owns

You know who owns what: the RACI splits eight capabilities between platform, data science, security, product and compliance.

PeopleTeams, on-call, skills

You know why quality on-call differs from classical SRE on-call and why its playbooks are written before the incident.

Business linesProduct, finance, risk, customer

You can tie an AI Act, NIS2 or GDPR obligation to the telemetry signal that proves it.

Why bridges matter →