Quiz: AI observability

Reading mode

EngineerDecision-maker

Ten questions to check what you remember from the guide. Each answer is corrected immediately, with an explanation and a link to the relevant lesson. Nothing is sent anywhere: your best score stays in this browser.

1. Question 1 of 10Classical observability has three signals: logs, metrics, traces. Which two signals does AI observability add?Several answers possible

   • Structured logs
   • Evaluations
   • Embeddings
   • User feedback

   Check

2. Question 2 of 10Bridge: business linesAn AI feature is up and responds within its latency budget. Which statements are true?Several answers possible

   • A 200 OK response guarantees there is no hallucination.
   • The system can be up, fast and wrong: you need a second axis, output quality.
   • A hundred perfect answers guarantee that the hundred and first will be perfect too.
   • Domain-specific quality requires custom evaluators, whose rubric must be written by the people who know what a good answer is.

   Check

3. Question 3 of 10According to the maturity roadmap, which level is a realistic target for a six-month program?One answer

   • Level 1: basic telemetry
   • Level 5: closed loop
   • Level 3: online evaluation
   • Level 2: structured tracing

   Check

4. Question 4 of 10What is the first of the seven implementation steps?One answer

   • Write down the questions the platform must answer
   • Instrument every call with as many attributes as possible
   • Deploy an OpenTelemetry Collector
   • Pick a commercial LLM observability product

   Check

5. Question 5 of 10In the OpenTelemetry Collector, where should the redaction processor sit relative to the batch processor?One answer

   • After batch, to redact whole batches at once
   • Before batch: redact first, batch second
   • Order does not matter as long as the backend also redacts

   Check

6. Question 6 of 10Which traces must be evaluated at 100%, whatever the evaluation sampling tier?Several answers possible

   • Traces with a high score
   • Error traces
   • Traces that received negative user feedback
   • A random 10% sample of traffic

   Check

7. Question 7 of 10You set a Cohen's kappa threshold of 0.7 in advance. On your calibration set, your LLM judge only reaches a kappa of 0.45 with human judgment. What does the guide recommend?One answer

   • Put the judge in production and watch the trend
   • Evaluate more production traces to smooth out the noise
   • Switch to self-consistency and consider the problem solved
   • Rework the judge prompt or change the judge model, then recalibrate

   Check

8. Question 8 of 10Bridge: businessIn the example at 100 requests per second in chapter 7, how many tokens does a single LLM judge evaluator, on 10% of traces at 1,500 tokens per judgment, consume per day?One answer

   • About 13 million tokens
   • About 1.3 billion tokens
   • About 1 TB

   Check

9. Question 9 of 10Bridge: organizationIn the proposed split of responsibilities, what does the ML team own?Several answers possible

   • Redaction rules and access policies
   • The evaluator catalog
   • Collectors, backends and alert routing
   • Quality thresholds and the regression set

   Check

10. Question 10 of 10Bridge: peopleEvaluations run, scores are stored, but nothing improves. What remedy does the guide propose?One answer

    • Add more attributes to every span
    • Build more detailed dashboards
    • Formalise a weekly review in which failed traces become regression set entries
    • Replace the stack with a commercial tool

    Check

Start over

Going further

• Guide overview
• Glossary
• Bridges
• Observing an LLM system: deployment playbooks (LLM, RAG, MCP)

BridgesWhat this page changes beyond the technical team.

BusinessValue, cost, risk

You can start from the questions of finance, product and leadership, and spot that the LLM judge, more than storage, drives the budget.

OrganizationWho decides, who owns

You know who owns what: platform the stack, ML the evaluators and thresholds, security redaction and audit, and everyone the weekly review.

PeopleTeams, on-call, skills

You know the closed loop is a team ritual: without a regular review of failed traces, the platform is just an archive.

Business linesProduct, finance, risk, customer

You know a system can be up, fast and wrong, and that only the business lines can write the rubric that says what a good answer is.

Why bridges matter →