Quiz: LLM observability

Reading mode

EngineerDecision-maker

Ten questions to check what you remember from the course. Each answer is corrected immediately, with an explanation and a link to the relevant lesson. Nothing is sent anywhere: your best score stays in this browser.

1. Question 1 of 10Why can a classic APM dashboard show perfect latency and zero errors while the assistant gives wrong answers to some users?One answer

   • Because APM probes cannot measure the latency of a streaming call.
   • Because LLM errors are semantic: the model is politely wrong, with a 200 OK.
   • Because LLM providers deliberately hide their 5xx error codes.

   Check

2. Question 2 of 10What are the three LLM observation axes that must be measured together?Several answers possible

   • Quality (relevance, faithfulness, hallucinations)
   • GPU utilization rate
   • Cost (tokens, euros per request, cache rate)
   • Performance (TTFT, p95 and p99 latency, error rate)

   Check

3. Question 3 of 10The OpenTelemetry GenAI conventions all have Development status. Which practices does module 2 recommend?Several answers possible

   • Translate attributes in the Collector rather than in dashboards and alerts.
   • Put the full prompt and response in a span attribute to make diagnosis easier.
   • Keep in-house extensions in your own namespace, for example app.*.
   • Wait for the stable version before instrumenting anything.

   Check

4. Question 4 of 10In a RAG pipeline, faithfulness is low while context recall is high. Where does the problem most likely lie?One answer

   • In generation: the context held the information, but the answer does not stick to it.
   • In retrieval: the right documents were not found.
   • In the embedding step, because it takes about 150 ms.

   Check

5. Question 5 of 10During a code cleanup, top_k goes from 5 to 1 and the context becomes insufficient. Which drift is this, and where do you find its cause?One answer

   • Data drift, visible in the distribution of questions asked.
   • Concept drift, visible by comparing faithfulness against ground truth.
   • Pipeline drift, visible in the attributes of the retrieval span.

   Check

6. Question 6 of 10Bridge: business linesA collective agreement has changed and the HR assistant still quotes the old version, while the whole infrastructure is green. Who is best placed to spot this concept drift?One answer

   • On-call, thanks to latency and error rate alerts.
   • The business lines, who know a rule or a price has changed, backed by user feedback and a judge on a sample.
   • The model provider, which updates its knowledge with every release.

   Check

7. Question 7 of 10An LLM judge has known biases (verbosity, self-preference, position, confidence). Which countermeasures does module 4 propose?Several answers possible

   • Use the evaluated model itself as the judge, to save tokens.
   • A structured evaluation prompt and a rigid output format.
   • A judge different from the evaluated model.
   • Cross-sampling with human annotations.

   Check

8. Question 8 of 10Bridge: businessTo discuss margin and pricing with product and finance, how does lab 4 establish the cost of each request?One answer

   • It waits for the provider's monthly invoice and splits it pro rata by number of requests.
   • It counts consumed tokens client side, and converts them into euros only with a dated price list supplied by the user.
   • It treats a self-hosted model as free and leaves it out of the analysis.

   Check

9. Question 9 of 10Bridge: organizationAccording to module 5, who watches what, and how often?Several answers possible

   • On-call continuously watches latency, error and token budget alerts.
   • Finance reviews cost per customer and per feature every month.
   • The AI quality committee checks GPU load every day.
   • The product team reviews quality score, user feedback and drift every week.

   Check

10. Question 10 of 10Bridge: peopleIn the lab 4 SLO example for a support assistant, which line changes on-call work and sets an LLM SLO apart from a classic one?One answer

    • Error rate, at most 2% over 15 min.
    • p95 latency, at most 3 s.
    • The quality score, at least 0.6 average faithfulness over 1 h.

    Check

Start over

Going further

• Course overview
• Observing an LLM system
• Glossary
• Bridges

BridgesWhat this page changes beyond the technical team.

BusinessValue, cost, risk

Every request consumes tokens, measured per customer and per feature and converted into euros with your own dated price list: margin per customer and per feature becomes something to discuss with product and finance.

OrganizationWho decides, who owns

On-call, product, finance and the AI quality committee each watch their own indicators at their own pace, and the logging and monitoring the AI Act requires are prepared from instrumentation onward.

PeopleTeams, on-call, skills

A quality SLO changes on-call work: faced with a hallucination, you do not restart, you roll back to the previous version.

Business linesProduct, finance, risk, customer

Data, concept and pipeline drifts are detected with the business lines, who know when a rule or a price has changed.

Why bridges matter →