Reading mode Engineer Decision-maker
Decision-maker reading: bridges first.
Ten questions to check what you remember from the guide. Each answer is corrected right away, with an explanation and a link to the relevant lesson. If a question stumps you, reread the linked lesson before trying again. Nothing is sent anywhere: your best score stays in this browser.
Question 1 of 10 Classical observability has three signals: logs, metrics, traces. Which two signals does AI observability add? Several answers possible Review: Lesson 1: Foundations
Question 2 of 10 Bridge: business lines An AI feature is up and responds within its latency budget. Which statements are true? Several answers possible Review: Lesson 1: Foundations
Question 3 of 10 According to the maturity roadmap, which level is a realistic target for a six-month program? One answer Review: Lesson 3: The maturity roadmap
Question 4 of 10 What is the first of the seven implementation steps? One answer Review: Lesson 4: Implementation in seven steps
Question 5 of 10 In the OpenTelemetry Collector, where should the redaction processor sit relative to the batch processor? One answer Review: Lesson 4: Implementation in seven steps
Question 6 of 10 Which traces must be evaluated at 100%, whatever the evaluation sampling tier? Several answers possible Review: Lesson 5: Evaluator design
Question 7 of 10 You set a Cohen's kappa threshold of 0.7 in advance. On your calibration set, your LLM judge only reaches a kappa of 0.45 with human judgment. What does the guide recommend? One answer Review: Lesson 5: Evaluator design
Question 8 of 10 Bridge: business In the example at 100 requests per second in chapter 7, how many tokens does a single LLM judge evaluator, on 10% of traces at 1,500 tokens per judgment, consume per day? One answer About 13 million tokens That would be the case with a 0.1% sample: 100 x 86,400 x 0.001 x 1,500, about 13 million.
About 1.3 billion tokens Yes: 100 x 86,400 x 0.10 = 864,000 judgments per day, times 1,500 tokens, about 1,296 million tokens, every day. Hence the case for tiering and sampling evaluators.
About 1 TB About 1 TB is the volume of traces kept in hot storage over 30 days, not a token consumption.
Review: Lesson 7: Operating the platform
Question 9 of 10 Bridge: organization In the proposed split of responsibilities, what does the ML team own? Several answers possible Review: Lesson 7: Operating the platform
Question 10 of 10 Bridge: people Evaluations run, scores are stored, but nothing improves. What remedy does the guide propose? One answer Review: Lesson 9: Anti-patterns and common pitfalls
Start over
Bridges What this page changes beyond the technical team.
BusinessValue, cost, risk You can start from the questions of finance, product and leadership. You also know that the LLM judge weighs more on the budget than storage does.
OrganizationWho decides, who owns You know who owns what: platform the stack, ML the evaluators and thresholds, security redaction and audit, while everyone owns the weekly review.
PeopleTeams, on-call, skills You know the closed loop is a team ritual: without a regular review of failed traces, the platform is just an archive.
Business linesProduct, finance, risk, customer You know a system can be up, fast and wrong; only the business lines can write the rubric that says what a good answer is. Why bridges matter →