Skip to content

TechnicalExpert3 min read

Is your AI's quality really proven?

Check whether a quality target is demonstrated on a sample: margin of uncertainty (Wilson interval), verdict against the target and required sample size.

For: engineers and SREs · business and productPrerequisites: Notions of SLOs. The statistics you need (proportion, margin of uncertainty) are explained on the page.

Reading mode

An illustrative scenario: the team building an AI assistant tells the steering committee that 98% of answers are faithful to the documents, above the 95% target. One question from the head of quality cools the room: “Out of how many answers?” Out of 50. The figure is true; the conclusion is not, at least not yet.

Think of an opinion poll. Asking 50 people gives a trend, with a margin of error so wide you cannot call a winner. With 1,000 people, the margin narrows. A quality target for an AI works the same way. It is phrased as an SLO (service level objective), a promise with a number: for example, 95% of answers must stay faithful to the sources. It is measured on a sample of evaluated answers: k answers judged good out of n. The proportion k / n is only an estimate, like a poll result.

The simulator helps settle one question: with the evaluated answers you have, can you say the target is met, that it is missed or neither? Comparing the observed proportion directly with the target leads to declaring the target met by luck on a small sample. The right comparison uses the range of uncertainty, called the confidence interval. More precisely, it uses its lower bound. The simulator computes it with the Wilson method, which holds up well on small samples as well as when the proportion is close to 0 or 1.

Sample
SLO and precision

Result

Observed proportion
Wilson interval
Verdict
Size for margin e
n ≈ (z / e)² p (1 - p), observed p
Size to demonstrate the target
if the observed proportion holds
Confidence interval and target
How the interval narrows as n grows, at constant proportion
Wilson intervalObserved proportionTargetcurrent n

Wilson score interval, without continuity correction. Assumption: independent responses and a representative sample; stratified sampling requires per-stratum weighting. The sizing formula n ≈ (z / e)² p (1 - p) is the classic normal approximation: it underestimates the need when p is close to 0 or 1.

  1. Start from the defaults: 475 good answers out of 500, exactly 95%, for a 95% target. The range runs from about 92.7% to 96.6%. The target lies inside: the verdict is “not demonstrated”. A proportion equal to the target never proves the target.
  2. Set k = 49 and n = 50, the case in the scenario. You observe 98%, but the lower bound drops to about 89.5%. An excellent score on a small sample demonstrates nothing.
  3. Go back to n = 500 and set k = 460. You observe 92% with an upper bound around 94.1%, below target. The target is missed and this time the sample is large enough to say so.
  4. Type 96 in the proportion field (with n = 500). The target is not demonstrated yet. The “Size to demonstrate the target” tile shows you would need about 1,800 evaluated answers if that proportion holds. Switch confidence to 99%: you need more than 3,000.

The target is met if the lower bound of the range is above it. It is violated if the upper bound is below it. In every other case, it is not demonstrated: you need more data, not a conclusion. In my view, that third verdict is the most useful of the three, because it stops you both from declaring an incident on noise and from reassuring people on too few answers.

Evaluating answers costs time or money. Sample size is therefore decided in advance, together with the target and the confidence level, by the service owner.

For technical teams: the formulas

The Wilson interval, with z = 1.645, 1.96 or 2.576 for 90, 95 or 99% confidence:

center = (p + z² / 2n) / (1 + z² / n)
half-width = z / (1 + z² / n) × sqrt( p (1 - p) / n + z² / 4n² )

To size the sample around a proportion p with a margin e, the approximation n ≈ (z / e)² p (1 - p) gives the order of magnitude: about 460 answers for p = 0.95 and e = 2 points at 95%.

The full method, including how to run the bound in a dedicated component and quality error budgets, is described in the SRE core: quality SLOs. See also the error budget simulator, the method calculator and the LLM, RAG and MCP page.