Skip to content

TechnicalExpert8 min read

Part IV. The SRE core: quality SLOs and error budgets

How do you commit to a quality level when you can only score part of the answers? Deterministic and probabilistic SLIs, quality SLOs with a Wilson bound, error budget and burn rate on quality, watching the judge.

For: engineers and SREs · architects · team managersPrerequisites: Have read part I; notions of SLO, error budget and burn rate.

An illustrative scenario. At a steering committee, the product lead shows a reassuring figure: the assistant’s average faithfulness score is 0.91 out of 1. A committee member asks how many answers are flatly wrong. Nobody can say. Another asks how many answers the average was computed on: about thirty, scored by hand the week before.

In my view, this is the most specific contribution of the method. The SRE discipline (site reliability engineering) has learned to commit to a numeric service level (the SLO) and to manage the margin for error it leaves, the error budget. That discipline applies to the quality of generative AI. One difference changes everything though: quality is not measured on every request. You estimate it on a sample, with an evaluator that can itself be wrong.

12. Deterministic versus probabilistic SLIs

Section titled “12. Deterministic versus probabilistic SLIs”

An SLI is the indicator you track to know whether the commitment holds. There are three families.

Service SLIs are deterministic: availability, latency, error ratio. You measure them on 100 percent of traffic, as in classical SRE. Each request either answered or did not, in so many milliseconds.

Quality SLIs are probabilistic: faithfulness, hallucination, refusal, relevance. You measure them on a sample, like a quality inspector who pulls a few bottles off the line instead of tasting every one. The evaluator that scores these answers is often a model itself, so it is non-deterministic: it may give the same answer two different scores.

Cost SLIs track the average cost per request, per session and per task, along with its drift.

The core difference fits in one word: a quality SLI is an estimate, with the uncertainty of any poll. Treating it as an exact measure is a frequent methodological mistake, covered as an anti-pattern (Part VIII, pitfall 3).

This part is the site’s reference for quality SLOs. The simplified thresholds shown elsewhere, for example in the labs or the guide, are pedagogical SLOs that point here.

Back to the committee’s 0.91. A mean of scores (“average faithfulness above 0.8”) does not tell you how many answers are bad. A few very bad answers hide behind many good ones; the mean barely moves when those extreme cases, called the tail, get worse. Nor does a mean give you a budget: you cannot derive an error budget or a burn rate from it.

An SLO counts good events among valid events. For quality, the good event is an answer whose score reaches a threshold set in advance. The SLI therefore becomes a proportion. Since that proportion is estimated on a sample, you compare it to the target through its lower confidence bound, meaning the lowest plausible value given the sample size.

Define a quality threshold tau (for example a minimum faithfulness score), then compute the share of answers that reach it.

SLI_quality = (number of sampled responses with score >= tau) / (sample size)

A poll of twenty people and a poll of a thousand can give the same percentage, yet you do not trust them equally. That is why you do not compare the raw value to the target. You compare the lower bound of the 95 percent confidence interval, computed with the Wilson method, which stays reliable on small samples.

SLO met <=> lower_bound_95(SLI_quality, n) >= target

For a margin of error e at 95 percent around a proportion p, the sample size is n ~= (1.96 / e)^2 * p * (1 - p). To estimate a faithfulness rate near 0.95 to within plus or minus 2 points, you need on the order of 450 to 500 evaluated responses per measurement window. The committee’s thirty answers were nowhere near.

I recommend stratified sampling: draw more from the at-risk segments (sensitive requests, new flows) rather than sampling uniformly across all traffic.

What sets the precision is the number of judgments per segment and per window, not the percentage of traffic. Illustrative example: 10 percent of a segment of 200 requests a day gives 20 judgments, far too few for the margin targeted above. 1 percent of a flow of 100,000 requests gives 1,000. The per-tier percentages in the guide, 5.7 are examples to convert this way: start from the margin you want, derive n per segment and per window, then each segment’s sampling rate.

Here is a typical wording for a quality SLO, ready to drop into a commitment document: “at least 95 percent of responses, estimated on a stratified sample of about 500 per day, score above the faithfulness threshold, with the lower bound of the 95 percent confidence interval above target, over a 7-day rolling window”.

The error budget works like a monthly spending allowance. If the target is 95 percent good answers, you allow yourself 5 percent bad ones: that is the allowance. The burn rate measures how fast you spend it.

quality_error_budget = 1 - target
burn_rate = (1 - SLI_quality) / (1 - target)

A burn rate of 1 consumes the budget exactly at the planned pace. Above 1, the allowance runs out before the end of the period.

You watch that pace over two windows at once, suited to the way quality moves. The short window (for example 1 h, with a high burn rate threshold) catches a sudden collapse after a prompt deployment or a model change. The long window (for example 24 h to 7 d, with a moderate threshold) catches slow drift, which is the characteristic illness of quality.

Do not copy these windows from latency. Quality moves slowly; its windows are longer than availability windows.

15. The evaluator problem (meta-observability)

Section titled “15. The evaluator problem (meta-observability)”

To score thousands of answers, teams often hand the job to another model: the LLM-as-judge. This is the blind spot of most setups, because that judge is itself a non-deterministic model that drifts. Picture a referee whose eyesight fades over the season while nobody checks: the scores change, the matches do not. If the judge drifts, the quality SLI is corrupted with nothing to show it. So you must observe the judge, hence the name meta-observability.

The method relies on a golden set, a reference set annotated by humans and stable over time. It plays the role of the standard weight kept at the office of weights and measures.

You then measure calibration, the agreement between the judge and the golden set. I recommend a chance-corrected measure such as Cohen’s kappa (see the guide, 5.4), tracked as a metric in its own right. A dropping agreement invalidates the quality measures.

You version the judge: the judge model version and the evaluation prompt are pinned. Any judge change is an event to trace, on the same footing as a production model change.

You recalibrate periodically, revalidating the judge against humans at a regular interval and after any version change.

That leaves the economics of ground truth. The bottleneck is human annotation, which is slow and costly. The strategy is to build the golden set once, use it to calibrate the judge, let the judge scale, then call on humans only for periodic revalidation and for cases where the judge is unsure.

The Wilson bound is not computed in MetricsQL. It lives in a quality SLO sidecar, a small program running alongside the rest. It reads the sampled scores (from the evaluation platform or the trace store), computes the bound over a rolling window and exposes it as an OTel metric. The SLO alert then reads that metric and fires when the lower bound drops below target, not on the point value.

For technical teams: computing the Wilson bound in Python
# quality SLO sidecar: Wilson lower bound exposed as an OTel metric
from math import sqrt
def wilson_lower(k, n, z=1.96):
if n == 0:
return 0.0
p = k / n
d = 1 + z*z/n
centre = p + z*z/(2*n)
margin = z * sqrt(p*(1-p)/n + z*z/(4*n*n))
return (centre - margin) / d
# k = sampled responses with score >= tau, n = sample size
# emitted as quality_slo.faithfulness.lower_bound (OTel gauge), outside
# the gen_ai.* namespace, reserved for the semantic conventions
slo_met = wilson_lower(k, n) >= target

The full lab of the method, in preparation (Part IX), will ship this sidecar. Its hallucination scenario is meant to drive the bound below target while latency stays green.

A RAG has a retrieval SLO and a generation SLO. For the user, the answer is only good if both hold at once: retrieval brought the right context and generation respected it. It is like a chain where every link must hold. Splitting the SLO lets you pin a degradation on the right link. In the same way, an agent has a task success SLO that combines the tool and model SLOs along its trajectory.

17. Quality gating in continuous integration

Section titled “17. Quality gating in continuous integration”

SLOs watch production. Their offline counterpart is gating, a checkpoint before release. Any change of prompt, model or tool version goes through an evaluation suite on a reference dataset before deployment. If the suite regresses beyond budget, the deployment is blocked. DeepEval, RAGAS and Promptfoo do this job.

In my view, a light evaluation that runs on every pull request beats a full quarterly evaluation you forget to run.

Look at your current quality indicator. If it is a mean, replace it with the share of answers above a threshold. Then count how many answers you score per day and per segment: below a few hundred, your figure is a poll too small to drive an alert. Finally, build a first golden set, even a modest one, so you can check your judge.


Revised on 4 October 2026: this part becomes the site’s reference for quality SLOs; added why a threshold on a mean is not an SLO, the judge sampling principle (number of judgments per segment and per window, with a pointer to guide 5.7) and chance-corrected agreement for calibration.

Revised on 25 October 2026: page rewritten for a wider audience (opening scenario, images, code folded away), with no change of substance.