Skip to content

TechnicalPractitioner9 min read

5. Evaluator design

Evaluator catalog, evaluator types, LLM-as-judge prompt patterns, calibration, score aggregation, judge ensembles and evaluation sampling.

For: engineers and SREsPrerequisites: Have read chapter 4 of the guide, especially step 5.

Back to the insurer’s assistant in our illustrative scenario. The team now wants to know, for each answer, whether it really rests on the policy. Nobody can reread thousands of conversations a day. So you need an automatic grader: an evaluator.

An evaluator is a program that scores an answer against a criterion. The criterion can be simple (is the answer valid JSON?) or subtle (does the answer say the same thing as the policy?). Evaluators are the most domain-specific part of an AI observability platform.

Off-the-shelf evaluators exist in Ragas, DeepEval or Phoenix evals. They cover the generic dimensions: faithfulness, toxicity, format. Domain-specific quality needs custom evaluators. These almost always take the form of an LLM judge, a model asked to grade another model’s answers using a carefully written rubric. This chapter covers the practical mechanics.

Here is a practical starting set for most deployments. You do not need to set it all up at once: the appendix suggests starting with three evaluators.

EvaluatorApplies toWhat it measures
FaithfulnessRAGWhether the answer is supported by the retrieved context
Answer relevancyLLM, RAGWhether the answer addresses the question asked
Context precisionRAGProportion of retrieved chunks that are relevant
Context recallRAGProportion of relevant chunks that were retrieved
ToxicityLLMPresence of harmful, harassing or hateful content
PII leakageLLMPresence of personal identifying data in output
Format complianceLLMConformance of output to a required schema
Hallucination rateLLM, RAGFabricated entities or claims not present in source
Refusal correctnessLLMWhether the model refused (or did not refuse) appropriately
Tool selectionAgentWhether the correct tool was chosen at each step
Argument validityAgentWhether tool arguments matched the schema
Loop boundAgentWhether the agent exceeded an acceptable iteration count
Domain rubricAnyCustom criteria specific to the application

There are three ways to build an evaluator, from the cheapest and most rigid to the most expensive and most flexible.

Rule-based evaluators rely on regular expressions (search patterns in text), schema validation or keyword presence. They are fast, cheap and always give the same result for the same input. But they only see the surface of the text.

Classifier-based evaluators use small models trained for one specific task: detecting toxicity, personal data, sentiment. They are fast at runtime and deterministic too. In return, they need labeled training data.

The LLM judge, finally, is a model that scores another model’s output against a written rubric. It is flexible, expensive and does not always give the same score. It is the only one that can judge meaning.

The common mistake is to hand everything to the LLM judge. Spotting a card number or a banned word is a regex job. Checking that a structured output respects its format is a job for JSON Schema, the standard language for describing JSON documents. I recommend keeping the LLM judge for criteria that genuinely require understanding the text.

Three patterns cover most use cases. Each one is summarized here and the full prompt sits in a collapsible section.

The judge answers 0 or 1. It is used to detect hallucinations or check refusal correctness. For format compliance of structured output, a JSON Schema check is still better; keep the judge for format criteria that require reading for meaning, such as tone or register.

This prompt asks the judge to return grounded as 1 only if every factual claim is supported by the context. An inference not stated in the context does not count as grounded.

For technical teams: the binary classification prompt
SYSTEM:
You are an evaluator. Given a question, a context and an answer, return
exactly one JSON object with the schema:
{"grounded": 0 or 1, "reason": "one short sentence"}
A grounded answer is one where every factual claim is supported by the
context. Inferences that are not stated in the context are not grounded.
USER:
Question: {question}
Context: {context}
Answer: {answer}

The judge returns a numeric score against a written rubric, like an examiner with a marking scheme. It is used for relevance, completeness or tone.

The rubric runs from 1 (off-topic) to 5 (correct, complete and well structured), through 2 (partial answer, key information missing), 3 (answer with errors or omissions) and 4 (correct and complete, but rough in form).

For technical teams: the rubric-based scoring prompt
SYSTEM:
Score the answer on a scale of 1 to 5 against the following rubric:
1 = irrelevant or off-topic
2 = partially addresses the question, missing key information
3 = addresses the question but with errors or omissions
4 = correct and complete, lacks polish
5 = correct, complete, well-structured
Return JSON: {"score": int, "reason": "short justification"}
USER:
Question: {question}
Answer: {answer}

The judge picks the better of two answers. It is used for A/B testing of prompts or models. It is easier to say which of two answers is better than to give one answer an absolute score, so this pattern is less sensitive to calibration.

The judge decides on factual correctness first, then completeness, then clarity.

For technical teams: the pairwise comparison prompt
SYSTEM:
Given a question and two candidate answers, return JSON:
{"winner": "A" or "B" or "tie", "reason": "short justification"}
Judge on factual correctness first, then completeness, then clarity.
USER:
Question: {question}
Answer A: {answer_a}
Answer B: {answer_b}

Pairwise judges have a position bias: they tend to favor the answer shown first, or second depending on the model. Run each pair twice with A and B swapped. Only declare a winner when both runs agree; otherwise, count a tie.

A judge has no absolute score. Its scores depend on the judge model, the prompt and the temperature. Calibrating means checking that it grades the way a human would, the way two teachers mark the same papers to see whether they agree.

The process has four stages.

  1. Build a calibration set whose right answers were established by humans. Order of magnitude: a few dozen to a few hundred traces (for example 50 to 200), with enough positive and negative cases. The rarer or subtler the criterion, the more you need.
  2. Run the judge on this set and measure its agreement with the humans using a chance-corrected statistic, such as Cohen’s kappa for binary or categorical verdicts. Raw agreement is misleading when one class dominates: if 90 percent of answers are grounded, a judge that always answers “grounded” gets 90 percent agreement without evaluating anything. Kappa removes the share of agreement due to chance. For numeric scores, use a rank correlation or a weighted kappa.
  3. Set the acceptance threshold before measuring, according to the stakes. As a reference point, the Landis and Koch (1977) scale calls a kappa between 0.61 and 0.80 “substantial” agreement; a threshold of 0.7 is an example, not a standard. Below your threshold, the judge prompt needs rework or the judge model needs changing.
  4. Recalibrate whenever the judge model changes. Provider updates shift judge behavior.

A per-trace score is for investigating one case. To monitor quality over time, you need to aggregate. I see three ways to do it.

The mean over a window is simple but hides the shape of the distribution: a few very bad answers vanish in the crowd. It is the default for stable evaluators.

A low percentile, often p10 or p25, looks at the tail of the worst answers rather than the average. The p10 is the score below which the 10 percent lowest-scored answers fall. It is better at catching regressions.

The rate below a threshold counts the share of traces scoring under a fixed value. This is the basis of a quality SLO, a numeric target stated as a proportion: for example “at least 95 percent of evaluated answers score above 0.8 on faithfulness over a rolling 7-day window” (illustrative values).

This proportion is estimated on a sample of evaluated answers. You therefore compare not its point value with the target but the lower bound of its confidence interval (Wilson bound), that is the lowest value consistent with the sample. Defining the quality SLO, sizing the sample, the error budget and the judge’s meta-observability are covered in the GenAI method, Part IV, which is the reference on this point.

A single judge call is noisy: two identical calls can give two different scores. Two strategies reduce that variance.

Self-consistency means running the same judge prompt three or five times and taking the majority. Judge cost is multiplied by three (or five) and variance drops markedly.

Judge ensembles use two different judge models and require them to agree, a bit like double marking. Confidence goes up on positive cases, but there are more disagreements to investigate.

I keep self-consistency for high-stakes evaluators, such as refusal or safety. For cheap continuous scorers, such as relevance, a single call is enough.

Running every evaluator on every trace is rarely affordable at scale. So you evaluate a sample, organized in cost tiers.

One principle holds across the site: what sets the precision of a quality indicator is the number of judgments per segment (feature, tenant, language, request type) and per period, not the sampled percentage. It works like an opinion poll: precision depends on how many people you ask, not on what share of the population they represent. The same 1 percent rate yields 100,000 judgments a day on 10 million requests, but only 20 on 2,000 requests, too few to track a proportion.

So first set the number of judgments needed per segment and per window (the GenAI method, Part IV gives the sizing formula). Derive the rate from it, then oversample rare or risky segments. The tiers below illustrate a cost-based organization; their percentages are examples.

  • Tier 1 (cheap, always on): rule-based format checks, classifier-based PII and toxicity detection. For example on 100 percent of traces.
  • Tier 2 (medium cost): small LLM-as-judge calls for faithfulness and relevance. For example on 10 percent of traces.
  • Tier 3 (high cost): large judge models or self-consistency for domain rubrics. For example on 1 percent of traces, plus 100 percent of traces flagged by Tier 1 or Tier 2.

Adjust these rates to your volume and budget. Check that they yield enough judgments per segment and per period for the precision you need.

Where to start? Pick one criterion that matters to the business, write its rubric with them, then have humans score fifty traces before you even plug in the judge.

Next: 6. The tools landscape.

Revised on 4 October 2026: quality SLO stated as a proportion with a link to the GenAI method (Wilson bound), judge sampling principle (number of judgments per segment and per period rather than percentage), tiers presented as examples.

Revised on 25 October 2026: opening on the illustrative scenario, terms explained (evaluator, regular expression, JSON Schema, kappa, percentile, Wilson bound), prompts moved into collapsible sections, types and aggregations written as prose.