TechnicalPractitioner5 min read
Module 4: continuous evaluation in production
Grade an LLM's answers before each release and continuously afterwards: offline and online evaluation, the LLM as a judge and its biases, four RAG-specific metrics, hallucination detection and user feedback.
For: engineers and SREsPrerequisites: Have read module 3 of the course on the three drifts.
One Friday, the team behind a support assistant deploys a new version of the prompt (an illustrative scenario). Before release, they tested it on three hundred questions whose right answer they know: no regression. The following Monday, negative feedback rises on a single feature, order tracking. The test questions hardly included any. The team ran the check before setting off; what it lacked was the warning light on the road.
This module covers both checks, the tool that automates them (one model grading another) and how to read the scores of a RAG.
Online or offline
Section titled “Online or offline”Offline evaluation is like a car’s roadworthiness test: you run it at each new version, before deployment, on a reference set, meaning a list of questions whose right answer you know (as an order of magnitude, a few hundred to a thousand question and answer pairs). It allows a rigorous comparison between versions. It costs more, since it often needs a human in the loop. It also cannot see drifts that appear after release to production.
Online evaluation is more like the warning lights on the dashboard: it runs continuously on real traffic, sampled (for example 1 to 5%, see below). It detects quickly and alerts in near real time. Its cost is kept under control by sampling, since each judgment consumes tokens. Its limit: there is no oracle, only another model doing the judging.
The two complement each other and answer different questions. In practice, you evaluate offline before each release to production and online continuously.
How many answers should be judged online? The principle used across the site: what makes a score precise is the number of judgments per segment and per period (per feature or per customer, per hour or per day), not the percentage of traffic. The 1 to 5% quoted here are examples. On heavy traffic, 1% already yields many judgments; on a rarely used segment, even 100% may yield only a few per day. That is exactly the order-tracking trap in the opening scenario. The design of sampling (evaluator tiers, traces that are always evaluated) has its reference in section 5.7 of the guide; this module keeps what is specific to the labs: judge biases and RAG diagnosis.
The two evaluations form a single loop:
flowchart TB
V["New version"] --> OFF{"Offline<br/>reference set"}
OFF -->|"regression"| FX["Fix"]
FX --> V
OFF -->|"compliant"| PR["Production"]
PR --> ON["Online<br/>1 to 5% sample,<br/>different judge"]
PR --> FB["User feedback<br/>linked to the trace_id"]
ON --> AL{"Score<br/>dropping"}
FB --> AL
AL -->|"yes"| EN["Investigation<br/>on the trace"]
EN -.->|"new cases"| OFF
The LLM as a judge
Section titled “The LLM as a judge”An LLM judge is a model you hand the marking to: you show it the question, the context and the answer; it returns a grade. It is fast and yields a score you can use as a metric. It also remains expensive and non-deterministic: pinning the judge model version and setting temperature to zero improve reproducibility without guaranteeing it.
Agreement with human judgment can be high. On comparisons of chatbot answers, Zheng et al. measure over 80% agreement between GPT-4 and human evaluators, as much as between humans (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023). This result does not carry over as is to your judge or your criteria: measure agreement on a sample you have had annotated.
Like any examiner, the judge has known biases, which must be offset:
| Bias | Effect |
|---|---|
| verbosity | prefers long answers |
| self-preference | prefers its own style |
| position | favors the first option in an A/B comparison |
| confidence | tolerates a well-written hallucination |
| hidden cost | the judge’s tokens sometimes exceed those of the evaluated model |
To offset them, I recommend a structured evaluation prompt, a rigid output format, a judge different from the evaluated model and cross-sampling with human annotations.
Four RAG-specific metrics
Section titled “Four RAG-specific metrics”In a RAG, a bad answer can come from two places: the search did not find the right documents, or the model did not know how to use them. Four metrics help tell the two apart. RAGAS, cited in the table, is an evaluation library that computes them.
| Metric | Question | Tools |
|---|---|---|
| Faithfulness | are the claims in the answer supported by the retrieved context? | RAGAS, Phoenix |
| Context recall | does the context contain the needed information? | RAGAS, ground truth |
| Context precision | is the context dense or noisy? | RAGAS |
| Answer relevance | does the answer address the question asked? | RAGAS, LLM judge |
Low faithfulness with low context recall points to retrieval: the right documents were not on the table. Low faithfulness with high context recall points to generation: the documents were there, the model strayed from them.
flowchart LR
F["Low faithfulness"] --> R{"Context recall"}
R -->|"low"| RET["Look at retrieval"]
R -->|"high"| GEN["Look at generation"]
Detecting hallucinations
Section titled “Detecting hallucinations”Three levels of detection are combined according to risk. The first relies on simple rules: a figure, a date or a proper name present in the answer yet absent from the context. The second uses specialized models, entailment classifiers that check whether the context logically supports what the answer claims. The third listens to weak signals: negative feedback, immediate rewordings of the same question, abandonment.
User feedback
Section titled “User feedback”Thumbs up or down is the cheapest and most direct signal. Link it to the trace through its trace_id: negative feedback then becomes an entry point to the exact context that produced the bad answer.
What to put in place
Section titled “What to put in place”Make offline evaluation a condition of every release to production and let online evaluation run continuously, with enough judgments on every segment that matters. An LLM judge is only useful if it differs from the evaluated model and its biases are offset. Read faithfulness and context recall together to know whether the problem comes from retrieval or generation. Finally, link every piece of user feedback to its trace: it gives the exact context of the error, which no aggregate score provides.
Revised on 2 October 2026: the unsourced correlation figure between judge and humans is replaced by the study by Zheng et al. (2023), sampling rates and reference set size are presented as orders of magnitude.
Revised on 4 October 2026: judge sampling principle stated (the number of judgments per segment and per period matters more than the percentage), rates presented as examples and pointer to section 5.7 of the guide.