Skip to content

TechnicalPractitioner5 min read

Module 4: continuous evaluation in production

Grade an LLM's answers before each release and continuously afterwards: offline and online evaluation, the LLM as a judge and its biases, four RAG-specific metrics, hallucination detection and user feedback.

For: engineers and SREsPrerequisites: Have read module 3 of the course on the three drifts.

One Friday, the team behind a support assistant deploys a new version of the prompt (an illustrative scenario). Before release, they tested it on three hundred questions whose right answer they know: no regression. The following Monday, negative feedback rises on a single feature, order tracking. The test questions hardly included any. The team ran the check before setting off; what it lacked was the warning light on the road.

This module covers both checks, the tool that automates them (one model grading another) and how to read the scores of a RAG.

Offline evaluation is like a car’s roadworthiness test: you run it at each new version, before deployment, on a reference set, meaning a list of questions whose right answer you know (as an order of magnitude, a few hundred to a thousand question and answer pairs). It allows a rigorous comparison between versions. It costs more, since it often needs a human in the loop. It also cannot see drifts that appear after release to production.

Online evaluation is more like the warning lights on the dashboard: it runs continuously on real traffic, sampled (for example 1 to 5%, see below). It detects quickly and alerts in near real time. Its cost is kept under control by sampling, since each judgment consumes tokens. Its limit: there is no oracle, only another model doing the judging.

The two complement each other and answer different questions. In practice, you evaluate offline before each release to production and online continuously.

How many answers should be judged online? The principle used across the site: what makes a score precise is the number of judgments per segment and per period (per feature or per customer, per hour or per day), not the percentage of traffic. The 1 to 5% quoted here are examples. On heavy traffic, 1% already yields many judgments; on a rarely used segment, even 100% may yield only a few per day. That is exactly the order-tracking trap in the opening scenario. The design of sampling (evaluator tiers, traces that are always evaluated) has its reference in section 5.7 of the guide; this module keeps what is specific to the labs: judge biases and RAG diagnosis.

The two evaluations form a single loop:

flowchart TB
  V["New version"] --> OFF{"Offline<br/>reference set"}
  OFF -->|"regression"| FX["Fix"]
  FX --> V
  OFF -->|"compliant"| PR["Production"]
  PR --> ON["Online<br/>1 to 5% sample,<br/>different judge"]
  PR --> FB["User feedback<br/>linked to the trace_id"]
  ON --> AL{"Score<br/>dropping"}
  FB --> AL
  AL -->|"yes"| EN["Investigation<br/>on the trace"]
  EN -.->|"new cases"| OFF

An LLM judge is a model you hand the marking to: you show it the question, the context and the answer; it returns a grade. It is fast and yields a score you can use as a metric. It also remains expensive and non-deterministic: pinning the judge model version and setting temperature to zero improve reproducibility without guaranteeing it.

Agreement with human judgment can be high. On comparisons of chatbot answers, Zheng et al. measure over 80% agreement between GPT-4 and human evaluators, as much as between humans (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023). This result does not carry over as is to your judge or your criteria: measure agreement on a sample you have had annotated.

Like any examiner, the judge has known biases, which must be offset:

BiasEffect
verbosityprefers long answers
self-preferenceprefers its own style
positionfavors the first option in an A/B comparison
confidencetolerates a well-written hallucination
hidden costthe judge’s tokens sometimes exceed those of the evaluated model

To offset them, I recommend a structured evaluation prompt, a rigid output format, a judge different from the evaluated model and cross-sampling with human annotations.

In a RAG, a bad answer can come from two places: the search did not find the right documents, or the model did not know how to use them. Four metrics help tell the two apart. RAGAS, cited in the table, is an evaluation library that computes them.

MetricQuestionTools
Faithfulnessare the claims in the answer supported by the retrieved context?RAGAS, Phoenix
Context recalldoes the context contain the needed information?RAGAS, ground truth
Context precisionis the context dense or noisy?RAGAS
Answer relevancedoes the answer address the question asked?RAGAS, LLM judge

Low faithfulness with low context recall points to retrieval: the right documents were not on the table. Low faithfulness with high context recall points to generation: the documents were there, the model strayed from them.

flowchart LR
  F["Low faithfulness"] --> R{"Context recall"}
  R -->|"low"| RET["Look at retrieval"]
  R -->|"high"| GEN["Look at generation"]

Three levels of detection are combined according to risk. The first relies on simple rules: a figure, a date or a proper name present in the answer yet absent from the context. The second uses specialized models, entailment classifiers that check whether the context logically supports what the answer claims. The third listens to weak signals: negative feedback, immediate rewordings of the same question, abandonment.

Thumbs up or down is the cheapest and most direct signal. Link it to the trace through its trace_id: negative feedback then becomes an entry point to the exact context that produced the bad answer.

Make offline evaluation a condition of every release to production and let online evaluation run continuously, with enough judgments on every segment that matters. An LLM judge is only useful if it differs from the evaluated model and its biases are offset. Read faithfulness and context recall together to know whether the problem comes from retrieval or generation. Finally, link every piece of user feedback to its trace: it gives the exact context of the error, which no aggregate score provides.

Revised on 2 October 2026: the unsourced correlation figure between judge and humans is replaced by the study by Zheng et al. (2023), sampling rates and reference set size are presented as orders of magnitude.

Revised on 4 October 2026: judge sampling principle stated (the number of judgments per segment and per period matters more than the percentage), rates presented as examples and pointer to section 5.7 of the guide.