Skip to content

TechnicalPractitioner5 min read

8. Versioning, replay and experimentation

Versioning everything that influences the answer, replaying old traces, shadow mode, A/B testing, canaries and regression gating in CI.

For: engineers and SREs · architectsPrerequisites: Have read chapters 3 to 5 of the guide.

Illustrative scenario. One Tuesday, the faithfulness of the insurer’s assistant drops sharply. The week before, three things changed: the team edited the prompt, the document index was rebuilt and the provider may have updated its model. Which one is to blame? If nobody recorded which version of each component served each answer, the question has no answer.

Once the platform reaches Level 3, three capabilities come within reach: versioning, replay and experimentation. Each one multiplies the value of the observability data already collected.

Everything that influences the answer should be versioned and its version recorded as a span attribute. Think of a dated recipe: if the cake fails, you want to know which version of the recipe, which flour and which oven were used. Without this, you cannot answer the scenario’s question: did the change of prompt version cause the regression?

  • Prompt templates (system and user), by content hash and by semantic version (a readable number such as v17).
  • Model identity, including the actually served version returned by the provider, not just the requested name.
  • The retriever index, by snapshot or build identifier.
  • The reranker model and its weights.
  • Tool schemas exposed to the agent.
  • Evaluator prompts, when an LLM judge is used.
  • The entire feature configuration, as a single config hash.
acme.prompt.version = "rag-answer-v17"
acme.prompt.hash = "sha256:..."
acme.retriever.index = "kb-prod-2026-05-12"
acme.retriever.version = "hybrid-v3"
acme.reranker.model = "bge-rerank-v2"
acme.config.hash = "sha256:..."
gen_ai.response.model = "provider-model-2026-03-15"

When a regression appears, the first query groups traces by acme.prompt.version and computes mean faithfulness for each group. If versions are recorded, the answer is one query away. If not, it is out of reach.

Replaying means running an old request through a different version of the system to compare answers, like re-watching an old match in slow motion under new refereeing rules. It is the foundation of safe rollouts.

Replay lets you test a new prompt version against last week’s traffic before deploying it. It also lets you compare a new model with the production model on identical inputs. You can re-evaluate historical traces with an updated evaluator to see the trend under the new rubric. Finally, you can reproduce a customer complaint to check that a fix works.

First, you need to have captured the full prompt and tool arguments at the time of the original trace, which assumes content capture was enabled (see chapter 4). Next, you need a way to bypass external calls whose results vary, by freezing MCP responses and retrieval results. Finally, you need a separate execution path that does not mix its traces with production ones.

The first pitfall is forgetting that retrieval results have changed since the original trace. Either replay against a frozen index or accept that retrieval differences are part of the comparison.

The second is replaying with the original temperature. Temperature controls how much randomness goes into the answer. At the original temperature, that sampling noise drowns small score differences. Use temperature zero or self-consistency. Temperature zero reduces variability without removing it: it is not deterministic with every provider and some reasoning models do not accept this parameter. In that case, self-consistency or several runs per trace are the only option.

The third is letting replay traces into production dashboards. Tag them explicitly as replay and filter them out of operational alerts.

8.3 Experimentation: shadow mode and A/B testing

Section titled “8.3 Experimentation: shadow mode and A/B testing”

Replay works on the past. To test a change on live traffic, there are three deployment patterns, from the most cautious to the most exposed.

In shadow mode, the new system runs in parallel with production, like a trainee pilot following the flight on a simulator wired to the real aircraft. Its answer is not returned to the user. Both answers are recorded with a shared trace ID, the evaluators score both and a comparison dashboard shows the differences. It suits high-confidence rollouts, where the new system is expected to match or beat production.

In an A/B test, a fraction of traffic is routed to the new system and the user sees the experimental answer. The trace attribute acme.experiment.variant records which arm the request was assigned to. Metrics and evaluator scores are aggregated by variant.

Before promoting a variant, the difference has to be statistically significant, meaning unlikely to be due to chance. A 1 percent change in faithfulness on 1,000 traces is not.

The canary takes its name from the birds once taken down mines to detect gas: a small exposure warns of danger before it reaches everyone. It is an A/B test that starts on a small fraction of traffic (1 to 5 percent), then widens gradually. Rollback is automatic if online evaluators detect a quality regression. It suits high-traffic features best, where the statistical signal arrives quickly.

The last maturity move: block a deployment based on offline evaluation results, like a roadworthiness test you must pass before getting back on the road. Continuous integration (CI) is the automated pipeline that builds and tests every change.

  1. Maintain a regression set that covers the failure modes seen in production.
  2. Run the candidate prompt or model against this set on every CI build.
  3. Score each output with the relevant evaluators.
  4. Fail the build if any evaluator score drops below the previous best.
  5. Treat the regression set as a living artifact: every triaged production failure becomes a new entry.

If you do only one thing after this chapter, add the prompt version and the actually served model version to every span. The rest can come later.

Next: 9. Anti-patterns and common pitfalls.

Revised on 25 October 2026: opening on an illustrative scenario, notions explained with images (dated recipe, flight simulator, canary, roadworthiness test), temperature and significance explained, lists written as prose.