Technical7 min read (+1 technical)

OpenTelemetry labs

The labs published on the forge around OpenTelemetry: a control plane for a fleet of Collectors, a RAG application instrumented with the GenAI conventions, an audit exporter for the Collector.

OpenTelemetry is the open standard that defines how an application produces and sends its observability data. At the heart of many setups sits the Collector: a program that works like a postal sorting center. It receives the data, sorts it, enriches it and forwards it to the right storage.

On the forge, the labs devoted to OpenTelemetry have their own group. It holds three of them today and each one answers a different question: how to drive a fleet of Collectors, how to instrument an AI application, how to prove what leaves the Collector.

The otel group on the forgeThe three repositories, public, to clone without an account.

The three labs at a glance

Lab	The question	What you handle	Prerequisites	State as of 10 October 2026
`REVERSOTEL`	Can you change observability backends without reconfiguring every agent?	OpAMP, Collector, fleet Git repository	Docker with Compose v2 (12 GB of memory allocated), Go 1.26, Git	published, with the reference report of the bench
`GENAIOTEL`	What do you see of a RAG application with the OpenTelemetry standard alone?	GenAI semantic conventions, Collector, OTTL	Docker with the Compose plugin	published; the latest version has not yet been replayed end to end under Docker
`OTELAUDITEXP`	How do you prove which data left the Collector, for whom and unaltered?	custom exporter, OpenTelemetry Collector Builder	Go 1.25 or newer	published

The figures and prerequisites on this page come from the README.md of each repository, read again on 10 October 2026. When a lab announces a measured result, I say which report it comes from.

REVERSOTEL: switching backends with a commit

Imagine switching electricity suppliers and having to rewire every socket in the house to do it. That is often where observability stands. OpenTelemetry has opened up collection: agents, protocols and formats are standard. Managing the fleet of agents is not. Their configuration, dashboards and alerts are often tied to a vendor, so changing backends turns into a project of its own.

REVERSOTEL is a control plane, a kind of control tower for Collectors. It drives them remotely through the OpAMP protocol from a Git repository. It stores no traces, no metrics and no logs itself. Destinations, roles, the data contract and the content (dashboards, alerts) are versioned text files. Changing backends means editing one line and recording that change in Git, which is called a commit.

The lab teaches you to drive Collectors remotely, with no intervention on the machines. You write a data contract that every Collector enforces. You also practice dual write: data goes to both the old and the new destination before you remove the old one, so that nothing is lost.

The repository ships a test bench you can replay. It switches a fleet of 21 Collectors from Elastic to VictoriaMetrics and VictoriaLogs, then back. The reference report published in the repository, dated 30 September 2026, shows 27 of 27 measures passing, including a switch of the 21 agents in 5.1 seconds and no loss on the witness stream. These are the results of that bench: the repository explains how to replay it to get your own.

For technical teams: running REVERSOTEL

git clone https://forge.erythix.tech/labstraining/otel/reversotel.git
cd reversotel
demo/run.sh

The scenario takes about 20 to 30 minutes, plus image builds on the first run. The fleet configuration files are YAML.

Following the fleet from Grafana

A control tower needs screens. A companion repository, Reversotel-app, adds three pages to Grafana to follow the fleet driven by REVERSOTEL. The first lists every Collector with its version, its health and the state of its configuration. The second details one Collector, down to its effective configuration. The third shows drift: the Collectors whose applied configuration differs from what Git asks for, with a line-by-line diff.

The app is read-only by design. Git stays the single source of truth: the app never changes a configuration, it only links to the relevant commit or file in the forge. It also works in an environment cut off from the Internet, without loading anything from outside.

For technical teams: requirements and limits

The app needs Grafana 11.6 or later; the repository reports tests on versions 11.6.16 and 13.2.4. A Go backend queries the REVERSOTEL REST API with a token, over TLS. Some fields (service name, contract version per agent, component health tree) are proposed additions to the REVERSOTEL API that do not exist yet: without them, the app shows “not reported”.

GENAIOTEL: tracing a RAG application with no proprietary SDK

A RAG application is an assistant that looks up documents in a knowledge base before asking a language model to write the answer, like a student checking their notes before answering an exam question. AI observability tools often come with their own code library to embed in the application (an SDK). The lab’s question is simple: what can you get with the standard alone?

The lab’s application emits standard signals only. These are spans (the steps of a trace) and metrics that follow the OpenTelemetry GenAI semantic conventions, meaning the shared names the standard gives to generative AI data. All enrichment then happens in the platform, with stock components: the Collector computes the cost of each step and a connector produces latency and error rate per model. The language model and the embeddings (turning text into vectors of numbers) are simulated and always give the same result. There is no API key and no model to download. What is real are the traces and metrics.

The lab teaches you to instrument a RAG pipeline with the standard attributes, including retries and the fallback to another model. It offers scenarios that answer production questions: a bad day, what the cache saves, an index that degrades with no error and no rise in latency. Finally, it separates two things often confused: observability while the application runs and evaluation of answer quality, done separately on a reference dataset.

The GenAI conventions are still in Development status, in other words liable to change. The lab follows their current text and isolates each attribute name in a constant, so that a rename stays a one-line change.

For technical teams: running GENAIOTEL

Cost per span is computed in the Collector with OTTL; latency and error rate per model come from the spanmetrics connector. Attributes follow the gen_ai.* namespace.

git clone https://forge.erythix.tech/labstraining/otel/genaiotel.git
cd genaiotel
make build
make demo

make demo runs the application alone. make up, make demo-load then make verify add the observability stack and check the chain link by link.

OTELAUDITEXP: proving what leaves the Collector

Think of registered mail: the recipient knows who sent it and when, with a guarantee the envelope was not opened on the way. In a regulated context, with several customers on the same platform, a team must be able to provide that kind of proof for its data: which logs, metrics and traces left the Collector, for which customer and unaltered. OTLP exporters, the components that send data to its destination using OpenTelemetry’s standard protocol, pass on raw data with no attestation at all.

The lab is a custom exporter for the Collector. It puts each batch of data in an envelope before sending it to a collection point. The envelope carries the customer identifier, the schema version, a sequence number that goes up with each send, a fingerprint of the content (a calculation that changes as soon as a single character changes), the emission time and the item count. The recipient recomputes the fingerprint to detect tampering and follows the numbers to spot a missing or replayed send.

You learn to write a Collector component on its official interfaces, then to build it. You also learn when a custom component is justified: in my view, only when the need cannot be expressed with existing components. Finally, the lab shows the limits of a proof, which the repository documents: sequence counters restart from zero on restart and a batch that mixes several customers is attributed to the first one found.

For technical teams: building OTELAUDITEXP

Each batch of logs, metrics or traces is serialized to OTLP JSON, wrapped, then posted to an HTTP sink. The fingerprint is SHA-256 and the sequence is kept per signal. The component is built with the OpenTelemetry Collector Builder.

git clone https://forge.erythix.tech/labstraining/otel/otelauditexp.git
cd otelauditexp
go install go.opentelemetry.io/collector/cmd/builder@v0.155.0
builder --config examples/builder-config.yaml

For the full reasoning on proof through logs, see Audit log integrity.

Which one to start with?

You are new to OpenTelemetry: I recommend GENAIOTEL. It only needs Docker and the application runs alone before you add the observability stack.
You already operate Collectors: REVERSOTEL, to see what OpAMP-driven management changes.
You work in a regulated context or you develop for the Collector: OTELAUDITEXP.

The matching articles are in the OpenTelemetry section. The other labs of the site are on the Labs page.