Skip to content

TechnicalBeginner7 min read (+1 technical)

OpenTelemetry end to end

What OpenTelemetry is and the journey of one trace step, from the application to storage, told like the journey of a letter. The explanation for everyone, then a Python instrumentation and a minimal Collector configuration for technical teams.

For: engineers and SREs · architects · team managersPrerequisites: Having read Observability in 10 minutes, or knowing what a metric, a log and a trace are. The collapsed section assumes you can read a YAML file.

Reading mode

Let’s go back to the slow payment from Observability in 10 minutes. In the trace, one line jumps out: the bank call takes almost four seconds. But that line did not write itself. Something produced it inside the payment service, packed it, sent it, sorted it and filed it somewhere it could be found again.

This page follows that journey from start to finish, for a single trace step: the bank call. The standard that organizes the whole route is called OpenTelemetry.

Think of a letter. Someone writes it. The post office puts it in an envelope, bundles it with others and hands it to a carrier. The envelope has a standard format that every sorting office can read. The sorting office checks it, sets aside what should not go out and sends each letter to the right mailbox. Finally the letter lands in a mailbox where it is filed and read.

A trace step makes exactly that journey. In the trade, this step is called a span: a piece of a trace, with a start time, an end time and a few details attached. Here is how the two map, step by step; the rest of the page goes through each one:

  • writing the letter is instrumentation, the code that creates the span;
  • the post office is the SDK, the library that prepares the shipment;
  • the envelope format is OTLP, the transport protocol;
  • the sorting office is the Collector;
  • the mailbox is storage, with its interface for reading.

OpenTelemetry, often shortened to OTel, is an open source project of the CNCF, the foundation that also hosts Kubernetes and Prometheus. It was born from the merger of two earlier projects, OpenTracing and OpenCensus (official overview).

What surprises people most: OpenTelemetry is not a storage or visualization tool. It produces and transports traces, metrics and logs, the way the postal service carries mail without making the mailboxes. Storage is your choice: Prometheus, Thanos, Mimir or VictoriaMetrics for metrics, Tempo or Jaeger for traces, Loki or Elasticsearch for logs, or a commercial solution.

The project brings together six building blocks. The specification is the text that says how everything must work, language by language. The API is the set of functions code calls to create a span or a metric. The SDK is the library that implements that API: it samples, batches and exports. OTLP is the transport protocol shared by all signals. The Collector is a standalone program that receives, transforms and forwards traces, metrics and logs. Finally, the semantic conventions are the list of standard attribute names and what they mean.

Here is the complete path, as it appears in the configuration shown further down.

flowchart LR
  A["Application<br/>payment service"] -->|"API"| S["SDK<br/>sampling, batching"]
  S -->|"OTLP"| R
  subgraph COL["Collector"]
    R["Receiver<br/>otlp"] --> P["Processors<br/>memory_limiter, batch"] --> E["Exporters"]
  end
  E -->|"OTLP"| B[("Trace<br/>storage")]
  E --> D["debug<br/>console"]

Instrumenting means adding to a program the code that produces traces, metrics and logs. There are two ways to do it.

Auto-instrumentation, known as zero-code, requires no code change. An agent or launcher wraps itself around well-known libraries (the web server, the HTTP client, the database driver) and creates spans on its own. In my view, it is the best way to start.

Manual instrumentation means calling the API in your own code. It adds what auto-instrumentation cannot guess: a business step such as payment authorization, a detail such as the card type.

In practice, you combine both. Auto-instrumentation gives the skeleton of the trace. Manual instrumentation gives it meaning.

The SDK receives finished spans. First it decides whether to keep them: this is sampling, which lets you keep only part of the traces when volume is high. Then it groups them into batches to limit the number of sends. It adds the resource attributes, which say which service produced them (more on that below). Then it hands them to an exporter, the component in charge of sending them.

OTLP (OpenTelemetry Protocol) is the transport format, the standard envelope. It runs over gRPC, a protocol programs use to talk to each other (port 4317 by default), or over HTTP (port 4318) (Collector configuration). The same envelope carries metrics, logs and traces.

The Collector is the sorting office. It is a program that sits between applications and storage. It has three kinds of components.

Receivers take data in: the otlp receiver listens on ports 4317 and 4318. Processors transform data, in the order they are listed. For example, memory_limiter refuses data when memory gets close to its limit so the Collector does not crash, while batch groups data before sending. Finally, exporters send data to one or more storage backends.

A pipeline connects receivers, processors and exporters for one signal type. You write one pipeline for traces, another for metrics.

The Collector is not mandatory: an SDK can export straight to a backend. I still recommend using it from day one. It is the single place where you protect, filter and route data without touching the applications.

The backend files spans by trace id, the number shared by every step of the same request. The interface then rebuilds the waterfall seen on the previous page. The trace explorer shows what that result looks like.

Two notions come up everywhere. They are simple, but they make all the difference when you search.

The resource is the sender’s address on the back of the envelope. It describes who is emitting and it is attached once and for all to everything the service produces: its name (service.name="payment"), its version (service.version="1.4.2"), its environment (deployment.environment.name="dev"). Without service.name, there is no way to know which service emitted a span.

Semantic conventions are the common language. They set the name and meaning of common attributes, the details attached to a span (semantic conventions). An HTTP request carries http.request.method and http.response.status_code. A route carries http.route. Everyone uses the same names, whatever the programming language.

The benefit is concrete. If a Java service and a Python service follow the conventions, a single query finds all HTTP 500 errors from both. If each invents its own names (status, httpCode, return_code), you need one query per team.

The conventions are still evolving and some names have changed in recent years. I recommend pinning the version you follow and reviewing it at every SDK upgrade.

The following example instruments a Python service and runs a Collector that prints received data to its console. It is collapsed below for teams who want to try it.

For technical teams: instrumenting a Python service and configuring the Collector

Instrument without touching code. The commands come from the OpenTelemetry Python zero-code documentation. app.py is your application.

Terminal window
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
export OTEL_SERVICE_NAME=payment
export OTEL_RESOURCE_ATTRIBUTES=service.version=1.4.2,deployment.environment.name=dev
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
opentelemetry-instrument python app.py

opentelemetry-bootstrap detects installed libraries and adds the matching instrumentations. opentelemetry-instrument starts the application with those instrumentations active. Here the resource comes from the OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES variables.

Add a business span by hand. To see the “authorize” step in the trace, with the card type, a few lines are enough:

from opentelemetry import trace
tracer = trace.get_tracer("payment")
def authorize(order):
with tracer.start_as_current_span("authorize_payment") as span:
span.set_attribute("app.payment.card_type", order.card_type)
return call_bank(order)

This span becomes the child of the HTTP span created automatically. The app. prefix flags an application-specific attribute, outside the standard conventions.

Configure the Collector.

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
batch: {}
exporters:
debug:
verbosity: detailed
# To a trace backend that accepts OTLP (Tempo, Jaeger, etc.).
otlp_grpc/traces:
endpoint: traces-backend:4317
tls:
insecure: true # lab only, never in production
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, otlp_grpc/traces]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug]

To read this file:

  • memory_limiter comes first: it must refuse data before any other processing. batch comes last, after processors that filter or transform.
  • The debug exporter prints each span to the Collector console. It is the simplest way to check that data arrives.
  • Recent Collector versions name the OTLP gRPC exporter otlp_grpc and the OTLP HTTP exporter otlp_http. The old names otlp and otlphttp are still accepted as deprecated aliases: some configurations on this site still use them. Check the name your version expects.

The 0.0.0.0 address listens on all interfaces. The Collector documentation uses it for convenience and recommends localhost when all clients are local.

If you are starting from scratch, I recommend this order. Turn on auto-instrumentation for one important service, payment for instance, without touching the code. Route its data through a Collector from day one, with the debug exporter to check that everything arrives. Give every service a clear name in its resource. Then add by hand the two or three business steps that matter. Finally, write down which version of the conventions you follow and name the person who owns the Collector configuration.