Skip to content

TechnicalPractitioner3 min read (+2 technical)

How is data sorted before it is stored?

Metrics, traces and logs flow through the Collector's sorting steps: turn one off and watch live what reaches storage, with the matching YAML configuration.

For: engineers and SREs · architectsPrerequisites: Know what an OpenTelemetry Collector is for.

Reading mode

An illustrative scenario: during a routine check, the data protection officer finds customer email addresses in plain text in the technical logs. The same month, the storage bill climbs for no obvious reason. Both problems share a cause: nothing sorted the data between the moment applications sent it and the moment it was stored.

That sorting happens in the OpenTelemetry Collector, a piece of software that receives signals from applications and passes them on to storage. Picture a mail sorting center. Parcels ride a conveyor belt past a row of stations: one throws out junk mail, another blacks out confidential details, a third sets aside items for the archive. In the Collector, these stations are called processors. It is the last place where you decide what happens to a piece of data before it becomes a line on the bill, exposed personal data or audit evidence.

In the diagram, each dot is a signal. Its color gives its type (metric, trace or log), its letter what it carries. Click a processor to turn it off and watch what reaches the end of the line.

metrictracelogerror (red ring)Hhealth checkDdebug log@contains an email*masked emailUuser_id labelAaudit / compliance

Click a processor (or Tab then Enter) to disable it.

Collector configuration

Recommended order: memory_limiter first; filtering and redaction early, so nothing sensitive or useless travels further; tail_sampling after redaction; batch last, right before export, or, instead, batching inside the exporter (sending_queue.batch).

What reaches the storage backends

Volume stored
0
of emitted signals
Sensitive data exposed
0
plain emails or user_id
Errors kept
0
of emitted errors
Audit logs in cold storage
0
of emitted audit logs
Export requests
0
calls to backends
Refused / lost
0
memory pressure or OOM
DestinationReceivedNoiseSensitiveErrors

Illustrative simulation: signal mix, 10% sampling and memory threshold are assumptions. Signals refused by memory_limiter are in reality retried by the sender, within the limits of its own queue.

  1. Turn off filter, the station that throws out junk mail. Dots marked H (automatic liveness checks, or health checks) and D (debug logs) reach Tempo and VictoriaLogs, the trace and log databases. The “Noise” column climbs: stored volume grows with nothing gained for diagnosis.
  2. Turn off transform, the station that blacks out sensitive details. Email addresses (@) arrive in clear while metrics keep the user’s identifier, the user_id label (U). The sensitive data counter turns red. That identifier is also a cost problem: on a metric, it multiplies the number of rows to store. The metrics bill simulator puts a figure on it.
  3. Turn off memory_limiter, then click “Traffic spike”. This station works like a doorman who stops letting people in when the room is full. Without it, the Collector’s memory overflows, the system kills it abruptly (an OOM, out of memory) and every signal being processed is lost. Turn it back on and spike again: the Collector refuses part of the sends, which applications resend a little later, but it stays up.
  4. Turn off tail_sampling, then routing. The first one picks which traces to keep once they are complete: every error, only part of the normal requests. Without it, every normal trace is stored even though all errors were already kept. The second one sets audit logs (A) aside for archiving. Without it, they stay in hot storage, which is fast but kept only briefly, so the “Audit logs in cold storage” rate drops to zero.

In a sorting center, you check there is room before unpacking parcels. You throw out junk mail before weighing anything and you seal the boxes just before the truck leaves, not before sorting. The Collector follows the same logic. The doorman (memory_limiter) goes first, to refuse data before spending memory on it. Filtering and masking come early: what is dropped early costs nothing further on, while no sensitive data should travel any further. Choosing which traces to keep comes next, on traces that are already clean. Grouping data into batches for sending (batch) comes last: batching data you are about to throw away is wasted work.

For technical teams: processor order, routing and versions

Processors run in the order they are listed in service.pipelines, not in the order of the processors section.

  • memory_limiter first: it must be able to refuse data before the Collector spends memory processing it.
  • Filter and redact early: what is dropped early costs nothing downstream. No sensitive data should travel through sampling, batching or an exporter.
  • tail_sampling after redaction: the decision is made on the complete trace (every error, a fraction of nominal traffic). It assumes all spans of a trace reach the same Collector, hence a loadbalancing exporter upstream as soon as you run several instances.
  • batch last, right before export. The alternative is to batch inside the exporter itself (sending_queue with batch: {}, off by default): this is the direction the project is taking to replace the processor (deprecation proposal), but as of 2 October 2026 the batch processor is still beta and not formally deprecated.

Routing to cold storage goes through the routing connector: the former routing processor was removed in favor of the connector. The exact syntax of its routing table has changed across Collector contrib versions: the configuration shown follows v0.162 (prefixed OTTL condition, log.attributes[...], no context field); check the documentation for your version before reusing it. The awss3 exporter targets S3-compatible storage outside AWS here, hence endpoint and s3_force_path_style.

The sorting center needs watching too: how many parcels come in, how many are refused, how many leave, whether the queue overflows, whether the warehouse runs out of room. The Collector publishes its own metrics to answer these questions.

Two checks do not rely on those metrics. For sensitive data, I recommend a scheduled search through stored logs and traces that looks for an email pattern and raises an alert if it finds anything. For audit, compare the number of audit logs emitted with the number of objects written to cold storage.

For technical teams: the Collector’s internal metrics

By default the Collector exposes its metrics in Prometheus format on port 8888, configurable under service.telemetry. Prefix otelcol_:

QuestionCollector internal metric (prefix otelcol_)
How many signals come in?receiver_accepted_spans, receiver_accepted_log_records, receiver_accepted_metric_points
How many are refused at the door?receiver_refused_*: refusals by memory_limiter or a saturated exporter
How many go out, how many fail?exporter_sent_*, exporter_send_failed_*
Is the export queue saturating?exporter_queue_size relative to exporter_queue_capacity
Is the Collector short of memory?process_memory_rss, process_runtime_heap_alloc_bytes and container restarts on the Kubernetes side

Exact names vary slightly across versions (_total suffixes, newer processor metric names): start from your Collector’s /metrics page. The email pattern search runs as a scheduled query in VictoriaLogs or Tempo.

Going further: the glossary, the technical view and on the cost side the business page.

Revised on 2 October 2026: configuration aligned with Collector contrib v0.162 (trace_conditions and log_conditions filter syntax, prefixed transform statements, otlp_grpc and otlp_http exporters, S3 storage endpoint), plus exporter-side batching mentioned.