TechnicalPractitioner3 min read
Which traces should you keep without storing everything?
Compare two ways of selecting traces at equal volume, at the start of the request or once the trace is complete: traces kept, storage, cost, Collector memory and share of errors retained.
For: engineers and SREs · architectsPrerequisites: Basic notions of distributed traces and the OpenTelemetry Collector.
An illustrative scenario: an angry customer calls support because their payment failed yesterday at 2:12 pm. The on-call engineer looks for the request’s trace, the detailed account of its journey between services. It does not exist. To keep costs down, the team keeps only one trace in ten, picked at random. This one fell through the cracks.
Keeping 100% of traces is expensive. Keeping 10% at random means throwing away 90% of incidents. Think of a security camera. Option one: keep one frame in ten, picked at random when it is shot. Storage shrinks, but the frame showing the break-in has a nine in ten chance of being gone. Option two: record everything into a buffer for a few seconds, then keep only the sequences where something happened, plus a few quiet ones for comparison.
For traces, the first option is called head sampling: the decision is made when the request starts, before knowing whether it will fail. The second is called tail sampling: you wait until the trace is complete, then decide. You keep every error, every slow trace and a small percentage of the rest. That intelligence has a price: the memory of the Collector, the software doing the sorting, which must hold every trace until the decision is made.
The simulator helps settle one question: for the same storage budget, which method keeps the traces you will need during an incident? Prices change fast and vary by contract: this guide gives none; use your provider’s dated price list. Without a price, the storage tile shows the volume to be billed in GB-months (one gigabyte kept for one month). Enter your price per GB-month to get an amount.
Result
Breakdown by category
| Category | Incoming | Kept (tail) | Kept (head) |
|---|
Assumptions: error and slowness are treated as independent (a trace can be both; it is then counted as an error). KB and GB in powers of 1,000. Real Collector memory is often several times the wire size (in-memory structures, traffic peaks): plan headroom. The storage price is empty by default: enter the one from your provider’s dated price list to get an amount.
Things to try
Section titled “Things to try”- Look at the “Errors kept” tile. With the defaults, tail sampling keeps 100% of errors. Head sampling, for the same stored volume, keeps only about 13%. The chart says the same thing: two bars of equal length, two very different contents. Both methods keep about 269 traces per second (13% of traffic), that is 466 GB per day and 6,985 GB-months to be billed over 15 days of retention.
- Untick “Keep 100% of error traces”. Errors are then picked at random like everything else. You are back to roughly head sampling behavior, with the memory cost on top.
- Raise the share of slow traces to 30%. The kept volume explodes, from 269 to 725 traces per second (36% of traffic) and from 6,985 to 18,789 GB-months. A badly tuned latency threshold cancels the saving: it should target the few truly slow requests, not a third of the traffic.
- Raise the wait before decision from 30 to 120 seconds. Collector memory quadruples, from 1.2 to 4.8 GB. Long traces, such as processing that carries on in the background, force a longer wait, hence bigger Collectors or traces spread across several instances.
How it works
Section titled “How it works”With head sampling every trace has the same chance of being kept: the sample looks like the traffic and rare errors stay rare. With tail sampling the decision is based on the trace content, so what matters for diagnosis can be kept first. In my view, it is the right method as soon as errors are rare and valuable, provided you accept its two practical constraints: all parts of a given trace must reach the same Collector, which in turn needs enough memory.
The simulator assumes error and slowness are independent. In reality they often go together (a service that stops responding yields a trace that is both slow and failed), which slightly reduces the kept volume.
For technical teams: load balancing and sizing
All spans of a trace must reach the same Collector: traces are spread across instances by trace ID. Memory scales roughly as traces/s × wait × size, with headroom for peaks.
Further reading: the business dimension on observability data cost, the Collector cost simulator, the metrics bill simulator and the glossary.
Revised on 2 October 2026: prices removed; storage shows in GB-months and in euros only with your price.