Skip to content
MTTL, Mean Time To Learn

TechnicalBeginner5 min read (+2 technical)Video

Cardinality explained

Why can an observability bill double while traffic stays flat? Because of one criterion too many on a metric. The explanation for everyone, with a video, then the details for technical teams and a lab that replays the scenario.

For: engineers and SREs · architects · team managersPrerequisites: None. The "For technical teams" section assumes some knowledge of Prometheus.

Reading mode

One Monday morning, a developer needs to understand why one particular customer is seeing errors. To move fast, they add the customer ID to a metric. The bug is found within the hour and the ticket closed. Three weeks later, the observability bill has doubled. Nobody connects the dots: traffic has not moved and the dashboards are green.

This scenario is made up, but it is a classic. It goes by a slightly off-putting name, cardinality. The idea itself is simple.

Read the video text

This quarter, the observability bill doubled. Traffic did not move. Nobody knows why.

A metric works like a tracking spreadsheet. Every different combination opens a new row. The number of rows is what we call cardinality.

In the lab, a simulated service counts its visits with four criteria. Rows multiply: 3 × 8 × 5 × 3 = 360 for this counter alone. With its other measurements, the service takes up about 2,000 rows. Nothing moves.

A developer adds the customer ID to debug a ticket. With 200 customers, this counter jumps to 72,000 rows. The service now takes up about 74,000 rows. Traffic has not changed.

Every row is stored, indexed and scanned on every search. More rows mean more storage, more memory, slower searches. In the lab: 37 times more storage. Billed per series, the bill follows.

Nobody comes forward to confess. You count the values of each column. The region looks suspicious, yet it will only ever have a few values. The customer ID has no limit. That is what drives the bill up.

First, the platform removes the column on the way in, before storage. Rows fall back to about 2,000 within minutes. Then you fix the code. Then you set a cap for next time.

What to remember so the bill no longer takes you by surprise.

What to remember:

  1. A column should have few possible values: region, page, outcome.
  2. Identifiers (customer, request) belong in traces and logs.
  3. Watch the number of rows and how fast it climbs.

Lab assumptions: one measurement every 15 seconds, 0.7 byte per measurement. Cost depends on your contract.

Cardinality explained, with the lab to replay it: meantimetolearn.com.

Music: “Radar Focus”, Blue Saga (Epidemic Sound).

A metric is a tracking spreadsheet

Section titled “A metric is a tracking spreadsheet”

Picture a spreadsheet where you count visits to a website. One column for the server that answered, one for the page requested, one for the outcome, as a response code (200 when all is well, 404 for a page not found, 500 for a server error…). Every different combination opens a new row: “server 1, cart page, 200” is one row; “server 1, cart page, 500” is another.

Monitoring tools work exactly like that. The columns are called labels, each row is called a series and the number of rows is the cardinality. Every row is stored, indexed and, with many vendors, paid for.

Rows multiply

Section titled “Rows multiply”

The number of rows does not add up; it multiplies. Three servers, eight pages and five possible outcomes already give 3 × 8 × 5 = 120 rows. Add three regions and you reach 360.

That is what the lab behind this page shows: a small simulated service, with a visit counter and these four criteria. Counting its other measurements, the service takes up about 2,000 rows. As long as nobody adds a column, that number does not move.

The column too many

Section titled “The column too many”

Back to Monday morning. Adding the customer ID means adding a column whose values never end: a new customer brings new rows, for every server, every page and every outcome.

In the lab, with only 200 customers, the counter goes from 360 to 72,000 rows and the service from about 2,000 to 74,000. Traffic is exactly the same. Nothing breaks, no usual dashboard turns red. Only the number of rows gives the problem away.

And 200 customers is not many. With 50,000 customers, do the math. A request ID or a full web address has the same effect, only worse.

What it costs

Section titled “What it costs”

More rows mean more storage, more memory to find the data and slower searches, often during an incident, when you need them most.

The lab puts a figure on storage: about 242 MB a month for the healthy service, about 8.9 GB with the extra column. Thirty-seven times more, for information nobody ever uses in aggregate. As elsewhere on this site, I give no prices: they vary from one contract to the next. If your vendor bills per series, the bill follows the same curve. The metrics bill simulator does the math with your own pricing.

Finding the culprit column

Section titled “Finding the culprit column”

Nobody comes forward to confess. The useful reflex is to count, for each column, how many different values it holds. In the lab, the “region” column is a false lead: it looks suspicious because it multiplies every row by three, yet it has only three values and will never have many more. The customer ID has 200 today, 300 next month, more the month after.

In my view, that is the point to remember: a column is not dangerous because it has many values at a given moment, but because its values have no limit.

In real life, the drift is often slow: the column arrives with a release, then the number of rows climbs day after day. An alert on how fast new rows appear warns you well before the bill does. Who is growing the metrics bill? turns this search into a method, with the tools that give the figures.

What to do?

Section titled “What to do?”

First, stop the leak: the platform can remove the column as data arrives, before storing it. In the lab, active rows fall back to about 2,000 within two or three minutes. What has already been written stays stored until the retention period ends: the bill does not drop overnight.

Then fix it for good in the application code, which otherwise keeps sending the column for nothing. The customer ID belongs elsewhere: in traces and logs, it costs much less and stays useful for investigations. To keep the link between the metric and one customer’s trace, there is a bridge: exemplars, covered below.

Finally, prepare for next time: a cap on rows per application limits the damage when a new extra column shows up. It comes at a price, since once reached it rejects legitimate rows too. It is a safety net, not a solution.

The most effective move is still not to let the column through: a written rule, checked automatically before each release, stating which columns a metric is allowed to carry. That is the subject of Governing observability data.

For technical teams: queries, configuration and caps

A series is identified by the metric name plus its full set of labels. For a histogram, each label combination also produces one series per bucket, plus _sum and _count: they are often the biggest contributors.

Find the metrics and labels responsible:

# The metrics with the most series
topk(10, count by (__name__) ({job="synthetic-app"}))


# The number of distinct values of each candidate label
count(count by (user_id) (http_requests_total))    # about 200
count(count by (region) (http_requests_total))     # 3
count(count by (endpoint) (http_requests_total))   # 8
count(count by (status) (http_requests_total))     # 5

Drop the label at ingestion, in a Prometheus-compatible agent (vmagent in the lab):

metric_relabel_configs:
  - regex: "user_id"
    action: labeldrop

This rule stops the leak, yet it has a flaw you need to know about. Once user_id is gone, the 72,000 series collapse onto 360 identical label sets: several series now carry exactly the same labels, with interleaved counter values. Depending on the storage, samples are rejected as duplicates or out of order, or rate() sees false counter resets and returns absurd values. The series count is right; the totals are not.

To keep correct totals, aggregate instead of dropping: vmagent stream aggregation sums the counters without the label before writing, or a recording rule does the same job afterwards. The lab ships this variant as an appendix (config/streamaggr.yml and docker-compose.streamaggr.yml):

- match: 'http_requests_total'
  interval: 30s
  by: [job, app_instance, endpoint, status]
  outputs: [total]

An OpenTelemetry Collector has the same trap: deleting the attribute with the attributes processor (action: delete) makes data points collide. The clean way is the OTTL function aggregate_on_attributes of the transform processor, available in recent versions of the contrib Collector (the lab does not use a Collector: check yours).

To link a metric to one customer’s trace without creating a series, use exemplars: a few samples of the metric carry the ID of the trace that produced them. The dashboard shows the error spike, one click opens the trace of a failing request and the customer ID can be read in that trace. That is exactly what the Monday morning developer needed.

For caps, the lab shows two vmagent guardrails: series_limit per target plus a global cap on new series per hour (-remoteWrite.maxHourlySeries). For alerts, it fires on active series, on their creation rate and on their growth ratio; the leak profile shows the last two firing before the absolute threshold.

Lab storage assumptions: a 15-second scrape interval (172,800 samples per series per month) and about 0.7 byte per sample after compression.

The lab

Section titled “The lab”

The cardinality-cost-lab replays the whole scenario in one to one and a half hours, on a machine with Docker: you start the healthy service, add the extra column, look for the culprit, fix it and measure the gain.

Terminal window
docker compose up -d --build   # the healthy service
make toxic                     # the extra column, all at once
make leak                      # the same, gradually over 5 minutes
make remediate                 # the fix
./validate.sh                  # replays the three steps and checks the thresholds
cardinality-cost-lab on the forgeThe lab repository: walkthrough, expected answers, load profiles, guardrails and gain report.
Metrics bill simulatorAdd or remove criteria and watch the number of rows, memory and cost change by orders of magnitude.

Sources

Section titled “Sources”