TechnicalBeginner5 min read (+2 technical)Video25 October 2026

Who is growing the metrics bill?

The metrics bill doubles while traffic stays flat: finance sees it first, only engineers can find the cause. A three-question method to find the metrics and labels responsible, then the monthly report that gets everyone on the same page.

For: engineers and SREs · architects · finance, risk and compliancePrerequisites: Having read Cardinality explained, or knowing what a series is. The "For technical teams" section assumes some knowledge of Prometheus.

Reading mode

End of the month at a software company. The financial controller opens the observability platform invoice: the “metrics” line has doubled in four weeks. They call the engineering team. Traffic has not moved, the dashboards are green, nobody changed anything “big”. This scenario is made up, yet it is common.

It is not an outage. When traffic stays flat, the most frequent cause is cardinality: someone added to a metric a label whose values never end. A customer ID, a request ID, a full web address, or the name of a pod that changes with every deployment. Cardinality explained tells you how the mechanism works. This page explains how to find the culprit.

Read the video text

End of the month: the metrics line on the invoice has doubled. Traffic has not moved. The dashboards are green.

Finance sees the increase first. It cannot see the cause. Engineers have the tools to see it. Nothing warns them. They need one shared question.

First, rule out the simple causes in five minutes. If nothing changed, the lead is one label too many.

Which metric grew the most? Because of which label? Since when? Above all: who owns it? Without an owner, nobody will decide to reduce it.

The answer fits on one page: the monthly report. The heaviest metrics, their trend, the heaviest label, their owner. Here, a full web address as a label: 180% more in one month. There, a metric nobody has read in five months.

Finance no longer reads an invoice line. It reads a list of decisions. Every decision has a name next to it: the owner’s. Teams see their metric before anyone comes to talk to them about it.

Without a report, the increase shows up on the invoice, often late. With the report, it shows up in the first month.

Made-up figures.

Who is growing the metrics bill? meantimetolearn.com.

Music: “Radar Focus”, Blue Saga (Epidemic Sound).

Those who see, those who pay

The hard part is mostly human. Finance feels the problem first, on the invoice, yet has no way to see its cause. Engineers have the tools to see it, yet nothing warns them: no screen turns red when a metric goes from a thousand to a hundred thousand series.

The way out is a question both sides understand. It fits in one sentence: which metrics grew the most, because of which label and since when? Then: who owns them?

Before blaming cardinality

A few simpler causes can be ruled out in five minutes by asking the engineering team:

are new machines, services or environments being monitored since last month?
has the collection interval been shortened (one sample every 15 seconds instead of every minute means four times more points)?
has data retention been extended?
has the contract or the billing model changed?

If the answer is no across the board, cardinality becomes the right lead. It matters even more if the vendor bills per series.

Three questions, in order

Think of an electricity bill that doubles. You do not start by taking every socket apart. You first look at which appliance uses the most, then at what changed on that appliance, then since when.

Which metric? A series is one row of the tracking spreadsheet: one combination of values, stored and often billed. Metrics databases can rank metrics by number of series. Often a handful of metrics alone weighs a large share of the total. That is where to look.

Which label? On the top metric, check how many distinct values each label takes. Most have a handful: a response code, a region, a service name. The culprit has thousands. And the number climbs every day.

Since when? A sudden jump points to a specific deployment, or to a library upgrade, which you can find in the release history. A slow, steady rise rather points to a label that piles up values: customers, pods, sessions.

The fourth question is not technical: who owns this metric? Without an owner, nobody will decide to reduce it.

The usual suspects

A few families of labels come up in almost every investigation:

IDs of people or business objects: customer, user, order, session;
unique technical IDs: request, trace or container ID;
full web addresses with their parameters, instead of the route template (/order/{id});
pod names in Kubernetes: every deployment creates new names, hence new series, even if the number of pods stays the same. This is called churn. It costs money even with a constant number of pods: some offers bill the active series over the period, or their hourly average, so every deployment adds series to the month’s count;
error messages copied as is into a label;
histograms crossed with too many labels: every combination is multiplied by the number of buckets;
attributes added by an upgrade: a new version of an OpenTelemetry SDK, instrumentation or exporter can add attributes, hence labels, without anyone having “changed anything”;
resource attributes turned into labels: when the configuration copies onto every series the attributes describing the machine or the pod (host name, container ID…), all series multiply at once.

One last suspect is quieter: the metric nobody looks at anymore. It does not necessarily grow, yet it costs money every month for nothing. Some tools can tell you when a metric was last queried.

The report that gets everyone on the same page

Once the method is known, the most useful step is to make it routine. A one-page monthly report is often enough: the ten metrics with the most series, how they changed over the month, the heaviest label on each one and their owner. For example (made-up figures):

Metric	Series	Over a month	Heaviest label	Owner
`http_server_request_duration_seconds_bucket`	412,000	+180%	`url`: 9,800 values	Payments team
`kube_pod_container_status_restarts_total`	96,000	+12%	`pod`: renewed with every deployment	Platform team
`app_cache_hits_total`	54,000	flat	`customer_id`	Catalog team, not read in five months

The first row calls for a quick decision: replace the full address with the route template. That is what the stable OpenTelemetry HTTP semantic conventions provide for: the http.server.request.duration metric carries the route template (http.route) rather than the full address, precisely to keep the number of series down. The third raises another question: should you keep a metric nobody reads?

Finance reads a list of decisions instead of an invoice line. Teams see their metric before anyone comes to talk to them about it. The next increase shows up after one month, not three.

For technical teams: where to find the numbers

Prometheus. The Status › TSDB Status page of the UI, or the matching API, gives the metrics and label=value pairs with the most series, the labels that have the most values and the memory those values take up:

curl -s 'http://prometheus:9090/api/v1/status/tsdb?limit=20'
# seriesCountByMetricName     : the heaviest metrics
# labelValueCountByLabelName  : the labels with the most values
# seriesCountByLabelValuePair : the heaviest label=value pairs
# memoryInBytesByLabelName    : the memory taken by each label's values

These figures cover the in-memory head block (the last few hours). To analyze a block already written to disk (the most recent one by default), including the labels most involved in churn, run promtool on the data directory (/prometheus in the official image). It opens the database read-only; on a busy production server, run it on a copy of the directory instead:

promtool tsdb analyze /prometheus
# Highest cardinality metric names, Highest cardinality labels,
# Label names most involved in churning

VictoriaMetrics. The Explore cardinality tab in vmui shows the same rankings for a given day, so you can compare with an earlier one. The API accepts a selector and a label to focus on:

curl -s 'http://victoriametrics:8428/api/v1/status/tsdb?topN=20&date=2026-10-25'
# -g: without it, curl interprets the brackets in match[]
curl -sg 'http://victoriametrics:8428/api/v1/status/tsdb?match[]=http_requests_total&focusLabel=user_id'

These addresses apply to the single-node version. In a cluster, the same API goes through vmselect: http://vmselect:8481/select/<accountID>/prometheus/api/v1/status/tsdb.

If usage tracking is enabled, the explorer also shows how many times each metric was queried and when it was last queried: the list of metrics nobody reads comes out in a few clicks.

In PromQL, for one metric, the number of values of a candidate label and the change over a week:

count(count by (user_id) (http_requests_total))
count(http_requests_total) / count(http_requests_total offset 7d)

Avoid count by (__name__) ({__name__=~".+"}) on a large database: the query scans every series and can become a problem itself. The status APIs above are made for this.

Resource attributes promoted to labels are controlled at ingestion. On the Prometheus side, the OTLP receiver only promotes the attributes listed in promote_resource_attributes; the promote_all_resource_attributes option promotes them all and the documentation warns about series churn. On the Collector side, the Prometheus exporter copies every resource attribute when resource_to_telemetry_conversion is enabled (a deprecated option, replaced by resource_constant_labels, which lets you choose the attributes).

Churn shows up in the rate of new series creation. On Prometheus:

rate(prometheus_tsdb_head_series_created_total[1h])

An alert on this rate warns you of a leak well before the invoice does; the lab in Cardinality explained shows an example.

Going further

Cardinality explained: the mechanism, with a video and a lab that replays the scenario.
Metrics bill simulator: add or remove labels and watch the number of series change.
Governing observability data: owners, series budgets and checks before release.

Sources

Prometheus, HTTP API: TSDB stats
Prometheus, promtool: the tsdb analyze command
VictoriaMetrics, cardinality explorer and /api/v1/status/tsdb
Prometheus, OTLP receiver configuration: promote_resource_attributes
OpenTelemetry Collector, Prometheus exporter: resource_constant_labels
OpenTelemetry, semantic conventions for HTTP metrics: http.server.request.duration and http.route
Prometheus, metric and label naming: the warning against labels with unlimited values