Skip to content

OrganizationPractitioner8 min read (+1 technical)

Governing observability data

A record card for every piece of data collected, checked automatically, volume ceilings, retention periods by type of data and a quarterly review: decide what gets collected before the bill or the auditor decides for you.

For: engineers and SREs · architects · team managers · finance, risk and compliancePrerequisites: No technical background needed. The page [Cardinality explained](/en/technical/cardinality/) helps with the section on volume ceilings.

Reading mode

An illustrative scenario. The observability tool’s bill has doubled in a year. Looking for the reason, the platform team finds that a single metric, added by a product team for a one-off need, carries the identifier of every customer. The same month, an audit finds email addresses in application logs kept for more than a year. Nobody had decided to keep them. Nobody had decided to delete them either.

Without governance, metrics (numbers tracked over time), logs (the lines of text applications write as they run) and traces (the path of a request from one service to the next) follow a natural slope. Every team adds metrics, attributes and logs. Nobody removes any. Retention stays at its default setting. The result shows first on the bill, then in slowing queries, sometimes in an audit.

Governing this data means deciding in advance what may be collected, in what quantity, for how long and where it goes, then checking automatically that those decisions are respected. I rely on four tools for this: the data contract, the cardinality budget, the retention policy and a quarterly review.

Think of the record card a warehouse keeps for every item it stores: who ordered it, what it is for, the maximum stock, which aisle it goes in. The data contract plays that role for each signal, meaning each metric, log or trace an application emits.

The glossary defines it as a written rule, checked in continuous integration, that sets for a signal the allowed attributes, the cardinality budget, the retention and the destination. Continuous integration is the automated pipeline that tests every code change before it is accepted. Attributes are the labels that come with a measurement, such as the service name or the response code. I recommend adding two fields to the contract: an owner and a note on any personal data the signal may contain.

Three choices deserve an explanation. The first is the “usage” field. Every signal must serve a purpose: an SLO (a quality-of-service objective with a number on it), an alert, a dashboard, a decision. A signal with no declared usage is the first candidate for removal at review time. This is the frugality rule proposed in lesson 3 of the CIO path.

The second is about attribute names. The OpenTelemetry semantic conventions provide standardized names and meanings; OpenTelemetry is the most widely used open standard for producing these signals. I recommend using those names rather than inventing an in-house vocabulary. Contracts then stay readable from one team to another and from one tool to another.

The third is forbidden attributes. The documentation of Prometheus, one of the most widely used metrics tools, explicitly advises against using labels to store high-cardinality dimensions such as user IDs or email addresses. Writing these bans into the contract makes them checkable.

For technical teams: an example contract

Here is what a contract for a metric can look like. The format is an illustrative example, to be adapted to your tooling; it does not follow the schema of any specific tool.

signal: http.server.request.duration
type: histogram
owner: payments-team
usage: [payments-slo, payments-dashboard]
allowed_attributes:
- service.name
- http.request.method
- http.response.status_code
- http.route # normalized route, never the raw path
forbidden_attributes:
- user.id # unbounded: belongs on traces
- url.full
cardinality_budget: 5000 # maximum active series for this signal
personal_data: no
retention: operations-class
destination: hot-metrics-storage

A contract that is only checked by eye, in review, ends up ignored. I recommend automated checks, from the simplest to the most complete. The first checks how metrics are named and formatted. The second checks that attributes follow the semantic conventions. The third compares what the code emits with the contracts and rejects a forbidden attribute or a signal with no owner or usage.

In my view, that third check is the most useful. It is also the one no generic tool provides as is, since it depends on your contracts. It can start small: a script that rejects a merge (bringing a change into the shared code) when a new signal has no contract.

For technical teams: the checks and possible tools
CheckWhat it blocksPossible tools
Naming and formatnon-compliant names, missing unitspromtool check metrics for metrics in Prometheus format
Convention complianceunknown or mistyped attributesOTel Weaver, which validates a semantic convention registry
Contractforbidden attributes, signal without owner or usagean in-house test comparing declared instrumentation with the contracts

Picture a sales tracking spreadsheet. One row per store is easy to read. One row per store and per product already makes it much bigger. With one row per store, per product and per customer, it becomes unmanageable. Cardinality is that number of rows: the number of distinct time series a metric produces, one series per combination of label values. It multiplies with every label you add, which the cardinality explosion simulator makes visible. The page Cardinality explained tells the story in detail.

A cardinality budget sets a ceiling of active series per signal, per service or per team. I recommend setting it at two levels. Per signal first, in the contract, to stop in review the attribute that would blow up a metric. Per team next, as an overall envelope, so the team makes its own trade-offs between its signals.

The budget must also be enforced while applications are running, because a mistake always gets through review one day. Collection tools can reject or clean up whatever exceeds the ceiling before the bill swells. The Collector cost simulator quantifies the effect of these levers.

That leaves exceptions. A budget with no possible exception gets bypassed. I recommend a short, written process: the team asks, justifies the usage and proposes a duration; the platform team estimates the impact; the decision is recorded in the contract with an end date. An exception without an end date becomes the rule.

For technical teams: enforcing the budget at runtime

On the Prometheus side, the sample_limit and label_limit scrape settings make the scrape of a target fail when it exceeds a ceiling (Prometheus configuration). In an OpenTelemetry Collector, the filter and transform processors drop or rewrite forbidden attributes before export.

At home, you throw away a till receipt after a few weeks, keep bank statements for a few years and payslips much longer. Observability data works the same way: not all of it has the same useful life. Retention is that storage period. I recommend defining a small number of retention classes and attaching each signal to a class in its contract.

“Hot” storage is fast and expensive, “cold” storage slow and cheap, “warm” in between. A “nominal” trace is one for a request that went fine; often only a sample of those is kept.

ClassExample signalsDurationStorage
Diagnosissampled nominal traces, authorized debug logsa few dayshot
Operationsoperational metrics, application logs, error tracesa few weekshot then warm
Trendaggregated metrics (per service, reduced resolution)several months to a few yearscold or compressed
Evidenceaudit logs, traces with evidential valueset by the retention policy and applicable regulationscold, immutable if needed

The durations in this table are illustrative orders of magnitude. Evidence-class durations are not chosen based on cost: they come from the company’s retention policy and sector regulations, as explained in lesson 2 of the CIO path. For the integrity of these logs, see audit log integrity.

Personal data needs special attention. The GDPR sets out the principles of data minimization and storage limitation (Regulation (EU) 2016/679, article 5). Applied to logs and traces, they mean that a user identifier, an IP address or request content should only be collected if it serves a declared usage. Nor should it be kept longer than that usage requires. The contract’s “personal data” field makes this choice explicit and checkable by the DPO.

Contracts and budgets age. I recommend a one-hour review every quarter. It brings together the platform team, product team champions and a FinOps representative (the function that manages infrastructure spending with the teams). Security and the DPO join at least once a year.

A typical agenda fits in seven items:

  1. Change in volume and cardinality per team since the previous review.
  2. The ten most expensive signals and their declared usage.
  3. Signals with no observed usage over the quarter: remove or justify.
  4. Exceptions that have reached their end date: close or renew with a new date.
  5. Incidents where a signal was missing: add it, with a contract.
  6. Retention and personal data deviations found.
  7. Decisions, with an owner and a date.

Item 5 matters as much as the others. Governance is not only about cutting. A post-mortem that concludes there was an observability gap must be able to add a signal quickly. See blameless post-mortems.

These indicators tell you whether governance holds in practice. An “active series” is one that is still receiving values.

IndicatorWhat it reveals
Share of signals covered by a contractthe real reach of governance
Share of signals with a declared and observed usagehow frugal collection is
Active series per team, against their budgetwhether cardinality budgets hold
Number of merges blocked by contract checkswhether rules are applied or bypassed
Open and expired exceptionsthe discipline of the exception process
Signals containing personal data outside any contractcompliance risk
Cost per team and cost per business transactionthe link with FinOps management in lesson 4
  • A data contract format is chosen and documented
  • Every new signal comes with a contract, an owner and a usage
  • Attribute names follow the OpenTelemetry semantic conventions
  • Continuous integration blocks a signal without a contract
  • A cardinality budget exists per team and is enforced at runtime
  • The exception process is written down and every exception has an end date
  • Every signal is attached to a retention class
  • The presence of personal data is declared and reviewed with the DPO
  • The quarterly review takes place and produces recorded decisions