OrganizationPractitioner8 min read (+1 technical)
Governing observability data
A record card for every piece of data collected, checked automatically, volume ceilings, retention periods by type of data and a quarterly review: decide what gets collected before the bill or the auditor decides for you.
For: engineers and SREs · architects · team managers · finance, risk and compliancePrerequisites: No technical background needed. The page [Cardinality explained](/en/technical/cardinality/) helps with the section on volume ceilings.
An illustrative scenario. The observability tool’s bill has doubled in a year. Looking for the reason, the platform team finds that a single metric, added by a product team for a one-off need, carries the identifier of every customer. The same month, an audit finds email addresses in application logs kept for more than a year. Nobody had decided to keep them. Nobody had decided to delete them either.
Without governance, metrics (numbers tracked over time), logs (the lines of text applications write as they run) and traces (the path of a request from one service to the next) follow a natural slope. Every team adds metrics, attributes and logs. Nobody removes any. Retention stays at its default setting. The result shows first on the bill, then in slowing queries, sometimes in an audit.
Governing this data means deciding in advance what may be collected, in what quantity, for how long and where it goes, then checking automatically that those decisions are respected. I rely on four tools for this: the data contract, the cardinality budget, the retention policy and a quarterly review.
The data contract
Section titled “The data contract”Think of the record card a warehouse keeps for every item it stores: who ordered it, what it is for, the maximum stock, which aisle it goes in. The data contract plays that role for each signal, meaning each metric, log or trace an application emits.
The glossary defines it as a written rule, checked in continuous integration, that sets for a signal the allowed attributes, the cardinality budget, the retention and the destination. Continuous integration is the automated pipeline that tests every code change before it is accepted. Attributes are the labels that come with a measurement, such as the service name or the response code. I recommend adding two fields to the contract: an owner and a note on any personal data the signal may contain.
Three choices deserve an explanation. The first is the “usage” field. Every signal must serve a purpose: an SLO (a quality-of-service objective with a number on it), an alert, a dashboard, a decision. A signal with no declared usage is the first candidate for removal at review time. This is the frugality rule proposed in lesson 3 of the CIO path.
The second is about attribute names. The OpenTelemetry semantic conventions provide standardized names and meanings; OpenTelemetry is the most widely used open standard for producing these signals. I recommend using those names rather than inventing an in-house vocabulary. Contracts then stay readable from one team to another and from one tool to another.
The third is forbidden attributes. The documentation of Prometheus, one of the most widely used metrics tools, explicitly advises against using labels to store high-cardinality dimensions such as user IDs or email addresses. Writing these bans into the contract makes them checkable.
For technical teams: an example contract
Here is what a contract for a metric can look like. The format is an illustrative example, to be adapted to your tooling; it does not follow the schema of any specific tool.
signal: http.server.request.durationtype: histogramowner: payments-teamusage: [payments-slo, payments-dashboard]allowed_attributes: - service.name - http.request.method - http.response.status_code - http.route # normalized route, never the raw pathforbidden_attributes: - user.id # unbounded: belongs on traces - url.fullcardinality_budget: 5000 # maximum active series for this signalpersonal_data: noretention: operations-classdestination: hot-metrics-storageChecking in continuous integration
Section titled “Checking in continuous integration”A contract that is only checked by eye, in review, ends up ignored. I recommend automated checks, from the simplest to the most complete. The first checks how metrics are named and formatted. The second checks that attributes follow the semantic conventions. The third compares what the code emits with the contracts and rejects a forbidden attribute or a signal with no owner or usage.
In my view, that third check is the most useful. It is also the one no generic tool provides as is, since it depends on your contracts. It can start small: a script that rejects a merge (bringing a change into the shared code) when a new signal has no contract.
For technical teams: the checks and possible tools
| Check | What it blocks | Possible tools |
|---|---|---|
| Naming and format | non-compliant names, missing units | promtool check metrics for metrics in Prometheus format |
| Convention compliance | unknown or mistyped attributes | OTel Weaver, which validates a semantic convention registry |
| Contract | forbidden attributes, signal without owner or usage | an in-house test comparing declared instrumentation with the contracts |
The cardinality budget
Section titled “The cardinality budget”Picture a sales tracking spreadsheet. One row per store is easy to read. One row per store and per product already makes it much bigger. With one row per store, per product and per customer, it becomes unmanageable. Cardinality is that number of rows: the number of distinct time series a metric produces, one series per combination of label values. It multiplies with every label you add, which the cardinality explosion simulator makes visible. The page Cardinality explained tells the story in detail.
A cardinality budget sets a ceiling of active series per signal, per service or per team. I recommend setting it at two levels. Per signal first, in the contract, to stop in review the attribute that would blow up a metric. Per team next, as an overall envelope, so the team makes its own trade-offs between its signals.
The budget must also be enforced while applications are running, because a mistake always gets through review one day. Collection tools can reject or clean up whatever exceeds the ceiling before the bill swells. The Collector cost simulator quantifies the effect of these levers.
That leaves exceptions. A budget with no possible exception gets bypassed. I recommend a short, written process: the team asks, justifies the usage and proposes a duration; the platform team estimates the impact; the decision is recorded in the contract with an end date. An exception without an end date becomes the rule.
For technical teams: enforcing the budget at runtime
On the Prometheus side, the sample_limit and label_limit scrape settings make the scrape of a target fail when it exceeds a ceiling (Prometheus configuration). In an OpenTelemetry Collector, the filter and transform processors drop or rewrite forbidden attributes before export.
Retention
Section titled “Retention”At home, you throw away a till receipt after a few weeks, keep bank statements for a few years and payslips much longer. Observability data works the same way: not all of it has the same useful life. Retention is that storage period. I recommend defining a small number of retention classes and attaching each signal to a class in its contract.
“Hot” storage is fast and expensive, “cold” storage slow and cheap, “warm” in between. A “nominal” trace is one for a request that went fine; often only a sample of those is kept.
| Class | Example signals | Duration | Storage |
|---|---|---|---|
| Diagnosis | sampled nominal traces, authorized debug logs | a few days | hot |
| Operations | operational metrics, application logs, error traces | a few weeks | hot then warm |
| Trend | aggregated metrics (per service, reduced resolution) | several months to a few years | cold or compressed |
| Evidence | audit logs, traces with evidential value | set by the retention policy and applicable regulations | cold, immutable if needed |
The durations in this table are illustrative orders of magnitude. Evidence-class durations are not chosen based on cost: they come from the company’s retention policy and sector regulations, as explained in lesson 2 of the CIO path. For the integrity of these logs, see audit log integrity.
Personal data needs special attention. The GDPR sets out the principles of data minimization and storage limitation (Regulation (EU) 2016/679, article 5). Applied to logs and traces, they mean that a user identifier, an IP address or request content should only be collected if it serves a declared usage. Nor should it be kept longer than that usage requires. The contract’s “personal data” field makes this choice explicit and checkable by the DPO.
The quarterly review
Section titled “The quarterly review”Contracts and budgets age. I recommend a one-hour review every quarter. It brings together the platform team, product team champions and a FinOps representative (the function that manages infrastructure spending with the teams). Security and the DPO join at least once a year.
A typical agenda fits in seven items:
- Change in volume and cardinality per team since the previous review.
- The ten most expensive signals and their declared usage.
- Signals with no observed usage over the quarter: remove or justify.
- Exceptions that have reached their end date: close or renew with a new date.
- Incidents where a signal was missing: add it, with a contract.
- Retention and personal data deviations found.
- Decisions, with an owner and a date.
Item 5 matters as much as the others. Governance is not only about cutting. A post-mortem that concludes there was an observability gap must be able to add a signal quickly. See blameless post-mortems.
What to measure
Section titled “What to measure”These indicators tell you whether governance holds in practice. An “active series” is one that is still receiving values.
| Indicator | What it reveals |
|---|---|
| Share of signals covered by a contract | the real reach of governance |
| Share of signals with a declared and observed usage | how frugal collection is |
| Active series per team, against their budget | whether cardinality budgets hold |
| Number of merges blocked by contract checks | whether rules are applied or bypassed |
| Open and expired exceptions | the discipline of the exception process |
| Signals containing personal data outside any contract | compliance risk |
| Cost per team and cost per business transaction | the link with FinOps management in lesson 4 |
Getting started
Section titled “Getting started”- A data contract format is chosen and documented
- Every new signal comes with a contract, an owner and a usage
- Attribute names follow the OpenTelemetry semantic conventions
- Continuous integration blocks a signal without a contract
- A cardinality budget exists per team and is enforced at runtime
- The exception process is written down and every exception has an end date
- Every signal is attached to a retention class
- The presence of personal data is declared and reviewed with the DPO
- The quarterly review takes place and produces recorded decisions
Going further
Section titled “Going further”- Who owns the contracts and who decides on overruns depends on the team model.
- The cardinality explosion simulator shows why a single unbounded attribute changes the order of magnitude of the bill.
- The collection pipeline diagram shows where the rules apply.
Sources
Section titled “Sources”- OpenTelemetry, Semantic Conventions and OTel Weaver.
- Prometheus, Metric and label naming and Configuration (
sample_limit,label_limit). - Regulation (EU) 2016/679 (GDPR), article 5.