Skip to content

BusinessPractitioner10 min read

4. Observability FinOps

Understand why the bill drifts, read it line by line, pull the technical and contractual levers, then set up governance that stops it from doubling quietly.

For: architects · finance, risk and compliance · executives and CIOsPrerequisites: Basic notions of metrics, logs and traces.

Reading mode

Observability was long sold as insurance. With volume-based billing, it looks more like a water bill: nobody decides to double consumption, yet a dripping tap in a room nobody visits is enough to push it up. FinOps, a blend of finance and operations, is the discipline of steering this kind of spending together with the people who generate it. This lesson gives the CIO a framework to take back control: understand the bill, pull the levers, set up governance.

The first reason lies in the billing model, which rewards verbosity. Platforms bill on a mix of units: hosts or containers, ingested volume (logs, traces, custom metrics), indexed volume, users, modules. When the unit is volume, every engineering decision costs money: a log level switched to debug, full tracing left on, an extra tag. An application that throws a lot of errors also produces a lot of logs. Technical debt then gets paid twice.

The second is cardinality, the number of different combinations a metric can take. A metric described by route, status code and method stays reasonable: a few dozen combinations. Add a user or session ID and the number of series, those curves the platform tracks and bills one by one, jumps by an order of magnitude. The cardinality simulator shows this live.

The third is the lack of a feedback loop. Those who instrument do not see the bill and those who pay it do not see the decisions that drive it. Without that link, drift is the natural slope.

These mechanisms apply to every solution. When self-managed, drift does not show up on a bill but in storage, compute and team time. It is just less visible.

An observability bill usually has seven to ten lines. Their relative weight varies by vendor and by your usage, which is why it pays to measure it in your own environment. The table below is a reading guide.

LineTypical unitWhat makes it drift
Hosts, containersflat fee per monitored unitautoscaling, test environments left running
Log ingestioningested volumedebug level, duplicated logs, non-production environments
Log indexing and retentionindexed volume, durationindexing everything instead of archiving
Custom metricsactive seriescardinality
Tracesingested or indexed spansfull tracing, long retention
Optional modulesper unit or per testmodules switched on for a trial and never revisited
Usersper seatlow in value, useful in negotiation
Security, SIEM, automationvariablerecent lines, to be shared with the CISO’s budget

The first exercise is simple and rarely done: ask for the bill broken down by line, by team and by environment over a rolling twelve months. Without that view, no serious discussion is possible.

A trace tells the story of a request’s journey through the system. You do not need to keep every one of those stories, any more than a quality check examines every part coming off a production line. There are two ways to sort them. Head sampling decides at the start of the trace, at random: it is simple, but it lets through the rare traces that matter most, the ones with errors or slowness. Tail sampling waits until the trace ends to decide. It keeps every error, every trace above a slowness threshold and a fraction of normal traffic. It requires an intermediate collection layer, usually an OpenTelemetry Collector. The sampling simulator compares the two at equal volume.

An authentication log and a debug log have neither the same value nor the same lifespan. Think of your office: current files on the desk, last year’s in a cupboard, archives in the basement. A three-tier scheme follows the same logic. The hot tier is indexed and searchable instantly, for a few days to a few weeks. The warm tier is archived and compressed, retrievable within a day. The cold tier is low-cost object storage, for compliance. The durations for each category follow the grid in lesson 2.

You do not use a unique identifier as a metric dimension: that information belongs in traces. I recommend publishing a cap on series per service and setting up an automated check that rejects a metric above the threshold.

Aggregating at the source, or at the Collector, rather than at the provider is like sending the day’s total instead of every till receipt: one value per second instead of one data point per request, with no loss for everyday dashboards.

A few dozen well-chosen business metrics are worth more than thousands of metrics emitted by default by a framework. For each metric, you should know what it measures, which decision it informs and who looks at it. A metric nobody looks at is pure spend.

The Collector cost simulator puts a figure on these levers for your volume.

Vendors grant discounts in exchange for a multi-year commitment or a committed volume. The trap is committing to an overestimated trajectory. In my view, the prudent approach is to commit below the usage observed over the last twelve months and to negotiate flexibility, both upward and downward.

Some clauses should be requested in every contract:

  1. a billing cap, with notification and the option to suspend ingestion rather than pay for an abnormal spike;
  2. a long notice period before any price increase;
  3. the right to audit your own usage through the API, without restriction;
  4. documented reversibility, with export in an open format;
  5. unit prices held if the offering changes or the vendor is acquired;
  6. a ban on training models on your data without explicit consent.

To keep negotiating power, the simplest route is OpenTelemetry instrumentation that sends to one or more interchangeable backends. It also lets you separate uses: one backend for day-to-day operations, cold storage for compliance.

Then there is benchmarking the market. Coming to a renewal with two or three comparable offers, on the same scope and the same requirements, changes the incumbent vendor’s attitude. The exercise takes a few weeks.

The FinOps Foundation describes three maturity levels, Crawl, Walk and Run, assessed capability by capability. It states that reaching Run everywhere is not a goal in itself (FinOps maturity model). The mapping that follows is my reading of that model applied to observability, not a definition from the Foundation.

At the Crawl level, you gain visibility: you know what you spend by team, by application and by environment. That requires tags enforced at ingestion (team, application, environment, criticality). At the Walk level, each team sees its monthly cost (known as showback) and answers for it in a review. At the Run level, you move to continuous optimization: internal chargeback, budgets per team, trade-offs handled as for the cloud.

Three models exist. FinOps can be centralized in the platform team, which is clear but creates a bottleneck. It can be federated, with a champion per team and a small central unit. Or it can be delegated to the vendor. That last model is comfortable, but the vendor has no interest in lowering its bill for good: it cannot be your main model.

I recommend a monthly drift review: thirty minutes on the month’s biggest increases, their causes and the decisions taken. Add a quarterly committee on usage forecasts and commitments. After any billing surprise, a blameless post-mortem with the team involved helps you understand what happened without pointing fingers.

Four indicators are enough to follow the whole picture.

IndicatorWhat it tells youExpected trend
Cost per business transactionthe cost of observability relative to activitystable or falling
Committed share of the billwhat is covered by negotiated commitmentsto be set according to your predictability
Cost per team or per servicewho consumes whatcompared month over month
Observability to infrastructure ratioobservability spend relative to infrastructure spendtracked over time, compared across entities

The ratchet effect is the sneakiest: a usage tier reached during a spike stays billed when usage comes back down. Negotiate the way down in the initial contract, not just the way up.

Non-production environments come next. Development, test and staging are often instrumented like production and left running overnight. Measure their share: it is often surprising.

Abandoned dashboards consume queries and therefore budget. An annual review to delete the ones nobody opens is an easy lever. Finally, tracing drift: full tracing switched on during an incident and never turned back down, as in the journal above.

The first is a quick bill audit, in two hours. Plot each line over twelve months, spot the three that grew the most and look for the decision that explains each one. If you cannot find it, visibility is your first project.

The second is a quick benchmark, over a few weeks: ask three competitors for an offer on a scope identical to yours and compare them with your bill.

The third is about governance and takes a quarter. Write a one-page FinOps RACI (who optimizes, who answers for the budget, who is consulted, who is informed), circulate it, then launch the monthly review.

To check where you stand, here is a checklist:

  • Costs broken down by team, application and environment
  • Standardized tags mandatory at ingestion
  • Traces tail-sampled rather than kept at 100%
  • Logs classified into hot, warm and cold tiers, with justified durations
  • Cap, reversibility and no-training clauses in the contract
  • At least two comparable offers known before the next renewal
  • Monthly drift review with the teams
  • Cost per business transaction tracked