BusinessPractitioner10 min read
4. Observability FinOps
Understand why the bill drifts, read it line by line, pull the technical and contractual levers, then set up governance that stops it from doubling quietly.
For: architects · finance, risk and compliance · executives and CIOsPrerequisites: Basic notions of metrics, logs and traces.
Observability was long sold as insurance. With volume-based billing, it looks more like a water bill: nobody decides to double consumption, yet a dripping tap in a room nobody visits is enough to push it up. FinOps, a blend of finance and operations, is the discipline of steering this kind of spending together with the people who generate it. This lesson gives the CIO a framework to take back control: understand the bill, pull the levers, set up governance.
Why does the bill drift?
Section titled “Why does the bill drift?”The first reason lies in the billing model, which rewards verbosity. Platforms bill on a mix of units: hosts or containers, ingested volume (logs, traces, custom metrics), indexed volume, users, modules. When the unit is volume, every engineering decision costs money: a log level switched to debug, full tracing left on, an extra tag. An application that throws a lot of errors also produces a lot of logs. Technical debt then gets paid twice.
The second is cardinality, the number of different combinations a metric can take. A metric described by route, status code and method stays reasonable: a few dozen combinations. Add a user or session ID and the number of series, those curves the platform tracks and bills one by one, jumps by an order of magnitude. The cardinality simulator shows this live.
The third is the lack of a feedback loop. Those who instrument do not see the bill and those who pay it do not see the decisions that drive it. Without that link, drift is the natural slope.
These mechanisms apply to every solution. When self-managed, drift does not show up on a bill but in storage, compute and team time. It is just less visible.
Reading a bill line by line
Section titled “Reading a bill line by line”An observability bill usually has seven to ten lines. Their relative weight varies by vendor and by your usage, which is why it pays to measure it in your own environment. The table below is a reading guide.
| Line | Typical unit | What makes it drift |
|---|---|---|
| Hosts, containers | flat fee per monitored unit | autoscaling, test environments left running |
| Log ingestion | ingested volume | debug level, duplicated logs, non-production environments |
| Log indexing and retention | indexed volume, duration | indexing everything instead of archiving |
| Custom metrics | active series | cardinality |
| Traces | ingested or indexed spans | full tracing, long retention |
| Optional modules | per unit or per test | modules switched on for a trial and never revisited |
| Users | per seat | low in value, useful in negotiation |
| Security, SIEM, automation | variable | recent lines, to be shared with the CISO’s budget |
The first exercise is simple and rarely done: ask for the bill broken down by line, by team and by environment over a rolling twelve months. Without that view, no serious discussion is possible.
The technical levers
Section titled “The technical levers”Trace sampling
Section titled “Trace sampling”A trace tells the story of a request’s journey through the system. You do not need to keep every one of those stories, any more than a quality check examines every part coming off a production line. There are two ways to sort them. Head sampling decides at the start of the trace, at random: it is simple, but it lets through the rare traces that matter most, the ones with errors or slowness. Tail sampling waits until the trace ends to decide. It keeps every error, every trace above a slowness threshold and a fraction of normal traffic. It requires an intermediate collection layer, usually an OpenTelemetry Collector. The sampling simulator compares the two at equal volume.
Log retention tiers
Section titled “Log retention tiers”An authentication log and a debug log have neither the same value nor the same lifespan. Think of your office: current files on the desk, last year’s in a cupboard, archives in the basement. A three-tier scheme follows the same logic. The hot tier is indexed and searchable instantly, for a few days to a few weeks. The warm tier is archived and compressed, retrievable within a day. The cold tier is low-cost object storage, for compliance. The durations for each category follow the grid in lesson 2.
Controlling cardinality
Section titled “Controlling cardinality”You do not use a unique identifier as a metric dimension: that information belongs in traces. I recommend publishing a cap on series per service and setting up an automated check that rejects a metric above the threshold.
Upstream aggregation
Section titled “Upstream aggregation”Aggregating at the source, or at the Collector, rather than at the provider is like sending the day’s total instead of every till receipt: one value per second instead of one data point per request, with no loss for everyday dashboards.
Measuring with intent
Section titled “Measuring with intent”A few dozen well-chosen business metrics are worth more than thousands of metrics emitted by default by a framework. For each metric, you should know what it measures, which decision it informs and who looks at it. A metric nobody looks at is pure spend.
The Collector cost simulator puts a figure on these levers for your volume.
The contractual levers
Section titled “The contractual levers”Vendors grant discounts in exchange for a multi-year commitment or a committed volume. The trap is committing to an overestimated trajectory. In my view, the prudent approach is to commit below the usage observed over the last twelve months and to negotiate flexibility, both upward and downward.
Some clauses should be requested in every contract:
- a billing cap, with notification and the option to suspend ingestion rather than pay for an abnormal spike;
- a long notice period before any price increase;
- the right to audit your own usage through the API, without restriction;
- documented reversibility, with export in an open format;
- unit prices held if the offering changes or the vendor is acquired;
- a ban on training models on your data without explicit consent.
To keep negotiating power, the simplest route is OpenTelemetry instrumentation that sends to one or more interchangeable backends. It also lets you separate uses: one backend for day-to-day operations, cold storage for compliance.
Then there is benchmarking the market. Coming to a renewal with two or three comparable offers, on the same scope and the same requirements, changes the incumbent vendor’s attitude. The exercise takes a few weeks.
FinOps governance
Section titled “FinOps governance”The FinOps Foundation describes three maturity levels, Crawl, Walk and Run, assessed capability by capability. It states that reaching Run everywhere is not a goal in itself (FinOps maturity model). The mapping that follows is my reading of that model applied to observability, not a definition from the Foundation.
At the Crawl level, you gain visibility: you know what you spend by team, by application and by environment. That requires tags enforced at ingestion (team, application, environment, criticality). At the Walk level, each team sees its monthly cost (known as showback) and answers for it in a review. At the Run level, you move to continuous optimization: internal chargeback, budgets per team, trade-offs handled as for the cloud.
Who does it?
Section titled “Who does it?”Three models exist. FinOps can be centralized in the platform team, which is clear but creates a bottleneck. It can be federated, with a champion per team and a small central unit. Or it can be delegated to the vendor. That last model is comfortable, but the vendor has no interest in lowering its bill for good: it cannot be your main model.
The rituals
Section titled “The rituals”I recommend a monthly drift review: thirty minutes on the month’s biggest increases, their causes and the decisions taken. Add a quarterly committee on usage forecasts and commitments. After any billing surprise, a blameless post-mortem with the team involved helps you understand what happened without pointing fingers.
The indicators
Section titled “The indicators”Four indicators are enough to follow the whole picture.
| Indicator | What it tells you | Expected trend |
|---|---|---|
| Cost per business transaction | the cost of observability relative to activity | stable or falling |
| Committed share of the bill | what is covered by negotiated commitments | to be set according to your predictability |
| Cost per team or per service | who consumes what | compared month over month |
| Observability to infrastructure ratio | observability spend relative to infrastructure spend | tracked over time, compared across entities |
Classic traps
Section titled “Classic traps”The ratchet effect is the sneakiest: a usage tier reached during a spike stays billed when usage comes back down. Negotiate the way down in the initial contract, not just the way up.
Non-production environments come next. Development, test and staging are often instrumented like production and left running overnight. Measure their share: it is often surprising.
Abandoned dashboards consume queries and therefore budget. An annual review to delete the ones nobody opens is an easy lever. Finally, tracing drift: full tracing switched on during an incident and never turned back down, as in the journal above.
Exercises
Section titled “Exercises”The first is a quick bill audit, in two hours. Plot each line over twelve months, spot the three that grew the most and look for the decision that explains each one. If you cannot find it, visibility is your first project.
The second is a quick benchmark, over a few weeks: ask three competitors for an offer on a scope identical to yours and compare them with your bill.
The third is about governance and takes a quarter. Write a one-page FinOps RACI (who optimizes, who answers for the budget, who is consulted, who is informed), circulate it, then launch the monthly review.
To check where you stand, here is a checklist:
- Costs broken down by team, application and environment
- Standardized tags mandatory at ingestion
- Traces tail-sampled rather than kept at 100%
- Logs classified into hot, warm and cold tiers, with justified durations
- Cap, reversibility and no-training clauses in the contract
- At least two comparable offers known before the next renewal
- Monthly drift review with the teams
- Cost per business transaction tracked
Going further
Section titled “Going further”- The Collector cost simulator, to put a figure on each lever for your volume.
- The business dimension.