BusinessBeginner11 min read (+1 technical)
What a business or product owner can ask of observability
The questions a business or product owner can ask of observability, what the engineering team must set up to answer them, how to turn a customer promise into a measurable objective and where to stand during incidents.
For: business and product · team managers · executives and CIOsPrerequisites: None.
Monday morning, executive committee. The day before, the website was slow for part of the afternoon. The sales director asks a simple question: “How many customers were affected?” The engineering team answers with what it has at hand. One server’s processor went up to 95%; the error rate doubled for twenty minutes. Nobody can say how many customers gave up on their order.
This scenario is made up, though it is common. The engineering team did nothing wrong: nobody had asked it to count customers. This page is for business and product owners who want to ask that kind of question ahead of time. It requires no technical knowledge. Each term is explained the first time it appears and the glossary collects the definitions.
Why does it concern you?
Section titled “Why does it concern you?”Think of a car dashboard. It only shows what the manufacturer decided to put there: without an oil gauge, you will not know the level is dropping. Observability works the same way. It is the ability to understand what happens inside an IT system from the data it emits. People often associate it with server outages. Yet it first answers business questions. Do our customers achieve what they came to do? How long do they wait? What does each use cost?
These answers do not appear on their own. The engineering team instruments the system, meaning it places the measurement points that will answer the questions it is asked. If nobody asks the business questions, it measures what it knows: processors, memory, technical errors. That is what happened at Monday’s committee. The Bridges page develops this idea and offers a translation dictionary between technical signals and business indicators.
Your role is therefore to frame the questions. The engineering team’s role is to make the answers possible.
The questions you can ask
Section titled “The questions you can ask”Three words come up in this section. A signal is a type of data emitted by the application. A trace tells the path of a request through all the services, like the tracking of a parcel moving from warehouse to warehouse. An attribute is a label stuck on that parcel, for example “journey step = payment”. The details are in Observability in 10 minutes.
For each question, I give the indicator that answers it, what to ask the team for and how much effort it takes in my view. The technical names are grouped in a collapsible section at the end.
“How many customers were affected by yesterday’s incident and for how long?”
Section titled ““How many customers were affected by yesterday’s incident and for how long?””The indicator is the number of distinct customers whose requests failed or were too slow, with the start time and the end time. It comes from traces and from logs (the application’s logbook), provided they carry a customer identifier or at least a customer segment.
So ask for a segment attribute on traces. If the data protection officer agrees, also ask for a pseudonymized customer identifier, meaning one replaced by a code that does not directly reveal who the person is.
A point of caution: this identifier goes on traces, not on metrics. A metric is a counter, a bit like a tracking sheet where each combination of labels opens a new row. Putting the customer identifier on it opens one row per customer and multiplies the storage cost. This is called cardinality.
In my view the effort is modest if traces already exist. It is heavier if the services must be traced first.
“Where in the purchase journey do customers drop out or wait?”
Section titled ““Where in the purchase journey do customers drop out or wait?””Picture a store that counts customers at the entrance of each aisle and then at the checkout, noting how long they spend in each place. The indicator does the same for an online journey: for each step, the share of customers who move on to the next one and the time they spend there.
It relies on traces carrying the journey step, linked to business events such as “cart confirmed” or “order paid”. Ask for a step attribute, the emission of those key events and a dashboard per step that anyone can read without training.
In my view this is a project of a few iterations. Move forward step by step, starting with the one that worries you most.
“Is the new feature slower than the old one?”
Section titled ““Is the new feature slower than the old one?””You compare response time between the two versions. The average is not enough: it hides the worst-served customers, the way the average wait at a counter hides the queue that builds up at lunchtime. Teams often look at p95, the time under which 95% of requests complete.
To compare, every piece of data must carry the service version. OpenTelemetry, the open collection standard, provides an attribute for this. If the new feature is rolled out gradually with a feature flag (a switch that turns it on for only part of your customers), the flag name is needed too.
The effort is low if the version is already filled in. I recommend making it a condition of every production release.
“How much does our AI feature cost per conversation?”
Section titled ““How much does our AI feature cost per conversation?””AI model providers bill per token, the unit of text they count, a bit like a taxi meter counts distance. The indicator is the number of tokens consumed per conversation multiplied by the unit price in your contract.
Traces of calls to the model can carry those tokens: OpenTelemetry’s conventions for generative AI cover the input and output tokens of each call. Ask for a conversation identifier on each call. Also ask for an attribute naming the feature concerned, so the cost can be allocated.
The effort is reasonable if the application goes through a shared layer to call models. Otherwise, plan for it from the design stage.
“Are we within our commitment to customers?”
Section titled ““Are we within our commitment to customers?””The indicator compares the service level actually observed with the promise and shows the margin left. It relies on success and response time measurements on the journeys covered by the promise. Ask for one SLO per important journey (the next section explains the term), tracked in a dashboard you can open yourself.
In my view this is the most structuring of the five questions, because it forces everyone to agree on the promise. That is the subject of the next section.
For technical teams: the attributes mentioned
The attribute names below are illustrative. Choose your own and write them down in a shared naming rule.
- Customer segment and pseudonymized customer identifier: on traces and logs, never as a metric label.
- Journey step: for example
journey.stepon traces, linked to the emission of the key business events. - Version: the OpenTelemetry resource attribute
service.versionon every piece of data, plus the feature flag name for a progressive rollout. - AI: the token usage attributes (input and output) from the GenAI semantic conventions, a conversation identifier and a feature attribute on each model call.
Turning a customer promise into an SLO
Section titled “Turning a customer promise into an SLO”A customer promise is like the timetable posted for a train line. The punctuality measured every day is the SLI (service level indicator): for an online store, for example, the share of payments that succeed in under 2 seconds. The SLO (service level objective) is the internal target you set on that indicator. The SLA (service level agreement) is the contractual commitment, with penalties if it is not met, like the refund promised to passengers when a train is late. The SLOs and alerting article covers them in detail.
I recommend working out the wording with the engineering team in four steps:
- Start from a customer sentence. For example, as an illustration: “A customer who pays sees the confirmation in under 2 seconds.”
- Ask what is measurable. The team says where it can measure and what the measurement does not see.
- Look at the current level before setting the target. A target chosen without measurement is either untenable or pointless.
- Write the final sentence. “99.9% of payments succeed in under 2 seconds, over a rolling 30 days” (illustrative example).
Aiming for 99.9% means accepting 0.1% failures. This tolerated share is called the error budget. Over 30 days, it amounts to about 43 minutes of full outage (computed value: 0.1% of 43,200 minutes).
Think of it as the contingency line in a project budget. While some remains, the team can ship new features and take risks. When it is used up, priority shifts to reliability. Google’s Site Reliability Workbook publishes an example error budget policy, a short document that writes this rule down in black and white. I recommend that the product owner sign it with the engineering team, before the first incident. The error budget simulator helps discuss it on a concrete case.
Your place in incidents and post-mortems
Section titled “Your place in incidents and post-mortems”During the incident, expect a single point of contact and regular updates, even when nothing has moved. Avoid contacting the people doing the repair directly: every call interrupts them. Bring what they cannot see instead. A key customer is affected, a commercial deadline is coming, a communication needs preparing.
After the incident comes the post-mortem, the meeting that analyzes what happened to draw actions from it, without looking for someone to blame. The template in The blameless post-mortem includes a business impact established with the business concerned. Your presence is therefore not a courtesy. Google’s chapter on post-mortem culture also states that any stakeholder can request one.
I recommend coming with three things: what customers experienced, what the incident cost in your view and what you would have needed to know sooner. The last point often becomes an instrumentation action. At Monday’s committee, it is the one that would have put the customer segment on traces.
AI features: quality and cost, not only uptime
Section titled “AI features: quality and cost, not only uptime”An AI assistant can respond quickly, without a single technical error yet still respond wrongly. Every light is green while the customer leaves with bad information. Uptime is therefore not enough. Chapter 1 of the AI observability guide adds two signals to the classic ones. The first is evaluations, which score the quality of answers. The second is user feedback.
For any AI feature, I recommend tracking quality, meaning the share of answers judged correct, with a numeric objective: a quality SLO. Also track cost per use, expressed in a unit that makes sense to your business (the conversation, the document processed or the customer). Finally, look at the share of answers that users flag as wrong or unhelpful.
The quality SLO simulator shows how to set an objective on quality. Chapter 3 of the guide helps locate your current level. To put a figure on an undetected degradation, see the GenAI method, part V and its exposure calculator.
Questions to ask your engineering team
Section titled “Questions to ask your engineering team”This table fits on one page and can serve as a meeting handout. The last column tells you how to check, without technical knowledge, that the answer is in place.
| Question | What to ask the team for | Sign that it is in place |
|---|---|---|
| How many customers did an incident affect? | a segment or a pseudonymized customer identifier on traces | the figure is available the next day, without a manual query |
| Where does the journey stall? | a step attribute and the key business events | a dashboard per step that you open yourself |
| Is the new version slower? | the version and the feature flag on every piece of data | a before and after comparison at every release |
| How much does AI cost per use? | tokens and a conversation identifier on each call | a cost per conversation tracked every month |
| Does the AI answer correctly? | evaluations and collection of user feedback | a quality SLO with an error budget |
| Are we keeping the customer promise? | one SLO per important journey | a visible error budget and a signed policy |
| Who decides when the budget is exhausted? | a written error budget policy | a name next to each decision |
| Will I be involved in post-mortems? | a systematic invitation when customers are affected | an impact section filled in with you |
If you do only one thing this week, I suggest picking the question you lacked an answer to during the last incident and asking it of your team, with this table in front of you.
Going further
Section titled “Going further”- Bridges: the questions from other departments and the translation dictionary.
- Observability in 10 minutes: the basic vocabulary on a slow payment example.
- SLOs and alerting: SLI, SLO, SLA and error budget in detail.
- The blameless post-mortem: the full process and template.
- The executive dashboard: an example of indicators presented to leadership.
- The maturity self-assessment: locate your organization.
- The MTTR business case: what faster detection is worth.
- The AI observability guide: chapters 1 and 3 to start.
Sources
Section titled “Sources”- Beyer, Jones, Petoff and Murphy (eds.), Site Reliability Engineering, O’Reilly, 2016: chapter 4, Service Level Objectives and chapter 15, Post-mortem Culture.
- Beyer, Murphy, Rensin, Kawahara and Thorne (eds.), The Site Reliability Workbook, O’Reilly, 2018: chapter 2, Implementing SLOs and the appendix Example Error Budget Policy.
- OpenTelemetry, semantic conventions for service resources (
service.versionattribute) and semantic conventions for generative AI (token usage). - The 43-minute budget is computed from a 30-day period, that is 43,200 minutes.