TechnicalBeginner5 min read
Observability in 10 minutes
Monitoring and observability, the signals (metrics, logs, traces, profiles, events), how to link them and what cardinality is, explained on a single example: a slow payment.
For: engineers and SREs · architects · team managers · business and productPrerequisites: None. Knowing what a web application is will do.
This page is for someone discovering the topic. It sets out the vocabulary you will find everywhere else on the site, with a single example from start to finish: a slow payment on an online shop.
A payment that drags
Section titled “A payment that drags”Picture an online shop. The customer clicks “Pay”. Usually the confirmation shows up in under a second. This morning, some customers wait several seconds and a few abandon their cart.
The scenario is made up and deliberately ordinary: it is exactly the kind of problem observability should help solve.
Behind the “Pay” button, several programs work together. They are called services and each one does one specific job. Here, the checkout service receives the order and reads the cart from a database. It then asks the payment service to have the bank approve the payment.
flowchart LR
C["Customer"] --> K["checkout"]
K --> DB[("Cart database")]
K --> P["payment"]
P --> B["Bank API"]
The question is simple: why are some payments slow?
Monitoring and observability
Section titled “Monitoring and observability”Monitoring is like the dashboard of a car. You picked a few warning lights in advance: the error rate, the response time. You set thresholds. When a threshold is crossed, an alert fires.
That works very well for planned questions. Is the service up? Is the response time above two seconds?
Our problem does not fit those boxes. Only some payments are slow. Maybe those of a specific bank, a country or a card type. Nobody had planned a warning light for that.
Observability is the ability to answer a question you had not planned, by querying the data already collected, without changing the code or waiting for the next release. It requires rich data that is linked together.
The two are not opposed. Monitoring tells you there is a problem; observability helps you understand which one.
The signals
Section titled “The signals”An application can tell what it is doing in several ways. These accounts are called signals. Three are classic and two more complete the picture.
Metrics: is there a problem?
Section titled “Metrics: is there a problem?”A metric is a number recorded at regular intervals, like an odometer you would read every minute: the number of payments, their average duration.
Each metric has a name and a few details called labels, for example the service concerned or the outcome (success, error). Metrics take little space and can be kept for a long time. They feed dashboards and alerts.
In our example, a chart shows that payment duration has been climbing since 9 am. It does not say why.
Traces: where is the time going?
Section titled “Traces: where is the time going?”A trace is like parcel tracking: it tells the journey of one request from service to service, step by step, with the arrival and departure time at each stop. Each step is called a span.
The steps nest inside each other: the checkout step contains the payment step, which contains the call to the bank. You can see at a glance where the time goes.
POST /checkout (checkout) |=========================================| 4,200 ms read cart (database) |=| 70 ms authorize (payment) |=======================================| 3,950 ms bank API call (bank) |=====================================| 3,850 msThe durations are illustrative. The trace shows that the payment service is waiting for the bank. The problem is neither in the database nor in checkout.
Logs: what exactly happened?
Section titled “Logs: what exactly happened?”A log is a dated line in a logbook that describes an event: “retrying connection to the bank, timeout exceeded”.
A log is far more helpful when it is written as fields (bank="bank-b", retry=2) rather than as free text: you can then filter and count it, like the columns of a spreadsheet.
In our example, the payment service logs show repeated retries towards a single bank. We have the likely cause.
Profiles and events
Section titled “Profiles and events”A profile measures where a program spends its computing time and memory, function by function. It answers the question “which part of the code is expensive?”. In OpenTelemetry, profiles are still under development (signals documentation).
An event marks a one-off fact: a deployment, a configuration change, a switch to another server. Placed on a chart, it answers “what changed at 9 am?”.
| Signal | Question it answers | In the example |
|---|---|---|
| Metric | Is there a problem, since when, how big? | payment duration rising since 9 am |
| Trace | Where, in which service, at which step? | time spent in the bank call |
| Log | What exactly happened? | repeated retries towards a single bank |
| Profile | Which code consumes resources? | useful if the service itself were slow |
| Event | What changed? | a payment deployment at 8:55 am |
Linking the signals
Section titled “Linking the signals”Taken separately, each signal gives one piece of the story. The value shows up when you can move from one to another in a click: from the rising chart to one slow request, then to the logs of that exact request.
To make this possible, every trace gets a unique number that travels with the request from service to service, like a parcel tracking number. If the logs carry the same number too, you find the log lines for the slow request in an instant. All signals must also refer to the service the same way, with the same name.
This is precisely what OpenTelemetry brings, the open standard presented in OpenTelemetry end to end: the same identifiers and the same names for every signal.
flowchart LR M["Metric<br/>payment duration"] -->|"exemplar"| T["Trace<br/>slow bank call"] T -->|"trace id"| L["Logs<br/>retries"] M -.->|"service.name"| L
For technical teams: trace id, exemplars and resource attributes
The trace number is called the trace id. It travels in an HTTP header standardized by the W3C (Trace Context). A metric can also point to a representative trace: this link is called an exemplar.
The shared service name goes through resource attributes, which describe who emits the signal: service.name, its version, the environment, the host. If metrics, logs and traces carry the same attributes, you filter them all the same way, for example service.name="payment".
Cardinality in a nutshell
Section titled “Cardinality in a nutshell”A metric splits into rows, one per combination of labels: one row for bank A with success, another for bank A with error and so on. With 20 banks and 3 possible outcomes, you get at most 60 rows. Add the customer identifier with 100,000 customers and that number is multiplied by 100,000 (illustrative calculation). Storage, slowness and often the bill follow (Prometheus documentation).
The rule I recommend: a unique identifier (customer, order, request) goes into traces and logs, never into the labels of a metric. The page Cardinality explained tells this scenario in detail, with a video and a lab.
Going further
Section titled “Going further”- OpenTelemetry end to end: follow a trace step from the application to storage.
- SLOs and alerting: decide when the slow payment deserves to wake someone up.
- Trace explorer: play with a real trace waterfall.
- The layers of observability and the clickable architecture: see where each signal is produced and stored.
- Cardinality simulator: measure the effect of a label on your own numbers.
- The glossary covers every term on this page.
Sources
Section titled “Sources”- OpenTelemetry, Signals: traces, metrics, logs, baggage and the status of profiles and events.
- W3C, Trace Context: format of the header that carries the trace identifier.
- Prometheus, Metric and label naming: every label combination creates a new time series.