Skip to content

TechnicalBeginner5 min read

Observability in 10 minutes

Monitoring and observability, the signals (metrics, logs, traces, profiles, events), how to link them and what cardinality is, explained on a single example: a slow payment.

For: engineers and SREs · architects · team managers · business and productPrerequisites: None. Knowing what a web application is will do.

Reading mode

This page is for someone discovering the topic. It sets out the vocabulary you will find everywhere else on the site, with a single example from start to finish: a slow payment on an online shop.

Picture an online shop. The customer clicks “Pay”. Usually the confirmation shows up in under a second. This morning, some customers wait several seconds and a few abandon their cart.

The scenario is made up and deliberately ordinary: it is exactly the kind of problem observability should help solve.

Behind the “Pay” button, several programs work together. They are called services and each one does one specific job. Here, the checkout service receives the order and reads the cart from a database. It then asks the payment service to have the bank approve the payment.

flowchart LR
  C["Customer"] --> K["checkout"]
  K --> DB[("Cart database")]
  K --> P["payment"]
  P --> B["Bank API"]

The question is simple: why are some payments slow?

Monitoring is like the dashboard of a car. You picked a few warning lights in advance: the error rate, the response time. You set thresholds. When a threshold is crossed, an alert fires.

That works very well for planned questions. Is the service up? Is the response time above two seconds?

Our problem does not fit those boxes. Only some payments are slow. Maybe those of a specific bank, a country or a card type. Nobody had planned a warning light for that.

Observability is the ability to answer a question you had not planned, by querying the data already collected, without changing the code or waiting for the next release. It requires rich data that is linked together.

The two are not opposed. Monitoring tells you there is a problem; observability helps you understand which one.

An application can tell what it is doing in several ways. These accounts are called signals. Three are classic and two more complete the picture.

A metric is a number recorded at regular intervals, like an odometer you would read every minute: the number of payments, their average duration.

Each metric has a name and a few details called labels, for example the service concerned or the outcome (success, error). Metrics take little space and can be kept for a long time. They feed dashboards and alerts.

In our example, a chart shows that payment duration has been climbing since 9 am. It does not say why.

A trace is like parcel tracking: it tells the journey of one request from service to service, step by step, with the arrival and departure time at each stop. Each step is called a span.

The steps nest inside each other: the checkout step contains the payment step, which contains the call to the bank. You can see at a glance where the time goes.

POST /checkout (checkout) |=========================================| 4,200 ms
read cart (database) |=| 70 ms
authorize (payment) |=======================================| 3,950 ms
bank API call (bank) |=====================================| 3,850 ms

The durations are illustrative. The trace shows that the payment service is waiting for the bank. The problem is neither in the database nor in checkout.

A log is a dated line in a logbook that describes an event: “retrying connection to the bank, timeout exceeded”.

A log is far more helpful when it is written as fields (bank="bank-b", retry=2) rather than as free text: you can then filter and count it, like the columns of a spreadsheet.

In our example, the payment service logs show repeated retries towards a single bank. We have the likely cause.

A profile measures where a program spends its computing time and memory, function by function. It answers the question “which part of the code is expensive?”. In OpenTelemetry, profiles are still under development (signals documentation).

An event marks a one-off fact: a deployment, a configuration change, a switch to another server. Placed on a chart, it answers “what changed at 9 am?”.

SignalQuestion it answersIn the example
MetricIs there a problem, since when, how big?payment duration rising since 9 am
TraceWhere, in which service, at which step?time spent in the bank call
LogWhat exactly happened?repeated retries towards a single bank
ProfileWhich code consumes resources?useful if the service itself were slow
EventWhat changed?a payment deployment at 8:55 am

Taken separately, each signal gives one piece of the story. The value shows up when you can move from one to another in a click: from the rising chart to one slow request, then to the logs of that exact request.

To make this possible, every trace gets a unique number that travels with the request from service to service, like a parcel tracking number. If the logs carry the same number too, you find the log lines for the slow request in an instant. All signals must also refer to the service the same way, with the same name.

This is precisely what OpenTelemetry brings, the open standard presented in OpenTelemetry end to end: the same identifiers and the same names for every signal.

flowchart LR
  M["Metric<br/>payment duration"] -->|"exemplar"| T["Trace<br/>slow bank call"]
  T -->|"trace id"| L["Logs<br/>retries"]
  M -.->|"service.name"| L
For technical teams: trace id, exemplars and resource attributes

The trace number is called the trace id. It travels in an HTTP header standardized by the W3C (Trace Context). A metric can also point to a representative trace: this link is called an exemplar.

The shared service name goes through resource attributes, which describe who emits the signal: service.name, its version, the environment, the host. If metrics, logs and traces carry the same attributes, you filter them all the same way, for example service.name="payment".

A metric splits into rows, one per combination of labels: one row for bank A with success, another for bank A with error and so on. With 20 banks and 3 possible outcomes, you get at most 60 rows. Add the customer identifier with 100,000 customers and that number is multiplied by 100,000 (illustrative calculation). Storage, slowness and often the bill follow (Prometheus documentation).

The rule I recommend: a unique identifier (customer, order, request) goes into traces and logs, never into the labels of a metric. The page Cardinality explained tells this scenario in detail, with a video and a lab.

  • OpenTelemetry, Signals: traces, metrics, logs, baggage and the status of profiles and events.
  • W3C, Trace Context: format of the header that carries the trace identifier.
  • Prometheus, Metric and label naming: every label combination creates a new time series.