PeopleBeginner7 min read
Which profiles should you hire for observability?
Five profiles that keep observability alive: observability engineer, SRE, platform engineer, AI observability, telemetry FinOps lead. Their missions, what sets them apart, the skills expected at each level and the right number of roles for the size of the team.
For: team managers · executives and CIOs · engineers and SREsPrerequisites: None. Technical terms are explained along the way.
Illustrative scenario. A company posts an opening for an “observability engineer”. The ad lists fifteen tools, from Prometheus to Kubernetes, with three vendors in between. Three months later, the person hired masters those tools but spends their days responding to incidents, because that is the team’s real problem: nobody owns the reliability of the payment service. The company was looking for an SRE and did not know it.
This page describes five profiles, what sets them apart and what is expected of them at each level. It opens a series on hiring: writing the job description, running the interview and the first 90 days. The job description generator puts it all together for a given context.
Start with the problem
Section titled “Start with the problem”Before choosing a title, write in one sentence the problem the new hire must solve. A few examples:
- “Our teams cannot tell why a request is slow: traces are missing and nobody knows how to read them.”
- “Our incidents last too long and nobody owns the reliability of payments.”
- “Each product team sets up its own monitoring; we pay for three tools and none of them is well kept.”
- “Our AI assistants answer badly on some days and we cannot measure it.”
- “The telemetry bill doubled in a year and nobody knows why.”
Each of these sentences calls for a different profile. The table below maps them.
Five profiles
Section titled “Five profiles”| Profile | The problem it solves | What it delivers | What it is not |
|---|---|---|---|
| Observability engineer | teams cannot see what happens in their services | instrumentation, the collection pipeline, reference dashboards and alerts, training for the teams | the person who answers every incident |
| SRE (site reliability engineer) | nobody owns the reliability of a service | reliability objectives (SLOs), an on-call rotation that holds, post-mortems that produce actions, automation of repetitive work | a renamed system administrator |
| Platform engineer | each team reinvents its tooling | a self-service internal platform (deployment, observability, security) run as a product | a team that does the others’ work for them |
| AI observability engineer | the quality of a model’s answers is not measured | tracing of model calls, quality evaluations, cost per request | a data scientist who trains models |
| Telemetry FinOps lead | the cost of observability drifts without explanation | cost per team and per signal, retention and sampling rules, a monthly report | a management controller |
SRE comes from Google’s book Site Reliability Engineering, which defines it as what happens when you ask software engineers to design an operations function. Platform engineering has its own definition in the CNCF white paper on platforms. The other three titles have no official reference: they describe needs that show up in job ads, with boundaries that vary from one company to the next.
Observability engineer
Section titled “Observability engineer”Makes services readable. Chooses and maintains the collection pipeline, often built around OpenTelemetry: SDKs in the applications, the Collector in the middle, storage at the end. Writes the naming conventions, the reference dashboards and the baseline alerts each team reuses. A large part of the job is teaching: showing teams how to instrument their code and read a trace.
The sign of success: product teams diagnose their own incidents without calling them.
Owns the reliability of one or more services. Negotiates a reliability objective with product (an SLO, for example “99.5% of payments succeed in under two seconds”) and the error budget that follows, the margin of failure the team allows itself. Holds or organizes on-call, runs post-mortems and reduces repetitive work. Observability is the main tool, not the deliverable.
The sign of success: incidents become rarer and shorter; the remaining ones teach the team something. See sustainable on-call and the blameless post-mortem.
Platform engineer
Section titled “Platform engineer”Builds the path product teams take to deploy and run their services: project templates, pipelines, environments, observability built in by default. Treats the platform as a product, with users (the developers), a roadmap and a measure of adoption.
The sign of success: a new service reaches production instrumented without anyone having to think about it.
AI observability engineer
Section titled “AI observability engineer”Measures what classic metrics miss: a wrong answer returned in 200 milliseconds with a success code. Traces calls to models (the OpenTelemetry GenAI conventions help, see observing an LLM, a RAG and MCP), sets up quality evaluations and tracks cost per request. Works with the teams that design the assistants and with those that run them.
The sign of success: a drop in quality is detected before users complain. The AI observability guide details the job.
Telemetry FinOps lead
Section titled “Telemetry FinOps lead”Makes the cost of observability readable and manageable. Ties the bill to teams and signals, spots the metrics that blow up (see who is growing the bill), sets retention and sampling rules. It is rarely a full-time role: often a mission given to an observability engineer or an SRE, together with the company’s FinOps team.
The sign of success: finance and engineers read the same monthly report and decide together.
Skills by level
Section titled “Skills by level”“Junior”, “mid-level” and “senior” mean very different things from one company to another. I recommend defining them by the scope of decisions rather than by years of experience.
| Area | Junior: applies | Mid-level: designs for the team | Senior: sets the rules for several teams |
|---|---|---|---|
| Instrumentation | adds metrics and spans following the conventions | chooses what to instrument in a service and writes the team’s conventions | defines shared conventions and gets them adopted |
| Collection and storage | deploys a Collector from a template | sizes the pipeline, tunes sampling and cardinality | weighs architectures, vendors and costs over several years |
| Alerts and SLOs | responds to an alert following the runbook | writes symptom-based alerts and SLOs with product | sets the company’s alerting and error budget policy |
| Incidents | takes part in the response, writes the timeline | coordinates an incident, runs the post-mortem | improves the incident process and trains coordinators |
| Diagnosis | reads a trace, a dashboard, logs | links the three signals to find a cause | diagnoses problems that cross several systems |
| Communication | explains what they did | trains the team, writes documentation others follow | convinces teams that do not report to them, talks to leadership and finance |
The last row matters as much as the others. An observability engineer who cannot get conventions adopted produces dashboards nobody opens.
One role or several?
Section titled “One role or several?”The number of roles depends on the size of the organization and on its team model. The guidelines below are indicative, to adjust to how critical the services are.
| Size of the engineering organization | What often works |
|---|---|
| fewer than 50 engineers | one role covering tooling and reliability, with on-call shared by the developers |
| 50 to 200 engineers | a small platform team that owns observability, SREs attached to the most critical services |
| more than 200 engineers | a separate observability team, SREs per domain, a telemetry FinOps lead, an AI observability profile if assistants are in production |
The most common trap is the single role that does everything: building the tooling, holding on-call for every service and answering finance’s questions. The urgent always wins; the tooling stops moving.
Hire or grow?
Section titled “Hire or grow?”The market for these profiles remains tight. Before opening a role, look at who could fill it internally. A system administrator who knows twenty years of the information system’s history, trained in OpenTelemetry and SLOs, often becomes a more useful SRE than an external hire. A developer who instrumented their own service is often the best possible observability engineer: they know the pain from the other side.
Lesson 5 of the CIO path describes the typical profiles of an operations team and the career paths that work for each.
Going further
Section titled “Going further”- How do you write an observability job description?: the structure, common mistakes and two templates.
- How do you run the interview?: questions, a hands-on exercise and an evaluation grid.
- How do you get the first 90 days right?: onboarding the new hire.
- The job description generator: a job description and an interview plan tailored to your context.