Skip to content

BusinessPractitioner2 min read

How do you compare solutions fairly?

Compare up to three observability solutions on six weighted criteria, with the same requirements for SaaS, open source and hybrid, then spot eliminating scores.

For: architects · finance, risk and compliance · executives and CIOsPrerequisites: None.

Reading mode

An illustrative scenario: after a very slick demo, part of the committee wants to sign with a vendor. The architect prefers an open source solution she knows well. Procurement asks which criteria the comparison rests on. Nobody has written them down. The discussion turns into a clash of tastes.

Choosing an observability solution is a commitment for several years. It is a bit like hiring. A good panel writes its interview grid before meeting the candidates, decides in advance which requirements are non-negotiable and judges on verifiable achievements rather than on how well someone talks. This scorecard applies the same discipline. It compares up to three options on six criteria, with the same standard for a SaaS offer (an online service run by the provider), an open source solution you run yourself or a hybrid architecture.

It helps settle one question: which option to pick, based on what evidence? It deliberately starts with identical scores. No real solution is prefilled: it is up to you to score them, evidence in hand.

Option names
CriterionWeightOption AOption BOption C
Portabilityingests and exports data in OpenTelemetry, open formats
Exit costmigrating history, dashboards, alerts; clause in the contract
Data sovereigntylocation, applicable law, known subprocessors
Cost predictability12- and 36-month simulations on your volumes, caps
Support and service levelsguaranteed response times, named contact, active community
EcosystemCI/CD, ITSM, security integrations; open API

Scores: 1 weak, 2 fair, 3 good, 4 excellent.

Result

Score from evidence (proof of concept on your data, contract, written answer to a requirements document) and keep the source of each score. Apply the same grid to a SaaS offer, a self-managed open source solution and a hybrid one. Nothing is sent: the grid stays in this browser.

  1. Name your options and set the weights before scoring. Weights set after the fact mostly serve to justify a choice already made.
  2. Give an option a score of 1 on data sovereignty. If the criterion’s weight reaches the eliminating threshold, the option is set aside, even with the best score. That is the equivalent of the non-negotiable requirement in hiring.
  3. Bring two options within five points of each other. The scorecard refuses to name a winner: the proof of concept, a trial on your own data, is what decides.
  4. Set portability to 0, meaning how easy it is to leave the solution one day. You will see how much the ranking depends on it. In my view, it is the criterion that conditions all the others in the long run.

I consider that the factor that weighs most on exit cost is not the commercial model but the instrumentation, meaning the code that produces data inside your applications. Collecting with OpenTelemetry, an open standard, lets you change storage backends without rewriting that code. This is an opinion, not a measurement; your costed exit plan will confirm it or not for your case. Whatever the option, require a proof of concept on your own data plus a costed exit plan.

Going further: the governance lesson of the CIO path, the FinOps lesson and the business dimension.