Skip to content
MTTL, Mean Time To Learn

TechnicalPractitioner

Clickable architecture

For: engineers and SREs · architectsPrerequisites: Basic notions of metrics, traces and logs.

Reading mode

An observability stack reads like a transport network: producers (the applications), a sorting yard (the OpenTelemetry Collector), warehouses specialized by signal type, and counters where people look and get alerted. The stack below is an example, not a default choice: each card names equivalent alternatives, open source or commercial, and the Collector lets you move from one to another. All its components are open source except Phoenix (Elastic License 2.0, source available), and all can run on sovereign infrastructure. Every component is clickable.

MetricTraceLogDashed: indirect link (scores pushed back as metrics).
PlatformMLSecurityProduct, application teamTypical split, to adapt: what matters is that each component carries a name.

Click a component, or reach it with Tab then press Enter, to open its card.

OTLPOn-callApplicationOpenTelemetry SDKProductOpenTelemetry CollectorOTLP 4317 / 4318PlatformVictoriaMetricsmetrics, port 8428PlatformGrafana Tempotraces, port 3200PlatformVictoriaLogs or Lokilogs, ports 9428 / 3100SecurityPhoenix or Langfuseevaluation, port 6006MLGrafanadashboards, port 3000Platformvmalert + Alertmanageralerts, ports 8880 / 9093Platform

Try this

Section titled “Try this”
  1. Follow a signal: start the animation. The metric, the trace and the log share the first leg (OTLP to the Collector), then split toward three different databases and meet again in Grafana. GenAI spans also go to the evaluation tool.
  2. Open the Collector: its minimal configuration shows the pipelines (metrics, traces, logs, plus a traces/genai pipeline that keeps only GenAI spans for the evaluation tool) and their exporters. It is the only place to change when switching backends.
  3. Turn on “Who owns what”: the platform team owns most components, but logs often belong to security and evaluation to the ML team. Ask who owns the masking rules in the Collector, or the labels that blow up cardinality.
  4. Compare ports: open Phoenix or Langfuse, then Grafana. Two classic clashes appear (4317 with the Collector, 3000 with Grafana): check them before deploying on a single host.

What to remember

Section titled “What to remember”
  • Everything goes through the Collector. Applications know a single OTLP address; routing, masking and sampling are decided in one place.
  • Each component watches itself. The three metrics on each card belong in a pipeline dashboard and in alerts: silently losing spans is an incident.
  • One name per component. Technical architecture is not enough: without a named owner, trade-offs (retention, cardinality, thresholds) have no decision maker.

Go further: the VictoriaMetrics for LLMs training, the LLM observability training, the GenAI method architecture and the trace explorer to see what this stack carries.

Revised on 2 October 2026: configurations checked against current versions (otlp_grpc and otlp_http exporter names, Tempo 3 retention, GenAI span filter toward the evaluation tool), Phoenix license clarified and alternatives named for each storage component.