Skip to content

TechnicalPractitioner4 min read

VictoriaMetrics as an LLM observability backend

A practical guide to using VictoriaMetrics as the metrics backend for production LLM workloads: architecture, installation, MetricsQL, dashboards, alerts, SLOs and operations.

For: engineers and SREs · architectsPrerequisites: Prometheus basics (metric types, PromQL), Docker Compose and some Python.

Reading mode

An LLM assistant goes into production. Three questions land on the table almost at once: how much does it cost, is it fast enough, are its answers still good? Answering them over time takes a place to keep metrics for months or even years and to query them quickly. That is the job of a metrics backend, the database that stores time series.

This course completes the “Go to production” step: it covers that component of the production stack and comes after the labs and the deployment playbooks.

It is meant for SRE, platform and MLOps engineers as well as observability architects. You need basic Prometheus knowledge (metric types, PromQL), Docker Compose and some Python. The goal is to run VictoriaMetrics as a production metrics backend for LLM and RAG workloads, from first deployment to alerting and SLOs.

The guide covers a concrete setup: OpenTelemetry Collector and vmagent for collection, VictoriaMetrics for storage, MetricsQL for queries, Grafana for dashboards, vmalert and Alertmanager for alerts. The architecture can be deployed in an environment cut off from the internet (air-gapped).

This course takes up and corrects my guide “VictoriaMetrics as an LLM Observability Backend” (version 3). Callouts in the lessons flag the points corrected since that first version. The labs rely on the public VMLLM lab on the forge.

By the end of this guide, you will be able to:

  • deploy VictoriaMetrics in Single mode for an LLM development environment;
  • choose between Single and Cluster mode based on your actual LLM workload;
  • configure the OTel Collector to VictoriaMetrics pipeline for LLM metrics collection;
  • instrument a Python/FastAPI application to emit the right LLM metrics;
  • write MetricsQL queries for the main use cases (cost, quality, latency, GPU);
  • build a Grafana LLM dashboard with four functional rows and real-time alerts;
  • configure vmalert with production-grade alert rules for LLM degradations;
  • optimize retention, storage and recording rules to minimize infrastructure costs.
  1. Why VictoriaMetrics, architecture and deployment mode: the LLM metrics profile, comparison with Prometheus, Thanos and Mimir, the VM + OTel architecture, Single versus Cluster.
  2. Installation and configuration: Docker Compose stack, OTel Collector with the spanmetrics connector, vmagent scraping.
  3. Metrics and instrumentation: LLM metrics taxonomy, naming, cardinality, Python instrumentation.
  4. MetricsQL and dashboards: cost, quality, latency and GPU queries, Grafana provisioning and panels.
  5. Alerts, recording rules and SLOs: vmalert ruleset, Alertmanager routing, recording rules, error budget and burn rate.
  6. Operations: retention and tuning, security and air-gap, migration from Prometheus, illustrative scenarios, FAQ.
  7. Labs, checklist and cheat sheet: four practical labs, go-live checklist, MetricsQL cheat sheet, references.
  8. Glossary.

This course names its application metrics llm_* and rag_* and emits them with the Prometheus client. These are not the names of the OpenTelemetry GenAI semantic conventions, meaning the standard names OpenTelemetry proposes for generative AI applications. Their status and history are kept in the article, section 2. The quantities measured are the same; only the names and units differ. If your stack also receives gen_ai.* metrics from an OpenTelemetry instrumentation, keep a single source per quantity and do the translation in the Collector or in recording rules rather than in every dashboard. The course names stay unchanged: they match the lab code.

Course metricEquivalent in the GenAI conventionsNote
llm_requests_totalnumber of observations of the gen_ai.client.operation.duration histogramfailure status is carried by error.type
llm_request_duration_msgen_ai.client.operation.durationthe conventions measure in seconds
llm_ttft_msgen_ai.client.operation.time_to_first_chunk (client side) or gen_ai.server.time_to_first_token (server side)in seconds
llm_tokens_total{type}gen_ai.client.token.usage broken down by gen_ai.token.type; on the unreleased main branch of the GenAI repository, gen_ai.client.inference.usage.input_tokens and gen_ai.client.inference.usage.output_tokenssee the article, section 2.1
llm_errors_totalerror.type attribute on the duration metrics
llm_cost_usd_totalnone: cost is not part of the conventionsapplication metric, keep it
rag_*no metric; retrieval spans with gen_ai.data_source.id and gen_ai.retrieval.top_kapplication metrics
model, provider labelsgen_ai.request.model, gen_ai.provider.name attributes

Revised on 4 October 2026: page moved to MDX with the progression box, positioning within the production stack, mapping between the llm_* metrics and the GenAI conventions, links to the guide, the labs, the article and the method.