VictoriaMetrics as an LLM observability backend

Reading mode

EngineerDecision-maker

Where this content fits

1. 1. DiscoverAI in cross-sectionthe five layers, where to look
2. 2. UnderstandUnderstanding generative AI observabilityconcepts and definitions
3. 3. PracticeLLM observability: the labshands-on, on a self-hosted stack
4. 4. Go to productionPlaybooks, then VictoriaMetrics for LLMs (you are here)production stack, OpenTelemetry conventions, licenses
5. 5. MasterGenAI observability methodquality SLOs, quantified impact, compliance, RACI
6. 6. Go beyondBeyond the LLM: GPUs and model qualityGPU infrastructure, classic ML, statistical drift

This course completes the “Go to production” step: it covers one component of the production stack (the metrics backend) and comes after the labs and the deployment playbooks.

Audience: SRE, platform and MLOps engineers, observability architects. Prerequisites: basic Prometheus knowledge (metric types, PromQL), Docker Compose, some Python. Goal: run VictoriaMetrics as a production metrics backend for LLM and RAG workloads, from first deployment to alerting and SLOs.

This guide covers a concrete setup: OpenTelemetry Collector and vmagent for collection, VictoriaMetrics for storage, MetricsQL for queries, Grafana for dashboards, vmalert and Alertmanager for alerts. The architecture can be deployed in an air-gapped environment.

This course takes up and corrects my guide “VictoriaMetrics as an LLM Observability Backend” (version 3). Callouts in the lessons flag the points corrected since that first version. The labs rely on the public VMLLM lab on the forge.

Learning objectives

By the end of this guide, you will be able to:

• deploy VictoriaMetrics in Single mode for an LLM development environment;
• choose between Single and Cluster mode based on your actual LLM workload;
• configure the OTel Collector to VictoriaMetrics pipeline for LLM metrics collection;
• instrument a Python/FastAPI application to emit the right LLM metrics;
• write MetricsQL queries for the main use cases (cost, quality, latency, GPU);
• build a Grafana LLM dashboard with four functional rows and real-time alerts;
• configure vmalert with production-grade alert rules for LLM degradations;
• optimize retention, storage and recording rules to minimize infrastructure costs.

Contents

1. Why VictoriaMetrics, architecture and deployment mode: the LLM metrics profile, comparison with Prometheus, Thanos and Mimir, the VM + OTel architecture, Single versus Cluster.
2. Installation and configuration: Docker Compose stack, OTel Collector with the spanmetrics connector, vmagent scraping.
3. Metrics and instrumentation: LLM metrics taxonomy, naming, cardinality, Python instrumentation.
4. MetricsQL and dashboards: cost, quality, latency and GPU queries, Grafana provisioning and panels.
5. Alerts, recording rules and SLOs: vmalert ruleset, Alertmanager routing, recording rules, error budget and burn rate.
6. Operations: retention and tuning, security and air-gap, migration from Prometheus, illustrative scenarios, FAQ.
7. Labs, checklist and cheat sheet: four practical labs, go-live checklist, MetricsQL cheat sheet, references.
8. Glossary.

Flag notation

VictoriaMetrics components (victoria-metrics, vmagent, vmalert, vmauth) document their flags with a single dash, for example -retentionPeriod. A double dash also works. This guide uses the single dash throughout. vmctl is the exception: it is a separate command-line tool whose options are written with a double dash.

Metric names and GenAI conventions

This course names its application metrics llm_* and rag_* and emits them with the Prometheus client. These are not the names of the OpenTelemetry GenAI semantic conventions, whose status and history are kept in the article, section 2. The correspondence principle: the quantities are the same, the names and units differ. If your stack also receives gen_ai.* metrics from an OpenTelemetry instrumentation, keep a single source per quantity and do the translation in the Collector or in recording rules rather than in every dashboard. The course names stay unchanged: they match the lab code.

Course metric	Equivalent in the GenAI conventions	Note
llm_requests_total	number of observations of the gen_ai.client.operation.duration histogram	failure status is carried by error.type
llm_request_duration_ms	gen_ai.client.operation.duration	the conventions measure in seconds
llm_ttft_ms	gen_ai.client.operation.time_to_first_chunk (client side) or gen_ai.server.time_to_first_token (server side)	in seconds
llm_tokens_total{type}	gen_ai.client.token.usage broken down by gen_ai.token.type; on the unreleased main branch of the GenAI repository, gen_ai.client.inference.usage.input_tokens and gen_ai.client.inference.usage.output_tokens	see the article, section 2.1
llm_errors_total	error.type attribute on the duration metrics
llm_cost_usd_total	none: cost is not part of the conventions	application metric, keep it
rag_*	no metric; retrieval spans with gen_ai.data_source.id and gen_ai.retrieval.top_k	application metrics
model, provider labels	gen_ai.request.model, gen_ai.provider.name attributes

Going further

• Understanding generative AI observability: concepts and definitions.
• LLM observability: the labs: hands-on, on a self-hosted stack.
• Observing an LLM system: deployment playbooks and the OpenTelemetry GenAI conventions.
• GenAI observability method: quality SLOs, quantified impact, compliance.

Revised on 4 October 2026: page moved to MDX with the progression box, positioning within the production stack, mapping between the llm_* metrics and the GenAI conventions, links to the guide, the labs, the article and the method.

BridgesWhat this page changes beyond the technical team.

BusinessValue, cost, risk

Cost metrics, kept for several years, feed internal chargeback and AI spending audits.

OrganizationWho decides, who owns

Single node versus cluster, retention and multi-tenant access are governance decisions as much as technical ones.

PeopleTeams, on-call, skills

Alert rules with runbooks and justified thresholds, rather than a wall of thresholds, protect on-call.

Business linesProduct, finance, risk, customer

Read-only dashboards for leadership and finance, built on the same data as operations.

Why bridges matter →