Skip to content

TechnicalPractitioner10 min read

Why VictoriaMetrics, architecture and deployment mode

What makes the metrics of an LLM application unusual, how VictoriaMetrics compares with Prometheus, Thanos and Mimir, the full architecture with OpenTelemetry and the choice between Single and Cluster mode.

For: engineers and SREs · architectsPrerequisites: Prometheus basics (metric types, PromQL).

Module 1: why VictoriaMetrics for LLM workloads

Section titled “Module 1: why VictoriaMetrics for LLM workloads”

Picture a platform team hosting an internal assistant. The scenario is illustrative. At first a single Prometheus is enough: a few services, a handful of metrics. Then the assistant opens up to six business teams, who use three models from two providers. Finance asks for cost per team and per month over the whole fiscal year. Six months later the Prometheus local disk is full, memory climbs with every new internal customer and nobody knows where to keep two years of cost history.

This module explains why that scenario arrives quickly with LLM workloads. It then compares the storage options that can handle it.

A metric reads like a tracking sheet: each combination of labels (the attributes that qualify a measurement, such as the model or the customer) opens a row called a time series. The number of series is the cardinality; the page Cardinality explained walks through it with that tracking-sheet image.

LLM applications in production fill that sheet in an unusual way, one that pushes classic monitoring backends to their limits.

LLM characteristicEffect on the metrics backend
High cardinalitymodel x provider x tenant x pipeline quickly yields thousands of time series and every histogram bucket multiplies that number.
Token histogramsWide distributions (10 to 100k tokens per request), with very spread-out buckets.
USD cost metricsFinancial metrics requiring float precision and aggregations by accounting period.
Long retentionCost audits and contractual SLAs: one to three years of retention may be required.
Ingestion burstsA completion batch can multiply the normal write rate within seconds; the size of the burst depends on the batch size and the scrape interval.
RAG quality metricsFloats, distributions, scores: write-intensive and frequently read for alerting.

A tenant here means an internal customer kept apart from the others (a team, a subsidiary). A histogram sorts each measurement into ranges (“under 100 tokens”, “under 1,000”…) and every range is one more series. RAG (retrieval-augmented generation) is the pattern where the application looks up documents before calling the model.

1.2 VictoriaMetrics compared with Prometheus, Thanos and Mimir

Section titled “1.2 VictoriaMetrics compared with Prometheus, Thanos and Mimir”

Prometheus on its own keeps its data on the local disk of a single machine. To keep history for a long time, survive the loss of an instance or separate several customers (multi-tenancy), it is usually paired with remote storage: Thanos, Grafana Mimir (derived from Cortex) or VictoriaMetrics. The useful comparison therefore sets these options against each other, on criteria you can check.

CriterionPrometheus alonePrometheus + ThanosGrafana MimirVictoriaMetrics
DeploymentSingle binaryPrometheus plus several Thanos components (sidecar or receive, querier, store gateway, compactor)Microservices, or monolithic mode in one binarySingle binary (Single), or three components (vminsert, vmselect, vmstorage) in Cluster
Storage and long retentionLocal disk, not replicated (default retention: 15 days)Object storage (S3, GCS, Azure…); downsampling by the compactorObject storageLocal or block disk, retention per instance (-retentionPeriod); backups to object storage with vmbackup; downsampling restricted to the Enterprise edition
High availabilityTwo identical instances, no native deduplicationDeduplication at query timeReplication at ingestionTwo instances fed by vmagent (Single); replication factor in Cluster
Multi-tenancyNoYes, with ReceiveNativeNative in Cluster (accountID)
Query languagePromQLPromQLPromQLMetricsQL, largely PromQL-compatible, with extensions (median_over_time, rollup_candlestick, outliers_mad…)
OTLP ingestionOTLP receiver (experimental since 2.47, built into Prometheus 3.x)Through Prometheus or a CollectorNative OTLP endpointOTLP endpoint on the HTTP port: /opentelemetry/v1/metrics on port 8428 (Single), /insert/<accountID>/opentelemetry/v1/metrics on vminsert (Cluster)
LicenseApache 2.0Apache 2.0AGPL 3.0Apache 2.0 (Single and Cluster); Enterprise edition under a commercial license (retention filters, downsampling, vmanomaly, vmauth mTLS…)
CompatibilityReferencePrometheus query APIPrometheus query and write APIsPrometheus query and write APIs: most existing dashboards work unchanged

Downsampling means keeping only one point every five minutes or every hour for old data, instead of every sample. OTLP is the OpenTelemetry wire protocol.

Hosted offerings (Grafana Cloud, Amazon Managed Service for Prometheus, Google Cloud Managed Service for Prometheus, VictoriaMetrics’ cloud offering, among others) build on these engines or on compatible APIs. The question then becomes cost per series and the country where the data lives.

On the numbers, the Prometheus documentation states an average of 1 to 2 bytes per sample (prometheus.io, Storage, accessed 2 October 2026). According to the vendor, VictoriaMetrics uses “up to 7x less RAM” than Prometheus, Thanos or Cortex under high cardinality (millions of unique time series) and requires “up to 7x less storage space” than the same tools (VictoriaMetrics documentation, accessed 2 October 2026). Both figures come from a benchmark published by the vendor itself, comparing Prometheus 2.22.2 and VictoriaMetrics 1.47.0 on node_exporter metrics (vendor article). They are maxima obtained on one specific workload with old versions. I recommend measuring on your own workload, with the versions you plan to run, before sizing anything.

Module 2: complete VM + OTel architecture for LLM

Section titled “Module 2: complete VM + OTel architecture for LLM”

Think of the pipeline as a postal network. Applications drop off their mail, sorting centres receive and label it, a warehouse stores it, a service keeps checking that nothing unusual is arriving and counters present it to readers. The architecture covers that whole path, from LLM sources to dashboards and alerts. It can run in an environment fully cut off from the internet (air-gapped).

StageComponentsWhat happens
SourcesLLM application (OTel SDK), vLLM, Ollama, LiteLLM, DCGM exporterEmit metrics and traces, or expose a /metrics endpoint
CollectionOTel Collector, vmagentReceive OTLP, scrape Prometheus endpoints, enrich, derive metrics from traces
StorageVictoriaMetrics (Single or Cluster)Stores series, serves MetricsQL, hosts vmui
Rulesvmalert, optionally vmanomalyEvaluates alerting and recording rules, writes results back
ConsumptionGrafana, AlertmanagerDashboards, alert routing

In practice, instrumented sources send their data to the OTel Collector over OTLP, either gRPC on port 4317 or HTTP on port 4318. The Collector filters, adds Kubernetes labels, builds metrics out of traces with the spanmetrics connector, then exports. VictoriaMetrics receives the data through Prometheus remote write (Prometheus’ protocol for writing to remote storage) and also accepts OTLP directly over HTTP, in both Single and Cluster mode.

Downstream, vmalert evaluates the MetricsQL rules and sends alerts to Alertmanager. Grafana reads VictoriaMetrics as a datasource, through the VictoriaMetrics plugin or the standard Prometheus datasource. vmanomaly, an optional Enterprise component, adds anomaly detection on LLM metrics.

ComponentPort(s)Role for LLM observability
OTel Collector4317 (gRPC), 4318 (HTTP)Normalizes and routes all metrics. The spanmetrics connector derives metrics from RAG traces.
VictoriaMetrics Single8428TSDB storage, MetricsQL API, remote write, direct OTLP, built-in vmui
vmagent8429Prometheus-style scraping. Suited to DCGM exporter, vLLM /metrics, LiteLLM
vmalert8880Evaluates MetricsQL alerting and recording rules, talks to Alertmanager
Alertmanager9093Alert routing (Slack, PagerDuty, email, webhook), deduplication and silencing
Grafana3000Dashboards, with the VictoriaMetrics plugin or the standard Prometheus datasource
vmanomaly (Enterprise)see its documentationAnomaly detection on LLM metrics using statistical and ML models. Optional.

A TSDB (time series database) is a database built for timestamped values. vmui is the web interface built into VictoriaMetrics, where you type queries by hand.

Module 3: Single node vs Cluster, choosing your deployment mode

Section titled “Module 3: Single node vs Cluster, choosing your deployment mode”

Single mode is like a small shop where the same person receives the goods, shelves them and serves customers. It is one binary that ingests, stores and queries. Cluster mode is a warehouse with three separate crews: vminsert receives (ingestion), vmstorage shelves (storage) and vmselect answers questions (queries). Each crew can grow and be doubled independently of the others.

The vendor documentation recommends the single-node version for ingestion rates lower than about a million data points per second. It advises to “think twice” before choosing the cluster version, which is harder to configure and operate (cluster documentation, accessed 2 October 2026). The LLM workloads described in this guide usually stay far below that rate. The choice of Cluster then mostly comes down to high availability and multi-tenancy.

CriterionSingle modeCluster mode
Ingestion rateRecommended below about one million data points per second (vendor documentation)Above that, or when one machine is no longer enough
High availabilityNot native (two instances fed by vmagent, or external solutions)Native: replicas per component, replication factor
Multi-tenancyNot native (separate instances or vmauth)Native (accountID in API paths)
Deployment1 container, 1 binaryThree component types to deploy
RetentionOne global retention per instanceOne global retention per vmstorage; per-tenant or per-series retention requires Enterprise (-retentionFilter)
Ops complexityLowHigher: several components to size and monitor
LicenseOpen source (Apache 2.0)Open source (Apache 2.0)

Back to the team from the opening. If its problem is a disk that is too small, a bigger machine and a properly set retention are often enough. Cluster becomes worthwhile when ingestion approaches a million data points per second, when one machine’s resources are no longer enough despite vertical scaling, or when an HA SLA or native isolation between tenants is required. Queries, the Prometheus-compatible API and Grafana dashboards stay the same after the move.