TechnicalPractitioner10 min read
Why VictoriaMetrics, architecture and deployment mode
What makes the metrics of an LLM application unusual, how VictoriaMetrics compares with Prometheus, Thanos and Mimir, the full architecture with OpenTelemetry and the choice between Single and Cluster mode.
For: engineers and SREs · architectsPrerequisites: Prometheus basics (metric types, PromQL).
Module 1: why VictoriaMetrics for LLM workloads
Section titled “Module 1: why VictoriaMetrics for LLM workloads”Picture a platform team hosting an internal assistant. The scenario is illustrative. At first a single Prometheus is enough: a few services, a handful of metrics. Then the assistant opens up to six business teams, who use three models from two providers. Finance asks for cost per team and per month over the whole fiscal year. Six months later the Prometheus local disk is full, memory climbs with every new internal customer and nobody knows where to keep two years of cost history.
This module explains why that scenario arrives quickly with LLM workloads. It then compares the storage options that can handle it.
1.1 The specific challenge of LLM metrics
Section titled “1.1 The specific challenge of LLM metrics”A metric reads like a tracking sheet: each combination of labels (the attributes that qualify a measurement, such as the model or the customer) opens a row called a time series. The number of series is the cardinality; the page Cardinality explained walks through it with that tracking-sheet image.
LLM applications in production fill that sheet in an unusual way, one that pushes classic monitoring backends to their limits.
| LLM characteristic | Effect on the metrics backend |
|---|---|
| High cardinality | model x provider x tenant x pipeline quickly yields thousands of time series and every histogram bucket multiplies that number. |
| Token histograms | Wide distributions (10 to 100k tokens per request), with very spread-out buckets. |
| USD cost metrics | Financial metrics requiring float precision and aggregations by accounting period. |
| Long retention | Cost audits and contractual SLAs: one to three years of retention may be required. |
| Ingestion bursts | A completion batch can multiply the normal write rate within seconds; the size of the burst depends on the batch size and the scrape interval. |
| RAG quality metrics | Floats, distributions, scores: write-intensive and frequently read for alerting. |
A tenant here means an internal customer kept apart from the others (a team, a subsidiary). A histogram sorts each measurement into ranges (“under 100 tokens”, “under 1,000”…) and every range is one more series. RAG (retrieval-augmented generation) is the pattern where the application looks up documents before calling the model.
1.2 VictoriaMetrics compared with Prometheus, Thanos and Mimir
Section titled “1.2 VictoriaMetrics compared with Prometheus, Thanos and Mimir”Prometheus on its own keeps its data on the local disk of a single machine. To keep history for a long time, survive the loss of an instance or separate several customers (multi-tenancy), it is usually paired with remote storage: Thanos, Grafana Mimir (derived from Cortex) or VictoriaMetrics. The useful comparison therefore sets these options against each other, on criteria you can check.
| Criterion | Prometheus alone | Prometheus + Thanos | Grafana Mimir | VictoriaMetrics |
|---|---|---|---|---|
| Deployment | Single binary | Prometheus plus several Thanos components (sidecar or receive, querier, store gateway, compactor) | Microservices, or monolithic mode in one binary | Single binary (Single), or three components (vminsert, vmselect, vmstorage) in Cluster |
| Storage and long retention | Local disk, not replicated (default retention: 15 days) | Object storage (S3, GCS, Azure…); downsampling by the compactor | Object storage | Local or block disk, retention per instance (-retentionPeriod); backups to object storage with vmbackup; downsampling restricted to the Enterprise edition |
| High availability | Two identical instances, no native deduplication | Deduplication at query time | Replication at ingestion | Two instances fed by vmagent (Single); replication factor in Cluster |
| Multi-tenancy | No | Yes, with Receive | Native | Native in Cluster (accountID) |
| Query language | PromQL | PromQL | PromQL | MetricsQL, largely PromQL-compatible, with extensions (median_over_time, rollup_candlestick, outliers_mad…) |
| OTLP ingestion | OTLP receiver (experimental since 2.47, built into Prometheus 3.x) | Through Prometheus or a Collector | Native OTLP endpoint | OTLP endpoint on the HTTP port: /opentelemetry/v1/metrics on port 8428 (Single), /insert/<accountID>/opentelemetry/v1/metrics on vminsert (Cluster) |
| License | Apache 2.0 | Apache 2.0 | AGPL 3.0 | Apache 2.0 (Single and Cluster); Enterprise edition under a commercial license (retention filters, downsampling, vmanomaly, vmauth mTLS…) |
| Compatibility | Reference | Prometheus query API | Prometheus query and write APIs | Prometheus query and write APIs: most existing dashboards work unchanged |
Downsampling means keeping only one point every five minutes or every hour for old data, instead of every sample. OTLP is the OpenTelemetry wire protocol.
Hosted offerings (Grafana Cloud, Amazon Managed Service for Prometheus, Google Cloud Managed Service for Prometheus, VictoriaMetrics’ cloud offering, among others) build on these engines or on compatible APIs. The question then becomes cost per series and the country where the data lives.
On the numbers, the Prometheus documentation states an average of 1 to 2 bytes per sample (prometheus.io, Storage, accessed 2 October 2026). According to the vendor, VictoriaMetrics uses “up to 7x less RAM” than Prometheus, Thanos or Cortex under high cardinality (millions of unique time series) and requires “up to 7x less storage space” than the same tools (VictoriaMetrics documentation, accessed 2 October 2026). Both figures come from a benchmark published by the vendor itself, comparing Prometheus 2.22.2 and VictoriaMetrics 1.47.0 on node_exporter metrics (vendor article). They are maxima obtained on one specific workload with old versions. I recommend measuring on your own workload, with the versions you plan to run, before sizing anything.
Module 2: complete VM + OTel architecture for LLM
Section titled “Module 2: complete VM + OTel architecture for LLM”2.1 Overview
Section titled “2.1 Overview”Think of the pipeline as a postal network. Applications drop off their mail, sorting centres receive and label it, a warehouse stores it, a service keeps checking that nothing unusual is arriving and counters present it to readers. The architecture covers that whole path, from LLM sources to dashboards and alerts. It can run in an environment fully cut off from the internet (air-gapped).
| Stage | Components | What happens |
|---|---|---|
| Sources | LLM application (OTel SDK), vLLM, Ollama, LiteLLM, DCGM exporter | Emit metrics and traces, or expose a /metrics endpoint |
| Collection | OTel Collector, vmagent | Receive OTLP, scrape Prometheus endpoints, enrich, derive metrics from traces |
| Storage | VictoriaMetrics (Single or Cluster) | Stores series, serves MetricsQL, hosts vmui |
| Rules | vmalert, optionally vmanomaly | Evaluates alerting and recording rules, writes results back |
| Consumption | Grafana, Alertmanager | Dashboards, alert routing |
In practice, instrumented sources send their data to the OTel Collector over OTLP, either gRPC on port 4317 or HTTP on port 4318. The Collector filters, adds Kubernetes labels, builds metrics out of traces with the spanmetrics connector, then exports. VictoriaMetrics receives the data through Prometheus remote write (Prometheus’ protocol for writing to remote storage) and also accepts OTLP directly over HTTP, in both Single and Cluster mode.
Downstream, vmalert evaluates the MetricsQL rules and sends alerts to Alertmanager. Grafana reads VictoriaMetrics as a datasource, through the VictoriaMetrics plugin or the standard Prometheus datasource. vmanomaly, an optional Enterprise component, adds anomaly detection on LLM metrics.
2.2 Components and roles
Section titled “2.2 Components and roles”| Component | Port(s) | Role for LLM observability |
|---|---|---|
| OTel Collector | 4317 (gRPC), 4318 (HTTP) | Normalizes and routes all metrics. The spanmetrics connector derives metrics from RAG traces. |
| VictoriaMetrics Single | 8428 | TSDB storage, MetricsQL API, remote write, direct OTLP, built-in vmui |
| vmagent | 8429 | Prometheus-style scraping. Suited to DCGM exporter, vLLM /metrics, LiteLLM |
| vmalert | 8880 | Evaluates MetricsQL alerting and recording rules, talks to Alertmanager |
| Alertmanager | 9093 | Alert routing (Slack, PagerDuty, email, webhook), deduplication and silencing |
| Grafana | 3000 | Dashboards, with the VictoriaMetrics plugin or the standard Prometheus datasource |
| vmanomaly (Enterprise) | see its documentation | Anomaly detection on LLM metrics using statistical and ML models. Optional. |
A TSDB (time series database) is a database built for timestamped values. vmui is the web interface built into VictoriaMetrics, where you type queries by hand.
Module 3: Single node vs Cluster, choosing your deployment mode
Section titled “Module 3: Single node vs Cluster, choosing your deployment mode”Single mode is like a small shop where the same person receives the goods, shelves them and serves customers. It is one binary that ingests, stores and queries. Cluster mode is a warehouse with three separate crews: vminsert receives (ingestion), vmstorage shelves (storage) and vmselect answers questions (queries). Each crew can grow and be doubled independently of the others.
3.1 Decision criteria for an LLM context
Section titled “3.1 Decision criteria for an LLM context”The vendor documentation recommends the single-node version for ingestion rates lower than about a million data points per second. It advises to “think twice” before choosing the cluster version, which is harder to configure and operate (cluster documentation, accessed 2 October 2026). The LLM workloads described in this guide usually stay far below that rate. The choice of Cluster then mostly comes down to high availability and multi-tenancy.
| Criterion | Single mode | Cluster mode |
|---|---|---|
| Ingestion rate | Recommended below about one million data points per second (vendor documentation) | Above that, or when one machine is no longer enough |
| High availability | Not native (two instances fed by vmagent, or external solutions) | Native: replicas per component, replication factor |
| Multi-tenancy | Not native (separate instances or vmauth) | Native (accountID in API paths) |
| Deployment | 1 container, 1 binary | Three component types to deploy |
| Retention | One global retention per instance | One global retention per vmstorage; per-tenant or per-series retention requires Enterprise (-retentionFilter) |
| Ops complexity | Low | Higher: several components to size and monitor |
| License | Open source (Apache 2.0) | Open source (Apache 2.0) |
When to move from Single to Cluster?
Section titled “When to move from Single to Cluster?”Back to the team from the opening. If its problem is a disk that is too small, a bigger machine and a properly set retention are often enough. Cluster becomes worthwhile when ingestion approaches a million data points per second, when one machine’s resources are no longer enough despite vertical scaling, or when an HA SLA or native isolation between tenants is required. Queries, the Prometheus-compatible API and Grafana dashboards stay the same after the move.